Re: [PATCH -v5 00/17] Rewrite x86/ftrace to use text_poke (and more)

[Masami Hiramatsu <mhiramat@kernel.org>]

On Tue, 26 Nov 2019 17:58:12 +0900
Masami Hiramatsu <mhiramat@kernel.org> wrote:
> Oops, I've tested a bit older kernel (with above change it seems to be fixed).
> I'll check the latest -tip.

Oops, it is reproduced.

+ wc -l
+ L=256
+ [ 256 -ne 256 ]
+ echo 1
[   75.852865] BUG: kernel NULL pointer dereference, address: 0000000000000000
[   75.855004] #PF: supervisor read access in kernel mode
[   75.856579] #PF: error_code(0x0000) - not-present page
[   75.858541] PGD 800000007bf60067 P4D 800000007bf60067 PUD 7bf5f067 PMD 0 
[   75.861420] Oops: 0000 [#1] PREEMPT SMP PTI
[   75.863008] CPU: 6 PID: 0 Comm: swapper/6 Not tainted 5.4.0-rc8+ #23
[   75.864881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014
[   75.870225] RIP: 0010:poke_int3_handler+0x39/0x100
[   75.871672] Code: 5b 5d c3 f6 87 88 00 00 00 03 75 f2 48 8b 87 80 00 00 00 48 89 fb 48 8d 68 ff 48 8b 05 80 98 72 01 83 fa 01 0f 8f 93 00 00 00 <48> 63 10 48 81 c2 00 00 00 81 48 39 d5 75 c5 0f b6 50 08 8d 4a 34
[   75.877927] RSP: 0018:ffffc900001a8eb8 EFLAGS: 00010046
[   75.879794] RAX: 0000000000000000 RBX: ffffc900001a8ee8 RCX: ffffffff81a00b57
[   75.881717] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffc900001a8ee8
[   75.883632] RBP: ffffffff81027635 R08: 0000000000000000 R09: 0000000000000000
[   75.885494] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[   75.887360] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[   75.889299] FS:  0000000000000000(0000) GS:ffff88807d980000(0000) knlGS:0000000000000000
[   75.891668] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   75.893657] CR2: 0000000000000000 CR3: 000000007a970000 CR4: 00000000000006a0
[   75.895307] Call Trace:
[   75.896126]  <IRQ>
[   75.897191]  do_int3+0xd/0xf0
[   75.898710]  int3+0x42/0x50
[   75.900306] RIP: 0010:sched_clock+0x6/0x10
[   75.901410] Code: d3 ea f6 c1 40 48 0f 45 c2 4c 01 c0 65 ff 0d 29 07 ff 7e 74 02 5d c3 e8 f0 a7 fd ff 5d c3 66 0f 1f 44 00 00 e8 ab f6 01 00 cc <90> c3 0f 1f 84 00 00 00 00 00 48 81 3d dd e5 21 01 c0 6f 02 81 0f
[   75.905389] RSP: 0018:ffffc900001a8fc8 EFLAGS: 00000016
[   75.906591] RAX: 00000011b208c72d RBX: 0000000000000000 RCX: 0000000000000000
[   75.908690] RDX: 0000000000000004 RSI: 0000000000000004 RDI: ffffffff8275d180
[   75.911009] RBP: 0000000000000000 R08: 0000002354a2e063 R09: 0000000000000001
[   75.912981] R10: 0000000000000000 R11: ffff88807d1a9f18 R12: 0000000000000000
[   75.914629] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[   75.916351]  ? sched_clock+0x6/0x10
[   75.917475]  ? sched_clock+0x5/0x10
[   75.918742]  sched_clock_cpu+0xe/0xd0
[   75.920304]  irq_exit+0xb3/0xc0
[   75.921634]  call_function_interrupt+0xf/0x20
[   75.922918]  </IRQ>
[   75.923987] RIP: 0010:default_idle+0x23/0x180
[   75.925386] Code: ff cc cc cc cc cc cc 41 55 41 54 55 53 e8 45 e8 c6 ff 0f 1f 44 00 00 e8 3b 25 95 ff e9 07 00 00 00 0f 00 2d 41 fd 5f 00 fb f4 <e8> 28 e8 c6 ff 89 c5 0f 1f 44 00 00 5b 5d 41 5c 41 5d c3 65 8b 05
[   75.931117] RSP: 0018:ffffc9000008feb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff03
[   75.933490] RAX: ffff88807d1a9680 RBX: 0000000000000006 RCX: 0000000000000000
[   75.935047] RDX: 0000000000000046 RSI: 0000000000000006 RDI: ffff88807d1a9680
[   75.936602] RBP: ffffffff8233dbc0 R08: 0000000000000001 R09: 0000000000000000
[   75.938136] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[   75.939726] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[   75.941784]  ? default_idle+0x15/0x180
[   75.943147]  default_idle_call+0x26/0x30
[   75.944539]  do_idle+0x1fe/0x220
[   75.945618]  ? trace_hardirqs_on+0x2c/0xf0
[   75.946699]  cpu_startup_entry+0x14/0x20
[   75.948054]  start_secondary+0x152/0x180
[   75.949257]  secondary_startup_64+0xb6/0xc0
[   75.950287] Modules linked in: [last unloaded: trace_printk]
[   75.951927] CR2: 0000000000000000
[   75.953002] ---[ end trace f347ec44ad8fffbb ]---
[   75.954322] RIP: 0010:poke_int3_handler+0x39/0x100
[   75.956041] Code: 5b 5d c3 f6 87 88 00 00 00 03 75 f2 48 8b 87 80 00 00 00 48 89 fb 48 8d 68 ff 48 8b 05 80 98 72 01 83 fa 01 0f 8f 93 00 00 00 <48> 63 10 48 81 c2 00 00 00 81 48 39 d5 75 c5 0f b6 50 08 8d 4a 34
[   75.960386] RSP: 0018:ffffc900001a8eb8 EFLAGS: 00010046
[   75.961904] RAX: 0000000000000000 RBX: ffffc900001a8ee8 RCX: ffffffff81a00b57
[   75.964994] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffc900001a8ee8
[   75.966836] RBP: ffffffff81027635 R08: 0000000000000000 R09: 0000000000000000
[   75.968460] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[   75.972194] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[   75.973849] FS:  0000000000000000(0000) GS:ffff88807d980000(0000) knlGS:0000000000000000
[   75.976592] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   75.978447] CR2: 0000000000000000 CR3: 000000007a970000 CR4: 00000000000006a0
[   75.980548] Kernel panic - not syncing: Fatal exception
[   75.982294] Kernel Offset: disabled
[   75.983427] ---[ end Kernel panic - not syncing: Fatal exception ]---
QEMU: Terminated
mhiramat@devnote2:~/ksrc/mincs$ cd work/linux/build-x86_64/
mhiramat@devnote2:~/ksrc/mincs/work/linux/build-x86_64$ eu-addr2line -e ./vmlinux poke_int3_handler+0x39
/home/mhiramat/ksrc/mincs/work/linux/linux/arch/x86/kernel/alternative.c:958:20

static inline void *text_poke_addr(struct text_poke_loc *tp)
{
        return _stext + tp->rel_addr; <------ Here is line #958
}

I applied following patch, but it seems not enough. While disabling 256 kprobes,
system was frozen (no BUG message).

Thank you,

diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 4552795a8df4..9505096e2cd1 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -1134,8 +1134,14 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries
 	 * sync_core() implies an smp_mb() and orders this store against
 	 * the writing of the new instruction.
 	 */
-	bp_patching.vec = NULL;
 	bp_patching.nr_entries = 0;
+	/*
+	 * This sync_core () ensures that all int3 handlers in progress
+	 * have finished. This allows poke_int3_handler () after this to
+	 * avoid touching bp_paching.vec by checking nr_entries == 0.
+	 */
+	text_poke_sync();
+	bp_patching.vec = NULL;
 }
 
 void text_poke_loc_init(struct text_poke_loc *tp, void *addr,


-- 
Masami Hiramatsu <mhiramat@kernel.org>