linux-rt-users.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* NMIs reported by console_blast.sh with 6.6.20-rt25
@ 2024-03-15 19:21 John B. Wyatt IV
  2024-03-22 12:29 ` Sebastian Andrzej Siewior
  0 siblings, 1 reply; 6+ messages in thread
From: John B. Wyatt IV @ 2024-03-15 19:21 UTC (permalink / raw)
  To: John Ogness
  Cc: Petr Mladek, Clark Williams, Juri Lelli, Derek Barbosa,
	Bruno Goncalves, John B. Wyatt IV, LKML, linux-rt-users

Hello John,

The real-time team at Red Hat is discussing backporting the rt patchset in 6.6
to RHEL 9/Stream 9. I decided to test v6.6.20-rt25 from stable-rt with
console_blast.sh. It reported similar NMIs from my testing of 6.7.0-rt6 with
that high cpu count server over uart 8250; which is expected since the patchset
is similar.

One interesting thing is that 6.7.0-rt6 fully preemptive + realtime tuned profile
did not return any NMIs while 6.6 did with that same configuration.

Another aspect I noticed during my testing. I did not set grub to
start with the realtime profile at boot for this machine. When I did set
it the second (and latter) NMI did not show for fully preemptive (the
3rd set at the bottom of this email). 

Caller info was enabled. No modifications to the source code were made.

I have not tested previous versions before 6.7.0-rt6 or 6.6.20-rt25;
with the exception of accidently testing 6.6.10-rt19. 6.6.10 also
reported NMIs during this test. If you wish to see these reports please
let me know.

Also, please let me know if I can help in any other way or if you want anything else
tested. I will be moving to 6.8 and using BUG_ON(system_state == SYSTEM_RUNNING);
in the scheduler next for my tests to see if the messages do not show.

Link to lscpu and previous NMI reports:
https://lore.kernel.org/linux-rt-users/ZdUmpblrzX0pcPjz@thinkpad2021/T/#mec258264be95dcba23970def7b248af0c6e942b2

-----------------------------
NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned realtime profile
-----------------------------

[ T2614] Kernel panic - not syncing: sysrq triggered crash
[   C56] NMI backtrace for cpu 56
[   C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_out (arch/x86/kernel/early_printk.c:105) 
[ C56] Code: 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 89 d0 0f b7 57 08 d3 e6 01 f2 ee <c3> cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90
All code
========
   0:	0f 1f 00             	nopl   (%rax)
   3:	90                   	nop
   4:	90                   	nop
   5:	90                   	nop
   6:	90                   	nop
   7:	90                   	nop
   8:	90                   	nop
   9:	90                   	nop
   a:	90                   	nop
   b:	90                   	nop
   c:	90                   	nop
   d:	90                   	nop
   e:	90                   	nop
   f:	90                   	nop
  10:	90                   	nop
  11:	90                   	nop
  12:	90                   	nop
  13:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
  18:	0f b6 8f c1 00 00 00 	movzbl 0xc1(%rdi),%ecx
  1f:	89 d0                	mov    %edx,%eax
  21:	0f b7 57 08          	movzwl 0x8(%rdi),%edx
  25:	d3 e6                	shl    %cl,%esi
  27:	01 f2                	add    %esi,%edx
  29:	ee                   	out    %al,(%dx)
  2a:*	c3                   	ret		<-- trapping instruction
  2b:	cc                   	int3
  2c:	cc                   	int3
  2d:	cc                   	int3
  2e:	cc                   	int3
  2f:	0f 1f 40 00          	nopl   0x0(%rax)
  33:	90                   	nop
  34:	90                   	nop
  35:	90                   	nop
  36:	90                   	nop
  37:	90                   	nop
  38:	90                   	nop
  39:	90                   	nop
  3a:	90                   	nop
  3b:	90                   	nop
  3c:	90                   	nop
  3d:	90                   	nop
  3e:	90                   	nop
  3f:	90                   	nop

Code starting with the faulting instruction
===========================================
   0:	c3                   	ret
   1:	cc                   	int3
   2:	cc                   	int3
   3:	cc                   	int3
   4:	cc                   	int3
   5:	0f 1f 40 00          	nopl   0x0(%rax)
   9:	90                   	nop
   a:	90                   	nop
   b:	90                   	nop
   c:	90                   	nop
   d:	90                   	nop
   e:	90                   	nop
   f:	90                   	nop
  10:	90                   	nop
  11:	90                   	nop
  12:	90                   	nop
  13:	90                   	nop
  14:	90                   	nop
  15:	90                   	nop
[   C56] RSP: 0018:ffa0000030a9fb28 EFLAGS: 00000002
[   C56] RAX: 000000000000005f RBX: ffffffffa6ac68c0 RCX: 0000000000000000
[   C56] RDX: 00000000000003f8 RSI: 0000000000000000 RDI: ffffffffa6ac68c0
[   C56] RBP: 000000000000005f R08: 303378302f623178 R09: 302b6d73615f6b72
[   C56] R10: 615f6b726f665f6d R11: 6f72665f74657220 R12: ffffffffa6ac68c0
[   C56] R13: ffffffffa69b9434 R14: 0000000000000000 R15: 0000000000000000
[   C56] FS:  00007f4a81d54740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000
[   C56] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   C56] CR2: 00007f4a81f30650 CR3: 000000011ea24004 CR4: 0000000000771ee0
[   C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[   C56] PKRU: 55555554
[   C56] Call Trace:
[   C56]  <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115) 
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1)) 
[ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149) 
[ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105) 
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347) 
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538) 
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458) 
[ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105) 
[ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105) 
[ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105) 
[   C56]  </NMI>
[   C56]  <TASK>
[ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347) 
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343) 
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134) 
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628) 
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940) 
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1)) 
[ C56] vprintk_emit (kernel/printk/printk.c:2414) 
[ C56] _printk (kernel/printk/printk.c:2474) 
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339) 
[ C56] ? _printk (kernel/printk/printk.c:2474) 
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154) 
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601) 
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165) 
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352) 
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) 
[ C56] vfs_write (fs/read_write.c:582) 
[ C56] ksys_write (fs/read_write.c:637) 
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81) 
[ C56] ? do_dup2 (fs/file.c:1142) 
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299) 
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88) 
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561) 
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) 
[   C56] RIP: 0033:0x7f4a81e5fc94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
   0:	c7 00 16 00 00 00    	movl   $0x16,(%rax)
   6:	b8 ff ff ff ff       	mov    $0xffffffff,%eax
   b:	c3                   	ret
   c:	66 2e 0f 1f 84 00 00 	cs nopw 0x0(%rax,%rax,1)
  13:	00 00 00 
  16:	f3 0f 1e fa          	endbr64
  1a:	80 3d d5 76 0d 00 00 	cmpb   $0x0,0xd76d5(%rip)        # 0xd76f6
  21:	74 13                	je     0x36
  23:	b8 01 00 00 00       	mov    $0x1,%eax
  28:	0f 05                	syscall
  2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
  30:	77 54                	ja     0x86
  32:	c3                   	ret
  33:	0f 1f 00             	nopl   (%rax)
  36:	55                   	push   %rbp
  37:	48 89 e5             	mov    %rsp,%rbp
  3a:	48 83 ec 20          	sub    $0x20,%rsp
  3e:	48                   	rex.W
  3f:	89                   	.byte 0x89

Code starting with the faulting instruction
===========================================
   0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
   6:	77 54                	ja     0x5c
   8:	c3                   	ret
   9:	0f 1f 00             	nopl   (%rax)
   c:	55                   	push   %rbp
   d:	48 89 e5             	mov    %rsp,%rbp
  10:	48 83 ec 20          	sub    $0x20,%rsp
  14:	48                   	rex.W
  15:	89                   	.byte 0x89
[   C56] RSP: 002b:00007ffc0e13b318 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[   C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4a81e5fc94
[   C56] RDX: 0000000000000002 RSI: 000055a865695490 RDI: 0000000000000001
[   C56] RBP: 00007ffc0e13b340 R08: 0000000000000410 R09: 0000000000000001
[   C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[   C56] R13: 000055a865695490 R14: 00007f4a81f305c0 R15: 00007f4a81f2df20
[   C56]  </TASK>
[   C56] NMI backtrace for cpu 56
[   C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_in (arch/x86/kernel/early_printk.c:100) 
[ C56] Code: cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 0f b7 57 08 d3 e6 01 f2 ec <0f> b6 c0 c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90
All code
========
   0:	cc                   	int3
   1:	0f 1f 40 00          	nopl   0x0(%rax)
   5:	90                   	nop
   6:	90                   	nop
   7:	90                   	nop
   8:	90                   	nop
   9:	90                   	nop
   a:	90                   	nop
   b:	90                   	nop
   c:	90                   	nop
   d:	90                   	nop
   e:	90                   	nop
   f:	90                   	nop
  10:	90                   	nop
  11:	90                   	nop
  12:	90                   	nop
  13:	90                   	nop
  14:	90                   	nop
  15:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
  1a:	0f b6 8f c1 00 00 00 	movzbl 0xc1(%rdi),%ecx
  21:	0f b7 57 08          	movzwl 0x8(%rdi),%edx
  25:	d3 e6                	shl    %cl,%esi
  27:	01 f2                	add    %esi,%edx
  29:	ec                   	in     (%dx),%al
  2a:*	0f b6 c0             	movzbl %al,%eax		<-- trapping instruction
  2d:	c3                   	ret
  2e:	cc                   	int3
  2f:	cc                   	int3
  30:	cc                   	int3
  31:	cc                   	int3
  32:	0f 1f 00             	nopl   (%rax)
  35:	90                   	nop
  36:	90                   	nop
  37:	90                   	nop
  38:	90                   	nop
  39:	90                   	nop
  3a:	90                   	nop
  3b:	90                   	nop
  3c:	90                   	nop
  3d:	90                   	nop
  3e:	90                   	nop
  3f:	90                   	nop

Code starting with the faulting instruction
===========================================
   0:	0f b6 c0             	movzbl %al,%eax
   3:	c3                   	ret
   4:	cc                   	int3
   5:	cc                   	int3
   6:	cc                   	int3
   7:	cc                   	int3
   8:	0f 1f 00             	nopl   (%rax)
   b:	90                   	nop
   c:	90                   	nop
   d:	90                   	nop
   e:	90                   	nop
   f:	90                   	nop
  10:	90                   	nop
  11:	90                   	nop
  12:	90                   	nop
  13:	90                   	nop
  14:	90                   	nop
  15:	90                   	nop
[   C56] RSP: 0018:ffa0000030a9fb00 EFLAGS: 00000002
[   C56] RAX: ffffffffa45cf400 RBX: 0000000000000000 RCX: 0000000000000000
[   C56] RDX: 00000000000003fd RSI: 0000000000000005 RDI: ffffffffa6ac68c0
[   C56] RBP: 0000000000002710 R08: 303578302f643278 R09: 302b6b726f665f6d
[   C56] R10: 302b6b726f665f6d R11: 6f72665f74657220 R12: 0000000000000020
[   C56] R13: ffffffffa6ac68c0 R14: 0000000000000000 R15: 0000000000000000
[   C56] FS:  00007f4a81d54740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000
[   C56] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   C56] CR2: 00007f4a81f30650 CR3: 000000011ea24004 CR4: 0000000000771ee0
[   C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[   C56] PKRU: 55555554
[   C56] Call Trace:
[   C56]  <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115) 
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1)) 
[ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149) 
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100) 
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347) 
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538) 
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458) 
[ C56] ? mem16_serial_out (drivers/tty/serial/8250/8250_port.c:382) 
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100) 
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100) 
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100) 
[   C56]  </NMI>
[   C56]  <TASK>
[ C56] wait_for_xmitr (drivers/tty/serial/8250/8250.h:117 drivers/tty/serial/8250/8250.h:139 drivers/tty/serial/8250/8250_port.c:2106 drivers/tty/serial/8250/8250_port.c:2124) 
[ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347) 
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343) 
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134) 
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628) 
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940) 
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1)) 
[ C56] vprintk_emit (kernel/printk/printk.c:2414) 
[ C56] _printk (kernel/printk/printk.c:2474) 
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339) 
[ C56] ? _printk (kernel/printk/printk.c:2474) 
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154) 
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601) 
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165) 
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352) 
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) 
[ C56] vfs_write (fs/read_write.c:582) 
[ C56] ksys_write (fs/read_write.c:637) 
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81) 
[ C56] ? do_dup2 (fs/file.c:1142) 
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299) 
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88) 
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561) 
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) 
[   C56] RIP: 0033:0x7f4a81e5fc94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
   0:	c7 00 16 00 00 00    	movl   $0x16,(%rax)
   6:	b8 ff ff ff ff       	mov    $0xffffffff,%eax
   b:	c3                   	ret
   c:	66 2e 0f 1f 84 00 00 	cs nopw 0x0(%rax,%rax,1)
  13:	00 00 00 
  16:	f3 0f 1e fa          	endbr64
  1a:	80 3d d5 76 0d 00 00 	cmpb   $0x0,0xd76d5(%rip)        # 0xd76f6
  21:	74 13                	je     0x36
  23:	b8 01 00 00 00       	mov    $0x1,%eax
  28:	0f 05                	syscall
  2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
  30:	77 54                	ja     0x86
  32:	c3                   	ret
  33:	0f 1f 00             	nopl   (%rax)
  36:	55                   	push   %rbp
  37:	48 89 e5             	mov    %rsp,%rbp
  3a:	48 83 ec 20          	sub    $0x20,%rsp
  3e:	48                   	rex.W
  3f:	89                   	.byte 0x89

Code starting with the faulting instruction
===========================================
   0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
   6:	77 54                	ja     0x5c
   8:	c3                   	ret
   9:	0f 1f 00             	nopl   (%rax)
   c:	55                   	push   %rbp
   d:	48 89 e5             	mov    %rsp,%rbp
  10:	48 83 ec 20          	sub    $0x20,%rsp
  14:	48                   	rex.W
  15:	89                   	.byte 0x89
[   C56] RSP: 002b:00007ffc0e13b318 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[   C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4a81e5fc94
[   C56] RDX: 0000000000000002 RSI: 000055a865695490 RDI: 0000000000000001
[   C56] RBP: 00007ffc0e13b340 R08: 0000000000000410 R09: 0000000000000001
[   C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[   C56] R13: 000055a865695490 R14: 00007f4a81f305c0 R15: 00007f4a81f2df20
[   C56]  </TASK>
[ T2614] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ T2614] Call Trace:
[ T2614]  <TASK>
[ T2614] dump_stack_lvl (lib/dump_stack.c:107) 
[ T2614] panic (kernel/panic.c:340) 
[ T2614] ? _printk (kernel/printk/printk.c:2474) 
[ T2614] sysrq_handle_crash (drivers/tty/sysrq.c:154) 
[ T2614] __handle_sysrq (drivers/tty/sysrq.c:601) 
[ T2614] write_sysrq_trigger (drivers/tty/sysrq.c:1165) 
[ T2614] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352) 
[ T2614] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) 
[ T2614] vfs_write (fs/read_write.c:582) 
[ T2614] ksys_write (fs/read_write.c:637) 
[ T2614] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81) 
[ T2614] ? do_dup2 (fs/file.c:1142) 
[ T2614] ? syscall_exit_to_user_mode (kernel/entry/common.c:299) 
[ T2614] ? do_syscall_64 (arch/x86/entry/common.c:88) 
[ T2614] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561) 
[ T2614] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) 
[ T2614] RIP: 0033:0x7f4a81e5fc94
[ T2614] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
   0:	c7 00 16 00 00 00    	movl   $0x16,(%rax)
   6:	b8 ff ff ff ff       	mov    $0xffffffff,%eax
   b:	c3                   	ret
   c:	66 2e 0f 1f 84 00 00 	cs nopw 0x0(%rax,%rax,1)
  13:	00 00 00 
  16:	f3 0f 1e fa          	endbr64
  1a:	80 3d d5 76 0d 00 00 	cmpb   $0x0,0xd76d5(%rip)        # 0xd76f6
  21:	74 13                	je     0x36
  23:	b8 01 00 00 00       	mov    $0x1,%eax
  28:	0f 05                	syscall
  2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
  30:	77 54                	ja     0x86
  32:	c3                   	ret
  33:	0f 1f 00             	nopl   (%rax)
  36:	55                   	push   %rbp
  37:	48 89 e5             	mov    %rsp,%rbp
  3a:	48 83 ec 20          	sub    $0x20,%rsp
  3e:	48                   	rex.W
  3f:	89                   	.byte 0x89

Code starting with the faulting instruction
===========================================
   0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
   6:	77 54                	ja     0x5c
   8:	c3                   	ret
   9:	0f 1f 00             	nopl   (%rax)
   c:	55                   	push   %rbp
   d:	48 89 e5             	mov    %rsp,%rbp
  10:	48 83 ec 20          	sub    $0x20,%rsp
  14:	48                   	rex.W
  15:	89                   	.byte 0x89
[ T2614] RSP: 002b:00007ffc0e13b318 EFLAGS: 00000202
[ T2614] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4a81e5fc94
[ T2614] RDX: 0000000000000002 RSI: 000055a865695490 RDI: 0000000000000001
[ T2614] RBP: 00007ffc0e13b340 R08: 0000000000000410 R09: 0000000000000001
[ T2614] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[ T2614] R13: 000055a865695490 R14: 00007f4a81f305c0 R15: 00007f4a81f2df20
[ T2614]  </TASK>
[ T2614] Shutting down cpus with NMI
[ T2614] Kernel Offset: disabled
[ T2614] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---
[disconnect]
^[]777;notify;Command completed;KRB5CCNAME=/dev/null console -M conserver-02.hosts.prod.psi.bos.redhat.com intel-eaglestream-spr-15.khw3.lab.eng.bos.redhat.com^[^[]777;precmd^[^[]0;j@thinkpad2021:~/BTWork/records/2023/12Dec-testprintkupstream/6.6.20-rt25^[^[]7;file://thinkpad2021/home/j/BTWork/records/2023/12Dec-testprintkupstream/6.6.20-rt25^[^[[?2004h^[[38;5;166mj^[[38;5;228m6.6.20-rt25^[[38;5;43m^[[38;5;71m
>$ ^[(B^[[mexit
^[[?2004l\r^[]777;preexec^[exit

Script done on 2024-03-08 12:25:38-05:00 [COMMAND_EXIT_CODE="0"]







-----------------------------
NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned throughput-performance profile
-----------------------------


[ T2546] sysrq: Trigger a crash
[ T2546] Kernel panic - not syncing: sysrq triggered crash
[   C56] NMI backtrace for cpu 56
[   C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_in (arch/x86/kernel/early_printk.c:100) 
[ C56] Code: cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 0f b7 57 08 d3 e6 01 f2 ec <0f> b6 c0 c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90
All code
========
   0:	cc                   	int3
   1:	0f 1f 40 00          	nopl   0x0(%rax)
   5:	90                   	nop
   6:	90                   	nop
   7:	90                   	nop
   8:	90                   	nop
   9:	90                   	nop
   a:	90                   	nop
   b:	90                   	nop
   c:	90                   	nop
   d:	90                   	nop
   e:	90                   	nop
   f:	90                   	nop
  10:	90                   	nop
  11:	90                   	nop
  12:	90                   	nop
  13:	90                   	nop
  14:	90                   	nop
  15:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
  1a:	0f b6 8f c1 00 00 00 	movzbl 0xc1(%rdi),%ecx
  21:	0f b7 57 08          	movzwl 0x8(%rdi),%edx
  25:	d3 e6                	shl    %cl,%esi
  27:	01 f2                	add    %esi,%edx
  29:	ec                   	in     (%dx),%al
  2a:*	0f b6 c0             	movzbl %al,%eax		<-- trapping instruction
  2d:	c3                   	ret
  2e:	cc                   	int3
  2f:	cc                   	int3
  30:	cc                   	int3
  31:	cc                   	int3
  32:	0f 1f 00             	nopl   (%rax)
  35:	90                   	nop
  36:	90                   	nop
  37:	90                   	nop
  38:	90                   	nop
  39:	90                   	nop
  3a:	90                   	nop
  3b:	90                   	nop
  3c:	90                   	nop
  3d:	90                   	nop
  3e:	90                   	nop
  3f:	90                   	nop

Code starting with the faulting instruction
===========================================
   0:	0f b6 c0             	movzbl %al,%eax
   3:	c3                   	ret
   4:	cc                   	int3
   5:	cc                   	int3
   6:	cc                   	int3
   7:	cc                   	int3
   8:	0f 1f 00             	nopl   (%rax)
   b:	90                   	nop
   c:	90                   	nop
   d:	90                   	nop
   e:	90                   	nop
   f:	90                   	nop
  10:	90                   	nop
  11:	90                   	nop
  12:	90                   	nop
  13:	90                   	nop
  14:	90                   	nop
  15:	90                   	nop
[   C56] RSP: 0018:ffa00000308e7ae8 EFLAGS: 00000002
[   C56] RAX: ffffffff8d7cf400 RBX: 0000000000000000 RCX: 0000000000000000
[   C56] RDX: 00000000000003fd RSI: 0000000000000005 RDI: ffffffff8fcc68c0
[   C56] RBP: 000000000000270f R08: 6469703e2d727275 R09: 2020202020202020
[   C56] R10: 00000000632e2020 R11: 00000000632e2020 R12: 0000000000000020
[   C56] R13: ffffffff8fcc68c0 R14: 0000000000000000 R15: 0000000000000000
[   C56] FS:  00007f4c17301740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000
[   C56] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   C56] CR2: 00007f4c174dd650 CR3: 000000010994c004 CR4: 0000000000771ee0
[   C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[   C56] PKRU: 55555554
[   C56] Call Trace:
[   C56]  <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115) 
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1)) 
[ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149) 
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100) 
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347) 
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538) 
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458) 
[ C56] ? mem16_serial_out (drivers/tty/serial/8250/8250_port.c:382) 
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100) 
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100) 
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100) 
[   C56]  </NMI>
[   C56]  <TASK>
[ C56] wait_for_xmitr (drivers/tty/serial/8250/8250.h:117 drivers/tty/serial/8250/8250.h:139 drivers/tty/serial/8250/8250_port.c:2106 drivers/tty/serial/8250/8250_port.c:2124) 
[ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347) 
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343) 
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134) 
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628) 
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940) 
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1)) 
[ C56] vprintk_emit (kernel/printk/printk.c:2414) 
[ C56] _printk (kernel/printk/printk.c:2474) 
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339) 
[ C56] ? _printk (kernel/printk/printk.c:2474) 
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154) 
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601) 
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165) 
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352) 
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) 
[ C56] vfs_write (fs/read_write.c:582) 
[ C56] ? __do_sys_newfstatat (fs/stat.c:466 (discriminator 1)) 
[ C56] ksys_write (fs/read_write.c:637) 
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81) 
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299) 
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88) 
[ C56] ? up_read (./arch/x86/include/asm/preempt.h:104 kernel/locking/rwsem.c:1354 kernel/locking/rwsem.c:1622) 
[ C56] ? do_user_addr_fault (arch/x86/mm/fault.c:1368) 
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561) 
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) 
[   C56] RIP: 0033:0x7f4c1740cc94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
   0:	c7 00 16 00 00 00    	movl   $0x16,(%rax)
   6:	b8 ff ff ff ff       	mov    $0xffffffff,%eax
   b:	c3                   	ret
   c:	66 2e 0f 1f 84 00 00 	cs nopw 0x0(%rax,%rax,1)
  13:	00 00 00 
  16:	f3 0f 1e fa          	endbr64
  1a:	80 3d d5 76 0d 00 00 	cmpb   $0x0,0xd76d5(%rip)        # 0xd76f6
  21:	74 13                	je     0x36
  23:	b8 01 00 00 00       	mov    $0x1,%eax
  28:	0f 05                	syscall
  2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
  30:	77 54                	ja     0x86
  32:	c3                   	ret
  33:	0f 1f 00             	nopl   (%rax)
  36:	55                   	push   %rbp
  37:	48 89 e5             	mov    %rsp,%rbp
  3a:	48 83 ec 20          	sub    $0x20,%rsp
  3e:	48                   	rex.W
  3f:	89                   	.byte 0x89

Code starting with the faulting instruction
===========================================
   0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
   6:	77 54                	ja     0x5c
   8:	c3                   	ret
   9:	0f 1f 00             	nopl   (%rax)
   c:	55                   	push   %rbp
   d:	48 89 e5             	mov    %rsp,%rbp
  10:	48 83 ec 20          	sub    $0x20,%rsp
  14:	48                   	rex.W
  15:	89                   	.byte 0x89
[   C56] RSP: 002b:00007ffe88b9b298 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[   C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4c1740cc94
[   C56] RDX: 0000000000000002 RSI: 00005634f890b490 RDI: 0000000000000001
[   C56] RBP: 00007ffe88b9b2c0 R08: 0000000000000410 R09: 0000000000000001
[   C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[   C56] R13: 00005634f890b490 R14: 00007f4c174dd5c0 R15: 00007f4c174daf20
[   C56]  </TASK>
[   C56] NMI backtrace for cpu 56
[   C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_in (arch/x86/kernel/early_printk.c:100) 
[ C56] Code: cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 0f b7 57 08 d3 e6 01 f2 ec <0f> b6 c0 c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90
All code
========
   0:	cc                   	int3
   1:	0f 1f 40 00          	nopl   0x0(%rax)
   5:	90                   	nop
   6:	90                   	nop
   7:	90                   	nop
   8:	90                   	nop
   9:	90                   	nop
   a:	90                   	nop
   b:	90                   	nop
   c:	90                   	nop
   d:	90                   	nop
   e:	90                   	nop
   f:	90                   	nop
  10:	90                   	nop
  11:	90                   	nop
  12:	90                   	nop
  13:	90                   	nop
  14:	90                   	nop
  15:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
  1a:	0f b6 8f c1 00 00 00 	movzbl 0xc1(%rdi),%ecx
  21:	0f b7 57 08          	movzwl 0x8(%rdi),%edx
  25:	d3 e6                	shl    %cl,%esi
  27:	01 f2                	add    %esi,%edx
  29:	ec                   	in     (%dx),%al
  2a:*	0f b6 c0             	movzbl %al,%eax		<-- trapping instruction
  2d:	c3                   	ret
  2e:	cc                   	int3
  2f:	cc                   	int3
  30:	cc                   	int3
  31:	cc                   	int3
  32:	0f 1f 00             	nopl   (%rax)
  35:	90                   	nop
  36:	90                   	nop
  37:	90                   	nop
  38:	90                   	nop
  39:	90                   	nop
  3a:	90                   	nop
  3b:	90                   	nop
  3c:	90                   	nop
  3d:	90                   	nop
  3e:	90                   	nop
  3f:	90                   	nop

Code starting with the faulting instruction
===========================================
   0:	0f b6 c0             	movzbl %al,%eax
   3:	c3                   	ret
   4:	cc                   	int3
   5:	cc                   	int3
   6:	cc                   	int3
   7:	cc                   	int3
   8:	0f 1f 00             	nopl   (%rax)
   b:	90                   	nop
   c:	90                   	nop
   d:	90                   	nop
   e:	90                   	nop
   f:	90                   	nop
  10:	90                   	nop
  11:	90                   	nop
  12:	90                   	nop
  13:	90                   	nop
  14:	90                   	nop
  15:	90                   	nop
[   C56] RSP: 0018:ffa00000308e7ae8 EFLAGS: 00000002
[   C56] RAX: ffffffff8d7cf400 RBX: 0000000000000000 RCX: 0000000000000000
[   C56] RDX: 00000000000003fd RSI: 0000000000000005 RDI: ffffffff8fcc68c0
[   C56] RBP: 000000000000270f R08: 64656c74746f7268 R09: 2020202020202020
[   C56] R10: 00000000742e2020 R11: 00000000742e2020 R12: 0000000000000020
[   C56] R13: ffffffff8fcc68c0 R14: 0000000000000000 R15: 0000000000000000
[   C56] FS:  00007f4c17301740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000
[   C56] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   C56] CR2: 00007f4c174dd650 CR3: 000000010994c004 CR4: 0000000000771ee0
[   C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[   C56] PKRU: 55555554
[   C56] Call Trace:
[   C56]  <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115) 
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1)) 
[ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149) 
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100) 
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347) 
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538) 
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458) 
[ C56] ? mem16_serial_out (drivers/tty/serial/8250/8250_port.c:382) 
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100) 
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100) 
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100) 
[   C56]  </NMI>
[   C56]  <TASK>
[ C56] wait_for_xmitr (drivers/tty/serial/8250/8250.h:117 drivers/tty/serial/8250/8250.h:139 drivers/tty/serial/8250/8250_port.c:2106 drivers/tty/serial/8250/8250_port.c:2124) 
[ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347) 
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343) 
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134) 
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628) 
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940) 
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1)) 
[ C56] vprintk_emit (kernel/printk/printk.c:2414) 
[ C56] _printk (kernel/printk/printk.c:2474) 
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339) 
[ C56] ? _printk (kernel/printk/printk.c:2474) 
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154) 
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601) 
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165) 
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352) 
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) 
[ C56] vfs_write (fs/read_write.c:582) 
[ C56] ? __do_sys_newfstatat (fs/stat.c:466 (discriminator 1)) 
[ C56] ksys_write (fs/read_write.c:637) 
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81) 
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299) 
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88) 
[ C56] ? up_read (./arch/x86/include/asm/preempt.h:104 kernel/locking/rwsem.c:1354 kernel/locking/rwsem.c:1622) 
[ C56] ? do_user_addr_fault (arch/x86/mm/fault.c:1368) 
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561) 
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) 
[   C56] RIP: 0033:0x7f4c1740cc94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
   0:	c7 00 16 00 00 00    	movl   $0x16,(%rax)
   6:	b8 ff ff ff ff       	mov    $0xffffffff,%eax
   b:	c3                   	ret
   c:	66 2e 0f 1f 84 00 00 	cs nopw 0x0(%rax,%rax,1)
  13:	00 00 00 
  16:	f3 0f 1e fa          	endbr64
  1a:	80 3d d5 76 0d 00 00 	cmpb   $0x0,0xd76d5(%rip)        # 0xd76f6
  21:	74 13                	je     0x36
  23:	b8 01 00 00 00       	mov    $0x1,%eax
  28:	0f 05                	syscall
  2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
  30:	77 54                	ja     0x86
  32:	c3                   	ret
  33:	0f 1f 00             	nopl   (%rax)
  36:	55                   	push   %rbp
  37:	48 89 e5             	mov    %rsp,%rbp
  3a:	48 83 ec 20          	sub    $0x20,%rsp
  3e:	48                   	rex.W
  3f:	89                   	.byte 0x89

Code starting with the faulting instruction
===========================================
   0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
   6:	77 54                	ja     0x5c
   8:	c3                   	ret
   9:	0f 1f 00             	nopl   (%rax)
   c:	55                   	push   %rbp
   d:	48 89 e5             	mov    %rsp,%rbp
  10:	48 83 ec 20          	sub    $0x20,%rsp
  14:	48                   	rex.W
  15:	89                   	.byte 0x89
[   C56] RSP: 002b:00007ffe88b9b298 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[   C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4c1740cc94
[   C56] RDX: 0000000000000002 RSI: 00005634f890b490 RDI: 0000000000000001
[   C56] RBP: 00007ffe88b9b2c0 R08: 0000000000000410 R09: 0000000000000001
[   C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[   C56] R13: 00005634f890b490 R14: 00007f4c174dd5c0 R15: 00007f4c174daf20
[   C56]  </TASK>
[ T2546] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ T2546] Call Trace:
[ T2546]  <TASK>
[ T2546] dump_stack_lvl (lib/dump_stack.c:107) 
[ T2546] panic (kernel/panic.c:340) 
[ T2546] ? _printk (kernel/printk/printk.c:2474) 
[ T2546] sysrq_handle_crash (drivers/tty/sysrq.c:154) 
[ T2546] __handle_sysrq (drivers/tty/sysrq.c:601) 
[ T2546] write_sysrq_trigger (drivers/tty/sysrq.c:1165) 
[ T2546] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352) 
[ T2546] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) 
[ T2546] vfs_write (fs/read_write.c:582) 
[ T2546] ? __do_sys_newfstatat (fs/stat.c:466 (discriminator 1)) 
[ T2546] ksys_write (fs/read_write.c:637) 
[ T2546] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81) 
[ T2546] ? syscall_exit_to_user_mode (kernel/entry/common.c:299) 
[ T2546] ? do_syscall_64 (arch/x86/entry/common.c:88) 
[ T2546] ? up_read (./arch/x86/include/asm/preempt.h:104 kernel/locking/rwsem.c:1354 kernel/locking/rwsem.c:1622) 
[ T2546] ? do_user_addr_fault (arch/x86/mm/fault.c:1368) 
[ T2546] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561) 
[ T2546] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) 
[ T2546] RIP: 0033:0x7f4c1740cc94
[ T2546] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
   0:	c7 00 16 00 00 00    	movl   $0x16,(%rax)
   6:	b8 ff ff ff ff       	mov    $0xffffffff,%eax
   b:	c3                   	ret
   c:	66 2e 0f 1f 84 00 00 	cs nopw 0x0(%rax,%rax,1)
  13:	00 00 00 
  16:	f3 0f 1e fa          	endbr64
  1a:	80 3d d5 76 0d 00 00 	cmpb   $0x0,0xd76d5(%rip)        # 0xd76f6
  21:	74 13                	je     0x36
  23:	b8 01 00 00 00       	mov    $0x1,%eax
  28:	0f 05                	syscall
  2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
  30:	77 54                	ja     0x86
  32:	c3                   	ret
  33:	0f 1f 00             	nopl   (%rax)
  36:	55                   	push   %rbp
  37:	48 89 e5             	mov    %rsp,%rbp
  3a:	48 83 ec 20          	sub    $0x20,%rsp
  3e:	48                   	rex.W
  3f:	89                   	.byte 0x89

Code starting with the faulting instruction
===========================================
   0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
   6:	77 54                	ja     0x5c
   8:	c3                   	ret
   9:	0f 1f 00             	nopl   (%rax)
   c:	55                   	push   %rbp
   d:	48 89 e5             	mov    %rsp,%rbp
  10:	48 83 ec 20          	sub    $0x20,%rsp
  14:	48                   	rex.W
  15:	89                   	.byte 0x89
[ T2546] RSP: 002b:00007ffe88b9b298 EFLAGS: 00000202
[ T2546] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4c1740cc94
[ T2546] RDX: 0000000000000002 RSI: 00005634f890b490 RDI: 0000000000000001
[ T2546] RBP: 00007ffe88b9b2c0 R08: 0000000000000410 R09: 0000000000000001
[ T2546] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[ T2546] R13: 00005634f890b490 R14: 00007f4c174dd5c0 R15: 00007f4c174daf20
[ T2546]  </TASK>
[ T2546] Shutting down cpus with NMI
[ T2546] Kernel Offset: disabled
[ T2546] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---





-----------------------------
NMI Backtrace for 6.6.20-rt25 fully preemptive with tuned realtime profile
-----------------------------

[ T3402] sysrq: Trigger a crash
[ T3406] 
[ T3402] Kernel panic - not syncing: sysrq triggered crash
[   C56] NMI backtrace for cpu 56
[   C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:delay_halt_tpause (arch/x86/lib/delay.c:118) 
[ C56] Code: cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 48 8d 04 37 31 c9 48 89 c2 48 c1 ea 20 66 0f ae f1 <c3> cc cc cc cc 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90
All code
========
   0:	cc                   	int3
   1:	0f 1f 00             	nopl   (%rax)
   4:	90                   	nop
   5:	90                   	nop
   6:	90                   	nop
   7:	90                   	nop
   8:	90                   	nop
   9:	90                   	nop
   a:	90                   	nop
   b:	90                   	nop
   c:	90                   	nop
   d:	90                   	nop
   e:	90                   	nop
   f:	90                   	nop
  10:	90                   	nop
  11:	90                   	nop
  12:	90                   	nop
  13:	90                   	nop
  14:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
  19:	48 8d 04 37          	lea    (%rdi,%rsi,1),%rax
  1d:	31 c9                	xor    %ecx,%ecx
  1f:	48 89 c2             	mov    %rax,%rdx
  22:	48 c1 ea 20          	shr    $0x20,%rdx
  26:	66 0f ae f1          	tpause %ecx
  2a:*	c3                   	ret		<-- trapping instruction
  2b:	cc                   	int3
  2c:	cc                   	int3
  2d:	cc                   	int3
  2e:	cc                   	int3
  2f:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
  34:	90                   	nop
  35:	90                   	nop
  36:	90                   	nop
  37:	90                   	nop
  38:	90                   	nop
  39:	90                   	nop
  3a:	90                   	nop
  3b:	90                   	nop
  3c:	90                   	nop
  3d:	90                   	nop
  3e:	90                   	nop
  3f:	90                   	nop

Code starting with the faulting instruction
===========================================
   0:	c3                   	ret
   1:	cc                   	int3
   2:	cc                   	int3
   3:	cc                   	int3
   4:	cc                   	int3
   5:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
   a:	90                   	nop
   b:	90                   	nop
   c:	90                   	nop
   d:	90                   	nop
   e:	90                   	nop
   f:	90                   	nop
  10:	90                   	nop
  11:	90                   	nop
  12:	90                   	nop
  13:	90                   	nop
  14:	90                   	nop
  15:	90                   	nop
[   C56] RSP: 0018:ffa0000039cfba80 EFLAGS: 00000002
[   C56] RAX: 00000099a649b2ae RBX: 00000000000006d5 RCX: 0000000000000000
[   C56] RDX: 0000000000000099 RSI: 00000000000006d5 RDI: 00000099a649abd9
[   C56] RBP: 00000099a649abd9 R08: 303178302f303178 R09: 302b646165726874
[   C56] R10: 6461657268746b5f R11: 7866705f5f203f20 R12: 0000000000000020
[   C56] R13: ffffffff9b8fd320 R14: 0000000000000000 R15: 0000000000000000
[   C56] FS:  00007f12ca52d740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000
[   C56] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   C56] CR2: 00007f12ca709650 CR3: 000000010fbe6003 CR4: 0000000000771ee0
[   C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[   C56] PKRU: 55555554
[   C56] Call Trace:
[   C56]  <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115) 
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1)) 
[ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149) 
[ C56] ? delay_halt_tpause (arch/x86/lib/delay.c:118) 
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347) 
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538) 
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458) 
[ C56] ? delay_halt_tpause (arch/x86/lib/delay.c:118) 
[ C56] ? delay_halt_tpause (arch/x86/lib/delay.c:118) 
[ C56] ? delay_halt_tpause (arch/x86/lib/delay.c:118) 
[   C56]  </NMI>
[   C56]  <TASK>
[ C56] delay_halt.part.0 (arch/x86/lib/delay.c:163) 
[ C56] wait_for_xmitr (./include/linux/nmi.h:147 drivers/tty/serial/8250/8250_port.c:2113 drivers/tty/serial/8250/8250_port.c:2124) 
[ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347) 
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343) 
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134) 
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628) 
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940) 
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1)) 
[ C56] vprintk_emit (kernel/printk/printk.c:2414) 
[ C56] _printk (kernel/printk/printk.c:2474) 
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339) 
[ C56] ? _printk (kernel/printk/printk.c:2474) 
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154) 
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601) 
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165) 
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352) 
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) 
[ C56] vfs_write (fs/read_write.c:582) 
[ C56] ksys_write (fs/read_write.c:637) 
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81) 
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299) 
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88) 
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88) 
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561) 
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) 
[   C56] RIP: 0033:0x7f12ca638c94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
   0:	c7 00 16 00 00 00    	movl   $0x16,(%rax)
   6:	b8 ff ff ff ff       	mov    $0xffffffff,%eax
   b:	c3                   	ret
   c:	66 2e 0f 1f 84 00 00 	cs nopw 0x0(%rax,%rax,1)
  13:	00 00 00 
  16:	f3 0f 1e fa          	endbr64
  1a:	80 3d d5 76 0d 00 00 	cmpb   $0x0,0xd76d5(%rip)        # 0xd76f6
  21:	74 13                	je     0x36
  23:	b8 01 00 00 00       	mov    $0x1,%eax
  28:	0f 05                	syscall
  2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
  30:	77 54                	ja     0x86
  32:	c3                   	ret
  33:	0f 1f 00             	nopl   (%rax)
  36:	55                   	push   %rbp
  37:	48 89 e5             	mov    %rsp,%rbp
  3a:	48 83 ec 20          	sub    $0x20,%rsp
  3e:	48                   	rex.W
  3f:	89                   	.byte 0x89

Code starting with the faulting instruction
===========================================
   0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
   6:	77 54                	ja     0x5c
   8:	c3                   	ret
   9:	0f 1f 00             	nopl   (%rax)
   c:	55                   	push   %rbp
   d:	48 89 e5             	mov    %rsp,%rbp
  10:	48 83 ec 20          	sub    $0x20,%rsp
  14:	48                   	rex.W
  15:	89                   	.byte 0x89
[   C56] RSP: 002b:00007ffce58c2ef8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[   C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f12ca638c94
[   C56] RDX: 0000000000000002 RSI: 000055b5493b8490 RDI: 0000000000000001
[   C56] RBP: 00007ffce58c2f20 R08: 0000000000000410 R09: 0000000000000001
[   C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[   C56] R13: 000055b5493b8490 R14: 00007f12ca7095c0 R15: 00007f12ca706f20
[   C56]  </TASK>
[   C56] NMI backtrace for cpu 56
[   C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:delay_halt.part.0 (arch/x86/lib/delay.c:169) 
[ C56] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 53 48 89 fb 0f 01 f9 66 90 48 c1 e2 20 48 09 c2 48 89 d5 eb 09 <48> 01 eb 48 89 c5 48 29 c3 48 8b 05 15 bd a0 00 48 89 de 48 89 ef
All code
========
   0:	90                   	nop
   1:	90                   	nop
   2:	90                   	nop
   3:	90                   	nop
   4:	90                   	nop
   5:	90                   	nop
   6:	90                   	nop
   7:	90                   	nop
   8:	90                   	nop
   9:	90                   	nop
   a:	90                   	nop
   b:	90                   	nop
   c:	90                   	nop
   d:	90                   	nop
   e:	90                   	nop
   f:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
  14:	55                   	push   %rbp
  15:	53                   	push   %rbx
  16:	48 89 fb             	mov    %rdi,%rbx
  19:	0f 01 f9             	rdtscp
  1c:	66 90                	xchg   %ax,%ax
  1e:	48 c1 e2 20          	shl    $0x20,%rdx
  22:	48 09 c2             	or     %rax,%rdx
  25:	48 89 d5             	mov    %rdx,%rbp
  28:	eb 09                	jmp    0x33
  2a:*	48 01 eb             	add    %rbp,%rbx		<-- trapping instruction
  2d:	48 89 c5             	mov    %rax,%rbp
  30:	48 29 c3             	sub    %rax,%rbx
  33:	48 8b 05 15 bd a0 00 	mov    0xa0bd15(%rip),%rax        # 0xa0bd4f
  3a:	48 89 de             	mov    %rbx,%rsi
  3d:	48 89 ef             	mov    %rbp,%rdi

Code starting with the faulting instruction
===========================================
   0:	48 01 eb             	add    %rbp,%rbx
   3:	48 89 c5             	mov    %rax,%rbp
   6:	48 29 c3             	sub    %rax,%rbx
   9:	48 8b 05 15 bd a0 00 	mov    0xa0bd15(%rip),%rax        # 0xa0bd25
  10:	48 89 de             	mov    %rbx,%rsi
  13:	48 89 ef             	mov    %rbp,%rdi
[   C56] RSP: 0018:ffa0000039cfba88 EFLAGS: 00000093
[   C56] RAX: 000000ec13deb29f RBX: 000000000000076d RCX: 0000000000004038
[   C56] RDX: 0000000000000050 RSI: 000000000000076d RDI: 000000ec13deb24f
[   C56] RBP: 000000ec13deb24f R08: 20203138312f7068 R09: 7570632020202020
[   C56] R10: 38312f7068757063 R11: 2020202020205320 R12: 0000000000000020
[   C56] R13: ffffffff9b8fd320 R14: 0000000000000000 R15: 0000000000000000
[   C56] FS:  00007f12ca52d740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000
[   C56] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   C56] CR2: 00007f12ca709650 CR3: 000000010fbe6003 CR4: 0000000000771ee0
[   C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[   C56] PKRU: 55555554
[   C56] Call Trace:
[   C56]  <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115) 
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1)) 
[ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149) 
[ C56] ? delay_halt.part.0 (arch/x86/lib/delay.c:169) 
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347) 
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538) 
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458) 
[ C56] ? delay_halt.part.0 (arch/x86/lib/delay.c:169) 
[ C56] ? delay_halt.part.0 (arch/x86/lib/delay.c:169) 
[ C56] ? delay_halt.part.0 (arch/x86/lib/delay.c:169) 
[   C56]  </NMI>
[   C56]  <TASK>
[ C56] wait_for_xmitr (./include/linux/nmi.h:147 drivers/tty/serial/8250/8250_port.c:2113 drivers/tty/serial/8250/8250_port.c:2124) 
[ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347) 
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343) 
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134) 
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628) 
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940) 
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1)) 
[ C56] vprintk_emit (kernel/printk/printk.c:2414) 
[ C56] _printk (kernel/printk/printk.c:2474) 
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339) 
[ C56] ? _printk (kernel/printk/printk.c:2474) 
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154) 
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601) 
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165) 
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352) 
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) 
[ C56] vfs_write (fs/read_write.c:582) 
[ C56] ksys_write (fs/read_write.c:637) 
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81) 
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299) 
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88) 
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88) 
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561) 
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) 
[   C56] RIP: 0033:0x7f12ca638c94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
   0:	c7 00 16 00 00 00    	movl   $0x16,(%rax)
   6:	b8 ff ff ff ff       	mov    $0xffffffff,%eax
   b:	c3                   	ret
   c:	66 2e 0f 1f 84 00 00 	cs nopw 0x0(%rax,%rax,1)
  13:	00 00 00 
  16:	f3 0f 1e fa          	endbr64
  1a:	80 3d d5 76 0d 00 00 	cmpb   $0x0,0xd76d5(%rip)        # 0xd76f6
  21:	74 13                	je     0x36
  23:	b8 01 00 00 00       	mov    $0x1,%eax
  28:	0f 05                	syscall
  2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
  30:	77 54                	ja     0x86
  32:	c3                   	ret
  33:	0f 1f 00             	nopl   (%rax)
  36:	55                   	push   %rbp
  37:	48 89 e5             	mov    %rsp,%rbp
  3a:	48 83 ec 20          	sub    $0x20,%rsp
  3e:	48                   	rex.W
  3f:	89                   	.byte 0x89

Code starting with the faulting instruction
===========================================
   0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
   6:	77 54                	ja     0x5c
   8:	c3                   	ret
   9:	0f 1f 00             	nopl   (%rax)
   c:	55                   	push   %rbp
   d:	48 89 e5             	mov    %rsp,%rbp
  10:	48 83 ec 20          	sub    $0x20,%rsp
  14:	48                   	rex.W
  15:	89                   	.byte 0x89
[   C56] RSP: 002b:00007ffce58c2ef8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[   C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f12ca638c94
[   C56] RDX: 0000000000000002 RSI: 000055b5493b8490 RDI: 0000000000000001
[   C56] RBP: 00007ffce58c2f20 R08: 0000000000000410 R09: 0000000000000001
[   C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[   C56] R13: 000055b5493b8490 R14: 00007f12ca7095c0 R15: 00007f12ca706f20
[   C56]  </TASK>
[ T3402] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ T3402] Call Trace:
[ T3402]  <TASK>
[ T3402] dump_stack_lvl (lib/dump_stack.c:107) 
[ T3402] panic (kernel/panic.c:340) 
[ T3402] ? _printk (kernel/printk/printk.c:2474) 
[ T3402] sysrq_handle_crash (drivers/tty/sysrq.c:154) 
[ T3402] __handle_sysrq (drivers/tty/sysrq.c:601) 
[ T3402] write_sysrq_trigger (drivers/tty/sysrq.c:1165) 
[ T3402] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352) 
[ T3402] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) 
[ T3402] vfs_write (fs/read_write.c:582) 
[ T3402] ksys_write (fs/read_write.c:637) 
[ T3402] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81) 
[ T3402] ? syscall_exit_to_user_mode (kernel/entry/common.c:299) 
[ T3402] ? do_syscall_64 (arch/x86/entry/common.c:88) 
[ T3402] ? do_syscall_64 (arch/x86/entry/common.c:88) 
[ T3402] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561) 
[ T3402] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) 
[ T3402] RIP: 0033:0x7f12ca638c94
[ T3402] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
   0:	c7 00 16 00 00 00    	movl   $0x16,(%rax)
   6:	b8 ff ff ff ff       	mov    $0xffffffff,%eax
   b:	c3                   	ret
   c:	66 2e 0f 1f 84 00 00 	cs nopw 0x0(%rax,%rax,1)
  13:	00 00 00 
  16:	f3 0f 1e fa          	endbr64
  1a:	80 3d d5 76 0d 00 00 	cmpb   $0x0,0xd76d5(%rip)        # 0xd76f6
  21:	74 13                	je     0x36
  23:	b8 01 00 00 00       	mov    $0x1,%eax
  28:	0f 05                	syscall
  2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
  30:	77 54                	ja     0x86
  32:	c3                   	ret
  33:	0f 1f 00             	nopl   (%rax)
  36:	55                   	push   %rbp
  37:	48 89 e5             	mov    %rsp,%rbp
  3a:	48 83 ec 20          	sub    $0x20,%rsp
  3e:	48                   	rex.W
  3f:	89                   	.byte 0x89

Code starting with the faulting instruction
===========================================
   0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
   6:	77 54                	ja     0x5c
   8:	c3                   	ret
   9:	0f 1f 00             	nopl   (%rax)
   c:	55                   	push   %rbp
   d:	48 89 e5             	mov    %rsp,%rbp
  10:	48 83 ec 20          	sub    $0x20,%rsp
  14:	48                   	rex.W
  15:	89                   	.byte 0x89
[ T3402] RSP: 002b:00007ffce58c2ef8 EFLAGS: 00000202
[ T3402] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f12ca638c94
[ T3402] RDX: 0000000000000002 RSI: 000055b5493b8490 RDI: 0000000000000001
[ T3402] RBP: 00007ffce58c2f20 R08: 0000000000000410 R09: 0000000000000001
[ T3402] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[ T3402] R13: 000055b5493b8490 R14: 00007f12ca7095c0 R15: 00007f12ca706f20
[ T3402]  </TASK>
[ T3402] Kernel Offset: disabled
[ T3402] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---

-- 
Sincerly,
John Wyatt
Software Engineer, Core Kernel


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: NMIs reported by console_blast.sh with 6.6.20-rt25
  2024-03-15 19:21 NMIs reported by console_blast.sh with 6.6.20-rt25 John B. Wyatt IV
@ 2024-03-22 12:29 ` Sebastian Andrzej Siewior
  2024-03-27 23:44   ` John B. Wyatt IV
  0 siblings, 1 reply; 6+ messages in thread
From: Sebastian Andrzej Siewior @ 2024-03-22 12:29 UTC (permalink / raw)
  To: John B. Wyatt IV
  Cc: John Ogness, Petr Mladek, Clark Williams, Juri Lelli,
	Derek Barbosa, Bruno Goncalves, John B. Wyatt IV, LKML,
	linux-rt-users

On 2024-03-15 15:21:19 [-0400], John B. Wyatt IV wrote:
> Hello John,
Hi John B.,

> The real-time team at Red Hat is discussing backporting the rt patchset in 6.6
> to RHEL 9/Stream 9. I decided to test v6.6.20-rt25 from stable-rt with
> console_blast.sh. It reported similar NMIs from my testing of 6.7.0-rt6 with
> that high cpu count server over uart 8250; which is expected since the patchset
> is similar.
> 
> One interesting thing is that 6.7.0-rt6 fully preemptive + realtime tuned profile
> did not return any NMIs while 6.6 did with that same configuration.

The thing is that console_blast.sh does this "show a backtrace on all
CPUs, please" which triggers NMIs on all CPUs for backtrace. I can't
imagine how you did obtain the backtraces without an NMI. Unless the
tuned profile disables this somehow.

> Another aspect I noticed during my testing. I did not set grub to
> start with the realtime profile at boot for this machine. When I did set
> it the second (and latter) NMI did not show for fully preemptive (the
> 3rd set at the bottom of this email). 
> 
> Caller info was enabled. No modifications to the source code were made.
> 
> I have not tested previous versions before 6.7.0-rt6 or 6.6.20-rt25;
> with the exception of accidently testing 6.6.10-rt19. 6.6.10 also
> reported NMIs during this test. If you wish to see these reports please
> let me know.

This NMI part has nothing todo with printk. If you need this clarified,
I would need a reproducer.

…
> -----------------------------
> NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned realtime profile
> -----------------------------
> 
> [ T2614] Kernel panic - not syncing: sysrq triggered crash
> [   C56] NMI backtrace for cpu 56
> [   C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
> [ C56] RIP: 0010:io_serial_out (arch/x86/kernel/early_printk.c:105) 
> [ C56] Code: 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 89 d0 0f b7 57 08 d3 e6 01 f2 ee <c3> cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90
> All code
> =======
>   12:	90                   	nop
>   13:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
>   18:	0f b6 8f c1 00 00 00 	movzbl 0xc1(%rdi),%ecx
>   1f:	89 d0                	mov    %edx,%eax
>   21:	0f b7 57 08          	movzwl 0x8(%rdi),%edx
>   25:	d3 e6                	shl    %cl,%esi
>   27:	01 f2                	add    %esi,%edx
>   29:	ee                   	out    %al,(%dx)
>   2a:*	c3                   	ret		<-- trapping instruction

where is this output from? The `ret' opcode usually does not cause a
trap. My guess is that the machine has been interrupted by an external
user at this position.
Side note: This is using early_printk, correct?
…

> [   C56] Call Trace:
> [   C56]  <NMI>
> [ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115) 
> [ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1)) 
> [ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149) 
> [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105) 
> [ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347) 
> [ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538) 
> [ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458) 
> [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105) 
> [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105) 
> [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105) 
> [   C56]  </NMI>

This looks okay. The NMI did the backtrace as expected.

> [   C56]  <TASK>
> [ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347) 
> [ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343) 
> [ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134) 
> [ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628) 
> [ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940) 
> [ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1)) 
> [ C56] vprintk_emit (kernel/printk/printk.c:2414) 
> [ C56] _printk (kernel/printk/printk.c:2474) 
> [ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339) 
> [ C56] ? _printk (kernel/printk/printk.c:2474) 
> [ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154) 
> [ C56] __handle_sysrq (drivers/tty/sysrq.c:601) 
> [ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165) 
> [ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352) 
> [ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) 
> [ C56] vfs_write (fs/read_write.c:582) 
> [ C56] ksys_write (fs/read_write.c:637) 
> [ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81) 
> [ C56] ? do_dup2 (fs/file.c:1142) 
> [ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299) 
> [ C56] ? do_syscall_64 (arch/x86/entry/common.c:88) 
> [ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561) 
> [ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) 

According to this, someone issued a `crash' via sysrq. Why?

…
> [ T2614] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---
yes exactly.
…
> NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned throughput-performance profile
> -----------------------------

This and the following backtrace shows the same picture: The CPU is
crashing due to proc/sysrq request and does CPU-backtraces via NMI and
polls in early_printk, waiting for the UART to become idle (probably).

I don't see an issue here so far.

…
> -- 
> Sincerly,
> John Wyatt
> Software Engineer, Core Kernel

Sebastian

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: NMIs reported by console_blast.sh with 6.6.20-rt25
  2024-03-22 12:29 ` Sebastian Andrzej Siewior
@ 2024-03-27 23:44   ` John B. Wyatt IV
  2024-04-02 10:34     ` Sebastian Andrzej Siewior
  0 siblings, 1 reply; 6+ messages in thread
From: John B. Wyatt IV @ 2024-03-27 23:44 UTC (permalink / raw)
  To: Sebastian Andrzej Siewior
  Cc: John Ogness, Petr Mladek, Clark Williams, Juri Lelli,
	Derek Barbosa, Bruno Goncalves, John B. Wyatt IV, LKML,
	linux-rt-users

On Fri, Mar 22, 2024 at 01:29:21PM +0100, Sebastian Andrzej Siewior wrote:
> On 2024-03-15 15:21:19 [-0400], John B. Wyatt IV wrote:
> The thing is that console_blast.sh does this "show a backtrace on all
> CPUs, please" which triggers NMIs on all CPUs for backtrace. I can't
> imagine how you did obtain the backtraces without an NMI. Unless the
> tuned profile disables this somehow.

I may have misapplied it. This is a very manual process right now due to
conserver.

> > -----------------------------
> > NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned realtime profile
> > -----------------------------
> > 
> > [ T2614] Kernel panic - not syncing: sysrq triggered crash
> > [   C56] NMI backtrace for cpu 56
> > [   C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
> > [ C56] RIP: 0010:io_serial_out (arch/x86/kernel/early_printk.c:105) 
> > [ C56] Code: 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 89 d0 0f b7 57 08 d3 e6 01 f2 ee <c3> cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90
> > All code
> > =======
> …
> >   12:	90                   	nop
> >   13:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
> >   18:	0f b6 8f c1 00 00 00 	movzbl 0xc1(%rdi),%ecx
> >   1f:	89 d0                	mov    %edx,%eax
> >   21:	0f b7 57 08          	movzwl 0x8(%rdi),%edx
> >   25:	d3 e6                	shl    %cl,%esi
> >   27:	01 f2                	add    %esi,%edx
> >   29:	ee                   	out    %al,(%dx)
> >   2a:*	c3                   	ret		<-- trapping instruction
> 
> where is this output from? The `ret' opcode usually does not cause a
> trap. My guess is that the machine has been interrupted by an external
> user at this position.

Just before the sysrq that crashes the system.

[ T2618] ret_from_fork_asm (arch/x86/entry/entry_64.S:312) 
[ T2618]  </TASK>
[ T2618] task:kworker/185:0   state:I stack:0     pid:1131  ppid:2      flags:0x00004000
[ T2618] Workqueue:  0x0 (events)
[ T2618] Call Trace:
[ T2618]  <TASK>
[ T2618] __schedule (kernel/sched/core.c:5404 kernel/sched/core.c:6718) 
[ T2618] schedule (kernel/sched/core.c:6796 kernel/sched/core.c:6810) 
[ T2618] worker_thread (kernel/workqueue.c:2734) 
[ T2618] ? __pfx_worker_thread (kernel/workqueue.c:2727) 
[ T2618] kthread (kernel/kthread.c:388) 
[ T2618] ? __pfx_kthread (kernel/kthread.c:341) 
[ T2618] ret_from_fork (arch/x86/kernel/process.c:147) 
[ T2618] ? __pfx_kthread (kernel/kthread.c:341) 
[ T2618] ret_from_fork_asm (arch/x86/entry/entry_64.S:312) 
[ T2618]  </TASK>
[ T2614] sysrq: Trigger a crash
[ T2618] task:kworker/185:0H  state:I
[ T2618]  stack:0     pid:1132  ppid:2      flags:0x00004000
[ T2614] Kernel panic - not syncing: sysrq triggered crash
[   C56] NMI backtrace for cpu 56
[   C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_out (arch/x86/kernel/early_printk.c:105) 
[ C56] Code: 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 89 d0 0f b7 57 08 d3 e6 01 f2 ee <c3> cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90

> Side note: This is using early_printk, correct?

I believe so, but it might be preempted? This is the part it stopped in.

static void io_serial_out(unsigned long addr, int offset, int value)
{
	outb(value, addr + offset);
}

> …
> 
> > [   C56] Call Trace:
> > [   C56]  <NMI>
> > [ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115) 
> > [ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1)) 
> > [ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149) 
> > [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105) 
> > [ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347) 
> > [ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538) 
> > [ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458) 
> > [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105) 
> > [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105) 
> > [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105) 
> > [   C56]  </NMI>
> 
> This looks okay. The NMI did the backtrace as expected.
> 
> > [   C56]  <TASK>
> > [ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347) 
> > [ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343) 
> > [ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134) 
> > [ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628) 
> > [ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940) 
> > [ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1)) 
> > [ C56] vprintk_emit (kernel/printk/printk.c:2414) 
> > [ C56] _printk (kernel/printk/printk.c:2474) 
> > [ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339) 
> > [ C56] ? _printk (kernel/printk/printk.c:2474) 
> > [ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154) 
> > [ C56] __handle_sysrq (drivers/tty/sysrq.c:601) 
> > [ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165) 
> > [ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352) 
> > [ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) 
> > [ C56] vfs_write (fs/read_write.c:582) 
> > [ C56] ksys_write (fs/read_write.c:637) 
> > [ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81) 
> > [ C56] ? do_dup2 (fs/file.c:1142) 
> > [ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299) 
> > [ C56] ? do_syscall_64 (arch/x86/entry/common.c:88) 
> > [ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561) 
> > [ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) 
> 
> According to this, someone issued a `crash' via sysrq. Why?
> 

This is part of the console_blast.sh script that John Ogness sent me.

Please see below:

#!/bin/bash
set -e

# This script is intended to be started with no arguments.
#
# It then calls itself to create a pinned process for each CPU. Those
# child processes will run in infinite loops of show-task-states via
# /proc/sysrq-trigger. This generates lots of contention on the console.
#
# Finally, it waits 10 seconds before crashing the system.

if [ $# -eq 0 ]; then
	cpus=$(($(nproc) - 1))
	for i in $(seq 0 $cpus); do
		$0 $i &
	done

	sleep 15
	echo c > /proc/sysrq-trigger
	exit 0
fi

# Setup the CPU mask for the provided CPU number. The real taskset(1)
# could simply use -c, but the busybox(1) variant does not support this.
mask=$(printf "0x%x\n" $((1 << $1)))
taskset -p $mask $$

# Wait a moment before beginning blast.
sleep 2

while true; do
	echo t > /proc/sysrq-trigger
done

> > NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned throughput-performance profile
> > -----------------------------
> 
> This and the following backtrace shows the same picture: The CPU is
> crashing due to proc/sysrq request and does CPU-backtraces via NMI and
> polls in early_printk, waiting for the UART to become idle (probably).
> 
> I don't see an issue here so far.

Luis Goncalves discussed it with me after reading your response. Thank
you for your help. The NMI was needed to flush the buffers upon the
system crashing itself. Does this part about NMI watchdog need to be
documented?

-- 
Sincerly,
John Wyatt
Software Engineer, Core Kernel


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: NMIs reported by console_blast.sh with 6.6.20-rt25
  2024-03-27 23:44   ` John B. Wyatt IV
@ 2024-04-02 10:34     ` Sebastian Andrzej Siewior
  2024-04-05  0:10       ` John B. Wyatt IV
  0 siblings, 1 reply; 6+ messages in thread
From: Sebastian Andrzej Siewior @ 2024-04-02 10:34 UTC (permalink / raw)
  To: John B. Wyatt IV
  Cc: John Ogness, Petr Mladek, Clark Williams, Juri Lelli,
	Derek Barbosa, Bruno Goncalves, John B. Wyatt IV, LKML,
	linux-rt-users

On 2024-03-27 19:44:20 [-0400], John B. Wyatt IV wrote:
> > where is this output from? The `ret' opcode usually does not cause a
> > trap. My guess is that the machine has been interrupted by an external
> > user at this position.
> 
> Just before the sysrq that crashes the system.

so this is intentional.

…
> > Side note: This is using early_printk, correct?
> 
> I believe so, but it might be preempted? This is the part it stopped in.
> 
> static void io_serial_out(unsigned long addr, int offset, int value)
> {
> 	outb(value, addr + offset);
> }

The function is invoked in NMU context so it can't be preempted.

> > According to this, someone issued a `crash' via sysrq. Why?
> > 
> 
> This is part of the console_blast.sh script that John Ogness sent me.
> 
> Please see below:
…

Okay. Then everything works as it should…

> > > NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned throughput-performance profile
> > > -----------------------------
> > 
> > This and the following backtrace shows the same picture: The CPU is
> > crashing due to proc/sysrq request and does CPU-backtraces via NMI and
> > polls in early_printk, waiting for the UART to become idle (probably).
> > 
> > I don't see an issue here so far.
> 
> Luis Goncalves discussed it with me after reading your response. Thank
> you for your help. The NMI was needed to flush the buffers upon the
> system crashing itself. Does this part about NMI watchdog need to be
> documented?

Not sure about that one. There is an _a_ _lot_ to be printed from NMI
and the NMI watchdog might trigger if nothing is triggering the
NMI-watchdog during the print job. Also, the crash was requested.

Sebastian

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: NMIs reported by console_blast.sh with 6.6.20-rt25
  2024-04-02 10:34     ` Sebastian Andrzej Siewior
@ 2024-04-05  0:10       ` John B. Wyatt IV
  2024-04-05  6:52         ` Sebastian Andrzej Siewior
  0 siblings, 1 reply; 6+ messages in thread
From: John B. Wyatt IV @ 2024-04-05  0:10 UTC (permalink / raw)
  To: Sebastian Andrzej Siewior
  Cc: John Ogness, Petr Mladek, Clark Williams, Juri Lelli,
	Derek Barbosa, Bruno Goncalves, John B. Wyatt IV, LKML,
	linux-rt-users

On Tue, Apr 02, 2024 at 12:34:14PM +0200, Sebastian Andrzej Siewior wrote:
> > Just before the sysrq that crashes the system.
> 
> so this is intentional.

Yes.

> > 
> > This is part of the console_blast.sh script that John Ogness sent me.
> > 
> > Please see below:
> …
> 
> Okay. Then everything works as it should…

Correct.

> 
> > > > NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned throughput-performance profile
> > > > -----------------------------
> > > 
> > > This and the following backtrace shows the same picture: The CPU is
> > > crashing due to proc/sysrq request and does CPU-backtraces via NMI and
> > > polls in early_printk, waiting for the UART to become idle (probably).
> > > 
> > > I don't see an issue here so far.
> > 
> > Luis Goncalves discussed it with me after reading your response. Thank
> > you for your help. The NMI was needed to flush the buffers upon the
> > system crashing itself. Does this part about NMI watchdog need to be
> > documented?
> 
> Not sure about that one. There is an _a_ _lot_ to be printed from NMI
> and the NMI watchdog might trigger if nothing is triggering the
> NMI-watchdog during the print job. Also, the crash was requested.

I reran the 6.6 test and no NMI was reported with fully preemptive and
the realtime tuned profile. It was my error; my apologies for that.

I did include more of logs if you want to see here for my 6.8 testing.

https://lore.kernel.org/linux-rt-users/ZgWL2UyknaE2T70C@thinkpad2021/T/#u

Sincerely,
John Wyatt
Software Engineer, Core Kernel
Red Hat


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: NMIs reported by console_blast.sh with 6.6.20-rt25
  2024-04-05  0:10       ` John B. Wyatt IV
@ 2024-04-05  6:52         ` Sebastian Andrzej Siewior
  0 siblings, 0 replies; 6+ messages in thread
From: Sebastian Andrzej Siewior @ 2024-04-05  6:52 UTC (permalink / raw)
  To: John B. Wyatt IV
  Cc: John Ogness, Petr Mladek, Clark Williams, Juri Lelli,
	Derek Barbosa, Bruno Goncalves, John B. Wyatt IV, LKML,
	linux-rt-users

On 2024-04-04 20:10:47 [-0400], John B. Wyatt IV wrote:
> I reran the 6.6 test and no NMI was reported with fully preemptive and
> the realtime tuned profile. It was my error; my apologies for that.

no worries. 

> I did include more of logs if you want to see here for my 6.8 testing.
> 
> https://lore.kernel.org/linux-rt-users/ZgWL2UyknaE2T70C@thinkpad2021/T/#u

I looked over it and it seems we are good.
Thanks for testing.

> Sincerely,
> John Wyatt
> Software Engineer, Core Kernel
> Red Hat

Sebastian

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2024-04-05  6:52 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-03-15 19:21 NMIs reported by console_blast.sh with 6.6.20-rt25 John B. Wyatt IV
2024-03-22 12:29 ` Sebastian Andrzej Siewior
2024-03-27 23:44   ` John B. Wyatt IV
2024-04-02 10:34     ` Sebastian Andrzej Siewior
2024-04-05  0:10       ` John B. Wyatt IV
2024-04-05  6:52         ` Sebastian Andrzej Siewior

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).