* Kernel 4.17.4 lockup
@ 2018-07-08 21:36 H.J. Lu
2018-07-09 14:54 ` Dave Hansen
0 siblings, 1 reply; 25+ messages in thread
From: H.J. Lu @ 2018-07-08 21:36 UTC (permalink / raw)
To: H. Peter Anvin, Matthew Wilcox, LKML
On 3 x86-64 machines, kernel 4.17.4 locked up under heavy load. 2 of them don't
have any kernel messages. One has
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: general protection
fault: 0000 [#1] SMP PTI
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Modules linked in:
rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache devlink ebtable_filter
ebtables ip6table_filter ip6_tables intel_rapl x86_pkg_temp_thermal
intel_powerclamp coretemp snd_hda_codec_hdmi snd_hda_codec_realtek
kvm_intel snd_hda_codec_generic snd_hda_intel kvm snd_hda_codec
snd_hda_core snd_hwdep irqbypass crct10dif_pclmul crc32_pclmul snd_seq
mei_wdt ghash_clmulni_intel snd_seq_device intel_cstate ppdev
intel_uncore iTCO_wdt gpio_ich iTCO_vendor_support snd_pcm
intel_rapl_perf snd_timer snd mei_me parport_pc joydev i2c_i801 mei
soundcore shpchp lpc_ich parport nfsd auth_rpcgss nfs_acl lockd grace
sunrpc i915 i2c_algo_bit drm_kms_helper r8169 drm crc32c_intel mii
video
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: CPU: 7 PID: 7093 Comm:
cc1 Not tainted 4.17.4-200.0.fc28.x86_64 #1
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Hardware name: Gigabyte
Technology Co., Ltd. H87M-D3H/H87M-D3H, BIOS F11 08/18/2015
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RIP:
0010:free_pages_and_swap_cache+0x29/0xb0
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RSP:
0018:ffffb2cd83ffbd58 EFLAGS: 00010202
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RAX: 0017fffe00040068
RBX: ffff93d4abb5ec80 RCX: 0000000000000000
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RDX: 0017fffe00040068
RSI: 00000000000001fe RDI: ffff93d51e3dd2a0
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RBP: 00000000000001fe
R08: fffff0809df82d20 R09: ffff93d51e5d5000
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R10: ffff93d51e5d5e20
R11: ffff93d51e5d5d00 R12: ffff93d4abb5e010
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R13: fffbf0809e304bc0
R14: ffff93d4abb5f000 R15: ffff93d4cbcee8f0
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: FS:
0000000000000000(0000) GS:ffff93d51e3c0000(0000)
knlGS:0000000000000000
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: CS: 0010 DS: 0000 ES:
0000 CR0: 0000000080050033
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: CR2: 00007ffb255e753c
CR3: 00000005e820a002 CR4: 00000000001606e0
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Call Trace:
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: tlb_flush_mmu_free+0x31/0x50
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: arch_tlb_finish_mmu+0x42/0x70
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: tlb_finish_mmu+0x1f/0x30
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: exit_mmap+0xca/0x190
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: mmput+0x5f/0x130
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: do_exit+0x280/0xae0
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: ? __do_page_fault+0x263/0x4e0
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: do_group_exit+0x3a/0xa0
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: __x64_sys_exit_group+0x14/0x20
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: do_syscall_64+0x65/0x160
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel:
entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RIP: 0033:0x7ffb2542b3c6
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RSP:
002b:00007ffd9e7e33b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RAX: ffffffffffffffda
RBX: 00007ffb2551c740 RCX: 00007ffb2542b3c6
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RDX: 0000000000000000
RSI: 000000000000003c RDI: 0000000000000000
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RBP: 0000000000000000
R08: 00000000000000e7 R09: fffffffffffffe70
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R10: 00007ffd9e7e3250
R11: 0000000000000246 R12: 00007ffb2551c740
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R13: 0000000000000037
R14: 00007ffb25525708 R15: 0000000000000000
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Code: 40 00 0f 1f 44 00
00 41 56 41 55 41 54 49 89 fc 55 89 f5 53 e8 29 99 fb ff 85 ed 7e 6b
8d 45 ff 4c 89 e3 4d 8d 74 c4 08 4c 8b 2b <49> 8b 55 20 48 8d 42 ff 83
e2 01 49 0f 44 c5 48 8b 48 20 48 8d
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RIP:
free_pages_and_swap_cache+0x29/0xb0 RSP: ffffb2cd83ffbd58
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: ---[ end trace
5960277fd8a3c0b5 ]---
Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Fixing recursive fault
but reboot is needed!
Kernel 4.16.x is OK. Is this a known issue?
--
H.J.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-08 21:36 Kernel 4.17.4 lockup H.J. Lu
@ 2018-07-09 14:54 ` Dave Hansen
2018-07-09 23:02 ` H.J. Lu
0 siblings, 1 reply; 25+ messages in thread
From: Dave Hansen @ 2018-07-09 14:54 UTC (permalink / raw)
To: H.J. Lu, H. Peter Anvin, Matthew Wilcox, LKML
On 07/09/2018 06:19 AM, Lu, Hongjiu wrote:
> On 3 x86-64 machines, kernel 4.17.4 locked up under heavy load. 2 of
them don't have any kernel messages. One has
Hi H.J.,
It'd be really handy if you could pastebin things like this, or attach a
text file with the oops. Your email wrapped the heck out of the oops and
I had to go and unwrap it to read it.
A full disassembly of free_pages_and_swap_cache() from the actual
vmlinux to account for differences between toolchains would be helpful.
It'll probably help me figure out what the loop counter was for
instance. It makes it a bit easier to read random oopses if you boot
with 'nokaslr' because the pointer types (kernel text, linear map,
vmemmap, etc...) stick out much more easily. Would you be able to boot
with it in the future?
We've had a bit of churn in that code, but nothing between 4.16 and 4.17
that really sticks out to me in the x86 code.
The general protection fault is a bit of an oddball. If I disassembled
right, it's trying to dereference %R13+20. That doesn't even cross a
page boundary, so it's a bit hard to fathom where the #GP would come from.
(mostly) unwrapped oops below.
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: general protection
>> fault: 0000 [#1] SMP PTI
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Modules linked in:
>> rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache devlink ebtable_filter
>> ebtables ip6table_filter ip6_tables intel_rapl x86_pkg_temp_thermal
>> intel_powerclamp coretemp snd_hda_codec_hdmi snd_hda_codec_realtek
>> kvm_intel snd_hda_codec_generic snd_hda_intel kvm snd_hda_codec
>> snd_hda_core snd_hwdep irqbypass crct10dif_pclmul crc32_pclmul snd_seq
>> mei_wdt ghash_clmulni_intel snd_seq_device intel_cstate ppdev
>> intel_uncore iTCO_wdt gpio_ich iTCO_vendor_support snd_pcm
>> intel_rapl_perf snd_timer snd mei_me parport_pc joydev i2c_i801 mei
>> soundcore shpchp lpc_ich parport nfsd auth_rpcgss nfs_acl lockd grace
>> sunrpc i915 i2c_algo_bit drm_kms_helper r8169 drm crc32c_intel mii
>> video
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: CPU: 7 PID: 7093 Comm:
>> cc1 Not tainted 4.17.4-200.0.fc28.x86_64 #1
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Hardware name: Gigabyte
>> Technology Co., Ltd. H87M-D3H/H87M-D3H, BIOS F11 08/18/2015
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RIP: 0010:free_pages_and_swap_cache+0x29/0xb0
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RSP: 0018:ffffb2cd83ffbd58 EFLAGS: 00010202
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RAX: 0017fffe00040068 RBX: ffff93d4abb5ec80 RCX: 0000000000000000
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RDX: 0017fffe00040068 RSI: 00000000000001fe RDI: ffff93d51e3dd2a0
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RBP: 00000000000001fe R08: fffff0809df82d20 R09: ffff93d51e5d5000
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R10: ffff93d51e5d5e20 R11: ffff93d51e5d5d00 R12: ffff93d4abb5e010
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R13: fffbf0809e304bc0 R14: ffff93d4abb5f000 R15: ffff93d4cbcee8f0
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: FS: 0000000000000000(0000) GS:ffff93d51e3c0000(0000) knlGS:0000000000000000
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: CR2: 00007ffb255e753c CR3: 00000005e820a002 CR4: 00000000001606e0
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Call Trace:
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: tlb_flush_mmu_free+0x31/0x50
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: arch_tlb_finish_mmu+0x42/0x70
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: tlb_finish_mmu+0x1f/0x30
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: exit_mmap+0xca/0x190
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: mmput+0x5f/0x130
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: do_exit+0x280/0xae0
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: ? __do_page_fault+0x263/0x4e0
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: do_group_exit+0x3a/0xa0
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: __x64_sys_exit_group+0x14/0x20
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: do_syscall_64+0x65/0x160
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RIP: 0033:0x7ffb2542b3c6
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RSP: 002b:00007ffd9e7e33b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RAX: ffffffffffffffda RBX: 00007ffb2551c740 RCX: 00007ffb2542b3c6
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RDX: 0000000000000000 RSI: 000000000000003c RDI: 0000000000000000
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RBP: 0000000000000000 R08: 00000000000000e7 R09: fffffffffffffe70
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R10: 00007ffd9e7e3250 R11: 0000000000000246 R12: 00007ffb2551c740
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R13: 0000000000000037 R14: 00007ffb25525708 R15: 0000000000000000
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Code: 40 00 0f 1f 44 00 00 41 56 41 55 41 54 49 89 fc 55 89 f5 53 e8 29 99 fb ff 85 ed 7e 6b 8d 45 ff 4c 89 e3 4d 8d 74 c4 08 4c 8b 2b <49> 8b 55 20 48 8d 42 ff 83 e2 01 49 0f 44 c5 48 8b 48 20 48 8d
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RIP: free_pages_and_swap_cache+0x29/0xb0 RSP: ffffb2cd83ffbd58
>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: ---[ end trace 5960277fd8a3c0b5 ]---
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-09 14:54 ` Dave Hansen
@ 2018-07-09 23:02 ` H.J. Lu
2018-07-10 0:44 ` Dave Hansen
0 siblings, 1 reply; 25+ messages in thread
From: H.J. Lu @ 2018-07-09 23:02 UTC (permalink / raw)
To: Dave Hansen; +Cc: H. Peter Anvin, Matthew Wilcox, LKML
On Mon, Jul 9, 2018 at 7:54 AM, Dave Hansen <dave.hansen@intel.com> wrote:
> On 07/09/2018 06:19 AM, Lu, Hongjiu wrote:
>> On 3 x86-64 machines, kernel 4.17.4 locked up under heavy load. 2 of
> them don't have any kernel messages. One has
>
> Hi H.J.,
>
> It'd be really handy if you could pastebin things like this, or attach a
> text file with the oops. Your email wrapped the heck out of the oops and
> I had to go and unwrap it to read it.
>
> A full disassembly of free_pages_and_swap_cache() from the actual
> vmlinux to account for differences between toolchains would be helpful.
> It'll probably help me figure out what the loop counter was for
> instance. It makes it a bit easier to read random oopses if you boot
> with 'nokaslr' because the pointer types (kernel text, linear map,
> vmemmap, etc...) stick out much more easily. Would you be able to boot
> with it in the future?
I will do that if it happens again.
> We've had a bit of churn in that code, but nothing between 4.16 and 4.17
> that really sticks out to me in the x86 code.
>
> The general protection fault is a bit of an oddball. If I disassembled
> right, it's trying to dereference %R13+20. That doesn't even cross a
This is correct.
> page boundary, so it's a bit hard to fathom where the #GP would come from.
(gdb) disass free_pages_and_swap_cache
Dump of assembler code for function free_pages_and_swap_cache:
0xffffffff8124c0d0 <+0>: callq 0xffffffff81a017a0 <__fentry__>
0xffffffff8124c0d5 <+5>: push %r14
0xffffffff8124c0d7 <+7>: push %r13
0xffffffff8124c0d9 <+9>: push %r12
0xffffffff8124c0db <+11>: mov %rdi,%r12
0xffffffff8124c0de <+14>: push %rbp
0xffffffff8124c0df <+15>: mov %esi,%ebp
0xffffffff8124c0e1 <+17>: push %rbx
0xffffffff8124c0e2 <+18>: callq 0xffffffff81205a10 <lru_add_drain>
0xffffffff8124c0e7 <+23>: test %ebp,%ebp
0xffffffff8124c0e9 <+25>: jle 0xffffffff8124c156
<free_pages_and_swap_cache+134>
0xffffffff8124c0eb <+27>: lea -0x1(%rbp),%eax
0xffffffff8124c0ee <+30>: mov %r12,%rbx
0xffffffff8124c0f1 <+33>: lea 0x8(%r12,%rax,8),%r14
0xffffffff8124c0f6 <+38>: mov (%rbx),%r13
0xffffffff8124c0f9 <+41>: mov 0x20(%r13),%rdx
<<<<<<<<<<<<<<<<<<<< GPF here.
0xffffffff8124c0fd <+45>: lea -0x1(%rdx),%rax
0xffffffff8124c101 <+49>: and $0x1,%edx
0xffffffff8124c104 <+52>: cmove %r13,%rax
0xffffffff8124c108 <+56>: mov 0x20(%rax),%rcx
0xffffffff8124c10c <+60>: lea -0x1(%rcx),%rdx
0xffffffff8124c110 <+64>: and $0x1,%ecx
0xffffffff8124c113 <+67>: cmove %rax,%rdx
0xffffffff8124c117 <+71>: mov (%rdx),%rdx
0xffffffff8124c11a <+74>: test $0x40000,%edx
0xffffffff8124c120 <+80>: je 0xffffffff8124c14d
<free_pages_and_swap_cache+125>
0xffffffff8124c122 <+82>: mov (%rax),%rax
0xffffffff8124c125 <+85>: test $0x2,%ah
0xffffffff8124c128 <+88>: je 0xffffffff8124c14d
<free_pages_and_swap_cache+125>
0xffffffff8124c12a <+90>: mov %r13,%rdi
0xffffffff8124c12d <+93>: callq 0xffffffff81218260 <page_mapped>
0xffffffff8124c132 <+98>: test %al,%al
0xffffffff8124c134 <+100>: jne 0xffffffff8124c14d
<free_pages_and_swap_cache+125>
0xffffffff8124c136 <+102>: mov 0x20(%r13),%rdx
0xffffffff8124c13a <+106>: lea -0x1(%rdx),%rax
0xffffffff8124c13e <+110>: and $0x1,%edx
0xffffffff8124c141 <+113>: cmove %r13,%rax
0xffffffff8124c145 <+117>: lock btsq $0x0,(%rax)
0xffffffff8124c14b <+123>: jae 0xffffffff8124c168
<free_pages_and_swap_cache+152>
0xffffffff8124c14d <+125>: add $0x8,%rbx
0xffffffff8124c151 <+129>: cmp %rbx,%r14
0xffffffff8124c154 <+132>: jne 0xffffffff8124c0f6
<free_pages_and_swap_cache+38>
0xffffffff8124c156 <+134>: pop %rbx
0xffffffff8124c157 <+135>: mov %ebp,%esi
0xffffffff8124c159 <+137>: mov %r12,%rdi
0xffffffff8124c15c <+140>: pop %rbp
0xffffffff8124c15d <+141>: pop %r12
0xffffffff8124c15f <+143>: pop %r13
0xffffffff8124c161 <+145>: pop %r14
0xffffffff8124c163 <+147>: jmpq 0xffffffff81204600 <release_pages>
0xffffffff8124c168 <+152>: mov %r13,%rdi
0xffffffff8124c16b <+155>: callq 0xffffffff81250aa0 <try_to_free_swap>
0xffffffff8124c170 <+160>: mov %r13,%rdi
0xffffffff8124c173 <+163>: callq 0xffffffff811ed140 <unlock_page>
0xffffffff8124c178 <+168>: jmp 0xffffffff8124c14d
<free_pages_and_swap_cache+125>
End of assembler dump.
(gdb)
> (mostly) unwrapped oops below.
>
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: general protection
>>> fault: 0000 [#1] SMP PTI
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Modules linked in:
>>> rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache devlink ebtable_filter
>>> ebtables ip6table_filter ip6_tables intel_rapl x86_pkg_temp_thermal
>>> intel_powerclamp coretemp snd_hda_codec_hdmi snd_hda_codec_realtek
>>> kvm_intel snd_hda_codec_generic snd_hda_intel kvm snd_hda_codec
>>> snd_hda_core snd_hwdep irqbypass crct10dif_pclmul crc32_pclmul snd_seq
>>> mei_wdt ghash_clmulni_intel snd_seq_device intel_cstate ppdev
>>> intel_uncore iTCO_wdt gpio_ich iTCO_vendor_support snd_pcm
>>> intel_rapl_perf snd_timer snd mei_me parport_pc joydev i2c_i801 mei
>>> soundcore shpchp lpc_ich parport nfsd auth_rpcgss nfs_acl lockd grace
>>> sunrpc i915 i2c_algo_bit drm_kms_helper r8169 drm crc32c_intel mii
>>> video
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: CPU: 7 PID: 7093 Comm:
>>> cc1 Not tainted 4.17.4-200.0.fc28.x86_64 #1
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Hardware name: Gigabyte
>>> Technology Co., Ltd. H87M-D3H/H87M-D3H, BIOS F11 08/18/2015
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RIP: 0010:free_pages_and_swap_cache+0x29/0xb0
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RSP: 0018:ffffb2cd83ffbd58 EFLAGS: 00010202
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RAX: 0017fffe00040068 RBX: ffff93d4abb5ec80 RCX: 0000000000000000
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RDX: 0017fffe00040068 RSI: 00000000000001fe RDI: ffff93d51e3dd2a0
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RBP: 00000000000001fe R08: fffff0809df82d20 R09: ffff93d51e5d5000
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R10: ffff93d51e5d5e20 R11: ffff93d51e5d5d00 R12: ffff93d4abb5e010
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R13: fffbf0809e304bc0 R14: ffff93d4abb5f000 R15: ffff93d4cbcee8f0
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: FS: 0000000000000000(0000) GS:ffff93d51e3c0000(0000) knlGS:0000000000000000
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: CR2: 00007ffb255e753c CR3: 00000005e820a002 CR4: 00000000001606e0
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Call Trace:
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: tlb_flush_mmu_free+0x31/0x50
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: arch_tlb_finish_mmu+0x42/0x70
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: tlb_finish_mmu+0x1f/0x30
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: exit_mmap+0xca/0x190
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: mmput+0x5f/0x130
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: do_exit+0x280/0xae0
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: ? __do_page_fault+0x263/0x4e0
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: do_group_exit+0x3a/0xa0
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: __x64_sys_exit_group+0x14/0x20
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: do_syscall_64+0x65/0x160
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RIP: 0033:0x7ffb2542b3c6
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RSP: 002b:00007ffd9e7e33b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RAX: ffffffffffffffda RBX: 00007ffb2551c740 RCX: 00007ffb2542b3c6
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RDX: 0000000000000000 RSI: 000000000000003c RDI: 0000000000000000
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RBP: 0000000000000000 R08: 00000000000000e7 R09: fffffffffffffe70
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R10: 00007ffd9e7e3250 R11: 0000000000000246 R12: 00007ffb2551c740
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R13: 0000000000000037 R14: 00007ffb25525708 R15: 0000000000000000
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Code: 40 00 0f 1f 44 00 00 41 56 41 55 41 54 49 89 fc 55 89 f5 53 e8 29 99 fb ff 85 ed 7e 6b 8d 45 ff 4c 89 e3 4d 8d 74 c4 08 4c 8b 2b <49> 8b 55 20 48 8d 42 ff 83 e2 01 49 0f 44 c5 48 8b 48 20 48 8d
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RIP: free_pages_and_swap_cache+0x29/0xb0 RSP: ffffb2cd83ffbd58
>>> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: ---[ end trace 5960277fd8a3c0b5 ]---
--
H.J.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-09 23:02 ` H.J. Lu
@ 2018-07-10 0:44 ` Dave Hansen
2018-07-10 2:14 ` H.J. Lu
0 siblings, 1 reply; 25+ messages in thread
From: Dave Hansen @ 2018-07-10 0:44 UTC (permalink / raw)
To: H.J. Lu
Cc: H. Peter Anvin, Matthew Wilcox, LKML, Andy Lutomirski,
Mel Gorman, Andrew Morton, Rik van Riel, Minchan Kim
... cc'ing a few folks who I know have been looking at this code
lately. The full oops is below if any of you want to take a look.
OK, well, annotating the disassembly a bit:
> (gdb) disass free_pages_and_swap_cache
> Dump of assembler code for function free_pages_and_swap_cache:
> 0xffffffff8124c0d0 <+0>: callq 0xffffffff81a017a0 <__fentry__>
> 0xffffffff8124c0d5 <+5>: push %r14
> 0xffffffff8124c0d7 <+7>: push %r13
> 0xffffffff8124c0d9 <+9>: push %r12
> 0xffffffff8124c0db <+11>: mov %rdi,%r12 // %r12 = pages
> 0xffffffff8124c0de <+14>: push %rbp
> 0xffffffff8124c0df <+15>: mov %esi,%ebp // %ebp = nr
> 0xffffffff8124c0e1 <+17>: push %rbx
> 0xffffffff8124c0e2 <+18>: callq 0xffffffff81205a10 <lru_add_drain>
> 0xffffffff8124c0e7 <+23>: test %ebp,%ebp // test nr==0
> 0xffffffff8124c0e9 <+25>: jle 0xffffffff8124c156 <free_pages_and_swap_cache+134>
> 0xffffffff8124c0eb <+27>: lea -0x1(%rbp),%eax
> 0xffffffff8124c0ee <+30>: mov %r12,%rbx // %rbx = pages
> 0xffffffff8124c0f1 <+33>: lea 0x8(%r12,%rax,8),%r14 // load &pages[nr] into %r14?
> 0xffffffff8124c0f6 <+38>: mov (%rbx),%r13 // %r13 = pages[i]
> 0xffffffff8124c0f9 <+41>: mov 0x20(%r13),%rdx //<<<<<<<<<<<<<<<<<<<< GPF here.
%r13 is 64-byte aligned, so looks like a halfway reasonable 'struct page *'.
%R14 looks OK (0xffff93d4abb5f000) because it points to the end of a
dynamically-allocated (not on-stack) mmu_gather_batch page. %RBX is
pointing 50 pages up from the start of the previous page. That makes it
the 48th page in pages[] after a pointer and two integers in the
beginning of the structure. That 48 is important because it's way
larger than the on-stack size of 8.
It's hard to make much sense of %R13 (pages[48] / 0xfffbf0809e304bc0)
because the vmemmap addresses get randomized. But, I _think_ that's too
high of an address for a 4-level paging vmemmap[] entry. Does anybody
else know offhand?
I'd really want to see this reproduced without KASLR to make the oops
easier to read. It would also be handy to try your workload with all
the pedantic debugging: KASAN, slab debugging, DEBUG_PAGE_ALLOC, etc...
and see if it still triggers.
Some relevant functions and structures below for reference.
void free_pages_and_swap_cache(struct page **pages, int nr)
{
for (i = 0; i < nr; i++)
free_swap_cache(pages[i]);
}
static void tlb_flush_mmu_free(struct mmu_gather *tlb)
{
for (batch = &tlb->local; batch && batch->nr;
batch = batch->next) {
free_pages_and_swap_cache(batch->pages, batch->nr);
}
zap_pte_range()
{
if (force_flush)
tlb_flush_mmu_free(tlb);
}
... all the way up to the on-stack-allocated mmu_gather:
void zap_page_range(struct vm_area_struct *vma, unsigned long start,
unsigned long size)
{
struct mmu_gather tlb;
#define MMU_GATHER_BUNDLE 8
struct mmu_gather {
...
struct mmu_gather_batch local;
struct page *__pages[MMU_GATHER_BUNDLE];
}
struct mmu_gather_batch {
struct mmu_gather_batch *next;
unsigned int nr;
unsigned int max;
struct page *pages[0];
};
#define MAX_GATHER_BATCH \
((PAGE_SIZE - sizeof(struct mmu_gather_batch)) / sizeof(void *))
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: general protection
> fault: 0000 [#1] SMP PTI
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Modules linked in:
> rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache devlink ebtable_filter
> ebtables ip6table_filter ip6_tables intel_rapl x86_pkg_temp_thermal
> intel_powerclamp coretemp snd_hda_codec_hdmi snd_hda_codec_realtek
> kvm_intel snd_hda_codec_generic snd_hda_intel kvm snd_hda_codec
> snd_hda_core snd_hwdep irqbypass crct10dif_pclmul crc32_pclmul snd_seq
> mei_wdt ghash_clmulni_intel snd_seq_device intel_cstate ppdev
> intel_uncore iTCO_wdt gpio_ich iTCO_vendor_support snd_pcm
> intel_rapl_perf snd_timer snd mei_me parport_pc joydev i2c_i801 mei
> soundcore shpchp lpc_ich parport nfsd auth_rpcgss nfs_acl lockd grace
> sunrpc i915 i2c_algo_bit drm_kms_helper r8169 drm crc32c_intel mii
> video
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: CPU: 7 PID: 7093 Comm:
> cc1 Not tainted 4.17.4-200.0.fc28.x86_64 #1
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Hardware name: Gigabyte
> Technology Co., Ltd. H87M-D3H/H87M-D3H, BIOS F11 08/18/2015
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RIP: 0010:free_pages_and_swap_cache+0x29/0xb0
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RSP: 0018:ffffb2cd83ffbd58 EFLAGS: 00010202
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RAX: 0017fffe00040068 RBX: ffff93d4abb5ec80 RCX: 0000000000000000
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RDX: 0017fffe00040068 RSI: 00000000000001fe RDI: ffff93d51e3dd2a0
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RBP: 00000000000001fe R08: fffff0809df82d20 R09: ffff93d51e5d5000
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R10: ffff93d51e5d5e20 R11: ffff93d51e5d5d00 R12: ffff93d4abb5e010
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R13: fffbf0809e304bc0 R14: ffff93d4abb5f000 R15: ffff93d4cbcee8f0
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: FS: 0000000000000000(0000) GS:ffff93d51e3c0000(0000) knlGS:0000000000000000
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: CR2: 00007ffb255e753c CR3: 00000005e820a002 CR4: 00000000001606e0
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Call Trace:
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: tlb_flush_mmu_free+0x31/0x50
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: arch_tlb_finish_mmu+0x42/0x70
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: tlb_finish_mmu+0x1f/0x30
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: exit_mmap+0xca/0x190
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: mmput+0x5f/0x130
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: do_exit+0x280/0xae0
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: ? __do_page_fault+0x263/0x4e0
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: do_group_exit+0x3a/0xa0
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: __x64_sys_exit_group+0x14/0x20
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: do_syscall_64+0x65/0x160
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RIP: 0033:0x7ffb2542b3c6
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RSP: 002b:00007ffd9e7e33b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RAX: ffffffffffffffda RBX: 00007ffb2551c740 RCX: 00007ffb2542b3c6
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RDX: 0000000000000000 RSI: 000000000000003c RDI: 0000000000000000
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RBP: 0000000000000000 R08: 00000000000000e7 R09: fffffffffffffe70
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R10: 00007ffd9e7e3250 R11: 0000000000000246 R12: 00007ffb2551c740
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: R13: 0000000000000037 R14: 00007ffb25525708 R15: 0000000000000000
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: Code: 40 00 0f 1f 44 00 00 41 56 41 55 41 54 49 89 fc 55 89 f5 53 e8 29 99 fb ff 85 ed 7e 6b 8d 45 ff 4c 89 e3 4d 8d 74 c4 08 4c 8b 2b <49> 8b 55 20 48 8d 42 ff 83 e2 01 49 0f 44 c5 48 8b 48 20 48 8d
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: RIP: free_pages_and_swap_cache+0x29/0xb0 RSP: ffffb2cd83ffbd58
> Jul 05 14:33:32 gnu-hsw-1.sc.intel.com kernel: ---[ end trace 5960277fd8a3c0b5 ]---
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-10 0:44 ` Dave Hansen
@ 2018-07-10 2:14 ` H.J. Lu
2018-07-10 3:47 ` Dave Hansen
0 siblings, 1 reply; 25+ messages in thread
From: H.J. Lu @ 2018-07-10 2:14 UTC (permalink / raw)
To: Dave Hansen
Cc: H. Peter Anvin, LKML, Andy Lutomirski, Mel Gorman, Andrew Morton,
Rik van Riel, Minchan Kim
On Mon, Jul 9, 2018 at 5:44 PM, Dave Hansen <dave.hansen@intel.com> wrote:
> ... cc'ing a few folks who I know have been looking at this code
> lately. The full oops is below if any of you want to take a look.
>
> OK, well, annotating the disassembly a bit:
>
>> (gdb) disass free_pages_and_swap_cache
>> Dump of assembler code for function free_pages_and_swap_cache:
>> 0xffffffff8124c0d0 <+0>: callq 0xffffffff81a017a0 <__fentry__>
>> 0xffffffff8124c0d5 <+5>: push %r14
>> 0xffffffff8124c0d7 <+7>: push %r13
>> 0xffffffff8124c0d9 <+9>: push %r12
>> 0xffffffff8124c0db <+11>: mov %rdi,%r12 // %r12 = pages
>> 0xffffffff8124c0de <+14>: push %rbp
>> 0xffffffff8124c0df <+15>: mov %esi,%ebp // %ebp = nr
>> 0xffffffff8124c0e1 <+17>: push %rbx
>> 0xffffffff8124c0e2 <+18>: callq 0xffffffff81205a10 <lru_add_drain>
>> 0xffffffff8124c0e7 <+23>: test %ebp,%ebp // test nr==0
>> 0xffffffff8124c0e9 <+25>: jle 0xffffffff8124c156 <free_pages_and_swap_cache+134>
>> 0xffffffff8124c0eb <+27>: lea -0x1(%rbp),%eax
>> 0xffffffff8124c0ee <+30>: mov %r12,%rbx // %rbx = pages
>> 0xffffffff8124c0f1 <+33>: lea 0x8(%r12,%rax,8),%r14 // load &pages[nr] into %r14?
>> 0xffffffff8124c0f6 <+38>: mov (%rbx),%r13 // %r13 = pages[i]
>> 0xffffffff8124c0f9 <+41>: mov 0x20(%r13),%rdx //<<<<<<<<<<<<<<<<<<<< GPF here.
> %r13 is 64-byte aligned, so looks like a halfway reasonable 'struct page *'.
>
> %R14 looks OK (0xffff93d4abb5f000) because it points to the end of a
> dynamically-allocated (not on-stack) mmu_gather_batch page. %RBX is
> pointing 50 pages up from the start of the previous page. That makes it
> the 48th page in pages[] after a pointer and two integers in the
> beginning of the structure. That 48 is important because it's way
> larger than the on-stack size of 8.
>
> It's hard to make much sense of %R13 (pages[48] / 0xfffbf0809e304bc0)
> because the vmemmap addresses get randomized. But, I _think_ that's too
> high of an address for a 4-level paging vmemmap[] entry. Does anybody
> else know offhand?
>
> I'd really want to see this reproduced without KASLR to make the oops
> easier to read. It would also be handy to try your workload with all
> the pedantic debugging: KASAN, slab debugging, DEBUG_PAGE_ALLOC, etc...
> and see if it still triggers.
How can I turn them on at boot time?
> Some relevant functions and structures below for reference.
>
> void free_pages_and_swap_cache(struct page **pages, int nr)
> {
> for (i = 0; i < nr; i++)
> free_swap_cache(pages[i]);
> }
>
>
> static void tlb_flush_mmu_free(struct mmu_gather *tlb)
> {
> for (batch = &tlb->local; batch && batch->nr;
> batch = batch->next) {
> free_pages_and_swap_cache(batch->pages, batch->nr);
> }
>
> zap_pte_range()
> {
> if (force_flush)
> tlb_flush_mmu_free(tlb);
> }
>
> ... all the way up to the on-stack-allocated mmu_gather:
>
> void zap_page_range(struct vm_area_struct *vma, unsigned long start,
> unsigned long size)
> {
> struct mmu_gather tlb;
>
>
> #define MMU_GATHER_BUNDLE 8
>
> struct mmu_gather {
> ...
> struct mmu_gather_batch local;
> struct page *__pages[MMU_GATHER_BUNDLE];
> }
>
> struct mmu_gather_batch {
> struct mmu_gather_batch *next;
> unsigned int nr;
> unsigned int max;
> struct page *pages[0];
> };
>
> #define MAX_GATHER_BATCH \
> ((PAGE_SIZE - sizeof(struct mmu_gather_batch)) / sizeof(void *))
>
--
H.J.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-10 2:14 ` H.J. Lu
@ 2018-07-10 3:47 ` Dave Hansen
[not found] ` <CAMe9rOrHowBX06nihdRRmEqhV8v7cs+PwVY7JYQFpUFOnHC71A@mail.gmail.com>
0 siblings, 1 reply; 25+ messages in thread
From: Dave Hansen @ 2018-07-10 3:47 UTC (permalink / raw)
To: H.J. Lu
Cc: H. Peter Anvin, LKML, Andy Lutomirski, Mel Gorman, Andrew Morton,
Rik van Riel, Minchan Kim
On 07/09/2018 07:14 PM, H.J. Lu wrote:
>> I'd really want to see this reproduced without KASLR to make the oops
>> easier to read. It would also be handy to try your workload with all
>> the pedantic debugging: KASAN, slab debugging, DEBUG_PAGE_ALLOC, etc...
>> and see if it still triggers.
> How can I turn them on at boot time?
The only thing you can add at boot time is slab debugging, and it's
probably the most useless of the three that I listed since you're not
actually seeing any slab corruption.
The rest are compile-time options.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
[not found] ` <CAMe9rOrHowBX06nihdRRmEqhV8v7cs+PwVY7JYQFpUFOnHC71A@mail.gmail.com>
@ 2018-07-11 15:13 ` Dave Hansen
2018-07-11 15:40 ` H.J. Lu
0 siblings, 1 reply; 25+ messages in thread
From: Dave Hansen @ 2018-07-11 15:13 UTC (permalink / raw)
To: H.J. Lu
Cc: H. Peter Anvin, LKML, Andy Lutomirski, Mel Gorman, Andrew Morton,
Rik van Riel, Minchan Kim
On 07/11/2018 07:56 AM, H.J. Lu wrote:
> On Mon, Jul 9, 2018 at 8:47 PM, Dave Hansen <dave.hansen@intel.com> wrote:
>> On 07/09/2018 07:14 PM, H.J. Lu wrote:
>>>> I'd really want to see this reproduced without KASLR to make the oops
>>>> easier to read. It would also be handy to try your workload with all
>>>> the pedantic debugging: KASAN, slab debugging, DEBUG_PAGE_ALLOC, etc...
>>>> and see if it still triggers.
>>> How can I turn them on at boot time?
>> The only thing you can add at boot time is slab debugging, and it's
>> probably the most useless of the three that I listed since you're not
>> actually seeing any slab corruption.
>>
>> The rest are compile-time options.
> I enabled KASAN, slab debugging, DEBUG_PAGE_ALLOC and disabled
> KASLR. Machine locked up. Here is the last kernel message before locking
> up.
KASAN looks to have caught it, although it scrolled off the screen. I
can certainly imagine the oops you saw earlier being caused by stack
corruption.
Sounds like we need to reproduce this in an environment that can
actually capture a real oops. Can you share more about your workload?
I'll see if I can get it to reproduce in a VM.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-11 15:13 ` Dave Hansen
@ 2018-07-11 15:40 ` H.J. Lu
2018-07-11 16:24 ` Dave Hansen
0 siblings, 1 reply; 25+ messages in thread
From: H.J. Lu @ 2018-07-11 15:40 UTC (permalink / raw)
To: Dave Hansen
Cc: H. Peter Anvin, LKML, Andy Lutomirski, Mel Gorman, Andrew Morton,
Rik van Riel, Minchan Kim
On Wed, Jul 11, 2018 at 8:13 AM, Dave Hansen <dave.hansen@intel.com> wrote:
> On 07/11/2018 07:56 AM, H.J. Lu wrote:
>> On Mon, Jul 9, 2018 at 8:47 PM, Dave Hansen <dave.hansen@intel.com> wrote:
>>> On 07/09/2018 07:14 PM, H.J. Lu wrote:
>>>>> I'd really want to see this reproduced without KASLR to make the oops
>>>>> easier to read. It would also be handy to try your workload with all
>>>>> the pedantic debugging: KASAN, slab debugging, DEBUG_PAGE_ALLOC, etc...
>>>>> and see if it still triggers.
>>>> How can I turn them on at boot time?
>>> The only thing you can add at boot time is slab debugging, and it's
>>> probably the most useless of the three that I listed since you're not
>>> actually seeing any slab corruption.
>>>
>>> The rest are compile-time options.
>> I enabled KASAN, slab debugging, DEBUG_PAGE_ALLOC and disabled
>> KASLR. Machine locked up. Here is the last kernel message before locking
>> up.
>
> KASAN looks to have caught it, although it scrolled off the screen. I
> can certainly imagine the oops you saw earlier being caused by stack
> corruption.
>
> Sounds like we need to reproduce this in an environment that can
> actually capture a real oops. Can you share more about your workload?
> I'll see if I can get it to reproduce in a VM.
This is a quad-core machine with HT and 6 GB RAM. The workload is
x32 GCC build and test with "make -j8". The bug is triggered during GCC
test after a couple hours. I have a script to set up my workload:
https://github.com/hjl-tools/gcc-regression
with a cron job
# It takes about 3 hour to bootstrap x86-64 GCC and 3 hour to run tests,
TIMEOUT=480
# Run it every hour,
30 * * * * /export/gnu/import/git/gcc-test-x32/gcc-build -mx32
--with-pic > /dev/null 2>&1
--
H.J.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-11 15:40 ` H.J. Lu
@ 2018-07-11 16:24 ` Dave Hansen
2018-07-11 16:29 ` H.J. Lu
0 siblings, 1 reply; 25+ messages in thread
From: Dave Hansen @ 2018-07-11 16:24 UTC (permalink / raw)
To: H.J. Lu
Cc: H. Peter Anvin, LKML, Andy Lutomirski, Mel Gorman, Andrew Morton,
Rik van Riel, Minchan Kim
On 07/11/2018 08:40 AM, H.J. Lu wrote:
> This is a quad-core machine with HT and 6 GB RAM. The workload is
> x32 GCC build and test with "make -j8". The bug is triggered during GCC
> test after a couple hours. I have a script to set up my workload:
>
> https://github.com/hjl-tools/gcc-regression
>
> with a cron job
>
> # It takes about 3 hour to bootstrap x86-64 GCC and 3 hour to run tests,
> TIMEOUT=480
> # Run it every hour,
> 30 * * * * /export/gnu/import/git/gcc-test-x32/gcc-build -mx32
> --with-pic > /dev/null 2>&1
Oh, fun, one of those.
How long does it take to reproduce?
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-11 16:24 ` Dave Hansen
@ 2018-07-11 16:29 ` H.J. Lu
2018-07-11 16:49 ` Dave Hansen
0 siblings, 1 reply; 25+ messages in thread
From: H.J. Lu @ 2018-07-11 16:29 UTC (permalink / raw)
To: Dave Hansen
Cc: H. Peter Anvin, LKML, Andy Lutomirski, Mel Gorman, Andrew Morton,
Rik van Riel, Minchan Kim
On Wed, Jul 11, 2018 at 9:24 AM, Dave Hansen <dave.hansen@intel.com> wrote:
> On 07/11/2018 08:40 AM, H.J. Lu wrote:
>> This is a quad-core machine with HT and 6 GB RAM. The workload is
>> x32 GCC build and test with "make -j8". The bug is triggered during GCC
>> test after a couple hours. I have a script to set up my workload:
>>
>> https://github.com/hjl-tools/gcc-regression
>>
>> with a cron job
>>
>> # It takes about 3 hour to bootstrap x86-64 GCC and 3 hour to run tests,
>> TIMEOUT=480
>> # Run it every hour,
>> 30 * * * * /export/gnu/import/git/gcc-test-x32/gcc-build -mx32
>> --with-pic > /dev/null 2>&1
>
> Oh, fun, one of those.
>
> How long does it take to reproduce?
About 5 hours.
--
H.J.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-11 16:29 ` H.J. Lu
@ 2018-07-11 16:49 ` Dave Hansen
2018-07-11 16:53 ` H.J. Lu
0 siblings, 1 reply; 25+ messages in thread
From: Dave Hansen @ 2018-07-11 16:49 UTC (permalink / raw)
To: H.J. Lu
Cc: H. Peter Anvin, LKML, Andy Lutomirski, Mel Gorman, Andrew Morton,
Rik van Riel, Minchan Kim
On 07/11/2018 09:29 AM, H.J. Lu wrote:
>>> # It takes about 3 hour to bootstrap x86-64 GCC and 3 hour to run tests,
>>> TIMEOUT=480
>>> # Run it every hour,
>>> 30 * * * * /export/gnu/import/git/gcc-test-x32/gcc-build -mx32
>>> --with-pic > /dev/null 2>&1
>> Oh, fun, one of those.
>>
>> How long does it take to reproduce?
> About 5 hours.
It would be much appreciated if you can get a console of some kind on
that system and try to get a full KASAN oops out of it. Serial, usb
debug, or maybe netconsole would probably work.
I'll also try to get it running on a big system and see if it triggers
faster there.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-11 16:49 ` Dave Hansen
@ 2018-07-11 16:53 ` H.J. Lu
[not found] ` <CAMe9rOpV89jWhvAZtqOJnc0eXzRYiLF5pLWRcdb7-kKLigj4rQ@mail.gmail.com>
0 siblings, 1 reply; 25+ messages in thread
From: H.J. Lu @ 2018-07-11 16:53 UTC (permalink / raw)
To: Dave Hansen
Cc: H. Peter Anvin, LKML, Andy Lutomirski, Mel Gorman, Andrew Morton,
Rik van Riel, Minchan Kim
On Wed, Jul 11, 2018 at 9:49 AM, Dave Hansen <dave.hansen@intel.com> wrote:
> On 07/11/2018 09:29 AM, H.J. Lu wrote:
>>>> # It takes about 3 hour to bootstrap x86-64 GCC and 3 hour to run tests,
>>>> TIMEOUT=480
>>>> # Run it every hour,
>>>> 30 * * * * /export/gnu/import/git/gcc-test-x32/gcc-build -mx32
>>>> --with-pic > /dev/null 2>&1
>>> Oh, fun, one of those.
>>>
>>> How long does it take to reproduce?
>> About 5 hours.
>
> It would be much appreciated if you can get a console of some kind on
> that system and try to get a full KASAN oops out of it. Serial, usb
> debug, or maybe netconsole would probably work.
I don't have serial ports. I will try netconsole.
> I'll also try to get it running on a big system and see if it triggers
> faster there.
I have seen it on machines with various amounts of cores and RAMs.
It triggers the fastest on 8 cores with 6GB RAM reliably.
--
H.J.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
[not found] ` <CAMe9rOpV89jWhvAZtqOJnc0eXzRYiLF5pLWRcdb7-kKLigj4rQ@mail.gmail.com>
@ 2018-07-11 17:36 ` Andy Lutomirski
2018-07-11 17:43 ` Dave Hansen
2018-07-11 17:50 ` Dave Hansen
2 siblings, 0 replies; 25+ messages in thread
From: Andy Lutomirski @ 2018-07-11 17:36 UTC (permalink / raw)
To: H.J. Lu
Cc: Dave Hansen, H. Peter Anvin, LKML, Andy Lutomirski, Mel Gorman,
Andrew Morton, Rik van Riel, Minchan Kim
On Wed, Jul 11, 2018 at 10:29 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Wed, Jul 11, 2018 at 9:53 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>> On Wed, Jul 11, 2018 at 9:49 AM, Dave Hansen <dave.hansen@intel.com> wrote:
>>> On 07/11/2018 09:29 AM, H.J. Lu wrote:
>>>>>> # It takes about 3 hour to bootstrap x86-64 GCC and 3 hour to run tests,
>>>>>> TIMEOUT=480
>>>>>> # Run it every hour,
>>>>>> 30 * * * * /export/gnu/import/git/gcc-test-x32/gcc-build -mx32
>>>>>> --with-pic > /dev/null 2>&1
>>>>> Oh, fun, one of those.
>>>>>
>>>>> How long does it take to reproduce?
>>>> About 5 hours.
>>>
>>> It would be much appreciated if you can get a console of some kind on
>>> that system and try to get a full KASAN oops out of it. Serial, usb
>>> debug, or maybe netconsole would probably work.
>>
>> I don't have serial ports. I will try netconsole.
>>
>>> I'll also try to get it running on a big system and see if it triggers
>>> faster there.
>>
>> I have seen it on machines with various amounts of cores and RAMs.
>> It triggers the fastest on 8 cores with 6GB RAM reliably.
>
> Here is the first kernel message.
I find myself wondering if this is somehow the same issue as below:
https://syzkaller.appspot.com/bug?id=2c33dff2a2138d709b43671603dc01d65b28a689
We could plausibly have a problem with the entry code, but I don't see
anything relevant in the changelog.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
[not found] ` <CAMe9rOpV89jWhvAZtqOJnc0eXzRYiLF5pLWRcdb7-kKLigj4rQ@mail.gmail.com>
2018-07-11 17:36 ` Andy Lutomirski
@ 2018-07-11 17:43 ` Dave Hansen
2018-07-11 17:50 ` H.J. Lu
2018-07-11 17:50 ` Dave Hansen
2 siblings, 1 reply; 25+ messages in thread
From: Dave Hansen @ 2018-07-11 17:43 UTC (permalink / raw)
To: H.J. Lu
Cc: H. Peter Anvin, LKML, Andy Lutomirski, Mel Gorman, Andrew Morton,
Rik van Riel, Minchan Kim
On 07/11/2018 10:29 AM, H.J. Lu wrote:
>> I have seen it on machines with various amounts of cores and RAMs.
>> It triggers the fastest on 8 cores with 6GB RAM reliably.
> Here is the first kernel message.
Does it trigger better with more RAM or less?
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
[not found] ` <CAMe9rOpV89jWhvAZtqOJnc0eXzRYiLF5pLWRcdb7-kKLigj4rQ@mail.gmail.com>
2018-07-11 17:36 ` Andy Lutomirski
2018-07-11 17:43 ` Dave Hansen
@ 2018-07-11 17:50 ` Dave Hansen
2018-07-11 18:31 ` Dave Jones
2 siblings, 1 reply; 25+ messages in thread
From: Dave Hansen @ 2018-07-11 17:50 UTC (permalink / raw)
To: H.J. Lu
Cc: H. Peter Anvin, LKML, Andy Lutomirski, Mel Gorman, Andrew Morton,
Rik van Riel, Minchan Kim
On 07/11/2018 10:29 AM, H.J. Lu wrote:
>> I have seen it on machines with various amounts of cores and RAMs.
>> It triggers the fastest on 8 cores with 6GB RAM reliably.
> Here is the first kernel message.
This looks like random corruption again. It's probably a bogus 'struct
page' that fails the move_freepages() pfn_valid() checks. I'm too lazy
to go reproduce the likely stack trace (not sure why it didn't show up
on your screen), but this could just be another symptom of the same
issue that caused the TLB batching oops.
My money is on this being some kind of odd stack corruption, maybe
interrupt-induced, but that's a total guess at this point.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-11 17:43 ` Dave Hansen
@ 2018-07-11 17:50 ` H.J. Lu
0 siblings, 0 replies; 25+ messages in thread
From: H.J. Lu @ 2018-07-11 17:50 UTC (permalink / raw)
To: Dave Hansen
Cc: H. Peter Anvin, LKML, Andy Lutomirski, Mel Gorman, Andrew Morton,
Rik van Riel, Minchan Kim
On Wed, Jul 11, 2018 at 10:43 AM, Dave Hansen <dave.hansen@intel.com> wrote:
> On 07/11/2018 10:29 AM, H.J. Lu wrote:
>>> I have seen it on machines with various amounts of cores and RAMs.
>>> It triggers the fastest on 8 cores with 6GB RAM reliably.
>> Here is the first kernel message.
>
> Does it trigger better with more RAM or less?
It triggers much more with 6GB RAM than with 32GB and 64GB.
--
H.J.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-11 17:50 ` Dave Hansen
@ 2018-07-11 18:31 ` Dave Jones
2018-07-11 23:07 ` Andy Lutomirski
0 siblings, 1 reply; 25+ messages in thread
From: Dave Jones @ 2018-07-11 18:31 UTC (permalink / raw)
To: Dave Hansen
Cc: H.J. Lu, H. Peter Anvin, LKML, Andy Lutomirski, Mel Gorman,
Andrew Morton, Rik van Riel, Minchan Kim
On Wed, Jul 11, 2018 at 10:50:22AM -0700, Dave Hansen wrote:
> On 07/11/2018 10:29 AM, H.J. Lu wrote:
> >> I have seen it on machines with various amounts of cores and RAMs.
> >> It triggers the fastest on 8 cores with 6GB RAM reliably.
> > Here is the first kernel message.
>
> This looks like random corruption again. It's probably a bogus 'struct
> page' that fails the move_freepages() pfn_valid() checks. I'm too lazy
> to go reproduce the likely stack trace (not sure why it didn't show up
> on your screen), but this could just be another symptom of the same
> issue that caused the TLB batching oops.
>
> My money is on this being some kind of odd stack corruption, maybe
> interrupt-induced, but that's a total guess at this point.
So, maybe related.. I reported this to linux-mm a few days ago:
When I ran an rsync on my machine I use for backups, it eventually
hits this trace..
kernel BUG at mm/page_alloc.c:2016!
invalid opcode: 0000 [#1] SMP RIP: move_freepages_block+0x120/0x2d0
CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.18.0-rc4-backup+ #1
Hardware name: ASUS All Series/Z97-DELUXE, BIOS 2602 08/18/2015
RIP: 0010:move_freepages_block+0x120/0x2d0
Code: 05 48 01 c8 74 3b f6 00 02 74 36 48 8b 03 48 c1 e8 3e 48 8d 0c 40 48 8b 86 c0 7f 00 00 48 c1 e8 3e 48 8d 04 40 48 39 c8 74 17 <0f> 0b 45 31 f6 48 83 c4 28 44 89 f0 5b 5d 41
5c 41 5d 41 5e 41 5f
RSP: 0018:ffff88043fac3af8 EFLAGS: 00010093
RAX: 0000000000000000 RBX: ffffea0002e20000 RCX: 0000000000000003
RDX: 0000000000000000 RSI: ffffea0002e20000 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffff88043fac3b5c R09: ffffffff9295e110
R10: ffff88043fdf4000 R11: ffffea0002e20008 R12: ffffea0002e20000
R13: ffffffff9295dd40 R14: 0000000000000008 R15: ffffea0002e27fc0
FS: 0000000000000000(0000) GS:ffff88043fac0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2a75f71fe8 CR3: 00000001e380f006 CR4: 00000000001606e0
Call Trace:
<IRQ>
? lock_acquire+0xe6/0x1dc
steal_suitable_fallback+0x152/0x1a0
get_page_from_freelist+0x1029/0x1650
? free_debug_processing+0x271/0x410
__alloc_pages_nodemask+0x111/0x310
page_frag_alloc+0x74/0x120
__netdev_alloc_skb+0x95/0x110
e1000_alloc_rx_buffers+0x225/0x2b0
e1000_clean_rx_irq+0x2ee/0x450
e1000e_poll+0x7c/0x2e0
net_rx_action+0x273/0x4d0
__do_softirq+0xc6/0x4d6
irq_exit+0xbb/0xc0
do_IRQ+0x60/0x110
common_interrupt+0xf/0xf
</IRQ>
RIP: 0010:cpuidle_enter_state+0xb5/0x390
Code: 89 04 24 0f 1f 44 00 00 31 ff e8 86 26 64 ff 80 7c 24 0f 00 0f 85 fb 01 00 00 e8 66 02 66 ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <48> 8b 0c 24 4c 29 f9 48 89 c8 48 c1 f9 3f 48
f7 ea b8 ff ff ff 7f
RSP: 0018:ffffc900000abe70 EFLAGS: 00000202
ORIG_RAX: ffffffffffffffdc
RAX: ffff880107fe8040 RBX: 0000000000000003 RCX: 0000000000000001
RDX: 20c49ba5e353f7cf RSI: 0000000000000001 RDI: ffff880107fe8040
RBP: ffff88043fae8c20 R08: 0000000000000001 R09: 0000000000000018
R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff928fb7d8
R13: 0000000000000003 R14: 0000000000000003 R15: 0000015e55aecf23
do_idle+0x128/0x230
cpu_startup_entry+0x6f/0x80
start_secondary+0x192/0x1f0
secondary_startup_64+0xa5/0xb0
NMI watchdog: Watchdog detected hard LOCKUP on cpu 4
Everything then locks up & rebooots.
It's fairly reproduceable, though every time I run it my rsync gets further, and eventually I suspect it
won't create enough load to reproduce.
2006 #ifndef CONFIG_HOLES_IN_ZONE
2007 /*
2008 * page_zone is not safe to call in this context when
2009 * CONFIG_HOLES_IN_ZONE is set. This bug check is probably redundant
2010 * anyway as we check zone boundaries in move_freepages_block().
2011 * Remove at a later date when no bug reports exist related to
2012 * grouping pages by mobility
2013 */
2014 VM_BUG_ON(pfn_valid(page_to_pfn(start_page)) &&
2015 pfn_valid(page_to_pfn(end_page)) &&
2016 page_zone(start_page) != page_zone(end_page));
2017 #endif
2018
I could trigger it fairly quickly last week, but it seemed dependant on just how much
rsync is actually transferring. (There are millions of files, and only a few thousand had changed)
When there's nothing changed, the rsync was running to completion every time.
Dave
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-11 18:31 ` Dave Jones
@ 2018-07-11 23:07 ` Andy Lutomirski
2018-07-11 23:14 ` Dave Hansen
0 siblings, 1 reply; 25+ messages in thread
From: Andy Lutomirski @ 2018-07-11 23:07 UTC (permalink / raw)
To: Dave Jones
Cc: Dave Hansen, H.J. Lu, H. Peter Anvin, LKML, Andy Lutomirski,
Mel Gorman, Andrew Morton, Rik van Riel, Minchan Kim
> On Jul 11, 2018, at 11:31 AM, Dave Jones <davej@codemonkey.org.uk> wrote:
>
>> On Wed, Jul 11, 2018 at 10:50:22AM -0700, Dave Hansen wrote:
>> On 07/11/2018 10:29 AM, H.J. Lu wrote:
>>>> I have seen it on machines with various amounts of cores and RAMs.
>>>> It triggers the fastest on 8 cores with 6GB RAM reliably.
>>> Here is the first kernel message.
>>
>> This looks like random corruption again. It's probably a bogus 'struct
>> page' that fails the move_freepages() pfn_valid() checks. I'm too lazy
>> to go reproduce the likely stack trace (not sure why it didn't show up
>> on your screen), but this could just be another symptom of the same
>> issue that caused the TLB batching oops.
>>
>> My money is on this being some kind of odd stack corruption, maybe
>> interrupt-induced, but that's a total guess at this point.
>
> So, maybe related.. I reported this to linux-mm a few days ago:
>
> When I ran an rsync on my machine I use for backups, it eventually
> hits this trace..
>
> kernel BUG at mm/page_alloc.c:2016!
> invalid opcode: 0000 [#1] SMP RIP: move_freepages_block+0x120/0x2d0
> CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.18.0-rc4-backup+ #1
> Hardware name: ASUS All Series/Z97-DELUXE, BIOS 2602 08/18/2015
> RIP: 0010:move_freepages_block+0x120/0x2d0
> Code: 05 48 01 c8 74 3b f6 00 02 74 36 48 8b 03 48 c1 e8 3e 48 8d 0c 40 48 8b 86 c0 7f 00 00 48 c1 e8 3e 48 8d 04 40 48 39 c8 74 17 <0f> 0b 45 31 f6 48 83 c4 28 44 89 f0 5b 5d 41
> 5c 41 5d 41 5e 41 5f
> RSP: 0018:ffff88043fac3af8 EFLAGS: 00010093
> RAX: 0000000000000000 RBX: ffffea0002e20000 RCX: 0000000000000003
> RDX: 0000000000000000 RSI: ffffea0002e20000 RDI: 0000000000000000
> RBP: 0000000000000000 R08: ffff88043fac3b5c R09: ffffffff9295e110
> R10: ffff88043fdf4000 R11: ffffea0002e20008 R12: ffffea0002e20000
> R13: ffffffff9295dd40 R14: 0000000000000008 R15: ffffea0002e27fc0
> FS: 0000000000000000(0000) GS:ffff88043fac0000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f2a75f71fe8 CR3: 00000001e380f006 CR4: 00000000001606e0
> Call Trace:
> <IRQ>
> ? lock_acquire+0xe6/0x1dc
> steal_suitable_fallback+0x152/0x1a0
> get_page_from_freelist+0x1029/0x1650
> ? free_debug_processing+0x271/0x410
> __alloc_pages_nodemask+0x111/0x310
> page_frag_alloc+0x74/0x120
> __netdev_alloc_skb+0x95/0x110
> e1000_alloc_rx_buffers+0x225/0x2b0
> e1000_clean_rx_irq+0x2ee/0x450
> e1000e_poll+0x7c/0x2e0
> net_rx_action+0x273/0x4d0
> __do_softirq+0xc6/0x4d6
> irq_exit+0xbb/0xc0
> do_IRQ+0x60/0x110
> common_interrupt+0xf/0xf
> </IRQ>
> RIP: 0010:cpuidle_enter_state+0xb5/0x390
> Code: 89 04 24 0f 1f 44 00 00 31 ff e8 86 26 64 ff 80 7c 24 0f 00 0f 85 fb 01 00 00 e8 66 02 66 ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <48> 8b 0c 24 4c 29 f9 48 89 c8 48 c1 f9 3f 48
> f7 ea b8 ff ff ff 7f
> RSP: 0018:ffffc900000abe70 EFLAGS: 00000202
> ORIG_RAX: ffffffffffffffdc
> RAX: ffff880107fe8040 RBX: 0000000000000003 RCX: 0000000000000001
> RDX: 20c49ba5e353f7cf RSI: 0000000000000001 RDI: ffff880107fe8040
> RBP: ffff88043fae8c20 R08: 0000000000000001 R09: 0000000000000018
> R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff928fb7d8
> R13: 0000000000000003 R14: 0000000000000003 R15: 0000015e55aecf23
> do_idle+0x128/0x230
> cpu_startup_entry+0x6f/0x80
> start_secondary+0x192/0x1f0
> secondary_startup_64+0xa5/0xb0
> NMI watchdog: Watchdog detected hard LOCKUP on cpu 4
>
> Everything then locks up & rebooots.
>
> It's fairly reproduceable, though every time I run it my rsync gets further, and eventually I suspect it
> won't create enough load to reproduce.
>
> 2006 #ifndef CONFIG_HOLES_IN_ZONE
> 2007 /*
> 2008 * page_zone is not safe to call in this context when
> 2009 * CONFIG_HOLES_IN_ZONE is set. This bug check is probably redundant
> 2010 * anyway as we check zone boundaries in move_freepages_block().
> 2011 * Remove at a later date when no bug reports exist related to
> 2012 * grouping pages by mobility
> 2013 */
> 2014 VM_BUG_ON(pfn_valid(page_to_pfn(start_page)) &&
> 2015 pfn_valid(page_to_pfn(end_page)) &&
> 2016 page_zone(start_page) != page_zone(end_page));
> 2017 #endif
> 2018
>
>
> I could trigger it fairly quickly last week, but it seemed dependant on just how much
> rsync is actually transferring. (There are millions of files, and only a few thousand had changed)
>
> When there's nothing changed, the rsync was running to completion every time.
>
>
Could the cause be an overflow of the IRQ stack? I’ve been meaning to put guard pages on all the special stacks for a while. Let me see if I can do that in the next couple days.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-11 23:07 ` Andy Lutomirski
@ 2018-07-11 23:14 ` Dave Hansen
2018-07-12 14:44 ` H.J. Lu
0 siblings, 1 reply; 25+ messages in thread
From: Dave Hansen @ 2018-07-11 23:14 UTC (permalink / raw)
To: Andy Lutomirski, Dave Jones
Cc: H.J. Lu, H. Peter Anvin, LKML, Andy Lutomirski, Mel Gorman,
Andrew Morton, Rik van Riel, Minchan Kim
On 07/11/2018 04:07 PM, Andy Lutomirski wrote:
> Could the cause be an overflow of the IRQ stack? I’ve been meaning
> to put guard pages on all the special stacks for a while. Let me see
> if I can do that in the next couple days.
But what would that overflow into? Wouldn't it most likely be another
interrupt stack since they're all allocated together?
This looks more like thread stack corruption.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-11 23:14 ` Dave Hansen
@ 2018-07-12 14:44 ` H.J. Lu
2018-07-12 23:28 ` H.J. Lu
0 siblings, 1 reply; 25+ messages in thread
From: H.J. Lu @ 2018-07-12 14:44 UTC (permalink / raw)
To: Dave Hansen
Cc: Andy Lutomirski, Dave Jones, H. Peter Anvin, LKML,
Andy Lutomirski, Mel Gorman, Andrew Morton, Rik van Riel,
Minchan Kim
On Wed, Jul 11, 2018 at 4:14 PM, Dave Hansen <dave.hansen@intel.com> wrote:
> On 07/11/2018 04:07 PM, Andy Lutomirski wrote:
>> Could the cause be an overflow of the IRQ stack? I’ve been meaning
>> to put guard pages on all the special stacks for a while. Let me see
>> if I can do that in the next couple days.
>
> But what would that overflow into? Wouldn't it most likely be another
> interrupt stack since they're all allocated together?
>
> This looks more like thread stack corruption.
I tried netconsole and got this:
[29369.552998] ------------[ cut here ]------------
[29369.560996] kernel BUG at mm/page_alloc.c:2019!
[29369.568980] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC KASAN PTI
[29369.576892] Modules linked in: netconsole xt_CHECKSUM
ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns
nf_conntrack_broadcast xt_CT ip6t_rpfilter ip6t_REJECT nf_reject_ipv6
xt_conntrack devlink ip_set nfnetlink ebtable_nat ebtable_broute
bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6
nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat
nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack
libcrc32c iptable_mangle iptable_raw iptable_security ebtable_filter
ebtables ip6table_filter ip6_tables sunrpc vfat fat intel_powerclamp
coretemp kvm_intel kvm irqbypass intel_cstate intel_uncore
snd_hda_codec_realtek snd_hda_codec_generic iTCO_wdt
iTCO_vendor_support gpio_ich snd_hda_intel joydev snd_hda_codec
snd_hda_core snd_hwdep mxm_wmi snd_seq snd_seq_device
[29369.627745] snd_pcm pcspkr snd_timer snd i2c_i801 soundcore
lpc_ich i5500_temp i7core_edac shpchp wmi acpi_cpufreq ata_generic
pata_acpi radeon crc32c_intel i2c_algo_bit drm_kms_helper
firewire_ohci firewire_core ttm crc_itu_t drm e1000e pata_marvell
[29369.645472] CPU: 1 PID: 3896 Comm: expect Tainted: G I
4.17.5+ #7
[29369.654333] Hardware name: /DX58SO, BIOS
SOX5810J.86A.5600.2013.0729.2250 07/29/2013
[29369.663320] RIP: 0010:move_freepages_block+0x246/0x4b0
[29369.672238] RSP: 0018:ffff8800b61f7178 EFLAGS: 00010002
[29369.681064] RAX: ffff8801af3d7000 RBX: ffffea00033c8000 RCX: 0000000000000000
[29369.690011] RDX: dffffc0000000000 RSI: ffffea00033cc000 RDI: ffffffff831d8ec0
[29369.698992] RBP: ffff8801af3d7680 R08: ffff8800b61f73c8 R09: ffffed0035e7af78
[29369.708025] R10: ffffed0035e7af78 R11: ffff8801af3d7bc3 R12: ffff8800b61f7228
before machine looked up.
--
H.J.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-12 14:44 ` H.J. Lu
@ 2018-07-12 23:28 ` H.J. Lu
2018-07-14 2:08 ` Andy Lutomirski
0 siblings, 1 reply; 25+ messages in thread
From: H.J. Lu @ 2018-07-12 23:28 UTC (permalink / raw)
To: Dave Hansen
Cc: Andy Lutomirski, Dave Jones, H. Peter Anvin, LKML,
Andy Lutomirski, Mel Gorman, Andrew Morton, Rik van Riel,
Minchan Kim
On Thu, Jul 12, 2018 at 7:44 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Wed, Jul 11, 2018 at 4:14 PM, Dave Hansen <dave.hansen@intel.com> wrote:
>> On 07/11/2018 04:07 PM, Andy Lutomirski wrote:
>>> Could the cause be an overflow of the IRQ stack? I’ve been meaning
>>> to put guard pages on all the special stacks for a while. Let me see
>>> if I can do that in the next couple days.
>>
>> But what would that overflow into? Wouldn't it most likely be another
>> interrupt stack since they're all allocated together?
>>
>> This looks more like thread stack corruption.
>
> I tried netconsole and got this:
>
> [29369.552998] ------------[ cut here ]------------
> [29369.560996] kernel BUG at mm/page_alloc.c:2019!
> [29369.568980] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC KASAN PTI
> [29369.576892] Modules linked in: netconsole xt_CHECKSUM
> ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns
> nf_conntrack_broadcast xt_CT ip6t_rpfilter ip6t_REJECT nf_reject_ipv6
> xt_conntrack devlink ip_set nfnetlink ebtable_nat ebtable_broute
> bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6
> nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat
> nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack
> libcrc32c iptable_mangle iptable_raw iptable_security ebtable_filter
> ebtables ip6table_filter ip6_tables sunrpc vfat fat intel_powerclamp
> coretemp kvm_intel kvm irqbypass intel_cstate intel_uncore
> snd_hda_codec_realtek snd_hda_codec_generic iTCO_wdt
> iTCO_vendor_support gpio_ich snd_hda_intel joydev snd_hda_codec
> snd_hda_core snd_hwdep mxm_wmi snd_seq snd_seq_device
> [29369.627745] snd_pcm pcspkr snd_timer snd i2c_i801 soundcore
> lpc_ich i5500_temp i7core_edac shpchp wmi acpi_cpufreq ata_generic
> pata_acpi radeon crc32c_intel i2c_algo_bit drm_kms_helper
> firewire_ohci firewire_core ttm crc_itu_t drm e1000e pata_marvell
> [29369.645472] CPU: 1 PID: 3896 Comm: expect Tainted: G I
> 4.17.5+ #7
> [29369.654333] Hardware name: /DX58SO, BIOS
> SOX5810J.86A.5600.2013.0729.2250 07/29/2013
> [29369.663320] RIP: 0010:move_freepages_block+0x246/0x4b0
> [29369.672238] RSP: 0018:ffff8800b61f7178 EFLAGS: 00010002
> [29369.681064] RAX: ffff8801af3d7000 RBX: ffffea00033c8000 RCX: 0000000000000000
> [29369.690011] RDX: dffffc0000000000 RSI: ffffea00033cc000 RDI: ffffffff831d8ec0
> [29369.698992] RBP: ffff8801af3d7680 R08: ffff8800b61f73c8 R09: ffffed0035e7af78
> [29369.708025] R10: ffffed0035e7af78 R11: ffff8801af3d7bc3 R12: ffff8800b61f7228
>
> before machine looked up.
4.17.6:
[24707.152017] kernel BUG at mm/page_alloc.c:2019!
[24707.157304] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC KASAN PTI
[24707.162573] Modules linked in: netconsole xt_CHECKSUM
ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns
nf_conntrack_broadcast xt_CT ip6t_rpfilter ip6t_REJECT nf_reject_ipv6
xt_conntrack devlink ip_set nfnetlink ebtable_nat ebtable_broute
bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6
nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat
nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack
libcrc32c iptable_mangle iptable_raw iptable_security ebtable_filter
ebtables ip6table_filter ip6_tables sunrpc vfat fat intel_powerclamp
coretemp kvm_intel kvm irqbypass intel_cstate intel_uncore
snd_hda_codec_realtek iTCO_wdt mxm_wmi gpio_ich iTCO_vendor_support
snd_hda_codec_generic snd_hda_intel snd_hda_codec joydev snd_hda_core
snd_hwdep snd_seq snd_seq_device
[24707.196565] snd_pcm pcspkr snd_timer snd i2c_i801 lpc_ich
soundcore i7core_edac i5500_temp shpchp wmi acpi_cpufreq ata_generic
pata_acpi radeon i2c_algo_bit drm_kms_helper crc32c_intel
firewire_ohci ttm firewire_core crc_itu_t drm pata_marvell e1000e
[24707.208944] CPU: 3 PID: 23661 Comm: cc1plus Tainted: G I
4.17.6+ #11
[24707.215277] Hardware name: /DX58SO, BIOS
SOX5810J.86A.5600.2013.0729.2250 07/29/2013
[24707.221715] RIP: 0010:move_freepages_block+0x246/0x4b0
[24707.228145] RSP: 0018:ffff880170387818 EFLAGS: 00010002
[24707.234614] RAX: ffff8801af3d7000 RBX: ffffea00033c8000 RCX: 0000000000000000
[24707.241158] RDX: dffffc0000000000 RSI: ffffea00033cf000 RDI: ffffffff831d9380
[24707.247735] RBP: ffff8801af3d7680 R08: ffff880170387a68 R09: ffffed0035e7af78
[24707.254233] R10: ffffed0035e7af78 R11: ffff8801af3d7bc3 R12: ffff8801703878c8
[24707.260650] R13: ffff8801af3d7680 R14: 0000000000000800 R15: 0000000000000000
[24707.267015] FS: 00007f18c4310c80(0000) GS:ffff880176ec0000(0000)
knlGS:0000000000000000
[24707.273381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[24707.279656] CR2: 00007f18c42a0000 CR3: 000000016b982000 CR4: 00000000000006e0
[24707.285919] Call Trace:
[24707.292076] ? unwind_next_frame+0x793/0x850
[24707.298199] steal_suitable_fallback+0x1f2/0x2a0
[24707.304344] ? ftrace_ops_trampoline+0x4c/0x80
[24707.310425] ? move_freepages_block+0x4b0/0x4b0
[24707.316423] ? is_bpf_text_address+0xa/0x20
[24707.322325] ? kernel_text_address+0x100/0x110
[24707.328105] ? find_suitable_fallback+0xd0/0x100
[24707.333828] get_page_from_freelist+0x189c/0x2540
[24707.339471] ? __isolate_free_page+0x2c0/0x2c0
[24707.345011] ? unlazy_walk+0xb8/0x160
[24707.350539] __alloc_pages_nodemask+0x1b0/0x3c0
[24707.355912] ? __alloc_pages_slowpath+0x1240/0x1240
[24707.361193] ? path_mountpoint+0x8d0/0x8d0
[24707.366500] ? deactivate_slab.isra.61+0x200/0x500
[24707.371712] ? policy_node+0x56/0x60
[24707.376838] new_slab+0x288/0x790
[24707.381853] ? __kasan_slab_free+0x143/0x180
[24707.386795] ___slab_alloc+0x3b4/0x550
[24707.391606] ? getname_flags+0x6c/0x2a0
[24707.396435] ? getname_flags+0x6c/0x2a0
[24707.401241] __slab_alloc+0x1c/0x30
[24707.406059] kmem_cache_alloc+0x223/0x260
[24707.410913] getname_flags+0x6c/0x2a0
[24707.415772] ? __ia32_sys_fstat+0x30/0x30
[24707.420655] user_path_at_empty+0x1d/0x40
[24707.425551] vfs_statx+0xb9/0x140
[24707.430489] ? vfs_statx_fd+0x80/0x80
[24707.435307] __do_sys_newlstat+0x77/0xd0
[24707.440144] ? __ia32_sys_newstat+0x30/0x30
[24707.445005] ? do_sys_open+0x16b/0x290
[24707.449881] do_syscall_64+0x7d/0x1c0
[24707.454778] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[24707.459716] RIP: 0033:0x7f18c43ff049
[24707.464670] RSP: 002b:00007fff75c7aaa8 EFLAGS: 00000246 ORIG_RAX:
0000000000000006
[24707.469625] RAX: ffffffffffffffda RBX: 00007fff75c7bbd0 RCX: 00007f18c43ff049
--
H.J.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-12 23:28 ` H.J. Lu
@ 2018-07-14 2:08 ` Andy Lutomirski
2018-07-16 16:05 ` H.J. Lu
0 siblings, 1 reply; 25+ messages in thread
From: Andy Lutomirski @ 2018-07-14 2:08 UTC (permalink / raw)
To: H.J. Lu
Cc: Dave Hansen, Dave Jones, H. Peter Anvin, LKML, Andy Lutomirski,
Mel Gorman, Andrew Morton, Rik van Riel, Minchan Kim
I'm not at all convinced that this is the problem, but the series here
will give a better diagnostic if the issue really is an IRQ stack
overflow:
https://git.kernel.org/pub/scm/linux/kernel/git/luto/linux.git/log/?h=x86/guard_pages
(link currently broken. should work soon.)
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-14 2:08 ` Andy Lutomirski
@ 2018-07-16 16:05 ` H.J. Lu
2018-07-20 21:35 ` Andy Lutomirski
0 siblings, 1 reply; 25+ messages in thread
From: H.J. Lu @ 2018-07-16 16:05 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Dave Hansen, Dave Jones, H. Peter Anvin, LKML, Andy Lutomirski,
Mel Gorman, Andrew Morton, Rik van Riel, Minchan Kim
[-- Attachment #1: Type: text/plain, Size: 383 bytes --]
On Fri, Jul 13, 2018 at 7:08 PM, Andy Lutomirski <luto@amacapital.net> wrote:
> I'm not at all convinced that this is the problem, but the series here
> will give a better diagnostic if the issue really is an IRQ stack
> overflow:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/luto/linux.git/log/?h=x86/guard_pages
>
This kernel won't boot with this configure file.
--
H.J.
[-- Attachment #2: config.debug.xz --]
[-- Type: application/x-xz, Size: 42236 bytes --]
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-16 16:05 ` H.J. Lu
@ 2018-07-20 21:35 ` Andy Lutomirski
2018-07-23 14:51 ` H.J. Lu
0 siblings, 1 reply; 25+ messages in thread
From: Andy Lutomirski @ 2018-07-20 21:35 UTC (permalink / raw)
To: H.J. Lu
Cc: Dave Hansen, Dave Jones, H. Peter Anvin, LKML, Andy Lutomirski,
Mel Gorman, Andrew Morton, Rik van Riel, Minchan Kim
> On Jul 16, 2018, at 6:05 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>
>> On Fri, Jul 13, 2018 at 7:08 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>> I'm not at all convinced that this is the problem, but the series here
>> will give a better diagnostic if the issue really is an IRQ stack
>> overflow:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/luto/linux.git/log/?h=x86/guard_pages
>>
>
> This kernel won't boot with this configure file.
Bah humbug. You’ll need to turn KASAN off (and maybe even compile it out entirely) if you want to test this. I need to bug the KASAN folks to unbreak their interaction with stacks in vmap space. It’s been broken literally forever, but I keep incorrectly imagining that it’s been fixed already.
>
> --
> H.J.
> <config.debug.xz>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Kernel 4.17.4 lockup
2018-07-20 21:35 ` Andy Lutomirski
@ 2018-07-23 14:51 ` H.J. Lu
0 siblings, 0 replies; 25+ messages in thread
From: H.J. Lu @ 2018-07-23 14:51 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Dave Hansen, Dave Jones, H. Peter Anvin, LKML, Andy Lutomirski,
Mel Gorman, Andrew Morton, Rik van Riel, Minchan Kim
On Fri, Jul 20, 2018 at 2:35 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>
>> On Jul 16, 2018, at 6:05 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>
>>> On Fri, Jul 13, 2018 at 7:08 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>>> I'm not at all convinced that this is the problem, but the series here
>>> will give a better diagnostic if the issue really is an IRQ stack
>>> overflow:
>>>
>>> https://git.kernel.org/pub/scm/linux/kernel/git/luto/linux.git/log/?h=x86/guard_pages
>>>
>>
>> This kernel won't boot with this configure file.
>
> Bah humbug. You’ll need to turn KASAN off (and maybe even compile it out entirely) if you want to test this. I need to bug the KASAN folks to unbreak their interaction with stacks in vmap space. It’s been broken literally forever, but I keep incorrectly imagining that it’s been fixed already.
>
This particular kernel oops has been fixed by kernel 4.17.7. But now
I am running into
another kernel oops:
https://bugzilla.redhat.com/show_bug.cgi?id=1597559
--
H.J.
^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2018-07-23 14:55 UTC | newest]
Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-07-08 21:36 Kernel 4.17.4 lockup H.J. Lu
2018-07-09 14:54 ` Dave Hansen
2018-07-09 23:02 ` H.J. Lu
2018-07-10 0:44 ` Dave Hansen
2018-07-10 2:14 ` H.J. Lu
2018-07-10 3:47 ` Dave Hansen
[not found] ` <CAMe9rOrHowBX06nihdRRmEqhV8v7cs+PwVY7JYQFpUFOnHC71A@mail.gmail.com>
2018-07-11 15:13 ` Dave Hansen
2018-07-11 15:40 ` H.J. Lu
2018-07-11 16:24 ` Dave Hansen
2018-07-11 16:29 ` H.J. Lu
2018-07-11 16:49 ` Dave Hansen
2018-07-11 16:53 ` H.J. Lu
[not found] ` <CAMe9rOpV89jWhvAZtqOJnc0eXzRYiLF5pLWRcdb7-kKLigj4rQ@mail.gmail.com>
2018-07-11 17:36 ` Andy Lutomirski
2018-07-11 17:43 ` Dave Hansen
2018-07-11 17:50 ` H.J. Lu
2018-07-11 17:50 ` Dave Hansen
2018-07-11 18:31 ` Dave Jones
2018-07-11 23:07 ` Andy Lutomirski
2018-07-11 23:14 ` Dave Hansen
2018-07-12 14:44 ` H.J. Lu
2018-07-12 23:28 ` H.J. Lu
2018-07-14 2:08 ` Andy Lutomirski
2018-07-16 16:05 ` H.J. Lu
2018-07-20 21:35 ` Andy Lutomirski
2018-07-23 14:51 ` H.J. Lu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).