linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* next-20190708: kernel BUG at lib/lockref.c:189, softlockups in shrink_dcache...?
@ 2019-07-10 20:13 Pavel Machek
  2019-07-11  0:09 ` Andrew Morton
  0 siblings, 1 reply; 4+ messages in thread
From: Pavel Machek @ 2019-07-10 20:13 UTC (permalink / raw)
  To: kernel list, sfr, Andrew Morton

[-- Attachment #1: Type: text/plain, Size: 12029 bytes --]

Hi!

I'm getting some nastyness from lockref / memory management.

Any ideas? Any ideas who to talk to?

									Pavel



[    0.000000] Linux version 5.2.0-next-20190708+ (pavel@pollux.denx.de) (gcc version 9.1.1 20190503 (Red Hat 9.1.1-1) (GCC)) #55 SMP Wed Jul 10 09:30:51 CEST 2019
[    0.000000] Disabled fast string operations
[    0.000000] x86/fpu: x87 FPU will use FXSAVE
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009efff] usable
...
[ 5347.093378] wlan0: authenticate with 5c:f4:ab:10:d2:bb
[ 5347.093511] wlan0: send auth to 5c:f4:ab:10:d2:bb (try 1/3)
[ 5347.096461] wlan0: authenticated
[ 5347.100328] wlan0: associate with 5c:f4:ab:10:d2:bb (try 1/3)
[ 5347.103232] wlan0: RX AssocResp from 5c:f4:ab:10:d2:bb (capab=0x411 status=0 aid=2)
[ 5347.105009] wlan0: associated
[ 6552.596393] ------------[ cut here ]------------
[ 6552.596412] kernel BUG at lib/lockref.c:189!
[ 6552.596433] invalid opcode: 0000 [#1] SMP PTI
[ 6552.596442] CPU: 0 PID: 8494 Comm: Chrome_ProcessL Not tainted 5.2.0-next-20190708+ #55
[ 6552.596447] Hardware name: LENOVO 17097HU/17097HU, BIOS 7BETD8WW (2.19 ) 03/31/2011
[ 6552.596463] EIP: lockref_mark_dead+0x10/0x20
[ 6552.596471] Code: c0 01 be 01 00 00 00 89 43 20 89 d8 e8 f9 5d 93 00 89 f0 5b 5e 5d c3 8d 76 00 8b 10 85 d2 74 0a c7 40 20 80 ff ff ff c3 66 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90 90 90 89 c2 83 e0 0f c0
[ 6552.596478] EAX: ef4b7830 EBX: ef4b77d8 ECX: 00000001 EDX: 00000000
[ 6552.596484] ESI: e647a3f0 EDI: ef4b7830 EBP: d9bfde0c ESP: d9bfddfc
[ 6552.596491] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010246
[ 6552.596497] CR0: 80050033 CR2: 071e03d0 CR3: 203e2000 CR4: 000006b0
[ 6552.596502] Call Trace:
[ 6552.596515]  ? __dentry_kill+0x1f/0x160
[ 6552.596524]  shrink_dcache_parent+0xb0/0x110
[ 6552.596532]  d_invalidate+0x48/0xc0
[ 6552.596541]  proc_flush_task+0x78/0x150
[ 6552.596552]  release_task+0x4f/0x450
[ 6552.596560]  wait_consider_task+0x7c9/0x850
[ 6552.596568]  do_wait+0x11b/0x230
[ 6552.596575]  kernel_wait4+0x8b/0x130
[ 6552.596583]  ? task_stopped_code+0x40/0x40
[ 6552.596590]  sys_waitpid+0x13/0x15
[ 6552.596597]  do_int80_syscall_32+0x4b/0xf0
[ 6552.596608]  entry_INT80_32+0x110/0x110
[ 6552.596614] EIP: 0xb7f8cc42
[ 6552.596621] Code: 65 8b 15 04 00 00 00 8b 0e 8b 0c ca 83 f9 ff 75 0c 89 04 24 89 f0 e8 b3 fe ff ff eb 05 8b 46 04 01 c8 83 c4 14 5b 5e c3 cd 80 <c3> 8d b6 00 00 00 00 8d bc 27 00 00 00 00 8b 1c 24 c3 8d b6 00 00
[ 6552.596628] EAX: ffffffda EBX: 0000212d ECX: 00000000 EDX: 00000000
[ 6552.596635] ESI: 8aec3b40 EDI: 09004d20 EBP: 8aec3158 ESP: 8aec3110
[ 6552.596641] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000293
[ 6552.596647] Modules linked in:
[ 6552.596658] ---[ end trace 8058c26986d91112 ]---
[ 6552.596667] EIP: lockref_mark_dead+0x10/0x20
[ 6552.596674] Code: c0 01 be 01 00 00 00 89 43 20 89 d8 e8 f9 5d 93 00 89 f0 5b 5e 5d c3 8d 76 00 8b 10 85 d2 74 0a c7 40 20 80 ff ff ff c3 66 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90 90 90 89 c2 83 e0 0f c0
[ 6552.596681] EAX: ef4b7830 EBX: ef4b77d8 ECX: 00000001 EDX: 00000000
[ 6552.596687] ESI: e647a3f0 EDI: ef4b7830 EBP: d9bfde0c ESP: d9bfddfc
[ 6552.596693] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010246
[ 6552.596700] CR0: 80050033 CR2: 071e03d0 CR3: 203e2000 CR4: 000006b0
[ 6580.173598] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [WorkerPool/7703:7703]
[ 6580.173608] Modules linked in:
[ 6580.173620] CPU: 1 PID: 7703 Comm: WorkerPool/7703 Tainted: G      D           5.2.0-next-20190708+ #55
[ 6580.173626] Hardware name: LENOVO 17097HU/17097HU, BIOS 7BETD8WW (2.19 ) 03/31/2011
[ 6580.173642] EIP: queued_spin_lock_slowpath+0x40/0x1d0
[ 6580.173651] Code: f0 0f ba 29 08 0f 92 c2 8b 01 0f b6 d2 c1 e2 08 30 e4 09 d0 a9 00 01 ff ff 75 27 85 c0 74 13 8b 01 84 c0 74 0d 8d 74 26 00 90 <f3> 90 8b 01 84 c0 75 f8 b8 01 00 00 00 66 89 01 c3 8d b4 26 00 00
[ 6580.173658] EAX: 00000101 EBX: e647a448 ECX: e647a448 EDX: 00000000
[ 6580.173664] ESI: ddd18358 EDI: e647a448 EBP: eefe1b44 ESP: eefe1b3c
[ 6580.173671] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00000202
[ 6580.173678] CR0: 80050033 CR2: 0a0e4748 CR3: 203e2000 CR4: 000006b0
[ 6580.173684] Call Trace:
[ 6580.173694]  ? do_raw_spin_lock+0xb9/0xc0
[ 6580.173705]  _raw_spin_lock+0x22/0x30
[ 6580.173715]  ? shrink_lock_dentry.part.0+0x9f/0xf0
[ 6580.173723]  shrink_lock_dentry.part.0+0x9f/0xf0
[ 6580.173731]  shrink_dentry_list+0x27/0x100
[ 6580.173739]  prune_dcache_sb+0x42/0x60
[ 6580.173747]  ? d_lru_shrink_move+0x50/0x50
[ 6580.173755]  super_cache_scan+0xc9/0x160
[ 6580.173763]  shrink_slab.constprop.0+0x1ae/0x390
[ 6580.173772]  shrink_node+0x7b1/0x9d0
[ 6580.173781]  do_try_to_free_pages+0xa1/0x250
[ 6580.173788]  ? do_try_to_free_pages+0xa1/0x250
[ 6580.173795]  try_to_free_pages+0x15e/0x430
[ 6580.173805]  __alloc_pages_nodemask+0x44d/0xd00
[ 6580.173816]  wp_page_copy+0x67/0x680
[ 6580.173825]  do_wp_page+0x75/0x450
[ 6580.173833]  ? _raw_spin_lock+0x22/0x30
[ 6580.173840]  handle_mm_fault+0x598/0xbd3
[ 6580.173848]  ? __lock_acquire.isra.0+0x2b8/0x4f0
[ 6580.173858]  __do_page_fault+0x15e/0x410
[ 6580.173865]  ? vmalloc_sync_all+0x220/0x220
[ 6580.173872]  do_page_fault+0x24/0x137
[ 6580.173879]  ? vmalloc_sync_all+0x220/0x220
[ 6580.173886]  common_exception+0x166/0x16e
[ 6580.173892] EIP: 0x8cad20
[ 6580.173899] Code: a8 73 5b fe 89 4c 24 04 c7 44 24 08 76 00 00 00 c7 04 24 01 00 00 00 89 44 24 4c e8 4a 98 ff ff 8b 44 24 4c 8b 4c 24 48 89 31 <89> 7e 04 e9 84 02 00 00 89 7c 24 08 89 44 24 04 89 34 24 e8 a8 ea
[ 6580.173907] EAX: 0f6c1230 EBX: 08d83524 ECX: 08fbc8c0 EDX: fffffb9b
[ 6580.173913] ESI: 0f85a000 EDI: fffffb9b EBP: b2af35d8 ESP: b2af30b0
[ 6580.173920] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00010246
[ 6608.173599] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [WorkerPool/7703:7703]
[ 6608.173607] Modules linked in:
[ 6608.173619] CPU: 1 PID: 7703 Comm: WorkerPool/7703 Tainted: G      D      L    5.2.0-next-20190708+ #55
[ 6608.173624] Hardware name: LENOVO 17097HU/17097HU, BIOS 7BETD8WW (2.19 ) 03/31/2011
[ 6608.173640] EIP: queued_spin_lock_slowpath+0x44/0x1d0
[ 6608.173648] Code: 08 0f 92 c2 8b 01 0f b6 d2 c1 e2 08 30 e4 09 d0 a9 00 01 ff ff 75 27 85 c0 74 13 8b 01 84 c0 74 0d 8d 74 26 00 90 f3 90 8b 01 <84> c0 75 f8 b8 01 00 00 00 66 89 01 c3 8d b4 26 00 00 00 00 f6 c4
[ 6608.173655] EAX: 00000101 EBX: e647a448 ECX: e647a448 EDX: 00000000
[ 6608.173661] ESI: ddd18358 EDI: e647a448 EBP: eefe1b44 ESP: eefe1b3c
[ 6608.173668] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00000202
[ 6608.173675] CR0: 80050033 CR2: 0a0e4748 CR3: 203e2000 CR4: 000006b0
[ 6608.173680] Call Trace:
[ 6608.173690]  ? do_raw_spin_lock+0xb9/0xc0
[ 6608.173701]  _raw_spin_lock+0x22/0x30
[ 6608.173712]  ? shrink_lock_dentry.part.0+0x9f/0xf0
[ 6608.173719]  shrink_lock_dentry.part.0+0x9f/0xf0
[ 6608.173727]  shrink_dentry_list+0x27/0x100
[ 6608.173735]  prune_dcache_sb+0x42/0x60
[ 6608.173743]  ? d_lru_shrink_move+0x50/0x50
[ 6608.173751]  super_cache_scan+0xc9/0x160
[ 6608.173760]  shrink_slab.constprop.0+0x1ae/0x390
[ 6608.173768]  shrink_node+0x7b1/0x9d0
[ 6608.173777]  do_try_to_free_pages+0xa1/0x250
[ 6608.173784]  ? do_try_to_free_pages+0xa1/0x250
[ 6608.173791]  try_to_free_pages+0x15e/0x430
[ 6608.173801]  __alloc_pages_nodemask+0x44d/0xd00
[ 6608.173813]  wp_page_copy+0x67/0x680
[ 6608.173822]  do_wp_page+0x75/0x450
[ 6608.173829]  ? _raw_spin_lock+0x22/0x30
[ 6608.173836]  handle_mm_fault+0x598/0xbd3
[ 6608.173844]  ? __lock_acquire.isra.0+0x2b8/0x4f0
[ 6608.173854]  __do_page_fault+0x15e/0x410
[ 6608.173862]  ? vmalloc_sync_all+0x220/0x220
[ 6608.173869]  do_page_fault+0x24/0x137
[ 6608.173876]  ? vmalloc_sync_all+0x220/0x220
[ 6608.173883]  common_exception+0x166/0x16e
[ 6608.173889] EIP: 0x8cad20
[ 6608.173896] Code: a8 73 5b fe 89 4c 24 04 c7 44 24 08 76 00 00 00 c7 04 24 01 00 00 00 89 44 24 4c e8 4a 98 ff ff 8b 44 24 4c 8b 4c 24 48 89 31 <89> 7e 04 e9 84 02 00 00 89 7c 24 08 89 44 24 04 89 34 24 e8 a8 ea
[ 6608.173903] EAX: 0f6c1230 EBX: 08d83524 ECX: 08fbc8c0 EDX: fffffb9b
[ 6608.173909] ESI: 0f85a000 EDI: fffffb9b EBP: b2af35d8 ESP: b2af30b0
[ 6608.173916] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00010246
[ 6612.628114] rcu: INFO: rcu_sched self-detected stall on CPU
[ 6612.628114] rcu: 	1-....: (14999 ticks this GP) idle=07a/1/0x40000002 softirq=778997/778997 fqs=7465 
[ 6612.628114] 	(t=15000 jiffies g=940129 q=41142)
[ 6612.628146] NMI backtrace for cpu 1
[ 6612.628146] CPU: 1 PID: 7703 Comm: WorkerPool/7703 Tainted: G      D      L    5.2.0-next-20190708+ #55
[ 6612.628146] Hardware name: LENOVO 17097HU/17097HU, BIOS 7BETD8WW (2.19 ) 03/31/2011
[ 6612.628165] Call Trace:
[ 6612.628165]  dump_stack+0x47/0x5e
[ 6612.628165]  nmi_cpu_backtrace.cold+0x10/0x48
[ 6612.628165]  ? lapic_can_unplug_cpu.cold+0x3d/0x3d
[ 6612.628165]  nmi_trigger_cpumask_backtrace+0x8f/0x91
[ 6612.628165]  arch_trigger_cpumask_backtrace+0x10/0x20
[ 6612.628165]  rcu_dump_cpu_stacks+0x6a/0x90
[ 6612.628165]  rcu_sched_clock_irq.cold+0x18f/0x352
[ 6612.628165]  update_process_times+0x23/0x60
[ 6612.628165]  tick_sched_handle.isra.0+0x2e/0x60
[ 6612.628165]  tick_sched_timer+0x3c/0x90
[ 6612.628165]  ? tick_sched_handle.isra.0+0x60/0x60
[ 6612.628165]  __hrtimer_run_queues+0x105/0x2e0
[ 6612.628165]  ? tick_sched_handle.isra.0+0x60/0x60
[ 6612.628165]  hrtimer_interrupt+0x10e/0x260
[ 6612.628165]  smp_apic_timer_interrupt+0x68/0x160
[ 6612.628165]  apic_timer_interrupt+0x11b/0x120
[ 6612.628165] EIP: queued_spin_lock_slowpath+0x42/0x1d0
[ 6612.628165] Code: ba 29 08 0f 92 c2 8b 01 0f b6 d2 c1 e2 08 30 e4 09 d0 a9 00 01 ff ff 75 27 85 c0 74 13 8b 01 84 c0 74 0d 8d 74 26 00 90 f3 90 <8b> 01 84 c0 75 f8 b8 01 00 00 00 66 89 01 c3 8d b4 26 00 00 00 00
[ 6612.628165] EAX: 00000101 EBX: e647a448 ECX: e647a448 EDX: 00000000
[ 6612.628165] ESI: ddd18358 EDI: e647a448 EBP: eefe1b44 ESP: eefe1b3c
[ 6612.628165] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00000202
[ 6612.628165]  ? queued_spin_lock_slowpath+0x42/0x1d0
[ 6612.628165]  ? do_raw_spin_lock+0xb9/0xc0
[ 6612.628165]  _raw_spin_lock+0x22/0x30
[ 6612.628165]  ? shrink_lock_dentry.part.0+0x9f/0xf0
[ 6612.628165]  shrink_lock_dentry.part.0+0x9f/0xf0
[ 6612.628165]  shrink_dentry_list+0x27/0x100
[ 6612.628165]  prune_dcache_sb+0x42/0x60
[ 6612.628165]  ? d_lru_shrink_move+0x50/0x50
[ 6612.628165]  super_cache_scan+0xc9/0x160
[ 6612.628165]  shrink_slab.constprop.0+0x1ae/0x390
[ 6612.628165]  shrink_node+0x7b1/0x9d0
[ 6612.628165]  do_try_to_free_pages+0xa1/0x250
[ 6612.628165]  ? do_try_to_free_pages+0xa1/0x250
[ 6612.628165]  try_to_free_pages+0x15e/0x430
[ 6612.628165]  __alloc_pages_nodemask+0x44d/0xd00
[ 6612.628165]  wp_page_copy+0x67/0x680
[ 6612.628165]  do_wp_page+0x75/0x450
[ 6612.628165]  ? _raw_spin_lock+0x22/0x30
[ 6612.628165]  handle_mm_fault+0x598/0xbd3
[ 6612.628165]  ? __lock_acquire.isra.0+0x2b8/0x4f0
[ 6612.628165]  __do_page_fault+0x15e/0x410
[ 6612.628165]  ? vmalloc_sync_all+0x220/0x220
[ 6612.628165]  do_page_fault+0x24/0x137
[ 6612.628165]  ? vmalloc_sync_all+0x220/0x220
[ 6612.628165]  common_exception+0x166/0x16e
[ 6612.628165] EIP: 0x8cad20
[ 6612.628165] Code: a8 73 5b fe 89 4c 24 04 c7 44 24 08 76 00 00 00 c7 04 24 01 00 00 00 89 44 24 4c e8 4a 98 ff ff 8b 44 24 4c 8b 4c 24 48 89 31 <89> 7e 04 e9 84 02 00 00 89 7c 24 08 89 44 24 04 89 34 24 e8 a8 ea
[ 6612.628165] EAX: 0f6c1230 EBX: 08d83524 ECX: 08fbc8c0 EDX: fffffb9b
[ 6612.628165] ESI: 0f85a000 EDI: fffffb9b EBP: b2af35d8 ESP: b2af30b0
[ 6612.628165] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00010246




-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: next-20190708: kernel BUG at lib/lockref.c:189, softlockups in shrink_dcache...?
  2019-07-10 20:13 next-20190708: kernel BUG at lib/lockref.c:189, softlockups in shrink_dcache...? Pavel Machek
@ 2019-07-11  0:09 ` Andrew Morton
  2019-07-11  0:25   ` Al Viro
  0 siblings, 1 reply; 4+ messages in thread
From: Andrew Morton @ 2019-07-11  0:09 UTC (permalink / raw)
  To: Pavel Machek; +Cc: kernel list, sfr, Al Viro

On Wed, 10 Jul 2019 22:13:11 +0200 Pavel Machek <pavel@ucw.cz> wrote:

> Hi!
> 
> I'm getting some nastyness from lockref / memory management.
> 
> Any ideas? Any ideas who to talk to?
> 

I'd be suspecting Al's a99d7580f66e737 ("Teach shrink_dcache_parent()
to cope with mixed-filesystem shrink lists").

This appears to have been added to -next around the -rc6 timeframe,
which seems awfully late?


> 
> 
> 
> [    0.000000] Linux version 5.2.0-next-20190708+ (pavel@pollux.denx.de) (gcc version 9.1.1 20190503 (Red Hat 9.1.1-1) (GCC)) #55 SMP Wed Jul 10 09:30:51 CEST 2019
> [    0.000000] Disabled fast string operations
> [    0.000000] x86/fpu: x87 FPU will use FXSAVE
> [    0.000000] BIOS-provided physical RAM map:
> [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009efff] usable
> ...
> [ 5347.093378] wlan0: authenticate with 5c:f4:ab:10:d2:bb
> [ 5347.093511] wlan0: send auth to 5c:f4:ab:10:d2:bb (try 1/3)
> [ 5347.096461] wlan0: authenticated
> [ 5347.100328] wlan0: associate with 5c:f4:ab:10:d2:bb (try 1/3)
> [ 5347.103232] wlan0: RX AssocResp from 5c:f4:ab:10:d2:bb (capab=0x411 status=0 aid=2)
> [ 5347.105009] wlan0: associated
> [ 6552.596393] ------------[ cut here ]------------
> [ 6552.596412] kernel BUG at lib/lockref.c:189!
> [ 6552.596433] invalid opcode: 0000 [#1] SMP PTI
> [ 6552.596442] CPU: 0 PID: 8494 Comm: Chrome_ProcessL Not tainted 5.2.0-next-20190708+ #55
> [ 6552.596447] Hardware name: LENOVO 17097HU/17097HU, BIOS 7BETD8WW (2.19 ) 03/31/2011
> [ 6552.596463] EIP: lockref_mark_dead+0x10/0x20
> [ 6552.596471] Code: c0 01 be 01 00 00 00 89 43 20 89 d8 e8 f9 5d 93 00 89 f0 5b 5e 5d c3 8d 76 00 8b 10 85 d2 74 0a c7 40 20 80 ff ff ff c3 66 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90 90 90 89 c2 83 e0 0f c0
> [ 6552.596478] EAX: ef4b7830 EBX: ef4b77d8 ECX: 00000001 EDX: 00000000
> [ 6552.596484] ESI: e647a3f0 EDI: ef4b7830 EBP: d9bfde0c ESP: d9bfddfc
> [ 6552.596491] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010246
> [ 6552.596497] CR0: 80050033 CR2: 071e03d0 CR3: 203e2000 CR4: 000006b0
> [ 6552.596502] Call Trace:
> [ 6552.596515]  ? __dentry_kill+0x1f/0x160
> [ 6552.596524]  shrink_dcache_parent+0xb0/0x110
> [ 6552.596532]  d_invalidate+0x48/0xc0
> [ 6552.596541]  proc_flush_task+0x78/0x150
> [ 6552.596552]  release_task+0x4f/0x450
> [ 6552.596560]  wait_consider_task+0x7c9/0x850
> [ 6552.596568]  do_wait+0x11b/0x230
> [ 6552.596575]  kernel_wait4+0x8b/0x130
> [ 6552.596583]  ? task_stopped_code+0x40/0x40
> [ 6552.596590]  sys_waitpid+0x13/0x15
> [ 6552.596597]  do_int80_syscall_32+0x4b/0xf0
> [ 6552.596608]  entry_INT80_32+0x110/0x110
> [ 6552.596614] EIP: 0xb7f8cc42
> [ 6552.596621] Code: 65 8b 15 04 00 00 00 8b 0e 8b 0c ca 83 f9 ff 75 0c 89 04 24 89 f0 e8 b3 fe ff ff eb 05 8b 46 04 01 c8 83 c4 14 5b 5e c3 cd 80 <c3> 8d b6 00 00 00 00 8d bc 27 00 00 00 00 8b 1c 24 c3 8d b6 00 00
> [ 6552.596628] EAX: ffffffda EBX: 0000212d ECX: 00000000 EDX: 00000000
> [ 6552.596635] ESI: 8aec3b40 EDI: 09004d20 EBP: 8aec3158 ESP: 8aec3110
> [ 6552.596641] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000293
> [ 6552.596647] Modules linked in:
> [ 6552.596658] ---[ end trace 8058c26986d91112 ]---
> [ 6552.596667] EIP: lockref_mark_dead+0x10/0x20
> [ 6552.596674] Code: c0 01 be 01 00 00 00 89 43 20 89 d8 e8 f9 5d 93 00 89 f0 5b 5e 5d c3 8d 76 00 8b 10 85 d2 74 0a c7 40 20 80 ff ff ff c3 66 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90 90 90 89 c2 83 e0 0f c0
> [ 6552.596681] EAX: ef4b7830 EBX: ef4b77d8 ECX: 00000001 EDX: 00000000
> [ 6552.596687] ESI: e647a3f0 EDI: ef4b7830 EBP: d9bfde0c ESP: d9bfddfc
> [ 6552.596693] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010246
> [ 6552.596700] CR0: 80050033 CR2: 071e03d0 CR3: 203e2000 CR4: 000006b0
> [ 6580.173598] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [WorkerPool/7703:7703]
> [ 6580.173608] Modules linked in:
> [ 6580.173620] CPU: 1 PID: 7703 Comm: WorkerPool/7703 Tainted: G      D           5.2.0-next-20190708+ #55
> [ 6580.173626] Hardware name: LENOVO 17097HU/17097HU, BIOS 7BETD8WW (2.19 ) 03/31/2011
> [ 6580.173642] EIP: queued_spin_lock_slowpath+0x40/0x1d0
> [ 6580.173651] Code: f0 0f ba 29 08 0f 92 c2 8b 01 0f b6 d2 c1 e2 08 30 e4 09 d0 a9 00 01 ff ff 75 27 85 c0 74 13 8b 01 84 c0 74 0d 8d 74 26 00 90 <f3> 90 8b 01 84 c0 75 f8 b8 01 00 00 00 66 89 01 c3 8d b4 26 00 00
> [ 6580.173658] EAX: 00000101 EBX: e647a448 ECX: e647a448 EDX: 00000000
> [ 6580.173664] ESI: ddd18358 EDI: e647a448 EBP: eefe1b44 ESP: eefe1b3c
> [ 6580.173671] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00000202
> [ 6580.173678] CR0: 80050033 CR2: 0a0e4748 CR3: 203e2000 CR4: 000006b0
> [ 6580.173684] Call Trace:
> [ 6580.173694]  ? do_raw_spin_lock+0xb9/0xc0
> [ 6580.173705]  _raw_spin_lock+0x22/0x30
> [ 6580.173715]  ? shrink_lock_dentry.part.0+0x9f/0xf0
> [ 6580.173723]  shrink_lock_dentry.part.0+0x9f/0xf0
> [ 6580.173731]  shrink_dentry_list+0x27/0x100
> [ 6580.173739]  prune_dcache_sb+0x42/0x60
> [ 6580.173747]  ? d_lru_shrink_move+0x50/0x50
> [ 6580.173755]  super_cache_scan+0xc9/0x160
> [ 6580.173763]  shrink_slab.constprop.0+0x1ae/0x390
> [ 6580.173772]  shrink_node+0x7b1/0x9d0
> [ 6580.173781]  do_try_to_free_pages+0xa1/0x250
> [ 6580.173788]  ? do_try_to_free_pages+0xa1/0x250
> [ 6580.173795]  try_to_free_pages+0x15e/0x430
> [ 6580.173805]  __alloc_pages_nodemask+0x44d/0xd00
> [ 6580.173816]  wp_page_copy+0x67/0x680
> [ 6580.173825]  do_wp_page+0x75/0x450
> [ 6580.173833]  ? _raw_spin_lock+0x22/0x30
> [ 6580.173840]  handle_mm_fault+0x598/0xbd3
> [ 6580.173848]  ? __lock_acquire.isra.0+0x2b8/0x4f0
> [ 6580.173858]  __do_page_fault+0x15e/0x410
> [ 6580.173865]  ? vmalloc_sync_all+0x220/0x220
> [ 6580.173872]  do_page_fault+0x24/0x137
> [ 6580.173879]  ? vmalloc_sync_all+0x220/0x220
> [ 6580.173886]  common_exception+0x166/0x16e
> [ 6580.173892] EIP: 0x8cad20
> [ 6580.173899] Code: a8 73 5b fe 89 4c 24 04 c7 44 24 08 76 00 00 00 c7 04 24 01 00 00 00 89 44 24 4c e8 4a 98 ff ff 8b 44 24 4c 8b 4c 24 48 89 31 <89> 7e 04 e9 84 02 00 00 89 7c 24 08 89 44 24 04 89 34 24 e8 a8 ea
> [ 6580.173907] EAX: 0f6c1230 EBX: 08d83524 ECX: 08fbc8c0 EDX: fffffb9b
> [ 6580.173913] ESI: 0f85a000 EDI: fffffb9b EBP: b2af35d8 ESP: b2af30b0
> [ 6580.173920] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00010246
> [ 6608.173599] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [WorkerPool/7703:7703]
> [ 6608.173607] Modules linked in:
> [ 6608.173619] CPU: 1 PID: 7703 Comm: WorkerPool/7703 Tainted: G      D      L    5.2.0-next-20190708+ #55
> [ 6608.173624] Hardware name: LENOVO 17097HU/17097HU, BIOS 7BETD8WW (2.19 ) 03/31/2011
> [ 6608.173640] EIP: queued_spin_lock_slowpath+0x44/0x1d0
> [ 6608.173648] Code: 08 0f 92 c2 8b 01 0f b6 d2 c1 e2 08 30 e4 09 d0 a9 00 01 ff ff 75 27 85 c0 74 13 8b 01 84 c0 74 0d 8d 74 26 00 90 f3 90 8b 01 <84> c0 75 f8 b8 01 00 00 00 66 89 01 c3 8d b4 26 00 00 00 00 f6 c4
> [ 6608.173655] EAX: 00000101 EBX: e647a448 ECX: e647a448 EDX: 00000000
> [ 6608.173661] ESI: ddd18358 EDI: e647a448 EBP: eefe1b44 ESP: eefe1b3c
> [ 6608.173668] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00000202
> [ 6608.173675] CR0: 80050033 CR2: 0a0e4748 CR3: 203e2000 CR4: 000006b0
> [ 6608.173680] Call Trace:
> [ 6608.173690]  ? do_raw_spin_lock+0xb9/0xc0
> [ 6608.173701]  _raw_spin_lock+0x22/0x30
> [ 6608.173712]  ? shrink_lock_dentry.part.0+0x9f/0xf0
> [ 6608.173719]  shrink_lock_dentry.part.0+0x9f/0xf0
> [ 6608.173727]  shrink_dentry_list+0x27/0x100
> [ 6608.173735]  prune_dcache_sb+0x42/0x60
> [ 6608.173743]  ? d_lru_shrink_move+0x50/0x50
> [ 6608.173751]  super_cache_scan+0xc9/0x160
> [ 6608.173760]  shrink_slab.constprop.0+0x1ae/0x390
> [ 6608.173768]  shrink_node+0x7b1/0x9d0
> [ 6608.173777]  do_try_to_free_pages+0xa1/0x250
> [ 6608.173784]  ? do_try_to_free_pages+0xa1/0x250
> [ 6608.173791]  try_to_free_pages+0x15e/0x430
> [ 6608.173801]  __alloc_pages_nodemask+0x44d/0xd00
> [ 6608.173813]  wp_page_copy+0x67/0x680
> [ 6608.173822]  do_wp_page+0x75/0x450
> [ 6608.173829]  ? _raw_spin_lock+0x22/0x30
> [ 6608.173836]  handle_mm_fault+0x598/0xbd3
> [ 6608.173844]  ? __lock_acquire.isra.0+0x2b8/0x4f0
> [ 6608.173854]  __do_page_fault+0x15e/0x410
> [ 6608.173862]  ? vmalloc_sync_all+0x220/0x220
> [ 6608.173869]  do_page_fault+0x24/0x137
> [ 6608.173876]  ? vmalloc_sync_all+0x220/0x220
> [ 6608.173883]  common_exception+0x166/0x16e
> [ 6608.173889] EIP: 0x8cad20
> [ 6608.173896] Code: a8 73 5b fe 89 4c 24 04 c7 44 24 08 76 00 00 00 c7 04 24 01 00 00 00 89 44 24 4c e8 4a 98 ff ff 8b 44 24 4c 8b 4c 24 48 89 31 <89> 7e 04 e9 84 02 00 00 89 7c 24 08 89 44 24 04 89 34 24 e8 a8 ea
> [ 6608.173903] EAX: 0f6c1230 EBX: 08d83524 ECX: 08fbc8c0 EDX: fffffb9b
> [ 6608.173909] ESI: 0f85a000 EDI: fffffb9b EBP: b2af35d8 ESP: b2af30b0
> [ 6608.173916] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00010246
> [ 6612.628114] rcu: INFO: rcu_sched self-detected stall on CPU
> [ 6612.628114] rcu: 	1-....: (14999 ticks this GP) idle=07a/1/0x40000002 softirq=778997/778997 fqs=7465 
> [ 6612.628114] 	(t=15000 jiffies g=940129 q=41142)
> [ 6612.628146] NMI backtrace for cpu 1
> [ 6612.628146] CPU: 1 PID: 7703 Comm: WorkerPool/7703 Tainted: G      D      L    5.2.0-next-20190708+ #55
> [ 6612.628146] Hardware name: LENOVO 17097HU/17097HU, BIOS 7BETD8WW (2.19 ) 03/31/2011
> [ 6612.628165] Call Trace:
> [ 6612.628165]  dump_stack+0x47/0x5e
> [ 6612.628165]  nmi_cpu_backtrace.cold+0x10/0x48
> [ 6612.628165]  ? lapic_can_unplug_cpu.cold+0x3d/0x3d
> [ 6612.628165]  nmi_trigger_cpumask_backtrace+0x8f/0x91
> [ 6612.628165]  arch_trigger_cpumask_backtrace+0x10/0x20
> [ 6612.628165]  rcu_dump_cpu_stacks+0x6a/0x90
> [ 6612.628165]  rcu_sched_clock_irq.cold+0x18f/0x352
> [ 6612.628165]  update_process_times+0x23/0x60
> [ 6612.628165]  tick_sched_handle.isra.0+0x2e/0x60
> [ 6612.628165]  tick_sched_timer+0x3c/0x90
> [ 6612.628165]  ? tick_sched_handle.isra.0+0x60/0x60
> [ 6612.628165]  __hrtimer_run_queues+0x105/0x2e0
> [ 6612.628165]  ? tick_sched_handle.isra.0+0x60/0x60
> [ 6612.628165]  hrtimer_interrupt+0x10e/0x260
> [ 6612.628165]  smp_apic_timer_interrupt+0x68/0x160
> [ 6612.628165]  apic_timer_interrupt+0x11b/0x120
> [ 6612.628165] EIP: queued_spin_lock_slowpath+0x42/0x1d0
> [ 6612.628165] Code: ba 29 08 0f 92 c2 8b 01 0f b6 d2 c1 e2 08 30 e4 09 d0 a9 00 01 ff ff 75 27 85 c0 74 13 8b 01 84 c0 74 0d 8d 74 26 00 90 f3 90 <8b> 01 84 c0 75 f8 b8 01 00 00 00 66 89 01 c3 8d b4 26 00 00 00 00
> [ 6612.628165] EAX: 00000101 EBX: e647a448 ECX: e647a448 EDX: 00000000
> [ 6612.628165] ESI: ddd18358 EDI: e647a448 EBP: eefe1b44 ESP: eefe1b3c
> [ 6612.628165] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00000202
> [ 6612.628165]  ? queued_spin_lock_slowpath+0x42/0x1d0
> [ 6612.628165]  ? do_raw_spin_lock+0xb9/0xc0
> [ 6612.628165]  _raw_spin_lock+0x22/0x30
> [ 6612.628165]  ? shrink_lock_dentry.part.0+0x9f/0xf0
> [ 6612.628165]  shrink_lock_dentry.part.0+0x9f/0xf0
> [ 6612.628165]  shrink_dentry_list+0x27/0x100
> [ 6612.628165]  prune_dcache_sb+0x42/0x60
> [ 6612.628165]  ? d_lru_shrink_move+0x50/0x50
> [ 6612.628165]  super_cache_scan+0xc9/0x160
> [ 6612.628165]  shrink_slab.constprop.0+0x1ae/0x390
> [ 6612.628165]  shrink_node+0x7b1/0x9d0
> [ 6612.628165]  do_try_to_free_pages+0xa1/0x250
> [ 6612.628165]  ? do_try_to_free_pages+0xa1/0x250
> [ 6612.628165]  try_to_free_pages+0x15e/0x430
> [ 6612.628165]  __alloc_pages_nodemask+0x44d/0xd00
> [ 6612.628165]  wp_page_copy+0x67/0x680
> [ 6612.628165]  do_wp_page+0x75/0x450
> [ 6612.628165]  ? _raw_spin_lock+0x22/0x30
> [ 6612.628165]  handle_mm_fault+0x598/0xbd3
> [ 6612.628165]  ? __lock_acquire.isra.0+0x2b8/0x4f0
> [ 6612.628165]  __do_page_fault+0x15e/0x410
> [ 6612.628165]  ? vmalloc_sync_all+0x220/0x220
> [ 6612.628165]  do_page_fault+0x24/0x137
> [ 6612.628165]  ? vmalloc_sync_all+0x220/0x220
> [ 6612.628165]  common_exception+0x166/0x16e
> [ 6612.628165] EIP: 0x8cad20
> [ 6612.628165] Code: a8 73 5b fe 89 4c 24 04 c7 44 24 08 76 00 00 00 c7 04 24 01 00 00 00 89 44 24 4c e8 4a 98 ff ff 8b 44 24 4c 8b 4c 24 48 89 31 <89> 7e 04 e9 84 02 00 00 89 7c 24 08 89 44 24 04 89 34 24 e8 a8 ea
> [ 6612.628165] EAX: 0f6c1230 EBX: 08d83524 ECX: 08fbc8c0 EDX: fffffb9b
> [ 6612.628165] ESI: 0f85a000 EDI: fffffb9b EBP: b2af35d8 ESP: b2af30b0
> [ 6612.628165] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00010246
> 
> 
> 
> 
> -- 
> (english) http://www.livejournal.com/~pavelmachek
> (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: next-20190708: kernel BUG at lib/lockref.c:189, softlockups in shrink_dcache...?
  2019-07-11  0:09 ` Andrew Morton
@ 2019-07-11  0:25   ` Al Viro
  2019-07-11  7:24     ` Pavel Machek
  0 siblings, 1 reply; 4+ messages in thread
From: Al Viro @ 2019-07-11  0:25 UTC (permalink / raw)
  To: Andrew Morton; +Cc: Pavel Machek, kernel list, sfr

On Wed, Jul 10, 2019 at 05:09:12PM -0700, Andrew Morton wrote:
> On Wed, 10 Jul 2019 22:13:11 +0200 Pavel Machek <pavel@ucw.cz> wrote:
> 
> > Hi!
> > 
> > I'm getting some nastyness from lockref / memory management.
> > 
> > Any ideas? Any ideas who to talk to?
> > 
> 
> I'd be suspecting Al's a99d7580f66e737 ("Teach shrink_dcache_parent()
> to cope with mixed-filesystem shrink lists").

It is, and it's already fixed.  See 9bdebc2bd1c4 with fixes folded in;
the incremental is

diff --git a/fs/dcache.c b/fs/dcache.c
index d8732cf2e302..01b8cae41a71 100644
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -1555,7 +1555,9 @@ void shrink_dcache_parent(struct dentry *parent)
 		d_walk(parent, &data, select_collect2);
 		if (data.victim) {
 			struct dentry *parent;
+			spin_lock(&data.victim->d_lock);
 			if (!shrink_lock_dentry(data.victim)) {
+				spin_unlock(&data.victim->d_lock);
 				rcu_read_unlock();
 			} else {
 				rcu_read_unlock();
 
> This appears to have been added to -next around the -rc6 timeframe,
> which seems awfully late?

I'm not sure if I'll be asking to pull that series this window,
actually ;-/

Sigh...  This cycle had been a mess - spent a month or so very unpleasantly
sick ;-/

^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: next-20190708: kernel BUG at lib/lockref.c:189, softlockups in shrink_dcache...?
  2019-07-11  0:25   ` Al Viro
@ 2019-07-11  7:24     ` Pavel Machek
  0 siblings, 0 replies; 4+ messages in thread
From: Pavel Machek @ 2019-07-11  7:24 UTC (permalink / raw)
  To: Al Viro; +Cc: Andrew Morton, kernel list, sfr

[-- Attachment #1: Type: text/plain, Size: 1331 bytes --]

On Thu 2019-07-11 01:25:27, Al Viro wrote:
> On Wed, Jul 10, 2019 at 05:09:12PM -0700, Andrew Morton wrote:
> > On Wed, 10 Jul 2019 22:13:11 +0200 Pavel Machek <pavel@ucw.cz> wrote:
> > 
> > > Hi!
> > > 
> > > I'm getting some nastyness from lockref / memory management.
> > > 
> > > Any ideas? Any ideas who to talk to?
> > > 
> > 
> > I'd be suspecting Al's a99d7580f66e737 ("Teach shrink_dcache_parent()
> > to cope with mixed-filesystem shrink lists").
> 
> It is, and it's already fixed.  See 9bdebc2bd1c4 with fixes folded in;
> the incremental is
> 
> diff --git a/fs/dcache.c b/fs/dcache.c
> index d8732cf2e302..01b8cae41a71 100644
> --- a/fs/dcache.c
> +++ b/fs/dcache.c
> @@ -1555,7 +1555,9 @@ void shrink_dcache_parent(struct dentry *parent)
>  		d_walk(parent, &data, select_collect2);
>  		if (data.victim) {
>  			struct dentry *parent;
> +			spin_lock(&data.victim->d_lock);
>  			if (!shrink_lock_dentry(data.victim)) {
> +				spin_unlock(&data.victim->d_lock);
>  				rcu_read_unlock();
>  			} else {
>  				rcu_read_unlock();

Thanks! It only happened once under high load, so it seems it will not
be coming back.
								Pavel
								
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2019-07-11  7:24 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-07-10 20:13 next-20190708: kernel BUG at lib/lockref.c:189, softlockups in shrink_dcache...? Pavel Machek
2019-07-11  0:09 ` Andrew Morton
2019-07-11  0:25   ` Al Viro
2019-07-11  7:24     ` Pavel Machek

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).