* Deadlock due to "cpuset: Make cpuset hotplug synchronous"
@ 2020-03-25 19:16 Qian Cai
2020-03-25 19:19 ` Tejun Heo
2020-03-25 20:20 ` Peter Zijlstra
0 siblings, 2 replies; 11+ messages in thread
From: Qian Cai @ 2020-03-25 19:16 UTC (permalink / raw)
To: Prateek Sood; +Cc: Li Zefan, Tejun Heo, cgroups, LKML, Peter Zijlstra
The linux-next commit a49e4629b5ed (“cpuset: Make cpuset hotplug synchronous”)
introduced real deadlocks with CPU hotplug as showed in the lockdep splat, since it is
now making a relation from cpu_hotplug_lock —> cgroup_mutex.
[17602.773334][ T15] ======================================================
[17602.780207][ T15] WARNING: possible circular locking dependency detected
[17602.787081][ T15] 5.6.0-rc7-next-20200325+ #13 Tainted: G L
[17602.794125][ T15] ------------------------------------------------------
[17602.800997][ T15] cpuhp/1/15 is trying to acquire lock:
[17602.806392][ T15] ffff900012cb7bf0 (cgroup_mutex){+.+.}-{3:3}, at: cgroup_transfer_tasks+0x130/0x2d8
[17602.815718][ T15]
[17602.815718][ T15] but task is already holding lock:
[17602.822934][ T15] ffff900012aeb2b0 (cpuhp_state-down){+.+.}-{0:0}, at: cpuhp_lock_acquire+0x8/0x48
[17602.832078][ T15]
[17602.832078][ T15] which lock already depends on the new lock.
[17602.832078][ T15]
[17602.842334][ T15]
[17602.842334][ T15] the existing dependency chain (in reverse order) is:
[17602.851200][ T15]
[17602.851200][ T15] -> #2 (cpuhp_state-down){+.+.}-{0:0}:
[17602.858780][ T15] lock_acquire+0xe4/0x25c
[17602.863570][ T15] cpuhp_lock_acquire+0x40/0x48
[17602.868794][ T15] cpuhp_kick_ap_work+0x64/0x118
[17602.874107][ T15] _cpu_down+0x16c/0x3c0
[17602.878722][ T15] cpu_down+0x54/0x74
[17602.883079][ T15] cpu_subsys_offline+0x24/0x30
[17602.888303][ T15] device_offline+0x94/0xf0
[17602.893180][ T15] online_store+0x80/0x110
[17602.897969][ T15] dev_attr_store+0x5c/0x78
[17602.902847][ T15] sysfs_kf_write+0x9c/0xcc
[17602.907724][ T15] kernfs_fop_write+0x228/0x32c
[17602.912950][ T15] __vfs_write+0x84/0x1d8
[17602.917652][ T15] vfs_write+0x13c/0x1b4
[17602.922268][ T15] ksys_write+0xb0/0x120
[17602.926884][ T15] __arm64_sys_write+0x54/0x88
[17602.932021][ T15] do_el0_svc+0x128/0x1dc
[17602.936725][ T15] el0_sync_handler+0x150/0x250
[17602.941948][ T15] el0_sync+0x164/0x180
[17602.946473][ T15]
[17602.946473][ T15] -> #1 (cpu_hotplug_lock){++++}-{0:0}:
[17602.954050][ T15] lock_acquire+0xe4/0x25c
[17602.958841][ T15] cpus_read_lock+0x50/0x154
[17602.963807][ T15] static_key_slow_inc+0x18/0x30
[17602.969117][ T15] mem_cgroup_css_alloc+0x824/0x8b0
[17602.974689][ T15] cgroup_apply_control_enable+0x1d8/0x56c
[17602.980867][ T15] cgroup_apply_control+0x40/0x344
[17602.986352][ T15] cgroup_subtree_control_write+0x664/0x69c
[17602.992618][ T15] cgroup_file_write+0x130/0x2e8
[17602.997928][ T15] kernfs_fop_write+0x228/0x32c
[17603.003152][ T15] __vfs_write+0x84/0x1d8
[17603.007854][ T15] vfs_write+0x13c/0x1b4
[17603.012470][ T15] ksys_write+0xb0/0x120
[17603.017087][ T15] __arm64_sys_write+0x54/0x88
[17603.022223][ T15] do_el0_svc+0x128/0x1dc
[17603.026926][ T15] el0_sync_handler+0x150/0x250
[17603.032149][ T15] el0_sync+0x164/0x180
[17603.036674][ T15]
[17603.036674][ T15] -> #0 (cgroup_mutex){+.+.}-{3:3}:
[17603.043904][ T15] validate_chain+0x17bc/0x38b4
[17603.049128][ T15] __lock_acquire+0xefc/0x1180
[17603.054265][ T15] lock_acquire+0xe4/0x25c
[17603.059056][ T15] __mutex_lock_common+0x12c/0xf38
[17603.064540][ T15] mutex_lock_nested+0x40/0x50
[17603.069678][ T15] cgroup_transfer_tasks+0x130/0x2d8
[17603.075337][ T15] cpuset_hotplug_update_tasks+0x6d4/0x794
[17603.081515][ T15] cpuset_hotplug+0x42c/0x5bc
[17603.086566][ T15] cpuset_update_active_cpus+0x14/0x1c
[17603.092400][ T15] sched_cpu_deactivate+0x144/0x208
[17603.097971][ T15] cpuhp_invoke_callback+0x1dc/0x534
[17603.103628][ T15] cpuhp_thread_fun+0x27c/0x36c
[17603.108852][ T15] smpboot_thread_fn+0x388/0x53c
[17603.114164][ T15] kthread+0x208/0x224
[17603.118606][ T15] ret_from_fork+0x10/0x18
[17603.123392][ T15]
[17603.123392][ T15] other info that might help us debug this:
[17603.123392][ T15]
[17603.133473][ T15] Chain exists of:
[17603.133473][ T15] cgroup_mutex --> cpu_hotplug_lock --> cpuhp_state-down
[17603.133473][ T15]
[17603.146256][ T15] Possible unsafe locking scenario:
[17603.146256][ T15]
[17603.153560][ T15] CPU0 CPU1
[17603.158779][ T15] ---- ----
[17603.164000][ T15] lock(cpuhp_state-down);
[17603.168357][ T15] lock(cpu_hotplug_lock);
[17603.175231][ T15] lock(cpuhp_state-down);
[17603.182103][ T15] lock(cgroup_mutex);
[17603.186111][ T15]
[17603.186111][ T15] *** DEADLOCK ***
[17603.186111][ T15]
[17603.194111][ T15] 2 locks held by cpuhp/1/15:
[17603.198636][ T15] #0: ffff900012ae9408 (cpu_hotplug_lock){++++}-{0:0}, at: lockdep_acquire_cpus_lock+0xc/0x3c
[17603.208821][ T15] #1: ffff900012aeb2b0 (cpuhp_state-down){+.+.}-{0:0}, at: cpuhp_lock_acquire+0x8/0x48
[17603.218397][ T15]
[17603.218397][ T15] stack backtrace:
[17603.224146][ T15] CPU: 1 PID: 15 Comm: cpuhp/1 Tainted: G L 5.6.0-rc7-next-20200325+ #13
[17603.233708][ T15] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.11 06/18/2019
[17603.244138][ T15] Call trace:
[17603.247278][ T15] dump_backtrace+0x0/0x224
[17603.251633][ T15] show_stack+0x20/0x2c
[17603.255643][ T15] dump_stack+0xfc/0x184
[17603.259739][ T15] print_circular_bug+0x3dc/0x438
[17603.264616][ T15] check_noncircular+0x284/0x28c
[17603.269405][ T15] validate_chain+0x17bc/0x38b4
[17603.274108][ T15] __lock_acquire+0xefc/0x1180
[17603.278724][ T15] lock_acquire+0xe4/0x25c
[17603.282993][ T15] __mutex_lock_common+0x12c/0xf38
[17603.287957][ T15] mutex_lock_nested+0x40/0x50
[17603.292573][ T15] cgroup_transfer_tasks+0x130/0x2d8
[17603.297711][ T15] cpuset_hotplug_update_tasks+0x6d4/0x794
remove_tasks_in_empty_cpuset at kernel/cgroup/cpuset.c:2932
(inlined by) hotplug_update_tasks_legacy at kernel/cgroup/cpuset.c:2973
(inlined by) cpuset_hotplug_update_tasks at kernel/cgroup/cpuset.c:3097
[17603.303368][ T15] cpuset_hotplug+0x42c/0x5bc
[17603.307897][ T15] cpuset_update_active_cpus+0x14/0x1c
cpuset_update_active_cpus at kernel/cgroup/cpuset.c:3230
[17603.313208][ T15] sched_cpu_deactivate+0x144/0x208
[17603.318258][ T15] cpuhp_invoke_callback+0x1dc/0x534
[17603.323394][ T15] cpuhp_thread_fun+0x27c/0x36c
[136806][ T15] [17608.539300][ T16] IRQ 90: no longer affine to CPU1
Once it happens, the system is doomed.
[18797.018637][T25534] LTP: starting cpuhotplug02 (cpuhotplug02.sh -c 1 -l 1)
[18808.649761][ T16] IRQ 90: no longer affine to CPU1
[18808.655134][ T16] process 74665 (cpuhotplug_do_s) no longer affine to cpu1
[18808.656233][T74643] CPU1: shutdown
[18808.673016][T74643] psci: CPU1 killed (polled 10 ms)
[18816.737743][ T0] Detected PIPT I-cache on CPU1
[18816.742479][ T0] GICv3: CPU1: found redistributor 100 region 0:0x0000000401080000
[18816.750416][ T0] CPU1: Booted secondary processor 0x0000000100 [0x431f0af1]
[18820.809117][T25534] LTP: starting cpuhotplug03 (cpuhotplug03.sh -c 1 -l 1)
[18839.239220][T74677] CPU1: shutdown
[18839.255936][T74677] psci: CPU1 killed (polled 10 ms)
[18962.232428][ T0] Detected PIPT I-cache on CPU1
[18962.237230][ T0] GICv3: CPU1: found redistributor 100 region 0:0x0000000401080000
[18962.245291][ T0] CPU1: Booted secondary processor 0x0000000100 [0x431f0af1]
[19245.173443][T25534] LTP: starting cpuhotplug04 (cpuhotplug04.sh -l 1)
[19266.168715][ T12] IRQ 159: no longer affine to CPU0
[19266.174935][T79171] CPU0: shutdown
[19266.191548][T79171] psci: CPU0 killed (polled 20 ms)
[19465.919614][ T1307] INFO: task systemd:1 blocked for more than 122 seconds.
[19465.926605][ T1307] Tainted: G W L 5.6.0-rc7-next-20200324+ #12
[19465.934271][ T1307] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[19465.942854][ T1307] systemd D24464 1 0 0x00000000
[19465.949086][ T1307] Call trace:
[19465.952287][ T1307] __switch_to+0x13c/0x178
[19465.956578][ T1307] __schedule+0x65c/0x6fc
[19465.960814][ T1307] schedule+0xe4/0x1c0
[19465.964757][ T1307] percpu_rwsem_wait+0x270/0x2f4
[19465.969600][ T1307] __percpu_down_read+0x64/0x94
[19465.974325][ T1307] cpus_read_lock+0x150/0x154
[19465.978876][ T1307] static_key_slow_inc+0x18/0x30
[19465.983719][ T1307] mem_cgroup_css_alloc+0x810/0x89c
[19465.988790][ T1307] cgroup_apply_control_enable+0x1d8/0x564
[19465.994501][ T1307] cgroup_mkdir+0x168/0x1ec
[19465.998879][ T1307] kernfs_iop_mkdir+0x90/0xc4
[19466.003463][ T1307] vfs_mkdir+0xcc/0x110
[19466.007492][ T1307] do_mkdirat+0xe4/0x174
[19466.011641][ T1307] __arm64_sys_mkdirat+0x54/0x88
[19466.016450][ T1307] do_el0_svc+0x128/0x1dc
[19466.020685][ T1307] el0_sync_handler+0xd0/0x268
[19466.025319][ T1307] el0_sync+0x164/0x180
[19466.029352][ T1307] INFO: task cpuhp/1:15 blocked for more than 122 seconds.
[19466.036451][ T1307] Tainted: G W L 5.6.0-rc7-next-20200324+ #12
[19466.044070][ T1307] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[19466.052643][ T1307] cpuhp/1 D27984 15 2 0x00000028
[19466.058876][ T1307] Call trace:
[19466.062070][ T1307] __switch_to+0x13c/0x178
[19466.066358][ T1307] __schedule+0x65c/0x6fc
[19466.070593][ T1307] schedule+0xe4/0x1c0
[19466.074534][ T1307] schedule_preempt_disabled+0x3c/0x68
[19466.079898][ T1307] __mutex_lock_common+0x6fc/0xf38
[19466.084882][ T1307] mutex_lock_nested+0x40/0x50
[19466.089552][ T1307] cgroup_transfer_tasks+0x130/0x2d8
[19466.094709][ T1307] cpuset_hotplug_update_tasks+0x6d4/0x794
[19466.100418][ T1307] cpuset_hotplug+0x42c/0x5bc
[19466.104967][ T1307] cpuset_update_active_cpus+0x14/0x1c
[19466.110333][ T1307] sched_cpu_deactivate+0x144/0x208
[19466.115402][ T1307] cpuhp_invoke_callback+0x1dc/0x534
[19466.120592][ T1307] cpuhp_thread_fun+0x27c/0x36c
[19466.125314][ T1307] smpboot_thread_fn+0x388/0x53c
[19466.130156][ T1307] kthread+0x1cc/0x1e8
[19466.134096][ T1307] ret_from_fork+0x10/0x18
[19466.139100][ T1307] INFO: task systemd-journal:2575 blocked for more than 123 seconds.
[19466.147101][ T1307] Tainted: G W L 5.6.0-rc7-next-20200324+ #12
[19466.154724][ T1307] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[19466.163296][ T1307] systemd-journal D24384 2575 1 0x00000800
[19466.169557][ T1307] Call trace:
[19466.172718][ T1307] __switch_to+0x13c/0x178
[19466.177007][ T1307] __schedule+0x65c/0x6fc
[19466.181261][ T1307] schedule+0xe4/0x1c0
[19466.185204][ T1307] schedule_preempt_disabled+0x3c/0x68
[19466.190569][ T1307] __mutex_lock_common+0x6fc/0xf38
[19466.195552][ T1307] mutex_lock_nested+0x40/0x50
[19466.200221][ T1307] proc_cgroup_show+0x64/0x328
[19466.204857][ T1307] proc_single_show+0x84/0xbc
[19466.209406][ T1307] seq_read+0x380/0x930
[19466.213467][ T1307] __vfs_read+0x84/0x1d0
[19466.217582][ T1307] vfs_read+0xec/0x120
[19466.221555][ T1307] ksys_read+0xb0/0x120
[19466.225583][ T1307] __arm64_sys_read+0x54/0x88
[19466.230164][ T1307] do_el0_svc+0x128/0x1dc
[19466.234365][ T1307] el0_sync_handler+0xd0/0x268
[19466.238999][ T1307] el0_sync+0x164/0x180
[19466.243094][ T1307] INFO: task irqbalance:3677 blocked for more than 123 seconds.
[19466.250656][ T1307] Tainted: G W L 5.6.0-rc7-next-20200324+ #12
[19466.258240][ T1307] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[19466.266818][ T1307] irqbalance D24592 3677 1 0x00000000
[19466.273080][ T1307] Call trace:
[19466.276241][ T1307] __switch_to+0x13c/0x178
[19466.280563][ T1307] __schedule+0x65c/0x6fc
[19466.284765][ T1307] schedule+0xe4/0x1c0
[19466.288706][ T1307] schedule_preempt_disabled+0x3c/0x68
[19466.294070][ T1307] __mutex_lock_common+0x6fc/0xf38
[19466.299053][ T1307] mutex_lock_nested+0x40/0x50
[19466.303723][ T1307] online_show+0x2c/0x6c
[19466.307837][ T1307] dev_attr_show+0x50/0x90
[19466.312158][ T1307] sysfs_kf_seq_show+0x198/0x2d4
[19466.316967][ T1307] kernfs_seq_show+0xa4/0xcc
[19466.321461][ T1307] seq_read+0x380/0x930
[19466.325489][ T1307] kernfs_fop_read+0xa8/0x35c
[19466.330070][ T1307] __vfs_read+0x84/0x1d0
[19466.334184][ T1307] vfs_read+0xec/0x120
[19466.338124][ T1307] ksys_read+0xb0/0x120
[19466.342185][ T1307] __arm64_sys_read+0x54/0x88
[19466.346733][ T1307] do_el0_svc+0x128/0x1dc
[19466.350967][ T1307] el0_sync_handler+0xd0/0x268
[19466.355602][ T1307] el0_sync+0x164/0x180
[19466.359988][ T1307] INFO: task kworker/1:1:74671 blocked for more than 123 seconds.
[19466.367661][ T1307] Tainted: G W L 5.6.0-rc7-next-20200324+ #12
[19466.375284][ T1307] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[19466.383856][ T1307] kworker/1:1 D29680 74671 2 0x00000028
[19466.390130][ T1307] Workqueue: events vmstat_shepherd
[19466.395202][ T1307] Call trace:
[19466.398362][ T1307] __switch_to+0x13c/0x178
[19466.402684][ T1307] __schedule+0x65c/0x6fc
[19466.406885][ T1307] schedule+0xe4/0x1c0
[19466.410860][ T1307] percpu_rwsem_wait+0x270/0x2f4
[19466.415669][ T1307] __percpu_down_read+0x64/0x94
[19466.420425][ T1307] cpus_read_lock+0x150/0x154
[19466.424973][ T1307] vmstat_shepherd+0x20/0x16c
[19466.429556][ T1307] process_one_work+0x2c0/0x4c4
[19466.434278][ T1307] worker_thread+0x320/0x4b4
[19466.438739][ T1307] kthread+0x1cc/0x1e8
[19466.442730][ T1307] ret_from_fork+0x10/0x18
[19466.447021][ T1307] INFO: task cpuhotplug04.sh:79171 blocked 00324+ #12
[19466.471271][ T13097][ T1307] __sule+0xe4/0x1c0
1a4/0x2b4
[19466.502725][ T1307] __wait_for_common+0x58/0xa8
[19466.507361][ T1307] wait_for_completion+0x2c/0x38
[19466.512210][ T1307] __cpuhp_kick_ap+0xe8/0xf8
[19466.516671][ T1307] cpuhp_kick_ap+0x34/0x6c
[19466.520992][ T1307] cpuhp_kick_ap_work+0xb4/0x118
[19466.525800][ T1307] _cpu_down+0x16c/0x3c0
[19466.529953][ T1307] cpu_down+0x54/0x74
[19466.533808][ T1307] cpu_subsys_offline+0x24/0x30
[19466.538529][ T1307] device_offline+0x94/0xf0
[19466.542937][ T1307] online_store+0x80/0x110
[19466.547224][ T1307] dev_attr_store+0x5c/0x78
[19466.551633][ T1307] sysfs_kf_write+0x9c/0xcc
[19466.556008][ T1307] kernfs_fop_write+0x228/0x32c
[19466.560769][ T1307] __vfs_write+0x84/0x1d8
[19466.564969][ T1307] vfs_write+0x13c/0x1b4
[19466.569083][ T1307] ksys_write+0xb0/0x120
[19466.573235][ T1307] __arm64_sys_write+0x54/0x88
[19466.577870][ T1307] do_el0_svc+0x128/0x1dc
[19466.582105][ T1307] el0_sync_handler+0xd0/0x268
[19466.586739][ T1307] el0_sync+0x164/0x180
[19466.590809][ T1307] INFO: lockdep is turned off.
[19588.799236][ T1307] INFO: task systemd:1 blocked for more than 245 seconds.
[19588.806218][ T1307] Tainted: G W L 5.6.0-rc7-next-20200324+ #12
[19588.813859][ T1307] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[19588.822433][ T1307] systemd D24464 1 0 0x00000000
[19588.828661][ T1307] Call trace:
[19588.831858][ T1307] __switch_to+0x13c/0x178
[19588.836146][ T1307] __schedule+0x65c/0x6fc
[19588.840382][ T1307] schedule+0xeown_read+0x64/0x+0x18/0x30
[195_enable+0x1d8/0x90/0xc4
[19588.05][ T1307] __arm64_sys_mkdirat+0x54/0x88
[19588.896014][ T1307] do_el0_svc+0x128/0x1dc
[19588.900249][ T1307] el0_sync_handler+0xd0/0x268
[19588.904883][ T1307] el0_sync+0x164/0x180
[19588.908915][ T1307] INFO: task cpuhp/1:15 blocked for more than 245 seconds.
[19588.916013][ T1307] Tainted: G W L 5.6.0-rc7-next-20200324+ #12
[19588.923632][ T1307] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[19588.932204][ T1307] cpuhp/1 D27984 15 2 0x00000028
[19588.938436][ T1307] Call trace:
[19588.941630][ T1307] __switch_to+0x13c/0x178
[19588.945919][ T1307] __schedule+0x65c/0x6fc
[19588.950154][ T1307] schedule+0xe4/0x1c0
[19588.954095][ T1307] schedule_preempt_disabled+0x3c/0x68
[19588.959459][ T1307] __mutex_lock_common+0x6fc/0xf38
[19588.964443][ T1307] mutex_lock_nested+0x40/0x50
[19588.969079][ T1307] cgroup_transfer_tasks+0x130/0x2d8
[19588.974269][ T1307] cpuset_hotplug_update_tasks+0x6d4/0x794
[19588.979980][ T1307] cpuset_hotplug+0x42c/0x5bc
[19588.984529][ T1307] cpuset_update_active_cpus+0x14/0x1c
[19588.989895][ T1307] sched_cpu_deactivate+0x144/0x208
[19588.994965][ T1307] cpuhp_invoke_callback+0x1dc/0x534
[19589.000154][ T1307] cpuhp_thread_fun+0x27c/0x36c
[19589.004876][ T1307] smpboot_thread_fn+0x388/0x53c
[19589.009717][ T1307] kthread+0x1cc/0x1e8
[19589.013657][ T1307] ret_from_fork+0x10/0x18
[19589.018654][ T1307] INFO: task systemd-journal:2575 blocked for more than 245 seconds.
[19589.026653][ T1307] Tainted: G W L 5.6.0-rc7-next-20200324+ #12
[19589.034277][ T1307] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[19589.042849][ T1307] systemd-journal D24384 2575 1 0x00000800
[19589.049076][ T1307] Call trace:
[19589.052271][ T1307] __switch_to+0x13c/0x178
[19589.056560][ T1307] __schedule+0x65c/0x6fc
[19589.060812][ T1307] schedule+0xe4/0x1c0
[19589.064754][ T1307] schedule_preempt_disabled+0x3c/0x68
[19589.070118][ T1307] __mutex_lock_common+0x6fc/0xf38
[19589.075101][ T1307] mutex_lock_nested+0x40/0x50
[19589.079770][ T1307] proc_cgroup_show+0x64/0x328
[19589.084406][ T1307] proc_single_show+0x84/0xbc
[19589.088953][ T1307] seq_read+0x380/0x930
[19589.093015][ T1307] __vfs_read+0x84/0x1d0
[19589.097129][ T1307] vfs_read+0xec/0x120
[19589.101102][ T1307] ksys_read+0xb0/0x120
[19589.105130][ T1307] __arm64_sys_read+0x54/0x88
[19589.109711][ T1307] do_el0_svc+0x128/0x1dc
[19589.113913][ T1307] el0_sync_handler+0xd0/0x268
[19589.118547][ T1307] el0_sync+0x164/0x180
[19589.122630][ T1307] INFO: task irqbalance:3677 blocked for more than 246 seconds.
[19589.130165][ T1307] Tainted: G W L 5.6.0-rc7-next-20200324+ #12
[19589.137750][ T1307] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[19589.146321][ T1307] irqbalance D24592 3677 1 0x00000000
[19589.152582][ T1307] Call trace:
[19589.155743][ T1307] __switch_to+0x13c/0x178
[19589.160065][ T1307] __schedule+0x65c/0x6fc
[19589.164266][ T1307] schedule+0xe4/0x1c0
[19589.168207][ T1307] schedule_preempt_disabled+0x3c/0x68
[19589.173571][ T1307] __mutex_lock_common+0x6fc/0xf38
[19589.178554][ T1307] mutex_lock_nested+0x40/0x50
[19589.183223][ T1307] online_show+0x2c/0x6c
[19589.187337][ T1307] dev_attr_show+0x50/0x90
[19589.191659][ T1307] sysfs_kf_seq_show+0x198/0x2d4
[19589.196469][ T1307] kernfs_seq_show+0xa4/0xcc
[19589.200963][ T1307] seq_read+0x380/0x930
[19589.204991][ T1307] kernfs_fop_read+0xa8/0x35c
[19589.209573][ T1307] __vfs_read+0x84/0x1d0
[19589.213687][ T1307] vfs_read+0xec/0x120
[19589.217627][ T1307] ksys_read+0xb0/0x120
[19589.221688][ T1307] __arm64_sys_read+0x54/0x88
[19589.226235][ T1307] do_el0_svc+0x128/0x1dc
[19589.230470][ T1307] el0_sync_handler+0xd0/0x268
[19589.235105][ T1307] el0_sync+0x164/0x180
[19589.239491][ T1307] INFO: lockdep is turned off.
[21665.996141][ T3790] process 3847 (rngd) no longer affine to cpu1
[25274.311890][ T3790] process 3846 (rngd) no longer affine to cpu0
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Deadlock due to "cpuset: Make cpuset hotplug synchronous"
2020-03-25 19:16 Deadlock due to "cpuset: Make cpuset hotplug synchronous" Qian Cai
@ 2020-03-25 19:19 ` Tejun Heo
2020-03-26 10:15 ` Qais Yousef
2020-04-02 2:00 ` Qian Cai
2020-03-25 20:20 ` Peter Zijlstra
1 sibling, 2 replies; 11+ messages in thread
From: Tejun Heo @ 2020-03-25 19:19 UTC (permalink / raw)
To: Qian Cai; +Cc: Prateek Sood, Li Zefan, cgroups, LKML, Peter Zijlstra
On Wed, Mar 25, 2020 at 03:16:56PM -0400, Qian Cai wrote:
> The linux-next commit a49e4629b5ed (“cpuset: Make cpuset hotplug synchronous”)
> introduced real deadlocks with CPU hotplug as showed in the lockdep splat, since it is
> now making a relation from cpu_hotplug_lock —> cgroup_mutex.
Prateek, can you please take a look? Given that the merge window is just around
the corner, we might have to revert and retry later if it can't be resolved
quickly.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Deadlock due to "cpuset: Make cpuset hotplug synchronous"
2020-03-25 19:16 Deadlock due to "cpuset: Make cpuset hotplug synchronous" Qian Cai
2020-03-25 19:19 ` Tejun Heo
@ 2020-03-25 20:20 ` Peter Zijlstra
1 sibling, 0 replies; 11+ messages in thread
From: Peter Zijlstra @ 2020-03-25 20:20 UTC (permalink / raw)
To: Qian Cai; +Cc: Prateek Sood, Li Zefan, Tejun Heo, cgroups, LKML
On Wed, Mar 25, 2020 at 03:16:56PM -0400, Qian Cai wrote:
> [17602.773334][ T15] ======================================================
> [17602.780207][ T15] WARNING: possible circular locking dependency detected
> [17602.787081][ T15] 5.6.0-rc7-next-20200325+ #13 Tainted: G L
> [17602.794125][ T15] ------------------------------------------------------
> [17602.800997][ T15] cpuhp/1/15 is trying to acquire lock:
> [17602.806392][ T15] ffff900012cb7bf0 (cgroup_mutex){+.+.}-{3:3}, at: cgroup_transfer_tasks+0x130/0x2d8
> [17602.815718][ T15]
> [17602.815718][ T15] but task is already holding lock:
> [17602.822934][ T15] ffff900012aeb2b0 (cpuhp_state-down){+.+.}-{0:0}, at: cpuhp_lock_acquire+0x8/0x48
> [17602.832078][ T15]
> [17602.832078][ T15] which lock already depends on the new lock.
> [17602.832078][ T15]
> [17602.842334][ T15]
> [17602.842334][ T15] the existing dependency chain (in reverse order) is:
> [17602.946473][ T15] -> #1 (cpu_hotplug_lock){++++}-{0:0}:
> [17602.954050][ T15] lock_acquire+0xe4/0x25c
> [17602.958841][ T15] cpus_read_lock+0x50/0x154
> [17602.963807][ T15] static_key_slow_inc+0x18/0x30
> [17602.969117][ T15] mem_cgroup_css_alloc+0x824/0x8b0
> [17602.974689][ T15] cgroup_apply_control_enable+0x1d8/0x56c
> [17602.980867][ T15] cgroup_apply_control+0x40/0x344
> [17602.986352][ T15] cgroup_subtree_control_write+0x664/0x69c
> [17602.992618][ T15] cgroup_file_write+0x130/0x2e8
> [17602.997928][ T15] kernfs_fop_write+0x228/0x32c
> [17603.003152][ T15] __vfs_write+0x84/0x1d8
> [17603.007854][ T15] vfs_write+0x13c/0x1b4
> [17603.012470][ T15] ksys_write+0xb0/0x120
> [17603.017087][ T15] __arm64_sys_write+0x54/0x88
> [17603.022223][ T15] do_el0_svc+0x128/0x1dc
> [17603.026926][ T15] el0_sync_handler+0x150/0x250
> [17603.032149][ T15] el0_sync+0x164/0x180
> [17603.036674][ T15]
> [17603.036674][ T15] -> #0 (cgroup_mutex){+.+.}-{3:3}:
> [17603.123392][ T15] other info that might help us debug this:
> [17603.123392][ T15]
> [17603.133473][ T15] Chain exists of:
> [17603.133473][ T15] cgroup_mutex --> cpu_hotplug_lock --> cpuhp_state-down
> [17603.133473][ T15]
> [17603.194111][ T15] 2 locks held by cpuhp/1/15:
> [17603.198636][ T15] #0: ffff900012ae9408 (cpu_hotplug_lock){++++}-{0:0}, at: lockdep_acquire_cpus_lock+0xc/0x3c
> [17603.218397][ T15] stack backtrace:
> [17603.224146][ T15] CPU: 1 PID: 15 Comm: cpuhp/1 Tainted: G L 5.6.0-rc7-next-20200325+ #13
> [17603.233708][ T15] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.11 06/18/2019
> [17603.244138][ T15] Call trace:
> [17603.287957][ T15] mutex_lock_nested+0x40/0x50
> [17603.292573][ T15] cgroup_transfer_tasks+0x130/0x2d8
> [17603.297711][ T15] cpuset_hotplug_update_tasks+0x6d4/0x794
> remove_tasks_in_empty_cpuset at kernel/cgroup/cpuset.c:2932
> (inlined by) hotplug_update_tasks_legacy at kernel/cgroup/cpuset.c:2973
> (inlined by) cpuset_hotplug_update_tasks at kernel/cgroup/cpuset.c:3097
> [17603.303368][ T15] cpuset_hotplug+0x42c/0x5bc
> [17603.307897][ T15] cpuset_update_active_cpus+0x14/0x1c
> cpuset_update_active_cpus at kernel/cgroup/cpuset.c:3230
> [17603.313208][ T15] sched_cpu_deactivate+0x144/0x208
> [17603.318258][ T15] cpuhp_invoke_callback+0x1dc/0x534
> [17603.323394][ T15] cpuhp_thread_fun+0x27c/0x36c
Hurmph,.. that one is vexing indeed. The only possible solution is
changing mem_cgroup_css_alloc(), but that also doesn't look too pretty.
I so hate hotplug...
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Deadlock due to "cpuset: Make cpuset hotplug synchronous"
2020-03-25 19:19 ` Tejun Heo
@ 2020-03-26 10:15 ` Qais Yousef
2020-04-03 14:55 ` Tejun Heo
2020-04-02 2:00 ` Qian Cai
1 sibling, 1 reply; 11+ messages in thread
From: Qais Yousef @ 2020-03-26 10:15 UTC (permalink / raw)
To: Tejun Heo; +Cc: Qian Cai, Prateek Sood, Li Zefan, cgroups, LKML, Peter Zijlstra
On 03/25/20 15:19, Tejun Heo wrote:
> On Wed, Mar 25, 2020 at 03:16:56PM -0400, Qian Cai wrote:
> > The linux-next commit a49e4629b5ed (“cpuset: Make cpuset hotplug synchronous”)
> > introduced real deadlocks with CPU hotplug as showed in the lockdep splat, since it is
> > now making a relation from cpu_hotplug_lock —> cgroup_mutex.
>
> Prateek, can you please take a look? Given that the merge window is just around
> the corner, we might have to revert and retry later if it can't be resolved
> quickly.
I've ran cpuset_hotplug and cpuhotplug LTP tests using next-20200325 but
couldn't reproduce it.
Hopefully that can be fixed, but if you had to revert it, do you mind picking
this instead to fix the LTP issue I encountered before?
https://lore.kernel.org/lkml/20200211141554.24181-1-qais.yousef@arm.com/
Thanks!
--
Qais Yousef
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Deadlock due to "cpuset: Make cpuset hotplug synchronous"
2020-03-25 19:19 ` Tejun Heo
2020-03-26 10:15 ` Qais Yousef
@ 2020-04-02 2:00 ` Qian Cai
2020-04-03 14:54 ` Tejun Heo
1 sibling, 1 reply; 11+ messages in thread
From: Qian Cai @ 2020-04-02 2:00 UTC (permalink / raw)
To: Tejun Heo; +Cc: Prateek Sood, Li Zefan, cgroups, LKML, Peter Zijlstra
> On Mar 25, 2020, at 3:19 PM, Tejun Heo <tj@kernel.org> wrote:
>
> On Wed, Mar 25, 2020 at 03:16:56PM -0400, Qian Cai wrote:
>> The linux-next commit a49e4629b5ed (“cpuset: Make cpuset hotplug synchronous”)
>> introduced real deadlocks with CPU hotplug as showed in the lockdep splat, since it is
>> now making a relation from cpu_hotplug_lock —> cgroup_mutex.
>
> Prateek, can you please take a look? Given that the merge window is just around
> the corner, we might have to revert and retry later if it can't be resolved
> quickly.
Tejun, can you back off this commit now given it seems nobody is trying to rescue it?
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Deadlock due to "cpuset: Make cpuset hotplug synchronous"
2020-04-02 2:00 ` Qian Cai
@ 2020-04-03 14:54 ` Tejun Heo
0 siblings, 0 replies; 11+ messages in thread
From: Tejun Heo @ 2020-04-03 14:54 UTC (permalink / raw)
To: Qian Cai; +Cc: Prateek Sood, Li Zefan, cgroups, LKML, Peter Zijlstra
On Wed, Apr 01, 2020 at 10:00:31PM -0400, Qian Cai wrote:
>
>
> > On Mar 25, 2020, at 3:19 PM, Tejun Heo <tj@kernel.org> wrote:
> >
> > On Wed, Mar 25, 2020 at 03:16:56PM -0400, Qian Cai wrote:
> >> The linux-next commit a49e4629b5ed (“cpuset: Make cpuset hotplug synchronous”)
> >> introduced real deadlocks with CPU hotplug as showed in the lockdep splat, since it is
> >> now making a relation from cpu_hotplug_lock —> cgroup_mutex.
> >
> > Prateek, can you please take a look? Given that the merge window is just around
> > the corner, we might have to revert and retry later if it can't be resolved
> > quickly.
>
> Tejun, can you back off this commit now given it seems nobody is trying to rescue it?
Yeah, gonna revert it before sending out the pull request.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Deadlock due to "cpuset: Make cpuset hotplug synchronous"
2020-03-26 10:15 ` Qais Yousef
@ 2020-04-03 14:55 ` Tejun Heo
2020-04-06 10:55 ` Qais Yousef
0 siblings, 1 reply; 11+ messages in thread
From: Tejun Heo @ 2020-04-03 14:55 UTC (permalink / raw)
To: Qais Yousef
Cc: Qian Cai, Prateek Sood, Li Zefan, cgroups, LKML, Peter Zijlstra
On Thu, Mar 26, 2020 at 10:15:32AM +0000, Qais Yousef wrote:
> On 03/25/20 15:19, Tejun Heo wrote:
> > On Wed, Mar 25, 2020 at 03:16:56PM -0400, Qian Cai wrote:
> > > The linux-next commit a49e4629b5ed (“cpuset: Make cpuset hotplug synchronous”)
> > > introduced real deadlocks with CPU hotplug as showed in the lockdep splat, since it is
> > > now making a relation from cpu_hotplug_lock —> cgroup_mutex.
> >
> > Prateek, can you please take a look? Given that the merge window is just around
> > the corner, we might have to revert and retry later if it can't be resolved
> > quickly.
>
> I've ran cpuset_hotplug and cpuhotplug LTP tests using next-20200325 but
> couldn't reproduce it.
>
> Hopefully that can be fixed, but if you had to revert it, do you mind picking
> this instead to fix the LTP issue I encountered before?
>
> https://lore.kernel.org/lkml/20200211141554.24181-1-qais.yousef@arm.com/
So, I'd rather not, for now anyway. It isn't a real problem and I don't wanna
add a wait vector there.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Deadlock due to "cpuset: Make cpuset hotplug synchronous"
2020-04-03 14:55 ` Tejun Heo
@ 2020-04-06 10:55 ` Qais Yousef
2020-04-06 14:34 ` Tejun Heo
0 siblings, 1 reply; 11+ messages in thread
From: Qais Yousef @ 2020-04-06 10:55 UTC (permalink / raw)
To: Tejun Heo; +Cc: Qian Cai, Prateek Sood, Li Zefan, cgroups, LKML, Peter Zijlstra
On 04/03/20 10:55, Tejun Heo wrote:
> On Thu, Mar 26, 2020 at 10:15:32AM +0000, Qais Yousef wrote:
> > On 03/25/20 15:19, Tejun Heo wrote:
> > > On Wed, Mar 25, 2020 at 03:16:56PM -0400, Qian Cai wrote:
> > > > The linux-next commit a49e4629b5ed (“cpuset: Make cpuset hotplug synchronous”)
> > > > introduced real deadlocks with CPU hotplug as showed in the lockdep splat, since it is
> > > > now making a relation from cpu_hotplug_lock —> cgroup_mutex.
> > >
> > > Prateek, can you please take a look? Given that the merge window is just around
> > > the corner, we might have to revert and retry later if it can't be resolved
> > > quickly.
> >
> > I've ran cpuset_hotplug and cpuhotplug LTP tests using next-20200325 but
> > couldn't reproduce it.
> >
> > Hopefully that can be fixed, but if you had to revert it, do you mind picking
> > this instead to fix the LTP issue I encountered before?
> >
> > https://lore.kernel.org/lkml/20200211141554.24181-1-qais.yousef@arm.com/
>
> So, I'd rather not, for now anyway. It isn't a real problem and I don't wanna
> add a wait vector there.
What would be the right approach to get a fix in then? We have been skipping
this test for a while and we'd like to enable it but this failure is a
blocking issue.
Android relies on cpuset and some devices use hotplug to manage thermal/power.
So it's an interesting combination to be able to test for us.
Thanks
--
Qais Yousef
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Deadlock due to "cpuset: Make cpuset hotplug synchronous"
2020-04-06 10:55 ` Qais Yousef
@ 2020-04-06 14:34 ` Tejun Heo
2020-04-07 13:07 ` Qais Yousef
0 siblings, 1 reply; 11+ messages in thread
From: Tejun Heo @ 2020-04-06 14:34 UTC (permalink / raw)
To: Qais Yousef
Cc: Qian Cai, Prateek Sood, Li Zefan, cgroups, LKML, Peter Zijlstra
On Mon, Apr 06, 2020 at 11:55:23AM +0100, Qais Yousef wrote:
> What would be the right approach to get a fix in then? We have been skipping
> this test for a while and we'd like to enable it but this failure is a
> blocking issue.
Update the test so that it accounts for the async nature of the operation?
--
tejun
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Deadlock due to "cpuset: Make cpuset hotplug synchronous"
2020-04-06 14:34 ` Tejun Heo
@ 2020-04-07 13:07 ` Qais Yousef
2020-04-07 13:35 ` Tejun Heo
0 siblings, 1 reply; 11+ messages in thread
From: Qais Yousef @ 2020-04-07 13:07 UTC (permalink / raw)
To: Tejun Heo; +Cc: Qian Cai, Prateek Sood, Li Zefan, cgroups, LKML, Peter Zijlstra
On 04/06/20 10:34, Tejun Heo wrote:
> On Mon, Apr 06, 2020 at 11:55:23AM +0100, Qais Yousef wrote:
> > What would be the right approach to get a fix in then? We have been skipping
> > this test for a while and we'd like to enable it but this failure is a
> > blocking issue.
>
> Update the test so that it accounts for the async nature of the operation?
With a sleep() between the hp ops and the read? Isn't this a nasty hack?
I was just trying to find out what can be done for 5.7. But based on your
response this is sadly a dead end. For no reason I can comprehend.
--
Qais Yousef
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Deadlock due to "cpuset: Make cpuset hotplug synchronous"
2020-04-07 13:07 ` Qais Yousef
@ 2020-04-07 13:35 ` Tejun Heo
0 siblings, 0 replies; 11+ messages in thread
From: Tejun Heo @ 2020-04-07 13:35 UTC (permalink / raw)
To: Qais Yousef
Cc: Qian Cai, Prateek Sood, Li Zefan, cgroups, LKML, Peter Zijlstra
On Tue, Apr 07, 2020 at 02:07:26PM +0100, Qais Yousef wrote:
> On 04/06/20 10:34, Tejun Heo wrote:
> > On Mon, Apr 06, 2020 at 11:55:23AM +0100, Qais Yousef wrote:
> > > What would be the right approach to get a fix in then? We have been skipping
> > > this test for a while and we'd like to enable it but this failure is a
> > > blocking issue.
> >
> > Update the test so that it accounts for the async nature of the operation?
>
> With a sleep() between the hp ops and the read? Isn't this a nasty hack?
Some operations are just async. Why is that inherently wrong?
> I was just trying to find out what can be done for 5.7. But based on your
> response this is sadly a dead end. For no reason I can comprehend.
Because it's creating real issues while trying to address a non-real problem.
It's dumb. Fix the non-real part.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2020-04-07 13:35 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-25 19:16 Deadlock due to "cpuset: Make cpuset hotplug synchronous" Qian Cai
2020-03-25 19:19 ` Tejun Heo
2020-03-26 10:15 ` Qais Yousef
2020-04-03 14:55 ` Tejun Heo
2020-04-06 10:55 ` Qais Yousef
2020-04-06 14:34 ` Tejun Heo
2020-04-07 13:07 ` Qais Yousef
2020-04-07 13:35 ` Tejun Heo
2020-04-02 2:00 ` Qian Cai
2020-04-03 14:54 ` Tejun Heo
2020-03-25 20:20 ` Peter Zijlstra
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).