From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753672AbdIDN1o (ORCPT ); Mon, 4 Sep 2017 09:27:44 -0400 Received: from mout.gmx.net ([212.227.17.21]:61515 "EHLO mout.gmx.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752835AbdIDN1m (ORCPT ); Mon, 4 Sep 2017 09:27:42 -0400 Message-ID: <1504531627.10288.5.camel@gmx.de> Subject: Re: hotplug lockdep splat (tip) From: Mike Galbraith To: Peter Zijlstra Cc: LKML , Thomas Gleixner , Byungchul Park Date: Mon, 04 Sep 2017 15:27:07 +0200 In-Reply-To: <20170904075502.74pgfd2twdfklspw@hirez.programming.kicks-ass.net> References: <1504350596.16793.44.camel@gmx.de> <1504421975.30792.32.camel@gmx.de> <20170904075502.74pgfd2twdfklspw@hirez.programming.kicks-ass.net> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.20.5 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:JprBk6xu4ljZcuQgnIw7qpJ8AdYqSafS3YJIu8PMY3uNDu+uSbG D8/d3uKZ+xTaHkt5eo/yL1go906KC4tlb3uHOt6GYz3qW8EmTwTTM719Z/sHWekYWhaMcJ/ EjLnUP1YIFsRCGQkUkZLz9VSvzXYEXMdm/D8Rqs83qFqG1fMTDUPeZfjg11KzusCKUI9fNY /rlkR6pO9wTBNfegqI0+g== X-UI-Out-Filterresults: notjunk:1;V01:K0:enzFzluXsBk=:2DAGmcMBZNvaAP7eGbIcc4 JWCajK4AaMRUAWDZ1vLUZNnOSHjTJBxh9jEsiFTWHhTWB3TBWnixWem7e8evgJGG+e+8Cz2Eb XbDFva25mgPfEa1I9phsV+RUl9/Yi8c9LIq4/JiemVFdEAQgzGTXgrPeLKqVLCMPFmQ68dOGS jgi1Yoo43VSOaqtDoQn/EcqUqxdbK/JlgTC30YMHIEIpTo2QrZA6HUkpfkDoT5MawjTp11pDl ek0jdYeYx4i21XuFv/1j0IBNO4gPI7fOAZtb9cXdWKRNRZFxIuA+4GC8oXCXXK6FkfP6jr9Vu V3hv4LROrLsyzNTmLzEXmAf5eq5UhpAvAkOrQALMG9viuN6zXf7P28xfvbpYeKBA+b0Clmn5k lkrDFqcxqr32OGxPMWJscAj3TL4TLIofdLNNT2Si2wGmVa7KMHs4uJ8ZSYJGCY4U5shsbucx7 p0SdpUnXRQS+gGlwCt2Wm8/jzPKF9JKYKnvcsysYhXQ39R3ObpQh3Ew6LYghy7vZr51na0jCH w4CUiaEtswZj3DptB+bwxFT1K07ApnoKK2r7IBLo3d/C19uV5MNZa8UR7igOS/KcrCffbbkdV KcU2NtxnqvSX89bSOqjcSURuXL9qQDQDO3poLhrwoYYG+7VIFKped3ZJHguV+R2UtHmuN+MV0 1n0MgC7A0dsym5tDTkfExALMfQSN84bKXP/vgHyRs+/tvOH7s+Rlxtm7eaoNEsG+6dKr9qi7U y263D3arPeXTl8O3Q2HA5zctwMU1CCVaKjk8ctvHH76NUfFE3qtN4O85t/HY7AEXbNR+IhIdn FjFWIOjiO0vBUufjjECEAjW2UdupGNzpNR+VNc82cybNMEVjrs= Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 2017-09-04 at 09:55 +0200, Peter Zijlstra wrote: > On Sun, Sep 03, 2017 at 08:59:35AM +0200, Mike Galbraith wrote: > > > > [ 126.626908] Unregister pv shared memory for cpu 1 > > [ 126.631041] > > [ 126.631269] ====================================================== > > [ 126.632442] WARNING: possible circular locking dependency detected > > [ 126.633599] 4.13.0.g06260ca-tip-lockdep #2 Tainted: G E > > [ 126.634380] ------------------------------------------------------ > > [ 126.635124] stress-cpu-hotp/3156 is trying to acquire lock: > > [ 126.635804] ((complete)&st->done){+.+.}, at: [] takedown_cpu+0x8a/0xf0 > > [ 126.636809] > > [ 126.636809] but task is already holding lock: > > [ 126.637567] (sparse_irq_lock){+.+.}, at: [] irq_lock_sparse+0x17/0x20 > > [ 126.638665] > > https://lkml.kernel.org/r/20170829193416.GC32112@worktop.programming.kicks-ass.net Well, flavor of gripe changed. [ 164.114290] ====================================================== [ 164.115146] WARNING: possible circular locking dependency detected [ 164.115751] 4.13.0.g90abd70-tip-lockdep #4 Tainted: G E [ 164.116348] ------------------------------------------------------ [ 164.116919] cpuhp/0/12 is trying to acquire lock: [ 164.117381] (cpuhp_state){+.+.}, at: [] cpuhp_thread_fun+0x2a/0x160 [ 164.118097] but now in release context of a crosslock acquired at the following: [ 164.118845] ((complete)&per_cpu_ptr(&cpuhp_state, i)->done#2){+.+.}, at: [] cpuhp_issue_call+0x13f/0x170 [ 164.119789] which lock already depends on the new lock. [ 164.120483] the existing dependency chain (in reverse order) is: [ 164.121121] -> #2 ((complete)&per_cpu_ptr(&cpuhp_state, i)->done#2){+.+.}: [ 164.121840] __lock_acquire+0x116a/0x11c0 [ 164.122382] lock_acquire+0xbd/0x200 [ 164.122741] wait_for_completion+0x53/0x1a0 [ 164.123181] takedown_cpu+0x8a/0xf0 [ 164.123533] cpuhp_invoke_callback+0x1f6/0x830 [ 164.123967] cpuhp_down_callbacks+0x42/0x80 [ 164.124433] _cpu_down+0x155/0x190 [ 164.124779] do_cpu_down+0x3e/0x60 [ 164.125127] cpu_down+0x10/0x20 [ 164.125456] cpu_subsys_offline+0x14/0x20 [ 164.125937] device_offline+0x8c/0xb0 [ 164.126487] online_store+0x40/0x80 [ 164.126943] dev_attr_store+0x18/0x30 [ 164.127335] sysfs_kf_write+0x44/0x60 [ 164.127703] kernfs_fop_write+0x113/0x1a0 [ 164.128141] __vfs_write+0x28/0x150 [ 164.128532] vfs_write+0xc7/0x1c0 [ 164.128871] SyS_write+0x49/0xa0 [ 164.129206] entry_SYSCALL_64_fastpath+0x1f/0xbe [ 164.129656] -> #1 (sparse_irq_lock){+.+.}: [ 164.130145] __lock_acquire+0x116a/0x11c0 [ 164.130546] lock_acquire+0xbd/0x200 [ 164.130913] __mutex_lock+0x70/0x8e0 [ 164.131276] mutex_lock_nested+0x1b/0x20 [ 164.131664] irq_lock_sparse+0x17/0x20 [ 164.132039] irq_affinity_online_cpu+0x18/0xd0 [ 164.132463] cpuhp_invoke_callback+0x1f6/0x830 [ 164.132928] cpuhp_up_callbacks+0x37/0xb0 [ 164.133487] cpuhp_thread_fun+0x14f/0x160 [ 164.133885] smpboot_thread_fn+0x11a/0x1e0 [ 164.134333] kthread+0x152/0x190 [ 164.134652] ret_from_fork+0x2a/0x40 [ 164.135010] -> #0 (cpuhp_state){+.+.}: [ 164.135454] cpuhp_thread_fun+0x54/0x160 [ 164.135834] smpboot_thread_fn+0x11a/0x1e0 [ 164.136235] kthread+0x152/0x190 [ 164.136558] ret_from_fork+0x2a/0x40 [ 164.136920] 0xffffffffffffffff [ 164.137239] other info that might help us debug this: [ 164.137927] Chain exists of: cpuhp_state --> sparse_irq_lock --> (complete)&per_cpu_ptr(&cpuhp_state, i)->done#2 [ 164.139136] Possible unsafe locking scenario by crosslock: [ 164.139898] CPU0 CPU1 [ 164.140298] ---- ---- [ 164.140689] lock(sparse_irq_lock); [ 164.141015] lock((complete)&per_cpu_ptr(&cpuhp_state, i)->done#2); [ 164.141565] lock(cpuhp_state); [ 164.142077] unlock((complete)&per_cpu_ptr(&cpuhp_state, i)->done#2); [ 164.142859] *** DEADLOCK *** [ 164.143382] 1 lock held by cpuhp/0/12: [ 164.143794] #0: (&x->wait#2){-...}, at: [] complete+0x1d/0x60 [ 164.144461] stack backtrace: [ 164.144837] CPU: 0 PID: 12 Comm: cpuhp/0 Tainted: G E 4.13.0.g90abd70-tip-lockdep #4 [ 164.145594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.0.0-prebuilt.qemu-project.org 04/01/2014 [ 164.146485] Call Trace: [ 164.146707] dump_stack+0x85/0xc5 [ 164.147004] print_circular_bug+0x2e9/0x2f7 [ 164.147369] check_prev_add+0x666/0x700 [ 164.147706] ? print_bfs_bug+0x40/0x40 [ 164.148045] ? trace_hardirqs_on+0xd/0x10 [ 164.148389] lock_commit_crosslock+0x410/0x5a0 [ 164.148780] complete+0x29/0x60 [ 164.149064] cpuhp_thread_fun+0xa1/0x160 [ 164.149433] ? cpuhp_thread_fun+0x2a/0x160 [ 164.149814] smpboot_thread_fn+0x11a/0x1e0 [ 164.150193] kthread+0x152/0x190 [ 164.150482] ? sort_range+0x30/0x30 [ 164.150781] ? kthread_create_on_node+0x40/0x40 [ 164.151184] ret_from_fork+0x2a/0x40