From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754865AbcG0JvA (ORCPT ); Wed, 27 Jul 2016 05:51:00 -0400 Received: from hqemgate15.nvidia.com ([216.228.121.64]:11663 "EHLO hqemgate15.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753285AbcG0Ju5 (ORCPT ); Wed, 27 Jul 2016 05:50:57 -0400 X-PGP-Universal: processed; by hqnvupgp08.nvidia.com on Wed, 27 Jul 2016 02:48:14 -0700 Subject: Re: [PATCH] timers/core: Correct callback order during CPU hot plug. To: Richard Cochran , References: <20160713153337.566790058@linutronix.de> <1469610498-25914-1-git-send-email-rcochran@linutronix.de> CC: , John Stultz , Linus Torvalds , Oleg Nesterov , "Peter Zijlstra" , Rasmus Villemoes , Sebastian Andrzej Siewior , Thomas Gleixner , "linux-tegra@vger.kernel.org" From: Jon Hunter Message-ID: <25aca1c4-5bc0-838c-ac69-45ec97031da2@nvidia.com> Date: Wed, 27 Jul 2016 10:50:42 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <1469610498-25914-1-git-send-email-rcochran@linutronix.de> X-Originating-IP: [10.26.11.53] X-ClientProxiedBy: UKMAIL101.nvidia.com (10.26.138.13) To UKMAIL101.nvidia.com (10.26.138.13) Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Richard, On 27/07/16 10:08, Richard Cochran wrote: > On the tear-down path, the dead CPU callback for the timers was > misplaced within the 'cpuhp_state' enumeration. There is a hidden > dependency between the timers and block multiqueue. The timers > callback must happen before the block multiqueue callback otherwise a > RCU stall occurs. > > This patch fixes the order within the hot plug state machine. > > Signed-off-by: Richard Cochran > Fixes: 24f73b99716a ("timers/core: Convert to hotplug state machine") > > Cc: John Stultz > Cc: Jon Hunter > Cc: Linus Torvalds > Cc: Oleg Nesterov > Cc: Peter Zijlstra > Cc: Rasmus Villemoes > Cc: Sebastian Andrzej Siewior > Cc: Thomas Gleixner > Cc: rt@linutronix.de > --- > include/linux/cpuhotplug.h | 2 +- > kernel/cpu.c | 15 ++++++++++----- > 2 files changed, 11 insertions(+), 6 deletions(-) > > diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h > index 6d405db..242bf53 100644 > --- a/include/linux/cpuhotplug.h > +++ b/include/linux/cpuhotplug.h > @@ -20,9 +20,9 @@ enum cpuhp_state { > CPUHP_PROFILE_PREPARE, > CPUHP_X2APIC_PREPARE, > CPUHP_SMPCFD_PREPARE, > - CPUHP_TIMERS_DEAD, > CPUHP_RCUTREE_PREP, > CPUHP_NOTIFY_PREPARE, > + CPUHP_TIMERS_DEAD, > CPUHP_BRINGUP_CPU, > CPUHP_AP_IDLE_DEAD, > CPUHP_AP_OFFLINE, > diff --git a/kernel/cpu.c b/kernel/cpu.c > index 67f4943..ec12b72 100644 > --- a/kernel/cpu.c > +++ b/kernel/cpu.c > @@ -1208,11 +1208,6 @@ static struct cpuhp_step cpuhp_bp_states[] = { > .startup = smpcfd_prepare_cpu, > .teardown = smpcfd_dead_cpu, > }, > - [CPUHP_TIMERS_DEAD] = { > - .name = "timers dead", > - .startup = NULL, > - .teardown = timers_dead_cpu, > - }, > [CPUHP_RCUTREE_PREP] = { > .name = "RCU-tree prepare", > .startup = rcutree_prepare_cpu, > @@ -1229,6 +1224,16 @@ static struct cpuhp_step cpuhp_bp_states[] = { > .skip_onerr = true, > .cant_stop = true, > }, > + /* > + * On the tear-down path, timers_dead_cpu() must be invoked > + * before blk_mq_queue_reinit_notify() from notify_dead(), > + * otherwise a RCU stall occurs. > + */ > + [CPUHP_TIMERS_DEAD] = { > + .name = "timers dead", > + .startup = NULL, > + .teardown = timers_dead_cpu, > + }, > /* Kicks the plugged cpu into life */ > [CPUHP_BRINGUP_CPU] = { > .name = "cpu:bringup", > This fixes the RCU stalls I was seeing during suspend on Tegra and so ... Tested-by: Jon Hunter Cheers! Jon -- nvpublic