From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752709AbbDCKuz (ORCPT ); Fri, 3 Apr 2015 06:50:55 -0400 Received: from mail-wi0-f176.google.com ([209.85.212.176]:38104 "EHLO mail-wi0-f176.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752566AbbDCKux (ORCPT ); Fri, 3 Apr 2015 06:50:53 -0400 Date: Fri, 3 Apr 2015 12:50:48 +0200 From: Ingo Molnar To: Preeti U Murthy Cc: nico@linaro.org, tglx@linutronix.de, hpa@zytor.com, linux-kernel@vger.kernel.org, linux-tip-commits@vger.kernel.org Subject: Re: [tip:timers/core] clockevents: Fix cpu_down() race for hrtimer based broadcasting Message-ID: <20150403105048.GA27855@gmail.com> References: <20150330092410.24979.59887.stgit@preeti.in.ibm.com> <551E6DBD.6070401@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <551E6DBD.6070401@linux.vnet.ibm.com> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org * Preeti U Murthy wrote: > On 04/02/2015 08:00 PM, tip-bot for Preeti U Murthy wrote: > > Commit-ID: 345527b1edce8df719e0884500c76832a18211c3 > > Gitweb: http://git.kernel.org/tip/345527b1edce8df719e0884500c76832a18211c3 > > Author: Preeti U Murthy > > AuthorDate: Mon, 30 Mar 2015 14:59:19 +0530 > > Committer: Ingo Molnar > > CommitDate: Thu, 2 Apr 2015 14:25:39 +0200 > > > > clockevents: Fix cpu_down() race for hrtimer based broadcasting > > > > It was found when doing a hotplug stress test on POWER, that the > > machine either hit softlockups or rcu_sched stall warnings. The > > issue was traced to commit: > > > > 7cba160ad789 ("powernv/cpuidle: Redesign idle states management") > > > > which exposed the cpu_down() race with hrtimer based broadcast mode: > > > > 5d1638acb9f6 ("tick: Introduce hrtimer based broadcast") > > > > The race is the following: > > > > Assume CPU1 is the CPU which holds the hrtimer broadcasting duty > > before it is taken down. > > > > CPU0 CPU1 > > > > cpu_down() take_cpu_down() > > disable_interrupts() > > > > cpu_die() > > > > while (CPU1 != CPU_DEAD) { > > msleep(100); > > switch_to_idle(); > > stop_cpu_timer(); > > schedule_broadcast(); > > } > > > > tick_cleanup_cpu_dead() > > take_over_broadcast() > > > > So after CPU1 disabled interrupts it cannot handle the broadcast > > hrtimer anymore, so CPU0 will be stuck forever. > > > > Fix this by explicitly taking over broadcast duty before cpu_die(). > > > > This is a temporary workaround. What we really want is a callback > > in the clockevent device which allows us to do that from the dying > > CPU by pushing the hrtimer onto a different cpu. That might involve > > an IPI and is definitely more complex than this immediate fix. > > > > Changelog was picked up from: > > > > https://lkml.org/lkml/2015/2/16/213 > > > > Suggested-by: Thomas Gleixner > > Tested-by: Nicolas Pitre > > Signed-off-by: Preeti U. Murthy > > Cc: linuxppc-dev@lists.ozlabs.org > > Cc: mpe@ellerman.id.au > > Cc: nicolas.pitre@linaro.org > > Cc: peterz@infradead.org > > Cc: rjw@rjwysocki.net > > Fixes: http://linuxppc.10917.n7.nabble.com/offlining-cpus-breakage-td88619.html > > Link: http://lkml.kernel.org/r/20150330092410.24979.59887.stgit@preeti.in.ibm.com > > [ Merged it to the latest timer tree, renamed the callback, tidied up the changelog. ] > > Signed-off-by: Ingo Molnar > > Can this be marked for stable too please? It can be forwarded to -stable once it has gone upstream in the merge window and has gone through some testing upstream as well. The breakage itself was introduced in the v3.19 merge window, so it's an older regression - and the fix is not simple either. Thanks, Ingo