LKML Archive on lore.kernel.org
 help / color / Atom feed
From: Thomas Gleixner <tglx@linutronix.de>
To: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: LKML <linux-kernel@vger.kernel.org>,
	Anna-Maria Gleixner <anna-maria@linutronix.de>,
	Sebastian Siewior <bigeasy@linutronix.de>,
	Peter Zijlstra <peterz@infradead.org>,
	Frederic Weisbecker <fweisbec@gmail.com>,
	Ingo Molnar <mingo@kernel.org>
Subject: Re: [patch 0/4] timer/nohz: Fix timer/nohz woes
Date: Wed, 27 Dec 2017 21:55:58 +0100 (CET)
Message-ID: <alpine.DEB.2.20.1712272152200.2431@nanos> (raw)
In-Reply-To: <20171224012120.GA4113@linux.vnet.ibm.com>

On Sat, 23 Dec 2017, Paul E. McKenney wrote:
> diff --git a/kernel/time/timer.c b/kernel/time/timer.c
> index ffebcf878fba..94cce780c574 100644
> --- a/kernel/time/timer.c
> +++ b/kernel/time/timer.c
> @@ -1875,6 +1875,7 @@ int timers_dead_cpu(unsigned int cpu)
>  
>  		BUG_ON(old_base->running_timer);
>  
> +		old_base->must_forward_clk = true;
>  		for (i = 0; i < WHEEL_SIZE; i++)
>  			migrate_timer_list(new_base, old_base->vectors + i);
>  

That's daft because base->next_expiry is also stale and that might cause
interesting issues in forward_timer_base(). But you are right, leaving the
base stale on cpu hotplug is not really a brilliant thing to do.

Instead of that workaround, I think we want a proper callback in the
prepare stage which brings the bases of the to be plugged CPU up to date.

Thanks,

	tglx

8<---------------------
Subject: timers: Reinitialize per cpu bases on hotplug
From: Thomas Gleixner <tglx@linutronix.de>
Date: Wed, 27 Dec 2017 21:37:25 +0100

The timer wheel bases are not (re)initialized on CPU hotplug. That leaves
them with a potentially stale clk and next_expiry valuem, which can cause
trouble then the CPU is plugged.

Add a prepare callback which forwards the clock, sets next_expiry to far in
the future and reset the control flags to a known state.

Set base->must_forward_clk so the first timer which is queued will try to
forward the clock to current jiffies.

Fixes: 500462a9de65 ("timers: Switch to a non-cascading wheel")
Reported-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: stable@vger.kernel.org
---
 include/linux/cpuhotplug.h |    2 +-
 include/linux/timer.h      |    4 +++-
 kernel/cpu.c               |    4 ++--
 kernel/time/timer.c        |   14 ++++++++++++++
 4 files changed, 20 insertions(+), 4 deletions(-)

--- a/include/linux/cpuhotplug.h
+++ b/include/linux/cpuhotplug.h
@@ -86,7 +86,7 @@ enum cpuhp_state {
 	CPUHP_MM_ZSWP_POOL_PREPARE,
 	CPUHP_KVM_PPC_BOOK3S_PREPARE,
 	CPUHP_ZCOMP_PREPARE,
-	CPUHP_TIMERS_DEAD,
+	CPUHP_TIMERS_PREPARE,
 	CPUHP_MIPS_SOC_PREPARE,
 	CPUHP_BP_PREPARE_DYN,
 	CPUHP_BP_PREPARE_DYN_END		= CPUHP_BP_PREPARE_DYN + 20,
--- a/include/linux/timer.h
+++ b/include/linux/timer.h
@@ -207,9 +207,11 @@ unsigned long round_jiffies_up(unsigned
 unsigned long round_jiffies_up_relative(unsigned long j);
 
 #ifdef CONFIG_HOTPLUG_CPU
+int timers_prepare_cpu(unsigned int cpu);
 int timers_dead_cpu(unsigned int cpu);
 #else
-#define timers_dead_cpu NULL
+#define timers_prepare_cpu	NULL
+#define timers_dead_cpu		NULL
 #endif
 
 #endif
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1277,9 +1277,9 @@ static struct cpuhp_step cpuhp_bp_states
 	 * before blk_mq_queue_reinit_notify() from notify_dead(),
 	 * otherwise a RCU stall occurs.
 	 */
-	[CPUHP_TIMERS_DEAD] = {
+	[CPUHP_TIMERS_PREPARE] = {
 		.name			= "timers:dead",
-		.startup.single		= NULL,
+		.startup.single		= timers_prepare_cpu,
 		.teardown.single	= timers_dead_cpu,
 	},
 	/* Kicks the plugged cpu into life */
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1853,6 +1853,20 @@ static void migrate_timer_list(struct ti
 	}
 }
 
+int timers_prepare_cpu(unsigned int cpu)
+{
+	struct timer_base *base;
+	int b;
+
+	for (b = 0; b < NR_BASES; b++) {
+		base = per_cpu_ptr(&timer_bases[b], cpu);
+		base->clk = jiffies;
+		base->next_expiry = base->clk + NEXT_TIMER_MAX_DELTA;
+		base->is_idle = false;
+		base->must_forward_clk = true;
+	}
+}
+
 int timers_dead_cpu(unsigned int cpu)
 {
 	struct timer_base *old_base;

  parent reply index

Thread overview: 23+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-12-22 14:51 Thomas Gleixner
2017-12-22 14:51 ` [patch 1/4] timer: Use deferrable base independent of base::nohz_active Thomas Gleixner
2017-12-25 16:24   ` Frederic Weisbecker
2017-12-29 22:45   ` [tip:timers/urgent] timers: " tip-bot for Anna-Maria Gleixner
2017-12-22 14:51 ` [patch 2/4] nohz: Prevent erroneous tick stop invocations Thomas Gleixner
2017-12-26 15:17   ` Frederic Weisbecker
2017-12-27 18:22     ` Thomas Gleixner
2017-12-27 18:24       ` Thomas Gleixner
2017-12-27 20:58         ` Thomas Gleixner
2017-12-29 16:12           ` Frederic Weisbecker
2017-12-29 22:46           ` [tip:timers/urgent] nohz: Prevent a timer interrupt storm in tick_nohz_stop_sched_tick() tip-bot for Thomas Gleixner
2017-12-22 14:51 ` [patch 3/4] timer: Invoke timer_start_debug() where it makes sense Thomas Gleixner
2017-12-29 22:47   ` [tip:timers/urgent] timers: " tip-bot for Thomas Gleixner
2017-12-22 14:51 ` [patch 4/4] timerqueue: Document return values of timerqueue_add/del() Thomas Gleixner
2017-12-29 22:47   ` [tip:timers/urgent] " tip-bot for Thomas Gleixner
2017-12-22 17:09 ` [patch 0/4] timer/nohz: Fix timer/nohz woes Paul E. McKenney
2017-12-24  1:21   ` Paul E. McKenney
2017-12-24  1:29     ` Paul E. McKenney
2018-01-05 19:41       ` Paul E. McKenney
2018-01-06 21:18         ` Thomas Gleixner
2018-01-06 23:21           ` Paul E. McKenney
2017-12-27 20:55     ` Thomas Gleixner [this message]
2017-12-29 22:46       ` [tip:timers/urgent] timers: Reinitialize per cpu bases on hotplug tip-bot for Thomas Gleixner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=alpine.DEB.2.20.1712272152200.2431@nanos \
    --to=tglx@linutronix.de \
    --cc=anna-maria@linutronix.de \
    --cc=bigeasy@linutronix.de \
    --cc=fweisbec@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=paulmck@linux.vnet.ibm.com \
    --cc=peterz@infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git
	git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git