[PATCH] tick/nohz: Fix the intention to stop full tick for offline CPU

[PATCH] tick/nohz: Fix the intention to stop full tick for offline CPU

[Wanpeng Li]

From: Wanpeng Li <wanpeng.li@hotmail.com>

Interrupt exit is the place to stop the tick: it happens after all 
events happening before and during the irq which are liable to update 
the dependency on the tick occurred. However, tick_nohz_stop_sched_tick() 
try to stop the full tick instead of bailing out in advance. 

As a result: 

WARNING: CPU: 3 PID: 31 at kernel/irq_work.c:124 irq_work_needs_cpu+0x86/0x90
CPU: 3 PID: 31 Comm: migration/3 Not tainted 4.8.0-rc5+ #46
Call Trace:
 <IRQ>  [<ffffffffb94477c9>] dump_stack+0x99/0xd0
 [<ffffffffb908d581>] __warn+0xd1/0xf0
 [<ffffffffb908d66d>] warn_slowpath_null+0x1d/0x20
 [<ffffffffb91a7446>] irq_work_needs_cpu+0x86/0x90
 [<ffffffffb912cbed>] tick_nohz_stop_sched_tick+0x2ad/0x430
 [<ffffffffb912d75e>] ? tick_nohz_irq_exit+0x10e/0x150
 [<ffffffffb906de05>] ? kvm_clock_read+0x25/0x30
 [<ffffffffb912d75e>] tick_nohz_irq_exit+0x10e/0x150
 [<ffffffffb9094a31>] irq_exit+0x81/0xf0
 [<ffffffffb98ca4cc>] do_IRQ+0x6c/0x120
 [<ffffffffb98c83d6>] common_interrupt+0x96/0x96
 <EOI>  [<ffffffffb9153fb2>] ? multi_cpu_stop+0xa2/0x160
 [<ffffffffb9153f10>] ? cpu_stop_park+0x40/0x40
 [<ffffffffb9153e44>] cpu_stopper_thread+0x74/0x100
 [<ffffffffb90b7e77>] smpboot_thread_fn+0x117/0x1d0
 [<ffffffffb90b7d60>] ? sort_range+0x30/0x30
 [<ffffffffb90b4111>] kthread+0x101/0x120
 [<ffffffffb90dea6d>] ? complete+0x1d/0x50
 [<ffffffffb98c7c8f>] ret_from_fork+0x1f/0x40
 [<ffffffffb90b4010>] ? kthread_create_on_node+0x250/0x250

Commit f7ea0fd639c2c4 (tick: Don't invoke tick_nohz_stop_sched_tick() if 
the cpu is offline) fixes stop idle tick for offline cpu, this patch adds 
the similar method to fix the intention to stop full tick for offline cpu.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
---
 kernel/time/tick-sched.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 2ec7c00..3bcb61b 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -186,10 +186,13 @@ static bool check_tick_dependency(atomic_t *dep)
 	return false;
 }
 
-static bool can_stop_full_tick(struct tick_sched *ts)
+static bool can_stop_full_tick(int cpu, struct tick_sched *ts)
 {
 	WARN_ON_ONCE(!irqs_disabled());
 
+	if (unlikely(!cpu_online(cpu)))
+		return false;
+
 	if (check_tick_dependency(&tick_dep_mask))
 		return false;
 
@@ -843,7 +846,7 @@ static void tick_nohz_full_update_tick(struct tick_sched *ts)
 	if (!ts->tick_stopped && ts->nohz_mode == NOHZ_MODE_INACTIVE)
 		return;
 
-	if (can_stop_full_tick(ts))
+	if (can_stop_full_tick(cpu, ts))
 		tick_nohz_stop_sched_tick(ts, ktime_get(), cpu);
 	else if (ts->tick_stopped)
 		tick_nohz_restart_sched_tick(ts, ktime_get());
-- 
1.9.1

Re: [PATCH] tick/nohz: Fix the intention to stop full tick for offline CPU

[Wanpeng Li]

Ping Thomas, :)
2016-09-07 18:51 GMT+08:00 Wanpeng Li <kernellwp@gmail.com>:
> From: Wanpeng Li <wanpeng.li@hotmail.com>
>
> Interrupt exit is the place to stop the tick: it happens after all
> events happening before and during the irq which are liable to update
> the dependency on the tick occurred. However, tick_nohz_stop_sched_tick()
> try to stop the full tick instead of bailing out in advance.
>
> As a result:
>
> WARNING: CPU: 3 PID: 31 at kernel/irq_work.c:124 irq_work_needs_cpu+0x86/0x90
> CPU: 3 PID: 31 Comm: migration/3 Not tainted 4.8.0-rc5+ #46
> Call Trace:
>  <IRQ>  [<ffffffffb94477c9>] dump_stack+0x99/0xd0
>  [<ffffffffb908d581>] __warn+0xd1/0xf0
>  [<ffffffffb908d66d>] warn_slowpath_null+0x1d/0x20
>  [<ffffffffb91a7446>] irq_work_needs_cpu+0x86/0x90
>  [<ffffffffb912cbed>] tick_nohz_stop_sched_tick+0x2ad/0x430
>  [<ffffffffb912d75e>] ? tick_nohz_irq_exit+0x10e/0x150
>  [<ffffffffb906de05>] ? kvm_clock_read+0x25/0x30
>  [<ffffffffb912d75e>] tick_nohz_irq_exit+0x10e/0x150
>  [<ffffffffb9094a31>] irq_exit+0x81/0xf0
>  [<ffffffffb98ca4cc>] do_IRQ+0x6c/0x120
>  [<ffffffffb98c83d6>] common_interrupt+0x96/0x96
>  <EOI>  [<ffffffffb9153fb2>] ? multi_cpu_stop+0xa2/0x160
>  [<ffffffffb9153f10>] ? cpu_stop_park+0x40/0x40
>  [<ffffffffb9153e44>] cpu_stopper_thread+0x74/0x100
>  [<ffffffffb90b7e77>] smpboot_thread_fn+0x117/0x1d0
>  [<ffffffffb90b7d60>] ? sort_range+0x30/0x30
>  [<ffffffffb90b4111>] kthread+0x101/0x120
>  [<ffffffffb90dea6d>] ? complete+0x1d/0x50
>  [<ffffffffb98c7c8f>] ret_from_fork+0x1f/0x40
>  [<ffffffffb90b4010>] ? kthread_create_on_node+0x250/0x250
>
> Commit f7ea0fd639c2c4 (tick: Don't invoke tick_nohz_stop_sched_tick() if
> the cpu is offline) fixes stop idle tick for offline cpu, this patch adds
> the similar method to fix the intention to stop full tick for offline cpu.
>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Frederic Weisbecker <fweisbec@gmail.com>
> Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
> ---
>  kernel/time/tick-sched.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
> index 2ec7c00..3bcb61b 100644
> --- a/kernel/time/tick-sched.c
> +++ b/kernel/time/tick-sched.c
> @@ -186,10 +186,13 @@ static bool check_tick_dependency(atomic_t *dep)
>         return false;
>  }
>
> -static bool can_stop_full_tick(struct tick_sched *ts)
> +static bool can_stop_full_tick(int cpu, struct tick_sched *ts)
>  {
>         WARN_ON_ONCE(!irqs_disabled());
>
> +       if (unlikely(!cpu_online(cpu)))
> +               return false;
> +
>         if (check_tick_dependency(&tick_dep_mask))
>                 return false;
>
> @@ -843,7 +846,7 @@ static void tick_nohz_full_update_tick(struct tick_sched *ts)
>         if (!ts->tick_stopped && ts->nohz_mode == NOHZ_MODE_INACTIVE)
>                 return;
>
> -       if (can_stop_full_tick(ts))
> +       if (can_stop_full_tick(cpu, ts))
>                 tick_nohz_stop_sched_tick(ts, ktime_get(), cpu);
>         else if (ts->tick_stopped)
>                 tick_nohz_restart_sched_tick(ts, ktime_get());
> --
> 1.9.1
>



-- 
Regards,
Wanpeng Li

[tip:timers/core] tick/nohz: Prevent stopping the tick on an offline CPU

[tip-bot for Wanpeng Li]

Commit-ID:  57ccdf449f962ab5fc8cbf26479402f13bdb8be7
Gitweb:     http://git.kernel.org/tip/57ccdf449f962ab5fc8cbf26479402f13bdb8be7
Author:     Wanpeng Li <wanpeng.li@hotmail.com>
AuthorDate: Wed, 7 Sep 2016 18:51:13 +0800
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Tue, 13 Sep 2016 17:53:52 +0200

tick/nohz: Prevent stopping the tick on an offline CPU

can_stop_full_tick() has no check for offline cpus. So it allows to stop
the tick on an offline cpu from the interrupt return path, which is wrong
and subsequently makes irq_work_needs_cpu() warn about being called for an
offline cpu.

Commit f7ea0fd639c2c4 ("tick: Don't invoke tick_nohz_stop_sched_tick() if
the cpu is offline") added prevention for can_stop_idle_tick(), but forgot
to do the same in can_stop_full_tick(). Add it.

[ tglx: Massaged changelog ]

Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Link: http://lkml.kernel.org/r/1473245473-4463-1-git-send-email-wanpeng.li@hotmail.com
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

---
 kernel/time/tick-sched.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 2ec7c00..3bcb61b 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -186,10 +186,13 @@ static bool check_tick_dependency(atomic_t *dep)
 	return false;
 }
 
-static bool can_stop_full_tick(struct tick_sched *ts)
+static bool can_stop_full_tick(int cpu, struct tick_sched *ts)
 {
 	WARN_ON_ONCE(!irqs_disabled());
 
+	if (unlikely(!cpu_online(cpu)))
+		return false;
+
 	if (check_tick_dependency(&tick_dep_mask))
 		return false;
 
@@ -843,7 +846,7 @@ static void tick_nohz_full_update_tick(struct tick_sched *ts)
 	if (!ts->tick_stopped && ts->nohz_mode == NOHZ_MODE_INACTIVE)
 		return;
 
-	if (can_stop_full_tick(ts))
+	if (can_stop_full_tick(cpu, ts))
 		tick_nohz_stop_sched_tick(ts, ktime_get(), cpu);
 	else if (ts->tick_stopped)
 		tick_nohz_restart_sched_tick(ts, ktime_get());