* [PATCH] sched/core: Fix kick offline cpu to do nohz idle load balance
@ 2016-10-10 4:10 Wanpeng Li
2016-10-10 8:34 ` Wanpeng Li
0 siblings, 1 reply; 3+ messages in thread
From: Wanpeng Li @ 2016-10-10 4:10 UTC (permalink / raw)
To: linux-kernel
Cc: Wanpeng Li, Ingo Molnar, Mike Galbraith, Peter Zijlstra, Thomas Gleixner
From: Wanpeng Li <wanpeng.li@hotmail.com>
WARNING: CPU: 0 PID: 3404 at arch/x86/kernel/smp.c:125 native_smp_send_reschedule+0x3f/0x50
CPU: 0 PID: 3404 Comm: qemu-system-x86 Not tainted 4.8.0+ #21
Call Trace:
__warn+0xd1/0xf0
warn_slowpath_null+0x1d/0x20
native_smp_send_reschedule+0x3f/0x50
trigger_load_balance+0x29c/0x4a0
? trigger_load_balance+0x72/0x4a0
scheduler_tick+0x9f/0xd0
? tick_sched_do_timer+0x50/0x50
update_process_times+0x47/0x60
tick_sched_handle.isra.24+0x25/0x60
tick_sched_timer+0x3d/0x70
__hrtimer_run_queues+0xf4/0x510
hrtimer_interrupt+0xb7/0x1d0
local_apic_timer_interrupt+0x35/0x60
smp_apic_timer_interrupt+0x3d/0x50
apic_timer_interrupt+0x96/0xa0
If there is a need to kick the idle load balancer, an ILB will be selected
to perform nohz idle load balance, however, if the selected ILB is in the
process of offline, smp_sched_reschedule() which generates a sched IPI will
splat as above.
CPU0 CPU1
find_new_ilb()
set_rq_offline()
smp_sched_reschedule() Oops
nohz_balance_exit_idle()
This patch fix it by exiting nohz idle balance before set cpu offline.
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
---
kernel/sched/core.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 94732d1..7c83f99 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7412,6 +7412,7 @@ int sched_cpu_dying(unsigned int cpu)
/* Handle pending wakeups and then migrate everything off */
sched_ttwu_pending();
+ nohz_balance_exit_idle(cpu);
raw_spin_lock_irqsave(&rq->lock, flags);
if (rq->rd) {
BUG_ON(!cpumask_test_cpu(cpu, rq->rd->span));
@@ -7422,7 +7423,6 @@ int sched_cpu_dying(unsigned int cpu)
raw_spin_unlock_irqrestore(&rq->lock, flags);
calc_load_migrate(rq);
update_max_interval();
- nohz_balance_exit_idle(cpu);
hrtick_clear(rq);
return 0;
}
--
1.9.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH] sched/core: Fix kick offline cpu to do nohz idle load balance
2016-10-10 4:10 [PATCH] sched/core: Fix kick offline cpu to do nohz idle load balance Wanpeng Li
@ 2016-10-10 8:34 ` Wanpeng Li
2016-10-10 12:02 ` Peter Zijlstra
0 siblings, 1 reply; 3+ messages in thread
From: Wanpeng Li @ 2016-10-10 8:34 UTC (permalink / raw)
To: linux-kernel
Cc: Wanpeng Li, Ingo Molnar, Mike Galbraith, Peter Zijlstra, Thomas Gleixner
2016-10-10 12:10 GMT+08:00 Wanpeng Li <kernellwp@gmail.com>:
> From: Wanpeng Li <wanpeng.li@hotmail.com>
>
> WARNING: CPU: 0 PID: 3404 at arch/x86/kernel/smp.c:125 native_smp_send_reschedule+0x3f/0x50
> CPU: 0 PID: 3404 Comm: qemu-system-x86 Not tainted 4.8.0+ #21
> Call Trace:
> __warn+0xd1/0xf0
> warn_slowpath_null+0x1d/0x20
> native_smp_send_reschedule+0x3f/0x50
> trigger_load_balance+0x29c/0x4a0
> ? trigger_load_balance+0x72/0x4a0
> scheduler_tick+0x9f/0xd0
> ? tick_sched_do_timer+0x50/0x50
> update_process_times+0x47/0x60
> tick_sched_handle.isra.24+0x25/0x60
> tick_sched_timer+0x3d/0x70
> __hrtimer_run_queues+0xf4/0x510
> hrtimer_interrupt+0xb7/0x1d0
> local_apic_timer_interrupt+0x35/0x60
> smp_apic_timer_interrupt+0x3d/0x50
> apic_timer_interrupt+0x96/0xa0
>
> If there is a need to kick the idle load balancer, an ILB will be selected
> to perform nohz idle load balance, however, if the selected ILB is in the
> process of offline, smp_sched_reschedule() which generates a sched IPI will
> splat as above.
>
> CPU0 CPU1
>
> find_new_ilb()
> set_rq_offline()
> smp_sched_reschedule() Oops
> nohz_balance_exit_idle()
>
> This patch fix it by exiting nohz idle balance before set cpu offline.
CPU 0 CPU1
find_new_ilb()
nohz_balance_exit_idle()
set_rq_offline()
smp_sched_reschedule()
It seems that the patch still can't avoid this race, so any proposal
is a great appreciated. :)
Regards,
Wanpeng Li
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH] sched/core: Fix kick offline cpu to do nohz idle load balance
2016-10-10 8:34 ` Wanpeng Li
@ 2016-10-10 12:02 ` Peter Zijlstra
0 siblings, 0 replies; 3+ messages in thread
From: Peter Zijlstra @ 2016-10-10 12:02 UTC (permalink / raw)
To: Wanpeng Li
Cc: linux-kernel, Wanpeng Li, Ingo Molnar, Mike Galbraith, Thomas Gleixner
On Mon, Oct 10, 2016 at 04:34:48PM +0800, Wanpeng Li wrote:
> > If there is a need to kick the idle load balancer, an ILB will be selected
> > to perform nohz idle load balance, however, if the selected ILB is in the
> > process of offline, smp_sched_reschedule() which generates a sched IPI will
> > splat as above.
> >
> > CPU0 CPU1
> >
> > find_new_ilb()
> > set_rq_offline()
> > smp_sched_reschedule() Oops
> > nohz_balance_exit_idle()
> >
> > This patch fix it by exiting nohz idle balance before set cpu offline.
>
> CPU 0 CPU1
>
> find_new_ilb()
> nohz_balance_exit_idle()
> set_rq_offline()
> smp_sched_reschedule()
>
> It seems that the patch still can't avoid this race, so any proposal
> is a great appreciated. :)
Not sure how this can happen, scheduler_tick() -> trigger_load_balance()
-> nohz_balancer_kick() is called with IRQs disabled, this too implies a
RCU-sched read side section.
And hotplug explicitly includes a rcu_sync_sched().
It would be find_new_ilb() is 'broken' in that it considers !active
CPUs. That's not immediately obvious.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2016-10-10 12:20 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-10-10 4:10 [PATCH] sched/core: Fix kick offline cpu to do nohz idle load balance Wanpeng Li
2016-10-10 8:34 ` Wanpeng Li
2016-10-10 12:02 ` Peter Zijlstra
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.