[PATCH] x86: Don't send RESCHEDULE_VECTOR to offlined cpus

[PATCH] x86: Don't send RESCHEDULE_VECTOR to offlined cpus

[Gautham R Shenoy]

x86: Don't send RESCHEDULE_VECTOR to offlined cpus.
From: Gautham R Shenoy <ego@in.ibm.com>

In the x86 native_smp_send_reschedule_function(), don't send the IPI if the
cpu has gone offline already. Warn nevertheless!!

Signed-off-by: Gautham R Shenoy <ego@in.ibm.com>
---

 arch/x86/kernel/smp_32.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kernel/smp_32.c b/arch/x86/kernel/smp_32.c
index dc0cde9..4df9042 100644
--- a/arch/x86/kernel/smp_32.c
+++ b/arch/x86/kernel/smp_32.c
@@ -472,7 +472,10 @@ void flush_tlb_all(void)
  */
 static void native_smp_send_reschedule(int cpu)
 {
-	WARN_ON(cpu_is_offline(cpu));
+	if (unlikely(cpu_is_offline(cpu))) {
+		WARN_ON(1);
+		return;
+	}
 	send_IPI_mask(cpumask_of_cpu(cpu), RESCHEDULE_VECTOR);
 }
 
-- 
Thanks and Regards
gautham

Re: [PATCH] x86: Don't send RESCHEDULE_VECTOR to offlined cpus

[Ingo Molnar]

* Gautham R Shenoy <ego@in.ibm.com> wrote:

> x86: Don't send RESCHEDULE_VECTOR to offlined cpus.
> From: Gautham R Shenoy <ego@in.ibm.com>
> 
> In the x86 native_smp_send_reschedule_function(), don't send the IPI 
> if the cpu has gone offline already. Warn nevertheless!!

have you seen this happen?

	Ingo

Re: [PATCH] x86: Don't send RESCHEDULE_VECTOR to offlined cpus

[Gautham R Shenoy]

On Mon, Mar 10, 2008 at 01:24:18PM +0100, Ingo Molnar wrote:
> 
> * Gautham R Shenoy <ego@in.ibm.com> wrote:
> 
> > x86: Don't send RESCHEDULE_VECTOR to offlined cpus.
> > From: Gautham R Shenoy <ego@in.ibm.com>
> > 
> > In the x86 native_smp_send_reschedule_function(), don't send the IPI 
> > if the cpu has gone offline already. Warn nevertheless!!
> 
> have you seen this happen?

Yup, this afternoon while running
cpu-hotplug stress tests (http://lkml.org/lkml/2008/3/3/11)
in parallel with kern-bench, I got this on the serial console.

------------[ cut here ]------------
WARNING: at arch/x86/kernel/smp_32.c:475 native_smp_send_reschedule+0x22/0x3f()
Modules linked in: dock
Pid: 4722, comm: kstopmachine Not tainted 2.6.25-rc3 #46
 [<c011f424>] warn_on_slowpath+0x41/0x51
 [<c013ac74>] ? __lock_acquire+0xaae/0xaf6
 [<c011a446>] ? resched_cpu+0x2c/0x6f
 [<c0111691>] native_smp_send_reschedule+0x22/0x3f
 [<c01186c7>] __resched_task+0x5f/0x63
 [<c011a479>] resched_cpu+0x5f/0x6f
 [<c011bb6d>] scheduler_tick+0x214/0x28f
 [<c012737b>] update_process_times+0x3d/0x49
 [<c013751d>] tick_sched_timer+0x6e/0xa6
 [<c01374af>] ? tick_sched_timer+0x0/0xa6
 [<c0131e8c>] __run_hrtimer+0x39/0x70
 [<c0132668>] hrtimer_interrupt+0xeb/0x154
 [<c01127bc>] smp_apic_timer_interrupt+0x6c/0x80
 [<c0143670>] ? stopmachine+0x0/0x98
 [<c0105553>] apic_timer_interrupt+0x33/0x38
 [<c0143670>] ? stopmachine+0x0/0x98
 [<c013007b>] ? sample_to_timespec+0x16/0x35
 [<c0143701>] ? stopmachine+0x91/0x98
 [<c01056eb>] kernel_thread_helper+0x7/0x10
 =======================
---[ end trace 1016b68a0b60b50b ]---
------------[ cut here ]------------

> 
> 	Ingo

-- 
Thanks and Regards
gautham

Re: [PATCH] x86: Don't send RESCHEDULE_VECTOR to offlined cpus

[Roel Kluin]

Gautham R Shenoy wrote:
> On Mon, Mar 10, 2008 at 01:24:18PM +0100, Ingo Molnar wrote:
>> * Gautham R Shenoy <ego@in.ibm.com> wrote:
>>
>>> x86: Don't send RESCHEDULE_VECTOR to offlined cpus.
>>> From: Gautham R Shenoy <ego@in.ibm.com>
>>>
>>> In the x86 native_smp_send_reschedule_function(), don't send the IPI 
>>> if the cpu has gone offline already. Warn nevertheless!!
>> have you seen this happen?
> 
> Yup, this afternoon while running
> cpu-hotplug stress tests (http://lkml.org/lkml/2008/3/3/11)
> in parallel with kern-bench, I got this on the serial console.

from arch/m32r/kernel/smp.c:134

void smp_send_reschedule(int cpu_id)
{
        WARN_ON(cpu_is_offline(cpu_id));
        send_IPI_mask(cpumask_of_cpu(cpu_id), RESCHEDULE_IPI, 1);
}

should this be changed as well?

Re: [PATCH] x86: Don't send RESCHEDULE_VECTOR to offlined cpus

[Gregory Haskins]

Gautham R Shenoy wrote:
> On Mon, Mar 10, 2008 at 01:24:18PM +0100, Ingo Molnar wrote:
>> * Gautham R Shenoy <ego@in.ibm.com> wrote:
>>
>>> x86: Don't send RESCHEDULE_VECTOR to offlined cpus.
>>> From: Gautham R Shenoy <ego@in.ibm.com>
>>>
>>> In the x86 native_smp_send_reschedule_function(), don't send the IPI 
>>> if the cpu has gone offline already. Warn nevertheless!!
>> have you seen this happen?
> 
> Yup, this afternoon while running
> cpu-hotplug stress tests (http://lkml.org/lkml/2008/3/3/11)
> in parallel with kern-bench, I got this on the serial console.

Note that this may be a side-effect of the bug we are chasing with the 
rd->online problem.

E.g. if the rd->online issue is causing us to route tasks to dead CPUs, 
this phenomenon would probably go away once rd->online is fixed (which I 
believe it is with the patches that were submitted).

Regards,
-Greg

Re: [PATCH] x86: Don't send RESCHEDULE_VECTOR to offlined cpus

[Ingo Molnar]

* Gregory Haskins <gregory.haskins.ml@gmail.com> wrote:

>> Yup, this afternoon while running cpu-hotplug stress tests 
>> (http://lkml.org/lkml/2008/3/3/11) in parallel with kern-bench, I got 
>> this on the serial console.
>
> Note that this may be a side-effect of the bug we are chasing with the 
> rd->online problem.
>
> E.g. if the rd->online issue is causing us to route tasks to dead 
> CPUs, this phenomenon would probably go away once rd->online is fixed 
> (which I believe it is with the patches that were submitted).

yeah, that makes sense. I've applied your patch nevertheless because it 
is the better failure mode for that code.

	Ingo