* [3.14] core onlining/hotplug regression
@ 2014-07-25 7:50 Daniel J Blueman
2014-07-25 9:05 ` Thomas Gleixner
0 siblings, 1 reply; 4+ messages in thread
From: Daniel J Blueman @ 2014-07-25 7:50 UTC (permalink / raw)
To: Oleg Nesterov, Thomas Gleixner, Peter Zijlstra, Hillf Danton
Cc: Borislav Petkov, Ingo Molnar, Igor Mammedov, Steffen Persvold, LKML
Hi Thomas et al,
On a larger x86 system with 1728 cores, 3.15(.6) asserts on
smpboot_thread_fn's td->cpu != smp_processor_id() consistently after
~1500 cores are online.
Reverting the only directly related changes I could find [1,2] doesn't
help. Debugging indicates there is a race where the created thread is
quickly migrated to core 0 when this occurs, since smp_processor_id
returns 0 in these cases. Thomas introduced a thread parked state to fix
related issues a year back. Linux 3.14(.13) boots just nice.
Full boot output is at:
https://resources.numascale.com/linux-315-thread-mig.txt
Any theories so far? I'll start bisecting when I have full access to the
system again in a week and I'll do some more debugging with intermittent
access before then.
Thanks,
Daniel
-- [1]
commit 81c98869faa5f3a9457c93efef908ef476326b31
Author: Nishanth Aravamudan <nacc@linux.vnet.ibm.com>
Date: Thu Apr 3 14:46:25 2014 -0700
kthread: ensure locality of task_struct allocations
-- [2]
commit 89f898c1e195fa6235c869bb457e500b7b3ac49d
Author: Igor Mammedov <imammedo@redhat.com>
Date: Thu Jun 5 15:42:43 2014 +0200
x86: Fix list/memory corruption on CPU hotplug
--
Daniel J Blueman
Principal Software Engineer, Numascale
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [3.14] core onlining/hotplug regression
2014-07-25 7:50 [3.14] core onlining/hotplug regression Daniel J Blueman
@ 2014-07-25 9:05 ` Thomas Gleixner
2014-07-25 9:36 ` Daniel J Blueman
2014-09-13 9:03 ` Daniel J Blueman
0 siblings, 2 replies; 4+ messages in thread
From: Thomas Gleixner @ 2014-07-25 9:05 UTC (permalink / raw)
To: Daniel J Blueman
Cc: Oleg Nesterov, Peter Zijlstra, Hillf Danton, Borislav Petkov,
Ingo Molnar, Igor Mammedov, Steffen Persvold, LKML
On Fri, 25 Jul 2014, Daniel J Blueman wrote:
> On a larger x86 system with 1728 cores, 3.15(.6) asserts on
> smpboot_thread_fn's td->cpu != smp_processor_id() consistently after ~1500
> cores are online.
>
> Reverting the only directly related changes I could find [1,2] doesn't help.
> Debugging indicates there is a race where the created thread is quickly
> migrated to core 0 when this occurs, since smp_processor_id returns 0 in these
> cases. Thomas introduced a thread parked state to fix related issues a year
> back. Linux 3.14(.13) boots just nice.
Weird. Commits [1,2] are definitely not the culprits.
> Full boot output is at:
> https://resources.numascale.com/linux-315-thread-mig.txt
Not really helpful, as we don't see what causes it. We just see the
wreckage.
> Any theories so far? I'll start bisecting when I have full access to the
> system again in a week and I'll do some more debugging with intermittent
> access before then.
One thing you could try is enabling tracing.
"ftrace=function ftrace_dump_on_oops"
It'll take a looooong time to spill out the traces, but that should
give us the root cause precisely.
Thanks,
tglx
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [3.14] core onlining/hotplug regression
2014-07-25 9:05 ` Thomas Gleixner
@ 2014-07-25 9:36 ` Daniel J Blueman
2014-09-13 9:03 ` Daniel J Blueman
1 sibling, 0 replies; 4+ messages in thread
From: Daniel J Blueman @ 2014-07-25 9:36 UTC (permalink / raw)
To: Thomas Gleixner
Cc: Oleg Nesterov, Peter Zijlstra, Hillf Danton, Borislav Petkov,
Ingo Molnar, Igor Mammedov, Steffen Persvold, LKML
On 07/25/2014 05:05 PM, Thomas Gleixner wrote:
> On Fri, 25 Jul 2014, Daniel J Blueman wrote:
>> On a larger x86 system with 1728 cores, 3.15(.6) asserts on
>> smpboot_thread_fn's td->cpu != smp_processor_id() consistently after ~1500
>> cores are online.
>>
>> Reverting the only directly related changes I could find [1,2] doesn't help.
>> Debugging indicates there is a race where the created thread is quickly
>> migrated to core 0 when this occurs, since smp_processor_id returns 0 in these
>> cases. Thomas introduced a thread parked state to fix related issues a year
>> back. Linux 3.14(.13) boots just nice.
>
> Weird. Commits [1,2] are definitely not the culprits.
>
>> Full boot output is at:
>> https://resources.numascale.com/linux-315-thread-mig.txt
>
> Not really helpful, as we don't see what causes it. We just see the
> wreckage.
>
>> Any theories so far? I'll start bisecting when I have full access to the
>> system again in a week and I'll do some more debugging with intermittent
>> access before then.
>
> One thing you could try is enabling tracing.
>
> "ftrace=function ftrace_dump_on_oops"
>
> It'll take a looooong time to spill out the traces, but that should
> give us the root cause precisely.
Good trick. I'll get this early next week and we'll see what's up.
Thanks,
Daniel
--
Daniel J Blueman
Principal Software Engineer, Numascale
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [3.14] core onlining/hotplug regression
2014-07-25 9:05 ` Thomas Gleixner
2014-07-25 9:36 ` Daniel J Blueman
@ 2014-09-13 9:03 ` Daniel J Blueman
1 sibling, 0 replies; 4+ messages in thread
From: Daniel J Blueman @ 2014-09-13 9:03 UTC (permalink / raw)
To: Thomas Gleixner, Lai Jiangshan
Cc: Oleg Nesterov, Peter Zijlstra, Hillf Danton, Borislav Petkov,
Ingo Molnar, Igor Mammedov, Steffen Persvold, LKML,
Andrew Morton
On 07/25/2014 05:05 PM, Thomas Gleixner wrote:
> On Fri, 25 Jul 2014, Daniel J Blueman wrote:
>> On a larger x86 system with 1728 cores, 3.15(.6) asserts on
>> smpboot_thread_fn's td->cpu != smp_processor_id() consistently after ~1500
>> cores are online.
>>
>> Reverting the only directly related changes I could find [1,2] doesn't help.
>> Debugging indicates there is a race where the created thread is quickly
>> migrated to core 0 when this occurs, since smp_processor_id returns 0 in these
>> cases. Thomas introduced a thread parked state to fix related issues a year
>> back. Linux 3.14(.13) boots just nice.
>
> Weird. Commits [1,2] are definitely not the culprits.
>
>> Full boot output is at:
>> https://resources.numascale.com/linux-315-thread-mig.txt
>
> Not really helpful, as we don't see what causes it. We just see the
> wreckage.
>
>> Any theories so far? I'll start bisecting when I have full access to the
>> system again in a week and I'll do some more debugging with intermittent
>> access before then.
>
> One thing you could try is enabling tracing.
>
> "ftrace=function ftrace_dump_on_oops"
>
> It'll take a looooong time to spill out the traces, but that should
> give us the root cause precisely.
It turns out that bisecting led to Lai's patch "Fix hotplug vs.
set_cpus_allowed_ptr()" [1]. Reverting it prevents the smpboot.c
BUG_ON(td->cpu != smp_processor_id()) in smpboot_thread_fn from tripping.
-- [1]
commit 6acbfb96976fc3350e30d964acb1dbbdf876d55e
Author: Lai Jiangshan <laijs@cn.fujitsu.com>
Date: Fri May 16 11:50:42 2014 +0800
sched: Fix hotplug vs. set_cpus_allowed_ptr()
Lai found that:
WARNING: CPU: 1 PID: 13 at arch/x86/kernel/smp.c:124
native_smp_send_reschedule+0x2d/0x4b()
...
migration_cpu_stop+0x1d/0x22
was caused by set_cpus_allowed_ptr() assuming that cpu_active_mask is
always a sub-set of cpu_online_mask.
This isn't true since 5fbd036b552f ("sched: Cleanup cpu_active
madness").
So set active and online at the same time to avoid this particular
problem.
Fixes: 5fbd036b552f ("sched: Cleanup cpu_active madness")
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Michael wang <wangyun@linux.vnet.ibm.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Cc: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
Cc: Toshi Kani <toshi.kani@hp.com>
Link: http://lkml.kernel.org/r/53758B12.8060609@cn.fujitsu.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
--
Daniel J Blueman
Principal Software Engineer, Numascale
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2014-09-13 9:03 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-07-25 7:50 [3.14] core onlining/hotplug regression Daniel J Blueman
2014-07-25 9:05 ` Thomas Gleixner
2014-07-25 9:36 ` Daniel J Blueman
2014-09-13 9:03 ` Daniel J Blueman
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.