From: Peter Zijlstra <a.p.zijlstra@chello.nl>
To: Anton Blanchard <anton@samba.org>
Cc: mahesh@linux.vnet.ibm.com, linux-kernel@vger.kernel.org,
linuxppc-dev@lists.ozlabs.org, mingo@elte.hu,
benh@kernel.crashing.org, torvalds@linux-foundation.org
Subject: Re: [regression] 3.0-rc boot failure -- bisected to cd4ea6ae3982
Date: Tue, 19 Jul 2011 12:21:34 +0200 [thread overview]
Message-ID: <1311070894.13765.180.camel@twins> (raw)
In-Reply-To: <20110719144451.79bc69ab@kryten>
On Tue, 2011-07-19 at 14:44 +1000, Anton Blanchard wrote:
>
> Our node distances are a bit arbitrary (I make them up based on
> information given to us in the device tree). In terms of memory we have
> a maximum of three levels. To give some gross estimates, on chip memory
> might be 30GB/sec, on node memory 10-15GB/sec and off node memory
> 5GB/sec.
>
> The only thing we tweak with node distances is to make sure we go into
> node reclaim before going off node:
>
> /*
> * Before going off node we want the VM to try and reclaim from the local
> * node. It does this if the remote distance is larger than RECLAIM_DISTANCE.
> * With the default REMOTE_DISTANCE of 20 and the default RECLAIM_DISTANCE of
> * 20, we never reclaim and go off node straight away.
> *
> * To fix this we choose a smaller value of RECLAIM_DISTANCE.
> */
> #define RECLAIM_DISTANCE 10
> node distances:
> node 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
> 0: 10 20 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 1: 20 10 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 2: 20 20 10 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 3: 20 20 20 10 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 4: 40 40 40 40 10 20 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 5: 40 40 40 40 20 10 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 6: 40 40 40 40 20 20 10 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 7: 40 40 40 40 20 20 20 10 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 8: 40 40 40 40 40 40 40 40 10 20 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 9: 40 40 40 40 40 40 40 40 20 10 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 10: 40 40 40 40 40 40 40 40 20 20 10 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 11: 40 40 40 40 40 40 40 40 20 20 20 10 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 12: 40 40 40 40 40 40 40 40 40 40 40 40 10 20 20 20 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 13: 40 40 40 40 40 40 40 40 40 40 40 40 20 10 20 20 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 14: 40 40 40 40 40 40 40 40 40 40 40 40 20 20 10 20 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 15: 40 40 40 40 40 40 40 40 40 40 40 40 20 20 20 10 40 40 40 40 40 40 40 40 40 40 40 40 0 0 0 0
> 16: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 10 20 20 20 40 40 40 40 40 40 40 40 0 0 0 0
> 17: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 10 20 20 40 40 40 40 40 40 40 40 0 0 0 0
> 18: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 10 20 40 40 40 40 40 40 40 40 0 0 0 0
> 19: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 20 10 40 40 40 40 40 40 40 40 0 0 0 0
> 20: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 10 20 20 20 40 40 40 40 0 0 0 0
> 21: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 10 20 20 40 40 40 40 0 0 0 0
> 22: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 10 20 40 40 40 40 0 0 0 0
> 23: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 20 10 40 40 40 40 0 0 0 0
> 24: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 25: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 26: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 27: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 28: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 10 20 20 20 0 0 0 0
> 29: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 10 20 20 0 0 0 0
> 30: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 10 20 0 0 0 0
> 31: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 20 10 0 0 0 0
That looks very strange indeed.. up to node 23 there is the normal
symmetric matrix with all the trace elements on 10 (as we would expect
for local access), and some 4x4 sub-matrix stacked around the trace with
20, suggesting a single hop distance, and the rest on 40 being
out-there.
But row 24-27 and column 28-31 are way weird, how can that ever be?
Aren't the inter-connects symmetric and thus mandating a fully symmetric
matrix? That is, how can traffic from node 23 (row) to node 28 (column)
have inf bandwidth (0) yet traffic from node 28 (row) to node 23
(column) have a multi-hop distance of 40.
So the idea I had to generate numa sched domains from the node distance
( http://marc.info/?l=linux-kernel&m=130218515520540 ), would that still
work for you? [it does assume a symmetric matrix ]
WARNING: multiple messages have this Message-ID (diff)
From: Peter Zijlstra <a.p.zijlstra@chello.nl>
To: Anton Blanchard <anton@samba.org>
Cc: mahesh@linux.vnet.ibm.com, linuxppc-dev@lists.ozlabs.org,
linux-kernel@vger.kernel.org, mingo@elte.hu,
torvalds@linux-foundation.org
Subject: Re: [regression] 3.0-rc boot failure -- bisected to cd4ea6ae3982
Date: Tue, 19 Jul 2011 12:21:34 +0200 [thread overview]
Message-ID: <1311070894.13765.180.camel@twins> (raw)
In-Reply-To: <20110719144451.79bc69ab@kryten>
On Tue, 2011-07-19 at 14:44 +1000, Anton Blanchard wrote:
>=20
> Our node distances are a bit arbitrary (I make them up based on
> information given to us in the device tree). In terms of memory we have
> a maximum of three levels. To give some gross estimates, on chip memory
> might be 30GB/sec, on node memory 10-15GB/sec and off node memory
> 5GB/sec.
>=20
> The only thing we tweak with node distances is to make sure we go into
> node reclaim before going off node:
>=20
> /*
> * Before going off node we want the VM to try and reclaim from the local
> * node. It does this if the remote distance is larger than RECLAIM_DISTA=
NCE.
> * With the default REMOTE_DISTANCE of 20 and the default RECLAIM_DISTANC=
E of
> * 20, we never reclaim and go off node straight away.
> *
> * To fix this we choose a smaller value of RECLAIM_DISTANCE.
> */
> #define RECLAIM_DISTANCE 10
> node distances:
> node 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 =
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31=20
> 0: 10 20 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 1: 20 10 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 2: 20 20 10 20 40 40 40 40 40 40 40 40 40 40 40 40 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 3: 20 20 20 10 40 40 40 40 40 40 40 40 40 40 40 40 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 4: 40 40 40 40 10 20 20 20 40 40 40 40 40 40 40 40 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 5: 40 40 40 40 20 10 20 20 40 40 40 40 40 40 40 40 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 6: 40 40 40 40 20 20 10 20 40 40 40 40 40 40 40 40 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 7: 40 40 40 40 20 20 20 10 40 40 40 40 40 40 40 40 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 8: 40 40 40 40 40 40 40 40 10 20 20 20 40 40 40 40 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 9: 40 40 40 40 40 40 40 40 20 10 20 20 40 40 40 40 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 10: 40 40 40 40 40 40 40 40 20 20 10 20 40 40 40 40 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 11: 40 40 40 40 40 40 40 40 20 20 20 10 40 40 40 40 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 12: 40 40 40 40 40 40 40 40 40 40 40 40 10 20 20 20 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 13: 40 40 40 40 40 40 40 40 40 40 40 40 20 10 20 20 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 14: 40 40 40 40 40 40 40 40 40 40 40 40 20 20 10 20 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 15: 40 40 40 40 40 40 40 40 40 40 40 40 20 20 20 10 40 =
40 40 40 40 40 40 40 40 40 40 40 0 0 0 0=20
> 16: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 10 =
20 20 20 40 40 40 40 40 40 40 40 0 0 0 0=20
> 17: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 =
10 20 20 40 40 40 40 40 40 40 40 0 0 0 0=20
> 18: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 =
20 10 20 40 40 40 40 40 40 40 40 0 0 0 0=20
> 19: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 =
20 20 10 40 40 40 40 40 40 40 40 0 0 0 0=20
> 20: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 =
40 40 40 10 20 20 20 40 40 40 40 0 0 0 0=20
> 21: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 =
40 40 40 20 10 20 20 40 40 40 40 0 0 0 0=20
> 22: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 =
40 40 40 20 20 10 20 40 40 40 40 0 0 0 0=20
> 23: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 =
40 40 40 20 20 20 10 40 40 40 40 0 0 0 0=20
> 24: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 =
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0=20
> 25: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 =
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0=20
> 26: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 =
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0=20
> 27: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 =
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0=20
> 28: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 =
40 40 40 40 40 40 40 10 20 20 20 0 0 0 0=20
> 29: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 =
40 40 40 40 40 40 40 20 10 20 20 0 0 0 0=20
> 30: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 =
40 40 40 40 40 40 40 20 20 10 20 0 0 0 0=20
> 31: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 =
40 40 40 40 40 40 40 20 20 20 10 0 0 0 0=20
That looks very strange indeed.. up to node 23 there is the normal
symmetric matrix with all the trace elements on 10 (as we would expect
for local access), and some 4x4 sub-matrix stacked around the trace with
20, suggesting a single hop distance, and the rest on 40 being
out-there.
But row 24-27 and column 28-31 are way weird, how can that ever be?
Aren't the inter-connects symmetric and thus mandating a fully symmetric
matrix? That is, how can traffic from node 23 (row) to node 28 (column)
have inf bandwidth (0) yet traffic from node 28 (row) to node 23
(column) have a multi-hop distance of 40.
So the idea I had to generate numa sched domains from the node distance
( http://marc.info/?l=3Dlinux-kernel&m=3D130218515520540 ), would that stil=
l
work for you? [it does assume a symmetric matrix ]
next prev parent reply other threads:[~2011-07-19 10:22 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-07-07 10:22 [regression] 3.0-rc boot failure -- bisected to cd4ea6ae3982 Mahesh J Salgaonkar
2011-07-07 10:22 ` Mahesh J Salgaonkar
2011-07-07 10:59 ` Peter Zijlstra
2011-07-07 10:59 ` Peter Zijlstra
2011-07-07 11:55 ` Mahesh J Salgaonkar
2011-07-07 11:55 ` Mahesh J Salgaonkar
2011-07-07 12:28 ` Peter Zijlstra
2011-07-07 12:28 ` Peter Zijlstra
2011-07-14 0:34 ` Anton Blanchard
2011-07-14 0:34 ` Anton Blanchard
2011-07-14 4:35 ` Anton Blanchard
2011-07-14 4:35 ` Anton Blanchard
2011-07-14 13:16 ` Peter Zijlstra
2011-07-14 13:16 ` Peter Zijlstra
2011-07-15 0:45 ` Anton Blanchard
2011-07-15 0:45 ` Anton Blanchard
2011-07-15 8:37 ` Peter Zijlstra
2011-07-15 8:37 ` Peter Zijlstra
2011-07-18 21:35 ` Peter Zijlstra
2011-07-18 21:35 ` Peter Zijlstra
2011-07-19 4:44 ` Anton Blanchard
2011-07-19 4:44 ` Anton Blanchard
2011-07-19 10:21 ` Peter Zijlstra [this message]
2011-07-19 10:21 ` Peter Zijlstra
2011-07-20 2:03 ` Anton Blanchard
2011-07-20 2:03 ` Anton Blanchard
2011-07-20 10:14 ` Anton Blanchard
2011-07-20 10:14 ` Anton Blanchard
2011-07-20 10:45 ` Peter Zijlstra
2011-07-20 10:45 ` Peter Zijlstra
2011-07-20 12:14 ` Anton Blanchard
2011-07-20 12:14 ` Anton Blanchard
2011-07-20 14:40 ` Linus Torvalds
2011-07-20 14:40 ` Linus Torvalds
2011-07-20 14:58 ` Peter Zijlstra
2011-07-20 14:58 ` Peter Zijlstra
2011-07-20 16:04 ` Linus Torvalds
2011-07-20 16:04 ` Linus Torvalds
2011-07-20 16:42 ` Ingo Molnar
2011-07-20 16:42 ` Ingo Molnar
2011-07-20 16:42 ` Peter Zijlstra
2011-07-20 16:42 ` Peter Zijlstra
2011-07-20 17:29 ` [tip:sched/urgent] sched: Avoid creating superfluous NUMA domains on non-NUMA systems tip-bot for Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1311070894.13765.180.camel@twins \
--to=a.p.zijlstra@chello.nl \
--cc=anton@samba.org \
--cc=benh@kernel.crashing.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=mahesh@linux.vnet.ibm.com \
--cc=mingo@elte.hu \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.