All of lore.kernel.org
 help / color / mirror / Atom feed
From: Anton Blanchard <anton@samba.org>
To: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: mahesh@linux.vnet.ibm.com, linux-kernel@vger.kernel.org,
	linuxppc-dev@lists.ozlabs.org, mingo@elte.hu,
	benh@kernel.crashing.org, torvalds@linux-foundation.org
Subject: Re: [regression] 3.0-rc boot failure -- bisected to cd4ea6ae3982
Date: Wed, 20 Jul 2011 12:03:26 +1000	[thread overview]
Message-ID: <20110720120326.093d9045@kryten> (raw)
In-Reply-To: <1311070894.13765.180.camel@twins>


Hi,

> That looks very strange indeed.. up to node 23 there is the normal
> symmetric matrix with all the trace elements on 10 (as we would expect
> for local access), and some 4x4 sub-matrix stacked around the trace
> with 20, suggesting a single hop distance, and the rest on 40 being
> out-there.
> 
> But row 24-27 and column 28-31 are way weird, how can that ever be?
> Aren't the inter-connects symmetric and thus mandating a fully
> symmetric matrix? That is, how can traffic from node 23 (row) to node
> 28 (column) have inf bandwidth (0) yet traffic from node 28 (row) to
> node 23 (column) have a multi-hop distance of 40.

Good point, it definitely makes no sense. It looks like a bug in
numactl, the raw data looks reasonable:

# cat /sys/devices/system/node/node?/distance node??/distance
10 20 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
20 10 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
20 20 10 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
20 20 20 10 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 10 20 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 20 10 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 20 20 10 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 20 20 20 10 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 10 20 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 20 10 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 20 20 10 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 20 20 20 10 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 10 20 20 20 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 20 10 20 20 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 20 20 10 20 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 20 20 20 10 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 10 20 20 20 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 10 20 20 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 10 20 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 20 10 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 10 20 20 20 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 10 20 20 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 10 20 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 20 10 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 10 20 20 20
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 10 20 20
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 10 20
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 20 10

Yet another bug to track down :(

> So the idea I had to generate numa sched domains from the node
> distance ( http://marc.info/?l=linux-kernel&m=130218515520540 ),
> would that still work for you? [it does assume a symmetric matrix ]

It should work for us and it makes our NUMA memory and scheduler
domains more consistent. Nice!

Anton

WARNING: multiple messages have this Message-ID (diff)
From: Anton Blanchard <anton@samba.org>
To: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: mahesh@linux.vnet.ibm.com, linuxppc-dev@lists.ozlabs.org,
	linux-kernel@vger.kernel.org, mingo@elte.hu,
	torvalds@linux-foundation.org
Subject: Re: [regression] 3.0-rc boot failure -- bisected to cd4ea6ae3982
Date: Wed, 20 Jul 2011 12:03:26 +1000	[thread overview]
Message-ID: <20110720120326.093d9045@kryten> (raw)
In-Reply-To: <1311070894.13765.180.camel@twins>


Hi,

> That looks very strange indeed.. up to node 23 there is the normal
> symmetric matrix with all the trace elements on 10 (as we would expect
> for local access), and some 4x4 sub-matrix stacked around the trace
> with 20, suggesting a single hop distance, and the rest on 40 being
> out-there.
> 
> But row 24-27 and column 28-31 are way weird, how can that ever be?
> Aren't the inter-connects symmetric and thus mandating a fully
> symmetric matrix? That is, how can traffic from node 23 (row) to node
> 28 (column) have inf bandwidth (0) yet traffic from node 28 (row) to
> node 23 (column) have a multi-hop distance of 40.

Good point, it definitely makes no sense. It looks like a bug in
numactl, the raw data looks reasonable:

# cat /sys/devices/system/node/node?/distance node??/distance
10 20 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
20 10 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
20 20 10 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
20 20 20 10 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 10 20 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 20 10 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 20 20 10 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 20 20 20 10 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 10 20 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 20 10 20 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 20 20 10 20 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 20 20 20 10 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 10 20 20 20 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 20 10 20 20 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 20 20 10 20 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 20 20 20 10 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 10 20 20 20 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 10 20 20 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 10 20 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 20 10 40 40 40 40 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 10 20 20 20 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 10 20 20 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 10 20 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 20 10 40 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 10 20 20 20
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 10 20 20
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 10 20
40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 20 20 10

Yet another bug to track down :(

> So the idea I had to generate numa sched domains from the node
> distance ( http://marc.info/?l=linux-kernel&m=130218515520540 ),
> would that still work for you? [it does assume a symmetric matrix ]

It should work for us and it makes our NUMA memory and scheduler
domains more consistent. Nice!

Anton

  reply	other threads:[~2011-07-20  2:03 UTC|newest]

Thread overview: 43+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2011-07-07 10:22 [regression] 3.0-rc boot failure -- bisected to cd4ea6ae3982 Mahesh J Salgaonkar
2011-07-07 10:22 ` Mahesh J Salgaonkar
2011-07-07 10:59 ` Peter Zijlstra
2011-07-07 10:59   ` Peter Zijlstra
2011-07-07 11:55   ` Mahesh J Salgaonkar
2011-07-07 11:55     ` Mahesh J Salgaonkar
2011-07-07 12:28     ` Peter Zijlstra
2011-07-07 12:28       ` Peter Zijlstra
2011-07-14  0:34   ` Anton Blanchard
2011-07-14  0:34     ` Anton Blanchard
2011-07-14  4:35     ` Anton Blanchard
2011-07-14  4:35       ` Anton Blanchard
2011-07-14 13:16       ` Peter Zijlstra
2011-07-14 13:16         ` Peter Zijlstra
2011-07-15  0:45         ` Anton Blanchard
2011-07-15  0:45           ` Anton Blanchard
2011-07-15  8:37           ` Peter Zijlstra
2011-07-15  8:37             ` Peter Zijlstra
2011-07-18 21:35           ` Peter Zijlstra
2011-07-18 21:35             ` Peter Zijlstra
2011-07-19  4:44             ` Anton Blanchard
2011-07-19  4:44               ` Anton Blanchard
2011-07-19 10:21               ` Peter Zijlstra
2011-07-19 10:21                 ` Peter Zijlstra
2011-07-20  2:03                 ` Anton Blanchard [this message]
2011-07-20  2:03                   ` Anton Blanchard
2011-07-20 10:14                 ` Anton Blanchard
2011-07-20 10:14                   ` Anton Blanchard
2011-07-20 10:45                   ` Peter Zijlstra
2011-07-20 10:45                     ` Peter Zijlstra
2011-07-20 12:14                     ` Anton Blanchard
2011-07-20 12:14                       ` Anton Blanchard
2011-07-20 14:40                       ` Linus Torvalds
2011-07-20 14:40                         ` Linus Torvalds
2011-07-20 14:58                         ` Peter Zijlstra
2011-07-20 14:58                           ` Peter Zijlstra
2011-07-20 16:04                           ` Linus Torvalds
2011-07-20 16:04                             ` Linus Torvalds
2011-07-20 16:42                             ` Ingo Molnar
2011-07-20 16:42                               ` Ingo Molnar
2011-07-20 16:42                             ` Peter Zijlstra
2011-07-20 16:42                               ` Peter Zijlstra
2011-07-20 17:29                               ` [tip:sched/urgent] sched: Avoid creating superfluous NUMA domains on non-NUMA systems tip-bot for Peter Zijlstra

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20110720120326.093d9045@kryten \
    --to=anton@samba.org \
    --cc=a.p.zijlstra@chello.nl \
    --cc=benh@kernel.crashing.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linuxppc-dev@lists.ozlabs.org \
    --cc=mahesh@linux.vnet.ibm.com \
    --cc=mingo@elte.hu \
    --cc=torvalds@linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.