From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755510AbbCFSMG (ORCPT ); Fri, 6 Mar 2015 13:12:06 -0500 Received: from mout.gmx.net ([212.227.15.19]:51851 "EHLO mout.gmx.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755411AbbCFSMD (ORCPT ); Fri, 6 Mar 2015 13:12:03 -0500 Message-ID: <1425665511.7562.36.camel@gmx.de> Subject: Re: NMI watchdog triggering during load_balance From: Mike Galbraith To: David Ahern Cc: Peter Zijlstra , Ingo Molnar , LKML Date: Fri, 06 Mar 2015 19:11:51 +0100 In-Reply-To: <54F9C155.3050309@oracle.com> References: <54F92788.6010007@oracle.com> <1425617559.16821.36.camel@gmx.de> <54F9C155.3050309@oracle.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.12.11 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:eUfgCTRc0FUDqUjX5wp47yB1NitCOeMiyeU2mHSy8+8S12MGq5n MD2QJj36uTgOuoLTkcvYnQPOEzXMYw/uKi5An2mf2nzh43Tm9MA9l5vWc00zjVcbJD3hE2Y qy2WUa7FiiIZE3XEKoNdH4uzRXScfN3phMMlFsedgCbPyvurJZuE0pC/J+II/z7n+nQQChu gnlbxbbcEIr+Ru0hgrtow== X-UI-Out-Filterresults: notjunk:1; Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, 2015-03-06 at 08:01 -0700, David Ahern wrote: > On 3/5/15 9:52 PM, Mike Galbraith wrote: > >> CPU970 attaching sched-domain: > >> domain 0: span 968-975 level SIBLING > >> groups: 8 single CPU groups > >> domain 1: span 968-975 level MC > >> groups: 1 group with 8 cpus > >> domain 2: span 768-1023 level CPU > >> groups: 4 groups with 256 cpus per group > > > > Wow, that topology is horrid. I'm not surprised that your box is > > writhing in agony. Can you twiddle that? > > > > twiddle that how? That was the question, _do_ you have any control, because that topology is toxic. I guess your reply means 'nope'. > The system has 4 physical cpus (sockets). Each cpu has 32 cores with 8 > threads per core and each cpu has 4 memory controllers. Thank god I've never met one of these, looks like the box from hell :) > If I disable SCHED_MC and CGROUPS_SCHED (group scheduling) there is a > noticeable improvement -- watchdog does not trigger and I do not get the > rq locks held for 2-3 seconds. But there is still fairly high cpu usage > for an idle system. Perhaps I should leave SCHED_MC on and disable > SCHED_SMT; I'll try that today. Well, if you disable SMT,your troubles _should_ shrink radically, as your box does. You should probably look at why you have CPU domains. You don't ever want to see that on a NUMA box. -Mike