From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1033011Ab2CSU3F (ORCPT ); Mon, 19 Mar 2012 16:29:05 -0400 Received: from mail-we0-f194.google.com ([74.125.82.194]:34303 "EHLO mail-we0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1759396Ab2CSU27 (ORCPT ); Mon, 19 Mar 2012 16:28:59 -0400 Date: Mon, 19 Mar 2012 21:28:49 +0100 From: Ingo Molnar To: Christoph Lameter Cc: Andrea Arcangeli , Peter Zijlstra , Avi Kivity , Linus Torvalds , Andrew Morton , Thomas Gleixner , Ingo Molnar , Paul Turner , Suresh Siddha , Mike Galbraith , "Paul E. McKenney" , Lai Jiangshan , Dan Smith , Bharata B Rao , Lee Schermerhorn , Rik van Riel , Johannes Weiner , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [RFC][PATCH 00/26] sched/numa Message-ID: <20120319202846.GA26555@gmail.com> References: <20120316144028.036474157@chello.nl> <4F670325.7080700@redhat.com> <1332155527.18960.292.camel@twins> <20120319130401.GI24602@redhat.com> <1332164371.18960.339.camel@twins> <20120319142046.GP24602@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org * Christoph Lameter wrote: > On Mon, 19 Mar 2012, Andrea Arcangeli wrote: > > > Yeah I'll try to fix that but it's massively complex and > > frankly benchmarking wise it won't help much fixing that... > > so it's beyond the end of my todo list. > > Well a word of caution here: SGI tried to implement automatic > migration schemes back in the 90's but they were never able to > show a general benefit of migration. The overhead added > because of auto migration often was not made up by true > acceleration of the applications running on the system. They > were able to tune the automatic migration to work on > particular classes of applications but it never turned out to > be generally advantageous. Obviously any such scheme must be a win in general for it to be default on. We don't have the numbers to justify that - and I'm sceptical whether it will be possible, but I'm willing to be surprised. I'm especially sceptical since most mainstream NUMA systems tend to have a low NUMA factor. Thus the actual cost of being NUMA is pretty low. That having said PeterZ's numbers showed some pretty good improvement for the streams workload: before: 512.8M after: 615.7M i.e. a +20% improvement on a not very heavily NUMA box. That kind of raw speedup of a CPU execution workload like streams is definitely not something to ignore out of hand. *IF* there is a good automatism that can activate it for the apps that are very likely to benefit from it then we can possibly do it. But a lot more measurements have to be done, and I'd be also very interested in the areas that regress. Otherwise, if no robust automation is possible, it will have to be opt-in, on a per app basis, with both programmatic and sysadmin knobs available. (who will hopefully make use if it...) That's the best we can do I think. > I wonder how we can verify that the automatic migration > schemes are a real benefit to the application? We have a > history of developing a kernel that decreases in performance > as development proceeds. How can we make sure that these > schemes are actually beneficial overall for all loads and do > not cause regressions elsewhere? [...] The usual way? Thanks, Ingo From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx117.postini.com [74.125.245.117]) by kanga.kvack.org (Postfix) with SMTP id 252496B0106 for ; Mon, 19 Mar 2012 16:29:00 -0400 (EDT) Received: by werm1 with SMTP id m1so1427148wer.2 for ; Mon, 19 Mar 2012 13:28:58 -0700 (PDT) Date: Mon, 19 Mar 2012 21:28:49 +0100 From: Ingo Molnar Subject: Re: [RFC][PATCH 00/26] sched/numa Message-ID: <20120319202846.GA26555@gmail.com> References: <20120316144028.036474157@chello.nl> <4F670325.7080700@redhat.com> <1332155527.18960.292.camel@twins> <20120319130401.GI24602@redhat.com> <1332164371.18960.339.camel@twins> <20120319142046.GP24602@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Christoph Lameter Cc: Andrea Arcangeli , Peter Zijlstra , Avi Kivity , Linus Torvalds , Andrew Morton , Thomas Gleixner , Ingo Molnar , Paul Turner , Suresh Siddha , Mike Galbraith , "Paul E. McKenney" , Lai Jiangshan , Dan Smith , Bharata B Rao , Lee Schermerhorn , Rik van Riel , Johannes Weiner , linux-kernel@vger.kernel.org, linux-mm@kvack.org * Christoph Lameter wrote: > On Mon, 19 Mar 2012, Andrea Arcangeli wrote: > > > Yeah I'll try to fix that but it's massively complex and > > frankly benchmarking wise it won't help much fixing that... > > so it's beyond the end of my todo list. > > Well a word of caution here: SGI tried to implement automatic > migration schemes back in the 90's but they were never able to > show a general benefit of migration. The overhead added > because of auto migration often was not made up by true > acceleration of the applications running on the system. They > were able to tune the automatic migration to work on > particular classes of applications but it never turned out to > be generally advantageous. Obviously any such scheme must be a win in general for it to be default on. We don't have the numbers to justify that - and I'm sceptical whether it will be possible, but I'm willing to be surprised. I'm especially sceptical since most mainstream NUMA systems tend to have a low NUMA factor. Thus the actual cost of being NUMA is pretty low. That having said PeterZ's numbers showed some pretty good improvement for the streams workload: before: 512.8M after: 615.7M i.e. a +20% improvement on a not very heavily NUMA box. That kind of raw speedup of a CPU execution workload like streams is definitely not something to ignore out of hand. *IF* there is a good automatism that can activate it for the apps that are very likely to benefit from it then we can possibly do it. But a lot more measurements have to be done, and I'd be also very interested in the areas that regress. Otherwise, if no robust automation is possible, it will have to be opt-in, on a per app basis, with both programmatic and sysadmin knobs available. (who will hopefully make use if it...) That's the best we can do I think. > I wonder how we can verify that the automatic migration > schemes are a real benefit to the application? We have a > history of developing a kernel that decreases in performance > as development proceeds. How can we make sure that these > schemes are actually beneficial overall for all loads and do > not cause regressions elsewhere? [...] The usual way? Thanks, Ingo -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org