From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6D6FC38A2A for ; Fri, 8 May 2020 09:22:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B0677208CA for ; Fri, 8 May 2020 09:22:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726712AbgEHJWR (ORCPT ); Fri, 8 May 2020 05:22:17 -0400 Received: from outbound-smtp18.blacknight.com ([46.22.139.245]:58387 "EHLO outbound-smtp18.blacknight.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725379AbgEHJWR (ORCPT ); Fri, 8 May 2020 05:22:17 -0400 Received: from mail.blacknight.com (pemlinmail06.blacknight.ie [81.17.255.152]) by outbound-smtp18.blacknight.com (Postfix) with ESMTPS id B00DC1C39C2 for ; Fri, 8 May 2020 10:22:15 +0100 (IST) Received: (qmail 2053 invoked from network); 8 May 2020 09:22:15 -0000 Received: from unknown (HELO techsingularity.net) (mgorman@techsingularity.net@[84.203.18.57]) by 81.17.254.9 with ESMTPSA (AES256-SHA encrypted, authenticated); 8 May 2020 09:22:15 -0000 Date: Fri, 8 May 2020 10:22:12 +0100 From: Mel Gorman To: Jirka Hladky Cc: Phil Auld , Peter Zijlstra , Ingo Molnar , Vincent Guittot , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Valentin Schneider , Hillf Danton , LKML , Douglas Shakshober , Waiman Long , Joe Mario , Bill Gray Subject: Re: [PATCH 00/13] Reconcile NUMA balancing decisions with the load balancer v6 Message-ID: <20200508092212.GE3758@techsingularity.net> References: <20200312214736.GA3818@techsingularity.net> <20200320152251.GC3818@techsingularity.net> <20200320163843.GD3818@techsingularity.net> <20200507155422.GD3758@techsingularity.net> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, May 07, 2020 at 06:29:44PM +0200, Jirka Hladky wrote: > Hi Mel, > > we are not targeting just OMP applications. We see the performance > degradation also for other workloads, like SPECjbb2005 and > SPECjvm2008. Even worse, it also affects a higher number of threads. > For example, comparing 5.7.0-0.rc2 against 5.6 kernel, on 4 NUMA > server with 2x AMD 7351 CPU, we see performance degradation 22% for 32 > threads (the system has 64 CPUs in total). We observe this degradation > only when we run a single SPECjbb binary. When running 4 SPECjbb > binaries in parallel, there is no change in performance between 5.6 > and 5.7. > Minimally I suggest confirming that it's really due to adjust_numa_imbalance() by making the function a no-op and retesting. I have found odd artifacts with it but I'm unsure how to proceed without causing problems elsehwere. For example, netperf on localhost in some cases reported a regression when the client and server were running on the same node. The problem appears to be that netserver completes its work faster when running local and goes idle more regularly. The cost of going idle and waking up builds up and a lower throughput is reported but I'm not sure if gaming an artifact like that is a good idea. > That's why we are asking for the kernel tunable, which we would add to > the tuned profile. We don't expect users to change this frequently but > rather to set the performance profile once based on the purpose of the > server. > > If you could prepare a patch for us, we would be more than happy to > test it extensively. Based on the results, we can then evaluate if > it's the way to go. Thoughts? > I would suggest simply disabling that function first to ensure that is really what is causing problems for you. -- Mel Gorman SUSE Labs