From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932542AbdK2Lzh (ORCPT ); Wed, 29 Nov 2017 06:55:37 -0500 Received: from mail-ve1eur01on0074.outbound.protection.outlook.com ([104.47.1.74]:34436 "EHLO EUR01-VE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932529AbdK2Lze (ORCPT ); Wed, 29 Nov 2017 06:55:34 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Morten.Rasmussen@arm.com; Date: Wed, 29 Nov 2017 11:55:23 +0000 From: Morten Rasmussen To: Vincent Guittot Cc: Peter Zijlstra , "mingo@redhat.com" , Dietmar Eggemann , mgalbraith@suse.de, linux-kernel Subject: Re: [PATCH] sched/topology: Set SD_PREFER_SIBLING consistently on non-NUMA levels Message-ID: <20171129115523.GF10381@e105550-lin.cambridge.arm.com> References: <1511803767-24572-1-git-send-email-morten.rasmussen@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Originating-IP: [217.140.96.140] X-ClientProxiedBy: VI1PR0101CA0044.eurprd01.prod.exchangelabs.com (2603:10a6:800:1f::12) To HE1PR0801MB1962.eurprd08.prod.outlook.com (2603:10a6:3:4f::16) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ee2a083a-1d2c-4879-dd7c-08d53720160a X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(5600026)(4604075)(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(48565401081)(2017052603199);SRVR:HE1PR0801MB1962; X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1962;3:DUHeksbFnOa1g/zctVvDvIyIbQo6k3mtej6lZJr/bqXqA960+zYecIDqd+Guxvnfi/lZOkBYsAR5TkRKcsDfNoK7OMEpU5y7BjwsSCVG5MSzRHK6Cn7HcnNQOOoUmo3mAg1M9iMoC/CYHZ7povmlC1g0yTIYcKW4Th39fhM86uCnqCGnVrZOYpNsmNvEZ1miimPCRGtgGYCuIUD5IC4FUZuVbwU/i7+awjlF9wG+wqYTzowCctS5IgYOn+ay8jS4;25:gXddZhNYHHa+iHnArgRKJFGA8dwBIVWz/TgwLCkunaa/qbWJZYIq/+B809FXZE5FGEqIvVFLT6meLhMJoS7Q5D99x6ynRAZtuiW0yW69xq5Jjd0G+RsHlF0nUccso49h4UojElIwlyvoEZamehq6hlKFSLOr2Gf32JidcKfpIT1jMPDVXLYkQjmGmuK99F9N38r3ZQ/DOzsHHZ13hC+IBx1iD+6aLnNB9l0uBkYnQyYq+20qHmV17psF50MOMbuw/d3OM28S8CiYXEXn6L9gV7hUp5Ef28hcKaH2ytAzA4sMaY107BXxL+6wJgDoxn2Yn99mxf+ZDvn5S60MMuQesWru+tg4FkV1aPpsIz44/lk=;31:FKHaB1rhY68np4IMWSqSt+RkHJvAnUULwylRpJOurUv+7FWhtRZDaL0SafO2QqQCZugXQz2vr/1r5lwCn8HYiWhl1RnI+3PsrSfI7lqDPCrcnvSsyQ+FJmFfKQ65O8XeWEv3lF25K+6sxsgvz+Bifenz9GTOrPSvbuuAMNH1RtDeeft3LlEGbkehXhfJLPYlMr+qWcs7B71h7KYF/7AGtgJzsQQm2gISqqVnjZXK18o= X-MS-TrafficTypeDiagnostic: HE1PR0801MB1962: X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1962;20:20/zO82d7J6f17pGZN9qY+xb3MLnjy17cInw8rvO9Zl37MZ1hF23z8Gl1Xo9nbRVq4xH7AGnIetYt09QU6eAKH2cTR5aTvkRmO8e+dkqCZ/sXAh5FsJ+hHMD0g2bubqel69RmqVu6cn4PjV3Iez1/rPP/GzLqeP4yGGpYAphBw4DMsCYZkHzP8BNqYfe7sP8gHvxM9wITF6Z18VVPqmJ6jehajOPhYiSzjZzDpznXmvqHQCy1S+nBEPgl8twyeCAzH34ycC4Ihhm7G3MaXRSV4KkcAD1QMOFTxu9HCQTfwMPNHhWo/cEHm6tiLLCoD0WsKpLcqLdhMsz8fj7yLqIjb/Le4Izunx173Ii/v0hsQiGYAroYgfIFgLLL0l+fzLxEnfT6bwc4g3G4WyMize2J+kDEJwp/5fkQNgaK4BuPxATGMyIaMqog5qnw/DH1tiPQoU8HsFUvAKbB4JPCazCij8XS/3l7IdnyhAKK+wBjrpn/GwZCkcDT0p9XMup8woG;4:0r2LdxISnLrxaLmOg3QIB1+FD3ttg7tUjk2U95Rp57w3aEqMK8gUMlwpxLgYHBCYsJuC+2qpFkhzH0sU34Z9kD9TKSYlBky5ESccjAUlfDBCVoCs7V4/FFOwe7mtsAvxu0i5GTzXLJqOhp7YGSniY/ZK9WHkOAYUYNtYLAfay54n4eBUJaHl/112iiV9iHryMVJb/78JFS1gDy+Wqe5GXN+DhvbJme5fpvN20BuMVgrpzWnZCDG+41RwoTUfG+DRZgldl4iYe8hLCNu/8EG27sQczCJN9gfum8/CLry3dHlb1l+5ZlQ6LJFEa+QWx63/K0YvfkEJJ//hawVveNij9bhXulajO9Hodr241pKS5ls= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(180628864354917)(17755550239193); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040450)(2401047)(5005006)(8121501046)(3002001)(3231022)(93006095)(93001095)(10201501046)(6055026)(6041248)(20161123558100)(20161123564025)(20161123562025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(20161123555025)(6072148)(201708071742011);SRVR:HE1PR0801MB1962;BCL:0;PCL:0;RULEID:(100000803101)(100110400095);SRVR:HE1PR0801MB1962; X-Forefront-PRVS: 05066DEDBB X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(6009001)(39860400002)(366004)(376002)(346002)(40434004)(199003)(24454002)(189002)(106356001)(83506002)(5660300001)(58126008)(105586002)(53936002)(6246003)(6666003)(25786009)(101416001)(2950100002)(6916009)(8936002)(4326008)(6116002)(76176999)(16526018)(53546010)(54906003)(97736004)(47776003)(23726003)(72206003)(478600001)(54356999)(66066001)(50986999)(81166006)(305945005)(316002)(8676002)(189998001)(7736002)(5890100001)(2906002)(3846002)(86362001)(33656002)(8746002)(229853002)(1076002)(7696005)(68736007)(50466002)(55016002)(81156014)(52116002)(18370500001);DIR:OUT;SFP:1101;SCL:1;SRVR:HE1PR0801MB1962;H:e105550-lin.cambridge.arm.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:3;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;HE1PR0801MB1962;23:2hkeht/D5nmsLFebuEQG4/3l2CLuBpEgUPbvtB2?= =?us-ascii?Q?lu2yN1U/ZZcklXKx9atnxLSGO1Shi6OzEFyJACxf8W7aHOR/okUY4RtRJAu0?= =?us-ascii?Q?6MWVJjYWT7XD1G581bnsafHjBcxWo/2wWBbWDRCV4RPrtDW5/Cd7n1Xqg0ka?= =?us-ascii?Q?kvOPidFGz1c8oECWktttq1d9N4zJ2fv9aXSs8U37q/pEx7o4+ZHqZITV/AyD?= =?us-ascii?Q?igp/v7eQEh+uHG6RK9eBwn8zQch909a1uSUIOpxxvIba8t05yMFjeakCsArf?= =?us-ascii?Q?nUl4w/iKKXnE1qh9C/NJVoRN5RJxbtVPDsshkL/3df+zbBFZlssebSCKiFw5?= =?us-ascii?Q?OsTBEirzqheeAOv50JbR7b1r5WscVl8Wy1XnbJdBc3XbDU1BD8lNlID0b5YP?= =?us-ascii?Q?7iY9lEtrDoEd0MpkBUZf3agOG8QvpD7hLBw1ZkgEU9psLpb6Bp9T7Rg3RnwM?= =?us-ascii?Q?rrVg/4fsF7litL+AmjbvmSygyrq/7tcBGiWr9mwMkfSLBbHlo1DxngD7aiZC?= =?us-ascii?Q?i5IqmDToUMbLA0bsyyClgrLOXt0QYgL1HjHBYfFEMlUUICEF9wxMfki5HsMh?= =?us-ascii?Q?DenxPZj5H6Lk70TdIsa5n36cxB780unLNBEiaM24TYwBwkMYxlFGfRXVuLXu?= =?us-ascii?Q?hFGwD3N2P1bNGySUU9Pxeupe/3mNbo9f6PNf89EQJSDi2xUUNDU2lacr+fHB?= =?us-ascii?Q?pY+eJgYyw45XPgE8DpIImt28Yx0ZZfy3XpK+md4BBcIvDKqTHZknrY4YRdij?= =?us-ascii?Q?6PlX4LHxqFRl4nea54i3fiUAuL2RBNpj0funVBPP4JO/6Ifss9AbnjgJBcsA?= =?us-ascii?Q?2frY7FuqNTtEFkGPbMemUws462lucUFzM1oHCskCGgc8yU0mrUAlD44oUOqL?= =?us-ascii?Q?t4sje/3pXNAFHQaz0Bpe3xBr3z1S9SU4grTSTGDkQKaosBsle6w2iGlxyonf?= =?us-ascii?Q?WiNVULAPN6MteY91TbEYTorJFHG5eyh0gt2inbnFUHiL9TKe/lm2ZMH+ujMW?= =?us-ascii?Q?57C+X5drUNKKaHecMuLjPDUXYeTakBky1UYUm+QWuXoZX0u1fDbV47pFpJYG?= =?us-ascii?Q?hR28Uqbt4J/LjCzzO/ARTFujKL56DDyPHzRrdz3QEFIDDBGN3EtYkAUMjxNU?= =?us-ascii?Q?wO6uraIHXNwem/1FhvHb23wPTfDElKorl35UooDyzaQGOmxukR0O1MeC39Sh?= =?us-ascii?Q?nu9HzfJg6vISIKg7XI+xbluyKmsqKHgol5wd8q7IZPcj1GeZV5BID5UOElCO?= =?us-ascii?Q?Df2YNstLrZLmZG1Qi/PyfkW4RF7yLzlXkx5Q+TSYlO16IVMfXP/wdDwByy8K?= =?us-ascii?Q?D3apgYF/7JSeO3HBRQBJaPPM=3D?= X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1962;6:rPbpa3C3Xoi9giosr2enBpW85VEkfcbzF1QZzMPYRM9LsWEheCBgMlJ8akgYEGMBC1XTaGqAljwfOcXvFDw83NEpKjkSapXBu4cBke+RWXeoSxpjNFb+RF0ckboSsRWMn8d5ObvmmHzkU40Hrzw6esDkA11IX4XxiGEfGhii6q7eZMpx8esdKzr1RST/+e+yMoVsNxdnI1euCu0vPSj/BvipaQraByE3OfWz4jQEqmMGu23b9PiQeLwA52NGhYLirfLfqRRogNuU1HJ+2/uvd0l2CeuOyWVKf1yHkhWeQ6xp0bVQWDyNQo41JoPf2w44hEBUZemkGGrPn0BKqVIvNqyTmrDyX8jIbZK26XTucFc=;5:Jkez548l0gHNrUiELNz4nqPQzdda16PKsnToQU0Xb/2UaNgeY0kn3CMqQmy1TgQ0GNVCvXZXDqu/wUHqvmEnkBpcM+vh+Zqq8LaojiR4Duc9V4eUYCfDuaHo2v7cs6hk55hYXLID+T+a8oxBMubcMO7oNVXw9iKrE4KvkIo4crc=;24:0xZmhQxAAtcldjQF+3oqkxEuTPM6Pt8GSnfALRli3l/tKj5sITtzQxP4RfwZ1k88ywaXUWHzgyW0LfEQUOJmeD7TFmwWlHoGb1j6XP0w/Mc=;7:kx4OaIFUXqbkxRXoT/wye+PJ6N8TOxOyq+yLKpawkmX8wXsLevZOZbssDDBrriPFi53ouWd15M/VAx8sbGa3kbJ+VKSu+Ok2t2SlR1rHWsEIOpQTRalb4AmrA7509B7SBNTFf9zN2+/irVTd7RLtFbq1F/jcxWiwPvwqhUM9NJ2v+HiNYCseeqU4LJXhgRisvbdJD/Dc3SIgetwtPGsZJGx2mXzgBbNOW3RTn7UyueoRGWqqXI8vHrAEj2X7zHgM SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Nov 2017 11:55:28.4522 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ee2a083a-1d2c-4879-dd7c-08d53720160a X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0801MB1962 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by nfs id vATBtfB9031886 On Wed, Nov 29, 2017 at 10:24:57AM +0100, Vincent Guittot wrote: > Hi Morten, > > On 27 November 2017 at 18:29, Morten Rasmussen wrote: > > SD_PREFER_SIBLING adds an additional bias towards spreading tasks on the > > _parent_ sched_domain even if a sched_group isn't overloaded. It is > > currently set on: > > > > 1. SMT level to promote spreading to sibling cores rather than using > > sibling HW-threads (caff37ef96eac7fe96a). > > > > 2. Non-NUMA levels which don't have the SD_SHARE_PKG_RESOURCES flag > > set (= DIE level in the default topology) as it was found to > > improve benchmarks on certain NUMA systems (6956dc568f34107f1d02b). > > So the goal is to have the last non-NUMA level with the flag so we can > spread between "DIE" The flag is already set on "DIE", the goal is to be consistent and set the flag consistently. As it is, it is set on SMT and DIE, but not MC: SMT: SD_PREFER_SIBLING MC: !SD_PREFER_SIBLING DIE: SD_PREFER_SIBLING So the goal is to spread consistently at all non-NUMA levels regardless of which levels are present as currently it is not that obvious when spreading happens on MC level as this list illustrates. > > > > > 3. Any non-NUMA level that inherits the flag due to elimination of > > its parent sched_domain level in the de-generate step of the > > sched_domain hierarchy set up (= MC level in the default > > topology). > > This is to ensure that the last non NUMA level has the flag when the > DIE one disappears but the goal is the same as 2. My point was just summarize the conditions for enabling spreading. It doesn't guarantee that the last non-NUMA level is spreading. For example, x86 numa-in-package topology doesn't have DIE at all, so there is nothing to inherit. Going the argument used to introduce the flag on DIE level, it should be set on MC for those systems. The whole thing really boils down to: Why do we want to not always spread at all levels inside a single node? It seems that we almost always do that already. So why not make it truly always? > > Preferring siblings seems to be a useful tweak for all non-NUMA levels, > > so we should enable it on all non-NUMA levels. As it is, it is possible > > to have it SMT and DIE, but not MC in between when using the default > > topology. > > So you want to extend it to all non NUMA level. And especially you > want to spread tasks in each MC groups when we have DIE and MC levels. > Have you got benchmark results to show improvement or is it just to > align topology configuration? It is purely to have consistent scheduling behaviour and make it clearer how system topology affect scheduling. SD_PREFER_SIBLING is currently a behavioral flag, which means that each architecture cannot modify it, it is set by a set of rules based on the number of levels used by the architecture and the topology flags set. So should an architecture have platform with two or more non-SMT non-NUMA levels it is currently not possible to make the scheduler spread on both levels. > The fact that this flag improves bench for SMT and NUMA level doesn't > mean that it will improve for MC level as well. We have the > wake_wide/wake_affine stuffs that tries to do similar thing > dynamically and it regularly improves/regresses benchmark like > sysbench or hackbench True. wake_wide/wake_affine in the wake-up path has some of the same behaviour as SD_PREFER_SIBLING has for periodic/idle/nohz balancing. However, that is already consistently enabled for all levels apart for NUMA levels where the reclaim distance is too high. So currently we do allow some spreading, but not consistently. I can't say that this patch won't cause any regressions. I was hoping someone would say that it is definitely a bad idea with a good reason, or we try it for the benefit of consistent flag setting. I'm happy to drop the patch if there is good explanation for why the flag is set like it is, but I think we have to make it controllable by each architecture then. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.