From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 364CCC433F5 for ; Mon, 14 Mar 2022 16:59:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=bKBe3CBCVRPQfLekGpwd0tZxDZgCpr0BMA4/Smd7cNU=; b=V5rdW4LVgBKSEp DVfFT4I20BMkPpL9iz6pZduZRu4ssoSdrPSDTffGoCu+S8NblmXY0hZrRZILF0oI+fCThQdimaXUV 05O+lh9TopROWf2KFzw49+l6n9LNJixplOmhdTTSc8JQr7cLlT+vuGwrVZN3LnleEmB6tAxThYsMD kw0eFZk1oVZBkSDnpEvNtt2cdOyjFYmtoMv6aDUhmiVRfpLYi8p7lnwfKAd6O5upYXXD3y6x6DuNS Y2wpVwsKlB9B+jztnQ1xDe4olm9U9cUtkl6OokyZtfTKsN3vDHuJGKFgvpwhBgcA5KROJ4tlFs4VY NWdsM+WhEPpxtNDyp8vA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTo0x-006CQG-DK; Mon, 14 Mar 2022 16:57:48 +0000 Received: from mail-bn7nam10on20706.outbound.protection.outlook.com ([2a01:111:f400:7e8a::706] helo=NAM10-BN7-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nTny7-006BDS-Io for linux-arm-kernel@lists.infradead.org; Mon, 14 Mar 2022 16:54:53 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KSRlTpkyPghqAorBPFGpZqFe8QGHwSpqEgnb6J8IIW7gt+ruvM0XOtetWL0x14s6+i/kn5vg0dYAxTd0Ouke73fJJOal7D8YN/pTid3QtCHdV+GGHTTrTMuXagvqH18pjlYn//Pbh71HGb6wYoE5XZk/TxYJxj0SeCj/sQTv+GJrB1GvBFICv8w22vKlPh/xpuCKr6cGY13h4BQk36KxHHy9ogralAGOm+u9y/kkqka2qJ5OX1N4h86HRHzLExYWXg4BihONfZ3ustHeFmC70BKVZWLnUl6VPTtd5ieaFUBVIuvrn8hQ2SRjdD/oR6EOXI89y3Pb38SslZqttdHW3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6+4DCNsbRr2l4aD6ESQNFY8UhlzPBu/4e8bB/iVmgWA=; b=FIlRs5ev2/ypuZLHbUPHrtBZijse2vkozKCuqFUlTI9m0/rj6Q7y0jWItBfmutEs0Xb5tJB4Qopc/p3f64caqUh9NQDA8HpbnSxkseQANRolBk62ylssGlUSvp3qcbjT88nFF/uyQgRF0KrCDkrQig+NySLaoPZUqXoag69/yLEvQ/+rHfg/tcMt7yDdYMyd5psqX+/OqKfviRjoDg7KcqB0u0iuDzznLhdTe5O4h+qm7sJOlGa6d7qAxY2aNaBiFZtUOeWEI+OCbJ8fZHdC0ZDHc2obt9xHQ6pEomlix3wJwTvtHYTTgVOnEXPkAW6s9rGL3OK9Om5EaloTuXLg+g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6+4DCNsbRr2l4aD6ESQNFY8UhlzPBu/4e8bB/iVmgWA=; b=Q/NoaxuuXeT1uZAJDcTXatAw7bj2ZDCqF1fJONfp4CjvRivrb9abPNVKeJIzwEvCSXgJqtrietou7BPgax61c7+KHa8Dvod4PyUVWpCb8CF80pJAVDOhCufMtu8MOC57jucqO/8SPkcHUuSmt+d4AVsj/yB963I37YSFUEsaWB4= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from SA0PR01MB6329.prod.exchangelabs.com (2603:10b6:806:ee::12) by BYAPR01MB4710.prod.exchangelabs.com (2603:10b6:a03:83::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5061.22; Mon, 14 Mar 2022 16:54:46 +0000 Received: from SA0PR01MB6329.prod.exchangelabs.com ([fe80::f56a:e18f:b6c4:ddb5]) by SA0PR01MB6329.prod.exchangelabs.com ([fe80::f56a:e18f:b6c4:ddb5%9]) with mapi id 15.20.5061.028; Mon, 14 Mar 2022 16:54:45 +0000 Date: Mon, 14 Mar 2022 09:54:42 -0700 From: Darren Hart To: Dietmar Eggemann Cc: Vincent Guittot , Will Deacon , LKML , Linux Arm , Sudeep Holla , Greg Kroah-Hartman , "Rafael J. Wysocki" , Catalin Marinas , Peter Zijlstra , Barry Song , Valentin Schneider , "D . Scott Phillips" , Ilkka Koskinen , stable@vger.kernel.org Subject: Re: [PATCH v3] topology: make core_mask include at least cluster_siblings Message-ID: References: <20220308103012.GA31267@willie-the-truck> <7ac47c67-0b5e-5caa-20bb-a0100a0cb78f@arm.com> <9398d7ad-30e7-890a-3e18-c3011c383585@arm.com> Content-Disposition: inline In-Reply-To: <9398d7ad-30e7-890a-3e18-c3011c383585@arm.com> X-ClientProxiedBy: CH2PR05CA0043.namprd05.prod.outlook.com (2603:10b6:610:38::20) To SA0PR01MB6329.prod.exchangelabs.com (2603:10b6:806:ee::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 525aedb0-8001-40eb-f5e1-08da05db57d4 X-MS-TrafficTypeDiagnostic: BYAPR01MB4710:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: n6SWHtYx4Wu+SS/aIW16JSeunovYI70+QyScKUMul7W3YcFdCt2J547TA5KXvFhuIB2G4JnhCWrntQacGE98yxjBefuRHShqynz79aHKGOVXXwePbnF2wnXG2CZvNtlDENo7lR4RQQ1TGsS6V7oCgI8FNS6YdXVKaXqzmzuOmBQe1V5FRJ1uVsR91LZ6izd+Q/RxEHxF7aHjxJFOcuAtEfIXyH4NRiFGCql233AHqMH7/5YqV3Aut+u4sDWnV2RMGzCCO652nbdEhp0lSS69OY+TmMUj425Jj8PmFdU9utVnfgTMt/t/DObGEe9XF1PfezUtiGo9uHvA8I5fsEO6Sv8aqNNOsi0xos1P6z9hmz2n/XfYfstzW/b3UeJm5d5BxbpEFVfkPgBZcf46D8ZqbMbEHgROTgHQQkpXBTXVSTgssVei/tAFAgkpRk/Ifi41XgMFojzK+07OM9Womm2qAZgr3lsnRDDrPJWSesp2bcF/df0D7QPUS0lzxmr0AGqRc0nRC1y1ArcmRxHa6nAW0oKL3QTgZRClmb3eOSX6pseJ00Htx8Q0NgssSTJ6XVqqqgXfdOE6gfq8UZRKgxAMj1iB4soG5DgDynCXKInRX1Bjbs7eZUCeem4Jt15MOcSBG3xigygWBLcG7AzcVtniQ+cgR6SYAPIocmJOBInDcryjIITHWxgpgJSgWCuZzmQfznlPwnBunHknNoZJMST+3A== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SA0PR01MB6329.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230001)(4636009)(7916004)(366004)(5660300002)(7416002)(66946007)(66556008)(66476007)(4326008)(8676002)(54906003)(6916009)(83380400001)(316002)(8936002)(33716001)(6512007)(86362001)(6666004)(26005)(186003)(508600001)(53546011)(52116002)(9686003)(38350700002)(6486002)(6506007)(2906002)(38100700002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?UV8l0cV2BPXhEAqiokoE3DXgaT6Aj4OHycIIOhIhu58HHfff7YgY59BtLeR7?= =?us-ascii?Q?dEMGBQAERGqYQIdEpVcgLm8vTpGDA8kyzdKx3+iLHwoShFFaVylTXwjAvptd?= =?us-ascii?Q?ugu5MIqGn2S66VNm2fV4Mz99+1M+h4Ei8UauxJKm/jCfC+k7l4oaW6kjKQe1?= =?us-ascii?Q?e0jm09XKhb9NCM19xEFDCH4lcEBGhLr9OUp82xFDtKk6FUdFLy0Xm6jyccz3?= =?us-ascii?Q?LugXg8Y/sKMQIkzDm+AHaP4nb5B8anmTqEaGjQ43+5MWYLMNamV2/E9GIdQW?= =?us-ascii?Q?Kuni49NFMfR1bXanBP9uap1TnuzqNk6jfkHcpxBsS9ogUaIQFXP78kqg7t05?= =?us-ascii?Q?A0mic6wS+p8r0MkpUS1Zhujntd/AmIp2LJh6bVhFwbuECJKSQVut27M/4VV8?= =?us-ascii?Q?0kcwX4te5z9ERsHNvqHthMvzF0Qc8reoC3rRD4PUs6PeX3/VDVCw67+IEXux?= =?us-ascii?Q?ZJgTcwUNd7TfR4n3or/mlU3UFnHdBt/t0kYax8N24k3z4dh4xUBNhZIh+AaR?= =?us-ascii?Q?/tH1YSeB1ytwoWvrJoFqDv7nOo/akLlCfTnsBvfgxIW1p3jvhutq/+zVWqlC?= =?us-ascii?Q?szPRC1aEbDGCEyAZHgzt4uj0YMylftYZmw9i5VYDvyyZ/DR/0Bad5so6xQ1O?= =?us-ascii?Q?Xi1Zs+g2tHN5j2HEY313nmiyAH+VwmoXkeshNcz6YBiMh83nxEv6NAJr9vLQ?= =?us-ascii?Q?YGu83eNEx4Tu7sjT+VyEXh7x7WMnNIr0P+oYyJwdWShx5Fq7l+CB+3cDILV8?= =?us-ascii?Q?MCO3YhDvg9eyAhEAO3L0pyQKLi6eYjhuia7JP4GSg5UiRgXV6ErpoKbZeXsP?= =?us-ascii?Q?AxTzc1VrWjjfFgJjVAZMplZiL5SrTTGn4nSGxqf8pgNZifPF1mrpS9E4VdWz?= =?us-ascii?Q?cJ7HzZkJ2k07sV5LXpUfs//0hMyvFJFi/cOzwzL4A78s0YJnqcuj5e78pdsU?= =?us-ascii?Q?eUBxZiYvUweylFenjhJeq653fQnrMEVjNppKFq+bRU48osWo80LozxuLejBH?= =?us-ascii?Q?/UWYCYwuqh7vkcwgsoWl00vppOdA0ZdhP/rFRoNMIeB6+OqcpLY8zZsvr6dL?= =?us-ascii?Q?lynlIcq1y6TogBfxz14r3rpFdatsTnkPzQFqaOYmttZVsGNJxx264Kxb4gtA?= =?us-ascii?Q?98sBaemB50BlryQXYcvVi7PUIFy4ZLvYDFpbGSAHBv+OMrQ/FNcLydfNgkk8?= =?us-ascii?Q?3j8cxD1Dsk+dlECz9yh0X0aZs3yGONtJ6EXUJdZAIwQJHc26CV8FQm5QFCPz?= =?us-ascii?Q?SAlx/YlIC+vZjGV6nvKlGk5GYCKda95OFGYJ7ZUbEfRC8TWfO22R9x/X8I8V?= =?us-ascii?Q?DMCO1ROZBSaaOD0jt8bt1f8Lx84TBHyikYOhj+Npyzr0208R06BQ4hxQuSkn?= =?us-ascii?Q?JHj69UzuZTNDxsvFZh2AFVIcm1kc9XIftcS4xqTMI0zVumik3YDiWezyhrhO?= =?us-ascii?Q?d1lcKhuorrBNGLTzeEpP+xk91Iw07lxfWqwkkJoovWzbQESqOxLcOFwT+zE5?= =?us-ascii?Q?8rs2vOlnOzCofPI=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 525aedb0-8001-40eb-f5e1-08da05db57d4 X-MS-Exchange-CrossTenant-AuthSource: SA0PR01MB6329.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Mar 2022 16:54:45.8709 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 0+wZNKKrZ5Vw01Ax5TnoK3hjiaVSyNV1KqlwL+2MRA3DBiXM0giiidpkpPPjUZ5nZHdvUlnMewJDm7MbH0tygPjnh3cxAZUWG+wVQHRrqoGeqa9SVI3SM+UUq5hxqe8T X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR01MB4710 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220314_095451_749308_6330D575 X-CRM114-Status: GOOD ( 44.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Mar 14, 2022 at 05:35:05PM +0100, Dietmar Eggemann wrote: > On 09/03/2022 19:26, Darren Hart wrote: > > On Wed, Mar 09, 2022 at 01:50:07PM +0100, Dietmar Eggemann wrote: > >> On 08/03/2022 18:49, Darren Hart wrote: > >>> On Tue, Mar 08, 2022 at 05:03:07PM +0100, Dietmar Eggemann wrote: > >>>> On 08/03/2022 12:04, Vincent Guittot wrote: > >>>>> On Tue, 8 Mar 2022 at 11:30, Will Deacon wrote: > > [...] > > >>>> I do not have any better idea than this tweak here either in case the > >>>> platform can't provide a cleaner setup. > >>> > >>> I'd argue The platform is describing itself accurately in ACPI PPTT > >>> terms. The topology doesn't fit nicely within the kernel abstractions > >>> today. This is an area where I hope to continue to improve things going > >>> forward. > >> > >> I see. And I assume lying about SCU/LLC boundaries in ACPI is not an > >> option since it messes up /sys/devices/system/cpu/cpu0/cache/index*/. > >> > >> [...] > > > > I'm not aware of a way to accurately describe the SCU topology in the PPTT, and > > the risk we run with lying about LLC topology is that lie has to be comprehended > > by all OSes and not conflict with other lies people may ask for. In general, I > > think it is preferable and more maintainable to describe the topology as > > accurately and honestly as we can within the existing platform mechanisms (PPTT, > > HMAT, etc) and work on the higher level abstractions to accommodate a broader > > set of topologies as they emerge (as well as working to more fully describe the > > topology with new platform level mechanisms as needed). > > > > As I mentioned, I intend to continue looking in to how to improve the current > > abstractions. For now, it sounds like we have agreement that this patch can be > > merged to address the BUG? > > What about swapping the CLS and MC cpumasks for such a machine? This > would avoid that the task scheduler has to deal with a system which has > CLS but no MC. We essentially promote the CLS cpumask up to MC in this > case. > > cat /sys/kernel/debug/sched/domains/cpu0/domain*/name > MC > ^^ > DIE > NUMA > > cat /sys/kernel/debug/sched/domains/cpu0# cat domain*/flags > SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_SHARE_PKG_RESOURCES SD_PREFER_SIBLING > ^^^^^^^^^^^^^^^^^^^^^^ > SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_PREFER_SIBLING > SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_SERIALIZE SD_OVERLAP SD_NUMA > > Only very lightly tested on Altra and Juno-r0 (DT). > > --->8--- > > From 54bef59e7f50fa41b7ae39190fd71af57209c27d Mon Sep 17 00:00:00 2001 > From: Dietmar Eggemann > Date: Mon, 14 Mar 2022 15:08:23 +0000 > Subject: [PATCH] arch_topology: Swap MC & CLS SD mask if MC weight==1 & > subset(MC,CLS) > > This avoids the issue of having a system with a CLS SD but no MC SD. > CLS should be sub-SD of MC. Hi Dietmar, Ultimately, this delivers the same result. I do think it imposes more complexity for everyone to address what as far as I'm aware only affect the one system. I don't think the term "Cluster" has a clear and universally understood definition, so I don't think it's a given that "CLS should be sub-SD of MC". I think this has been assumed, and that assumption has mostly held up, but this is an abstraction, and the abstraction should follow the physical topologies rather than the other way around in my opinion. If that's the primary motivation for this approach, I don't think it justifies the additional complexity. All told, I prefer the 2 line change contained within cpu_coregroup_mask() which handles the one known exception with minimal impact. It's easy enough to come back to this to address more cases with a more complex solution if needed in the future - but I prefer to introduce the least amount of complexity as possible to address the known issues, especially if the end result is the same and the cost is paid by the affected systems. Thanks, > > The cpumask under /sys/devices/system/cpu/cpu*/cache/index* and > /sys/devices/system/cpu/cpu*/topology are not changed by this. > > Signed-off-by: Dietmar Eggemann > --- > drivers/base/arch_topology.c | 30 ++++++++++++++++++++++++++++-- > 1 file changed, 28 insertions(+), 2 deletions(-) > > diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c > index 976154140f0b..9af90a5625c7 100644 > --- a/drivers/base/arch_topology.c > +++ b/drivers/base/arch_topology.c > @@ -614,7 +614,7 @@ static int __init parse_dt_topology(void) > struct cpu_topology cpu_topology[NR_CPUS]; > EXPORT_SYMBOL_GPL(cpu_topology); > > -const struct cpumask *cpu_coregroup_mask(int cpu) > +const struct cpumask *_cpu_coregroup_mask(int cpu) > { > const cpumask_t *core_mask = cpumask_of_node(cpu_to_node(cpu)); > > @@ -631,11 +631,37 @@ const struct cpumask *cpu_coregroup_mask(int cpu) > return core_mask; > } > > -const struct cpumask *cpu_clustergroup_mask(int cpu) > +const struct cpumask *_cpu_clustergroup_mask(int cpu) > { > return &cpu_topology[cpu].cluster_sibling; > } > > +static int > +swap_masks(const cpumask_t *core_mask, const cpumask_t *cluster_mask) > +{ > + if (cpumask_weight(core_mask) == 1 && > + cpumask_subset(core_mask, cluster_mask)) > + return 1; > + > + return 0; > +} > + > +const struct cpumask *cpu_coregroup_mask(int cpu) > +{ > + const cpumask_t *cluster_mask = _cpu_clustergroup_mask(cpu); > + const cpumask_t *core_mask = _cpu_coregroup_mask(cpu); > + > + return swap_masks(core_mask, cluster_mask) ? cluster_mask : core_mask; > +} > + > +const struct cpumask *cpu_clustergroup_mask(int cpu) > +{ > + const cpumask_t *cluster_mask = _cpu_clustergroup_mask(cpu); > + const cpumask_t *core_mask = _cpu_coregroup_mask(cpu); > + > + return swap_masks(core_mask, cluster_mask) ? core_mask : cluster_mask; > +} > + > void update_siblings_masks(unsigned int cpuid) > { > struct cpu_topology *cpu_topo, *cpuid_topo = &cpu_topology[cpuid]; > -- > 2.25.1 -- Darren Hart Ampere Computing / OS and Kernel _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel