From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56311C433ED for ; Mon, 3 May 2021 06:21:48 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C793A61176 for ; Mon, 3 May 2021 06:21:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C793A61176 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=hisilicon.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References:Message-ID:Date: Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=9lfVYMIGPQqEyeWYeu+RYVuUY++rjJcelnujc7ZWCJ4=; b=la9+31DRvEoZokL9iEwLR/Grh fjvIfMVfJ1bNwMDi8shnqqvpGGmJwVXOSS5nyFyEcPKOZDVOh2o1vLgr4MeRzs22oJtlwDHDmlDZO gpEnKVrsAxU2CJjJ55+DGg0dkDFcNPQwhy7dU3UuDOMEV4JQSJvY+gVoWGPJ4cOYIqFh0zWeuiI7R KDl/qCrQXLoUGh2zTkiM0+m6aoY74v8F6dnlv5lpmoWt4+x7w5CRyV8vRWENZGMEru+GBhKlWpIis drSd21H8xBGkcvZYR0QlrwDhyGlPjkB6qr9fmTp/YIDdAg5Lfmw1IIzoNy5A3PSzi2Om/M43hNbpV I1dGaCXCw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1ldRvR-00DGrs-8c; Mon, 03 May 2021 06:19:26 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1ldRvK-00DGr9-5f for linux-arm-kernel@desiato.infradead.org; Mon, 03 May 2021 06:19:18 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=MIME-Version: Content-Transfer-Encoding:Content-Type:In-Reply-To:References:Message-ID:Date :Subject:CC:To:From:Sender:Reply-To:Content-ID:Content-Description; bh=Ev+I19ICApG784BP5d76VRbQ0egpo/+CVAUQaBVOm3s=; b=MJJancvNkrdDcBrQBkzIrACgPR l7b6JvxHefr0jXI4QvRMyp+WjobYDOU+VCjxYBIT4dZPp6iTg0WEmrI7z5gaXwtXQD7TXfcRzyxvY GBl66wNPHzHgRvaaE+sJYL3WLY8uHclLvO7M2TEqLbKHMx6clx1iSKYXEzTN5QQ1NIo9x4J4UcaHg awl4f+kYaySMb06TjbkLq5dCjCsk0VewVpnMDNzS/EAWGsZjClSJBEfsufXs/q7CtBTibsmyFbTKR eJtJTe4yyfX/frxr8iNfr2AuF+d1rvwxshhtVZDFkJpx5nZ57NLHu2mzH3QmV30OYtcI51Xf2qjWZ UR3KB/Ng==; Received: from frasgout.his.huawei.com ([185.176.79.56]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1ldRvG-002rXu-RX for linux-arm-kernel@lists.infradead.org; Mon, 03 May 2021 06:19:16 +0000 Received: from fraeml713-chm.china.huawei.com (unknown [172.18.147.201]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4FYXb16vn4z6tm7b; Mon, 3 May 2021 14:08:13 +0800 (CST) Received: from lhreml718-chm.china.huawei.com (10.201.108.69) by fraeml713-chm.china.huawei.com (10.206.15.32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Mon, 3 May 2021 08:19:04 +0200 Received: from dggemi761-chm.china.huawei.com (10.1.198.147) by lhreml718-chm.china.huawei.com (10.201.108.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.2176.2; Mon, 3 May 2021 07:19:02 +0100 Received: from dggemi761-chm.china.huawei.com ([10.9.49.202]) by dggemi761-chm.china.huawei.com ([10.9.49.202]) with mapi id 15.01.2176.012; Mon, 3 May 2021 14:19:00 +0800 From: "Song Bao Hua (Barry Song)" To: Dietmar Eggemann , Vincent Guittot CC: "tim.c.chen@linux.intel.com" , "catalin.marinas@arm.com" , "will@kernel.org" , "rjw@rjwysocki.net" , "bp@alien8.de" , "tglx@linutronix.de" , "mingo@redhat.com" , "lenb@kernel.org" , "peterz@infradead.org" , "rostedt@goodmis.org" , "bsegall@google.com" , "mgorman@suse.de" , "msys.mizuma@gmail.com" , "valentin.schneider@arm.com" , "gregkh@linuxfoundation.org" , Jonathan Cameron , "juri.lelli@redhat.com" , "mark.rutland@arm.com" , "sudeep.holla@arm.com" , "aubrey.li@linux.intel.com" , "linux-arm-kernel@lists.infradead.org" , "linux-kernel@vger.kernel.org" , "linux-acpi@vger.kernel.org" , "x86@kernel.org" , "xuwei (O)" , "Zengtao (B)" , "guodong.xu@linaro.org" , yangyicong , "Liguozhu (Kenneth)" , "linuxarm@openeuler.org" , "hpa@zytor.com" Subject: RE: [RFC PATCH v6 3/4] scheduler: scan idle cpu in cluster for tasks within one LLC Thread-Topic: [RFC PATCH v6 3/4] scheduler: scan idle cpu in cluster for tasks within one LLC Thread-Index: AQHXPa2htgkN1X7dCEatlQZ1LQ756arRBrBQ Date: Mon, 3 May 2021 06:19:00 +0000 Message-ID: <51df51d84c764d9292146e11d1031b08@hisilicon.com> References: <20210420001844.9116-1-song.bao.hua@hisilicon.com> <20210420001844.9116-4-song.bao.hua@hisilicon.com> <80f489f9-8c88-95d8-8241-f0cfd2c2ac66@arm.com> <8b5277d9-e367-566d-6bd1-44ac78d21d3f@arm.com> <185746c4d02a485ca8f3509439328b26@hisilicon.com> In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.126.203.132] MIME-Version: 1.0 X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210502_231915_316642_4C9C30E5 X-CRM114-Status: GOOD ( 26.68 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org > -----Original Message----- > From: Dietmar Eggemann [mailto:dietmar.eggemann@arm.com] > Sent: Friday, April 30, 2021 10:43 PM > To: Song Bao Hua (Barry Song) ; Vincent Guittot > > Cc: tim.c.chen@linux.intel.com; catalin.marinas@arm.com; will@kernel.org; > rjw@rjwysocki.net; bp@alien8.de; tglx@linutronix.de; mingo@redhat.com; > lenb@kernel.org; peterz@infradead.org; rostedt@goodmis.org; > bsegall@google.com; mgorman@suse.de; msys.mizuma@gmail.com; > valentin.schneider@arm.com; gregkh@linuxfoundation.org; Jonathan Cameron > ; juri.lelli@redhat.com; mark.rutland@arm.com; > sudeep.holla@arm.com; aubrey.li@linux.intel.com; > linux-arm-kernel@lists.infradead.org; linux-kernel@vger.kernel.org; > linux-acpi@vger.kernel.org; x86@kernel.org; xuwei (O) ; > Zengtao (B) ; guodong.xu@linaro.org; yangyicong > ; Liguozhu (Kenneth) ; > linuxarm@openeuler.org; hpa@zytor.com > Subject: Re: [RFC PATCH v6 3/4] scheduler: scan idle cpu in cluster for tasks > within one LLC > > On 29/04/2021 00:41, Song Bao Hua (Barry Song) wrote: > > > > > >> -----Original Message----- > >> From: Dietmar Eggemann [mailto:dietmar.eggemann@arm.com] > > [...] > > >>>>> From: Dietmar Eggemann [mailto:dietmar.eggemann@arm.com] > >> > >> [...] > >> > >>>>> On 20/04/2021 02:18, Barry Song wrote: > > [...] > > > Though we will never go to slow path, wake_wide() will affect want_affine, > > so eventually affect the "new_cpu"? > > yes. > > > > > for_each_domain(cpu, tmp) { > > /* > > * If both 'cpu' and 'prev_cpu' are part of this domain, > > * cpu is a valid SD_WAKE_AFFINE target. > > */ > > if (want_affine && (tmp->flags & SD_WAKE_AFFINE) && > > cpumask_test_cpu(prev_cpu, sched_domain_span(tmp))) { > > if (cpu != prev_cpu) > > new_cpu = wake_affine(tmp, p, cpu, prev_cpu, sync); > > > > sd = NULL; /* Prefer wake_affine over balance flags */ > > break; > > } > > > > if (tmp->flags & sd_flag) > > sd = tmp; > > else if (!want_affine) > > break; > > } > > > > If wake_affine is false, the above won't execute, new_cpu(target) will > > always be "prev_cpu"? so when task size > cluster size in wake_wide(), > > this means we won't pull the wakee to the cluster of waker? It seems > > sensible. > > What is `task size` here? > > The criterion is `!(slave < factor || master < slave * factor)` or > `slave >= factor && master >= slave * factor` to wake wide. > Yes. For "task size", I actually mean a bundle of waker-wakee tasks which can make "slave >= factor && master >= slave * factor" either true or false, then change the target cpu where we are going to scan from. Now since I have moved to cluster level when tasks have been in same LLC level, it seems it would be more sensible to use "cluster_size" as factor? > I see that since you effectively change the sched domain size from LLC > to CLUSTER (e.g. 24->6) for wakeups with cpu and prev_cpu sharing LLC > (hence the `numactl -N 0` in your workload), wake_wide() has to take > CLUSTER size into consideration. > > I was wondering if you saw wake_wide() returning 1 with your use cases: > > numactl -N 0 /usr/lib/lmbench/bin/stream -P [6,12] -M 1024M -N 5 I couldn't make wake_wide return 1 by the above stream command. And I can't reproduce it by a 1:1(monogamous) hackbench "-f 1". But I am able to reproduce this issue by a M:N hackbench, for example: numactl -N 0 hackbench -p -T -f 10 -l 20000 -g 1 hackbench will create 10 senders which will send messages to 10 receivers. (Each sender can send messages to all 10 receivers.) I've often seen flips like: waker wakee 1501 39 1509 17 11 1320 13 2016 11, 13, 17 is smaller than LLC but larger than cluster. So the wake_wide() using cluster factor will return 1, on the other hand, if we always use llc_size as factor, it will return 0. However, it seems the change in wake_wide() could bring some negative influence to M:N relationship(-f 10) according to tests made today by: numactl -N 0 hackbench -p -T -f 10 -l 20000 -g $1 g = 1 2 3 4 cluster_size 0.5768 0.6578 0.8117 1.0119 LLC_size 0.5479 0.6162 0.6922 0.7754 Always using llc_size as factor in wake_wide still shows better result in the 10:10 polygamous hackbench. So it seems the `slave >= factor && master >= slave * factor` isn't a suitable criterion for cluster size? Thanks Barry _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel