From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5529FC433ED for ; Thu, 13 May 2021 12:34:49 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CFB57613C8 for ; Thu, 13 May 2021 12:34:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CFB57613C8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:From: References:Cc:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=GiXZwvUYMAlcgb8wH17cjVOlpdOuJMwDUXPJKZfup7M=; b=p2g+GTq/ez+0On+GMgfrZkvoE 5Krje9tx3C5Zc+HjSwcm/OfeiY7aZmflfyVzWyMJaRgEaFl4Z0jQVLRMYCPj4zRF23OUM5vvE8aVr +1HzKq3JbWg6ZNUYaA7lcP/yVI82aU13mf/jZoX8BjQIIIYfwcPfDXZlpx6vlYWF9o1ySy6rtTZdt a05S0yTfLjsGu9mCC8P1N01m3ZYr9Les29FkIT9CBZz/AMqU9f4aEOkMUG+2e5iXPjeY3BWqvTuc1 sKmrYJUKlE+dlK0uCxGbn+2FrU1YF4R0nVJ6I8sFgWibESCpApJ3q8e8HeoqaMIjvjrjazav6K+Ei qKjs4urwQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lhAW9-005WRq-SG; Thu, 13 May 2021 12:32:42 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lhAW6-005WQw-OC for linux-arm-kernel@desiato.infradead.org; Thu, 13 May 2021 12:32:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:From:References:Cc:To: Subject:Sender:Reply-To:Content-ID:Content-Description; bh=0EWAMiMxEOI0xe8i/B+NUrNZZXZcE3h69VE5YnlxWMw=; b=LnJabest6dvdoLmS1jGTJ78Jgo nMsgdHsYZsGBKWLQ2+saDyr8mOjJC9bn401jjgfEahWzD+KImBaV6T9uzCIR6Xf8xK5wnoLUc/MCu piI47Py5BVqnaZERfX5tTBxXLYueWR/vDDdj0EtuMtbhX6H4cgSNPrCTz2ViOY+wMiI2UJOut+3Iq 8PvwtlpCZvxfVDSXHxnwh8IofYs/tfLmgD8PquSsMHl6al1J4wGlj5ZBpFBxetv2N0buvmQV/J2q8 3KRk8nBdd8ltiHLk3o0xEF5211NepF1bNRc66HU2c4YmRAQO8Vv6+Jtlu2vAaDslziKS8jheevc5t tufmVjBA==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lhAW3-00BFAm-HZ for linux-arm-kernel@lists.infradead.org; Thu, 13 May 2021 12:32:37 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 35DA0169E; Thu, 13 May 2021 05:32:27 -0700 (PDT) Received: from [192.168.178.6] (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3D7AE3F73B; Thu, 13 May 2021 05:32:22 -0700 (PDT) Subject: Re: [RFC PATCH v6 3/4] scheduler: scan idle cpu in cluster for tasks within one LLC To: "Song Bao Hua (Barry Song)" , Vincent Guittot Cc: "tim.c.chen@linux.intel.com" , "catalin.marinas@arm.com" , "will@kernel.org" , "rjw@rjwysocki.net" , "bp@alien8.de" , "tglx@linutronix.de" , "mingo@redhat.com" , "lenb@kernel.org" , "peterz@infradead.org" , "rostedt@goodmis.org" , "bsegall@google.com" , "mgorman@suse.de" , "msys.mizuma@gmail.com" , "valentin.schneider@arm.com" , "gregkh@linuxfoundation.org" , Jonathan Cameron , "juri.lelli@redhat.com" , "mark.rutland@arm.com" , "sudeep.holla@arm.com" , "aubrey.li@linux.intel.com" , "linux-arm-kernel@lists.infradead.org" , "linux-kernel@vger.kernel.org" , "linux-acpi@vger.kernel.org" , "x86@kernel.org" , "xuwei (O)" , "Zengtao (B)" , "guodong.xu@linaro.org" , yangyicong , "Liguozhu (Kenneth)" , "linuxarm@openeuler.org" , "hpa@zytor.com" References: <20210420001844.9116-1-song.bao.hua@hisilicon.com> <20210420001844.9116-4-song.bao.hua@hisilicon.com> <80f489f9-8c88-95d8-8241-f0cfd2c2ac66@arm.com> <8b5277d9-e367-566d-6bd1-44ac78d21d3f@arm.com> <185746c4d02a485ca8f3509439328b26@hisilicon.com> <4d1f063504b1420c9f836d1f1a7f8e77@hisilicon.com> <142c7192-cde8-6dbe-bb9d-f0fce21ec959@arm.com> From: Dietmar Eggemann Message-ID: <45cce983-79ca-392a-f590-9168da7aefab@arm.com> Date: Thu, 13 May 2021 14:32:20 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210513_053235_685549_38245B03 X-CRM114-Status: GOOD ( 18.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 07/05/2021 15:07, Song Bao Hua (Barry Song) wrote: > > >> -----Original Message----- >> From: Dietmar Eggemann [mailto:dietmar.eggemann@arm.com] [...] >> On 03/05/2021 13:35, Song Bao Hua (Barry Song) wrote: >> >> [...] >> >>>> From: Song Bao Hua (Barry Song) >> >> [...] >> >>>>> From: Dietmar Eggemann [mailto:dietmar.eggemann@arm.com] >> >> [...] >> >>>>> On 29/04/2021 00:41, Song Bao Hua (Barry Song) wrote: >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Dietmar Eggemann [mailto:dietmar.eggemann@arm.com] >>>>> >>>>> [...] >>>>> >>>>>>>>>> From: Dietmar Eggemann [mailto:dietmar.eggemann@arm.com] >>>>>>> >>>>>>> [...] >>>>>>> >>>>>>>>>> On 20/04/2021 02:18, Barry Song wrote: >> >> [...] >> >>> >>> On the other hand, according to "sched: Implement smarter wake-affine logic" >>> >> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/ >> ?id=62470419 >>> >>> Proper factor in wake_wide is mainly beneficial of 1:n tasks like >> postgresql/pgbench. >>> So using the smaller cluster size as factor might help make wake_affine false >> so >>> improve pgbench. >>> >>> From the commit log, while clients = 2*cpus, the commit made the biggest >>> improvement. In my case, It should be clients=48 for a machine whose LLC >>> size is 24. >>> >>> In Linux, I created a 240MB database and ran "pgbench -c 48 -S -T 20 pgbench" >>> under two different scenarios: >>> 1. page cache always hit, so no real I/O for database read >>> 2. echo 3 > /proc/sys/vm/drop_caches >>> >>> For case 1, using cluster_size and using llc_size will result in similar >>> tps= ~108000, all of 24 cpus have 100% cpu utilization. >>> >>> For case 2, using llc_size still shows better performance. >>> >>> tps for each test round(cluster size as factor in wake_wide): >>> 1398.450887 1275.020401 1632.542437 1412.241627 1611.095692 1381.354294 >> 1539.877146 >>> avg tps = 1464 >>> >>> tps for each test round(llc size as factor in wake_wide): >>> 1718.402983 1443.169823 1502.353823 1607.415861 1597.396924 1745.651814 >> 1876.802168 >>> avg tps = 1641 (+12%) >>> >>> so it seems using cluster_size as factor in "slave >= factor && master >= >> slave * >>> factor" isn't a good choice for my machine at least. >> >> So SD size = 4 (instead of 24) seems to be too small for `-c 48`. >> >> Just curious, have you seen the benefit of using wake wide on SD size = >> 24 (LLC) compared to not using it at all? > > At least in my benchmark made today, I have not seen any benefit to use > llc_size. Always returning 0 in wake_wide() seems to be much better. > > postgres@ubuntu:$pgbench -i pgbench > postgres@pgbench:$ pgbench -T 120 -c 48 pgbench > > using llc_size, it got to 123tps > always returning 0 in wake_wide(), it got to 158tps > > actually, I really couldn't reproduce the performance improvement > the commit "sched: Implement smarter wake-affine logic" mentioned. > on the other hand, the commit log didn't present the pgbench command > parameter used. I guess the benchmark result will highly depend on > the command parameter and disk I/O speed. I see. And it was a way smaller machine (12 CPUs) back then. You could run pgbench via mmtests https://github.com/gormanm/mmtests. I.e the `timed-ro-medium` test. mmtests# ./run-mmtests.sh --config ./configs/config-db-pgbench-timed-ro-medium test_tag /shellpacks/shellpack-bench-pgbench contains all the individual test steps. Something you could use as a template for your pgbench standalone tests as well. I ran this test on an Intel Xeon E5-2690 v2 with 40 CPUs and 64GB of memory on v5.12 vanilla and w/o wakewide. The test uses `scale_factor = 2570` on this machine. I guess this relates to ~41GB? At least this was the size of the: #mmtests/work/testdisk/data/pgdata directory when the test started. mmtests/work/log# ../../compare-kernels.sh --baseline base --compare wo_wakewide | grep ^Hmean #clients v5.12 vanilla v5.12 w/o wakewide Hmean 1 10903.88 ( 0.00%) 10792.59 * -1.02%* Hmean 6 28480.60 ( 0.00%) 27954.97 * -1.85%* Hmean 12 49197.55 ( 0.00%) 47758.16 * -2.93%* Hmean 22 72902.37 ( 0.00%) 71314.01 * -2.18%* Hmean 30 75468.16 ( 0.00%) 75929.17 * 0.61%* Hmean 48 60155.58 ( 0.00%) 60471.91 * 0.53%* Hmean 80 62202.38 ( 0.00%) 60814.76 * -2.23%* So there are some improvements w/ wakewide but nothing of the scale showed in the original wakewide patch. I'm not an expert on how to set up these pgbench tests though. So maybe other pgbench related mmtests configs or some more fine-grained tuning can produce bigger diffs? _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel