From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 838DAC63777 for ; Thu, 3 Dec 2020 09:40:47 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 060F520872 for ; Thu, 3 Dec 2020 09:40:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 060F520872 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From:In-Reply-To: References:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=5fqwalFGFlRKWxlg3srf4kF0qegD2M+h1ajF3km+aCo=; b=YBEOu4+zYodEM1opGaiTvBmU4 nZ1/IlymfEdt/SNSRu7F5J/BZR+l1+AWnasqkSf2dMoiT4KAdL7vKGSrOyAV3YSuMl5cegPNml6P7 o1cdlITy2nH17contvi/kwhKLmVZlAzyNAqlqqRY3RGCdWRuR+3APzo8Lxru5KbOFFceLxBpglikS fVYkxqqX9U1ntqSLc17PjzaMi+E01YfqMyxDQbh7PuF+38FkhCpz82/gm4G9s/cdHigxYsb8/QFK5 XPrxkXlL5Aw+vUkHf45dlEtb3PvXoCAJ9Kk7rXZj2162lW66ET19pzeq+2u4LiCQKgqAsQjWF5ofi tdQupiDvw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkl5B-0006GB-KT; Thu, 03 Dec 2020 09:39:25 +0000 Received: from mail-lj1-x242.google.com ([2a00:1450:4864:20::242]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkl57-0006Ej-QF for linux-arm-kernel@lists.infradead.org; Thu, 03 Dec 2020 09:39:24 +0000 Received: by mail-lj1-x242.google.com with SMTP id y16so1836514ljk.1 for ; Thu, 03 Dec 2020 01:39:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=87JXtw/NuImj4CEAwcS3K0+LyEdyqYc6Ik1YBJGXgHQ=; b=ncAP/4WR3lka3FQV72J3WYpVAM9JcUNBy4yuvRt4oNx4TneevIQByJTgtvfDNqhNRS UhC1GHbWB7yQ3KmyYzHEaz1Hz0dChlFEOi+WCIlC6DzfBbJMoA6/OxFk8FQ/lYjQkW+p xYqbuYmQJ8FuisU3PaVDl1RCGeDNfbl+KjK45jSim2+cjZiBpkz+e3pfXsTxq7o86sDh AMQobK96jPwkWv2DVTymeH8ZHKPSnzRlAWzAsdIltZhDn3PS6Zybd9ikxjCSVVOXKe0T V4OkiPDVQeH9vNu8ymPsa9nk9xawZFhrhqRZd85cizxsKbB9sy9+Hs7TelFr3liYOynX XBYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=87JXtw/NuImj4CEAwcS3K0+LyEdyqYc6Ik1YBJGXgHQ=; b=NjVVUqB7arAsFJspiomFUtS1r03AVqjMUYUqxG1L1CN0sapADykvhaHY/RkLF/MPUf 6CkH1Gfx8hx6kILmDSqh54vMTh+9+Tbf1LxmHyWdDgu+Ti+WarWmq2AoPa5jdG1GaTcJ thUsklrCkVKjxGZ2O98M9+jZhRTk+ADFB810U7FPwCRWEz7Dt54QfqE5AyvJpPIYRfhu vcnbZI8fB4q5QN0FM0kiWamM44NjOSK+WD5+7K7nKlphZFtoBnpK3ELYkaSHzyrtCLmy acvuPqcJDoEX/hRsdM+/xcYUPKlhUSPVjNMvKwe5XxumfjnzhpzksBK5xgnsn9YQULHp YY3w== X-Gm-Message-State: AOAM531Hs0ubr74j5gM4tqVWqRSvC35DEXlGNUlHaI/EvhvivRJ9HtB+ qFDYqTM61+VDnY4ScZhXm5MWaZt8Ha/PzI9iaH+kAw== X-Google-Smtp-Source: ABdhPJyhx4nIcs643IMbl0F4Dj7bkGFKz8NBBvEA0cDs5cb7dKUi4K5TnYSyPZJ1oHzhBarqJ7Sm79DATF/DhoNyG7U= X-Received: by 2002:a2e:8050:: with SMTP id p16mr911186ljg.69.1606988360003; Thu, 03 Dec 2020 01:39:20 -0800 (PST) MIME-Version: 1.0 References: <20201201025944.18260-1-song.bao.hua@hisilicon.com> <20201201025944.18260-3-song.bao.hua@hisilicon.com> <414fbd167b214452b925ac674575f0d6@hisilicon.com> In-Reply-To: From: Vincent Guittot Date: Thu, 3 Dec 2020 10:39:08 +0100 Message-ID: Subject: Re: [RFC PATCH v2 2/2] scheduler: add scheduler level for clusters To: "Song Bao Hua (Barry Song)" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201203_043921_913216_2D182D53 X-CRM114-Status: GOOD ( 25.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Juri Lelli , Mark Rutland , "Zengtao \(B\)" , Peter Zijlstra , Catalin Marinas , Jonathan Cameron , "Rafael J. Wysocki" , linux-kernel , Steven Rostedt , Dietmar Eggemann , Ben Segall , ACPI Devel Maling List , Ingo Molnar , Linuxarm , Mel Gorman , "xuwei \(O\)" , "gregkh@linuxfoundation.org" , Will Deacon , Valentin Schneider , LAK , "Cc: Len Brown" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, 3 Dec 2020 at 10:11, Song Bao Hua (Barry Song) wrote: > > > > > -----Original Message----- > > From: Vincent Guittot [mailto:vincent.guittot@linaro.org] > > Sent: Thursday, December 3, 2020 10:04 PM > > To: Song Bao Hua (Barry Song) > > Cc: Valentin Schneider ; Catalin Marinas > > ; Will Deacon ; Rafael J. Wysocki > > ; Cc: Len Brown ; > > gregkh@linuxfoundation.org; Jonathan Cameron ; > > Ingo Molnar ; Peter Zijlstra ; Juri > > Lelli ; Dietmar Eggemann ; > > Steven Rostedt ; Ben Segall ; Mel > > Gorman ; Mark Rutland ; LAK > > ; linux-kernel > > ; ACPI Devel Maling List > > ; Linuxarm ; xuwei (O) > > ; Zengtao (B) > > Subject: Re: [RFC PATCH v2 2/2] scheduler: add scheduler level for clusters > > > > On Wed, 2 Dec 2020 at 21:58, Song Bao Hua (Barry Song) > > wrote: > > > > > > > > > > > Sorry. Please ignore this. I added some printk here while testing > > > > one numa. Will update you the data in another email. > > > > > > Re-tested in one NUMA node(cpu0-cpu23): > > > > > > g=1 > > > Running in threaded mode with 1 groups using 40 file descriptors > > > Each sender will pass 100000 messages of 100 bytes > > > w/o: 7.689 7.485 7.485 7.458 7.524 7.539 7.738 7.693 7.568 7.674=7.5853 > > > w/ : 7.516 7.941 7.374 7.963 7.881 7.910 7.420 7.556 7.695 7.441=7.6697 > > > w/ but dropped select_idle_cluster: > > > 7.752 7.739 7.739 7.571 7.545 7.685 7.407 7.580 7.605 7.487=7.611 > > > > > > g=2 > > > Running in threaded mode with 2 groups using 40 file descriptors > > > Each sender will pass 100000 messages of 100 bytes > > > w/o: 10.127 10.119 10.070 10.196 10.057 10.111 10.045 10.164 10.162 > > > 9.955=10.1006 > > > w/ : 9.694 9.654 9.612 9.649 9.686 9.734 9.607 9.842 9.690 9.710=9.6878 > > > w/ but dropped select_idle_cluster: > > > 9.877 10.069 9.951 9.918 9.947 9.790 9.906 9.820 9.863 9.906=9.9047 > > > > > > g=3 > > > Running in threaded mode with 3 groups using 40 file descriptors > > > Each sender will pass 100000 messages of 100 bytes > > > w/o: 15.885 15.254 15.932 15.647 16.120 15.878 15.857 15.759 15.674 > > > 15.721=15.7727 > > > w/ : 14.974 14.657 13.969 14.985 14.728 15.665 15.191 14.995 14.946 > > > 14.895=14.9005 > > > w/ but dropped select_idle_cluster: > > > 15.405 15.177 15.373 15.187 15.450 15.540 15.278 15.628 15.228 > > 15.325=15.3591 > > > > > > g=4 > > > Running in threaded mode with 4 groups using 40 file descriptors > > > Each sender will pass 100000 messages of 100 bytes > > > w/o: 20.014 21.025 21.119 21.235 19.767 20.971 20.962 20.914 21.090 > > 21.090=20.8187 > > > w/ : 20.331 20.608 20.338 20.445 20.456 20.146 20.693 20.797 21.381 > > 20.452=20.5647 > > > w/ but dropped select_idle_cluster: > > > 19.814 20.126 20.229 20.350 20.750 20.404 19.957 19.888 20.226 > > 20.562=20.2306 > > > > > > > I assume that you have run this on v5.9 as previous tests. > > Yep > > > The results don't show any real benefit of select_idle_cluster() > > inside a node whereas this is where we could expect most of the > > benefit. We have to understand why we have such an impact on numa > > tests only. > > There is a 4-5.5% increase while g=2 and g=3. my point was with vs without select_idle_cluster() but still having a cluster domain level In this case, the diff is -0.8% for g=1 +2.2% for g=2, +3% for g=3 and -1.7% for g=4 > > Regarding the huge increase in NUMA case, at the first beginning, I suspect > we have wrong llc domain. For example, if cpu0's llc domain span > cpu0-cpu47, then select_idle_cpu() is running in wrong range while > it should run in cpu0-cpu23. > > But after printing the llc domain's span, I find it is completely right. > Cpu0's llc span: cpu0-cpu23 > Cpu24's llc span: cpu24-cpu47 Have you checked that the cluster mask was also correct ? > > Maybe I need more trace data to figure out if select_idle_cpu() is running > correctly. For example, maybe I can figure out if it is always returning -1, > or it returns -1 very often? yes, could be interesting to check how often select_idle_cpu return -1 > > Or do you have any idea? tracking migration across nod could help to understand too Vincent > > > > > > > Thanks > > > Barry > > Thanks > Barry > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel