From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3455FC64E7B for ; Thu, 3 Dec 2020 09:55:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D551520644 for ; Thu, 3 Dec 2020 09:55:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388645AbgLCJzJ (ORCPT ); Thu, 3 Dec 2020 04:55:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53570 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387531AbgLCJzI (ORCPT ); Thu, 3 Dec 2020 04:55:08 -0500 Received: from mail-lj1-x242.google.com (mail-lj1-x242.google.com [IPv6:2a00:1450:4864:20::242]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D1CFC061A4D for ; Thu, 3 Dec 2020 01:54:22 -0800 (PST) Received: by mail-lj1-x242.google.com with SMTP id r18so1881861ljc.2 for ; Thu, 03 Dec 2020 01:54:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=wEueHcV7l/ctvMvBAXQKBWYAUOQJ5GZrrhWLBwBeD4o=; b=rNuyqmJ5C2MhebdamXLkPH+dVt6AdIFw5GuyIWWpgzKbEhsEab0J+yt5FA89LvtYEy AXOv9K+LUJ5KkzCL3CzUABhpWP3EMkVFmEtzDVrsJeoPWKoYVodZK++P9LyEtM1gs4a7 WSKRNoY913ctUeHc0ZvLRDPlvHCGFLXocy4F0xlfXtkv1WB9PaRZ2D9dhAn1ly0AF6fJ rrW9fi1J0Rd3cgDM2xBQcF3wwgBdqRfnP8j8pibuqGD1twuxmEscFPq+nBbh0b46jhG0 14Ugpem3J6WbxQs3pk/bsox0x6Pv7rt28Eh/I03moh6cGtcjyKfOYH/9vbynfFzfz/sH wPxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=wEueHcV7l/ctvMvBAXQKBWYAUOQJ5GZrrhWLBwBeD4o=; b=j/+GDGkLsbJJ6W8/JLHxGznbQkuHv2NmcxW7qw+82IzNHGbdxNT1k1tp7QrLe4w5gp kVFzeLujgv8ybNAfyEp+IutUbJz1bGY9jngO5vcLuz6RAp7DpIMxoLtnW/EHxff2bVbk Zlo6Wb3BMemM6fNd7P0fCi00JzeiuiMlx2+BASWhIho7uPSdc8ZxEz75WIIo38kFGegP RdF2tfxB2AzL6RfGZlEvuh+G6EjWByJyh8lZnNc1vlvb3DPvYQuiM55D/VS8+FyXG9F4 tlD2kUENQQgA3kxIIfAI0M1SfwYzrKWALkVlvRfHOsALyFSlvag5c4QbT+OdYpCY3Qw+ Xxpw== X-Gm-Message-State: AOAM533KqpNyj2SeP+iZvnQolz6RfgtLoKUhC9bt6uobQ7KYW+T/Lvhz ihtK9DL86t5fL9S7BW2Nx4AhCPBBzwzc+bVCwszLsQ== X-Google-Smtp-Source: ABdhPJwdjaSsw7GE05hNk15bu9q/1poMmL86Ci331MBFX4yFGGVAd9RviAK5X2nqsWakHczDAvzczO+p7F7KynjSr28= X-Received: by 2002:a2e:b8d0:: with SMTP id s16mr884517ljp.423.1606989260776; Thu, 03 Dec 2020 01:54:20 -0800 (PST) MIME-Version: 1.0 References: <20201201025944.18260-1-song.bao.hua@hisilicon.com> <20201201025944.18260-3-song.bao.hua@hisilicon.com> <414fbd167b214452b925ac674575f0d6@hisilicon.com> In-Reply-To: From: Vincent Guittot Date: Thu, 3 Dec 2020 10:54:09 +0100 Message-ID: Subject: Re: [RFC PATCH v2 2/2] scheduler: add scheduler level for clusters To: "Song Bao Hua (Barry Song)" Cc: Valentin Schneider , Catalin Marinas , Will Deacon , "Rafael J. Wysocki" , "Cc: Len Brown" , "gregkh@linuxfoundation.org" , Jonathan Cameron , Ingo Molnar , Peter Zijlstra , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Mark Rutland , LAK , linux-kernel , ACPI Devel Maling List , Linuxarm , "xuwei (O)" , "Zengtao (B)" Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 3 Dec 2020 at 10:39, Vincent Guittot wrote: > > On Thu, 3 Dec 2020 at 10:11, Song Bao Hua (Barry Song) > wrote: > > > > > > > > > -----Original Message----- > > > From: Vincent Guittot [mailto:vincent.guittot@linaro.org] > > > Sent: Thursday, December 3, 2020 10:04 PM > > > To: Song Bao Hua (Barry Song) > > > Cc: Valentin Schneider ; Catalin Marinas > > > ; Will Deacon ; Rafael J. Wysocki > > > ; Cc: Len Brown ; > > > gregkh@linuxfoundation.org; Jonathan Cameron ; > > > Ingo Molnar ; Peter Zijlstra ; Juri > > > Lelli ; Dietmar Eggemann ; > > > Steven Rostedt ; Ben Segall ; Mel > > > Gorman ; Mark Rutland ; LAK > > > ; linux-kernel > > > ; ACPI Devel Maling List > > > ; Linuxarm ; xuwei (O) > > > ; Zengtao (B) > > > Subject: Re: [RFC PATCH v2 2/2] scheduler: add scheduler level for clusters > > > > > > On Wed, 2 Dec 2020 at 21:58, Song Bao Hua (Barry Song) > > > wrote: > > > > > > > > > > > > > > Sorry. Please ignore this. I added some printk here while testing > > > > > one numa. Will update you the data in another email. > > > > > > > > Re-tested in one NUMA node(cpu0-cpu23): > > > > > > > > g=1 > > > > Running in threaded mode with 1 groups using 40 file descriptors > > > > Each sender will pass 100000 messages of 100 bytes > > > > w/o: 7.689 7.485 7.485 7.458 7.524 7.539 7.738 7.693 7.568 7.674=7.5853 > > > > w/ : 7.516 7.941 7.374 7.963 7.881 7.910 7.420 7.556 7.695 7.441=7.6697 > > > > w/ but dropped select_idle_cluster: > > > > 7.752 7.739 7.739 7.571 7.545 7.685 7.407 7.580 7.605 7.487=7.611 > > > > > > > > g=2 > > > > Running in threaded mode with 2 groups using 40 file descriptors > > > > Each sender will pass 100000 messages of 100 bytes > > > > w/o: 10.127 10.119 10.070 10.196 10.057 10.111 10.045 10.164 10.162 > > > > 9.955=10.1006 > > > > w/ : 9.694 9.654 9.612 9.649 9.686 9.734 9.607 9.842 9.690 9.710=9.6878 > > > > w/ but dropped select_idle_cluster: > > > > 9.877 10.069 9.951 9.918 9.947 9.790 9.906 9.820 9.863 9.906=9.9047 > > > > > > > > g=3 > > > > Running in threaded mode with 3 groups using 40 file descriptors > > > > Each sender will pass 100000 messages of 100 bytes > > > > w/o: 15.885 15.254 15.932 15.647 16.120 15.878 15.857 15.759 15.674 > > > > 15.721=15.7727 > > > > w/ : 14.974 14.657 13.969 14.985 14.728 15.665 15.191 14.995 14.946 > > > > 14.895=14.9005 > > > > w/ but dropped select_idle_cluster: > > > > 15.405 15.177 15.373 15.187 15.450 15.540 15.278 15.628 15.228 > > > 15.325=15.3591 > > > > > > > > g=4 > > > > Running in threaded mode with 4 groups using 40 file descriptors > > > > Each sender will pass 100000 messages of 100 bytes > > > > w/o: 20.014 21.025 21.119 21.235 19.767 20.971 20.962 20.914 21.090 > > > 21.090=20.8187 > > > > w/ : 20.331 20.608 20.338 20.445 20.456 20.146 20.693 20.797 21.381 > > > 20.452=20.5647 > > > > w/ but dropped select_idle_cluster: > > > > 19.814 20.126 20.229 20.350 20.750 20.404 19.957 19.888 20.226 > > > 20.562=20.2306 > > > > > > > > > > I assume that you have run this on v5.9 as previous tests. > > > > Yep > > > > > The results don't show any real benefit of select_idle_cluster() > > > inside a node whereas this is where we could expect most of the > > > benefit. We have to understand why we have such an impact on numa > > > tests only. > > > > There is a 4-5.5% increase while g=2 and g=3. > > my point was with vs without select_idle_cluster() but still having a > cluster domain level > In this case, the diff is -0.8% for g=1 +2.2% for g=2, +3% for g=3 and > -1.7% for g=4 > > > > > Regarding the huge increase in NUMA case, at the first beginning, I suspect > > we have wrong llc domain. For example, if cpu0's llc domain span > > cpu0-cpu47, then select_idle_cpu() is running in wrong range while > > it should run in cpu0-cpu23. > > > > But after printing the llc domain's span, I find it is completely right. > > Cpu0's llc span: cpu0-cpu23 > > Cpu24's llc span: cpu24-cpu47 > > Have you checked that the cluster mask was also correct ? > > > > > Maybe I need more trace data to figure out if select_idle_cpu() is running > > correctly. For example, maybe I can figure out if it is always returning -1, > > or it returns -1 very often? > > yes, could be interesting to check how often select_idle_cpu return -1 > > > > > Or do you have any idea? > > tracking migration across nod could help to understand too Also the v6 of https://lkml.org/lkml/2020/11/26/187 might also help you > > Vincent > > > > > > > > > > > Thanks > > > > Barry > > > > Thanks > > Barry > >