From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB96AC433EF for ; Fri, 10 Sep 2021 01:19:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 43322610C9 for ; Fri, 10 Sep 2021 01:19:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 43322610C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id B377D6B0071; Thu, 9 Sep 2021 21:19:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE586900002; Thu, 9 Sep 2021 21:19:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9AC316B0073; Thu, 9 Sep 2021 21:19:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0085.hostedemail.com [216.40.44.85]) by kanga.kvack.org (Postfix) with ESMTP id 8C5406B0071 for ; Thu, 9 Sep 2021 21:19:20 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 41E8E31E50 for ; Fri, 10 Sep 2021 01:19:20 +0000 (UTC) X-FDA: 78569905680.16.B461871 Received: from mail-lj1-f171.google.com (mail-lj1-f171.google.com [209.85.208.171]) by imf11.hostedemail.com (Postfix) with ESMTP id 001B1F0000B5 for ; Fri, 10 Sep 2021 01:19:19 +0000 (UTC) Received: by mail-lj1-f171.google.com with SMTP id s12so593539ljg.0 for ; Thu, 09 Sep 2021 18:19:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=fFtcZqqQNlz6Fo1j4RT+wXd6hyzZuXZbR1hNqy5igjU=; b=hSRL+WyUJXwUcnrNYMNnEI15VJGfds+1s1dPvNus6FbCMsiOovqAZAZXi5JWAJIABQ 7r5S0cGi+wc4+f/6yCU3ReOpb1EqhwMWWjSn6WNb5Cphsb74F/lMDCVHVQ6kw7vlHst7 Lp8QkTTHnevTzIo/hhIWlt7GGNiHHkBDXaCZ3YXotxGb1uGOrwZpxo+WWvaU7KICXprz NtzDx5s8FA6PJDIiTxTdHExWXrnchcMuyHkrzUTqcsNRdbTbM3kkluQ6I2VBD95QsmV7 o7NbATkRtuCMYnaEiStX9scb8/ZNHqIgXKd+WuySR0Yet5cKm0/qB7quBPU4R/+vde2w V6HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=fFtcZqqQNlz6Fo1j4RT+wXd6hyzZuXZbR1hNqy5igjU=; b=B7N+C1hNiqfRqC6wa05tjwpMz0nPffi3QYS2Wmjl9GCL2YDzBPceJNKlxw4rvJ/qeJ 0l0apOWZomHdriMrmJj4vU/NIw6EApJntyYO0lb9ozuIpRDS9XYNMaOZZms1Xm+akl+T hvQgC7QhKpA7MWHhh58Ubz3SrEZSuSvA50hUSnAVdxI4frgZKNetcT4QzvkpDy/itmMd udMaZb0as0I9ba6epGwJrbNFKOMFRMWHZqqgqRNMTLS4IJtaCzEoC6le2HrQA9ZBgfYu 0vx0+oPst/9uP7kYA8sLNw6ueWKL66/f3rRBAvRhIarCnoIhvI5l2LbI5yNMcfvWoBX8 x9Cg== X-Gm-Message-State: AOAM531Zq+XN68a7tzbieML4ZiKMWu7v+nY3Lqnft2qyrEGQK+0+XKZW rdqHmfqfGtFtR6XJf48b2CFrGtEyCcbimDSKul+6Yw== X-Google-Smtp-Source: ABdhPJx0EHLqcZYEMME+jzHmD+0Avn+MXVsNojtjDl7JoLwOqEcfuh3FmEWGaZm5g9a/nos0JiO3xDsUyMNrRUFGRHg= X-Received: by 2002:a2e:a363:: with SMTP id i3mr2267599ljn.86.1631236758167; Thu, 09 Sep 2021 18:19:18 -0700 (PDT) MIME-Version: 1.0 References: <20210902215504.dSSfDKJZu%akpm@linux-foundation.org> <20210905124439.GA15026@xsang-OptiPlex-9020> <20210907033000.GA88160@shbuild999.sh.intel.com> <20210910010842.GA94434@shbuild999.sh.intel.com> In-Reply-To: <20210910010842.GA94434@shbuild999.sh.intel.com> From: Shakeel Butt Date: Thu, 9 Sep 2021 18:19:06 -0700 Message-ID: Subject: Re: [memcg] 45208c9105: aim7.jobs-per-min -14.0% regression To: Feng Tang Cc: kernel test robot , Andrew Morton , 0day robot , Marek Szyprowski , Hillf Danton , Huang Ying , Johannes Weiner , Michal Hocko , "Michal Koutn??" , Muchun Song , Roman Gushchin , Tejun Heo , LKML , lkp@lists.01.org, Xing Zhengjun , Linux MM , mm-commits@vger.kernel.org, Linus Torvalds Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: ushzts43f8c7dmg4mbz889n9od4biuss Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=hSRL+WyU; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of shakeelb@google.com designates 209.85.208.171 as permitted sender) smtp.mailfrom=shakeelb@google.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 001B1F0000B5 X-HE-Tag: 1631236759-490761 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Sep 9, 2021 at 6:08 PM Feng Tang wrote: > > On Thu, Sep 09, 2021 at 05:43:40PM -0700, Shakeel Butt wrote: > > On Mon, Sep 6, 2021 at 8:30 PM Feng Tang wrote: > > > > > > Hi Shakeel, > > > > > > On Sun, Sep 05, 2021 at 03:15:46PM -0700, Shakeel Butt wrote: > > > > On Sun, Sep 5, 2021 at 5:27 AM kernel test robot wrote: > > > [...] > > > > > ========================================================================================= > > > > > compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase/ucode: > > > > > gcc-9/performance/1BRD_48G/xfs/x86_64-rhel-8.3/3000/debian-10.4-x86_64-20200603.cgz/lkp-icl-2sp2/disk_rr/aim7/0xd000280 > > > > > > > > > > commit: > > > > > 3c28c7680e ("memcg: switch lruvec stats to rstat") > > > > > 45208c9105 ("memcg: infrastructure to flush memcg stats") > > > > > > > > I am looking into this. I was hoping we have resolution for [1] as > > > > these patches touch similar data structures. > > > > > > > > [1] https://lore.kernel.org/all/20210811031734.GA5193@xsang-OptiPlex-9020/T/#u > > > > > > I tried 2 debug methods for that 36.4% vm-scalability regression: > > > > > > 1. Disable the HW cache prefetcher, no effect on this case > > > 2. relayout and add padding to 'struct cgroup_subsys_state', reduce > > > the regression to 3.1% > > > > > > > Thanks Feng but it seems like the issue for this commit is different. > > Rearranging the layout didn't help. Actually the cause of slowdown is > > the call to queue_work() inside __mod_memcg_lruvec_state(). > > > > At the moment, queue_work() is called after 32 updates. I changed it > > to 128 and the slowdown of will-it-scale:page_fault[1|2|3] halved > > (from around 10% to 5%). I am unable to run reaim or > > will-it-scale:fallocate2 as I was getting weird errors. > > > > Feng, is it possible for you to run these benchmarks with the change > > (basically changing MEMCG_CHARGE_BATCH to 128 in the if condition > > before queue_work() inside __mod_memcg_lruvec_state())? > > When I checked this, I tried different changes, including this batch > number change :), but it didn't recover the regression (the regression > is slightly reduced to about 12%) > > Please check if my patch is what you want to test: Yes, the following patch is what I want to test. > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c > index 4d8c9af..a50a69a 100644 > --- a/mm/memcontrol.c > +++ b/mm/memcontrol.c > @@ -682,7 +682,8 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, > > /* Update lruvec */ > __this_cpu_add(pn->lruvec_stats_percpu->state[idx], val); > - if (!(__this_cpu_inc_return(stats_flush_threshold) % MEMCG_CHARGE_BATCH)) > +// if (!(__this_cpu_inc_return(stats_flush_threshold) % MEMCG_CHARGE_BATCH)) > + if (!(__this_cpu_inc_return(stats_flush_threshold) % 128)) > queue_work(system_unbound_wq, &stats_flush_work); > } > Another change we can try is to remove this specific queue_work() altogether because this is the only significant change for the workload. That will give us the base performance number. If that also has regression then there are more issues to debug. Thanks a lot for your help.