From: Feng Tang <feng.tang@intel.com> To: Roman Gushchin <roman.gushchin@linux.dev> Cc: Shakeel Butt <shakeelb@google.com>, Eric Dumazet <edumazet@google.com>, Linux MM <linux-mm@kvack.org>, Andrew Morton <akpm@linux-foundation.org>, Michal Hocko <mhocko@kernel.org>, Johannes Weiner <hannes@cmpxchg.org>, Muchun Song <songmuchun@bytedance.com>, Jakub Kicinski <kuba@kernel.org>, Xin Long <lucien.xin@gmail.com>, Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>, kernel test robot <oliver.sang@intel.com>, Soheil Hassas Yeganeh <soheil@google.com>, LKML <linux-kernel@vger.kernel.org>, network dev <netdev@vger.kernel.org>, linux-s390@vger.kernel.org, MPTCP Upstream <mptcp@lists.linux.dev>, "linux-sctp @ vger . kernel . org" <linux-sctp@vger.kernel.org>, lkp@lists.01.org, kbuild test robot <lkp@intel.com>, Huang Ying <ying.huang@intel.com>, Xing Zhengjun <zhengjun.xing@linux.intel.com>, Yin Fengwei <fengwei.yin@intel.com>, Ying Xu <yinxu@redhat.com> Subject: Re: [net] 4890b686f4: netperf.Throughput_Mbps -69.4% regression Date: Tue, 5 Jul 2022 13:03:26 +0800 [thread overview] Message-ID: <20220705050326.GF62281@shbuild999.sh.intel.com> (raw) In-Reply-To: <YsIeYzEuj95PWMWO@castle> On Sun, Jul 03, 2022 at 03:55:31PM -0700, Roman Gushchin wrote: > On Sun, Jul 03, 2022 at 06:43:53PM +0800, Feng Tang wrote: > > Hi Shakeel, > > > > On Fri, Jul 01, 2022 at 08:47:29AM -0700, Shakeel Butt wrote: > > > On Mon, Jun 27, 2022 at 8:49 PM Feng Tang <feng.tang@intel.com> wrote: > > > > I just tested it, it does perform better (the 4th is with your patch), > > > > some perf-profile data is also listed. > > > > > > > > 7c80b038d23e1f4c 4890b686f4088c90432149bd6de 332b589c49656a45881bca4ecc0 e719635902654380b23ffce908d > > > > ---------------- --------------------------- --------------------------- --------------------------- > > > > 15722 -69.5% 4792 -40.8% 9300 -27.9% 11341 netperf.Throughput_Mbps > > > > > > > > 0.00 +0.3 0.26 ± 5% +0.5 0.51 +1.3 1.27 ± 2%pp.self.__sk_mem_raise_allocated > > > > 0.00 +0.3 0.32 ± 15% +1.7 1.74 ± 2% +0.4 0.40 ± 2% pp.self.propagate_protected_usage > > > > 0.00 +0.8 0.82 ± 7% +0.9 0.90 +0.8 0.84 pp.self.__mod_memcg_state > > > > 0.00 +1.2 1.24 ± 4% +1.0 1.01 +1.4 1.44 pp.self.try_charge_memcg > > > > 0.00 +2.1 2.06 +2.1 2.13 +2.1 2.11 pp.self.page_counter_uncharge > > > > 0.00 +2.1 2.14 ± 4% +2.7 2.71 +2.6 2.60 ± 2% pp.self.page_counter_try_charge > > > > 1.12 ± 4% +3.1 4.24 +1.1 2.22 +1.4 2.51 pp.self.native_queued_spin_lock_slowpath > > > > 0.28 ± 9% +3.8 4.06 ± 4% +0.2 0.48 +0.4 0.68 pp.self.sctp_eat_data > > > > 0.00 +8.2 8.23 +0.8 0.83 +1.3 1.26 pp.self.__sk_mem_reduce_allocated > > > > > > > > And the size of 'mem_cgroup' is increased from 4224 Bytes to 4608. > > > > > > Hi Feng, can you please try two more configurations? Take Eric's patch > > > of adding ____cacheline_aligned_in_smp in page_counter and for first > > > increase MEMCG_CHARGE_BATCH to 64 and for second increase it to 128. > > > Basically batch increases combined with Eric's patch. > > > > With increasing batch to 128, the regression could be reduced to -12.4%. > > If we're going to bump it, I wonder if we should scale it dynamically depending > on the size of the memory cgroup? I think it makes sense, or also make it a configurable parameter? From the test reports of 0Day, these charging/counting play critical role in performance (easy to see up to 60% performance effect). If user only wants memcg for isolating things or doesn't care charging/stats, these seem to be extra taxes. For bumping to 64 or 128, universal improvement is expected with the only concern of accuracy. Thanks, Feng > Thanks!
WARNING: multiple messages have this Message-ID (diff)
From: Feng Tang <feng.tang@intel.com> To: lkp@lists.01.org Subject: Re: [net] 4890b686f4: netperf.Throughput_Mbps -69.4% regression Date: Tue, 05 Jul 2022 13:03:26 +0800 [thread overview] Message-ID: <20220705050326.GF62281@shbuild999.sh.intel.com> (raw) In-Reply-To: <YsIeYzEuj95PWMWO@castle> [-- Attachment #1: Type: text/plain, Size: 3118 bytes --] On Sun, Jul 03, 2022 at 03:55:31PM -0700, Roman Gushchin wrote: > On Sun, Jul 03, 2022 at 06:43:53PM +0800, Feng Tang wrote: > > Hi Shakeel, > > > > On Fri, Jul 01, 2022 at 08:47:29AM -0700, Shakeel Butt wrote: > > > On Mon, Jun 27, 2022 at 8:49 PM Feng Tang <feng.tang@intel.com> wrote: > > > > I just tested it, it does perform better (the 4th is with your patch), > > > > some perf-profile data is also listed. > > > > > > > > 7c80b038d23e1f4c 4890b686f4088c90432149bd6de 332b589c49656a45881bca4ecc0 e719635902654380b23ffce908d > > > > ---------------- --------------------------- --------------------------- --------------------------- > > > > 15722 -69.5% 4792 -40.8% 9300 -27.9% 11341 netperf.Throughput_Mbps > > > > > > > > 0.00 +0.3 0.26 ± 5% +0.5 0.51 +1.3 1.27 ± 2%pp.self.__sk_mem_raise_allocated > > > > 0.00 +0.3 0.32 ± 15% +1.7 1.74 ± 2% +0.4 0.40 ± 2% pp.self.propagate_protected_usage > > > > 0.00 +0.8 0.82 ± 7% +0.9 0.90 +0.8 0.84 pp.self.__mod_memcg_state > > > > 0.00 +1.2 1.24 ± 4% +1.0 1.01 +1.4 1.44 pp.self.try_charge_memcg > > > > 0.00 +2.1 2.06 +2.1 2.13 +2.1 2.11 pp.self.page_counter_uncharge > > > > 0.00 +2.1 2.14 ± 4% +2.7 2.71 +2.6 2.60 ± 2% pp.self.page_counter_try_charge > > > > 1.12 ± 4% +3.1 4.24 +1.1 2.22 +1.4 2.51 pp.self.native_queued_spin_lock_slowpath > > > > 0.28 ± 9% +3.8 4.06 ± 4% +0.2 0.48 +0.4 0.68 pp.self.sctp_eat_data > > > > 0.00 +8.2 8.23 +0.8 0.83 +1.3 1.26 pp.self.__sk_mem_reduce_allocated > > > > > > > > And the size of 'mem_cgroup' is increased from 4224 Bytes to 4608. > > > > > > Hi Feng, can you please try two more configurations? Take Eric's patch > > > of adding ____cacheline_aligned_in_smp in page_counter and for first > > > increase MEMCG_CHARGE_BATCH to 64 and for second increase it to 128. > > > Basically batch increases combined with Eric's patch. > > > > With increasing batch to 128, the regression could be reduced to -12.4%. > > If we're going to bump it, I wonder if we should scale it dynamically depending > on the size of the memory cgroup? I think it makes sense, or also make it a configurable parameter? From the test reports of 0Day, these charging/counting play critical role in performance (easy to see up to 60% performance effect). If user only wants memcg for isolating things or doesn't care charging/stats, these seem to be extra taxes. For bumping to 64 or 128, universal improvement is expected with the only concern of accuracy. Thanks, Feng > Thanks!
next prev parent reply other threads:[~2022-07-05 5:03 UTC|newest] Thread overview: 70+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-06-19 15:04 [net] 4890b686f4: netperf.Throughput_Mbps -69.4% regression kernel test robot 2022-06-19 15:04 ` kernel test robot 2022-06-23 0:28 ` Jakub Kicinski 2022-06-23 0:28 ` Jakub Kicinski 2022-06-23 3:08 ` Xin Long 2022-06-23 3:08 ` Xin Long 2022-06-23 22:50 ` Xin Long 2022-06-23 22:50 ` Xin Long 2022-06-24 1:57 ` Jakub Kicinski 2022-06-24 1:57 ` Jakub Kicinski 2022-06-24 4:13 ` Eric Dumazet 2022-06-24 4:13 ` Eric Dumazet 2022-06-24 4:22 ` Eric Dumazet 2022-06-24 4:22 ` Eric Dumazet 2022-06-24 5:13 ` Feng Tang 2022-06-24 5:13 ` Feng Tang 2022-06-24 5:45 ` Eric Dumazet 2022-06-24 5:45 ` Eric Dumazet 2022-06-24 6:00 ` Feng Tang 2022-06-24 6:00 ` Feng Tang 2022-06-24 6:07 ` Eric Dumazet 2022-06-24 6:07 ` Eric Dumazet 2022-06-24 6:34 ` Shakeel Butt 2022-06-24 6:34 ` Shakeel Butt 2022-06-24 7:06 ` Feng Tang 2022-06-24 7:06 ` Feng Tang 2022-06-24 14:43 ` Shakeel Butt 2022-06-24 14:43 ` Shakeel Butt 2022-06-25 2:36 ` Feng Tang 2022-06-25 2:36 ` Feng Tang 2022-06-27 2:38 ` Feng Tang 2022-06-27 2:38 ` Feng Tang 2022-06-27 8:46 ` Eric Dumazet 2022-06-27 8:46 ` Eric Dumazet 2022-06-27 12:34 ` Feng Tang 2022-06-27 12:34 ` Feng Tang 2022-06-27 14:07 ` Eric Dumazet 2022-06-27 14:07 ` Eric Dumazet 2022-06-27 14:48 ` Feng Tang 2022-06-27 14:48 ` Feng Tang 2022-06-27 16:25 ` Eric Dumazet 2022-06-27 16:25 ` Eric Dumazet 2022-06-27 16:48 ` Shakeel Butt 2022-06-27 16:48 ` Shakeel Butt 2022-06-27 17:05 ` Eric Dumazet 2022-06-27 17:05 ` Eric Dumazet 2022-06-28 1:46 ` Roman Gushchin 2022-06-28 1:46 ` Roman Gushchin 2022-06-28 3:49 ` Feng Tang 2022-06-28 3:49 ` Feng Tang 2022-07-01 15:47 ` Shakeel Butt 2022-07-01 15:47 ` Shakeel Butt 2022-07-03 10:43 ` Feng Tang 2022-07-03 10:43 ` Feng Tang 2022-07-03 22:55 ` Roman Gushchin 2022-07-03 22:55 ` Roman Gushchin 2022-07-05 5:03 ` Feng Tang [this message] 2022-07-05 5:03 ` Feng Tang 2022-08-16 5:52 ` Oliver Sang 2022-08-16 5:52 ` Oliver Sang 2022-08-16 15:55 ` Shakeel Butt 2022-08-16 15:55 ` Shakeel Butt 2022-06-27 14:52 ` Shakeel Butt 2022-06-27 14:52 ` Shakeel Butt 2022-06-27 14:56 ` Eric Dumazet 2022-06-27 14:56 ` Eric Dumazet 2022-06-27 15:12 ` Feng Tang 2022-06-27 15:12 ` Feng Tang 2022-06-27 16:25 ` Shakeel Butt 2022-06-27 16:25 ` Shakeel Butt
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20220705050326.GF62281@shbuild999.sh.intel.com \ --to=feng.tang@intel.com \ --cc=akpm@linux-foundation.org \ --cc=edumazet@google.com \ --cc=fengwei.yin@intel.com \ --cc=hannes@cmpxchg.org \ --cc=kuba@kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linux-s390@vger.kernel.org \ --cc=linux-sctp@vger.kernel.org \ --cc=lkp@intel.com \ --cc=lkp@lists.01.org \ --cc=lucien.xin@gmail.com \ --cc=marcelo.leitner@gmail.com \ --cc=mhocko@kernel.org \ --cc=mptcp@lists.linux.dev \ --cc=netdev@vger.kernel.org \ --cc=oliver.sang@intel.com \ --cc=roman.gushchin@linux.dev \ --cc=shakeelb@google.com \ --cc=soheil@google.com \ --cc=songmuchun@bytedance.com \ --cc=ying.huang@intel.com \ --cc=yinxu@redhat.com \ --cc=zhengjun.xing@linux.intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.