From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-yb1-f182.google.com (mail-yb1-f182.google.com [209.85.219.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 27D032100 for ; Mon, 27 Jun 2022 08:46:34 +0000 (UTC) Received: by mail-yb1-f182.google.com with SMTP id v185so6005698ybe.8 for ; Mon, 27 Jun 2022 01:46:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=4vsOnIc+C2gY1UVlbVz/JDciMvgYEap1WYGcSJm7nik=; b=izm5TvVaySgpwnEQF9/y67fAWYu9B2xhHcC5MrbsqfKRyBWcvUJOefttP3U8C7H5yK cf/c+G/MDFeL9d3jWR3ZM1W+uRZZqwp2hTAlphjRcQ4y5gzmknF6zrBjrzS5Oq9593TH prMhKu/+/nH//vvJzE+A8myFZw0Vf2YB5togDcAyh2pSj1lg7+NXGygkUac2Pq5BEBXO 5+wwtXrzIkpK6w7dEXtmMbM9imlU0I4M6KLt1i3bZoyGQ30Vb87yxIZpNyIgjCBXAtgw DSd6AsFRpnuauM1qBq8q0T/kLZUm8tLjA1oB1Fx/TPENkDLwNudIWg8vpilEWofAkp3b qS3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=4vsOnIc+C2gY1UVlbVz/JDciMvgYEap1WYGcSJm7nik=; b=EUufD1xz664nFDVf6mfVxQFbGJ4gsg7pCTDhNXw1qlH9X1Thzuf1RWHIdbZIUE7DfQ I0z2Fih31NizUWFxJF4YnlbtE6BLmOxlQolDj/aehtwhbwRkWphwsKOpsgZRZdYd5cVC 7Jkwby3XSoz0CDXGLlQgC1fvKEr8f2ghS2J6ZBZNNg3PJlLwuZMtBl7OreNTuQHeMRfa T5zDjrIjpBaIlvqkETkKVsA9RSQh3TVJkXALkyFkvygdFKl3OWZgfM+QmQbxZJ2pAtmZ UebqHqrNnwEUATYN/NxcFzhad5O6IXBphFhuHMp6e3p6d0ZQQKVk2SFaONfB5uWcuXHb 3KGw== X-Gm-Message-State: AJIora+r393XwH46DQ9U8HQy4+D2fgKsdB4xZtenb/pu7y3YcjeYUfro v28hrm+d2KB7pefYkedsNUlogD9lNdwa8agd9LoBBw== X-Google-Smtp-Source: AGRyM1tmdR+Xbj5kJ69vq4foOyHTSytqN33Xz6MWmkpMcamFreKrameDkaAL4/75vWqZ0Cye+/NLca9lf7qZmNQ/HPM= X-Received: by 2002:a25:6c5:0:b0:669:a17a:2289 with SMTP id 188-20020a2506c5000000b00669a17a2289mr12352386ybg.231.1656319592797; Mon, 27 Jun 2022 01:46:32 -0700 (PDT) Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <20220619150456.GB34471@xsang-OptiPlex-9020> <20220622172857.37db0d29@kernel.org> <20220623185730.25b88096@kernel.org> <20220624070656.GE79500@shbuild999.sh.intel.com> <20220624144358.lqt2ffjdry6p5u4d@google.com> <20220625023642.GA40868@shbuild999.sh.intel.com> <20220627023812.GA29314@shbuild999.sh.intel.com> In-Reply-To: <20220627023812.GA29314@shbuild999.sh.intel.com> From: Eric Dumazet Date: Mon, 27 Jun 2022 10:46:21 +0200 Message-ID: Subject: Re: [net] 4890b686f4: netperf.Throughput_Mbps -69.4% regression To: Feng Tang Cc: Shakeel Butt , Linux MM , Andrew Morton , Roman Gushchin , Michal Hocko , Johannes Weiner , Muchun Song , Jakub Kicinski , Xin Long , Marcelo Ricardo Leitner , kernel test robot , Soheil Hassas Yeganeh , LKML , network dev , linux-s390@vger.kernel.org, MPTCP Upstream , "linux-sctp @ vger . kernel . org" , lkp@lists.01.org, kbuild test robot , Huang Ying , Xing Zhengjun , Yin Fengwei , Ying Xu Content-Type: text/plain; charset="UTF-8" On Mon, Jun 27, 2022 at 4:38 AM Feng Tang wrote: > > On Sat, Jun 25, 2022 at 10:36:42AM +0800, Feng Tang wrote: > > On Fri, Jun 24, 2022 at 02:43:58PM +0000, Shakeel Butt wrote: > > > On Fri, Jun 24, 2022 at 03:06:56PM +0800, Feng Tang wrote: > > > > On Thu, Jun 23, 2022 at 11:34:15PM -0700, Shakeel Butt wrote: > > > [...] > > > > > > > > > > Feng, can you please explain the memcg setup on these test machines > > > > > and if the tests are run in root or non-root memcg? > > > > > > > > I don't know the exact setup, Philip/Oliver from 0Day can correct me. > > > > > > > > I logged into a test box which runs netperf test, and it seems to be > > > > cgoup v1 and non-root memcg. The netperf tasks all sit in dir: > > > > '/sys/fs/cgroup/memory/system.slice/lkp-bootstrap.service' > > > > > > > > > > Thanks Feng. Can you check the value of memory.kmem.tcp.max_usage_in_bytes > > > in /sys/fs/cgroup/memory/system.slice/lkp-bootstrap.service after making > > > sure that the netperf test has already run? > > > > memory.kmem.tcp.max_usage_in_bytes:0 > > Sorry, I made a mistake that in the original report from Oliver, it > was 'cgroup v2' with a 'debian-11.1' rootfs. > > When you asked about cgroup info, I tried the job on another tbox, and > the original 'job.yaml' didn't work, so I kept the 'netperf' test > parameters and started a new job which somehow run with a 'debian-10.4' > rootfs and acutally run with cgroup v1. > > And as you mentioned cgroup version does make a big difference, that > with v1, the regression is reduced to 1% ~ 5% on different generations > of test platforms. Eric mentioned they also got regression report, > but much smaller one, maybe it's due to the cgroup version? This was using the current net-next tree. Used recipe was something like: Make sure cgroup2 is mounted or mount it by mount -t cgroup2 none $MOUNT_POINT. Enable memory controller by echo +memory > $MOUNT_POINT/cgroup.subtree_control. Create a cgroup by mkdir $MOUNT_POINT/job. Jump into that cgroup by echo $$ > $MOUNT_POINT/job/cgroup.procs. The regression was smaller than 1%, so considered noise compared to the benefits of the bug fix. > > Thanks, > Feng > > > And here is more memcg stats (let me know if you want to check more) > > > > > If this is non-zero then network memory accounting is enabled and the > > > slowdown is expected. > > > > >From the perf-profile data in original report, both > > __sk_mem_raise_allocated() and __sk_mem_reduce_allocated() are called > > much more often, which call memcg charge/uncharge functions. > > > > IIUC, the call chain is: > > > > __sk_mem_raise_allocated > > sk_memory_allocated_add > > mem_cgroup_charge_skmem > > charge memcg->tcpmem (for cgroup v2) > > try_charge memcg (for v1) > > > > Also from Eric's one earlier commit log: > > > > " > > net: implement per-cpu reserves for memory_allocated > > ... > > This means we are going to call sk_memory_allocated_add() > > and sk_memory_allocated_sub() more often. > > ... > > " > > > > So this slowdown is related to the more calling of charge/uncharge? > > > > Thanks, > > Feng > > > > > > And the rootfs is a debian based rootfs > > > > > > > > Thanks, > > > > Feng > > > > > > > > > > > > > thanks, > > > > > Shakeel