From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEFA1C433ED for ; Thu, 8 Apr 2021 21:08:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 43EEA61175 for ; Thu, 8 Apr 2021 21:08:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 43EEA61175 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B30C56B0036; Thu, 8 Apr 2021 17:08:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AB9F76B006C; Thu, 8 Apr 2021 17:08:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90C436B006E; Thu, 8 Apr 2021 17:08:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0200.hostedemail.com [216.40.44.200]) by kanga.kvack.org (Postfix) with ESMTP id 708086B0036 for ; Thu, 8 Apr 2021 17:08:27 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id EFF661801FA4B for ; Thu, 8 Apr 2021 21:08:26 +0000 (UTC) X-FDA: 78010438212.27.35F7A30 Received: from mail-lf1-f45.google.com (mail-lf1-f45.google.com [209.85.167.45]) by imf13.hostedemail.com (Postfix) with ESMTP id 5F601E000118 for ; Thu, 8 Apr 2021 21:08:24 +0000 (UTC) Received: by mail-lf1-f45.google.com with SMTP id n8so6333916lfh.1 for ; Thu, 08 Apr 2021 14:08:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=mHTf5ssdkIiRDssyNc9X1VWJob3DAQAHF/gT3LBrwK4=; b=K/2mj2WZnop4FMfN5auFI3pSXePveJpdMuXsG94EKe79lp0ntZCo4nrOpfAe3cD7zZ EAOCOhkQPiteudf67bLl1Xb5omVMKb0zTVI091C70TA+rMYlc/5Z+JlE/pap0Pun/ZvU OkHlTT9LSMp2KCTY5isvFeg9T4xFaTdXDH6Ip4whYUQsSPvCotnTg6JTG0mwR6eARGV9 W/UdSM+Zaxqx7oom4CcfDZpI8lsM0MEFo6eQkuu8SE3Vnhw6LtLXb1q1VWaDh+CyXUeH pbs9Irk0M9erAhjswUFKHAvQJT0Ya4Nwfd+2LK5qoGY+oX7xDjD2DuIVYBmoa/XEWmW/ Plqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=mHTf5ssdkIiRDssyNc9X1VWJob3DAQAHF/gT3LBrwK4=; b=regrYpiZOs9rQOH5/sMjKEHYd/0dheUS/1EJ03yqFvXyClxCgy8PDrNYIGTCnv0lLi gxUPxyFZbvhO7dAc9iYmV39DFJ5BR9IcWuYMt3z8tsRi9kacQ5s3xIVvWHa3NpeB984x shsojqLhcg0oGN6HU9r/tmu51pfNQXHf3Ax9QjJtTX9n7Ika83BVubaiyKnf5cuMXsZ8 Q0ko00LVV4sZs9nb8K2CBjGdOeb+R2iEjxKmrECnOEeUvCIib4vKXYTWu0DirV1Z1f/Q gheILBjYkuBt/U8sTg5WDWR1CXqBTnplIm8ilfKWh01YkYgY9ZrIlCp6Vc0O4LEj9Bc1 Nbag== X-Gm-Message-State: AOAM531af/Rh1wBDCbEgeFmZp1eA7PXjW2lqnigmj+bz4zLILnpPQpo/ bjyGX6upUcbfCsXLcb2Qe7mnMkExvbzJf7VQMLNGsg== X-Google-Smtp-Source: ABdhPJyvolIigLxCDZRZlYFjolv6wvU8kltQD0URGdOZg186bPPwq8th2JOzr7NDip7h6wQXMX1mecS1D41flyHh8Eo= X-Received: by 2002:a19:3804:: with SMTP id f4mr8369747lfa.117.1617916104718; Thu, 08 Apr 2021 14:08:24 -0700 (PDT) MIME-Version: 1.0 References: <20210408193948.vfktg3azh2wrt56t@gabell> In-Reply-To: From: Shakeel Butt Date: Thu, 8 Apr 2021 14:08:13 -0700 Message-ID: Subject: Re: memcg: performance degradation since v5.9 To: Roman Gushchin Cc: Masayoshi Mizuma , Johannes Weiner , Michal Hocko , Vladimir Davydov , Cgroups , Linux MM Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: og81rw5ktju8bmdfz3u7m4bh8aanmuet X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5F601E000118 Received-SPF: none (google.com>: No applicable sender policy available) receiver=imf13; identity=mailfrom; envelope-from=""; helo=mail-lf1-f45.google.com; client-ip=209.85.167.45 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617916104-843895 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Apr 8, 2021 at 1:54 PM Roman Gushchin wrote: > > On Thu, Apr 08, 2021 at 03:39:48PM -0400, Masayoshi Mizuma wrote: > > Hello, > > > > I detected a performance degradation issue for a benchmark of PostgresSQL [1], > > and the issue seems to be related to object level memory cgroup [2]. > > I would appreciate it if you could give me some ideas to solve it. > > > > The benchmark shows the transaction per second (tps) and the tps for v5.9 > > and later kernel get about 10%-20% smaller than v5.8. > > > > The benchmark does sendto() and recvfrom() system calls repeatedly, > > and the duration of the system calls get longer than v5.8. > > The result of perf trace of the benchmark is as follows: > > > > - v5.8 > > > > syscall calls errors total min avg max stddev > > (msec) (msec) (msec) (msec) (%) > > --------------- -------- ------ -------- --------- --------- --------- ------ > > sendto 699574 0 2595.220 0.001 0.004 0.462 0.03% > > recvfrom 1391089 694427 2163.458 0.001 0.002 0.442 0.04% > > > > - v5.9 > > > > syscall calls errors total min avg max stddev > > (msec) (msec) (msec) (msec) (%) > > --------------- -------- ------ -------- --------- --------- --------- ------ > > sendto 699187 0 3316.948 0.002 0.005 0.044 0.02% > > recvfrom 1397042 698828 2464.995 0.001 0.002 0.025 0.04% > > > > - v5.12-rc6 > > > > syscall calls errors total min avg max stddev > > (msec) (msec) (msec) (msec) (%) > > --------------- -------- ------ -------- --------- --------- --------- ------ > > sendto 699445 0 3015.642 0.002 0.004 0.027 0.02% > > recvfrom 1395929 697909 2338.783 0.001 0.002 0.024 0.03% > > Can you please explain how to read these numbers? Or at least put a % regression. > > I bisected the kernel patches, then I found the patch series, which add > > object level memory cgroup support, causes the degradation. > > > > I confirmed the delay with a kernel module which just runs > > kmem_cache_alloc/kmem_cache_free as follows. The duration is about > > 2-3 times than v5.8. > > > > dummy_cache = KMEM_CACHE(dummy, SLAB_ACCOUNT); > > for (i = 0; i < 100000000; i++) > > { > > p = kmem_cache_alloc(dummy_cache, GFP_KERNEL); > > kmem_cache_free(dummy_cache, p); > > } > > > > It seems that the object accounting work in slab_pre_alloc_hook() and > > slab_post_alloc_hook() is the overhead. > > > > cgroup.nokmem kernel parameter doesn't work for my case because it disables > > all of kmem accounting. The patch is somewhat doing that i.e. disabling memcg accounting for slab. > > > > The degradation is gone when I apply a patch (at the bottom of this email) > > that adds a kernel parameter that expects to fallback to the page level > > accounting, however, I'm not sure it's a good approach though... > > Hello Masayoshi! > > Thank you for the report! > > It's not a secret that per-object accounting is more expensive than a per-page > allocation. I had micro-benchmark results similar to yours: accounted > allocations are about 2x slower. But in general it tends to not affect real > workloads, because the cost of allocations is still low and tends to be only > a small fraction of the whole cpu load. And because it brings up significant > benefits: 40%+ slab memory savings, less fragmentation, more stable workingset, > etc, real workloads tend to perform on pair or better. > > So my first question is if you see the regression in any real workload > or it's only about the benchmark? > > Second, I'll try to take a look into the benchmark to figure out why it's > affected so badly, but I'm not sure we can easily fix it. If you have any > ideas what kind of objects the benchmark is allocating in big numbers, > please let me know. > One idea would be to increase MEMCG_CHARGE_BATCH.