From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 298CDC433E0 for ; Thu, 11 Mar 2021 19:09:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 86AAD64E60 for ; Thu, 11 Mar 2021 19:09:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 86AAD64E60 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D56038D02E4; Thu, 11 Mar 2021 14:09:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CDE7C8D02B2; Thu, 11 Mar 2021 14:09:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B171D8D02E4; Thu, 11 Mar 2021 14:09:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0024.hostedemail.com [216.40.44.24]) by kanga.kvack.org (Postfix) with ESMTP id 8E0E88D02B2 for ; Thu, 11 Mar 2021 14:09:02 -0500 (EST) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 406118249980 for ; Thu, 11 Mar 2021 19:09:02 +0000 (UTC) X-FDA: 77908530924.08.FABC9A4 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by imf28.hostedemail.com (Postfix) with ESMTP id B5FE22000D8E for ; Thu, 11 Mar 2021 19:09:01 +0000 (UTC) Received: by mail-pf1-f176.google.com with SMTP id x7so188306pfi.7 for ; Thu, 11 Mar 2021 11:09:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=tdhYJCYiOt2eSgI8PfK4axYaL2BrnhaEAO2Wlh0WvDk=; b=nkBpaQcgW9RD/fIGp5Tdbkr5WTg1wBsbwaUQV7yY3y/LPIFCyfx4v36reJKQuA53SM TLMBz5oHk9fNzyQdW0EevD1FmwpvpbVZ8TfpZWVXjgZ5lJC52aYEcgBm3/n6wSEsD7vN +7MwtIdbWmR9wVzlPW0AQGjkQlBlOO32/Rek16G0LSXch98dKfsyUB0xmDQfbKaPtCbp pA+m5X2soGbLm/2bEqarfQVLB2buA8mbEBcY1K6TP7xyqe5iadoJuO538gc7dZsdHQ3c OT9jiG9x5tmNK5KFg4O3Lm82WpCVKCRUY3TQgHGIaNAzAyW2rPVJta434DJbBsAy3ecy tRtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=tdhYJCYiOt2eSgI8PfK4axYaL2BrnhaEAO2Wlh0WvDk=; b=Is9jusG+r9mpqynJOgOaX8JV9b1tOxfWk5YuuN1EIWO7GmzzX0PFdYzZ1ggrXSHcIR b8eCpKF+gATeVbJwQuUJ08Ct1h4+c8S+ROUjT0Q/IsSBRfuBxqhLh7KqZwEBz3kAlh96 jo/sYp0Vf/lpakJVLHCfWNocxjL5zHds9GIxE7zO5h8BmNJtXS/dx3ECKHxEM1FTrI0q SbmX3JI9gTdOdPwfu0TQ3QybJJyLR73zpgG0mgad7xOD68O94KnhBeyf6it+Cd7yM4jV b2g6u9rlgsbCtndYctfEJIypgxTu+2d2T/z91OFHaZJR6b3jBgSDk+de3xfZDtJ1NM2L eYqg== X-Gm-Message-State: AOAM5313Cp+RpRZGisS2JpiciRqcP8aaCEb6M4Jyp4x+nu+MUP/alDVM ZIfZdFqhzpttXRoV21H3UHw= X-Google-Smtp-Source: ABdhPJyUbP6xv80HKAJya3p5gm4s9WjVOXrNwskZg3sKRrEtQuXYKtNBqp20EDyyoHhz0h2vkihVwQ== X-Received: by 2002:a63:c02:: with SMTP id b2mr8396753pgl.53.1615489739228; Thu, 11 Mar 2021 11:08:59 -0800 (PST) Received: from localhost.localdomain (c-73-93-239-127.hsd1.ca.comcast.net. [73.93.239.127]) by smtp.gmail.com with ESMTPSA id t12sm2999111pfe.203.2021.03.11.11.08.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Mar 2021 11:08:58 -0800 (PST) From: Yang Shi To: guro@fb.com, ktkhai@virtuozzo.com, vbabka@suse.cz, shakeelb@google.com, david@fromorbit.com, hannes@cmpxchg.org, mhocko@suse.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [v10 PATCH 00/13] Make shrinker's nr_deferred memcg aware Date: Thu, 11 Mar 2021 11:08:32 -0800 Message-Id: <20210311190845.9708-1-shy828301@gmail.com> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 X-Stat-Signature: 3c68p87spk115azzqn7zo5i7sst7jh9b X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B5FE22000D8E Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf28; identity=mailfrom; envelope-from=""; helo=mail-pf1-f176.google.com; client-ip=209.85.210.176 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615489741-59217 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Changelog v9 --> v10: * Rebased onto next-20210310 of linux-next to solve the conflict with "mm: memcontrol: switch to rstat" series.=20 v8 --> v9: * Used two parameters form of kvfree_rcu() per Shakeel. * Amended commit log about allocating shrinker_info for root memcg fo= r patch 09/13 per Shakeel. * Colleted more ack and review tags. * Rebased onto 5.12-rc2. v7 --> v8: * Added lockdep assert in expand_shrinker_info() per Roman. * Added patch 05/13 to use kvfree_rcu() instead of call_rcu() per Rom= an and Kirill. * Moved rwsem acquire/release out of unregister_memcg_shrinker() per = Roman. * Renamed count_nr_deferred_{memcg} to xchg_nr_deferred_{memcg} per R= oman. * Fixed the next_deferred logic per Vlastimil. * Misc minor code cleanup, refactor and spelling correction per Roman and Shakeel. * Collected more ack and review tags from Roman, Shakeel and Vlastimi= l. v6 --> v7: * Expanded shrinker_info in a batch of BITS_PER_LONG per Kirill. * Added patch 06/12 to introduce a helper for dereferencing shrinker_= info per Kirill. * Renamed set_nr_deferred_memcg to add_nr_deferred_memcg per Kirill. * Collected Acked-by from Kirill. v5 --> v6: * Rebased on top of https://lore.kernel.org/linux-mm/1611216029-34397= -1-git-send-email-abaci-bugfix@linux.alibaba.com/ per Kirill. * Don't register shrinker idr with NULL and remove idr_replace() per = Vlastimil. * Move nr_deferred before map to guarantee the alignment per Vlastimi= l. * Misc minor code cleanup and refactor per Kirill and Vlastimil. * Added Acked-by from Vlastimil for path #1, #2, #3, #5, #9 and #10. v4 --> v5: * Incorporated the comments from Kirill. * Rebased to v5.11-rc5. v3 --> v4: * Removed "memcg_" prefix for shrinker_maps related functions per Rom= an. * Use write lock instead of read lock per Kirill. Also removed Johann= es's ack since write lock is used. * Incorporated the comments from Kirill. * Removed RFC. * Rebased to v5.11-rc4. v2 --> v3: * Moved shrinker_maps related code to vmscan.c per Dave. * Removed memcg_shrinker_map_size. Calcuated the size of map via shri= nker_nr_max per Johannes. * Consolidated shrinker_deferred with shrinker_maps into one struct p= er Dave. * Simplified the nr_deferred related code. * Dropped the memory barrier from v2. * Moved nr_deferred reparent code to vmscan.c per Dave. * Added test coverage information in patch #11. Dave is concerned abo= ut the potential regression. I didn't notice regression with my tests, but= suggestions about more test coverage is definitely welcome. And it may help spo= t regression with this patch in -mm tree then linux-next tree so I keep it in th= is version. * The code cleanup and consolidation resulted in the series grow to 1= 1 patches. * Rebased onto 5.11-rc2.=20 v1 --> v2: * Use shrinker->flags to store the new SHRINKER_REGISTERED flag per R= oman. * Folded patch #1 into patch #6 per Roman. * Added memory barrier to prevent shrink_slab_memcg from seeing NULL = shrinker_maps/ shrinker_deferred per Kirill. * Removed memcg_shrinker_map_mutex. Protcted shrinker_map/shrinker_de= ferred allocations from expand with shrinker_rwsem per Johannes. Recently huge amount one-off slab drop was seen on some vfs metadata heav= y workloads, it turned out there were huge amount accumulated nr_deferred objects seen= by the shrinker. On our production machine, I saw absurd number of nr_deferred shown as th= e below tracing result:=20 <...>-48776 [032] .... 27970562.458916: mm_shrink_slab_start: super_cache_scan+0x0/0x1a0 ffff9a83046f3458: nid: 0 objects to shrink 2531805877005 gfp_flags GFP_HIGHUSER_MOVABLE pgs_scanned 32 lru_pgs 9300 cache items 1667 delta 11 total_scan 833 There are 2.5 trillion deferred objects on one node, assuming all of them are dentry (192 bytes per object), so the total size of deferred on one node is ~480TB. It is definitely ridiculous. I managed to reproduce this problem with kernel build workload plus negat= ive dentry generator. First step, run the below kernel build test script: NR_CPUS=3D`cat /proc/cpuinfo | grep -e processor | wc -l` cd /root/Buildarea/linux-stable for i in `seq 1500`; do cgcreate -g memory:kern_build echo 4G > /sys/fs/cgroup/memory/kern_build/memory.limit_in_bytes echo 3 > /proc/sys/vm/drop_caches cgexec -g memory:kern_build make clean > /dev/null 2>&1 cgexec -g memory:kern_build make -j$NR_CPUS > /dev/null 2>&1 cgdelete -g memory:kern_build done Then run the below negative dentry generator script: NR_CPUS=3D`cat /proc/cpuinfo | grep -e processor | wc -l` mkdir /sys/fs/cgroup/memory/test echo $$ > /sys/fs/cgroup/memory/test/tasks for i in `seq $NR_CPUS`; do while true; do FILE=3D`head /dev/urandom | tr -dc A-Za-z0-9 | head -c 64= ` cat $FILE 2>/dev/null done & done Then kswapd will shrink half of dentry cache in just one loop as the belo= w tracing result showed: kswapd0-475 [028] .... 305968.252561: mm_shrink_slab_start: super_cach= e_scan+0x0/0x190 0000000024acf00c: nid: 0 objects to shrink 4994376020 gfp_flags GFP_KERNEL cache items 93689873 de= lta 45746 total_scan 46844936 priority 12 kswapd0-475 [021] .... 306013.099399: mm_shrink_slab_end: super_cache_= scan+0x0/0x190 0000000024acf00c: nid: 0 unused scan count 4994376020 new scan count 4947576838 total_scan 8 last shrinke= r return val 46844928 There were huge number of deferred objects before the shrinker was called= , the behavior does match the code but it might be not desirable from the user's stand o= f point. The excessive amount of nr_deferred might be accumulated due to various r= easons, for example: * GFP_NOFS allocation * Significant times of small amount scan (< scan_batch, 1024 for vfs = metadata) However the LRUs of slabs are per memcg (memcg-aware shrinkers) but the d= eferred objects is per shrinker, this may have some bad effects: * Poor isolation among memcgs. Some memcgs which happen to have frequ= ent limit reclaim may get nr_deferred accumulated to a huge number, then othe= r innocent memcgs may take the fall. In our case the main workload was hit. * Unbounded deferred objects. There is no cap for deferred objects, i= t can outgrow ridiculously as the tracing result showed. * Easy to get out of control. Although shrinkers take into account de= ferred objects, but it can go out of control easily. One misconfigured memcg could = incur absurd=20 amount of deferred objects in a period of time. * Sort of reclaim problems, i.e. over reclaim, long reclaim latency, = etc. There may be hundred GB slab caches for vfe metadata heavy workload, shrink half= of them may take minutes. We observed latency spike due to the prolonged reclaim. These issues also have been discussed in https://lore.kernel.org/linux-mm= /20200916185823.5347-1-shy828301@gmail.com/. The patchset is the outcome of that discussion. So this patchset makes nr_deferred per-memcg to tackle the problem. It do= es: * Have memcg_shrinker_deferred per memcg per node, just like what shr= inker_map does. Instead it is an atomic_long_t array, each element represent = one shrinker even though the shrinker is not memcg aware, this simplifies the im= plementation. For memcg aware shrinkers, the deferred objects are just accumulate= d to its own memcg. The shrinkers just see nr_deferred from its own memcg. Non m= emcg aware shrinkers still use global nr_deferred from struct shrinker. * Once the memcg is offlined, its nr_deferred will be reparented to i= ts parent along with LRUs. * The root memcg has memcg_shrinker_deferred array too. It simplifies= the handling of reparenting to root memcg. * Cap nr_deferred to 2x of the length of lru. The idea is borrowed fr= om Dave Chinner's series (https://lore.kernel.org/linux-xfs/20191031234618.15403-1-da= vid@fromorbit.com/) The downside is each memcg has to allocate extra memory to store the nr_d= eferred array. On our production environment, there are typically around 40 shrinkers, s= o each memcg needs ~320 bytes. 10K memcgs would need ~3.2MB memory. It seems fine. We have been running the patched kernel on some hosts of our fleet (test = and production) for months, it works very well. The monitor data shows the working set is sus= tained as expected. Yang Shi (13): mm: vmscan: use nid from shrink_control for tracepoint mm: vmscan: consolidate shrinker_maps handling code mm: vmscan: use shrinker_rwsem to protect shrinker_maps allocation mm: vmscan: remove memcg_shrinker_map_size mm: vmscan: use kvfree_rcu instead of call_rcu mm: memcontrol: rename shrinker_map to shrinker_info mm: vmscan: add shrinker_info_protected() helper mm: vmscan: use a new flag to indicate shrinker is registered mm: vmscan: add per memcg shrinker nr_deferred mm: vmscan: use per memcg nr_deferred of shrinker mm: vmscan: don't need allocate shrinker->nr_deferred for memcg awa= re shrinkers mm: memcontrol: reparent nr_deferred when memcg offline mm: vmscan: shrink deferred objects proportional to priority include/linux/memcontrol.h | 23 +++--- include/linux/shrinker.h | 7 +- mm/huge_memory.c | 4 +- mm/list_lru.c | 6 +- mm/memcontrol.c | 130 +------------------------------ mm/vmscan.c | 394 +++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++------------------------ 6 files changed, 319 insertions(+), 245 deletions(-)