From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D2FEC001B0 for ; Wed, 19 Jul 2023 17:46:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9585828007A; Wed, 19 Jul 2023 13:46:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9077828004C; Wed, 19 Jul 2023 13:46:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CF8F28007A; Wed, 19 Jul 2023 13:46:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6E80728004C for ; Wed, 19 Jul 2023 13:46:21 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1F441403BD for ; Wed, 19 Jul 2023 17:46:21 +0000 (UTC) X-FDA: 81029090562.06.6B9A548 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 559F0180017 for ; Wed, 19 Jul 2023 17:46:18 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=qQcHhj0f; spf=pass (imf24.hostedemail.com: domain of 3aCG4ZAoKCLcvlpovXejbadlldib.Zljifkru-jjhsXZh.lod@flex--yosryahmed.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3aCG4ZAoKCLcvlpovXejbadlldib.Zljifkru-jjhsXZh.lod@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689788778; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=6e7KuyR5qZ6xCXotERLteVBuiZtk31A4XosqK7/beqs=; b=OizEMNwT3kidDy1sr7Hipur7hxJY0QfYEULMkwUENP6kqM0bivu6KUKiwBA2r4I1DktDBx oVioyzI0jyd6F13X1AxyA5XkiJlcOrzSOVOrfFjNZt22D2eQ7uJYDpefXOKyNX62MkVgbX QvVt3HWLuYDptxe4cuz+11XQ6s0tq3s= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=qQcHhj0f; spf=pass (imf24.hostedemail.com: domain of 3aCG4ZAoKCLcvlpovXejbadlldib.Zljifkru-jjhsXZh.lod@flex--yosryahmed.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3aCG4ZAoKCLcvlpovXejbadlldib.Zljifkru-jjhsXZh.lod@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689788778; a=rsa-sha256; cv=none; b=ByWvZAF2mbBI8pJmcNGDep5D7g7T2FEah0EaEoVSdoaiIqlSZXjyEYIxEnUNpfHZVknwdn zUByEe6r80bCYSZ2VbmChEUqnL686NxiqCmIHdCltMWiREfoC9A1pnuSd6A8eIDtBY8Yo2 AM1xo6GcgFRDtkQwupIILKMafdfOR4I= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-1b89f6463deso56163185ad.1 for ; Wed, 19 Jul 2023 10:46:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689788777; x=1692380777; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=6e7KuyR5qZ6xCXotERLteVBuiZtk31A4XosqK7/beqs=; b=qQcHhj0foniGWDDkDgux7VVWqWaFxDGN+dbzuBsFo30NW6axCLOXrULxehYv6yMHwO jH8UyWpvMU89UheqpLWuY6AceR7QLFibsdGjWeBHNc4UVV7JAvKCHe+DohloqVnlnsji ogeDFvWIQSpEqF11Bn77AHuju0IchimGYa8p1bIZvDMDediUM4PW9ts4U8YkzJIa5x9a tQYoEjbHB3KdO3hxgnScIFJG5V/1KNIo6cJRQRkAbFtYttlUH1CHzaGUF3Qhf/MJ7iLy wMWj+q+IOdC7KZsTRwCYrC2HWbvNHpWR762aG94pnusjfJNKUEQG9aOKJYgksrUzGeX1 Qhag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689788777; x=1692380777; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=6e7KuyR5qZ6xCXotERLteVBuiZtk31A4XosqK7/beqs=; b=IGZCp1gBhbS5d+psJmFJKTc14xxczf0DyPFZj6PjooirIkJPikwRA0qdZE3gfyi17J LkPaCjSvHubreHJ2SmpI02EAuob34T6E7Jgj1vBYzhhi7VdLNC/ClnWvOCu3WdXb3gDn wlSUMFZ5eViPfG71SLxoPafct6ccqpbCQcH0DetoPn+V3jSREdphZDvkWABB7Q1JbciW zine6hhyNFlDQxJ7uEtTKCpaOFXlICty97zw8Kkx+sRuF7ldbssdnP15Vd0loS6EuK8d 71Yso6e2oSmADzkASOogr8/C3dEK+/xAvq3ebQhZwXEdb5h4oHX8MiMJKs5Und7GGSJ7 iElg== X-Gm-Message-State: ABy/qLZZYDsgXt0k1xTn3EVqtj4+Een/sc5hFmfX9aq/W8CzrZnpSdTp zA0lD22vBvcA1vvtOS2a6d90Opsy3K2k8DLU X-Google-Smtp-Source: APBJJlEUiRxTpsrvbnGGBXTcFBfqoa+OttLra1eEzXRriYhTXWucHEY3BX2wQBiQaWUu6OO/SRuy5386wIsNmSAq X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:903:2450:b0:1b3:e4f1:1b3f with SMTP id l16-20020a170903245000b001b3e4f11b3fmr17242pls.2.1689788776892; Wed, 19 Jul 2023 10:46:16 -0700 (PDT) Date: Wed, 19 Jul 2023 17:46:13 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230719174613.3062124-1-yosryahmed@google.com> Subject: [PATCH] mm: memcg: use rstat for non-hierarchical stats From: Yosry Ahmed To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 559F0180017 X-Rspam-User: X-Stat-Signature: 84mogbcxbfgt8f7wncxefb5wrupony1j X-Rspamd-Server: rspam01 X-HE-Tag: 1689788778-200444 X-HE-Meta: U2FsdGVkX1+yUG3Mpgs+R/03/OEAXkTIntqXBAJ3oewtA4yhi705QZIQJU7MGxoR9dvdoKZ54QYOzpS/4tJi+TLvzZ+fiWoigcR+iqdmZJzdwLE/SB/ZBD/lJjzILQr+G6El0CDHTjaKOz3uF7YlyYpdFCk1R6DHno1oBdO//WAHQ4OdWtGDsS5XqkMkxau8eWaoEIIz1ykacXVOEVb1t44bdorsgIuS83yqlm1Em+9JuDFO0DFjLTZdkU27iRqNelDDDauj4nowsRvWkf/8UYlEX9wXzn16aMXsYQr+iUEzrdRoCeR1LyqFMAl4GBXKYScFgg02isP5qHPRU7AFHGXCNwTWv57MS1qNkkRWw861stvhDopZ8Lca4NKitetPWozzWlBe98qE8wH2c4GngWd513Q+iFV188vIUpKjSZyd71pMaedeEe4n435KKZn/400zTuNdAFrECoxXAiJ/4bgX6wNAZrHUM28Z3PHxCLRgYQROAirUjSoYiL7S4b/qrLbJDSf5kxT8v/5mowsn8Xjfv2bTASPyq/rWe6lxWjopISMn+88F5EywS8J2vHqY9P5nPpSikO3JjWFkvAOVpHsaIHnCGoFgqT1Ae0IFX8makBNLD65ASR6oe7SoiQi1UfhFxqocgLo1ZCph8tKqK71XZGGg2gUCCAIstCnm0k6fKa6qf5p5hMh2P00jJrHBc96tKf5q3hn3y3yOeZx7J2m3kdQfdTaQTbyfoS1SGtlMfhuTLgxrk3r9Sj7z8g4a9b76qMCbWL+XpCaQme9H70HtgWpI7bYVVCBufh7jeJ5CmIjbiCYQh5sU3dZdjsr51y290TIWUTHkWU58IUknBmHsBn9N7JcYipzzh9Y0CnuFKEJRKG1bM0skUrmnxH6cSntbct3Evv7sdSPDJqAkzoPpVqLHKsCvpFLMNV1iilkcFZiSiMUxYt75crG6BYPEbBIAdBMc5RmwRZgTBG/ qQu4+mS2 qad3hGBI49ZKGv53kINbRSmX7JqG+z8tFnyU3sItdsMGKes4rpoQ2pk0Rs/S15sDLvgeMTXGSiwiRGCxvZZgs5nMuogu9pBC0ulhDN4vRcUtxj4kYDPhiUd5g3s0f3EyMe/YzJH7NvihB+aYTCxxnstsU6wXniMAW5WelRjv/SdUyyDBPaSpTbnvsBTbJDgGSmJZHmyvaO7b8w1L1NeKElJSgJlMFSkZGTL6uEuFSyYBWiVrmtTboUKU0pzsk/+dEEHiM8IGT22FP+qFjoyUFgHftPFrjkVoezjETqD4O4P1nwHDA52pwB6MyQbt8qZfumib12oN8DlssWNoHnA9ZdPcKIOCwFFl8LqBKFLuEe967Hf1k3XVubYNeEvC1F2tVIrmqjdA5sjXx3XAV8z3oz4WdJVvyB1P4gC8Yqf8XQlWUOZCXUx7U9cdreprGrNSh5cD8NhluCtbWPSjtap264HhbCfxZoFBzXtFgRDU5d4m3byNCgtMAsqhEQbPtjKr2jy2P6ql90B3KbDOiaCRGkHTq6E6j1bImrxdwkMJq4D12zGC8UYPQQpfhVsBwrF1SWS+E536f6S6gzjDwDbq3sLdDIwkMJt+wLrqQkEGaIZgf70WKdTNHP0+vcJI122SMNeORBY0PtjlUJy4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, memcg uses rstat to maintain hierarchical stats. The rstat framework keeps track of which cgroups have updates on which cpus. For non-hierarchical stats, as memcg moved to rstat, they are no longer readily available as counters. Instead, the percpu counters for a given stat need to be summed to get the non-hierarchical stat value. This causes a performance regression when reading non-hierarchical stats on kernels where memcg moved to using rstat. This is especially visible when reading memory.stat on cgroup v1. There are also some code paths internal to the kernel that read such non-hierarchical stats. It is inefficient to iterate and sum counters in all cpus when the rstat framework knows exactly when a percpu counter has an update. Instead, maintain cpu-aggregated non-hierarchical counters for each stat. During an rstat flush, keep those updated as well. When reading non-hierarchical stats, we no longer need to iterate cpus, we just need to read the maintainer counters, similar to hierarchical stats. A caveat is that we now a stats flush before reading local/non-hierarchical stats through {memcg/lruvec}_page_state_local() or memcg_events_local(), where we previously only needed a flush to read hierarchical stats. Most contexts reading non-hierarchical stats are already doing a flush, add a flush to the only missing context in count_shadow_nodes(). With this patch, reading memory.stat from 1000 memcgs is 3x faster on a machine with 256 cpus on cgroup v1: # for i in $(seq 1000); do mkdir /sys/fs/cgroup/memory/cg$i; done # time cat /dev/cgroup/memory/cg*/memory.stat > /dev/null real 0m0.125s user 0m0.005s sys 0m0.120s After: real 0m0.032s user 0m0.005s sys 0m0.027s Signed-off-by: Yosry Ahmed --- include/linux/memcontrol.h | 7 ++++--- mm/memcontrol.c | 32 +++++++++++++++++++------------- mm/workingset.c | 1 + 3 files changed, 24 insertions(+), 16 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 5818af8eca5a..a9f2861a57a5 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -112,6 +112,9 @@ struct lruvec_stats { /* Aggregated (CPU and subtree) state */ long state[NR_VM_NODE_STAT_ITEMS]; + /* Non-hierarchical (CPU aggregated) state */ + long state_local[NR_VM_NODE_STAT_ITEMS]; + /* Pending child counts during tree propagation */ long state_pending[NR_VM_NODE_STAT_ITEMS]; }; @@ -1020,14 +1023,12 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, { struct mem_cgroup_per_node *pn; long x = 0; - int cpu; if (mem_cgroup_disabled()) return node_page_state(lruvec_pgdat(lruvec), idx); pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - for_each_possible_cpu(cpu) - x += per_cpu(pn->lruvec_stats_percpu->state[idx], cpu); + x = READ_ONCE(pn->lruvec_stats.state_local[idx]); #ifdef CONFIG_SMP if (x < 0) x = 0; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e8ca4bdcb03c..90a22637818e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -742,6 +742,10 @@ struct memcg_vmstats { long state[MEMCG_NR_STAT]; unsigned long events[NR_MEMCG_EVENTS]; + /* Non-hierarchical (CPU aggregated) page state & events */ + long state_local[MEMCG_NR_STAT]; + unsigned long events_local[NR_MEMCG_EVENTS]; + /* Pending child counts during tree propagation */ long state_pending[MEMCG_NR_STAT]; unsigned long events_pending[NR_MEMCG_EVENTS]; @@ -775,11 +779,8 @@ void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val) /* idx can be of type enum memcg_stat_item or node_stat_item. */ static unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) { - long x = 0; - int cpu; + long x = READ_ONCE(memcg->vmstats->state_local[idx]); - for_each_possible_cpu(cpu) - x += per_cpu(memcg->vmstats_percpu->state[idx], cpu); #ifdef CONFIG_SMP if (x < 0) x = 0; @@ -926,16 +927,12 @@ static unsigned long memcg_events(struct mem_cgroup *memcg, int event) static unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) { - long x = 0; - int cpu; int index = memcg_events_index(event); if (index < 0) return 0; - for_each_possible_cpu(cpu) - x += per_cpu(memcg->vmstats_percpu->events[index], cpu); - return x; + return READ_ONCE(memcg->vmstats->events_local[index]); } static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, @@ -5526,7 +5523,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) struct mem_cgroup *memcg = mem_cgroup_from_css(css); struct mem_cgroup *parent = parent_mem_cgroup(memcg); struct memcg_vmstats_percpu *statc; - long delta, v; + long delta, delta_cpu, v; int i, nid; statc = per_cpu_ptr(memcg->vmstats_percpu, cpu); @@ -5542,9 +5539,11 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) memcg->vmstats->state_pending[i] = 0; /* Add CPU changes on this level since the last flush */ + delta_cpu = 0; v = READ_ONCE(statc->state[i]); if (v != statc->state_prev[i]) { - delta += v - statc->state_prev[i]; + delta_cpu = v - statc->state_prev[i]; + delta += delta_cpu; statc->state_prev[i] = v; } @@ -5553,6 +5552,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) /* Aggregate counts on this level and propagate upwards */ memcg->vmstats->state[i] += delta; + memcg->vmstats->state_local[i] += delta_cpu; if (parent) parent->vmstats->state_pending[i] += delta; } @@ -5562,9 +5562,11 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) if (delta) memcg->vmstats->events_pending[i] = 0; + delta_cpu = 0; v = READ_ONCE(statc->events[i]); if (v != statc->events_prev[i]) { - delta += v - statc->events_prev[i]; + delta_cpu = v - statc->events_prev[i]; + delta += delta_cpu; statc->events_prev[i] = v; } @@ -5572,6 +5574,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) continue; memcg->vmstats->events[i] += delta; + memcg->vmstats->events_local[i] += delta_cpu; if (parent) parent->vmstats->events_pending[i] += delta; } @@ -5591,9 +5594,11 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) if (delta) pn->lruvec_stats.state_pending[i] = 0; + delta_cpu = 0; v = READ_ONCE(lstatc->state[i]); if (v != lstatc->state_prev[i]) { - delta += v - lstatc->state_prev[i]; + delta_cpu = v - lstatc->state_prev[i]; + delta += delta_cpu; lstatc->state_prev[i] = v; } @@ -5601,6 +5606,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) continue; pn->lruvec_stats.state[i] += delta; + pn->lruvec_stats.state_local[i] += delta_cpu; if (ppn) ppn->lruvec_stats.state_pending[i] += delta; } diff --git a/mm/workingset.c b/mm/workingset.c index 4686ae363000..da58a26d0d4d 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -664,6 +664,7 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker, struct lruvec *lruvec; int i; + mem_cgroup_flush_stats(); lruvec = mem_cgroup_lruvec(sc->memcg, NODE_DATA(sc->nid)); for (pages = 0, i = 0; i < NR_LRU_LISTS; i++) pages += lruvec_page_state_local(lruvec, -- 2.41.0.255.g8b1d071c50-goog