From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF900C433FE for ; Sun, 6 Dec 2020 08:31:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6F56423108 for ; Sun, 6 Dec 2020 08:31:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6F56423108 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 06DBA8D0003; Sun, 6 Dec 2020 03:31:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0448B6B0071; Sun, 6 Dec 2020 03:31:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E772A8D0003; Sun, 6 Dec 2020 03:31:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0110.hostedemail.com [216.40.44.110]) by kanga.kvack.org (Postfix) with ESMTP id D121A6B006E for ; Sun, 6 Dec 2020 03:31:48 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 93486181AEF10 for ; Sun, 6 Dec 2020 08:31:48 +0000 (UTC) X-FDA: 77562189096.07.bean13_0e002ca273d4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id 6F81F1803F9AA for ; Sun, 6 Dec 2020 08:31:48 +0000 (UTC) X-HE-Tag: bean13_0e002ca273d4 X-Filterd-Recvd-Size: 8766 Received: from mail-pj1-f65.google.com (mail-pj1-f65.google.com [209.85.216.65]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Sun, 6 Dec 2020 08:31:47 +0000 (UTC) Received: by mail-pj1-f65.google.com with SMTP id m5so5671196pjv.5 for ; Sun, 06 Dec 2020 00:31:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cL9BVbVQLgAGW6GpXAhL1X3CA8w9xPKU91tirmQZrTs=; b=x8fvuLarCFgaQTRPR4ffz/JJqgV+fhAwSGezh2EEVl+aDi7Dz1rixLAZS3bee4/Cjk yTGJNzm19fwdVNSUefGAt/1Wpp4o+VzhKWewiGyppNqXwi0CgEQsI/hZ1WPDO5K5ydIQ eRGdBfLsH4lswDedLsHI/qspqiRz4LmeZfVhabRi4uvmhk4Dq7H60Zlrvh1Rt7dIufz3 J1FWj8pGKuya9oJVeHptAgoqD1GoWwEJChbD0OAh6qXkKyV5GZqTOeqfmnt/JzlOVNcR rYXgLQfzO6U/LyVw7BGLIdIbkmEnYVEFW95PERZbdV2rsZqnmmBlkC2DdbYTpQk4MRQ9 H+4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cL9BVbVQLgAGW6GpXAhL1X3CA8w9xPKU91tirmQZrTs=; b=iX4k5fqx3EepMk7MltSU6hJGNDfDS/ft0CJS2yncPK9yLfBvckrEkeWeIEPufQfKVx QpqNDIIKrBo5R29UaK3Acgs1vv1xqfjtD48FOHZOkMYQpnW7ehLK1jMtuOo0sZJVDiTk FUdn4nxV2XyGT0Me+i/2jwP33cFtf+0/FMInsu+QIcgD3gy6xEktPvslXrD+DNGYfyyG I+5oZJFZgVpRuA9wo4PJMBSGbJ/HnDUSoN7cDW1wNejYZBk+pHKAhSzqSSUJKIm+xMiq jbcc/FtHyLoU3wRVCdTUw/g19AZK35z3z4/t42rf6g/ewhHggyEzUAPHrN2oaamrl/8X l1Pg== X-Gm-Message-State: AOAM533MbfilCbkNriQYI70vyafs6a75MDAfIAPnSnKm9VyPPtaYPwNm vypW3oGuruKNBPRlJ2EEOgWbAg== X-Google-Smtp-Source: ABdhPJzglciP6WEF9pyhvIJXUJQdKJTvf5/6EhbcvbjvVcthuUO6f4w5qWd4LOe+Qy42RJZ84jcY1Q== X-Received: by 2002:a17:902:be11:b029:da:ba30:5791 with SMTP id r17-20020a170902be11b02900daba305791mr11119985pls.13.1607243507099; Sun, 06 Dec 2020 00:31:47 -0800 (PST) Received: from localhost.localdomain ([103.136.221.70]) by smtp.gmail.com with ESMTPSA id c2sm10229107pfa.59.2020.12.06.00.31.39 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 06 Dec 2020 00:31:46 -0800 (PST) From: Muchun Song To: gregkh@linuxfoundation.org, rafael@kernel.org, adobriyan@gmail.com, akpm@linux-foundation.org, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, hughd@google.com, will@kernel.org, guro@fb.com, rppt@kernel.org, tglx@linutronix.de, esyr@redhat.com, peterx@redhat.com, krisman@collabora.com, surenb@google.com, avagin@openvz.org, elver@google.com, rdunlap@infradead.org, iamjoonsoo.kim@lge.com Cc: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, Muchun Song Subject: [PATCH v2 10/12] mm: memcontrol: scale stat_threshold for byted-sized vmstat Date: Sun, 6 Dec 2020 16:29:46 +0800 Message-Id: <20201206082948.11812-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20201206082948.11812-1-songmuchun@bytedance.com> References: <20201206082948.11812-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some vmstat counters are being accounted in bytes not pages, so the stat_threshold should also scale to bytes. The vmstat counters are already long type for memcg (can reference to struct lruvec_stat). For the global per-node vmstat counters also can scale to long. But the maximum vmstat threshold is 125, so the type of s32 is enough. Signed-off-by: Muchun Song --- include/linux/mmzone.h | 17 ++++++----------- include/linux/vmstat.h | 1 - mm/vmstat.c | 24 +++++++++++++----------- 3 files changed, 19 insertions(+), 23 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 1f9c83778629..d53328551225 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -216,17 +216,12 @@ enum node_stat_item { */ static __always_inline bool vmstat_item_in_bytes(int idx) { - /* - * Global and per-node slab counters track slab pages. - * It's expected that changes are multiples of PAGE_SIZE. - * Internally values are stored in pages. - * - * Per-memcg and per-lruvec counters track memory, consumed - * by individual slab objects. These counters are actually - * byte-precise. - */ return (idx =3D=3D NR_SLAB_RECLAIMABLE_B || - idx =3D=3D NR_SLAB_UNRECLAIMABLE_B); + idx =3D=3D NR_SLAB_UNRECLAIMABLE_B || +#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) + idx =3D=3D NR_KERNEL_SCS_B || +#endif + idx =3D=3D NR_KERNEL_STACK_B); } =20 /* @@ -340,7 +335,7 @@ struct per_cpu_pageset { =20 struct per_cpu_nodestat { s8 stat_threshold; - s8 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS]; + s32 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS]; }; =20 #endif /* !__GENERATING_BOUNDS.H */ diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index fd1a3d5d4926..afd84dc2398c 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -211,7 +211,6 @@ static inline unsigned long global_node_page_state(en= um node_stat_item item) { long x =3D atomic_long_read(&vm_node_stat[item]); =20 - VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); #ifdef CONFIG_SMP if (x < 0) x =3D 0; diff --git a/mm/vmstat.c b/mm/vmstat.c index 7fb0c7cb9516..25751b1d8e2e 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -341,13 +341,15 @@ void __mod_node_page_state(struct pglist_data *pgda= t, enum node_stat_item item, long delta) { struct per_cpu_nodestat __percpu *pcp =3D pgdat->per_cpu_nodestats; - s8 __percpu *p =3D pcp->vm_node_stat_diff + item; + s32 __percpu *p =3D pcp->vm_node_stat_diff + item; long x; long t; =20 x =3D delta + __this_cpu_read(*p); =20 t =3D __this_cpu_read(pcp->stat_threshold); + if (vmstat_item_in_bytes(item)) + t <<=3D PAGE_SHIFT; =20 if (unlikely(abs(x) > t)) { node_page_state_add(x, pgdat, item); @@ -399,15 +401,15 @@ void __inc_zone_state(struct zone *zone, enum zone_= stat_item item) void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item ite= m) { struct per_cpu_nodestat __percpu *pcp =3D pgdat->per_cpu_nodestats; - s8 __percpu *p =3D pcp->vm_node_stat_diff + item; - s8 v, t; + s32 __percpu *p =3D pcp->vm_node_stat_diff + item; + s32 v, t; =20 VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); =20 v =3D __this_cpu_inc_return(*p); t =3D __this_cpu_read(pcp->stat_threshold); if (unlikely(v > t)) { - s8 overstep =3D t >> 1; + s32 overstep =3D t >> 1; =20 node_page_state_add(v + overstep, pgdat, item); __this_cpu_write(*p, -overstep); @@ -445,8 +447,8 @@ void __dec_zone_state(struct zone *zone, enum zone_st= at_item item) void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item ite= m) { struct per_cpu_nodestat __percpu *pcp =3D pgdat->per_cpu_nodestats; - s8 __percpu *p =3D pcp->vm_node_stat_diff + item; - s8 v, t; + s32 __percpu *p =3D pcp->vm_node_stat_diff + item; + s32 v, t; =20 VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); =20 @@ -546,7 +548,7 @@ static inline void mod_node_state(struct pglist_data = *pgdat, enum node_stat_item item, int delta, int overstep_mode) { struct per_cpu_nodestat __percpu *pcp =3D pgdat->per_cpu_nodestats; - s8 __percpu *p =3D pcp->vm_node_stat_diff + item; + s32 __percpu *p =3D pcp->vm_node_stat_diff + item; long o, n, t, z; =20 do { @@ -563,6 +565,8 @@ static inline void mod_node_state(struct pglist_data = *pgdat, * for all cpus in a node. */ t =3D this_cpu_read(pcp->stat_threshold); + if (vmstat_item_in_bytes(item)) + t <<=3D PAGE_SHIFT; =20 o =3D this_cpu_read(*p); n =3D delta + o; @@ -829,7 +833,7 @@ static int refresh_cpu_vm_stats(bool do_pagesets) struct per_cpu_nodestat __percpu *p =3D pgdat->per_cpu_nodestats; =20 for (i =3D 0; i < NR_VM_NODE_STAT_ITEMS; i++) { - int v; + s32 v; =20 v =3D this_cpu_xchg(p->vm_node_stat_diff[i], 0); if (v) { @@ -899,7 +903,7 @@ void cpu_vm_stats_fold(int cpu) =20 for (i =3D 0; i < NR_VM_NODE_STAT_ITEMS; i++) if (p->vm_node_stat_diff[i]) { - int v; + s32 v; =20 v =3D p->vm_node_stat_diff[i]; p->vm_node_stat_diff[i] =3D 0; @@ -1017,8 +1021,6 @@ unsigned long node_page_state(struct pglist_data *p= gdat, { long x =3D atomic_long_read(&pgdat->vm_stat[item]); =20 - VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); - #ifdef CONFIG_SMP if (x < 0) x =3D 0; --=20 2.11.0