From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1580DC433DB for ; Fri, 19 Feb 2021 09:11:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 67E3A64EB1 for ; Fri, 19 Feb 2021 09:11:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 67E3A64EB1 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=suse.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D3D548D0006; Fri, 19 Feb 2021 04:11:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CEEFA8D0001; Fri, 19 Feb 2021 04:11:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C2CA18D0006; Fri, 19 Feb 2021 04:11:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0204.hostedemail.com [216.40.44.204]) by kanga.kvack.org (Postfix) with ESMTP id AE2968D0001 for ; Fri, 19 Feb 2021 04:11:10 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 7A0326C0B for ; Fri, 19 Feb 2021 09:11:10 +0000 (UTC) X-FDA: 77834448300.22.B52F3D2 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf22.hostedemail.com (Postfix) with ESMTP id ADBC4C000C6F for ; Fri, 19 Feb 2021 09:11:07 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1613725868; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=HK3gOMvSsOerVFX2l1WmjV6t+45SWNGfAyfy9D0NI4I=; b=l0gWdfjvB023KeFjTf+7ola/XiStFn7J6ZbR1Yk8PqvRYBWWwvxUJ6gADxWP5iemoq8Wne k/GZWOdhCUSjNyryLo2qew+rt6guUCiEVRRZRacQT4V82Hj0pqrhSdEHFx7JcVJPKOZUyu 1YDuCYmEasD1wBLaqseB+XSD8qvxG4E= Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 6F7E3B114; Fri, 19 Feb 2021 09:11:08 +0000 (UTC) Date: Fri, 19 Feb 2021 10:11:02 +0100 From: Michal Hocko To: Tim Chen Cc: Andrew Morton , Johannes Weiner , Vladimir Davydov , Dave Hansen , Ying Huang , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2 2/3] mm: Force update of mem cgroup soft limit tree on usage excess Message-ID: References: <06f1f92f1f7d4e57c4e20c97f435252c16c60a27.1613584277.git.tim.c.chen@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <06f1f92f1f7d4e57c4e20c97f435252c16c60a27.1613584277.git.tim.c.chen@linux.intel.com> X-Stat-Signature: bnq89iddns5mmsiohhh6wm6ehj5396da X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: ADBC4C000C6F Received-SPF: none (suse.com>: No applicable sender policy available) receiver=imf22; identity=mailfrom; envelope-from=""; helo=mx2.suse.de; client-ip=195.135.220.15 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1613725867-392370 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed 17-02-21 12:41:35, Tim Chen wrote: > To rate limit updates to the mem cgroup soft limit tree, we only perform > updates every SOFTLIMIT_EVENTS_TARGET (defined as 1024) memory events. > > However, this sampling based updates may miss a critical update: i.e. when > the mem cgroup first exceeded its limit but it was not on the soft limit tree. > It should be on the tree at that point so it could be subjected to soft > limit page reclaim. If the mem cgroup had few memory events compared with > other mem cgroups, we may not update it and place in on the mem cgroup > soft limit tree for many memory events. And this mem cgroup excess > usage could creep up and the mem cgroup could be hidden from the soft > limit page reclaim for a long time. > > Fix this issue by forcing an update to the mem cgroup soft limit tree if a > mem cgroup has exceeded its memory soft limit but it is not on the mem > cgroup soft limit tree. Let me copy your clarification from the other reply (this should go to the changelog btw.): > The sceanrio I saw was we have multiple cgroups running pmbench. > One cgroup exceeded the soft limit and soft reclaim is active on > that cgroup. So there are a whole bunch of memcg events associated > with that cgroup. Then another cgroup starts to exceed its > soft limit. > > Memory is accessed at a much lower frequency > for the second cgroup. The memcg event update was not triggered for the > second cgroup as the memcg event update didn't happened on the 1024th sample. > The second cgroup was not placed on the soft limit tree and we didn't > try to reclaim the excess pages. > > As time goes on, we saw that the first cgroup was kept close to its > soft limit due to reclaim activities, while the second cgroup's memory > usage slowly creep up as it keeps getting missed from the soft limit tree > update as the update didn't fall on the modulo 1024 sample. As a result, > the memory usage of the second cgroup keeps growing over the soft limit > for a long time due to its relatively rare occurrence. Soft limit is evaluated every THRESHOLDS_EVENTS_TARGET * SOFTLIMIT_EVENTS_TARGET. If all events correspond with a newly charged memory and the last event was just about the soft limit boundary then we should be bound by 128k pages (512M and much more if this were huge pages) which is a lot! I haven't realized this was that much. Now I see the problem. This would be a useful information for the changelog. Your fix is focusing on the over-the-limit boundary which will solve the problem but wouldn't that lead to to updates happening too often in pathological situation when a memcg would get reclaimed immediatelly? One way around that would be to lower the SOFTLIMIT_EVENTS_TARGET. Have you tried that? Do we even need a separate treshold for soft limit, why cannot we simply update the tree each MEM_CGROUP_TARGET_THRESH? > Reviewed-by: Ying Huang > Signed-off-by: Tim Chen > --- > mm/memcontrol.c | 11 +++++++++-- > 1 file changed, 9 insertions(+), 2 deletions(-) > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c > index a51bf90732cb..d72449eeb85a 100644 > --- a/mm/memcontrol.c > +++ b/mm/memcontrol.c > @@ -985,15 +985,22 @@ static bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg, > */ > static void memcg_check_events(struct mem_cgroup *memcg, struct page *page) > { > + struct mem_cgroup_per_node *mz; > + bool force_update = false; > + > + mz = mem_cgroup_nodeinfo(memcg, page_to_nid(page)); > + if (mz && !mz->on_tree && soft_limit_excess(mz->memcg) > 0) > + force_update = true; > + > /* threshold event is triggered in finer grain than soft limit */ > - if (unlikely(mem_cgroup_event_ratelimit(memcg, > + if (unlikely((force_update) || mem_cgroup_event_ratelimit(memcg, > MEM_CGROUP_TARGET_THRESH))) { > bool do_softlimit; > > do_softlimit = mem_cgroup_event_ratelimit(memcg, > MEM_CGROUP_TARGET_SOFTLIMIT); > mem_cgroup_threshold(memcg); > - if (unlikely(do_softlimit)) > + if (unlikely((force_update) || do_softlimit)) > mem_cgroup_update_tree(memcg, page); > } > } > -- > 2.20.1 -- Michal Hocko SUSE Labs