From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E41CC433DB for ; Thu, 18 Feb 2021 19:51:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A54D264EB4 for ; Thu, 18 Feb 2021 19:51:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A54D264EB4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EE1E66B0006; Thu, 18 Feb 2021 14:51:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E908B6B006C; Thu, 18 Feb 2021 14:51:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D581F6B006E; Thu, 18 Feb 2021 14:51:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0134.hostedemail.com [216.40.44.134]) by kanga.kvack.org (Postfix) with ESMTP id BAD106B0006 for ; Thu, 18 Feb 2021 14:51:32 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 8045C181AF5F7 for ; Thu, 18 Feb 2021 19:51:32 +0000 (UTC) X-FDA: 77832433224.01.05EF906 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf27.hostedemail.com (Postfix) with ESMTP id 7A28E80192DD for ; Thu, 18 Feb 2021 19:51:27 +0000 (UTC) IronPort-SDR: 3q7KtfOLlQhRstjVpuYDr59QGr05Wqwwv/yL76wgvVYCO4Td36AbUbqTwUXE5B9XJ+0Blw0kor ksGrZmh6W8Iw== X-IronPort-AV: E=McAfee;i="6000,8403,9899"; a="202885154" X-IronPort-AV: E=Sophos;i="5.81,187,1610438400"; d="scan'208";a="202885154" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Feb 2021 11:51:29 -0800 IronPort-SDR: Laxqy638jBahXM/Nn1uEjjSLbDexQjGW13cAai0nmMz4tPOw4aaSlZ/8c1Uy4n9jHYq8NZ4hb4 q3aVtuPVbnDA== X-IronPort-AV: E=Sophos;i="5.81,187,1610438400"; d="scan'208";a="428366282" Received: from schen9-mobl.amr.corp.intel.com ([10.254.101.217]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Feb 2021 11:51:29 -0800 Subject: Re: [PATCH v2 1/3] mm: Fix dropped memcg from mem cgroup soft limit tree To: Michal Hocko Cc: Andrew Morton , Johannes Weiner , Vladimir Davydov , Dave Hansen , Ying Huang , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org References: <8d35206601ccf0e1fe021d24405b2a0c2f4e052f.1613584277.git.tim.c.chen@linux.intel.com> From: Tim Chen Message-ID: <286b1ade-46d9-2fca-1641-69de4694deb1@linux.intel.com> Date: Thu, 18 Feb 2021 11:51:27 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 7A28E80192DD X-Stat-Signature: yu3tz8oby6fkzi8g8zpfk9wc61yjrh5g Received-SPF: none (linux.intel.com>: No applicable sender policy available) receiver=imf27; identity=mailfrom; envelope-from=""; helo=mga01.intel.com; client-ip=192.55.52.88 X-HE-DKIM-Result: none/none X-HE-Tag: 1613677887-11730 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2/18/21 11:13 AM, Michal Hocko wrote: > On Thu 18-02-21 10:30:20, Tim Chen wrote: >> >> >> On 2/18/21 12:24 AM, Michal Hocko wrote: >> >>> >>> I have already acked this patch in the previous version along with Fixes >>> tag. It seems that my review feedback has been completely ignored also >>> for other patches in this series. >> >> Michal, >> >> My apology. Our mail system screwed up and there are some mail missing >> from our mail system that I completely missed your mail. >> Only saw them now after I looked into the lore.kernel.org. > > I see. My apology for suspecting you from ignoring my review. > >> Responding to your comment: >> >>> Have you observed this happening in the real life? I do agree that the >>> threshold based updates of the tree is not ideal but the whole soft >>> reclaim code is far from optimal. So why do we care only now? The >>> feature is essentially dead and fine tuning it sounds like a step back >>> to me. >> >> Yes, I did see the issue mentioned in patch 2 breaking soft limit >> reclaim for cgroup v1. There are still some of our customers using >> cgroup v1 so we will like to fix this if possible. > > It would be great to see more details. > The sceanrio I saw was we have multiple cgroups running pmbench. One cgroup exceeded the soft limit and soft reclaim is active on that cgroup. So there are a whole bunch of memcg events associated with that cgroup. Then another cgroup starts to exceed its soft limit. Memory is accessed at a much lower frequency for the second cgroup. The memcg event update was not triggered for the second cgroup as the memcg event update didn't happened on the 1024th sample. The second cgroup was not placed on the soft limit tree and we didn't try to reclaim the excess pages. As time goes on, we saw that the first cgroup was kept close to its soft limit due to reclaim activities, while the second cgroup's memory usage slowly creep up as it keeps getting missed from the soft limit tree update as the update didn't fall on the modulo 1024 sample. As a result, the memory usage of the second cgroup keeps growing over the soft limit for a long time due to its relatively rare occurrence. >> For patch 3 regarding the uncharge_batch, it >> is more of an observation that we should uncharge in batch of same node >> and not prompted by actual workload. >> Thinking more about this, the worst that could happen >> is we could have some entries in the soft limit tree that overestimate >> the memory used. The worst that could happen is a soft page reclaim >> on that cgroup. The overhead from extra memcg event update could >> be more than a soft page reclaim pass. So let's drop patch 3 >> for now. > > I would still prefer to handle that in the soft limit reclaim path and > check each memcg for the soft limit reclaim excess before the reclaim. > Something like this? diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 8bddee75f5cb..b50cae3b2a1a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3472,6 +3472,14 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, if (!mz) break; + /* + * Soft limit tree is updated based on memcg events sampling. + * We could have missed some updates on page uncharge and + * the cgroup is below soft limit. Skip useless soft reclaim. + */ + if (!soft_limit_excess(mz->memcg)) + continue; + nr_scanned = 0; reclaimed = mem_cgroup_soft_reclaim(mz->memcg, pgdat, Tim