From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70088C433EF for ; Fri, 1 Apr 2022 09:12:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232706AbiDAJOT (ORCPT ); Fri, 1 Apr 2022 05:14:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46188 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232156AbiDAJOS (ORCPT ); Fri, 1 Apr 2022 05:14:18 -0400 Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9DC2265EAB for ; Fri, 1 Apr 2022 02:12:28 -0700 (PDT) Received: by mail-pj1-x102d.google.com with SMTP id y16so1875065pju.4 for ; Fri, 01 Apr 2022 02:12:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=TTgjpbxi+atnp9axNVFkq7OGFubJ8g3qn22nFzknXc0=; b=BJ7yd3RqMpeyZP/nXaJIqtS0xR9zz6WFUei6+PbJx29crBeMBoajAHTDBFqT/NP/E9 oDskgt1nDRh5HaFRpNsKLfCBPspbWgH8cCGdFcCBdjHXQsgZe7KYIL2BHqwf89YC2KqE FU1D88az6OaXiSSpAJa0xhtQvr3n9BKcuFwA/YBZ/Jyf7/x9liXJJ0BGx8kvbP4eC8sV bnF/eQBu+fZeYpJPE1wiLaT1wHA93zccpcqbztUDXVgRcYURXvHnX+RbafbV1iWb4j5D gn0lTg7NhGi/NMUl3PMdZ11PMU3EVUfR9yFC1YuR3rOGIjL/88llWWZvDc4Oi9MsFlpW BsNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=TTgjpbxi+atnp9axNVFkq7OGFubJ8g3qn22nFzknXc0=; b=S3G0PpvaLAgle7U8szleJuVMclENhi+h4ebsag1fkrhiopNC4Qd0JrmkJClxBAt9rg ukWzMllQzlEFmsZDQxn9Un8LcqGuUYLKRiP0Q2aekbBVSY4s5VNwHzo9/8/OG1YklNBw c00leq+tiCCP1ypP6Q5QCaePmtVbDQ6uptEx/AF1v4DM4aX5R4+w8Z/atCxlYMqg5b3i VuTDgf37CdUNegSw/rGIZK31r+zTRnwcgjg97VlOKLtD9T+5v9Kt+0yhpLx2v5nrPbhk 60umCMNEaPwjK3Z+cZS/n6PR8b9WmlBizL6u2bMCPqazl5GMAFboF7rgOHIhqZNC09P7 R5lg== X-Gm-Message-State: AOAM532G8f+0RTk7y+ziXIr+O2iTFVJpeVW83DIdHqWHRN3XWxbekDvR U2fm/QWsAU0RP1ukkZVHPbyMo93qPqsWepxqhVFRNQ== X-Google-Smtp-Source: ABdhPJzopBl16/r1LJ0fm04p7JCvGevCb43y19JegZ3PEwne5+cHWxKVLGwBpnfGLKYJXujMrPBfvrr2QbDLUvnpMww= X-Received: by 2002:a17:902:f24b:b0:154:de5:a24c with SMTP id j11-20020a170902f24b00b001540de5a24cmr9507262plc.162.1648804348061; Fri, 01 Apr 2022 02:12:28 -0700 (PDT) MIME-Version: 1.0 References: <20220331084151.2600229-1-yosryahmed@google.com> In-Reply-To: From: Yosry Ahmed Date: Fri, 1 Apr 2022 02:11:51 -0700 Message-ID: Subject: Re: [PATCH resend] memcg: introduce per-memcg reclaim interface To: Roman Gushchin Cc: Johannes Weiner , Michal Hocko , Shakeel Butt , Andrew Morton , David Rientjes , Tejun Heo , Zefan Li , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Linux-MM , Jonathan Corbet , Yu Zhao , Dave Hansen , Wei Xu , Greg Thelen Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Mar 31, 2022 at 10:25 AM Roman Gushchin wrote: > > On Thu, Mar 31, 2022 at 08:41:51AM +0000, Yosry Ahmed wrote: > > From: Shakeel Butt > > > > Introduce an memcg interface to trigger memory reclaim on a memory cgroup. > > > > Use case: Proactive Reclaim > > --------------------------- > > > > A userspace proactive reclaimer can continuously probe the memcg to > > reclaim a small amount of memory. This gives more accurate and > > up-to-date workingset estimation as the LRUs are continuously > > sorted and can potentially provide more deterministic memory > > overcommit behavior. The memory overcommit controller can provide > > more proactive response to the changing behavior of the running > > applications instead of being reactive. > > > > A userspace reclaimer's purpose in this case is not a complete replacement > > for kswapd or direct reclaim, it is to proactively identify memory savings > > opportunities and reclaim some amount of cold pages set by the policy > > to free up the memory for more demanding jobs or scheduling new jobs. > > > > A user space proactive reclaimer is used in Google data centers. > > Additionally, Meta's TMO paper recently referenced a very similar > > interface used for user space proactive reclaim: > > https://dl.acm.org/doi/pdf/10.1145/3503222.3507731 > > > > Benefits of a user space reclaimer: > > ----------------------------------- > > > > 1) More flexible on who should be charged for the cpu of the memory > > reclaim. For proactive reclaim, it makes more sense to be centralized. > > > > 2) More flexible on dedicating the resources (like cpu). The memory > > overcommit controller can balance the cost between the cpu usage and > > the memory reclaimed. > > > > 3) Provides a way to the applications to keep their LRUs sorted, so, > > under memory pressure better reclaim candidates are selected. This also > > gives more accurate and uptodate notion of working set for an > > application. > > > > Why memory.high is not enough? > > ------------------------------ > > > > - memory.high can be used to trigger reclaim in a memcg and can > > potentially be used for proactive reclaim. > > However there is a big downside in using memory.high. It can potentially > > introduce high reclaim stalls in the target application as the > > allocations from the processes or the threads of the application can hit > > the temporary memory.high limit. > > > > - Userspace proactive reclaimers usually use feedback loops to decide > > how much memory to proactively reclaim from a workload. The metrics > > used for this are usually either refaults or PSI, and these metrics > > will become messy if the application gets throttled by hitting the > > high limit. > > > > - memory.high is a stateful interface, if the userspace proactive > > reclaimer crashes for any reason while triggering reclaim it can leave > > the application in a bad state. > > > > - If a workload is rapidly expanding, setting memory.high to proactively > > reclaim memory can result in actually reclaiming more memory than > > intended. > > > > The benefits of such interface and shortcomings of existing interface > > were further discussed in this RFC thread: > > https://lore.kernel.org/linux-mm/5df21376-7dd1-bf81-8414-32a73cea45dd@google.com/ > > Hello! > > I'm totally up for the proposed feature! It makes total sense and is proved > to be useful, let's add it. > > > > > Interface: > > ---------- > > > > Introducing a very simple memcg interface 'echo 10M > memory.reclaim' to > > trigger reclaim in the target memory cgroup. > > > > > > Possible Extensions: > > -------------------- > > > > - This interface can be extended with an additional parameter or flags > > to allow specifying one or more types of memory to reclaim from (e.g. > > file, anon, ..). > > > > - The interface can also be extended with a node mask to reclaim from > > specific nodes. This has use cases for reclaim-based demotion in memory > > tiering systens. > > > > - A similar per-node interface can also be added to support proactive > > reclaim and reclaim-based demotion in systems without memcg. > > Maybe an option to specify a timeout? That might simplify the userspace part. > Also, please please add a test to selftests/cgroup/memcg tests. > It will also provide an example on how the userspace can use the feature. > Hi Roman, thanks for taking the time to review this! A timeout can be a good extension, I will add it to the commit message in the next version in possible extensions. I will add a test in v2, thanks! > > > > > For now, let's keep things simple by adding the basic functionality. > > What I'm worried about is how we gonna extend it? How do you see the interface > with 2-3 extensions from the list above? All these extensions look very > reasonable to me, so we'll likely have to implement them soon. So let's think > about the extensibility now. > My idea is to have these extensions as optional positional arguments (like Wei suggested), so that the interface does not get too complicated for users who don't care about tuning these options. If this is the case then I think there is nothing to worry about. Otherwise, if you think some of these options make sense to be a required argument instead, we can rethink the initial interface. > I wonder if it makes more sense to introduce a sys_reclaim() syscall instead? > In the end, such a feature might make sense on the system level too. > Yes, there is the drop_caches sysctl, but it's too radical for many cases. > I think in the RFC discussion there was consensus to add both a per-memcg knob, as well as per-node / per-system knobs (through sysfs or syscalls) later. Wei also points out that it's not common for a syscall to have a cgroup argument. > > > > [yosryahmed@google.com: refreshed to current master, updated commit > > message based on recent discussions and use cases] > > Signed-off-by: Shakeel Butt > > Signed-off-by: Yosry Ahmed > > --- > > Documentation/admin-guide/cgroup-v2.rst | 9 ++++++ > > mm/memcontrol.c | 37 +++++++++++++++++++++++++ > > 2 files changed, 46 insertions(+) > > > > diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst > > index 69d7a6983f78..925aaabb2247 100644 > > --- a/Documentation/admin-guide/cgroup-v2.rst > > +++ b/Documentation/admin-guide/cgroup-v2.rst > > @@ -1208,6 +1208,15 @@ PAGE_SIZE multiple when read back. > > high limit is used and monitored properly, this limit's > > utility is limited to providing the final safety net. > > > > + memory.reclaim > > + A write-only file which exists on non-root cgroups. > > + > > + This is a simple interface to trigger memory reclaim in the > > + target cgroup. Write the number of bytes to reclaim to this > > + file and the kernel will try to reclaim that much memory. > > + Please note that the kernel can over or under reclaim from > > + the target cgroup. > > + > > memory.oom.group > > A read-write single value file which exists on non-root > > cgroups. The default value is "0". > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c > > index 725f76723220..994849fab7df 100644 > > --- a/mm/memcontrol.c > > +++ b/mm/memcontrol.c > > @@ -6355,6 +6355,38 @@ static ssize_t memory_oom_group_write(struct kernfs_open_file *of, > > return nbytes; > > } > > > > +static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf, > > + size_t nbytes, loff_t off) > > +{ > > + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); > > + unsigned int nr_retries = MAX_RECLAIM_RETRIES; > > + unsigned long nr_to_reclaim, nr_reclaimed = 0; > > + int err; > > + > > + buf = strstrip(buf); > > + err = page_counter_memparse(buf, "", &nr_to_reclaim); > > + if (err) > > + return err; > > + > > + while (nr_reclaimed < nr_to_reclaim) { > > + unsigned long reclaimed; > > + > > + if (signal_pending(current)) > > + break; > > + > > + reclaimed = try_to_free_mem_cgroup_pages(memcg, > > + nr_to_reclaim - nr_reclaimed, > > + GFP_KERNEL, true); > > + > > + if (!reclaimed && !nr_retries--) > > + break; > > + > > + nr_reclaimed += reclaimed; > > + } > > + > > + return nbytes; > > +} > > + > > static struct cftype memory_files[] = { > > { > > .name = "current", > > @@ -6413,6 +6445,11 @@ static struct cftype memory_files[] = { > > .seq_show = memory_oom_group_show, > > .write = memory_oom_group_write, > > }, > > + { > > + .name = "reclaim", > > + .flags = CFTYPE_NOT_ON_ROOT | CFTYPE_NS_DELEGATABLE, > > + .write = memory_reclaim, > > Btw, why not on root? From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yosry Ahmed Subject: Re: [PATCH resend] memcg: introduce per-memcg reclaim interface Date: Fri, 1 Apr 2022 02:11:51 -0700 Message-ID: References: <20220331084151.2600229-1-yosryahmed@google.com> Mime-Version: 1.0 Return-path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=TTgjpbxi+atnp9axNVFkq7OGFubJ8g3qn22nFzknXc0=; b=BJ7yd3RqMpeyZP/nXaJIqtS0xR9zz6WFUei6+PbJx29crBeMBoajAHTDBFqT/NP/E9 oDskgt1nDRh5HaFRpNsKLfCBPspbWgH8cCGdFcCBdjHXQsgZe7KYIL2BHqwf89YC2KqE FU1D88az6OaXiSSpAJa0xhtQvr3n9BKcuFwA/YBZ/Jyf7/x9liXJJ0BGx8kvbP4eC8sV bnF/eQBu+fZeYpJPE1wiLaT1wHA93zccpcqbztUDXVgRcYURXvHnX+RbafbV1iWb4j5D gn0lTg7NhGi/NMUl3PMdZ11PMU3EVUfR9yFC1YuR3rOGIjL/88llWWZvDc4Oi9MsFlpW BsNQ== In-Reply-To: List-ID: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Roman Gushchin Cc: Johannes Weiner , Michal Hocko , Shakeel Butt , Andrew Morton , David Rientjes , Tejun Heo , Zefan Li , cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-doc-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Linux-MM , Jonathan Corbet , Yu Zhao , Dave Hansen , Wei Xu , Greg Thelen On Thu, Mar 31, 2022 at 10:25 AM Roman Gushchin wrote: > > On Thu, Mar 31, 2022 at 08:41:51AM +0000, Yosry Ahmed wrote: > > From: Shakeel Butt > > > > Introduce an memcg interface to trigger memory reclaim on a memory cgroup. > > > > Use case: Proactive Reclaim > > --------------------------- > > > > A userspace proactive reclaimer can continuously probe the memcg to > > reclaim a small amount of memory. This gives more accurate and > > up-to-date workingset estimation as the LRUs are continuously > > sorted and can potentially provide more deterministic memory > > overcommit behavior. The memory overcommit controller can provide > > more proactive response to the changing behavior of the running > > applications instead of being reactive. > > > > A userspace reclaimer's purpose in this case is not a complete replacement > > for kswapd or direct reclaim, it is to proactively identify memory savings > > opportunities and reclaim some amount of cold pages set by the policy > > to free up the memory for more demanding jobs or scheduling new jobs. > > > > A user space proactive reclaimer is used in Google data centers. > > Additionally, Meta's TMO paper recently referenced a very similar > > interface used for user space proactive reclaim: > > https://dl.acm.org/doi/pdf/10.1145/3503222.3507731 > > > > Benefits of a user space reclaimer: > > ----------------------------------- > > > > 1) More flexible on who should be charged for the cpu of the memory > > reclaim. For proactive reclaim, it makes more sense to be centralized. > > > > 2) More flexible on dedicating the resources (like cpu). The memory > > overcommit controller can balance the cost between the cpu usage and > > the memory reclaimed. > > > > 3) Provides a way to the applications to keep their LRUs sorted, so, > > under memory pressure better reclaim candidates are selected. This also > > gives more accurate and uptodate notion of working set for an > > application. > > > > Why memory.high is not enough? > > ------------------------------ > > > > - memory.high can be used to trigger reclaim in a memcg and can > > potentially be used for proactive reclaim. > > However there is a big downside in using memory.high. It can potentially > > introduce high reclaim stalls in the target application as the > > allocations from the processes or the threads of the application can hit > > the temporary memory.high limit. > > > > - Userspace proactive reclaimers usually use feedback loops to decide > > how much memory to proactively reclaim from a workload. The metrics > > used for this are usually either refaults or PSI, and these metrics > > will become messy if the application gets throttled by hitting the > > high limit. > > > > - memory.high is a stateful interface, if the userspace proactive > > reclaimer crashes for any reason while triggering reclaim it can leave > > the application in a bad state. > > > > - If a workload is rapidly expanding, setting memory.high to proactively > > reclaim memory can result in actually reclaiming more memory than > > intended. > > > > The benefits of such interface and shortcomings of existing interface > > were further discussed in this RFC thread: > > https://lore.kernel.org/linux-mm/5df21376-7dd1-bf81-8414-32a73cea45dd-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org/ > > Hello! > > I'm totally up for the proposed feature! It makes total sense and is proved > to be useful, let's add it. > > > > > Interface: > > ---------- > > > > Introducing a very simple memcg interface 'echo 10M > memory.reclaim' to > > trigger reclaim in the target memory cgroup. > > > > > > Possible Extensions: > > -------------------- > > > > - This interface can be extended with an additional parameter or flags > > to allow specifying one or more types of memory to reclaim from (e.g. > > file, anon, ..). > > > > - The interface can also be extended with a node mask to reclaim from > > specific nodes. This has use cases for reclaim-based demotion in memory > > tiering systens. > > > > - A similar per-node interface can also be added to support proactive > > reclaim and reclaim-based demotion in systems without memcg. > > Maybe an option to specify a timeout? That might simplify the userspace part. > Also, please please add a test to selftests/cgroup/memcg tests. > It will also provide an example on how the userspace can use the feature. > Hi Roman, thanks for taking the time to review this! A timeout can be a good extension, I will add it to the commit message in the next version in possible extensions. I will add a test in v2, thanks! > > > > > For now, let's keep things simple by adding the basic functionality. > > What I'm worried about is how we gonna extend it? How do you see the interface > with 2-3 extensions from the list above? All these extensions look very > reasonable to me, so we'll likely have to implement them soon. So let's think > about the extensibility now. > My idea is to have these extensions as optional positional arguments (like Wei suggested), so that the interface does not get too complicated for users who don't care about tuning these options. If this is the case then I think there is nothing to worry about. Otherwise, if you think some of these options make sense to be a required argument instead, we can rethink the initial interface. > I wonder if it makes more sense to introduce a sys_reclaim() syscall instead? > In the end, such a feature might make sense on the system level too. > Yes, there is the drop_caches sysctl, but it's too radical for many cases. > I think in the RFC discussion there was consensus to add both a per-memcg knob, as well as per-node / per-system knobs (through sysfs or syscalls) later. Wei also points out that it's not common for a syscall to have a cgroup argument. > > > > [yosryahmed-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org: refreshed to current master, updated commit > > message based on recent discussions and use cases] > > Signed-off-by: Shakeel Butt > > Signed-off-by: Yosry Ahmed > > --- > > Documentation/admin-guide/cgroup-v2.rst | 9 ++++++ > > mm/memcontrol.c | 37 +++++++++++++++++++++++++ > > 2 files changed, 46 insertions(+) > > > > diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst > > index 69d7a6983f78..925aaabb2247 100644 > > --- a/Documentation/admin-guide/cgroup-v2.rst > > +++ b/Documentation/admin-guide/cgroup-v2.rst > > @@ -1208,6 +1208,15 @@ PAGE_SIZE multiple when read back. > > high limit is used and monitored properly, this limit's > > utility is limited to providing the final safety net. > > > > + memory.reclaim > > + A write-only file which exists on non-root cgroups. > > + > > + This is a simple interface to trigger memory reclaim in the > > + target cgroup. Write the number of bytes to reclaim to this > > + file and the kernel will try to reclaim that much memory. > > + Please note that the kernel can over or under reclaim from > > + the target cgroup. > > + > > memory.oom.group > > A read-write single value file which exists on non-root > > cgroups. The default value is "0". > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c > > index 725f76723220..994849fab7df 100644 > > --- a/mm/memcontrol.c > > +++ b/mm/memcontrol.c > > @@ -6355,6 +6355,38 @@ static ssize_t memory_oom_group_write(struct kernfs_open_file *of, > > return nbytes; > > } > > > > +static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf, > > + size_t nbytes, loff_t off) > > +{ > > + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); > > + unsigned int nr_retries = MAX_RECLAIM_RETRIES; > > + unsigned long nr_to_reclaim, nr_reclaimed = 0; > > + int err; > > + > > + buf = strstrip(buf); > > + err = page_counter_memparse(buf, "", &nr_to_reclaim); > > + if (err) > > + return err; > > + > > + while (nr_reclaimed < nr_to_reclaim) { > > + unsigned long reclaimed; > > + > > + if (signal_pending(current)) > > + break; > > + > > + reclaimed = try_to_free_mem_cgroup_pages(memcg, > > + nr_to_reclaim - nr_reclaimed, > > + GFP_KERNEL, true); > > + > > + if (!reclaimed && !nr_retries--) > > + break; > > + > > + nr_reclaimed += reclaimed; > > + } > > + > > + return nbytes; > > +} > > + > > static struct cftype memory_files[] = { > > { > > .name = "current", > > @@ -6413,6 +6445,11 @@ static struct cftype memory_files[] = { > > .seq_show = memory_oom_group_show, > > .write = memory_oom_group_write, > > }, > > + { > > + .name = "reclaim", > > + .flags = CFTYPE_NOT_ON_ROOT | CFTYPE_NS_DELEGATABLE, > > + .write = memory_reclaim, > > Btw, why not on root?