From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2296CC433E5 for ; Fri, 10 Jul 2020 14:12:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A39B720842 for ; Fri, 10 Jul 2020 14:12:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vuH+P3Lx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A39B720842 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4D2AD8D000F; Fri, 10 Jul 2020 10:12:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 483328D0001; Fri, 10 Jul 2020 10:12:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 371F48D000F; Fri, 10 Jul 2020 10:12:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0107.hostedemail.com [216.40.44.107]) by kanga.kvack.org (Postfix) with ESMTP id 216698D0001 for ; Fri, 10 Jul 2020 10:12:37 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id C962818019255 for ; Fri, 10 Jul 2020 14:12:36 +0000 (UTC) X-FDA: 77022356712.21.glue74_3413bdf26ecf Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id 9D7E3180442CD for ; Fri, 10 Jul 2020 14:12:36 +0000 (UTC) X-HE-Tag: glue74_3413bdf26ecf X-Filterd-Recvd-Size: 6216 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by imf46.hostedemail.com (Postfix) with ESMTP for ; Fri, 10 Jul 2020 14:12:35 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id h19so6574609ljg.13 for ; Fri, 10 Jul 2020 07:12:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=PKCWrTfr3M3VE4zZraZncAFHMeh1ED72eMf914LZfYE=; b=vuH+P3LxQOhQzesxYlIlfiMh1qXv9/xVVXJv14axnvi9nFSaX0pINMUMb3BuL/K29E 8xjQ+MPMBLBSys5Npfb+b9A9SWsdvjMFZ7AbFQD1KRDWUkNX308eSDqg0AAz9UaLwj6J qpQMQZ9PxlFmIuGPQ4NI9buSv9aP9P3EfCInUHeaVaVTIT3Tfr6qDd57DUnSvOHMkzJQ JGMdTceGMmn1A5xPQfaZQjJGeajN0FPeNvmAmWY0OTp8XsOxyg/waavYXcu30KyNqXdC p8TkcQvqHrlLWeHu8EJeBhkqH72QPFopaz4sIjtAXuicSSLd4hYpCIa9O7Ct/DKWR8Wt 9/Tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=PKCWrTfr3M3VE4zZraZncAFHMeh1ED72eMf914LZfYE=; b=AVcGmxUT4LuE6nMYyEtjot6FfwtLOrpQAqIm3wpStZ6jHr/KhZt8mgmkdqGJqJCp/C zFcIpcVfHLwpU431l72Wz3jHz3Nro70avpaMp3JUnDzFbE1F+MF3f+64ArfwrePt70rt XNHaSBTlqLJ3HqAx+7qyN0drbZHSg1e2RrVSIzD2reKSD+0ZWdS7E81gZy/PY7xmCips Aqd3IsWkyO4ngLZa/HkK/C11qgTHmqE/GOm8xZqnBpNVa5ScEKBht/a0lNEx2bh03hFL KBZARfsZ4cHiS+hV7rnJXIiV33zBAiniiKzqtQEIGjbFuLamZyZNRtACtlDnt/hDEmHG bVUw== X-Gm-Message-State: AOAM5307dcUDkb7IJgUgkXSWTYx0lziJYUNytCtIDsTKKLuWKIB6heeD NdBnThc28e2gAj0xUUKvquky51y0Hqq6Fi1HkMQewg== X-Google-Smtp-Source: ABdhPJzlre8tOErvRW+mpIkqnqyceO+sOQTbl0pF6vq8FjFLcm4k96A3c2vvmcqnvoMOopZqXZ7b09p1tN16Pya7kWQ= X-Received: by 2002:a05:651c:10f:: with SMTP id a15mr38227429ljb.192.1594390354195; Fri, 10 Jul 2020 07:12:34 -0700 (PDT) MIME-Version: 1.0 References: <20200709194718.189231-1-guro@fb.com> <20200710122917.GB3022@dhcp22.suse.cz> In-Reply-To: <20200710122917.GB3022@dhcp22.suse.cz> From: Shakeel Butt Date: Fri, 10 Jul 2020 07:12:22 -0700 Message-ID: Subject: Re: [PATCH] mm: memcontrol: avoid workload stalls when lowering memory.high To: Michal Hocko Cc: Roman Gushchin , Andrew Morton , Johannes Weiner , Linux MM , Kernel Team , LKML , Domas Mituzas , Tejun Heo , Chris Down Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 9D7E3180442CD X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jul 10, 2020 at 5:29 AM Michal Hocko wrote: > > On Thu 09-07-20 12:47:18, Roman Gushchin wrote: > > Memory.high limit is implemented in a way such that the kernel > > penalizes all threads which are allocating a memory over the limit. > > Forcing all threads into the synchronous reclaim and adding some > > artificial delays allows to slow down the memory consumption and > > potentially give some time for userspace oom handlers/resource control > > agents to react. > > > > It works nicely if the memory usage is hitting the limit from below, > > however it works sub-optimal if a user adjusts memory.high to a value > > way below the current memory usage. It basically forces all workload > > threads (doing any memory allocations) into the synchronous reclaim > > and sleep. This makes the workload completely unresponsive for > > a long period of time and can also lead to a system-wide contention on > > lru locks. It can happen even if the workload is not actually tight on > > memory and has, for example, a ton of cold pagecache. > > > > In the current implementation writing to memory.high causes an atomic > > update of page counter's high value followed by an attempt to reclaim > > enough memory to fit into the new limit. To fix the problem described > > above, all we need is to change the order of execution: try to push > > the memory usage under the limit first, and only then set the new > > high limit. > > Shakeel would this help with your pro-active reclaim usecase? It would > require to reset the high limit right after the reclaim returns which is > quite ugly but it would at least not require a completely new interface. > You would simply do > high = current - to_reclaim > echo $high > memory.high > echo infinity > memory.high # To prevent direct reclaim > # allocation stalls > This will reduce the chance of stalls but the interface is still non-delegatable i.e. applications can not change their own memory.high for the use-cases like application controlled proactive reclaim and uswapd. One more ugly fix would be to add one more layer of cgroup and the application use memory.high of that layer to fulfill such use-cases. I think providing a new interface would allow us to have a much cleaner solution than to settle on a bunch of ugly hacks. > The primary reason to set the high limit in advance was to catch > potential runaways more effectively because they would just get > throttled while memory_high_write is reclaiming. With this change > the reclaim here might be just playing never ending catch up. On the > plus side a break out from the reclaim loop would just enforce the limit > so if the operation takes too long then the reclaim burden will move > over to consumers eventually. So I do not see any real danger. > > > Signed-off-by: Roman Gushchin > > Reported-by: Domas Mituzas > > Cc: Johannes Weiner > > Cc: Michal Hocko > > Cc: Tejun Heo > > Cc: Shakeel Butt > > Cc: Chris Down > > Acked-by: Michal Hocko > This patch seems reasonable on its own. Reviewed-by: Shakeel Butt