From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4337C433E0 for ; Tue, 14 Jul 2020 08:41:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6442021897 for ; Tue, 14 Jul 2020 08:41:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6442021897 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 123496B0003; Tue, 14 Jul 2020 04:41:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D3F96B0005; Tue, 14 Jul 2020 04:41:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F2AFE6B0006; Tue, 14 Jul 2020 04:41:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0082.hostedemail.com [216.40.44.82]) by kanga.kvack.org (Postfix) with ESMTP id DE8546B0003 for ; Tue, 14 Jul 2020 04:41:27 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 9F2DC2DFC for ; Tue, 14 Jul 2020 08:41:27 +0000 (UTC) X-FDA: 77036037414.30.need52_250b60226eef Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id 76666180B3C8E for ; Tue, 14 Jul 2020 08:41:27 +0000 (UTC) X-HE-Tag: need52_250b60226eef X-Filterd-Recvd-Size: 5912 Received: from mail-wr1-f65.google.com (mail-wr1-f65.google.com [209.85.221.65]) by imf17.hostedemail.com (Postfix) with ESMTP for ; Tue, 14 Jul 2020 08:41:26 +0000 (UTC) Received: by mail-wr1-f65.google.com with SMTP id k6so20288709wrn.3 for ; Tue, 14 Jul 2020 01:41:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=DZJI4J4GWPCiHcryb6MzhVJv5JF+Q8LkCIIU9I8lCYw=; b=ntYZfaYZJSaBl9cWOrLvMoeQmpgwJ7mWPaax93CPiEkgSdXH9i1jkhMBeI4cBLOXVu q2rtXovQzOoYeQ2REUNKmCI3jqy076e4bh5qL9/Cn3gZHnNA0/3nHv46pAJK7vARUd9N 66czHUdCCFPuBQAJ1fXnG9DlqhOqHAxCKm4OP4snHVAnAix+lFEX6AP7klkk5QGzOFdS vnGUJFCIG0HMKWIH26ybn/UliVoFnyGLEok/3gzBtki03CpIXEIxClfp64qjYkgTqTWF ISa+OO7QG+TpHUna3ARCSZRVEDcqddvPBupchC1cuwLyxxEiiR1oeNQWzqrMAK9NXbcT AViw== X-Gm-Message-State: AOAM530sTP2WO8ojII5EhsxFlbp/+GDoIEcD1tH6a9hqxqnuZQ/cipK7 /LNicX6uWh7mwTtzH3gJlp8= X-Google-Smtp-Source: ABdhPJxTNcf8EkhjrdL/FnjXZ4Igi2NSjQyCDEFbKe9Yox0PnbOgIh5YD0E1iH0E7yiJmmONgB/Bqw== X-Received: by 2002:adf:bc41:: with SMTP id a1mr3749906wrh.186.1594716086018; Tue, 14 Jul 2020 01:41:26 -0700 (PDT) Received: from localhost (ip-37-188-148-171.eurotel.cz. [37.188.148.171]) by smtp.gmail.com with ESMTPSA id j4sm28721035wrp.51.2020.07.14.01.41.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jul 2020 01:41:25 -0700 (PDT) Date: Tue, 14 Jul 2020 10:41:23 +0200 From: Michal Hocko To: Shakeel Butt Cc: Roman Gushchin , Andrew Morton , Johannes Weiner , Linux MM , Kernel Team , LKML , Domas Mituzas , Tejun Heo , Chris Down Subject: Re: [PATCH] mm: memcontrol: avoid workload stalls when lowering memory.high Message-ID: <20200714084123.GG24642@dhcp22.suse.cz> References: <20200709194718.189231-1-guro@fb.com> <20200710122917.GB3022@dhcp22.suse.cz> <20200710184205.GB350256@carbon.dhcp.thefacebook.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 76666180B3C8E X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri 10-07-20 12:19:37, Shakeel Butt wrote: > On Fri, Jul 10, 2020 at 11:42 AM Roman Gushchin wrote: > > > > On Fri, Jul 10, 2020 at 07:12:22AM -0700, Shakeel Butt wrote: > > > On Fri, Jul 10, 2020 at 5:29 AM Michal Hocko wrote: > > > > > > > > On Thu 09-07-20 12:47:18, Roman Gushchin wrote: > > > > > Memory.high limit is implemented in a way such that the kernel > > > > > penalizes all threads which are allocating a memory over the limit. > > > > > Forcing all threads into the synchronous reclaim and adding some > > > > > artificial delays allows to slow down the memory consumption and > > > > > potentially give some time for userspace oom handlers/resource control > > > > > agents to react. > > > > > > > > > > It works nicely if the memory usage is hitting the limit from below, > > > > > however it works sub-optimal if a user adjusts memory.high to a value > > > > > way below the current memory usage. It basically forces all workload > > > > > threads (doing any memory allocations) into the synchronous reclaim > > > > > and sleep. This makes the workload completely unresponsive for > > > > > a long period of time and can also lead to a system-wide contention on > > > > > lru locks. It can happen even if the workload is not actually tight on > > > > > memory and has, for example, a ton of cold pagecache. > > > > > > > > > > In the current implementation writing to memory.high causes an atomic > > > > > update of page counter's high value followed by an attempt to reclaim > > > > > enough memory to fit into the new limit. To fix the problem described > > > > > above, all we need is to change the order of execution: try to push > > > > > the memory usage under the limit first, and only then set the new > > > > > high limit. > > > > > > > > Shakeel would this help with your pro-active reclaim usecase? It would > > > > require to reset the high limit right after the reclaim returns which is > > > > quite ugly but it would at least not require a completely new interface. > > > > You would simply do > > > > high = current - to_reclaim > > > > echo $high > memory.high > > > > echo infinity > memory.high # To prevent direct reclaim > > > > # allocation stalls > > > > > > > > > > This will reduce the chance of stalls but the interface is still > > > non-delegatable i.e. applications can not change their own memory.high > > > for the use-cases like application controlled proactive reclaim and > > > uswapd. > > > > Can you, please, elaborate a bit more on this? I didn't understand > > why. > > > > Sure. Do we want memory.high a CFTYPE_NS_DELEGATABLE type file? I > don't think so otherwise any job on a system can change their > memory.high and can adversely impact the isolation and memory > scheduling of the system. Is this really the case? There should always be a parent cgroup that overrides the setting. Also you can always set the hard limit if you do not want to add another layer of cgroup in the hierarchy before delegation. Or am I missing something? -- Michal Hocko SUSE Labs