From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.4 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4652AC2D0EF for ; Fri, 17 Apr 2020 16:11:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ECC8F2078E for ; Fri, 17 Apr 2020 16:11:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ThYKyseG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ECC8F2078E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 70B7F8E0032; Fri, 17 Apr 2020 12:11:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6BBE18E0023; Fri, 17 Apr 2020 12:11:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5AAD68E0032; Fri, 17 Apr 2020 12:11:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0231.hostedemail.com [216.40.44.231]) by kanga.kvack.org (Postfix) with ESMTP id 3EF3C8E0023 for ; Fri, 17 Apr 2020 12:11:48 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 001E540C0 for ; Fri, 17 Apr 2020 16:11:47 +0000 (UTC) X-FDA: 76717837854.18.touch95_8414672e0b83d X-HE-Tag: touch95_8414672e0b83d X-Filterd-Recvd-Size: 4694 Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Fri, 17 Apr 2020 16:11:47 +0000 (UTC) Received: by mail-lj1-f195.google.com with SMTP id k21so2626850ljh.2 for ; Fri, 17 Apr 2020 09:11:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=c1APeQIZSwYvgzGtI49u613t4lCqukuDkbPUvWkvPzU=; b=ThYKyseGRAxgJFZMUU4L4u54kmHxMZkY8P61z911368jzyeY/e9JAfj8tF2wCkj6bA V5ZduIWFzynFO88OwEbWxf1JjarT+nqTzButGm04VyflH+r9Ibs22c7zQFOCl29NhUP+ 6mL1TkTeUxBY9Ifi3JreaImUiuqjAcM2j0HdBQxnr+ggLD+6Y/gO+PXQsc25/Bu5zTir mtlHksZ3pCs3gTsN1zuxUHgTZ+LTvYuY5D0UkbrML+Oqx56cP8qPYsHh+w1C5u5R7jFV DqPtL7fhjmjx5VfGf9JIqqDMx5KZwBJUp60FWDviwDPbWVNroD/VGG/oNusPZ+fsQ0vD vEJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=c1APeQIZSwYvgzGtI49u613t4lCqukuDkbPUvWkvPzU=; b=CH2+w+8I9bEfFYMTIIXqMNFAQ8ShjH23M2xrBp9RGNzMv4kXQpzY9FLgX9YqR8Pfb8 kuIuke9Z1o9oECf1YU407ps6Ryk9dTdbvKIxY7VxqNykwNSIUrzYsyPuZAoCbIAGcMDp dL3KXegMP7pzMOSBQ5VoORzrBOAUvOxiL1ps4g301E2gJcbXzmH5ZnHNu3bcMTUfdF0g WR2LyWQFt+Qif1EAxg1rHumkcceQ9hXU+6JL7c3zUNTmIX6F8rKepqDioaiVIUwATOrF iv/Bqod19NxoeNTGrup2H7qwWEDfSXeZsF65Ozs189hjH0TXuFNAMsuJyC9cqDsDZsst 3nKg== X-Gm-Message-State: AGi0PubansQbUAzQ6m30bwPjE+MzEWKS24LR+l+qXvVsHagQqVC/GEpf IL0747FidBlx3PzzomMtyDS0bY5wctjViPpepn2nvw== X-Google-Smtp-Source: APiQypIMx30LE6PxP+XRRTzQjNJKcLRuAZj4sZfr/V0Hfw0WOjOF/DEC8pHQo1LUqncf2p/NKk3A4RN1eV/CC2a6XOo= X-Received: by 2002:a2e:b6cf:: with SMTP id m15mr2549490ljo.168.1587139905662; Fri, 17 Apr 2020 09:11:45 -0700 (PDT) MIME-Version: 1.0 References: <20200417010617.927266-1-kuba@kernel.org> In-Reply-To: <20200417010617.927266-1-kuba@kernel.org> From: Shakeel Butt Date: Fri, 17 Apr 2020 09:11:33 -0700 Message-ID: Subject: Re: [PATCH 0/3] memcg: Slow down swap allocation as the available space gets depleted To: Jakub Kicinski Cc: Andrew Morton , Linux MM , Kernel Team , Tejun Heo , Johannes Weiner , Chris Down , Cgroups Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Apr 16, 2020 at 6:06 PM Jakub Kicinski wrote: > > Tejun describes the problem as follows: > > When swap runs out, there's an abrupt change in system behavior - > the anonymous memory suddenly becomes unmanageable which readily > breaks any sort of memory isolation and can bring down the whole > system. Can you please add more info on this abrupt change in system behavior and what do you mean by anon memory becoming unmanageable? Once the system is in global reclaim and doing swapping the memory isolation is already broken. Here I am assuming you are talking about memcg limit reclaim and memcg limits are overcommitted. Shouldn't running out of swap will trigger the OOM earlier which should be better than impacting the whole system. > To avoid that, oomd [1] monitors free swap space and triggers > kills when it drops below the specific threshold (e.g. 15%). > > While this works, it's far from ideal: > - Depending on IO performance and total swap size, a given > headroom might not be enough or too much. > - oomd has to monitor swap depletion in addition to the usual > pressure metrics and it currently doesn't consider memory.swap.max. > > Solve this by adapting the same approach that memory.high uses - > slow down allocation as the resource gets depleted turning the > depletion behavior from abrupt cliff one to gradual degradation > observable through memory pressure metric. > > [1] https://github.com/facebookincubator/oomd > > Jakub Kicinski (3): > mm: prepare for swap over-high accounting and penalty calculation > mm: move penalty delay clamping out of calculate_high_delay() > mm: automatically penalize tasks with high swap use > > include/linux/memcontrol.h | 4 + > mm/memcontrol.c | 166 ++++++++++++++++++++++++++++--------- > 2 files changed, 131 insertions(+), 39 deletions(-) > > -- > 2.25.2 > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Shakeel Butt Subject: Re: [PATCH 0/3] memcg: Slow down swap allocation as the available space gets depleted Date: Fri, 17 Apr 2020 09:11:33 -0700 Message-ID: References: <20200417010617.927266-1-kuba@kernel.org> Mime-Version: 1.0 Return-path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=c1APeQIZSwYvgzGtI49u613t4lCqukuDkbPUvWkvPzU=; b=ThYKyseGRAxgJFZMUU4L4u54kmHxMZkY8P61z911368jzyeY/e9JAfj8tF2wCkj6bA V5ZduIWFzynFO88OwEbWxf1JjarT+nqTzButGm04VyflH+r9Ibs22c7zQFOCl29NhUP+ 6mL1TkTeUxBY9Ifi3JreaImUiuqjAcM2j0HdBQxnr+ggLD+6Y/gO+PXQsc25/Bu5zTir mtlHksZ3pCs3gTsN1zuxUHgTZ+LTvYuY5D0UkbrML+Oqx56cP8qPYsHh+w1C5u5R7jFV DqPtL7fhjmjx5VfGf9JIqqDMx5KZwBJUp60FWDviwDPbWVNroD/VGG/oNusPZ+fsQ0vD vEJg== In-Reply-To: <20200417010617.927266-1-kuba-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> Sender: cgroups-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-ID: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Jakub Kicinski Cc: Andrew Morton , Linux MM , Kernel Team , Tejun Heo , Johannes Weiner , Chris Down , Cgroups On Thu, Apr 16, 2020 at 6:06 PM Jakub Kicinski wrote: > > Tejun describes the problem as follows: > > When swap runs out, there's an abrupt change in system behavior - > the anonymous memory suddenly becomes unmanageable which readily > breaks any sort of memory isolation and can bring down the whole > system. Can you please add more info on this abrupt change in system behavior and what do you mean by anon memory becoming unmanageable? Once the system is in global reclaim and doing swapping the memory isolation is already broken. Here I am assuming you are talking about memcg limit reclaim and memcg limits are overcommitted. Shouldn't running out of swap will trigger the OOM earlier which should be better than impacting the whole system. > To avoid that, oomd [1] monitors free swap space and triggers > kills when it drops below the specific threshold (e.g. 15%). > > While this works, it's far from ideal: > - Depending on IO performance and total swap size, a given > headroom might not be enough or too much. > - oomd has to monitor swap depletion in addition to the usual > pressure metrics and it currently doesn't consider memory.swap.max. > > Solve this by adapting the same approach that memory.high uses - > slow down allocation as the resource gets depleted turning the > depletion behavior from abrupt cliff one to gradual degradation > observable through memory pressure metric. > > [1] https://github.com/facebookincubator/oomd > > Jakub Kicinski (3): > mm: prepare for swap over-high accounting and penalty calculation > mm: move penalty delay clamping out of calculate_high_delay() > mm: automatically penalize tasks with high swap use > > include/linux/memcontrol.h | 4 + > mm/memcontrol.c | 166 ++++++++++++++++++++++++++++--------- > 2 files changed, 131 insertions(+), 39 deletions(-) > > -- > 2.25.2 >