From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.6 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D0BDC433E0 for ; Tue, 2 Jun 2020 02:34:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8E82D20671 for ; Tue, 2 Jun 2020 02:34:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cFKWs76d" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8E82D20671 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F099A80007; Mon, 1 Jun 2020 22:34:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EBADC8E0006; Mon, 1 Jun 2020 22:34:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD0A080007; Mon, 1 Jun 2020 22:34:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0163.hostedemail.com [216.40.44.163]) by kanga.kvack.org (Postfix) with ESMTP id C535C8E0006 for ; Mon, 1 Jun 2020 22:34:23 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 7CD38180AD802 for ; Tue, 2 Jun 2020 02:34:23 +0000 (UTC) X-FDA: 76882702806.13.trail51_2648b7cd7f833 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id 60D9718140B67 for ; Tue, 2 Jun 2020 02:34:23 +0000 (UTC) X-HE-Tag: trail51_2648b7cd7f833 X-Filterd-Recvd-Size: 11247 Received: from mail-qk1-f196.google.com (mail-qk1-f196.google.com [209.85.222.196]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Tue, 2 Jun 2020 02:34:22 +0000 (UTC) Received: by mail-qk1-f196.google.com with SMTP id c14so10160403qka.11 for ; Mon, 01 Jun 2020 19:34:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=WIV812zXdt1sSNO21HBL8cR2iY4avQ4bfEsobtUNfCM=; b=cFKWs76db7QYS3TDdaWNc4V2cCDKOVhKW6Mya5Z6lI0jqEdEbjKD7MCzoE6a6qp8cT nGSlPo+VgqvwNLi5iJ/+oAmtnBl1D8aMVa1vZsXhB+OwPVk6B60yA3cP4QCgFVzUcdxC p9zeA6Hm2sPP/EmLCupJj0dPHow+inE7kLTsnO5CqWSSKxejDWH5mdlSAb1Fda8oB90H Zb44vhn9fSWPdzVwsNEic+sU13jS/9XZvNPJ7zq8X5bsDJlX9HTBM8PRH8k0GuO2tG0i TPB5rmgr6CgE07F/4Vs4/EGjTD0J/WvsgWRZwlqGCfOv6AmyJGGa68hWiUaaZIPaAGVq jPUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=WIV812zXdt1sSNO21HBL8cR2iY4avQ4bfEsobtUNfCM=; b=TAjp/OQAZftn/oJbcqmkom075hcPT1tbdC0mxkJMMEoldgnhjPAUnRJjAKQvXyEhcf qyr8j60HAhtbXieKE0T3kl3cv/Y8Yv0WQgbvJte/9R9Bngv1U9BK41uZGfya0J/Liilc KQfQ3dy+RQYTS8AOkr3tNcB4MT6FpZyN+m0r1AqCmsclFk0Ft/BPVKDQaGLkhhzmaKbD vbnVZUBcjRYY+bIYbjTX8HOD7QyBD2tvwklypSQ/T6ucawz8M8QiovV6AyF6htCW+MIB 731QOsjEtrLGCxOHNS37mXXuZvxNgYTuU0iZuO4mTIOmAD8yEuzdN5sn+acVvKkc0XBL BKBQ== X-Gm-Message-State: AOAM5303oyFvHg6cD83aTp5Plrwccb7WcNx+ryHAVuczckX9SxUZUJoK iFCEP/WZTEY5s8cs2cdc8nJZJNv0pBtCXRPtqqk= X-Google-Smtp-Source: ABdhPJw9+5hNfnvvSMNcTloFXeduRgAnxY7dVQ+MrEu5zMcyrl36HQroBoKPx8+xID0m7vPttHPDe7pPHxF6rMuRI/Q= X-Received: by 2002:a05:620a:164c:: with SMTP id c12mr20956489qko.343.1591065262176; Mon, 01 Jun 2020 19:34:22 -0700 (PDT) MIME-Version: 1.0 References: <20200520232525.798933-1-hannes@cmpxchg.org> <20200520232525.798933-6-hannes@cmpxchg.org> <20200527134333.GF6781@cmpxchg.org> <20200528170155.GA69521@cmpxchg.org> <20200529151228.GA92892@cmpxchg.org> <20200601155615.GA131075@cmpxchg.org> In-Reply-To: <20200601155615.GA131075@cmpxchg.org> From: Joonsoo Kim Date: Tue, 2 Jun 2020 11:34:17 +0900 Message-ID: Subject: Re: [PATCH 05/14] mm: workingset: let cache workingset challenge anon To: Johannes Weiner Cc: Linux Memory Management List , Rik van Riel , Minchan Kim , Michal Hocko , Andrew Morton , Joonsoo Kim , LKML , kernel-team@fb.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 60D9718140B67 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000003, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: 2020=EB=85=84 6=EC=9B=94 2=EC=9D=BC (=ED=99=94) =EC=98=A4=EC=A0=84 12:56, J= ohannes Weiner =EB=8B=98=EC=9D=B4 =EC=9E=91=EC=84=B1: > > On Mon, Jun 01, 2020 at 03:14:24PM +0900, Joonsoo Kim wrote: > > 2020=EB=85=84 5=EC=9B=94 30=EC=9D=BC (=ED=86=A0) =EC=98=A4=EC=A0=84 12:= 12, Johannes Weiner =EB=8B=98=EC=9D=B4 =EC=9E=91=EC=84= =B1: > > > > > > On Fri, May 29, 2020 at 03:48:00PM +0900, Joonsoo Kim wrote: > > > > 2020=EB=85=84 5=EC=9B=94 29=EC=9D=BC (=EA=B8=88) =EC=98=A4=EC=A0=84= 2:02, Johannes Weiner =EB=8B=98=EC=9D=B4 =EC=9E=91=EC= =84=B1: > > > > > On Thu, May 28, 2020 at 04:16:50PM +0900, Joonsoo Kim wrote: > > > > > > 2020=EB=85=84 5=EC=9B=94 27=EC=9D=BC (=EC=88=98) =EC=98=A4=ED= =9B=84 10:43, Johannes Weiner =EB=8B=98=EC=9D=B4 =EC=9E= =91=EC=84=B1: > > > > > > > On Wed, May 27, 2020 at 11:06:47AM +0900, Joonsoo Kim wrote: > > > > > The only way they could get reclaimed is if their access distance= ends > > > > > up bigger than the file cache. But if that's the case, then the > > > > > workingset is overcommitted, and none of the pages qualify for re= claim > > > > > protection. Picking a subset to protect against the rest is arbit= rary. > > > > > > > > In the fixed example, although other file (500 MB) is repeatedly ac= cessed, > > > > it's not workingset. If we have unified list (file + anon), access = distance of > > > > Pn will be larger than whole memory size. Therefore, it's not overc= ommitted > > > > workingset and this patch wrongly try to activate it. As I said bef= ore, > > > > without considering inactive_age for anon list, this calculation ca= n not be > > > > correct. > > > > > > You're right. If we don't take anon age into account, the activations > > > could be over-eager; however, so would counting IO cost and exerting > > > pressure on anon be, which means my previous patch to split these two > > > wouldn't fix fundamental the problem you're pointing out. We simply > > > > Splitting would not fix the fundamental problem (over-eager) but it wou= ld > > greatly weaken the problem. Just counting IO cost doesn't break the > > active/inactive separation in file list. It does cause more scan on ano= n list > > but I think that it's endurable. > > I think the split is a good idea. > > The only thing I'm not sure yet is if we can get away without an > additional page flag if the active flag cannot be reused to denote > thrashing. I'll keep at it, maybe I can figure something out. > > But I think it would be follow-up work. > > > > have to take anon age into account for the refaults to be comparable. > > > > Yes, taking anon age into account is also a good candidate to fix the p= roblem. > > Okay, good. > > > > However, your example cannot have a completely silent stable state. A= s > > > we stop workingset aging, the refault distances will slowly increase > > > again. We will always have a bit of churn, and rightfully so, because > > > the workingset *could* go stale. > > > > > > That's the same situation in my cache-only example above. Anytime you > > > have a subset of pages that by itself could fit into memory, but can'= t > > > because of an established workingset, ongoing sampling is necessary. > > > > > > But the rate definitely needs to reduce as we detect that in-memory > > > pages are indeed hot. Otherwise we cause more churn than is required > > > for an appropriate rate of workingset sampling. > > > > > > How about the patch below? It looks correct, but I will have to re-ru= n > > > my tests to make sure I / we are not missing anything. > > > > Much better! It may solve my concern mostly. > > Okay thanks for confirming. I'll send a proper version to Andrew. Okay. > > But, I still think that modified refault activation equation isn't > > safe. The next > > problem I found is related to the scan ratio limit patch ("limit the ra= nge of > > LRU type balancing") on this series. See the below example. > > > > anon: Hot (X M) > > file: Hot (200 M) / dummy (200 M) > > P: 1200 M (3 parts, each one 400 M, P1, P2, P3) > > Access Pattern: A -> F(H) -> P1 -> A -> F(H) -> P2 -> ... -> > > > > Without this patch, A and F(H) are kept on the memory and look like > > it's correct. > > > > With this patch and below fix, refault equation for Pn would be: > > > > Refault dist of Pn =3D 1200 (from file non-resident) + 1200 * anon scan > > ratio (from anon non-resident) > > anon + active file =3D X + 200 > > 1200 + 1200 * anon scan ratio (0.5 ~ 2) < X + 200 > > That doesn't look quite right to me. The anon part of the refault > distance is driven by X, so the left-hand of this formula contains X > as well. > > 1000 file (1200M reuse distance, 200M in-core size) + F(H) reactivations = + X * scan ratio < X + 1000 As I said before, there is no X on left-hand of this formula. To access all Pn and re-access P1, we need 1200M file list scan and reclaim. More scan isn't nee= ded. With your patch "limit the range of LRU type balancing", scan ratio between file/anon list is limited to 0.5 ~ 2.0, so, maximum anon scan would be 1200 M * 2.0, that is, 2400 M and not bounded by X. That means that file list cannot be stable with some X. > Activations persist as long as anon isn't fully scanned and it isn't > established yet that it's fully hot. Meaning, we optimistically assume > the refaulting pages can be workingset until we're proven wrong. > > > According to the size of X, Pn's refault result would be different. Pn = could > > be activated with large enough X and then F(H) could be evicted. In ide= al > > case (unified list), for this example, Pn should not be activated in an= y X. > > Yes. The active/iocost split would allow us to be smarter about it. > > > This is a fundamental problem since we have two list type (file/anon) a= nd > > scan ratio limit is required. Anyway, we need to take care of this real= ity and > > the way most safe is to count IO cost instead of doing activation in th= is > > 'non-resident dist < (active + anon list)' case. > > Agreed here again. > > > Again, for this patch, I'm not confident myself so please let me know i= f I'm > > wrong. > > As far as this patch goes, I think it's important to look at the > bigger picture. > > We need to have convergence first before being able to worry about > optimizing. Stable states are optimizations, but false stable states > are correctness problems. > > For the longest time, we scanned active pages unconditionally during > page reclaim. This was always safe in the sense that it wouldn't get > stuck on a stale workingset, but it incurs unnecessary workingset > churn when reclaim is driven by use-once patterns. > > We optimized the latter too aggressively, and as a result caused > situations where we indefinitely fail to cache the hottest > data. That's not really a workable trade-off. > > With the active/iocost split you're suggesting, we can reasonably > optimize your example scenario. But we can't do it if the flipside > means complete failure to transition between in-memory sets. > > So I think we should go ahead with this patch (with the anon age > recognition fixed, because that's a correctness issue), and follow it > up with the stable state optimization to shrink anon first. If my lastly found example is a correct example (your confirm is required), it is also related to the correctness issue since cold pages causes eviction of the hot pages repeatedly. In this case, they (without patch, with patch) all have some correctness issue so we need to judge which one is better in terms of overall impact. I don't have strong opinion about it so it's up to you to decide the way to= go. Thanks.