From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EB02C433E1 for ; Mon, 1 Jun 2020 20:44:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8F1DE206E2 for ; Mon, 1 Jun 2020 20:44:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=cmpxchg-org.20150623.gappssmtp.com header.i=@cmpxchg-org.20150623.gappssmtp.com header.b="c5d1DsFj" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8F1DE206E2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1AA8D80007; Mon, 1 Jun 2020 16:44:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 135758E0006; Mon, 1 Jun 2020 16:44:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F15F980007; Mon, 1 Jun 2020 16:44:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0030.hostedemail.com [216.40.44.30]) by kanga.kvack.org (Postfix) with ESMTP id D43CC8E0006 for ; Mon, 1 Jun 2020 16:44:57 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 98191EAA4B for ; Mon, 1 Jun 2020 20:44:57 +0000 (UTC) X-FDA: 76881822234.29.list42_2b31091028e43 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 75A6B18086E44 for ; Mon, 1 Jun 2020 20:44:57 +0000 (UTC) X-HE-Tag: list42_2b31091028e43 X-Filterd-Recvd-Size: 14280 Received: from mail-qt1-f195.google.com (mail-qt1-f195.google.com [209.85.160.195]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Mon, 1 Jun 2020 20:44:56 +0000 (UTC) Received: by mail-qt1-f195.google.com with SMTP id k22so8864073qtm.6 for ; Mon, 01 Jun 2020 13:44:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=WX6vrgdXsTGNrYeiwfYIi7rHNabcj7FuMmAuNNQlces=; b=c5d1DsFj0nNPRuIrAbjcGrjMZFSlE7mxt8AP1AOeqOE1SyQQ6yzOnmxsL+zkINmMgS QlauWET3DH37bCvNbv15tP8W1iLK0r2deoocUb+gryo3ILSa0KX0TFCcvwXMSUzxsJui xVIKFSwNcdWhjmR59VuLzQLmPfbLkH4kzQHS5b69OdGh2N/2K6n4P2BRqidhAztH3dzN DH0fq/9RhiDc4adE9DP8heGKwo0ozpYlCHqevzmCRRUvLGdi5AIeMU0coxVBOeRO7zQI rvgqGBBsdkNNLeOk0A9YGsMfbvxf4JTiqVJvYkQC3qtOg7lMX2DtpfJIgM/+/jFW5fjI GOjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=WX6vrgdXsTGNrYeiwfYIi7rHNabcj7FuMmAuNNQlces=; b=S5TsTopXM0Yx0CByXgJXhBcRdpM4stxkFUHsbSWjtpIY9nRVq9suy08h41IwidcGRA 8w44qKR9cVdR5SEZBH5S4Vt/AoxPsgj7izVFucTjZrwejNbPnD+E8AkBx768UqH/b/KM YRPgyXBZ8YyiPrvNUBjpNz7WoPBPMX16d/rwJwmCmGOasn8hl2Z/zdYbK0IeD+LW0lDG LhmkT1GaEHPyaLM8UqXgrnqoK4O4+1095yjka4S3cY/jbfP1cLgsXH4p3QKxshtEcYS8 sQBQhQYEMsImN1KF/9KdL1ViDLAR3+wHTYFSJFAxcgzB3rI7YYUJX4wwlcLsHfHnGIVz XmaA== X-Gm-Message-State: AOAM531W5ezDkJKZucZ/4dCMnql5mHLLw3VjIhxIwwV2ncujQ5KrQfPu QMv50EE7T488GGxbHONXfk4QRQ== X-Google-Smtp-Source: ABdhPJx1byJgYP1AEleCHDYStQWli8NMJXupxh2z98H9LSqNSlzxrKuTtyufKdmMkVDN6802Cobw5Q== X-Received: by 2002:ac8:311d:: with SMTP id g29mr24976546qtb.242.1591044295733; Mon, 01 Jun 2020 13:44:55 -0700 (PDT) Received: from localhost (70.44.39.90.res-cmts.bus.ptd.net. [70.44.39.90]) by smtp.gmail.com with ESMTPSA id k12sm303772qkk.58.2020.06.01.13.44.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Jun 2020 13:44:54 -0700 (PDT) Date: Mon, 1 Jun 2020 16:44:24 -0400 From: Johannes Weiner To: Joonsoo Kim Cc: Linux Memory Management List , Rik van Riel , Minchan Kim , Michal Hocko , Andrew Morton , Joonsoo Kim , LKML , kernel-team@fb.com Subject: Re: [PATCH 05/14] mm: workingset: let cache workingset challenge anon Message-ID: <20200601204424.GA212944@cmpxchg.org> References: <20200520232525.798933-1-hannes@cmpxchg.org> <20200520232525.798933-6-hannes@cmpxchg.org> <20200527134333.GF6781@cmpxchg.org> <20200528170155.GA69521@cmpxchg.org> <20200529151228.GA92892@cmpxchg.org> <20200601155615.GA131075@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20200601155615.GA131075@cmpxchg.org> X-Rspamd-Queue-Id: 75A6B18086E44 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jun 01, 2020 at 11:56:17AM -0400, Johannes Weiner wrote: > On Mon, Jun 01, 2020 at 03:14:24PM +0900, Joonsoo Kim wrote: > > 2020=EB=85=84 5=EC=9B=94 30=EC=9D=BC (=ED=86=A0) =EC=98=A4=EC=A0=84 1= 2:12, Johannes Weiner =EB=8B=98=EC=9D=B4 =EC=9E=91=EC= =84=B1: > > > However, your example cannot have a completely silent stable state.= As > > > we stop workingset aging, the refault distances will slowly increas= e > > > again. We will always have a bit of churn, and rightfully so, becau= se > > > the workingset *could* go stale. > > > > > > That's the same situation in my cache-only example above. Anytime y= ou > > > have a subset of pages that by itself could fit into memory, but ca= n't > > > because of an established workingset, ongoing sampling is necessary= . > > > > > > But the rate definitely needs to reduce as we detect that in-memory > > > pages are indeed hot. Otherwise we cause more churn than is require= d > > > for an appropriate rate of workingset sampling. > > > > > > How about the patch below? It looks correct, but I will have to re-= run > > > my tests to make sure I / we are not missing anything. > >=20 > > Much better! It may solve my concern mostly. >=20 > Okay thanks for confirming. I'll send a proper version to Andrew. I sent the below patch through my battery of tests to make sure it doesn't break anything fundamental, and they didn't. For the convergence test, the overload test, and the kernel build, the difference, if any, is in the noise, which is expected. Andrew, absent any objections, can you please fold the below into the original patch? It's a bit noisy due to the rename of inactive_age now that it's used for anon as well (frankly, it was a misnomer from the start), and switching the interface to take an lruvec instead of a memcg and a pgdat, which works better for the callers (I stole parent_lruvec() from "mm: vmscan: determine anon/file pressure balance at the reclaim root" later in the series, so that isn't new code). The main change is the two bits in __remove_mapping and move_pages_to_lru, so that shadow entries are aged when anon is reclaimed or rotated. --- >From a8faceabc1454dfd878caee2a8422493d937a394 Mon Sep 17 00:00:00 2001 From: Johannes Weiner Date: Mon, 1 Jun 2020 14:04:09 -0400 Subject: [PATCH] mm: workingset: let cache workingset challenge anon fix We compare refault distances to active_file + anon. But age of the non-resident information is only driven by the file LRU. As a result, we may overestimate the recency of any incoming refaults and activate them too eagerly, causing unnecessary LRU churn in certain situations. Make anon aging drive nonresident age as well to address that. Reported-by: Joonsoo Kim Signed-off-by: Johannes Weiner --- include/linux/memcontrol.h | 13 +++++++++++ include/linux/mmzone.h | 4 ++-- include/linux/swap.h | 1 + mm/vmscan.c | 3 +++ mm/workingset.c | 46 ++++++++++++++++++++++---------------- 5 files changed, 46 insertions(+), 21 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 32a0b4d47540..d982c80da157 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1303,6 +1303,19 @@ static inline void dec_lruvec_page_state(struct pa= ge *page, mod_lruvec_page_state(page, idx, -1); } =20 +static inline struct lruvec *parent_lruvec(struct lruvec *lruvec) +{ + struct mem_cgroup *memcg; + + memcg =3D lruvec_memcg(lruvec); + if (!memcg) + return NULL; + memcg =3D parent_mem_cgroup(memcg); + if (!memcg) + return NULL; + return mem_cgroup_lruvec(memcg, lruvec_pgdat(lruvec)); +} + #ifdef CONFIG_CGROUP_WRITEBACK =20 struct wb_domain *mem_cgroup_wb_domain(struct bdi_writeback *wb); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index c1fbda9ddd1f..7cccbb8bc2d7 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -262,8 +262,8 @@ enum lruvec_flags { struct lruvec { struct list_head lists[NR_LRU_LISTS]; struct zone_reclaim_stat reclaim_stat; - /* Evictions & activations on the inactive file list */ - atomic_long_t inactive_age; + /* Non-resident age, driven by LRU movement */ + atomic_long_t nonresident_age; /* Refaults at the time of last reclaim cycle */ unsigned long refaults; /* Various lruvec state flags (enum lruvec_flags) */ diff --git a/include/linux/swap.h b/include/linux/swap.h index 30fd4641890e..f0d2dca28c99 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -312,6 +312,7 @@ struct vma_swap_readahead { }; =20 /* linux/mm/workingset.c */ +void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_= pages); void *workingset_eviction(struct page *page, struct mem_cgroup *target_m= emcg); void workingset_refault(struct page *page, void *shadow); void workingset_activation(struct page *page); diff --git a/mm/vmscan.c b/mm/vmscan.c index 43f88b1a4f14..3033f1c951cd 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -898,6 +898,7 @@ static int __remove_mapping(struct address_space *map= ping, struct page *page, __delete_from_swap_cache(page, swap); xa_unlock_irqrestore(&mapping->i_pages, flags); put_swap_page(page, swap); + workingset_eviction(page, target_memcg); } else { void (*freepage)(struct page *); void *shadow =3D NULL; @@ -1876,6 +1877,8 @@ static unsigned noinline_for_stack move_pages_to_lr= u(struct lruvec *lruvec, list_add(&page->lru, &pages_to_free); } else { nr_moved +=3D nr_pages; + if (PageActive(page)) + workingset_age_nonresident(lruvec, nr_pages); } } =20 diff --git a/mm/workingset.c b/mm/workingset.c index e69865739539..98b91e623b85 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -156,8 +156,8 @@ * * Implementation * - * For each node's file LRU lists, a counter for inactive evictions - * and activations is maintained (node->inactive_age). + * For each node's LRU lists, a counter for inactive evictions and + * activations is maintained (node->nonresident_age). * * On eviction, a snapshot of this counter (along with some bits to * identify the node) is stored in the now empty page cache @@ -213,7 +213,17 @@ static void unpack_shadow(void *shadow, int *memcgid= p, pg_data_t **pgdat, *workingsetp =3D workingset; } =20 -static void advance_inactive_age(struct mem_cgroup *memcg, pg_data_t *pg= dat) +/** + * workingset_age_nonresident - age non-resident entries as LRU ages + * @memcg: the lruvec that was aged + * @nr_pages: the number of pages to count + * + * As in-memory pages are aged, non-resident pages need to be aged as + * well, in order for the refault distances later on to be comparable + * to the in-memory dimensions. This function allows reclaim and LRU + * operations to drive the non-resident aging along in parallel. + */ +void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_= pages) { /* * Reclaiming a cgroup means reclaiming all its children in a @@ -227,11 +237,8 @@ static void advance_inactive_age(struct mem_cgroup *= memcg, pg_data_t *pgdat) * the root cgroup's, age as well. */ do { - struct lruvec *lruvec; - - lruvec =3D mem_cgroup_lruvec(memcg, pgdat); - atomic_long_inc(&lruvec->inactive_age); - } while (memcg && (memcg =3D parent_mem_cgroup(memcg))); + atomic_long_add(nr_pages, &lruvec->nonresident_age); + } while ((lruvec =3D parent_lruvec(lruvec))); } =20 /** @@ -254,12 +261,11 @@ void *workingset_eviction(struct page *page, struct= mem_cgroup *target_memcg) VM_BUG_ON_PAGE(page_count(page), page); VM_BUG_ON_PAGE(!PageLocked(page), page); =20 - advance_inactive_age(page_memcg(page), pgdat); - lruvec =3D mem_cgroup_lruvec(target_memcg, pgdat); + workingset_age_nonresident(lruvec, hpage_nr_pages(page)); /* XXX: target_memcg can be NULL, go through lruvec */ memcgid =3D mem_cgroup_id(lruvec_memcg(lruvec)); - eviction =3D atomic_long_read(&lruvec->inactive_age); + eviction =3D atomic_long_read(&lruvec->nonresident_age); return pack_shadow(memcgid, pgdat, eviction, PageWorkingset(page)); } =20 @@ -309,20 +315,20 @@ void workingset_refault(struct page *page, void *sh= adow) if (!mem_cgroup_disabled() && !eviction_memcg) goto out; eviction_lruvec =3D mem_cgroup_lruvec(eviction_memcg, pgdat); - refault =3D atomic_long_read(&eviction_lruvec->inactive_age); + refault =3D atomic_long_read(&eviction_lruvec->nonresident_age); =20 /* * Calculate the refault distance * * The unsigned subtraction here gives an accurate distance - * across inactive_age overflows in most cases. There is a + * across nonresident_age overflows in most cases. There is a * special case: usually, shadow entries have a short lifetime * and are either refaulted or reclaimed along with the inode * before they get too old. But it is not impossible for the - * inactive_age to lap a shadow entry in the field, which can - * then result in a false small refault distance, leading to a - * false activation should this old entry actually refault - * again. However, earlier kernels used to deactivate + * nonresident_age to lap a shadow entry in the field, which + * can then result in a false small refault distance, leading + * to a false activation should this old entry actually + * refault again. However, earlier kernels used to deactivate * unconditionally with *every* reclaim invocation for the * longest time, so the occasional inappropriate activation * leading to pressure on the active list is not a problem. @@ -359,7 +365,7 @@ void workingset_refault(struct page *page, void *shad= ow) goto out; =20 SetPageActive(page); - advance_inactive_age(memcg, pgdat); + workingset_age_nonresident(lruvec, hpage_nr_pages(page)); inc_lruvec_state(lruvec, WORKINGSET_ACTIVATE); =20 /* Page was active prior to eviction */ @@ -378,6 +384,7 @@ void workingset_refault(struct page *page, void *shad= ow) void workingset_activation(struct page *page) { struct mem_cgroup *memcg; + struct lruvec *lruvec; =20 rcu_read_lock(); /* @@ -390,7 +397,8 @@ void workingset_activation(struct page *page) memcg =3D page_memcg_rcu(page); if (!mem_cgroup_disabled() && !memcg) goto out; - advance_inactive_age(memcg, page_pgdat(page)); + lruvec =3D mem_cgroup_page_lruvec(page, page_pgdat(page)); + workingset_age_nonresident(lruvec, hpage_nr_pages(page)); out: rcu_read_unlock(); } --=20 2.26.2