From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 982DAC433DF for ; Wed, 27 May 2020 13:52:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3F7D5208C3 for ; Wed, 27 May 2020 13:52:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=cmpxchg-org.20150623.gappssmtp.com header.i=@cmpxchg-org.20150623.gappssmtp.com header.b="SQkna51R" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3F7D5208C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C8366800DC; Wed, 27 May 2020 09:52:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C345E80010; Wed, 27 May 2020 09:52:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AFAEA800DC; Wed, 27 May 2020 09:52:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0247.hostedemail.com [216.40.44.247]) by kanga.kvack.org (Postfix) with ESMTP id 9952C80010 for ; Wed, 27 May 2020 09:52:20 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 5DE42180AD822 for ; Wed, 27 May 2020 13:52:20 +0000 (UTC) X-FDA: 76862638440.03.maid24_1f1825826d52 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id 3809D28A4E8 for ; Wed, 27 May 2020 13:52:20 +0000 (UTC) X-HE-Tag: maid24_1f1825826d52 X-Filterd-Recvd-Size: 8254 Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Wed, 27 May 2020 13:52:19 +0000 (UTC) Received: by mail-pf1-f193.google.com with SMTP id y18so11831102pfl.9 for ; Wed, 27 May 2020 06:52:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=IO5JhFmIPCTYFLrUzI9rZeUIf5+8TuwzK/GlIcJJS98=; b=SQkna51R7cwNgROOrm1TMMvR+y+K6+JlvOcSNGkgvqFFTS/q8pcYbviD2+iO5Zhbuq 5XwtvcvruQ8GBv556BflApV3T8mCgB6Rat4ZU2uqQJ9r/S3bM+VZF8qr/sb4qX3CkPsL HgudQQSThhK9w3iyn1NgPzErW7bOz5mFbHc0sg1kv1JBxbEr7T2RmJ0t7GKFxx19Peit E8GEjlGS+eAInm90g7ZO3RLKXwedXBBTcZvbt8HLfoaaHmkfIfxrJnEkhy1U2JOAAlm+ 4tyhf5LBv29vAaqjvJC4Ma09l8IZh5qRD/bme174FEwtHgJHoJ7xThl8x6uTHUxILDN/ B9SQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=IO5JhFmIPCTYFLrUzI9rZeUIf5+8TuwzK/GlIcJJS98=; b=gjmLaBdKd+VPc5+uloXye8D4YQG88STNKm/DetrXqqKAFaeSPruIZT1xTwZe69agV9 7mzpeQVoqMfMUq9LhjuwYGF88I+ZrYf+WQddrUjjBZ5gcZjhoLylLpyzqnIabZOlerNE mLjWgwMOqyzyerFueENPDRAMCLoasP7L3adhgxG+f8SVr6tj6Nl+Lf8JrvaZ1y9UgBeD 0+0GWmZ8TpgPE3a02+kqZqWQtL/YCuX5Kls5cbDpm9DB7A56zZAoc0R6EMDzgNQESY6Z XlXGvWTgYA6k+ascoyUlQhf80021bTxThmnbioLOg5xsY4iu03URwyOe1V+LF0yN/KBZ BOug== X-Gm-Message-State: AOAM532s9j9CUVliYiS3Dz+MjUc18hlkvjeix7Vne3H0gzfKYN5uHaMw hSQJUGbuvGeus9YMDeU/lQCLy1Yo/LE= X-Google-Smtp-Source: ABdhPJx6REUHl/zGkCE1Q/7yFh4jvtGaH1lpWxAb8Ty6lkqqea1s81qP1qKnsPDFktJE1n0Mq5zjEA== X-Received: by 2002:a0c:b492:: with SMTP id c18mr25889059qve.139.1590587038483; Wed, 27 May 2020 06:43:58 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::1:8a69]) by smtp.gmail.com with ESMTPSA id g13sm2152129qki.95.2020.05.27.06.43.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 May 2020 06:43:57 -0700 (PDT) Date: Wed, 27 May 2020 09:43:33 -0400 From: Johannes Weiner To: Joonsoo Kim Cc: Linux Memory Management List , Rik van Riel , Minchan Kim , Michal Hocko , Andrew Morton , Joonsoo Kim , LKML , kernel-team@fb.com Subject: Re: [PATCH 05/14] mm: workingset: let cache workingset challenge anon Message-ID: <20200527134333.GF6781@cmpxchg.org> References: <20200520232525.798933-1-hannes@cmpxchg.org> <20200520232525.798933-6-hannes@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 3809D28A4E8 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, May 27, 2020 at 11:06:47AM +0900, Joonsoo Kim wrote: > 2020=EB=85=84 5=EC=9B=94 21=EC=9D=BC (=EB=AA=A9) =EC=98=A4=EC=A0=84 8:2= 6, Johannes Weiner =EB=8B=98=EC=9D=B4 =EC=9E=91=EC=84= =B1: > > > > We activate cache refaults with reuse distances in pages smaller than > > the size of the total cache. This allows new pages with competitive > > access frequencies to establish themselves, as well as challenge and > > potentially displace pages on the active list that have gone cold. > > > > However, that assumes that active cache can only replace other active > > cache in a competition for the hottest memory. This is not a great > > default assumption. The page cache might be thrashing while there are > > enough completely cold and unused anonymous pages sitting around that > > we'd only have to write to swap once to stop all IO from the cache. > > > > Activate cache refaults when their reuse distance in pages is smaller > > than the total userspace workingset, including anonymous pages. >=20 > Hmm... I'm not sure the correctness of this change. >=20 > IIUC, this patch leads to more activations in the file list and more ac= tivations > here will challenge the anon list since rotation ratio for the file > list will be increased. Yes. > However, this change breaks active/inactive concept of the file list. > active/inactive > separation is implemented by in-list refault distance. anon list size h= as > no direct connection with refault distance of the file list so using > anon list size > to detect workingset for file page breaks the concept. This is intentional, because there IS a connection: they both take up space in RAM, and they both cost IO to bring back once reclaimed. When file is refaulting, it means we need to make more space for cache. That space can come from stale active file pages. But what if active cache is all hot, and meanwhile there are cold anon pages that we could swap out once and then serve everything from RAM? When file is refaulting, we should find the coldest data that is taking up RAM and kick it out. It doesn't matter whether it's file or anon: the goal is to free up RAM with the least amount of IO risk. Remember that the file/anon split, and the inactive/active split, are there to optimize reclaim. It doesn't mean that these memory pools are independent from each other. The file list is split in two because of use-once cache. The anon and file lists are split because of different IO patterns, because we may not have swap etc. But once we are out of use-once cache, have swap space available, and have corrected for the different cost of IO, there needs to be a relative order between all pages in the system to find the optimal candidates to reclaim. > My suspicion is started by this counter example. >=20 > Environment: > anon: 500 MB (so hot) / 500 MB (so hot) > file: 50 MB (hot) / 50 MB (cold) >=20 > Think about the situation that there is periodical access to other file= (100 MB) > with low frequency (refault distance is 500 MB) >=20 > Without your change, this periodical access doesn't make thrashing for = cached > active file page since refault distance of periodical access is larger > than the size of > the active file list. However, with your change, it causes thrashing > on the file list. It doesn't cause thrashing. It causes scanning because that 100M file IS thrashing: with or without my patch, that refault IO is occuring. What this patch acknowledges is that the 100M file COULD fit fully into memory, and not require any IO to serve, IFF 100M of the active file or anon pages were cold and could be reclaimed or swapped out. In your example, the anon set is hot. We'll scan it slowly (at the rate of IO from the other file) and rotate the pages that are in use - which would be all of them. Likewise for the file - there will be some deactivations, but mark_page_accessed() or the second chance algorithm in page_check_references() for mapped will keep the hottest pages active. In a slightly modified example, 400M of the anon set is hot and 100M cold. Without my patch, we would never look for them and the second file would be IO-bound forever. After my patch, we would scan anon, eventually find the cold pages, swap them out, and then serve the entire workingset from memory. Does it cause more scanning than before in your scenario? Yes, but we don't even know it's your scenario instead of mine until we actually sample references of all memory. Not scanning is a false stable state. And your scenario could easily change over time. Even if the amount of repeatedly accessed pages stays larger than memory, and will always require IO to serve, the relative access frequencies could change. Some pages could become hotter, others colder. Without scanning, we wouldn't adapt the LRU ordering and cause more IO than necessary.