linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed
From: Alex Shi <alex.shi@linux.alibaba.com>
To: cgroups@vger.kernel.org, akpm@linux-foundation.org,
	mgorman@techsingularity.net, tj@kernel.org, hughd@google.com,
	khlebnikov@yandex-team.ru, daniel.m.jordan@oracle.com,
	yang.shi@linux.alibaba.com, willy@infradead.org,
	hannes@cmpxchg.org, lkp@intel.com
Cc: Alex Shi <alex.shi@linux.alibaba.com>,
	linux-kernel@vger.kernel.org, linux-mm@kvack.org
Subject: [PATCH v9 08/20] mm/lru: add page isolation precondition in __isolate_lru_page
Date: Mon,  2 Mar 2020 19:00:18 +0800	[thread overview]
Message-ID: <1583146830-169516-9-git-send-email-alex.shi@linux.alibaba.com> (raw)
In-Reply-To: <1583146830-169516-1-git-send-email-alex.shi@linux.alibaba.com>

Johannes Weiner has suggested:
"
So here is a crazy idea that may be worth exploring:

Right now, pgdat->lru_lock protects both PageLRU *and* the lruvec's
linked list.

Can we make PageLRU atomic and use it to stabilize the lru_lock
instead, and then use the lru_lock only serialize list operations?
...
"

Yes, this patch is doing so on  __isolate_lru_page which is the core
page isolation func in compaction and shrinking path.

This patch move clear page lru action before compaction getting lru_lock,
makes it as a necessary condition for page isolation. Hence, PageLRU may
be cleared druing shrink_inactive_list path for isolation reason. If so,
we can skip that page's in reclaim.

It's a preparation for later per memcg lru_lock change.

Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Alex Shi <alex.shi@linux.alibaba.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mm@kvack.org
---
 include/linux/swap.h |  2 +-
 mm/compaction.c      | 25 +++++++++++++++++--------
 mm/vmscan.c          | 48 ++++++++++++++++++++++++++----------------------
 3 files changed, 44 insertions(+), 31 deletions(-)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index c555e8f161ad..69f0794f1da3 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -351,7 +351,7 @@ extern void lru_cache_add_active_or_unevictable(struct page *page,
 extern unsigned long zone_reclaimable_pages(struct zone *zone);
 extern unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
 					gfp_t gfp_mask, nodemask_t *mask);
-extern int __isolate_lru_page(struct page *page, isolate_mode_t mode);
+extern int __isolate_lru_page_prepare(struct page *page, isolate_mode_t mode);
 extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
 						  unsigned long nr_pages,
 						  gfp_t gfp_mask,
diff --git a/mm/compaction.c b/mm/compaction.c
index 672d3c78c6ab..1baba328d089 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -948,6 +948,23 @@ static bool too_many_isolated(pg_data_t *pgdat)
 		if (!(cc->gfp_mask & __GFP_FS) && page_mapping(page))
 			goto isolate_fail;
 
+		if (__isolate_lru_page_prepare(page, isolate_mode) != 0)
+			goto isolate_fail;
+
+		/*
+		 * Be careful not to clear PageLRU until after we're
+		 * sure the page is not being freed elsewhere -- the
+		 * page release code relies on it.
+		 */
+		if (unlikely(!get_page_unless_zero(page)))
+			goto isolate_fail;
+
+		/* Try isolate the page */
+		if (!TestClearPageLRU(page)) {
+			put_page(page);
+			goto isolate_fail;
+		}
+
 		/* If we already hold the lock, we can skip some rechecking */
 		if (!locked) {
 			locked = compact_lock_irqsave(&pgdat->lru_lock,
@@ -960,10 +977,6 @@ static bool too_many_isolated(pg_data_t *pgdat)
 					goto isolate_abort;
 			}
 
-			/* Recheck PageLRU and PageCompound under lock */
-			if (!PageLRU(page))
-				goto isolate_fail;
-
 			/*
 			 * Page become compound since the non-locked check,
 			 * and it's on LRU. It can only be a THP so the order
@@ -977,10 +990,6 @@ static bool too_many_isolated(pg_data_t *pgdat)
 
 		lruvec = mem_cgroup_page_lruvec(page, pgdat);
 
-		/* Try isolate the page */
-		if (__isolate_lru_page(page, isolate_mode) != 0)
-			goto isolate_fail;
-
 		VM_BUG_ON_PAGE(PageCompound(page), page);
 
 		/* Successfully isolated */
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 8958454d50fe..bc2ec3fe4f48 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1522,20 +1522,20 @@ unsigned long reclaim_clean_pages_from_list(struct zone *zone,
  *
  * returns 0 on success, -ve errno on failure.
  */
-int __isolate_lru_page(struct page *page, isolate_mode_t mode)
+int __isolate_lru_page_prepare(struct page *page, isolate_mode_t mode)
 {
 	int ret = -EINVAL;
 
-	/* Only take pages on the LRU. */
-	if (!PageLRU(page))
-		return ret;
-
 	/* Compaction should not handle unevictable pages but CMA can do so */
 	if (PageUnevictable(page) && !(mode & ISOLATE_UNEVICTABLE))
 		return ret;
 
 	ret = -EBUSY;
 
+	/* Only take pages on the LRU. */
+	if (!PageLRU(page))
+		return ret;
+
 	/*
 	 * To minimise LRU disruption, the caller can indicate that it only
 	 * wants to isolate pages it will be able to operate on without
@@ -1576,20 +1576,9 @@ int __isolate_lru_page(struct page *page, isolate_mode_t mode)
 	if ((mode & ISOLATE_UNMAPPED) && page_mapped(page))
 		return ret;
 
-	if (likely(get_page_unless_zero(page))) {
-		/*
-		 * Be careful not to clear PageLRU until after we're
-		 * sure the page is not being freed elsewhere -- the
-		 * page release code relies on it.
-		 */
-		ClearPageLRU(page);
-		ret = 0;
-	}
-
-	return ret;
+	return 0;
 }
 
-
 /*
  * Update LRU sizes after isolating pages. The LRU size updates must
  * be complete before mem_cgroup_update_lru_size due to a santity check.
@@ -1653,8 +1642,6 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan,
 		page = lru_to_page(src);
 		prefetchw_prev_lru_page(page, src, flags);
 
-		VM_BUG_ON_PAGE(!PageLRU(page), page);
-
 		nr_pages = compound_nr(page);
 		total_scan += nr_pages;
 
@@ -1675,17 +1662,34 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan,
 		 * only when the page is being freed somewhere else.
 		 */
 		scan += nr_pages;
-		switch (__isolate_lru_page(page, mode)) {
+		switch (__isolate_lru_page_prepare(page, mode)) {
 		case 0:
+			/*
+			 * Be careful not to clear PageLRU until after we're
+			 * sure the page is not being freed elsewhere -- the
+			 * page release code relies on it.
+			 */
+			if (unlikely(!get_page_unless_zero(page)))
+				goto busy;
+
+			if (!TestClearPageLRU(page)) {
+				/*
+				 * This page may in other isolation path,
+				 * but we still hold lru_lock.
+				 */
+				put_page(page);
+				goto busy;
+			}
+
 			nr_taken += nr_pages;
 			nr_zone_taken[page_zonenum(page)] += nr_pages;
 			list_move(&page->lru, dst);
 			break;
-
+busy:
 		case -EBUSY:
 			/* else it is being freed elsewhere */
 			list_move(&page->lru, src);
-			continue;
+			break;
 
 		default:
 			BUG();
-- 
1.8.3.1



  parent reply	other threads:[~2020-03-02 11:01 UTC|newest]

Thread overview: 32+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <1583146830-169516-1-git-send-email-alex.shi@linux.alibaba.com>
2020-03-02 11:00 ` [PATCH v9 01/20] mm/vmscan: remove unnecessary lruvec adding Alex Shi
2020-03-02 11:00 ` [PATCH v9 02/20] mm/memcg: fold lock_page_lru into commit_charge Alex Shi
2020-03-02 11:00 ` [PATCH v9 03/20] mm/page_idle: no unlikely double check for idle page counting Alex Shi
2020-03-02 11:00 ` [PATCH v9 04/20] mm/thp: move lru_add_page_tail func to huge_memory.c Alex Shi
2020-03-04  7:47   ` Kirill A. Shutemov
2020-03-04  8:13     ` Alex Shi
2020-03-02 11:00 ` [PATCH v9 05/20] mm/thp: clean up lru_add_page_tail Alex Shi
2020-03-02 11:00 ` [PATCH v9 06/20] mm/thp: narrow lru locking Alex Shi
2020-03-04  8:02   ` Kirill A. Shutemov
2020-03-04  8:51     ` Alex Shi
2020-03-02 11:00 ` [PATCH v9 07/20] mm/lru: introduce TestClearPageLRU Alex Shi
2020-03-02 22:11   ` Andrew Morton
2020-03-03  4:11     ` Alex Shi
2020-03-04  0:46       ` Andrew Morton
2020-03-04  7:06         ` Alex Shi
2020-03-04  9:03           ` Rong Chen
2020-03-04  9:37             ` Alex Shi
2020-03-02 11:00 ` Alex Shi [this message]
2020-03-02 11:00 ` [PATCH v9 09/20] mm/mlock: ClearPageLRU before get lru lock in munlock page isolation Alex Shi
2020-03-02 11:00 ` [PATCH v9 10/20] mm/lru: take PageLRU first in moving page between lru lists Alex Shi
2020-03-02 11:00 ` [PATCH v9 11/20] mm/memcg: move SetPageLRU out of lru_lock in commit_charge Alex Shi
2020-03-02 11:00 ` [PATCH v9 12/20] mm/mlock: clean up __munlock_isolate_lru_page Alex Shi
2020-03-02 11:00 ` [PATCH v9 13/20] mm/lru: replace pgdat lru_lock with lruvec lock Alex Shi
2020-03-02 11:00 ` [PATCH v9 14/20] mm/lru: introduce the relock_page_lruvec function Alex Shi
2020-03-02 11:00 ` [PATCH v9 15/20] mm/mlock: optimize munlock_pagevec by relocking Alex Shi
2020-03-02 11:00 ` [PATCH v9 16/20] mm/swap: only change the lru_lock iff page's lruvec is different Alex Shi
2020-03-02 11:00 ` [PATCH v9 17/20] mm/pgdat: remove pgdat lru_lock Alex Shi
2020-03-02 11:00 ` [PATCH v9 18/20] mm/lru: revise the comments of lru_lock Alex Shi
2020-03-02 11:00 ` [PATCH v9 19/20] mm/lru: add debug checking for page memcg moving Alex Shi
2020-03-02 11:00 ` [PATCH v9 20/20] mm/memcg: add debug checking in lock_page_memcg Alex Shi
2020-03-04  3:13 ` [PATCH v9 02/20] mm/memcg: fold lock_page_lru into commit_charge Hillf Danton
2020-03-04  7:19   ` Alex Shi

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1583146830-169516-9-git-send-email-alex.shi@linux.alibaba.com \
    --to=alex.shi@linux.alibaba.com \
    --cc=akpm@linux-foundation.org \
    --cc=cgroups@vger.kernel.org \
    --cc=daniel.m.jordan@oracle.com \
    --cc=hannes@cmpxchg.org \
    --cc=hughd@google.com \
    --cc=khlebnikov@yandex-team.ru \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=lkp@intel.com \
    --cc=mgorman@techsingularity.net \
    --cc=tj@kernel.org \
    --cc=willy@infradead.org \
    --cc=yang.shi@linux.alibaba.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).