linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Mel Gorman <mgorman@techsingularity.net>
To: Linux-MM <linux-mm@kvack.org>
Cc: David Rientjes <rientjes@google.com>,
	Andrea Arcangeli <aarcange@redhat.com>,
	Vlastimil Babka <vbabka@suse.cz>,
	ying.huang@intel.com, kirill@shutemov.name,
	Andrew Morton <akpm@linux-foundation.org>,
	Linux List Kernel Mailing <linux-kernel@vger.kernel.org>,
	Mel Gorman <mgorman@techsingularity.net>
Subject: [PATCH 21/25] mm, compaction: Round-robin the order while searching the free lists for a target
Date: Fri,  4 Jan 2019 12:50:07 +0000	[thread overview]
Message-ID: <20190104125011.16071-22-mgorman@techsingularity.net> (raw)
In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net>

As compaction proceeds and creates high-order blocks, the free list
search gets less efficient as the larger blocks are used as compaction
targets. Eventually, the larger blocks will be behind the migration
scanner for partially migrated pageblocks and the search fails. This
patch round-robins what orders are searched so that larger blocks can be
ignored and find smaller blocks that can be used as migration targets.

The overall impact was small on 1-socket but it avoids corner cases where
the migration/free scanners meet prematurely or situations where many of
the pageblocks encountered by the free scanner are almost full instead of
being properly packed. Previous testing had indicated that without this
patch there were occasional large spikes in the free scanner without this
patch. By co-incidence, the 2-socket results showed a 54% reduction in
the free scanner but will not be universally true.

Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
 mm/compaction.c | 33 ++++++++++++++++++++++++++++++---
 mm/internal.h   |  3 ++-
 2 files changed, 32 insertions(+), 4 deletions(-)

diff --git a/mm/compaction.c b/mm/compaction.c
index 6c5552c6d8f9..652e249168b1 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -1154,6 +1154,24 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long
 		set_pageblock_skip(page);
 }
 
+/* Search orders in round-robin fashion */
+static int next_search_order(struct compact_control *cc, int order)
+{
+	order--;
+	if (order < 0)
+		order = cc->order - 1;
+
+	/* Search wrapped around? */
+	if (order == cc->search_order) {
+		cc->search_order--;
+		if (cc->search_order < 0)
+			cc->search_order = cc->order - 1;
+		return -1;
+	}
+
+	return order;
+}
+
 static unsigned long
 fast_isolate_freepages(struct compact_control *cc)
 {
@@ -1186,9 +1204,15 @@ fast_isolate_freepages(struct compact_control *cc)
 	if (WARN_ON_ONCE(min_pfn > low_pfn))
 		low_pfn = min_pfn;
 
-	for (order = cc->order - 1;
-	     order >= 0 && !page;
-	     order--) {
+	/*
+	 * Search starts from the last successful isolation order or the next
+	 * order to search after a previous failure
+	 */
+	cc->search_order = min_t(unsigned int, cc->order - 1, cc->search_order);
+
+	for (order = cc->search_order;
+	     !page && order >= 0;
+	     order = next_search_order(cc, order)) {
 		struct free_area *area = &cc->zone->free_area[order];
 		struct list_head *freelist;
 		struct page *freepage;
@@ -1211,6 +1235,7 @@ fast_isolate_freepages(struct compact_control *cc)
 
 			if (pfn >= low_pfn) {
 				cc->fast_search_fail = 0;
+				cc->search_order = order;
 				page = freepage;
 				break;
 			}
@@ -2146,6 +2171,7 @@ static enum compact_result compact_zone_order(struct zone *zone, int order,
 		.total_migrate_scanned = 0,
 		.total_free_scanned = 0,
 		.order = order,
+		.search_order = order,
 		.gfp_mask = gfp_mask,
 		.zone = zone,
 		.mode = (prio == COMPACT_PRIO_ASYNC) ?
@@ -2385,6 +2411,7 @@ static void kcompactd_do_work(pg_data_t *pgdat)
 	struct zone *zone;
 	struct compact_control cc = {
 		.order = pgdat->kcompactd_max_order,
+		.search_order = pgdat->kcompactd_max_order,
 		.total_migrate_scanned = 0,
 		.total_free_scanned = 0,
 		.classzone_idx = pgdat->kcompactd_classzone_idx,
diff --git a/mm/internal.h b/mm/internal.h
index e5ca2a10b8ad..d028abd8a8f3 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -191,7 +191,8 @@ struct compact_control {
 	struct zone *zone;
 	unsigned long total_migrate_scanned;
 	unsigned long total_free_scanned;
-	unsigned int fast_search_fail;	/* failures to use free list searches */
+	unsigned short fast_search_fail;/* failures to use free list searches */
+	unsigned short search_order;	/* order to start a fast search at */
 	const gfp_t gfp_mask;		/* gfp mask of a direct compactor */
 	int order;			/* order a direct compactor needs */
 	int migratetype;		/* migratetype of direct compactor */
-- 
2.16.4


  parent reply	other threads:[~2019-01-04 12:53 UTC|newest]

Thread overview: 75+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-01-04 12:49 [PATCH 00/25] Increase success rates and reduce latency of compaction v2 Mel Gorman
2019-01-04 12:49 ` [PATCH 01/25] mm, compaction: Shrink compact_control Mel Gorman
2019-01-04 12:49 ` [PATCH 02/25] mm, compaction: Rearrange compact_control Mel Gorman
2019-01-04 12:49 ` [PATCH 03/25] mm, compaction: Remove last_migrated_pfn from compact_control Mel Gorman
2019-01-04 12:49 ` [PATCH 04/25] mm, compaction: Remove unnecessary zone parameter in some instances Mel Gorman
2019-01-15 11:43   ` Vlastimil Babka
2019-01-04 12:49 ` [PATCH 05/25] mm, compaction: Rename map_pages to split_map_pages Mel Gorman
2019-01-15 11:59   ` Vlastimil Babka
2019-01-04 12:49 ` [PATCH 06/25] mm, compaction: Skip pageblocks with reserved pages Mel Gorman
2019-01-15 12:10   ` Vlastimil Babka
2019-01-15 12:50     ` Mel Gorman
2019-01-16  9:42       ` Mel Gorman
2019-01-04 12:49 ` [PATCH 07/25] mm, migrate: Immediately fail migration of a page with no migration handler Mel Gorman
2019-01-04 12:49 ` [PATCH 08/25] mm, compaction: Always finish scanning of a full pageblock Mel Gorman
2019-01-04 12:49 ` [PATCH 09/25] mm, compaction: Use the page allocator bulk-free helper for lists of pages Mel Gorman
2019-01-15 12:39   ` Vlastimil Babka
2019-01-16  9:46     ` Mel Gorman
2019-01-04 12:49 ` [PATCH 10/25] mm, compaction: Ignore the fragmentation avoidance boost for isolation and compaction Mel Gorman
2019-01-15 13:18   ` Vlastimil Babka
2019-01-04 12:49 ` [PATCH 11/25] mm, compaction: Use free lists to quickly locate a migration source Mel Gorman
2019-01-16 13:15   ` Vlastimil Babka
2019-01-16 14:33     ` Mel Gorman
2019-01-16 15:00       ` Vlastimil Babka
2019-01-16 15:43         ` Mel Gorman
2019-01-04 12:49 ` [PATCH 12/25] mm, compaction: Keep migration source private to a single compaction instance Mel Gorman
2019-01-16 15:45   ` Vlastimil Babka
2019-01-16 16:15     ` Mel Gorman
2019-01-17  9:29       ` Vlastimil Babka
2019-01-17  9:40   ` Vlastimil Babka
2019-01-04 12:49 ` [PATCH 13/25] mm, compaction: Use free lists to quickly locate a migration target Mel Gorman
2019-01-17 14:36   ` Vlastimil Babka
2019-01-17 15:51     ` Mel Gorman
2019-01-04 12:50 ` [PATCH 14/25] mm, compaction: Avoid rescanning the same pageblock multiple times Mel Gorman
2019-01-17 15:16   ` Vlastimil Babka
2019-01-17 16:00     ` Mel Gorman
2019-01-04 12:50 ` [PATCH 15/25] mm, compaction: Finish pageblock scanning on contention Mel Gorman
2019-01-17 16:38   ` Vlastimil Babka
2019-01-17 17:11     ` Mel Gorman
2019-01-18  8:57       ` Vlastimil Babka
2019-01-04 12:50 ` [PATCH 16/25] mm, compaction: Check early for huge pages encountered by the migration scanner Mel Gorman
2019-01-17 17:01   ` Vlastimil Babka
2019-01-17 17:35     ` Mel Gorman
2019-01-04 12:50 ` [PATCH 17/25] mm, compaction: Keep cached migration PFNs synced for unusable pageblocks Mel Gorman
2019-01-17 17:17   ` Vlastimil Babka
2019-01-17 17:37     ` Mel Gorman
2019-01-04 12:50 ` [PATCH 18/25] mm, compaction: Rework compact_should_abort as compact_check_resched Mel Gorman
2019-01-17 17:27   ` Vlastimil Babka
2019-01-04 12:50 ` [PATCH 19/25] mm, compaction: Do not consider a need to reschedule as contention Mel Gorman
2019-01-17 17:33   ` Vlastimil Babka
2019-01-17 18:05     ` Mel Gorman
2019-01-04 12:50 ` [PATCH 20/25] mm, compaction: Reduce unnecessary skipping of migration target scanner Mel Gorman
2019-01-17 17:58   ` Vlastimil Babka
2019-01-17 19:39     ` Mel Gorman
2019-01-18  9:09       ` Vlastimil Babka
2019-01-04 12:50 ` Mel Gorman [this message]
2019-01-18  9:17   ` [PATCH 21/25] mm, compaction: Round-robin the order while searching the free lists for a target Vlastimil Babka
2019-01-04 12:50 ` [PATCH 22/25] mm, compaction: Sample pageblocks for free pages Mel Gorman
2019-01-18 10:38   ` Vlastimil Babka
2019-01-18 13:44     ` Mel Gorman
2019-01-04 12:50 ` [PATCH 23/25] mm, compaction: Be selective about what pageblocks to clear skip hints Mel Gorman
2019-01-18 12:55   ` Vlastimil Babka
2019-01-18 14:10     ` Mel Gorman
2019-01-04 12:50 ` [PATCH 24/25] mm, compaction: Capture a page under direct compaction Mel Gorman
2019-01-18 13:40   ` Vlastimil Babka
2019-01-18 14:39     ` Mel Gorman
2019-01-04 12:50 ` [PATCH 25/25] mm, compaction: Do not direct compact remote memory Mel Gorman
2019-01-18 13:51   ` Vlastimil Babka
2019-01-18 14:46     ` Mel Gorman
2019-01-07 23:43 ` [PATCH 00/25] Increase success rates and reduce latency of compaction v2 Andrew Morton
2019-01-08  9:12   ` Mel Gorman
2019-01-09 11:13 ` [PATCH] mm, compaction: Use free lists to quickly locate a migration target -fix Mel Gorman
2019-01-09 19:27   ` Andrew Morton
2019-01-09 21:26     ` Mel Gorman
2019-01-09 11:15 ` [PATCH] mm, compaction: Finish pageblock scanning on contention -fix Mel Gorman
2019-01-09 11:16 ` [PATCH] mm, compaction: Round-robin the order while searching the free lists for a target -fix Mel Gorman

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190104125011.16071-22-mgorman@techsingularity.net \
    --to=mgorman@techsingularity.net \
    --cc=aarcange@redhat.com \
    --cc=akpm@linux-foundation.org \
    --cc=kirill@shutemov.name \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=rientjes@google.com \
    --cc=vbabka@suse.cz \
    --cc=ying.huang@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).