From: Mel Gorman <mgorman@techsingularity.net>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: David Rientjes <rientjes@google.com>,
Andrea Arcangeli <aarcange@redhat.com>,
Vlastimil Babka <vbabka@suse.cz>,
Linux List Kernel Mailing <linux-kernel@vger.kernel.org>,
Linux-MM <linux-mm@kvack.org>,
Mel Gorman <mgorman@techsingularity.net>
Subject: [PATCH 20/22] mm, compaction: Sample pageblocks for free pages
Date: Fri, 18 Jan 2019 17:51:34 +0000 [thread overview]
Message-ID: <20190118175136.31341-21-mgorman@techsingularity.net> (raw)
In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net>
Once fast searching finishes, there is a possibility that the linear
scanner is scanning full blocks found by the fast scanner earlier. This
patch uses an adaptive stride to sample pageblocks for free pages. The
more consecutive full pageblocks encountered, the larger the stride until
a pageblock with free pages is found. The scanners might meet slightly
sooner but it is an acceptable risk given that the search of the free
lists may still encounter the pages and adjust the cached PFN of the free
scanner accordingly.
5.0.0-rc1 5.0.0-rc1
roundrobin-v3r17 samplefree-v3r17
Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%*
Amean fault-both-3 2752.37 ( 0.00%) 2729.95 ( 0.81%)
Amean fault-both-5 4341.69 ( 0.00%) 4397.80 ( -1.29%)
Amean fault-both-7 6308.75 ( 0.00%) 6097.61 ( 3.35%)
Amean fault-both-12 10241.81 ( 0.00%) 9407.15 ( 8.15%)
Amean fault-both-18 13736.09 ( 0.00%) 10857.63 * 20.96%*
Amean fault-both-24 16853.95 ( 0.00%) 13323.24 * 20.95%*
Amean fault-both-30 15862.61 ( 0.00%) 17345.44 ( -9.35%)
Amean fault-both-32 18450.85 ( 0.00%) 16892.00 ( 8.45%)
The latency is mildly improved offseting some overhead from earlier
patches that are prerequisites for the rest of the series. However,
a major impact is on the free scan rate with an 82% reduction.
5.0.0-rc1 5.0.0-rc1
roundrobin-v3r17 samplefree-v3r17
Compaction migrate scanned 21607271 20116887
Compaction free scanned 95336406 16668703
It's also the first time in the series where the number of pages scanned
by the migration scanner is greater than the free scanner due to the
increased search efficiency.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/compaction.c | 28 ++++++++++++++++++++++------
1 file changed, 22 insertions(+), 6 deletions(-)
diff --git a/mm/compaction.c b/mm/compaction.c
index 7d39ff08269a..74bf620e3dcd 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -440,6 +440,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc,
unsigned long *start_pfn,
unsigned long end_pfn,
struct list_head *freelist,
+ unsigned int stride,
bool strict)
{
int nr_scanned = 0, total_isolated = 0;
@@ -449,10 +450,14 @@ static unsigned long isolate_freepages_block(struct compact_control *cc,
unsigned long blockpfn = *start_pfn;
unsigned int order;
+ /* Strict mode is for isolation, speed is secondary */
+ if (strict)
+ stride = 1;
+
cursor = pfn_to_page(blockpfn);
/* Isolate free pages. */
- for (; blockpfn < end_pfn; blockpfn++, cursor++) {
+ for (; blockpfn < end_pfn; blockpfn += stride, cursor += stride) {
int isolated;
struct page *page = cursor;
@@ -614,7 +619,7 @@ isolate_freepages_range(struct compact_control *cc,
break;
isolated = isolate_freepages_block(cc, &isolate_start_pfn,
- block_end_pfn, &freelist, true);
+ block_end_pfn, &freelist, 0, true);
/*
* In strict mode, isolate_freepages_block() returns 0 if
@@ -1132,7 +1137,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long
/* Scan before */
if (start_pfn != pfn) {
- isolate_freepages_block(cc, &start_pfn, pfn, &cc->freepages, false);
+ isolate_freepages_block(cc, &start_pfn, pfn, &cc->freepages, 1, false);
if (cc->nr_freepages >= cc->nr_migratepages)
return;
}
@@ -1140,7 +1145,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long
/* Scan after */
start_pfn = pfn + nr_isolated;
if (start_pfn != end_pfn)
- isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, false);
+ isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, 1, false);
/* Skip this pageblock in the future as it's full or nearly full */
if (cc->nr_freepages < cc->nr_migratepages)
@@ -1332,6 +1337,7 @@ static void isolate_freepages(struct compact_control *cc)
unsigned long block_end_pfn; /* end of current pageblock */
unsigned long low_pfn; /* lowest pfn scanner is able to scan */
struct list_head *freelist = &cc->freepages;
+ unsigned int stride;
/* Try a small search of the free lists for a candidate */
isolate_start_pfn = fast_isolate_freepages(cc);
@@ -1354,6 +1360,7 @@ static void isolate_freepages(struct compact_control *cc)
block_end_pfn = min(block_start_pfn + pageblock_nr_pages,
zone_end_pfn(zone));
low_pfn = pageblock_end_pfn(cc->migrate_pfn);
+ stride = cc->mode == MIGRATE_ASYNC ? COMPACT_CLUSTER_MAX : 1;
/*
* Isolate free pages until enough are available to migrate the
@@ -1364,6 +1371,8 @@ static void isolate_freepages(struct compact_control *cc)
block_end_pfn = block_start_pfn,
block_start_pfn -= pageblock_nr_pages,
isolate_start_pfn = block_start_pfn) {
+ unsigned long nr_isolated;
+
/*
* This can iterate a massively long zone without finding any
* suitable migration targets, so periodically check resched.
@@ -1385,8 +1394,8 @@ static void isolate_freepages(struct compact_control *cc)
continue;
/* Found a block suitable for isolating free pages from. */
- isolate_freepages_block(cc, &isolate_start_pfn, block_end_pfn,
- freelist, false);
+ nr_isolated = isolate_freepages_block(cc, &isolate_start_pfn,
+ block_end_pfn, freelist, stride, false);
/* Update the skip hint if the full pageblock was scanned */
if (isolate_start_pfn == block_end_pfn)
@@ -1410,6 +1419,13 @@ static void isolate_freepages(struct compact_control *cc)
*/
break;
}
+
+ /* Adjust stride depending on isolation */
+ if (nr_isolated) {
+ stride = 1;
+ continue;
+ }
+ stride = min_t(unsigned int, COMPACT_CLUSTER_MAX, stride << 1);
}
/*
--
2.16.4
next prev parent reply other threads:[~2019-01-18 17:55 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-18 17:51 [PATCH 00/22] Increase success rates and reduce latency of compaction v3 Mel Gorman
2019-01-18 17:51 ` [PATCH 01/22] mm, compaction: Shrink compact_control Mel Gorman
2019-01-18 17:51 ` [PATCH 02/22] mm, compaction: Rearrange compact_control Mel Gorman
2019-01-18 17:51 ` [PATCH 03/22] mm, compaction: Remove last_migrated_pfn from compact_control Mel Gorman
2019-01-18 17:51 ` [PATCH 04/22] mm, compaction: Remove unnecessary zone parameter in some instances Mel Gorman
2019-01-18 17:51 ` [PATCH 05/22] mm, compaction: Rename map_pages to split_map_pages Mel Gorman
2019-01-18 17:51 ` [PATCH 06/22] mm, migrate: Immediately fail migration of a page with no migration handler Mel Gorman
2019-01-18 17:51 ` [PATCH 07/22] mm, compaction: Always finish scanning of a full pageblock Mel Gorman
2019-01-18 17:51 ` [PATCH 08/22] mm, compaction: Ignore the fragmentation avoidance boost for isolation and compaction Mel Gorman
2019-01-18 17:51 ` [PATCH 09/22] mm, compaction: Use free lists to quickly locate a migration source Mel Gorman
2019-01-31 13:55 ` Vlastimil Babka
2019-01-31 14:12 ` Vlastimil Babka
2019-02-01 15:06 ` Mel Gorman
2019-02-04 8:55 ` [PATCH] mm, compaction: Use free lists to quickly locate a migration source -fix Mel Gorman
2019-02-08 17:10 ` [PATCH 09/22] mm, compaction: Use free lists to quickly locate a migration source Vlastimil Babka
2019-01-18 17:51 ` [PATCH 10/22] mm, compaction: Keep migration source private to a single compaction instance Mel Gorman
2019-01-18 17:51 ` [PATCH 11/22] mm, compaction: Use free lists to quickly locate a migration target Mel Gorman
2019-01-31 14:52 ` Vlastimil Babka
2019-02-01 14:51 ` Mel Gorman
2019-02-01 14:58 ` Vlastimil Babka
2019-02-04 12:01 ` [PATCH] mm, compaction: Use free lists to quickly locate a migration source -fix Mel Gorman
2021-01-12 5:19 ` [PATCH] mm, compaction: make sure we isolate a valid freepage when high_pfn is used Rokudo Yan
2021-01-12 9:10 ` Mel Gorman
2021-01-12 9:47 ` [PATCH] mm, compaction: move high_pfn to the for loop scope Rokudo Yan
2021-01-12 10:45 ` Mel Gorman
2021-01-12 10:48 ` Vlastimil Babka
2021-01-12 22:27 ` Andrew Morton
2021-01-18 7:41 ` Rokudo Yan
2021-01-18 9:42 ` Mel Gorman
2019-01-18 17:51 ` [PATCH 12/22] mm, compaction: Avoid rescanning the same pageblock multiple times Mel Gorman
2019-01-18 17:51 ` [PATCH 13/22] mm, compaction: Finish pageblock scanning on contention Mel Gorman
2019-01-18 17:51 ` [PATCH 14/22] mm, compaction: Check early for huge pages encountered by the migration scanner Mel Gorman
2019-01-18 17:51 ` [PATCH 15/22] mm, compaction: Keep cached migration PFNs synced for unusable pageblocks Mel Gorman
2019-01-18 17:51 ` [PATCH 16/22] mm, compaction: Rework compact_should_abort as compact_check_resched Mel Gorman
2019-01-18 17:51 ` [PATCH 17/22] mm, compaction: Do not consider a need to reschedule as contention Mel Gorman
2019-01-18 17:51 ` [PATCH 18/22] mm, compaction: Reduce premature advancement of the migration target scanner Mel Gorman
2019-01-18 17:51 ` [PATCH 19/22] mm, compaction: Round-robin the order while searching the free lists for a target Mel Gorman
2019-01-18 17:51 ` Mel Gorman [this message]
2019-01-31 15:39 ` [PATCH 20/22] mm, compaction: Sample pageblocks for free pages Vlastimil Babka
2019-01-18 17:51 ` [PATCH 21/22] mm, compaction: Be selective about what pageblocks to clear skip hints Mel Gorman
2019-01-31 15:45 ` Vlastimil Babka
2019-01-18 17:51 ` [PATCH 22/22] mm, compaction: Capture a page under direct compaction Mel Gorman
2019-01-31 16:11 ` Vlastimil Babka
2019-02-01 14:38 ` [PATCH] mm, compaction: Capture a page under direct compaction -fix Mel Gorman
2019-01-24 8:53 ` [PATCH 00/22] Increase success rates and reduce latency of compaction v3 Mel Gorman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190118175136.31341-21-mgorman@techsingularity.net \
--to=mgorman@techsingularity.net \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=rientjes@google.com \
--cc=vbabka@suse.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).