* [PATCH 1/6] mm: compaction: Abort compaction loop if lock is contended or run too long
2012-09-20 14:04 [PATCH 0/6] Reduce compaction scanning and lock contention Mel Gorman
@ 2012-09-20 14:04 ` Mel Gorman
2012-09-20 18:53 ` Rik van Riel
2012-09-20 14:04 ` [PATCH 2/6] mm: compaction: Acquire the zone->lru_lock as late as possible Mel Gorman
` (5 subsequent siblings)
6 siblings, 1 reply; 20+ messages in thread
From: Mel Gorman @ 2012-09-20 14:04 UTC (permalink / raw)
To: Richard Davies, Shaohua Li
Cc: Rik van Riel, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML, Mel Gorman
From: Shaohua Li <shli@fusionio.com>
Changelog since V2
o Fix BUG_ON triggered due to pages left on cc.migratepages
o Make compact_zone_order() require non-NULL arg `contended'
Changelog since V1
o only abort the compaction if lock is contended or run too long
o Rearranged the code by Andrea Arcangeli.
isolate_migratepages_range() might isolate no pages if for example when
zone->lru_lock is contended and running asynchronous compaction. In this
case, we should abort compaction, otherwise, compact_zone will run a
useless loop and make zone->lru_lock is even contended.
[minchan@kernel.org: Putback pages isolated for migration if aborting]
[akpm@linux-foundation.org: compact_zone_order requires non-NULL arg contended]
Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
Signed-off-by: Shaohua Li <shli@fusionio.com>
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/compaction.c | 17 ++++++++++++-----
mm/internal.h | 2 +-
2 files changed, 13 insertions(+), 6 deletions(-)
diff --git a/mm/compaction.c b/mm/compaction.c
index 7fcd3a5..a8de20d 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -70,8 +70,7 @@ static bool compact_checklock_irqsave(spinlock_t *lock, unsigned long *flags,
/* async aborts if taking too long or contended */
if (!cc->sync) {
- if (cc->contended)
- *cc->contended = true;
+ cc->contended = true;
return false;
}
@@ -634,7 +633,7 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone,
/* Perform the isolation */
low_pfn = isolate_migratepages_range(zone, cc, low_pfn, end_pfn);
- if (!low_pfn)
+ if (!low_pfn || cc->contended)
return ISOLATE_ABORT;
cc->migrate_pfn = low_pfn;
@@ -787,6 +786,8 @@ static int compact_zone(struct zone *zone, struct compact_control *cc)
switch (isolate_migratepages(zone, cc)) {
case ISOLATE_ABORT:
ret = COMPACT_PARTIAL;
+ putback_lru_pages(&cc->migratepages);
+ cc->nr_migratepages = 0;
goto out;
case ISOLATE_NONE:
continue;
@@ -831,6 +832,7 @@ static unsigned long compact_zone_order(struct zone *zone,
int order, gfp_t gfp_mask,
bool sync, bool *contended)
{
+ unsigned long ret;
struct compact_control cc = {
.nr_freepages = 0,
.nr_migratepages = 0,
@@ -838,12 +840,17 @@ static unsigned long compact_zone_order(struct zone *zone,
.migratetype = allocflags_to_migratetype(gfp_mask),
.zone = zone,
.sync = sync,
- .contended = contended,
};
INIT_LIST_HEAD(&cc.freepages);
INIT_LIST_HEAD(&cc.migratepages);
- return compact_zone(zone, &cc);
+ ret = compact_zone(zone, &cc);
+
+ VM_BUG_ON(!list_empty(&cc.freepages));
+ VM_BUG_ON(!list_empty(&cc.migratepages));
+
+ *contended = cc.contended;
+ return ret;
}
int sysctl_extfrag_threshold = 500;
diff --git a/mm/internal.h b/mm/internal.h
index b8c91b3..4bd7c0e 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -130,7 +130,7 @@ struct compact_control {
int order; /* order a direct compactor needs */
int migratetype; /* MOVABLE, RECLAIMABLE etc */
struct zone *zone;
- bool *contended; /* True if a lock was contended */
+ bool contended; /* True if a lock was contended */
};
unsigned long
--
1.7.9.2
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH 1/6] mm: compaction: Abort compaction loop if lock is contended or run too long
2012-09-20 14:04 ` [PATCH 1/6] mm: compaction: Abort compaction loop if lock is contended or run too long Mel Gorman
@ 2012-09-20 18:53 ` Rik van Riel
0 siblings, 0 replies; 20+ messages in thread
From: Rik van Riel @ 2012-09-20 18:53 UTC (permalink / raw)
To: Mel Gorman
Cc: Richard Davies, Shaohua Li, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML
On 09/20/2012 10:04 AM, Mel Gorman wrote:
> From: Shaohua Li <shli@fusionio.com>
>
> Changelog since V2
> o Fix BUG_ON triggered due to pages left on cc.migratepages
> o Make compact_zone_order() require non-NULL arg `contended'
>
> Changelog since V1
> o only abort the compaction if lock is contended or run too long
> o Rearranged the code by Andrea Arcangeli.
>
> isolate_migratepages_range() might isolate no pages if for example when
> zone->lru_lock is contended and running asynchronous compaction. In this
> case, we should abort compaction, otherwise, compact_zone will run a
> useless loop and make zone->lru_lock is even contended.
>
> [minchan@kernel.org: Putback pages isolated for migration if aborting]
> [akpm@linux-foundation.org: compact_zone_order requires non-NULL arg contended]
> Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
> Signed-off-by: Shaohua Li <shli@fusionio.com>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH 2/6] mm: compaction: Acquire the zone->lru_lock as late as possible
2012-09-20 14:04 [PATCH 0/6] Reduce compaction scanning and lock contention Mel Gorman
2012-09-20 14:04 ` [PATCH 1/6] mm: compaction: Abort compaction loop if lock is contended or run too long Mel Gorman
@ 2012-09-20 14:04 ` Mel Gorman
2012-09-20 18:54 ` Rik van Riel
2012-09-20 14:04 ` [PATCH 3/6] mm: compaction: Acquire the zone->lock " Mel Gorman
` (4 subsequent siblings)
6 siblings, 1 reply; 20+ messages in thread
From: Mel Gorman @ 2012-09-20 14:04 UTC (permalink / raw)
To: Richard Davies, Shaohua Li
Cc: Rik van Riel, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML, Mel Gorman
Compactions migrate scanner acquires the zone->lru_lock when scanning a range
of pages looking for LRU pages to acquire. It does this even if there are
no LRU pages in the range. If multiple processes are compacting then this
can cause severe locking contention. To make matters worse commit b2eef8c0
(mm: compaction: minimise the time IRQs are disabled while isolating pages
for migration) releases the lru_lock every SWAP_CLUSTER_MAX pages that are
scanned.
This patch makes two changes to how the migrate scanner acquires the LRU
lock. First, it only releases the LRU lock every SWAP_CLUSTER_MAX pages if
the lock is contended. This reduces the number of times it unnecessarily
disables and re-enables IRQs. The second is that it defers acquiring the
LRU lock for as long as possible. If there are no LRU pages or the only
LRU pages are transhuge then the LRU lock will not be acquired at all
which reduces contention on zone->lru_lock.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/compaction.c | 63 +++++++++++++++++++++++++++++++++++++------------------
1 file changed, 43 insertions(+), 20 deletions(-)
diff --git a/mm/compaction.c b/mm/compaction.c
index a8de20d..6450c3e 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -50,6 +50,11 @@ static inline bool migrate_async_suitable(int migratetype)
return is_migrate_cma(migratetype) || migratetype == MIGRATE_MOVABLE;
}
+static inline bool should_release_lock(spinlock_t *lock)
+{
+ return need_resched() || spin_is_contended(lock);
+}
+
/*
* Compaction requires the taking of some coarse locks that are potentially
* very heavily contended. Check if the process needs to be scheduled or
@@ -62,7 +67,7 @@ static inline bool migrate_async_suitable(int migratetype)
static bool compact_checklock_irqsave(spinlock_t *lock, unsigned long *flags,
bool locked, struct compact_control *cc)
{
- if (need_resched() || spin_is_contended(lock)) {
+ if (should_release_lock(lock)) {
if (locked) {
spin_unlock_irqrestore(lock, *flags);
locked = false;
@@ -275,7 +280,7 @@ isolate_migratepages_range(struct zone *zone, struct compact_control *cc,
isolate_mode_t mode = 0;
struct lruvec *lruvec;
unsigned long flags;
- bool locked;
+ bool locked = false;
/*
* Ensure that there are not too many pages isolated from the LRU
@@ -295,23 +300,17 @@ isolate_migratepages_range(struct zone *zone, struct compact_control *cc,
/* Time to isolate some pages for migration */
cond_resched();
- spin_lock_irqsave(&zone->lru_lock, flags);
- locked = true;
for (; low_pfn < end_pfn; low_pfn++) {
struct page *page;
/* give a chance to irqs before checking need_resched() */
- if (!((low_pfn+1) % SWAP_CLUSTER_MAX)) {
- spin_unlock_irqrestore(&zone->lru_lock, flags);
- locked = false;
+ if (locked && !((low_pfn+1) % SWAP_CLUSTER_MAX)) {
+ if (should_release_lock(&zone->lru_lock)) {
+ spin_unlock_irqrestore(&zone->lru_lock, flags);
+ locked = false;
+ }
}
- /* Check if it is ok to still hold the lock */
- locked = compact_checklock_irqsave(&zone->lru_lock, &flags,
- locked, cc);
- if (!locked)
- break;
-
/*
* migrate_pfn does not necessarily start aligned to a
* pageblock. Ensure that pfn_valid is called when moving
@@ -351,21 +350,38 @@ isolate_migratepages_range(struct zone *zone, struct compact_control *cc,
pageblock_nr = low_pfn >> pageblock_order;
if (!cc->sync && last_pageblock_nr != pageblock_nr &&
!migrate_async_suitable(get_pageblock_migratetype(page))) {
- low_pfn += pageblock_nr_pages;
- low_pfn = ALIGN(low_pfn, pageblock_nr_pages) - 1;
- last_pageblock_nr = pageblock_nr;
- continue;
+ goto next_pageblock;
}
+ /* Check may be lockless but that's ok as we recheck later */
if (!PageLRU(page))
continue;
/*
- * PageLRU is set, and lru_lock excludes isolation,
- * splitting and collapsing (collapsing has already
- * happened if PageLRU is set).
+ * PageLRU is set. lru_lock normally excludes isolation
+ * splitting and collapsing (collapsing has already happened
+ * if PageLRU is set) but the lock is not necessarily taken
+ * here and it is wasteful to take it just to check transhuge.
+ * Check transhuge without lock and skip if it's either a
+ * transhuge or hugetlbfs page.
*/
if (PageTransHuge(page)) {
+ if (!locked)
+ goto next_pageblock;
+ low_pfn += (1 << compound_order(page)) - 1;
+ continue;
+ }
+
+ /* Check if it is ok to still hold the lock */
+ locked = compact_checklock_irqsave(&zone->lru_lock, &flags,
+ locked, cc);
+ if (!locked)
+ break;
+
+ /* Recheck PageLRU and PageTransHuge under lock */
+ if (!PageLRU(page))
+ continue;
+ if (PageTransHuge(page)) {
low_pfn += (1 << compound_order(page)) - 1;
continue;
}
@@ -392,6 +408,13 @@ isolate_migratepages_range(struct zone *zone, struct compact_control *cc,
++low_pfn;
break;
}
+
+ continue;
+
+next_pageblock:
+ low_pfn += pageblock_nr_pages;
+ low_pfn = ALIGN(low_pfn, pageblock_nr_pages) - 1;
+ last_pageblock_nr = pageblock_nr;
}
acct_isolated(zone, locked, cc);
--
1.7.9.2
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH 2/6] mm: compaction: Acquire the zone->lru_lock as late as possible
2012-09-20 14:04 ` [PATCH 2/6] mm: compaction: Acquire the zone->lru_lock as late as possible Mel Gorman
@ 2012-09-20 18:54 ` Rik van Riel
0 siblings, 0 replies; 20+ messages in thread
From: Rik van Riel @ 2012-09-20 18:54 UTC (permalink / raw)
To: Mel Gorman
Cc: Richard Davies, Shaohua Li, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML
On 09/20/2012 10:04 AM, Mel Gorman wrote:
> Compactions migrate scanner acquires the zone->lru_lock when scanning a range
> of pages looking for LRU pages to acquire. It does this even if there are
> no LRU pages in the range. If multiple processes are compacting then this
> can cause severe locking contention. To make matters worse commit b2eef8c0
> (mm: compaction: minimise the time IRQs are disabled while isolating pages
> for migration) releases the lru_lock every SWAP_CLUSTER_MAX pages that are
> scanned.
>
> This patch makes two changes to how the migrate scanner acquires the LRU
> lock. First, it only releases the LRU lock every SWAP_CLUSTER_MAX pages if
> the lock is contended. This reduces the number of times it unnecessarily
> disables and re-enables IRQs. The second is that it defers acquiring the
> LRU lock for as long as possible. If there are no LRU pages or the only
> LRU pages are transhuge then the LRU lock will not be acquired at all
> which reduces contention on zone->lru_lock.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH 3/6] mm: compaction: Acquire the zone->lock as late as possible
2012-09-20 14:04 [PATCH 0/6] Reduce compaction scanning and lock contention Mel Gorman
2012-09-20 14:04 ` [PATCH 1/6] mm: compaction: Abort compaction loop if lock is contended or run too long Mel Gorman
2012-09-20 14:04 ` [PATCH 2/6] mm: compaction: Acquire the zone->lru_lock as late as possible Mel Gorman
@ 2012-09-20 14:04 ` Mel Gorman
2012-09-20 18:54 ` Rik van Riel
2012-09-20 14:04 ` [PATCH 4/6] Revert "mm: have order > 0 compaction start off where it left" Mel Gorman
` (3 subsequent siblings)
6 siblings, 1 reply; 20+ messages in thread
From: Mel Gorman @ 2012-09-20 14:04 UTC (permalink / raw)
To: Richard Davies, Shaohua Li
Cc: Rik van Riel, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML, Mel Gorman
Compactions free scanner acquires the zone->lock when checking for PageBuddy
pages and isolating them. It does this even if there are no PageBuddy pages
in the range.
This patch defers acquiring the zone lock for as long as possible. In the
event there are no free pages in the pageblock then the lock will not be
acquired at all which reduces contention on zone->lock.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/compaction.c | 141 +++++++++++++++++++++++++++++--------------------------
1 file changed, 75 insertions(+), 66 deletions(-)
diff --git a/mm/compaction.c b/mm/compaction.c
index 6450c3e..70c7cbd 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -89,10 +89,26 @@ static bool compact_checklock_irqsave(spinlock_t *lock, unsigned long *flags,
return true;
}
-static inline bool compact_trylock_irqsave(spinlock_t *lock,
- unsigned long *flags, struct compact_control *cc)
+/* Returns true if the page is within a block suitable for migration to */
+static bool suitable_migration_target(struct page *page)
{
- return compact_checklock_irqsave(lock, flags, false, cc);
+
+ int migratetype = get_pageblock_migratetype(page);
+
+ /* Don't interfere with memory hot-remove or the min_free_kbytes blocks */
+ if (migratetype == MIGRATE_ISOLATE || migratetype == MIGRATE_RESERVE)
+ return false;
+
+ /* If the page is a large free page, then allow migration */
+ if (PageBuddy(page) && page_order(page) >= pageblock_order)
+ return true;
+
+ /* If the block is MIGRATE_MOVABLE or MIGRATE_CMA, allow migration */
+ if (migrate_async_suitable(migratetype))
+ return true;
+
+ /* Otherwise skip the block */
+ return false;
}
/*
@@ -101,13 +117,16 @@ static inline bool compact_trylock_irqsave(spinlock_t *lock,
* pages inside of the pageblock (even though it may still end up isolating
* some pages).
*/
-static unsigned long isolate_freepages_block(unsigned long blockpfn,
+static unsigned long isolate_freepages_block(struct compact_control *cc,
+ unsigned long blockpfn,
unsigned long end_pfn,
struct list_head *freelist,
bool strict)
{
int nr_scanned = 0, total_isolated = 0;
struct page *cursor;
+ unsigned long flags;
+ bool locked = false;
cursor = pfn_to_page(blockpfn);
@@ -116,23 +135,38 @@ static unsigned long isolate_freepages_block(unsigned long blockpfn,
int isolated, i;
struct page *page = cursor;
- if (!pfn_valid_within(blockpfn)) {
- if (strict)
- return 0;
- continue;
- }
+ if (!pfn_valid_within(blockpfn))
+ goto strict_check;
nr_scanned++;
- if (!PageBuddy(page)) {
- if (strict)
- return 0;
- continue;
- }
+ if (!PageBuddy(page))
+ goto strict_check;
+
+ /*
+ * The zone lock must be held to isolate freepages. This
+ * unfortunately this is a very coarse lock and can be
+ * heavily contended if there are parallel allocations
+ * or parallel compactions. For async compaction do not
+ * spin on the lock and we acquire the lock as late as
+ * possible.
+ */
+ locked = compact_checklock_irqsave(&cc->zone->lock, &flags,
+ locked, cc);
+ if (!locked)
+ break;
+
+ /* Recheck this is a suitable migration target under lock */
+ if (!strict && !suitable_migration_target(page))
+ break;
+
+ /* Recheck this is a buddy page under lock */
+ if (!PageBuddy(page))
+ goto strict_check;
/* Found a free page, break it into order-0 pages */
isolated = split_free_page(page);
if (!isolated && strict)
- return 0;
+ goto strict_check;
total_isolated += isolated;
for (i = 0; i < isolated; i++) {
list_add(&page->lru, freelist);
@@ -144,9 +178,23 @@ static unsigned long isolate_freepages_block(unsigned long blockpfn,
blockpfn += isolated - 1;
cursor += isolated - 1;
}
+
+ continue;
+
+strict_check:
+ /* Abort isolation if the caller requested strict isolation */
+ if (strict) {
+ total_isolated = 0;
+ goto out;
+ }
}
trace_mm_compaction_isolate_freepages(nr_scanned, total_isolated);
+
+out:
+ if (locked)
+ spin_unlock_irqrestore(&cc->zone->lock, flags);
+
return total_isolated;
}
@@ -166,13 +214,18 @@ static unsigned long isolate_freepages_block(unsigned long blockpfn,
unsigned long
isolate_freepages_range(unsigned long start_pfn, unsigned long end_pfn)
{
- unsigned long isolated, pfn, block_end_pfn, flags;
+ unsigned long isolated, pfn, block_end_pfn;
struct zone *zone = NULL;
LIST_HEAD(freelist);
+ struct compact_control cc;
if (pfn_valid(start_pfn))
zone = page_zone(pfn_to_page(start_pfn));
+ /* cc needed for isolate_freepages_block to acquire zone->lock */
+ cc.zone = zone;
+ cc.sync = true;
+
for (pfn = start_pfn; pfn < end_pfn; pfn += isolated) {
if (!pfn_valid(pfn) || zone != page_zone(pfn_to_page(pfn)))
break;
@@ -184,10 +237,8 @@ isolate_freepages_range(unsigned long start_pfn, unsigned long end_pfn)
block_end_pfn = ALIGN(pfn + 1, pageblock_nr_pages);
block_end_pfn = min(block_end_pfn, end_pfn);
- spin_lock_irqsave(&zone->lock, flags);
- isolated = isolate_freepages_block(pfn, block_end_pfn,
+ isolated = isolate_freepages_block(&cc, pfn, block_end_pfn,
&freelist, true);
- spin_unlock_irqrestore(&zone->lock, flags);
/*
* In strict mode, isolate_freepages_block() returns 0 if
@@ -429,29 +480,6 @@ next_pageblock:
#endif /* CONFIG_COMPACTION || CONFIG_CMA */
#ifdef CONFIG_COMPACTION
-
-/* Returns true if the page is within a block suitable for migration to */
-static bool suitable_migration_target(struct page *page)
-{
-
- int migratetype = get_pageblock_migratetype(page);
-
- /* Don't interfere with memory hot-remove or the min_free_kbytes blocks */
- if (migratetype == MIGRATE_ISOLATE || migratetype == MIGRATE_RESERVE)
- return false;
-
- /* If the page is a large free page, then allow migration */
- if (PageBuddy(page) && page_order(page) >= pageblock_order)
- return true;
-
- /* If the block is MIGRATE_MOVABLE or MIGRATE_CMA, allow migration */
- if (migrate_async_suitable(migratetype))
- return true;
-
- /* Otherwise skip the block */
- return false;
-}
-
/*
* Returns the start pfn of the last page block in a zone. This is the starting
* point for full compaction of a zone. Compaction searches for free pages from
@@ -475,7 +503,6 @@ static void isolate_freepages(struct zone *zone,
{
struct page *page;
unsigned long high_pfn, low_pfn, pfn, zone_end_pfn, end_pfn;
- unsigned long flags;
int nr_freepages = cc->nr_freepages;
struct list_head *freelist = &cc->freepages;
@@ -523,30 +550,12 @@ static void isolate_freepages(struct zone *zone,
if (!suitable_migration_target(page))
continue;
- /*
- * Found a block suitable for isolating free pages from. Now
- * we disabled interrupts, double check things are ok and
- * isolate the pages. This is to minimise the time IRQs
- * are disabled
- */
+ /* Found a block suitable for isolating free pages from */
isolated = 0;
-
- /*
- * The zone lock must be held to isolate freepages. This
- * unfortunately this is a very coarse lock and can be
- * heavily contended if there are parallel allocations
- * or parallel compactions. For async compaction do not
- * spin on the lock
- */
- if (!compact_trylock_irqsave(&zone->lock, &flags, cc))
- break;
- if (suitable_migration_target(page)) {
- end_pfn = min(pfn + pageblock_nr_pages, zone_end_pfn);
- isolated = isolate_freepages_block(pfn, end_pfn,
- freelist, false);
- nr_freepages += isolated;
- }
- spin_unlock_irqrestore(&zone->lock, flags);
+ end_pfn = min(pfn + pageblock_nr_pages, zone_end_pfn);
+ isolated = isolate_freepages_block(cc, pfn, end_pfn,
+ freelist, false);
+ nr_freepages += isolated;
/*
* Record the highest PFN we isolated pages from. When next
--
1.7.9.2
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH 3/6] mm: compaction: Acquire the zone->lock as late as possible
2012-09-20 14:04 ` [PATCH 3/6] mm: compaction: Acquire the zone->lock " Mel Gorman
@ 2012-09-20 18:54 ` Rik van Riel
0 siblings, 0 replies; 20+ messages in thread
From: Rik van Riel @ 2012-09-20 18:54 UTC (permalink / raw)
To: Mel Gorman
Cc: Richard Davies, Shaohua Li, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML
On 09/20/2012 10:04 AM, Mel Gorman wrote:
> Compactions free scanner acquires the zone->lock when checking for PageBuddy
> pages and isolating them. It does this even if there are no PageBuddy pages
> in the range.
>
> This patch defers acquiring the zone lock for as long as possible. In the
> event there are no free pages in the pageblock then the lock will not be
> acquired at all which reduces contention on zone->lock.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH 4/6] Revert "mm: have order > 0 compaction start off where it left"
2012-09-20 14:04 [PATCH 0/6] Reduce compaction scanning and lock contention Mel Gorman
` (2 preceding siblings ...)
2012-09-20 14:04 ` [PATCH 3/6] mm: compaction: Acquire the zone->lock " Mel Gorman
@ 2012-09-20 14:04 ` Mel Gorman
2012-09-20 18:54 ` Rik van Riel
2012-09-20 14:04 ` [PATCH 5/6] mm: compaction: Cache if a pageblock was scanned and no pages were isolated Mel Gorman
` (2 subsequent siblings)
6 siblings, 1 reply; 20+ messages in thread
From: Mel Gorman @ 2012-09-20 14:04 UTC (permalink / raw)
To: Richard Davies, Shaohua Li
Cc: Rik van Riel, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML, Mel Gorman
This reverts commit 7db8889a (mm: have order > 0 compaction start off
where it left) and commit de74f1cc (mm: have order > 0 compaction start
near a pageblock with free pages). These patches were a good idea and
tests confirmed that they massively reduced the amount of scanning but
the implementation is complex and tricky to understand. A later patch
will cache what pageblocks should be skipped and reimplements the
concept of compact_cached_free_pfn on top for both migration and
free scanners.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
include/linux/mmzone.h | 4 ---
mm/compaction.c | 65 ++++--------------------------------------------
mm/internal.h | 6 -----
mm/page_alloc.c | 5 ----
4 files changed, 5 insertions(+), 75 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 2daa54f..603d0b5 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -368,10 +368,6 @@ struct zone {
*/
spinlock_t lock;
int all_unreclaimable; /* All pages pinned */
-#if defined CONFIG_COMPACTION || defined CONFIG_CMA
- /* pfn where the last incremental compaction isolated free pages */
- unsigned long compact_cached_free_pfn;
-#endif
#ifdef CONFIG_MEMORY_HOTPLUG
/* see spanned/present_pages for more description */
seqlock_t span_seqlock;
diff --git a/mm/compaction.c b/mm/compaction.c
index 70c7cbd..6058822 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -481,20 +481,6 @@ next_pageblock:
#endif /* CONFIG_COMPACTION || CONFIG_CMA */
#ifdef CONFIG_COMPACTION
/*
- * Returns the start pfn of the last page block in a zone. This is the starting
- * point for full compaction of a zone. Compaction searches for free pages from
- * the end of each zone, while isolate_freepages_block scans forward inside each
- * page block.
- */
-static unsigned long start_free_pfn(struct zone *zone)
-{
- unsigned long free_pfn;
- free_pfn = zone->zone_start_pfn + zone->spanned_pages;
- free_pfn &= ~(pageblock_nr_pages-1);
- return free_pfn;
-}
-
-/*
* Based on information in the current compact_control, find blocks
* suitable for isolating free pages from and then isolate them.
*/
@@ -562,19 +548,8 @@ static void isolate_freepages(struct zone *zone,
* looking for free pages, the search will restart here as
* page migration may have returned some pages to the allocator
*/
- if (isolated) {
+ if (isolated)
high_pfn = max(high_pfn, pfn);
-
- /*
- * If the free scanner has wrapped, update
- * compact_cached_free_pfn to point to the highest
- * pageblock with free pages. This reduces excessive
- * scanning of full pageblocks near the end of the
- * zone
- */
- if (cc->order > 0 && cc->wrapped)
- zone->compact_cached_free_pfn = high_pfn;
- }
}
/* split_free_page does not map the pages */
@@ -582,11 +557,6 @@ static void isolate_freepages(struct zone *zone,
cc->free_pfn = high_pfn;
cc->nr_freepages = nr_freepages;
-
- /* If compact_cached_free_pfn is reset then set it now */
- if (cc->order > 0 && !cc->wrapped &&
- zone->compact_cached_free_pfn == start_free_pfn(zone))
- zone->compact_cached_free_pfn = high_pfn;
}
/*
@@ -682,26 +652,8 @@ static int compact_finished(struct zone *zone,
if (fatal_signal_pending(current))
return COMPACT_PARTIAL;
- /*
- * A full (order == -1) compaction run starts at the beginning and
- * end of a zone; it completes when the migrate and free scanner meet.
- * A partial (order > 0) compaction can start with the free scanner
- * at a random point in the zone, and may have to restart.
- */
- if (cc->free_pfn <= cc->migrate_pfn) {
- if (cc->order > 0 && !cc->wrapped) {
- /* We started partway through; restart at the end. */
- unsigned long free_pfn = start_free_pfn(zone);
- zone->compact_cached_free_pfn = free_pfn;
- cc->free_pfn = free_pfn;
- cc->wrapped = 1;
- return COMPACT_CONTINUE;
- }
- return COMPACT_COMPLETE;
- }
-
- /* We wrapped around and ended up where we started. */
- if (cc->wrapped && cc->free_pfn <= cc->start_free_pfn)
+ /* Compaction run completes if the migrate and free scanner meet */
+ if (cc->free_pfn <= cc->migrate_pfn)
return COMPACT_COMPLETE;
/*
@@ -799,15 +751,8 @@ static int compact_zone(struct zone *zone, struct compact_control *cc)
/* Setup to move all movable pages to the end of the zone */
cc->migrate_pfn = zone->zone_start_pfn;
-
- if (cc->order > 0) {
- /* Incremental compaction. Start where the last one stopped. */
- cc->free_pfn = zone->compact_cached_free_pfn;
- cc->start_free_pfn = cc->free_pfn;
- } else {
- /* Order == -1 starts at the end of the zone. */
- cc->free_pfn = start_free_pfn(zone);
- }
+ cc->free_pfn = cc->migrate_pfn + zone->spanned_pages;
+ cc->free_pfn &= ~(pageblock_nr_pages-1);
migrate_prep_local();
diff --git a/mm/internal.h b/mm/internal.h
index 4bd7c0e..04ab01a 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -118,14 +118,8 @@ struct compact_control {
unsigned long nr_freepages; /* Number of isolated free pages */
unsigned long nr_migratepages; /* Number of pages to migrate */
unsigned long free_pfn; /* isolate_freepages search base */
- unsigned long start_free_pfn; /* where we started the search */
unsigned long migrate_pfn; /* isolate_migratepages search base */
bool sync; /* Synchronous migration */
- bool wrapped; /* Order > 0 compactions are
- incremental, once free_pfn
- and migrate_pfn meet, we restart
- from the top of the zone;
- remember we wrapped around. */
int order; /* order a direct compactor needs */
int migratetype; /* MOVABLE, RECLAIMABLE etc */
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index c66fb87..46b2db3 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4438,11 +4438,6 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat,
zone->spanned_pages = size;
zone->present_pages = realsize;
-#if defined CONFIG_COMPACTION || defined CONFIG_CMA
- zone->compact_cached_free_pfn = zone->zone_start_pfn +
- zone->spanned_pages;
- zone->compact_cached_free_pfn &= ~(pageblock_nr_pages-1);
-#endif
#ifdef CONFIG_NUMA
zone->node = nid;
zone->min_unmapped_pages = (realsize*sysctl_min_unmapped_ratio)
--
1.7.9.2
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH 4/6] Revert "mm: have order > 0 compaction start off where it left"
2012-09-20 14:04 ` [PATCH 4/6] Revert "mm: have order > 0 compaction start off where it left" Mel Gorman
@ 2012-09-20 18:54 ` Rik van Riel
0 siblings, 0 replies; 20+ messages in thread
From: Rik van Riel @ 2012-09-20 18:54 UTC (permalink / raw)
To: Mel Gorman
Cc: Richard Davies, Shaohua Li, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML
On 09/20/2012 10:04 AM, Mel Gorman wrote:
> This reverts commit 7db8889a (mm: have order > 0 compaction start off
> where it left) and commit de74f1cc (mm: have order > 0 compaction start
> near a pageblock with free pages). These patches were a good idea and
> tests confirmed that they massively reduced the amount of scanning but
> the implementation is complex and tricky to understand. A later patch
> will cache what pageblocks should be skipped and reimplements the
> concept of compact_cached_free_pfn on top for both migration and
> free scanners.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
Sure, it makes your next patches easier...
Acked-by: Rik van Riel <riel@redhat.com>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH 5/6] mm: compaction: Cache if a pageblock was scanned and no pages were isolated
2012-09-20 14:04 [PATCH 0/6] Reduce compaction scanning and lock contention Mel Gorman
` (3 preceding siblings ...)
2012-09-20 14:04 ` [PATCH 4/6] Revert "mm: have order > 0 compaction start off where it left" Mel Gorman
@ 2012-09-20 14:04 ` Mel Gorman
2012-09-20 18:55 ` Rik van Riel
2012-09-20 14:04 ` [PATCH 6/6] mm: compaction: Restart compaction from near where it left off Mel Gorman
2012-09-21 9:13 ` [PATCH 0/6] Reduce compaction scanning and lock contention Richard Davies
6 siblings, 1 reply; 20+ messages in thread
From: Mel Gorman @ 2012-09-20 14:04 UTC (permalink / raw)
To: Richard Davies, Shaohua Li
Cc: Rik van Riel, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML, Mel Gorman
When compaction was implemented it was known that scanning could potentially
be excessive. The ideal was that a counter be maintained for each pageblock
but maintaining this information would incur a severe penalty due to a
shared writable cache line. It has reached the point where the scanning
costs are an serious problem, particularly on long-lived systems where a
large process starts and allocates a large number of THPs at the same time.
Instead of using a shared counter, this patch adds another bit to the
pageblock flags called PG_migrate_skip. If a pageblock is scanned by
either migrate or free scanner and 0 pages were isolated, the pageblock
is marked to be skipped in the future. When scanning, this bit is checked
before any scanning takes place and the block skipped if set.
The main difficulty with a patch like this is "when to ignore the cached
information?" If it's ignored too often, the scanning rates will still
be excessive. If the information is too stale then allocations will fail
that might have otherwise succeeded. In this patch
o CMA always ignores the information
o If the migrate and free scanner meet then the cached information will
be discarded if it's at least 5 seconds since the last time the cache
was discarded
o If there are a large number of allocation failures, discard the cache.
The time-based heuristic is very clumsy but there are few choices for a
better event. Depending solely on multiple allocation failures still allows
excessive scanning when THP allocations are failing in quick succession
due to memory pressure. Waiting until memory pressure is relieved would
cause compaction to continually fail instead of using reclaim/compaction
to try allocate the page. The time-based mechanism is clumsy but a better
option is not obvious.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
include/linux/mmzone.h | 3 ++
include/linux/pageblock-flags.h | 19 +++++++-
mm/compaction.c | 93 +++++++++++++++++++++++++++++++++++++--
mm/internal.h | 1 +
mm/page_alloc.c | 1 +
5 files changed, 111 insertions(+), 6 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 603d0b5..a456361 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -368,6 +368,9 @@ struct zone {
*/
spinlock_t lock;
int all_unreclaimable; /* All pages pinned */
+#if defined CONFIG_COMPACTION || defined CONFIG_CMA
+ unsigned long compact_blockskip_expire;
+#endif
#ifdef CONFIG_MEMORY_HOTPLUG
/* see spanned/present_pages for more description */
seqlock_t span_seqlock;
diff --git a/include/linux/pageblock-flags.h b/include/linux/pageblock-flags.h
index 19ef95d..eed27f4 100644
--- a/include/linux/pageblock-flags.h
+++ b/include/linux/pageblock-flags.h
@@ -30,6 +30,9 @@ enum pageblock_bits {
PB_migrate,
PB_migrate_end = PB_migrate + 3 - 1,
/* 3 bits required for migrate types */
+#ifdef CONFIG_COMPACTION
+ PB_migrate_skip,/* If set the block is skipped by compaction */
+#endif /* CONFIG_COMPACTION */
NR_PAGEBLOCK_BITS
};
@@ -65,10 +68,22 @@ unsigned long get_pageblock_flags_group(struct page *page,
void set_pageblock_flags_group(struct page *page, unsigned long flags,
int start_bitidx, int end_bitidx);
+#ifdef CONFIG_COMPACTION
+#define get_pageblock_skip(page) \
+ get_pageblock_flags_group(page, PB_migrate_skip, \
+ PB_migrate_skip + 1)
+#define clear_pageblock_skip(page) \
+ set_pageblock_flags_group(page, 0, PB_migrate_skip, \
+ PB_migrate_skip + 1)
+#define set_pageblock_skip(page) \
+ set_pageblock_flags_group(page, 1, PB_migrate_skip, \
+ PB_migrate_skip + 1)
+#endif /* CONFIG_COMPACTION */
+
#define get_pageblock_flags(page) \
- get_pageblock_flags_group(page, 0, NR_PAGEBLOCK_BITS-1)
+ get_pageblock_flags_group(page, 0, PB_migrate_end)
#define set_pageblock_flags(page, flags) \
set_pageblock_flags_group(page, flags, \
- 0, NR_PAGEBLOCK_BITS-1)
+ 0, PB_migrate_end)
#endif /* PAGEBLOCK_FLAGS_H */
diff --git a/mm/compaction.c b/mm/compaction.c
index 6058822..fae0011 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -50,6 +50,64 @@ static inline bool migrate_async_suitable(int migratetype)
return is_migrate_cma(migratetype) || migratetype == MIGRATE_MOVABLE;
}
+/* Returns true if the pageblock should be scanned for pages to isolate. */
+static inline bool isolation_suitable(struct compact_control *cc,
+ struct page *page)
+{
+ if (cc->ignore_skip_hint)
+ return true;
+
+ return !get_pageblock_skip(page);
+}
+
+/*
+ * This function is called to clear all cached information on pageblocks that
+ * should be skipped for page isolation when the migrate and free page scanner
+ * meet.
+ */
+static void reset_isolation_suitable(struct zone *zone)
+{
+ unsigned long start_pfn = zone->zone_start_pfn;
+ unsigned long end_pfn = zone->zone_start_pfn + zone->spanned_pages;
+ unsigned long pfn;
+
+ /*
+ * Do not reset more than once every five seconds. If allocations are
+ * failing sufficiently quickly to allow this to happen then continually
+ * scanning for compaction is not going to help. The choice of five
+ * seconds is arbitrary but will mitigate excessive scanning.
+ */
+ if (time_before(jiffies, zone->compact_blockskip_expire))
+ return;
+ zone->compact_blockskip_expire = jiffies + (HZ * 5);
+
+ /* Walk the zone and mark every pageblock as suitable for isolation */
+ for (pfn = start_pfn; pfn < end_pfn; pfn += pageblock_nr_pages) {
+ struct page *page;
+ if (!pfn_valid(pfn))
+ continue;
+
+ page = pfn_to_page(pfn);
+ if (zone != page_zone(page))
+ continue;
+
+ clear_pageblock_skip(page);
+ }
+}
+
+/*
+ * If no pages were isolated then mark this pageblock to be skipped in the
+ * future. The information is later cleared by reset_isolation_suitable().
+ */
+static void update_pageblock_skip(struct page *page, unsigned long nr_isolated)
+{
+ if (!page)
+ return;
+
+ if (!nr_isolated)
+ set_pageblock_skip(page);
+}
+
static inline bool should_release_lock(spinlock_t *lock)
{
return need_resched() || spin_is_contended(lock);
@@ -124,7 +182,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc,
bool strict)
{
int nr_scanned = 0, total_isolated = 0;
- struct page *cursor;
+ struct page *cursor, *valid_page = NULL;
unsigned long flags;
bool locked = false;
@@ -138,6 +196,8 @@ static unsigned long isolate_freepages_block(struct compact_control *cc,
if (!pfn_valid_within(blockpfn))
goto strict_check;
nr_scanned++;
+ if (!valid_page)
+ valid_page = page;
if (!PageBuddy(page))
goto strict_check;
@@ -195,6 +255,10 @@ out:
if (locked)
spin_unlock_irqrestore(&cc->zone->lock, flags);
+ /* Update the pageblock-skip if the whole pageblock was scanned */
+ if (blockpfn == end_pfn)
+ update_pageblock_skip(valid_page, total_isolated);
+
return total_isolated;
}
@@ -332,6 +396,7 @@ isolate_migratepages_range(struct zone *zone, struct compact_control *cc,
struct lruvec *lruvec;
unsigned long flags;
bool locked = false;
+ struct page *page = NULL, *valid_page = NULL;
/*
* Ensure that there are not too many pages isolated from the LRU
@@ -352,7 +417,6 @@ isolate_migratepages_range(struct zone *zone, struct compact_control *cc,
/* Time to isolate some pages for migration */
cond_resched();
for (; low_pfn < end_pfn; low_pfn++) {
- struct page *page;
/* give a chance to irqs before checking need_resched() */
if (locked && !((low_pfn+1) % SWAP_CLUSTER_MAX)) {
@@ -389,6 +453,14 @@ isolate_migratepages_range(struct zone *zone, struct compact_control *cc,
if (page_zone(page) != zone)
continue;
+ if (!valid_page)
+ valid_page = page;
+
+ /* If isolation recently failed, do not retry */
+ pageblock_nr = low_pfn >> pageblock_order;
+ if (!isolation_suitable(cc, page))
+ goto next_pageblock;
+
/* Skip if free */
if (PageBuddy(page))
continue;
@@ -398,7 +470,6 @@ isolate_migratepages_range(struct zone *zone, struct compact_control *cc,
* migration is optimistic to see if the minimum amount of work
* satisfies the allocation
*/
- pageblock_nr = low_pfn >> pageblock_order;
if (!cc->sync && last_pageblock_nr != pageblock_nr &&
!migrate_async_suitable(get_pageblock_migratetype(page))) {
goto next_pageblock;
@@ -473,6 +544,10 @@ next_pageblock:
if (locked)
spin_unlock_irqrestore(&zone->lru_lock, flags);
+ /* Update the pageblock-skip if the whole pageblock was scanned */
+ if (low_pfn == end_pfn)
+ update_pageblock_skip(valid_page, nr_isolated);
+
trace_mm_compaction_isolate_migratepages(nr_scanned, nr_isolated);
return low_pfn;
@@ -536,6 +611,10 @@ static void isolate_freepages(struct zone *zone,
if (!suitable_migration_target(page))
continue;
+ /* If isolation recently failed, do not retry */
+ if (!isolation_suitable(cc, page))
+ continue;
+
/* Found a block suitable for isolating free pages from */
isolated = 0;
end_pfn = min(pfn + pageblock_nr_pages, zone_end_pfn);
@@ -653,8 +732,10 @@ static int compact_finished(struct zone *zone,
return COMPACT_PARTIAL;
/* Compaction run completes if the migrate and free scanner meet */
- if (cc->free_pfn <= cc->migrate_pfn)
+ if (cc->free_pfn <= cc->migrate_pfn) {
+ reset_isolation_suitable(cc->zone);
return COMPACT_COMPLETE;
+ }
/*
* order == -1 is expected when compacting via
@@ -754,6 +835,10 @@ static int compact_zone(struct zone *zone, struct compact_control *cc)
cc->free_pfn = cc->migrate_pfn + zone->spanned_pages;
cc->free_pfn &= ~(pageblock_nr_pages-1);
+ /* Clear pageblock skip if there are numerous alloc failures */
+ if (zone->compact_defer_shift == COMPACT_MAX_DEFER_SHIFT)
+ reset_isolation_suitable(zone);
+
migrate_prep_local();
while ((ret = compact_finished(zone, cc)) == COMPACT_CONTINUE) {
diff --git a/mm/internal.h b/mm/internal.h
index 04ab01a..aeae7eb 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -120,6 +120,7 @@ struct compact_control {
unsigned long free_pfn; /* isolate_freepages search base */
unsigned long migrate_pfn; /* isolate_migratepages search base */
bool sync; /* Synchronous migration */
+ bool ignore_skip_hint; /* Scan blocks even if marked skip */
int order; /* order a direct compactor needs */
int migratetype; /* MOVABLE, RECLAIMABLE etc */
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 46b2db3..75fee02 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5640,6 +5640,7 @@ static int __alloc_contig_migrate_range(unsigned long start, unsigned long end)
.order = -1,
.zone = page_zone(pfn_to_page(start)),
.sync = true,
+ .ignore_skip_hint = true,
};
INIT_LIST_HEAD(&cc.migratepages);
--
1.7.9.2
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH 5/6] mm: compaction: Cache if a pageblock was scanned and no pages were isolated
2012-09-20 14:04 ` [PATCH 5/6] mm: compaction: Cache if a pageblock was scanned and no pages were isolated Mel Gorman
@ 2012-09-20 18:55 ` Rik van Riel
0 siblings, 0 replies; 20+ messages in thread
From: Rik van Riel @ 2012-09-20 18:55 UTC (permalink / raw)
To: Mel Gorman
Cc: Richard Davies, Shaohua Li, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML
On 09/20/2012 10:04 AM, Mel Gorman wrote:
> When compaction was implemented it was known that scanning could potentially
> be excessive. The ideal was that a counter be maintained for each pageblock
> but maintaining this information would incur a severe penalty due to a
> shared writable cache line. It has reached the point where the scanning
> costs are an serious problem, particularly on long-lived systems where a
> large process starts and allocates a large number of THPs at the same time.
>
> Instead of using a shared counter, this patch adds another bit to the
> pageblock flags called PG_migrate_skip. If a pageblock is scanned by
> either migrate or free scanner and 0 pages were isolated, the pageblock
> is marked to be skipped in the future. When scanning, this bit is checked
> before any scanning takes place and the block skipped if set.
>
> The main difficulty with a patch like this is "when to ignore the cached
> information?" If it's ignored too often, the scanning rates will still
> be excessive. If the information is too stale then allocations will fail
> that might have otherwise succeeded. In this patch
Big hammer, but I guess it is effective...
Acked-by: Rik van Riel <riel@redhat.com>
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH 6/6] mm: compaction: Restart compaction from near where it left off
2012-09-20 14:04 [PATCH 0/6] Reduce compaction scanning and lock contention Mel Gorman
` (4 preceding siblings ...)
2012-09-20 14:04 ` [PATCH 5/6] mm: compaction: Cache if a pageblock was scanned and no pages were isolated Mel Gorman
@ 2012-09-20 14:04 ` Mel Gorman
2012-09-20 18:57 ` Rik van Riel
2012-09-21 9:13 ` [PATCH 0/6] Reduce compaction scanning and lock contention Richard Davies
6 siblings, 1 reply; 20+ messages in thread
From: Mel Gorman @ 2012-09-20 14:04 UTC (permalink / raw)
To: Richard Davies, Shaohua Li
Cc: Rik van Riel, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML, Mel Gorman
This is almost entirely based on Rik's previous patches and discussions
with him about how this might be implemented.
Order > 0 compaction stops when enough free pages of the correct page
order have been coalesced. When doing subsequent higher order allocations,
it is possible for compaction to be invoked many times.
However, the compaction code always starts out looking for things to compact
at the start of the zone, and for free pages to compact things to at the
end of the zone.
This can cause quadratic behaviour, with isolate_freepages starting at
the end of the zone each time, even though previous invocations of the
compaction code already filled up all free memory on that end of the zone.
This can cause isolate_freepages to take enormous amounts of CPU with
certain workloads on larger memory systems.
This patch caches where the migration and free scanner should start from on
subsequent compaction invocations using the pageblock-skip information. When
compaction starts it begins from the cached restart points and will
update the cached restart points until a page is isolated or a pageblock
is skipped that would have been scanned by synchronous compaction.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
include/linux/mmzone.h | 4 ++++
mm/compaction.c | 54 ++++++++++++++++++++++++++++++++++++++++--------
mm/internal.h | 4 ++++
3 files changed, 53 insertions(+), 9 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index a456361..e7792a3 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -370,6 +370,10 @@ struct zone {
int all_unreclaimable; /* All pages pinned */
#if defined CONFIG_COMPACTION || defined CONFIG_CMA
unsigned long compact_blockskip_expire;
+
+ /* pfns where compaction scanners should start */
+ unsigned long compact_cached_free_pfn;
+ unsigned long compact_cached_migrate_pfn;
#endif
#ifdef CONFIG_MEMORY_HOTPLUG
/* see spanned/present_pages for more description */
diff --git a/mm/compaction.c b/mm/compaction.c
index fae0011..45a17c9 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -79,6 +79,9 @@ static void reset_isolation_suitable(struct zone *zone)
*/
if (time_before(jiffies, zone->compact_blockskip_expire))
return;
+
+ zone->compact_cached_migrate_pfn = start_pfn;
+ zone->compact_cached_free_pfn = end_pfn;
zone->compact_blockskip_expire = jiffies + (HZ * 5);
/* Walk the zone and mark every pageblock as suitable for isolation */
@@ -99,13 +102,29 @@ static void reset_isolation_suitable(struct zone *zone)
* If no pages were isolated then mark this pageblock to be skipped in the
* future. The information is later cleared by reset_isolation_suitable().
*/
-static void update_pageblock_skip(struct page *page, unsigned long nr_isolated)
+static void update_pageblock_skip(struct compact_control *cc,
+ struct page *page, unsigned long nr_isolated,
+ bool migrate_scanner)
{
+ struct zone *zone = cc->zone;
if (!page)
return;
- if (!nr_isolated)
+ if (!nr_isolated) {
+ unsigned long pfn = page_to_pfn(page);
set_pageblock_skip(page);
+
+ /* Update where compaction should restart */
+ if (migrate_scanner) {
+ if (!cc->finished_update_migrate &&
+ pfn > zone->compact_cached_migrate_pfn)
+ zone->compact_cached_migrate_pfn = pfn;
+ } else {
+ if (!cc->finished_update_free &&
+ pfn < zone->compact_cached_free_pfn)
+ zone->compact_cached_free_pfn = pfn;
+ }
+ }
}
static inline bool should_release_lock(spinlock_t *lock)
@@ -257,7 +276,7 @@ out:
/* Update the pageblock-skip if the whole pageblock was scanned */
if (blockpfn == end_pfn)
- update_pageblock_skip(valid_page, total_isolated);
+ update_pageblock_skip(cc, valid_page, total_isolated, false);
return total_isolated;
}
@@ -472,6 +491,7 @@ isolate_migratepages_range(struct zone *zone, struct compact_control *cc,
*/
if (!cc->sync && last_pageblock_nr != pageblock_nr &&
!migrate_async_suitable(get_pageblock_migratetype(page))) {
+ cc->finished_update_migrate = true;
goto next_pageblock;
}
@@ -520,6 +540,7 @@ isolate_migratepages_range(struct zone *zone, struct compact_control *cc,
VM_BUG_ON(PageTransCompound(page));
/* Successfully isolated */
+ cc->finished_update_migrate = true;
del_page_from_lru_list(page, lruvec, page_lru(page));
list_add(&page->lru, migratelist);
cc->nr_migratepages++;
@@ -546,7 +567,7 @@ next_pageblock:
/* Update the pageblock-skip if the whole pageblock was scanned */
if (low_pfn == end_pfn)
- update_pageblock_skip(valid_page, nr_isolated);
+ update_pageblock_skip(cc, valid_page, nr_isolated, true);
trace_mm_compaction_isolate_migratepages(nr_scanned, nr_isolated);
@@ -627,8 +648,10 @@ static void isolate_freepages(struct zone *zone,
* looking for free pages, the search will restart here as
* page migration may have returned some pages to the allocator
*/
- if (isolated)
+ if (isolated) {
+ cc->finished_update_free = true;
high_pfn = max(high_pfn, pfn);
+ }
}
/* split_free_page does not map the pages */
@@ -818,6 +841,8 @@ unsigned long compaction_suitable(struct zone *zone, int order)
static int compact_zone(struct zone *zone, struct compact_control *cc)
{
int ret;
+ unsigned long start_pfn = zone->zone_start_pfn;
+ unsigned long end_pfn = zone->zone_start_pfn + zone->spanned_pages;
ret = compaction_suitable(zone, cc->order);
switch (ret) {
@@ -830,10 +855,21 @@ static int compact_zone(struct zone *zone, struct compact_control *cc)
;
}
- /* Setup to move all movable pages to the end of the zone */
- cc->migrate_pfn = zone->zone_start_pfn;
- cc->free_pfn = cc->migrate_pfn + zone->spanned_pages;
- cc->free_pfn &= ~(pageblock_nr_pages-1);
+ /*
+ * Setup to move all movable pages to the end of the zone. Used cached
+ * information on where the scanners should start but check that it
+ * is initialised by ensuring the values are within zone boundaries.
+ */
+ cc->migrate_pfn = zone->compact_cached_migrate_pfn;
+ cc->free_pfn = zone->compact_cached_free_pfn;
+ if (cc->free_pfn < start_pfn || cc->free_pfn > end_pfn) {
+ cc->free_pfn = end_pfn & ~(pageblock_nr_pages-1);
+ zone->compact_cached_free_pfn = cc->free_pfn;
+ }
+ if (cc->migrate_pfn < start_pfn || cc->migrate_pfn > end_pfn) {
+ cc->migrate_pfn = start_pfn;
+ zone->compact_cached_migrate_pfn = cc->migrate_pfn;
+ }
/* Clear pageblock skip if there are numerous alloc failures */
if (zone->compact_defer_shift == COMPACT_MAX_DEFER_SHIFT)
diff --git a/mm/internal.h b/mm/internal.h
index aeae7eb..d5c1163 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -121,6 +121,10 @@ struct compact_control {
unsigned long migrate_pfn; /* isolate_migratepages search base */
bool sync; /* Synchronous migration */
bool ignore_skip_hint; /* Scan blocks even if marked skip */
+ bool finished_update_free; /* True when the zone cached pfns are
+ * no longer being updated
+ */
+ bool finished_update_migrate;
int order; /* order a direct compactor needs */
int migratetype; /* MOVABLE, RECLAIMABLE etc */
--
1.7.9.2
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH 6/6] mm: compaction: Restart compaction from near where it left off
2012-09-20 14:04 ` [PATCH 6/6] mm: compaction: Restart compaction from near where it left off Mel Gorman
@ 2012-09-20 18:57 ` Rik van Riel
0 siblings, 0 replies; 20+ messages in thread
From: Rik van Riel @ 2012-09-20 18:57 UTC (permalink / raw)
To: Mel Gorman
Cc: Richard Davies, Shaohua Li, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML
On 09/20/2012 10:04 AM, Mel Gorman wrote:
> This is almost entirely based on Rik's previous patches and discussions
> with him about how this might be implemented.
>
> Order > 0 compaction stops when enough free pages of the correct page
> order have been coalesced. When doing subsequent higher order allocations,
> it is possible for compaction to be invoked many times.
>
> However, the compaction code always starts out looking for things to compact
> at the start of the zone, and for free pages to compact things to at the
> end of the zone.
>
> This can cause quadratic behaviour, with isolate_freepages starting at
> the end of the zone each time, even though previous invocations of the
> compaction code already filled up all free memory on that end of the zone.
> This can cause isolate_freepages to take enormous amounts of CPU with
> certain workloads on larger memory systems.
>
> This patch caches where the migration and free scanner should start from on
> subsequent compaction invocations using the pageblock-skip information. When
> compaction starts it begins from the cached restart points and will
> update the cached restart points until a page is isolated or a pageblock
> is skipped that would have been scanned by synchronous compaction.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
Together with patch 5/6, this has the effect of
skipping compaction in a zone if the free and
isolate markers have met, and it has been less
than 5 seconds since the "skip" information was
reset.
Compaction on zones where we cycle through more
slowly can continue, even when this particular
zone is experiencing problems, so I guess this
is desired behaviour...
Acked-by: Rik van Riel <riel@redhat.com>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 0/6] Reduce compaction scanning and lock contention
2012-09-20 14:04 [PATCH 0/6] Reduce compaction scanning and lock contention Mel Gorman
` (5 preceding siblings ...)
2012-09-20 14:04 ` [PATCH 6/6] mm: compaction: Restart compaction from near where it left off Mel Gorman
@ 2012-09-21 9:13 ` Richard Davies
2012-09-21 9:15 ` Richard Davies
` (3 more replies)
6 siblings, 4 replies; 20+ messages in thread
From: Richard Davies @ 2012-09-21 9:13 UTC (permalink / raw)
To: Mel Gorman
Cc: Shaohua Li, Rik van Riel, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML
Hi Mel,
Thank you for this series. I have applied on clean 3.6-rc5 and tested, and
it works well for me - the lock contention is (still) gone and
isolate_freepages_block is much reduced.
Here is a typical test with these patches:
# grep -F '[k]' report | head -8
65.20% qemu-kvm [kernel.kallsyms] [k] clear_page_c
2.18% qemu-kvm [kernel.kallsyms] [k] isolate_freepages_block
1.56% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
1.40% qemu-kvm [kernel.kallsyms] [k] svm_vcpu_run
1.38% swapper [kernel.kallsyms] [k] default_idle
1.35% qemu-kvm [kernel.kallsyms] [k] get_page_from_freelist
0.74% ksmd [kernel.kallsyms] [k] memcmp
0.72% qemu-kvm [kernel.kallsyms] [k] free_pages_prepare
I did manage to get a couple which were slightly worse, but nothing like as
bad as before. Here are the results:
# grep -F '[k]' report | head -8
45.60% qemu-kvm [kernel.kallsyms] [k] clear_page_c
11.26% qemu-kvm [kernel.kallsyms] [k] isolate_freepages_block
3.21% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
2.27% ksmd [kernel.kallsyms] [k] memcmp
2.02% swapper [kernel.kallsyms] [k] default_idle
1.58% qemu-kvm [kernel.kallsyms] [k] svm_vcpu_run
1.30% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock_irqsave
1.09% qemu-kvm [kernel.kallsyms] [k] get_page_from_freelist
# grep -F '[k]' report | head -8
61.29% qemu-kvm [kernel.kallsyms] [k] clear_page_c
4.52% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock_irqsave
2.64% qemu-kvm [kernel.kallsyms] [k] copy_page_c
1.61% swapper [kernel.kallsyms] [k] default_idle
1.57% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
1.18% qemu-kvm [kernel.kallsyms] [k] get_page_from_freelist
1.18% qemu-kvm [kernel.kallsyms] [k] isolate_freepages_block
1.11% qemu-kvm [kernel.kallsyms] [k] svm_vcpu_run
I will follow up with the detailed traces for these three tests.
Thank you!
Richard.
^ permalink raw reply [flat|nested] 20+ messages in thread* Re: [PATCH 0/6] Reduce compaction scanning and lock contention
2012-09-21 9:13 ` [PATCH 0/6] Reduce compaction scanning and lock contention Richard Davies
@ 2012-09-21 9:15 ` Richard Davies
2012-09-21 9:17 ` Richard Davies
` (2 subsequent siblings)
3 siblings, 0 replies; 20+ messages in thread
From: Richard Davies @ 2012-09-21 9:15 UTC (permalink / raw)
To: Mel Gorman
Cc: Shaohua Li, Rik van Riel, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML
Richard Davies wrote:
> Here is a typical test with these patches:
>
> # grep -F '[k]' report | head -8
> 65.20% qemu-kvm [kernel.kallsyms] [k] clear_page_c
> 2.18% qemu-kvm [kernel.kallsyms] [k] isolate_freepages_block
> 1.56% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
> 1.40% qemu-kvm [kernel.kallsyms] [k] svm_vcpu_run
> 1.38% swapper [kernel.kallsyms] [k] default_idle
> 1.35% qemu-kvm [kernel.kallsyms] [k] get_page_from_freelist
> 0.74% ksmd [kernel.kallsyms] [k] memcmp
> 0.72% qemu-kvm [kernel.kallsyms] [k] free_pages_prepare
# ========
# captured on: Fri Sep 21 08:29:36 2012
# os release : 3.6.0-rc5-elastic+
# perf version : 3.5.2
# arch : x86_64
# nrcpus online : 16
# nrcpus avail : 16
# cpudesc : AMD Opteron(tm) Processor 6128
# cpuid : AuthenticAMD,16,9,1
# total memory : 131973276 kB
# cmdline : /home/root/bin/perf record -g -a
# event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, id = { 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64 }
# HEADER_CPU_TOPOLOGY info available, use -I to display
# HEADER_NUMA_TOPOLOGY info available, use -I to display
# ========
#
# Samples: 837K of event 'cycles'
# Event count (approx.): 290328521160
#
# Overhead Command Shared Object Symbol
# ........ ............... .................... ..........................................
#
65.20% qemu-kvm [kernel.kallsyms] [k] clear_page_c
|
--- clear_page_c
|
|--98.02%-- do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--56.65%-- 0x10100000002
| |
| |--43.35%-- 0x10100000006
| --0.00%-- [...]
|
--1.98%-- __alloc_pages_nodemask
|
|--91.16%-- alloc_pages_vma
| handle_pte_fault
| |
| |--99.74%-- handle_mm_fault
| | |
| | |--99.93%-- __get_user_pages
| | | get_user_page_nowait
| | | hva_to_pfn.isra.17
| | | __gfn_to_pfn
| | | gfn_to_pfn_async
| | | try_async_pf
| | | tdp_page_fault
| | | kvm_mmu_page_fault
| | | pf_interception
| | | handle_exit
| | | kvm_arch_vcpu_ioctl_run
| | | kvm_vcpu_ioctl
| | | do_vfs_ioctl
| | | sys_ioctl
| | | system_call_fastpath
| | | ioctl
| | | |
| | | |--86.42%-- 0x10100000006
| | | |
| | | --13.58%-- 0x10100000002
| | --0.07%-- [...]
| --0.26%-- [...]
|
--8.84%-- alloc_pages_current
|
|--99.73%-- pte_alloc_one
| |
| |--97.60%-- do_huge_pmd_anonymous_page
| | handle_mm_fault
| | __get_user_pages
| | get_user_page_nowait
| | hva_to_pfn.isra.17
| | __gfn_to_pfn
| | gfn_to_pfn_async
| | try_async_pf
| | tdp_page_fault
| | kvm_mmu_page_fault
| | pf_interception
| | handle_exit
| | kvm_arch_vcpu_ioctl_run
| | kvm_vcpu_ioctl
| | do_vfs_ioctl
| | sys_ioctl
| | system_call_fastpath
| | ioctl
| | |
| | |--60.84%-- 0x10100000002
| | |
| | --39.16%-- 0x10100000006
| |
| --2.40%-- __pte_alloc
| do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| 0x10100000006
--0.27%-- [...]
2.18% qemu-kvm [kernel.kallsyms] [k] isolate_freepages_block
|
--- isolate_freepages_block
compaction_alloc
migrate_pages
compact_zone
compact_zone_order
try_to_compact_pages
__alloc_pages_direct_compact
__alloc_pages_nodemask
alloc_pages_vma
do_huge_pmd_anonymous_page
handle_mm_fault
__get_user_pages
get_user_page_nowait
hva_to_pfn.isra.17
__gfn_to_pfn
gfn_to_pfn_async
try_async_pf
tdp_page_fault
kvm_mmu_page_fault
pf_interception
handle_exit
kvm_arch_vcpu_ioctl_run
kvm_vcpu_ioctl
do_vfs_ioctl
sys_ioctl
system_call_fastpath
ioctl
|
|--92.03%-- 0x10100000006
|
--7.97%-- 0x10100000002
1.58% qemu-kvm qemu-kvm [.] 0x000000000015b95b
|
|--2.54%-- 0x652b11
|
|--2.19%-- 0x55b9ba
| |
| --100.00%-- 0x0
|
|--2.04%-- 0x56b990
| |
| |--71.82%-- 0x100000008
| |
| |--25.52%-- 0xfed00000
| | |
| | --100.00%-- 0x0
| |
| |--2.11%-- 0xfee00000
| |
| --0.55%-- 0x100000009
|
|--0.95%-- 0x5ac46a
| |
| |--89.44%-- 0x10100000002
| |
| --10.56%-- 0x10100000006
|
|--0.75%-- 0x5ad00b
| |
| |--93.06%-- 0x10100000002
| |
| --6.94%-- 0x10100000006
|
|--0.73%-- 0x56b999
| |
| |--68.98%-- 0x100000008
| |
| |--29.58%-- 0xfed00000
| | |
| | --100.00%-- 0x0
| |
| --1.43%-- 0xfee00000
|
|--0.73%-- 0x664a8f
| 0x6e6f6d
|
|--0.70%-- 0x5acf38
| |
| |--78.42%-- 0x10100000002
| |
| |--20.00%-- 0x10100000006
| |
| --1.58%-- 0x0
|
|--0.65%-- 0x458de4
| |
| |--33.45%-- 0x2f1a310
| | 0x0
| |
| |--27.60%-- 0x3014310
| | 0x0
| |
| |--23.23%-- 0x2b1b310
| | 0x0
| |
| |--5.04%-- 0x0
| |
| |--2.08%-- 0x840f01fa8338578b
| |
| |--1.68%-- 0x2f1f450
| | 0x0
| |
| |--1.27%-- 0x80d504c748000000
| |
| |--1.12%-- 0x78840ff685450040
| |
| |--1.05%-- 0x48ffef5b8ae8c031
| |
| |--0.92%-- 0x3cd5cc05c70000
| |
| |--0.88%-- 0x200bd7e0f05
| |
| |--0.88%-- 0x1dc0be00791856ba
| |
| --0.79%-- 0x485390fff4d921e8
|
|--0.63%-- 0x664a82
| |
| |--95.74%-- 0x6e6f6d
| |
| --4.26%-- 0x2540627568006563
|
|--0.63%-- 0x664a85
| 0x6e6f6d
|
|--0.51%-- 0x594ce4
| |
| |--45.13%-- 0x2b1b310
| | 0x0
| |
| |--24.91%-- 0x3014310
| | 0x0
| |
| |--24.18%-- 0x2f1a310
| | 0x0
| |
| |--1.73%-- 0x0
| |
| |--1.64%-- 0x7000000
| | 0x2a1e3c0
| |
| |--1.43%-- 0x75005a6a2f053348
| |
| --0.97%-- 0x440f4c1045894908
|
|--0.51%-- 0x52fb44
| |
| |--56.61%-- 0x10100000002
| |
| |--34.56%-- 0x0
| | |
| | --100.00%-- 0x2f3e590
| |
| --8.83%-- 0x10100000006
--86.43%-- [...]
1.56% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
|
|--41.47%-- tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--76.14%-- 0x10100000006
| |
| --23.86%-- 0x10100000002
|
|--11.06%-- kvm_mmu_load
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--53.21%-- 0x10100000002
| |
| --46.79%-- 0x10100000006
|
|--7.91%-- follow_page
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--86.50%-- 0x10100000006
| |
| --13.50%-- 0x10100000002
|
|--6.97%-- mmu_free_roots
| |
| |--76.89%-- nonpaging_free
| | kvm_mmu_reset_context
| | kvm_set_cr4
| | emulator_set_cr
| | em_cr_write
| | x86_emulate_insn
| | x86_emulate_instruction
| | emulate_on_interception
| | cr_interception
| | handle_exit
| | kvm_arch_vcpu_ioctl_run
| | kvm_vcpu_ioctl
| | do_vfs_ioctl
| | sys_ioctl
| | system_call_fastpath
| | ioctl
| | |
| | |--50.62%-- 0x10100000006
| | |
| | --49.38%-- 0x10100000002
| |
| --23.11%-- kvm_mmu_unload
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--52.88%-- 0x10100000002
| |
| --47.12%-- 0x10100000006
|
|--6.91%-- kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--82.74%-- 0x10100000006
| |
| --17.26%-- 0x10100000002
|
|--3.62%-- yield_to
| kvm_vcpu_yield_to
| kvm_vcpu_on_spin
| pause_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--54.40%-- 0x10100000006
| |
| --45.60%-- 0x10100000002
|
|--2.15%-- kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--61.48%-- 0x10100000002
| |
| --38.52%-- 0x10100000006
|
|--1.99%-- grab_super_passive
| prune_super
| shrink_slab
| try_to_free_pages
| __alloc_pages_nodemask
| alloc_pages_vma
| do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--86.97%-- 0x10100000006
| |
| --13.03%-- 0x10100000002
|
|--1.95%-- put_super
| drop_super
| prune_super
| shrink_slab
| try_to_free_pages
| __alloc_pages_nodemask
| alloc_pages_vma
| do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--86.46%-- 0x10100000006
| |
| --13.54%-- 0x10100000002
|
|--1.93%-- __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--84.17%-- 0x10100000006
| |
| --15.83%-- 0x10100000002
|
|--1.78%-- handle_pte_fault
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--83.19%-- 0x10100000006
| |
| --16.81%-- 0x10100000002
|
|--1.33%-- free_pcppages_bulk
| |
| |--83.72%-- drain_pages
| | |
| | |--99.41%-- drain_local_pages
| | | generic_smp_call_function_interrupt
| | | smp_call_function_interrupt
| | | call_function_interrupt
| | | |
| | | |--42.28%-- kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | |
| | | | |--83.98%-- 0x10100000006
| | | | |
| | | | --16.02%-- 0x10100000002
| | | |
| | | |--8.49%-- __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | handle_pte_fault
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | |
| | | | |--80.36%-- 0x10100000006
| | | | |
| | | | --19.64%-- 0x10100000002
| | | |
| | | |--6.53%-- __remove_mapping
| | | | shrink_page_list
| | | | shrink_inactive_list
| | | | shrink_lruvec
| | | | try_to_free_pages
| | | | __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--6.20%-- buffer_migrate_page
| | | | move_to_new_page
| | | | migrate_pages
| | | | compact_zone
| | | | compact_zone_order
| | | | try_to_compact_pages
| | | | __alloc_pages_direct_compact
| | | | __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | |
| | | | |--83.97%-- 0x10100000006
| | | | |
| | | | --16.03%-- 0x10100000002
| | | |
| | | |--5.34%-- __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | |
| | | | |--79.01%-- 0x10100000006
| | | | |
| | | | --20.99%-- 0x10100000002
| | | |
| | | |--3.98%-- compact_checklock_irqsave
| | | | isolate_freepages_block
| | | | compaction_alloc
| | | | migrate_pages
| | | | compact_zone
| | | | compact_zone_order
| | | | try_to_compact_pages
| | | | __alloc_pages_direct_compact
| | | | __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--3.48%-- do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | |
| | | | |--83.45%-- 0x10100000006
| | | | |
| | | | --16.55%-- 0x10100000002
| | | |
| | | |--2.91%-- hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | |
| | | | |--79.66%-- 0x10100000006
| | | | |
| | | | --20.34%-- 0x10100000002
| | | |
| | | |--2.85%-- tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--2.45%-- compaction_alloc
| | | | migrate_pages
| | | | compact_zone
| | | | compact_zone_order
| | | | try_to_compact_pages
| | | | __alloc_pages_direct_compact
| | | | __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | |
| | | | |--59.53%-- 0x10100000006
| | | | |
| | | | --40.47%-- 0x10100000002
| | | |
| | | |--2.14%-- kvm_vcpu_yield_to
| | | | kvm_vcpu_on_spin
| | | | pause_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--1.15%-- handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--0.99%-- putback_lru_page
| | | | migrate_pages
| | | | compact_zone
| | | | compact_zone_order
| | | | try_to_compact_pages
| | | | __alloc_pages_direct_compact
| | | | __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--0.98%-- timer_gettime
| | | |
| | | |--0.97%-- native_flush_tlb_others
| | | | flush_tlb_page
| | | | ptep_clear_flush_young
| | | | page_referenced_one
| | | | page_referenced
| | | | shrink_active_list
| | | | shrink_lruvec
| | | | try_to_free_pages
| | | | __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--0.97%-- try_to_free_buffers
| | | | jbd2_journal_try_to_free_buffers
| | | | ext4_releasepage
| | | | try_to_release_page
| | | | shrink_page_list
| | | | shrink_inactive_list
| | | | shrink_lruvec
| | | | try_to_free_pages
| | | | __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--0.86%-- shrink_inactive_list
| | | | shrink_lruvec
| | | | try_to_free_pages
| | | | __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--0.86%-- __mutex_lock_slowpath
| | | | mutex_lock
| | | | page_lock_anon_vma
| | | | page_referenced
| | | | shrink_active_list
| | | | shrink_lruvec
| | | | try_to_free_pages
| | | | __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000002
| | | |
| | | |--0.83%-- kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--0.62%-- __direct_map.isra.103
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--0.61%-- gfn_to_hva
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--0.60%-- mmu_set_spte.isra.100
| | | | __direct_map.isra.103
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--0.59%-- mmu_spte_update
| | | | set_spte
| | | | mmu_set_spte.isra.100
| | | | __direct_map.isra.103
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--0.59%-- kvm_vcpu_on_spin
^ permalink raw reply [flat|nested] 20+ messages in thread* Re: [PATCH 0/6] Reduce compaction scanning and lock contention
2012-09-21 9:13 ` [PATCH 0/6] Reduce compaction scanning and lock contention Richard Davies
2012-09-21 9:15 ` Richard Davies
@ 2012-09-21 9:17 ` Richard Davies
2012-09-21 9:55 ` Mel Gorman
2012-09-21 9:18 ` Richard Davies
2012-09-21 9:35 ` Mel Gorman
3 siblings, 1 reply; 20+ messages in thread
From: Richard Davies @ 2012-09-21 9:17 UTC (permalink / raw)
To: Mel Gorman
Cc: Shaohua Li, Rik van Riel, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML
Richard Davies wrote:
> I did manage to get a couple which were slightly worse, but nothing like as
> bad as before. Here are the results:
>
> # grep -F '[k]' report | head -8
> 45.60% qemu-kvm [kernel.kallsyms] [k] clear_page_c
> 11.26% qemu-kvm [kernel.kallsyms] [k] isolate_freepages_block
> 3.21% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
> 2.27% ksmd [kernel.kallsyms] [k] memcmp
> 2.02% swapper [kernel.kallsyms] [k] default_idle
> 1.58% qemu-kvm [kernel.kallsyms] [k] svm_vcpu_run
> 1.30% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock_irqsave
> 1.09% qemu-kvm [kernel.kallsyms] [k] get_page_from_freelist
# ========
# captured on: Fri Sep 21 08:17:52 2012
# os release : 3.6.0-rc5-elastic+
# perf version : 3.5.2
# arch : x86_64
# nrcpus online : 16
# nrcpus avail : 16
# cpudesc : AMD Opteron(tm) Processor 6128
# cpuid : AuthenticAMD,16,9,1
# total memory : 131973276 kB
# cmdline : /home/root/bin/perf record -g -a
# event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, id = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16 }
# HEADER_CPU_TOPOLOGY info available, use -I to display
# HEADER_NUMA_TOPOLOGY info available, use -I to display
# ========
#
# Samples: 283K of event 'cycles'
# Event count (approx.): 109057976176
#
# Overhead Command Shared Object Symbol
# ........ ............. .................... ..............................................
#
45.60% qemu-kvm [kernel.kallsyms] [k] clear_page_c
|
--- clear_page_c
|
|--93.35%-- do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--91.13%-- 0x10100000006
| |
| --8.87%-- 0x10100000002
|
--6.65%-- __alloc_pages_nodemask
|
|--98.71%-- alloc_pages_vma
| handle_pte_fault
| |
| |--99.78%-- handle_mm_fault
| | __get_user_pages
| | get_user_page_nowait
| | hva_to_pfn.isra.17
| | __gfn_to_pfn
| | gfn_to_pfn_async
| | try_async_pf
| | tdp_page_fault
| | kvm_mmu_page_fault
| | pf_interception
| | handle_exit
| | kvm_arch_vcpu_ioctl_run
| | kvm_vcpu_ioctl
| | do_vfs_ioctl
| | sys_ioctl
| | system_call_fastpath
| | ioctl
| | |
| | |--98.94%-- 0x10100000006
| | |
| | --1.06%-- 0x10100000002
| --0.22%-- [...]
|
--1.29%-- alloc_pages_current
pte_alloc_one
|
|--80.44%-- do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--89.97%-- 0x10100000006
| |
| --10.03%-- 0x10100000002
|
--19.56%-- __pte_alloc
do_huge_pmd_anonymous_page
handle_mm_fault
__get_user_pages
get_user_page_nowait
hva_to_pfn.isra.17
__gfn_to_pfn
gfn_to_pfn_async
try_async_pf
tdp_page_fault
kvm_mmu_page_fault
pf_interception
handle_exit
kvm_arch_vcpu_ioctl_run
kvm_vcpu_ioctl
do_vfs_ioctl
sys_ioctl
system_call_fastpath
ioctl
0x10100000006
11.26% qemu-kvm [kernel.kallsyms] [k] isolate_freepages_block
|
--- isolate_freepages_block
compaction_alloc
migrate_pages
compact_zone
compact_zone_order
try_to_compact_pages
__alloc_pages_direct_compact
__alloc_pages_nodemask
alloc_pages_vma
do_huge_pmd_anonymous_page
handle_mm_fault
__get_user_pages
get_user_page_nowait
hva_to_pfn.isra.17
__gfn_to_pfn
gfn_to_pfn_async
try_async_pf
tdp_page_fault
kvm_mmu_page_fault
pf_interception
handle_exit
kvm_arch_vcpu_ioctl_run
kvm_vcpu_ioctl
do_vfs_ioctl
sys_ioctl
system_call_fastpath
ioctl
|
|--96.34%-- 0x10100000006
|
--3.66%-- 0x10100000002
3.21% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
|
|--39.96%-- tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--99.27%-- 0x10100000006
| |
| --0.73%-- 0x10100000002
|
|--8.69%-- follow_page
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--98.41%-- 0x10100000006
| |
| --1.59%-- 0x10100000002
|
|--8.12%-- kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--99.54%-- 0x10100000006
| --0.46%-- [...]
|
|--7.52%-- kvm_mmu_load
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--99.16%-- 0x10100000006
| |
| --0.84%-- 0x10100000002
|
|--7.42%-- grab_super_passive
| prune_super
| shrink_slab
| try_to_free_pages
| __alloc_pages_nodemask
| alloc_pages_vma
| do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--93.29%-- 0x10100000006
| |
| --6.71%-- 0x10100000002
|
|--7.14%-- put_super
| drop_super
| prune_super
| shrink_slab
| try_to_free_pages
| __alloc_pages_nodemask
| alloc_pages_vma
| do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--94.13%-- 0x10100000006
| |
| --5.87%-- 0x10100000002
|
|--5.17%-- mmu_free_roots
| nonpaging_free
| kvm_mmu_reset_context
| kvm_set_cr4
| emulator_set_cr
| em_cr_write
| x86_emulate_insn
| x86_emulate_instruction
| emulate_on_interception
| cr_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--99.48%-- 0x10100000006
| |
| --0.52%-- 0x10100000002
|
|--2.82%-- yield_to
| kvm_vcpu_yield_to
| kvm_vcpu_on_spin
| pause_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--94.58%-- 0x10100000006
| |
| --5.42%-- 0x10100000002
|
|--2.00%-- __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--99.46%-- 0x10100000006
| |
| --0.54%-- 0x10100000002
|
|--1.69%-- free_pcppages_bulk
| |
| |--77.53%-- drain_pages
| | |
| | |--95.77%-- drain_local_pages
| | | |
| | | |--97.90%-- generic_smp_call_function_interrupt
| | | | smp_call_function_interrupt
| | | | call_function_interrupt
| | | | |
| | | | |--23.37%-- kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | |
| | | | | |--97.22%-- 0x10100000006
| | | | | |
| | | | | --2.78%-- 0x10100000002
| | | | |
| | | | |--17.80%-- __remove_mapping
| | | | | shrink_page_list
| | | | | shrink_inactive_list
| | | | | shrink_lruvec
| | | | | try_to_free_pages
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | |
| | | | | |--93.60%-- 0x10100000006
| | | | | |
| | | | | --6.40%-- 0x10100000002
| | | | |
| | | | |--8.81%-- do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--5.95%-- __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | |
| | | | | |--80.66%-- handle_pte_fault
| | | | | | handle_mm_fault
| | | | | | __get_user_pages
| | | | | | get_user_page_nowait
| | | | | | hva_to_pfn.isra.17
| | | | | | __gfn_to_pfn
| | | | | | gfn_to_pfn_async
| | | | | | try_async_pf
| | | | | | tdp_page_fault
| | | | | | kvm_mmu_page_fault
| | | | | | pf_interception
| | | | | | handle_exit
| | | | | | kvm_arch_vcpu_ioctl_run
| | | | | | kvm_vcpu_ioctl
| | | | | | do_vfs_ioctl
| | | | | | sys_ioctl
| | | | | | system_call_fastpath
| | | | | | ioctl
| | | | | | 0x10100000006
| | | | | |
| | | | | --19.34%-- do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--4.38%-- try_to_free_buffers
| | | | | jbd2_journal_try_to_free_buffers
| | | | | ext4_releasepage
| | | | | try_to_release_page
| | | | | shrink_page_list
| | | | | shrink_inactive_list
| | | | | shrink_lruvec
| | | | | try_to_free_pages
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--4.00%-- isolate_migratepages_range
| | | | | compact_zone
| | | | | compact_zone_order
| | | | | try_to_compact_pages
| | | | | __alloc_pages_direct_compact
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--3.37%-- shrink_inactive_list
| | | | | shrink_lruvec
| | | | | try_to_free_pages
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--3.33%-- free_hot_cold_page_list
| | | | | shrink_page_list
| | | | | shrink_inactive_list
| | | | | shrink_lruvec
| | | | | try_to_free_pages
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--2.31%-- compaction_alloc
| | | | | migrate_pages
| | | | | compact_zone
| | | | | compact_zone_order
| | | | | try_to_compact_pages
| | | | | __alloc_pages_direct_compact
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--2.22%-- compact_checklock_irqsave
| | | | | isolate_migratepages_range
| | | | | compact_zone
| | | | | compact_zone_order
| | | | | try_to_compact_pages
| | | | | __alloc_pages_direct_compact
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--2.19%-- shrink_page_list
| | | | | shrink_inactive_list
| | | | | shrink_lruvec
| | | | | try_to_free_pages
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--2.02%-- buffer_migrate_page
| | | | | move_to_new_page
| | | | | migrate_pages
| | | | | compact_zone
| | | | | compact_zone_order
| | | | | try_to_compact_pages
| | | | | __alloc_pages_direct_compact
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | |
| | | | | |--55.61%-- 0x10100000002
| | | | | |
| | | | | --44.39%-- 0x10100000006
| | | | |
| | | | |--1.42%-- mmu_set_spte.isra.100
| | | | | __direct_map.isra.103
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--1.16%-- on_each_cpu_mask
| | | | | drain_all_pages
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000002
| | | | |
| | | | |--1.16%-- compact_zone_order
| | | | | try_to_compact_pages
| | | | | __alloc_pages_direct_compact
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--1.15%-- compact_zone
| | | | | compact_zone_order
| | | | | try_to_compact_pages
| | | | | __alloc_pages_direct_compact
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--1.15%-- grab_super_passive
| | | | | prune_super
| | | | | shrink_slab
| | | | | try_to_free_pages
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--1.15%-- drop_super
| | | | | prune_super
| | | | | shrink_slab
| | | | | try_to_free_pages
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--1.14%-- finish_task_switch
| | | | | __schedule
| | | | | schedule
| | | | | schedule_preempt_disabled
| | | | | __mutex_lock_slowpath
| | | | | mutex_lock
| | | | | rmap_walk
| | | | | move_to_new_page
| | | | | migrate_pages
| | | | | compact_zone
| | | | | compact_zone_order
| | | | | try_to_compact_pages
| | | | | __alloc_pages_direct_compact
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--1.13%-- smp_call_function_many
| | | | | native_flush_tlb_others
| | | | | flush_tlb_page
| | | | | ptep_clear_flush
| | | | | try_to_unmap_one
| | | | | try_to_unmap_anon
| | | | | try_to_unmap
| | | | | migrate_pages
| | | | | compact_zone
| | | | | compact_zone_order
| | | | | try_to_compact_pages
| | | | | __alloc_pages_direct_compact
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--1.13%-- release_pages
| | | | | pagevec_lru_move_fn
| | | | | __pagevec_lru_add
| | | | | __lru_cache_add
| | | | | lru_cache_add_lru
| | | | | putback_lru_page
| | | | | migrate_pages
| | | | | compact_zone
| | | | | compact_zone_order
| | | | | try_to_compact_pages
| | | | | __alloc_pages_direct_compact
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--1.10%-- move_to_new_page
| | | | | migrate_pages
| | | | | compact_zone
| | | | | compact_zone_order
| | | | | try_to_compact_pages
| | | | | __alloc_pages_direct_compact
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--1.09%-- free_hot_cold_page
| | | | | free_hot_cold_page_list
| | | | | shrink_page_list
| | | | | shrink_inactive_list
| | | | | shrink_lruvec
| | | | | try_to_free_pages
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--0.98%-- kvm_host_page_size
| | | | | mapping_level.isra.88
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | |--0.94%-- pagevec_lru_move_fn
| | | | | __pagevec_lru_add
| | | | | __lru_cache_add
| | | | | lru_cache_add_lru
| | | | | page_add_new_anon_rmap
| | | | | handle_pte_fault
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
^ permalink raw reply [flat|nested] 20+ messages in thread* Re: [PATCH 0/6] Reduce compaction scanning and lock contention
2012-09-21 9:17 ` Richard Davies
@ 2012-09-21 9:55 ` Mel Gorman
0 siblings, 0 replies; 20+ messages in thread
From: Mel Gorman @ 2012-09-21 9:55 UTC (permalink / raw)
To: Richard Davies
Cc: Shaohua Li, Rik van Riel, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML
On Fri, Sep 21, 2012 at 10:17:01AM +0100, Richard Davies wrote:
> Richard Davies wrote:
> > I did manage to get a couple which were slightly worse, but nothing like as
> > bad as before. Here are the results:
> >
> > # grep -F '[k]' report | head -8
> > 45.60% qemu-kvm [kernel.kallsyms] [k] clear_page_c
> > 11.26% qemu-kvm [kernel.kallsyms] [k] isolate_freepages_block
> > 3.21% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
> > 2.27% ksmd [kernel.kallsyms] [k] memcmp
> > 2.02% swapper [kernel.kallsyms] [k] default_idle
> > 1.58% qemu-kvm [kernel.kallsyms] [k] svm_vcpu_run
> > 1.30% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock_irqsave
> > 1.09% qemu-kvm [kernel.kallsyms] [k] get_page_from_freelist
>
> # ========
> # captured on: Fri Sep 21 08:17:52 2012
> # os release : 3.6.0-rc5-elastic+
> # perf version : 3.5.2
> # arch : x86_64
> # nrcpus online : 16
> # nrcpus avail : 16
> # cpudesc : AMD Opteron(tm) Processor 6128
> # cpuid : AuthenticAMD,16,9,1
> # total memory : 131973276 kB
> # cmdline : /home/root/bin/perf record -g -a
> # event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, id = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16 }
> # HEADER_CPU_TOPOLOGY info available, use -I to display
> # HEADER_NUMA_TOPOLOGY info available, use -I to display
> # ========
> #
> # Samples: 283K of event 'cycles'
> # Event count (approx.): 109057976176
> #
> # Overhead Command Shared Object Symbol
> # ........ ............. .................... ..............................................
> #
> 45.60% qemu-kvm [kernel.kallsyms] [k] clear_page_c
> |
> --- clear_page_c
> |
> |--93.35%-- do_huge_pmd_anonymous_page
This is unavoidable. If THP was disabled, the cost would still be
incurred, just on base pages instead of huge pages.
> <SNIP>
> 11.26% qemu-kvm [kernel.kallsyms] [k] isolate_freepages_block
> |
> --- isolate_freepages_block
> compaction_alloc
> migrate_pages
> compact_zone
> compact_zone_order
> try_to_compact_pages
> __alloc_pages_direct_compact
> __alloc_pages_nodemask
> alloc_pages_vma
> do_huge_pmd_anonymous_page
And this is showing that we're still spending a lot of time scanning
for free pages to isolate. I do not have a great idea on how this can be
reduced further without interfering with the page allocator.
One ok idea I considered in the past was using the buddy lists to find
free pages quickly but there is first the problem that the buddy lists
themselves may need to be searched and now that the zone lock is not held
during the scan it would be particularly difficult. The harder problem is
deciding when compaction "finishes". I'll put more thought into it over
the weekend and see if something falls out but I'm not going to hold up
this series waiting for inspiration.
> 3.21% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
> |
> --- _raw_spin_lock
> |
> |--39.96%-- tdp_page_fault
Nothing very interesting here until...
> |--1.69%-- free_pcppages_bulk
> | |
> | |--77.53%-- drain_pages
> | | |
> | | |--95.77%-- drain_local_pages
> | | | |
> | | | |--97.90%-- generic_smp_call_function_interrupt
> | | | | smp_call_function_interrupt
> | | | | call_function_interrupt
> | | | | |
> | | | | |--23.37%-- kvm_vcpu_ioctl
> | | | | | do_vfs_ioctl
> | | | | | sys_ioctl
> | | | | | system_call_fastpath
> | | | | | ioctl
> | | | | | |
> | | | | | |--97.22%-- 0x10100000006
> | | | | | |
> | | | | | --2.78%-- 0x10100000002
> | | | | |
> | | | | |--17.80%-- __remove_mapping
> | | | | | shrink_page_list
> | | | | | shrink_inactive_list
> | | | | | shrink_lruvec
> | | | | | try_to_free_pages
> | | | | | __alloc_pages_nodemask
> | | | | | alloc_pages_vma
> | | | | | do_huge_pmd_anonymous_page
This whole section is interesting simply because it shows the per-cpu
draining cost. It's low enough that I'm not going to put much thought
into it but it's not often the per-cpu allocator sticks out like this.
Thanks Richard.
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 0/6] Reduce compaction scanning and lock contention
2012-09-21 9:13 ` [PATCH 0/6] Reduce compaction scanning and lock contention Richard Davies
2012-09-21 9:15 ` Richard Davies
2012-09-21 9:17 ` Richard Davies
@ 2012-09-21 9:18 ` Richard Davies
2012-09-21 9:35 ` Mel Gorman
3 siblings, 0 replies; 20+ messages in thread
From: Richard Davies @ 2012-09-21 9:18 UTC (permalink / raw)
To: Mel Gorman
Cc: Shaohua Li, Rik van Riel, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML
Richard Davies wrote:
> I did manage to get a couple which were slightly worse, but nothing like as
> bad as before. Here are the results:
...
> # grep -F '[k]' report | head -8
> 61.29% qemu-kvm [kernel.kallsyms] [k] clear_page_c
> 4.52% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock_irqsave
> 2.64% qemu-kvm [kernel.kallsyms] [k] copy_page_c
> 1.61% swapper [kernel.kallsyms] [k] default_idle
> 1.57% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
> 1.18% qemu-kvm [kernel.kallsyms] [k] get_page_from_freelist
> 1.18% qemu-kvm [kernel.kallsyms] [k] isolate_freepages_block
> 1.11% qemu-kvm [kernel.kallsyms] [k] svm_vcpu_run
# ========
# captured on: Fri Sep 21 08:40:48 2012
# os release : 3.6.0-rc5-elastic+
# perf version : 3.5.2
# arch : x86_64
# nrcpus online : 16
# nrcpus avail : 16
# cpudesc : AMD Opteron(tm) Processor 6128
# cpuid : AuthenticAMD,16,9,1
# total memory : 131973276 kB
# cmdline : /home/root/bin/perf record -g -a
# event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, id = { 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112 }
# HEADER_CPU_TOPOLOGY info available, use -I to display
# HEADER_NUMA_TOPOLOGY info available, use -I to display
# ========
#
# Samples: 914K of event 'cycles'
# Event count (approx.): 328377288871
#
# Overhead Command Shared Object Symbol
# ........ ............. .................... ..............................................
#
61.29% qemu-kvm [kernel.kallsyms] [k] clear_page_c
|
--- clear_page_c
|
|--98.26%-- do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--67.45%-- 0x10100000006
| |
| --32.55%-- 0x10100000002
|
--1.74%-- __alloc_pages_nodemask
|
|--91.69%-- alloc_pages_vma
| handle_pte_fault
| |
| |--99.76%-- handle_mm_fault
| | |
| | |--99.78%-- __get_user_pages
| | | get_user_page_nowait
| | | hva_to_pfn.isra.17
| | | __gfn_to_pfn
| | | gfn_to_pfn_async
| | | try_async_pf
| | | tdp_page_fault
| | | kvm_mmu_page_fault
| | | pf_interception
| | | handle_exit
| | | kvm_arch_vcpu_ioctl_run
| | | kvm_vcpu_ioctl
| | | do_vfs_ioctl
| | | sys_ioctl
| | | system_call_fastpath
| | | ioctl
| | | |
| | | |--98.56%-- 0x10100000006
| | | |
| | | --1.44%-- 0x10100000002
| | --0.22%-- [...]
| --0.24%-- [...]
|
--8.31%-- alloc_pages_current
|
|--99.43%-- pte_alloc_one
| |
| |--97.72%-- do_huge_pmd_anonymous_page
| | handle_mm_fault
| | __get_user_pages
| | get_user_page_nowait
| | hva_to_pfn.isra.17
| | __gfn_to_pfn
| | gfn_to_pfn_async
| | try_async_pf
| | tdp_page_fault
| | kvm_mmu_page_fault
| | pf_interception
| | handle_exit
| | kvm_arch_vcpu_ioctl_run
| | kvm_vcpu_ioctl
| | do_vfs_ioctl
| | sys_ioctl
| | system_call_fastpath
| | ioctl
| | |
| | |--56.04%-- 0x10100000006
| | |
| | --43.96%-- 0x10100000002
| |
| --2.28%-- __pte_alloc
| do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| 0x10100000006
--0.57%-- [...]
4.52% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
|
|--90.15%-- compact_checklock_irqsave
| |
| |--99.77%-- isolate_migratepages_range
| | compact_zone
| | compact_zone_order
| | try_to_compact_pages
| | __alloc_pages_direct_compact
| | __alloc_pages_nodemask
| | alloc_pages_vma
| | do_huge_pmd_anonymous_page
| | handle_mm_fault
| | __get_user_pages
| | get_user_page_nowait
| | hva_to_pfn.isra.17
| | __gfn_to_pfn
| | gfn_to_pfn_async
| | try_async_pf
| | tdp_page_fault
| | kvm_mmu_page_fault
| | pf_interception
| | handle_exit
| | kvm_arch_vcpu_ioctl_run
| | kvm_vcpu_ioctl
| | do_vfs_ioctl
| | sys_ioctl
| | system_call_fastpath
| | ioctl
| | |
| | |--94.58%-- 0x10100000006
| | |
| | --5.42%-- 0x10100000002
| --0.23%-- [...]
|
|--3.60%-- isolate_migratepages_range
| compact_zone
| compact_zone_order
| try_to_compact_pages
| __alloc_pages_direct_compact
| __alloc_pages_nodemask
| alloc_pages_vma
| do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--91.30%-- 0x10100000006
| |
| --8.70%-- 0x10100000002
|
|--2.26%-- pagevec_lru_move_fn
| __pagevec_lru_add
| |
| |--96.55%-- __lru_cache_add
| | lru_cache_add_lru
| | |
| | |--96.23%-- putback_lru_page
| | | |
| | | |--98.46%-- migrate_pages
| | | | compact_zone
| | | | compact_zone_order
| | | | try_to_compact_pages
| | | | __alloc_pages_direct_compact
| | | | __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | |
| | | | |--91.70%-- 0x10100000006
| | | | |
| | | | --8.30%-- 0x10100000002
| | | |
| | | --1.54%-- putback_lru_pages
| | | compact_zone
| | | compact_zone_order
| | | try_to_compact_pages
| | | __alloc_pages_direct_compact
| | | __alloc_pages_nodemask
| | | alloc_pages_vma
| | | do_huge_pmd_anonymous_page
| | | handle_mm_fault
| | | __get_user_pages
| | | get_user_page_nowait
| | | hva_to_pfn.isra.17
| | | __gfn_to_pfn
| | | gfn_to_pfn_async
| | | try_async_pf
| | | tdp_page_fault
| | | kvm_mmu_page_fault
| | | pf_interception
| | | handle_exit
| | | kvm_arch_vcpu_ioctl_run
| | | kvm_vcpu_ioctl
| | | do_vfs_ioctl
| | | sys_ioctl
| | | system_call_fastpath
| | | ioctl
| | | 0x10100000006
| | |
| | --3.77%-- page_add_new_anon_rmap
| | handle_pte_fault
| | handle_mm_fault
| | __get_user_pages
| | get_user_page_nowait
| | hva_to_pfn.isra.17
| | __gfn_to_pfn
| | gfn_to_pfn_async
| | try_async_pf
| | tdp_page_fault
| | kvm_mmu_page_fault
| | pf_interception
| | handle_exit
| | kvm_arch_vcpu_ioctl_run
| | kvm_vcpu_ioctl
| | do_vfs_ioctl
| | sys_ioctl
| | system_call_fastpath
| | ioctl
| | |
| | |--97.54%-- 0x10100000006
| | |
| | --2.46%-- 0x10100000002
| |
| --3.45%-- lru_add_drain_cpu
| lru_add_drain
| migrate_prep_local
| compact_zone
| compact_zone_order
| try_to_compact_pages
| __alloc_pages_direct_compact
| __alloc_pages_nodemask
| alloc_pages_vma
| do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| 0x10100000006
|
|--1.58%-- release_pages
| pagevec_lru_move_fn
| __pagevec_lru_add
| |
| |--98.25%-- __lru_cache_add
| | lru_cache_add_lru
| | putback_lru_page
| | |
| | |--99.42%-- migrate_pages
| | | compact_zone
| | | compact_zone_order
| | | try_to_compact_pages
| | | __alloc_pages_direct_compact
| | | __alloc_pages_nodemask
| | | alloc_pages_vma
| | | do_huge_pmd_anonymous_page
| | | handle_mm_fault
| | | __get_user_pages
| | | get_user_page_nowait
| | | hva_to_pfn.isra.17
| | | __gfn_to_pfn
| | | gfn_to_pfn_async
| | | try_async_pf
| | | tdp_page_fault
| | | kvm_mmu_page_fault
| | | pf_interception
| | | handle_exit
| | | kvm_arch_vcpu_ioctl_run
| | | kvm_vcpu_ioctl
| | | do_vfs_ioctl
| | | sys_ioctl
| | | system_call_fastpath
| | | ioctl
| | | |
| | | |--95.23%-- 0x10100000006
| | | |
| | | --4.77%-- 0x10100000002
| | |
| | --0.58%-- putback_lru_pages
| | compact_zone
| | compact_zone_order
| | try_to_compact_pages
| | __alloc_pages_direct_compact
| | __alloc_pages_nodemask
| | alloc_pages_vma
| | do_huge_pmd_anonymous_page
| | handle_mm_fault
| | __get_user_pages
| | get_user_page_nowait
| | hva_to_pfn.isra.17
| | __gfn_to_pfn
| | gfn_to_pfn_async
| | try_async_pf
| | tdp_page_fault
| | kvm_mmu_page_fault
| | pf_interception
| | handle_exit
| | kvm_arch_vcpu_ioctl_run
| | kvm_vcpu_ioctl
| | do_vfs_ioctl
| | sys_ioctl
| | system_call_fastpath
| | ioctl
| | 0x10100000006
| |
| --1.75%-- lru_add_drain_cpu
| lru_add_drain
| migrate_prep_local
| compact_zone
| compact_zone_order
| try_to_compact_pages
| __alloc_pages_direct_compact
| __alloc_pages_nodemask
| alloc_pages_vma
| do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--80.09%-- 0x10100000006
| |
| --19.91%-- 0x10100000002
|
|--0.99%-- __page_cache_release.part.11
| __put_single_page
| put_page
| putback_lru_page
| migrate_pages
| compact_zone
| compact_zone_order
| try_to_compact_pages
| __alloc_pages_direct_compact
| __alloc_pages_nodemask
| alloc_pages_vma
| do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--92.20%-- 0x10100000006
| |
| --7.80%-- 0x10100000002
--1.42%-- [...]
2.64% qemu-kvm [kernel.kallsyms] [k] copy_page_c
|
--- copy_page_c
|
|--75.79%-- buffer_migrate_page
| move_to_new_page
| migrate_pages
| compact_zone
| compact_zone_order
| try_to_compact_pages
| __alloc_pages_direct_compact
| __alloc_pages_nodemask
| alloc_pages_vma
| do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--98.86%-- 0x10100000006
| |
| --1.14%-- 0x10100000002
|
|--24.08%-- migrate_page
| |
| |--82.12%-- buffer_migrate_page
| | move_to_new_page
| | migrate_pages
| | compact_zone
| | compact_zone_order
| | try_to_compact_pages
| | __alloc_pages_direct_compact
| | __alloc_pages_nodemask
| | alloc_pages_vma
| | do_huge_pmd_anonymous_page
| | handle_mm_fault
| | __get_user_pages
| | get_user_page_nowait
| | hva_to_pfn.isra.17
| | __gfn_to_pfn
| | gfn_to_pfn_async
| | try_async_pf
| | tdp_page_fault
| | kvm_mmu_page_fault
| | pf_interception
| | handle_exit
| | kvm_arch_vcpu_ioctl_run
| | kvm_vcpu_ioctl
| | do_vfs_ioctl
| | sys_ioctl
| | system_call_fastpath
| | ioctl
| | |
| | |--98.08%-- 0x10100000006
| | |
| | --1.92%-- 0x10100000002
| |
| --17.88%-- move_to_new_page
| migrate_pages
| compact_zone
| compact_zone_order
| try_to_compact_pages
| __alloc_pages_direct_compact
| __alloc_pages_nodemask
| alloc_pages_vma
| do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--99.21%-- 0x10100000006
| |
| --0.79%-- 0x10100000002
--0.13%-- [...]
1.61% swapper [kernel.kallsyms] [k] default_idle
|
--- default_idle
|
|--99.76%-- cpu_idle
| |
| |--77.77%-- start_secondary
| |
| --22.23%-- rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
--0.24%-- [...]
1.57% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
|
|--37.29%-- tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--89.54%-- 0x10100000006
| |
| --10.46%-- 0x10100000002
|
|--9.09%-- kvm_mmu_load
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--59.75%-- 0x10100000006
| |
| --40.25%-- 0x10100000002
|
|--8.00%-- follow_page
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--98.39%-- 0x10100000006
| |
| --1.61%-- 0x10100000002
|
|--7.17%-- kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--94.45%-- 0x10100000006
| |
| --5.55%-- 0x10100000002
|
|--5.65%-- mmu_free_roots
| |
| |--76.51%-- nonpaging_free
| | kvm_mmu_reset_context
| | kvm_set_cr4
| | emulator_set_cr
| | em_cr_write
| | x86_emulate_insn
| | x86_emulate_instruction
| | emulate_on_interception
| | cr_interception
| | handle_exit
| | kvm_arch_vcpu_ioctl_run
| | kvm_vcpu_ioctl
| | do_vfs_ioctl
| | sys_ioctl
| | system_call_fastpath
| | ioctl
| | |
| | |--64.15%-- 0x10100000006
| | |
| | --35.85%-- 0x10100000002
| |
| --23.49%-- kvm_mmu_unload
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--73.65%-- 0x10100000006
| |
| --26.35%-- 0x10100000002
|
|--4.61%-- put_super
| drop_super
| prune_super
| shrink_slab
| try_to_free_pages
| __alloc_pages_nodemask
| alloc_pages_vma
| do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--93.31%-- 0x10100000006
| |
| --6.69%-- 0x10100000002
|
|--4.60%-- grab_super_passive
| prune_super
| shrink_slab
| try_to_free_pages
| __alloc_pages_nodemask
| alloc_pages_vma
| do_huge_pmd_anonymous_page
| handle_mm_fault
| __get_user_pages
| get_user_page_nowait
| hva_to_pfn.isra.17
| __gfn_to_pfn
| gfn_to_pfn_async
| try_async_pf
| tdp_page_fault
| kvm_mmu_page_fault
| pf_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--95.07%-- 0x10100000006
| |
| --4.93%-- 0x10100000002
|
|--3.94%-- yield_to
| kvm_vcpu_yield_to
| kvm_vcpu_on_spin
| pause_interception
| handle_exit
| kvm_arch_vcpu_ioctl_run
| kvm_vcpu_ioctl
| do_vfs_ioctl
| sys_ioctl
| system_call_fastpath
| ioctl
| |
| |--86.93%-- 0x10100000006
| |
| --13.07%-- 0x10100000002
|
|--3.08%-- free_pcppages_bulk
| |
| |--89.90%-- drain_pages
| | |
| | |--95.03%-- drain_local_pages
| | | generic_smp_call_function_interrupt
| | | smp_call_function_interrupt
| | | call_function_interrupt
| | | |
| | | |--29.99%-- buffer_migrate_page
| | | | move_to_new_page
| | | | migrate_pages
| | | | compact_zone
| | | | compact_zone_order
| | | | try_to_compact_pages
| | | | __alloc_pages_direct_compact
| | | | __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--13.18%-- kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | |
| | | | |--98.30%-- 0x10100000006
| | | | |
| | | | --1.70%-- 0x10100000002
| | | |
| | | |--8.86%-- compact_checklock_irqsave
| | | | |
| | | | |--65.32%-- isolate_migratepages_range
| | | | | compact_zone
| | | | | compact_zone_order
| | | | | try_to_compact_pages
| | | | | __alloc_pages_direct_compact
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | |
| | | | | |--94.13%-- 0x10100000006
| | | | | |
| | | | | --5.87%-- 0x10100000002
| | | | |
| | | | --34.68%-- isolate_freepages_block
| | | | compaction_alloc
| | | | migrate_pages
| | | | compact_zone
| | | | compact_zone_order
| | | | try_to_compact_pages
| | | | __alloc_pages_direct_compact
| | | | __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--8.21%-- migrate_page
| | | | |
| | | | |--91.65%-- buffer_migrate_page
| | | | | move_to_new_page
| | | | | migrate_pages
| | | | | compact_zone
| | | | | compact_zone_order
| | | | | try_to_compact_pages
| | | | | __alloc_pages_direct_compact
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
| | | | | handle_exit
| | | | | kvm_arch_vcpu_ioctl_run
| | | | | kvm_vcpu_ioctl
| | | | | do_vfs_ioctl
| | | | | sys_ioctl
| | | | | system_call_fastpath
| | | | | ioctl
| | | | | 0x10100000006
| | | | |
| | | | --8.35%-- move_to_new_page
| | | | migrate_pages
| | | | compact_zone
| | | | compact_zone_order
| | | | try_to_compact_pages
| | | | __alloc_pages_direct_compact
| | | | __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--6.16%-- do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--5.39%-- isolate_migratepages_range
| | | | compact_zone
| | | | compact_zone_order
| | | | try_to_compact_pages
| | | | __alloc_pages_direct_compact
| | | | __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--3.59%-- compaction_alloc
| | | | migrate_pages
| | | | compact_zone
| | | | compact_zone_order
| | | | try_to_compact_pages
| | | | __alloc_pages_direct_compact
| | | | __alloc_pages_nodemask
| | | | alloc_pages_vma
| | | | do_huge_pmd_anonymous_page
| | | | handle_mm_fault
| | | | __get_user_pages
| | | | get_user_page_nowait
| | | | hva_to_pfn.isra.17
| | | | __gfn_to_pfn
| | | | gfn_to_pfn_async
| | | | try_async_pf
| | | | tdp_page_fault
| | | | kvm_mmu_page_fault
| | | | pf_interception
| | | | handle_exit
| | | | kvm_arch_vcpu_ioctl_run
| | | | kvm_vcpu_ioctl
| | | | do_vfs_ioctl
| | | | sys_ioctl
| | | | system_call_fastpath
| | | | ioctl
| | | | 0x10100000006
| | | |
| | | |--3.19%-- migrate_page_move_mapping.part.16
| | | | |
| | | | |--89.86%-- buffer_migrate_page
| | | | | move_to_new_page
| | | | | migrate_pages
| | | | | compact_zone
| | | | | compact_zone_order
| | | | | try_to_compact_pages
| | | | | __alloc_pages_direct_compact
| | | | | __alloc_pages_nodemask
| | | | | alloc_pages_vma
| | | | | do_huge_pmd_anonymous_page
| | | | | handle_mm_fault
| | | | | __get_user_pages
| | | | | get_user_page_nowait
| | | | | hva_to_pfn.isra.17
| | | | | __gfn_to_pfn
| | | | | gfn_to_pfn_async
| | | | | try_async_pf
| | | | | tdp_page_fault
| | | | | kvm_mmu_page_fault
| | | | | pf_interception
^ permalink raw reply [flat|nested] 20+ messages in thread* Re: [PATCH 0/6] Reduce compaction scanning and lock contention
2012-09-21 9:13 ` [PATCH 0/6] Reduce compaction scanning and lock contention Richard Davies
` (2 preceding siblings ...)
2012-09-21 9:18 ` Richard Davies
@ 2012-09-21 9:35 ` Mel Gorman
2012-09-21 9:49 ` Richard Davies
3 siblings, 1 reply; 20+ messages in thread
From: Mel Gorman @ 2012-09-21 9:35 UTC (permalink / raw)
To: Richard Davies
Cc: Shaohua Li, Rik van Riel, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML
On Fri, Sep 21, 2012 at 10:13:33AM +0100, Richard Davies wrote:
> Hi Mel,
>
> Thank you for this series. I have applied on clean 3.6-rc5 and tested, and
> it works well for me - the lock contention is (still) gone and
> isolate_freepages_block is much reduced.
>
Excellent!
> Here is a typical test with these patches:
>
> # grep -F '[k]' report | head -8
> 65.20% qemu-kvm [kernel.kallsyms] [k] clear_page_c
> 2.18% qemu-kvm [kernel.kallsyms] [k] isolate_freepages_block
> 1.56% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
> 1.40% qemu-kvm [kernel.kallsyms] [k] svm_vcpu_run
> 1.38% swapper [kernel.kallsyms] [k] default_idle
> 1.35% qemu-kvm [kernel.kallsyms] [k] get_page_from_freelist
> 0.74% ksmd [kernel.kallsyms] [k] memcmp
> 0.72% qemu-kvm [kernel.kallsyms] [k] free_pages_prepare
>
Ok, so that is more or less acceptable. I would like to reduce the scanning
even further but I'll take this as a start -- largely because I do not have
any new good ideas on how it could be reduced further without incurring
a large cost in the page allocator :)
> I did manage to get a couple which were slightly worse, but nothing like as
> bad as before. Here are the results:
>
> # grep -F '[k]' report | head -8
> 45.60% qemu-kvm [kernel.kallsyms] [k] clear_page_c
> 11.26% qemu-kvm [kernel.kallsyms] [k] isolate_freepages_block
> 3.21% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
> 2.27% ksmd [kernel.kallsyms] [k] memcmp
> 2.02% swapper [kernel.kallsyms] [k] default_idle
> 1.58% qemu-kvm [kernel.kallsyms] [k] svm_vcpu_run
> 1.30% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock_irqsave
> 1.09% qemu-kvm [kernel.kallsyms] [k] get_page_from_freelist
>
> # grep -F '[k]' report | head -8
> 61.29% qemu-kvm [kernel.kallsyms] [k] clear_page_c
> 4.52% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock_irqsave
> 2.64% qemu-kvm [kernel.kallsyms] [k] copy_page_c
> 1.61% swapper [kernel.kallsyms] [k] default_idle
> 1.57% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
> 1.18% qemu-kvm [kernel.kallsyms] [k] get_page_from_freelist
> 1.18% qemu-kvm [kernel.kallsyms] [k] isolate_freepages_block
> 1.11% qemu-kvm [kernel.kallsyms] [k] svm_vcpu_run
>
>
Were the boot times acceptable even when these slightly worse figures
were recorded?
> I will follow up with the detailed traces for these three tests.
>
> Thank you!
>
Thank you for the detailed reporting and the testing, it's much
appreciated. I've already rebased the patches to Andrew's tree and tested
them overnight and the figures look good on my side. I'll update the
changelog and push them shortly.
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 20+ messages in thread* Re: [PATCH 0/6] Reduce compaction scanning and lock contention
2012-09-21 9:35 ` Mel Gorman
@ 2012-09-21 9:49 ` Richard Davies
0 siblings, 0 replies; 20+ messages in thread
From: Richard Davies @ 2012-09-21 9:49 UTC (permalink / raw)
To: Mel Gorman
Cc: Shaohua Li, Rik van Riel, Avi Kivity, QEMU-devel, KVM, Linux-MM, LKML
Mel Gorman wrote:
> > I did manage to get a couple which were slightly worse, but nothing like as
> > bad as before. Here are the results:
> >
> > # grep -F '[k]' report | head -8
> > 45.60% qemu-kvm [kernel.kallsyms] [k] clear_page_c
> > 11.26% qemu-kvm [kernel.kallsyms] [k] isolate_freepages_block
> > 3.21% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
> > 2.27% ksmd [kernel.kallsyms] [k] memcmp
> > 2.02% swapper [kernel.kallsyms] [k] default_idle
> > 1.58% qemu-kvm [kernel.kallsyms] [k] svm_vcpu_run
> > 1.30% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock_irqsave
> > 1.09% qemu-kvm [kernel.kallsyms] [k] get_page_from_freelist
> >
> > # grep -F '[k]' report | head -8
> > 61.29% qemu-kvm [kernel.kallsyms] [k] clear_page_c
> > 4.52% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock_irqsave
> > 2.64% qemu-kvm [kernel.kallsyms] [k] copy_page_c
> > 1.61% swapper [kernel.kallsyms] [k] default_idle
> > 1.57% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock
> > 1.18% qemu-kvm [kernel.kallsyms] [k] get_page_from_freelist
> > 1.18% qemu-kvm [kernel.kallsyms] [k] isolate_freepages_block
> > 1.11% qemu-kvm [kernel.kallsyms] [k] svm_vcpu_run
>
> Were the boot times acceptable even when these slightly worse figures
> were recorded?
Yes, they were 10-20% slower as you might expect from the traces, rather
than a factor slower.
> Thank you for the detailed reporting and the testing, it's much
> appreciated. I've already rebased the patches to Andrew's tree and tested
> them overnight and the figures look good on my side. I'll update the
> changelog and push them shortly.
Great. On my side, I'm delighted that senior kernel developers such as you,
Rik and Avi took our bug report seriously and helped fix it!
Thank you,
Richard.
^ permalink raw reply [flat|nested] 20+ messages in thread