Re: [PATCH 28/28] mm, page_alloc: Defer debugging checks of pages allocated from the PCP

[Vlastimil Babka <vbabka@suse.cz>]

On 04/15/2016 11:07 AM, Mel Gorman wrote:
> Every page allocated checks a number of page fields for validity. This
> catches corruption bugs of pages that are already freed but it is expensive.
> This patch weakens the debugging check by checking PCP pages only when
> the PCP lists are being refilled. All compound pages are checked. This
> potentially avoids debugging checks entirely if the PCP lists are never
> emptied and refilled so some corruption issues may be missed. Full checking
> requires DEBUG_VM.
> 
> With the two deferred debugging patches applied, the impact to a page
> allocator microbenchmark is
> 
>                                             4.6.0-rc3                  4.6.0-rc3
>                                           inline-v3r6            deferalloc-v3r7
> Min      alloc-odr0-1               344.00 (  0.00%)           317.00 (  7.85%)
> Min      alloc-odr0-2               248.00 (  0.00%)           231.00 (  6.85%)
> Min      alloc-odr0-4               209.00 (  0.00%)           192.00 (  8.13%)
> Min      alloc-odr0-8               181.00 (  0.00%)           166.00 (  8.29%)
> Min      alloc-odr0-16              168.00 (  0.00%)           154.00 (  8.33%)
> Min      alloc-odr0-32              161.00 (  0.00%)           148.00 (  8.07%)
> Min      alloc-odr0-64              158.00 (  0.00%)           145.00 (  8.23%)
> Min      alloc-odr0-128             156.00 (  0.00%)           143.00 (  8.33%)
> Min      alloc-odr0-256             168.00 (  0.00%)           154.00 (  8.33%)
> Min      alloc-odr0-512             178.00 (  0.00%)           167.00 (  6.18%)
> Min      alloc-odr0-1024            186.00 (  0.00%)           174.00 (  6.45%)
> Min      alloc-odr0-2048            192.00 (  0.00%)           180.00 (  6.25%)
> Min      alloc-odr0-4096            198.00 (  0.00%)           184.00 (  7.07%)
> Min      alloc-odr0-8192            200.00 (  0.00%)           188.00 (  6.00%)
> Min      alloc-odr0-16384           201.00 (  0.00%)           188.00 (  6.47%)
> Min      free-odr0-1                189.00 (  0.00%)           180.00 (  4.76%)
> Min      free-odr0-2                132.00 (  0.00%)           126.00 (  4.55%)
> Min      free-odr0-4                104.00 (  0.00%)            99.00 (  4.81%)
> Min      free-odr0-8                 90.00 (  0.00%)            85.00 (  5.56%)
> Min      free-odr0-16                84.00 (  0.00%)            80.00 (  4.76%)
> Min      free-odr0-32                80.00 (  0.00%)            76.00 (  5.00%)
> Min      free-odr0-64                78.00 (  0.00%)            74.00 (  5.13%)
> Min      free-odr0-128               77.00 (  0.00%)            73.00 (  5.19%)
> Min      free-odr0-256               94.00 (  0.00%)            91.00 (  3.19%)
> Min      free-odr0-512              108.00 (  0.00%)           112.00 ( -3.70%)
> Min      free-odr0-1024             115.00 (  0.00%)           118.00 ( -2.61%)
> Min      free-odr0-2048             120.00 (  0.00%)           125.00 ( -4.17%)
> Min      free-odr0-4096             123.00 (  0.00%)           129.00 ( -4.88%)
> Min      free-odr0-8192             126.00 (  0.00%)           130.00 ( -3.17%)
> Min      free-odr0-16384            126.00 (  0.00%)           131.00 ( -3.97%)
> 
> Note that the free paths for large numbers of pages is impacted as the
> debugging cost gets shifted into that path when the page data is no longer
> necessarily cache-hot.
> 
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>

Acked-by: Vlastimil Babka <vbabka@suse.cz>

Unlike the free path, there are no duplications here, which is nice.
Some un-inlining of bad page check should still work here though imho:

>From afdefd87f2d8d07cba4bd2a2f3531dc8bb0b7a19 Mon Sep 17 00:00:00 2001
From: Vlastimil Babka <vbabka@suse.cz>
Date: Wed, 27 Apr 2016 15:47:29 +0200
Subject: [PATCH] mm, page_alloc: uninline the bad page part of
 check_new_page()

Bad pages should be rare so the code handling them doesn't need to be inline
for performance reasons. Put it to separate function which returns void.
This also assumes that the initial page_expected_state() result will match the
result of the thorough check, i.e. the page doesn't become "good" in the
meanwhile. This matches the same expectations already in place in
free_pages_check().

!DEBUG_VM bloat-o-meter:

add/remove: 1/0 grow/shrink: 0/1 up/down: 134/-274 (-140)
function                                     old     new   delta
check_new_page_bad                             -     134    +134
get_page_from_freelist                      3468    3194    -274

Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
 mm/page_alloc.c | 33 +++++++++++++++++----------------
 1 file changed, 17 insertions(+), 16 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 2b3aefdfcaa2..755ec9465d8a 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1648,19 +1648,11 @@ static inline void expand(struct zone *zone, struct page *page,
 	}
 }
 
-/*
- * This page is about to be returned from the page allocator
- */
-static inline int check_new_page(struct page *page)
+static void check_new_page_bad(struct page *page)
 {
-	const char *bad_reason;
-	unsigned long bad_flags;
+	const char *bad_reason = NULL;
+	unsigned long bad_flags = 0;
 
-	if (page_expected_state(page, PAGE_FLAGS_CHECK_AT_PREP|__PG_HWPOISON))
-		return 0;
-
-	bad_reason = NULL;
-	bad_flags = 0;
 	if (unlikely(atomic_read(&page->_mapcount) != -1))
 		bad_reason = "nonzero mapcount";
 	if (unlikely(page->mapping != NULL))
@@ -1679,11 +1671,20 @@ static inline int check_new_page(struct page *page)
 	if (unlikely(page->mem_cgroup))
 		bad_reason = "page still charged to cgroup";
 #endif
-	if (unlikely(bad_reason)) {
-		bad_page(page, bad_reason, bad_flags);
-		return 1;
-	}
-	return 0;
+	bad_page(page, bad_reason, bad_flags);
+}
+
+/*
+ * This page is about to be returned from the page allocator
+ */
+static inline int check_new_page(struct page *page)
+{
+	if (likely(page_expected_state(page,
+				PAGE_FLAGS_CHECK_AT_PREP|__PG_HWPOISON)))
+		return 0;
+
+	check_new_page_bad(page);
+	return 1;
 }
 
 static inline bool free_pages_prezeroed(bool poisoned)
-- 
2.8.1