[PATCH 1/2] mm: free large amount of 0-order pages in workqueue

[PATCH 1/2] mm: free large amount of 0-order pages in workqueue

[Sasha Levin]

Freeing pages became a rather costly operation, specially when multiple debug
options are enabled. This causes hangs when an attempt to free a large amount
of 0-order is made. Two examples are vfree()ing large block of memory, and
punching a hole in a shmem filesystem.

To avoid that, move any free operations that involve batching pages into a
list to a workqueue handler where they could be freed later.

Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
---
 mm/page_alloc.c |   50 ++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 46 insertions(+), 4 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 5bd9711..812ca75 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1586,10 +1586,11 @@ out:
 	local_irq_restore(flags);
 }
 
-/*
- * Free a list of 0-order pages
- */
-void free_hot_cold_page_list(struct list_head *list, bool cold)
+static LIST_HEAD(free_hot_page_list);
+static LIST_HEAD(free_cold_page_list);
+static DEFINE_SPINLOCK(free_page_lock);
+
+static void __free_hot_cold_page_list(struct list_head *list, bool cold)
 {
 	struct page *page, *next;
 
@@ -1599,6 +1600,47 @@ void free_hot_cold_page_list(struct list_head *list, bool cold)
 	}
 }
 
+static void free_page_lists_work(struct work_struct *work)
+{
+	LIST_HEAD(hot_pages);
+	LIST_HEAD(cold_pages);
+	unsigned long flags;
+
+	spin_lock_irqsave(&free_page_lock, flags);
+	list_cut_position(&hot_pages, &free_hot_page_list,
+					free_hot_page_list.prev);
+	list_cut_position(&cold_pages, &free_cold_page_list,
+					free_cold_page_list.prev);
+	spin_unlock_irqrestore(&free_page_lock, flags);
+
+	__free_hot_cold_page_list(&hot_pages, false);
+	__free_hot_cold_page_list(&cold_pages, true);
+}
+
+static DECLARE_WORK(free_page_work, free_page_lists_work);
+
+/*
+ * Free a list of 0-order pages
+ */
+void free_hot_cold_page_list(struct list_head *list, bool cold)
+{
+	unsigned long flags;
+
+	if (unlikely(!keventd_up())) {
+		__free_hot_cold_page_list(list, cold);
+		return;
+	}
+
+	spin_lock_irqsave(&free_page_lock, flags);
+	if(cold)
+		list_splice_tail(list, &free_cold_page_list);
+	else
+		list_splice_tail(list, &free_hot_page_list);
+	spin_unlock_irqrestore(&free_page_lock, flags);
+
+	schedule_work(&free_page_work);
+}
+
 /*
  * split_page takes a non-compound higher-order page, and splits it into
  * n (1<<order) sub-pages: page[0..n]
-- 
1.7.10.4

[PATCH 2/2] mm: __free_pages batch up 0-order pages for freeing

[Sasha Levin]

Rather than calling free_hot_cold_page() for every page, batch them up in a
list and pass them on to free_hot_cold_page_list(). This will let us defer
them to a workqueue.

Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
---
 mm/page_alloc.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 812ca75..e58e795 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2997,12 +2997,16 @@ EXPORT_SYMBOL(get_zeroed_page);
 
 void __free_pages(struct page *page, unsigned int order)
 {
+	LIST_HEAD(hot_cold_pages);
+
 	if (put_page_testzero(page)) {
 		if (order == 0)
-			free_hot_cold_page(page, false);
+			list_add(&page->lru, &hot_cold_pages);
 		else
 			__free_pages_ok(page, order);
 	}
+
+	free_hot_cold_page_list(&hot_cold_pages, false);
 }
 
 EXPORT_SYMBOL(__free_pages);
-- 
1.7.10.4

Re: [PATCH 2/2] mm: __free_pages batch up 0-order pages for freeing

[Rasmus Villemoes]

On Wed, Apr 01 2015, Sasha Levin <sasha.levin@oracle.com> wrote:

> Rather than calling free_hot_cold_page() for every page, batch them up in a
> list and pass them on to free_hot_cold_page_list(). This will let us defer
> them to a workqueue.
>
> Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
> ---
>  mm/page_alloc.c |    6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 812ca75..e58e795 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -2997,12 +2997,16 @@ EXPORT_SYMBOL(get_zeroed_page);
>  
>  void __free_pages(struct page *page, unsigned int order)
>  {
> +	LIST_HEAD(hot_cold_pages);
> +
>  	if (put_page_testzero(page)) {
>  		if (order == 0)
> -			free_hot_cold_page(page, false);
> +			list_add(&page->lru, &hot_cold_pages);
>  		else
>  			__free_pages_ok(page, order);
>  	}
> +
> +	free_hot_cold_page_list(&hot_cold_pages, false);

Is there a reason to do this function call when the list is empty? In
other words, why can't this just be done inside the if (order == 0)?

Rasmus

Re: [PATCH 1/2] mm: free large amount of 0-order pages in workqueue

[Andrew Morton]

On Tue, 31 Mar 2015 18:11:32 -0400 Sasha Levin <sasha.levin@oracle.com> wrote:

> Freeing pages became a rather costly operation, specially when multiple debug
> options are enabled. This causes hangs when an attempt to free a large amount
> of 0-order is made. Two examples are vfree()ing large block of memory, and
> punching a hole in a shmem filesystem.
> 
> To avoid that, move any free operations that involve batching pages into a
> list to a workqueue handler where they could be freed later.

eek.

__free_pages() is going to be a hot path for someone - it has 500+
callsites.

And this patch might cause problems for rt_prio() tasks which run for a
long time, starving out the workqueue thread.  And probably other stuff
I didn't think of...

What whacky debug option is actually causing this?  Full-page poisoning?



Stick a cond_resched() in __vunmap() ;)