* [PATCH 1/2] mm: free large amount of 0-order pages in workqueue
@ 2015-03-31 22:11 Sasha Levin
2015-03-31 22:11 ` [PATCH 2/2] mm: __free_pages batch up 0-order pages for freeing Sasha Levin
2015-03-31 22:31 ` [PATCH 1/2] mm: free large amount of 0-order pages in workqueue Andrew Morton
0 siblings, 2 replies; 4+ messages in thread
From: Sasha Levin @ 2015-03-31 22:11 UTC (permalink / raw)
To: linux-kernel
Cc: mhocko, Sasha Levin, Andrew Morton, Mel Gorman, Vlastimil Babka,
Johannes Weiner, David Rientjes, Joonsoo Kim,
open list:MEMORY MANAGEMENT
Freeing pages became a rather costly operation, specially when multiple debug
options are enabled. This causes hangs when an attempt to free a large amount
of 0-order is made. Two examples are vfree()ing large block of memory, and
punching a hole in a shmem filesystem.
To avoid that, move any free operations that involve batching pages into a
list to a workqueue handler where they could be freed later.
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
---
mm/page_alloc.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++----
1 file changed, 46 insertions(+), 4 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 5bd9711..812ca75 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1586,10 +1586,11 @@ out:
local_irq_restore(flags);
}
-/*
- * Free a list of 0-order pages
- */
-void free_hot_cold_page_list(struct list_head *list, bool cold)
+static LIST_HEAD(free_hot_page_list);
+static LIST_HEAD(free_cold_page_list);
+static DEFINE_SPINLOCK(free_page_lock);
+
+static void __free_hot_cold_page_list(struct list_head *list, bool cold)
{
struct page *page, *next;
@@ -1599,6 +1600,47 @@ void free_hot_cold_page_list(struct list_head *list, bool cold)
}
}
+static void free_page_lists_work(struct work_struct *work)
+{
+ LIST_HEAD(hot_pages);
+ LIST_HEAD(cold_pages);
+ unsigned long flags;
+
+ spin_lock_irqsave(&free_page_lock, flags);
+ list_cut_position(&hot_pages, &free_hot_page_list,
+ free_hot_page_list.prev);
+ list_cut_position(&cold_pages, &free_cold_page_list,
+ free_cold_page_list.prev);
+ spin_unlock_irqrestore(&free_page_lock, flags);
+
+ __free_hot_cold_page_list(&hot_pages, false);
+ __free_hot_cold_page_list(&cold_pages, true);
+}
+
+static DECLARE_WORK(free_page_work, free_page_lists_work);
+
+/*
+ * Free a list of 0-order pages
+ */
+void free_hot_cold_page_list(struct list_head *list, bool cold)
+{
+ unsigned long flags;
+
+ if (unlikely(!keventd_up())) {
+ __free_hot_cold_page_list(list, cold);
+ return;
+ }
+
+ spin_lock_irqsave(&free_page_lock, flags);
+ if(cold)
+ list_splice_tail(list, &free_cold_page_list);
+ else
+ list_splice_tail(list, &free_hot_page_list);
+ spin_unlock_irqrestore(&free_page_lock, flags);
+
+ schedule_work(&free_page_work);
+}
+
/*
* split_page takes a non-compound higher-order page, and splits it into
* n (1<<order) sub-pages: page[0..n]
--
1.7.10.4
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH 2/2] mm: __free_pages batch up 0-order pages for freeing
2015-03-31 22:11 [PATCH 1/2] mm: free large amount of 0-order pages in workqueue Sasha Levin
@ 2015-03-31 22:11 ` Sasha Levin
2015-04-01 12:48 ` Rasmus Villemoes
2015-03-31 22:31 ` [PATCH 1/2] mm: free large amount of 0-order pages in workqueue Andrew Morton
1 sibling, 1 reply; 4+ messages in thread
From: Sasha Levin @ 2015-03-31 22:11 UTC (permalink / raw)
To: linux-kernel
Cc: mhocko, Sasha Levin, Andrew Morton, Mel Gorman, Vlastimil Babka,
Johannes Weiner, David Rientjes, Joonsoo Kim,
open list:MEMORY MANAGEMENT
Rather than calling free_hot_cold_page() for every page, batch them up in a
list and pass them on to free_hot_cold_page_list(). This will let us defer
them to a workqueue.
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
---
mm/page_alloc.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 812ca75..e58e795 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2997,12 +2997,16 @@ EXPORT_SYMBOL(get_zeroed_page);
void __free_pages(struct page *page, unsigned int order)
{
+ LIST_HEAD(hot_cold_pages);
+
if (put_page_testzero(page)) {
if (order == 0)
- free_hot_cold_page(page, false);
+ list_add(&page->lru, &hot_cold_pages);
else
__free_pages_ok(page, order);
}
+
+ free_hot_cold_page_list(&hot_cold_pages, false);
}
EXPORT_SYMBOL(__free_pages);
--
1.7.10.4
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH 1/2] mm: free large amount of 0-order pages in workqueue
2015-03-31 22:11 [PATCH 1/2] mm: free large amount of 0-order pages in workqueue Sasha Levin
2015-03-31 22:11 ` [PATCH 2/2] mm: __free_pages batch up 0-order pages for freeing Sasha Levin
@ 2015-03-31 22:31 ` Andrew Morton
1 sibling, 0 replies; 4+ messages in thread
From: Andrew Morton @ 2015-03-31 22:31 UTC (permalink / raw)
To: Sasha Levin
Cc: linux-kernel, mhocko, Mel Gorman, Vlastimil Babka,
Johannes Weiner, David Rientjes, Joonsoo Kim,
open list:MEMORY MANAGEMENT
On Tue, 31 Mar 2015 18:11:32 -0400 Sasha Levin <sasha.levin@oracle.com> wrote:
> Freeing pages became a rather costly operation, specially when multiple debug
> options are enabled. This causes hangs when an attempt to free a large amount
> of 0-order is made. Two examples are vfree()ing large block of memory, and
> punching a hole in a shmem filesystem.
>
> To avoid that, move any free operations that involve batching pages into a
> list to a workqueue handler where they could be freed later.
eek.
__free_pages() is going to be a hot path for someone - it has 500+
callsites.
And this patch might cause problems for rt_prio() tasks which run for a
long time, starving out the workqueue thread. And probably other stuff
I didn't think of...
What whacky debug option is actually causing this? Full-page poisoning?
Stick a cond_resched() in __vunmap() ;)
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH 2/2] mm: __free_pages batch up 0-order pages for freeing
2015-03-31 22:11 ` [PATCH 2/2] mm: __free_pages batch up 0-order pages for freeing Sasha Levin
@ 2015-04-01 12:48 ` Rasmus Villemoes
0 siblings, 0 replies; 4+ messages in thread
From: Rasmus Villemoes @ 2015-04-01 12:48 UTC (permalink / raw)
To: Sasha Levin
Cc: linux-kernel, mhocko, Andrew Morton, Mel Gorman, Vlastimil Babka,
Johannes Weiner, David Rientjes, Joonsoo Kim,
open list:MEMORY MANAGEMENT
On Wed, Apr 01 2015, Sasha Levin <sasha.levin@oracle.com> wrote:
> Rather than calling free_hot_cold_page() for every page, batch them up in a
> list and pass them on to free_hot_cold_page_list(). This will let us defer
> them to a workqueue.
>
> Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
> ---
> mm/page_alloc.c | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 812ca75..e58e795 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -2997,12 +2997,16 @@ EXPORT_SYMBOL(get_zeroed_page);
>
> void __free_pages(struct page *page, unsigned int order)
> {
> + LIST_HEAD(hot_cold_pages);
> +
> if (put_page_testzero(page)) {
> if (order == 0)
> - free_hot_cold_page(page, false);
> + list_add(&page->lru, &hot_cold_pages);
> else
> __free_pages_ok(page, order);
> }
> +
> + free_hot_cold_page_list(&hot_cold_pages, false);
Is there a reason to do this function call when the list is empty? In
other words, why can't this just be done inside the if (order == 0)?
Rasmus
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2015-04-01 12:48 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-03-31 22:11 [PATCH 1/2] mm: free large amount of 0-order pages in workqueue Sasha Levin
2015-03-31 22:11 ` [PATCH 2/2] mm: __free_pages batch up 0-order pages for freeing Sasha Levin
2015-04-01 12:48 ` Rasmus Villemoes
2015-03-31 22:31 ` [PATCH 1/2] mm: free large amount of 0-order pages in workqueue Andrew Morton
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).