linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Marek Szyprowski <m.szyprowski@samsung.com>
To: linux-kernel@vger.kernel.org,
	linux-arm-kernel@lists.infradead.org,
	linux-media@vger.kernel.org, linux-mm@kvack.org,
	linaro-mm-sig@lists.linaro.org
Cc: Michal Nazarewicz <mina86@mina86.com>,
	Marek Szyprowski <m.szyprowski@samsung.com>,
	Kyungmin Park <kyungmin.park@samsung.com>,
	Russell King <linux@arm.linux.org.uk>,
	Andrew Morton <akpm@linux-foundation.org>,
	KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>,
	Daniel Walker <dwalker@codeaurora.org>,
	Mel Gorman <mel@csn.ul.ie>, Arnd Bergmann <arnd@arndb.de>,
	Jesse Barker <jesse.barker@linaro.org>,
	Jonathan Corbet <corbet@lwn.net>,
	Shariq Hasnain <shariq.hasnain@linaro.org>,
	Chunsang Jeong <chunsang.jeong@linaro.org>,
	Dave Hansen <dave@linux.vnet.ibm.com>,
	Benjamin Gaignard <benjamin.gaignard@linaro.org>
Subject: [PATCH 06/15] mm: page_alloc: introduce alloc_contig_range()
Date: Thu, 26 Jan 2012 10:00:48 +0100	[thread overview]
Message-ID: <1327568457-27734-7-git-send-email-m.szyprowski@samsung.com> (raw)
In-Reply-To: <1327568457-27734-1-git-send-email-m.szyprowski@samsung.com>

From: Michal Nazarewicz <mina86@mina86.com>

This commit adds the alloc_contig_range() function which tries
to allocate given range of pages.  It tries to migrate all
already allocated pages that fall in the range thus freeing them.
Once all pages in the range are freed they are removed from the
buddy system thus allocated for the caller to use.

Signed-off-by: Michal Nazarewicz <mina86@mina86.com>
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
---
 include/linux/page-isolation.h |    7 ++
 mm/page_alloc.c                |  183 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 190 insertions(+), 0 deletions(-)

diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h
index 8c02c2b..430cf61 100644
--- a/include/linux/page-isolation.h
+++ b/include/linux/page-isolation.h
@@ -39,5 +39,12 @@ extern void update_pcp_isolate_block(unsigned long pfn);
 extern int set_migratetype_isolate(struct page *page);
 extern void unset_migratetype_isolate(struct page *page);
 
+#ifdef CONFIG_CMA
+
+/* The below functions must be run on a range from a single zone. */
+extern int alloc_contig_range(unsigned long start, unsigned long end);
+extern void free_contig_range(unsigned long pfn, unsigned nr_pages);
+
+#endif
 
 #endif
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 70709e7..b4f50532 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -57,6 +57,7 @@
 #include <linux/ftrace_event.h>
 #include <linux/memcontrol.h>
 #include <linux/prefetch.h>
+#include <linux/migrate.h>
 #include <linux/page-debug-flags.h>
 
 #include <asm/tlbflush.h>
@@ -5488,6 +5489,188 @@ out:
 	spin_unlock_irqrestore(&zone->lock, flags);
 }
 
+#ifdef CONFIG_CMA
+
+static unsigned long pfn_align_to_maxpage_down(unsigned long pfn)
+{
+	return pfn & ~(MAX_ORDER_NR_PAGES - 1);
+}
+
+static unsigned long pfn_align_to_maxpage_up(unsigned long pfn)
+{
+	return ALIGN(pfn, MAX_ORDER_NR_PAGES);
+}
+
+static struct page *
+__alloc_contig_migrate_alloc(struct page *page, unsigned long private,
+			     int **resultp)
+{
+	return alloc_page(GFP_HIGHUSER_MOVABLE);
+}
+
+/* [start, end) must belong to a single zone. */
+static int __alloc_contig_migrate_range(unsigned long start, unsigned long end)
+{
+	/* This function is based on compact_zone() from compaction.c. */
+
+	unsigned long pfn = start;
+	unsigned int tries = 0;
+	int ret = 0;
+
+	struct compact_control cc = {
+		.nr_migratepages = 0,
+		.order = -1,
+		.zone = page_zone(pfn_to_page(start)),
+		.sync = true,
+	};
+	INIT_LIST_HEAD(&cc.migratepages);
+
+	migrate_prep_local();
+
+	while (pfn < end || !list_empty(&cc.migratepages)) {
+		if (fatal_signal_pending(current)) {
+			ret = -EINTR;
+			break;
+		}
+
+		if (list_empty(&cc.migratepages)) {
+			cc.nr_migratepages = 0;
+			pfn = isolate_migratepages_range(cc.zone, &cc,
+							 pfn, end);
+			if (!pfn) {
+				ret = -EINTR;
+				break;
+			}
+			tries = 0;
+		} else if (++tries == 5) {
+			ret = ret < 0 ? ret : -EBUSY;
+			break;
+		}
+
+		ret = migrate_pages(&cc.migratepages,
+				    __alloc_contig_migrate_alloc,
+				    0, false, true);
+	}
+
+	putback_lru_pages(&cc.migratepages);
+	return ret;
+}
+
+/**
+ * alloc_contig_range() -- tries to allocate given range of pages
+ * @start:	start PFN to allocate
+ * @end:	one-past-the-last PFN to allocate
+ *
+ * The PFN range does not have to be pageblock or MAX_ORDER_NR_PAGES
+ * aligned, however it's the caller's responsibility to guarantee that
+ * we are the only thread that changes migrate type of pageblocks the
+ * pages fall in.
+ *
+ * The PFN range must belong to a single zone.
+ *
+ * Returns zero on success or negative error code.  On success all
+ * pages which PFN is in [start, end) are allocated for the caller and
+ * need to be freed with free_contig_range().
+ */
+int alloc_contig_range(unsigned long start, unsigned long end)
+{
+	unsigned long outer_start, outer_end;
+	int ret = 0, order;
+
+	/*
+	 * What we do here is we mark all pageblocks in range as
+	 * MIGRATE_ISOLATE.  Because of the way page allocator work, we
+	 * align the range to MAX_ORDER pages so that page allocator
+	 * won't try to merge buddies from different pageblocks and
+	 * change MIGRATE_ISOLATE to some other migration type.
+	 *
+	 * Once the pageblocks are marked as MIGRATE_ISOLATE, we
+	 * migrate the pages from an unaligned range (ie. pages that
+	 * we are interested in).  This will put all the pages in
+	 * range back to page allocator as MIGRATE_ISOLATE.
+	 *
+	 * When this is done, we take the pages in range from page
+	 * allocator removing them from the buddy system.  This way
+	 * page allocator will never consider using them.
+	 *
+	 * This lets us mark the pageblocks back as
+	 * MIGRATE_CMA/MIGRATE_MOVABLE so that free pages in the
+	 * MAX_ORDER aligned range but not in the unaligned, original
+	 * range are put back to page allocator so that buddy can use
+	 * them.
+	 */
+
+	ret = start_isolate_page_range(pfn_align_to_maxpage_down(start),
+				       pfn_align_to_maxpage_up(end));
+	if (ret)
+		goto done;
+
+	ret = __alloc_contig_migrate_range(start, end);
+	if (ret)
+		goto done;
+
+	/*
+	 * Pages from [start, end) are within a MAX_ORDER_NR_PAGES
+	 * aligned blocks that are marked as MIGRATE_ISOLATE.  What's
+	 * more, all pages in [start, end) are free in page allocator.
+	 * What we are going to do is to allocate all pages from
+	 * [start, end) (that is remove them from page allocater).
+	 *
+	 * The only problem is that pages at the beginning and at the
+	 * end of interesting range may be not aligned with pages that
+	 * page allocator holds, ie. they can be part of higher order
+	 * pages.  Because of this, we reserve the bigger range and
+	 * once this is done free the pages we are not interested in.
+	 */
+
+	lru_add_drain_all();
+	drain_all_pages();
+
+	order = 0;
+	outer_start = start;
+	while (!PageBuddy(pfn_to_page(outer_start))) {
+		if (WARN_ON(++order >= MAX_ORDER)) {
+			ret = -EINVAL;
+			goto done;
+		}
+		outer_start &= ~0UL << order;
+	}
+
+	/* Make sure the range is really isolated. */
+	if (test_pages_isolated(outer_start, end)) {
+		pr_warn("__alloc_contig_migrate_range: test_pages_isolated(%lx, %lx) failed\n",
+		       outer_start, end);
+		ret = -EBUSY;
+		goto done;
+	}
+
+	outer_end = isolate_freepages_range(outer_start, end);
+	if (!outer_end) {
+		ret = -EBUSY;
+		goto done;
+	}
+
+	/* Free head and tail (if any) */
+	if (start != outer_start)
+		free_contig_range(outer_start, start - outer_start);
+	if (end != outer_end)
+		free_contig_range(end, outer_end - end);
+
+done:
+	undo_isolate_page_range(pfn_align_to_maxpage_down(start),
+				pfn_align_to_maxpage_up(end));
+	return ret;
+}
+
+void free_contig_range(unsigned long pfn, unsigned nr_pages)
+{
+	for (; nr_pages--; ++pfn)
+		__free_page(pfn_to_page(pfn));
+}
+
+#endif
+
+
 #ifdef CONFIG_MEMORY_HOTREMOVE
 /*
  * All pages in the range must be isolated before calling this.
-- 
1.7.1.569.g6f426


  parent reply	other threads:[~2012-01-26  9:10 UTC|newest]

Thread overview: 62+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-01-26  9:00 [PATCHv19 00/15] Contiguous Memory Allocator Marek Szyprowski
2012-01-26  9:00 ` [PATCH 01/15] mm: page_alloc: remove trailing whitespace Marek Szyprowski
2012-01-30 10:59   ` Mel Gorman
2012-01-26  9:00 ` [PATCH 02/15] mm: page_alloc: update migrate type of pages on pcp when isolating Marek Szyprowski
2012-01-30 11:15   ` Mel Gorman
2012-01-30 15:41     ` Michal Nazarewicz
2012-01-30 16:14       ` Mel Gorman
2012-01-31 16:23         ` Marek Szyprowski
2012-02-02 12:47           ` Mel Gorman
2012-02-02 19:53             ` Michal Nazarewicz
2012-02-03  9:31               ` Marek Szyprowski
2012-02-03 11:27               ` Mel Gorman
2012-01-26  9:00 ` [PATCH 03/15] mm: compaction: introduce isolate_migratepages_range() Marek Szyprowski
2012-01-30 11:24   ` Mel Gorman
2012-01-30 12:42     ` Michal Nazarewicz
2012-01-30 13:25       ` Mel Gorman
2012-01-26  9:00 ` [PATCH 04/15] mm: compaction: introduce isolate_freepages_range() Marek Szyprowski
2012-01-30 11:48   ` Mel Gorman
2012-01-30 11:55     ` Mel Gorman
2012-01-26  9:00 ` [PATCH 05/15] mm: compaction: export some of the functions Marek Szyprowski
2012-01-30 11:57   ` Mel Gorman
2012-01-30 12:33     ` Michal Nazarewicz
2012-01-26  9:00 ` Marek Szyprowski [this message]
2012-01-30 12:11   ` [PATCH 06/15] mm: page_alloc: introduce alloc_contig_range() Mel Gorman
2012-01-26  9:00 ` [PATCH 07/15] mm: page_alloc: change fallbacks array handling Marek Szyprowski
2012-01-30 12:12   ` Mel Gorman
2012-01-26  9:00 ` [PATCH 08/15] mm: mmzone: MIGRATE_CMA migration type added Marek Szyprowski
2012-01-30 12:35   ` Mel Gorman
2012-01-30 13:06     ` Michal Nazarewicz
2012-01-30 14:52       ` Mel Gorman
2012-01-26  9:00 ` [PATCH 09/15] mm: page_isolation: MIGRATE_CMA isolation functions added Marek Szyprowski
2012-01-26  9:00 ` [PATCH 10/15] mm: extract reclaim code from __alloc_pages_direct_reclaim() Marek Szyprowski
2012-01-30 12:42   ` Mel Gorman
2012-01-26  9:00 ` [PATCH 11/15] mm: trigger page reclaim in alloc_contig_range() to stabilize watermarks Marek Szyprowski
2012-01-30 13:05   ` Mel Gorman
2012-01-31 17:15     ` Marek Szyprowski
2012-01-26  9:00 ` [PATCH 12/15] drivers: add Contiguous Memory Allocator Marek Szyprowski
2012-01-27  9:44   ` [Linaro-mm-sig] " Ohad Ben-Cohen
2012-01-27 10:53     ` Marek Szyprowski
2012-01-27 14:27       ` Clark, Rob
2012-01-27 14:51         ` Marek Szyprowski
2012-01-27 14:59           ` Ohad Ben-Cohen
2012-01-27 15:17             ` Marek Szyprowski
2012-01-28 18:57               ` Ohad Ben-Cohen
2012-01-30  7:43                 ` Marek Szyprowski
2012-01-30  9:16                   ` Ohad Ben-Cohen
2012-01-27 14:56       ` Ohad Ben-Cohen
2012-01-26  9:00 ` [PATCH 13/15] X86: integrate CMA with DMA-mapping subsystem Marek Szyprowski
2012-01-26  9:00 ` [PATCH 14/15] ARM: " Marek Szyprowski
2012-01-26  9:00 ` [PATCH 15/15] ARM: Samsung: use CMA for 2 memory banks for s5p-mfc device Marek Szyprowski
2012-01-26 15:31 ` [PATCHv19 00/15] Contiguous Memory Allocator Arnd Bergmann
2012-01-26 15:38   ` Michal Nazarewicz
2012-01-26 15:48   ` Marek Szyprowski
2012-01-28  0:26   ` Andrew Morton
2012-01-29 18:09     ` Rob Clark
2012-01-29 20:32       ` Anca Emanuel
2012-01-29 20:51     ` Arnd Bergmann
2012-01-30 13:25     ` Mel Gorman
2012-01-30 15:43       ` Michal Nazarewicz
     [not found]         ` <CA+M3ks7h1t6DbPSAhPN6LJ5Dw84hSukfWG16avh2eZL+o4caJg@mail.gmail.com>
2012-02-01  8:47           ` Marek Szyprowski
2012-02-10 18:10     ` Marek Szyprowski
2012-02-03 12:18 [PATCHv20 " Marek Szyprowski
2012-02-03 12:18 ` [PATCH 06/15] mm: page_alloc: introduce alloc_contig_range() Marek Szyprowski

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1327568457-27734-7-git-send-email-m.szyprowski@samsung.com \
    --to=m.szyprowski@samsung.com \
    --cc=akpm@linux-foundation.org \
    --cc=arnd@arndb.de \
    --cc=benjamin.gaignard@linaro.org \
    --cc=chunsang.jeong@linaro.org \
    --cc=corbet@lwn.net \
    --cc=dave@linux.vnet.ibm.com \
    --cc=dwalker@codeaurora.org \
    --cc=jesse.barker@linaro.org \
    --cc=kamezawa.hiroyu@jp.fujitsu.com \
    --cc=kyungmin.park@samsung.com \
    --cc=linaro-mm-sig@lists.linaro.org \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-media@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux@arm.linux.org.uk \
    --cc=mel@csn.ul.ie \
    --cc=mina86@mina86.com \
    --cc=shariq.hasnain@linaro.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).