[PATCH 0/6] Enable parallel page migration

[PATCH 0/6] Enable parallel page migration

[Anshuman Khandual]

	This patch series is base on the work posted by Zi Yan back in
November 2016 (https://lkml.org/lkml/2016/11/22/457) but includes some
amount clean up and re-organization. This series depends on THP migration
optimization patch series posted by Naoya Horiguchi on 8th November 2016
(https://lwn.net/Articles/705879/). Though Zi Yan has recently reposted
V3 of the THP migration patch series (https://lwn.net/Articles/713667/),
this series is yet to be rebased.

	Primary motivation behind this patch series is to achieve higher
bandwidth of memory migration when ever possible using multi threaded
instead of a single threaded copy. Did all the experiments using a two
socket X86 sytsem (Intel(R) Xeon(R) CPU E5-2650). All the experiments
here have same allocation size 4K * 100000 (which did not split evenly
for the 2MB huge pages). Here are the results.

Vanilla:

Moved 100000 normal pages in 247.000000 msecs 1.544412 GBs
Moved 100000 normal pages in 238.000000 msecs 1.602814 GBs
Moved 195 huge pages in 252.000000 msecs 1.513769 GBs
Moved 195 huge pages in 257.000000 msecs 1.484318 GBs

THP migration improvements:

Moved 100000 normal pages in 302.000000 msecs 1.263145 GBs
Moved 100000 normal pages in 262.000000 msecs 1.455991 GBs
Moved 195 huge pages in 120.000000 msecs 3.178914 GBs
Moved 195 huge pages in 129.000000 msecs 2.957130 GBs

THP migration improvements + Multi threaded page copy:

Moved 100000 normal pages in 1589.000000 msecs 0.240069 GBs **
Moved 100000 normal pages in 1932.000000 msecs 0.197448 GBs **
Moved 195 huge pages in 54.000000 msecs 7.064254 GBs ***
Moved 195 huge pages in 86.000000 msecs 4.435694 GBs ***


**      Using multi threaded copy can be detrimental to performance if
	used for regular pages which are way too small. But then the
	framework provides the means to use it if some kernel/driver
	caller or user application wants to use it.

***     These applications have used the new MPOL_MF_MOVE_MT flag while
	calling the system calls like mbind() and move_pages().

On POWER8 the improvements are similar when tested with a draft patch
which enables migration at PMD level. Not putting out the results here
as the kernel is not stable with the that draft patch and crashes some
times. We are working on enabling PMD level migration on POWER8 and will
test this series out thoroughly when its ready.

Patch Series Description::

Patch 1: Add new parameter to migrate_page_copy and copy_huge_page so
	 that it can differentiate between when to use single threaded
	 version (MIGRATE_ST) or multi threaded version (MIGRATE_MT).

Patch 2: Make migrate_mode types non-exclusive.

Patch 3: Add copy_pages_mthread function which does the actual multi
	 threaded copy. This involves splitting the copy work into
	 chunks, selecting threads and submitting copy jobs in the
	 work queues.

Patch 4: Add new migrate mode MIGRATE_MT to be used by higher level
	 migration functions.

Patch 5: Add new migration flag MPOL_MF_MOVE_MT for migration system
	 calls to be used in the user space.

Patch 6: Define global mt_page_copy tunable which turns on the multi
	 threaded page copy no matter what for all migrations on the
	 system.

Outstanding Issues::

Issue 1: The usefulness of the global multi threaded copy tunable i.e
	 vm.mt_page_copy. It makes sense and helps in validating the
	 framework. Should this be moved to debugfs instead ?

Issue 2: We choose nr_copythreads = 8 as maximum number of threads on
	 a node can be 8 on any architecture (Which is on POWER8 if
	 I am not missing any other arch which might have equal or
	 more number of threads per node). It just denotes max number
	 of threads and we will be adjusted based on cpumask_weight
	 value on destination node. Can we do better, suggestions ?

Issue 3: Multi threaded page migration works best with threads allocated
	 at different physical cores, not all in the same hyper-threaded
	 core. Work queues submitted jobs consume scheduler slots from
	 the given thread to execute the copy. This can interfere with
	 scheduling and affect some already running tasks on the system.
	 Should we be looking into arch topology information, scheduler
	 cpu idle details to decide on which threads to use before going
	 for multi threaded copy ? Abort multi threaded copy and fallback
	 to regular copy at times when the parameters are not good ?

Any comments, suggestions are welcome.

Zi Yan (6):
  mm/migrate: Add new mode parameter to migrate_page_copy() function
  mm/migrate: Make migrate_mode types non-exclussive
  mm/migrate: Add copy_pages_mthread function
  mm/migrate: Add new migrate mode MIGRATE_MT
  mm/migrate: Add new migration flag MPOL_MF_MOVE_MT for syscalls
  sysctl: Add global tunable mt_page_copy

 fs/aio.c                       |  2 +-
 fs/f2fs/data.c                 |  2 +-
 fs/hugetlbfs/inode.c           |  2 +-
 fs/ubifs/file.c                |  2 +-
 include/linux/highmem.h        |  2 +
 include/linux/migrate.h        |  6 ++-
 include/linux/migrate_mode.h   |  8 ++--
 include/uapi/linux/mempolicy.h |  4 +-
 kernel/sysctl.c                | 10 +++++
 mm/Makefile                    |  2 +
 mm/compaction.c                | 20 +++++-----
 mm/copy_pages_mthread.c        | 87 ++++++++++++++++++++++++++++++++++++++++++
 mm/mempolicy.c                 |  7 +++-
 mm/migrate.c                   | 81 +++++++++++++++++++++++++++------------
 14 files changed, 190 insertions(+), 45 deletions(-)
 create mode 100644 mm/copy_pages_mthread.c

-- 
2.9.3

[PATCH 1/6] mm/migrate: Add new mode parameter to migrate_page_copy() function

[Anshuman Khandual]

From: Zi Yan <ziy@nvidia.com>

This is a prerequisite change required to make page migration framewok
copy in different modes like the default single threaded or the new
multi threaded one yet to be introduced in follow up patches. This
does not change any existing functionality. Only migrate_page_copy()
and copy_huge_page() function's signatures are affected.

Signed-off-by: Zi Yan <zi.yan@cs.rutgers.edu>
Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
 fs/aio.c                     |  2 +-
 fs/f2fs/data.c               |  2 +-
 fs/hugetlbfs/inode.c         |  2 +-
 fs/ubifs/file.c              |  2 +-
 include/linux/migrate.h      |  6 ++++--
 include/linux/migrate_mode.h |  1 +
 mm/migrate.c                 | 14 ++++++++------
 7 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/fs/aio.c b/fs/aio.c
index 873b4ca..ba3f6eb 100644
--- a/fs/aio.c
+++ b/fs/aio.c
@@ -418,7 +418,7 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,
 	 * events from being lost.
 	 */
 	spin_lock_irqsave(&ctx->completion_lock, flags);
-	migrate_page_copy(new, old);
+	migrate_page_copy(new, old, MIGRATE_ST);
 	BUG_ON(ctx->ring_pages[idx] != old);
 	ctx->ring_pages[idx] = new;
 	spin_unlock_irqrestore(&ctx->completion_lock, flags);
diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index 9ac2625..ad41356 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -1997,7 +1997,7 @@ int f2fs_migrate_page(struct address_space *mapping,
 		SetPagePrivate(newpage);
 	set_page_private(newpage, page_private(page));
 
-	migrate_page_copy(newpage, page);
+	migrate_page_copy(newpage, page, MIGRATE_ST);
 
 	return MIGRATEPAGE_SUCCESS;
 }
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 54de77e..0e16512f 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -850,7 +850,7 @@ static int hugetlbfs_migrate_page(struct address_space *mapping,
 	rc = migrate_huge_page_move_mapping(mapping, newpage, page);
 	if (rc != MIGRATEPAGE_SUCCESS)
 		return rc;
-	migrate_page_copy(newpage, page);
+	migrate_page_copy(newpage, page, MIGRATE_ST);
 
 	return MIGRATEPAGE_SUCCESS;
 }
diff --git a/fs/ubifs/file.c b/fs/ubifs/file.c
index b0d7837..293616f 100644
--- a/fs/ubifs/file.c
+++ b/fs/ubifs/file.c
@@ -1482,7 +1482,7 @@ static int ubifs_migrate_page(struct address_space *mapping,
 		SetPagePrivate(newpage);
 	}
 
-	migrate_page_copy(newpage, page);
+	migrate_page_copy(newpage, page, MIGRATE_ST);
 	return MIGRATEPAGE_SUCCESS;
 }
 #endif
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index ae8d475..d843b8f 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -42,7 +42,8 @@ extern void putback_movable_page(struct page *page);
 
 extern int migrate_prep(void);
 extern int migrate_prep_local(void);
-extern void migrate_page_copy(struct page *newpage, struct page *page);
+extern void migrate_page_copy(struct page *newpage, struct page *page,
+			enum migrate_mode mode);
 extern int migrate_huge_page_move_mapping(struct address_space *mapping,
 				  struct page *newpage, struct page *page);
 extern int migrate_page_move_mapping(struct address_space *mapping,
@@ -61,7 +62,8 @@ static inline int migrate_prep(void) { return -ENOSYS; }
 static inline int migrate_prep_local(void) { return -ENOSYS; }
 
 static inline void migrate_page_copy(struct page *newpage,
-				     struct page *page) {}
+				     struct page *page,
+				     enum migrate_mode mode) {}
 
 static inline int migrate_huge_page_move_mapping(struct address_space *mapping,
 				  struct page *newpage, struct page *page)
diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h
index ebf3d89..b3b9acb 100644
--- a/include/linux/migrate_mode.h
+++ b/include/linux/migrate_mode.h
@@ -11,6 +11,7 @@ enum migrate_mode {
 	MIGRATE_ASYNC,
 	MIGRATE_SYNC_LIGHT,
 	MIGRATE_SYNC,
+	MIGRATE_ST
 };
 
 #endif		/* MIGRATE_MODE_H_INCLUDED */
diff --git a/mm/migrate.c b/mm/migrate.c
index 87f4d0f..13fa938 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -589,7 +589,8 @@ static void __copy_gigantic_page(struct page *dst, struct page *src,
 	}
 }
 
-static void copy_huge_page(struct page *dst, struct page *src)
+static void copy_huge_page(struct page *dst, struct page *src,
+				enum migrate_mode mode)
 {
 	int i;
 	int nr_pages;
@@ -618,12 +619,13 @@ static void copy_huge_page(struct page *dst, struct page *src)
 /*
  * Copy the page to its new location
  */
-void migrate_page_copy(struct page *newpage, struct page *page)
+void migrate_page_copy(struct page *newpage, struct page *page,
+					   enum migrate_mode mode)
 {
 	int cpupid;
 
 	if (PageHuge(page) || PageTransHuge(page))
-		copy_huge_page(newpage, page);
+		copy_huge_page(newpage, page, mode);
 	else
 		copy_highpage(newpage, page);
 
@@ -705,7 +707,7 @@ int migrate_page(struct address_space *mapping,
 	if (rc != MIGRATEPAGE_SUCCESS)
 		return rc;
 
-	migrate_page_copy(newpage, page);
+	migrate_page_copy(newpage, page, mode);
 	return MIGRATEPAGE_SUCCESS;
 }
 EXPORT_SYMBOL(migrate_page);
@@ -755,7 +757,7 @@ int buffer_migrate_page(struct address_space *mapping,
 
 	SetPagePrivate(newpage);
 
-	migrate_page_copy(newpage, page);
+	migrate_page_copy(newpage, page, MIGRATE_ST);
 
 	bh = head;
 	do {
@@ -1968,7 +1970,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
 	/* anon mapping, we can simply copy page->mapping to the new page: */
 	new_page->mapping = page->mapping;
 	new_page->index = page->index;
-	migrate_page_copy(new_page, page);
+	migrate_page_copy(new_page, page, MIGRATE_ST);
 	WARN_ON(PageLRU(new_page));
 
 	/* Recheck the target PMD */
-- 
2.9.3

[PATCH 2/6] mm/migrate: Make migrate_mode types non-exclusive

[Anshuman Khandual]

From: Zi Yan <ziy@nvidia.com>

It basically changes the enum declaration from numbers to bit positions
so that they can be used in combination which was not the case earlier.
No functionality has been changed.

Signed-off-by: Zi Yan <zi.yan@cs.rutgers.edu>
Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
 include/linux/migrate_mode.h |  8 ++++----
 mm/compaction.c              | 20 ++++++++++----------
 mm/migrate.c                 | 14 +++++++-------
 3 files changed, 21 insertions(+), 21 deletions(-)

diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h
index b3b9acb..89c1700 100644
--- a/include/linux/migrate_mode.h
+++ b/include/linux/migrate_mode.h
@@ -8,10 +8,10 @@
  * MIGRATE_SYNC will block when migrating pages
  */
 enum migrate_mode {
-	MIGRATE_ASYNC,
-	MIGRATE_SYNC_LIGHT,
-	MIGRATE_SYNC,
-	MIGRATE_ST
+	MIGRATE_ASYNC		= 1<<0,
+	MIGRATE_SYNC_LIGHT	= 1<<1,
+	MIGRATE_SYNC		= 1<<2,
+	MIGRATE_ST		= 1<<3,
 };
 
 #endif		/* MIGRATE_MODE_H_INCLUDED */
diff --git a/mm/compaction.c b/mm/compaction.c
index 949198d..1a481af 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -296,7 +296,7 @@ static void update_pageblock_skip(struct compact_control *cc,
 	if (migrate_scanner) {
 		if (pfn > zone->compact_cached_migrate_pfn[0])
 			zone->compact_cached_migrate_pfn[0] = pfn;
-		if (cc->mode != MIGRATE_ASYNC &&
+		if (!(cc->mode & MIGRATE_ASYNC) &&
 		    pfn > zone->compact_cached_migrate_pfn[1])
 			zone->compact_cached_migrate_pfn[1] = pfn;
 	} else {
@@ -329,7 +329,7 @@ static void update_pageblock_skip(struct compact_control *cc,
 static bool compact_trylock_irqsave(spinlock_t *lock, unsigned long *flags,
 						struct compact_control *cc)
 {
-	if (cc->mode == MIGRATE_ASYNC) {
+	if (cc->mode & MIGRATE_ASYNC) {
 		if (!spin_trylock_irqsave(lock, *flags)) {
 			cc->contended = true;
 			return false;
@@ -370,7 +370,7 @@ static bool compact_unlock_should_abort(spinlock_t *lock,
 	}
 
 	if (need_resched()) {
-		if (cc->mode == MIGRATE_ASYNC) {
+		if (cc->mode & MIGRATE_ASYNC) {
 			cc->contended = true;
 			return true;
 		}
@@ -393,7 +393,7 @@ static inline bool compact_should_abort(struct compact_control *cc)
 {
 	/* async compaction aborts if contended */
 	if (need_resched()) {
-		if (cc->mode == MIGRATE_ASYNC) {
+		if (cc->mode & MIGRATE_ASYNC) {
 			cc->contended = true;
 			return true;
 		}
@@ -688,7 +688,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 	 */
 	while (unlikely(too_many_isolated(zone))) {
 		/* async migration should just abort */
-		if (cc->mode == MIGRATE_ASYNC)
+		if (cc->mode & MIGRATE_ASYNC)
 			return 0;
 
 		congestion_wait(BLK_RW_ASYNC, HZ/10);
@@ -700,7 +700,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 	if (compact_should_abort(cc))
 		return 0;
 
-	if (cc->direct_compaction && (cc->mode == MIGRATE_ASYNC)) {
+	if (cc->direct_compaction && (cc->mode & MIGRATE_ASYNC)) {
 		skip_on_failure = true;
 		next_skip_pfn = block_end_pfn(low_pfn, cc->order);
 	}
@@ -1195,7 +1195,7 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone,
 	struct page *page;
 	const isolate_mode_t isolate_mode =
 		(sysctl_compact_unevictable_allowed ? ISOLATE_UNEVICTABLE : 0) |
-		(cc->mode != MIGRATE_SYNC ? ISOLATE_ASYNC_MIGRATE : 0);
+		(!(cc->mode & MIGRATE_SYNC) ? ISOLATE_ASYNC_MIGRATE : 0);
 
 	/*
 	 * Start at where we last stopped, or beginning of the zone as
@@ -1241,7 +1241,7 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone,
 		 * Async compaction is optimistic to see if the minimum amount
 		 * of work satisfies the allocation.
 		 */
-		if (cc->mode == MIGRATE_ASYNC &&
+		if ((cc->mode & MIGRATE_ASYNC) &&
 		    !migrate_async_suitable(get_pageblock_migratetype(page)))
 			continue;
 
@@ -1481,7 +1481,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro
 	unsigned long start_pfn = zone->zone_start_pfn;
 	unsigned long end_pfn = zone_end_pfn(zone);
 	const int migratetype = gfpflags_to_migratetype(cc->gfp_mask);
-	const bool sync = cc->mode != MIGRATE_ASYNC;
+	const bool sync = !(cc->mode & MIGRATE_ASYNC);
 
 	ret = compaction_suitable(zone, cc->order, cc->alloc_flags,
 							cc->classzone_idx);
@@ -1577,7 +1577,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro
 			 * order-aligned block, so skip the rest of it.
 			 */
 			if (cc->direct_compaction &&
-						(cc->mode == MIGRATE_ASYNC)) {
+						(cc->mode & MIGRATE_ASYNC)) {
 				cc->migrate_pfn = block_end_pfn(
 						cc->migrate_pfn - 1, cc->order);
 				/* Draining pcplists is useless in this case */
diff --git a/mm/migrate.c b/mm/migrate.c
index 13fa938..63c3682 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -352,7 +352,7 @@ static bool buffer_migrate_lock_buffers(struct buffer_head *head,
 	struct buffer_head *bh = head;
 
 	/* Simple case, sync compaction */
-	if (mode != MIGRATE_ASYNC) {
+	if (!(mode & MIGRATE_ASYNC)) {
 		do {
 			get_bh(bh);
 			lock_buffer(bh);
@@ -453,7 +453,7 @@ int migrate_page_move_mapping(struct address_space *mapping,
 	 * the mapping back due to an elevated page count, we would have to
 	 * block waiting on other references to be dropped.
 	 */
-	if (mode == MIGRATE_ASYNC && head &&
+	if ((mode & MIGRATE_ASYNC) && head &&
 			!buffer_migrate_lock_buffers(head, mode)) {
 		page_ref_unfreeze(page, expected_count);
 		spin_unlock_irq(&mapping->tree_lock);
@@ -739,7 +739,7 @@ int buffer_migrate_page(struct address_space *mapping,
 	 * with an IRQ-safe spinlock held. In the sync case, the buffers
 	 * need to be locked now
 	 */
-	if (mode != MIGRATE_ASYNC)
+	if (!(mode & MIGRATE_ASYNC))
 		BUG_ON(!buffer_migrate_lock_buffers(head, mode));
 
 	ClearPagePrivate(page);
@@ -821,7 +821,7 @@ static int fallback_migrate_page(struct address_space *mapping,
 {
 	if (PageDirty(page)) {
 		/* Only writeback pages in full synchronous migration */
-		if (mode != MIGRATE_SYNC)
+		if (!(mode & MIGRATE_SYNC))
 			return -EBUSY;
 		return writeout(mapping, page);
 	}
@@ -930,7 +930,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
 	bool is_lru = !__PageMovable(page);
 
 	if (!trylock_page(page)) {
-		if (!force || mode == MIGRATE_ASYNC)
+		if (!force || (mode & MIGRATE_ASYNC))
 			goto out;
 
 		/*
@@ -959,7 +959,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
 		 * the retry loop is too short and in the sync-light case,
 		 * the overhead of stalling is too much
 		 */
-		if (mode != MIGRATE_SYNC) {
+		if (!(mode & MIGRATE_SYNC)) {
 			rc = -EBUSY;
 			goto out_unlock;
 		}
@@ -1229,7 +1229,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page,
 		return -ENOMEM;
 
 	if (!trylock_page(hpage)) {
-		if (!force || mode != MIGRATE_SYNC)
+		if (!force || !(mode & MIGRATE_SYNC))
 			goto out;
 		lock_page(hpage);
 	}
-- 
2.9.3

[PATCH 3/6] mm/migrate: Add copy_pages_mthread function

[Anshuman Khandual]

From: Zi Yan <ziy@nvidia.com>

This change adds a new function copy_pages_mthread to enable multi threaded
page copy which can be utilized during migration. This function splits the
page copy request into multiple threads which will handle individual chunk
and send them as jobs to system_highpri_wq work queue.

Signed-off-by: Zi Yan <zi.yan@cs.rutgers.edu>
Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
 include/linux/highmem.h |  2 ++
 mm/Makefile             |  2 ++
 mm/copy_pages_mthread.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 91 insertions(+)
 create mode 100644 mm/copy_pages_mthread.c

diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index bb3f329..e1f4f1b 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -236,6 +236,8 @@ static inline void copy_user_highpage(struct page *to, struct page *from,
 
 #endif
 
+int copy_pages_mthread(struct page *to, struct page *from, int nr_pages);
+
 static inline void copy_highpage(struct page *to, struct page *from)
 {
 	char *vfrom, *vto;
diff --git a/mm/Makefile b/mm/Makefile
index 295bd7a..cc27e76 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -41,6 +41,8 @@ obj-y			:= filemap.o mempool.o oom_kill.o \
 
 obj-y += init-mm.o
 
+obj-y += copy_pages_mthread.o
+
 ifdef CONFIG_NO_BOOTMEM
 	obj-y		+= nobootmem.o
 else
diff --git a/mm/copy_pages_mthread.c b/mm/copy_pages_mthread.c
new file mode 100644
index 0000000..9ad2ef6
--- /dev/null
+++ b/mm/copy_pages_mthread.c
@@ -0,0 +1,87 @@
+/*
+ * This implements parallel page copy function through multi
+ * threaded work queues.
+ *
+ * Copyright (C) Zi Yan <ziy@nvidia.com>, Nov 2016
+ *
+ * Licensed under the terms of the GNU GPL, version 2.
+ */
+#include <linux/highmem.h>
+#include <linux/workqueue.h>
+#include <linux/slab.h>
+#include <linux/freezer.h>
+
+/*
+ * nr_copythreads can be the highest number of threads for given
+ * node on any architecture. The actual number of copy threads
+ * will be limited by the cpumask weight of the target node.
+ */
+unsigned int nr_copythreads = 8;
+
+struct copy_info {
+	struct work_struct copy_work;
+	char *to;
+	char *from;
+	unsigned long chunk_size;
+};
+
+static void copy_pages(char *vto, char *vfrom, unsigned long size)
+{
+	memcpy(vto, vfrom, size);
+}
+
+static void copythread(struct work_struct *work)
+{
+	struct copy_info *info = (struct copy_info *) work;
+
+	copy_pages(info->to, info->from, info->chunk_size);
+}
+
+int copy_pages_mthread(struct page *to, struct page *from, int nr_pages)
+{
+	struct cpumask *cpumask;
+	struct copy_info *work_items;
+	char *vto, *vfrom;
+	unsigned long i, cthreads, cpu, node, chunk_size;
+	int cpu_id_list[32] = {0};
+
+	node = page_to_nid(to);
+	cpumask = cpumask_of_node(node);
+	cthreads = nr_copythreads;
+	cthreads = min_t(unsigned int, cthreads, cpumask_weight(cpumask));
+	cthreads = (cthreads / 2) * 2;
+	work_items = kcalloc(cthreads, sizeof(struct copy_info), GFP_KERNEL);
+	if (!work_items)
+		return -ENOMEM;
+
+	i = 0;
+	for_each_cpu(cpu, cpumask) {
+		if (i >= cthreads)
+			break;
+		cpu_id_list[i] = cpu;
+		++i;
+	}
+
+	vfrom = kmap(from);
+	vto = kmap(to);
+	chunk_size = PAGE_SIZE * nr_pages / cthreads;
+
+	for (i = 0; i < cthreads; ++i) {
+		INIT_WORK((struct work_struct *) &work_items[i], copythread);
+
+		work_items[i].to = vto + i * chunk_size;
+		work_items[i].from = vfrom + i * chunk_size;
+		work_items[i].chunk_size = chunk_size;
+
+		queue_work_on(cpu_id_list[i], system_highpri_wq,
+					(struct work_struct *) &work_items[i]);
+	}
+
+	for (i = 0; i < cthreads; ++i)
+		flush_work((struct work_struct *) &work_items[i]);
+
+	kunmap(to);
+	kunmap(from);
+	kfree(work_items);
+	return 0;
+}
-- 
2.9.3

[PATCH 4/6] mm/migrate: Add new migrate mode MIGRATE_MT

[Anshuman Khandual]

From: Zi Yan <ziy@nvidia.com>

This change adds a new migration mode called MIGRATE_MT to enable multi
threaded page copy implementation inside copy_huge_page() function by
selectively calling copy_pages_mthread() when requested. But it still
falls back using the regular page copy mechanism instead the previous
multi threaded attempt fails. It also attempts multi threaded copy for
regular pages.

Signed-off-by: Zi Yan <zi.yan@cs.rutgers.edu>
Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
 include/linux/migrate_mode.h |  1 +
 mm/migrate.c                 | 25 ++++++++++++++++++-------
 2 files changed, 19 insertions(+), 7 deletions(-)

diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h
index 89c1700..d344ad6 100644
--- a/include/linux/migrate_mode.h
+++ b/include/linux/migrate_mode.h
@@ -12,6 +12,7 @@ enum migrate_mode {
 	MIGRATE_SYNC_LIGHT	= 1<<1,
 	MIGRATE_SYNC		= 1<<2,
 	MIGRATE_ST		= 1<<3,
+	MIGRATE_MT		= 1<<4,
 };
 
 #endif		/* MIGRATE_MODE_H_INCLUDED */
diff --git a/mm/migrate.c b/mm/migrate.c
index 63c3682..6ac3572 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -594,6 +594,7 @@ static void copy_huge_page(struct page *dst, struct page *src,
 {
 	int i;
 	int nr_pages;
+	int rc = -EFAULT;
 
 	if (PageHuge(src)) {
 		/* hugetlbfs page */
@@ -610,10 +611,14 @@ static void copy_huge_page(struct page *dst, struct page *src,
 		nr_pages = hpage_nr_pages(src);
 	}
 
-	for (i = 0; i < nr_pages; i++) {
-		cond_resched();
-		copy_highpage(dst + i, src + i);
-	}
+	if (mode & MIGRATE_MT)
+		rc = copy_pages_mthread(dst, src, nr_pages);
+
+	if (rc)
+		for (i = 0; i < nr_pages; i++) {
+			cond_resched();
+			copy_highpage(dst + i, src + i);
+		}
 }
 
 /*
@@ -624,10 +629,16 @@ void migrate_page_copy(struct page *newpage, struct page *page,
 {
 	int cpupid;
 
-	if (PageHuge(page) || PageTransHuge(page))
+	if (PageHuge(page) || PageTransHuge(page)) {
 		copy_huge_page(newpage, page, mode);
-	else
-		copy_highpage(newpage, page);
+	} else {
+		if (mode & MIGRATE_MT) {
+			if (copy_pages_mthread(newpage, page, 1))
+				copy_highpage(newpage, page);
+		} else {
+			copy_highpage(newpage, page);
+		}
+	}
 
 	if (PageError(page))
 		SetPageError(newpage);
-- 
2.9.3

[PATCH 5/6] mm/migrate: Add new migration flag MPOL_MF_MOVE_MT for syscalls

[Anshuman Khandual]

From: Zi Yan <ziy@nvidia.com>

This change adds a new mode flag MPOL_MF_MOVE_MT for migration system
calls like move_pages() and mbind() which indicates request for using
the multi threaded copy method.

Signed-off-by: Zi Yan <zi.yan@cs.rutgers.edu>
Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
 include/uapi/linux/mempolicy.h |  4 +++-
 mm/mempolicy.c                 |  7 ++++++-
 mm/migrate.c                   | 14 ++++++++++----
 3 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h
index 9cd8b21..8f1db2e 100644
--- a/include/uapi/linux/mempolicy.h
+++ b/include/uapi/linux/mempolicy.h
@@ -53,10 +53,12 @@ enum mpol_rebind_step {
 #define MPOL_MF_MOVE_ALL (1<<2)	/* Move every page to conform to policy */
 #define MPOL_MF_LAZY	 (1<<3)	/* Modifies '_MOVE:  lazy migrate on fault */
 #define MPOL_MF_INTERNAL (1<<4)	/* Internal flags start here */
+#define MPOL_MF_MOVE_MT  (1<<6)	/* Use multi-threaded page copy routine */
 
 #define MPOL_MF_VALID	(MPOL_MF_STRICT   | 	\
 			 MPOL_MF_MOVE     | 	\
-			 MPOL_MF_MOVE_ALL)
+			 MPOL_MF_MOVE_ALL |	\
+			 MPOL_MF_MOVE_MT)
 
 /*
  * Internal flags that share the struct mempolicy flags word with
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 1e7873e..ef7c586 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1244,9 +1244,14 @@ static long do_mbind(unsigned long start, unsigned long len,
 		int nr_failed = 0;
 
 		if (!list_empty(&pagelist)) {
+			enum migrate_mode mode = MIGRATE_SYNC;
+
+			if (flags & MPOL_MF_MOVE_MT)
+				mode |= MIGRATE_MT;
+
 			WARN_ON_ONCE(flags & MPOL_MF_LAZY);
 			nr_failed = migrate_pages(&pagelist, new_page, NULL,
-				start, MIGRATE_SYNC, MR_MEMPOLICY_MBIND);
+					start, mode, MR_MEMPOLICY_MBIND);
 			if (nr_failed)
 				putback_movable_pages(&pagelist);
 		}
diff --git a/mm/migrate.c b/mm/migrate.c
index 6ac3572..660c4b2 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1429,11 +1429,16 @@ static struct page *new_page_node(struct page *p, unsigned long private,
  */
 static int do_move_page_to_node_array(struct mm_struct *mm,
 				      struct page_to_node *pm,
-				      int migrate_all)
+				      int migrate_all,
+					  int migrate_use_mt)
 {
 	int err;
 	struct page_to_node *pp;
 	LIST_HEAD(pagelist);
+	enum migrate_mode mode = MIGRATE_SYNC;
+
+	if (migrate_use_mt)
+		mode |= MIGRATE_MT;
 
 	down_read(&mm->mmap_sem);
 
@@ -1501,7 +1506,7 @@ static int do_move_page_to_node_array(struct mm_struct *mm,
 	err = 0;
 	if (!list_empty(&pagelist)) {
 		err = migrate_pages(&pagelist, new_page_node, NULL,
-				(unsigned long)pm, MIGRATE_SYNC, MR_SYSCALL);
+				(unsigned long)pm, mode, MR_SYSCALL);
 		if (err)
 			putback_movable_pages(&pagelist);
 	}
@@ -1578,7 +1583,8 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes,
 
 		/* Migrate this chunk */
 		err = do_move_page_to_node_array(mm, pm,
-						 flags & MPOL_MF_MOVE_ALL);
+						 flags & MPOL_MF_MOVE_ALL,
+						 flags & MPOL_MF_MOVE_MT);
 		if (err < 0)
 			goto out_pm;
 
@@ -1685,7 +1691,7 @@ SYSCALL_DEFINE6(move_pages, pid_t, pid, unsigned long, nr_pages,
 	nodemask_t task_nodes;
 
 	/* Check flags */
-	if (flags & ~(MPOL_MF_MOVE|MPOL_MF_MOVE_ALL))
+	if (flags & ~(MPOL_MF_MOVE|MPOL_MF_MOVE_ALL|MPOL_MF_MOVE_MT))
 		return -EINVAL;
 
 	if ((flags & MPOL_MF_MOVE_ALL) && !capable(CAP_SYS_NICE))
-- 
2.9.3

[PATCH 6/6] sysctl: Add global tunable mt_page_copy

[Anshuman Khandual]

From: Zi Yan <ziy@nvidia.com>

A new global sysctl tunable 'mt_page_copy' is added which will override
syscall specific requests and enable multi threaded page copy during
all migrations on the system. This tunable is disabled by default.

Signed-off-by: Zi Yan <zi.yan@cs.rutgers.edu>
Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
 kernel/sysctl.c | 10 ++++++++++
 mm/migrate.c    | 14 ++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/kernel/sysctl.c b/kernel/sysctl.c
index 1aea594..e5f7ca9 100644
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -98,6 +98,7 @@
 #if defined(CONFIG_SYSCTL)
 
 /* External variables not in a header file. */
+extern int mt_page_copy;
 extern int suid_dumpable;
 #ifdef CONFIG_COREDUMP
 extern int core_uses_pid;
@@ -1346,6 +1347,15 @@ static struct ctl_table vm_table[] = {
 		.proc_handler   = &hugetlb_mempolicy_sysctl_handler,
 	},
 #endif
+	{
+		.procname	= "mt_page_copy",
+		.data		= &mt_page_copy,
+		.maxlen		= sizeof(mt_page_copy),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+		.extra1		= &zero,
+		.extra2		= &one,
+	},
 	 {
 		.procname	= "hugetlb_shm_group",
 		.data		= &sysctl_hugetlb_shm_group,
diff --git a/mm/migrate.c b/mm/migrate.c
index 660c4b2..75b6d7a 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -48,6 +48,8 @@
 
 #include "internal.h"
 
+int mt_page_copy;
+
 /*
  * migrate_prep() needs to be called before we start compiling a list of pages
  * to be migrated using isolate_lru_page(). If scheduling work on other CPUs is
@@ -611,6 +613,9 @@ static void copy_huge_page(struct page *dst, struct page *src,
 		nr_pages = hpage_nr_pages(src);
 	}
 
+	if (mt_page_copy)
+		mode |= MIGRATE_MT;
+
 	if (mode & MIGRATE_MT)
 		rc = copy_pages_mthread(dst, src, nr_pages);
 
@@ -629,6 +634,9 @@ void migrate_page_copy(struct page *newpage, struct page *page,
 {
 	int cpupid;
 
+	if (mt_page_copy)
+		mode |= MIGRATE_MT;
+
 	if (PageHuge(page) || PageTransHuge(page)) {
 		copy_huge_page(newpage, page, mode);
 	} else {
@@ -695,6 +703,12 @@ void migrate_page_copy(struct page *newpage, struct page *page,
 }
 EXPORT_SYMBOL(migrate_page_copy);
 
+static int __init mt_page_copy_init(void)
+{
+	mt_page_copy = 0;
+	return 0;
+}
+subsys_initcall(mt_page_copy_init);
 /************************************************************
  *                    Migration functions
  ***********************************************************/
-- 
2.9.3

Re: [PATCH 3/6] mm/migrate: Add copy_pages_mthread function

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 1864 bytes --]

Hi Zi,

[auto build test WARNING on linus/master]
[also build test WARNING on v4.10-rc8 next-20170216]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Enable-parallel-page-migration/20170217-200523
config: i386-tinyconfig (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All warnings (new ones prefixed by >>):

   mm/copy_pages_mthread.c: In function 'copy_pages_mthread':
>> mm/copy_pages_mthread.c:49:10: warning: assignment discards 'const' qualifier from pointer target type [-Wdiscarded-qualifiers]
     cpumask = cpumask_of_node(node);
             ^

vim +/const +49 mm/copy_pages_mthread.c

    33	static void copythread(struct work_struct *work)
    34	{
    35		struct copy_info *info = (struct copy_info *) work;
    36	
    37		copy_pages(info->to, info->from, info->chunk_size);
    38	}
    39	
    40	int copy_pages_mthread(struct page *to, struct page *from, int nr_pages)
    41	{
    42		struct cpumask *cpumask;
    43		struct copy_info *work_items;
    44		char *vto, *vfrom;
    45		unsigned long i, cthreads, cpu, node, chunk_size;
    46		int cpu_id_list[32] = {0};
    47	
    48		node = page_to_nid(to);
  > 49		cpumask = cpumask_of_node(node);
    50		cthreads = nr_copythreads;
    51		cthreads = min_t(unsigned int, cthreads, cpumask_weight(cpumask));
    52		cthreads = (cthreads / 2) * 2;
    53		work_items = kcalloc(cthreads, sizeof(struct copy_info), GFP_KERNEL);
    54		if (!work_items)
    55			return -ENOMEM;
    56	
    57		i = 0;

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 6397 bytes --]

Re: [PATCH 6/6] sysctl: Add global tunable mt_page_copy

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 786 bytes --]

Hi Zi,

[auto build test ERROR on linus/master]
[also build test ERROR on v4.10-rc8 next-20170217]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Enable-parallel-page-migration/20170217-200523
config: i386-randconfig-a0-02131010 (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All errors (new ones prefixed by >>):

>> kernel/built-in.o:(.data+0x1ec8): undefined reference to `mt_page_copy'

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 26466 bytes --]

Re: [PATCH 0/6] Enable parallel page migration

[Balbir Singh]

On Fri, Feb 17, 2017 at 04:54:47PM +0530, Anshuman Khandual wrote:
> 	This patch series is base on the work posted by Zi Yan back in
> November 2016 (https://lkml.org/lkml/2016/11/22/457) but includes some
> amount clean up and re-organization. This series depends on THP migration
> optimization patch series posted by Naoya Horiguchi on 8th November 2016
> (https://lwn.net/Articles/705879/). Though Zi Yan has recently reposted
> V3 of the THP migration patch series (https://lwn.net/Articles/713667/),
> this series is yet to be rebased.
> 
> 	Primary motivation behind this patch series is to achieve higher
> bandwidth of memory migration when ever possible using multi threaded
> instead of a single threaded copy. Did all the experiments using a two
> socket X86 sytsem (Intel(R) Xeon(R) CPU E5-2650). All the experiments
> here have same allocation size 4K * 100000 (which did not split evenly
> for the 2MB huge pages). Here are the results.
> 
> Vanilla:
> 
> Moved 100000 normal pages in 247.000000 msecs 1.544412 GBs
> Moved 100000 normal pages in 238.000000 msecs 1.602814 GBs
> Moved 195 huge pages in 252.000000 msecs 1.513769 GBs
> Moved 195 huge pages in 257.000000 msecs 1.484318 GBs
> 
> THP migration improvements:
> 
> Moved 100000 normal pages in 302.000000 msecs 1.263145 GBs

Is there a decrease here for normal pages?

> Moved 100000 normal pages in 262.000000 msecs 1.455991 GBs
> Moved 195 huge pages in 120.000000 msecs 3.178914 GBs
> Moved 195 huge pages in 129.000000 msecs 2.957130 GBs
> 
> THP migration improvements + Multi threaded page copy:
> 
> Moved 100000 normal pages in 1589.000000 msecs 0.240069 GBs **

Ditto?

> Moved 100000 normal pages in 1932.000000 msecs 0.197448 GBs **
> Moved 195 huge pages in 54.000000 msecs 7.064254 GBs ***
> Moved 195 huge pages in 86.000000 msecs 4.435694 GBs ***
>

Could you also comment on the CPU utilization impact of these
patches. 

Balbir Singh.

Re: [PATCH 0/6] Enable parallel page migration

[Anshuman Khandual]

On 02/22/2017 10:34 AM, Balbir Singh wrote:
> On Fri, Feb 17, 2017 at 04:54:47PM +0530, Anshuman Khandual wrote:
>> 	This patch series is base on the work posted by Zi Yan back in
>> November 2016 (https://lkml.org/lkml/2016/11/22/457) but includes some
>> amount clean up and re-organization. This series depends on THP migration
>> optimization patch series posted by Naoya Horiguchi on 8th November 2016
>> (https://lwn.net/Articles/705879/). Though Zi Yan has recently reposted
>> V3 of the THP migration patch series (https://lwn.net/Articles/713667/),
>> this series is yet to be rebased.
>>
>> 	Primary motivation behind this patch series is to achieve higher
>> bandwidth of memory migration when ever possible using multi threaded
>> instead of a single threaded copy. Did all the experiments using a two
>> socket X86 sytsem (Intel(R) Xeon(R) CPU E5-2650). All the experiments
>> here have same allocation size 4K * 100000 (which did not split evenly
>> for the 2MB huge pages). Here are the results.
>>
>> Vanilla:
>>
>> Moved 100000 normal pages in 247.000000 msecs 1.544412 GBs
>> Moved 100000 normal pages in 238.000000 msecs 1.602814 GBs
>> Moved 195 huge pages in 252.000000 msecs 1.513769 GBs
>> Moved 195 huge pages in 257.000000 msecs 1.484318 GBs
>>
>> THP migration improvements:
>>
>> Moved 100000 normal pages in 302.000000 msecs 1.263145 GBs
> 
> Is there a decrease here for normal pages?

Yeah.

> 
>> Moved 100000 normal pages in 262.000000 msecs 1.455991 GBs
>> Moved 195 huge pages in 120.000000 msecs 3.178914 GBs
>> Moved 195 huge pages in 129.000000 msecs 2.957130 GBs
>>
>> THP migration improvements + Multi threaded page copy:
>>
>> Moved 100000 normal pages in 1589.000000 msecs 0.240069 GBs **
> 
> Ditto?

Yeah, I have already mentioned about this after these data in
the cover letter. This new flag is controlled from user space
while invoking the system calls. Users should be careful in
using it for scenarios where its useful and avoid it for cases
where it hurts.

> 
>> Moved 100000 normal pages in 1932.000000 msecs 0.197448 GBs **
>> Moved 195 huge pages in 54.000000 msecs 7.064254 GBs ***
>> Moved 195 huge pages in 86.000000 msecs 4.435694 GBs ***
>>
> 
> Could you also comment on the CPU utilization impact of these
> patches. 

Yeah, it really makes sense to analyze this impact. I have mentioned
about this in the outstanding issues section of the series. But what
exactly we need to analyze from CPU utilization impact point of view
? Like whats the probability that the work queue requested jobs will
throw some tasks from the run queue and make them starve for some
more time ? Could you please give some details on this ?

Re: [PATCH 0/6] Enable parallel page migration

[Balbir Singh]

On 22/02/17 16:55, Anshuman Khandual wrote:
> On 02/22/2017 10:34 AM, Balbir Singh wrote:
>> On Fri, Feb 17, 2017 at 04:54:47PM +0530, Anshuman Khandual wrote:
>>> 	This patch series is base on the work posted by Zi Yan back in
>>> November 2016 (https://lkml.org/lkml/2016/11/22/457) but includes some
>>> amount clean up and re-organization. This series depends on THP migration
>>> optimization patch series posted by Naoya Horiguchi on 8th November 2016
>>> (https://lwn.net/Articles/705879/). Though Zi Yan has recently reposted
>>> V3 of the THP migration patch series (https://lwn.net/Articles/713667/),
>>> this series is yet to be rebased.
>>>
>>> 	Primary motivation behind this patch series is to achieve higher
>>> bandwidth of memory migration when ever possible using multi threaded
>>> instead of a single threaded copy. Did all the experiments using a two
>>> socket X86 sytsem (Intel(R) Xeon(R) CPU E5-2650). All the experiments
>>> here have same allocation size 4K * 100000 (which did not split evenly
>>> for the 2MB huge pages). Here are the results.
>>>
>>> Vanilla:
>>>
>>> Moved 100000 normal pages in 247.000000 msecs 1.544412 GBs
>>> Moved 100000 normal pages in 238.000000 msecs 1.602814 GBs
>>> Moved 195 huge pages in 252.000000 msecs 1.513769 GBs
>>> Moved 195 huge pages in 257.000000 msecs 1.484318 GBs
>>>
>>> THP migration improvements:
>>>
>>> Moved 100000 normal pages in 302.000000 msecs 1.263145 GBs
>>
>> Is there a decrease here for normal pages?
> 
> Yeah.
> 
>>
>>> Moved 100000 normal pages in 262.000000 msecs 1.455991 GBs
>>> Moved 195 huge pages in 120.000000 msecs 3.178914 GBs
>>> Moved 195 huge pages in 129.000000 msecs 2.957130 GBs
>>>
>>> THP migration improvements + Multi threaded page copy:
>>>
>>> Moved 100000 normal pages in 1589.000000 msecs 0.240069 GBs **
>>
>> Ditto?
> 
> Yeah, I have already mentioned about this after these data in
> the cover letter. This new flag is controlled from user space
> while invoking the system calls. Users should be careful in
> using it for scenarios where its useful and avoid it for cases
> where it hurts.

Fair enough, I wonder if _MT should be disabled for normal pages
and allow only THP migration. I think it might be worth evaluating
the overheads

> 
>>
>>> Moved 100000 normal pages in 1932.000000 msecs 0.197448 GBs **
>>> Moved 195 huge pages in 54.000000 msecs 7.064254 GBs ***
>>> Moved 195 huge pages in 86.000000 msecs 4.435694 GBs ***
>>>
>>
>> Could you also comment on the CPU utilization impact of these
>> patches. 
> 
> Yeah, it really makes sense to analyze this impact. I have mentioned
> about this in the outstanding issues section of the series. But what
> exactly we need to analyze from CPU utilization impact point of view
> ? Like whats the probability that the work queue requested jobs will
> throw some tasks from the run queue and make them starve for some
> more time ? Could you please give some details on this ?
> 

I wonder if the CPU utilization is so high that its hurting the CPU
(system time) at the cost of increased migration speeds. We may need
a trade-off (see my comment above)

Balbir Singh.

Re: [PATCH 6/6] sysctl: Add global tunable mt_page_copy

[Anshuman Khandual]

On 02/17/2017 09:00 PM, kbuild test robot wrote:
> Hi Zi,
> 
> [auto build test ERROR on linus/master]
> [also build test ERROR on v4.10-rc8 next-20170217]
> [if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
> 
> url:    https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Enable-parallel-page-migration/20170217-200523
> config: i386-randconfig-a0-02131010 (attached as .config)
> compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901

Though I dont have the same compiler, I am unable to reproduce this
build failure exactly. The build fails but for a different symbol.
I have the following gcc version but does it really make a
difference with respect to finding the symbol etc ?

gcc (Ubuntu 4.9.2-10ubuntu13) 4.9.2


mm/memory.c: In function ‘copy_pmd_range’:
mm/memory.c:1002:3: error: implicit declaration of function
‘pmd_related’ [-Werror=implicit-function-declaration]
   if (pmd_related(*src_pmd)) {
   ^
cc1: some warnings being treated as errors
scripts/Makefile.build:294: recipe for target 'mm/memory.o' failed

Re: [PATCH 3/6] mm/migrate: Add copy_pages_mthread function

[Anshuman Khandual]

On 02/17/2017 05:57 PM, kbuild test robot wrote:
> Hi Zi,
> 
> [auto build test WARNING on linus/master]
> [also build test WARNING on v4.10-rc8 next-20170216]
> [if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
> 
> url:    https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Enable-parallel-page-migration/20170217-200523
> config: i386-tinyconfig (attached as .config)
> compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
> reproduce:
>         # save the attached .config to linux build tree
>         make ARCH=i386 
> 
> All warnings (new ones prefixed by >>):
> 
>    mm/copy_pages_mthread.c: In function 'copy_pages_mthread':
>>> mm/copy_pages_mthread.c:49:10: warning: assignment discards 'const' qualifier from pointer target type [-Wdiscarded-qualifiers]
>      cpumask = cpumask_of_node(node);

My bad. This fixes the above warning. Will fix it up next time
around.

diff --git a/mm/copy_pages_mthread.c b/mm/copy_pages_mthread.c
index 9ad2ef6..46b22b1 100644
--- a/mm/copy_pages_mthread.c
+++ b/mm/copy_pages_mthread.c
@@ -46,7 +46,7 @@ int copy_pages_mthread(struct page *to, struct page
*from, int nr_pages)
        int cpu_id_list[32] = {0};

        node = page_to_nid(to);
-       cpumask = cpumask_of_node(node);
+       cpumask = (struct cpumask *) cpumask_of_node(node);
        cthreads = nr_copythreads;
        cthreads = min_t(unsigned int, cthreads, cpumask_weight(cpumask));
        cthreads = (cthreads / 2) * 2;

Re: [PATCH 0/6] Enable parallel page migration

[Anshuman Khandual]

On 02/17/2017 04:54 PM, Anshuman Khandual wrote:
> 	This patch series is base on the work posted by Zi Yan back in
> November 2016 (https://lkml.org/lkml/2016/11/22/457) but includes some
> amount clean up and re-organization. This series depends on THP migration
> optimization patch series posted by Naoya Horiguchi on 8th November 2016
> (https://lwn.net/Articles/705879/). Though Zi Yan has recently reposted
> V3 of the THP migration patch series (https://lwn.net/Articles/713667/),
> this series is yet to be rebased.
> 
> 	Primary motivation behind this patch series is to achieve higher
> bandwidth of memory migration when ever possible using multi threaded
> instead of a single threaded copy. Did all the experiments using a two
> socket X86 sytsem (Intel(R) Xeon(R) CPU E5-2650). All the experiments
> here have same allocation size 4K * 100000 (which did not split evenly
> for the 2MB huge pages). Here are the results.
> 
> Vanilla:
> 
> Moved 100000 normal pages in 247.000000 msecs 1.544412 GBs
> Moved 100000 normal pages in 238.000000 msecs 1.602814 GBs
> Moved 195 huge pages in 252.000000 msecs 1.513769 GBs
> Moved 195 huge pages in 257.000000 msecs 1.484318 GBs
> 
> THP migration improvements:
> 
> Moved 100000 normal pages in 302.000000 msecs 1.263145 GBs
> Moved 100000 normal pages in 262.000000 msecs 1.455991 GBs
> Moved 195 huge pages in 120.000000 msecs 3.178914 GBs
> Moved 195 huge pages in 129.000000 msecs 2.957130 GBs
> 
> THP migration improvements + Multi threaded page copy:
> 
> Moved 100000 normal pages in 1589.000000 msecs 0.240069 GBs **
> Moved 100000 normal pages in 1932.000000 msecs 0.197448 GBs **
> Moved 195 huge pages in 54.000000 msecs 7.064254 GBs ***
> Moved 195 huge pages in 86.000000 msecs 4.435694 GBs ***
> 
> 
> **      Using multi threaded copy can be detrimental to performance if
> 	used for regular pages which are way too small. But then the
> 	framework provides the means to use it if some kernel/driver
> 	caller or user application wants to use it.
> 
> ***     These applications have used the new MPOL_MF_MOVE_MT flag while
> 	calling the system calls like mbind() and move_pages().
> 
> On POWER8 the improvements are similar when tested with a draft patch
> which enables migration at PMD level. Not putting out the results here
> as the kernel is not stable with the that draft patch and crashes some
> times. We are working on enabling PMD level migration on POWER8 and will
> test this series out thoroughly when its ready.
> 
> Patch Series Description::
> 
> Patch 1: Add new parameter to migrate_page_copy and copy_huge_page so
> 	 that it can differentiate between when to use single threaded
> 	 version (MIGRATE_ST) or multi threaded version (MIGRATE_MT).
> 
> Patch 2: Make migrate_mode types non-exclusive.
> 
> Patch 3: Add copy_pages_mthread function which does the actual multi
> 	 threaded copy. This involves splitting the copy work into
> 	 chunks, selecting threads and submitting copy jobs in the
> 	 work queues.
> 
> Patch 4: Add new migrate mode MIGRATE_MT to be used by higher level
> 	 migration functions.
> 
> Patch 5: Add new migration flag MPOL_MF_MOVE_MT for migration system
> 	 calls to be used in the user space.
> 
> Patch 6: Define global mt_page_copy tunable which turns on the multi
> 	 threaded page copy no matter what for all migrations on the
> 	 system.
> 
> Outstanding Issues::
> 
> Issue 1: The usefulness of the global multi threaded copy tunable i.e
> 	 vm.mt_page_copy. It makes sense and helps in validating the
> 	 framework. Should this be moved to debugfs instead ?
> 
> Issue 2: We choose nr_copythreads = 8 as maximum number of threads on
> 	 a node can be 8 on any architecture (Which is on POWER8 if
> 	 I am not missing any other arch which might have equal or
> 	 more number of threads per node). It just denotes max number
> 	 of threads and we will be adjusted based on cpumask_weight
> 	 value on destination node. Can we do better, suggestions ?
> 
> Issue 3: Multi threaded page migration works best with threads allocated
> 	 at different physical cores, not all in the same hyper-threaded
> 	 core. Work queues submitted jobs consume scheduler slots from
> 	 the given thread to execute the copy. This can interfere with
> 	 scheduling and affect some already running tasks on the system.
> 	 Should we be looking into arch topology information, scheduler
> 	 cpu idle details to decide on which threads to use before going
> 	 for multi threaded copy ? Abort multi threaded copy and fallback
> 	 to regular copy at times when the parameters are not good ?
> 
> Any comments, suggestions are welcome.

Hello Vlastimil/Michal/Minchan/Mel/Dave,

Apart from the comments from Naoya on a different thread posted by Zi
Yan, I did not get any more review comments on this series. Could you
please kindly have a look on the over all design and its benefits from
page migration performance point of view and let me know your views.
Thank you.

+ Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>


Regards
Anshuman

[PATCH 1/6] mm/migrate: Add new mode parameter to migrate_page_copy() function

[Anshuman Khandual]

From: Zi Yan <ziy@nvidia.com>

This is a prerequisite change required to make page migration framewok
copy in different modes like the default single threaded or the new
multi threaded one yet to be introduced in follow up patches. This
does not change any existing functionality. Only migrate_page_copy()
and copy_huge_page() function's signatures are affected.

Signed-off-by: Zi Yan <zi.yan@cs.rutgers.edu>
Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
* Updated include/linux/migrate_mode.h comment as per Naoya

 fs/aio.c                     |  2 +-
 fs/f2fs/data.c               |  2 +-
 fs/hugetlbfs/inode.c         |  2 +-
 fs/ubifs/file.c              |  2 +-
 include/linux/migrate.h      |  6 ++++--
 include/linux/migrate_mode.h |  2 ++
 mm/migrate.c                 | 14 ++++++++------
 7 files changed, 18 insertions(+), 12 deletions(-)

diff --git a/fs/aio.c b/fs/aio.c
index 873b4ca..ba3f6eb 100644
--- a/fs/aio.c
+++ b/fs/aio.c
@@ -418,7 +418,7 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,
 	 * events from being lost.
 	 */
 	spin_lock_irqsave(&ctx->completion_lock, flags);
-	migrate_page_copy(new, old);
+	migrate_page_copy(new, old, MIGRATE_ST);
 	BUG_ON(ctx->ring_pages[idx] != old);
 	ctx->ring_pages[idx] = new;
 	spin_unlock_irqrestore(&ctx->completion_lock, flags);
diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index 9ac2625..ad41356 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -1997,7 +1997,7 @@ int f2fs_migrate_page(struct address_space *mapping,
 		SetPagePrivate(newpage);
 	set_page_private(newpage, page_private(page));
 
-	migrate_page_copy(newpage, page);
+	migrate_page_copy(newpage, page, MIGRATE_ST);
 
 	return MIGRATEPAGE_SUCCESS;
 }
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 54de77e..0e16512f 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -850,7 +850,7 @@ static int hugetlbfs_migrate_page(struct address_space *mapping,
 	rc = migrate_huge_page_move_mapping(mapping, newpage, page);
 	if (rc != MIGRATEPAGE_SUCCESS)
 		return rc;
-	migrate_page_copy(newpage, page);
+	migrate_page_copy(newpage, page, MIGRATE_ST);
 
 	return MIGRATEPAGE_SUCCESS;
 }
diff --git a/fs/ubifs/file.c b/fs/ubifs/file.c
index b0d7837..293616f 100644
--- a/fs/ubifs/file.c
+++ b/fs/ubifs/file.c
@@ -1482,7 +1482,7 @@ static int ubifs_migrate_page(struct address_space *mapping,
 		SetPagePrivate(newpage);
 	}
 
-	migrate_page_copy(newpage, page);
+	migrate_page_copy(newpage, page, MIGRATE_ST);
 	return MIGRATEPAGE_SUCCESS;
 }
 #endif
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index ae8d475..d843b8f 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -42,7 +42,8 @@ extern void putback_movable_page(struct page *page);
 
 extern int migrate_prep(void);
 extern int migrate_prep_local(void);
-extern void migrate_page_copy(struct page *newpage, struct page *page);
+extern void migrate_page_copy(struct page *newpage, struct page *page,
+			enum migrate_mode mode);
 extern int migrate_huge_page_move_mapping(struct address_space *mapping,
 				  struct page *newpage, struct page *page);
 extern int migrate_page_move_mapping(struct address_space *mapping,
@@ -61,7 +62,8 @@ static inline int migrate_prep(void) { return -ENOSYS; }
 static inline int migrate_prep_local(void) { return -ENOSYS; }
 
 static inline void migrate_page_copy(struct page *newpage,
-				     struct page *page) {}
+				     struct page *page,
+				     enum migrate_mode mode) {}
 
 static inline int migrate_huge_page_move_mapping(struct address_space *mapping,
 				  struct page *newpage, struct page *page)
diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h
index ebf3d89..deaeba5 100644
--- a/include/linux/migrate_mode.h
+++ b/include/linux/migrate_mode.h
@@ -6,11 +6,13 @@
  *	on most operations but not ->writepage as the potential stall time
  *	is too significant
  * MIGRATE_SYNC will block when migrating pages
+ * MIGRATE_ST will use single thread when migrating pages
  */
 enum migrate_mode {
 	MIGRATE_ASYNC,
 	MIGRATE_SYNC_LIGHT,
 	MIGRATE_SYNC,
+	MIGRATE_ST
 };
 
 #endif		/* MIGRATE_MODE_H_INCLUDED */
diff --git a/mm/migrate.c b/mm/migrate.c
index e8ae11a..5ef4aa4 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -631,7 +631,8 @@ static void __copy_gigantic_page(struct page *dst, struct page *src,
 	}
 }
 
-static void copy_huge_page(struct page *dst, struct page *src)
+static void copy_huge_page(struct page *dst, struct page *src,
+				enum migrate_mode mode)
 {
 	int i;
 	int nr_pages;
@@ -660,12 +661,13 @@ static void copy_huge_page(struct page *dst, struct page *src)
 /*
  * Copy the page to its new location
  */
-void migrate_page_copy(struct page *newpage, struct page *page)
+void migrate_page_copy(struct page *newpage, struct page *page,
+					   enum migrate_mode mode)
 {
 	int cpupid;
 
 	if (PageHuge(page) || PageTransHuge(page))
-		copy_huge_page(newpage, page);
+		copy_huge_page(newpage, page, mode);
 	else
 		copy_highpage(newpage, page);
 
@@ -747,7 +749,7 @@ int migrate_page(struct address_space *mapping,
 	if (rc != MIGRATEPAGE_SUCCESS)
 		return rc;
 
-	migrate_page_copy(newpage, page);
+	migrate_page_copy(newpage, page, mode);
 	return MIGRATEPAGE_SUCCESS;
 }
 EXPORT_SYMBOL(migrate_page);
@@ -797,7 +799,7 @@ int buffer_migrate_page(struct address_space *mapping,
 
 	SetPagePrivate(newpage);
 
-	migrate_page_copy(newpage, page);
+	migrate_page_copy(newpage, page, MIGRATE_ST);
 
 	bh = head;
 	do {
@@ -2029,7 +2031,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
 	/* anon mapping, we can simply copy page->mapping to the new page: */
 	new_page->mapping = page->mapping;
 	new_page->index = page->index;
-	migrate_page_copy(new_page, page);
+	migrate_page_copy(new_page, page, MIGRATE_ST);
 	WARN_ON(PageLRU(new_page));
 
 	/* Recheck the target PMD */
-- 
2.1.4

[PATCH 3/6] mm/migrate: Add copy_pages_mthread function

[Anshuman Khandual]

From: Zi Yan <ziy@nvidia.com>

This change adds a new function copy_pages_mthread to enable multi threaded
page copy which can be utilized during migration. This function splits the
page copy request into multiple threads which will handle individual chunk
and send them as jobs to system_highpri_wq work queue.

Signed-off-by: Zi Yan <zi.yan@cs.rutgers.edu>
Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
* Updated cthread calculations, taking care of divide by zero issues,
  picking up the right single thread, defining NR_COPYTHREADS, fixing
  the build problem on i386 etc.

 include/linux/highmem.h |  2 ++
 mm/Makefile             |  2 ++
 mm/copy_pages_mthread.c | 95 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 99 insertions(+)
 create mode 100644 mm/copy_pages_mthread.c

diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index bb3f329..e1f4f1b 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -236,6 +236,8 @@ static inline void copy_user_highpage(struct page *to, struct page *from,
 
 #endif
 
+int copy_pages_mthread(struct page *to, struct page *from, int nr_pages);
+
 static inline void copy_highpage(struct page *to, struct page *from)
 {
 	char *vfrom, *vto;
diff --git a/mm/Makefile b/mm/Makefile
index 295bd7a..cc27e76 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -41,6 +41,8 @@ obj-y			:= filemap.o mempool.o oom_kill.o \
 
 obj-y += init-mm.o
 
+obj-y += copy_pages_mthread.o
+
 ifdef CONFIG_NO_BOOTMEM
 	obj-y		+= nobootmem.o
 else
diff --git a/mm/copy_pages_mthread.c b/mm/copy_pages_mthread.c
new file mode 100644
index 0000000..5af861c
--- /dev/null
+++ b/mm/copy_pages_mthread.c
@@ -0,0 +1,95 @@
+/*
+ * This implements parallel page copy function through multi
+ * threaded work queues.
+ *
+ * Copyright (C) Zi Yan <ziy@nvidia.com>, Nov 2016
+ *
+ * Licensed under the terms of the GNU GPL, version 2.
+ */
+#include <linux/highmem.h>
+#include <linux/workqueue.h>
+#include <linux/slab.h>
+#include <linux/freezer.h>
+
+/*
+ * NR_COPYTHREADS can be the highest number of threads for given
+ * node on any architecture. The actual number of copy threads
+ * will be limited by the cpumask weight of the target node.
+ */
+#define NR_COPYTHREADS 8
+
+struct copy_info {
+	struct work_struct copy_work;
+	char *to;
+	char *from;
+	unsigned long chunk_size;
+};
+
+static void copy_pages(char *vto, char *vfrom, unsigned long size)
+{
+	memcpy(vto, vfrom, size);
+}
+
+static void copythread(struct work_struct *work)
+{
+	struct copy_info *info = (struct copy_info *) work;
+
+	copy_pages(info->to, info->from, info->chunk_size);
+}
+
+int copy_pages_mthread(struct page *to, struct page *from, int nr_pages)
+{
+	struct cpumask *cpumask;
+	struct copy_info *work_items;
+	char *vto, *vfrom;
+	unsigned long i, cthreads, cpu, node, chunk_size;
+	int cpu_id_list[NR_COPYTHREADS] = {0};
+
+	node = page_to_nid(to);
+	cpumask = (struct cpumask *) cpumask_of_node(node);
+	cthreads = min_t(unsigned int, NR_COPYTHREADS, cpumask_weight(cpumask));
+	cthreads = (cthreads / 2) * 2;
+	if (!cthreads)
+		cthreads = 1;
+
+	work_items = kcalloc(cthreads, sizeof(struct copy_info), GFP_KERNEL);
+	if (!work_items)
+		return -ENOMEM;
+
+	/*
+	 * XXX: On a memory-only CPU-less NUMA node it will
+	 * just fallback using cpu[0] in a single threaded
+	 * manner to do the page copy. On single CPU target
+	 * node that CPU will be used for the page copy.
+	 */
+	i = 0;
+	for_each_cpu(cpu, cpumask) {
+		if (i >= cthreads)
+			break;
+		cpu_id_list[i] = cpu;
+		++i;
+	}
+
+	vfrom = kmap(from);
+	vto = kmap(to);
+	chunk_size = PAGE_SIZE * nr_pages / cthreads;
+
+	for (i = 0; i < cthreads; ++i) {
+		INIT_WORK((struct work_struct *) &work_items[i], copythread);
+
+		work_items[i].to = vto + i * chunk_size;
+		work_items[i].from = vfrom + i * chunk_size;
+		work_items[i].chunk_size = chunk_size;
+
+		queue_work_on(cpu_id_list[i], system_highpri_wq,
+					(struct work_struct *) &work_items[i]);
+	}
+
+	for (i = 0; i < cthreads; ++i)
+		flush_work((struct work_struct *) &work_items[i]);
+
+	kunmap(to);
+	kunmap(from);
+	kfree(work_items);
+	return 0;
+}
-- 
2.1.4

[PATCH 5/6] mm/migrate: Add new migration flag MPOL_MF_MOVE_MT for syscalls

[Anshuman Khandual]

From: Zi Yan <ziy@nvidia.com>

This change adds a new mode flag MPOL_MF_MOVE_MT for migration system
calls like move_pages() and mbind() which indicates request for using
the multi threaded copy method.

Signed-off-by: Zi Yan <zi.yan@cs.rutgers.edu>
Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
* Updated include/linux/migrate_mode.h comment as per Naoya

 include/uapi/linux/mempolicy.h |  4 +++-
 mm/mempolicy.c                 |  7 ++++++-
 mm/migrate.c                   | 14 ++++++++++----
 3 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h
index 9cd8b21..8f1db2e 100644
--- a/include/uapi/linux/mempolicy.h
+++ b/include/uapi/linux/mempolicy.h
@@ -53,10 +53,12 @@ enum mpol_rebind_step {
 #define MPOL_MF_MOVE_ALL (1<<2)	/* Move every page to conform to policy */
 #define MPOL_MF_LAZY	 (1<<3)	/* Modifies '_MOVE:  lazy migrate on fault */
 #define MPOL_MF_INTERNAL (1<<4)	/* Internal flags start here */
+#define MPOL_MF_MOVE_MT  (1<<6)	/* Use multi-threaded page copy routine */
 
 #define MPOL_MF_VALID	(MPOL_MF_STRICT   | 	\
 			 MPOL_MF_MOVE     | 	\
-			 MPOL_MF_MOVE_ALL)
+			 MPOL_MF_MOVE_ALL |	\
+			 MPOL_MF_MOVE_MT)
 
 /*
  * Internal flags that share the struct mempolicy flags word with
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index d880dc6..2d06ee2 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1300,9 +1300,14 @@ static long do_mbind(unsigned long start, unsigned long len,
 		int nr_failed = 0;
 
 		if (!list_empty(&pagelist)) {
+			enum migrate_mode mode = MIGRATE_SYNC;
+
+			if (flags & MPOL_MF_MOVE_MT)
+				mode |= MIGRATE_MT;
+
 			WARN_ON_ONCE(flags & MPOL_MF_LAZY);
 			nr_failed = migrate_pages(&pagelist, new_page, NULL,
-				start, MIGRATE_SYNC, MR_MEMPOLICY_MBIND);
+					start, mode, MR_MEMPOLICY_MBIND);
 			if (nr_failed)
 				putback_movable_pages(&pagelist);
 		}
diff --git a/mm/migrate.c b/mm/migrate.c
index 187065e..7449f7d 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1481,11 +1481,16 @@ static struct page *new_page_node(struct page *p, unsigned long private,
  */
 static int do_move_page_to_node_array(struct mm_struct *mm,
 				      struct page_to_node *pm,
-				      int migrate_all)
+				      int migrate_all,
+					  int migrate_use_mt)
 {
 	int err;
 	struct page_to_node *pp;
 	LIST_HEAD(pagelist);
+	enum migrate_mode mode = MIGRATE_SYNC;
+
+	if (migrate_use_mt)
+		mode |= MIGRATE_MT;
 
 	down_read(&mm->mmap_sem);
 
@@ -1562,7 +1567,7 @@ static int do_move_page_to_node_array(struct mm_struct *mm,
 	err = 0;
 	if (!list_empty(&pagelist)) {
 		err = migrate_pages(&pagelist, new_page_node, NULL,
-				(unsigned long)pm, MIGRATE_SYNC, MR_SYSCALL);
+				(unsigned long)pm, mode, MR_SYSCALL);
 		if (err)
 			putback_movable_pages(&pagelist);
 	}
@@ -1639,7 +1644,8 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes,
 
 		/* Migrate this chunk */
 		err = do_move_page_to_node_array(mm, pm,
-						 flags & MPOL_MF_MOVE_ALL);
+						 flags & MPOL_MF_MOVE_ALL,
+						 flags & MPOL_MF_MOVE_MT);
 		if (err < 0)
 			goto out_pm;
 
@@ -1746,7 +1752,7 @@ SYSCALL_DEFINE6(move_pages, pid_t, pid, unsigned long, nr_pages,
 	nodemask_t task_nodes;
 
 	/* Check flags */
-	if (flags & ~(MPOL_MF_MOVE|MPOL_MF_MOVE_ALL))
+	if (flags & ~(MPOL_MF_MOVE|MPOL_MF_MOVE_ALL|MPOL_MF_MOVE_MT))
 		return -EINVAL;
 
 	if ((flags & MPOL_MF_MOVE_ALL) && !capable(CAP_SYS_NICE))
-- 
2.1.4

Re: [PATCH 0/6] Enable parallel page migration

[Mel Gorman]

On Wed, Mar 08, 2017 at 09:34:27PM +0530, Anshuman Khandual wrote:
> > Any comments, suggestions are welcome.
> 
> Hello Vlastimil/Michal/Minchan/Mel/Dave,
> 
> Apart from the comments from Naoya on a different thread posted by Zi
> Yan, I did not get any more review comments on this series. Could you
> please kindly have a look on the over all design and its benefits from
> page migration performance point of view and let me know your views.
> Thank you.
> 

I didn't look into the patches in detail except to get a general feel
for how it works and I'm not convinced that it's a good idea at all.

I accept that memory bandwidth utilisation may be higher as a result but
consider the impact. THP migrations are relatively rare and when they
occur, it's in the context of a single thread. To parallelise the copy,
an allocation, kmap and workqueue invocation are required. There may be a
long delay before the workqueue item can start which may exceed the time
to do a single copy if the CPUs on a node are saturated. Furthermore, a
single thread can preempt operations of other unrelated threads and incur
CPU cache pollution and future misses on unrelated CPUs. It's compounded by
the fact that a high priority system workqueue is used to do the operation,
one that is used for CPU hotplug operations and rolling back when a netdevice
fails to be registered. It treats a hugepage copy as an essential operation
that can preempt all other work which is very questionable.

The series leader has no details on a workload that is bottlenecked by
THP migrations and even if it is, the primary question should be *why*
THP migrations are so frequent and alleviating that instead of
preempting multiple CPUs to do the work.

-- 
Mel Gorman
SUSE Labs

Re: [PATCH 0/6] Enable parallel page migration

[David Nellans]

On 03/09/2017 09:09 AM, Mel Gorman wrote:
> I didn't look into the patches in detail except to get a general feel
> for how it works and I'm not convinced that it's a good idea at all.
>
> I accept that memory bandwidth utilisation may be higher as a result but
> consider the impact. THP migrations are relatively rare and when they
> occur, it's in the context of a single thread. To parallelise the copy,
> an allocation, kmap and workqueue invocation are required. There may be a
> long delay before the workqueue item can start which may exceed the time
> to do a single copy if the CPUs on a node are saturated. Furthermore, a
> single thread can preempt operations of other unrelated threads and incur
> CPU cache pollution and future misses on unrelated CPUs. It's compounded by
> the fact that a high priority system workqueue is used to do the operation,
> one that is used for CPU hotplug operations and rolling back when a netdevice
> fails to be registered. It treats a hugepage copy as an essential operation
> that can preempt all other work which is very questionable.
>
> The series leader has no details on a workload that is bottlenecked by
> THP migrations and even if it is, the primary question should be *why*
> THP migrations are so frequent and alleviating that instead of
> preempting multiple CPUs to do the work.
>
>
Mel - I sense on going frustration around some of the THP migration,
migration acceleration, CDM, and other patches.  Here is a 10k foot
description that I hope adds to what John & Anshuman have said in other
threads.

Vendors are currently providing systems that have both traditional
DDR3/4 memory (lets call it 100GB/s) and high bandwidth memory (HBM)
(lets call it 1TB/s) within a single system.  GPUs have been doing this
with HBM on the GPU and DDR on the CPU complex, but they've been
attached via PCIe and thus HBM has been GPU private memory.  The GPU has
managed this memory by effectively mlocking pages on the CPU and copying
the data into the GPU while its being computed on and then copying it
back to the CPU when CPU faults on trying to touch it or the GPU is
done.  Because HBM is limited in capacity (10's of GB max) versus DDR3
(100's+ GB), runtimes like Nvidia's unified memory dynamically page
memory in and out of the GPU to get the benefits of high bandwidth,
while still allowing access a total footprint of system memory.  Its
effectively page protection based CPU/GPU memory coherence.

PCIe attached GPUs+HBM are the bulk or whats out there today and will
continue to be, so there are efforts to try and improve how GPUs (and
other devices in the same PCIe boat) interact with -mm given the
limitations of PCIe (see HMM).

Jumping to what is essentially a different platform - there will be
systems where that same GPU HBM memory is now part of the OS controlled
memory (aka NUMA node) because these systems have a cache coherent link
attaching them (could be NVLINK, QPI, CAPI, HT, or something else)  This
HBM zone might have CPU cores in it, it might have GPU cores in it, or
an FPGA, its not necessarily GPU specific.  NVIDIA has talked about
systems that look like this, as has Intel (KNL with flat memory), and
there are likely others. Systems like this can be thought of (just for
exampke) as 2 NUMA node box where you've got 100GB/s of bandwidth on one
node, 1TB/s on the other, connected via some cache coherent link. That
link is probably order 100GB/s max too (maybe lower, but certainly not
1TB/s yet).

Cores (CPU/GPU/FPGA) can access either NUMA node via the coherent link
(just like a multi-socket CPU box) but you also want to be able to
optimize page placement so that hot pages physically get migrated into
the HBM node from the DDR node. The expectation is that on such systems
either the user, a daemon, or kernel/autonuma is going to be migrating
(TH)pages between the NUMA zones to optimize overall system
bandwidth/throughput.  Because of the 10x discrepancy in memory
bandwidth, despite the best paging policies to optimize for page
locality in the HBM nodes, pages will often still be moving at a high
rate between zones.  This differs from a traditional NUMA system where
moving a page from one 100GB/s node to the other 100GB/s node has
dubious value, like you say.

To your specific question - what workloads benefit from this improved
migration throughput and why THPs?  We have seen that there can be a
1.7x improvement in GPU perf by improving NVLink page migration
bandwidth from 6GB/s->32.5GB/s.  In comparison, 4KB page migration on
x86 over QPI (today) gets < 100MB/s of throughput even though QPI and
NVLink can provide 32GB/s+.  We couldn't cripple the link enough to get
down to ~100MB/s, but obviously using small base page sizes at 100MB/s
of migration throughput would kill performance.  So good THP
functionality + good migration throughput appear critical to us (and
maybe KNL too?).

https://devblogs.nvidia.com/parallelforall/beyond-gpu-memory-limits-unified-memory-pascal/

There's a laundry list of things to make it work well related to THPs,
migration policy and eviction, numa zone isolation, etc.  We hope
building functionality into mm + autonuma is useful for everyone so that
less of it needs to live in proprietary runtimes.

Hope that helps with 10k foot view and a specific use case on why at
least NVIDIA is very interested in optimizing kernel NUMA+THP functionality.

Re: [PATCH 0/6] Enable parallel page migration

[Mel Gorman]

On Thu, Mar 09, 2017 at 11:38:00AM -0600, David Nellans wrote:
> On 03/09/2017 09:09 AM, Mel Gorman wrote:
> > I didn't look into the patches in detail except to get a general feel
> > for how it works and I'm not convinced that it's a good idea at all.
> >
> > I accept that memory bandwidth utilisation may be higher as a result but
> > consider the impact. THP migrations are relatively rare and when they
> > occur, it's in the context of a single thread. To parallelise the copy,
> > an allocation, kmap and workqueue invocation are required. There may be a
> > long delay before the workqueue item can start which may exceed the time
> > to do a single copy if the CPUs on a node are saturated. Furthermore, a
> > single thread can preempt operations of other unrelated threads and incur
> > CPU cache pollution and future misses on unrelated CPUs. It's compounded by
> > the fact that a high priority system workqueue is used to do the operation,
> > one that is used for CPU hotplug operations and rolling back when a netdevice
> > fails to be registered. It treats a hugepage copy as an essential operation
> > that can preempt all other work which is very questionable.
> >
> > The series leader has no details on a workload that is bottlenecked by
> > THP migrations and even if it is, the primary question should be *why*
> > THP migrations are so frequent and alleviating that instead of
> > preempting multiple CPUs to do the work.
> >
> >
>
> Mel - I sense on going frustration around some of the THP migration,
> migration acceleration, CDM, and other patches.  Here is a 10k foot
> description that I hope adds to what John & Anshuman have said in other
> threads.
> 

Hi David,

I recognise the motivation for some of these patches but disagree on the
mechanisms used, more on this later.

> Vendors are currently providing systems that have both traditional
> DDR3/4 memory (lets call it 100GB/s) and high bandwidth memory (HBM)
> (lets call it 1TB/s) within a single system.  GPUs have been doing this
> with HBM on the GPU and DDR on the CPU complex, but they've been
> attached via PCIe and thus HBM has been GPU private memory. 

I completely understand although I'd point out that HBM is slightly
different in that it could be expressed in terms of a hierarchical node
system whereby some nodes migrate to each other -- from faster to slower by
a "migrate on LRU reclaim" and from slower to faster with automatic NUMA
balancing using sampling. However, HBM is extremely specific and dealing
with that is not necessarily compatible with devices that are not coherent.

> <SNIP>

Again, I understand the motivation and have no further comment to make.
In the interest of trying to be helpful, I'll propose an alternative to
this series and expand upon why I think it's problematic.

> the HBM node from the DDR node. The expectation is that on such systems
> either the user, a daemon, or kernel/autonuma is going to be migrating
> (TH)pages between the NUMA zones to optimize overall system
> bandwidth/throughput.  Because of the 10x discrepancy in memory
> bandwidth, despite the best paging policies to optimize for page
> locality in the HBM nodes, pages will often still be moving at a high
> rate between zones.  This differs from a traditional NUMA system where
> moving a page from one 100GB/s node to the other 100GB/s node has
> dubious value, like you say.
> 
> To your specific question - what workloads benefit from this improved
> migration throughput and why THPs? 

So the key potential issue here in my mind is that THP migration is too slow
in some cases. What I object to is improving that using a high priority
workqueue that potentially starves other CPUs and pollutes their cache
which is generally very expensive.

Lets look at the core of what copy_huge_page does in mm/migrate.c which
is the function that gets parallelised by the series in question. For
a !HIGHMEM system, it's woefully inefficient. Historically, it was an
implementation that would work generically which was fine but maybe not
for future systems. It was also fine back when hugetlbfs was the only huge
page implementation and COW operations were incredibly rare on the grounds
due to the risk that they could terminate the process with prejudice.

The function takes a huge page, splits it into PAGE_SIZE chunks, kmap_atomics
the source and destination for each PAGE_SIZE chunk and copies it. The
parallelised version does one kmap and copies it in chunks assuming the
THP is fully mapped and accessible. Fundamentally, this is broken in the
generic sense as the kmap is not guaranteed to make the whole page necessary
but it happens to work on !highmem systems.  What is more important to
note is that it's multiple preempt and pagefault enables and disables
on a per-page basis that happens 512 times (for THP on x86-64 at least),
all of which are expensive operations depending on the kernel config and
I suspect that the parallisation is actually masking that stupid overhead.

At the very least, I would have expected an initial attempt of one patch that
optimised for !highmem systems to ignore kmap, simply disable preempt (if
that is even necessary, I didn't check) and copy a pinned physical->physical
page as a single copy without looping on a PAGE_SIZE basis and see how
much that gained. Do it initially for THP only and worry about gigantic
pages when or if that is a problem.

That would be patch 1 of a series.  Maybe that'll be enough, maybe not but
I feel it's important to optimise the serialised case as much as possible
before considering parallelisation to highlight and justify why it's
necessary[1]. If nothing else, what if two CPUs both parallelise a migration
at the same time and end up preempting each other? Between that and the
workqueue setup, it's potentially much slower than an optimised serial copy.

It would be tempting to experiment but the test case was not even included
with the series (maybe it's somewhere else)[2]. While it's obvious how
such a test case could be constructed, it feels unnecessary to construct
it when it should be in the changelog.

To some extent, CDM suffered from the same pushback. Like this series,
it introduced something new, complex and with high maintenance overhead
without considering whether the existing mechanisms (cpuset, mempolicies
or some combination of both) or out-of tree proposals such as HMM that
could be added to and finalised[3].

[1] If nothing else, it would make it clear to the reviewer that
    additional complexity is 100% justified which is woefully missing in
    this series.

[2] For complex series that have an alleged performance improvement, it
    should always be possible to supply a test case that can demonstrate
    that. Granted, this does not always happen but at least for my own
    series I have the test case in question automated and can point people
    to the repository that stores the test case and if necessary, supply
    instructions on how to reproduce the results.

[3] Preferably with an in-tree user because the lack of such a user was
    one of the major factors that gave HMM a kicking.

-- 
Mel Gorman
SUSE Labs

Re: [PATCH 0/6] Enable parallel page migration

[Zi Yan]

[-- Attachment #1: Type: text/plain, Size: 5035 bytes --]

Hi Mel,

Thanks for pointing out the problems in this patchset.

It was my intern project done in NVIDIA last summer. I only used
micro-benchmarks to demonstrate the big memory bandwidth utilization gap
between base page migration and THP migration along with serialized page
migration vs parallel page migration.

Here are cross-socket serialized page migration results from calling
move_pages() syscall:

In x86_64, a Intel two-socket E5-2640v3 box,
single 4KB base page migration takes 62.47 us, using 0.06 GB/s BW,
single 2MB THP migration takes 658.54 us, using 2.97 GB/s BW,
512 4KB base page migration takes 1987.38 us, using 0.98 GB/s BW.

In ppc64, a two-socket Power8 box,
single 64KB base page migration takes 49.3 us, using 1.24 GB/s BW,
single 16MB THP migration takes 2202.17 us, using 7.10 GB/s BW,
256 64KB base page migration takes 2543.65 us, using 6.14 GB/s BW.

THP migration is not slow at all when compared to a group of equivalent
base page migrations.

For 1-thread vs 8-thread THP migration:
In x86_64,
1-thread 2MB THP migration takes 658.54 us, using 2.97 GB/s BW,
8-thread 2MB THP migration takes 227.76 us, using 8.58 GB/s BW.

In ppc64,
1-thread 16MB THP migration takes 2202.17 us, using 7.10 GB/s BW,
8-thread 16MB THP migration takes 1223.87 us, using 12.77 GB/s BW.

This big increase on BW utilization is the motivation of pushing this
patchset.

> 
> So the key potential issue here in my mind is that THP migration is too slow
> in some cases. What I object to is improving that using a high priority
> workqueue that potentially starves other CPUs and pollutes their cache
> which is generally very expensive.

I might not completely agree with this. Using a high priority workqueue
can guarantee page migration work is done ASAP. Otherwise, we completely
lose the speedup brought by parallel page migration, if data copy
threads have to wait.

I understand your concern on CPU utilization impact. I think checking
CPU utilization and only using idle CPUs could potentially avoid this
problem.

> 
> Lets look at the core of what copy_huge_page does in mm/migrate.c which
> is the function that gets parallelised by the series in question. For
> a !HIGHMEM system, it's woefully inefficient. Historically, it was an
> implementation that would work generically which was fine but maybe not
> for future systems. It was also fine back when hugetlbfs was the only huge
> page implementation and COW operations were incredibly rare on the grounds
> due to the risk that they could terminate the process with prejudice.
> 
> The function takes a huge page, splits it into PAGE_SIZE chunks, kmap_atomics
> the source and destination for each PAGE_SIZE chunk and copies it. The
> parallelised version does one kmap and copies it in chunks assuming the
> THP is fully mapped and accessible. Fundamentally, this is broken in the
> generic sense as the kmap is not guaranteed to make the whole page necessary
> but it happens to work on !highmem systems.  What is more important to
> note is that it's multiple preempt and pagefault enables and disables
> on a per-page basis that happens 512 times (for THP on x86-64 at least),
> all of which are expensive operations depending on the kernel config and
> I suspect that the parallisation is actually masking that stupid overhead.

You are right on kmap, I think making this patchset depend on !HIGHMEM
can avoid the problem. It might not make sense to kmap potentially 512
base pages to migrate a THP in a system with highmem.

> 
> At the very least, I would have expected an initial attempt of one patch that
> optimised for !highmem systems to ignore kmap, simply disable preempt (if
> that is even necessary, I didn't check) and copy a pinned physical->physical
> page as a single copy without looping on a PAGE_SIZE basis and see how
> much that gained. Do it initially for THP only and worry about gigantic
> pages when or if that is a problem.

I can try this out to show how much improvement we can obtain from
existing THP migration, which is shown in the data above.

> 
> That would be patch 1 of a series.  Maybe that'll be enough, maybe not but
> I feel it's important to optimise the serialised case as much as possible
> before considering parallelisation to highlight and justify why it's
> necessary[1]. If nothing else, what if two CPUs both parallelise a migration
> at the same time and end up preempting each other? Between that and the
> workqueue setup, it's potentially much slower than an optimised serial copy.
> 
> It would be tempting to experiment but the test case was not even included
> with the series (maybe it's somewhere else)[2]. While it's obvious how
> such a test case could be constructed, it feels unnecessary to construct
> it when it should be in the changelog.

Do you mean performing multiple parallel page migrations at the same
time and show all the page migration time?


-- 
Best Regards,
Yan Zi


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 537 bytes --]

Re: [kbuild-all] [PATCH 6/6] sysctl: Add global tunable mt_page_copy

[Ye Xiaolong]

On 03/08, Anshuman Khandual wrote:
>On 02/17/2017 09:00 PM, kbuild test robot wrote:
>> Hi Zi,
>> 
>> [auto build test ERROR on linus/master]
>> [also build test ERROR on v4.10-rc8 next-20170217]
>> [if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
>> 
>> url:    https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Enable-parallel-page-migration/20170217-200523
>> config: i386-randconfig-a0-02131010 (attached as .config)
>> compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
>
>Though I dont have the same compiler, I am unable to reproduce this
>build failure exactly. The build fails but for a different symbol.

I think previous "undefined reference to `mt_page_copy'" error is due to kbuild
bot didn't set CONFIG_MIGRATION (see attached config in original mail) since it
is a randconfig test.

Thanks,
Xiaolong

>I have the following gcc version but does it really make a
>difference with respect to finding the symbol etc ?
>
>gcc (Ubuntu 4.9.2-10ubuntu13) 4.9.2
>
>
>mm/memory.c: In function ‘copy_pmd_range’:
>mm/memory.c:1002:3: error: implicit declaration of function
>‘pmd_related’ [-Werror=implicit-function-declaration]
>   if (pmd_related(*src_pmd)) {
>   ^
>cc1: some warnings being treated as errors
>scripts/Makefile.build:294: recipe for target 'mm/memory.o' failed
>
>_______________________________________________
>kbuild-all mailing list
>kbuild-all@lists.01.org
>https://lists.01.org/mailman/listinfo/kbuild-all

Re: [kbuild-all] [PATCH 6/6] sysctl: Add global tunable mt_page_copy

[Anshuman Khandual]

On 03/10/2017 06:42 AM, Ye Xiaolong wrote:
> On 03/08, Anshuman Khandual wrote:
>> On 02/17/2017 09:00 PM, kbuild test robot wrote:
>>> Hi Zi,
>>>
>>> [auto build test ERROR on linus/master]
>>> [also build test ERROR on v4.10-rc8 next-20170217]
>>> [if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
>>>
>>> url:    https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Enable-parallel-page-migration/20170217-200523
>>> config: i386-randconfig-a0-02131010 (attached as .config)
>>> compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
>> Though I dont have the same compiler, I am unable to reproduce this
>> build failure exactly. The build fails but for a different symbol.
> I think previous "undefined reference to `mt_page_copy'" error is due to kbuild
> bot didn't set CONFIG_MIGRATION (see attached config in original mail) since it
> is a randconfig test.

If CONFIG_MIGRATION is not set then mm/migrate.c never gets compiled
and the symbol 'mt_page_copy' is never exported for kernel/sysctl.c
based extern variable to use. Sure, will fix it by keeping all the
code in kernel/sysctl.c within CONFIG_MIGRATION config. Thanks for
pointing it out.

Re: [PATCH 0/6] Enable parallel page migration

[Michal Hocko]

On Thu 09-03-17 15:09:04, Mel Gorman wrote:
> On Wed, Mar 08, 2017 at 09:34:27PM +0530, Anshuman Khandual wrote:
> > > Any comments, suggestions are welcome.
> > 
> > Hello Vlastimil/Michal/Minchan/Mel/Dave,
> > 
> > Apart from the comments from Naoya on a different thread posted by Zi
> > Yan, I did not get any more review comments on this series. Could you
> > please kindly have a look on the over all design and its benefits from
> > page migration performance point of view and let me know your views.
> > Thank you.
> > 
> 
> I didn't look into the patches in detail except to get a general feel
> for how it works and I'm not convinced that it's a good idea at all.
> 
> I accept that memory bandwidth utilisation may be higher as a result but
> consider the impact. THP migrations are relatively rare and when they
> occur, it's in the context of a single thread. To parallelise the copy,
> an allocation, kmap and workqueue invocation are required. There may be a
> long delay before the workqueue item can start which may exceed the time
> to do a single copy if the CPUs on a node are saturated. Furthermore, a
> single thread can preempt operations of other unrelated threads and incur
> CPU cache pollution and future misses on unrelated CPUs. It's compounded by
> the fact that a high priority system workqueue is used to do the operation,
> one that is used for CPU hotplug operations and rolling back when a netdevice
> fails to be registered. It treats a hugepage copy as an essential operation
> that can preempt all other work which is very questionable.
> 
> The series leader has no details on a workload that is bottlenecked by
> THP migrations and even if it is, the primary question should be *why*
> THP migrations are so frequent and alleviating that instead of
> preempting multiple CPUs to do the work.

FWIW I very much agree here and the follow up reply. Making migration
itself parallel is a hard task. You should start simple and optimize the
current code first and each step accompany with numbers. Parallel
migration should be the very last step - if it is needed at all of
course. I am quite skeptical that a reasonable parallel load balancing
is achievable without a large maintenance cost and/or predictable
behavior. 
-- 
Michal Hocko
SUSE Labs

Re: [PATCH 0/6] Enable parallel page migration

[Mel Gorman]

On Thu, Mar 09, 2017 at 05:46:16PM -0600, Zi Yan wrote:
> Hi Mel,
> 
> Thanks for pointing out the problems in this patchset.
> 
> It was my intern project done in NVIDIA last summer. I only used
> micro-benchmarks to demonstrate the big memory bandwidth utilization gap
> between base page migration and THP migration along with serialized page
> migration vs parallel page migration.
> 

The measurement itself is not a problem. It clearly shows why you were
doing it and indicates that it's possible.

> <SNIP>
> This big increase on BW utilization is the motivation of pushing this
> patchset.
> 

As before, I have no problem with the motivation, my problem is with the
approach and in particular that the serialised case was not optimised first.

> > 
> > So the key potential issue here in my mind is that THP migration is too slow
> > in some cases. What I object to is improving that using a high priority
> > workqueue that potentially starves other CPUs and pollutes their cache
> > which is generally very expensive.
> 
> I might not completely agree with this. Using a high priority workqueue
> can guarantee page migration work is done ASAP.

Yes, but at the cost of stalling other operations that are happening at
the same tiime. The series assumes that the migration is definitely the
most important operation going on at the moment.

> Otherwise, we completely
> lose the speedup brought by parallel page migration, if data copy
> threads have to wait.
> 

And conversely, if important threads were running on the other CPUs at
the time the migration started then they might be equally unhappy.

> I understand your concern on CPU utilization impact. I think checking
> CPU utilization and only using idle CPUs could potentially avoid this
> problem.
> 

That will be costly to detect actually. It would require poking into the
scheduler core and incurring a number of cache misses for a race-prone
operation that may not succeed. Even if you do it, it'll still be
brought up that the serialised case should be optimised first.

> > The function takes a huge page, splits it into PAGE_SIZE chunks, kmap_atomics
> > the source and destination for each PAGE_SIZE chunk and copies it. The
> > parallelised version does one kmap and copies it in chunks assuming the
> > THP is fully mapped and accessible. Fundamentally, this is broken in the
> > generic sense as the kmap is not guaranteed to make the whole page necessary
> > but it happens to work on !highmem systems.  What is more important to
> > note is that it's multiple preempt and pagefault enables and disables
> > on a per-page basis that happens 512 times (for THP on x86-64 at least),
> > all of which are expensive operations depending on the kernel config and
> > I suspect that the parallisation is actually masking that stupid overhead.
> 
> You are right on kmap, I think making this patchset depend on !HIGHMEM
> can avoid the problem. It might not make sense to kmap potentially 512
> base pages to migrate a THP in a system with highmem.
> 

One concern I have is that the series benefitted the most by simply batching
all those operations even if it was not intended.

> > At the very least, I would have expected an initial attempt of one patch that
> > optimised for !highmem systems to ignore kmap, simply disable preempt (if
> > that is even necessary, I didn't check) and copy a pinned physical->physical
> > page as a single copy without looping on a PAGE_SIZE basis and see how
> > much that gained. Do it initially for THP only and worry about gigantic
> > pages when or if that is a problem.
> 
> I can try this out to show how much improvement we can obtain from
> existing THP migration, which is shown in the data above.
> 

It would be important to do so. There would need to be absolute proof
that parallelisation is required and even then the concerns about
interfering with workloads on other CPUs is not going to be easy to
handle.

> > That would be patch 1 of a series.  Maybe that'll be enough, maybe not but
> > I feel it's important to optimise the serialised case as much as possible
> > before considering parallelisation to highlight and justify why it's
> > necessary[1]. If nothing else, what if two CPUs both parallelise a migration
> > at the same time and end up preempting each other? Between that and the
> > workqueue setup, it's potentially much slower than an optimised serial copy.
> > 
> > It would be tempting to experiment but the test case was not even included
> > with the series (maybe it's somewhere else)[2]. While it's obvious how
> > such a test case could be constructed, it feels unnecessary to construct
> > it when it should be in the changelog.
> 
> Do you mean performing multiple parallel page migrations at the same
> time and show all the page migration time?

I mean that the test case that was used to generate the bandwidth
utilisation figures should be included.

-- 
Mel Gorman
SUSE Labs

Re: [PATCH 0/6] Enable parallel page migration

[Michal Hocko]

On Fri 10-03-17 14:07:16, Mel Gorman wrote:
> On Thu, Mar 09, 2017 at 05:46:16PM -0600, Zi Yan wrote:
[...]
> > I understand your concern on CPU utilization impact. I think checking
> > CPU utilization and only using idle CPUs could potentially avoid this
> > problem.
> > 
> 
> That will be costly to detect actually. It would require poking into the
> scheduler core and incurring a number of cache misses for a race-prone
> operation that may not succeed. Even if you do it, it'll still be
> brought up that the serialised case should be optimised first.

do not forget that seeing idle cpus is not a sufficient criterion to use
it for parallel migration. There might be other policies you are not
aware of from the MM code to keep them idle (power saving and who knows
what else). Developing a reasonable strategy for spreading the load to
different CPUs is really hard, much harder than you can imaging I
suspect (just look at how hard it was and I long it took to get to a
reasonable scheduler driven frequency scaling/power governors).
-- 
Michal Hocko
SUSE Labs