linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v2 0/5] propose auto-run mode of ksm and its tests
@ 2022-08-12 10:11 cgel.zte
  2022-08-12 10:11 ` [PATCH v2 1/5] ksm: add a auto-run mode of ksm cgel.zte
                   ` (4 more replies)
  0 siblings, 5 replies; 6+ messages in thread
From: cgel.zte @ 2022-08-12 10:11 UTC (permalink / raw)
  To: akpm, willy; +Cc: hughd, izik.eidus, linux-kernel, linux-mm, xu.xin16

From: xu xin <xu.xin16@zte.com.cn>

The following patch series bring a NEW running state "auto mode". In
traditional KSM, whether ksmd works scanning and merging or not
depends on the sysfs klob ksm_run.

Most of time, letting ksmd run is not very much needed, for example,
when memory is sufficient, because it increases the delays of COW for
user applications and also consume extra cpu resource. Besides, the fixed
pages_to_scan is not always good. When there are a lot of same pages,
the default pages_to_scan makes ksmd so slow to merge them.

The four patches try to optimize the above two points with a auto mode.
It's a lightweight optimization to KSM, not like uksm or pksm and so on.

The tests show:
	The convergence time of merging time of auto-mode reduces can low to
	HALF of normal-mode. The cpu consumption of auto-mode also get less
	than normal-mode.

There are two type of TEST which have different shape of samepage areas.
One is full-samepages vm areas, and the other is sparse-samepage areas.
Both of the madvise area'size are 300MB.

Here is test log:
****************** TEST 1 ******************
Now the shape of memory area is full-samepages:
[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]

Start to test normal-run ksmd...
ksm current cpu total slice: 15274
ksm normal-run's merging time: 45.596018 seconds
ksm current cpu total slice: 16370
Start to test auto-run ksmd...
ksm auto-run's merging time: 17.599391 seconds
ksm current cpu total slice: 16488

****************** TEST 2 ******************
Now the shape of memory area is sparse-samepages:
[xx]          [xx]          [xx]   

Start to test normal-run ksmd...
ksm current cpu total slice: 16532
ksm normal-run's merging time: 21.625451 seconds
ksm current cpu total slice: 16833
Start to test auto-run ksmd...
ksm auto-run's merging time: 17.684379 seconds
ksm current cpu total slice: 16950

Change log
==========
for v2:
	1. We adjust the scan-enhanced algorithm, abstract its implementation
	into a single function, which is clearer and smoother.
	2. Add tests of auto-mode of ksm to verify the effectiveness.

xu xin (5):
  ksm: add a auto-run mode of ksm
  ksm: implement scan-enhanced algorithm
  ksm: let ksmd auto-work with memory threshold
  ksm: show ksmd status for auto mode
  ksm: add tests of ksm auto mode

 mm/ksm.c                                   | 250 ++++++++++++++++++-
 tools/testing/selftests/vm/.gitignore      |   1 +
 tools/testing/selftests/vm/Makefile        |   1 +
 tools/testing/selftests/vm/test-ksm-auto.c | 273 +++++++++++++++++++++
 4 files changed, 519 insertions(+), 6 deletions(-)
 create mode 100644 tools/testing/selftests/vm/test-ksm-auto.c

-- 
2.25.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH v2 1/5] ksm: add a auto-run mode of ksm
  2022-08-12 10:11 [PATCH v2 0/5] propose auto-run mode of ksm and its tests cgel.zte
@ 2022-08-12 10:11 ` cgel.zte
  2022-08-12 10:12 ` [PATCH v2 2/5] ksm: implement scan-enhanced algorithm cgel.zte
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 6+ messages in thread
From: cgel.zte @ 2022-08-12 10:11 UTC (permalink / raw)
  To: akpm, willy; +Cc: hughd, izik.eidus, linux-kernel, linux-mm, xu.xin16, CGEL

From: xu xin <xu.xin16@zte.com.cn>

Add a new running state auto-mode to ksm. This is to pave the way
for subsequent real optimization features.

Use it by: echo 8 > /sys/kernel/mm/ksm/run

Signed-off-by: xu xin <xu.xin16@zte.com.cn>
Signed-off-by: CGEL <cgel.zte@gmail.com>
---
 mm/ksm.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/mm/ksm.c b/mm/ksm.c
index 2f315c69fa2c..c80d908221a4 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -290,6 +290,7 @@ static int ksm_nr_node_ids = 1;
 #define KSM_RUN_MERGE	1
 #define KSM_RUN_UNMERGE	2
 #define KSM_RUN_OFFLINE	4
+#define KSM_RUN_AUTO	8
 static unsigned long ksm_run = KSM_RUN_STOP;
 static void wait_while_offlining(void);
 
@@ -2416,7 +2417,9 @@ static void ksm_do_scan(unsigned int scan_npages)
 
 static int ksmd_should_run(void)
 {
-	return (ksm_run & KSM_RUN_MERGE) && !list_empty(&ksm_mm_head.mm_list);
+	if (!list_empty(&ksm_mm_head.mm_list))
+		return ksm_run & KSM_RUN_AUTO || ksm_run & KSM_RUN_MERGE;
+	return 0;
 }
 
 static int ksm_scan_thread(void *nothing)
@@ -2916,7 +2919,7 @@ static ssize_t run_store(struct kobject *kobj, struct kobj_attribute *attr,
 	err = kstrtouint(buf, 10, &flags);
 	if (err)
 		return -EINVAL;
-	if (flags > KSM_RUN_UNMERGE)
+	if (flags > KSM_RUN_UNMERGE && flags != KSM_RUN_AUTO)
 		return -EINVAL;
 
 	/*
@@ -2942,7 +2945,7 @@ static ssize_t run_store(struct kobject *kobj, struct kobj_attribute *attr,
 	}
 	mutex_unlock(&ksm_thread_mutex);
 
-	if (flags & KSM_RUN_MERGE)
+	if (flags & KSM_RUN_MERGE || flags & KSM_RUN_AUTO)
 		wake_up_interruptible(&ksm_thread_wait);
 
 	return count;
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 6+ messages in thread

* [PATCH v2 2/5] ksm: implement scan-enhanced algorithm
  2022-08-12 10:11 [PATCH v2 0/5] propose auto-run mode of ksm and its tests cgel.zte
  2022-08-12 10:11 ` [PATCH v2 1/5] ksm: add a auto-run mode of ksm cgel.zte
@ 2022-08-12 10:12 ` cgel.zte
  2022-08-12 10:12 ` [PATCH v2 3/5] ksm: let ksmd auto-work with memory threshold cgel.zte
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 6+ messages in thread
From: cgel.zte @ 2022-08-12 10:12 UTC (permalink / raw)
  To: akpm, willy; +Cc: hughd, izik.eidus, linux-kernel, linux-mm, xu.xin16, CGEL

From: xu xin <xu.xin16@zte.com.cn>

Implement the scan-enhanced algorithm of auto mode. In this algorithm,
after every time of scanning, if new ksm pages are obtained, it will
double pages_to_scan for the next scanning until the general
multiplying factor is not less than max_scanning_factor. If no new ksm
pages are obtained, then reset pages_to_scan to the default value.

We add the sysfs klob of max_scanning_factor to limit scanning factor's
excessive growth.

Signed-off-by: CGEL <cgel.zte@gmail.com>
Signed-off-by: xu xin <xu.xin16@zte.com.cn>
---
 mm/ksm.c | 100 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 98 insertions(+), 2 deletions(-)

diff --git a/mm/ksm.c b/mm/ksm.c
index c80d908221a4..f416f168a6da 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -131,6 +131,10 @@ struct mm_slot {
  * @address: the next address inside that to be scanned
  * @rmap_list: link to the next rmap to be scanned in the rmap_list
  * @seqnr: count of completed full scans (needed when removing unstable node)
+ * @new_ksmpages: count of the new merged KSM pages in the current scanning
+ *	              of mm_lists (cleared after every turn of ksm_do_scan() ends)
+ * @prev_ksmpages: the record of the new merged KSM pages in the last turn of
+ *	               scanning by ksm_do_scan().
  *
  * There is only the one ksm_scan instance of this cursor structure.
  */
@@ -139,6 +143,8 @@ struct ksm_scan {
 	unsigned long address;
 	struct rmap_item **rmap_list;
 	unsigned long seqnr;
+	unsigned long new_ksmpages;
+	unsigned long prev_ksmpages;
 };
 
 /**
@@ -277,6 +283,19 @@ static unsigned int zero_checksum __read_mostly;
 /* Whether to merge empty (zeroed) pages with actual zero pages */
 static bool ksm_use_zero_pages __read_mostly;
 
+/*
+ * Work in auto-mode.
+ * The multiplicative factor of pages_to_scan.
+ * Real pages to scan equals to the product of scanning_factor
+ * and pages_to_scan
+ */
+#define INIT_SCANNING_FACTOR	1
+static unsigned int scanning_factor = INIT_SCANNING_FACTOR;
+
+/* The upper limit of scanning_factor */
+#define DEFAULT_MAX_SCANNING_FACTOR	16
+static unsigned int max_scanning_factor	= DEFAULT_MAX_SCANNING_FACTOR;
+
 #ifdef CONFIG_NUMA
 /* Zeroed when merging across nodes is not allowed */
 static unsigned int ksm_merge_across_nodes = 1;
@@ -2031,6 +2050,8 @@ static void stable_tree_append(struct rmap_item *rmap_item,
 	rmap_item->address |= STABLE_FLAG;
 	hlist_add_head(&rmap_item->hlist, &stable_node->hlist);
 
+	ksm_scan.new_ksmpages++;
+
 	if (rmap_item->hlist.next)
 		ksm_pages_sharing++;
 	else
@@ -2422,6 +2443,41 @@ static int ksmd_should_run(void)
 	return 0;
 }
 
+/*
+ * Work in auto mode, the scan-enhanced algorithm.
+ * current_factor: the current scanning_factor.
+ * return: the scanning_factor caculated by scan-enhanced algorithm.
+ */
+static unsigned int scan_enhanced_algorithm(unsigned int current_factor)
+{
+	unsigned int next_factor;
+	unsigned int max, min;
+
+	/*
+	 * The calculation is divied into three cases as follows:
+	 *
+	 * Case 1: when new_ksmpages > prev_ksmpages * 1/2, get the
+	 *         next factor by double the current factor.
+	 * Case 2: when 0 < new_ksmpages < prev_ksmpages * 1/2, keep
+	 *         the factor unchanged.
+	 * Case 3: when new_ksmpages equals 0, then get the next
+	 *         factor by halfing the current factor.
+	 */
+	max = READ_ONCE(max_scanning_factor);
+	min = INIT_SCANNING_FACTOR;
+	if (ksm_scan.new_ksmpages * 2 > ksm_scan.prev_ksmpages) {
+		next_factor = current_factor << 1; /* Doubling */
+		if (next_factor > max)
+			next_factor = max;
+	} else if (ksm_scan.new_ksmpages == 0) {
+		next_factor = current_factor >> 1; /* Halfing */
+		next_factor = next_factor < min ? min : next_factor;
+	} else
+		next_factor = current_factor;
+
+	return next_factor;
+}
+
 static int ksm_scan_thread(void *nothing)
 {
 	unsigned int sleep_ms;
@@ -2432,8 +2488,19 @@ static int ksm_scan_thread(void *nothing)
 	while (!kthread_should_stop()) {
 		mutex_lock(&ksm_thread_mutex);
 		wait_while_offlining();
-		if (ksmd_should_run())
-			ksm_do_scan(ksm_thread_pages_to_scan);
+		if (ksmd_should_run()) {
+			if (ksm_run & KSM_RUN_AUTO) {
+				ksm_do_scan(ksm_thread_pages_to_scan * scanning_factor);
+
+				scanning_factor = scan_enhanced_algorithm(scanning_factor);
+				/*
+				 * Reset ksm_scan.new_ksmpages after
+				 * updating scanning_factor by scan_enhanced_algorithm.
+				 */
+				ksm_scan.new_ksmpages = 0;
+			} else
+				ksm_do_scan(ksm_thread_pages_to_scan);
+		}
 		mutex_unlock(&ksm_thread_mutex);
 
 		try_to_freeze();
@@ -2904,6 +2971,34 @@ static ssize_t pages_to_scan_store(struct kobject *kobj,
 }
 KSM_ATTR(pages_to_scan);
 
+static ssize_t max_scanning_factor_show(struct kobject *kobj,
+						struct kobj_attribute *attr, char *buf)
+{
+	return sysfs_emit(buf, "%u\n", max_scanning_factor);
+}
+
+static ssize_t max_scanning_factor_store(struct kobject *kobj,
+								struct kobj_attribute *attr,
+								const char *buf, size_t count)
+{
+		unsigned int value, max;
+		int err;
+
+		err = kstrtouint(buf, 10, &value);
+		if (err)
+			return -EINVAL;
+
+		max = totalram_pages() / ksm_thread_pages_to_scan;
+
+		if (value < 1 && value > max)
+			return -EINVAL;
+
+		max_scanning_factor = value;
+
+		return count;
+}
+KSM_ATTR(max_scanning_factor);
+
 static ssize_t run_show(struct kobject *kobj, struct kobj_attribute *attr,
 			char *buf)
 {
@@ -3161,6 +3256,7 @@ KSM_ATTR_RO(full_scans);
 static struct attribute *ksm_attrs[] = {
 	&sleep_millisecs_attr.attr,
 	&pages_to_scan_attr.attr,
+	&max_scanning_factor_attr.attr,
 	&run_attr.attr,
 	&pages_shared_attr.attr,
 	&pages_sharing_attr.attr,
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 6+ messages in thread

* [PATCH v2 3/5] ksm: let ksmd auto-work with memory threshold
  2022-08-12 10:11 [PATCH v2 0/5] propose auto-run mode of ksm and its tests cgel.zte
  2022-08-12 10:11 ` [PATCH v2 1/5] ksm: add a auto-run mode of ksm cgel.zte
  2022-08-12 10:12 ` [PATCH v2 2/5] ksm: implement scan-enhanced algorithm cgel.zte
@ 2022-08-12 10:12 ` cgel.zte
  2022-08-12 10:12 ` [PATCH v2 4/5] ksm: show ksmd status for auto mode cgel.zte
  2022-08-12 10:12 ` [PATCH v2 5/5] ksm: add tests of ksm " cgel.zte
  4 siblings, 0 replies; 6+ messages in thread
From: cgel.zte @ 2022-08-12 10:12 UTC (permalink / raw)
  To: akpm, willy; +Cc: hughd, izik.eidus, linux-kernel, linux-mm, xu.xin16, CGEL

From: xu xin <xu.xin16@zte.com.cn>

When memory is sufficient, merging pages to save memory is not very
much needed, and it also inceases delays of COW for user application.

So set a memory threshold, when free memory is lower than the threshold,
ksmd will be triggered to compare and merge pages. And to avoid ping-pong
effect due to the threshold, ksmd needs to try to merge pages until free
memory is larger than (threshold + total_memory * 1/16).

Before free memory is lower than the threshold, ksmd will still scan pages
at a very low speed, to calculate their checksum but not to compare and
merge pages.

        |
        |       ----(Threshold + total_memory/16)--------
        |                              |
------Threshold------                  |
        |                              |
        |_____ksmd try to merge pages__|

We also add a new sysfs klob auto_threshold_percent for user to be able
to tune.

Signed-off-by: xu xin <xu.xin16@zte.com.cn>
Signed-off-by: CGEL <cgel.zte@gmail.com>
---
 mm/ksm.c | 103 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 101 insertions(+), 2 deletions(-)

diff --git a/mm/ksm.c b/mm/ksm.c
index f416f168a6da..c5fd4f520f4a 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -296,6 +296,17 @@ static unsigned int scanning_factor = INIT_SCANNING_FACTOR;
 #define DEFAULT_MAX_SCANNING_FACTOR	16
 static unsigned int max_scanning_factor	= DEFAULT_MAX_SCANNING_FACTOR;
 
+/*
+ * Work in auto mode.
+ * Value: 0~100. Default 20 means "20%". When free memory is lower
+ * than this total memory * ksm_auto_threshold/100, auto_triggered
+ * will be set true.
+ */
+unsigned int ksm_auto_threshold = 20;
+
+/* Work in auto-mode. Whether trigger ksmd to compare and merge pages */
+static bool auto_triggered;
+
 #ifdef CONFIG_NUMA
 /* Zeroed when merging across nodes is not allowed */
 static unsigned int ksm_merge_across_nodes = 1;
@@ -2431,11 +2442,61 @@ static void ksm_do_scan(unsigned int scan_npages)
 		rmap_item = scan_get_next_rmap_item(&page);
 		if (!rmap_item)
 			return;
-		cmp_and_merge_page(page, rmap_item);
+		if (ksm_run & KSM_RUN_AUTO  && !auto_triggered) {
+			/*
+			 * This should happens only when ksm_run is KSM_RUN_AUTO
+			 * and free memory threshold still not reached.
+			 * The reason to calculate it's checksum is to reduce the
+			 * waiting time the rmap_item is added to unstable tree.
+			 */
+			rmap_item->oldchecksum = calc_checksum(page);
+		} else
+			cmp_and_merge_page(page, rmap_item);
+
 		put_page(page);
 	}
 }
 
+#define RIGHT_SHIFT_FOUR_BIT	4
+/* Work in auto mode, should reset auto_triggered ? */
+static bool should_stop_ksmd_to_merge(void)
+{
+	unsigned long total_ram_pages, free_pages;
+	unsigned int threshold;
+
+	total_ram_pages = totalram_pages();
+	free_pages = global_zone_page_state(NR_FREE_PAGES);
+	threshold = READ_ONCE(ksm_auto_threshold);
+
+	return free_pages > (total_ram_pages * threshold / 100) +
+		        (total_ram_pages >> RIGHT_SHIFT_FOUR_BIT);
+}
+
+/* Work in auto mode, should ksmd start to merge ? */
+static bool should_trigger_ksmd_to_merge(void)
+{
+	unsigned long total_ram_pages, free_pages;
+	unsigned int threshold;
+
+	total_ram_pages = totalram_pages();
+	free_pages = global_zone_page_state(NR_FREE_PAGES);
+	threshold = READ_ONCE(ksm_auto_threshold);
+
+	return free_pages < (total_ram_pages * threshold / 100);
+}
+
+static inline void trigger_ksmd_to_merge(void)
+{
+	if (!auto_triggered)
+		auto_triggered = true;
+}
+
+static inline void stop_ksmd_to_merge(void)
+{
+	if (auto_triggered)
+		auto_triggered = false;
+}
+
 static int ksmd_should_run(void)
 {
 	if (!list_empty(&ksm_mm_head.mm_list))
@@ -2478,6 +2539,8 @@ static unsigned int scan_enhanced_algorithm(unsigned int current_factor)
 	return next_factor;
 }
 
+#define SLOW_SCAN_PAGES	5 /* Used when ksmd is not triggered to merge*/
+
 static int ksm_scan_thread(void *nothing)
 {
 	unsigned int sleep_ms;
@@ -2490,7 +2553,10 @@ static int ksm_scan_thread(void *nothing)
 		wait_while_offlining();
 		if (ksmd_should_run()) {
 			if (ksm_run & KSM_RUN_AUTO) {
-				ksm_do_scan(ksm_thread_pages_to_scan * scanning_factor);
+				if (!auto_triggered)
+					ksm_do_scan(SLOW_SCAN_PAGES);
+				else
+					ksm_do_scan(ksm_thread_pages_to_scan * scanning_factor);
 
 				scanning_factor = scan_enhanced_algorithm(scanning_factor);
 				/*
@@ -2498,6 +2564,11 @@ static int ksm_scan_thread(void *nothing)
 				 * updating scanning_factor by scan_enhanced_algorithm.
 				 */
 				ksm_scan.new_ksmpages = 0;
+
+				if (should_trigger_ksmd_to_merge())
+					trigger_ksmd_to_merge();
+				else if (should_stop_ksmd_to_merge())
+					stop_ksmd_to_merge();
 			} else
 				ksm_do_scan(ksm_thread_pages_to_scan);
 		}
@@ -3047,6 +3118,32 @@ static ssize_t run_store(struct kobject *kobj, struct kobj_attribute *attr,
 }
 KSM_ATTR(run);
 
+static ssize_t auto_threshold_show(struct kobject *kobj,
+						struct kobj_attribute *attr, char *buf)
+{
+	return sysfs_emit(buf, "%u\n", ksm_auto_threshold);
+}
+
+static ssize_t auto_threshold_store(struct kobject *kobj,
+								struct kobj_attribute *attr,
+								const char *buf, size_t count)
+{
+	unsigned int value;
+	int err;
+
+	err = kstrtouint(buf, 10, &value);
+	if (err)
+		return -EINVAL;
+
+	if (value > 100)
+		return -EINVAL;
+
+	ksm_auto_threshold = value;
+
+	return count;
+}
+KSM_ATTR(auto_threshold);
+
 #ifdef CONFIG_NUMA
 static ssize_t merge_across_nodes_show(struct kobject *kobj,
 				       struct kobj_attribute *attr, char *buf)
@@ -3258,6 +3355,7 @@ static struct attribute *ksm_attrs[] = {
 	&pages_to_scan_attr.attr,
 	&max_scanning_factor_attr.attr,
 	&run_attr.attr,
+	&auto_threshold_attr.attr,
 	&pages_shared_attr.attr,
 	&pages_sharing_attr.attr,
 	&pages_unshared_attr.attr,
@@ -3289,6 +3387,7 @@ static int __init ksm_init(void)
 	zero_checksum = calc_checksum(ZERO_PAGE(0));
 	/* Default to false for backwards compatibility */
 	ksm_use_zero_pages = false;
+	auto_triggered = false;
 
 	err = ksm_slab_init();
 	if (err)
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 6+ messages in thread

* [PATCH v2 4/5] ksm: show ksmd status for auto mode
  2022-08-12 10:11 [PATCH v2 0/5] propose auto-run mode of ksm and its tests cgel.zte
                   ` (2 preceding siblings ...)
  2022-08-12 10:12 ` [PATCH v2 3/5] ksm: let ksmd auto-work with memory threshold cgel.zte
@ 2022-08-12 10:12 ` cgel.zte
  2022-08-12 10:12 ` [PATCH v2 5/5] ksm: add tests of ksm " cgel.zte
  4 siblings, 0 replies; 6+ messages in thread
From: cgel.zte @ 2022-08-12 10:12 UTC (permalink / raw)
  To: akpm, willy; +Cc: hughd, izik.eidus, linux-kernel, linux-mm, xu.xin16, CGEL

From: xu xin <xu.xin16@zte.com.cn>

Add a sysfs interface of ksmd_status to show some details related
with auto-mode.

Signed-off-by: xu xin <xu.xin16@zte.com.cn>
Signed-off-by: CGEL <cgel.zte@gmail.com>
---
 mm/ksm.c | 42 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 41 insertions(+), 1 deletion(-)

diff --git a/mm/ksm.c b/mm/ksm.c
index c5fd4f520f4a..478bcf26bfcd 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -307,6 +307,9 @@ unsigned int ksm_auto_threshold = 20;
 /* Work in auto-mode. Whether trigger ksmd to compare and merge pages */
 static bool auto_triggered;
 
+/* Count of times that ksmd is triggered due to low free memory */
+static unsigned long triggered_times;
+
 #ifdef CONFIG_NUMA
 /* Zeroed when merging across nodes is not allowed */
 static unsigned int ksm_merge_across_nodes = 1;
@@ -2487,8 +2490,10 @@ static bool should_trigger_ksmd_to_merge(void)
 
 static inline void trigger_ksmd_to_merge(void)
 {
-	if (!auto_triggered)
+	if (!auto_triggered) {
+		triggered_times++;
 		auto_triggered = true;
+	}
 }
 
 static inline void stop_ksmd_to_merge(void)
@@ -3118,6 +3123,40 @@ static ssize_t run_store(struct kobject *kobj, struct kobj_attribute *attr,
 }
 KSM_ATTR(run);
 
+static ssize_t ksmd_status_show(struct kobject *kobj,
+				struct kobj_attribute *attr, char *buf)
+{
+	int len = 0;
+	unsigned int mergeable_mms = 0;
+	struct list_head *pos;
+
+	list_for_each(pos, &ksm_mm_head.mm_list)
+		mergeable_mms++;
+
+	if (ksm_run & KSM_RUN_AUTO) {
+		len += sysfs_emit_at(buf, len, "mode: auto\n");
+		len += sysfs_emit_at(buf, len, "auto_triggered: %d\n",
+						      auto_triggered);
+		len += sysfs_emit_at(buf, len, "mergeable_mms: %u\n",
+						       mergeable_mms);
+		len += sysfs_emit_at(buf, len, "scanning_factor: %u\n",
+						       scanning_factor);
+		len += sysfs_emit_at(buf, len, "triggered_times: %lu\n",
+						       triggered_times);
+	} else if (ksm_run & KSM_RUN_MERGE) {
+		len += sysfs_emit_at(buf, len, "mode: on\n");
+		len += sysfs_emit_at(buf, len, "mergeable_mms: %u\n",
+							mergeable_mms);
+	} else if (ksm_run & KSM_RUN_UNMERGE)
+		len += sysfs_emit_at(buf, len, "mode: unmerge\n");
+	else
+		len += sysfs_emit_at(buf, len, "mode: off\n");
+
+
+	return len;
+}
+KSM_ATTR_RO(ksmd_status);
+
 static ssize_t auto_threshold_show(struct kobject *kobj,
 						struct kobj_attribute *attr, char *buf)
 {
@@ -3355,6 +3394,7 @@ static struct attribute *ksm_attrs[] = {
 	&pages_to_scan_attr.attr,
 	&max_scanning_factor_attr.attr,
 	&run_attr.attr,
+	&ksmd_status_attr.attr,
 	&auto_threshold_attr.attr,
 	&pages_shared_attr.attr,
 	&pages_sharing_attr.attr,
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 6+ messages in thread

* [PATCH v2 5/5] ksm: add tests of ksm auto mode
  2022-08-12 10:11 [PATCH v2 0/5] propose auto-run mode of ksm and its tests cgel.zte
                   ` (3 preceding siblings ...)
  2022-08-12 10:12 ` [PATCH v2 4/5] ksm: show ksmd status for auto mode cgel.zte
@ 2022-08-12 10:12 ` cgel.zte
  4 siblings, 0 replies; 6+ messages in thread
From: cgel.zte @ 2022-08-12 10:12 UTC (permalink / raw)
  To: akpm, willy; +Cc: hughd, izik.eidus, linux-kernel, linux-mm, xu.xin16, CGEL

From: xu xin <xu.xin16@zte.com.cn>

Add tests to verify the effectiveness and cpu consumption of auto
mode of KSM. The tests requires no other MERGEABLE-madvised vm areas
in system than the areas our testing process allocs.

Signed-off-by: xu xin <xu.xin16@zte.com.cn>
Signed-off-by: CGEL <cgel.zte@gmail.com>
---
 tools/testing/selftests/vm/.gitignore      |   1 +
 tools/testing/selftests/vm/Makefile        |   1 +
 tools/testing/selftests/vm/test-ksm-auto.c | 273 +++++++++++++++++++++
 3 files changed, 275 insertions(+)
 create mode 100644 tools/testing/selftests/vm/test-ksm-auto.c

diff --git a/tools/testing/selftests/vm/.gitignore b/tools/testing/selftests/vm/.gitignore
index 31e5eea2a9b9..1cd8816c055d 100644
--- a/tools/testing/selftests/vm/.gitignore
+++ b/tools/testing/selftests/vm/.gitignore
@@ -34,3 +34,4 @@ local_config.*
 soft-dirty
 split_huge_page_test
 ksm_tests
+test-ksm-auto
diff --git a/tools/testing/selftests/vm/Makefile b/tools/testing/selftests/vm/Makefile
index d9fa6a9ea584..002d9482c37f 100644
--- a/tools/testing/selftests/vm/Makefile
+++ b/tools/testing/selftests/vm/Makefile
@@ -54,6 +54,7 @@ TEST_GEN_FILES += userfaultfd
 TEST_GEN_PROGS += soft-dirty
 TEST_GEN_PROGS += split_huge_page_test
 TEST_GEN_FILES += ksm_tests
+TEST_GEN_FILES += test-ksm-auto
 
 ifeq ($(MACHINE),x86_64)
 CAN_BUILD_I386 := $(shell ./../x86/check_cc.sh "$(CC)" ../x86/trivial_32bit_program.c -m32)
diff --git a/tools/testing/selftests/vm/test-ksm-auto.c b/tools/testing/selftests/vm/test-ksm-auto.c
new file mode 100644
index 000000000000..0d71593e008e
--- /dev/null
+++ b/tools/testing/selftests/vm/test-ksm-auto.c
@@ -0,0 +1,273 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <errno.h>
+#include <fcntl.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <ucontext.h>
+#include <sys/time.h>
+#include <sys/mman.h>
+
+#define KSM_CLEAR_MODE  "2\n"
+#define KSM_NORMAL_MODE "1\n"
+#define KSM_AUTO_MODE   "8\n"
+
+#define PAGESIZE	(4*1024)
+/* Don't change the value, it will afffect the result */
+#define TOTAL_MADVISE_SIZE	(300*1024*1024)
+
+char *ksm_run_file = "/sys/kernel/mm/ksm/run";
+char *ksm_auto_threshold_file = "/sys/kernel/mm/ksm/auto_threshold";
+char *ksm_pages_volatile_file = "/sys/kernel/mm/ksm/pages_volatile";
+char *ksm_pages_sharing_file = "/sys/kernel/mm/ksm/pages_sharing";
+
+#define SHAPE_FULL          1
+#define SHAPE_SPARSE        2
+/* They are related to the shape of memory */
+int final_pages[3] = {0, 76500, 42};
+
+static char *mmap_and_madvise(long long size, int advise)
+{
+	char *ptr;
+	int err;
+
+	err = 0;
+
+	ptr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
+	if (!ptr)
+		return NULL;
+
+	err = madvise(ptr, size, advise);
+	if (err) {
+		perror("Madvise failed\n");
+		free(ptr);
+		return NULL;
+	}
+
+	return ptr;
+}
+
+void make_samepage_ares(char *ptr, int size, int shape_type)
+{
+	int i, j;
+	char rnd_num;
+
+	switch (shape_type) {
+	case SHAPE_FULL:
+		for (i = 0; i < (size / PAGESIZE); i++)
+			memset(ptr + (i * PAGESIZE), 0x1, PAGESIZE);
+		break;
+	case SHAPE_SPARSE:
+		/* Make pages different */
+		j = 0;
+		for (i = 1; i < (size / PAGESIZE); i++) {
+			ptr[i * PAGESIZE + (j%PAGESIZE)] = j%127 + 1;
+			j++;
+		}
+		for (i = 0; i < (size / PAGESIZE); i += 1800)
+			memset(ptr + (i * PAGESIZE), -1, PAGESIZE);
+	}
+
+	return;
+}
+
+int read_file(char *file, char *buffer, int buf_len)
+{
+	FILE *fp;
+	size_t result;
+	long lSize;
+
+	fp = fopen(file, "r");
+	if (!fp)
+		return -1;
+
+	fseek(fp, 0, SEEK_END);
+	lSize = ftell(fp);
+	rewind(fp);
+
+	memset(buffer, 0, buf_len);
+	result = fread(buffer, 1, buf_len, fp);
+	if (result == 0)
+		return -1;
+
+	fclose(fp);
+
+	return 0;
+}
+
+int write_file(char *file, const char *buffer, int len)
+{
+	FILE *fp;
+	size_t result;
+
+	fp = fopen(file, "w+");
+	if (!fp)
+		return -1;
+
+	result = fwrite(buffer, len, 1, fp);
+	if (result == 0)
+		return -1;
+
+	fclose(fp);
+
+	return 0;
+}
+
+static inline void get_orig_info(int *run, int *auto_threshold)
+{
+	char buffer[50];
+
+	/* Read the original state of ksm/run */
+	if (read_file(ksm_run_file, buffer, sizeof(buffer))) {
+		printf("read file %s failed\n", ksm_run_file);
+		exit(1);
+	}
+	*run = atoi(buffer);
+
+	if (read_file(ksm_auto_threshold_file, buffer, sizeof(buffer))) {
+		printf("read file: %s failed\n", ksm_auto_threshold_file);
+		exit(1);
+	}
+	*auto_threshold = atoi(buffer);
+}
+
+static inline void restore_orig_state(int run, int auto_threshold)
+{
+	char buffer[50];
+
+	/* restore the original state */
+	memset(buffer, 0, sizeof(buffer));
+	snprintf(buffer, sizeof(buffer) - 1, "%d\n", run);
+	if (write_file(ksm_run_file, buffer, sizeof(buffer))) {
+		printf("write file %s failed\n", ksm_run_file);
+		exit(1);
+	}
+
+	memset(buffer, 0, sizeof(buffer));
+	snprintf(buffer, sizeof(buffer) - 1, "%d\n", auto_threshold);
+	if (write_file(ksm_auto_threshold_file, buffer, sizeof(buffer))) {
+		printf("write file %s failed\n", ksm_run_file);
+		exit(1);
+	}
+}
+
+void set_ksmd_run_mode(char *mode)
+{
+	if (write_file(ksm_run_file, mode, 2)) {
+		printf("Failed: write 1 to %s\n", ksm_auto_threshold_file);
+		exit(1);
+	}
+}
+
+static inline void wait_ksmpages_converged(int final_pages)
+{
+	int pages_sharing;
+	char buffer[50];
+
+	for (;;) {
+		if (read_file(ksm_pages_sharing_file, buffer, sizeof(buffer))) {
+			printf("read file %s failed\n", ksm_pages_sharing_file);
+			exit(1);
+		}
+
+		pages_sharing = atoi(buffer);
+		if (pages_sharing >= final_pages)
+			break;
+	}
+}
+
+void print_shape(int shape_type)
+{
+	switch (shape_type) {
+	case SHAPE_FULL:
+		printf("Now the shape of memory area is full-samepages:\n");
+		printf("[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]\n\n");
+		break;
+	case SHAPE_SPARSE:
+		printf("Now the shape of memory area is sparse-samepages:\n");
+		printf("[xx]          [xx]          [xx]   \n\n");
+		break;
+	}
+}
+
+void print_ksmd_cpu_comsuption(void)
+{
+	system("(ps x| grep \"ksmd\" | grep -v grep | awk \'{print $1}\' |"
+	       " xargs -i cat /proc/{}/stat) | awk \'{print \"ksm current "
+		   "cpu total slice: \"  $14+$15+$16+$17}\'");
+}
+
+void test_ksmd_performance(char *madvise_area, int shape_type)
+{
+	struct timeval tv_start, tv_end;
+
+	make_samepage_ares(madvise_area, TOTAL_MADVISE_SIZE, shape_type);
+	print_shape(shape_type);
+
+	/********* Start to time ksmd's normal-run mode **********/
+	printf("Start to test normal-run ksmd...\n");
+
+	print_ksmd_cpu_comsuption();
+
+	set_ksmd_run_mode(KSM_CLEAR_MODE);
+	set_ksmd_run_mode(KSM_NORMAL_MODE);
+
+	gettimeofday(&tv_start, NULL);
+
+	wait_ksmpages_converged(final_pages[shape_type]);
+
+	gettimeofday(&tv_end, NULL);
+	printf("ksm normal-run's merging time: %lf seconds\n",
+	       ((tv_end.tv_sec * 1000000 + tv_end.tv_usec) -
+	       (tv_start.tv_sec * 1000000 + tv_start.tv_usec))/1000000.0);
+
+	/******* Start to time ksmd's auto-run mode **********/
+	print_ksmd_cpu_comsuption();
+
+	printf("Start to test auto-run ksmd...\n");
+	set_ksmd_run_mode(KSM_CLEAR_MODE);
+	set_ksmd_run_mode(KSM_AUTO_MODE);
+	if (write_file(ksm_auto_threshold_file, "99\n", 2))
+		printf("Failed: write 1 to %s\n", ksm_auto_threshold_file);
+
+	gettimeofday(&tv_start, NULL);
+
+	wait_ksmpages_converged(shape_type);
+
+	gettimeofday(&tv_end, NULL);
+	printf("ksm auto-run's merging time: %lf seconds\n",
+		((tv_end.tv_sec * 1000000 + tv_end.tv_usec) -
+		(tv_start.tv_sec * 1000000 + tv_start.tv_usec))/1000000.0);
+
+	print_ksmd_cpu_comsuption();
+}
+
+int main(int argc, char **argv)
+{
+	char *madvise_area;
+	int orig_run, orig_auto_threshold;
+
+	/* Get the original state of ksm */
+	get_orig_info(&orig_run, &orig_auto_threshold);
+	printf("Now we mmap 300MB anouymous memory for testing.\n"
+		"There are two type of TEST which have different shape of\n"
+		"samepage areas.\n"
+		"Note: the test requires no other MERGEABLE-madvised vm areas\n"
+		"in system than the areas our testing process allocs.\n");
+	madvise_area = mmap_and_madvise(TOTAL_MADVISE_SIZE, MADV_MERGEABLE);
+	if (!madvise_area) {
+		printf("madvise failed\n");
+		exit(1);
+	}
+
+	printf("\n****************** TEST 1 ******************\n");
+	test_ksmd_performance(madvise_area, SHAPE_FULL);
+	printf("\n****************** TEST 2 ******************\n");
+	test_ksmd_performance(madvise_area, SHAPE_SPARSE);
+
+	/* Restore the original state */
+	restore_orig_state(orig_run, orig_auto_threshold);
+
+	return 0;
+}
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2022-08-12 10:13 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-08-12 10:11 [PATCH v2 0/5] propose auto-run mode of ksm and its tests cgel.zte
2022-08-12 10:11 ` [PATCH v2 1/5] ksm: add a auto-run mode of ksm cgel.zte
2022-08-12 10:12 ` [PATCH v2 2/5] ksm: implement scan-enhanced algorithm cgel.zte
2022-08-12 10:12 ` [PATCH v2 3/5] ksm: let ksmd auto-work with memory threshold cgel.zte
2022-08-12 10:12 ` [PATCH v2 4/5] ksm: show ksmd status for auto mode cgel.zte
2022-08-12 10:12 ` [PATCH v2 5/5] ksm: add tests of ksm " cgel.zte

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).