[RFCv3][PATCH 1/4] replace string_get_size() arrays

[RFCv3][PATCH 1/4] replace string_get_size() arrays

[Dave Hansen]

Instead of explicitly storing the entire string for each
possible units, just store the thing that varies: the
first character.

We have to special-case the 'B' unit (index==0).

This shaves about 100 bytes off of my .o file.

Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>
---

 linux-2.6.git-dave/lib/string_helpers.c |   30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)

diff -puN lib/string_helpers.c~string_get_size-pow2 lib/string_helpers.c
--- linux-2.6.git/lib/string_helpers.c~string_get_size-pow2	2011-09-30 16:50:31.628981352 -0700
+++ linux-2.6.git-dave/lib/string_helpers.c	2011-09-30 17:04:02.211607364 -0700
@@ -8,6 +8,23 @@
 #include <linux/module.h>
 #include <linux/string_helpers.h>
 
+static const char byte_units[] = "_KMGTPEZY";
+
+static char *__units_str(enum string_size_units unit, char *buf, int index)
+{
+	int place = 0;
+
+	/* index=0 is plain 'B' with no other unit */
+	if (index) {
+		buf[place++] = byte_units[index];
+		if (unit == STRING_UNITS_2)
+			buf[place++] = 'i';
+	}
+	buf[place++] = 'B';
+	buf[place++] = '\0';
+	return buf;
+}
+
 /**
  * string_get_size - get the size in the specified units
  * @size:	The size to be converted
@@ -23,26 +40,19 @@
 int string_get_size(u64 size, const enum string_size_units units,
 		    char *buf, int len)
 {
-	const char *units_10[] = { "B", "kB", "MB", "GB", "TB", "PB",
-				   "EB", "ZB", "YB", NULL};
-	const char *units_2[] = {"B", "KiB", "MiB", "GiB", "TiB", "PiB",
-				 "EiB", "ZiB", "YiB", NULL };
-	const char **units_str[] = {
-		[STRING_UNITS_10] =  units_10,
-		[STRING_UNITS_2] = units_2,
-	};
 	const unsigned int divisor[] = {
 		[STRING_UNITS_10] = 1000,
 		[STRING_UNITS_2] = 1024,
 	};
 	int i, j;
 	u64 remainder = 0, sf_cap;
+	char unit_buf[4];
 	char tmp[8];
 
 	tmp[0] = '\0';
 	i = 0;
 	if (size >= divisor[units]) {
-		while (size >= divisor[units] && units_str[units][i]) {
+		while (size >= divisor[units] && (i < strlen(byte_units))) {
 			remainder = do_div(size, divisor[units]);
 			i++;
 		}
@@ -61,7 +71,7 @@ int string_get_size(u64 size, const enum
 	}
 
 	snprintf(buf, len, "%lld%s %s", (unsigned long long)size,
-		 tmp, units_str[units][i]);
+		 tmp, __units_str(units, unit_buf, i));
 
 	return 0;
 }
diff -puN include/linux/string_helpers.h~string_get_size-pow2 include/linux/string_helpers.h
_

[RFCv3][PATCH 1/4] replace string_get_size() arrays

[Dave Hansen]

Instead of explicitly storing the entire string for each
possible units, just store the thing that varies: the
first character.

We have to special-case the 'B' unit (index==0).

This shaves about 100 bytes off of my .o file.

Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>
---

 linux-2.6.git-dave/lib/string_helpers.c |   30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)

diff -puN lib/string_helpers.c~string_get_size-pow2 lib/string_helpers.c
--- linux-2.6.git/lib/string_helpers.c~string_get_size-pow2	2011-09-30 16:50:31.628981352 -0700
+++ linux-2.6.git-dave/lib/string_helpers.c	2011-09-30 17:04:02.211607364 -0700
@@ -8,6 +8,23 @@
 #include <linux/module.h>
 #include <linux/string_helpers.h>
 
+static const char byte_units[] = "_KMGTPEZY";
+
+static char *__units_str(enum string_size_units unit, char *buf, int index)
+{
+	int place = 0;
+
+	/* index=0 is plain 'B' with no other unit */
+	if (index) {
+		buf[place++] = byte_units[index];
+		if (unit == STRING_UNITS_2)
+			buf[place++] = 'i';
+	}
+	buf[place++] = 'B';
+	buf[place++] = '\0';
+	return buf;
+}
+
 /**
  * string_get_size - get the size in the specified units
  * @size:	The size to be converted
@@ -23,26 +40,19 @@
 int string_get_size(u64 size, const enum string_size_units units,
 		    char *buf, int len)
 {
-	const char *units_10[] = { "B", "kB", "MB", "GB", "TB", "PB",
-				   "EB", "ZB", "YB", NULL};
-	const char *units_2[] = {"B", "KiB", "MiB", "GiB", "TiB", "PiB",
-				 "EiB", "ZiB", "YiB", NULL };
-	const char **units_str[] = {
-		[STRING_UNITS_10] =  units_10,
-		[STRING_UNITS_2] = units_2,
-	};
 	const unsigned int divisor[] = {
 		[STRING_UNITS_10] = 1000,
 		[STRING_UNITS_2] = 1024,
 	};
 	int i, j;
 	u64 remainder = 0, sf_cap;
+	char unit_buf[4];
 	char tmp[8];
 
 	tmp[0] = '\0';
 	i = 0;
 	if (size >= divisor[units]) {
-		while (size >= divisor[units] && units_str[units][i]) {
+		while (size >= divisor[units] && (i < strlen(byte_units))) {
 			remainder = do_div(size, divisor[units]);
 			i++;
 		}
@@ -61,7 +71,7 @@ int string_get_size(u64 size, const enum
 	}
 
 	snprintf(buf, len, "%lld%s %s", (unsigned long long)size,
-		 tmp, units_str[units][i]);
+		 tmp, __units_str(units, unit_buf, i));
 
 	return 0;
 }
diff -puN include/linux/string_helpers.h~string_get_size-pow2 include/linux/string_helpers.h
_

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[RFCv3][PATCH 2/4] add string_get_size_pow2()

[Dave Hansen]

This is a specialized version of string_get_size().

It only works on powers-of-two, and only outputs in
KiB/MiB/etc...  Doing it this way means that we do
not have to do any division like string_get_size()
does.

Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>
---

 linux-2.6.git-dave/include/linux/string_helpers.h |    1 
 linux-2.6.git-dave/lib/string_helpers.c           |   23 ++++++++++++++++++++++
 2 files changed, 24 insertions(+)

diff -puN include/linux/string_helpers.h~string_get_size-pow2-1 include/linux/string_helpers.h
--- linux-2.6.git/include/linux/string_helpers.h~string_get_size-pow2-1	2011-09-30 17:03:00.511708995 -0700
+++ linux-2.6.git-dave/include/linux/string_helpers.h	2011-09-30 17:03:00.535708956 -0700
@@ -10,6 +10,7 @@ enum string_size_units {
 	STRING_UNITS_2,		/* use binary powers of 2^10 */
 };
 
+u64 string_get_size_pow2(u64 size, char *unit_ret);
 int string_get_size(u64 size, enum string_size_units units,
 		    char *buf, int len);
 
diff -puN lib/string_helpers.c~string_get_size-pow2-1 lib/string_helpers.c
--- linux-2.6.git/lib/string_helpers.c~string_get_size-pow2-1	2011-09-30 17:03:00.515708988 -0700
+++ linux-2.6.git-dave/lib/string_helpers.c	2011-09-30 17:03:00.535708956 -0700
@@ -25,6 +25,29 @@ static char *__units_str(enum string_siz
 	return buf;
 }
 
+u64 string_get_size_pow2(u64 size, char *unit_ret)
+{
+	int log2;
+	int unit_index;
+
+	if (!size) {
+		__units_str(STRING_UNITS_2, unit_ret, 0);
+		return 0;
+	} else {
+		log2 = ilog2(size);
+	}
+
+	/* KiB is log2=0->9, MiB is 10->19, etc... */
+	unit_index = log2 / 10;
+	__units_str(STRING_UNITS_2, unit_ret, unit_index);
+
+	/* 512 aka 2^9 is the largest integer without
+	 * overflowing to the next power-of-two, so
+	 * use %10 to make it max out there */
+	return (1 << (log2 % 10));
+}
+EXPORT_SYMBOL(string_get_size_pow2);
+
 /**
  * string_get_size - get the size in the specified units
  * @size:	The size to be converted
_

[RFCv3][PATCH 2/4] add string_get_size_pow2()

[Dave Hansen]

This is a specialized version of string_get_size().

It only works on powers-of-two, and only outputs in
KiB/MiB/etc...  Doing it this way means that we do
not have to do any division like string_get_size()
does.

Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>
---

 linux-2.6.git-dave/include/linux/string_helpers.h |    1 
 linux-2.6.git-dave/lib/string_helpers.c           |   23 ++++++++++++++++++++++
 2 files changed, 24 insertions(+)

diff -puN include/linux/string_helpers.h~string_get_size-pow2-1 include/linux/string_helpers.h
--- linux-2.6.git/include/linux/string_helpers.h~string_get_size-pow2-1	2011-09-30 17:03:00.511708995 -0700
+++ linux-2.6.git-dave/include/linux/string_helpers.h	2011-09-30 17:03:00.535708956 -0700
@@ -10,6 +10,7 @@ enum string_size_units {
 	STRING_UNITS_2,		/* use binary powers of 2^10 */
 };
 
+u64 string_get_size_pow2(u64 size, char *unit_ret);
 int string_get_size(u64 size, enum string_size_units units,
 		    char *buf, int len);
 
diff -puN lib/string_helpers.c~string_get_size-pow2-1 lib/string_helpers.c
--- linux-2.6.git/lib/string_helpers.c~string_get_size-pow2-1	2011-09-30 17:03:00.515708988 -0700
+++ linux-2.6.git-dave/lib/string_helpers.c	2011-09-30 17:03:00.535708956 -0700
@@ -25,6 +25,29 @@ static char *__units_str(enum string_siz
 	return buf;
 }
 
+u64 string_get_size_pow2(u64 size, char *unit_ret)
+{
+	int log2;
+	int unit_index;
+
+	if (!size) {
+		__units_str(STRING_UNITS_2, unit_ret, 0);
+		return 0;
+	} else {
+		log2 = ilog2(size);
+	}
+
+	/* KiB is log2=0->9, MiB is 10->19, etc... */
+	unit_index = log2 / 10;
+	__units_str(STRING_UNITS_2, unit_ret, unit_index);
+
+	/* 512 aka 2^9 is the largest integer without
+	 * overflowing to the next power-of-two, so
+	 * use %10 to make it max out there */
+	return (1 << (log2 % 10));
+}
+EXPORT_SYMBOL(string_get_size_pow2);
+
 /**
  * string_get_size - get the size in the specified units
  * @size:	The size to be converted
_

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[RFCv3][PATCH 3/4] add seq_print_pow2() function

[Dave Hansen]

In order to get nice, human-readable output, we are going to
use MiB/KiB, etc... in numa_maps.  Introduce a helper to do
the conversion from a raw integer over to a string.

I thought about doing this as a new printk() format specifier.
That would be interesting, but it's hard to argue with this
since it's so short and sweet.

Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>
---

 linux-2.6.git-dave/fs/seq_file.c            |   11 +++++++++++
 linux-2.6.git-dave/include/linux/seq_file.h |    2 ++
 2 files changed, 13 insertions(+)

diff -puN fs/seq_file.c~add-seq_print_size fs/seq_file.c
--- linux-2.6.git/fs/seq_file.c~add-seq_print_size	2011-09-30 16:41:04.169957332 -0700
+++ linux-2.6.git-dave/fs/seq_file.c	2011-09-30 16:41:04.181957311 -0700
@@ -386,6 +386,17 @@ int seq_printf(struct seq_file *m, const
 }
 EXPORT_SYMBOL(seq_printf);
 
+/*
+ * Prints output with KiB/MiB/etc... suffixes
+ */
+int seq_print_pow2(struct seq_file *seq, u64 size)
+{
+	u64 shifted_size;
+	char unit_str[4];
+	shifted_size = string_get_size_pow2(size, unit_str);
+	return seq_printf(seq, "%llu%s", shifted_size, unit_str);
+}
+
 /**
  *	mangle_path -	mangle and copy path to buffer beginning
  *	@s: buffer start
diff -puN include/linux/seq_file.h~add-seq_print_size include/linux/seq_file.h
--- linux-2.6.git/include/linux/seq_file.h~add-seq_print_size	2011-09-30 16:41:04.173957325 -0700
+++ linux-2.6.git-dave/include/linux/seq_file.h	2011-09-30 16:41:04.181957311 -0700
@@ -3,6 +3,7 @@
 
 #include <linux/types.h>
 #include <linux/string.h>
+#include <linux/string_helpers.h>
 #include <linux/mutex.h>
 #include <linux/cpumask.h>
 #include <linux/nodemask.h>
@@ -83,6 +84,7 @@ int seq_escape(struct seq_file *, const 
 int seq_putc(struct seq_file *m, char c);
 int seq_puts(struct seq_file *m, const char *s);
 int seq_write(struct seq_file *seq, const void *data, size_t len);
+int seq_print_pow2(struct seq_file *seq, u64 size);
 
 int seq_printf(struct seq_file *, const char *, ...)
 	__attribute__ ((format (printf,2,3)));
_

[RFCv3][PATCH 3/4] add seq_print_pow2() function

[Dave Hansen]

In order to get nice, human-readable output, we are going to
use MiB/KiB, etc... in numa_maps.  Introduce a helper to do
the conversion from a raw integer over to a string.

I thought about doing this as a new printk() format specifier.
That would be interesting, but it's hard to argue with this
since it's so short and sweet.

Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>
---

 linux-2.6.git-dave/fs/seq_file.c            |   11 +++++++++++
 linux-2.6.git-dave/include/linux/seq_file.h |    2 ++
 2 files changed, 13 insertions(+)

diff -puN fs/seq_file.c~add-seq_print_size fs/seq_file.c
--- linux-2.6.git/fs/seq_file.c~add-seq_print_size	2011-09-30 16:41:04.169957332 -0700
+++ linux-2.6.git-dave/fs/seq_file.c	2011-09-30 16:41:04.181957311 -0700
@@ -386,6 +386,17 @@ int seq_printf(struct seq_file *m, const
 }
 EXPORT_SYMBOL(seq_printf);
 
+/*
+ * Prints output with KiB/MiB/etc... suffixes
+ */
+int seq_print_pow2(struct seq_file *seq, u64 size)
+{
+	u64 shifted_size;
+	char unit_str[4];
+	shifted_size = string_get_size_pow2(size, unit_str);
+	return seq_printf(seq, "%llu%s", shifted_size, unit_str);
+}
+
 /**
  *	mangle_path -	mangle and copy path to buffer beginning
  *	@s: buffer start
diff -puN include/linux/seq_file.h~add-seq_print_size include/linux/seq_file.h
--- linux-2.6.git/include/linux/seq_file.h~add-seq_print_size	2011-09-30 16:41:04.173957325 -0700
+++ linux-2.6.git-dave/include/linux/seq_file.h	2011-09-30 16:41:04.181957311 -0700
@@ -3,6 +3,7 @@
 
 #include <linux/types.h>
 #include <linux/string.h>
+#include <linux/string_helpers.h>
 #include <linux/mutex.h>
 #include <linux/cpumask.h>
 #include <linux/nodemask.h>
@@ -83,6 +84,7 @@ int seq_escape(struct seq_file *, const 
 int seq_putc(struct seq_file *m, char c);
 int seq_puts(struct seq_file *m, const char *s);
 int seq_write(struct seq_file *seq, const void *data, size_t len);
+int seq_print_pow2(struct seq_file *seq, u64 size);
 
 int seq_printf(struct seq_file *, const char *, ...)
 	__attribute__ ((format (printf,2,3)));
_

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[Dave Hansen]

The output of /proc/$pid/numa_maps is in terms of number of pages
like anon=22 or dirty=54.  Here's some output:

7f4680000000 default file=/hugetlb/bigfile anon=50 dirty=50 N0=50
7f7659600000 default file=/anon_hugepage\040(deleted) anon=50 dirty=50 N0=50
7fff8d425000 default stack anon=50 dirty=50 N0=50

Looks like we have a stack and a couple of anonymous hugetlbfs
areas page which both use the same amount of memory.  They don't.

The 'bigfile' uses 1GB pages and takes up ~50GB of space.  The
anon_hugepage uses 2MB pages and takes up ~100MB of space while
the stack uses normal 4k pages.  You can go over to smaps to
figure out what the page size _really_ is with KernelPageSize
or MMUPageSize.  But, I think this is a pretty nasty and
counterintuitive interface as it stands.

The following patch adds a pagesize= field.  Note that this only
shows the kernel's notion of page size.  For transparent
hugepages, it still shows the base page size.  Here's some real
output.  Note the anon_hugepage in there.

# cat /proc/`pidof memknobs`/numa_maps
00400000 default file=/root/memknobs pagesize=4KiB dirty=3 active=2 N0=3
00602000 default file=/root/memknobs pagesize=4KiB anon=1 dirty=1 N0=1
00603000 default file=/root/memknobs pagesize=4KiB anon=1 dirty=1 N0=1
00604000 default heap pagesize=4KiB anon=6 dirty=6 N0=6
7f6766216000 default file=/lib/libc-2.9.so pagesize=4KiB mapped=98 mapmax=25 active=97 N0=98
7f676637e000 default file=/lib/libc-2.9.so
7f676657e000 default file=/lib/libc-2.9.so pagesize=4KiB anon=4 dirty=4 N0=4
7f6766582000 default file=/lib/libc-2.9.so pagesize=4KiB anon=1 dirty=1 N0=1
7f6766583000 default pagesize=4KiB anon=3 dirty=3 N0=3
7f6766588000 default file=/lib/ld-2.9.so pagesize=4KiB mapped=25 mapmax=24 N0=25
7f676679d000 default pagesize=4KiB anon=2 dirty=2 N0=2
7f67667a3000 default pagesize=4KiB anon=4 dirty=4 N0=4
7f67667a7000 default file=/lib/ld-2.9.so pagesize=4KiB anon=1 dirty=1 N0=1
7f67667a8000 default file=/lib/ld-2.9.so pagesize=4KiB anon=1 dirty=1 N0=1
7f6766800000 default file=/anon_hugepage\040(deleted) pagesize=2MiB anon=10 dirty=10 N0=10
7fff5b948000 default stack pagesize=4KiB anon=2 dirty=2 N0=2
7fff5b96d000 default

Signed-off-by: Dave Haneen <dave@linux.vnet.ibm.com>
---

 linux-2.6.git-dave/fs/proc/task_mmu.c |    5 +++++
 1 file changed, 5 insertions(+)

diff -puN fs/proc/task_mmu.c~show-page-size fs/proc/task_mmu.c
--- linux-2.6.git/fs/proc/task_mmu.c~show-page-size	2011-09-30 16:41:06.125953955 -0700
+++ linux-2.6.git-dave/fs/proc/task_mmu.c	2011-09-30 16:41:06.133953941 -0700
@@ -1044,6 +1044,11 @@ static int show_numa_map(struct seq_file
 	if (!md->pages)
 		goto out;
 
+	/* Only interesting for hugetlbfs pages.
+	 * Transparent hugepages are still pagesize=4k */
+	seq_puts(m, " pagesize=");
+	seq_print_pow2(m, vma_kernel_pagesize(vma));
+
 	if (md->anon)
 		seq_printf(m, " anon=%lu", md->anon);
 
_

[RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[Dave Hansen]

The output of /proc/$pid/numa_maps is in terms of number of pages
like anon=22 or dirty=54.  Here's some output:

7f4680000000 default file=/hugetlb/bigfile anon=50 dirty=50 N0=50
7f7659600000 default file=/anon_hugepage\040(deleted) anon=50 dirty=50 N0=50
7fff8d425000 default stack anon=50 dirty=50 N0=50

Looks like we have a stack and a couple of anonymous hugetlbfs
areas page which both use the same amount of memory.  They don't.

The 'bigfile' uses 1GB pages and takes up ~50GB of space.  The
anon_hugepage uses 2MB pages and takes up ~100MB of space while
the stack uses normal 4k pages.  You can go over to smaps to
figure out what the page size _really_ is with KernelPageSize
or MMUPageSize.  But, I think this is a pretty nasty and
counterintuitive interface as it stands.

The following patch adds a pagesize= field.  Note that this only
shows the kernel's notion of page size.  For transparent
hugepages, it still shows the base page size.  Here's some real
output.  Note the anon_hugepage in there.

# cat /proc/`pidof memknobs`/numa_maps
00400000 default file=/root/memknobs pagesize=4KiB dirty=3 active=2 N0=3
00602000 default file=/root/memknobs pagesize=4KiB anon=1 dirty=1 N0=1
00603000 default file=/root/memknobs pagesize=4KiB anon=1 dirty=1 N0=1
00604000 default heap pagesize=4KiB anon=6 dirty=6 N0=6
7f6766216000 default file=/lib/libc-2.9.so pagesize=4KiB mapped=98 mapmax=25 active=97 N0=98
7f676637e000 default file=/lib/libc-2.9.so
7f676657e000 default file=/lib/libc-2.9.so pagesize=4KiB anon=4 dirty=4 N0=4
7f6766582000 default file=/lib/libc-2.9.so pagesize=4KiB anon=1 dirty=1 N0=1
7f6766583000 default pagesize=4KiB anon=3 dirty=3 N0=3
7f6766588000 default file=/lib/ld-2.9.so pagesize=4KiB mapped=25 mapmax=24 N0=25
7f676679d000 default pagesize=4KiB anon=2 dirty=2 N0=2
7f67667a3000 default pagesize=4KiB anon=4 dirty=4 N0=4
7f67667a7000 default file=/lib/ld-2.9.so pagesize=4KiB anon=1 dirty=1 N0=1
7f67667a8000 default file=/lib/ld-2.9.so pagesize=4KiB anon=1 dirty=1 N0=1
7f6766800000 default file=/anon_hugepage\040(deleted) pagesize=2MiB anon=10 dirty=10 N0=10
7fff5b948000 default stack pagesize=4KiB anon=2 dirty=2 N0=2
7fff5b96d000 default

Signed-off-by: Dave Haneen <dave@linux.vnet.ibm.com>
---

 linux-2.6.git-dave/fs/proc/task_mmu.c |    5 +++++
 1 file changed, 5 insertions(+)

diff -puN fs/proc/task_mmu.c~show-page-size fs/proc/task_mmu.c
--- linux-2.6.git/fs/proc/task_mmu.c~show-page-size	2011-09-30 16:41:06.125953955 -0700
+++ linux-2.6.git-dave/fs/proc/task_mmu.c	2011-09-30 16:41:06.133953941 -0700
@@ -1044,6 +1044,11 @@ static int show_numa_map(struct seq_file
 	if (!md->pages)
 		goto out;
 
+	/* Only interesting for hugetlbfs pages.
+	 * Transparent hugepages are still pagesize=4k */
+	seq_puts(m, " pagesize=");
+	seq_print_pow2(m, vma_kernel_pagesize(vma));
+
 	if (md->anon)
 		seq_printf(m, " anon=%lu", md->anon);
 
_

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 1/4] replace string_get_size() arrays

[Joe Perches]

On Fri, 2011-09-30 at 17:08 -0700, Dave Hansen wrote:
> Instead of explicitly storing the entire string for each
> possible units, just store the thing that varies: the
> first character.

trivia

> diff -puN lib/string_helpers.c~string_get_size-pow2 lib/string_helpers.c
> --- linux-2.6.git/lib/string_helpers.c~string_get_size-pow2	2011-09-30 16:50:31.628981352 -0700
> +++ linux-2.6.git-dave/lib/string_helpers.c	2011-09-30 17:04:02.211607364 -0700
> @@ -8,6 +8,23 @@
>  #include <linux/module.h>
>  #include <linux/string_helpers.h>
>  
> +static const char byte_units[] = "_KMGTPEZY";

u64 could be up to ~1.8**19 decimal
zetta and yotta are not possible or necessary.
u128 maybe someday, but then other changes
would be necessary.

> +static char *__units_str(enum string_size_units unit, char *buf, int index)
> +{
> +	int place = 0;
> +
> +	/* index=0 is plain 'B' with no other unit */
> +	if (index) {
> +		buf[place++] = byte_units[index];

unbounded index (doesn't matter currently, it will for u128)

> @@ -23,26 +40,19 @@
>  int string_get_size(u64 size, const enum string_size_units units,
>  		    char *buf, int len)
[]
>  	const unsigned int divisor[] = {
>  		[STRING_UNITS_10] = 1000,
>  		[STRING_UNITS_2] = 1024,
>  	};

static const or it might be better to use
as that would make the code clearer.

	unsigned int divisor = (string_size_units == STRING_UNITS_2) ? 1024 : 1000;

> @@ -61,7 +71,7 @@ int string_get_size(u64 size, const enum
>  	}
>  
>  	snprintf(buf, len, "%lld%s %s", (unsigned long long)size,

%llu

Re: [RFCv3][PATCH 1/4] replace string_get_size() arrays

[Joe Perches]

On Fri, 2011-09-30 at 17:08 -0700, Dave Hansen wrote:
> Instead of explicitly storing the entire string for each
> possible units, just store the thing that varies: the
> first character.

trivia

> diff -puN lib/string_helpers.c~string_get_size-pow2 lib/string_helpers.c
> --- linux-2.6.git/lib/string_helpers.c~string_get_size-pow2	2011-09-30 16:50:31.628981352 -0700
> +++ linux-2.6.git-dave/lib/string_helpers.c	2011-09-30 17:04:02.211607364 -0700
> @@ -8,6 +8,23 @@
>  #include <linux/module.h>
>  #include <linux/string_helpers.h>
>  
> +static const char byte_units[] = "_KMGTPEZY";

u64 could be up to ~1.8**19 decimal
zetta and yotta are not possible or necessary.
u128 maybe someday, but then other changes
would be necessary.

> +static char *__units_str(enum string_size_units unit, char *buf, int index)
> +{
> +	int place = 0;
> +
> +	/* index=0 is plain 'B' with no other unit */
> +	if (index) {
> +		buf[place++] = byte_units[index];

unbounded index (doesn't matter currently, it will for u128)

> @@ -23,26 +40,19 @@
>  int string_get_size(u64 size, const enum string_size_units units,
>  		    char *buf, int len)
[]
>  	const unsigned int divisor[] = {
>  		[STRING_UNITS_10] = 1000,
>  		[STRING_UNITS_2] = 1024,
>  	};

static const or it might be better to use
as that would make the code clearer.

	unsigned int divisor = (string_size_units == STRING_UNITS_2) ? 1024 : 1000;

> @@ -61,7 +71,7 @@ int string_get_size(u64 size, const enum
>  	}
>  
>  	snprintf(buf, len, "%lld%s %s", (unsigned long long)size,

%llu

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 1/4] replace string_get_size() arrays

[Joe Perches]

On Fri, 2011-09-30 at 17:08 -0700, Dave Hansen wrote:
> Instead of explicitly storing the entire string for each
> possible units, just store the thing that varies: the
> first character.

trivia

> diff -puN lib/string_helpers.c~string_get_size-pow2 lib/string_helpers.c
> --- linux-2.6.git/lib/string_helpers.c~string_get_size-pow2	2011-09-30 16:50:31.628981352 -0700
> +++ linux-2.6.git-dave/lib/string_helpers.c	2011-09-30 17:04:02.211607364 -0700
> @@ -8,6 +8,23 @@
>  #include <linux/module.h>
>  #include <linux/string_helpers.h>
>  
> +static const char byte_units[] = "_KMGTPEZY";

u64 could be up to ~1.8**19 decimal
zetta and yotta are not possible or necessary.
u128 maybe someday, but then other changes
would be necessary too.

> +static char *__units_str(enum string_size_units unit, char *buf, int index)
> +{
> +	int place = 0;
> +
> +	/* index=0 is plain 'B' with no other unit */
> +	if (index) {
> +		buf[place++] = byte_units[index];

index is unbounded (doesn't matter currently, it will for u128)

> @@ -23,26 +40,19 @@
>  int string_get_size(u64 size, const enum string_size_units units,
>  		    char *buf, int len)
[]
>  	const unsigned int divisor[] = {
>  		[STRING_UNITS_10] = 1000,
>  		[STRING_UNITS_2] = 1024,
>  	};

static const or it might be better to use
	unsigned int divisor = (string_size_units == STRING_UNITS_2) ? 1024 : 1000;
as that would make the code clearer in a
couple of uses of divisor[] later.

> @@ -61,7 +71,7 @@ int string_get_size(u64 size, const enum
>  	}
>  
>  	snprintf(buf, len, "%lld%s %s", (unsigned long long)size,

%llu

Re: [RFCv3][PATCH 1/4] replace string_get_size() arrays

[Joe Perches]

On Fri, 2011-09-30 at 17:08 -0700, Dave Hansen wrote:
> Instead of explicitly storing the entire string for each
> possible units, just store the thing that varies: the
> first character.

trivia

> diff -puN lib/string_helpers.c~string_get_size-pow2 lib/string_helpers.c
> --- linux-2.6.git/lib/string_helpers.c~string_get_size-pow2	2011-09-30 16:50:31.628981352 -0700
> +++ linux-2.6.git-dave/lib/string_helpers.c	2011-09-30 17:04:02.211607364 -0700
> @@ -8,6 +8,23 @@
>  #include <linux/module.h>
>  #include <linux/string_helpers.h>
>  
> +static const char byte_units[] = "_KMGTPEZY";

u64 could be up to ~1.8**19 decimal
zetta and yotta are not possible or necessary.
u128 maybe someday, but then other changes
would be necessary too.

> +static char *__units_str(enum string_size_units unit, char *buf, int index)
> +{
> +	int place = 0;
> +
> +	/* index=0 is plain 'B' with no other unit */
> +	if (index) {
> +		buf[place++] = byte_units[index];

index is unbounded (doesn't matter currently, it will for u128)

> @@ -23,26 +40,19 @@
>  int string_get_size(u64 size, const enum string_size_units units,
>  		    char *buf, int len)
[]
>  	const unsigned int divisor[] = {
>  		[STRING_UNITS_10] = 1000,
>  		[STRING_UNITS_2] = 1024,
>  	};

static const or it might be better to use
	unsigned int divisor = (string_size_units == STRING_UNITS_2) ? 1024 : 1000;
as that would make the code clearer in a
couple of uses of divisor[] later.

> @@ -61,7 +71,7 @@ int string_get_size(u64 size, const enum
>  	}
>  
>  	snprintf(buf, len, "%lld%s %s", (unsigned long long)size,

%llu


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 1/4] replace string_get_size() arrays

[Dave Hansen]

On Sat, 2011-10-01 at 12:33 -0700, Joe Perches wrote:
> On Fri, 2011-09-30 at 17:08 -0700, Dave Hansen wrote:
> > Instead of explicitly storing the entire string for each
> > possible units, just store the thing that varies: the
> > first character.
> 
> trivia

I'm not sure what you mean by that.

> > diff -puN lib/string_helpers.c~string_get_size-pow2 lib/string_helpers.c
> > --- linux-2.6.git/lib/string_helpers.c~string_get_size-pow2	2011-09-30 16:50:31.628981352 -0700
> > +++ linux-2.6.git-dave/lib/string_helpers.c	2011-09-30 17:04:02.211607364 -0700
> > @@ -8,6 +8,23 @@
> >  #include <linux/module.h>
> >  #include <linux/string_helpers.h>
> >  
> > +static const char byte_units[] = "_KMGTPEZY";
> 
> u64 could be up to ~1.8**19 decimal
> zetta and yotta are not possible or necessary.
> u128 maybe someday, but then other changes
> would be necessary too.

Right, but we're only handling u64.

> > +static char *__units_str(enum string_size_units unit, char *buf, int index)
> > +{
> > +	int place = 0;
> > +
> > +	/* index=0 is plain 'B' with no other unit */
> > +	if (index) {
> > +		buf[place++] = byte_units[index];
> 
> index is unbounded (doesn't matter currently, it will for u128)

It's bound by the division or the log2 at least.  You do have to know
what you're passing in to __units_str, just like you had to know what
you were indexing with in to units_2[] and units_10[].

Is there something else you'd like to see done here?  We can
bounds-check index, but that seems a bit unnecessary since it's static
and the two callers are visible on the same page of code.

> > @@ -23,26 +40,19 @@
> >  int string_get_size(u64 size, const enum string_size_units units,
> >  		    char *buf, int len)
> []
> >  	const unsigned int divisor[] = {
> >  		[STRING_UNITS_10] = 1000,
> >  		[STRING_UNITS_2] = 1024,
> >  	};
> 
> static const or it might be better to use
> 	unsigned int divisor = (string_size_units == STRING_UNITS_2) ? 1024 : 1000;
> as that would make the code clearer in a
> couple of uses of divisor[] later.
> 
> > @@ -61,7 +71,7 @@ int string_get_size(u64 size, const enum
> >  	}
> >  
> >  	snprintf(buf, len, "%lld%s %s", (unsigned long long)size,
> 
> %llu

These two are about existing code, and not really necessary for this
set.  They'd make good follow-on patches, though.

-- Dave

Re: [RFCv3][PATCH 1/4] replace string_get_size() arrays

[Dave Hansen]

On Sat, 2011-10-01 at 12:33 -0700, Joe Perches wrote:
> On Fri, 2011-09-30 at 17:08 -0700, Dave Hansen wrote:
> > Instead of explicitly storing the entire string for each
> > possible units, just store the thing that varies: the
> > first character.
> 
> trivia

I'm not sure what you mean by that.

> > diff -puN lib/string_helpers.c~string_get_size-pow2 lib/string_helpers.c
> > --- linux-2.6.git/lib/string_helpers.c~string_get_size-pow2	2011-09-30 16:50:31.628981352 -0700
> > +++ linux-2.6.git-dave/lib/string_helpers.c	2011-09-30 17:04:02.211607364 -0700
> > @@ -8,6 +8,23 @@
> >  #include <linux/module.h>
> >  #include <linux/string_helpers.h>
> >  
> > +static const char byte_units[] = "_KMGTPEZY";
> 
> u64 could be up to ~1.8**19 decimal
> zetta and yotta are not possible or necessary.
> u128 maybe someday, but then other changes
> would be necessary too.

Right, but we're only handling u64.

> > +static char *__units_str(enum string_size_units unit, char *buf, int index)
> > +{
> > +	int place = 0;
> > +
> > +	/* index=0 is plain 'B' with no other unit */
> > +	if (index) {
> > +		buf[place++] = byte_units[index];
> 
> index is unbounded (doesn't matter currently, it will for u128)

It's bound by the division or the log2 at least.  You do have to know
what you're passing in to __units_str, just like you had to know what
you were indexing with in to units_2[] and units_10[].

Is there something else you'd like to see done here?  We can
bounds-check index, but that seems a bit unnecessary since it's static
and the two callers are visible on the same page of code.

> > @@ -23,26 +40,19 @@
> >  int string_get_size(u64 size, const enum string_size_units units,
> >  		    char *buf, int len)
> []
> >  	const unsigned int divisor[] = {
> >  		[STRING_UNITS_10] = 1000,
> >  		[STRING_UNITS_2] = 1024,
> >  	};
> 
> static const or it might be better to use
> 	unsigned int divisor = (string_size_units == STRING_UNITS_2) ? 1024 : 1000;
> as that would make the code clearer in a
> couple of uses of divisor[] later.
> 
> > @@ -61,7 +71,7 @@ int string_get_size(u64 size, const enum
> >  	}
> >  
> >  	snprintf(buf, len, "%lld%s %s", (unsigned long long)size,
> 
> %llu

These two are about existing code, and not really necessary for this
set.  They'd make good follow-on patches, though.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 1/4] replace string_get_size() arrays

[Joe Perches]

On Tue, 2011-10-04 at 12:35 -0700, Dave Hansen wrote:
> On Sat, 2011-10-01 at 12:33 -0700, Joe Perches wrote:
> > On Fri, 2011-09-30 at 17:08 -0700, Dave Hansen wrote:
> > > Instead of explicitly storing the entire string for each
> > > possible units, just store the thing that varies: the
> > > first character.
> > trivia
> I'm not sure what you mean by that.

that what followed wasn't a declaration of defect, just trivial.

> > > diff -puN lib/string_helpers.c~string_get_size-pow2 lib/string_helpers.c
> > > --- linux-2.6.git/lib/string_helpers.c~string_get_size-pow2	2011-09-30 16:50:31.628981352 -0700
> > > +++ linux-2.6.git-dave/lib/string_helpers.c	2011-09-30 17:04:02.211607364 -0700
> > > @@ -8,6 +8,23 @@
> > >  #include <linux/module.h>
> > >  #include <linux/string_helpers.h>
> > >  
> > > +static const char byte_units[] = "_KMGTPEZY";
> > 
> > u64 could be up to ~1.8**19 decimal
> > zetta and yotta are not possible or necessary.
> > u128 maybe someday, but then other changes
> > would be necessary too.
> 
> Right, but we're only handling u64.

So the declaration should be:

	static const char byte_units[] = " KMGTPE";

Re: [RFCv3][PATCH 1/4] replace string_get_size() arrays

[Joe Perches]

On Tue, 2011-10-04 at 12:35 -0700, Dave Hansen wrote:
> On Sat, 2011-10-01 at 12:33 -0700, Joe Perches wrote:
> > On Fri, 2011-09-30 at 17:08 -0700, Dave Hansen wrote:
> > > Instead of explicitly storing the entire string for each
> > > possible units, just store the thing that varies: the
> > > first character.
> > trivia
> I'm not sure what you mean by that.

that what followed wasn't a declaration of defect, just trivial.

> > > diff -puN lib/string_helpers.c~string_get_size-pow2 lib/string_helpers.c
> > > --- linux-2.6.git/lib/string_helpers.c~string_get_size-pow2	2011-09-30 16:50:31.628981352 -0700
> > > +++ linux-2.6.git-dave/lib/string_helpers.c	2011-09-30 17:04:02.211607364 -0700
> > > @@ -8,6 +8,23 @@
> > >  #include <linux/module.h>
> > >  #include <linux/string_helpers.h>
> > >  
> > > +static const char byte_units[] = "_KMGTPEZY";
> > 
> > u64 could be up to ~1.8**19 decimal
> > zetta and yotta are not possible or necessary.
> > u128 maybe someday, but then other changes
> > would be necessary too.
> 
> Right, but we're only handling u64.

So the declaration should be:

	static const char byte_units[] = " KMGTPE";


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 1/4] replace string_get_size() arrays

[Dave Hansen]

On Tue, 2011-10-04 at 13:42 -0700, Joe Perches wrote:
> > Right, but we're only handling u64.
> 
> So the declaration should be:
> 
>         static const char byte_units[] = " KMGTPE";

I guess that's worth a comment.  But that first character doesn't get
used.  There were two alternatives:

	static const char byte_units[] = "_KMGTPE";

or something along the lines of:

+	static const char byte_units[] = "KMGTPE";
...
+	index--;
+       /* index=-1 is plain 'B' with no other unit */
+       if (index >= 0) {

We don't ever _actually_ look at the space (or underscore).  I figured
the _ was nicer since it would be _obvious_ if it ever got printed out
somehow.  

-- Dave

Re: [RFCv3][PATCH 1/4] replace string_get_size() arrays

[Dave Hansen]

On Tue, 2011-10-04 at 13:42 -0700, Joe Perches wrote:
> > Right, but we're only handling u64.
> 
> So the declaration should be:
> 
>         static const char byte_units[] = " KMGTPE";

I guess that's worth a comment.  But that first character doesn't get
used.  There were two alternatives:

	static const char byte_units[] = "_KMGTPE";

or something along the lines of:

+	static const char byte_units[] = "KMGTPE";
...
+	index--;
+       /* index=-1 is plain 'B' with no other unit */
+       if (index >= 0) {

We don't ever _actually_ look at the space (or underscore).  I figured
the _ was nicer since it would be _obvious_ if it ever got printed out
somehow.  

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 1/4] replace string_get_size() arrays

[Joe Perches]

On Tue, 2011-10-04 at 13:51 -0700, Dave Hansen wrote:
> On Tue, 2011-10-04 at 13:42 -0700, Joe Perches wrote:
> > > Right, but we're only handling u64.
> > So the declaration should be:
> >         static const char byte_units[] = " KMGTPE";
> I guess that's worth a comment.  But that first character doesn't get
> used.  There were two alternatives:
> 	static const char byte_units[] = "_KMGTPE";

or
	static const char byte_units[] = { 0, 'K', 'M', 'G', 'T', 'P', 'E' };

and use ARRAY_SIZE(byte_units) not strlen(byte_units)
for array size maximum.

> or something along the lines of:
> +	static const char byte_units[] = "KMGTPE";
> ...
> +	index--;
> +       /* index=-1 is plain 'B' with no other unit */
> +       if (index >= 0) {
> 
> We don't ever _actually_ look at the space (or underscore).  I figured
> the _ was nicer since it would be _obvious_ if it ever got printed out
> somehow.  

shrug.  It's all the same stuff.

cheers, Joe

Re: [RFCv3][PATCH 1/4] replace string_get_size() arrays

[Joe Perches]

On Tue, 2011-10-04 at 13:51 -0700, Dave Hansen wrote:
> On Tue, 2011-10-04 at 13:42 -0700, Joe Perches wrote:
> > > Right, but we're only handling u64.
> > So the declaration should be:
> >         static const char byte_units[] = " KMGTPE";
> I guess that's worth a comment.  But that first character doesn't get
> used.  There were two alternatives:
> 	static const char byte_units[] = "_KMGTPE";

or
	static const char byte_units[] = { 0, 'K', 'M', 'G', 'T', 'P', 'E' };

and use ARRAY_SIZE(byte_units) not strlen(byte_units)
for array size maximum.

> or something along the lines of:
> +	static const char byte_units[] = "KMGTPE";
> ...
> +	index--;
> +       /* index=-1 is plain 'B' with no other unit */
> +       if (index >= 0) {
> 
> We don't ever _actually_ look at the space (or underscore).  I figured
> the _ was nicer since it would be _obvious_ if it ever got printed out
> somehow.  

shrug.  It's all the same stuff.

cheers, Joe

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[David Rientjes]

On Fri, 30 Sep 2011, Dave Hansen wrote:

> 
> The output of /proc/$pid/numa_maps is in terms of number of pages
> like anon=22 or dirty=54.  Here's some output:
> 
> 7f4680000000 default file=/hugetlb/bigfile anon=50 dirty=50 N0=50
> 7f7659600000 default file=/anon_hugepage\040(deleted) anon=50 dirty=50 N0=50
> 7fff8d425000 default stack anon=50 dirty=50 N0=50
> 
> Looks like we have a stack and a couple of anonymous hugetlbfs
> areas page which both use the same amount of memory.  They don't.
> 
> The 'bigfile' uses 1GB pages and takes up ~50GB of space.  The
> anon_hugepage uses 2MB pages and takes up ~100MB of space while
> the stack uses normal 4k pages.  You can go over to smaps to
> figure out what the page size _really_ is with KernelPageSize
> or MMUPageSize.  But, I think this is a pretty nasty and
> counterintuitive interface as it stands.
> 
> The following patch adds a pagesize= field.  Note that this only
> shows the kernel's notion of page size.  For transparent
> hugepages, it still shows the base page size.  Here's some real
> output.  Note the anon_hugepage in there.
> 
> # cat /proc/`pidof memknobs`/numa_maps
> 00400000 default file=/root/memknobs pagesize=4KiB dirty=3 active=2 N0=3
> 00602000 default file=/root/memknobs pagesize=4KiB anon=1 dirty=1 N0=1
> 00603000 default file=/root/memknobs pagesize=4KiB anon=1 dirty=1 N0=1
> 00604000 default heap pagesize=4KiB anon=6 dirty=6 N0=6
> 7f6766216000 default file=/lib/libc-2.9.so pagesize=4KiB mapped=98 mapmax=25 active=97 N0=98
> 7f676637e000 default file=/lib/libc-2.9.so
> 7f676657e000 default file=/lib/libc-2.9.so pagesize=4KiB anon=4 dirty=4 N0=4
> 7f6766582000 default file=/lib/libc-2.9.so pagesize=4KiB anon=1 dirty=1 N0=1
> 7f6766583000 default pagesize=4KiB anon=3 dirty=3 N0=3
> 7f6766588000 default file=/lib/ld-2.9.so pagesize=4KiB mapped=25 mapmax=24 N0=25
> 7f676679d000 default pagesize=4KiB anon=2 dirty=2 N0=2
> 7f67667a3000 default pagesize=4KiB anon=4 dirty=4 N0=4
> 7f67667a7000 default file=/lib/ld-2.9.so pagesize=4KiB anon=1 dirty=1 N0=1
> 7f67667a8000 default file=/lib/ld-2.9.so pagesize=4KiB anon=1 dirty=1 N0=1
> 7f6766800000 default file=/anon_hugepage\040(deleted) pagesize=2MiB anon=10 dirty=10 N0=10
> 7fff5b948000 default stack pagesize=4KiB anon=2 dirty=2 N0=2
> 7fff5b96d000 default
> 

This would be great if all the /proc/pid/numa_maps consumers were human, 
but unfortuantely that's not the case.  

I understand that this patchset was probably the result of me asking for 
the pagesize= to be specified in each line and using pagesize=4K and 
pagesize=2M as examples, but that exact usage is probably not what we 
want.

As long as there are scripts that go through and read this information 
(we have some internally), expressing them with differing units just makes 
it more difficult to parse.  I'd rather them just be the byte count.

That way, 1G pages would just show pagesize=1073741824.  I don't think 
that's too long and is much easier to parse systematically.

> Signed-off-by: Dave Haneen <dave@linux.vnet.ibm.com>
> ---
> 
>  linux-2.6.git-dave/fs/proc/task_mmu.c |    5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff -puN fs/proc/task_mmu.c~show-page-size fs/proc/task_mmu.c
> --- linux-2.6.git/fs/proc/task_mmu.c~show-page-size	2011-09-30 16:41:06.125953955 -0700
> +++ linux-2.6.git-dave/fs/proc/task_mmu.c	2011-09-30 16:41:06.133953941 -0700
> @@ -1044,6 +1044,11 @@ static int show_numa_map(struct seq_file
>  	if (!md->pages)
>  		goto out;
>  
> +	/* Only interesting for hugetlbfs pages.
> +	 * Transparent hugepages are still pagesize=4k */

Strange comment style (and may not be 4K if thp is extended beyond x86).

> +	seq_puts(m, " pagesize=");
> +	seq_print_pow2(m, vma_kernel_pagesize(vma));
> +
>  	if (md->anon)
>  		seq_printf(m, " anon=%lu", md->anon);
>  

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[David Rientjes]

On Fri, 30 Sep 2011, Dave Hansen wrote:

> 
> The output of /proc/$pid/numa_maps is in terms of number of pages
> like anon=22 or dirty=54.  Here's some output:
> 
> 7f4680000000 default file=/hugetlb/bigfile anon=50 dirty=50 N0=50
> 7f7659600000 default file=/anon_hugepage\040(deleted) anon=50 dirty=50 N0=50
> 7fff8d425000 default stack anon=50 dirty=50 N0=50
> 
> Looks like we have a stack and a couple of anonymous hugetlbfs
> areas page which both use the same amount of memory.  They don't.
> 
> The 'bigfile' uses 1GB pages and takes up ~50GB of space.  The
> anon_hugepage uses 2MB pages and takes up ~100MB of space while
> the stack uses normal 4k pages.  You can go over to smaps to
> figure out what the page size _really_ is with KernelPageSize
> or MMUPageSize.  But, I think this is a pretty nasty and
> counterintuitive interface as it stands.
> 
> The following patch adds a pagesize= field.  Note that this only
> shows the kernel's notion of page size.  For transparent
> hugepages, it still shows the base page size.  Here's some real
> output.  Note the anon_hugepage in there.
> 
> # cat /proc/`pidof memknobs`/numa_maps
> 00400000 default file=/root/memknobs pagesize=4KiB dirty=3 active=2 N0=3
> 00602000 default file=/root/memknobs pagesize=4KiB anon=1 dirty=1 N0=1
> 00603000 default file=/root/memknobs pagesize=4KiB anon=1 dirty=1 N0=1
> 00604000 default heap pagesize=4KiB anon=6 dirty=6 N0=6
> 7f6766216000 default file=/lib/libc-2.9.so pagesize=4KiB mapped=98 mapmax=25 active=97 N0=98
> 7f676637e000 default file=/lib/libc-2.9.so
> 7f676657e000 default file=/lib/libc-2.9.so pagesize=4KiB anon=4 dirty=4 N0=4
> 7f6766582000 default file=/lib/libc-2.9.so pagesize=4KiB anon=1 dirty=1 N0=1
> 7f6766583000 default pagesize=4KiB anon=3 dirty=3 N0=3
> 7f6766588000 default file=/lib/ld-2.9.so pagesize=4KiB mapped=25 mapmax=24 N0=25
> 7f676679d000 default pagesize=4KiB anon=2 dirty=2 N0=2
> 7f67667a3000 default pagesize=4KiB anon=4 dirty=4 N0=4
> 7f67667a7000 default file=/lib/ld-2.9.so pagesize=4KiB anon=1 dirty=1 N0=1
> 7f67667a8000 default file=/lib/ld-2.9.so pagesize=4KiB anon=1 dirty=1 N0=1
> 7f6766800000 default file=/anon_hugepage\040(deleted) pagesize=2MiB anon=10 dirty=10 N0=10
> 7fff5b948000 default stack pagesize=4KiB anon=2 dirty=2 N0=2
> 7fff5b96d000 default
> 

This would be great if all the /proc/pid/numa_maps consumers were human, 
but unfortuantely that's not the case.  

I understand that this patchset was probably the result of me asking for 
the pagesize= to be specified in each line and using pagesize=4K and 
pagesize=2M as examples, but that exact usage is probably not what we 
want.

As long as there are scripts that go through and read this information 
(we have some internally), expressing them with differing units just makes 
it more difficult to parse.  I'd rather them just be the byte count.

That way, 1G pages would just show pagesize=1073741824.  I don't think 
that's too long and is much easier to parse systematically.

> Signed-off-by: Dave Haneen <dave@linux.vnet.ibm.com>
> ---
> 
>  linux-2.6.git-dave/fs/proc/task_mmu.c |    5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff -puN fs/proc/task_mmu.c~show-page-size fs/proc/task_mmu.c
> --- linux-2.6.git/fs/proc/task_mmu.c~show-page-size	2011-09-30 16:41:06.125953955 -0700
> +++ linux-2.6.git-dave/fs/proc/task_mmu.c	2011-09-30 16:41:06.133953941 -0700
> @@ -1044,6 +1044,11 @@ static int show_numa_map(struct seq_file
>  	if (!md->pages)
>  		goto out;
>  
> +	/* Only interesting for hugetlbfs pages.
> +	 * Transparent hugepages are still pagesize=4k */

Strange comment style (and may not be 4K if thp is extended beyond x86).

> +	seq_puts(m, " pagesize=");
> +	seq_print_pow2(m, vma_kernel_pagesize(vma));
> +
>  	if (md->anon)
>  		seq_printf(m, " anon=%lu", md->anon);
>  

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 1/4] replace string_get_size() arrays

[David Rientjes]

On Fri, 30 Sep 2011, Dave Hansen wrote:

> 
> Instead of explicitly storing the entire string for each
> possible units, just store the thing that varies: the
> first character.
> 
> We have to special-case the 'B' unit (index==0).
> 
> This shaves about 100 bytes off of my .o file.
> 

It shaved more than that from my .o file, but should we really be 
optimizing this for text size?  __unit_str() would be replacing what used 
to be a read of stack-allocated memory and make string_get_size() more 
expensive of a function and more complex code.

> Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>
> ---
> 
>  linux-2.6.git-dave/lib/string_helpers.c |   30 ++++++++++++++++++++----------
>  1 file changed, 20 insertions(+), 10 deletions(-)
> 
> diff -puN lib/string_helpers.c~string_get_size-pow2 lib/string_helpers.c
> --- linux-2.6.git/lib/string_helpers.c~string_get_size-pow2	2011-09-30 16:50:31.628981352 -0700
> +++ linux-2.6.git-dave/lib/string_helpers.c	2011-09-30 17:04:02.211607364 -0700
> @@ -8,6 +8,23 @@
>  #include <linux/module.h>
>  #include <linux/string_helpers.h>
>  
> +static const char byte_units[] = "_KMGTPEZY";
> +
> +static char *__units_str(enum string_size_units unit, char *buf, int index)
> +{
> +	int place = 0;
> +
> +	/* index=0 is plain 'B' with no other unit */
> +	if (index) {
> +		buf[place++] = byte_units[index];
> +		if (unit == STRING_UNITS_2)
> +			buf[place++] = 'i';
> +	}
> +	buf[place++] = 'B';
> +	buf[place++] = '\0';
> +	return buf;
> +}
> +
>  /**
>   * string_get_size - get the size in the specified units
>   * @size:	The size to be converted
> @@ -23,26 +40,19 @@
>  int string_get_size(u64 size, const enum string_size_units units,
>  		    char *buf, int len)
>  {
> -	const char *units_10[] = { "B", "kB", "MB", "GB", "TB", "PB",
> -				   "EB", "ZB", "YB", NULL};
> -	const char *units_2[] = {"B", "KiB", "MiB", "GiB", "TiB", "PiB",
> -				 "EiB", "ZiB", "YiB", NULL };
> -	const char **units_str[] = {
> -		[STRING_UNITS_10] =  units_10,
> -		[STRING_UNITS_2] = units_2,
> -	};
>  	const unsigned int divisor[] = {
>  		[STRING_UNITS_10] = 1000,
>  		[STRING_UNITS_2] = 1024,
>  	};
>  	int i, j;
>  	u64 remainder = 0, sf_cap;
> +	char unit_buf[4];
>  	char tmp[8];
>  
>  	tmp[0] = '\0';
>  	i = 0;
>  	if (size >= divisor[units]) {
> -		while (size >= divisor[units] && units_str[units][i]) {
> +		while (size >= divisor[units] && (i < strlen(byte_units))) {
>  			remainder = do_div(size, divisor[units]);
>  			i++;
>  		}
> @@ -61,7 +71,7 @@ int string_get_size(u64 size, const enum
>  	}
>  
>  	snprintf(buf, len, "%lld%s %s", (unsigned long long)size,
> -		 tmp, units_str[units][i]);
> +		 tmp, __units_str(units, unit_buf, i));
>  
>  	return 0;
>  }

Re: [RFCv3][PATCH 1/4] replace string_get_size() arrays

[David Rientjes]

On Fri, 30 Sep 2011, Dave Hansen wrote:

> 
> Instead of explicitly storing the entire string for each
> possible units, just store the thing that varies: the
> first character.
> 
> We have to special-case the 'B' unit (index==0).
> 
> This shaves about 100 bytes off of my .o file.
> 

It shaved more than that from my .o file, but should we really be 
optimizing this for text size?  __unit_str() would be replacing what used 
to be a read of stack-allocated memory and make string_get_size() more 
expensive of a function and more complex code.

> Signed-off-by: Dave Hansen <dave@linux.vnet.ibm.com>
> ---
> 
>  linux-2.6.git-dave/lib/string_helpers.c |   30 ++++++++++++++++++++----------
>  1 file changed, 20 insertions(+), 10 deletions(-)
> 
> diff -puN lib/string_helpers.c~string_get_size-pow2 lib/string_helpers.c
> --- linux-2.6.git/lib/string_helpers.c~string_get_size-pow2	2011-09-30 16:50:31.628981352 -0700
> +++ linux-2.6.git-dave/lib/string_helpers.c	2011-09-30 17:04:02.211607364 -0700
> @@ -8,6 +8,23 @@
>  #include <linux/module.h>
>  #include <linux/string_helpers.h>
>  
> +static const char byte_units[] = "_KMGTPEZY";
> +
> +static char *__units_str(enum string_size_units unit, char *buf, int index)
> +{
> +	int place = 0;
> +
> +	/* index=0 is plain 'B' with no other unit */
> +	if (index) {
> +		buf[place++] = byte_units[index];
> +		if (unit == STRING_UNITS_2)
> +			buf[place++] = 'i';
> +	}
> +	buf[place++] = 'B';
> +	buf[place++] = '\0';
> +	return buf;
> +}
> +
>  /**
>   * string_get_size - get the size in the specified units
>   * @size:	The size to be converted
> @@ -23,26 +40,19 @@
>  int string_get_size(u64 size, const enum string_size_units units,
>  		    char *buf, int len)
>  {
> -	const char *units_10[] = { "B", "kB", "MB", "GB", "TB", "PB",
> -				   "EB", "ZB", "YB", NULL};
> -	const char *units_2[] = {"B", "KiB", "MiB", "GiB", "TiB", "PiB",
> -				 "EiB", "ZiB", "YiB", NULL };
> -	const char **units_str[] = {
> -		[STRING_UNITS_10] =  units_10,
> -		[STRING_UNITS_2] = units_2,
> -	};
>  	const unsigned int divisor[] = {
>  		[STRING_UNITS_10] = 1000,
>  		[STRING_UNITS_2] = 1024,
>  	};
>  	int i, j;
>  	u64 remainder = 0, sf_cap;
> +	char unit_buf[4];
>  	char tmp[8];
>  
>  	tmp[0] = '\0';
>  	i = 0;
>  	if (size >= divisor[units]) {
> -		while (size >= divisor[units] && units_str[units][i]) {
> +		while (size >= divisor[units] && (i < strlen(byte_units))) {
>  			remainder = do_div(size, divisor[units]);
>  			i++;
>  		}
> @@ -61,7 +71,7 @@ int string_get_size(u64 size, const enum
>  	}
>  
>  	snprintf(buf, len, "%lld%s %s", (unsigned long long)size,
> -		 tmp, units_str[units][i]);
> +		 tmp, __units_str(units, unit_buf, i));
>  
>  	return 0;
>  }

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[Eric Dumazet]

Le mardi 04 octobre 2011 à 23:50 -0700, David Rientjes a écrit :

> This would be great if all the /proc/pid/numa_maps consumers were human, 
> but unfortuantely that's not the case.  
> 
> I understand that this patchset was probably the result of me asking for 
> the pagesize= to be specified in each line and using pagesize=4K and 
> pagesize=2M as examples, but that exact usage is probably not what we 
> want.
> 
> As long as there are scripts that go through and read this information 
> (we have some internally), expressing them with differing units just makes 
> it more difficult to parse.  I'd rather them just be the byte count.
> 
> That way, 1G pages would just show pagesize=1073741824.  I don't think 
> that's too long and is much easier to parse systematically.
> 

Hmm... Thats sounds strange.

Are you saying you cant change your scripts [But you'll have to anyway
to parse pagesize=] ?

I routinely use "cat /proc/xxx/numa_maps", and am stuck when a kernel
displays nothing (it happened on some debian released kernels)

Seeing pagesize=1GB is slightly better for human, and not that hard to
parse for a program.

By the way, "pagesize=4KiB" are just noise if you ask me, thats the
default PAGE_SIZE. This also breaks old scripts :)

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[Eric Dumazet]

Le mardi 04 octobre 2011 A  23:50 -0700, David Rientjes a A(C)crit :

> This would be great if all the /proc/pid/numa_maps consumers were human, 
> but unfortuantely that's not the case.  
> 
> I understand that this patchset was probably the result of me asking for 
> the pagesize= to be specified in each line and using pagesize=4K and 
> pagesize=2M as examples, but that exact usage is probably not what we 
> want.
> 
> As long as there are scripts that go through and read this information 
> (we have some internally), expressing them with differing units just makes 
> it more difficult to parse.  I'd rather them just be the byte count.
> 
> That way, 1G pages would just show pagesize=1073741824.  I don't think 
> that's too long and is much easier to parse systematically.
> 

Hmm... Thats sounds strange.

Are you saying you cant change your scripts [But you'll have to anyway
to parse pagesize=] ?

I routinely use "cat /proc/xxx/numa_maps", and am stuck when a kernel
displays nothing (it happened on some debian released kernels)

Seeing pagesize=1GB is slightly better for human, and not that hard to
parse for a program.

By the way, "pagesize=4KiB" are just noise if you ask me, thats the
default PAGE_SIZE. This also breaks old scripts :)



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[David Rientjes]

On Wed, 5 Oct 2011, Eric Dumazet wrote:

> > This would be great if all the /proc/pid/numa_maps consumers were human, 
> > but unfortuantely that's not the case.  
> > 
> > I understand that this patchset was probably the result of me asking for 
> > the pagesize= to be specified in each line and using pagesize=4K and 
> > pagesize=2M as examples, but that exact usage is probably not what we 
> > want.
> > 
> > As long as there are scripts that go through and read this information 
> > (we have some internally), expressing them with differing units just makes 
> > it more difficult to parse.  I'd rather them just be the byte count.
> > 
> > That way, 1G pages would just show pagesize=1073741824.  I don't think 
> > that's too long and is much easier to parse systematically.
> > 
> 
> Hmm... Thats sounds strange.
> 
> Are you saying you cant change your scripts [But you'll have to anyway
> to parse pagesize=] ?
> 

pagesize= isn't absolutely required, you can already get the thp stats 
from /proc/pid/smaps.  A script can then parse /proc/pid/numa_maps for the 
same vmas to determine the NUMA locality.  Adding pagesize= to numa_maps 
then doesn't change the script at all if it's robust.

> I routinely use "cat /proc/xxx/numa_maps", and am stuck when a kernel
> displays nothing (it happened on some debian released kernels)
> 
> Seeing pagesize=1GB is slightly better for human, and not that hard to
> parse for a program.
> 

Why on earth do we want to convert a byte value into a string so a script 
can convert it the other way around?  Do you have a hard time parsing 
4096, 2097152, and 1073741824 to be 4K, 2M, and 1G respectively?  Then you 
better not use the hugepage sysfs interface to configure multiple hugepage 
sizes because they're all specified in kB!

> By the way, "pagesize=4KiB" are just noise if you ask me, thats the
> default PAGE_SIZE. This also breaks old scripts :)
> 

-ENOPARSE.  It better not break any old script, I don't know why it would.  
Anything that emits a field=value tuple should be parsed as such.  I doubt 
you can find a well-distributed example since numa_maps output includes 
file, mapped, mapmax, anon, and dirty fields and not a single line of 
output would have all of them.

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[David Rientjes]

On Wed, 5 Oct 2011, Eric Dumazet wrote:

> > This would be great if all the /proc/pid/numa_maps consumers were human, 
> > but unfortuantely that's not the case.  
> > 
> > I understand that this patchset was probably the result of me asking for 
> > the pagesize= to be specified in each line and using pagesize=4K and 
> > pagesize=2M as examples, but that exact usage is probably not what we 
> > want.
> > 
> > As long as there are scripts that go through and read this information 
> > (we have some internally), expressing them with differing units just makes 
> > it more difficult to parse.  I'd rather them just be the byte count.
> > 
> > That way, 1G pages would just show pagesize=1073741824.  I don't think 
> > that's too long and is much easier to parse systematically.
> > 
> 
> Hmm... Thats sounds strange.
> 
> Are you saying you cant change your scripts [But you'll have to anyway
> to parse pagesize=] ?
> 

pagesize= isn't absolutely required, you can already get the thp stats 
from /proc/pid/smaps.  A script can then parse /proc/pid/numa_maps for the 
same vmas to determine the NUMA locality.  Adding pagesize= to numa_maps 
then doesn't change the script at all if it's robust.

> I routinely use "cat /proc/xxx/numa_maps", and am stuck when a kernel
> displays nothing (it happened on some debian released kernels)
> 
> Seeing pagesize=1GB is slightly better for human, and not that hard to
> parse for a program.
> 

Why on earth do we want to convert a byte value into a string so a script 
can convert it the other way around?  Do you have a hard time parsing 
4096, 2097152, and 1073741824 to be 4K, 2M, and 1G respectively?  Then you 
better not use the hugepage sysfs interface to configure multiple hugepage 
sizes because they're all specified in kB!

> By the way, "pagesize=4KiB" are just noise if you ask me, thats the
> default PAGE_SIZE. This also breaks old scripts :)
> 

-ENOPARSE.  It better not break any old script, I don't know why it would.  
Anything that emits a field=value tuple should be parsed as such.  I doubt 
you can find a well-distributed example since numa_maps output includes 
file, mapped, mapmax, anon, and dirty fields and not a single line of 
output would have all of them.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[Eric Dumazet]

Le mercredi 05 octobre 2011 à 00:23 -0700, David Rientjes a écrit :

> Why on earth do we want to convert a byte value into a string so a script 
> can convert it the other way around?  Do you have a hard time parsing 
> 4096, 2097152, and 1073741824 to be 4K, 2M, and 1G respectively?  

Yes I do. I dont have in my head all possible 2^X values, but K, M, G,
T : thats ok (less neurons needed)

You focus on current x86_64 hardware.

Some arches have lot of different choices. (powerpc has 64K, 16M, 16GB
pages)

In 10 years, you'll have pagesize=549755813888, or maybe
pagesize=8589934592

I pretty much prefer pagesize=512GB and pagesize=8TB

This is consistent with usual conventions and practice.

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[Eric Dumazet]

Le mercredi 05 octobre 2011 A  00:23 -0700, David Rientjes a A(C)crit :

> Why on earth do we want to convert a byte value into a string so a script 
> can convert it the other way around?  Do you have a hard time parsing 
> 4096, 2097152, and 1073741824 to be 4K, 2M, and 1G respectively?  

Yes I do. I dont have in my head all possible 2^X values, but K, M, G,
T : thats ok (less neurons needed)

You focus on current x86_64 hardware.

Some arches have lot of different choices. (powerpc has 64K, 16M, 16GB
pages)

In 10 years, you'll have pagesize=549755813888, or maybe
pagesize=8589934592

I pretty much prefer pagesize=512GB and pagesize=8TB

This is consistent with usual conventions and practice.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[Dave Hansen]

On Tue, 2011-10-04 at 23:50 -0700, David Rientjes wrote:
> That way, 1G pages would just show pagesize=1073741824.  I don't think 
> that's too long and is much easier to parse systematically. 

OK, I'll switch it back.

-- Dave

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[Dave Hansen]

On Tue, 2011-10-04 at 23:50 -0700, David Rientjes wrote:
> That way, 1G pages would just show pagesize=1073741824.  I don't think 
> that's too long and is much easier to parse systematically. 

OK, I'll switch it back.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[Dave Hansen]

On Wed, 2011-10-05 at 09:09 +0200, Eric Dumazet wrote:
> By the way, "pagesize=4KiB" are just noise if you ask me, thats the
> default PAGE_SIZE. This also breaks old scripts :)

How does it break old scripts?

-- Dave

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[Dave Hansen]

On Wed, 2011-10-05 at 09:09 +0200, Eric Dumazet wrote:
> By the way, "pagesize=4KiB" are just noise if you ask me, thats the
> default PAGE_SIZE. This also breaks old scripts :)

How does it break old scripts?

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[Eric Dumazet]

Le mercredi 05 octobre 2011 à 08:22 -0700, Dave Hansen a écrit :
> On Wed, 2011-10-05 at 09:09 +0200, Eric Dumazet wrote:
> > By the way, "pagesize=4KiB" are just noise if you ask me, thats the
> > default PAGE_SIZE. This also breaks old scripts :)
> 
> How does it break old scripts?
> 

Old scripts just parse numa_maps, and on typical machines where
hugepages are not used, they dont have to care about page size.
They assume pages are 4KB.

Adding a new word (pagesize=...) might break them, but personally I dont
care.

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[Eric Dumazet]

Le mercredi 05 octobre 2011 A  08:22 -0700, Dave Hansen a A(C)crit :
> On Wed, 2011-10-05 at 09:09 +0200, Eric Dumazet wrote:
> > By the way, "pagesize=4KiB" are just noise if you ask me, thats the
> > default PAGE_SIZE. This also breaks old scripts :)
> 
> How does it break old scripts?
> 

Old scripts just parse numa_maps, and on typical machines where
hugepages are not used, they dont have to care about page size.
They assume pages are 4KB.

Adding a new word (pagesize=...) might break them, but personally I dont
care.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[David Rientjes]

On Wed, 5 Oct 2011, Eric Dumazet wrote:

> > Why on earth do we want to convert a byte value into a string so a script 
> > can convert it the other way around?  Do you have a hard time parsing 
> > 4096, 2097152, and 1073741824 to be 4K, 2M, and 1G respectively?  
> 
> Yes I do. I dont have in my head all possible 2^X values, but K, M, G,
> T : thats ok (less neurons needed)
> 
> You focus on current x86_64 hardware.
> 
> Some arches have lot of different choices. (powerpc has 64K, 16M, 16GB
> pages)
> 
> In 10 years, you'll have pagesize=549755813888, or maybe
> pagesize=8589934592
> 
> I pretty much prefer pagesize=512GB and pagesize=8TB
> 
> This is consistent with usual conventions and practice.
> 

I'm indifferent whether it's displayed in bytes (so a script could do 
pagesize * anon, for example, and find the exact amount of anonymous 
memory for that vma without needing smaps) or in KB like /proc/pid/smaps, 
grep Hugepagesize /proc/meminfo, and ls /sys/kernel/mm/hugepages.

In other words, pagesize= in /proc/pid/numa_maps is the least of your 
worries if you're serious about this: you would have already struggled 
with smaps, meminfo, and the sysfs interface for reserving the hugepages 
in the first place.

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[David Rientjes]

On Wed, 5 Oct 2011, Eric Dumazet wrote:

> > Why on earth do we want to convert a byte value into a string so a script 
> > can convert it the other way around?  Do you have a hard time parsing 
> > 4096, 2097152, and 1073741824 to be 4K, 2M, and 1G respectively?  
> 
> Yes I do. I dont have in my head all possible 2^X values, but K, M, G,
> T : thats ok (less neurons needed)
> 
> You focus on current x86_64 hardware.
> 
> Some arches have lot of different choices. (powerpc has 64K, 16M, 16GB
> pages)
> 
> In 10 years, you'll have pagesize=549755813888, or maybe
> pagesize=8589934592
> 
> I pretty much prefer pagesize=512GB and pagesize=8TB
> 
> This is consistent with usual conventions and practice.
> 

I'm indifferent whether it's displayed in bytes (so a script could do 
pagesize * anon, for example, and find the exact amount of anonymous 
memory for that vma without needing smaps) or in KB like /proc/pid/smaps, 
grep Hugepagesize /proc/meminfo, and ls /sys/kernel/mm/hugepages.

In other words, pagesize= in /proc/pid/numa_maps is the least of your 
worries if you're serious about this: you would have already struggled 
with smaps, meminfo, and the sysfs interface for reserving the hugepages 
in the first place.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[David Rientjes]

On Wed, 5 Oct 2011, Eric Dumazet wrote:

> > How does it break old scripts?
> > 
> 
> Old scripts just parse numa_maps, and on typical machines where
> hugepages are not used, they dont have to care about page size.
> They assume pages are 4KB.
> 
> Adding a new word (pagesize=...) might break them, but personally I dont
> care.
> 

If your script is only parsing numa_maps, then Dave's effort is actually 
allowing them to be fixed rather than breaking them.  We could silently 
continue to export the page counts without specifying the size (hugetlb 
pages are counted in their true hugepage size, THP pages are counted in 
PAGE_SIZE units), but then a script would always be broken unless they use 
smaps as well.  Dave's addition of pagesize allows numa_maps to stand on 
its own and actually be useful when hugepages are used.

Re: [RFCv3][PATCH 4/4] show page size in /proc/$pid/numa_maps

[David Rientjes]

On Wed, 5 Oct 2011, Eric Dumazet wrote:

> > How does it break old scripts?
> > 
> 
> Old scripts just parse numa_maps, and on typical machines where
> hugepages are not used, they dont have to care about page size.
> They assume pages are 4KB.
> 
> Adding a new word (pagesize=...) might break them, but personally I dont
> care.
> 

If your script is only parsing numa_maps, then Dave's effort is actually 
allowing them to be fixed rather than breaking them.  We could silently 
continue to export the page counts without specifying the size (hugetlb 
pages are counted in their true hugepage size, THP pages are counted in 
PAGE_SIZE units), but then a script would always be broken unless they use 
smaps as well.  Dave's addition of pagesize allows numa_maps to stand on 
its own and actually be useful when hugepages are used.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>