[RFC 0/2] Delay initializing of large sections of memory

[RFC 0/2] Delay initializing of large sections of memory

[Nathan Zimmer]

This rfc patch set delays initializing large sections of memory until we have
started cpus.  This has the effect of reducing startup times on large memory
systems.  On 16TB it can take over an hour to boot and most of that time
is spent initializing memory.

We avoid that bottleneck by delaying initialization until after we have 
started multiple cpus and can initialize in a multithreaded manner.
This allows us to actually reduce boot time rather then just moving around
the point of initialization.

Mike and I have worked on this set for a while, with him doing the most of the
heavy lifting, and are eager for some feedback.

Mike Travis (2):
  x86_64, mm: Delay initializing large portion of memory
  x86_64, mm: Reinsert the absent memory

 Documentation/kernel-parameters.txt |  15 ++
 arch/x86/Kconfig                    |  10 ++
 arch/x86/include/asm/e820.h         |  16 +-
 arch/x86/kernel/e820.c              | 292 +++++++++++++++++++++++++++++++++++-
 drivers/base/memory.c               |  83 ++++++++++
 include/linux/memory.h              |   5 +
 6 files changed, 413 insertions(+), 8 deletions(-)

-- 
1.8.2.1

[RFC 1/2] x86_64, mm: Delay initializing large portion of memory

[Nathan Zimmer]

On a 16TB system it can takes upwards of two hours to boot the system with
about 60% of the time being spent initializing memory.  This patch delays
initializing a large portion of memory until after the system is booted.
This can significantly reduce the time it takes the boot the system down
to the 15 to 30 minute range.

Signed-off-by: Mike Travis <travis@sgi.com>
Signed-off-by: Nathan Zimmer <nzimmer@sgi.com>
Cc: Rob Landley <rob@landley.net>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
---
 Documentation/kernel-parameters.txt |  15 ++++
 arch/x86/Kconfig                    |  10 +++
 arch/x86/include/asm/e820.h         |  16 +++-
 arch/x86/kernel/e820.c              | 163 ++++++++++++++++++++++++++++++++++--
 4 files changed, 196 insertions(+), 8 deletions(-)

diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 2fe6e76..77b8195 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -706,6 +706,21 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			Defaults to the default architecture's huge page size
 			if not specified.
 
+	delay_mem_init=B:M:n:l:h
+			This delays the initialization of a large portion of
+			memory by inserting it into the "absent" memory list.
+			This allows the system to boot up much faster at the
+			expense of the time needed to add this absent memory
+			after the system has booted.  That however can be done
+			in parallel with other operations.
+			Format: B:M:n:l:h
+			    (1 << B) is the block size (bsize)
+			    	     ['0' indicates use the default 128M]
+			    (1 << M) is the address space per node
+			    (n * bsize) is minimum sized node memory to slice
+			    (l * bisze) is low memory to leave on node
+			    (h * bisze) is high memory to leave on node
+
 	dhash_entries=	[KNL]
 			Set number of hash buckets for dentry cache.
 
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 685692c..28b6b2c 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1566,6 +1566,16 @@ config EFI_STUB
 
 	  See Documentation/x86/efi-stub.txt for more information.
 
+config DELAY_MEM_INIT
+	bool "Delay memory initialization"
+	depends on EFI && MEMORY_HOTPLUG_SPARSE
+	---help---
+	  This  option delays initializing a large portion of memory
+	  until after the system is booted.  This can significantly
+	  reduce the time it takes the boot the system when there
+	  is a significant amount of memory present.  Systems with
+	  8TB or more of memory benefit the most.
+
 config SECCOMP
 	def_bool y
 	prompt "Enable seccomp to safely compute untrusted bytecode"
diff --git a/arch/x86/include/asm/e820.h b/arch/x86/include/asm/e820.h
index cccd07f..05278d8 100644
--- a/arch/x86/include/asm/e820.h
+++ b/arch/x86/include/asm/e820.h
@@ -18,8 +18,6 @@ extern int e820_any_mapped(u64 start, u64 end, unsigned type);
 extern int e820_all_mapped(u64 start, u64 end, unsigned type);
 extern void e820_add_region(u64 start, u64 size, int type);
 extern void e820_print_map(char *who);
-extern int
-sanitize_e820_map(struct e820entry *biosmap, int max_nr_map, u32 *pnr_map);
 extern u64 e820_update_range(u64 start, u64 size, unsigned old_type,
 			       unsigned new_type);
 extern u64 e820_remove_range(u64 start, u64 size, unsigned old_type,
@@ -31,6 +29,20 @@ extern int e820_search_gap(unsigned long *gapstart, unsigned long *gapsize,
 struct setup_data;
 extern void parse_e820_ext(struct setup_data *data);
 
+extern int
+__sanitize_e820_map(struct e820entry *biosmap, int max_nr_map, u32 *pnr_map);
+
+#ifdef CONFIG_DELAY_MEM_INIT
+extern int
+sanitize_e820_map(struct e820entry *biosmap, int max_nr_map, u32 *pnr_map);
+#else
+static inline int sanitize_e820_map(struct e820entry *biosmap, int max_nr_map,
+				    u32 *pnr_map)
+{
+	return __sanitize_e820_map(biosmap, max_nr_map, pnr_map);
+}
+#endif /* CONFIG_DELAY_MEM_INIT */
+
 #if defined(CONFIG_X86_64) || \
 	(defined(CONFIG_X86_32) && defined(CONFIG_HIBERNATION))
 extern void e820_mark_nosave_regions(unsigned long limit_pfn);
diff --git a/arch/x86/kernel/e820.c b/arch/x86/kernel/e820.c
index d32abea..3752dc5 100644
--- a/arch/x86/kernel/e820.c
+++ b/arch/x86/kernel/e820.c
@@ -21,6 +21,10 @@
 #include <linux/memblock.h>
 #include <linux/sort.h>
 
+#ifdef CONFIG_DELAY_MEM_INIT
+#include <linux/memory.h>
+#endif
+
 #include <asm/e820.h>
 #include <asm/proto.h>
 #include <asm/setup.h>
@@ -41,6 +45,9 @@
  */
 struct e820map e820;
 struct e820map e820_saved;
+#ifdef CONFIG_DELAY_MEM_INIT
+struct e820map e820_absent;
+#endif
 
 /* For PCI or other memory-mapped resources */
 unsigned long pci_mem_start = 0xaeedbabe;
@@ -155,20 +162,25 @@ static void __init e820_print_type(u32 type)
 	}
 }
 
-void __init e820_print_map(char *who)
+static void __init __e820_print_map(char *who, struct e820map *e820x)
 {
 	int i;
 
-	for (i = 0; i < e820.nr_map; i++) {
+	for (i = 0; i < e820x->nr_map; i++) {
 		printk(KERN_INFO "%s: [mem %#018Lx-%#018Lx] ", who,
-		       (unsigned long long) e820.map[i].addr,
+		       (unsigned long long) e820x->map[i].addr,
 		       (unsigned long long)
-		       (e820.map[i].addr + e820.map[i].size - 1));
-		e820_print_type(e820.map[i].type);
+		       (e820x->map[i].addr + e820x->map[i].size - 1));
+		e820_print_type(e820x->map[i].type);
 		printk(KERN_CONT "\n");
 	}
 }
 
+void __init e820_print_map(char *who)
+{
+	__e820_print_map(who, &e820);
+}
+
 /*
  * Sanitize the BIOS e820 map.
  *
@@ -252,7 +264,7 @@ static int __init cpcompare(const void *a, const void *b)
 	return (ap->addr != ap->pbios->addr) - (bp->addr != bp->pbios->addr);
 }
 
-int __init sanitize_e820_map(struct e820entry *biosmap, int max_nr_map,
+int __init __sanitize_e820_map(struct e820entry *biosmap, int max_nr_map,
 			     u32 *pnr_map)
 {
 	static struct change_member change_point_list[2*E820_X_MAX] __initdata;
@@ -378,6 +390,145 @@ int __init sanitize_e820_map(struct e820entry *biosmap, int max_nr_map,
 	return 0;
 }
 
+#ifdef CONFIG_DELAY_MEM_INIT
+static u64 block_size;		/* block size */
+static u64 mem_per_node;	/* mem_per_node address increment */
+static u64 min_region_size;	/* min size of region to slice from */
+static u64 pre_region_size;	/* multiply bsize for node low memory */
+static u64 post_region_size;	/* multiply bsize for node high memory */
+
+static int __init setup_delay_mem_init(char *str)
+{
+	int bbits, mpnbits, minmult, premult, postmult;
+
+	if (sscanf(str, "%d:%d:%d:%d:%d", &bbits, &mpnbits,
+		&minmult, &premult, &postmult) != 5)
+		goto error;
+
+	if (!bbits)	/* default block size */
+		bbits = MIN_MEMORY_BLOCK_SIZE;
+
+	if (!mpnbits || !minmult || !premult || !postmult)
+		goto error;
+
+	/* (The '+1' because memory can end on a non-block size boundary) */
+	if (bbits > mpnbits || (premult + postmult + 1) > minmult
+	    || bbits > MAX_PHYSMEM_BITS || mpnbits > MAX_PHYSMEM_BITS)
+		goto error;
+
+	block_size = 1ull << bbits;
+	mem_per_node = 1ull << mpnbits;
+	min_region_size = block_size * minmult;
+	pre_region_size = block_size * premult;
+	post_region_size = block_size * postmult;
+	pr_info("e820: delay_mem_init=%s (bsize:%llx mem_incr:%llx)\n",
+		str, block_size, mem_per_node);
+	return 0;
+
+error:
+	mem_per_node = 0;
+	pr_err("e820: Invalid parameter: delay_mem_init=%s\n", str);
+	return -EINVAL;
+}
+early_param("delay_mem_init", setup_delay_mem_init);
+
+/* Check if region already present */
+static int __init e820_region_found(struct e820map *e820x,
+					u64 start, u64 size, int type)
+{
+	u64 end = start + size;
+	int i;
+
+	for (i = 0; i < e820x->nr_map; i++) {
+		struct e820entry *ei = &e820x->map[i];
+		u64 ei_end;
+
+		if (type != ei->type)
+			continue;
+
+		ei_end = ei->addr + ei->size;
+		if (ei->addr <= start && end <= ei_end)
+			return 1;
+	}
+	return 0;
+}
+
+/* Move region of memory from e820 to e820_absent */
+static void __init add_e820_absent(u64 start, u64 size, int type)
+{
+	u64 bsize = block_size;
+	u64 size0, addr1, size1, addr2, size2;
+
+	/* start/size0 ... addr1/size1 [removed region] ... addr2/size2 */
+	addr1 = start + pre_region_size;
+	size0 = addr1 - start;
+
+	addr2 = ((start + size) & ~(bsize - 1)) - post_region_size;
+	size2 = (size + start) - addr2;
+	size1 = size - size0 - size2;
+
+	if (size1 && !(size1 & (bsize - 1))) {
+		if (e820_region_found(&e820, addr1, size1, type))
+			e820_remove_range(addr1, size1, type, 1);
+
+		if (!e820_region_found(&e820_absent, addr1, size1, type))
+			__e820_add_region(&e820_absent, addr1, size1, type);
+	}
+}
+
+/* Move memory from Node 1 thru last Node - 1 to the "absent" list */
+static void __init setup_e820_absent(void)
+{
+	u64 last_base = 0;
+	u64 last_size = 0;
+	u64 last_start = 0;
+	int i;
+
+	if (!mem_per_node)
+		return;
+
+	for (i = 0; i < e820.nr_map; i++) {
+		u64 start = e820.map[i].addr;
+		u64 size = e820.map[i].size;
+		int type = e820.map[i].type;
+
+		if (type != E820_RAM)
+			continue;
+
+		if (start > last_start && (start & (mem_per_node - 1)) == 0) {
+			if (last_base) {
+				add_e820_absent(last_base, last_size, type);
+				last_base = 0;
+			}
+			if (size >= min_region_size) {
+				last_base = start;
+				last_size = size;
+			}
+			last_start = start;
+		}
+	}
+	if (last_start)
+		__e820_print_map("e820_absent", &e820_absent);
+	else
+		pr_info("e820: No memory found to move to absent list\n");
+}
+
+/* Check for possible "absent" (delayed) memory after each addition */
+int __init sanitize_e820_map(struct e820entry *biosmap, int max_nr_map,
+			     u32 *pnr_map)
+{
+	int ret;
+
+	ret = __sanitize_e820_map(biosmap, max_nr_map, pnr_map);
+
+	if (ret == 0 && biosmap == e820.map) {
+		setup_e820_absent();
+		ret = __sanitize_e820_map(biosmap, max_nr_map, pnr_map);
+	}
+	return ret;
+}
+#endif /* CONFIG_DELAY_MEM_INIT */
+
 static int __init __append_e820_map(struct e820entry *biosmap, int nr_map)
 {
 	while (nr_map) {
-- 
1.8.2.1

Re: [RFC 1/2] x86_64, mm: Delay initializing large portion of memory

[Rob Landley]

On 06/21/2013 11:25:33 AM, Nathan Zimmer wrote:
> On a 16TB system it can takes upwards of two hours to boot the system  
> with
> about 60% of the time being spent initializing memory.  This patch  
> delays
> initializing a large portion of memory until after the system is  
> booted.
> This can significantly reduce the time it takes the boot the system  
> down
> to the 15 to 30 minute range.

Why is this conditional? Initialize the minimum amount of memory to  
bring up each NUMA node, and then have each processor initialize its  
own memory. I would have thought it was already doing this...


> +	delay_mem_init=B:M:n:l:h
> +			This delays the initialization of a large  
> portion of
> +			memory by inserting it into the "absent" memory  
> list.
> +			This allows the system to boot up much faster  
> at the
> +			expense of the time needed to add this absent  
> memory
> +			after the system has booted.  That however can  
> be done
> +			in parallel with other operations.

This seems like a giant advertisement primarily aimed at repeating why  
you think we need to merge the patch, not explaining what it is or how  
to use it.

I would rephrase:

			Defer memory initialization until after SMP  
init (so
			large memory ranges can be initialized in  
parallel) by
			moving memory not needed during boot to the  
"absent" list.

And I repeat: why do we need to micromanage this? It sounds like all  
NUMA systems should do something like this. (Single-threaded memory  
initialization in an SMP system is kind of weird.)

> +			Format: B:M:n:l:h
> +			    (1 << B) is the block size (bsize)
> +			    	     ['0' indicates use the default  
> 128M]
> +			    (1 << M) is the address space per node
> +			    (n * bsize) is minimum sized node memory to  
> slice
> +			    (l * bisze) is low memory to leave on node
> +			    (h * bisze) is high memory to leave on node

I don't understand this in the slightest. I understand "low memory to  
leave on the node", I have no idea why there are four other parameters.


> +config DELAY_MEM_INIT
> +	bool "Delay memory initialization"
> +	depends on EFI && MEMORY_HOTPLUG_SPARSE
> +	---help---
> +	  This  option delays initializing a large portion of memory
> +	  until after the system is booted.  This can significantly
> +	  reduce the time it takes the boot the system when there
> +	  is a significant amount of memory present.  Systems with
> +	  8TB or more of memory benefit the most.

I can see an SMP phone wanting to use this to shave a quarter second  
off its boot time. Your "large portion of memory" description is a bit  
myopic.

Rob

[RFC 2/2] x86_64, mm: Reinsert the absent memory

[Nathan Zimmer]

The memory we set aside in the previous patch needs to be reinserted.
We start this process via late_initcall so we will have multiple cpus to do
the work.

Signed-off-by: Mike Travis <travis@sgi.com>
Signed-off-by: Nathan Zimmer <nzimmer@sgi.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org> 
Cc: Yinghai Lu <yinghai@kernel.org>
---
 arch/x86/kernel/e820.c | 129 +++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/base/memory.c  |  83 +++++++++++++++++++++++++++++++
 include/linux/memory.h |   5 ++
 3 files changed, 217 insertions(+)

diff --git a/arch/x86/kernel/e820.c b/arch/x86/kernel/e820.c
index 3752dc5..d31039d 100644
--- a/arch/x86/kernel/e820.c
+++ b/arch/x86/kernel/e820.c
@@ -23,6 +23,7 @@
 
 #ifdef CONFIG_DELAY_MEM_INIT
 #include <linux/memory.h>
+#include <linux/delay.h>
 #endif
 
 #include <asm/e820.h>
@@ -397,6 +398,22 @@ static u64 min_region_size;	/* min size of region to slice from */
 static u64 pre_region_size;	/* multiply bsize for node low memory */
 static u64 post_region_size;	/* multiply bsize for node high memory */
 
+static unsigned long add_absent_work_start_time;
+static unsigned long add_absent_work_stop_time;
+static unsigned int add_absent_job_count;
+static atomic_t add_absent_work_count;
+
+struct absent_work {
+	struct work_struct	work;
+	struct absent_work	*next;
+	atomic_t		busy;
+	int			cpu;
+	int			node;
+	int			index;
+};
+static DEFINE_PER_CPU(struct absent_work, absent_work);
+static struct absent_work *first_absent_work;
+
 static int __init setup_delay_mem_init(char *str)
 {
 	int bbits, mpnbits, minmult, premult, postmult;
@@ -527,6 +544,118 @@ int __init sanitize_e820_map(struct e820entry *biosmap, int max_nr_map,
 	}
 	return ret;
 }
+
+/* Assign a cpu for this memory chunk and get the per_cpu absent_work struct */
+static struct absent_work *get_absent_work(int node)
+{
+	int cpu;
+
+	for_each_cpu(cpu, cpumask_of_node(node)) {
+		struct absent_work *aws = &per_cpu(absent_work, cpu);
+		if (aws->node)
+			continue;
+		aws->cpu = cpu;
+		aws->node = node;
+		return aws;
+	}
+
+	/* (if this becomes a problem, we can use a cpu on another node) */
+	pr_crit("e820: No CPU on Node %d to schedule absent_work\n", node);
+	return NULL;
+}
+
+/* Count of 'not done' processes */
+static int count_absent_work_notdone(void)
+{
+	struct absent_work *aws;
+	int notdone = 0;
+
+	for (aws = first_absent_work; aws; aws = aws->next)
+		if (atomic_read(&aws->busy) < 2)
+			notdone++;
+
+	return notdone;
+}
+
+/* The absent_work thread */
+static void add_absent_memory_work(struct work_struct *work)
+{
+	struct absent_work *aws;
+	u64 phys_addr, size;
+	int ret;
+
+	aws = container_of(work, struct absent_work, work);
+
+	phys_addr = e820_absent.map[aws->index].addr;
+	size = e820_absent.map[aws->index].size;
+	ret = memory_add_absent(aws->node, phys_addr, size);
+	if (ret)
+		pr_crit("e820: Error %d adding absent memory %llx %llx (%d)\n",
+			ret, phys_addr, size, aws->node);
+
+	atomic_set(&aws->busy, 2);
+	atomic_dec(&add_absent_work_count);
+
+	/* if no one is waiting, then snap stop time */
+	if (!count_absent_work_notdone())
+		add_absent_work_stop_time = get_seconds();
+}
+
+/* Initialize absent_work threads */
+static int add_absent_memory(void)
+{
+	struct absent_work *aws = NULL;
+	int cpu, i;
+
+	add_absent_work_start_time = get_seconds();
+	add_absent_work_stop_time = 0;
+	atomic_set(&add_absent_work_count, 0);
+
+	for_each_online_cpu(cpu) {
+		struct absent_work *aws = &per_cpu(absent_work, cpu);
+		aws->node = 0;
+	}
+
+	/* setup each work thread */
+	for (i = 0; i < e820_absent.nr_map; i++) {
+		u64 phys_addr = e820_absent.map[i].addr;
+		int node = memory_add_physaddr_to_nid(phys_addr);
+
+		if (!node_online(node))
+			continue;
+
+		if (!aws) {
+			aws = get_absent_work(node);
+			first_absent_work = aws;
+		} else {
+			aws->next = get_absent_work(node);
+			aws = aws->next;
+		}
+
+		if (!aws)
+			continue;
+
+		INIT_WORK(&aws->work, add_absent_memory_work);
+		atomic_set(&aws->busy, 0);
+		aws->index = i;
+
+		/* schedule absent_work thread */
+		if (!schedule_work_on(aws->cpu, &aws->work))
+			BUG();
+	}
+
+
+	pr_info("e820: Add absent memory started\n");
+
+	return 0;
+}
+
+/* Called during bootup to start adding absent_mem early */
+static int absent_memory_init(void)
+{
+	return add_absent_memory();
+}
+late_initcall(absent_memory_init);
 #endif /* CONFIG_DELAY_MEM_INIT */
 
 static int __init __append_e820_map(struct e820entry *biosmap, int nr_map)
diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index 14f8a69..5b4245a 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -442,6 +442,89 @@ static inline int memory_probe_init(void)
 }
 #endif
 
+#ifdef CONFIG_DELAY_MEM_INIT
+static struct memory_block *memory_get_block(u64 phys_addr,
+					struct memory_block *last_mem_blk)
+{
+	unsigned long pfn = phys_addr >> PAGE_SHIFT;
+	struct memory_block *mem_blk = NULL;
+	struct mem_section *mem_sect;
+	unsigned long section_nr = pfn_to_section_nr(pfn);
+
+	if (!present_section_nr(section_nr))
+		return NULL;
+
+	mem_sect = __nr_to_section(section_nr);
+	mem_blk = find_memory_block_hinted(mem_sect, last_mem_blk);
+	return mem_blk;
+}
+
+/* addr and size must be aligned on memory_block_size boundaries */
+int memory_add_absent(int nid, u64 phys_addr, u64 size)
+{
+	struct memory_block *mem = NULL;
+	struct page *first_page;
+	unsigned long block_sz;
+	unsigned long nr_pages;
+	unsigned long start_pfn;
+	int ret;
+
+	block_sz = get_memory_block_size();
+	if (phys_addr & (block_sz - 1) || size & (block_sz - 1))
+		return -EINVAL;
+
+	/* memory already present? */
+	if (memory_get_block(phys_addr, NULL))
+		return -EBUSY;
+
+	ret = add_memory(nid, phys_addr, size);
+	if (ret)
+		return ret;
+
+	/* grab first block to use for onlining process */
+	mem = memory_get_block(phys_addr, NULL);
+	if (!mem)
+		return -ENOMEM;
+
+	first_page = pfn_to_page(mem->start_section_nr << PFN_SECTION_SHIFT);
+	start_pfn = page_to_pfn(first_page);
+	nr_pages = size >> PAGE_SHIFT;
+
+	ret = online_pages(start_pfn, nr_pages, ONLINE_KEEP);
+	if (ret)
+		return ret;
+
+	for (;;) {
+		/* we already have first block from above */
+		mutex_lock(&mem->state_mutex);
+		if (mem->state == MEM_OFFLINE) {
+			mem->state = MEM_ONLINE;
+			kobject_uevent(&mem->dev.kobj, KOBJ_ONLINE);
+		}
+		mutex_unlock(&mem->state_mutex);
+
+		phys_addr += block_sz;
+		size -= block_sz;
+		if (!size)
+			break;
+
+		mem = memory_get_block(phys_addr, mem);
+		if (mem)
+			continue;
+
+		pr_err("memory_get_block failed at %llx\n", phys_addr);
+		return -EFAULT;
+	}
+	return 0;
+}
+
+#else
+static inline int start_add_absent_init(void)
+{
+	return 0;
+}
+#endif /* CONFIG_DELAY_MEM_INIT */
+
 #ifdef CONFIG_MEMORY_FAILURE
 /*
  * Support for offlining pages of memory
diff --git a/include/linux/memory.h b/include/linux/memory.h
index 85c31a8..a000c54 100644
--- a/include/linux/memory.h
+++ b/include/linux/memory.h
@@ -128,6 +128,11 @@ extern struct memory_block *find_memory_block(struct mem_section *);
 enum mem_add_context { BOOT, HOTPLUG };
 #endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
 
+#ifdef CONFIG_DELAY_MEM_INIT
+extern int memory_add_absent(int nid, u64 phys_addr, u64 size);
+#endif
+
+
 #ifdef CONFIG_MEMORY_HOTPLUG
 #define hotplug_memory_notifier(fn, pri) ({		\
 	static __meminitdata struct notifier_block fn##_mem_nb =\
-- 
1.8.2.1

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Ingo Molnar]

* Nathan Zimmer <nzimmer@sgi.com> wrote:

> The memory we set aside in the previous patch needs to be reinserted.
> We start this process via late_initcall so we will have multiple cpus to do
> the work.
> 
> Signed-off-by: Mike Travis <travis@sgi.com>
> Signed-off-by: Nathan Zimmer <nzimmer@sgi.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Cc: Andrew Morton <akpm@linux-foundation.org> 
> Cc: Yinghai Lu <yinghai@kernel.org>
> ---
>  arch/x86/kernel/e820.c | 129 +++++++++++++++++++++++++++++++++++++++++++++++++
>  drivers/base/memory.c  |  83 +++++++++++++++++++++++++++++++
>  include/linux/memory.h |   5 ++
>  3 files changed, 217 insertions(+)
> 
> diff --git a/arch/x86/kernel/e820.c b/arch/x86/kernel/e820.c
> index 3752dc5..d31039d 100644
> --- a/arch/x86/kernel/e820.c
> +++ b/arch/x86/kernel/e820.c
> @@ -23,6 +23,7 @@
>  
>  #ifdef CONFIG_DELAY_MEM_INIT
>  #include <linux/memory.h>
> +#include <linux/delay.h>
>  #endif
>  
>  #include <asm/e820.h>
> @@ -397,6 +398,22 @@ static u64 min_region_size;	/* min size of region to slice from */
>  static u64 pre_region_size;	/* multiply bsize for node low memory */
>  static u64 post_region_size;	/* multiply bsize for node high memory */
>  
> +static unsigned long add_absent_work_start_time;
> +static unsigned long add_absent_work_stop_time;
> +static unsigned int add_absent_job_count;
> +static atomic_t add_absent_work_count;
> +
> +struct absent_work {
> +	struct work_struct	work;
> +	struct absent_work	*next;
> +	atomic_t		busy;
> +	int			cpu;
> +	int			node;
> +	int			index;
> +};
> +static DEFINE_PER_CPU(struct absent_work, absent_work);
> +static struct absent_work *first_absent_work;

That's 4.5 GB/sec initialization speed - that feels a bit slow and the 
boot time effect should be felt on smaller 'a couple of gigabytes' desktop 
boxes as well. Do we know exactly where the 2 hours of boot time on a 32 
TB system is spent?

While you cannot profile the boot process (yet), you could try your 
delayed patch and run a "perf record -g" call-graph profiling of the 
late-time initialization routines. What does 'perf report' show?

Delayed initialization makes sense I guess because 32 TB is a lot of 
memory - I'm just wondering whether there's some low hanging fruits left 
in the mem init code, that code is certainly not optimized for 
performance.

Plus with a struct page size of around 64 bytes (?) 32 TB of RAM has 512 
GB of struct page arrays alone. Initializing those will take quite some 
time as well - and I suspect they are allocated via zeroing them first. If 
that memset() exists then getting rid of it might be a good move as well.

Yet another thing to consider would be to implement an initialization 
speedup of 3 orders of magnitude: initialize on the large page (2MB) 
grandularity and on-demand delay the initialization of the 4K granular 
struct pages [but still allocating them] - which I suspect are a good 
chunk of the overhead? That way we could initialize in 2MB steps and speed 
up the 2 hours bootup of 32 TB of RAM to 14 seconds...

[ The cost would be one more branch in the buddy allocator, to detect
  not-yet-initialized 2 MB chunks as we encounter them. Acceptable I 
  think. ]

Thanks,

	Ingo

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Ingo Molnar]

* Ingo Molnar <mingo@kernel.org> wrote:

> Yet another thing to consider would be to implement an initialization 
> speedup of 3 orders of magnitude: initialize on the large page (2MB) 
> grandularity and on-demand delay the initialization of the 4K granular 
> struct pages [but still allocating them] - which I suspect are a good 
> chunk of the overhead? That way we could initialize in 2MB steps and speed 
> up the 2 hours bootup of 32 TB of RAM to 14 seconds...
> 
> [ The cost would be one more branch in the buddy allocator, to detect
>   not-yet-initialized 2 MB chunks as we encounter them. Acceptable I 
>   think. ]

One advantage of this scheme would be that we could use it on pretty much 
any box, it would provide instant boot time speedups everywhere [a couple 
of hundred msecs on a small 4GB box - significant I think] - and would 
spread out and parallelize initialization to later stages.

Thanks,

	Ingo

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[H. Peter Anvin]

On 06/23/2013 02:32 AM, Ingo Molnar wrote:
> 
> * Ingo Molnar <mingo@kernel.org> wrote:
> 
>> Yet another thing to consider would be to implement an initialization 
>> speedup of 3 orders of magnitude: initialize on the large page (2MB) 
>> grandularity and on-demand delay the initialization of the 4K granular 
>> struct pages [but still allocating them] - which I suspect are a good 
>> chunk of the overhead? That way we could initialize in 2MB steps and speed 
>> up the 2 hours bootup of 32 TB of RAM to 14 seconds...
>>
>> [ The cost would be one more branch in the buddy allocator, to detect
>>   not-yet-initialized 2 MB chunks as we encounter them. Acceptable I 
>>   think. ]
> 
> One advantage of this scheme would be that we could use it on pretty much 
> any box, it would provide instant boot time speedups everywhere [a couple 
> of hundred msecs on a small 4GB box - significant I think] - and would 
> spread out and parallelize initialization to later stages.
> 

Even better if we could start at the 1 GB level, which most of these
really huge machines will have hardware support for.

	-hpa

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Ingo Molnar]

* H. Peter Anvin <hpa@zytor.com> wrote:

> On 06/23/2013 02:32 AM, Ingo Molnar wrote:
> > 
> > * Ingo Molnar <mingo@kernel.org> wrote:
> > 
> >> Yet another thing to consider would be to implement an initialization 
> >> speedup of 3 orders of magnitude: initialize on the large page (2MB) 
> >> grandularity and on-demand delay the initialization of the 4K granular 
> >> struct pages [but still allocating them] - which I suspect are a good 
> >> chunk of the overhead? That way we could initialize in 2MB steps and speed 
> >> up the 2 hours bootup of 32 TB of RAM to 14 seconds...
> >>
> >> [ The cost would be one more branch in the buddy allocator, to detect
> >>   not-yet-initialized 2 MB chunks as we encounter them. Acceptable I 
> >>   think. ]
> > 
> > One advantage of this scheme would be that we could use it on pretty much 
> > any box, it would provide instant boot time speedups everywhere [a couple 
> > of hundred msecs on a small 4GB box - significant I think] - and would 
> > spread out and parallelize initialization to later stages.
> 
> Even better if we could start at the 1 GB level, which most of these 
> really huge machines will have hardware support for.

That might be a bit too granular: if we hit such an uninitialized block of 
memory we'd have to process 262,144 pages - potentially from an IRQ 
handler that does GFP_ATOMIC... or other latency critical code.

With 2MB we'd have to on-demand initialize 512 pages, which shouldn't show 
up during normal use.

Thanks,

	Ingo

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[H. Peter Anvin]

On 06/24/2013 12:39 PM, Ingo Molnar wrote:
> 
> That might be a bit too granular: if we hit such an uninitialized block of 
> memory we'd have to process 262,144 pages - potentially from an IRQ 
> handler that does GFP_ATOMIC... or other latency critical code.
> 
> With 2MB we'd have to on-demand initialize 512 pages, which shouldn't show 
> up during normal use.
> 

No... you split the gigabyte page into 511 2M pages and 512 4K pages.

	-hpa

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Ingo Molnar]

* H. Peter Anvin <hpa@zytor.com> wrote:

> On 06/24/2013 12:39 PM, Ingo Molnar wrote:
> > 
> > That might be a bit too granular: if we hit such an uninitialized block of 
> > memory we'd have to process 262,144 pages - potentially from an IRQ 
> > handler that does GFP_ATOMIC... or other latency critical code.
> > 
> > With 2MB we'd have to on-demand initialize 512 pages, which shouldn't show 
> > up during normal use.
> > 
> 
> No... you split the gigabyte page into 511 2M pages and 512 4K pages.

Oh, double deferred initialization. That should solve the 32 exabyte 
problem as well I guess ;-)

Thanks,

	Ingo

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Nathan Zimmer]

On Sun, Jun 23, 2013 at 11:28:40AM +0200, Ingo Molnar wrote:
> 
> That's 4.5 GB/sec initialization speed - that feels a bit slow and the 
> boot time effect should be felt on smaller 'a couple of gigabytes' desktop 
> boxes as well. Do we know exactly where the 2 hours of boot time on a 32 
> TB system is spent?
> 
There are other several spots that could be improved on a large system but 
memory initialization is by far the biggest.

> While you cannot profile the boot process (yet), you could try your 
> delayed patch and run a "perf record -g" call-graph profiling of the 
> late-time initialization routines. What does 'perf report' show?
> 

I have some data from earlier runs.
memmap_init_zone was the function that was the biggest hitter by far.
Parts of it could certianly are low hanging fruit, set_pageblock_migratetype
for example.
However it seems for a larger system SetPageReserved will be the largest
consumer of cycles.  On a 1TB system I just booted it was around 50% of time
spent in memmap_init_zone.


perf seems to struggle with 512 cpus, but I did get some data.
It seems to indicate similar data to what I found in earlier experiments.
Lots of time in memmap_init_zone,
Some are waiting on locks, this guy seems to be representative of that.

-      0.14%    kworker/160:1  [kernel.kallsyms]        [k] mspin_lock                               ▒
   + mspin_lock                                                                                      ▒
   + __mutex_lock_slowpath                                                                           ▒
   - mutex_lock                                                                                      ▒
      - 99.69% online_pages             

> Delayed initialization makes sense I guess because 32 TB is a lot of 
> memory - I'm just wondering whether there's some low hanging fruits left 
> in the mem init code, that code is certainly not optimized for 
> performance.
>
> Plus with a struct page size of around 64 bytes (?) 32 TB of RAM has 512 
> GB of struct page arrays alone. Initializing those will take quite some 
> time as well - and I suspect they are allocated via zeroing them first. If 
> that memset() exists then getting rid of it might be a good move as well.
> 
> Yet another thing to consider would be to implement an initialization 
> speedup of 3 orders of magnitude: initialize on the large page (2MB) 
> grandularity and on-demand delay the initialization of the 4K granular 
> struct pages [but still allocating them] - which I suspect are a good 
> chunk of the overhead? That way we could initialize in 2MB steps and speed 
> up the 2 hours bootup of 32 TB of RAM to 14 seconds...
> 
> [ The cost would be one more branch in the buddy allocator, to detect
>   not-yet-initialized 2 MB chunks as we encounter them. Acceptable I 
>   think. ]
> 
> Thanks,
> 
> 	Ingo

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Ingo Molnar]

* Nathan Zimmer <nzimmer@sgi.com> wrote:

> On Sun, Jun 23, 2013 at 11:28:40AM +0200, Ingo Molnar wrote:
> > 
> > That's 4.5 GB/sec initialization speed - that feels a bit slow and the 
> > boot time effect should be felt on smaller 'a couple of gigabytes' 
> > desktop boxes as well. Do we know exactly where the 2 hours of boot 
> > time on a 32 TB system is spent?
> 
> There are other several spots that could be improved on a large system 
> but memory initialization is by far the biggest.

My feeling is that deferred/on-demand initialization triggered from the 
buddy allocator is the better long term solution.

That will also make it much easier to profile/test memory init 
performance: boot up a large system and run a simple testprogram that 
allocates a lot of RAM.

( It will also make people want to optimize the initialization sequence 
  better, as it will be part of any freshly booted system's memory 
  allocation overhead. )

Thanks,

	Ingo

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[H. Peter Anvin]

I have to say I really like this concept. It should have some very nice properties including perhaps making THP work better?

Ingo Molnar <mingo@kernel.org> wrote:

>
>* Nathan Zimmer <nzimmer@sgi.com> wrote:
>
>> On Sun, Jun 23, 2013 at 11:28:40AM +0200, Ingo Molnar wrote:
>> > 
>> > That's 4.5 GB/sec initialization speed - that feels a bit slow and
>the 
>> > boot time effect should be felt on smaller 'a couple of gigabytes' 
>> > desktop boxes as well. Do we know exactly where the 2 hours of boot
>
>> > time on a 32 TB system is spent?
>> 
>> There are other several spots that could be improved on a large
>system 
>> but memory initialization is by far the biggest.
>
>My feeling is that deferred/on-demand initialization triggered from the
>
>buddy allocator is the better long term solution.
>
>That will also make it much easier to profile/test memory init 
>performance: boot up a large system and run a simple testprogram that 
>allocates a lot of RAM.
>
>( It will also make people want to optimize the initialization sequence
>
>  better, as it will be part of any freshly booted system's memory 
>  allocation overhead. )
>
>Thanks,
>
>	Ingo

-- 
Sent from my mobile phone. Please excuse brevity and lack of formatting.

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Mike Travis]

Thanks.  I went through a few iterations until finalizing
on the current patches.  Originally the memory was inserted
using the memory probe interface, but at 2G per addition,
it was still very slow.  By bypassing the memory driver
interface, and going directly to add_memory and memory_online,
along with Nathan's changes to unlock the memory mutex on
a per node basis, it sped up dramatically.


On 6/25/2013 8:07 AM, H. Peter Anvin wrote:
> I have to say I really like this concept. It should have some very nice properties including perhaps making THP work better?
> 
> Ingo Molnar <mingo@kernel.org> wrote:
> 
>>
>> * Nathan Zimmer <nzimmer@sgi.com> wrote:
>>
>>> On Sun, Jun 23, 2013 at 11:28:40AM +0200, Ingo Molnar wrote:
>>>>
>>>> That's 4.5 GB/sec initialization speed - that feels a bit slow and
>> the 
>>>> boot time effect should be felt on smaller 'a couple of gigabytes' 
>>>> desktop boxes as well. Do we know exactly where the 2 hours of boot
>>
>>>> time on a 32 TB system is spent?
>>>
>>> There are other several spots that could be improved on a large
>> system 
>>> but memory initialization is by far the biggest.
>>
>> My feeling is that deferred/on-demand initialization triggered from the
>>
>> buddy allocator is the better long term solution.
>>
>> That will also make it much easier to profile/test memory init 
>> performance: boot up a large system and run a simple testprogram that 
>> allocates a lot of RAM.
>>
>> ( It will also make people want to optimize the initialization sequence
>>
>>  better, as it will be part of any freshly booted system's memory 
>>  allocation overhead. )
>>
>> Thanks,
>>
>> 	Ingo
> 

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Mike Travis]

On 6/25/2013 12:38 AM, Ingo Molnar wrote:
> 
> * Nathan Zimmer <nzimmer@sgi.com> wrote:
> 
>> On Sun, Jun 23, 2013 at 11:28:40AM +0200, Ingo Molnar wrote:
>>>
>>> That's 4.5 GB/sec initialization speed - that feels a bit slow and the 
>>> boot time effect should be felt on smaller 'a couple of gigabytes' 
>>> desktop boxes as well. Do we know exactly where the 2 hours of boot 
>>> time on a 32 TB system is spent?
>>
>> There are other several spots that could be improved on a large system 
>> but memory initialization is by far the biggest.
> 
> My feeling is that deferred/on-demand initialization triggered from the 
> buddy allocator is the better long term solution.

I haven't caught up with all of Nathan's changes yet (just
got back from vacation), but there was an option to either
start the memory insertion on boot, or trigger it later
using the /sys/.../memory interface.  There is also a monitor
program that calculates the memory insertion rate.  This was
extremely useful to determine how changes in the kernel
affected the rate.

> 
> That will also make it much easier to profile/test memory init 
> performance: boot up a large system and run a simple testprogram that 
> allocates a lot of RAM.
> 
> ( It will also make people want to optimize the initialization sequence 
>   better, as it will be part of any freshly booted system's memory 
>   allocation overhead. )
> 
> Thanks,
> 
> 	Ingo
> 

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[H. Peter Anvin]

On 06/25/2013 10:22 AM, Mike Travis wrote:
> 
> On 6/25/2013 12:38 AM, Ingo Molnar wrote:
>>
>> * Nathan Zimmer <nzimmer@sgi.com> wrote:
>>
>>> On Sun, Jun 23, 2013 at 11:28:40AM +0200, Ingo Molnar wrote:
>>>>
>>>> That's 4.5 GB/sec initialization speed - that feels a bit slow and the 
>>>> boot time effect should be felt on smaller 'a couple of gigabytes' 
>>>> desktop boxes as well. Do we know exactly where the 2 hours of boot 
>>>> time on a 32 TB system is spent?
>>>
>>> There are other several spots that could be improved on a large system 
>>> but memory initialization is by far the biggest.
>>
>> My feeling is that deferred/on-demand initialization triggered from the 
>> buddy allocator is the better long term solution.
> 
> I haven't caught up with all of Nathan's changes yet (just
> got back from vacation), but there was an option to either
> start the memory insertion on boot, or trigger it later
> using the /sys/.../memory interface.  There is also a monitor
> program that calculates the memory insertion rate.  This was
> extremely useful to determine how changes in the kernel
> affected the rate.
> 

Sorry, I *totally* did not follow that comment.  It seemed like a
complete non-sequitur?

	-hpa

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Mike Travis]

On 6/25/2013 11:43 AM, H. Peter Anvin wrote:
> On 06/25/2013 10:22 AM, Mike Travis wrote:
>>
>> On 6/25/2013 12:38 AM, Ingo Molnar wrote:
>>>
>>> * Nathan Zimmer <nzimmer@sgi.com> wrote:
>>>
>>>> On Sun, Jun 23, 2013 at 11:28:40AM +0200, Ingo Molnar wrote:
>>>>>
>>>>> That's 4.5 GB/sec initialization speed - that feels a bit slow and the 
>>>>> boot time effect should be felt on smaller 'a couple of gigabytes' 
>>>>> desktop boxes as well. Do we know exactly where the 2 hours of boot 
>>>>> time on a 32 TB system is spent?
>>>>
>>>> There are other several spots that could be improved on a large system 
>>>> but memory initialization is by far the biggest.
>>>
>>> My feeling is that deferred/on-demand initialization triggered from the 
>>> buddy allocator is the better long term solution.
>>
>> I haven't caught up with all of Nathan's changes yet (just
>> got back from vacation), but there was an option to either
>> start the memory insertion on boot, or trigger it later
>> using the /sys/.../memory interface.  There is also a monitor
>> program that calculates the memory insertion rate.  This was
>> extremely useful to determine how changes in the kernel
>> affected the rate.
>>
> 
> Sorry, I *totally* did not follow that comment.  It seemed like a
> complete non-sequitur?
> 
> 	-hpa

It was I who was not following the question.  I'm still reverting
back to "work mode".

[There is more code in a separate patch that Nate has not sent
yet that instructs the kernel to start adding memory as early
as possible, or not.  That way you can start the insertion process
later and monitor it's progress to determine how changes in the
kernel affect that process.  It is controlled by a separate
CONFIG option.]


> 
> 

[RFC] Transparent on-demand memory setup initialization embedded in the (GFP) buddy allocator

[Ingo Molnar]

(Changed the subject, to make it more apparent what we are talking about.)

* Mike Travis <travis@sgi.com> wrote:

> On 6/25/2013 11:43 AM, H. Peter Anvin wrote:
> > On 06/25/2013 10:22 AM, Mike Travis wrote:
> >>
> >> On 6/25/2013 12:38 AM, Ingo Molnar wrote:
> >>>
> >>> * Nathan Zimmer <nzimmer@sgi.com> wrote:
> >>>
> >>>> On Sun, Jun 23, 2013 at 11:28:40AM +0200, Ingo Molnar wrote:
> >>>>>
> >>>>> That's 4.5 GB/sec initialization speed - that feels a bit slow and the 
> >>>>> boot time effect should be felt on smaller 'a couple of gigabytes' 
> >>>>> desktop boxes as well. Do we know exactly where the 2 hours of boot 
> >>>>> time on a 32 TB system is spent?
> >>>>
> >>>> There are other several spots that could be improved on a large system 
> >>>> but memory initialization is by far the biggest.
> >>>
> >>> My feeling is that deferred/on-demand initialization triggered from the 
> >>> buddy allocator is the better long term solution.
> >>
> >> I haven't caught up with all of Nathan's changes yet (just
> >> got back from vacation), but there was an option to either
> >> start the memory insertion on boot, or trigger it later
> >> using the /sys/.../memory interface.  There is also a monitor
> >> program that calculates the memory insertion rate.  This was
> >> extremely useful to determine how changes in the kernel
> >> affected the rate.
> >>
> > 
> > Sorry, I *totally* did not follow that comment.  It seemed like a
> > complete non-sequitur?
> > 
> > 	-hpa
> 
> It was I who was not following the question.  I'm still reverting
> back to "work mode".
> 
> [There is more code in a separate patch that Nate has not sent
> yet that instructs the kernel to start adding memory as early
> as possible, or not.  That way you can start the insertion process
> later and monitor it's progress to determine how changes in the
> kernel affect that process.  It is controlled by a separate
> CONFIG option.]

So, just to repeat (and expand upon) the solution hpa and me suggests: 
it's not based on /sys, delayed initialization lists or any similar 
(essentially memory hot plug based) approach.

It's a transparent on-demand initialization scheme based on only 
initializing the very early memory setup in 1GB (2MB) steps (not in 4K 
steps like we do it today).

Any subsequent split-up initialization is done on-demand, in alloc_pages() 
et al, initilizing a batch of 512 (or 1024) struct page head's when an 
uninitialized portion is first encountered.

This leaves the principle logic of early init largely untouched, we still 
have the same amount of RAM during and after bootup, except that on 32 TB 
systems we don't spend ~2 hours initializing 8,589,934,592 page heads.

This scheme could be implemented by introducing a new PG_initialized flag, 
which is seen by an unlikely() branch in alloc_pages() and which triggers 
the on-demand initialization of pages.

[ It could probably be made zero-cost for the post-initialization state:
  we already check a bunch of rare PG_ flags, one more flag would not 
  introduce any new branch in the page allocation hot path. ]

It's a technically different solution from what was submitted in this 
thread.

Cons:

 - it works after bootup, via GFP. If done in a simple fashion it adds one 
   more branch to the GFP fastpath. [ If done a bit more cleverly it can 
   merge into an existing unlikely() branch and become essentially 
   zero-cost for the fastpath. ]

 - it adds an initialization non-determinism to GFP, to the tune of
   initializing ~512 page heads when RAM is utilized first.

 - initialization is done when memory is needed - not during or shortly 
   after bootup. This (slightly) increases first-use overhead. [I don't 
   think this factor is significant - and I think we'll quickly see 
   speedups to initialization, once the overhead becomes more easily 
   measurable.]

Pros:

 - it's transparent to the boot process. ('free' shows the same full
   amount of RAM all the time, there's no weird effects of RAM coming
   online asynchronously. You see all the RAM you have - etc.)

 - it helps the boot time of every single Linux system, not just large RAM
   ones. On a smallish, 4GB system memory init can take up precious
   hundreds of milliseconds, so this is a practical issue.

 - it spreads initialization overhead to later portions of the system's 
   life time: when there's typically more idle time and more paralellism
   available.

 - initialization overhead, because it's a natural part of first-time 
   memory allocation with this scheme, becomes more measurable (and thus 
   more prominently optimized) than any deferred lists processed in the 
   background.

 - as an added bonus it probably speeds up your usecase even more than the
   patches you are providing: on a 32 TB system the primary initialization
   would only have to enumerate memory, allocate page heads and buddy
   bitmaps, and initialize the 1GB granular page heads: there's only 32768
   of them.

So unless I overlooked some factor this scheme would be unconditional 
goodness for everyone.

Thanks,

	Ingo

Re: [RFC] Transparent on-demand memory setup initialization embedded in the (GFP) buddy allocator

[Andrew Morton]

On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar <mingo@kernel.org> wrote:

> except that on 32 TB 
> systems we don't spend ~2 hours initializing 8,589,934,592 page heads.

That's about a million a second which is crazy slow - even my prehistoric desktop
is 100x faster than that.

Where's all this time actually being spent?

Re: [RFC] Transparent on-demand memory setup initialization embedded in the (GFP) buddy allocator

[Ingo Molnar]

* Andrew Morton <akpm@linux-foundation.org> wrote:

> On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar <mingo@kernel.org> wrote:
> 
> > except that on 32 TB 
> > systems we don't spend ~2 hours initializing 8,589,934,592 page heads.
> 
> That's about a million a second which is crazy slow - even my 
> prehistoric desktop is 100x faster than that.
> 
> Where's all this time actually being spent?

See the earlier part of the thread - apparently it's spent initializing 
the page heads - remote NUMA node misses from a single boot CPU, going 
across a zillion cross-connects? I guess there's some other low hanging 
fruits as well - so making this easier to profile would be nice. The 
profile posted was not really usable.

Btw., NUMA locality would be another advantage of on-demand 
initialization: actual users of RAM tend to allocate node-local 
(especially on large clusters), so any overhead will be naturally lower.

Thanks,

	Ingo

Re: [RFC] Transparent on-demand memory setup initialization embedded in the (GFP) buddy allocator

[Nathan Zimmer]

On Wed, Jun 26, 2013 at 03:37:15PM +0200, Ingo Molnar wrote:
> 
> * Andrew Morton <akpm@linux-foundation.org> wrote:
> 
> > On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar <mingo@kernel.org> wrote:
> > 
> > > except that on 32 TB 
> > > systems we don't spend ~2 hours initializing 8,589,934,592 page heads.
> > 
> > That's about a million a second which is crazy slow - even my 
> > prehistoric desktop is 100x faster than that.
> > 
> > Where's all this time actually being spent?
> 
> See the earlier part of the thread - apparently it's spent initializing 
> the page heads - remote NUMA node misses from a single boot CPU, going 
> across a zillion cross-connects? I guess there's some other low hanging 
> fruits as well - so making this easier to profile would be nice. The 
> profile posted was not really usable.
> 
That is correct, from what I am seeing, using crude cycle counters, there is
far more time spent on the later nodes, i.e. memory near the boot node is 
initialized a lot faster then remote memory.

I think the other low hanging fruits are currently being drowned out by the
lack of locality.

Nate

> Btw., NUMA locality would be another advantage of on-demand 
> initialization: actual users of RAM tend to allocate node-local 
> (especially on large clusters), so any overhead will be naturally lower.
> 
> Thanks,
> 
> 	Ingo

Re: [RFC] Transparent on-demand memory setup initialization embedded in the (GFP) buddy allocator

[Mike Travis]

On 6/26/2013 6:37 AM, Ingo Molnar wrote:
> 
> * Andrew Morton <akpm@linux-foundation.org> wrote:
> 
>> On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar <mingo@kernel.org> wrote:
>>
>>> except that on 32 TB 
>>> systems we don't spend ~2 hours initializing 8,589,934,592 page heads.
>>
>> That's about a million a second which is crazy slow - even my 
>> prehistoric desktop is 100x faster than that.
>>
>> Where's all this time actually being spent?
> 
> See the earlier part of the thread - apparently it's spent initializing 
> the page heads - remote NUMA node misses from a single boot CPU, going 
> across a zillion cross-connects? I guess there's some other low hanging 
> fruits as well - so making this easier to profile would be nice. The 
> profile posted was not really usable.

This is one advantage of delayed memory init.  I can do it under
the profiler.  I will put everything together to accomplish this
and then send a perf report.

> 
> Btw., NUMA locality would be another advantage of on-demand 
> initialization: actual users of RAM tend to allocate node-local 
> (especially on large clusters), so any overhead will be naturally lower.
> 
> Thanks,
> 
> 	Ingo
> 

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Ingo Molnar]

* Nathan Zimmer <nzimmer@sgi.com> wrote:

> perf seems to struggle with 512 cpus, but I did get some data.

Btw., mind outlining in what way it struggles?

Thanks,

	Ingo

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Nathan Zimmer]

On Wed, Jun 26, 2013 at 02:14:30PM +0200, Ingo Molnar wrote:
> 
> * Nathan Zimmer <nzimmer@sgi.com> wrote:
> 
> > perf seems to struggle with 512 cpus, but I did get some data.
> 
> Btw., mind outlining in what way it struggles?
> 
> Thanks,
> 
> 	Ingo

System interactivity take a nose dive, including the ability to halt perf.
The commands do get through but they take a while.
Other processes tend to make signifigantly reduced progress.
Also it tends to slow down all running processes. 
My guess it is the NMIs overwhelming the system but I have not found a good way
to profile perf gone wild so it is only a guess.

The issue mitigate somewhat by restrictring the number of cpus perf is 
recording or by not using the -g option, but sometimes I do need data on the
whole system.


Nate

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Dave Hansen]

On 06/26/2013 07:49 AM, Nathan Zimmer wrote:
> My guess it is the NMIs overwhelming the system but I have not found a good way
> to profile perf gone wild so it is only a guess.

I've got an 80-core system and the symptoms sound similar to perf issues
I'm seeing.  Dropping the sample rate helped for me, and there's a patch
in -tip at the moment to do it automatically:

> http://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/commit/?id=14c63f17b1fde5a575a28e96547a22b451c71fb5

Looks like you've also found that debugging NMIs is fun. :)

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Nathan Zimmer]

On 06/26/2013 10:12 AM, Dave Hansen wrote:
> On 06/26/2013 07:49 AM, Nathan Zimmer wrote:
>> My guess it is the NMIs overwhelming the system but I have not found a good way
>> to profile perf gone wild so it is only a guess.
> I've got an 80-core system and the symptoms sound similar to perf issues
> I'm seeing.  Dropping the sample rate helped for me, and there's a patch
> in -tip at the moment to do it automatically:
>
>> http://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/commit/?id=14c63f17b1fde5a575a28e96547a22b451c71fb5
> Looks like you've also found that debugging NMIs is fun. :)
More fun then one person can handle.
I'll have to take this patch for a spin.

Thanks,
     Nate

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Ingo Molnar]

* Nathan Zimmer <nzimmer@sgi.com> wrote:

> On 06/26/2013 10:12 AM, Dave Hansen wrote:
> >On 06/26/2013 07:49 AM, Nathan Zimmer wrote:
> >>My guess it is the NMIs overwhelming the system but I have not found a good way
> >>to profile perf gone wild so it is only a guess.
> >I've got an 80-core system and the symptoms sound similar to perf issues
> >I'm seeing.  Dropping the sample rate helped for me, and there's a patch
> >in -tip at the moment to do it automatically:
> >
> >>http://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/commit/?id=14c63f17b1fde5a575a28e96547a22b451c71fb5
> >Looks like you've also found that debugging NMIs is fun. :)
> More fun then one person can handle.
> I'll have to take this patch for a spin.

Please try latest -tip:master, there's a series of patches from Dave that 
address this problem category.

Does it only happen with -g, or with regular perf record and/or with perf 
top as well?

Thanks,

	Ingo

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Nathan Zimmer]

On 06/26/2013 10:58 AM, Ingo Molnar wrote:
> * Nathan Zimmer <nzimmer@sgi.com> wrote:
>
>> On 06/26/2013 10:12 AM, Dave Hansen wrote:
>>> On 06/26/2013 07:49 AM, Nathan Zimmer wrote:
>>>> My guess it is the NMIs overwhelming the system but I have not found a good way
>>>> to profile perf gone wild so it is only a guess.
>>> I've got an 80-core system and the symptoms sound similar to perf issues
>>> I'm seeing.  Dropping the sample rate helped for me, and there's a patch
>>> in -tip at the moment to do it automatically:
>>>
>>>> http://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/commit/?id=14c63f17b1fde5a575a28e96547a22b451c71fb5
>>> Looks like you've also found that debugging NMIs is fun. :)
>> More fun then one person can handle.
>> I'll have to take this patch for a spin.
> Please try latest -tip:master, there's a series of patches from Dave that
> address this problem category.
>
> Does it only happen with -g, or with regular perf record and/or with perf
> top as well?
>
> Thanks,
>
> 	Ingo

It happens in all those cases, but I notice it most when doing a perf 
record -g -a
I'll try the tip series and see how it compares.

Thanks
     Nate

Re: [RFC 2/2] x86_64, mm: Reinsert the absent memory

[Mike Travis]

On 6/26/2013 5:14 AM, Ingo Molnar wrote:
> 
> * Nathan Zimmer <nzimmer@sgi.com> wrote:
> 
>> perf seems to struggle with 512 cpus, but I did get some data.
> 
> Btw., mind outlining in what way it struggles?
> 
> Thanks,
> 
> 	Ingo
> 

I submitted some patches to Jason Wessel that updated the community
support for KDB & NMI quite awhile ago that addressed the issues
with perf and friends on UV.  But I have not heard back from him in a
couple of months.  Is there a new maintainer for NMI/PERF/KDB etc.?

The primary problem is that the current UV NMI handler is in the
primary NMI notifier chain causing excessive reads of a register in
the UV hub.  When perf is running these approach the millions per
second, and the MMIO read is not only expensive but also distrupts
the primary HUB activities of directing NUMA traffic.  Thus the
system slows down considerably, and perf can even lose events.

We've even updated the UV BIOS to make this a faster path but it
needs the newer NMI handler to use it.  Perhaps I should resubmit
them directly to you?

Thanks,
Mike

Re: [RFC 0/2] Delay initializing of large sections of memory

[Greg KH]

On Fri, Jun 21, 2013 at 11:25:32AM -0500, Nathan Zimmer wrote:
> This rfc patch set delays initializing large sections of memory until we have
> started cpus.  This has the effect of reducing startup times on large memory
> systems.  On 16TB it can take over an hour to boot and most of that time
> is spent initializing memory.
> 
> We avoid that bottleneck by delaying initialization until after we have 
> started multiple cpus and can initialize in a multithreaded manner.
> This allows us to actually reduce boot time rather then just moving around
> the point of initialization.
> 
> Mike and I have worked on this set for a while, with him doing the most of the
> heavy lifting, and are eager for some feedback.

Why make this a config option at all, why not just always do this if the
memory size is larger than some specific number (like 8TB?)

Otherwise the distros will always enable this option, and having it be a
configuration choice doesn't make any sense.

thanks,

greg k-h

Re: [RFC 0/2] Delay initializing of large sections of memory

[H. Peter Anvin]

On 06/21/2013 09:51 AM, Greg KH wrote:
> On Fri, Jun 21, 2013 at 11:25:32AM -0500, Nathan Zimmer wrote:
>> This rfc patch set delays initializing large sections of memory until we have
>> started cpus.  This has the effect of reducing startup times on large memory
>> systems.  On 16TB it can take over an hour to boot and most of that time
>> is spent initializing memory.
>>
>> We avoid that bottleneck by delaying initialization until after we have 
>> started multiple cpus and can initialize in a multithreaded manner.
>> This allows us to actually reduce boot time rather then just moving around
>> the point of initialization.
>>
>> Mike and I have worked on this set for a while, with him doing the most of the
>> heavy lifting, and are eager for some feedback.
> 
> Why make this a config option at all, why not just always do this if the
> memory size is larger than some specific number (like 8TB?)
> 
> Otherwise the distros will always enable this option, and having it be a
> configuration choice doesn't make any sense.
> 

Since you made it a compile time option, it would be good to know how
much code it adds, but otherwise I agree with Greg here... this really
shouldn't need to be an option.  It *especially* shouldn't need to be a
hand-set runtime option (which looks quite complex, to boot.)

I suspect the cutoff for this should be a lot lower than 8 TB even, more
like 128 GB or so.  The only concern is to not set the cutoff so low
that we can end up running out of memory or with suboptimal NUMA
placement just because of this.

Also, in case it is not bloody obvious: whatever memory the kernel image
was loaded into MUST be considered "online", even if it is loaded way high.

	-hpa

Re: [RFC 0/2] Delay initializing of large sections of memory

[Nathan Zimmer]

On 06/21/2013 12:03 PM, H. Peter Anvin wrote:
> On 06/21/2013 09:51 AM, Greg KH wrote:
>> On Fri, Jun 21, 2013 at 11:25:32AM -0500, Nathan Zimmer wrote:
>>> This rfc patch set delays initializing large sections of memory until we have
>>> started cpus.  This has the effect of reducing startup times on large memory
>>> systems.  On 16TB it can take over an hour to boot and most of that time
>>> is spent initializing memory.
>>>
>>> We avoid that bottleneck by delaying initialization until after we have
>>> started multiple cpus and can initialize in a multithreaded manner.
>>> This allows us to actually reduce boot time rather then just moving around
>>> the point of initialization.
>>>
>>> Mike and I have worked on this set for a while, with him doing the most of the
>>> heavy lifting, and are eager for some feedback.
>> Why make this a config option at all, why not just always do this if the
>> memory size is larger than some specific number (like 8TB?)
>>
>> Otherwise the distros will always enable this option, and having it be a
>> configuration choice doesn't make any sense.
>>
> Since you made it a compile time option, it would be good to know how
> much code it adds, but otherwise I agree with Greg here... this really
> shouldn't need to be an option.  It *especially* shouldn't need to be a
> hand-set runtime option (which looks quite complex, to boot.)
The patchset as a whole is just over 400 lines so it doesn't add alot.
If I were to pull the .config option it would probably remove 30 lines.

The command line option is too complex but some of the data I haven't 
found a way
to get at runtime yet.


>
> I suspect the cutoff for this should be a lot lower than 8 TB even, more
> like 128 GB or so.  The only concern is to not set the cutoff so low
> that we can end up running out of memory or with suboptimal NUMA
> placement just because of this.
Even at lower amounts of ram there is an positive impact.I it knocks 
time off
boot even at as small as a 1TB of ram.

> Also, in case it is not bloody obvious: whatever memory the kernel image
> was loaded into MUST be considered "online", even if it is loaded way high.
>
> 	-hpa
>
>
>
>


Ok

Re: [RFC 0/2] Delay initializing of large sections of memory

[H. Peter Anvin]

On 06/21/2013 10:18 AM, Nathan Zimmer wrote:
>>>
>> Since you made it a compile time option, it would be good to know how
>> much code it adds, but otherwise I agree with Greg here... this really
>> shouldn't need to be an option.  It *especially* shouldn't need to be a
>> hand-set runtime option (which looks quite complex, to boot.)
> The patchset as a whole is just over 400 lines so it doesn't add alot.
> If I were to pull the .config option it would probably remove 30 lines.

I'm more concerned about bytes of code.

> The command line option is too complex but some of the data I haven't
> found a way to get at runtime yet.

I think that is probably key.

>> I suspect the cutoff for this should be a lot lower than 8 TB even, more
>> like 128 GB or so.  The only concern is to not set the cutoff so low
>> that we can end up running out of memory or with suboptimal NUMA
>> placement just because of this.
> Even at lower amounts of ram there is an positive impact.I it knocks
> time off
> boot even at as small as a 1TB of ram.

I am not surprised.

	-hpa

Re: [RFC 0/2] Delay initializing of large sections of memory

[Nathan Zimmer]

On 06/21/2013 12:28 PM, H. Peter Anvin wrote:
> On 06/21/2013 10:18 AM, Nathan Zimmer wrote:
>>> Since you made it a compile time option, it would be good to know how
>>> much code it adds, but otherwise I agree with Greg here... this really
>>> shouldn't need to be an option.  It *especially* shouldn't need to be a
>>> hand-set runtime option (which looks quite complex, to boot.)
>> The patchset as a whole is just over 400 lines so it doesn't add alot.
>> If I were to pull the .config option it would probably remove 30 lines.
> I'm more concerned about bytes of code.
Oh, The difference is just under 32k.
371843425 Jun 21 14:08 vmlinux.o /* DELAY_MEM_INIT is not set  */
371875600 Jun 21 14:36 vmlinux.o /* DELAY_MEM_INIT=y */

>
>> The command line option is too complex but some of the data I haven't
>> found a way to get at runtime yet.
> I think that is probably key.
>
>>> I suspect the cutoff for this should be a lot lower than 8 TB even, more
>>> like 128 GB or so.  The only concern is to not set the cutoff so low
>>> that we can end up running out of memory or with suboptimal NUMA
>>> placement just because of this.
>> Even at lower amounts of ram there is an positive impact.I it knocks
>> time off
>> boot even at as small as a 1TB of ram.
> I am not surprised.
>
> 	-hpa
>

Re: [RFC 0/2] Delay initializing of large sections of memory

[H. Peter Anvin]

Is this init code?  32K of unconditional runtime addition isn't completely trivial.

Nathan Zimmer <nzimmer@sgi.com> wrote:

>On 06/21/2013 12:28 PM, H. Peter Anvin wrote:
>> On 06/21/2013 10:18 AM, Nathan Zimmer wrote:
>>>> Since you made it a compile time option, it would be good to know
>how
>>>> much code it adds, but otherwise I agree with Greg here... this
>really
>>>> shouldn't need to be an option.  It *especially* shouldn't need to
>be a
>>>> hand-set runtime option (which looks quite complex, to boot.)
>>> The patchset as a whole is just over 400 lines so it doesn't add
>alot.
>>> If I were to pull the .config option it would probably remove 30
>lines.
>> I'm more concerned about bytes of code.
>Oh, The difference is just under 32k.
>371843425 Jun 21 14:08 vmlinux.o /* DELAY_MEM_INIT is not set  */
>371875600 Jun 21 14:36 vmlinux.o /* DELAY_MEM_INIT=y */
>
>>
>>> The command line option is too complex but some of the data I
>haven't
>>> found a way to get at runtime yet.
>> I think that is probably key.
>>
>>>> I suspect the cutoff for this should be a lot lower than 8 TB even,
>more
>>>> like 128 GB or so.  The only concern is to not set the cutoff so
>low
>>>> that we can end up running out of memory or with suboptimal NUMA
>>>> placement just because of this.
>>> Even at lower amounts of ram there is an positive impact.I it knocks
>>> time off
>>> boot even at as small as a 1TB of ram.
>> I am not surprised.
>>
>> 	-hpa
>>

-- 
Sent from my mobile phone. Please excuse brevity and lack of formatting.

Re: [RFC 0/2] Delay initializing of large sections of memory

[Nathan Zimmer]

On Fri, Jun 21, 2013 at 01:08:06PM -0700, H. Peter Anvin wrote:
> Is this init code?  32K of unconditional runtime addition isn't completely trivial.

Some of it is init code but not all.
I am guessing 24k of that is actually runtime.

> 
> Nathan Zimmer <nzimmer@sgi.com> wrote:
> 
> >On 06/21/2013 12:28 PM, H. Peter Anvin wrote:
> >> On 06/21/2013 10:18 AM, Nathan Zimmer wrote:
> >>>> Since you made it a compile time option, it would be good to know
> >how
> >>>> much code it adds, but otherwise I agree with Greg here... this
> >really
> >>>> shouldn't need to be an option.  It *especially* shouldn't need to
> >be a
> >>>> hand-set runtime option (which looks quite complex, to boot.)
> >>> The patchset as a whole is just over 400 lines so it doesn't add
> >alot.
> >>> If I were to pull the .config option it would probably remove 30
> >lines.
> >> I'm more concerned about bytes of code.
> >Oh, The difference is just under 32k.
> >371843425 Jun 21 14:08 vmlinux.o /* DELAY_MEM_INIT is not set  */
> >371875600 Jun 21 14:36 vmlinux.o /* DELAY_MEM_INIT=y */
> >
> >>
> >>> The command line option is too complex but some of the data I
> >haven't
> >>> found a way to get at runtime yet.
> >> I think that is probably key.
> >>
> >>>> I suspect the cutoff for this should be a lot lower than 8 TB even,
> >more
> >>>> like 128 GB or so.  The only concern is to not set the cutoff so
> >low
> >>>> that we can end up running out of memory or with suboptimal NUMA
> >>>> placement just because of this.
> >>> Even at lower amounts of ram there is an positive impact.I it knocks
> >>> time off
> >>> boot even at as small as a 1TB of ram.
> >> I am not surprised.
> >>
> >> 	-hpa
> >>
> 
> -- 
> Sent from my mobile phone. Please excuse brevity and lack of formatting.

Re: [RFC 0/2] Delay initializing of large sections of memory

[Mike Travis]

On 6/21/2013 1:08 PM, H. Peter Anvin wrote:
> Is this init code?  32K of unconditional runtime addition isn't completely trivial.

The delay functions that move memory to the absent list are __init
but the read back of the list and memory insertion are not.  BTW, this
option is only available with the memory hotplug option which depends
on the sparse memory option.  So any usage of these distro configs will
normally be on enterprise class machines.

> 
> Nathan Zimmer <nzimmer@sgi.com> wrote:
> 
>> On 06/21/2013 12:28 PM, H. Peter Anvin wrote:
>>> On 06/21/2013 10:18 AM, Nathan Zimmer wrote:
>>>>> Since you made it a compile time option, it would be good to know
>> how
>>>>> much code it adds, but otherwise I agree with Greg here... this
>> really
>>>>> shouldn't need to be an option.  It *especially* shouldn't need to
>> be a
>>>>> hand-set runtime option (which looks quite complex, to boot.)
>>>> The patchset as a whole is just over 400 lines so it doesn't add
>> alot.
>>>> If I were to pull the .config option it would probably remove 30
>> lines.
>>> I'm more concerned about bytes of code.
>> Oh, The difference is just under 32k.
>> 371843425 Jun 21 14:08 vmlinux.o /* DELAY_MEM_INIT is not set  */
>> 371875600 Jun 21 14:36 vmlinux.o /* DELAY_MEM_INIT=y */
>>
>>>
>>>> The command line option is too complex but some of the data I
>> haven't
>>>> found a way to get at runtime yet.
>>> I think that is probably key.
>>>
>>>>> I suspect the cutoff for this should be a lot lower than 8 TB even,
>> more
>>>>> like 128 GB or so.  The only concern is to not set the cutoff so
>> low
>>>>> that we can end up running out of memory or with suboptimal NUMA
>>>>> placement just because of this.
>>>> Even at lower amounts of ram there is an positive impact.I it knocks
>>>> time off
>>>> boot even at as small as a 1TB of ram.
>>> I am not surprised.
>>>
>>> 	-hpa
>>>
> 

Re: [RFC 0/2] Delay initializing of large sections of memory

[Mike Travis]

On 6/21/2013 10:18 AM, Nathan Zimmer wrote:
> On 06/21/2013 12:03 PM, H. Peter Anvin wrote:
>> On 06/21/2013 09:51 AM, Greg KH wrote:
>>> On Fri, Jun 21, 2013 at 11:25:32AM -0500, Nathan Zimmer wrote:
>>>> This rfc patch set delays initializing large sections of memory
>>>> until we have
>>>> started cpus.  This has the effect of reducing startup times on
>>>> large memory
>>>> systems.  On 16TB it can take over an hour to boot and most of that
>>>> time
>>>> is spent initializing memory.

On 32TB we went from 2:25 to around 20 minutes.

>>>> We avoid that bottleneck by delaying initialization until after we have
>>>> started multiple cpus and can initialize in a multithreaded manner.
>>>> This allows us to actually reduce boot time rather then just moving
>>>> around
>>>> the point of initialization.
>>>>
>>>> Mike and I have worked on this set for a while, with him doing the
>>>> most of the
>>>> heavy lifting, and are eager for some feedback.
>>> Why make this a config option at all, why not just always do this if the
>>> memory size is larger than some specific number (like 8TB?)
>>>
>>> Otherwise the distros will always enable this option, and having it be a
>>> configuration choice doesn't make any sense.
>>>
>> Since you made it a compile time option, it would be good to know how
>> much code it adds, but otherwise I agree with Greg here... this really
>> shouldn't need to be an option.  It *especially* shouldn't need to be a
>> hand-set runtime option (which looks quite complex, to boot.)
> The patchset as a whole is just over 400 lines so it doesn't add alot.
> If I were to pull the .config option it would probably remove 30 lines.
> 
> The command line option is too complex but some of the data I
> haven't found a way to get at runtime yet.

Specifically, the physical address space of each node and whether the
block size is 128M or 2G is needed.  The other params are really there as
a fallback as we have not yet verified on the largest possible machine.
The parameter is intended to be set by a configurator.  There are far
too many kernel params to leave them to chance to be set correctly.
On UV we use a utility called 'uvconfig'.

What we could do is default the values unless specifically set?
Perhaps only set the node address space?  Delaying the memory
insertion is mostly a debug aid for debugging the insertion functions.

>>
>> I suspect the cutoff for this should be a lot lower than 8 TB even, more
>> like 128 GB or so.  The only concern is to not set the cutoff so low
>> that we can end up running out of memory or with suboptimal NUMA
>> placement just because of this.

Exactly.

We test regularly on a machine that has ~4TB and the speedup is
negligible.  The problem seems to occur as the count of memory blocks
is increased over some limit.  I think going much lower might start
getting in the way of other things, like constructing transparent
huge pages, etc.

Also notice that Node 0 and the last Node already have all their memory.
There are just too many other types in the memmap that it wasn't worth
the hassle.  So unless the system has at least 6 or 8 nodes you're not
gaining much.

> Even at lower amounts of ram there is an positive impact.I it knocks
> time off
> boot even at as small as a 1TB of ram.
> 
>> Also, in case it is not bloody obvious: whatever memory the kernel image
>> was loaded into MUST be considered "online", even if it is loaded way
>> high.

Good point, we should add a check since we have that info at boot time.
Other checks might be if this is a kdump kernel, or even perhaps a KVM
kernel boot (though giving it 16TB is pretty wild.)

Thanks!
>>
>>     -hpa

Re: [RFC 0/2] Delay initializing of large sections of memory

[Yinghai Lu]

On Fri, Jun 21, 2013 at 10:03 AM, H. Peter Anvin <hpa@zytor.com> wrote:
> On 06/21/2013 09:51 AM, Greg KH wrote:
>
> I suspect the cutoff for this should be a lot lower than 8 TB even, more
> like 128 GB or so.  The only concern is to not set the cutoff so low
> that we can end up running out of memory or with suboptimal NUMA
> placement just because of this.

I would suggest another way:
only boot the system with boot node (include cpu, ram and pci root buses).
then after boot, could add other nodes.

or something like pxm_mask or node_mask to select boot nodes.

Thanks

Yinghai

Re: [RFC 0/2] Delay initializing of large sections of memory

[Greg KH]

On Fri, Jun 21, 2013 at 11:44:22AM -0700, Yinghai Lu wrote:
> On Fri, Jun 21, 2013 at 10:03 AM, H. Peter Anvin <hpa@zytor.com> wrote:
> > On 06/21/2013 09:51 AM, Greg KH wrote:
> >
> > I suspect the cutoff for this should be a lot lower than 8 TB even, more
> > like 128 GB or so.  The only concern is to not set the cutoff so low
> > that we can end up running out of memory or with suboptimal NUMA
> > placement just because of this.
> 
> I would suggest another way:
> only boot the system with boot node (include cpu, ram and pci root buses).
> then after boot, could add other nodes.

What exactly do you mean by "after boot"?  Often, the boot process of
userspace needs those additional cpus and ram in order to initialize
everything (like the pci devices) properly.

thanks,

greg k-h

Re: [RFC 0/2] Delay initializing of large sections of memory

[Yinghai Lu]

On Fri, Jun 21, 2013 at 11:50 AM, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Fri, Jun 21, 2013 at 11:44:22AM -0700, Yinghai Lu wrote:
>> On Fri, Jun 21, 2013 at 10:03 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>> > On 06/21/2013 09:51 AM, Greg KH wrote:
>> >
>> > I suspect the cutoff for this should be a lot lower than 8 TB even, more
>> > like 128 GB or so.  The only concern is to not set the cutoff so low
>> > that we can end up running out of memory or with suboptimal NUMA
>> > placement just because of this.
>>
>> I would suggest another way:
>> only boot the system with boot node (include cpu, ram and pci root buses).
>> then after boot, could add other nodes.
>
> What exactly do you mean by "after boot"?  Often, the boot process of
> userspace needs those additional cpus and ram in order to initialize
> everything (like the pci devices) properly.

I mean for Intel cpu have cpu and memory controller and IIO.
every IIO is one peer pci root bus.
So scan root bus that are not with boot node later.

in this way we can keep all numa etc on the place when online ram, cpu, pci...

For example if we have 32 sockets system, most time for boot is with *BIOS*
instead of OS. In those kind of system boot is like this way:
only first two sockets get booted from bios to OS.
later use hot add every other two sockets.

that will also make BIOS simpler, and it need to support hot-add for
services purpose anyway.

Yinghai

Re: [RFC 0/2] Delay initializing of large sections of memory

[Nathan Zimmer]

On 06/21/2013 02:10 PM, Yinghai Lu wrote:
> On Fri, Jun 21, 2013 at 11:50 AM, Greg KH <gregkh@linuxfoundation.org> wrote:
>> On Fri, Jun 21, 2013 at 11:44:22AM -0700, Yinghai Lu wrote:
>>> On Fri, Jun 21, 2013 at 10:03 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>>>> On 06/21/2013 09:51 AM, Greg KH wrote:
>>>>
>>>> I suspect the cutoff for this should be a lot lower than 8 TB even, more
>>>> like 128 GB or so.  The only concern is to not set the cutoff so low
>>>> that we can end up running out of memory or with suboptimal NUMA
>>>> placement just because of this.
>>> I would suggest another way:
>>> only boot the system with boot node (include cpu, ram and pci root buses).
>>> then after boot, could add other nodes.
>> What exactly do you mean by "after boot"?  Often, the boot process of
>> userspace needs those additional cpus and ram in order to initialize
>> everything (like the pci devices) properly.
> I mean for Intel cpu have cpu and memory controller and IIO.
> every IIO is one peer pci root bus.
> So scan root bus that are not with boot node later.
>
> in this way we can keep all numa etc on the place when online ram, cpu, pci...
>
> For example if we have 32 sockets system, most time for boot is with *BIOS*
> instead of OS. In those kind of system boot is like this way:
> only first two sockets get booted from bios to OS.
> later use hot add every other two sockets.
>
> that will also make BIOS simpler, and it need to support hot-add for
> services purpose anyway.
>
> Yinghai

Yes the hot add path was one option we looked at and it did shorten boot 
times but the goal I had here is to get from power on to having the full 
machine available as quick as possible. Several clients need significant 
portions of ram for their key workloads.  So that guided my thoughts on 
this patch.

Re: [RFC 0/2] Delay initializing of large sections of memory

[Yinghai Lu]

On Fri, Jun 21, 2013 at 12:19 PM, Nathan Zimmer <nzimmer@sgi.com> wrote:
> On 06/21/2013 02:10 PM, Yinghai Lu wrote:
>> in this way we can keep all numa etc on the place when online ram, cpu,
>> pci...
>>
>> For example if we have 32 sockets system, most time for boot is with
>> *BIOS*
>> instead of OS. In those kind of system boot is like this way:
>> only first two sockets get booted from bios to OS.
>> later use hot add every other two sockets.
>>
>> that will also make BIOS simpler, and it need to support hot-add for
>> services purpose anyway.
>>
> Yes the hot add path was one option we looked at and it did shorten boot
> times but the goal I had here is to get from power on to having the full
> machine available as quick as possible. Several clients need significant
> portions of ram for their key workloads.  So that guided my thoughts on this
> patch.

you mean boot small number sockets + hot add still take more time?
hot add ram still in serialize mode?

Yinghai

Re: [RFC 0/2] Delay initializing of large sections of memory

[Nathan Zimmer]

On Fri, Jun 21, 2013 at 01:28:11PM -0700, Yinghai Lu wrote:
> On Fri, Jun 21, 2013 at 12:19 PM, Nathan Zimmer <nzimmer@sgi.com> wrote:
> > On 06/21/2013 02:10 PM, Yinghai Lu wrote:
> >> in this way we can keep all numa etc on the place when online ram, cpu,
> >> pci...
> >>
> >> For example if we have 32 sockets system, most time for boot is with
> >> *BIOS*
> >> instead of OS. In those kind of system boot is like this way:
> >> only first two sockets get booted from bios to OS.
> >> later use hot add every other two sockets.
> >>
> >> that will also make BIOS simpler, and it need to support hot-add for
> >> services purpose anyway.
> >>
> > Yes the hot add path was one option we looked at and it did shorten boot
> > times but the goal I had here is to get from power on to having the full
> > machine available as quick as possible. Several clients need significant
> > portions of ram for their key workloads.  So that guided my thoughts on this
> > patch.
> 
> you mean boot small number sockets + hot add still take more time?
> hot add ram still in serialize mode?
> 
> Yinghai

Yes the hot add path is still fairly serial.
lock_memory_hotplug() is still a global lock, which effectivly serializes
online pages.  Since the hotplug lock could done per node but for that to be 
effective something would need to be done with lock_system_sleep() since is
called by lock_memory_hotplug().

Nate

Re: [RFC 0/2] Delay initializing of large sections of memory

[Mike Travis]

On 6/21/2013 11:50 AM, Greg KH wrote:
> On Fri, Jun 21, 2013 at 11:44:22AM -0700, Yinghai Lu wrote:
>> On Fri, Jun 21, 2013 at 10:03 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>>> On 06/21/2013 09:51 AM, Greg KH wrote:
>>>
>>> I suspect the cutoff for this should be a lot lower than 8 TB even, more
>>> like 128 GB or so.  The only concern is to not set the cutoff so low
>>> that we can end up running out of memory or with suboptimal NUMA
>>> placement just because of this.
>>
>> I would suggest another way:
>> only boot the system with boot node (include cpu, ram and pci root buses).
>> then after boot, could add other nodes.
> 
> What exactly do you mean by "after boot"?  Often, the boot process of
> userspace needs those additional cpus and ram in order to initialize
> everything (like the pci devices) properly.

Exactly.  That's why I left both low and high memory on each node.

> 
> thanks,
> 
> greg k-h
> 

Re: [RFC 0/2] Delay initializing of large sections of memory

[Yinghai Lu]

On Fri, Jun 21, 2013 at 2:30 PM, Mike Travis <travis@sgi.com> wrote:
>
>
> On 6/21/2013 11:50 AM, Greg KH wrote:
>> On Fri, Jun 21, 2013 at 11:44:22AM -0700, Yinghai Lu wrote:
>>> On Fri, Jun 21, 2013 at 10:03 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>>>> On 06/21/2013 09:51 AM, Greg KH wrote:
>>>>
>>>> I suspect the cutoff for this should be a lot lower than 8 TB even, more
>>>> like 128 GB or so.  The only concern is to not set the cutoff so low
>>>> that we can end up running out of memory or with suboptimal NUMA
>>>> placement just because of this.
>>>
>>> I would suggest another way:
>>> only boot the system with boot node (include cpu, ram and pci root buses).
>>> then after boot, could add other nodes.
>>
>> What exactly do you mean by "after boot"?  Often, the boot process of
>> userspace needs those additional cpus and ram in order to initialize
>> everything (like the pci devices) properly.
>
> Exactly.  That's why I left both low and high memory on each node.

looks like you assume every node have same ram, and before booting you
you need to know memory layout to append the boot command line.

We have patchset that moving srat table parse early.
git://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git
for-x86-mm
https://git.kernel.org/cgit/linux/kernel/git/yinghai/linux-yinghai.git/log/?h=for-x86-mm

on top that, we could make your patch pass more simple command like
1/2^n of every node, and only need to pass n instead.

Thanks

Yinghai

Re: [RFC 0/2] Delay initializing of large sections of memory

[Mike Travis]

On 6/21/2013 5:23 PM, Yinghai Lu wrote:
> On Fri, Jun 21, 2013 at 2:30 PM, Mike Travis <travis@sgi.com> wrote:
>>
>>
>> On 6/21/2013 11:50 AM, Greg KH wrote:
>>> On Fri, Jun 21, 2013 at 11:44:22AM -0700, Yinghai Lu wrote:
>>>> On Fri, Jun 21, 2013 at 10:03 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>>>>> On 06/21/2013 09:51 AM, Greg KH wrote:
>>>>>
>>>>> I suspect the cutoff for this should be a lot lower than 8 TB even, more
>>>>> like 128 GB or so.  The only concern is to not set the cutoff so low
>>>>> that we can end up running out of memory or with suboptimal NUMA
>>>>> placement just because of this.
>>>>
>>>> I would suggest another way:
>>>> only boot the system with boot node (include cpu, ram and pci root buses).
>>>> then after boot, could add other nodes.
>>>
>>> What exactly do you mean by "after boot"?  Often, the boot process of
>>> userspace needs those additional cpus and ram in order to initialize
>>> everything (like the pci devices) properly.
>>
>> Exactly.  That's why I left both low and high memory on each node.
> 
> looks like you assume every node have same ram, and before booting you
> you need to know memory layout to append the boot command line.
> 
> We have patchset that moving srat table parse early.
> git://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git
> for-x86-mm
> https://git.kernel.org/cgit/linux/kernel/git/yinghai/linux-yinghai.git/log/?h=for-x86-mm
> 
> on top that, we could make your patch pass more simple command like
> 1/2^n of every node, and only need to pass n instead.

The two params that I couldn't figure out how to provide except via kernel
param option was the memory block size (128M or 2G) and the physical
address space per node.  The other 3 params can be automatically
setup by a script when the total system size is known.  As soon as we
verify on the 32TB system and surmise what will be needed for 64TB,
then those 3 params can probably disappear.




> 
> Thanks
> 
> Yinghai
> 

Re: [RFC 0/2] Delay initializing of large sections of memory

[H. Peter Anvin]

On 06/25/2013 10:35 AM, Mike Travis wrote:
> 
> The two params that I couldn't figure out how to provide except via kernel
> param option was the memory block size (128M or 2G) and the physical
> address space per node.  The other 3 params can be automatically
> setup by a script when the total system size is known.  As soon as we
> verify on the 32TB system and surmise what will be needed for 64TB,
> then those 3 params can probably disappear.
> 

"Setup by script" is a no-go.  You *have* the total system size already,
it is in the e820 tables (anything which isn't in e820 is hotplug, that
automagically gets deferred.)

However, please consider Ingo's counterproposal of doing this via the
buddy allocator, i.e. hugepages being broken on demand.  That is a
*very* powerful model, although would require more infrastructure.

	-hpa

Re: [RFC 0/2] Delay initializing of large sections of memory

[Mike Travis]

On 6/25/2013 11:17 AM, H. Peter Anvin wrote:
> On 06/25/2013 10:35 AM, Mike Travis wrote:
>>
>> The two params that I couldn't figure out how to provide except via kernel
>> param option was the memory block size (128M or 2G) and the physical
>> address space per node.  The other 3 params can be automatically
>> setup by a script when the total system size is known.  As soon as we
>> verify on the 32TB system and surmise what will be needed for 64TB,
>> then those 3 params can probably disappear.
>>
> 
> "Setup by script" is a no-go.  You *have* the total system size already,
> it is in the e820 tables (anything which isn't in e820 is hotplug, that
> automagically gets deferred.)

Okay, I'll figure something out.  If Yinghai's SRAT patch can help
with the node address space, then I might be able to determine if
the system is a UV which is the only system I see that uses 2G
memory blocks.  (Or make get_memory_block_size() a global.)

Then a simple param to start the insertion early or defer it until
the system is fully up is still useful, and that's easy to understand.

[I think we still want to keep the actual process of moving memory
to the absent list an option, yes?  If for no other reason except
to rule out this code when a problem crops up.  Or at least have a
way to disable the process if it's CONFIG'd in.]

> 
> However, please consider Ingo's counterproposal of doing this via the
> buddy allocator, i.e. hugepages being broken on demand.  That is a
> *very* powerful model, although would require more infrastructure.

We will certainly continue to make improvements as larger system sizes
become more commonplace (and customers continue to complain :).  But
we are cutting it close to including this into the nextgen distro
releases, so it would have to be a follow on project.  (I've been
working on this patch since last November.)

Thanks,
Mike

> 
> 	-hpa
> 
> 

Re: [RFC 0/2] Delay initializing of large sections of memory

[Yinghai Lu]

On Tue, Jun 25, 2013 at 11:17 AM, H. Peter Anvin <hpa@zytor.com> wrote:
> On 06/25/2013 10:35 AM, Mike Travis wrote:

> However, please consider Ingo's counterproposal of doing this via the
> buddy allocator, i.e. hugepages being broken on demand.  That is a
> *very* powerful model, although would require more infrastructure.

Can you or Ingo elaborate more about the buddy allocator proposal?

Thanks

Re: [RFC 0/2] Delay initializing of large sections of memory

[H. Peter Anvin]

On 06/25/2013 11:40 AM, Yinghai Lu wrote:
> On Tue, Jun 25, 2013 at 11:17 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>> On 06/25/2013 10:35 AM, Mike Travis wrote:
> 
>> However, please consider Ingo's counterproposal of doing this via the
>> buddy allocator, i.e. hugepages being broken on demand.  That is a
>> *very* powerful model, although would require more infrastructure.
> 
> Can you or Ingo elaborate more about the buddy allocator proposal?
> 

Start by initializing 1G hyperpages only, but mark them so that the
allocator knows that if it needs to break them apart it has to
initialize the page structures for the 2M subpages.

Same thing with 2M -> 4K.

	-hpa

Re: [RFC 0/2] Delay initializing of large sections of memory

[Mike Travis]

On 6/25/2013 11:44 AM, H. Peter Anvin wrote:
> On 06/25/2013 11:40 AM, Yinghai Lu wrote:
>> On Tue, Jun 25, 2013 at 11:17 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>>> On 06/25/2013 10:35 AM, Mike Travis wrote:
>>
>>> However, please consider Ingo's counterproposal of doing this via the
>>> buddy allocator, i.e. hugepages being broken on demand.  That is a
>>> *very* powerful model, although would require more infrastructure.
>>
>> Can you or Ingo elaborate more about the buddy allocator proposal?
>>
> 
> Start by initializing 1G hyperpages only, but mark them so that the
> allocator knows that if it needs to break them apart it has to
> initialize the page structures for the 2M subpages.
> 
> Same thing with 2M -> 4K.
> 
> 	-hpa
> 
> 

It is worth experimenting with but the big question would be,
if it still avoids the very expensive "memmap_init_zone" and
it's sub-functions using huge expanses of memory.  I'll do some
experimenting as soon as I can.  Our 32TB system is being
brought back to 16TB (we found a number of problems as we
get closer and closer to the 64TB limit), but that's still
a significant size.

Re: [RFC 0/2] Delay initializing of large sections of memory

[Yinghai Lu]

> It is worth experimenting with but the big question would be,
> if it still avoids the very expensive "memmap_init_zone" and
> it's sub-functions using huge expanses of memory.  I'll do some
> experimenting as soon as I can.  Our 32TB system is being
> brought back to 16TB (we found a number of problems as we
> get closer and closer to the 64TB limit), but that's still
> a significant size.

According to Intel SDM, CPU could support 52bits physical addressing.

So how linux kernel will handle it? as we only have 48bits virtual addressing.

Yinghai

Re: [RFC 0/2] Delay initializing of large sections of memory

[H. Peter Anvin]

On 06/25/2013 12:03 PM, Yinghai Lu wrote:
>> It is worth experimenting with but the big question would be,
>> if it still avoids the very expensive "memmap_init_zone" and
>> it's sub-functions using huge expanses of memory.  I'll do some
>> experimenting as soon as I can.  Our 32TB system is being
>> brought back to 16TB (we found a number of problems as we
>> get closer and closer to the 64TB limit), but that's still
>> a significant size.
> 
> According to Intel SDM, CPU could support 52bits physical addressing.
> 
> So how linux kernel will handle it? as we only have 48bits virtual addressing.
> 

The Linux kernel will not support more than V-2 bits of physical address
space for any number V of virtual address bits.

	-hpa

Re: [RFC 0/2] Delay initializing of large sections of memory

[Yinghai Lu]

On Tue, Jun 25, 2013 at 12:09 PM, H. Peter Anvin <hpa@zytor.com> wrote:
>> According to Intel SDM, CPU could support 52bits physical addressing.
>>
>> So how linux kernel will handle it? as we only have 48bits virtual addressing.
>>
>
> The Linux kernel will not support more than V-2 bits of physical address
> space for any number V of virtual address bits.

ok, then will need cpu support 57bits virtual addressing, and level 5
page table?

Yinghai

Re: [RFC 0/2] Delay initializing of large sections of memory

[Yinghai Lu]

On Tue, Jun 25, 2013 at 11:58 AM, Mike Travis <travis@sgi.com> wrote:
> experimenting as soon as I can.  Our 32TB system is being
> brought back to 16TB (we found a number of problems as we
> get closer and closer to the 64TB limit), but that's still
> a significant size.

Hi, Mike,

Can you post e820 memory map on system that have 32TiB or more?

Is there one range size more than 16TiB? like [16TiB, 32TiB)...

Thanks

Yinghai

Re: [RFC 0/2] Delay initializing of large sections of memory

[Robin Holt]

On Wed, Jun 26, 2013 at 11:37:43PM -0700, Yinghai Lu wrote:
> On Tue, Jun 25, 2013 at 11:58 AM, Mike Travis <travis@sgi.com> wrote:
> > experimenting as soon as I can.  Our 32TB system is being
> > brought back to 16TB (we found a number of problems as we
> > get closer and closer to the 64TB limit), but that's still
> > a significant size.
> 
> Hi, Mike,
> 
> Can you post e820 memory map on system that have 32TiB or more?
> 
> Is there one range size more than 16TiB? like [16TiB, 32TiB)...

[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000007f000 (usable)
[    0.000000]  BIOS-e820: 000000000007f000 - 0000000000080000 (reserved)
[    0.000000]  BIOS-e820: 0000000000080000 - 00000000000a0000 (usable)
[    0.000000]  BIOS-e820: 0000000000100000 - 000000007abb1000 (usable)
[    0.000000]  BIOS-e820: 000000007abb1000 - 000000007abb2000 (reserved)
[    0.000000]  BIOS-e820: 000000007abb2000 - 000000007abc5000 (usable)
[    0.000000]  BIOS-e820: 000000007abc5000 - 000000007ae05000 (reserved)
[    0.000000]  BIOS-e820: 000000007ae05000 - 000000007bb13000 (usable)
[    0.000000]  BIOS-e820: 000000007bb13000 - 000000007bc13000 (reserved)
[    0.000000]  BIOS-e820: 000000007bc13000 - 000000007bd13000 (unusable)
[    0.000000]  BIOS-e820: 000000007bd13000 - 000000007be13000 (reserved)
[    0.000000]  BIOS-e820: 000000007be13000 - 000000007d1f3000 (ACPI NVS)
[    0.000000]  BIOS-e820: 000000007d1f3000 - 000000007e000000 (ACPI data)
[    0.000000]  BIOS-e820: 000000007e000000 - 000000007e2ee000 (usable)
[    0.000000]  BIOS-e820: 000000007e2ee000 - 000000007ef3f000 (ACPI data)
[    0.000000]  BIOS-e820: 000000007ef3f000 - 000000007f000000 (usable)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000001f80000000 (usable)
[    0.000000]  BIOS-e820: 0000004000000000 - 0000005eff000000 (usable)
[    0.000000]  BIOS-e820: 0000008000000000 - 0000009eff000000 (usable)
[    0.000000]  BIOS-e820: 000000c000000000 - 000000deff000000 (usable)
[    0.000000]  BIOS-e820: 0000010000000000 - 0000011eff000000 (usable)
[    0.000000]  BIOS-e820: 0000014000000000 - 0000015eff000000 (usable)
[    0.000000]  BIOS-e820: 0000018000000000 - 0000019eff000000 (usable)
[    0.000000]  BIOS-e820: 000001c000000000 - 000001deff000000 (usable)
[    0.000000]  BIOS-e820: 0000020000000000 - 0000021eff000000 (usable)
[    0.000000]  BIOS-e820: 0000024000000000 - 0000025eff000000 (usable)
[    0.000000]  BIOS-e820: 0000028000000000 - 0000029eff000000 (usable)
[    0.000000]  BIOS-e820: 000002c000000000 - 000002deff000000 (usable)
[    0.000000]  BIOS-e820: 0000030000000000 - 0000031eff000000 (usable)
[    0.000000]  BIOS-e820: 0000034000000000 - 0000035eff000000 (usable)
[    0.000000]  BIOS-e820: 0000038000000000 - 0000039eff000000 (usable)
[    0.000000]  BIOS-e820: 000003c000000000 - 000003deff000000 (usable)
[    0.000000]  BIOS-e820: 0000040000000000 - 0000041eff000000 (usable)
[    0.000000]  BIOS-e820: 0000044000000000 - 0000045eff000000 (usable)
[    0.000000]  BIOS-e820: 0000048000000000 - 0000049eff000000 (usable)
[    0.000000]  BIOS-e820: 000004c000000000 - 000004deff000000 (usable)
[    0.000000]  BIOS-e820: 0000050000000000 - 0000051eff000000 (usable)
[    0.000000]  BIOS-e820: 0000054000000000 - 0000055eff000000 (usable)
[    0.000000]  BIOS-e820: 0000058000000000 - 0000059eff000000 (usable)
[    0.000000]  BIOS-e820: 000005c000000000 - 000005deff000000 (usable)
[    0.000000]  BIOS-e820: 0000060000000000 - 0000061eff000000 (usable)
[    0.000000]  BIOS-e820: 0000064000000000 - 0000065eff000000 (usable)
[    0.000000]  BIOS-e820: 0000068000000000 - 0000069eff000000 (usable)
[    0.000000]  BIOS-e820: 000006c000000000 - 000006deff000000 (usable)
[    0.000000]  BIOS-e820: 0000070000000000 - 0000071eff000000 (usable)
[    0.000000]  BIOS-e820: 0000074000000000 - 0000075eff000000 (usable)
[    0.000000]  BIOS-e820: 0000078000000000 - 0000079eff000000 (usable)
[    0.000000]  BIOS-e820: 000007c000000000 - 000007deff000000 (usable)
[    0.000000]  BIOS-e820: 0000080000000000 - 0000081eff000000 (usable)
[    0.000000]  BIOS-e820: 0000084000000000 - 0000085eff000000 (usable)
[    0.000000]  BIOS-e820: 0000088000000000 - 0000089eff000000 (usable)
[    0.000000]  BIOS-e820: 000008c000000000 - 000008deff000000 (usable)
[    0.000000]  BIOS-e820: 0000090000000000 - 0000091eff000000 (usable)
[    0.000000]  BIOS-e820: 0000094000000000 - 0000095eff000000 (usable)
[    0.000000]  BIOS-e820: 0000098000000000 - 0000099eff000000 (usable)
[    0.000000]  BIOS-e820: 000009c000000000 - 000009deff000000 (usable)
[    0.000000]  BIOS-e820: 00000a0000000000 - 00000a1eff000000 (usable)
[    0.000000]  BIOS-e820: 00000a4000000000 - 00000a5eff000000 (usable)
[    0.000000]  BIOS-e820: 00000a8000000000 - 00000a9eff000000 (usable)
[    0.000000]  BIOS-e820: 00000ac000000000 - 00000adeff000000 (usable)
[    0.000000]  BIOS-e820: 00000b0000000000 - 00000b1eff000000 (usable)
[    0.000000]  BIOS-e820: 00000b4000000000 - 00000b5eff000000 (usable)
[    0.000000]  BIOS-e820: 00000b8000000000 - 00000b9eff000000 (usable)
[    0.000000]  BIOS-e820: 00000bc000000000 - 00000bdeff000000 (usable)
[    0.000000]  BIOS-e820: 00000c0000000000 - 00000c1eff000000 (usable)
[    0.000000]  BIOS-e820: 00000c4000000000 - 00000c5eff000000 (usable)
[    0.000000]  BIOS-e820: 00000c8000000000 - 00000c9eff000000 (usable)
[    0.000000]  BIOS-e820: 00000cc000000000 - 00000cdeff000000 (usable)
[    0.000000]  BIOS-e820: 00000d0000000000 - 00000d1eff000000 (usable)
[    0.000000]  BIOS-e820: 00000d4000000000 - 00000d5eff000000 (usable)
[    0.000000]  BIOS-e820: 00000d8000000000 - 00000d9eff000000 (usable)
[    0.000000]  BIOS-e820: 00000dc000000000 - 00000ddeff000000 (usable)
[    0.000000]  BIOS-e820: 00000e0000000000 - 00000e1eff000000 (usable)
[    0.000000]  BIOS-e820: 00000e4000000000 - 00000e5eff000000 (usable)
[    0.000000]  BIOS-e820: 00000e8000000000 - 00000e9eff000000 (usable)
[    0.000000]  BIOS-e820: 00000ec000000000 - 00000edeff000000 (usable)
[    0.000000]  BIOS-e820: 00000f0000000000 - 00000f1eff000000 (usable)
[    0.000000]  BIOS-e820: 00000f4000000000 - 00000f5eff000000 (usable)
[    0.000000]  BIOS-e820: 00000f8000000000 - 00000f9eff000000 (usable)
[    0.000000]  BIOS-e820: 00000fc000000000 - 00000fdeff000000 (usable)
[    0.000000]  BIOS-e820: 0000100000000000 - 0000101eff000000 (usable)
[    0.000000]  BIOS-e820: 0000104000000000 - 0000105eff000000 (usable)
[    0.000000]  BIOS-e820: 0000108000000000 - 0000109eff000000 (usable)
[    0.000000]  BIOS-e820: 000010c000000000 - 000010deff000000 (usable)
[    0.000000]  BIOS-e820: 0000110000000000 - 0000111eff000000 (usable)
[    0.000000]  BIOS-e820: 0000114000000000 - 0000115eff000000 (usable)
[    0.000000]  BIOS-e820: 0000118000000000 - 0000119eff000000 (usable)
[    0.000000]  BIOS-e820: 000011c000000000 - 000011deff000000 (usable)
[    0.000000]  BIOS-e820: 0000120000000000 - 0000121eff000000 (usable)
[    0.000000]  BIOS-e820: 0000124000000000 - 0000125eff000000 (usable)
[    0.000000]  BIOS-e820: 0000128000000000 - 0000129eff000000 (usable)
[    0.000000]  BIOS-e820: 000012c000000000 - 000012deff000000 (usable)
[    0.000000]  BIOS-e820: 0000130000000000 - 0000131eff000000 (usable)
[    0.000000]  BIOS-e820: 0000134000000000 - 0000135eff000000 (usable)
[    0.000000]  BIOS-e820: 0000138000000000 - 0000139eff000000 (usable)
[    0.000000]  BIOS-e820: 000013c000000000 - 000013deff000000 (usable)
[    0.000000]  BIOS-e820: 0000140000000000 - 0000141eff000000 (usable)
[    0.000000]  BIOS-e820: 0000144000000000 - 0000145eff000000 (usable)
[    0.000000]  BIOS-e820: 0000148000000000 - 0000149eff000000 (usable)
[    0.000000]  BIOS-e820: 000014c000000000 - 000014deff000000 (usable)
[    0.000000]  BIOS-e820: 0000150000000000 - 0000151eff000000 (usable)
[    0.000000]  BIOS-e820: 0000154000000000 - 0000155eff000000 (usable)
[    0.000000]  BIOS-e820: 0000158000000000 - 0000159eff000000 (usable)
[    0.000000]  BIOS-e820: 000015c000000000 - 000015deff000000 (usable)
[    0.000000]  BIOS-e820: 0000160000000000 - 0000161eff000000 (usable)
[    0.000000]  BIOS-e820: 0000164000000000 - 0000165eff000000 (usable)
[    0.000000]  BIOS-e820: 0000168000000000 - 0000169eff000000 (usable)
[    0.000000]  BIOS-e820: 000016c000000000 - 000016deff000000 (usable)
[    0.000000]  BIOS-e820: 0000170000000000 - 0000171eff000000 (usable)
[    0.000000]  BIOS-e820: 0000174000000000 - 0000175eff000000 (usable)
[    0.000000]  BIOS-e820: 0000178000000000 - 0000179eff000000 (usable)
[    0.000000]  BIOS-e820: 000017c000000000 - 000017deff000000 (usable)
[    0.000000]  BIOS-e820: 0000180000000000 - 0000181eff000000 (usable)
[    0.000000]  BIOS-e820: 0000184000000000 - 0000185eff000000 (usable)
[    0.000000]  BIOS-e820: 0000188000000000 - 0000189eff000000 (usable)
[    0.000000]  BIOS-e820: 000018c000000000 - 000018deff000000 (usable)
[    0.000000]  BIOS-e820: 0000190000000000 - 0000191eff000000 (usable)
[    0.000000]  BIOS-e820: 0000194000000000 - 0000195eff000000 (usable)
[    0.000000]  BIOS-e820: 0000198000000000 - 0000199eff000000 (usable)
[    0.000000]  BIOS-e820: 000019c000000000 - 000019deff000000 (usable)
[    0.000000]  BIOS-e820: 00001a0000000000 - 00001a1eff000000 (usable)
[    0.000000]  BIOS-e820: 00001a4000000000 - 00001a5eff000000 (usable)
[    0.000000]  BIOS-e820: 00001a8000000000 - 00001a9eff000000 (usable)
[    0.000000]  BIOS-e820: 00001ac000000000 - 00001adeff000000 (usable)
[    0.000000]  BIOS-e820: 00001b0000000000 - 00001b1eff000000 (usable)
[    0.000000]  BIOS-e820: 00001b4000000000 - 00001b5eff000000 (usable)
[    0.000000]  BIOS-e820: 00001b8000000000 - 00001b9eff000000 (usable)
[    0.000000]  BIOS-e820: 00001bc000000000 - 00001bdeff000000 (usable)
[    0.000000] bootconsole [earlyser0] enabled
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] EFI v2.30 by EDK II
[    0.000000]  ACPI=0x7ef3e000  ACPI 2.0=0x7ef3e014  SMBIOS=0x7be0f000  UVsystab=0x7bbcb000 
[    0.000000] Kernel-defined memdesc doesn't match the one from EFI!
[    0.000000] EFI: mem00: type=3, attr=0xf, range=[0x0000000000000000-0x0000000000001000) (0MB)
[    0.000000] EFI: mem01: type=7, attr=0xf, range=[0x0000000000001000-0x000000000005f000) (0MB)
[    0.000000] EFI: mem02: type=4, attr=0xf, range=[0x000000000005f000-0x0000000000060000) (0MB)
[    0.000000] EFI: mem03: type=3, attr=0xf, range=[0x0000000000060000-0x000000000007f000) (0MB)
[    0.000000] EFI: mem04: type=0, attr=0xf, range=[0x000000000007f000-0x0000000000080000) (0MB)
[    0.000000] EFI: mem05: type=3, attr=0xf, range=[0x0000000000080000-0x00000000000a0000) (0MB)
[    0.000000] EFI: mem06: type=2, attr=0xf, range=[0x0000000000100000-0x0000000000400000) (3MB)
[    0.000000] EFI: mem07: type=3, attr=0xf, range=[0x0000000000400000-0x00000000004ee000) (0MB)
[    0.000000] EFI: mem08: type=7, attr=0xf, range=[0x00000000004ee000-0x0000000000c00000) (7MB)
[    0.000000] EFI: mem09: type=3, attr=0xf, range=[0x0000000000c00000-0x0000000001000000) (4MB)
[    0.000000] EFI: mem10: type=2, attr=0xf, range=[0x0000000001000000-0x0000000001800000) (8MB)
[    0.000000] EFI: mem11: type=7, attr=0xf, range=[0x0000000001800000-0x0000000064821000) (1584MB)
[    0.000000] EFI: mem12: type=2, attr=0xf, range=[0x0000000064821000-0x0000000065051000) (8MB)
[    0.000000] EFI: mem13: type=1, attr=0xf, range=[0x0000000065051000-0x000000006508f000) (0MB)
[    0.000000] EFI: mem14: type=4, attr=0xf, range=[0x000000006508f000-0x000000006514c000) (0MB)
[    0.000000] EFI: mem15: type=1, attr=0xf, range=[0x000000006514c000-0x0000000065209000) (0MB)
[    0.000000] EFI: mem16: type=4, attr=0xf, range=[0x0000000065209000-0x00000000652ab000) (0MB)
[    0.000000] EFI: mem17: type=7, attr=0xf, range=[0x00000000652ab000-0x00000000652ad000) (0MB)
[    0.000000] EFI: mem18: type=4, attr=0xf, range=[0x00000000652ad000-0x00000000652ae000) (0MB)
[    0.000000] EFI: mem19: type=7, attr=0xf, range=[0x00000000652ae000-0x00000000652b6000) (0MB)
[    0.000000] EFI: mem20: type=4, attr=0xf, range=[0x00000000652b6000-0x00000000652bb000) (0MB)
[    0.000000] EFI: mem21: type=7, attr=0xf, range=[0x00000000652bb000-0x00000000652bc000) (0MB)
[    0.000000] EFI: mem22: type=4, attr=0xf, range=[0x00000000652bc000-0x00000000652d3000) (0MB)
[    0.000000] EFI: mem23: type=3, attr=0xf, range=[0x00000000652d3000-0x00000000652d6000) (0MB)
[    0.000000] EFI: mem24: type=4, attr=0xf, range=[0x00000000652d6000-0x00000000652e4000) (0MB)
[    0.000000] EFI: mem25: type=3, attr=0xf, range=[0x00000000652e4000-0x00000000652f8000) (0MB)
[    0.000000] EFI: mem26: type=4, attr=0xf, range=[0x00000000652f8000-0x0000000066a99000) (23MB)
[    0.000000] EFI: mem27: type=3, attr=0xf, range=[0x0000000066a99000-0x0000000066a9f000) (0MB)
[    0.000000] EFI: mem28: type=4, attr=0xf, range=[0x0000000066a9f000-0x000000006814e000) (22MB)
[    0.000000] EFI: mem29: type=3, attr=0xf, range=[0x000000006814e000-0x00000000681b7000) (0MB)
[    0.000000] EFI: mem30: type=4, attr=0xf, range=[0x00000000681b7000-0x0000000068307000) (1MB)
[    0.000000] EFI: mem31: type=3, attr=0xf, range=[0x0000000068307000-0x000000006830a000) (0MB)
[    0.000000] EFI: mem32: type=4, attr=0xf, range=[0x000000006830a000-0x0000000068338000) (0MB)
[    0.000000] EFI: mem33: type=3, attr=0xf, range=[0x0000000068338000-0x0000000068369000) (0MB)
[    0.000000] EFI: mem34: type=4, attr=0xf, range=[0x0000000068369000-0x0000000068390000) (0MB)
[    0.000000] EFI: mem35: type=3, attr=0xf, range=[0x0000000068390000-0x0000000068393000) (0MB)
[    0.000000] EFI: mem36: type=4, attr=0xf, range=[0x0000000068393000-0x000000006839c000) (0MB)
[    0.000000] EFI: mem37: type=3, attr=0xf, range=[0x000000006839c000-0x00000000683b0000) (0MB)
[    0.000000] EFI: mem38: type=4, attr=0xf, range=[0x00000000683b0000-0x00000000683b1000) (0MB)
[    0.000000] EFI: mem39: type=3, attr=0xf, range=[0x00000000683b1000-0x00000000683b4000) (0MB)
[    0.000000] EFI: mem40: type=4, attr=0xf, range=[0x00000000683b4000-0x00000000683b5000) (0MB)
[    0.000000] EFI: mem41: type=3, attr=0xf, range=[0x00000000683b5000-0x00000000683bd000) (0MB)
[    0.000000] EFI: mem42: type=4, attr=0xf, range=[0x00000000683bd000-0x00000000683be000) (0MB)
[    0.000000] EFI: mem43: type=3, attr=0xf, range=[0x00000000683be000-0x00000000683c9000) (0MB)
[    0.000000] EFI: mem44: type=4, attr=0xf, range=[0x00000000683c9000-0x00000000683ca000) (0MB)
[    0.000000] EFI: mem45: type=3, attr=0xf, range=[0x00000000683ca000-0x00000000683cc000) (0MB)
[    0.000000] EFI: mem46: type=4, attr=0xf, range=[0x00000000683cc000-0x00000000683ce000) (0MB)
[    0.000000] EFI: mem47: type=3, attr=0xf, range=[0x00000000683ce000-0x00000000683d4000) (0MB)
[    0.000000] EFI: mem48: type=4, attr=0xf, range=[0x00000000683d4000-0x00000000683d9000) (0MB)
[    0.000000] EFI: mem49: type=3, attr=0xf, range=[0x00000000683d9000-0x00000000683dc000) (0MB)
[    0.000000] EFI: mem50: type=4, attr=0xf, range=[0x00000000683dc000-0x00000000683df000) (0MB)
[    0.000000] EFI: mem51: type=3, attr=0xf, range=[0x00000000683df000-0x00000000683eb000) (0MB)
[    0.000000] EFI: mem52: type=4, attr=0xf, range=[0x00000000683eb000-0x000000006840e000) (0MB)
[    0.000000] EFI: mem53: type=3, attr=0xf, range=[0x000000006840e000-0x0000000068415000) (0MB)
[    0.000000] EFI: mem54: type=4, attr=0xf, range=[0x0000000068415000-0x000000006841b000) (0MB)
[    0.000000] EFI: mem55: type=3, attr=0xf, range=[0x000000006841b000-0x0000000068420000) (0MB)
[    0.000000] EFI: mem56: type=4, attr=0xf, range=[0x0000000068420000-0x0000000068423000) (0MB)
[    0.000000] EFI: mem57: type=3, attr=0xf, range=[0x0000000068423000-0x0000000068436000) (0MB)
[    0.000000] EFI: mem58: type=4, attr=0xf, range=[0x0000000068436000-0x0000000068437000) (0MB)
[    0.000000] EFI: mem59: type=3, attr=0xf, range=[0x0000000068437000-0x000000006843f000) (0MB)
[    0.000000] EFI: mem60: type=4, attr=0xf, range=[0x000000006843f000-0x0000000068440000) (0MB)
[    0.000000] EFI: mem61: type=3, attr=0xf, range=[0x0000000068440000-0x0000000068459000) (0MB)
[    0.000000] EFI: mem62: type=4, attr=0xf, range=[0x0000000068459000-0x000000006845b000) (0MB)
[    0.000000] EFI: mem63: type=3, attr=0xf, range=[0x000000006845b000-0x0000000068475000) (0MB)
[    0.000000] EFI: mem64: type=4, attr=0xf, range=[0x0000000068475000-0x0000000068478000) (0MB)
[    0.000000] EFI: mem65: type=3, attr=0xf, range=[0x0000000068478000-0x0000000068493000) (0MB)
[    0.000000] EFI: mem66: type=4, attr=0xf, range=[0x0000000068493000-0x0000000068495000) (0MB)
[    0.000000] EFI: mem67: type=3, attr=0xf, range=[0x0000000068495000-0x000000006849b000) (0MB)
[    0.000000] EFI: mem68: type=4, attr=0xf, range=[0x000000006849b000-0x000000006849c000) (0MB)
[    0.000000] EFI: mem69: type=3, attr=0xf, range=[0x000000006849c000-0x00000000684a9000) (0MB)
[    0.000000] EFI: mem70: type=4, attr=0xf, range=[0x00000000684a9000-0x00000000684ab000) (0MB)
[    0.000000] EFI: mem71: type=3, attr=0xf, range=[0x00000000684ab000-0x00000000684c2000) (0MB)
[    0.000000] EFI: mem72: type=4, attr=0xf, range=[0x00000000684c2000-0x00000000684c6000) (0MB)
[    0.000000] EFI: mem73: type=3, attr=0xf, range=[0x00000000684c6000-0x00000000684e5000) (0MB)
[    0.000000] EFI: mem74: type=4, attr=0xf, range=[0x00000000684e5000-0x00000000684e6000) (0MB)
[    0.000000] EFI: mem75: type=3, attr=0xf, range=[0x00000000684e6000-0x00000000684e8000) (0MB)
[    0.000000] EFI: mem76: type=4, attr=0xf, range=[0x00000000684e8000-0x00000000684ef000) (0MB)
[    0.000000] EFI: mem77: type=3, attr=0xf, range=[0x00000000684ef000-0x0000000068524000) (0MB)
[    0.000000] EFI: mem78: type=4, attr=0xf, range=[0x0000000068524000-0x0000000068525000) (0MB)
[    0.000000] EFI: mem79: type=3, attr=0xf, range=[0x0000000068525000-0x000000006855f000) (0MB)
[    0.000000] EFI: mem80: type=4, attr=0xf, range=[0x000000006855f000-0x0000000068560000) (0MB)
[    0.000000] EFI: mem81: type=3, attr=0xf, range=[0x0000000068560000-0x000000006856b000) (0MB)
[    0.000000] EFI: mem82: type=4, attr=0xf, range=[0x000000006856b000-0x000000006856d000) (0MB)
[    0.000000] EFI: mem83: type=3, attr=0xf, range=[0x000000006856d000-0x0000000068577000) (0MB)
[    0.000000] EFI: mem84: type=4, attr=0xf, range=[0x0000000068577000-0x0000000068578000) (0MB)
[    0.000000] EFI: mem85: type=3, attr=0xf, range=[0x0000000068578000-0x000000006858a000) (0MB)
[    0.000000] EFI: mem86: type=4, attr=0xf, range=[0x000000006858a000-0x0000000068632000) (0MB)
[    0.000000] EFI: mem87: type=3, attr=0xf, range=[0x0000000068632000-0x0000000068643000) (0MB)
[    0.000000] EFI: mem88: type=4, attr=0xf, range=[0x0000000068643000-0x0000000068805000) (1MB)
[    0.000000] EFI: mem89: type=3, attr=0xf, range=[0x0000000068805000-0x0000000068809000) (0MB)
[    0.000000] EFI: mem90: type=7, attr=0xf, range=[0x0000000068809000-0x000000006880a000) (0MB)
[    0.000000] EFI: mem91: type=4, attr=0xf, range=[0x000000006880a000-0x000000006880c000) (0MB)
[    0.000000] EFI: mem92: type=7, attr=0xf, range=[0x000000006880c000-0x000000006880d000) (0MB)
[    0.000000] EFI: mem93: type=4, attr=0xf, range=[0x000000006880d000-0x000000006880e000) (0MB)
[    0.000000] EFI: mem94: type=7, attr=0xf, range=[0x000000006880e000-0x0000000068838000) (0MB)
[    0.000000] EFI: mem95: type=2, attr=0xf, range=[0x0000000068838000-0x0000000068850000) (0MB)
[    0.000000] EFI: mem96: type=4, attr=0xf, range=[0x0000000068850000-0x00000000691e6000) (9MB)
[    0.000000] EFI: mem97: type=3, attr=0xf, range=[0x00000000691e6000-0x00000000692a4000) (0MB)
[    0.000000] EFI: mem98: type=4, attr=0xf, range=[0x00000000692a4000-0x00000000692ad000) (0MB)
[    0.000000] EFI: mem99: type=3, attr=0xf, range=[0x00000000692ad000-0x00000000692c2000) (0MB)
[    0.000000] EFI: mem100: type=4, attr=0xf, range=[0x00000000692c2000-0x00000000692d5000) (0MB)
[    0.000000] EFI: mem101: type=3, attr=0xf, range=[0x00000000692d5000-0x0000000069305000) (0MB)
[    0.000000] EFI: mem102: type=4, attr=0xf, range=[0x0000000069305000-0x000000006932b000) (0MB)
[    0.000000] EFI: mem103: type=7, attr=0xf, range=[0x000000006932b000-0x000000006932c000) (0MB)
[    0.000000] EFI: mem104: type=4, attr=0xf, range=[0x000000006932c000-0x0000000069339000) (0MB)
[    0.000000] EFI: mem105: type=3, attr=0xf, range=[0x0000000069339000-0x0000000069342000) (0MB)
[    0.000000] EFI: mem106: type=4, attr=0xf, range=[0x0000000069342000-0x0000000069347000) (0MB)
[    0.000000] EFI: mem107: type=3, attr=0xf, range=[0x0000000069347000-0x0000000069348000) (0MB)
[    0.000000] EFI: mem108: type=4, attr=0xf, range=[0x0000000069348000-0x000000006934e000) (0MB)
[    0.000000] EFI: mem109: type=7, attr=0xf, range=[0x000000006934e000-0x0000000069352000) (0MB)
[    0.000000] EFI: mem110: type=2, attr=0xf, range=[0x0000000069352000-0x0000000069357000) (0MB)
[    0.000000] EFI: mem111: type=7, attr=0xf, range=[0x0000000069357000-0x000000006935b000) (0MB)
[    0.000000] EFI: mem112: type=2, attr=0xf, range=[0x000000006935b000-0x000000006936d000) (0MB)
[    0.000000] EFI: mem113: type=4, attr=0xf, range=[0x000000006936d000-0x00000000693b9000) (0MB)
[    0.000000] EFI: mem114: type=7, attr=0xf, range=[0x00000000693b9000-0x00000000693bb000) (0MB)
[    0.000000] EFI: mem115: type=2, attr=0xf, range=[0x00000000693bb000-0x00000000693c1000) (0MB)
[    0.000000] EFI: mem116: type=7, attr=0xf, range=[0x00000000693c1000-0x00000000693c2000) (0MB)
[    0.000000] EFI: mem117: type=4, attr=0xf, range=[0x00000000693c2000-0x00000000693db000) (0MB)
[    0.000000] EFI: mem118: type=2, attr=0xf, range=[0x00000000693db000-0x00000000693dc000) (0MB)
[    0.000000] EFI: mem119: type=4, attr=0xf, range=[0x00000000693dc000-0x00000000693f0000) (0MB)
[    0.000000] EFI: mem120: type=2, attr=0xf, range=[0x00000000693f0000-0x0000000069405000) (0MB)
[    0.000000] EFI: mem121: type=4, attr=0xf, range=[0x0000000069405000-0x0000000069406000) (0MB)
[    0.000000] EFI: mem122: type=2, attr=0xf, range=[0x0000000069406000-0x0000000069409000) (0MB)
[    0.000000] EFI: mem123: type=4, attr=0xf, range=[0x0000000069409000-0x000000006943a000) (0MB)
[    0.000000] EFI: mem124: type=3, attr=0xf, range=[0x000000006943a000-0x0000000069474000) (0MB)
[    0.000000] EFI: mem125: type=4, attr=0xf, range=[0x0000000069474000-0x0000000069475000) (0MB)
[    0.000000] EFI: mem126: type=3, attr=0xf, range=[0x0000000069475000-0x000000006947e000) (0MB)
[    0.000000] EFI: mem127: type=4, attr=0xf, range=[0x000000006947e000-0x0000000069480000) (0MB)
[    0.000000] EFI: mem128: type=3, attr=0xf, range=[0x0000000069480000-0x0000000069495000) (0MB)
[    0.000000] EFI: mem129: type=4, attr=0xf, range=[0x0000000069495000-0x0000000069499000) (0MB)
[    0.000000] EFI: mem130: type=3, attr=0xf, range=[0x0000000069499000-0x00000000694aa000) (0MB)
[    0.000000] EFI: mem131: type=4, attr=0xf, range=[0x00000000694aa000-0x00000000694b7000) (0MB)
[    0.000000] EFI: mem132: type=3, attr=0xf, range=[0x00000000694b7000-0x00000000694ba000) (0MB)
[    0.000000] EFI: mem133: type=4, attr=0xf, range=[0x00000000694ba000-0x00000000694bd000) (0MB)
[    0.000000] EFI: mem134: type=3, attr=0xf, range=[0x00000000694bd000-0x00000000694de000) (0MB)
[    0.000000] EFI: mem135: type=4, attr=0xf, range=[0x00000000694de000-0x00000000698de000) (4MB)
[    0.000000] EFI: mem136: type=3, attr=0xf, range=[0x00000000698de000-0x00000000698e4000) (0MB)
[    0.000000] EFI: mem137: type=4, attr=0xf, range=[0x00000000698e4000-0x00000000698e7000) (0MB)
[    0.000000] EFI: mem138: type=3, attr=0xf, range=[0x00000000698e7000-0x00000000698e9000) (0MB)
[    0.000000] EFI: mem139: type=4, attr=0xf, range=[0x00000000698e9000-0x00000000698eb000) (0MB)
[    0.000000] EFI: mem140: type=3, attr=0xf, range=[0x00000000698eb000-0x00000000698ed000) (0MB)
[    0.000000] EFI: mem141: type=4, attr=0xf, range=[0x00000000698ed000-0x00000000698ee000) (0MB)
[    0.000000] EFI: mem142: type=3, attr=0xf, range=[0x00000000698ee000-0x00000000698f0000) (0MB)
[    0.000000] EFI: mem143: type=4, attr=0xf, range=[0x00000000698f0000-0x00000000698f1000) (0MB)
[    0.000000] EFI: mem144: type=3, attr=0xf, range=[0x00000000698f1000-0x0000000069907000) (0MB)
[    0.000000] EFI: mem145: type=4, attr=0xf, range=[0x0000000069907000-0x000000006994e000) (0MB)
[    0.000000] EFI: mem146: type=3, attr=0xf, range=[0x000000006994e000-0x000000006998b000) (0MB)
[    0.000000] EFI: mem147: type=4, attr=0xf, range=[0x000000006998b000-0x00000000699d8000) (0MB)
[    0.000000] EFI: mem148: type=3, attr=0xf, range=[0x00000000699d8000-0x00000000699da000) (0MB)
[    0.000000] EFI: mem149: type=4, attr=0xf, range=[0x00000000699da000-0x00000000699db000) (0MB)
[    0.000000] EFI: mem150: type=3, attr=0xf, range=[0x00000000699db000-0x00000000699ed000) (0MB)
[    0.000000] EFI: mem151: type=4, attr=0xf, range=[0x00000000699ed000-0x0000000069a25000) (0MB)
[    0.000000] EFI: mem152: type=3, attr=0xf, range=[0x0000000069a25000-0x0000000069a29000) (0MB)
[    0.000000] EFI: mem153: type=4, attr=0xf, range=[0x0000000069a29000-0x000000007abb1000) (273MB)
[    0.000000] EFI: mem154: type=6, attr=0x800000000000000f, range=[0x000000007abb1000-0x000000007abb2000) (0MB)
[    0.000000] EFI: mem155: type=4, attr=0xf, range=[0x000000007abb2000-0x000000007abc5000) (0MB)
[    0.000000] EFI: mem156: type=6, attr=0x800000000000000f, range=[0x000000007abc5000-0x000000007ae05000) (2MB)
[    0.000000] EFI: mem157: type=4, attr=0xf, range=[0x000000007ae05000-0x000000007bb13000) (13MB)
[    0.000000] EFI: mem158: type=6, attr=0x800000000000000f, range=[0x000000007bb13000-0x000000007bc13000) (1MB)
[    0.000000] EFI: mem159: type=5, attr=0x800000000000000f, range=[0x000000007bc13000-0x000000007bd13000) (1MB)
[    0.000000] EFI: mem160: type=0, attr=0xf, range=[0x000000007bd13000-0x000000007be13000) (1MB)
[    0.000000] EFI: mem161: type=10, attr=0xf, range=[0x000000007be13000-0x000000007d1f3000) (19MB)
[    0.000000] EFI: mem162: type=9, attr=0xf, range=[0x000000007d1f3000-0x000000007e000000) (14MB)
[    0.000000] EFI: mem163: type=2, attr=0xf, range=[0x000000007e000000-0x000000007e2ee000) (2MB)
[    0.000000] EFI: mem164: type=9, attr=0xf, range=[0x000000007e2ee000-0x000000007ef3f000) (12MB)
[    0.000000] EFI: mem165: type=4, attr=0xf, range=[0x000000007ef3f000-0x000000007f000000) (0MB)
[    0.000000] EFI: mem166: type=7, attr=0xf, range=[0x0000000100000000-0x0000001f80000000) (124928MB)
[    0.000000] EFI: mem167: type=7, attr=0xf, range=[0x0000004000000000-0x0000005eff000000) (126960MB)
[    0.000000] EFI: mem168: type=7, attr=0xf, range=[0x0000008000000000-0x0000009eff000000) (126960MB)
[    0.000000] EFI: mem169: type=7, attr=0xf, range=[0x000000c000000000-0x000000deff000000) (126960MB)
[    0.000000] EFI: mem170: type=7, attr=0xf, range=[0x0000010000000000-0x0000011eff000000) (126960MB)
[    0.000000] EFI: mem171: type=7, attr=0xf, range=[0x0000014000000000-0x0000015eff000000) (126960MB)
[    0.000000] EFI: mem172: type=7, attr=0xf, range=[0x0000018000000000-0x0000019eff000000) (126960MB)
[    0.000000] EFI: mem173: type=7, attr=0xf, range=[0x000001c000000000-0x000001deff000000) (126960MB)
[    0.000000] EFI: mem174: type=7, attr=0xf, range=[0x0000020000000000-0x0000021eff000000) (126960MB)
[    0.000000] EFI: mem175: type=7, attr=0xf, range=[0x0000024000000000-0x0000025eff000000) (126960MB)
[    0.000000] EFI: mem176: type=7, attr=0xf, range=[0x0000028000000000-0x0000029eff000000) (126960MB)
[    0.000000] EFI: mem177: type=7, attr=0xf, range=[0x000002c000000000-0x000002deff000000) (126960MB)
[    0.000000] EFI: mem178: type=7, attr=0xf, range=[0x0000030000000000-0x0000031eff000000) (126960MB)
[    0.000000] EFI: mem179: type=7, attr=0xf, range=[0x0000034000000000-0x0000035eff000000) (126960MB)
[    0.000000] EFI: mem180: type=7, attr=0xf, range=[0x0000038000000000-0x0000039eff000000) (126960MB)
[    0.000000] EFI: mem181: type=7, attr=0xf, range=[0x000003c000000000-0x000003deff000000) (126960MB)
[    0.000000] EFI: mem182: type=7, attr=0xf, range=[0x0000040000000000-0x0000041eff000000) (126960MB)
[    0.000000] EFI: mem183: type=7, attr=0xf, range=[0x0000044000000000-0x0000045eff000000) (126960MB)
[    0.000000] EFI: mem184: type=7, attr=0xf, range=[0x0000048000000000-0x0000049eff000000) (126960MB)
[    0.000000] EFI: mem185: type=7, attr=0xf, range=[0x000004c000000000-0x000004deff000000) (126960MB)
[    0.000000] EFI: mem186: type=7, attr=0xf, range=[0x0000050000000000-0x0000051eff000000) (126960MB)
[    0.000000] EFI: mem187: type=7, attr=0xf, range=[0x0000054000000000-0x0000055eff000000) (126960MB)
[    0.000000] EFI: mem188: type=7, attr=0xf, range=[0x0000058000000000-0x0000059eff000000) (126960MB)
[    0.000000] EFI: mem189: type=7, attr=0xf, range=[0x000005c000000000-0x000005deff000000) (126960MB)
[    0.000000] EFI: mem190: type=7, attr=0xf, range=[0x0000060000000000-0x0000061eff000000) (126960MB)
[    0.000000] EFI: mem191: type=7, attr=0xf, range=[0x0000064000000000-0x0000065eff000000) (126960MB)
[    0.000000] EFI: mem192: type=7, attr=0xf, range=[0x0000068000000000-0x0000069eff000000) (126960MB)
[    0.000000] EFI: mem193: type=7, attr=0xf, range=[0x000006c000000000-0x000006deff000000) (126960MB)
[    0.000000] EFI: mem194: type=7, attr=0xf, range=[0x0000070000000000-0x0000071eff000000) (126960MB)
[    0.000000] EFI: mem195: type=7, attr=0xf, range=[0x0000074000000000-0x0000075eff000000) (126960MB)
[    0.000000] EFI: mem196: type=7, attr=0xf, range=[0x0000078000000000-0x0000079eff000000) (126960MB)
[    0.000000] EFI: mem197: type=7, attr=0xf, range=[0x000007c000000000-0x000007deff000000) (126960MB)
[    0.000000] EFI: mem198: type=7, attr=0xf, range=[0x0000080000000000-0x0000081eff000000) (126960MB)
[    0.000000] EFI: mem199: type=7, attr=0xf, range=[0x0000084000000000-0x0000085eff000000) (126960MB)
[    0.000000] EFI: mem200: type=7, attr=0xf, range=[0x0000088000000000-0x0000089eff000000) (126960MB)
[    0.000000] EFI: mem201: type=7, attr=0xf, range=[0x000008c000000000-0x000008deff000000) (126960MB)
[    0.000000] EFI: mem202: type=7, attr=0xf, range=[0x0000090000000000-0x0000091eff000000) (126960MB)
[    0.000000] EFI: mem203: type=7, attr=0xf, range=[0x0000094000000000-0x0000095eff000000) (126960MB)
[    0.000000] EFI: mem204: type=7, attr=0xf, range=[0x0000098000000000-0x0000099eff000000) (126960MB)
[    0.000000] EFI: mem205: type=7, attr=0xf, range=[0x000009c000000000-0x000009deff000000) (126960MB)
[    0.000000] EFI: mem206: type=7, attr=0xf, range=[0x00000a0000000000-0x00000a1eff000000) (126960MB)
[    0.000000] EFI: mem207: type=7, attr=0xf, range=[0x00000a4000000000-0x00000a5eff000000) (126960MB)
[    0.000000] EFI: mem208: type=7, attr=0xf, range=[0x00000a8000000000-0x00000a9eff000000) (126960MB)
[    0.000000] EFI: mem209: type=7, attr=0xf, range=[0x00000ac000000000-0x00000adeff000000) (126960MB)
[    0.000000] EFI: mem210: type=7, attr=0xf, range=[0x00000b0000000000-0x00000b1eff000000) (126960MB)
[    0.000000] EFI: mem211: type=7, attr=0xf, range=[0x00000b4000000000-0x00000b5eff000000) (126960MB)
[    0.000000] EFI: mem212: type=7, attr=0xf, range=[0x00000b8000000000-0x00000b9eff000000) (126960MB)
[    0.000000] EFI: mem213: type=7, attr=0xf, range=[0x00000bc000000000-0x00000bdeff000000) (126960MB)
[    0.000000] EFI: mem214: type=7, attr=0xf, range=[0x00000c0000000000-0x00000c1eff000000) (126960MB)
[    0.000000] EFI: mem215: type=7, attr=0xf, range=[0x00000c4000000000-0x00000c5eff000000) (126960MB)
[    0.000000] EFI: mem216: type=7, attr=0xf, range=[0x00000c8000000000-0x00000c9eff000000) (126960MB)
[    0.000000] EFI: mem217: type=7, attr=0xf, range=[0x00000cc000000000-0x00000cdeff000000) (126960MB)
[    0.000000] EFI: mem218: type=7, attr=0xf, range=[0x00000d0000000000-0x00000d1eff000000) (126960MB)
[    0.000000] EFI: mem219: type=7, attr=0xf, range=[0x00000d4000000000-0x00000d5eff000000) (126960MB)
[    0.000000] EFI: mem220: type=7, attr=0xf, range=[0x00000d8000000000-0x00000d9eff000000) (126960MB)
[    0.000000] EFI: mem221: type=7, attr=0xf, range=[0x00000dc000000000-0x00000ddeff000000) (126960MB)
[    0.000000] EFI: mem222: type=7, attr=0xf, range=[0x00000e0000000000-0x00000e1eff000000) (126960MB)
[    0.000000] EFI: mem223: type=7, attr=0xf, range=[0x00000e4000000000-0x00000e5eff000000) (126960MB)
[    0.000000] EFI: mem224: type=7, attr=0xf, range=[0x00000e8000000000-0x00000e9eff000000) (126960MB)
[    0.000000] EFI: mem225: type=7, attr=0xf, range=[0x00000ec000000000-0x00000edeff000000) (126960MB)
[    0.000000] EFI: mem226: type=7, attr=0xf, range=[0x00000f0000000000-0x00000f1eff000000) (126960MB)
[    0.000000] EFI: mem227: type=7, attr=0xf, range=[0x00000f4000000000-0x00000f5eff000000) (126960MB)
[    0.000000] EFI: mem228: type=7, attr=0xf, range=[0x00000f8000000000-0x00000f9eff000000) (126960MB)
[    0.000000] EFI: mem229: type=7, attr=0xf, range=[0x00000fc000000000-0x00000fdeff000000) (126960MB)
[    0.000000] EFI: mem230: type=7, attr=0xf, range=[0x0000100000000000-0x0000101eff000000) (126960MB)
[    0.000000] EFI: mem231: type=7, attr=0xf, range=[0x0000104000000000-0x0000105eff000000) (126960MB)
[    0.000000] EFI: mem232: type=7, attr=0xf, range=[0x0000108000000000-0x0000109eff000000) (126960MB)
[    0.000000] EFI: mem233: type=7, attr=0xf, range=[0x000010c000000000-0x000010deff000000) (126960MB)
[    0.000000] EFI: mem234: type=7, attr=0xf, range=[0x0000110000000000-0x0000111eff000000) (126960MB)
[    0.000000] EFI: mem235: type=7, attr=0xf, range=[0x0000114000000000-0x0000115eff000000) (126960MB)
[    0.000000] EFI: mem236: type=7, attr=0xf, range=[0x0000118000000000-0x0000119eff000000) (126960MB)
[    0.000000] EFI: mem237: type=7, attr=0xf, range=[0x000011c000000000-0x000011deff000000) (126960MB)
[    0.000000] EFI: mem238: type=7, attr=0xf, range=[0x0000120000000000-0x0000121eff000000) (126960MB)
[    0.000000] EFI: mem239: type=7, attr=0xf, range=[0x0000124000000000-0x0000125eff000000) (126960MB)
[    0.000000] EFI: mem240: type=7, attr=0xf, range=[0x0000128000000000-0x0000129eff000000) (126960MB)
[    0.000000] EFI: mem241: type=7, attr=0xf, range=[0x000012c000000000-0x000012deff000000) (126960MB)
[    0.000000] EFI: mem242: type=7, attr=0xf, range=[0x0000130000000000-0x0000131eff000000) (126960MB)
[    0.000000] EFI: mem243: type=7, attr=0xf, range=[0x0000134000000000-0x0000135eff000000) (126960MB)
[    0.000000] EFI: mem244: type=7, attr=0xf, range=[0x0000138000000000-0x0000139eff000000) (126960MB)
[    0.000000] EFI: mem245: type=7, attr=0xf, range=[0x000013c000000000-0x000013deff000000) (126960MB)
[    0.000000] EFI: mem246: type=7, attr=0xf, range=[0x0000140000000000-0x0000141eff000000) (126960MB)
[    0.000000] EFI: mem247: type=7, attr=0xf, range=[0x0000144000000000-0x0000145eff000000) (126960MB)
[    0.000000] EFI: mem248: type=7, attr=0xf, range=[0x0000148000000000-0x0000149eff000000) (126960MB)
[    0.000000] EFI: mem249: type=7, attr=0xf, range=[0x000014c000000000-0x000014deff000000) (126960MB)
[    0.000000] EFI: mem250: type=7, attr=0xf, range=[0x0000150000000000-0x0000151eff000000) (126960MB)
[    0.000000] EFI: mem251: type=7, attr=0xf, range=[0x0000154000000000-0x0000155eff000000) (126960MB)
[    0.000000] EFI: mem252: type=7, attr=0xf, range=[0x0000158000000000-0x0000159eff000000) (126960MB)
[    0.000000] EFI: mem253: type=7, attr=0xf, range=[0x000015c000000000-0x000015deff000000) (126960MB)
[    0.000000] EFI: mem254: type=7, attr=0xf, range=[0x0000160000000000-0x0000161eff000000) (126960MB)
[    0.000000] EFI: mem255: type=7, attr=0xf, range=[0x0000164000000000-0x0000165eff000000) (126960MB)
[    0.000000] EFI: mem256: type=7, attr=0xf, range=[0x0000168000000000-0x0000169eff000000) (126960MB)
[    0.000000] EFI: mem257: type=7, attr=0xf, range=[0x000016c000000000-0x000016deff000000) (126960MB)
[    0.000000] EFI: mem258: type=7, attr=0xf, range=[0x0000170000000000-0x0000171eff000000) (126960MB)
[    0.000000] EFI: mem259: type=7, attr=0xf, range=[0x0000174000000000-0x0000175eff000000) (126960MB)
[    0.000000] EFI: mem260: type=7, attr=0xf, range=[0x0000178000000000-0x0000179eff000000) (126960MB)
[    0.000000] EFI: mem261: type=7, attr=0xf, range=[0x000017c000000000-0x000017deff000000) (126960MB)
[    0.000000] EFI: mem262: type=7, attr=0xf, range=[0x0000180000000000-0x0000181eff000000) (126960MB)
[    0.000000] EFI: mem263: type=7, attr=0xf, range=[0x0000184000000000-0x0000185eff000000) (126960MB)
[    0.000000] EFI: mem264: type=7, attr=0xf, range=[0x0000188000000000-0x0000189eff000000) (126960MB)
[    0.000000] EFI: mem265: type=7, attr=0xf, range=[0x000018c000000000-0x000018deff000000) (126960MB)
[    0.000000] EFI: mem266: type=7, attr=0xf, range=[0x0000190000000000-0x0000191eff000000) (126960MB)
[    0.000000] EFI: mem267: type=7, attr=0xf, range=[0x0000194000000000-0x0000195eff000000) (126960MB)
[    0.000000] EFI: mem268: type=7, attr=0xf, range=[0x0000198000000000-0x0000199eff000000) (126960MB)
[    0.000000] EFI: mem269: type=7, attr=0xf, range=[0x000019c000000000-0x000019deff000000) (126960MB)
[    0.000000] EFI: mem270: type=7, attr=0xf, range=[0x00001a0000000000-0x00001a1eff000000) (126960MB)
[    0.000000] EFI: mem271: type=7, attr=0xf, range=[0x00001a4000000000-0x00001a5eff000000) (126960MB)
[    0.000000] EFI: mem272: type=7, attr=0xf, range=[0x00001a8000000000-0x00001a9eff000000) (126960MB)
[    0.000000] EFI: mem273: type=7, attr=0xf, range=[0x00001ac000000000-0x00001adeff000000) (126960MB)
[    0.000000] EFI: mem274: type=7, attr=0xf, range=[0x00001b0000000000-0x00001b1eff000000) (126960MB)
[    0.000000] EFI: mem275: type=7, attr=0xf, range=[0x00001b4000000000-0x00001b5eff000000) (126960MB)
[    0.000000] EFI: mem276: type=7, attr=0xf, range=[0x00001b8000000000-0x00001b9eff000000) (126960MB)
[    0.000000] EFI: mem277: type=7, attr=0xf, range=[0x00001bc000000000-0x00001bdeff000000) (126960MB)
[    0.000000] EFI: mem278: type=7, attr=0xf, range=[0x00001c0000000000-0x00001c1eff000000) (126960MB)
[    0.000000] EFI: mem279: type=7, attr=0xf, range=[0x00001c4000000000-0x00001c5eff000000) (126960MB)
[    0.000000] EFI: mem280: type=7, attr=0xf, range=[0x00001c8000000000-0x00001c9eff000000) (126960MB)
[    0.000000] EFI: mem281: type=7, attr=0xf, range=[0x00001cc000000000-0x00001cdeff000000) (126960MB)
[    0.000000] EFI: mem282: type=7, attr=0xf, range=[0x00001d0000000000-0x00001d1eff000000) (126960MB)
[    0.000000] EFI: mem283: type=7, attr=0xf, range=[0x00001d4000000000-0x00001d5eff000000) (126960MB)
[    0.000000] EFI: mem284: type=7, attr=0xf, range=[0x00001d8000000000-0x00001d9eff000000) (126960MB)
[    0.000000] EFI: mem285: type=7, attr=0xf, range=[0x00001dc000000000-0x00001ddeff000000) (126960MB)
[    0.000000] EFI: mem286: type=7, attr=0xf, range=[0x00001e0000000000-0x00001e1eff000000) (126960MB)
[    0.000000] EFI: mem287: type=7, attr=0xf, range=[0x00001e4000000000-0x00001e5eff000000) (126960MB)
[    0.000000] EFI: mem288: type=7, attr=0xf, range=[0x00001e8000000000-0x00001e9eff000000) (126960MB)
[    0.000000] EFI: mem289: type=7, attr=0xf, range=[0x00001ec000000000-0x00001edeff000000) (126960MB)
[    0.000000] EFI: mem290: type=7, attr=0xf, range=[0x00001f0000000000-0x00001f1eff000000) (126960MB)
[    0.000000] EFI: mem291: type=7, attr=0xf, range=[0x00001f4000000000-0x00001f5eff000000) (126960MB)
[    0.000000] EFI: mem292: type=7, attr=0xf, range=[0x00001f8000000000-0x00001f9eff000000) (126960MB)
[    0.000000] EFI: mem293: type=7, attr=0xf, range=[0x00001fc000000000-0x00001fdeff000000) (126960MB)
[    0.000000] EFI: mem294: type=7, attr=0xf, range=[0x0000200000000000-0x0000201eff000000) (126960MB)
[    0.000000] EFI: mem295: type=7, attr=0xf, range=[0x0000204000000000-0x0000205eff000000) (126960MB)
[    0.000000] EFI: mem296: type=7, attr=0xf, range=[0x0000208000000000-0x0000209eff000000) (126960MB)
[    0.000000] EFI: mem297: type=7, attr=0xf, range=[0x000020c000000000-0x000020deff000000) (126960MB)
[    0.000000] EFI: mem298: type=7, attr=0xf, range=[0x0000210000000000-0x0000211eff000000) (126960MB)
[    0.000000] EFI: mem299: type=7, attr=0xf, range=[0x0000214000000000-0x0000215eff000000) (126960MB)
[    0.000000] EFI: mem300: type=7, attr=0xf, range=[0x0000218000000000-0x0000219eff000000) (126960MB)
[    0.000000] EFI: mem301: type=7, attr=0xf, range=[0x000021c000000000-0x000021deff000000) (126960MB)
[    0.000000] EFI: mem302: type=7, attr=0xf, range=[0x0000220000000000-0x0000221eff000000) (126960MB)
[    0.000000] EFI: mem303: type=7, attr=0xf, range=[0x0000224000000000-0x0000225eff000000) (126960MB)
[    0.000000] EFI: mem304: type=7, attr=0xf, range=[0x0000228000000000-0x0000229eff000000) (126960MB)
[    0.000000] EFI: mem305: type=7, attr=0xf, range=[0x000022c000000000-0x000022deff000000) (126960MB)
[    0.000000] EFI: mem306: type=7, attr=0xf, range=[0x0000230000000000-0x0000231eff000000) (126960MB)
[    0.000000] EFI: mem307: type=7, attr=0xf, range=[0x0000234000000000-0x0000235eff000000) (126960MB)
[    0.000000] EFI: mem308: type=7, attr=0xf, range=[0x0000238000000000-0x0000239eff000000) (126960MB)
[    0.000000] EFI: mem309: type=7, attr=0xf, range=[0x000023c000000000-0x000023deff000000) (126960MB)
[    0.000000] EFI: mem310: type=7, attr=0xf, range=[0x0000240000000000-0x0000241eff000000) (126960MB)
[    0.000000] EFI: mem311: type=7, attr=0xf, range=[0x0000244000000000-0x0000245eff000000) (126960MB)
[    0.000000] EFI: mem312: type=7, attr=0xf, range=[0x0000248000000000-0x0000249eff000000) (126960MB)
[    0.000000] EFI: mem313: type=7, attr=0xf, range=[0x000024c000000000-0x000024deff000000) (126960MB)
[    0.000000] EFI: mem314: type=7, attr=0xf, range=[0x0000250000000000-0x0000251eff000000) (126960MB)
[    0.000000] EFI: mem315: type=7, attr=0xf, range=[0x0000254000000000-0x0000255eff000000) (126960MB)
[    0.000000] EFI: mem316: type=7, attr=0xf, range=[0x0000258000000000-0x0000259eff000000) (126960MB)
[    0.000000] EFI: mem317: type=7, attr=0xf, range=[0x000025c000000000-0x000025deff000000) (126960MB)
[    0.000000] EFI: mem318: type=7, attr=0xf, range=[0x0000260000000000-0x0000261eff000000) (126960MB)
[    0.000000] EFI: mem319: type=7, attr=0xf, range=[0x0000264000000000-0x0000265eff000000) (126960MB)
[    0.000000] EFI: mem320: type=7, attr=0xf, range=[0x0000268000000000-0x0000269eff000000) (126960MB)
[    0.000000] EFI: mem321: type=7, attr=0xf, range=[0x000026c000000000-0x000026deff000000) (126960MB)
[    0.000000] EFI: mem322: type=7, attr=0xf, range=[0x0000270000000000-0x0000271eff000000) (126960MB)
[    0.000000] EFI: mem323: type=7, attr=0xf, range=[0x0000274000000000-0x0000275eff000000) (126960MB)
[    0.000000] EFI: mem324: type=7, attr=0xf, range=[0x0000278000000000-0x0000279eff000000) (126960MB)
[    0.000000] EFI: mem325: type=7, attr=0xf, range=[0x000027c000000000-0x000027deff000000) (126960MB)
[    0.000000] EFI: mem326: type=7, attr=0xf, range=[0x0000280000000000-0x0000281eff000000) (126960MB)
[    0.000000] EFI: mem327: type=7, attr=0xf, range=[0x0000284000000000-0x0000285eff000000) (126960MB)
[    0.000000] EFI: mem328: type=7, attr=0xf, range=[0x0000288000000000-0x0000289eff000000) (126960MB)
[    0.000000] EFI: mem329: type=7, attr=0xf, range=[0x000028c000000000-0x000028deff000000) (126960MB)
[    0.000000] EFI: mem330: type=7, attr=0xf, range=[0x0000290000000000-0x0000291eff000000) (126960MB)
[    0.000000] EFI: mem331: type=7, attr=0xf, range=[0x0000294000000000-0x0000295eff000000) (126960MB)
[    0.000000] EFI: mem332: type=7, attr=0xf, range=[0x0000298000000000-0x0000299eff000000) (126960MB)
[    0.000000] EFI: mem333: type=7, attr=0xf, range=[0x000029c000000000-0x000029deff000000) (126960MB)
[    0.000000] EFI: mem334: type=7, attr=0xf, range=[0x00002a0000000000-0x00002a1eff000000) (126960MB)
[    0.000000] EFI: mem335: type=7, attr=0xf, range=[0x00002a4000000000-0x00002a5eff000000) (126960MB)
[    0.000000] EFI: mem336: type=7, attr=0xf, range=[0x00002a8000000000-0x00002a9eff000000) (126960MB)
[    0.000000] EFI: mem337: type=7, attr=0xf, range=[0x00002ac000000000-0x00002adeff000000) (126960MB)
[    0.000000] EFI: mem338: type=7, attr=0xf, range=[0x00002b0000000000-0x00002b1eff000000) (126960MB)
[    0.000000] EFI: mem339: type=7, attr=0xf, range=[0x00002b4000000000-0x00002b5eff000000) (126960MB)
[    0.000000] EFI: mem340: type=7, attr=0xf, range=[0x00002b8000000000-0x00002b9eff000000) (126960MB)
[    0.000000] EFI: mem341: type=7, attr=0xf, range=[0x00002bc000000000-0x00002bdeff000000) (126960MB)
[    0.000000] EFI: mem342: type=7, attr=0xf, range=[0x00002c0000000000-0x00002c1eff000000) (126960MB)
[    0.000000] EFI: mem343: type=7, attr=0xf, range=[0x00002c4000000000-0x00002c5eff000000) (126960MB)
[    0.000000] EFI: mem344: type=7, attr=0xf, range=[0x00002c8000000000-0x00002c9eff000000) (126960MB)
[    0.000000] EFI: mem345: type=7, attr=0xf, range=[0x00002cc000000000-0x00002cdeff000000) (126960MB)
[    0.000000] EFI: mem346: type=7, attr=0xf, range=[0x00002d0000000000-0x00002d1eff000000) (126960MB)
[    0.000000] EFI: mem347: type=7, attr=0xf, range=[0x00002d4000000000-0x00002d5eff000000) (126960MB)
[    0.000000] EFI: mem348: type=7, attr=0xf, range=[0x00002d8000000000-0x00002d9eff000000) (126960MB)
[    0.000000] EFI: mem349: type=7, attr=0xf, range=[0x00002dc000000000-0x00002ddeff000000) (126960MB)
[    0.000000] EFI: mem350: type=7, attr=0xf, range=[0x00002e0000000000-0x00002e1eff000000) (126960MB)
[    0.000000] EFI: mem351: type=7, attr=0xf, range=[0x00002e4000000000-0x00002e5eff000000) (126960MB)
[    0.000000] EFI: mem352: type=7, attr=0xf, range=[0x00002e8000000000-0x00002e9eff000000) (126960MB)
[    0.000000] EFI: mem353: type=7, attr=0xf, range=[0x00002ec000000000-0x00002edeff000000) (126960MB)
[    0.000000] EFI: mem354: type=7, attr=0xf, range=[0x00002f0000000000-0x00002f1eff000000) (126960MB)
[    0.000000] EFI: mem355: type=7, attr=0xf, range=[0x00002f4000000000-0x00002f5eff000000) (126960MB)
[    0.000000] EFI: mem356: type=7, attr=0xf, range=[0x00002f8000000000-0x00002f9eff000000) (126960MB)
[    0.000000] EFI: mem357: type=7, attr=0xf, range=[0x00002fc000000000-0x00002fdeff000000) (126960MB)
[    0.000000] EFI: mem358: type=7, attr=0xf, range=[0x0000300000000000-0x0000301eff000000) (126960MB)
[    0.000000] EFI: mem359: type=7, attr=0xf, range=[0x0000304000000000-0x0000305eff000000) (126960MB)
[    0.000000] EFI: mem360: type=7, attr=0xf, range=[0x0000308000000000-0x0000309eff000000) (126960MB)
[    0.000000] EFI: mem361: type=7, attr=0xf, range=[0x000030c000000000-0x000030deff000000) (126960MB)
[    0.000000] EFI: mem362: type=7, attr=0xf, range=[0x0000310000000000-0x0000311eff000000) (126960MB)
[    0.000000] EFI: mem363: type=7, attr=0xf, range=[0x0000314000000000-0x0000315eff000000) (126960MB)
[    0.000000] EFI: mem364: type=7, attr=0xf, range=[0x0000318000000000-0x0000319eff000000) (126960MB)
[    0.000000] EFI: mem365: type=7, attr=0xf, range=[0x000031c000000000-0x000031deff000000) (126960MB)
[    0.000000] EFI: mem366: type=7, attr=0xf, range=[0x0000320000000000-0x0000321eff000000) (126960MB)
[    0.000000] EFI: mem367: type=7, attr=0xf, range=[0x0000324000000000-0x0000325eff000000) (126960MB)
[    0.000000] EFI: mem368: type=7, attr=0xf, range=[0x0000328000000000-0x0000329eff000000) (126960MB)
[    0.000000] EFI: mem369: type=7, attr=0xf, range=[0x000032c000000000-0x000032deff000000) (126960MB)
[    0.000000] EFI: mem370: type=7, attr=0xf, range=[0x0000330000000000-0x0000331eff000000) (126960MB)
[    0.000000] EFI: mem371: type=7, attr=0xf, range=[0x0000334000000000-0x0000335eff000000) (126960MB)
[    0.000000] EFI: mem372: type=7, attr=0xf, range=[0x0000338000000000-0x0000339eff000000) (126960MB)
[    0.000000] EFI: mem373: type=7, attr=0xf, range=[0x000033c000000000-0x000033deff000000) (126960MB)
[    0.000000] EFI: mem374: type=7, attr=0xf, range=[0x0000340000000000-0x0000341eff000000) (126960MB)
[    0.000000] EFI: mem375: type=7, attr=0xf, range=[0x0000344000000000-0x0000345eff000000) (126960MB)
[    0.000000] EFI: mem376: type=7, attr=0xf, range=[0x0000348000000000-0x0000349eff000000) (126960MB)
[    0.000000] EFI: mem377: type=7, attr=0xf, range=[0x000034c000000000-0x000034deff000000) (126960MB)
[    0.000000] EFI: mem378: type=7, attr=0xf, range=[0x0000350000000000-0x0000351eff000000) (126960MB)
[    0.000000] EFI: mem379: type=7, attr=0xf, range=[0x0000354000000000-0x0000355eff000000) (126960MB)
[    0.000000] EFI: mem380: type=7, attr=0xf, range=[0x0000358000000000-0x0000359eff000000) (126960MB)
[    0.000000] EFI: mem381: type=7, attr=0xf, range=[0x000035c000000000-0x000035deff000000) (126960MB)
[    0.000000] EFI: mem382: type=7, attr=0xf, range=[0x0000360000000000-0x0000361eff000000) (126960MB)
[    0.000000] EFI: mem383: type=7, attr=0xf, range=[0x0000364000000000-0x0000365eff000000) (126960MB)
[    0.000000] EFI: mem384: type=7, attr=0xf, range=[0x0000368000000000-0x0000369eff000000) (126960MB)
[    0.000000] EFI: mem385: type=7, attr=0xf, range=[0x0000370000000000-0x0000371eff000000) (126960MB)
[    0.000000] EFI: mem386: type=7, attr=0xf, range=[0x0000378000000000-0x0000379eff000000) (126960MB)
[    0.000000] EFI: mem387: type=7, attr=0xf, range=[0x000037c000000000-0x000037deff000000) (126960MB)
[    0.000000] EFI: mem388: type=7, attr=0xf, range=[0x0000380000000000-0x0000381eff000000) (126960MB)
[    0.000000] EFI: mem389: type=7, attr=0xf, range=[0x0000384000000000-0x0000385eff000000) (126960MB)
[    0.000000] EFI: mem390: type=7, attr=0xf, range=[0x0000388000000000-0x0000389eff000000) (126960MB)
[    0.000000] EFI: mem391: type=7, attr=0xf, range=[0x000038c000000000-0x000038deff000000) (126960MB)
[    0.000000] EFI: mem392: type=7, attr=0xf, range=[0x0000390000000000-0x0000391eff000000) (126960MB)
[    0.000000] EFI: mem393: type=7, attr=0xf, range=[0x0000394000000000-0x0000395eff000000) (126960MB)
[    0.000000] EFI: mem394: type=7, attr=0xf, range=[0x0000398000000000-0x0000399eff000000) (126960MB)
[    0.000000] EFI: mem395: type=7, attr=0xf, range=[0x000039c000000000-0x000039deff000000) (126960MB)
[    0.000000] EFI: mem396: type=7, attr=0xf, range=[0x00003a0000000000-0x00003a1eff000000) (126960MB)
[    0.000000] EFI: mem397: type=7, attr=0xf, range=[0x00003a4000000000-0x00003a5eff000000) (126960MB)
[    0.000000] EFI: mem398: type=7, attr=0xf, range=[0x00003a8000000000-0x00003a9eff000000) (126960MB)
[    0.000000] EFI: mem399: type=7, attr=0xf, range=[0x00003ac000000000-0x00003adeff000000) (126960MB)
[    0.000000] EFI: mem400: type=7, attr=0xf, range=[0x00003b0000000000-0x00003b1eff000000) (126960MB)
[    0.000000] EFI: mem401: type=7, attr=0xf, range=[0x00003b4000000000-0x00003b5eff000000) (126960MB)
[    0.000000] EFI: mem402: type=7, attr=0xf, range=[0x00003b8000000000-0x00003b9eff000000) (126960MB)
[    0.000000] EFI: mem403: type=7, attr=0xf, range=[0x00003bc000000000-0x00003bdeff000000) (126960MB)
[    0.000000] EFI: mem404: type=7, attr=0xf, range=[0x00003c0000000000-0x00003c1eff000000) (126960MB)
[    0.000000] EFI: mem405: type=7, attr=0xf, range=[0x00003c4000000000-0x00003c5eff000000) (126960MB)
[    0.000000] EFI: mem406: type=7, attr=0xf, range=[0x00003c8000000000-0x00003c9eff000000) (126960MB)
[    0.000000] EFI: mem407: type=7, attr=0xf, range=[0x00003cc000000000-0x00003cdeff000000) (126960MB)
[    0.000000] EFI: mem408: type=7, attr=0xf, range=[0x00003d0000000000-0x00003d1eff000000) (126960MB)
[    0.000000] EFI: mem409: type=7, attr=0xf, range=[0x00003d4000000000-0x00003d5eff000000) (126960MB)
[    0.000000] EFI: mem410: type=7, attr=0xf, range=[0x00003d8000000000-0x00003d9eff000000) (126960MB)
[    0.000000] EFI: mem411: type=7, attr=0xf, range=[0x00003dc000000000-0x00003ddeff000000) (126960MB)
[    0.000000] EFI: mem412: type=7, attr=0xf, range=[0x00003e0000000000-0x00003e1eff000000) (126960MB)
[    0.000000] EFI: mem413: type=7, attr=0xf, range=[0x00003e4000000000-0x00003e5eff000000) (126960MB)
[    0.000000] EFI: mem414: type=7, attr=0xf, range=[0x00003e8000000000-0x00003e9eff000000) (126960MB)
[    0.000000] EFI: mem415: type=7, attr=0xf, range=[0x00003ec000000000-0x00003edeff000000) (126960MB)
[    0.000000] EFI: mem416: type=7, attr=0xf, range=[0x00003f0000000000-0x00003f1eff000000) (126960MB)
[    0.000000] EFI: mem417: type=7, attr=0xf, range=[0x00003f4000000000-0x00003f5eff000000) (126960MB)
[    0.000000] EFI: mem418: type=7, attr=0xf, range=[0x00003f8000000000-0x00003f9eff000000) (126960MB)
[    0.000000] EFI: mem419: type=7, attr=0xf, range=[0x00003fc000000000-0x00003fdeff000000) (126960MB)
[    0.000000] EFI: mem420: type=11, attr=0x8000000000000000, range=[0x0000000080000000-0x0000000090000000) (256MB)
[    0.000000] EFI: mem421: type=11, attr=0x8000000000000001, range=[0x00000000f8000000-0x00000000fe000000) (96MB)
[    0.000000] EFI: mem422: type=11, attr=0x8000000000000001, range=[0x00000000fed1c000-0x00000000fed20000) (0MB)
[    0.000000] EFI: mem423: type=11, attr=0x8000000000000001, range=[0x00003ff000000000-0x00003ff003e00000) (62MB)
[    0.000000] EFI: mem424: type=11, attr=0x8000000000000001, range=[0x00003ff003f00000-0x00003ff007e00000) (63MB)
[    0.000000] EFI: mem425: type=11, attr=0x8000000000000001, range=[0x00003ff007f00000-0x00003ff008000000) (1MB)
[    0.000000] EFI: mem426: type=11, attr=0x8000000000000001, range=[0x00003ff00bf00000-0x00003ff00c000000) (1MB)
[    0.000000] EFI: mem427: type=11, attr=0x8000000000000001, range=[0x00003ff00ff00000-0x00003ff010000000) (1MB)
[    0.000000] EFI: mem428: type=11, attr=0x8000000000000001, range=[0x00003ff013f00000-0x00003ff014000000) (1MB)
[    0.000000] EFI: mem429: type=11, attr=0x8000000000000001, range=[0x00003ff017f00000-0x00003ff018000000) (1MB)
[    0.000000] EFI: mem430: type=11, attr=0x8000000000000001, range=[0x00003ff01bf00000-0x00003ff01c000000) (1MB)
[    0.000000] EFI: mem431: type=11, attr=0x8000000000000001, range=[0x00003ff01ff00000-0x00003ff020000000) (1MB)
[    0.000000] EFI: mem432: type=11, attr=0x8000000000000001, range=[0x00003ff023f00000-0x00003ff024000000) (1MB)
[    0.000000] EFI: mem433: type=11, attr=0x8000000000000001, range=[0x00003ff027f00000-0x00003ff028000000) (1MB)
[    0.000000] EFI: mem434: type=11, attr=0x8000000000000001, range=[0x00003ff02bf00000-0x00003ff02c000000) (1MB)
[    0.000000] EFI: mem435: type=11, attr=0x8000000000000001, range=[0x00003ff02ff00000-0x00003ff030000000) (1MB)
[    0.000000] EFI: mem436: type=11, attr=0x8000000000000001, range=[0x00003ff033f00000-0x00003ff034000000) (1MB)
[    0.000000] EFI: mem437: type=11, attr=0x8000000000000001, range=[0x00003ff037f00000-0x00003ff038000000) (1MB)
[    0.000000] EFI: mem438: type=11, attr=0x8000000000000001, range=[0x00003ff03bf00000-0x00003ff03c000000) (1MB)
[    0.000000] EFI: mem439: type=11, attr=0x8000000000000001, range=[0x00003ff03ff00000-0x00003ff040000000) (1MB)
[    0.000000] EFI: mem440: type=11, attr=0x8000000000000001, range=[0x00003ff043f00000-0x00003ff044000000) (1MB)
[    0.000000] EFI: mem441: type=11, attr=0x8000000000000001, range=[0x00003ff047f00000-0x00003ff048000000) (1MB)
[    0.000000] EFI: mem442: type=11, attr=0x8000000000000001, range=[0x00003ff04bf00000-0x00003ff04c000000) (1MB)
[    0.000000] EFI: mem443: type=11, attr=0x8000000000000001, range=[0x00003ff04ff00000-0x00003ff050000000) (1MB)
[    0.000000] EFI: mem444: type=11, attr=0x8000000000000001, range=[0x00003ff053f00000-0x00003ff054000000) (1MB)
[    0.000000] EFI: mem445: type=11, attr=0x8000000000000001, range=[0x00003ff057f00000-0x00003ff058000000) (1MB)
[    0.000000] EFI: mem446: type=11, attr=0x8000000000000001, range=[0x00003ff05bf00000-0x00003ff05c000000) (1MB)
[    0.000000] EFI: mem447: type=11, attr=0x8000000000000001, range=[0x00003ff05ff00000-0x00003ff060000000) (1MB)
[    0.000000] EFI: mem448: type=11, attr=0x8000000000000001, range=[0x00003ff063f00000-0x00003ff064000000) (1MB)
[    0.000000] EFI: mem449: type=11, attr=0x8000000000000001, range=[0x00003ff067f00000-0x00003ff068000000) (1MB)
[    0.000000] EFI: mem450: type=11, attr=0x8000000000000001, range=[0x00003ff06bf00000-0x00003ff06c000000) (1MB)
[    0.000000] EFI: mem451: type=11, attr=0x8000000000000001, range=[0x00003ff06ff00000-0x00003ff070000000) (1MB)
[    0.000000] EFI: mem452: type=11, attr=0x8000000000000001, range=[0x00003ff073f00000-0x00003ff074000000) (1MB)
[    0.000000] EFI: mem453: type=11, attr=0x8000000000000001, range=[0x00003ff077f00000-0x00003ff078000000) (1MB)
[    0.000000] EFI: mem454: type=11, attr=0x8000000000000001, range=[0x00003ff07bf00000-0x00003ff07c000000) (1MB)
[    0.000000] EFI: mem455: type=11, attr=0x8000000000000001, range=[0x00003ff07ff00000-0x00003ff080000000) (1MB)
[    0.000000] EFI: mem456: type=11, attr=0x8000000000000001, range=[0x00003ff083f00000-0x00003ff084000000) (1MB)
[    0.000000] EFI: mem457: type=11, attr=0x8000000000000001, range=[0x00003ff087f00000-0x00003ff088000000) (1MB)
[    0.000000] EFI: mem458: type=11, attr=0x8000000000000001, range=[0x00003ff08bf00000-0x00003ff08c000000) (1MB)
[    0.000000] EFI: mem459: type=11, attr=0x8000000000000001, range=[0x00003ff08ff00000-0x00003ff090000000) (1MB)
[    0.000000] EFI: mem460: type=11, attr=0x8000000000000001, range=[0x00003ff093f00000-0x00003ff094000000) (1MB)
[    0.000000] EFI: mem461: type=11, attr=0x8000000000000001, range=[0x00003ff097f00000-0x00003ff098000000) (1MB)
[    0.000000] EFI: mem462: type=11, attr=0x8000000000000001, range=[0x00003ff09bf00000-0x00003ff09c000000) (1MB)
[    0.000000] EFI: mem463: type=11, attr=0x8000000000000001, range=[0x00003ff09ff00000-0x00003ff0a0000000) (1MB)
[    0.000000] EFI: mem464: type=11, attr=0x8000000000000001, range=[0x00003ff0a3f00000-0x00003ff0a4000000) (1MB)
[    0.000000] EFI: mem465: type=11, attr=0x8000000000000001, range=[0x00003ff0a7f00000-0x00003ff0a8000000) (1MB)
[    0.000000] EFI: mem466: type=11, attr=0x8000000000000001, range=[0x00003ff0abf00000-0x00003ff0ac000000) (1MB)
[    0.000000] EFI: mem467: type=11, attr=0x8000000000000001, range=[0x00003ff0aff00000-0x00003ff0b0000000) (1MB)
[    0.000000] EFI: mem468: type=11, attr=0x8000000000000001, range=[0x00003ff0b3f00000-0x00003ff0b4000000) (1MB)
[    0.000000] EFI: mem469: type=11, attr=0x8000000000000001, range=[0x00003ff0b7f00000-0x00003ff0b8000000) (1MB)
[    0.000000] EFI: mem470: type=11, attr=0x8000000000000001, range=[0x00003ff0bbf00000-0x00003ff0bc000000) (1MB)
[    0.000000] EFI: mem471: type=11, attr=0x8000000000000001, range=[0x00003ff0bff00000-0x00003ff0c0000000) (1MB)
[    0.000000] EFI: mem472: type=11, attr=0x8000000000000001, range=[0x00003ff0c3f00000-0x00003ff0c4000000) (1MB)
[    0.000000] EFI: mem473: type=11, attr=0x8000000000000001, range=[0x00003ff0c7f00000-0x00003ff0c8000000) (1MB)
[    0.000000] EFI: mem474: type=11, attr=0x8000000000000001, range=[0x00003ff0cbf00000-0x00003ff0cc000000) (1MB)
[    0.000000] EFI: mem475: type=11, attr=0x8000000000000001, range=[0x00003ff0cff00000-0x00003ff0d0000000) (1MB)
[    0.000000] EFI: mem476: type=11, attr=0x8000000000000001, range=[0x00003ff0d3f00000-0x00003ff0d4000000) (1MB)
[    0.000000] EFI: mem477: type=11, attr=0x8000000000000001, range=[0x00003ff0d7f00000-0x00003ff0d8000000) (1MB)
[    0.000000] EFI: mem478: type=11, attr=0x8000000000000001, range=[0x00003ff0dbf00000-0x00003ff0dc000000) (1MB)
[    0.000000] EFI: mem479: type=11, attr=0x8000000000000001, range=[0x00003ff0dff00000-0x00003ff0e0000000) (1MB)
[    0.000000] EFI: mem480: type=11, attr=0x8000000000000001, range=[0x00003ff0e3f00000-0x00003ff0e4000000) (1MB)
[    0.000000] EFI: mem481: type=11, attr=0x8000000000000001, range=[0x00003ff0e7f00000-0x00003ff0e8000000) (1MB)
[    0.000000] EFI: mem482: type=11, attr=0x8000000000000001, range=[0x00003ff0ebf00000-0x00003ff0ec000000) (1MB)
[    0.000000] EFI: mem483: type=11, attr=0x8000000000000001, range=[0x00003ff0eff00000-0x00003ff0f0000000) (1MB)
[    0.000000] EFI: mem484: type=11, attr=0x8000000000000001, range=[0x00003ff0f3f00000-0x00003ff0f4000000) (1MB)
[    0.000000] EFI: mem485: type=11, attr=0x8000000000000001, range=[0x00003ff0f7f00000-0x00003ff0f8000000) (1MB)
[    0.000000] EFI: mem486: type=11, attr=0x8000000000000001, range=[0x00003ff0fbf00000-0x00003ff0fc000000) (1MB)
[    0.000000] EFI: mem487: type=11, attr=0x8000000000000001, range=[0x00003ff0fff00000-0x00003ff100000000) (1MB)
[    0.000000] EFI: mem488: type=11, attr=0x8000000000000001, range=[0x00003ff103f00000-0x00003ff104000000) (1MB)
[    0.000000] EFI: mem489: type=11, attr=0x8000000000000001, range=[0x00003ff107f00000-0x00003ff108000000) (1MB)
[    0.000000] EFI: mem490: type=11, attr=0x8000000000000001, range=[0x00003ff10bf00000-0x00003ff10c000000) (1MB)
[    0.000000] EFI: mem491: type=11, attr=0x8000000000000001, range=[0x00003ff10ff00000-0x00003ff110000000) (1MB)
[    0.000000] EFI: mem492: type=11, attr=0x8000000000000001, range=[0x00003ff113f00000-0x00003ff114000000) (1MB)
[    0.000000] EFI: mem493: type=11, attr=0x8000000000000001, range=[0x00003ff117f00000-0x00003ff118000000) (1MB)
[    0.000000] EFI: mem494: type=11, attr=0x8000000000000001, range=[0x00003ff11bf00000-0x00003ff11c000000) (1MB)
[    0.000000] EFI: mem495: type=11, attr=0x8000000000000001, range=[0x00003ff11ff00000-0x00003ff120000000) (1MB)
[    0.000000] EFI: mem496: type=11, attr=0x8000000000000001, range=[0x00003ff123f00000-0x00003ff124000000) (1MB)
[    0.000000] EFI: mem497: type=11, attr=0x8000000000000001, range=[0x00003ff127f00000-0x00003ff128000000) (1MB)
[    0.000000] EFI: mem498: type=11, attr=0x8000000000000001, range=[0x00003ff12bf00000-0x00003ff12c000000) (1MB)
[    0.000000] EFI: mem499: type=11, attr=0x8000000000000001, range=[0x00003ff12ff00000-0x00003ff130000000) (1MB)
[    0.000000] EFI: mem500: type=11, attr=0x8000000000000001, range=[0x00003ff133f00000-0x00003ff134000000) (1MB)
[    0.000000] EFI: mem501: type=11, attr=0x8000000000000001, range=[0x00003ff137f00000-0x00003ff138000000) (1MB)
[    0.000000] EFI: mem502: type=11, attr=0x8000000000000001, range=[0x00003ff13bf00000-0x00003ff13c000000) (1MB)
[    0.000000] EFI: mem503: type=11, attr=0x8000000000000001, range=[0x00003ff13ff00000-0x00003ff140000000) (1MB)
[    0.000000] EFI: mem504: type=11, attr=0x8000000000000001, range=[0x00003ff143f00000-0x00003ff144000000) (1MB)
[    0.000000] EFI: mem505: type=11, attr=0x8000000000000001, range=[0x00003ff147f00000-0x00003ff148000000) (1MB)
[    0.000000] EFI: mem506: type=11, attr=0x8000000000000001, range=[0x00003ff14bf00000-0x00003ff14c000000) (1MB)
[    0.000000] EFI: mem507: type=11, attr=0x8000000000000001, range=[0x00003ff14ff00000-0x00003ff150000000) (1MB)
[    0.000000] EFI: mem508: type=11, attr=0x8000000000000001, range=[0x00003ff153f00000-0x00003ff154000000) (1MB)
[    0.000000] EFI: mem509: type=11, attr=0x8000000000000001, range=[0x00003ff157f00000-0x00003ff158000000) (1MB)
[    0.000000] EFI: mem510: type=11, attr=0x8000000000000001, range=[0x00003ff15bf00000-0x00003ff15c000000) (1MB)
[    0.000000] EFI: mem511: type=11, attr=0x8000000000000001, range=[0x00003ff15ff00000-0x00003ff160000000) (1MB)
[    0.000000] EFI: mem512: type=11, attr=0x8000000000000001, range=[0x00003ff163f00000-0x00003ff164000000) (1MB)
[    0.000000] EFI: mem513: type=11, attr=0x8000000000000001, range=[0x00003ff167f00000-0x00003ff168000000) (1MB)
[    0.000000] EFI: mem514: type=11, attr=0x8000000000000001, range=[0x00003ff16bf00000-0x00003ff16c000000) (1MB)
[    0.000000] EFI: mem515: type=11, attr=0x8000000000000001, range=[0x00003ff16ff00000-0x00003ff170000000) (1MB)
[    0.000000] EFI: mem516: type=11, attr=0x8000000000000001, range=[0x00003ff173f00000-0x00003ff174000000) (1MB)
[    0.000000] EFI: mem517: type=11, attr=0x8000000000000001, range=[0x00003ff177f00000-0x00003ff178000000) (1MB)
[    0.000000] EFI: mem518: type=11, attr=0x8000000000000001, range=[0x00003ff17bf00000-0x00003ff17c000000) (1MB)
[    0.000000] EFI: mem519: type=11, attr=0x8000000000000001, range=[0x00003ff17ff00000-0x00003ff180000000) (1MB)
[    0.000000] EFI: mem520: type=11, attr=0x8000000000000001, range=[0x00003ff183f00000-0x00003ff184000000) (1MB)
[    0.000000] EFI: mem521: type=11, attr=0x8000000000000001, range=[0x00003ff187f00000-0x00003ff188000000) (1MB)
[    0.000000] EFI: mem522: type=11, attr=0x8000000000000001, range=[0x00003ff18bf00000-0x00003ff18c000000) (1MB)
[    0.000000] EFI: mem523: type=11, attr=0x8000000000000001, range=[0x00003ff18ff00000-0x00003ff190000000) (1MB)
[    0.000000] EFI: mem524: type=11, attr=0x8000000000000001, range=[0x00003ff193f00000-0x00003ff194000000) (1MB)
[    0.000000] EFI: mem525: type=11, attr=0x8000000000000001, range=[0x00003ff197f00000-0x00003ff198000000) (1MB)
[    0.000000] EFI: mem526: type=11, attr=0x8000000000000001, range=[0x00003ff19bf00000-0x00003ff19c000000) (1MB)
[    0.000000] EFI: mem527: type=11, attr=0x8000000000000001, range=[0x00003ff19ff00000-0x00003ff1a0000000) (1MB)
[    0.000000] EFI: mem528: type=11, attr=0x8000000000000001, range=[0x00003ff1a3f00000-0x00003ff1a4000000) (1MB)
[    0.000000] EFI: mem529: type=11, attr=0x8000000000000001, range=[0x00003ff1a7f00000-0x00003ff1a8000000) (1MB)
[    0.000000] EFI: mem530: type=11, attr=0x8000000000000001, range=[0x00003ff1abf00000-0x00003ff1ac000000) (1MB)
[    0.000000] EFI: mem531: type=11, attr=0x8000000000000001, range=[0x00003ff1aff00000-0x00003ff1b0000000) (1MB)
[    0.000000] EFI: mem532: type=11, attr=0x8000000000000001, range=[0x00003ff1b3f00000-0x00003ff1b4000000) (1MB)
[    0.000000] EFI: mem533: type=11, attr=0x8000000000000001, range=[0x00003ff1b7f00000-0x00003ff1b8000000) (1MB)
[    0.000000] EFI: mem534: type=11, attr=0x8000000000000001, range=[0x00003ff1bbf00000-0x00003ff1bc000000) (1MB)
[    0.000000] EFI: mem535: type=11, attr=0x8000000000000001, range=[0x00003ff1bff00000-0x00003ff1c0000000) (1MB)
[    0.000000] EFI: mem536: type=11, attr=0x8000000000000001, range=[0x00003ff1c3f00000-0x00003ff1c4000000) (1MB)
[    0.000000] EFI: mem537: type=11, attr=0x8000000000000001, range=[0x00003ff1c7f00000-0x00003ff1c8000000) (1MB)
[    0.000000] EFI: mem538: type=11, attr=0x8000000000000001, range=[0x00003ff1cbf00000-0x00003ff1cc000000) (1MB)
[    0.000000] EFI: mem539: type=11, attr=0x8000000000000001, range=[0x00003ff1cff00000-0x00003ff1d0000000) (1MB)
[    0.000000] EFI: mem540: type=11, attr=0x8000000000000001, range=[0x00003ff1d3f00000-0x00003ff1d4000000) (1MB)
[    0.000000] EFI: mem541: type=11, attr=0x8000000000000001, range=[0x00003ff1d7f00000-0x00003ff1d8000000) (1MB)
[    0.000000] EFI: mem542: type=11, attr=0x8000000000000001, range=[0x00003ff1dbf00000-0x00003ff1dc000000) (1MB)
[    0.000000] EFI: mem543: type=11, attr=0x8000000000000001, range=[0x00003ff1dff00000-0x00003ff1e0000000) (1MB)
[    0.000000] EFI: mem544: type=11, attr=0x8000000000000001, range=[0x00003ff1e3f00000-0x00003ff1e4000000) (1MB)
[    0.000000] EFI: mem545: type=11, attr=0x8000000000000001, range=[0x00003ff1e7f00000-0x00003ff1e8000000) (1MB)
[    0.000000] EFI: mem546: type=11, attr=0x8000000000000001, range=[0x00003ff1ebf00000-0x00003ff1ec000000) (1MB)
[    0.000000] EFI: mem547: type=11, attr=0x8000000000000001, range=[0x00003ff1eff00000-0x00003ff1f0000000) (1MB)
[    0.000000] EFI: mem548: type=11, attr=0x8000000000000001, range=[0x00003ff1f3f00000-0x00003ff1f4000000) (1MB)
[    0.000000] EFI: mem549: type=11, attr=0x8000000000000001, range=[0x00003ff1f7f00000-0x00003ff1f8000000) (1MB)
[    0.000000] EFI: mem550: type=11, attr=0x8000000000000001, range=[0x00003ff1fbf00000-0x00003ff1fc000000) (1MB)
[    0.000000] EFI: mem551: type=11, attr=0x8000000000000001, range=[0x00003ff1fff00000-0x00003ff200000000) (1MB)
[    0.000000] EFI: mem552: type=11, attr=0x8000000000000001, range=[0x00003ff203f00000-0x00003ff204000000) (1MB)
[    0.000000] EFI: mem553: type=11, attr=0x8000000000000001, range=[0x00003ff207f00000-0x00003ff208000000) (1MB)
[    0.000000] EFI: mem554: type=11, attr=0x8000000000000001, range=[0x00003ff20bf00000-0x00003ff20c000000) (1MB)
[    0.000000] EFI: mem555: type=11, attr=0x8000000000000001, range=[0x00003ff20ff00000-0x00003ff210000000) (1MB)
[    0.000000] EFI: mem556: type=11, attr=0x8000000000000001, range=[0x00003ff213f00000-0x00003ff214000000) (1MB)
[    0.000000] EFI: mem557: type=11, attr=0x8000000000000001, range=[0x00003ff217f00000-0x00003ff218000000) (1MB)
[    0.000000] EFI: mem558: type=11, attr=0x8000000000000001, range=[0x00003ff21bf00000-0x00003ff21c000000) (1MB)
[    0.000000] EFI: mem559: type=11, attr=0x8000000000000001, range=[0x00003ff21ff00000-0x00003ff220000000) (1MB)
[    0.000000] EFI: mem560: type=11, attr=0x8000000000000001, range=[0x00003ff223f00000-0x00003ff224000000) (1MB)
[    0.000000] EFI: mem561: type=11, attr=0x8000000000000001, range=[0x00003ff227f00000-0x00003ff228000000) (1MB)
[    0.000000] EFI: mem562: type=11, attr=0x8000000000000001, range=[0x00003ff22bf00000-0x00003ff22c000000) (1MB)
[    0.000000] EFI: mem563: type=11, attr=0x8000000000000001, range=[0x00003ff22ff00000-0x00003ff230000000) (1MB)
[    0.000000] EFI: mem564: type=11, attr=0x8000000000000001, range=[0x00003ff233f00000-0x00003ff234000000) (1MB)
[    0.000000] EFI: mem565: type=11, attr=0x8000000000000001, range=[0x00003ff237f00000-0x00003ff238000000) (1MB)
[    0.000000] EFI: mem566: type=11, attr=0x8000000000000001, range=[0x00003ff23bf00000-0x00003ff23c000000) (1MB)
[    0.000000] EFI: mem567: type=11, attr=0x8000000000000001, range=[0x00003ff23ff00000-0x00003ff240000000) (1MB)
[    0.000000] EFI: mem568: type=11, attr=0x8000000000000001, range=[0x00003ff243f00000-0x00003ff244000000) (1MB)
[    0.000000] EFI: mem569: type=11, attr=0x8000000000000001, range=[0x00003ff247f00000-0x00003ff248000000) (1MB)
[    0.000000] EFI: mem570: type=11, attr=0x8000000000000001, range=[0x00003ff24bf00000-0x00003ff24c000000) (1MB)
[    0.000000] EFI: mem571: type=11, attr=0x8000000000000001, range=[0x00003ff24ff00000-0x00003ff250000000) (1MB)
[    0.000000] EFI: mem572: type=11, attr=0x8000000000000001, range=[0x00003ff253f00000-0x00003ff254000000) (1MB)
[    0.000000] EFI: mem573: type=11, attr=0x8000000000000001, range=[0x00003ff257f00000-0x00003ff258000000) (1MB)
[    0.000000] EFI: mem574: type=11, attr=0x8000000000000001, range=[0x00003ff25bf00000-0x00003ff25c000000) (1MB)
[    0.000000] EFI: mem575: type=11, attr=0x8000000000000001, range=[0x00003ff25ff00000-0x00003ff260000000) (1MB)
[    0.000000] EFI: mem576: type=11, attr=0x8000000000000001, range=[0x00003ff263f00000-0x00003ff264000000) (1MB)
[    0.000000] EFI: mem577: type=11, attr=0x8000000000000001, range=[0x00003ff267f00000-0x00003ff268000000) (1MB)
[    0.000000] EFI: mem578: type=11, attr=0x8000000000000001, range=[0x00003ff26bf00000-0x00003ff26c000000) (1MB)
[    0.000000] EFI: mem579: type=11, attr=0x8000000000000001, range=[0x00003ff26ff00000-0x00003ff270000000) (1MB)
[    0.000000] EFI: mem580: type=11, attr=0x8000000000000001, range=[0x00003ff273f00000-0x00003ff274000000) (1MB)
[    0.000000] EFI: mem581: type=11, attr=0x8000000000000001, range=[0x00003ff277f00000-0x00003ff278000000) (1MB)
[    0.000000] EFI: mem582: type=11, attr=0x8000000000000001, range=[0x00003ff27bf00000-0x00003ff27c000000) (1MB)
[    0.000000] EFI: mem583: type=11, attr=0x8000000000000001, range=[0x00003ff27ff00000-0x00003ff280000000) (1MB)
[    0.000000] EFI: mem584: type=11, attr=0x8000000000000001, range=[0x00003ff283f00000-0x00003ff284000000) (1MB)
[    0.000000] EFI: mem585: type=11, attr=0x8000000000000001, range=[0x00003ff287f00000-0x00003ff288000000) (1MB)
[    0.000000] EFI: mem586: type=11, attr=0x8000000000000001, range=[0x00003ff28bf00000-0x00003ff28c000000) (1MB)
[    0.000000] EFI: mem587: type=11, attr=0x8000000000000001, range=[0x00003ff28ff00000-0x00003ff290000000) (1MB)
[    0.000000] EFI: mem588: type=11, attr=0x8000000000000001, range=[0x00003ff293f00000-0x00003ff294000000) (1MB)
[    0.000000] EFI: mem589: type=11, attr=0x8000000000000001, range=[0x00003ff297f00000-0x00003ff298000000) (1MB)
[    0.000000] EFI: mem590: type=11, attr=0x8000000000000001, range=[0x00003ff29bf00000-0x00003ff29c000000) (1MB)
[    0.000000] EFI: mem591: type=11, attr=0x8000000000000001, range=[0x00003ff29ff00000-0x00003ff2a0000000) (1MB)
[    0.000000] EFI: mem592: type=11, attr=0x8000000000000001, range=[0x00003ff2a3f00000-0x00003ff2a4000000) (1MB)
[    0.000000] EFI: mem593: type=11, attr=0x8000000000000001, range=[0x00003ff2a7f00000-0x00003ff2a8000000) (1MB)
[    0.000000] EFI: mem594: type=11, attr=0x8000000000000001, range=[0x00003ff2abf00000-0x00003ff2ac000000) (1MB)
[    0.000000] EFI: mem595: type=11, attr=0x8000000000000001, range=[0x00003ff2aff00000-0x00003ff2b0000000) (1MB)
[    0.000000] EFI: mem596: type=11, attr=0x8000000000000001, range=[0x00003ff2b3f00000-0x00003ff2b4000000) (1MB)
[    0.000000] EFI: mem597: type=11, attr=0x8000000000000001, range=[0x00003ff2b7f00000-0x00003ff2b8000000) (1MB)
[    0.000000] EFI: mem598: type=11, attr=0x8000000000000001, range=[0x00003ff2bbf00000-0x00003ff2bc000000) (1MB)
[    0.000000] EFI: mem599: type=11, attr=0x8000000000000001, range=[0x00003ff2bff00000-0x00003ff2c0000000) (1MB)
[    0.000000] EFI: mem600: type=11, attr=0x8000000000000001, range=[0x00003ff2c3f00000-0x00003ff2c4000000) (1MB)
[    0.000000] EFI: mem601: type=11, attr=0x8000000000000001, range=[0x00003ff2c7f00000-0x00003ff2c8000000) (1MB)
[    0.000000] EFI: mem602: type=11, attr=0x8000000000000001, range=[0x00003ff2cbf00000-0x00003ff2cc000000) (1MB)
[    0.000000] EFI: mem603: type=11, attr=0x8000000000000001, range=[0x00003ff2cff00000-0x00003ff2d0000000) (1MB)
[    0.000000] EFI: mem604: type=11, attr=0x8000000000000001, range=[0x00003ff2d3f00000-0x00003ff2d4000000) (1MB)
[    0.000000] EFI: mem605: type=11, attr=0x8000000000000001, range=[0x00003ff2d7f00000-0x00003ff2d8000000) (1MB)
[    0.000000] EFI: mem606: type=11, attr=0x8000000000000001, range=[0x00003ff2dbf00000-0x00003ff2dc000000) (1MB)
[    0.000000] EFI: mem607: type=11, attr=0x8000000000000001, range=[0x00003ff2dff00000-0x00003ff2e0000000) (1MB)
[    0.000000] EFI: mem608: type=11, attr=0x8000000000000001, range=[0x00003ff2e3f00000-0x00003ff2e4000000) (1MB)
[    0.000000] EFI: mem609: type=11, attr=0x8000000000000001, range=[0x00003ff2e7f00000-0x00003ff2e8000000) (1MB)
[    0.000000] EFI: mem610: type=11, attr=0x8000000000000001, range=[0x00003ff2ebf00000-0x00003ff2ec000000) (1MB)
[    0.000000] EFI: mem611: type=11, attr=0x8000000000000001, range=[0x00003ff2eff00000-0x00003ff2f0000000) (1MB)
[    0.000000] EFI: mem612: type=11, attr=0x8000000000000001, range=[0x00003ff2f3f00000-0x00003ff2f4000000) (1MB)
[    0.000000] EFI: mem613: type=11, attr=0x8000000000000001, range=[0x00003ff2f7f00000-0x00003ff2f8000000) (1MB)
[    0.000000] EFI: mem614: type=11, attr=0x8000000000000001, range=[0x00003ff2fbf00000-0x00003ff2fc000000) (1MB)
[    0.000000] EFI: mem615: type=11, attr=0x8000000000000001, range=[0x00003ff2fff00000-0x00003ff300000000) (1MB)
[    0.000000] EFI: mem616: type=11, attr=0x8000000000000001, range=[0x00003ff303f00000-0x00003ff304000000) (1MB)
[    0.000000] EFI: mem617: type=11, attr=0x8000000000000001, range=[0x00003ff307f00000-0x00003ff308000000) (1MB)
[    0.000000] EFI: mem618: type=11, attr=0x8000000000000001, range=[0x00003ff30bf00000-0x00003ff30c000000) (1MB)
[    0.000000] EFI: mem619: type=11, attr=0x8000000000000001, range=[0x00003ff30ff00000-0x00003ff310000000) (1MB)
[    0.000000] EFI: mem620: type=11, attr=0x8000000000000001, range=[0x00003ff313f00000-0x00003ff314000000) (1MB)
[    0.000000] EFI: mem621: type=11, attr=0x8000000000000001, range=[0x00003ff317f00000-0x00003ff318000000) (1MB)
[    0.000000] EFI: mem622: type=11, attr=0x8000000000000001, range=[0x00003ff31bf00000-0x00003ff31c000000) (1MB)
[    0.000000] EFI: mem623: type=11, attr=0x8000000000000001, range=[0x00003ff31ff00000-0x00003ff320000000) (1MB)
[    0.000000] EFI: mem624: type=11, attr=0x8000000000000001, range=[0x00003ff323f00000-0x00003ff324000000) (1MB)
[    0.000000] EFI: mem625: type=11, attr=0x8000000000000001, range=[0x00003ff327f00000-0x00003ff328000000) (1MB)
[    0.000000] EFI: mem626: type=11, attr=0x8000000000000001, range=[0x00003ff32bf00000-0x00003ff32c000000) (1MB)
[    0.000000] EFI: mem627: type=11, attr=0x8000000000000001, range=[0x00003ff32ff00000-0x00003ff330000000) (1MB)
[    0.000000] EFI: mem628: type=11, attr=0x8000000000000001, range=[0x00003ff333f00000-0x00003ff334000000) (1MB)
[    0.000000] EFI: mem629: type=11, attr=0x8000000000000001, range=[0x00003ff337f00000-0x00003ff338000000) (1MB)
[    0.000000] EFI: mem630: type=11, attr=0x8000000000000001, range=[0x00003ff33bf00000-0x00003ff33c000000) (1MB)
[    0.000000] EFI: mem631: type=11, attr=0x8000000000000001, range=[0x00003ff33ff00000-0x00003ff340000000) (1MB)
[    0.000000] EFI: mem632: type=11, attr=0x8000000000000001, range=[0x00003ff343f00000-0x00003ff344000000) (1MB)
[    0.000000] EFI: mem633: type=11, attr=0x8000000000000001, range=[0x00003ff347f00000-0x00003ff348000000) (1MB)
[    0.000000] EFI: mem634: type=11, attr=0x8000000000000001, range=[0x00003ff34bf00000-0x00003ff34c000000) (1MB)
[    0.000000] EFI: mem635: type=11, attr=0x8000000000000001, range=[0x00003ff34ff00000-0x00003ff350000000) (1MB)
[    0.000000] EFI: mem636: type=11, attr=0x8000000000000001, range=[0x00003ff353f00000-0x00003ff354000000) (1MB)
[    0.000000] EFI: mem637: type=11, attr=0x8000000000000001, range=[0x00003ff357f00000-0x00003ff358000000) (1MB)
[    0.000000] EFI: mem638: type=11, attr=0x8000000000000001, range=[0x00003ff35bf00000-0x00003ff35c000000) (1MB)
[    0.000000] EFI: mem639: type=11, attr=0x8000000000000001, range=[0x00003ff35ff00000-0x00003ff360000000) (1MB)
[    0.000000] EFI: mem640: type=11, attr=0x8000000000000001, range=[0x00003ff363f00000-0x00003ff364000000) (1MB)
[    0.000000] EFI: mem641: type=11, attr=0x8000000000000001, range=[0x00003ff367f00000-0x00003ff368000000) (1MB)
[    0.000000] EFI: mem642: type=11, attr=0x8000000000000001, range=[0x00003ff36bf00000-0x00003ff36c000000) (1MB)
[    0.000000] EFI: mem643: type=11, attr=0x8000000000000001, range=[0x00003ff36ff00000-0x00003ff370000000) (1MB)
[    0.000000] EFI: mem644: type=11, attr=0x8000000000000001, range=[0x00003ff373f00000-0x00003ff374000000) (1MB)
[    0.000000] EFI: mem645: type=11, attr=0x8000000000000001, range=[0x00003ff377f00000-0x00003ff378000000) (1MB)
[    0.000000] EFI: mem646: type=11, attr=0x8000000000000001, range=[0x00003ff37bf00000-0x00003ff37c000000) (1MB)
[    0.000000] EFI: mem647: type=11, attr=0x8000000000000001, range=[0x00003ff37ff00000-0x00003ff380000000) (1MB)
[    0.000000] EFI: mem648: type=11, attr=0x8000000000000001, range=[0x00003ff383f00000-0x00003ff384000000) (1MB)
[    0.000000] EFI: mem649: type=11, attr=0x8000000000000001, range=[0x00003ff387f00000-0x00003ff388000000) (1MB)
[    0.000000] EFI: mem650: type=11, attr=0x8000000000000001, range=[0x00003ff38bf00000-0x00003ff38c000000) (1MB)
[    0.000000] EFI: mem651: type=11, attr=0x8000000000000001, range=[0x00003ff38ff00000-0x00003ff390000000) (1MB)
[    0.000000] EFI: mem652: type=11, attr=0x8000000000000001, range=[0x00003ff393f00000-0x00003ff394000000) (1MB)
[    0.000000] EFI: mem653: type=11, attr=0x8000000000000001, range=[0x00003ff397f00000-0x00003ff398000000) (1MB)
[    0.000000] EFI: mem654: type=11, attr=0x8000000000000001, range=[0x00003ff39bf00000-0x00003ff39c000000) (1MB)
[    0.000000] EFI: mem655: type=11, attr=0x8000000000000001, range=[0x00003ff39ff00000-0x00003ff3a0000000) (1MB)
[    0.000000] EFI: mem656: type=11, attr=0x8000000000000001, range=[0x00003ff3a3f00000-0x00003ff3a4000000) (1MB)
[    0.000000] EFI: mem657: type=11, attr=0x8000000000000001, range=[0x00003ff3a7f00000-0x00003ff3a8000000) (1MB)
[    0.000000] EFI: mem658: type=11, attr=0x8000000000000001, range=[0x00003ff3abf00000-0x00003ff3ac000000) (1MB)
[    0.000000] EFI: mem659: type=11, attr=0x8000000000000001, range=[0x00003ff3aff00000-0x00003ff3b0000000) (1MB)
[    0.000000] EFI: mem660: type=11, attr=0x8000000000000001, range=[0x00003ff3b3f00000-0x00003ff3b4000000) (1MB)
[    0.000000] EFI: mem661: type=11, attr=0x8000000000000001, range=[0x00003ff3b7f00000-0x00003ff3b8000000) (1MB)
[    0.000000] EFI: mem662: type=11, attr=0x8000000000000001, range=[0x00003ff3bbf00000-0x00003ff3bc000000) (1MB)
[    0.000000] EFI: mem663: type=11, attr=0x8000000000000001, range=[0x00003ff3bff00000-0x00003ff3c0000000) (1MB)
[    0.000000] EFI: mem664: type=11, attr=0x8000000000000001, range=[0x00003ff3c3f00000-0x00003ff3c4000000) (1MB)
[    0.000000] EFI: mem665: type=11, attr=0x8000000000000001, range=[0x00003ff3c7f00000-0x00003ff3c8000000) (1MB)
[    0.000000] EFI: mem666: type=11, attr=0x8000000000000001, range=[0x00003ff3cbf00000-0x00003ff3cc000000) (1MB)
[    0.000000] EFI: mem667: type=11, attr=0x8000000000000001, range=[0x00003ff3cff00000-0x00003ff3d0000000) (1MB)
[    0.000000] EFI: mem668: type=11, attr=0x8000000000000001, range=[0x00003ff3d3f00000-0x00003ff3d4000000) (1MB)
[    0.000000] EFI: mem669: type=11, attr=0x8000000000000001, range=[0x00003ff3d7f00000-0x00003ff3d8000000) (1MB)
[    0.000000] EFI: mem670: type=11, attr=0x8000000000000001, range=[0x00003ff3dbf00000-0x00003ff3dc000000) (1MB)
[    0.000000] EFI: mem671: type=11, attr=0x8000000000000001, range=[0x00003ff3dff00000-0x00003ff3e0000000) (1MB)
[    0.000000] EFI: mem672: type=11, attr=0x8000000000000001, range=[0x00003ff3e3f00000-0x00003ff3e4000000) (1MB)
[    0.000000] EFI: mem673: type=11, attr=0x8000000000000001, range=[0x00003ff3e7f00000-0x00003ff3e8000000) (1MB)
[    0.000000] EFI: mem674: type=11, attr=0x8000000000000001, range=[0x00003ff3ebf00000-0x00003ff3ec000000) (1MB)
[    0.000000] EFI: mem675: type=11, attr=0x8000000000000001, range=[0x00003ff3eff00000-0x00003ff3f0000000) (1MB)
[    0.000000] EFI: mem676: type=11, attr=0x8000000000000001, range=[0x00003ff3f3f00000-0x00003ff3f4000000) (1MB)
[    0.000000] EFI: mem677: type=11, attr=0x8000000000000001, range=[0x00003ff3f7f00000-0x00003ff3f8000000) (1MB)
[    0.000000] EFI: mem678: type=11, attr=0x8000000000000001, range=[0x00003ff3fbf00000-0x00003ff3fc000000) (1MB)
[    0.000000] EFI: mem679: type=11, attr=0x8000000000000001, range=[0x00003ff3fff00000-0x00003ff403000000) (49MB)
[    0.000000] EFI: mem680: type=11, attr=0x8000000000000001, range=[0x00003ff404000000-0x00003ff407000000) (48MB)
[    0.000000] EFI: mem681: type=11, attr=0x8000000000000001, range=[0x00003ff408000000-0x00003ff40b000000) (48MB)
[    0.000000] EFI: mem682: type=11, attr=0x8000000000000001, range=[0x00003ff40c000000-0x00003ff40f000000) (48MB)
[    0.000000] EFI: mem683: type=11, attr=0x8000000000000001, range=[0x00003ff410000000-0x00003ff413000000) (48MB)
[    0.000000] EFI: mem684: type=11, attr=0x8000000000000001, range=[0x00003ff414000000-0x00003ff417000000) (48MB)
[    0.000000] EFI: mem685: type=11, attr=0x8000000000000001, range=[0x00003ff418000000-0x00003ff41b000000) (48MB)
[    0.000000] EFI: mem686: type=11, attr=0x8000000000000001, range=[0x00003ff41c000000-0x00003ff41f000000) (48MB)
[    0.000000] EFI: mem687: type=11, attr=0x8000000000000001, range=[0x00003ff420000000-0x00003ff423000000) (48MB)
[    0.000000] EFI: mem688: type=11, attr=0x8000000000000001, range=[0x00003ff424000000-0x00003ff427000000) (48MB)
[    0.000000] EFI: mem689: type=11, attr=0x8000000000000001, range=[0x00003ff428000000-0x00003ff42b000000) (48MB)
[    0.000000] EFI: mem690: type=11, attr=0x8000000000000001, range=[0x00003ff42c000000-0x00003ff42f000000) (48MB)
[    0.000000] EFI: mem691: type=11, attr=0x8000000000000001, range=[0x00003ff430000000-0x00003ff433000000) (48MB)
[    0.000000] EFI: mem692: type=11, attr=0x8000000000000001, range=[0x00003ff434000000-0x00003ff437000000) (48MB)
[    0.000000] EFI: mem693: type=11, attr=0x8000000000000001, range=[0x00003ff438000000-0x00003ff43b000000) (48MB)
[    0.000000] EFI: mem694: type=11, attr=0x8000000000000001, range=[0x00003ff43c000000-0x00003ff43f000000) (48MB)
[    0.000000] EFI: mem695: type=11, attr=0x8000000000000001, range=[0x00003ff440000000-0x00003ff443000000) (48MB)
[    0.000000] EFI: mem696: type=11, attr=0x8000000000000001, range=[0x00003ff444000000-0x00003ff447000000) (48MB)
[    0.000000] EFI: mem697: type=11, attr=0x8000000000000001, range=[0x00003ff448000000-0x00003ff44b000000) (48MB)
[    0.000000] EFI: mem698: type=11, attr=0x8000000000000001, range=[0x00003ff44c000000-0x00003ff44f000000) (48MB)
[    0.000000] EFI: mem699: type=11, attr=0x8000000000000001, range=[0x00003ff450000000-0x00003ff453000000) (48MB)
[    0.000000] EFI: mem700: type=11, attr=0x8000000000000001, range=[0x00003ff454000000-0x00003ff457000000) (48MB)
[    0.000000] EFI: mem701: type=11, attr=0x8000000000000001, range=[0x00003ff458000000-0x00003ff45b000000) (48MB)
[    0.000000] EFI: mem702: type=11, attr=0x8000000000000001, range=[0x00003ff45c000000-0x00003ff45f000000) (48MB)
[    0.000000] EFI: mem703: type=11, attr=0x8000000000000001, range=[0x00003ff460000000-0x00003ff463000000) (48MB)
[    0.000000] EFI: mem704: type=11, attr=0x8000000000000001, range=[0x00003ff464000000-0x00003ff467000000) (48MB)
[    0.000000] EFI: mem705: type=11, attr=0x8000000000000001, range=[0x00003ff468000000-0x00003ff46b000000) (48MB)
[    0.000000] EFI: mem706: type=11, attr=0x8000000000000001, range=[0x00003ff46c000000-0x00003ff46f000000) (48MB)
[    0.000000] EFI: mem707: type=11, attr=0x8000000000000001, range=[0x00003ff470000000-0x00003ff473000000) (48MB)
[    0.000000] EFI: mem708: type=11, attr=0x8000000000000001, range=[0x00003ff474000000-0x00003ff477000000) (48MB)
[    0.000000] EFI: mem709: type=11, attr=0x8000000000000001, range=[0x00003ff478000000-0x00003ff47b000000) (48MB)
[    0.000000] EFI: mem710: type=11, attr=0x8000000000000001, range=[0x00003ff47c000000-0x00003ff47f000000) (48MB)
[    0.000000] EFI: mem711: type=11, attr=0x8000000000000001, range=[0x00003ff480000000-0x00003ff483000000) (48MB)
[    0.000000] EFI: mem712: type=11, attr=0x8000000000000001, range=[0x00003ff484000000-0x00003ff487000000) (48MB)
[    0.000000] EFI: mem713: type=11, attr=0x8000000000000001, range=[0x00003ff488000000-0x00003ff48b000000) (48MB)
[    0.000000] EFI: mem714: type=11, attr=0x8000000000000001, range=[0x00003ff48c000000-0x00003ff48f000000) (48MB)
[    0.000000] EFI: mem715: type=11, attr=0x8000000000000001, range=[0x00003ff490000000-0x00003ff493000000) (48MB)
[    0.000000] EFI: mem716: type=11, attr=0x8000000000000001, range=[0x00003ff494000000-0x00003ff497000000) (48MB)
[    0.000000] EFI: mem717: type=11, attr=0x8000000000000001, range=[0x00003ff498000000-0x00003ff49b000000) (48MB)
[    0.000000] EFI: mem718: type=11, attr=0x8000000000000001, range=[0x00003ff49c000000-0x00003ff49f000000) (48MB)
[    0.000000] EFI: mem719: type=11, attr=0x8000000000000001, range=[0x00003ff4a0000000-0x00003ff4a3000000) (48MB)
[    0.000000] EFI: mem720: type=11, attr=0x8000000000000001, range=[0x00003ff4a4000000-0x00003ff4a7000000) (48MB)
[    0.000000] EFI: mem721: type=11, attr=0x8000000000000001, range=[0x00003ff4a8000000-0x00003ff4ab000000) (48MB)
[    0.000000] EFI: mem722: type=11, attr=0x8000000000000001, range=[0x00003ff4ac000000-0x00003ff4af000000) (48MB)
[    0.000000] EFI: mem723: type=11, attr=0x8000000000000001, range=[0x00003ff4b0000000-0x00003ff4b3000000) (48MB)
[    0.000000] EFI: mem724: type=11, attr=0x8000000000000001, range=[0x00003ff4b4000000-0x00003ff4b7000000) (48MB)
[    0.000000] EFI: mem725: type=11, attr=0x8000000000000001, range=[0x00003ff4b8000000-0x00003ff4bb000000) (48MB)
[    0.000000] EFI: mem726: type=11, attr=0x8000000000000001, range=[0x00003ff4bc000000-0x00003ff4bf000000) (48MB)
[    0.000000] EFI: mem727: type=11, attr=0x8000000000000001, range=[0x00003ff4c0000000-0x00003ff4c3000000) (48MB)
[    0.000000] EFI: mem728: type=11, attr=0x8000000000000001, range=[0x00003ff4c4000000-0x00003ff4c7000000) (48MB)
[    0.000000] EFI: mem729: type=11, attr=0x8000000000000001, range=[0x00003ff4c8000000-0x00003ff4cb000000) (48MB)
[    0.000000] EFI: mem730: type=11, attr=0x8000000000000001, range=[0x00003ff4cc000000-0x00003ff4cf000000) (48MB)
[    0.000000] EFI: mem731: type=11, attr=0x8000000000000001, range=[0x00003ff4d0000000-0x00003ff4d3000000) (48MB)
[    0.000000] EFI: mem732: type=11, attr=0x8000000000000001, range=[0x00003ff4d4000000-0x00003ff4d7000000) (48MB)
[    0.000000] EFI: mem733: type=11, attr=0x8000000000000001, range=[0x00003ff4d8000000-0x00003ff4db000000) (48MB)
[    0.000000] EFI: mem734: type=11, attr=0x8000000000000001, range=[0x00003ff4dc000000-0x00003ff4df000000) (48MB)
[    0.000000] EFI: mem735: type=11, attr=0x8000000000000001, range=[0x00003ff4e0000000-0x00003ff4e3000000) (48MB)
[    0.000000] EFI: mem736: type=11, attr=0x8000000000000001, range=[0x00003ff4e4000000-0x00003ff4e7000000) (48MB)
[    0.000000] EFI: mem737: type=11, attr=0x8000000000000001, range=[0x00003ff4e8000000-0x00003ff4eb000000) (48MB)
[    0.000000] EFI: mem738: type=11, attr=0x8000000000000001, range=[0x00003ff4ec000000-0x00003ff4ef000000) (48MB)
[    0.000000] EFI: mem739: type=11, attr=0x8000000000000001, range=[0x00003ff4f0000000-0x00003ff4f3000000) (48MB)
[    0.000000] EFI: mem740: type=11, attr=0x8000000000000001, range=[0x00003ff4f4000000-0x00003ff4f7000000) (48MB)
[    0.000000] EFI: mem741: type=11, attr=0x8000000000000001, range=[0x00003ff4f8000000-0x00003ff4fb000000) (48MB)
[    0.000000] EFI: mem742: type=11, attr=0x8000000000000001, range=[0x00003ff4fc000000-0x00003ff4ff000000) (48MB)
[    0.000000] EFI: mem743: type=11, attr=0x8000000000000001, range=[0x00003ff500000000-0x00003ff503000000) (48MB)
[    0.000000] EFI: mem744: type=11, attr=0x8000000000000001, range=[0x00003ff504000000-0x00003ff507000000) (48MB)
[    0.000000] EFI: mem745: type=11, attr=0x8000000000000001, range=[0x00003ff508000000-0x00003ff50b000000) (48MB)
[    0.000000] EFI: mem746: type=11, attr=0x8000000000000001, range=[0x00003ff50c000000-0x00003ff50f000000) (48MB)
[    0.000000] EFI: mem747: type=11, attr=0x8000000000000001, range=[0x00003ff510000000-0x00003ff513000000) (48MB)
[    0.000000] EFI: mem748: type=11, attr=0x8000000000000001, range=[0x00003ff514000000-0x00003ff517000000) (48MB)
[    0.000000] EFI: mem749: type=11, attr=0x8000000000000001, range=[0x00003ff518000000-0x00003ff51b000000) (48MB)
[    0.000000] EFI: mem750: type=11, attr=0x8000000000000001, range=[0x00003ff51c000000-0x00003ff51f000000) (48MB)
[    0.000000] EFI: mem751: type=11, attr=0x8000000000000001, range=[0x00003ff520000000-0x00003ff523000000) (48MB)
[    0.000000] EFI: mem752: type=11, attr=0x8000000000000001, range=[0x00003ff524000000-0x00003ff527000000) (48MB)
[    0.000000] EFI: mem753: type=11, attr=0x8000000000000001, range=[0x00003ff528000000-0x00003ff52b000000) (48MB)
[    0.000000] EFI: mem754: type=11, attr=0x8000000000000001, range=[0x00003ff52c000000-0x00003ff52f000000) (48MB)
[    0.000000] EFI: mem755: type=11, attr=0x8000000000000001, range=[0x00003ff530000000-0x00003ff533000000) (48MB)
[    0.000000] EFI: mem756: type=11, attr=0x8000000000000001, range=[0x00003ff534000000-0x00003ff537000000) (48MB)
[    0.000000] EFI: mem757: type=11, attr=0x8000000000000001, range=[0x00003ff538000000-0x00003ff53b000000) (48MB)
[    0.000000] EFI: mem758: type=11, attr=0x8000000000000001, range=[0x00003ff53c000000-0x00003ff53f000000) (48MB)
[    0.000000] EFI: mem759: type=11, attr=0x8000000000000001, range=[0x00003ff540000000-0x00003ff543000000) (48MB)
[    0.000000] EFI: mem760: type=11, attr=0x8000000000000001, range=[0x00003ff544000000-0x00003ff547000000) (48MB)
[    0.000000] EFI: mem761: type=11, attr=0x8000000000000001, range=[0x00003ff548000000-0x00003ff54b000000) (48MB)
[    0.000000] EFI: mem762: type=11, attr=0x8000000000000001, range=[0x00003ff54c000000-0x00003ff54f000000) (48MB)
[    0.000000] EFI: mem763: type=11, attr=0x8000000000000001, range=[0x00003ff550000000-0x00003ff553000000) (48MB)
[    0.000000] EFI: mem764: type=11, attr=0x8000000000000001, range=[0x00003ff554000000-0x00003ff557000000) (48MB)
[    0.000000] EFI: mem765: type=11, attr=0x8000000000000001, range=[0x00003ff558000000-0x00003ff55b000000) (48MB)
[    0.000000] EFI: mem766: type=11, attr=0x8000000000000001, range=[0x00003ff55c000000-0x00003ff55f000000) (48MB)
[    0.000000] EFI: mem767: type=11, attr=0x8000000000000001, range=[0x00003ff560000000-0x00003ff563000000) (48MB)
[    0.000000] EFI: mem768: type=11, attr=0x8000000000000001, range=[0x00003ff564000000-0x00003ff567000000) (48MB)
[    0.000000] EFI: mem769: type=11, attr=0x8000000000000001, range=[0x00003ff568000000-0x00003ff56b000000) (48MB)
[    0.000000] EFI: mem770: type=11, attr=0x8000000000000001, range=[0x00003ff56c000000-0x00003ff56f000000) (48MB)
[    0.000000] EFI: mem771: type=11, attr=0x8000000000000001, range=[0x00003ff570000000-0x00003ff573000000) (48MB)
[    0.000000] EFI: mem772: type=11, attr=0x8000000000000001, range=[0x00003ff574000000-0x00003ff577000000) (48MB)
[    0.000000] EFI: mem773: type=11, attr=0x8000000000000001, range=[0x00003ff578000000-0x00003ff57b000000) (48MB)
[    0.000000] EFI: mem774: type=11, attr=0x8000000000000001, range=[0x00003ff57c000000-0x00003ff57f000000) (48MB)
[    0.000000] EFI: mem775: type=11, attr=0x8000000000000001, range=[0x00003ff580000000-0x00003ff583000000) (48MB)
[    0.000000] EFI: mem776: type=11, attr=0x8000000000000001, range=[0x00003ff584000000-0x00003ff587000000) (48MB)
[    0.000000] EFI: mem777: type=11, attr=0x8000000000000001, range=[0x00003ff588000000-0x00003ff58b000000) (48MB)
[    0.000000] EFI: mem778: type=11, attr=0x8000000000000001, range=[0x00003ff58c000000-0x00003ff58f000000) (48MB)
[    0.000000] EFI: mem779: type=11, attr=0x8000000000000001, range=[0x00003ff590000000-0x00003ff593000000) (48MB)
[    0.000000] EFI: mem780: type=11, attr=0x8000000000000001, range=[0x00003ff594000000-0x00003ff597000000) (48MB)
[    0.000000] EFI: mem781: type=11, attr=0x8000000000000001, range=[0x00003ff598000000-0x00003ff59b000000) (48MB)
[    0.000000] EFI: mem782: type=11, attr=0x8000000000000001, range=[0x00003ff59c000000-0x00003ff59f000000) (48MB)
[    0.000000] EFI: mem783: type=11, attr=0x8000000000000001, range=[0x00003ff5a0000000-0x00003ff5a3000000) (48MB)
[    0.000000] EFI: mem784: type=11, attr=0x8000000000000001, range=[0x00003ff5a4000000-0x00003ff5a7000000) (48MB)
[    0.000000] EFI: mem785: type=11, attr=0x8000000000000001, range=[0x00003ff5a8000000-0x00003ff5ab000000) (48MB)
[    0.000000] EFI: mem786: type=11, attr=0x8000000000000001, range=[0x00003ff5ac000000-0x00003ff5af000000) (48MB)
[    0.000000] EFI: mem787: type=11, attr=0x8000000000000001, range=[0x00003ff5b0000000-0x00003ff5b3000000) (48MB)
[    0.000000] EFI: mem788: type=11, attr=0x8000000000000001, range=[0x00003ff5b4000000-0x00003ff5b7000000) (48MB)
[    0.000000] EFI: mem789: type=11, attr=0x8000000000000001, range=[0x00003ff5b8000000-0x00003ff5bb000000) (48MB)
[    0.000000] EFI: mem790: type=11, attr=0x8000000000000001, range=[0x00003ff5bc000000-0x00003ff5bf000000) (48MB)
[    0.000000] EFI: mem791: type=11, attr=0x8000000000000001, range=[0x00003ff5c0000000-0x00003ff5c3000000) (48MB)
[    0.000000] EFI: mem792: type=11, attr=0x8000000000000001, range=[0x00003ff5c4000000-0x00003ff5c7000000) (48MB)
[    0.000000] EFI: mem793: type=11, attr=0x8000000000000001, range=[0x00003ff5c8000000-0x00003ff5cb000000) (48MB)
[    0.000000] EFI: mem794: type=11, attr=0x8000000000000001, range=[0x00003ff5cc000000-0x00003ff5cf000000) (48MB)
[    0.000000] EFI: mem795: type=11, attr=0x8000000000000001, range=[0x00003ff5d0000000-0x00003ff5d3000000) (48MB)
[    0.000000] EFI: mem796: type=11, attr=0x8000000000000001, range=[0x00003ff5d4000000-0x00003ff5d7000000) (48MB)
[    0.000000] EFI: mem797: type=11, attr=0x8000000000000001, range=[0x00003ff5d8000000-0x00003ff5db000000) (48MB)
[    0.000000] EFI: mem798: type=11, attr=0x8000000000000001, range=[0x00003ff5dc000000-0x00003ff5df000000) (48MB)
[    0.000000] EFI: mem799: type=11, attr=0x8000000000000001, range=[0x00003ff5e0000000-0x00003ff5e3000000) (48MB)
[    0.000000] EFI: mem800: type=11, attr=0x8000000000000001, range=[0x00003ff5e4000000-0x00003ff5e7000000) (48MB)
[    0.000000] EFI: mem801: type=11, attr=0x8000000000000001, range=[0x00003ff5e8000000-0x00003ff5eb000000) (48MB)
[    0.000000] EFI: mem802: type=11, attr=0x8000000000000001, range=[0x00003ff5ec000000-0x00003ff5ef000000) (48MB)
[    0.000000] EFI: mem803: type=11, attr=0x8000000000000001, range=[0x00003ff5f0000000-0x00003ff5f3000000) (48MB)
[    0.000000] EFI: mem804: type=11, attr=0x8000000000000001, range=[0x00003ff5f4000000-0x00003ff5f7000000) (48MB)
[    0.000000] EFI: mem805: type=11, attr=0x8000000000000001, range=[0x00003ff5f8000000-0x00003ff5fb000000) (48MB)
[    0.000000] EFI: mem806: type=11, attr=0x8000000000000001, range=[0x00003ff5fc000000-0x00003ff5ff000000) (48MB)
[    0.000000] EFI: mem807: type=11, attr=0x8000000000000001, range=[0x00003ff600000000-0x00003ff603000000) (48MB)
[    0.000000] EFI: mem808: type=11, attr=0x8000000000000001, range=[0x00003ff604000000-0x00003ff607000000) (48MB)
[    0.000000] EFI: mem809: type=11, attr=0x8000000000000001, range=[0x00003ff608000000-0x00003ff60b000000) (48MB)
[    0.000000] EFI: mem810: type=11, attr=0x8000000000000001, range=[0x00003ff60c000000-0x00003ff60f000000) (48MB)
[    0.000000] EFI: mem811: type=11, attr=0x8000000000000001, range=[0x00003ff610000000-0x00003ff613000000) (48MB)
[    0.000000] EFI: mem812: type=11, attr=0x8000000000000001, range=[0x00003ff614000000-0x00003ff617000000) (48MB)
[    0.000000] EFI: mem813: type=11, attr=0x8000000000000001, range=[0x00003ff618000000-0x00003ff61b000000) (48MB)
[    0.000000] EFI: mem814: type=11, attr=0x8000000000000001, range=[0x00003ff61c000000-0x00003ff61f000000) (48MB)
[    0.000000] EFI: mem815: type=11, attr=0x8000000000000001, range=[0x00003ff620000000-0x00003ff623000000) (48MB)
[    0.000000] EFI: mem816: type=11, attr=0x8000000000000001, range=[0x00003ff624000000-0x00003ff627000000) (48MB)
[    0.000000] EFI: mem817: type=11, attr=0x8000000000000001, range=[0x00003ff628000000-0x00003ff62b000000) (48MB)
[    0.000000] EFI: mem818: type=11, attr=0x8000000000000001, range=[0x00003ff62c000000-0x00003ff62f000000) (48MB)
[    0.000000] EFI: mem819: type=11, attr=0x8000000000000001, range=[0x00003ff630000000-0x00003ff633000000) (48MB)
[    0.000000] EFI: mem820: type=11, attr=0x8000000000000001, range=[0x00003ff634000000-0x00003ff637000000) (48MB)
[    0.000000] EFI: mem821: type=11, attr=0x8000000000000001, range=[0x00003ff638000000-0x00003ff63b000000) (48MB)
[    0.000000] EFI: mem822: type=11, attr=0x8000000000000001, range=[0x00003ff63c000000-0x00003ff63f000000) (48MB)
[    0.000000] EFI: mem823: type=11, attr=0x8000000000000001, range=[0x00003ff640000000-0x00003ff643000000) (48MB)
[    0.000000] EFI: mem824: type=11, attr=0x8000000000000001, range=[0x00003ff644000000-0x00003ff647000000) (48MB)
[    0.000000] EFI: mem825: type=11, attr=0x8000000000000001, range=[0x00003ff648000000-0x00003ff64b000000) (48MB)
[    0.000000] EFI: mem826: type=11, attr=0x8000000000000001, range=[0x00003ff64c000000-0x00003ff64f000000) (48MB)
[    0.000000] EFI: mem827: type=11, attr=0x8000000000000001, range=[0x00003ff650000000-0x00003ff653000000) (48MB)
[    0.000000] EFI: mem828: type=11, attr=0x8000000000000001, range=[0x00003ff654000000-0x00003ff657000000) (48MB)
[    0.000000] EFI: mem829: type=11, attr=0x8000000000000001, range=[0x00003ff658000000-0x00003ff65b000000) (48MB)
[    0.000000] EFI: mem830: type=11, attr=0x8000000000000001, range=[0x00003ff65c000000-0x00003ff65f000000) (48MB)
[    0.000000] EFI: mem831: type=11, attr=0x8000000000000001, range=[0x00003ff660000000-0x00003ff663000000) (48MB)
[    0.000000] EFI: mem832: type=11, attr=0x8000000000000001, range=[0x00003ff664000000-0x00003ff667000000) (48MB)
[    0.000000] EFI: mem833: type=11, attr=0x8000000000000001, range=[0x00003ff668000000-0x00003ff66b000000) (48MB)
[    0.000000] EFI: mem834: type=11, attr=0x8000000000000001, range=[0x00003ff66c000000-0x00003ff66f000000) (48MB)
[    0.000000] EFI: mem835: type=11, attr=0x8000000000000001, range=[0x00003ff670000000-0x00003ff673000000) (48MB)
[    0.000000] EFI: mem836: type=11, attr=0x8000000000000001, range=[0x00003ff674000000-0x00003ff677000000) (48MB)
[    0.000000] EFI: mem837: type=11, attr=0x8000000000000001, range=[0x00003ff678000000-0x00003ff67b000000) (48MB)
[    0.000000] EFI: mem838: type=11, attr=0x8000000000000001, range=[0x00003ff67c000000-0x00003ff67f000000) (48MB)
[    0.000000] EFI: mem839: type=11, attr=0x8000000000000001, range=[0x00003ff680000000-0x00003ff683000000) (48MB)
[    0.000000] EFI: mem840: type=11, attr=0x8000000000000001, range=[0x00003ff684000000-0x00003ff687000000) (48MB)
[    0.000000] EFI: mem841: type=11, attr=0x8000000000000001, range=[0x00003ff688000000-0x00003ff68b000000) (48MB)
[    0.000000] EFI: mem842: type=11, attr=0x8000000000000001, range=[0x00003ff68c000000-0x00003ff68f000000) (48MB)
[    0.000000] EFI: mem843: type=11, attr=0x8000000000000001, range=[0x00003ff690000000-0x00003ff693000000) (48MB)
[    0.000000] EFI: mem844: type=11, attr=0x8000000000000001, range=[0x00003ff694000000-0x00003ff697000000) (48MB)
[    0.000000] EFI: mem845: type=11, attr=0x8000000000000001, range=[0x00003ff698000000-0x00003ff69b000000) (48MB)
[    0.000000] EFI: mem846: type=11, attr=0x8000000000000001, range=[0x00003ff69c000000-0x00003ff69f000000) (48MB)
[    0.000000] EFI: mem847: type=11, attr=0x8000000000000001, range=[0x00003ff6a0000000-0x00003ff6a3000000) (48MB)
[    0.000000] EFI: mem848: type=11, attr=0x8000000000000001, range=[0x00003ff6a4000000-0x00003ff6a7000000) (48MB)
[    0.000000] EFI: mem849: type=11, attr=0x8000000000000001, range=[0x00003ff6a8000000-0x00003ff6ab000000) (48MB)
[    0.000000] EFI: mem850: type=11, attr=0x8000000000000001, range=[0x00003ff6ac000000-0x00003ff6af000000) (48MB)
[    0.000000] EFI: mem851: type=11, attr=0x8000000000000001, range=[0x00003ff6b0000000-0x00003ff6b3000000) (48MB)
[    0.000000] EFI: mem852: type=11, attr=0x8000000000000001, range=[0x00003ff6b4000000-0x00003ff6b7000000) (48MB)
[    0.000000] EFI: mem853: type=11, attr=0x8000000000000001, range=[0x00003ff6b8000000-0x00003ff6bb000000) (48MB)
[    0.000000] EFI: mem854: type=11, attr=0x8000000000000001, range=[0x00003ff6bc000000-0x00003ff6bf000000) (48MB)
[    0.000000] EFI: mem855: type=11, attr=0x8000000000000001, range=[0x00003ff6c0000000-0x00003ff6c3000000) (48MB)
[    0.000000] EFI: mem856: type=11, attr=0x8000000000000001, range=[0x00003ff6c4000000-0x00003ff6c7000000) (48MB)
[    0.000000] EFI: mem857: type=11, attr=0x8000000000000001, range=[0x00003ff6c8000000-0x00003ff6cb000000) (48MB)
[    0.000000] EFI: mem858: type=11, attr=0x8000000000000001, range=[0x00003ff6cc000000-0x00003ff6cf000000) (48MB)
[    0.000000] EFI: mem859: type=11, attr=0x8000000000000001, range=[0x00003ff6d0000000-0x00003ff6d3000000) (48MB)
[    0.000000] EFI: mem860: type=11, attr=0x8000000000000001, range=[0x00003ff6d4000000-0x00003ff6d7000000) (48MB)
[    0.000000] EFI: mem861: type=11, attr=0x8000000000000001, range=[0x00003ff6d8000000-0x00003ff6db000000) (48MB)
[    0.000000] EFI: mem862: type=11, attr=0x8000000000000001, range=[0x00003ff6dc000000-0x00003ff6df000000) (48MB)
[    0.000000] EFI: mem863: type=11, attr=0x8000000000000001, range=[0x00003ff6e0000000-0x00003ff6e3000000) (48MB)
[    0.000000] EFI: mem864: type=11, attr=0x8000000000000001, range=[0x00003ff6e4000000-0x00003ff6e7000000) (48MB)
[    0.000000] EFI: mem865: type=11, attr=0x8000000000000001, range=[0x00003ff6e8000000-0x00003ff6eb000000) (48MB)
[    0.000000] EFI: mem866: type=11, attr=0x8000000000000001, range=[0x00003ff6ec000000-0x00003ff6ef000000) (48MB)
[    0.000000] EFI: mem867: type=11, attr=0x8000000000000001, range=[0x00003ff6f0000000-0x00003ff6f3000000) (48MB)
[    0.000000] EFI: mem868: type=11, attr=0x8000000000000001, range=[0x00003ff6f4000000-0x00003ff6f7000000) (48MB)
[    0.000000] EFI: mem869: type=11, attr=0x8000000000000001, range=[0x00003ff6f8000000-0x00003ff6fb000000) (48MB)
[    0.000000] EFI: mem870: type=11, attr=0x8000000000000001, range=[0x00003ff6fc000000-0x00003ff6ff000000) (48MB)
[    0.000000] EFI: mem871: type=11, attr=0x8000000000000001, range=[0x00003ff700000000-0x00003ff703000000) (48MB)
[    0.000000] EFI: mem872: type=11, attr=0x8000000000000001, range=[0x00003ff704000000-0x00003ff707000000) (48MB)
[    0.000000] EFI: mem873: type=11, attr=0x8000000000000001, range=[0x00003ff708000000-0x00003ff70b000000) (48MB)
[    0.000000] EFI: mem874: type=11, attr=0x8000000000000001, range=[0x00003ff70c000000-0x00003ff70f000000) (48MB)
[    0.000000] EFI: mem875: type=11, attr=0x8000000000000001, range=[0x00003ff710000000-0x00003ff713000000) (48MB)
[    0.000000] EFI: mem876: type=11, attr=0x8000000000000001, range=[0x00003ff714000000-0x00003ff717000000) (48MB)
[    0.000000] EFI: mem877: type=11, attr=0x8000000000000001, range=[0x00003ff718000000-0x00003ff71b000000) (48MB)
[    0.000000] EFI: mem878: type=11, attr=0x8000000000000001, range=[0x00003ff71c000000-0x00003ff71f000000) (48MB)
[    0.000000] EFI: mem879: type=11, attr=0x8000000000000001, range=[0x00003ff720000000-0x00003ff723000000) (48MB)
[    0.000000] EFI: mem880: type=11, attr=0x8000000000000001, range=[0x00003ff724000000-0x00003ff727000000) (48MB)
[    0.000000] EFI: mem881: type=11, attr=0x8000000000000001, range=[0x00003ff728000000-0x00003ff72b000000) (48MB)
[    0.000000] EFI: mem882: type=11, attr=0x8000000000000001, range=[0x00003ff72c000000-0x00003ff72f000000) (48MB)
[    0.000000] EFI: mem883: type=11, attr=0x8000000000000001, range=[0x00003ff730000000-0x00003ff733000000) (48MB)
[    0.000000] EFI: mem884: type=11, attr=0x8000000000000001, range=[0x00003ff734000000-0x00003ff737000000) (48MB)
[    0.000000] EFI: mem885: type=11, attr=0x8000000000000001, range=[0x00003ff738000000-0x00003ff73b000000) (48MB)
[    0.000000] EFI: mem886: type=11, attr=0x8000000000000001, range=[0x00003ff73c000000-0x00003ff73f000000) (48MB)
[    0.000000] EFI: mem887: type=11, attr=0x8000000000000001, range=[0x00003ff740000000-0x00003ff743000000) (48MB)
[    0.000000] EFI: mem888: type=11, attr=0x8000000000000001, range=[0x00003ff744000000-0x00003ff747000000) (48MB)
[    0.000000] EFI: mem889: type=11, attr=0x8000000000000001, range=[0x00003ff748000000-0x00003ff74b000000) (48MB)
[    0.000000] EFI: mem890: type=11, attr=0x8000000000000001, range=[0x00003ff74c000000-0x00003ff74f000000) (48MB)
[    0.000000] EFI: mem891: type=11, attr=0x8000000000000001, range=[0x00003ff750000000-0x00003ff753000000) (48MB)
[    0.000000] EFI: mem892: type=11, attr=0x8000000000000001, range=[0x00003ff754000000-0x00003ff757000000) (48MB)
[    0.000000] EFI: mem893: type=11, attr=0x8000000000000001, range=[0x00003ff758000000-0x00003ff75b000000) (48MB)
[    0.000000] EFI: mem894: type=11, attr=0x8000000000000001, range=[0x00003ff75c000000-0x00003ff75f000000) (48MB)
[    0.000000] EFI: mem895: type=11, attr=0x8000000000000001, range=[0x00003ff760000000-0x00003ff763000000) (48MB)
[    0.000000] EFI: mem896: type=11, attr=0x8000000000000001, range=[0x00003ff764000000-0x00003ff767000000) (48MB)
[    0.000000] EFI: mem897: type=11, attr=0x8000000000000001, range=[0x00003ff768000000-0x00003ff76b000000) (48MB)
[    0.000000] EFI: mem898: type=11, attr=0x8000000000000001, range=[0x00003ff76c000000-0x00003ff76f000000) (48MB)
[    0.000000] EFI: mem899: type=11, attr=0x8000000000000001, range=[0x00003ff770000000-0x00003ff773000000) (48MB)
[    0.000000] EFI: mem900: type=11, attr=0x8000000000000001, range=[0x00003ff774000000-0x00003ff777000000) (48MB)
[    0.000000] EFI: mem901: type=11, attr=0x8000000000000001, range=[0x00003ff778000000-0x00003ff77b000000) (48MB)
[    0.000000] EFI: mem902: type=11, attr=0x8000000000000001, range=[0x00003ff77c000000-0x00003ff77f000000) (48MB)
[    0.000000] EFI: mem903: type=11, attr=0x8000000000000001, range=[0x00003ff780000000-0x00003ff783000000) (48MB)
[    0.000000] EFI: mem904: type=11, attr=0x8000000000000001, range=[0x00003ff784000000-0x00003ff787000000) (48MB)
[    0.000000] EFI: mem905: type=11, attr=0x8000000000000001, range=[0x00003ff788000000-0x00003ff78b000000) (48MB)
[    0.000000] EFI: mem906: type=11, attr=0x8000000000000001, range=[0x00003ff78c000000-0x00003ff78f000000) (48MB)
[    0.000000] EFI: mem907: type=11, attr=0x8000000000000001, range=[0x00003ff790000000-0x00003ff793000000) (48MB)
[    0.000000] EFI: mem908: type=11, attr=0x8000000000000001, range=[0x00003ff794000000-0x00003ff797000000) (48MB)
[    0.000000] EFI: mem909: type=11, attr=0x8000000000000001, range=[0x00003ff798000000-0x00003ff79b000000) (48MB)
[    0.000000] EFI: mem910: type=11, attr=0x8000000000000001, range=[0x00003ff79c000000-0x00003ff79f000000) (48MB)
[    0.000000] EFI: mem911: type=11, attr=0x8000000000000001, range=[0x00003ff7a0000000-0x00003ff7a3000000) (48MB)
[    0.000000] EFI: mem912: type=11, attr=0x8000000000000001, range=[0x00003ff7a4000000-0x00003ff7a7000000) (48MB)
[    0.000000] EFI: mem913: type=11, attr=0x8000000000000001, range=[0x00003ff7a8000000-0x00003ff7ab000000) (48MB)
[    0.000000] EFI: mem914: type=11, attr=0x8000000000000001, range=[0x00003ff7ac000000-0x00003ff7af000000) (48MB)
[    0.000000] EFI: mem915: type=11, attr=0x8000000000000001, range=[0x00003ff7b0000000-0x00003ff7b3000000) (48MB)
[    0.000000] EFI: mem916: type=11, attr=0x8000000000000001, range=[0x00003ff7b4000000-0x00003ff7b7000000) (48MB)
[    0.000000] EFI: mem917: type=11, attr=0x8000000000000001, range=[0x00003ff7b8000000-0x00003ff7bb000000) (48MB)
[    0.000000] EFI: mem918: type=11, attr=0x8000000000000001, range=[0x00003ff7bc000000-0x00003ff7bf000000) (48MB)
[    0.000000] EFI: mem919: type=11, attr=0x8000000000000001, range=[0x00003ff7c0000000-0x00003ff7c3000000) (48MB)
[    0.000000] EFI: mem920: type=11, attr=0x8000000000000001, range=[0x00003ff7c4000000-0x00003ff7c7000000) (48MB)
[    0.000000] EFI: mem921: type=11, attr=0x8000000000000001, range=[0x00003ff7c8000000-0x00003ff7cb000000) (48MB)
[    0.000000] EFI: mem922: type=11, attr=0x8000000000000001, range=[0x00003ff7cc000000-0x00003ff7cf000000) (48MB)
[    0.000000] EFI: mem923: type=11, attr=0x8000000000000001, range=[0x00003ff7d0000000-0x00003ff7d3000000) (48MB)
[    0.000000] EFI: mem924: type=11, attr=0x8000000000000001, range=[0x00003ff7d4000000-0x00003ff7d7000000) (48MB)
[    0.000000] EFI: mem925: type=11, attr=0x8000000000000001, range=[0x00003ff7d8000000-0x00003ff7db000000) (48MB)
[    0.000000] EFI: mem926: type=11, attr=0x8000000000000001, range=[0x00003ff7dc000000-0x00003ff7df000000) (48MB)
[    0.000000] EFI: mem927: type=11, attr=0x8000000000000001, range=[0x00003ff7e0000000-0x00003ff7e3000000) (48MB)
[    0.000000] EFI: mem928: type=11, attr=0x8000000000000001, range=[0x00003ff7e4000000-0x00003ff7e7000000) (48MB)
[    0.000000] EFI: mem929: type=11, attr=0x8000000000000001, range=[0x00003ff7e8000000-0x00003ff7eb000000) (48MB)
[    0.000000] EFI: mem930: type=11, attr=0x8000000000000001, range=[0x00003ff7ec000000-0x00003ff7ef000000) (48MB)
[    0.000000] EFI: mem931: type=11, attr=0x8000000000000001, range=[0x00003ff7f0000000-0x00003ff7f3000000) (48MB)
[    0.000000] EFI: mem932: type=11, attr=0x8000000000000001, range=[0x00003ff7f4000000-0x00003ff7f7000000) (48MB)
[    0.000000] EFI: mem933: type=11, attr=0x8000000000000001, range=[0x00003ff7f8000000-0x00003ff7fb000000) (48MB)
[    0.000000] EFI: mem934: type=11, attr=0x8000000000000001, range=[0x00003ff7fc000000-0x00003ff7ff000000) (48MB)

Robin

Re: [RFC 0/2] Delay initializing of large sections of memory

[Mike Travis]

On 6/26/2013 11:37 PM, Yinghai Lu wrote:
> On Tue, Jun 25, 2013 at 11:58 AM, Mike Travis <travis@sgi.com> wrote:
>> experimenting as soon as I can.  Our 32TB system is being
>> brought back to 16TB (we found a number of problems as we
>> get closer and closer to the 64TB limit), but that's still
>> a significant size.
> 
> Hi, Mike,
> 
> Can you post e820 memory map on system that have 32TiB or more?
> 
> Is there one range size more than 16TiB? like [16TiB, 32TiB)...
> 
> Thanks
> 
> Yinghai
> 

Here is (was) the 32T system with 2048 cores with HT disabled:

BIOS-e820: 0000000000000000 - 000000000007f000 (usable)
BIOS-e820: 000000000007f000 - 0000000000080000 (reserved)
BIOS-e820: 0000000000080000 - 00000000000a0000 (usable)
BIOS-e820: 0000000000100000 - 000000007ad54000 (usable)
BIOS-e820: 000000007ad54000 - 000000007ad55000 (reserved)
BIOS-e820: 000000007ad55000 - 000000007ad68000 (usable)
BIOS-e820: 000000007ad68000 - 000000007afa8000 (reserved)
BIOS-e820: 000000007afa8000 - 000000007bcb7000 (usable)
BIOS-e820: 000000007bcb7000 - 000000007bdb7000 (reserved)
BIOS-e820: 000000007bdb7000 - 000000007beb7000 (unusable)
BIOS-e820: 000000007beb7000 - 000000007bfb7000 (reserved)
BIOS-e820: 000000007bfb7000 - 000000007d1b7000 (ACPI NVS)
BIOS-e820: 000000007d1b7000 - 000000007e000000 (ACPI data)
BIOS-e820: 000000007e000000 - 000000007e318000 (usable)
BIOS-e820: 000000007e318000 - 000000007ef49000 (ACPI data)
BIOS-e820: 000000007ef49000 - 000000007f000000 (usable)
BIOS-e820: 0000000100000000 - 0000001e00000000 (usable)
BIOS-e820: 0000002000000000 - 0000003dff000000 (usable)
BIOS-e820: 0000004000000000 - 0000005dff000000 (usable)
BIOS-e820: 0000006000000000 - 0000007dff000000 (usable)

....
BIOS-e820: 00000e0000000000 - 00000e1dff000000 (usable)
BIOS-e820: 00000e2000000000 - 00000e3dff000000 (usable)
BIOS-e820: 00000e4000000000 - 00000e5dff000000 (usable)
BIOS-e820: 00000e6000000000 - 00000e7dff000000 (usable)

Note that on UV, there is some memory reserved at the end
of each node that is used for the directory RAM by the
UV HUB.  That is why the ranges are not contiguous.  I'd
have to double check with the BIOS guys, but I don't believe
they would have collapsed ranges across nodes even if the
directory RAM did not exist.

-Mike

Re: [RFC 0/2] Delay initializing of large sections of memory

[Ingo Molnar]

* Yinghai Lu <yinghai@kernel.org> wrote:

> On Tue, Jun 25, 2013 at 11:17 AM, H. Peter Anvin <hpa@zytor.com> wrote:
> > On 06/25/2013 10:35 AM, Mike Travis wrote:
> 
> > However, please consider Ingo's counterproposal of doing this via the
> > buddy allocator, i.e. hugepages being broken on demand.  That is a
> > *very* powerful model, although would require more infrastructure.
> 
> Can you or Ingo elaborate more about the buddy allocator proposal?

Ok, I just wrote a fuller proposal, with cons and pros listed as well.

I think it's a winner but hey.

Thanks,

	Ingo

Re: [RFC 0/2] Delay initializing of large sections of memory

[Yinghai Lu]

On Tue, Jun 25, 2013 at 10:35 AM, Mike Travis <travis@sgi.com> wrote:
>
>
> On 6/21/2013 5:23 PM, Yinghai Lu wrote:
>> On Fri, Jun 21, 2013 at 2:30 PM, Mike Travis <travis@sgi.com> wrote:
>>> Exactly.  That's why I left both low and high memory on each node.
>>
>> looks like you assume every node have same ram, and before booting you
>> you need to know memory layout to append the boot command line.
>>
>> We have patchset that moving srat table parse early.
>> git://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git
>> for-x86-mm
>> https://git.kernel.org/cgit/linux/kernel/git/yinghai/linux-yinghai.git/log/?h=for-x86-mm
>>
>> on top that, we could make your patch pass more simple command like
>> 1/2^n of every node, and only need to pass n instead.
>
> The two params that I couldn't figure out how to provide except via kernel
> param option was the memory block size (128M or 2G) and the physical
> address space per node.  The other 3 params can be automatically
> setup by a script when the total system size is known.  As soon as we
> verify on the 32TB system and surmise what will be needed for 64TB,
> then those 3 params can probably disappear.

our "numa parsing early" patchset could provide "physical address
space per node",
also can calculate memory block size via alignment detection from numa info.

with that, user only can pass "delay_init_mem" only.

Thanks

Yinghai

Re: [RFC 0/2] Delay initializing of large sections of memory

[Mike Travis]

On 6/25/2013 11:38 AM, Yinghai Lu wrote:
> On Tue, Jun 25, 2013 at 10:35 AM, Mike Travis <travis@sgi.com> wrote:
>>
>>
>> On 6/21/2013 5:23 PM, Yinghai Lu wrote:
>>> On Fri, Jun 21, 2013 at 2:30 PM, Mike Travis <travis@sgi.com> wrote:
>>>> Exactly.  That's why I left both low and high memory on each node.
>>>
>>> looks like you assume every node have same ram, and before booting you
>>> you need to know memory layout to append the boot command line.
>>>
>>> We have patchset that moving srat table parse early.
>>> git://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git
>>> for-x86-mm
>>> https://git.kernel.org/cgit/linux/kernel/git/yinghai/linux-yinghai.git/log/?h=for-x86-mm
>>>
>>> on top that, we could make your patch pass more simple command like
>>> 1/2^n of every node, and only need to pass n instead.
>>
>> The two params that I couldn't figure out how to provide except via kernel
>> param option was the memory block size (128M or 2G) and the physical
>> address space per node.  The other 3 params can be automatically
>> setup by a script when the total system size is known.  As soon as we
>> verify on the 32TB system and surmise what will be needed for 64TB,
>> then those 3 params can probably disappear.
> 
> our "numa parsing early" patchset could provide "physical address
> space per node",
> also can calculate memory block size via alignment detection from numa info.

Thanks!  I'll try it out.
> 
> with that, user only can pass "delay_init_mem" only.
> 
> Thanks
> 
> Yinghai
> 

Re: [RFC 0/2] Delay initializing of large sections of memory

[Yinghai Lu]

On Fri, Jun 21, 2013 at 9:25 AM, Nathan Zimmer <nzimmer@sgi.com> wrote:
> This rfc patch set delays initializing large sections of memory until we have
> started cpus.  This has the effect of reducing startup times on large memory
> systems.  On 16TB it can take over an hour to boot and most of that time
> is spent initializing memory.

One hour on system with 16T ram? BIOS or OS?

I use wall clock to check bootime on one system with 3T and 16 pcie cards,
Linus only takes about 3m and 30 seconds from bootloader.

wonder if you boot delay is with so many cpu get onlined in serialized mode.

so can you try boot your system with "maxcpus=128" to get the boot time with
wall clock ?

Thanks

Yinghai

Re: [RFC 0/2] Delay initializing of large sections of memory

[Greg Kroah-Hartman]

On Fri, Jun 21, 2013 at 11:36:21AM -0700, Yinghai Lu wrote:
> On Fri, Jun 21, 2013 at 9:25 AM, Nathan Zimmer <nzimmer@sgi.com> wrote:
> > This rfc patch set delays initializing large sections of memory until we have
> > started cpus.  This has the effect of reducing startup times on large memory
> > systems.  On 16TB it can take over an hour to boot and most of that time
> > is spent initializing memory.
> 
> One hour on system with 16T ram? BIOS or OS?
> 
> I use wall clock to check bootime on one system with 3T and 16 pcie cards,
> Linus only takes about 3m and 30 seconds from bootloader.
> 
> wonder if you boot delay is with so many cpu get onlined in serialized mode.
> 
> so can you try boot your system with "maxcpus=128" to get the boot time with
> wall clock ?

Why use the "wall clock" when we have the wonderful bootchart tools and
scripts that do this all for you, and can tell you exactly what part of
the kernel is taking what time, to help with fixing issues like this?

thanks,

greg k-h

Re: [RFC 0/2] Delay initializing of large sections of memory

[Yinghai Lu]

On Fri, Jun 21, 2013 at 11:44 AM, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Fri, Jun 21, 2013 at 11:36:21AM -0700, Yinghai Lu wrote:
>> On Fri, Jun 21, 2013 at 9:25 AM, Nathan Zimmer <nzimmer@sgi.com> wrote:
>> > This rfc patch set delays initializing large sections of memory until we have
>> > started cpus.  This has the effect of reducing startup times on large memory
>> > systems.  On 16TB it can take over an hour to boot and most of that time
>> > is spent initializing memory.
>>
>> One hour on system with 16T ram? BIOS or OS?
>>
>> I use wall clock to check bootime on one system with 3T and 16 pcie cards,
>> Linus only takes about 3m and 30 seconds from bootloader.
>>
>> wonder if you boot delay is with so many cpu get onlined in serialized mode.
>>
>> so can you try boot your system with "maxcpus=128" to get the boot time with
>> wall clock ?
>
> Why use the "wall clock" when we have the wonderful bootchart tools and
> scripts that do this all for you, and can tell you exactly what part of
> the kernel is taking what time, to help with fixing issues like this?

bootchart is not completed.

printk timestamp come after mem get initialized.
....
[    0.004000] tsc: Fast TSC calibration using PIT

before that stamp are all 0.

Yinghai

Re: [RFC 0/2] Delay initializing of large sections of memory

[Mike Travis]

On 6/21/2013 12:00 PM, Yinghai Lu wrote:
> On Fri, Jun 21, 2013 at 11:44 AM, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
>> On Fri, Jun 21, 2013 at 11:36:21AM -0700, Yinghai Lu wrote:
>>> On Fri, Jun 21, 2013 at 9:25 AM, Nathan Zimmer <nzimmer@sgi.com> wrote:
>>>> This rfc patch set delays initializing large sections of memory until we have
>>>> started cpus.  This has the effect of reducing startup times on large memory
>>>> systems.  On 16TB it can take over an hour to boot and most of that time
>>>> is spent initializing memory.
>>>
>>> One hour on system with 16T ram? BIOS or OS?
>>>
>>> I use wall clock to check bootime on one system with 3T and 16 pcie cards,
>>> Linus only takes about 3m and 30 seconds from bootloader.
>>>
>>> wonder if you boot delay is with so many cpu get onlined in serialized mode.
>>>
>>> so can you try boot your system with "maxcpus=128" to get the boot time with
>>> wall clock ?
>>
>> Why use the "wall clock" when we have the wonderful bootchart tools and
>> scripts that do this all for you, and can tell you exactly what part of
>> the kernel is taking what time, to help with fixing issues like this?
> 
> bootchart is not completed.
> 
> printk timestamp come after mem get initialized.
> ....
> [    0.004000] tsc: Fast TSC calibration using PIT
> 
> before that stamp are all 0.
> 
> Yinghai
> 

On UV the system console function has an option to include
timestamps, both sequential in HH:MM:SS and deltas to 100ms.
So we get both the BIOS and system times before printk time
is active.  We also have a custom "script" command that adds
timing info.

Re: [RFC 0/2] Delay initializing of large sections of memory

[Mike Travis]

On 6/21/2013 11:36 AM, Yinghai Lu wrote:
> On Fri, Jun 21, 2013 at 9:25 AM, Nathan Zimmer <nzimmer@sgi.com> wrote:
>> This rfc patch set delays initializing large sections of memory until we have
>> started cpus.  This has the effect of reducing startup times on large memory
>> systems.  On 16TB it can take over an hour to boot and most of that time
>> is spent initializing memory.
> 
> One hour on system with 16T ram? BIOS or OS?

The BIOS is about 20 minutes *before* the 1+ hour.  (When we started this
UV project way back when, 8TB took over 4 hours before we threw in the towel.)
> 
> I use wall clock to check bootime on one system with 3T and 16 pcie cards,
> Linus only takes about 3m and 30 seconds from bootloader.

I can send some stats on where various delays are but most of it was in
memory initialization.  On average UV nodes carry 128 or 256G per node,
so 12 nodes would take about 3 or 4 minutes, perhaps more.
> 
> wonder if you boot delay is with so many cpu get onlined in serialized mode.

Nope.  If that was the case, delaying memory but initializing all the
cpus would not affect the time.
> 
> so can you try boot your system with "maxcpus=128" to get the boot time with
> wall clock ?

We could try if you really think it will provide any useful info.  Not sure
exactly how having memory on nodes with no active cpus will react.
> 
> Thanks
> 
> Yinghai
> 

Re: [RFC] Transparent on-demand memory setup initialization embedded in the (GFP) buddy allocator

[Daniel J Blueman]

On Wednesday, June 26, 2013 9:30:02 PM UTC+8, Andrew Morton wrote:
 >
 > On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar <mi...@kernel.org> wrote:
 >
 > > except that on 32 TB
 > > systems we don't spend ~2 hours initializing 8,589,934,592 page heads.
 >
 > That's about a million a second which is crazy slow - even my 
prehistoric desktop
 > is 100x faster than that.
 >
 > Where's all this time actually being spent?

The complexity of a directory-lookup architecture to make the 
(intrinsically unscalable) cache-coherency protocol scalable gives you a 
~1us roundtrip to remote NUMA nodes.

Probably a lot of time is spent in some memsets, and RMW cycles which 
are setting page bits, which are intrinsically synchronous, so the 
initialising core can't get to 12 or so outstanding memory transactions.

Since EFI memory ranges have a flag to state if they are zerod (which 
may be a fair assumption for memory on non-bootstrap processor NUMA 
nodes), we can probably collapse the RMWs to just writes.

A normal write will require a coherency cycle, then a fetch and a 
writeback when it's evicted from the cache. For this purpose, 
non-temporal writes would eliminate the cache line fetch and give a 
massive increase in bandwidth. We wouldn't even need a store-fence as 
the initialising core is the only one online.

Daniel
-- 
Daniel J Blueman
Principal Software Engineer, Numascale Asia

Re: [RFC] Transparent on-demand memory setup initialization embedded in the (GFP) buddy allocator

[Nathan Zimmer]

On 06/26/2013 10:35 PM, Daniel J Blueman wrote:
> On Wednesday, June 26, 2013 9:30:02 PM UTC+8, Andrew Morton wrote:
> >
> > On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar <mi...@kernel.org> 
> wrote:
> >
> > > except that on 32 TB
> > > systems we don't spend ~2 hours initializing 8,589,934,592 page 
> heads.
> >
> > That's about a million a second which is crazy slow - even my 
> prehistoric desktop
> > is 100x faster than that.
> >
> > Where's all this time actually being spent?
>
> The complexity of a directory-lookup architecture to make the 
> (intrinsically unscalable) cache-coherency protocol scalable gives you 
> a ~1us roundtrip to remote NUMA nodes.
>
> Probably a lot of time is spent in some memsets, and RMW cycles which 
> are setting page bits, which are intrinsically synchronous, so the 
> initialising core can't get to 12 or so outstanding memory transactions.
>
> Since EFI memory ranges have a flag to state if they are zerod (which 
> may be a fair assumption for memory on non-bootstrap processor NUMA 
> nodes), we can probably collapse the RMWs to just writes.
>
> A normal write will require a coherency cycle, then a fetch and a 
> writeback when it's evicted from the cache. For this purpose, 
> non-temporal writes would eliminate the cache line fetch and give a 
> massive increase in bandwidth. We wouldn't even need a store-fence as 
> the initialising core is the only one online.
>
> Daniel

Could you elaborate a bit more? or suggest a specific area to look at?

After some experiments with trying to just set some fields in the struct 
page directly I haven't been able to produce any improvements.  Of 
course there is lots about the area which I don't have much experience with.

Nate

Re: [RFC] Transparent on-demand memory setup initialization embedded in the (GFP) buddy allocator

[Ingo Molnar]

* Nathan Zimmer <nzimmer@sgi.com> wrote:

> On 06/26/2013 10:35 PM, Daniel J Blueman wrote:
> >On Wednesday, June 26, 2013 9:30:02 PM UTC+8, Andrew Morton wrote:
> >>
> >> On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar
> ><mi...@kernel.org> wrote:
> >>
> >> > except that on 32 TB
> >> > systems we don't spend ~2 hours initializing 8,589,934,592
> >page heads.
> >>
> >> That's about a million a second which is crazy slow - even my
> >prehistoric desktop
> >> is 100x faster than that.
> >>
> >> Where's all this time actually being spent?
> >
> > The complexity of a directory-lookup architecture to make the 
> > (intrinsically unscalable) cache-coherency protocol scalable gives you 
> > a ~1us roundtrip to remote NUMA nodes.
> >
> > Probably a lot of time is spent in some memsets, and RMW cycles which 
> > are setting page bits, which are intrinsically synchronous, so the 
> > initialising core can't get to 12 or so outstanding memory 
> > transactions.
> >
> > Since EFI memory ranges have a flag to state if they are zerod (which 
> > may be a fair assumption for memory on non-bootstrap processor NUMA 
> > nodes), we can probably collapse the RMWs to just writes.
> >
> > A normal write will require a coherency cycle, then a fetch and a 
> > writeback when it's evicted from the cache. For this purpose, 
> > non-temporal writes would eliminate the cache line fetch and give a 
> > massive increase in bandwidth. We wouldn't even need a store-fence as 
> > the initialising core is the only one online.
> 
> Could you elaborate a bit more? or suggest a specific area to look at?
> 
> After some experiments with trying to just set some fields in the struct 
> page directly I haven't been able to produce any improvements.  Of 
> course there is lots about the area which I don't have much experience 
> with.

Any such improvement will at most be in the 10-20% range.

I'd suggest first concentrating on the 1000-fold boot time initialization 
speedup that the buddy allocator delayed initialization can offer, and 
speeding up whatever remains after that stage - in a much more 
development-friendly environment. (You'll be able to run 'perf record 
./calloc-1TB' after bootup and get meaningful results, etc.)

Thanks,

	Ingo

Re: [RFC] Transparent on-demand memory setup initialization embedded in the (GFP) buddy allocator

[Nathan Zimmer]

On Sat, Jun 29, 2013 at 09:24:41AM +0200, Ingo Molnar wrote:
> 
> * Nathan Zimmer <nzimmer@sgi.com> wrote:
> 
> > On 06/26/2013 10:35 PM, Daniel J Blueman wrote:
> > >On Wednesday, June 26, 2013 9:30:02 PM UTC+8, Andrew Morton wrote:
> > >>
> > >> On Wed, 26 Jun 2013 11:22:48 +0200 Ingo Molnar
> > ><mi...@kernel.org> wrote:
> > >>
> > >> > except that on 32 TB
> > >> > systems we don't spend ~2 hours initializing 8,589,934,592
> > >page heads.
> > >>
> > >> That's about a million a second which is crazy slow - even my
> > >prehistoric desktop
> > >> is 100x faster than that.
> > >>
> > >> Where's all this time actually being spent?
> > >
> > > The complexity of a directory-lookup architecture to make the 
> > > (intrinsically unscalable) cache-coherency protocol scalable gives you 
> > > a ~1us roundtrip to remote NUMA nodes.
> > >
> > > Probably a lot of time is spent in some memsets, and RMW cycles which 
> > > are setting page bits, which are intrinsically synchronous, so the 
> > > initialising core can't get to 12 or so outstanding memory 
> > > transactions.
> > >
> > > Since EFI memory ranges have a flag to state if they are zerod (which 
> > > may be a fair assumption for memory on non-bootstrap processor NUMA 
> > > nodes), we can probably collapse the RMWs to just writes.
> > >
> > > A normal write will require a coherency cycle, then a fetch and a 
> > > writeback when it's evicted from the cache. For this purpose, 
> > > non-temporal writes would eliminate the cache line fetch and give a 
> > > massive increase in bandwidth. We wouldn't even need a store-fence as 
> > > the initialising core is the only one online.
> > 
> > Could you elaborate a bit more? or suggest a specific area to look at?
> > 
> > After some experiments with trying to just set some fields in the struct 
> > page directly I haven't been able to produce any improvements.  Of 
> > course there is lots about the area which I don't have much experience 
> > with.
> 
> Any such improvement will at most be in the 10-20% range.
> 
> I'd suggest first concentrating on the 1000-fold boot time initialization 
> speedup that the buddy allocator delayed initialization can offer, and 
> speeding up whatever remains after that stage - in a much more 
> development-friendly environment. (You'll be able to run 'perf record 
> ./calloc-1TB' after bootup and get meaningful results, etc.)
> 
> Thanks,
> 
> 	Ingo

I had been focusing on the bigger gains but my attention had been diverted by
hope of an easy, alibiet smaller, win.


I have been experimenting with the patch proper, I am just doing 2MB pages for
the moment.  The improvement is vast,  I'll worry about proper numbers once I
think I have a fully working patch.

Some progress is being made on the real patch.  I think the memory is
being set up correctly, On aligned pages setting the up the page as normal
plus setting new PG_ flag. 

Right now I am trying to sort out free_pages_prepare and free_pages_check.

Thanks,
Nate