Re: [PATCH v4 4/4] Use 2GB memory block size on large-memory x86-64 systems

From: Yinghai Lu <yinghai@kernel.org>
To: "Luck, Tony" <tony.luck@intel.com>
Cc: Daniel J Blueman <daniel@numascale.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@redhat.com>, "H. Peter Anvin" <hpa@zytor.com>,
	Bjorn Helgaas <bhelgaas@google.com>,
	"the arch/x86 maintainers" <x86@kernel.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	Steffen Persvold <sp@numascale.com>
Subject: Re: [PATCH v4 4/4] Use 2GB memory block size on large-memory x86-64 systems
Date: Fri, 21 Aug 2015 11:38:13 -0700	[thread overview]
Message-ID: <CAE9FiQXWXPexLM8xma8Cb9R9hpR2+tOwzGtmJxzH5uuyPraDpQ@mail.gmail.com> (raw)
In-Reply-To: <20150821181910.GA31378@agluck-desk.sc.intel.com>

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]

On Fri, Aug 21, 2015 at 11:19 AM, Luck, Tony <tony.luck@intel.com> wrote:
> On Tue, Nov 04, 2014 at 04:29:44PM +0800, Daniel J Blueman wrote:
>> On large-memory x86-64 systems of 64GB or more with memory hot-plug
>> enabled, use a 2GB memory block size. Eg with 64GB memory, this reduces
>> the number of directories in /sys/devices/system/memory from 512 to 32,
>> making it more manageable, and reducing the creation time accordingly.
>>
>> This caveat is that the memory can't be offlined (for hotplug or otherwise)
>> with finer 128MB granularity, but this is unimportant due to the high
>> memory densities generally used with such large-memory systems, where
>> eg a single DIMM is the order of 16GB.
>
> git bisect points to this commit as the cause of a panic on my
> machine:
>
> [    4.518415] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
> [    4.525882] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
> [    4.536280] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820
> [    4.544344] PCI: Using configuration type 1 for base access
> [    4.550778] BUG: unable to handle kernel paging request at ffffea0078000020
> [    4.558572] IP: [<ffffffff8142ab0d>] register_mem_sect_under_
...
> so the older code will look at max_pfn and set memory block size:
>
> [    3.021752] memory block size : 256MB
>
> I think the problem is more connected to the strange max_pfn rather
> than the holes ... but will defer to wiser heads.
>
> If the problem is with max_pfn ... I don't think it is a safe assumption
> that systems with >64GB memory will have 2GB aligned max_pfn.

That commit could be reverted.
According to
https://lkml.org/lkml/2014/11/10/123

I had attached patch for my test setups for a while.

Yinghai

[-- Attachment #2: revert_commit_bdee237.patch --]
[-- Type: text/x-patch, Size: 1521 bytes --]

Subject: [PATCH] x86, mm: put memory block size probing back

commit bdee237c0343 ("x86: mm: Use 2GB memory block size on large-memory x86-64 systems")
let system with more than 64GiB ram just use 2G as memory block
size without probing.

found one system: has memory map like:
[0x00000000-0x60000000)
[0x100000000-0x20a0000000)

We should use 0x20000000 in this case. So can not assume system with big
memory have 2g tail anymore.

So revert it to put probing back.

Fixes: bdee237c0343 ("x86: mm: Use 2GB memory block size on large-memory x86-64 systems")
Signed-off-by: Yinghai Lu <yinghai@kernel.org>

---
 arch/x86/mm/init_64.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

Index: linux-2.6/arch/x86/mm/init_64.c
===================================================================

--- linux-2.6.orig/arch/x86/mm/init_64.c
+++ linux-2.6/arch/x86/mm/init_64.c
@@ -52,6 +52,7 @@
 #include <asm/numa.h>
 #include <asm/cacheflush.h>
 #include <asm/init.h>
+#include <asm/uv/uv.h>
 #include <asm/setup.h>
 
 #include "mm_internal.h"
@@ -1204,10 +1205,12 @@ static unsigned long probe_memory_block_
 	/* start from 2g */
 	unsigned long bz = 1UL<<31;
 
-	if (totalram_pages >= (64ULL << (30 - PAGE_SHIFT))) {
-		pr_info("Using 2GB memory block size for large-memory system\n");
+#ifdef CONFIG_X86_UV
+	if (is_uv_system()) {
+		printk(KERN_INFO "UV: memory block size 2GB\n");
 		return 2UL * 1024 * 1024 * 1024;
 	}
+#endif
 
 	/* less than 64g installed */
 	if ((max_pfn << PAGE_SHIFT) < (16UL << 32))
