[BUG] 2.6.26-rc1 lost half the RAM on UltraSPARC 5

[BUG] 2.6.26-rc1 lost half the RAM on UltraSPARC 5

[Mikael Pettersson]

Now that 2.6.26-rc1 boots on my Ultra5, I noticed that it
reports having only 128MB RAM, while earlier kernels reported
the correct amount: 256MB.

A diff of the dmesg output from 2.6.25 and 2.6.26-rc1 shows:

--- dmesg-2.6.25	2008-05-07 19:41:26.000000000 +0200
+++ dmesg-2.6.26-rc1	2008-05-07 19:41:26.000000000 +0200
@@ -1,24 +1,36 @@
 PROMLIB: Sun IEEE Boot Prom 'OBP 3.25.3 2000/06/29 14:12'
 PROMLIB: Root node compatible: 
-Linux version 2.6.25 (mikpe@sparge) (gcc version 4.2.3) #1 Thu Apr 17 19:53:16 CEST 2008
+Linux version 2.6.26-rc1 (mikpe@sparge) (gcc version 4.2.3) #1 Wed May 7 17:41:33 CEST 2008
 console [earlyprom0] enabled
 ARCH: SUN4U
 Ethernet address: 08:00:20:fd:ec:1f
 Kernel: Using 1 locked TLB entries for main kernel image.
 Remapping the kernel... done.
-[0000000200000000-fffff80000400000] page_structs=262144 node=0 entry=0/0
-[0000000200000000-fffff80000800000] page_structs=262144 node=0 entry=1/0
-[0000000200000000-fffff80000c00000] page_structs=262144 node=0 entry=2/0
-[0000000200000000-fffff80001000000] page_structs=262144 node=0 entry=3/0
 OF stdout device is: /pci@1f,0/pci@1,1/SUNW,m64B@2
-PROM: Built device tree with 46848 bytes of memory.
-On node 0 totalpages: 32298
+PROM: Built device tree with 46641 bytes of memory.
+Top of RAM: 0x17f46000, Total RAM: 0xff40000
+Memory hole size: 128MB
+Entering add_active_range(0, 0, 16384) 0 entries of 256 used
+Entering add_active_range(0, 32768, 49023) 1 entries of 256 used
+Entering add_active_range(0, 49024, 49053) 2 entries of 256 used
+Entering add_active_range(0, 49055, 49059) 3 entries of 256 used
+[0000000200000000-fffff80000400000] page_structs=131072 node=0 entry=0/0
+[0000000200000000-fffff80000800000] page_structs=131072 node=0 entry=1/0
+Zone PFN ranges:
+  Normal          0 ->    49059
+Movable zone start PFN for each node
+early_node_map[4] active PFN ranges
+    0:        0 ->    16384
+    0:    32768 ->    49023
+    0:    49024 ->    49053
+    0:    49055 ->    49059
+On node 0 totalpages: 32672
   Normal zone: 335 pages used for memmap
   Normal zone: 0 pages reserved
-  Normal zone: 31963 pages, LIFO batch:7
+  Normal zone: 32337 pages, LIFO batch:7
   Movable zone: 0 pages used for memmap
 Booting Linux...
-Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 31963
+Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32337
 Kernel command line: ro root=/dev/sda5
 PID hash table entries: 1024 (order: 10, 8192 bytes)
 clocksource: mult[28000] shift[16]
@@ -27,11 +39,11 @@
 console handover: boot [earlyprom0] -> real [tty0]
 Dentry cache hash table entries: 32768 (order: 5, 262144 bytes)
 Inode-cache hash table entries: 16384 (order: 4, 131072 bytes)
-Memory: 237256k available (1928k kernel code, 744k data, 144k init) [fffff80000000000,0000000017f46000]
-SLUB: Genslabs=13, HWalign=32, Order=0-2, MinObjects=8, CPUs=1, Nodes=1
-Calibrating delay using timer specific routine.. 800.81 BogoMIPS (lpj=4004065)
+Memory: 127016k available (1920k kernel code, 744k data, 152k init) [fffff80000000000,0000000017f46000]
+SLUB: Genslabs=13, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
+Calibrating delay using timer specific routine.. 800.99 BogoMIPS (lpj=4004959)
 Mount-cache hash table entries: 512
-net_namespace: 280 bytes
+net_namespace: 344 bytes
 NET: Registered protocol family 16
 PCI: Probing for controllers.
 /pci@1f,0: SABRE PCI Bus Module

This was with my 2.6.25 .config + make oldconfig. I also built
2.6.26-rc1 with arch/sparc64/defconfig + minor tweaks to suit
my machine, but that one also only found 128MB RAM.

dmesg logs and .configs are in <http://user.it.uu.se/~mikpe/linux/sparge/>

/Mikael

Re: [BUG] 2.6.26-rc1 lost half the RAM on UltraSPARC 5

[David Miller]

From: Mikael Pettersson <mikpe@it.uu.se>
Date: Wed, 7 May 2008 20:36:55 +0200

> Now that 2.6.26-rc1 boots on my Ultra5, I noticed that it
> reports having only 128MB RAM, while earlier kernels reported
> the correct amount: 256MB.
> 
> A diff of the dmesg output from 2.6.25 and 2.6.26-rc1 shows:

Thanks for the report, I'll try to track this one down.

I have a machine configured similarly to your's, so this ought
to not be too difficult to fix... I hope. :-)

Re: [BUG] 2.6.26-rc1 lost half the RAM on UltraSPARC 5

[David Miller]

From: Mikael Pettersson <mikpe@it.uu.se>
Date: Wed, 7 May 2008 20:36:55 +0200

> Now that 2.6.26-rc1 boots on my Ultra5, I noticed that it
> reports having only 128MB RAM, while earlier kernels reported
> the correct amount: 256MB.
> 
> A diff of the dmesg output from 2.6.25 and 2.6.26-rc1 shows:

Try as I might I couldn't reproduce this, although I did find another
bug along the way.

But that's OK, we'll add some debugging and fetch the necessary
information from your machine.

The good news is that the early bootup does see all 256MB of your
memory:

Top of RAM: 0x17f46000, Total RAM: 0xff40000
Memory hole size: 128MB
Entering add_active_range(0, 0, 16384) 0 entries of 256 used
Entering add_active_range(0, 32768, 49023) 1 entries of 256 used
Entering add_active_range(0, 49024, 49053) 2 entries of 256 used
Entering add_active_range(0, 49055, 49059) 3 entries of 256 used

That "0xff40000" value is 267649024 decimal, and the size of the page
ranges registered next match up.

And yet we get:

Memory: 127016k available (1920k kernel code, 744k data, 152k init) [fffff80000000000,0000000017f46000]

which is strange.

Between these two events there is only a handfull of bootmem
allocations, which together should not total 128MB on your
machine. :-)

We have some existing debugging, which I'd like you to enable on the
boot command line.  Simply add "numa=debug" and that'll get some more
vebose information.

Please also add the debugging patch below.

Thanks!

diff --git a/lib/lmb.c b/lib/lmb.c
index 83287d3..3f55973 100644
--- a/lib/lmb.c
+++ b/lib/lmb.c
@@ -19,6 +19,8 @@
 
 struct lmb lmb;
 
+#define DEBUG
+
 void lmb_dump_all(void)
 {
 #ifdef DEBUG
@@ -29,7 +31,7 @@ void lmb_dump_all(void)
 	pr_debug("    memory.size		  = 0x%llx\n",
 	    (unsigned long long)lmb.memory.size);
 	for (i=0; i < lmb.memory.cnt ;i++) {
-		pr_debug("    memory.region[0x%x].base       = 0x%llx\n",
+		pr_debug("    memory.region[0x%lx].base       = 0x%llx\n",
 		    i, (unsigned long long)lmb.memory.region[i].base);
 		pr_debug("		      .size     = 0x%llx\n",
 		    (unsigned long long)lmb.memory.region[i].size);
@@ -38,7 +40,7 @@ void lmb_dump_all(void)
 	pr_debug("    reserved.cnt	  = 0x%lx\n", lmb.reserved.cnt);
 	pr_debug("    reserved.size	  = 0x%lx\n", lmb.reserved.size);
 	for (i=0; i < lmb.reserved.cnt ;i++) {
-		pr_debug("    reserved.region[0x%x].base       = 0x%llx\n",
+		pr_debug("    reserved.region[0x%lx].base       = 0x%llx\n",
 		    i, (unsigned long long)lmb.reserved.region[i].base);
 		pr_debug("		      .size     = 0x%llx\n",
 		    (unsigned long long)lmb.reserved.region[i].size);
diff --git a/mm/bootmem.c b/mm/bootmem.c
index e8fb927..78e6216 100644
--- a/mm/bootmem.c
+++ b/mm/bootmem.c
@@ -242,6 +242,10 @@ __alloc_bootmem_core(struct bootmem_data *bdata, unsigned long size,
 	unsigned long node_boot_start;
 	void *node_bootmem_map;
 
+#if 1
+	printk(KERN_ERR "__alloc_bootmem_core: size(%lu) align(%lu) goal[0x%lx] limit[0x%lx]\n",
+	       size, align, goal, limit);
+#endif
 	if (!size) {
 		printk("__alloc_bootmem_core(): zero-sized request\n");
 		BUG();
@@ -369,6 +373,17 @@ found:
 	return ret;
 }
 
+static void report_range(unsigned long *start, unsigned long pfn)
+{
+	if (*start == ~0UL)
+		return;
+
+	printk(KERN_ERR "free_all_bootmem_core: Freed pfn range [0x%lx --> 0x%lx]\n",
+	       *start, pfn);
+
+	*start = ~0UL;
+}
+
 static unsigned long __init free_all_bootmem_core(pg_data_t *pgdat)
 {
 	struct page *page;
@@ -377,6 +392,7 @@ static unsigned long __init free_all_bootmem_core(pg_data_t *pgdat)
 	unsigned long i, count, total = 0;
 	unsigned long idx;
 	unsigned long *map; 
+	unsigned long debug_start_pfn = ~0UL;
 	int gofast = 0;
 
 	BUG_ON(!bdata->node_bootmem_map);
@@ -390,12 +406,17 @@ static unsigned long __init free_all_bootmem_core(pg_data_t *pgdat)
 	if (bdata->node_boot_start == 0 ||
 	    ffs(bdata->node_boot_start) - PAGE_SHIFT > ffs(BITS_PER_LONG))
 		gofast = 1;
+	printk(KERN_ERR "free_all_bootmem_core: pfn[0x%lx] idx[0x%lx] gofast(%d) "
+	       "node_boot_start[0x%lx]\n",
+	       pfn, idx, gofast, bdata->node_boot_start);
 	for (i = 0; i < idx; ) {
 		unsigned long v = ~map[i / BITS_PER_LONG];
 
 		if (gofast && v == ~0UL) {
 			int order;
 
+			if (debug_start_pfn == ~0UL)
+				debug_start_pfn = pfn;
 			page = pfn_to_page(pfn);
 			count += BITS_PER_LONG;
 			order = ffs(BITS_PER_LONG) - 1;
@@ -403,20 +424,27 @@ static unsigned long __init free_all_bootmem_core(pg_data_t *pgdat)
 			i += BITS_PER_LONG;
 			page += BITS_PER_LONG;
 		} else if (v) {
-			unsigned long m;
+			unsigned long m, debug_index;
 
 			page = pfn_to_page(pfn);
-			for (m = 1; m && i < idx; m<<=1, page++, i++) {
+			debug_index = 0;
+			for (m = 1; m && i < idx; m<<=1, page++, i++, debug_index++) {
 				if (v & m) {
+					if (debug_start_pfn == ~0UL)
+						debug_start_pfn = pfn + debug_index;
 					count++;
 					__free_pages_bootmem(page, 0);
+				} else {
+					report_range(&debug_start_pfn, pfn + debug_index);
 				}
 			}
 		} else {
+			report_range(&debug_start_pfn, pfn);
 			i += BITS_PER_LONG;
 		}
 		pfn += BITS_PER_LONG;
 	}
+	report_range(&debug_start_pfn, pfn);
 	total += count;
 
 	/*

Re: [BUG] 2.6.26-rc1 lost half the RAM on UltraSPARC 5

[Mikael Pettersson]

David Miller writes:
 > From: Mikael Pettersson <mikpe@it.uu.se>
 > Date: Wed, 7 May 2008 20:36:55 +0200
 > 
 > > Now that 2.6.26-rc1 boots on my Ultra5, I noticed that it
 > > reports having only 128MB RAM, while earlier kernels reported
 > > the correct amount: 256MB.
 > > 
 > > A diff of the dmesg output from 2.6.25 and 2.6.26-rc1 shows:
 > 
 > Try as I might I couldn't reproduce this, although I did find another
 > bug along the way.
 > 
 > But that's OK, we'll add some debugging and fetch the necessary
 > information from your machine.
 > 
 > The good news is that the early bootup does see all 256MB of your
 > memory:
 > 
 > Top of RAM: 0x17f46000, Total RAM: 0xff40000
 > Memory hole size: 128MB
 > Entering add_active_range(0, 0, 16384) 0 entries of 256 used
 > Entering add_active_range(0, 32768, 49023) 1 entries of 256 used
 > Entering add_active_range(0, 49024, 49053) 2 entries of 256 used
 > Entering add_active_range(0, 49055, 49059) 3 entries of 256 used
 > 
 > That "0xff40000" value is 267649024 decimal, and the size of the page
 > ranges registered next match up.
 > 
 > And yet we get:
 > 
 > Memory: 127016k available (1920k kernel code, 744k data, 152k init) [fffff80000000000,0000000017f46000]
 > 
 > which is strange.
 > 
 > Between these two events there is only a handfull of bootmem
 > allocations, which together should not total 128MB on your
 > machine. :-)
 > 
 > We have some existing debugging, which I'd like you to enable on the
 > boot command line.  Simply add "numa=debug" and that'll get some more
 > vebose information.
 > 
 > Please also add the debugging patch below.

Right, 2.6.26-rc2 plus your debugging patch and booted with numa=debug
prints the following:

[    0.000000] PROMLIB: Sun IEEE Boot Prom 'OBP 3.25.3 2000/06/29 14:12'
[    0.000000] PROMLIB: Root node compatible: 
[    0.000000] Linux version 2.6.26-rc2-test (mikpe@sparge) (gcc version 4.2.3) #1 Mon May 12 19:22:10 CEST 2008
[    0.000000] console [earlyprom0] enabled
[    0.000000] ARCH: SUN4U
[    0.000000] Ethernet address: 08:00:20:fd:ec:1f
[    0.000000] Found ramdisk at physical address 0x10800000, size 3683778
[    0.000000] Kernel: Using 1 locked TLB entries for main kernel image.
[    0.000000] Remapping the kernel... done.
[    0.000000] OF stdout device is: /pci@1f,0/pci@1,1/SUNW,m64B@2
[    0.000000] PROM: Built device tree with 46850 bytes of memory.
[    0.000000] bootmem_init_numa()
[    0.000000] bootmem_init_nonnuma()
[    0.000000] Top of RAM: 0x17f46000, Total RAM: 0xff40000
[    0.000000] Memory hole size: 128MB
[    0.000000] Initializing tables for non-numa.
[    0.000000] Entering add_active_range(0, 0, 16384) 0 entries of 256 used
[    0.000000] Entering add_active_range(0, 32768, 49023) 1 entries of 256 used
[    0.000000] Entering add_active_range(0, 49024, 49053) 2 entries of 256 used
[    0.000000] Entering add_active_range(0, 49055, 49059) 3 entries of 256 used
[    0.000000] bootmem_init_one_node(0)
[    0.000000]   init_bootmem_node(0, 3ffe, 0, bfa3)
[    0.000000]   free_bootmem_with_active_regions(0, bfa3)
[    0.000000]   trim_reserved_in_node(0)
[    0.000000]     reserve_range_in_node(nid[0],start[7ffc000],end[7ffe000]
[    0.000000]       MATCH reserving range [7ffc000:7ffe000]
[    0.000000]     reserve_range_in_node(nid[0],start[7fff040],end[8000000]
[    0.000000]       MATCH reserving range [7fff040:8001040]
[    0.000000]     reserve_range_in_node(nid[0],start[10000000],end[10359f20]
[    0.000000]       MATCH reserving range [10000000:1035a000]
[    0.000000]     reserve_range_in_node(nid[0],start[107e8cc0],end[213835c2]
[    0.000000]       MATCH reserving range [107e8cc0:21384cc0]
[    0.000000]   sparse_memory_present_with_active_regions(0)
[    0.000000] __alloc_bootmem_core: size(8192) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000] __alloc_bootmem_core: size(32768) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000] __alloc_bootmem_core: size(48) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000] __alloc_bootmem_core: size(4194304) align(4194304) goal[0x7ffffffffff] limit[0x0]
[    0.000000] [0000000200000000-fffff80000400000] page_structs=131072 node=0 entry=0/0
[    0.000000] __alloc_bootmem_core: size(4194304) align(4194304) goal[0x7ffffffffff] limit[0x0]
[    0.000000] [0000000200000000-fffff80000800000] page_structs=131072 node=0 entry=1/0
[    0.000000] Zone PFN ranges:
[    0.000000]   Normal          0 ->    49059
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[4] active PFN ranges
[    0.000000]     0:        0 ->    16384
[    0.000000]     0:    32768 ->    49023
[    0.000000]     0:    49024 ->    49053
[    0.000000]     0:    49055 ->    49059
[    0.000000] On node 0 totalpages: 32672
[    0.000000]   Normal zone: 335 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 32337 pages, LIFO batch:7
[    0.000000] __alloc_bootmem_core: size(4096) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000]   Movable zone: 0 pages used for memmap
[    0.000000] Booting Linux...
[    0.000000] __alloc_bootmem_core: size(29) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000] __alloc_bootmem_core: size(29) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000] __alloc_bootmem_core: size(64) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 32337
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: ro root=/dev/sda5 numa=debug
[    0.000000] __alloc_bootmem_core: size(32768) align(64) goal[0x7ffffffffff] limit[0x0]
[    0.000000] PID hash table entries: 1024 (order: 10, 8192 bytes)
[    0.000000] __alloc_bootmem_core: size(8192) align(64) goal[0x7ffffffffff] limit[0x0]
[   25.221347] clocksource: mult[28000] shift[16]
[   25.321537] clockevent: mult[66666666] shift[32]
[   25.422936] __alloc_bootmem_core: size(424) align(64) goal[0x7ffffffffff] limit[0x0]
[   25.422963] __alloc_bootmem_core: size(4000) align(64) goal[0x7ffffffffff] limit[0x0]
[   25.423055] Console: colour dummy device 80x25
[   25.910678] console handover: boot [earlyprom0] -> real [tty0]
[   26.019608] __alloc_bootmem_core: size(262144) align(64) goal[0x7ffffffffff] limit[0x0]
[   26.021182] Dentry cache hash table entries: 32768 (order: 5, 262144 bytes)
[   26.021716] __alloc_bootmem_core: size(131072) align(64) goal[0x7ffffffffff] limit[0x0]
[   26.022488] Inode-cache hash table entries: 16384 (order: 4, 131072 bytes)
[   26.022748] __alloc_bootmem_core: size(16) align(64) goal[0x7ffffffffff] limit[0x0]
[   26.026505] free_all_bootmem_core: pfn[0x0] idx[0xbfa3] gofast(1) node_boot_start[0x0]
[   26.026758] free_all_bootmem_core: Freed pfn range [0x3a --> 0x200]
[   26.031939] free_all_bootmem_core: Freed pfn range [0x600 --> 0x3ffe]
[   26.032216] free_all_bootmem_core: Freed pfn range [0x81ad --> 0x83f4]
[   26.032341] Memory: 127064k available (2032k kernel code, 800k data, 152k init) [fffff80000000000,0000000017f46000]
[   26.032566] SLUB: Genslabs=14, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=16
[   26.179691] Calibrating delay using timer specific routine.. 800.75 BogoMIPS (lpj=4003753)
[   26.180038] Mount-cache hash table entries: 512
[   26.182670] net_namespace: 344 bytes
[   26.183637] NET: Registered protocol family 16
[   26.197822] PCI: Probing for controllers.
[   26.201140] /pci@1f,0: SABRE PCI Bus Module
[   26.201192] /pci@1f,0: PCI IO[1fe02000000] MEM[1ff00000000]
[   26.201270] PCI: Scanning PBM /pci@1f,0
[   26.205349] ebus0: [auxio] [power] [SUNW,pll] [se] [su] [su] [ecpp] [fdthree] [eeprom] [flashprom] [SUNW,CS4231]
[   26.208785] power: Control reg at 1fff1724000
[   26.214386] SCSI subsystem initialized
[   26.214637] libata version 3.00 loaded.
[   26.230239] AUXIO: Found device at /pci@1f,0/pci@1,1/ebus@1/auxio@14,726000
[   26.230582] /pci@1f,0/pci@1,1/ebus@1/eeprom@14,0: Clock regs at 000001fff1000000
[   26.233300] NET: Registered protocol family 2
[   26.550041] IP route cache hash table entries: 2048 (order: 1, 16384 bytes)
[   26.551261] TCP established hash table entries: 8192 (order: 4, 131072 bytes)
[   26.551948] TCP bind hash table entries: 8192 (order: 3, 65536 bytes)
[   26.552300] TCP: Hash tables configured (established 8192 bind 8192)
[   26.552354] TCP reno registered
[   26.580007] NET: Registered protocol family 1
[   26.580627] Unpacking initramfs... done
[   27.455017] Freeing initrd memory: 3597k freed
[   27.455794] Mini RTC Driver
[   27.480248] msgmni has been set to 255 for ipc namespace 00000000006b07d0
[   27.481545] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
[   27.481631] io scheduler noop registered
[   27.481677] io scheduler anticipatory registered (default)
[   27.481725] io scheduler deadline registered
[   27.482589] io scheduler cfq registered
[   27.483143] atyfb: 3D RAGE PRO (Mach64 GP, PQFP, PCI) [0x4750 rev 0x7c]
[   27.484393] atyfb: 4M SGRAM (1:1), 14.31818 MHz XTAL, 230 MHz PLL, 100 Mhz MCLK, 100 MHz XCLK
[   27.484829] fbcon: ATY Mach64 (fb0) is primary device
[   27.562718] Console: switching to colour frame buffer device 160x64
[   27.593689] atyfb: fb0: ATY Mach64 frame buffer device on PCI
[   27.610070] /pci@1f,0/pci@1,1/ebus@1/su@14,3083f8: Keyboard port at 1fff13083f8, irq 6
[   27.610530] /pci@1f,0/pci@1,1/ebus@1/su@14,3062f8: Mouse port at 1fff13062f8, irq 7
[   27.611497] f006159c: ttyS0 at MMIO 0x1fff1400000 (irq = 5) is a SAB82532 V3.2
[   27.612311] f006159c: ttyS1 at MMIO 0x1fff1400040 (irq = 5) is a SAB82532 V3.2
[   27.613450] Driver 'sd' needs updating - please use bus_type methods
[   27.614385] scsi0 : pata_cmd64x
[   27.615098] scsi1 : pata_cmd64x
[   27.615556] ata1: PATA max MWDMA2 cmd 0x1fe02c00000 ctl 0x1fe02c00008 bmdma 0x1fe02c00020 irq 14
[   27.615910] ata2: PATA max MWDMA2 cmd 0x1fe02c00010 ctl 0x1fe02c00018 bmdma 0x1fe02c00028 irq 14
[   27.616320] pata_cmd64x: active 10 recovery 10 setup 3.
[   27.616343] pata_cmd64x: active 10 recovery 10 setup 3.
[   27.790122] ata1.00: ATA-4: ST320420A, 3.21, max UDMA/66
[   27.790346] ata1.00: 39851760 sectors, multi 0: LBA 
[   27.790603] pata_cmd64x: active 3 recovery 1 setup 1.
[   27.790631] pata_cmd64x: active 3 recovery 1 setup 1.
[   27.830069] ata1.00: configured for MWDMA2
[   27.830306] pata_cmd64x: active 10 recovery 10 setup 3.
[   27.830329] pata_cmd64x: active 10 recovery 10 setup 3.
[   28.010111] ata2.00: ATAPI: CRD-8483B, 1.00, max UDMA/33
[   28.010367] pata_cmd64x: active 3 recovery 1 setup 1.
[   28.010392] pata_cmd64x: active 3 recovery 1 setup 1.
[   28.030077] ata2.00: configured for MWDMA2
[   28.030703] scsi 0:0:0:0: Direct-Access     ATA      ST320420A        3.21 PQ: 0 ANSI: 5
[   28.031739] sd 0:0:0:0: [sda] 39851760 512-byte hardware sectors (20404 MB)
[   28.032111] sd 0:0:0:0: [sda] Write Protect is off
[   28.032317] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[   28.032479] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   28.033106] sd 0:0:0:0: [sda] 39851760 512-byte hardware sectors (20404 MB)
[   28.033465] sd 0:0:0:0: [sda] Write Protect is off
[   28.033676] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[   28.033833] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   28.034192]  sda: sda1 sda2 sda3 sda4 sda5
[   28.040646] sd 0:0:0:0: [sda] Attached SCSI disk
[   28.041747] scsi 1:0:0:0: CD-ROM            LG       CD-ROM CRD-8483B 1.00 PQ: 0 ANSI: 5
[   28.044017] mice: PS/2 mouse device common for all mice
[   28.694315] input: Sun Type 5 keyboard as /class/input/input0
[   28.721130] input: Sun Mouse as /class/input/input1
[   28.752541] TCP cubic registered
[   28.752695] NET: Registered protocol family 17
[   29.749831] kjournald starting.  Commit interval 5 seconds
[   29.757309] EXT3-fs: mounted filesystem with ordered data mode.
[   36.303502] PCI: Enabling device: (0000:01:01.1), cmd 2
[   36.303535] sunhme.c:v3.00 June 23, 2006 David S. Miller (davem@davemloft.net)
[   36.305722] eth0: HAPPY MEAL (PCI/CheerIO) 10/100BaseT Ethernet 08:00:20:fd:ec:1f
[   39.701765] EXT3 FS on sda5, internal journal
[   39.919869] kjournald starting.  Commit interval 5 seconds
[   39.920286] EXT3 FS on sda1, internal journal
[   39.920311] EXT3-fs: mounted filesystem with ordered data mode.
[   39.938232] kjournald starting.  Commit interval 5 seconds
[   39.938753] EXT3 FS on sda2, internal journal
[   39.938776] EXT3-fs: mounted filesystem with ordered data mode.
[   40.628505] Adding 1048808k swap on /dev/sda4.  Priority:-1 extents:1 across:1048808k
[   46.099830] eth0: Link is up using internal transceiver at 100Mb/s, Full Duplex.

One thing that bothers me is the "Memory hole size: 128MB" message.
It wasn't present in 2.6.25, and the amount reported is the exact
same amount of RAM I've lost. Or maybe I'm paranoid :-)

/Mikael

Re: [BUG] 2.6.26-rc1 lost half the RAM on UltraSPARC 5

[David Miller]

From: Mikael Pettersson <mikpe@it.uu.se>
Date: Mon, 12 May 2008 21:06:41 +0200

> David Miller writes:
>  > We have some existing debugging, which I'd like you to enable on the
>  > boot command line.  Simply add "numa=debug" and that'll get some more
>  > vebose information.
>  > 
>  > Please also add the debugging patch below.
> 
> Right, 2.6.26-rc2 plus your debugging patch and booted with numa=debug
> prints the following:

Thanks, I'll look over this!

> One thing that bothers me is the "Memory hole size: 128MB" message.
> It wasn't present in 2.6.25, and the amount reported is the exact
> same amount of RAM I've lost. Or maybe I'm paranoid :-)

It's completely new and harmless.  You have memory at 0->128MB and
at 256MB->384MB.  That gap from 128MB->256MB is the hole is it talking
about.

It's computed very simply, it is the difference between the highest
value physical address of available memory and the total amount of
available memory.

Don't let it scare you, it's 2GB on one of my machines :-)

Re: [BUG] 2.6.26-rc1 lost half the RAM on UltraSPARC 5

[David Miller]

From: Mikael Pettersson <mikpe@it.uu.se>
Date: Mon, 12 May 2008 21:06:41 +0200

> David Miller writes:
>  > Please also add the debugging patch below.
> 
> Right, 2.6.26-rc2 plus your debugging patch and booted with numa=debug
> prints the following:

Hmmm... my debugging patch had this:

diff --git a/lib/lmb.c b/lib/lmb.c
index 83287d3..3f55973 100644
--- a/lib/lmb.c
+++ b/lib/lmb.c
 ...
+#define DEBUG
+

which should have resulted in lmb_dump_all() printing some userful
debugging messages.  I validated that it did so on my machine with the
patch applied, but they appear nowhere in your logs :(

paginig_init() in arch/sparc64/mm/init.c calls lmb_analyze() them lmb_dump_all().

Those messages go out with KERN_DEBUG log level, maybe messages at
that level were trimmed by your log capture for some reason?

In any event I think I know the area that's cause some kind of problem.
It looks like lmb_alloc() has a case where it will reserve the wrong
amount of memory, or something like that.

You can remove the debugging patch I sent you, and try this one instead.
Please make sure KERN_DEBUG messages make it into the log :-)

diff --git a/arch/sparc64/mm/init.c b/arch/sparc64/mm/init.c
index a9828d7..a628a99 100644
--- a/arch/sparc64/mm/init.c
+++ b/arch/sparc64/mm/init.c
@@ -1353,6 +1353,8 @@ static void __init bootmem_init_one_node(int nid)
 
 	numadbg("bootmem_init_one_node(%d)\n", nid);
 
+	lmb_dump_all();
+
 	p = NODE_DATA(nid);
 
 	if (p->node_spanned_pages) {
diff --git a/lib/lmb.c b/lib/lmb.c
index 83287d3..d8c84f3 100644
--- a/lib/lmb.c
+++ b/lib/lmb.c
@@ -17,6 +17,8 @@
 
 #define LMB_ALLOC_ANYWHERE	0
 
+#define DEBUG
+
 struct lmb lmb;
 
 void lmb_dump_all(void)
@@ -29,7 +31,7 @@ void lmb_dump_all(void)
 	pr_debug("    memory.size		  = 0x%llx\n",
 	    (unsigned long long)lmb.memory.size);
 	for (i=0; i < lmb.memory.cnt ;i++) {
-		pr_debug("    memory.region[0x%x].base       = 0x%llx\n",
+		pr_debug("    memory.region[0x%lx].base       = 0x%llx\n",
 		    i, (unsigned long long)lmb.memory.region[i].base);
 		pr_debug("		      .size     = 0x%llx\n",
 		    (unsigned long long)lmb.memory.region[i].size);
@@ -38,7 +40,7 @@ void lmb_dump_all(void)
 	pr_debug("    reserved.cnt	  = 0x%lx\n", lmb.reserved.cnt);
 	pr_debug("    reserved.size	  = 0x%lx\n", lmb.reserved.size);
 	for (i=0; i < lmb.reserved.cnt ;i++) {
-		pr_debug("    reserved.region[0x%x].base       = 0x%llx\n",
+		pr_debug("    reserved.region[0x%lx].base       = 0x%llx\n",
 		    i, (unsigned long long)lmb.reserved.region[i].base);
 		pr_debug("		      .size     = 0x%llx\n",
 		    (unsigned long long)lmb.reserved.region[i].size);

Re: [BUG] 2.6.26-rc1 lost half the RAM on UltraSPARC 5

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Mon, 12 May 2008 15:31:43 -0700 (PDT)

> From: Mikael Pettersson <mikpe@it.uu.se>
> Date: Mon, 12 May 2008 21:06:41 +0200
> 
> > David Miller writes:
> >  > Please also add the debugging patch below.
> > 
> > Right, 2.6.26-rc2 plus your debugging patch and booted with numa=debug
> > prints the following:
> 
> Hmmm... my debugging patch had this:
> 
> diff --git a/lib/lmb.c b/lib/lmb.c
> index 83287d3..3f55973 100644
> --- a/lib/lmb.c
> +++ b/lib/lmb.c
>  ...
> +#define DEBUG
> +

Nevermind I see why the messages don't show up.  Hold on for a second
and I'll send an updated debugging patch.

Thanks.

Re: [BUG] 2.6.26-rc1 lost half the RAM on UltraSPARC 5

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Mon, 12 May 2008 15:36:28 -0700 (PDT)

> Nevermind I see why the messages don't show up.  Hold on for a second
> and I'll send an updated debugging patch.

Try this patch instead, thanks!

diff --git a/arch/sparc64/mm/init.c b/arch/sparc64/mm/init.c
index a9828d7..a628a99 100644
--- a/arch/sparc64/mm/init.c
+++ b/arch/sparc64/mm/init.c
@@ -1353,6 +1353,8 @@ static void __init bootmem_init_one_node(int nid)
 
 	numadbg("bootmem_init_one_node(%d)\n", nid);
 
+	lmb_dump_all();
+
 	p = NODE_DATA(nid);
 
 	if (p->node_spanned_pages) {
diff --git a/lib/lmb.c b/lib/lmb.c
index 83287d3..f294bbc 100644
--- a/lib/lmb.c
+++ b/lib/lmb.c
@@ -10,6 +10,8 @@
  *      2 of the License, or (at your option) any later version.
  */
 
+#define DEBUG
+
 #include <linux/kernel.h>
 #include <linux/init.h>
 #include <linux/bitops.h>
@@ -29,7 +31,7 @@ void lmb_dump_all(void)
 	pr_debug("    memory.size		  = 0x%llx\n",
 	    (unsigned long long)lmb.memory.size);
 	for (i=0; i < lmb.memory.cnt ;i++) {
-		pr_debug("    memory.region[0x%x].base       = 0x%llx\n",
+		pr_debug("    memory.region[0x%lx].base       = 0x%llx\n",
 		    i, (unsigned long long)lmb.memory.region[i].base);
 		pr_debug("		      .size     = 0x%llx\n",
 		    (unsigned long long)lmb.memory.region[i].size);
@@ -38,7 +40,7 @@ void lmb_dump_all(void)
 	pr_debug("    reserved.cnt	  = 0x%lx\n", lmb.reserved.cnt);
 	pr_debug("    reserved.size	  = 0x%lx\n", lmb.reserved.size);
 	for (i=0; i < lmb.reserved.cnt ;i++) {
-		pr_debug("    reserved.region[0x%x].base       = 0x%llx\n",
+		pr_debug("    reserved.region[0x%lx].base       = 0x%llx\n",
 		    i, (unsigned long long)lmb.reserved.region[i].base);
 		pr_debug("		      .size     = 0x%llx\n",
 		    (unsigned long long)lmb.reserved.region[i].size);

Re: [BUG] 2.6.26-rc1 lost half the RAM on UltraSPARC 5

[Mikael Pettersson]

David Miller writes:
 > From: David Miller <davem@davemloft.net>
 > Date: Mon, 12 May 2008 15:36:28 -0700 (PDT)
 > 
 > > Nevermind I see why the messages don't show up.  Hold on for a second
 > > and I'll send an updated debugging patch.
 > 
 > Try this patch instead, thanks!
 > 
 > diff --git a/arch/sparc64/mm/init.c b/arch/sparc64/mm/init.c
 > index a9828d7..a628a99 100644
 > --- a/arch/sparc64/mm/init.c
 > +++ b/arch/sparc64/mm/init.c
 > @@ -1353,6 +1353,8 @@ static void __init bootmem_init_one_node(int nid)
 >  
 >  	numadbg("bootmem_init_one_node(%d)\n", nid);
 >  
 > +	lmb_dump_all();
 > +
 >  	p = NODE_DATA(nid);
 >  
 >  	if (p->node_spanned_pages) {
 > diff --git a/lib/lmb.c b/lib/lmb.c
 > index 83287d3..f294bbc 100644
 > --- a/lib/lmb.c
 > +++ b/lib/lmb.c
 > @@ -10,6 +10,8 @@
 >   *      2 of the License, or (at your option) any later version.
 >   */
 >  
 > +#define DEBUG
 > +
 >  #include <linux/kernel.h>
 >  #include <linux/init.h>
 >  #include <linux/bitops.h>
 > @@ -29,7 +31,7 @@ void lmb_dump_all(void)
 >  	pr_debug("    memory.size		  = 0x%llx\n",
 >  	    (unsigned long long)lmb.memory.size);
 >  	for (i=0; i < lmb.memory.cnt ;i++) {
 > -		pr_debug("    memory.region[0x%x].base       = 0x%llx\n",
 > +		pr_debug("    memory.region[0x%lx].base       = 0x%llx\n",
 >  		    i, (unsigned long long)lmb.memory.region[i].base);
 >  		pr_debug("		      .size     = 0x%llx\n",
 >  		    (unsigned long long)lmb.memory.region[i].size);
 > @@ -38,7 +40,7 @@ void lmb_dump_all(void)
 >  	pr_debug("    reserved.cnt	  = 0x%lx\n", lmb.reserved.cnt);
 >  	pr_debug("    reserved.size	  = 0x%lx\n", lmb.reserved.size);
 >  	for (i=0; i < lmb.reserved.cnt ;i++) {
 > -		pr_debug("    reserved.region[0x%x].base       = 0x%llx\n",
 > +		pr_debug("    reserved.region[0x%lx].base       = 0x%llx\n",
 >  		    i, (unsigned long long)lmb.reserved.region[i].base);
 >  		pr_debug("		      .size     = 0x%llx\n",
 >  		    (unsigned long long)lmb.reserved.region[i].size);
 > 

Ok, reverting previous patch and applying this done results in:

PROMLIB: Sun IEEE Boot Prom 'OBP 3.25.3 2000/06/29 14:12'
PROMLIB: Root node compatible: 
Linux version 2.6.26-rc2-test (mikpe@sparge) (gcc version 4.2.3) #2 Tue May 13 18:26:56 CEST 2008
console [earlyprom0] enabled
ARCH: SUN4U
Ethernet address: 08:00:20:fd:ec:1f
Found ramdisk at physical address 0x10800000, size 3683665
lmb_dump_all:
    memory.cnt		  = 0x4
    memory.size		  = 0xff40000
    memory.region[0x0].base       = 0x0
		      .size     = 0x8000000
    memory.region[0x1].base       = 0x10000000
		      .size     = 0x7efe000
    memory.region[0x2].base       = 0x17f00000
		      .size     = 0x3a000
    memory.region[0x3].base       = 0x17f3e000
		      .size     = 0x8000
    reserved.cnt	  = 0x2
    reserved.size	  = 0x0
    reserved.region[0x0].base       = 0x10000000
		      .size     = 0x35bf60
    reserved.region[0x1].base       = 0x10800000
		      .size     = 0x10b83551
Kernel: Using 1 locked TLB entries for main kernel image.
Remapping the kernel... done.
OF stdout device is: /pci@1f,0/pci@1,1/SUNW,m64B@2
PROM: Built device tree with 46641 bytes of memory.
bootmem_init_numa()
bootmem_init_nonnuma()
Top of RAM: 0x17f46000, Total RAM: 0xff40000
Memory hole size: 128MB
Initializing tables for non-numa.
Entering add_active_range(0, 0, 16384) 0 entries of 256 used
Entering add_active_range(0, 32768, 49023) 1 entries of 256 used
Entering add_active_range(0, 49024, 49053) 2 entries of 256 used
Entering add_active_range(0, 49055, 49059) 3 entries of 256 used
bootmem_init_one_node(0)
lmb_dump_all:
    memory.cnt		  = 0x4
    memory.size		  = 0xff40000
    memory.region[0x0].base       = 0x0
		      .size     = 0x8000000
    memory.region[0x1].base       = 0x10000000
		      .size     = 0x7efe000
    memory.region[0x2].base       = 0x17f00000
		      .size     = 0x3a000
    memory.region[0x3].base       = 0x17f3e000
		      .size     = 0x8000
    reserved.cnt	  = 0x4
    reserved.size	  = 0x0
    reserved.region[0x0].base       = 0x7ffc000
		      .size     = 0x2000
    reserved.region[0x1].base       = 0x7fff040
		      .size     = 0xfc0
    reserved.region[0x2].base       = 0x10000000
		      .size     = 0x35bf60
    reserved.region[0x3].base       = 0x107e8e00
		      .size     = 0x10b9a751
  init_bootmem_node(0, 3ffe, 0, bfa3)
  free_bootmem_with_active_regions(0, bfa3)
  trim_reserved_in_node(0)
    reserve_range_in_node(nid[0],start[7ffc000],end[7ffe000]
      MATCH reserving range [7ffc000:7ffe000]
    reserve_range_in_node(nid[0],start[7fff040],end[8000000]
      MATCH reserving range [7fff040:8001040]
    reserve_range_in_node(nid[0],start[10000000],end[1035bf60]
      MATCH reserving range [10000000:1035c000]
    reserve_range_in_node(nid[0],start[107e8e00],end[21383551]
      MATCH reserving range [107e8e00:21384e00]
  sparse_memory_present_with_active_regions(0)
[0000000200000000-fffff80000400000] page_structs=131072 node=0 entry=0/0
[0000000200000000-fffff80000800000] page_structs=131072 node=0 entry=1/0
Zone PFN ranges:
  Normal          0 ->    49059
Movable zone start PFN for each node
early_node_map[4] active PFN ranges
    0:        0 ->    16384
    0:    32768 ->    49023
    0:    49024 ->    49053
    0:    49055 ->    49059
On node 0 totalpages: 32672
  Normal zone: 335 pages used for memmap
  Normal zone: 0 pages reserved
  Normal zone: 32337 pages, LIFO batch:7
  Movable zone: 0 pages used for memmap
Booting Linux...
Built 1 zonelists in Node order, mobility grouping on.  Total pages: 32337
Policy zone: Normal
Kernel command line: ro root=/dev/sda5 numa=debug
PID hash table entries: 1024 (order: 10, 8192 bytes)
clocksource: mult[28000] shift[16]
clockevent: mult[66666666] shift[32]
Console: colour dummy device 80x25
console handover: boot [earlyprom0] -> real [tty0]
Dentry cache hash table entries: 32768 (order: 5, 262144 bytes)
Inode-cache hash table entries: 16384 (order: 4, 131072 bytes)
Memory: 127056k available (2040k kernel code, 808k data, 152k init) [fffff80000000000,0000000017f46000]
SLUB: Genslabs=14, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=16
Calibrating delay using timer specific routine.. 800.75 BogoMIPS (lpj=4003776)
Mount-cache hash table entries: 512
net_namespace: 344 bytes
NET: Registered protocol family 16
PCI: Probing for controllers.
/pci@1f,0: SABRE PCI Bus Module
/pci@1f,0: PCI IO[1fe02000000] MEM[1ff00000000]
PCI: Scanning PBM /pci@1f,0
ebus0: [auxio] [power] [SUNW,pll] [se] [su] [su] [ecpp] [fdthree] [eeprom] [flashprom] [SUNW,CS4231]
power: Control reg at 1fff1724000
SCSI subsystem initialized
libata version 3.00 loaded.
AUXIO: Found device at /pci@1f,0/pci@1,1/ebus@1/auxio@14,726000
/pci@1f,0/pci@1,1/ebus@1/eeprom@14,0: Clock regs at 000001fff1000000
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 16384 bytes)
TCP established hash table entries: 8192 (order: 4, 131072 bytes)
TCP bind hash table entries: 8192 (order: 3, 65536 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
NET: Registered protocol family 1
Unpacking initramfs... done
Freeing initrd memory: 3597k freed
Mini RTC Driver
msgmni has been set to 255 for ipc namespace 00000000006b27c8
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
atyfb: 3D RAGE PRO (Mach64 GP, PQFP, PCI) [0x4750 rev 0x7c]
atyfb: 4M SGRAM (1:1), 14.31818 MHz XTAL, 230 MHz PLL, 100 Mhz MCLK, 100 MHz XCLK
fbcon: ATY Mach64 (fb0) is primary device
Console: switching to colour frame buffer device 144x56
atyfb: fb0: ATY Mach64 frame buffer device on PCI
/pci@1f,0/pci@1,1/ebus@1/su@14,3083f8: Keyboard port at 1fff13083f8, irq 6
/pci@1f,0/pci@1,1/ebus@1/su@14,3062f8: Mouse port at 1fff13062f8, irq 7
f006159c: ttyS0 at MMIO 0x1fff1400000 (irq = 5) is a SAB82532 V3.2
f006159c: ttyS1 at MMIO 0x1fff1400040 (irq = 5) is a SAB82532 V3.2
Driver 'sd' needs updating - please use bus_type methods
scsi0 : pata_cmd64x
scsi1 : pata_cmd64x
ata1: PATA max MWDMA2 cmd 0x1fe02c00000 ctl 0x1fe02c00008 bmdma 0x1fe02c00020 irq 14
ata2: PATA max MWDMA2 cmd 0x1fe02c00010 ctl 0x1fe02c00018 bmdma 0x1fe02c00028 irq 14
pata_cmd64x: active 10 recovery 10 setup 3.
pata_cmd64x: active 10 recovery 10 setup 3.
ata1.00: ATA-4: ST320420A, 3.21, max UDMA/66
ata1.00: 39851760 sectors, multi 0: LBA 
pata_cmd64x: active 3 recovery 1 setup 1.
pata_cmd64x: active 3 recovery 1 setup 1.
ata1.00: configured for MWDMA2
pata_cmd64x: active 10 recovery 10 setup 3.
pata_cmd64x: active 10 recovery 10 setup 3.
ata2.00: ATAPI: CRD-8483B, 1.00, max UDMA/33
pata_cmd64x: active 3 recovery 1 setup 1.
pata_cmd64x: active 3 recovery 1 setup 1.
ata2.00: configured for MWDMA2
scsi 0:0:0:0: Direct-Access     ATA      ST320420A        3.21 PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 39851760 512-byte hardware sectors (20404 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 0:0:0:0: [sda] 39851760 512-byte hardware sectors (20404 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda: sda1 sda2 sda3 sda4 sda5
sd 0:0:0:0: [sda] Attached SCSI disk
scsi 1:0:0:0: CD-ROM            LG       CD-ROM CRD-8483B 1.00 PQ: 0 ANSI: 5
mice: PS/2 mouse device common for all mice
input: Sun Type 5 keyboard as /class/input/input0
input: Sun Mouse as /class/input/input1
TCP cubic registered
NET: Registered protocol family 17
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
PCI: Enabling device: (0000:01:01.1), cmd 2
sunhme.c:v3.00 June 23, 2006 David S. Miller (davem@davemloft.net)
eth0: HAPPY MEAL (PCI/CheerIO) 10/100BaseT Ethernet 08:00:20:fd:ec:1f
EXT3 FS on sda5, internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 1048808k swap on /dev/sda4.  Priority:-1 extents:1 across:1048808k
eth0: Link is up using internal transceiver at 100Mb/s, Full Duplex.

I also tried your two lmb patches from today (had to revert this patch first),
but they made no significant difference.

/Mikael

Re: [BUG] 2.6.26-rc1 lost half the RAM on UltraSPARC 5

[David Miller]

From: Mikael Pettersson <mikpe@it.uu.se>
Date: Tue, 13 May 2008 21:31:19 +0200

> David Miller writes:
>  > Try this patch instead, thanks!
> 
> Ok, reverting previous patch and applying this done results in:
 ...
> lmb_dump_all:
>     memory.cnt		  = 0x4
>     memory.size		  = 0xff40000
>     memory.region[0x0].base       = 0x0
> 		      .size     = 0x8000000
>     memory.region[0x1].base       = 0x10000000
> 		      .size     = 0x7efe000
>     memory.region[0x2].base       = 0x17f00000
> 		      .size     = 0x3a000
>     memory.region[0x3].base       = 0x17f3e000
> 		      .size     = 0x8000
>     reserved.cnt	  = 0x4
>     reserved.size	  = 0x0
>     reserved.region[0x0].base       = 0x7ffc000
> 		      .size     = 0x2000
>     reserved.region[0x1].base       = 0x7fff040
> 		      .size     = 0xfc0
>     reserved.region[0x2].base       = 0x10000000
> 		      .size     = 0x35bf60
>     reserved.region[0x3].base       = 0x107e8e00
> 		      .size     = 0x10b9a751

Yeah, those last two reserved regions are where your 128MB went to,
as I suspected.

Thanks I'll try to figure out where to go from here.

Re: [BUG] 2.6.26-rc1 lost half the RAM on UltraSPARC 5

[David Miller]

From: Mikael Pettersson <mikpe@it.uu.se>
Date: Tue, 13 May 2008 21:31:19 +0200

Ok, Mikael, I think I figured out this bug:

> Found ramdisk at physical address 0x10800000, size 3683665
> lmb_dump_all:
>     memory.cnt		  = 0x4
>     memory.size		  = 0xff40000
>     memory.region[0x0].base       = 0x0
> 		      .size     = 0x8000000
>     memory.region[0x1].base       = 0x10000000
> 		      .size     = 0x7efe000
>     memory.region[0x2].base       = 0x17f00000
> 		      .size     = 0x3a000
>     memory.region[0x3].base       = 0x17f3e000
> 		      .size     = 0x8000
>     reserved.cnt	  = 0x2
>     reserved.size	  = 0x0
>     reserved.region[0x0].base       = 0x10000000
> 		      .size     = 0x35bf60
>     reserved.region[0x1].base       = 0x10800000
> 		      .size     = 0x10b83551

>From the very beginning your higher RAM is gone.
It's correct that some memory should be reserved,
for the ramdisk, but not 128MB :-)

The ramdisk is just under 4MB in size, so something is fishy here.

And indeed, I'm reserving the wrong length.  Please try this
patch:

sparc64: Fix lmb_reserve() args in find_ramdisk().

This fixes the missing ram regression reported by
Mikael Pettersson <mikpe@it.uu.se>, much thanks for
all of this help in diagnosing this.

The second argument to lmb_reserve() is a size,
not an end address bounds.

Signed-off-by: David S. Miller <davem@davemloft.net>
---
 arch/sparc64/mm/init.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/arch/sparc64/mm/init.c b/arch/sparc64/mm/init.c
index a9828d7..3c7b947 100644
--- a/arch/sparc64/mm/init.c
+++ b/arch/sparc64/mm/init.c
@@ -768,7 +768,7 @@ static void __init find_ramdisk(unsigned long phys_base)
 		initrd_start = ramdisk_image;
 		initrd_end = ramdisk_image + sparc_ramdisk_size;
 
-		lmb_reserve(initrd_start, initrd_end);
+		lmb_reserve(initrd_start, sparc_ramdisk_size);
 
 		initrd_start += PAGE_OFFSET;
 		initrd_end += PAGE_OFFSET;
-- 
1.5.5.1.57.g5909c

Re: [BUG] 2.6.26-rc1 lost half the RAM on UltraSPARC 5

[Mikael Pettersson]

David Miller writes:
 > From: Mikael Pettersson <mikpe@it.uu.se>
 > Date: Tue, 13 May 2008 21:31:19 +0200
 > 
 > Ok, Mikael, I think I figured out this bug:
 > 
 > > Found ramdisk at physical address 0x10800000, size 3683665
 > > lmb_dump_all:
 > >     memory.cnt		  = 0x4
 > >     memory.size		  = 0xff40000
 > >     memory.region[0x0].base       = 0x0
 > > 		      .size     = 0x8000000
 > >     memory.region[0x1].base       = 0x10000000
 > > 		      .size     = 0x7efe000
 > >     memory.region[0x2].base       = 0x17f00000
 > > 		      .size     = 0x3a000
 > >     memory.region[0x3].base       = 0x17f3e000
 > > 		      .size     = 0x8000
 > >     reserved.cnt	  = 0x2
 > >     reserved.size	  = 0x0
 > >     reserved.region[0x0].base       = 0x10000000
 > > 		      .size     = 0x35bf60
 > >     reserved.region[0x1].base       = 0x10800000
 > > 		      .size     = 0x10b83551
 > 
 > >From the very beginning your higher RAM is gone.
 > It's correct that some memory should be reserved,
 > for the ramdisk, but not 128MB :-)
 > 
 > The ramdisk is just under 4MB in size, so something is fishy here.
 > 
 > And indeed, I'm reserving the wrong length.  Please try this
 > patch:
 > 
 > sparc64: Fix lmb_reserve() args in find_ramdisk().
 > 
 > This fixes the missing ram regression reported by
 > Mikael Pettersson <mikpe@it.uu.se>, much thanks for
 > all of this help in diagnosing this.
 > 
 > The second argument to lmb_reserve() is a size,
 > not an end address bounds.
 > 
 > Signed-off-by: David S. Miller <davem@davemloft.net>
 > ---
 >  arch/sparc64/mm/init.c |    2 +-
 >  1 files changed, 1 insertions(+), 1 deletions(-)
 > 
 > diff --git a/arch/sparc64/mm/init.c b/arch/sparc64/mm/init.c
 > index a9828d7..3c7b947 100644
 > --- a/arch/sparc64/mm/init.c
 > +++ b/arch/sparc64/mm/init.c
 > @@ -768,7 +768,7 @@ static void __init find_ramdisk(unsigned long phys_base)
 >  		initrd_start = ramdisk_image;
 >  		initrd_end = ramdisk_image + sparc_ramdisk_size;
 >  
 > -		lmb_reserve(initrd_start, initrd_end);
 > +		lmb_reserve(initrd_start, sparc_ramdisk_size);
 >  
 >  		initrd_start += PAGE_OFFSET;
 >  		initrd_end += PAGE_OFFSET;
 > -- 
 > 1.5.5.1.57.g5909c
 > 

Thanks Dave. As the dmesg diff below shows, this patch fixed
my lost RAM issue. I actually have about 8KB more available
now than I had with 2.6.25.

Tested-by: Mikael Pettersson <mikpe@it.uu.se>

--- dmesg-2.6.26-rc2-a	2008-05-15 16:12:39.000000000 +0200
+++ dmesg-2.6.26-rc2-b	2008-05-15 16:18:37.000000000 +0200
@@ -1,14 +1,14 @@
 PROMLIB: Sun IEEE Boot Prom 'OBP 3.25.3 2000/06/29 14:12'
 PROMLIB: Root node compatible: 
-Linux version 2.6.26-rc2 (mikpe@sparge) (gcc version 4.2.3) #1 Wed May 14 22:27:26 CEST 2008
+Linux version 2.6.26-rc2 (mikpe@sparge) (gcc version 4.2.3) #2 Thu May 15 16:14:24 CEST 2008
 console [earlyprom0] enabled
 ARCH: SUN4U
 Ethernet address: 08:00:20:fd:ec:1f
-Found ramdisk at physical address 0x10800000, size 3780573
+Found ramdisk at physical address 0x10800000, size 3780646
 Kernel: Using 1 locked TLB entries for main kernel image.
 Remapping the kernel... done.
 OF stdout device is: /pci@1f,0/pci@1,1/SUNW,m64B@2
-PROM: Built device tree with 42146 bytes of memory.
+PROM: Built device tree with 42148 bytes of memory.
 bootmem_init_nonnuma()
 Top of RAM: 0x17f46000, Total RAM: 0xff42000
 Memory hole size: 128MB
@@ -25,8 +25,12 @@
       MATCH reserving range [7ffe000:8000000]
     reserve_range_in_node(nid[0],start[10000000],end[1030ed28]
       MATCH reserving range [10000000:1030ed28]
-    reserve_range_in_node(nid[0],start[107eb3c0],end[2139afdd]
-      MATCH reserving range [107eb3c0:2139afdd]
+    reserve_range_in_node(nid[0],start[10800000],end[10b9b026]
+      MATCH reserving range [10800000:10b9b026]
+    reserve_range_in_node(nid[0],start[17f2f3c0],end[17f3c000]
+      MATCH reserving range [17f2f3c0:17f3c000]
+    reserve_range_in_node(nid[0],start[17f3e000],end[17f46000]
+      MATCH reserving range [17f3e000:17f46000]
   sparse_memory_present_with_active_regions(0)
 [0000000200000000-fffff80000400000] page_structs=131072 node=0 entry=0/0
 [0000000200000000-fffff80000800000] page_structs=131072 node=0 entry=1/0
@@ -53,9 +57,9 @@
 console handover: boot [earlyprom0] -> real [tty0]
 Dentry cache hash table entries: 32768 (order: 5, 262144 bytes)
 Inode-cache hash table entries: 16384 (order: 4, 131072 bytes)
-Memory: 127016k available (1920k kernel code, 744k data, 152k init) [fffff80000000000,0000000017f46000]
+Memory: 245448k available (1920k kernel code, 744k data, 152k init) [fffff80000000000,0000000017f46000]
...

Re: [BUG] 2.6.26-rc1 lost half the RAM on UltraSPARC 5

[Mikael Pettersson]

Mikael Pettersson writes:
 > Thanks Dave. As the dmesg diff below shows, this patch fixed
 > my lost RAM issue. I actually have about 8KB more available

Doh! s/KB/MB/ of course.

 > now than I had with 2.6.25.

Re: [BUG] 2.6.26-rc1 lost half the RAM on UltraSPARC 5

[David Miller]

From: Mikael Pettersson <mikpe@it.uu.se>
Date: Thu, 15 May 2008 20:43:10 +0200

> Thanks Dave. As the dmesg diff below shows, this patch fixed
> my lost RAM issue. I actually have about 8KB more available
> now than I had with 2.6.25.

Don't spend it all in one place :-)

> Tested-by: Mikael Pettersson <mikpe@it.uu.se>

Thanks a lot for all of your help.