From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751537AbbEBQGI (ORCPT ); Sat, 2 May 2015 12:06:08 -0400 Received: from numascale.com ([213.162.240.84]:55859 "EHLO numascale.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750994AbbEBQGG (ORCPT ); Sat, 2 May 2015 12:06:06 -0400 Date: Sun, 03 May 2015 00:05:49 +0800 From: Daniel J Blueman Subject: Re: [PATCH 0/13] Parallel struct page initialisation v4 To: Waiman Long , Mel Gorman Cc: Andrew Morton , Nathan Zimmer , Dave Hansen , Scott Norton , Linux-MM , LKML Message-Id: <1430582749.21217.0@cpanel21.proisp.no> In-Reply-To: <1430556732.28355.0@cpanel21.proisp.no> References: <1430556732.28355.0@cpanel21.proisp.no> X-Mailer: geary/0.8.3 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cpanel21.proisp.no X-AntiAbuse: Original Domain - vger.kernel.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - numascale.com X-Get-Message-Sender-Via: cpanel21.proisp.no: authenticated_id: daniel@numascale.com X-Source: X-Source-Args: X-Source-Dir: Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, May 2, 2015 at 4:52 PM, Daniel J Blueman wrote: > On Sat, May 2, 2015 at 8:09 AM, Waiman Long > wrote: >> On 05/01/2015 06:02 PM, Waiman Long wrote: >>> >>> Bad news! >>> >>> I tried your patch on a 24-TB DragonHawk and got an out of memory >>> panic. The kernel log messages were: >>> : >>> [ 80.126186] CPU 474: hi: 186, btch: 31 usd: 0 >>> [ 80.131457] CPU 475: hi: 186, btch: 31 usd: 0 >>> [ 80.136726] CPU 476: hi: 186, btch: 31 usd: 0 >>> [ 80.141997] CPU 477: hi: 186, btch: 31 usd: 0 >>> [ 80.147267] CPU 478: hi: 186, btch: 31 usd: 0 >>> [ 80.152538] CPU 479: hi: 186, btch: 31 usd: 0 >>> [ 80.157813] active_anon:0 inactive_anon:0 isolated_anon:0 >>> [ 80.157813] active_file:0 inactive_file:0 isolated_file:0 >>> [ 80.157813] unevictable:0 dirty:0 writeback:0 unstable:0 >>> [ 80.157813] free:209 slab_reclaimable:7 slab_unreclaimable:42986 >>> [ 80.157813] mapped:0 shmem:0 pagetables:0 bounce:0 >>> [ 80.157813] free_cma:0 >>> [ 80.190428] Node 0 DMA free:568kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:15988kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB >>> mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:14928kB kernel_stack:400kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.233475] lowmem_reserve[]: 0 0 0 0 >>> [ 80.237542] Node 0 DMA32 free:20kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1961924kB managed:1333604kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB >>> slab_unreclaimable:101664kB kernel_stack:50176kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.281456] lowmem_reserve[]: 0 0 0 0 >>> [ 80.285527] Node 0 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1608515580kB managed:2097148kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:4kB >>> slab_unreclaimable:948kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.328958] lowmem_reserve[]: 0 0 0 0 >>> [ 80.333031] Node 1 Normal free:248kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612732kB managed:2228220kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB >>> slab_unreclaimable:46240kB kernel_stack:3232kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.377256] lowmem_reserve[]: 0 0 0 0 >>> [ 80.381325] Node 2 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:612kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.424764] lowmem_reserve[]: 0 0 0 0 >>> [ 80.428842] Node 3 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:600kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.472293] lowmem_reserve[]: 0 0 0 0 >>> [ 80.476360] Node 4 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:620kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.519803] lowmem_reserve[]: 0 0 0 0 >>> [ 80.523875] Node 5 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:584kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.567312] lowmem_reserve[]: 0 0 0 0 >>> [ 80.571379] Node 6 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:556kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.614814] lowmem_reserve[]: 0 0 0 0 >>> [ 80.618881] Node 7 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:556kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.662316] lowmem_reserve[]: 0 0 0 0 >>> [ 80.666390] Node 8 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:572kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.709827] lowmem_reserve[]: 0 0 0 0 >>> [ 80.713898] Node 9 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:572kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.757336] lowmem_reserve[]: 0 0 0 0 >>> [ 80.761407] Node 10 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:564kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.804941] lowmem_reserve[]: 0 0 0 0 >>> [ 80.809015] Node 11 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:572kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.852548] lowmem_reserve[]: 0 0 0 0 >>> [ 80.856620] Node 12 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:616kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.900158] lowmem_reserve[]: 0 0 0 0 >>> [ 80.904236] Node 13 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:592kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.947765] lowmem_reserve[]: 0 0 0 0 >>> [ 80.951847] Node 14 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:600kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.995380] lowmem_reserve[]: 0 0 0 0 >>> [ 80.999448] Node 15 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:548kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 81.042974] lowmem_reserve[]: 0 0 0 0 >>> [ 81.047044] Node 0 DMA: 132*4kB (U) 5*8kB (U) 0*16kB 0*32kB >>> 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 568kB >>> [ 81.059632] Node 0 DMA32: 5*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB >>> [ 81.071733] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.083443] Node 1 Normal: 52*4kB (U) 5*8kB (U) 0*16kB 0*32kB >>> 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 248kB >>> [ 81.096227] Node 2 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.107935] Node 3 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.119643] Node 4 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.131347] Node 5 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.143056] Node 6 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.154767] Node 7 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.166473] Node 8 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.178179] Node 9 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.189893] Node 10 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.201695] Node 11 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.213496] Node 12 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.225324] Node 13 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.237130] Node 14 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.248926] Node 15 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.260726] 0 total pagecache pages >>> [ 81.264565] 0 pages in swap cache >>> [ 81.268212] Swap cache stats: add 0, delete 0, find 0/0 >>> [ 81.273962] Free swap = 0kB >>> [ 81.277125] Total swap = 0kB >>> [ 81.280341] 6442421132 pages RAM >>> [ 81.283888] 0 pages HighMem/MovableOnly >>> [ 81.288109] 6433662383 pages reserved >>> [ 81.292135] 0 pages hwpoisoned >>> [ 81.295491] [ pid ] uid tgid total_vm rss nr_ptes >>> nr_pmds swapents oom_score_adj name >>> [ 81.305245] Kernel panic - not syncing: Out of memory and no >>> killable processes... >>> [ 81.305245] >>> [ 81.315200] CPU: 240 PID: 1 Comm: swapper/0 Not tainted >>> 4.0.1-pmm-bigsmp #1 >>> [ 81.322856] Hardware name: HP Superdome2 16s x86, BIOS Bundle: >>> 006.000.042 SFW: 015.099.000 04/01/2015 >>> [ 81.333096] 0000000000000000 ffff8800044c79c8 ffffffff8151b0c9 >>> ffff8800044c7a48 >>> [ 81.341262] ffffffff8151ae1e 0000000000000008 ffff8800044c7a58 >>> ffff8800044c79f8 >>> [ 81.349428] ffffffff810785c3 ffffffff81a13480 0000000000000000 >>> ffff8800001001d0 >>> [ 81.357595] Call Trace: >>> [ 81.360287] [] dump_stack+0x68/0x77 >>> [ 81.365942] [] panic+0xb9/0x219 >>> [ 81.371213] [] ? >>> __blocking_notifier_call_chain+0x63/0x80 >>> [ 81.378971] [] __out_of_memory+0x34e/0x350 >>> [ 81.385292] [] out_of_memory+0x5e/0x90 >>> [ 81.391230] [] >>> __alloc_pages_slowpath+0x6be/0x740 >>> [ 81.398219] [] >>> __alloc_pages_nodemask+0x23c/0x250 >>> [ 81.405212] [] kmem_getpages+0x56/0x110 >>> [ 81.411246] [] fallback_alloc+0x164/0x200 >>> [ 81.417474] [] ____cache_alloc_node+0x8d/0x170 >>> [ 81.424179] [] >>> kmem_cache_alloc_trace+0x17b/0x240 >>> [ 81.431169] [] init_memory_block+0x3a/0x110 >>> [ 81.437586] [] memory_dev_init+0xd7/0x13d >>> [ 81.443810] [] driver_init+0x2f/0x37 >>> [ 81.449556] [] do_basic_setup+0x29/0xd5 >>> [ 81.455597] [] ? sched_init_smp+0x140/0x147 >>> [ 81.462015] [] >>> kernel_init_freeable+0x20e/0x297 >>> [ 81.468815] [] ? rest_init+0x80/0x80 >>> [ 81.474565] [] kernel_init+0x9/0xf0 >>> [ 81.480216] [] ret_from_fork+0x58/0x90 >>> [ 81.486156] [] ? rest_init+0x80/0x80 >>> [ 81.492350] ---[ end Kernel panic - not syncing: Out of memory >>> and no killable processes... >>> [ 81.492350] >>> >>> -Longman >> >> I increased the pre-initialized memory per node in >> update_defer_init() of mm/page_alloc.c from 2G to 4G. Now I am able >> to boot the 24-TB machine without error. The 12-TB has 0.75TB/node, >> while the 24-TB machine has 1.5TB/node. I would suggest something >> like pre-initializing 1G per 0.25TB/node. In this way, it will scale >> properly with the memory size. >> >> Before the patch, the boot time from elilo prompt to ssh login was >> 694s. After the patch, the boot up time was 346s, a saving of 348s >> (about 50%). > > I second scaling the up-front init with the zone size. The 7TB system > I was booting has only 32GB per NUMA node, which at 1GB per 0.25TB > would work out at 128MB up-front init per-NUMA-node, which worked > nice and booted faster yet. > > Even booting with 64MB per NUMA node worked great, so there is > adequate margin for the 8 cores, just I guess we'd need to enforce a > minimum of eg 64MB or so. Varying the synchronous per-NUMA-node initialisation (with non-temporal patch, but that just removes a constant from PMD init), from kernel load to login prompt on this 7TB, 1728-core system takes: 512MB 699.2s 256MB 680.3s 128MB 661.7s 64MB 663.6s 32MB 667.8s So, in this case 128MB per NUMA node gives more locality than 64MB, so should be a good minimum, and matches Waiman's scaling suggestion. Thanks, Daniel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-la0-f48.google.com (mail-la0-f48.google.com [209.85.215.48]) by kanga.kvack.org (Postfix) with ESMTP id F21096B0038 for ; Sat, 2 May 2015 12:06:07 -0400 (EDT) Received: by lagv1 with SMTP id v1so80402778lag.3 for ; Sat, 02 May 2015 09:06:07 -0700 (PDT) Received: from numascale.com (numascale.com. [213.162.240.84]) by mx.google.com with ESMTPS id aw7si6325843lbc.80.2015.05.02.09.06.04 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 May 2015 09:06:05 -0700 (PDT) Date: Sun, 03 May 2015 00:05:49 +0800 From: Daniel J Blueman Subject: Re: [PATCH 0/13] Parallel struct page initialisation v4 Message-Id: <1430582749.21217.0@cpanel21.proisp.no> In-Reply-To: <1430556732.28355.0@cpanel21.proisp.no> References: <1430556732.28355.0@cpanel21.proisp.no> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Sender: owner-linux-mm@kvack.org List-ID: To: Waiman Long , Mel Gorman Cc: Andrew Morton , Nathan Zimmer , Dave Hansen , Scott Norton , Linux-MM , LKML On Sat, May 2, 2015 at 4:52 PM, Daniel J Blueman wrote: > On Sat, May 2, 2015 at 8:09 AM, Waiman Long > wrote: >> On 05/01/2015 06:02 PM, Waiman Long wrote: >>> >>> Bad news! >>> >>> I tried your patch on a 24-TB DragonHawk and got an out of memory >>> panic. The kernel log messages were: >>> : >>> [ 80.126186] CPU 474: hi: 186, btch: 31 usd: 0 >>> [ 80.131457] CPU 475: hi: 186, btch: 31 usd: 0 >>> [ 80.136726] CPU 476: hi: 186, btch: 31 usd: 0 >>> [ 80.141997] CPU 477: hi: 186, btch: 31 usd: 0 >>> [ 80.147267] CPU 478: hi: 186, btch: 31 usd: 0 >>> [ 80.152538] CPU 479: hi: 186, btch: 31 usd: 0 >>> [ 80.157813] active_anon:0 inactive_anon:0 isolated_anon:0 >>> [ 80.157813] active_file:0 inactive_file:0 isolated_file:0 >>> [ 80.157813] unevictable:0 dirty:0 writeback:0 unstable:0 >>> [ 80.157813] free:209 slab_reclaimable:7 slab_unreclaimable:42986 >>> [ 80.157813] mapped:0 shmem:0 pagetables:0 bounce:0 >>> [ 80.157813] free_cma:0 >>> [ 80.190428] Node 0 DMA free:568kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:15988kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB >>> mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:14928kB kernel_stack:400kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.233475] lowmem_reserve[]: 0 0 0 0 >>> [ 80.237542] Node 0 DMA32 free:20kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1961924kB managed:1333604kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB >>> slab_unreclaimable:101664kB kernel_stack:50176kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.281456] lowmem_reserve[]: 0 0 0 0 >>> [ 80.285527] Node 0 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1608515580kB managed:2097148kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:4kB >>> slab_unreclaimable:948kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.328958] lowmem_reserve[]: 0 0 0 0 >>> [ 80.333031] Node 1 Normal free:248kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612732kB managed:2228220kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB >>> slab_unreclaimable:46240kB kernel_stack:3232kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.377256] lowmem_reserve[]: 0 0 0 0 >>> [ 80.381325] Node 2 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:612kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.424764] lowmem_reserve[]: 0 0 0 0 >>> [ 80.428842] Node 3 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:600kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.472293] lowmem_reserve[]: 0 0 0 0 >>> [ 80.476360] Node 4 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:620kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.519803] lowmem_reserve[]: 0 0 0 0 >>> [ 80.523875] Node 5 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:584kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.567312] lowmem_reserve[]: 0 0 0 0 >>> [ 80.571379] Node 6 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:556kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.614814] lowmem_reserve[]: 0 0 0 0 >>> [ 80.618881] Node 7 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:556kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.662316] lowmem_reserve[]: 0 0 0 0 >>> [ 80.666390] Node 8 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:572kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.709827] lowmem_reserve[]: 0 0 0 0 >>> [ 80.713898] Node 9 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:572kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.757336] lowmem_reserve[]: 0 0 0 0 >>> [ 80.761407] Node 10 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:564kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.804941] lowmem_reserve[]: 0 0 0 0 >>> [ 80.809015] Node 11 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:572kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.852548] lowmem_reserve[]: 0 0 0 0 >>> [ 80.856620] Node 12 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:616kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.900158] lowmem_reserve[]: 0 0 0 0 >>> [ 80.904236] Node 13 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:592kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.947765] lowmem_reserve[]: 0 0 0 0 >>> [ 80.951847] Node 14 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:600kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 80.995380] lowmem_reserve[]: 0 0 0 0 >>> [ 80.999448] Node 15 Normal free:0kB min:0kB low:0kB high:0kB >>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB >>> unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB >>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB >>> slab_unreclaimable:548kB kernel_stack:0kB pagetables:0kB >>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB >>> pages_scanned:0 all_unreclaimable? yes >>> [ 81.042974] lowmem_reserve[]: 0 0 0 0 >>> [ 81.047044] Node 0 DMA: 132*4kB (U) 5*8kB (U) 0*16kB 0*32kB >>> 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 568kB >>> [ 81.059632] Node 0 DMA32: 5*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB >>> [ 81.071733] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.083443] Node 1 Normal: 52*4kB (U) 5*8kB (U) 0*16kB 0*32kB >>> 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 248kB >>> [ 81.096227] Node 2 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.107935] Node 3 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.119643] Node 4 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.131347] Node 5 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.143056] Node 6 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.154767] Node 7 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.166473] Node 8 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.178179] Node 9 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.189893] Node 10 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.201695] Node 11 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.213496] Node 12 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.225324] Node 13 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.237130] Node 14 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.248926] Node 15 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB >>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB >>> [ 81.260726] 0 total pagecache pages >>> [ 81.264565] 0 pages in swap cache >>> [ 81.268212] Swap cache stats: add 0, delete 0, find 0/0 >>> [ 81.273962] Free swap = 0kB >>> [ 81.277125] Total swap = 0kB >>> [ 81.280341] 6442421132 pages RAM >>> [ 81.283888] 0 pages HighMem/MovableOnly >>> [ 81.288109] 6433662383 pages reserved >>> [ 81.292135] 0 pages hwpoisoned >>> [ 81.295491] [ pid ] uid tgid total_vm rss nr_ptes >>> nr_pmds swapents oom_score_adj name >>> [ 81.305245] Kernel panic - not syncing: Out of memory and no >>> killable processes... >>> [ 81.305245] >>> [ 81.315200] CPU: 240 PID: 1 Comm: swapper/0 Not tainted >>> 4.0.1-pmm-bigsmp #1 >>> [ 81.322856] Hardware name: HP Superdome2 16s x86, BIOS Bundle: >>> 006.000.042 SFW: 015.099.000 04/01/2015 >>> [ 81.333096] 0000000000000000 ffff8800044c79c8 ffffffff8151b0c9 >>> ffff8800044c7a48 >>> [ 81.341262] ffffffff8151ae1e 0000000000000008 ffff8800044c7a58 >>> ffff8800044c79f8 >>> [ 81.349428] ffffffff810785c3 ffffffff81a13480 0000000000000000 >>> ffff8800001001d0 >>> [ 81.357595] Call Trace: >>> [ 81.360287] [] dump_stack+0x68/0x77 >>> [ 81.365942] [] panic+0xb9/0x219 >>> [ 81.371213] [] ? >>> __blocking_notifier_call_chain+0x63/0x80 >>> [ 81.378971] [] __out_of_memory+0x34e/0x350 >>> [ 81.385292] [] out_of_memory+0x5e/0x90 >>> [ 81.391230] [] >>> __alloc_pages_slowpath+0x6be/0x740 >>> [ 81.398219] [] >>> __alloc_pages_nodemask+0x23c/0x250 >>> [ 81.405212] [] kmem_getpages+0x56/0x110 >>> [ 81.411246] [] fallback_alloc+0x164/0x200 >>> [ 81.417474] [] ____cache_alloc_node+0x8d/0x170 >>> [ 81.424179] [] >>> kmem_cache_alloc_trace+0x17b/0x240 >>> [ 81.431169] [] init_memory_block+0x3a/0x110 >>> [ 81.437586] [] memory_dev_init+0xd7/0x13d >>> [ 81.443810] [] driver_init+0x2f/0x37 >>> [ 81.449556] [] do_basic_setup+0x29/0xd5 >>> [ 81.455597] [] ? sched_init_smp+0x140/0x147 >>> [ 81.462015] [] >>> kernel_init_freeable+0x20e/0x297 >>> [ 81.468815] [] ? rest_init+0x80/0x80 >>> [ 81.474565] [] kernel_init+0x9/0xf0 >>> [ 81.480216] [] ret_from_fork+0x58/0x90 >>> [ 81.486156] [] ? rest_init+0x80/0x80 >>> [ 81.492350] ---[ end Kernel panic - not syncing: Out of memory >>> and no killable processes... >>> [ 81.492350] >>> >>> -Longman >> >> I increased the pre-initialized memory per node in >> update_defer_init() of mm/page_alloc.c from 2G to 4G. Now I am able >> to boot the 24-TB machine without error. The 12-TB has 0.75TB/node, >> while the 24-TB machine has 1.5TB/node. I would suggest something >> like pre-initializing 1G per 0.25TB/node. In this way, it will scale >> properly with the memory size. >> >> Before the patch, the boot time from elilo prompt to ssh login was >> 694s. After the patch, the boot up time was 346s, a saving of 348s >> (about 50%). > > I second scaling the up-front init with the zone size. The 7TB system > I was booting has only 32GB per NUMA node, which at 1GB per 0.25TB > would work out at 128MB up-front init per-NUMA-node, which worked > nice and booted faster yet. > > Even booting with 64MB per NUMA node worked great, so there is > adequate margin for the 8 cores, just I guess we'd need to enforce a > minimum of eg 64MB or so. Varying the synchronous per-NUMA-node initialisation (with non-temporal patch, but that just removes a constant from PMD init), from kernel load to login prompt on this 7TB, 1728-core system takes: 512MB 699.2s 256MB 680.3s 128MB 661.7s 64MB 663.6s 32MB 667.8s So, in this case 128MB per NUMA node gives more locality than 64MB, so should be a good minimum, and matches Waiman's scaling suggestion. Thanks, Daniel -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org