Re: [5.6.0-rc2-next-20200218/powerpc] Boot failure on POWER9

From: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
To: Michael Ellerman <mpe@ellerman.id.au>
Cc: Sachin Sant <sachinp@linux.vnet.ibm.com>,
	Michal Hocko <mhocko@kernel.org>,
	Benjamin Herrenschmidt <benh@kernel.crashing.org>,
	Paul Mackerras <paulus@samba.org>,
	Pekka Enberg <penberg@kernel.org>,
	Linux-Next Mailing List <linux-next@vger.kernel.org>,
	David Rientjes <rientjes@google.com>,
	Christopher Lameter <cl@linux.com>,
	linuxppc-dev@lists.ozlabs.org,
	Joonsoo Kim <iamjoonsoo.kim@lge.com>,
	Kirill Tkhai <ktkhai@virtuozzo.com>,
	Vlastimil Babka <vbabka@suse.cz>
Subject: Re: [5.6.0-rc2-next-20200218/powerpc] Boot failure on POWER9
Date: Fri, 13 Mar 2020 16:42:46 +0530	[thread overview]
Message-ID: <20200313111246.GB25144@linux.vnet.ibm.com> (raw)
In-Reply-To: <875zf8y1i1.fsf@mpe.ellerman.id.au>

* Michael Ellerman <mpe@ellerman.id.au> [2020-03-13 21:48:06]:

> Sachin Sant <sachinp@linux.vnet.ibm.com> writes:
> >> The patch below might work. Sachin can you test this? I tried faking up
> >> a system with a memoryless node zero but couldn't get it to even start
> >> booting.
> >> 
> > The patch did not help. The kernel crashed during
> > the boot with the same call trace.
> >
> > BUG_ON() introduced with the patch was not triggered.
> 
> OK, that's weird.
> 
> I eventually managed to get a memoryless node going in sim, and it
> appears to work there.
> 
> eg in dmesg:
> 
>   [    0.000000][    T0] numa:   NODE_DATA [mem 0x2000fffa2f80-0x2000fffa7fff]
>   [    0.000000][    T0] numa:     NODE_DATA(0) on node 1
>   [    0.000000][    T0] numa:   NODE_DATA [mem 0x2000fff9df00-0x2000fffa2f7f]
>   ...
>   [    0.000000][    T0] Early memory node ranges
>   [    0.000000][    T0]   node   1: [mem 0x0000000000000000-0x00000000ffffffff]
>   [    0.000000][    T0]   node   1: [mem 0x0000200000000000-0x00002000ffffffff]
>   [    0.000000][    T0] Could not find start_pfn for node 0
>   [    0.000000][    T0] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
>   [    0.000000][    T0] On node 0 totalpages: 0
>   [    0.000000][    T0] Initmem setup node 1 [mem 0x0000000000000000-0x00002000ffffffff]
>   [    0.000000][    T0] On node 1 totalpages: 131072
>   
>   # dmesg | grep set_numa
>   [    0.000000][    T0] set_numa_mem: mem node for 0 = 1
>   [    0.005654][    T0] set_numa_mem: mem node for 1 = 1
> 
> So is the problem more than just node zero having no memory?
> 

The problem would happen with possible nodes which are not yet present. i.e
no cpus, no memory attached to those nodes.

Please look at
http://lore.kernel.org/lkml/20200312131438.GB3277@linux.vnet.ibm.com/t/#u
for more details.

The summary being: pgdat/Node_Data for such nodes is not allocated. Hence
the node_present_pages(nid) called  where nid is a possible but not yet
present node fails. Currently node_present_pages(nid) and node_to_mem_node
don't seem to be equipped to handle possible but not present nodes.

> cheers

-- 
Thanks and Regards
Srikar Dronamraju