RE: [PATCH 1/3] arm64/numa: set numa_off to false when numa node is fake

From: Justin He <Justin.He@arm.com>
To: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
Cc: Catalin Marinas <Catalin.Marinas@arm.com>,
	Will Deacon <will@kernel.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Mike Rapoport <rppt@linux.ibm.com>, Baoquan He <bhe@redhat.com>,
	Chuhong Yuan <hslester96@gmail.com>,
	"linux-arm-kernel@lists.infradead.org" 
	<linux-arm-kernel@lists.infradead.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-mm@kvack.org" <linux-mm@kvack.org>,
	Kaly Xin <Kaly.Xin@arm.com>, David Hildenbrand <david@redhat.com>
Subject: RE: [PATCH 1/3] arm64/numa: set numa_off to false when numa node is fake
Date: Mon, 6 Jul 2020 12:47:51 +0000	[thread overview]
Message-ID: <AM6PR08MB4069BCD0E17BD37CC5591C63F7690@AM6PR08MB4069.eurprd08.prod.outlook.com> (raw)
In-Reply-To: <20200706114605.000050ac@Huawei.com>

Hi Jonathan, thanks for the comments.

> -----Original Message-----
> From: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
> Sent: Monday, July 6, 2020 6:46 PM
> To: Justin He <Justin.He@arm.com>
> Cc: Catalin Marinas <Catalin.Marinas@arm.com>; Will Deacon
> <will@kernel.org>; Andrew Morton <akpm@linux-foundation.org>; Mike
> Rapoport <rppt@linux.ibm.com>; Baoquan He <bhe@redhat.com>; Chuhong Yuan
> <hslester96@gmail.com>; linux-arm-kernel@lists.infradead.org; linux-
> kernel@vger.kernel.org; linux-mm@kvack.org; Kaly Xin <Kaly.Xin@arm.com>
> Subject: Re: [PATCH 1/3] arm64/numa: set numa_off to false when numa node
> is fake
> 
> On Mon, 6 Jul 2020 11:29:21 +0100
> Jonathan Cameron <Jonathan.Cameron@Huawei.com> wrote:
> 
> > On Mon, 6 Jul 2020 09:19:45 +0800
> > Jia He <justin.he@arm.com> wrote:
> >
> > Hi,
> >
> > > Previously, numa_off is set to true unconditionally in
> dummy_numa_init(),
> > > even if there is a fake numa node.
> > >
> > > But acpi will translate node id to NUMA_NO_NODE(-1) in
> acpi_map_pxm_to_node()
> > > because it regards numa_off as turning off the numa node.
> >
> > That is correct.  It is operating exactly as it should, if SRAT hasn't
> been parsed
> > and you are on ACPI platform there are no nodes.  They cannot be created
> at
> > some later date.  The dummy code doesn't change this. It just does
> enough to carry
> > on operating with no specified nodes.
> >
> > >
> > > Without this patch, pmem can't be probed as a RAM device on arm64 if
> SRAT table
> > > isn't present.
> > >
> > > $ndctl create-namespace -fe namespace0.0 --mode=devdax --map=dev -s 1g
> -a 64K
> > > kmem dax0.0: rejecting DAX region [mem 0x240400000-0x2bfffffff] with
> invalid node: -1
> > > kmem: probe of dax0.0 failed with error -22
> > >
> > > This fixes it by setting numa_off to false.
> >
> > Without the SRAT protection patch [1] you may well run into problems

Sorry, doesn't quite understand here. Do you mean your [1] can resolve this
issue? But acpi_map_pxm_to_node() has returned with NUMA_NO_NODE after
following check:
	if (pxm < 0 || pxm >= MAX_PXM_DOMAINS || numa_off)
		return NUMA_NO_NODE;
Seems even with your [1] patch, it is not helpful? Thanks for clarification
if my understanding is wrong.
[1] https://patchwork.kernel.org/patch/11632063/

> > because someone somewhere will have _PXM in a DSDT but will
> > have a non existent SRAT.   We had this happen on an AMD platform when
> we
> > tried to introduce working _PXM support for PCI. [2]
> >
> > So whilst this seems superficially safe, I'd definitely be crossing your
> fingers.
> > Note, at that time I proposed putting the numa_off = false into the x86
> code
> > path precisely to cut out that possibility (was rejected at the time, at
> least
> > partly because the clarifications to the ACPI spec were not pubilc.)
> >
> > The patch in [1] should sort things out however by ensuring we only
> create
> > new domains where we should actually be doing so. However, in your case
> > it will return NUMA_NO_NODE anyway so this isn't the right way to fix
> things.

Okay, let me try to summarize, there might be 3 possible fixing ways:
1. this patch, seems it is not satisfied by you and David 😉
2. my previous proposal [2], similar as what David suggested
3. remove numa_off check in acpi_map_pxm_to_node()
e.g.
...
	if (pxm < 0 || pxm >= MAX_PXM_DOMAINS /*|| numa_off*/)
		return NUMA_NO_NODE;

[2] https://lkml.org/lkml/2019/8/16/367

--
Cheers,
Justin (Jia He)