RE: [PATCH v2 2/9] x86: numa: check the node id consistently for x86

From: Salil Mehta <salil.mehta@huawei.com>
To: Peter Zijlstra <peterz@infradead.org>,
	linyunsheng <linyunsheng@huawei.com>
Cc: "dalias@libc.org" <dalias@libc.org>,
	"linux-sh@vger.kernel.org" <linux-sh@vger.kernel.org>,
	"catalin.marinas@arm.com" <catalin.marinas@arm.com>,
	"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
	"heiko.carstens@de.ibm.com" <heiko.carstens@de.ibm.com>,
	Linuxarm <linuxarm@huawei.com>,
	"jiaxun.yang@flygoat.com" <jiaxun.yang@flygoat.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"mwb@linux.vnet.ibm.com" <mwb@linux.vnet.ibm.com>,
	"paulus@samba.org" <paulus@samba.org>,
	"hpa@zytor.com" <hpa@zytor.com>,
	"sparclinux@vger.kernel.org" <sparclinux@vger.kernel.org>,
	"chenhc@lemote.com" <chenhc@lemote.com>,
	"will@kernel.org" <will@kernel.org>, "cai@lca.pw" <cai@lca.pw>,
	"linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
	"ysato@users.sourceforge.jp" <ysato@users.sourceforge.jp>,
	"x86@kernel.org" <x86@kernel.org>,
	"rppt@linux.ibm.com" <rppt@linux.ibm.com>,
	"borntraeger@de.ibm.com" <borntraeger@de.ibm.com>,
	"dledford@redhat.com" <dledford@redhat.com>,
	"mingo@redhat.com" <mingo@redhat.com>,
	"jeffrey.t.kirsher@intel.com" <jeffrey.t.kirsher@intel.com>,
	"jhogan@kernel.org" <jhogan@kernel.org>,
	"nfont@linux.vnet.ibm.com" <nfont@linux.vnet.ibm.com>,
	"mattst88@gmail.com" <mattst88@gmail.com>,
	"len.brown@intel.com" <len.brown@intel.com>,
	"gor@linux.ibm.com" <gor@linux.ibm.com>,
	"anshuman.khandual@arm.com" <anshuman.khandual@arm.com>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"bp@alien8.de" <bp@alien8.de>,
	"luto@kernel.org" <luto@kernel.org>,
	"tglx@linutronix.de" <tglx@linutronix.de>,
	"naveen.n.rao@linux.vnet.ibm.com"
	<naveen.n.rao@linux.vnet.ibm.com>,
	"linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>,
	"rth@twiddle.net" <rth@twiddle.net>,
	"axboe@kernel.dk" <axboe@kernel.dk>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	"Rafael J. Wysocki" <rjw@rjwysocki.net>,
	"linux-mips@vger.kernel.org" <linux-mips@vger.kernel.org>,
	"ralf@linux-mips.org" <ralf@linux-mips.org>,
	"tbogendoerfer@suse.de" <tbogendoerfer@suse.de>,
	"paul.burton@mips.com" <paul.burton@mips.com>,
	"linux-alpha@vger.kernel.org" <linux-alpha@vger.kernel.org>,
	"ink@jurassic.park.msu.ru" <ink@jurassic.park.msu.ru>,
	"akpm@linux-foundation.org" <akpm@linux-foundation.org>,
	"linuxppc-dev@lists.ozlabs.org" <linuxppc-dev@lists.ozlabs.org>,
	"davem@davemloft.net" <davem@davemloft.net>
Subject: RE: [PATCH v2 2/9] x86: numa: check the node id consistently for x86
Date: Tue, 3 Sep 2019 12:15:24 +0000	[thread overview]
Message-ID: <3bc19c01095545ddbe2ba424f5488b4d@huawei.com> (raw)
In-Reply-To: <20190903071111.GU2369@hirez.programming.kicks-ass.net>

> From: Linuxarm [mailto:linuxarm-bounces@huawei.com] On Behalf Of Peter Zijlstra
> Sent: Tuesday, September 3, 2019 8:11 AM
> 
> On Tue, Sep 03, 2019 at 02:19:04PM +0800, Yunsheng Lin wrote:
> > On 2019/9/2 20:56, Peter Zijlstra wrote:
> > > On Mon, Sep 02, 2019 at 08:25:24PM +0800, Yunsheng Lin wrote:
> > >> On 2019/9/2 15:25, Peter Zijlstra wrote:
> > >>> On Mon, Sep 02, 2019 at 01:46:51PM +0800, Yunsheng Lin wrote:
> > >>>> On 2019/9/1 0:12, Peter Zijlstra wrote:
> > >>>
> > >>>>> 1) because even it is not set, the device really does belong to a node.
> > >>>>> It is impossible a device will have magic uniform access to memory when
> > >>>>> CPUs cannot.
> > >>>>
> > >>>> So it means dev_to_node() will return either NUMA_NO_NODE or a
> > >>>> valid node id?
> > >>>
> > >>> NUMA_NO_NODE := -1, which is not a valid node number. It is also, like> I
> > >>> said, not a valid device location on a NUMA system.
> > >>>
> > >>> Just because ACPI/BIOS is shit, doesn't mean the device doesn't have a
> > >>> node association. It just means we don't know and might have to guess.
> > >>
> > >> How do we guess the device's location when ACPI/BIOS does not set it?
> > >
> > > See device_add(), it looks to the device's parent and on NO_NODE, puts
> > > it there.
> > >
> > > Lacking any hints, just stick it to node0 and print a FW_BUG or
> > > something.
> > >
> > >> It seems dev_to_node() does not do anything about that and leave the
> > >> job to the caller or whatever function that get called with its return
> > >> value, such as cpumask_of_node().
> > >
> > > Well, dev_to_node() doesn't do anything; nor should it. It are the
> > > callers of set_dev_node() that should be taking care.
> > >
> > > Also note how device_add() sets the device node to the parent device's
> > > node on NUMA_NO_NODE. Arguably we should change it to complain when it
> > > finds NUMA_NO_NODE and !parent.
> >
> > Is it possible that the node id set by device_add() become invalid
> > if the node is offlined, then dev_to_node() may return a invalid
> > node id.
> 
> In that case I would expect the device to go away too. Once the memory
> controller goes away, the PCI bus connected to it cannot continue to
> function.

I am not sure if this is *exactly* true on our system as NUMA nodes are
part of the SoCs and devices could still be used even if all the memory
and CPUs part of the node are turned off. Although, it is highly unlikely
anybody would do that(maybe could be debated for the Power Management case?) 

Best Regards
Salil