[PATCH v4 0/2] Fix and enable pmem as RAM device on arm64

[PATCH v4 0/2] Fix and enable pmem as RAM device on arm64

[Jia He]

This fixies a few issues when I tried to enable pmem as RAM device on arm64.

To use memory_add_physaddr_to_nid as a fallback nid, it would be better
implement a general version (__weak) in mm/memory_hotplug. After that, arm64/
sh/s390 can simply use the general version, and PowerPC/ia64/x86 will use
arch specific version.

Tested on ThunderX2 host/qemu "-M virt" guest with a nvdimm device. The
memblocks from the dax pmem device can be either hot-added or hot-removed
on arm64 guest. Also passed the compilation test on x86.

Changes:
v4: - remove "device-dax: use fallback nid when numa_node is invalid", wait
      for Dan Williams' phys_addr_to_target_node() patch
    - folder v3 patch1-4 into single one, no functional changes
v3: https://lkml.org/lkml/2020/7/8/1541
    - introduce general version memory_add_physaddr_to_nid, refine the arch
      specific one
    - fix an uninitialization bug in v2 device-dax patch
v2: https://lkml.org/lkml/2020/7/7/71
    - Drop unnecessary patch to harden try_offline_node
    - Use new solution(by David) to fix dev->target_node=-1 during probing
    - Refine the mem_hotplug_begin/done patch

v1: https://lkml.org/lkml/2020/7/5/381

Jia He (2):
  mm/memory_hotplug: introduce default dummy
    memory_add_physaddr_to_nid()
  mm/memory_hotplug: fix unpaired mem_hotplug_begin/done

 arch/arm64/mm/numa.c | 10 ----------
 arch/ia64/mm/numa.c  |  2 --
 arch/sh/mm/init.c    |  9 ---------
 arch/x86/mm/numa.c   |  1 -
 mm/memory_hotplug.c  | 15 ++++++++++++---
 5 files changed, 12 insertions(+), 25 deletions(-)

-- 
2.17.1

[PATCH 1/2] mm/memory_hotplug: introduce default dummy memory_add_physaddr_to_nid()

[Jia He]

This is to introduce a general dummy helper. memory_add_physaddr_to_nid()
is a fallback option to get the nid in case NUMA_NO_NID is detected.

After this patch, arm64/sh/s390 can simply use the general dummy version.
PowerPC/x86/ia64 will still use their specific version.

This is the preparation to set a fallback value for dev_dax->target_node.

Reviewed-by: David Hildenbrand <david@redhat.com>
Signed-off-by: Jia He <justin.he@arm.com>
---
 arch/arm64/mm/numa.c | 10 ----------
 arch/ia64/mm/numa.c  |  2 --
 arch/sh/mm/init.c    |  9 ---------
 arch/x86/mm/numa.c   |  1 -
 mm/memory_hotplug.c  | 10 ++++++++++
 5 files changed, 10 insertions(+), 22 deletions(-)

diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
index aafcee3e3f7e..73f8b49d485c 100644
--- a/arch/arm64/mm/numa.c
+++ b/arch/arm64/mm/numa.c
@@ -461,13 +461,3 @@ void __init arm64_numa_init(void)
 
 	numa_init(dummy_numa_init);
 }
-
-/*
- * We hope that we will be hotplugging memory on nodes we already know about,
- * such that acpi_get_node() succeeds and we never fall back to this...
- */
-int memory_add_physaddr_to_nid(u64 addr)
-{
-	pr_warn("Unknown node for memory at 0x%llx, assuming node 0\n", addr);
-	return 0;
-}
diff --git a/arch/ia64/mm/numa.c b/arch/ia64/mm/numa.c
index 5e1015eb6d0d..f34964271101 100644
--- a/arch/ia64/mm/numa.c
+++ b/arch/ia64/mm/numa.c
@@ -106,7 +106,5 @@ int memory_add_physaddr_to_nid(u64 addr)
 		return 0;
 	return nid;
 }
-
-EXPORT_SYMBOL_GPL(memory_add_physaddr_to_nid);
 #endif
 #endif
diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
index a70ba0fdd0b3..f75932ba87a6 100644
--- a/arch/sh/mm/init.c
+++ b/arch/sh/mm/init.c
@@ -430,15 +430,6 @@ int arch_add_memory(int nid, u64 start, u64 size,
 	return ret;
 }
 
-#ifdef CONFIG_NUMA
-int memory_add_physaddr_to_nid(u64 addr)
-{
-	/* Node 0 for now.. */
-	return 0;
-}
-EXPORT_SYMBOL_GPL(memory_add_physaddr_to_nid);
-#endif
-
 void arch_remove_memory(int nid, u64 start, u64 size,
 			struct vmem_altmap *altmap)
 {
diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c
index 8ee952038c80..2a6e62af4636 100644
--- a/arch/x86/mm/numa.c
+++ b/arch/x86/mm/numa.c
@@ -929,5 +929,4 @@ int memory_add_physaddr_to_nid(u64 start)
 		nid = numa_meminfo.blk[0].nid;
 	return nid;
 }
-EXPORT_SYMBOL_GPL(memory_add_physaddr_to_nid);
 #endif
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index da374cd3d45b..b49ab743d914 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -350,6 +350,16 @@ int __ref __add_pages(int nid, unsigned long pfn, unsigned long nr_pages,
 	return err;
 }
 
+#ifdef CONFIG_NUMA
+int __weak memory_add_physaddr_to_nid(u64 start)
+{
+	pr_info_once("Unknown target node for memory at 0x%llx, assuming node 0\n",
+			start);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(memory_add_physaddr_to_nid);
+#endif
+
 /* find the smallest valid pfn in the range [start_pfn, end_pfn) */
 static unsigned long find_smallest_section_pfn(int nid, struct zone *zone,
 				     unsigned long start_pfn,
-- 
2.17.1

[PATCH 2/2] mm/memory_hotplug: fix unpaired mem_hotplug_begin/done

[Jia He]

When check_memblock_offlined_cb() returns failed rc(e.g. the memblock is
online at that time), mem_hotplug_begin/done is unpaired in such case.

Therefore a warning:
 Call Trace:
  percpu_up_write+0x33/0x40
  try_remove_memory+0x66/0x120
  ? _cond_resched+0x19/0x30
  remove_memory+0x2b/0x40
  dev_dax_kmem_remove+0x36/0x72 [kmem]
  device_release_driver_internal+0xf0/0x1c0
  device_release_driver+0x12/0x20
  bus_remove_device+0xe1/0x150
  device_del+0x17b/0x3e0
  unregister_dev_dax+0x29/0x60
  devm_action_release+0x15/0x20
  release_nodes+0x19a/0x1e0
  devres_release_all+0x3f/0x50
  device_release_driver_internal+0x100/0x1c0
  driver_detach+0x4c/0x8f
  bus_remove_driver+0x5c/0xd0
  driver_unregister+0x31/0x50
  dax_pmem_exit+0x10/0xfe0 [dax_pmem]

Fixes: f1037ec0cc8a ("mm/memory_hotplug: fix remove_memory() lockdep splat")
Cc: stable@vger.kernel.org # v5.6+
Signed-off-by: Jia He <justin.he@arm.com>
Reviewed-by: David Hildenbrand <david@redhat.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Acked-by: Dan Williams <dan.j.williams@intel.com>
---
 mm/memory_hotplug.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index b49ab743d914..3e0645387daf 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1752,7 +1752,7 @@ static int __ref try_remove_memory(int nid, u64 start, u64 size)
 	 */
 	rc = walk_memory_blocks(start, size, NULL, check_memblock_offlined_cb);
 	if (rc)
-		goto done;
+		return rc;
 
 	/* remove memmap entry */
 	firmware_map_remove(start, start + size, "System RAM");
@@ -1776,9 +1776,8 @@ static int __ref try_remove_memory(int nid, u64 start, u64 size)
 
 	try_offline_node(nid);
 
-done:
 	mem_hotplug_done();
-	return rc;
+	return 0;
 }
 
 /**
-- 
2.17.1

Re: [PATCH v4 0/2] Fix and enable pmem as RAM device on arm64

[David Hildenbrand]

On 10.07.20 05:16, Jia He wrote:
> This fixies a few issues when I tried to enable pmem as RAM device on arm64.
> 
> To use memory_add_physaddr_to_nid as a fallback nid, it would be better
> implement a general version (__weak) in mm/memory_hotplug. After that, arm64/
> sh/s390 can simply use the general version, and PowerPC/ia64/x86 will use
> arch specific version.
> 
> Tested on ThunderX2 host/qemu "-M virt" guest with a nvdimm device. The
> memblocks from the dax pmem device can be either hot-added or hot-removed
> on arm64 guest. Also passed the compilation test on x86.
> 
> Changes:
> v4: - remove "device-dax: use fallback nid when numa_node is invalid", wait
>       for Dan Williams' phys_addr_to_target_node() patch

So, this series no longer does what it promises? "Fix and enable pmem as
RAM device on arm64"

-- 
Thanks,

David / dhildenb

RE: [PATCH v4 0/2] Fix and enable pmem as RAM device on arm64

[Justin He]

DQpIaSBEYXZpZA0KPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBEYXZpZCBI
aWxkZW5icmFuZCA8ZGF2aWRAcmVkaGF0LmNvbT4NCj4gU2VudDogRnJpZGF5LCBKdWx5IDEwLCAy
MDIwIDQ6MzAgUE0NCj4gVG86IEp1c3RpbiBIZSA8SnVzdGluLkhlQGFybS5jb20+OyBDYXRhbGlu
IE1hcmluYXMNCj4gPENhdGFsaW4uTWFyaW5hc0Bhcm0uY29tPjsgV2lsbCBEZWFjb24gPHdpbGxA
a2VybmVsLm9yZz47IFRvbnkgTHVjaw0KPiA8dG9ueS5sdWNrQGludGVsLmNvbT47IEZlbmdodWEg
WXUgPGZlbmdodWEueXVAaW50ZWwuY29tPjsgWW9zaGlub3JpIFNhdG8NCj4gPHlzYXRvQHVzZXJz
LnNvdXJjZWZvcmdlLmpwPjsgUmljaCBGZWxrZXIgPGRhbGlhc0BsaWJjLm9yZz47IERhdmUgSGFu
c2VuDQo+IDxkYXZlLmhhbnNlbkBsaW51eC5pbnRlbC5jb20+OyBBbmR5IEx1dG9taXJza2kgPGx1
dG9Aa2VybmVsLm9yZz47IFBldGVyDQo+IFppamxzdHJhIDxwZXRlcnpAaW5mcmFkZWFkLm9yZz47
IFRob21hcyBHbGVpeG5lciA8dGdseEBsaW51dHJvbml4LmRlPjsNCj4gSW5nbyBNb2xuYXIgPG1p
bmdvQHJlZGhhdC5jb20+OyBCb3Jpc2xhdiBQZXRrb3YgPGJwQGFsaWVuOC5kZT4NCj4gQ2M6IHg4
NkBrZXJuZWwub3JnOyBILiBQZXRlciBBbnZpbiA8aHBhQHp5dG9yLmNvbT47IERhbiBXaWxsaWFt
cw0KPiA8ZGFuLmoud2lsbGlhbXNAaW50ZWwuY29tPjsgVmlzaGFsIFZlcm1hIDx2aXNoYWwubC52
ZXJtYUBpbnRlbC5jb20+OyBEYXZlDQo+IEppYW5nIDxkYXZlLmppYW5nQGludGVsLmNvbT47IEFu
ZHJldyBNb3J0b24gPGFrcG1AbGludXgtZm91bmRhdGlvbi5vcmc+Ow0KPiBCYW9xdWFuIEhlIDxi
aGVAcmVkaGF0LmNvbT47IENodWhvbmcgWXVhbiA8aHNsZXN0ZXI5NkBnbWFpbC5jb20+OyBNaWtl
DQo+IFJhcG9wb3J0IDxycHB0QGxpbnV4LmlibS5jb20+OyBMb2dhbiBHdW50aG9ycGUgPGxvZ2Fu
Z0BkZWx0YXRlZS5jb20+Ow0KPiBNYXNhaGlybyBZYW1hZGEgPG1hc2FoaXJveUBrZXJuZWwub3Jn
PjsgTWljaGFsIEhvY2tvIDxtaG9ja29Ac3VzZS5jb20+Ow0KPiBsaW51eC1hcm0ta2VybmVsQGxp
c3RzLmluZnJhZGVhZC5vcmc7IGxpbnV4LWtlcm5lbEB2Z2VyLmtlcm5lbC5vcmc7IGxpbnV4LQ0K
PiBpYTY0QHZnZXIua2VybmVsLm9yZzsgbGludXgtc2hAdmdlci5rZXJuZWwub3JnOyBsaW51eC1u
dmRpbW1AbGlzdHMuMDEub3JnOw0KPiBsaW51eC1tbUBrdmFjay5vcmc7IEpvbmF0aGFuIENhbWVy
b24gPEpvbmF0aGFuLkNhbWVyb25ASHVhd2VpLmNvbT47IEthbHkNCj4gWGluIDxLYWx5LlhpbkBh
cm0uY29tPg0KPiBTdWJqZWN0OiBSZTogW1BBVENIIHY0IDAvMl0gRml4IGFuZCBlbmFibGUgcG1l
bSBhcyBSQU0gZGV2aWNlIG9uIGFybTY0DQo+IA0KPiBPbiAxMC4wNy4yMCAwNToxNiwgSmlhIEhl
IHdyb3RlOg0KPiA+IFRoaXMgZml4aWVzIGEgZmV3IGlzc3VlcyB3aGVuIEkgdHJpZWQgdG8gZW5h
YmxlIHBtZW0gYXMgUkFNIGRldmljZSBvbg0KPiBhcm02NC4NCj4gPg0KPiA+IFRvIHVzZSBtZW1v
cnlfYWRkX3BoeXNhZGRyX3RvX25pZCBhcyBhIGZhbGxiYWNrIG5pZCwgaXQgd291bGQgYmUgYmV0
dGVyDQo+ID4gaW1wbGVtZW50IGEgZ2VuZXJhbCB2ZXJzaW9uIChfX3dlYWspIGluIG1tL21lbW9y
eV9ob3RwbHVnLiBBZnRlciB0aGF0LA0KPiBhcm02NC8NCj4gPiBzaC9zMzkwIGNhbiBzaW1wbHkg
dXNlIHRoZSBnZW5lcmFsIHZlcnNpb24sIGFuZCBQb3dlclBDL2lhNjQveDg2IHdpbGwNCj4gdXNl
DQo+ID4gYXJjaCBzcGVjaWZpYyB2ZXJzaW9uLg0KPiA+DQo+ID4gVGVzdGVkIG9uIFRodW5kZXJY
MiBob3N0L3FlbXUgIi1NIHZpcnQiIGd1ZXN0IHdpdGggYSBudmRpbW0gZGV2aWNlLiBUaGUNCj4g
PiBtZW1ibG9ja3MgZnJvbSB0aGUgZGF4IHBtZW0gZGV2aWNlIGNhbiBiZSBlaXRoZXIgaG90LWFk
ZGVkIG9yIGhvdC0NCj4gcmVtb3ZlZA0KPiA+IG9uIGFybTY0IGd1ZXN0LiBBbHNvIHBhc3NlZCB0
aGUgY29tcGlsYXRpb24gdGVzdCBvbiB4ODYuDQo+ID4NCj4gPiBDaGFuZ2VzOg0KPiA+IHY0OiAt
IHJlbW92ZSAiZGV2aWNlLWRheDogdXNlIGZhbGxiYWNrIG5pZCB3aGVuIG51bWFfbm9kZSBpcyBp
bnZhbGlkIiwNCj4gd2FpdA0KPiA+ICAgICAgIGZvciBEYW4gV2lsbGlhbXMnIHBoeXNfYWRkcl90
b190YXJnZXRfbm9kZSgpIHBhdGNoDQo+IA0KPiBTbywgdGhpcyBzZXJpZXMgbm8gbG9uZ2VyIGRv
ZXMgd2hhdCBpdCBwcm9taXNlcz8gIkZpeCBhbmQgZW5hYmxlIHBtZW0gYXMNCj4gUkFNIGRldmlj
ZSBvbiBhcm02NCINCj4gDQpIbW0sIGEgbGl0dGxlIGJpdCBhd2t3YXJkIGJ1dCBzZWVtcyBubyBs
b25nIHdoYXQgaXQgcHJvbWlzZXMuIEhvdyBhYm91dA0Kc2VuZGluZyBwYXRjaDEgcGF0Y2gyIGlu
ZGl2aWR1YWxseSB3aXRob3V0IHRoaXMgY292ZXItbGV0dGVyPw0KDQotLQ0KQ2hlZXJzLA0KSnVz
dGluIChKaWEgSGUpDQoNCg0K

Re: [PATCH v4 0/2] Fix and enable pmem as RAM device on arm64

[Dan Williams]

On Thu, Jul 9, 2020 at 8:17 PM Jia He <justin.he@arm.com> wrote:
>
> This fixies a few issues when I tried to enable pmem as RAM device on arm64.
>
> To use memory_add_physaddr_to_nid as a fallback nid, it would be better
> implement a general version (__weak) in mm/memory_hotplug. After that, arm64/
> sh/s390 can simply use the general version, and PowerPC/ia64/x86 will use
> arch specific version.
>
> Tested on ThunderX2 host/qemu "-M virt" guest with a nvdimm device. The
> memblocks from the dax pmem device can be either hot-added or hot-removed
> on arm64 guest. Also passed the compilation test on x86.
>
> Changes:
> v4: - remove "device-dax: use fallback nid when numa_node is invalid", wait
>       for Dan Williams' phys_addr_to_target_node() patch

FWIW, I put these patches through a 0-day run overnight and will be
posting them today.