nvdimm.lists.linux.dev archive mirror
 help / color / mirror / Atom feed
* [PATCH v2] device-dax: use fallback nid when numa node is invalid
@ 2021-09-10 12:46 Jia He
  2021-09-10 15:42 ` Dan Williams
  0 siblings, 1 reply; 6+ messages in thread
From: Jia He @ 2021-09-10 12:46 UTC (permalink / raw)
  To: Dan Williams, Vishal Verma, Dave Jiang, David Hildenbrand
  Cc: nvdimm, linux-kernel, Jia He

Previously, numa_off was set unconditionally in dummy_numa_init()
even with a fake numa node. Then ACPI sets node id as NUMA_NO_NODE(-1)
after acpi_map_pxm_to_node() because it regards numa_off as turning
off the numa node. Hence dev_dax->target_node is NUMA_NO_NODE on
arm64 with fake numa case.

Without this patch, pmem can't be probed as RAM devices on arm64 if
SRAT table isn't present:
  $ndctl create-namespace -fe namespace0.0 --mode=devdax --map=dev -s 1g -a 64K
  kmem dax0.0: rejecting DAX region [mem 0x240400000-0x2bfffffff] with invalid node: -1
  kmem: probe of dax0.0 failed with error -22

This fixes it by using fallback memory_add_physaddr_to_nid() as nid.

Suggested-by: David Hildenbrand <david@redhat.com>
Signed-off-by: Jia He <justin.he@arm.com>
---
v2: - rebase it based on David's "memory group" patch.
    - drop the changes in dev_dax_kmem_remove() since nid had been 
      removed in remove_memory().
 drivers/dax/kmem.c | 31 +++++++++++++++++--------------
 1 file changed, 17 insertions(+), 14 deletions(-)

diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
index a37622060fff..e4836eb7539e 100644
--- a/drivers/dax/kmem.c
+++ b/drivers/dax/kmem.c
@@ -47,20 +47,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
 	unsigned long total_len = 0;
 	struct dax_kmem_data *data;
 	int i, rc, mapped = 0;
-	int numa_node;
-
-	/*
-	 * Ensure good NUMA information for the persistent memory.
-	 * Without this check, there is a risk that slow memory
-	 * could be mixed in a node with faster memory, causing
-	 * unavoidable performance issues.
-	 */
-	numa_node = dev_dax->target_node;
-	if (numa_node < 0) {
-		dev_warn(dev, "rejecting DAX region with invalid node: %d\n",
-				numa_node);
-		return -EINVAL;
-	}
+	int numa_node = dev_dax->target_node;
 
 	for (i = 0; i < dev_dax->nr_range; i++) {
 		struct range range;
@@ -71,6 +58,22 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
 					i, range.start, range.end);
 			continue;
 		}
+
+		/*
+		 * Ensure good NUMA information for the persistent memory.
+		 * Without this check, there is a risk but not fatal that slow
+		 * memory could be mixed in a node with faster memory, causing
+		 * unavoidable performance issues. Warn this and use fallback
+		 * node id.
+		 */
+		if (numa_node < 0) {
+			int new_node = memory_add_physaddr_to_nid(range.start);
+
+			dev_info(dev, "changing nid from %d to %d for DAX region [%#llx-%#llx]\n",
+				 numa_node, new_node, range.start, range.end);
+			numa_node = new_node;
+		}
+
 		total_len += range_len(&range);
 	}
 
-- 
2.17.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH v2] device-dax: use fallback nid when numa node is invalid
  2021-09-10 12:46 [PATCH v2] device-dax: use fallback nid when numa node is invalid Jia He
@ 2021-09-10 15:42 ` Dan Williams
  2021-09-14  2:06   ` Justin He
  0 siblings, 1 reply; 6+ messages in thread
From: Dan Williams @ 2021-09-10 15:42 UTC (permalink / raw)
  To: Jia He
  Cc: Vishal Verma, Dave Jiang, David Hildenbrand, Linux NVDIMM,
	Linux Kernel Mailing List

On Fri, Sep 10, 2021 at 5:46 AM Jia He <justin.he@arm.com> wrote:
>
> Previously, numa_off was set unconditionally in dummy_numa_init()
> even with a fake numa node. Then ACPI sets node id as NUMA_NO_NODE(-1)
> after acpi_map_pxm_to_node() because it regards numa_off as turning
> off the numa node. Hence dev_dax->target_node is NUMA_NO_NODE on
> arm64 with fake numa case.
>
> Without this patch, pmem can't be probed as RAM devices on arm64 if
> SRAT table isn't present:
>   $ndctl create-namespace -fe namespace0.0 --mode=devdax --map=dev -s 1g -a 64K
>   kmem dax0.0: rejecting DAX region [mem 0x240400000-0x2bfffffff] with invalid node: -1
>   kmem: probe of dax0.0 failed with error -22
>
> This fixes it by using fallback memory_add_physaddr_to_nid() as nid.
>
> Suggested-by: David Hildenbrand <david@redhat.com>
> Signed-off-by: Jia He <justin.he@arm.com>
> ---
> v2: - rebase it based on David's "memory group" patch.
>     - drop the changes in dev_dax_kmem_remove() since nid had been
>       removed in remove_memory().
>  drivers/dax/kmem.c | 31 +++++++++++++++++--------------
>  1 file changed, 17 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
> index a37622060fff..e4836eb7539e 100644
> --- a/drivers/dax/kmem.c
> +++ b/drivers/dax/kmem.c
> @@ -47,20 +47,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
>         unsigned long total_len = 0;
>         struct dax_kmem_data *data;
>         int i, rc, mapped = 0;
> -       int numa_node;
> -
> -       /*
> -        * Ensure good NUMA information for the persistent memory.
> -        * Without this check, there is a risk that slow memory
> -        * could be mixed in a node with faster memory, causing
> -        * unavoidable performance issues.
> -        */
> -       numa_node = dev_dax->target_node;
> -       if (numa_node < 0) {
> -               dev_warn(dev, "rejecting DAX region with invalid node: %d\n",
> -                               numa_node);
> -               return -EINVAL;
> -       }
> +       int numa_node = dev_dax->target_node;
>
>         for (i = 0; i < dev_dax->nr_range; i++) {
>                 struct range range;
> @@ -71,6 +58,22 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
>                                         i, range.start, range.end);
>                         continue;
>                 }
> +
> +               /*
> +                * Ensure good NUMA information for the persistent memory.
> +                * Without this check, there is a risk but not fatal that slow
> +                * memory could be mixed in a node with faster memory, causing
> +                * unavoidable performance issues. Warn this and use fallback
> +                * node id.
> +                */
> +               if (numa_node < 0) {
> +                       int new_node = memory_add_physaddr_to_nid(range.start);
> +
> +                       dev_info(dev, "changing nid from %d to %d for DAX region [%#llx-%#llx]\n",
> +                                numa_node, new_node, range.start, range.end);
> +                       numa_node = new_node;
> +               }
> +
>                 total_len += range_len(&range);

This fallback change belongs where the parent region for the namespace
adopts its target_node, because it's not clear
memory_add_physaddr_to_nid() is the right fallback in all situations.
Here is where this setting is happening currently:

drivers/acpi/nfit/core.c:3004:          ndr_desc->target_node =
pxm_to_node(spa->proximity_domain);
drivers/acpi/nfit/core.c:3007:          ndr_desc->target_node = NUMA_NO_NODE;
drivers/nvdimm/e820.c:29:       ndr_desc.target_node = nid;
drivers/nvdimm/of_pmem.c:58:            ndr_desc.target_node =
ndr_desc.numa_node;
drivers/nvdimm/region_devs.c:1127:      nd_region->target_node =
ndr_desc->target_node;

...where is this pmem region originating on this arm64 platform?

^ permalink raw reply	[flat|nested] 6+ messages in thread

* RE: [PATCH v2] device-dax: use fallback nid when numa node is invalid
  2021-09-10 15:42 ` Dan Williams
@ 2021-09-14  2:06   ` Justin He
  2021-09-15  5:15     ` Dan Williams
  0 siblings, 1 reply; 6+ messages in thread
From: Justin He @ 2021-09-14  2:06 UTC (permalink / raw)
  To: Dan Williams
  Cc: Vishal Verma, Dave Jiang, David Hildenbrand, Linux NVDIMM,
	Linux Kernel Mailing List, nd

Hi Dan,

> -----Original Message-----
> From: Dan Williams <dan.j.williams@intel.com>
> Sent: Friday, September 10, 2021 11:42 PM
> To: Justin He <Justin.He@arm.com>
> Cc: Vishal Verma <vishal.l.verma@intel.com>; Dave Jiang
> <dave.jiang@intel.com>; David Hildenbrand <david@redhat.com>; Linux NVDIMM
> <nvdimm@lists.linux.dev>; Linux Kernel Mailing List <linux-
> kernel@vger.kernel.org>
> Subject: Re: [PATCH v2] device-dax: use fallback nid when numa node is
> invalid
> 
> On Fri, Sep 10, 2021 at 5:46 AM Jia He <justin.he@arm.com> wrote:
> >
> > Previously, numa_off was set unconditionally in dummy_numa_init()
> > even with a fake numa node. Then ACPI sets node id as NUMA_NO_NODE(-1)
> > after acpi_map_pxm_to_node() because it regards numa_off as turning
> > off the numa node. Hence dev_dax->target_node is NUMA_NO_NODE on
> > arm64 with fake numa case.
> >
> > Without this patch, pmem can't be probed as RAM devices on arm64 if
> > SRAT table isn't present:
> >   $ndctl create-namespace -fe namespace0.0 --mode=devdax --map=dev -s 1g
> -a 64K
> >   kmem dax0.0: rejecting DAX region [mem 0x240400000-0x2bfffffff] with
> invalid node: -1
> >   kmem: probe of dax0.0 failed with error -22
> >
> > This fixes it by using fallback memory_add_physaddr_to_nid() as nid.
> >
> > Suggested-by: David Hildenbrand <david@redhat.com>
> > Signed-off-by: Jia He <justin.he@arm.com>
> > ---
> > v2: - rebase it based on David's "memory group" patch.
> >     - drop the changes in dev_dax_kmem_remove() since nid had been
> >       removed in remove_memory().
> >  drivers/dax/kmem.c | 31 +++++++++++++++++--------------
> >  1 file changed, 17 insertions(+), 14 deletions(-)
> >
> > diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
> > index a37622060fff..e4836eb7539e 100644
> > --- a/drivers/dax/kmem.c
> > +++ b/drivers/dax/kmem.c
> > @@ -47,20 +47,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
> >         unsigned long total_len = 0;
> >         struct dax_kmem_data *data;
> >         int i, rc, mapped = 0;
> > -       int numa_node;
> > -
> > -       /*
> > -        * Ensure good NUMA information for the persistent memory.
> > -        * Without this check, there is a risk that slow memory
> > -        * could be mixed in a node with faster memory, causing
> > -        * unavoidable performance issues.
> > -        */
> > -       numa_node = dev_dax->target_node;
> > -       if (numa_node < 0) {
> > -               dev_warn(dev, "rejecting DAX region with invalid
> node: %d\n",
> > -                               numa_node);
> > -               return -EINVAL;
> > -       }
> > +       int numa_node = dev_dax->target_node;
> >
> >         for (i = 0; i < dev_dax->nr_range; i++) {
> >                 struct range range;
> > @@ -71,6 +58,22 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
> >                                         i, range.start, range.end);
> >                         continue;
> >                 }
> > +
> > +               /*
> > +                * Ensure good NUMA information for the persistent
> memory.
> > +                * Without this check, there is a risk but not fatal
> that slow
> > +                * memory could be mixed in a node with faster memory,
> causing
> > +                * unavoidable performance issues. Warn this and use
> fallback
> > +                * node id.
> > +                */
> > +               if (numa_node < 0) {
> > +                       int new_node =
> memory_add_physaddr_to_nid(range.start);
> > +
> > +                       dev_info(dev, "changing nid from %d to %d for
> DAX region [%#llx-%#llx]\n",
> > +                                numa_node, new_node, range.start,
> range.end);
> > +                       numa_node = new_node;
> > +               }
> > +
> >                 total_len += range_len(&range);
> 
> This fallback change belongs where the parent region for the namespace
> adopts its target_node, because it's not clear
> memory_add_physaddr_to_nid() is the right fallback in all situations.
> Here is where this setting is happening currently:
> 
> drivers/acpi/nfit/core.c:3004:          ndr_desc->target_node =
> pxm_to_node(spa->proximity_domain);
On my local arm64 guest('virt' machine type), the target_node is
set to -1 at this line.
That is:
The condition "spa->flags & ACPI_NFIT_PROXIMITY_VALID" is hit.

> drivers/acpi/nfit/core.c:3007:          ndr_desc->target_node =
> NUMA_NO_NODE;
> drivers/nvdimm/e820.c:29:       ndr_desc.target_node = nid;
> drivers/nvdimm/of_pmem.c:58:            ndr_desc.target_node =
> ndr_desc.numa_node;
> drivers/nvdimm/region_devs.c:1127:      nd_region->target_node =
> ndr_desc->target_node;


Sorry,Dan. I thought I missed your previous mail:

=========================================
Looks like it is the NFIT driver, thanks.

If you're getting NUMA_NO_NODE in dax_kmem from the NFIT driver in
means your ACPI NFIT table is failing to populate correct numa
information. You could try the following to fix it up, but I think the
real problem is that your platform BIOS needs to add the proper numa
data.

diff --git a/drivers/acpi/nfit/core.c b/drivers/acpi/nfit/core.c
index fb775b967c52..d3a0cec635b1 100644
--- a/drivers/acpi/nfit/core.c
+++ b/drivers/acpi/nfit/core.c
@@ -3005,15 +3005,8 @@ static int acpi_nfit_register_region(struct
acpi_nfit_desc *acpi_desc,
        ndr_desc->res = &res;
        ndr_desc->provider_data = nfit_spa;
        ndr_desc->attr_groups = acpi_nfit_region_attribute_groups;
-       if (spa->flags & ACPI_NFIT_PROXIMITY_VALID) {
-               ndr_desc->numa_node = acpi_map_pxm_to_online_node(
-                                               spa->proximity_domain);
-               ndr_desc->target_node = acpi_map_pxm_to_node(
-                               spa->proximity_domain);
-       } else {
-               ndr_desc->numa_node = NUMA_NO_NODE;
-               ndr_desc->target_node = NUMA_NO_NODE;
-       }
+       ndr_desc->numa_node = memory_add_physaddr_to_nid(spa->address);
+       ndr_desc->target_node = phys_to_target_node(spa->address);

        /*
         * Persistence domain bits are hierarchical, if
===================================================

Do you still suggest fixing like this?


--
Cheers,
Justin (Jia He)




^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH v2] device-dax: use fallback nid when numa node is invalid
  2021-09-14  2:06   ` Justin He
@ 2021-09-15  5:15     ` Dan Williams
  2021-09-15  6:50       ` Justin He
  0 siblings, 1 reply; 6+ messages in thread
From: Dan Williams @ 2021-09-15  5:15 UTC (permalink / raw)
  To: Justin He
  Cc: Vishal Verma, Dave Jiang, David Hildenbrand, Linux NVDIMM,
	Linux Kernel Mailing List, nd

On Mon, Sep 13, 2021 at 7:06 PM Justin He <Justin.He@arm.com> wrote:
>
> Hi Dan,
>
> > -----Original Message-----
> > From: Dan Williams <dan.j.williams@intel.com>
> > Sent: Friday, September 10, 2021 11:42 PM
> > To: Justin He <Justin.He@arm.com>
> > Cc: Vishal Verma <vishal.l.verma@intel.com>; Dave Jiang
> > <dave.jiang@intel.com>; David Hildenbrand <david@redhat.com>; Linux NVDIMM
> > <nvdimm@lists.linux.dev>; Linux Kernel Mailing List <linux-
> > kernel@vger.kernel.org>
> > Subject: Re: [PATCH v2] device-dax: use fallback nid when numa node is
> > invalid
> >
> > On Fri, Sep 10, 2021 at 5:46 AM Jia He <justin.he@arm.com> wrote:
> > >
> > > Previously, numa_off was set unconditionally in dummy_numa_init()
> > > even with a fake numa node. Then ACPI sets node id as NUMA_NO_NODE(-1)
> > > after acpi_map_pxm_to_node() because it regards numa_off as turning
> > > off the numa node. Hence dev_dax->target_node is NUMA_NO_NODE on
> > > arm64 with fake numa case.
> > >
> > > Without this patch, pmem can't be probed as RAM devices on arm64 if
> > > SRAT table isn't present:
> > >   $ndctl create-namespace -fe namespace0.0 --mode=devdax --map=dev -s 1g
> > -a 64K
> > >   kmem dax0.0: rejecting DAX region [mem 0x240400000-0x2bfffffff] with
> > invalid node: -1
> > >   kmem: probe of dax0.0 failed with error -22
> > >
> > > This fixes it by using fallback memory_add_physaddr_to_nid() as nid.
> > >
> > > Suggested-by: David Hildenbrand <david@redhat.com>
> > > Signed-off-by: Jia He <justin.he@arm.com>
> > > ---
> > > v2: - rebase it based on David's "memory group" patch.
> > >     - drop the changes in dev_dax_kmem_remove() since nid had been
> > >       removed in remove_memory().
> > >  drivers/dax/kmem.c | 31 +++++++++++++++++--------------
> > >  1 file changed, 17 insertions(+), 14 deletions(-)
> > >
> > > diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
> > > index a37622060fff..e4836eb7539e 100644
> > > --- a/drivers/dax/kmem.c
> > > +++ b/drivers/dax/kmem.c
> > > @@ -47,20 +47,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
> > >         unsigned long total_len = 0;
> > >         struct dax_kmem_data *data;
> > >         int i, rc, mapped = 0;
> > > -       int numa_node;
> > > -
> > > -       /*
> > > -        * Ensure good NUMA information for the persistent memory.
> > > -        * Without this check, there is a risk that slow memory
> > > -        * could be mixed in a node with faster memory, causing
> > > -        * unavoidable performance issues.
> > > -        */
> > > -       numa_node = dev_dax->target_node;
> > > -       if (numa_node < 0) {
> > > -               dev_warn(dev, "rejecting DAX region with invalid
> > node: %d\n",
> > > -                               numa_node);
> > > -               return -EINVAL;
> > > -       }
> > > +       int numa_node = dev_dax->target_node;
> > >
> > >         for (i = 0; i < dev_dax->nr_range; i++) {
> > >                 struct range range;
> > > @@ -71,6 +58,22 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
> > >                                         i, range.start, range.end);
> > >                         continue;
> > >                 }
> > > +
> > > +               /*
> > > +                * Ensure good NUMA information for the persistent
> > memory.
> > > +                * Without this check, there is a risk but not fatal
> > that slow
> > > +                * memory could be mixed in a node with faster memory,
> > causing
> > > +                * unavoidable performance issues. Warn this and use
> > fallback
> > > +                * node id.
> > > +                */
> > > +               if (numa_node < 0) {
> > > +                       int new_node =
> > memory_add_physaddr_to_nid(range.start);
> > > +
> > > +                       dev_info(dev, "changing nid from %d to %d for
> > DAX region [%#llx-%#llx]\n",
> > > +                                numa_node, new_node, range.start,
> > range.end);
> > > +                       numa_node = new_node;
> > > +               }
> > > +
> > >                 total_len += range_len(&range);
> >
> > This fallback change belongs where the parent region for the namespace
> > adopts its target_node, because it's not clear
> > memory_add_physaddr_to_nid() is the right fallback in all situations.
> > Here is where this setting is happening currently:
> >
> > drivers/acpi/nfit/core.c:3004:          ndr_desc->target_node =
> > pxm_to_node(spa->proximity_domain);
> On my local arm64 guest('virt' machine type), the target_node is
> set to -1 at this line.
> That is:
> The condition "spa->flags & ACPI_NFIT_PROXIMITY_VALID" is hit.
>
> > drivers/acpi/nfit/core.c:3007:          ndr_desc->target_node =
> > NUMA_NO_NODE;
> > drivers/nvdimm/e820.c:29:       ndr_desc.target_node = nid;
> > drivers/nvdimm/of_pmem.c:58:            ndr_desc.target_node =
> > ndr_desc.numa_node;
> > drivers/nvdimm/region_devs.c:1127:      nd_region->target_node =
> > ndr_desc->target_node;
>
>
> Sorry,Dan. I thought I missed your previous mail:
>
> =========================================
> Looks like it is the NFIT driver, thanks.
>
> If you're getting NUMA_NO_NODE in dax_kmem from the NFIT driver in
> means your ACPI NFIT table is failing to populate correct numa
> information. You could try the following to fix it up, but I think the
> real problem is that your platform BIOS needs to add the proper numa
> data.
>
> diff --git a/drivers/acpi/nfit/core.c b/drivers/acpi/nfit/core.c
> index fb775b967c52..d3a0cec635b1 100644
> --- a/drivers/acpi/nfit/core.c
> +++ b/drivers/acpi/nfit/core.c
> @@ -3005,15 +3005,8 @@ static int acpi_nfit_register_region(struct
> acpi_nfit_desc *acpi_desc,
>         ndr_desc->res = &res;
>         ndr_desc->provider_data = nfit_spa;
>         ndr_desc->attr_groups = acpi_nfit_region_attribute_groups;
> -       if (spa->flags & ACPI_NFIT_PROXIMITY_VALID) {
> -               ndr_desc->numa_node = acpi_map_pxm_to_online_node(
> -                                               spa->proximity_domain);
> -               ndr_desc->target_node = acpi_map_pxm_to_node(
> -                               spa->proximity_domain);
> -       } else {
> -               ndr_desc->numa_node = NUMA_NO_NODE;
> -               ndr_desc->target_node = NUMA_NO_NODE;
> -       }
> +       ndr_desc->numa_node = memory_add_physaddr_to_nid(spa->address);
> +       ndr_desc->target_node = phys_to_target_node(spa->address);
>
>         /*
>          * Persistence domain bits are hierarchical, if
> ===================================================
>
> Do you still suggest fixing like this?

Are you saying that ACPI_NFIT_PROXIMITY_VALID is not set on your
platform, or that pxm_to_node() returns NUMA_NO_NODE?

I would expect something like this:

diff --git a/drivers/acpi/nfit/core.c b/drivers/acpi/nfit/core.c
index a3ef6cce644c..95de7dc18ed8 100644
--- a/drivers/acpi/nfit/core.c
+++ b/drivers/acpi/nfit/core.c
@@ -3007,6 +3007,15 @@ static int acpi_nfit_register_region(struct
acpi_nfit_desc *acpi_desc,
                ndr_desc->target_node = NUMA_NO_NODE;
        }

+       /*
+        * Fallback to address based numa information if node lookup
+        * failed
+        */
+       if (ndr_desc->numa_node == NUMA_NO_NODE)
+               ndr_desc->numa_node = memory_add_physaddr_to_nid(spa->address);
+       if (ndr_desc->target_node == NUMA_NO_NODE)
+               phys_to_target_node(spa->address);
+
        /*
         * Persistence domain bits are hierarchical, if
         * ACPI_NFIT_CAPABILITY_CACHE_FLUSH is set then

^ permalink raw reply	[flat|nested] 6+ messages in thread

* RE: [PATCH v2] device-dax: use fallback nid when numa node is invalid
  2021-09-15  5:15     ` Dan Williams
@ 2021-09-15  6:50       ` Justin He
  2021-09-15 16:22         ` Dan Williams
  0 siblings, 1 reply; 6+ messages in thread
From: Justin He @ 2021-09-15  6:50 UTC (permalink / raw)
  To: Dan Williams
  Cc: Vishal Verma, Dave Jiang, David Hildenbrand, Linux NVDIMM,
	Linux Kernel Mailing List, nd



> -----Original Message-----
> From: Dan Williams <dan.j.williams@intel.com>
> Sent: Wednesday, September 15, 2021 1:16 PM
> To: Justin He <Justin.He@arm.com>
> Cc: Vishal Verma <vishal.l.verma@intel.com>; Dave Jiang
> <dave.jiang@intel.com>; David Hildenbrand <david@redhat.com>; Linux NVDIMM
> <nvdimm@lists.linux.dev>; Linux Kernel Mailing List <linux-
> kernel@vger.kernel.org>; nd <nd@arm.com>
> Subject: Re: [PATCH v2] device-dax: use fallback nid when numa node is
> invalid
> 
> On Mon, Sep 13, 2021 at 7:06 PM Justin He <Justin.He@arm.com> wrote:
> >
> > Hi Dan,
> >
> > > -----Original Message-----
> > > From: Dan Williams <dan.j.williams@intel.com>
> > > Sent: Friday, September 10, 2021 11:42 PM
> > > To: Justin He <Justin.He@arm.com>
> > > Cc: Vishal Verma <vishal.l.verma@intel.com>; Dave Jiang
> > > <dave.jiang@intel.com>; David Hildenbrand <david@redhat.com>; Linux
> NVDIMM
> > > <nvdimm@lists.linux.dev>; Linux Kernel Mailing List <linux-
> > > kernel@vger.kernel.org>
> > > Subject: Re: [PATCH v2] device-dax: use fallback nid when numa node is
> > > invalid
> > >
> > > On Fri, Sep 10, 2021 at 5:46 AM Jia He <justin.he@arm.com> wrote:
> > > >
> > > > Previously, numa_off was set unconditionally in dummy_numa_init()
> > > > even with a fake numa node. Then ACPI sets node id as NUMA_NO_NODE(-1)
> > > > after acpi_map_pxm_to_node() because it regards numa_off as turning
> > > > off the numa node. Hence dev_dax->target_node is NUMA_NO_NODE on
> > > > arm64 with fake numa case.
> > > >
> > > > Without this patch, pmem can't be probed as RAM devices on arm64 if
> > > > SRAT table isn't present:
> > > >   $ndctl create-namespace -fe namespace0.0 --mode=devdax --map=dev -s
> 1g
> > > -a 64K
> > > >   kmem dax0.0: rejecting DAX region [mem 0x240400000-0x2bfffffff]
> with
> > > invalid node: -1
> > > >   kmem: probe of dax0.0 failed with error -22
> > > >
> > > > This fixes it by using fallback memory_add_physaddr_to_nid() as nid.
> > > >
> > > > Suggested-by: David Hildenbrand <david@redhat.com>
> > > > Signed-off-by: Jia He <justin.he@arm.com>
> > > > ---
> > > > v2: - rebase it based on David's "memory group" patch.
> > > >     - drop the changes in dev_dax_kmem_remove() since nid had been
> > > >       removed in remove_memory().
> > > >  drivers/dax/kmem.c | 31 +++++++++++++++++--------------
> > > >  1 file changed, 17 insertions(+), 14 deletions(-)
> > > >
> > > > diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
> > > > index a37622060fff..e4836eb7539e 100644
> > > > --- a/drivers/dax/kmem.c
> > > > +++ b/drivers/dax/kmem.c
> > > > @@ -47,20 +47,7 @@ static int dev_dax_kmem_probe(struct dev_dax
> *dev_dax)
> > > >         unsigned long total_len = 0;
> > > >         struct dax_kmem_data *data;
> > > >         int i, rc, mapped = 0;
> > > > -       int numa_node;
> > > > -
> > > > -       /*
> > > > -        * Ensure good NUMA information for the persistent memory.
> > > > -        * Without this check, there is a risk that slow memory
> > > > -        * could be mixed in a node with faster memory, causing
> > > > -        * unavoidable performance issues.
> > > > -        */
> > > > -       numa_node = dev_dax->target_node;
> > > > -       if (numa_node < 0) {
> > > > -               dev_warn(dev, "rejecting DAX region with invalid
> > > node: %d\n",
> > > > -                               numa_node);
> > > > -               return -EINVAL;
> > > > -       }
> > > > +       int numa_node = dev_dax->target_node;
> > > >
> > > >         for (i = 0; i < dev_dax->nr_range; i++) {
> > > >                 struct range range;
> > > > @@ -71,6 +58,22 @@ static int dev_dax_kmem_probe(struct dev_dax
> *dev_dax)
> > > >                                         i, range.start, range.end);
> > > >                         continue;
> > > >                 }
> > > > +
> > > > +               /*
> > > > +                * Ensure good NUMA information for the persistent
> > > memory.
> > > > +                * Without this check, there is a risk but not fatal
> > > that slow
> > > > +                * memory could be mixed in a node with faster memory,
> > > causing
> > > > +                * unavoidable performance issues. Warn this and use
> > > fallback
> > > > +                * node id.
> > > > +                */
> > > > +               if (numa_node < 0) {
> > > > +                       int new_node =
> > > memory_add_physaddr_to_nid(range.start);
> > > > +
> > > > +                       dev_info(dev, "changing nid from %d to %d for
> > > DAX region [%#llx-%#llx]\n",
> > > > +                                numa_node, new_node, range.start,
> > > range.end);
> > > > +                       numa_node = new_node;
> > > > +               }
> > > > +
> > > >                 total_len += range_len(&range);
> > >
> > > This fallback change belongs where the parent region for the namespace
> > > adopts its target_node, because it's not clear
> > > memory_add_physaddr_to_nid() is the right fallback in all situations.
> > > Here is where this setting is happening currently:
> > >
> > > drivers/acpi/nfit/core.c:3004:          ndr_desc->target_node =
> > > pxm_to_node(spa->proximity_domain);
> > On my local arm64 guest('virt' machine type), the target_node is
> > set to -1 at this line.
> > That is:
> > The condition "spa->flags & ACPI_NFIT_PROXIMITY_VALID" is hit.
> >
> > > drivers/acpi/nfit/core.c:3007:          ndr_desc->target_node =
> > > NUMA_NO_NODE;
> > > drivers/nvdimm/e820.c:29:       ndr_desc.target_node = nid;
> > > drivers/nvdimm/of_pmem.c:58:            ndr_desc.target_node =
> > > ndr_desc.numa_node;
> > > drivers/nvdimm/region_devs.c:1127:      nd_region->target_node =
> > > ndr_desc->target_node;
> >
> >
> > Sorry,Dan. I thought I missed your previous mail:
> >
> > =========================================
> > Looks like it is the NFIT driver, thanks.
> >
> > If you're getting NUMA_NO_NODE in dax_kmem from the NFIT driver in
> > means your ACPI NFIT table is failing to populate correct numa
> > information. You could try the following to fix it up, but I think the
> > real problem is that your platform BIOS needs to add the proper numa
> > data.
> >
> > diff --git a/drivers/acpi/nfit/core.c b/drivers/acpi/nfit/core.c
> > index fb775b967c52..d3a0cec635b1 100644
> > --- a/drivers/acpi/nfit/core.c
> > +++ b/drivers/acpi/nfit/core.c
> > @@ -3005,15 +3005,8 @@ static int acpi_nfit_register_region(struct
> > acpi_nfit_desc *acpi_desc,
> >         ndr_desc->res = &res;
> >         ndr_desc->provider_data = nfit_spa;
> >         ndr_desc->attr_groups = acpi_nfit_region_attribute_groups;
> > -       if (spa->flags & ACPI_NFIT_PROXIMITY_VALID) {
> > -               ndr_desc->numa_node = acpi_map_pxm_to_online_node(
> > -                                               spa->proximity_domain);
> > -               ndr_desc->target_node = acpi_map_pxm_to_node(
> > -                               spa->proximity_domain);
> > -       } else {
> > -               ndr_desc->numa_node = NUMA_NO_NODE;
> > -               ndr_desc->target_node = NUMA_NO_NODE;
> > -       }
> > +       ndr_desc->numa_node = memory_add_physaddr_to_nid(spa->address);
> > +       ndr_desc->target_node = phys_to_target_node(spa->address);
> >
> >         /*
> >          * Persistence domain bits are hierarchical, if
> > ===================================================
> >
> > Do you still suggest fixing like this?
> 
> Are you saying that ACPI_NFIT_PROXIMITY_VALID is not set on your
> platform, or that pxm_to_node() returns NUMA_NO_NODE?
> 
Latter,  ACPI_NFIT_PROXIMITY_VALID is *set* in my case.

> I would expect something like this:
> 
> diff --git a/drivers/acpi/nfit/core.c b/drivers/acpi/nfit/core.c
> index a3ef6cce644c..95de7dc18ed8 100644
> --- a/drivers/acpi/nfit/core.c
> +++ b/drivers/acpi/nfit/core.c
> @@ -3007,6 +3007,15 @@ static int acpi_nfit_register_region(struct
> acpi_nfit_desc *acpi_desc,
>                 ndr_desc->target_node = NUMA_NO_NODE;
>         }
> 
> +       /*
> +        * Fallback to address based numa information if node lookup
> +        * failed
> +        */
> +       if (ndr_desc->numa_node == NUMA_NO_NODE)
> +               ndr_desc->numa_node = memory_add_physaddr_to_nid(spa-
> >address);
> +       if (ndr_desc->target_node == NUMA_NO_NODE)
> +               phys_to_target_node(spa->address);
> +

Would it better to add a dev_info() here to report this node id changing?

--
Cheers,
Justin (Jia He)


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH v2] device-dax: use fallback nid when numa node is invalid
  2021-09-15  6:50       ` Justin He
@ 2021-09-15 16:22         ` Dan Williams
  0 siblings, 0 replies; 6+ messages in thread
From: Dan Williams @ 2021-09-15 16:22 UTC (permalink / raw)
  To: Justin He
  Cc: Vishal Verma, Dave Jiang, David Hildenbrand, Linux NVDIMM,
	Linux Kernel Mailing List, nd

On Tue, Sep 14, 2021 at 11:51 PM Justin He <Justin.He@arm.com> wrote:
[..]
> > > diff --git a/drivers/acpi/nfit/core.c b/drivers/acpi/nfit/core.c
> > > index fb775b967c52..d3a0cec635b1 100644
> > > --- a/drivers/acpi/nfit/core.c
> > > +++ b/drivers/acpi/nfit/core.c
> > > @@ -3005,15 +3005,8 @@ static int acpi_nfit_register_region(struct
> > > acpi_nfit_desc *acpi_desc,
> > >         ndr_desc->res = &res;
> > >         ndr_desc->provider_data = nfit_spa;
> > >         ndr_desc->attr_groups = acpi_nfit_region_attribute_groups;
> > > -       if (spa->flags & ACPI_NFIT_PROXIMITY_VALID) {
> > > -               ndr_desc->numa_node = acpi_map_pxm_to_online_node(
> > > -                                               spa->proximity_domain);
> > > -               ndr_desc->target_node = acpi_map_pxm_to_node(
> > > -                               spa->proximity_domain);
> > > -       } else {
> > > -               ndr_desc->numa_node = NUMA_NO_NODE;
> > > -               ndr_desc->target_node = NUMA_NO_NODE;
> > > -       }
> > > +       ndr_desc->numa_node = memory_add_physaddr_to_nid(spa->address);
> > > +       ndr_desc->target_node = phys_to_target_node(spa->address);
> > >
> > >         /*
> > >          * Persistence domain bits are hierarchical, if
> > > ===================================================
> > >
> > > Do you still suggest fixing like this?
> >
> > Are you saying that ACPI_NFIT_PROXIMITY_VALID is not set on your
> > platform, or that pxm_to_node() returns NUMA_NO_NODE?
> >
> Latter,  ACPI_NFIT_PROXIMITY_VALID is *set* in my case.
>
> > I would expect something like this:
> >
> > diff --git a/drivers/acpi/nfit/core.c b/drivers/acpi/nfit/core.c
> > index a3ef6cce644c..95de7dc18ed8 100644
> > --- a/drivers/acpi/nfit/core.c
> > +++ b/drivers/acpi/nfit/core.c
> > @@ -3007,6 +3007,15 @@ static int acpi_nfit_register_region(struct
> > acpi_nfit_desc *acpi_desc,
> >                 ndr_desc->target_node = NUMA_NO_NODE;
> >         }
> >
> > +       /*
> > +        * Fallback to address based numa information if node lookup
> > +        * failed
> > +        */
> > +       if (ndr_desc->numa_node == NUMA_NO_NODE)
> > +               ndr_desc->numa_node = memory_add_physaddr_to_nid(spa-
> > >address);
> > +       if (ndr_desc->target_node == NUMA_NO_NODE)
> > +               phys_to_target_node(spa->address);
> > +
>
> Would it better to add a dev_info() here to report this node id changing?

Yes, given all the possibilities here, a dev_info() reporting the
final result of the node mapping is justifiable.

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2021-09-15 16:22 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-10 12:46 [PATCH v2] device-dax: use fallback nid when numa node is invalid Jia He
2021-09-10 15:42 ` Dan Williams
2021-09-14  2:06   ` Justin He
2021-09-15  5:15     ` Dan Williams
2021-09-15  6:50       ` Justin He
2021-09-15 16:22         ` Dan Williams

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).