[Patch v4] PCI: hv: Fix NUMA node assignment when kernel boots with custom NUMA topology

[Patch v4] PCI: hv: Fix NUMA node assignment when kernel boots with custom NUMA topology

[longli]

From: Long Li <longli@microsoft.com>

When kernel boots with a NUMA topology with some NUMA nodes offline, the PCI
driver should only set an online NUMA node on the device. This can happen
during KDUMP where some NUMA nodes are not made online by the KDUMP kernel.

This patch also fixes the case where kernel is booting with "numa=off".

Fixes: 999dd956d838 ("PCI: hv: Add support for protocol 1.3 and support PCI_BUS_RELATIONS2")
Signed-off-by: Long Li <longli@microsoft.com>
Reviewed-by: Michael Kelley <mikelley@microsoft.com>
---
Change log:
v2: use numa_map_to_online_node() to assign a node to device (suggested by
Michael Kelly <mikelley@microsoft.com>)
v3: add "Fixes" and check for num_possible_nodes()
v4: fix commit message format

 drivers/pci/controller/pci-hyperv.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
index 20ea2ee330b8..ae0bc2fee4ca 100644
--- a/drivers/pci/controller/pci-hyperv.c
+++ b/drivers/pci/controller/pci-hyperv.c
@@ -2155,8 +2155,17 @@ static void hv_pci_assign_numa_node(struct hv_pcibus_device *hbus)
 		if (!hv_dev)
 			continue;
 
-		if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY)
-			set_dev_node(&dev->dev, hv_dev->desc.virtual_numa_node);
+		if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY &&
+		    hv_dev->desc.virtual_numa_node < num_possible_nodes())
+			/*
+			 * The kernel may boot with some NUMA nodes offline
+			 * (e.g. in a KDUMP kernel) or with NUMA disabled via
+			 * "numa=off". In those cases, adjust the host provided
+			 * NUMA node to a valid NUMA node used by the kernel.
+			 */
+			set_dev_node(&dev->dev,
+				     numa_map_to_online_node(
+					     hv_dev->desc.virtual_numa_node));
 
 		put_pcichild(hv_dev);
 	}
-- 
2.25.1

RE: [EXTERNAL] [Patch v4] PCI: hv: Fix NUMA node assignment when kernel boots with custom NUMA topology

[Purna Pavan Chandra Aekkaladevi]

-----Original Message-----
From: longli@linuxonhyperv.com <longli@linuxonhyperv.com> 
Sent: Thursday, January 27, 2022 7:14 AM
To: linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; linux-hyperv@vger.kernel.org; Purna Pavan Chandra Aekkaladevi <paekkaladevi@microsoft.com>
Cc: Long Li <longli@microsoft.com>
Subject: [EXTERNAL] [Patch v4] PCI: hv: Fix NUMA node assignment when kernel boots with custom NUMA topology

[You don't often get email from longli@linuxonhyperv.com. Learn why this is important at http://aka.ms/LearnAboutSenderIdentification.]

From: Long Li <longli@microsoft.com>

When kernel boots with a NUMA topology with some NUMA nodes offline, the PCI driver should only set an online NUMA node on the device. This can happen during KDUMP where some NUMA nodes are not made online by the KDUMP kernel.

This patch also fixes the case where kernel is booting with "numa=off".

Fixes: 999dd956d838 ("PCI: hv: Add support for protocol 1.3 and support PCI_BUS_RELATIONS2")
Signed-off-by: Long Li <longli@microsoft.com>
Reviewed-by: Michael Kelley <mikelley@microsoft.com>
Tested-by: Purna Pavan Chandra Aekkaladevi <paekkaladevi@microsoft.com>
---
Change log:
v2: use numa_map_to_online_node() to assign a node to device (suggested by Michael Kelly <mikelley@microsoft.com>)
v3: add "Fixes" and check for num_possible_nodes()
v4: fix commit message format

 drivers/pci/controller/pci-hyperv.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
index 20ea2ee330b8..ae0bc2fee4ca 100644
--- a/drivers/pci/controller/pci-hyperv.c
+++ b/drivers/pci/controller/pci-hyperv.c
@@ -2155,8 +2155,17 @@ static void hv_pci_assign_numa_node(struct hv_pcibus_device *hbus)
                if (!hv_dev)
                        continue;

-               if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY)
-                       set_dev_node(&dev->dev, hv_dev->desc.virtual_numa_node);
+               if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY &&
+                   hv_dev->desc.virtual_numa_node < num_possible_nodes())
+                       /*
+                        * The kernel may boot with some NUMA nodes offline
+                        * (e.g. in a KDUMP kernel) or with NUMA disabled via
+                        * "numa=off". In those cases, adjust the host provided
+                        * NUMA node to a valid NUMA node used by the kernel.
+                        */
+                       set_dev_node(&dev->dev,
+                                    numa_map_to_online_node(
+                                            
+ hv_dev->desc.virtual_numa_node));

                put_pcichild(hv_dev);
        }
--
2.25.1

Re: [Patch v4] PCI: hv: Fix NUMA node assignment when kernel boots with custom NUMA topology

[Wei Liu]

On Wed, Jan 26, 2022 at 05:43:34PM -0800, longli@linuxonhyperv.com wrote:
> From: Long Li <longli@microsoft.com>
> 
> When kernel boots with a NUMA topology with some NUMA nodes offline, the PCI
> driver should only set an online NUMA node on the device. This can happen
> during KDUMP where some NUMA nodes are not made online by the KDUMP kernel.
> 
> This patch also fixes the case where kernel is booting with "numa=off".
> 
> Fixes: 999dd956d838 ("PCI: hv: Add support for protocol 1.3 and support PCI_BUS_RELATIONS2")
> Signed-off-by: Long Li <longli@microsoft.com>
> Reviewed-by: Michael Kelley <mikelley@microsoft.com>

CC PCI maintainers.

Hi Bjorn and Lorenzo

Are you going to pick this up? This is a fix we would like to see
upstream sooner rather than later.

I can pick this up via hyperv-fixes if that suits you, since this only
affect pci-hyperv.c.

Thanks,
Wei.

> ---
> Change log:
> v2: use numa_map_to_online_node() to assign a node to device (suggested by
> Michael Kelly <mikelley@microsoft.com>)
> v3: add "Fixes" and check for num_possible_nodes()
> v4: fix commit message format
> 
>  drivers/pci/controller/pci-hyperv.c | 13 +++++++++++--
>  1 file changed, 11 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
> index 20ea2ee330b8..ae0bc2fee4ca 100644
> --- a/drivers/pci/controller/pci-hyperv.c
> +++ b/drivers/pci/controller/pci-hyperv.c
> @@ -2155,8 +2155,17 @@ static void hv_pci_assign_numa_node(struct hv_pcibus_device *hbus)
>  		if (!hv_dev)
>  			continue;
>  
> -		if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY)
> -			set_dev_node(&dev->dev, hv_dev->desc.virtual_numa_node);
> +		if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY &&
> +		    hv_dev->desc.virtual_numa_node < num_possible_nodes())
> +			/*
> +			 * The kernel may boot with some NUMA nodes offline
> +			 * (e.g. in a KDUMP kernel) or with NUMA disabled via
> +			 * "numa=off". In those cases, adjust the host provided
> +			 * NUMA node to a valid NUMA node used by the kernel.
> +			 */
> +			set_dev_node(&dev->dev,
> +				     numa_map_to_online_node(
> +					     hv_dev->desc.virtual_numa_node));
>  
>  		put_pcichild(hv_dev);
>  	}
> -- 
> 2.25.1
> 

Re: [Patch v4] PCI: hv: Fix NUMA node assignment when kernel boots with custom NUMA topology

[Lorenzo Pieralisi]

On Wed, Jan 26, 2022 at 05:43:34PM -0800, longli@linuxonhyperv.com wrote:
> From: Long Li <longli@microsoft.com>
> 
> When kernel boots with a NUMA topology with some NUMA nodes offline, the PCI
> driver should only set an online NUMA node on the device. This can happen
> during KDUMP where some NUMA nodes are not made online by the KDUMP kernel.
> 
> This patch also fixes the case where kernel is booting with "numa=off".
> 
> Fixes: 999dd956d838 ("PCI: hv: Add support for protocol 1.3 and support PCI_BUS_RELATIONS2")
> Signed-off-by: Long Li <longli@microsoft.com>
> Reviewed-by: Michael Kelley <mikelley@microsoft.com>
> ---
> Change log:
> v2: use numa_map_to_online_node() to assign a node to device (suggested by
> Michael Kelly <mikelley@microsoft.com>)
> v3: add "Fixes" and check for num_possible_nodes()
> v4: fix commit message format
> 
>  drivers/pci/controller/pci-hyperv.c | 13 +++++++++++--
>  1 file changed, 11 insertions(+), 2 deletions(-)

Feel free to pick it up:

Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

> diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
> index 20ea2ee330b8..ae0bc2fee4ca 100644
> --- a/drivers/pci/controller/pci-hyperv.c
> +++ b/drivers/pci/controller/pci-hyperv.c
> @@ -2155,8 +2155,17 @@ static void hv_pci_assign_numa_node(struct hv_pcibus_device *hbus)
>  		if (!hv_dev)
>  			continue;
>  
> -		if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY)
> -			set_dev_node(&dev->dev, hv_dev->desc.virtual_numa_node);
> +		if (hv_dev->desc.flags & HV_PCI_DEVICE_FLAG_NUMA_AFFINITY &&
> +		    hv_dev->desc.virtual_numa_node < num_possible_nodes())
> +			/*
> +			 * The kernel may boot with some NUMA nodes offline
> +			 * (e.g. in a KDUMP kernel) or with NUMA disabled via
> +			 * "numa=off". In those cases, adjust the host provided
> +			 * NUMA node to a valid NUMA node used by the kernel.
> +			 */
> +			set_dev_node(&dev->dev,
> +				     numa_map_to_online_node(
> +					     hv_dev->desc.virtual_numa_node));
>  
>  		put_pcichild(hv_dev);
>  	}
> -- 
> 2.25.1
> 

Re: [Patch v4] PCI: hv: Fix NUMA node assignment when kernel boots with custom NUMA topology

[Wei Liu]

On Thu, Feb 03, 2022 at 12:42:46PM +0000, Lorenzo Pieralisi wrote:
> On Wed, Jan 26, 2022 at 05:43:34PM -0800, longli@linuxonhyperv.com wrote:
> > From: Long Li <longli@microsoft.com>
> > 
> > When kernel boots with a NUMA topology with some NUMA nodes offline, the PCI
> > driver should only set an online NUMA node on the device. This can happen
> > during KDUMP where some NUMA nodes are not made online by the KDUMP kernel.
> > 
> > This patch also fixes the case where kernel is booting with "numa=off".
> > 
> > Fixes: 999dd956d838 ("PCI: hv: Add support for protocol 1.3 and support PCI_BUS_RELATIONS2")
> > Signed-off-by: Long Li <longli@microsoft.com>
> > Reviewed-by: Michael Kelley <mikelley@microsoft.com>
> > ---
> > Change log:
> > v2: use numa_map_to_online_node() to assign a node to device (suggested by
> > Michael Kelly <mikelley@microsoft.com>)
> > v3: add "Fixes" and check for num_possible_nodes()
> > v4: fix commit message format
> > 
> >  drivers/pci/controller/pci-hyperv.c | 13 +++++++++++--
> >  1 file changed, 11 insertions(+), 2 deletions(-)
> 
> Feel free to pick it up:
> 
> Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

Thanks Lorenzo.