[PATCH V3] PCI: vmd: Fix secondary bus reset for Intel bridges

[PATCH V3] PCI: vmd: Fix secondary bus reset for Intel bridges

[francisco.munoz.ruiz]

From: Francisco Munoz <francisco.munoz.ruiz@linux.intel.com>

The reset was never applied in the current implementation because Intel
Bridges owned by VMD are parentless. Internally, pci_reset_bus() applies
a reset to the parent of the PCI device supplied as argument, but in this
case it failed because there wasn't a parent.

In more detail, this change allows the VMD driver to enumerate NVMe devices
in pass-through configurations when guest reboots are performed. Commit id
6aab5622296b ("PCI: vmd: Clean up domain before enumeration") attempted to
fix this, but later we discovered that the code inside pci_reset_bus() wasn’t
triggering secondary bus resets.  Therefore, we updated the parameters passed
to it, and now NVMe SSDs attached to VMD bridges are properly enumerated in
VT-d pass-through scenarios.

Signed-off-by: Francisco Munoz <francisco.munoz.ruiz@linux.intel.com>
Reviewed-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Reviewed-by: Jonathan Derrick <jonathan.derrick@linux.dev>
---
V3:
    - Add WARN_ON
    - Include Jonathan as reviewer
    - Update commit message
V2:
    - Update commit message

 drivers/pci/controller/vmd.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index e06e9f4fc50f..2406be6644f3 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -859,8 +859,17 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
 	pci_scan_child_bus(vmd->bus);
 	vmd_domain_reset(vmd);
-	list_for_each_entry(child, &vmd->bus->children, node)
-		pci_reset_bus(child->self);
+
+	list_for_each_entry(child, &vmd->bus->children, node) {
+		if (!list_empty(&child->devices)) {
+			ret = pci_reset_bus(list_first_entry(&child->devices,
+							     struct pci_dev,
+							     bus_list));
+			WARN_ON(ret);
+			break;
+		}
+	}
+
 	pci_assign_unassigned_bus_resources(vmd->bus);
 
 	/*
-- 
2.25.1

Re: [PATCH V3] PCI: vmd: Fix secondary bus reset for Intel bridges

[Munoz Ruiz, Francisco]

On 11/3/2022 1:14 PM, francisco.munoz.ruiz@linux.intel.com wrote:
> From: Francisco Munoz <francisco.munoz.ruiz@linux.intel.com>
> 
> The reset was never applied in the current implementation because Intel
> Bridges owned by VMD are parentless. Internally, pci_reset_bus() applies
> a reset to the parent of the PCI device supplied as argument, but in this
> case it failed because there wasn't a parent.
> 
> In more detail, this change allows the VMD driver to enumerate NVMe devices
> in pass-through configurations when guest reboots are performed. Commit id
> 6aab5622296b ("PCI: vmd: Clean up domain before enumeration") attempted to
> fix this, but later we discovered that the code inside pci_reset_bus() wasn’t
> triggering secondary bus resets.  Therefore, we updated the parameters passed
> to it, and now NVMe SSDs attached to VMD bridges are properly enumerated in
> VT-d pass-through scenarios.
> 
> Signed-off-by: Francisco Munoz <francisco.munoz.ruiz@linux.intel.com>
> Reviewed-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> Reviewed-by: Jonathan Derrick <jonathan.derrick@linux.dev>
> ---
> V3:
>     - Add WARN_ON
>     - Include Jonathan as reviewer
>     - Update commit message
> V2:
>     - Update commit message
> 
>  drivers/pci/controller/vmd.c | 13 +++++++++++--
>  1 file changed, 11 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
> index e06e9f4fc50f..2406be6644f3 100644
> --- a/drivers/pci/controller/vmd.c
> +++ b/drivers/pci/controller/vmd.c
> @@ -859,8 +859,17 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
>  
>  	pci_scan_child_bus(vmd->bus);
>  	vmd_domain_reset(vmd);
> -	list_for_each_entry(child, &vmd->bus->children, node)
> -		pci_reset_bus(child->self);
> +
> +	list_for_each_entry(child, &vmd->bus->children, node) {
> +		if (!list_empty(&child->devices)) {
> +			ret = pci_reset_bus(list_first_entry(&child->devices,
> +							     struct pci_dev,
> +							     bus_list));
> +			WARN_ON(ret);
> +			break;
> +		}
> +	}
> +
>  	pci_assign_unassigned_bus_resources(vmd->bus);
>  
>  	/*


Hi,

Just a gentle reminder for this one

Best wishes,
Francisco.

Re: [PATCH V3] PCI: vmd: Fix secondary bus reset for Intel bridges

[Lorenzo Pieralisi]

On Mon, Nov 28, 2022 at 03:38:24PM -0800, Munoz Ruiz, Francisco wrote:
> On 11/3/2022 1:14 PM, francisco.munoz.ruiz@linux.intel.com wrote:
> > From: Francisco Munoz <francisco.munoz.ruiz@linux.intel.com>
> > 
> > The reset was never applied in the current implementation because Intel
> > Bridges owned by VMD are parentless. Internally, pci_reset_bus() applies
> > a reset to the parent of the PCI device supplied as argument, but in this
> > case it failed because there wasn't a parent.

Please add a comment in the *code* as well and explain why it is so.

> > In more detail, this change allows the VMD driver to enumerate NVMe devices
> > in pass-through configurations when guest reboots are performed. Commit id
> > 6aab5622296b ("PCI: vmd: Clean up domain before enumeration") attempted to
> > fix this, but later we discovered that the code inside pci_reset_bus() wasn’t
> > triggering secondary bus resets.  Therefore, we updated the parameters passed
> > to it, and now NVMe SSDs attached to VMD bridges are properly enumerated in
> > VT-d pass-through scenarios.

I am not sure this helps much - I'd rather explain why the device
hierarchy in VMD is as it is - when that's explained is rather clear
why the current reset is NOP and this patch is needed.

> > Signed-off-by: Francisco Munoz <francisco.munoz.ruiz@linux.intel.com>
> > Reviewed-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> > Reviewed-by: Jonathan Derrick <jonathan.derrick@linux.dev>
> > ---
> > V3:
> >     - Add WARN_ON
> >     - Include Jonathan as reviewer
> >     - Update commit message
> > V2:
> >     - Update commit message
> > 
> >  drivers/pci/controller/vmd.c | 13 +++++++++++--
> >  1 file changed, 11 insertions(+), 2 deletions(-)
> > 
> > diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
> > index e06e9f4fc50f..2406be6644f3 100644
> > --- a/drivers/pci/controller/vmd.c
> > +++ b/drivers/pci/controller/vmd.c
> > @@ -859,8 +859,17 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
> >  
> >  	pci_scan_child_bus(vmd->bus);
> >  	vmd_domain_reset(vmd);
> > -	list_for_each_entry(child, &vmd->bus->children, node)
> > -		pci_reset_bus(child->self);
> > +
> > +	list_for_each_entry(child, &vmd->bus->children, node) {
> > +		if (!list_empty(&child->devices)) {
> > +			ret = pci_reset_bus(list_first_entry(&child->devices,
> > +							     struct pci_dev,
> > +							     bus_list));
> > +			WARN_ON(ret);

Technically you are adding a WARN_ON() here to make sure that a failed
reset is detected - I am not sure a backtrace is really required,
a pci_warn() maybe ?

Lorenzo

> > +			break;
> > +		}
> > +	}
> > +
> >  	pci_assign_unassigned_bus_resources(vmd->bus);
> >  
> >  	/*
> 
> 
> Hi,
> 
> Just a gentle reminder for this one
> 
> Best wishes,
> Francisco.