[PATCH v2 0/2] PCI: dwc: Add support for 64-bit MSI target addresses

[PATCH v2 0/2] PCI: dwc: Add support for 64-bit MSI target addresses

[Will McVicker]

Hi All,

I have updated the patch series based on the build error reported by the
kernel test robot which found an issues with the new inline function when
PCIE_DW_HOST isn't set. Additionally, I have updated the DMA alloc error
handling for the first patch which was reported to me directly by Isaac
Manjarres. Basically, we don't need to check for a DMA mapping error since
dma_alloc_coherent() will free the memory when a mapping error occurs.

Please take a look and let me know if you have any questions or concerns.

Thanks,
Will

Reported-by: kernel test robot <lkp@intel.com>
Reported-by: Isaac J. Manjarres <isaacmanjarres@google.com>

Will McVicker (2):
  PCI: dwc: drop dependency on ZONE_DMA32
  PCI: dwc: add support for 64-bit MSI target address

v2:
  * Fixed build error caught by kernel test robot
  * Fixed error handling reported by Isaac Manjarres

 .../pci/controller/dwc/pcie-designware-host.c | 37 +++++++++++--------
 drivers/pci/controller/dwc/pcie-designware.c  |  9 +++++
 drivers/pci/controller/dwc/pcie-designware.h  |  1 +
 3 files changed, 31 insertions(+), 16 deletions(-)


base-commit: d4252071b97d2027d246f6a82cbee4d52f618b47
-- 
2.37.1.559.g78731f0fdb-goog

[PATCH v2 1/2] PCI: dwc: drop dependency on ZONE_DMA32

[Will McVicker]

Re-work the msi_msg DMA allocation logic to use dma_alloc_coherent()
which uses the coherent DMA mask to try and return an allocation within
the DMA mask limits. This allows kernel configurations that disable
ZONE_DMA32 to continue supporting a 32-bit DMA mask. Without this patch,
the PCIe host device will fail to probe when ZONE_DMA32 is disabled.

Fixes: 35797e672ff0 ("PCI: dwc: Fix MSI msi_msg DMA mapping")
Reported-by: Isaac J. Manjarres <isaacmanjarres@google.com>
Signed-off-by: Will McVicker <willmcvicker@google.com>
---
 .../pci/controller/dwc/pcie-designware-host.c | 23 ++++++++-----------
 1 file changed, 9 insertions(+), 14 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index 7746f94a715f..8f2222f51671 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -272,9 +272,9 @@ static void dw_pcie_free_msi(struct dw_pcie_rp *pp)
 		struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
 		struct device *dev = pci->dev;
 
-		dma_unmap_page(dev, pp->msi_data, PAGE_SIZE, DMA_FROM_DEVICE);
-		if (pp->msi_page)
-			__free_page(pp->msi_page);
+		dma_free_coherent(dev, PAGE_SIZE, pp->msi_page, pp->msi_data);
+		pp->msi_data = 0;
+		pp->msi_page = NULL;
 	}
 }
 
@@ -375,22 +375,17 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)
 						    dw_chained_msi_isr, pp);
 	}
 
-	ret = dma_set_mask(dev, DMA_BIT_MASK(32));
+	ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32));
 	if (ret)
 		dev_warn(dev, "Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n");
 
-	pp->msi_page = alloc_page(GFP_DMA32);
-	pp->msi_data = dma_map_page(dev, pp->msi_page, 0,
-				    PAGE_SIZE, DMA_FROM_DEVICE);
-	ret = dma_mapping_error(dev, pp->msi_data);
-	if (ret) {
-		dev_err(pci->dev, "Failed to map MSI data\n");
-		__free_page(pp->msi_page);
-		pp->msi_page = NULL;
+	pp->msi_page = dma_alloc_coherent(dev, PAGE_SIZE, &pp->msi_data,
+					  GFP_KERNEL);
+	if (!pp->msi_page) {
+		dev_err(dev, "Failed to alloc and map MSI data\n");
 		pp->msi_data = 0;
 		dw_pcie_free_msi(pp);
-
-		return ret;
+		return -ENOMEM;
 	}
 
 	return 0;
-- 
2.37.1.559.g78731f0fdb-goog

[PATCH v2 2/2] PCI: dwc: add support for 64-bit MSI target address

[Will McVicker]

Since not all devices require a 32-bit MSI address, add support to the
PCIe host driver to allow setting the DMA mask to 64-bits. This allows
kernels to disable ZONE_DMA32 and bounce buffering (swiotlb) without
risking not being able to get a 32-bit address during DMA allocation.
Basically, in the slim chance that there are no 32-bit allocations
available, the current PCIe host driver will fail to allocate the
msi_msg page due to a DMA address overflow (seen in [1]). With this
patch, the PCIe driver can advertise 64-bit support via it's MSI
capabilities to hint to the PCIe host driver to set the DMA mask to
64-bits.

[1] https://lore.kernel.org/all/Yo0soniFborDl7+C@google.com/

Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Will McVicker <willmcvicker@google.com>
---
 drivers/pci/controller/dwc/pcie-designware-host.c | 14 ++++++++++++--
 drivers/pci/controller/dwc/pcie-designware.c      |  9 +++++++++
 drivers/pci/controller/dwc/pcie-designware.h      |  1 +
 3 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index 8f2222f51671..2b9b63489415 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -338,6 +338,8 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)
 	struct platform_device *pdev = to_platform_device(dev);
 	int ret;
 	u32 ctrl, num_ctrls;
+	bool msi_64b = false;
+	u16 msi_capabilities;
 
 	for (ctrl = 0; ctrl < MAX_MSI_CTRLS; ctrl++)
 		pp->irq_mask[ctrl] = ~0;
@@ -375,9 +377,17 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)
 						    dw_chained_msi_isr, pp);
 	}
 
-	ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32));
+	msi_capabilities = dw_pcie_msi_capabilities(pci);
+	if (msi_capabilities & PCI_MSI_FLAGS_ENABLE)
+		msi_64b = msi_capabilities & PCI_MSI_FLAGS_64BIT ? true : false;
+
+	dev_dbg(dev, "Setting MSI DMA mask to %s-bit.\n",
+		msi_64b ? "64" : "32");
+	ret = dma_set_mask_and_coherent(dev, msi_64b ?
+					DMA_BIT_MASK(64) : DMA_BIT_MASK(32));
 	if (ret)
-		dev_warn(dev, "Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n");
+		dev_warn(dev, "Failed to set DMA mask to %s-bit.\n",
+			 msi_64b ? "64" : "32");
 
 	pp->msi_page = dma_alloc_coherent(dev, PAGE_SIZE, &pp->msi_data,
 					  GFP_KERNEL);
diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
index c6725c519a47..8ed402307d7f 100644
--- a/drivers/pci/controller/dwc/pcie-designware.c
+++ b/drivers/pci/controller/dwc/pcie-designware.c
@@ -82,6 +82,15 @@ u8 dw_pcie_find_capability(struct dw_pcie *pci, u8 cap)
 }
 EXPORT_SYMBOL_GPL(dw_pcie_find_capability);
 
+u16 dw_pcie_msi_capabilities(struct dw_pcie *pci)
+{
+	u8 offset;
+
+	offset = dw_pcie_find_capability(pci, PCI_CAP_ID_MSI);
+	return dw_pcie_readw_dbi(pci, offset + PCI_MSI_FLAGS);
+}
+EXPORT_SYMBOL_GPL(dw_pcie_msi_capabilities);
+
 static u16 dw_pcie_find_next_ext_capability(struct dw_pcie *pci, u16 start,
 					    u8 cap)
 {
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index 09b887093a84..4631e26ba6c5 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -333,6 +333,7 @@ void dw_pcie_version_detect(struct dw_pcie *pci);
 
 u8 dw_pcie_find_capability(struct dw_pcie *pci, u8 cap);
 u16 dw_pcie_find_ext_capability(struct dw_pcie *pci, u8 cap);
+u16 dw_pcie_msi_capabilities(struct dw_pcie *pci);
 
 int dw_pcie_read(void __iomem *addr, int size, u32 *val);
 int dw_pcie_write(void __iomem *addr, int size, u32 val);
-- 
2.37.1.559.g78731f0fdb-goog

Re: [PATCH v2 1/2] PCI: dwc: drop dependency on ZONE_DMA32

[Rob Herring]

On Wed, Aug 10, 2022 at 06:35:34PM +0000, Will McVicker wrote:
> Re-work the msi_msg DMA allocation logic to use dma_alloc_coherent()
> which uses the coherent DMA mask to try and return an allocation within
> the DMA mask limits. This allows kernel configurations that disable
> ZONE_DMA32 to continue supporting a 32-bit DMA mask. Without this patch,
> the PCIe host device will fail to probe when ZONE_DMA32 is disabled.
> 
> Fixes: 35797e672ff0 ("PCI: dwc: Fix MSI msi_msg DMA mapping")
> Reported-by: Isaac J. Manjarres <isaacmanjarres@google.com>
> Signed-off-by: Will McVicker <willmcvicker@google.com>
> ---
>  .../pci/controller/dwc/pcie-designware-host.c | 23 ++++++++-----------
>  1 file changed, 9 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
> index 7746f94a715f..8f2222f51671 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> @@ -272,9 +272,9 @@ static void dw_pcie_free_msi(struct dw_pcie_rp *pp)
>  		struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
>  		struct device *dev = pci->dev;
>  
> -		dma_unmap_page(dev, pp->msi_data, PAGE_SIZE, DMA_FROM_DEVICE);
> -		if (pp->msi_page)
> -			__free_page(pp->msi_page);
> +		dma_free_coherent(dev, PAGE_SIZE, pp->msi_page, pp->msi_data);
> +		pp->msi_data = 0;
> +		pp->msi_page = NULL;
>  	}
>  }
>  
> @@ -375,22 +375,17 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)
>  						    dw_chained_msi_isr, pp);
>  	}
>  
> -	ret = dma_set_mask(dev, DMA_BIT_MASK(32));
> +	ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32));
>  	if (ret)
>  		dev_warn(dev, "Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n");
>  
> -	pp->msi_page = alloc_page(GFP_DMA32);
> -	pp->msi_data = dma_map_page(dev, pp->msi_page, 0,
> -				    PAGE_SIZE, DMA_FROM_DEVICE);
> -	ret = dma_mapping_error(dev, pp->msi_data);
> -	if (ret) {
> -		dev_err(pci->dev, "Failed to map MSI data\n");
> -		__free_page(pp->msi_page);
> -		pp->msi_page = NULL;
> +	pp->msi_page = dma_alloc_coherent(dev, PAGE_SIZE, &pp->msi_data,
> +					  GFP_KERNEL);

You can use the managed version, dmam_alloc_coherent(), and avoid the 
freeing yourself. Also with that, I think you don't need 'msi_page'?

Also, no need to alloc a whole page. A u32 or u64? should be fine. The 
write never makes it to memory, so doesn't really matter.

> +	if (!pp->msi_page) {
> +		dev_err(dev, "Failed to alloc and map MSI data\n");
>  		pp->msi_data = 0;
>  		dw_pcie_free_msi(pp);
> -
> -		return ret;
> +		return -ENOMEM;
>  	}
>  
>  	return 0;
> -- 
> 2.37.1.559.g78731f0fdb-goog
> 

Re: [PATCH v2 2/2] PCI: dwc: add support for 64-bit MSI target address

[Rob Herring]

On Wed, Aug 10, 2022 at 06:35:35PM +0000, Will McVicker wrote:
> Since not all devices require a 32-bit MSI address, add support to the
> PCIe host driver to allow setting the DMA mask to 64-bits. This allows
> kernels to disable ZONE_DMA32 and bounce buffering (swiotlb) without
> risking not being able to get a 32-bit address during DMA allocation.
> Basically, in the slim chance that there are no 32-bit allocations
> available, the current PCIe host driver will fail to allocate the
> msi_msg page due to a DMA address overflow (seen in [1]). With this
> patch, the PCIe driver can advertise 64-bit support via it's MSI
> capabilities to hint to the PCIe host driver to set the DMA mask to
> 64-bits.
> 
> [1] https://lore.kernel.org/all/Yo0soniFborDl7+C@google.com/
> 
> Reported-by: kernel test robot <lkp@intel.com>
> Signed-off-by: Will McVicker <willmcvicker@google.com>
> ---
>  drivers/pci/controller/dwc/pcie-designware-host.c | 14 ++++++++++++--
>  drivers/pci/controller/dwc/pcie-designware.c      |  9 +++++++++
>  drivers/pci/controller/dwc/pcie-designware.h      |  1 +
>  3 files changed, 22 insertions(+), 2 deletions(-)

Reviewed-by: Rob Herring <robh@kernel.org>

Re: [PATCH v2 1/2] PCI: dwc: drop dependency on ZONE_DMA32

[William McVicker]

On 08/10/2022, Rob Herring wrote:
> On Wed, Aug 10, 2022 at 06:35:34PM +0000, Will McVicker wrote:
> > Re-work the msi_msg DMA allocation logic to use dma_alloc_coherent()
> > which uses the coherent DMA mask to try and return an allocation within
> > the DMA mask limits. This allows kernel configurations that disable
> > ZONE_DMA32 to continue supporting a 32-bit DMA mask. Without this patch,
> > the PCIe host device will fail to probe when ZONE_DMA32 is disabled.
> > 
> > Fixes: 35797e672ff0 ("PCI: dwc: Fix MSI msi_msg DMA mapping")
> > Reported-by: Isaac J. Manjarres <isaacmanjarres@google.com>
> > Signed-off-by: Will McVicker <willmcvicker@google.com>
> > ---
> >  .../pci/controller/dwc/pcie-designware-host.c | 23 ++++++++-----------
> >  1 file changed, 9 insertions(+), 14 deletions(-)
> > 
> > diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
> > index 7746f94a715f..8f2222f51671 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> > @@ -272,9 +272,9 @@ static void dw_pcie_free_msi(struct dw_pcie_rp *pp)
> >  		struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> >  		struct device *dev = pci->dev;
> >  
> > -		dma_unmap_page(dev, pp->msi_data, PAGE_SIZE, DMA_FROM_DEVICE);
> > -		if (pp->msi_page)
> > -			__free_page(pp->msi_page);
> > +		dma_free_coherent(dev, PAGE_SIZE, pp->msi_page, pp->msi_data);
> > +		pp->msi_data = 0;
> > +		pp->msi_page = NULL;
> >  	}
> >  }
> >  
> > @@ -375,22 +375,17 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)
> >  						    dw_chained_msi_isr, pp);
> >  	}
> >  
> > -	ret = dma_set_mask(dev, DMA_BIT_MASK(32));
> > +	ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32));
> >  	if (ret)
> >  		dev_warn(dev, "Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n");
> >  
> > -	pp->msi_page = alloc_page(GFP_DMA32);
> > -	pp->msi_data = dma_map_page(dev, pp->msi_page, 0,
> > -				    PAGE_SIZE, DMA_FROM_DEVICE);
> > -	ret = dma_mapping_error(dev, pp->msi_data);
> > -	if (ret) {
> > -		dev_err(pci->dev, "Failed to map MSI data\n");
> > -		__free_page(pp->msi_page);
> > -		pp->msi_page = NULL;
> > +	pp->msi_page = dma_alloc_coherent(dev, PAGE_SIZE, &pp->msi_data,
> > +					  GFP_KERNEL);
> 
> You can use the managed version, dmam_alloc_coherent(), and avoid the 
> freeing yourself. Also with that, I think you don't need 'msi_page'?
> 
> Also, no need to alloc a whole page. A u32 or u64? should be fine. The 
> write never makes it to memory, so doesn't really matter.
> 
> > +	if (!pp->msi_page) {
> > +		dev_err(dev, "Failed to alloc and map MSI data\n");
> >  		pp->msi_data = 0;
> >  		dw_pcie_free_msi(pp);
> > -
> > -		return ret;
> > +		return -ENOMEM;
> >  	}
> >  
> >  	return 0;
> > -- 
> > 2.37.1.559.g78731f0fdb-goog
> > 

Thanks for the review Rob! I've updated the series to address your concerns.

Regards,
Will