[PATCH v4 1/1] PCI: endpoint: pci-epf-vntb: Fix transfer failure for fixed size BARs

[PATCH v4 1/1] PCI: endpoint: pci-epf-vntb: Fix transfer failure for fixed size BARs

[Frank Li]

For the inbound MEM/IO TLPs, iATU on the endpoint expects the target
address to be aligned to the size of the BAR. For configurable BARs, there
is no issue because both host and endpoint will know the exact size of the
BAR region. But for fixed size BARs available in some controllers, if the
BAR size advertised by the endpoint is not same as of the actual BAR size
used in the controller, then the MEM/IO TLPs generated by the host will not
be translated properly by the endpoint iATU.

So if the fixed size BARs are available in endpoint controllers, always use
the actual BAR size.

This only fixes doorbell (DB) BAR. A similar fix is needed for memory map
windows(MW) BARs.

Fixes: e35f56bb0330 ("PCI: endpoint: Support NTB transfer between RC and EP")
Cc: stable@vger.kernel.org
Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Signed-off-by: Frank Li <Frank.Li@nxp.com>
---

Notes:
    Change from v3 to v4
    - add notes: fixing the issue only for DB BAR and a similar fix is needed
    for other MW BARs
    - Add Manivannan Sadhasivam's review tag
    
    Change from v2 to v3
    - rework commti message
    - add fixes and cc stable
    - return -ENOMEN when request size > fix bar size
    Change from v1 to v2
    - Remove unnessary set align when fix_bar_size.

 drivers/pci/endpoint/functions/pci-epf-vntb.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
index 3f60128560ed0..85120978fb8c9 100644
--- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
@@ -550,6 +550,14 @@ static int epf_ntb_db_bar_init(struct epf_ntb *ntb)
 
 	barno = ntb->epf_ntb_bar[BAR_DB];
 
+	if (epc_features->bar_fixed_size[barno]) {
+		if (size > epc_features->bar_fixed_size[barno]) {
+			dev_err(dev, "Fixed BAR%d is too small for doorbell\n", barno);
+			return -ENOMEM;
+		}
+		size = epc_features->bar_fixed_size[barno];
+	}
+
 	mw_addr = pci_epf_alloc_space(ntb->epf, size, barno, align, 0);
 	if (!mw_addr) {
 		dev_err(dev, "Failed to allocate OB address\n");
-- 
2.34.1

Re: [PATCH v4 1/1] PCI: endpoint: pci-epf-vntb: Fix transfer failure for fixed size BARs

[Frank Li]

On Mon, Jan 08, 2024 at 10:10:15AM -0500, Frank Li wrote:
> For the inbound MEM/IO TLPs, iATU on the endpoint expects the target
> address to be aligned to the size of the BAR. For configurable BARs, there
> is no issue because both host and endpoint will know the exact size of the
> BAR region. But for fixed size BARs available in some controllers, if the
> BAR size advertised by the endpoint is not same as of the actual BAR size
> used in the controller, then the MEM/IO TLPs generated by the host will not
> be translated properly by the endpoint iATU.
> 
> So if the fixed size BARs are available in endpoint controllers, always use
> the actual BAR size.
> 
> This only fixes doorbell (DB) BAR. A similar fix is needed for memory map
> windows(MW) BARs.
> 
> Fixes: e35f56bb0330 ("PCI: endpoint: Support NTB transfer between RC and EP")
> Cc: stable@vger.kernel.org
> Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---

@lpieralisi:

	Could you please pick this fix patch?

Frank

> 
> Notes:
>     Change from v3 to v4
>     - add notes: fixing the issue only for DB BAR and a similar fix is needed
>     for other MW BARs
>     - Add Manivannan Sadhasivam's review tag
>     
>     Change from v2 to v3
>     - rework commti message
>     - add fixes and cc stable
>     - return -ENOMEN when request size > fix bar size
>     Change from v1 to v2
>     - Remove unnessary set align when fix_bar_size.
> 
>  drivers/pci/endpoint/functions/pci-epf-vntb.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> index 3f60128560ed0..85120978fb8c9 100644
> --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> @@ -550,6 +550,14 @@ static int epf_ntb_db_bar_init(struct epf_ntb *ntb)
>  
>  	barno = ntb->epf_ntb_bar[BAR_DB];
>  
> +	if (epc_features->bar_fixed_size[barno]) {
> +		if (size > epc_features->bar_fixed_size[barno]) {
> +			dev_err(dev, "Fixed BAR%d is too small for doorbell\n", barno);
> +			return -ENOMEM;
> +		}
> +		size = epc_features->bar_fixed_size[barno];
> +	}
> +
>  	mw_addr = pci_epf_alloc_space(ntb->epf, size, barno, align, 0);
>  	if (!mw_addr) {
>  		dev_err(dev, "Failed to allocate OB address\n");
> -- 
> 2.34.1
> 

Re: [PATCH v4 1/1] PCI: endpoint: pci-epf-vntb: Fix transfer failure for fixed size BARs

[Niklas Cassel]

On Thu, Jan 25, 2024 at 10:33:19AM -0500, Frank Li wrote:
> On Mon, Jan 08, 2024 at 10:10:15AM -0500, Frank Li wrote:
> > For the inbound MEM/IO TLPs, iATU on the endpoint expects the target
> > address to be aligned to the size of the BAR. For configurable BARs, there
> > is no issue because both host and endpoint will know the exact size of the
> > BAR region. But for fixed size BARs available in some controllers, if the
> > BAR size advertised by the endpoint is not same as of the actual BAR size
> > used in the controller, then the MEM/IO TLPs generated by the host will not
> > be translated properly by the endpoint iATU.
> > 
> > So if the fixed size BARs are available in endpoint controllers, always use
> > the actual BAR size.
> > 
> > This only fixes doorbell (DB) BAR. A similar fix is needed for memory map
> > windows(MW) BARs.
> > 
> > Fixes: e35f56bb0330 ("PCI: endpoint: Support NTB transfer between RC and EP")
> > Cc: stable@vger.kernel.org
> > Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > ---
> 
> @lpieralisi:
> 
> 	Could you please pick this fix patch?


Hello Frank, Lorenzo, Mani,


Frank, sorry for not seeing this patch earlier.

Could you please see if this series fixes your issue:
https://lore.kernel.org/linux-pci/20240130193214.713739-1-cassel@kernel.org/T/#t

I think it is nicer since:
1) It doesn't add copy pasted code from pci-epf-test.c to pci-epf-vntb.c.
2) I would expect it to handle both the doorbell BAR and the MW BARs.


Kind regards,
Niklas

Re: [PATCH v4 1/1] PCI: endpoint: pci-epf-vntb: Fix transfer failure for fixed size BARs

[Frank Li]

On Tue, Jan 30, 2024 at 08:43:24PM +0100, Niklas Cassel wrote:
> On Thu, Jan 25, 2024 at 10:33:19AM -0500, Frank Li wrote:
> > On Mon, Jan 08, 2024 at 10:10:15AM -0500, Frank Li wrote:
> > > For the inbound MEM/IO TLPs, iATU on the endpoint expects the target
> > > address to be aligned to the size of the BAR. For configurable BARs, there
> > > is no issue because both host and endpoint will know the exact size of the
> > > BAR region. But for fixed size BARs available in some controllers, if the
> > > BAR size advertised by the endpoint is not same as of the actual BAR size
> > > used in the controller, then the MEM/IO TLPs generated by the host will not
> > > be translated properly by the endpoint iATU.
> > > 
> > > So if the fixed size BARs are available in endpoint controllers, always use
> > > the actual BAR size.
> > > 
> > > This only fixes doorbell (DB) BAR. A similar fix is needed for memory map
> > > windows(MW) BARs.
> > > 
> > > Fixes: e35f56bb0330 ("PCI: endpoint: Support NTB transfer between RC and EP")
> > > Cc: stable@vger.kernel.org
> > > Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > ---
> > 
> > @lpieralisi:
> > 
> > 	Could you please pick this fix patch?
> 
> 
> Hello Frank, Lorenzo, Mani,
> 
> 
> Frank, sorry for not seeing this patch earlier.
> 
> Could you please see if this series fixes your issue:
> https://lore.kernel.org/linux-pci/20240130193214.713739-1-cassel@kernel.org/T/#t
> 
> I think it is nicer since:
> 1) It doesn't add copy pasted code from pci-epf-test.c to pci-epf-vntb.c.
> 2) I would expect it to handle both the doorbell BAR and the MW BARs.

Agree! Thanks

Frank
> 
> 
> Kind regards,
> Niklas