linux-arm-kernel.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v3 00/11] *** SUBJECT HERE ***
@ 2019-09-02  3:17 Xiaowei Bao
  2019-09-02  3:17 ` [PATCH v3 01/11] PCI: designware-ep: Add multiple PFs support for DWC Xiaowei Bao
                   ` (11 more replies)
  0 siblings, 12 replies; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-02  3:17 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, leoyang.li, kishon,
	lorenzo.pieralisi, minghuan.Lian, mingkai.hu, roy.zang,
	jingoohan1, gustavo.pimentel, linux-pci, devicetree,
	linux-kernel, linux-arm-kernel, linuxppc-dev
  Cc: gregkh, zhiqiang.hou, Xiaowei Bao, arnd

*** BLURB HERE ***

Xiaowei Bao (11):
  PCI: designware-ep: Add multiple PFs support for DWC
  PCI: designware-ep: Add the doorbell mode of MSI-X in EP mode
  PCI: designware-ep: Move the function of getting MSI capability
    forward
  PCI: designware-ep: Modify MSI and MSIX CAP way of finding
  dt-bindings: pci: layerscape-pci: add compatible strings for ls1088a
    and ls2088a
  PCI: layerscape: Fix some format issue of the code
  PCI: layerscape: Modify the way of getting capability with different
    PEX
  PCI: layerscape: Modify the MSIX to the doorbell mode
  PCI: layerscape: Add EP mode support for ls1088a and ls2088a
  arm64: dts: layerscape: Add PCIe EP node for ls1088a
  misc: pci_endpoint_test: Add LS1088a in pci_device_id table

 .../devicetree/bindings/pci/layerscape-pci.txt     |   4 +-
 arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi     |  31 +++
 drivers/misc/pci_endpoint_test.c                   |   1 +
 drivers/pci/controller/dwc/pci-layerscape-ep.c     | 100 ++++++--
 drivers/pci/controller/dwc/pcie-designware-ep.c    | 255 +++++++++++++++++----
 drivers/pci/controller/dwc/pcie-designware.c       |  59 +++--
 drivers/pci/controller/dwc/pcie-designware.h       |  48 +++-
 7 files changed, 404 insertions(+), 94 deletions(-)

-- 
2.9.5


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* [PATCH v3 01/11] PCI: designware-ep: Add multiple PFs support for DWC
  2019-09-02  3:17 [PATCH v3 00/11] *** SUBJECT HERE *** Xiaowei Bao
@ 2019-09-02  3:17 ` Xiaowei Bao
  2019-09-02 16:26   ` Andrew Murray
  2019-09-02  3:17 ` [PATCH v3 02/11] PCI: designware-ep: Add the doorbell mode of MSI-X in EP mode Xiaowei Bao
                   ` (10 subsequent siblings)
  11 siblings, 1 reply; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-02  3:17 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, leoyang.li, kishon,
	lorenzo.pieralisi, minghuan.Lian, mingkai.hu, roy.zang,
	jingoohan1, gustavo.pimentel, linux-pci, devicetree,
	linux-kernel, linux-arm-kernel, linuxppc-dev
  Cc: gregkh, zhiqiang.hou, Xiaowei Bao, arnd

Add multiple PFs support for DWC, different PF have different config space
we use pf-offset property which get from the DTS to access the different pF
config space.

Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
---
v2:
 - Remove duplicate redundant code.
 - Reimplement the PF config space access way.
v3:
 - Integrate duplicate code for func_select.
 - Move PCIE_ATU_FUNC_NUM(pf) (pf << 20) to ((pf) << 20).
 - Add the comments for func_conf_select function.

 drivers/pci/controller/dwc/pcie-designware-ep.c | 123 ++++++++++++++++--------
 drivers/pci/controller/dwc/pcie-designware.c    |  59 ++++++++----
 drivers/pci/controller/dwc/pcie-designware.h    |  18 +++-
 3 files changed, 142 insertions(+), 58 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index 65f4792..eb851c2 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -19,12 +19,26 @@ void dw_pcie_ep_linkup(struct dw_pcie_ep *ep)
 	pci_epc_linkup(epc);
 }
 
-static void __dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar,
-				   int flags)
+static unsigned int dw_pcie_ep_func_select(struct dw_pcie_ep *ep, u8 func_no)
+{
+	unsigned int func_offset = 0;
+
+	if (ep->ops->func_conf_select)
+		func_offset = ep->ops->func_conf_select(ep, func_no);
+
+	return func_offset;
+}
+
+static void __dw_pcie_ep_reset_bar(struct dw_pcie *pci, u8 func_no,
+				   enum pci_barno bar, int flags)
 {
 	u32 reg;
+	unsigned int func_offset = 0;
+	struct dw_pcie_ep *ep = &pci->ep;
+
+	func_offset = dw_pcie_ep_func_select(ep, func_no);
 
-	reg = PCI_BASE_ADDRESS_0 + (4 * bar);
+	reg = func_offset + PCI_BASE_ADDRESS_0 + (4 * bar);
 	dw_pcie_dbi_ro_wr_en(pci);
 	dw_pcie_writel_dbi2(pci, reg, 0x0);
 	dw_pcie_writel_dbi(pci, reg, 0x0);
@@ -37,7 +51,12 @@ static void __dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar,
 
 void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar)
 {
-	__dw_pcie_ep_reset_bar(pci, bar, 0);
+	u8 func_no, funcs;
+
+	funcs = pci->ep.epc->max_functions;
+
+	for (func_no = 0; func_no < funcs; func_no++)
+		__dw_pcie_ep_reset_bar(pci, func_no, bar, 0);
 }
 
 static int dw_pcie_ep_write_header(struct pci_epc *epc, u8 func_no,
@@ -45,28 +64,31 @@ static int dw_pcie_ep_write_header(struct pci_epc *epc, u8 func_no,
 {
 	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
+	unsigned int func_offset = 0;
+
+	func_offset = dw_pcie_ep_func_select(ep, func_no);
 
 	dw_pcie_dbi_ro_wr_en(pci);
-	dw_pcie_writew_dbi(pci, PCI_VENDOR_ID, hdr->vendorid);
-	dw_pcie_writew_dbi(pci, PCI_DEVICE_ID, hdr->deviceid);
-	dw_pcie_writeb_dbi(pci, PCI_REVISION_ID, hdr->revid);
-	dw_pcie_writeb_dbi(pci, PCI_CLASS_PROG, hdr->progif_code);
-	dw_pcie_writew_dbi(pci, PCI_CLASS_DEVICE,
+	dw_pcie_writew_dbi(pci, func_offset + PCI_VENDOR_ID, hdr->vendorid);
+	dw_pcie_writew_dbi(pci, func_offset + PCI_DEVICE_ID, hdr->deviceid);
+	dw_pcie_writeb_dbi(pci, func_offset + PCI_REVISION_ID, hdr->revid);
+	dw_pcie_writeb_dbi(pci, func_offset + PCI_CLASS_PROG, hdr->progif_code);
+	dw_pcie_writew_dbi(pci, func_offset + PCI_CLASS_DEVICE,
 			   hdr->subclass_code | hdr->baseclass_code << 8);
-	dw_pcie_writeb_dbi(pci, PCI_CACHE_LINE_SIZE,
+	dw_pcie_writeb_dbi(pci, func_offset + PCI_CACHE_LINE_SIZE,
 			   hdr->cache_line_size);
-	dw_pcie_writew_dbi(pci, PCI_SUBSYSTEM_VENDOR_ID,
+	dw_pcie_writew_dbi(pci, func_offset + PCI_SUBSYSTEM_VENDOR_ID,
 			   hdr->subsys_vendor_id);
-	dw_pcie_writew_dbi(pci, PCI_SUBSYSTEM_ID, hdr->subsys_id);
-	dw_pcie_writeb_dbi(pci, PCI_INTERRUPT_PIN,
+	dw_pcie_writew_dbi(pci, func_offset + PCI_SUBSYSTEM_ID, hdr->subsys_id);
+	dw_pcie_writeb_dbi(pci, func_offset + PCI_INTERRUPT_PIN,
 			   hdr->interrupt_pin);
 	dw_pcie_dbi_ro_wr_dis(pci);
 
 	return 0;
 }
 
-static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, enum pci_barno bar,
-				  dma_addr_t cpu_addr,
+static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no,
+				  enum pci_barno bar, dma_addr_t cpu_addr,
 				  enum dw_pcie_as_type as_type)
 {
 	int ret;
@@ -79,7 +101,7 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, enum pci_barno bar,
 		return -EINVAL;
 	}
 
-	ret = dw_pcie_prog_inbound_atu(pci, free_win, bar, cpu_addr,
+	ret = dw_pcie_prog_inbound_atu(pci, func_no, free_win, bar, cpu_addr,
 				       as_type);
 	if (ret < 0) {
 		dev_err(pci->dev, "Failed to program IB window\n");
@@ -92,7 +114,8 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, enum pci_barno bar,
 	return 0;
 }
 
-static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, phys_addr_t phys_addr,
+static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, u8 func_no,
+				   phys_addr_t phys_addr,
 				   u64 pci_addr, size_t size)
 {
 	u32 free_win;
@@ -104,8 +127,8 @@ static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, phys_addr_t phys_addr,
 		return -EINVAL;
 	}
 
-	dw_pcie_prog_outbound_atu(pci, free_win, PCIE_ATU_TYPE_MEM,
-				  phys_addr, pci_addr, size);
+	dw_pcie_prog_ep_outbound_atu(pci, func_no, free_win, PCIE_ATU_TYPE_MEM,
+				     phys_addr, pci_addr, size);
 
 	set_bit(free_win, ep->ob_window_map);
 	ep->outbound_addr[free_win] = phys_addr;
@@ -121,7 +144,7 @@ static void dw_pcie_ep_clear_bar(struct pci_epc *epc, u8 func_no,
 	enum pci_barno bar = epf_bar->barno;
 	u32 atu_index = ep->bar_to_atu[bar];
 
-	__dw_pcie_ep_reset_bar(pci, bar, epf_bar->flags);
+	__dw_pcie_ep_reset_bar(pci, func_no, bar, epf_bar->flags);
 
 	dw_pcie_disable_atu(pci, atu_index, DW_PCIE_REGION_INBOUND);
 	clear_bit(atu_index, ep->ib_window_map);
@@ -137,14 +160,20 @@ static int dw_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no,
 	size_t size = epf_bar->size;
 	int flags = epf_bar->flags;
 	enum dw_pcie_as_type as_type;
-	u32 reg = PCI_BASE_ADDRESS_0 + (4 * bar);
+	u32 reg;
+	unsigned int func_offset = 0;
+
+	func_offset = dw_pcie_ep_func_select(ep, func_no);
+
+	reg = PCI_BASE_ADDRESS_0 + (4 * bar) + func_offset;
 
 	if (!(flags & PCI_BASE_ADDRESS_SPACE))
 		as_type = DW_PCIE_AS_MEM;
 	else
 		as_type = DW_PCIE_AS_IO;
 
-	ret = dw_pcie_ep_inbound_atu(ep, bar, epf_bar->phys_addr, as_type);
+	ret = dw_pcie_ep_inbound_atu(ep, func_no, bar,
+				     epf_bar->phys_addr, as_type);
 	if (ret)
 		return ret;
 
@@ -202,7 +231,7 @@ static int dw_pcie_ep_map_addr(struct pci_epc *epc, u8 func_no,
 	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 
-	ret = dw_pcie_ep_outbound_atu(ep, addr, pci_addr, size);
+	ret = dw_pcie_ep_outbound_atu(ep, func_no, addr, pci_addr, size);
 	if (ret) {
 		dev_err(pci->dev, "Failed to enable address\n");
 		return ret;
@@ -216,11 +245,14 @@ static int dw_pcie_ep_get_msi(struct pci_epc *epc, u8 func_no)
 	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	u32 val, reg;
+	unsigned int func_offset = 0;
 
 	if (!ep->msi_cap)
 		return -EINVAL;
 
-	reg = ep->msi_cap + PCI_MSI_FLAGS;
+	func_offset = dw_pcie_ep_func_select(ep, func_no);
+
+	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
 	val = dw_pcie_readw_dbi(pci, reg);
 	if (!(val & PCI_MSI_FLAGS_ENABLE))
 		return -EINVAL;
@@ -235,11 +267,14 @@ static int dw_pcie_ep_set_msi(struct pci_epc *epc, u8 func_no, u8 interrupts)
 	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	u32 val, reg;
+	unsigned int func_offset = 0;
 
 	if (!ep->msi_cap)
 		return -EINVAL;
 
-	reg = ep->msi_cap + PCI_MSI_FLAGS;
+	func_offset = dw_pcie_ep_func_select(ep, func_no);
+
+	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
 	val = dw_pcie_readw_dbi(pci, reg);
 	val &= ~PCI_MSI_FLAGS_QMASK;
 	val |= (interrupts << 1) & PCI_MSI_FLAGS_QMASK;
@@ -255,11 +290,14 @@ static int dw_pcie_ep_get_msix(struct pci_epc *epc, u8 func_no)
 	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	u32 val, reg;
+	unsigned int func_offset = 0;
 
 	if (!ep->msix_cap)
 		return -EINVAL;
 
-	reg = ep->msix_cap + PCI_MSIX_FLAGS;
+	func_offset = dw_pcie_ep_func_select(ep, func_no);
+
+	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
 	val = dw_pcie_readw_dbi(pci, reg);
 	if (!(val & PCI_MSIX_FLAGS_ENABLE))
 		return -EINVAL;
@@ -274,11 +312,14 @@ static int dw_pcie_ep_set_msix(struct pci_epc *epc, u8 func_no, u16 interrupts)
 	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	u32 val, reg;
+	unsigned int func_offset = 0;
 
 	if (!ep->msix_cap)
 		return -EINVAL;
 
-	reg = ep->msix_cap + PCI_MSIX_FLAGS;
+	func_offset = dw_pcie_ep_func_select(ep, func_no);
+
+	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
 	val = dw_pcie_readw_dbi(pci, reg);
 	val &= ~PCI_MSIX_FLAGS_QSIZE;
 	val |= interrupts;
@@ -365,6 +406,7 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	struct pci_epc *epc = ep->epc;
 	unsigned int aligned_offset;
+	unsigned int func_offset = 0;
 	u16 msg_ctrl, msg_data;
 	u32 msg_addr_lower, msg_addr_upper, reg;
 	u64 msg_addr;
@@ -374,20 +416,22 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
 	if (!ep->msi_cap)
 		return -EINVAL;
 
+	func_offset = dw_pcie_ep_func_select(ep, func_no);
+
 	/* Raise MSI per the PCI Local Bus Specification Revision 3.0, 6.8.1. */
-	reg = ep->msi_cap + PCI_MSI_FLAGS;
+	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
 	msg_ctrl = dw_pcie_readw_dbi(pci, reg);
 	has_upper = !!(msg_ctrl & PCI_MSI_FLAGS_64BIT);
-	reg = ep->msi_cap + PCI_MSI_ADDRESS_LO;
+	reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_LO;
 	msg_addr_lower = dw_pcie_readl_dbi(pci, reg);
 	if (has_upper) {
-		reg = ep->msi_cap + PCI_MSI_ADDRESS_HI;
+		reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_HI;
 		msg_addr_upper = dw_pcie_readl_dbi(pci, reg);
-		reg = ep->msi_cap + PCI_MSI_DATA_64;
+		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_64;
 		msg_data = dw_pcie_readw_dbi(pci, reg);
 	} else {
 		msg_addr_upper = 0;
-		reg = ep->msi_cap + PCI_MSI_DATA_32;
+		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_32;
 		msg_data = dw_pcie_readw_dbi(pci, reg);
 	}
 	aligned_offset = msg_addr_lower & (epc->mem->page_size - 1);
@@ -406,11 +450,12 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
 }
 
 int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
-			     u16 interrupt_num)
+			      u16 interrupt_num)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	struct pci_epc *epc = ep->epc;
 	u16 tbl_offset, bir;
+	unsigned int func_offset = 0;
 	u32 bar_addr_upper, bar_addr_lower;
 	u32 msg_addr_upper, msg_addr_lower;
 	u32 reg, msg_data, vec_ctrl;
@@ -418,12 +463,14 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
 	void __iomem *msix_tbl;
 	int ret;
 
-	reg = ep->msix_cap + PCI_MSIX_TABLE;
+	func_offset = dw_pcie_ep_func_select(ep, func_no);
+
+	reg = ep->msix_cap + func_offset + PCI_MSIX_TABLE;
 	tbl_offset = dw_pcie_readl_dbi(pci, reg);
 	bir = (tbl_offset & PCI_MSIX_TABLE_BIR);
 	tbl_offset &= PCI_MSIX_TABLE_OFFSET;
 
-	reg = PCI_BASE_ADDRESS_0 + (4 * bir);
+	reg = PCI_BASE_ADDRESS_0 + func_offset + (4 * bir);
 	bar_addr_upper = 0;
 	bar_addr_lower = dw_pcie_readl_dbi(pci, reg);
 	reg_u64 = (bar_addr_lower & PCI_BASE_ADDRESS_MEM_TYPE_MASK);
@@ -559,13 +606,13 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 	ep->epc = epc;
 	epc_set_drvdata(epc, ep);
 
-	if (ep->ops->ep_init)
-		ep->ops->ep_init(ep);
-
 	ret = of_property_read_u8(np, "max-functions", &epc->max_functions);
 	if (ret < 0)
 		epc->max_functions = 1;
 
+	if (ep->ops->ep_init)
+		ep->ops->ep_init(ep);
+
 	ret = __pci_epc_mem_init(epc, ep->phys_base, ep->addr_size,
 				 ep->page_size);
 	if (ret < 0) {
diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
index 143cb6c..ede2e75 100644
--- a/drivers/pci/controller/dwc/pcie-designware.c
+++ b/drivers/pci/controller/dwc/pcie-designware.c
@@ -238,9 +238,10 @@ static void dw_pcie_writel_ob_unroll(struct dw_pcie *pci, u32 index, u32 reg,
 	dw_pcie_writel_atu(pci, offset + reg, val);
 }
 
-static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
-					     int type, u64 cpu_addr,
-					     u64 pci_addr, u32 size)
+static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, u8 func_no,
+					     int index, int type,
+					     u64 cpu_addr, u64 pci_addr,
+					     u32 size)
 {
 	u32 retries, val;
 
@@ -255,7 +256,7 @@ static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
 	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_UPPER_TARGET,
 				 upper_32_bits(pci_addr));
 	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1,
-				 type);
+				 type | PCIE_ATU_FUNC_NUM(func_no));
 	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL2,
 				 PCIE_ATU_ENABLE);
 
@@ -274,8 +275,9 @@ static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
 	dev_err(pci->dev, "Outbound iATU is not being enabled\n");
 }
 
-void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
-			       u64 cpu_addr, u64 pci_addr, u32 size)
+static void __dw_pcie_prog_outbound_atu(struct dw_pcie *pci, u8 func_no,
+					int index, int type, u64 cpu_addr,
+					u64 pci_addr, u32 size)
 {
 	u32 retries, val;
 
@@ -283,8 +285,8 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
 		cpu_addr = pci->ops->cpu_addr_fixup(pci, cpu_addr);
 
 	if (pci->iatu_unroll_enabled) {
-		dw_pcie_prog_outbound_atu_unroll(pci, index, type, cpu_addr,
-						 pci_addr, size);
+		dw_pcie_prog_outbound_atu_unroll(pci, func_no, index, type,
+						 cpu_addr, pci_addr, size);
 		return;
 	}
 
@@ -300,7 +302,8 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
 			   lower_32_bits(pci_addr));
 	dw_pcie_writel_dbi(pci, PCIE_ATU_UPPER_TARGET,
 			   upper_32_bits(pci_addr));
-	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type);
+	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type |
+			   PCIE_ATU_FUNC_NUM(func_no));
 	dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE);
 
 	/*
@@ -317,6 +320,21 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
 	dev_err(pci->dev, "Outbound iATU is not being enabled\n");
 }
 
+void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
+			       u64 cpu_addr, u64 pci_addr, u32 size)
+{
+	__dw_pcie_prog_outbound_atu(pci, 0, index, type,
+				    cpu_addr, pci_addr, size);
+}
+
+void dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int index,
+				  int type, u64 cpu_addr, u64 pci_addr,
+				  u32 size)
+{
+	__dw_pcie_prog_outbound_atu(pci, func_no, index, type,
+				    cpu_addr, pci_addr, size);
+}
+
 static u32 dw_pcie_readl_ib_unroll(struct dw_pcie *pci, u32 index, u32 reg)
 {
 	u32 offset = PCIE_GET_ATU_INB_UNR_REG_OFFSET(index);
@@ -332,8 +350,8 @@ static void dw_pcie_writel_ib_unroll(struct dw_pcie *pci, u32 index, u32 reg,
 	dw_pcie_writel_atu(pci, offset + reg, val);
 }
 
-static int dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, int index,
-					   int bar, u64 cpu_addr,
+static int dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, u8 func_no,
+					   int index, int bar, u64 cpu_addr,
 					   enum dw_pcie_as_type as_type)
 {
 	int type;
@@ -355,8 +373,10 @@ static int dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, int index,
 		return -EINVAL;
 	}
 
-	dw_pcie_writel_ib_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1, type);
+	dw_pcie_writel_ib_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1, type |
+				 PCIE_ATU_FUNC_NUM(func_no));
 	dw_pcie_writel_ib_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL2,
+				 PCIE_ATU_FUNC_NUM_MATCH_EN |
 				 PCIE_ATU_ENABLE |
 				 PCIE_ATU_BAR_MODE_ENABLE | (bar << 8));
 
@@ -377,14 +397,15 @@ static int dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, int index,
 	return -EBUSY;
 }
 
-int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int bar,
-			     u64 cpu_addr, enum dw_pcie_as_type as_type)
+int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, u8 func_no, int index,
+			     int bar, u64 cpu_addr,
+			     enum dw_pcie_as_type as_type)
 {
 	int type;
 	u32 retries, val;
 
 	if (pci->iatu_unroll_enabled)
-		return dw_pcie_prog_inbound_atu_unroll(pci, index, bar,
+		return dw_pcie_prog_inbound_atu_unroll(pci, func_no, index, bar,
 						       cpu_addr, as_type);
 
 	dw_pcie_writel_dbi(pci, PCIE_ATU_VIEWPORT, PCIE_ATU_REGION_INBOUND |
@@ -403,9 +424,11 @@ int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int bar,
 		return -EINVAL;
 	}
 
-	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type);
-	dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE
-			   | PCIE_ATU_BAR_MODE_ENABLE | (bar << 8));
+	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type |
+			   PCIE_ATU_FUNC_NUM(func_no));
+	dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE |
+			   PCIE_ATU_FUNC_NUM_MATCH_EN |
+			   PCIE_ATU_BAR_MODE_ENABLE | (bar << 8));
 
 	/*
 	 * Make sure ATU enable takes effect before any subsequent config
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index 5a18e94..6aca0bb 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -71,9 +71,11 @@
 #define PCIE_ATU_TYPE_IO		0x2
 #define PCIE_ATU_TYPE_CFG0		0x4
 #define PCIE_ATU_TYPE_CFG1		0x5
+#define PCIE_ATU_FUNC_NUM(pf)           ((pf) << 20)
 #define PCIE_ATU_CR2			0x908
 #define PCIE_ATU_ENABLE			BIT(31)
 #define PCIE_ATU_BAR_MODE_ENABLE	BIT(30)
+#define PCIE_ATU_FUNC_NUM_MATCH_EN      BIT(19)
 #define PCIE_ATU_LOWER_BASE		0x90C
 #define PCIE_ATU_UPPER_BASE		0x910
 #define PCIE_ATU_LIMIT			0x914
@@ -206,6 +208,14 @@ struct dw_pcie_ep_ops {
 	int	(*raise_irq)(struct dw_pcie_ep *ep, u8 func_no,
 			     enum pci_epc_irq_type type, u16 interrupt_num);
 	const struct pci_epc_features* (*get_features)(struct dw_pcie_ep *ep);
+	/*
+	 * Provide a method to implement the different func config space
+	 * access for different platform, if different func have different
+	 * offset, return the offset of func. if use write a register way
+	 * return a 0, and implement code in callback function of platform
+	 * driver.
+	 */
+	unsigned int (*func_conf_select)(struct dw_pcie_ep *ep, u8 func_no);
 };
 
 struct dw_pcie_ep {
@@ -277,8 +287,12 @@ int dw_pcie_wait_for_link(struct dw_pcie *pci);
 void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index,
 			       int type, u64 cpu_addr, u64 pci_addr,
 			       u32 size);
-int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int bar,
-			     u64 cpu_addr, enum dw_pcie_as_type as_type);
+void dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int index,
+				  int type, u64 cpu_addr, u64 pci_addr,
+				  u32 size);
+int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, u8 func_no, int index,
+			     int bar, u64 cpu_addr,
+			     enum dw_pcie_as_type as_type);
 void dw_pcie_disable_atu(struct dw_pcie *pci, int index,
 			 enum dw_pcie_region_type type);
 void dw_pcie_setup(struct dw_pcie *pci);
-- 
2.9.5


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 42+ messages in thread

* [PATCH v3 02/11] PCI: designware-ep: Add the doorbell mode of MSI-X in EP mode
  2019-09-02  3:17 [PATCH v3 00/11] *** SUBJECT HERE *** Xiaowei Bao
  2019-09-02  3:17 ` [PATCH v3 01/11] PCI: designware-ep: Add multiple PFs support for DWC Xiaowei Bao
@ 2019-09-02  3:17 ` Xiaowei Bao
  2019-09-02  3:17 ` [PATCH v3 03/11] PCI: designware-ep: Move the function of getting MSI capability forward Xiaowei Bao
                   ` (9 subsequent siblings)
  11 siblings, 0 replies; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-02  3:17 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, leoyang.li, kishon,
	lorenzo.pieralisi, minghuan.Lian, mingkai.hu, roy.zang,
	jingoohan1, gustavo.pimentel, linux-pci, devicetree,
	linux-kernel, linux-arm-kernel, linuxppc-dev
  Cc: gregkh, zhiqiang.hou, Xiaowei Bao, arnd

Add the doorbell mode of MSI-X in EP mode.

Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
Reviewed-by: Andrew Murray <andrew.murray@arm.com>
---
v2:
 - Remove the macro of no used.
v3:
 - No change.

 drivers/pci/controller/dwc/pcie-designware-ep.c | 14 ++++++++++++++
 drivers/pci/controller/dwc/pcie-designware.h    | 12 ++++++++++++
 2 files changed, 26 insertions(+)

diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index eb851c2..55b23ce 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -449,6 +449,20 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
 	return 0;
 }
 
+int dw_pcie_ep_raise_msix_irq_doorbell(struct dw_pcie_ep *ep, u8 func_no,
+				       u16 interrupt_num)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
+	u32 msg_data;
+
+	msg_data = (func_no << PCIE_MSIX_DOORBELL_PF_SHIFT) |
+		   (interrupt_num - 1);
+
+	dw_pcie_writel_dbi(pci, PCIE_MSIX_DOORBELL, msg_data);
+
+	return 0;
+}
+
 int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
 			      u16 interrupt_num)
 {
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index 6aca0bb..56789be 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -88,6 +88,9 @@
 #define PCIE_MISC_CONTROL_1_OFF		0x8BC
 #define PCIE_DBI_RO_WR_EN		BIT(0)
 
+#define PCIE_MSIX_DOORBELL		0x948
+#define PCIE_MSIX_DOORBELL_PF_SHIFT	24
+
 #define PCIE_PL_CHK_REG_CONTROL_STATUS			0xB20
 #define PCIE_PL_CHK_REG_CHK_REG_START			BIT(0)
 #define PCIE_PL_CHK_REG_CHK_REG_CONTINUOUS		BIT(1)
@@ -419,6 +422,8 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
 			     u8 interrupt_num);
 int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
 			     u16 interrupt_num);
+int dw_pcie_ep_raise_msix_irq_doorbell(struct dw_pcie_ep *ep, u8 func_no,
+				       u16 interrupt_num);
 void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar);
 #else
 static inline void dw_pcie_ep_linkup(struct dw_pcie_ep *ep)
@@ -451,6 +456,13 @@ static inline int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
 	return 0;
 }
 
+static inline int dw_pcie_ep_raise_msix_irq_doorbell(struct dw_pcie_ep *ep,
+						     u8 func_no,
+						     u16 interrupt_num)
+{
+	return 0;
+}
+
 static inline void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar)
 {
 }
-- 
2.9.5


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 42+ messages in thread

* [PATCH v3 03/11] PCI: designware-ep: Move the function of getting MSI capability forward
  2019-09-02  3:17 [PATCH v3 00/11] *** SUBJECT HERE *** Xiaowei Bao
  2019-09-02  3:17 ` [PATCH v3 01/11] PCI: designware-ep: Add multiple PFs support for DWC Xiaowei Bao
  2019-09-02  3:17 ` [PATCH v3 02/11] PCI: designware-ep: Add the doorbell mode of MSI-X in EP mode Xiaowei Bao
@ 2019-09-02  3:17 ` Xiaowei Bao
  2019-09-02  3:17 ` [PATCH v3 04/11] PCI: designware-ep: Modify MSI and MSIX CAP way of finding Xiaowei Bao
                   ` (8 subsequent siblings)
  11 siblings, 0 replies; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-02  3:17 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, leoyang.li, kishon,
	lorenzo.pieralisi, minghuan.Lian, mingkai.hu, roy.zang,
	jingoohan1, gustavo.pimentel, linux-pci, devicetree,
	linux-kernel, linux-arm-kernel, linuxppc-dev
  Cc: gregkh, zhiqiang.hou, Xiaowei Bao, arnd

Move the function of getting MSI capability to the front of init
function, because the init function of the EP platform driver will use
the return value by the function of getting MSI capability.

Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
Reviewed-by: Andrew Murray <andrew.murray@arm.com>
---
v2:
 - No change.
v3:
 - No change.

 drivers/pci/controller/dwc/pcie-designware-ep.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index 55b23ce..c3bc7bd 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -624,6 +624,10 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 	if (ret < 0)
 		epc->max_functions = 1;
 
+	ep->msi_cap = dw_pcie_find_capability(pci, PCI_CAP_ID_MSI);
+
+	ep->msix_cap = dw_pcie_find_capability(pci, PCI_CAP_ID_MSIX);
+
 	if (ep->ops->ep_init)
 		ep->ops->ep_init(ep);
 
@@ -640,9 +644,6 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 		dev_err(dev, "Failed to reserve memory for MSI/MSI-X\n");
 		return -ENOMEM;
 	}
-	ep->msi_cap = dw_pcie_find_capability(pci, PCI_CAP_ID_MSI);
-
-	ep->msix_cap = dw_pcie_find_capability(pci, PCI_CAP_ID_MSIX);
 
 	offset = dw_pcie_ep_find_ext_capability(pci, PCI_EXT_CAP_ID_REBAR);
 	if (offset) {
-- 
2.9.5


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 42+ messages in thread

* [PATCH v3 04/11] PCI: designware-ep: Modify MSI and MSIX CAP way of finding
  2019-09-02  3:17 [PATCH v3 00/11] *** SUBJECT HERE *** Xiaowei Bao
                   ` (2 preceding siblings ...)
  2019-09-02  3:17 ` [PATCH v3 03/11] PCI: designware-ep: Move the function of getting MSI capability forward Xiaowei Bao
@ 2019-09-02  3:17 ` Xiaowei Bao
  2019-09-02 15:07   ` Andrew Murray
  2019-09-02  3:17 ` [PATCH v3 05/11] dt-bindings: pci: layerscape-pci: add compatible strings for ls1088a and ls2088a Xiaowei Bao
                   ` (7 subsequent siblings)
  11 siblings, 1 reply; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-02  3:17 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, leoyang.li, kishon,
	lorenzo.pieralisi, minghuan.Lian, mingkai.hu, roy.zang,
	jingoohan1, gustavo.pimentel, linux-pci, devicetree,
	linux-kernel, linux-arm-kernel, linuxppc-dev
  Cc: gregkh, zhiqiang.hou, Xiaowei Bao, arnd

Each PF of EP device should have it's own MSI or MSIX capabitily
struct, so create a dw_pcie_ep_func struct and remover the msi_cap
and msix_cap to this struce, and manage the PFs with a list.

Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
---
v1:
 - This is a new patch, to fix the issue of MSI and MSIX CAP way of
   finding.
v2:
 - No change.
v3:
 - No change.

 drivers/pci/controller/dwc/pcie-designware-ep.c | 135 +++++++++++++++++++++---
 drivers/pci/controller/dwc/pcie-designware.h    |  18 +++-
 2 files changed, 134 insertions(+), 19 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index c3bc7bd..144eb12 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -19,6 +19,19 @@ void dw_pcie_ep_linkup(struct dw_pcie_ep *ep)
 	pci_epc_linkup(epc);
 }
 
+struct dw_pcie_ep_func *
+dw_pcie_ep_get_func_from_ep(struct dw_pcie_ep *ep, u8 func_no)
+{
+	struct dw_pcie_ep_func *ep_func;
+
+	list_for_each_entry(ep_func, &ep->func_list, list) {
+		if (ep_func->func_no == func_no)
+			return ep_func;
+	}
+
+	return NULL;
+}
+
 static unsigned int dw_pcie_ep_func_select(struct dw_pcie_ep *ep, u8 func_no)
 {
 	unsigned int func_offset = 0;
@@ -59,6 +72,47 @@ void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar)
 		__dw_pcie_ep_reset_bar(pci, func_no, bar, 0);
 }
 
+static u8 __dw_pcie_ep_find_next_cap(struct dw_pcie_ep *ep, u8 func_no,
+		u8 cap_ptr, u8 cap)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
+	unsigned int func_offset = 0;
+	u8 cap_id, next_cap_ptr;
+	u16 reg;
+
+	if (!cap_ptr)
+		return 0;
+
+	func_offset = dw_pcie_ep_func_select(ep, func_no);
+
+	reg = dw_pcie_readw_dbi(pci, func_offset + cap_ptr);
+	cap_id = (reg & 0x00ff);
+
+	if (cap_id > PCI_CAP_ID_MAX)
+		return 0;
+
+	if (cap_id == cap)
+		return cap_ptr;
+
+	next_cap_ptr = (reg & 0xff00) >> 8;
+	return __dw_pcie_ep_find_next_cap(ep, func_no, next_cap_ptr, cap);
+}
+
+static u8 dw_pcie_ep_find_capability(struct dw_pcie_ep *ep, u8 func_no, u8 cap)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
+	unsigned int func_offset = 0;
+	u8 next_cap_ptr;
+	u16 reg;
+
+	func_offset = dw_pcie_ep_func_select(ep, func_no);
+
+	reg = dw_pcie_readw_dbi(pci, func_offset + PCI_CAPABILITY_LIST);
+	next_cap_ptr = (reg & 0x00ff);
+
+	return __dw_pcie_ep_find_next_cap(ep, func_no, next_cap_ptr, cap);
+}
+
 static int dw_pcie_ep_write_header(struct pci_epc *epc, u8 func_no,
 				   struct pci_epf_header *hdr)
 {
@@ -246,13 +300,18 @@ static int dw_pcie_ep_get_msi(struct pci_epc *epc, u8 func_no)
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	u32 val, reg;
 	unsigned int func_offset = 0;
+	struct dw_pcie_ep_func *ep_func;
 
-	if (!ep->msi_cap)
+	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
+	if (!ep_func)
+		return -EINVAL;
+
+	if (!ep_func->msi_cap)
 		return -EINVAL;
 
 	func_offset = dw_pcie_ep_func_select(ep, func_no);
 
-	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
+	reg = ep_func->msi_cap + func_offset + PCI_MSI_FLAGS;
 	val = dw_pcie_readw_dbi(pci, reg);
 	if (!(val & PCI_MSI_FLAGS_ENABLE))
 		return -EINVAL;
@@ -268,13 +327,18 @@ static int dw_pcie_ep_set_msi(struct pci_epc *epc, u8 func_no, u8 interrupts)
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	u32 val, reg;
 	unsigned int func_offset = 0;
+	struct dw_pcie_ep_func *ep_func;
+
+	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
+	if (!ep_func)
+		return -EINVAL;
 
-	if (!ep->msi_cap)
+	if (!ep_func->msi_cap)
 		return -EINVAL;
 
 	func_offset = dw_pcie_ep_func_select(ep, func_no);
 
-	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
+	reg = ep_func->msi_cap + func_offset + PCI_MSI_FLAGS;
 	val = dw_pcie_readw_dbi(pci, reg);
 	val &= ~PCI_MSI_FLAGS_QMASK;
 	val |= (interrupts << 1) & PCI_MSI_FLAGS_QMASK;
@@ -291,13 +355,18 @@ static int dw_pcie_ep_get_msix(struct pci_epc *epc, u8 func_no)
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	u32 val, reg;
 	unsigned int func_offset = 0;
+	struct dw_pcie_ep_func *ep_func;
+
+	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
+	if (!ep_func)
+		return -EINVAL;
 
-	if (!ep->msix_cap)
+	if (!ep_func->msix_cap)
 		return -EINVAL;
 
 	func_offset = dw_pcie_ep_func_select(ep, func_no);
 
-	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
+	reg = ep_func->msix_cap + func_offset + PCI_MSIX_FLAGS;
 	val = dw_pcie_readw_dbi(pci, reg);
 	if (!(val & PCI_MSIX_FLAGS_ENABLE))
 		return -EINVAL;
@@ -313,13 +382,18 @@ static int dw_pcie_ep_set_msix(struct pci_epc *epc, u8 func_no, u16 interrupts)
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	u32 val, reg;
 	unsigned int func_offset = 0;
+	struct dw_pcie_ep_func *ep_func;
 
-	if (!ep->msix_cap)
+	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
+	if (!ep_func)
+		return -EINVAL;
+
+	if (!ep_func->msix_cap)
 		return -EINVAL;
 
 	func_offset = dw_pcie_ep_func_select(ep, func_no);
 
-	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
+	reg = ep_func->msix_cap + func_offset + PCI_MSIX_FLAGS;
 	val = dw_pcie_readw_dbi(pci, reg);
 	val &= ~PCI_MSIX_FLAGS_QSIZE;
 	val |= interrupts;
@@ -404,6 +478,7 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
 			     u8 interrupt_num)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
+	struct dw_pcie_ep_func *ep_func;
 	struct pci_epc *epc = ep->epc;
 	unsigned int aligned_offset;
 	unsigned int func_offset = 0;
@@ -413,25 +488,29 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
 	bool has_upper;
 	int ret;
 
-	if (!ep->msi_cap)
+	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
+	if (!ep_func)
+		return -EINVAL;
+
+	if (!ep_func->msi_cap)
 		return -EINVAL;
 
 	func_offset = dw_pcie_ep_func_select(ep, func_no);
 
 	/* Raise MSI per the PCI Local Bus Specification Revision 3.0, 6.8.1. */
-	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
+	reg = ep_func->msi_cap + func_offset + PCI_MSI_FLAGS;
 	msg_ctrl = dw_pcie_readw_dbi(pci, reg);
 	has_upper = !!(msg_ctrl & PCI_MSI_FLAGS_64BIT);
-	reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_LO;
+	reg = ep_func->msi_cap + func_offset + PCI_MSI_ADDRESS_LO;
 	msg_addr_lower = dw_pcie_readl_dbi(pci, reg);
 	if (has_upper) {
-		reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_HI;
+		reg = ep_func->msi_cap + func_offset + PCI_MSI_ADDRESS_HI;
 		msg_addr_upper = dw_pcie_readl_dbi(pci, reg);
-		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_64;
+		reg = ep_func->msi_cap + func_offset + PCI_MSI_DATA_64;
 		msg_data = dw_pcie_readw_dbi(pci, reg);
 	} else {
 		msg_addr_upper = 0;
-		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_32;
+		reg = ep_func->msi_cap + func_offset + PCI_MSI_DATA_32;
 		msg_data = dw_pcie_readw_dbi(pci, reg);
 	}
 	aligned_offset = msg_addr_lower & (epc->mem->page_size - 1);
@@ -467,6 +546,7 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
 			      u16 interrupt_num)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
+	struct dw_pcie_ep_func *ep_func;
 	struct pci_epc *epc = ep->epc;
 	u16 tbl_offset, bir;
 	unsigned int func_offset = 0;
@@ -477,9 +557,16 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
 	void __iomem *msix_tbl;
 	int ret;
 
+	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
+	if (!ep_func)
+		return -EINVAL;
+
+	if (!ep_func->msix_cap)
+		return -EINVAL;
+
 	func_offset = dw_pcie_ep_func_select(ep, func_no);
 
-	reg = ep->msix_cap + func_offset + PCI_MSIX_TABLE;
+	reg = ep_func->msix_cap + func_offset + PCI_MSIX_TABLE;
 	tbl_offset = dw_pcie_readl_dbi(pci, reg);
 	bir = (tbl_offset & PCI_MSIX_TABLE_BIR);
 	tbl_offset &= PCI_MSIX_TABLE_OFFSET;
@@ -558,6 +645,7 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 	int i;
 	int ret;
 	u32 reg;
+	u8 func_no;
 	void *addr;
 	unsigned int nbars;
 	unsigned int offset;
@@ -565,6 +653,9 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	struct device *dev = pci->dev;
 	struct device_node *np = dev->of_node;
+	struct dw_pcie_ep_func *ep_func;
+
+	INIT_LIST_HEAD(&ep->func_list);
 
 	if (!pci->dbi_base || !pci->dbi_base2) {
 		dev_err(dev, "dbi_base/dbi_base2 is not populated\n");
@@ -624,9 +715,19 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 	if (ret < 0)
 		epc->max_functions = 1;
 
-	ep->msi_cap = dw_pcie_find_capability(pci, PCI_CAP_ID_MSI);
+	for (func_no = 0; func_no < epc->max_functions; func_no++) {
+		ep_func = devm_kzalloc(dev, sizeof(*ep_func), GFP_KERNEL);
+		if (!ep_func)
+			return -ENOMEM;
 
-	ep->msix_cap = dw_pcie_find_capability(pci, PCI_CAP_ID_MSIX);
+		ep_func->func_no = func_no;
+		ep_func->msi_cap = dw_pcie_ep_find_capability(ep, func_no,
+							      PCI_CAP_ID_MSI);
+		ep_func->msix_cap = dw_pcie_ep_find_capability(ep, func_no,
+							       PCI_CAP_ID_MSIX);
+
+		list_add_tail(&ep_func->list, &ep->func_list);
+	}
 
 	if (ep->ops->ep_init)
 		ep->ops->ep_init(ep);
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index 56789be..a57743c 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -221,8 +221,16 @@ struct dw_pcie_ep_ops {
 	unsigned int (*func_conf_select)(struct dw_pcie_ep *ep, u8 func_no);
 };
 
+struct dw_pcie_ep_func {
+	struct list_head	list;
+	u8			func_no;
+	u8			msi_cap;	/* MSI capability offset */
+	u8			msix_cap;	/* MSI-X capability offset */
+};
+
 struct dw_pcie_ep {
 	struct pci_epc		*epc;
+	struct list_head	func_list;
 	const struct dw_pcie_ep_ops *ops;
 	phys_addr_t		phys_base;
 	size_t			addr_size;
@@ -235,8 +243,6 @@ struct dw_pcie_ep {
 	u32			num_ob_windows;
 	void __iomem		*msi_mem;
 	phys_addr_t		msi_mem_phys;
-	u8			msi_cap;	/* MSI capability offset */
-	u8			msix_cap;	/* MSI-X capability offset */
 };
 
 struct dw_pcie_ops {
@@ -425,6 +431,8 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
 int dw_pcie_ep_raise_msix_irq_doorbell(struct dw_pcie_ep *ep, u8 func_no,
 				       u16 interrupt_num);
 void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar);
+struct dw_pcie_ep_func *
+dw_pcie_ep_get_func_from_ep(struct dw_pcie_ep *ep, u8 func_no);
 #else
 static inline void dw_pcie_ep_linkup(struct dw_pcie_ep *ep)
 {
@@ -466,5 +474,11 @@ static inline int dw_pcie_ep_raise_msix_irq_doorbell(struct dw_pcie_ep *ep,
 static inline void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar)
 {
 }
+
+struct dw_pcie_ep_func *
+dw_pcie_ep_get_func_from_ep(struct dw_pcie_ep *ep, u8 func_no)
+{
+	return NULL;
+}
 #endif
 #endif /* _PCIE_DESIGNWARE_H */
-- 
2.9.5


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 42+ messages in thread

* [PATCH v3 05/11] dt-bindings: pci: layerscape-pci: add compatible strings for ls1088a and ls2088a
  2019-09-02  3:17 [PATCH v3 00/11] *** SUBJECT HERE *** Xiaowei Bao
                   ` (3 preceding siblings ...)
  2019-09-02  3:17 ` [PATCH v3 04/11] PCI: designware-ep: Modify MSI and MSIX CAP way of finding Xiaowei Bao
@ 2019-09-02  3:17 ` Xiaowei Bao
  2019-09-02 12:31   ` Andrew Murray
  2019-09-02  3:17 ` [PATCH v3 06/11] PCI: layerscape: Fix some format issue of the code Xiaowei Bao
                   ` (6 subsequent siblings)
  11 siblings, 1 reply; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-02  3:17 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, leoyang.li, kishon,
	lorenzo.pieralisi, minghuan.Lian, mingkai.hu, roy.zang,
	jingoohan1, gustavo.pimentel, linux-pci, devicetree,
	linux-kernel, linux-arm-kernel, linuxppc-dev
  Cc: gregkh, zhiqiang.hou, Xiaowei Bao, arnd

Add compatible strings for ls1088a and ls2088a.

Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
---
v2:
 - No change.
v3:
 - Use one valid combination of compatible strings.

 Documentation/devicetree/bindings/pci/layerscape-pci.txt | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Documentation/devicetree/bindings/pci/layerscape-pci.txt b/Documentation/devicetree/bindings/pci/layerscape-pci.txt
index e20ceaa..762ae41 100644
--- a/Documentation/devicetree/bindings/pci/layerscape-pci.txt
+++ b/Documentation/devicetree/bindings/pci/layerscape-pci.txt
@@ -22,7 +22,9 @@ Required properties:
         "fsl,ls1043a-pcie"
         "fsl,ls1012a-pcie"
   EP mode:
-	"fsl,ls1046a-pcie-ep", "fsl,ls-pcie-ep"
+	"fsl,ls1046a-pcie-ep" "fsl,ls-pcie-ep"
+	"fsl,ls1088a-pcie-ep" "fsl,ls-pcie-ep"
+	"fsl,ls2088a-pcie-ep" "fsl,ls-pcie-ep"
 - reg: base addresses and lengths of the PCIe controller register blocks.
 - interrupts: A list of interrupt outputs of the controller. Must contain an
   entry for each entry in the interrupt-names property.
-- 
2.9.5


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 42+ messages in thread

* [PATCH v3 06/11] PCI: layerscape: Fix some format issue of the code
  2019-09-02  3:17 [PATCH v3 00/11] *** SUBJECT HERE *** Xiaowei Bao
                   ` (4 preceding siblings ...)
  2019-09-02  3:17 ` [PATCH v3 05/11] dt-bindings: pci: layerscape-pci: add compatible strings for ls1088a and ls2088a Xiaowei Bao
@ 2019-09-02  3:17 ` Xiaowei Bao
  2019-09-02  3:17 ` [PATCH v3 07/11] PCI: layerscape: Modify the way of getting capability with different PEX Xiaowei Bao
                   ` (5 subsequent siblings)
  11 siblings, 0 replies; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-02  3:17 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, leoyang.li, kishon,
	lorenzo.pieralisi, minghuan.Lian, mingkai.hu, roy.zang,
	jingoohan1, gustavo.pimentel, linux-pci, devicetree,
	linux-kernel, linux-arm-kernel, linuxppc-dev
  Cc: gregkh, zhiqiang.hou, Xiaowei Bao, arnd

Fix some format issue of the code in EP driver.

Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
Reviewed-by: Andrew Murray <andrew.murray@arm.com>
---
v2:
 - No change.
v3:
 - No change.

 drivers/pci/controller/dwc/pci-layerscape-ep.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c b/drivers/pci/controller/dwc/pci-layerscape-ep.c
index ca9aa45..a9c552e 100644
--- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
+++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
@@ -63,7 +63,7 @@ static void ls_pcie_ep_init(struct dw_pcie_ep *ep)
 }
 
 static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
-				  enum pci_epc_irq_type type, u16 interrupt_num)
+				enum pci_epc_irq_type type, u16 interrupt_num)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 
@@ -87,7 +87,7 @@ static const struct dw_pcie_ep_ops pcie_ep_ops = {
 };
 
 static int __init ls_add_pcie_ep(struct ls_pcie_ep *pcie,
-					struct platform_device *pdev)
+				 struct platform_device *pdev)
 {
 	struct dw_pcie *pci = pcie->pci;
 	struct device *dev = pci->dev;
-- 
2.9.5


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 42+ messages in thread

* [PATCH v3 07/11] PCI: layerscape: Modify the way of getting capability with different PEX
  2019-09-02  3:17 [PATCH v3 00/11] *** SUBJECT HERE *** Xiaowei Bao
                   ` (5 preceding siblings ...)
  2019-09-02  3:17 ` [PATCH v3 06/11] PCI: layerscape: Fix some format issue of the code Xiaowei Bao
@ 2019-09-02  3:17 ` Xiaowei Bao
  2019-09-02 13:37   ` Andrew Murray
  2019-09-02  3:17 ` [PATCH v3 08/11] PCI: layerscape: Modify the MSIX to the doorbell mode Xiaowei Bao
                   ` (4 subsequent siblings)
  11 siblings, 1 reply; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-02  3:17 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, leoyang.li, kishon,
	lorenzo.pieralisi, minghuan.Lian, mingkai.hu, roy.zang,
	jingoohan1, gustavo.pimentel, linux-pci, devicetree,
	linux-kernel, linux-arm-kernel, linuxppc-dev
  Cc: gregkh, zhiqiang.hou, Xiaowei Bao, arnd

The different PCIe controller in one board may be have different
capability of MSI or MSIX, so change the way of getting the MSI
capability, make it more flexible.

Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
---
v2:
 - Remove the repeated assignment code.
v3:
 - Use ep_func msi_cap and msix_cap to decide the msi_capable and
   msix_capable of pci_epc_features struct.

 drivers/pci/controller/dwc/pci-layerscape-ep.c | 31 +++++++++++++++++++-------
 1 file changed, 23 insertions(+), 8 deletions(-)

diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c b/drivers/pci/controller/dwc/pci-layerscape-ep.c
index a9c552e..1e07287 100644
--- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
+++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
@@ -22,6 +22,7 @@
 
 struct ls_pcie_ep {
 	struct dw_pcie		*pci;
+	struct pci_epc_features	*ls_epc;
 };
 
 #define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
@@ -40,26 +41,31 @@ static const struct of_device_id ls_pcie_ep_of_match[] = {
 	{ },
 };
 
-static const struct pci_epc_features ls_pcie_epc_features = {
-	.linkup_notifier = false,
-	.msi_capable = true,
-	.msix_capable = false,
-	.bar_fixed_64bit = (1 << BAR_2) | (1 << BAR_4),
-};
-
 static const struct pci_epc_features*
 ls_pcie_ep_get_features(struct dw_pcie_ep *ep)
 {
-	return &ls_pcie_epc_features;
+	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
+	struct ls_pcie_ep *pcie = to_ls_pcie_ep(pci);
+
+	return pcie->ls_epc;
 }
 
 static void ls_pcie_ep_init(struct dw_pcie_ep *ep)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
+	struct ls_pcie_ep *pcie = to_ls_pcie_ep(pci);
+	struct dw_pcie_ep_func *ep_func;
 	enum pci_barno bar;
 
+	ep_func = dw_pcie_ep_get_func_from_ep(ep, 0);
+	if (!ep_func)
+		return;
+
 	for (bar = BAR_0; bar <= BAR_5; bar++)
 		dw_pcie_ep_reset_bar(pci, bar);
+
+	pcie->ls_epc->msi_capable = ep_func->msi_cap ? true : false;
+	pcie->ls_epc->msix_capable = ep_func->msix_cap ? true : false;
 }
 
 static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
@@ -119,6 +125,7 @@ static int __init ls_pcie_ep_probe(struct platform_device *pdev)
 	struct device *dev = &pdev->dev;
 	struct dw_pcie *pci;
 	struct ls_pcie_ep *pcie;
+	struct pci_epc_features *ls_epc;
 	struct resource *dbi_base;
 	int ret;
 
@@ -130,6 +137,10 @@ static int __init ls_pcie_ep_probe(struct platform_device *pdev)
 	if (!pci)
 		return -ENOMEM;
 
+	ls_epc = devm_kzalloc(dev, sizeof(*ls_epc), GFP_KERNEL);
+	if (!ls_epc)
+		return -ENOMEM;
+
 	dbi_base = platform_get_resource_byname(pdev, IORESOURCE_MEM, "regs");
 	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_base);
 	if (IS_ERR(pci->dbi_base))
@@ -140,6 +151,10 @@ static int __init ls_pcie_ep_probe(struct platform_device *pdev)
 	pci->ops = &ls_pcie_ep_ops;
 	pcie->pci = pci;
 
+	ls_epc->bar_fixed_64bit = (1 << BAR_2) | (1 << BAR_4),
+
+	pcie->ls_epc = ls_epc;
+
 	platform_set_drvdata(pdev, pcie);
 
 	ret = ls_add_pcie_ep(pcie, pdev);
-- 
2.9.5


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 42+ messages in thread

* [PATCH v3 08/11] PCI: layerscape: Modify the MSIX to the doorbell mode
  2019-09-02  3:17 [PATCH v3 00/11] *** SUBJECT HERE *** Xiaowei Bao
                   ` (6 preceding siblings ...)
  2019-09-02  3:17 ` [PATCH v3 07/11] PCI: layerscape: Modify the way of getting capability with different PEX Xiaowei Bao
@ 2019-09-02  3:17 ` Xiaowei Bao
  2019-09-02 12:01   ` Andrew Murray
  2019-09-02  3:17 ` [PATCH v3 09/11] PCI: layerscape: Add EP mode support for ls1088a and ls2088a Xiaowei Bao
                   ` (3 subsequent siblings)
  11 siblings, 1 reply; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-02  3:17 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, leoyang.li, kishon,
	lorenzo.pieralisi, minghuan.Lian, mingkai.hu, roy.zang,
	jingoohan1, gustavo.pimentel, linux-pci, devicetree,
	linux-kernel, linux-arm-kernel, linuxppc-dev
  Cc: gregkh, zhiqiang.hou, Xiaowei Bao, arnd

dw_pcie_ep_raise_msix_irq was never called in the exisitng driver
before, because the ls1046a platform don't support the MSIX feature
and msix_capable was always set to false.
Now that add the ls1088a platform with MSIX support, but the existing
dw_pcie_ep_raise_msix_irq doesn't work, so use the doorbell method to
support the MSIX feature.

Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
---
v2: 
 - No change
v3:
 - Modify the commit message make it clearly.

 drivers/pci/controller/dwc/pci-layerscape-ep.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c b/drivers/pci/controller/dwc/pci-layerscape-ep.c
index 1e07287..5f0cb99 100644
--- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
+++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
@@ -79,7 +79,8 @@ static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 	case PCI_EPC_IRQ_MSI:
 		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
 	case PCI_EPC_IRQ_MSIX:
-		return dw_pcie_ep_raise_msix_irq(ep, func_no, interrupt_num);
+		return dw_pcie_ep_raise_msix_irq_doorbell(ep, func_no,
+							  interrupt_num);
 	default:
 		dev_err(pci->dev, "UNKNOWN IRQ type\n");
 		return -EINVAL;
-- 
2.9.5


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 42+ messages in thread

* [PATCH v3 09/11] PCI: layerscape: Add EP mode support for ls1088a and ls2088a
  2019-09-02  3:17 [PATCH v3 00/11] *** SUBJECT HERE *** Xiaowei Bao
                   ` (7 preceding siblings ...)
  2019-09-02  3:17 ` [PATCH v3 08/11] PCI: layerscape: Modify the MSIX to the doorbell mode Xiaowei Bao
@ 2019-09-02  3:17 ` Xiaowei Bao
  2019-09-02 12:46   ` Andrew Murray
  2019-09-02  3:17 ` [PATCH v3 10/11] arm64: dts: layerscape: Add PCIe EP node for ls1088a Xiaowei Bao
                   ` (2 subsequent siblings)
  11 siblings, 1 reply; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-02  3:17 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, leoyang.li, kishon,
	lorenzo.pieralisi, minghuan.Lian, mingkai.hu, roy.zang,
	jingoohan1, gustavo.pimentel, linux-pci, devicetree,
	linux-kernel, linux-arm-kernel, linuxppc-dev
  Cc: gregkh, zhiqiang.hou, Xiaowei Bao, arnd

Add PCIe EP mode support for ls1088a and ls2088a, there are some
difference between LS1 and LS2 platform, so refactor the code of
the EP driver.

Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
---
v2: 
 - This is a new patch for supporting the ls1088a and ls2088a platform.
v3:
 - Adjust the some struct assignment order in probe function.

 drivers/pci/controller/dwc/pci-layerscape-ep.c | 72 +++++++++++++++++++-------
 1 file changed, 53 insertions(+), 19 deletions(-)

diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c b/drivers/pci/controller/dwc/pci-layerscape-ep.c
index 5f0cb99..723bbe5 100644
--- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
+++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
@@ -20,27 +20,29 @@
 
 #define PCIE_DBI2_OFFSET		0x1000	/* DBI2 base address*/
 
-struct ls_pcie_ep {
-	struct dw_pcie		*pci;
-	struct pci_epc_features	*ls_epc;
+#define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
+
+struct ls_pcie_ep_drvdata {
+	u32				func_offset;
+	const struct dw_pcie_ep_ops	*ops;
+	const struct dw_pcie_ops	*dw_pcie_ops;
 };
 
-#define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
+struct ls_pcie_ep {
+	struct dw_pcie			*pci;
+	struct pci_epc_features		*ls_epc;
+	const struct ls_pcie_ep_drvdata *drvdata;
+};
 
 static int ls_pcie_establish_link(struct dw_pcie *pci)
 {
 	return 0;
 }
 
-static const struct dw_pcie_ops ls_pcie_ep_ops = {
+static const struct dw_pcie_ops dw_ls_pcie_ep_ops = {
 	.start_link = ls_pcie_establish_link,
 };
 
-static const struct of_device_id ls_pcie_ep_of_match[] = {
-	{ .compatible = "fsl,ls-pcie-ep",},
-	{ },
-};
-
 static const struct pci_epc_features*
 ls_pcie_ep_get_features(struct dw_pcie_ep *ep)
 {
@@ -87,10 +89,39 @@ static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 	}
 }
 
-static const struct dw_pcie_ep_ops pcie_ep_ops = {
+static unsigned int ls_pcie_ep_func_conf_select(struct dw_pcie_ep *ep,
+						u8 func_no)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
+	struct ls_pcie_ep *pcie = to_ls_pcie_ep(pci);
+
+	WARN_ON(func_no && !pcie->drvdata->func_offset);
+	return pcie->drvdata->func_offset * func_no;
+}
+
+static const struct dw_pcie_ep_ops ls_pcie_ep_ops = {
 	.ep_init = ls_pcie_ep_init,
 	.raise_irq = ls_pcie_ep_raise_irq,
 	.get_features = ls_pcie_ep_get_features,
+	.func_conf_select = ls_pcie_ep_func_conf_select,
+};
+
+static const struct ls_pcie_ep_drvdata ls1_ep_drvdata = {
+	.ops = &ls_pcie_ep_ops,
+	.dw_pcie_ops = &dw_ls_pcie_ep_ops,
+};
+
+static const struct ls_pcie_ep_drvdata ls2_ep_drvdata = {
+	.func_offset = 0x20000,
+	.ops = &ls_pcie_ep_ops,
+	.dw_pcie_ops = &dw_ls_pcie_ep_ops,
+};
+
+static const struct of_device_id ls_pcie_ep_of_match[] = {
+	{ .compatible = "fsl,ls1046a-pcie-ep", .data = &ls1_ep_drvdata },
+	{ .compatible = "fsl,ls1088a-pcie-ep", .data = &ls2_ep_drvdata },
+	{ .compatible = "fsl,ls2088a-pcie-ep", .data = &ls2_ep_drvdata },
+	{ },
 };
 
 static int __init ls_add_pcie_ep(struct ls_pcie_ep *pcie,
@@ -103,7 +134,7 @@ static int __init ls_add_pcie_ep(struct ls_pcie_ep *pcie,
 	int ret;
 
 	ep = &pci->ep;
-	ep->ops = &pcie_ep_ops;
+	ep->ops = pcie->drvdata->ops;
 
 	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "addr_space");
 	if (!res)
@@ -142,20 +173,23 @@ static int __init ls_pcie_ep_probe(struct platform_device *pdev)
 	if (!ls_epc)
 		return -ENOMEM;
 
-	dbi_base = platform_get_resource_byname(pdev, IORESOURCE_MEM, "regs");
-	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_base);
-	if (IS_ERR(pci->dbi_base))
-		return PTR_ERR(pci->dbi_base);
+	pcie->drvdata = of_device_get_match_data(dev);
 
-	pci->dbi_base2 = pci->dbi_base + PCIE_DBI2_OFFSET;
 	pci->dev = dev;
-	pci->ops = &ls_pcie_ep_ops;
-	pcie->pci = pci;
+	pci->ops = pcie->drvdata->dw_pcie_ops;
 
 	ls_epc->bar_fixed_64bit = (1 << BAR_2) | (1 << BAR_4),
 
+	pcie->pci = pci;
 	pcie->ls_epc = ls_epc;
 
+	dbi_base = platform_get_resource_byname(pdev, IORESOURCE_MEM, "regs");
+	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_base);
+	if (IS_ERR(pci->dbi_base))
+		return PTR_ERR(pci->dbi_base);
+
+	pci->dbi_base2 = pci->dbi_base + PCIE_DBI2_OFFSET;
+
 	platform_set_drvdata(pdev, pcie);
 
 	ret = ls_add_pcie_ep(pcie, pdev);
-- 
2.9.5


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 42+ messages in thread

* [PATCH v3 10/11] arm64: dts: layerscape: Add PCIe EP node for ls1088a
  2019-09-02  3:17 [PATCH v3 00/11] *** SUBJECT HERE *** Xiaowei Bao
                   ` (8 preceding siblings ...)
  2019-09-02  3:17 ` [PATCH v3 09/11] PCI: layerscape: Add EP mode support for ls1088a and ls2088a Xiaowei Bao
@ 2019-09-02  3:17 ` Xiaowei Bao
  2019-09-02 13:06   ` Andrew Murray
  2019-09-02  3:17 ` [PATCH v3 11/11] misc: pci_endpoint_test: Add LS1088a in pci_device_id table Xiaowei Bao
  2019-09-02  3:52 ` [PATCH v3 00/11] *** SUBJECT HERE *** Z.q. Hou
  11 siblings, 1 reply; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-02  3:17 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, leoyang.li, kishon,
	lorenzo.pieralisi, minghuan.Lian, mingkai.hu, roy.zang,
	jingoohan1, gustavo.pimentel, linux-pci, devicetree,
	linux-kernel, linux-arm-kernel, linuxppc-dev
  Cc: gregkh, zhiqiang.hou, Xiaowei Bao, arnd

Add PCIe EP node for ls1088a to support EP mode.

Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
---
v2:
 - Remove the pf-offset proparty.
v3:
 - No change.
 
 arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi | 31 ++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi b/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
index c676d07..da246ab 100644
--- a/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
+++ b/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
@@ -483,6 +483,17 @@
 			status = "disabled";
 		};
 
+		pcie_ep@3400000 {
+			compatible = "fsl,ls1088a-pcie-ep","fsl,ls-pcie-ep";
+			reg = <0x00 0x03400000 0x0 0x00100000
+			       0x20 0x00000000 0x8 0x00000000>;
+			reg-names = "regs", "addr_space";
+			num-ib-windows = <24>;
+			num-ob-windows = <128>;
+			max-functions = /bits/ 8 <2>;
+			status = "disabled";
+		};
+
 		pcie@3500000 {
 			compatible = "fsl,ls1088a-pcie";
 			reg = <0x00 0x03500000 0x0 0x00100000   /* controller registers */
@@ -508,6 +519,16 @@
 			status = "disabled";
 		};
 
+		pcie_ep@3500000 {
+			compatible = "fsl,ls1088a-pcie-ep","fsl,ls-pcie-ep";
+			reg = <0x00 0x03500000 0x0 0x00100000
+			       0x28 0x00000000 0x8 0x00000000>;
+			reg-names = "regs", "addr_space";
+			num-ib-windows = <6>;
+			num-ob-windows = <8>;
+			status = "disabled";
+		};
+
 		pcie@3600000 {
 			compatible = "fsl,ls1088a-pcie";
 			reg = <0x00 0x03600000 0x0 0x00100000   /* controller registers */
@@ -533,6 +554,16 @@
 			status = "disabled";
 		};
 
+		pcie_ep@3600000 {
+			compatible = "fsl,ls1088a-pcie-ep","fsl,ls-pcie-ep";
+			reg = <0x00 0x03600000 0x0 0x00100000
+			       0x30 0x00000000 0x8 0x00000000>;
+			reg-names = "regs", "addr_space";
+			num-ib-windows = <6>;
+			num-ob-windows = <8>;
+			status = "disabled";
+		};
+
 		smmu: iommu@5000000 {
 			compatible = "arm,mmu-500";
 			reg = <0 0x5000000 0 0x800000>;
-- 
2.9.5


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 42+ messages in thread

* [PATCH v3 11/11] misc: pci_endpoint_test: Add LS1088a in pci_device_id table
  2019-09-02  3:17 [PATCH v3 00/11] *** SUBJECT HERE *** Xiaowei Bao
                   ` (9 preceding siblings ...)
  2019-09-02  3:17 ` [PATCH v3 10/11] arm64: dts: layerscape: Add PCIe EP node for ls1088a Xiaowei Bao
@ 2019-09-02  3:17 ` Xiaowei Bao
  2019-09-02 12:54   ` Andrew Murray
  2019-09-02  3:52 ` [PATCH v3 00/11] *** SUBJECT HERE *** Z.q. Hou
  11 siblings, 1 reply; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-02  3:17 UTC (permalink / raw)
  To: robh+dt, mark.rutland, shawnguo, leoyang.li, kishon,
	lorenzo.pieralisi, minghuan.Lian, mingkai.hu, roy.zang,
	jingoohan1, gustavo.pimentel, linux-pci, devicetree,
	linux-kernel, linux-arm-kernel, linuxppc-dev
  Cc: gregkh, zhiqiang.hou, Xiaowei Bao, arnd

Add LS1088a in pci_device_id table so that pci-epf-test can be used
for testing PCIe EP in LS1088a.

Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
---
v2:
 - No change.
v3:
 - No change.
 
 drivers/misc/pci_endpoint_test.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
index 6e208a0..d531951 100644
--- a/drivers/misc/pci_endpoint_test.c
+++ b/drivers/misc/pci_endpoint_test.c
@@ -793,6 +793,7 @@ static const struct pci_device_id pci_endpoint_test_tbl[] = {
 	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_DRA74x) },
 	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_DRA72x) },
 	{ PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, 0x81c0) },
+	{ PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, 0x80c0) },
 	{ PCI_DEVICE_DATA(SYNOPSYS, EDDA, NULL) },
 	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_AM654),
 	  .driver_data = (kernel_ulong_t)&am654_data
-- 
2.9.5


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 00/11] *** SUBJECT HERE ***
  2019-09-02  3:17 [PATCH v3 00/11] *** SUBJECT HERE *** Xiaowei Bao
                   ` (10 preceding siblings ...)
  2019-09-02  3:17 ` [PATCH v3 11/11] misc: pci_endpoint_test: Add LS1088a in pci_device_id table Xiaowei Bao
@ 2019-09-02  3:52 ` Z.q. Hou
  2019-09-02  3:54   ` Xiaowei Bao
  11 siblings, 1 reply; 42+ messages in thread
From: Z.q. Hou @ 2019-09-02  3:52 UTC (permalink / raw)
  To: Xiaowei Bao, robh+dt, mark.rutland, shawnguo, Leo Li, kishon,
	lorenzo.pieralisi, M.h. Lian, Mingkai Hu, Roy Zang, jingoohan1,
	gustavo.pimentel, linux-pci, devicetree, linux-kernel,
	linux-arm-kernel, linuxppc-dev
  Cc: gregkh, Xiaowei Bao, arnd

Xiaowei,

> -----Original Message-----
> From: Xiaowei Bao <xiaowei.bao@nxp.com>
> Sent: 2019年9月2日 11:17
> To: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org;
> Leo Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> M.h. Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>;
> Roy Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org
> Cc: arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> <zhiqiang.hou@nxp.com>; Xiaowei Bao <xiaowei.bao@nxp.com>
> Subject: [PATCH v3 00/11] *** SUBJECT HERE ***
> 
> *** BLURB HERE ***

Add subject and blurb for this series.

Thanks,
Zhiqiang

> 
> Xiaowei Bao (11):
>   PCI: designware-ep: Add multiple PFs support for DWC
>   PCI: designware-ep: Add the doorbell mode of MSI-X in EP mode
>   PCI: designware-ep: Move the function of getting MSI capability
>     forward
>   PCI: designware-ep: Modify MSI and MSIX CAP way of finding
>   dt-bindings: pci: layerscape-pci: add compatible strings for ls1088a
>     and ls2088a
>   PCI: layerscape: Fix some format issue of the code
>   PCI: layerscape: Modify the way of getting capability with different
>     PEX
>   PCI: layerscape: Modify the MSIX to the doorbell mode
>   PCI: layerscape: Add EP mode support for ls1088a and ls2088a
>   arm64: dts: layerscape: Add PCIe EP node for ls1088a
>   misc: pci_endpoint_test: Add LS1088a in pci_device_id table
> 
>  .../devicetree/bindings/pci/layerscape-pci.txt     |   4 +-
>  arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi     |  31 +++
>  drivers/misc/pci_endpoint_test.c                   |   1 +
>  drivers/pci/controller/dwc/pci-layerscape-ep.c     | 100 ++++++--
>  drivers/pci/controller/dwc/pcie-designware-ep.c    | 255
> +++++++++++++++++----
>  drivers/pci/controller/dwc/pcie-designware.c       |  59 +++--
>  drivers/pci/controller/dwc/pcie-designware.h       |  48 +++-
>  7 files changed, 404 insertions(+), 94 deletions(-)
> 
> --
> 2.9.5

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 00/11] *** SUBJECT HERE ***
  2019-09-02  3:52 ` [PATCH v3 00/11] *** SUBJECT HERE *** Z.q. Hou
@ 2019-09-02  3:54   ` Xiaowei Bao
  0 siblings, 0 replies; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-02  3:54 UTC (permalink / raw)
  To: Z.q. Hou, robh+dt, mark.rutland, shawnguo, Leo Li, kishon,
	lorenzo.pieralisi, M.h. Lian, Mingkai Hu, Roy Zang, jingoohan1,
	gustavo.pimentel, linux-pci, devicetree, linux-kernel,
	linux-arm-kernel, linuxppc-dev
  Cc: gregkh, arnd



> -----Original Message-----
> From: Z.q. Hou
> Sent: 2019年9月2日 11:52
> To: Xiaowei Bao <xiaowei.bao@nxp.com>; robh+dt@kernel.org;
> mark.rutland@arm.com; shawnguo@kernel.org; Leo Li
> <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h. Lian
> <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy Zang
> <roy.zang@nxp.com>; jingoohan1@gmail.com;
> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org
> Cc: arnd@arndb.de; gregkh@linuxfoundation.org; Xiaowei Bao
> <xiaowei.bao@nxp.com>
> Subject: RE: [PATCH v3 00/11] *** SUBJECT HERE ***
> 
> Xiaowei,
> 
> > -----Original Message-----
> > From: Xiaowei Bao <xiaowei.bao@nxp.com>
> > Sent: 2019年9月2日 11:17
> > To: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org;
> Leo
> > Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > M.h. Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>;
> > Roy Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> > gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> > devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org
> > Cc: arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> > <zhiqiang.hou@nxp.com>; Xiaowei Bao <xiaowei.bao@nxp.com>
> > Subject: [PATCH v3 00/11] *** SUBJECT HERE ***
> >
> > *** BLURB HERE ***
> 
> Add subject and blurb for this series.

OK, thanks.


> 
> Thanks,
> Zhiqiang
> 
> >
> > Xiaowei Bao (11):
> >   PCI: designware-ep: Add multiple PFs support for DWC
> >   PCI: designware-ep: Add the doorbell mode of MSI-X in EP mode
> >   PCI: designware-ep: Move the function of getting MSI capability
> >     forward
> >   PCI: designware-ep: Modify MSI and MSIX CAP way of finding
> >   dt-bindings: pci: layerscape-pci: add compatible strings for ls1088a
> >     and ls2088a
> >   PCI: layerscape: Fix some format issue of the code
> >   PCI: layerscape: Modify the way of getting capability with different
> >     PEX
> >   PCI: layerscape: Modify the MSIX to the doorbell mode
> >   PCI: layerscape: Add EP mode support for ls1088a and ls2088a
> >   arm64: dts: layerscape: Add PCIe EP node for ls1088a
> >   misc: pci_endpoint_test: Add LS1088a in pci_device_id table
> >
> >  .../devicetree/bindings/pci/layerscape-pci.txt     |   4 +-
> >  arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi     |  31 +++
> >  drivers/misc/pci_endpoint_test.c                   |   1 +
> >  drivers/pci/controller/dwc/pci-layerscape-ep.c     | 100 ++++++--
> >  drivers/pci/controller/dwc/pcie-designware-ep.c    | 255
> > +++++++++++++++++----
> >  drivers/pci/controller/dwc/pcie-designware.c       |  59 +++--
> >  drivers/pci/controller/dwc/pcie-designware.h       |  48 +++-
> >  7 files changed, 404 insertions(+), 94 deletions(-)
> >
> > --
> > 2.9.5

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* Re: [PATCH v3 08/11] PCI: layerscape: Modify the MSIX to the doorbell mode
  2019-09-02  3:17 ` [PATCH v3 08/11] PCI: layerscape: Modify the MSIX to the doorbell mode Xiaowei Bao
@ 2019-09-02 12:01   ` Andrew Murray
  2019-09-12 11:24     ` Gustavo Pimentel
  0 siblings, 1 reply; 42+ messages in thread
From: Andrew Murray @ 2019-09-02 12:01 UTC (permalink / raw)
  To: Xiaowei Bao
  Cc: mark.rutland, roy.zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, zhiqiang.hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, minghuan.Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, leoyang.li, shawnguo, mingkai.hu

On Mon, Sep 02, 2019 at 11:17:13AM +0800, Xiaowei Bao wrote:
> dw_pcie_ep_raise_msix_irq was never called in the exisitng driver
> before, because the ls1046a platform don't support the MSIX feature
> and msix_capable was always set to false.
> Now that add the ls1088a platform with MSIX support, but the existing
> dw_pcie_ep_raise_msix_irq doesn't work, so use the doorbell method to
> support the MSIX feature.
> 
> Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>

Reviewed-by: Andrew Murray <andrew.murray@arm.com>

> ---
> v2: 
>  - No change
> v3:
>  - Modify the commit message make it clearly.
> 
>  drivers/pci/controller/dwc/pci-layerscape-ep.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> index 1e07287..5f0cb99 100644
> --- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> +++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> @@ -79,7 +79,8 @@ static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	case PCI_EPC_IRQ_MSI:
>  		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
>  	case PCI_EPC_IRQ_MSIX:
> -		return dw_pcie_ep_raise_msix_irq(ep, func_no, interrupt_num);
> +		return dw_pcie_ep_raise_msix_irq_doorbell(ep, func_no,
> +							  interrupt_num);
>  	default:
>  		dev_err(pci->dev, "UNKNOWN IRQ type\n");
>  		return -EINVAL;
> -- 
> 2.9.5
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* Re: [PATCH v3 05/11] dt-bindings: pci: layerscape-pci: add compatible strings for ls1088a and ls2088a
  2019-09-02  3:17 ` [PATCH v3 05/11] dt-bindings: pci: layerscape-pci: add compatible strings for ls1088a and ls2088a Xiaowei Bao
@ 2019-09-02 12:31   ` Andrew Murray
  2019-09-03  1:33     ` Xiaowei Bao
  0 siblings, 1 reply; 42+ messages in thread
From: Andrew Murray @ 2019-09-02 12:31 UTC (permalink / raw)
  To: Xiaowei Bao
  Cc: mark.rutland, roy.zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, zhiqiang.hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, minghuan.Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, leoyang.li, shawnguo, mingkai.hu

On Mon, Sep 02, 2019 at 11:17:10AM +0800, Xiaowei Bao wrote:
> Add compatible strings for ls1088a and ls2088a.
> 
> Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> ---
> v2:
>  - No change.
> v3:
>  - Use one valid combination of compatible strings.
> 
>  Documentation/devicetree/bindings/pci/layerscape-pci.txt | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/devicetree/bindings/pci/layerscape-pci.txt b/Documentation/devicetree/bindings/pci/layerscape-pci.txt
> index e20ceaa..762ae41 100644
> --- a/Documentation/devicetree/bindings/pci/layerscape-pci.txt
> +++ b/Documentation/devicetree/bindings/pci/layerscape-pci.txt
> @@ -22,7 +22,9 @@ Required properties:
>          "fsl,ls1043a-pcie"
>          "fsl,ls1012a-pcie"
>    EP mode:
> -	"fsl,ls1046a-pcie-ep", "fsl,ls-pcie-ep"
> +	"fsl,ls1046a-pcie-ep" "fsl,ls-pcie-ep"
> +	"fsl,ls1088a-pcie-ep" "fsl,ls-pcie-ep"
> +	"fsl,ls2088a-pcie-ep" "fsl,ls-pcie-ep"

This isn't consistent with "[PATCH v3 09/11] PCI: layerscape: Add EP mode..."
as that patch drops the fallback "fsl,ls-pcie-ep". Either the fallback must
be preserved in the driver, or you need to drop it here.

What if there are existing users that depend on the fallback?

(I'm also not sure if that comma should have been dropped).

Thanks,

Andrew Murray

>  - reg: base addresses and lengths of the PCIe controller register blocks.
>  - interrupts: A list of interrupt outputs of the controller. Must contain an
>    entry for each entry in the interrupt-names property.
> -- 
> 2.9.5
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* Re: [PATCH v3 09/11] PCI: layerscape: Add EP mode support for ls1088a and ls2088a
  2019-09-02  3:17 ` [PATCH v3 09/11] PCI: layerscape: Add EP mode support for ls1088a and ls2088a Xiaowei Bao
@ 2019-09-02 12:46   ` Andrew Murray
  2019-09-03  1:47     ` Xiaowei Bao
  0 siblings, 1 reply; 42+ messages in thread
From: Andrew Murray @ 2019-09-02 12:46 UTC (permalink / raw)
  To: Xiaowei Bao
  Cc: mark.rutland, roy.zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, zhiqiang.hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, minghuan.Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, leoyang.li, shawnguo, mingkai.hu

On Mon, Sep 02, 2019 at 11:17:14AM +0800, Xiaowei Bao wrote:
> Add PCIe EP mode support for ls1088a and ls2088a, there are some
> difference between LS1 and LS2 platform, so refactor the code of
> the EP driver.
> 
> Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> ---
> v2: 
>  - This is a new patch for supporting the ls1088a and ls2088a platform.
> v3:
>  - Adjust the some struct assignment order in probe function.
> 
>  drivers/pci/controller/dwc/pci-layerscape-ep.c | 72 +++++++++++++++++++-------
>  1 file changed, 53 insertions(+), 19 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> index 5f0cb99..723bbe5 100644
> --- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> +++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> @@ -20,27 +20,29 @@
>  
>  #define PCIE_DBI2_OFFSET		0x1000	/* DBI2 base address*/
>  
> -struct ls_pcie_ep {
> -	struct dw_pcie		*pci;
> -	struct pci_epc_features	*ls_epc;
> +#define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
> +
> +struct ls_pcie_ep_drvdata {
> +	u32				func_offset;
> +	const struct dw_pcie_ep_ops	*ops;
> +	const struct dw_pcie_ops	*dw_pcie_ops;
>  };
>  
> -#define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
> +struct ls_pcie_ep {
> +	struct dw_pcie			*pci;
> +	struct pci_epc_features		*ls_epc;
> +	const struct ls_pcie_ep_drvdata *drvdata;
> +};
>  
>  static int ls_pcie_establish_link(struct dw_pcie *pci)
>  {
>  	return 0;
>  }
>  
> -static const struct dw_pcie_ops ls_pcie_ep_ops = {
> +static const struct dw_pcie_ops dw_ls_pcie_ep_ops = {
>  	.start_link = ls_pcie_establish_link,
>  };
>  
> -static const struct of_device_id ls_pcie_ep_of_match[] = {
> -	{ .compatible = "fsl,ls-pcie-ep",},
> -	{ },
> -};
> -
>  static const struct pci_epc_features*
>  ls_pcie_ep_get_features(struct dw_pcie_ep *ep)
>  {
> @@ -87,10 +89,39 @@ static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	}
>  }
>  
> -static const struct dw_pcie_ep_ops pcie_ep_ops = {
> +static unsigned int ls_pcie_ep_func_conf_select(struct dw_pcie_ep *ep,
> +						u8 func_no)
> +{
> +	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> +	struct ls_pcie_ep *pcie = to_ls_pcie_ep(pci);
> +
> +	WARN_ON(func_no && !pcie->drvdata->func_offset);
> +	return pcie->drvdata->func_offset * func_no;
> +}
> +
> +static const struct dw_pcie_ep_ops ls_pcie_ep_ops = {
>  	.ep_init = ls_pcie_ep_init,
>  	.raise_irq = ls_pcie_ep_raise_irq,
>  	.get_features = ls_pcie_ep_get_features,
> +	.func_conf_select = ls_pcie_ep_func_conf_select,
> +};
> +
> +static const struct ls_pcie_ep_drvdata ls1_ep_drvdata = {
> +	.ops = &ls_pcie_ep_ops,
> +	.dw_pcie_ops = &dw_ls_pcie_ep_ops,
> +};
> +
> +static const struct ls_pcie_ep_drvdata ls2_ep_drvdata = {
> +	.func_offset = 0x20000,
> +	.ops = &ls_pcie_ep_ops,
> +	.dw_pcie_ops = &dw_ls_pcie_ep_ops,
> +};
> +
> +static const struct of_device_id ls_pcie_ep_of_match[] = {
> +	{ .compatible = "fsl,ls1046a-pcie-ep", .data = &ls1_ep_drvdata },
> +	{ .compatible = "fsl,ls1088a-pcie-ep", .data = &ls2_ep_drvdata },
> +	{ .compatible = "fsl,ls2088a-pcie-ep", .data = &ls2_ep_drvdata },
> +	{ },

This removes support for "fsl,ls-pcie-ep" - was that intentional? If you do
plan to drop it please make sure you explain why in the commit message. See
also my comments in your dt-binding patch.

Thanks,

Andrew Murray

>  };
>  
>  static int __init ls_add_pcie_ep(struct ls_pcie_ep *pcie,
> @@ -103,7 +134,7 @@ static int __init ls_add_pcie_ep(struct ls_pcie_ep *pcie,
>  	int ret;
>  
>  	ep = &pci->ep;
> -	ep->ops = &pcie_ep_ops;
> +	ep->ops = pcie->drvdata->ops;
>  
>  	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "addr_space");
>  	if (!res)
> @@ -142,20 +173,23 @@ static int __init ls_pcie_ep_probe(struct platform_device *pdev)
>  	if (!ls_epc)
>  		return -ENOMEM;
>  
> -	dbi_base = platform_get_resource_byname(pdev, IORESOURCE_MEM, "regs");
> -	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_base);
> -	if (IS_ERR(pci->dbi_base))
> -		return PTR_ERR(pci->dbi_base);
> +	pcie->drvdata = of_device_get_match_data(dev);
>  
> -	pci->dbi_base2 = pci->dbi_base + PCIE_DBI2_OFFSET;
>  	pci->dev = dev;
> -	pci->ops = &ls_pcie_ep_ops;
> -	pcie->pci = pci;
> +	pci->ops = pcie->drvdata->dw_pcie_ops;
>  
>  	ls_epc->bar_fixed_64bit = (1 << BAR_2) | (1 << BAR_4),
>  
> +	pcie->pci = pci;
>  	pcie->ls_epc = ls_epc;
>  
> +	dbi_base = platform_get_resource_byname(pdev, IORESOURCE_MEM, "regs");
> +	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_base);
> +	if (IS_ERR(pci->dbi_base))
> +		return PTR_ERR(pci->dbi_base);
> +
> +	pci->dbi_base2 = pci->dbi_base + PCIE_DBI2_OFFSET;
> +
>  	platform_set_drvdata(pdev, pcie);
>  
>  	ret = ls_add_pcie_ep(pcie, pdev);
> -- 
> 2.9.5
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* Re: [PATCH v3 11/11] misc: pci_endpoint_test: Add LS1088a in pci_device_id table
  2019-09-02  3:17 ` [PATCH v3 11/11] misc: pci_endpoint_test: Add LS1088a in pci_device_id table Xiaowei Bao
@ 2019-09-02 12:54   ` Andrew Murray
  2019-09-03  1:52     ` Xiaowei Bao
  0 siblings, 1 reply; 42+ messages in thread
From: Andrew Murray @ 2019-09-02 12:54 UTC (permalink / raw)
  To: Xiaowei Bao
  Cc: mark.rutland, roy.zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, zhiqiang.hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, minghuan.Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, leoyang.li, shawnguo, mingkai.hu

On Mon, Sep 02, 2019 at 11:17:16AM +0800, Xiaowei Bao wrote:
> Add LS1088a in pci_device_id table so that pci-epf-test can be used
> for testing PCIe EP in LS1088a.
> 
> Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> ---
> v2:
>  - No change.
> v3:
>  - No change.
>  
>  drivers/misc/pci_endpoint_test.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
> index 6e208a0..d531951 100644
> --- a/drivers/misc/pci_endpoint_test.c
> +++ b/drivers/misc/pci_endpoint_test.c
> @@ -793,6 +793,7 @@ static const struct pci_device_id pci_endpoint_test_tbl[] = {
>  	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_DRA74x) },
>  	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_DRA72x) },
>  	{ PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, 0x81c0) },
> +	{ PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, 0x80c0) },

The Freescale PCI devices are the only devices in this table that don't
have a define for their device ID. I think a define should be created
for both of the device IDs above.

Thanks,

Andrew Murray

>  	{ PCI_DEVICE_DATA(SYNOPSYS, EDDA, NULL) },
>  	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_AM654),
>  	  .driver_data = (kernel_ulong_t)&am654_data
> -- 
> 2.9.5
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* Re: [PATCH v3 10/11] arm64: dts: layerscape: Add PCIe EP node for ls1088a
  2019-09-02  3:17 ` [PATCH v3 10/11] arm64: dts: layerscape: Add PCIe EP node for ls1088a Xiaowei Bao
@ 2019-09-02 13:06   ` Andrew Murray
  2019-09-03  2:01     ` Xiaowei Bao
  0 siblings, 1 reply; 42+ messages in thread
From: Andrew Murray @ 2019-09-02 13:06 UTC (permalink / raw)
  To: Xiaowei Bao
  Cc: mark.rutland, roy.zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, zhiqiang.hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, minghuan.Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, leoyang.li, shawnguo, mingkai.hu

On Mon, Sep 02, 2019 at 11:17:15AM +0800, Xiaowei Bao wrote:
> Add PCIe EP node for ls1088a to support EP mode.
> 
> Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> ---
> v2:
>  - Remove the pf-offset proparty.
> v3:
>  - No change.
>  
>  arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi | 31 ++++++++++++++++++++++++++
>  1 file changed, 31 insertions(+)
> 
> diff --git a/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi b/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
> index c676d07..da246ab 100644
> --- a/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
> +++ b/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
> @@ -483,6 +483,17 @@
>  			status = "disabled";
>  		};
>  
> +		pcie_ep@3400000 {
> +			compatible = "fsl,ls1088a-pcie-ep","fsl,ls-pcie-ep";

Here you specify a fallback "fsl,ls-pcie-ep" that is removed by this series.

Besides that, this looks OK.

Thanks,

Andrew Murray

> +			reg = <0x00 0x03400000 0x0 0x00100000
> +			       0x20 0x00000000 0x8 0x00000000>;
> +			reg-names = "regs", "addr_space";
> +			num-ib-windows = <24>;
> +			num-ob-windows = <128>;
> +			max-functions = /bits/ 8 <2>;
> +			status = "disabled";
> +		};
> +
>  		pcie@3500000 {
>  			compatible = "fsl,ls1088a-pcie";
>  			reg = <0x00 0x03500000 0x0 0x00100000   /* controller registers */
> @@ -508,6 +519,16 @@
>  			status = "disabled";
>  		};
>  
> +		pcie_ep@3500000 {
> +			compatible = "fsl,ls1088a-pcie-ep","fsl,ls-pcie-ep";
> +			reg = <0x00 0x03500000 0x0 0x00100000
> +			       0x28 0x00000000 0x8 0x00000000>;
> +			reg-names = "regs", "addr_space";
> +			num-ib-windows = <6>;
> +			num-ob-windows = <8>;
> +			status = "disabled";
> +		};
> +
>  		pcie@3600000 {
>  			compatible = "fsl,ls1088a-pcie";
>  			reg = <0x00 0x03600000 0x0 0x00100000   /* controller registers */
> @@ -533,6 +554,16 @@
>  			status = "disabled";
>  		};
>  
> +		pcie_ep@3600000 {
> +			compatible = "fsl,ls1088a-pcie-ep","fsl,ls-pcie-ep";
> +			reg = <0x00 0x03600000 0x0 0x00100000
> +			       0x30 0x00000000 0x8 0x00000000>;
> +			reg-names = "regs", "addr_space";
> +			num-ib-windows = <6>;
> +			num-ob-windows = <8>;
> +			status = "disabled";
> +		};
> +
>  		smmu: iommu@5000000 {
>  			compatible = "arm,mmu-500";
>  			reg = <0 0x5000000 0 0x800000>;
> -- 
> 2.9.5
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* Re: [PATCH v3 07/11] PCI: layerscape: Modify the way of getting capability with different PEX
  2019-09-02  3:17 ` [PATCH v3 07/11] PCI: layerscape: Modify the way of getting capability with different PEX Xiaowei Bao
@ 2019-09-02 13:37   ` Andrew Murray
  2019-09-03  2:13     ` Xiaowei Bao
  0 siblings, 1 reply; 42+ messages in thread
From: Andrew Murray @ 2019-09-02 13:37 UTC (permalink / raw)
  To: Xiaowei Bao
  Cc: mark.rutland, roy.zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, zhiqiang.hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, minghuan.Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, leoyang.li, shawnguo, mingkai.hu

On Mon, Sep 02, 2019 at 11:17:12AM +0800, Xiaowei Bao wrote:
> The different PCIe controller in one board may be have different
> capability of MSI or MSIX, so change the way of getting the MSI
> capability, make it more flexible.
> 
> Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>

Please see the comments I just made to Kishon's feedback in the thread for
this patch in series v2.

Thanks,

Andrew Murray

> ---
> v2:
>  - Remove the repeated assignment code.
> v3:
>  - Use ep_func msi_cap and msix_cap to decide the msi_capable and
>    msix_capable of pci_epc_features struct.
> 
>  drivers/pci/controller/dwc/pci-layerscape-ep.c | 31 +++++++++++++++++++-------
>  1 file changed, 23 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> index a9c552e..1e07287 100644
> --- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> +++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> @@ -22,6 +22,7 @@
>  
>  struct ls_pcie_ep {
>  	struct dw_pcie		*pci;
> +	struct pci_epc_features	*ls_epc;
>  };
>  
>  #define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
> @@ -40,26 +41,31 @@ static const struct of_device_id ls_pcie_ep_of_match[] = {
>  	{ },
>  };
>  
> -static const struct pci_epc_features ls_pcie_epc_features = {
> -	.linkup_notifier = false,
> -	.msi_capable = true,
> -	.msix_capable = false,
> -	.bar_fixed_64bit = (1 << BAR_2) | (1 << BAR_4),
> -};
> -
>  static const struct pci_epc_features*
>  ls_pcie_ep_get_features(struct dw_pcie_ep *ep)
>  {
> -	return &ls_pcie_epc_features;
> +	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> +	struct ls_pcie_ep *pcie = to_ls_pcie_ep(pci);
> +
> +	return pcie->ls_epc;
>  }
>  
>  static void ls_pcie_ep_init(struct dw_pcie_ep *ep)
>  {
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> +	struct ls_pcie_ep *pcie = to_ls_pcie_ep(pci);
> +	struct dw_pcie_ep_func *ep_func;
>  	enum pci_barno bar;
>  
> +	ep_func = dw_pcie_ep_get_func_from_ep(ep, 0);
> +	if (!ep_func)
> +		return;
> +
>  	for (bar = BAR_0; bar <= BAR_5; bar++)
>  		dw_pcie_ep_reset_bar(pci, bar);
> +
> +	pcie->ls_epc->msi_capable = ep_func->msi_cap ? true : false;
> +	pcie->ls_epc->msix_capable = ep_func->msix_cap ? true : false;
>  }
>  
>  static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> @@ -119,6 +125,7 @@ static int __init ls_pcie_ep_probe(struct platform_device *pdev)
>  	struct device *dev = &pdev->dev;
>  	struct dw_pcie *pci;
>  	struct ls_pcie_ep *pcie;
> +	struct pci_epc_features *ls_epc;
>  	struct resource *dbi_base;
>  	int ret;
>  
> @@ -130,6 +137,10 @@ static int __init ls_pcie_ep_probe(struct platform_device *pdev)
>  	if (!pci)
>  		return -ENOMEM;
>  
> +	ls_epc = devm_kzalloc(dev, sizeof(*ls_epc), GFP_KERNEL);
> +	if (!ls_epc)
> +		return -ENOMEM;
> +
>  	dbi_base = platform_get_resource_byname(pdev, IORESOURCE_MEM, "regs");
>  	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_base);
>  	if (IS_ERR(pci->dbi_base))
> @@ -140,6 +151,10 @@ static int __init ls_pcie_ep_probe(struct platform_device *pdev)
>  	pci->ops = &ls_pcie_ep_ops;
>  	pcie->pci = pci;
>  
> +	ls_epc->bar_fixed_64bit = (1 << BAR_2) | (1 << BAR_4),
> +
> +	pcie->ls_epc = ls_epc;
> +
>  	platform_set_drvdata(pdev, pcie);
>  
>  	ret = ls_add_pcie_ep(pcie, pdev);
> -- 
> 2.9.5
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* Re: [PATCH v3 04/11] PCI: designware-ep: Modify MSI and MSIX CAP way of finding
  2019-09-02  3:17 ` [PATCH v3 04/11] PCI: designware-ep: Modify MSI and MSIX CAP way of finding Xiaowei Bao
@ 2019-09-02 15:07   ` Andrew Murray
  2019-09-03  2:33     ` Xiaowei Bao
  0 siblings, 1 reply; 42+ messages in thread
From: Andrew Murray @ 2019-09-02 15:07 UTC (permalink / raw)
  To: Xiaowei Bao
  Cc: mark.rutland, roy.zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, zhiqiang.hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, minghuan.Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, leoyang.li, shawnguo, mingkai.hu

On Mon, Sep 02, 2019 at 11:17:09AM +0800, Xiaowei Bao wrote:
> Each PF of EP device should have it's own MSI or MSIX capabitily
> struct, so create a dw_pcie_ep_func struct and remover the msi_cap

remover?

> and msix_cap to this struce, and manage the PFs with a list.

struce?

> 
> Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> ---
> v1:
>  - This is a new patch, to fix the issue of MSI and MSIX CAP way of
>    finding.
> v2:
>  - No change.
> v3:
>  - No change.

This makes it look like you introduced the patch in v1 and haven't
changed it since.

I think it's more common to have a history like this:

---
v3:
 - Introduced new patch, to fix the issue of MSI and MSIX CAP way of
   finding.


> 
>  drivers/pci/controller/dwc/pcie-designware-ep.c | 135 +++++++++++++++++++++---
>  drivers/pci/controller/dwc/pcie-designware.h    |  18 +++-
>  2 files changed, 134 insertions(+), 19 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
> index c3bc7bd..144eb12 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> @@ -19,6 +19,19 @@ void dw_pcie_ep_linkup(struct dw_pcie_ep *ep)
>  	pci_epc_linkup(epc);
>  }
>  
> +struct dw_pcie_ep_func *
> +dw_pcie_ep_get_func_from_ep(struct dw_pcie_ep *ep, u8 func_no)
> +{
> +	struct dw_pcie_ep_func *ep_func;
> +
> +	list_for_each_entry(ep_func, &ep->func_list, list) {
> +		if (ep_func->func_no == func_no)
> +			return ep_func;
> +	}
> +
> +	return NULL;
> +}
> +
>  static unsigned int dw_pcie_ep_func_select(struct dw_pcie_ep *ep, u8 func_no)
>  {
>  	unsigned int func_offset = 0;
> @@ -59,6 +72,47 @@ void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar)
>  		__dw_pcie_ep_reset_bar(pci, func_no, bar, 0);
>  }
>  
> +static u8 __dw_pcie_ep_find_next_cap(struct dw_pcie_ep *ep, u8 func_no,
> +		u8 cap_ptr, u8 cap)
> +{
> +	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> +	unsigned int func_offset = 0;
> +	u8 cap_id, next_cap_ptr;
> +	u16 reg;
> +
> +	if (!cap_ptr)
> +		return 0;
> +
> +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> +
> +	reg = dw_pcie_readw_dbi(pci, func_offset + cap_ptr);
> +	cap_id = (reg & 0x00ff);
> +
> +	if (cap_id > PCI_CAP_ID_MAX)
> +		return 0;
> +
> +	if (cap_id == cap)
> +		return cap_ptr;
> +
> +	next_cap_ptr = (reg & 0xff00) >> 8;
> +	return __dw_pcie_ep_find_next_cap(ep, func_no, next_cap_ptr, cap);
> +}

Which tree have you based this patchset on? v5.3-rc3 and pci/dwc both already
have this function (without the func_no). See beb4641a787d
("PCI: dwc: Add MSI-X callbacks handler").

> +
> +static u8 dw_pcie_ep_find_capability(struct dw_pcie_ep *ep, u8 func_no, u8 cap)
> +{
> +	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> +	unsigned int func_offset = 0;
> +	u8 next_cap_ptr;
> +	u16 reg;
> +
> +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> +
> +	reg = dw_pcie_readw_dbi(pci, func_offset + PCI_CAPABILITY_LIST);
> +	next_cap_ptr = (reg & 0x00ff);
> +
> +	return __dw_pcie_ep_find_next_cap(ep, func_no, next_cap_ptr, cap);
> +}
> +
>  static int dw_pcie_ep_write_header(struct pci_epc *epc, u8 func_no,
>  				   struct pci_epf_header *hdr)
>  {
> @@ -246,13 +300,18 @@ static int dw_pcie_ep_get_msi(struct pci_epc *epc, u8 func_no)
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  	u32 val, reg;
>  	unsigned int func_offset = 0;
> +	struct dw_pcie_ep_func *ep_func;
>  
> -	if (!ep->msi_cap)
> +	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
> +	if (!ep_func)
> +		return -EINVAL;
> +
> +	if (!ep_func->msi_cap)
>  		return -EINVAL;
>  
>  	func_offset = dw_pcie_ep_func_select(ep, func_no);
>  
> -	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
> +	reg = ep_func->msi_cap + func_offset + PCI_MSI_FLAGS;
>  	val = dw_pcie_readw_dbi(pci, reg);
>  	if (!(val & PCI_MSI_FLAGS_ENABLE))
>  		return -EINVAL;
> @@ -268,13 +327,18 @@ static int dw_pcie_ep_set_msi(struct pci_epc *epc, u8 func_no, u8 interrupts)
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  	u32 val, reg;
>  	unsigned int func_offset = 0;
> +	struct dw_pcie_ep_func *ep_func;
> +
> +	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
> +	if (!ep_func)
> +		return -EINVAL;
>  
> -	if (!ep->msi_cap)
> +	if (!ep_func->msi_cap)
>  		return -EINVAL;
>  
>  	func_offset = dw_pcie_ep_func_select(ep, func_no);
>  
> -	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
> +	reg = ep_func->msi_cap + func_offset + PCI_MSI_FLAGS;
>  	val = dw_pcie_readw_dbi(pci, reg);
>  	val &= ~PCI_MSI_FLAGS_QMASK;
>  	val |= (interrupts << 1) & PCI_MSI_FLAGS_QMASK;
> @@ -291,13 +355,18 @@ static int dw_pcie_ep_get_msix(struct pci_epc *epc, u8 func_no)
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  	u32 val, reg;
>  	unsigned int func_offset = 0;
> +	struct dw_pcie_ep_func *ep_func;
> +
> +	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
> +	if (!ep_func)
> +		return -EINVAL;
>  
> -	if (!ep->msix_cap)
> +	if (!ep_func->msix_cap)
>  		return -EINVAL;
>  
>  	func_offset = dw_pcie_ep_func_select(ep, func_no);
>  
> -	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
> +	reg = ep_func->msix_cap + func_offset + PCI_MSIX_FLAGS;
>  	val = dw_pcie_readw_dbi(pci, reg);
>  	if (!(val & PCI_MSIX_FLAGS_ENABLE))
>  		return -EINVAL;
> @@ -313,13 +382,18 @@ static int dw_pcie_ep_set_msix(struct pci_epc *epc, u8 func_no, u16 interrupts)
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  	u32 val, reg;
>  	unsigned int func_offset = 0;
> +	struct dw_pcie_ep_func *ep_func;
>  
> -	if (!ep->msix_cap)
> +	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
> +	if (!ep_func)
> +		return -EINVAL;
> +
> +	if (!ep_func->msix_cap)
>  		return -EINVAL;
>  
>  	func_offset = dw_pcie_ep_func_select(ep, func_no);
>  
> -	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
> +	reg = ep_func->msix_cap + func_offset + PCI_MSIX_FLAGS;
>  	val = dw_pcie_readw_dbi(pci, reg);
>  	val &= ~PCI_MSIX_FLAGS_QSIZE;
>  	val |= interrupts;
> @@ -404,6 +478,7 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
>  			     u8 interrupt_num)
>  {
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> +	struct dw_pcie_ep_func *ep_func;
>  	struct pci_epc *epc = ep->epc;
>  	unsigned int aligned_offset;
>  	unsigned int func_offset = 0;
> @@ -413,25 +488,29 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	bool has_upper;
>  	int ret;
>  
> -	if (!ep->msi_cap)
> +	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
> +	if (!ep_func)
> +		return -EINVAL;
> +
> +	if (!ep_func->msi_cap)
>  		return -EINVAL;
>  
>  	func_offset = dw_pcie_ep_func_select(ep, func_no);
>  
>  	/* Raise MSI per the PCI Local Bus Specification Revision 3.0, 6.8.1. */
> -	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
> +	reg = ep_func->msi_cap + func_offset + PCI_MSI_FLAGS;
>  	msg_ctrl = dw_pcie_readw_dbi(pci, reg);
>  	has_upper = !!(msg_ctrl & PCI_MSI_FLAGS_64BIT);
> -	reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_LO;
> +	reg = ep_func->msi_cap + func_offset + PCI_MSI_ADDRESS_LO;
>  	msg_addr_lower = dw_pcie_readl_dbi(pci, reg);
>  	if (has_upper) {
> -		reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_HI;
> +		reg = ep_func->msi_cap + func_offset + PCI_MSI_ADDRESS_HI;
>  		msg_addr_upper = dw_pcie_readl_dbi(pci, reg);
> -		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_64;
> +		reg = ep_func->msi_cap + func_offset + PCI_MSI_DATA_64;
>  		msg_data = dw_pcie_readw_dbi(pci, reg);
>  	} else {
>  		msg_addr_upper = 0;
> -		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_32;
> +		reg = ep_func->msi_cap + func_offset + PCI_MSI_DATA_32;
>  		msg_data = dw_pcie_readw_dbi(pci, reg);
>  	}
>  	aligned_offset = msg_addr_lower & (epc->mem->page_size - 1);
> @@ -467,6 +546,7 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
>  			      u16 interrupt_num)
>  {
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> +	struct dw_pcie_ep_func *ep_func;
>  	struct pci_epc *epc = ep->epc;
>  	u16 tbl_offset, bir;
>  	unsigned int func_offset = 0;
> @@ -477,9 +557,16 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	void __iomem *msix_tbl;
>  	int ret;
>  
> +	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
> +	if (!ep_func)
> +		return -EINVAL;
> +
> +	if (!ep_func->msix_cap)
> +		return -EINVAL;
> +
>  	func_offset = dw_pcie_ep_func_select(ep, func_no);
>  
> -	reg = ep->msix_cap + func_offset + PCI_MSIX_TABLE;
> +	reg = ep_func->msix_cap + func_offset + PCI_MSIX_TABLE;
>  	tbl_offset = dw_pcie_readl_dbi(pci, reg);
>  	bir = (tbl_offset & PCI_MSIX_TABLE_BIR);
>  	tbl_offset &= PCI_MSIX_TABLE_OFFSET;
> @@ -558,6 +645,7 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
>  	int i;
>  	int ret;
>  	u32 reg;
> +	u8 func_no;
>  	void *addr;
>  	unsigned int nbars;
>  	unsigned int offset;
> @@ -565,6 +653,9 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  	struct device *dev = pci->dev;
>  	struct device_node *np = dev->of_node;
> +	struct dw_pcie_ep_func *ep_func;
> +
> +	INIT_LIST_HEAD(&ep->func_list);
>  
>  	if (!pci->dbi_base || !pci->dbi_base2) {
>  		dev_err(dev, "dbi_base/dbi_base2 is not populated\n");
> @@ -624,9 +715,19 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
>  	if (ret < 0)
>  		epc->max_functions = 1;
>  
> -	ep->msi_cap = dw_pcie_find_capability(pci, PCI_CAP_ID_MSI);
> +	for (func_no = 0; func_no < epc->max_functions; func_no++) {
> +		ep_func = devm_kzalloc(dev, sizeof(*ep_func), GFP_KERNEL);
> +		if (!ep_func)
> +			return -ENOMEM;
>  
> -	ep->msix_cap = dw_pcie_find_capability(pci, PCI_CAP_ID_MSIX);
> +		ep_func->func_no = func_no;
> +		ep_func->msi_cap = dw_pcie_ep_find_capability(ep, func_no,
> +							      PCI_CAP_ID_MSI);
> +		ep_func->msix_cap = dw_pcie_ep_find_capability(ep, func_no,
> +							       PCI_CAP_ID_MSIX);
> +
> +		list_add_tail(&ep_func->list, &ep->func_list);
> +	}

Whilst your patch addresses the issue of giving each function the ability to
have differing capabilities - I feel that this solution doesn't go deep enough.

In my view the root issue here is that 'struct dw_pcie_ep' represents both a
EP controller and a *single* EP function. I think that there should be a
representation for an EP controller and a representation for a EP function
(i.e. some separation). Thus allowing one EP controller to have many EP
functions. This isn't too dissimilar to host bridges and their functions.
Others here may have different views.

It may be unlikely now, but EP functions belonging to the same bit of IP may
have differing functionality - your approach addresses that for MSI/MSI
capabilities, but what about other differences?

(It would be really nice as well if an EP controller could provide config
read/write ops such that existing functions in the core such as
__pci_find_next_capability could be reused - instead of copying them
like dw_pcie_ep_find_capability. However I don't think this is feasible.)

Thanks,

Andrew Murray

>  
>  	if (ep->ops->ep_init)
>  		ep->ops->ep_init(ep);
> diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
> index 56789be..a57743c 100644
> --- a/drivers/pci/controller/dwc/pcie-designware.h
> +++ b/drivers/pci/controller/dwc/pcie-designware.h
> @@ -221,8 +221,16 @@ struct dw_pcie_ep_ops {
>  	unsigned int (*func_conf_select)(struct dw_pcie_ep *ep, u8 func_no);
>  };
>  
> +struct dw_pcie_ep_func {
> +	struct list_head	list;
> +	u8			func_no;
> +	u8			msi_cap;	/* MSI capability offset */
> +	u8			msix_cap;	/* MSI-X capability offset */
> +};
> +
>  struct dw_pcie_ep {
>  	struct pci_epc		*epc;
> +	struct list_head	func_list;
>  	const struct dw_pcie_ep_ops *ops;
>  	phys_addr_t		phys_base;
>  	size_t			addr_size;
> @@ -235,8 +243,6 @@ struct dw_pcie_ep {
>  	u32			num_ob_windows;
>  	void __iomem		*msi_mem;
>  	phys_addr_t		msi_mem_phys;
> -	u8			msi_cap;	/* MSI capability offset */
> -	u8			msix_cap;	/* MSI-X capability offset */
>  };
>  
>  struct dw_pcie_ops {
> @@ -425,6 +431,8 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
>  int dw_pcie_ep_raise_msix_irq_doorbell(struct dw_pcie_ep *ep, u8 func_no,
>  				       u16 interrupt_num);
>  void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar);
> +struct dw_pcie_ep_func *
> +dw_pcie_ep_get_func_from_ep(struct dw_pcie_ep *ep, u8 func_no);
>  #else
>  static inline void dw_pcie_ep_linkup(struct dw_pcie_ep *ep)
>  {
> @@ -466,5 +474,11 @@ static inline int dw_pcie_ep_raise_msix_irq_doorbell(struct dw_pcie_ep *ep,
>  static inline void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar)
>  {
>  }
> +
> +struct dw_pcie_ep_func *
> +dw_pcie_ep_get_func_from_ep(struct dw_pcie_ep *ep, u8 func_no)
> +{
> +	return NULL;
> +}
>  #endif
>  #endif /* _PCIE_DESIGNWARE_H */
> -- 
> 2.9.5
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* Re: [PATCH v3 01/11] PCI: designware-ep: Add multiple PFs support for DWC
  2019-09-02  3:17 ` [PATCH v3 01/11] PCI: designware-ep: Add multiple PFs support for DWC Xiaowei Bao
@ 2019-09-02 16:26   ` Andrew Murray
  2019-09-03  3:43     ` Xiaowei Bao
  0 siblings, 1 reply; 42+ messages in thread
From: Andrew Murray @ 2019-09-02 16:26 UTC (permalink / raw)
  To: Xiaowei Bao
  Cc: mark.rutland, roy.zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, zhiqiang.hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, minghuan.Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, leoyang.li, shawnguo, mingkai.hu

On Mon, Sep 02, 2019 at 11:17:06AM +0800, Xiaowei Bao wrote:
> Add multiple PFs support for DWC, different PF have different config space
> we use pf-offset property which get from the DTS to access the different pF

This needs to be updated as this no longer comes from the DT.

> config space.
> 
> Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>


We're assuming:

 - The offset address (func_offset) between PF's in the memory map can be
   different between different DWC implementations. And also that it's
   possible for DWC implementations to address PFs without using an offset.

 - The current approach is preferable to adding DWC EP driver callbacks
   for writing to the EP config space (e.g. a variant of dw_pcie_writew_dbi
   that takes a func number).

I'm keen to hear feedback from Jingoo/Gustavo on this.

Thanks,

Andrew Murray

> ---
> v2:
>  - Remove duplicate redundant code.
>  - Reimplement the PF config space access way.
> v3:
>  - Integrate duplicate code for func_select.
>  - Move PCIE_ATU_FUNC_NUM(pf) (pf << 20) to ((pf) << 20).
>  - Add the comments for func_conf_select function.
> 
>  drivers/pci/controller/dwc/pcie-designware-ep.c | 123 ++++++++++++++++--------
>  drivers/pci/controller/dwc/pcie-designware.c    |  59 ++++++++----
>  drivers/pci/controller/dwc/pcie-designware.h    |  18 +++-
>  3 files changed, 142 insertions(+), 58 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
> index 65f4792..eb851c2 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> @@ -19,12 +19,26 @@ void dw_pcie_ep_linkup(struct dw_pcie_ep *ep)
>  	pci_epc_linkup(epc);
>  }
>  
> -static void __dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar,
> -				   int flags)
> +static unsigned int dw_pcie_ep_func_select(struct dw_pcie_ep *ep, u8 func_no)
> +{
> +	unsigned int func_offset = 0;
> +
> +	if (ep->ops->func_conf_select)
> +		func_offset = ep->ops->func_conf_select(ep, func_no);
> +
> +	return func_offset;
> +}
> +
> +static void __dw_pcie_ep_reset_bar(struct dw_pcie *pci, u8 func_no,
> +				   enum pci_barno bar, int flags)
>  {
>  	u32 reg;
> +	unsigned int func_offset = 0;
> +	struct dw_pcie_ep *ep = &pci->ep;
> +
> +	func_offset = dw_pcie_ep_func_select(ep, func_no);
>  
> -	reg = PCI_BASE_ADDRESS_0 + (4 * bar);
> +	reg = func_offset + PCI_BASE_ADDRESS_0 + (4 * bar);
>  	dw_pcie_dbi_ro_wr_en(pci);
>  	dw_pcie_writel_dbi2(pci, reg, 0x0);
>  	dw_pcie_writel_dbi(pci, reg, 0x0);
> @@ -37,7 +51,12 @@ static void __dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar,
>  
>  void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar)
>  {
> -	__dw_pcie_ep_reset_bar(pci, bar, 0);
> +	u8 func_no, funcs;
> +
> +	funcs = pci->ep.epc->max_functions;
> +
> +	for (func_no = 0; func_no < funcs; func_no++)
> +		__dw_pcie_ep_reset_bar(pci, func_no, bar, 0);
>  }
>  
>  static int dw_pcie_ep_write_header(struct pci_epc *epc, u8 func_no,
> @@ -45,28 +64,31 @@ static int dw_pcie_ep_write_header(struct pci_epc *epc, u8 func_no,
>  {
>  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> +	unsigned int func_offset = 0;
> +
> +	func_offset = dw_pcie_ep_func_select(ep, func_no);
>  
>  	dw_pcie_dbi_ro_wr_en(pci);
> -	dw_pcie_writew_dbi(pci, PCI_VENDOR_ID, hdr->vendorid);
> -	dw_pcie_writew_dbi(pci, PCI_DEVICE_ID, hdr->deviceid);
> -	dw_pcie_writeb_dbi(pci, PCI_REVISION_ID, hdr->revid);
> -	dw_pcie_writeb_dbi(pci, PCI_CLASS_PROG, hdr->progif_code);
> -	dw_pcie_writew_dbi(pci, PCI_CLASS_DEVICE,
> +	dw_pcie_writew_dbi(pci, func_offset + PCI_VENDOR_ID, hdr->vendorid);
> +	dw_pcie_writew_dbi(pci, func_offset + PCI_DEVICE_ID, hdr->deviceid);
> +	dw_pcie_writeb_dbi(pci, func_offset + PCI_REVISION_ID, hdr->revid);
> +	dw_pcie_writeb_dbi(pci, func_offset + PCI_CLASS_PROG, hdr->progif_code);
> +	dw_pcie_writew_dbi(pci, func_offset + PCI_CLASS_DEVICE,
>  			   hdr->subclass_code | hdr->baseclass_code << 8);
> -	dw_pcie_writeb_dbi(pci, PCI_CACHE_LINE_SIZE,
> +	dw_pcie_writeb_dbi(pci, func_offset + PCI_CACHE_LINE_SIZE,
>  			   hdr->cache_line_size);
> -	dw_pcie_writew_dbi(pci, PCI_SUBSYSTEM_VENDOR_ID,
> +	dw_pcie_writew_dbi(pci, func_offset + PCI_SUBSYSTEM_VENDOR_ID,
>  			   hdr->subsys_vendor_id);
> -	dw_pcie_writew_dbi(pci, PCI_SUBSYSTEM_ID, hdr->subsys_id);
> -	dw_pcie_writeb_dbi(pci, PCI_INTERRUPT_PIN,
> +	dw_pcie_writew_dbi(pci, func_offset + PCI_SUBSYSTEM_ID, hdr->subsys_id);
> +	dw_pcie_writeb_dbi(pci, func_offset + PCI_INTERRUPT_PIN,
>  			   hdr->interrupt_pin);
>  	dw_pcie_dbi_ro_wr_dis(pci);
>  
>  	return 0;
>  }
>  
> -static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, enum pci_barno bar,
> -				  dma_addr_t cpu_addr,
> +static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no,
> +				  enum pci_barno bar, dma_addr_t cpu_addr,
>  				  enum dw_pcie_as_type as_type)
>  {
>  	int ret;
> @@ -79,7 +101,7 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, enum pci_barno bar,
>  		return -EINVAL;
>  	}
>  
> -	ret = dw_pcie_prog_inbound_atu(pci, free_win, bar, cpu_addr,
> +	ret = dw_pcie_prog_inbound_atu(pci, func_no, free_win, bar, cpu_addr,
>  				       as_type);
>  	if (ret < 0) {
>  		dev_err(pci->dev, "Failed to program IB window\n");
> @@ -92,7 +114,8 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, enum pci_barno bar,
>  	return 0;
>  }
>  
> -static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, phys_addr_t phys_addr,
> +static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, u8 func_no,
> +				   phys_addr_t phys_addr,
>  				   u64 pci_addr, size_t size)
>  {
>  	u32 free_win;
> @@ -104,8 +127,8 @@ static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, phys_addr_t phys_addr,
>  		return -EINVAL;
>  	}
>  
> -	dw_pcie_prog_outbound_atu(pci, free_win, PCIE_ATU_TYPE_MEM,
> -				  phys_addr, pci_addr, size);
> +	dw_pcie_prog_ep_outbound_atu(pci, func_no, free_win, PCIE_ATU_TYPE_MEM,
> +				     phys_addr, pci_addr, size);
>  
>  	set_bit(free_win, ep->ob_window_map);
>  	ep->outbound_addr[free_win] = phys_addr;
> @@ -121,7 +144,7 @@ static void dw_pcie_ep_clear_bar(struct pci_epc *epc, u8 func_no,
>  	enum pci_barno bar = epf_bar->barno;
>  	u32 atu_index = ep->bar_to_atu[bar];
>  
> -	__dw_pcie_ep_reset_bar(pci, bar, epf_bar->flags);
> +	__dw_pcie_ep_reset_bar(pci, func_no, bar, epf_bar->flags);
>  
>  	dw_pcie_disable_atu(pci, atu_index, DW_PCIE_REGION_INBOUND);
>  	clear_bit(atu_index, ep->ib_window_map);
> @@ -137,14 +160,20 @@ static int dw_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no,
>  	size_t size = epf_bar->size;
>  	int flags = epf_bar->flags;
>  	enum dw_pcie_as_type as_type;
> -	u32 reg = PCI_BASE_ADDRESS_0 + (4 * bar);
> +	u32 reg;
> +	unsigned int func_offset = 0;
> +
> +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> +
> +	reg = PCI_BASE_ADDRESS_0 + (4 * bar) + func_offset;
>  
>  	if (!(flags & PCI_BASE_ADDRESS_SPACE))
>  		as_type = DW_PCIE_AS_MEM;
>  	else
>  		as_type = DW_PCIE_AS_IO;
>  
> -	ret = dw_pcie_ep_inbound_atu(ep, bar, epf_bar->phys_addr, as_type);
> +	ret = dw_pcie_ep_inbound_atu(ep, func_no, bar,
> +				     epf_bar->phys_addr, as_type);
>  	if (ret)
>  		return ret;
>  
> @@ -202,7 +231,7 @@ static int dw_pcie_ep_map_addr(struct pci_epc *epc, u8 func_no,
>  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  
> -	ret = dw_pcie_ep_outbound_atu(ep, addr, pci_addr, size);
> +	ret = dw_pcie_ep_outbound_atu(ep, func_no, addr, pci_addr, size);
>  	if (ret) {
>  		dev_err(pci->dev, "Failed to enable address\n");
>  		return ret;
> @@ -216,11 +245,14 @@ static int dw_pcie_ep_get_msi(struct pci_epc *epc, u8 func_no)
>  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  	u32 val, reg;
> +	unsigned int func_offset = 0;
>  
>  	if (!ep->msi_cap)
>  		return -EINVAL;
>  
> -	reg = ep->msi_cap + PCI_MSI_FLAGS;
> +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> +
> +	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
>  	val = dw_pcie_readw_dbi(pci, reg);
>  	if (!(val & PCI_MSI_FLAGS_ENABLE))
>  		return -EINVAL;
> @@ -235,11 +267,14 @@ static int dw_pcie_ep_set_msi(struct pci_epc *epc, u8 func_no, u8 interrupts)
>  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  	u32 val, reg;
> +	unsigned int func_offset = 0;
>  
>  	if (!ep->msi_cap)
>  		return -EINVAL;
>  
> -	reg = ep->msi_cap + PCI_MSI_FLAGS;
> +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> +
> +	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
>  	val = dw_pcie_readw_dbi(pci, reg);
>  	val &= ~PCI_MSI_FLAGS_QMASK;
>  	val |= (interrupts << 1) & PCI_MSI_FLAGS_QMASK;
> @@ -255,11 +290,14 @@ static int dw_pcie_ep_get_msix(struct pci_epc *epc, u8 func_no)
>  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  	u32 val, reg;
> +	unsigned int func_offset = 0;
>  
>  	if (!ep->msix_cap)
>  		return -EINVAL;
>  
> -	reg = ep->msix_cap + PCI_MSIX_FLAGS;
> +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> +
> +	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
>  	val = dw_pcie_readw_dbi(pci, reg);
>  	if (!(val & PCI_MSIX_FLAGS_ENABLE))
>  		return -EINVAL;
> @@ -274,11 +312,14 @@ static int dw_pcie_ep_set_msix(struct pci_epc *epc, u8 func_no, u16 interrupts)
>  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  	u32 val, reg;
> +	unsigned int func_offset = 0;
>  
>  	if (!ep->msix_cap)
>  		return -EINVAL;
>  
> -	reg = ep->msix_cap + PCI_MSIX_FLAGS;
> +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> +
> +	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
>  	val = dw_pcie_readw_dbi(pci, reg);
>  	val &= ~PCI_MSIX_FLAGS_QSIZE;
>  	val |= interrupts;
> @@ -365,6 +406,7 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  	struct pci_epc *epc = ep->epc;
>  	unsigned int aligned_offset;
> +	unsigned int func_offset = 0;
>  	u16 msg_ctrl, msg_data;
>  	u32 msg_addr_lower, msg_addr_upper, reg;
>  	u64 msg_addr;
> @@ -374,20 +416,22 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	if (!ep->msi_cap)
>  		return -EINVAL;
>  
> +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> +
>  	/* Raise MSI per the PCI Local Bus Specification Revision 3.0, 6.8.1. */
> -	reg = ep->msi_cap + PCI_MSI_FLAGS;
> +	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
>  	msg_ctrl = dw_pcie_readw_dbi(pci, reg);
>  	has_upper = !!(msg_ctrl & PCI_MSI_FLAGS_64BIT);
> -	reg = ep->msi_cap + PCI_MSI_ADDRESS_LO;
> +	reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_LO;
>  	msg_addr_lower = dw_pcie_readl_dbi(pci, reg);
>  	if (has_upper) {
> -		reg = ep->msi_cap + PCI_MSI_ADDRESS_HI;
> +		reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_HI;
>  		msg_addr_upper = dw_pcie_readl_dbi(pci, reg);
> -		reg = ep->msi_cap + PCI_MSI_DATA_64;
> +		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_64;
>  		msg_data = dw_pcie_readw_dbi(pci, reg);
>  	} else {
>  		msg_addr_upper = 0;
> -		reg = ep->msi_cap + PCI_MSI_DATA_32;
> +		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_32;
>  		msg_data = dw_pcie_readw_dbi(pci, reg);
>  	}
>  	aligned_offset = msg_addr_lower & (epc->mem->page_size - 1);
> @@ -406,11 +450,12 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
>  }
>  
>  int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
> -			     u16 interrupt_num)
> +			      u16 interrupt_num)
>  {
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  	struct pci_epc *epc = ep->epc;
>  	u16 tbl_offset, bir;
> +	unsigned int func_offset = 0;
>  	u32 bar_addr_upper, bar_addr_lower;
>  	u32 msg_addr_upper, msg_addr_lower;
>  	u32 reg, msg_data, vec_ctrl;
> @@ -418,12 +463,14 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	void __iomem *msix_tbl;
>  	int ret;
>  
> -	reg = ep->msix_cap + PCI_MSIX_TABLE;
> +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> +
> +	reg = ep->msix_cap + func_offset + PCI_MSIX_TABLE;
>  	tbl_offset = dw_pcie_readl_dbi(pci, reg);
>  	bir = (tbl_offset & PCI_MSIX_TABLE_BIR);
>  	tbl_offset &= PCI_MSIX_TABLE_OFFSET;
>  
> -	reg = PCI_BASE_ADDRESS_0 + (4 * bir);
> +	reg = PCI_BASE_ADDRESS_0 + func_offset + (4 * bir);
>  	bar_addr_upper = 0;
>  	bar_addr_lower = dw_pcie_readl_dbi(pci, reg);
>  	reg_u64 = (bar_addr_lower & PCI_BASE_ADDRESS_MEM_TYPE_MASK);
> @@ -559,13 +606,13 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
>  	ep->epc = epc;
>  	epc_set_drvdata(epc, ep);
>  
> -	if (ep->ops->ep_init)
> -		ep->ops->ep_init(ep);
> -
>  	ret = of_property_read_u8(np, "max-functions", &epc->max_functions);
>  	if (ret < 0)
>  		epc->max_functions = 1;
>  
> +	if (ep->ops->ep_init)
> +		ep->ops->ep_init(ep);
> +
>  	ret = __pci_epc_mem_init(epc, ep->phys_base, ep->addr_size,
>  				 ep->page_size);
>  	if (ret < 0) {
> diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
> index 143cb6c..ede2e75 100644
> --- a/drivers/pci/controller/dwc/pcie-designware.c
> +++ b/drivers/pci/controller/dwc/pcie-designware.c
> @@ -238,9 +238,10 @@ static void dw_pcie_writel_ob_unroll(struct dw_pcie *pci, u32 index, u32 reg,
>  	dw_pcie_writel_atu(pci, offset + reg, val);
>  }
>  
> -static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
> -					     int type, u64 cpu_addr,
> -					     u64 pci_addr, u32 size)
> +static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, u8 func_no,
> +					     int index, int type,
> +					     u64 cpu_addr, u64 pci_addr,
> +					     u32 size)
>  {
>  	u32 retries, val;
>  
> @@ -255,7 +256,7 @@ static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
>  	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_UPPER_TARGET,
>  				 upper_32_bits(pci_addr));
>  	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1,
> -				 type);
> +				 type | PCIE_ATU_FUNC_NUM(func_no));
>  	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL2,
>  				 PCIE_ATU_ENABLE);
>  
> @@ -274,8 +275,9 @@ static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
>  	dev_err(pci->dev, "Outbound iATU is not being enabled\n");
>  }
>  
> -void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
> -			       u64 cpu_addr, u64 pci_addr, u32 size)
> +static void __dw_pcie_prog_outbound_atu(struct dw_pcie *pci, u8 func_no,
> +					int index, int type, u64 cpu_addr,
> +					u64 pci_addr, u32 size)
>  {
>  	u32 retries, val;
>  
> @@ -283,8 +285,8 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
>  		cpu_addr = pci->ops->cpu_addr_fixup(pci, cpu_addr);
>  
>  	if (pci->iatu_unroll_enabled) {
> -		dw_pcie_prog_outbound_atu_unroll(pci, index, type, cpu_addr,
> -						 pci_addr, size);
> +		dw_pcie_prog_outbound_atu_unroll(pci, func_no, index, type,
> +						 cpu_addr, pci_addr, size);
>  		return;
>  	}
>  
> @@ -300,7 +302,8 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
>  			   lower_32_bits(pci_addr));
>  	dw_pcie_writel_dbi(pci, PCIE_ATU_UPPER_TARGET,
>  			   upper_32_bits(pci_addr));
> -	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type);
> +	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type |
> +			   PCIE_ATU_FUNC_NUM(func_no));
>  	dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE);
>  
>  	/*
> @@ -317,6 +320,21 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
>  	dev_err(pci->dev, "Outbound iATU is not being enabled\n");
>  }
>  
> +void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
> +			       u64 cpu_addr, u64 pci_addr, u32 size)
> +{
> +	__dw_pcie_prog_outbound_atu(pci, 0, index, type,
> +				    cpu_addr, pci_addr, size);
> +}
> +
> +void dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> +				  int type, u64 cpu_addr, u64 pci_addr,
> +				  u32 size)
> +{
> +	__dw_pcie_prog_outbound_atu(pci, func_no, index, type,
> +				    cpu_addr, pci_addr, size);
> +}
> +
>  static u32 dw_pcie_readl_ib_unroll(struct dw_pcie *pci, u32 index, u32 reg)
>  {
>  	u32 offset = PCIE_GET_ATU_INB_UNR_REG_OFFSET(index);
> @@ -332,8 +350,8 @@ static void dw_pcie_writel_ib_unroll(struct dw_pcie *pci, u32 index, u32 reg,
>  	dw_pcie_writel_atu(pci, offset + reg, val);
>  }
>  
> -static int dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, int index,
> -					   int bar, u64 cpu_addr,
> +static int dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, u8 func_no,
> +					   int index, int bar, u64 cpu_addr,
>  					   enum dw_pcie_as_type as_type)
>  {
>  	int type;
> @@ -355,8 +373,10 @@ static int dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, int index,
>  		return -EINVAL;
>  	}
>  
> -	dw_pcie_writel_ib_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1, type);
> +	dw_pcie_writel_ib_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1, type |
> +				 PCIE_ATU_FUNC_NUM(func_no));
>  	dw_pcie_writel_ib_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL2,
> +				 PCIE_ATU_FUNC_NUM_MATCH_EN |
>  				 PCIE_ATU_ENABLE |
>  				 PCIE_ATU_BAR_MODE_ENABLE | (bar << 8));
>  
> @@ -377,14 +397,15 @@ static int dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, int index,
>  	return -EBUSY;
>  }
>  
> -int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int bar,
> -			     u64 cpu_addr, enum dw_pcie_as_type as_type)
> +int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> +			     int bar, u64 cpu_addr,
> +			     enum dw_pcie_as_type as_type)
>  {
>  	int type;
>  	u32 retries, val;
>  
>  	if (pci->iatu_unroll_enabled)
> -		return dw_pcie_prog_inbound_atu_unroll(pci, index, bar,
> +		return dw_pcie_prog_inbound_atu_unroll(pci, func_no, index, bar,
>  						       cpu_addr, as_type);
>  
>  	dw_pcie_writel_dbi(pci, PCIE_ATU_VIEWPORT, PCIE_ATU_REGION_INBOUND |
> @@ -403,9 +424,11 @@ int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int bar,
>  		return -EINVAL;
>  	}
>  
> -	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type);
> -	dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE
> -			   | PCIE_ATU_BAR_MODE_ENABLE | (bar << 8));
> +	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type |
> +			   PCIE_ATU_FUNC_NUM(func_no));
> +	dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE |
> +			   PCIE_ATU_FUNC_NUM_MATCH_EN |
> +			   PCIE_ATU_BAR_MODE_ENABLE | (bar << 8));
>  
>  	/*
>  	 * Make sure ATU enable takes effect before any subsequent config
> diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
> index 5a18e94..6aca0bb 100644
> --- a/drivers/pci/controller/dwc/pcie-designware.h
> +++ b/drivers/pci/controller/dwc/pcie-designware.h
> @@ -71,9 +71,11 @@
>  #define PCIE_ATU_TYPE_IO		0x2
>  #define PCIE_ATU_TYPE_CFG0		0x4
>  #define PCIE_ATU_TYPE_CFG1		0x5
> +#define PCIE_ATU_FUNC_NUM(pf)           ((pf) << 20)
>  #define PCIE_ATU_CR2			0x908
>  #define PCIE_ATU_ENABLE			BIT(31)
>  #define PCIE_ATU_BAR_MODE_ENABLE	BIT(30)
> +#define PCIE_ATU_FUNC_NUM_MATCH_EN      BIT(19)
>  #define PCIE_ATU_LOWER_BASE		0x90C
>  #define PCIE_ATU_UPPER_BASE		0x910
>  #define PCIE_ATU_LIMIT			0x914
> @@ -206,6 +208,14 @@ struct dw_pcie_ep_ops {
>  	int	(*raise_irq)(struct dw_pcie_ep *ep, u8 func_no,
>  			     enum pci_epc_irq_type type, u16 interrupt_num);
>  	const struct pci_epc_features* (*get_features)(struct dw_pcie_ep *ep);
> +	/*
> +	 * Provide a method to implement the different func config space
> +	 * access for different platform, if different func have different
> +	 * offset, return the offset of func. if use write a register way
> +	 * return a 0, and implement code in callback function of platform
> +	 * driver.
> +	 */
> +	unsigned int (*func_conf_select)(struct dw_pcie_ep *ep, u8 func_no);
>  };
>  
>  struct dw_pcie_ep {
> @@ -277,8 +287,12 @@ int dw_pcie_wait_for_link(struct dw_pcie *pci);
>  void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index,
>  			       int type, u64 cpu_addr, u64 pci_addr,
>  			       u32 size);
> -int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int bar,
> -			     u64 cpu_addr, enum dw_pcie_as_type as_type);
> +void dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> +				  int type, u64 cpu_addr, u64 pci_addr,
> +				  u32 size);
> +int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> +			     int bar, u64 cpu_addr,
> +			     enum dw_pcie_as_type as_type);
>  void dw_pcie_disable_atu(struct dw_pcie *pci, int index,
>  			 enum dw_pcie_region_type type);
>  void dw_pcie_setup(struct dw_pcie *pci);
> -- 
> 2.9.5
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 05/11] dt-bindings: pci: layerscape-pci: add compatible strings for ls1088a and ls2088a
  2019-09-02 12:31   ` Andrew Murray
@ 2019-09-03  1:33     ` Xiaowei Bao
  0 siblings, 0 replies; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-03  1:33 UTC (permalink / raw)
  To: Andrew Murray
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, M.h. Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, Leo Li, shawnguo, Mingkai Hu



> -----Original Message-----
> From: Andrew Murray <andrew.murray@arm.com>
> Sent: 2019年9月2日 20:32
> To: Xiaowei Bao <xiaowei.bao@nxp.com>
> Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> <zhiqiang.hou@nxp.com>
> Subject: Re: [PATCH v3 05/11] dt-bindings: pci: layerscape-pci: add compatible
> strings for ls1088a and ls2088a
> 
> On Mon, Sep 02, 2019 at 11:17:10AM +0800, Xiaowei Bao wrote:
> > Add compatible strings for ls1088a and ls2088a.
> >
> > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > ---
> > v2:
> >  - No change.
> > v3:
> >  - Use one valid combination of compatible strings.
> >
> >  Documentation/devicetree/bindings/pci/layerscape-pci.txt | 4 +++-
> >  1 file changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/Documentation/devicetree/bindings/pci/layerscape-pci.txt
> > b/Documentation/devicetree/bindings/pci/layerscape-pci.txt
> > index e20ceaa..762ae41 100644
> > --- a/Documentation/devicetree/bindings/pci/layerscape-pci.txt
> > +++ b/Documentation/devicetree/bindings/pci/layerscape-pci.txt
> > @@ -22,7 +22,9 @@ Required properties:
> >          "fsl,ls1043a-pcie"
> >          "fsl,ls1012a-pcie"
> >    EP mode:
> > -	"fsl,ls1046a-pcie-ep", "fsl,ls-pcie-ep"
> > +	"fsl,ls1046a-pcie-ep" "fsl,ls-pcie-ep"
> > +	"fsl,ls1088a-pcie-ep" "fsl,ls-pcie-ep"
> > +	"fsl,ls2088a-pcie-ep" "fsl,ls-pcie-ep"
> 
> This isn't consistent with "[PATCH v3 09/11] PCI: layerscape: Add EP mode..."
> as that patch drops the fallback "fsl,ls-pcie-ep". Either the fallback must be
> preserved in the driver, or you need to drop it here.
> 
> What if there are existing users that depend on the fallback?
> 
> (I'm also not sure if that comma should have been dropped).

Hi Andrew,

Thanks for your comments, I lose the comma.

Thanks 
Xiaowei

> 
> Thanks,
> 
> Andrew Murray
> 
> >  - reg: base addresses and lengths of the PCIe controller register blocks.
> >  - interrupts: A list of interrupt outputs of the controller. Must contain an
> >    entry for each entry in the interrupt-names property.
> > --
> > 2.9.5
> >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 09/11] PCI: layerscape: Add EP mode support for ls1088a and ls2088a
  2019-09-02 12:46   ` Andrew Murray
@ 2019-09-03  1:47     ` Xiaowei Bao
  2019-09-12 12:49       ` Andrew Murray
  0 siblings, 1 reply; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-03  1:47 UTC (permalink / raw)
  To: Andrew Murray
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, M.h. Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, Leo Li, shawnguo, Mingkai Hu



> -----Original Message-----
> From: Andrew Murray <andrew.murray@arm.com>
> Sent: 2019年9月2日 20:46
> To: Xiaowei Bao <xiaowei.bao@nxp.com>
> Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> <zhiqiang.hou@nxp.com>
> Subject: Re: [PATCH v3 09/11] PCI: layerscape: Add EP mode support for
> ls1088a and ls2088a
> 
> On Mon, Sep 02, 2019 at 11:17:14AM +0800, Xiaowei Bao wrote:
> > Add PCIe EP mode support for ls1088a and ls2088a, there are some
> > difference between LS1 and LS2 platform, so refactor the code of the
> > EP driver.
> >
> > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > ---
> > v2:
> >  - This is a new patch for supporting the ls1088a and ls2088a platform.
> > v3:
> >  - Adjust the some struct assignment order in probe function.
> >
> >  drivers/pci/controller/dwc/pci-layerscape-ep.c | 72
> > +++++++++++++++++++-------
> >  1 file changed, 53 insertions(+), 19 deletions(-)
> >
> > diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > index 5f0cb99..723bbe5 100644
> > --- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > +++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > @@ -20,27 +20,29 @@
> >
> >  #define PCIE_DBI2_OFFSET		0x1000	/* DBI2 base address*/
> >
> > -struct ls_pcie_ep {
> > -	struct dw_pcie		*pci;
> > -	struct pci_epc_features	*ls_epc;
> > +#define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
> > +
> > +struct ls_pcie_ep_drvdata {
> > +	u32				func_offset;
> > +	const struct dw_pcie_ep_ops	*ops;
> > +	const struct dw_pcie_ops	*dw_pcie_ops;
> >  };
> >
> > -#define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
> > +struct ls_pcie_ep {
> > +	struct dw_pcie			*pci;
> > +	struct pci_epc_features		*ls_epc;
> > +	const struct ls_pcie_ep_drvdata *drvdata; };
> >
> >  static int ls_pcie_establish_link(struct dw_pcie *pci)  {
> >  	return 0;
> >  }
> >
> > -static const struct dw_pcie_ops ls_pcie_ep_ops = {
> > +static const struct dw_pcie_ops dw_ls_pcie_ep_ops = {
> >  	.start_link = ls_pcie_establish_link,  };
> >
> > -static const struct of_device_id ls_pcie_ep_of_match[] = {
> > -	{ .compatible = "fsl,ls-pcie-ep",},
> > -	{ },
> > -};
> > -
> >  static const struct pci_epc_features*  ls_pcie_ep_get_features(struct
> > dw_pcie_ep *ep)  { @@ -87,10 +89,39 @@ static int
> > ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> >  	}
> >  }
> >
> > -static const struct dw_pcie_ep_ops pcie_ep_ops = {
> > +static unsigned int ls_pcie_ep_func_conf_select(struct dw_pcie_ep *ep,
> > +						u8 func_no)
> > +{
> > +	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > +	struct ls_pcie_ep *pcie = to_ls_pcie_ep(pci);
> > +
> > +	WARN_ON(func_no && !pcie->drvdata->func_offset);
> > +	return pcie->drvdata->func_offset * func_no; }
> > +
> > +static const struct dw_pcie_ep_ops ls_pcie_ep_ops = {
> >  	.ep_init = ls_pcie_ep_init,
> >  	.raise_irq = ls_pcie_ep_raise_irq,
> >  	.get_features = ls_pcie_ep_get_features,
> > +	.func_conf_select = ls_pcie_ep_func_conf_select, };
> > +
> > +static const struct ls_pcie_ep_drvdata ls1_ep_drvdata = {
> > +	.ops = &ls_pcie_ep_ops,
> > +	.dw_pcie_ops = &dw_ls_pcie_ep_ops,
> > +};
> > +
> > +static const struct ls_pcie_ep_drvdata ls2_ep_drvdata = {
> > +	.func_offset = 0x20000,
> > +	.ops = &ls_pcie_ep_ops,
> > +	.dw_pcie_ops = &dw_ls_pcie_ep_ops,
> > +};
> > +
> > +static const struct of_device_id ls_pcie_ep_of_match[] = {
> > +	{ .compatible = "fsl,ls1046a-pcie-ep", .data = &ls1_ep_drvdata },
> > +	{ .compatible = "fsl,ls1088a-pcie-ep", .data = &ls2_ep_drvdata },
> > +	{ .compatible = "fsl,ls2088a-pcie-ep", .data = &ls2_ep_drvdata },
> > +	{ },
> 
> This removes support for "fsl,ls-pcie-ep" - was that intentional? If you do plan
> to drop it please make sure you explain why in the commit message. See also
> my comments in your dt-binding patch.

In fact, the u-boot will fixup the status property to 'status = enabled' in PCI node of 
the DTS base on "fsl,ls-pcie-ep" compatible, so "fsl,ls-pcie-ep" is used, I used this
compatible before, because the driver only support the LS1046a, but this time, I add
the LS1088a and LS2088a support, and these two boards have some difference form
LS1046a, so I changed the compatible. I am not sure whether need to add "fsl,ls-pcie-ep"
in there, could you give some advice, thanks a lot.

Thanks 
Xiaowei
 
> 
> Thanks,
> 
> Andrew Murray
> 
> >  };
> >
> >  static int __init ls_add_pcie_ep(struct ls_pcie_ep *pcie, @@ -103,7
> > +134,7 @@ static int __init ls_add_pcie_ep(struct ls_pcie_ep *pcie,
> >  	int ret;
> >
> >  	ep = &pci->ep;
> > -	ep->ops = &pcie_ep_ops;
> > +	ep->ops = pcie->drvdata->ops;
> >
> >  	res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> "addr_space");
> >  	if (!res)
> > @@ -142,20 +173,23 @@ static int __init ls_pcie_ep_probe(struct
> platform_device *pdev)
> >  	if (!ls_epc)
> >  		return -ENOMEM;
> >
> > -	dbi_base = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> "regs");
> > -	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_base);
> > -	if (IS_ERR(pci->dbi_base))
> > -		return PTR_ERR(pci->dbi_base);
> > +	pcie->drvdata = of_device_get_match_data(dev);
> >
> > -	pci->dbi_base2 = pci->dbi_base + PCIE_DBI2_OFFSET;
> >  	pci->dev = dev;
> > -	pci->ops = &ls_pcie_ep_ops;
> > -	pcie->pci = pci;
> > +	pci->ops = pcie->drvdata->dw_pcie_ops;
> >
> >  	ls_epc->bar_fixed_64bit = (1 << BAR_2) | (1 << BAR_4),
> >
> > +	pcie->pci = pci;
> >  	pcie->ls_epc = ls_epc;
> >
> > +	dbi_base = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> "regs");
> > +	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_base);
> > +	if (IS_ERR(pci->dbi_base))
> > +		return PTR_ERR(pci->dbi_base);
> > +
> > +	pci->dbi_base2 = pci->dbi_base + PCIE_DBI2_OFFSET;
> > +
> >  	platform_set_drvdata(pdev, pcie);
> >
> >  	ret = ls_add_pcie_ep(pcie, pdev);
> > --
> > 2.9.5
> >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 11/11] misc: pci_endpoint_test: Add LS1088a in pci_device_id table
  2019-09-02 12:54   ` Andrew Murray
@ 2019-09-03  1:52     ` Xiaowei Bao
  2019-09-12 12:59       ` Andrew Murray
  0 siblings, 1 reply; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-03  1:52 UTC (permalink / raw)
  To: Andrew Murray
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, M.h. Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, Leo Li, shawnguo, Mingkai Hu



> -----Original Message-----
> From: Andrew Murray <andrew.murray@arm.com>
> Sent: 2019年9月2日 20:55
> To: Xiaowei Bao <xiaowei.bao@nxp.com>
> Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> <zhiqiang.hou@nxp.com>
> Subject: Re: [PATCH v3 11/11] misc: pci_endpoint_test: Add LS1088a in
> pci_device_id table
> 
> On Mon, Sep 02, 2019 at 11:17:16AM +0800, Xiaowei Bao wrote:
> > Add LS1088a in pci_device_id table so that pci-epf-test can be used
> > for testing PCIe EP in LS1088a.
> >
> > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > ---
> > v2:
> >  - No change.
> > v3:
> >  - No change.
> >
> >  drivers/misc/pci_endpoint_test.c | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/drivers/misc/pci_endpoint_test.c
> > b/drivers/misc/pci_endpoint_test.c
> > index 6e208a0..d531951 100644
> > --- a/drivers/misc/pci_endpoint_test.c
> > +++ b/drivers/misc/pci_endpoint_test.c
> > @@ -793,6 +793,7 @@ static const struct pci_device_id
> pci_endpoint_test_tbl[] = {
> >  	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_DRA74x) },
> >  	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_DRA72x) },
> >  	{ PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, 0x81c0) },
> > +	{ PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, 0x80c0) },
> 
> The Freescale PCI devices are the only devices in this table that don't have a
> define for their device ID. I think a define should be created for both of the
> device IDs above.

OK, but I only define in this file, I am not sure this can define in include/linux/pci_ids.h
file 

Thanks 
Xiaowei

> 
> Thanks,
> 
> Andrew Murray
> 
> >  	{ PCI_DEVICE_DATA(SYNOPSYS, EDDA, NULL) },
> >  	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_AM654),
> >  	  .driver_data = (kernel_ulong_t)&am654_data
> > --
> > 2.9.5
> >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 10/11] arm64: dts: layerscape: Add PCIe EP node for ls1088a
  2019-09-02 13:06   ` Andrew Murray
@ 2019-09-03  2:01     ` Xiaowei Bao
  2019-09-12 13:01       ` Andrew Murray
  0 siblings, 1 reply; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-03  2:01 UTC (permalink / raw)
  To: Andrew Murray
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, M.h. Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, Leo Li, shawnguo, Mingkai Hu



> -----Original Message-----
> From: Andrew Murray <andrew.murray@arm.com>
> Sent: 2019年9月2日 21:06
> To: Xiaowei Bao <xiaowei.bao@nxp.com>
> Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> <zhiqiang.hou@nxp.com>
> Subject: Re: [PATCH v3 10/11] arm64: dts: layerscape: Add PCIe EP node for
> ls1088a
> 
> On Mon, Sep 02, 2019 at 11:17:15AM +0800, Xiaowei Bao wrote:
> > Add PCIe EP node for ls1088a to support EP mode.
> >
> > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > ---
> > v2:
> >  - Remove the pf-offset proparty.
> > v3:
> >  - No change.
> >
> >  arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi | 31
> ++++++++++++++++++++++++++
> >  1 file changed, 31 insertions(+)
> >
> > diff --git a/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
> b/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
> > index c676d07..da246ab 100644
> > --- a/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
> > +++ b/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
> > @@ -483,6 +483,17 @@
> >  			status = "disabled";
> >  		};
> >
> > +		pcie_ep@3400000 {
> > +			compatible = "fsl,ls1088a-pcie-ep","fsl,ls-pcie-ep";
> 
> Here you specify a fallback "fsl,ls-pcie-ep" that is removed by this series.
> 
> Besides that, this looks OK.

As explained, the "fsl,ls-pcie-ep" is needed, due to the u-boot will fixup the status
property base on this compatible, I think we reserve this compatible is helpfully,
if delate this compatible, I have to modify the code of bootloader.

Thanks 
XIaowei

> 
> Thanks,
> 
> Andrew Murray
> 
> > +			reg = <0x00 0x03400000 0x0 0x00100000
> > +			       0x20 0x00000000 0x8 0x00000000>;
> > +			reg-names = "regs", "addr_space";
> > +			num-ib-windows = <24>;
> > +			num-ob-windows = <128>;
> > +			max-functions = /bits/ 8 <2>;
> > +			status = "disabled";
> > +		};
> > +
> >  		pcie@3500000 {
> >  			compatible = "fsl,ls1088a-pcie";
> >  			reg = <0x00 0x03500000 0x0 0x00100000   /* controller
> registers */
> > @@ -508,6 +519,16 @@
> >  			status = "disabled";
> >  		};
> >
> > +		pcie_ep@3500000 {
> > +			compatible = "fsl,ls1088a-pcie-ep","fsl,ls-pcie-ep";
> > +			reg = <0x00 0x03500000 0x0 0x00100000
> > +			       0x28 0x00000000 0x8 0x00000000>;
> > +			reg-names = "regs", "addr_space";
> > +			num-ib-windows = <6>;
> > +			num-ob-windows = <8>;
> > +			status = "disabled";
> > +		};
> > +
> >  		pcie@3600000 {
> >  			compatible = "fsl,ls1088a-pcie";
> >  			reg = <0x00 0x03600000 0x0 0x00100000   /* controller
> registers */
> > @@ -533,6 +554,16 @@
> >  			status = "disabled";
> >  		};
> >
> > +		pcie_ep@3600000 {
> > +			compatible = "fsl,ls1088a-pcie-ep","fsl,ls-pcie-ep";
> > +			reg = <0x00 0x03600000 0x0 0x00100000
> > +			       0x30 0x00000000 0x8 0x00000000>;
> > +			reg-names = "regs", "addr_space";
> > +			num-ib-windows = <6>;
> > +			num-ob-windows = <8>;
> > +			status = "disabled";
> > +		};
> > +
> >  		smmu: iommu@5000000 {
> >  			compatible = "arm,mmu-500";
> >  			reg = <0 0x5000000 0 0x800000>;
> > --
> > 2.9.5
> >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 07/11] PCI: layerscape: Modify the way of getting capability with different PEX
  2019-09-02 13:37   ` Andrew Murray
@ 2019-09-03  2:13     ` Xiaowei Bao
  0 siblings, 0 replies; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-03  2:13 UTC (permalink / raw)
  To: Andrew Murray
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, M.h. Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, Leo Li, shawnguo, Mingkai Hu



> -----Original Message-----
> From: Andrew Murray <andrew.murray@arm.com>
> Sent: 2019年9月2日 21:38
> To: Xiaowei Bao <xiaowei.bao@nxp.com>
> Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> <zhiqiang.hou@nxp.com>
> Subject: Re: [PATCH v3 07/11] PCI: layerscape: Modify the way of getting
> capability with different PEX
> 
> On Mon, Sep 02, 2019 at 11:17:12AM +0800, Xiaowei Bao wrote:
> > The different PCIe controller in one board may be have different
> > capability of MSI or MSIX, so change the way of getting the MSI
> > capability, make it more flexible.
> >
> > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> 
> Please see the comments I just made to Kishon's feedback in the thread for
> this patch in series v2.

I have reply the comments in series v2, expect Kishon's feedback.

Thanks
Xiaowei

> 
> Thanks,
> 
> Andrew Murray
> 
> > ---
> > v2:
> >  - Remove the repeated assignment code.
> > v3:
> >  - Use ep_func msi_cap and msix_cap to decide the msi_capable and
> >    msix_capable of pci_epc_features struct.
> >
> >  drivers/pci/controller/dwc/pci-layerscape-ep.c | 31
> > +++++++++++++++++++-------
> >  1 file changed, 23 insertions(+), 8 deletions(-)
> >
> > diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > index a9c552e..1e07287 100644
> > --- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > +++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > @@ -22,6 +22,7 @@
> >
> >  struct ls_pcie_ep {
> >  	struct dw_pcie		*pci;
> > +	struct pci_epc_features	*ls_epc;
> >  };
> >
> >  #define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
> > @@ -40,26 +41,31 @@ static const struct of_device_id
> ls_pcie_ep_of_match[] = {
> >  	{ },
> >  };
> >
> > -static const struct pci_epc_features ls_pcie_epc_features = {
> > -	.linkup_notifier = false,
> > -	.msi_capable = true,
> > -	.msix_capable = false,
> > -	.bar_fixed_64bit = (1 << BAR_2) | (1 << BAR_4),
> > -};
> > -
> >  static const struct pci_epc_features*  ls_pcie_ep_get_features(struct
> > dw_pcie_ep *ep)  {
> > -	return &ls_pcie_epc_features;
> > +	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > +	struct ls_pcie_ep *pcie = to_ls_pcie_ep(pci);
> > +
> > +	return pcie->ls_epc;
> >  }
> >
> >  static void ls_pcie_ep_init(struct dw_pcie_ep *ep)  {
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > +	struct ls_pcie_ep *pcie = to_ls_pcie_ep(pci);
> > +	struct dw_pcie_ep_func *ep_func;
> >  	enum pci_barno bar;
> >
> > +	ep_func = dw_pcie_ep_get_func_from_ep(ep, 0);
> > +	if (!ep_func)
> > +		return;
> > +
> >  	for (bar = BAR_0; bar <= BAR_5; bar++)
> >  		dw_pcie_ep_reset_bar(pci, bar);
> > +
> > +	pcie->ls_epc->msi_capable = ep_func->msi_cap ? true : false;
> > +	pcie->ls_epc->msix_capable = ep_func->msix_cap ? true : false;
> >  }
> >
> >  static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no, @@
> > -119,6 +125,7 @@ static int __init ls_pcie_ep_probe(struct platform_device
> *pdev)
> >  	struct device *dev = &pdev->dev;
> >  	struct dw_pcie *pci;
> >  	struct ls_pcie_ep *pcie;
> > +	struct pci_epc_features *ls_epc;
> >  	struct resource *dbi_base;
> >  	int ret;
> >
> > @@ -130,6 +137,10 @@ static int __init ls_pcie_ep_probe(struct
> platform_device *pdev)
> >  	if (!pci)
> >  		return -ENOMEM;
> >
> > +	ls_epc = devm_kzalloc(dev, sizeof(*ls_epc), GFP_KERNEL);
> > +	if (!ls_epc)
> > +		return -ENOMEM;
> > +
> >  	dbi_base = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> "regs");
> >  	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_base);
> >  	if (IS_ERR(pci->dbi_base))
> > @@ -140,6 +151,10 @@ static int __init ls_pcie_ep_probe(struct
> platform_device *pdev)
> >  	pci->ops = &ls_pcie_ep_ops;
> >  	pcie->pci = pci;
> >
> > +	ls_epc->bar_fixed_64bit = (1 << BAR_2) | (1 << BAR_4),
> > +
> > +	pcie->ls_epc = ls_epc;
> > +
> >  	platform_set_drvdata(pdev, pcie);
> >
> >  	ret = ls_add_pcie_ep(pcie, pdev);
> > --
> > 2.9.5
> >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 04/11] PCI: designware-ep: Modify MSI and MSIX CAP way of finding
  2019-09-02 15:07   ` Andrew Murray
@ 2019-09-03  2:33     ` Xiaowei Bao
  0 siblings, 0 replies; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-03  2:33 UTC (permalink / raw)
  To: Andrew Murray
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, M.h. Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, Leo Li, shawnguo, Mingkai Hu



> -----Original Message-----
> From: Andrew Murray <andrew.murray@arm.com>
> Sent: 2019年9月2日 23:07
> To: Xiaowei Bao <xiaowei.bao@nxp.com>
> Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> <zhiqiang.hou@nxp.com>
> Subject: Re: [PATCH v3 04/11] PCI: designware-ep: Modify MSI and MSIX CAP
> way of finding
> 
> On Mon, Sep 02, 2019 at 11:17:09AM +0800, Xiaowei Bao wrote:
> > Each PF of EP device should have it's own MSI or MSIX capabitily
> > struct, so create a dw_pcie_ep_func struct and remover the msi_cap
> 
> remover?

Sorry. ^_^


> 
> > and msix_cap to this struce, and manage the PFs with a list.
> 
> struce?
> 
> >
> > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > ---
> > v1:
> >  - This is a new patch, to fix the issue of MSI and MSIX CAP way of
> >    finding.
> > v2:
> >  - No change.
> > v3:
> >  - No change.
> 
> This makes it look like you introduced the patch in v1 and haven't changed it
> since.
> 
> I think it's more common to have a history like this:
> 
> ---
> v3:
>  - Introduced new patch, to fix the issue of MSI and MSIX CAP way of
>    finding.

OK, thanks, I am not clear the rules, thanks a lot for your help.

Thanks 
Xiaowei

> 
> 
> >
> >  drivers/pci/controller/dwc/pcie-designware-ep.c | 135
> +++++++++++++++++++++---
> >  drivers/pci/controller/dwc/pcie-designware.h    |  18 +++-
> >  2 files changed, 134 insertions(+), 19 deletions(-)
> >
> > diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c
> > b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > index c3bc7bd..144eb12 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > @@ -19,6 +19,19 @@ void dw_pcie_ep_linkup(struct dw_pcie_ep *ep)
> >  	pci_epc_linkup(epc);
> >  }
> >
> > +struct dw_pcie_ep_func *
> > +dw_pcie_ep_get_func_from_ep(struct dw_pcie_ep *ep, u8 func_no) {
> > +	struct dw_pcie_ep_func *ep_func;
> > +
> > +	list_for_each_entry(ep_func, &ep->func_list, list) {
> > +		if (ep_func->func_no == func_no)
> > +			return ep_func;
> > +	}
> > +
> > +	return NULL;
> > +}
> > +
> >  static unsigned int dw_pcie_ep_func_select(struct dw_pcie_ep *ep, u8
> > func_no)  {
> >  	unsigned int func_offset = 0;
> > @@ -59,6 +72,47 @@ void dw_pcie_ep_reset_bar(struct dw_pcie *pci,
> enum pci_barno bar)
> >  		__dw_pcie_ep_reset_bar(pci, func_no, bar, 0);  }
> >
> > +static u8 __dw_pcie_ep_find_next_cap(struct dw_pcie_ep *ep, u8
> func_no,
> > +		u8 cap_ptr, u8 cap)
> > +{
> > +	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > +	unsigned int func_offset = 0;
> > +	u8 cap_id, next_cap_ptr;
> > +	u16 reg;
> > +
> > +	if (!cap_ptr)
> > +		return 0;
> > +
> > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > +
> > +	reg = dw_pcie_readw_dbi(pci, func_offset + cap_ptr);
> > +	cap_id = (reg & 0x00ff);
> > +
> > +	if (cap_id > PCI_CAP_ID_MAX)
> > +		return 0;
> > +
> > +	if (cap_id == cap)
> > +		return cap_ptr;
> > +
> > +	next_cap_ptr = (reg & 0xff00) >> 8;
> > +	return __dw_pcie_ep_find_next_cap(ep, func_no, next_cap_ptr, cap); }
> 
> Which tree have you based this patchset on? v5.3-rc3 and pci/dwc both
> already have this function (without the func_no). See beb4641a787d
> ("PCI: dwc: Add MSI-X callbacks handler").

There is a commit 7a6854f68 in the latest kernel.

Thanks 
Xiaowei

> 
> > +
> > +static u8 dw_pcie_ep_find_capability(struct dw_pcie_ep *ep, u8
> > +func_no, u8 cap) {
> > +	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > +	unsigned int func_offset = 0;
> > +	u8 next_cap_ptr;
> > +	u16 reg;
> > +
> > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > +
> > +	reg = dw_pcie_readw_dbi(pci, func_offset + PCI_CAPABILITY_LIST);
> > +	next_cap_ptr = (reg & 0x00ff);
> > +
> > +	return __dw_pcie_ep_find_next_cap(ep, func_no, next_cap_ptr, cap); }
> > +
> >  static int dw_pcie_ep_write_header(struct pci_epc *epc, u8 func_no,
> >  				   struct pci_epf_header *hdr)
> >  {
> > @@ -246,13 +300,18 @@ static int dw_pcie_ep_get_msi(struct pci_epc
> *epc, u8 func_no)
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >  	u32 val, reg;
> >  	unsigned int func_offset = 0;
> > +	struct dw_pcie_ep_func *ep_func;
> >
> > -	if (!ep->msi_cap)
> > +	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
> > +	if (!ep_func)
> > +		return -EINVAL;
> > +
> > +	if (!ep_func->msi_cap)
> >  		return -EINVAL;
> >
> >  	func_offset = dw_pcie_ep_func_select(ep, func_no);
> >
> > -	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
> > +	reg = ep_func->msi_cap + func_offset + PCI_MSI_FLAGS;
> >  	val = dw_pcie_readw_dbi(pci, reg);
> >  	if (!(val & PCI_MSI_FLAGS_ENABLE))
> >  		return -EINVAL;
> > @@ -268,13 +327,18 @@ static int dw_pcie_ep_set_msi(struct pci_epc
> *epc, u8 func_no, u8 interrupts)
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >  	u32 val, reg;
> >  	unsigned int func_offset = 0;
> > +	struct dw_pcie_ep_func *ep_func;
> > +
> > +	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
> > +	if (!ep_func)
> > +		return -EINVAL;
> >
> > -	if (!ep->msi_cap)
> > +	if (!ep_func->msi_cap)
> >  		return -EINVAL;
> >
> >  	func_offset = dw_pcie_ep_func_select(ep, func_no);
> >
> > -	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
> > +	reg = ep_func->msi_cap + func_offset + PCI_MSI_FLAGS;
> >  	val = dw_pcie_readw_dbi(pci, reg);
> >  	val &= ~PCI_MSI_FLAGS_QMASK;
> >  	val |= (interrupts << 1) & PCI_MSI_FLAGS_QMASK; @@ -291,13 +355,18
> > @@ static int dw_pcie_ep_get_msix(struct pci_epc *epc, u8 func_no)
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >  	u32 val, reg;
> >  	unsigned int func_offset = 0;
> > +	struct dw_pcie_ep_func *ep_func;
> > +
> > +	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
> > +	if (!ep_func)
> > +		return -EINVAL;
> >
> > -	if (!ep->msix_cap)
> > +	if (!ep_func->msix_cap)
> >  		return -EINVAL;
> >
> >  	func_offset = dw_pcie_ep_func_select(ep, func_no);
> >
> > -	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
> > +	reg = ep_func->msix_cap + func_offset + PCI_MSIX_FLAGS;
> >  	val = dw_pcie_readw_dbi(pci, reg);
> >  	if (!(val & PCI_MSIX_FLAGS_ENABLE))
> >  		return -EINVAL;
> > @@ -313,13 +382,18 @@ static int dw_pcie_ep_set_msix(struct pci_epc
> *epc, u8 func_no, u16 interrupts)
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >  	u32 val, reg;
> >  	unsigned int func_offset = 0;
> > +	struct dw_pcie_ep_func *ep_func;
> >
> > -	if (!ep->msix_cap)
> > +	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
> > +	if (!ep_func)
> > +		return -EINVAL;
> > +
> > +	if (!ep_func->msix_cap)
> >  		return -EINVAL;
> >
> >  	func_offset = dw_pcie_ep_func_select(ep, func_no);
> >
> > -	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
> > +	reg = ep_func->msix_cap + func_offset + PCI_MSIX_FLAGS;
> >  	val = dw_pcie_readw_dbi(pci, reg);
> >  	val &= ~PCI_MSIX_FLAGS_QSIZE;
> >  	val |= interrupts;
> > @@ -404,6 +478,7 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep
> *ep, u8 func_no,
> >  			     u8 interrupt_num)
> >  {
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > +	struct dw_pcie_ep_func *ep_func;
> >  	struct pci_epc *epc = ep->epc;
> >  	unsigned int aligned_offset;
> >  	unsigned int func_offset = 0;
> > @@ -413,25 +488,29 @@ int dw_pcie_ep_raise_msi_irq(struct
> dw_pcie_ep *ep, u8 func_no,
> >  	bool has_upper;
> >  	int ret;
> >
> > -	if (!ep->msi_cap)
> > +	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
> > +	if (!ep_func)
> > +		return -EINVAL;
> > +
> > +	if (!ep_func->msi_cap)
> >  		return -EINVAL;
> >
> >  	func_offset = dw_pcie_ep_func_select(ep, func_no);
> >
> >  	/* Raise MSI per the PCI Local Bus Specification Revision 3.0, 6.8.1. */
> > -	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
> > +	reg = ep_func->msi_cap + func_offset + PCI_MSI_FLAGS;
> >  	msg_ctrl = dw_pcie_readw_dbi(pci, reg);
> >  	has_upper = !!(msg_ctrl & PCI_MSI_FLAGS_64BIT);
> > -	reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_LO;
> > +	reg = ep_func->msi_cap + func_offset + PCI_MSI_ADDRESS_LO;
> >  	msg_addr_lower = dw_pcie_readl_dbi(pci, reg);
> >  	if (has_upper) {
> > -		reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_HI;
> > +		reg = ep_func->msi_cap + func_offset + PCI_MSI_ADDRESS_HI;
> >  		msg_addr_upper = dw_pcie_readl_dbi(pci, reg);
> > -		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_64;
> > +		reg = ep_func->msi_cap + func_offset + PCI_MSI_DATA_64;
> >  		msg_data = dw_pcie_readw_dbi(pci, reg);
> >  	} else {
> >  		msg_addr_upper = 0;
> > -		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_32;
> > +		reg = ep_func->msi_cap + func_offset + PCI_MSI_DATA_32;
> >  		msg_data = dw_pcie_readw_dbi(pci, reg);
> >  	}
> >  	aligned_offset = msg_addr_lower & (epc->mem->page_size - 1); @@
> > -467,6 +546,7 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep,
> u8 func_no,
> >  			      u16 interrupt_num)
> >  {
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > +	struct dw_pcie_ep_func *ep_func;
> >  	struct pci_epc *epc = ep->epc;
> >  	u16 tbl_offset, bir;
> >  	unsigned int func_offset = 0;
> > @@ -477,9 +557,16 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep
> *ep, u8 func_no,
> >  	void __iomem *msix_tbl;
> >  	int ret;
> >
> > +	ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no);
> > +	if (!ep_func)
> > +		return -EINVAL;
> > +
> > +	if (!ep_func->msix_cap)
> > +		return -EINVAL;
> > +
> >  	func_offset = dw_pcie_ep_func_select(ep, func_no);
> >
> > -	reg = ep->msix_cap + func_offset + PCI_MSIX_TABLE;
> > +	reg = ep_func->msix_cap + func_offset + PCI_MSIX_TABLE;
> >  	tbl_offset = dw_pcie_readl_dbi(pci, reg);
> >  	bir = (tbl_offset & PCI_MSIX_TABLE_BIR);
> >  	tbl_offset &= PCI_MSIX_TABLE_OFFSET; @@ -558,6 +645,7 @@ int
> > dw_pcie_ep_init(struct dw_pcie_ep *ep)
> >  	int i;
> >  	int ret;
> >  	u32 reg;
> > +	u8 func_no;
> >  	void *addr;
> >  	unsigned int nbars;
> >  	unsigned int offset;
> > @@ -565,6 +653,9 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >  	struct device *dev = pci->dev;
> >  	struct device_node *np = dev->of_node;
> > +	struct dw_pcie_ep_func *ep_func;
> > +
> > +	INIT_LIST_HEAD(&ep->func_list);
> >
> >  	if (!pci->dbi_base || !pci->dbi_base2) {
> >  		dev_err(dev, "dbi_base/dbi_base2 is not populated\n"); @@ -624,9
> > +715,19 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
> >  	if (ret < 0)
> >  		epc->max_functions = 1;
> >
> > -	ep->msi_cap = dw_pcie_find_capability(pci, PCI_CAP_ID_MSI);
> > +	for (func_no = 0; func_no < epc->max_functions; func_no++) {
> > +		ep_func = devm_kzalloc(dev, sizeof(*ep_func), GFP_KERNEL);
> > +		if (!ep_func)
> > +			return -ENOMEM;
> >
> > -	ep->msix_cap = dw_pcie_find_capability(pci, PCI_CAP_ID_MSIX);
> > +		ep_func->func_no = func_no;
> > +		ep_func->msi_cap = dw_pcie_ep_find_capability(ep, func_no,
> > +							      PCI_CAP_ID_MSI);
> > +		ep_func->msix_cap = dw_pcie_ep_find_capability(ep, func_no,
> > +							       PCI_CAP_ID_MSIX);
> > +
> > +		list_add_tail(&ep_func->list, &ep->func_list);
> > +	}
> 
> Whilst your patch addresses the issue of giving each function the ability to
> have differing capabilities - I feel that this solution doesn't go deep enough.
> 
> In my view the root issue here is that 'struct dw_pcie_ep' represents both a
> EP controller and a *single* EP function. I think that there should be a
> representation for an EP controller and a representation for a EP function (i.e.
> some separation). Thus allowing one EP controller to have many EP functions.
> This isn't too dissimilar to host bridges and their functions.
> Others here may have different views.
> 
> It may be unlikely now, but EP functions belonging to the same bit of IP may
> have differing functionality - your approach addresses that for MSI/MSI
> capabilities, but what about other differences?

Sorry, I am not clear about the other differences is? are there have any defect?

> 
> (It would be really nice as well if an EP controller could provide config
> read/write ops such that existing functions in the core such as
> __pci_find_next_capability could be reused - instead of copying them like
> dw_pcie_ep_find_capability. However I don't think this is feasible.)

Yes, agree you viewpoint, but this is the best way that I can thought ^_^,
may be others can give the best solution.

Thanks
Xiaowei

> 
> Thanks,
> 
> Andrew Murray
> 
> >
> >  	if (ep->ops->ep_init)
> >  		ep->ops->ep_init(ep);
> > diff --git a/drivers/pci/controller/dwc/pcie-designware.h
> > b/drivers/pci/controller/dwc/pcie-designware.h
> > index 56789be..a57743c 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware.h
> > +++ b/drivers/pci/controller/dwc/pcie-designware.h
> > @@ -221,8 +221,16 @@ struct dw_pcie_ep_ops {
> >  	unsigned int (*func_conf_select)(struct dw_pcie_ep *ep, u8 func_no);
> > };
> >
> > +struct dw_pcie_ep_func {
> > +	struct list_head	list;
> > +	u8			func_no;
> > +	u8			msi_cap;	/* MSI capability offset */
> > +	u8			msix_cap;	/* MSI-X capability offset */
> > +};
> > +
> >  struct dw_pcie_ep {
> >  	struct pci_epc		*epc;
> > +	struct list_head	func_list;
> >  	const struct dw_pcie_ep_ops *ops;
> >  	phys_addr_t		phys_base;
> >  	size_t			addr_size;
> > @@ -235,8 +243,6 @@ struct dw_pcie_ep {
> >  	u32			num_ob_windows;
> >  	void __iomem		*msi_mem;
> >  	phys_addr_t		msi_mem_phys;
> > -	u8			msi_cap;	/* MSI capability offset */
> > -	u8			msix_cap;	/* MSI-X capability offset */
> >  };
> >
> >  struct dw_pcie_ops {
> > @@ -425,6 +431,8 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep
> > *ep, u8 func_no,  int dw_pcie_ep_raise_msix_irq_doorbell(struct
> dw_pcie_ep *ep, u8 func_no,
> >  				       u16 interrupt_num);
> >  void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar);
> > +struct dw_pcie_ep_func *
> > +dw_pcie_ep_get_func_from_ep(struct dw_pcie_ep *ep, u8 func_no);
> >  #else
> >  static inline void dw_pcie_ep_linkup(struct dw_pcie_ep *ep)  { @@
> > -466,5 +474,11 @@ static inline int
> > dw_pcie_ep_raise_msix_irq_doorbell(struct dw_pcie_ep *ep,  static
> > inline void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno
> > bar)  {  }
> > +
> > +struct dw_pcie_ep_func *
> > +dw_pcie_ep_get_func_from_ep(struct dw_pcie_ep *ep, u8 func_no) {
> > +	return NULL;
> > +}
> >  #endif
> >  #endif /* _PCIE_DESIGNWARE_H */
> > --
> > 2.9.5
> >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 01/11] PCI: designware-ep: Add multiple PFs support for DWC
  2019-09-02 16:26   ` Andrew Murray
@ 2019-09-03  3:43     ` Xiaowei Bao
  2019-09-26 10:29       ` Andrew Murray
  0 siblings, 1 reply; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-03  3:43 UTC (permalink / raw)
  To: Andrew Murray
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, M.h. Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, Leo Li, shawnguo, Mingkai Hu



> -----Original Message-----
> From: Andrew Murray <andrew.murray@arm.com>
> Sent: 2019年9月3日 0:26
> To: Xiaowei Bao <xiaowei.bao@nxp.com>
> Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> gregkh@linuxfoundation.org; Z.q. Hou <zhiqiang.hou@nxp.com>;
> arnd@arndb.de
> Subject: Re: [PATCH v3 01/11] PCI: designware-ep: Add multiple PFs support
> for DWC
> 
> On Mon, Sep 02, 2019 at 11:17:06AM +0800, Xiaowei Bao wrote:
> > Add multiple PFs support for DWC, different PF have different config
> > space we use pf-offset property which get from the DTS to access the
> > different pF
> 
> This needs to be updated as this no longer comes from the DT.

Yes, thanks

Thanks
Xiaowei

> 
> > config space.
> >
> > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> 
> 
> We're assuming:
> 
>  - The offset address (func_offset) between PF's in the memory map can be
>    different between different DWC implementations. And also that it's
>    possible for DWC implementations to address PFs without using an offset.
> 
>  - The current approach is preferable to adding DWC EP driver callbacks
>    for writing to the EP config space (e.g. a variant of dw_pcie_writew_dbi
>    that takes a func number).

Even if use the a variant of dw_pcie_writew_dbi, we also need a offset value form
different platform, due to the different platform may be have different implement
about this, so I am not sure how to implement the variant of dw_pcie_writew_dbi?
  
> 
> I'm keen to hear feedback from Jingoo/Gustavo on this.

OK, expect the feedback.

Thanks 
Xiaowei

> 
> Thanks,
> 
> Andrew Murray
> 
> > ---
> > v2:
> >  - Remove duplicate redundant code.
> >  - Reimplement the PF config space access way.
> > v3:
> >  - Integrate duplicate code for func_select.
> >  - Move PCIE_ATU_FUNC_NUM(pf) (pf << 20) to ((pf) << 20).
> >  - Add the comments for func_conf_select function.
> >
> >  drivers/pci/controller/dwc/pcie-designware-ep.c | 123
> ++++++++++++++++--------
> >  drivers/pci/controller/dwc/pcie-designware.c    |  59 ++++++++----
> >  drivers/pci/controller/dwc/pcie-designware.h    |  18 +++-
> >  3 files changed, 142 insertions(+), 58 deletions(-)
> >
> > diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c
> > b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > index 65f4792..eb851c2 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > @@ -19,12 +19,26 @@ void dw_pcie_ep_linkup(struct dw_pcie_ep *ep)
> >  	pci_epc_linkup(epc);
> >  }
> >
> > -static void __dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno
> bar,
> > -				   int flags)
> > +static unsigned int dw_pcie_ep_func_select(struct dw_pcie_ep *ep, u8
> > +func_no) {
> > +	unsigned int func_offset = 0;
> > +
> > +	if (ep->ops->func_conf_select)
> > +		func_offset = ep->ops->func_conf_select(ep, func_no);
> > +
> > +	return func_offset;
> > +}
> > +
> > +static void __dw_pcie_ep_reset_bar(struct dw_pcie *pci, u8 func_no,
> > +				   enum pci_barno bar, int flags)
> >  {
> >  	u32 reg;
> > +	unsigned int func_offset = 0;
> > +	struct dw_pcie_ep *ep = &pci->ep;
> > +
> > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> >
> > -	reg = PCI_BASE_ADDRESS_0 + (4 * bar);
> > +	reg = func_offset + PCI_BASE_ADDRESS_0 + (4 * bar);
> >  	dw_pcie_dbi_ro_wr_en(pci);
> >  	dw_pcie_writel_dbi2(pci, reg, 0x0);
> >  	dw_pcie_writel_dbi(pci, reg, 0x0);
> > @@ -37,7 +51,12 @@ static void __dw_pcie_ep_reset_bar(struct dw_pcie
> > *pci, enum pci_barno bar,
> >
> >  void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar)  {
> > -	__dw_pcie_ep_reset_bar(pci, bar, 0);
> > +	u8 func_no, funcs;
> > +
> > +	funcs = pci->ep.epc->max_functions;
> > +
> > +	for (func_no = 0; func_no < funcs; func_no++)
> > +		__dw_pcie_ep_reset_bar(pci, func_no, bar, 0);
> >  }
> >
> >  static int dw_pcie_ep_write_header(struct pci_epc *epc, u8 func_no,
> > @@ -45,28 +64,31 @@ static int dw_pcie_ep_write_header(struct pci_epc
> > *epc, u8 func_no,  {
> >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > +	unsigned int func_offset = 0;
> > +
> > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> >
> >  	dw_pcie_dbi_ro_wr_en(pci);
> > -	dw_pcie_writew_dbi(pci, PCI_VENDOR_ID, hdr->vendorid);
> > -	dw_pcie_writew_dbi(pci, PCI_DEVICE_ID, hdr->deviceid);
> > -	dw_pcie_writeb_dbi(pci, PCI_REVISION_ID, hdr->revid);
> > -	dw_pcie_writeb_dbi(pci, PCI_CLASS_PROG, hdr->progif_code);
> > -	dw_pcie_writew_dbi(pci, PCI_CLASS_DEVICE,
> > +	dw_pcie_writew_dbi(pci, func_offset + PCI_VENDOR_ID, hdr->vendorid);
> > +	dw_pcie_writew_dbi(pci, func_offset + PCI_DEVICE_ID, hdr->deviceid);
> > +	dw_pcie_writeb_dbi(pci, func_offset + PCI_REVISION_ID, hdr->revid);
> > +	dw_pcie_writeb_dbi(pci, func_offset + PCI_CLASS_PROG,
> hdr->progif_code);
> > +	dw_pcie_writew_dbi(pci, func_offset + PCI_CLASS_DEVICE,
> >  			   hdr->subclass_code | hdr->baseclass_code << 8);
> > -	dw_pcie_writeb_dbi(pci, PCI_CACHE_LINE_SIZE,
> > +	dw_pcie_writeb_dbi(pci, func_offset + PCI_CACHE_LINE_SIZE,
> >  			   hdr->cache_line_size);
> > -	dw_pcie_writew_dbi(pci, PCI_SUBSYSTEM_VENDOR_ID,
> > +	dw_pcie_writew_dbi(pci, func_offset + PCI_SUBSYSTEM_VENDOR_ID,
> >  			   hdr->subsys_vendor_id);
> > -	dw_pcie_writew_dbi(pci, PCI_SUBSYSTEM_ID, hdr->subsys_id);
> > -	dw_pcie_writeb_dbi(pci, PCI_INTERRUPT_PIN,
> > +	dw_pcie_writew_dbi(pci, func_offset + PCI_SUBSYSTEM_ID,
> hdr->subsys_id);
> > +	dw_pcie_writeb_dbi(pci, func_offset + PCI_INTERRUPT_PIN,
> >  			   hdr->interrupt_pin);
> >  	dw_pcie_dbi_ro_wr_dis(pci);
> >
> >  	return 0;
> >  }
> >
> > -static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, enum
> pci_barno bar,
> > -				  dma_addr_t cpu_addr,
> > +static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no,
> > +				  enum pci_barno bar, dma_addr_t cpu_addr,
> >  				  enum dw_pcie_as_type as_type)
> >  {
> >  	int ret;
> > @@ -79,7 +101,7 @@ static int dw_pcie_ep_inbound_atu(struct
> dw_pcie_ep *ep, enum pci_barno bar,
> >  		return -EINVAL;
> >  	}
> >
> > -	ret = dw_pcie_prog_inbound_atu(pci, free_win, bar, cpu_addr,
> > +	ret = dw_pcie_prog_inbound_atu(pci, func_no, free_win, bar,
> > +cpu_addr,
> >  				       as_type);
> >  	if (ret < 0) {
> >  		dev_err(pci->dev, "Failed to program IB window\n"); @@ -92,7
> +114,8
> > @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, enum
> pci_barno bar,
> >  	return 0;
> >  }
> >
> > -static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, phys_addr_t
> > phys_addr,
> > +static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, u8 func_no,
> > +				   phys_addr_t phys_addr,
> >  				   u64 pci_addr, size_t size)
> >  {
> >  	u32 free_win;
> > @@ -104,8 +127,8 @@ static int dw_pcie_ep_outbound_atu(struct
> dw_pcie_ep *ep, phys_addr_t phys_addr,
> >  		return -EINVAL;
> >  	}
> >
> > -	dw_pcie_prog_outbound_atu(pci, free_win, PCIE_ATU_TYPE_MEM,
> > -				  phys_addr, pci_addr, size);
> > +	dw_pcie_prog_ep_outbound_atu(pci, func_no, free_win,
> PCIE_ATU_TYPE_MEM,
> > +				     phys_addr, pci_addr, size);
> >
> >  	set_bit(free_win, ep->ob_window_map);
> >  	ep->outbound_addr[free_win] = phys_addr; @@ -121,7 +144,7 @@
> static
> > void dw_pcie_ep_clear_bar(struct pci_epc *epc, u8 func_no,
> >  	enum pci_barno bar = epf_bar->barno;
> >  	u32 atu_index = ep->bar_to_atu[bar];
> >
> > -	__dw_pcie_ep_reset_bar(pci, bar, epf_bar->flags);
> > +	__dw_pcie_ep_reset_bar(pci, func_no, bar, epf_bar->flags);
> >
> >  	dw_pcie_disable_atu(pci, atu_index, DW_PCIE_REGION_INBOUND);
> >  	clear_bit(atu_index, ep->ib_window_map); @@ -137,14 +160,20 @@
> > static int dw_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no,
> >  	size_t size = epf_bar->size;
> >  	int flags = epf_bar->flags;
> >  	enum dw_pcie_as_type as_type;
> > -	u32 reg = PCI_BASE_ADDRESS_0 + (4 * bar);
> > +	u32 reg;
> > +	unsigned int func_offset = 0;
> > +
> > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > +
> > +	reg = PCI_BASE_ADDRESS_0 + (4 * bar) + func_offset;
> >
> >  	if (!(flags & PCI_BASE_ADDRESS_SPACE))
> >  		as_type = DW_PCIE_AS_MEM;
> >  	else
> >  		as_type = DW_PCIE_AS_IO;
> >
> > -	ret = dw_pcie_ep_inbound_atu(ep, bar, epf_bar->phys_addr, as_type);
> > +	ret = dw_pcie_ep_inbound_atu(ep, func_no, bar,
> > +				     epf_bar->phys_addr, as_type);
> >  	if (ret)
> >  		return ret;
> >
> > @@ -202,7 +231,7 @@ static int dw_pcie_ep_map_addr(struct pci_epc
> *epc, u8 func_no,
> >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >
> > -	ret = dw_pcie_ep_outbound_atu(ep, addr, pci_addr, size);
> > +	ret = dw_pcie_ep_outbound_atu(ep, func_no, addr, pci_addr, size);
> >  	if (ret) {
> >  		dev_err(pci->dev, "Failed to enable address\n");
> >  		return ret;
> > @@ -216,11 +245,14 @@ static int dw_pcie_ep_get_msi(struct pci_epc
> *epc, u8 func_no)
> >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >  	u32 val, reg;
> > +	unsigned int func_offset = 0;
> >
> >  	if (!ep->msi_cap)
> >  		return -EINVAL;
> >
> > -	reg = ep->msi_cap + PCI_MSI_FLAGS;
> > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > +
> > +	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
> >  	val = dw_pcie_readw_dbi(pci, reg);
> >  	if (!(val & PCI_MSI_FLAGS_ENABLE))
> >  		return -EINVAL;
> > @@ -235,11 +267,14 @@ static int dw_pcie_ep_set_msi(struct pci_epc
> *epc, u8 func_no, u8 interrupts)
> >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >  	u32 val, reg;
> > +	unsigned int func_offset = 0;
> >
> >  	if (!ep->msi_cap)
> >  		return -EINVAL;
> >
> > -	reg = ep->msi_cap + PCI_MSI_FLAGS;
> > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > +
> > +	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
> >  	val = dw_pcie_readw_dbi(pci, reg);
> >  	val &= ~PCI_MSI_FLAGS_QMASK;
> >  	val |= (interrupts << 1) & PCI_MSI_FLAGS_QMASK; @@ -255,11 +290,14
> > @@ static int dw_pcie_ep_get_msix(struct pci_epc *epc, u8 func_no)
> >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >  	u32 val, reg;
> > +	unsigned int func_offset = 0;
> >
> >  	if (!ep->msix_cap)
> >  		return -EINVAL;
> >
> > -	reg = ep->msix_cap + PCI_MSIX_FLAGS;
> > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > +
> > +	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
> >  	val = dw_pcie_readw_dbi(pci, reg);
> >  	if (!(val & PCI_MSIX_FLAGS_ENABLE))
> >  		return -EINVAL;
> > @@ -274,11 +312,14 @@ static int dw_pcie_ep_set_msix(struct pci_epc
> *epc, u8 func_no, u16 interrupts)
> >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >  	u32 val, reg;
> > +	unsigned int func_offset = 0;
> >
> >  	if (!ep->msix_cap)
> >  		return -EINVAL;
> >
> > -	reg = ep->msix_cap + PCI_MSIX_FLAGS;
> > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > +
> > +	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
> >  	val = dw_pcie_readw_dbi(pci, reg);
> >  	val &= ~PCI_MSIX_FLAGS_QSIZE;
> >  	val |= interrupts;
> > @@ -365,6 +406,7 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep
> *ep, u8 func_no,
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >  	struct pci_epc *epc = ep->epc;
> >  	unsigned int aligned_offset;
> > +	unsigned int func_offset = 0;
> >  	u16 msg_ctrl, msg_data;
> >  	u32 msg_addr_lower, msg_addr_upper, reg;
> >  	u64 msg_addr;
> > @@ -374,20 +416,22 @@ int dw_pcie_ep_raise_msi_irq(struct
> dw_pcie_ep *ep, u8 func_no,
> >  	if (!ep->msi_cap)
> >  		return -EINVAL;
> >
> > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > +
> >  	/* Raise MSI per the PCI Local Bus Specification Revision 3.0, 6.8.1. */
> > -	reg = ep->msi_cap + PCI_MSI_FLAGS;
> > +	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
> >  	msg_ctrl = dw_pcie_readw_dbi(pci, reg);
> >  	has_upper = !!(msg_ctrl & PCI_MSI_FLAGS_64BIT);
> > -	reg = ep->msi_cap + PCI_MSI_ADDRESS_LO;
> > +	reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_LO;
> >  	msg_addr_lower = dw_pcie_readl_dbi(pci, reg);
> >  	if (has_upper) {
> > -		reg = ep->msi_cap + PCI_MSI_ADDRESS_HI;
> > +		reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_HI;
> >  		msg_addr_upper = dw_pcie_readl_dbi(pci, reg);
> > -		reg = ep->msi_cap + PCI_MSI_DATA_64;
> > +		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_64;
> >  		msg_data = dw_pcie_readw_dbi(pci, reg);
> >  	} else {
> >  		msg_addr_upper = 0;
> > -		reg = ep->msi_cap + PCI_MSI_DATA_32;
> > +		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_32;
> >  		msg_data = dw_pcie_readw_dbi(pci, reg);
> >  	}
> >  	aligned_offset = msg_addr_lower & (epc->mem->page_size - 1); @@
> > -406,11 +450,12 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep,
> > u8 func_no,  }
> >
> >  int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
> > -			     u16 interrupt_num)
> > +			      u16 interrupt_num)
> >  {
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >  	struct pci_epc *epc = ep->epc;
> >  	u16 tbl_offset, bir;
> > +	unsigned int func_offset = 0;
> >  	u32 bar_addr_upper, bar_addr_lower;
> >  	u32 msg_addr_upper, msg_addr_lower;
> >  	u32 reg, msg_data, vec_ctrl;
> > @@ -418,12 +463,14 @@ int dw_pcie_ep_raise_msix_irq(struct
> dw_pcie_ep *ep, u8 func_no,
> >  	void __iomem *msix_tbl;
> >  	int ret;
> >
> > -	reg = ep->msix_cap + PCI_MSIX_TABLE;
> > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > +
> > +	reg = ep->msix_cap + func_offset + PCI_MSIX_TABLE;
> >  	tbl_offset = dw_pcie_readl_dbi(pci, reg);
> >  	bir = (tbl_offset & PCI_MSIX_TABLE_BIR);
> >  	tbl_offset &= PCI_MSIX_TABLE_OFFSET;
> >
> > -	reg = PCI_BASE_ADDRESS_0 + (4 * bir);
> > +	reg = PCI_BASE_ADDRESS_0 + func_offset + (4 * bir);
> >  	bar_addr_upper = 0;
> >  	bar_addr_lower = dw_pcie_readl_dbi(pci, reg);
> >  	reg_u64 = (bar_addr_lower & PCI_BASE_ADDRESS_MEM_TYPE_MASK);
> @@
> > -559,13 +606,13 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
> >  	ep->epc = epc;
> >  	epc_set_drvdata(epc, ep);
> >
> > -	if (ep->ops->ep_init)
> > -		ep->ops->ep_init(ep);
> > -
> >  	ret = of_property_read_u8(np, "max-functions", &epc->max_functions);
> >  	if (ret < 0)
> >  		epc->max_functions = 1;
> >
> > +	if (ep->ops->ep_init)
> > +		ep->ops->ep_init(ep);
> > +
> >  	ret = __pci_epc_mem_init(epc, ep->phys_base, ep->addr_size,
> >  				 ep->page_size);
> >  	if (ret < 0) {
> > diff --git a/drivers/pci/controller/dwc/pcie-designware.c
> > b/drivers/pci/controller/dwc/pcie-designware.c
> > index 143cb6c..ede2e75 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware.c
> > @@ -238,9 +238,10 @@ static void dw_pcie_writel_ob_unroll(struct
> dw_pcie *pci, u32 index, u32 reg,
> >  	dw_pcie_writel_atu(pci, offset + reg, val);  }
> >
> > -static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int
> index,
> > -					     int type, u64 cpu_addr,
> > -					     u64 pci_addr, u32 size)
> > +static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, u8
> func_no,
> > +					     int index, int type,
> > +					     u64 cpu_addr, u64 pci_addr,
> > +					     u32 size)
> >  {
> >  	u32 retries, val;
> >
> > @@ -255,7 +256,7 @@ static void
> dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
> >  	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_UPPER_TARGET,
> >  				 upper_32_bits(pci_addr));
> >  	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1,
> > -				 type);
> > +				 type | PCIE_ATU_FUNC_NUM(func_no));
> >  	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL2,
> >  				 PCIE_ATU_ENABLE);
> >
> > @@ -274,8 +275,9 @@ static void
> dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
> >  	dev_err(pci->dev, "Outbound iATU is not being enabled\n");  }
> >
> > -void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
> > -			       u64 cpu_addr, u64 pci_addr, u32 size)
> > +static void __dw_pcie_prog_outbound_atu(struct dw_pcie *pci, u8
> func_no,
> > +					int index, int type, u64 cpu_addr,
> > +					u64 pci_addr, u32 size)
> >  {
> >  	u32 retries, val;
> >
> > @@ -283,8 +285,8 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie
> *pci, int index, int type,
> >  		cpu_addr = pci->ops->cpu_addr_fixup(pci, cpu_addr);
> >
> >  	if (pci->iatu_unroll_enabled) {
> > -		dw_pcie_prog_outbound_atu_unroll(pci, index, type, cpu_addr,
> > -						 pci_addr, size);
> > +		dw_pcie_prog_outbound_atu_unroll(pci, func_no, index, type,
> > +						 cpu_addr, pci_addr, size);
> >  		return;
> >  	}
> >
> > @@ -300,7 +302,8 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie
> *pci, int index, int type,
> >  			   lower_32_bits(pci_addr));
> >  	dw_pcie_writel_dbi(pci, PCIE_ATU_UPPER_TARGET,
> >  			   upper_32_bits(pci_addr));
> > -	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type);
> > +	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type |
> > +			   PCIE_ATU_FUNC_NUM(func_no));
> >  	dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE);
> >
> >  	/*
> > @@ -317,6 +320,21 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie
> *pci, int index, int type,
> >  	dev_err(pci->dev, "Outbound iATU is not being enabled\n");  }
> >
> > +void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
> > +			       u64 cpu_addr, u64 pci_addr, u32 size) {
> > +	__dw_pcie_prog_outbound_atu(pci, 0, index, type,
> > +				    cpu_addr, pci_addr, size);
> > +}
> > +
> > +void dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int
> index,
> > +				  int type, u64 cpu_addr, u64 pci_addr,
> > +				  u32 size)
> > +{
> > +	__dw_pcie_prog_outbound_atu(pci, func_no, index, type,
> > +				    cpu_addr, pci_addr, size);
> > +}
> > +
> >  static u32 dw_pcie_readl_ib_unroll(struct dw_pcie *pci, u32 index,
> > u32 reg)  {
> >  	u32 offset = PCIE_GET_ATU_INB_UNR_REG_OFFSET(index);
> > @@ -332,8 +350,8 @@ static void dw_pcie_writel_ib_unroll(struct dw_pcie
> *pci, u32 index, u32 reg,
> >  	dw_pcie_writel_atu(pci, offset + reg, val);  }
> >
> > -static int dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, int index,
> > -					   int bar, u64 cpu_addr,
> > +static int dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, u8
> func_no,
> > +					   int index, int bar, u64 cpu_addr,
> >  					   enum dw_pcie_as_type as_type)  {
> >  	int type;
> > @@ -355,8 +373,10 @@ static int dw_pcie_prog_inbound_atu_unroll(struct
> dw_pcie *pci, int index,
> >  		return -EINVAL;
> >  	}
> >
> > -	dw_pcie_writel_ib_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1,
> type);
> > +	dw_pcie_writel_ib_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1,
> type |
> > +				 PCIE_ATU_FUNC_NUM(func_no));
> >  	dw_pcie_writel_ib_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL2,
> > +				 PCIE_ATU_FUNC_NUM_MATCH_EN |
> >  				 PCIE_ATU_ENABLE |
> >  				 PCIE_ATU_BAR_MODE_ENABLE | (bar << 8));
> >
> > @@ -377,14 +397,15 @@ static int
> dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, int index,
> >  	return -EBUSY;
> >  }
> >
> > -int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int bar,
> > -			     u64 cpu_addr, enum dw_pcie_as_type as_type)
> > +int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> > +			     int bar, u64 cpu_addr,
> > +			     enum dw_pcie_as_type as_type)
> >  {
> >  	int type;
> >  	u32 retries, val;
> >
> >  	if (pci->iatu_unroll_enabled)
> > -		return dw_pcie_prog_inbound_atu_unroll(pci, index, bar,
> > +		return dw_pcie_prog_inbound_atu_unroll(pci, func_no, index, bar,
> >  						       cpu_addr, as_type);
> >
> >  	dw_pcie_writel_dbi(pci, PCIE_ATU_VIEWPORT,
> PCIE_ATU_REGION_INBOUND |
> > @@ -403,9 +424,11 @@ int dw_pcie_prog_inbound_atu(struct dw_pcie
> *pci, int index, int bar,
> >  		return -EINVAL;
> >  	}
> >
> > -	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type);
> > -	dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE
> > -			   | PCIE_ATU_BAR_MODE_ENABLE | (bar << 8));
> > +	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type |
> > +			   PCIE_ATU_FUNC_NUM(func_no));
> > +	dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE |
> > +			   PCIE_ATU_FUNC_NUM_MATCH_EN |
> > +			   PCIE_ATU_BAR_MODE_ENABLE | (bar << 8));
> >
> >  	/*
> >  	 * Make sure ATU enable takes effect before any subsequent config
> > diff --git a/drivers/pci/controller/dwc/pcie-designware.h
> > b/drivers/pci/controller/dwc/pcie-designware.h
> > index 5a18e94..6aca0bb 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware.h
> > +++ b/drivers/pci/controller/dwc/pcie-designware.h
> > @@ -71,9 +71,11 @@
> >  #define PCIE_ATU_TYPE_IO		0x2
> >  #define PCIE_ATU_TYPE_CFG0		0x4
> >  #define PCIE_ATU_TYPE_CFG1		0x5
> > +#define PCIE_ATU_FUNC_NUM(pf)           ((pf) << 20)
> >  #define PCIE_ATU_CR2			0x908
> >  #define PCIE_ATU_ENABLE			BIT(31)
> >  #define PCIE_ATU_BAR_MODE_ENABLE	BIT(30)
> > +#define PCIE_ATU_FUNC_NUM_MATCH_EN      BIT(19)
> >  #define PCIE_ATU_LOWER_BASE		0x90C
> >  #define PCIE_ATU_UPPER_BASE		0x910
> >  #define PCIE_ATU_LIMIT			0x914
> > @@ -206,6 +208,14 @@ struct dw_pcie_ep_ops {
> >  	int	(*raise_irq)(struct dw_pcie_ep *ep, u8 func_no,
> >  			     enum pci_epc_irq_type type, u16 interrupt_num);
> >  	const struct pci_epc_features* (*get_features)(struct dw_pcie_ep
> > *ep);
> > +	/*
> > +	 * Provide a method to implement the different func config space
> > +	 * access for different platform, if different func have different
> > +	 * offset, return the offset of func. if use write a register way
> > +	 * return a 0, and implement code in callback function of platform
> > +	 * driver.
> > +	 */
> > +	unsigned int (*func_conf_select)(struct dw_pcie_ep *ep, u8 func_no);
> >  };
> >
> >  struct dw_pcie_ep {
> > @@ -277,8 +287,12 @@ int dw_pcie_wait_for_link(struct dw_pcie *pci);
> > void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index,
> >  			       int type, u64 cpu_addr, u64 pci_addr,
> >  			       u32 size);
> > -int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int bar,
> > -			     u64 cpu_addr, enum dw_pcie_as_type as_type);
> > +void dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int
> index,
> > +				  int type, u64 cpu_addr, u64 pci_addr,
> > +				  u32 size);
> > +int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> > +			     int bar, u64 cpu_addr,
> > +			     enum dw_pcie_as_type as_type);
> >  void dw_pcie_disable_atu(struct dw_pcie *pci, int index,
> >  			 enum dw_pcie_region_type type);
> >  void dw_pcie_setup(struct dw_pcie *pci);
> > --
> > 2.9.5
> >
> >
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists
> > .infradead.org%2Fmailman%2Flistinfo%2Flinux-arm-kernel&amp;data=02%
> 7C0
> >
> 1%7Cxiaowei.bao%40nxp.com%7C99eef14a525040ed3eab08d72fc244f1%7C
> 686ea1d
> >
> 3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637030383769341270&amp;sd
> ata=ck2EC
> > %2FJYCjWErvbUM%2FT%2BoVMANMwyLRI4gVRssdnd04w%3D&amp;reser
> ved=0
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 08/11] PCI: layerscape: Modify the MSIX to the doorbell mode
  2019-09-02 12:01   ` Andrew Murray
@ 2019-09-12 11:24     ` Gustavo Pimentel
  2019-09-14  6:37       ` Xiaowei Bao
  0 siblings, 1 reply; 42+ messages in thread
From: Gustavo Pimentel @ 2019-09-12 11:24 UTC (permalink / raw)
  To: Andrew Murray, Xiaowei Bao
  Cc: mark.rutland, roy.zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, zhiqiang.hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, minghuan.Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel@synopsys.com, leoyang.li, shawnguo, mingkai.hu

Hi,

Sorry for the delay I was in parental leave and I'm still trying not to 
drown in the mailing list emails... 😊

On Mon, Sep 2, 2019 at 13:1:47, Andrew Murray <andrew.murray@arm.com> 
wrote:

> On Mon, Sep 02, 2019 at 11:17:13AM +0800, Xiaowei Bao wrote:
> > dw_pcie_ep_raise_msix_irq was never called in the exisitng driver
> > before, because the ls1046a platform don't support the MSIX feature
> > and msix_capable was always set to false.
> > Now that add the ls1088a platform with MSIX support, but the existing
> > dw_pcie_ep_raise_msix_irq doesn't work, so use the doorbell method to
> > support the MSIX feature.

Hum... the implementation of msix implementation did work on my use case, 
however, at the time the setup used for developing and testing the 
implementation only had one PF (by default 0). Perhaps this could was is 
causing the different behavior between our setups.

You have more than one PF, right?

If I remember correctly, msix feature support entered on kernel 4.19 
version and it worked quite well at the time, but I didn't test since 
there (I've to manage time to be able to retest it again), I'm didn't 
seen any patch that could interfere with this.

Regards,
Gustavo


> > 
> > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> 
> Reviewed-by: Andrew Murray <andrew.murray@arm.com>
> 
> > ---
> > v2: 
> >  - No change
> > v3:
> >  - Modify the commit message make it clearly.
> > 
> >  drivers/pci/controller/dwc/pci-layerscape-ep.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > index 1e07287..5f0cb99 100644
> > --- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > +++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > @@ -79,7 +79,8 @@ static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> >  	case PCI_EPC_IRQ_MSI:
> >  		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
> >  	case PCI_EPC_IRQ_MSIX:
> > -		return dw_pcie_ep_raise_msix_irq(ep, func_no, interrupt_num);
> > +		return dw_pcie_ep_raise_msix_irq_doorbell(ep, func_no,
> > +							  interrupt_num);
> >  	default:
> >  		dev_err(pci->dev, "UNKNOWN IRQ type\n");
> >  		return -EINVAL;
> > -- 
> > 2.9.5
> > 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* Re: [PATCH v3 09/11] PCI: layerscape: Add EP mode support for ls1088a and ls2088a
  2019-09-03  1:47     ` Xiaowei Bao
@ 2019-09-12 12:49       ` Andrew Murray
  2019-09-14  4:10         ` Xiaowei Bao
  0 siblings, 1 reply; 42+ messages in thread
From: Andrew Murray @ 2019-09-12 12:49 UTC (permalink / raw)
  To: Xiaowei Bao
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, M.h. Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, Leo Li, shawnguo, Mingkai Hu

On Tue, Sep 03, 2019 at 01:47:36AM +0000, Xiaowei Bao wrote:
> 
> 
> > -----Original Message-----
> > From: Andrew Murray <andrew.murray@arm.com>
> > Sent: 2019年9月2日 20:46
> > To: Xiaowei Bao <xiaowei.bao@nxp.com>
> > Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> > Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> > Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> > Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> > gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> > devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> > arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> > <zhiqiang.hou@nxp.com>
> > Subject: Re: [PATCH v3 09/11] PCI: layerscape: Add EP mode support for
> > ls1088a and ls2088a
> > 
> > On Mon, Sep 02, 2019 at 11:17:14AM +0800, Xiaowei Bao wrote:
> > > Add PCIe EP mode support for ls1088a and ls2088a, there are some
> > > difference between LS1 and LS2 platform, so refactor the code of the
> > > EP driver.
> > >
> > > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > ---
> > > v2:
> > >  - This is a new patch for supporting the ls1088a and ls2088a platform.
> > > v3:
> > >  - Adjust the some struct assignment order in probe function.
> > >
> > >  drivers/pci/controller/dwc/pci-layerscape-ep.c | 72
> > > +++++++++++++++++++-------
> > >  1 file changed, 53 insertions(+), 19 deletions(-)
> > >
> > > diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > index 5f0cb99..723bbe5 100644
> > > --- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > +++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > @@ -20,27 +20,29 @@
> > >
> > >  #define PCIE_DBI2_OFFSET		0x1000	/* DBI2 base address*/
> > >
> > > -struct ls_pcie_ep {
> > > -	struct dw_pcie		*pci;
> > > -	struct pci_epc_features	*ls_epc;
> > > +#define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
> > > +
> > > +struct ls_pcie_ep_drvdata {
> > > +	u32				func_offset;
> > > +	const struct dw_pcie_ep_ops	*ops;
> > > +	const struct dw_pcie_ops	*dw_pcie_ops;
> > >  };
> > >
> > > -#define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
> > > +struct ls_pcie_ep {
> > > +	struct dw_pcie			*pci;
> > > +	struct pci_epc_features		*ls_epc;
> > > +	const struct ls_pcie_ep_drvdata *drvdata; };
> > >
> > >  static int ls_pcie_establish_link(struct dw_pcie *pci)  {
> > >  	return 0;
> > >  }
> > >
> > > -static const struct dw_pcie_ops ls_pcie_ep_ops = {
> > > +static const struct dw_pcie_ops dw_ls_pcie_ep_ops = {
> > >  	.start_link = ls_pcie_establish_link,  };
> > >
> > > -static const struct of_device_id ls_pcie_ep_of_match[] = {
> > > -	{ .compatible = "fsl,ls-pcie-ep",},
> > > -	{ },
> > > -};
> > > -
> > >  static const struct pci_epc_features*  ls_pcie_ep_get_features(struct
> > > dw_pcie_ep *ep)  { @@ -87,10 +89,39 @@ static int
> > > ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> > >  	}
> > >  }
> > >
> > > -static const struct dw_pcie_ep_ops pcie_ep_ops = {
> > > +static unsigned int ls_pcie_ep_func_conf_select(struct dw_pcie_ep *ep,
> > > +						u8 func_no)
> > > +{
> > > +	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > > +	struct ls_pcie_ep *pcie = to_ls_pcie_ep(pci);
> > > +
> > > +	WARN_ON(func_no && !pcie->drvdata->func_offset);
> > > +	return pcie->drvdata->func_offset * func_no; }
> > > +
> > > +static const struct dw_pcie_ep_ops ls_pcie_ep_ops = {
> > >  	.ep_init = ls_pcie_ep_init,
> > >  	.raise_irq = ls_pcie_ep_raise_irq,
> > >  	.get_features = ls_pcie_ep_get_features,
> > > +	.func_conf_select = ls_pcie_ep_func_conf_select, };
> > > +
> > > +static const struct ls_pcie_ep_drvdata ls1_ep_drvdata = {
> > > +	.ops = &ls_pcie_ep_ops,
> > > +	.dw_pcie_ops = &dw_ls_pcie_ep_ops,
> > > +};
> > > +
> > > +static const struct ls_pcie_ep_drvdata ls2_ep_drvdata = {
> > > +	.func_offset = 0x20000,
> > > +	.ops = &ls_pcie_ep_ops,
> > > +	.dw_pcie_ops = &dw_ls_pcie_ep_ops,
> > > +};
> > > +
> > > +static const struct of_device_id ls_pcie_ep_of_match[] = {
> > > +	{ .compatible = "fsl,ls1046a-pcie-ep", .data = &ls1_ep_drvdata },
> > > +	{ .compatible = "fsl,ls1088a-pcie-ep", .data = &ls2_ep_drvdata },
> > > +	{ .compatible = "fsl,ls2088a-pcie-ep", .data = &ls2_ep_drvdata },
> > > +	{ },
> > 
> > This removes support for "fsl,ls-pcie-ep" - was that intentional? If you do plan
> > to drop it please make sure you explain why in the commit message. See also
> > my comments in your dt-binding patch.
> 
> In fact, the u-boot will fixup the status property to 'status = enabled' in PCI node of 
> the DTS base on "fsl,ls-pcie-ep" compatible, so "fsl,ls-pcie-ep" is used, I used this
> compatible before, because the driver only support the LS1046a, but this time, I add
> the LS1088a and LS2088a support, and these two boards have some difference form
> LS1046a, so I changed the compatible. I am not sure whether need to add "fsl,ls-pcie-ep"
> in there, could you give some advice, thanks a lot.

It sounds like "fsl,ls-pcie-ep" can be a fallback for "fsl,ls1046a-pcie-ep".

I'm assuming that if someone used "fsl,ls1046a-pcie-ep" on ls1088a or ls2088a
hardware it would still work, but without the multiple PF support.

I.e. if "fsl,ls-pcie-ep" is given, treat it as ls1046a.

Thanks,

Andrew Murray

> 
> Thanks 
> Xiaowei
>  
> > 
> > Thanks,
> > 
> > Andrew Murray
> > 
> > >  };
> > >
> > >  static int __init ls_add_pcie_ep(struct ls_pcie_ep *pcie, @@ -103,7
> > > +134,7 @@ static int __init ls_add_pcie_ep(struct ls_pcie_ep *pcie,
> > >  	int ret;
> > >
> > >  	ep = &pci->ep;
> > > -	ep->ops = &pcie_ep_ops;
> > > +	ep->ops = pcie->drvdata->ops;
> > >
> > >  	res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> > "addr_space");
> > >  	if (!res)
> > > @@ -142,20 +173,23 @@ static int __init ls_pcie_ep_probe(struct
> > platform_device *pdev)
> > >  	if (!ls_epc)
> > >  		return -ENOMEM;
> > >
> > > -	dbi_base = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> > "regs");
> > > -	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_base);
> > > -	if (IS_ERR(pci->dbi_base))
> > > -		return PTR_ERR(pci->dbi_base);
> > > +	pcie->drvdata = of_device_get_match_data(dev);
> > >
> > > -	pci->dbi_base2 = pci->dbi_base + PCIE_DBI2_OFFSET;
> > >  	pci->dev = dev;
> > > -	pci->ops = &ls_pcie_ep_ops;
> > > -	pcie->pci = pci;
> > > +	pci->ops = pcie->drvdata->dw_pcie_ops;
> > >
> > >  	ls_epc->bar_fixed_64bit = (1 << BAR_2) | (1 << BAR_4),
> > >
> > > +	pcie->pci = pci;
> > >  	pcie->ls_epc = ls_epc;
> > >
> > > +	dbi_base = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> > "regs");
> > > +	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_base);
> > > +	if (IS_ERR(pci->dbi_base))
> > > +		return PTR_ERR(pci->dbi_base);
> > > +
> > > +	pci->dbi_base2 = pci->dbi_base + PCIE_DBI2_OFFSET;
> > > +
> > >  	platform_set_drvdata(pdev, pcie);
> > >
> > >  	ret = ls_add_pcie_ep(pcie, pdev);
> > > --
> > > 2.9.5
> > >

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* Re: [PATCH v3 11/11] misc: pci_endpoint_test: Add LS1088a in pci_device_id table
  2019-09-03  1:52     ` Xiaowei Bao
@ 2019-09-12 12:59       ` Andrew Murray
  2019-09-14  4:13         ` Xiaowei Bao
  0 siblings, 1 reply; 42+ messages in thread
From: Andrew Murray @ 2019-09-12 12:59 UTC (permalink / raw)
  To: Xiaowei Bao, helgaas
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, M.h. Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, Leo Li, shawnguo, Mingkai Hu

On Tue, Sep 03, 2019 at 01:52:30AM +0000, Xiaowei Bao wrote:
> 
> 
> > -----Original Message-----
> > From: Andrew Murray <andrew.murray@arm.com>
> > Sent: 2019年9月2日 20:55
> > To: Xiaowei Bao <xiaowei.bao@nxp.com>
> > Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> > Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> > Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> > Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> > gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> > devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> > arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> > <zhiqiang.hou@nxp.com>
> > Subject: Re: [PATCH v3 11/11] misc: pci_endpoint_test: Add LS1088a in
> > pci_device_id table
> > 
> > On Mon, Sep 02, 2019 at 11:17:16AM +0800, Xiaowei Bao wrote:
> > > Add LS1088a in pci_device_id table so that pci-epf-test can be used
> > > for testing PCIe EP in LS1088a.
> > >
> > > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > ---
> > > v2:
> > >  - No change.
> > > v3:
> > >  - No change.
> > >
> > >  drivers/misc/pci_endpoint_test.c | 1 +
> > >  1 file changed, 1 insertion(+)
> > >
> > > diff --git a/drivers/misc/pci_endpoint_test.c
> > > b/drivers/misc/pci_endpoint_test.c
> > > index 6e208a0..d531951 100644
> > > --- a/drivers/misc/pci_endpoint_test.c
> > > +++ b/drivers/misc/pci_endpoint_test.c
> > > @@ -793,6 +793,7 @@ static const struct pci_device_id
> > pci_endpoint_test_tbl[] = {
> > >  	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_DRA74x) },
> > >  	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_DRA72x) },
> > >  	{ PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, 0x81c0) },
> > > +	{ PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, 0x80c0) },
> > 
> > The Freescale PCI devices are the only devices in this table that don't have a
> > define for their device ID. I think a define should be created for both of the
> > device IDs above.
> 
> OK, but I only define in this file, I am not sure this can define in include/linux/pci_ids.h
> file 

This file seems a little inconsistent...

 - Two of the TI device IDs are defined in pci_ids.h and only used in pci_endpoint_test.c
 - One of the TI device IDs are defined in pci_endpoint_test.c and only used there
 - The Freescale device ID is hardcoded and only used in pci_endpoint_test.c

The header in pci_ids.h has a comment suggestion definitions are only added where used
in multiple files - yet I don't think this holds true.

Bjorn - do you have a suggestion?

Thanks,

Andrew Murray

> 
> Thanks 
> Xiaowei
> 
> > 
> > Thanks,
> > 
> > Andrew Murray
> > 
> > >  	{ PCI_DEVICE_DATA(SYNOPSYS, EDDA, NULL) },
> > >  	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_AM654),
> > >  	  .driver_data = (kernel_ulong_t)&am654_data
> > > --
> > > 2.9.5
> > >

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* Re: [PATCH v3 10/11] arm64: dts: layerscape: Add PCIe EP node for ls1088a
  2019-09-03  2:01     ` Xiaowei Bao
@ 2019-09-12 13:01       ` Andrew Murray
  2019-09-14  4:15         ` Xiaowei Bao
  0 siblings, 1 reply; 42+ messages in thread
From: Andrew Murray @ 2019-09-12 13:01 UTC (permalink / raw)
  To: Xiaowei Bao
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, M.h. Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, Leo Li, shawnguo, Mingkai Hu

On Tue, Sep 03, 2019 at 02:01:32AM +0000, Xiaowei Bao wrote:
> 
> 
> > -----Original Message-----
> > From: Andrew Murray <andrew.murray@arm.com>
> > Sent: 2019年9月2日 21:06
> > To: Xiaowei Bao <xiaowei.bao@nxp.com>
> > Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> > Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> > Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> > Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> > gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> > devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> > arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> > <zhiqiang.hou@nxp.com>
> > Subject: Re: [PATCH v3 10/11] arm64: dts: layerscape: Add PCIe EP node for
> > ls1088a
> > 
> > On Mon, Sep 02, 2019 at 11:17:15AM +0800, Xiaowei Bao wrote:
> > > Add PCIe EP node for ls1088a to support EP mode.
> > >
> > > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > ---
> > > v2:
> > >  - Remove the pf-offset proparty.
> > > v3:
> > >  - No change.
> > >
> > >  arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi | 31
> > ++++++++++++++++++++++++++
> > >  1 file changed, 31 insertions(+)
> > >
> > > diff --git a/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
> > b/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
> > > index c676d07..da246ab 100644
> > > --- a/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
> > > +++ b/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
> > > @@ -483,6 +483,17 @@
> > >  			status = "disabled";
> > >  		};
> > >
> > > +		pcie_ep@3400000 {
> > > +			compatible = "fsl,ls1088a-pcie-ep","fsl,ls-pcie-ep";
> > 
> > Here you specify a fallback "fsl,ls-pcie-ep" that is removed by this series.
> > 
> > Besides that, this looks OK.
> 
> As explained, the "fsl,ls-pcie-ep" is needed, due to the u-boot will fixup the status
> property base on this compatible, I think we reserve this compatible is helpfully,
> if delate this compatible, I have to modify the code of bootloader.

I assume you mean that u-boot fixes up "fsl,ls-pcie-ep" *only* for ls1046a
devices?

Thanks,

Andrew Murray

> 
> Thanks 
> XIaowei
> 
> > 
> > Thanks,
> > 
> > Andrew Murray
> > 
> > > +			reg = <0x00 0x03400000 0x0 0x00100000
> > > +			       0x20 0x00000000 0x8 0x00000000>;
> > > +			reg-names = "regs", "addr_space";
> > > +			num-ib-windows = <24>;
> > > +			num-ob-windows = <128>;
> > > +			max-functions = /bits/ 8 <2>;
> > > +			status = "disabled";
> > > +		};
> > > +
> > >  		pcie@3500000 {
> > >  			compatible = "fsl,ls1088a-pcie";
> > >  			reg = <0x00 0x03500000 0x0 0x00100000   /* controller
> > registers */
> > > @@ -508,6 +519,16 @@
> > >  			status = "disabled";
> > >  		};
> > >
> > > +		pcie_ep@3500000 {
> > > +			compatible = "fsl,ls1088a-pcie-ep","fsl,ls-pcie-ep";
> > > +			reg = <0x00 0x03500000 0x0 0x00100000
> > > +			       0x28 0x00000000 0x8 0x00000000>;
> > > +			reg-names = "regs", "addr_space";
> > > +			num-ib-windows = <6>;
> > > +			num-ob-windows = <8>;
> > > +			status = "disabled";
> > > +		};
> > > +
> > >  		pcie@3600000 {
> > >  			compatible = "fsl,ls1088a-pcie";
> > >  			reg = <0x00 0x03600000 0x0 0x00100000   /* controller
> > registers */
> > > @@ -533,6 +554,16 @@
> > >  			status = "disabled";
> > >  		};
> > >
> > > +		pcie_ep@3600000 {
> > > +			compatible = "fsl,ls1088a-pcie-ep","fsl,ls-pcie-ep";
> > > +			reg = <0x00 0x03600000 0x0 0x00100000
> > > +			       0x30 0x00000000 0x8 0x00000000>;
> > > +			reg-names = "regs", "addr_space";
> > > +			num-ib-windows = <6>;
> > > +			num-ob-windows = <8>;
> > > +			status = "disabled";
> > > +		};
> > > +
> > >  		smmu: iommu@5000000 {
> > >  			compatible = "arm,mmu-500";
> > >  			reg = <0 0x5000000 0 0x800000>;
> > > --
> > > 2.9.5
> > >

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 09/11] PCI: layerscape: Add EP mode support for ls1088a and ls2088a
  2019-09-12 12:49       ` Andrew Murray
@ 2019-09-14  4:10         ` Xiaowei Bao
  2019-09-16 14:37           ` Andrew Murray
  0 siblings, 1 reply; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-14  4:10 UTC (permalink / raw)
  To: Andrew Murray
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, M.h. Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, Leo Li, shawnguo, Mingkai Hu



> -----Original Message-----
> From: Andrew Murray <andrew.murray@arm.com>
> Sent: 2019年9月12日 20:50
> To: Xiaowei Bao <xiaowei.bao@nxp.com>
> Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> <zhiqiang.hou@nxp.com>
> Subject: Re: [PATCH v3 09/11] PCI: layerscape: Add EP mode support for
> ls1088a and ls2088a
> 
> On Tue, Sep 03, 2019 at 01:47:36AM +0000, Xiaowei Bao wrote:
> >
> >
> > > -----Original Message-----
> > > From: Andrew Murray <andrew.murray@arm.com>
> > > Sent: 2019年9月2日 20:46
> > > To: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org;
> > > Leo Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> M.h.
> > > Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> > > Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> > > gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> > > devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> > > arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> > > <zhiqiang.hou@nxp.com>
> > > Subject: Re: [PATCH v3 09/11] PCI: layerscape: Add EP mode support
> > > for ls1088a and ls2088a
> > >
> > > On Mon, Sep 02, 2019 at 11:17:14AM +0800, Xiaowei Bao wrote:
> > > > Add PCIe EP mode support for ls1088a and ls2088a, there are some
> > > > difference between LS1 and LS2 platform, so refactor the code of
> > > > the EP driver.
> > > >
> > > > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > > ---
> > > > v2:
> > > >  - This is a new patch for supporting the ls1088a and ls2088a platform.
> > > > v3:
> > > >  - Adjust the some struct assignment order in probe function.
> > > >
> > > >  drivers/pci/controller/dwc/pci-layerscape-ep.c | 72
> > > > +++++++++++++++++++-------
> > > >  1 file changed, 53 insertions(+), 19 deletions(-)
> > > >
> > > > diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > index 5f0cb99..723bbe5 100644
> > > > --- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > +++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > @@ -20,27 +20,29 @@
> > > >
> > > >  #define PCIE_DBI2_OFFSET		0x1000	/* DBI2 base address*/
> > > >
> > > > -struct ls_pcie_ep {
> > > > -	struct dw_pcie		*pci;
> > > > -	struct pci_epc_features	*ls_epc;
> > > > +#define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
> > > > +
> > > > +struct ls_pcie_ep_drvdata {
> > > > +	u32				func_offset;
> > > > +	const struct dw_pcie_ep_ops	*ops;
> > > > +	const struct dw_pcie_ops	*dw_pcie_ops;
> > > >  };
> > > >
> > > > -#define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
> > > > +struct ls_pcie_ep {
> > > > +	struct dw_pcie			*pci;
> > > > +	struct pci_epc_features		*ls_epc;
> > > > +	const struct ls_pcie_ep_drvdata *drvdata; };
> > > >
> > > >  static int ls_pcie_establish_link(struct dw_pcie *pci)  {
> > > >  	return 0;
> > > >  }
> > > >
> > > > -static const struct dw_pcie_ops ls_pcie_ep_ops = {
> > > > +static const struct dw_pcie_ops dw_ls_pcie_ep_ops = {
> > > >  	.start_link = ls_pcie_establish_link,  };
> > > >
> > > > -static const struct of_device_id ls_pcie_ep_of_match[] = {
> > > > -	{ .compatible = "fsl,ls-pcie-ep",},
> > > > -	{ },
> > > > -};
> > > > -
> > > >  static const struct pci_epc_features*
> > > > ls_pcie_ep_get_features(struct dw_pcie_ep *ep)  { @@ -87,10 +89,39
> > > > @@ static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> > > >  	}
> > > >  }
> > > >
> > > > -static const struct dw_pcie_ep_ops pcie_ep_ops = {
> > > > +static unsigned int ls_pcie_ep_func_conf_select(struct dw_pcie_ep
> *ep,
> > > > +						u8 func_no)
> > > > +{
> > > > +	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > > > +	struct ls_pcie_ep *pcie = to_ls_pcie_ep(pci);
> > > > +
> > > > +	WARN_ON(func_no && !pcie->drvdata->func_offset);
> > > > +	return pcie->drvdata->func_offset * func_no; }
> > > > +
> > > > +static const struct dw_pcie_ep_ops ls_pcie_ep_ops = {
> > > >  	.ep_init = ls_pcie_ep_init,
> > > >  	.raise_irq = ls_pcie_ep_raise_irq,
> > > >  	.get_features = ls_pcie_ep_get_features,
> > > > +	.func_conf_select = ls_pcie_ep_func_conf_select, };
> > > > +
> > > > +static const struct ls_pcie_ep_drvdata ls1_ep_drvdata = {
> > > > +	.ops = &ls_pcie_ep_ops,
> > > > +	.dw_pcie_ops = &dw_ls_pcie_ep_ops, };
> > > > +
> > > > +static const struct ls_pcie_ep_drvdata ls2_ep_drvdata = {
> > > > +	.func_offset = 0x20000,
> > > > +	.ops = &ls_pcie_ep_ops,
> > > > +	.dw_pcie_ops = &dw_ls_pcie_ep_ops, };
> > > > +
> > > > +static const struct of_device_id ls_pcie_ep_of_match[] = {
> > > > +	{ .compatible = "fsl,ls1046a-pcie-ep", .data = &ls1_ep_drvdata },
> > > > +	{ .compatible = "fsl,ls1088a-pcie-ep", .data = &ls2_ep_drvdata },
> > > > +	{ .compatible = "fsl,ls2088a-pcie-ep", .data = &ls2_ep_drvdata },
> > > > +	{ },
> > >
> > > This removes support for "fsl,ls-pcie-ep" - was that intentional? If
> > > you do plan to drop it please make sure you explain why in the
> > > commit message. See also my comments in your dt-binding patch.
> >
> > In fact, the u-boot will fixup the status property to 'status =
> > enabled' in PCI node of the DTS base on "fsl,ls-pcie-ep" compatible,
> > so "fsl,ls-pcie-ep" is used, I used this compatible before, because
> > the driver only support the LS1046a, but this time, I add the LS1088a
> > and LS2088a support, and these two boards have some difference form
> LS1046a, so I changed the compatible. I am not sure whether need to add
> "fsl,ls-pcie-ep"
> > in there, could you give some advice, thanks a lot.
> 
> It sounds like "fsl,ls-pcie-ep" can be a fallback for "fsl,ls1046a-pcie-ep".

This is not a fallback, the compatible "fsl,ls1046a-pcie-ep" is used by bootloader,
the bootloader will modify the status property, the bootloader code get the
PCI_HEADER_TYPE(0xe) of config space to decide enable which node(EP or RC)
status property. At the beginning, we plan to use one compatible "fsl,ls1046a-pcie-ep"
support all NXP's platform, but actually, due to the difference of each platform,
it is difficult.

> 
> I'm assuming that if someone used "fsl,ls1046a-pcie-ep" on ls1088a or
> ls2088a hardware it would still work, but without the multiple PF support.
> 

I think the EP driver will not work if use current code, due to the current driver
need driver data. 

> I.e. if "fsl,ls-pcie-ep" is given, treat it as ls1046a.



> 
> Thanks,
> 
> Andrew Murray
> 
> >
> > Thanks
> > Xiaowei
> >
> > >
> > > Thanks,
> > >
> > > Andrew Murray
> > >
> > > >  };
> > > >
> > > >  static int __init ls_add_pcie_ep(struct ls_pcie_ep *pcie, @@
> > > > -103,7
> > > > +134,7 @@ static int __init ls_add_pcie_ep(struct ls_pcie_ep
> > > > +*pcie,
> > > >  	int ret;
> > > >
> > > >  	ep = &pci->ep;
> > > > -	ep->ops = &pcie_ep_ops;
> > > > +	ep->ops = pcie->drvdata->ops;
> > > >
> > > >  	res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> > > "addr_space");
> > > >  	if (!res)
> > > > @@ -142,20 +173,23 @@ static int __init ls_pcie_ep_probe(struct
> > > platform_device *pdev)
> > > >  	if (!ls_epc)
> > > >  		return -ENOMEM;
> > > >
> > > > -	dbi_base = platform_get_resource_byname(pdev,
> IORESOURCE_MEM,
> > > "regs");
> > > > -	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_base);
> > > > -	if (IS_ERR(pci->dbi_base))
> > > > -		return PTR_ERR(pci->dbi_base);
> > > > +	pcie->drvdata = of_device_get_match_data(dev);
> > > >
> > > > -	pci->dbi_base2 = pci->dbi_base + PCIE_DBI2_OFFSET;
> > > >  	pci->dev = dev;
> > > > -	pci->ops = &ls_pcie_ep_ops;
> > > > -	pcie->pci = pci;
> > > > +	pci->ops = pcie->drvdata->dw_pcie_ops;
> > > >
> > > >  	ls_epc->bar_fixed_64bit = (1 << BAR_2) | (1 << BAR_4),
> > > >
> > > > +	pcie->pci = pci;
> > > >  	pcie->ls_epc = ls_epc;
> > > >
> > > > +	dbi_base = platform_get_resource_byname(pdev,
> IORESOURCE_MEM,
> > > "regs");
> > > > +	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_base);
> > > > +	if (IS_ERR(pci->dbi_base))
> > > > +		return PTR_ERR(pci->dbi_base);
> > > > +
> > > > +	pci->dbi_base2 = pci->dbi_base + PCIE_DBI2_OFFSET;
> > > > +
> > > >  	platform_set_drvdata(pdev, pcie);
> > > >
> > > >  	ret = ls_add_pcie_ep(pcie, pdev);
> > > > --
> > > > 2.9.5
> > > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 11/11] misc: pci_endpoint_test: Add LS1088a in pci_device_id table
  2019-09-12 12:59       ` Andrew Murray
@ 2019-09-14  4:13         ` Xiaowei Bao
  0 siblings, 0 replies; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-14  4:13 UTC (permalink / raw)
  To: Andrew Murray, helgaas
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, M.h. Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, Leo Li, shawnguo, Mingkai Hu



> -----Original Message-----
> From: Andrew Murray <andrew.murray@arm.com>
> Sent: 2019年9月12日 21:00
> To: Xiaowei Bao <xiaowei.bao@nxp.com>; helgaas@kernel.org
> Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> <zhiqiang.hou@nxp.com>
> Subject: Re: [PATCH v3 11/11] misc: pci_endpoint_test: Add LS1088a in
> pci_device_id table
> 
> On Tue, Sep 03, 2019 at 01:52:30AM +0000, Xiaowei Bao wrote:
> >
> >
> > > -----Original Message-----
> > > From: Andrew Murray <andrew.murray@arm.com>
> > > Sent: 2019年9月2日 20:55
> > > To: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org;
> > > Leo Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> M.h.
> > > Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> > > Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> > > gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> > > devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> > > arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> > > <zhiqiang.hou@nxp.com>
> > > Subject: Re: [PATCH v3 11/11] misc: pci_endpoint_test: Add LS1088a
> > > in pci_device_id table
> > >
> > > On Mon, Sep 02, 2019 at 11:17:16AM +0800, Xiaowei Bao wrote:
> > > > Add LS1088a in pci_device_id table so that pci-epf-test can be
> > > > used for testing PCIe EP in LS1088a.
> > > >
> > > > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > > ---
> > > > v2:
> > > >  - No change.
> > > > v3:
> > > >  - No change.
> > > >
> > > >  drivers/misc/pci_endpoint_test.c | 1 +
> > > >  1 file changed, 1 insertion(+)
> > > >
> > > > diff --git a/drivers/misc/pci_endpoint_test.c
> > > > b/drivers/misc/pci_endpoint_test.c
> > > > index 6e208a0..d531951 100644
> > > > --- a/drivers/misc/pci_endpoint_test.c
> > > > +++ b/drivers/misc/pci_endpoint_test.c
> > > > @@ -793,6 +793,7 @@ static const struct pci_device_id
> > > pci_endpoint_test_tbl[] = {
> > > >  	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_DRA74x) },
> > > >  	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_DRA72x) },
> > > >  	{ PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, 0x81c0) },
> > > > +	{ PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, 0x80c0) },
> > >
> > > The Freescale PCI devices are the only devices in this table that
> > > don't have a define for their device ID. I think a define should be
> > > created for both of the device IDs above.
> >
> > OK, but I only define in this file, I am not sure this can define in
> > include/linux/pci_ids.h file
> 
> This file seems a little inconsistent...
> 
>  - Two of the TI device IDs are defined in pci_ids.h and only used in
> pci_endpoint_test.c
>  - One of the TI device IDs are defined in pci_endpoint_test.c and only used
> there
>  - The Freescale device ID is hardcoded and only used in pci_endpoint_test.c
> 
> The header in pci_ids.h has a comment suggestion definitions are only added
> where used in multiple files - yet I don't think this holds true.

Thanks Andrew.


> 
> Bjorn - do you have a suggestion?
> 
> Thanks,
> 
> Andrew Murray
> 
> >
> > Thanks
> > Xiaowei
> >
> > >
> > > Thanks,
> > >
> > > Andrew Murray
> > >
> > > >  	{ PCI_DEVICE_DATA(SYNOPSYS, EDDA, NULL) },
> > > >  	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_AM654),
> > > >  	  .driver_data = (kernel_ulong_t)&am654_data
> > > > --
> > > > 2.9.5
> > > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 10/11] arm64: dts: layerscape: Add PCIe EP node for ls1088a
  2019-09-12 13:01       ` Andrew Murray
@ 2019-09-14  4:15         ` Xiaowei Bao
  0 siblings, 0 replies; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-14  4:15 UTC (permalink / raw)
  To: Andrew Murray
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, M.h. Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, Leo Li, shawnguo, Mingkai Hu



> -----Original Message-----
> From: Andrew Murray <andrew.murray@arm.com>
> Sent: 2019年9月12日 21:02
> To: Xiaowei Bao <xiaowei.bao@nxp.com>
> Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> <zhiqiang.hou@nxp.com>
> Subject: Re: [PATCH v3 10/11] arm64: dts: layerscape: Add PCIe EP node for
> ls1088a
> 
> On Tue, Sep 03, 2019 at 02:01:32AM +0000, Xiaowei Bao wrote:
> >
> >
> > > -----Original Message-----
> > > From: Andrew Murray <andrew.murray@arm.com>
> > > Sent: 2019年9月2日 21:06
> > > To: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org;
> > > Leo Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> M.h.
> > > Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> > > Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> > > gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> > > devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> > > arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> > > <zhiqiang.hou@nxp.com>
> > > Subject: Re: [PATCH v3 10/11] arm64: dts: layerscape: Add PCIe EP
> > > node for ls1088a
> > >
> > > On Mon, Sep 02, 2019 at 11:17:15AM +0800, Xiaowei Bao wrote:
> > > > Add PCIe EP node for ls1088a to support EP mode.
> > > >
> > > > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > > ---
> > > > v2:
> > > >  - Remove the pf-offset proparty.
> > > > v3:
> > > >  - No change.
> > > >
> > > >  arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi | 31
> > > ++++++++++++++++++++++++++
> > > >  1 file changed, 31 insertions(+)
> > > >
> > > > diff --git a/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
> > > b/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
> > > > index c676d07..da246ab 100644
> > > > --- a/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
> > > > +++ b/arch/arm64/boot/dts/freescale/fsl-ls1088a.dtsi
> > > > @@ -483,6 +483,17 @@
> > > >  			status = "disabled";
> > > >  		};
> > > >
> > > > +		pcie_ep@3400000 {
> > > > +			compatible = "fsl,ls1088a-pcie-ep","fsl,ls-pcie-ep";
> > >
> > > Here you specify a fallback "fsl,ls-pcie-ep" that is removed by this series.
> > >
> > > Besides that, this looks OK.
> >
> > As explained, the "fsl,ls-pcie-ep" is needed, due to the u-boot will
> > fixup the status property base on this compatible, I think we reserve
> > this compatible is helpfully, if delate this compatible, I have to modify the
> code of bootloader.
> 
> I assume you mean that u-boot fixes up "fsl,ls-pcie-ep" *only* for ls1046a
> devices?

No, all Layerscape platform of NXP.

> 
> Thanks,
> 
> Andrew Murray
> 
> >
> > Thanks
> > XIaowei
> >
> > >
> > > Thanks,
> > >
> > > Andrew Murray
> > >
> > > > +			reg = <0x00 0x03400000 0x0 0x00100000
> > > > +			       0x20 0x00000000 0x8 0x00000000>;
> > > > +			reg-names = "regs", "addr_space";
> > > > +			num-ib-windows = <24>;
> > > > +			num-ob-windows = <128>;
> > > > +			max-functions = /bits/ 8 <2>;
> > > > +			status = "disabled";
> > > > +		};
> > > > +
> > > >  		pcie@3500000 {
> > > >  			compatible = "fsl,ls1088a-pcie";
> > > >  			reg = <0x00 0x03500000 0x0 0x00100000   /* controller
> > > registers */
> > > > @@ -508,6 +519,16 @@
> > > >  			status = "disabled";
> > > >  		};
> > > >
> > > > +		pcie_ep@3500000 {
> > > > +			compatible = "fsl,ls1088a-pcie-ep","fsl,ls-pcie-ep";
> > > > +			reg = <0x00 0x03500000 0x0 0x00100000
> > > > +			       0x28 0x00000000 0x8 0x00000000>;
> > > > +			reg-names = "regs", "addr_space";
> > > > +			num-ib-windows = <6>;
> > > > +			num-ob-windows = <8>;
> > > > +			status = "disabled";
> > > > +		};
> > > > +
> > > >  		pcie@3600000 {
> > > >  			compatible = "fsl,ls1088a-pcie";
> > > >  			reg = <0x00 0x03600000 0x0 0x00100000   /* controller
> > > registers */
> > > > @@ -533,6 +554,16 @@
> > > >  			status = "disabled";
> > > >  		};
> > > >
> > > > +		pcie_ep@3600000 {
> > > > +			compatible = "fsl,ls1088a-pcie-ep","fsl,ls-pcie-ep";
> > > > +			reg = <0x00 0x03600000 0x0 0x00100000
> > > > +			       0x30 0x00000000 0x8 0x00000000>;
> > > > +			reg-names = "regs", "addr_space";
> > > > +			num-ib-windows = <6>;
> > > > +			num-ob-windows = <8>;
> > > > +			status = "disabled";
> > > > +		};
> > > > +
> > > >  		smmu: iommu@5000000 {
> > > >  			compatible = "arm,mmu-500";
> > > >  			reg = <0 0x5000000 0 0x800000>;
> > > > --
> > > > 2.9.5
> > > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 08/11] PCI: layerscape: Modify the MSIX to the doorbell mode
  2019-09-12 11:24     ` Gustavo Pimentel
@ 2019-09-14  6:37       ` Xiaowei Bao
  2019-09-16  8:54         ` Gustavo Pimentel
  0 siblings, 1 reply; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-14  6:37 UTC (permalink / raw)
  To: Gustavo Pimentel, Andrew Murray
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-kernel, kishon,
	M.h. Lian, robh+dt, gregkh, linux-arm-kernel, linux-pci, Leo Li,
	shawnguo, Mingkai Hu



> -----Original Message-----
> From: Gustavo Pimentel <Gustavo.Pimentel@synopsys.com>
> Sent: 2019年9月12日 19:24
> To: Andrew Murray <andrew.murray@arm.com>; Xiaowei Bao
> <xiaowei.bao@nxp.com>
> Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> <zhiqiang.hou@nxp.com>
> Subject: RE: [PATCH v3 08/11] PCI: layerscape: Modify the MSIX to the
> doorbell mode
> 
> Hi,
> 
> Sorry for the delay I was in parental leave and I'm still trying not to drown in
> the mailing list emails... 😊
> 
> On Mon, Sep 2, 2019 at 13:1:47, Andrew Murray <andrew.murray@arm.com>
> wrote:
> 
> > On Mon, Sep 02, 2019 at 11:17:13AM +0800, Xiaowei Bao wrote:
> > > dw_pcie_ep_raise_msix_irq was never called in the exisitng driver
> > > before, because the ls1046a platform don't support the MSIX feature
> > > and msix_capable was always set to false.
> > > Now that add the ls1088a platform with MSIX support, but the
> > > existing dw_pcie_ep_raise_msix_irq doesn't work, so use the doorbell
> > > method to support the MSIX feature.
> 
> Hum... the implementation of msix implementation did work on my use case,
> however, at the time the setup used for developing and testing the
> implementation only had one PF (by default 0). Perhaps this could was is
> causing the different behavior between our setups.
> 
> You have more than one PF, right?

Yes, I have two PFs.

Thanks
Xiaowei

> 
> If I remember correctly, msix feature support entered on kernel 4.19 version
> and it worked quite well at the time, but I didn't test since there (I've to
> manage time to be able to retest it again), I'm didn't seen any patch that
> could interfere with this.
> 
> Regards,
> Gustavo
> 
> 
> > >
> > > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> >
> > Reviewed-by: Andrew Murray <andrew.murray@arm.com>
> >
> > > ---
> > > v2:
> > >  - No change
> > > v3:
> > >  - Modify the commit message make it clearly.
> > >
> > >  drivers/pci/controller/dwc/pci-layerscape-ep.c | 3 ++-
> > >  1 file changed, 2 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > index 1e07287..5f0cb99 100644
> > > --- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > +++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > @@ -79,7 +79,8 @@ static int ls_pcie_ep_raise_irq(struct dw_pcie_ep
> *ep, u8 func_no,
> > >  	case PCI_EPC_IRQ_MSI:
> > >  		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
> > >  	case PCI_EPC_IRQ_MSIX:
> > > -		return dw_pcie_ep_raise_msix_irq(ep, func_no, interrupt_num);
> > > +		return dw_pcie_ep_raise_msix_irq_doorbell(ep, func_no,
> > > +							  interrupt_num);
> > >  	default:
> > >  		dev_err(pci->dev, "UNKNOWN IRQ type\n");
> > >  		return -EINVAL;
> > > --
> > > 2.9.5
> > >
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 08/11] PCI: layerscape: Modify the MSIX to the doorbell mode
  2019-09-14  6:37       ` Xiaowei Bao
@ 2019-09-16  8:54         ` Gustavo Pimentel
  0 siblings, 0 replies; 42+ messages in thread
From: Gustavo Pimentel @ 2019-09-16  8:54 UTC (permalink / raw)
  To: Xiaowei Bao, Gustavo Pimentel, Andrew Murray
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-kernel, kishon,
	M.h. Lian, robh+dt, gregkh, linux-arm-kernel, linux-pci, Leo Li,
	shawnguo, Mingkai Hu

On Sat, Sep 14, 2019 at 7:37:54, Xiaowei Bao <xiaowei.bao@nxp.com> wrote:

> 
> 
> > -----Original Message-----
> > From: Gustavo Pimentel <Gustavo.Pimentel@synopsys.com>
> > Sent: 2019年9月12日 19:24
> > To: Andrew Murray <andrew.murray@arm.com>; Xiaowei Bao
> > <xiaowei.bao@nxp.com>
> > Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> > Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> > Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> > Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> > gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> > devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> > arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> > <zhiqiang.hou@nxp.com>
> > Subject: RE: [PATCH v3 08/11] PCI: layerscape: Modify the MSIX to the
> > doorbell mode
> > 
> > Hi,
> > 
> > Sorry for the delay I was in parental leave and I'm still trying not to drown in
> > the mailing list emails... 😊
> > 
> > On Mon, Sep 2, 2019 at 13:1:47, Andrew Murray <andrew.murray@arm.com>
> > wrote:
> > 
> > > On Mon, Sep 02, 2019 at 11:17:13AM +0800, Xiaowei Bao wrote:
> > > > dw_pcie_ep_raise_msix_irq was never called in the exisitng driver
> > > > before, because the ls1046a platform don't support the MSIX feature
> > > > and msix_capable was always set to false.
> > > > Now that add the ls1088a platform with MSIX support, but the
> > > > existing dw_pcie_ep_raise_msix_irq doesn't work, so use the doorbell
> > > > method to support the MSIX feature.
> > 
> > Hum... the implementation of msix implementation did work on my use case,
> > however, at the time the setup used for developing and testing the
> > implementation only had one PF (by default 0). Perhaps this could was is
> > causing the different behavior between our setups.
> > 
> > You have more than one PF, right?
> 
> Yes, I have two PFs.

Probably that's the reason why my MSI-X raise function implementation 
didn't work on your case.

> 
> Thanks
> Xiaowei
> 
> > 
> > If I remember correctly, msix feature support entered on kernel 4.19 version
> > and it worked quite well at the time, but I didn't test since there (I've to
> > manage time to be able to retest it again), I'm didn't seen any patch that
> > could interfere with this.
> > 
> > Regards,
> > Gustavo
> > 
> > 
> > > >
> > > > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > >
> > > Reviewed-by: Andrew Murray <andrew.murray@arm.com>
> > >
> > > > ---
> > > > v2:
> > > >  - No change
> > > > v3:
> > > >  - Modify the commit message make it clearly.
> > > >
> > > >  drivers/pci/controller/dwc/pci-layerscape-ep.c | 3 ++-
> > > >  1 file changed, 2 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > index 1e07287..5f0cb99 100644
> > > > --- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > +++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > @@ -79,7 +79,8 @@ static int ls_pcie_ep_raise_irq(struct dw_pcie_ep
> > *ep, u8 func_no,
> > > >  	case PCI_EPC_IRQ_MSI:
> > > >  		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
> > > >  	case PCI_EPC_IRQ_MSIX:
> > > > -		return dw_pcie_ep_raise_msix_irq(ep, func_no, interrupt_num);
> > > > +		return dw_pcie_ep_raise_msix_irq_doorbell(ep, func_no,
> > > > +							  interrupt_num);
> > > >  	default:
> > > >  		dev_err(pci->dev, "UNKNOWN IRQ type\n");
> > > >  		return -EINVAL;
> > > > --
> > > > 2.9.5
> > > >
> > 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* Re: [PATCH v3 09/11] PCI: layerscape: Add EP mode support for ls1088a and ls2088a
  2019-09-14  4:10         ` Xiaowei Bao
@ 2019-09-16 14:37           ` Andrew Murray
  2019-09-18  3:17             ` Xiaowei Bao
  0 siblings, 1 reply; 42+ messages in thread
From: Andrew Murray @ 2019-09-16 14:37 UTC (permalink / raw)
  To: Xiaowei Bao, robh+dt
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, M.h. Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, Leo Li, shawnguo, Mingkai Hu

On Sat, Sep 14, 2019 at 04:10:22AM +0000, Xiaowei Bao wrote:
> 
> 
> > -----Original Message-----
> > From: Andrew Murray <andrew.murray@arm.com>
> > Sent: 2019年9月12日 20:50
> > To: Xiaowei Bao <xiaowei.bao@nxp.com>
> > Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> > Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> > Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> > Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> > gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> > devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> > arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> > <zhiqiang.hou@nxp.com>
> > Subject: Re: [PATCH v3 09/11] PCI: layerscape: Add EP mode support for
> > ls1088a and ls2088a
> > 
> > On Tue, Sep 03, 2019 at 01:47:36AM +0000, Xiaowei Bao wrote:
> > >
> > >
> > > > -----Original Message-----
> > > > From: Andrew Murray <andrew.murray@arm.com>
> > > > Sent: 2019年9月2日 20:46
> > > > To: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > > Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org;
> > > > Leo Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > M.h.
> > > > Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> > > > Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> > > > gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> > > > devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > > linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> > > > arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> > > > <zhiqiang.hou@nxp.com>
> > > > Subject: Re: [PATCH v3 09/11] PCI: layerscape: Add EP mode support
> > > > for ls1088a and ls2088a
> > > >
> > > > On Mon, Sep 02, 2019 at 11:17:14AM +0800, Xiaowei Bao wrote:
> > > > > Add PCIe EP mode support for ls1088a and ls2088a, there are some
> > > > > difference between LS1 and LS2 platform, so refactor the code of
> > > > > the EP driver.
> > > > >
> > > > > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > > > ---
> > > > > v2:
> > > > >  - This is a new patch for supporting the ls1088a and ls2088a platform.
> > > > > v3:
> > > > >  - Adjust the some struct assignment order in probe function.
> > > > >
> > > > >  drivers/pci/controller/dwc/pci-layerscape-ep.c | 72
> > > > > +++++++++++++++++++-------
> > > > >  1 file changed, 53 insertions(+), 19 deletions(-)
> > > > >
> > > > > diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > > b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > > index 5f0cb99..723bbe5 100644
> > > > > --- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > > +++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > > @@ -20,27 +20,29 @@
> > > > >
> > > > >  #define PCIE_DBI2_OFFSET		0x1000	/* DBI2 base address*/
> > > > >
> > > > > -struct ls_pcie_ep {
> > > > > -	struct dw_pcie		*pci;
> > > > > -	struct pci_epc_features	*ls_epc;
> > > > > +#define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
> > > > > +
> > > > > +struct ls_pcie_ep_drvdata {
> > > > > +	u32				func_offset;
> > > > > +	const struct dw_pcie_ep_ops	*ops;
> > > > > +	const struct dw_pcie_ops	*dw_pcie_ops;
> > > > >  };
> > > > >
> > > > > -#define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
> > > > > +struct ls_pcie_ep {
> > > > > +	struct dw_pcie			*pci;
> > > > > +	struct pci_epc_features		*ls_epc;
> > > > > +	const struct ls_pcie_ep_drvdata *drvdata; };
> > > > >
> > > > >  static int ls_pcie_establish_link(struct dw_pcie *pci)  {
> > > > >  	return 0;
> > > > >  }
> > > > >
> > > > > -static const struct dw_pcie_ops ls_pcie_ep_ops = {
> > > > > +static const struct dw_pcie_ops dw_ls_pcie_ep_ops = {
> > > > >  	.start_link = ls_pcie_establish_link,  };
> > > > >
> > > > > -static const struct of_device_id ls_pcie_ep_of_match[] = {
> > > > > -	{ .compatible = "fsl,ls-pcie-ep",},
> > > > > -	{ },
> > > > > -};
> > > > > -
> > > > >  static const struct pci_epc_features*
> > > > > ls_pcie_ep_get_features(struct dw_pcie_ep *ep)  { @@ -87,10 +89,39
> > > > > @@ static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> > > > >  	}
> > > > >  }
> > > > >
> > > > > -static const struct dw_pcie_ep_ops pcie_ep_ops = {
> > > > > +static unsigned int ls_pcie_ep_func_conf_select(struct dw_pcie_ep
> > *ep,
> > > > > +						u8 func_no)
> > > > > +{
> > > > > +	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > > > > +	struct ls_pcie_ep *pcie = to_ls_pcie_ep(pci);
> > > > > +
> > > > > +	WARN_ON(func_no && !pcie->drvdata->func_offset);
> > > > > +	return pcie->drvdata->func_offset * func_no; }
> > > > > +
> > > > > +static const struct dw_pcie_ep_ops ls_pcie_ep_ops = {
> > > > >  	.ep_init = ls_pcie_ep_init,
> > > > >  	.raise_irq = ls_pcie_ep_raise_irq,
> > > > >  	.get_features = ls_pcie_ep_get_features,
> > > > > +	.func_conf_select = ls_pcie_ep_func_conf_select, };
> > > > > +
> > > > > +static const struct ls_pcie_ep_drvdata ls1_ep_drvdata = {
> > > > > +	.ops = &ls_pcie_ep_ops,
> > > > > +	.dw_pcie_ops = &dw_ls_pcie_ep_ops, };
> > > > > +
> > > > > +static const struct ls_pcie_ep_drvdata ls2_ep_drvdata = {
> > > > > +	.func_offset = 0x20000,
> > > > > +	.ops = &ls_pcie_ep_ops,
> > > > > +	.dw_pcie_ops = &dw_ls_pcie_ep_ops, };
> > > > > +
> > > > > +static const struct of_device_id ls_pcie_ep_of_match[] = {
> > > > > +	{ .compatible = "fsl,ls1046a-pcie-ep", .data = &ls1_ep_drvdata },
> > > > > +	{ .compatible = "fsl,ls1088a-pcie-ep", .data = &ls2_ep_drvdata },
> > > > > +	{ .compatible = "fsl,ls2088a-pcie-ep", .data = &ls2_ep_drvdata },
> > > > > +	{ },
> > > >
> > > > This removes support for "fsl,ls-pcie-ep" - was that intentional? If
> > > > you do plan to drop it please make sure you explain why in the
> > > > commit message. See also my comments in your dt-binding patch.
> > >
> > > In fact, the u-boot will fixup the status property to 'status =
> > > enabled' in PCI node of the DTS base on "fsl,ls-pcie-ep" compatible,
> > > so "fsl,ls-pcie-ep" is used, I used this compatible before, because
> > > the driver only support the LS1046a, but this time, I add the LS1088a
> > > and LS2088a support, and these two boards have some difference form
> > LS1046a, so I changed the compatible. I am not sure whether need to add
> > "fsl,ls-pcie-ep"
> > > in there, could you give some advice, thanks a lot.
> > 
> > It sounds like "fsl,ls-pcie-ep" can be a fallback for "fsl,ls1046a-pcie-ep".
> 
> This is not a fallback, the compatible "fsl,ls1046a-pcie-ep" is used by bootloader,
> the bootloader will modify the status property, the bootloader code get the
> PCI_HEADER_TYPE(0xe) of config space to decide enable which node(EP or RC)
> status property. At the beginning, we plan to use one compatible "fsl,ls1046a-pcie-ep"
> support all NXP's platform, but actually, due to the difference of each platform,
> it is difficult.

I've looked at the U-Boot source [1] and device trees, I think I understand
what happens here.

The DT describes disabled nodes for both fsl,lsXXXXX-pcie and
fsl,lxXXXXX-pcie-ep. U-Boot looks at the nodes and compares with the actual PCI
config space to determine the current hardware configuration type. It will
then *enable* either the RC or EP.

However U-Boot currently only looks for a compatible string with "fsl,ls-pcie"
or "fsl,ls-pcie-ep". This is why the DT needs to describe a PCI node as both
"fsl,lsXXXXX-pcie-ep" and "fsl,ls-pcie" - the first for kernel and the second
for the U-Boot. (The second is no longer needed by the kernel driver as you
are now using the more specific names).

Looking again at your bindings patch...

diff --git a/Documentation/devicetree/bindings/pci/layerscape-pci.txt b/Documentation/devicetree/bindings/pci/layerscape-pci.txt
index e20ceaa..762ae41 100644
--- a/Documentation/devicetree/bindings/pci/layerscape-pci.txt
+++ b/Documentation/devicetree/bindings/pci/layerscape-pci.txt
@@ -22,7 +22,9 @@  Required properties:
         "fsl,ls1043a-pcie"
         "fsl,ls1012a-pcie"
   EP mode:
-	"fsl,ls1046a-pcie-ep", "fsl,ls-pcie-ep"
+	"fsl,ls1046a-pcie-ep" "fsl,ls-pcie-ep"
+	"fsl,ls1088a-pcie-ep" "fsl,ls-pcie-ep"
+	"fsl,ls2088a-pcie-ep" "fsl,ls-pcie-ep"

... the "fsl,ls-pcie-ep" is added *only* to the EP mode.

But doesn't U-Boot need "fsl,ls-pcie-ep" added to each of the RC modes as
well, to ensure they are set to enabled before booting the kernel?

Rob - Do we document compatible names like this that are used in the DT
but not used by the kernel?

In any case, prior to this series it would have been possible to use a
ls1046a device with a DT that has only string "fsl,ls-pcie-ep" - now that
doesn't work. If this is of concern then &ls1_ep_drvdata should also be
used for fsl,ls-pcie-ep.

Thanks,

Andrew Murray

[1] https://gitlab.denx.de/u-boot/u-boot/blob/master/drivers/pci/pcie_layerscape_fixup.c

> 
> > 
> > I'm assuming that if someone used "fsl,ls1046a-pcie-ep" on ls1088a or
> > ls2088a hardware it would still work, but without the multiple PF support.
> > 
> 
> I think the EP driver will not work if use current code, due to the current driver
> need driver data. 
> 
> > I.e. if "fsl,ls-pcie-ep" is given, treat it as ls1046a.
> 
> 
> 
> > 
> > Thanks,
> > 
> > Andrew Murray
> > 
> > >
> > > Thanks
> > > Xiaowei
> > >
> > > >
> > > > Thanks,
> > > >
> > > > Andrew Murray
> > > >
> > > > >  };
> > > > >
> > > > >  static int __init ls_add_pcie_ep(struct ls_pcie_ep *pcie, @@
> > > > > -103,7
> > > > > +134,7 @@ static int __init ls_add_pcie_ep(struct ls_pcie_ep
> > > > > +*pcie,
> > > > >  	int ret;
> > > > >
> > > > >  	ep = &pci->ep;
> > > > > -	ep->ops = &pcie_ep_ops;
> > > > > +	ep->ops = pcie->drvdata->ops;
> > > > >
> > > > >  	res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> > > > "addr_space");
> > > > >  	if (!res)
> > > > > @@ -142,20 +173,23 @@ static int __init ls_pcie_ep_probe(struct
> > > > platform_device *pdev)
> > > > >  	if (!ls_epc)
> > > > >  		return -ENOMEM;
> > > > >
> > > > > -	dbi_base = platform_get_resource_byname(pdev,
> > IORESOURCE_MEM,
> > > > "regs");
> > > > > -	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_base);
> > > > > -	if (IS_ERR(pci->dbi_base))
> > > > > -		return PTR_ERR(pci->dbi_base);
> > > > > +	pcie->drvdata = of_device_get_match_data(dev);
> > > > >
> > > > > -	pci->dbi_base2 = pci->dbi_base + PCIE_DBI2_OFFSET;
> > > > >  	pci->dev = dev;
> > > > > -	pci->ops = &ls_pcie_ep_ops;
> > > > > -	pcie->pci = pci;
> > > > > +	pci->ops = pcie->drvdata->dw_pcie_ops;
> > > > >
> > > > >  	ls_epc->bar_fixed_64bit = (1 << BAR_2) | (1 << BAR_4),
> > > > >
> > > > > +	pcie->pci = pci;
> > > > >  	pcie->ls_epc = ls_epc;
> > > > >
> > > > > +	dbi_base = platform_get_resource_byname(pdev,
> > IORESOURCE_MEM,
> > > > "regs");
> > > > > +	pci->dbi_base = devm_pci_remap_cfg_resource(dev, dbi_base);
> > > > > +	if (IS_ERR(pci->dbi_base))
> > > > > +		return PTR_ERR(pci->dbi_base);
> > > > > +
> > > > > +	pci->dbi_base2 = pci->dbi_base + PCIE_DBI2_OFFSET;
> > > > > +
> > > > >  	platform_set_drvdata(pdev, pcie);
> > > > >
> > > > >  	ret = ls_add_pcie_ep(pcie, pdev);
> > > > > --
> > > > > 2.9.5
> > > > >

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 09/11] PCI: layerscape: Add EP mode support for ls1088a and ls2088a
  2019-09-16 14:37           ` Andrew Murray
@ 2019-09-18  3:17             ` Xiaowei Bao
  0 siblings, 0 replies; 42+ messages in thread
From: Xiaowei Bao @ 2019-09-18  3:17 UTC (permalink / raw)
  To: Andrew Murray, robh+dt
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	jingoohan1, Z.q. Hou, linuxppc-dev, linux-pci, linux-kernel,
	kishon, M.h. Lian, robh+dt, gregkh, linux-arm-kernel,
	gustavo.pimentel, Leo Li, shawnguo, Mingkai Hu



> -----Original Message-----
> From: Andrew Murray <andrew.murray@arm.com>
> Sent: 2019年9月16日 22:38
> To: Xiaowei Bao <xiaowei.bao@nxp.com>; robh+dt@kernel.org
> Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> <zhiqiang.hou@nxp.com>
> Subject: Re: [PATCH v3 09/11] PCI: layerscape: Add EP mode support for
> ls1088a and ls2088a
> 
> On Sat, Sep 14, 2019 at 04:10:22AM +0000, Xiaowei Bao wrote:
> >
> >
> > > -----Original Message-----
> > > From: Andrew Murray <andrew.murray@arm.com>
> > > Sent: 2019年9月12日 20:50
> > > To: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org;
> > > Leo Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> M.h.
> > > Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> > > Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> > > gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> > > devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> > > arnd@arndb.de; gregkh@linuxfoundation.org; Z.q. Hou
> > > <zhiqiang.hou@nxp.com>
> > > Subject: Re: [PATCH v3 09/11] PCI: layerscape: Add EP mode support
> > > for ls1088a and ls2088a
> > >
> > > On Tue, Sep 03, 2019 at 01:47:36AM +0000, Xiaowei Bao wrote:
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Andrew Murray <andrew.murray@arm.com>
> > > > > Sent: 2019年9月2日 20:46
> > > > > To: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > > > Cc: robh+dt@kernel.org; mark.rutland@arm.com;
> > > > > shawnguo@kernel.org; Leo Li <leoyang.li@nxp.com>; kishon@ti.com;
> > > > > lorenzo.pieralisi@arm.com;
> > > M.h.
> > > > > Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>;
> > > > > Roy Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> > > > > gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> > > > > devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > > > linux-arm-kernel@lists.infradead.org;
> > > > > linuxppc-dev@lists.ozlabs.org; arnd@arndb.de;
> > > > > gregkh@linuxfoundation.org; Z.q. Hou <zhiqiang.hou@nxp.com>
> > > > > Subject: Re: [PATCH v3 09/11] PCI: layerscape: Add EP mode
> > > > > support for ls1088a and ls2088a
> > > > >
> > > > > On Mon, Sep 02, 2019 at 11:17:14AM +0800, Xiaowei Bao wrote:
> > > > > > Add PCIe EP mode support for ls1088a and ls2088a, there are
> > > > > > some difference between LS1 and LS2 platform, so refactor the
> > > > > > code of the EP driver.
> > > > > >
> > > > > > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > > > > ---
> > > > > > v2:
> > > > > >  - This is a new patch for supporting the ls1088a and ls2088a
> platform.
> > > > > > v3:
> > > > > >  - Adjust the some struct assignment order in probe function.
> > > > > >
> > > > > >  drivers/pci/controller/dwc/pci-layerscape-ep.c | 72
> > > > > > +++++++++++++++++++-------
> > > > > >  1 file changed, 53 insertions(+), 19 deletions(-)
> > > > > >
> > > > > > diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > > > b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > > > index 5f0cb99..723bbe5 100644
> > > > > > --- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > > > +++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > > > > > @@ -20,27 +20,29 @@
> > > > > >
> > > > > >  #define PCIE_DBI2_OFFSET		0x1000	/* DBI2 base address*/
> > > > > >
> > > > > > -struct ls_pcie_ep {
> > > > > > -	struct dw_pcie		*pci;
> > > > > > -	struct pci_epc_features	*ls_epc;
> > > > > > +#define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
> > > > > > +
> > > > > > +struct ls_pcie_ep_drvdata {
> > > > > > +	u32				func_offset;
> > > > > > +	const struct dw_pcie_ep_ops	*ops;
> > > > > > +	const struct dw_pcie_ops	*dw_pcie_ops;
> > > > > >  };
> > > > > >
> > > > > > -#define to_ls_pcie_ep(x)	dev_get_drvdata((x)->dev)
> > > > > > +struct ls_pcie_ep {
> > > > > > +	struct dw_pcie			*pci;
> > > > > > +	struct pci_epc_features		*ls_epc;
> > > > > > +	const struct ls_pcie_ep_drvdata *drvdata; };
> > > > > >
> > > > > >  static int ls_pcie_establish_link(struct dw_pcie *pci)  {
> > > > > >  	return 0;
> > > > > >  }
> > > > > >
> > > > > > -static const struct dw_pcie_ops ls_pcie_ep_ops = {
> > > > > > +static const struct dw_pcie_ops dw_ls_pcie_ep_ops = {
> > > > > >  	.start_link = ls_pcie_establish_link,  };
> > > > > >
> > > > > > -static const struct of_device_id ls_pcie_ep_of_match[] = {
> > > > > > -	{ .compatible = "fsl,ls-pcie-ep",},
> > > > > > -	{ },
> > > > > > -};
> > > > > > -
> > > > > >  static const struct pci_epc_features*
> > > > > > ls_pcie_ep_get_features(struct dw_pcie_ep *ep)  { @@ -87,10
> > > > > > +89,39 @@ static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep,
> u8 func_no,
> > > > > >  	}
> > > > > >  }
> > > > > >
> > > > > > -static const struct dw_pcie_ep_ops pcie_ep_ops = {
> > > > > > +static unsigned int ls_pcie_ep_func_conf_select(struct
> > > > > > +dw_pcie_ep
> > > *ep,
> > > > > > +						u8 func_no)
> > > > > > +{
> > > > > > +	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > > > > > +	struct ls_pcie_ep *pcie = to_ls_pcie_ep(pci);
> > > > > > +
> > > > > > +	WARN_ON(func_no && !pcie->drvdata->func_offset);
> > > > > > +	return pcie->drvdata->func_offset * func_no; }
> > > > > > +
> > > > > > +static const struct dw_pcie_ep_ops ls_pcie_ep_ops = {
> > > > > >  	.ep_init = ls_pcie_ep_init,
> > > > > >  	.raise_irq = ls_pcie_ep_raise_irq,
> > > > > >  	.get_features = ls_pcie_ep_get_features,
> > > > > > +	.func_conf_select = ls_pcie_ep_func_conf_select, };
> > > > > > +
> > > > > > +static const struct ls_pcie_ep_drvdata ls1_ep_drvdata = {
> > > > > > +	.ops = &ls_pcie_ep_ops,
> > > > > > +	.dw_pcie_ops = &dw_ls_pcie_ep_ops, };
> > > > > > +
> > > > > > +static const struct ls_pcie_ep_drvdata ls2_ep_drvdata = {
> > > > > > +	.func_offset = 0x20000,
> > > > > > +	.ops = &ls_pcie_ep_ops,
> > > > > > +	.dw_pcie_ops = &dw_ls_pcie_ep_ops, };
> > > > > > +
> > > > > > +static const struct of_device_id ls_pcie_ep_of_match[] = {
> > > > > > +	{ .compatible = "fsl,ls1046a-pcie-ep", .data =
> &ls1_ep_drvdata },
> > > > > > +	{ .compatible = "fsl,ls1088a-pcie-ep", .data =
> &ls2_ep_drvdata },
> > > > > > +	{ .compatible = "fsl,ls2088a-pcie-ep", .data =
> &ls2_ep_drvdata },
> > > > > > +	{ },
> > > > >
> > > > > This removes support for "fsl,ls-pcie-ep" - was that
> > > > > intentional? If you do plan to drop it please make sure you
> > > > > explain why in the commit message. See also my comments in your
> dt-binding patch.
> > > >
> > > > In fact, the u-boot will fixup the status property to 'status =
> > > > enabled' in PCI node of the DTS base on "fsl,ls-pcie-ep"
> > > > compatible, so "fsl,ls-pcie-ep" is used, I used this compatible
> > > > before, because the driver only support the LS1046a, but this
> > > > time, I add the LS1088a and LS2088a support, and these two boards
> > > > have some difference form
> > > LS1046a, so I changed the compatible. I am not sure whether need to
> > > add "fsl,ls-pcie-ep"
> > > > in there, could you give some advice, thanks a lot.
> > >
> > > It sounds like "fsl,ls-pcie-ep" can be a fallback for "fsl,ls1046a-pcie-ep".
> >
> > This is not a fallback, the compatible "fsl,ls1046a-pcie-ep" is used
> > by bootloader, the bootloader will modify the status property, the
> > bootloader code get the
> > PCI_HEADER_TYPE(0xe) of config space to decide enable which node(EP or
> > RC) status property. At the beginning, we plan to use one compatible
> "fsl,ls1046a-pcie-ep"
> > support all NXP's platform, but actually, due to the difference of
> > each platform, it is difficult.
> 
> I've looked at the U-Boot source [1] and device trees, I think I understand
> what happens here.
> 
> The DT describes disabled nodes for both fsl,lsXXXXX-pcie and
> fsl,lxXXXXX-pcie-ep. U-Boot looks at the nodes and compares with the actual
> PCI config space to determine the current hardware configuration type. It will
> then *enable* either the RC or EP.

Yes, you are correct.

> 
> However U-Boot currently only looks for a compatible string with "fsl,ls-pcie"
> or "fsl,ls-pcie-ep". This is why the DT needs to describe a PCI node as both
> "fsl,lsXXXXX-pcie-ep" and "fsl,ls-pcie" - the first for kernel and the second for
> the U-Boot. (The second is no longer needed by the kernel driver as you are
> now using the more specific names).

The fact is that the compatible "fsl,ls-pcie-ep" is used for match the driver at first,
and community of opensource advice reserve the compatible "fsl,lsXXXXX-pcie-ep"
for other platforms, at present, the compatible "fsl,ls-pcie-ep" is also used to 
match the driver, and "fsl,lsXXXXX-pcie-ep" used to difference the platforms, please
refer to the drivers/pci/controller/dwc/pci-layerscape.c, it is similar.
 



> 
> Looking again at your bindings patch...
> 
> diff --git a/Documentation/devicetree/bindings/pci/layerscape-pci.txt
> b/Documentation/devicetree/bindings/pci/layerscape-pci.txt
> index e20ceaa..762ae41 100644
> --- a/Documentation/devicetree/bindings/pci/layerscape-pci.txt
> +++ b/Documentation/devicetree/bindings/pci/layerscape-pci.txt
> @@ -22,7 +22,9 @@  Required properties:
>          "fsl,ls1043a-pcie"
>          "fsl,ls1012a-pcie"
>    EP mode:
> -	"fsl,ls1046a-pcie-ep", "fsl,ls-pcie-ep"
> +	"fsl,ls1046a-pcie-ep" "fsl,ls-pcie-ep"
> +	"fsl,ls1088a-pcie-ep" "fsl,ls-pcie-ep"
> +	"fsl,ls2088a-pcie-ep" "fsl,ls-pcie-ep"
> 
> ... the "fsl,ls-pcie-ep" is added *only* to the EP mode.
> 
> But doesn't U-Boot need "fsl,ls-pcie-ep" added to each of the RC modes as
> well, to ensure they are set to enabled before booting the kernel?
> 
> Rob - Do we document compatible names like this that are used in the DT but
> not used by the kernel?
> 
> In any case, prior to this series it would have been possible to use a ls1046a
> device with a DT that has only string "fsl,ls-pcie-ep" - now that doesn't work.
> If this is of concern then &ls1_ep_drvdata should also be used for
> fsl,ls-pcie-ep.
> 
> Thanks,
> 
> Andrew Murray
> 
> [1]
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.
> denx.de%2Fu-boot%2Fu-boot%2Fblob%2Fmaster%2Fdrivers%2Fpci%2Fpcie_l
> ayerscape_fixup.c&amp;data=02%7C01%7Cxiaowei.bao%40nxp.com%7Ccf3
> 3c1b76cb74f1cb41308d73ab37330%7C686ea1d3bc2b4c6fa92cd99c5c30163
> 5%7C0%7C1%7C637042414749368155&amp;sdata=uVvfeT8AnOlPS%2ByBZ9
> Y4%2BHNJIA6MPx2sfWUBiD75IRY%3D&amp;reserved=0
> 
> >
> > >
> > > I'm assuming that if someone used "fsl,ls1046a-pcie-ep" on ls1088a
> > > or ls2088a hardware it would still work, but without the multiple PF
> support.
> > >
> >
> > I think the EP driver will not work if use current code, due to the
> > current driver need driver data.
> >
> > > I.e. if "fsl,ls-pcie-ep" is given, treat it as ls1046a.
> >
> >
> >
> > >
> > > Thanks,
> > >
> > > Andrew Murray
> > >
> > > >
> > > > Thanks
> > > > Xiaowei
> > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Andrew Murray
> > > > >
> > > > > >  };
> > > > > >
> > > > > >  static int __init ls_add_pcie_ep(struct ls_pcie_ep *pcie, @@
> > > > > > -103,7
> > > > > > +134,7 @@ static int __init ls_add_pcie_ep(struct ls_pcie_ep
> > > > > > +*pcie,
> > > > > >  	int ret;
> > > > > >
> > > > > >  	ep = &pci->ep;
> > > > > > -	ep->ops = &pcie_ep_ops;
> > > > > > +	ep->ops = pcie->drvdata->ops;
> > > > > >
> > > > > >  	res = platform_get_resource_byname(pdev,
> IORESOURCE_MEM,
> > > > > "addr_space");
> > > > > >  	if (!res)
> > > > > > @@ -142,20 +173,23 @@ static int __init
> > > > > > ls_pcie_ep_probe(struct
> > > > > platform_device *pdev)
> > > > > >  	if (!ls_epc)
> > > > > >  		return -ENOMEM;
> > > > > >
> > > > > > -	dbi_base = platform_get_resource_byname(pdev,
> > > IORESOURCE_MEM,
> > > > > "regs");
> > > > > > -	pci->dbi_base = devm_pci_remap_cfg_resource(dev,
> dbi_base);
> > > > > > -	if (IS_ERR(pci->dbi_base))
> > > > > > -		return PTR_ERR(pci->dbi_base);
> > > > > > +	pcie->drvdata = of_device_get_match_data(dev);
> > > > > >
> > > > > > -	pci->dbi_base2 = pci->dbi_base + PCIE_DBI2_OFFSET;
> > > > > >  	pci->dev = dev;
> > > > > > -	pci->ops = &ls_pcie_ep_ops;
> > > > > > -	pcie->pci = pci;
> > > > > > +	pci->ops = pcie->drvdata->dw_pcie_ops;
> > > > > >
> > > > > >  	ls_epc->bar_fixed_64bit = (1 << BAR_2) | (1 << BAR_4),
> > > > > >
> > > > > > +	pcie->pci = pci;
> > > > > >  	pcie->ls_epc = ls_epc;
> > > > > >
> > > > > > +	dbi_base = platform_get_resource_byname(pdev,
> > > IORESOURCE_MEM,
> > > > > "regs");
> > > > > > +	pci->dbi_base = devm_pci_remap_cfg_resource(dev,
> dbi_base);
> > > > > > +	if (IS_ERR(pci->dbi_base))
> > > > > > +		return PTR_ERR(pci->dbi_base);
> > > > > > +
> > > > > > +	pci->dbi_base2 = pci->dbi_base + PCIE_DBI2_OFFSET;
> > > > > > +
> > > > > >  	platform_set_drvdata(pdev, pcie);
> > > > > >
> > > > > >  	ret = ls_add_pcie_ep(pcie, pdev);
> > > > > > --
> > > > > > 2.9.5
> > > > > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* Re: [PATCH v3 01/11] PCI: designware-ep: Add multiple PFs support for DWC
  2019-09-03  3:43     ` Xiaowei Bao
@ 2019-09-26 10:29       ` Andrew Murray
  2019-09-26 13:38         ` Gustavo Pimentel
  0 siblings, 1 reply; 42+ messages in thread
From: Andrew Murray @ 2019-09-26 10:29 UTC (permalink / raw)
  To: Xiaowei Bao, jingoohan1, gustavo.pimentel
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	linux-pci, Z.q. Hou, linuxppc-dev, linux-kernel, kishon,
	M.h. Lian, robh+dt, linux-arm-kernel, gregkh, Leo Li, shawnguo,
	Mingkai Hu

On Tue, Sep 03, 2019 at 03:43:15AM +0000, Xiaowei Bao wrote:
> 
> 
> > -----Original Message-----
> > From: Andrew Murray <andrew.murray@arm.com>
> > Sent: 2019年9月3日 0:26
> > To: Xiaowei Bao <xiaowei.bao@nxp.com>
> > Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> > Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> > Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> > Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> > gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> > devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> > linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> > gregkh@linuxfoundation.org; Z.q. Hou <zhiqiang.hou@nxp.com>;
> > arnd@arndb.de
> > Subject: Re: [PATCH v3 01/11] PCI: designware-ep: Add multiple PFs support
> > for DWC
> > 
> > On Mon, Sep 02, 2019 at 11:17:06AM +0800, Xiaowei Bao wrote:
> > > Add multiple PFs support for DWC, different PF have different config
> > > space we use pf-offset property which get from the DTS to access the
> > > different pF
> > 
> > This needs to be updated as this no longer comes from the DT.
> 
> Yes, thanks
> 
> Thanks
> Xiaowei
> 
> > 
> > > config space.
> > >
> > > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > 
> > 
> > We're assuming:
> > 
> >  - The offset address (func_offset) between PF's in the memory map can be
> >    different between different DWC implementations. And also that it's
> >    possible for DWC implementations to address PFs without using an offset.
> > 
> >  - The current approach is preferable to adding DWC EP driver callbacks
> >    for writing to the EP config space (e.g. a variant of dw_pcie_writew_dbi
> >    that takes a func number).
> 
> Even if use the a variant of dw_pcie_writew_dbi, we also need a offset value form
> different platform, due to the different platform may be have different implement
> about this, so I am not sure how to implement the variant of dw_pcie_writew_dbi?
>   
> > 
> > I'm keen to hear feedback from Jingoo/Gustavo on this.
> 
> OK, expect the feedback.

Hi Jingoo/Gustavo,

I'm keen for your review/feedback on this.

Thanks,

Andrew Murray

> 
> Thanks 
> Xiaowei
> 
> > 
> > Thanks,
> > 
> > Andrew Murray
> > 
> > > ---
> > > v2:
> > >  - Remove duplicate redundant code.
> > >  - Reimplement the PF config space access way.
> > > v3:
> > >  - Integrate duplicate code for func_select.
> > >  - Move PCIE_ATU_FUNC_NUM(pf) (pf << 20) to ((pf) << 20).
> > >  - Add the comments for func_conf_select function.
> > >
> > >  drivers/pci/controller/dwc/pcie-designware-ep.c | 123
> > ++++++++++++++++--------
> > >  drivers/pci/controller/dwc/pcie-designware.c    |  59 ++++++++----
> > >  drivers/pci/controller/dwc/pcie-designware.h    |  18 +++-
> > >  3 files changed, 142 insertions(+), 58 deletions(-)
> > >
> > > diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c
> > > b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > > index 65f4792..eb851c2 100644
> > > --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> > > +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > > @@ -19,12 +19,26 @@ void dw_pcie_ep_linkup(struct dw_pcie_ep *ep)
> > >  	pci_epc_linkup(epc);
> > >  }
> > >
> > > -static void __dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno
> > bar,
> > > -				   int flags)
> > > +static unsigned int dw_pcie_ep_func_select(struct dw_pcie_ep *ep, u8
> > > +func_no) {
> > > +	unsigned int func_offset = 0;
> > > +
> > > +	if (ep->ops->func_conf_select)
> > > +		func_offset = ep->ops->func_conf_select(ep, func_no);
> > > +
> > > +	return func_offset;
> > > +}
> > > +
> > > +static void __dw_pcie_ep_reset_bar(struct dw_pcie *pci, u8 func_no,
> > > +				   enum pci_barno bar, int flags)
> > >  {
> > >  	u32 reg;
> > > +	unsigned int func_offset = 0;
> > > +	struct dw_pcie_ep *ep = &pci->ep;
> > > +
> > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > >
> > > -	reg = PCI_BASE_ADDRESS_0 + (4 * bar);
> > > +	reg = func_offset + PCI_BASE_ADDRESS_0 + (4 * bar);
> > >  	dw_pcie_dbi_ro_wr_en(pci);
> > >  	dw_pcie_writel_dbi2(pci, reg, 0x0);
> > >  	dw_pcie_writel_dbi(pci, reg, 0x0);
> > > @@ -37,7 +51,12 @@ static void __dw_pcie_ep_reset_bar(struct dw_pcie
> > > *pci, enum pci_barno bar,
> > >
> > >  void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar)  {
> > > -	__dw_pcie_ep_reset_bar(pci, bar, 0);
> > > +	u8 func_no, funcs;
> > > +
> > > +	funcs = pci->ep.epc->max_functions;
> > > +
> > > +	for (func_no = 0; func_no < funcs; func_no++)
> > > +		__dw_pcie_ep_reset_bar(pci, func_no, bar, 0);
> > >  }
> > >
> > >  static int dw_pcie_ep_write_header(struct pci_epc *epc, u8 func_no,
> > > @@ -45,28 +64,31 @@ static int dw_pcie_ep_write_header(struct pci_epc
> > > *epc, u8 func_no,  {
> > >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > > +	unsigned int func_offset = 0;
> > > +
> > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > >
> > >  	dw_pcie_dbi_ro_wr_en(pci);
> > > -	dw_pcie_writew_dbi(pci, PCI_VENDOR_ID, hdr->vendorid);
> > > -	dw_pcie_writew_dbi(pci, PCI_DEVICE_ID, hdr->deviceid);
> > > -	dw_pcie_writeb_dbi(pci, PCI_REVISION_ID, hdr->revid);
> > > -	dw_pcie_writeb_dbi(pci, PCI_CLASS_PROG, hdr->progif_code);
> > > -	dw_pcie_writew_dbi(pci, PCI_CLASS_DEVICE,
> > > +	dw_pcie_writew_dbi(pci, func_offset + PCI_VENDOR_ID, hdr->vendorid);
> > > +	dw_pcie_writew_dbi(pci, func_offset + PCI_DEVICE_ID, hdr->deviceid);
> > > +	dw_pcie_writeb_dbi(pci, func_offset + PCI_REVISION_ID, hdr->revid);
> > > +	dw_pcie_writeb_dbi(pci, func_offset + PCI_CLASS_PROG,
> > hdr->progif_code);
> > > +	dw_pcie_writew_dbi(pci, func_offset + PCI_CLASS_DEVICE,
> > >  			   hdr->subclass_code | hdr->baseclass_code << 8);
> > > -	dw_pcie_writeb_dbi(pci, PCI_CACHE_LINE_SIZE,
> > > +	dw_pcie_writeb_dbi(pci, func_offset + PCI_CACHE_LINE_SIZE,
> > >  			   hdr->cache_line_size);
> > > -	dw_pcie_writew_dbi(pci, PCI_SUBSYSTEM_VENDOR_ID,
> > > +	dw_pcie_writew_dbi(pci, func_offset + PCI_SUBSYSTEM_VENDOR_ID,
> > >  			   hdr->subsys_vendor_id);
> > > -	dw_pcie_writew_dbi(pci, PCI_SUBSYSTEM_ID, hdr->subsys_id);
> > > -	dw_pcie_writeb_dbi(pci, PCI_INTERRUPT_PIN,
> > > +	dw_pcie_writew_dbi(pci, func_offset + PCI_SUBSYSTEM_ID,
> > hdr->subsys_id);
> > > +	dw_pcie_writeb_dbi(pci, func_offset + PCI_INTERRUPT_PIN,
> > >  			   hdr->interrupt_pin);
> > >  	dw_pcie_dbi_ro_wr_dis(pci);
> > >
> > >  	return 0;
> > >  }
> > >
> > > -static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, enum
> > pci_barno bar,
> > > -				  dma_addr_t cpu_addr,
> > > +static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no,
> > > +				  enum pci_barno bar, dma_addr_t cpu_addr,
> > >  				  enum dw_pcie_as_type as_type)
> > >  {
> > >  	int ret;
> > > @@ -79,7 +101,7 @@ static int dw_pcie_ep_inbound_atu(struct
> > dw_pcie_ep *ep, enum pci_barno bar,
> > >  		return -EINVAL;
> > >  	}
> > >
> > > -	ret = dw_pcie_prog_inbound_atu(pci, free_win, bar, cpu_addr,
> > > +	ret = dw_pcie_prog_inbound_atu(pci, func_no, free_win, bar,
> > > +cpu_addr,
> > >  				       as_type);
> > >  	if (ret < 0) {
> > >  		dev_err(pci->dev, "Failed to program IB window\n"); @@ -92,7
> > +114,8
> > > @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, enum
> > pci_barno bar,
> > >  	return 0;
> > >  }
> > >
> > > -static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, phys_addr_t
> > > phys_addr,
> > > +static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, u8 func_no,
> > > +				   phys_addr_t phys_addr,
> > >  				   u64 pci_addr, size_t size)
> > >  {
> > >  	u32 free_win;
> > > @@ -104,8 +127,8 @@ static int dw_pcie_ep_outbound_atu(struct
> > dw_pcie_ep *ep, phys_addr_t phys_addr,
> > >  		return -EINVAL;
> > >  	}
> > >
> > > -	dw_pcie_prog_outbound_atu(pci, free_win, PCIE_ATU_TYPE_MEM,
> > > -				  phys_addr, pci_addr, size);
> > > +	dw_pcie_prog_ep_outbound_atu(pci, func_no, free_win,
> > PCIE_ATU_TYPE_MEM,
> > > +				     phys_addr, pci_addr, size);
> > >
> > >  	set_bit(free_win, ep->ob_window_map);
> > >  	ep->outbound_addr[free_win] = phys_addr; @@ -121,7 +144,7 @@
> > static
> > > void dw_pcie_ep_clear_bar(struct pci_epc *epc, u8 func_no,
> > >  	enum pci_barno bar = epf_bar->barno;
> > >  	u32 atu_index = ep->bar_to_atu[bar];
> > >
> > > -	__dw_pcie_ep_reset_bar(pci, bar, epf_bar->flags);
> > > +	__dw_pcie_ep_reset_bar(pci, func_no, bar, epf_bar->flags);
> > >
> > >  	dw_pcie_disable_atu(pci, atu_index, DW_PCIE_REGION_INBOUND);
> > >  	clear_bit(atu_index, ep->ib_window_map); @@ -137,14 +160,20 @@
> > > static int dw_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no,
> > >  	size_t size = epf_bar->size;
> > >  	int flags = epf_bar->flags;
> > >  	enum dw_pcie_as_type as_type;
> > > -	u32 reg = PCI_BASE_ADDRESS_0 + (4 * bar);
> > > +	u32 reg;
> > > +	unsigned int func_offset = 0;
> > > +
> > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > +
> > > +	reg = PCI_BASE_ADDRESS_0 + (4 * bar) + func_offset;
> > >
> > >  	if (!(flags & PCI_BASE_ADDRESS_SPACE))
> > >  		as_type = DW_PCIE_AS_MEM;
> > >  	else
> > >  		as_type = DW_PCIE_AS_IO;
> > >
> > > -	ret = dw_pcie_ep_inbound_atu(ep, bar, epf_bar->phys_addr, as_type);
> > > +	ret = dw_pcie_ep_inbound_atu(ep, func_no, bar,
> > > +				     epf_bar->phys_addr, as_type);
> > >  	if (ret)
> > >  		return ret;
> > >
> > > @@ -202,7 +231,7 @@ static int dw_pcie_ep_map_addr(struct pci_epc
> > *epc, u8 func_no,
> > >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > >
> > > -	ret = dw_pcie_ep_outbound_atu(ep, addr, pci_addr, size);
> > > +	ret = dw_pcie_ep_outbound_atu(ep, func_no, addr, pci_addr, size);
> > >  	if (ret) {
> > >  		dev_err(pci->dev, "Failed to enable address\n");
> > >  		return ret;
> > > @@ -216,11 +245,14 @@ static int dw_pcie_ep_get_msi(struct pci_epc
> > *epc, u8 func_no)
> > >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > >  	u32 val, reg;
> > > +	unsigned int func_offset = 0;
> > >
> > >  	if (!ep->msi_cap)
> > >  		return -EINVAL;
> > >
> > > -	reg = ep->msi_cap + PCI_MSI_FLAGS;
> > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > +
> > > +	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
> > >  	val = dw_pcie_readw_dbi(pci, reg);
> > >  	if (!(val & PCI_MSI_FLAGS_ENABLE))
> > >  		return -EINVAL;
> > > @@ -235,11 +267,14 @@ static int dw_pcie_ep_set_msi(struct pci_epc
> > *epc, u8 func_no, u8 interrupts)
> > >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > >  	u32 val, reg;
> > > +	unsigned int func_offset = 0;
> > >
> > >  	if (!ep->msi_cap)
> > >  		return -EINVAL;
> > >
> > > -	reg = ep->msi_cap + PCI_MSI_FLAGS;
> > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > +
> > > +	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
> > >  	val = dw_pcie_readw_dbi(pci, reg);
> > >  	val &= ~PCI_MSI_FLAGS_QMASK;
> > >  	val |= (interrupts << 1) & PCI_MSI_FLAGS_QMASK; @@ -255,11 +290,14
> > > @@ static int dw_pcie_ep_get_msix(struct pci_epc *epc, u8 func_no)
> > >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > >  	u32 val, reg;
> > > +	unsigned int func_offset = 0;
> > >
> > >  	if (!ep->msix_cap)
> > >  		return -EINVAL;
> > >
> > > -	reg = ep->msix_cap + PCI_MSIX_FLAGS;
> > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > +
> > > +	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
> > >  	val = dw_pcie_readw_dbi(pci, reg);
> > >  	if (!(val & PCI_MSIX_FLAGS_ENABLE))
> > >  		return -EINVAL;
> > > @@ -274,11 +312,14 @@ static int dw_pcie_ep_set_msix(struct pci_epc
> > *epc, u8 func_no, u16 interrupts)
> > >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > >  	u32 val, reg;
> > > +	unsigned int func_offset = 0;
> > >
> > >  	if (!ep->msix_cap)
> > >  		return -EINVAL;
> > >
> > > -	reg = ep->msix_cap + PCI_MSIX_FLAGS;
> > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > +
> > > +	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
> > >  	val = dw_pcie_readw_dbi(pci, reg);
> > >  	val &= ~PCI_MSIX_FLAGS_QSIZE;
> > >  	val |= interrupts;
> > > @@ -365,6 +406,7 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep
> > *ep, u8 func_no,
> > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > >  	struct pci_epc *epc = ep->epc;
> > >  	unsigned int aligned_offset;
> > > +	unsigned int func_offset = 0;
> > >  	u16 msg_ctrl, msg_data;
> > >  	u32 msg_addr_lower, msg_addr_upper, reg;
> > >  	u64 msg_addr;
> > > @@ -374,20 +416,22 @@ int dw_pcie_ep_raise_msi_irq(struct
> > dw_pcie_ep *ep, u8 func_no,
> > >  	if (!ep->msi_cap)
> > >  		return -EINVAL;
> > >
> > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > +
> > >  	/* Raise MSI per the PCI Local Bus Specification Revision 3.0, 6.8.1. */
> > > -	reg = ep->msi_cap + PCI_MSI_FLAGS;
> > > +	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
> > >  	msg_ctrl = dw_pcie_readw_dbi(pci, reg);
> > >  	has_upper = !!(msg_ctrl & PCI_MSI_FLAGS_64BIT);
> > > -	reg = ep->msi_cap + PCI_MSI_ADDRESS_LO;
> > > +	reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_LO;
> > >  	msg_addr_lower = dw_pcie_readl_dbi(pci, reg);
> > >  	if (has_upper) {
> > > -		reg = ep->msi_cap + PCI_MSI_ADDRESS_HI;
> > > +		reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_HI;
> > >  		msg_addr_upper = dw_pcie_readl_dbi(pci, reg);
> > > -		reg = ep->msi_cap + PCI_MSI_DATA_64;
> > > +		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_64;
> > >  		msg_data = dw_pcie_readw_dbi(pci, reg);
> > >  	} else {
> > >  		msg_addr_upper = 0;
> > > -		reg = ep->msi_cap + PCI_MSI_DATA_32;
> > > +		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_32;
> > >  		msg_data = dw_pcie_readw_dbi(pci, reg);
> > >  	}
> > >  	aligned_offset = msg_addr_lower & (epc->mem->page_size - 1); @@
> > > -406,11 +450,12 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep,
> > > u8 func_no,  }
> > >
> > >  int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
> > > -			     u16 interrupt_num)
> > > +			      u16 interrupt_num)
> > >  {
> > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > >  	struct pci_epc *epc = ep->epc;
> > >  	u16 tbl_offset, bir;
> > > +	unsigned int func_offset = 0;
> > >  	u32 bar_addr_upper, bar_addr_lower;
> > >  	u32 msg_addr_upper, msg_addr_lower;
> > >  	u32 reg, msg_data, vec_ctrl;
> > > @@ -418,12 +463,14 @@ int dw_pcie_ep_raise_msix_irq(struct
> > dw_pcie_ep *ep, u8 func_no,
> > >  	void __iomem *msix_tbl;
> > >  	int ret;
> > >
> > > -	reg = ep->msix_cap + PCI_MSIX_TABLE;
> > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > +
> > > +	reg = ep->msix_cap + func_offset + PCI_MSIX_TABLE;
> > >  	tbl_offset = dw_pcie_readl_dbi(pci, reg);
> > >  	bir = (tbl_offset & PCI_MSIX_TABLE_BIR);
> > >  	tbl_offset &= PCI_MSIX_TABLE_OFFSET;
> > >
> > > -	reg = PCI_BASE_ADDRESS_0 + (4 * bir);
> > > +	reg = PCI_BASE_ADDRESS_0 + func_offset + (4 * bir);
> > >  	bar_addr_upper = 0;
> > >  	bar_addr_lower = dw_pcie_readl_dbi(pci, reg);
> > >  	reg_u64 = (bar_addr_lower & PCI_BASE_ADDRESS_MEM_TYPE_MASK);
> > @@
> > > -559,13 +606,13 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
> > >  	ep->epc = epc;
> > >  	epc_set_drvdata(epc, ep);
> > >
> > > -	if (ep->ops->ep_init)
> > > -		ep->ops->ep_init(ep);
> > > -
> > >  	ret = of_property_read_u8(np, "max-functions", &epc->max_functions);
> > >  	if (ret < 0)
> > >  		epc->max_functions = 1;
> > >
> > > +	if (ep->ops->ep_init)
> > > +		ep->ops->ep_init(ep);
> > > +
> > >  	ret = __pci_epc_mem_init(epc, ep->phys_base, ep->addr_size,
> > >  				 ep->page_size);
> > >  	if (ret < 0) {
> > > diff --git a/drivers/pci/controller/dwc/pcie-designware.c
> > > b/drivers/pci/controller/dwc/pcie-designware.c
> > > index 143cb6c..ede2e75 100644
> > > --- a/drivers/pci/controller/dwc/pcie-designware.c
> > > +++ b/drivers/pci/controller/dwc/pcie-designware.c
> > > @@ -238,9 +238,10 @@ static void dw_pcie_writel_ob_unroll(struct
> > dw_pcie *pci, u32 index, u32 reg,
> > >  	dw_pcie_writel_atu(pci, offset + reg, val);  }
> > >
> > > -static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int
> > index,
> > > -					     int type, u64 cpu_addr,
> > > -					     u64 pci_addr, u32 size)
> > > +static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, u8
> > func_no,
> > > +					     int index, int type,
> > > +					     u64 cpu_addr, u64 pci_addr,
> > > +					     u32 size)
> > >  {
> > >  	u32 retries, val;
> > >
> > > @@ -255,7 +256,7 @@ static void
> > dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
> > >  	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_UPPER_TARGET,
> > >  				 upper_32_bits(pci_addr));
> > >  	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1,
> > > -				 type);
> > > +				 type | PCIE_ATU_FUNC_NUM(func_no));
> > >  	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL2,
> > >  				 PCIE_ATU_ENABLE);
> > >
> > > @@ -274,8 +275,9 @@ static void
> > dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
> > >  	dev_err(pci->dev, "Outbound iATU is not being enabled\n");  }
> > >
> > > -void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
> > > -			       u64 cpu_addr, u64 pci_addr, u32 size)
> > > +static void __dw_pcie_prog_outbound_atu(struct dw_pcie *pci, u8
> > func_no,
> > > +					int index, int type, u64 cpu_addr,
> > > +					u64 pci_addr, u32 size)
> > >  {
> > >  	u32 retries, val;
> > >
> > > @@ -283,8 +285,8 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie
> > *pci, int index, int type,
> > >  		cpu_addr = pci->ops->cpu_addr_fixup(pci, cpu_addr);
> > >
> > >  	if (pci->iatu_unroll_enabled) {
> > > -		dw_pcie_prog_outbound_atu_unroll(pci, index, type, cpu_addr,
> > > -						 pci_addr, size);
> > > +		dw_pcie_prog_outbound_atu_unroll(pci, func_no, index, type,
> > > +						 cpu_addr, pci_addr, size);
> > >  		return;
> > >  	}
> > >
> > > @@ -300,7 +302,8 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie
> > *pci, int index, int type,
> > >  			   lower_32_bits(pci_addr));
> > >  	dw_pcie_writel_dbi(pci, PCIE_ATU_UPPER_TARGET,
> > >  			   upper_32_bits(pci_addr));
> > > -	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type);
> > > +	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type |
> > > +			   PCIE_ATU_FUNC_NUM(func_no));
> > >  	dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE);
> > >
> > >  	/*
> > > @@ -317,6 +320,21 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie
> > *pci, int index, int type,
> > >  	dev_err(pci->dev, "Outbound iATU is not being enabled\n");  }
> > >
> > > +void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
> > > +			       u64 cpu_addr, u64 pci_addr, u32 size) {
> > > +	__dw_pcie_prog_outbound_atu(pci, 0, index, type,
> > > +				    cpu_addr, pci_addr, size);
> > > +}
> > > +
> > > +void dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int
> > index,
> > > +				  int type, u64 cpu_addr, u64 pci_addr,
> > > +				  u32 size)
> > > +{
> > > +	__dw_pcie_prog_outbound_atu(pci, func_no, index, type,
> > > +				    cpu_addr, pci_addr, size);
> > > +}
> > > +
> > >  static u32 dw_pcie_readl_ib_unroll(struct dw_pcie *pci, u32 index,
> > > u32 reg)  {
> > >  	u32 offset = PCIE_GET_ATU_INB_UNR_REG_OFFSET(index);
> > > @@ -332,8 +350,8 @@ static void dw_pcie_writel_ib_unroll(struct dw_pcie
> > *pci, u32 index, u32 reg,
> > >  	dw_pcie_writel_atu(pci, offset + reg, val);  }
> > >
> > > -static int dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, int index,
> > > -					   int bar, u64 cpu_addr,
> > > +static int dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, u8
> > func_no,
> > > +					   int index, int bar, u64 cpu_addr,
> > >  					   enum dw_pcie_as_type as_type)  {
> > >  	int type;
> > > @@ -355,8 +373,10 @@ static int dw_pcie_prog_inbound_atu_unroll(struct
> > dw_pcie *pci, int index,
> > >  		return -EINVAL;
> > >  	}
> > >
> > > -	dw_pcie_writel_ib_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1,
> > type);
> > > +	dw_pcie_writel_ib_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1,
> > type |
> > > +				 PCIE_ATU_FUNC_NUM(func_no));
> > >  	dw_pcie_writel_ib_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL2,
> > > +				 PCIE_ATU_FUNC_NUM_MATCH_EN |
> > >  				 PCIE_ATU_ENABLE |
> > >  				 PCIE_ATU_BAR_MODE_ENABLE | (bar << 8));
> > >
> > > @@ -377,14 +397,15 @@ static int
> > dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, int index,
> > >  	return -EBUSY;
> > >  }
> > >
> > > -int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int bar,
> > > -			     u64 cpu_addr, enum dw_pcie_as_type as_type)
> > > +int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> > > +			     int bar, u64 cpu_addr,
> > > +			     enum dw_pcie_as_type as_type)
> > >  {
> > >  	int type;
> > >  	u32 retries, val;
> > >
> > >  	if (pci->iatu_unroll_enabled)
> > > -		return dw_pcie_prog_inbound_atu_unroll(pci, index, bar,
> > > +		return dw_pcie_prog_inbound_atu_unroll(pci, func_no, index, bar,
> > >  						       cpu_addr, as_type);
> > >
> > >  	dw_pcie_writel_dbi(pci, PCIE_ATU_VIEWPORT,
> > PCIE_ATU_REGION_INBOUND |
> > > @@ -403,9 +424,11 @@ int dw_pcie_prog_inbound_atu(struct dw_pcie
> > *pci, int index, int bar,
> > >  		return -EINVAL;
> > >  	}
> > >
> > > -	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type);
> > > -	dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE
> > > -			   | PCIE_ATU_BAR_MODE_ENABLE | (bar << 8));
> > > +	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type |
> > > +			   PCIE_ATU_FUNC_NUM(func_no));
> > > +	dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE |
> > > +			   PCIE_ATU_FUNC_NUM_MATCH_EN |
> > > +			   PCIE_ATU_BAR_MODE_ENABLE | (bar << 8));
> > >
> > >  	/*
> > >  	 * Make sure ATU enable takes effect before any subsequent config
> > > diff --git a/drivers/pci/controller/dwc/pcie-designware.h
> > > b/drivers/pci/controller/dwc/pcie-designware.h
> > > index 5a18e94..6aca0bb 100644
> > > --- a/drivers/pci/controller/dwc/pcie-designware.h
> > > +++ b/drivers/pci/controller/dwc/pcie-designware.h
> > > @@ -71,9 +71,11 @@
> > >  #define PCIE_ATU_TYPE_IO		0x2
> > >  #define PCIE_ATU_TYPE_CFG0		0x4
> > >  #define PCIE_ATU_TYPE_CFG1		0x5
> > > +#define PCIE_ATU_FUNC_NUM(pf)           ((pf) << 20)
> > >  #define PCIE_ATU_CR2			0x908
> > >  #define PCIE_ATU_ENABLE			BIT(31)
> > >  #define PCIE_ATU_BAR_MODE_ENABLE	BIT(30)
> > > +#define PCIE_ATU_FUNC_NUM_MATCH_EN      BIT(19)
> > >  #define PCIE_ATU_LOWER_BASE		0x90C
> > >  #define PCIE_ATU_UPPER_BASE		0x910
> > >  #define PCIE_ATU_LIMIT			0x914
> > > @@ -206,6 +208,14 @@ struct dw_pcie_ep_ops {
> > >  	int	(*raise_irq)(struct dw_pcie_ep *ep, u8 func_no,
> > >  			     enum pci_epc_irq_type type, u16 interrupt_num);
> > >  	const struct pci_epc_features* (*get_features)(struct dw_pcie_ep
> > > *ep);
> > > +	/*
> > > +	 * Provide a method to implement the different func config space
> > > +	 * access for different platform, if different func have different
> > > +	 * offset, return the offset of func. if use write a register way
> > > +	 * return a 0, and implement code in callback function of platform
> > > +	 * driver.
> > > +	 */
> > > +	unsigned int (*func_conf_select)(struct dw_pcie_ep *ep, u8 func_no);
> > >  };
> > >
> > >  struct dw_pcie_ep {
> > > @@ -277,8 +287,12 @@ int dw_pcie_wait_for_link(struct dw_pcie *pci);
> > > void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index,
> > >  			       int type, u64 cpu_addr, u64 pci_addr,
> > >  			       u32 size);
> > > -int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int bar,
> > > -			     u64 cpu_addr, enum dw_pcie_as_type as_type);
> > > +void dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int
> > index,
> > > +				  int type, u64 cpu_addr, u64 pci_addr,
> > > +				  u32 size);
> > > +int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> > > +			     int bar, u64 cpu_addr,
> > > +			     enum dw_pcie_as_type as_type);
> > >  void dw_pcie_disable_atu(struct dw_pcie *pci, int index,
> > >  			 enum dw_pcie_region_type type);
> > >  void dw_pcie_setup(struct dw_pcie *pci);
> > > --
> > > 2.9.5
> > >
> > >
> > > _______________________________________________
> > > linux-arm-kernel mailing list
> > > linux-arm-kernel@lists.infradead.org
> > > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists
> > > .infradead.org%2Fmailman%2Flistinfo%2Flinux-arm-kernel&amp;data=02%
> > 7C0
> > >
> > 1%7Cxiaowei.bao%40nxp.com%7C99eef14a525040ed3eab08d72fc244f1%7C
> > 686ea1d
> > >
> > 3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637030383769341270&amp;sd
> > ata=ck2EC
> > > %2FJYCjWErvbUM%2FT%2BoVMANMwyLRI4gVRssdnd04w%3D&amp;reser
> > ved=0

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

* RE: [PATCH v3 01/11] PCI: designware-ep: Add multiple PFs support for DWC
  2019-09-26 10:29       ` Andrew Murray
@ 2019-09-26 13:38         ` Gustavo Pimentel
  0 siblings, 0 replies; 42+ messages in thread
From: Gustavo Pimentel @ 2019-09-26 13:38 UTC (permalink / raw)
  To: Andrew Murray, Xiaowei Bao, jingoohan1, gustavo.pimentel@synopsys.com
  Cc: mark.rutland, Roy Zang, lorenzo.pieralisi, arnd, devicetree,
	linux-pci, Z.q. Hou, linuxppc-dev, linux-kernel, kishon,
	M.h. Lian, robh+dt, linux-arm-kernel, gregkh, Leo Li, shawnguo,
	Mingkai Hu

On Thu, Sep 26, 2019 at 11:29:46, Andrew Murray <andrew.murray@arm.com> 
wrote:

> On Tue, Sep 03, 2019 at 03:43:15AM +0000, Xiaowei Bao wrote:
> > 
> > 
> > > -----Original Message-----
> > > From: Andrew Murray <andrew.murray@arm.com>
> > > Sent: 2019年9月3日 0:26
> > > To: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > Cc: robh+dt@kernel.org; mark.rutland@arm.com; shawnguo@kernel.org; Leo
> > > Li <leoyang.li@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com; M.h.
> > > Lian <minghuan.lian@nxp.com>; Mingkai Hu <mingkai.hu@nxp.com>; Roy
> > > Zang <roy.zang@nxp.com>; jingoohan1@gmail.com;
> > > gustavo.pimentel@synopsys.com; linux-pci@vger.kernel.org;
> > > devicetree@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > linux-arm-kernel@lists.infradead.org; linuxppc-dev@lists.ozlabs.org;
> > > gregkh@linuxfoundation.org; Z.q. Hou <zhiqiang.hou@nxp.com>;
> > > arnd@arndb.de
> > > Subject: Re: [PATCH v3 01/11] PCI: designware-ep: Add multiple PFs support
> > > for DWC
> > > 
> > > On Mon, Sep 02, 2019 at 11:17:06AM +0800, Xiaowei Bao wrote:
> > > > Add multiple PFs support for DWC, different PF have different config
> > > > space we use pf-offset property which get from the DTS to access the
> > > > different pF
> > > 
> > > This needs to be updated as this no longer comes from the DT.
> > 
> > Yes, thanks
> > 
> > Thanks
> > Xiaowei
> > 
> > > 
> > > > config space.
> > > >
> > > > Signed-off-by: Xiaowei Bao <xiaowei.bao@nxp.com>
> > > 
> > > 
> > > We're assuming:
> > > 
> > >  - The offset address (func_offset) between PF's in the memory map can be
> > >    different between different DWC implementations. And also that it's
> > >    possible for DWC implementations to address PFs without using an offset.
> > > 
> > >  - The current approach is preferable to adding DWC EP driver callbacks
> > >    for writing to the EP config space (e.g. a variant of dw_pcie_writew_dbi
> > >    that takes a func number).
> > 
> > Even if use the a variant of dw_pcie_writew_dbi, we also need a offset value form
> > different platform, due to the different platform may be have different implement
> > about this, so I am not sure how to implement the variant of dw_pcie_writew_dbi?
> >   
> > > 
> > > I'm keen to hear feedback from Jingoo/Gustavo on this.
> > 
> > OK, expect the feedback.
> 
> Hi Jingoo/Gustavo,
> 
> I'm keen for your review/feedback on this.

Hi,

This ep section was created by Kishon initially, he is maintainer of it.
But it seems ok to me. I've seen the patch and sounds good to me. I've 
given my Acked on version 4.

Regards,
Gustavo

> 
> Thanks,
> 
> Andrew Murray
> 
> > 
> > Thanks 
> > Xiaowei
> > 
> > > 
> > > Thanks,
> > > 
> > > Andrew Murray
> > > 
> > > > ---
> > > > v2:
> > > >  - Remove duplicate redundant code.
> > > >  - Reimplement the PF config space access way.
> > > > v3:
> > > >  - Integrate duplicate code for func_select.
> > > >  - Move PCIE_ATU_FUNC_NUM(pf) (pf << 20) to ((pf) << 20).
> > > >  - Add the comments for func_conf_select function.
> > > >
> > > >  drivers/pci/controller/dwc/pcie-designware-ep.c | 123
> > > ++++++++++++++++--------
> > > >  drivers/pci/controller/dwc/pcie-designware.c    |  59 ++++++++----
> > > >  drivers/pci/controller/dwc/pcie-designware.h    |  18 +++-
> > > >  3 files changed, 142 insertions(+), 58 deletions(-)
> > > >
> > > > diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c
> > > > b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > > > index 65f4792..eb851c2 100644
> > > > --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> > > > +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > > > @@ -19,12 +19,26 @@ void dw_pcie_ep_linkup(struct dw_pcie_ep *ep)
> > > >  	pci_epc_linkup(epc);
> > > >  }
> > > >
> > > > -static void __dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno
> > > bar,
> > > > -				   int flags)
> > > > +static unsigned int dw_pcie_ep_func_select(struct dw_pcie_ep *ep, u8
> > > > +func_no) {
> > > > +	unsigned int func_offset = 0;
> > > > +
> > > > +	if (ep->ops->func_conf_select)
> > > > +		func_offset = ep->ops->func_conf_select(ep, func_no);
> > > > +
> > > > +	return func_offset;
> > > > +}
> > > > +
> > > > +static void __dw_pcie_ep_reset_bar(struct dw_pcie *pci, u8 func_no,
> > > > +				   enum pci_barno bar, int flags)
> > > >  {
> > > >  	u32 reg;
> > > > +	unsigned int func_offset = 0;
> > > > +	struct dw_pcie_ep *ep = &pci->ep;
> > > > +
> > > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > >
> > > > -	reg = PCI_BASE_ADDRESS_0 + (4 * bar);
> > > > +	reg = func_offset + PCI_BASE_ADDRESS_0 + (4 * bar);
> > > >  	dw_pcie_dbi_ro_wr_en(pci);
> > > >  	dw_pcie_writel_dbi2(pci, reg, 0x0);
> > > >  	dw_pcie_writel_dbi(pci, reg, 0x0);
> > > > @@ -37,7 +51,12 @@ static void __dw_pcie_ep_reset_bar(struct dw_pcie
> > > > *pci, enum pci_barno bar,
> > > >
> > > >  void dw_pcie_ep_reset_bar(struct dw_pcie *pci, enum pci_barno bar)  {
> > > > -	__dw_pcie_ep_reset_bar(pci, bar, 0);
> > > > +	u8 func_no, funcs;
> > > > +
> > > > +	funcs = pci->ep.epc->max_functions;
> > > > +
> > > > +	for (func_no = 0; func_no < funcs; func_no++)
> > > > +		__dw_pcie_ep_reset_bar(pci, func_no, bar, 0);
> > > >  }
> > > >
> > > >  static int dw_pcie_ep_write_header(struct pci_epc *epc, u8 func_no,
> > > > @@ -45,28 +64,31 @@ static int dw_pcie_ep_write_header(struct pci_epc
> > > > *epc, u8 func_no,  {
> > > >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > > > +	unsigned int func_offset = 0;
> > > > +
> > > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > >
> > > >  	dw_pcie_dbi_ro_wr_en(pci);
> > > > -	dw_pcie_writew_dbi(pci, PCI_VENDOR_ID, hdr->vendorid);
> > > > -	dw_pcie_writew_dbi(pci, PCI_DEVICE_ID, hdr->deviceid);
> > > > -	dw_pcie_writeb_dbi(pci, PCI_REVISION_ID, hdr->revid);
> > > > -	dw_pcie_writeb_dbi(pci, PCI_CLASS_PROG, hdr->progif_code);
> > > > -	dw_pcie_writew_dbi(pci, PCI_CLASS_DEVICE,
> > > > +	dw_pcie_writew_dbi(pci, func_offset + PCI_VENDOR_ID, hdr->vendorid);
> > > > +	dw_pcie_writew_dbi(pci, func_offset + PCI_DEVICE_ID, hdr->deviceid);
> > > > +	dw_pcie_writeb_dbi(pci, func_offset + PCI_REVISION_ID, hdr->revid);
> > > > +	dw_pcie_writeb_dbi(pci, func_offset + PCI_CLASS_PROG,
> > > hdr->progif_code);
> > > > +	dw_pcie_writew_dbi(pci, func_offset + PCI_CLASS_DEVICE,
> > > >  			   hdr->subclass_code | hdr->baseclass_code << 8);
> > > > -	dw_pcie_writeb_dbi(pci, PCI_CACHE_LINE_SIZE,
> > > > +	dw_pcie_writeb_dbi(pci, func_offset + PCI_CACHE_LINE_SIZE,
> > > >  			   hdr->cache_line_size);
> > > > -	dw_pcie_writew_dbi(pci, PCI_SUBSYSTEM_VENDOR_ID,
> > > > +	dw_pcie_writew_dbi(pci, func_offset + PCI_SUBSYSTEM_VENDOR_ID,
> > > >  			   hdr->subsys_vendor_id);
> > > > -	dw_pcie_writew_dbi(pci, PCI_SUBSYSTEM_ID, hdr->subsys_id);
> > > > -	dw_pcie_writeb_dbi(pci, PCI_INTERRUPT_PIN,
> > > > +	dw_pcie_writew_dbi(pci, func_offset + PCI_SUBSYSTEM_ID,
> > > hdr->subsys_id);
> > > > +	dw_pcie_writeb_dbi(pci, func_offset + PCI_INTERRUPT_PIN,
> > > >  			   hdr->interrupt_pin);
> > > >  	dw_pcie_dbi_ro_wr_dis(pci);
> > > >
> > > >  	return 0;
> > > >  }
> > > >
> > > > -static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, enum
> > > pci_barno bar,
> > > > -				  dma_addr_t cpu_addr,
> > > > +static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no,
> > > > +				  enum pci_barno bar, dma_addr_t cpu_addr,
> > > >  				  enum dw_pcie_as_type as_type)
> > > >  {
> > > >  	int ret;
> > > > @@ -79,7 +101,7 @@ static int dw_pcie_ep_inbound_atu(struct
> > > dw_pcie_ep *ep, enum pci_barno bar,
> > > >  		return -EINVAL;
> > > >  	}
> > > >
> > > > -	ret = dw_pcie_prog_inbound_atu(pci, free_win, bar, cpu_addr,
> > > > +	ret = dw_pcie_prog_inbound_atu(pci, func_no, free_win, bar,
> > > > +cpu_addr,
> > > >  				       as_type);
> > > >  	if (ret < 0) {
> > > >  		dev_err(pci->dev, "Failed to program IB window\n"); @@ -92,7
> > > +114,8
> > > > @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, enum
> > > pci_barno bar,
> > > >  	return 0;
> > > >  }
> > > >
> > > > -static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, phys_addr_t
> > > > phys_addr,
> > > > +static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, u8 func_no,
> > > > +				   phys_addr_t phys_addr,
> > > >  				   u64 pci_addr, size_t size)
> > > >  {
> > > >  	u32 free_win;
> > > > @@ -104,8 +127,8 @@ static int dw_pcie_ep_outbound_atu(struct
> > > dw_pcie_ep *ep, phys_addr_t phys_addr,
> > > >  		return -EINVAL;
> > > >  	}
> > > >
> > > > -	dw_pcie_prog_outbound_atu(pci, free_win, PCIE_ATU_TYPE_MEM,
> > > > -				  phys_addr, pci_addr, size);
> > > > +	dw_pcie_prog_ep_outbound_atu(pci, func_no, free_win,
> > > PCIE_ATU_TYPE_MEM,
> > > > +				     phys_addr, pci_addr, size);
> > > >
> > > >  	set_bit(free_win, ep->ob_window_map);
> > > >  	ep->outbound_addr[free_win] = phys_addr; @@ -121,7 +144,7 @@
> > > static
> > > > void dw_pcie_ep_clear_bar(struct pci_epc *epc, u8 func_no,
> > > >  	enum pci_barno bar = epf_bar->barno;
> > > >  	u32 atu_index = ep->bar_to_atu[bar];
> > > >
> > > > -	__dw_pcie_ep_reset_bar(pci, bar, epf_bar->flags);
> > > > +	__dw_pcie_ep_reset_bar(pci, func_no, bar, epf_bar->flags);
> > > >
> > > >  	dw_pcie_disable_atu(pci, atu_index, DW_PCIE_REGION_INBOUND);
> > > >  	clear_bit(atu_index, ep->ib_window_map); @@ -137,14 +160,20 @@
> > > > static int dw_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no,
> > > >  	size_t size = epf_bar->size;
> > > >  	int flags = epf_bar->flags;
> > > >  	enum dw_pcie_as_type as_type;
> > > > -	u32 reg = PCI_BASE_ADDRESS_0 + (4 * bar);
> > > > +	u32 reg;
> > > > +	unsigned int func_offset = 0;
> > > > +
> > > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > > +
> > > > +	reg = PCI_BASE_ADDRESS_0 + (4 * bar) + func_offset;
> > > >
> > > >  	if (!(flags & PCI_BASE_ADDRESS_SPACE))
> > > >  		as_type = DW_PCIE_AS_MEM;
> > > >  	else
> > > >  		as_type = DW_PCIE_AS_IO;
> > > >
> > > > -	ret = dw_pcie_ep_inbound_atu(ep, bar, epf_bar->phys_addr, as_type);
> > > > +	ret = dw_pcie_ep_inbound_atu(ep, func_no, bar,
> > > > +				     epf_bar->phys_addr, as_type);
> > > >  	if (ret)
> > > >  		return ret;
> > > >
> > > > @@ -202,7 +231,7 @@ static int dw_pcie_ep_map_addr(struct pci_epc
> > > *epc, u8 func_no,
> > > >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > > >
> > > > -	ret = dw_pcie_ep_outbound_atu(ep, addr, pci_addr, size);
> > > > +	ret = dw_pcie_ep_outbound_atu(ep, func_no, addr, pci_addr, size);
> > > >  	if (ret) {
> > > >  		dev_err(pci->dev, "Failed to enable address\n");
> > > >  		return ret;
> > > > @@ -216,11 +245,14 @@ static int dw_pcie_ep_get_msi(struct pci_epc
> > > *epc, u8 func_no)
> > > >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > > >  	u32 val, reg;
> > > > +	unsigned int func_offset = 0;
> > > >
> > > >  	if (!ep->msi_cap)
> > > >  		return -EINVAL;
> > > >
> > > > -	reg = ep->msi_cap + PCI_MSI_FLAGS;
> > > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > > +
> > > > +	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
> > > >  	val = dw_pcie_readw_dbi(pci, reg);
> > > >  	if (!(val & PCI_MSI_FLAGS_ENABLE))
> > > >  		return -EINVAL;
> > > > @@ -235,11 +267,14 @@ static int dw_pcie_ep_set_msi(struct pci_epc
> > > *epc, u8 func_no, u8 interrupts)
> > > >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > > >  	u32 val, reg;
> > > > +	unsigned int func_offset = 0;
> > > >
> > > >  	if (!ep->msi_cap)
> > > >  		return -EINVAL;
> > > >
> > > > -	reg = ep->msi_cap + PCI_MSI_FLAGS;
> > > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > > +
> > > > +	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
> > > >  	val = dw_pcie_readw_dbi(pci, reg);
> > > >  	val &= ~PCI_MSI_FLAGS_QMASK;
> > > >  	val |= (interrupts << 1) & PCI_MSI_FLAGS_QMASK; @@ -255,11 +290,14
> > > > @@ static int dw_pcie_ep_get_msix(struct pci_epc *epc, u8 func_no)
> > > >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > > >  	u32 val, reg;
> > > > +	unsigned int func_offset = 0;
> > > >
> > > >  	if (!ep->msix_cap)
> > > >  		return -EINVAL;
> > > >
> > > > -	reg = ep->msix_cap + PCI_MSIX_FLAGS;
> > > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > > +
> > > > +	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
> > > >  	val = dw_pcie_readw_dbi(pci, reg);
> > > >  	if (!(val & PCI_MSIX_FLAGS_ENABLE))
> > > >  		return -EINVAL;
> > > > @@ -274,11 +312,14 @@ static int dw_pcie_ep_set_msix(struct pci_epc
> > > *epc, u8 func_no, u16 interrupts)
> > > >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> > > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > > >  	u32 val, reg;
> > > > +	unsigned int func_offset = 0;
> > > >
> > > >  	if (!ep->msix_cap)
> > > >  		return -EINVAL;
> > > >
> > > > -	reg = ep->msix_cap + PCI_MSIX_FLAGS;
> > > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > > +
> > > > +	reg = ep->msix_cap + func_offset + PCI_MSIX_FLAGS;
> > > >  	val = dw_pcie_readw_dbi(pci, reg);
> > > >  	val &= ~PCI_MSIX_FLAGS_QSIZE;
> > > >  	val |= interrupts;
> > > > @@ -365,6 +406,7 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep
> > > *ep, u8 func_no,
> > > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > > >  	struct pci_epc *epc = ep->epc;
> > > >  	unsigned int aligned_offset;
> > > > +	unsigned int func_offset = 0;
> > > >  	u16 msg_ctrl, msg_data;
> > > >  	u32 msg_addr_lower, msg_addr_upper, reg;
> > > >  	u64 msg_addr;
> > > > @@ -374,20 +416,22 @@ int dw_pcie_ep_raise_msi_irq(struct
> > > dw_pcie_ep *ep, u8 func_no,
> > > >  	if (!ep->msi_cap)
> > > >  		return -EINVAL;
> > > >
> > > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > > +
> > > >  	/* Raise MSI per the PCI Local Bus Specification Revision 3.0, 6.8.1. */
> > > > -	reg = ep->msi_cap + PCI_MSI_FLAGS;
> > > > +	reg = ep->msi_cap + func_offset + PCI_MSI_FLAGS;
> > > >  	msg_ctrl = dw_pcie_readw_dbi(pci, reg);
> > > >  	has_upper = !!(msg_ctrl & PCI_MSI_FLAGS_64BIT);
> > > > -	reg = ep->msi_cap + PCI_MSI_ADDRESS_LO;
> > > > +	reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_LO;
> > > >  	msg_addr_lower = dw_pcie_readl_dbi(pci, reg);
> > > >  	if (has_upper) {
> > > > -		reg = ep->msi_cap + PCI_MSI_ADDRESS_HI;
> > > > +		reg = ep->msi_cap + func_offset + PCI_MSI_ADDRESS_HI;
> > > >  		msg_addr_upper = dw_pcie_readl_dbi(pci, reg);
> > > > -		reg = ep->msi_cap + PCI_MSI_DATA_64;
> > > > +		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_64;
> > > >  		msg_data = dw_pcie_readw_dbi(pci, reg);
> > > >  	} else {
> > > >  		msg_addr_upper = 0;
> > > > -		reg = ep->msi_cap + PCI_MSI_DATA_32;
> > > > +		reg = ep->msi_cap + func_offset + PCI_MSI_DATA_32;
> > > >  		msg_data = dw_pcie_readw_dbi(pci, reg);
> > > >  	}
> > > >  	aligned_offset = msg_addr_lower & (epc->mem->page_size - 1); @@
> > > > -406,11 +450,12 @@ int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep,
> > > > u8 func_no,  }
> > > >
> > > >  int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
> > > > -			     u16 interrupt_num)
> > > > +			      u16 interrupt_num)
> > > >  {
> > > >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > > >  	struct pci_epc *epc = ep->epc;
> > > >  	u16 tbl_offset, bir;
> > > > +	unsigned int func_offset = 0;
> > > >  	u32 bar_addr_upper, bar_addr_lower;
> > > >  	u32 msg_addr_upper, msg_addr_lower;
> > > >  	u32 reg, msg_data, vec_ctrl;
> > > > @@ -418,12 +463,14 @@ int dw_pcie_ep_raise_msix_irq(struct
> > > dw_pcie_ep *ep, u8 func_no,
> > > >  	void __iomem *msix_tbl;
> > > >  	int ret;
> > > >
> > > > -	reg = ep->msix_cap + PCI_MSIX_TABLE;
> > > > +	func_offset = dw_pcie_ep_func_select(ep, func_no);
> > > > +
> > > > +	reg = ep->msix_cap + func_offset + PCI_MSIX_TABLE;
> > > >  	tbl_offset = dw_pcie_readl_dbi(pci, reg);
> > > >  	bir = (tbl_offset & PCI_MSIX_TABLE_BIR);
> > > >  	tbl_offset &= PCI_MSIX_TABLE_OFFSET;
> > > >
> > > > -	reg = PCI_BASE_ADDRESS_0 + (4 * bir);
> > > > +	reg = PCI_BASE_ADDRESS_0 + func_offset + (4 * bir);
> > > >  	bar_addr_upper = 0;
> > > >  	bar_addr_lower = dw_pcie_readl_dbi(pci, reg);
> > > >  	reg_u64 = (bar_addr_lower & PCI_BASE_ADDRESS_MEM_TYPE_MASK);
> > > @@
> > > > -559,13 +606,13 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
> > > >  	ep->epc = epc;
> > > >  	epc_set_drvdata(epc, ep);
> > > >
> > > > -	if (ep->ops->ep_init)
> > > > -		ep->ops->ep_init(ep);
> > > > -
> > > >  	ret = of_property_read_u8(np, "max-functions", &epc->max_functions);
> > > >  	if (ret < 0)
> > > >  		epc->max_functions = 1;
> > > >
> > > > +	if (ep->ops->ep_init)
> > > > +		ep->ops->ep_init(ep);
> > > > +
> > > >  	ret = __pci_epc_mem_init(epc, ep->phys_base, ep->addr_size,
> > > >  				 ep->page_size);
> > > >  	if (ret < 0) {
> > > > diff --git a/drivers/pci/controller/dwc/pcie-designware.c
> > > > b/drivers/pci/controller/dwc/pcie-designware.c
> > > > index 143cb6c..ede2e75 100644
> > > > --- a/drivers/pci/controller/dwc/pcie-designware.c
> > > > +++ b/drivers/pci/controller/dwc/pcie-designware.c
> > > > @@ -238,9 +238,10 @@ static void dw_pcie_writel_ob_unroll(struct
> > > dw_pcie *pci, u32 index, u32 reg,
> > > >  	dw_pcie_writel_atu(pci, offset + reg, val);  }
> > > >
> > > > -static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int
> > > index,
> > > > -					     int type, u64 cpu_addr,
> > > > -					     u64 pci_addr, u32 size)
> > > > +static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, u8
> > > func_no,
> > > > +					     int index, int type,
> > > > +					     u64 cpu_addr, u64 pci_addr,
> > > > +					     u32 size)
> > > >  {
> > > >  	u32 retries, val;
> > > >
> > > > @@ -255,7 +256,7 @@ static void
> > > dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
> > > >  	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_UPPER_TARGET,
> > > >  				 upper_32_bits(pci_addr));
> > > >  	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1,
> > > > -				 type);
> > > > +				 type | PCIE_ATU_FUNC_NUM(func_no));
> > > >  	dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL2,
> > > >  				 PCIE_ATU_ENABLE);
> > > >
> > > > @@ -274,8 +275,9 @@ static void
> > > dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index,
> > > >  	dev_err(pci->dev, "Outbound iATU is not being enabled\n");  }
> > > >
> > > > -void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
> > > > -			       u64 cpu_addr, u64 pci_addr, u32 size)
> > > > +static void __dw_pcie_prog_outbound_atu(struct dw_pcie *pci, u8
> > > func_no,
> > > > +					int index, int type, u64 cpu_addr,
> > > > +					u64 pci_addr, u32 size)
> > > >  {
> > > >  	u32 retries, val;
> > > >
> > > > @@ -283,8 +285,8 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie
> > > *pci, int index, int type,
> > > >  		cpu_addr = pci->ops->cpu_addr_fixup(pci, cpu_addr);
> > > >
> > > >  	if (pci->iatu_unroll_enabled) {
> > > > -		dw_pcie_prog_outbound_atu_unroll(pci, index, type, cpu_addr,
> > > > -						 pci_addr, size);
> > > > +		dw_pcie_prog_outbound_atu_unroll(pci, func_no, index, type,
> > > > +						 cpu_addr, pci_addr, size);
> > > >  		return;
> > > >  	}
> > > >
> > > > @@ -300,7 +302,8 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie
> > > *pci, int index, int type,
> > > >  			   lower_32_bits(pci_addr));
> > > >  	dw_pcie_writel_dbi(pci, PCIE_ATU_UPPER_TARGET,
> > > >  			   upper_32_bits(pci_addr));
> > > > -	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type);
> > > > +	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type |
> > > > +			   PCIE_ATU_FUNC_NUM(func_no));
> > > >  	dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE);
> > > >
> > > >  	/*
> > > > @@ -317,6 +320,21 @@ void dw_pcie_prog_outbound_atu(struct dw_pcie
> > > *pci, int index, int type,
> > > >  	dev_err(pci->dev, "Outbound iATU is not being enabled\n");  }
> > > >
> > > > +void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
> > > > +			       u64 cpu_addr, u64 pci_addr, u32 size) {
> > > > +	__dw_pcie_prog_outbound_atu(pci, 0, index, type,
> > > > +				    cpu_addr, pci_addr, size);
> > > > +}
> > > > +
> > > > +void dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int
> > > index,
> > > > +				  int type, u64 cpu_addr, u64 pci_addr,
> > > > +				  u32 size)
> > > > +{
> > > > +	__dw_pcie_prog_outbound_atu(pci, func_no, index, type,
> > > > +				    cpu_addr, pci_addr, size);
> > > > +}
> > > > +
> > > >  static u32 dw_pcie_readl_ib_unroll(struct dw_pcie *pci, u32 index,
> > > > u32 reg)  {
> > > >  	u32 offset = PCIE_GET_ATU_INB_UNR_REG_OFFSET(index);
> > > > @@ -332,8 +350,8 @@ static void dw_pcie_writel_ib_unroll(struct dw_pcie
> > > *pci, u32 index, u32 reg,
> > > >  	dw_pcie_writel_atu(pci, offset + reg, val);  }
> > > >
> > > > -static int dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, int index,
> > > > -					   int bar, u64 cpu_addr,
> > > > +static int dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, u8
> > > func_no,
> > > > +					   int index, int bar, u64 cpu_addr,
> > > >  					   enum dw_pcie_as_type as_type)  {
> > > >  	int type;
> > > > @@ -355,8 +373,10 @@ static int dw_pcie_prog_inbound_atu_unroll(struct
> > > dw_pcie *pci, int index,
> > > >  		return -EINVAL;
> > > >  	}
> > > >
> > > > -	dw_pcie_writel_ib_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1,
> > > type);
> > > > +	dw_pcie_writel_ib_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1,
> > > type |
> > > > +				 PCIE_ATU_FUNC_NUM(func_no));
> > > >  	dw_pcie_writel_ib_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL2,
> > > > +				 PCIE_ATU_FUNC_NUM_MATCH_EN |
> > > >  				 PCIE_ATU_ENABLE |
> > > >  				 PCIE_ATU_BAR_MODE_ENABLE | (bar << 8));
> > > >
> > > > @@ -377,14 +397,15 @@ static int
> > > dw_pcie_prog_inbound_atu_unroll(struct dw_pcie *pci, int index,
> > > >  	return -EBUSY;
> > > >  }
> > > >
> > > > -int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int bar,
> > > > -			     u64 cpu_addr, enum dw_pcie_as_type as_type)
> > > > +int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> > > > +			     int bar, u64 cpu_addr,
> > > > +			     enum dw_pcie_as_type as_type)
> > > >  {
> > > >  	int type;
> > > >  	u32 retries, val;
> > > >
> > > >  	if (pci->iatu_unroll_enabled)
> > > > -		return dw_pcie_prog_inbound_atu_unroll(pci, index, bar,
> > > > +		return dw_pcie_prog_inbound_atu_unroll(pci, func_no, index, bar,
> > > >  						       cpu_addr, as_type);
> > > >
> > > >  	dw_pcie_writel_dbi(pci, PCIE_ATU_VIEWPORT,
> > > PCIE_ATU_REGION_INBOUND |
> > > > @@ -403,9 +424,11 @@ int dw_pcie_prog_inbound_atu(struct dw_pcie
> > > *pci, int index, int bar,
> > > >  		return -EINVAL;
> > > >  	}
> > > >
> > > > -	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type);
> > > > -	dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE
> > > > -			   | PCIE_ATU_BAR_MODE_ENABLE | (bar << 8));
> > > > +	dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type |
> > > > +			   PCIE_ATU_FUNC_NUM(func_no));
> > > > +	dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE |
> > > > +			   PCIE_ATU_FUNC_NUM_MATCH_EN |
> > > > +			   PCIE_ATU_BAR_MODE_ENABLE | (bar << 8));
> > > >
> > > >  	/*
> > > >  	 * Make sure ATU enable takes effect before any subsequent config
> > > > diff --git a/drivers/pci/controller/dwc/pcie-designware.h
> > > > b/drivers/pci/controller/dwc/pcie-designware.h
> > > > index 5a18e94..6aca0bb 100644
> > > > --- a/drivers/pci/controller/dwc/pcie-designware.h
> > > > +++ b/drivers/pci/controller/dwc/pcie-designware.h
> > > > @@ -71,9 +71,11 @@
> > > >  #define PCIE_ATU_TYPE_IO		0x2
> > > >  #define PCIE_ATU_TYPE_CFG0		0x4
> > > >  #define PCIE_ATU_TYPE_CFG1		0x5
> > > > +#define PCIE_ATU_FUNC_NUM(pf)           ((pf) << 20)
> > > >  #define PCIE_ATU_CR2			0x908
> > > >  #define PCIE_ATU_ENABLE			BIT(31)
> > > >  #define PCIE_ATU_BAR_MODE_ENABLE	BIT(30)
> > > > +#define PCIE_ATU_FUNC_NUM_MATCH_EN      BIT(19)
> > > >  #define PCIE_ATU_LOWER_BASE		0x90C
> > > >  #define PCIE_ATU_UPPER_BASE		0x910
> > > >  #define PCIE_ATU_LIMIT			0x914
> > > > @@ -206,6 +208,14 @@ struct dw_pcie_ep_ops {
> > > >  	int	(*raise_irq)(struct dw_pcie_ep *ep, u8 func_no,
> > > >  			     enum pci_epc_irq_type type, u16 interrupt_num);
> > > >  	const struct pci_epc_features* (*get_features)(struct dw_pcie_ep
> > > > *ep);
> > > > +	/*
> > > > +	 * Provide a method to implement the different func config space
> > > > +	 * access for different platform, if different func have different
> > > > +	 * offset, return the offset of func. if use write a register way
> > > > +	 * return a 0, and implement code in callback function of platform
> > > > +	 * driver.
> > > > +	 */
> > > > +	unsigned int (*func_conf_select)(struct dw_pcie_ep *ep, u8 func_no);
> > > >  };
> > > >
> > > >  struct dw_pcie_ep {
> > > > @@ -277,8 +287,12 @@ int dw_pcie_wait_for_link(struct dw_pcie *pci);
> > > > void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index,
> > > >  			       int type, u64 cpu_addr, u64 pci_addr,
> > > >  			       u32 size);
> > > > -int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int bar,
> > > > -			     u64 cpu_addr, enum dw_pcie_as_type as_type);
> > > > +void dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int
> > > index,
> > > > +				  int type, u64 cpu_addr, u64 pci_addr,
> > > > +				  u32 size);
> > > > +int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> > > > +			     int bar, u64 cpu_addr,
> > > > +			     enum dw_pcie_as_type as_type);
> > > >  void dw_pcie_disable_atu(struct dw_pcie *pci, int index,
> > > >  			 enum dw_pcie_region_type type);
> > > >  void dw_pcie_setup(struct dw_pcie *pci);
> > > > --
> > > > 2.9.5
> > > >
> > > >
> > > > _______________________________________________
> > > > linux-arm-kernel mailing list
> > > > linux-arm-kernel@lists.infradead.org
> > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__eur01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Flists&d=DwIDaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=5KEPBnDGAa12Ywyt1Lmak3JdU8Sb2dXNFJ0Q8OZFOrk&s=SOL5A-18vfzblTMMJojZKBLwCjZIi9g5VBreF5sN1do&e= 
> > > > .infradead.org%2Fmailman%2Flistinfo%2Flinux-arm-kernel&amp;data=02%
> > > 7C0
> > > >
> > > 1%7Cxiaowei.bao%40nxp.com%7C99eef14a525040ed3eab08d72fc244f1%7C
> > > 686ea1d
> > > >
> > > 3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637030383769341270&amp;sd
> > > ata=ck2EC
> > > > %2FJYCjWErvbUM%2FT%2BoVMANMwyLRI4gVRssdnd04w%3D&amp;reser
> > > ved=0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 42+ messages in thread

end of thread, other threads:[~2019-09-26 13:38 UTC | newest]

Thread overview: 42+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-02  3:17 [PATCH v3 00/11] *** SUBJECT HERE *** Xiaowei Bao
2019-09-02  3:17 ` [PATCH v3 01/11] PCI: designware-ep: Add multiple PFs support for DWC Xiaowei Bao
2019-09-02 16:26   ` Andrew Murray
2019-09-03  3:43     ` Xiaowei Bao
2019-09-26 10:29       ` Andrew Murray
2019-09-26 13:38         ` Gustavo Pimentel
2019-09-02  3:17 ` [PATCH v3 02/11] PCI: designware-ep: Add the doorbell mode of MSI-X in EP mode Xiaowei Bao
2019-09-02  3:17 ` [PATCH v3 03/11] PCI: designware-ep: Move the function of getting MSI capability forward Xiaowei Bao
2019-09-02  3:17 ` [PATCH v3 04/11] PCI: designware-ep: Modify MSI and MSIX CAP way of finding Xiaowei Bao
2019-09-02 15:07   ` Andrew Murray
2019-09-03  2:33     ` Xiaowei Bao
2019-09-02  3:17 ` [PATCH v3 05/11] dt-bindings: pci: layerscape-pci: add compatible strings for ls1088a and ls2088a Xiaowei Bao
2019-09-02 12:31   ` Andrew Murray
2019-09-03  1:33     ` Xiaowei Bao
2019-09-02  3:17 ` [PATCH v3 06/11] PCI: layerscape: Fix some format issue of the code Xiaowei Bao
2019-09-02  3:17 ` [PATCH v3 07/11] PCI: layerscape: Modify the way of getting capability with different PEX Xiaowei Bao
2019-09-02 13:37   ` Andrew Murray
2019-09-03  2:13     ` Xiaowei Bao
2019-09-02  3:17 ` [PATCH v3 08/11] PCI: layerscape: Modify the MSIX to the doorbell mode Xiaowei Bao
2019-09-02 12:01   ` Andrew Murray
2019-09-12 11:24     ` Gustavo Pimentel
2019-09-14  6:37       ` Xiaowei Bao
2019-09-16  8:54         ` Gustavo Pimentel
2019-09-02  3:17 ` [PATCH v3 09/11] PCI: layerscape: Add EP mode support for ls1088a and ls2088a Xiaowei Bao
2019-09-02 12:46   ` Andrew Murray
2019-09-03  1:47     ` Xiaowei Bao
2019-09-12 12:49       ` Andrew Murray
2019-09-14  4:10         ` Xiaowei Bao
2019-09-16 14:37           ` Andrew Murray
2019-09-18  3:17             ` Xiaowei Bao
2019-09-02  3:17 ` [PATCH v3 10/11] arm64: dts: layerscape: Add PCIe EP node for ls1088a Xiaowei Bao
2019-09-02 13:06   ` Andrew Murray
2019-09-03  2:01     ` Xiaowei Bao
2019-09-12 13:01       ` Andrew Murray
2019-09-14  4:15         ` Xiaowei Bao
2019-09-02  3:17 ` [PATCH v3 11/11] misc: pci_endpoint_test: Add LS1088a in pci_device_id table Xiaowei Bao
2019-09-02 12:54   ` Andrew Murray
2019-09-03  1:52     ` Xiaowei Bao
2019-09-12 12:59       ` Andrew Murray
2019-09-14  4:13         ` Xiaowei Bao
2019-09-02  3:52 ` [PATCH v3 00/11] *** SUBJECT HERE *** Z.q. Hou
2019-09-02  3:54   ` Xiaowei Bao

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).