[PATCH v16 00/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe support

[PATCH v16 00/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe support

[Yoshihiro Shimoda]

Add R-Car S4-8 (R-Car Gen4) PCIe Host and Endpoint support.
To support them, modify PCIe DesignWare common codes.

Changes from v15:
https://lore.kernel.org/linux-pci/20230509124156.150200-1-yoshihiro.shimoda.uh@renesas.com/
 - Based on next-20230509 + pci.git / next branch (the commit 174977dc80b7
   ("Merge branch 'pci/controller/vmd'"))
 - (no change, JFYI) Based on the following cleanups patches:
   [PATCH v4 00/14] PCI: dwc: Relatively simple fixes and cleanups
   https://lore.kernel.org/linux-pci/20230414021832.13167-1-Sergey.Semin@baikalelectronics.ru/
 - Modify the code comments in patch 8/22.

Changes from v14:
https://lore.kernel.org/linux-pci/20230426045557.3613826-1-yoshihiro.shimoda.uh@renesas.com/
 - Based on next-20230508.
 - (no change, JFYI) Based on the following cleanups patches:
   [PATCH v4 00/14] PCI: dwc: Relatively simple fixes and cleanups
   https://lore.kernel.org/linux-pci/20230414021832.13167-1-Sergey.Semin@baikalelectronics.ru/
 - Add Reviewed-by from Serge in the patch {4,5,15,}/21.
 - Drop PCI_EXP_LNKCAP_MLW handling of pcie-tegra194.c because
   pcie-designware.c takes care of it.
 - Change subjects in the patch {5,6,7,8,10}/21.
 - Drop dw_pcie_prog_ep_outbound_atu().
 - Modify dw_pcie_link_set_max_link_width() to improve code readability.
 - Move the retrain code to .start_link().
 - Fix some minor issues.

Yoshihiro Shimoda (22):
  PCI: Add PCI_EXP_LNKCAP_MLW macros
  PCI: Add PCI_HEADER_TYPE_MULTI_FUNC
  PCI: Add INTx Mechanism Messages macros
  PCI: Rename PCI_EPC_IRQ_LEGACY to PCI_EPC_IRQ_INTX
  PCI: dwc: Rename "legacy_irq" to "INTx_irq"
  PCI: dwc: Change arguments of dw_pcie_prog_outbound_atu()
  PCI: dwc: Add outbound MSG TLPs support
  PCI: designware-ep: Add INTx IRQs support
  PCI: dwc: Add dw_pcie_link_set_max_link_width()
  PCI: dwc: Modify PCIE_PORT_LINK_CONTROL handling
  PCI: dwc: Add dw_pcie_link_set_max_cap_width()
  PCI: tegra194: Drop PCI_EXP_LNKSTA_NLW setting.
  PCI: dwc: Add EDMA_UNROLL capability flag
  PCI: dwc: Expose dw_pcie_ep_exit() to module
  PCI: dwc: Introduce .ep_pre_init() and .ep_deinit()
  dt-bindings: PCI: dwc: Update maxItems of reg and reg-names
  dt-bindings: PCI: renesas: Add R-Car Gen4 PCIe Host
  dt-bindings: PCI: renesas: Add R-Car Gen4 PCIe Endpoint
  PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support
  PCI: rcar-gen4-ep: Add R-Car Gen4 PCIe Endpoint support
  MAINTAINERS: Update PCI DRIVER FOR RENESAS R-CAR for R-Car Gen4
  misc: pci_endpoint_test: Add Device ID for R-Car S4-8 PCIe controller

 .../bindings/pci/rcar-gen4-pci-ep.yaml        |  98 +++++++++
 .../bindings/pci/rcar-gen4-pci-host.yaml      | 109 ++++++++++
 .../bindings/pci/snps,dw-pcie-ep.yaml         |   4 +-
 .../devicetree/bindings/pci/snps,dw-pcie.yaml |   4 +-
 MAINTAINERS                                   |   1 +
 drivers/misc/pci_endpoint_test.c              |   4 +
 .../pci/controller/cadence/pcie-cadence-ep.c  |   2 +-
 drivers/pci/controller/dwc/Kconfig            |  18 ++
 drivers/pci/controller/dwc/Makefile           |   4 +
 drivers/pci/controller/dwc/pci-dra7xx.c       |   2 +-
 drivers/pci/controller/dwc/pci-imx6.c         |   4 +-
 drivers/pci/controller/dwc/pci-keystone.c     |   2 +-
 .../pci/controller/dwc/pci-layerscape-ep.c    |   4 +-
 drivers/pci/controller/dwc/pcie-artpec6.c     |   2 +-
 .../pci/controller/dwc/pcie-designware-ep.c   |  98 +++++++--
 .../pci/controller/dwc/pcie-designware-host.c |  52 +++--
 .../pci/controller/dwc/pcie-designware-plat.c |   4 +-
 drivers/pci/controller/dwc/pcie-designware.c  | 160 ++++++++-------
 drivers/pci/controller/dwc/pcie-designware.h  |  33 ++-
 drivers/pci/controller/dwc/pcie-keembay.c     |   2 +-
 drivers/pci/controller/dwc/pcie-qcom-ep.c     |   4 +-
 .../pci/controller/dwc/pcie-rcar-gen4-ep.c    | 166 +++++++++++++++
 .../pci/controller/dwc/pcie-rcar-gen4-host.c  | 141 +++++++++++++
 drivers/pci/controller/dwc/pcie-rcar-gen4.c   | 190 ++++++++++++++++++
 drivers/pci/controller/dwc/pcie-rcar-gen4.h   |  46 +++++
 drivers/pci/controller/dwc/pcie-tegra194.c    |   8 +-
 drivers/pci/controller/dwc/pcie-uniphier-ep.c |   2 +-
 drivers/pci/controller/pcie-rcar-ep.c         |   2 +-
 drivers/pci/controller/pcie-rockchip-ep.c     |   2 +-
 drivers/pci/endpoint/functions/pci-epf-test.c |  10 +-
 drivers/pci/pci.h                             |  18 ++
 drivers/pci/probe.c                           |   2 +-
 drivers/pci/quirks.c                          |   4 +-
 include/linux/pci-epc.h                       |   4 +-
 include/uapi/linux/pci_regs.h                 |   7 +
 35 files changed, 1061 insertions(+), 152 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml
 create mode 100644 Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.yaml
 create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4-ep.c
 create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
 create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.c
 create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.h

-- 
2.25.1

[PATCH v16 01/22] PCI: Add PCI_EXP_LNKCAP_MLW macros

[Yoshihiro Shimoda]

Add macros defining Maximum Link Width bits in Link Capabilities
Register.

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
---
 include/uapi/linux/pci_regs.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
index dc2000e0fe3a..5d48413ac28f 100644
--- a/include/uapi/linux/pci_regs.h
+++ b/include/uapi/linux/pci_regs.h
@@ -538,6 +538,12 @@
 #define  PCI_EXP_LNKCAP_SLS_16_0GB 0x00000004 /* LNKCAP2 SLS Vector bit 3 */
 #define  PCI_EXP_LNKCAP_SLS_32_0GB 0x00000005 /* LNKCAP2 SLS Vector bit 4 */
 #define  PCI_EXP_LNKCAP_SLS_64_0GB 0x00000006 /* LNKCAP2 SLS Vector bit 5 */
+#define  PCI_EXP_LNKCAP_MLW_X1	0x00000010 /* Maximum Link Width x1 */
+#define  PCI_EXP_LNKCAP_MLW_X2	0x00000020 /* Maximum Link Width x2 */
+#define  PCI_EXP_LNKCAP_MLW_X4	0x00000040 /* Maximum Link Width x4 */
+#define  PCI_EXP_LNKCAP_MLW_X8	0x00000080 /* Maximum Link Width x8 */
+#define  PCI_EXP_LNKCAP_MLW_X12	0x000000c0 /* Maximum Link Width x12 */
+#define  PCI_EXP_LNKCAP_MLW_X16	0x00000100 /* Maximum Link Width x16 */
 #define  PCI_EXP_LNKCAP_MLW	0x000003f0 /* Maximum Link Width */
 #define  PCI_EXP_LNKCAP_ASPMS	0x00000c00 /* ASPM Support */
 #define  PCI_EXP_LNKCAP_ASPM_L0S 0x00000400 /* ASPM L0s Support */
-- 
2.25.1

[PATCH v16 02/22] PCI: Add PCI_HEADER_TYPE_MULTI_FUNC

[Yoshihiro Shimoda]

Add PCI_HEADER_TYPE_MULTI_FUNC macro which is "Multi-Function Device"
of Header Type Register.

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
 drivers/pci/probe.c           | 2 +-
 drivers/pci/quirks.c          | 4 ++--
 include/uapi/linux/pci_regs.h | 1 +
 3 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 0b2826c4a832..56f01b48fb81 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1836,7 +1836,7 @@ int pci_setup_device(struct pci_dev *dev)
 	dev->dev.parent = dev->bus->bridge;
 	dev->dev.bus = &pci_bus_type;
 	dev->hdr_type = hdr_type & 0x7f;
-	dev->multifunction = !!(hdr_type & 0x80);
+	dev->multifunction = !!(hdr_type & PCI_HEADER_TYPE_MULTI_FUNC);
 	dev->error_state = pci_channel_io_normal;
 	set_pcie_port_type(dev);
 
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index c1239706eeaf..ce8bc6788674 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -1750,7 +1750,7 @@ static void quirk_jmicron_ata(struct pci_dev *pdev)
 	/* Update pdev accordingly */
 	pci_read_config_byte(pdev, PCI_HEADER_TYPE, &hdr);
 	pdev->hdr_type = hdr & 0x7f;
-	pdev->multifunction = !!(hdr & 0x80);
+	pdev->multifunction = !!(hdr & PCI_HEADER_TYPE_MULTI_FUNC);
 
 	pci_read_config_dword(pdev, PCI_CLASS_REVISION, &class);
 	pdev->class = class >> 8;
@@ -5569,7 +5569,7 @@ static void quirk_nvidia_hda(struct pci_dev *gpu)
 
 	/* The GPU becomes a multi-function device when the HDA is enabled */
 	pci_read_config_byte(gpu, PCI_HEADER_TYPE, &hdr_type);
-	gpu->multifunction = !!(hdr_type & 0x80);
+	gpu->multifunction = !!(hdr_type & PCI_HEADER_TYPE_MULTI_FUNC);
 }
 DECLARE_PCI_FIXUP_CLASS_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
 			       PCI_BASE_CLASS_DISPLAY, 16, quirk_nvidia_hda);
diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
index 5d48413ac28f..a302b67d2834 100644
--- a/include/uapi/linux/pci_regs.h
+++ b/include/uapi/linux/pci_regs.h
@@ -80,6 +80,7 @@
 #define  PCI_HEADER_TYPE_NORMAL		0
 #define  PCI_HEADER_TYPE_BRIDGE		1
 #define  PCI_HEADER_TYPE_CARDBUS	2
+#define  PCI_HEADER_TYPE_MULTI_FUNC	0x80
 
 #define PCI_BIST		0x0f	/* 8 bits */
 #define  PCI_BIST_CODE_MASK	0x0f	/* Return result */
-- 
2.25.1

[PATCH v16 03/22] PCI: Add INTx Mechanism Messages macros

[Yoshihiro Shimoda]

Add "Message Routing" and "INTx Mechanism Messages" macros to send
a message by a PCIe driver.

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
 drivers/pci/pci.h | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 2475098f6518..67badc40e90b 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -11,6 +11,24 @@
 
 #define PCI_VSEC_ID_INTEL_TBT	0x1234	/* Thunderbolt */
 
+/* Message Routing */
+#define PCI_MSG_ROUTING_RC	0
+#define PCI_MSG_ROUTING_ADDR	1
+#define PCI_MSG_ROUTING_ID	2
+#define PCI_MSG_ROUTING_BC	3
+#define PCI_MSG_ROUTING_LOCAL	4
+#define PCI_MSG_ROUTING_GATHER	5
+
+/* INTx Mechanism Messages */
+#define PCI_CODE_ASSERT_INTA	0x20
+#define PCI_CODE_ASSERT_INTB	0x21
+#define PCI_CODE_ASSERT_INTC	0x22
+#define PCI_CODE_ASSERT_INTD	0x23
+#define PCI_CODE_DEASSERT_INTA	0x24
+#define PCI_CODE_DEASSERT_INTB	0x25
+#define PCI_CODE_DEASSERT_INTC	0x26
+#define PCI_CODE_DEASSERT_INTD	0x27
+
 extern const unsigned char pcie_link_speed[];
 extern bool pci_early_dump;
 
-- 
2.25.1

[PATCH v16 04/22] PCI: Rename PCI_EPC_IRQ_LEGACY to PCI_EPC_IRQ_INTX

[Yoshihiro Shimoda]

Using "INTx" instead of "legacy" is more specific. So, rename
PCI_EPC_IRQ_LEGACY to PCI_EPC_IRQ_INTX.

Suggested-by: Bjorn Helgaas <helgaas@kernel.org>
Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Cc: Tom Joseph <tjoseph@cadence.com>
Cc: Vignesh Raghavendra <vigneshr@ti.com>
Cc: Richard Zhu <hongxing.zhu@nxp.com>
Cc: Lucas Stach <l.stach@pengutronix.de>
Cc: Shawn Guo <shawnguo@kernel.org>
Cc: Sascha Hauer <s.hauer@pengutronix.de>
Cc: Pengutronix Kernel Team <kernel@pengutronix.de>
Cc: Fabio Estevam <festevam@gmail.com>
Cc: NXP Linux Team <linux-imx@nxp.com>
Cc: Minghuan Lian <minghuan.Lian@nxp.com>
Cc: Mingkai Hu <mingkai.hu@nxp.com>
Cc: Roy Zang <roy.zang@nxp.com>
Cc: Jingoo Han <jingoohan1@gmail.com>
Cc: Srikanth Thokala <srikanth.thokala@intel.com>
Cc: Thierry Reding <thierry.reding@gmail.com>
Cc: Jonathan Hunter <jonathanh@nvidia.com>
Cc: Kunihiko Hayashi <hayashi.kunihiko@socionext.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Marek Vasut <marek.vasut+renesas@gmail.com>
Cc: Shawn Lin <shawn.lin@rock-chips.com>
Cc: Heiko Stuebner <heiko@sntech.de>
Cc: Kishon Vijay Abraham I <kishon@kernel.org>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Acked-by: Jesper Nilsson <jesper.nilsson@axis.com> # ARTPEC
Reviewed-by: Serge Semin <fancer.lancer@gmail.com>
---
 drivers/pci/controller/cadence/pcie-cadence-ep.c  |  2 +-
 drivers/pci/controller/dwc/pci-dra7xx.c           |  2 +-
 drivers/pci/controller/dwc/pci-imx6.c             |  2 +-
 drivers/pci/controller/dwc/pci-keystone.c         |  2 +-
 drivers/pci/controller/dwc/pci-layerscape-ep.c    |  2 +-
 drivers/pci/controller/dwc/pcie-artpec6.c         |  2 +-
 drivers/pci/controller/dwc/pcie-designware-plat.c |  2 +-
 drivers/pci/controller/dwc/pcie-keembay.c         |  2 +-
 drivers/pci/controller/dwc/pcie-qcom-ep.c         |  2 +-
 drivers/pci/controller/dwc/pcie-tegra194.c        |  2 +-
 drivers/pci/controller/dwc/pcie-uniphier-ep.c     |  2 +-
 drivers/pci/controller/pcie-rcar-ep.c             |  2 +-
 drivers/pci/controller/pcie-rockchip-ep.c         |  2 +-
 drivers/pci/endpoint/functions/pci-epf-test.c     | 10 +++++-----
 include/linux/pci-epc.h                           |  4 ++--
 15 files changed, 20 insertions(+), 20 deletions(-)

diff --git a/drivers/pci/controller/cadence/pcie-cadence-ep.c b/drivers/pci/controller/cadence/pcie-cadence-ep.c
index b8b655d4047e..2af8eb4e6d91 100644
--- a/drivers/pci/controller/cadence/pcie-cadence-ep.c
+++ b/drivers/pci/controller/cadence/pcie-cadence-ep.c
@@ -539,7 +539,7 @@ static int cdns_pcie_ep_raise_irq(struct pci_epc *epc, u8 fn, u8 vfn,
 	struct device *dev = pcie->dev;
 
 	switch (type) {
-	case PCI_EPC_IRQ_LEGACY:
+	case PCI_EPC_IRQ_INTX:
 		if (vfn > 0) {
 			dev_err(dev, "Cannot raise legacy interrupts for VF\n");
 			return -EINVAL;
diff --git a/drivers/pci/controller/dwc/pci-dra7xx.c b/drivers/pci/controller/dwc/pci-dra7xx.c
index 4ae807e7cf79..b42fb1cc8bc8 100644
--- a/drivers/pci/controller/dwc/pci-dra7xx.c
+++ b/drivers/pci/controller/dwc/pci-dra7xx.c
@@ -410,7 +410,7 @@ static int dra7xx_pcie_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 	struct dra7xx_pcie *dra7xx = to_dra7xx_pcie(pci);
 
 	switch (type) {
-	case PCI_EPC_IRQ_LEGACY:
+	case PCI_EPC_IRQ_INTX:
 		dra7xx_pcie_raise_legacy_irq(dra7xx);
 		break;
 	case PCI_EPC_IRQ_MSI:
diff --git a/drivers/pci/controller/dwc/pci-imx6.c b/drivers/pci/controller/dwc/pci-imx6.c
index 52906f999f2b..1f39e733ce19 100644
--- a/drivers/pci/controller/dwc/pci-imx6.c
+++ b/drivers/pci/controller/dwc/pci-imx6.c
@@ -1062,7 +1062,7 @@ static int imx6_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 
 	switch (type) {
-	case PCI_EPC_IRQ_LEGACY:
+	case PCI_EPC_IRQ_INTX:
 		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
 	case PCI_EPC_IRQ_MSI:
 		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
diff --git a/drivers/pci/controller/dwc/pci-keystone.c b/drivers/pci/controller/dwc/pci-keystone.c
index 78818853af9e..3806f5530937 100644
--- a/drivers/pci/controller/dwc/pci-keystone.c
+++ b/drivers/pci/controller/dwc/pci-keystone.c
@@ -908,7 +908,7 @@ static int ks_pcie_am654_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 	struct keystone_pcie *ks_pcie = to_keystone_pcie(pci);
 
 	switch (type) {
-	case PCI_EPC_IRQ_LEGACY:
+	case PCI_EPC_IRQ_INTX:
 		ks_pcie_am654_raise_legacy_irq(ks_pcie);
 		break;
 	case PCI_EPC_IRQ_MSI:
diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c b/drivers/pci/controller/dwc/pci-layerscape-ep.c
index c640db60edc6..ab3306e206d8 100644
--- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
+++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
@@ -65,7 +65,7 @@ static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 
 	switch (type) {
-	case PCI_EPC_IRQ_LEGACY:
+	case PCI_EPC_IRQ_INTX:
 		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
 	case PCI_EPC_IRQ_MSI:
 		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
diff --git a/drivers/pci/controller/dwc/pcie-artpec6.c b/drivers/pci/controller/dwc/pcie-artpec6.c
index 98102079e26d..128cb1118e3a 100644
--- a/drivers/pci/controller/dwc/pcie-artpec6.c
+++ b/drivers/pci/controller/dwc/pcie-artpec6.c
@@ -357,7 +357,7 @@ static int artpec6_pcie_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 
 	switch (type) {
-	case PCI_EPC_IRQ_LEGACY:
+	case PCI_EPC_IRQ_INTX:
 		dev_err(pci->dev, "EP cannot trigger legacy IRQs\n");
 		return -EINVAL;
 	case PCI_EPC_IRQ_MSI:
diff --git a/drivers/pci/controller/dwc/pcie-designware-plat.c b/drivers/pci/controller/dwc/pcie-designware-plat.c
index 1fcfb840f238..fc3b02949218 100644
--- a/drivers/pci/controller/dwc/pcie-designware-plat.c
+++ b/drivers/pci/controller/dwc/pcie-designware-plat.c
@@ -48,7 +48,7 @@ static int dw_plat_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 
 	switch (type) {
-	case PCI_EPC_IRQ_LEGACY:
+	case PCI_EPC_IRQ_INTX:
 		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
 	case PCI_EPC_IRQ_MSI:
 		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
diff --git a/drivers/pci/controller/dwc/pcie-keembay.c b/drivers/pci/controller/dwc/pcie-keembay.c
index f90f36bac018..ceb940b327cb 100644
--- a/drivers/pci/controller/dwc/pcie-keembay.c
+++ b/drivers/pci/controller/dwc/pcie-keembay.c
@@ -290,7 +290,7 @@ static int keembay_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 
 	switch (type) {
-	case PCI_EPC_IRQ_LEGACY:
+	case PCI_EPC_IRQ_INTX:
 		/* Legacy interrupts are not supported in Keem Bay */
 		dev_err(pci->dev, "Legacy IRQ is not supported\n");
 		return -EINVAL;
diff --git a/drivers/pci/controller/dwc/pcie-qcom-ep.c b/drivers/pci/controller/dwc/pcie-qcom-ep.c
index 19b32839ea26..077afce48d0b 100644
--- a/drivers/pci/controller/dwc/pcie-qcom-ep.c
+++ b/drivers/pci/controller/dwc/pcie-qcom-ep.c
@@ -658,7 +658,7 @@ static int qcom_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 
 	switch (type) {
-	case PCI_EPC_IRQ_LEGACY:
+	case PCI_EPC_IRQ_INTX:
 		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
 	case PCI_EPC_IRQ_MSI:
 		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
diff --git a/drivers/pci/controller/dwc/pcie-tegra194.c b/drivers/pci/controller/dwc/pcie-tegra194.c
index 09825b4a075e..4adba379b83d 100644
--- a/drivers/pci/controller/dwc/pcie-tegra194.c
+++ b/drivers/pci/controller/dwc/pcie-tegra194.c
@@ -1980,7 +1980,7 @@ static int tegra_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 	struct tegra_pcie_dw *pcie = to_tegra_pcie(pci);
 
 	switch (type) {
-	case PCI_EPC_IRQ_LEGACY:
+	case PCI_EPC_IRQ_INTX:
 		return tegra_pcie_ep_raise_legacy_irq(pcie, interrupt_num);
 
 	case PCI_EPC_IRQ_MSI:
diff --git a/drivers/pci/controller/dwc/pcie-uniphier-ep.c b/drivers/pci/controller/dwc/pcie-uniphier-ep.c
index 4d0a587c0ba5..7787eedf87f4 100644
--- a/drivers/pci/controller/dwc/pcie-uniphier-ep.c
+++ b/drivers/pci/controller/dwc/pcie-uniphier-ep.c
@@ -262,7 +262,7 @@ static int uniphier_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 
 	switch (type) {
-	case PCI_EPC_IRQ_LEGACY:
+	case PCI_EPC_IRQ_INTX:
 		return uniphier_pcie_ep_raise_legacy_irq(ep);
 	case PCI_EPC_IRQ_MSI:
 		return uniphier_pcie_ep_raise_msi_irq(ep, func_no,
diff --git a/drivers/pci/controller/pcie-rcar-ep.c b/drivers/pci/controller/pcie-rcar-ep.c
index f9682df1da61..fbdf3d85301c 100644
--- a/drivers/pci/controller/pcie-rcar-ep.c
+++ b/drivers/pci/controller/pcie-rcar-ep.c
@@ -408,7 +408,7 @@ static int rcar_pcie_ep_raise_irq(struct pci_epc *epc, u8 fn, u8 vfn,
 	struct rcar_pcie_endpoint *ep = epc_get_drvdata(epc);
 
 	switch (type) {
-	case PCI_EPC_IRQ_LEGACY:
+	case PCI_EPC_IRQ_INTX:
 		return rcar_pcie_ep_assert_intx(ep, fn, 0);
 
 	case PCI_EPC_IRQ_MSI:
diff --git a/drivers/pci/controller/pcie-rockchip-ep.c b/drivers/pci/controller/pcie-rockchip-ep.c
index d1a200b93b2b..ef9d1f6c382a 100644
--- a/drivers/pci/controller/pcie-rockchip-ep.c
+++ b/drivers/pci/controller/pcie-rockchip-ep.c
@@ -477,7 +477,7 @@ static int rockchip_pcie_ep_raise_irq(struct pci_epc *epc, u8 fn, u8 vfn,
 	struct rockchip_pcie_ep *ep = epc_get_drvdata(epc);
 
 	switch (type) {
-	case PCI_EPC_IRQ_LEGACY:
+	case PCI_EPC_IRQ_INTX:
 		return rockchip_pcie_ep_send_legacy_irq(ep, fn, 0);
 	case PCI_EPC_IRQ_MSI:
 		return rockchip_pcie_ep_send_msi_irq(ep, fn, interrupt_num);
diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c b/drivers/pci/endpoint/functions/pci-epf-test.c
index 623b08caa998..6beb3f2b0afb 100644
--- a/drivers/pci/endpoint/functions/pci-epf-test.c
+++ b/drivers/pci/endpoint/functions/pci-epf-test.c
@@ -19,11 +19,11 @@
 #include <linux/pci-epf.h>
 #include <linux/pci_regs.h>
 
-#define IRQ_TYPE_LEGACY			0
+#define IRQ_TYPE_INTX			0
 #define IRQ_TYPE_MSI			1
 #define IRQ_TYPE_MSIX			2
 
-#define COMMAND_RAISE_LEGACY_IRQ	BIT(0)
+#define COMMAND_RAISE_INTX_IRQ		BIT(0)
 #define COMMAND_RAISE_MSI_IRQ		BIT(1)
 #define COMMAND_RAISE_MSIX_IRQ		BIT(2)
 #define COMMAND_READ			BIT(3)
@@ -600,9 +600,9 @@ static void pci_epf_test_raise_irq(struct pci_epf_test *epf_test,
 	WRITE_ONCE(reg->status, status);
 
 	switch (reg->irq_type) {
-	case IRQ_TYPE_LEGACY:
+	case IRQ_TYPE_INTX:
 		pci_epc_raise_irq(epc, epf->func_no, epf->vfunc_no,
-				  PCI_EPC_IRQ_LEGACY, 0);
+				  PCI_EPC_IRQ_INTX, 0);
 		break;
 	case IRQ_TYPE_MSI:
 		count = pci_epc_get_msi(epc, epf->func_no, epf->vfunc_no);
@@ -659,7 +659,7 @@ static void pci_epf_test_cmd_handler(struct work_struct *work)
 	}
 
 	switch (command) {
-	case COMMAND_RAISE_LEGACY_IRQ:
+	case COMMAND_RAISE_INTX_IRQ:
 	case COMMAND_RAISE_MSI_IRQ:
 	case COMMAND_RAISE_MSIX_IRQ:
 		pci_epf_test_raise_irq(epf_test, reg);
diff --git a/include/linux/pci-epc.h b/include/linux/pci-epc.h
index 301bb0e53707..c2572a93d73d 100644
--- a/include/linux/pci-epc.h
+++ b/include/linux/pci-epc.h
@@ -21,7 +21,7 @@ enum pci_epc_interface_type {
 
 enum pci_epc_irq_type {
 	PCI_EPC_IRQ_UNKNOWN,
-	PCI_EPC_IRQ_LEGACY,
+	PCI_EPC_IRQ_INTX,
 	PCI_EPC_IRQ_MSI,
 	PCI_EPC_IRQ_MSIX,
 };
@@ -54,7 +54,7 @@ pci_epc_interface_string(enum pci_epc_interface_type type)
  *	     MSI-X capability register
  * @get_msix: ops to get the number of MSI-X interrupts allocated by the RC
  *	     from the MSI-X capability register
- * @raise_irq: ops to raise a legacy, MSI or MSI-X interrupt
+ * @raise_irq: ops to raise an INTx, MSI or MSI-X interrupt
  * @map_msi_irq: ops to map physical address to MSI address and return MSI data
  * @start: ops to start the PCI link
  * @stop: ops to stop the PCI link
-- 
2.25.1

[PATCH v16 05/22] PCI: dwc: Rename "legacy_irq" to "INTx_irq"

[Yoshihiro Shimoda]

Using "INTx" instead of "legacy" is more specific. So, rename
dw_pcie_ep_raise_legacy_irq() to dw_pcie_ep_raise_intx_irq().

Suggested-by: Bjorn Helgaas <helgaas@kernel.org>
Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Reviewed-by: Serge Semin <fancer.lancer@gmail.com>
---
 drivers/pci/controller/dwc/pci-imx6.c             | 2 +-
 drivers/pci/controller/dwc/pci-layerscape-ep.c    | 2 +-
 drivers/pci/controller/dwc/pcie-designware-ep.c   | 6 +++---
 drivers/pci/controller/dwc/pcie-designware-plat.c | 2 +-
 drivers/pci/controller/dwc/pcie-designware.h      | 4 ++--
 drivers/pci/controller/dwc/pcie-qcom-ep.c         | 2 +-
 6 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/drivers/pci/controller/dwc/pci-imx6.c b/drivers/pci/controller/dwc/pci-imx6.c
index 1f39e733ce19..0831f3947220 100644
--- a/drivers/pci/controller/dwc/pci-imx6.c
+++ b/drivers/pci/controller/dwc/pci-imx6.c
@@ -1063,7 +1063,7 @@ static int imx6_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 
 	switch (type) {
 	case PCI_EPC_IRQ_INTX:
-		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
+		return dw_pcie_ep_raise_intx_irq(ep, func_no);
 	case PCI_EPC_IRQ_MSI:
 		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
 	case PCI_EPC_IRQ_MSIX:
diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c b/drivers/pci/controller/dwc/pci-layerscape-ep.c
index ab3306e206d8..3d58fc1670b4 100644
--- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
+++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
@@ -66,7 +66,7 @@ static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 
 	switch (type) {
 	case PCI_EPC_IRQ_INTX:
-		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
+		return dw_pcie_ep_raise_intx_irq(ep, func_no);
 	case PCI_EPC_IRQ_MSI:
 		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
 	case PCI_EPC_IRQ_MSIX:
diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index f9182f8d552f..27278010ecec 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -479,16 +479,16 @@ static const struct pci_epc_ops epc_ops = {
 	.get_features		= dw_pcie_ep_get_features,
 };
 
-int dw_pcie_ep_raise_legacy_irq(struct dw_pcie_ep *ep, u8 func_no)
+int dw_pcie_ep_raise_intx_irq(struct dw_pcie_ep *ep, u8 func_no)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	struct device *dev = pci->dev;
 
-	dev_err(dev, "EP cannot trigger legacy IRQs\n");
+	dev_err(dev, "EP cannot trigger INTx IRQs\n");
 
 	return -EINVAL;
 }
-EXPORT_SYMBOL_GPL(dw_pcie_ep_raise_legacy_irq);
+EXPORT_SYMBOL_GPL(dw_pcie_ep_raise_intx_irq);
 
 int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
 			     u8 interrupt_num)
diff --git a/drivers/pci/controller/dwc/pcie-designware-plat.c b/drivers/pci/controller/dwc/pcie-designware-plat.c
index fc3b02949218..2689ff7939e4 100644
--- a/drivers/pci/controller/dwc/pcie-designware-plat.c
+++ b/drivers/pci/controller/dwc/pcie-designware-plat.c
@@ -49,7 +49,7 @@ static int dw_plat_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 
 	switch (type) {
 	case PCI_EPC_IRQ_INTX:
-		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
+		return dw_pcie_ep_raise_intx_irq(ep, func_no);
 	case PCI_EPC_IRQ_MSI:
 		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
 	case PCI_EPC_IRQ_MSIX:
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index adad0ea61799..9acf6c40d252 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -550,7 +550,7 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep);
 int dw_pcie_ep_init_complete(struct dw_pcie_ep *ep);
 void dw_pcie_ep_init_notify(struct dw_pcie_ep *ep);
 void dw_pcie_ep_exit(struct dw_pcie_ep *ep);
-int dw_pcie_ep_raise_legacy_irq(struct dw_pcie_ep *ep, u8 func_no);
+int dw_pcie_ep_raise_intx_irq(struct dw_pcie_ep *ep, u8 func_no);
 int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
 			     u8 interrupt_num);
 int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
@@ -583,7 +583,7 @@ static inline void dw_pcie_ep_exit(struct dw_pcie_ep *ep)
 {
 }
 
-static inline int dw_pcie_ep_raise_legacy_irq(struct dw_pcie_ep *ep, u8 func_no)
+static inline int dw_pcie_ep_raise_intx_irq(struct dw_pcie_ep *ep, u8 func_no)
 {
 	return 0;
 }
diff --git a/drivers/pci/controller/dwc/pcie-qcom-ep.c b/drivers/pci/controller/dwc/pcie-qcom-ep.c
index 077afce48d0b..3061e5e13476 100644
--- a/drivers/pci/controller/dwc/pcie-qcom-ep.c
+++ b/drivers/pci/controller/dwc/pcie-qcom-ep.c
@@ -659,7 +659,7 @@ static int qcom_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
 
 	switch (type) {
 	case PCI_EPC_IRQ_INTX:
-		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
+		return dw_pcie_ep_raise_intx_irq(ep, func_no);
 	case PCI_EPC_IRQ_MSI:
 		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
 	default:
-- 
2.25.1

[PATCH v16 06/22] PCI: dwc: Change arguments of dw_pcie_prog_outbound_atu()

[Yoshihiro Shimoda]

To add more arguments to the dw_pcie_prog_outbound_atu() in
the future, introduce struct dw_pcie_ob_atu_cfg and change
the argument. And, drop dw_pcie_prog_ep_outbound_atu(). No behavior
changes.

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
 .../pci/controller/dwc/pcie-designware-ep.c   | 21 +++++---
 .../pci/controller/dwc/pcie-designware-host.c | 52 +++++++++++++------
 drivers/pci/controller/dwc/pcie-designware.c  | 49 ++++++-----------
 drivers/pci/controller/dwc/pcie-designware.h  | 15 ++++--
 4 files changed, 77 insertions(+), 60 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index 27278010ecec..fe2e0d765be9 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -182,9 +182,8 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no, int type,
 	return 0;
 }
 
-static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, u8 func_no,
-				   phys_addr_t phys_addr,
-				   u64 pci_addr, size_t size)
+static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep,
+				   struct dw_pcie_ob_atu_cfg *atu)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	u32 free_win;
@@ -196,13 +195,13 @@ static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, u8 func_no,
 		return -EINVAL;
 	}
 
-	ret = dw_pcie_prog_ep_outbound_atu(pci, func_no, free_win, PCIE_ATU_TYPE_MEM,
-					   phys_addr, pci_addr, size);
+	atu->index = free_win;
+	ret = dw_pcie_prog_outbound_atu(pci, atu);
 	if (ret)
 		return ret;
 
 	set_bit(free_win, ep->ob_window_map);
-	ep->outbound_addr[free_win] = phys_addr;
+	ep->outbound_addr[free_win] = atu->cpu_addr;
 
 	return 0;
 }
@@ -305,8 +304,14 @@ static int dw_pcie_ep_map_addr(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
 	int ret;
 	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
-
-	ret = dw_pcie_ep_outbound_atu(ep, func_no, addr, pci_addr, size);
+	struct dw_pcie_ob_atu_cfg atu = { 0 };
+
+	atu.func_no = func_no;
+	atu.type = PCIE_ATU_TYPE_MEM;
+	atu.cpu_addr = addr;
+	atu.pci_addr = pci_addr;
+	atu.size = size;
+	ret = dw_pcie_ep_outbound_atu(ep, &atu);
 	if (ret) {
 		dev_err(pci->dev, "Failed to enable address\n");
 		return ret;
diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index 5718b4bb67f0..676216d540fe 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -544,6 +544,7 @@ static void __iomem *dw_pcie_other_conf_map_bus(struct pci_bus *bus,
 {
 	struct dw_pcie_rp *pp = bus->sysdata;
 	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct dw_pcie_ob_atu_cfg atu = { 0 };
 	int type, ret;
 	u32 busdev;
 
@@ -566,8 +567,12 @@ static void __iomem *dw_pcie_other_conf_map_bus(struct pci_bus *bus,
 	else
 		type = PCIE_ATU_TYPE_CFG1;
 
-	ret = dw_pcie_prog_outbound_atu(pci, 0, type, pp->cfg0_base, busdev,
-					pp->cfg0_size);
+	atu.type = type;
+	atu.cpu_addr = pp->cfg0_base;
+	atu.pci_addr = busdev;
+	atu.size = pp->cfg0_size;
+
+	ret = dw_pcie_prog_outbound_atu(pci, &atu);
 	if (ret)
 		return NULL;
 
@@ -579,6 +584,7 @@ static int dw_pcie_rd_other_conf(struct pci_bus *bus, unsigned int devfn,
 {
 	struct dw_pcie_rp *pp = bus->sysdata;
 	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct dw_pcie_ob_atu_cfg atu = { 0 };
 	int ret;
 
 	ret = pci_generic_config_read(bus, devfn, where, size, val);
@@ -586,9 +592,12 @@ static int dw_pcie_rd_other_conf(struct pci_bus *bus, unsigned int devfn,
 		return ret;
 
 	if (pp->cfg0_io_shared) {
-		ret = dw_pcie_prog_outbound_atu(pci, 0, PCIE_ATU_TYPE_IO,
-						pp->io_base, pp->io_bus_addr,
-						pp->io_size);
+		atu.type = PCIE_ATU_TYPE_IO;
+		atu.cpu_addr = pp->io_base;
+		atu.pci_addr = pp->io_bus_addr;
+		atu.size = pp->io_size;
+
+		ret = dw_pcie_prog_outbound_atu(pci, &atu);
 		if (ret)
 			return PCIBIOS_SET_FAILED;
 	}
@@ -601,6 +610,7 @@ static int dw_pcie_wr_other_conf(struct pci_bus *bus, unsigned int devfn,
 {
 	struct dw_pcie_rp *pp = bus->sysdata;
 	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct dw_pcie_ob_atu_cfg atu = { 0 };
 	int ret;
 
 	ret = pci_generic_config_write(bus, devfn, where, size, val);
@@ -608,9 +618,12 @@ static int dw_pcie_wr_other_conf(struct pci_bus *bus, unsigned int devfn,
 		return ret;
 
 	if (pp->cfg0_io_shared) {
-		ret = dw_pcie_prog_outbound_atu(pci, 0, PCIE_ATU_TYPE_IO,
-						pp->io_base, pp->io_bus_addr,
-						pp->io_size);
+		atu.type = PCIE_ATU_TYPE_IO;
+		atu.cpu_addr = pp->io_base;
+		atu.pci_addr = pp->io_bus_addr;
+		atu.size = pp->io_size;
+
+		ret = dw_pcie_prog_outbound_atu(pci, &atu);
 		if (ret)
 			return PCIBIOS_SET_FAILED;
 	}
@@ -645,6 +658,7 @@ static struct pci_ops dw_pcie_ops = {
 static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct dw_pcie_ob_atu_cfg atu = { 0 };
 	struct resource_entry *entry;
 	int i, ret;
 
@@ -672,10 +686,13 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
 		if (pci->num_ob_windows <= ++i)
 			break;
 
-		ret = dw_pcie_prog_outbound_atu(pci, i, PCIE_ATU_TYPE_MEM,
-						entry->res->start,
-						entry->res->start - entry->offset,
-						resource_size(entry->res));
+		atu.index = i;
+		atu.type = PCIE_ATU_TYPE_MEM;
+		atu.cpu_addr = entry->res->start;
+		atu.pci_addr = entry->res->start - entry->offset;
+		atu.size = resource_size(entry->res);
+
+		ret = dw_pcie_prog_outbound_atu(pci, &atu);
 		if (ret) {
 			dev_err(pci->dev, "Failed to set MEM range %pr\n",
 				entry->res);
@@ -685,10 +702,13 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
 
 	if (pp->io_size) {
 		if (pci->num_ob_windows > ++i) {
-			ret = dw_pcie_prog_outbound_atu(pci, i, PCIE_ATU_TYPE_IO,
-							pp->io_base,
-							pp->io_bus_addr,
-							pp->io_size);
+			atu.index = i;
+			atu.type = PCIE_ATU_TYPE_IO;
+			atu.cpu_addr = pp->io_base;
+			atu.pci_addr = pp->io_bus_addr;
+			atu.size = pp->io_size;
+
+			ret = dw_pcie_prog_outbound_atu(pci, &atu);
 			if (ret) {
 				dev_err(pci->dev, "Failed to set IO range %pr\n",
 					entry->res);
diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
index ede166645289..bd85a73d354b 100644
--- a/drivers/pci/controller/dwc/pcie-designware.c
+++ b/drivers/pci/controller/dwc/pcie-designware.c
@@ -464,56 +464,56 @@ static inline u32 dw_pcie_enable_ecrc(u32 val)
 	return val | PCIE_ATU_TD;
 }
 
-static int __dw_pcie_prog_outbound_atu(struct dw_pcie *pci, u8 func_no,
-				       int index, int type, u64 cpu_addr,
-				       u64 pci_addr, u64 size)
+int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
+			      const struct dw_pcie_ob_atu_cfg *atu)
 {
+	u64 cpu_addr = atu->cpu_addr;
 	u32 retries, val;
 	u64 limit_addr;
 
 	if (pci->ops && pci->ops->cpu_addr_fixup)
 		cpu_addr = pci->ops->cpu_addr_fixup(pci, cpu_addr);
 
-	limit_addr = cpu_addr + size - 1;
+	limit_addr = cpu_addr + atu->size - 1;
 
 	if ((limit_addr & ~pci->region_limit) != (cpu_addr & ~pci->region_limit) ||
 	    !IS_ALIGNED(cpu_addr, pci->region_align) ||
-	    !IS_ALIGNED(pci_addr, pci->region_align) || !size) {
+	    !IS_ALIGNED(atu->pci_addr, pci->region_align) || !atu->size) {
 		return -EINVAL;
 	}
 
-	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_LOWER_BASE,
+	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_LOWER_BASE,
 			      lower_32_bits(cpu_addr));
-	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_UPPER_BASE,
+	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_BASE,
 			      upper_32_bits(cpu_addr));
 
-	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_LIMIT,
+	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_LIMIT,
 			      lower_32_bits(limit_addr));
 	if (dw_pcie_ver_is_ge(pci, 460A))
-		dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_UPPER_LIMIT,
+		dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_LIMIT,
 				      upper_32_bits(limit_addr));
 
-	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_LOWER_TARGET,
-			      lower_32_bits(pci_addr));
-	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_UPPER_TARGET,
-			      upper_32_bits(pci_addr));
+	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_LOWER_TARGET,
+			      lower_32_bits(atu->pci_addr));
+	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_TARGET,
+			      upper_32_bits(atu->pci_addr));
 
-	val = type | PCIE_ATU_FUNC_NUM(func_no);
+	val = atu->type | PCIE_ATU_FUNC_NUM(atu->func_no);
 	if (upper_32_bits(limit_addr) > upper_32_bits(cpu_addr) &&
 	    dw_pcie_ver_is_ge(pci, 460A))
 		val |= PCIE_ATU_INCREASE_REGION_SIZE;
 	if (dw_pcie_ver_is(pci, 490A))
 		val = dw_pcie_enable_ecrc(val);
-	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_REGION_CTRL1, val);
+	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL1, val);
 
-	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_REGION_CTRL2, PCIE_ATU_ENABLE);
+	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2, PCIE_ATU_ENABLE);
 
 	/*
 	 * Make sure ATU enable takes effect before any subsequent config
 	 * and I/O accesses.
 	 */
 	for (retries = 0; retries < LINK_WAIT_MAX_IATU_RETRIES; retries++) {
-		val = dw_pcie_readl_atu_ob(pci, index, PCIE_ATU_REGION_CTRL2);
+		val = dw_pcie_readl_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2);
 		if (val & PCIE_ATU_ENABLE)
 			return 0;
 
@@ -525,21 +525,6 @@ static int __dw_pcie_prog_outbound_atu(struct dw_pcie *pci, u8 func_no,
 	return -ETIMEDOUT;
 }
 
-int dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
-			      u64 cpu_addr, u64 pci_addr, u64 size)
-{
-	return __dw_pcie_prog_outbound_atu(pci, 0, index, type,
-					   cpu_addr, pci_addr, size);
-}
-
-int dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int index,
-				 int type, u64 cpu_addr, u64 pci_addr,
-				 u64 size)
-{
-	return __dw_pcie_prog_outbound_atu(pci, func_no, index, type,
-					   cpu_addr, pci_addr, size);
-}
-
 static inline u32 dw_pcie_readl_atu_ib(struct dw_pcie *pci, u32 index, u32 reg)
 {
 	return dw_pcie_readl_atu(pci, PCIE_ATU_REGION_DIR_IB, index, reg);
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index 9acf6c40d252..b8fa099bbed3 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -291,6 +291,15 @@ enum dw_pcie_core_rst {
 	DW_PCIE_NUM_CORE_RSTS
 };
 
+struct dw_pcie_ob_atu_cfg {
+	int index;
+	int type;
+	u8 func_no;
+	u64 cpu_addr;
+	u64 pci_addr;
+	u64 size;
+};
+
 struct dw_pcie_host_ops {
 	int (*host_init)(struct dw_pcie_rp *pp);
 	void (*host_deinit)(struct dw_pcie_rp *pp);
@@ -419,10 +428,8 @@ void dw_pcie_write_dbi2(struct dw_pcie *pci, u32 reg, size_t size, u32 val);
 int dw_pcie_link_up(struct dw_pcie *pci);
 void dw_pcie_upconfig_setup(struct dw_pcie *pci);
 int dw_pcie_wait_for_link(struct dw_pcie *pci);
-int dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
-			      u64 cpu_addr, u64 pci_addr, u64 size);
-int dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int index,
-				 int type, u64 cpu_addr, u64 pci_addr, u64 size);
+int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
+			      const struct dw_pcie_ob_atu_cfg *atu);
 int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int type,
 			     u64 cpu_addr, u64 pci_addr, u64 size);
 int dw_pcie_prog_ep_inbound_atu(struct dw_pcie *pci, u8 func_no, int index,
-- 
2.25.1

[PATCH v16 07/22] PCI: dwc: Add outbound MSG TLPs support

[Yoshihiro Shimoda]

Add "code" and "routing" into struct dw_pcie_outbound_atu for
sending MSG by iATU in the PCIe endpoint mode in near the future.
PCIE_ATU_INHIBIT_PAYLOAD is set to issue TLP type of Msg instead of
MsgD. PCIE_ATU_HEADER_SUB_ENABLE is set to issue the translated
TLP header by using PCIE_ATU_{LOW,UPP}ER_TARGET registers' values.

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
 drivers/pci/controller/dwc/pcie-designware.c | 7 +++++--
 drivers/pci/controller/dwc/pcie-designware.h | 5 +++++
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
index bd85a73d354b..a7c724ba7385 100644
--- a/drivers/pci/controller/dwc/pcie-designware.c
+++ b/drivers/pci/controller/dwc/pcie-designware.c
@@ -498,7 +498,7 @@ int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
 	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_TARGET,
 			      upper_32_bits(atu->pci_addr));
 
-	val = atu->type | PCIE_ATU_FUNC_NUM(atu->func_no);
+	val = atu->type | atu->routing | PCIE_ATU_FUNC_NUM(atu->func_no);
 	if (upper_32_bits(limit_addr) > upper_32_bits(cpu_addr) &&
 	    dw_pcie_ver_is_ge(pci, 460A))
 		val |= PCIE_ATU_INCREASE_REGION_SIZE;
@@ -506,7 +506,10 @@ int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
 		val = dw_pcie_enable_ecrc(val);
 	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL1, val);
 
-	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2, PCIE_ATU_ENABLE);
+	val = PCIE_ATU_ENABLE;
+	if (atu->type == PCIE_ATU_TYPE_MSG)
+		val |= PCIE_ATU_INHIBIT_PAYLOAD | PCIE_ATU_HEADER_SUB_ENABLE | atu->code;
+	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2, val);
 
 	/*
 	 * Make sure ATU enable takes effect before any subsequent config
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index b8fa099bbed3..c83d1d176e62 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -150,11 +150,14 @@
 #define PCIE_ATU_TYPE_IO		0x2
 #define PCIE_ATU_TYPE_CFG0		0x4
 #define PCIE_ATU_TYPE_CFG1		0x5
+#define PCIE_ATU_TYPE_MSG		0x10
 #define PCIE_ATU_TD			BIT(8)
 #define PCIE_ATU_FUNC_NUM(pf)           ((pf) << 20)
 #define PCIE_ATU_REGION_CTRL2		0x004
 #define PCIE_ATU_ENABLE			BIT(31)
 #define PCIE_ATU_BAR_MODE_ENABLE	BIT(30)
+#define PCIE_ATU_INHIBIT_PAYLOAD	BIT(22)
+#define PCIE_ATU_HEADER_SUB_ENABLE	BIT(21)
 #define PCIE_ATU_FUNC_NUM_MATCH_EN      BIT(19)
 #define PCIE_ATU_LOWER_BASE		0x008
 #define PCIE_ATU_UPPER_BASE		0x00C
@@ -295,6 +298,8 @@ struct dw_pcie_ob_atu_cfg {
 	int index;
 	int type;
 	u8 func_no;
+	u8 code;
+	u8 routing;
 	u64 cpu_addr;
 	u64 pci_addr;
 	u64 size;
-- 
2.25.1

[PATCH v16 08/22] PCI: designware-ep: Add INTx IRQs support

[Yoshihiro Shimoda]

Add support for triggering INTx IRQs by using outbound iATU.
Outbound iATU is utilized to send assert and de-assert INTx TLPs.
The message is generated based on the payloadless Msg TLP with type
0x14, where 0x4 is the routing code implying the Terminate at
Receiver message. The message code is specified as b1000xx for
the INTx assertion and b1001xx for the INTx de-assertion.

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
 .../pci/controller/dwc/pcie-designware-ep.c   | 66 +++++++++++++++++--
 drivers/pci/controller/dwc/pcie-designware.h  |  2 +
 2 files changed, 64 insertions(+), 4 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index fe2e0d765be9..0abc0073b1cf 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -6,9 +6,11 @@
  * Author: Kishon Vijay Abraham I <kishon@ti.com>
  */
 
+#include <linux/delay.h>
 #include <linux/of.h>
 #include <linux/platform_device.h>
 
+#include "../../pci.h"
 #include "pcie-designware.h"
 #include <linux/pci-epc.h>
 #include <linux/pci-epf.h>
@@ -484,14 +486,57 @@ static const struct pci_epc_ops epc_ops = {
 	.get_features		= dw_pcie_ep_get_features,
 };
 
+static int dw_pcie_ep_send_msg(struct dw_pcie_ep *ep, u8 func_no, u8 code,
+			       u8 routing)
+{
+	struct dw_pcie_ob_atu_cfg atu = { 0 };
+	struct pci_epc *epc = ep->epc;
+	int ret;
+
+	atu.func_no = func_no;
+	atu.code = code;
+	atu.routing = routing;
+	atu.type = PCIE_ATU_TYPE_MSG;
+	atu.cpu_addr = ep->intx_mem_phys;
+	atu.size = epc->mem->window.page_size;
+
+	ret = dw_pcie_ep_outbound_atu(ep, &atu);
+	if (ret)
+		return ret;
+
+	writel(0, ep->intx_mem);
+
+	dw_pcie_ep_unmap_addr(epc, func_no, 0, ep->intx_mem_phys);
+
+	return 0;
+}
+
 int dw_pcie_ep_raise_intx_irq(struct dw_pcie_ep *ep, u8 func_no)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	struct device *dev = pci->dev;
+	int ret;
 
-	dev_err(dev, "EP cannot trigger INTx IRQs\n");
+	if (!ep->intx_mem) {
+		dev_err(dev, "INTx not supported\n");
+		return -EOPNOTSUPP;
+	}
 
-	return -EINVAL;
+	/*
+	 * According to the document of PCIe, the INTx emulation should be
+	 * level-triggered. However, the Linux PCIe Endpoint framework only
+	 * supports edge-triggered for now. So, this function asserts INTx for
+	 * 50 usec, and then deasserts it.
+	 */
+	ret = dw_pcie_ep_send_msg(ep, func_no, PCI_CODE_ASSERT_INTA,
+				  PCI_MSG_ROUTING_LOCAL);
+	if (ret)
+		return ret;
+
+	usleep_range(50, 100);
+
+	return dw_pcie_ep_send_msg(ep, func_no, PCI_CODE_DEASSERT_INTA,
+				   PCI_MSG_ROUTING_LOCAL);
 }
 EXPORT_SYMBOL_GPL(dw_pcie_ep_raise_intx_irq);
 
@@ -622,6 +667,10 @@ void dw_pcie_ep_exit(struct dw_pcie_ep *ep)
 
 	dw_pcie_edma_remove(pci);
 
+	if (ep->intx_mem)
+		pci_epc_mem_free_addr(epc, ep->intx_mem_phys, ep->intx_mem,
+				      epc->mem->window.page_size);
+
 	pci_epc_mem_free_addr(epc, ep->msi_mem_phys, ep->msi_mem,
 			      epc->mem->window.page_size);
 
@@ -793,9 +842,14 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 		goto err_exit_epc_mem;
 	}
 
+	ep->intx_mem = pci_epc_mem_alloc_addr(epc, &ep->intx_mem_phys,
+					      epc->mem->window.page_size);
+	if (!ep->intx_mem)
+		dev_warn(dev, "Failed to reserve memory for INTx\n");
+
 	ret = dw_pcie_edma_detect(pci);
 	if (ret)
-		goto err_free_epc_mem;
+		goto err_free_epc_mem_intx;
 
 	if (ep->ops->get_features) {
 		epc_features = ep->ops->get_features(ep);
@@ -812,7 +866,11 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 err_remove_edma:
 	dw_pcie_edma_remove(pci);
 
-err_free_epc_mem:
+err_free_epc_mem_intx:
+	if (ep->intx_mem)
+		pci_epc_mem_free_addr(epc, ep->intx_mem_phys, ep->intx_mem,
+				      epc->mem->window.page_size);
+
 	pci_epc_mem_free_addr(epc, ep->msi_mem_phys, ep->msi_mem,
 			      epc->mem->window.page_size);
 
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index c83d1d176e62..06e044e2163a 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -369,6 +369,8 @@ struct dw_pcie_ep {
 	unsigned long		*ob_window_map;
 	void __iomem		*msi_mem;
 	phys_addr_t		msi_mem_phys;
+	void __iomem		*intx_mem;
+	phys_addr_t		intx_mem_phys;
 	struct pci_epf_bar	*epf_bar[PCI_STD_NUM_BARS];
 };
 
-- 
2.25.1

[PATCH v16 09/22] PCI: dwc: Add dw_pcie_link_set_max_link_width()

[Yoshihiro Shimoda]

To improve code readability, add dw_pcie_link_set_max_link_width().

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
---
 drivers/pci/controller/dwc/pcie-designware.c | 46 ++++++++++++--------
 1 file changed, 28 insertions(+), 18 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
index a7c724ba7385..68aefbbcd68c 100644
--- a/drivers/pci/controller/dwc/pcie-designware.c
+++ b/drivers/pci/controller/dwc/pcie-designware.c
@@ -719,6 +719,33 @@ static void dw_pcie_link_set_max_speed(struct dw_pcie *pci, u32 link_gen)
 	dw_pcie_writel_dbi(pci, offset + PCI_EXP_LNKCAP, cap | link_speed);
 }
 
+static void dw_pcie_link_set_max_link_width(struct dw_pcie *pci, u32 num_lanes)
+{
+	u32 lwsc;
+
+	if (!num_lanes)
+		return;
+
+	/* Set link width speed control register */
+	lwsc = dw_pcie_readl_dbi(pci, PCIE_LINK_WIDTH_SPEED_CONTROL);
+	lwsc &= ~PORT_LOGIC_LINK_WIDTH_MASK;
+	switch (num_lanes) {
+	case 1:
+		lwsc |= PORT_LOGIC_LINK_WIDTH_1_LANES;
+		break;
+	case 2:
+		lwsc |= PORT_LOGIC_LINK_WIDTH_2_LANES;
+		break;
+	case 4:
+		lwsc |= PORT_LOGIC_LINK_WIDTH_4_LANES;
+		break;
+	case 8:
+		lwsc |= PORT_LOGIC_LINK_WIDTH_8_LANES;
+		break;
+	}
+	dw_pcie_writel_dbi(pci, PCIE_LINK_WIDTH_SPEED_CONTROL, lwsc);
+}
+
 void dw_pcie_iatu_detect(struct dw_pcie *pci)
 {
 	int max_region, ob, ib;
@@ -1026,22 +1053,5 @@ void dw_pcie_setup(struct dw_pcie *pci)
 	}
 	dw_pcie_writel_dbi(pci, PCIE_PORT_LINK_CONTROL, val);
 
-	/* Set link width speed control register */
-	val = dw_pcie_readl_dbi(pci, PCIE_LINK_WIDTH_SPEED_CONTROL);
-	val &= ~PORT_LOGIC_LINK_WIDTH_MASK;
-	switch (pci->num_lanes) {
-	case 1:
-		val |= PORT_LOGIC_LINK_WIDTH_1_LANES;
-		break;
-	case 2:
-		val |= PORT_LOGIC_LINK_WIDTH_2_LANES;
-		break;
-	case 4:
-		val |= PORT_LOGIC_LINK_WIDTH_4_LANES;
-		break;
-	case 8:
-		val |= PORT_LOGIC_LINK_WIDTH_8_LANES;
-		break;
-	}
-	dw_pcie_writel_dbi(pci, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
+	dw_pcie_link_set_max_link_width(pci, pci->num_lanes);
 }
-- 
2.25.1

[PATCH v16 10/22] PCI: dwc: Modify PCIE_PORT_LINK_CONTROL handling

[Yoshihiro Shimoda]

To improve code readability, modify PCIE_PORT_LINK_CONTROL handling.

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
 drivers/pci/controller/dwc/pcie-designware.c | 40 +++++++-------------
 1 file changed, 13 insertions(+), 27 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
index 68aefbbcd68c..5dc423dd2f21 100644
--- a/drivers/pci/controller/dwc/pcie-designware.c
+++ b/drivers/pci/controller/dwc/pcie-designware.c
@@ -721,28 +721,40 @@ static void dw_pcie_link_set_max_speed(struct dw_pcie *pci, u32 link_gen)
 
 static void dw_pcie_link_set_max_link_width(struct dw_pcie *pci, u32 num_lanes)
 {
-	u32 lwsc;
+	u32 lwsc, plc;
 
 	if (!num_lanes)
 		return;
 
+	/* Set the number of lanes */
+	plc = dw_pcie_readl_dbi(pci, PCIE_PORT_LINK_CONTROL);
+	plc &= ~PORT_LINK_MODE_MASK;
+
 	/* Set link width speed control register */
 	lwsc = dw_pcie_readl_dbi(pci, PCIE_LINK_WIDTH_SPEED_CONTROL);
 	lwsc &= ~PORT_LOGIC_LINK_WIDTH_MASK;
 	switch (num_lanes) {
 	case 1:
+		plc |= PORT_LINK_MODE_1_LANES;
 		lwsc |= PORT_LOGIC_LINK_WIDTH_1_LANES;
 		break;
 	case 2:
+		plc |= PORT_LINK_MODE_2_LANES;
 		lwsc |= PORT_LOGIC_LINK_WIDTH_2_LANES;
 		break;
 	case 4:
+		plc |= PORT_LINK_MODE_4_LANES;
 		lwsc |= PORT_LOGIC_LINK_WIDTH_4_LANES;
 		break;
 	case 8:
+		plc |= PORT_LINK_MODE_8_LANES;
 		lwsc |= PORT_LOGIC_LINK_WIDTH_8_LANES;
 		break;
+	default:
+		dev_err(pci->dev, "num-lanes %u: invalid value\n", num_lanes);
+		return;
 	}
+	dw_pcie_writel_dbi(pci, PCIE_PORT_LINK_CONTROL, plc);
 	dw_pcie_writel_dbi(pci, PCIE_LINK_WIDTH_SPEED_CONTROL, lwsc);
 }
 
@@ -1027,31 +1039,5 @@ void dw_pcie_setup(struct dw_pcie *pci)
 	val |= PORT_LINK_DLL_LINK_EN;
 	dw_pcie_writel_dbi(pci, PCIE_PORT_LINK_CONTROL, val);
 
-	if (!pci->num_lanes) {
-		dev_dbg(pci->dev, "Using h/w default number of lanes\n");
-		return;
-	}
-
-	/* Set the number of lanes */
-	val &= ~PORT_LINK_MODE_MASK;
-	switch (pci->num_lanes) {
-	case 1:
-		val |= PORT_LINK_MODE_1_LANES;
-		break;
-	case 2:
-		val |= PORT_LINK_MODE_2_LANES;
-		break;
-	case 4:
-		val |= PORT_LINK_MODE_4_LANES;
-		break;
-	case 8:
-		val |= PORT_LINK_MODE_8_LANES;
-		break;
-	default:
-		dev_err(pci->dev, "num-lanes %u: invalid value\n", pci->num_lanes);
-		return;
-	}
-	dw_pcie_writel_dbi(pci, PCIE_PORT_LINK_CONTROL, val);
-
 	dw_pcie_link_set_max_link_width(pci, pci->num_lanes);
 }
-- 
2.25.1

[PATCH v16 11/22] PCI: dwc: Add dw_pcie_link_set_max_cap_width()

[Yoshihiro Shimoda]

Add dw_pcie_link_set_max_cap_width() to set PCI_EXP_LNKCAP_MLW.
In accordance with the DW PCIe RC/EP HW manuals [1,2,3,...] aside with
the PORT_LINK_CTRL_OFF.LINK_CAPABLE and GEN2_CTRL_OFF.NUM_OF_LANES[8:0]
field there is another one which needs to be updated. It's
LINK_CAPABILITIES_REG.PCIE_CAP_MAX_LINK_WIDTH. If it isn't done at
the very least the maximum link-width capability CSR won't expose
the actual maximum capability.

[1] DesignWare Cores PCI Express Controller Databook - DWC PCIe Root Port,
    Version 4.60a, March 2015, p.1032
[2] DesignWare Cores PCI Express Controller Databook - DWC PCIe Root Port,
    Version 4.70a, March 2016, p.1065
[3] DesignWare Cores PCI Express Controller Databook - DWC PCIe Root Port,
    Version 4.90a, March 2016, p.1057
...
[X] DesignWare Cores PCI Express Controller Databook - DWC PCIe Endpoint,
      Version 5.40a, March 2019, p.1396
[X+1] DesignWare Cores PCI Express Controller Databook - DWC PCIe Root Port,
      Version 5.40a, March 2019, p.1266

The commit description is suggested by Serge Semin.

Suggested-by: Serge Semin <fancer.lancer@gmail.com>
Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
 drivers/pci/controller/dwc/pcie-designware.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
index 5dc423dd2f21..8b2978c6eb23 100644
--- a/drivers/pci/controller/dwc/pcie-designware.c
+++ b/drivers/pci/controller/dwc/pcie-designware.c
@@ -758,6 +758,21 @@ static void dw_pcie_link_set_max_link_width(struct dw_pcie *pci, u32 num_lanes)
 	dw_pcie_writel_dbi(pci, PCIE_LINK_WIDTH_SPEED_CONTROL, lwsc);
 }
 
+static void dw_pcie_link_set_max_cap_width(struct dw_pcie *pci, int num_lanes)
+{
+	u32 val;
+	u8 cap;
+
+	if (!num_lanes)
+		return;
+
+	cap = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP);
+	val = dw_pcie_readl_dbi(pci, cap + PCI_EXP_LNKCAP);
+	val &= ~PCI_EXP_LNKCAP_MLW;
+	val |= num_lanes << PCI_EXP_LNKSTA_NLW_SHIFT;
+	dw_pcie_writel_dbi(pci, cap + PCI_EXP_LNKCAP, val);
+}
+
 void dw_pcie_iatu_detect(struct dw_pcie *pci)
 {
 	int max_region, ob, ib;
@@ -1040,4 +1055,5 @@ void dw_pcie_setup(struct dw_pcie *pci)
 	dw_pcie_writel_dbi(pci, PCIE_PORT_LINK_CONTROL, val);
 
 	dw_pcie_link_set_max_link_width(pci, pci->num_lanes);
+	dw_pcie_link_set_max_cap_width(pci, pci->num_lanes);
 }
-- 
2.25.1

[PATCH v16 12/22] PCI: tegra194: Drop PCI_EXP_LNKSTA_NLW setting.

[Yoshihiro Shimoda]

dw_pcie_setup() will set PCI_EXP_LNKSTA_NLW to PCI_EXP_LNKCAP register
so that drop such setting from tegra_pcie_dw_host_init().

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Cc: Thierry Reding <thierry.reding@gmail.com>
Cc: Jonathan Hunter <jonathanh@nvidia.com>
---
 drivers/pci/controller/dwc/pcie-tegra194.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-tegra194.c b/drivers/pci/controller/dwc/pcie-tegra194.c
index 4adba379b83d..723a22ccd58c 100644
--- a/drivers/pci/controller/dwc/pcie-tegra194.c
+++ b/drivers/pci/controller/dwc/pcie-tegra194.c
@@ -901,12 +901,6 @@ static int tegra_pcie_dw_host_init(struct dw_pcie_rp *pp)
 		AMBA_ERROR_RESPONSE_CRS_SHIFT);
 	dw_pcie_writel_dbi(pci, PORT_LOGIC_AMBA_ERROR_RESPONSE_DEFAULT, val);
 
-	/* Configure Max lane width from DT */
-	val = dw_pcie_readl_dbi(pci, pcie->pcie_cap_base + PCI_EXP_LNKCAP);
-	val &= ~PCI_EXP_LNKCAP_MLW;
-	val |= (pcie->num_lanes << PCI_EXP_LNKSTA_NLW_SHIFT);
-	dw_pcie_writel_dbi(pci, pcie->pcie_cap_base + PCI_EXP_LNKCAP, val);
-
 	/* Clear Slot Clock Configuration bit if SRNS configuration */
 	if (pcie->enable_srns) {
 		val_16 = dw_pcie_readw_dbi(pci, pcie->pcie_cap_base +
-- 
2.25.1

[PATCH v16 13/22] PCI: dwc: Add EDMA_UNROLL capability flag

[Yoshihiro Shimoda]

Renesas R-Car Gen4 PCIe controllers have an unexpected register value on
the dbi+0x97b register. So, add a new capability flag "EDMA_UNROLL"
which would force the unrolled eDMA mapping for the problematic
device, as suggested by Serge Semin.

Suggested-by: Serge Semin <fancer.lancer@gmail.com>
Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
 drivers/pci/controller/dwc/pcie-designware.c | 8 +++++++-
 drivers/pci/controller/dwc/pcie-designware.h | 5 +++--
 2 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
index 8b2978c6eb23..e405bfae0191 100644
--- a/drivers/pci/controller/dwc/pcie-designware.c
+++ b/drivers/pci/controller/dwc/pcie-designware.c
@@ -881,8 +881,14 @@ static int dw_pcie_edma_find_chip(struct dw_pcie *pci)
 	 * Indirect eDMA CSRs access has been completely removed since v5.40a
 	 * thus no space is now reserved for the eDMA channels viewport and
 	 * former DMA CTRL register is no longer fixed to FFs.
+	 *
+	 * Note that Renesas R-Car S4-8's PCIe controllers for unknown reason
+	 * may have zeros in the eDMA CTRL register even though the HW-manual
+	 * explicitly states there must FFs if the unrolled mapping is enabled.
+	 * For such cases the low-level drivers are supposed to manually
+	 * activate the unrolled mapping to bypass the auto-detection procedure.
 	 */
-	if (dw_pcie_ver_is_ge(pci, 540A))
+	if (dw_pcie_ver_is_ge(pci, 540A) || dw_pcie_cap_is(pci, EDMA_UNROLL))
 		val = 0xFFFFFFFF;
 	else
 		val = dw_pcie_readl_dbi(pci, PCIE_DMA_VIEWPORT_BASE + PCIE_DMA_CTRL);
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index 06e044e2163a..2639206b4c18 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -54,8 +54,9 @@
 
 /* DWC PCIe controller capabilities */
 #define DW_PCIE_CAP_REQ_RES		0
-#define DW_PCIE_CAP_IATU_UNROLL		1
-#define DW_PCIE_CAP_CDM_CHECK		2
+#define DW_PCIE_CAP_EDMA_UNROLL		1
+#define DW_PCIE_CAP_IATU_UNROLL		2
+#define DW_PCIE_CAP_CDM_CHECK		3
 
 #define dw_pcie_cap_is(_pci, _cap) \
 	test_bit(DW_PCIE_CAP_ ## _cap, &(_pci)->caps)
-- 
2.25.1

[PATCH v16 14/22] PCI: dwc: Expose dw_pcie_ep_exit() to module

[Yoshihiro Shimoda]

Since no PCIe controller drivers call this, this change is not required
for now. But, Renesas R-Car Gen4 PCIe controller driver will call this
and if the controller driver is built as a kernel module, the following
build error happens. So, expose dw_pcie_ep_exit() for it.

ERROR: modpost: "dw_pcie_ep_exit" [drivers/pci/controller/dwc/pcie-rcar-gen4-ep-drv.ko] undefined!

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
 drivers/pci/controller/dwc/pcie-designware-ep.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index 0abc0073b1cf..023938468b5d 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -676,6 +676,7 @@ void dw_pcie_ep_exit(struct dw_pcie_ep *ep)
 
 	pci_epc_mem_exit(epc);
 }
+EXPORT_SYMBOL_GPL(dw_pcie_ep_exit);
 
 static unsigned int dw_pcie_ep_find_ext_capability(struct dw_pcie *pci, int cap)
 {
-- 
2.25.1

[PATCH v16 15/22] PCI: dwc: Introduce .ep_pre_init() and .ep_deinit()

[Yoshihiro Shimoda]

Renesas R-Car Gen4 PCIe controllers require vender-specific
initialization before .ep_init(). To use dw->dbi and dw->num-lanes
in the initialization code, introduce .ep_pre_init() into struct
dw_pcie_ep_ops. Also introduce .ep_deinit() to disable the controller
by using vender-specific de-initialization.

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
 drivers/pci/controller/dwc/pcie-designware-ep.c | 6 ++++++
 drivers/pci/controller/dwc/pcie-designware.h    | 2 ++
 2 files changed, 8 insertions(+)

diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index 023938468b5d..2728e17bc66e 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -665,6 +665,9 @@ void dw_pcie_ep_exit(struct dw_pcie_ep *ep)
 	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
 	struct pci_epc *epc = ep->epc;
 
+	if (ep->ops->ep_deinit)
+		ep->ops->ep_deinit(ep);
+
 	dw_pcie_edma_remove(pci);
 
 	if (ep->intx_mem)
@@ -778,6 +781,9 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
 	ep->phys_base = res->start;
 	ep->addr_size = resource_size(res);
 
+	if (ep->ops->ep_pre_init)
+		ep->ops->ep_pre_init(ep);
+
 	dw_pcie_version_detect(pci);
 
 	dw_pcie_iatu_detect(pci);
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index 2639206b4c18..c8a0fd97731f 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -336,7 +336,9 @@ struct dw_pcie_rp {
 };
 
 struct dw_pcie_ep_ops {
+	void	(*ep_pre_init)(struct dw_pcie_ep *ep);
 	void	(*ep_init)(struct dw_pcie_ep *ep);
+	void	(*ep_deinit)(struct dw_pcie_ep *ep);
 	int	(*raise_irq)(struct dw_pcie_ep *ep, u8 func_no,
 			     enum pci_epc_irq_type type, u16 interrupt_num);
 	const struct pci_epc_features* (*get_features)(struct dw_pcie_ep *ep);
-- 
2.25.1

[PATCH v16 16/22] dt-bindings: PCI: dwc: Update maxItems of reg and reg-names

[Yoshihiro Shimoda]

Update maxItems of reg and reg-names on both host and endpoint
for supporting Renesas R-Car Gen4 PCIe controllers later.

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Acked-by: Rob Herring <robh@kernel.org>
Acked-by: Manivannan Sadhasivam <mani@kernel.org>
Reviewed-by: Serge Semin <fancer.lancer@gmail.com>
---
 Documentation/devicetree/bindings/pci/snps,dw-pcie-ep.yaml | 4 ++--
 Documentation/devicetree/bindings/pci/snps,dw-pcie.yaml    | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/Documentation/devicetree/bindings/pci/snps,dw-pcie-ep.yaml b/Documentation/devicetree/bindings/pci/snps,dw-pcie-ep.yaml
index 8fc2151691a4..cb727f60be0b 100644
--- a/Documentation/devicetree/bindings/pci/snps,dw-pcie-ep.yaml
+++ b/Documentation/devicetree/bindings/pci/snps,dw-pcie-ep.yaml
@@ -33,11 +33,11 @@ properties:
       normal controller functioning. iATU memory IO region is also required
       if the space is unrolled (IP-core version >= 4.80a).
     minItems: 2
-    maxItems: 5
+    maxItems: 6
 
   reg-names:
     minItems: 2
-    maxItems: 5
+    maxItems: 6
     items:
       oneOf:
         - description:
diff --git a/Documentation/devicetree/bindings/pci/snps,dw-pcie.yaml b/Documentation/devicetree/bindings/pci/snps,dw-pcie.yaml
index 1a83f0f65f19..0bfcfd6ccb5f 100644
--- a/Documentation/devicetree/bindings/pci/snps,dw-pcie.yaml
+++ b/Documentation/devicetree/bindings/pci/snps,dw-pcie.yaml
@@ -33,11 +33,11 @@ properties:
       are required for the normal controller work. iATU memory IO region is
       also required if the space is unrolled (IP-core version >= 4.80a).
     minItems: 2
-    maxItems: 5
+    maxItems: 6
 
   reg-names:
     minItems: 2
-    maxItems: 5
+    maxItems: 6
     items:
       oneOf:
         - description:
-- 
2.25.1

[PATCH v16 17/22] dt-bindings: PCI: renesas: Add R-Car Gen4 PCIe Host

[Yoshihiro Shimoda]

Document bindings for Renesas R-Car Gen4 and R-Car S4-8 (R8A779F0)
PCIe host module.

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Reviewed-by: Rob Herring <robh@kernel.org>
Reviewed-by: Serge Semin <fancer.lancer@gmail.com>
---
 .../bindings/pci/rcar-gen4-pci-host.yaml      | 109 ++++++++++++++++++
 1 file changed, 109 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.yaml

diff --git a/Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.yaml b/Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.yaml
new file mode 100644
index 000000000000..6bbdbf4fa617
--- /dev/null
+++ b/Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.yaml
@@ -0,0 +1,109 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+# Copyright (C) 2022-2023 Renesas Electronics Corp.
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/pci/rcar-gen4-pci-host.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Renesas R-Car Gen4 PCIe Host
+
+maintainers:
+  - Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
+
+allOf:
+  - $ref: snps,dw-pcie.yaml#
+
+properties:
+  compatible:
+    items:
+      - const: renesas,r8a779f0-pcie   # R-Car S4-8
+      - const: renesas,rcar-gen4-pcie  # R-Car Gen4
+
+  reg:
+    maxItems: 6
+
+  reg-names:
+    items:
+      - const: dbi
+      - const: dbi2
+      - const: atu
+      - const: dma
+      - const: app
+      - const: config
+
+  interrupts:
+    maxItems: 4
+
+  interrupt-names:
+    items:
+      - const: msi
+      - const: dma
+      - const: sft_ce
+      - const: app
+
+  clocks:
+    maxItems: 1
+
+  power-domains:
+    maxItems: 1
+
+  resets:
+    maxItems: 1
+
+  max-link-speed:
+    maximum: 4
+
+  num-lanes:
+    maximum: 4
+
+required:
+  - compatible
+  - reg
+  - interrupts
+  - clocks
+  - power-domains
+
+unevaluatedProperties: false
+
+examples:
+  - |
+    #include <dt-bindings/clock/r8a779f0-cpg-mssr.h>
+    #include <dt-bindings/interrupt-controller/arm-gic.h>
+    #include <dt-bindings/power/r8a779f0-sysc.h>
+
+    soc {
+        #address-cells = <2>;
+        #size-cells = <2>;
+
+        pcie: pcie@e65d0000 {
+            compatible = "renesas,r8a779f0-pcie", "renesas,rcar-gen4-pcie";
+            reg = <0 0xe65d0000 0 0x1000>, <0 0xe65d2000 0 0x0800>,
+                  <0 0xe65d3000 0 0x2000>, <0 0xe65d5000 0 0x1200>,
+                  <0 0xe65d6200 0 0x0e00>, <0 0xfe000000 0 0x400000>;
+            reg-names = "dbi", "dbi2", "atu", "dma", "app", "config";
+            #address-cells = <3>;
+            #size-cells = <2>;
+            bus-range = <0x00 0xff>;
+            device_type = "pci";
+            ranges = <0x01000000 0 0x00000000 0 0xfe000000 0 0x00400000>,
+                     <0x02000000 0 0x30000000 0 0x30000000 0 0x10000000>;
+            dma-ranges = <0x42000000 0 0x00000000 0 0x00000000 1 0x00000000>;
+            interrupts = <GIC_SPI 416 IRQ_TYPE_LEVEL_HIGH>,
+                         <GIC_SPI 417 IRQ_TYPE_LEVEL_HIGH>,
+                         <GIC_SPI 418 IRQ_TYPE_LEVEL_HIGH>,
+                         <GIC_SPI 422 IRQ_TYPE_LEVEL_HIGH>;
+            interrupt-names = "msi", "dma", "sft_ce", "app";
+            #interrupt-cells = <1>;
+            interrupt-map-mask = <0 0 0 7>;
+            interrupt-map = <0 0 0 1 &gic GIC_SPI 416 IRQ_TYPE_LEVEL_HIGH>,
+                            <0 0 0 2 &gic GIC_SPI 416 IRQ_TYPE_LEVEL_HIGH>,
+                            <0 0 0 3 &gic GIC_SPI 416 IRQ_TYPE_LEVEL_HIGH>,
+                            <0 0 0 4 &gic GIC_SPI 416 IRQ_TYPE_LEVEL_HIGH>;
+            clocks = <&cpg CPG_MOD 624>;
+            power-domains = <&sysc R8A779F0_PD_ALWAYS_ON>;
+            resets = <&cpg 624>;
+            num-lanes = <2>;
+            snps,enable-cdm-check;
+            max-link-speed = <4>;
+        };
+    };
-- 
2.25.1

[PATCH v16 18/22] dt-bindings: PCI: renesas: Add R-Car Gen4 PCIe Endpoint

[Yoshihiro Shimoda]

Document bindings for Renesas R-Car Gen4 and R-Car S4-8 (R8A779F0)
PCIe endpoint module.

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Reviewed-by: Rob Herring <robh@kernel.org>
Reviewed-by: Serge Semin <fancer.lancer@gmail.com>
Acked-by: Manivannan Sadhasivam <mani@kernel.org>
---
 .../bindings/pci/rcar-gen4-pci-ep.yaml        | 98 +++++++++++++++++++
 1 file changed, 98 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml

diff --git a/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml b/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml
new file mode 100644
index 000000000000..0453bdcf9645
--- /dev/null
+++ b/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml
@@ -0,0 +1,98 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+# Copyright (C) 2022-2023 Renesas Electronics Corp.
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/pci/rcar-gen4-pci-ep.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Renesas R-Car Gen4 PCIe Endpoint
+
+maintainers:
+  - Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
+
+allOf:
+  - $ref: snps,dw-pcie-ep.yaml#
+
+properties:
+  compatible:
+    items:
+      - const: renesas,r8a779f0-pcie-ep   # R-Car S4-8
+      - const: renesas,rcar-gen4-pcie-ep  # R-Car Gen4
+
+  reg:
+    maxItems: 6
+
+  reg-names:
+    items:
+      - const: dbi
+      - const: dbi2
+      - const: atu
+      - const: dma
+      - const: app
+      - const: addr_space
+
+  interrupts:
+    maxItems: 3
+
+  interrupt-names:
+    items:
+      - const: dma
+      - const: sft_ce
+      - const: app
+
+  power-domains:
+    maxItems: 1
+
+  resets:
+    maxItems: 1
+
+  clocks:
+    maxItems: 1
+
+  max-functions:
+    maximum: 2
+
+  max-link-speed:
+    maximum: 4
+
+  num-lanes:
+    maximum: 4
+
+required:
+  - compatible
+  - reg
+  - reg-names
+  - interrupts
+  - resets
+  - power-domains
+  - clocks
+
+unevaluatedProperties: false
+
+examples:
+  - |
+    #include <dt-bindings/clock/r8a779f0-cpg-mssr.h>
+    #include <dt-bindings/interrupt-controller/arm-gic.h>
+    #include <dt-bindings/power/r8a779f0-sysc.h>
+
+    soc {
+        #address-cells = <2>;
+        #size-cells = <2>;
+
+        pcie0_ep: pcie-ep@e65d0000 {
+            compatible = "renesas,r8a779f0-pcie-ep", "renesas,rcar-gen4-pcie-ep";
+            reg = <0 0xe65d0000 0 0x2000>, <0 0xe65d2800 0 0x0800>,
+                  <0 0xe65d3000 0 0x2000>, <0 0xe65d5000 0 0x1200>,
+                  <0 0xe65d6200 0 0x0e00>, <0 0xfe000000 0 0x400000>;
+            reg-names = "dbi", "dbi2", "atu", "dma", "app", "addr_space";
+            interrupts = <GIC_SPI 417 IRQ_TYPE_LEVEL_HIGH>,
+                         <GIC_SPI 418 IRQ_TYPE_LEVEL_HIGH>,
+                         <GIC_SPI 422 IRQ_TYPE_LEVEL_HIGH>;
+            interrupt-names = "dma", "sft_ce", "app";
+            clocks = <&cpg CPG_MOD 624>;
+            power-domains = <&sysc R8A779F0_PD_ALWAYS_ON>;
+            resets = <&cpg 624>;
+            num-lanes = <2>;
+            max-link-speed = <4>;
+        };
+    };
-- 
2.25.1

[PATCH v16 19/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support

[Yoshihiro Shimoda]

Add R-Car Gen4 PCIe Host support. This controller is based on
Synopsys DesignWare PCIe, but this controller has vendor-specific
registers so that requires initialization code like mode setting
and retraining and so on.

To reduce code delta, adds some helper functions which are used by
both the host driver and the endpoint driver (which is added
immediately afterwards) into a separate file.

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
 drivers/pci/controller/dwc/Kconfig            |   9 +
 drivers/pci/controller/dwc/Makefile           |   2 +
 .../pci/controller/dwc/pcie-rcar-gen4-host.c  | 141 +++++++++++++
 drivers/pci/controller/dwc/pcie-rcar-gen4.c   | 190 ++++++++++++++++++
 drivers/pci/controller/dwc/pcie-rcar-gen4.h   |  46 +++++
 5 files changed, 388 insertions(+)
 create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
 create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.c
 create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.h

diff --git a/drivers/pci/controller/dwc/Kconfig b/drivers/pci/controller/dwc/Kconfig
index ab96da43e0c2..64d4d37bc891 100644
--- a/drivers/pci/controller/dwc/Kconfig
+++ b/drivers/pci/controller/dwc/Kconfig
@@ -415,4 +415,13 @@ config PCIE_VISCONTI_HOST
 	  Say Y here if you want PCIe controller support on Toshiba Visconti SoC.
 	  This driver supports TMPV7708 SoC.
 
+config PCIE_RCAR_GEN4
+	tristate "Renesas R-Car Gen4 PCIe Host controller"
+	depends on ARCH_RENESAS || COMPILE_TEST
+	depends on PCI_MSI
+	select PCIE_DW_HOST
+	help
+	  Say Y here if you want PCIe host controller support on R-Car Gen4 SoCs.
+	  This uses the DesignWare core.
+
 endmenu
diff --git a/drivers/pci/controller/dwc/Makefile b/drivers/pci/controller/dwc/Makefile
index bf5c311875a1..486cf706b53d 100644
--- a/drivers/pci/controller/dwc/Makefile
+++ b/drivers/pci/controller/dwc/Makefile
@@ -26,6 +26,8 @@ obj-$(CONFIG_PCIE_TEGRA194) += pcie-tegra194.o
 obj-$(CONFIG_PCIE_UNIPHIER) += pcie-uniphier.o
 obj-$(CONFIG_PCIE_UNIPHIER_EP) += pcie-uniphier-ep.o
 obj-$(CONFIG_PCIE_VISCONTI_HOST) += pcie-visconti.o
+pcie-rcar-gen4-host-drv-objs := pcie-rcar-gen4.o pcie-rcar-gen4-host.o
+obj-$(CONFIG_PCIE_RCAR_GEN4) += pcie-rcar-gen4-host-drv.o
 
 # The following drivers are for devices that use the generic ACPI
 # pci_root.c driver but don't support standard ECAM config access.
diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
new file mode 100644
index 000000000000..df7d80f1874f
--- /dev/null
+++ b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
@@ -0,0 +1,141 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * PCIe host controller driver for Renesas R-Car Gen4 Series SoCs
+ * Copyright (C) 2022-2023 Renesas Electronics Corporation
+ */
+
+#include <linux/delay.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/of_device.h>
+#include <linux/pci.h>
+#include <linux/platform_device.h>
+
+#include "pcie-rcar-gen4.h"
+#include "pcie-designware.h"
+
+static int rcar_gen4_pcie_host_init(struct dw_pcie_rp *pp)
+{
+	struct dw_pcie *dw = to_dw_pcie_from_pp(pp);
+	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
+	int ret;
+	u32 val;
+
+	gpiod_set_value_cansleep(dw->pe_rst, 1);
+
+	ret = rcar_gen4_pcie_set_device_type(rcar, true, dw->num_lanes);
+	if (ret < 0)
+		return ret;
+
+	dw_pcie_dbi_ro_wr_en(dw);
+
+	/*
+	 * According to the section 3.5.7.2 "RC Mode" in DWC PCIe Dual Mode
+	 * Rev.5.20a, we should disable two BARs to avoid unnecessary memory
+	 * assignment during device enumeration.
+	 */
+	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_0, 0x0);
+	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_1, 0x0);
+
+	dw_pcie_dbi_ro_wr_dis(dw);
+
+	if (IS_ENABLED(CONFIG_PCI_MSI)) {
+		/* Enable MSI interrupt signal */
+		val = readl(rcar->base + PCIEINTSTS0EN);
+		val |= MSI_CTRL_INT;
+		writel(val, rcar->base + PCIEINTSTS0EN);
+	}
+
+	msleep(100);	/* pe_rst requires 100msec delay */
+
+	gpiod_set_value_cansleep(dw->pe_rst, 0);
+
+	return 0;
+}
+
+static const struct dw_pcie_host_ops rcar_gen4_pcie_host_ops = {
+	.host_init = rcar_gen4_pcie_host_init,
+};
+
+static int rcar_gen4_add_dw_pcie_rp(struct rcar_gen4_pcie *rcar,
+				   struct platform_device *pdev)
+{
+	struct dw_pcie *dw = &rcar->dw;
+	struct dw_pcie_rp *pp = &dw->pp;
+
+	pp->num_vectors = MAX_MSI_IRQS;
+	pp->ops = &rcar_gen4_pcie_host_ops;
+	dw_pcie_cap_set(dw, REQ_RES);
+
+	return dw_pcie_host_init(pp);
+}
+
+static void rcar_gen4_remove_dw_pcie_rp(struct rcar_gen4_pcie *rcar)
+{
+	dw_pcie_host_deinit(&rcar->dw.pp);
+	gpiod_set_value_cansleep(rcar->dw.pe_rst, 1);
+}
+
+static int rcar_gen4_pcie_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct rcar_gen4_pcie *rcar;
+	int err;
+
+	rcar = rcar_gen4_pcie_devm_alloc(dev);
+	if (!rcar)
+		return -ENOMEM;
+
+	err = rcar_gen4_pcie_get_resources(rcar, pdev);
+	if (err < 0) {
+		dev_err(dev, "Failed to request resource: %d\n", err);
+		return err;
+	}
+
+	platform_set_drvdata(pdev, rcar);
+
+	err = rcar_gen4_pcie_prepare(rcar);
+	if (err < 0)
+		return err;
+
+	rcar->needs_retrain = true;
+	err = rcar_gen4_add_dw_pcie_rp(rcar, pdev);
+	if (err < 0)
+		goto err_add;
+
+	return 0;
+
+err_add:
+	rcar_gen4_pcie_unprepare(rcar);
+
+	return err;
+}
+
+static int rcar_gen4_pcie_remove(struct platform_device *pdev)
+{
+	struct rcar_gen4_pcie *rcar = platform_get_drvdata(pdev);
+
+	rcar_gen4_remove_dw_pcie_rp(rcar);
+	rcar_gen4_pcie_unprepare(rcar);
+
+	return 0;
+}
+
+static const struct of_device_id rcar_gen4_pcie_of_match[] = {
+	{ .compatible = "renesas,rcar-gen4-pcie", },
+	{},
+};
+
+static struct platform_driver rcar_gen4_pcie_driver = {
+	.driver = {
+		.name = "pcie-rcar-gen4",
+		.of_match_table = rcar_gen4_pcie_of_match,
+		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
+	},
+	.probe = rcar_gen4_pcie_probe,
+	.remove = rcar_gen4_pcie_remove,
+};
+module_platform_driver(rcar_gen4_pcie_driver);
+
+MODULE_DESCRIPTION("Renesas R-Car Gen4 PCIe host controller driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.c b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
new file mode 100644
index 000000000000..35923fda8ed5
--- /dev/null
+++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
@@ -0,0 +1,190 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
+ * Copyright (C) 2022-2023 Renesas Electronics Corporation
+ */
+
+#include <linux/delay.h>
+#include <linux/io.h>
+#include <linux/of_device.h>
+#include <linux/pci.h>
+#include <linux/pm_runtime.h>
+#include <linux/reset.h>
+
+#include "pcie-rcar-gen4.h"
+#include "pcie-designware.h"
+
+/* Renesas-specific */
+#define PCIERSTCTRL1		0x0014
+#define  APP_HOLD_PHY_RST	BIT(16)
+#define  APP_LTSSM_ENABLE	BIT(0)
+
+#define RETRAIN_MAX_CHECK	10
+#define RETRAIN_MAX_RETRIES	10
+
+static void rcar_gen4_pcie_ltssm_enable(struct rcar_gen4_pcie *rcar,
+					bool enable)
+{
+	u32 val;
+
+	val = readl(rcar->base + PCIERSTCTRL1);
+	if (enable) {
+		val |= APP_LTSSM_ENABLE;
+		val &= ~APP_HOLD_PHY_RST;
+	} else {
+		val &= ~APP_LTSSM_ENABLE;
+		val |= APP_HOLD_PHY_RST;
+	}
+	writel(val, rcar->base + PCIERSTCTRL1);
+}
+
+static bool rcar_gen4_pcie_check_retrain_link(struct dw_pcie *dw)
+{
+	u8 offset = dw_pcie_find_capability(dw, PCI_CAP_ID_EXP);
+	u32 lnkcap = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCAP);
+	u32 lnkctl = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCTL);
+	u16 lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
+	int i;
+
+	if ((lnksta & PCI_EXP_LNKSTA_CLS) == (lnkcap & PCI_EXP_LNKCAP_SLS))
+		return true;
+
+	lnkctl |= PCI_EXP_LNKCTL_RL;
+	dw_pcie_writel_dbi(dw, offset + PCI_EXP_LNKCTL, lnkctl);
+
+	for (i = 0; i < RETRAIN_MAX_CHECK; i++) {
+		lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
+		if (lnksta & PCI_EXP_LNKSTA_LT)
+			return true;
+		usleep_range(1000, 1100);
+	}
+
+	return false;
+}
+
+static int rcar_gen4_pcie_link_up(struct dw_pcie *dw)
+{
+	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
+	u32 val, mask;
+
+	val = readl(rcar->base + PCIEINTSTS0);
+	mask = RDLH_LINK_UP | SMLH_LINK_UP;
+
+	return (val & mask) == mask;
+}
+
+static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
+{
+	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
+	int i;
+
+	rcar_gen4_pcie_ltssm_enable(rcar, true);
+
+	/*
+	 * Require retraining here. Otherwise RDLH_LINK_UP of PCIEINTSTS0 which
+	 * is this controller specific register may not be set.
+	 */
+	if (rcar->needs_retrain) {
+		for (i = 0; i < RETRAIN_MAX_RETRIES; i++) {
+			if (rcar_gen4_pcie_check_retrain_link(dw))
+				return 0;
+			msleep(100);
+		}
+
+		return -ETIMEDOUT;	/* Failed */
+	}
+
+	return 0;
+}
+
+static void rcar_gen4_pcie_stop_link(struct dw_pcie *dw)
+{
+	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
+
+	rcar_gen4_pcie_ltssm_enable(rcar, false);
+}
+
+int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
+				   int num_lanes)
+{
+	u32 val;
+
+	/* Note: Assume the rcar->rst which is Cold-reset is asserted here */
+	val = readl(rcar->base + PCIEMSR0);
+	if (rc)
+		val |= DEVICE_TYPE_RC;
+	else
+		val |= DEVICE_TYPE_EP;
+
+	if (num_lanes < 4)
+		val |= BIFUR_MOD_SET_ON;
+
+	writel(val, rcar->base + PCIEMSR0);
+
+	return reset_control_deassert(rcar->rst);
+}
+
+int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *rcar)
+{
+	struct device *dev = rcar->dw.dev;
+	int err;
+
+	pm_runtime_enable(dev);
+	err = pm_runtime_resume_and_get(dev);
+	if (err < 0) {
+		dev_err(dev, "Failed to resume/get Runtime PM\n");
+		pm_runtime_disable(dev);
+	}
+
+	return err;
+}
+
+void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *rcar)
+{
+	struct device *dev = rcar->dw.dev;
+
+	if (!reset_control_status(rcar->rst))
+		reset_control_assert(rcar->rst);
+	pm_runtime_put(dev);
+	pm_runtime_disable(dev);
+}
+
+int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
+				 struct platform_device *pdev)
+{
+	struct device *dev = rcar->dw.dev;
+
+	/* Renesas-specific registers */
+	rcar->base = devm_platform_ioremap_resource_byname(pdev, "app");
+	if (IS_ERR(rcar->base))
+		return PTR_ERR(rcar->base);
+
+	rcar->rst = devm_reset_control_get(dev, NULL);
+	if (IS_ERR(rcar->rst)) {
+		dev_err(dev, "Failed to get Cold-reset\n");
+		return PTR_ERR(rcar->rst);
+	}
+
+	return 0;
+}
+
+static const struct dw_pcie_ops dw_pcie_ops = {
+	.start_link = rcar_gen4_pcie_start_link,
+	.stop_link = rcar_gen4_pcie_stop_link,
+	.link_up = rcar_gen4_pcie_link_up,
+};
+
+struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev)
+{
+	struct rcar_gen4_pcie *rcar;
+
+	rcar = devm_kzalloc(dev, sizeof(*rcar), GFP_KERNEL);
+	if (!rcar)
+		return NULL;
+
+	rcar->dw.dev = dev;
+	rcar->dw.ops = &dw_pcie_ops;
+	dw_pcie_cap_set(&rcar->dw, EDMA_UNROLL);
+
+	return rcar;
+}
diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.h b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
new file mode 100644
index 000000000000..fec3f18609f4
--- /dev/null
+++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
@@ -0,0 +1,46 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
+ * Copyright (C) 2022-2023 Renesas Electronics Corporation
+ */
+
+#ifndef _PCIE_RCAR_GEN4_H_
+#define _PCIE_RCAR_GEN4_H_
+
+#include <linux/io.h>
+#include <linux/pci.h>
+#include <linux/reset.h>
+
+#include "pcie-designware.h"
+
+/* Renesas-specific */
+#define PCIEMSR0		0x0000
+#define  BIFUR_MOD_SET_ON	BIT(0)
+#define  DEVICE_TYPE_EP		0
+#define  DEVICE_TYPE_RC		BIT(4)
+
+#define PCIEINTSTS0		0x0084
+#define PCIEINTSTS0EN		0x0310
+#define  MSI_CTRL_INT		BIT(26)
+#define  SMLH_LINK_UP		BIT(7)
+#define  RDLH_LINK_UP		BIT(6)
+#define PCIEDMAINTSTSEN		0x0314
+#define  PCIEDMAINTSTSEN_INIT	GENMASK(15, 0)
+
+struct rcar_gen4_pcie {
+	struct dw_pcie		dw;
+	void __iomem		*base;
+	struct reset_control	*rst;
+	bool			needs_retrain;
+};
+#define to_rcar_gen4_pcie(x)	dev_get_drvdata((x)->dev)
+
+int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
+				   int num_lanes);
+int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *pcie);
+void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *pcie);
+int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
+				 struct platform_device *pdev);
+struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev);
+
+#endif /* _PCIE_RCAR_GEN4_H_ */
-- 
2.25.1

[PATCH v16 20/22] PCI: rcar-gen4-ep: Add R-Car Gen4 PCIe Endpoint support

[Yoshihiro Shimoda]

Add R-Car Gen4 PCIe Endpoint support. This controller is based on
Synopsys DesignWare PCIe.

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
---
 drivers/pci/controller/dwc/Kconfig            |   9 +
 drivers/pci/controller/dwc/Makefile           |   2 +
 .../pci/controller/dwc/pcie-rcar-gen4-ep.c    | 166 ++++++++++++++++++
 3 files changed, 177 insertions(+)
 create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4-ep.c

diff --git a/drivers/pci/controller/dwc/Kconfig b/drivers/pci/controller/dwc/Kconfig
index 64d4d37bc891..4d877cd18374 100644
--- a/drivers/pci/controller/dwc/Kconfig
+++ b/drivers/pci/controller/dwc/Kconfig
@@ -424,4 +424,13 @@ config PCIE_RCAR_GEN4
 	  Say Y here if you want PCIe host controller support on R-Car Gen4 SoCs.
 	  This uses the DesignWare core.
 
+config PCIE_RCAR_GEN4_EP
+	tristate "Renesas R-Car Gen4 PCIe Endpoint controller"
+	depends on ARCH_RENESAS || COMPILE_TEST
+	depends on PCI_ENDPOINT
+	select PCIE_DW_EP
+	help
+	  Say Y here if you want PCIe endpoint controller support on R-Car Gen4
+	  SoCs. This uses the DesignWare core.
+
 endmenu
diff --git a/drivers/pci/controller/dwc/Makefile b/drivers/pci/controller/dwc/Makefile
index 486cf706b53d..0fb0bde26ac4 100644
--- a/drivers/pci/controller/dwc/Makefile
+++ b/drivers/pci/controller/dwc/Makefile
@@ -28,6 +28,8 @@ obj-$(CONFIG_PCIE_UNIPHIER_EP) += pcie-uniphier-ep.o
 obj-$(CONFIG_PCIE_VISCONTI_HOST) += pcie-visconti.o
 pcie-rcar-gen4-host-drv-objs := pcie-rcar-gen4.o pcie-rcar-gen4-host.o
 obj-$(CONFIG_PCIE_RCAR_GEN4) += pcie-rcar-gen4-host-drv.o
+pcie-rcar-gen4-ep-drv-objs := pcie-rcar-gen4.o pcie-rcar-gen4-ep.o
+obj-$(CONFIG_PCIE_RCAR_GEN4_EP) += pcie-rcar-gen4-ep-drv.o
 
 # The following drivers are for devices that use the generic ACPI
 # pci_root.c driver but don't support standard ECAM config access.
diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4-ep.c b/drivers/pci/controller/dwc/pcie-rcar-gen4-ep.c
new file mode 100644
index 000000000000..710bbc9e61a5
--- /dev/null
+++ b/drivers/pci/controller/dwc/pcie-rcar-gen4-ep.c
@@ -0,0 +1,166 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * PCIe Endpoint driver for Renesas R-Car Gen4 Series SoCs
+ * Copyright (C) 2022-2023 Renesas Electronics Corporation
+ */
+
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/of_device.h>
+#include <linux/pci.h>
+#include <linux/platform_device.h>
+
+#include "pcie-rcar-gen4.h"
+#include "pcie-designware.h"
+
+static void rcar_gen4_pcie_ep_pre_init(struct dw_pcie_ep *ep)
+{
+	struct dw_pcie *dw = to_dw_pcie_from_ep(ep);
+	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
+	u8 val;
+
+	rcar_gen4_pcie_set_device_type(rcar, false, dw->num_lanes);
+
+	dw_pcie_dbi_ro_wr_en(dw);
+
+	/* Single function */
+	val = dw_pcie_readb_dbi(dw, PCI_HEADER_TYPE);
+	val &= ~PCI_HEADER_TYPE_MULTI_FUNC;
+	dw_pcie_writeb_dbi(dw, PCI_HEADER_TYPE, val);
+
+	dw_pcie_dbi_ro_wr_dis(dw);
+
+	writel(PCIEDMAINTSTSEN_INIT, rcar->base + PCIEDMAINTSTSEN);
+}
+
+static void rcar_gen4_pcie_ep_deinit(struct dw_pcie_ep *ep)
+{
+	struct dw_pcie *dw = to_dw_pcie_from_ep(ep);
+	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
+
+	writel(0, rcar->base + PCIEDMAINTSTSEN);
+}
+
+static int rcar_gen4_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
+				       enum pci_epc_irq_type type,
+				       u16 interrupt_num)
+{
+	struct dw_pcie *dw = to_dw_pcie_from_ep(ep);
+
+	switch (type) {
+	case PCI_EPC_IRQ_INTX:
+		return dw_pcie_ep_raise_intx_irq(ep, func_no);
+	case PCI_EPC_IRQ_MSI:
+		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
+	default:
+		dev_err(dw->dev, "Unknown IRQ type\n");
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static const struct pci_epc_features rcar_gen4_pcie_epc_features = {
+	.linkup_notifier = false,
+	.msi_capable = true,
+	.msix_capable = false,
+	.reserved_bar = 1 << BAR_5,
+	.align = SZ_1M,
+};
+
+static const struct pci_epc_features*
+rcar_gen4_pcie_ep_get_features(struct dw_pcie_ep *ep)
+{
+	return &rcar_gen4_pcie_epc_features;
+}
+
+static const struct dw_pcie_ep_ops pcie_ep_ops = {
+	.ep_pre_init = rcar_gen4_pcie_ep_pre_init,
+	.ep_deinit = rcar_gen4_pcie_ep_deinit,
+	.raise_irq = rcar_gen4_pcie_ep_raise_irq,
+	.get_features = rcar_gen4_pcie_ep_get_features,
+};
+
+static int rcar_gen4_add_pcie_ep(struct rcar_gen4_pcie *rcar,
+				 struct platform_device *pdev)
+{
+	struct dw_pcie_ep *ep = &rcar->dw.ep;
+	int ret;
+
+	ep->ops = &pcie_ep_ops;
+
+	ret = dw_pcie_ep_init(ep);
+	if (ret) {
+		dev_err(&pdev->dev, "Failed to initialize endpoint\n");
+		return ret;
+	}
+
+	return 0;
+}
+
+static void rcar_gen4_remove_pcie_ep(struct rcar_gen4_pcie *rcar)
+{
+	dw_pcie_ep_exit(&rcar->dw.ep);
+}
+
+static int rcar_gen4_pcie_ep_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct rcar_gen4_pcie *rcar;
+	int err;
+
+	rcar = rcar_gen4_pcie_devm_alloc(dev);
+	if (!rcar)
+		return -ENOMEM;
+
+	err = rcar_gen4_pcie_get_resources(rcar, pdev);
+	if (err < 0) {
+		dev_err(dev, "Failed to request resource: %d\n", err);
+		return err;
+	}
+
+	platform_set_drvdata(pdev, rcar);
+
+	err = rcar_gen4_pcie_prepare(rcar);
+	if (err < 0)
+		return err;
+
+	err = rcar_gen4_add_pcie_ep(rcar, pdev);
+	if (err < 0)
+		goto err_add;
+
+	return 0;
+
+err_add:
+	rcar_gen4_pcie_unprepare(rcar);
+
+	return err;
+}
+
+static int rcar_gen4_pcie_ep_remove(struct platform_device *pdev)
+{
+	struct rcar_gen4_pcie *rcar = platform_get_drvdata(pdev);
+
+	rcar_gen4_remove_pcie_ep(rcar);
+	rcar_gen4_pcie_unprepare(rcar);
+
+	return 0;
+}
+
+static const struct of_device_id rcar_gen4_pcie_of_match[] = {
+	{ .compatible = "renesas,rcar-gen4-pcie-ep", },
+	{},
+};
+
+static struct platform_driver rcar_gen4_pcie_ep_driver = {
+	.driver = {
+		.name = "pcie-rcar-gen4-ep",
+		.of_match_table = rcar_gen4_pcie_of_match,
+	},
+	.probe = rcar_gen4_pcie_ep_probe,
+	.remove = rcar_gen4_pcie_ep_remove,
+};
+module_platform_driver(rcar_gen4_pcie_ep_driver);
+
+MODULE_DESCRIPTION("Renesas R-Car Gen4 PCIe endpoint controller driver");
+MODULE_LICENSE("GPL");
-- 
2.25.1

[PATCH v16 21/22] MAINTAINERS: Update PCI DRIVER FOR RENESAS R-CAR for R-Car Gen4

[Yoshihiro Shimoda]

Update this entry for R-Car Gen4's source code.

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Acked-by: Manivannan Sadhasivam <mani@kernel.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 893391ddde6b..9b4db32ab2cf 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -16165,6 +16165,7 @@ L:	linux-renesas-soc@vger.kernel.org
 S:	Maintained
 F:	Documentation/devicetree/bindings/pci/*rcar*
 F:	drivers/pci/controller/*rcar*
+F:	drivers/pci/controller/dwc/*rcar*
 
 PCI DRIVER FOR SAMSUNG EXYNOS
 M:	Jingoo Han <jingoohan1@gmail.com>
-- 
2.25.1

[PATCH v16 22/22] misc: pci_endpoint_test: Add Device ID for R-Car S4-8 PCIe controller

[Yoshihiro Shimoda]

Add Renesas R8A779F0 in pci_device_id table so that pci-epf-test
can be used for testing PCIe EP on R-Car S4-8.

Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Acked-by: Manivannan Sadhasivam <mani@kernel.org>
---
 drivers/misc/pci_endpoint_test.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
index ed4d0ef5e5c3..150083dab71a 100644
--- a/drivers/misc/pci_endpoint_test.c
+++ b/drivers/misc/pci_endpoint_test.c
@@ -81,6 +81,7 @@
 #define PCI_DEVICE_ID_RENESAS_R8A774B1		0x002b
 #define PCI_DEVICE_ID_RENESAS_R8A774C0		0x002d
 #define PCI_DEVICE_ID_RENESAS_R8A774E1		0x0025
+#define PCI_DEVICE_ID_RENESAS_R8A779F0		0x0031
 
 static DEFINE_IDA(pci_endpoint_test_ida);
 
@@ -990,6 +991,9 @@ static const struct pci_device_id pci_endpoint_test_tbl[] = {
 	{ PCI_DEVICE(PCI_VENDOR_ID_RENESAS, PCI_DEVICE_ID_RENESAS_R8A774B1),},
 	{ PCI_DEVICE(PCI_VENDOR_ID_RENESAS, PCI_DEVICE_ID_RENESAS_R8A774C0),},
 	{ PCI_DEVICE(PCI_VENDOR_ID_RENESAS, PCI_DEVICE_ID_RENESAS_R8A774E1),},
+	{ PCI_DEVICE(PCI_VENDOR_ID_RENESAS, PCI_DEVICE_ID_RENESAS_R8A779F0),
+	  .driver_data = (kernel_ulong_t)&default_data,
+	},
 	{ PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_J721E),
 	  .driver_data = (kernel_ulong_t)&j721e_data,
 	},
-- 
2.25.1

Re: [PATCH v16 18/22] dt-bindings: PCI: renesas: Add R-Car Gen4 PCIe Endpoint

[Krzysztof Kozlowski]

On Wed, 10 May 2023 15:22:30 +0900, Yoshihiro Shimoda wrote:
> Document bindings for Renesas R-Car Gen4 and R-Car S4-8 (R8A779F0)
> PCIe endpoint module.
> 
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> Reviewed-by: Rob Herring <robh@kernel.org>
> Reviewed-by: Serge Semin <fancer.lancer@gmail.com>
> Acked-by: Manivannan Sadhasivam <mani@kernel.org>
> ---
>  .../bindings/pci/rcar-gen4-pci-ep.yaml        | 98 +++++++++++++++++++
>  1 file changed, 98 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml
> 

BTW, you keep not-ccing me. Since long time. I don't understand why. I
don't have the emails in inbox, so I won't be responding to your
patchset.


My bot found errors running 'make DT_CHECKER_FLAGS=-m dt_binding_check'
on your patch (DT_CHECKER_FLAGS is new in v5.13):

yamllint warnings/errors:

dtschema/dtc warnings/errors:
/builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.example.dtb: pcie-ep@e65d0000: reg: [[0, 3864854528, 0, 8192], [0, 3864864768, 0, 2048], [0, 3864866816, 0, 8192], [0, 3864875008, 0, 4608], [0, 3864879616, 0, 3584], [0, 4261412864, 0, 4194304]] is too long
	From schema: /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml
/builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.example.dtb: pcie-ep@e65d0000: reg-names: ['dbi', 'dbi2', 'atu', 'dma', 'app', 'addr_space'] is too long
	From schema: /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml

doc reference errors (make refcheckdocs):
Documentation/usb/gadget_uvc.rst: Documentation/userspace-api/media/v4l/pixfmt-packed.yuv.rst
MAINTAINERS: Documentation/devicetree/bindings/pwm/pwm-apple.yaml

See https://patchwork.ozlabs.org/patch/1779260

This check can fail if there are any dependencies. The base for a patch
series is generally the most recent rc1.

If you already ran 'make dt_binding_check' and didn't see the above
error(s), then make sure 'yamllint' is installed and dt-schema is up to
date:

pip3 install dtschema --upgrade

Please check and re-submit.

Re: [PATCH v16 17/22] dt-bindings: PCI: renesas: Add R-Car Gen4 PCIe Host

[Krzysztof Kozlowski]

On Wed, 10 May 2023 15:22:29 +0900, Yoshihiro Shimoda wrote:
> Document bindings for Renesas R-Car Gen4 and R-Car S4-8 (R8A779F0)
> PCIe host module.
> 
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> Reviewed-by: Rob Herring <robh@kernel.org>
> Reviewed-by: Serge Semin <fancer.lancer@gmail.com>
> ---
>  .../bindings/pci/rcar-gen4-pci-host.yaml      | 109 ++++++++++++++++++
>  1 file changed, 109 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.yaml
> 

My bot found errors running 'make DT_CHECKER_FLAGS=-m dt_binding_check'
on your patch (DT_CHECKER_FLAGS is new in v5.13):

yamllint warnings/errors:

dtschema/dtc warnings/errors:
/builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.example.dtb: pcie@e65d0000: reg: [[0, 3864854528, 0, 4096], [0, 3864862720, 0, 2048], [0, 3864866816, 0, 8192], [0, 3864875008, 0, 4608], [0, 3864879616, 0, 3584], [0, 4261412864, 0, 4194304]] is too long
	From schema: /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.yaml
/builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.example.dtb: pcie@e65d0000: reg-names: ['dbi', 'dbi2', 'atu', 'dma', 'app', 'config'] is too long
	From schema: /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.yaml
/builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.example.dtb: pcie@e65d0000: Unevaluated properties are not allowed ('#address-cells', '#interrupt-cells', '#size-cells', 'bus-range', 'device_type', 'dma-ranges', 'interrupt-map', 'interrupt-map-mask', 'ranges', 'snps,enable-cdm-check' were unexpected)
	From schema: /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.yaml


See https://patchwork.ozlabs.org/patch/1779258

This check can fail if there are any dependencies. The base for a patch
series is generally the most recent rc1.

If you already ran 'make dt_binding_check' and didn't see the above
error(s), then make sure 'yamllint' is installed and dt-schema is up to
date:

pip3 install dtschema --upgrade

Please check and re-submit.

RE: [PATCH v16 18/22] dt-bindings: PCI: renesas: Add R-Car Gen4 PCIe Endpoint

[Yoshihiro Shimoda]

Hi Krzysztof,

> From: Krzysztof Kozlowski, Sent: Wednesday, May 10, 2023 7:03 PM
>
> On Wed, 10 May 2023 15:22:30 +0900, Yoshihiro Shimoda wrote:
> > Document bindings for Renesas R-Car Gen4 and R-Car S4-8 (R8A779F0)
> > PCIe endpoint module.
> >
> > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > Reviewed-by: Rob Herring <robh@kernel.org>
> > Reviewed-by: Serge Semin <fancer.lancer@gmail.com>
> > Acked-by: Manivannan Sadhasivam <mani@kernel.org>
> > ---
> >  .../bindings/pci/rcar-gen4-pci-ep.yaml        | 98 +++++++++++++++++++
> >  1 file changed, 98 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml
> >
>
> BTW, you keep not-ccing me. Since long time. I don't understand why. I
> don't have the emails in inbox, so I won't be responding to your
> patchset.

I'm sorry. This is my bad. My using script for sending PCI patches didn't describe
your email address and I didn't realize that until now. Today I added your email
address and Conor's email address into my script.

>
> My bot found errors running 'make DT_CHECKER_FLAGS=-m dt_binding_check'
> on your patch (DT_CHECKER_FLAGS is new in v5.13):
>
> yamllint warnings/errors:
>
> dtschema/dtc warnings/errors:
> /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.example.dtb:
> pcie-ep@e65d0000: reg: [[0, 3864854528, 0, 8192], [0, 3864864768, 0, 2048], [0, 3864866816, 0, 8192], [0, 3864875008,
> 0, 4608], [0, 3864879616, 0, 3584], [0, 4261412864, 0, 4194304]] is too long
>       From schema:
> /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml
> /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.example.dtb:
> pcie-ep@e65d0000: reg-names: ['dbi', 'dbi2', 'atu', 'dma', 'app', 'addr_space'] is too long
>       From schema:
> /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml

I believe these errors disappear if we apply the patch [16/22] of this patch series.
So, we can ignore these errors.

Best regards,
Yoshihiro Shimoda

> doc reference errors (make refcheckdocs):
> Documentation/usb/gadget_uvc.rst: Documentation/userspace-api/media/v4l/pixfmt-packed.yuv.rst
> MAINTAINERS: Documentation/devicetree/bindings/pwm/pwm-apple.yaml
>
> See
> https://patchwork.ozlabs.org/patch/1779260
> 1%7Cyoshihiro.shimoda.uh%40renesas.com%7C84515e5d9f5045c6e2a408db513dc57e%7C53d82571da1947e49cb4625a166a4a2a%7C0%7C0
> %7C638193097971078712%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3
> 000%7C%7C%7C&sdata=57twFUl%2FDPMnxVgxTs%2FJCH5tXNmWqdRyoNMO5yqt8Zc%3D&reserved=0
>
> This check can fail if there are any dependencies. The base for a patch
> series is generally the most recent rc1.
>
> If you already ran 'make dt_binding_check' and didn't see the above
> error(s), then make sure 'yamllint' is installed and dt-schema is up to
> date:
>
> pip3 install dtschema --upgrade
>
> Please check and re-submit.

RE: [PATCH v16 17/22] dt-bindings: PCI: renesas: Add R-Car Gen4 PCIe Host

[Yoshihiro Shimoda]

Hi

> From: Krzysztof Kozlowski, Sent: Wednesday, May 10, 2023 7:04 PM
>
> On Wed, 10 May 2023 15:22:29 +0900, Yoshihiro Shimoda wrote:
> > Document bindings for Renesas R-Car Gen4 and R-Car S4-8 (R8A779F0)
> > PCIe host module.
> >
> > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > Reviewed-by: Rob Herring <robh@kernel.org>
> > Reviewed-by: Serge Semin <fancer.lancer@gmail.com>
> > ---
> >  .../bindings/pci/rcar-gen4-pci-host.yaml      | 109 ++++++++++++++++++
> >  1 file changed, 109 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.yaml
> >
>
> My bot found errors running 'make DT_CHECKER_FLAGS=-m dt_binding_check'
> on your patch (DT_CHECKER_FLAGS is new in v5.13):
>
> yamllint warnings/errors:
>
> dtschema/dtc warnings/errors:
> /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.example.dtb:
> pcie@e65d0000: reg: [[0, 3864854528, 0, 4096], [0, 3864862720, 0, 2048], [0, 3864866816, 0, 8192], [0, 3864875008, 0,
> 4608], [0, 3864879616, 0, 3584], [0, 4261412864, 0, 4194304]] is too long
>       From schema:
> /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.yaml
> /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.example.dtb:
> pcie@e65d0000: reg-names: ['dbi', 'dbi2', 'atu', 'dma', 'app', 'config'] is too long
>       From schema:
> /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.yaml
> /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.example.dtb:
> pcie@e65d0000: Unevaluated properties are not allowed ('#address-cells', '#interrupt-cells', '#size-cells', 'bus-range',
> 'device_type', 'dma-ranges', 'interrupt-map', 'interrupt-map-mask', 'ranges', 'snps,enable-cdm-check' were unexpected)
>       From schema:
> /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.yaml

I'm sorry. This is my bad. My using script for sending PCI patches didn't describe
your email address and I didn't realize that until now. Today I added your email
address and Conor's email address into my script.

Best regards,
Yoshihiro Shimoda

>
> See
> https://patchwork.ozlabs.org/patch/1779258
> 1%7Cyoshihiro.shimoda.uh%40renesas.com%7Ccce74f6aab8246aafb3408db513dd82d%7C53d82571da1947e49cb4625a166a4a2a%7C0%7C0
> %7C638193098276177092%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3
> 000%7C%7C%7C&sdata=rr5rnbZBGfOK87p1dv8T4pEEXyzHuOiKktG9SQZrVrs%3D&reserved=0
>
> This check can fail if there are any dependencies. The base for a patch
> series is generally the most recent rc1.
>
> If you already ran 'make dt_binding_check' and didn't see the above
> error(s), then make sure 'yamllint' is installed and dt-schema is up to
> date:
>
> pip3 install dtschema --upgrade
>
> Please check and re-submit.

Re: [PATCH v16 18/22] dt-bindings: PCI: renesas: Add R-Car Gen4 PCIe Endpoint

[Krzysztof Kozlowski]

On 11/05/2023 02:23, Yoshihiro Shimoda wrote:
> Hi Krzysztof,
> 
>> From: Krzysztof Kozlowski, Sent: Wednesday, May 10, 2023 7:03 PM
>>
>> On Wed, 10 May 2023 15:22:30 +0900, Yoshihiro Shimoda wrote:
>>> Document bindings for Renesas R-Car Gen4 and R-Car S4-8 (R8A779F0)
>>> PCIe endpoint module.
>>>
>>> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
>>> Reviewed-by: Rob Herring <robh@kernel.org>
>>> Reviewed-by: Serge Semin <fancer.lancer@gmail.com>
>>> Acked-by: Manivannan Sadhasivam <mani@kernel.org>
>>> ---
>>>  .../bindings/pci/rcar-gen4-pci-ep.yaml        | 98 +++++++++++++++++++
>>>  1 file changed, 98 insertions(+)
>>>  create mode 100644 Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml
>>>
>>
>> BTW, you keep not-ccing me. Since long time. I don't understand why. I
>> don't have the emails in inbox, so I won't be responding to your
>> patchset.
> 
> I'm sorry. This is my bad. My using script for sending PCI patches didn't describe
> your email address and I didn't realize that until now. Today I added your email
> address and Conor's email address into my script.
> 
>>
>> My bot found errors running 'make DT_CHECKER_FLAGS=-m dt_binding_check'
>> on your patch (DT_CHECKER_FLAGS is new in v5.13):
>>
>> yamllint warnings/errors:
>>
>> dtschema/dtc warnings/errors:
>> /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.example.dtb:
>> pcie-ep@e65d0000: reg: [[0, 3864854528, 0, 8192], [0, 3864864768, 0, 2048], [0, 3864866816, 0, 8192], [0, 3864875008,
>> 0, 4608], [0, 3864879616, 0, 3584], [0, 4261412864, 0, 4194304]] is too long
>>       From schema:
>> /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml
>> /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.example.dtb:
>> pcie-ep@e65d0000: reg-names: ['dbi', 'dbi2', 'atu', 'dma', 'app', 'addr_space'] is too long
>>       From schema:
>> /builds/robherring/dt-review-ci/linux/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml
> 
> I believe these errors disappear if we apply the patch [16/22] of this patch series.
> So, we can ignore these errors.

Yes, seems so.

Best regards,
Krzysztof

Re: [PATCH v16 00/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe support

[Serge Semin]

Hi Yoshihiro

On Wed, May 10, 2023 at 03:22:12PM +0900, Yoshihiro Shimoda wrote:
> Add R-Car S4-8 (R-Car Gen4) PCIe Host and Endpoint support.
> To support them, modify PCIe DesignWare common codes.

Thanks for the updated series. I'll have a look at it on this week.

-Serge(y)

> 
> Changes from v15:
> https://lore.kernel.org/linux-pci/20230509124156.150200-1-yoshihiro.shimoda.uh@renesas.com/
>  - Based on next-20230509 + pci.git / next branch (the commit 174977dc80b7
>    ("Merge branch 'pci/controller/vmd'"))
>  - (no change, JFYI) Based on the following cleanups patches:
>    [PATCH v4 00/14] PCI: dwc: Relatively simple fixes and cleanups
>    https://lore.kernel.org/linux-pci/20230414021832.13167-1-Sergey.Semin@baikalelectronics.ru/
>  - Modify the code comments in patch 8/22.
> 
> Changes from v14:
> https://lore.kernel.org/linux-pci/20230426045557.3613826-1-yoshihiro.shimoda.uh@renesas.com/
>  - Based on next-20230508.
>  - (no change, JFYI) Based on the following cleanups patches:
>    [PATCH v4 00/14] PCI: dwc: Relatively simple fixes and cleanups
>    https://lore.kernel.org/linux-pci/20230414021832.13167-1-Sergey.Semin@baikalelectronics.ru/
>  - Add Reviewed-by from Serge in the patch {4,5,15,}/21.
>  - Drop PCI_EXP_LNKCAP_MLW handling of pcie-tegra194.c because
>    pcie-designware.c takes care of it.
>  - Change subjects in the patch {5,6,7,8,10}/21.
>  - Drop dw_pcie_prog_ep_outbound_atu().
>  - Modify dw_pcie_link_set_max_link_width() to improve code readability.
>  - Move the retrain code to .start_link().
>  - Fix some minor issues.
> 
> Yoshihiro Shimoda (22):
>   PCI: Add PCI_EXP_LNKCAP_MLW macros
>   PCI: Add PCI_HEADER_TYPE_MULTI_FUNC
>   PCI: Add INTx Mechanism Messages macros
>   PCI: Rename PCI_EPC_IRQ_LEGACY to PCI_EPC_IRQ_INTX
>   PCI: dwc: Rename "legacy_irq" to "INTx_irq"
>   PCI: dwc: Change arguments of dw_pcie_prog_outbound_atu()
>   PCI: dwc: Add outbound MSG TLPs support
>   PCI: designware-ep: Add INTx IRQs support
>   PCI: dwc: Add dw_pcie_link_set_max_link_width()
>   PCI: dwc: Modify PCIE_PORT_LINK_CONTROL handling
>   PCI: dwc: Add dw_pcie_link_set_max_cap_width()
>   PCI: tegra194: Drop PCI_EXP_LNKSTA_NLW setting.
>   PCI: dwc: Add EDMA_UNROLL capability flag
>   PCI: dwc: Expose dw_pcie_ep_exit() to module
>   PCI: dwc: Introduce .ep_pre_init() and .ep_deinit()
>   dt-bindings: PCI: dwc: Update maxItems of reg and reg-names
>   dt-bindings: PCI: renesas: Add R-Car Gen4 PCIe Host
>   dt-bindings: PCI: renesas: Add R-Car Gen4 PCIe Endpoint
>   PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support
>   PCI: rcar-gen4-ep: Add R-Car Gen4 PCIe Endpoint support
>   MAINTAINERS: Update PCI DRIVER FOR RENESAS R-CAR for R-Car Gen4
>   misc: pci_endpoint_test: Add Device ID for R-Car S4-8 PCIe controller
> 
>  .../bindings/pci/rcar-gen4-pci-ep.yaml        |  98 +++++++++
>  .../bindings/pci/rcar-gen4-pci-host.yaml      | 109 ++++++++++
>  .../bindings/pci/snps,dw-pcie-ep.yaml         |   4 +-
>  .../devicetree/bindings/pci/snps,dw-pcie.yaml |   4 +-
>  MAINTAINERS                                   |   1 +
>  drivers/misc/pci_endpoint_test.c              |   4 +
>  .../pci/controller/cadence/pcie-cadence-ep.c  |   2 +-
>  drivers/pci/controller/dwc/Kconfig            |  18 ++
>  drivers/pci/controller/dwc/Makefile           |   4 +
>  drivers/pci/controller/dwc/pci-dra7xx.c       |   2 +-
>  drivers/pci/controller/dwc/pci-imx6.c         |   4 +-
>  drivers/pci/controller/dwc/pci-keystone.c     |   2 +-
>  .../pci/controller/dwc/pci-layerscape-ep.c    |   4 +-
>  drivers/pci/controller/dwc/pcie-artpec6.c     |   2 +-
>  .../pci/controller/dwc/pcie-designware-ep.c   |  98 +++++++--
>  .../pci/controller/dwc/pcie-designware-host.c |  52 +++--
>  .../pci/controller/dwc/pcie-designware-plat.c |   4 +-
>  drivers/pci/controller/dwc/pcie-designware.c  | 160 ++++++++-------
>  drivers/pci/controller/dwc/pcie-designware.h  |  33 ++-
>  drivers/pci/controller/dwc/pcie-keembay.c     |   2 +-
>  drivers/pci/controller/dwc/pcie-qcom-ep.c     |   4 +-
>  .../pci/controller/dwc/pcie-rcar-gen4-ep.c    | 166 +++++++++++++++
>  .../pci/controller/dwc/pcie-rcar-gen4-host.c  | 141 +++++++++++++
>  drivers/pci/controller/dwc/pcie-rcar-gen4.c   | 190 ++++++++++++++++++
>  drivers/pci/controller/dwc/pcie-rcar-gen4.h   |  46 +++++
>  drivers/pci/controller/dwc/pcie-tegra194.c    |   8 +-
>  drivers/pci/controller/dwc/pcie-uniphier-ep.c |   2 +-
>  drivers/pci/controller/pcie-rcar-ep.c         |   2 +-
>  drivers/pci/controller/pcie-rockchip-ep.c     |   2 +-
>  drivers/pci/endpoint/functions/pci-epf-test.c |  10 +-
>  drivers/pci/pci.h                             |  18 ++
>  drivers/pci/probe.c                           |   2 +-
>  drivers/pci/quirks.c                          |   4 +-
>  include/linux/pci-epc.h                       |   4 +-
>  include/uapi/linux/pci_regs.h                 |   7 +
>  35 files changed, 1061 insertions(+), 152 deletions(-)
>  create mode 100644 Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml
>  create mode 100644 Documentation/devicetree/bindings/pci/rcar-gen4-pci-host.yaml
>  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4-ep.c
>  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
>  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.c
>  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.h
> 
> -- 
> 2.25.1
> 

Re: [PATCH v16 01/22] PCI: Add PCI_EXP_LNKCAP_MLW macros

[Serge Semin]

On Wed, May 10, 2023 at 03:22:13PM +0900, Yoshihiro Shimoda wrote:
> Add macros defining Maximum Link Width bits in Link Capabilities
> Register.
> 
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> Acked-by: Bjorn Helgaas <bhelgaas@google.com>

You haven't been using these macros in the following up patches since
v9. Why do you keep submitting this change then? I would suggest to
drop the patch especially seeing the PCI_EXP_LNKCAP_MLW field directly
encodes the link width thus these macros unlikely will be of much use.

-Serge(y)

> ---
>  include/uapi/linux/pci_regs.h | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
> index dc2000e0fe3a..5d48413ac28f 100644
> --- a/include/uapi/linux/pci_regs.h
> +++ b/include/uapi/linux/pci_regs.h
> @@ -538,6 +538,12 @@
>  #define  PCI_EXP_LNKCAP_SLS_16_0GB 0x00000004 /* LNKCAP2 SLS Vector bit 3 */
>  #define  PCI_EXP_LNKCAP_SLS_32_0GB 0x00000005 /* LNKCAP2 SLS Vector bit 4 */
>  #define  PCI_EXP_LNKCAP_SLS_64_0GB 0x00000006 /* LNKCAP2 SLS Vector bit 5 */
> +#define  PCI_EXP_LNKCAP_MLW_X1	0x00000010 /* Maximum Link Width x1 */
> +#define  PCI_EXP_LNKCAP_MLW_X2	0x00000020 /* Maximum Link Width x2 */
> +#define  PCI_EXP_LNKCAP_MLW_X4	0x00000040 /* Maximum Link Width x4 */
> +#define  PCI_EXP_LNKCAP_MLW_X8	0x00000080 /* Maximum Link Width x8 */
> +#define  PCI_EXP_LNKCAP_MLW_X12	0x000000c0 /* Maximum Link Width x12 */
> +#define  PCI_EXP_LNKCAP_MLW_X16	0x00000100 /* Maximum Link Width x16 */
>  #define  PCI_EXP_LNKCAP_MLW	0x000003f0 /* Maximum Link Width */
>  #define  PCI_EXP_LNKCAP_ASPMS	0x00000c00 /* ASPM Support */
>  #define  PCI_EXP_LNKCAP_ASPM_L0S 0x00000400 /* ASPM L0s Support */
> -- 
> 2.25.1
> 

Re: [PATCH v16 03/22] PCI: Add INTx Mechanism Messages macros

[Serge Semin]

On Wed, May 10, 2023 at 03:22:15PM +0900, Yoshihiro Shimoda wrote:
> Add "Message Routing" and "INTx Mechanism Messages" macros to send
> a message by a PCIe driver.
> 
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> ---
>  drivers/pci/pci.h | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
> 
> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> index 2475098f6518..67badc40e90b 100644
> --- a/drivers/pci/pci.h
> +++ b/drivers/pci/pci.h
> @@ -11,6 +11,24 @@
>  
>  #define PCI_VSEC_ID_INTEL_TBT	0x1234	/* Thunderbolt */
>  

> +/* Message Routing */

Call it "Implicit Message Routing (r[2:0])" as per the specification.

> +#define PCI_MSG_ROUTING_RC	0
> +#define PCI_MSG_ROUTING_ADDR	1
> +#define PCI_MSG_ROUTING_ID	2
> +#define PCI_MSG_ROUTING_BC	3
> +#define PCI_MSG_ROUTING_LOCAL	4
> +#define PCI_MSG_ROUTING_GATHER	5

IMO prefix like this PCI_MSG_TYPE_R_{RC,ADDR,ID,BC,...} would be a bit
better since it would indicate that this routing flags are a sub-field of
the Message Type field. Bjorn?

> +
> +/* INTx Mechanism Messages */
> +#define PCI_CODE_ASSERT_INTA	0x20
> +#define PCI_CODE_ASSERT_INTB	0x21
> +#define PCI_CODE_ASSERT_INTC	0x22
> +#define PCI_CODE_ASSERT_INTD	0x23
> +#define PCI_CODE_DEASSERT_INTA	0x24
> +#define PCI_CODE_DEASSERT_INTB	0x25
> +#define PCI_CODE_DEASSERT_INTC	0x26
> +#define PCI_CODE_DEASSERT_INTD	0x27

IMO Prefix PCI_MSG_CODE_... would be a bit more descriptive since per
the specification the respective message field is called "Message
Code" and not just "Code". Bjorn?

-Serge(y)

> +
>  extern const unsigned char pcie_link_speed[];
>  extern bool pci_early_dump;
>  
> -- 
> 2.25.1
> 

Re: [PATCH v16 04/22] PCI: Rename PCI_EPC_IRQ_LEGACY to PCI_EPC_IRQ_INTX

[Serge Semin]

On Wed, May 10, 2023 at 03:22:16PM +0900, Yoshihiro Shimoda wrote:
> Using "INTx" instead of "legacy" is more specific. So, rename
> PCI_EPC_IRQ_LEGACY to PCI_EPC_IRQ_INTX.
> 
> Suggested-by: Bjorn Helgaas <helgaas@kernel.org>
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>

> Cc: Tom Joseph <tjoseph@cadence.com>
> Cc: Vignesh Raghavendra <vigneshr@ti.com>
> Cc: Richard Zhu <hongxing.zhu@nxp.com>
> Cc: Lucas Stach <l.stach@pengutronix.de>
> Cc: Shawn Guo <shawnguo@kernel.org>
> Cc: Sascha Hauer <s.hauer@pengutronix.de>
> Cc: Pengutronix Kernel Team <kernel@pengutronix.de>
> Cc: Fabio Estevam <festevam@gmail.com>
> Cc: NXP Linux Team <linux-imx@nxp.com>
> Cc: Minghuan Lian <minghuan.Lian@nxp.com>
> Cc: Mingkai Hu <mingkai.hu@nxp.com>
> Cc: Roy Zang <roy.zang@nxp.com>
> Cc: Jingoo Han <jingoohan1@gmail.com>
> Cc: Srikanth Thokala <srikanth.thokala@intel.com>
> Cc: Thierry Reding <thierry.reding@gmail.com>
> Cc: Jonathan Hunter <jonathanh@nvidia.com>
> Cc: Kunihiko Hayashi <hayashi.kunihiko@socionext.com>
> Cc: Masami Hiramatsu <mhiramat@kernel.org>
> Cc: Marek Vasut <marek.vasut+renesas@gmail.com>
> Cc: Shawn Lin <shawn.lin@rock-chips.com>
> Cc: Heiko Stuebner <heiko@sntech.de>
> Cc: Kishon Vijay Abraham I <kishon@kernel.org>

I doubt that that long Cc-tags list of much use in this case
especially seeing the change is mainly relevant to the PCIe
subsystem core. In order to still let git send-email to create a
pre-defined Cc-list I normally move all the Cc'es to be below the
"---" line. Thus the specified developers will be added by the
send-email tool to the list of recipients but git am will ignore
everything below "---" so the mainline commit won't have these
Cc-tags.

-Serge(y)

> Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
> Acked-by: Jesper Nilsson <jesper.nilsson@axis.com> # ARTPEC
> Reviewed-by: Serge Semin <fancer.lancer@gmail.com>
> ---
>  drivers/pci/controller/cadence/pcie-cadence-ep.c  |  2 +-
>  drivers/pci/controller/dwc/pci-dra7xx.c           |  2 +-
>  drivers/pci/controller/dwc/pci-imx6.c             |  2 +-
>  drivers/pci/controller/dwc/pci-keystone.c         |  2 +-
>  drivers/pci/controller/dwc/pci-layerscape-ep.c    |  2 +-
>  drivers/pci/controller/dwc/pcie-artpec6.c         |  2 +-
>  drivers/pci/controller/dwc/pcie-designware-plat.c |  2 +-
>  drivers/pci/controller/dwc/pcie-keembay.c         |  2 +-
>  drivers/pci/controller/dwc/pcie-qcom-ep.c         |  2 +-
>  drivers/pci/controller/dwc/pcie-tegra194.c        |  2 +-
>  drivers/pci/controller/dwc/pcie-uniphier-ep.c     |  2 +-
>  drivers/pci/controller/pcie-rcar-ep.c             |  2 +-
>  drivers/pci/controller/pcie-rockchip-ep.c         |  2 +-
>  drivers/pci/endpoint/functions/pci-epf-test.c     | 10 +++++-----
>  include/linux/pci-epc.h                           |  4 ++--
>  15 files changed, 20 insertions(+), 20 deletions(-)
> 
> diff --git a/drivers/pci/controller/cadence/pcie-cadence-ep.c b/drivers/pci/controller/cadence/pcie-cadence-ep.c
> index b8b655d4047e..2af8eb4e6d91 100644
> --- a/drivers/pci/controller/cadence/pcie-cadence-ep.c
> +++ b/drivers/pci/controller/cadence/pcie-cadence-ep.c
> @@ -539,7 +539,7 @@ static int cdns_pcie_ep_raise_irq(struct pci_epc *epc, u8 fn, u8 vfn,
>  	struct device *dev = pcie->dev;
>  
>  	switch (type) {
> -	case PCI_EPC_IRQ_LEGACY:
> +	case PCI_EPC_IRQ_INTX:
>  		if (vfn > 0) {
>  			dev_err(dev, "Cannot raise legacy interrupts for VF\n");
>  			return -EINVAL;
> diff --git a/drivers/pci/controller/dwc/pci-dra7xx.c b/drivers/pci/controller/dwc/pci-dra7xx.c
> index 4ae807e7cf79..b42fb1cc8bc8 100644
> --- a/drivers/pci/controller/dwc/pci-dra7xx.c
> +++ b/drivers/pci/controller/dwc/pci-dra7xx.c
> @@ -410,7 +410,7 @@ static int dra7xx_pcie_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	struct dra7xx_pcie *dra7xx = to_dra7xx_pcie(pci);
>  
>  	switch (type) {
> -	case PCI_EPC_IRQ_LEGACY:
> +	case PCI_EPC_IRQ_INTX:
>  		dra7xx_pcie_raise_legacy_irq(dra7xx);
>  		break;
>  	case PCI_EPC_IRQ_MSI:
> diff --git a/drivers/pci/controller/dwc/pci-imx6.c b/drivers/pci/controller/dwc/pci-imx6.c
> index 52906f999f2b..1f39e733ce19 100644
> --- a/drivers/pci/controller/dwc/pci-imx6.c
> +++ b/drivers/pci/controller/dwc/pci-imx6.c
> @@ -1062,7 +1062,7 @@ static int imx6_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  
>  	switch (type) {
> -	case PCI_EPC_IRQ_LEGACY:
> +	case PCI_EPC_IRQ_INTX:
>  		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
>  	case PCI_EPC_IRQ_MSI:
>  		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
> diff --git a/drivers/pci/controller/dwc/pci-keystone.c b/drivers/pci/controller/dwc/pci-keystone.c
> index 78818853af9e..3806f5530937 100644
> --- a/drivers/pci/controller/dwc/pci-keystone.c
> +++ b/drivers/pci/controller/dwc/pci-keystone.c
> @@ -908,7 +908,7 @@ static int ks_pcie_am654_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	struct keystone_pcie *ks_pcie = to_keystone_pcie(pci);
>  
>  	switch (type) {
> -	case PCI_EPC_IRQ_LEGACY:
> +	case PCI_EPC_IRQ_INTX:
>  		ks_pcie_am654_raise_legacy_irq(ks_pcie);
>  		break;
>  	case PCI_EPC_IRQ_MSI:
> diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> index c640db60edc6..ab3306e206d8 100644
> --- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> +++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> @@ -65,7 +65,7 @@ static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  
>  	switch (type) {
> -	case PCI_EPC_IRQ_LEGACY:
> +	case PCI_EPC_IRQ_INTX:
>  		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
>  	case PCI_EPC_IRQ_MSI:
>  		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
> diff --git a/drivers/pci/controller/dwc/pcie-artpec6.c b/drivers/pci/controller/dwc/pcie-artpec6.c
> index 98102079e26d..128cb1118e3a 100644
> --- a/drivers/pci/controller/dwc/pcie-artpec6.c
> +++ b/drivers/pci/controller/dwc/pcie-artpec6.c
> @@ -357,7 +357,7 @@ static int artpec6_pcie_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  
>  	switch (type) {
> -	case PCI_EPC_IRQ_LEGACY:
> +	case PCI_EPC_IRQ_INTX:
>  		dev_err(pci->dev, "EP cannot trigger legacy IRQs\n");
>  		return -EINVAL;
>  	case PCI_EPC_IRQ_MSI:
> diff --git a/drivers/pci/controller/dwc/pcie-designware-plat.c b/drivers/pci/controller/dwc/pcie-designware-plat.c
> index 1fcfb840f238..fc3b02949218 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-plat.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-plat.c
> @@ -48,7 +48,7 @@ static int dw_plat_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  
>  	switch (type) {
> -	case PCI_EPC_IRQ_LEGACY:
> +	case PCI_EPC_IRQ_INTX:
>  		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
>  	case PCI_EPC_IRQ_MSI:
>  		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
> diff --git a/drivers/pci/controller/dwc/pcie-keembay.c b/drivers/pci/controller/dwc/pcie-keembay.c
> index f90f36bac018..ceb940b327cb 100644
> --- a/drivers/pci/controller/dwc/pcie-keembay.c
> +++ b/drivers/pci/controller/dwc/pcie-keembay.c
> @@ -290,7 +290,7 @@ static int keembay_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  
>  	switch (type) {
> -	case PCI_EPC_IRQ_LEGACY:
> +	case PCI_EPC_IRQ_INTX:
>  		/* Legacy interrupts are not supported in Keem Bay */
>  		dev_err(pci->dev, "Legacy IRQ is not supported\n");
>  		return -EINVAL;
> diff --git a/drivers/pci/controller/dwc/pcie-qcom-ep.c b/drivers/pci/controller/dwc/pcie-qcom-ep.c
> index 19b32839ea26..077afce48d0b 100644
> --- a/drivers/pci/controller/dwc/pcie-qcom-ep.c
> +++ b/drivers/pci/controller/dwc/pcie-qcom-ep.c
> @@ -658,7 +658,7 @@ static int qcom_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  
>  	switch (type) {
> -	case PCI_EPC_IRQ_LEGACY:
> +	case PCI_EPC_IRQ_INTX:
>  		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
>  	case PCI_EPC_IRQ_MSI:
>  		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
> diff --git a/drivers/pci/controller/dwc/pcie-tegra194.c b/drivers/pci/controller/dwc/pcie-tegra194.c
> index 09825b4a075e..4adba379b83d 100644
> --- a/drivers/pci/controller/dwc/pcie-tegra194.c
> +++ b/drivers/pci/controller/dwc/pcie-tegra194.c
> @@ -1980,7 +1980,7 @@ static int tegra_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	struct tegra_pcie_dw *pcie = to_tegra_pcie(pci);
>  
>  	switch (type) {
> -	case PCI_EPC_IRQ_LEGACY:
> +	case PCI_EPC_IRQ_INTX:
>  		return tegra_pcie_ep_raise_legacy_irq(pcie, interrupt_num);
>  
>  	case PCI_EPC_IRQ_MSI:
> diff --git a/drivers/pci/controller/dwc/pcie-uniphier-ep.c b/drivers/pci/controller/dwc/pcie-uniphier-ep.c
> index 4d0a587c0ba5..7787eedf87f4 100644
> --- a/drivers/pci/controller/dwc/pcie-uniphier-ep.c
> +++ b/drivers/pci/controller/dwc/pcie-uniphier-ep.c
> @@ -262,7 +262,7 @@ static int uniphier_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  
>  	switch (type) {
> -	case PCI_EPC_IRQ_LEGACY:
> +	case PCI_EPC_IRQ_INTX:
>  		return uniphier_pcie_ep_raise_legacy_irq(ep);
>  	case PCI_EPC_IRQ_MSI:
>  		return uniphier_pcie_ep_raise_msi_irq(ep, func_no,
> diff --git a/drivers/pci/controller/pcie-rcar-ep.c b/drivers/pci/controller/pcie-rcar-ep.c
> index f9682df1da61..fbdf3d85301c 100644
> --- a/drivers/pci/controller/pcie-rcar-ep.c
> +++ b/drivers/pci/controller/pcie-rcar-ep.c
> @@ -408,7 +408,7 @@ static int rcar_pcie_ep_raise_irq(struct pci_epc *epc, u8 fn, u8 vfn,
>  	struct rcar_pcie_endpoint *ep = epc_get_drvdata(epc);
>  
>  	switch (type) {
> -	case PCI_EPC_IRQ_LEGACY:
> +	case PCI_EPC_IRQ_INTX:
>  		return rcar_pcie_ep_assert_intx(ep, fn, 0);
>  
>  	case PCI_EPC_IRQ_MSI:
> diff --git a/drivers/pci/controller/pcie-rockchip-ep.c b/drivers/pci/controller/pcie-rockchip-ep.c
> index d1a200b93b2b..ef9d1f6c382a 100644
> --- a/drivers/pci/controller/pcie-rockchip-ep.c
> +++ b/drivers/pci/controller/pcie-rockchip-ep.c
> @@ -477,7 +477,7 @@ static int rockchip_pcie_ep_raise_irq(struct pci_epc *epc, u8 fn, u8 vfn,
>  	struct rockchip_pcie_ep *ep = epc_get_drvdata(epc);
>  
>  	switch (type) {
> -	case PCI_EPC_IRQ_LEGACY:
> +	case PCI_EPC_IRQ_INTX:
>  		return rockchip_pcie_ep_send_legacy_irq(ep, fn, 0);
>  	case PCI_EPC_IRQ_MSI:
>  		return rockchip_pcie_ep_send_msi_irq(ep, fn, interrupt_num);
> diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c b/drivers/pci/endpoint/functions/pci-epf-test.c
> index 623b08caa998..6beb3f2b0afb 100644
> --- a/drivers/pci/endpoint/functions/pci-epf-test.c
> +++ b/drivers/pci/endpoint/functions/pci-epf-test.c
> @@ -19,11 +19,11 @@
>  #include <linux/pci-epf.h>
>  #include <linux/pci_regs.h>
>  
> -#define IRQ_TYPE_LEGACY			0
> +#define IRQ_TYPE_INTX			0
>  #define IRQ_TYPE_MSI			1
>  #define IRQ_TYPE_MSIX			2
>  
> -#define COMMAND_RAISE_LEGACY_IRQ	BIT(0)
> +#define COMMAND_RAISE_INTX_IRQ		BIT(0)
>  #define COMMAND_RAISE_MSI_IRQ		BIT(1)
>  #define COMMAND_RAISE_MSIX_IRQ		BIT(2)
>  #define COMMAND_READ			BIT(3)
> @@ -600,9 +600,9 @@ static void pci_epf_test_raise_irq(struct pci_epf_test *epf_test,
>  	WRITE_ONCE(reg->status, status);
>  
>  	switch (reg->irq_type) {
> -	case IRQ_TYPE_LEGACY:
> +	case IRQ_TYPE_INTX:
>  		pci_epc_raise_irq(epc, epf->func_no, epf->vfunc_no,
> -				  PCI_EPC_IRQ_LEGACY, 0);
> +				  PCI_EPC_IRQ_INTX, 0);
>  		break;
>  	case IRQ_TYPE_MSI:
>  		count = pci_epc_get_msi(epc, epf->func_no, epf->vfunc_no);
> @@ -659,7 +659,7 @@ static void pci_epf_test_cmd_handler(struct work_struct *work)
>  	}
>  
>  	switch (command) {
> -	case COMMAND_RAISE_LEGACY_IRQ:
> +	case COMMAND_RAISE_INTX_IRQ:
>  	case COMMAND_RAISE_MSI_IRQ:
>  	case COMMAND_RAISE_MSIX_IRQ:
>  		pci_epf_test_raise_irq(epf_test, reg);
> diff --git a/include/linux/pci-epc.h b/include/linux/pci-epc.h
> index 301bb0e53707..c2572a93d73d 100644
> --- a/include/linux/pci-epc.h
> +++ b/include/linux/pci-epc.h
> @@ -21,7 +21,7 @@ enum pci_epc_interface_type {
>  
>  enum pci_epc_irq_type {
>  	PCI_EPC_IRQ_UNKNOWN,
> -	PCI_EPC_IRQ_LEGACY,
> +	PCI_EPC_IRQ_INTX,
>  	PCI_EPC_IRQ_MSI,
>  	PCI_EPC_IRQ_MSIX,
>  };
> @@ -54,7 +54,7 @@ pci_epc_interface_string(enum pci_epc_interface_type type)
>   *	     MSI-X capability register
>   * @get_msix: ops to get the number of MSI-X interrupts allocated by the RC
>   *	     from the MSI-X capability register
> - * @raise_irq: ops to raise a legacy, MSI or MSI-X interrupt
> + * @raise_irq: ops to raise an INTx, MSI or MSI-X interrupt
>   * @map_msi_irq: ops to map physical address to MSI address and return MSI data
>   * @start: ops to start the PCI link
>   * @stop: ops to stop the PCI link
> -- 
> 2.25.1
> 

Re: [PATCH v16 06/22] PCI: dwc: Change arguments of dw_pcie_prog_outbound_atu()

[Serge Semin]

On Wed, May 10, 2023 at 03:22:18PM +0900, Yoshihiro Shimoda wrote:
> To add more arguments to the dw_pcie_prog_outbound_atu() in
> the future, introduce struct dw_pcie_ob_atu_cfg and change
> the argument. And, drop dw_pcie_prog_ep_outbound_atu(). No behavior
> changes.

I doubt anyone not being aware of the change context will understand
your message. More details would help with that: why the conversion
was necessary, how come the dw_pcie_prog_ep_outbound_atu() function
turns to be redundant, what additional parameters will be added
afterwards so this patch turns to be a preparation patch for that, etc.

Other than that the change looks good.

-Serge(y)

> 
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> ---
>  .../pci/controller/dwc/pcie-designware-ep.c   | 21 +++++---
>  .../pci/controller/dwc/pcie-designware-host.c | 52 +++++++++++++------
>  drivers/pci/controller/dwc/pcie-designware.c  | 49 ++++++-----------
>  drivers/pci/controller/dwc/pcie-designware.h  | 15 ++++--
>  4 files changed, 77 insertions(+), 60 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
> index 27278010ecec..fe2e0d765be9 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> @@ -182,9 +182,8 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no, int type,
>  	return 0;
>  }
>  
> -static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, u8 func_no,
> -				   phys_addr_t phys_addr,
> -				   u64 pci_addr, size_t size)
> +static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep,
> +				   struct dw_pcie_ob_atu_cfg *atu)
>  {
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  	u32 free_win;
> @@ -196,13 +195,13 @@ static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, u8 func_no,
>  		return -EINVAL;
>  	}
>  
> -	ret = dw_pcie_prog_ep_outbound_atu(pci, func_no, free_win, PCIE_ATU_TYPE_MEM,
> -					   phys_addr, pci_addr, size);
> +	atu->index = free_win;
> +	ret = dw_pcie_prog_outbound_atu(pci, atu);
>  	if (ret)
>  		return ret;
>  
>  	set_bit(free_win, ep->ob_window_map);
> -	ep->outbound_addr[free_win] = phys_addr;
> +	ep->outbound_addr[free_win] = atu->cpu_addr;
>  
>  	return 0;
>  }
> @@ -305,8 +304,14 @@ static int dw_pcie_ep_map_addr(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
>  	int ret;
>  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> -
> -	ret = dw_pcie_ep_outbound_atu(ep, func_no, addr, pci_addr, size);
> +	struct dw_pcie_ob_atu_cfg atu = { 0 };
> +
> +	atu.func_no = func_no;
> +	atu.type = PCIE_ATU_TYPE_MEM;
> +	atu.cpu_addr = addr;
> +	atu.pci_addr = pci_addr;
> +	atu.size = size;
> +	ret = dw_pcie_ep_outbound_atu(ep, &atu);
>  	if (ret) {
>  		dev_err(pci->dev, "Failed to enable address\n");
>  		return ret;
> diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
> index 5718b4bb67f0..676216d540fe 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> @@ -544,6 +544,7 @@ static void __iomem *dw_pcie_other_conf_map_bus(struct pci_bus *bus,
>  {
>  	struct dw_pcie_rp *pp = bus->sysdata;
>  	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> +	struct dw_pcie_ob_atu_cfg atu = { 0 };
>  	int type, ret;
>  	u32 busdev;
>  
> @@ -566,8 +567,12 @@ static void __iomem *dw_pcie_other_conf_map_bus(struct pci_bus *bus,
>  	else
>  		type = PCIE_ATU_TYPE_CFG1;
>  
> -	ret = dw_pcie_prog_outbound_atu(pci, 0, type, pp->cfg0_base, busdev,
> -					pp->cfg0_size);
> +	atu.type = type;
> +	atu.cpu_addr = pp->cfg0_base;
> +	atu.pci_addr = busdev;
> +	atu.size = pp->cfg0_size;
> +
> +	ret = dw_pcie_prog_outbound_atu(pci, &atu);
>  	if (ret)
>  		return NULL;
>  
> @@ -579,6 +584,7 @@ static int dw_pcie_rd_other_conf(struct pci_bus *bus, unsigned int devfn,
>  {
>  	struct dw_pcie_rp *pp = bus->sysdata;
>  	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> +	struct dw_pcie_ob_atu_cfg atu = { 0 };
>  	int ret;
>  
>  	ret = pci_generic_config_read(bus, devfn, where, size, val);
> @@ -586,9 +592,12 @@ static int dw_pcie_rd_other_conf(struct pci_bus *bus, unsigned int devfn,
>  		return ret;
>  
>  	if (pp->cfg0_io_shared) {
> -		ret = dw_pcie_prog_outbound_atu(pci, 0, PCIE_ATU_TYPE_IO,
> -						pp->io_base, pp->io_bus_addr,
> -						pp->io_size);
> +		atu.type = PCIE_ATU_TYPE_IO;
> +		atu.cpu_addr = pp->io_base;
> +		atu.pci_addr = pp->io_bus_addr;
> +		atu.size = pp->io_size;
> +
> +		ret = dw_pcie_prog_outbound_atu(pci, &atu);
>  		if (ret)
>  			return PCIBIOS_SET_FAILED;
>  	}
> @@ -601,6 +610,7 @@ static int dw_pcie_wr_other_conf(struct pci_bus *bus, unsigned int devfn,
>  {
>  	struct dw_pcie_rp *pp = bus->sysdata;
>  	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> +	struct dw_pcie_ob_atu_cfg atu = { 0 };
>  	int ret;
>  
>  	ret = pci_generic_config_write(bus, devfn, where, size, val);
> @@ -608,9 +618,12 @@ static int dw_pcie_wr_other_conf(struct pci_bus *bus, unsigned int devfn,
>  		return ret;
>  
>  	if (pp->cfg0_io_shared) {
> -		ret = dw_pcie_prog_outbound_atu(pci, 0, PCIE_ATU_TYPE_IO,
> -						pp->io_base, pp->io_bus_addr,
> -						pp->io_size);
> +		atu.type = PCIE_ATU_TYPE_IO;
> +		atu.cpu_addr = pp->io_base;
> +		atu.pci_addr = pp->io_bus_addr;
> +		atu.size = pp->io_size;
> +
> +		ret = dw_pcie_prog_outbound_atu(pci, &atu);
>  		if (ret)
>  			return PCIBIOS_SET_FAILED;
>  	}
> @@ -645,6 +658,7 @@ static struct pci_ops dw_pcie_ops = {
>  static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
>  {
>  	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> +	struct dw_pcie_ob_atu_cfg atu = { 0 };
>  	struct resource_entry *entry;
>  	int i, ret;
>  
> @@ -672,10 +686,13 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
>  		if (pci->num_ob_windows <= ++i)
>  			break;
>  
> -		ret = dw_pcie_prog_outbound_atu(pci, i, PCIE_ATU_TYPE_MEM,
> -						entry->res->start,
> -						entry->res->start - entry->offset,
> -						resource_size(entry->res));
> +		atu.index = i;
> +		atu.type = PCIE_ATU_TYPE_MEM;
> +		atu.cpu_addr = entry->res->start;
> +		atu.pci_addr = entry->res->start - entry->offset;
> +		atu.size = resource_size(entry->res);
> +
> +		ret = dw_pcie_prog_outbound_atu(pci, &atu);
>  		if (ret) {
>  			dev_err(pci->dev, "Failed to set MEM range %pr\n",
>  				entry->res);
> @@ -685,10 +702,13 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
>  
>  	if (pp->io_size) {
>  		if (pci->num_ob_windows > ++i) {
> -			ret = dw_pcie_prog_outbound_atu(pci, i, PCIE_ATU_TYPE_IO,
> -							pp->io_base,
> -							pp->io_bus_addr,
> -							pp->io_size);
> +			atu.index = i;
> +			atu.type = PCIE_ATU_TYPE_IO;
> +			atu.cpu_addr = pp->io_base;
> +			atu.pci_addr = pp->io_bus_addr;
> +			atu.size = pp->io_size;
> +
> +			ret = dw_pcie_prog_outbound_atu(pci, &atu);
>  			if (ret) {
>  				dev_err(pci->dev, "Failed to set IO range %pr\n",
>  					entry->res);
> diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
> index ede166645289..bd85a73d354b 100644
> --- a/drivers/pci/controller/dwc/pcie-designware.c
> +++ b/drivers/pci/controller/dwc/pcie-designware.c
> @@ -464,56 +464,56 @@ static inline u32 dw_pcie_enable_ecrc(u32 val)
>  	return val | PCIE_ATU_TD;
>  }
>  
> -static int __dw_pcie_prog_outbound_atu(struct dw_pcie *pci, u8 func_no,
> -				       int index, int type, u64 cpu_addr,
> -				       u64 pci_addr, u64 size)
> +int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
> +			      const struct dw_pcie_ob_atu_cfg *atu)
>  {
> +	u64 cpu_addr = atu->cpu_addr;
>  	u32 retries, val;
>  	u64 limit_addr;
>  
>  	if (pci->ops && pci->ops->cpu_addr_fixup)
>  		cpu_addr = pci->ops->cpu_addr_fixup(pci, cpu_addr);
>  
> -	limit_addr = cpu_addr + size - 1;
> +	limit_addr = cpu_addr + atu->size - 1;
>  
>  	if ((limit_addr & ~pci->region_limit) != (cpu_addr & ~pci->region_limit) ||
>  	    !IS_ALIGNED(cpu_addr, pci->region_align) ||
> -	    !IS_ALIGNED(pci_addr, pci->region_align) || !size) {
> +	    !IS_ALIGNED(atu->pci_addr, pci->region_align) || !atu->size) {
>  		return -EINVAL;
>  	}
>  
> -	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_LOWER_BASE,
> +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_LOWER_BASE,
>  			      lower_32_bits(cpu_addr));
> -	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_UPPER_BASE,
> +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_BASE,
>  			      upper_32_bits(cpu_addr));
>  
> -	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_LIMIT,
> +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_LIMIT,
>  			      lower_32_bits(limit_addr));
>  	if (dw_pcie_ver_is_ge(pci, 460A))
> -		dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_UPPER_LIMIT,
> +		dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_LIMIT,
>  				      upper_32_bits(limit_addr));
>  
> -	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_LOWER_TARGET,
> -			      lower_32_bits(pci_addr));
> -	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_UPPER_TARGET,
> -			      upper_32_bits(pci_addr));
> +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_LOWER_TARGET,
> +			      lower_32_bits(atu->pci_addr));
> +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_TARGET,
> +			      upper_32_bits(atu->pci_addr));
>  
> -	val = type | PCIE_ATU_FUNC_NUM(func_no);
> +	val = atu->type | PCIE_ATU_FUNC_NUM(atu->func_no);
>  	if (upper_32_bits(limit_addr) > upper_32_bits(cpu_addr) &&
>  	    dw_pcie_ver_is_ge(pci, 460A))
>  		val |= PCIE_ATU_INCREASE_REGION_SIZE;
>  	if (dw_pcie_ver_is(pci, 490A))
>  		val = dw_pcie_enable_ecrc(val);
> -	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_REGION_CTRL1, val);
> +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL1, val);
>  
> -	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_REGION_CTRL2, PCIE_ATU_ENABLE);
> +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2, PCIE_ATU_ENABLE);
>  
>  	/*
>  	 * Make sure ATU enable takes effect before any subsequent config
>  	 * and I/O accesses.
>  	 */
>  	for (retries = 0; retries < LINK_WAIT_MAX_IATU_RETRIES; retries++) {
> -		val = dw_pcie_readl_atu_ob(pci, index, PCIE_ATU_REGION_CTRL2);
> +		val = dw_pcie_readl_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2);
>  		if (val & PCIE_ATU_ENABLE)
>  			return 0;
>  
> @@ -525,21 +525,6 @@ static int __dw_pcie_prog_outbound_atu(struct dw_pcie *pci, u8 func_no,
>  	return -ETIMEDOUT;
>  }
>  
> -int dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
> -			      u64 cpu_addr, u64 pci_addr, u64 size)
> -{
> -	return __dw_pcie_prog_outbound_atu(pci, 0, index, type,
> -					   cpu_addr, pci_addr, size);
> -}
> -
> -int dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> -				 int type, u64 cpu_addr, u64 pci_addr,
> -				 u64 size)
> -{
> -	return __dw_pcie_prog_outbound_atu(pci, func_no, index, type,
> -					   cpu_addr, pci_addr, size);
> -}
> -
>  static inline u32 dw_pcie_readl_atu_ib(struct dw_pcie *pci, u32 index, u32 reg)
>  {
>  	return dw_pcie_readl_atu(pci, PCIE_ATU_REGION_DIR_IB, index, reg);
> diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
> index 9acf6c40d252..b8fa099bbed3 100644
> --- a/drivers/pci/controller/dwc/pcie-designware.h
> +++ b/drivers/pci/controller/dwc/pcie-designware.h
> @@ -291,6 +291,15 @@ enum dw_pcie_core_rst {
>  	DW_PCIE_NUM_CORE_RSTS
>  };
>  
> +struct dw_pcie_ob_atu_cfg {
> +	int index;
> +	int type;
> +	u8 func_no;
> +	u64 cpu_addr;
> +	u64 pci_addr;
> +	u64 size;
> +};
> +
>  struct dw_pcie_host_ops {
>  	int (*host_init)(struct dw_pcie_rp *pp);
>  	void (*host_deinit)(struct dw_pcie_rp *pp);
> @@ -419,10 +428,8 @@ void dw_pcie_write_dbi2(struct dw_pcie *pci, u32 reg, size_t size, u32 val);
>  int dw_pcie_link_up(struct dw_pcie *pci);
>  void dw_pcie_upconfig_setup(struct dw_pcie *pci);
>  int dw_pcie_wait_for_link(struct dw_pcie *pci);
> -int dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
> -			      u64 cpu_addr, u64 pci_addr, u64 size);
> -int dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> -				 int type, u64 cpu_addr, u64 pci_addr, u64 size);
> +int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
> +			      const struct dw_pcie_ob_atu_cfg *atu);
>  int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int type,
>  			     u64 cpu_addr, u64 pci_addr, u64 size);
>  int dw_pcie_prog_ep_inbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> -- 
> 2.25.1
> 

RE: [PATCH v16 01/22] PCI: Add PCI_EXP_LNKCAP_MLW macros

[Yoshihiro Shimoda]

Hello Serge,

> From: Serge Semin, Sent: Monday, June 5, 2023 7:51 AM
> 
> On Wed, May 10, 2023 at 03:22:13PM +0900, Yoshihiro Shimoda wrote:
> > Add macros defining Maximum Link Width bits in Link Capabilities
> > Register.
> >
> > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> 
> You haven't been using these macros in the following up patches since
> v9. Why do you keep submitting this change then? I would suggest to
> drop the patch especially seeing the PCI_EXP_LNKCAP_MLW field directly
> encodes the link width thus these macros unlikely will be of much use.

Thank you for your review! You're correct. I didn't realized that
the macros were not used on the patch series..

However, I also realized that the patch 11/22 used the PCI_EXP_LNKSTA_NLW_SHIFT
macro for the PCI_EXP_LNKCAP register. So, I'm thinking that I should modify
this patch as adding PCI_EXP_LNKCAP_MLW_SHIFT instead. But, what do you think?

--- on the patch 11/22 ---
> +	cap = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP);
> +	val = dw_pcie_readl_dbi(pci, cap + PCI_EXP_LNKCAP);
> +	val &= ~PCI_EXP_LNKCAP_MLW;
> +	val |= num_lanes << PCI_EXP_LNKSTA_NLW_SHIFT;

Perhaps, this should be PCI_EXP_LNKCAP_MLW_SHIFT instead.
---

Best regards,
Yoshihiro Shimoda

> -Serge(y)
> 
> > ---
> >  include/uapi/linux/pci_regs.h | 6 ++++++
> >  1 file changed, 6 insertions(+)
> >
> > diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
> > index dc2000e0fe3a..5d48413ac28f 100644
> > --- a/include/uapi/linux/pci_regs.h
> > +++ b/include/uapi/linux/pci_regs.h
> > @@ -538,6 +538,12 @@
> >  #define  PCI_EXP_LNKCAP_SLS_16_0GB 0x00000004 /* LNKCAP2 SLS Vector bit 3 */
> >  #define  PCI_EXP_LNKCAP_SLS_32_0GB 0x00000005 /* LNKCAP2 SLS Vector bit 4 */
> >  #define  PCI_EXP_LNKCAP_SLS_64_0GB 0x00000006 /* LNKCAP2 SLS Vector bit 5 */
> > +#define  PCI_EXP_LNKCAP_MLW_X1	0x00000010 /* Maximum Link Width x1 */
> > +#define  PCI_EXP_LNKCAP_MLW_X2	0x00000020 /* Maximum Link Width x2 */
> > +#define  PCI_EXP_LNKCAP_MLW_X4	0x00000040 /* Maximum Link Width x4 */
> > +#define  PCI_EXP_LNKCAP_MLW_X8	0x00000080 /* Maximum Link Width x8 */
> > +#define  PCI_EXP_LNKCAP_MLW_X12	0x000000c0 /* Maximum Link Width x12 */
> > +#define  PCI_EXP_LNKCAP_MLW_X16	0x00000100 /* Maximum Link Width x16 */
> >  #define  PCI_EXP_LNKCAP_MLW	0x000003f0 /* Maximum Link Width */
> >  #define  PCI_EXP_LNKCAP_ASPMS	0x00000c00 /* ASPM Support */
> >  #define  PCI_EXP_LNKCAP_ASPM_L0S 0x00000400 /* ASPM L0s Support */
> > --
> > 2.25.1
> >

Re: [PATCH v16 07/22] PCI: dwc: Add outbound MSG TLPs support

[Serge Semin]

On Wed, May 10, 2023 at 03:22:19PM +0900, Yoshihiro Shimoda wrote:
> Add "code" and "routing" into struct dw_pcie_outbound_atu for

structure name has been changed.

> sending MSG by iATU in the PCIe endpoint mode in near the future.
> PCIE_ATU_INHIBIT_PAYLOAD is set to issue TLP type of Msg instead of
> MsgD.

So your implementation implies the data-less messages only. This note
should have been added at least to the commit log. Ideally it would be
useful to have an in-situ comment above the code setting these flags.

> PCIE_ATU_HEADER_SUB_ENABLE is set to issue the translated
> TLP header by using PCIE_ATU_{LOW,UPP}ER_TARGET registers' values.

Why is that needed? Please elaborate in the patch log.

Other than that the change looks good.

* I'll get back to the series review tomorrow.

-Serge(y)

> 
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> ---
>  drivers/pci/controller/dwc/pcie-designware.c | 7 +++++--
>  drivers/pci/controller/dwc/pcie-designware.h | 5 +++++
>  2 files changed, 10 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
> index bd85a73d354b..a7c724ba7385 100644
> --- a/drivers/pci/controller/dwc/pcie-designware.c
> +++ b/drivers/pci/controller/dwc/pcie-designware.c
> @@ -498,7 +498,7 @@ int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
>  	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_TARGET,
>  			      upper_32_bits(atu->pci_addr));
>  
> -	val = atu->type | PCIE_ATU_FUNC_NUM(atu->func_no);
> +	val = atu->type | atu->routing | PCIE_ATU_FUNC_NUM(atu->func_no);
>  	if (upper_32_bits(limit_addr) > upper_32_bits(cpu_addr) &&
>  	    dw_pcie_ver_is_ge(pci, 460A))
>  		val |= PCIE_ATU_INCREASE_REGION_SIZE;
> @@ -506,7 +506,10 @@ int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
>  		val = dw_pcie_enable_ecrc(val);
>  	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL1, val);
>  
> -	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2, PCIE_ATU_ENABLE);
> +	val = PCIE_ATU_ENABLE;
> +	if (atu->type == PCIE_ATU_TYPE_MSG)
> +		val |= PCIE_ATU_INHIBIT_PAYLOAD | PCIE_ATU_HEADER_SUB_ENABLE | atu->code;
> +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2, val);
>  
>  	/*
>  	 * Make sure ATU enable takes effect before any subsequent config
> diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
> index b8fa099bbed3..c83d1d176e62 100644
> --- a/drivers/pci/controller/dwc/pcie-designware.h
> +++ b/drivers/pci/controller/dwc/pcie-designware.h
> @@ -150,11 +150,14 @@
>  #define PCIE_ATU_TYPE_IO		0x2
>  #define PCIE_ATU_TYPE_CFG0		0x4
>  #define PCIE_ATU_TYPE_CFG1		0x5
> +#define PCIE_ATU_TYPE_MSG		0x10
>  #define PCIE_ATU_TD			BIT(8)
>  #define PCIE_ATU_FUNC_NUM(pf)           ((pf) << 20)
>  #define PCIE_ATU_REGION_CTRL2		0x004
>  #define PCIE_ATU_ENABLE			BIT(31)
>  #define PCIE_ATU_BAR_MODE_ENABLE	BIT(30)
> +#define PCIE_ATU_INHIBIT_PAYLOAD	BIT(22)
> +#define PCIE_ATU_HEADER_SUB_ENABLE	BIT(21)
>  #define PCIE_ATU_FUNC_NUM_MATCH_EN      BIT(19)
>  #define PCIE_ATU_LOWER_BASE		0x008
>  #define PCIE_ATU_UPPER_BASE		0x00C
> @@ -295,6 +298,8 @@ struct dw_pcie_ob_atu_cfg {
>  	int index;
>  	int type;
>  	u8 func_no;
> +	u8 code;
> +	u8 routing;
>  	u64 cpu_addr;
>  	u64 pci_addr;
>  	u64 size;
> -- 
> 2.25.1
> 

Re: [PATCH v16 01/22] PCI: Add PCI_EXP_LNKCAP_MLW macros

[Serge Semin]

On Mon, Jun 05, 2023 at 12:14:26AM +0000, Yoshihiro Shimoda wrote:
> Hello Serge,
> 
> > From: Serge Semin, Sent: Monday, June 5, 2023 7:51 AM
> > 
> > On Wed, May 10, 2023 at 03:22:13PM +0900, Yoshihiro Shimoda wrote:
> > > Add macros defining Maximum Link Width bits in Link Capabilities
> > > Register.
> > >
> > > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > > Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> > 
> > You haven't been using these macros in the following up patches since
> > v9. Why do you keep submitting this change then? I would suggest to
> > drop the patch especially seeing the PCI_EXP_LNKCAP_MLW field directly
> > encodes the link width thus these macros unlikely will be of much use.
> 
> Thank you for your review! You're correct. I didn't realized that
> the macros were not used on the patch series..
> 
> However, I also realized that the patch 11/22 used the PCI_EXP_LNKSTA_NLW_SHIFT
> macro for the PCI_EXP_LNKCAP register. So, I'm thinking that I should modify
> this patch as adding PCI_EXP_LNKCAP_MLW_SHIFT instead. But, what do you think?
> 
> --- on the patch 11/22 ---
> > +	cap = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP);
> > +	val = dw_pcie_readl_dbi(pci, cap + PCI_EXP_LNKCAP);
> > +	val &= ~PCI_EXP_LNKCAP_MLW;
> > +	val |= num_lanes << PCI_EXP_LNKSTA_NLW_SHIFT;
> 
> Perhaps, this should be PCI_EXP_LNKCAP_MLW_SHIFT instead.

I'll get to that patch tomorrow, but in any case there is no need in
defining an additional macro here. There is a handy helper FIELD_PREP()
for that: FIELD_PREP(PCI_EXP_LNKSTA_MLW, num_lanes).

IMO for the same reason the PCI_EXP_LNKSTA_NLW_* macros are
unnecessary and can be easily either dropped or replaced by the
FIELD_{PREP,GET} macros usage.

-Serge(y)

> ---
> 
> Best regards,
> Yoshihiro Shimoda
> 
> > -Serge(y)
> > 
> > > ---
> > >  include/uapi/linux/pci_regs.h | 6 ++++++
> > >  1 file changed, 6 insertions(+)
> > >
> > > diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
> > > index dc2000e0fe3a..5d48413ac28f 100644
> > > --- a/include/uapi/linux/pci_regs.h
> > > +++ b/include/uapi/linux/pci_regs.h
> > > @@ -538,6 +538,12 @@
> > >  #define  PCI_EXP_LNKCAP_SLS_16_0GB 0x00000004 /* LNKCAP2 SLS Vector bit 3 */
> > >  #define  PCI_EXP_LNKCAP_SLS_32_0GB 0x00000005 /* LNKCAP2 SLS Vector bit 4 */
> > >  #define  PCI_EXP_LNKCAP_SLS_64_0GB 0x00000006 /* LNKCAP2 SLS Vector bit 5 */
> > > +#define  PCI_EXP_LNKCAP_MLW_X1	0x00000010 /* Maximum Link Width x1 */
> > > +#define  PCI_EXP_LNKCAP_MLW_X2	0x00000020 /* Maximum Link Width x2 */
> > > +#define  PCI_EXP_LNKCAP_MLW_X4	0x00000040 /* Maximum Link Width x4 */
> > > +#define  PCI_EXP_LNKCAP_MLW_X8	0x00000080 /* Maximum Link Width x8 */
> > > +#define  PCI_EXP_LNKCAP_MLW_X12	0x000000c0 /* Maximum Link Width x12 */
> > > +#define  PCI_EXP_LNKCAP_MLW_X16	0x00000100 /* Maximum Link Width x16 */
> > >  #define  PCI_EXP_LNKCAP_MLW	0x000003f0 /* Maximum Link Width */
> > >  #define  PCI_EXP_LNKCAP_ASPMS	0x00000c00 /* ASPM Support */
> > >  #define  PCI_EXP_LNKCAP_ASPM_L0S 0x00000400 /* ASPM L0s Support */
> > > --
> > > 2.25.1
> > >

RE: [PATCH v16 01/22] PCI: Add PCI_EXP_LNKCAP_MLW macros

[Yoshihiro Shimoda]

Hello Serge,

> From: Serge Semin, Sent: Monday, June 5, 2023 9:26 AM
> 
> On Mon, Jun 05, 2023 at 12:14:26AM +0000, Yoshihiro Shimoda wrote:
> > Hello Serge,
> >
> > > From: Serge Semin, Sent: Monday, June 5, 2023 7:51 AM
> > >
> > > On Wed, May 10, 2023 at 03:22:13PM +0900, Yoshihiro Shimoda wrote:
> > > > Add macros defining Maximum Link Width bits in Link Capabilities
> > > > Register.
> > > >
> > > > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > > > Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> > >
> > > You haven't been using these macros in the following up patches since
> > > v9. Why do you keep submitting this change then? I would suggest to
> > > drop the patch especially seeing the PCI_EXP_LNKCAP_MLW field directly
> > > encodes the link width thus these macros unlikely will be of much use.
> >
> > Thank you for your review! You're correct. I didn't realized that
> > the macros were not used on the patch series..
> >
> > However, I also realized that the patch 11/22 used the PCI_EXP_LNKSTA_NLW_SHIFT
> > macro for the PCI_EXP_LNKCAP register. So, I'm thinking that I should modify
> > this patch as adding PCI_EXP_LNKCAP_MLW_SHIFT instead. But, what do you think?
> >
> > --- on the patch 11/22 ---
> > > +	cap = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP);
> > > +	val = dw_pcie_readl_dbi(pci, cap + PCI_EXP_LNKCAP);
> > > +	val &= ~PCI_EXP_LNKCAP_MLW;
> > > +	val |= num_lanes << PCI_EXP_LNKSTA_NLW_SHIFT;
> >
> > Perhaps, this should be PCI_EXP_LNKCAP_MLW_SHIFT instead.
> 
> I'll get to that patch tomorrow, but in any case there is no need in
> defining an additional macro here. There is a handy helper FIELD_PREP()
> for that: FIELD_PREP(PCI_EXP_LNKSTA_MLW, num_lanes).

Thank you for the reply. TBH, I didn't know the helper FIELD_PREP(),
but now I understood it. Since the pci_regs.h has already had
the PCI_EXP_LNKCAP_MLW macro, we don't need to add any macro here.

> IMO for the same reason the PCI_EXP_LNKSTA_NLW_* macros are
> unnecessary and can be easily either dropped or replaced by the
> FIELD_{PREP,GET} macros usage.

I got it.

Best regards,
Yoshihiro Shimoda

> -Serge(y)
> 
> > ---
> >
> > Best regards,
> > Yoshihiro Shimoda
> >
> > > -Serge(y)
> > >
> > > > ---
> > > >  include/uapi/linux/pci_regs.h | 6 ++++++
> > > >  1 file changed, 6 insertions(+)
> > > >
> > > > diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
> > > > index dc2000e0fe3a..5d48413ac28f 100644
> > > > --- a/include/uapi/linux/pci_regs.h
> > > > +++ b/include/uapi/linux/pci_regs.h
> > > > @@ -538,6 +538,12 @@
> > > >  #define  PCI_EXP_LNKCAP_SLS_16_0GB 0x00000004 /* LNKCAP2 SLS Vector bit 3 */
> > > >  #define  PCI_EXP_LNKCAP_SLS_32_0GB 0x00000005 /* LNKCAP2 SLS Vector bit 4 */
> > > >  #define  PCI_EXP_LNKCAP_SLS_64_0GB 0x00000006 /* LNKCAP2 SLS Vector bit 5 */
> > > > +#define  PCI_EXP_LNKCAP_MLW_X1	0x00000010 /* Maximum Link Width x1 */
> > > > +#define  PCI_EXP_LNKCAP_MLW_X2	0x00000020 /* Maximum Link Width x2 */
> > > > +#define  PCI_EXP_LNKCAP_MLW_X4	0x00000040 /* Maximum Link Width x4 */
> > > > +#define  PCI_EXP_LNKCAP_MLW_X8	0x00000080 /* Maximum Link Width x8 */
> > > > +#define  PCI_EXP_LNKCAP_MLW_X12	0x000000c0 /* Maximum Link Width x12 */
> > > > +#define  PCI_EXP_LNKCAP_MLW_X16	0x00000100 /* Maximum Link Width x16 */
> > > >  #define  PCI_EXP_LNKCAP_MLW	0x000003f0 /* Maximum Link Width */
> > > >  #define  PCI_EXP_LNKCAP_ASPMS	0x00000c00 /* ASPM Support */
> > > >  #define  PCI_EXP_LNKCAP_ASPM_L0S 0x00000400 /* ASPM L0s Support */
> > > > --
> > > > 2.25.1
> > > >

RE: [PATCH v16 03/22] PCI: Add INTx Mechanism Messages macros

[Yoshihiro Shimoda]

Hello Serge,

> From: Serge Semin, Sent: Monday, June 5, 2023 8:08 AM
> 
> On Wed, May 10, 2023 at 03:22:15PM +0900, Yoshihiro Shimoda wrote:
> > Add "Message Routing" and "INTx Mechanism Messages" macros to send
> > a message by a PCIe driver.
> >
> > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > ---
> >  drivers/pci/pci.h | 18 ++++++++++++++++++
> >  1 file changed, 18 insertions(+)
> >
> > diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> > index 2475098f6518..67badc40e90b 100644
> > --- a/drivers/pci/pci.h
> > +++ b/drivers/pci/pci.h
> > @@ -11,6 +11,24 @@
> >
> >  #define PCI_VSEC_ID_INTEL_TBT	0x1234	/* Thunderbolt */
> >
> 
> > +/* Message Routing */
> 
> Call it "Implicit Message Routing (r[2:0])" as per the specification.

I got it.

> > +#define PCI_MSG_ROUTING_RC	0
> > +#define PCI_MSG_ROUTING_ADDR	1
> > +#define PCI_MSG_ROUTING_ID	2
> > +#define PCI_MSG_ROUTING_BC	3
> > +#define PCI_MSG_ROUTING_LOCAL	4
> > +#define PCI_MSG_ROUTING_GATHER	5
> 
> IMO prefix like this PCI_MSG_TYPE_R_{RC,ADDR,ID,BC,...} would be a bit
> better since it would indicate that this routing flags are a sub-field of
> the Message Type field. Bjorn?

I got it. I'll rename them.

> > +
> > +/* INTx Mechanism Messages */
> > +#define PCI_CODE_ASSERT_INTA	0x20
> > +#define PCI_CODE_ASSERT_INTB	0x21
> > +#define PCI_CODE_ASSERT_INTC	0x22
> > +#define PCI_CODE_ASSERT_INTD	0x23
> > +#define PCI_CODE_DEASSERT_INTA	0x24
> > +#define PCI_CODE_DEASSERT_INTB	0x25
> > +#define PCI_CODE_DEASSERT_INTC	0x26
> > +#define PCI_CODE_DEASSERT_INTD	0x27
> 
> IMO Prefix PCI_MSG_CODE_... would be a bit more descriptive since per
> the specification the respective message field is called "Message
> Code" and not just "Code". Bjorn?

I got it. I'll rename them.

Best regards,
Yoshihiro Shimoda

> -Serge(y)
> 
> > +
> >  extern const unsigned char pcie_link_speed[];
> >  extern bool pci_early_dump;
> >
> > --
> > 2.25.1
> >

RE: [PATCH v16 04/22] PCI: Rename PCI_EPC_IRQ_LEGACY to PCI_EPC_IRQ_INTX

[Yoshihiro Shimoda]

Hello Serge,

> From: Serge Semin, Sent: Monday, June 5, 2023 8:23 AM
> 
> On Wed, May 10, 2023 at 03:22:16PM +0900, Yoshihiro Shimoda wrote:
> > Using "INTx" instead of "legacy" is more specific. So, rename
> > PCI_EPC_IRQ_LEGACY to PCI_EPC_IRQ_INTX.
> >
> > Suggested-by: Bjorn Helgaas <helgaas@kernel.org>
> > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> 
> > Cc: Tom Joseph <tjoseph@cadence.com>
> > Cc: Vignesh Raghavendra <vigneshr@ti.com>
> > Cc: Richard Zhu <hongxing.zhu@nxp.com>
> > Cc: Lucas Stach <l.stach@pengutronix.de>
> > Cc: Shawn Guo <shawnguo@kernel.org>
> > Cc: Sascha Hauer <s.hauer@pengutronix.de>
> > Cc: Pengutronix Kernel Team <kernel@pengutronix.de>
> > Cc: Fabio Estevam <festevam@gmail.com>
> > Cc: NXP Linux Team <linux-imx@nxp.com>
> > Cc: Minghuan Lian <minghuan.Lian@nxp.com>
> > Cc: Mingkai Hu <mingkai.hu@nxp.com>
> > Cc: Roy Zang <roy.zang@nxp.com>
> > Cc: Jingoo Han <jingoohan1@gmail.com>
> > Cc: Srikanth Thokala <srikanth.thokala@intel.com>
> > Cc: Thierry Reding <thierry.reding@gmail.com>
> > Cc: Jonathan Hunter <jonathanh@nvidia.com>
> > Cc: Kunihiko Hayashi <hayashi.kunihiko@socionext.com>
> > Cc: Masami Hiramatsu <mhiramat@kernel.org>
> > Cc: Marek Vasut <marek.vasut+renesas@gmail.com>
> > Cc: Shawn Lin <shawn.lin@rock-chips.com>
> > Cc: Heiko Stuebner <heiko@sntech.de>
> > Cc: Kishon Vijay Abraham I <kishon@kernel.org>
> 
> I doubt that that long Cc-tags list of much use in this case
> especially seeing the change is mainly relevant to the PCIe
> subsystem core. In order to still let git send-email to create a
> pre-defined Cc-list I normally move all the Cc'es to be below the
> "---" line. Thus the specified developers will be added by the
> send-email tool to the list of recipients but git am will ignore
> everything below "---" so the mainline commit won't have these
> Cc-tags.

Thank you for your suggestion. I didn't know such a method.
I'll do that on v17.

Best regards,
Yoshihiro Shimoda

> -Serge(y)
> 
> > Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
> > Acked-by: Jesper Nilsson <jesper.nilsson@axis.com> # ARTPEC
> > Reviewed-by: Serge Semin <fancer.lancer@gmail.com>
> > ---
> >  drivers/pci/controller/cadence/pcie-cadence-ep.c  |  2 +-
> >  drivers/pci/controller/dwc/pci-dra7xx.c           |  2 +-
> >  drivers/pci/controller/dwc/pci-imx6.c             |  2 +-
> >  drivers/pci/controller/dwc/pci-keystone.c         |  2 +-
> >  drivers/pci/controller/dwc/pci-layerscape-ep.c    |  2 +-
> >  drivers/pci/controller/dwc/pcie-artpec6.c         |  2 +-
> >  drivers/pci/controller/dwc/pcie-designware-plat.c |  2 +-
> >  drivers/pci/controller/dwc/pcie-keembay.c         |  2 +-
> >  drivers/pci/controller/dwc/pcie-qcom-ep.c         |  2 +-
> >  drivers/pci/controller/dwc/pcie-tegra194.c        |  2 +-
> >  drivers/pci/controller/dwc/pcie-uniphier-ep.c     |  2 +-
> >  drivers/pci/controller/pcie-rcar-ep.c             |  2 +-
> >  drivers/pci/controller/pcie-rockchip-ep.c         |  2 +-
> >  drivers/pci/endpoint/functions/pci-epf-test.c     | 10 +++++-----
> >  include/linux/pci-epc.h                           |  4 ++--
> >  15 files changed, 20 insertions(+), 20 deletions(-)
> >
> > diff --git a/drivers/pci/controller/cadence/pcie-cadence-ep.c b/drivers/pci/controller/cadence/pcie-cadence-ep.c
> > index b8b655d4047e..2af8eb4e6d91 100644
> > --- a/drivers/pci/controller/cadence/pcie-cadence-ep.c
> > +++ b/drivers/pci/controller/cadence/pcie-cadence-ep.c
> > @@ -539,7 +539,7 @@ static int cdns_pcie_ep_raise_irq(struct pci_epc *epc, u8 fn, u8 vfn,
> >  	struct device *dev = pcie->dev;
> >
> >  	switch (type) {
> > -	case PCI_EPC_IRQ_LEGACY:
> > +	case PCI_EPC_IRQ_INTX:
> >  		if (vfn > 0) {
> >  			dev_err(dev, "Cannot raise legacy interrupts for VF\n");
> >  			return -EINVAL;
> > diff --git a/drivers/pci/controller/dwc/pci-dra7xx.c b/drivers/pci/controller/dwc/pci-dra7xx.c
> > index 4ae807e7cf79..b42fb1cc8bc8 100644
> > --- a/drivers/pci/controller/dwc/pci-dra7xx.c
> > +++ b/drivers/pci/controller/dwc/pci-dra7xx.c
> > @@ -410,7 +410,7 @@ static int dra7xx_pcie_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> >  	struct dra7xx_pcie *dra7xx = to_dra7xx_pcie(pci);
> >
> >  	switch (type) {
> > -	case PCI_EPC_IRQ_LEGACY:
> > +	case PCI_EPC_IRQ_INTX:
> >  		dra7xx_pcie_raise_legacy_irq(dra7xx);
> >  		break;
> >  	case PCI_EPC_IRQ_MSI:
> > diff --git a/drivers/pci/controller/dwc/pci-imx6.c b/drivers/pci/controller/dwc/pci-imx6.c
> > index 52906f999f2b..1f39e733ce19 100644
> > --- a/drivers/pci/controller/dwc/pci-imx6.c
> > +++ b/drivers/pci/controller/dwc/pci-imx6.c
> > @@ -1062,7 +1062,7 @@ static int imx6_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >
> >  	switch (type) {
> > -	case PCI_EPC_IRQ_LEGACY:
> > +	case PCI_EPC_IRQ_INTX:
> >  		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
> >  	case PCI_EPC_IRQ_MSI:
> >  		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
> > diff --git a/drivers/pci/controller/dwc/pci-keystone.c b/drivers/pci/controller/dwc/pci-keystone.c
> > index 78818853af9e..3806f5530937 100644
> > --- a/drivers/pci/controller/dwc/pci-keystone.c
> > +++ b/drivers/pci/controller/dwc/pci-keystone.c
> > @@ -908,7 +908,7 @@ static int ks_pcie_am654_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> >  	struct keystone_pcie *ks_pcie = to_keystone_pcie(pci);
> >
> >  	switch (type) {
> > -	case PCI_EPC_IRQ_LEGACY:
> > +	case PCI_EPC_IRQ_INTX:
> >  		ks_pcie_am654_raise_legacy_irq(ks_pcie);
> >  		break;
> >  	case PCI_EPC_IRQ_MSI:
> > diff --git a/drivers/pci/controller/dwc/pci-layerscape-ep.c b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > index c640db60edc6..ab3306e206d8 100644
> > --- a/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > +++ b/drivers/pci/controller/dwc/pci-layerscape-ep.c
> > @@ -65,7 +65,7 @@ static int ls_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >
> >  	switch (type) {
> > -	case PCI_EPC_IRQ_LEGACY:
> > +	case PCI_EPC_IRQ_INTX:
> >  		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
> >  	case PCI_EPC_IRQ_MSI:
> >  		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
> > diff --git a/drivers/pci/controller/dwc/pcie-artpec6.c b/drivers/pci/controller/dwc/pcie-artpec6.c
> > index 98102079e26d..128cb1118e3a 100644
> > --- a/drivers/pci/controller/dwc/pcie-artpec6.c
> > +++ b/drivers/pci/controller/dwc/pcie-artpec6.c
> > @@ -357,7 +357,7 @@ static int artpec6_pcie_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >
> >  	switch (type) {
> > -	case PCI_EPC_IRQ_LEGACY:
> > +	case PCI_EPC_IRQ_INTX:
> >  		dev_err(pci->dev, "EP cannot trigger legacy IRQs\n");
> >  		return -EINVAL;
> >  	case PCI_EPC_IRQ_MSI:
> > diff --git a/drivers/pci/controller/dwc/pcie-designware-plat.c b/drivers/pci/controller/dwc/pcie-designware-plat.c
> > index 1fcfb840f238..fc3b02949218 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware-plat.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware-plat.c
> > @@ -48,7 +48,7 @@ static int dw_plat_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >
> >  	switch (type) {
> > -	case PCI_EPC_IRQ_LEGACY:
> > +	case PCI_EPC_IRQ_INTX:
> >  		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
> >  	case PCI_EPC_IRQ_MSI:
> >  		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
> > diff --git a/drivers/pci/controller/dwc/pcie-keembay.c b/drivers/pci/controller/dwc/pcie-keembay.c
> > index f90f36bac018..ceb940b327cb 100644
> > --- a/drivers/pci/controller/dwc/pcie-keembay.c
> > +++ b/drivers/pci/controller/dwc/pcie-keembay.c
> > @@ -290,7 +290,7 @@ static int keembay_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >
> >  	switch (type) {
> > -	case PCI_EPC_IRQ_LEGACY:
> > +	case PCI_EPC_IRQ_INTX:
> >  		/* Legacy interrupts are not supported in Keem Bay */
> >  		dev_err(pci->dev, "Legacy IRQ is not supported\n");
> >  		return -EINVAL;
> > diff --git a/drivers/pci/controller/dwc/pcie-qcom-ep.c b/drivers/pci/controller/dwc/pcie-qcom-ep.c
> > index 19b32839ea26..077afce48d0b 100644
> > --- a/drivers/pci/controller/dwc/pcie-qcom-ep.c
> > +++ b/drivers/pci/controller/dwc/pcie-qcom-ep.c
> > @@ -658,7 +658,7 @@ static int qcom_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >
> >  	switch (type) {
> > -	case PCI_EPC_IRQ_LEGACY:
> > +	case PCI_EPC_IRQ_INTX:
> >  		return dw_pcie_ep_raise_legacy_irq(ep, func_no);
> >  	case PCI_EPC_IRQ_MSI:
> >  		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
> > diff --git a/drivers/pci/controller/dwc/pcie-tegra194.c b/drivers/pci/controller/dwc/pcie-tegra194.c
> > index 09825b4a075e..4adba379b83d 100644
> > --- a/drivers/pci/controller/dwc/pcie-tegra194.c
> > +++ b/drivers/pci/controller/dwc/pcie-tegra194.c
> > @@ -1980,7 +1980,7 @@ static int tegra_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> >  	struct tegra_pcie_dw *pcie = to_tegra_pcie(pci);
> >
> >  	switch (type) {
> > -	case PCI_EPC_IRQ_LEGACY:
> > +	case PCI_EPC_IRQ_INTX:
> >  		return tegra_pcie_ep_raise_legacy_irq(pcie, interrupt_num);
> >
> >  	case PCI_EPC_IRQ_MSI:
> > diff --git a/drivers/pci/controller/dwc/pcie-uniphier-ep.c b/drivers/pci/controller/dwc/pcie-uniphier-ep.c
> > index 4d0a587c0ba5..7787eedf87f4 100644
> > --- a/drivers/pci/controller/dwc/pcie-uniphier-ep.c
> > +++ b/drivers/pci/controller/dwc/pcie-uniphier-ep.c
> > @@ -262,7 +262,7 @@ static int uniphier_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >
> >  	switch (type) {
> > -	case PCI_EPC_IRQ_LEGACY:
> > +	case PCI_EPC_IRQ_INTX:
> >  		return uniphier_pcie_ep_raise_legacy_irq(ep);
> >  	case PCI_EPC_IRQ_MSI:
> >  		return uniphier_pcie_ep_raise_msi_irq(ep, func_no,
> > diff --git a/drivers/pci/controller/pcie-rcar-ep.c b/drivers/pci/controller/pcie-rcar-ep.c
> > index f9682df1da61..fbdf3d85301c 100644
> > --- a/drivers/pci/controller/pcie-rcar-ep.c
> > +++ b/drivers/pci/controller/pcie-rcar-ep.c
> > @@ -408,7 +408,7 @@ static int rcar_pcie_ep_raise_irq(struct pci_epc *epc, u8 fn, u8 vfn,
> >  	struct rcar_pcie_endpoint *ep = epc_get_drvdata(epc);
> >
> >  	switch (type) {
> > -	case PCI_EPC_IRQ_LEGACY:
> > +	case PCI_EPC_IRQ_INTX:
> >  		return rcar_pcie_ep_assert_intx(ep, fn, 0);
> >
> >  	case PCI_EPC_IRQ_MSI:
> > diff --git a/drivers/pci/controller/pcie-rockchip-ep.c b/drivers/pci/controller/pcie-rockchip-ep.c
> > index d1a200b93b2b..ef9d1f6c382a 100644
> > --- a/drivers/pci/controller/pcie-rockchip-ep.c
> > +++ b/drivers/pci/controller/pcie-rockchip-ep.c
> > @@ -477,7 +477,7 @@ static int rockchip_pcie_ep_raise_irq(struct pci_epc *epc, u8 fn, u8 vfn,
> >  	struct rockchip_pcie_ep *ep = epc_get_drvdata(epc);
> >
> >  	switch (type) {
> > -	case PCI_EPC_IRQ_LEGACY:
> > +	case PCI_EPC_IRQ_INTX:
> >  		return rockchip_pcie_ep_send_legacy_irq(ep, fn, 0);
> >  	case PCI_EPC_IRQ_MSI:
> >  		return rockchip_pcie_ep_send_msi_irq(ep, fn, interrupt_num);
> > diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c b/drivers/pci/endpoint/functions/pci-epf-test.c
> > index 623b08caa998..6beb3f2b0afb 100644
> > --- a/drivers/pci/endpoint/functions/pci-epf-test.c
> > +++ b/drivers/pci/endpoint/functions/pci-epf-test.c
> > @@ -19,11 +19,11 @@
> >  #include <linux/pci-epf.h>
> >  #include <linux/pci_regs.h>
> >
> > -#define IRQ_TYPE_LEGACY			0
> > +#define IRQ_TYPE_INTX			0
> >  #define IRQ_TYPE_MSI			1
> >  #define IRQ_TYPE_MSIX			2
> >
> > -#define COMMAND_RAISE_LEGACY_IRQ	BIT(0)
> > +#define COMMAND_RAISE_INTX_IRQ		BIT(0)
> >  #define COMMAND_RAISE_MSI_IRQ		BIT(1)
> >  #define COMMAND_RAISE_MSIX_IRQ		BIT(2)
> >  #define COMMAND_READ			BIT(3)
> > @@ -600,9 +600,9 @@ static void pci_epf_test_raise_irq(struct pci_epf_test *epf_test,
> >  	WRITE_ONCE(reg->status, status);
> >
> >  	switch (reg->irq_type) {
> > -	case IRQ_TYPE_LEGACY:
> > +	case IRQ_TYPE_INTX:
> >  		pci_epc_raise_irq(epc, epf->func_no, epf->vfunc_no,
> > -				  PCI_EPC_IRQ_LEGACY, 0);
> > +				  PCI_EPC_IRQ_INTX, 0);
> >  		break;
> >  	case IRQ_TYPE_MSI:
> >  		count = pci_epc_get_msi(epc, epf->func_no, epf->vfunc_no);
> > @@ -659,7 +659,7 @@ static void pci_epf_test_cmd_handler(struct work_struct *work)
> >  	}
> >
> >  	switch (command) {
> > -	case COMMAND_RAISE_LEGACY_IRQ:
> > +	case COMMAND_RAISE_INTX_IRQ:
> >  	case COMMAND_RAISE_MSI_IRQ:
> >  	case COMMAND_RAISE_MSIX_IRQ:
> >  		pci_epf_test_raise_irq(epf_test, reg);
> > diff --git a/include/linux/pci-epc.h b/include/linux/pci-epc.h
> > index 301bb0e53707..c2572a93d73d 100644
> > --- a/include/linux/pci-epc.h
> > +++ b/include/linux/pci-epc.h
> > @@ -21,7 +21,7 @@ enum pci_epc_interface_type {
> >
> >  enum pci_epc_irq_type {
> >  	PCI_EPC_IRQ_UNKNOWN,
> > -	PCI_EPC_IRQ_LEGACY,
> > +	PCI_EPC_IRQ_INTX,
> >  	PCI_EPC_IRQ_MSI,
> >  	PCI_EPC_IRQ_MSIX,
> >  };
> > @@ -54,7 +54,7 @@ pci_epc_interface_string(enum pci_epc_interface_type type)
> >   *	     MSI-X capability register
> >   * @get_msix: ops to get the number of MSI-X interrupts allocated by the RC
> >   *	     from the MSI-X capability register
> > - * @raise_irq: ops to raise a legacy, MSI or MSI-X interrupt
> > + * @raise_irq: ops to raise an INTx, MSI or MSI-X interrupt
> >   * @map_msi_irq: ops to map physical address to MSI address and return MSI data
> >   * @start: ops to start the PCI link
> >   * @stop: ops to stop the PCI link
> > --
> > 2.25.1
> >

Re: [PATCH v16 03/22] PCI: Add INTx Mechanism Messages macros

[Serge Semin]

On Mon, Jun 05, 2023 at 02:10:14AM +0000, Yoshihiro Shimoda wrote:
> Hello Serge,
> 
> > From: Serge Semin, Sent: Monday, June 5, 2023 8:08 AM
> > 
> > On Wed, May 10, 2023 at 03:22:15PM +0900, Yoshihiro Shimoda wrote:
> > > Add "Message Routing" and "INTx Mechanism Messages" macros to send
> > > a message by a PCIe driver.
> > >
> > > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > > ---
> > >  drivers/pci/pci.h | 18 ++++++++++++++++++
> > >  1 file changed, 18 insertions(+)
> > >
> > > diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> > > index 2475098f6518..67badc40e90b 100644
> > > --- a/drivers/pci/pci.h
> > > +++ b/drivers/pci/pci.h
> > > @@ -11,6 +11,24 @@
> > >
> > >  #define PCI_VSEC_ID_INTEL_TBT	0x1234	/* Thunderbolt */
> > >
> > 
> > > +/* Message Routing */
> > 
> > Call it "Implicit Message Routing (r[2:0])" as per the specification.
> 
> I got it.
> 
> > > +#define PCI_MSG_ROUTING_RC	0
> > > +#define PCI_MSG_ROUTING_ADDR	1
> > > +#define PCI_MSG_ROUTING_ID	2
> > > +#define PCI_MSG_ROUTING_BC	3
> > > +#define PCI_MSG_ROUTING_LOCAL	4
> > > +#define PCI_MSG_ROUTING_GATHER	5
> > 
> > IMO prefix like this PCI_MSG_TYPE_R_{RC,ADDR,ID,BC,...} would be a bit
> > better since it would indicate that this routing flags are a sub-field of
> > the Message Type field. Bjorn?
> 
> I got it. I'll rename them.
> 
> > > +
> > > +/* INTx Mechanism Messages */
> > > +#define PCI_CODE_ASSERT_INTA	0x20
> > > +#define PCI_CODE_ASSERT_INTB	0x21
> > > +#define PCI_CODE_ASSERT_INTC	0x22
> > > +#define PCI_CODE_ASSERT_INTD	0x23
> > > +#define PCI_CODE_DEASSERT_INTA	0x24
> > > +#define PCI_CODE_DEASSERT_INTB	0x25
> > > +#define PCI_CODE_DEASSERT_INTC	0x26
> > > +#define PCI_CODE_DEASSERT_INTD	0x27
> > 
> > IMO Prefix PCI_MSG_CODE_... would be a bit more descriptive since per
> > the specification the respective message field is called "Message
> > Code" and not just "Code". Bjorn?
> 
> I got it. I'll rename them.

I would suggest to wait for Bjorn' opinion about this for sometime
since the macros will be defined in the PCIe common header file.

-Serge(y)

> 
> Best regards,
> Yoshihiro Shimoda
> 
> > -Serge(y)
> > 
> > > +
> > >  extern const unsigned char pcie_link_speed[];
> > >  extern bool pci_early_dump;
> > >
> > > --
> > > 2.25.1
> > >

RE: [PATCH v16 03/22] PCI: Add INTx Mechanism Messages macros

[Yoshihiro Shimoda]

Hello Serge,

> From: Serge Semin, Sent: Monday, June 5, 2023 4:25 PM
> 
> On Mon, Jun 05, 2023 at 02:10:14AM +0000, Yoshihiro Shimoda wrote:
> > Hello Serge,
> >
> > > From: Serge Semin, Sent: Monday, June 5, 2023 8:08 AM
> > >
> > > On Wed, May 10, 2023 at 03:22:15PM +0900, Yoshihiro Shimoda wrote:
> > > > Add "Message Routing" and "INTx Mechanism Messages" macros to send
> > > > a message by a PCIe driver.
> > > >
> > > > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > > > ---
> > > >  drivers/pci/pci.h | 18 ++++++++++++++++++
> > > >  1 file changed, 18 insertions(+)
> > > >
> > > > diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> > > > index 2475098f6518..67badc40e90b 100644
> > > > --- a/drivers/pci/pci.h
> > > > +++ b/drivers/pci/pci.h
> > > > @@ -11,6 +11,24 @@
> > > >
> > > >  #define PCI_VSEC_ID_INTEL_TBT	0x1234	/* Thunderbolt */
> > > >
> > >
> > > > +/* Message Routing */
> > >
> > > Call it "Implicit Message Routing (r[2:0])" as per the specification.
> >
> > I got it.
> >
> > > > +#define PCI_MSG_ROUTING_RC	0
> > > > +#define PCI_MSG_ROUTING_ADDR	1
> > > > +#define PCI_MSG_ROUTING_ID	2
> > > > +#define PCI_MSG_ROUTING_BC	3
> > > > +#define PCI_MSG_ROUTING_LOCAL	4
> > > > +#define PCI_MSG_ROUTING_GATHER	5
> > >
> > > IMO prefix like this PCI_MSG_TYPE_R_{RC,ADDR,ID,BC,...} would be a bit
> > > better since it would indicate that this routing flags are a sub-field of
> > > the Message Type field. Bjorn?
> >
> > I got it. I'll rename them.
> >
> > > > +
> > > > +/* INTx Mechanism Messages */
> > > > +#define PCI_CODE_ASSERT_INTA	0x20
> > > > +#define PCI_CODE_ASSERT_INTB	0x21
> > > > +#define PCI_CODE_ASSERT_INTC	0x22
> > > > +#define PCI_CODE_ASSERT_INTD	0x23
> > > > +#define PCI_CODE_DEASSERT_INTA	0x24
> > > > +#define PCI_CODE_DEASSERT_INTB	0x25
> > > > +#define PCI_CODE_DEASSERT_INTC	0x26
> > > > +#define PCI_CODE_DEASSERT_INTD	0x27
> > >
> > > IMO Prefix PCI_MSG_CODE_... would be a bit more descriptive since per
> > > the specification the respective message field is called "Message
> > > Code" and not just "Code". Bjorn?
> >
> > I got it. I'll rename them.
> 
> I would suggest to wait for Bjorn' opinion about this for sometime
> since the macros will be defined in the PCIe common header file.

I got it. I will also wait for Bjorn's opinion.

Best regards,
Yoshihiro Shimoda

> -Serge(y)
> 
> >
> > Best regards,
> > Yoshihiro Shimoda
> >
> > > -Serge(y)
> > >
> > > > +
> > > >  extern const unsigned char pcie_link_speed[];
> > > >  extern bool pci_early_dump;
> > > >
> > > > --
> > > > 2.25.1
> > > >

Re: [PATCH v16 08/22] PCI: designware-ep: Add INTx IRQs support

[Serge Semin]

On Wed, May 10, 2023 at 03:22:20PM +0900, Yoshihiro Shimoda wrote:
> Add support for triggering INTx IRQs by using outbound iATU.
> Outbound iATU is utilized to send assert and de-assert INTx TLPs.
> The message is generated based on the payloadless Msg TLP with type
> 0x14, where 0x4 is the routing code implying the Terminate at
> Receiver message. The message code is specified as b1000xx for
> the INTx assertion and b1001xx for the INTx de-assertion.
> 
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> ---
>  .../pci/controller/dwc/pcie-designware-ep.c   | 66 +++++++++++++++++--
>  drivers/pci/controller/dwc/pcie-designware.h  |  2 +
>  2 files changed, 64 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
> index fe2e0d765be9..0abc0073b1cf 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> @@ -6,9 +6,11 @@
>   * Author: Kishon Vijay Abraham I <kishon@ti.com>
>   */
>  
> +#include <linux/delay.h>
>  #include <linux/of.h>
>  #include <linux/platform_device.h>
>  
> +#include "../../pci.h"
>  #include "pcie-designware.h"
>  #include <linux/pci-epc.h>
>  #include <linux/pci-epf.h>
> @@ -484,14 +486,57 @@ static const struct pci_epc_ops epc_ops = {
>  	.get_features		= dw_pcie_ep_get_features,
>  };
>  
> +static int dw_pcie_ep_send_msg(struct dw_pcie_ep *ep, u8 func_no, u8 code,
> +			       u8 routing)
> +{
> +	struct dw_pcie_ob_atu_cfg atu = { 0 };
> +	struct pci_epc *epc = ep->epc;
> +	int ret;
> +
> +	atu.func_no = func_no;
> +	atu.code = code;
> +	atu.routing = routing;
> +	atu.type = PCIE_ATU_TYPE_MSG;
> +	atu.cpu_addr = ep->intx_mem_phys;
> +	atu.size = epc->mem->window.page_size;
> +
> +	ret = dw_pcie_ep_outbound_atu(ep, &atu);
> +	if (ret)
> +		return ret;
> +
> +	writel(0, ep->intx_mem);
> +
> +	dw_pcie_ep_unmap_addr(epc, func_no, 0, ep->intx_mem_phys);
> +
> +	return 0;
> +}
> +
>  int dw_pcie_ep_raise_intx_irq(struct dw_pcie_ep *ep, u8 func_no)
>  {
>  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
>  	struct device *dev = pci->dev;
> +	int ret;
>  
> -	dev_err(dev, "EP cannot trigger INTx IRQs\n");
> +	if (!ep->intx_mem) {
> +		dev_err(dev, "INTx not supported\n");
> +		return -EOPNOTSUPP;
> +	}
>  
> -	return -EINVAL;
> +	/*

> +	 * According to the document of PCIe, the INTx emulation should be
> +	 * level-triggered. However, the Linux PCIe Endpoint framework only
> +	 * supports edge-triggered for now. So, this function asserts INTx for
> +	 * 50 usec, and then deasserts it.

"Document of PCIe"?)
What about the next message:
Even though the PCI bus specification implies the level-triggered INTx
interrupts the kernel PCIe endpoint framework has a single
PCI_EPC_IRQ_INTx flag defined for the legacy IRQs simulation. Thus
this function sends the Deassert_INTx PCIe TLP after the Assert_INTx
message with the 50 usec duration basically implementing the
rising-edge triggering IRQ. Hopefully the interrupt controller will
still be able to register the incoming IRQ event...

Other than that the change looks good.

-Serge(y)

> +	 */
> +	ret = dw_pcie_ep_send_msg(ep, func_no, PCI_CODE_ASSERT_INTA,
> +				  PCI_MSG_ROUTING_LOCAL);
> +	if (ret)
> +		return ret;
> +
> +	usleep_range(50, 100);
> +
> +	return dw_pcie_ep_send_msg(ep, func_no, PCI_CODE_DEASSERT_INTA,
> +				   PCI_MSG_ROUTING_LOCAL);
>  }
>  EXPORT_SYMBOL_GPL(dw_pcie_ep_raise_intx_irq);
>  
> @@ -622,6 +667,10 @@ void dw_pcie_ep_exit(struct dw_pcie_ep *ep)
>  
>  	dw_pcie_edma_remove(pci);
>  
> +	if (ep->intx_mem)
> +		pci_epc_mem_free_addr(epc, ep->intx_mem_phys, ep->intx_mem,
> +				      epc->mem->window.page_size);
> +
>  	pci_epc_mem_free_addr(epc, ep->msi_mem_phys, ep->msi_mem,
>  			      epc->mem->window.page_size);
>  
> @@ -793,9 +842,14 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
>  		goto err_exit_epc_mem;
>  	}
>  
> +	ep->intx_mem = pci_epc_mem_alloc_addr(epc, &ep->intx_mem_phys,
> +					      epc->mem->window.page_size);
> +	if (!ep->intx_mem)
> +		dev_warn(dev, "Failed to reserve memory for INTx\n");
> +
>  	ret = dw_pcie_edma_detect(pci);
>  	if (ret)
> -		goto err_free_epc_mem;
> +		goto err_free_epc_mem_intx;
>  
>  	if (ep->ops->get_features) {
>  		epc_features = ep->ops->get_features(ep);
> @@ -812,7 +866,11 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
>  err_remove_edma:
>  	dw_pcie_edma_remove(pci);
>  
> -err_free_epc_mem:
> +err_free_epc_mem_intx:
> +	if (ep->intx_mem)
> +		pci_epc_mem_free_addr(epc, ep->intx_mem_phys, ep->intx_mem,
> +				      epc->mem->window.page_size);
> +
>  	pci_epc_mem_free_addr(epc, ep->msi_mem_phys, ep->msi_mem,
>  			      epc->mem->window.page_size);
>  
> diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
> index c83d1d176e62..06e044e2163a 100644
> --- a/drivers/pci/controller/dwc/pcie-designware.h
> +++ b/drivers/pci/controller/dwc/pcie-designware.h
> @@ -369,6 +369,8 @@ struct dw_pcie_ep {
>  	unsigned long		*ob_window_map;
>  	void __iomem		*msi_mem;
>  	phys_addr_t		msi_mem_phys;
> +	void __iomem		*intx_mem;
> +	phys_addr_t		intx_mem_phys;
>  	struct pci_epf_bar	*epf_bar[PCI_STD_NUM_BARS];
>  };
>  
> -- 
> 2.25.1
> 

Re: [PATCH v16 10/22] PCI: dwc: Modify PCIE_PORT_LINK_CONTROL handling

[Serge Semin]

On Wed, May 10, 2023 at 03:22:22PM +0900, Yoshihiro Shimoda wrote:
> To improve code readability, modify PCIE_PORT_LINK_CONTROL handling.

So basically you are doing the same update as in the Patch 9:
detaching the already implemented link width setups into a separate
method. Why do you split them up into the incremental updates? Just
squash this patch into the patch 9. The resultant patch would be an
atomic update and a preparation before adding the PCI_EXP_LNKCAP field
update. The later would lead to the fully coherent maximum link width
setup method in accordance with the DW PCIe hardware manual.

-Serge(y)

> 
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> ---
>  drivers/pci/controller/dwc/pcie-designware.c | 40 +++++++-------------
>  1 file changed, 13 insertions(+), 27 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
> index 68aefbbcd68c..5dc423dd2f21 100644
> --- a/drivers/pci/controller/dwc/pcie-designware.c
> +++ b/drivers/pci/controller/dwc/pcie-designware.c
> @@ -721,28 +721,40 @@ static void dw_pcie_link_set_max_speed(struct dw_pcie *pci, u32 link_gen)
>  
>  static void dw_pcie_link_set_max_link_width(struct dw_pcie *pci, u32 num_lanes)
>  {
> -	u32 lwsc;
> +	u32 lwsc, plc;
>  
>  	if (!num_lanes)
>  		return;
>  
> +	/* Set the number of lanes */
> +	plc = dw_pcie_readl_dbi(pci, PCIE_PORT_LINK_CONTROL);
> +	plc &= ~PORT_LINK_MODE_MASK;
> +
>  	/* Set link width speed control register */
>  	lwsc = dw_pcie_readl_dbi(pci, PCIE_LINK_WIDTH_SPEED_CONTROL);
>  	lwsc &= ~PORT_LOGIC_LINK_WIDTH_MASK;
>  	switch (num_lanes) {
>  	case 1:
> +		plc |= PORT_LINK_MODE_1_LANES;
>  		lwsc |= PORT_LOGIC_LINK_WIDTH_1_LANES;
>  		break;
>  	case 2:
> +		plc |= PORT_LINK_MODE_2_LANES;
>  		lwsc |= PORT_LOGIC_LINK_WIDTH_2_LANES;
>  		break;
>  	case 4:
> +		plc |= PORT_LINK_MODE_4_LANES;
>  		lwsc |= PORT_LOGIC_LINK_WIDTH_4_LANES;
>  		break;
>  	case 8:
> +		plc |= PORT_LINK_MODE_8_LANES;
>  		lwsc |= PORT_LOGIC_LINK_WIDTH_8_LANES;
>  		break;
> +	default:
> +		dev_err(pci->dev, "num-lanes %u: invalid value\n", num_lanes);
> +		return;
>  	}
> +	dw_pcie_writel_dbi(pci, PCIE_PORT_LINK_CONTROL, plc);
>  	dw_pcie_writel_dbi(pci, PCIE_LINK_WIDTH_SPEED_CONTROL, lwsc);
>  }
>  
> @@ -1027,31 +1039,5 @@ void dw_pcie_setup(struct dw_pcie *pci)
>  	val |= PORT_LINK_DLL_LINK_EN;
>  	dw_pcie_writel_dbi(pci, PCIE_PORT_LINK_CONTROL, val);
>  
> -	if (!pci->num_lanes) {
> -		dev_dbg(pci->dev, "Using h/w default number of lanes\n");
> -		return;
> -	}
> -
> -	/* Set the number of lanes */
> -	val &= ~PORT_LINK_MODE_MASK;
> -	switch (pci->num_lanes) {
> -	case 1:
> -		val |= PORT_LINK_MODE_1_LANES;
> -		break;
> -	case 2:
> -		val |= PORT_LINK_MODE_2_LANES;
> -		break;
> -	case 4:
> -		val |= PORT_LINK_MODE_4_LANES;
> -		break;
> -	case 8:
> -		val |= PORT_LINK_MODE_8_LANES;
> -		break;
> -	default:
> -		dev_err(pci->dev, "num-lanes %u: invalid value\n", pci->num_lanes);
> -		return;
> -	}
> -	dw_pcie_writel_dbi(pci, PCIE_PORT_LINK_CONTROL, val);
> -
>  	dw_pcie_link_set_max_link_width(pci, pci->num_lanes);
>  }
> -- 
> 2.25.1
> 

Re: [PATCH v16 11/22] PCI: dwc: Add dw_pcie_link_set_max_cap_width()

[Serge Semin]

On Wed, May 10, 2023 at 03:22:23PM +0900, Yoshihiro Shimoda wrote:
> Add dw_pcie_link_set_max_cap_width() to set PCI_EXP_LNKCAP_MLW.
> In accordance with the DW PCIe RC/EP HW manuals [1,2,3,...] aside with
> the PORT_LINK_CTRL_OFF.LINK_CAPABLE and GEN2_CTRL_OFF.NUM_OF_LANES[8:0]
> field there is another one which needs to be updated. It's
> LINK_CAPABILITIES_REG.PCIE_CAP_MAX_LINK_WIDTH. If it isn't done at
> the very least the maximum link-width capability CSR won't expose
> the actual maximum capability.
> 
> [1] DesignWare Cores PCI Express Controller Databook - DWC PCIe Root Port,
>     Version 4.60a, March 2015, p.1032
> [2] DesignWare Cores PCI Express Controller Databook - DWC PCIe Root Port,
>     Version 4.70a, March 2016, p.1065
> [3] DesignWare Cores PCI Express Controller Databook - DWC PCIe Root Port,
>     Version 4.90a, March 2016, p.1057
> ...
> [X] DesignWare Cores PCI Express Controller Databook - DWC PCIe Endpoint,
>       Version 5.40a, March 2019, p.1396
> [X+1] DesignWare Cores PCI Express Controller Databook - DWC PCIe Root Port,
>       Version 5.40a, March 2019, p.1266
> 
> The commit description is suggested by Serge Semin.
> 
> Suggested-by: Serge Semin <fancer.lancer@gmail.com>
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> ---
>  drivers/pci/controller/dwc/pcie-designware.c | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
> index 5dc423dd2f21..8b2978c6eb23 100644
> --- a/drivers/pci/controller/dwc/pcie-designware.c
> +++ b/drivers/pci/controller/dwc/pcie-designware.c
> @@ -758,6 +758,21 @@ static void dw_pcie_link_set_max_link_width(struct dw_pcie *pci, u32 num_lanes)
>  	dw_pcie_writel_dbi(pci, PCIE_LINK_WIDTH_SPEED_CONTROL, lwsc);
>  }
>  
> +static void dw_pcie_link_set_max_cap_width(struct dw_pcie *pci, int num_lanes)
> +{

> +	u32 val;
> +	u8 cap;
> +
> +	if (!num_lanes)
> +		return;
> +
> +	cap = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP);
> +	val = dw_pcie_readl_dbi(pci, cap + PCI_EXP_LNKCAP);
> +	val &= ~PCI_EXP_LNKCAP_MLW;
> +	val |= num_lanes << PCI_EXP_LNKSTA_NLW_SHIFT;
> +	dw_pcie_writel_dbi(pci, cap + PCI_EXP_LNKCAP, val);

Once again. Please move this code to the
dw_pcie_link_set_max_link_width() method as I already asked here:
https://lore.kernel.org/linux-pci/20230501195753.o3qfcs7qyergccnr@mobilestation/
There is no point in creating a separate method for the action which
is implied by the already defined and incomplete
dw_pcie_link_set_max_link_width() function.

Also as we already agreed please replace the hard-coded bitwise shift
operation with the FIELD_PREP(PCI_EXP_LNKCAP_MLW, num_lanes)
statement.

-Serge(y)

> +}
> +
>  void dw_pcie_iatu_detect(struct dw_pcie *pci)
>  {
>  	int max_region, ob, ib;
> @@ -1040,4 +1055,5 @@ void dw_pcie_setup(struct dw_pcie *pci)
>  	dw_pcie_writel_dbi(pci, PCIE_PORT_LINK_CONTROL, val);
>  
>  	dw_pcie_link_set_max_link_width(pci, pci->num_lanes);
> +	dw_pcie_link_set_max_cap_width(pci, pci->num_lanes);
>  }
> -- 
> 2.25.1
> 

Re: [PATCH v16 12/22] PCI: tegra194: Drop PCI_EXP_LNKSTA_NLW setting.

[Serge Semin]

On Wed, May 10, 2023 at 03:22:24PM +0900, Yoshihiro Shimoda wrote:
> dw_pcie_setup() will set PCI_EXP_LNKSTA_NLW to PCI_EXP_LNKCAP register
> so that drop such setting from tegra_pcie_dw_host_init().
> 
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> Cc: Thierry Reding <thierry.reding@gmail.com>
> Cc: Jonathan Hunter <jonathanh@nvidia.com>

AFAICS this update isn't supposed to bring any regression since the
tegra_pcie_dw_host_init() call is always followed by the
dw_pcie_setup_rc() method invocation which thanks to your patches now
fully performs the link width setups. So from my point of view:

Reviewed-by: Serge Semin <fancer.lancer@gmail.com>

Though testing this on a real hw would be very welcome.

-Serge(y)

> ---
>  drivers/pci/controller/dwc/pcie-tegra194.c | 6 ------
>  1 file changed, 6 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-tegra194.c b/drivers/pci/controller/dwc/pcie-tegra194.c
> index 4adba379b83d..723a22ccd58c 100644
> --- a/drivers/pci/controller/dwc/pcie-tegra194.c
> +++ b/drivers/pci/controller/dwc/pcie-tegra194.c
> @@ -901,12 +901,6 @@ static int tegra_pcie_dw_host_init(struct dw_pcie_rp *pp)
>  		AMBA_ERROR_RESPONSE_CRS_SHIFT);
>  	dw_pcie_writel_dbi(pci, PORT_LOGIC_AMBA_ERROR_RESPONSE_DEFAULT, val);
>  
> -	/* Configure Max lane width from DT */
> -	val = dw_pcie_readl_dbi(pci, pcie->pcie_cap_base + PCI_EXP_LNKCAP);
> -	val &= ~PCI_EXP_LNKCAP_MLW;
> -	val |= (pcie->num_lanes << PCI_EXP_LNKSTA_NLW_SHIFT);
> -	dw_pcie_writel_dbi(pci, pcie->pcie_cap_base + PCI_EXP_LNKCAP, val);
> -
>  	/* Clear Slot Clock Configuration bit if SRNS configuration */
>  	if (pcie->enable_srns) {
>  		val_16 = dw_pcie_readw_dbi(pci, pcie->pcie_cap_base +
> -- 
> 2.25.1
> 

Re: [PATCH v16 13/22] PCI: dwc: Add EDMA_UNROLL capability flag

[Serge Semin]

On Wed, May 10, 2023 at 03:22:25PM +0900, Yoshihiro Shimoda wrote:
> Renesas R-Car Gen4 PCIe controllers have an unexpected register value on
> the dbi+0x97b register. So, add a new capability flag "EDMA_UNROLL"
> which would force the unrolled eDMA mapping for the problematic
> device, 

> as suggested by Serge Semin.

Drop this. Suggested-by tag already means that.

> 
> Suggested-by: Serge Semin <fancer.lancer@gmail.com>
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> ---
>  drivers/pci/controller/dwc/pcie-designware.c | 8 +++++++-
>  drivers/pci/controller/dwc/pcie-designware.h | 5 +++--
>  2 files changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
> index 8b2978c6eb23..e405bfae0191 100644
> --- a/drivers/pci/controller/dwc/pcie-designware.c
> +++ b/drivers/pci/controller/dwc/pcie-designware.c
> @@ -881,8 +881,14 @@ static int dw_pcie_edma_find_chip(struct dw_pcie *pci)
>  	 * Indirect eDMA CSRs access has been completely removed since v5.40a
>  	 * thus no space is now reserved for the eDMA channels viewport and
>  	 * former DMA CTRL register is no longer fixed to FFs.
> +	 *
> +	 * Note that Renesas R-Car S4-8's PCIe controllers for unknown reason

> +	 * may have zeros in the eDMA CTRL register even though the HW-manual

s/may have/have
(your comment is about a particular device which for sure has the
denoted problem.)

Other than that the change looks good. Thanks!
Reviewed-by: Serge Semin <fancer.lancer@gmail.com>

-Serge(y)

> +	 * explicitly states there must FFs if the unrolled mapping is enabled.
> +	 * For such cases the low-level drivers are supposed to manually
> +	 * activate the unrolled mapping to bypass the auto-detection procedure.
>  	 */
> -	if (dw_pcie_ver_is_ge(pci, 540A))
> +	if (dw_pcie_ver_is_ge(pci, 540A) || dw_pcie_cap_is(pci, EDMA_UNROLL))
>  		val = 0xFFFFFFFF;
>  	else
>  		val = dw_pcie_readl_dbi(pci, PCIE_DMA_VIEWPORT_BASE + PCIE_DMA_CTRL);
> diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
> index 06e044e2163a..2639206b4c18 100644
> --- a/drivers/pci/controller/dwc/pcie-designware.h
> +++ b/drivers/pci/controller/dwc/pcie-designware.h
> @@ -54,8 +54,9 @@
>  
>  /* DWC PCIe controller capabilities */
>  #define DW_PCIE_CAP_REQ_RES		0
> -#define DW_PCIE_CAP_IATU_UNROLL		1
> -#define DW_PCIE_CAP_CDM_CHECK		2
> +#define DW_PCIE_CAP_EDMA_UNROLL		1
> +#define DW_PCIE_CAP_IATU_UNROLL		2
> +#define DW_PCIE_CAP_CDM_CHECK		3
>  
>  #define dw_pcie_cap_is(_pci, _cap) \
>  	test_bit(DW_PCIE_CAP_ ## _cap, &(_pci)->caps)
> -- 
> 2.25.1
> 

Re: [PATCH v16 14/22] PCI: dwc: Expose dw_pcie_ep_exit() to module

[Serge Semin]

On Wed, May 10, 2023 at 03:22:26PM +0900, Yoshihiro Shimoda wrote:
> Since no PCIe controller drivers call this, this change is not required
> for now. But, Renesas R-Car Gen4 PCIe controller driver will call this
> and if the controller driver is built as a kernel module, the following
> build error happens. So, expose dw_pcie_ep_exit() for it.
> 
> ERROR: modpost: "dw_pcie_ep_exit" [drivers/pci/controller/dwc/pcie-rcar-gen4-ep-drv.ko] undefined!
> 
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>

Looks good. Thanks!
Reviewed-by: Serge Semin <fancer.lancer@gmail.com>

-Serge(y)

> ---
>  drivers/pci/controller/dwc/pcie-designware-ep.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
> index 0abc0073b1cf..023938468b5d 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> @@ -676,6 +676,7 @@ void dw_pcie_ep_exit(struct dw_pcie_ep *ep)
>  
>  	pci_epc_mem_exit(epc);
>  }
> +EXPORT_SYMBOL_GPL(dw_pcie_ep_exit);
>  
>  static unsigned int dw_pcie_ep_find_ext_capability(struct dw_pcie *pci, int cap)
>  {
> -- 
> 2.25.1
> 

Re: [PATCH v16 19/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support

[Serge Semin]

On Wed, May 10, 2023 at 03:22:31PM +0900, Yoshihiro Shimoda wrote:
> Add R-Car Gen4 PCIe Host support. This controller is based on
> Synopsys DesignWare PCIe, but this controller has vendor-specific
> registers so that requires initialization code like mode setting
> and retraining and so on.
> 
> To reduce code delta, adds some helper functions which are used by
> both the host driver and the endpoint driver (which is added
> immediately afterwards) into a separate file.
> 
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> ---
>  drivers/pci/controller/dwc/Kconfig            |   9 +
>  drivers/pci/controller/dwc/Makefile           |   2 +
>  .../pci/controller/dwc/pcie-rcar-gen4-host.c  | 141 +++++++++++++
>  drivers/pci/controller/dwc/pcie-rcar-gen4.c   | 190 ++++++++++++++++++
>  drivers/pci/controller/dwc/pcie-rcar-gen4.h   |  46 +++++
>  5 files changed, 388 insertions(+)
>  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
>  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.c
>  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.h
> 
> diff --git a/drivers/pci/controller/dwc/Kconfig b/drivers/pci/controller/dwc/Kconfig
> index ab96da43e0c2..64d4d37bc891 100644
> --- a/drivers/pci/controller/dwc/Kconfig
> +++ b/drivers/pci/controller/dwc/Kconfig
> @@ -415,4 +415,13 @@ config PCIE_VISCONTI_HOST
>  	  Say Y here if you want PCIe controller support on Toshiba Visconti SoC.
>  	  This driver supports TMPV7708 SoC.
>  
> +config PCIE_RCAR_GEN4
> +	tristate "Renesas R-Car Gen4 PCIe Host controller"
> +	depends on ARCH_RENESAS || COMPILE_TEST
> +	depends on PCI_MSI
> +	select PCIE_DW_HOST
> +	help
> +	  Say Y here if you want PCIe host controller support on R-Car Gen4 SoCs.
> +	  This uses the DesignWare core.
> +
>  endmenu
> diff --git a/drivers/pci/controller/dwc/Makefile b/drivers/pci/controller/dwc/Makefile
> index bf5c311875a1..486cf706b53d 100644
> --- a/drivers/pci/controller/dwc/Makefile
> +++ b/drivers/pci/controller/dwc/Makefile
> @@ -26,6 +26,8 @@ obj-$(CONFIG_PCIE_TEGRA194) += pcie-tegra194.o
>  obj-$(CONFIG_PCIE_UNIPHIER) += pcie-uniphier.o
>  obj-$(CONFIG_PCIE_UNIPHIER_EP) += pcie-uniphier-ep.o
>  obj-$(CONFIG_PCIE_VISCONTI_HOST) += pcie-visconti.o
> +pcie-rcar-gen4-host-drv-objs := pcie-rcar-gen4.o pcie-rcar-gen4-host.o
> +obj-$(CONFIG_PCIE_RCAR_GEN4) += pcie-rcar-gen4-host-drv.o
>  
>  # The following drivers are for devices that use the generic ACPI
>  # pci_root.c driver but don't support standard ECAM config access.
> diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> new file mode 100644
> index 000000000000..df7d80f1874f
> --- /dev/null
> +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> @@ -0,0 +1,141 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * PCIe host controller driver for Renesas R-Car Gen4 Series SoCs
> + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> + */
> +
> +#include <linux/delay.h>
> +#include <linux/interrupt.h>
> +#include <linux/module.h>
> +#include <linux/of_device.h>
> +#include <linux/pci.h>
> +#include <linux/platform_device.h>
> +
> +#include "pcie-rcar-gen4.h"
> +#include "pcie-designware.h"
> +
> +static int rcar_gen4_pcie_host_init(struct dw_pcie_rp *pp)
> +{
> +	struct dw_pcie *dw = to_dw_pcie_from_pp(pp);
> +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> +	int ret;
> +	u32 val;
> +
> +	gpiod_set_value_cansleep(dw->pe_rst, 1);
> +
> +	ret = rcar_gen4_pcie_set_device_type(rcar, true, dw->num_lanes);
> +	if (ret < 0)
> +		return ret;
> +

> +	dw_pcie_dbi_ro_wr_en(dw);

Are you sure dw_pcie_dbi_ro_wr_en() and dw_pcie_dbi_ro_wr_dis() are
needed? In accordance with the DW PCIe Dual-mode HW manual the BARx
registers are W-only over the DBI2 map with no need in setting the
DBI_RO_WR_EN flag.

Please check that on your hardware.

> +
> +	/*
> +	 * According to the section 3.5.7.2 "RC Mode" in DWC PCIe Dual Mode
> +	 * Rev.5.20a, we should disable two BARs to avoid unnecessary memory
> +	 * assignment during device enumeration.
> +	 */
> +	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_0, 0x0);
> +	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_1, 0x0);
> +

> +	dw_pcie_dbi_ro_wr_dis(dw);

ditto

> +
> +	if (IS_ENABLED(CONFIG_PCI_MSI)) {
> +		/* Enable MSI interrupt signal */
> +		val = readl(rcar->base + PCIEINTSTS0EN);
> +		val |= MSI_CTRL_INT;
> +		writel(val, rcar->base + PCIEINTSTS0EN);
> +	}
> +
> +	msleep(100);	/* pe_rst requires 100msec delay */
> +
> +	gpiod_set_value_cansleep(dw->pe_rst, 0);
> +
> +	return 0;
> +}
> +
> +static const struct dw_pcie_host_ops rcar_gen4_pcie_host_ops = {
> +	.host_init = rcar_gen4_pcie_host_init,
> +};
> +
> +static int rcar_gen4_add_dw_pcie_rp(struct rcar_gen4_pcie *rcar,
> +				   struct platform_device *pdev)
> +{
> +	struct dw_pcie *dw = &rcar->dw;
> +	struct dw_pcie_rp *pp = &dw->pp;
> +
> +	pp->num_vectors = MAX_MSI_IRQS;
> +	pp->ops = &rcar_gen4_pcie_host_ops;
> +	dw_pcie_cap_set(dw, REQ_RES);
> +
> +	return dw_pcie_host_init(pp);
> +}
> +
> +static void rcar_gen4_remove_dw_pcie_rp(struct rcar_gen4_pcie *rcar)
> +{
> +	dw_pcie_host_deinit(&rcar->dw.pp);
> +	gpiod_set_value_cansleep(rcar->dw.pe_rst, 1);
> +}
> +
> +static int rcar_gen4_pcie_probe(struct platform_device *pdev)
> +{
> +	struct device *dev = &pdev->dev;
> +	struct rcar_gen4_pcie *rcar;
> +	int err;
> +
> +	rcar = rcar_gen4_pcie_devm_alloc(dev);
> +	if (!rcar)
> +		return -ENOMEM;
> +
> +	err = rcar_gen4_pcie_get_resources(rcar, pdev);
> +	if (err < 0) {
> +		dev_err(dev, "Failed to request resource: %d\n", err);
> +		return err;
> +	}
> +
> +	platform_set_drvdata(pdev, rcar);
> +
> +	err = rcar_gen4_pcie_prepare(rcar);
> +	if (err < 0)
> +		return err;
> +
> +	rcar->needs_retrain = true;
> +	err = rcar_gen4_add_dw_pcie_rp(rcar, pdev);
> +	if (err < 0)
> +		goto err_add;
> +
> +	return 0;
> +
> +err_add:
> +	rcar_gen4_pcie_unprepare(rcar);
> +
> +	return err;
> +}
> +
> +static int rcar_gen4_pcie_remove(struct platform_device *pdev)
> +{
> +	struct rcar_gen4_pcie *rcar = platform_get_drvdata(pdev);
> +
> +	rcar_gen4_remove_dw_pcie_rp(rcar);
> +	rcar_gen4_pcie_unprepare(rcar);
> +
> +	return 0;
> +}
> +
> +static const struct of_device_id rcar_gen4_pcie_of_match[] = {
> +	{ .compatible = "renesas,rcar-gen4-pcie", },
> +	{},
> +};
> +
> +static struct platform_driver rcar_gen4_pcie_driver = {
> +	.driver = {
> +		.name = "pcie-rcar-gen4",
> +		.of_match_table = rcar_gen4_pcie_of_match,
> +		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
> +	},
> +	.probe = rcar_gen4_pcie_probe,
> +	.remove = rcar_gen4_pcie_remove,
> +};
> +module_platform_driver(rcar_gen4_pcie_driver);
> +
> +MODULE_DESCRIPTION("Renesas R-Car Gen4 PCIe host controller driver");
> +MODULE_LICENSE("GPL");
> diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.c b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> new file mode 100644
> index 000000000000..35923fda8ed5
> --- /dev/null
> +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> @@ -0,0 +1,190 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
> + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> + */
> +
> +#include <linux/delay.h>
> +#include <linux/io.h>
> +#include <linux/of_device.h>
> +#include <linux/pci.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/reset.h>
> +
> +#include "pcie-rcar-gen4.h"
> +#include "pcie-designware.h"
> +
> +/* Renesas-specific */
> +#define PCIERSTCTRL1		0x0014
> +#define  APP_HOLD_PHY_RST	BIT(16)
> +#define  APP_LTSSM_ENABLE	BIT(0)
> +
> +#define RETRAIN_MAX_CHECK	10
> +#define RETRAIN_MAX_RETRIES	10
> +
> +static void rcar_gen4_pcie_ltssm_enable(struct rcar_gen4_pcie *rcar,
> +					bool enable)
> +{
> +	u32 val;
> +
> +	val = readl(rcar->base + PCIERSTCTRL1);
> +	if (enable) {
> +		val |= APP_LTSSM_ENABLE;

> +		val &= ~APP_HOLD_PHY_RST;

What about moving the APP_HOLD_PHY_RST de-assertion to the
rcar_gen4_pcie_set_device_type() method? In accordance with the
"3.1 Initialization" chapter it's supposed to be done before
performing the DBI programming and activating the link training.

> +	} else {
> +		val &= ~APP_LTSSM_ENABLE;
> +		val |= APP_HOLD_PHY_RST;
> +	}
> +	writel(val, rcar->base + PCIERSTCTRL1);
> +}
> +
> +static bool rcar_gen4_pcie_check_retrain_link(struct dw_pcie *dw)
> +{
> +	u8 offset = dw_pcie_find_capability(dw, PCI_CAP_ID_EXP);
> +	u32 lnkcap = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCAP);
> +	u32 lnkctl = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCTL);
> +	u16 lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> +	int i;
> +

> +	if ((lnksta & PCI_EXP_LNKSTA_CLS) == (lnkcap & PCI_EXP_LNKCAP_SLS))
> +		return true;
> +
> +	lnkctl |= PCI_EXP_LNKCTL_RL;
> +	dw_pcie_writel_dbi(dw, offset + PCI_EXP_LNKCTL, lnkctl);
> +
> +	for (i = 0; i < RETRAIN_MAX_CHECK; i++) {
> +		lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> +		if (lnksta & PCI_EXP_LNKSTA_LT)
> +			return true;
> +		usleep_range(1000, 1100);
> +	}

I'll ask one more time because you didn't respond to my previous note
about this. Are you sure that this is needed? Did you try
the approach described in "3.13 Gen2/3/4/5 Speed Modes" with
de-asserting/asserting the GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag?
I keep asking because the same problem we used to have on our hardware
until we found out that the DIRECT_SPEED_CHANGE flag helped to train
the link right to the speed specified in the capabilities.

So here is what presumably you'll need to do (based on the
"3.1 Initialization" and "3.13 Gen2/3/4/5 Speed Modes" chapters of
the DW PCIe DM hw-manual):
1. Make sure the controller is in the power-down/reset state.
2. Select device_type (EP or RP).
3. De-assert the controller reset.
4. Clear PHY-reset flag in the app registers.
5. Perform some controller initializations.
6. Enable LTSSM to start link training.
7. Set GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag one more time.

1-4 are supposed to be done in rcar_gen4_pcie_host_init().
5 is performed in the framework of the DW PCIe core driver.
6-7 should be done in rcar_gen4_pcie_start_link().

Note 1. GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag is already set on stage
5 in the framework of the dw_pcie_setup_rc() method. But in our case
it only caused having the Gen.2 link speed. Adding stage 7 helped to
get stable Gen.3 link. So please try the denoted approach. If it works
what about adding stage 7 twice in order to get Gen.4 speed?
(waiting for the DIRECT_SPEED_CHANGE flag being auto-cleared and then
set it up again?)

Note 2. GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag is defined as
PCIE_LINK_WIDTH_SPEED_CONTROL.PORT_LOGIC_SPEED_CHANGE macros in the DW
PCIe core driver.

Note 3. If what is suggested above works well then you won't need to
have the heavy rcar_gen4_pcie_check_retrain_link() method in the way
you have it defined.

> +
> +	return false;
> +}
> +
> +static int rcar_gen4_pcie_link_up(struct dw_pcie *dw)
> +{
> +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> +	u32 val, mask;
> +
> +	val = readl(rcar->base + PCIEINTSTS0);
> +	mask = RDLH_LINK_UP | SMLH_LINK_UP;
> +
> +	return (val & mask) == mask;
> +}
> +
> +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> +{
> +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> +	int i;
> +
> +	rcar_gen4_pcie_ltssm_enable(rcar, true);
> +
> +	/*
> +	 * Require retraining here. Otherwise RDLH_LINK_UP of PCIEINTSTS0 which
> +	 * is this controller specific register may not be set.
> +	 */
> +	if (rcar->needs_retrain) {
> +		for (i = 0; i < RETRAIN_MAX_RETRIES; i++) {
> +			if (rcar_gen4_pcie_check_retrain_link(dw))
> +				return 0;
> +			msleep(100);
> +		}
> +
> +		return -ETIMEDOUT;	/* Failed */
> +	}
> +
> +	return 0;
> +}
> +
> +static void rcar_gen4_pcie_stop_link(struct dw_pcie *dw)
> +{
> +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> +
> +	rcar_gen4_pcie_ltssm_enable(rcar, false);
> +}
> +

> +int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
> +				   int num_lanes)

1. Number of lanes is already defined in the rcar_gen4_pcie.dw.num_lanes field.
What about using it from there instead of passing it as an argument?
2. DW PCIe core driver has a very handy enum defined:
dw_pcie_device_mode. It describes the controller modes (End-point,
Root port, etc). What about adding the mode field right to the
rcar_gen4_pcie structure and initializing it in someplace in probe() ?
3. Based on the function semantic it's better to be named as something
like rcar_gen4_pcie_init_device() or even rcar_gen4_pcie_basic_init().


> +{
> +	u32 val;
> +

> +	/* Note: Assume the rcar->rst which is Cold-reset is asserted here */

What about directly asserting it here then? In accordance with the DW
PCIe DM manual the "device_type" input must be set before the DM
controller is powered up (basically un-reset). What if the controller
reset is already de-asserted, but you are going to changes its mode?
In that case the mode won't be changed and you'll end up with
unpredictable results.

> +	val = readl(rcar->base + PCIEMSR0);
> +	if (rc)
> +		val |= DEVICE_TYPE_RC;
> +	else
> +		val |= DEVICE_TYPE_EP;
> +
> +	if (num_lanes < 4)
> +		val |= BIFUR_MOD_SET_ON;
> +
> +	writel(val, rcar->base + PCIEMSR0);
> +
> +	return reset_control_deassert(rcar->rst);
> +}
> +
> +int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *rcar)
> +{
> +	struct device *dev = rcar->dw.dev;
> +	int err;
> +
> +	pm_runtime_enable(dev);
> +	err = pm_runtime_resume_and_get(dev);
> +	if (err < 0) {
> +		dev_err(dev, "Failed to resume/get Runtime PM\n");
> +		pm_runtime_disable(dev);
> +	}
> +
> +	return err;
> +}
> +
> +void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *rcar)
> +{
> +	struct device *dev = rcar->dw.dev;
> +
> +	if (!reset_control_status(rcar->rst))
> +		reset_control_assert(rcar->rst);
> +	pm_runtime_put(dev);
> +	pm_runtime_disable(dev);
> +}
> +
> +int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
> +				 struct platform_device *pdev)
> +{
> +	struct device *dev = rcar->dw.dev;
> +
> +	/* Renesas-specific registers */
> +	rcar->base = devm_platform_ioremap_resource_byname(pdev, "app");
> +	if (IS_ERR(rcar->base))
> +		return PTR_ERR(rcar->base);
> +

> +	rcar->rst = devm_reset_control_get(dev, NULL);
> +	if (IS_ERR(rcar->rst)) {
> +		dev_err(dev, "Failed to get Cold-reset\n");

So AFAICS your platform is equipped with the DWC_pcie_clkrst.v module.
Thus all the resets are appropriately cleared by a single flag:
power_up_rst_n. What about using the named reset in this case with the
"pwr" name? Thus you'll be able to drop the manual
devm_reset_control_get() invocation and instead use the reset-resources
requested in the framework of the generic dw_pcie_get_resources()
method? Note you'll need to move the dw_pcie_cap_set(dw, REQ_RES);
statement to rcar_gen4_pcie_devm_alloc() then and drop the
rcar_gen4_pcie.rst field afterwords.

By the way I don't see you requesting and enabling the reference
clock in your driver but the bindings imply the clock source. How
come?

> +		return PTR_ERR(rcar->rst);
> +	}
> +
> +	return 0;
> +}
> +
> +static const struct dw_pcie_ops dw_pcie_ops = {
> +	.start_link = rcar_gen4_pcie_start_link,
> +	.stop_link = rcar_gen4_pcie_stop_link,
> +	.link_up = rcar_gen4_pcie_link_up,
> +};
> +
> +struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev)
> +{
> +	struct rcar_gen4_pcie *rcar;
> +
> +	rcar = devm_kzalloc(dev, sizeof(*rcar), GFP_KERNEL);
> +	if (!rcar)
> +		return NULL;
> +
> +	rcar->dw.dev = dev;
> +	rcar->dw.ops = &dw_pcie_ops;
> +	dw_pcie_cap_set(&rcar->dw, EDMA_UNROLL);
> +
> +	return rcar;
> +}
> diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.h b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
> new file mode 100644
> index 000000000000..fec3f18609f4
> --- /dev/null
> +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
> @@ -0,0 +1,46 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
> + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> + */
> +
> +#ifndef _PCIE_RCAR_GEN4_H_
> +#define _PCIE_RCAR_GEN4_H_
> +
> +#include <linux/io.h>
> +#include <linux/pci.h>
> +#include <linux/reset.h>
> +
> +#include "pcie-designware.h"
> +
> +/* Renesas-specific */
> +#define PCIEMSR0		0x0000
> +#define  BIFUR_MOD_SET_ON	BIT(0)
> +#define  DEVICE_TYPE_EP		0
> +#define  DEVICE_TYPE_RC		BIT(4)
> +
> +#define PCIEINTSTS0		0x0084
> +#define PCIEINTSTS0EN		0x0310
> +#define  MSI_CTRL_INT		BIT(26)
> +#define  SMLH_LINK_UP		BIT(7)
> +#define  RDLH_LINK_UP		BIT(6)
> +#define PCIEDMAINTSTSEN		0x0314
> +#define  PCIEDMAINTSTSEN_INIT	GENMASK(15, 0)
> +

> +struct rcar_gen4_pcie {

As I mentioned above this structure can be extended with the enum
dw_pcie_device_mode field thus dropping the boolean argument from the
rcar_gen4_pcie_set_device_type() method.

> +	struct dw_pcie		dw;

As I already mentioned above the dw.num_lanes could be used instead of
passing it as the rcar_gen4_pcie_set_device_type() argument.

-Serge(y)

> +	void __iomem		*base;
> +	struct reset_control	*rst;
> +	bool			needs_retrain;
> +};
> +#define to_rcar_gen4_pcie(x)	dev_get_drvdata((x)->dev)
> +
> +int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
> +				   int num_lanes);
> +int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *pcie);
> +void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *pcie);
> +int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
> +				 struct platform_device *pdev);
> +struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev);
> +
> +#endif /* _PCIE_RCAR_GEN4_H_ */
> -- 
> 2.25.1
> 

Re: [PATCH v16 18/22] dt-bindings: PCI: renesas: Add R-Car Gen4 PCIe Endpoint

[Serge Semin]

On Wed, May 10, 2023 at 03:22:30PM +0900, Yoshihiro Shimoda wrote:
> Document bindings for Renesas R-Car Gen4 and R-Car S4-8 (R8A779F0)
> PCIe endpoint module.
> 
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> Reviewed-by: Rob Herring <robh@kernel.org>
> Reviewed-by: Serge Semin <fancer.lancer@gmail.com>
> Acked-by: Manivannan Sadhasivam <mani@kernel.org>
> ---
>  .../bindings/pci/rcar-gen4-pci-ep.yaml        | 98 +++++++++++++++++++
>  1 file changed, 98 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml
> 
> diff --git a/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml b/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml
> new file mode 100644
> index 000000000000..0453bdcf9645
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/pci/rcar-gen4-pci-ep.yaml
> @@ -0,0 +1,98 @@
> +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +# Copyright (C) 2022-2023 Renesas Electronics Corp.
> +%YAML 1.2
> +---
> +$id: http://devicetree.org/schemas/pci/rcar-gen4-pci-ep.yaml#
> +$schema: http://devicetree.org/meta-schemas/core.yaml#
> +
> +title: Renesas R-Car Gen4 PCIe Endpoint
> +
> +maintainers:
> +  - Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> +
> +allOf:
> +  - $ref: snps,dw-pcie-ep.yaml#
> +
> +properties:
> +  compatible:
> +    items:
> +      - const: renesas,r8a779f0-pcie-ep   # R-Car S4-8
> +      - const: renesas,rcar-gen4-pcie-ep  # R-Car Gen4
> +
> +  reg:
> +    maxItems: 6
> +
> +  reg-names:
> +    items:
> +      - const: dbi
> +      - const: dbi2
> +      - const: atu
> +      - const: dma
> +      - const: app
> +      - const: addr_space
> +
> +  interrupts:
> +    maxItems: 3
> +
> +  interrupt-names:
> +    items:
> +      - const: dma
> +      - const: sft_ce
> +      - const: app
> +
> +  power-domains:
> +    maxItems: 1
> +
> +  resets:
> +    maxItems: 1
> +
> +  clocks:
> +    maxItems: 1
> +
> +  max-functions:
> +    maximum: 2
> +
> +  max-link-speed:
> +    maximum: 4
> +
> +  num-lanes:
> +    maximum: 4
> +
> +required:
> +  - compatible
> +  - reg
> +  - reg-names
> +  - interrupts
> +  - resets
> +  - power-domains
> +  - clocks
> +
> +unevaluatedProperties: false
> +
> +examples:
> +  - |
> +    #include <dt-bindings/clock/r8a779f0-cpg-mssr.h>
> +    #include <dt-bindings/interrupt-controller/arm-gic.h>
> +    #include <dt-bindings/power/r8a779f0-sysc.h>
> +
> +    soc {
> +        #address-cells = <2>;
> +        #size-cells = <2>;
> +
> +        pcie0_ep: pcie-ep@e65d0000 {
> +            compatible = "renesas,r8a779f0-pcie-ep", "renesas,rcar-gen4-pcie-ep";

> +            reg = <0 0xe65d0000 0 0x2000>, <0 0xe65d2800 0 0x0800>,
> +                  <0 0xe65d3000 0 0x2000>, <0 0xe65d5000 0 0x1200>,
> +                  <0 0xe65d6200 0 0x0e00>, <0 0xfe000000 0 0x400000>;
> +            reg-names = "dbi", "dbi2", "atu", "dma", "app", "addr_space";

Just figured this out. I see you defining the dbi2 as <0 0xe65d2800 0
0x0800>. But sometime before you mentioned that your device has the
next CSRs layout:
! +0x0000 : Function 0 (common address in Root complex and Endpoint mode)
  +0x1000 : Function 1 (Endpoint mode only)
  +0x2000 : Shadow register for Function 0
! +0x2800 : Shadow register for Function 1
it means you have the dbi space defined for both functions meanwhile
the dbi2 space defined for _function #1_ only. Moreover
your DW PCIe End-point driver disables the multi-function feature
support. So AFAICS either you have wrong DW PCIe EP example node or
the node is wrong in your platform DTS too and you have a malfunction
end-point mode. Am I missing something? In any case based on the your
End-point driver implementation dbi2 is supposed to be defined at the
0xe65d2000 base address.

-Serge(y)

> +            interrupts = <GIC_SPI 417 IRQ_TYPE_LEVEL_HIGH>,
> +                         <GIC_SPI 418 IRQ_TYPE_LEVEL_HIGH>,
> +                         <GIC_SPI 422 IRQ_TYPE_LEVEL_HIGH>;
> +            interrupt-names = "dma", "sft_ce", "app";
> +            clocks = <&cpg CPG_MOD 624>;
> +            power-domains = <&sysc R8A779F0_PD_ALWAYS_ON>;
> +            resets = <&cpg 624>;
> +            num-lanes = <2>;
> +            max-link-speed = <4>;
> +        };
> +    };
> -- 
> 2.25.1
> 

Re: [PATCH v16 20/22] PCI: rcar-gen4-ep: Add R-Car Gen4 PCIe Endpoint support

[Serge Semin]

On Wed, May 10, 2023 at 03:22:32PM +0900, Yoshihiro Shimoda wrote:
> Add R-Car Gen4 PCIe Endpoint support. This controller is based on
> Synopsys DesignWare PCIe.
> 
> Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> ---
>  drivers/pci/controller/dwc/Kconfig            |   9 +
>  drivers/pci/controller/dwc/Makefile           |   2 +
>  .../pci/controller/dwc/pcie-rcar-gen4-ep.c    | 166 ++++++++++++++++++
>  3 files changed, 177 insertions(+)
>  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4-ep.c
> 
> diff --git a/drivers/pci/controller/dwc/Kconfig b/drivers/pci/controller/dwc/Kconfig
> index 64d4d37bc891..4d877cd18374 100644
> --- a/drivers/pci/controller/dwc/Kconfig
> +++ b/drivers/pci/controller/dwc/Kconfig
> @@ -424,4 +424,13 @@ config PCIE_RCAR_GEN4
>  	  Say Y here if you want PCIe host controller support on R-Car Gen4 SoCs.
>  	  This uses the DesignWare core.
>  
> +config PCIE_RCAR_GEN4_EP
> +	tristate "Renesas R-Car Gen4 PCIe Endpoint controller"
> +	depends on ARCH_RENESAS || COMPILE_TEST
> +	depends on PCI_ENDPOINT
> +	select PCIE_DW_EP
> +	help
> +	  Say Y here if you want PCIe endpoint controller support on R-Car Gen4
> +	  SoCs. This uses the DesignWare core.
> +
>  endmenu
> diff --git a/drivers/pci/controller/dwc/Makefile b/drivers/pci/controller/dwc/Makefile
> index 486cf706b53d..0fb0bde26ac4 100644
> --- a/drivers/pci/controller/dwc/Makefile
> +++ b/drivers/pci/controller/dwc/Makefile
> @@ -28,6 +28,8 @@ obj-$(CONFIG_PCIE_UNIPHIER_EP) += pcie-uniphier-ep.o
>  obj-$(CONFIG_PCIE_VISCONTI_HOST) += pcie-visconti.o
>  pcie-rcar-gen4-host-drv-objs := pcie-rcar-gen4.o pcie-rcar-gen4-host.o
>  obj-$(CONFIG_PCIE_RCAR_GEN4) += pcie-rcar-gen4-host-drv.o
> +pcie-rcar-gen4-ep-drv-objs := pcie-rcar-gen4.o pcie-rcar-gen4-ep.o
> +obj-$(CONFIG_PCIE_RCAR_GEN4_EP) += pcie-rcar-gen4-ep-drv.o
>  
>  # The following drivers are for devices that use the generic ACPI
>  # pci_root.c driver but don't support standard ECAM config access.
> diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4-ep.c b/drivers/pci/controller/dwc/pcie-rcar-gen4-ep.c
> new file mode 100644
> index 000000000000..710bbc9e61a5
> --- /dev/null
> +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4-ep.c
> @@ -0,0 +1,166 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * PCIe Endpoint driver for Renesas R-Car Gen4 Series SoCs
> + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> + */
> +
> +#include <linux/interrupt.h>
> +#include <linux/module.h>
> +#include <linux/of_device.h>
> +#include <linux/pci.h>
> +#include <linux/platform_device.h>
> +
> +#include "pcie-rcar-gen4.h"
> +#include "pcie-designware.h"
> +
> +static void rcar_gen4_pcie_ep_pre_init(struct dw_pcie_ep *ep)
> +{
> +	struct dw_pcie *dw = to_dw_pcie_from_ep(ep);
> +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> +	u8 val;
> +
> +	rcar_gen4_pcie_set_device_type(rcar, false, dw->num_lanes);
> +
> +	dw_pcie_dbi_ro_wr_en(dw);
> +

> +	/* Single function */
> +	val = dw_pcie_readb_dbi(dw, PCI_HEADER_TYPE);
> +	val &= ~PCI_HEADER_TYPE_MULTI_FUNC;
> +	dw_pcie_writeb_dbi(dw, PCI_HEADER_TYPE, val);

Just wondering. Did you try to implement the multi-function-ness
support? It don't seem to be much problematic (implementing
func_conf_select() callback and possibly some trick to distinguish dbi
and dbi2 spaces).

BTW based on your End-point DT-bindings the "max-functions" property
is permitted to have value 2. Thus your platforms may have both functions
activated and exposed to user-space meanwhile your driver doesn't seem
to support it...

-Serge(y)

> +
> +	dw_pcie_dbi_ro_wr_dis(dw);
> +
> +	writel(PCIEDMAINTSTSEN_INIT, rcar->base + PCIEDMAINTSTSEN);
> +}
> +
> +static void rcar_gen4_pcie_ep_deinit(struct dw_pcie_ep *ep)
> +{
> +	struct dw_pcie *dw = to_dw_pcie_from_ep(ep);
> +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> +
> +	writel(0, rcar->base + PCIEDMAINTSTSEN);
> +}
> +
> +static int rcar_gen4_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
> +				       enum pci_epc_irq_type type,
> +				       u16 interrupt_num)
> +{
> +	struct dw_pcie *dw = to_dw_pcie_from_ep(ep);
> +
> +	switch (type) {
> +	case PCI_EPC_IRQ_INTX:
> +		return dw_pcie_ep_raise_intx_irq(ep, func_no);
> +	case PCI_EPC_IRQ_MSI:
> +		return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
> +	default:
> +		dev_err(dw->dev, "Unknown IRQ type\n");
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static const struct pci_epc_features rcar_gen4_pcie_epc_features = {
> +	.linkup_notifier = false,
> +	.msi_capable = true,
> +	.msix_capable = false,
> +	.reserved_bar = 1 << BAR_5,
> +	.align = SZ_1M,
> +};
> +
> +static const struct pci_epc_features*
> +rcar_gen4_pcie_ep_get_features(struct dw_pcie_ep *ep)
> +{
> +	return &rcar_gen4_pcie_epc_features;
> +}
> +
> +static const struct dw_pcie_ep_ops pcie_ep_ops = {
> +	.ep_pre_init = rcar_gen4_pcie_ep_pre_init,
> +	.ep_deinit = rcar_gen4_pcie_ep_deinit,
> +	.raise_irq = rcar_gen4_pcie_ep_raise_irq,
> +	.get_features = rcar_gen4_pcie_ep_get_features,
> +};
> +
> +static int rcar_gen4_add_pcie_ep(struct rcar_gen4_pcie *rcar,
> +				 struct platform_device *pdev)
> +{
> +	struct dw_pcie_ep *ep = &rcar->dw.ep;
> +	int ret;
> +
> +	ep->ops = &pcie_ep_ops;
> +
> +	ret = dw_pcie_ep_init(ep);
> +	if (ret) {
> +		dev_err(&pdev->dev, "Failed to initialize endpoint\n");
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static void rcar_gen4_remove_pcie_ep(struct rcar_gen4_pcie *rcar)
> +{
> +	dw_pcie_ep_exit(&rcar->dw.ep);
> +}
> +
> +static int rcar_gen4_pcie_ep_probe(struct platform_device *pdev)
> +{
> +	struct device *dev = &pdev->dev;
> +	struct rcar_gen4_pcie *rcar;
> +	int err;
> +
> +	rcar = rcar_gen4_pcie_devm_alloc(dev);
> +	if (!rcar)
> +		return -ENOMEM;
> +
> +	err = rcar_gen4_pcie_get_resources(rcar, pdev);
> +	if (err < 0) {
> +		dev_err(dev, "Failed to request resource: %d\n", err);
> +		return err;
> +	}
> +
> +	platform_set_drvdata(pdev, rcar);
> +
> +	err = rcar_gen4_pcie_prepare(rcar);
> +	if (err < 0)
> +		return err;
> +
> +	err = rcar_gen4_add_pcie_ep(rcar, pdev);
> +	if (err < 0)
> +		goto err_add;
> +
> +	return 0;
> +
> +err_add:
> +	rcar_gen4_pcie_unprepare(rcar);
> +
> +	return err;
> +}
> +
> +static int rcar_gen4_pcie_ep_remove(struct platform_device *pdev)
> +{
> +	struct rcar_gen4_pcie *rcar = platform_get_drvdata(pdev);
> +
> +	rcar_gen4_remove_pcie_ep(rcar);
> +	rcar_gen4_pcie_unprepare(rcar);
> +
> +	return 0;
> +}
> +
> +static const struct of_device_id rcar_gen4_pcie_of_match[] = {
> +	{ .compatible = "renesas,rcar-gen4-pcie-ep", },
> +	{},
> +};
> +
> +static struct platform_driver rcar_gen4_pcie_ep_driver = {
> +	.driver = {
> +		.name = "pcie-rcar-gen4-ep",
> +		.of_match_table = rcar_gen4_pcie_of_match,
> +	},
> +	.probe = rcar_gen4_pcie_ep_probe,
> +	.remove = rcar_gen4_pcie_ep_remove,
> +};
> +module_platform_driver(rcar_gen4_pcie_ep_driver);
> +
> +MODULE_DESCRIPTION("Renesas R-Car Gen4 PCIe endpoint controller driver");
> +MODULE_LICENSE("GPL");
> -- 
> 2.25.1
> 

RE: [PATCH v16 08/22] PCI: designware-ep: Add INTx IRQs support

[Yoshihiro Shimoda]

Hello Serge,

> From: Serge Semin, Sent: Monday, June 5, 2023 5:06 PM
> 
> On Wed, May 10, 2023 at 03:22:20PM +0900, Yoshihiro Shimoda wrote:
> > Add support for triggering INTx IRQs by using outbound iATU.
> > Outbound iATU is utilized to send assert and de-assert INTx TLPs.
> > The message is generated based on the payloadless Msg TLP with type
> > 0x14, where 0x4 is the routing code implying the Terminate at
> > Receiver message. The message code is specified as b1000xx for
> > the INTx assertion and b1001xx for the INTx de-assertion.
> >
> > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > ---
> >  .../pci/controller/dwc/pcie-designware-ep.c   | 66 +++++++++++++++++--
> >  drivers/pci/controller/dwc/pcie-designware.h  |  2 +
> >  2 files changed, 64 insertions(+), 4 deletions(-)
> >
> > diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > index fe2e0d765be9..0abc0073b1cf 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > @@ -6,9 +6,11 @@
> >   * Author: Kishon Vijay Abraham I <kishon@ti.com>
> >   */
> >
> > +#include <linux/delay.h>
> >  #include <linux/of.h>
> >  #include <linux/platform_device.h>
> >
> > +#include "../../pci.h"
> >  #include "pcie-designware.h"
> >  #include <linux/pci-epc.h>
> >  #include <linux/pci-epf.h>
> > @@ -484,14 +486,57 @@ static const struct pci_epc_ops epc_ops = {
> >  	.get_features		= dw_pcie_ep_get_features,
> >  };
> >
> > +static int dw_pcie_ep_send_msg(struct dw_pcie_ep *ep, u8 func_no, u8 code,
> > +			       u8 routing)
> > +{
> > +	struct dw_pcie_ob_atu_cfg atu = { 0 };
> > +	struct pci_epc *epc = ep->epc;
> > +	int ret;
> > +
> > +	atu.func_no = func_no;
> > +	atu.code = code;
> > +	atu.routing = routing;
> > +	atu.type = PCIE_ATU_TYPE_MSG;
> > +	atu.cpu_addr = ep->intx_mem_phys;
> > +	atu.size = epc->mem->window.page_size;
> > +
> > +	ret = dw_pcie_ep_outbound_atu(ep, &atu);
> > +	if (ret)
> > +		return ret;
> > +
> > +	writel(0, ep->intx_mem);
> > +
> > +	dw_pcie_ep_unmap_addr(epc, func_no, 0, ep->intx_mem_phys);
> > +
> > +	return 0;
> > +}
> > +
> >  int dw_pcie_ep_raise_intx_irq(struct dw_pcie_ep *ep, u8 func_no)
> >  {
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >  	struct device *dev = pci->dev;
> > +	int ret;
> >
> > -	dev_err(dev, "EP cannot trigger INTx IRQs\n");
> > +	if (!ep->intx_mem) {
> > +		dev_err(dev, "INTx not supported\n");
> > +		return -EOPNOTSUPP;
> > +	}
> >
> > -	return -EINVAL;
> > +	/*
> 
> > +	 * According to the document of PCIe, the INTx emulation should be
> > +	 * level-triggered. However, the Linux PCIe Endpoint framework only
> > +	 * supports edge-triggered for now. So, this function asserts INTx for
> > +	 * 50 usec, and then deasserts it.
> 
> "Document of PCIe"?)
> What about the next message:
> Even though the PCI bus specification implies the level-triggered INTx
> interrupts the kernel PCIe endpoint framework has a single
> PCI_EPC_IRQ_INTx flag defined for the legacy IRQs simulation. Thus
> this function sends the Deassert_INTx PCIe TLP after the Assert_INTx
> message with the 50 usec duration basically implementing the
> rising-edge triggering IRQ. Hopefully the interrupt controller will
> still be able to register the incoming IRQ event...

Thank you for your suggestion! I'll use your comment on v17.

> Other than that the change looks good.

Thanks!

Best regards,
Yoshihiro Shimoda

> -Serge(y)
> 
> > +	 */
> > +	ret = dw_pcie_ep_send_msg(ep, func_no, PCI_CODE_ASSERT_INTA,
> > +				  PCI_MSG_ROUTING_LOCAL);
> > +	if (ret)
> > +		return ret;
> > +
> > +	usleep_range(50, 100);
> > +
> > +	return dw_pcie_ep_send_msg(ep, func_no, PCI_CODE_DEASSERT_INTA,
> > +				   PCI_MSG_ROUTING_LOCAL);
> >  }
> >  EXPORT_SYMBOL_GPL(dw_pcie_ep_raise_intx_irq);
> >
> > @@ -622,6 +667,10 @@ void dw_pcie_ep_exit(struct dw_pcie_ep *ep)
> >
> >  	dw_pcie_edma_remove(pci);
> >
> > +	if (ep->intx_mem)
> > +		pci_epc_mem_free_addr(epc, ep->intx_mem_phys, ep->intx_mem,
> > +				      epc->mem->window.page_size);
> > +
> >  	pci_epc_mem_free_addr(epc, ep->msi_mem_phys, ep->msi_mem,
> >  			      epc->mem->window.page_size);
> >
> > @@ -793,9 +842,14 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
> >  		goto err_exit_epc_mem;
> >  	}
> >
> > +	ep->intx_mem = pci_epc_mem_alloc_addr(epc, &ep->intx_mem_phys,
> > +					      epc->mem->window.page_size);
> > +	if (!ep->intx_mem)
> > +		dev_warn(dev, "Failed to reserve memory for INTx\n");
> > +
> >  	ret = dw_pcie_edma_detect(pci);
> >  	if (ret)
> > -		goto err_free_epc_mem;
> > +		goto err_free_epc_mem_intx;
> >
> >  	if (ep->ops->get_features) {
> >  		epc_features = ep->ops->get_features(ep);
> > @@ -812,7 +866,11 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep)
> >  err_remove_edma:
> >  	dw_pcie_edma_remove(pci);
> >
> > -err_free_epc_mem:
> > +err_free_epc_mem_intx:
> > +	if (ep->intx_mem)
> > +		pci_epc_mem_free_addr(epc, ep->intx_mem_phys, ep->intx_mem,
> > +				      epc->mem->window.page_size);
> > +
> >  	pci_epc_mem_free_addr(epc, ep->msi_mem_phys, ep->msi_mem,
> >  			      epc->mem->window.page_size);
> >
> > diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
> > index c83d1d176e62..06e044e2163a 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware.h
> > +++ b/drivers/pci/controller/dwc/pcie-designware.h
> > @@ -369,6 +369,8 @@ struct dw_pcie_ep {
> >  	unsigned long		*ob_window_map;
> >  	void __iomem		*msi_mem;
> >  	phys_addr_t		msi_mem_phys;
> > +	void __iomem		*intx_mem;
> > +	phys_addr_t		intx_mem_phys;
> >  	struct pci_epf_bar	*epf_bar[PCI_STD_NUM_BARS];
> >  };
> >
> > --
> > 2.25.1
> >

RE: [PATCH v16 10/22] PCI: dwc: Modify PCIE_PORT_LINK_CONTROL handling

[Yoshihiro Shimoda]

Hello Serge,

> From: Serge Semin, Sent: Monday, June 5, 2023 7:53 PM
> 
> On Wed, May 10, 2023 at 03:22:22PM +0900, Yoshihiro Shimoda wrote:
> > To improve code readability, modify PCIE_PORT_LINK_CONTROL handling.
> 
> So basically you are doing the same update as in the Patch 9:
> detaching the already implemented link width setups into a separate
> method. Why do you split them up into the incremental updates?

I thought that splitting them was review-friendly. But, it's wrong...

> Just
> squash this patch into the patch 9. The resultant patch would be an
> atomic update and a preparation before adding the PCI_EXP_LNKCAP field
> update. The later would lead to the fully coherent maximum link width
> setup method in accordance with the DW PCIe hardware manual.

I got it.

Best regards,
Yoshihiro Shimoda

> -Serge(y)
> 
> >
> > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > ---
> >  drivers/pci/controller/dwc/pcie-designware.c | 40 +++++++-------------
> >  1 file changed, 13 insertions(+), 27 deletions(-)
> >
> > diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
> > index 68aefbbcd68c..5dc423dd2f21 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware.c
> > @@ -721,28 +721,40 @@ static void dw_pcie_link_set_max_speed(struct dw_pcie *pci, u32 link_gen)
> >
> >  static void dw_pcie_link_set_max_link_width(struct dw_pcie *pci, u32 num_lanes)
> >  {
> > -	u32 lwsc;
> > +	u32 lwsc, plc;
> >
> >  	if (!num_lanes)
> >  		return;
> >
> > +	/* Set the number of lanes */
> > +	plc = dw_pcie_readl_dbi(pci, PCIE_PORT_LINK_CONTROL);
> > +	plc &= ~PORT_LINK_MODE_MASK;
> > +
> >  	/* Set link width speed control register */
> >  	lwsc = dw_pcie_readl_dbi(pci, PCIE_LINK_WIDTH_SPEED_CONTROL);
> >  	lwsc &= ~PORT_LOGIC_LINK_WIDTH_MASK;
> >  	switch (num_lanes) {
> >  	case 1:
> > +		plc |= PORT_LINK_MODE_1_LANES;
> >  		lwsc |= PORT_LOGIC_LINK_WIDTH_1_LANES;
> >  		break;
> >  	case 2:
> > +		plc |= PORT_LINK_MODE_2_LANES;
> >  		lwsc |= PORT_LOGIC_LINK_WIDTH_2_LANES;
> >  		break;
> >  	case 4:
> > +		plc |= PORT_LINK_MODE_4_LANES;
> >  		lwsc |= PORT_LOGIC_LINK_WIDTH_4_LANES;
> >  		break;
> >  	case 8:
> > +		plc |= PORT_LINK_MODE_8_LANES;
> >  		lwsc |= PORT_LOGIC_LINK_WIDTH_8_LANES;
> >  		break;
> > +	default:
> > +		dev_err(pci->dev, "num-lanes %u: invalid value\n", num_lanes);
> > +		return;
> >  	}
> > +	dw_pcie_writel_dbi(pci, PCIE_PORT_LINK_CONTROL, plc);
> >  	dw_pcie_writel_dbi(pci, PCIE_LINK_WIDTH_SPEED_CONTROL, lwsc);
> >  }
> >
> > @@ -1027,31 +1039,5 @@ void dw_pcie_setup(struct dw_pcie *pci)
> >  	val |= PORT_LINK_DLL_LINK_EN;
> >  	dw_pcie_writel_dbi(pci, PCIE_PORT_LINK_CONTROL, val);
> >
> > -	if (!pci->num_lanes) {
> > -		dev_dbg(pci->dev, "Using h/w default number of lanes\n");
> > -		return;
> > -	}
> > -
> > -	/* Set the number of lanes */
> > -	val &= ~PORT_LINK_MODE_MASK;
> > -	switch (pci->num_lanes) {
> > -	case 1:
> > -		val |= PORT_LINK_MODE_1_LANES;
> > -		break;
> > -	case 2:
> > -		val |= PORT_LINK_MODE_2_LANES;
> > -		break;
> > -	case 4:
> > -		val |= PORT_LINK_MODE_4_LANES;
> > -		break;
> > -	case 8:
> > -		val |= PORT_LINK_MODE_8_LANES;
> > -		break;
> > -	default:
> > -		dev_err(pci->dev, "num-lanes %u: invalid value\n", pci->num_lanes);
> > -		return;
> > -	}
> > -	dw_pcie_writel_dbi(pci, PCIE_PORT_LINK_CONTROL, val);
> > -
> >  	dw_pcie_link_set_max_link_width(pci, pci->num_lanes);
> >  }
> > --
> > 2.25.1
> >

RE: [PATCH v16 11/22] PCI: dwc: Add dw_pcie_link_set_max_cap_width()

[Yoshihiro Shimoda]

Hello Serge,

> From: Serge Semin, Sent: Monday, June 5, 2023 7:59 PM
> 
> On Wed, May 10, 2023 at 03:22:23PM +0900, Yoshihiro Shimoda wrote:
> > Add dw_pcie_link_set_max_cap_width() to set PCI_EXP_LNKCAP_MLW.
> > In accordance with the DW PCIe RC/EP HW manuals [1,2,3,...] aside with
> > the PORT_LINK_CTRL_OFF.LINK_CAPABLE and GEN2_CTRL_OFF.NUM_OF_LANES[8:0]
> > field there is another one which needs to be updated. It's
> > LINK_CAPABILITIES_REG.PCIE_CAP_MAX_LINK_WIDTH. If it isn't done at
> > the very least the maximum link-width capability CSR won't expose
> > the actual maximum capability.
> >
> > [1] DesignWare Cores PCI Express Controller Databook - DWC PCIe Root Port,
> >     Version 4.60a, March 2015, p.1032
> > [2] DesignWare Cores PCI Express Controller Databook - DWC PCIe Root Port,
> >     Version 4.70a, March 2016, p.1065
> > [3] DesignWare Cores PCI Express Controller Databook - DWC PCIe Root Port,
> >     Version 4.90a, March 2016, p.1057
> > ...
> > [X] DesignWare Cores PCI Express Controller Databook - DWC PCIe Endpoint,
> >       Version 5.40a, March 2019, p.1396
> > [X+1] DesignWare Cores PCI Express Controller Databook - DWC PCIe Root Port,
> >       Version 5.40a, March 2019, p.1266
> >
> > The commit description is suggested by Serge Semin.
> >
> > Suggested-by: Serge Semin <fancer.lancer@gmail.com>
> > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > ---
> >  drivers/pci/controller/dwc/pcie-designware.c | 16 ++++++++++++++++
> >  1 file changed, 16 insertions(+)
> >
> > diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
> > index 5dc423dd2f21..8b2978c6eb23 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware.c
> > @@ -758,6 +758,21 @@ static void dw_pcie_link_set_max_link_width(struct dw_pcie *pci, u32 num_lanes)
> >  	dw_pcie_writel_dbi(pci, PCIE_LINK_WIDTH_SPEED_CONTROL, lwsc);
> >  }
> >
> > +static void dw_pcie_link_set_max_cap_width(struct dw_pcie *pci, int num_lanes)
> > +{
> 
> > +	u32 val;
> > +	u8 cap;
> > +
> > +	if (!num_lanes)
> > +		return;
> > +
> > +	cap = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP);
> > +	val = dw_pcie_readl_dbi(pci, cap + PCI_EXP_LNKCAP);
> > +	val &= ~PCI_EXP_LNKCAP_MLW;
> > +	val |= num_lanes << PCI_EXP_LNKSTA_NLW_SHIFT;
> > +	dw_pcie_writel_dbi(pci, cap + PCI_EXP_LNKCAP, val);
> 
> Once again. Please move this code to the
> dw_pcie_link_set_max_link_width() method as I already asked here:
<snip URL>

I'm sorry for misunderstanding your previous comment...

> There is no point in creating a separate method for the action which
> is implied by the already defined and incomplete
> dw_pcie_link_set_max_link_width() function.
> 
> Also as we already agreed please replace the hard-coded bitwise shift
> operation with the FIELD_PREP(PCI_EXP_LNKCAP_MLW, num_lanes)
> statement.

I got it.

Best regards,
Yoshihiro Shimoda

> -Serge(y)
> 
> > +}
> > +
> >  void dw_pcie_iatu_detect(struct dw_pcie *pci)
> >  {
> >  	int max_region, ob, ib;
> > @@ -1040,4 +1055,5 @@ void dw_pcie_setup(struct dw_pcie *pci)
> >  	dw_pcie_writel_dbi(pci, PCIE_PORT_LINK_CONTROL, val);
> >
> >  	dw_pcie_link_set_max_link_width(pci, pci->num_lanes);
> > +	dw_pcie_link_set_max_cap_width(pci, pci->num_lanes);
> >  }
> > --
> > 2.25.1
> >

RE: [PATCH v16 13/22] PCI: dwc: Add EDMA_UNROLL capability flag

[Yoshihiro Shimoda]

Hello Serge,

> From: Serge Semin, Sent: Monday, June 5, 2023 8:16 PM
> 
> On Wed, May 10, 2023 at 03:22:25PM +0900, Yoshihiro Shimoda wrote:
> > Renesas R-Car Gen4 PCIe controllers have an unexpected register value on
> > the dbi+0x97b register. So, add a new capability flag "EDMA_UNROLL"
> > which would force the unrolled eDMA mapping for the problematic
> > device,
> 
> > as suggested by Serge Semin.
> 
> Drop this. Suggested-by tag already means that.

I got it.

> >
> > Suggested-by: Serge Semin <fancer.lancer@gmail.com>
> > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > ---
> >  drivers/pci/controller/dwc/pcie-designware.c | 8 +++++++-
> >  drivers/pci/controller/dwc/pcie-designware.h | 5 +++--
> >  2 files changed, 10 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
> > index 8b2978c6eb23..e405bfae0191 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware.c
> > @@ -881,8 +881,14 @@ static int dw_pcie_edma_find_chip(struct dw_pcie *pci)
> >  	 * Indirect eDMA CSRs access has been completely removed since v5.40a
> >  	 * thus no space is now reserved for the eDMA channels viewport and
> >  	 * former DMA CTRL register is no longer fixed to FFs.
> > +	 *
> > +	 * Note that Renesas R-Car S4-8's PCIe controllers for unknown reason
> 
> > +	 * may have zeros in the eDMA CTRL register even though the HW-manual
> 
> s/may have/have

I'll fix it.

> (your comment is about a particular device which for sure has the
> denoted problem.)

I understood it.

> Other than that the change looks good. Thanks!
> Reviewed-by: Serge Semin <fancer.lancer@gmail.com>

Thank you very much for your review!

Best regards,
Yoshihiro Shimoda

> -Serge(y)
> 
> > +	 * explicitly states there must FFs if the unrolled mapping is enabled.
> > +	 * For such cases the low-level drivers are supposed to manually
> > +	 * activate the unrolled mapping to bypass the auto-detection procedure.
> >  	 */
> > -	if (dw_pcie_ver_is_ge(pci, 540A))
> > +	if (dw_pcie_ver_is_ge(pci, 540A) || dw_pcie_cap_is(pci, EDMA_UNROLL))
> >  		val = 0xFFFFFFFF;
> >  	else
> >  		val = dw_pcie_readl_dbi(pci, PCIE_DMA_VIEWPORT_BASE + PCIE_DMA_CTRL);
> > diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
> > index 06e044e2163a..2639206b4c18 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware.h
> > +++ b/drivers/pci/controller/dwc/pcie-designware.h
> > @@ -54,8 +54,9 @@
> >
> >  /* DWC PCIe controller capabilities */
> >  #define DW_PCIE_CAP_REQ_RES		0
> > -#define DW_PCIE_CAP_IATU_UNROLL		1
> > -#define DW_PCIE_CAP_CDM_CHECK		2
> > +#define DW_PCIE_CAP_EDMA_UNROLL		1
> > +#define DW_PCIE_CAP_IATU_UNROLL		2
> > +#define DW_PCIE_CAP_CDM_CHECK		3
> >
> >  #define dw_pcie_cap_is(_pci, _cap) \
> >  	test_bit(DW_PCIE_CAP_ ## _cap, &(_pci)->caps)
> > --
> > 2.25.1
> >

RE: [PATCH v16 19/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support

[Yoshihiro Shimoda]

Hello Serge,

> From: Serge Semin, Sent: Monday, June 5, 2023 11:39 PM
> 
> On Wed, May 10, 2023 at 03:22:31PM +0900, Yoshihiro Shimoda wrote:
> > Add R-Car Gen4 PCIe Host support. This controller is based on
> > Synopsys DesignWare PCIe, but this controller has vendor-specific
> > registers so that requires initialization code like mode setting
> > and retraining and so on.
> >
> > To reduce code delta, adds some helper functions which are used by
> > both the host driver and the endpoint driver (which is added
> > immediately afterwards) into a separate file.
> >
> > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > ---
> >  drivers/pci/controller/dwc/Kconfig            |   9 +
> >  drivers/pci/controller/dwc/Makefile           |   2 +
> >  .../pci/controller/dwc/pcie-rcar-gen4-host.c  | 141 +++++++++++++
> >  drivers/pci/controller/dwc/pcie-rcar-gen4.c   | 190 ++++++++++++++++++
> >  drivers/pci/controller/dwc/pcie-rcar-gen4.h   |  46 +++++
> >  5 files changed, 388 insertions(+)
> >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.c
> >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.h
> >
> > diff --git a/drivers/pci/controller/dwc/Kconfig b/drivers/pci/controller/dwc/Kconfig
> > index ab96da43e0c2..64d4d37bc891 100644
> > --- a/drivers/pci/controller/dwc/Kconfig
> > +++ b/drivers/pci/controller/dwc/Kconfig
> > @@ -415,4 +415,13 @@ config PCIE_VISCONTI_HOST
> >  	  Say Y here if you want PCIe controller support on Toshiba Visconti SoC.
> >  	  This driver supports TMPV7708 SoC.
> >
> > +config PCIE_RCAR_GEN4
> > +	tristate "Renesas R-Car Gen4 PCIe Host controller"
> > +	depends on ARCH_RENESAS || COMPILE_TEST
> > +	depends on PCI_MSI
> > +	select PCIE_DW_HOST
> > +	help
> > +	  Say Y here if you want PCIe host controller support on R-Car Gen4 SoCs.
> > +	  This uses the DesignWare core.
> > +
> >  endmenu
> > diff --git a/drivers/pci/controller/dwc/Makefile b/drivers/pci/controller/dwc/Makefile
> > index bf5c311875a1..486cf706b53d 100644
> > --- a/drivers/pci/controller/dwc/Makefile
> > +++ b/drivers/pci/controller/dwc/Makefile
> > @@ -26,6 +26,8 @@ obj-$(CONFIG_PCIE_TEGRA194) += pcie-tegra194.o
> >  obj-$(CONFIG_PCIE_UNIPHIER) += pcie-uniphier.o
> >  obj-$(CONFIG_PCIE_UNIPHIER_EP) += pcie-uniphier-ep.o
> >  obj-$(CONFIG_PCIE_VISCONTI_HOST) += pcie-visconti.o
> > +pcie-rcar-gen4-host-drv-objs := pcie-rcar-gen4.o pcie-rcar-gen4-host.o
> > +obj-$(CONFIG_PCIE_RCAR_GEN4) += pcie-rcar-gen4-host-drv.o
> >
> >  # The following drivers are for devices that use the generic ACPI
> >  # pci_root.c driver but don't support standard ECAM config access.
> > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > new file mode 100644
> > index 000000000000..df7d80f1874f
> > --- /dev/null
> > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > @@ -0,0 +1,141 @@
> > +// SPDX-License-Identifier: GPL-2.0-only
> > +/*
> > + * PCIe host controller driver for Renesas R-Car Gen4 Series SoCs
> > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > + */
> > +
> > +#include <linux/delay.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/module.h>
> > +#include <linux/of_device.h>
> > +#include <linux/pci.h>
> > +#include <linux/platform_device.h>
> > +
> > +#include "pcie-rcar-gen4.h"
> > +#include "pcie-designware.h"
> > +
> > +static int rcar_gen4_pcie_host_init(struct dw_pcie_rp *pp)
> > +{
> > +	struct dw_pcie *dw = to_dw_pcie_from_pp(pp);
> > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > +	int ret;
> > +	u32 val;
> > +
> > +	gpiod_set_value_cansleep(dw->pe_rst, 1);
> > +
> > +	ret = rcar_gen4_pcie_set_device_type(rcar, true, dw->num_lanes);
> > +	if (ret < 0)
> > +		return ret;
> > +
> 
> > +	dw_pcie_dbi_ro_wr_en(dw);
> 
> Are you sure dw_pcie_dbi_ro_wr_en() and dw_pcie_dbi_ro_wr_dis() are
> needed? In accordance with the DW PCIe Dual-mode HW manual the BARx
> registers are W-only over the DBI2 map with no need in setting the
> DBI_RO_WR_EN flag.
> 
> Please check that on your hardware.

You're correct. They are not needed. So, I'll drop this on v17.

> > +
> > +	/*
> > +	 * According to the section 3.5.7.2 "RC Mode" in DWC PCIe Dual Mode
> > +	 * Rev.5.20a, we should disable two BARs to avoid unnecessary memory
> > +	 * assignment during device enumeration.
> > +	 */
> > +	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_0, 0x0);
> > +	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_1, 0x0);
> > +
> 
> > +	dw_pcie_dbi_ro_wr_dis(dw);
> 
> ditto

I'll drop this too.

> > +
> > +	if (IS_ENABLED(CONFIG_PCI_MSI)) {
> > +		/* Enable MSI interrupt signal */
> > +		val = readl(rcar->base + PCIEINTSTS0EN);
> > +		val |= MSI_CTRL_INT;
> > +		writel(val, rcar->base + PCIEINTSTS0EN);
> > +	}
> > +
> > +	msleep(100);	/* pe_rst requires 100msec delay */
> > +
> > +	gpiod_set_value_cansleep(dw->pe_rst, 0);
> > +
> > +	return 0;
> > +}
> > +
> > +static const struct dw_pcie_host_ops rcar_gen4_pcie_host_ops = {
> > +	.host_init = rcar_gen4_pcie_host_init,
> > +};
> > +
> > +static int rcar_gen4_add_dw_pcie_rp(struct rcar_gen4_pcie *rcar,
> > +				   struct platform_device *pdev)
> > +{
> > +	struct dw_pcie *dw = &rcar->dw;
> > +	struct dw_pcie_rp *pp = &dw->pp;
> > +
> > +	pp->num_vectors = MAX_MSI_IRQS;
> > +	pp->ops = &rcar_gen4_pcie_host_ops;
> > +	dw_pcie_cap_set(dw, REQ_RES);
> > +
> > +	return dw_pcie_host_init(pp);
> > +}
> > +
> > +static void rcar_gen4_remove_dw_pcie_rp(struct rcar_gen4_pcie *rcar)
> > +{
> > +	dw_pcie_host_deinit(&rcar->dw.pp);
> > +	gpiod_set_value_cansleep(rcar->dw.pe_rst, 1);
> > +}
> > +
> > +static int rcar_gen4_pcie_probe(struct platform_device *pdev)
> > +{
> > +	struct device *dev = &pdev->dev;
> > +	struct rcar_gen4_pcie *rcar;
> > +	int err;
> > +
> > +	rcar = rcar_gen4_pcie_devm_alloc(dev);
> > +	if (!rcar)
> > +		return -ENOMEM;
> > +
> > +	err = rcar_gen4_pcie_get_resources(rcar, pdev);
> > +	if (err < 0) {
> > +		dev_err(dev, "Failed to request resource: %d\n", err);
> > +		return err;
> > +	}
> > +
> > +	platform_set_drvdata(pdev, rcar);
> > +
> > +	err = rcar_gen4_pcie_prepare(rcar);
> > +	if (err < 0)
> > +		return err;
> > +
> > +	rcar->needs_retrain = true;
> > +	err = rcar_gen4_add_dw_pcie_rp(rcar, pdev);
> > +	if (err < 0)
> > +		goto err_add;
> > +
> > +	return 0;
> > +
> > +err_add:
> > +	rcar_gen4_pcie_unprepare(rcar);
> > +
> > +	return err;
> > +}
> > +
> > +static int rcar_gen4_pcie_remove(struct platform_device *pdev)
> > +{
> > +	struct rcar_gen4_pcie *rcar = platform_get_drvdata(pdev);
> > +
> > +	rcar_gen4_remove_dw_pcie_rp(rcar);
> > +	rcar_gen4_pcie_unprepare(rcar);
> > +
> > +	return 0;
> > +}
> > +
> > +static const struct of_device_id rcar_gen4_pcie_of_match[] = {
> > +	{ .compatible = "renesas,rcar-gen4-pcie", },
> > +	{},
> > +};
> > +
> > +static struct platform_driver rcar_gen4_pcie_driver = {
> > +	.driver = {
> > +		.name = "pcie-rcar-gen4",
> > +		.of_match_table = rcar_gen4_pcie_of_match,
> > +		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
> > +	},
> > +	.probe = rcar_gen4_pcie_probe,
> > +	.remove = rcar_gen4_pcie_remove,
> > +};
> > +module_platform_driver(rcar_gen4_pcie_driver);
> > +
> > +MODULE_DESCRIPTION("Renesas R-Car Gen4 PCIe host controller driver");
> > +MODULE_LICENSE("GPL");
> > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.c b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > new file mode 100644
> > index 000000000000..35923fda8ed5
> > --- /dev/null
> > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > @@ -0,0 +1,190 @@
> > +// SPDX-License-Identifier: GPL-2.0-only
> > +/*
> > + * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
> > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > + */
> > +
> > +#include <linux/delay.h>
> > +#include <linux/io.h>
> > +#include <linux/of_device.h>
> > +#include <linux/pci.h>
> > +#include <linux/pm_runtime.h>
> > +#include <linux/reset.h>
> > +
> > +#include "pcie-rcar-gen4.h"
> > +#include "pcie-designware.h"
> > +
> > +/* Renesas-specific */
> > +#define PCIERSTCTRL1		0x0014
> > +#define  APP_HOLD_PHY_RST	BIT(16)
> > +#define  APP_LTSSM_ENABLE	BIT(0)
> > +
> > +#define RETRAIN_MAX_CHECK	10
> > +#define RETRAIN_MAX_RETRIES	10
> > +
> > +static void rcar_gen4_pcie_ltssm_enable(struct rcar_gen4_pcie *rcar,
> > +					bool enable)
> > +{
> > +	u32 val;
> > +
> > +	val = readl(rcar->base + PCIERSTCTRL1);
> > +	if (enable) {
> > +		val |= APP_LTSSM_ENABLE;
> 
> > +		val &= ~APP_HOLD_PHY_RST;
> 
> What about moving the APP_HOLD_PHY_RST de-assertion to the
> rcar_gen4_pcie_set_device_type() method? In accordance with the
> "3.1 Initialization" chapter it's supposed to be done before
> performing the DBI programming and activating the link training.

IIUC, the "3.1 Initialization" said app_hold_phy_rst = 1 before
performing the DBI programming. So, it is assertion. Also, the SoC
documentation described the initializing procedure as the follows:
 app_ltssm_enable = 1
 app_hold_phy_rst = 0
So, I would like to keep them in the function.

> > +	} else {
> > +		val &= ~APP_LTSSM_ENABLE;
> > +		val |= APP_HOLD_PHY_RST;
> > +	}
> > +	writel(val, rcar->base + PCIERSTCTRL1);
> > +}
> > +
> > +static bool rcar_gen4_pcie_check_retrain_link(struct dw_pcie *dw)
> > +{
> > +	u8 offset = dw_pcie_find_capability(dw, PCI_CAP_ID_EXP);
> > +	u32 lnkcap = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCAP);
> > +	u32 lnkctl = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCTL);
> > +	u16 lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> > +	int i;
> > +
> 
> > +	if ((lnksta & PCI_EXP_LNKSTA_CLS) == (lnkcap & PCI_EXP_LNKCAP_SLS))
> > +		return true;
> > +
> > +	lnkctl |= PCI_EXP_LNKCTL_RL;
> > +	dw_pcie_writel_dbi(dw, offset + PCI_EXP_LNKCTL, lnkctl);
> > +
> > +	for (i = 0; i < RETRAIN_MAX_CHECK; i++) {
> > +		lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> > +		if (lnksta & PCI_EXP_LNKSTA_LT)
> > +			return true;
> > +		usleep_range(1000, 1100);
> > +	}
> 
> I'll ask one more time because you didn't respond to my previous note
> about this.

I'm sorry. I completely overlooked the previous note.

> Are you sure that this is needed? Did you try
> the approach described in "3.13 Gen2/3/4/5 Speed Modes" with
> de-asserting/asserting the GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag?

I tried this setting, but it doesn't work. I'll investigate this setting more.

> I keep asking because the same problem we used to have on our hardware
> until we found out that the DIRECT_SPEED_CHANGE flag helped to train
> the link right to the speed specified in the capabilities.
> 
> So here is what presumably you'll need to do (based on the
> "3.1 Initialization" and "3.13 Gen2/3/4/5 Speed Modes" chapters of
> the DW PCIe DM hw-manual):
> 1. Make sure the controller is in the power-down/reset state.
> 2. Select device_type (EP or RP).
> 3. De-assert the controller reset.
> 4. Clear PHY-reset flag in the app registers.
> 5. Perform some controller initializations.
> 6. Enable LTSSM to start link training.
> 7. Set GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag one more time.
> 
> 1-4 are supposed to be done in rcar_gen4_pcie_host_init().
> 5 is performed in the framework of the DW PCIe core driver.
> 6-7 should be done in rcar_gen4_pcie_start_link().
> 
> Note 1. GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag is already set on stage
> 5 in the framework of the dw_pcie_setup_rc() method. But in our case
> it only caused having the Gen.2 link speed. Adding stage 7 helped to
> get stable Gen.3 link. So please try the denoted approach. If it works
> what about adding stage 7 twice in order to get Gen.4 speed?
> (waiting for the DIRECT_SPEED_CHANGE flag being auto-cleared and then
> set it up again?)
> 
> Note 2. GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag is defined as
> PCIE_LINK_WIDTH_SPEED_CONTROL.PORT_LOGIC_SPEED_CHANGE macros in the DW
> PCIe core driver.
> 
> Note 3. If what is suggested above works well then you won't need to
> have the heavy rcar_gen4_pcie_check_retrain_link() method in the way
> you have it defined.

Thank you very much for your comments!

> > +
> > +	return false;
> > +}
> > +
> > +static int rcar_gen4_pcie_link_up(struct dw_pcie *dw)
> > +{
> > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > +	u32 val, mask;
> > +
> > +	val = readl(rcar->base + PCIEINTSTS0);
> > +	mask = RDLH_LINK_UP | SMLH_LINK_UP;
> > +
> > +	return (val & mask) == mask;
> > +}
> > +
> > +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > +{
> > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > +	int i;
> > +
> > +	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > +
> > +	/*
> > +	 * Require retraining here. Otherwise RDLH_LINK_UP of PCIEINTSTS0 which
> > +	 * is this controller specific register may not be set.
> > +	 */
> > +	if (rcar->needs_retrain) {
> > +		for (i = 0; i < RETRAIN_MAX_RETRIES; i++) {
> > +			if (rcar_gen4_pcie_check_retrain_link(dw))
> > +				return 0;
> > +			msleep(100);
> > +		}
> > +
> > +		return -ETIMEDOUT;	/* Failed */
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static void rcar_gen4_pcie_stop_link(struct dw_pcie *dw)
> > +{
> > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > +
> > +	rcar_gen4_pcie_ltssm_enable(rcar, false);
> > +}
> > +
> 
> > +int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
> > +				   int num_lanes)
> 
> 1. Number of lanes is already defined in the rcar_gen4_pcie.dw.num_lanes field.
> What about using it from there instead of passing it as an argument?
> 2. DW PCIe core driver has a very handy enum defined:
> dw_pcie_device_mode. It describes the controller modes (End-point,
> Root port, etc). What about adding the mode field right to the
> rcar_gen4_pcie structure and initializing it in someplace in probe() ?
> 3. Based on the function semantic it's better to be named as something
> like rcar_gen4_pcie_init_device() or even rcar_gen4_pcie_basic_init().

Thank you for your comments! I'll modify the function.

> 
> > +{
> > +	u32 val;
> > +
> 
> > +	/* Note: Assume the rcar->rst which is Cold-reset is asserted here */
> 
> What about directly asserting it here then? In accordance with the DW
> PCIe DM manual the "device_type" input must be set before the DM
> controller is powered up (basically un-reset). What if the controller
> reset is already de-asserted, but you are going to changes its mode?
> In that case the mode won't be changed and you'll end up with
> unpredictable results.

Thank you for your comment. We should add asserting it here as you mentioned.

> > +	val = readl(rcar->base + PCIEMSR0);
> > +	if (rc)
> > +		val |= DEVICE_TYPE_RC;
> > +	else
> > +		val |= DEVICE_TYPE_EP;
> > +
> > +	if (num_lanes < 4)
> > +		val |= BIFUR_MOD_SET_ON;
> > +
> > +	writel(val, rcar->base + PCIEMSR0);
> > +
> > +	return reset_control_deassert(rcar->rst);
> > +}
> > +
> > +int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *rcar)
> > +{
> > +	struct device *dev = rcar->dw.dev;
> > +	int err;
> > +
> > +	pm_runtime_enable(dev);
> > +	err = pm_runtime_resume_and_get(dev);
> > +	if (err < 0) {
> > +		dev_err(dev, "Failed to resume/get Runtime PM\n");
> > +		pm_runtime_disable(dev);
> > +	}
> > +
> > +	return err;
> > +}
> > +
> > +void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *rcar)
> > +{
> > +	struct device *dev = rcar->dw.dev;
> > +
> > +	if (!reset_control_status(rcar->rst))
> > +		reset_control_assert(rcar->rst);
> > +	pm_runtime_put(dev);
> > +	pm_runtime_disable(dev);
> > +}
> > +
> > +int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
> > +				 struct platform_device *pdev)
> > +{
> > +	struct device *dev = rcar->dw.dev;
> > +
> > +	/* Renesas-specific registers */
> > +	rcar->base = devm_platform_ioremap_resource_byname(pdev, "app");
> > +	if (IS_ERR(rcar->base))
> > +		return PTR_ERR(rcar->base);
> > +
> 
> > +	rcar->rst = devm_reset_control_get(dev, NULL);
> > +	if (IS_ERR(rcar->rst)) {
> > +		dev_err(dev, "Failed to get Cold-reset\n");
> 
> So AFAICS your platform is equipped with the DWC_pcie_clkrst.v module.
> Thus all the resets are appropriately cleared by a single flag:
> power_up_rst_n. What about using the named reset in this case with the
> "pwr" name? Thus you'll be able to drop the manual
> devm_reset_control_get() invocation and instead use the reset-resources
> requested in the framework of the generic dw_pcie_get_resources()
> method? Note you'll need to move the dw_pcie_cap_set(dw, REQ_RES);
> statement to rcar_gen4_pcie_devm_alloc() then and drop the
> rcar_gen4_pcie.rst field afterwords.

Thank you for your suggestion! Using "pwr" can work on my environment.

> By the way I don't see you requesting and enabling the reference
> clock in your driver but the bindings imply the clock source. How
> come?

For now, I used gpio-hog to enable the reference clock. But, it seem
I should use "ref" clock for it. So, I'll fix it too.

> > +		return PTR_ERR(rcar->rst);
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static const struct dw_pcie_ops dw_pcie_ops = {
> > +	.start_link = rcar_gen4_pcie_start_link,
> > +	.stop_link = rcar_gen4_pcie_stop_link,
> > +	.link_up = rcar_gen4_pcie_link_up,
> > +};
> > +
> > +struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev)
> > +{
> > +	struct rcar_gen4_pcie *rcar;
> > +
> > +	rcar = devm_kzalloc(dev, sizeof(*rcar), GFP_KERNEL);
> > +	if (!rcar)
> > +		return NULL;
> > +
> > +	rcar->dw.dev = dev;
> > +	rcar->dw.ops = &dw_pcie_ops;
> > +	dw_pcie_cap_set(&rcar->dw, EDMA_UNROLL);
> > +
> > +	return rcar;
> > +}
> > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.h b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > new file mode 100644
> > index 000000000000..fec3f18609f4
> > --- /dev/null
> > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > @@ -0,0 +1,46 @@
> > +/* SPDX-License-Identifier: GPL-2.0-only */
> > +/*
> > + * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
> > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > + */
> > +
> > +#ifndef _PCIE_RCAR_GEN4_H_
> > +#define _PCIE_RCAR_GEN4_H_
> > +
> > +#include <linux/io.h>
> > +#include <linux/pci.h>
> > +#include <linux/reset.h>
> > +
> > +#include "pcie-designware.h"
> > +
> > +/* Renesas-specific */
> > +#define PCIEMSR0		0x0000
> > +#define  BIFUR_MOD_SET_ON	BIT(0)
> > +#define  DEVICE_TYPE_EP		0
> > +#define  DEVICE_TYPE_RC		BIT(4)
> > +
> > +#define PCIEINTSTS0		0x0084
> > +#define PCIEINTSTS0EN		0x0310
> > +#define  MSI_CTRL_INT		BIT(26)
> > +#define  SMLH_LINK_UP		BIT(7)
> > +#define  RDLH_LINK_UP		BIT(6)
> > +#define PCIEDMAINTSTSEN		0x0314
> > +#define  PCIEDMAINTSTSEN_INIT	GENMASK(15, 0)
> > +
> 
> > +struct rcar_gen4_pcie {
> 
> As I mentioned above this structure can be extended with the enum
> dw_pcie_device_mode field thus dropping the boolean argument from the
> rcar_gen4_pcie_set_device_type() method.

I got it. I'll fix this.

> > +	struct dw_pcie		dw;
> 
> As I already mentioned above the dw.num_lanes could be used instead of
> passing it as the rcar_gen4_pcie_set_device_type() argument.

I'll fix this too.

Best regards,
Yoshihiro Shimoda

> -Serge(y)
> 
> > +	void __iomem		*base;
> > +	struct reset_control	*rst;
> > +	bool			needs_retrain;
> > +};
> > +#define to_rcar_gen4_pcie(x)	dev_get_drvdata((x)->dev)
> > +
> > +int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
> > +				   int num_lanes);
> > +int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *pcie);
> > +void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *pcie);
> > +int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
> > +				 struct platform_device *pdev);
> > +struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev);
> > +
> > +#endif /* _PCIE_RCAR_GEN4_H_ */
> > --
> > 2.25.1
> >

Re: [PATCH v16 19/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support

[Serge Semin]

On Wed, Jun 07, 2023 at 02:59:20AM +0000, Yoshihiro Shimoda wrote:
> Hello Serge,
> 
> > From: Serge Semin, Sent: Monday, June 5, 2023 11:39 PM
> > 
> > On Wed, May 10, 2023 at 03:22:31PM +0900, Yoshihiro Shimoda wrote:
> > > Add R-Car Gen4 PCIe Host support. This controller is based on
> > > Synopsys DesignWare PCIe, but this controller has vendor-specific
> > > registers so that requires initialization code like mode setting
> > > and retraining and so on.
> > >
> > > To reduce code delta, adds some helper functions which are used by
> > > both the host driver and the endpoint driver (which is added
> > > immediately afterwards) into a separate file.
> > >
> > > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > > ---
> > >  drivers/pci/controller/dwc/Kconfig            |   9 +
> > >  drivers/pci/controller/dwc/Makefile           |   2 +
> > >  .../pci/controller/dwc/pcie-rcar-gen4-host.c  | 141 +++++++++++++
> > >  drivers/pci/controller/dwc/pcie-rcar-gen4.c   | 190 ++++++++++++++++++
> > >  drivers/pci/controller/dwc/pcie-rcar-gen4.h   |  46 +++++
> > >  5 files changed, 388 insertions(+)
> > >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > >
> > > diff --git a/drivers/pci/controller/dwc/Kconfig b/drivers/pci/controller/dwc/Kconfig
> > > index ab96da43e0c2..64d4d37bc891 100644
> > > --- a/drivers/pci/controller/dwc/Kconfig
> > > +++ b/drivers/pci/controller/dwc/Kconfig
> > > @@ -415,4 +415,13 @@ config PCIE_VISCONTI_HOST
> > >  	  Say Y here if you want PCIe controller support on Toshiba Visconti SoC.
> > >  	  This driver supports TMPV7708 SoC.
> > >
> > > +config PCIE_RCAR_GEN4
> > > +	tristate "Renesas R-Car Gen4 PCIe Host controller"
> > > +	depends on ARCH_RENESAS || COMPILE_TEST
> > > +	depends on PCI_MSI
> > > +	select PCIE_DW_HOST
> > > +	help
> > > +	  Say Y here if you want PCIe host controller support on R-Car Gen4 SoCs.
> > > +	  This uses the DesignWare core.
> > > +
> > >  endmenu
> > > diff --git a/drivers/pci/controller/dwc/Makefile b/drivers/pci/controller/dwc/Makefile
> > > index bf5c311875a1..486cf706b53d 100644
> > > --- a/drivers/pci/controller/dwc/Makefile
> > > +++ b/drivers/pci/controller/dwc/Makefile
> > > @@ -26,6 +26,8 @@ obj-$(CONFIG_PCIE_TEGRA194) += pcie-tegra194.o
> > >  obj-$(CONFIG_PCIE_UNIPHIER) += pcie-uniphier.o
> > >  obj-$(CONFIG_PCIE_UNIPHIER_EP) += pcie-uniphier-ep.o
> > >  obj-$(CONFIG_PCIE_VISCONTI_HOST) += pcie-visconti.o
> > > +pcie-rcar-gen4-host-drv-objs := pcie-rcar-gen4.o pcie-rcar-gen4-host.o
> > > +obj-$(CONFIG_PCIE_RCAR_GEN4) += pcie-rcar-gen4-host-drv.o
> > >
> > >  # The following drivers are for devices that use the generic ACPI
> > >  # pci_root.c driver but don't support standard ECAM config access.
> > > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > > new file mode 100644
> > > index 000000000000..df7d80f1874f
> > > --- /dev/null
> > > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > > @@ -0,0 +1,141 @@
> > > +// SPDX-License-Identifier: GPL-2.0-only
> > > +/*
> > > + * PCIe host controller driver for Renesas R-Car Gen4 Series SoCs
> > > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > > + */
> > > +
> > > +#include <linux/delay.h>
> > > +#include <linux/interrupt.h>
> > > +#include <linux/module.h>
> > > +#include <linux/of_device.h>
> > > +#include <linux/pci.h>
> > > +#include <linux/platform_device.h>
> > > +
> > > +#include "pcie-rcar-gen4.h"
> > > +#include "pcie-designware.h"
> > > +
> > > +static int rcar_gen4_pcie_host_init(struct dw_pcie_rp *pp)
> > > +{
> > > +	struct dw_pcie *dw = to_dw_pcie_from_pp(pp);
> > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > +	int ret;
> > > +	u32 val;
> > > +
> > > +	gpiod_set_value_cansleep(dw->pe_rst, 1);
> > > +
> > > +	ret = rcar_gen4_pcie_set_device_type(rcar, true, dw->num_lanes);
> > > +	if (ret < 0)
> > > +		return ret;
> > > +
> > 
> > > +	dw_pcie_dbi_ro_wr_en(dw);
> > 
> > Are you sure dw_pcie_dbi_ro_wr_en() and dw_pcie_dbi_ro_wr_dis() are
> > needed? In accordance with the DW PCIe Dual-mode HW manual the BARx
> > registers are W-only over the DBI2 map with no need in setting the
> > DBI_RO_WR_EN flag.
> > 
> > Please check that on your hardware.
> 
> You're correct. They are not needed. So, I'll drop this on v17.
> 
> > > +
> > > +	/*
> > > +	 * According to the section 3.5.7.2 "RC Mode" in DWC PCIe Dual Mode
> > > +	 * Rev.5.20a, we should disable two BARs to avoid unnecessary memory
> > > +	 * assignment during device enumeration.
> > > +	 */
> > > +	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_0, 0x0);
> > > +	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_1, 0x0);
> > > +
> > 
> > > +	dw_pcie_dbi_ro_wr_dis(dw);
> > 
> > ditto
> 
> I'll drop this too.
> 
> > > +
> > > +	if (IS_ENABLED(CONFIG_PCI_MSI)) {
> > > +		/* Enable MSI interrupt signal */
> > > +		val = readl(rcar->base + PCIEINTSTS0EN);
> > > +		val |= MSI_CTRL_INT;
> > > +		writel(val, rcar->base + PCIEINTSTS0EN);
> > > +	}
> > > +
> > > +	msleep(100);	/* pe_rst requires 100msec delay */
> > > +
> > > +	gpiod_set_value_cansleep(dw->pe_rst, 0);
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static const struct dw_pcie_host_ops rcar_gen4_pcie_host_ops = {
> > > +	.host_init = rcar_gen4_pcie_host_init,
> > > +};
> > > +
> > > +static int rcar_gen4_add_dw_pcie_rp(struct rcar_gen4_pcie *rcar,
> > > +				   struct platform_device *pdev)
> > > +{
> > > +	struct dw_pcie *dw = &rcar->dw;
> > > +	struct dw_pcie_rp *pp = &dw->pp;
> > > +
> > > +	pp->num_vectors = MAX_MSI_IRQS;
> > > +	pp->ops = &rcar_gen4_pcie_host_ops;
> > > +	dw_pcie_cap_set(dw, REQ_RES);
> > > +
> > > +	return dw_pcie_host_init(pp);
> > > +}
> > > +
> > > +static void rcar_gen4_remove_dw_pcie_rp(struct rcar_gen4_pcie *rcar)
> > > +{
> > > +	dw_pcie_host_deinit(&rcar->dw.pp);
> > > +	gpiod_set_value_cansleep(rcar->dw.pe_rst, 1);
> > > +}
> > > +
> > > +static int rcar_gen4_pcie_probe(struct platform_device *pdev)
> > > +{
> > > +	struct device *dev = &pdev->dev;
> > > +	struct rcar_gen4_pcie *rcar;
> > > +	int err;
> > > +
> > > +	rcar = rcar_gen4_pcie_devm_alloc(dev);
> > > +	if (!rcar)
> > > +		return -ENOMEM;
> > > +
> > > +	err = rcar_gen4_pcie_get_resources(rcar, pdev);
> > > +	if (err < 0) {
> > > +		dev_err(dev, "Failed to request resource: %d\n", err);
> > > +		return err;
> > > +	}
> > > +
> > > +	platform_set_drvdata(pdev, rcar);
> > > +
> > > +	err = rcar_gen4_pcie_prepare(rcar);
> > > +	if (err < 0)
> > > +		return err;
> > > +
> > > +	rcar->needs_retrain = true;
> > > +	err = rcar_gen4_add_dw_pcie_rp(rcar, pdev);
> > > +	if (err < 0)
> > > +		goto err_add;
> > > +
> > > +	return 0;
> > > +
> > > +err_add:
> > > +	rcar_gen4_pcie_unprepare(rcar);
> > > +
> > > +	return err;
> > > +}
> > > +
> > > +static int rcar_gen4_pcie_remove(struct platform_device *pdev)
> > > +{
> > > +	struct rcar_gen4_pcie *rcar = platform_get_drvdata(pdev);
> > > +
> > > +	rcar_gen4_remove_dw_pcie_rp(rcar);
> > > +	rcar_gen4_pcie_unprepare(rcar);
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static const struct of_device_id rcar_gen4_pcie_of_match[] = {
> > > +	{ .compatible = "renesas,rcar-gen4-pcie", },
> > > +	{},
> > > +};
> > > +
> > > +static struct platform_driver rcar_gen4_pcie_driver = {
> > > +	.driver = {
> > > +		.name = "pcie-rcar-gen4",
> > > +		.of_match_table = rcar_gen4_pcie_of_match,
> > > +		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
> > > +	},
> > > +	.probe = rcar_gen4_pcie_probe,
> > > +	.remove = rcar_gen4_pcie_remove,
> > > +};
> > > +module_platform_driver(rcar_gen4_pcie_driver);
> > > +
> > > +MODULE_DESCRIPTION("Renesas R-Car Gen4 PCIe host controller driver");
> > > +MODULE_LICENSE("GPL");
> > > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.c b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > > new file mode 100644
> > > index 000000000000..35923fda8ed5
> > > --- /dev/null
> > > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > > @@ -0,0 +1,190 @@
> > > +// SPDX-License-Identifier: GPL-2.0-only
> > > +/*
> > > + * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
> > > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > > + */
> > > +
> > > +#include <linux/delay.h>
> > > +#include <linux/io.h>
> > > +#include <linux/of_device.h>
> > > +#include <linux/pci.h>
> > > +#include <linux/pm_runtime.h>
> > > +#include <linux/reset.h>
> > > +
> > > +#include "pcie-rcar-gen4.h"
> > > +#include "pcie-designware.h"
> > > +
> > > +/* Renesas-specific */
> > > +#define PCIERSTCTRL1		0x0014
> > > +#define  APP_HOLD_PHY_RST	BIT(16)
> > > +#define  APP_LTSSM_ENABLE	BIT(0)
> > > +
> > > +#define RETRAIN_MAX_CHECK	10
> > > +#define RETRAIN_MAX_RETRIES	10
> > > +
> > > +static void rcar_gen4_pcie_ltssm_enable(struct rcar_gen4_pcie *rcar,
> > > +					bool enable)
> > > +{
> > > +	u32 val;
> > > +
> > > +	val = readl(rcar->base + PCIERSTCTRL1);
> > > +	if (enable) {
> > > +		val |= APP_LTSSM_ENABLE;
> > 
> > > +		val &= ~APP_HOLD_PHY_RST;
> > 
> > What about moving the APP_HOLD_PHY_RST de-assertion to the
> > rcar_gen4_pcie_set_device_type() method? In accordance with the
> > "3.1 Initialization" chapter it's supposed to be done before
> > performing the DBI programming and activating the link training.
> 

> IIUC, the "3.1 Initialization" said app_hold_phy_rst = 1 before
> performing the DBI programming. So, it is assertion. Also, the SoC
> documentation described the initializing procedure as the follows:
>  app_ltssm_enable = 1
>  app_hold_phy_rst = 0
> So, I would like to keep them in the function.

Indeed. I was wrong. Sorry for the misleading comment.

> 
> > > +	} else {
> > > +		val &= ~APP_LTSSM_ENABLE;
> > > +		val |= APP_HOLD_PHY_RST;
> > > +	}
> > > +	writel(val, rcar->base + PCIERSTCTRL1);
> > > +}
> > > +
> > > +static bool rcar_gen4_pcie_check_retrain_link(struct dw_pcie *dw)
> > > +{
> > > +	u8 offset = dw_pcie_find_capability(dw, PCI_CAP_ID_EXP);
> > > +	u32 lnkcap = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCAP);
> > > +	u32 lnkctl = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCTL);
> > > +	u16 lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> > > +	int i;
> > > +
> > 
> > > +	if ((lnksta & PCI_EXP_LNKSTA_CLS) == (lnkcap & PCI_EXP_LNKCAP_SLS))
> > > +		return true;
> > > +
> > > +	lnkctl |= PCI_EXP_LNKCTL_RL;
> > > +	dw_pcie_writel_dbi(dw, offset + PCI_EXP_LNKCTL, lnkctl);
> > > +
> > > +	for (i = 0; i < RETRAIN_MAX_CHECK; i++) {
> > > +		lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> > > +		if (lnksta & PCI_EXP_LNKSTA_LT)
> > > +			return true;
> > > +		usleep_range(1000, 1100);
> > > +	}
> > 
> > I'll ask one more time because you didn't respond to my previous note
> > about this.
> 
> I'm sorry. I completely overlooked the previous note.
> 
> > Are you sure that this is needed? Did you try
> > the approach described in "3.13 Gen2/3/4/5 Speed Modes" with
> > de-asserting/asserting the GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag?
> 
> I tried this setting, but it doesn't work. I'll investigate this setting more.
> 
> > I keep asking because the same problem we used to have on our hardware
> > until we found out that the DIRECT_SPEED_CHANGE flag helped to train
> > the link right to the speed specified in the capabilities.
> > 
> > So here is what presumably you'll need to do (based on the
> > "3.1 Initialization" and "3.13 Gen2/3/4/5 Speed Modes" chapters of
> > the DW PCIe DM hw-manual):
> > 1. Make sure the controller is in the power-down/reset state.
> > 2. Select device_type (EP or RP).
> > 3. De-assert the controller reset.
> > 4. Clear PHY-reset flag in the app registers.
> > 5. Perform some controller initializations.
> > 6. Enable LTSSM to start link training.
> > 7. Set GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag one more time.
> > 
> > 1-4 are supposed to be done in rcar_gen4_pcie_host_init().
> > 5 is performed in the framework of the DW PCIe core driver.
> > 6-7 should be done in rcar_gen4_pcie_start_link().
> > 
> > Note 1. GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag is already set on stage
> > 5 in the framework of the dw_pcie_setup_rc() method. But in our case
> > it only caused having the Gen.2 link speed. Adding stage 7 helped to
> > get stable Gen.3 link. So please try the denoted approach. If it works
> > what about adding stage 7 twice in order to get Gen.4 speed?
> > (waiting for the DIRECT_SPEED_CHANGE flag being auto-cleared and then
> > set it up again?)
> > 
> > Note 2. GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag is defined as
> > PCIE_LINK_WIDTH_SPEED_CONTROL.PORT_LOGIC_SPEED_CHANGE macros in the DW
> > PCIe core driver.
> > 
> > Note 3. If what is suggested above works well then you won't need to
> > have the heavy rcar_gen4_pcie_check_retrain_link() method in the way
> > you have it defined.
> 
> Thank you very much for your comments!

Please see the GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE description for details
of how the flag is supposed to be de-asserted and asserted in order to
initiate the direct speed change.

> 
> > > +
> > > +	return false;
> > > +}
> > > +
> > > +static int rcar_gen4_pcie_link_up(struct dw_pcie *dw)
> > > +{
> > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > +	u32 val, mask;
> > > +
> > > +	val = readl(rcar->base + PCIEINTSTS0);
> > > +	mask = RDLH_LINK_UP | SMLH_LINK_UP;
> > > +
> > > +	return (val & mask) == mask;
> > > +}
> > > +
> > > +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > +{
> > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > +	int i;
> > > +
> > > +	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > +
> > > +	/*
> > > +	 * Require retraining here. Otherwise RDLH_LINK_UP of PCIEINTSTS0 which
> > > +	 * is this controller specific register may not be set.
> > > +	 */
> > > +	if (rcar->needs_retrain) {
> > > +		for (i = 0; i < RETRAIN_MAX_RETRIES; i++) {
> > > +			if (rcar_gen4_pcie_check_retrain_link(dw))
> > > +				return 0;
> > > +			msleep(100);
> > > +		}
> > > +
> > > +		return -ETIMEDOUT;	/* Failed */
> > > +	}
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static void rcar_gen4_pcie_stop_link(struct dw_pcie *dw)
> > > +{
> > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > +
> > > +	rcar_gen4_pcie_ltssm_enable(rcar, false);
> > > +}
> > > +
> > 
> > > +int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
> > > +				   int num_lanes)
> > 
> > 1. Number of lanes is already defined in the rcar_gen4_pcie.dw.num_lanes field.
> > What about using it from there instead of passing it as an argument?
> > 2. DW PCIe core driver has a very handy enum defined:
> > dw_pcie_device_mode. It describes the controller modes (End-point,
> > Root port, etc). What about adding the mode field right to the
> > rcar_gen4_pcie structure and initializing it in someplace in probe() ?
> > 3. Based on the function semantic it's better to be named as something
> > like rcar_gen4_pcie_init_device() or even rcar_gen4_pcie_basic_init().
> 
> Thank you for your comments! I'll modify the function.
> 
> > 
> > > +{
> > > +	u32 val;
> > > +
> > 
> > > +	/* Note: Assume the rcar->rst which is Cold-reset is asserted here */
> > 
> > What about directly asserting it here then? In accordance with the DW
> > PCIe DM manual the "device_type" input must be set before the DM
> > controller is powered up (basically un-reset). What if the controller
> > reset is already de-asserted, but you are going to changes its mode?
> > In that case the mode won't be changed and you'll end up with
> > unpredictable results.
> 
> Thank you for your comment. We should add asserting it here as you mentioned.
> 
> > > +	val = readl(rcar->base + PCIEMSR0);
> > > +	if (rc)
> > > +		val |= DEVICE_TYPE_RC;
> > > +	else
> > > +		val |= DEVICE_TYPE_EP;
> > > +
> > > +	if (num_lanes < 4)
> > > +		val |= BIFUR_MOD_SET_ON;
> > > +
> > > +	writel(val, rcar->base + PCIEMSR0);
> > > +
> > > +	return reset_control_deassert(rcar->rst);
> > > +}
> > > +
> > > +int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *rcar)
> > > +{
> > > +	struct device *dev = rcar->dw.dev;
> > > +	int err;
> > > +
> > > +	pm_runtime_enable(dev);
> > > +	err = pm_runtime_resume_and_get(dev);
> > > +	if (err < 0) {
> > > +		dev_err(dev, "Failed to resume/get Runtime PM\n");
> > > +		pm_runtime_disable(dev);
> > > +	}
> > > +
> > > +	return err;
> > > +}
> > > +
> > > +void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *rcar)
> > > +{
> > > +	struct device *dev = rcar->dw.dev;
> > > +
> > > +	if (!reset_control_status(rcar->rst))
> > > +		reset_control_assert(rcar->rst);
> > > +	pm_runtime_put(dev);
> > > +	pm_runtime_disable(dev);
> > > +}
> > > +
> > > +int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
> > > +				 struct platform_device *pdev)
> > > +{
> > > +	struct device *dev = rcar->dw.dev;
> > > +
> > > +	/* Renesas-specific registers */
> > > +	rcar->base = devm_platform_ioremap_resource_byname(pdev, "app");
> > > +	if (IS_ERR(rcar->base))
> > > +		return PTR_ERR(rcar->base);
> > > +
> > 
> > > +	rcar->rst = devm_reset_control_get(dev, NULL);
> > > +	if (IS_ERR(rcar->rst)) {
> > > +		dev_err(dev, "Failed to get Cold-reset\n");
> > 
> > So AFAICS your platform is equipped with the DWC_pcie_clkrst.v module.
> > Thus all the resets are appropriately cleared by a single flag:
> > power_up_rst_n. What about using the named reset in this case with the
> > "pwr" name? Thus you'll be able to drop the manual
> > devm_reset_control_get() invocation and instead use the reset-resources
> > requested in the framework of the generic dw_pcie_get_resources()
> > method? Note you'll need to move the dw_pcie_cap_set(dw, REQ_RES);
> > statement to rcar_gen4_pcie_devm_alloc() then and drop the
> > rcar_gen4_pcie.rst field afterwords.
> 
> Thank you for your suggestion! Using "pwr" can work on my environment.
> 
> > By the way I don't see you requesting and enabling the reference
> > clock in your driver but the bindings imply the clock source. How
> > come?
> 

> For now, I used gpio-hog to enable the reference clock. But, it seem
> I should use "ref" clock for it. So, I'll fix it too.

Not sure what gpio-hog you are talking about. Do you mean the pe_rst
signal or some another gpio? I failed to see of how pe_rst is
connected to the clock source. In anyway directly handling the clock
source would be more portable choice.

-Serge(y)

> 
> > > +		return PTR_ERR(rcar->rst);
> > > +	}
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static const struct dw_pcie_ops dw_pcie_ops = {
> > > +	.start_link = rcar_gen4_pcie_start_link,
> > > +	.stop_link = rcar_gen4_pcie_stop_link,
> > > +	.link_up = rcar_gen4_pcie_link_up,
> > > +};
> > > +
> > > +struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev)
> > > +{
> > > +	struct rcar_gen4_pcie *rcar;
> > > +
> > > +	rcar = devm_kzalloc(dev, sizeof(*rcar), GFP_KERNEL);
> > > +	if (!rcar)
> > > +		return NULL;
> > > +
> > > +	rcar->dw.dev = dev;
> > > +	rcar->dw.ops = &dw_pcie_ops;
> > > +	dw_pcie_cap_set(&rcar->dw, EDMA_UNROLL);
> > > +
> > > +	return rcar;
> > > +}
> > > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.h b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > > new file mode 100644
> > > index 000000000000..fec3f18609f4
> > > --- /dev/null
> > > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > > @@ -0,0 +1,46 @@
> > > +/* SPDX-License-Identifier: GPL-2.0-only */
> > > +/*
> > > + * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
> > > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > > + */
> > > +
> > > +#ifndef _PCIE_RCAR_GEN4_H_
> > > +#define _PCIE_RCAR_GEN4_H_
> > > +
> > > +#include <linux/io.h>
> > > +#include <linux/pci.h>
> > > +#include <linux/reset.h>
> > > +
> > > +#include "pcie-designware.h"
> > > +
> > > +/* Renesas-specific */
> > > +#define PCIEMSR0		0x0000
> > > +#define  BIFUR_MOD_SET_ON	BIT(0)
> > > +#define  DEVICE_TYPE_EP		0
> > > +#define  DEVICE_TYPE_RC		BIT(4)
> > > +
> > > +#define PCIEINTSTS0		0x0084
> > > +#define PCIEINTSTS0EN		0x0310
> > > +#define  MSI_CTRL_INT		BIT(26)
> > > +#define  SMLH_LINK_UP		BIT(7)
> > > +#define  RDLH_LINK_UP		BIT(6)
> > > +#define PCIEDMAINTSTSEN		0x0314
> > > +#define  PCIEDMAINTSTSEN_INIT	GENMASK(15, 0)
> > > +
> > 
> > > +struct rcar_gen4_pcie {
> > 
> > As I mentioned above this structure can be extended with the enum
> > dw_pcie_device_mode field thus dropping the boolean argument from the
> > rcar_gen4_pcie_set_device_type() method.
> 
> I got it. I'll fix this.
> 
> > > +	struct dw_pcie		dw;
> > 
> > As I already mentioned above the dw.num_lanes could be used instead of
> > passing it as the rcar_gen4_pcie_set_device_type() argument.
> 
> I'll fix this too.
> 
> Best regards,
> Yoshihiro Shimoda
> 
> > -Serge(y)
> > 
> > > +	void __iomem		*base;
> > > +	struct reset_control	*rst;
> > > +	bool			needs_retrain;
> > > +};
> > > +#define to_rcar_gen4_pcie(x)	dev_get_drvdata((x)->dev)
> > > +
> > > +int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
> > > +				   int num_lanes);
> > > +int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *pcie);
> > > +void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *pcie);
> > > +int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
> > > +				 struct platform_device *pdev);
> > > +struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev);
> > > +
> > > +#endif /* _PCIE_RCAR_GEN4_H_ */
> > > --
> > > 2.25.1
> > >

RE: [PATCH v16 19/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support

[Yoshihiro Shimoda]

Hello Serge,

> From: Serge Semin, Sent: Wednesday, June 7, 2023 9:16 PM
> 
> On Wed, Jun 07, 2023 at 02:59:20AM +0000, Yoshihiro Shimoda wrote:
> > Hello Serge,
> >
> > > From: Serge Semin, Sent: Monday, June 5, 2023 11:39 PM
> > >
> > > On Wed, May 10, 2023 at 03:22:31PM +0900, Yoshihiro Shimoda wrote:
> > > > Add R-Car Gen4 PCIe Host support. This controller is based on
> > > > Synopsys DesignWare PCIe, but this controller has vendor-specific
> > > > registers so that requires initialization code like mode setting
> > > > and retraining and so on.
> > > >
> > > > To reduce code delta, adds some helper functions which are used by
> > > > both the host driver and the endpoint driver (which is added
> > > > immediately afterwards) into a separate file.
> > > >
> > > > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > > > ---
> > > >  drivers/pci/controller/dwc/Kconfig            |   9 +
> > > >  drivers/pci/controller/dwc/Makefile           |   2 +
> > > >  .../pci/controller/dwc/pcie-rcar-gen4-host.c  | 141 +++++++++++++
> > > >  drivers/pci/controller/dwc/pcie-rcar-gen4.c   | 190 ++++++++++++++++++
> > > >  drivers/pci/controller/dwc/pcie-rcar-gen4.h   |  46 +++++
> > > >  5 files changed, 388 insertions(+)
> > > >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > > >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > > >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > > >
> > > > diff --git a/drivers/pci/controller/dwc/Kconfig b/drivers/pci/controller/dwc/Kconfig
> > > > index ab96da43e0c2..64d4d37bc891 100644
> > > > --- a/drivers/pci/controller/dwc/Kconfig
> > > > +++ b/drivers/pci/controller/dwc/Kconfig
> > > > @@ -415,4 +415,13 @@ config PCIE_VISCONTI_HOST
> > > >  	  Say Y here if you want PCIe controller support on Toshiba Visconti SoC.
> > > >  	  This driver supports TMPV7708 SoC.
> > > >
> > > > +config PCIE_RCAR_GEN4
> > > > +	tristate "Renesas R-Car Gen4 PCIe Host controller"
> > > > +	depends on ARCH_RENESAS || COMPILE_TEST
> > > > +	depends on PCI_MSI
> > > > +	select PCIE_DW_HOST
> > > > +	help
> > > > +	  Say Y here if you want PCIe host controller support on R-Car Gen4 SoCs.
> > > > +	  This uses the DesignWare core.
> > > > +
> > > >  endmenu
> > > > diff --git a/drivers/pci/controller/dwc/Makefile b/drivers/pci/controller/dwc/Makefile
> > > > index bf5c311875a1..486cf706b53d 100644
> > > > --- a/drivers/pci/controller/dwc/Makefile
> > > > +++ b/drivers/pci/controller/dwc/Makefile
> > > > @@ -26,6 +26,8 @@ obj-$(CONFIG_PCIE_TEGRA194) += pcie-tegra194.o
> > > >  obj-$(CONFIG_PCIE_UNIPHIER) += pcie-uniphier.o
> > > >  obj-$(CONFIG_PCIE_UNIPHIER_EP) += pcie-uniphier-ep.o
> > > >  obj-$(CONFIG_PCIE_VISCONTI_HOST) += pcie-visconti.o
> > > > +pcie-rcar-gen4-host-drv-objs := pcie-rcar-gen4.o pcie-rcar-gen4-host.o
> > > > +obj-$(CONFIG_PCIE_RCAR_GEN4) += pcie-rcar-gen4-host-drv.o
> > > >
> > > >  # The following drivers are for devices that use the generic ACPI
> > > >  # pci_root.c driver but don't support standard ECAM config access.
> > > > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > > > new file mode 100644
> > > > index 000000000000..df7d80f1874f
> > > > --- /dev/null
> > > > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > > > @@ -0,0 +1,141 @@
> > > > +// SPDX-License-Identifier: GPL-2.0-only
> > > > +/*
> > > > + * PCIe host controller driver for Renesas R-Car Gen4 Series SoCs
> > > > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > > > + */
> > > > +
> > > > +#include <linux/delay.h>
> > > > +#include <linux/interrupt.h>
> > > > +#include <linux/module.h>
> > > > +#include <linux/of_device.h>
> > > > +#include <linux/pci.h>
> > > > +#include <linux/platform_device.h>
> > > > +
> > > > +#include "pcie-rcar-gen4.h"
> > > > +#include "pcie-designware.h"
> > > > +
> > > > +static int rcar_gen4_pcie_host_init(struct dw_pcie_rp *pp)
> > > > +{
> > > > +	struct dw_pcie *dw = to_dw_pcie_from_pp(pp);
> > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > +	int ret;
> > > > +	u32 val;
> > > > +
> > > > +	gpiod_set_value_cansleep(dw->pe_rst, 1);
> > > > +
> > > > +	ret = rcar_gen4_pcie_set_device_type(rcar, true, dw->num_lanes);
> > > > +	if (ret < 0)
> > > > +		return ret;
> > > > +
> > >
> > > > +	dw_pcie_dbi_ro_wr_en(dw);
> > >
> > > Are you sure dw_pcie_dbi_ro_wr_en() and dw_pcie_dbi_ro_wr_dis() are
> > > needed? In accordance with the DW PCIe Dual-mode HW manual the BARx
> > > registers are W-only over the DBI2 map with no need in setting the
> > > DBI_RO_WR_EN flag.
> > >
> > > Please check that on your hardware.
> >
> > You're correct. They are not needed. So, I'll drop this on v17.
> >
> > > > +
> > > > +	/*
> > > > +	 * According to the section 3.5.7.2 "RC Mode" in DWC PCIe Dual Mode
> > > > +	 * Rev.5.20a, we should disable two BARs to avoid unnecessary memory
> > > > +	 * assignment during device enumeration.
> > > > +	 */
> > > > +	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_0, 0x0);
> > > > +	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_1, 0x0);
> > > > +
> > >
> > > > +	dw_pcie_dbi_ro_wr_dis(dw);
> > >
> > > ditto
> >
> > I'll drop this too.
> >
> > > > +
> > > > +	if (IS_ENABLED(CONFIG_PCI_MSI)) {
> > > > +		/* Enable MSI interrupt signal */
> > > > +		val = readl(rcar->base + PCIEINTSTS0EN);
> > > > +		val |= MSI_CTRL_INT;
> > > > +		writel(val, rcar->base + PCIEINTSTS0EN);
> > > > +	}
> > > > +
> > > > +	msleep(100);	/* pe_rst requires 100msec delay */
> > > > +
> > > > +	gpiod_set_value_cansleep(dw->pe_rst, 0);
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static const struct dw_pcie_host_ops rcar_gen4_pcie_host_ops = {
> > > > +	.host_init = rcar_gen4_pcie_host_init,
> > > > +};
> > > > +
> > > > +static int rcar_gen4_add_dw_pcie_rp(struct rcar_gen4_pcie *rcar,
> > > > +				   struct platform_device *pdev)
> > > > +{
> > > > +	struct dw_pcie *dw = &rcar->dw;
> > > > +	struct dw_pcie_rp *pp = &dw->pp;
> > > > +
> > > > +	pp->num_vectors = MAX_MSI_IRQS;
> > > > +	pp->ops = &rcar_gen4_pcie_host_ops;
> > > > +	dw_pcie_cap_set(dw, REQ_RES);
> > > > +
> > > > +	return dw_pcie_host_init(pp);
> > > > +}
> > > > +
> > > > +static void rcar_gen4_remove_dw_pcie_rp(struct rcar_gen4_pcie *rcar)
> > > > +{
> > > > +	dw_pcie_host_deinit(&rcar->dw.pp);
> > > > +	gpiod_set_value_cansleep(rcar->dw.pe_rst, 1);
> > > > +}
> > > > +
> > > > +static int rcar_gen4_pcie_probe(struct platform_device *pdev)
> > > > +{
> > > > +	struct device *dev = &pdev->dev;
> > > > +	struct rcar_gen4_pcie *rcar;
> > > > +	int err;
> > > > +
> > > > +	rcar = rcar_gen4_pcie_devm_alloc(dev);
> > > > +	if (!rcar)
> > > > +		return -ENOMEM;
> > > > +
> > > > +	err = rcar_gen4_pcie_get_resources(rcar, pdev);
> > > > +	if (err < 0) {
> > > > +		dev_err(dev, "Failed to request resource: %d\n", err);
> > > > +		return err;
> > > > +	}
> > > > +
> > > > +	platform_set_drvdata(pdev, rcar);
> > > > +
> > > > +	err = rcar_gen4_pcie_prepare(rcar);
> > > > +	if (err < 0)
> > > > +		return err;
> > > > +
> > > > +	rcar->needs_retrain = true;
> > > > +	err = rcar_gen4_add_dw_pcie_rp(rcar, pdev);
> > > > +	if (err < 0)
> > > > +		goto err_add;
> > > > +
> > > > +	return 0;
> > > > +
> > > > +err_add:
> > > > +	rcar_gen4_pcie_unprepare(rcar);
> > > > +
> > > > +	return err;
> > > > +}
> > > > +
> > > > +static int rcar_gen4_pcie_remove(struct platform_device *pdev)
> > > > +{
> > > > +	struct rcar_gen4_pcie *rcar = platform_get_drvdata(pdev);
> > > > +
> > > > +	rcar_gen4_remove_dw_pcie_rp(rcar);
> > > > +	rcar_gen4_pcie_unprepare(rcar);
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static const struct of_device_id rcar_gen4_pcie_of_match[] = {
> > > > +	{ .compatible = "renesas,rcar-gen4-pcie", },
> > > > +	{},
> > > > +};
> > > > +
> > > > +static struct platform_driver rcar_gen4_pcie_driver = {
> > > > +	.driver = {
> > > > +		.name = "pcie-rcar-gen4",
> > > > +		.of_match_table = rcar_gen4_pcie_of_match,
> > > > +		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
> > > > +	},
> > > > +	.probe = rcar_gen4_pcie_probe,
> > > > +	.remove = rcar_gen4_pcie_remove,
> > > > +};
> > > > +module_platform_driver(rcar_gen4_pcie_driver);
> > > > +
> > > > +MODULE_DESCRIPTION("Renesas R-Car Gen4 PCIe host controller driver");
> > > > +MODULE_LICENSE("GPL");
> > > > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.c b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > > > new file mode 100644
> > > > index 000000000000..35923fda8ed5
> > > > --- /dev/null
> > > > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > > > @@ -0,0 +1,190 @@
> > > > +// SPDX-License-Identifier: GPL-2.0-only
> > > > +/*
> > > > + * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
> > > > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > > > + */
> > > > +
> > > > +#include <linux/delay.h>
> > > > +#include <linux/io.h>
> > > > +#include <linux/of_device.h>
> > > > +#include <linux/pci.h>
> > > > +#include <linux/pm_runtime.h>
> > > > +#include <linux/reset.h>
> > > > +
> > > > +#include "pcie-rcar-gen4.h"
> > > > +#include "pcie-designware.h"
> > > > +
> > > > +/* Renesas-specific */
> > > > +#define PCIERSTCTRL1		0x0014
> > > > +#define  APP_HOLD_PHY_RST	BIT(16)
> > > > +#define  APP_LTSSM_ENABLE	BIT(0)
> > > > +
> > > > +#define RETRAIN_MAX_CHECK	10
> > > > +#define RETRAIN_MAX_RETRIES	10
> > > > +
> > > > +static void rcar_gen4_pcie_ltssm_enable(struct rcar_gen4_pcie *rcar,
> > > > +					bool enable)
> > > > +{
> > > > +	u32 val;
> > > > +
> > > > +	val = readl(rcar->base + PCIERSTCTRL1);
> > > > +	if (enable) {
> > > > +		val |= APP_LTSSM_ENABLE;
> > >
> > > > +		val &= ~APP_HOLD_PHY_RST;
> > >
> > > What about moving the APP_HOLD_PHY_RST de-assertion to the
> > > rcar_gen4_pcie_set_device_type() method? In accordance with the
> > > "3.1 Initialization" chapter it's supposed to be done before
> > > performing the DBI programming and activating the link training.
> >
> 
> > IIUC, the "3.1 Initialization" said app_hold_phy_rst = 1 before
> > performing the DBI programming. So, it is assertion. Also, the SoC
> > documentation described the initializing procedure as the follows:
> >  app_ltssm_enable = 1
> >  app_hold_phy_rst = 0
> > So, I would like to keep them in the function.
> 
> Indeed. I was wrong. Sorry for the misleading comment.

No problem. Thank you for the confirmation!

> >
> > > > +	} else {
> > > > +		val &= ~APP_LTSSM_ENABLE;
> > > > +		val |= APP_HOLD_PHY_RST;
> > > > +	}
> > > > +	writel(val, rcar->base + PCIERSTCTRL1);
> > > > +}
> > > > +
> > > > +static bool rcar_gen4_pcie_check_retrain_link(struct dw_pcie *dw)
> > > > +{
> > > > +	u8 offset = dw_pcie_find_capability(dw, PCI_CAP_ID_EXP);
> > > > +	u32 lnkcap = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCAP);
> > > > +	u32 lnkctl = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCTL);
> > > > +	u16 lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> > > > +	int i;
> > > > +
> > >
> > > > +	if ((lnksta & PCI_EXP_LNKSTA_CLS) == (lnkcap & PCI_EXP_LNKCAP_SLS))
> > > > +		return true;
> > > > +
> > > > +	lnkctl |= PCI_EXP_LNKCTL_RL;
> > > > +	dw_pcie_writel_dbi(dw, offset + PCI_EXP_LNKCTL, lnkctl);
> > > > +
> > > > +	for (i = 0; i < RETRAIN_MAX_CHECK; i++) {
> > > > +		lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> > > > +		if (lnksta & PCI_EXP_LNKSTA_LT)
> > > > +			return true;
> > > > +		usleep_range(1000, 1100);
> > > > +	}
> > >
> > > I'll ask one more time because you didn't respond to my previous note
> > > about this.
> >
> > I'm sorry. I completely overlooked the previous note.
> >
> > > Are you sure that this is needed? Did you try
> > > the approach described in "3.13 Gen2/3/4/5 Speed Modes" with
> > > de-asserting/asserting the GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag?
> >
> > I tried this setting, but it doesn't work. I'll investigate this setting more.
> >
> > > I keep asking because the same problem we used to have on our hardware
> > > until we found out that the DIRECT_SPEED_CHANGE flag helped to train
> > > the link right to the speed specified in the capabilities.
> > >
> > > So here is what presumably you'll need to do (based on the
> > > "3.1 Initialization" and "3.13 Gen2/3/4/5 Speed Modes" chapters of
> > > the DW PCIe DM hw-manual):
> > > 1. Make sure the controller is in the power-down/reset state.
> > > 2. Select device_type (EP or RP).
> > > 3. De-assert the controller reset.
> > > 4. Clear PHY-reset flag in the app registers.
> > > 5. Perform some controller initializations.
> > > 6. Enable LTSSM to start link training.
> > > 7. Set GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag one more time.
> > >
> > > 1-4 are supposed to be done in rcar_gen4_pcie_host_init().
> > > 5 is performed in the framework of the DW PCIe core driver.
> > > 6-7 should be done in rcar_gen4_pcie_start_link().
> > >
> > > Note 1. GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag is already set on stage
> > > 5 in the framework of the dw_pcie_setup_rc() method. But in our case
> > > it only caused having the Gen.2 link speed. Adding stage 7 helped to
> > > get stable Gen.3 link. So please try the denoted approach. If it works
> > > what about adding stage 7 twice in order to get Gen.4 speed?
> > > (waiting for the DIRECT_SPEED_CHANGE flag being auto-cleared and then
> > > set it up again?)
> > >
> > > Note 2. GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag is defined as
> > > PCIE_LINK_WIDTH_SPEED_CONTROL.PORT_LOGIC_SPEED_CHANGE macros in the DW
> > > PCIe core driver.
> > >
> > > Note 3. If what is suggested above works well then you won't need to
> > > have the heavy rcar_gen4_pcie_check_retrain_link() method in the way
> > > you have it defined.
> >
> > Thank you very much for your comments!
> 
> Please see the GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE description for details
> of how the flag is supposed to be de-asserted and asserted in order to
> initiate the direct speed change.

After I modified the start_link() like below, it also works. Is the code
acceptable? (Sorry all tabs are replaced to spaces...)
----------------------------------------------------------------------------
static bool rcar_gen4_pcie_check_current_link(struct dw_pcie *dw)
{
        u8 offset = dw_pcie_find_capability(dw, PCI_CAP_ID_EXP);
        u32 lnkcap = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCAP);
        u16 lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);

        if ((lnksta & PCI_EXP_LNKSTA_CLS) == (lnkcap & PCI_EXP_LNKCAP_SLS))
                return true;

        return false;
}

static void rcar_gen4_pcie_speed_change(struct dw_pcie *dw)
{
        u32 val;

        val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
        val &= ~PORT_LOGIC_SPEED_CHANGE;
        dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);

        val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
        val |= PORT_LOGIC_SPEED_CHANGE;
        dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
}

static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
{
        struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
        int i;

        rcar_gen4_pcie_ltssm_enable(rcar, true);

        /*
         * Require direct speed change here. Otherwise RDLH_LINK_UP of
         * PCIEINTSTS0 which is this controller specific register may not
         * be set.
         */
        if (rcar->needs_speed_change) {
                for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
                        rcar_gen4_pcie_speed_change(dw);
                        msleep(100);
                        if (rcar_gen4_pcie_check_current_link(dw))
                                return 0;
                }

                return -ETIMEDOUT;      /* Failed */
        }
------------------------------------------------------------------

> >
> > > > +
> > > > +	return false;
> > > > +}
> > > > +
> > > > +static int rcar_gen4_pcie_link_up(struct dw_pcie *dw)
> > > > +{
> > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > +	u32 val, mask;
> > > > +
> > > > +	val = readl(rcar->base + PCIEINTSTS0);
> > > > +	mask = RDLH_LINK_UP | SMLH_LINK_UP;
> > > > +
> > > > +	return (val & mask) == mask;
> > > > +}
> > > > +
> > > > +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > > +{
> > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > +	int i;
> > > > +
> > > > +	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > +
> > > > +	/*
> > > > +	 * Require retraining here. Otherwise RDLH_LINK_UP of PCIEINTSTS0 which
> > > > +	 * is this controller specific register may not be set.
> > > > +	 */
> > > > +	if (rcar->needs_retrain) {
> > > > +		for (i = 0; i < RETRAIN_MAX_RETRIES; i++) {
> > > > +			if (rcar_gen4_pcie_check_retrain_link(dw))
> > > > +				return 0;
> > > > +			msleep(100);
> > > > +		}
> > > > +
> > > > +		return -ETIMEDOUT;	/* Failed */
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static void rcar_gen4_pcie_stop_link(struct dw_pcie *dw)
> > > > +{
> > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > +
> > > > +	rcar_gen4_pcie_ltssm_enable(rcar, false);
> > > > +}
> > > > +
> > >
> > > > +int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
> > > > +				   int num_lanes)
> > >
> > > 1. Number of lanes is already defined in the rcar_gen4_pcie.dw.num_lanes field.
> > > What about using it from there instead of passing it as an argument?
> > > 2. DW PCIe core driver has a very handy enum defined:
> > > dw_pcie_device_mode. It describes the controller modes (End-point,
> > > Root port, etc). What about adding the mode field right to the
> > > rcar_gen4_pcie structure and initializing it in someplace in probe() ?
> > > 3. Based on the function semantic it's better to be named as something
> > > like rcar_gen4_pcie_init_device() or even rcar_gen4_pcie_basic_init().
> >
> > Thank you for your comments! I'll modify the function.
> >
> > >
> > > > +{
> > > > +	u32 val;
> > > > +
> > >
> > > > +	/* Note: Assume the rcar->rst which is Cold-reset is asserted here */
> > >
> > > What about directly asserting it here then? In accordance with the DW
> > > PCIe DM manual the "device_type" input must be set before the DM
> > > controller is powered up (basically un-reset). What if the controller
> > > reset is already de-asserted, but you are going to changes its mode?
> > > In that case the mode won't be changed and you'll end up with
> > > unpredictable results.
> >
> > Thank you for your comment. We should add asserting it here as you mentioned.
> >
> > > > +	val = readl(rcar->base + PCIEMSR0);
> > > > +	if (rc)
> > > > +		val |= DEVICE_TYPE_RC;
> > > > +	else
> > > > +		val |= DEVICE_TYPE_EP;
> > > > +
> > > > +	if (num_lanes < 4)
> > > > +		val |= BIFUR_MOD_SET_ON;
> > > > +
> > > > +	writel(val, rcar->base + PCIEMSR0);
> > > > +
> > > > +	return reset_control_deassert(rcar->rst);
> > > > +}
> > > > +
> > > > +int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *rcar)
> > > > +{
> > > > +	struct device *dev = rcar->dw.dev;
> > > > +	int err;
> > > > +
> > > > +	pm_runtime_enable(dev);
> > > > +	err = pm_runtime_resume_and_get(dev);
> > > > +	if (err < 0) {
> > > > +		dev_err(dev, "Failed to resume/get Runtime PM\n");
> > > > +		pm_runtime_disable(dev);
> > > > +	}
> > > > +
> > > > +	return err;
> > > > +}
> > > > +
> > > > +void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *rcar)
> > > > +{
> > > > +	struct device *dev = rcar->dw.dev;
> > > > +
> > > > +	if (!reset_control_status(rcar->rst))
> > > > +		reset_control_assert(rcar->rst);
> > > > +	pm_runtime_put(dev);
> > > > +	pm_runtime_disable(dev);
> > > > +}
> > > > +
> > > > +int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
> > > > +				 struct platform_device *pdev)
> > > > +{
> > > > +	struct device *dev = rcar->dw.dev;
> > > > +
> > > > +	/* Renesas-specific registers */
> > > > +	rcar->base = devm_platform_ioremap_resource_byname(pdev, "app");
> > > > +	if (IS_ERR(rcar->base))
> > > > +		return PTR_ERR(rcar->base);
> > > > +
> > >
> > > > +	rcar->rst = devm_reset_control_get(dev, NULL);
> > > > +	if (IS_ERR(rcar->rst)) {
> > > > +		dev_err(dev, "Failed to get Cold-reset\n");
> > >
> > > So AFAICS your platform is equipped with the DWC_pcie_clkrst.v module.
> > > Thus all the resets are appropriately cleared by a single flag:
> > > power_up_rst_n. What about using the named reset in this case with the
> > > "pwr" name? Thus you'll be able to drop the manual
> > > devm_reset_control_get() invocation and instead use the reset-resources
> > > requested in the framework of the generic dw_pcie_get_resources()
> > > method? Note you'll need to move the dw_pcie_cap_set(dw, REQ_RES);
> > > statement to rcar_gen4_pcie_devm_alloc() then and drop the
> > > rcar_gen4_pcie.rst field afterwords.
> >
> > Thank you for your suggestion! Using "pwr" can work on my environment.
> >
> > > By the way I don't see you requesting and enabling the reference
> > > clock in your driver but the bindings imply the clock source. How
> > > come?
> >
> 
> > For now, I used gpio-hog to enable the reference clock. But, it seem
> > I should use "ref" clock for it. So, I'll fix it too.
> 
> Not sure what gpio-hog you are talking about. Do you mean the pe_rst
> signal or some another gpio? I failed to see of how pe_rst is
> connected to the clock source. In anyway directly handling the clock
> source would be more portable choice.

Sorry for lacking information. I described a gpio node like below
and then the gpio will be high automatically, and the reference clock
will be output. But, this is completely independent from pci.
---
&gpio2 {
        pci-clkreq0-hog {
                gpio-hog;
                gpios = <15 GPIO_ACTIVE_LOW>;
                output-high;
        };
};
---

Now I could implement the clock handling by using "gpio-gate-clock".
So, I'll drop the gpio-hog for the reference clock.

Best regards,
Yoshihiro Shimoda

> -Serge(y)
> 
> >
> > > > +		return PTR_ERR(rcar->rst);
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static const struct dw_pcie_ops dw_pcie_ops = {
> > > > +	.start_link = rcar_gen4_pcie_start_link,
> > > > +	.stop_link = rcar_gen4_pcie_stop_link,
> > > > +	.link_up = rcar_gen4_pcie_link_up,
> > > > +};
> > > > +
> > > > +struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev)
> > > > +{
> > > > +	struct rcar_gen4_pcie *rcar;
> > > > +
> > > > +	rcar = devm_kzalloc(dev, sizeof(*rcar), GFP_KERNEL);
> > > > +	if (!rcar)
> > > > +		return NULL;
> > > > +
> > > > +	rcar->dw.dev = dev;
> > > > +	rcar->dw.ops = &dw_pcie_ops;
> > > > +	dw_pcie_cap_set(&rcar->dw, EDMA_UNROLL);
> > > > +
> > > > +	return rcar;
> > > > +}
> > > > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.h b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > > > new file mode 100644
> > > > index 000000000000..fec3f18609f4
> > > > --- /dev/null
> > > > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > > > @@ -0,0 +1,46 @@
> > > > +/* SPDX-License-Identifier: GPL-2.0-only */
> > > > +/*
> > > > + * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
> > > > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > > > + */
> > > > +
> > > > +#ifndef _PCIE_RCAR_GEN4_H_
> > > > +#define _PCIE_RCAR_GEN4_H_
> > > > +
> > > > +#include <linux/io.h>
> > > > +#include <linux/pci.h>
> > > > +#include <linux/reset.h>
> > > > +
> > > > +#include "pcie-designware.h"
> > > > +
> > > > +/* Renesas-specific */
> > > > +#define PCIEMSR0		0x0000
> > > > +#define  BIFUR_MOD_SET_ON	BIT(0)
> > > > +#define  DEVICE_TYPE_EP		0
> > > > +#define  DEVICE_TYPE_RC		BIT(4)
> > > > +
> > > > +#define PCIEINTSTS0		0x0084
> > > > +#define PCIEINTSTS0EN		0x0310
> > > > +#define  MSI_CTRL_INT		BIT(26)
> > > > +#define  SMLH_LINK_UP		BIT(7)
> > > > +#define  RDLH_LINK_UP		BIT(6)
> > > > +#define PCIEDMAINTSTSEN		0x0314
> > > > +#define  PCIEDMAINTSTSEN_INIT	GENMASK(15, 0)
> > > > +
> > >
> > > > +struct rcar_gen4_pcie {
> > >
> > > As I mentioned above this structure can be extended with the enum
> > > dw_pcie_device_mode field thus dropping the boolean argument from the
> > > rcar_gen4_pcie_set_device_type() method.
> >
> > I got it. I'll fix this.
> >
> > > > +	struct dw_pcie		dw;
> > >
> > > As I already mentioned above the dw.num_lanes could be used instead of
> > > passing it as the rcar_gen4_pcie_set_device_type() argument.
> >
> > I'll fix this too.
> >
> > Best regards,
> > Yoshihiro Shimoda
> >
> > > -Serge(y)
> > >
> > > > +	void __iomem		*base;
> > > > +	struct reset_control	*rst;
> > > > +	bool			needs_retrain;
> > > > +};
> > > > +#define to_rcar_gen4_pcie(x)	dev_get_drvdata((x)->dev)
> > > > +
> > > > +int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
> > > > +				   int num_lanes);
> > > > +int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *pcie);
> > > > +void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *pcie);
> > > > +int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
> > > > +				 struct platform_device *pdev);
> > > > +struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev);
> > > > +
> > > > +#endif /* _PCIE_RCAR_GEN4_H_ */
> > > > --
> > > > 2.25.1
> > > >

Re: [PATCH v16 19/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support

[Serge Semin]

On Thu, Jun 08, 2023 at 08:47:16AM +0000, Yoshihiro Shimoda wrote:
> Hello Serge,
> 
> > From: Serge Semin, Sent: Wednesday, June 7, 2023 9:16 PM
> > 
> > On Wed, Jun 07, 2023 at 02:59:20AM +0000, Yoshihiro Shimoda wrote:
> > > Hello Serge,
> > >
> > > > From: Serge Semin, Sent: Monday, June 5, 2023 11:39 PM
> > > >
> > > > On Wed, May 10, 2023 at 03:22:31PM +0900, Yoshihiro Shimoda wrote:
> > > > > Add R-Car Gen4 PCIe Host support. This controller is based on
> > > > > Synopsys DesignWare PCIe, but this controller has vendor-specific
> > > > > registers so that requires initialization code like mode setting
> > > > > and retraining and so on.
> > > > >
> > > > > To reduce code delta, adds some helper functions which are used by
> > > > > both the host driver and the endpoint driver (which is added
> > > > > immediately afterwards) into a separate file.
> > > > >
> > > > > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > > > > ---
> > > > >  drivers/pci/controller/dwc/Kconfig            |   9 +
> > > > >  drivers/pci/controller/dwc/Makefile           |   2 +
> > > > >  .../pci/controller/dwc/pcie-rcar-gen4-host.c  | 141 +++++++++++++
> > > > >  drivers/pci/controller/dwc/pcie-rcar-gen4.c   | 190 ++++++++++++++++++
> > > > >  drivers/pci/controller/dwc/pcie-rcar-gen4.h   |  46 +++++
> > > > >  5 files changed, 388 insertions(+)
> > > > >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > > > >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > > > >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > > > >
> > > > > diff --git a/drivers/pci/controller/dwc/Kconfig b/drivers/pci/controller/dwc/Kconfig
> > > > > index ab96da43e0c2..64d4d37bc891 100644
> > > > > --- a/drivers/pci/controller/dwc/Kconfig
> > > > > +++ b/drivers/pci/controller/dwc/Kconfig
> > > > > @@ -415,4 +415,13 @@ config PCIE_VISCONTI_HOST
> > > > >  	  Say Y here if you want PCIe controller support on Toshiba Visconti SoC.
> > > > >  	  This driver supports TMPV7708 SoC.
> > > > >
> > > > > +config PCIE_RCAR_GEN4
> > > > > +	tristate "Renesas R-Car Gen4 PCIe Host controller"
> > > > > +	depends on ARCH_RENESAS || COMPILE_TEST
> > > > > +	depends on PCI_MSI
> > > > > +	select PCIE_DW_HOST
> > > > > +	help
> > > > > +	  Say Y here if you want PCIe host controller support on R-Car Gen4 SoCs.
> > > > > +	  This uses the DesignWare core.
> > > > > +
> > > > >  endmenu
> > > > > diff --git a/drivers/pci/controller/dwc/Makefile b/drivers/pci/controller/dwc/Makefile
> > > > > index bf5c311875a1..486cf706b53d 100644
> > > > > --- a/drivers/pci/controller/dwc/Makefile
> > > > > +++ b/drivers/pci/controller/dwc/Makefile
> > > > > @@ -26,6 +26,8 @@ obj-$(CONFIG_PCIE_TEGRA194) += pcie-tegra194.o
> > > > >  obj-$(CONFIG_PCIE_UNIPHIER) += pcie-uniphier.o
> > > > >  obj-$(CONFIG_PCIE_UNIPHIER_EP) += pcie-uniphier-ep.o
> > > > >  obj-$(CONFIG_PCIE_VISCONTI_HOST) += pcie-visconti.o
> > > > > +pcie-rcar-gen4-host-drv-objs := pcie-rcar-gen4.o pcie-rcar-gen4-host.o
> > > > > +obj-$(CONFIG_PCIE_RCAR_GEN4) += pcie-rcar-gen4-host-drv.o
> > > > >
> > > > >  # The following drivers are for devices that use the generic ACPI
> > > > >  # pci_root.c driver but don't support standard ECAM config access.
> > > > > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > > > > new file mode 100644
> > > > > index 000000000000..df7d80f1874f
> > > > > --- /dev/null
> > > > > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > > > > @@ -0,0 +1,141 @@
> > > > > +// SPDX-License-Identifier: GPL-2.0-only
> > > > > +/*
> > > > > + * PCIe host controller driver for Renesas R-Car Gen4 Series SoCs
> > > > > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > > > > + */
> > > > > +
> > > > > +#include <linux/delay.h>
> > > > > +#include <linux/interrupt.h>
> > > > > +#include <linux/module.h>
> > > > > +#include <linux/of_device.h>
> > > > > +#include <linux/pci.h>
> > > > > +#include <linux/platform_device.h>
> > > > > +
> > > > > +#include "pcie-rcar-gen4.h"
> > > > > +#include "pcie-designware.h"
> > > > > +
> > > > > +static int rcar_gen4_pcie_host_init(struct dw_pcie_rp *pp)
> > > > > +{
> > > > > +	struct dw_pcie *dw = to_dw_pcie_from_pp(pp);
> > > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > +	int ret;
> > > > > +	u32 val;
> > > > > +
> > > > > +	gpiod_set_value_cansleep(dw->pe_rst, 1);
> > > > > +
> > > > > +	ret = rcar_gen4_pcie_set_device_type(rcar, true, dw->num_lanes);
> > > > > +	if (ret < 0)
> > > > > +		return ret;
> > > > > +
> > > >
> > > > > +	dw_pcie_dbi_ro_wr_en(dw);
> > > >
> > > > Are you sure dw_pcie_dbi_ro_wr_en() and dw_pcie_dbi_ro_wr_dis() are
> > > > needed? In accordance with the DW PCIe Dual-mode HW manual the BARx
> > > > registers are W-only over the DBI2 map with no need in setting the
> > > > DBI_RO_WR_EN flag.
> > > >
> > > > Please check that on your hardware.
> > >
> > > You're correct. They are not needed. So, I'll drop this on v17.
> > >
> > > > > +
> > > > > +	/*
> > > > > +	 * According to the section 3.5.7.2 "RC Mode" in DWC PCIe Dual Mode
> > > > > +	 * Rev.5.20a, we should disable two BARs to avoid unnecessary memory
> > > > > +	 * assignment during device enumeration.
> > > > > +	 */
> > > > > +	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_0, 0x0);
> > > > > +	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_1, 0x0);
> > > > > +
> > > >
> > > > > +	dw_pcie_dbi_ro_wr_dis(dw);
> > > >
> > > > ditto
> > >
> > > I'll drop this too.
> > >
> > > > > +
> > > > > +	if (IS_ENABLED(CONFIG_PCI_MSI)) {
> > > > > +		/* Enable MSI interrupt signal */
> > > > > +		val = readl(rcar->base + PCIEINTSTS0EN);
> > > > > +		val |= MSI_CTRL_INT;
> > > > > +		writel(val, rcar->base + PCIEINTSTS0EN);
> > > > > +	}
> > > > > +
> > > > > +	msleep(100);	/* pe_rst requires 100msec delay */
> > > > > +
> > > > > +	gpiod_set_value_cansleep(dw->pe_rst, 0);
> > > > > +
> > > > > +	return 0;
> > > > > +}
> > > > > +
> > > > > +static const struct dw_pcie_host_ops rcar_gen4_pcie_host_ops = {
> > > > > +	.host_init = rcar_gen4_pcie_host_init,
> > > > > +};
> > > > > +
> > > > > +static int rcar_gen4_add_dw_pcie_rp(struct rcar_gen4_pcie *rcar,
> > > > > +				   struct platform_device *pdev)
> > > > > +{
> > > > > +	struct dw_pcie *dw = &rcar->dw;
> > > > > +	struct dw_pcie_rp *pp = &dw->pp;
> > > > > +
> > > > > +	pp->num_vectors = MAX_MSI_IRQS;
> > > > > +	pp->ops = &rcar_gen4_pcie_host_ops;
> > > > > +	dw_pcie_cap_set(dw, REQ_RES);
> > > > > +
> > > > > +	return dw_pcie_host_init(pp);
> > > > > +}
> > > > > +
> > > > > +static void rcar_gen4_remove_dw_pcie_rp(struct rcar_gen4_pcie *rcar)
> > > > > +{
> > > > > +	dw_pcie_host_deinit(&rcar->dw.pp);
> > > > > +	gpiod_set_value_cansleep(rcar->dw.pe_rst, 1);
> > > > > +}
> > > > > +
> > > > > +static int rcar_gen4_pcie_probe(struct platform_device *pdev)
> > > > > +{
> > > > > +	struct device *dev = &pdev->dev;
> > > > > +	struct rcar_gen4_pcie *rcar;
> > > > > +	int err;
> > > > > +
> > > > > +	rcar = rcar_gen4_pcie_devm_alloc(dev);
> > > > > +	if (!rcar)
> > > > > +		return -ENOMEM;
> > > > > +
> > > > > +	err = rcar_gen4_pcie_get_resources(rcar, pdev);
> > > > > +	if (err < 0) {
> > > > > +		dev_err(dev, "Failed to request resource: %d\n", err);
> > > > > +		return err;
> > > > > +	}
> > > > > +
> > > > > +	platform_set_drvdata(pdev, rcar);
> > > > > +
> > > > > +	err = rcar_gen4_pcie_prepare(rcar);
> > > > > +	if (err < 0)
> > > > > +		return err;
> > > > > +
> > > > > +	rcar->needs_retrain = true;
> > > > > +	err = rcar_gen4_add_dw_pcie_rp(rcar, pdev);
> > > > > +	if (err < 0)
> > > > > +		goto err_add;
> > > > > +
> > > > > +	return 0;
> > > > > +
> > > > > +err_add:
> > > > > +	rcar_gen4_pcie_unprepare(rcar);
> > > > > +
> > > > > +	return err;
> > > > > +}
> > > > > +
> > > > > +static int rcar_gen4_pcie_remove(struct platform_device *pdev)
> > > > > +{
> > > > > +	struct rcar_gen4_pcie *rcar = platform_get_drvdata(pdev);
> > > > > +
> > > > > +	rcar_gen4_remove_dw_pcie_rp(rcar);
> > > > > +	rcar_gen4_pcie_unprepare(rcar);
> > > > > +
> > > > > +	return 0;
> > > > > +}
> > > > > +
> > > > > +static const struct of_device_id rcar_gen4_pcie_of_match[] = {
> > > > > +	{ .compatible = "renesas,rcar-gen4-pcie", },
> > > > > +	{},
> > > > > +};
> > > > > +
> > > > > +static struct platform_driver rcar_gen4_pcie_driver = {
> > > > > +	.driver = {
> > > > > +		.name = "pcie-rcar-gen4",
> > > > > +		.of_match_table = rcar_gen4_pcie_of_match,
> > > > > +		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
> > > > > +	},
> > > > > +	.probe = rcar_gen4_pcie_probe,
> > > > > +	.remove = rcar_gen4_pcie_remove,
> > > > > +};
> > > > > +module_platform_driver(rcar_gen4_pcie_driver);
> > > > > +
> > > > > +MODULE_DESCRIPTION("Renesas R-Car Gen4 PCIe host controller driver");
> > > > > +MODULE_LICENSE("GPL");
> > > > > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.c b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > > > > new file mode 100644
> > > > > index 000000000000..35923fda8ed5
> > > > > --- /dev/null
> > > > > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > > > > @@ -0,0 +1,190 @@
> > > > > +// SPDX-License-Identifier: GPL-2.0-only
> > > > > +/*
> > > > > + * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
> > > > > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > > > > + */
> > > > > +
> > > > > +#include <linux/delay.h>
> > > > > +#include <linux/io.h>
> > > > > +#include <linux/of_device.h>
> > > > > +#include <linux/pci.h>
> > > > > +#include <linux/pm_runtime.h>
> > > > > +#include <linux/reset.h>
> > > > > +
> > > > > +#include "pcie-rcar-gen4.h"
> > > > > +#include "pcie-designware.h"
> > > > > +
> > > > > +/* Renesas-specific */
> > > > > +#define PCIERSTCTRL1		0x0014
> > > > > +#define  APP_HOLD_PHY_RST	BIT(16)
> > > > > +#define  APP_LTSSM_ENABLE	BIT(0)
> > > > > +
> > > > > +#define RETRAIN_MAX_CHECK	10
> > > > > +#define RETRAIN_MAX_RETRIES	10
> > > > > +
> > > > > +static void rcar_gen4_pcie_ltssm_enable(struct rcar_gen4_pcie *rcar,
> > > > > +					bool enable)
> > > > > +{
> > > > > +	u32 val;
> > > > > +
> > > > > +	val = readl(rcar->base + PCIERSTCTRL1);
> > > > > +	if (enable) {
> > > > > +		val |= APP_LTSSM_ENABLE;
> > > >
> > > > > +		val &= ~APP_HOLD_PHY_RST;
> > > >
> > > > What about moving the APP_HOLD_PHY_RST de-assertion to the
> > > > rcar_gen4_pcie_set_device_type() method? In accordance with the
> > > > "3.1 Initialization" chapter it's supposed to be done before
> > > > performing the DBI programming and activating the link training.
> > >
> > 
> > > IIUC, the "3.1 Initialization" said app_hold_phy_rst = 1 before
> > > performing the DBI programming. So, it is assertion. Also, the SoC
> > > documentation described the initializing procedure as the follows:
> > >  app_ltssm_enable = 1
> > >  app_hold_phy_rst = 0
> > > So, I would like to keep them in the function.
> > 
> > Indeed. I was wrong. Sorry for the misleading comment.
> 
> No problem. Thank you for the confirmation!
> 
> > >
> > > > > +	} else {
> > > > > +		val &= ~APP_LTSSM_ENABLE;
> > > > > +		val |= APP_HOLD_PHY_RST;
> > > > > +	}
> > > > > +	writel(val, rcar->base + PCIERSTCTRL1);
> > > > > +}
> > > > > +
> > > > > +static bool rcar_gen4_pcie_check_retrain_link(struct dw_pcie *dw)
> > > > > +{
> > > > > +	u8 offset = dw_pcie_find_capability(dw, PCI_CAP_ID_EXP);
> > > > > +	u32 lnkcap = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCAP);
> > > > > +	u32 lnkctl = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCTL);
> > > > > +	u16 lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> > > > > +	int i;
> > > > > +
> > > >
> > > > > +	if ((lnksta & PCI_EXP_LNKSTA_CLS) == (lnkcap & PCI_EXP_LNKCAP_SLS))
> > > > > +		return true;
> > > > > +
> > > > > +	lnkctl |= PCI_EXP_LNKCTL_RL;
> > > > > +	dw_pcie_writel_dbi(dw, offset + PCI_EXP_LNKCTL, lnkctl);
> > > > > +
> > > > > +	for (i = 0; i < RETRAIN_MAX_CHECK; i++) {
> > > > > +		lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> > > > > +		if (lnksta & PCI_EXP_LNKSTA_LT)
> > > > > +			return true;
> > > > > +		usleep_range(1000, 1100);
> > > > > +	}
> > > >
> > > > I'll ask one more time because you didn't respond to my previous note
> > > > about this.
> > >
> > > I'm sorry. I completely overlooked the previous note.
> > >
> > > > Are you sure that this is needed? Did you try
> > > > the approach described in "3.13 Gen2/3/4/5 Speed Modes" with
> > > > de-asserting/asserting the GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag?
> > >
> > > I tried this setting, but it doesn't work. I'll investigate this setting more.
> > >
> > > > I keep asking because the same problem we used to have on our hardware
> > > > until we found out that the DIRECT_SPEED_CHANGE flag helped to train
> > > > the link right to the speed specified in the capabilities.
> > > >
> > > > So here is what presumably you'll need to do (based on the
> > > > "3.1 Initialization" and "3.13 Gen2/3/4/5 Speed Modes" chapters of
> > > > the DW PCIe DM hw-manual):
> > > > 1. Make sure the controller is in the power-down/reset state.
> > > > 2. Select device_type (EP or RP).
> > > > 3. De-assert the controller reset.
> > > > 4. Clear PHY-reset flag in the app registers.
> > > > 5. Perform some controller initializations.
> > > > 6. Enable LTSSM to start link training.
> > > > 7. Set GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag one more time.
> > > >
> > > > 1-4 are supposed to be done in rcar_gen4_pcie_host_init().
> > > > 5 is performed in the framework of the DW PCIe core driver.
> > > > 6-7 should be done in rcar_gen4_pcie_start_link().
> > > >
> > > > Note 1. GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag is already set on stage
> > > > 5 in the framework of the dw_pcie_setup_rc() method. But in our case
> > > > it only caused having the Gen.2 link speed. Adding stage 7 helped to
> > > > get stable Gen.3 link. So please try the denoted approach. If it works
> > > > what about adding stage 7 twice in order to get Gen.4 speed?
> > > > (waiting for the DIRECT_SPEED_CHANGE flag being auto-cleared and then
> > > > set it up again?)
> > > >
> > > > Note 2. GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag is defined as
> > > > PCIE_LINK_WIDTH_SPEED_CONTROL.PORT_LOGIC_SPEED_CHANGE macros in the DW
> > > > PCIe core driver.
> > > >
> > > > Note 3. If what is suggested above works well then you won't need to
> > > > have the heavy rcar_gen4_pcie_check_retrain_link() method in the way
> > > > you have it defined.
> > >
> > > Thank you very much for your comments!
> > 
> > Please see the GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE description for details
> > of how the flag is supposed to be de-asserted and asserted in order to
> > initiate the direct speed change.
> 
> After I modified the start_link() like below, it also works. Is the code
> acceptable? (Sorry all tabs are replaced to spaces...)

Looks good, but still there are some questions.

> ----------------------------------------------------------------------------
> static bool rcar_gen4_pcie_check_current_link(struct dw_pcie *dw)
> {
>         u8 offset = dw_pcie_find_capability(dw, PCI_CAP_ID_EXP);
>         u32 lnkcap = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCAP);
>         u16 lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> 

>         if ((lnksta & PCI_EXP_LNKSTA_CLS) == (lnkcap & PCI_EXP_LNKCAP_SLS))
>                 return true;

AFAICS depending on the link partner speed capabilities this may never
happen.

PCI_EXP_LNKCAP_SLS - Max Link Speed. This field indicates the maximum Link
speed of the associated Port.
PCI_EXP_LNKSTA_CLS - Current Link Speed. This field indicates the negotiated
Link speed of the given PCI Express Link

What if a link partner has the speed capability weaker than the link
speed of the Root Port? If so then the current link speed will never
reach the max link speed value.

Of course this can be fixed by specifying a correct "max-link-speed"
property, but what if a platform has a cold-swappable port connected to
the root port? Since any device can be attached you'll never be able
to predict its capabilities beforahead.

> 
>         return false;
> }
> 
> static void rcar_gen4_pcie_speed_change(struct dw_pcie *dw)
> {
>         u32 val;
> 
>         val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
>         val &= ~PORT_LOGIC_SPEED_CHANGE;
>         dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
> 
>         val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
>         val |= PORT_LOGIC_SPEED_CHANGE;
>         dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
> }
> 
> static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> {
>         struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
>         int i;
> 
>         rcar_gen4_pcie_ltssm_enable(rcar, true);
> 
>         /*
>          * Require direct speed change here. Otherwise RDLH_LINK_UP of
>          * PCIEINTSTS0 which is this controller specific register may not
>          * be set.
>          */

>         if (rcar->needs_speed_change) {

Seeing this is specified for the root port only what about
replacing the statement with just test whether the rcar_gen4_pcie.mode ==
DW_PCIE_RC_TYPE? Thus you'll be ablt to drop the needs_speed_change field.

BTW Just curious. Why is the loop below enabled for the Root Port
only? What about the end-point controller? It's the same hardware
after all..

>                 for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
>                         rcar_gen4_pcie_speed_change(dw);
>                         msleep(100);
>                         if (rcar_gen4_pcie_check_current_link(dw))
>                                 return 0;
>                 }

Did you trace how many iterations this loop normally takes? Is it
constant or varies for the same platform setup and a connected link
partner? Does the number of iterations depend on the target link speed
specified via the "max-link-speed" property?

I am just trying to understand whether we can completely get rid from
the rcar_gen4_pcie_check_current_link() method and have it replaced
with several rcar_gen4_pcie_speed_change() calls. The current link
state would have been checked in the framework of the
dw_pcie_wait_for_link() method which calls dw_pcie_link_up() and your
rcar_gen4_pcie_link_up() in order to make sure the link is actually up.

-Serge(y)

> 
>                 return -ETIMEDOUT;      /* Failed */
>         }
> ------------------------------------------------------------------
> 
> > >
> > > > > +
> > > > > +	return false;
> > > > > +}
> > > > > +
> > > > > +static int rcar_gen4_pcie_link_up(struct dw_pcie *dw)
> > > > > +{
> > > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > +	u32 val, mask;
> > > > > +
> > > > > +	val = readl(rcar->base + PCIEINTSTS0);
> > > > > +	mask = RDLH_LINK_UP | SMLH_LINK_UP;
> > > > > +
> > > > > +	return (val & mask) == mask;
> > > > > +}
> > > > > +
> > > > > +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > > > +{
> > > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > +	int i;
> > > > > +
> > > > > +	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > > +
> > > > > +	/*
> > > > > +	 * Require retraining here. Otherwise RDLH_LINK_UP of PCIEINTSTS0 which
> > > > > +	 * is this controller specific register may not be set.
> > > > > +	 */
> > > > > +	if (rcar->needs_retrain) {
> > > > > +		for (i = 0; i < RETRAIN_MAX_RETRIES; i++) {
> > > > > +			if (rcar_gen4_pcie_check_retrain_link(dw))
> > > > > +				return 0;
> > > > > +			msleep(100);
> > > > > +		}
> > > > > +
> > > > > +		return -ETIMEDOUT;	/* Failed */
> > > > > +	}
> > > > > +
> > > > > +	return 0;
> > > > > +}
> > > > > +
> > > > > +static void rcar_gen4_pcie_stop_link(struct dw_pcie *dw)
> > > > > +{
> > > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > +
> > > > > +	rcar_gen4_pcie_ltssm_enable(rcar, false);
> > > > > +}
> > > > > +
> > > >
> > > > > +int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
> > > > > +				   int num_lanes)
> > > >
> > > > 1. Number of lanes is already defined in the rcar_gen4_pcie.dw.num_lanes field.
> > > > What about using it from there instead of passing it as an argument?
> > > > 2. DW PCIe core driver has a very handy enum defined:
> > > > dw_pcie_device_mode. It describes the controller modes (End-point,
> > > > Root port, etc). What about adding the mode field right to the
> > > > rcar_gen4_pcie structure and initializing it in someplace in probe() ?
> > > > 3. Based on the function semantic it's better to be named as something
> > > > like rcar_gen4_pcie_init_device() or even rcar_gen4_pcie_basic_init().
> > >
> > > Thank you for your comments! I'll modify the function.
> > >
> > > >
> > > > > +{
> > > > > +	u32 val;
> > > > > +
> > > >
> > > > > +	/* Note: Assume the rcar->rst which is Cold-reset is asserted here */
> > > >
> > > > What about directly asserting it here then? In accordance with the DW
> > > > PCIe DM manual the "device_type" input must be set before the DM
> > > > controller is powered up (basically un-reset). What if the controller
> > > > reset is already de-asserted, but you are going to changes its mode?
> > > > In that case the mode won't be changed and you'll end up with
> > > > unpredictable results.
> > >
> > > Thank you for your comment. We should add asserting it here as you mentioned.
> > >
> > > > > +	val = readl(rcar->base + PCIEMSR0);
> > > > > +	if (rc)
> > > > > +		val |= DEVICE_TYPE_RC;
> > > > > +	else
> > > > > +		val |= DEVICE_TYPE_EP;
> > > > > +
> > > > > +	if (num_lanes < 4)
> > > > > +		val |= BIFUR_MOD_SET_ON;
> > > > > +
> > > > > +	writel(val, rcar->base + PCIEMSR0);
> > > > > +
> > > > > +	return reset_control_deassert(rcar->rst);
> > > > > +}
> > > > > +
> > > > > +int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *rcar)
> > > > > +{
> > > > > +	struct device *dev = rcar->dw.dev;
> > > > > +	int err;
> > > > > +
> > > > > +	pm_runtime_enable(dev);
> > > > > +	err = pm_runtime_resume_and_get(dev);
> > > > > +	if (err < 0) {
> > > > > +		dev_err(dev, "Failed to resume/get Runtime PM\n");
> > > > > +		pm_runtime_disable(dev);
> > > > > +	}
> > > > > +
> > > > > +	return err;
> > > > > +}
> > > > > +
> > > > > +void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *rcar)
> > > > > +{
> > > > > +	struct device *dev = rcar->dw.dev;
> > > > > +
> > > > > +	if (!reset_control_status(rcar->rst))
> > > > > +		reset_control_assert(rcar->rst);
> > > > > +	pm_runtime_put(dev);
> > > > > +	pm_runtime_disable(dev);
> > > > > +}
> > > > > +
> > > > > +int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
> > > > > +				 struct platform_device *pdev)
> > > > > +{
> > > > > +	struct device *dev = rcar->dw.dev;
> > > > > +
> > > > > +	/* Renesas-specific registers */
> > > > > +	rcar->base = devm_platform_ioremap_resource_byname(pdev, "app");
> > > > > +	if (IS_ERR(rcar->base))
> > > > > +		return PTR_ERR(rcar->base);
> > > > > +
> > > >
> > > > > +	rcar->rst = devm_reset_control_get(dev, NULL);
> > > > > +	if (IS_ERR(rcar->rst)) {
> > > > > +		dev_err(dev, "Failed to get Cold-reset\n");
> > > >
> > > > So AFAICS your platform is equipped with the DWC_pcie_clkrst.v module.
> > > > Thus all the resets are appropriately cleared by a single flag:
> > > > power_up_rst_n. What about using the named reset in this case with the
> > > > "pwr" name? Thus you'll be able to drop the manual
> > > > devm_reset_control_get() invocation and instead use the reset-resources
> > > > requested in the framework of the generic dw_pcie_get_resources()
> > > > method? Note you'll need to move the dw_pcie_cap_set(dw, REQ_RES);
> > > > statement to rcar_gen4_pcie_devm_alloc() then and drop the
> > > > rcar_gen4_pcie.rst field afterwords.
> > >
> > > Thank you for your suggestion! Using "pwr" can work on my environment.
> > >
> > > > By the way I don't see you requesting and enabling the reference
> > > > clock in your driver but the bindings imply the clock source. How
> > > > come?
> > >
> > 
> > > For now, I used gpio-hog to enable the reference clock. But, it seem
> > > I should use "ref" clock for it. So, I'll fix it too.
> > 
> > Not sure what gpio-hog you are talking about. Do you mean the pe_rst
> > signal or some another gpio? I failed to see of how pe_rst is
> > connected to the clock source. In anyway directly handling the clock
> > source would be more portable choice.
> 
> Sorry for lacking information. I described a gpio node like below
> and then the gpio will be high automatically, and the reference clock
> will be output. But, this is completely independent from pci.
> ---
> &gpio2 {
>         pci-clkreq0-hog {
>                 gpio-hog;
>                 gpios = <15 GPIO_ACTIVE_LOW>;
>                 output-high;
>         };
> };
> ---
> 
> Now I could implement the clock handling by using "gpio-gate-clock".
> So, I'll drop the gpio-hog for the reference clock.
> 
> Best regards,
> Yoshihiro Shimoda
> 
> > -Serge(y)
> > 
> > >
> > > > > +		return PTR_ERR(rcar->rst);
> > > > > +	}
> > > > > +
> > > > > +	return 0;
> > > > > +}
> > > > > +
> > > > > +static const struct dw_pcie_ops dw_pcie_ops = {
> > > > > +	.start_link = rcar_gen4_pcie_start_link,
> > > > > +	.stop_link = rcar_gen4_pcie_stop_link,
> > > > > +	.link_up = rcar_gen4_pcie_link_up,
> > > > > +};
> > > > > +
> > > > > +struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev)
> > > > > +{
> > > > > +	struct rcar_gen4_pcie *rcar;
> > > > > +
> > > > > +	rcar = devm_kzalloc(dev, sizeof(*rcar), GFP_KERNEL);
> > > > > +	if (!rcar)
> > > > > +		return NULL;
> > > > > +
> > > > > +	rcar->dw.dev = dev;
> > > > > +	rcar->dw.ops = &dw_pcie_ops;
> > > > > +	dw_pcie_cap_set(&rcar->dw, EDMA_UNROLL);
> > > > > +
> > > > > +	return rcar;
> > > > > +}
> > > > > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.h b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > > > > new file mode 100644
> > > > > index 000000000000..fec3f18609f4
> > > > > --- /dev/null
> > > > > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > > > > @@ -0,0 +1,46 @@
> > > > > +/* SPDX-License-Identifier: GPL-2.0-only */
> > > > > +/*
> > > > > + * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
> > > > > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > > > > + */
> > > > > +
> > > > > +#ifndef _PCIE_RCAR_GEN4_H_
> > > > > +#define _PCIE_RCAR_GEN4_H_
> > > > > +
> > > > > +#include <linux/io.h>
> > > > > +#include <linux/pci.h>
> > > > > +#include <linux/reset.h>
> > > > > +
> > > > > +#include "pcie-designware.h"
> > > > > +
> > > > > +/* Renesas-specific */
> > > > > +#define PCIEMSR0		0x0000
> > > > > +#define  BIFUR_MOD_SET_ON	BIT(0)
> > > > > +#define  DEVICE_TYPE_EP		0
> > > > > +#define  DEVICE_TYPE_RC		BIT(4)
> > > > > +
> > > > > +#define PCIEINTSTS0		0x0084
> > > > > +#define PCIEINTSTS0EN		0x0310
> > > > > +#define  MSI_CTRL_INT		BIT(26)
> > > > > +#define  SMLH_LINK_UP		BIT(7)
> > > > > +#define  RDLH_LINK_UP		BIT(6)
> > > > > +#define PCIEDMAINTSTSEN		0x0314
> > > > > +#define  PCIEDMAINTSTSEN_INIT	GENMASK(15, 0)
> > > > > +
> > > >
> > > > > +struct rcar_gen4_pcie {
> > > >
> > > > As I mentioned above this structure can be extended with the enum
> > > > dw_pcie_device_mode field thus dropping the boolean argument from the
> > > > rcar_gen4_pcie_set_device_type() method.
> > >
> > > I got it. I'll fix this.
> > >
> > > > > +	struct dw_pcie		dw;
> > > >
> > > > As I already mentioned above the dw.num_lanes could be used instead of
> > > > passing it as the rcar_gen4_pcie_set_device_type() argument.
> > >
> > > I'll fix this too.
> > >
> > > Best regards,
> > > Yoshihiro Shimoda
> > >
> > > > -Serge(y)
> > > >
> > > > > +	void __iomem		*base;
> > > > > +	struct reset_control	*rst;
> > > > > +	bool			needs_retrain;
> > > > > +};
> > > > > +#define to_rcar_gen4_pcie(x)	dev_get_drvdata((x)->dev)
> > > > > +
> > > > > +int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
> > > > > +				   int num_lanes);
> > > > > +int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *pcie);
> > > > > +void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *pcie);
> > > > > +int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
> > > > > +				 struct platform_device *pdev);
> > > > > +struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev);
> > > > > +
> > > > > +#endif /* _PCIE_RCAR_GEN4_H_ */
> > > > > --
> > > > > 2.25.1
> > > > >

RE: [PATCH v16 19/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support

[Yoshihiro Shimoda]

Hello Serge,

> From: Serge Semin, Sent: Thursday, June 8, 2023 9:11 PM
> 
> On Thu, Jun 08, 2023 at 08:47:16AM +0000, Yoshihiro Shimoda wrote:
> > Hello Serge,
> >
> > > From: Serge Semin, Sent: Wednesday, June 7, 2023 9:16 PM
> > >
> > > On Wed, Jun 07, 2023 at 02:59:20AM +0000, Yoshihiro Shimoda wrote:
> > > > Hello Serge,
> > > >
> > > > > From: Serge Semin, Sent: Monday, June 5, 2023 11:39 PM
> > > > >
> > > > > On Wed, May 10, 2023 at 03:22:31PM +0900, Yoshihiro Shimoda wrote:
> > > > > > Add R-Car Gen4 PCIe Host support. This controller is based on
> > > > > > Synopsys DesignWare PCIe, but this controller has vendor-specific
> > > > > > registers so that requires initialization code like mode setting
> > > > > > and retraining and so on.
> > > > > >
> > > > > > To reduce code delta, adds some helper functions which are used by
> > > > > > both the host driver and the endpoint driver (which is added
> > > > > > immediately afterwards) into a separate file.
> > > > > >
> > > > > > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > > > > > ---
> > > > > >  drivers/pci/controller/dwc/Kconfig            |   9 +
> > > > > >  drivers/pci/controller/dwc/Makefile           |   2 +
> > > > > >  .../pci/controller/dwc/pcie-rcar-gen4-host.c  | 141 +++++++++++++
> > > > > >  drivers/pci/controller/dwc/pcie-rcar-gen4.c   | 190 ++++++++++++++++++
> > > > > >  drivers/pci/controller/dwc/pcie-rcar-gen4.h   |  46 +++++
> > > > > >  5 files changed, 388 insertions(+)
> > > > > >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > > > > >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > > > > >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > > > > >
> > > > > > diff --git a/drivers/pci/controller/dwc/Kconfig b/drivers/pci/controller/dwc/Kconfig
> > > > > > index ab96da43e0c2..64d4d37bc891 100644
> > > > > > --- a/drivers/pci/controller/dwc/Kconfig
> > > > > > +++ b/drivers/pci/controller/dwc/Kconfig
> > > > > > @@ -415,4 +415,13 @@ config PCIE_VISCONTI_HOST
> > > > > >  	  Say Y here if you want PCIe controller support on Toshiba Visconti SoC.
> > > > > >  	  This driver supports TMPV7708 SoC.
> > > > > >
> > > > > > +config PCIE_RCAR_GEN4
> > > > > > +	tristate "Renesas R-Car Gen4 PCIe Host controller"
> > > > > > +	depends on ARCH_RENESAS || COMPILE_TEST
> > > > > > +	depends on PCI_MSI
> > > > > > +	select PCIE_DW_HOST
> > > > > > +	help
> > > > > > +	  Say Y here if you want PCIe host controller support on R-Car Gen4 SoCs.
> > > > > > +	  This uses the DesignWare core.
> > > > > > +
> > > > > >  endmenu
> > > > > > diff --git a/drivers/pci/controller/dwc/Makefile b/drivers/pci/controller/dwc/Makefile
> > > > > > index bf5c311875a1..486cf706b53d 100644
> > > > > > --- a/drivers/pci/controller/dwc/Makefile
> > > > > > +++ b/drivers/pci/controller/dwc/Makefile
> > > > > > @@ -26,6 +26,8 @@ obj-$(CONFIG_PCIE_TEGRA194) += pcie-tegra194.o
> > > > > >  obj-$(CONFIG_PCIE_UNIPHIER) += pcie-uniphier.o
> > > > > >  obj-$(CONFIG_PCIE_UNIPHIER_EP) += pcie-uniphier-ep.o
> > > > > >  obj-$(CONFIG_PCIE_VISCONTI_HOST) += pcie-visconti.o
> > > > > > +pcie-rcar-gen4-host-drv-objs := pcie-rcar-gen4.o pcie-rcar-gen4-host.o
> > > > > > +obj-$(CONFIG_PCIE_RCAR_GEN4) += pcie-rcar-gen4-host-drv.o
> > > > > >
> > > > > >  # The following drivers are for devices that use the generic ACPI
> > > > > >  # pci_root.c driver but don't support standard ECAM config access.
> > > > > > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > > > > > new file mode 100644
> > > > > > index 000000000000..df7d80f1874f
> > > > > > --- /dev/null
> > > > > > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > > > > > @@ -0,0 +1,141 @@
> > > > > > +// SPDX-License-Identifier: GPL-2.0-only
> > > > > > +/*
> > > > > > + * PCIe host controller driver for Renesas R-Car Gen4 Series SoCs
> > > > > > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > > > > > + */
> > > > > > +
> > > > > > +#include <linux/delay.h>
> > > > > > +#include <linux/interrupt.h>
> > > > > > +#include <linux/module.h>
> > > > > > +#include <linux/of_device.h>
> > > > > > +#include <linux/pci.h>
> > > > > > +#include <linux/platform_device.h>
> > > > > > +
> > > > > > +#include "pcie-rcar-gen4.h"
> > > > > > +#include "pcie-designware.h"
> > > > > > +
> > > > > > +static int rcar_gen4_pcie_host_init(struct dw_pcie_rp *pp)
> > > > > > +{
> > > > > > +	struct dw_pcie *dw = to_dw_pcie_from_pp(pp);
> > > > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > > +	int ret;
> > > > > > +	u32 val;
> > > > > > +
> > > > > > +	gpiod_set_value_cansleep(dw->pe_rst, 1);
> > > > > > +
> > > > > > +	ret = rcar_gen4_pcie_set_device_type(rcar, true, dw->num_lanes);
> > > > > > +	if (ret < 0)
> > > > > > +		return ret;
> > > > > > +
> > > > >
> > > > > > +	dw_pcie_dbi_ro_wr_en(dw);
> > > > >
> > > > > Are you sure dw_pcie_dbi_ro_wr_en() and dw_pcie_dbi_ro_wr_dis() are
> > > > > needed? In accordance with the DW PCIe Dual-mode HW manual the BARx
> > > > > registers are W-only over the DBI2 map with no need in setting the
> > > > > DBI_RO_WR_EN flag.
> > > > >
> > > > > Please check that on your hardware.
> > > >
> > > > You're correct. They are not needed. So, I'll drop this on v17.
> > > >
> > > > > > +
> > > > > > +	/*
> > > > > > +	 * According to the section 3.5.7.2 "RC Mode" in DWC PCIe Dual Mode
> > > > > > +	 * Rev.5.20a, we should disable two BARs to avoid unnecessary memory
> > > > > > +	 * assignment during device enumeration.
> > > > > > +	 */
> > > > > > +	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_0, 0x0);
> > > > > > +	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_1, 0x0);
> > > > > > +
> > > > >
> > > > > > +	dw_pcie_dbi_ro_wr_dis(dw);
> > > > >
> > > > > ditto
> > > >
> > > > I'll drop this too.
> > > >
> > > > > > +
> > > > > > +	if (IS_ENABLED(CONFIG_PCI_MSI)) {
> > > > > > +		/* Enable MSI interrupt signal */
> > > > > > +		val = readl(rcar->base + PCIEINTSTS0EN);
> > > > > > +		val |= MSI_CTRL_INT;
> > > > > > +		writel(val, rcar->base + PCIEINTSTS0EN);
> > > > > > +	}
> > > > > > +
> > > > > > +	msleep(100);	/* pe_rst requires 100msec delay */
> > > > > > +
> > > > > > +	gpiod_set_value_cansleep(dw->pe_rst, 0);
> > > > > > +
> > > > > > +	return 0;
> > > > > > +}
> > > > > > +
> > > > > > +static const struct dw_pcie_host_ops rcar_gen4_pcie_host_ops = {
> > > > > > +	.host_init = rcar_gen4_pcie_host_init,
> > > > > > +};
> > > > > > +
> > > > > > +static int rcar_gen4_add_dw_pcie_rp(struct rcar_gen4_pcie *rcar,
> > > > > > +				   struct platform_device *pdev)
> > > > > > +{
> > > > > > +	struct dw_pcie *dw = &rcar->dw;
> > > > > > +	struct dw_pcie_rp *pp = &dw->pp;
> > > > > > +
> > > > > > +	pp->num_vectors = MAX_MSI_IRQS;
> > > > > > +	pp->ops = &rcar_gen4_pcie_host_ops;
> > > > > > +	dw_pcie_cap_set(dw, REQ_RES);
> > > > > > +
> > > > > > +	return dw_pcie_host_init(pp);
> > > > > > +}
> > > > > > +
> > > > > > +static void rcar_gen4_remove_dw_pcie_rp(struct rcar_gen4_pcie *rcar)
> > > > > > +{
> > > > > > +	dw_pcie_host_deinit(&rcar->dw.pp);
> > > > > > +	gpiod_set_value_cansleep(rcar->dw.pe_rst, 1);
> > > > > > +}
> > > > > > +
> > > > > > +static int rcar_gen4_pcie_probe(struct platform_device *pdev)
> > > > > > +{
> > > > > > +	struct device *dev = &pdev->dev;
> > > > > > +	struct rcar_gen4_pcie *rcar;
> > > > > > +	int err;
> > > > > > +
> > > > > > +	rcar = rcar_gen4_pcie_devm_alloc(dev);
> > > > > > +	if (!rcar)
> > > > > > +		return -ENOMEM;
> > > > > > +
> > > > > > +	err = rcar_gen4_pcie_get_resources(rcar, pdev);
> > > > > > +	if (err < 0) {
> > > > > > +		dev_err(dev, "Failed to request resource: %d\n", err);
> > > > > > +		return err;
> > > > > > +	}
> > > > > > +
> > > > > > +	platform_set_drvdata(pdev, rcar);
> > > > > > +
> > > > > > +	err = rcar_gen4_pcie_prepare(rcar);
> > > > > > +	if (err < 0)
> > > > > > +		return err;
> > > > > > +
> > > > > > +	rcar->needs_retrain = true;
> > > > > > +	err = rcar_gen4_add_dw_pcie_rp(rcar, pdev);
> > > > > > +	if (err < 0)
> > > > > > +		goto err_add;
> > > > > > +
> > > > > > +	return 0;
> > > > > > +
> > > > > > +err_add:
> > > > > > +	rcar_gen4_pcie_unprepare(rcar);
> > > > > > +
> > > > > > +	return err;
> > > > > > +}
> > > > > > +
> > > > > > +static int rcar_gen4_pcie_remove(struct platform_device *pdev)
> > > > > > +{
> > > > > > +	struct rcar_gen4_pcie *rcar = platform_get_drvdata(pdev);
> > > > > > +
> > > > > > +	rcar_gen4_remove_dw_pcie_rp(rcar);
> > > > > > +	rcar_gen4_pcie_unprepare(rcar);
> > > > > > +
> > > > > > +	return 0;
> > > > > > +}
> > > > > > +
> > > > > > +static const struct of_device_id rcar_gen4_pcie_of_match[] = {
> > > > > > +	{ .compatible = "renesas,rcar-gen4-pcie", },
> > > > > > +	{},
> > > > > > +};
> > > > > > +
> > > > > > +static struct platform_driver rcar_gen4_pcie_driver = {
> > > > > > +	.driver = {
> > > > > > +		.name = "pcie-rcar-gen4",
> > > > > > +		.of_match_table = rcar_gen4_pcie_of_match,
> > > > > > +		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
> > > > > > +	},
> > > > > > +	.probe = rcar_gen4_pcie_probe,
> > > > > > +	.remove = rcar_gen4_pcie_remove,
> > > > > > +};
> > > > > > +module_platform_driver(rcar_gen4_pcie_driver);
> > > > > > +
> > > > > > +MODULE_DESCRIPTION("Renesas R-Car Gen4 PCIe host controller driver");
> > > > > > +MODULE_LICENSE("GPL");
> > > > > > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.c b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > > > > > new file mode 100644
> > > > > > index 000000000000..35923fda8ed5
> > > > > > --- /dev/null
> > > > > > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > > > > > @@ -0,0 +1,190 @@
> > > > > > +// SPDX-License-Identifier: GPL-2.0-only
> > > > > > +/*
> > > > > > + * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
> > > > > > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > > > > > + */
> > > > > > +
> > > > > > +#include <linux/delay.h>
> > > > > > +#include <linux/io.h>
> > > > > > +#include <linux/of_device.h>
> > > > > > +#include <linux/pci.h>
> > > > > > +#include <linux/pm_runtime.h>
> > > > > > +#include <linux/reset.h>
> > > > > > +
> > > > > > +#include "pcie-rcar-gen4.h"
> > > > > > +#include "pcie-designware.h"
> > > > > > +
> > > > > > +/* Renesas-specific */
> > > > > > +#define PCIERSTCTRL1		0x0014
> > > > > > +#define  APP_HOLD_PHY_RST	BIT(16)
> > > > > > +#define  APP_LTSSM_ENABLE	BIT(0)
> > > > > > +
> > > > > > +#define RETRAIN_MAX_CHECK	10
> > > > > > +#define RETRAIN_MAX_RETRIES	10
> > > > > > +
> > > > > > +static void rcar_gen4_pcie_ltssm_enable(struct rcar_gen4_pcie *rcar,
> > > > > > +					bool enable)
> > > > > > +{
> > > > > > +	u32 val;
> > > > > > +
> > > > > > +	val = readl(rcar->base + PCIERSTCTRL1);
> > > > > > +	if (enable) {
> > > > > > +		val |= APP_LTSSM_ENABLE;
> > > > >
> > > > > > +		val &= ~APP_HOLD_PHY_RST;
> > > > >
> > > > > What about moving the APP_HOLD_PHY_RST de-assertion to the
> > > > > rcar_gen4_pcie_set_device_type() method? In accordance with the
> > > > > "3.1 Initialization" chapter it's supposed to be done before
> > > > > performing the DBI programming and activating the link training.
> > > >
> > >
> > > > IIUC, the "3.1 Initialization" said app_hold_phy_rst = 1 before
> > > > performing the DBI programming. So, it is assertion. Also, the SoC
> > > > documentation described the initializing procedure as the follows:
> > > >  app_ltssm_enable = 1
> > > >  app_hold_phy_rst = 0
> > > > So, I would like to keep them in the function.
> > >
> > > Indeed. I was wrong. Sorry for the misleading comment.
> >
> > No problem. Thank you for the confirmation!
> >
> > > >
> > > > > > +	} else {
> > > > > > +		val &= ~APP_LTSSM_ENABLE;
> > > > > > +		val |= APP_HOLD_PHY_RST;
> > > > > > +	}
> > > > > > +	writel(val, rcar->base + PCIERSTCTRL1);
> > > > > > +}
> > > > > > +
> > > > > > +static bool rcar_gen4_pcie_check_retrain_link(struct dw_pcie *dw)
> > > > > > +{
> > > > > > +	u8 offset = dw_pcie_find_capability(dw, PCI_CAP_ID_EXP);
> > > > > > +	u32 lnkcap = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCAP);
> > > > > > +	u32 lnkctl = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCTL);
> > > > > > +	u16 lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> > > > > > +	int i;
> > > > > > +
> > > > >
> > > > > > +	if ((lnksta & PCI_EXP_LNKSTA_CLS) == (lnkcap & PCI_EXP_LNKCAP_SLS))
> > > > > > +		return true;
> > > > > > +
> > > > > > +	lnkctl |= PCI_EXP_LNKCTL_RL;
> > > > > > +	dw_pcie_writel_dbi(dw, offset + PCI_EXP_LNKCTL, lnkctl);
> > > > > > +
> > > > > > +	for (i = 0; i < RETRAIN_MAX_CHECK; i++) {
> > > > > > +		lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> > > > > > +		if (lnksta & PCI_EXP_LNKSTA_LT)
> > > > > > +			return true;
> > > > > > +		usleep_range(1000, 1100);
> > > > > > +	}
> > > > >
> > > > > I'll ask one more time because you didn't respond to my previous note
> > > > > about this.
> > > >
> > > > I'm sorry. I completely overlooked the previous note.
> > > >
> > > > > Are you sure that this is needed? Did you try
> > > > > the approach described in "3.13 Gen2/3/4/5 Speed Modes" with
> > > > > de-asserting/asserting the GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag?
> > > >
> > > > I tried this setting, but it doesn't work. I'll investigate this setting more.
> > > >
> > > > > I keep asking because the same problem we used to have on our hardware
> > > > > until we found out that the DIRECT_SPEED_CHANGE flag helped to train
> > > > > the link right to the speed specified in the capabilities.
> > > > >
> > > > > So here is what presumably you'll need to do (based on the
> > > > > "3.1 Initialization" and "3.13 Gen2/3/4/5 Speed Modes" chapters of
> > > > > the DW PCIe DM hw-manual):
> > > > > 1. Make sure the controller is in the power-down/reset state.
> > > > > 2. Select device_type (EP or RP).
> > > > > 3. De-assert the controller reset.
> > > > > 4. Clear PHY-reset flag in the app registers.
> > > > > 5. Perform some controller initializations.
> > > > > 6. Enable LTSSM to start link training.
> > > > > 7. Set GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag one more time.
> > > > >
> > > > > 1-4 are supposed to be done in rcar_gen4_pcie_host_init().
> > > > > 5 is performed in the framework of the DW PCIe core driver.
> > > > > 6-7 should be done in rcar_gen4_pcie_start_link().
> > > > >
> > > > > Note 1. GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag is already set on stage
> > > > > 5 in the framework of the dw_pcie_setup_rc() method. But in our case
> > > > > it only caused having the Gen.2 link speed. Adding stage 7 helped to
> > > > > get stable Gen.3 link. So please try the denoted approach. If it works
> > > > > what about adding stage 7 twice in order to get Gen.4 speed?
> > > > > (waiting for the DIRECT_SPEED_CHANGE flag being auto-cleared and then
> > > > > set it up again?)
> > > > >
> > > > > Note 2. GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag is defined as
> > > > > PCIE_LINK_WIDTH_SPEED_CONTROL.PORT_LOGIC_SPEED_CHANGE macros in the DW
> > > > > PCIe core driver.
> > > > >
> > > > > Note 3. If what is suggested above works well then you won't need to
> > > > > have the heavy rcar_gen4_pcie_check_retrain_link() method in the way
> > > > > you have it defined.
> > > >
> > > > Thank you very much for your comments!
> > >
> > > Please see the GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE description for details
> > > of how the flag is supposed to be de-asserted and asserted in order to
> > > initiate the direct speed change.
> >
> > After I modified the start_link() like below, it also works. Is the code
> > acceptable? (Sorry all tabs are replaced to spaces...)
> 
> Looks good, but still there are some questions.
> 
> > ----------------------------------------------------------------------------
> > static bool rcar_gen4_pcie_check_current_link(struct dw_pcie *dw)
> > {
> >         u8 offset = dw_pcie_find_capability(dw, PCI_CAP_ID_EXP);
> >         u32 lnkcap = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCAP);
> >         u16 lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> >
> 
> >         if ((lnksta & PCI_EXP_LNKSTA_CLS) == (lnkcap & PCI_EXP_LNKCAP_SLS))
> >                 return true;
> 
> AFAICS depending on the link partner speed capabilities this may never
> happen.
> 
> PCI_EXP_LNKCAP_SLS - Max Link Speed. This field indicates the maximum Link
> speed of the associated Port.
> PCI_EXP_LNKSTA_CLS - Current Link Speed. This field indicates the negotiated
> Link speed of the given PCI Express Link
> 
> What if a link partner has the speed capability weaker than the link
> speed of the Root Port? If so then the current link speed will never
> reach the max link speed value.

Thank you very much for your comments! You're correct. This code cannot
work correctly if a link partner has the speed capability weaker than the link speed...

> Of course this can be fixed by specifying a correct "max-link-speed"
> property, but what if a platform has a cold-swappable port connected to
> the root port? Since any device can be attached you'll never be able
> to predict its capabilities beforahead.

You're correct. So, I'll fix the code somehow.

> >
> >         return false;
> > }
> >
> > static void rcar_gen4_pcie_speed_change(struct dw_pcie *dw)
> > {
> >         u32 val;
> >
> >         val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
> >         val &= ~PORT_LOGIC_SPEED_CHANGE;
> >         dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
> >
> >         val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
> >         val |= PORT_LOGIC_SPEED_CHANGE;
> >         dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
> > }
> >
> > static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > {
> >         struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> >         int i;
> >
> >         rcar_gen4_pcie_ltssm_enable(rcar, true);
> >
> >         /*
> >          * Require direct speed change here. Otherwise RDLH_LINK_UP of
> >          * PCIEINTSTS0 which is this controller specific register may not
> >          * be set.
> >          */
> 
> >         if (rcar->needs_speed_change) {
> 
> Seeing this is specified for the root port only what about
> replacing the statement with just test whether the rcar_gen4_pcie.mode ==
> DW_PCIE_RC_TYPE? Thus you'll be ablt to drop the needs_speed_change field.

Thank you for the comment. I'll fix it.

> BTW Just curious. Why is the loop below enabled for the Root Port
> only? What about the end-point controller? It's the same hardware
> after all..

This is reused from v16 and then it used "link retraining" which is only for
the Root Port. As you mentioned, it seems endpoint controller is also needed
if we use direct speed change.

> >                 for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> >                         rcar_gen4_pcie_speed_change(dw);
> >                         msleep(100);
> >                         if (rcar_gen4_pcie_check_current_link(dw))
> >                                 return 0;
> >                 }
> 
> Did you trace how many iterations this loop normally takes?

i = 0 or 1 (if the max-link-speed is suitable for a connected device.)

> Is it
> constant or varies for the same platform setup and a connected link
> partner? Does the number of iterations depend on the target link speed
> specified via the "max-link-speed" property?

This is not related to the "max-link-speed". It seems to related to
a link partner.
		LinkCap	max-link-speed	loop
Device A		4	4		1
Device A		4	3		1
Device B		3	3		0

> I am just trying to understand whether we can completely get rid from
> the rcar_gen4_pcie_check_current_link() method and have it replaced
> with several rcar_gen4_pcie_speed_change() calls. The current link
> state would have been checked in the framework of the
> dw_pcie_wait_for_link() method which calls dw_pcie_link_up() and your
> rcar_gen4_pcie_link_up() in order to make sure the link is actually up.

Thank you for your comment! I'll investigate it.

Best regards,
Yoshihiro Shimoda

> -Serge(y)
> 
> >
> >                 return -ETIMEDOUT;      /* Failed */
> >         }
> > ------------------------------------------------------------------
> >
> > > >
> > > > > > +
> > > > > > +	return false;
> > > > > > +}
> > > > > > +
> > > > > > +static int rcar_gen4_pcie_link_up(struct dw_pcie *dw)
> > > > > > +{
> > > > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > > +	u32 val, mask;
> > > > > > +
> > > > > > +	val = readl(rcar->base + PCIEINTSTS0);
> > > > > > +	mask = RDLH_LINK_UP | SMLH_LINK_UP;
> > > > > > +
> > > > > > +	return (val & mask) == mask;
> > > > > > +}
> > > > > > +
> > > > > > +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > > > > +{
> > > > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > > +	int i;
> > > > > > +
> > > > > > +	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > > > +
> > > > > > +	/*
> > > > > > +	 * Require retraining here. Otherwise RDLH_LINK_UP of PCIEINTSTS0 which
> > > > > > +	 * is this controller specific register may not be set.
> > > > > > +	 */
> > > > > > +	if (rcar->needs_retrain) {
> > > > > > +		for (i = 0; i < RETRAIN_MAX_RETRIES; i++) {
> > > > > > +			if (rcar_gen4_pcie_check_retrain_link(dw))
> > > > > > +				return 0;
> > > > > > +			msleep(100);
> > > > > > +		}
> > > > > > +
> > > > > > +		return -ETIMEDOUT;	/* Failed */
> > > > > > +	}
> > > > > > +
> > > > > > +	return 0;
> > > > > > +}
> > > > > > +
> > > > > > +static void rcar_gen4_pcie_stop_link(struct dw_pcie *dw)
> > > > > > +{
> > > > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > > +
> > > > > > +	rcar_gen4_pcie_ltssm_enable(rcar, false);
> > > > > > +}
> > > > > > +
> > > > >
> > > > > > +int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
> > > > > > +				   int num_lanes)
> > > > >
> > > > > 1. Number of lanes is already defined in the rcar_gen4_pcie.dw.num_lanes field.
> > > > > What about using it from there instead of passing it as an argument?
> > > > > 2. DW PCIe core driver has a very handy enum defined:
> > > > > dw_pcie_device_mode. It describes the controller modes (End-point,
> > > > > Root port, etc). What about adding the mode field right to the
> > > > > rcar_gen4_pcie structure and initializing it in someplace in probe() ?
> > > > > 3. Based on the function semantic it's better to be named as something
> > > > > like rcar_gen4_pcie_init_device() or even rcar_gen4_pcie_basic_init().
> > > >
> > > > Thank you for your comments! I'll modify the function.
> > > >
> > > > >
> > > > > > +{
> > > > > > +	u32 val;
> > > > > > +
> > > > >
> > > > > > +	/* Note: Assume the rcar->rst which is Cold-reset is asserted here */
> > > > >
> > > > > What about directly asserting it here then? In accordance with the DW
> > > > > PCIe DM manual the "device_type" input must be set before the DM
> > > > > controller is powered up (basically un-reset). What if the controller
> > > > > reset is already de-asserted, but you are going to changes its mode?
> > > > > In that case the mode won't be changed and you'll end up with
> > > > > unpredictable results.
> > > >
> > > > Thank you for your comment. We should add asserting it here as you mentioned.
> > > >
> > > > > > +	val = readl(rcar->base + PCIEMSR0);
> > > > > > +	if (rc)
> > > > > > +		val |= DEVICE_TYPE_RC;
> > > > > > +	else
> > > > > > +		val |= DEVICE_TYPE_EP;
> > > > > > +
> > > > > > +	if (num_lanes < 4)
> > > > > > +		val |= BIFUR_MOD_SET_ON;
> > > > > > +
> > > > > > +	writel(val, rcar->base + PCIEMSR0);
> > > > > > +
> > > > > > +	return reset_control_deassert(rcar->rst);
> > > > > > +}
> > > > > > +
> > > > > > +int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *rcar)
> > > > > > +{
> > > > > > +	struct device *dev = rcar->dw.dev;
> > > > > > +	int err;
> > > > > > +
> > > > > > +	pm_runtime_enable(dev);
> > > > > > +	err = pm_runtime_resume_and_get(dev);
> > > > > > +	if (err < 0) {
> > > > > > +		dev_err(dev, "Failed to resume/get Runtime PM\n");
> > > > > > +		pm_runtime_disable(dev);
> > > > > > +	}
> > > > > > +
> > > > > > +	return err;
> > > > > > +}
> > > > > > +
> > > > > > +void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *rcar)
> > > > > > +{
> > > > > > +	struct device *dev = rcar->dw.dev;
> > > > > > +
> > > > > > +	if (!reset_control_status(rcar->rst))
> > > > > > +		reset_control_assert(rcar->rst);
> > > > > > +	pm_runtime_put(dev);
> > > > > > +	pm_runtime_disable(dev);
> > > > > > +}
> > > > > > +
> > > > > > +int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
> > > > > > +				 struct platform_device *pdev)
> > > > > > +{
> > > > > > +	struct device *dev = rcar->dw.dev;
> > > > > > +
> > > > > > +	/* Renesas-specific registers */
> > > > > > +	rcar->base = devm_platform_ioremap_resource_byname(pdev, "app");
> > > > > > +	if (IS_ERR(rcar->base))
> > > > > > +		return PTR_ERR(rcar->base);
> > > > > > +
> > > > >
> > > > > > +	rcar->rst = devm_reset_control_get(dev, NULL);
> > > > > > +	if (IS_ERR(rcar->rst)) {
> > > > > > +		dev_err(dev, "Failed to get Cold-reset\n");
> > > > >
> > > > > So AFAICS your platform is equipped with the DWC_pcie_clkrst.v module.
> > > > > Thus all the resets are appropriately cleared by a single flag:
> > > > > power_up_rst_n. What about using the named reset in this case with the
> > > > > "pwr" name? Thus you'll be able to drop the manual
> > > > > devm_reset_control_get() invocation and instead use the reset-resources
> > > > > requested in the framework of the generic dw_pcie_get_resources()
> > > > > method? Note you'll need to move the dw_pcie_cap_set(dw, REQ_RES);
> > > > > statement to rcar_gen4_pcie_devm_alloc() then and drop the
> > > > > rcar_gen4_pcie.rst field afterwords.
> > > >
> > > > Thank you for your suggestion! Using "pwr" can work on my environment.
> > > >
> > > > > By the way I don't see you requesting and enabling the reference
> > > > > clock in your driver but the bindings imply the clock source. How
> > > > > come?
> > > >
> > >
> > > > For now, I used gpio-hog to enable the reference clock. But, it seem
> > > > I should use "ref" clock for it. So, I'll fix it too.
> > >
> > > Not sure what gpio-hog you are talking about. Do you mean the pe_rst
> > > signal or some another gpio? I failed to see of how pe_rst is
> > > connected to the clock source. In anyway directly handling the clock
> > > source would be more portable choice.
> >
> > Sorry for lacking information. I described a gpio node like below
> > and then the gpio will be high automatically, and the reference clock
> > will be output. But, this is completely independent from pci.
> > ---
> > &gpio2 {
> >         pci-clkreq0-hog {
> >                 gpio-hog;
> >                 gpios = <15 GPIO_ACTIVE_LOW>;
> >                 output-high;
> >         };
> > };
> > ---
> >
> > Now I could implement the clock handling by using "gpio-gate-clock".
> > So, I'll drop the gpio-hog for the reference clock.
> >
> > Best regards,
> > Yoshihiro Shimoda
> >
> > > -Serge(y)
> > >
> > > >
> > > > > > +		return PTR_ERR(rcar->rst);
> > > > > > +	}
> > > > > > +
> > > > > > +	return 0;
> > > > > > +}
> > > > > > +
> > > > > > +static const struct dw_pcie_ops dw_pcie_ops = {
> > > > > > +	.start_link = rcar_gen4_pcie_start_link,
> > > > > > +	.stop_link = rcar_gen4_pcie_stop_link,
> > > > > > +	.link_up = rcar_gen4_pcie_link_up,
> > > > > > +};
> > > > > > +
> > > > > > +struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev)
> > > > > > +{
> > > > > > +	struct rcar_gen4_pcie *rcar;
> > > > > > +
> > > > > > +	rcar = devm_kzalloc(dev, sizeof(*rcar), GFP_KERNEL);
> > > > > > +	if (!rcar)
> > > > > > +		return NULL;
> > > > > > +
> > > > > > +	rcar->dw.dev = dev;
> > > > > > +	rcar->dw.ops = &dw_pcie_ops;
> > > > > > +	dw_pcie_cap_set(&rcar->dw, EDMA_UNROLL);
> > > > > > +
> > > > > > +	return rcar;
> > > > > > +}
> > > > > > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.h b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > > > > > new file mode 100644
> > > > > > index 000000000000..fec3f18609f4
> > > > > > --- /dev/null
> > > > > > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > > > > > @@ -0,0 +1,46 @@
> > > > > > +/* SPDX-License-Identifier: GPL-2.0-only */
> > > > > > +/*
> > > > > > + * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
> > > > > > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > > > > > + */
> > > > > > +
> > > > > > +#ifndef _PCIE_RCAR_GEN4_H_
> > > > > > +#define _PCIE_RCAR_GEN4_H_
> > > > > > +
> > > > > > +#include <linux/io.h>
> > > > > > +#include <linux/pci.h>
> > > > > > +#include <linux/reset.h>
> > > > > > +
> > > > > > +#include "pcie-designware.h"
> > > > > > +
> > > > > > +/* Renesas-specific */
> > > > > > +#define PCIEMSR0		0x0000
> > > > > > +#define  BIFUR_MOD_SET_ON	BIT(0)
> > > > > > +#define  DEVICE_TYPE_EP		0
> > > > > > +#define  DEVICE_TYPE_RC		BIT(4)
> > > > > > +
> > > > > > +#define PCIEINTSTS0		0x0084
> > > > > > +#define PCIEINTSTS0EN		0x0310
> > > > > > +#define  MSI_CTRL_INT		BIT(26)
> > > > > > +#define  SMLH_LINK_UP		BIT(7)
> > > > > > +#define  RDLH_LINK_UP		BIT(6)
> > > > > > +#define PCIEDMAINTSTSEN		0x0314
> > > > > > +#define  PCIEDMAINTSTSEN_INIT	GENMASK(15, 0)
> > > > > > +
> > > > >
> > > > > > +struct rcar_gen4_pcie {
> > > > >
> > > > > As I mentioned above this structure can be extended with the enum
> > > > > dw_pcie_device_mode field thus dropping the boolean argument from the
> > > > > rcar_gen4_pcie_set_device_type() method.
> > > >
> > > > I got it. I'll fix this.
> > > >
> > > > > > +	struct dw_pcie		dw;
> > > > >
> > > > > As I already mentioned above the dw.num_lanes could be used instead of
> > > > > passing it as the rcar_gen4_pcie_set_device_type() argument.
> > > >
> > > > I'll fix this too.
> > > >
> > > > Best regards,
> > > > Yoshihiro Shimoda
> > > >
> > > > > -Serge(y)
> > > > >
> > > > > > +	void __iomem		*base;
> > > > > > +	struct reset_control	*rst;
> > > > > > +	bool			needs_retrain;
> > > > > > +};
> > > > > > +#define to_rcar_gen4_pcie(x)	dev_get_drvdata((x)->dev)
> > > > > > +
> > > > > > +int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
> > > > > > +				   int num_lanes);
> > > > > > +int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *pcie);
> > > > > > +void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *pcie);
> > > > > > +int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
> > > > > > +				 struct platform_device *pdev);
> > > > > > +struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev);
> > > > > > +
> > > > > > +#endif /* _PCIE_RCAR_GEN4_H_ */
> > > > > > --
> > > > > > 2.25.1
> > > > > >

Re: [PATCH v16 19/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support

[Serge Semin]

On Fri, Jun 09, 2023 at 06:29:39AM +0000, Yoshihiro Shimoda wrote:
> Hello Serge,
> 
> > From: Serge Semin, Sent: Thursday, June 8, 2023 9:11 PM
> > 
> > On Thu, Jun 08, 2023 at 08:47:16AM +0000, Yoshihiro Shimoda wrote:
> > > Hello Serge,
> > >
> > > > From: Serge Semin, Sent: Wednesday, June 7, 2023 9:16 PM
> > > >
> > > > On Wed, Jun 07, 2023 at 02:59:20AM +0000, Yoshihiro Shimoda wrote:
> > > > > Hello Serge,
> > > > >
> > > > > > From: Serge Semin, Sent: Monday, June 5, 2023 11:39 PM
> > > > > >
> > > > > > On Wed, May 10, 2023 at 03:22:31PM +0900, Yoshihiro Shimoda wrote:
> > > > > > > Add R-Car Gen4 PCIe Host support. This controller is based on
> > > > > > > Synopsys DesignWare PCIe, but this controller has vendor-specific
> > > > > > > registers so that requires initialization code like mode setting
> > > > > > > and retraining and so on.
> > > > > > >
> > > > > > > To reduce code delta, adds some helper functions which are used by
> > > > > > > both the host driver and the endpoint driver (which is added
> > > > > > > immediately afterwards) into a separate file.
> > > > > > >
> > > > > > > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > > > > > > ---
> > > > > > >  drivers/pci/controller/dwc/Kconfig            |   9 +
> > > > > > >  drivers/pci/controller/dwc/Makefile           |   2 +
> > > > > > >  .../pci/controller/dwc/pcie-rcar-gen4-host.c  | 141 +++++++++++++
> > > > > > >  drivers/pci/controller/dwc/pcie-rcar-gen4.c   | 190 ++++++++++++++++++
> > > > > > >  drivers/pci/controller/dwc/pcie-rcar-gen4.h   |  46 +++++
> > > > > > >  5 files changed, 388 insertions(+)
> > > > > > >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > > > > > >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > > > > > >  create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > > > > > >
> > > > > > > diff --git a/drivers/pci/controller/dwc/Kconfig b/drivers/pci/controller/dwc/Kconfig
> > > > > > > index ab96da43e0c2..64d4d37bc891 100644
> > > > > > > --- a/drivers/pci/controller/dwc/Kconfig
> > > > > > > +++ b/drivers/pci/controller/dwc/Kconfig
> > > > > > > @@ -415,4 +415,13 @@ config PCIE_VISCONTI_HOST
> > > > > > >  	  Say Y here if you want PCIe controller support on Toshiba Visconti SoC.
> > > > > > >  	  This driver supports TMPV7708 SoC.
> > > > > > >
> > > > > > > +config PCIE_RCAR_GEN4
> > > > > > > +	tristate "Renesas R-Car Gen4 PCIe Host controller"
> > > > > > > +	depends on ARCH_RENESAS || COMPILE_TEST
> > > > > > > +	depends on PCI_MSI
> > > > > > > +	select PCIE_DW_HOST
> > > > > > > +	help
> > > > > > > +	  Say Y here if you want PCIe host controller support on R-Car Gen4 SoCs.
> > > > > > > +	  This uses the DesignWare core.
> > > > > > > +
> > > > > > >  endmenu
> > > > > > > diff --git a/drivers/pci/controller/dwc/Makefile b/drivers/pci/controller/dwc/Makefile
> > > > > > > index bf5c311875a1..486cf706b53d 100644
> > > > > > > --- a/drivers/pci/controller/dwc/Makefile
> > > > > > > +++ b/drivers/pci/controller/dwc/Makefile
> > > > > > > @@ -26,6 +26,8 @@ obj-$(CONFIG_PCIE_TEGRA194) += pcie-tegra194.o
> > > > > > >  obj-$(CONFIG_PCIE_UNIPHIER) += pcie-uniphier.o
> > > > > > >  obj-$(CONFIG_PCIE_UNIPHIER_EP) += pcie-uniphier-ep.o
> > > > > > >  obj-$(CONFIG_PCIE_VISCONTI_HOST) += pcie-visconti.o
> > > > > > > +pcie-rcar-gen4-host-drv-objs := pcie-rcar-gen4.o pcie-rcar-gen4-host.o
> > > > > > > +obj-$(CONFIG_PCIE_RCAR_GEN4) += pcie-rcar-gen4-host-drv.o
> > > > > > >
> > > > > > >  # The following drivers are for devices that use the generic ACPI
> > > > > > >  # pci_root.c driver but don't support standard ECAM config access.
> > > > > > > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > > > > > > new file mode 100644
> > > > > > > index 000000000000..df7d80f1874f
> > > > > > > --- /dev/null
> > > > > > > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > > > > > > @@ -0,0 +1,141 @@
> > > > > > > +// SPDX-License-Identifier: GPL-2.0-only
> > > > > > > +/*
> > > > > > > + * PCIe host controller driver for Renesas R-Car Gen4 Series SoCs
> > > > > > > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > > > > > > + */
> > > > > > > +
> > > > > > > +#include <linux/delay.h>
> > > > > > > +#include <linux/interrupt.h>
> > > > > > > +#include <linux/module.h>
> > > > > > > +#include <linux/of_device.h>
> > > > > > > +#include <linux/pci.h>
> > > > > > > +#include <linux/platform_device.h>
> > > > > > > +
> > > > > > > +#include "pcie-rcar-gen4.h"
> > > > > > > +#include "pcie-designware.h"
> > > > > > > +
> > > > > > > +static int rcar_gen4_pcie_host_init(struct dw_pcie_rp *pp)
> > > > > > > +{
> > > > > > > +	struct dw_pcie *dw = to_dw_pcie_from_pp(pp);
> > > > > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > > > +	int ret;
> > > > > > > +	u32 val;
> > > > > > > +
> > > > > > > +	gpiod_set_value_cansleep(dw->pe_rst, 1);
> > > > > > > +
> > > > > > > +	ret = rcar_gen4_pcie_set_device_type(rcar, true, dw->num_lanes);
> > > > > > > +	if (ret < 0)
> > > > > > > +		return ret;
> > > > > > > +
> > > > > >
> > > > > > > +	dw_pcie_dbi_ro_wr_en(dw);
> > > > > >
> > > > > > Are you sure dw_pcie_dbi_ro_wr_en() and dw_pcie_dbi_ro_wr_dis() are
> > > > > > needed? In accordance with the DW PCIe Dual-mode HW manual the BARx
> > > > > > registers are W-only over the DBI2 map with no need in setting the
> > > > > > DBI_RO_WR_EN flag.
> > > > > >
> > > > > > Please check that on your hardware.
> > > > >
> > > > > You're correct. They are not needed. So, I'll drop this on v17.
> > > > >
> > > > > > > +
> > > > > > > +	/*
> > > > > > > +	 * According to the section 3.5.7.2 "RC Mode" in DWC PCIe Dual Mode
> > > > > > > +	 * Rev.5.20a, we should disable two BARs to avoid unnecessary memory
> > > > > > > +	 * assignment during device enumeration.
> > > > > > > +	 */
> > > > > > > +	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_0, 0x0);
> > > > > > > +	dw_pcie_writel_dbi2(dw, PCI_BASE_ADDRESS_1, 0x0);
> > > > > > > +
> > > > > >
> > > > > > > +	dw_pcie_dbi_ro_wr_dis(dw);
> > > > > >
> > > > > > ditto
> > > > >
> > > > > I'll drop this too.
> > > > >
> > > > > > > +
> > > > > > > +	if (IS_ENABLED(CONFIG_PCI_MSI)) {
> > > > > > > +		/* Enable MSI interrupt signal */
> > > > > > > +		val = readl(rcar->base + PCIEINTSTS0EN);
> > > > > > > +		val |= MSI_CTRL_INT;
> > > > > > > +		writel(val, rcar->base + PCIEINTSTS0EN);
> > > > > > > +	}
> > > > > > > +
> > > > > > > +	msleep(100);	/* pe_rst requires 100msec delay */
> > > > > > > +
> > > > > > > +	gpiod_set_value_cansleep(dw->pe_rst, 0);
> > > > > > > +
> > > > > > > +	return 0;
> > > > > > > +}
> > > > > > > +
> > > > > > > +static const struct dw_pcie_host_ops rcar_gen4_pcie_host_ops = {
> > > > > > > +	.host_init = rcar_gen4_pcie_host_init,
> > > > > > > +};
> > > > > > > +
> > > > > > > +static int rcar_gen4_add_dw_pcie_rp(struct rcar_gen4_pcie *rcar,
> > > > > > > +				   struct platform_device *pdev)
> > > > > > > +{
> > > > > > > +	struct dw_pcie *dw = &rcar->dw;
> > > > > > > +	struct dw_pcie_rp *pp = &dw->pp;
> > > > > > > +
> > > > > > > +	pp->num_vectors = MAX_MSI_IRQS;
> > > > > > > +	pp->ops = &rcar_gen4_pcie_host_ops;
> > > > > > > +	dw_pcie_cap_set(dw, REQ_RES);
> > > > > > > +
> > > > > > > +	return dw_pcie_host_init(pp);
> > > > > > > +}
> > > > > > > +
> > > > > > > +static void rcar_gen4_remove_dw_pcie_rp(struct rcar_gen4_pcie *rcar)
> > > > > > > +{
> > > > > > > +	dw_pcie_host_deinit(&rcar->dw.pp);
> > > > > > > +	gpiod_set_value_cansleep(rcar->dw.pe_rst, 1);
> > > > > > > +}
> > > > > > > +
> > > > > > > +static int rcar_gen4_pcie_probe(struct platform_device *pdev)
> > > > > > > +{
> > > > > > > +	struct device *dev = &pdev->dev;
> > > > > > > +	struct rcar_gen4_pcie *rcar;
> > > > > > > +	int err;
> > > > > > > +
> > > > > > > +	rcar = rcar_gen4_pcie_devm_alloc(dev);
> > > > > > > +	if (!rcar)
> > > > > > > +		return -ENOMEM;
> > > > > > > +
> > > > > > > +	err = rcar_gen4_pcie_get_resources(rcar, pdev);
> > > > > > > +	if (err < 0) {
> > > > > > > +		dev_err(dev, "Failed to request resource: %d\n", err);
> > > > > > > +		return err;
> > > > > > > +	}
> > > > > > > +
> > > > > > > +	platform_set_drvdata(pdev, rcar);
> > > > > > > +
> > > > > > > +	err = rcar_gen4_pcie_prepare(rcar);
> > > > > > > +	if (err < 0)
> > > > > > > +		return err;
> > > > > > > +
> > > > > > > +	rcar->needs_retrain = true;
> > > > > > > +	err = rcar_gen4_add_dw_pcie_rp(rcar, pdev);
> > > > > > > +	if (err < 0)
> > > > > > > +		goto err_add;
> > > > > > > +
> > > > > > > +	return 0;
> > > > > > > +
> > > > > > > +err_add:
> > > > > > > +	rcar_gen4_pcie_unprepare(rcar);
> > > > > > > +
> > > > > > > +	return err;
> > > > > > > +}
> > > > > > > +
> > > > > > > +static int rcar_gen4_pcie_remove(struct platform_device *pdev)
> > > > > > > +{
> > > > > > > +	struct rcar_gen4_pcie *rcar = platform_get_drvdata(pdev);
> > > > > > > +
> > > > > > > +	rcar_gen4_remove_dw_pcie_rp(rcar);
> > > > > > > +	rcar_gen4_pcie_unprepare(rcar);
> > > > > > > +
> > > > > > > +	return 0;
> > > > > > > +}
> > > > > > > +
> > > > > > > +static const struct of_device_id rcar_gen4_pcie_of_match[] = {
> > > > > > > +	{ .compatible = "renesas,rcar-gen4-pcie", },
> > > > > > > +	{},
> > > > > > > +};
> > > > > > > +
> > > > > > > +static struct platform_driver rcar_gen4_pcie_driver = {
> > > > > > > +	.driver = {
> > > > > > > +		.name = "pcie-rcar-gen4",
> > > > > > > +		.of_match_table = rcar_gen4_pcie_of_match,
> > > > > > > +		.probe_type = PROBE_PREFER_ASYNCHRONOUS,
> > > > > > > +	},
> > > > > > > +	.probe = rcar_gen4_pcie_probe,
> > > > > > > +	.remove = rcar_gen4_pcie_remove,
> > > > > > > +};
> > > > > > > +module_platform_driver(rcar_gen4_pcie_driver);
> > > > > > > +
> > > > > > > +MODULE_DESCRIPTION("Renesas R-Car Gen4 PCIe host controller driver");
> > > > > > > +MODULE_LICENSE("GPL");
> > > > > > > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.c b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > > > > > > new file mode 100644
> > > > > > > index 000000000000..35923fda8ed5
> > > > > > > --- /dev/null
> > > > > > > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > > > > > > @@ -0,0 +1,190 @@
> > > > > > > +// SPDX-License-Identifier: GPL-2.0-only
> > > > > > > +/*
> > > > > > > + * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
> > > > > > > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > > > > > > + */
> > > > > > > +
> > > > > > > +#include <linux/delay.h>
> > > > > > > +#include <linux/io.h>
> > > > > > > +#include <linux/of_device.h>
> > > > > > > +#include <linux/pci.h>
> > > > > > > +#include <linux/pm_runtime.h>
> > > > > > > +#include <linux/reset.h>
> > > > > > > +
> > > > > > > +#include "pcie-rcar-gen4.h"
> > > > > > > +#include "pcie-designware.h"
> > > > > > > +
> > > > > > > +/* Renesas-specific */
> > > > > > > +#define PCIERSTCTRL1		0x0014
> > > > > > > +#define  APP_HOLD_PHY_RST	BIT(16)
> > > > > > > +#define  APP_LTSSM_ENABLE	BIT(0)
> > > > > > > +
> > > > > > > +#define RETRAIN_MAX_CHECK	10
> > > > > > > +#define RETRAIN_MAX_RETRIES	10
> > > > > > > +
> > > > > > > +static void rcar_gen4_pcie_ltssm_enable(struct rcar_gen4_pcie *rcar,
> > > > > > > +					bool enable)
> > > > > > > +{
> > > > > > > +	u32 val;
> > > > > > > +
> > > > > > > +	val = readl(rcar->base + PCIERSTCTRL1);
> > > > > > > +	if (enable) {
> > > > > > > +		val |= APP_LTSSM_ENABLE;
> > > > > >
> > > > > > > +		val &= ~APP_HOLD_PHY_RST;
> > > > > >
> > > > > > What about moving the APP_HOLD_PHY_RST de-assertion to the
> > > > > > rcar_gen4_pcie_set_device_type() method? In accordance with the
> > > > > > "3.1 Initialization" chapter it's supposed to be done before
> > > > > > performing the DBI programming and activating the link training.
> > > > >
> > > >
> > > > > IIUC, the "3.1 Initialization" said app_hold_phy_rst = 1 before
> > > > > performing the DBI programming. So, it is assertion. Also, the SoC
> > > > > documentation described the initializing procedure as the follows:
> > > > >  app_ltssm_enable = 1
> > > > >  app_hold_phy_rst = 0
> > > > > So, I would like to keep them in the function.
> > > >
> > > > Indeed. I was wrong. Sorry for the misleading comment.
> > >
> > > No problem. Thank you for the confirmation!
> > >
> > > > >
> > > > > > > +	} else {
> > > > > > > +		val &= ~APP_LTSSM_ENABLE;
> > > > > > > +		val |= APP_HOLD_PHY_RST;
> > > > > > > +	}
> > > > > > > +	writel(val, rcar->base + PCIERSTCTRL1);
> > > > > > > +}
> > > > > > > +
> > > > > > > +static bool rcar_gen4_pcie_check_retrain_link(struct dw_pcie *dw)
> > > > > > > +{
> > > > > > > +	u8 offset = dw_pcie_find_capability(dw, PCI_CAP_ID_EXP);
> > > > > > > +	u32 lnkcap = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCAP);
> > > > > > > +	u32 lnkctl = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCTL);
> > > > > > > +	u16 lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> > > > > > > +	int i;
> > > > > > > +
> > > > > >
> > > > > > > +	if ((lnksta & PCI_EXP_LNKSTA_CLS) == (lnkcap & PCI_EXP_LNKCAP_SLS))
> > > > > > > +		return true;
> > > > > > > +
> > > > > > > +	lnkctl |= PCI_EXP_LNKCTL_RL;
> > > > > > > +	dw_pcie_writel_dbi(dw, offset + PCI_EXP_LNKCTL, lnkctl);
> > > > > > > +
> > > > > > > +	for (i = 0; i < RETRAIN_MAX_CHECK; i++) {
> > > > > > > +		lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> > > > > > > +		if (lnksta & PCI_EXP_LNKSTA_LT)
> > > > > > > +			return true;
> > > > > > > +		usleep_range(1000, 1100);
> > > > > > > +	}
> > > > > >
> > > > > > I'll ask one more time because you didn't respond to my previous note
> > > > > > about this.
> > > > >
> > > > > I'm sorry. I completely overlooked the previous note.
> > > > >
> > > > > > Are you sure that this is needed? Did you try
> > > > > > the approach described in "3.13 Gen2/3/4/5 Speed Modes" with
> > > > > > de-asserting/asserting the GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag?
> > > > >
> > > > > I tried this setting, but it doesn't work. I'll investigate this setting more.
> > > > >
> > > > > > I keep asking because the same problem we used to have on our hardware
> > > > > > until we found out that the DIRECT_SPEED_CHANGE flag helped to train
> > > > > > the link right to the speed specified in the capabilities.
> > > > > >
> > > > > > So here is what presumably you'll need to do (based on the
> > > > > > "3.1 Initialization" and "3.13 Gen2/3/4/5 Speed Modes" chapters of
> > > > > > the DW PCIe DM hw-manual):
> > > > > > 1. Make sure the controller is in the power-down/reset state.
> > > > > > 2. Select device_type (EP or RP).
> > > > > > 3. De-assert the controller reset.
> > > > > > 4. Clear PHY-reset flag in the app registers.
> > > > > > 5. Perform some controller initializations.
> > > > > > 6. Enable LTSSM to start link training.
> > > > > > 7. Set GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag one more time.
> > > > > >
> > > > > > 1-4 are supposed to be done in rcar_gen4_pcie_host_init().
> > > > > > 5 is performed in the framework of the DW PCIe core driver.
> > > > > > 6-7 should be done in rcar_gen4_pcie_start_link().
> > > > > >
> > > > > > Note 1. GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag is already set on stage
> > > > > > 5 in the framework of the dw_pcie_setup_rc() method. But in our case
> > > > > > it only caused having the Gen.2 link speed. Adding stage 7 helped to
> > > > > > get stable Gen.3 link. So please try the denoted approach. If it works
> > > > > > what about adding stage 7 twice in order to get Gen.4 speed?
> > > > > > (waiting for the DIRECT_SPEED_CHANGE flag being auto-cleared and then
> > > > > > set it up again?)
> > > > > >
> > > > > > Note 2. GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag is defined as
> > > > > > PCIE_LINK_WIDTH_SPEED_CONTROL.PORT_LOGIC_SPEED_CHANGE macros in the DW
> > > > > > PCIe core driver.
> > > > > >
> > > > > > Note 3. If what is suggested above works well then you won't need to
> > > > > > have the heavy rcar_gen4_pcie_check_retrain_link() method in the way
> > > > > > you have it defined.
> > > > >
> > > > > Thank you very much for your comments!
> > > >
> > > > Please see the GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE description for details
> > > > of how the flag is supposed to be de-asserted and asserted in order to
> > > > initiate the direct speed change.
> > >
> > > After I modified the start_link() like below, it also works. Is the code
> > > acceptable? (Sorry all tabs are replaced to spaces...)
> > 
> > Looks good, but still there are some questions.
> > 
> > > ----------------------------------------------------------------------------
> > > static bool rcar_gen4_pcie_check_current_link(struct dw_pcie *dw)
> > > {
> > >         u8 offset = dw_pcie_find_capability(dw, PCI_CAP_ID_EXP);
> > >         u32 lnkcap = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCAP);
> > >         u16 lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> > >
> > 
> > >         if ((lnksta & PCI_EXP_LNKSTA_CLS) == (lnkcap & PCI_EXP_LNKCAP_SLS))
> > >                 return true;
> > 
> > AFAICS depending on the link partner speed capabilities this may never
> > happen.
> > 
> > PCI_EXP_LNKCAP_SLS - Max Link Speed. This field indicates the maximum Link
> > speed of the associated Port.
> > PCI_EXP_LNKSTA_CLS - Current Link Speed. This field indicates the negotiated
> > Link speed of the given PCI Express Link
> > 
> > What if a link partner has the speed capability weaker than the link
> > speed of the Root Port? If so then the current link speed will never
> > reach the max link speed value.
> 
> Thank you very much for your comments! You're correct. This code cannot
> work correctly if a link partner has the speed capability weaker than the link speed...
> 
> > Of course this can be fixed by specifying a correct "max-link-speed"
> > property, but what if a platform has a cold-swappable port connected to
> > the root port? Since any device can be attached you'll never be able
> > to predict its capabilities beforahead.
> 
> You're correct. So, I'll fix the code somehow.
> 
> > >
> > >         return false;
> > > }
> > >
> > > static void rcar_gen4_pcie_speed_change(struct dw_pcie *dw)
> > > {
> > >         u32 val;
> > >
> > >         val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
> > >         val &= ~PORT_LOGIC_SPEED_CHANGE;
> > >         dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
> > >
> > >         val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
> > >         val |= PORT_LOGIC_SPEED_CHANGE;
> > >         dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
> > > }
> > >
> > > static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > {
> > >         struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > >         int i;
> > >
> > >         rcar_gen4_pcie_ltssm_enable(rcar, true);
> > >
> > >         /*
> > >          * Require direct speed change here. Otherwise RDLH_LINK_UP of
> > >          * PCIEINTSTS0 which is this controller specific register may not
> > >          * be set.
> > >          */
> > 
> > >         if (rcar->needs_speed_change) {
> > 
> > Seeing this is specified for the root port only what about
> > replacing the statement with just test whether the rcar_gen4_pcie.mode ==
> > DW_PCIE_RC_TYPE? Thus you'll be ablt to drop the needs_speed_change field.
> 
> Thank you for the comment. I'll fix it.
> 
> > BTW Just curious. Why is the loop below enabled for the Root Port
> > only? What about the end-point controller? It's the same hardware
> > after all..
> 
> This is reused from v16 and then it used "link retraining" which is only for
> the Root Port. As you mentioned, it seems endpoint controller is also needed
> if we use direct speed change.
> 
> > >                 for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > >                         rcar_gen4_pcie_speed_change(dw);
> > >                         msleep(100);
> > >                         if (rcar_gen4_pcie_check_current_link(dw))
> > >                                 return 0;
> > >                 }
> > 
> > Did you trace how many iterations this loop normally takes?
> 
> i = 0 or 1 (if the max-link-speed is suitable for a connected device.)
> 
> > Is it
> > constant or varies for the same platform setup and a connected link
> > partner? Does the number of iterations depend on the target link speed
> > specified via the "max-link-speed" property?
> 

> This is not related to the "max-link-speed". It seems to related to
> a link partner.
> 		LinkCap	max-link-speed	loop
> Device A		4	4		1
> Device A		4	3		1
> Device B		3	3		0

Great! If so I would have just left a single unconditional
rcar_gen4_pcie_speed_change() call placed right after the
rcar_gen4_pcie_ltssm_enable() method with no delays afterwards. These
methods would have been invoked in the framework of
dw_pcie_start_link() after which the dw_pcie_wait_for_link() method is
called with several checks parted with the ~100ms delay. It will make
sure that at least some link is up with the link state printed to the
system log. If for some reason the performance degradation happens
then it will be up to the system administrator to investigate what was
wrong. Your driver did as much is it could to reach the best link gen.

-Serge(y)

> 
> > I am just trying to understand whether we can completely get rid from
> > the rcar_gen4_pcie_check_current_link() method and have it replaced
> > with several rcar_gen4_pcie_speed_change() calls. The current link
> > state would have been checked in the framework of the
> > dw_pcie_wait_for_link() method which calls dw_pcie_link_up() and your
> > rcar_gen4_pcie_link_up() in order to make sure the link is actually up.
> 
> Thank you for your comment! I'll investigate it.
> 
> Best regards,
> Yoshihiro Shimoda
> 
> > -Serge(y)
> > 
> > >
> > >                 return -ETIMEDOUT;      /* Failed */
> > >         }
> > > ------------------------------------------------------------------
> > >
> > > > >
> > > > > > > +
> > > > > > > +	return false;
> > > > > > > +}
> > > > > > > +
> > > > > > > +static int rcar_gen4_pcie_link_up(struct dw_pcie *dw)
> > > > > > > +{
> > > > > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > > > +	u32 val, mask;
> > > > > > > +
> > > > > > > +	val = readl(rcar->base + PCIEINTSTS0);
> > > > > > > +	mask = RDLH_LINK_UP | SMLH_LINK_UP;
> > > > > > > +
> > > > > > > +	return (val & mask) == mask;
> > > > > > > +}
> > > > > > > +
> > > > > > > +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > > > > > +{
> > > > > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > > > +	int i;
> > > > > > > +
> > > > > > > +	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > > > > +
> > > > > > > +	/*
> > > > > > > +	 * Require retraining here. Otherwise RDLH_LINK_UP of PCIEINTSTS0 which
> > > > > > > +	 * is this controller specific register may not be set.
> > > > > > > +	 */
> > > > > > > +	if (rcar->needs_retrain) {
> > > > > > > +		for (i = 0; i < RETRAIN_MAX_RETRIES; i++) {
> > > > > > > +			if (rcar_gen4_pcie_check_retrain_link(dw))
> > > > > > > +				return 0;
> > > > > > > +			msleep(100);
> > > > > > > +		}
> > > > > > > +
> > > > > > > +		return -ETIMEDOUT;	/* Failed */
> > > > > > > +	}
> > > > > > > +
> > > > > > > +	return 0;
> > > > > > > +}
> > > > > > > +
> > > > > > > +static void rcar_gen4_pcie_stop_link(struct dw_pcie *dw)
> > > > > > > +{
> > > > > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > > > +
> > > > > > > +	rcar_gen4_pcie_ltssm_enable(rcar, false);
> > > > > > > +}
> > > > > > > +
> > > > > >
> > > > > > > +int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
> > > > > > > +				   int num_lanes)
> > > > > >
> > > > > > 1. Number of lanes is already defined in the rcar_gen4_pcie.dw.num_lanes field.
> > > > > > What about using it from there instead of passing it as an argument?
> > > > > > 2. DW PCIe core driver has a very handy enum defined:
> > > > > > dw_pcie_device_mode. It describes the controller modes (End-point,
> > > > > > Root port, etc). What about adding the mode field right to the
> > > > > > rcar_gen4_pcie structure and initializing it in someplace in probe() ?
> > > > > > 3. Based on the function semantic it's better to be named as something
> > > > > > like rcar_gen4_pcie_init_device() or even rcar_gen4_pcie_basic_init().
> > > > >
> > > > > Thank you for your comments! I'll modify the function.
> > > > >
> > > > > >
> > > > > > > +{
> > > > > > > +	u32 val;
> > > > > > > +
> > > > > >
> > > > > > > +	/* Note: Assume the rcar->rst which is Cold-reset is asserted here */
> > > > > >
> > > > > > What about directly asserting it here then? In accordance with the DW
> > > > > > PCIe DM manual the "device_type" input must be set before the DM
> > > > > > controller is powered up (basically un-reset). What if the controller
> > > > > > reset is already de-asserted, but you are going to changes its mode?
> > > > > > In that case the mode won't be changed and you'll end up with
> > > > > > unpredictable results.
> > > > >
> > > > > Thank you for your comment. We should add asserting it here as you mentioned.
> > > > >
> > > > > > > +	val = readl(rcar->base + PCIEMSR0);
> > > > > > > +	if (rc)
> > > > > > > +		val |= DEVICE_TYPE_RC;
> > > > > > > +	else
> > > > > > > +		val |= DEVICE_TYPE_EP;
> > > > > > > +
> > > > > > > +	if (num_lanes < 4)
> > > > > > > +		val |= BIFUR_MOD_SET_ON;
> > > > > > > +
> > > > > > > +	writel(val, rcar->base + PCIEMSR0);
> > > > > > > +
> > > > > > > +	return reset_control_deassert(rcar->rst);
> > > > > > > +}
> > > > > > > +
> > > > > > > +int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *rcar)
> > > > > > > +{
> > > > > > > +	struct device *dev = rcar->dw.dev;
> > > > > > > +	int err;
> > > > > > > +
> > > > > > > +	pm_runtime_enable(dev);
> > > > > > > +	err = pm_runtime_resume_and_get(dev);
> > > > > > > +	if (err < 0) {
> > > > > > > +		dev_err(dev, "Failed to resume/get Runtime PM\n");
> > > > > > > +		pm_runtime_disable(dev);
> > > > > > > +	}
> > > > > > > +
> > > > > > > +	return err;
> > > > > > > +}
> > > > > > > +
> > > > > > > +void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *rcar)
> > > > > > > +{
> > > > > > > +	struct device *dev = rcar->dw.dev;
> > > > > > > +
> > > > > > > +	if (!reset_control_status(rcar->rst))
> > > > > > > +		reset_control_assert(rcar->rst);
> > > > > > > +	pm_runtime_put(dev);
> > > > > > > +	pm_runtime_disable(dev);
> > > > > > > +}
> > > > > > > +
> > > > > > > +int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
> > > > > > > +				 struct platform_device *pdev)
> > > > > > > +{
> > > > > > > +	struct device *dev = rcar->dw.dev;
> > > > > > > +
> > > > > > > +	/* Renesas-specific registers */
> > > > > > > +	rcar->base = devm_platform_ioremap_resource_byname(pdev, "app");
> > > > > > > +	if (IS_ERR(rcar->base))
> > > > > > > +		return PTR_ERR(rcar->base);
> > > > > > > +
> > > > > >
> > > > > > > +	rcar->rst = devm_reset_control_get(dev, NULL);
> > > > > > > +	if (IS_ERR(rcar->rst)) {
> > > > > > > +		dev_err(dev, "Failed to get Cold-reset\n");
> > > > > >
> > > > > > So AFAICS your platform is equipped with the DWC_pcie_clkrst.v module.
> > > > > > Thus all the resets are appropriately cleared by a single flag:
> > > > > > power_up_rst_n. What about using the named reset in this case with the
> > > > > > "pwr" name? Thus you'll be able to drop the manual
> > > > > > devm_reset_control_get() invocation and instead use the reset-resources
> > > > > > requested in the framework of the generic dw_pcie_get_resources()
> > > > > > method? Note you'll need to move the dw_pcie_cap_set(dw, REQ_RES);
> > > > > > statement to rcar_gen4_pcie_devm_alloc() then and drop the
> > > > > > rcar_gen4_pcie.rst field afterwords.
> > > > >
> > > > > Thank you for your suggestion! Using "pwr" can work on my environment.
> > > > >
> > > > > > By the way I don't see you requesting and enabling the reference
> > > > > > clock in your driver but the bindings imply the clock source. How
> > > > > > come?
> > > > >
> > > >
> > > > > For now, I used gpio-hog to enable the reference clock. But, it seem
> > > > > I should use "ref" clock for it. So, I'll fix it too.
> > > >
> > > > Not sure what gpio-hog you are talking about. Do you mean the pe_rst
> > > > signal or some another gpio? I failed to see of how pe_rst is
> > > > connected to the clock source. In anyway directly handling the clock
> > > > source would be more portable choice.
> > >
> > > Sorry for lacking information. I described a gpio node like below
> > > and then the gpio will be high automatically, and the reference clock
> > > will be output. But, this is completely independent from pci.
> > > ---
> > > &gpio2 {
> > >         pci-clkreq0-hog {
> > >                 gpio-hog;
> > >                 gpios = <15 GPIO_ACTIVE_LOW>;
> > >                 output-high;
> > >         };
> > > };
> > > ---
> > >
> > > Now I could implement the clock handling by using "gpio-gate-clock".
> > > So, I'll drop the gpio-hog for the reference clock.
> > >
> > > Best regards,
> > > Yoshihiro Shimoda
> > >
> > > > -Serge(y)
> > > >
> > > > >
> > > > > > > +		return PTR_ERR(rcar->rst);
> > > > > > > +	}
> > > > > > > +
> > > > > > > +	return 0;
> > > > > > > +}
> > > > > > > +
> > > > > > > +static const struct dw_pcie_ops dw_pcie_ops = {
> > > > > > > +	.start_link = rcar_gen4_pcie_start_link,
> > > > > > > +	.stop_link = rcar_gen4_pcie_stop_link,
> > > > > > > +	.link_up = rcar_gen4_pcie_link_up,
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev)
> > > > > > > +{
> > > > > > > +	struct rcar_gen4_pcie *rcar;
> > > > > > > +
> > > > > > > +	rcar = devm_kzalloc(dev, sizeof(*rcar), GFP_KERNEL);
> > > > > > > +	if (!rcar)
> > > > > > > +		return NULL;
> > > > > > > +
> > > > > > > +	rcar->dw.dev = dev;
> > > > > > > +	rcar->dw.ops = &dw_pcie_ops;
> > > > > > > +	dw_pcie_cap_set(&rcar->dw, EDMA_UNROLL);
> > > > > > > +
> > > > > > > +	return rcar;
> > > > > > > +}
> > > > > > > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.h b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > > > > > > new file mode 100644
> > > > > > > index 000000000000..fec3f18609f4
> > > > > > > --- /dev/null
> > > > > > > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.h
> > > > > > > @@ -0,0 +1,46 @@
> > > > > > > +/* SPDX-License-Identifier: GPL-2.0-only */
> > > > > > > +/*
> > > > > > > + * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
> > > > > > > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > > > > > > + */
> > > > > > > +
> > > > > > > +#ifndef _PCIE_RCAR_GEN4_H_
> > > > > > > +#define _PCIE_RCAR_GEN4_H_
> > > > > > > +
> > > > > > > +#include <linux/io.h>
> > > > > > > +#include <linux/pci.h>
> > > > > > > +#include <linux/reset.h>
> > > > > > > +
> > > > > > > +#include "pcie-designware.h"
> > > > > > > +
> > > > > > > +/* Renesas-specific */
> > > > > > > +#define PCIEMSR0		0x0000
> > > > > > > +#define  BIFUR_MOD_SET_ON	BIT(0)
> > > > > > > +#define  DEVICE_TYPE_EP		0
> > > > > > > +#define  DEVICE_TYPE_RC		BIT(4)
> > > > > > > +
> > > > > > > +#define PCIEINTSTS0		0x0084
> > > > > > > +#define PCIEINTSTS0EN		0x0310
> > > > > > > +#define  MSI_CTRL_INT		BIT(26)
> > > > > > > +#define  SMLH_LINK_UP		BIT(7)
> > > > > > > +#define  RDLH_LINK_UP		BIT(6)
> > > > > > > +#define PCIEDMAINTSTSEN		0x0314
> > > > > > > +#define  PCIEDMAINTSTSEN_INIT	GENMASK(15, 0)
> > > > > > > +
> > > > > >
> > > > > > > +struct rcar_gen4_pcie {
> > > > > >
> > > > > > As I mentioned above this structure can be extended with the enum
> > > > > > dw_pcie_device_mode field thus dropping the boolean argument from the
> > > > > > rcar_gen4_pcie_set_device_type() method.
> > > > >
> > > > > I got it. I'll fix this.
> > > > >
> > > > > > > +	struct dw_pcie		dw;
> > > > > >
> > > > > > As I already mentioned above the dw.num_lanes could be used instead of
> > > > > > passing it as the rcar_gen4_pcie_set_device_type() argument.
> > > > >
> > > > > I'll fix this too.
> > > > >
> > > > > Best regards,
> > > > > Yoshihiro Shimoda
> > > > >
> > > > > > -Serge(y)
> > > > > >
> > > > > > > +	void __iomem		*base;
> > > > > > > +	struct reset_control	*rst;
> > > > > > > +	bool			needs_retrain;
> > > > > > > +};
> > > > > > > +#define to_rcar_gen4_pcie(x)	dev_get_drvdata((x)->dev)
> > > > > > > +
> > > > > > > +int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
> > > > > > > +				   int num_lanes);
> > > > > > > +int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *pcie);
> > > > > > > +void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *pcie);
> > > > > > > +int rcar_gen4_pcie_get_resources(struct rcar_gen4_pcie *rcar,
> > > > > > > +				 struct platform_device *pdev);
> > > > > > > +struct rcar_gen4_pcie *rcar_gen4_pcie_devm_alloc(struct device *dev);
> > > > > > > +
> > > > > > > +#endif /* _PCIE_RCAR_GEN4_H_ */
> > > > > > > --
> > > > > > > 2.25.1
> > > > > > >

RE: [PATCH v16 19/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support

[Yoshihiro Shimoda]

Hello Serge,

> From: Serge Semin, Sent: Friday, June 9, 2023 7:54 PM
<snip>
> > > > static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > > {
> > > >         struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > >         int i;
> > > >
> > > >         rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > >
> > > >         /*
> > > >          * Require direct speed change here. Otherwise RDLH_LINK_UP of
> > > >          * PCIEINTSTS0 which is this controller specific register may not
> > > >          * be set.
> > > >          */
> > >
> > > >         if (rcar->needs_speed_change) {
> > >
> > > Seeing this is specified for the root port only what about
> > > replacing the statement with just test whether the rcar_gen4_pcie.mode ==
> > > DW_PCIE_RC_TYPE? Thus you'll be ablt to drop the needs_speed_change field.
> >
> > Thank you for the comment. I'll fix it.
> >
> > > BTW Just curious. Why is the loop below enabled for the Root Port
> > > only? What about the end-point controller? It's the same hardware
> > > after all..
> >
> > This is reused from v16 and then it used "link retraining" which is only for
> > the Root Port. As you mentioned, it seems endpoint controller is also needed
> > if we use direct speed change.
> >
> > > >                 for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > > >                         rcar_gen4_pcie_speed_change(dw);
> > > >                         msleep(100);
> > > >                         if (rcar_gen4_pcie_check_current_link(dw))
> > > >                                 return 0;
> > > >                 }
> > >
> > > Did you trace how many iterations this loop normally takes?
> >
> > i = 0 or 1 (if the max-link-speed is suitable for a connected device.)
> >
> > > Is it
> > > constant or varies for the same platform setup and a connected link
> > > partner? Does the number of iterations depend on the target link speed
> > > specified via the "max-link-speed" property?
> >
> 
> > This is not related to the "max-link-speed". It seems to related to
> > a link partner.
> > 		LinkCap	max-link-speed	loop
> > Device A		4	4		1
> > Device A		4	3		1
> > Device B		3	3		0
> 
> Great! If so I would have just left a single unconditional
> rcar_gen4_pcie_speed_change() call placed right after the
> rcar_gen4_pcie_ltssm_enable() method with no delays afterwards. These
> methods would have been invoked in the framework of
> dw_pcie_start_link() after which the dw_pcie_wait_for_link() method is
> called with several checks parted with the ~100ms delay. It will make
> sure that at least some link is up with the link state printed to the
> system log. If for some reason the performance degradation happens
> then it will be up to the system administrator to investigate what was
> wrong. Your driver did as much is it could to reach the best link gen.

IIUC, is your suggestion like the following code?
---
	rcar_gen4_pcie_ltssm_enable(rcar, true);
	if (!dw_pcie_wait_for_link(dw)) {
		rcar_gen4_pcie_speed_change(dw);
		return 0;
	}
---

Unfortunately, it doesn't work correctly...
The following code can work correctly. The value of i is still 1 on the device A.
What do you think that the following code is acceptable?
---
	rcar_gen4_pcie_ltssm_enable(rcar, true);
	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
		msleep(100);
		rcar_gen4_pcie_speed_change(dw);
		if (dw_pcie_link_up(dw)) {
			printk("%s:%d\n", __func__, i);
			return 0;
		}
	}
---

Best regards,
Yoshihiro Shimoda

> -Serge(y)

Re: [PATCH v16 19/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support

[Serge Semin]

On Mon, Jun 12, 2023 at 01:19:02PM +0000, Yoshihiro Shimoda wrote:
> Hello Serge,
> 
> > From: Serge Semin, Sent: Friday, June 9, 2023 7:54 PM
> <snip>
> > > > > static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > > > {
> > > > >         struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > >         int i;
> > > > >
> > > > >         rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > >
> > > > >         /*
> > > > >          * Require direct speed change here. Otherwise RDLH_LINK_UP of
> > > > >          * PCIEINTSTS0 which is this controller specific register may not
> > > > >          * be set.
> > > > >          */
> > > >
> > > > >         if (rcar->needs_speed_change) {
> > > >
> > > > Seeing this is specified for the root port only what about
> > > > replacing the statement with just test whether the rcar_gen4_pcie.mode ==
> > > > DW_PCIE_RC_TYPE? Thus you'll be ablt to drop the needs_speed_change field.
> > >
> > > Thank you for the comment. I'll fix it.
> > >
> > > > BTW Just curious. Why is the loop below enabled for the Root Port
> > > > only? What about the end-point controller? It's the same hardware
> > > > after all..
> > >
> > > This is reused from v16 and then it used "link retraining" which is only for
> > > the Root Port. As you mentioned, it seems endpoint controller is also needed
> > > if we use direct speed change.
> > >
> > > > >                 for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > > > >                         rcar_gen4_pcie_speed_change(dw);
> > > > >                         msleep(100);
> > > > >                         if (rcar_gen4_pcie_check_current_link(dw))
> > > > >                                 return 0;
> > > > >                 }
> > > >
> > > > Did you trace how many iterations this loop normally takes?
> > >
> > > i = 0 or 1 (if the max-link-speed is suitable for a connected device.)
> > >
> > > > Is it
> > > > constant or varies for the same platform setup and a connected link
> > > > partner? Does the number of iterations depend on the target link speed
> > > > specified via the "max-link-speed" property?
> > >
> > 
> > > This is not related to the "max-link-speed". It seems to related to
> > > a link partner.
> > > 		LinkCap	max-link-speed	loop
> > > Device A		4	4		1
> > > Device A		4	3		1
> > > Device B		3	3		0
> > 
> > Great! If so I would have just left a single unconditional
> > rcar_gen4_pcie_speed_change() call placed right after the
> > rcar_gen4_pcie_ltssm_enable() method with no delays afterwards. These
> > methods would have been invoked in the framework of
> > dw_pcie_start_link() after which the dw_pcie_wait_for_link() method is
> > called with several checks parted with the ~100ms delay. It will make
> > sure that at least some link is up with the link state printed to the
> > system log. If for some reason the performance degradation happens
> > then it will be up to the system administrator to investigate what was
> > wrong. Your driver did as much is it could to reach the best link gen.
> 
> IIUC, is your suggestion like the following code?
> ---
> 	rcar_gen4_pcie_ltssm_enable(rcar, true);
> 	if (!dw_pcie_wait_for_link(dw)) {
> 		rcar_gen4_pcie_speed_change(dw);
> 		return 0;
> 	}
> ---
> 
> Unfortunately, it doesn't work correctly...
> The following code can work correctly. The value of i is still 1 on the device A.
> What do you think that the following code is acceptable?
> ---
> 	rcar_gen4_pcie_ltssm_enable(rcar, true);
> 	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> 		msleep(100);
> 		rcar_gen4_pcie_speed_change(dw);
> 		if (dw_pcie_link_up(dw)) {
> 			printk("%s:%d\n", __func__, i);
> 			return 0;
> 		}
> 	}
> ---

My idea was to implement something like this:

+static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
+{
+	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
+
+	rcar_gen4_pcie_ltssm_enable(rcar, true);
+
+	rcar_gen4_pcie_speed_change(dw);
+
+	return 0;
+}

and retain the rcar_gen4_pcie_link_up() method as is.

* Note: originally your loop used to have the msleep() call performed
after the first rcar_gen4_pcie_speed_change() invocation. Thus the
delay can be dropped if there is only one iteration implemented (see
further to understand why).

You don't need to wait for the link to actually get up in the
start_link() callback because there is the link_up() callback, which
is called from the dw_pcie_wait_for_link() method during the generic
DWC PCIe setup procedure. See:

dw_pcie_host_init():
+-> ops->host_init()
+-> ...
+-> dw_pcie_setup_rc()
|   +-> ...
|   +-> dw_pcie_setup()
|   +-> ...
+-> if !dw_pcie_link_up()
|   |   +-> ops->link_up()
|   +-> dw_pcie_start_link()
|       +-> ops->start_link()
+-> dw_pcie_wait_for_link();   // See, wait-procedure is already performed
|   +-> loop 10 times          // for you in the core driver together
|       +-> dw_pcie_link_up()  // with the delays between the checks
|           +-> ops->link_up()
|       +-> msleep(~100)
+-> ...

-Serge(y)

> 
> Best regards,
> Yoshihiro Shimoda
> 
> > -Serge(y)
> 

RE: [PATCH v16 19/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support

[Yoshihiro Shimoda]

Hello Serge,

 From: Serge Semin, Sent: Tuesday, June 13, 2023 4:52 AM
> 
> On Mon, Jun 12, 2023 at 01:19:02PM +0000, Yoshihiro Shimoda wrote:
> > Hello Serge,
> >
> > > From: Serge Semin, Sent: Friday, June 9, 2023 7:54 PM
> > <snip>
> > > > > > static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > > > > {
> > > > > >         struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > >         int i;
> > > > > >
> > > > > >         rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > > >
> > > > > >         /*
> > > > > >          * Require direct speed change here. Otherwise RDLH_LINK_UP of
> > > > > >          * PCIEINTSTS0 which is this controller specific register may not
> > > > > >          * be set.
> > > > > >          */
> > > > >
> > > > > >         if (rcar->needs_speed_change) {
> > > > >
> > > > > Seeing this is specified for the root port only what about
> > > > > replacing the statement with just test whether the rcar_gen4_pcie.mode ==
> > > > > DW_PCIE_RC_TYPE? Thus you'll be ablt to drop the needs_speed_change field.
> > > >
> > > > Thank you for the comment. I'll fix it.
> > > >
> > > > > BTW Just curious. Why is the loop below enabled for the Root Port
> > > > > only? What about the end-point controller? It's the same hardware
> > > > > after all..
> > > >
> > > > This is reused from v16 and then it used "link retraining" which is only for
> > > > the Root Port. As you mentioned, it seems endpoint controller is also needed
> > > > if we use direct speed change.
> > > >
> > > > > >                 for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > > > > >                         rcar_gen4_pcie_speed_change(dw);
> > > > > >                         msleep(100);
> > > > > >                         if (rcar_gen4_pcie_check_current_link(dw))
> > > > > >                                 return 0;
> > > > > >                 }
> > > > >
> > > > > Did you trace how many iterations this loop normally takes?
> > > >
> > > > i = 0 or 1 (if the max-link-speed is suitable for a connected device.)
> > > >
> > > > > Is it
> > > > > constant or varies for the same platform setup and a connected link
> > > > > partner? Does the number of iterations depend on the target link speed
> > > > > specified via the "max-link-speed" property?
> > > >
> > >
> > > > This is not related to the "max-link-speed". It seems to related to
> > > > a link partner.
> > > > 		LinkCap	max-link-speed	loop
> > > > Device A		4	4		1
> > > > Device A		4	3		1
> > > > Device B		3	3		0
> > >
> > > Great! If so I would have just left a single unconditional
> > > rcar_gen4_pcie_speed_change() call placed right after the
> > > rcar_gen4_pcie_ltssm_enable() method with no delays afterwards. These
> > > methods would have been invoked in the framework of
> > > dw_pcie_start_link() after which the dw_pcie_wait_for_link() method is
> > > called with several checks parted with the ~100ms delay. It will make
> > > sure that at least some link is up with the link state printed to the
> > > system log. If for some reason the performance degradation happens
> > > then it will be up to the system administrator to investigate what was
> > > wrong. Your driver did as much is it could to reach the best link gen.
> >
> > IIUC, is your suggestion like the following code?
> > ---
> > 	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > 	if (!dw_pcie_wait_for_link(dw)) {
> > 		rcar_gen4_pcie_speed_change(dw);
> > 		return 0;
> > 	}
> > ---
> >
> > Unfortunately, it doesn't work correctly...
> > The following code can work correctly. The value of i is still 1 on the device A.
> > What do you think that the following code is acceptable?
> > ---
> > 	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > 	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > 		msleep(100);
> > 		rcar_gen4_pcie_speed_change(dw);
> > 		if (dw_pcie_link_up(dw)) {
> > 			printk("%s:%d\n", __func__, i);
> > 			return 0;
> > 		}
> > 	}
> > ---
> 
> My idea was to implement something like this:
> 
> +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> +{
> +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> +
> +	rcar_gen4_pcie_ltssm_enable(rcar, true);
> +
> +	rcar_gen4_pcie_speed_change(dw);
> +
> +	return 0;
> +}
> 
> and retain the rcar_gen4_pcie_link_up() method as is.

Unfortunately, such a code doesn't work on my environment...

> * Note: originally your loop used to have the msleep() call performed
> after the first rcar_gen4_pcie_speed_change() invocation. Thus the
> delay can be dropped if there is only one iteration implemented (see
> further to understand why).

Calling rcar_gen4_pcie_speed_change() multiple times is required on
my environment. I thought msleep(100) was quite long so that I tried
other wait interval like below:

 msleep(1) : about 5 loops is needed for link. (about 5 msec.)
 usleep_range(100, 110) : about 400 loops is needed for link. (about 40 msec.)
 usleep_range(500, 600) : about 80 loops is needed for link. (about 40 msec.)

The delay timing doesn't seems important. Both cases below can work correctly.
--- case 1 ---
	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
		rcar_gen4_pcie_speed_change(dw);
		if (dw_pcie_link_up(dw)) {
			printk("%s:%d\n", __func__, i); // will be removed
			return 0;
		}
		msleep(1);
	}
---
--- case 2 ---
	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
		rcar_gen4_pcie_speed_change(dw);
		msleep(1);
		if (dw_pcie_link_up(dw)) {
			printk("%s:%d\n", __func__, i); // will be removed
			return 0;
		}
	}
---

So, I'll use case 1 for it.

> You don't need to wait for the link to actually get up in the
> start_link() callback because there is the link_up() callback, which
> is called from the dw_pcie_wait_for_link() method during the generic
> DWC PCIe setup procedure. See:

Since the procedure will call rcar_gen4_pcie_speed_change() from
->start_link() once, my environment cannot work correctly...

Best regards,
Yoshihiro Shimoda

> dw_pcie_host_init():
> +-> ops->host_init()
> +-> ...
> +-> dw_pcie_setup_rc()
> |   +-> ...
> |   +-> dw_pcie_setup()
> |   +-> ...
> +-> if !dw_pcie_link_up()
> |   |   +-> ops->link_up()
> |   +-> dw_pcie_start_link()
> |       +-> ops->start_link()
> +-> dw_pcie_wait_for_link();   // See, wait-procedure is already performed
> |   +-> loop 10 times          // for you in the core driver together
> |       +-> dw_pcie_link_up()  // with the delays between the checks
> |           +-> ops->link_up()
> |       +-> msleep(~100)
> +-> ...
> 
> -Serge(y)
> 
> >
> > Best regards,
> > Yoshihiro Shimoda
> >
> > > -Serge(y)
> >

Re: [PATCH v16 19/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support

[Serge Semin]

On Wed, Jun 14, 2023 at 02:30:13AM +0000, Yoshihiro Shimoda wrote:
> Hello Serge,
> 
>  From: Serge Semin, Sent: Tuesday, June 13, 2023 4:52 AM
> > 
> > On Mon, Jun 12, 2023 at 01:19:02PM +0000, Yoshihiro Shimoda wrote:
> > > Hello Serge,
> > >
> > > > From: Serge Semin, Sent: Friday, June 9, 2023 7:54 PM
> > > <snip>
> > > > > > > static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > > > > > {
> > > > > > >         struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > > >         int i;
> > > > > > >
> > > > > > >         rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > > > >
> > > > > > >         /*
> > > > > > >          * Require direct speed change here. Otherwise RDLH_LINK_UP of
> > > > > > >          * PCIEINTSTS0 which is this controller specific register may not
> > > > > > >          * be set.
> > > > > > >          */
> > > > > >
> > > > > > >         if (rcar->needs_speed_change) {
> > > > > >
> > > > > > Seeing this is specified for the root port only what about
> > > > > > replacing the statement with just test whether the rcar_gen4_pcie.mode ==
> > > > > > DW_PCIE_RC_TYPE? Thus you'll be ablt to drop the needs_speed_change field.
> > > > >
> > > > > Thank you for the comment. I'll fix it.
> > > > >
> > > > > > BTW Just curious. Why is the loop below enabled for the Root Port
> > > > > > only? What about the end-point controller? It's the same hardware
> > > > > > after all..
> > > > >
> > > > > This is reused from v16 and then it used "link retraining" which is only for
> > > > > the Root Port. As you mentioned, it seems endpoint controller is also needed
> > > > > if we use direct speed change.
> > > > >
> > > > > > >                 for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > > > > > >                         rcar_gen4_pcie_speed_change(dw);
> > > > > > >                         msleep(100);
> > > > > > >                         if (rcar_gen4_pcie_check_current_link(dw))
> > > > > > >                                 return 0;
> > > > > > >                 }
> > > > > >
> > > > > > Did you trace how many iterations this loop normally takes?
> > > > >
> > > > > i = 0 or 1 (if the max-link-speed is suitable for a connected device.)
> > > > >
> > > > > > Is it
> > > > > > constant or varies for the same platform setup and a connected link
> > > > > > partner? Does the number of iterations depend on the target link speed
> > > > > > specified via the "max-link-speed" property?
> > > > >
> > > >
> > > > > This is not related to the "max-link-speed". It seems to related to
> > > > > a link partner.
> > > > > 		LinkCap	max-link-speed	loop
> > > > > Device A		4	4		1
> > > > > Device A		4	3		1
> > > > > Device B		3	3		0
> > > >
> > > > Great! If so I would have just left a single unconditional
> > > > rcar_gen4_pcie_speed_change() call placed right after the
> > > > rcar_gen4_pcie_ltssm_enable() method with no delays afterwards. These
> > > > methods would have been invoked in the framework of
> > > > dw_pcie_start_link() after which the dw_pcie_wait_for_link() method is
> > > > called with several checks parted with the ~100ms delay. It will make
> > > > sure that at least some link is up with the link state printed to the
> > > > system log. If for some reason the performance degradation happens
> > > > then it will be up to the system administrator to investigate what was
> > > > wrong. Your driver did as much is it could to reach the best link gen.
> > >
> > > IIUC, is your suggestion like the following code?
> > > ---
> > > 	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > 	if (!dw_pcie_wait_for_link(dw)) {
> > > 		rcar_gen4_pcie_speed_change(dw);
> > > 		return 0;
> > > 	}
> > > ---
> > >
> > > Unfortunately, it doesn't work correctly...
> > > The following code can work correctly. The value of i is still 1 on the device A.
> > > What do you think that the following code is acceptable?
> > > ---
> > > 	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > 	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > > 		msleep(100);
> > > 		rcar_gen4_pcie_speed_change(dw);
> > > 		if (dw_pcie_link_up(dw)) {
> > > 			printk("%s:%d\n", __func__, i);
> > > 			return 0;
> > > 		}
> > > 	}
> > > ---
> > 
> > My idea was to implement something like this:
> > 
> > +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > +{
> > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > +
> > +	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > +
> > +	rcar_gen4_pcie_speed_change(dw);
> > +
> > +	return 0;
> > +}
> > 
> > and retain the rcar_gen4_pcie_link_up() method as is.
> 
> Unfortunately, such a code doesn't work on my environment...
> 
> > * Note: originally your loop used to have the msleep() call performed
> > after the first rcar_gen4_pcie_speed_change() invocation. Thus the
> > delay can be dropped if there is only one iteration implemented (see
> > further to understand why).
> 
> Calling rcar_gen4_pcie_speed_change() multiple times is required on
> my environment. I thought msleep(100) was quite long so that I tried
> other wait interval like below:
> 
>  msleep(1) : about 5 loops is needed for link. (about 5 msec.)
>  usleep_range(100, 110) : about 400 loops is needed for link. (about 40 msec.)
>  usleep_range(500, 600) : about 80 loops is needed for link. (about 40 msec.)
> 
> The delay timing doesn't seems important. Both cases below can work correctly.
> --- case 1 ---
> 	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> 		rcar_gen4_pcie_speed_change(dw);
> 		if (dw_pcie_link_up(dw)) {
> 			printk("%s:%d\n", __func__, i); // will be removed
> 			return 0;
> 		}

> 		msleep(1);

Why? Just set it to 5 ms. In anyway please see the next message.

> 	}
> ---
> --- case 2 ---
> 	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> 		rcar_gen4_pcie_speed_change(dw);
> 		msleep(1);
> 		if (dw_pcie_link_up(dw)) {
> 			printk("%s:%d\n", __func__, i); // will be removed
> 			return 0;
> 		}
> 	}
> ---
> 
> So, I'll use case 1 for it.

Ah. I think I get it now. Your spreadsheet:

                LinkCap max-link-speed  loop
Device A           4          4           1
Device A           4          3           1
Device B           3          3           0

actually meant (loop+1) iterations. So in case of Gen4 you'll need
three speed changes (one already enabled in the dw_pcie_setup_rc()
method and another two ones are performed in your loop). Similarly in
case of Gen3 you'll need only one iteration. I bet you won't need to
call rcar_gen4_pcie_speed_change() at all if gen2 needs to be trained.
Could you try it out?

Anyway based on what you discovered and on my experience working with
that controller, there should be as many
GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag changes as the target speed
value, i.e. no flag switch if Gen1 is required, one flag switch if
Gen2 is required and so on. Although I failed to find any explicit
statement about that in the HW-manual.

In addition to the above I've found out that
GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE field is actually self cleared when
the speed change occurs (see the register description in the HW
reference manual). We can use it to implement the
dw_pcie_link_up()-independent link training algorithm like this:

+#define RCAR_RETRAIN_MAX_CHECK		10
+#define RCAR_LINK_SPEED_MAX		4
+
+static bool rcar_gen4_pcie_speed_change(struct dw_pcie *dw)
+{
+	u32 val;
+	int i;
+
+	val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
+	val &= ~PORT_LOGIC_SPEED_CHANGE;
+	dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
+
+	val |= PORT_LOGIC_SPEED_CHANGE;
+	dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
+
+	for (i = 0; i < RCAR_SPEED_CHANGE_WAIT_RETRIES; i++) {
+		val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
+		if (!(val & PORT_LOGIC_SPEED_CHANGE))
+			return true;
+
+		msleep(1);
+	}
+
+	return false;
+}
+
+static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
+{
+	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
+	int i, changes;
+
+	rcar_gen4_pcie_ltssm_enable(rcar, true);
+
+	changes = min_not_zero(dw->link_gen, RCAR_LINK_SPEED_MAX);
+	for (i = 0; i < changes; ++i) {
+		if (!rcar_gen4_pcie_speed_change(dw))
+			break;
+	}
+
+	return 0;
+}

Note 1. The actual link state will be checked in the framework of the
dw_pcie_wait_for_link() function, by means of dw_pcie_link_up().

Note 2. RCAR_LINK_SPEED_MAX is deliberately set to 4 because DW PCIe
EP core driver doesn't set the PORT_LOGIC_SPEED_CHANGE flag. In case
of the DW PCIe Root Port at most 3 iterations should be enough.

Note 3. Please use the RCAR_ prefix for the vendor-specific macros.
It concerns the entire series.

Could you try out the code suggested above?

-Serge(y)

> 
> > You don't need to wait for the link to actually get up in the
> > start_link() callback because there is the link_up() callback, which
> > is called from the dw_pcie_wait_for_link() method during the generic
> > DWC PCIe setup procedure. See:
> 
> Since the procedure will call rcar_gen4_pcie_speed_change() from
> ->start_link() once, my environment cannot work correctly...
> 
> Best regards,
> Yoshihiro Shimoda
> 
> > dw_pcie_host_init():
> > +-> ops->host_init()
> > +-> ...
> > +-> dw_pcie_setup_rc()
> > |   +-> ...
> > |   +-> dw_pcie_setup()
> > |   +-> ...
> > +-> if !dw_pcie_link_up()
> > |   |   +-> ops->link_up()
> > |   +-> dw_pcie_start_link()
> > |       +-> ops->start_link()
> > +-> dw_pcie_wait_for_link();   // See, wait-procedure is already performed
> > |   +-> loop 10 times          // for you in the core driver together
> > |       +-> dw_pcie_link_up()  // with the delays between the checks
> > |           +-> ops->link_up()
> > |       +-> msleep(~100)
> > +-> ...
> > 
> > -Serge(y)
> > 
> > >
> > > Best regards,
> > > Yoshihiro Shimoda
> > >
> > > > -Serge(y)
> > >

Re: [PATCH v16 19/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support

[Serge Semin]

On Wed, Jun 14, 2023 at 02:39:29PM +0300, Serge Semin wrote:
> On Wed, Jun 14, 2023 at 02:30:13AM +0000, Yoshihiro Shimoda wrote:
> > Hello Serge,
> > 
> >  From: Serge Semin, Sent: Tuesday, June 13, 2023 4:52 AM
> > > 
> > > On Mon, Jun 12, 2023 at 01:19:02PM +0000, Yoshihiro Shimoda wrote:
> > > > Hello Serge,
> > > >
> > > > > From: Serge Semin, Sent: Friday, June 9, 2023 7:54 PM
> > > > <snip>
> > > > > > > > static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > > > > > > {
> > > > > > > >         struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > > > >         int i;
> > > > > > > >
> > > > > > > >         rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > > > > >
> > > > > > > >         /*
> > > > > > > >          * Require direct speed change here. Otherwise RDLH_LINK_UP of
> > > > > > > >          * PCIEINTSTS0 which is this controller specific register may not
> > > > > > > >          * be set.
> > > > > > > >          */
> > > > > > >
> > > > > > > >         if (rcar->needs_speed_change) {
> > > > > > >
> > > > > > > Seeing this is specified for the root port only what about
> > > > > > > replacing the statement with just test whether the rcar_gen4_pcie.mode ==
> > > > > > > DW_PCIE_RC_TYPE? Thus you'll be ablt to drop the needs_speed_change field.
> > > > > >
> > > > > > Thank you for the comment. I'll fix it.
> > > > > >
> > > > > > > BTW Just curious. Why is the loop below enabled for the Root Port
> > > > > > > only? What about the end-point controller? It's the same hardware
> > > > > > > after all..
> > > > > >
> > > > > > This is reused from v16 and then it used "link retraining" which is only for
> > > > > > the Root Port. As you mentioned, it seems endpoint controller is also needed
> > > > > > if we use direct speed change.
> > > > > >
> > > > > > > >                 for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > > > > > > >                         rcar_gen4_pcie_speed_change(dw);
> > > > > > > >                         msleep(100);
> > > > > > > >                         if (rcar_gen4_pcie_check_current_link(dw))
> > > > > > > >                                 return 0;
> > > > > > > >                 }
> > > > > > >
> > > > > > > Did you trace how many iterations this loop normally takes?
> > > > > >
> > > > > > i = 0 or 1 (if the max-link-speed is suitable for a connected device.)
> > > > > >
> > > > > > > Is it
> > > > > > > constant or varies for the same platform setup and a connected link
> > > > > > > partner? Does the number of iterations depend on the target link speed
> > > > > > > specified via the "max-link-speed" property?
> > > > > >
> > > > >
> > > > > > This is not related to the "max-link-speed". It seems to related to
> > > > > > a link partner.
> > > > > > 		LinkCap	max-link-speed	loop
> > > > > > Device A		4	4		1
> > > > > > Device A		4	3		1
> > > > > > Device B		3	3		0
> > > > >
> > > > > Great! If so I would have just left a single unconditional
> > > > > rcar_gen4_pcie_speed_change() call placed right after the
> > > > > rcar_gen4_pcie_ltssm_enable() method with no delays afterwards. These
> > > > > methods would have been invoked in the framework of
> > > > > dw_pcie_start_link() after which the dw_pcie_wait_for_link() method is
> > > > > called with several checks parted with the ~100ms delay. It will make
> > > > > sure that at least some link is up with the link state printed to the
> > > > > system log. If for some reason the performance degradation happens
> > > > > then it will be up to the system administrator to investigate what was
> > > > > wrong. Your driver did as much is it could to reach the best link gen.
> > > >
> > > > IIUC, is your suggestion like the following code?
> > > > ---
> > > > 	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > 	if (!dw_pcie_wait_for_link(dw)) {
> > > > 		rcar_gen4_pcie_speed_change(dw);
> > > > 		return 0;
> > > > 	}
> > > > ---
> > > >
> > > > Unfortunately, it doesn't work correctly...
> > > > The following code can work correctly. The value of i is still 1 on the device A.
> > > > What do you think that the following code is acceptable?
> > > > ---
> > > > 	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > 	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > > > 		msleep(100);
> > > > 		rcar_gen4_pcie_speed_change(dw);
> > > > 		if (dw_pcie_link_up(dw)) {
> > > > 			printk("%s:%d\n", __func__, i);
> > > > 			return 0;
> > > > 		}
> > > > 	}
> > > > ---
> > > 
> > > My idea was to implement something like this:
> > > 
> > > +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > +{
> > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > +
> > > +	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > +
> > > +	rcar_gen4_pcie_speed_change(dw);
> > > +
> > > +	return 0;
> > > +}
> > > 
> > > and retain the rcar_gen4_pcie_link_up() method as is.
> > 
> > Unfortunately, such a code doesn't work on my environment...
> > 
> > > * Note: originally your loop used to have the msleep() call performed
> > > after the first rcar_gen4_pcie_speed_change() invocation. Thus the
> > > delay can be dropped if there is only one iteration implemented (see
> > > further to understand why).
> > 
> > Calling rcar_gen4_pcie_speed_change() multiple times is required on
> > my environment. I thought msleep(100) was quite long so that I tried
> > other wait interval like below:
> > 
> >  msleep(1) : about 5 loops is needed for link. (about 5 msec.)
> >  usleep_range(100, 110) : about 400 loops is needed for link. (about 40 msec.)
> >  usleep_range(500, 600) : about 80 loops is needed for link. (about 40 msec.)
> > 
> > The delay timing doesn't seems important. Both cases below can work correctly.
> > --- case 1 ---
> > 	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > 		rcar_gen4_pcie_speed_change(dw);
> > 		if (dw_pcie_link_up(dw)) {
> > 			printk("%s:%d\n", __func__, i); // will be removed
> > 			return 0;
> > 		}
> 
> > 		msleep(1);
> 
> Why? Just set it to 5 ms. In anyway please see the next message.
> 
> > 	}
> > ---
> > --- case 2 ---
> > 	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > 		rcar_gen4_pcie_speed_change(dw);
> > 		msleep(1);
> > 		if (dw_pcie_link_up(dw)) {
> > 			printk("%s:%d\n", __func__, i); // will be removed
> > 			return 0;
> > 		}
> > 	}
> > ---
> > 
> > So, I'll use case 1 for it.
> 
> Ah. I think I get it now. Your spreadsheet:
> 
>                 LinkCap max-link-speed  loop
> Device A           4          4           1
> Device A           4          3           1
> Device B           3          3           0
> 
> actually meant (loop+1) iterations. So in case of Gen4 you'll need
> three speed changes (one already enabled in the dw_pcie_setup_rc()
> method and another two ones are performed in your loop). Similarly in
> case of Gen3 you'll need only one iteration. I bet you won't need to
> call rcar_gen4_pcie_speed_change() at all if gen2 needs to be trained.
> Could you try it out?
> 
> Anyway based on what you discovered and on my experience working with
> that controller, there should be as many
> GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag changes as the target speed
> value, i.e. no flag switch if Gen1 is required, one flag switch if
> Gen2 is required and so on. Although I failed to find any explicit
> statement about that in the HW-manual.
> 
> In addition to the above I've found out that
> GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE field is actually self cleared when
> the speed change occurs (see the register description in the HW
> reference manual). We can use it to implement the
> dw_pcie_link_up()-independent link training algorithm like this:
> 
> +#define RCAR_RETRAIN_MAX_CHECK		10
> +#define RCAR_LINK_SPEED_MAX		4
> +
> +static bool rcar_gen4_pcie_speed_change(struct dw_pcie *dw)
> +{
> +	u32 val;
> +	int i;
> +
> +	val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
> +	val &= ~PORT_LOGIC_SPEED_CHANGE;
> +	dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
> +
> +	val |= PORT_LOGIC_SPEED_CHANGE;
> +	dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
> +
> +	for (i = 0; i < RCAR_SPEED_CHANGE_WAIT_RETRIES; i++) {
> +		val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
> +		if (!(val & PORT_LOGIC_SPEED_CHANGE))
> +			return true;
> +
> +		msleep(1);
> +	}
> +
> +	return false;
> +}
> +
> +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> +{
> +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> +	int i, changes;
> +
> +	rcar_gen4_pcie_ltssm_enable(rcar, true);
> +

> +	changes = min_not_zero(dw->link_gen, RCAR_LINK_SPEED_MAX);

This should have been:
+changes = min_not_zero(dw->link_gen, RCAR_LINK_SPEED_MAX) - 1;
because Gen1 doesn't need any speed change action.

But this part can be further improved. Instead of the code above the
next snipped can be implemented:

+changes = min_not_zero(dw->link_gen, RCAR_LINK_SPEED_MAX) - 1;
+if (changes && rcar->mode == DW_PCIE_RC_TYPE)
+		changes -= 1;

It takes into account that the GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE
flag is already set in the dw_pcie_setup_rc() method. So Gen2 will be
trained with no need in addition actions. If it's supported of course.

-Serge(y)

> +	for (i = 0; i < changes; ++i) {
> +		if (!rcar_gen4_pcie_speed_change(dw))
> +			break;
> +	}
> +
> +	return 0;
> +}
> 
> Note 1. The actual link state will be checked in the framework of the
> dw_pcie_wait_for_link() function, by means of dw_pcie_link_up().
> 
> Note 2. RCAR_LINK_SPEED_MAX is deliberately set to 4 because DW PCIe
> EP core driver doesn't set the PORT_LOGIC_SPEED_CHANGE flag. In case
> of the DW PCIe Root Port at most 3 iterations should be enough.
> 
> Note 3. Please use the RCAR_ prefix for the vendor-specific macros.
> It concerns the entire series.
> 
> Could you try out the code suggested above?
> 
> -Serge(y)
> 
> > 
> > > You don't need to wait for the link to actually get up in the
> > > start_link() callback because there is the link_up() callback, which
> > > is called from the dw_pcie_wait_for_link() method during the generic
> > > DWC PCIe setup procedure. See:
> > 
> > Since the procedure will call rcar_gen4_pcie_speed_change() from
> > ->start_link() once, my environment cannot work correctly...
> > 
> > Best regards,
> > Yoshihiro Shimoda
> > 
> > > dw_pcie_host_init():
> > > +-> ops->host_init()
> > > +-> ...
> > > +-> dw_pcie_setup_rc()
> > > |   +-> ...
> > > |   +-> dw_pcie_setup()
> > > |   +-> ...
> > > +-> if !dw_pcie_link_up()
> > > |   |   +-> ops->link_up()
> > > |   +-> dw_pcie_start_link()
> > > |       +-> ops->start_link()
> > > +-> dw_pcie_wait_for_link();   // See, wait-procedure is already performed
> > > |   +-> loop 10 times          // for you in the core driver together
> > > |       +-> dw_pcie_link_up()  // with the delays between the checks
> > > |           +-> ops->link_up()
> > > |       +-> msleep(~100)
> > > +-> ...
> > > 
> > > -Serge(y)
> > > 
> > > >
> > > > Best regards,
> > > > Yoshihiro Shimoda
> > > >
> > > > > -Serge(y)
> > > >

RE: [PATCH v16 19/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support

[Yoshihiro Shimoda]

Hello Serge,

> From: Serge Semin, Sent: Thursday, June 15, 2023 4:32 AM
> 
> On Wed, Jun 14, 2023 at 02:39:29PM +0300, Serge Semin wrote:
> > On Wed, Jun 14, 2023 at 02:30:13AM +0000, Yoshihiro Shimoda wrote:
> > > Hello Serge,
> > >
> > >  From: Serge Semin, Sent: Tuesday, June 13, 2023 4:52 AM
> > > >
> > > > On Mon, Jun 12, 2023 at 01:19:02PM +0000, Yoshihiro Shimoda wrote:
> > > > > Hello Serge,
> > > > >
> > > > > > From: Serge Semin, Sent: Friday, June 9, 2023 7:54 PM
> > > > > <snip>
> > > > > > > > > static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > > > > > > > {
> > > > > > > > >         struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > > > > >         int i;
> > > > > > > > >
> > > > > > > > >         rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > > > > > >
> > > > > > > > >         /*
> > > > > > > > >          * Require direct speed change here. Otherwise RDLH_LINK_UP of
> > > > > > > > >          * PCIEINTSTS0 which is this controller specific register may not
> > > > > > > > >          * be set.
> > > > > > > > >          */
> > > > > > > >
> > > > > > > > >         if (rcar->needs_speed_change) {
> > > > > > > >
> > > > > > > > Seeing this is specified for the root port only what about
> > > > > > > > replacing the statement with just test whether the rcar_gen4_pcie.mode ==
> > > > > > > > DW_PCIE_RC_TYPE? Thus you'll be ablt to drop the needs_speed_change field.
> > > > > > >
> > > > > > > Thank you for the comment. I'll fix it.
> > > > > > >
> > > > > > > > BTW Just curious. Why is the loop below enabled for the Root Port
> > > > > > > > only? What about the end-point controller? It's the same hardware
> > > > > > > > after all..
> > > > > > >
> > > > > > > This is reused from v16 and then it used "link retraining" which is only for
> > > > > > > the Root Port. As you mentioned, it seems endpoint controller is also needed
> > > > > > > if we use direct speed change.
> > > > > > >
> > > > > > > > >                 for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > > > > > > > >                         rcar_gen4_pcie_speed_change(dw);
> > > > > > > > >                         msleep(100);
> > > > > > > > >                         if (rcar_gen4_pcie_check_current_link(dw))
> > > > > > > > >                                 return 0;
> > > > > > > > >                 }
> > > > > > > >
> > > > > > > > Did you trace how many iterations this loop normally takes?
> > > > > > >
> > > > > > > i = 0 or 1 (if the max-link-speed is suitable for a connected device.)
> > > > > > >
> > > > > > > > Is it
> > > > > > > > constant or varies for the same platform setup and a connected link
> > > > > > > > partner? Does the number of iterations depend on the target link speed
> > > > > > > > specified via the "max-link-speed" property?
> > > > > > >
> > > > > >
> > > > > > > This is not related to the "max-link-speed". It seems to related to
> > > > > > > a link partner.
> > > > > > > 		LinkCap	max-link-speed	loop
> > > > > > > Device A		4	4		1
> > > > > > > Device A		4	3		1
> > > > > > > Device B		3	3		0
> > > > > >
> > > > > > Great! If so I would have just left a single unconditional
> > > > > > rcar_gen4_pcie_speed_change() call placed right after the
> > > > > > rcar_gen4_pcie_ltssm_enable() method with no delays afterwards. These
> > > > > > methods would have been invoked in the framework of
> > > > > > dw_pcie_start_link() after which the dw_pcie_wait_for_link() method is
> > > > > > called with several checks parted with the ~100ms delay. It will make
> > > > > > sure that at least some link is up with the link state printed to the
> > > > > > system log. If for some reason the performance degradation happens
> > > > > > then it will be up to the system administrator to investigate what was
> > > > > > wrong. Your driver did as much is it could to reach the best link gen.
> > > > >
> > > > > IIUC, is your suggestion like the following code?
> > > > > ---
> > > > > 	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > > 	if (!dw_pcie_wait_for_link(dw)) {
> > > > > 		rcar_gen4_pcie_speed_change(dw);
> > > > > 		return 0;
> > > > > 	}
> > > > > ---
> > > > >
> > > > > Unfortunately, it doesn't work correctly...
> > > > > The following code can work correctly. The value of i is still 1 on the device A.
> > > > > What do you think that the following code is acceptable?
> > > > > ---
> > > > > 	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > > 	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > > > > 		msleep(100);
> > > > > 		rcar_gen4_pcie_speed_change(dw);
> > > > > 		if (dw_pcie_link_up(dw)) {
> > > > > 			printk("%s:%d\n", __func__, i);
> > > > > 			return 0;
> > > > > 		}
> > > > > 	}
> > > > > ---
> > > >
> > > > My idea was to implement something like this:
> > > >
> > > > +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > > +{
> > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > +
> > > > +	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > +
> > > > +	rcar_gen4_pcie_speed_change(dw);
> > > > +
> > > > +	return 0;
> > > > +}
> > > >
> > > > and retain the rcar_gen4_pcie_link_up() method as is.
> > >
> > > Unfortunately, such a code doesn't work on my environment...
> > >
> > > > * Note: originally your loop used to have the msleep() call performed
> > > > after the first rcar_gen4_pcie_speed_change() invocation. Thus the
> > > > delay can be dropped if there is only one iteration implemented (see
> > > > further to understand why).
> > >
> > > Calling rcar_gen4_pcie_speed_change() multiple times is required on
> > > my environment. I thought msleep(100) was quite long so that I tried
> > > other wait interval like below:
> > >
> > >  msleep(1) : about 5 loops is needed for link. (about 5 msec.)
> > >  usleep_range(100, 110) : about 400 loops is needed for link. (about 40 msec.)
> > >  usleep_range(500, 600) : about 80 loops is needed for link. (about 40 msec.)
> > >
> > > The delay timing doesn't seems important. Both cases below can work correctly.
> > > --- case 1 ---
> > > 	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > > 		rcar_gen4_pcie_speed_change(dw);
> > > 		if (dw_pcie_link_up(dw)) {
> > > 			printk("%s:%d\n", __func__, i); // will be removed
> > > 			return 0;
> > > 		}
> >
> > > 		msleep(1);
> >
> > Why? Just set it to 5 ms. In anyway please see the next message.
> >
> > > 	}
> > > ---
> > > --- case 2 ---
> > > 	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > > 		rcar_gen4_pcie_speed_change(dw);
> > > 		msleep(1);
> > > 		if (dw_pcie_link_up(dw)) {
> > > 			printk("%s:%d\n", __func__, i); // will be removed
> > > 			return 0;
> > > 		}
> > > 	}
> > > ---
> > >
> > > So, I'll use case 1 for it.
> >
> > Ah. I think I get it now. Your spreadsheet:
> >
> >                 LinkCap max-link-speed  loop
> > Device A           4          4           1
> > Device A           4          3           1
> > Device B           3          3           0
> >
> > actually meant (loop+1) iterations. So in case of Gen4 you'll need
> > three speed changes (one already enabled in the dw_pcie_setup_rc()
> > method and another two ones are performed in your loop). Similarly in
> > case of Gen3 you'll need only one iteration. I bet you won't need to
> > call rcar_gen4_pcie_speed_change() at all if gen2 needs to be trained.
> > Could you try it out?
> >
> > Anyway based on what you discovered and on my experience working with
> > that controller, there should be as many
> > GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag changes as the target speed
> > value, i.e. no flag switch if Gen1 is required, one flag switch if
> > Gen2 is required and so on. Although I failed to find any explicit
> > statement about that in the HW-manual.
> >
> > In addition to the above I've found out that
> > GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE field is actually self cleared when
> > the speed change occurs (see the register description in the HW
> > reference manual). We can use it to implement the
> > dw_pcie_link_up()-independent link training algorithm like this:
> >
> > +#define RCAR_RETRAIN_MAX_CHECK		10
> > +#define RCAR_LINK_SPEED_MAX		4
> > +
> > +static bool rcar_gen4_pcie_speed_change(struct dw_pcie *dw)
> > +{
> > +	u32 val;
> > +	int i;
> > +
> > +	val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
> > +	val &= ~PORT_LOGIC_SPEED_CHANGE;
> > +	dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
> > +
> > +	val |= PORT_LOGIC_SPEED_CHANGE;
> > +	dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
> > +
> > +	for (i = 0; i < RCAR_SPEED_CHANGE_WAIT_RETRIES; i++) {
> > +		val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
> > +		if (!(val & PORT_LOGIC_SPEED_CHANGE))
> > +			return true;
> > +
> > +		msleep(1);
> > +	}
> > +
> > +	return false;
> > +}
> > +
> > +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > +{
> > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > +	int i, changes;
> > +
> > +	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > +
> 
> > +	changes = min_not_zero(dw->link_gen, RCAR_LINK_SPEED_MAX);
> 
> This should have been:
> +changes = min_not_zero(dw->link_gen, RCAR_LINK_SPEED_MAX) - 1;
> because Gen1 doesn't need any speed change action.
> 
> But this part can be further improved. Instead of the code above the
> next snipped can be implemented:
> 
> +changes = min_not_zero(dw->link_gen, RCAR_LINK_SPEED_MAX) - 1;
> +if (changes && rcar->mode == DW_PCIE_RC_TYPE)
> +		changes -= 1;
> 
> It takes into account that the GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE
> flag is already set in the dw_pcie_setup_rc() method. So Gen2 will be
> trained with no need in addition actions. If it's supported of course.

Thank you very much for your comments and suggestion! The suggestion code
works correctly on my environment.

Best regards,
Yoshihiro Shimoda

P.S.
So, I'm investigating endpoint mode issues which you commented now.

> -Serge(y)
> 
> > +	for (i = 0; i < changes; ++i) {
> > +		if (!rcar_gen4_pcie_speed_change(dw))
> > +			break;
> > +	}
> > +
> > +	return 0;
> > +}
> >
> > Note 1. The actual link state will be checked in the framework of the
> > dw_pcie_wait_for_link() function, by means of dw_pcie_link_up().
> >
> > Note 2. RCAR_LINK_SPEED_MAX is deliberately set to 4 because DW PCIe
> > EP core driver doesn't set the PORT_LOGIC_SPEED_CHANGE flag. In case
> > of the DW PCIe Root Port at most 3 iterations should be enough.
> >
> > Note 3. Please use the RCAR_ prefix for the vendor-specific macros.
> > It concerns the entire series.
> >
> > Could you try out the code suggested above?
> >
> > -Serge(y)
> >
> > >
> > > > You don't need to wait for the link to actually get up in the
> > > > start_link() callback because there is the link_up() callback, which
> > > > is called from the dw_pcie_wait_for_link() method during the generic
> > > > DWC PCIe setup procedure. See:
> > >
> > > Since the procedure will call rcar_gen4_pcie_speed_change() from
> > > ->start_link() once, my environment cannot work correctly...
> > >
> > > Best regards,
> > > Yoshihiro Shimoda
> > >
> > > > dw_pcie_host_init():
> > > > +-> ops->host_init()
> > > > +-> ...
> > > > +-> dw_pcie_setup_rc()
> > > > |   +-> ...
> > > > |   +-> dw_pcie_setup()
> > > > |   +-> ...
> > > > +-> if !dw_pcie_link_up()
> > > > |   |   +-> ops->link_up()
> > > > |   +-> dw_pcie_start_link()
> > > > |       +-> ops->start_link()
> > > > +-> dw_pcie_wait_for_link();   // See, wait-procedure is already performed
> > > > |   +-> loop 10 times          // for you in the core driver together
> > > > |       +-> dw_pcie_link_up()  // with the delays between the checks
> > > > |           +-> ops->link_up()
> > > > |       +-> msleep(~100)
> > > > +-> ...
> > > >
> > > > -Serge(y)
> > > >
> > > > >
> > > > > Best regards,
> > > > > Yoshihiro Shimoda
> > > > >
> > > > > > -Serge(y)
> > > > >

Re: [PATCH v16 19/22] PCI: rcar-gen4: Add R-Car Gen4 PCIe Host support

[Serge Semin]

On Tue, Jun 20, 2023 at 12:02:08PM +0000, Yoshihiro Shimoda wrote:
> Hello Serge,
> 
> > From: Serge Semin, Sent: Thursday, June 15, 2023 4:32 AM
> > 
> > On Wed, Jun 14, 2023 at 02:39:29PM +0300, Serge Semin wrote:
> > > On Wed, Jun 14, 2023 at 02:30:13AM +0000, Yoshihiro Shimoda wrote:
> > > > Hello Serge,
> > > >
> > > >  From: Serge Semin, Sent: Tuesday, June 13, 2023 4:52 AM
> > > > >
> > > > > On Mon, Jun 12, 2023 at 01:19:02PM +0000, Yoshihiro Shimoda wrote:
> > > > > > Hello Serge,
> > > > > >
> > > > > > > From: Serge Semin, Sent: Friday, June 9, 2023 7:54 PM
> > > > > > <snip>
> > > > > > > > > > static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > > > > > > > > {
> > > > > > > > > >         struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > > > > > >         int i;
> > > > > > > > > >
> > > > > > > > > >         rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > > > > > > >
> > > > > > > > > >         /*
> > > > > > > > > >          * Require direct speed change here. Otherwise RDLH_LINK_UP of
> > > > > > > > > >          * PCIEINTSTS0 which is this controller specific register may not
> > > > > > > > > >          * be set.
> > > > > > > > > >          */
> > > > > > > > >
> > > > > > > > > >         if (rcar->needs_speed_change) {
> > > > > > > > >
> > > > > > > > > Seeing this is specified for the root port only what about
> > > > > > > > > replacing the statement with just test whether the rcar_gen4_pcie.mode ==
> > > > > > > > > DW_PCIE_RC_TYPE? Thus you'll be ablt to drop the needs_speed_change field.
> > > > > > > >
> > > > > > > > Thank you for the comment. I'll fix it.
> > > > > > > >
> > > > > > > > > BTW Just curious. Why is the loop below enabled for the Root Port
> > > > > > > > > only? What about the end-point controller? It's the same hardware
> > > > > > > > > after all..
> > > > > > > >
> > > > > > > > This is reused from v16 and then it used "link retraining" which is only for
> > > > > > > > the Root Port. As you mentioned, it seems endpoint controller is also needed
> > > > > > > > if we use direct speed change.
> > > > > > > >
> > > > > > > > > >                 for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > > > > > > > > >                         rcar_gen4_pcie_speed_change(dw);
> > > > > > > > > >                         msleep(100);
> > > > > > > > > >                         if (rcar_gen4_pcie_check_current_link(dw))
> > > > > > > > > >                                 return 0;
> > > > > > > > > >                 }
> > > > > > > > >
> > > > > > > > > Did you trace how many iterations this loop normally takes?
> > > > > > > >
> > > > > > > > i = 0 or 1 (if the max-link-speed is suitable for a connected device.)
> > > > > > > >
> > > > > > > > > Is it
> > > > > > > > > constant or varies for the same platform setup and a connected link
> > > > > > > > > partner? Does the number of iterations depend on the target link speed
> > > > > > > > > specified via the "max-link-speed" property?
> > > > > > > >
> > > > > > >
> > > > > > > > This is not related to the "max-link-speed". It seems to related to
> > > > > > > > a link partner.
> > > > > > > > 		LinkCap	max-link-speed	loop
> > > > > > > > Device A		4	4		1
> > > > > > > > Device A		4	3		1
> > > > > > > > Device B		3	3		0
> > > > > > >
> > > > > > > Great! If so I would have just left a single unconditional
> > > > > > > rcar_gen4_pcie_speed_change() call placed right after the
> > > > > > > rcar_gen4_pcie_ltssm_enable() method with no delays afterwards. These
> > > > > > > methods would have been invoked in the framework of
> > > > > > > dw_pcie_start_link() after which the dw_pcie_wait_for_link() method is
> > > > > > > called with several checks parted with the ~100ms delay. It will make
> > > > > > > sure that at least some link is up with the link state printed to the
> > > > > > > system log. If for some reason the performance degradation happens
> > > > > > > then it will be up to the system administrator to investigate what was
> > > > > > > wrong. Your driver did as much is it could to reach the best link gen.
> > > > > >
> > > > > > IIUC, is your suggestion like the following code?
> > > > > > ---
> > > > > > 	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > > > 	if (!dw_pcie_wait_for_link(dw)) {
> > > > > > 		rcar_gen4_pcie_speed_change(dw);
> > > > > > 		return 0;
> > > > > > 	}
> > > > > > ---
> > > > > >
> > > > > > Unfortunately, it doesn't work correctly...
> > > > > > The following code can work correctly. The value of i is still 1 on the device A.
> > > > > > What do you think that the following code is acceptable?
> > > > > > ---
> > > > > > 	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > > > 	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > > > > > 		msleep(100);
> > > > > > 		rcar_gen4_pcie_speed_change(dw);
> > > > > > 		if (dw_pcie_link_up(dw)) {
> > > > > > 			printk("%s:%d\n", __func__, i);
> > > > > > 			return 0;
> > > > > > 		}
> > > > > > 	}
> > > > > > ---
> > > > >
> > > > > My idea was to implement something like this:
> > > > >
> > > > > +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > > > +{
> > > > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > > > +
> > > > > +	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > > > +
> > > > > +	rcar_gen4_pcie_speed_change(dw);
> > > > > +
> > > > > +	return 0;
> > > > > +}
> > > > >
> > > > > and retain the rcar_gen4_pcie_link_up() method as is.
> > > >
> > > > Unfortunately, such a code doesn't work on my environment...
> > > >
> > > > > * Note: originally your loop used to have the msleep() call performed
> > > > > after the first rcar_gen4_pcie_speed_change() invocation. Thus the
> > > > > delay can be dropped if there is only one iteration implemented (see
> > > > > further to understand why).
> > > >
> > > > Calling rcar_gen4_pcie_speed_change() multiple times is required on
> > > > my environment. I thought msleep(100) was quite long so that I tried
> > > > other wait interval like below:
> > > >
> > > >  msleep(1) : about 5 loops is needed for link. (about 5 msec.)
> > > >  usleep_range(100, 110) : about 400 loops is needed for link. (about 40 msec.)
> > > >  usleep_range(500, 600) : about 80 loops is needed for link. (about 40 msec.)
> > > >
> > > > The delay timing doesn't seems important. Both cases below can work correctly.
> > > > --- case 1 ---
> > > > 	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > > > 		rcar_gen4_pcie_speed_change(dw);
> > > > 		if (dw_pcie_link_up(dw)) {
> > > > 			printk("%s:%d\n", __func__, i); // will be removed
> > > > 			return 0;
> > > > 		}
> > >
> > > > 		msleep(1);
> > >
> > > Why? Just set it to 5 ms. In anyway please see the next message.
> > >
> > > > 	}
> > > > ---
> > > > --- case 2 ---
> > > > 	for (i = 0; i < SPEED_CHANGE_MAX_RETRIES; i++) {
> > > > 		rcar_gen4_pcie_speed_change(dw);
> > > > 		msleep(1);
> > > > 		if (dw_pcie_link_up(dw)) {
> > > > 			printk("%s:%d\n", __func__, i); // will be removed
> > > > 			return 0;
> > > > 		}
> > > > 	}
> > > > ---
> > > >
> > > > So, I'll use case 1 for it.
> > >
> > > Ah. I think I get it now. Your spreadsheet:
> > >
> > >                 LinkCap max-link-speed  loop
> > > Device A           4          4           1
> > > Device A           4          3           1
> > > Device B           3          3           0
> > >
> > > actually meant (loop+1) iterations. So in case of Gen4 you'll need
> > > three speed changes (one already enabled in the dw_pcie_setup_rc()
> > > method and another two ones are performed in your loop). Similarly in
> > > case of Gen3 you'll need only one iteration. I bet you won't need to
> > > call rcar_gen4_pcie_speed_change() at all if gen2 needs to be trained.
> > > Could you try it out?
> > >
> > > Anyway based on what you discovered and on my experience working with
> > > that controller, there should be as many
> > > GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE flag changes as the target speed
> > > value, i.e. no flag switch if Gen1 is required, one flag switch if
> > > Gen2 is required and so on. Although I failed to find any explicit
> > > statement about that in the HW-manual.
> > >
> > > In addition to the above I've found out that
> > > GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE field is actually self cleared when
> > > the speed change occurs (see the register description in the HW
> > > reference manual). We can use it to implement the
> > > dw_pcie_link_up()-independent link training algorithm like this:
> > >
> > > +#define RCAR_RETRAIN_MAX_CHECK		10
> > > +#define RCAR_LINK_SPEED_MAX		4
> > > +
> > > +static bool rcar_gen4_pcie_speed_change(struct dw_pcie *dw)
> > > +{
> > > +	u32 val;
> > > +	int i;
> > > +
> > > +	val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
> > > +	val &= ~PORT_LOGIC_SPEED_CHANGE;
> > > +	dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
> > > +
> > > +	val |= PORT_LOGIC_SPEED_CHANGE;
> > > +	dw_pcie_writel_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL, val);
> > > +
> > > +	for (i = 0; i < RCAR_SPEED_CHANGE_WAIT_RETRIES; i++) {
> > > +		val = dw_pcie_readl_dbi(dw, PCIE_LINK_WIDTH_SPEED_CONTROL);
> > > +		if (!(val & PORT_LOGIC_SPEED_CHANGE))
> > > +			return true;
> > > +
> > > +		msleep(1);
> > > +	}
> > > +
> > > +	return false;
> > > +}
> > > +
> > > +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > > +{
> > > +	struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > > +	int i, changes;
> > > +
> > > +	rcar_gen4_pcie_ltssm_enable(rcar, true);
> > > +
> > 
> > > +	changes = min_not_zero(dw->link_gen, RCAR_LINK_SPEED_MAX);
> > 
> > This should have been:
> > +changes = min_not_zero(dw->link_gen, RCAR_LINK_SPEED_MAX) - 1;
> > because Gen1 doesn't need any speed change action.
> > 
> > But this part can be further improved. Instead of the code above the
> > next snipped can be implemented:
> > 
> > +changes = min_not_zero(dw->link_gen, RCAR_LINK_SPEED_MAX) - 1;
> > +if (changes && rcar->mode == DW_PCIE_RC_TYPE)
> > +		changes -= 1;
> > 
> > It takes into account that the GEN2_CTRL_OFF.DIRECT_SPEED_CHANGE
> > flag is already set in the dw_pcie_setup_rc() method. So Gen2 will be
> > trained with no need in addition actions. If it's supported of course.
> 

> Thank you very much for your comments and suggestion! The suggestion code
> works correctly on my environment.

Awesome! Glad we've finally settled this.

-Serge(y)

> 
> Best regards,
> Yoshihiro Shimoda
> 
> P.S.
> So, I'm investigating endpoint mode issues which you commented now.
> 
> > -Serge(y)
> > 
> > > +	for (i = 0; i < changes; ++i) {
> > > +		if (!rcar_gen4_pcie_speed_change(dw))
> > > +			break;
> > > +	}
> > > +
> > > +	return 0;
> > > +}
> > >
> > > Note 1. The actual link state will be checked in the framework of the
> > > dw_pcie_wait_for_link() function, by means of dw_pcie_link_up().
> > >
> > > Note 2. RCAR_LINK_SPEED_MAX is deliberately set to 4 because DW PCIe
> > > EP core driver doesn't set the PORT_LOGIC_SPEED_CHANGE flag. In case
> > > of the DW PCIe Root Port at most 3 iterations should be enough.
> > >
> > > Note 3. Please use the RCAR_ prefix for the vendor-specific macros.
> > > It concerns the entire series.
> > >
> > > Could you try out the code suggested above?
> > >
> > > -Serge(y)
> > >
> > > >
> > > > > You don't need to wait for the link to actually get up in the
> > > > > start_link() callback because there is the link_up() callback, which
> > > > > is called from the dw_pcie_wait_for_link() method during the generic
> > > > > DWC PCIe setup procedure. See:
> > > >
> > > > Since the procedure will call rcar_gen4_pcie_speed_change() from
> > > > ->start_link() once, my environment cannot work correctly...
> > > >
> > > > Best regards,
> > > > Yoshihiro Shimoda
> > > >
> > > > > dw_pcie_host_init():
> > > > > +-> ops->host_init()
> > > > > +-> ...
> > > > > +-> dw_pcie_setup_rc()
> > > > > |   +-> ...
> > > > > |   +-> dw_pcie_setup()
> > > > > |   +-> ...
> > > > > +-> if !dw_pcie_link_up()
> > > > > |   |   +-> ops->link_up()
> > > > > |   +-> dw_pcie_start_link()
> > > > > |       +-> ops->start_link()
> > > > > +-> dw_pcie_wait_for_link();   // See, wait-procedure is already performed
> > > > > |   +-> loop 10 times          // for you in the core driver together
> > > > > |       +-> dw_pcie_link_up()  // with the delays between the checks
> > > > > |           +-> ops->link_up()
> > > > > |       +-> msleep(~100)
> > > > > +-> ...
> > > > >
> > > > > -Serge(y)
> > > > >
> > > > > >
> > > > > > Best regards,
> > > > > > Yoshihiro Shimoda
> > > > > >
> > > > > > > -Serge(y)
> > > > > >

RE: [PATCH v16 06/22] PCI: dwc: Change arguments of dw_pcie_prog_outbound_atu()

[Yoshihiro Shimoda]

Hi Serge,

> From: Serge Semin, Sent: Monday, June 5, 2023 8:56 AM
> 
> On Wed, May 10, 2023 at 03:22:18PM +0900, Yoshihiro Shimoda wrote:
> > To add more arguments to the dw_pcie_prog_outbound_atu() in
> > the future, introduce struct dw_pcie_ob_atu_cfg and change
> > the argument. And, drop dw_pcie_prog_ep_outbound_atu(). No behavior
> > changes.
> 
> I doubt anyone not being aware of the change context will understand
> your message. More details would help with that: why the conversion
> was necessary, how come the dw_pcie_prog_ep_outbound_atu() function
> turns to be redundant, what additional parameters will be added
> afterwards so this patch turns to be a preparation patch for that, etc.

I got it. I'll add such explanations.

> Other than that the change looks good.

Thank you for your review!

Best regards,
Yoshihiro Shimoda

> -Serge(y)
> 
> >
> > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > ---
> >  .../pci/controller/dwc/pcie-designware-ep.c   | 21 +++++---
> >  .../pci/controller/dwc/pcie-designware-host.c | 52 +++++++++++++------
> >  drivers/pci/controller/dwc/pcie-designware.c  | 49 ++++++-----------
> >  drivers/pci/controller/dwc/pcie-designware.h  | 15 ++++--
> >  4 files changed, 77 insertions(+), 60 deletions(-)
> >
> > diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > index 27278010ecec..fe2e0d765be9 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> > @@ -182,9 +182,8 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no, int type,
> >  	return 0;
> >  }
> >
> > -static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, u8 func_no,
> > -				   phys_addr_t phys_addr,
> > -				   u64 pci_addr, size_t size)
> > +static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep,
> > +				   struct dw_pcie_ob_atu_cfg *atu)
> >  {
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> >  	u32 free_win;
> > @@ -196,13 +195,13 @@ static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, u8 func_no,
> >  		return -EINVAL;
> >  	}
> >
> > -	ret = dw_pcie_prog_ep_outbound_atu(pci, func_no, free_win, PCIE_ATU_TYPE_MEM,
> > -					   phys_addr, pci_addr, size);
> > +	atu->index = free_win;
> > +	ret = dw_pcie_prog_outbound_atu(pci, atu);
> >  	if (ret)
> >  		return ret;
> >
> >  	set_bit(free_win, ep->ob_window_map);
> > -	ep->outbound_addr[free_win] = phys_addr;
> > +	ep->outbound_addr[free_win] = atu->cpu_addr;
> >
> >  	return 0;
> >  }
> > @@ -305,8 +304,14 @@ static int dw_pcie_ep_map_addr(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
> >  	int ret;
> >  	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
> >  	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
> > -
> > -	ret = dw_pcie_ep_outbound_atu(ep, func_no, addr, pci_addr, size);
> > +	struct dw_pcie_ob_atu_cfg atu = { 0 };
> > +
> > +	atu.func_no = func_no;
> > +	atu.type = PCIE_ATU_TYPE_MEM;
> > +	atu.cpu_addr = addr;
> > +	atu.pci_addr = pci_addr;
> > +	atu.size = size;
> > +	ret = dw_pcie_ep_outbound_atu(ep, &atu);
> >  	if (ret) {
> >  		dev_err(pci->dev, "Failed to enable address\n");
> >  		return ret;
> > diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
> > index 5718b4bb67f0..676216d540fe 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> > @@ -544,6 +544,7 @@ static void __iomem *dw_pcie_other_conf_map_bus(struct pci_bus *bus,
> >  {
> >  	struct dw_pcie_rp *pp = bus->sysdata;
> >  	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> > +	struct dw_pcie_ob_atu_cfg atu = { 0 };
> >  	int type, ret;
> >  	u32 busdev;
> >
> > @@ -566,8 +567,12 @@ static void __iomem *dw_pcie_other_conf_map_bus(struct pci_bus *bus,
> >  	else
> >  		type = PCIE_ATU_TYPE_CFG1;
> >
> > -	ret = dw_pcie_prog_outbound_atu(pci, 0, type, pp->cfg0_base, busdev,
> > -					pp->cfg0_size);
> > +	atu.type = type;
> > +	atu.cpu_addr = pp->cfg0_base;
> > +	atu.pci_addr = busdev;
> > +	atu.size = pp->cfg0_size;
> > +
> > +	ret = dw_pcie_prog_outbound_atu(pci, &atu);
> >  	if (ret)
> >  		return NULL;
> >
> > @@ -579,6 +584,7 @@ static int dw_pcie_rd_other_conf(struct pci_bus *bus, unsigned int devfn,
> >  {
> >  	struct dw_pcie_rp *pp = bus->sysdata;
> >  	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> > +	struct dw_pcie_ob_atu_cfg atu = { 0 };
> >  	int ret;
> >
> >  	ret = pci_generic_config_read(bus, devfn, where, size, val);
> > @@ -586,9 +592,12 @@ static int dw_pcie_rd_other_conf(struct pci_bus *bus, unsigned int devfn,
> >  		return ret;
> >
> >  	if (pp->cfg0_io_shared) {
> > -		ret = dw_pcie_prog_outbound_atu(pci, 0, PCIE_ATU_TYPE_IO,
> > -						pp->io_base, pp->io_bus_addr,
> > -						pp->io_size);
> > +		atu.type = PCIE_ATU_TYPE_IO;
> > +		atu.cpu_addr = pp->io_base;
> > +		atu.pci_addr = pp->io_bus_addr;
> > +		atu.size = pp->io_size;
> > +
> > +		ret = dw_pcie_prog_outbound_atu(pci, &atu);
> >  		if (ret)
> >  			return PCIBIOS_SET_FAILED;
> >  	}
> > @@ -601,6 +610,7 @@ static int dw_pcie_wr_other_conf(struct pci_bus *bus, unsigned int devfn,
> >  {
> >  	struct dw_pcie_rp *pp = bus->sysdata;
> >  	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> > +	struct dw_pcie_ob_atu_cfg atu = { 0 };
> >  	int ret;
> >
> >  	ret = pci_generic_config_write(bus, devfn, where, size, val);
> > @@ -608,9 +618,12 @@ static int dw_pcie_wr_other_conf(struct pci_bus *bus, unsigned int devfn,
> >  		return ret;
> >
> >  	if (pp->cfg0_io_shared) {
> > -		ret = dw_pcie_prog_outbound_atu(pci, 0, PCIE_ATU_TYPE_IO,
> > -						pp->io_base, pp->io_bus_addr,
> > -						pp->io_size);
> > +		atu.type = PCIE_ATU_TYPE_IO;
> > +		atu.cpu_addr = pp->io_base;
> > +		atu.pci_addr = pp->io_bus_addr;
> > +		atu.size = pp->io_size;
> > +
> > +		ret = dw_pcie_prog_outbound_atu(pci, &atu);
> >  		if (ret)
> >  			return PCIBIOS_SET_FAILED;
> >  	}
> > @@ -645,6 +658,7 @@ static struct pci_ops dw_pcie_ops = {
> >  static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
> >  {
> >  	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> > +	struct dw_pcie_ob_atu_cfg atu = { 0 };
> >  	struct resource_entry *entry;
> >  	int i, ret;
> >
> > @@ -672,10 +686,13 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
> >  		if (pci->num_ob_windows <= ++i)
> >  			break;
> >
> > -		ret = dw_pcie_prog_outbound_atu(pci, i, PCIE_ATU_TYPE_MEM,
> > -						entry->res->start,
> > -						entry->res->start - entry->offset,
> > -						resource_size(entry->res));
> > +		atu.index = i;
> > +		atu.type = PCIE_ATU_TYPE_MEM;
> > +		atu.cpu_addr = entry->res->start;
> > +		atu.pci_addr = entry->res->start - entry->offset;
> > +		atu.size = resource_size(entry->res);
> > +
> > +		ret = dw_pcie_prog_outbound_atu(pci, &atu);
> >  		if (ret) {
> >  			dev_err(pci->dev, "Failed to set MEM range %pr\n",
> >  				entry->res);
> > @@ -685,10 +702,13 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
> >
> >  	if (pp->io_size) {
> >  		if (pci->num_ob_windows > ++i) {
> > -			ret = dw_pcie_prog_outbound_atu(pci, i, PCIE_ATU_TYPE_IO,
> > -							pp->io_base,
> > -							pp->io_bus_addr,
> > -							pp->io_size);
> > +			atu.index = i;
> > +			atu.type = PCIE_ATU_TYPE_IO;
> > +			atu.cpu_addr = pp->io_base;
> > +			atu.pci_addr = pp->io_bus_addr;
> > +			atu.size = pp->io_size;
> > +
> > +			ret = dw_pcie_prog_outbound_atu(pci, &atu);
> >  			if (ret) {
> >  				dev_err(pci->dev, "Failed to set IO range %pr\n",
> >  					entry->res);
> > diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
> > index ede166645289..bd85a73d354b 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware.c
> > @@ -464,56 +464,56 @@ static inline u32 dw_pcie_enable_ecrc(u32 val)
> >  	return val | PCIE_ATU_TD;
> >  }
> >
> > -static int __dw_pcie_prog_outbound_atu(struct dw_pcie *pci, u8 func_no,
> > -				       int index, int type, u64 cpu_addr,
> > -				       u64 pci_addr, u64 size)
> > +int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
> > +			      const struct dw_pcie_ob_atu_cfg *atu)
> >  {
> > +	u64 cpu_addr = atu->cpu_addr;
> >  	u32 retries, val;
> >  	u64 limit_addr;
> >
> >  	if (pci->ops && pci->ops->cpu_addr_fixup)
> >  		cpu_addr = pci->ops->cpu_addr_fixup(pci, cpu_addr);
> >
> > -	limit_addr = cpu_addr + size - 1;
> > +	limit_addr = cpu_addr + atu->size - 1;
> >
> >  	if ((limit_addr & ~pci->region_limit) != (cpu_addr & ~pci->region_limit) ||
> >  	    !IS_ALIGNED(cpu_addr, pci->region_align) ||
> > -	    !IS_ALIGNED(pci_addr, pci->region_align) || !size) {
> > +	    !IS_ALIGNED(atu->pci_addr, pci->region_align) || !atu->size) {
> >  		return -EINVAL;
> >  	}
> >
> > -	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_LOWER_BASE,
> > +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_LOWER_BASE,
> >  			      lower_32_bits(cpu_addr));
> > -	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_UPPER_BASE,
> > +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_BASE,
> >  			      upper_32_bits(cpu_addr));
> >
> > -	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_LIMIT,
> > +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_LIMIT,
> >  			      lower_32_bits(limit_addr));
> >  	if (dw_pcie_ver_is_ge(pci, 460A))
> > -		dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_UPPER_LIMIT,
> > +		dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_LIMIT,
> >  				      upper_32_bits(limit_addr));
> >
> > -	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_LOWER_TARGET,
> > -			      lower_32_bits(pci_addr));
> > -	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_UPPER_TARGET,
> > -			      upper_32_bits(pci_addr));
> > +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_LOWER_TARGET,
> > +			      lower_32_bits(atu->pci_addr));
> > +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_TARGET,
> > +			      upper_32_bits(atu->pci_addr));
> >
> > -	val = type | PCIE_ATU_FUNC_NUM(func_no);
> > +	val = atu->type | PCIE_ATU_FUNC_NUM(atu->func_no);
> >  	if (upper_32_bits(limit_addr) > upper_32_bits(cpu_addr) &&
> >  	    dw_pcie_ver_is_ge(pci, 460A))
> >  		val |= PCIE_ATU_INCREASE_REGION_SIZE;
> >  	if (dw_pcie_ver_is(pci, 490A))
> >  		val = dw_pcie_enable_ecrc(val);
> > -	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_REGION_CTRL1, val);
> > +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL1, val);
> >
> > -	dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_REGION_CTRL2, PCIE_ATU_ENABLE);
> > +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2, PCIE_ATU_ENABLE);
> >
> >  	/*
> >  	 * Make sure ATU enable takes effect before any subsequent config
> >  	 * and I/O accesses.
> >  	 */
> >  	for (retries = 0; retries < LINK_WAIT_MAX_IATU_RETRIES; retries++) {
> > -		val = dw_pcie_readl_atu_ob(pci, index, PCIE_ATU_REGION_CTRL2);
> > +		val = dw_pcie_readl_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2);
> >  		if (val & PCIE_ATU_ENABLE)
> >  			return 0;
> >
> > @@ -525,21 +525,6 @@ static int __dw_pcie_prog_outbound_atu(struct dw_pcie *pci, u8 func_no,
> >  	return -ETIMEDOUT;
> >  }
> >
> > -int dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
> > -			      u64 cpu_addr, u64 pci_addr, u64 size)
> > -{
> > -	return __dw_pcie_prog_outbound_atu(pci, 0, index, type,
> > -					   cpu_addr, pci_addr, size);
> > -}
> > -
> > -int dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> > -				 int type, u64 cpu_addr, u64 pci_addr,
> > -				 u64 size)
> > -{
> > -	return __dw_pcie_prog_outbound_atu(pci, func_no, index, type,
> > -					   cpu_addr, pci_addr, size);
> > -}
> > -
> >  static inline u32 dw_pcie_readl_atu_ib(struct dw_pcie *pci, u32 index, u32 reg)
> >  {
> >  	return dw_pcie_readl_atu(pci, PCIE_ATU_REGION_DIR_IB, index, reg);
> > diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
> > index 9acf6c40d252..b8fa099bbed3 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware.h
> > +++ b/drivers/pci/controller/dwc/pcie-designware.h
> > @@ -291,6 +291,15 @@ enum dw_pcie_core_rst {
> >  	DW_PCIE_NUM_CORE_RSTS
> >  };
> >
> > +struct dw_pcie_ob_atu_cfg {
> > +	int index;
> > +	int type;
> > +	u8 func_no;
> > +	u64 cpu_addr;
> > +	u64 pci_addr;
> > +	u64 size;
> > +};
> > +
> >  struct dw_pcie_host_ops {
> >  	int (*host_init)(struct dw_pcie_rp *pp);
> >  	void (*host_deinit)(struct dw_pcie_rp *pp);
> > @@ -419,10 +428,8 @@ void dw_pcie_write_dbi2(struct dw_pcie *pci, u32 reg, size_t size, u32 val);
> >  int dw_pcie_link_up(struct dw_pcie *pci);
> >  void dw_pcie_upconfig_setup(struct dw_pcie *pci);
> >  int dw_pcie_wait_for_link(struct dw_pcie *pci);
> > -int dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
> > -			      u64 cpu_addr, u64 pci_addr, u64 size);
> > -int dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> > -				 int type, u64 cpu_addr, u64 pci_addr, u64 size);
> > +int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
> > +			      const struct dw_pcie_ob_atu_cfg *atu);
> >  int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int type,
> >  			     u64 cpu_addr, u64 pci_addr, u64 size);
> >  int dw_pcie_prog_ep_inbound_atu(struct dw_pcie *pci, u8 func_no, int index,
> > --
> > 2.25.1
> >

RE: [PATCH v16 07/22] PCI: dwc: Add outbound MSG TLPs support

[Yoshihiro Shimoda]

Hello Serge,

> From: Serge Semin, Sent: Monday, June 5, 2023 9:16 AM
> 
> On Wed, May 10, 2023 at 03:22:19PM +0900, Yoshihiro Shimoda wrote:
> > Add "code" and "routing" into struct dw_pcie_outbound_atu for
> 
> structure name has been changed.

Oops. I'll fix it.

> > sending MSG by iATU in the PCIe endpoint mode in near the future.
> > PCIE_ATU_INHIBIT_PAYLOAD is set to issue TLP type of Msg instead of
> > MsgD.
> 
> So your implementation implies the data-less messages only. This note
> should have been added at least to the commit log. Ideally it would be
> useful to have an in-situ comment above the code setting these flags.

I got it. I'll add such explanation at both the commit log and code.

> > PCIE_ATU_HEADER_SUB_ENABLE is set to issue the translated
> > TLP header by using PCIE_ATU_{LOW,UPP}ER_TARGET registers' values.
> 
> Why is that needed? Please elaborate in the patch log.

Thank you for your comment. I confirmed that this was not needed. So,
I'll drop this comment and related code.

> Other than that the change looks good.

Thank you for your review!

Best regards,
Yoshihiro Shimoda

> * I'll get back to the series review tomorrow.
> 
> -Serge(y)
> 
> >
> > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
> > ---
> >  drivers/pci/controller/dwc/pcie-designware.c | 7 +++++--
> >  drivers/pci/controller/dwc/pcie-designware.h | 5 +++++
> >  2 files changed, 10 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
> > index bd85a73d354b..a7c724ba7385 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware.c
> > @@ -498,7 +498,7 @@ int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
> >  	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_TARGET,
> >  			      upper_32_bits(atu->pci_addr));
> >
> > -	val = atu->type | PCIE_ATU_FUNC_NUM(atu->func_no);
> > +	val = atu->type | atu->routing | PCIE_ATU_FUNC_NUM(atu->func_no);
> >  	if (upper_32_bits(limit_addr) > upper_32_bits(cpu_addr) &&
> >  	    dw_pcie_ver_is_ge(pci, 460A))
> >  		val |= PCIE_ATU_INCREASE_REGION_SIZE;
> > @@ -506,7 +506,10 @@ int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
> >  		val = dw_pcie_enable_ecrc(val);
> >  	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL1, val);
> >
> > -	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2, PCIE_ATU_ENABLE);
> > +	val = PCIE_ATU_ENABLE;
> > +	if (atu->type == PCIE_ATU_TYPE_MSG)
> > +		val |= PCIE_ATU_INHIBIT_PAYLOAD | PCIE_ATU_HEADER_SUB_ENABLE | atu->code;
> > +	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2, val);
> >
> >  	/*
> >  	 * Make sure ATU enable takes effect before any subsequent config
> > diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
> > index b8fa099bbed3..c83d1d176e62 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware.h
> > +++ b/drivers/pci/controller/dwc/pcie-designware.h
> > @@ -150,11 +150,14 @@
> >  #define PCIE_ATU_TYPE_IO		0x2
> >  #define PCIE_ATU_TYPE_CFG0		0x4
> >  #define PCIE_ATU_TYPE_CFG1		0x5
> > +#define PCIE_ATU_TYPE_MSG		0x10
> >  #define PCIE_ATU_TD			BIT(8)
> >  #define PCIE_ATU_FUNC_NUM(pf)           ((pf) << 20)
> >  #define PCIE_ATU_REGION_CTRL2		0x004
> >  #define PCIE_ATU_ENABLE			BIT(31)
> >  #define PCIE_ATU_BAR_MODE_ENABLE	BIT(30)
> > +#define PCIE_ATU_INHIBIT_PAYLOAD	BIT(22)
> > +#define PCIE_ATU_HEADER_SUB_ENABLE	BIT(21)
> >  #define PCIE_ATU_FUNC_NUM_MATCH_EN      BIT(19)
> >  #define PCIE_ATU_LOWER_BASE		0x008
> >  #define PCIE_ATU_UPPER_BASE		0x00C
> > @@ -295,6 +298,8 @@ struct dw_pcie_ob_atu_cfg {
> >  	int index;
> >  	int type;
> >  	u8 func_no;
> > +	u8 code;
> > +	u8 routing;
> >  	u64 cpu_addr;
> >  	u64 pci_addr;
> >  	u64 size;
> > --
> > 2.25.1
> >