[PATCH v4 0/7] Add support for MHI Endpoint function driver

[PATCH v4 0/7] Add support for MHI Endpoint function driver

[Manivannan Sadhasivam]

Hello,

This series adds support for Modem Host Interface (MHI) Endpoint function
driver and few updates to the PCI endpoint core.

MHI
===

MHI is the communication protocol used by the host machines to control and
communicate with the Qualcomm modems/WLAN devices over any high speed physical
bus like PCIe. In Linux kernel, MHI is modeled as a bus driver [1] and there
are two instances of MHI used in a typical setup.

1. MHI host - MHI implementation for the host machines like x86/ARM64.
2. MHI Endpoint - MHI implementation for the endpoint devices like modems.

MHI EPF
=======

The MHI Endpoint function driver (MHI EPF) is used on the MHI endpoint devices
like modems. The MHI EPF driver sits in between the PCIe EP and MHI EP bus and
carries out all of the PCIe related activities like BAR config, PCIe Event
handling, MMIO read/write etc,... for the MHI EP bus.

Below is the simple representation of the setup:


                 +----------------------------------------------------+
                 |                  Endpoint CPU                      |                   
                 |                                                    |
+------------+   |   +------------+   +-----------+   +-----------+   |
|            |   |   |            |   |           |   |           |   |
|            |   |   |   MHI EP   |   |           |   |           |   | PCIe Bus
|  Modem DSP +---+---+    Bus     +---+  MHI EPF  +---+  PCIe EP  +---+---------
|            |   |   |            |   |           |   |           |   |
|            |   |   |            |   |           |   |           |   |
+------------+   |   +------------+   +-----------+   +-----------+   |
                 |                                                    |
                 |                                                    |
                 +----------------------------------------------------+

The data packets will be read from the Modem DSP by the MHI stack and will be
transmitted to the host machine over PCIe bus with the help of MHI EPF driver.

Test setup
==========

This series has been tested on Snapdragon X55 modem a.k.a SDX55 connected to
the ARM64 host machine.

Thanks,
Mani

[1] https://www.kernel.org/doc/html/latest/mhi/mhi.html

Changes in v4:

* Collected review tag from Kishon
* Changed the IP_SW0 channel numbers as per latest MHI spec

Changes in v3:

* Fixed the probe function of EPF_VNTB driver

Changes in v2:

* Rebased on top of v6.3-rc1
* Switched to the new callback interface for passing events from EPC to EPF
* Dropped one patch related to notifier

Manivannan Sadhasivam (7):
  PCI: endpoint: Pass EPF device ID to the probe function
  PCI: endpoint: Warn and return if EPC is started/stopped multiple
    times
  PCI: endpoint: Add linkdown notifier support
  PCI: endpoint: Add BME notifier support
  PCI: qcom-ep: Add support for Link down notification
  PCI: qcom-ep: Add support for BME notification
  PCI: endpoint: Add PCI Endpoint function driver for MHI bus

 drivers/pci/controller/dwc/pcie-qcom-ep.c     |   2 +
 drivers/pci/endpoint/functions/Kconfig        |  10 +
 drivers/pci/endpoint/functions/Makefile       |   1 +
 drivers/pci/endpoint/functions/pci-epf-mhi.c  | 454 ++++++++++++++++++
 drivers/pci/endpoint/functions/pci-epf-ntb.c  |   3 +-
 drivers/pci/endpoint/functions/pci-epf-test.c |   2 +-
 drivers/pci/endpoint/functions/pci-epf-vntb.c |   2 +-
 drivers/pci/endpoint/pci-ep-cfs.c             |   3 +
 drivers/pci/endpoint/pci-epc-core.c           |  52 ++
 drivers/pci/endpoint/pci-epf-core.c           |   8 +-
 include/linux/pci-epc.h                       |   2 +
 include/linux/pci-epf.h                       |   8 +-
 12 files changed, 540 insertions(+), 7 deletions(-)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-mhi.c

-- 
2.25.1

[PATCH v4 1/7] PCI: endpoint: Pass EPF device ID to the probe function

[Manivannan Sadhasivam]

Currently, the EPF probe function doesn't get the device ID argument needed
to correctly identify the device table ID of the EPF device.

When multiple entries are added to the "struct pci_epf_device_id" table,
the probe function needs to identify the correct one. And the only way to
do so is by storing the correct device ID in "struct pci_epf" during
"pci_epf_match_id()" and passing that to probe().

Reviewed-by: Kishon Vijay Abraham I <kishon@kernel.org>
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
---
 drivers/pci/endpoint/functions/pci-epf-ntb.c  | 3 ++-
 drivers/pci/endpoint/functions/pci-epf-test.c | 2 +-
 drivers/pci/endpoint/functions/pci-epf-vntb.c | 2 +-
 drivers/pci/endpoint/pci-epf-core.c           | 8 +++++---
 include/linux/pci-epf.h                       | 4 +++-
 5 files changed, 12 insertions(+), 7 deletions(-)

diff --git a/drivers/pci/endpoint/functions/pci-epf-ntb.c b/drivers/pci/endpoint/functions/pci-epf-ntb.c
index 9a00448c7e61..980b4ecf19a2 100644
--- a/drivers/pci/endpoint/functions/pci-epf-ntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-ntb.c
@@ -2075,11 +2075,12 @@ static struct config_group *epf_ntb_add_cfs(struct pci_epf *epf,
 /**
  * epf_ntb_probe() - Probe NTB function driver
  * @epf: NTB endpoint function device
+ * @id: NTB endpoint function device ID
  *
  * Probe NTB function driver when endpoint function bus detects a NTB
  * endpoint function.
  */
-static int epf_ntb_probe(struct pci_epf *epf)
+static int epf_ntb_probe(struct pci_epf *epf, const struct pci_epf_device_id *id)
 {
 	struct epf_ntb *ntb;
 	struct device *dev;
diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c b/drivers/pci/endpoint/functions/pci-epf-test.c
index 0f9d2ec822ac..d5fcc78a5b73 100644
--- a/drivers/pci/endpoint/functions/pci-epf-test.c
+++ b/drivers/pci/endpoint/functions/pci-epf-test.c
@@ -980,7 +980,7 @@ static const struct pci_epf_device_id pci_epf_test_ids[] = {
 	{},
 };
 
-static int pci_epf_test_probe(struct pci_epf *epf)
+static int pci_epf_test_probe(struct pci_epf *epf, const struct pci_epf_device_id *id)
 {
 	struct pci_epf_test *epf_test;
 	struct device *dev = &epf->dev;
diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
index b7c7a8af99f4..122eb7a12028 100644
--- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
@@ -1401,7 +1401,7 @@ static struct pci_epf_ops epf_ntb_ops = {
  *
  * Returns: Zero for success, or an error code in case of failure
  */
-static int epf_ntb_probe(struct pci_epf *epf)
+static int epf_ntb_probe(struct pci_epf *epf, const struct pci_epf_device_id *id)
 {
 	struct epf_ntb *ntb;
 	struct device *dev;
diff --git a/drivers/pci/endpoint/pci-epf-core.c b/drivers/pci/endpoint/pci-epf-core.c
index 2036e38be093..924564288c9a 100644
--- a/drivers/pci/endpoint/pci-epf-core.c
+++ b/drivers/pci/endpoint/pci-epf-core.c
@@ -494,11 +494,13 @@ static const struct device_type pci_epf_type = {
 };
 
 static int
-pci_epf_match_id(const struct pci_epf_device_id *id, const struct pci_epf *epf)
+pci_epf_match_id(const struct pci_epf_device_id *id, struct pci_epf *epf)
 {
 	while (id->name[0]) {
-		if (strcmp(epf->name, id->name) == 0)
+		if (strcmp(epf->name, id->name) == 0) {
+			epf->id = id;
 			return true;
+		}
 		id++;
 	}
 
@@ -526,7 +528,7 @@ static int pci_epf_device_probe(struct device *dev)
 
 	epf->driver = driver;
 
-	return driver->probe(epf);
+	return driver->probe(epf, epf->id);
 }
 
 static void pci_epf_device_remove(struct device *dev)
diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
index a215dc8ce693..bc613f0df7e3 100644
--- a/include/linux/pci-epf.h
+++ b/include/linux/pci-epf.h
@@ -89,7 +89,7 @@ struct pci_epc_event_ops {
  * @id_table: identifies EPF devices for probing
  */
 struct pci_epf_driver {
-	int	(*probe)(struct pci_epf *epf);
+	int	(*probe)(struct pci_epf *epf, const struct pci_epf_device_id *id);
 	void	(*remove)(struct pci_epf *epf);
 
 	struct device_driver	driver;
@@ -131,6 +131,7 @@ struct pci_epf_bar {
  * @epc: the EPC device to which this EPF device is bound
  * @epf_pf: the physical EPF device to which this virtual EPF device is bound
  * @driver: the EPF driver to which this EPF device is bound
+ * @id: Pointer to the EPF device ID
  * @list: to add pci_epf as a list of PCI endpoint functions to pci_epc
  * @lock: mutex to protect pci_epf_ops
  * @sec_epc: the secondary EPC device to which this EPF device is bound
@@ -158,6 +159,7 @@ struct pci_epf {
 	struct pci_epc		*epc;
 	struct pci_epf		*epf_pf;
 	struct pci_epf_driver	*driver;
+	const struct pci_epf_device_id *id;
 	struct list_head	list;
 	/* mutex to protect against concurrent access of pci_epf_ops */
 	struct mutex		lock;
-- 
2.25.1

[PATCH v4 2/7] PCI: endpoint: Warn and return if EPC is started/stopped multiple times

[Manivannan Sadhasivam]

When the EPC is started or stopped multiple times from configfs, just emit
a once time warning and return. There is no need to call the EPC start/stop
functions in those cases.

Reviewed-by: Kishon Vijay Abraham I <kishon@kernel.org>
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
---
 drivers/pci/endpoint/pci-ep-cfs.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/pci/endpoint/pci-ep-cfs.c b/drivers/pci/endpoint/pci-ep-cfs.c
index 4b8ac0ac84d5..62c8e09c59f4 100644
--- a/drivers/pci/endpoint/pci-ep-cfs.c
+++ b/drivers/pci/endpoint/pci-ep-cfs.c
@@ -178,6 +178,9 @@ static ssize_t pci_epc_start_store(struct config_item *item, const char *page,
 	if (kstrtobool(page, &start) < 0)
 		return -EINVAL;
 
+	if (WARN_ON_ONCE(start == epc_group->start))
+		return 0;
+
 	if (!start) {
 		pci_epc_stop(epc);
 		epc_group->start = 0;
-- 
2.25.1

[PATCH v4 3/7] PCI: endpoint: Add linkdown notifier support

[Manivannan Sadhasivam]

Add support to notify the EPF device about the linkdown event from the
EPC device.

Reviewed-by: Kishon Vijay Abraham I <kishon@kernel.org>
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
---
 drivers/pci/endpoint/pci-epc-core.c | 26 ++++++++++++++++++++++++++
 include/linux/pci-epc.h             |  1 +
 include/linux/pci-epf.h             |  2 ++
 3 files changed, 29 insertions(+)

diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
index 46c9a5c3ca14..1ecbe2b1d3bd 100644
--- a/drivers/pci/endpoint/pci-epc-core.c
+++ b/drivers/pci/endpoint/pci-epc-core.c
@@ -706,6 +706,32 @@ void pci_epc_linkup(struct pci_epc *epc)
 }
 EXPORT_SYMBOL_GPL(pci_epc_linkup);
 
+/**
+ * pci_epc_linkdown() - Notify the EPF device that EPC device has dropped the
+ *			connection with the Root Complex.
+ * @epc: the EPC device which has dropped the link with the host
+ *
+ * Invoke to Notify the EPF device that the EPC device has dropped the
+ * connection with the Root Complex.
+ */
+void pci_epc_linkdown(struct pci_epc *epc)
+{
+	struct pci_epf *epf;
+
+	if (!epc || IS_ERR(epc))
+		return;
+
+	mutex_lock(&epc->list_lock);
+	list_for_each_entry(epf, &epc->pci_epf, list) {
+		mutex_lock(&epf->lock);
+		if (epf->event_ops && epf->event_ops->link_down)
+			epf->event_ops->link_down(epf);
+		mutex_unlock(&epf->lock);
+	}
+	mutex_unlock(&epc->list_lock);
+}
+EXPORT_SYMBOL_GPL(pci_epc_linkdown);
+
 /**
  * pci_epc_init_notify() - Notify the EPF device that EPC device's core
  *			   initialization is completed.
diff --git a/include/linux/pci-epc.h b/include/linux/pci-epc.h
index 301bb0e53707..63a6cc5e5282 100644
--- a/include/linux/pci-epc.h
+++ b/include/linux/pci-epc.h
@@ -203,6 +203,7 @@ void pci_epc_destroy(struct pci_epc *epc);
 int pci_epc_add_epf(struct pci_epc *epc, struct pci_epf *epf,
 		    enum pci_epc_interface_type type);
 void pci_epc_linkup(struct pci_epc *epc);
+void pci_epc_linkdown(struct pci_epc *epc);
 void pci_epc_init_notify(struct pci_epc *epc);
 void pci_epc_remove_epf(struct pci_epc *epc, struct pci_epf *epf,
 			enum pci_epc_interface_type type);
diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
index bc613f0df7e3..f8e5a63d0c83 100644
--- a/include/linux/pci-epf.h
+++ b/include/linux/pci-epf.h
@@ -71,10 +71,12 @@ struct pci_epf_ops {
  * struct pci_epf_event_ops - Callbacks for capturing the EPC events
  * @core_init: Callback for the EPC initialization complete event
  * @link_up: Callback for the EPC link up event
+ * @link_down: Callback for the EPC link down event
  */
 struct pci_epc_event_ops {
 	int (*core_init)(struct pci_epf *epf);
 	int (*link_up)(struct pci_epf *epf);
+	int (*link_down)(struct pci_epf *epf);
 };
 
 /**
-- 
2.25.1

[PATCH v4 4/7] PCI: endpoint: Add BME notifier support

[Manivannan Sadhasivam]

Add support to notify the EPF device about the Bus Master Enable (BME)
event received by the EPC device from the Root complex.

Reviewed-by: Kishon Vijay Abraham I <kishon@kernel.org>
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
---
 drivers/pci/endpoint/pci-epc-core.c | 26 ++++++++++++++++++++++++++
 include/linux/pci-epc.h             |  1 +
 include/linux/pci-epf.h             |  2 ++
 3 files changed, 29 insertions(+)

diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
index 1ecbe2b1d3bd..ca8f838fa51f 100644
--- a/drivers/pci/endpoint/pci-epc-core.c
+++ b/drivers/pci/endpoint/pci-epc-core.c
@@ -758,6 +758,32 @@ void pci_epc_init_notify(struct pci_epc *epc)
 }
 EXPORT_SYMBOL_GPL(pci_epc_init_notify);
 
+/**
+ * pci_epc_bme_notify() - Notify the EPF device that the EPC device has received
+ *			  the BME event from the Root complex
+ * @epc: the EPC device that received the BME event
+ *
+ * Invoke to Notify the EPF device that the EPC device has received the Bus
+ * Master Enable (BME) event from the Root complex
+ */
+void pci_epc_bme_notify(struct pci_epc *epc)
+{
+	struct pci_epf *epf;
+
+	if (!epc || IS_ERR(epc))
+		return;
+
+	mutex_lock(&epc->list_lock);
+	list_for_each_entry(epf, &epc->pci_epf, list) {
+		mutex_lock(&epf->lock);
+		if (epf->event_ops && epf->event_ops->bme)
+			epf->event_ops->bme(epf);
+		mutex_unlock(&epf->lock);
+	}
+	mutex_unlock(&epc->list_lock);
+}
+EXPORT_SYMBOL_GPL(pci_epc_bme_notify);
+
 /**
  * pci_epc_destroy() - destroy the EPC device
  * @epc: the EPC device that has to be destroyed
diff --git a/include/linux/pci-epc.h b/include/linux/pci-epc.h
index 63a6cc5e5282..5cb694031072 100644
--- a/include/linux/pci-epc.h
+++ b/include/linux/pci-epc.h
@@ -205,6 +205,7 @@ int pci_epc_add_epf(struct pci_epc *epc, struct pci_epf *epf,
 void pci_epc_linkup(struct pci_epc *epc);
 void pci_epc_linkdown(struct pci_epc *epc);
 void pci_epc_init_notify(struct pci_epc *epc);
+void pci_epc_bme_notify(struct pci_epc *epc);
 void pci_epc_remove_epf(struct pci_epc *epc, struct pci_epf *epf,
 			enum pci_epc_interface_type type);
 int pci_epc_write_header(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
index f8e5a63d0c83..f34b3b32a0e7 100644
--- a/include/linux/pci-epf.h
+++ b/include/linux/pci-epf.h
@@ -72,11 +72,13 @@ struct pci_epf_ops {
  * @core_init: Callback for the EPC initialization complete event
  * @link_up: Callback for the EPC link up event
  * @link_down: Callback for the EPC link down event
+ * @bme: Callback for the EPC BME (Bus Master Enable) event
  */
 struct pci_epc_event_ops {
 	int (*core_init)(struct pci_epf *epf);
 	int (*link_up)(struct pci_epf *epf);
 	int (*link_down)(struct pci_epf *epf);
+	int (*bme)(struct pci_epf *epf);
 };
 
 /**
-- 
2.25.1

[PATCH v4 5/7] PCI: qcom-ep: Add support for Link down notification

[Manivannan Sadhasivam]

Add support to pass Link down notification to Endpoint function driver
so that the LINK_DOWN event can be processed by the function.

Reviewed-by: Kishon Vijay Abraham I <kishon@kernel.org>
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
---
 drivers/pci/controller/dwc/pcie-qcom-ep.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/pci/controller/dwc/pcie-qcom-ep.c b/drivers/pci/controller/dwc/pcie-qcom-ep.c
index 19b32839ea26..4ce01ff7527c 100644
--- a/drivers/pci/controller/dwc/pcie-qcom-ep.c
+++ b/drivers/pci/controller/dwc/pcie-qcom-ep.c
@@ -569,6 +569,7 @@ static irqreturn_t qcom_pcie_ep_global_irq_thread(int irq, void *data)
 	if (FIELD_GET(PARF_INT_ALL_LINK_DOWN, status)) {
 		dev_dbg(dev, "Received Linkdown event\n");
 		pcie_ep->link_status = QCOM_PCIE_EP_LINK_DOWN;
+		pci_epc_linkdown(pci->ep.epc);
 	} else if (FIELD_GET(PARF_INT_ALL_BME, status)) {
 		dev_dbg(dev, "Received BME event. Link is enabled!\n");
 		pcie_ep->link_status = QCOM_PCIE_EP_LINK_ENABLED;
-- 
2.25.1

[PATCH v4 6/7] PCI: qcom-ep: Add support for BME notification

[Manivannan Sadhasivam]

Add support to pass BME (Bus Master Enable) notification to Endpoint
function driver so that the BME event can be processed by the function.

Reviewed-by: Kishon Vijay Abraham I <kishon@kernel.org>
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
---
 drivers/pci/controller/dwc/pcie-qcom-ep.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/pci/controller/dwc/pcie-qcom-ep.c b/drivers/pci/controller/dwc/pcie-qcom-ep.c
index 4ce01ff7527c..1435f516d3f7 100644
--- a/drivers/pci/controller/dwc/pcie-qcom-ep.c
+++ b/drivers/pci/controller/dwc/pcie-qcom-ep.c
@@ -573,6 +573,7 @@ static irqreturn_t qcom_pcie_ep_global_irq_thread(int irq, void *data)
 	} else if (FIELD_GET(PARF_INT_ALL_BME, status)) {
 		dev_dbg(dev, "Received BME event. Link is enabled!\n");
 		pcie_ep->link_status = QCOM_PCIE_EP_LINK_ENABLED;
+		pci_epc_bme_notify(pci->ep.epc);
 	} else if (FIELD_GET(PARF_INT_ALL_PM_TURNOFF, status)) {
 		dev_dbg(dev, "Received PM Turn-off event! Entering L23\n");
 		val = readl_relaxed(pcie_ep->parf + PARF_PM_CTRL);
-- 
2.25.1

[PATCH v4 7/7] PCI: endpoint: Add PCI Endpoint function driver for MHI bus

[Manivannan Sadhasivam]

Add PCI Endpoint driver for the Qualcomm MHI (Modem Host Interface) bus.
The driver implements the MHI function over PCI in the endpoint device
such as SDX55 modem. The MHI endpoint function driver acts as a
controller driver for the MHI Endpoint stack and carries out all PCI
related activities like mapping the host memory using iATU, triggering
MSIs etc...

Reviewed-by: Kishon Vijay Abraham I <kishon@kernel.org>
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
---
 drivers/pci/endpoint/functions/Kconfig       |  10 +
 drivers/pci/endpoint/functions/Makefile      |   1 +
 drivers/pci/endpoint/functions/pci-epf-mhi.c | 454 +++++++++++++++++++
 3 files changed, 465 insertions(+)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-mhi.c

diff --git a/drivers/pci/endpoint/functions/Kconfig b/drivers/pci/endpoint/functions/Kconfig
index 9fd560886871..f5171b4fabbe 100644
--- a/drivers/pci/endpoint/functions/Kconfig
+++ b/drivers/pci/endpoint/functions/Kconfig
@@ -37,3 +37,13 @@ config PCI_EPF_VNTB
 	  between PCI Root Port and PCIe Endpoint.
 
 	  If in doubt, say "N" to disable Endpoint NTB driver.
+
+config PCI_EPF_MHI
+	tristate "PCI Endpoint driver for MHI bus"
+	depends on PCI_ENDPOINT && MHI_BUS_EP
+	help
+	   Enable this configuration option to enable the PCI Endpoint
+	   driver for Modem Host Interface (MHI) bus in Qualcomm Endpoint
+	   devices such as SDX55.
+
+	   If in doubt, say "N" to disable Endpoint driver for MHI bus.
diff --git a/drivers/pci/endpoint/functions/Makefile b/drivers/pci/endpoint/functions/Makefile
index 5c13001deaba..696473fce50e 100644
--- a/drivers/pci/endpoint/functions/Makefile
+++ b/drivers/pci/endpoint/functions/Makefile
@@ -6,3 +6,4 @@
 obj-$(CONFIG_PCI_EPF_TEST)		+= pci-epf-test.o
 obj-$(CONFIG_PCI_EPF_NTB)		+= pci-epf-ntb.o
 obj-$(CONFIG_PCI_EPF_VNTB) 		+= pci-epf-vntb.o
+obj-$(CONFIG_PCI_EPF_MHI)		+= pci-epf-mhi.o
diff --git a/drivers/pci/endpoint/functions/pci-epf-mhi.c b/drivers/pci/endpoint/functions/pci-epf-mhi.c
new file mode 100644
index 000000000000..df924fb10e4d
--- /dev/null
+++ b/drivers/pci/endpoint/functions/pci-epf-mhi.c
@@ -0,0 +1,454 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * PCI EPF driver for MHI Endpoint devices
+ *
+ * Copyright (C) 2022 Linaro Ltd.
+ * Author: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
+ */
+
+#include <linux/mhi_ep.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/pci-epc.h>
+#include <linux/pci-epf.h>
+
+#define MHI_VERSION_1_0 0x01000000
+
+struct pci_epf_mhi_ep_info {
+	const struct mhi_ep_cntrl_config *config;
+	struct pci_epf_header *epf_header;
+	enum pci_barno bar_num;
+	u32 epf_flags;
+	u32 msi_count;
+	u32 mru;
+};
+
+#define MHI_EP_CHANNEL_CONFIG_UL(ch_num, ch_name)	\
+	{						\
+		.num = ch_num,				\
+		.name = ch_name,			\
+		.dir = DMA_TO_DEVICE,			\
+	}
+
+#define MHI_EP_CHANNEL_CONFIG_DL(ch_num, ch_name)	\
+	{						\
+		.num = ch_num,				\
+		.name = ch_name,			\
+		.dir = DMA_FROM_DEVICE,			\
+	}
+
+static const struct mhi_ep_channel_config mhi_v1_channels[] = {
+	MHI_EP_CHANNEL_CONFIG_UL(0, "LOOPBACK"),
+	MHI_EP_CHANNEL_CONFIG_DL(1, "LOOPBACK"),
+	MHI_EP_CHANNEL_CONFIG_UL(2, "SAHARA"),
+	MHI_EP_CHANNEL_CONFIG_DL(3, "SAHARA"),
+	MHI_EP_CHANNEL_CONFIG_UL(4, "DIAG"),
+	MHI_EP_CHANNEL_CONFIG_DL(5, "DIAG"),
+	MHI_EP_CHANNEL_CONFIG_UL(6, "SSR"),
+	MHI_EP_CHANNEL_CONFIG_DL(7, "SSR"),
+	MHI_EP_CHANNEL_CONFIG_UL(8, "QDSS"),
+	MHI_EP_CHANNEL_CONFIG_DL(9, "QDSS"),
+	MHI_EP_CHANNEL_CONFIG_UL(10, "EFS"),
+	MHI_EP_CHANNEL_CONFIG_DL(11, "EFS"),
+	MHI_EP_CHANNEL_CONFIG_UL(12, "MBIM"),
+	MHI_EP_CHANNEL_CONFIG_DL(13, "MBIM"),
+	MHI_EP_CHANNEL_CONFIG_UL(14, "QMI"),
+	MHI_EP_CHANNEL_CONFIG_DL(15, "QMI"),
+	MHI_EP_CHANNEL_CONFIG_UL(16, "QMI"),
+	MHI_EP_CHANNEL_CONFIG_DL(17, "QMI"),
+	MHI_EP_CHANNEL_CONFIG_UL(18, "IP-CTRL-1"),
+	MHI_EP_CHANNEL_CONFIG_DL(19, "IP-CTRL-1"),
+	MHI_EP_CHANNEL_CONFIG_UL(20, "IPCR"),
+	MHI_EP_CHANNEL_CONFIG_DL(21, "IPCR"),
+	MHI_EP_CHANNEL_CONFIG_UL(32, "DUN"),
+	MHI_EP_CHANNEL_CONFIG_DL(33, "DUN"),
+	MHI_EP_CHANNEL_CONFIG_UL(46, "IP_SW0"),
+	MHI_EP_CHANNEL_CONFIG_DL(47, "IP_SW0"),
+};
+
+static const struct mhi_ep_cntrl_config mhi_v1_config = {
+	.max_channels = 128,
+	.num_channels = ARRAY_SIZE(mhi_v1_channels),
+	.ch_cfg = mhi_v1_channels,
+	.mhi_version = MHI_VERSION_1_0,
+};
+
+static struct pci_epf_header sdx55_header = {
+	.vendorid = PCI_VENDOR_ID_QCOM,
+	.deviceid = 0x0306,
+	.baseclass_code = PCI_BASE_CLASS_COMMUNICATION,
+	.subclass_code = PCI_CLASS_COMMUNICATION_MODEM & 0xff,
+	.interrupt_pin	= PCI_INTERRUPT_INTA,
+};
+
+static const struct pci_epf_mhi_ep_info sdx55_info = {
+	.config = &mhi_v1_config,
+	.epf_header = &sdx55_header,
+	.bar_num = BAR_0,
+	.epf_flags = PCI_BASE_ADDRESS_MEM_TYPE_32,
+	.msi_count = 32,
+	.mru = 0x8000,
+};
+
+struct pci_epf_mhi {
+	const struct pci_epf_mhi_ep_info *info;
+	struct mhi_ep_cntrl mhi_cntrl;
+	struct pci_epf *epf;
+	struct mutex lock;
+	void __iomem *mmio;
+	resource_size_t mmio_phys;
+	u32 mmio_size;
+	int irq;
+	bool mhi_registered;
+};
+
+static int pci_epf_mhi_alloc_map(struct mhi_ep_cntrl *mhi_cntrl, u64 pci_addr,
+				 phys_addr_t *phys_ptr, void __iomem **virt, size_t size)
+{
+	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
+	struct pci_epf *epf = epf_mhi->epf;
+	struct pci_epc *epc = epf_mhi->epf->epc;
+	size_t offset = pci_addr & (epc->mem->window.page_size - 1);
+	void __iomem *virt_addr;
+	phys_addr_t phys_addr;
+	int ret;
+
+	virt_addr = pci_epc_mem_alloc_addr(epc, &phys_addr, size + offset);
+	if (!virt_addr)
+		return -ENOMEM;
+
+	ret = pci_epc_map_addr(epc, epf->func_no, epf->vfunc_no, phys_addr, pci_addr - offset,
+			       size + offset);
+	if (ret) {
+		pci_epc_mem_free_addr(epc, phys_addr, virt_addr, size + offset);
+
+		return ret;
+	}
+
+	*phys_ptr = phys_addr + offset;
+	*virt = virt_addr + offset;
+
+	return 0;
+}
+
+static void pci_epf_mhi_unmap_free(struct mhi_ep_cntrl *mhi_cntrl, u64 pci_addr,
+				   phys_addr_t phys_addr, void __iomem *virt_addr, size_t size)
+{
+	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
+	struct pci_epf *epf = epf_mhi->epf;
+	struct pci_epc *epc = epf->epc;
+	size_t offset = pci_addr & (epc->mem->window.page_size - 1);
+
+	pci_epc_unmap_addr(epc, epf->func_no, epf->vfunc_no, phys_addr - offset);
+	pci_epc_mem_free_addr(epc, phys_addr - offset, virt_addr - offset, size + offset);
+}
+
+static void pci_epf_mhi_raise_irq(struct mhi_ep_cntrl *mhi_cntrl, u32 vector)
+{
+	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
+	struct pci_epf *epf = epf_mhi->epf;
+	struct pci_epc *epc = epf->epc;
+
+	/*
+	 * Vector is incremented by 1 here as the DWC core will decrement it before
+	 * writing to iATU.
+	 */
+	pci_epc_raise_irq(epc, epf->func_no, epf->vfunc_no, PCI_EPC_IRQ_MSI, vector + 1);
+}
+
+static int pci_epf_mhi_read_from_host(struct mhi_ep_cntrl *mhi_cntrl, u64 from, void __iomem *to,
+			       size_t size)
+{
+	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
+	struct pci_epf *epf = epf_mhi->epf;
+	struct pci_epc *epc = epf_mhi->epf->epc;
+	void __iomem *tre_buf;
+	phys_addr_t tre_phys;
+	size_t offset = from % 0x1000;
+	int ret;
+
+	mutex_lock(&epf_mhi->lock);
+
+	tre_buf = pci_epc_mem_alloc_addr(epc, &tre_phys, size + offset);
+	if (!tre_buf) {
+		mutex_unlock(&epf_mhi->lock);
+		return -ENOMEM;
+	}
+
+	ret = pci_epc_map_addr(epc, epf->func_no, epf->vfunc_no, tre_phys, from - offset,
+			       size + offset);
+	if (ret) {
+		pci_epc_mem_free_addr(epc, tre_phys, tre_buf, size + offset);
+		mutex_unlock(&epf_mhi->lock);
+		return ret;
+	}
+
+	memcpy_fromio(to, tre_buf + offset, size);
+
+	pci_epc_unmap_addr(epc, epf->func_no, epf->vfunc_no, tre_phys);
+	pci_epc_mem_free_addr(epc, tre_phys, tre_buf, size + offset);
+
+	mutex_unlock(&epf_mhi->lock);
+
+	return 0;
+}
+
+static int pci_epf_mhi_write_to_host(struct mhi_ep_cntrl *mhi_cntrl, void __iomem *from, u64 to,
+			      size_t size)
+{
+	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
+	struct pci_epf *epf = epf_mhi->epf;
+	struct pci_epc *epc = epf_mhi->epf->epc;
+	void __iomem *tre_buf;
+	phys_addr_t tre_phys;
+	size_t offset = to % 0x1000;
+	int ret;
+
+	mutex_lock(&epf_mhi->lock);
+
+	tre_buf = pci_epc_mem_alloc_addr(epc, &tre_phys, size + offset);
+	if (!tre_buf) {
+		mutex_unlock(&epf_mhi->lock);
+		return -ENOMEM;
+	}
+
+	ret = pci_epc_map_addr(epc, epf->func_no, epf->vfunc_no, tre_phys, to - offset,
+			       size + offset);
+	if (ret) {
+		pci_epc_mem_free_addr(epc, tre_phys, tre_buf, size + offset);
+		mutex_unlock(&epf_mhi->lock);
+		return ret;
+	}
+
+	memcpy_toio(tre_buf + offset, from, size);
+
+	pci_epc_unmap_addr(epc, epf->func_no, epf->vfunc_no, tre_phys);
+	pci_epc_mem_free_addr(epc, tre_phys, tre_buf, size + offset);
+
+	mutex_unlock(&epf_mhi->lock);
+
+	return 0;
+}
+
+static int pci_epf_mhi_core_init(struct pci_epf *epf)
+{
+	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
+	const struct pci_epf_mhi_ep_info *info = epf_mhi->info;
+	struct pci_epf_bar *epf_bar = &epf->bar[info->bar_num];
+	struct pci_epc *epc = epf->epc;
+	struct device *dev = &epf->dev;
+	int ret;
+
+	epf_bar->phys_addr = epf_mhi->mmio_phys;
+	epf_bar->size = epf_mhi->mmio_size;
+	epf_bar->barno = info->bar_num;
+	epf_bar->flags = info->epf_flags;
+	ret = pci_epc_set_bar(epc, epf->func_no, epf->vfunc_no, epf_bar);
+	if (ret) {
+		dev_err(dev, "Failed to set BAR: %d\n", ret);
+		return ret;
+	}
+
+	ret = pci_epc_set_msi(epc, epf->func_no, epf->vfunc_no,
+			      order_base_2(info->msi_count));
+	if (ret) {
+		dev_err(dev, "Failed to set MSI configuration: %d\n", ret);
+		return ret;
+	}
+
+	ret = pci_epc_write_header(epc, epf->func_no, epf->vfunc_no, epf->header);
+	if (ret) {
+		dev_err(dev, "Failed to set Configuration header: %d\n", ret);
+		return ret;
+	}
+
+	return 0;
+}
+
+static int pci_epf_mhi_link_up(struct pci_epf *epf)
+{
+	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
+	const struct pci_epf_mhi_ep_info *info = epf_mhi->info;
+	struct mhi_ep_cntrl *mhi_cntrl = &epf_mhi->mhi_cntrl;
+	struct pci_epc *epc = epf->epc;
+	struct device *dev = &epf->dev;
+	int ret;
+
+	mhi_cntrl->mmio = epf_mhi->mmio;
+	mhi_cntrl->irq = epf_mhi->irq;
+	mhi_cntrl->mru = info->mru;
+
+	/* Assign the struct dev of PCI EP as MHI controller device */
+	mhi_cntrl->cntrl_dev = epc->dev.parent;
+	mhi_cntrl->raise_irq = pci_epf_mhi_raise_irq;
+	mhi_cntrl->alloc_map = pci_epf_mhi_alloc_map;
+	mhi_cntrl->unmap_free = pci_epf_mhi_unmap_free;
+	mhi_cntrl->read_from_host = pci_epf_mhi_read_from_host;
+	mhi_cntrl->write_to_host = pci_epf_mhi_write_to_host;
+
+	/* Register the MHI EP controller */
+	ret = mhi_ep_register_controller(mhi_cntrl, info->config);
+	if (ret) {
+		dev_err(dev, "Failed to register MHI EP controller: %d\n", ret);
+		return ret;
+	}
+
+	epf_mhi->mhi_registered = true;
+
+	return 0;
+}
+
+static int pci_epf_mhi_link_down(struct pci_epf *epf)
+{
+	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
+	struct mhi_ep_cntrl *mhi_cntrl = &epf_mhi->mhi_cntrl;
+
+	if (epf_mhi->mhi_registered) {
+		mhi_ep_power_down(mhi_cntrl);
+		mhi_ep_unregister_controller(mhi_cntrl);
+		epf_mhi->mhi_registered = false;
+	}
+
+	return 0;
+}
+
+static int pci_epf_mhi_bme(struct pci_epf *epf)
+{
+	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
+	struct mhi_ep_cntrl *mhi_cntrl = &epf_mhi->mhi_cntrl;
+	struct device *dev = &epf->dev;
+	int ret;
+
+	/* Power up the MHI EP stack if link is up and stack is in power down state */
+	if (!mhi_cntrl->enabled && epf_mhi->mhi_registered) {
+		ret = mhi_ep_power_up(mhi_cntrl);
+		if (ret) {
+			dev_err(dev, "Failed to power up MHI EP: %d\n", ret);
+			mhi_ep_unregister_controller(mhi_cntrl);
+			epf_mhi->mhi_registered = false;
+		}
+	}
+
+	return 0;
+}
+
+static int pci_epf_mhi_bind(struct pci_epf *epf)
+{
+	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
+	struct pci_epc *epc = epf->epc;
+	struct platform_device *pdev = to_platform_device(epc->dev.parent);
+	struct device *dev = &epf->dev;
+	struct resource *res;
+	int ret;
+
+	if (WARN_ON_ONCE(!epc))
+		return -EINVAL;
+
+	/* Get MMIO base address from Endpoint controller */
+	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "mmio");
+	epf_mhi->mmio_phys = res->start;
+	epf_mhi->mmio_size = resource_size(res);
+
+	epf_mhi->mmio = ioremap_wc(epf_mhi->mmio_phys, epf_mhi->mmio_size);
+	if (IS_ERR(epf_mhi->mmio))
+		return PTR_ERR(epf_mhi->mmio);
+
+	ret = platform_get_irq_byname(pdev, "doorbell");
+	if (ret < 0) {
+		dev_err(dev, "Failed to get Doorbell IRQ\n");
+		iounmap(epf_mhi->mmio);
+		return ret;
+	}
+
+	epf_mhi->irq = ret;
+
+	return 0;
+}
+
+static void pci_epf_mhi_unbind(struct pci_epf *epf)
+{
+	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
+	const struct pci_epf_mhi_ep_info *info = epf_mhi->info;
+	struct pci_epf_bar *epf_bar = &epf->bar[info->bar_num];
+	struct mhi_ep_cntrl *mhi_cntrl = &epf_mhi->mhi_cntrl;
+	struct pci_epc *epc = epf->epc;
+
+	/*
+	 * Forcefully power down the MHI EP stack. Only way to bring the MHI EP stack
+	 * back to working state after successive bind is by getting BME from host.
+	 */
+	if (epf_mhi->mhi_registered) {
+		mhi_ep_power_down(mhi_cntrl);
+		mhi_ep_unregister_controller(mhi_cntrl);
+		epf_mhi->mhi_registered = false;
+	}
+
+	iounmap(epf_mhi->mmio);
+	pci_epc_clear_bar(epc, epf->func_no, epf->vfunc_no, epf_bar);
+}
+
+static struct pci_epc_event_ops pci_epf_mhi_event_ops = {
+	.core_init = pci_epf_mhi_core_init,
+	.link_up = pci_epf_mhi_link_up,
+	.link_down = pci_epf_mhi_link_down,
+	.bme = pci_epf_mhi_bme,
+};
+
+static int pci_epf_mhi_probe(struct pci_epf *epf, const struct pci_epf_device_id *id)
+{
+	struct pci_epf_mhi_ep_info *info = (struct pci_epf_mhi_ep_info *) id->driver_data;
+	struct pci_epf_mhi *epf_mhi;
+	struct device *dev = &epf->dev;
+
+	epf_mhi = devm_kzalloc(dev, sizeof(*epf_mhi), GFP_KERNEL);
+	if (!epf_mhi)
+		return -ENOMEM;
+
+	epf->header = info->epf_header;
+	epf_mhi->info = info;
+	epf_mhi->epf = epf;
+
+	epf->event_ops = &pci_epf_mhi_event_ops;
+
+	mutex_init(&epf_mhi->lock);
+
+	epf_set_drvdata(epf, epf_mhi);
+
+	return 0;
+}
+
+static const struct pci_epf_device_id pci_epf_mhi_ids[] = {
+	{
+		.name = "sdx55", .driver_data = (kernel_ulong_t) &sdx55_info,
+	},
+	{},
+};
+
+static struct pci_epf_ops pci_epf_mhi_ops = {
+	.unbind	= pci_epf_mhi_unbind,
+	.bind	= pci_epf_mhi_bind,
+};
+
+static struct pci_epf_driver pci_epf_mhi_driver = {
+	.driver.name	= "pci_epf_mhi",
+	.probe		= pci_epf_mhi_probe,
+	.id_table	= pci_epf_mhi_ids,
+	.ops		= &pci_epf_mhi_ops,
+	.owner		= THIS_MODULE,
+};
+
+static int __init pci_epf_mhi_init(void)
+{
+	return pci_epf_register_driver(&pci_epf_mhi_driver);
+}
+module_init(pci_epf_mhi_init);
+
+static void __exit pci_epf_mhi_exit(void)
+{
+	pci_epf_unregister_driver(&pci_epf_mhi_driver);
+}
+module_exit(pci_epf_mhi_exit);
+
+MODULE_DESCRIPTION("PCI EPF driver for MHI Endpoint devices");
+MODULE_AUTHOR("Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>");
+MODULE_LICENSE("GPL");
-- 
2.25.1

RE: [EXT] [PATCH v4 1/7] PCI: endpoint: Pass EPF device ID to the probe function

[Frank Li]

> 
> Currently, the EPF probe function doesn't get the device ID argument
> needed
> to correctly identify the device table ID of the EPF device.
> 
> When multiple entries are added to the "struct pci_epf_device_id" table,
> the probe function needs to identify the correct one. And the only way to
> do so is by storing the correct device ID in "struct pci_epf" during
> "pci_epf_match_id()" and passing that to probe().
> 
> Reviewed-by: Kishon Vijay Abraham I <kishon@kernel.org>
> Signed-off-by: Manivannan Sadhasivam
> <manivannan.sadhasivam@linaro.org>

Reviewed-by: Frank Li <Frank.Li@nxp.com>

> ---
>  drivers/pci/endpoint/functions/pci-epf-ntb.c  | 3 ++-
>  drivers/pci/endpoint/functions/pci-epf-test.c | 2 +-
>  drivers/pci/endpoint/functions/pci-epf-vntb.c | 2 +-
>  drivers/pci/endpoint/pci-epf-core.c           | 8 +++++---
>  include/linux/pci-epf.h                       | 4 +++-
>  5 files changed, 12 insertions(+), 7 deletions(-)
> 

RE: [EXT] [PATCH v4 4/7] PCI: endpoint: Add BME notifier support

[Frank Li]

> @@ -72,11 +72,13 @@ struct pci_epf_ops {
>   * @core_init: Callback for the EPC initialization complete event
>   * @link_up: Callback for the EPC link up event
>   * @link_down: Callback for the EPC link down event
> + * @bme: Callback for the EPC BME (Bus Master Enable) event
>   */
>  struct pci_epc_event_ops {
>         int (*core_init)(struct pci_epf *epf);
>         int (*link_up)(struct pci_epf *epf);
>         int (*link_down)(struct pci_epf *epf);
> +       int (*bme)(struct pci_epf *epf);

I posted a doorbell from host to EP at
https://lore.kernel.org/imx/20230426203436.1277307-2-Frank.Li@nxp.com/T/#u

Can we consider consolidate these notification to one function and distinguished by
EVENT_ID in future?

Best regards
Frank Li

>  };
> 
>  /**
> --
> 2.25.1

Re: [EXT] [PATCH v4 4/7] PCI: endpoint: Add BME notifier support

[Manivannan Sadhasivam]

On Fri, May 19, 2023 at 06:19:44PM +0000, Frank Li wrote:
> 
> > @@ -72,11 +72,13 @@ struct pci_epf_ops {
> >   * @core_init: Callback for the EPC initialization complete event
> >   * @link_up: Callback for the EPC link up event
> >   * @link_down: Callback for the EPC link down event
> > + * @bme: Callback for the EPC BME (Bus Master Enable) event
> >   */
> >  struct pci_epc_event_ops {
> >         int (*core_init)(struct pci_epf *epf);
> >         int (*link_up)(struct pci_epf *epf);
> >         int (*link_down)(struct pci_epf *epf);
> > +       int (*bme)(struct pci_epf *epf);
> 
> I posted a doorbell from host to EP at
> https://lore.kernel.org/imx/20230426203436.1277307-2-Frank.Li@nxp.com/T/#u
> 
> Can we consider consolidate these notification to one function and distinguished by
> EVENT_ID in future?
> 

My preference is to keep a separate callback for each event as it makes the code
look better instead of clubbing everything in a single callback separated by a
switch case.

- Mani

> Best regards
> Frank Li
> 
> >  };
> > 
> >  /**
> > --
> > 2.25.1
> 

-- 
மணிவண்ணன் சதாசிவம்

RE: [EXT] [PATCH v4 4/7] PCI: endpoint: Add BME notifier support

[Frank Li]

> >
> > Can we consider consolidate these notification to one function and
> distinguished by
> > EVENT_ID in future?
> >
> 
> My preference is to keep a separate callback for each event as it makes the
> code
> look better instead of clubbing everything in a single callback separated by a
> switch case.
> 

Okay, that is not big deal. I hope these can be merged into pci tree soon. 
So I can add more notification at layerscape platform.

> - Mani
> 
> > Best regards
> > Frank Li
> >
> > >  };
> > >
> > >  /**
> > > --
> > > 2.25.1
> >
> 
> --
> மணிவண்ணன் சதாசிவம்

Re: [PATCH v4 1/7] PCI: endpoint: Pass EPF device ID to the probe function

[Lorenzo Pieralisi]

On Fri, May 19, 2023 at 08:12:09PM +0530, Manivannan Sadhasivam wrote:
> Currently, the EPF probe function doesn't get the device ID argument needed
> to correctly identify the device table ID of the EPF device.
> 
> When multiple entries are added to the "struct pci_epf_device_id" table,
> the probe function needs to identify the correct one. And the only way to
> do so is by storing the correct device ID in "struct pci_epf" during
> "pci_epf_match_id()" and passing that to probe().
> 
> Reviewed-by: Kishon Vijay Abraham I <kishon@kernel.org>
> Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> ---
>  drivers/pci/endpoint/functions/pci-epf-ntb.c  | 3 ++-
>  drivers/pci/endpoint/functions/pci-epf-test.c | 2 +-
>  drivers/pci/endpoint/functions/pci-epf-vntb.c | 2 +-
>  drivers/pci/endpoint/pci-epf-core.c           | 8 +++++---
>  include/linux/pci-epf.h                       | 4 +++-
>  5 files changed, 12 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/pci/endpoint/functions/pci-epf-ntb.c b/drivers/pci/endpoint/functions/pci-epf-ntb.c
> index 9a00448c7e61..980b4ecf19a2 100644
> --- a/drivers/pci/endpoint/functions/pci-epf-ntb.c
> +++ b/drivers/pci/endpoint/functions/pci-epf-ntb.c
> @@ -2075,11 +2075,12 @@ static struct config_group *epf_ntb_add_cfs(struct pci_epf *epf,
>  /**
>   * epf_ntb_probe() - Probe NTB function driver
>   * @epf: NTB endpoint function device
> + * @id: NTB endpoint function device ID
>   *
>   * Probe NTB function driver when endpoint function bus detects a NTB
>   * endpoint function.
>   */
> -static int epf_ntb_probe(struct pci_epf *epf)
> +static int epf_ntb_probe(struct pci_epf *epf, const struct pci_epf_device_id *id)
>  {
>  	struct epf_ntb *ntb;
>  	struct device *dev;
> diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c b/drivers/pci/endpoint/functions/pci-epf-test.c
> index 0f9d2ec822ac..d5fcc78a5b73 100644
> --- a/drivers/pci/endpoint/functions/pci-epf-test.c
> +++ b/drivers/pci/endpoint/functions/pci-epf-test.c
> @@ -980,7 +980,7 @@ static const struct pci_epf_device_id pci_epf_test_ids[] = {
>  	{},
>  };
>  
> -static int pci_epf_test_probe(struct pci_epf *epf)
> +static int pci_epf_test_probe(struct pci_epf *epf, const struct pci_epf_device_id *id)
>  {
>  	struct pci_epf_test *epf_test;
>  	struct device *dev = &epf->dev;
> diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> index b7c7a8af99f4..122eb7a12028 100644
> --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> @@ -1401,7 +1401,7 @@ static struct pci_epf_ops epf_ntb_ops = {
>   *
>   * Returns: Zero for success, or an error code in case of failure
>   */
> -static int epf_ntb_probe(struct pci_epf *epf)
> +static int epf_ntb_probe(struct pci_epf *epf, const struct pci_epf_device_id *id)
>  {
>  	struct epf_ntb *ntb;
>  	struct device *dev;
> diff --git a/drivers/pci/endpoint/pci-epf-core.c b/drivers/pci/endpoint/pci-epf-core.c
> index 2036e38be093..924564288c9a 100644
> --- a/drivers/pci/endpoint/pci-epf-core.c
> +++ b/drivers/pci/endpoint/pci-epf-core.c
> @@ -494,11 +494,13 @@ static const struct device_type pci_epf_type = {
>  };
>  
>  static int
> -pci_epf_match_id(const struct pci_epf_device_id *id, const struct pci_epf *epf)
> +pci_epf_match_id(const struct pci_epf_device_id *id, struct pci_epf *epf)
>  {
>  	while (id->name[0]) {
> -		if (strcmp(epf->name, id->name) == 0)
> +		if (strcmp(epf->name, id->name) == 0) {
> +			epf->id = id;
>  			return true;
> +		}
>  		id++;
>  	}

I disagree with this patch's intent. The match function should not
change the parameters state. We should export this function to drivers
so that upon probe they can retrieve the matching id themselves,
as other bus interfaces do IMO.

Thanks,
Lorenzo

> @@ -526,7 +528,7 @@ static int pci_epf_device_probe(struct device *dev)
>  
>  	epf->driver = driver;
>  
> -	return driver->probe(epf);
> +	return driver->probe(epf, epf->id);
>  }
>  
>  static void pci_epf_device_remove(struct device *dev)
> diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
> index a215dc8ce693..bc613f0df7e3 100644
> --- a/include/linux/pci-epf.h
> +++ b/include/linux/pci-epf.h
> @@ -89,7 +89,7 @@ struct pci_epc_event_ops {
>   * @id_table: identifies EPF devices for probing
>   */
>  struct pci_epf_driver {
> -	int	(*probe)(struct pci_epf *epf);
> +	int	(*probe)(struct pci_epf *epf, const struct pci_epf_device_id *id);
>  	void	(*remove)(struct pci_epf *epf);
>  
>  	struct device_driver	driver;
> @@ -131,6 +131,7 @@ struct pci_epf_bar {
>   * @epc: the EPC device to which this EPF device is bound
>   * @epf_pf: the physical EPF device to which this virtual EPF device is bound
>   * @driver: the EPF driver to which this EPF device is bound
> + * @id: Pointer to the EPF device ID
>   * @list: to add pci_epf as a list of PCI endpoint functions to pci_epc
>   * @lock: mutex to protect pci_epf_ops
>   * @sec_epc: the secondary EPC device to which this EPF device is bound
> @@ -158,6 +159,7 @@ struct pci_epf {
>  	struct pci_epc		*epc;
>  	struct pci_epf		*epf_pf;
>  	struct pci_epf_driver	*driver;
> +	const struct pci_epf_device_id *id;
>  	struct list_head	list;
>  	/* mutex to protect against concurrent access of pci_epf_ops */
>  	struct mutex		lock;
> -- 
> 2.25.1
> 

Re: [PATCH v4 7/7] PCI: endpoint: Add PCI Endpoint function driver for MHI bus

[Lorenzo Pieralisi]

On Fri, May 19, 2023 at 08:12:15PM +0530, Manivannan Sadhasivam wrote:
> Add PCI Endpoint driver for the Qualcomm MHI (Modem Host Interface) bus.
> The driver implements the MHI function over PCI in the endpoint device
> such as SDX55 modem. The MHI endpoint function driver acts as a
> controller driver for the MHI Endpoint stack and carries out all PCI
> related activities like mapping the host memory using iATU, triggering
> MSIs etc...
> 
> Reviewed-by: Kishon Vijay Abraham I <kishon@kernel.org>
> Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> ---
>  drivers/pci/endpoint/functions/Kconfig       |  10 +
>  drivers/pci/endpoint/functions/Makefile      |   1 +
>  drivers/pci/endpoint/functions/pci-epf-mhi.c | 454 +++++++++++++++++++
>  3 files changed, 465 insertions(+)
>  create mode 100644 drivers/pci/endpoint/functions/pci-epf-mhi.c
> 
> diff --git a/drivers/pci/endpoint/functions/Kconfig b/drivers/pci/endpoint/functions/Kconfig
> index 9fd560886871..f5171b4fabbe 100644
> --- a/drivers/pci/endpoint/functions/Kconfig
> +++ b/drivers/pci/endpoint/functions/Kconfig
> @@ -37,3 +37,13 @@ config PCI_EPF_VNTB
>  	  between PCI Root Port and PCIe Endpoint.
>  
>  	  If in doubt, say "N" to disable Endpoint NTB driver.
> +
> +config PCI_EPF_MHI
> +	tristate "PCI Endpoint driver for MHI bus"
> +	depends on PCI_ENDPOINT && MHI_BUS_EP
> +	help
> +	   Enable this configuration option to enable the PCI Endpoint
> +	   driver for Modem Host Interface (MHI) bus in Qualcomm Endpoint
> +	   devices such as SDX55.
> +
> +	   If in doubt, say "N" to disable Endpoint driver for MHI bus.
> diff --git a/drivers/pci/endpoint/functions/Makefile b/drivers/pci/endpoint/functions/Makefile
> index 5c13001deaba..696473fce50e 100644
> --- a/drivers/pci/endpoint/functions/Makefile
> +++ b/drivers/pci/endpoint/functions/Makefile
> @@ -6,3 +6,4 @@
>  obj-$(CONFIG_PCI_EPF_TEST)		+= pci-epf-test.o
>  obj-$(CONFIG_PCI_EPF_NTB)		+= pci-epf-ntb.o
>  obj-$(CONFIG_PCI_EPF_VNTB) 		+= pci-epf-vntb.o
> +obj-$(CONFIG_PCI_EPF_MHI)		+= pci-epf-mhi.o
> diff --git a/drivers/pci/endpoint/functions/pci-epf-mhi.c b/drivers/pci/endpoint/functions/pci-epf-mhi.c
> new file mode 100644
> index 000000000000..df924fb10e4d
> --- /dev/null
> +++ b/drivers/pci/endpoint/functions/pci-epf-mhi.c
> @@ -0,0 +1,454 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * PCI EPF driver for MHI Endpoint devices
> + *
> + * Copyright (C) 2022 Linaro Ltd.
> + * Author: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> + */
> +
> +#include <linux/mhi_ep.h>
> +#include <linux/module.h>
> +#include <linux/platform_device.h>
> +#include <linux/pci-epc.h>
> +#include <linux/pci-epf.h>
> +
> +#define MHI_VERSION_1_0 0x01000000
> +
> +struct pci_epf_mhi_ep_info {
> +	const struct mhi_ep_cntrl_config *config;
> +	struct pci_epf_header *epf_header;
> +	enum pci_barno bar_num;
> +	u32 epf_flags;
> +	u32 msi_count;
> +	u32 mru;
> +};
> +
> +#define MHI_EP_CHANNEL_CONFIG_UL(ch_num, ch_name)	\
> +	{						\
> +		.num = ch_num,				\
> +		.name = ch_name,			\
> +		.dir = DMA_TO_DEVICE,			\
> +	}
> +
> +#define MHI_EP_CHANNEL_CONFIG_DL(ch_num, ch_name)	\
> +	{						\
> +		.num = ch_num,				\
> +		.name = ch_name,			\
> +		.dir = DMA_FROM_DEVICE,			\
> +	}

You can define it as:

#define MHI_EP_CHANNEL_CONFIG(ch_num, ch_name, dir)	\
	{						\
		.num = ch_num,				\
		.name = ch_name,			\
		.dir = dir,				\
	}

#define MHI_EP_CHANNEL_CONFIG_DL(ch_num, ch_name)	\
	MHI_EP_CHANNEL_CONFIG(ch_num, ch_name, DMA_FROM_DEVICE)


etc.

		
> +
> +static const struct mhi_ep_channel_config mhi_v1_channels[] = {
> +	MHI_EP_CHANNEL_CONFIG_UL(0, "LOOPBACK"),
> +	MHI_EP_CHANNEL_CONFIG_DL(1, "LOOPBACK"),
> +	MHI_EP_CHANNEL_CONFIG_UL(2, "SAHARA"),
> +	MHI_EP_CHANNEL_CONFIG_DL(3, "SAHARA"),
> +	MHI_EP_CHANNEL_CONFIG_UL(4, "DIAG"),
> +	MHI_EP_CHANNEL_CONFIG_DL(5, "DIAG"),
> +	MHI_EP_CHANNEL_CONFIG_UL(6, "SSR"),
> +	MHI_EP_CHANNEL_CONFIG_DL(7, "SSR"),
> +	MHI_EP_CHANNEL_CONFIG_UL(8, "QDSS"),
> +	MHI_EP_CHANNEL_CONFIG_DL(9, "QDSS"),
> +	MHI_EP_CHANNEL_CONFIG_UL(10, "EFS"),
> +	MHI_EP_CHANNEL_CONFIG_DL(11, "EFS"),
> +	MHI_EP_CHANNEL_CONFIG_UL(12, "MBIM"),
> +	MHI_EP_CHANNEL_CONFIG_DL(13, "MBIM"),
> +	MHI_EP_CHANNEL_CONFIG_UL(14, "QMI"),
> +	MHI_EP_CHANNEL_CONFIG_DL(15, "QMI"),
> +	MHI_EP_CHANNEL_CONFIG_UL(16, "QMI"),
> +	MHI_EP_CHANNEL_CONFIG_DL(17, "QMI"),
> +	MHI_EP_CHANNEL_CONFIG_UL(18, "IP-CTRL-1"),
> +	MHI_EP_CHANNEL_CONFIG_DL(19, "IP-CTRL-1"),
> +	MHI_EP_CHANNEL_CONFIG_UL(20, "IPCR"),
> +	MHI_EP_CHANNEL_CONFIG_DL(21, "IPCR"),
> +	MHI_EP_CHANNEL_CONFIG_UL(32, "DUN"),
> +	MHI_EP_CHANNEL_CONFIG_DL(33, "DUN"),
> +	MHI_EP_CHANNEL_CONFIG_UL(46, "IP_SW0"),
> +	MHI_EP_CHANNEL_CONFIG_DL(47, "IP_SW0"),
> +};
> +
> +static const struct mhi_ep_cntrl_config mhi_v1_config = {
> +	.max_channels = 128,
> +	.num_channels = ARRAY_SIZE(mhi_v1_channels),
> +	.ch_cfg = mhi_v1_channels,
> +	.mhi_version = MHI_VERSION_1_0,
> +};
> +
> +static struct pci_epf_header sdx55_header = {
> +	.vendorid = PCI_VENDOR_ID_QCOM,
> +	.deviceid = 0x0306,
> +	.baseclass_code = PCI_BASE_CLASS_COMMUNICATION,
> +	.subclass_code = PCI_CLASS_COMMUNICATION_MODEM & 0xff,
> +	.interrupt_pin	= PCI_INTERRUPT_INTA,
> +};
> +
> +static const struct pci_epf_mhi_ep_info sdx55_info = {
> +	.config = &mhi_v1_config,
> +	.epf_header = &sdx55_header,
> +	.bar_num = BAR_0,
> +	.epf_flags = PCI_BASE_ADDRESS_MEM_TYPE_32,
> +	.msi_count = 32,
> +	.mru = 0x8000,
> +};
> +
> +struct pci_epf_mhi {
> +	const struct pci_epf_mhi_ep_info *info;
> +	struct mhi_ep_cntrl mhi_cntrl;
> +	struct pci_epf *epf;
> +	struct mutex lock;
> +	void __iomem *mmio;
> +	resource_size_t mmio_phys;
> +	u32 mmio_size;
> +	int irq;
> +	bool mhi_registered;

Do we really need this variable ? Can't it be inferred from
the framework ?

> +};
> +
> +static int pci_epf_mhi_alloc_map(struct mhi_ep_cntrl *mhi_cntrl, u64 pci_addr,
> +				 phys_addr_t *phys_ptr, void __iomem **virt, size_t size)
> +{
> +	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
> +	struct pci_epf *epf = epf_mhi->epf;
> +	struct pci_epc *epc = epf_mhi->epf->epc;
> +	size_t offset = pci_addr & (epc->mem->window.page_size - 1);
> +	void __iomem *virt_addr;
> +	phys_addr_t phys_addr;
> +	int ret;
> +
> +	virt_addr = pci_epc_mem_alloc_addr(epc, &phys_addr, size + offset);
> +	if (!virt_addr)
> +		return -ENOMEM;
> +
> +	ret = pci_epc_map_addr(epc, epf->func_no, epf->vfunc_no, phys_addr, pci_addr - offset,
> +			       size + offset);
> +	if (ret) {
> +		pci_epc_mem_free_addr(epc, phys_addr, virt_addr, size + offset);
> +
> +		return ret;
> +	}
> +
> +	*phys_ptr = phys_addr + offset;
> +	*virt = virt_addr + offset;
> +
> +	return 0;
> +}
> +
> +static void pci_epf_mhi_unmap_free(struct mhi_ep_cntrl *mhi_cntrl, u64 pci_addr,
> +				   phys_addr_t phys_addr, void __iomem *virt_addr, size_t size)
> +{
> +	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
> +	struct pci_epf *epf = epf_mhi->epf;
> +	struct pci_epc *epc = epf->epc;
> +	size_t offset = pci_addr & (epc->mem->window.page_size - 1);
> +
> +	pci_epc_unmap_addr(epc, epf->func_no, epf->vfunc_no, phys_addr - offset);
> +	pci_epc_mem_free_addr(epc, phys_addr - offset, virt_addr - offset, size + offset);
> +}
> +
> +static void pci_epf_mhi_raise_irq(struct mhi_ep_cntrl *mhi_cntrl, u32 vector)
> +{
> +	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
> +	struct pci_epf *epf = epf_mhi->epf;
> +	struct pci_epc *epc = epf->epc;
> +
> +	/*
> +	 * Vector is incremented by 1 here as the DWC core will decrement it before
> +	 * writing to iATU.

This isn't OK. It is an API, you can't write code explicitly relying on
the underlying implementation. I assume the API is not well specified,
that's why we need these tricks ?

> +	 */
> +	pci_epc_raise_irq(epc, epf->func_no, epf->vfunc_no, PCI_EPC_IRQ_MSI, vector + 1);
> +}
> +
> +static int pci_epf_mhi_read_from_host(struct mhi_ep_cntrl *mhi_cntrl, u64 from, void __iomem *to,
> +			       size_t size)
> +{
> +	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
> +	struct pci_epf *epf = epf_mhi->epf;
> +	struct pci_epc *epc = epf_mhi->epf->epc;
> +	void __iomem *tre_buf;
> +	phys_addr_t tre_phys;
> +	size_t offset = from % 0x1000;

Explain what 0x1000 represents - make it a macro if that
helps

> +	int ret;
> +
> +	mutex_lock(&epf_mhi->lock);
> +
> +	tre_buf = pci_epc_mem_alloc_addr(epc, &tre_phys, size + offset);
> +	if (!tre_buf) {
> +		mutex_unlock(&epf_mhi->lock);
> +		return -ENOMEM;
> +	}
> +
> +	ret = pci_epc_map_addr(epc, epf->func_no, epf->vfunc_no, tre_phys, from - offset,
> +			       size + offset);
> +	if (ret) {
> +		pci_epc_mem_free_addr(epc, tre_phys, tre_buf, size + offset);
> +		mutex_unlock(&epf_mhi->lock);

I'd prefer a single goto label instead of repeating the code for every
branch taken but that's up to you.

> +		return ret;
> +	}
> +
> +	memcpy_fromio(to, tre_buf + offset, size);
> +
> +	pci_epc_unmap_addr(epc, epf->func_no, epf->vfunc_no, tre_phys);
> +	pci_epc_mem_free_addr(epc, tre_phys, tre_buf, size + offset);
> +
> +	mutex_unlock(&epf_mhi->lock);
> +
> +	return 0;
> +}
> +
> +static int pci_epf_mhi_write_to_host(struct mhi_ep_cntrl *mhi_cntrl, void __iomem *from, u64 to,
> +			      size_t size)
> +{
> +	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
> +	struct pci_epf *epf = epf_mhi->epf;
> +	struct pci_epc *epc = epf_mhi->epf->epc;
> +	void __iomem *tre_buf;
> +	phys_addr_t tre_phys;
> +	size_t offset = to % 0x1000;

Ditto as above.

> +	int ret;
> +
> +	mutex_lock(&epf_mhi->lock);
> +
> +	tre_buf = pci_epc_mem_alloc_addr(epc, &tre_phys, size + offset);
> +	if (!tre_buf) {
> +		mutex_unlock(&epf_mhi->lock);
> +		return -ENOMEM;
> +	}
> +
> +	ret = pci_epc_map_addr(epc, epf->func_no, epf->vfunc_no, tre_phys, to - offset,
> +			       size + offset);
> +	if (ret) {
> +		pci_epc_mem_free_addr(epc, tre_phys, tre_buf, size + offset);
> +		mutex_unlock(&epf_mhi->lock);

Ditto.

> +		return ret;
> +	}
> +
> +	memcpy_toio(tre_buf + offset, from, size);
> +
> +	pci_epc_unmap_addr(epc, epf->func_no, epf->vfunc_no, tre_phys);
> +	pci_epc_mem_free_addr(epc, tre_phys, tre_buf, size + offset);
> +
> +	mutex_unlock(&epf_mhi->lock);
> +
> +	return 0;
> +}
> +
> +static int pci_epf_mhi_core_init(struct pci_epf *epf)
> +{
> +	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
> +	const struct pci_epf_mhi_ep_info *info = epf_mhi->info;
> +	struct pci_epf_bar *epf_bar = &epf->bar[info->bar_num];
> +	struct pci_epc *epc = epf->epc;
> +	struct device *dev = &epf->dev;
> +	int ret;
> +
> +	epf_bar->phys_addr = epf_mhi->mmio_phys;
> +	epf_bar->size = epf_mhi->mmio_size;
> +	epf_bar->barno = info->bar_num;
> +	epf_bar->flags = info->epf_flags;
> +	ret = pci_epc_set_bar(epc, epf->func_no, epf->vfunc_no, epf_bar);
> +	if (ret) {
> +		dev_err(dev, "Failed to set BAR: %d\n", ret);
> +		return ret;
> +	}
> +
> +	ret = pci_epc_set_msi(epc, epf->func_no, epf->vfunc_no,
> +			      order_base_2(info->msi_count));
> +	if (ret) {
> +		dev_err(dev, "Failed to set MSI configuration: %d\n", ret);
> +		return ret;
> +	}
> +
> +	ret = pci_epc_write_header(epc, epf->func_no, epf->vfunc_no, epf->header);
> +	if (ret) {
> +		dev_err(dev, "Failed to set Configuration header: %d\n", ret);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int pci_epf_mhi_link_up(struct pci_epf *epf)
> +{
> +	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
> +	const struct pci_epf_mhi_ep_info *info = epf_mhi->info;
> +	struct mhi_ep_cntrl *mhi_cntrl = &epf_mhi->mhi_cntrl;
> +	struct pci_epc *epc = epf->epc;
> +	struct device *dev = &epf->dev;
> +	int ret;
> +
> +	mhi_cntrl->mmio = epf_mhi->mmio;
> +	mhi_cntrl->irq = epf_mhi->irq;
> +	mhi_cntrl->mru = info->mru;
> +
> +	/* Assign the struct dev of PCI EP as MHI controller device */
> +	mhi_cntrl->cntrl_dev = epc->dev.parent;
> +	mhi_cntrl->raise_irq = pci_epf_mhi_raise_irq;
> +	mhi_cntrl->alloc_map = pci_epf_mhi_alloc_map;
> +	mhi_cntrl->unmap_free = pci_epf_mhi_unmap_free;
> +	mhi_cntrl->read_from_host = pci_epf_mhi_read_from_host;
> +	mhi_cntrl->write_to_host = pci_epf_mhi_write_to_host;
> +
> +	/* Register the MHI EP controller */
> +	ret = mhi_ep_register_controller(mhi_cntrl, info->config);
> +	if (ret) {
> +		dev_err(dev, "Failed to register MHI EP controller: %d\n", ret);
> +		return ret;
> +	}
> +
> +	epf_mhi->mhi_registered = true;

I don't like this. It should be part of mhi_ep_register_controller();

> +
> +	return 0;
> +}
> +
> +static int pci_epf_mhi_link_down(struct pci_epf *epf)
> +{
> +	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
> +	struct mhi_ep_cntrl *mhi_cntrl = &epf_mhi->mhi_cntrl;
> +
> +	if (epf_mhi->mhi_registered) {
> +		mhi_ep_power_down(mhi_cntrl);
> +		mhi_ep_unregister_controller(mhi_cntrl);
> +		epf_mhi->mhi_registered = false;
> +	}
> +
> +	return 0;
> +}
> +
> +static int pci_epf_mhi_bme(struct pci_epf *epf)
> +{
> +	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
> +	struct mhi_ep_cntrl *mhi_cntrl = &epf_mhi->mhi_cntrl;
> +	struct device *dev = &epf->dev;
> +	int ret;
> +
> +	/* Power up the MHI EP stack if link is up and stack is in power down state */
> +	if (!mhi_cntrl->enabled && epf_mhi->mhi_registered) {
> +		ret = mhi_ep_power_up(mhi_cntrl);
> +		if (ret) {
> +			dev_err(dev, "Failed to power up MHI EP: %d\n", ret);
> +			mhi_ep_unregister_controller(mhi_cntrl);
> +			epf_mhi->mhi_registered = false;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static int pci_epf_mhi_bind(struct pci_epf *epf)
> +{
> +	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
> +	struct pci_epc *epc = epf->epc;
> +	struct platform_device *pdev = to_platform_device(epc->dev.parent);
> +	struct device *dev = &epf->dev;
> +	struct resource *res;
> +	int ret;
> +
> +	if (WARN_ON_ONCE(!epc))
> +		return -EINVAL;
> +
> +	/* Get MMIO base address from Endpoint controller */
> +	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "mmio");
> +	epf_mhi->mmio_phys = res->start;
> +	epf_mhi->mmio_size = resource_size(res);
> +
> +	epf_mhi->mmio = ioremap_wc(epf_mhi->mmio_phys, epf_mhi->mmio_size);

ioremap_wc(). Why wc mappings ? Please explain.

Thanks,
Lorenzo

> +	if (IS_ERR(epf_mhi->mmio))
> +		return PTR_ERR(epf_mhi->mmio);
> +
> +	ret = platform_get_irq_byname(pdev, "doorbell");
> +	if (ret < 0) {
> +		dev_err(dev, "Failed to get Doorbell IRQ\n");
> +		iounmap(epf_mhi->mmio);
> +		return ret;
> +	}
> +
> +	epf_mhi->irq = ret;
> +
> +	return 0;
> +}
> +
> +static void pci_epf_mhi_unbind(struct pci_epf *epf)
> +{
> +	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
> +	const struct pci_epf_mhi_ep_info *info = epf_mhi->info;
> +	struct pci_epf_bar *epf_bar = &epf->bar[info->bar_num];
> +	struct mhi_ep_cntrl *mhi_cntrl = &epf_mhi->mhi_cntrl;
> +	struct pci_epc *epc = epf->epc;
> +
> +	/*
> +	 * Forcefully power down the MHI EP stack. Only way to bring the MHI EP stack
> +	 * back to working state after successive bind is by getting BME from host.
> +	 */
> +	if (epf_mhi->mhi_registered) {
> +		mhi_ep_power_down(mhi_cntrl);
> +		mhi_ep_unregister_controller(mhi_cntrl);
> +		epf_mhi->mhi_registered = false;
> +	}
> +
> +	iounmap(epf_mhi->mmio);
> +	pci_epc_clear_bar(epc, epf->func_no, epf->vfunc_no, epf_bar);
> +}
> +
> +static struct pci_epc_event_ops pci_epf_mhi_event_ops = {
> +	.core_init = pci_epf_mhi_core_init,
> +	.link_up = pci_epf_mhi_link_up,
> +	.link_down = pci_epf_mhi_link_down,
> +	.bme = pci_epf_mhi_bme,
> +};
> +
> +static int pci_epf_mhi_probe(struct pci_epf *epf, const struct pci_epf_device_id *id)
> +{
> +	struct pci_epf_mhi_ep_info *info = (struct pci_epf_mhi_ep_info *) id->driver_data;
> +	struct pci_epf_mhi *epf_mhi;
> +	struct device *dev = &epf->dev;
> +
> +	epf_mhi = devm_kzalloc(dev, sizeof(*epf_mhi), GFP_KERNEL);
> +	if (!epf_mhi)
> +		return -ENOMEM;
> +
> +	epf->header = info->epf_header;
> +	epf_mhi->info = info;
> +	epf_mhi->epf = epf;
> +
> +	epf->event_ops = &pci_epf_mhi_event_ops;
> +
> +	mutex_init(&epf_mhi->lock);
> +
> +	epf_set_drvdata(epf, epf_mhi);
> +
> +	return 0;
> +}
> +
> +static const struct pci_epf_device_id pci_epf_mhi_ids[] = {
> +	{
> +		.name = "sdx55", .driver_data = (kernel_ulong_t) &sdx55_info,
> +	},
> +	{},
> +};
> +
> +static struct pci_epf_ops pci_epf_mhi_ops = {
> +	.unbind	= pci_epf_mhi_unbind,
> +	.bind	= pci_epf_mhi_bind,
> +};
> +
> +static struct pci_epf_driver pci_epf_mhi_driver = {
> +	.driver.name	= "pci_epf_mhi",
> +	.probe		= pci_epf_mhi_probe,
> +	.id_table	= pci_epf_mhi_ids,
> +	.ops		= &pci_epf_mhi_ops,
> +	.owner		= THIS_MODULE,
> +};
> +
> +static int __init pci_epf_mhi_init(void)
> +{
> +	return pci_epf_register_driver(&pci_epf_mhi_driver);
> +}
> +module_init(pci_epf_mhi_init);
> +
> +static void __exit pci_epf_mhi_exit(void)
> +{
> +	pci_epf_unregister_driver(&pci_epf_mhi_driver);
> +}
> +module_exit(pci_epf_mhi_exit);
> +
> +MODULE_DESCRIPTION("PCI EPF driver for MHI Endpoint devices");
> +MODULE_AUTHOR("Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>");
> +MODULE_LICENSE("GPL");
> -- 
> 2.25.1
> 

Re: [PATCH v4 1/7] PCI: endpoint: Pass EPF device ID to the probe function

[Manivannan Sadhasivam]

On Mon, May 29, 2023 at 11:36:03AM +0200, Lorenzo Pieralisi wrote:
> On Fri, May 19, 2023 at 08:12:09PM +0530, Manivannan Sadhasivam wrote:
> > Currently, the EPF probe function doesn't get the device ID argument needed
> > to correctly identify the device table ID of the EPF device.
> > 
> > When multiple entries are added to the "struct pci_epf_device_id" table,
> > the probe function needs to identify the correct one. And the only way to
> > do so is by storing the correct device ID in "struct pci_epf" during
> > "pci_epf_match_id()" and passing that to probe().
> > 
> > Reviewed-by: Kishon Vijay Abraham I <kishon@kernel.org>
> > Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> > ---
> >  drivers/pci/endpoint/functions/pci-epf-ntb.c  | 3 ++-
> >  drivers/pci/endpoint/functions/pci-epf-test.c | 2 +-
> >  drivers/pci/endpoint/functions/pci-epf-vntb.c | 2 +-
> >  drivers/pci/endpoint/pci-epf-core.c           | 8 +++++---
> >  include/linux/pci-epf.h                       | 4 +++-
> >  5 files changed, 12 insertions(+), 7 deletions(-)
> > 
> > diff --git a/drivers/pci/endpoint/functions/pci-epf-ntb.c b/drivers/pci/endpoint/functions/pci-epf-ntb.c
> > index 9a00448c7e61..980b4ecf19a2 100644
> > --- a/drivers/pci/endpoint/functions/pci-epf-ntb.c
> > +++ b/drivers/pci/endpoint/functions/pci-epf-ntb.c
> > @@ -2075,11 +2075,12 @@ static struct config_group *epf_ntb_add_cfs(struct pci_epf *epf,
> >  /**
> >   * epf_ntb_probe() - Probe NTB function driver
> >   * @epf: NTB endpoint function device
> > + * @id: NTB endpoint function device ID
> >   *
> >   * Probe NTB function driver when endpoint function bus detects a NTB
> >   * endpoint function.
> >   */
> > -static int epf_ntb_probe(struct pci_epf *epf)
> > +static int epf_ntb_probe(struct pci_epf *epf, const struct pci_epf_device_id *id)
> >  {
> >  	struct epf_ntb *ntb;
> >  	struct device *dev;
> > diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c b/drivers/pci/endpoint/functions/pci-epf-test.c
> > index 0f9d2ec822ac..d5fcc78a5b73 100644
> > --- a/drivers/pci/endpoint/functions/pci-epf-test.c
> > +++ b/drivers/pci/endpoint/functions/pci-epf-test.c
> > @@ -980,7 +980,7 @@ static const struct pci_epf_device_id pci_epf_test_ids[] = {
> >  	{},
> >  };
> >  
> > -static int pci_epf_test_probe(struct pci_epf *epf)
> > +static int pci_epf_test_probe(struct pci_epf *epf, const struct pci_epf_device_id *id)
> >  {
> >  	struct pci_epf_test *epf_test;
> >  	struct device *dev = &epf->dev;
> > diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > index b7c7a8af99f4..122eb7a12028 100644
> > --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
> > @@ -1401,7 +1401,7 @@ static struct pci_epf_ops epf_ntb_ops = {
> >   *
> >   * Returns: Zero for success, or an error code in case of failure
> >   */
> > -static int epf_ntb_probe(struct pci_epf *epf)
> > +static int epf_ntb_probe(struct pci_epf *epf, const struct pci_epf_device_id *id)
> >  {
> >  	struct epf_ntb *ntb;
> >  	struct device *dev;
> > diff --git a/drivers/pci/endpoint/pci-epf-core.c b/drivers/pci/endpoint/pci-epf-core.c
> > index 2036e38be093..924564288c9a 100644
> > --- a/drivers/pci/endpoint/pci-epf-core.c
> > +++ b/drivers/pci/endpoint/pci-epf-core.c
> > @@ -494,11 +494,13 @@ static const struct device_type pci_epf_type = {
> >  };
> >  
> >  static int
> > -pci_epf_match_id(const struct pci_epf_device_id *id, const struct pci_epf *epf)
> > +pci_epf_match_id(const struct pci_epf_device_id *id, struct pci_epf *epf)
> >  {
> >  	while (id->name[0]) {
> > -		if (strcmp(epf->name, id->name) == 0)
> > +		if (strcmp(epf->name, id->name) == 0) {
> > +			epf->id = id;
> >  			return true;
> > +		}
> >  		id++;
> >  	}
> 
> I disagree with this patch's intent. The match function should not
> change the parameters state. We should export this function to drivers
> so that upon probe they can retrieve the matching id themselves,
> as other bus interfaces do IMO.
> 

Ok, if you do not want to change the parameters inside match function, then I
could move the matching part to a separate function and make it to return the
correct pci_epf_device_id pointer. Based on this, the match() function can
return bool if the match is found or not and the pci_epf_device_probe() function
can pass the returned id to the driver's probe function. This is what being done
for PCI driver as well [1].

- Mani

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/pci-driver.c#n415

> Thanks,
> Lorenzo
> 
> > @@ -526,7 +528,7 @@ static int pci_epf_device_probe(struct device *dev)
> >  
> >  	epf->driver = driver;
> >  
> > -	return driver->probe(epf);
> > +	return driver->probe(epf, epf->id);
> >  }
> >  
> >  static void pci_epf_device_remove(struct device *dev)
> > diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
> > index a215dc8ce693..bc613f0df7e3 100644
> > --- a/include/linux/pci-epf.h
> > +++ b/include/linux/pci-epf.h
> > @@ -89,7 +89,7 @@ struct pci_epc_event_ops {
> >   * @id_table: identifies EPF devices for probing
> >   */
> >  struct pci_epf_driver {
> > -	int	(*probe)(struct pci_epf *epf);
> > +	int	(*probe)(struct pci_epf *epf, const struct pci_epf_device_id *id);
> >  	void	(*remove)(struct pci_epf *epf);
> >  
> >  	struct device_driver	driver;
> > @@ -131,6 +131,7 @@ struct pci_epf_bar {
> >   * @epc: the EPC device to which this EPF device is bound
> >   * @epf_pf: the physical EPF device to which this virtual EPF device is bound
> >   * @driver: the EPF driver to which this EPF device is bound
> > + * @id: Pointer to the EPF device ID
> >   * @list: to add pci_epf as a list of PCI endpoint functions to pci_epc
> >   * @lock: mutex to protect pci_epf_ops
> >   * @sec_epc: the secondary EPC device to which this EPF device is bound
> > @@ -158,6 +159,7 @@ struct pci_epf {
> >  	struct pci_epc		*epc;
> >  	struct pci_epf		*epf_pf;
> >  	struct pci_epf_driver	*driver;
> > +	const struct pci_epf_device_id *id;
> >  	struct list_head	list;
> >  	/* mutex to protect against concurrent access of pci_epf_ops */
> >  	struct mutex		lock;
> > -- 
> > 2.25.1
> > 

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v4 7/7] PCI: endpoint: Add PCI Endpoint function driver for MHI bus

[Manivannan Sadhasivam]

On Mon, May 29, 2023 at 03:11:17PM +0200, Lorenzo Pieralisi wrote:
> On Fri, May 19, 2023 at 08:12:15PM +0530, Manivannan Sadhasivam wrote:
> > Add PCI Endpoint driver for the Qualcomm MHI (Modem Host Interface) bus.
> > The driver implements the MHI function over PCI in the endpoint device
> > such as SDX55 modem. The MHI endpoint function driver acts as a
> > controller driver for the MHI Endpoint stack and carries out all PCI
> > related activities like mapping the host memory using iATU, triggering
> > MSIs etc...
> > 
> > Reviewed-by: Kishon Vijay Abraham I <kishon@kernel.org>
> > Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> > ---
> >  drivers/pci/endpoint/functions/Kconfig       |  10 +
> >  drivers/pci/endpoint/functions/Makefile      |   1 +
> >  drivers/pci/endpoint/functions/pci-epf-mhi.c | 454 +++++++++++++++++++
> >  3 files changed, 465 insertions(+)
> >  create mode 100644 drivers/pci/endpoint/functions/pci-epf-mhi.c
> > 
> > diff --git a/drivers/pci/endpoint/functions/Kconfig b/drivers/pci/endpoint/functions/Kconfig
> > index 9fd560886871..f5171b4fabbe 100644
> > --- a/drivers/pci/endpoint/functions/Kconfig
> > +++ b/drivers/pci/endpoint/functions/Kconfig
> > @@ -37,3 +37,13 @@ config PCI_EPF_VNTB
> >  	  between PCI Root Port and PCIe Endpoint.
> >  
> >  	  If in doubt, say "N" to disable Endpoint NTB driver.
> > +
> > +config PCI_EPF_MHI
> > +	tristate "PCI Endpoint driver for MHI bus"
> > +	depends on PCI_ENDPOINT && MHI_BUS_EP
> > +	help
> > +	   Enable this configuration option to enable the PCI Endpoint
> > +	   driver for Modem Host Interface (MHI) bus in Qualcomm Endpoint
> > +	   devices such as SDX55.
> > +
> > +	   If in doubt, say "N" to disable Endpoint driver for MHI bus.
> > diff --git a/drivers/pci/endpoint/functions/Makefile b/drivers/pci/endpoint/functions/Makefile
> > index 5c13001deaba..696473fce50e 100644
> > --- a/drivers/pci/endpoint/functions/Makefile
> > +++ b/drivers/pci/endpoint/functions/Makefile
> > @@ -6,3 +6,4 @@
> >  obj-$(CONFIG_PCI_EPF_TEST)		+= pci-epf-test.o
> >  obj-$(CONFIG_PCI_EPF_NTB)		+= pci-epf-ntb.o
> >  obj-$(CONFIG_PCI_EPF_VNTB) 		+= pci-epf-vntb.o
> > +obj-$(CONFIG_PCI_EPF_MHI)		+= pci-epf-mhi.o
> > diff --git a/drivers/pci/endpoint/functions/pci-epf-mhi.c b/drivers/pci/endpoint/functions/pci-epf-mhi.c
> > new file mode 100644
> > index 000000000000..df924fb10e4d
> > --- /dev/null
> > +++ b/drivers/pci/endpoint/functions/pci-epf-mhi.c

[...]

> > +#define MHI_EP_CHANNEL_CONFIG_UL(ch_num, ch_name)	\
> > +	{						\
> > +		.num = ch_num,				\
> > +		.name = ch_name,			\
> > +		.dir = DMA_TO_DEVICE,			\
> > +	}
> > +
> > +#define MHI_EP_CHANNEL_CONFIG_DL(ch_num, ch_name)	\
> > +	{						\
> > +		.num = ch_num,				\
> > +		.name = ch_name,			\
> > +		.dir = DMA_FROM_DEVICE,			\
> > +	}
> 
> You can define it as:
> 
> #define MHI_EP_CHANNEL_CONFIG(ch_num, ch_name, dir)	\
> 	{						\
> 		.num = ch_num,				\
> 		.name = ch_name,			\
> 		.dir = dir,				\
> 	}
> 
> #define MHI_EP_CHANNEL_CONFIG_DL(ch_num, ch_name)	\
> 	MHI_EP_CHANNEL_CONFIG(ch_num, ch_name, DMA_FROM_DEVICE)
> 
> 
> etc.

Ok.

> 
> 		
> > +
> > +static const struct mhi_ep_channel_config mhi_v1_channels[] = {
> > +	MHI_EP_CHANNEL_CONFIG_UL(0, "LOOPBACK"),
> > +	MHI_EP_CHANNEL_CONFIG_DL(1, "LOOPBACK"),
> > +	MHI_EP_CHANNEL_CONFIG_UL(2, "SAHARA"),
> > +	MHI_EP_CHANNEL_CONFIG_DL(3, "SAHARA"),
> > +	MHI_EP_CHANNEL_CONFIG_UL(4, "DIAG"),
> > +	MHI_EP_CHANNEL_CONFIG_DL(5, "DIAG"),
> > +	MHI_EP_CHANNEL_CONFIG_UL(6, "SSR"),
> > +	MHI_EP_CHANNEL_CONFIG_DL(7, "SSR"),
> > +	MHI_EP_CHANNEL_CONFIG_UL(8, "QDSS"),
> > +	MHI_EP_CHANNEL_CONFIG_DL(9, "QDSS"),
> > +	MHI_EP_CHANNEL_CONFIG_UL(10, "EFS"),
> > +	MHI_EP_CHANNEL_CONFIG_DL(11, "EFS"),
> > +	MHI_EP_CHANNEL_CONFIG_UL(12, "MBIM"),
> > +	MHI_EP_CHANNEL_CONFIG_DL(13, "MBIM"),
> > +	MHI_EP_CHANNEL_CONFIG_UL(14, "QMI"),
> > +	MHI_EP_CHANNEL_CONFIG_DL(15, "QMI"),
> > +	MHI_EP_CHANNEL_CONFIG_UL(16, "QMI"),
> > +	MHI_EP_CHANNEL_CONFIG_DL(17, "QMI"),
> > +	MHI_EP_CHANNEL_CONFIG_UL(18, "IP-CTRL-1"),
> > +	MHI_EP_CHANNEL_CONFIG_DL(19, "IP-CTRL-1"),
> > +	MHI_EP_CHANNEL_CONFIG_UL(20, "IPCR"),
> > +	MHI_EP_CHANNEL_CONFIG_DL(21, "IPCR"),
> > +	MHI_EP_CHANNEL_CONFIG_UL(32, "DUN"),
> > +	MHI_EP_CHANNEL_CONFIG_DL(33, "DUN"),
> > +	MHI_EP_CHANNEL_CONFIG_UL(46, "IP_SW0"),
> > +	MHI_EP_CHANNEL_CONFIG_DL(47, "IP_SW0"),
> > +};
> > +
> > +static const struct mhi_ep_cntrl_config mhi_v1_config = {
> > +	.max_channels = 128,
> > +	.num_channels = ARRAY_SIZE(mhi_v1_channels),
> > +	.ch_cfg = mhi_v1_channels,
> > +	.mhi_version = MHI_VERSION_1_0,
> > +};
> > +
> > +static struct pci_epf_header sdx55_header = {
> > +	.vendorid = PCI_VENDOR_ID_QCOM,
> > +	.deviceid = 0x0306,
> > +	.baseclass_code = PCI_BASE_CLASS_COMMUNICATION,
> > +	.subclass_code = PCI_CLASS_COMMUNICATION_MODEM & 0xff,
> > +	.interrupt_pin	= PCI_INTERRUPT_INTA,
> > +};
> > +
> > +static const struct pci_epf_mhi_ep_info sdx55_info = {
> > +	.config = &mhi_v1_config,
> > +	.epf_header = &sdx55_header,
> > +	.bar_num = BAR_0,
> > +	.epf_flags = PCI_BASE_ADDRESS_MEM_TYPE_32,
> > +	.msi_count = 32,
> > +	.mru = 0x8000,
> > +};
> > +
> > +struct pci_epf_mhi {
> > +	const struct pci_epf_mhi_ep_info *info;
> > +	struct mhi_ep_cntrl mhi_cntrl;
> > +	struct pci_epf *epf;
> > +	struct mutex lock;
> > +	void __iomem *mmio;
> > +	resource_size_t mmio_phys;
> > +	u32 mmio_size;
> > +	int irq;
> > +	bool mhi_registered;
> 
> Do we really need this variable ? Can't it be inferred from
> the framework ?
> 

Looks like yes. More below.

> > +};
> > +
> > +static int pci_epf_mhi_alloc_map(struct mhi_ep_cntrl *mhi_cntrl, u64 pci_addr,
> > +				 phys_addr_t *phys_ptr, void __iomem **virt, size_t size)
> > +{
> > +	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
> > +	struct pci_epf *epf = epf_mhi->epf;
> > +	struct pci_epc *epc = epf_mhi->epf->epc;
> > +	size_t offset = pci_addr & (epc->mem->window.page_size - 1);
> > +	void __iomem *virt_addr;
> > +	phys_addr_t phys_addr;
> > +	int ret;
> > +
> > +	virt_addr = pci_epc_mem_alloc_addr(epc, &phys_addr, size + offset);
> > +	if (!virt_addr)
> > +		return -ENOMEM;
> > +
> > +	ret = pci_epc_map_addr(epc, epf->func_no, epf->vfunc_no, phys_addr, pci_addr - offset,
> > +			       size + offset);
> > +	if (ret) {
> > +		pci_epc_mem_free_addr(epc, phys_addr, virt_addr, size + offset);
> > +
> > +		return ret;
> > +	}
> > +
> > +	*phys_ptr = phys_addr + offset;
> > +	*virt = virt_addr + offset;
> > +
> > +	return 0;
> > +}
> > +
> > +static void pci_epf_mhi_unmap_free(struct mhi_ep_cntrl *mhi_cntrl, u64 pci_addr,
> > +				   phys_addr_t phys_addr, void __iomem *virt_addr, size_t size)
> > +{
> > +	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
> > +	struct pci_epf *epf = epf_mhi->epf;
> > +	struct pci_epc *epc = epf->epc;
> > +	size_t offset = pci_addr & (epc->mem->window.page_size - 1);
> > +
> > +	pci_epc_unmap_addr(epc, epf->func_no, epf->vfunc_no, phys_addr - offset);
> > +	pci_epc_mem_free_addr(epc, phys_addr - offset, virt_addr - offset, size + offset);
> > +}
> > +
> > +static void pci_epf_mhi_raise_irq(struct mhi_ep_cntrl *mhi_cntrl, u32 vector)
> > +{
> > +	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
> > +	struct pci_epf *epf = epf_mhi->epf;
> > +	struct pci_epc *epc = epf->epc;
> > +
> > +	/*
> > +	 * Vector is incremented by 1 here as the DWC core will decrement it before
> > +	 * writing to iATU.
> 
> This isn't OK. It is an API, you can't write code explicitly relying on
> the underlying implementation. I assume the API is not well specified,
> that's why we need these tricks ?
> 

Well, this is not an API issue but rather an implementation detail of the DWC EP
core driver. The DWC driver expects the interrupt vectors to be 1 based, so it
decrements it before writing to the MSI address:
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-designware-ep.c#n537

> > +	 */
> > +	pci_epc_raise_irq(epc, epf->func_no, epf->vfunc_no, PCI_EPC_IRQ_MSI, vector + 1);
> > +}
> > +
> > +static int pci_epf_mhi_read_from_host(struct mhi_ep_cntrl *mhi_cntrl, u64 from, void __iomem *to,
> > +			       size_t size)
> > +{
> > +	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
> > +	struct pci_epf *epf = epf_mhi->epf;
> > +	struct pci_epc *epc = epf_mhi->epf->epc;
> > +	void __iomem *tre_buf;
> > +	phys_addr_t tre_phys;
> > +	size_t offset = from % 0x1000;
> 
> Explain what 0x1000 represents - make it a macro if that
> helps
> 

Ok.

> > +	int ret;
> > +
> > +	mutex_lock(&epf_mhi->lock);
> > +
> > +	tre_buf = pci_epc_mem_alloc_addr(epc, &tre_phys, size + offset);
> > +	if (!tre_buf) {
> > +		mutex_unlock(&epf_mhi->lock);
> > +		return -ENOMEM;
> > +	}
> > +
> > +	ret = pci_epc_map_addr(epc, epf->func_no, epf->vfunc_no, tre_phys, from - offset,
> > +			       size + offset);
> > +	if (ret) {
> > +		pci_epc_mem_free_addr(epc, tre_phys, tre_buf, size + offset);
> > +		mutex_unlock(&epf_mhi->lock);
> 
> I'd prefer a single goto label instead of repeating the code for every
> branch taken but that's up to you.
> 

Ok.

> > +		return ret;
> > +	}
> > +
> > +	memcpy_fromio(to, tre_buf + offset, size);
> > +
> > +	pci_epc_unmap_addr(epc, epf->func_no, epf->vfunc_no, tre_phys);
> > +	pci_epc_mem_free_addr(epc, tre_phys, tre_buf, size + offset);
> > +
> > +	mutex_unlock(&epf_mhi->lock);
> > +
> > +	return 0;
> > +}
> > +

[...]

> > +static int pci_epf_mhi_link_up(struct pci_epf *epf)
> > +{
> > +	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
> > +	const struct pci_epf_mhi_ep_info *info = epf_mhi->info;
> > +	struct mhi_ep_cntrl *mhi_cntrl = &epf_mhi->mhi_cntrl;
> > +	struct pci_epc *epc = epf->epc;
> > +	struct device *dev = &epf->dev;
> > +	int ret;
> > +
> > +	mhi_cntrl->mmio = epf_mhi->mmio;
> > +	mhi_cntrl->irq = epf_mhi->irq;
> > +	mhi_cntrl->mru = info->mru;
> > +
> > +	/* Assign the struct dev of PCI EP as MHI controller device */
> > +	mhi_cntrl->cntrl_dev = epc->dev.parent;
> > +	mhi_cntrl->raise_irq = pci_epf_mhi_raise_irq;
> > +	mhi_cntrl->alloc_map = pci_epf_mhi_alloc_map;
> > +	mhi_cntrl->unmap_free = pci_epf_mhi_unmap_free;
> > +	mhi_cntrl->read_from_host = pci_epf_mhi_read_from_host;
> > +	mhi_cntrl->write_to_host = pci_epf_mhi_write_to_host;
> > +
> > +	/* Register the MHI EP controller */
> > +	ret = mhi_ep_register_controller(mhi_cntrl, info->config);
> > +	if (ret) {
> > +		dev_err(dev, "Failed to register MHI EP controller: %d\n", ret);
> > +		return ret;
> > +	}
> > +
> > +	epf_mhi->mhi_registered = true;
> 
> I don't like this. It should be part of mhi_ep_register_controller();
> 

Hmm, I could use the "mhi_dev" pointer instead of a dedicated variable.

> > +
> > +	return 0;
> > +}
> > +
> > +static int pci_epf_mhi_link_down(struct pci_epf *epf)
> > +{
> > +	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
> > +	struct mhi_ep_cntrl *mhi_cntrl = &epf_mhi->mhi_cntrl;
> > +
> > +	if (epf_mhi->mhi_registered) {
> > +		mhi_ep_power_down(mhi_cntrl);
> > +		mhi_ep_unregister_controller(mhi_cntrl);
> > +		epf_mhi->mhi_registered = false;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static int pci_epf_mhi_bme(struct pci_epf *epf)
> > +{
> > +	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
> > +	struct mhi_ep_cntrl *mhi_cntrl = &epf_mhi->mhi_cntrl;
> > +	struct device *dev = &epf->dev;
> > +	int ret;
> > +
> > +	/* Power up the MHI EP stack if link is up and stack is in power down state */
> > +	if (!mhi_cntrl->enabled && epf_mhi->mhi_registered) {
> > +		ret = mhi_ep_power_up(mhi_cntrl);
> > +		if (ret) {
> > +			dev_err(dev, "Failed to power up MHI EP: %d\n", ret);
> > +			mhi_ep_unregister_controller(mhi_cntrl);
> > +			epf_mhi->mhi_registered = false;
> > +		}
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static int pci_epf_mhi_bind(struct pci_epf *epf)
> > +{
> > +	struct pci_epf_mhi *epf_mhi = epf_get_drvdata(epf);
> > +	struct pci_epc *epc = epf->epc;
> > +	struct platform_device *pdev = to_platform_device(epc->dev.parent);
> > +	struct device *dev = &epf->dev;
> > +	struct resource *res;
> > +	int ret;
> > +
> > +	if (WARN_ON_ONCE(!epc))
> > +		return -EINVAL;
> > +
> > +	/* Get MMIO base address from Endpoint controller */
> > +	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "mmio");
> > +	epf_mhi->mmio_phys = res->start;
> > +	epf_mhi->mmio_size = resource_size(res);
> > +
> > +	epf_mhi->mmio = ioremap_wc(epf_mhi->mmio_phys, epf_mhi->mmio_size);
> 
> ioremap_wc(). Why wc mappings ? Please explain.
> 

I saw a reference to write combined access requirement for the MMIO region but
couldn't verify it now. Will revert to non_wc mapping.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v4 7/7] PCI: endpoint: Add PCI Endpoint function driver for MHI bus

[Damien Le Moal]

On 5/30/23 02:38, Manivannan Sadhasivam wrote:
[...]
>>> +static void pci_epf_mhi_unmap_free(struct mhi_ep_cntrl *mhi_cntrl, u64 pci_addr,
>>> +				   phys_addr_t phys_addr, void __iomem *virt_addr, size_t size)
>>> +{
>>> +	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
>>> +	struct pci_epf *epf = epf_mhi->epf;
>>> +	struct pci_epc *epc = epf->epc;
>>> +	size_t offset = pci_addr & (epc->mem->window.page_size - 1);
>>> +
>>> +	pci_epc_unmap_addr(epc, epf->func_no, epf->vfunc_no, phys_addr - offset);
>>> +	pci_epc_mem_free_addr(epc, phys_addr - offset, virt_addr - offset, size + offset);
>>> +}
>>> +
>>> +static void pci_epf_mhi_raise_irq(struct mhi_ep_cntrl *mhi_cntrl, u32 vector)
>>> +{
>>> +	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
>>> +	struct pci_epf *epf = epf_mhi->epf;
>>> +	struct pci_epc *epc = epf->epc;
>>> +
>>> +	/*
>>> +	 * Vector is incremented by 1 here as the DWC core will decrement it before
>>> +	 * writing to iATU.
>>
>> This isn't OK. It is an API, you can't write code explicitly relying on
>> the underlying implementation. I assume the API is not well specified,
>> that's why we need these tricks ?
>>
> 
> Well, this is not an API issue but rather an implementation detail of the DWC EP
> core driver. The DWC driver expects the interrupt vectors to be 1 based, so it
> decrements it before writing to the MSI address:
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-designware-ep.c#n537

Then the driver should be fixed so that this peculiarity is not visible at the
user API level, resulting in a uniform usage of the API for all functions
regardless of the controller being used.

-- 
Damien Le Moal
Western Digital Research

Re: [PATCH v4 7/7] PCI: endpoint: Add PCI Endpoint function driver for MHI bus

[Manivannan Sadhasivam]

On Tue, May 30, 2023 at 08:37:07AM +0900, Damien Le Moal wrote:
> On 5/30/23 02:38, Manivannan Sadhasivam wrote:
> [...]
> >>> +static void pci_epf_mhi_unmap_free(struct mhi_ep_cntrl *mhi_cntrl, u64 pci_addr,
> >>> +				   phys_addr_t phys_addr, void __iomem *virt_addr, size_t size)
> >>> +{
> >>> +	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
> >>> +	struct pci_epf *epf = epf_mhi->epf;
> >>> +	struct pci_epc *epc = epf->epc;
> >>> +	size_t offset = pci_addr & (epc->mem->window.page_size - 1);
> >>> +
> >>> +	pci_epc_unmap_addr(epc, epf->func_no, epf->vfunc_no, phys_addr - offset);
> >>> +	pci_epc_mem_free_addr(epc, phys_addr - offset, virt_addr - offset, size + offset);
> >>> +}
> >>> +
> >>> +static void pci_epf_mhi_raise_irq(struct mhi_ep_cntrl *mhi_cntrl, u32 vector)
> >>> +{
> >>> +	struct pci_epf_mhi *epf_mhi = container_of(mhi_cntrl, struct pci_epf_mhi, mhi_cntrl);
> >>> +	struct pci_epf *epf = epf_mhi->epf;
> >>> +	struct pci_epc *epc = epf->epc;
> >>> +
> >>> +	/*
> >>> +	 * Vector is incremented by 1 here as the DWC core will decrement it before
> >>> +	 * writing to iATU.
> >>
> >> This isn't OK. It is an API, you can't write code explicitly relying on
> >> the underlying implementation. I assume the API is not well specified,
> >> that's why we need these tricks ?
> >>
> > 
> > Well, this is not an API issue but rather an implementation detail of the DWC EP
> > core driver. The DWC driver expects the interrupt vectors to be 1 based, so it
> > decrements it before writing to the MSI address:
> > [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-designware-ep.c#n537
> 
> Then the driver should be fixed so that this peculiarity is not visible at the
> user API level, resulting in a uniform usage of the API for all functions
> regardless of the controller being used.
> 

Just checked with Kishon offline and confirmed that this is an API behavior.
I also checked other drivers and they all doing the same.

But unfortunately, this behavior is not documented in the API. Will add a patch
for that.

- Mani

> -- 
> Damien Le Moal
> Western Digital Research
> 

-- 
மணிவண்ணன் சதாசிவம்