[PATCH v2 0/5] Add RC-to-EP doorbell with platform MSI controller

[PATCH v2 0/5] Add RC-to-EP doorbell with platform MSI controller

[Frank Li]

┌────────────┐   ┌───────────────────────────────────┐   ┌────────────────┐
│            │   │                                   │   │                │
│            │   │ PCI Endpoint                      │   │ PCI Host       │
│            │   │                                   │   │                │
│            │◄──┤ 1.platform_msi_domain_alloc_irqs()│   │                │
│            │   │                                   │   │                │
│ MSI        ├──►│ 2.write_msi_msg()                 ├──►├─BAR<n>         │
│ Controller │   │   update doorbell register address│   │                │
│            │   │   for BAR                         │   │                │
│            │   │                                   │   │ 3. Write BAR<n>│
│            │◄──┼───────────────────────────────────┼───┤                │
│            │   │                                   │   │                │
│            ├──►│ 4.Irq Handle                      │   │                │
│            │   │                                   │   │                │
│            │   │                                   │   │                │
└────────────┘   └───────────────────────────────────┘   └────────────────┘

This patches based on old https://lore.kernel.org/imx/20221124055036.1630573-1-Frank.Li@nxp.com/

Original patch only target to vntb driver. But actually it is common
method.

This patches add new API to pci-epf-core, so any EP driver can use it.

The key point is comments from Thomas Gleixner, who suggest use new
PCI/IMS. But arm platform change still not be merged yet.

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git devmsi-v2-arm

So I still use existed method implement RC to EP doorbell.

If Thomas Gleixner want to continue work on devmsi-v2-arm, I can help test
and update this patch.

Change from v1 to v2
- Add missed patch for endpont/pci-epf-test.c
- Move alloc and free to epc driver from epf.
- Provide general help function for EPC driver to alloc platform msi irq.
- Fixed manivannan's comments.

Frank Li (5):
  PCI: endpoint: Add RC-to-EP doorbell support using platform MSI
    controller
  PCI: dwc: add doorbell support by use MSI controller
  PCI: endpoint: pci-epf-test: add doorbell test
  misc: pci_endpoint_test: Add doorbell test case
  tools: PCI: Add 'B' option for test doorbell

 drivers/misc/pci_endpoint_test.c              |  48 +++++
 .../pci/controller/dwc/pcie-designware-ep.c   |   2 +
 drivers/pci/endpoint/functions/pci-epf-test.c |  59 +++++-
 drivers/pci/endpoint/pci-epc-core.c           | 192 ++++++++++++++++++
 drivers/pci/endpoint/pci-epf-core.c           |  44 ++++
 include/linux/pci-epc.h                       |   6 +
 include/linux/pci-epf.h                       |   7 +
 include/uapi/linux/pcitest.h                  |   1 +
 tools/pci/pcitest.c                           |  16 +-
 9 files changed, 373 insertions(+), 2 deletions(-)

-- 
2.34.1

[PATCH v2 1/5] PCI: endpoint: Add RC-to-EP doorbell support using platform MSI controller

[Frank Li]

This commit introduces a common method for sending messages from the Root
Complex (RC) to the Endpoint (EP) by utilizing the platform MSI interrupt
controller, such as ARM GIC, as an EP doorbell. Maps the memory assigned
for the BAR region by the PCI host to the message address of the platform
MSI interrupt controller in the PCI EP. As a result, when the PCI RC writes
to the BAR region, it triggers an IRQ at the EP. This implementation serves
as a common method for all endpoint function drivers.

However, it currently supports only one EP physical function due to
limitations in ARM MSI/IMS readiness.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
 drivers/pci/endpoint/pci-epc-core.c | 192 ++++++++++++++++++++++++++++
 drivers/pci/endpoint/pci-epf-core.c |  44 +++++++
 include/linux/pci-epc.h             |   6 +
 include/linux/pci-epf.h             |   7 +
 4 files changed, 249 insertions(+)

diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
index 5a4a8b0be6262..d336a99c6a94f 100644
--- a/drivers/pci/endpoint/pci-epc-core.c
+++ b/drivers/pci/endpoint/pci-epc-core.c
@@ -10,6 +10,7 @@
 #include <linux/slab.h>
 #include <linux/module.h>
 
+#include <linux/msi.h>
 #include <linux/pci-epc.h>
 #include <linux/pci-epf.h>
 #include <linux/pci-ep-cfs.h>
@@ -783,6 +784,197 @@ void pci_epc_bme_notify(struct pci_epc *epc)
 }
 EXPORT_SYMBOL_GPL(pci_epc_bme_notify);
 
+/**
+ * pci_epc_alloc_doorbell() - alloc an address space to let RC trigger EP side IRQ by write data to
+ *			      the space.
+ *
+ * @epc: the EPC device that need doorbell address and data from RC.
+ * @func_no: the physical endpoint function number in the EPC device.
+ * @vfunc_no: the virtual endpoint function number in the physical function.
+ * @num_msgs: the total number of doorbell messages
+ *
+ * Return: 0 success, other is failure
+ */
+int pci_epc_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
+{
+	int ret;
+
+	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
+		return -EINVAL;
+
+	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
+		return -EINVAL;
+
+	if (!epc->ops->alloc_doorbell)
+		return 0;
+
+	mutex_lock(&epc->lock);
+	ret = epc->ops->alloc_doorbell(epc, func_no, vfunc_no, num_msgs);
+	mutex_unlock(&epc->lock);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(pci_epc_alloc_doorbell);
+
+/**
+ * pci_epc_free_doorbell() - free resource allocated by pci_epc_alloc_doorbell()
+ *
+ * @epc: the EPC device that need doorbell address and data from RC.
+ * @func_no: the physical endpoint function number in the EPC device.
+ * @vfunc_no: the virtual endpoint function number in the physical function.
+ *
+ * Return: 0 success, other is failure
+ */
+void pci_epc_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
+{
+	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
+		return;
+
+	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
+		return;
+
+	if (!epc->ops->free_doorbell)
+		return;
+
+	mutex_lock(&epc->lock);
+	epc->ops->free_doorbell(epc, func_no, vfunc_no);
+	mutex_unlock(&epc->lock);
+}
+EXPORT_SYMBOL_GPL(pci_epc_free_doorbell);
+
+static irqreturn_t pci_epf_generic_doorbell_handler(int irq, void *data)
+{
+	struct pci_epf *epf = data;
+
+	if (epf->event_ops && epf->event_ops->doorbell)
+		epf->event_ops->doorbell(epf, irq - epf->virq_base);
+
+	return IRQ_HANDLED;
+}
+
+static void pci_epc_generic_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
+{
+	struct pci_epc *epc = NULL;
+	struct class_dev_iter iter;
+	struct pci_epf *epf;
+	struct device *dev;
+
+	class_dev_iter_init(&iter, pci_epc_class, NULL, NULL);
+	while ((dev = class_dev_iter_next(&iter))) {
+		if (dev->parent != desc->dev)
+			continue;
+
+		epc = to_pci_epc(dev);
+
+		class_dev_iter_exit(&iter);
+		break;
+	}
+
+	if (!epc)
+		return;
+
+	/* Only support one EPF for doorbell */
+	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
+
+	if (!epf)
+		return;
+
+	if (epf->msg && desc->msi_index < epf->num_msgs)
+		epf->msg[desc->msi_index] = *msg;
+}
+
+
+/**
+ * pci_epc_generic_alloc_doorbell() - Common help function. Allocate address space from MSI
+ *                                    controller
+ *
+ * @epc: the EPC device that need doorbell address and data from RC.
+ * @func_no: the physical endpoint function number in the EPC device.
+ * @vfunc_no: the virtual endpoint function number in the physical function.
+ * @num_msgs: the total number of doorbell messages
+ *
+ * Remark: use this function only if EPC driver just register one EPC device.
+ *
+ * Return: 0 success, other is failure
+ */
+int pci_epc_generic_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
+{
+	struct pci_epf *epf;
+	struct device *dev;
+	int virq, last;
+	int ret;
+	int i;
+
+	if (IS_ERR_OR_NULL(epc))
+		return -EINVAL;
+
+	/* Currently only support one func and one vfunc for doorbell */
+	if (func_no || vfunc_no)
+		return -EINVAL;
+
+	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
+	if (!epf)
+		return -EINVAL;
+
+	dev = epc->dev.parent;
+	ret = platform_msi_domain_alloc_irqs(dev, num_msgs, pci_epc_generic_write_msi_msg);
+	if (ret) {
+		dev_err(dev, "Failed to allocate MSI\n");
+		return -ENOMEM;
+	}
+
+	last = -1;
+	for (i = 0; i < num_msgs; i++) {
+		virq = msi_get_virq(dev, i);
+		if (i == 0)
+			epf->virq_base = virq;
+
+		ret = request_irq(virq, pci_epf_generic_doorbell_handler, 0,
+				  kasprintf(GFP_KERNEL, "pci-epc-doorbell%d", i), epf);
+
+		if (ret) {
+			dev_err(dev, "Failed to request doorbell\n");
+			goto err_free_irq;
+		}
+		last = i;
+	}
+
+	return 0;
+
+err_free_irq:
+	for (i = 0; i < last; i++)
+		kfree(free_irq(epf->virq_base + i, epf));
+	platform_msi_domain_free_irqs(dev);
+
+	return -EINVAL;
+}
+EXPORT_SYMBOL_GPL(pci_epc_generic_alloc_doorbell);
+
+/**
+ * pci_epc_generic_free_doorbell() - Common help function. Free resource created by
+ *				     pci_epc_generic_alloc_doorbell()
+ *
+ * @epc: the EPC device that need doorbell address and data from RC.
+ * @func_no: the physical endpoint function number in the EPC device.
+ * @vfunc_no: the virtual endpoint function number in the physical function.
+ * @num_msgs: the total number of doorbell messages
+ */
+void pci_epc_generic_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
+{
+	struct pci_epf *epf;
+	int i;
+
+	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
+	if (!epf)
+		return;
+
+	for (i = 0; i < epf->num_msgs; i++)
+		kfree(free_irq(epf->virq_base + i, epf));
+
+	platform_msi_domain_free_irqs(epc->dev.parent);
+}
+EXPORT_SYMBOL_GPL(pci_epc_generic_free_doorbell);
+
 /**
  * pci_epc_destroy() - destroy the EPC device
  * @epc: the EPC device that has to be destroyed
diff --git a/drivers/pci/endpoint/pci-epf-core.c b/drivers/pci/endpoint/pci-epf-core.c
index 2c32de6679377..eab82c6b0119a 100644
--- a/drivers/pci/endpoint/pci-epf-core.c
+++ b/drivers/pci/endpoint/pci-epf-core.c
@@ -10,6 +10,7 @@
 #include <linux/dma-mapping.h>
 #include <linux/slab.h>
 #include <linux/module.h>
+#include <linux/msi.h>
 
 #include <linux/pci-epc.h>
 #include <linux/pci-epf.h>
@@ -300,6 +301,49 @@ void *pci_epf_alloc_space(struct pci_epf *epf, size_t size, enum pci_barno bar,
 }
 EXPORT_SYMBOL_GPL(pci_epf_alloc_space);
 
+int pci_epf_alloc_doorbell(struct pci_epf *epf, u16 num_msgs)
+{
+	struct pci_epc *epc;
+	struct device *dev;
+	int ret;
+
+	epc = epf->epc;
+	dev = &epc->dev;
+
+	epf->msg = kcalloc(num_msgs, sizeof(struct msi_msg), GFP_KERNEL);
+	if (!epf->msg)
+		return -ENOMEM;
+
+	epf->num_msgs = num_msgs;
+
+	ret = pci_epc_alloc_doorbell(epc, epf->func_no, epf->vfunc_no, num_msgs);
+	if (ret)
+		goto err_free_mem;
+
+	return ret;
+
+err_free_mem:
+	kfree(epf->msg);
+	epf->msg = NULL;
+	epf->num_msgs = 0;
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(pci_epf_alloc_doorbell);
+
+void pci_epf_free_doorbell(struct pci_epf *epf)
+{
+	struct pci_epc *epc;
+
+	epc = epf->epc;
+	pci_epc_free_doorbell(epc, epf->func_no, epf->vfunc_no);
+
+	kfree(epf->msg);
+	epf->msg = NULL;
+	epf->num_msgs = 0;
+}
+EXPORT_SYMBOL_GPL(pci_epf_free_doorbell);
+
 static void pci_epf_remove_cfs(struct pci_epf_driver *driver)
 {
 	struct config_group *group, *tmp;
diff --git a/include/linux/pci-epc.h b/include/linux/pci-epc.h
index 5cb6940310729..605fb0debd6bc 100644
--- a/include/linux/pci-epc.h
+++ b/include/linux/pci-epc.h
@@ -88,6 +88,8 @@ struct pci_epc_ops {
 	void	(*stop)(struct pci_epc *epc);
 	const struct pci_epc_features* (*get_features)(struct pci_epc *epc,
 						       u8 func_no, u8 vfunc_no);
+	int	(*alloc_doorbell)(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs);
+	void	(*free_doorbell)(struct pci_epc *epc, u8 func_no, u8 vfunc_no);
 	struct module *owner;
 };
 
@@ -251,4 +253,8 @@ void __iomem *pci_epc_mem_alloc_addr(struct pci_epc *epc,
 				     phys_addr_t *phys_addr, size_t size);
 void pci_epc_mem_free_addr(struct pci_epc *epc, phys_addr_t phys_addr,
 			   void __iomem *virt_addr, size_t size);
+int pci_epc_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs);
+void pci_epc_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no);
+int pci_epc_generic_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs);
+void pci_epc_generic_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no);
 #endif /* __LINUX_PCI_EPC_H */
diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
index 3f44b6aec4770..485c146a5efe2 100644
--- a/include/linux/pci-epf.h
+++ b/include/linux/pci-epf.h
@@ -79,6 +79,7 @@ struct pci_epc_event_ops {
 	int (*link_up)(struct pci_epf *epf);
 	int (*link_down)(struct pci_epf *epf);
 	int (*bme)(struct pci_epf *epf);
+	int (*doorbell)(struct pci_epf *epf, int index);
 };
 
 /**
@@ -180,6 +181,9 @@ struct pci_epf {
 	unsigned long		vfunction_num_map;
 	struct list_head	pci_vepf;
 	const struct pci_epc_event_ops *event_ops;
+	struct msi_msg *msg;
+	u16 num_msgs;
+	int virq_base;
 };
 
 /**
@@ -223,4 +227,7 @@ int pci_epf_bind(struct pci_epf *epf);
 void pci_epf_unbind(struct pci_epf *epf);
 int pci_epf_add_vepf(struct pci_epf *epf_pf, struct pci_epf *epf_vf);
 void pci_epf_remove_vepf(struct pci_epf *epf_pf, struct pci_epf *epf_vf);
+int pci_epf_alloc_doorbell(struct pci_epf *epf, u16 nums);
+void pci_epf_free_doorbell(struct pci_epf *epf);
+
 #endif /* __LINUX_PCI_EPF_H */
-- 
2.34.1

[PATCH v2 2/5] PCI: dwc: add doorbell support by use MSI controller

[Frank Li]

Call pci_epc_generic_alloc_doorbell() to alloc an platform MSI address
space.The Endpoint (EP) function drver can maps the memory assigned for the
BAR region by the PCI host to the message address of the platform MSI
interrupt controller in the PCI EP. As a result, when the PCI RC writes to
the BAR region, it triggers an IRQ at the EP.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
 drivers/pci/controller/dwc/pcie-designware-ep.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index f9182f8d552f4..4f6be5f4ba18a 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -477,6 +477,8 @@ static const struct pci_epc_ops epc_ops = {
 	.start			= dw_pcie_ep_start,
 	.stop			= dw_pcie_ep_stop,
 	.get_features		= dw_pcie_ep_get_features,
+	.alloc_doorbell		= pci_epc_generic_alloc_doorbell,
+	.free_doorbell		= pci_epc_generic_free_doorbell,
 };
 
 int dw_pcie_ep_raise_legacy_irq(struct dw_pcie_ep *ep, u8 func_no)
-- 
2.34.1

[PATCH v2 3/5] PCI: endpoint: pci-epf-test: add doorbell test

[Frank Li]

Add three register: doorbell_bar, doorbell_addr, doorbell_data,
doorbell_done. Call pci_epf_alloc_doorbell() all a doorbell address space.

Root complex(RC) side driver can trigger pci-epc-test's doorbell callback
handler by write doorbell_data to mapped doorbell_bar's address space.

pci-epc-test will set doorbell_done in doorbell callback.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
 drivers/pci/endpoint/functions/pci-epf-test.c | 59 ++++++++++++++++++-
 1 file changed, 58 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c b/drivers/pci/endpoint/functions/pci-epf-test.c
index 1f0d2b84296a3..566549919b87b 100644
--- a/drivers/pci/endpoint/functions/pci-epf-test.c
+++ b/drivers/pci/endpoint/functions/pci-epf-test.c
@@ -11,6 +11,7 @@
 #include <linux/dmaengine.h>
 #include <linux/io.h>
 #include <linux/module.h>
+#include <linux/msi.h>
 #include <linux/slab.h>
 #include <linux/pci_ids.h>
 #include <linux/random.h>
@@ -39,17 +40,21 @@
 #define STATUS_IRQ_RAISED		BIT(6)
 #define STATUS_SRC_ADDR_INVALID		BIT(7)
 #define STATUS_DST_ADDR_INVALID		BIT(8)
+#define STATUS_DOORBELL_SUCCESS		BIT(9)
 
 #define FLAG_USE_DMA			BIT(0)
 
 #define TIMER_RESOLUTION		1
 
+#define MAGIC_VERSION_MASK		GENMASK(7, 0)
+
 static struct workqueue_struct *kpcitest_workqueue;
 
 struct pci_epf_test {
 	void			*reg[PCI_STD_NUM_BARS];
 	struct pci_epf		*epf;
 	enum pci_barno		test_reg_bar;
+	enum pci_barno		doorbell_bar;
 	size_t			msix_table_offset;
 	struct delayed_work	cmd_handler;
 	struct dma_chan		*dma_chan_tx;
@@ -74,6 +79,9 @@ struct pci_epf_test_reg {
 	u32	irq_type;
 	u32	irq_number;
 	u32	flags;
+	u32	doorbell_bar;
+	u32	doorbell_addr;
+	u32	doorbell_data;
 } __packed;
 
 static struct pci_epf_header test_header = {
@@ -693,6 +701,8 @@ static void pci_epf_test_unbind(struct pci_epf *epf)
 	struct pci_epf_bar *epf_bar;
 	int bar;
 
+	pci_epf_free_doorbell(epf);
+
 	cancel_delayed_work(&epf_test->cmd_handler);
 	pci_epf_test_clean_dma_chan(epf_test);
 	for (bar = 0; bar < PCI_STD_NUM_BARS; bar++) {
@@ -808,9 +818,22 @@ static int pci_epf_test_link_up(struct pci_epf *epf)
 	return 0;
 }
 
+static int pci_epf_test_doorbell(struct pci_epf *epf, int index)
+{
+	struct pci_epf_test *epf_test = epf_get_drvdata(epf);
+	enum pci_barno test_reg_bar = epf_test->test_reg_bar;
+	struct pci_epf_test_reg *reg = epf_test->reg[test_reg_bar];
+
+	reg->status |= STATUS_DOORBELL_SUCCESS;
+	pci_epf_test_raise_irq(epf_test, reg);
+
+	return 0;
+}
+
 static const struct pci_epc_event_ops pci_epf_test_event_ops = {
 	.core_init = pci_epf_test_core_init,
 	.link_up = pci_epf_test_link_up,
+	.doorbell = pci_epf_test_doorbell,
 };
 
 static int pci_epf_test_alloc_space(struct pci_epf *epf)
@@ -859,7 +882,7 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf)
 		epf_bar = &epf->bar[bar];
 		add = (epf_bar->flags & PCI_BASE_ADDRESS_MEM_TYPE_64) ? 2 : 1;
 
-		if (bar == test_reg_bar)
+		if (bar == test_reg_bar || bar == epf_test->doorbell_bar)
 			continue;
 
 		if (!!(epc_features->reserved_bar & (1 << bar)))
@@ -900,9 +923,14 @@ static int pci_epf_test_bind(struct pci_epf *epf)
 	struct pci_epf_test *epf_test = epf_get_drvdata(epf);
 	const struct pci_epc_features *epc_features;
 	enum pci_barno test_reg_bar = BAR_0;
+	enum pci_barno doorbell_bar = NO_BAR;
 	struct pci_epc *epc = epf->epc;
 	bool linkup_notifier = false;
 	bool core_init_notifier = false;
+	struct pci_epf_test_reg *reg;
+	struct msi_msg *msg;
+	u64 doorbell_addr;
+	u32 align;
 
 	if (WARN_ON_ONCE(!epc))
 		return -EINVAL;
@@ -923,10 +951,39 @@ static int pci_epf_test_bind(struct pci_epf *epf)
 	epf_test->test_reg_bar = test_reg_bar;
 	epf_test->epc_features = epc_features;
 
+	align = epc_features->align;
+	align = align ? align : 128;
+
+	ret = pci_epf_alloc_doorbell(epf, 1);
+	if (!ret) {
+		msg = epf->msg;
+		doorbell_bar = pci_epc_get_next_free_bar(epc_features, test_reg_bar + 1);
+
+		if (doorbell_bar > 0) {
+			epf_test->doorbell_bar = doorbell_bar;
+			doorbell_addr = msg->address_hi;
+			doorbell_addr <<= 32;
+			doorbell_addr |= msg->address_lo;
+			epf->bar[doorbell_bar].phys_addr = round_down(doorbell_addr, align);
+			epf->bar[doorbell_bar].barno = doorbell_bar;
+			epf->bar[doorbell_bar].size = align;
+		} else {
+			pci_epf_free_doorbell(epf);
+		}
+	}
+
 	ret = pci_epf_test_alloc_space(epf);
 	if (ret)
 		return ret;
 
+	reg = epf_test->reg[test_reg_bar];
+	reg->magic |= FIELD_PREP(MAGIC_VERSION_MASK, 0x1);
+	if (doorbell_bar > 0) {
+		reg->doorbell_addr = doorbell_addr & (align - 1);
+		reg->doorbell_data = msg->data;
+		reg->doorbell_bar = doorbell_bar;
+	}
+
 	if (!core_init_notifier) {
 		ret = pci_epf_test_core_init(epf);
 		if (ret)
-- 
2.34.1

[PATCH v2 4/5] misc: pci_endpoint_test: Add doorbell test case

[Frank Li]

Using bit 0..7 of magic as version number in pci_endpoint_test struct to
support older driver versions. Save to 'version' field of struct
pci_endpoint_test to prevent reading non-existent address.

Add three registers: PCIE_ENDPOINT_TEST_DB_BAR, PCIE_ENDPOINT_TEST_DB_ADDR,
PCIE_ENDPOINT_TEST_DB_DATA.

Write data from PCI_ENDPOINT_TEST_DB_DATA to address from
PCI_ENDPOINT_TEST_DB_ADDR to trigger doorbell and wait for endpoint
feedback.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
 drivers/misc/pci_endpoint_test.c | 48 ++++++++++++++++++++++++++++++++
 include/uapi/linux/pcitest.h     |  1 +
 2 files changed, 49 insertions(+)

diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
index ed4d0ef5e5c31..ed0b025132d17 100644
--- a/drivers/misc/pci_endpoint_test.c
+++ b/drivers/misc/pci_endpoint_test.c
@@ -33,6 +33,8 @@
 #define IRQ_TYPE_MSIX				2
 
 #define PCI_ENDPOINT_TEST_MAGIC			0x0
+#define PCI_MAGIC_VERSION_MASK			GENMASK(7, 0)
+#define PCI_ENDPOINT_TEST_V1			0x1
 
 #define PCI_ENDPOINT_TEST_COMMAND		0x4
 #define COMMAND_RAISE_LEGACY_IRQ		BIT(0)
@@ -52,6 +54,7 @@
 #define STATUS_IRQ_RAISED			BIT(6)
 #define STATUS_SRC_ADDR_INVALID			BIT(7)
 #define STATUS_DST_ADDR_INVALID			BIT(8)
+#define STATUS_DOORBELL_SUCCESS			BIT(9)
 
 #define PCI_ENDPOINT_TEST_LOWER_SRC_ADDR	0x0c
 #define PCI_ENDPOINT_TEST_UPPER_SRC_ADDR	0x10
@@ -66,7 +69,12 @@
 #define PCI_ENDPOINT_TEST_IRQ_NUMBER		0x28
 
 #define PCI_ENDPOINT_TEST_FLAGS			0x2c
+#define PCI_ENDPOINT_TEST_DB_BAR		0x30
+#define PCI_ENDPOINT_TEST_DB_ADDR		0x34
+#define PCI_ENDPOINT_TEST_DB_DATA		0x38
+
 #define FLAG_USE_DMA				BIT(0)
+#define FLAG_SUPPORT_DOORBELL			BIT(1)
 
 #define PCI_DEVICE_ID_TI_AM654			0xb00c
 #define PCI_DEVICE_ID_TI_J7200			0xb00f
@@ -102,6 +110,7 @@ enum pci_barno {
 	BAR_3,
 	BAR_4,
 	BAR_5,
+	NO_BAR = -1,
 };
 
 struct pci_endpoint_test {
@@ -118,6 +127,7 @@ struct pci_endpoint_test {
 	enum pci_barno test_reg_bar;
 	size_t alignment;
 	const char *name;
+	u8 version;
 };
 
 struct pci_endpoint_test_data {
@@ -713,6 +723,38 @@ static bool pci_endpoint_test_set_irq(struct pci_endpoint_test *test,
 	return false;
 }
 
+static bool pci_endpoint_test_doorbell(struct pci_endpoint_test *test)
+{
+	enum pci_barno bar;
+	u32 data, status;
+	u32 addr;
+
+	if (test->version < PCI_ENDPOINT_TEST_V1)
+		return false;
+
+	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
+	if (bar == NO_BAR)
+		return false;
+
+	data = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_DATA);
+	addr = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_ADDR);
+	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
+
+	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_TYPE, irq_type);
+	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_NUMBER, 1);
+
+	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_STATUS, 0);
+	pci_endpoint_test_bar_writel(test, bar, addr, data);
+
+	wait_for_completion_timeout(&test->irq_raised, msecs_to_jiffies(1000));
+
+	status = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_STATUS);
+	if (status & STATUS_DOORBELL_SUCCESS)
+		return true;
+
+	return false;
+}
+
 static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
 				    unsigned long arg)
 {
@@ -760,6 +802,9 @@ static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
 	case PCITEST_CLEAR_IRQ:
 		ret = pci_endpoint_test_clear_irq(test);
 		break;
+	case PCITEST_DOORBELL:
+		ret = pci_endpoint_test_doorbell(test);
+		break;
 	}
 
 ret:
@@ -887,6 +932,9 @@ static int pci_endpoint_test_probe(struct pci_dev *pdev,
 	misc_device->parent = &pdev->dev;
 	misc_device->fops = &pci_endpoint_test_fops;
 
+	test->version = FIELD_GET(PCI_MAGIC_VERSION_MASK,
+				  pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_MAGIC));
+
 	err = misc_register(misc_device);
 	if (err) {
 		dev_err(dev, "Failed to register device\n");
diff --git a/include/uapi/linux/pcitest.h b/include/uapi/linux/pcitest.h
index f9c1af8d141b4..479ca1aa3ae0b 100644
--- a/include/uapi/linux/pcitest.h
+++ b/include/uapi/linux/pcitest.h
@@ -20,6 +20,7 @@
 #define PCITEST_SET_IRQTYPE	_IOW('P', 0x8, int)
 #define PCITEST_GET_IRQTYPE	_IO('P', 0x9)
 #define PCITEST_CLEAR_IRQ	_IO('P', 0x10)
+#define PCITEST_DOORBELL	_IO('P', 0x11)
 
 #define PCITEST_FLAGS_USE_DMA	0x00000001
 
-- 
2.34.1

[PATCH v2 5/5] tools: PCI: Add 'B' option for test doorbell

[Frank Li]

Add doorbell test support.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
 tools/pci/pcitest.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/tools/pci/pcitest.c b/tools/pci/pcitest.c
index 441b542346354..215d0aa8a09fe 100644
--- a/tools/pci/pcitest.c
+++ b/tools/pci/pcitest.c
@@ -36,6 +36,7 @@ struct pci_test {
 	bool		copy;
 	unsigned long	size;
 	bool		use_dma;
+	bool		doorbell;
 };
 
 static int run_test(struct pci_test *test)
@@ -149,6 +150,15 @@ static int run_test(struct pci_test *test)
 			fprintf(stdout, "%s\n", result[ret]);
 	}
 
+	if (test->doorbell) {
+		ret = ioctl(fd, PCITEST_DOORBELL, 0);
+		fprintf(stdout, "Push doorbell\t\t");
+		if (ret < 0)
+			fprintf(stdout, "TEST FAILED\n");
+		else
+			fprintf(stdout, "%s\n", result[ret]);
+	}
+
 	fflush(stdout);
 	close(fd);
 	return (ret < 0) ? ret : 1 - ret; /* return 0 if test succeeded */
@@ -174,7 +184,7 @@ int main(int argc, char **argv)
 	/* set default endpoint device */
 	test->device = "/dev/pci-endpoint-test.0";
 
-	while ((c = getopt(argc, argv, "D:b:m:x:i:deIlhrwcs:")) != EOF)
+	while ((c = getopt(argc, argv, "D:b:m:x:i:BdeIlhrwcs:")) != EOF)
 	switch (c) {
 	case 'D':
 		test->device = optarg;
@@ -224,6 +234,9 @@ int main(int argc, char **argv)
 	case 'd':
 		test->use_dma = true;
 		continue;
+	case 'B':
+		test->doorbell = true;
+		continue;
 	case 'h':
 	default:
 usage:
@@ -243,6 +256,7 @@ int main(int argc, char **argv)
 			"\t-w			Write buffer test\n"
 			"\t-c			Copy buffer test\n"
 			"\t-s <size>		Size of buffer {default: 100KB}\n"
+			"\t-B			Doorbell test\n"
 			"\t-h			Print this help message\n",
 			argv[0]);
 		return -EINVAL;
-- 
2.34.1

Re: [PATCH v2 0/5] Add RC-to-EP doorbell with platform MSI controller

[Frank Li]

On Mon, Sep 11, 2023 at 06:09:15PM -0400, Frank Li wrote:
> ┌────────────┐   ┌───────────────────────────────────┐   ┌────────────────┐
> │            │   │                                   │   │                │
> │            │   │ PCI Endpoint                      │   │ PCI Host       │
> │            │   │                                   │   │                │
> │            │◄──┤ 1.platform_msi_domain_alloc_irqs()│   │                │
> │            │   │                                   │   │                │
> │ MSI        ├──►│ 2.write_msi_msg()                 ├──►├─BAR<n>         │
> │ Controller │   │   update doorbell register address│   │                │
> │            │   │   for BAR                         │   │                │
> │            │   │                                   │   │ 3. Write BAR<n>│
> │            │◄──┼───────────────────────────────────┼───┤                │
> │            │   │                                   │   │                │
> │            ├──►│ 4.Irq Handle                      │   │                │
> │            │   │                                   │   │                │
> │            │   │                                   │   │                │
> └────────────┘   └───────────────────────────────────┘   └────────────────┘

@mani:
	Do you have chance to review this patch again?

Frank

> 
> This patches based on old https://lore.kernel.org/imx/20221124055036.1630573-1-Frank.Li@nxp.com/
> 
> Original patch only target to vntb driver. But actually it is common
> method.
> 
> This patches add new API to pci-epf-core, so any EP driver can use it.
> 
> The key point is comments from Thomas Gleixner, who suggest use new
> PCI/IMS. But arm platform change still not be merged yet.
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git devmsi-v2-arm
> 
> So I still use existed method implement RC to EP doorbell.
> 
> If Thomas Gleixner want to continue work on devmsi-v2-arm, I can help test
> and update this patch.
> 
> Change from v1 to v2
> - Add missed patch for endpont/pci-epf-test.c
> - Move alloc and free to epc driver from epf.
> - Provide general help function for EPC driver to alloc platform msi irq.
> - Fixed manivannan's comments.
> 
> Frank Li (5):
>   PCI: endpoint: Add RC-to-EP doorbell support using platform MSI
>     controller
>   PCI: dwc: add doorbell support by use MSI controller
>   PCI: endpoint: pci-epf-test: add doorbell test
>   misc: pci_endpoint_test: Add doorbell test case
>   tools: PCI: Add 'B' option for test doorbell
> 
>  drivers/misc/pci_endpoint_test.c              |  48 +++++
>  .../pci/controller/dwc/pcie-designware-ep.c   |   2 +
>  drivers/pci/endpoint/functions/pci-epf-test.c |  59 +++++-
>  drivers/pci/endpoint/pci-epc-core.c           | 192 ++++++++++++++++++
>  drivers/pci/endpoint/pci-epf-core.c           |  44 ++++
>  include/linux/pci-epc.h                       |   6 +
>  include/linux/pci-epf.h                       |   7 +
>  include/uapi/linux/pcitest.h                  |   1 +
>  tools/pci/pcitest.c                           |  16 +-
>  9 files changed, 373 insertions(+), 2 deletions(-)
> 
> -- 
> 2.34.1
> 

Re: [PATCH v2 1/5] PCI: endpoint: Add RC-to-EP doorbell support using platform MSI controller

[Kishon Vijay Abraham I]

Hi Frank,

On 9/12/2023 3:39 AM, Frank Li wrote:
> This commit introduces a common method for sending messages from the Root
> Complex (RC) to the Endpoint (EP) by utilizing the platform MSI interrupt
> controller, such as ARM GIC, as an EP doorbell. Maps the memory assigned
> for the BAR region by the PCI host to the message address of the platform
> MSI interrupt controller in the PCI EP. As a result, when the PCI RC writes
> to the BAR region, it triggers an IRQ at the EP. This implementation serves
> as a common method for all endpoint function drivers.

This would help avoid the polling used in current EPF drivers. Thanks!
> 
> However, it currently supports only one EP physical function due to
> limitations in ARM MSI/IMS readiness.

Any such platform or architecture restrictions should not be handled in 
the endpoint core layer.
> 
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---
>   drivers/pci/endpoint/pci-epc-core.c | 192 ++++++++++++++++++++++++++++
>   drivers/pci/endpoint/pci-epf-core.c |  44 +++++++
>   include/linux/pci-epc.h             |   6 +
>   include/linux/pci-epf.h             |   7 +
>   4 files changed, 249 insertions(+)
> 
> diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
> index 5a4a8b0be6262..d336a99c6a94f 100644
> --- a/drivers/pci/endpoint/pci-epc-core.c
> +++ b/drivers/pci/endpoint/pci-epc-core.c
> @@ -10,6 +10,7 @@
>   #include <linux/slab.h>
>   #include <linux/module.h>
>   
> +#include <linux/msi.h>
>   #include <linux/pci-epc.h>
>   #include <linux/pci-epf.h>
>   #include <linux/pci-ep-cfs.h>
> @@ -783,6 +784,197 @@ void pci_epc_bme_notify(struct pci_epc *epc)
>   }
>   EXPORT_SYMBOL_GPL(pci_epc_bme_notify);
>   
> +/**
> + * pci_epc_alloc_doorbell() - alloc an address space to let RC trigger EP side IRQ by write data to
> + *			      the space.
> + *
> + * @epc: the EPC device that need doorbell address and data from RC.
> + * @func_no: the physical endpoint function number in the EPC device.
> + * @vfunc_no: the virtual endpoint function number in the physical function.
> + * @num_msgs: the total number of doorbell messages
> + *
> + * Return: 0 success, other is failure
> + */
> +int pci_epc_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> +{
> +	int ret;
> +
> +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> +		return -EINVAL;
> +
> +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> +		return -EINVAL;
> +
> +	if (!epc->ops->alloc_doorbell)
> +		return 0;
> +
> +	mutex_lock(&epc->lock);
> +	ret = epc->ops->alloc_doorbell(epc, func_no, vfunc_no, num_msgs);
> +	mutex_unlock(&epc->lock);
> +
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(pci_epc_alloc_doorbell);
> +
> +/**
> + * pci_epc_free_doorbell() - free resource allocated by pci_epc_alloc_doorbell()
> + *
> + * @epc: the EPC device that need doorbell address and data from RC.
> + * @func_no: the physical endpoint function number in the EPC device.
> + * @vfunc_no: the virtual endpoint function number in the physical function.
> + *
> + * Return: 0 success, other is failure
> + */
> +void pci_epc_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
> +{
> +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> +		return;
> +
> +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> +		return;
> +
> +	if (!epc->ops->free_doorbell)
> +		return;
> +
> +	mutex_lock(&epc->lock);
> +	epc->ops->free_doorbell(epc, func_no, vfunc_no);
> +	mutex_unlock(&epc->lock);
> +}
> +EXPORT_SYMBOL_GPL(pci_epc_free_doorbell);
> +
> +static irqreturn_t pci_epf_generic_doorbell_handler(int irq, void *data)
> +{
> +	struct pci_epf *epf = data;
> +
> +	if (epf->event_ops && epf->event_ops->doorbell)
> +		epf->event_ops->doorbell(epf, irq - epf->virq_base);
> +
> +	return IRQ_HANDLED;
> +}

IMO the handler should be directly implemented in the EPF drivers. There 
should be one API which returns the virq and the msi_msg to the EPF 
driver and the EPF driver should do request_irq.
> +
> +static void pci_epc_generic_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
> +{
> +	struct pci_epc *epc = NULL;
> +	struct class_dev_iter iter;
> +	struct pci_epf *epf;
> +	struct device *dev;
> +
> +	class_dev_iter_init(&iter, pci_epc_class, NULL, NULL);
> +	while ((dev = class_dev_iter_next(&iter))) {
> +		if (dev->parent != desc->dev)
> +			continue;

Ideally the msi_desc should be associated directly with the EPF device.
> +
> +		epc = to_pci_epc(dev);
> +
> +		class_dev_iter_exit(&iter);
> +		break;
> +	}
> +
> +	if (!epc)
> +		return;
> +
> +	/* Only support one EPF for doorbell */
> +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> +
> +	if (!epf)
> +		return;

If this is a platform restriction, this should be moved elsewhere.
> +
> +	if (epf->msg && desc->msi_index < epf->num_msgs)
> +		epf->msg[desc->msi_index] = *msg;
> +}
> +
> +
> +/**
> + * pci_epc_generic_alloc_doorbell() - Common help function. Allocate address space from MSI
> + *                                    controller
> + *
> + * @epc: the EPC device that need doorbell address and data from RC.
> + * @func_no: the physical endpoint function number in the EPC device.
> + * @vfunc_no: the virtual endpoint function number in the physical function.
> + * @num_msgs: the total number of doorbell messages
> + *
> + * Remark: use this function only if EPC driver just register one EPC device.
> + *
> + * Return: 0 success, other is failure
> + */
> +int pci_epc_generic_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> +{
> +	struct pci_epf *epf;
> +	struct device *dev;
> +	int virq, last;
> +	int ret;
> +	int i;
> +
> +	if (IS_ERR_OR_NULL(epc))
> +		return -EINVAL;
> +
> +	/* Currently only support one func and one vfunc for doorbell */
> +	if (func_no || vfunc_no)
> +		return -EINVAL;
> +
> +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> +	if (!epf)
> +		return -EINVAL;
> +
> +	dev = epc->dev.parent;
> +	ret = platform_msi_domain_alloc_irqs(dev, num_msgs, pci_epc_generic_write_msi_msg);
> +	if (ret) {
> +		dev_err(dev, "Failed to allocate MSI\n");
> +		return -ENOMEM;
> +	}

The alloc_irqs should be for a EPF device IMO.
> +
> +	last = -1;
> +	for (i = 0; i < num_msgs; i++) {
> +		virq = msi_get_virq(dev, i);
> +		if (i == 0)
> +			epf->virq_base = virq;
> +
> +		ret = request_irq(virq, pci_epf_generic_doorbell_handler, 0,
> +				  kasprintf(GFP_KERNEL, "pci-epc-doorbell%d", i), epf);
> +
> +		if (ret) {
> +			dev_err(dev, "Failed to request doorbell\n");
> +			goto err_free_irq;
> +		}
> +		last = i;
> +	}
> +
> +	return 0;
> +
> +err_free_irq:
> +	for (i = 0; i < last; i++)
> +		kfree(free_irq(epf->virq_base + i, epf));
> +	platform_msi_domain_free_irqs(dev);
> +
> +	return -EINVAL;
> +}

Thanks,
Kishon

Re: [PATCH v2 3/5] PCI: endpoint: pci-epf-test: add doorbell test

[Kishon Vijay Abraham I]

Hi Frank,

On 9/12/2023 3:39 AM, Frank Li wrote:
> Add three register: doorbell_bar, doorbell_addr, doorbell_data,
> doorbell_done. Call pci_epf_alloc_doorbell() all a doorbell address space.
> 
> Root complex(RC) side driver can trigger pci-epc-test's doorbell callback
> handler by write doorbell_data to mapped doorbell_bar's address space.
> 
> pci-epc-test will set doorbell_done in doorbell callback.
> 
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---
>   drivers/pci/endpoint/functions/pci-epf-test.c | 59 ++++++++++++++++++-
>   1 file changed, 58 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c b/drivers/pci/endpoint/functions/pci-epf-test.c
> index 1f0d2b84296a3..566549919b87b 100644
> --- a/drivers/pci/endpoint/functions/pci-epf-test.c
> +++ b/drivers/pci/endpoint/functions/pci-epf-test.c
> @@ -11,6 +11,7 @@
>   #include <linux/dmaengine.h>
>   #include <linux/io.h>
>   #include <linux/module.h>
> +#include <linux/msi.h>
>   #include <linux/slab.h>
>   #include <linux/pci_ids.h>
>   #include <linux/random.h>
> @@ -39,17 +40,21 @@
>   #define STATUS_IRQ_RAISED		BIT(6)
>   #define STATUS_SRC_ADDR_INVALID		BIT(7)
>   #define STATUS_DST_ADDR_INVALID		BIT(8)
> +#define STATUS_DOORBELL_SUCCESS		BIT(9)
>   
>   #define FLAG_USE_DMA			BIT(0)
>   
>   #define TIMER_RESOLUTION		1
>   
> +#define MAGIC_VERSION_MASK		GENMASK(7, 0)
> +
>   static struct workqueue_struct *kpcitest_workqueue;
>   
>   struct pci_epf_test {
>   	void			*reg[PCI_STD_NUM_BARS];
>   	struct pci_epf		*epf;
>   	enum pci_barno		test_reg_bar;
> +	enum pci_barno		doorbell_bar;
>   	size_t			msix_table_offset;
>   	struct delayed_work	cmd_handler;
>   	struct dma_chan		*dma_chan_tx;
> @@ -74,6 +79,9 @@ struct pci_epf_test_reg {
>   	u32	irq_type;
>   	u32	irq_number;
>   	u32	flags;
> +	u32	doorbell_bar;

You could extend test_reg_bar for doorbell to avoid using additional BAR.


> +	u32	doorbell_addr;
> +	u32	doorbell_data;
>   } __packed;
>   

Thanks,
Kishon

Re: [PATCH v2 3/5] PCI: endpoint: pci-epf-test: add doorbell test

[Frank Li]

On Fri, Sep 29, 2023 at 03:03:35PM +0530, Kishon Vijay Abraham I wrote:
> Hi Frank,
> 
> On 9/12/2023 3:39 AM, Frank Li wrote:
> > Add three register: doorbell_bar, doorbell_addr, doorbell_data,
> > doorbell_done. Call pci_epf_alloc_doorbell() all a doorbell address space.
> > 
> > Root complex(RC) side driver can trigger pci-epc-test's doorbell callback
> > handler by write doorbell_data to mapped doorbell_bar's address space.
> > 
> > pci-epc-test will set doorbell_done in doorbell callback.
> > 
> > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > ---
> >   drivers/pci/endpoint/functions/pci-epf-test.c | 59 ++++++++++++++++++-
> >   1 file changed, 58 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c b/drivers/pci/endpoint/functions/pci-epf-test.c
> > index 1f0d2b84296a3..566549919b87b 100644
> > --- a/drivers/pci/endpoint/functions/pci-epf-test.c
> > +++ b/drivers/pci/endpoint/functions/pci-epf-test.c
> > @@ -11,6 +11,7 @@
> >   #include <linux/dmaengine.h>
> >   #include <linux/io.h>
> >   #include <linux/module.h>
> > +#include <linux/msi.h>
> >   #include <linux/slab.h>
> >   #include <linux/pci_ids.h>
> >   #include <linux/random.h>
> > @@ -39,17 +40,21 @@
> >   #define STATUS_IRQ_RAISED		BIT(6)
> >   #define STATUS_SRC_ADDR_INVALID		BIT(7)
> >   #define STATUS_DST_ADDR_INVALID		BIT(8)
> > +#define STATUS_DOORBELL_SUCCESS		BIT(9)
> >   #define FLAG_USE_DMA			BIT(0)
> >   #define TIMER_RESOLUTION		1
> > +#define MAGIC_VERSION_MASK		GENMASK(7, 0)
> > +
> >   static struct workqueue_struct *kpcitest_workqueue;
> >   struct pci_epf_test {
> >   	void			*reg[PCI_STD_NUM_BARS];
> >   	struct pci_epf		*epf;
> >   	enum pci_barno		test_reg_bar;
> > +	enum pci_barno		doorbell_bar;
> >   	size_t			msix_table_offset;
> >   	struct delayed_work	cmd_handler;
> >   	struct dma_chan		*dma_chan_tx;
> > @@ -74,6 +79,9 @@ struct pci_epf_test_reg {
> >   	u32	irq_type;
> >   	u32	irq_number;
> >   	u32	flags;
> > +	u32	doorbell_bar;
> 
> You could extend test_reg_bar for doorbell to avoid using additional BAR.

It is sperated physical address space. So far, epc still not support map
two difference physcial address space into a bar.

So I have to use a sperate pci bar for doorbell.

Frank

> 
> 
> > +	u32	doorbell_addr;
> > +	u32	doorbell_data;
> >   } __packed;
> 
> Thanks,
> Kishon

Re: [PATCH v2 1/5] PCI: endpoint: Add RC-to-EP doorbell support using platform MSI controller

[Frank Li]

On Fri, Sep 29, 2023 at 03:00:16PM +0530, Kishon Vijay Abraham I wrote:
> Hi Frank,
> 
> On 9/12/2023 3:39 AM, Frank Li wrote:
> > This commit introduces a common method for sending messages from the Root
> > Complex (RC) to the Endpoint (EP) by utilizing the platform MSI interrupt
> > controller, such as ARM GIC, as an EP doorbell. Maps the memory assigned
> > for the BAR region by the PCI host to the message address of the platform
> > MSI interrupt controller in the PCI EP. As a result, when the PCI RC writes
> > to the BAR region, it triggers an IRQ at the EP. This implementation serves
> > as a common method for all endpoint function drivers.
> 
> This would help avoid the polling used in current EPF drivers. Thanks!
> > 
> > However, it currently supports only one EP physical function due to
> > limitations in ARM MSI/IMS readiness.
> 
> Any such platform or architecture restrictions should not be handled in the
> endpoint core layer.
> > 
> > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > ---
> >   drivers/pci/endpoint/pci-epc-core.c | 192 ++++++++++++++++++++++++++++
> >   drivers/pci/endpoint/pci-epf-core.c |  44 +++++++
> >   include/linux/pci-epc.h             |   6 +
> >   include/linux/pci-epf.h             |   7 +
> >   4 files changed, 249 insertions(+)
> > 
> > diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
> > index 5a4a8b0be6262..d336a99c6a94f 100644
> > --- a/drivers/pci/endpoint/pci-epc-core.c
> > +++ b/drivers/pci/endpoint/pci-epc-core.c
> > @@ -10,6 +10,7 @@
> >   #include <linux/slab.h>
> >   #include <linux/module.h>
> > +#include <linux/msi.h>
> >   #include <linux/pci-epc.h>
> >   #include <linux/pci-epf.h>
> >   #include <linux/pci-ep-cfs.h>
> > @@ -783,6 +784,197 @@ void pci_epc_bme_notify(struct pci_epc *epc)
> >   }
> >   EXPORT_SYMBOL_GPL(pci_epc_bme_notify);
> > +/**
> > + * pci_epc_alloc_doorbell() - alloc an address space to let RC trigger EP side IRQ by write data to
> > + *			      the space.
> > + *
> > + * @epc: the EPC device that need doorbell address and data from RC.
> > + * @func_no: the physical endpoint function number in the EPC device.
> > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > + * @num_msgs: the total number of doorbell messages
> > + *
> > + * Return: 0 success, other is failure
> > + */
> > +int pci_epc_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > +{
> > +	int ret;
> > +
> > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > +		return -EINVAL;
> > +
> > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > +		return -EINVAL;
> > +
> > +	if (!epc->ops->alloc_doorbell)
> > +		return 0;
> > +
> > +	mutex_lock(&epc->lock);
> > +	ret = epc->ops->alloc_doorbell(epc, func_no, vfunc_no, num_msgs);
> > +	mutex_unlock(&epc->lock);
> > +
> > +	return ret;
> > +}
> > +EXPORT_SYMBOL_GPL(pci_epc_alloc_doorbell);
> > +
> > +/**
> > + * pci_epc_free_doorbell() - free resource allocated by pci_epc_alloc_doorbell()
> > + *
> > + * @epc: the EPC device that need doorbell address and data from RC.
> > + * @func_no: the physical endpoint function number in the EPC device.
> > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > + *
> > + * Return: 0 success, other is failure
> > + */
> > +void pci_epc_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
> > +{
> > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > +		return;
> > +
> > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > +		return;
> > +
> > +	if (!epc->ops->free_doorbell)
> > +		return;
> > +
> > +	mutex_lock(&epc->lock);
> > +	epc->ops->free_doorbell(epc, func_no, vfunc_no);
> > +	mutex_unlock(&epc->lock);
> > +}
> > +EXPORT_SYMBOL_GPL(pci_epc_free_doorbell);
> > +
> > +static irqreturn_t pci_epf_generic_doorbell_handler(int irq, void *data)
> > +{
> > +	struct pci_epf *epf = data;
> > +
> > +	if (epf->event_ops && epf->event_ops->doorbell)
> > +		epf->event_ops->doorbell(epf, irq - epf->virq_base);
> > +
> > +	return IRQ_HANDLED;
> > +}
> 
> IMO the handler should be directly implemented in the EPF drivers. There
> should be one API which returns the virq and the msi_msg to the EPF driver
> and the EPF driver should do request_irq.
> > +
> > +static void pci_epc_generic_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
> > +{
> > +	struct pci_epc *epc = NULL;
> > +	struct class_dev_iter iter;
> > +	struct pci_epf *epf;
> > +	struct device *dev;
> > +
> > +	class_dev_iter_init(&iter, pci_epc_class, NULL, NULL);
> > +	while ((dev = class_dev_iter_next(&iter))) {
> > +		if (dev->parent != desc->dev)
> > +			continue;
> 
> Ideally the msi_desc should be associated directly with the EPF device.

The key problem is platform_msi_domain_alloc_irqs only support put epc
driver's dev into desc.

IMS may resolve this problem, but ARM IMS progress is quite slow.

> > +
> > +		epc = to_pci_epc(dev);
> > +
> > +		class_dev_iter_exit(&iter);
> > +		break;
> > +	}
> > +
> > +	if (!epc)
> > +		return;
> > +
> > +	/* Only support one EPF for doorbell */
> > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > +
> > +	if (!epf)
> > +		return;
> 
> If this is a platform restriction, this should be moved elsewhere.

where is good place? 

> > +
> > +	if (epf->msg && desc->msi_index < epf->num_msgs)
> > +		epf->msg[desc->msi_index] = *msg;
> > +}
> > +
> > +
> > +/**
> > + * pci_epc_generic_alloc_doorbell() - Common help function. Allocate address space from MSI
> > + *                                    controller
> > + *
> > + * @epc: the EPC device that need doorbell address and data from RC.
> > + * @func_no: the physical endpoint function number in the EPC device.
> > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > + * @num_msgs: the total number of doorbell messages
> > + *
> > + * Remark: use this function only if EPC driver just register one EPC device.
> > + *
> > + * Return: 0 success, other is failure
> > + */
> > +int pci_epc_generic_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > +{
> > +	struct pci_epf *epf;
> > +	struct device *dev;
> > +	int virq, last;
> > +	int ret;
> > +	int i;
> > +
> > +	if (IS_ERR_OR_NULL(epc))
> > +		return -EINVAL;
> > +
> > +	/* Currently only support one func and one vfunc for doorbell */
> > +	if (func_no || vfunc_no)
> > +		return -EINVAL;
> > +
> > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > +	if (!epf)
> > +		return -EINVAL;
> > +
> > +	dev = epc->dev.parent;
> > +	ret = platform_msi_domain_alloc_irqs(dev, num_msgs, pci_epc_generic_write_msi_msg);
> > +	if (ret) {
> > +		dev_err(dev, "Failed to allocate MSI\n");
> > +		return -ENOMEM;
> > +	}
> 
> The alloc_irqs should be for a EPF device IMO.

The GIC ITS driver need of_node information to alloc msi for such devices.
But EPF device have not of_node information.

> > +
> > +	last = -1;
> > +	for (i = 0; i < num_msgs; i++) {
> > +		virq = msi_get_virq(dev, i);
> > +		if (i == 0)
> > +			epf->virq_base = virq;
> > +
> > +		ret = request_irq(virq, pci_epf_generic_doorbell_handler, 0,
> > +				  kasprintf(GFP_KERNEL, "pci-epc-doorbell%d", i), epf);
> > +
> > +		if (ret) {
> > +			dev_err(dev, "Failed to request doorbell\n");
> > +			goto err_free_irq;
> > +		}
> > +		last = i;
> > +	}
> > +
> > +	return 0;
> > +
> > +err_free_irq:
> > +	for (i = 0; i < last; i++)
> > +		kfree(free_irq(epf->virq_base + i, epf));
> > +	platform_msi_domain_free_irqs(dev);
> > +
> > +	return -EINVAL;
> > +}
> 
> Thanks,
> Kishon

Re: [PATCH v2 0/5] Add RC-to-EP doorbell with platform MSI controller

[Manivannan Sadhasivam]

On Wed, Sep 20, 2023 at 05:39:48PM -0400, Frank Li wrote:
> On Mon, Sep 11, 2023 at 06:09:15PM -0400, Frank Li wrote:
> > ┌────────────┐   ┌───────────────────────────────────┐   ┌────────────────┐
> > │            │   │                                   │   │                │
> > │            │   │ PCI Endpoint                      │   │ PCI Host       │
> > │            │   │                                   │   │                │
> > │            │◄──┤ 1.platform_msi_domain_alloc_irqs()│   │                │
> > │            │   │                                   │   │                │
> > │ MSI        ├──►│ 2.write_msi_msg()                 ├──►├─BAR<n>         │
> > │ Controller │   │   update doorbell register address│   │                │
> > │            │   │   for BAR                         │   │                │
> > │            │   │                                   │   │ 3. Write BAR<n>│
> > │            │◄──┼───────────────────────────────────┼───┤                │
> > │            │   │                                   │   │                │
> > │            ├──►│ 4.Irq Handle                      │   │                │
> > │            │   │                                   │   │                │
> > │            │   │                                   │   │                │
> > └────────────┘   └───────────────────────────────────┘   └────────────────┘
> 
> @mani:
> 	Do you have chance to review this patch again?

I was on vacation for past few weeks. Will take a look in the coming week.

- Mani

> 
> Frank
> 
> > 
> > This patches based on old https://lore.kernel.org/imx/20221124055036.1630573-1-Frank.Li@nxp.com/
> > 
> > Original patch only target to vntb driver. But actually it is common
> > method.
> > 
> > This patches add new API to pci-epf-core, so any EP driver can use it.
> > 
> > The key point is comments from Thomas Gleixner, who suggest use new
> > PCI/IMS. But arm platform change still not be merged yet.
> > 
> > git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git devmsi-v2-arm
> > 
> > So I still use existed method implement RC to EP doorbell.
> > 
> > If Thomas Gleixner want to continue work on devmsi-v2-arm, I can help test
> > and update this patch.
> > 
> > Change from v1 to v2
> > - Add missed patch for endpont/pci-epf-test.c
> > - Move alloc and free to epc driver from epf.
> > - Provide general help function for EPC driver to alloc platform msi irq.
> > - Fixed manivannan's comments.
> > 
> > Frank Li (5):
> >   PCI: endpoint: Add RC-to-EP doorbell support using platform MSI
> >     controller
> >   PCI: dwc: add doorbell support by use MSI controller
> >   PCI: endpoint: pci-epf-test: add doorbell test
> >   misc: pci_endpoint_test: Add doorbell test case
> >   tools: PCI: Add 'B' option for test doorbell
> > 
> >  drivers/misc/pci_endpoint_test.c              |  48 +++++
> >  .../pci/controller/dwc/pcie-designware-ep.c   |   2 +
> >  drivers/pci/endpoint/functions/pci-epf-test.c |  59 +++++-
> >  drivers/pci/endpoint/pci-epc-core.c           | 192 ++++++++++++++++++
> >  drivers/pci/endpoint/pci-epf-core.c           |  44 ++++
> >  include/linux/pci-epc.h                       |   6 +
> >  include/linux/pci-epf.h                       |   7 +
> >  include/uapi/linux/pcitest.h                  |   1 +
> >  tools/pci/pcitest.c                           |  16 +-
> >  9 files changed, 373 insertions(+), 2 deletions(-)
> > 
> > -- 
> > 2.34.1
> > 

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 0/5] Add RC-to-EP doorbell with platform MSI controller

[Frank Li]

On Sat, Sep 30, 2023 at 11:02:49AM +0200, Manivannan Sadhasivam wrote:
> On Wed, Sep 20, 2023 at 05:39:48PM -0400, Frank Li wrote:
> > On Mon, Sep 11, 2023 at 06:09:15PM -0400, Frank Li wrote:
> > > ┌────────────┐   ┌───────────────────────────────────┐   ┌────────────────┐
> > > │            │   │                                   │   │                │
> > > │            │   │ PCI Endpoint                      │   │ PCI Host       │
> > > │            │   │                                   │   │                │
> > > │            │◄──┤ 1.platform_msi_domain_alloc_irqs()│   │                │
> > > │            │   │                                   │   │                │
> > > │ MSI        ├──►│ 2.write_msi_msg()                 ├──►├─BAR<n>         │
> > > │ Controller │   │   update doorbell register address│   │                │
> > > │            │   │   for BAR                         │   │                │
> > > │            │   │                                   │   │ 3. Write BAR<n>│
> > > │            │◄──┼───────────────────────────────────┼───┤                │
> > > │            │   │                                   │   │                │
> > > │            ├──►│ 4.Irq Handle                      │   │                │
> > > │            │   │                                   │   │                │
> > > │            │   │                                   │   │                │
> > > └────────────┘   └───────────────────────────────────┘   └────────────────┘
> > 
> > @mani:
> > 	Do you have chance to review this patch again?
> 
> I was on vacation for past few weeks. Will take a look in the coming week.

Ping?

Frank

> 
> - Mani
> 
> > 
> > Frank
> > 
> > > 
> > > This patches based on old https://lore.kernel.org/imx/20221124055036.1630573-1-Frank.Li@nxp.com/
> > > 
> > > Original patch only target to vntb driver. But actually it is common
> > > method.
> > > 
> > > This patches add new API to pci-epf-core, so any EP driver can use it.
> > > 
> > > The key point is comments from Thomas Gleixner, who suggest use new
> > > PCI/IMS. But arm platform change still not be merged yet.
> > > 
> > > git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git devmsi-v2-arm
> > > 
> > > So I still use existed method implement RC to EP doorbell.
> > > 
> > > If Thomas Gleixner want to continue work on devmsi-v2-arm, I can help test
> > > and update this patch.
> > > 
> > > Change from v1 to v2
> > > - Add missed patch for endpont/pci-epf-test.c
> > > - Move alloc and free to epc driver from epf.
> > > - Provide general help function for EPC driver to alloc platform msi irq.
> > > - Fixed manivannan's comments.
> > > 
> > > Frank Li (5):
> > >   PCI: endpoint: Add RC-to-EP doorbell support using platform MSI
> > >     controller
> > >   PCI: dwc: add doorbell support by use MSI controller
> > >   PCI: endpoint: pci-epf-test: add doorbell test
> > >   misc: pci_endpoint_test: Add doorbell test case
> > >   tools: PCI: Add 'B' option for test doorbell
> > > 
> > >  drivers/misc/pci_endpoint_test.c              |  48 +++++
> > >  .../pci/controller/dwc/pcie-designware-ep.c   |   2 +
> > >  drivers/pci/endpoint/functions/pci-epf-test.c |  59 +++++-
> > >  drivers/pci/endpoint/pci-epc-core.c           | 192 ++++++++++++++++++
> > >  drivers/pci/endpoint/pci-epf-core.c           |  44 ++++
> > >  include/linux/pci-epc.h                       |   6 +
> > >  include/linux/pci-epf.h                       |   7 +
> > >  include/uapi/linux/pcitest.h                  |   1 +
> > >  tools/pci/pcitest.c                           |  16 +-
> > >  9 files changed, 373 insertions(+), 2 deletions(-)
> > > 
> > > -- 
> > > 2.34.1
> > > 
> 
> -- 
> மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 0/5] Add RC-to-EP doorbell with platform MSI controller

[Frank Li]

On Tue, Oct 10, 2023 at 10:21:24AM -0400, Frank Li wrote:
> On Sat, Sep 30, 2023 at 11:02:49AM +0200, Manivannan Sadhasivam wrote:
> > On Wed, Sep 20, 2023 at 05:39:48PM -0400, Frank Li wrote:
> > > On Mon, Sep 11, 2023 at 06:09:15PM -0400, Frank Li wrote:
> > > > ┌────────────┐   ┌───────────────────────────────────┐   ┌────────────────┐
> > > > │            │   │                                   │   │                │
> > > > │            │   │ PCI Endpoint                      │   │ PCI Host       │
> > > > │            │   │                                   │   │                │
> > > > │            │◄──┤ 1.platform_msi_domain_alloc_irqs()│   │                │
> > > > │            │   │                                   │   │                │
> > > > │ MSI        ├──►│ 2.write_msi_msg()                 ├──►├─BAR<n>         │
> > > > │ Controller │   │   update doorbell register address│   │                │
> > > > │            │   │   for BAR                         │   │                │
> > > > │            │   │                                   │   │ 3. Write BAR<n>│
> > > > │            │◄──┼───────────────────────────────────┼───┤                │
> > > > │            │   │                                   │   │                │
> > > > │            ├──►│ 4.Irq Handle                      │   │                │
> > > > │            │   │                                   │   │                │
> > > > │            │   │                                   │   │                │
> > > > └────────────┘   └───────────────────────────────────┘   └────────────────┘
> > > 
> > > @mani:
> > > 	Do you have chance to review this patch again?
> > 
> > I was on vacation for past few weeks. Will take a look in the coming week.
> 
> Ping?
> 
> Frank

@mani: did you have chance to look at this?

> 
> > 
> > - Mani
> > 
> > > 
> > > Frank
> > > 
> > > > 
> > > > This patches based on old https://lore.kernel.org/imx/20221124055036.1630573-1-Frank.Li@nxp.com/
> > > > 
> > > > Original patch only target to vntb driver. But actually it is common
> > > > method.
> > > > 
> > > > This patches add new API to pci-epf-core, so any EP driver can use it.
> > > > 
> > > > The key point is comments from Thomas Gleixner, who suggest use new
> > > > PCI/IMS. But arm platform change still not be merged yet.
> > > > 
> > > > git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git devmsi-v2-arm
> > > > 
> > > > So I still use existed method implement RC to EP doorbell.
> > > > 
> > > > If Thomas Gleixner want to continue work on devmsi-v2-arm, I can help test
> > > > and update this patch.
> > > > 
> > > > Change from v1 to v2
> > > > - Add missed patch for endpont/pci-epf-test.c
> > > > - Move alloc and free to epc driver from epf.
> > > > - Provide general help function for EPC driver to alloc platform msi irq.
> > > > - Fixed manivannan's comments.
> > > > 
> > > > Frank Li (5):
> > > >   PCI: endpoint: Add RC-to-EP doorbell support using platform MSI
> > > >     controller
> > > >   PCI: dwc: add doorbell support by use MSI controller
> > > >   PCI: endpoint: pci-epf-test: add doorbell test
> > > >   misc: pci_endpoint_test: Add doorbell test case
> > > >   tools: PCI: Add 'B' option for test doorbell
> > > > 
> > > >  drivers/misc/pci_endpoint_test.c              |  48 +++++
> > > >  .../pci/controller/dwc/pcie-designware-ep.c   |   2 +
> > > >  drivers/pci/endpoint/functions/pci-epf-test.c |  59 +++++-
> > > >  drivers/pci/endpoint/pci-epc-core.c           | 192 ++++++++++++++++++
> > > >  drivers/pci/endpoint/pci-epf-core.c           |  44 ++++
> > > >  include/linux/pci-epc.h                       |   6 +
> > > >  include/linux/pci-epf.h                       |   7 +
> > > >  include/uapi/linux/pcitest.h                  |   1 +
> > > >  tools/pci/pcitest.c                           |  16 +-
> > > >  9 files changed, 373 insertions(+), 2 deletions(-)
> > > > 
> > > > -- 
> > > > 2.34.1
> > > > 
> > 
> > -- 
> > மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 0/5] Add RC-to-EP doorbell with platform MSI controller

[Manivannan Sadhasivam]

On Mon, Oct 16, 2023 at 10:50:58AM -0400, Frank Li wrote:
> On Tue, Oct 10, 2023 at 10:21:24AM -0400, Frank Li wrote:
> > On Sat, Sep 30, 2023 at 11:02:49AM +0200, Manivannan Sadhasivam wrote:
> > > On Wed, Sep 20, 2023 at 05:39:48PM -0400, Frank Li wrote:
> > > > On Mon, Sep 11, 2023 at 06:09:15PM -0400, Frank Li wrote:
> > > > > ┌────────────┐   ┌───────────────────────────────────┐   ┌────────────────┐
> > > > > │            │   │                                   │   │                │
> > > > > │            │   │ PCI Endpoint                      │   │ PCI Host       │
> > > > > │            │   │                                   │   │                │
> > > > > │            │◄──┤ 1.platform_msi_domain_alloc_irqs()│   │                │
> > > > > │            │   │                                   │   │                │
> > > > > │ MSI        ├──►│ 2.write_msi_msg()                 ├──►├─BAR<n>         │
> > > > > │ Controller │   │   update doorbell register address│   │                │
> > > > > │            │   │   for BAR                         │   │                │
> > > > > │            │   │                                   │   │ 3. Write BAR<n>│
> > > > > │            │◄──┼───────────────────────────────────┼───┤                │
> > > > > │            │   │                                   │   │                │
> > > > > │            ├──►│ 4.Irq Handle                      │   │                │
> > > > > │            │   │                                   │   │                │
> > > > > │            │   │                                   │   │                │
> > > > > └────────────┘   └───────────────────────────────────┘   └────────────────┘
> > > > 
> > > > @mani:
> > > > 	Do you have chance to review this patch again?
> > > 
> > > I was on vacation for past few weeks. Will take a look in the coming week.
> > 
> > Ping?
> > 
> > Frank
> 
> @mani: did you have chance to look at this?
> 

Sorry for the long delay. I will take a look tomorrow.

- Mani

> > 
> > > 
> > > - Mani
> > > 
> > > > 
> > > > Frank
> > > > 
> > > > > 
> > > > > This patches based on old https://lore.kernel.org/imx/20221124055036.1630573-1-Frank.Li@nxp.com/
> > > > > 
> > > > > Original patch only target to vntb driver. But actually it is common
> > > > > method.
> > > > > 
> > > > > This patches add new API to pci-epf-core, so any EP driver can use it.
> > > > > 
> > > > > The key point is comments from Thomas Gleixner, who suggest use new
> > > > > PCI/IMS. But arm platform change still not be merged yet.
> > > > > 
> > > > > git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git devmsi-v2-arm
> > > > > 
> > > > > So I still use existed method implement RC to EP doorbell.
> > > > > 
> > > > > If Thomas Gleixner want to continue work on devmsi-v2-arm, I can help test
> > > > > and update this patch.
> > > > > 
> > > > > Change from v1 to v2
> > > > > - Add missed patch for endpont/pci-epf-test.c
> > > > > - Move alloc and free to epc driver from epf.
> > > > > - Provide general help function for EPC driver to alloc platform msi irq.
> > > > > - Fixed manivannan's comments.
> > > > > 
> > > > > Frank Li (5):
> > > > >   PCI: endpoint: Add RC-to-EP doorbell support using platform MSI
> > > > >     controller
> > > > >   PCI: dwc: add doorbell support by use MSI controller
> > > > >   PCI: endpoint: pci-epf-test: add doorbell test
> > > > >   misc: pci_endpoint_test: Add doorbell test case
> > > > >   tools: PCI: Add 'B' option for test doorbell
> > > > > 
> > > > >  drivers/misc/pci_endpoint_test.c              |  48 +++++
> > > > >  .../pci/controller/dwc/pcie-designware-ep.c   |   2 +
> > > > >  drivers/pci/endpoint/functions/pci-epf-test.c |  59 +++++-
> > > > >  drivers/pci/endpoint/pci-epc-core.c           | 192 ++++++++++++++++++
> > > > >  drivers/pci/endpoint/pci-epf-core.c           |  44 ++++
> > > > >  include/linux/pci-epc.h                       |   6 +
> > > > >  include/linux/pci-epf.h                       |   7 +
> > > > >  include/uapi/linux/pcitest.h                  |   1 +
> > > > >  tools/pci/pcitest.c                           |  16 +-
> > > > >  9 files changed, 373 insertions(+), 2 deletions(-)
> > > > > 
> > > > > -- 
> > > > > 2.34.1
> > > > > 
> > > 
> > > -- 
> > > மணிவண்ணன் சதாசிவம்

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/5] PCI: endpoint: Add RC-to-EP doorbell support using platform MSI controller

[Manivannan Sadhasivam]

On Mon, Sep 11, 2023 at 06:09:16PM -0400, Frank Li wrote:
> This commit introduces a common method for sending messages from the Root
> Complex (RC) to the Endpoint (EP) by utilizing the platform MSI interrupt
> controller, such as ARM GIC, as an EP doorbell. Maps the memory assigned
> for the BAR region by the PCI host to the message address of the platform
> MSI interrupt controller in the PCI EP. As a result, when the PCI RC writes

"Doorbell feature is implemented by mapping the EP's MSI interrupt controller
message address to a dedicated BAR in the EPC core. It is the responsibility
of the EPF driver to pass the actual message data to be written by the host to
the doorbell BAR region through its own logic."

> to the BAR region, it triggers an IRQ at the EP. This implementation serves
> as a common method for all endpoint function drivers.
> 
> However, it currently supports only one EP physical function due to
> limitations in ARM MSI/IMS readiness.
> 
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---
>  drivers/pci/endpoint/pci-epc-core.c | 192 ++++++++++++++++++++++++++++
>  drivers/pci/endpoint/pci-epf-core.c |  44 +++++++
>  include/linux/pci-epc.h             |   6 +
>  include/linux/pci-epf.h             |   7 +
>  4 files changed, 249 insertions(+)
> 
> diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
> index 5a4a8b0be6262..d336a99c6a94f 100644
> --- a/drivers/pci/endpoint/pci-epc-core.c
> +++ b/drivers/pci/endpoint/pci-epc-core.c
> @@ -10,6 +10,7 @@
>  #include <linux/slab.h>
>  #include <linux/module.h>
>  
> +#include <linux/msi.h>
>  #include <linux/pci-epc.h>
>  #include <linux/pci-epf.h>
>  #include <linux/pci-ep-cfs.h>
> @@ -783,6 +784,197 @@ void pci_epc_bme_notify(struct pci_epc *epc)
>  }
>  EXPORT_SYMBOL_GPL(pci_epc_bme_notify);
>  
> +/**
> + * pci_epc_alloc_doorbell() - alloc an address space to let RC trigger EP side IRQ by write data to
> + *			      the space.

"Allocate platform specific doorbell IRQs to be used by the host to trigger
doorbells on EP."

> + *
> + * @epc: the EPC device that need doorbell address and data from RC.

EPC device for which the doorbell needs to be allocated

> + * @func_no: the physical endpoint function number in the EPC device.
> + * @vfunc_no: the virtual endpoint function number in the physical function.
> + * @num_msgs: the total number of doorbell messages

s/num_msgs/num_db

> + *
> + * Return: 0 success, other is failure
> + */
> +int pci_epc_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> +{
> +	int ret;
> +
> +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> +		return -EINVAL;
> +
> +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> +		return -EINVAL;
> +
> +	if (!epc->ops->alloc_doorbell)
> +		return 0;

You mentioned 0 is a success. So if there is no callback, you want to return
success?

> +
> +	mutex_lock(&epc->lock);
> +	ret = epc->ops->alloc_doorbell(epc, func_no, vfunc_no, num_msgs);

Why can't you just call the generic function here and in other places instead of
implementing callbacks? I do not see a necessity for EPC specific callbacks. If
there is one, please specify.

> +	mutex_unlock(&epc->lock);
> +
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(pci_epc_alloc_doorbell);
> +
> +/**
> + * pci_epc_free_doorbell() - free resource allocated by pci_epc_alloc_doorbell()
> + *
> + * @epc: the EPC device that need doorbell address and data from RC.

Same as above.

> + * @func_no: the physical endpoint function number in the EPC device.
> + * @vfunc_no: the virtual endpoint function number in the physical function.
> + *
> + * Return: 0 success, other is failure
> + */
> +void pci_epc_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
> +{
> +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> +		return;
> +
> +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> +		return;
> +
> +	if (!epc->ops->free_doorbell)
> +		return;
> +
> +	mutex_lock(&epc->lock);
> +	epc->ops->free_doorbell(epc, func_no, vfunc_no);

Same as suggested above.

> +	mutex_unlock(&epc->lock);
> +}
> +EXPORT_SYMBOL_GPL(pci_epc_free_doorbell);
> +
> +static irqreturn_t pci_epf_generic_doorbell_handler(int irq, void *data)
> +{
> +	struct pci_epf *epf = data;
> +
> +	if (epf->event_ops && epf->event_ops->doorbell)
> +		epf->event_ops->doorbell(epf, irq - epf->virq_base);

Same as suggested above.

> +
> +	return IRQ_HANDLED;
> +}
> +
> +static void pci_epc_generic_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
> +{
> +	struct pci_epc *epc = NULL;
> +	struct class_dev_iter iter;
> +	struct pci_epf *epf;
> +	struct device *dev;
> +
> +	class_dev_iter_init(&iter, pci_epc_class, NULL, NULL);
> +	while ((dev = class_dev_iter_next(&iter))) {
> +		if (dev->parent != desc->dev)
> +			continue;
> +
> +		epc = to_pci_epc(dev);
> +
> +		class_dev_iter_exit(&iter);
> +		break;
> +	}
> +
> +	if (!epc)
> +		return;
> +
> +	/* Only support one EPF for doorbell */
> +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> +

No need of this newline

> +	if (!epf)
> +		return;
> +
> +	if (epf->msg && desc->msi_index < epf->num_msgs)
> +		epf->msg[desc->msi_index] = *msg;
> +}
> +
> +

Remove extra newline

> +/**
> + * pci_epc_generic_alloc_doorbell() - Common help function. Allocate address space from MSI
> + *                                    controller
> + *
> + * @epc: the EPC device that need doorbell address and data from RC.
> + * @func_no: the physical endpoint function number in the EPC device.
> + * @vfunc_no: the virtual endpoint function number in the physical function.
> + * @num_msgs: the total number of doorbell messages
> + *

Same comment as for pci_epc_alloc_doorbell()

> + * Remark: use this function only if EPC driver just register one EPC device.
> + *
> + * Return: 0 success, other is failure
> + */
> +int pci_epc_generic_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> +{
> +	struct pci_epf *epf;
> +	struct device *dev;
> +	int virq, last;
> +	int ret;
> +	int i;
> +
> +	if (IS_ERR_OR_NULL(epc))
> +		return -EINVAL;
> +
> +	/* Currently only support one func and one vfunc for doorbell */
> +	if (func_no || vfunc_no)
> +		return -EINVAL;
> +
> +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> +	if (!epf)
> +		return -EINVAL;
> +
> +	dev = epc->dev.parent;
> +	ret = platform_msi_domain_alloc_irqs(dev, num_msgs, pci_epc_generic_write_msi_msg);
> +	if (ret) {
> +		dev_err(dev, "Failed to allocate MSI\n");
> +		return -ENOMEM;
> +	}
> +
> +	last = -1;
> +	for (i = 0; i < num_msgs; i++) {

You should iterate over msi_desc as below:

        msi_lock_descs(dev);
        msi_for_each_desc(desc, dev, MSI_DESC_ALL) {
		...
	}
	msi_unlock_descs(dev);

> +		virq = msi_get_virq(dev, i);
> +		if (i == 0)
> +			epf->virq_base = virq;
> +
> +		ret = request_irq(virq, pci_epf_generic_doorbell_handler, 0,

	request_irq(desc->irq, ...)

> +				  kasprintf(GFP_KERNEL, "pci-epc-doorbell%d", i), epf);
> +
> +		if (ret) {
> +			dev_err(dev, "Failed to request doorbell\n");
> +			goto err_free_irq;
> +		}
> +		last = i;
> +	}
> +
> +	return 0;
> +
> +err_free_irq:
> +	for (i = 0; i < last; i++)
> +		kfree(free_irq(epf->virq_base + i, epf));
> +	platform_msi_domain_free_irqs(dev);
> +
> +	return -EINVAL;

	return ret;

> +}
> +EXPORT_SYMBOL_GPL(pci_epc_generic_alloc_doorbell);
> +

[...]

> diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
> index 3f44b6aec4770..485c146a5efe2 100644
> --- a/include/linux/pci-epf.h
> +++ b/include/linux/pci-epf.h
> @@ -79,6 +79,7 @@ struct pci_epc_event_ops {
>  	int (*link_up)(struct pci_epf *epf);
>  	int (*link_down)(struct pci_epf *epf);
>  	int (*bme)(struct pci_epf *epf);
> +	int (*doorbell)(struct pci_epf *epf, int index);

kdoc missing.

>  };
>  
>  /**
> @@ -180,6 +181,9 @@ struct pci_epf {
>  	unsigned long		vfunction_num_map;
>  	struct list_head	pci_vepf;
>  	const struct pci_epc_event_ops *event_ops;
> +	struct msi_msg *msg;
> +	u16 num_msgs;

num_db

You also need to add kdoc for each new member.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/5] PCI: endpoint: Add RC-to-EP doorbell support using platform MSI controller

[Frank Li]

On Wed, Oct 18, 2023 at 12:07:22AM +0530, Manivannan Sadhasivam wrote:
> On Mon, Sep 11, 2023 at 06:09:16PM -0400, Frank Li wrote:
> > This commit introduces a common method for sending messages from the Root
> > Complex (RC) to the Endpoint (EP) by utilizing the platform MSI interrupt
> > controller, such as ARM GIC, as an EP doorbell. Maps the memory assigned
> > for the BAR region by the PCI host to the message address of the platform
> > MSI interrupt controller in the PCI EP. As a result, when the PCI RC writes
> 
> "Doorbell feature is implemented by mapping the EP's MSI interrupt controller
> message address to a dedicated BAR in the EPC core. It is the responsibility
> of the EPF driver to pass the actual message data to be written by the host to
> the doorbell BAR region through its own logic."
> 
> > to the BAR region, it triggers an IRQ at the EP. This implementation serves
> > as a common method for all endpoint function drivers.
> > 
> > However, it currently supports only one EP physical function due to
> > limitations in ARM MSI/IMS readiness.
> > 
> > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > ---
> >  drivers/pci/endpoint/pci-epc-core.c | 192 ++++++++++++++++++++++++++++
> >  drivers/pci/endpoint/pci-epf-core.c |  44 +++++++
> >  include/linux/pci-epc.h             |   6 +
> >  include/linux/pci-epf.h             |   7 +
> >  4 files changed, 249 insertions(+)
> > 
> > diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
> > index 5a4a8b0be6262..d336a99c6a94f 100644
> > --- a/drivers/pci/endpoint/pci-epc-core.c
> > +++ b/drivers/pci/endpoint/pci-epc-core.c
> > @@ -10,6 +10,7 @@
> >  #include <linux/slab.h>
> >  #include <linux/module.h>
> >  
> > +#include <linux/msi.h>
> >  #include <linux/pci-epc.h>
> >  #include <linux/pci-epf.h>
> >  #include <linux/pci-ep-cfs.h>
> > @@ -783,6 +784,197 @@ void pci_epc_bme_notify(struct pci_epc *epc)
> >  }
> >  EXPORT_SYMBOL_GPL(pci_epc_bme_notify);
> >  
> > +/**
> > + * pci_epc_alloc_doorbell() - alloc an address space to let RC trigger EP side IRQ by write data to
> > + *			      the space.
> 
> "Allocate platform specific doorbell IRQs to be used by the host to trigger
> doorbells on EP."
> 
> > + *
> > + * @epc: the EPC device that need doorbell address and data from RC.
> 
> EPC device for which the doorbell needs to be allocated
> 
> > + * @func_no: the physical endpoint function number in the EPC device.
> > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > + * @num_msgs: the total number of doorbell messages
> 
> s/num_msgs/num_db
> 
> > + *
> > + * Return: 0 success, other is failure
> > + */
> > +int pci_epc_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > +{
> > +	int ret;
> > +
> > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > +		return -EINVAL;
> > +
> > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > +		return -EINVAL;
> > +
> > +	if (!epc->ops->alloc_doorbell)
> > +		return 0;
> 
> You mentioned 0 is a success. So if there is no callback, you want to return
> success?
> 
> > +
> > +	mutex_lock(&epc->lock);
> > +	ret = epc->ops->alloc_doorbell(epc, func_no, vfunc_no, num_msgs);
> 
> Why can't you just call the generic function here and in other places instead of
> implementing callbacks? I do not see a necessity for EPC specific callbacks. If
> there is one, please specify.

1. Refer v1 your comments.
https://lore.kernel.org/imx/20230906145227.GC5930@thinkpad/
2. Maybe some ep controller have built-in doorbell support. Write to some
address to trigger doorbell irq.

Frank

> 
> > +	mutex_unlock(&epc->lock);
> > +
> > +	return ret;
> > +}
> > +EXPORT_SYMBOL_GPL(pci_epc_alloc_doorbell);
> > +
> > +/**
> > + * pci_epc_free_doorbell() - free resource allocated by pci_epc_alloc_doorbell()
> > + *
> > + * @epc: the EPC device that need doorbell address and data from RC.
> 
> Same as above.
> 
> > + * @func_no: the physical endpoint function number in the EPC device.
> > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > + *
> > + * Return: 0 success, other is failure
> > + */
> > +void pci_epc_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
> > +{
> > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > +		return;
> > +
> > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > +		return;
> > +
> > +	if (!epc->ops->free_doorbell)
> > +		return;
> > +
> > +	mutex_lock(&epc->lock);
> > +	epc->ops->free_doorbell(epc, func_no, vfunc_no);
> 
> Same as suggested above.
> 
> > +	mutex_unlock(&epc->lock);
> > +}
> > +EXPORT_SYMBOL_GPL(pci_epc_free_doorbell);
> > +
> > +static irqreturn_t pci_epf_generic_doorbell_handler(int irq, void *data)
> > +{
> > +	struct pci_epf *epf = data;
> > +
> > +	if (epf->event_ops && epf->event_ops->doorbell)
> > +		epf->event_ops->doorbell(epf, irq - epf->virq_base);
> 
> Same as suggested above.
> 
> > +
> > +	return IRQ_HANDLED;
> > +}
> > +
> > +static void pci_epc_generic_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
> > +{
> > +	struct pci_epc *epc = NULL;
> > +	struct class_dev_iter iter;
> > +	struct pci_epf *epf;
> > +	struct device *dev;
> > +
> > +	class_dev_iter_init(&iter, pci_epc_class, NULL, NULL);
> > +	while ((dev = class_dev_iter_next(&iter))) {
> > +		if (dev->parent != desc->dev)
> > +			continue;
> > +
> > +		epc = to_pci_epc(dev);
> > +
> > +		class_dev_iter_exit(&iter);
> > +		break;
> > +	}
> > +
> > +	if (!epc)
> > +		return;
> > +
> > +	/* Only support one EPF for doorbell */
> > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > +
> 
> No need of this newline
> 
> > +	if (!epf)
> > +		return;
> > +
> > +	if (epf->msg && desc->msi_index < epf->num_msgs)
> > +		epf->msg[desc->msi_index] = *msg;
> > +}
> > +
> > +
> 
> Remove extra newline
> 
> > +/**
> > + * pci_epc_generic_alloc_doorbell() - Common help function. Allocate address space from MSI
> > + *                                    controller
> > + *
> > + * @epc: the EPC device that need doorbell address and data from RC.
> > + * @func_no: the physical endpoint function number in the EPC device.
> > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > + * @num_msgs: the total number of doorbell messages
> > + *
> 
> Same comment as for pci_epc_alloc_doorbell()
> 
> > + * Remark: use this function only if EPC driver just register one EPC device.
> > + *
> > + * Return: 0 success, other is failure
> > + */
> > +int pci_epc_generic_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > +{
> > +	struct pci_epf *epf;
> > +	struct device *dev;
> > +	int virq, last;
> > +	int ret;
> > +	int i;
> > +
> > +	if (IS_ERR_OR_NULL(epc))
> > +		return -EINVAL;
> > +
> > +	/* Currently only support one func and one vfunc for doorbell */
> > +	if (func_no || vfunc_no)
> > +		return -EINVAL;
> > +
> > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > +	if (!epf)
> > +		return -EINVAL;
> > +
> > +	dev = epc->dev.parent;
> > +	ret = platform_msi_domain_alloc_irqs(dev, num_msgs, pci_epc_generic_write_msi_msg);
> > +	if (ret) {
> > +		dev_err(dev, "Failed to allocate MSI\n");
> > +		return -ENOMEM;
> > +	}
> > +
> > +	last = -1;
> > +	for (i = 0; i < num_msgs; i++) {
> 
> You should iterate over msi_desc as below:
> 
>         msi_lock_descs(dev);
>         msi_for_each_desc(desc, dev, MSI_DESC_ALL) {
> 		...
> 	}
> 	msi_unlock_descs(dev);
> 
> > +		virq = msi_get_virq(dev, i);
> > +		if (i == 0)
> > +			epf->virq_base = virq;
> > +
> > +		ret = request_irq(virq, pci_epf_generic_doorbell_handler, 0,
> 
> 	request_irq(desc->irq, ...)
> 
> > +				  kasprintf(GFP_KERNEL, "pci-epc-doorbell%d", i), epf);
> > +
> > +		if (ret) {
> > +			dev_err(dev, "Failed to request doorbell\n");
> > +			goto err_free_irq;
> > +		}
> > +		last = i;
> > +	}
> > +
> > +	return 0;
> > +
> > +err_free_irq:
> > +	for (i = 0; i < last; i++)
> > +		kfree(free_irq(epf->virq_base + i, epf));
> > +	platform_msi_domain_free_irqs(dev);
> > +
> > +	return -EINVAL;
> 
> 	return ret;
> 
> > +}
> > +EXPORT_SYMBOL_GPL(pci_epc_generic_alloc_doorbell);
> > +
> 
> [...]
> 
> > diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
> > index 3f44b6aec4770..485c146a5efe2 100644
> > --- a/include/linux/pci-epf.h
> > +++ b/include/linux/pci-epf.h
> > @@ -79,6 +79,7 @@ struct pci_epc_event_ops {
> >  	int (*link_up)(struct pci_epf *epf);
> >  	int (*link_down)(struct pci_epf *epf);
> >  	int (*bme)(struct pci_epf *epf);
> > +	int (*doorbell)(struct pci_epf *epf, int index);
> 
> kdoc missing.
> 
> >  };
> >  
> >  /**
> > @@ -180,6 +181,9 @@ struct pci_epf {
> >  	unsigned long		vfunction_num_map;
> >  	struct list_head	pci_vepf;
> >  	const struct pci_epc_event_ops *event_ops;
> > +	struct msi_msg *msg;
> > +	u16 num_msgs;
> 
> num_db
> 
> You also need to add kdoc for each new member.
> 
> - Mani
> 
> -- 
> மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/5] PCI: endpoint: Add RC-to-EP doorbell support using platform MSI controller

[Manivannan Sadhasivam]

On Tue, Oct 17, 2023 at 02:55:57PM -0400, Frank Li wrote:
> On Wed, Oct 18, 2023 at 12:07:22AM +0530, Manivannan Sadhasivam wrote:
> > On Mon, Sep 11, 2023 at 06:09:16PM -0400, Frank Li wrote:
> > > This commit introduces a common method for sending messages from the Root
> > > Complex (RC) to the Endpoint (EP) by utilizing the platform MSI interrupt
> > > controller, such as ARM GIC, as an EP doorbell. Maps the memory assigned
> > > for the BAR region by the PCI host to the message address of the platform
> > > MSI interrupt controller in the PCI EP. As a result, when the PCI RC writes
> > 
> > "Doorbell feature is implemented by mapping the EP's MSI interrupt controller
> > message address to a dedicated BAR in the EPC core. It is the responsibility
> > of the EPF driver to pass the actual message data to be written by the host to
> > the doorbell BAR region through its own logic."
> > 
> > > to the BAR region, it triggers an IRQ at the EP. This implementation serves
> > > as a common method for all endpoint function drivers.
> > > 
> > > However, it currently supports only one EP physical function due to
> > > limitations in ARM MSI/IMS readiness.
> > > 
> > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > ---
> > >  drivers/pci/endpoint/pci-epc-core.c | 192 ++++++++++++++++++++++++++++
> > >  drivers/pci/endpoint/pci-epf-core.c |  44 +++++++
> > >  include/linux/pci-epc.h             |   6 +
> > >  include/linux/pci-epf.h             |   7 +
> > >  4 files changed, 249 insertions(+)
> > > 
> > > diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
> > > index 5a4a8b0be6262..d336a99c6a94f 100644
> > > --- a/drivers/pci/endpoint/pci-epc-core.c
> > > +++ b/drivers/pci/endpoint/pci-epc-core.c
> > > @@ -10,6 +10,7 @@
> > >  #include <linux/slab.h>
> > >  #include <linux/module.h>
> > >  
> > > +#include <linux/msi.h>
> > >  #include <linux/pci-epc.h>
> > >  #include <linux/pci-epf.h>
> > >  #include <linux/pci-ep-cfs.h>
> > > @@ -783,6 +784,197 @@ void pci_epc_bme_notify(struct pci_epc *epc)
> > >  }
> > >  EXPORT_SYMBOL_GPL(pci_epc_bme_notify);
> > >  
> > > +/**
> > > + * pci_epc_alloc_doorbell() - alloc an address space to let RC trigger EP side IRQ by write data to
> > > + *			      the space.
> > 
> > "Allocate platform specific doorbell IRQs to be used by the host to trigger
> > doorbells on EP."
> > 
> > > + *
> > > + * @epc: the EPC device that need doorbell address and data from RC.
> > 
> > EPC device for which the doorbell needs to be allocated
> > 
> > > + * @func_no: the physical endpoint function number in the EPC device.
> > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > + * @num_msgs: the total number of doorbell messages
> > 
> > s/num_msgs/num_db
> > 
> > > + *
> > > + * Return: 0 success, other is failure
> > > + */
> > > +int pci_epc_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > > +{
> > > +	int ret;
> > > +
> > > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > > +		return -EINVAL;
> > > +
> > > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > > +		return -EINVAL;
> > > +
> > > +	if (!epc->ops->alloc_doorbell)
> > > +		return 0;
> > 
> > You mentioned 0 is a success. So if there is no callback, you want to return
> > success?
> > 
> > > +
> > > +	mutex_lock(&epc->lock);
> > > +	ret = epc->ops->alloc_doorbell(epc, func_no, vfunc_no, num_msgs);
> > 
> > Why can't you just call the generic function here and in other places instead of
> > implementing callbacks? I do not see a necessity for EPC specific callbacks. If
> > there is one, please specify.
> 
> 1. Refer v1 your comments.
> https://lore.kernel.org/imx/20230906145227.GC5930@thinkpad/

I do not find where I suggested the callback approach.

> 2. Maybe some ep controller have built-in doorbell support. Write to some
> address to trigger doorbell irq.
> 

We will handle it whenever such EP controllers arrive. Until then, let's keep it
simple.

- Mani

> Frank
> 
> > 
> > > +	mutex_unlock(&epc->lock);
> > > +
> > > +	return ret;
> > > +}
> > > +EXPORT_SYMBOL_GPL(pci_epc_alloc_doorbell);
> > > +
> > > +/**
> > > + * pci_epc_free_doorbell() - free resource allocated by pci_epc_alloc_doorbell()
> > > + *
> > > + * @epc: the EPC device that need doorbell address and data from RC.
> > 
> > Same as above.
> > 
> > > + * @func_no: the physical endpoint function number in the EPC device.
> > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > + *
> > > + * Return: 0 success, other is failure
> > > + */
> > > +void pci_epc_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
> > > +{
> > > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > > +		return;
> > > +
> > > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > > +		return;
> > > +
> > > +	if (!epc->ops->free_doorbell)
> > > +		return;
> > > +
> > > +	mutex_lock(&epc->lock);
> > > +	epc->ops->free_doorbell(epc, func_no, vfunc_no);
> > 
> > Same as suggested above.
> > 
> > > +	mutex_unlock(&epc->lock);
> > > +}
> > > +EXPORT_SYMBOL_GPL(pci_epc_free_doorbell);
> > > +
> > > +static irqreturn_t pci_epf_generic_doorbell_handler(int irq, void *data)
> > > +{
> > > +	struct pci_epf *epf = data;
> > > +
> > > +	if (epf->event_ops && epf->event_ops->doorbell)
> > > +		epf->event_ops->doorbell(epf, irq - epf->virq_base);
> > 
> > Same as suggested above.
> > 
> > > +
> > > +	return IRQ_HANDLED;
> > > +}
> > > +
> > > +static void pci_epc_generic_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
> > > +{
> > > +	struct pci_epc *epc = NULL;
> > > +	struct class_dev_iter iter;
> > > +	struct pci_epf *epf;
> > > +	struct device *dev;
> > > +
> > > +	class_dev_iter_init(&iter, pci_epc_class, NULL, NULL);
> > > +	while ((dev = class_dev_iter_next(&iter))) {
> > > +		if (dev->parent != desc->dev)
> > > +			continue;
> > > +
> > > +		epc = to_pci_epc(dev);
> > > +
> > > +		class_dev_iter_exit(&iter);
> > > +		break;
> > > +	}
> > > +
> > > +	if (!epc)
> > > +		return;
> > > +
> > > +	/* Only support one EPF for doorbell */
> > > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > > +
> > 
> > No need of this newline
> > 
> > > +	if (!epf)
> > > +		return;
> > > +
> > > +	if (epf->msg && desc->msi_index < epf->num_msgs)
> > > +		epf->msg[desc->msi_index] = *msg;
> > > +}
> > > +
> > > +
> > 
> > Remove extra newline
> > 
> > > +/**
> > > + * pci_epc_generic_alloc_doorbell() - Common help function. Allocate address space from MSI
> > > + *                                    controller
> > > + *
> > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > + * @func_no: the physical endpoint function number in the EPC device.
> > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > + * @num_msgs: the total number of doorbell messages
> > > + *
> > 
> > Same comment as for pci_epc_alloc_doorbell()
> > 
> > > + * Remark: use this function only if EPC driver just register one EPC device.
> > > + *
> > > + * Return: 0 success, other is failure
> > > + */
> > > +int pci_epc_generic_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > > +{
> > > +	struct pci_epf *epf;
> > > +	struct device *dev;
> > > +	int virq, last;
> > > +	int ret;
> > > +	int i;
> > > +
> > > +	if (IS_ERR_OR_NULL(epc))
> > > +		return -EINVAL;
> > > +
> > > +	/* Currently only support one func and one vfunc for doorbell */
> > > +	if (func_no || vfunc_no)
> > > +		return -EINVAL;
> > > +
> > > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > > +	if (!epf)
> > > +		return -EINVAL;
> > > +
> > > +	dev = epc->dev.parent;
> > > +	ret = platform_msi_domain_alloc_irqs(dev, num_msgs, pci_epc_generic_write_msi_msg);
> > > +	if (ret) {
> > > +		dev_err(dev, "Failed to allocate MSI\n");
> > > +		return -ENOMEM;
> > > +	}
> > > +
> > > +	last = -1;
> > > +	for (i = 0; i < num_msgs; i++) {
> > 
> > You should iterate over msi_desc as below:
> > 
> >         msi_lock_descs(dev);
> >         msi_for_each_desc(desc, dev, MSI_DESC_ALL) {
> > 		...
> > 	}
> > 	msi_unlock_descs(dev);
> > 
> > > +		virq = msi_get_virq(dev, i);
> > > +		if (i == 0)
> > > +			epf->virq_base = virq;
> > > +
> > > +		ret = request_irq(virq, pci_epf_generic_doorbell_handler, 0,
> > 
> > 	request_irq(desc->irq, ...)
> > 
> > > +				  kasprintf(GFP_KERNEL, "pci-epc-doorbell%d", i), epf);
> > > +
> > > +		if (ret) {
> > > +			dev_err(dev, "Failed to request doorbell\n");
> > > +			goto err_free_irq;
> > > +		}
> > > +		last = i;
> > > +	}
> > > +
> > > +	return 0;
> > > +
> > > +err_free_irq:
> > > +	for (i = 0; i < last; i++)
> > > +		kfree(free_irq(epf->virq_base + i, epf));
> > > +	platform_msi_domain_free_irqs(dev);
> > > +
> > > +	return -EINVAL;
> > 
> > 	return ret;
> > 
> > > +}
> > > +EXPORT_SYMBOL_GPL(pci_epc_generic_alloc_doorbell);
> > > +
> > 
> > [...]
> > 
> > > diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
> > > index 3f44b6aec4770..485c146a5efe2 100644
> > > --- a/include/linux/pci-epf.h
> > > +++ b/include/linux/pci-epf.h
> > > @@ -79,6 +79,7 @@ struct pci_epc_event_ops {
> > >  	int (*link_up)(struct pci_epf *epf);
> > >  	int (*link_down)(struct pci_epf *epf);
> > >  	int (*bme)(struct pci_epf *epf);
> > > +	int (*doorbell)(struct pci_epf *epf, int index);
> > 
> > kdoc missing.
> > 
> > >  };
> > >  
> > >  /**
> > > @@ -180,6 +181,9 @@ struct pci_epf {
> > >  	unsigned long		vfunction_num_map;
> > >  	struct list_head	pci_vepf;
> > >  	const struct pci_epc_event_ops *event_ops;
> > > +	struct msi_msg *msg;
> > > +	u16 num_msgs;
> > 
> > num_db
> > 
> > You also need to add kdoc for each new member.
> > 
> > - Mani
> > 
> > -- 
> > மணிவண்ணன் சதாசிவம்

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/5] PCI: endpoint: Add RC-to-EP doorbell support using platform MSI controller

[Frank Li]

On Thu, Oct 19, 2023 at 08:34:41PM +0530, Manivannan Sadhasivam wrote:
> On Tue, Oct 17, 2023 at 02:55:57PM -0400, Frank Li wrote:
> > On Wed, Oct 18, 2023 at 12:07:22AM +0530, Manivannan Sadhasivam wrote:
> > > On Mon, Sep 11, 2023 at 06:09:16PM -0400, Frank Li wrote:
> > > > This commit introduces a common method for sending messages from the Root
> > > > Complex (RC) to the Endpoint (EP) by utilizing the platform MSI interrupt
> > > > controller, such as ARM GIC, as an EP doorbell. Maps the memory assigned
> > > > for the BAR region by the PCI host to the message address of the platform
> > > > MSI interrupt controller in the PCI EP. As a result, when the PCI RC writes
> > > 
> > > "Doorbell feature is implemented by mapping the EP's MSI interrupt controller
> > > message address to a dedicated BAR in the EPC core. It is the responsibility
> > > of the EPF driver to pass the actual message data to be written by the host to
> > > the doorbell BAR region through its own logic."
> > > 
> > > > to the BAR region, it triggers an IRQ at the EP. This implementation serves
> > > > as a common method for all endpoint function drivers.
> > > > 
> > > > However, it currently supports only one EP physical function due to
> > > > limitations in ARM MSI/IMS readiness.
> > > > 
> > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > ---
> > > >  drivers/pci/endpoint/pci-epc-core.c | 192 ++++++++++++++++++++++++++++
> > > >  drivers/pci/endpoint/pci-epf-core.c |  44 +++++++
> > > >  include/linux/pci-epc.h             |   6 +
> > > >  include/linux/pci-epf.h             |   7 +
> > > >  4 files changed, 249 insertions(+)
> > > > 
> > > > diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
> > > > index 5a4a8b0be6262..d336a99c6a94f 100644
> > > > --- a/drivers/pci/endpoint/pci-epc-core.c
> > > > +++ b/drivers/pci/endpoint/pci-epc-core.c
> > > > @@ -10,6 +10,7 @@
> > > >  #include <linux/slab.h>
> > > >  #include <linux/module.h>
> > > >  
> > > > +#include <linux/msi.h>
> > > >  #include <linux/pci-epc.h>
> > > >  #include <linux/pci-epf.h>
> > > >  #include <linux/pci-ep-cfs.h>
> > > > @@ -783,6 +784,197 @@ void pci_epc_bme_notify(struct pci_epc *epc)
> > > >  }
> > > >  EXPORT_SYMBOL_GPL(pci_epc_bme_notify);
> > > >  
> > > > +/**
> > > > + * pci_epc_alloc_doorbell() - alloc an address space to let RC trigger EP side IRQ by write data to
> > > > + *			      the space.
> > > 
> > > "Allocate platform specific doorbell IRQs to be used by the host to trigger
> > > doorbells on EP."
> > > 
> > > > + *
> > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > 
> > > EPC device for which the doorbell needs to be allocated
> > > 
> > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > + * @num_msgs: the total number of doorbell messages
> > > 
> > > s/num_msgs/num_db
> > > 
> > > > + *
> > > > + * Return: 0 success, other is failure
> > > > + */
> > > > +int pci_epc_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > > > +{
> > > > +	int ret;
> > > > +
> > > > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > > > +		return -EINVAL;
> > > > +
> > > > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > > > +		return -EINVAL;
> > > > +
> > > > +	if (!epc->ops->alloc_doorbell)
> > > > +		return 0;
> > > 
> > > You mentioned 0 is a success. So if there is no callback, you want to return
> > > success?
> > > 
> > > > +
> > > > +	mutex_lock(&epc->lock);
> > > > +	ret = epc->ops->alloc_doorbell(epc, func_no, vfunc_no, num_msgs);
> > > 
> > > Why can't you just call the generic function here and in other places instead of
> > > implementing callbacks? I do not see a necessity for EPC specific callbacks. If
> > > there is one, please specify.
> > 
> > 1. Refer v1 your comments.
> > https://lore.kernel.org/imx/20230906145227.GC5930@thinkpad/
> 
> I do not find where I suggested the callback approach.

	> > > If that, Each EPF driver need do duplicate work. 
	> > > 
	> > 
	> > Yes, and that's how it should be. EPF core has no job in supplying the of_node.
	> > It is the responsibility of the EPF drivers as they depend on OF for platform
	> > support.
	> 
	> EPF driver still not depend on OF. such pci-epf-test, which was probed by
	> configfs.
	> 

	Hmm, yeah. Then it should be part of the EPC driver.

	Sorry for the confusion.

Here, all "EPF" should be "EPC". The key problem is of_node. EPC core have
not of_node, EPC core's parent driver (like dwc-ep driver) have of_node. 

pci_epc_generic_alloc_doorbell(dev), dev is probed by platform driver, such
as dwc-ep, which have of_node,  EPC core will create child device.

dwc-ep device
 |- epc core device

we can direct call pci_epc_generic_alloc_doorbell(epc->parent) here.

I may miss understand what your means. I think you want to dwc-ep
(with of_node) handle these alloc functions. 

> 
> > 2. Maybe some ep controller have built-in doorbell support. Write to some
> > address to trigger doorbell irq.
> > 
> 
> We will handle it whenever such EP controllers arrive. Until then, let's keep it
> simple.
> 
> - Mani
> 
> > Frank
> > 
> > > 
> > > > +	mutex_unlock(&epc->lock);
> > > > +
> > > > +	return ret;
> > > > +}
> > > > +EXPORT_SYMBOL_GPL(pci_epc_alloc_doorbell);
> > > > +
> > > > +/**
> > > > + * pci_epc_free_doorbell() - free resource allocated by pci_epc_alloc_doorbell()
> > > > + *
> > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > 
> > > Same as above.
> > > 
> > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > + *
> > > > + * Return: 0 success, other is failure
> > > > + */
> > > > +void pci_epc_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
> > > > +{
> > > > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > > > +		return;
> > > > +
> > > > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > > > +		return;
> > > > +
> > > > +	if (!epc->ops->free_doorbell)
> > > > +		return;
> > > > +
> > > > +	mutex_lock(&epc->lock);
> > > > +	epc->ops->free_doorbell(epc, func_no, vfunc_no);
> > > 
> > > Same as suggested above.
> > > 
> > > > +	mutex_unlock(&epc->lock);
> > > > +}
> > > > +EXPORT_SYMBOL_GPL(pci_epc_free_doorbell);
> > > > +
> > > > +static irqreturn_t pci_epf_generic_doorbell_handler(int irq, void *data)
> > > > +{
> > > > +	struct pci_epf *epf = data;
> > > > +
> > > > +	if (epf->event_ops && epf->event_ops->doorbell)
> > > > +		epf->event_ops->doorbell(epf, irq - epf->virq_base);
> > > 
> > > Same as suggested above.
> > > 
> > > > +
> > > > +	return IRQ_HANDLED;
> > > > +}
> > > > +
> > > > +static void pci_epc_generic_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
> > > > +{
> > > > +	struct pci_epc *epc = NULL;
> > > > +	struct class_dev_iter iter;
> > > > +	struct pci_epf *epf;
> > > > +	struct device *dev;
> > > > +
> > > > +	class_dev_iter_init(&iter, pci_epc_class, NULL, NULL);
> > > > +	while ((dev = class_dev_iter_next(&iter))) {
> > > > +		if (dev->parent != desc->dev)
> > > > +			continue;
> > > > +
> > > > +		epc = to_pci_epc(dev);
> > > > +
> > > > +		class_dev_iter_exit(&iter);
> > > > +		break;
> > > > +	}
> > > > +
> > > > +	if (!epc)
> > > > +		return;
> > > > +
> > > > +	/* Only support one EPF for doorbell */
> > > > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > > > +
> > > 
> > > No need of this newline
> > > 
> > > > +	if (!epf)
> > > > +		return;
> > > > +
> > > > +	if (epf->msg && desc->msi_index < epf->num_msgs)
> > > > +		epf->msg[desc->msi_index] = *msg;
> > > > +}
> > > > +
> > > > +
> > > 
> > > Remove extra newline
> > > 
> > > > +/**
> > > > + * pci_epc_generic_alloc_doorbell() - Common help function. Allocate address space from MSI
> > > > + *                                    controller
> > > > + *
> > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > + * @num_msgs: the total number of doorbell messages
> > > > + *
> > > 
> > > Same comment as for pci_epc_alloc_doorbell()
> > > 
> > > > + * Remark: use this function only if EPC driver just register one EPC device.
> > > > + *
> > > > + * Return: 0 success, other is failure
> > > > + */
> > > > +int pci_epc_generic_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > > > +{
> > > > +	struct pci_epf *epf;
> > > > +	struct device *dev;
> > > > +	int virq, last;
> > > > +	int ret;
> > > > +	int i;
> > > > +
> > > > +	if (IS_ERR_OR_NULL(epc))
> > > > +		return -EINVAL;
> > > > +
> > > > +	/* Currently only support one func and one vfunc for doorbell */
> > > > +	if (func_no || vfunc_no)
> > > > +		return -EINVAL;
> > > > +
> > > > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > > > +	if (!epf)
> > > > +		return -EINVAL;
> > > > +
> > > > +	dev = epc->dev.parent;
> > > > +	ret = platform_msi_domain_alloc_irqs(dev, num_msgs, pci_epc_generic_write_msi_msg);
> > > > +	if (ret) {
> > > > +		dev_err(dev, "Failed to allocate MSI\n");
> > > > +		return -ENOMEM;
> > > > +	}
> > > > +
> > > > +	last = -1;
> > > > +	for (i = 0; i < num_msgs; i++) {
> > > 
> > > You should iterate over msi_desc as below:
> > > 
> > >         msi_lock_descs(dev);
> > >         msi_for_each_desc(desc, dev, MSI_DESC_ALL) {
> > > 		...
> > > 	}
> > > 	msi_unlock_descs(dev);
> > > 
> > > > +		virq = msi_get_virq(dev, i);
> > > > +		if (i == 0)
> > > > +			epf->virq_base = virq;
> > > > +
> > > > +		ret = request_irq(virq, pci_epf_generic_doorbell_handler, 0,
> > > 
> > > 	request_irq(desc->irq, ...)
> > > 
> > > > +				  kasprintf(GFP_KERNEL, "pci-epc-doorbell%d", i), epf);
> > > > +
> > > > +		if (ret) {
> > > > +			dev_err(dev, "Failed to request doorbell\n");
> > > > +			goto err_free_irq;
> > > > +		}
> > > > +		last = i;
> > > > +	}
> > > > +
> > > > +	return 0;
> > > > +
> > > > +err_free_irq:
> > > > +	for (i = 0; i < last; i++)
> > > > +		kfree(free_irq(epf->virq_base + i, epf));
> > > > +	platform_msi_domain_free_irqs(dev);
> > > > +
> > > > +	return -EINVAL;
> > > 
> > > 	return ret;
> > > 
> > > > +}
> > > > +EXPORT_SYMBOL_GPL(pci_epc_generic_alloc_doorbell);
> > > > +
> > > 
> > > [...]
> > > 
> > > > diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
> > > > index 3f44b6aec4770..485c146a5efe2 100644
> > > > --- a/include/linux/pci-epf.h
> > > > +++ b/include/linux/pci-epf.h
> > > > @@ -79,6 +79,7 @@ struct pci_epc_event_ops {
> > > >  	int (*link_up)(struct pci_epf *epf);
> > > >  	int (*link_down)(struct pci_epf *epf);
> > > >  	int (*bme)(struct pci_epf *epf);
> > > > +	int (*doorbell)(struct pci_epf *epf, int index);
> > > 
> > > kdoc missing.
> > > 
> > > >  };
> > > >  
> > > >  /**
> > > > @@ -180,6 +181,9 @@ struct pci_epf {
> > > >  	unsigned long		vfunction_num_map;
> > > >  	struct list_head	pci_vepf;
> > > >  	const struct pci_epc_event_ops *event_ops;
> > > > +	struct msi_msg *msg;
> > > > +	u16 num_msgs;
> > > 
> > > num_db
> > > 
> > > You also need to add kdoc for each new member.
> > > 
> > > - Mani
> > > 
> > > -- 
> > > மணிவண்ணன் சதாசிவம்
> 
> -- 
> மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/5] PCI: endpoint: Add RC-to-EP doorbell support using platform MSI controller

[Manivannan Sadhasivam]

On Thu, Oct 19, 2023 at 12:00:22PM -0400, Frank Li wrote:
> On Thu, Oct 19, 2023 at 08:34:41PM +0530, Manivannan Sadhasivam wrote:
> > On Tue, Oct 17, 2023 at 02:55:57PM -0400, Frank Li wrote:
> > > On Wed, Oct 18, 2023 at 12:07:22AM +0530, Manivannan Sadhasivam wrote:
> > > > On Mon, Sep 11, 2023 at 06:09:16PM -0400, Frank Li wrote:
> > > > > This commit introduces a common method for sending messages from the Root
> > > > > Complex (RC) to the Endpoint (EP) by utilizing the platform MSI interrupt
> > > > > controller, such as ARM GIC, as an EP doorbell. Maps the memory assigned
> > > > > for the BAR region by the PCI host to the message address of the platform
> > > > > MSI interrupt controller in the PCI EP. As a result, when the PCI RC writes
> > > > 
> > > > "Doorbell feature is implemented by mapping the EP's MSI interrupt controller
> > > > message address to a dedicated BAR in the EPC core. It is the responsibility
> > > > of the EPF driver to pass the actual message data to be written by the host to
> > > > the doorbell BAR region through its own logic."
> > > > 
> > > > > to the BAR region, it triggers an IRQ at the EP. This implementation serves
> > > > > as a common method for all endpoint function drivers.
> > > > > 
> > > > > However, it currently supports only one EP physical function due to
> > > > > limitations in ARM MSI/IMS readiness.
> > > > > 
> > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > > ---
> > > > >  drivers/pci/endpoint/pci-epc-core.c | 192 ++++++++++++++++++++++++++++
> > > > >  drivers/pci/endpoint/pci-epf-core.c |  44 +++++++
> > > > >  include/linux/pci-epc.h             |   6 +
> > > > >  include/linux/pci-epf.h             |   7 +
> > > > >  4 files changed, 249 insertions(+)
> > > > > 
> > > > > diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
> > > > > index 5a4a8b0be6262..d336a99c6a94f 100644
> > > > > --- a/drivers/pci/endpoint/pci-epc-core.c
> > > > > +++ b/drivers/pci/endpoint/pci-epc-core.c
> > > > > @@ -10,6 +10,7 @@
> > > > >  #include <linux/slab.h>
> > > > >  #include <linux/module.h>
> > > > >  
> > > > > +#include <linux/msi.h>
> > > > >  #include <linux/pci-epc.h>
> > > > >  #include <linux/pci-epf.h>
> > > > >  #include <linux/pci-ep-cfs.h>
> > > > > @@ -783,6 +784,197 @@ void pci_epc_bme_notify(struct pci_epc *epc)
> > > > >  }
> > > > >  EXPORT_SYMBOL_GPL(pci_epc_bme_notify);
> > > > >  
> > > > > +/**
> > > > > + * pci_epc_alloc_doorbell() - alloc an address space to let RC trigger EP side IRQ by write data to
> > > > > + *			      the space.
> > > > 
> > > > "Allocate platform specific doorbell IRQs to be used by the host to trigger
> > > > doorbells on EP."
> > > > 
> > > > > + *
> > > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > 
> > > > EPC device for which the doorbell needs to be allocated
> > > > 
> > > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > > + * @num_msgs: the total number of doorbell messages
> > > > 
> > > > s/num_msgs/num_db
> > > > 
> > > > > + *
> > > > > + * Return: 0 success, other is failure
> > > > > + */
> > > > > +int pci_epc_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > > > > +{
> > > > > +	int ret;
> > > > > +
> > > > > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > > > > +		return -EINVAL;
> > > > > +
> > > > > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > > > > +		return -EINVAL;
> > > > > +
> > > > > +	if (!epc->ops->alloc_doorbell)
> > > > > +		return 0;
> > > > 
> > > > You mentioned 0 is a success. So if there is no callback, you want to return
> > > > success?
> > > > 
> > > > > +
> > > > > +	mutex_lock(&epc->lock);
> > > > > +	ret = epc->ops->alloc_doorbell(epc, func_no, vfunc_no, num_msgs);
> > > > 
> > > > Why can't you just call the generic function here and in other places instead of
> > > > implementing callbacks? I do not see a necessity for EPC specific callbacks. If
> > > > there is one, please specify.
> > > 
> > > 1. Refer v1 your comments.
> > > https://lore.kernel.org/imx/20230906145227.GC5930@thinkpad/
> > 
> > I do not find where I suggested the callback approach.
> 
> 	> > > If that, Each EPF driver need do duplicate work. 
> 	> > > 
> 	> > 
> 	> > Yes, and that's how it should be. EPF core has no job in supplying the of_node.
> 	> > It is the responsibility of the EPF drivers as they depend on OF for platform
> 	> > support.
> 	> 
> 	> EPF driver still not depend on OF. such pci-epf-test, which was probed by
> 	> configfs.
> 	> 
> 
> 	Hmm, yeah. Then it should be part of the EPC driver.
> 
> 	Sorry for the confusion.
> 
> Here, all "EPF" should be "EPC". The key problem is of_node. EPC core have
> not of_node, EPC core's parent driver (like dwc-ep driver) have of_node. 
> 
> pci_epc_generic_alloc_doorbell(dev), dev is probed by platform driver, such
> as dwc-ep, which have of_node,  EPC core will create child device.
> 
> dwc-ep device
>  |- epc core device
> 
> we can direct call pci_epc_generic_alloc_doorbell(epc->parent) here.
> 
> I may miss understand what your means. I think you want to dwc-ep
> (with of_node) handle these alloc functions. 
> 

My comment was to have just one function definition. But looking at it again, I
think it is better to move all the (alloc, free, write_msg) definitions to
dwc-ep, since the contents of those functions are not EPC core specific.

In the EPC core, you can still have the callbacks specific to each EPC. This
also solves your of_node problem.

- Mani

> > 
> > > 2. Maybe some ep controller have built-in doorbell support. Write to some
> > > address to trigger doorbell irq.
> > > 
> > 
> > We will handle it whenever such EP controllers arrive. Until then, let's keep it
> > simple.
> > 
> > - Mani
> > 
> > > Frank
> > > 
> > > > 
> > > > > +	mutex_unlock(&epc->lock);
> > > > > +
> > > > > +	return ret;
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(pci_epc_alloc_doorbell);
> > > > > +
> > > > > +/**
> > > > > + * pci_epc_free_doorbell() - free resource allocated by pci_epc_alloc_doorbell()
> > > > > + *
> > > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > 
> > > > Same as above.
> > > > 
> > > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > > + *
> > > > > + * Return: 0 success, other is failure
> > > > > + */
> > > > > +void pci_epc_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
> > > > > +{
> > > > > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > > > > +		return;
> > > > > +
> > > > > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > > > > +		return;
> > > > > +
> > > > > +	if (!epc->ops->free_doorbell)
> > > > > +		return;
> > > > > +
> > > > > +	mutex_lock(&epc->lock);
> > > > > +	epc->ops->free_doorbell(epc, func_no, vfunc_no);
> > > > 
> > > > Same as suggested above.
> > > > 
> > > > > +	mutex_unlock(&epc->lock);
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(pci_epc_free_doorbell);
> > > > > +
> > > > > +static irqreturn_t pci_epf_generic_doorbell_handler(int irq, void *data)
> > > > > +{
> > > > > +	struct pci_epf *epf = data;
> > > > > +
> > > > > +	if (epf->event_ops && epf->event_ops->doorbell)
> > > > > +		epf->event_ops->doorbell(epf, irq - epf->virq_base);
> > > > 
> > > > Same as suggested above.
> > > > 
> > > > > +
> > > > > +	return IRQ_HANDLED;
> > > > > +}
> > > > > +
> > > > > +static void pci_epc_generic_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
> > > > > +{
> > > > > +	struct pci_epc *epc = NULL;
> > > > > +	struct class_dev_iter iter;
> > > > > +	struct pci_epf *epf;
> > > > > +	struct device *dev;
> > > > > +
> > > > > +	class_dev_iter_init(&iter, pci_epc_class, NULL, NULL);
> > > > > +	while ((dev = class_dev_iter_next(&iter))) {
> > > > > +		if (dev->parent != desc->dev)
> > > > > +			continue;
> > > > > +
> > > > > +		epc = to_pci_epc(dev);
> > > > > +
> > > > > +		class_dev_iter_exit(&iter);
> > > > > +		break;
> > > > > +	}
> > > > > +
> > > > > +	if (!epc)
> > > > > +		return;
> > > > > +
> > > > > +	/* Only support one EPF for doorbell */
> > > > > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > > > > +
> > > > 
> > > > No need of this newline
> > > > 
> > > > > +	if (!epf)
> > > > > +		return;
> > > > > +
> > > > > +	if (epf->msg && desc->msi_index < epf->num_msgs)
> > > > > +		epf->msg[desc->msi_index] = *msg;
> > > > > +}
> > > > > +
> > > > > +
> > > > 
> > > > Remove extra newline
> > > > 
> > > > > +/**
> > > > > + * pci_epc_generic_alloc_doorbell() - Common help function. Allocate address space from MSI
> > > > > + *                                    controller
> > > > > + *
> > > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > > + * @num_msgs: the total number of doorbell messages
> > > > > + *
> > > > 
> > > > Same comment as for pci_epc_alloc_doorbell()
> > > > 
> > > > > + * Remark: use this function only if EPC driver just register one EPC device.
> > > > > + *
> > > > > + * Return: 0 success, other is failure
> > > > > + */
> > > > > +int pci_epc_generic_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > > > > +{
> > > > > +	struct pci_epf *epf;
> > > > > +	struct device *dev;
> > > > > +	int virq, last;
> > > > > +	int ret;
> > > > > +	int i;
> > > > > +
> > > > > +	if (IS_ERR_OR_NULL(epc))
> > > > > +		return -EINVAL;
> > > > > +
> > > > > +	/* Currently only support one func and one vfunc for doorbell */
> > > > > +	if (func_no || vfunc_no)
> > > > > +		return -EINVAL;
> > > > > +
> > > > > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > > > > +	if (!epf)
> > > > > +		return -EINVAL;
> > > > > +
> > > > > +	dev = epc->dev.parent;
> > > > > +	ret = platform_msi_domain_alloc_irqs(dev, num_msgs, pci_epc_generic_write_msi_msg);
> > > > > +	if (ret) {
> > > > > +		dev_err(dev, "Failed to allocate MSI\n");
> > > > > +		return -ENOMEM;
> > > > > +	}
> > > > > +
> > > > > +	last = -1;
> > > > > +	for (i = 0; i < num_msgs; i++) {
> > > > 
> > > > You should iterate over msi_desc as below:
> > > > 
> > > >         msi_lock_descs(dev);
> > > >         msi_for_each_desc(desc, dev, MSI_DESC_ALL) {
> > > > 		...
> > > > 	}
> > > > 	msi_unlock_descs(dev);
> > > > 
> > > > > +		virq = msi_get_virq(dev, i);
> > > > > +		if (i == 0)
> > > > > +			epf->virq_base = virq;
> > > > > +
> > > > > +		ret = request_irq(virq, pci_epf_generic_doorbell_handler, 0,
> > > > 
> > > > 	request_irq(desc->irq, ...)
> > > > 
> > > > > +				  kasprintf(GFP_KERNEL, "pci-epc-doorbell%d", i), epf);
> > > > > +
> > > > > +		if (ret) {
> > > > > +			dev_err(dev, "Failed to request doorbell\n");
> > > > > +			goto err_free_irq;
> > > > > +		}
> > > > > +		last = i;
> > > > > +	}
> > > > > +
> > > > > +	return 0;
> > > > > +
> > > > > +err_free_irq:
> > > > > +	for (i = 0; i < last; i++)
> > > > > +		kfree(free_irq(epf->virq_base + i, epf));
> > > > > +	platform_msi_domain_free_irqs(dev);
> > > > > +
> > > > > +	return -EINVAL;
> > > > 
> > > > 	return ret;
> > > > 
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(pci_epc_generic_alloc_doorbell);
> > > > > +
> > > > 
> > > > [...]
> > > > 
> > > > > diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
> > > > > index 3f44b6aec4770..485c146a5efe2 100644
> > > > > --- a/include/linux/pci-epf.h
> > > > > +++ b/include/linux/pci-epf.h
> > > > > @@ -79,6 +79,7 @@ struct pci_epc_event_ops {
> > > > >  	int (*link_up)(struct pci_epf *epf);
> > > > >  	int (*link_down)(struct pci_epf *epf);
> > > > >  	int (*bme)(struct pci_epf *epf);
> > > > > +	int (*doorbell)(struct pci_epf *epf, int index);
> > > > 
> > > > kdoc missing.
> > > > 
> > > > >  };
> > > > >  
> > > > >  /**
> > > > > @@ -180,6 +181,9 @@ struct pci_epf {
> > > > >  	unsigned long		vfunction_num_map;
> > > > >  	struct list_head	pci_vepf;
> > > > >  	const struct pci_epc_event_ops *event_ops;
> > > > > +	struct msi_msg *msg;
> > > > > +	u16 num_msgs;
> > > > 
> > > > num_db
> > > > 
> > > > You also need to add kdoc for each new member.
> > > > 
> > > > - Mani
> > > > 
> > > > -- 
> > > > மணிவண்ணன் சதாசிவம்
> > 
> > -- 
> > மணிவண்ணன் சதாசிவம்

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/5] PCI: endpoint: Add RC-to-EP doorbell support using platform MSI controllery

[Frank Li]

On Thu, Oct 19, 2023 at 10:53:47PM +0530, Manivannan Sadhasivam wrote:
> On Thu, Oct 19, 2023 at 12:00:22PM -0400, Frank Li wrote:
> > On Thu, Oct 19, 2023 at 08:34:41PM +0530, Manivannan Sadhasivam wrote:
> > > On Tue, Oct 17, 2023 at 02:55:57PM -0400, Frank Li wrote:
> > > > On Wed, Oct 18, 2023 at 12:07:22AM +0530, Manivannan Sadhasivam wrote:
> > > > > On Mon, Sep 11, 2023 at 06:09:16PM -0400, Frank Li wrote:
> > > > > > This commit introduces a common method for sending messages from the Root
> > > > > > Complex (RC) to the Endpoint (EP) by utilizing the platform MSI interrupt
> > > > > > controller, such as ARM GIC, as an EP doorbell. Maps the memory assigned
> > > > > > for the BAR region by the PCI host to the message address of the platform
> > > > > > MSI interrupt controller in the PCI EP. As a result, when the PCI RC writes
> > > > > 
> > > > > "Doorbell feature is implemented by mapping the EP's MSI interrupt controller
> > > > > message address to a dedicated BAR in the EPC core. It is the responsibility
> > > > > of the EPF driver to pass the actual message data to be written by the host to
> > > > > the doorbell BAR region through its own logic."
> > > > > 
> > > > > > to the BAR region, it triggers an IRQ at the EP. This implementation serves
> > > > > > as a common method for all endpoint function drivers.
> > > > > > 
> > > > > > However, it currently supports only one EP physical function due to
> > > > > > limitations in ARM MSI/IMS readiness.
> > > > > > 
> > > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > > > ---
> > > > > >  drivers/pci/endpoint/pci-epc-core.c | 192 ++++++++++++++++++++++++++++
> > > > > >  drivers/pci/endpoint/pci-epf-core.c |  44 +++++++
> > > > > >  include/linux/pci-epc.h             |   6 +
> > > > > >  include/linux/pci-epf.h             |   7 +
> > > > > >  4 files changed, 249 insertions(+)
> > > > > > 
> > > > > > diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
> > > > > > index 5a4a8b0be6262..d336a99c6a94f 100644
> > > > > > --- a/drivers/pci/endpoint/pci-epc-core.c
> > > > > > +++ b/drivers/pci/endpoint/pci-epc-core.c
> > > > > > @@ -10,6 +10,7 @@
> > > > > >  #include <linux/slab.h>
> > > > > >  #include <linux/module.h>
> > > > > >  
> > > > > > +#include <linux/msi.h>
> > > > > >  #include <linux/pci-epc.h>
> > > > > >  #include <linux/pci-epf.h>
> > > > > >  #include <linux/pci-ep-cfs.h>
> > > > > > @@ -783,6 +784,197 @@ void pci_epc_bme_notify(struct pci_epc *epc)
> > > > > >  }
> > > > > >  EXPORT_SYMBOL_GPL(pci_epc_bme_notify);
> > > > > >  
> > > > > > +/**
> > > > > > + * pci_epc_alloc_doorbell() - alloc an address space to let RC trigger EP side IRQ by write data to
> > > > > > + *			      the space.
> > > > > 
> > > > > "Allocate platform specific doorbell IRQs to be used by the host to trigger
> > > > > doorbells on EP."
> > > > > 
> > > > > > + *
> > > > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > > 
> > > > > EPC device for which the doorbell needs to be allocated
> > > > > 
> > > > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > > > + * @num_msgs: the total number of doorbell messages
> > > > > 
> > > > > s/num_msgs/num_db
> > > > > 
> > > > > > + *
> > > > > > + * Return: 0 success, other is failure
> > > > > > + */
> > > > > > +int pci_epc_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > > > > > +{
> > > > > > +	int ret;
> > > > > > +
> > > > > > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > > > > > +		return -EINVAL;
> > > > > > +
> > > > > > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > > > > > +		return -EINVAL;
> > > > > > +
> > > > > > +	if (!epc->ops->alloc_doorbell)
> > > > > > +		return 0;
> > > > > 
> > > > > You mentioned 0 is a success. So if there is no callback, you want to return
> > > > > success?
> > > > > 
> > > > > > +
> > > > > > +	mutex_lock(&epc->lock);
> > > > > > +	ret = epc->ops->alloc_doorbell(epc, func_no, vfunc_no, num_msgs);
> > > > > 
> > > > > Why can't you just call the generic function here and in other places instead of
> > > > > implementing callbacks? I do not see a necessity for EPC specific callbacks. If
> > > > > there is one, please specify.
> > > > 
> > > > 1. Refer v1 your comments.
> > > > https://lore.kernel.org/imx/20230906145227.GC5930@thinkpad/
> > > 
> > > I do not find where I suggested the callback approach.
> > 
> > 	> > > If that, Each EPF driver need do duplicate work. 
> > 	> > > 
> > 	> > 
> > 	> > Yes, and that's how it should be. EPF core has no job in supplying the of_node.
> > 	> > It is the responsibility of the EPF drivers as they depend on OF for platform
> > 	> > support.
> > 	> 
> > 	> EPF driver still not depend on OF. such pci-epf-test, which was probed by
> > 	> configfs.
> > 	> 
> > 
> > 	Hmm, yeah. Then it should be part of the EPC driver.
> > 
> > 	Sorry for the confusion.
> > 
> > Here, all "EPF" should be "EPC". The key problem is of_node. EPC core have
> > not of_node, EPC core's parent driver (like dwc-ep driver) have of_node. 
> > 
> > pci_epc_generic_alloc_doorbell(dev), dev is probed by platform driver, such
> > as dwc-ep, which have of_node,  EPC core will create child device.
> > 
> > dwc-ep device
> >  |- epc core device
> > 
> > we can direct call pci_epc_generic_alloc_doorbell(epc->parent) here.
> > 
> > I may miss understand what your means. I think you want to dwc-ep
> > (with of_node) handle these alloc functions. 
> > 
> 
> My comment was to have just one function definition. But looking at it again, I
> think it is better to move all the (alloc, free, write_msg) definitions to
> dwc-ep, since the contents of those functions are not EPC core specific.

There are still problem. (alloc, free, write_msg) is quite common for all
controller and the system with MSI.

If move these into dwc-ep,  cdns or other controller have to duplicate 
these codes.

If you think it is not EPC core specific, how about create new help files?

Frank

> 
> In the EPC core, you can still have the callbacks specific to each EPC. This
> also solves your of_node problem.
> 
> - Mani
> 
> > > 
> > > > 2. Maybe some ep controller have built-in doorbell support. Write to some
> > > > address to trigger doorbell irq.
> > > > 
> > > 
> > > We will handle it whenever such EP controllers arrive. Until then, let's keep it
> > > simple.
> > > 
> > > - Mani
> > > 
> > > > Frank
> > > > 
> > > > > 
> > > > > > +	mutex_unlock(&epc->lock);
> > > > > > +
> > > > > > +	return ret;
> > > > > > +}
> > > > > > +EXPORT_SYMBOL_GPL(pci_epc_alloc_doorbell);
> > > > > > +
> > > > > > +/**
> > > > > > + * pci_epc_free_doorbell() - free resource allocated by pci_epc_alloc_doorbell()
> > > > > > + *
> > > > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > > 
> > > > > Same as above.
> > > > > 
> > > > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > > > + *
> > > > > > + * Return: 0 success, other is failure
> > > > > > + */
> > > > > > +void pci_epc_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
> > > > > > +{
> > > > > > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > > > > > +		return;
> > > > > > +
> > > > > > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > > > > > +		return;
> > > > > > +
> > > > > > +	if (!epc->ops->free_doorbell)
> > > > > > +		return;
> > > > > > +
> > > > > > +	mutex_lock(&epc->lock);
> > > > > > +	epc->ops->free_doorbell(epc, func_no, vfunc_no);
> > > > > 
> > > > > Same as suggested above.
> > > > > 
> > > > > > +	mutex_unlock(&epc->lock);
> > > > > > +}
> > > > > > +EXPORT_SYMBOL_GPL(pci_epc_free_doorbell);
> > > > > > +
> > > > > > +static irqreturn_t pci_epf_generic_doorbell_handler(int irq, void *data)
> > > > > > +{
> > > > > > +	struct pci_epf *epf = data;
> > > > > > +
> > > > > > +	if (epf->event_ops && epf->event_ops->doorbell)
> > > > > > +		epf->event_ops->doorbell(epf, irq - epf->virq_base);
> > > > > 
> > > > > Same as suggested above.
> > > > > 
> > > > > > +
> > > > > > +	return IRQ_HANDLED;
> > > > > > +}
> > > > > > +
> > > > > > +static void pci_epc_generic_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
> > > > > > +{
> > > > > > +	struct pci_epc *epc = NULL;
> > > > > > +	struct class_dev_iter iter;
> > > > > > +	struct pci_epf *epf;
> > > > > > +	struct device *dev;
> > > > > > +
> > > > > > +	class_dev_iter_init(&iter, pci_epc_class, NULL, NULL);
> > > > > > +	while ((dev = class_dev_iter_next(&iter))) {
> > > > > > +		if (dev->parent != desc->dev)
> > > > > > +			continue;
> > > > > > +
> > > > > > +		epc = to_pci_epc(dev);
> > > > > > +
> > > > > > +		class_dev_iter_exit(&iter);
> > > > > > +		break;
> > > > > > +	}
> > > > > > +
> > > > > > +	if (!epc)
> > > > > > +		return;
> > > > > > +
> > > > > > +	/* Only support one EPF for doorbell */
> > > > > > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > > > > > +
> > > > > 
> > > > > No need of this newline
> > > > > 
> > > > > > +	if (!epf)
> > > > > > +		return;
> > > > > > +
> > > > > > +	if (epf->msg && desc->msi_index < epf->num_msgs)
> > > > > > +		epf->msg[desc->msi_index] = *msg;
> > > > > > +}
> > > > > > +
> > > > > > +
> > > > > 
> > > > > Remove extra newline
> > > > > 
> > > > > > +/**
> > > > > > + * pci_epc_generic_alloc_doorbell() - Common help function. Allocate address space from MSI
> > > > > > + *                                    controller
> > > > > > + *
> > > > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > > > + * @num_msgs: the total number of doorbell messages
> > > > > > + *
> > > > > 
> > > > > Same comment as for pci_epc_alloc_doorbell()
> > > > > 
> > > > > > + * Remark: use this function only if EPC driver just register one EPC device.
> > > > > > + *
> > > > > > + * Return: 0 success, other is failure
> > > > > > + */
> > > > > > +int pci_epc_generic_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > > > > > +{
> > > > > > +	struct pci_epf *epf;
> > > > > > +	struct device *dev;
> > > > > > +	int virq, last;
> > > > > > +	int ret;
> > > > > > +	int i;
> > > > > > +
> > > > > > +	if (IS_ERR_OR_NULL(epc))
> > > > > > +		return -EINVAL;
> > > > > > +
> > > > > > +	/* Currently only support one func and one vfunc for doorbell */
> > > > > > +	if (func_no || vfunc_no)
> > > > > > +		return -EINVAL;
> > > > > > +
> > > > > > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > > > > > +	if (!epf)
> > > > > > +		return -EINVAL;
> > > > > > +
> > > > > > +	dev = epc->dev.parent;
> > > > > > +	ret = platform_msi_domain_alloc_irqs(dev, num_msgs, pci_epc_generic_write_msi_msg);
> > > > > > +	if (ret) {
> > > > > > +		dev_err(dev, "Failed to allocate MSI\n");
> > > > > > +		return -ENOMEM;
> > > > > > +	}
> > > > > > +
> > > > > > +	last = -1;
> > > > > > +	for (i = 0; i < num_msgs; i++) {
> > > > > 
> > > > > You should iterate over msi_desc as below:
> > > > > 
> > > > >         msi_lock_descs(dev);
> > > > >         msi_for_each_desc(desc, dev, MSI_DESC_ALL) {
> > > > > 		...
> > > > > 	}
> > > > > 	msi_unlock_descs(dev);
> > > > > 
> > > > > > +		virq = msi_get_virq(dev, i);
> > > > > > +		if (i == 0)
> > > > > > +			epf->virq_base = virq;
> > > > > > +
> > > > > > +		ret = request_irq(virq, pci_epf_generic_doorbell_handler, 0,
> > > > > 
> > > > > 	request_irq(desc->irq, ...)
> > > > > 
> > > > > > +				  kasprintf(GFP_KERNEL, "pci-epc-doorbell%d", i), epf);
> > > > > > +
> > > > > > +		if (ret) {
> > > > > > +			dev_err(dev, "Failed to request doorbell\n");
> > > > > > +			goto err_free_irq;
> > > > > > +		}
> > > > > > +		last = i;
> > > > > > +	}
> > > > > > +
> > > > > > +	return 0;
> > > > > > +
> > > > > > +err_free_irq:
> > > > > > +	for (i = 0; i < last; i++)
> > > > > > +		kfree(free_irq(epf->virq_base + i, epf));
> > > > > > +	platform_msi_domain_free_irqs(dev);
> > > > > > +
> > > > > > +	return -EINVAL;
> > > > > 
> > > > > 	return ret;
> > > > > 
> > > > > > +}
> > > > > > +EXPORT_SYMBOL_GPL(pci_epc_generic_alloc_doorbell);
> > > > > > +
> > > > > 
> > > > > [...]
> > > > > 
> > > > > > diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
> > > > > > index 3f44b6aec4770..485c146a5efe2 100644
> > > > > > --- a/include/linux/pci-epf.h
> > > > > > +++ b/include/linux/pci-epf.h
> > > > > > @@ -79,6 +79,7 @@ struct pci_epc_event_ops {
> > > > > >  	int (*link_up)(struct pci_epf *epf);
> > > > > >  	int (*link_down)(struct pci_epf *epf);
> > > > > >  	int (*bme)(struct pci_epf *epf);
> > > > > > +	int (*doorbell)(struct pci_epf *epf, int index);
> > > > > 
> > > > > kdoc missing.
> > > > > 
> > > > > >  };
> > > > > >  
> > > > > >  /**
> > > > > > @@ -180,6 +181,9 @@ struct pci_epf {
> > > > > >  	unsigned long		vfunction_num_map;
> > > > > >  	struct list_head	pci_vepf;
> > > > > >  	const struct pci_epc_event_ops *event_ops;
> > > > > > +	struct msi_msg *msg;
> > > > > > +	u16 num_msgs;
> > > > > 
> > > > > num_db
> > > > > 
> > > > > You also need to add kdoc for each new member.
> > > > > 
> > > > > - Mani
> > > > > 
> > > > > -- 
> > > > > மணிவண்ணன் சதாசிவம்
> > > 
> > > -- 
> > > மணிவண்ணன் சதாசிவம்
> 
> -- 
> மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/5] PCI: endpoint: Add RC-to-EP doorbell support using platform MSI controllery

[Manivannan Sadhasivam]

On Thu, Oct 19, 2023 at 02:11:22PM -0400, Frank Li wrote:
> On Thu, Oct 19, 2023 at 10:53:47PM +0530, Manivannan Sadhasivam wrote:
> > On Thu, Oct 19, 2023 at 12:00:22PM -0400, Frank Li wrote:
> > > On Thu, Oct 19, 2023 at 08:34:41PM +0530, Manivannan Sadhasivam wrote:
> > > > On Tue, Oct 17, 2023 at 02:55:57PM -0400, Frank Li wrote:
> > > > > On Wed, Oct 18, 2023 at 12:07:22AM +0530, Manivannan Sadhasivam wrote:
> > > > > > On Mon, Sep 11, 2023 at 06:09:16PM -0400, Frank Li wrote:
> > > > > > > This commit introduces a common method for sending messages from the Root
> > > > > > > Complex (RC) to the Endpoint (EP) by utilizing the platform MSI interrupt
> > > > > > > controller, such as ARM GIC, as an EP doorbell. Maps the memory assigned
> > > > > > > for the BAR region by the PCI host to the message address of the platform
> > > > > > > MSI interrupt controller in the PCI EP. As a result, when the PCI RC writes
> > > > > > 
> > > > > > "Doorbell feature is implemented by mapping the EP's MSI interrupt controller
> > > > > > message address to a dedicated BAR in the EPC core. It is the responsibility
> > > > > > of the EPF driver to pass the actual message data to be written by the host to
> > > > > > the doorbell BAR region through its own logic."
> > > > > > 
> > > > > > > to the BAR region, it triggers an IRQ at the EP. This implementation serves
> > > > > > > as a common method for all endpoint function drivers.
> > > > > > > 
> > > > > > > However, it currently supports only one EP physical function due to
> > > > > > > limitations in ARM MSI/IMS readiness.
> > > > > > > 
> > > > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > > > > ---
> > > > > > >  drivers/pci/endpoint/pci-epc-core.c | 192 ++++++++++++++++++++++++++++
> > > > > > >  drivers/pci/endpoint/pci-epf-core.c |  44 +++++++
> > > > > > >  include/linux/pci-epc.h             |   6 +
> > > > > > >  include/linux/pci-epf.h             |   7 +
> > > > > > >  4 files changed, 249 insertions(+)
> > > > > > > 
> > > > > > > diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
> > > > > > > index 5a4a8b0be6262..d336a99c6a94f 100644
> > > > > > > --- a/drivers/pci/endpoint/pci-epc-core.c
> > > > > > > +++ b/drivers/pci/endpoint/pci-epc-core.c
> > > > > > > @@ -10,6 +10,7 @@
> > > > > > >  #include <linux/slab.h>
> > > > > > >  #include <linux/module.h>
> > > > > > >  
> > > > > > > +#include <linux/msi.h>
> > > > > > >  #include <linux/pci-epc.h>
> > > > > > >  #include <linux/pci-epf.h>
> > > > > > >  #include <linux/pci-ep-cfs.h>
> > > > > > > @@ -783,6 +784,197 @@ void pci_epc_bme_notify(struct pci_epc *epc)
> > > > > > >  }
> > > > > > >  EXPORT_SYMBOL_GPL(pci_epc_bme_notify);
> > > > > > >  
> > > > > > > +/**
> > > > > > > + * pci_epc_alloc_doorbell() - alloc an address space to let RC trigger EP side IRQ by write data to
> > > > > > > + *			      the space.
> > > > > > 
> > > > > > "Allocate platform specific doorbell IRQs to be used by the host to trigger
> > > > > > doorbells on EP."
> > > > > > 
> > > > > > > + *
> > > > > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > > > 
> > > > > > EPC device for which the doorbell needs to be allocated
> > > > > > 
> > > > > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > > > > + * @num_msgs: the total number of doorbell messages
> > > > > > 
> > > > > > s/num_msgs/num_db
> > > > > > 
> > > > > > > + *
> > > > > > > + * Return: 0 success, other is failure
> > > > > > > + */
> > > > > > > +int pci_epc_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > > > > > > +{
> > > > > > > +	int ret;
> > > > > > > +
> > > > > > > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > > > > > > +		return -EINVAL;
> > > > > > > +
> > > > > > > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > > > > > > +		return -EINVAL;
> > > > > > > +
> > > > > > > +	if (!epc->ops->alloc_doorbell)
> > > > > > > +		return 0;
> > > > > > 
> > > > > > You mentioned 0 is a success. So if there is no callback, you want to return
> > > > > > success?
> > > > > > 
> > > > > > > +
> > > > > > > +	mutex_lock(&epc->lock);
> > > > > > > +	ret = epc->ops->alloc_doorbell(epc, func_no, vfunc_no, num_msgs);
> > > > > > 
> > > > > > Why can't you just call the generic function here and in other places instead of
> > > > > > implementing callbacks? I do not see a necessity for EPC specific callbacks. If
> > > > > > there is one, please specify.
> > > > > 
> > > > > 1. Refer v1 your comments.
> > > > > https://lore.kernel.org/imx/20230906145227.GC5930@thinkpad/
> > > > 
> > > > I do not find where I suggested the callback approach.
> > > 
> > > 	> > > If that, Each EPF driver need do duplicate work. 
> > > 	> > > 
> > > 	> > 
> > > 	> > Yes, and that's how it should be. EPF core has no job in supplying the of_node.
> > > 	> > It is the responsibility of the EPF drivers as they depend on OF for platform
> > > 	> > support.
> > > 	> 
> > > 	> EPF driver still not depend on OF. such pci-epf-test, which was probed by
> > > 	> configfs.
> > > 	> 
> > > 
> > > 	Hmm, yeah. Then it should be part of the EPC driver.
> > > 
> > > 	Sorry for the confusion.
> > > 
> > > Here, all "EPF" should be "EPC". The key problem is of_node. EPC core have
> > > not of_node, EPC core's parent driver (like dwc-ep driver) have of_node. 
> > > 
> > > pci_epc_generic_alloc_doorbell(dev), dev is probed by platform driver, such
> > > as dwc-ep, which have of_node,  EPC core will create child device.
> > > 
> > > dwc-ep device
> > >  |- epc core device
> > > 
> > > we can direct call pci_epc_generic_alloc_doorbell(epc->parent) here.
> > > 
> > > I may miss understand what your means. I think you want to dwc-ep
> > > (with of_node) handle these alloc functions. 
> > > 
> > 
> > My comment was to have just one function definition. But looking at it again, I
> > think it is better to move all the (alloc, free, write_msg) definitions to
> > dwc-ep, since the contents of those functions are not EPC core specific.
> 
> There are still problem. (alloc, free, write_msg) is quite common for all
> controller and the system with MSI.
> 
> If move these into dwc-ep,  cdns or other controller have to duplicate 
> these codes.
> 
> If you think it is not EPC core specific, how about create new help files?
> 

Hmm, that sounds good to me. I think the best place would be:
drivers/pci/endpoint/pci-ep-msi.c

Reason is, we cannot have this generic code under drivers/pci/controller/ as it
is not a standalone PCI controller but a platform MSI controller. So having it
under pci/endpoint/ makes much sense to me.

And this is not specific to EPF drivers as well, so we cannot have it under
pci/endpoint/functions/.

- Mani

> Frank
> 
> > 
> > In the EPC core, you can still have the callbacks specific to each EPC. This
> > also solves your of_node problem.
> > 
> > - Mani
> > 
> > > > 
> > > > > 2. Maybe some ep controller have built-in doorbell support. Write to some
> > > > > address to trigger doorbell irq.
> > > > > 
> > > > 
> > > > We will handle it whenever such EP controllers arrive. Until then, let's keep it
> > > > simple.
> > > > 
> > > > - Mani
> > > > 
> > > > > Frank
> > > > > 
> > > > > > 
> > > > > > > +	mutex_unlock(&epc->lock);
> > > > > > > +
> > > > > > > +	return ret;
> > > > > > > +}
> > > > > > > +EXPORT_SYMBOL_GPL(pci_epc_alloc_doorbell);
> > > > > > > +
> > > > > > > +/**
> > > > > > > + * pci_epc_free_doorbell() - free resource allocated by pci_epc_alloc_doorbell()
> > > > > > > + *
> > > > > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > > > 
> > > > > > Same as above.
> > > > > > 
> > > > > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > > > > + *
> > > > > > > + * Return: 0 success, other is failure
> > > > > > > + */
> > > > > > > +void pci_epc_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
> > > > > > > +{
> > > > > > > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > > > > > > +		return;
> > > > > > > +
> > > > > > > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > > > > > > +		return;
> > > > > > > +
> > > > > > > +	if (!epc->ops->free_doorbell)
> > > > > > > +		return;
> > > > > > > +
> > > > > > > +	mutex_lock(&epc->lock);
> > > > > > > +	epc->ops->free_doorbell(epc, func_no, vfunc_no);
> > > > > > 
> > > > > > Same as suggested above.
> > > > > > 
> > > > > > > +	mutex_unlock(&epc->lock);
> > > > > > > +}
> > > > > > > +EXPORT_SYMBOL_GPL(pci_epc_free_doorbell);
> > > > > > > +
> > > > > > > +static irqreturn_t pci_epf_generic_doorbell_handler(int irq, void *data)
> > > > > > > +{
> > > > > > > +	struct pci_epf *epf = data;
> > > > > > > +
> > > > > > > +	if (epf->event_ops && epf->event_ops->doorbell)
> > > > > > > +		epf->event_ops->doorbell(epf, irq - epf->virq_base);
> > > > > > 
> > > > > > Same as suggested above.
> > > > > > 
> > > > > > > +
> > > > > > > +	return IRQ_HANDLED;
> > > > > > > +}
> > > > > > > +
> > > > > > > +static void pci_epc_generic_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
> > > > > > > +{
> > > > > > > +	struct pci_epc *epc = NULL;
> > > > > > > +	struct class_dev_iter iter;
> > > > > > > +	struct pci_epf *epf;
> > > > > > > +	struct device *dev;
> > > > > > > +
> > > > > > > +	class_dev_iter_init(&iter, pci_epc_class, NULL, NULL);
> > > > > > > +	while ((dev = class_dev_iter_next(&iter))) {
> > > > > > > +		if (dev->parent != desc->dev)
> > > > > > > +			continue;
> > > > > > > +
> > > > > > > +		epc = to_pci_epc(dev);
> > > > > > > +
> > > > > > > +		class_dev_iter_exit(&iter);
> > > > > > > +		break;
> > > > > > > +	}
> > > > > > > +
> > > > > > > +	if (!epc)
> > > > > > > +		return;
> > > > > > > +
> > > > > > > +	/* Only support one EPF for doorbell */
> > > > > > > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > > > > > > +
> > > > > > 
> > > > > > No need of this newline
> > > > > > 
> > > > > > > +	if (!epf)
> > > > > > > +		return;
> > > > > > > +
> > > > > > > +	if (epf->msg && desc->msi_index < epf->num_msgs)
> > > > > > > +		epf->msg[desc->msi_index] = *msg;
> > > > > > > +}
> > > > > > > +
> > > > > > > +
> > > > > > 
> > > > > > Remove extra newline
> > > > > > 
> > > > > > > +/**
> > > > > > > + * pci_epc_generic_alloc_doorbell() - Common help function. Allocate address space from MSI
> > > > > > > + *                                    controller
> > > > > > > + *
> > > > > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > > > > + * @num_msgs: the total number of doorbell messages
> > > > > > > + *
> > > > > > 
> > > > > > Same comment as for pci_epc_alloc_doorbell()
> > > > > > 
> > > > > > > + * Remark: use this function only if EPC driver just register one EPC device.
> > > > > > > + *
> > > > > > > + * Return: 0 success, other is failure
> > > > > > > + */
> > > > > > > +int pci_epc_generic_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > > > > > > +{
> > > > > > > +	struct pci_epf *epf;
> > > > > > > +	struct device *dev;
> > > > > > > +	int virq, last;
> > > > > > > +	int ret;
> > > > > > > +	int i;
> > > > > > > +
> > > > > > > +	if (IS_ERR_OR_NULL(epc))
> > > > > > > +		return -EINVAL;
> > > > > > > +
> > > > > > > +	/* Currently only support one func and one vfunc for doorbell */
> > > > > > > +	if (func_no || vfunc_no)
> > > > > > > +		return -EINVAL;
> > > > > > > +
> > > > > > > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > > > > > > +	if (!epf)
> > > > > > > +		return -EINVAL;
> > > > > > > +
> > > > > > > +	dev = epc->dev.parent;
> > > > > > > +	ret = platform_msi_domain_alloc_irqs(dev, num_msgs, pci_epc_generic_write_msi_msg);
> > > > > > > +	if (ret) {
> > > > > > > +		dev_err(dev, "Failed to allocate MSI\n");
> > > > > > > +		return -ENOMEM;
> > > > > > > +	}
> > > > > > > +
> > > > > > > +	last = -1;
> > > > > > > +	for (i = 0; i < num_msgs; i++) {
> > > > > > 
> > > > > > You should iterate over msi_desc as below:
> > > > > > 
> > > > > >         msi_lock_descs(dev);
> > > > > >         msi_for_each_desc(desc, dev, MSI_DESC_ALL) {
> > > > > > 		...
> > > > > > 	}
> > > > > > 	msi_unlock_descs(dev);
> > > > > > 
> > > > > > > +		virq = msi_get_virq(dev, i);
> > > > > > > +		if (i == 0)
> > > > > > > +			epf->virq_base = virq;
> > > > > > > +
> > > > > > > +		ret = request_irq(virq, pci_epf_generic_doorbell_handler, 0,
> > > > > > 
> > > > > > 	request_irq(desc->irq, ...)
> > > > > > 
> > > > > > > +				  kasprintf(GFP_KERNEL, "pci-epc-doorbell%d", i), epf);
> > > > > > > +
> > > > > > > +		if (ret) {
> > > > > > > +			dev_err(dev, "Failed to request doorbell\n");
> > > > > > > +			goto err_free_irq;
> > > > > > > +		}
> > > > > > > +		last = i;
> > > > > > > +	}
> > > > > > > +
> > > > > > > +	return 0;
> > > > > > > +
> > > > > > > +err_free_irq:
> > > > > > > +	for (i = 0; i < last; i++)
> > > > > > > +		kfree(free_irq(epf->virq_base + i, epf));
> > > > > > > +	platform_msi_domain_free_irqs(dev);
> > > > > > > +
> > > > > > > +	return -EINVAL;
> > > > > > 
> > > > > > 	return ret;
> > > > > > 
> > > > > > > +}
> > > > > > > +EXPORT_SYMBOL_GPL(pci_epc_generic_alloc_doorbell);
> > > > > > > +
> > > > > > 
> > > > > > [...]
> > > > > > 
> > > > > > > diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
> > > > > > > index 3f44b6aec4770..485c146a5efe2 100644
> > > > > > > --- a/include/linux/pci-epf.h
> > > > > > > +++ b/include/linux/pci-epf.h
> > > > > > > @@ -79,6 +79,7 @@ struct pci_epc_event_ops {
> > > > > > >  	int (*link_up)(struct pci_epf *epf);
> > > > > > >  	int (*link_down)(struct pci_epf *epf);
> > > > > > >  	int (*bme)(struct pci_epf *epf);
> > > > > > > +	int (*doorbell)(struct pci_epf *epf, int index);
> > > > > > 
> > > > > > kdoc missing.
> > > > > > 
> > > > > > >  };
> > > > > > >  
> > > > > > >  /**
> > > > > > > @@ -180,6 +181,9 @@ struct pci_epf {
> > > > > > >  	unsigned long		vfunction_num_map;
> > > > > > >  	struct list_head	pci_vepf;
> > > > > > >  	const struct pci_epc_event_ops *event_ops;
> > > > > > > +	struct msi_msg *msg;
> > > > > > > +	u16 num_msgs;
> > > > > > 
> > > > > > num_db
> > > > > > 
> > > > > > You also need to add kdoc for each new member.
> > > > > > 
> > > > > > - Mani
> > > > > > 
> > > > > > -- 
> > > > > > மணிவண்ணன் சதாசிவம்
> > > > 
> > > > -- 
> > > > மணிவண்ணன் சதாசிவம்
> > 
> > -- 
> > மணிவண்ணன் சதாசிவம்

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 3/5] PCI: endpoint: pci-epf-test: add doorbell test

[Manivannan Sadhasivam]

On Mon, Sep 11, 2023 at 06:09:18PM -0400, Frank Li wrote:

Subject could be,

PCI: endpoint: pci-epf-test: Add doorbell support

> Add three register: doorbell_bar, doorbell_addr, doorbell_data,
> doorbell_done. Call pci_epf_alloc_doorbell() all a doorbell address space.
> 
> Root complex(RC) side driver can trigger pci-epc-test's doorbell callback
> handler by write doorbell_data to mapped doorbell_bar's address space.
> 
> pci-epc-test will set doorbell_done in doorbell callback.
> 

How about,

Add doorbell support to the EPF test driver by introducing 3 new registers:

doorbell_bar
doorbell_addr
doorbell_data

The PCI RC driver can trigger the doorbell on the EP side by writing the
content of "doorbell_data" to the address specified by the "doorbell_addr"
register in the "doorbell_bar" BAR region.

> Signed-off-by: Frank Li <Frank.Li@nxp.com>

You should also update Documentation/PCI/endpoint/pci-test-* files in a separate
commit with doorbell support.

> ---
>  drivers/pci/endpoint/functions/pci-epf-test.c | 59 ++++++++++++++++++-
>  1 file changed, 58 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c b/drivers/pci/endpoint/functions/pci-epf-test.c
> index 1f0d2b84296a3..566549919b87b 100644
> --- a/drivers/pci/endpoint/functions/pci-epf-test.c
> +++ b/drivers/pci/endpoint/functions/pci-epf-test.c
> @@ -11,6 +11,7 @@
>  #include <linux/dmaengine.h>
>  #include <linux/io.h>
>  #include <linux/module.h>
> +#include <linux/msi.h>
>  #include <linux/slab.h>
>  #include <linux/pci_ids.h>
>  #include <linux/random.h>
> @@ -39,17 +40,21 @@
>  #define STATUS_IRQ_RAISED		BIT(6)
>  #define STATUS_SRC_ADDR_INVALID		BIT(7)
>  #define STATUS_DST_ADDR_INVALID		BIT(8)
> +#define STATUS_DOORBELL_SUCCESS		BIT(9)
>  
>  #define FLAG_USE_DMA			BIT(0)
>  
>  #define TIMER_RESOLUTION		1
>  
> +#define MAGIC_VERSION_MASK		GENMASK(7, 0)
> +
>  static struct workqueue_struct *kpcitest_workqueue;
>  
>  struct pci_epf_test {
>  	void			*reg[PCI_STD_NUM_BARS];
>  	struct pci_epf		*epf;
>  	enum pci_barno		test_reg_bar;
> +	enum pci_barno		doorbell_bar;
>  	size_t			msix_table_offset;
>  	struct delayed_work	cmd_handler;
>  	struct dma_chan		*dma_chan_tx;
> @@ -74,6 +79,9 @@ struct pci_epf_test_reg {
>  	u32	irq_type;
>  	u32	irq_number;
>  	u32	flags;
> +	u32	doorbell_bar;
> +	u32	doorbell_addr;
> +	u32	doorbell_data;
>  } __packed;
>  
>  static struct pci_epf_header test_header = {
> @@ -693,6 +701,8 @@ static void pci_epf_test_unbind(struct pci_epf *epf)
>  	struct pci_epf_bar *epf_bar;
>  	int bar;
>  
> +	pci_epf_free_doorbell(epf);
> +
>  	cancel_delayed_work(&epf_test->cmd_handler);
>  	pci_epf_test_clean_dma_chan(epf_test);
>  	for (bar = 0; bar < PCI_STD_NUM_BARS; bar++) {
> @@ -808,9 +818,22 @@ static int pci_epf_test_link_up(struct pci_epf *epf)
>  	return 0;
>  }
>  
> +static int pci_epf_test_doorbell(struct pci_epf *epf, int index)
> +{
> +	struct pci_epf_test *epf_test = epf_get_drvdata(epf);
> +	enum pci_barno test_reg_bar = epf_test->test_reg_bar;
> +	struct pci_epf_test_reg *reg = epf_test->reg[test_reg_bar];
> +
> +	reg->status |= STATUS_DOORBELL_SUCCESS;
> +	pci_epf_test_raise_irq(epf_test, reg);
> +
> +	return 0;
> +}
> +
>  static const struct pci_epc_event_ops pci_epf_test_event_ops = {
>  	.core_init = pci_epf_test_core_init,
>  	.link_up = pci_epf_test_link_up,
> +	.doorbell = pci_epf_test_doorbell,

I would like to pass this callback directly to the pci_epf_alloc_doorbell() API.
 Would that be feasible?

>  };
>  
>  static int pci_epf_test_alloc_space(struct pci_epf *epf)
> @@ -859,7 +882,7 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf)
>  		epf_bar = &epf->bar[bar];
>  		add = (epf_bar->flags & PCI_BASE_ADDRESS_MEM_TYPE_64) ? 2 : 1;
>  
> -		if (bar == test_reg_bar)
> +		if (bar == test_reg_bar || bar == epf_test->doorbell_bar)
>  			continue;
>  
>  		if (!!(epc_features->reserved_bar & (1 << bar)))
> @@ -900,9 +923,14 @@ static int pci_epf_test_bind(struct pci_epf *epf)
>  	struct pci_epf_test *epf_test = epf_get_drvdata(epf);
>  	const struct pci_epc_features *epc_features;
>  	enum pci_barno test_reg_bar = BAR_0;
> +	enum pci_barno doorbell_bar = NO_BAR;
>  	struct pci_epc *epc = epf->epc;
>  	bool linkup_notifier = false;
>  	bool core_init_notifier = false;
> +	struct pci_epf_test_reg *reg;
> +	struct msi_msg *msg;
> +	u64 doorbell_addr;
> +	u32 align;
>  
>  	if (WARN_ON_ONCE(!epc))
>  		return -EINVAL;
> @@ -923,10 +951,39 @@ static int pci_epf_test_bind(struct pci_epf *epf)
>  	epf_test->test_reg_bar = test_reg_bar;
>  	epf_test->epc_features = epc_features;
>  
> +	align = epc_features->align;
> +	align = align ? align : 128;
> +
> +	ret = pci_epf_alloc_doorbell(epf, 1);

This should be renamed as pci_epc_alloc_doorbell() as per comment on patch 1/3.
Also, the "msi_msg" pointer should be part of the EPC struct.

> +	if (!ret) {
> +		msg = epf->msg;
> +		doorbell_bar = pci_epc_get_next_free_bar(epc_features, test_reg_bar + 1);
> +
> +		if (doorbell_bar > 0) {
> +			epf_test->doorbell_bar = doorbell_bar;
> +			doorbell_addr = msg->address_hi;
> +			doorbell_addr <<= 32;
> +			doorbell_addr |= msg->address_lo;
> +			epf->bar[doorbell_bar].phys_addr = round_down(doorbell_addr, align);
> +			epf->bar[doorbell_bar].barno = doorbell_bar;
> +			epf->bar[doorbell_bar].size = align;
> +		} else {
> +			pci_epf_free_doorbell(epf);

This one too should be renamed. 

> +		}
> +	}
> +
>  	ret = pci_epf_test_alloc_space(epf);

This one too.

>  	if (ret)
>  		return ret;
>  
> +	reg = epf_test->reg[test_reg_bar];
> +	reg->magic |= FIELD_PREP(MAGIC_VERSION_MASK, 0x1);

Why are you writing this register? This register serves for the purpose of
testing BAR0.

- Mani

> +	if (doorbell_bar > 0) {
> +		reg->doorbell_addr = doorbell_addr & (align - 1);
> +		reg->doorbell_data = msg->data;
> +		reg->doorbell_bar = doorbell_bar;
> +	}
> +
>  	if (!core_init_notifier) {
>  		ret = pci_epf_test_core_init(epf);
>  		if (ret)
> -- 
> 2.34.1
> 

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 4/5] misc: pci_endpoint_test: Add doorbell test case

[Manivannan Sadhasivam]

On Mon, Sep 11, 2023 at 06:09:19PM -0400, Frank Li wrote:
> Using bit 0..7 of magic as version number in pci_endpoint_test struct to
> support older driver versions. Save to 'version' field of struct
> pci_endpoint_test to prevent reading non-existent address.
> 

Since both drivers are in the kernel, I don't see a necessity to maintain
compatibility. Does it make sense to load drivers of previous kernel revision
with a new kernel?

> Add three registers: PCIE_ENDPOINT_TEST_DB_BAR, PCIE_ENDPOINT_TEST_DB_ADDR,
> PCIE_ENDPOINT_TEST_DB_DATA.
> 

This patch is not adding these registers and not this driver also. So this
statement is wrong.

> Write data from PCI_ENDPOINT_TEST_DB_DATA to address from
> PCI_ENDPOINT_TEST_DB_ADDR to trigger doorbell and wait for endpoint
> feedback.
> 

You can reuse a part of the commit description I suggested for previous patch.

Rest looks good to me.

- Mani

> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---
>  drivers/misc/pci_endpoint_test.c | 48 ++++++++++++++++++++++++++++++++
>  include/uapi/linux/pcitest.h     |  1 +
>  2 files changed, 49 insertions(+)
> 
> diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
> index ed4d0ef5e5c31..ed0b025132d17 100644
> --- a/drivers/misc/pci_endpoint_test.c
> +++ b/drivers/misc/pci_endpoint_test.c
> @@ -33,6 +33,8 @@
>  #define IRQ_TYPE_MSIX				2
>  
>  #define PCI_ENDPOINT_TEST_MAGIC			0x0
> +#define PCI_MAGIC_VERSION_MASK			GENMASK(7, 0)
> +#define PCI_ENDPOINT_TEST_V1			0x1
>  
>  #define PCI_ENDPOINT_TEST_COMMAND		0x4
>  #define COMMAND_RAISE_LEGACY_IRQ		BIT(0)
> @@ -52,6 +54,7 @@
>  #define STATUS_IRQ_RAISED			BIT(6)
>  #define STATUS_SRC_ADDR_INVALID			BIT(7)
>  #define STATUS_DST_ADDR_INVALID			BIT(8)
> +#define STATUS_DOORBELL_SUCCESS			BIT(9)
>  
>  #define PCI_ENDPOINT_TEST_LOWER_SRC_ADDR	0x0c
>  #define PCI_ENDPOINT_TEST_UPPER_SRC_ADDR	0x10
> @@ -66,7 +69,12 @@
>  #define PCI_ENDPOINT_TEST_IRQ_NUMBER		0x28
>  
>  #define PCI_ENDPOINT_TEST_FLAGS			0x2c
> +#define PCI_ENDPOINT_TEST_DB_BAR		0x30
> +#define PCI_ENDPOINT_TEST_DB_ADDR		0x34
> +#define PCI_ENDPOINT_TEST_DB_DATA		0x38
> +
>  #define FLAG_USE_DMA				BIT(0)
> +#define FLAG_SUPPORT_DOORBELL			BIT(1)
>  
>  #define PCI_DEVICE_ID_TI_AM654			0xb00c
>  #define PCI_DEVICE_ID_TI_J7200			0xb00f
> @@ -102,6 +110,7 @@ enum pci_barno {
>  	BAR_3,
>  	BAR_4,
>  	BAR_5,
> +	NO_BAR = -1,
>  };
>  
>  struct pci_endpoint_test {
> @@ -118,6 +127,7 @@ struct pci_endpoint_test {
>  	enum pci_barno test_reg_bar;
>  	size_t alignment;
>  	const char *name;
> +	u8 version;
>  };
>  
>  struct pci_endpoint_test_data {
> @@ -713,6 +723,38 @@ static bool pci_endpoint_test_set_irq(struct pci_endpoint_test *test,
>  	return false;
>  }
>  
> +static bool pci_endpoint_test_doorbell(struct pci_endpoint_test *test)
> +{
> +	enum pci_barno bar;
> +	u32 data, status;
> +	u32 addr;
> +
> +	if (test->version < PCI_ENDPOINT_TEST_V1)
> +		return false;
> +
> +	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
> +	if (bar == NO_BAR)
> +		return false;
> +
> +	data = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_DATA);
> +	addr = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_ADDR);
> +	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
> +
> +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_TYPE, irq_type);
> +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_NUMBER, 1);
> +
> +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_STATUS, 0);
> +	pci_endpoint_test_bar_writel(test, bar, addr, data);
> +
> +	wait_for_completion_timeout(&test->irq_raised, msecs_to_jiffies(1000));
> +
> +	status = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_STATUS);
> +	if (status & STATUS_DOORBELL_SUCCESS)
> +		return true;
> +
> +	return false;
> +}
> +
>  static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
>  				    unsigned long arg)
>  {
> @@ -760,6 +802,9 @@ static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
>  	case PCITEST_CLEAR_IRQ:
>  		ret = pci_endpoint_test_clear_irq(test);
>  		break;
> +	case PCITEST_DOORBELL:
> +		ret = pci_endpoint_test_doorbell(test);
> +		break;
>  	}
>  
>  ret:
> @@ -887,6 +932,9 @@ static int pci_endpoint_test_probe(struct pci_dev *pdev,
>  	misc_device->parent = &pdev->dev;
>  	misc_device->fops = &pci_endpoint_test_fops;
>  
> +	test->version = FIELD_GET(PCI_MAGIC_VERSION_MASK,
> +				  pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_MAGIC));
> +
>  	err = misc_register(misc_device);
>  	if (err) {
>  		dev_err(dev, "Failed to register device\n");
> diff --git a/include/uapi/linux/pcitest.h b/include/uapi/linux/pcitest.h
> index f9c1af8d141b4..479ca1aa3ae0b 100644
> --- a/include/uapi/linux/pcitest.h
> +++ b/include/uapi/linux/pcitest.h
> @@ -20,6 +20,7 @@
>  #define PCITEST_SET_IRQTYPE	_IOW('P', 0x8, int)
>  #define PCITEST_GET_IRQTYPE	_IO('P', 0x9)
>  #define PCITEST_CLEAR_IRQ	_IO('P', 0x10)
> +#define PCITEST_DOORBELL	_IO('P', 0x11)
>  
>  #define PCITEST_FLAGS_USE_DMA	0x00000001
>  
> -- 
> 2.34.1
> 

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 5/5] tools: PCI: Add 'B' option for test doorbell

[Manivannan Sadhasivam]

On Mon, Sep 11, 2023 at 06:09:20PM -0400, Frank Li wrote:
> Add doorbell test support.
> 
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---
>  tools/pci/pcitest.c | 16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/pci/pcitest.c b/tools/pci/pcitest.c
> index 441b542346354..215d0aa8a09fe 100644
> --- a/tools/pci/pcitest.c
> +++ b/tools/pci/pcitest.c
> @@ -36,6 +36,7 @@ struct pci_test {
>  	bool		copy;
>  	unsigned long	size;
>  	bool		use_dma;
> +	bool		doorbell;
>  };
>  
>  static int run_test(struct pci_test *test)
> @@ -149,6 +150,15 @@ static int run_test(struct pci_test *test)
>  			fprintf(stdout, "%s\n", result[ret]);
>  	}
>  
> +	if (test->doorbell) {
> +		ret = ioctl(fd, PCITEST_DOORBELL, 0);
> +		fprintf(stdout, "Push doorbell\t\t");

"Ringing doorbell on the EP".

- Mani

> +		if (ret < 0)
> +			fprintf(stdout, "TEST FAILED\n");
> +		else
> +			fprintf(stdout, "%s\n", result[ret]);
> +	}
> +
>  	fflush(stdout);
>  	close(fd);
>  	return (ret < 0) ? ret : 1 - ret; /* return 0 if test succeeded */
> @@ -174,7 +184,7 @@ int main(int argc, char **argv)
>  	/* set default endpoint device */
>  	test->device = "/dev/pci-endpoint-test.0";
>  
> -	while ((c = getopt(argc, argv, "D:b:m:x:i:deIlhrwcs:")) != EOF)
> +	while ((c = getopt(argc, argv, "D:b:m:x:i:BdeIlhrwcs:")) != EOF)
>  	switch (c) {
>  	case 'D':
>  		test->device = optarg;
> @@ -224,6 +234,9 @@ int main(int argc, char **argv)
>  	case 'd':
>  		test->use_dma = true;
>  		continue;
> +	case 'B':
> +		test->doorbell = true;
> +		continue;
>  	case 'h':
>  	default:
>  usage:
> @@ -243,6 +256,7 @@ int main(int argc, char **argv)
>  			"\t-w			Write buffer test\n"
>  			"\t-c			Copy buffer test\n"
>  			"\t-s <size>		Size of buffer {default: 100KB}\n"
> +			"\t-B			Doorbell test\n"
>  			"\t-h			Print this help message\n",
>  			argv[0]);
>  		return -EINVAL;
> -- 
> 2.34.1
> 

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/5] PCI: endpoint: Add RC-to-EP doorbell support using platform MSI controllery

[Frank Li]

On Fri, Oct 20, 2023 at 10:42:15PM +0530, Manivannan Sadhasivam wrote:
> On Thu, Oct 19, 2023 at 02:11:22PM -0400, Frank Li wrote:
> > On Thu, Oct 19, 2023 at 10:53:47PM +0530, Manivannan Sadhasivam wrote:
> > > On Thu, Oct 19, 2023 at 12:00:22PM -0400, Frank Li wrote:
> > > > On Thu, Oct 19, 2023 at 08:34:41PM +0530, Manivannan Sadhasivam wrote:
> > > > > On Tue, Oct 17, 2023 at 02:55:57PM -0400, Frank Li wrote:
> > > > > > On Wed, Oct 18, 2023 at 12:07:22AM +0530, Manivannan Sadhasivam wrote:
> > > > > > > On Mon, Sep 11, 2023 at 06:09:16PM -0400, Frank Li wrote:
> > > > > > > > This commit introduces a common method for sending messages from the Root
> > > > > > > > Complex (RC) to the Endpoint (EP) by utilizing the platform MSI interrupt
> > > > > > > > controller, such as ARM GIC, as an EP doorbell. Maps the memory assigned
> > > > > > > > for the BAR region by the PCI host to the message address of the platform
> > > > > > > > MSI interrupt controller in the PCI EP. As a result, when the PCI RC writes
> > > > > > > 
> > > > > > > "Doorbell feature is implemented by mapping the EP's MSI interrupt controller
> > > > > > > message address to a dedicated BAR in the EPC core. It is the responsibility
> > > > > > > of the EPF driver to pass the actual message data to be written by the host to
> > > > > > > the doorbell BAR region through its own logic."
> > > > > > > 
> > > > > > > > to the BAR region, it triggers an IRQ at the EP. This implementation serves
> > > > > > > > as a common method for all endpoint function drivers.
> > > > > > > > 
> > > > > > > > However, it currently supports only one EP physical function due to
> > > > > > > > limitations in ARM MSI/IMS readiness.
> > > > > > > > 
> > > > > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > > > > > ---
> > > > > > > >  drivers/pci/endpoint/pci-epc-core.c | 192 ++++++++++++++++++++++++++++
> > > > > > > >  drivers/pci/endpoint/pci-epf-core.c |  44 +++++++
> > > > > > > >  include/linux/pci-epc.h             |   6 +
> > > > > > > >  include/linux/pci-epf.h             |   7 +
> > > > > > > >  4 files changed, 249 insertions(+)
> > > > > > > > 
> > > > > > > > diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
> > > > > > > > index 5a4a8b0be6262..d336a99c6a94f 100644
> > > > > > > > --- a/drivers/pci/endpoint/pci-epc-core.c
> > > > > > > > +++ b/drivers/pci/endpoint/pci-epc-core.c
> > > > > > > > @@ -10,6 +10,7 @@
> > > > > > > >  #include <linux/slab.h>
> > > > > > > >  #include <linux/module.h>
> > > > > > > >  
> > > > > > > > +#include <linux/msi.h>
> > > > > > > >  #include <linux/pci-epc.h>
> > > > > > > >  #include <linux/pci-epf.h>
> > > > > > > >  #include <linux/pci-ep-cfs.h>
> > > > > > > > @@ -783,6 +784,197 @@ void pci_epc_bme_notify(struct pci_epc *epc)
> > > > > > > >  }
> > > > > > > >  EXPORT_SYMBOL_GPL(pci_epc_bme_notify);
> > > > > > > >  
> > > > > > > > +/**
> > > > > > > > + * pci_epc_alloc_doorbell() - alloc an address space to let RC trigger EP side IRQ by write data to
> > > > > > > > + *			      the space.
> > > > > > > 
> > > > > > > "Allocate platform specific doorbell IRQs to be used by the host to trigger
> > > > > > > doorbells on EP."
> > > > > > > 
> > > > > > > > + *
> > > > > > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > > > > 
> > > > > > > EPC device for which the doorbell needs to be allocated
> > > > > > > 
> > > > > > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > > > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > > > > > + * @num_msgs: the total number of doorbell messages
> > > > > > > 
> > > > > > > s/num_msgs/num_db
> > > > > > > 
> > > > > > > > + *
> > > > > > > > + * Return: 0 success, other is failure
> > > > > > > > + */
> > > > > > > > +int pci_epc_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > > > > > > > +{
> > > > > > > > +	int ret;
> > > > > > > > +
> > > > > > > > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > > > > > > > +		return -EINVAL;
> > > > > > > > +
> > > > > > > > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > > > > > > > +		return -EINVAL;
> > > > > > > > +
> > > > > > > > +	if (!epc->ops->alloc_doorbell)
> > > > > > > > +		return 0;
> > > > > > > 
> > > > > > > You mentioned 0 is a success. So if there is no callback, you want to return
> > > > > > > success?
> > > > > > > 
> > > > > > > > +
> > > > > > > > +	mutex_lock(&epc->lock);
> > > > > > > > +	ret = epc->ops->alloc_doorbell(epc, func_no, vfunc_no, num_msgs);
> > > > > > > 
> > > > > > > Why can't you just call the generic function here and in other places instead of
> > > > > > > implementing callbacks? I do not see a necessity for EPC specific callbacks. If
> > > > > > > there is one, please specify.
> > > > > > 
> > > > > > 1. Refer v1 your comments.
> > > > > > https://lore.kernel.org/imx/20230906145227.GC5930@thinkpad/
> > > > > 
> > > > > I do not find where I suggested the callback approach.
> > > > 
> > > > 	> > > If that, Each EPF driver need do duplicate work. 
> > > > 	> > > 
> > > > 	> > 
> > > > 	> > Yes, and that's how it should be. EPF core has no job in supplying the of_node.
> > > > 	> > It is the responsibility of the EPF drivers as they depend on OF for platform
> > > > 	> > support.
> > > > 	> 
> > > > 	> EPF driver still not depend on OF. such pci-epf-test, which was probed by
> > > > 	> configfs.
> > > > 	> 
> > > > 
> > > > 	Hmm, yeah. Then it should be part of the EPC driver.
> > > > 
> > > > 	Sorry for the confusion.
> > > > 
> > > > Here, all "EPF" should be "EPC". The key problem is of_node. EPC core have
> > > > not of_node, EPC core's parent driver (like dwc-ep driver) have of_node. 
> > > > 
> > > > pci_epc_generic_alloc_doorbell(dev), dev is probed by platform driver, such
> > > > as dwc-ep, which have of_node,  EPC core will create child device.
> > > > 
> > > > dwc-ep device
> > > >  |- epc core device
> > > > 
> > > > we can direct call pci_epc_generic_alloc_doorbell(epc->parent) here.
> > > > 
> > > > I may miss understand what your means. I think you want to dwc-ep
> > > > (with of_node) handle these alloc functions. 
> > > > 
> > > 
> > > My comment was to have just one function definition. But looking at it again, I
> > > think it is better to move all the (alloc, free, write_msg) definitions to
> > > dwc-ep, since the contents of those functions are not EPC core specific.
> > 
> > There are still problem. (alloc, free, write_msg) is quite common for all
> > controller and the system with MSI.
> > 
> > If move these into dwc-ep,  cdns or other controller have to duplicate 
> > these codes.
> > 
> > If you think it is not EPC core specific, how about create new help files?
> > 
> 
> Hmm, that sounds good to me. I think the best place would be:
> drivers/pci/endpoint/pci-ep-msi.c

How about header file?

int pci_epc_generic_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs);     
void pci_epc_generic_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no);

Is it in include/linux/pci-epc.h, just 2 lines.

Frank
> 
> Reason is, we cannot have this generic code under drivers/pci/controller/ as it
> is not a standalone PCI controller but a platform MSI controller. So having it
> under pci/endpoint/ makes much sense to me.
> 
> And this is not specific to EPF drivers as well, so we cannot have it under
> pci/endpoint/functions/.
> 
> - Mani
> 
> > Frank
> > 
> > > 
> > > In the EPC core, you can still have the callbacks specific to each EPC. This
> > > also solves your of_node problem.
> > > 
> > > - Mani
> > > 
> > > > > 
> > > > > > 2. Maybe some ep controller have built-in doorbell support. Write to some
> > > > > > address to trigger doorbell irq.
> > > > > > 
> > > > > 
> > > > > We will handle it whenever such EP controllers arrive. Until then, let's keep it
> > > > > simple.
> > > > > 
> > > > > - Mani
> > > > > 
> > > > > > Frank
> > > > > > 
> > > > > > > 
> > > > > > > > +	mutex_unlock(&epc->lock);
> > > > > > > > +
> > > > > > > > +	return ret;
> > > > > > > > +}
> > > > > > > > +EXPORT_SYMBOL_GPL(pci_epc_alloc_doorbell);
> > > > > > > > +
> > > > > > > > +/**
> > > > > > > > + * pci_epc_free_doorbell() - free resource allocated by pci_epc_alloc_doorbell()
> > > > > > > > + *
> > > > > > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > > > > 
> > > > > > > Same as above.
> > > > > > > 
> > > > > > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > > > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > > > > > + *
> > > > > > > > + * Return: 0 success, other is failure
> > > > > > > > + */
> > > > > > > > +void pci_epc_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
> > > > > > > > +{
> > > > > > > > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > > > > > > > +		return;
> > > > > > > > +
> > > > > > > > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > > > > > > > +		return;
> > > > > > > > +
> > > > > > > > +	if (!epc->ops->free_doorbell)
> > > > > > > > +		return;
> > > > > > > > +
> > > > > > > > +	mutex_lock(&epc->lock);
> > > > > > > > +	epc->ops->free_doorbell(epc, func_no, vfunc_no);
> > > > > > > 
> > > > > > > Same as suggested above.
> > > > > > > 
> > > > > > > > +	mutex_unlock(&epc->lock);
> > > > > > > > +}
> > > > > > > > +EXPORT_SYMBOL_GPL(pci_epc_free_doorbell);
> > > > > > > > +
> > > > > > > > +static irqreturn_t pci_epf_generic_doorbell_handler(int irq, void *data)
> > > > > > > > +{
> > > > > > > > +	struct pci_epf *epf = data;
> > > > > > > > +
> > > > > > > > +	if (epf->event_ops && epf->event_ops->doorbell)
> > > > > > > > +		epf->event_ops->doorbell(epf, irq - epf->virq_base);
> > > > > > > 
> > > > > > > Same as suggested above.
> > > > > > > 
> > > > > > > > +
> > > > > > > > +	return IRQ_HANDLED;
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +static void pci_epc_generic_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
> > > > > > > > +{
> > > > > > > > +	struct pci_epc *epc = NULL;
> > > > > > > > +	struct class_dev_iter iter;
> > > > > > > > +	struct pci_epf *epf;
> > > > > > > > +	struct device *dev;
> > > > > > > > +
> > > > > > > > +	class_dev_iter_init(&iter, pci_epc_class, NULL, NULL);
> > > > > > > > +	while ((dev = class_dev_iter_next(&iter))) {
> > > > > > > > +		if (dev->parent != desc->dev)
> > > > > > > > +			continue;
> > > > > > > > +
> > > > > > > > +		epc = to_pci_epc(dev);
> > > > > > > > +
> > > > > > > > +		class_dev_iter_exit(&iter);
> > > > > > > > +		break;
> > > > > > > > +	}
> > > > > > > > +
> > > > > > > > +	if (!epc)
> > > > > > > > +		return;
> > > > > > > > +
> > > > > > > > +	/* Only support one EPF for doorbell */
> > > > > > > > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > > > > > > > +
> > > > > > > 
> > > > > > > No need of this newline
> > > > > > > 
> > > > > > > > +	if (!epf)
> > > > > > > > +		return;
> > > > > > > > +
> > > > > > > > +	if (epf->msg && desc->msi_index < epf->num_msgs)
> > > > > > > > +		epf->msg[desc->msi_index] = *msg;
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +
> > > > > > > 
> > > > > > > Remove extra newline
> > > > > > > 
> > > > > > > > +/**
> > > > > > > > + * pci_epc_generic_alloc_doorbell() - Common help function. Allocate address space from MSI
> > > > > > > > + *                                    controller
> > > > > > > > + *
> > > > > > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > > > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > > > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > > > > > + * @num_msgs: the total number of doorbell messages
> > > > > > > > + *
> > > > > > > 
> > > > > > > Same comment as for pci_epc_alloc_doorbell()
> > > > > > > 
> > > > > > > > + * Remark: use this function only if EPC driver just register one EPC device.
> > > > > > > > + *
> > > > > > > > + * Return: 0 success, other is failure
> > > > > > > > + */
> > > > > > > > +int pci_epc_generic_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > > > > > > > +{
> > > > > > > > +	struct pci_epf *epf;
> > > > > > > > +	struct device *dev;
> > > > > > > > +	int virq, last;
> > > > > > > > +	int ret;
> > > > > > > > +	int i;
> > > > > > > > +
> > > > > > > > +	if (IS_ERR_OR_NULL(epc))
> > > > > > > > +		return -EINVAL;
> > > > > > > > +
> > > > > > > > +	/* Currently only support one func and one vfunc for doorbell */
> > > > > > > > +	if (func_no || vfunc_no)
> > > > > > > > +		return -EINVAL;
> > > > > > > > +
> > > > > > > > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > > > > > > > +	if (!epf)
> > > > > > > > +		return -EINVAL;
> > > > > > > > +
> > > > > > > > +	dev = epc->dev.parent;
> > > > > > > > +	ret = platform_msi_domain_alloc_irqs(dev, num_msgs, pci_epc_generic_write_msi_msg);
> > > > > > > > +	if (ret) {
> > > > > > > > +		dev_err(dev, "Failed to allocate MSI\n");
> > > > > > > > +		return -ENOMEM;
> > > > > > > > +	}
> > > > > > > > +
> > > > > > > > +	last = -1;
> > > > > > > > +	for (i = 0; i < num_msgs; i++) {
> > > > > > > 
> > > > > > > You should iterate over msi_desc as below:
> > > > > > > 
> > > > > > >         msi_lock_descs(dev);
> > > > > > >         msi_for_each_desc(desc, dev, MSI_DESC_ALL) {
> > > > > > > 		...
> > > > > > > 	}
> > > > > > > 	msi_unlock_descs(dev);
> > > > > > > 
> > > > > > > > +		virq = msi_get_virq(dev, i);
> > > > > > > > +		if (i == 0)
> > > > > > > > +			epf->virq_base = virq;
> > > > > > > > +
> > > > > > > > +		ret = request_irq(virq, pci_epf_generic_doorbell_handler, 0,
> > > > > > > 
> > > > > > > 	request_irq(desc->irq, ...)
> > > > > > > 
> > > > > > > > +				  kasprintf(GFP_KERNEL, "pci-epc-doorbell%d", i), epf);
> > > > > > > > +
> > > > > > > > +		if (ret) {
> > > > > > > > +			dev_err(dev, "Failed to request doorbell\n");
> > > > > > > > +			goto err_free_irq;
> > > > > > > > +		}
> > > > > > > > +		last = i;
> > > > > > > > +	}
> > > > > > > > +
> > > > > > > > +	return 0;
> > > > > > > > +
> > > > > > > > +err_free_irq:
> > > > > > > > +	for (i = 0; i < last; i++)
> > > > > > > > +		kfree(free_irq(epf->virq_base + i, epf));
> > > > > > > > +	platform_msi_domain_free_irqs(dev);
> > > > > > > > +
> > > > > > > > +	return -EINVAL;
> > > > > > > 
> > > > > > > 	return ret;
> > > > > > > 
> > > > > > > > +}
> > > > > > > > +EXPORT_SYMBOL_GPL(pci_epc_generic_alloc_doorbell);
> > > > > > > > +
> > > > > > > 
> > > > > > > [...]
> > > > > > > 
> > > > > > > > diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
> > > > > > > > index 3f44b6aec4770..485c146a5efe2 100644
> > > > > > > > --- a/include/linux/pci-epf.h
> > > > > > > > +++ b/include/linux/pci-epf.h
> > > > > > > > @@ -79,6 +79,7 @@ struct pci_epc_event_ops {
> > > > > > > >  	int (*link_up)(struct pci_epf *epf);
> > > > > > > >  	int (*link_down)(struct pci_epf *epf);
> > > > > > > >  	int (*bme)(struct pci_epf *epf);
> > > > > > > > +	int (*doorbell)(struct pci_epf *epf, int index);
> > > > > > > 
> > > > > > > kdoc missing.
> > > > > > > 
> > > > > > > >  };
> > > > > > > >  
> > > > > > > >  /**
> > > > > > > > @@ -180,6 +181,9 @@ struct pci_epf {
> > > > > > > >  	unsigned long		vfunction_num_map;
> > > > > > > >  	struct list_head	pci_vepf;
> > > > > > > >  	const struct pci_epc_event_ops *event_ops;
> > > > > > > > +	struct msi_msg *msg;
> > > > > > > > +	u16 num_msgs;
> > > > > > > 
> > > > > > > num_db
> > > > > > > 
> > > > > > > You also need to add kdoc for each new member.
> > > > > > > 
> > > > > > > - Mani
> > > > > > > 
> > > > > > > -- 
> > > > > > > மணிவண்ணன் சதாசிவம்
> > > > > 
> > > > > -- 
> > > > > மணிவண்ணன் சதாசிவம்
> > > 
> > > -- 
> > > மணிவண்ணன் சதாசிவம்
> 
> -- 
> மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 4/5] misc: pci_endpoint_test: Add doorbell test case

[Manivannan Sadhasivam]

On Fri, Oct 20, 2023 at 11:23:04PM +0530, Manivannan Sadhasivam wrote:
> On Mon, Sep 11, 2023 at 06:09:19PM -0400, Frank Li wrote:
> > Using bit 0..7 of magic as version number in pci_endpoint_test struct to
> > support older driver versions. Save to 'version' field of struct
> > pci_endpoint_test to prevent reading non-existent address.
> > 
> 
> Since both drivers are in the kernel, I don't see a necessity to maintain
> compatibility. Does it make sense to load drivers of previous kernel revision
> with a new kernel?
> 

Shoot... Sorry, I completely forgot that one is EP and another is host. Yes, we
do need to maintain compatibility.

But can't we use the doorbell register contents to determine that?

- Mani

> > Add three registers: PCIE_ENDPOINT_TEST_DB_BAR, PCIE_ENDPOINT_TEST_DB_ADDR,
> > PCIE_ENDPOINT_TEST_DB_DATA.
> > 
> 
> This patch is not adding these registers and not this driver also. So this
> statement is wrong.
> 
> > Write data from PCI_ENDPOINT_TEST_DB_DATA to address from
> > PCI_ENDPOINT_TEST_DB_ADDR to trigger doorbell and wait for endpoint
> > feedback.
> > 
> 
> You can reuse a part of the commit description I suggested for previous patch.
> 
> Rest looks good to me.
> 
> - Mani
> 
> > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > ---
> >  drivers/misc/pci_endpoint_test.c | 48 ++++++++++++++++++++++++++++++++
> >  include/uapi/linux/pcitest.h     |  1 +
> >  2 files changed, 49 insertions(+)
> > 
> > diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
> > index ed4d0ef5e5c31..ed0b025132d17 100644
> > --- a/drivers/misc/pci_endpoint_test.c
> > +++ b/drivers/misc/pci_endpoint_test.c
> > @@ -33,6 +33,8 @@
> >  #define IRQ_TYPE_MSIX				2
> >  
> >  #define PCI_ENDPOINT_TEST_MAGIC			0x0
> > +#define PCI_MAGIC_VERSION_MASK			GENMASK(7, 0)
> > +#define PCI_ENDPOINT_TEST_V1			0x1
> >  
> >  #define PCI_ENDPOINT_TEST_COMMAND		0x4
> >  #define COMMAND_RAISE_LEGACY_IRQ		BIT(0)
> > @@ -52,6 +54,7 @@
> >  #define STATUS_IRQ_RAISED			BIT(6)
> >  #define STATUS_SRC_ADDR_INVALID			BIT(7)
> >  #define STATUS_DST_ADDR_INVALID			BIT(8)
> > +#define STATUS_DOORBELL_SUCCESS			BIT(9)
> >  
> >  #define PCI_ENDPOINT_TEST_LOWER_SRC_ADDR	0x0c
> >  #define PCI_ENDPOINT_TEST_UPPER_SRC_ADDR	0x10
> > @@ -66,7 +69,12 @@
> >  #define PCI_ENDPOINT_TEST_IRQ_NUMBER		0x28
> >  
> >  #define PCI_ENDPOINT_TEST_FLAGS			0x2c
> > +#define PCI_ENDPOINT_TEST_DB_BAR		0x30
> > +#define PCI_ENDPOINT_TEST_DB_ADDR		0x34
> > +#define PCI_ENDPOINT_TEST_DB_DATA		0x38
> > +
> >  #define FLAG_USE_DMA				BIT(0)
> > +#define FLAG_SUPPORT_DOORBELL			BIT(1)
> >  
> >  #define PCI_DEVICE_ID_TI_AM654			0xb00c
> >  #define PCI_DEVICE_ID_TI_J7200			0xb00f
> > @@ -102,6 +110,7 @@ enum pci_barno {
> >  	BAR_3,
> >  	BAR_4,
> >  	BAR_5,
> > +	NO_BAR = -1,
> >  };
> >  
> >  struct pci_endpoint_test {
> > @@ -118,6 +127,7 @@ struct pci_endpoint_test {
> >  	enum pci_barno test_reg_bar;
> >  	size_t alignment;
> >  	const char *name;
> > +	u8 version;
> >  };
> >  
> >  struct pci_endpoint_test_data {
> > @@ -713,6 +723,38 @@ static bool pci_endpoint_test_set_irq(struct pci_endpoint_test *test,
> >  	return false;
> >  }
> >  
> > +static bool pci_endpoint_test_doorbell(struct pci_endpoint_test *test)
> > +{
> > +	enum pci_barno bar;
> > +	u32 data, status;
> > +	u32 addr;
> > +
> > +	if (test->version < PCI_ENDPOINT_TEST_V1)
> > +		return false;
> > +
> > +	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
> > +	if (bar == NO_BAR)
> > +		return false;
> > +
> > +	data = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_DATA);
> > +	addr = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_ADDR);
> > +	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
> > +
> > +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_TYPE, irq_type);
> > +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_NUMBER, 1);
> > +
> > +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_STATUS, 0);
> > +	pci_endpoint_test_bar_writel(test, bar, addr, data);
> > +
> > +	wait_for_completion_timeout(&test->irq_raised, msecs_to_jiffies(1000));
> > +
> > +	status = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_STATUS);
> > +	if (status & STATUS_DOORBELL_SUCCESS)
> > +		return true;
> > +
> > +	return false;
> > +}
> > +
> >  static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
> >  				    unsigned long arg)
> >  {
> > @@ -760,6 +802,9 @@ static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
> >  	case PCITEST_CLEAR_IRQ:
> >  		ret = pci_endpoint_test_clear_irq(test);
> >  		break;
> > +	case PCITEST_DOORBELL:
> > +		ret = pci_endpoint_test_doorbell(test);
> > +		break;
> >  	}
> >  
> >  ret:
> > @@ -887,6 +932,9 @@ static int pci_endpoint_test_probe(struct pci_dev *pdev,
> >  	misc_device->parent = &pdev->dev;
> >  	misc_device->fops = &pci_endpoint_test_fops;
> >  
> > +	test->version = FIELD_GET(PCI_MAGIC_VERSION_MASK,
> > +				  pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_MAGIC));
> > +
> >  	err = misc_register(misc_device);
> >  	if (err) {
> >  		dev_err(dev, "Failed to register device\n");
> > diff --git a/include/uapi/linux/pcitest.h b/include/uapi/linux/pcitest.h
> > index f9c1af8d141b4..479ca1aa3ae0b 100644
> > --- a/include/uapi/linux/pcitest.h
> > +++ b/include/uapi/linux/pcitest.h
> > @@ -20,6 +20,7 @@
> >  #define PCITEST_SET_IRQTYPE	_IOW('P', 0x8, int)
> >  #define PCITEST_GET_IRQTYPE	_IO('P', 0x9)
> >  #define PCITEST_CLEAR_IRQ	_IO('P', 0x10)
> > +#define PCITEST_DOORBELL	_IO('P', 0x11)
> >  
> >  #define PCITEST_FLAGS_USE_DMA	0x00000001
> >  
> > -- 
> > 2.34.1
> > 
> 
> -- 
> மணிவண்ணன் சதாசிவம்

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 4/5] misc: pci_endpoint_test: Add doorbell test case

[Frank Li]

On Fri, Oct 20, 2023 at 11:31:39PM +0530, Manivannan Sadhasivam wrote:
> On Fri, Oct 20, 2023 at 11:23:04PM +0530, Manivannan Sadhasivam wrote:
> > On Mon, Sep 11, 2023 at 06:09:19PM -0400, Frank Li wrote:
> > > Using bit 0..7 of magic as version number in pci_endpoint_test struct to
> > > support older driver versions. Save to 'version' field of struct
> > > pci_endpoint_test to prevent reading non-existent address.
> > > 
> > 
> > Since both drivers are in the kernel, I don't see a necessity to maintain
> > compatibility. Does it make sense to load drivers of previous kernel revision
> > with a new kernel?
> > 
> 
> Shoot... Sorry, I completely forgot that one is EP and another is host. Yes, we
> do need to maintain compatibility.
> 
> But can't we use the doorbell register contents to determine that?

Doorbell register is not exist at old EP driver. If old EP driver register
size is 64Byte,  doorbell register is 64 - 68.

Read unexisted, or unmapped space will cause kernel dump or other side
effects.

Frank

> 
> - Mani
> 
> > > Add three registers: PCIE_ENDPOINT_TEST_DB_BAR, PCIE_ENDPOINT_TEST_DB_ADDR,
> > > PCIE_ENDPOINT_TEST_DB_DATA.
> > > 
> > 
> > This patch is not adding these registers and not this driver also. So this
> > statement is wrong.
> > 
> > > Write data from PCI_ENDPOINT_TEST_DB_DATA to address from
> > > PCI_ENDPOINT_TEST_DB_ADDR to trigger doorbell and wait for endpoint
> > > feedback.
> > > 
> > 
> > You can reuse a part of the commit description I suggested for previous patch.
> > 
> > Rest looks good to me.
> > 
> > - Mani
> > 
> > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > ---
> > >  drivers/misc/pci_endpoint_test.c | 48 ++++++++++++++++++++++++++++++++
> > >  include/uapi/linux/pcitest.h     |  1 +
> > >  2 files changed, 49 insertions(+)
> > > 
> > > diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
> > > index ed4d0ef5e5c31..ed0b025132d17 100644
> > > --- a/drivers/misc/pci_endpoint_test.c
> > > +++ b/drivers/misc/pci_endpoint_test.c
> > > @@ -33,6 +33,8 @@
> > >  #define IRQ_TYPE_MSIX				2
> > >  
> > >  #define PCI_ENDPOINT_TEST_MAGIC			0x0
> > > +#define PCI_MAGIC_VERSION_MASK			GENMASK(7, 0)
> > > +#define PCI_ENDPOINT_TEST_V1			0x1
> > >  
> > >  #define PCI_ENDPOINT_TEST_COMMAND		0x4
> > >  #define COMMAND_RAISE_LEGACY_IRQ		BIT(0)
> > > @@ -52,6 +54,7 @@
> > >  #define STATUS_IRQ_RAISED			BIT(6)
> > >  #define STATUS_SRC_ADDR_INVALID			BIT(7)
> > >  #define STATUS_DST_ADDR_INVALID			BIT(8)
> > > +#define STATUS_DOORBELL_SUCCESS			BIT(9)
> > >  
> > >  #define PCI_ENDPOINT_TEST_LOWER_SRC_ADDR	0x0c
> > >  #define PCI_ENDPOINT_TEST_UPPER_SRC_ADDR	0x10
> > > @@ -66,7 +69,12 @@
> > >  #define PCI_ENDPOINT_TEST_IRQ_NUMBER		0x28
> > >  
> > >  #define PCI_ENDPOINT_TEST_FLAGS			0x2c
> > > +#define PCI_ENDPOINT_TEST_DB_BAR		0x30
> > > +#define PCI_ENDPOINT_TEST_DB_ADDR		0x34
> > > +#define PCI_ENDPOINT_TEST_DB_DATA		0x38
> > > +
> > >  #define FLAG_USE_DMA				BIT(0)
> > > +#define FLAG_SUPPORT_DOORBELL			BIT(1)
> > >  
> > >  #define PCI_DEVICE_ID_TI_AM654			0xb00c
> > >  #define PCI_DEVICE_ID_TI_J7200			0xb00f
> > > @@ -102,6 +110,7 @@ enum pci_barno {
> > >  	BAR_3,
> > >  	BAR_4,
> > >  	BAR_5,
> > > +	NO_BAR = -1,
> > >  };
> > >  
> > >  struct pci_endpoint_test {
> > > @@ -118,6 +127,7 @@ struct pci_endpoint_test {
> > >  	enum pci_barno test_reg_bar;
> > >  	size_t alignment;
> > >  	const char *name;
> > > +	u8 version;
> > >  };
> > >  
> > >  struct pci_endpoint_test_data {
> > > @@ -713,6 +723,38 @@ static bool pci_endpoint_test_set_irq(struct pci_endpoint_test *test,
> > >  	return false;
> > >  }
> > >  
> > > +static bool pci_endpoint_test_doorbell(struct pci_endpoint_test *test)
> > > +{
> > > +	enum pci_barno bar;
> > > +	u32 data, status;
> > > +	u32 addr;
> > > +
> > > +	if (test->version < PCI_ENDPOINT_TEST_V1)
> > > +		return false;
> > > +
> > > +	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
> > > +	if (bar == NO_BAR)
> > > +		return false;
> > > +
> > > +	data = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_DATA);
> > > +	addr = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_ADDR);
> > > +	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
> > > +
> > > +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_TYPE, irq_type);
> > > +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_NUMBER, 1);
> > > +
> > > +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_STATUS, 0);
> > > +	pci_endpoint_test_bar_writel(test, bar, addr, data);
> > > +
> > > +	wait_for_completion_timeout(&test->irq_raised, msecs_to_jiffies(1000));
> > > +
> > > +	status = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_STATUS);
> > > +	if (status & STATUS_DOORBELL_SUCCESS)
> > > +		return true;
> > > +
> > > +	return false;
> > > +}
> > > +
> > >  static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
> > >  				    unsigned long arg)
> > >  {
> > > @@ -760,6 +802,9 @@ static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
> > >  	case PCITEST_CLEAR_IRQ:
> > >  		ret = pci_endpoint_test_clear_irq(test);
> > >  		break;
> > > +	case PCITEST_DOORBELL:
> > > +		ret = pci_endpoint_test_doorbell(test);
> > > +		break;
> > >  	}
> > >  
> > >  ret:
> > > @@ -887,6 +932,9 @@ static int pci_endpoint_test_probe(struct pci_dev *pdev,
> > >  	misc_device->parent = &pdev->dev;
> > >  	misc_device->fops = &pci_endpoint_test_fops;
> > >  
> > > +	test->version = FIELD_GET(PCI_MAGIC_VERSION_MASK,
> > > +				  pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_MAGIC));
> > > +
> > >  	err = misc_register(misc_device);
> > >  	if (err) {
> > >  		dev_err(dev, "Failed to register device\n");
> > > diff --git a/include/uapi/linux/pcitest.h b/include/uapi/linux/pcitest.h
> > > index f9c1af8d141b4..479ca1aa3ae0b 100644
> > > --- a/include/uapi/linux/pcitest.h
> > > +++ b/include/uapi/linux/pcitest.h
> > > @@ -20,6 +20,7 @@
> > >  #define PCITEST_SET_IRQTYPE	_IOW('P', 0x8, int)
> > >  #define PCITEST_GET_IRQTYPE	_IO('P', 0x9)
> > >  #define PCITEST_CLEAR_IRQ	_IO('P', 0x10)
> > > +#define PCITEST_DOORBELL	_IO('P', 0x11)
> > >  
> > >  #define PCITEST_FLAGS_USE_DMA	0x00000001
> > >  
> > > -- 
> > > 2.34.1
> > > 
> > 
> > -- 
> > மணிவண்ணன் சதாசிவம்
> 
> -- 
> மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 1/5] PCI: endpoint: Add RC-to-EP doorbell support using platform MSI controllery

[Manivannan Sadhasivam]

On Fri, Oct 20, 2023 at 02:00:06PM -0400, Frank Li wrote:
> On Fri, Oct 20, 2023 at 10:42:15PM +0530, Manivannan Sadhasivam wrote:
> > On Thu, Oct 19, 2023 at 02:11:22PM -0400, Frank Li wrote:
> > > On Thu, Oct 19, 2023 at 10:53:47PM +0530, Manivannan Sadhasivam wrote:
> > > > On Thu, Oct 19, 2023 at 12:00:22PM -0400, Frank Li wrote:
> > > > > On Thu, Oct 19, 2023 at 08:34:41PM +0530, Manivannan Sadhasivam wrote:
> > > > > > On Tue, Oct 17, 2023 at 02:55:57PM -0400, Frank Li wrote:
> > > > > > > On Wed, Oct 18, 2023 at 12:07:22AM +0530, Manivannan Sadhasivam wrote:
> > > > > > > > On Mon, Sep 11, 2023 at 06:09:16PM -0400, Frank Li wrote:
> > > > > > > > > This commit introduces a common method for sending messages from the Root
> > > > > > > > > Complex (RC) to the Endpoint (EP) by utilizing the platform MSI interrupt
> > > > > > > > > controller, such as ARM GIC, as an EP doorbell. Maps the memory assigned
> > > > > > > > > for the BAR region by the PCI host to the message address of the platform
> > > > > > > > > MSI interrupt controller in the PCI EP. As a result, when the PCI RC writes
> > > > > > > > 
> > > > > > > > "Doorbell feature is implemented by mapping the EP's MSI interrupt controller
> > > > > > > > message address to a dedicated BAR in the EPC core. It is the responsibility
> > > > > > > > of the EPF driver to pass the actual message data to be written by the host to
> > > > > > > > the doorbell BAR region through its own logic."
> > > > > > > > 
> > > > > > > > > to the BAR region, it triggers an IRQ at the EP. This implementation serves
> > > > > > > > > as a common method for all endpoint function drivers.
> > > > > > > > > 
> > > > > > > > > However, it currently supports only one EP physical function due to
> > > > > > > > > limitations in ARM MSI/IMS readiness.
> > > > > > > > > 
> > > > > > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > > > > > > ---
> > > > > > > > >  drivers/pci/endpoint/pci-epc-core.c | 192 ++++++++++++++++++++++++++++
> > > > > > > > >  drivers/pci/endpoint/pci-epf-core.c |  44 +++++++
> > > > > > > > >  include/linux/pci-epc.h             |   6 +
> > > > > > > > >  include/linux/pci-epf.h             |   7 +
> > > > > > > > >  4 files changed, 249 insertions(+)
> > > > > > > > > 
> > > > > > > > > diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c
> > > > > > > > > index 5a4a8b0be6262..d336a99c6a94f 100644
> > > > > > > > > --- a/drivers/pci/endpoint/pci-epc-core.c
> > > > > > > > > +++ b/drivers/pci/endpoint/pci-epc-core.c
> > > > > > > > > @@ -10,6 +10,7 @@
> > > > > > > > >  #include <linux/slab.h>
> > > > > > > > >  #include <linux/module.h>
> > > > > > > > >  
> > > > > > > > > +#include <linux/msi.h>
> > > > > > > > >  #include <linux/pci-epc.h>
> > > > > > > > >  #include <linux/pci-epf.h>
> > > > > > > > >  #include <linux/pci-ep-cfs.h>
> > > > > > > > > @@ -783,6 +784,197 @@ void pci_epc_bme_notify(struct pci_epc *epc)
> > > > > > > > >  }
> > > > > > > > >  EXPORT_SYMBOL_GPL(pci_epc_bme_notify);
> > > > > > > > >  
> > > > > > > > > +/**
> > > > > > > > > + * pci_epc_alloc_doorbell() - alloc an address space to let RC trigger EP side IRQ by write data to
> > > > > > > > > + *			      the space.
> > > > > > > > 
> > > > > > > > "Allocate platform specific doorbell IRQs to be used by the host to trigger
> > > > > > > > doorbells on EP."
> > > > > > > > 
> > > > > > > > > + *
> > > > > > > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > > > > > 
> > > > > > > > EPC device for which the doorbell needs to be allocated
> > > > > > > > 
> > > > > > > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > > > > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > > > > > > + * @num_msgs: the total number of doorbell messages
> > > > > > > > 
> > > > > > > > s/num_msgs/num_db
> > > > > > > > 
> > > > > > > > > + *
> > > > > > > > > + * Return: 0 success, other is failure
> > > > > > > > > + */
> > > > > > > > > +int pci_epc_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > > > > > > > > +{
> > > > > > > > > +	int ret;
> > > > > > > > > +
> > > > > > > > > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > > > > > > > > +		return -EINVAL;
> > > > > > > > > +
> > > > > > > > > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > > > > > > > > +		return -EINVAL;
> > > > > > > > > +
> > > > > > > > > +	if (!epc->ops->alloc_doorbell)
> > > > > > > > > +		return 0;
> > > > > > > > 
> > > > > > > > You mentioned 0 is a success. So if there is no callback, you want to return
> > > > > > > > success?
> > > > > > > > 
> > > > > > > > > +
> > > > > > > > > +	mutex_lock(&epc->lock);
> > > > > > > > > +	ret = epc->ops->alloc_doorbell(epc, func_no, vfunc_no, num_msgs);
> > > > > > > > 
> > > > > > > > Why can't you just call the generic function here and in other places instead of
> > > > > > > > implementing callbacks? I do not see a necessity for EPC specific callbacks. If
> > > > > > > > there is one, please specify.
> > > > > > > 
> > > > > > > 1. Refer v1 your comments.
> > > > > > > https://lore.kernel.org/imx/20230906145227.GC5930@thinkpad/
> > > > > > 
> > > > > > I do not find where I suggested the callback approach.
> > > > > 
> > > > > 	> > > If that, Each EPF driver need do duplicate work. 
> > > > > 	> > > 
> > > > > 	> > 
> > > > > 	> > Yes, and that's how it should be. EPF core has no job in supplying the of_node.
> > > > > 	> > It is the responsibility of the EPF drivers as they depend on OF for platform
> > > > > 	> > support.
> > > > > 	> 
> > > > > 	> EPF driver still not depend on OF. such pci-epf-test, which was probed by
> > > > > 	> configfs.
> > > > > 	> 
> > > > > 
> > > > > 	Hmm, yeah. Then it should be part of the EPC driver.
> > > > > 
> > > > > 	Sorry for the confusion.
> > > > > 
> > > > > Here, all "EPF" should be "EPC". The key problem is of_node. EPC core have
> > > > > not of_node, EPC core's parent driver (like dwc-ep driver) have of_node. 
> > > > > 
> > > > > pci_epc_generic_alloc_doorbell(dev), dev is probed by platform driver, such
> > > > > as dwc-ep, which have of_node,  EPC core will create child device.
> > > > > 
> > > > > dwc-ep device
> > > > >  |- epc core device
> > > > > 
> > > > > we can direct call pci_epc_generic_alloc_doorbell(epc->parent) here.
> > > > > 
> > > > > I may miss understand what your means. I think you want to dwc-ep
> > > > > (with of_node) handle these alloc functions. 
> > > > > 
> > > > 
> > > > My comment was to have just one function definition. But looking at it again, I
> > > > think it is better to move all the (alloc, free, write_msg) definitions to
> > > > dwc-ep, since the contents of those functions are not EPC core specific.
> > > 
> > > There are still problem. (alloc, free, write_msg) is quite common for all
> > > controller and the system with MSI.
> > > 
> > > If move these into dwc-ep,  cdns or other controller have to duplicate 
> > > these codes.
> > > 
> > > If you think it is not EPC core specific, how about create new help files?
> > > 
> > 
> > Hmm, that sounds good to me. I think the best place would be:
> > drivers/pci/endpoint/pci-ep-msi.c
> 
> How about header file?
> 
> int pci_epc_generic_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs);     
> void pci_epc_generic_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no);
> 
> Is it in include/linux/pci-epc.h, just 2 lines.
> 

Sorry, I don't understand what you are suggesting. Can you elaborate?

My suggestion was to place the function definitions in
drivers/pci/endpoint/pci-ep-msi.c, create a separate header
include/linux/pci-ep-msi.h and place the declarations there. The EPC drivers
want to use generic MSI support should include this header.

- Mani

> Frank
> > 
> > Reason is, we cannot have this generic code under drivers/pci/controller/ as it
> > is not a standalone PCI controller but a platform MSI controller. So having it
> > under pci/endpoint/ makes much sense to me.
> > 
> > And this is not specific to EPF drivers as well, so we cannot have it under
> > pci/endpoint/functions/.
> > 
> > - Mani
> > 
> > > Frank
> > > 
> > > > 
> > > > In the EPC core, you can still have the callbacks specific to each EPC. This
> > > > also solves your of_node problem.
> > > > 
> > > > - Mani
> > > > 
> > > > > > 
> > > > > > > 2. Maybe some ep controller have built-in doorbell support. Write to some
> > > > > > > address to trigger doorbell irq.
> > > > > > > 
> > > > > > 
> > > > > > We will handle it whenever such EP controllers arrive. Until then, let's keep it
> > > > > > simple.
> > > > > > 
> > > > > > - Mani
> > > > > > 
> > > > > > > Frank
> > > > > > > 
> > > > > > > > 
> > > > > > > > > +	mutex_unlock(&epc->lock);
> > > > > > > > > +
> > > > > > > > > +	return ret;
> > > > > > > > > +}
> > > > > > > > > +EXPORT_SYMBOL_GPL(pci_epc_alloc_doorbell);
> > > > > > > > > +
> > > > > > > > > +/**
> > > > > > > > > + * pci_epc_free_doorbell() - free resource allocated by pci_epc_alloc_doorbell()
> > > > > > > > > + *
> > > > > > > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > > > > > 
> > > > > > > > Same as above.
> > > > > > > > 
> > > > > > > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > > > > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > > > > > > + *
> > > > > > > > > + * Return: 0 success, other is failure
> > > > > > > > > + */
> > > > > > > > > +void pci_epc_free_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no)
> > > > > > > > > +{
> > > > > > > > > +	if (IS_ERR_OR_NULL(epc) || func_no >= epc->max_functions)
> > > > > > > > > +		return;
> > > > > > > > > +
> > > > > > > > > +	if (vfunc_no > 0 && (!epc->max_vfs || vfunc_no > epc->max_vfs[func_no]))
> > > > > > > > > +		return;
> > > > > > > > > +
> > > > > > > > > +	if (!epc->ops->free_doorbell)
> > > > > > > > > +		return;
> > > > > > > > > +
> > > > > > > > > +	mutex_lock(&epc->lock);
> > > > > > > > > +	epc->ops->free_doorbell(epc, func_no, vfunc_no);
> > > > > > > > 
> > > > > > > > Same as suggested above.
> > > > > > > > 
> > > > > > > > > +	mutex_unlock(&epc->lock);
> > > > > > > > > +}
> > > > > > > > > +EXPORT_SYMBOL_GPL(pci_epc_free_doorbell);
> > > > > > > > > +
> > > > > > > > > +static irqreturn_t pci_epf_generic_doorbell_handler(int irq, void *data)
> > > > > > > > > +{
> > > > > > > > > +	struct pci_epf *epf = data;
> > > > > > > > > +
> > > > > > > > > +	if (epf->event_ops && epf->event_ops->doorbell)
> > > > > > > > > +		epf->event_ops->doorbell(epf, irq - epf->virq_base);
> > > > > > > > 
> > > > > > > > Same as suggested above.
> > > > > > > > 
> > > > > > > > > +
> > > > > > > > > +	return IRQ_HANDLED;
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > +static void pci_epc_generic_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
> > > > > > > > > +{
> > > > > > > > > +	struct pci_epc *epc = NULL;
> > > > > > > > > +	struct class_dev_iter iter;
> > > > > > > > > +	struct pci_epf *epf;
> > > > > > > > > +	struct device *dev;
> > > > > > > > > +
> > > > > > > > > +	class_dev_iter_init(&iter, pci_epc_class, NULL, NULL);
> > > > > > > > > +	while ((dev = class_dev_iter_next(&iter))) {
> > > > > > > > > +		if (dev->parent != desc->dev)
> > > > > > > > > +			continue;
> > > > > > > > > +
> > > > > > > > > +		epc = to_pci_epc(dev);
> > > > > > > > > +
> > > > > > > > > +		class_dev_iter_exit(&iter);
> > > > > > > > > +		break;
> > > > > > > > > +	}
> > > > > > > > > +
> > > > > > > > > +	if (!epc)
> > > > > > > > > +		return;
> > > > > > > > > +
> > > > > > > > > +	/* Only support one EPF for doorbell */
> > > > > > > > > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > > > > > > > > +
> > > > > > > > 
> > > > > > > > No need of this newline
> > > > > > > > 
> > > > > > > > > +	if (!epf)
> > > > > > > > > +		return;
> > > > > > > > > +
> > > > > > > > > +	if (epf->msg && desc->msi_index < epf->num_msgs)
> > > > > > > > > +		epf->msg[desc->msi_index] = *msg;
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > +
> > > > > > > > 
> > > > > > > > Remove extra newline
> > > > > > > > 
> > > > > > > > > +/**
> > > > > > > > > + * pci_epc_generic_alloc_doorbell() - Common help function. Allocate address space from MSI
> > > > > > > > > + *                                    controller
> > > > > > > > > + *
> > > > > > > > > + * @epc: the EPC device that need doorbell address and data from RC.
> > > > > > > > > + * @func_no: the physical endpoint function number in the EPC device.
> > > > > > > > > + * @vfunc_no: the virtual endpoint function number in the physical function.
> > > > > > > > > + * @num_msgs: the total number of doorbell messages
> > > > > > > > > + *
> > > > > > > > 
> > > > > > > > Same comment as for pci_epc_alloc_doorbell()
> > > > > > > > 
> > > > > > > > > + * Remark: use this function only if EPC driver just register one EPC device.
> > > > > > > > > + *
> > > > > > > > > + * Return: 0 success, other is failure
> > > > > > > > > + */
> > > > > > > > > +int pci_epc_generic_alloc_doorbell(struct pci_epc *epc, u8 func_no, u8 vfunc_no, int num_msgs)
> > > > > > > > > +{
> > > > > > > > > +	struct pci_epf *epf;
> > > > > > > > > +	struct device *dev;
> > > > > > > > > +	int virq, last;
> > > > > > > > > +	int ret;
> > > > > > > > > +	int i;
> > > > > > > > > +
> > > > > > > > > +	if (IS_ERR_OR_NULL(epc))
> > > > > > > > > +		return -EINVAL;
> > > > > > > > > +
> > > > > > > > > +	/* Currently only support one func and one vfunc for doorbell */
> > > > > > > > > +	if (func_no || vfunc_no)
> > > > > > > > > +		return -EINVAL;
> > > > > > > > > +
> > > > > > > > > +	epf = list_first_entry_or_null(&epc->pci_epf, struct pci_epf, list);
> > > > > > > > > +	if (!epf)
> > > > > > > > > +		return -EINVAL;
> > > > > > > > > +
> > > > > > > > > +	dev = epc->dev.parent;
> > > > > > > > > +	ret = platform_msi_domain_alloc_irqs(dev, num_msgs, pci_epc_generic_write_msi_msg);
> > > > > > > > > +	if (ret) {
> > > > > > > > > +		dev_err(dev, "Failed to allocate MSI\n");
> > > > > > > > > +		return -ENOMEM;
> > > > > > > > > +	}
> > > > > > > > > +
> > > > > > > > > +	last = -1;
> > > > > > > > > +	for (i = 0; i < num_msgs; i++) {
> > > > > > > > 
> > > > > > > > You should iterate over msi_desc as below:
> > > > > > > > 
> > > > > > > >         msi_lock_descs(dev);
> > > > > > > >         msi_for_each_desc(desc, dev, MSI_DESC_ALL) {
> > > > > > > > 		...
> > > > > > > > 	}
> > > > > > > > 	msi_unlock_descs(dev);
> > > > > > > > 
> > > > > > > > > +		virq = msi_get_virq(dev, i);
> > > > > > > > > +		if (i == 0)
> > > > > > > > > +			epf->virq_base = virq;
> > > > > > > > > +
> > > > > > > > > +		ret = request_irq(virq, pci_epf_generic_doorbell_handler, 0,
> > > > > > > > 
> > > > > > > > 	request_irq(desc->irq, ...)
> > > > > > > > 
> > > > > > > > > +				  kasprintf(GFP_KERNEL, "pci-epc-doorbell%d", i), epf);
> > > > > > > > > +
> > > > > > > > > +		if (ret) {
> > > > > > > > > +			dev_err(dev, "Failed to request doorbell\n");
> > > > > > > > > +			goto err_free_irq;
> > > > > > > > > +		}
> > > > > > > > > +		last = i;
> > > > > > > > > +	}
> > > > > > > > > +
> > > > > > > > > +	return 0;
> > > > > > > > > +
> > > > > > > > > +err_free_irq:
> > > > > > > > > +	for (i = 0; i < last; i++)
> > > > > > > > > +		kfree(free_irq(epf->virq_base + i, epf));
> > > > > > > > > +	platform_msi_domain_free_irqs(dev);
> > > > > > > > > +
> > > > > > > > > +	return -EINVAL;
> > > > > > > > 
> > > > > > > > 	return ret;
> > > > > > > > 
> > > > > > > > > +}
> > > > > > > > > +EXPORT_SYMBOL_GPL(pci_epc_generic_alloc_doorbell);
> > > > > > > > > +
> > > > > > > > 
> > > > > > > > [...]
> > > > > > > > 
> > > > > > > > > diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h
> > > > > > > > > index 3f44b6aec4770..485c146a5efe2 100644
> > > > > > > > > --- a/include/linux/pci-epf.h
> > > > > > > > > +++ b/include/linux/pci-epf.h
> > > > > > > > > @@ -79,6 +79,7 @@ struct pci_epc_event_ops {
> > > > > > > > >  	int (*link_up)(struct pci_epf *epf);
> > > > > > > > >  	int (*link_down)(struct pci_epf *epf);
> > > > > > > > >  	int (*bme)(struct pci_epf *epf);
> > > > > > > > > +	int (*doorbell)(struct pci_epf *epf, int index);
> > > > > > > > 
> > > > > > > > kdoc missing.
> > > > > > > > 
> > > > > > > > >  };
> > > > > > > > >  
> > > > > > > > >  /**
> > > > > > > > > @@ -180,6 +181,9 @@ struct pci_epf {
> > > > > > > > >  	unsigned long		vfunction_num_map;
> > > > > > > > >  	struct list_head	pci_vepf;
> > > > > > > > >  	const struct pci_epc_event_ops *event_ops;
> > > > > > > > > +	struct msi_msg *msg;
> > > > > > > > > +	u16 num_msgs;
> > > > > > > > 
> > > > > > > > num_db
> > > > > > > > 
> > > > > > > > You also need to add kdoc for each new member.
> > > > > > > > 
> > > > > > > > - Mani
> > > > > > > > 
> > > > > > > > -- 
> > > > > > > > மணிவண்ணன் சதாசிவம்
> > > > > > 
> > > > > > -- 
> > > > > > மணிவண்ணன் சதாசிவம்
> > > > 
> > > > -- 
> > > > மணிவண்ணன் சதாசிவம்
> > 
> > -- 
> > மணிவண்ணன் சதாசிவம்

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 4/5] misc: pci_endpoint_test: Add doorbell test case

[Manivannan Sadhasivam]

On Fri, Oct 20, 2023 at 02:07:02PM -0400, Frank Li wrote:
> On Fri, Oct 20, 2023 at 11:31:39PM +0530, Manivannan Sadhasivam wrote:
> > On Fri, Oct 20, 2023 at 11:23:04PM +0530, Manivannan Sadhasivam wrote:
> > > On Mon, Sep 11, 2023 at 06:09:19PM -0400, Frank Li wrote:
> > > > Using bit 0..7 of magic as version number in pci_endpoint_test struct to
> > > > support older driver versions. Save to 'version' field of struct
> > > > pci_endpoint_test to prevent reading non-existent address.
> > > > 
> > > 
> > > Since both drivers are in the kernel, I don't see a necessity to maintain
> > > compatibility. Does it make sense to load drivers of previous kernel revision
> > > with a new kernel?
> > > 
> > 
> > Shoot... Sorry, I completely forgot that one is EP and another is host. Yes, we
> > do need to maintain compatibility.
> > 
> > But can't we use the doorbell register contents to determine that?
> 
> Doorbell register is not exist at old EP driver. If old EP driver register
> size is 64Byte,  doorbell register is 64 - 68.
>

Yes, I know!
 
> Read unexisted, or unmapped space will cause kernel dump or other side
> effects.
> 

For sure it won't be unmapped as the BAR0 size is 512B. I thought we could infer
something from the uninitialized registers. I need to think about other options.
But changing the semantics of MAGIC register is a no-go.

- Mani

> Frank
> 
> > 
> > - Mani
> > 
> > > > Add three registers: PCIE_ENDPOINT_TEST_DB_BAR, PCIE_ENDPOINT_TEST_DB_ADDR,
> > > > PCIE_ENDPOINT_TEST_DB_DATA.
> > > > 
> > > 
> > > This patch is not adding these registers and not this driver also. So this
> > > statement is wrong.
> > > 
> > > > Write data from PCI_ENDPOINT_TEST_DB_DATA to address from
> > > > PCI_ENDPOINT_TEST_DB_ADDR to trigger doorbell and wait for endpoint
> > > > feedback.
> > > > 
> > > 
> > > You can reuse a part of the commit description I suggested for previous patch.
> > > 
> > > Rest looks good to me.
> > > 
> > > - Mani
> > > 
> > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > ---
> > > >  drivers/misc/pci_endpoint_test.c | 48 ++++++++++++++++++++++++++++++++
> > > >  include/uapi/linux/pcitest.h     |  1 +
> > > >  2 files changed, 49 insertions(+)
> > > > 
> > > > diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
> > > > index ed4d0ef5e5c31..ed0b025132d17 100644
> > > > --- a/drivers/misc/pci_endpoint_test.c
> > > > +++ b/drivers/misc/pci_endpoint_test.c
> > > > @@ -33,6 +33,8 @@
> > > >  #define IRQ_TYPE_MSIX				2
> > > >  
> > > >  #define PCI_ENDPOINT_TEST_MAGIC			0x0
> > > > +#define PCI_MAGIC_VERSION_MASK			GENMASK(7, 0)
> > > > +#define PCI_ENDPOINT_TEST_V1			0x1
> > > >  
> > > >  #define PCI_ENDPOINT_TEST_COMMAND		0x4
> > > >  #define COMMAND_RAISE_LEGACY_IRQ		BIT(0)
> > > > @@ -52,6 +54,7 @@
> > > >  #define STATUS_IRQ_RAISED			BIT(6)
> > > >  #define STATUS_SRC_ADDR_INVALID			BIT(7)
> > > >  #define STATUS_DST_ADDR_INVALID			BIT(8)
> > > > +#define STATUS_DOORBELL_SUCCESS			BIT(9)
> > > >  
> > > >  #define PCI_ENDPOINT_TEST_LOWER_SRC_ADDR	0x0c
> > > >  #define PCI_ENDPOINT_TEST_UPPER_SRC_ADDR	0x10
> > > > @@ -66,7 +69,12 @@
> > > >  #define PCI_ENDPOINT_TEST_IRQ_NUMBER		0x28
> > > >  
> > > >  #define PCI_ENDPOINT_TEST_FLAGS			0x2c
> > > > +#define PCI_ENDPOINT_TEST_DB_BAR		0x30
> > > > +#define PCI_ENDPOINT_TEST_DB_ADDR		0x34
> > > > +#define PCI_ENDPOINT_TEST_DB_DATA		0x38
> > > > +
> > > >  #define FLAG_USE_DMA				BIT(0)
> > > > +#define FLAG_SUPPORT_DOORBELL			BIT(1)
> > > >  
> > > >  #define PCI_DEVICE_ID_TI_AM654			0xb00c
> > > >  #define PCI_DEVICE_ID_TI_J7200			0xb00f
> > > > @@ -102,6 +110,7 @@ enum pci_barno {
> > > >  	BAR_3,
> > > >  	BAR_4,
> > > >  	BAR_5,
> > > > +	NO_BAR = -1,
> > > >  };
> > > >  
> > > >  struct pci_endpoint_test {
> > > > @@ -118,6 +127,7 @@ struct pci_endpoint_test {
> > > >  	enum pci_barno test_reg_bar;
> > > >  	size_t alignment;
> > > >  	const char *name;
> > > > +	u8 version;
> > > >  };
> > > >  
> > > >  struct pci_endpoint_test_data {
> > > > @@ -713,6 +723,38 @@ static bool pci_endpoint_test_set_irq(struct pci_endpoint_test *test,
> > > >  	return false;
> > > >  }
> > > >  
> > > > +static bool pci_endpoint_test_doorbell(struct pci_endpoint_test *test)
> > > > +{
> > > > +	enum pci_barno bar;
> > > > +	u32 data, status;
> > > > +	u32 addr;
> > > > +
> > > > +	if (test->version < PCI_ENDPOINT_TEST_V1)
> > > > +		return false;
> > > > +
> > > > +	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
> > > > +	if (bar == NO_BAR)
> > > > +		return false;
> > > > +
> > > > +	data = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_DATA);
> > > > +	addr = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_ADDR);
> > > > +	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
> > > > +
> > > > +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_TYPE, irq_type);
> > > > +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_NUMBER, 1);
> > > > +
> > > > +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_STATUS, 0);
> > > > +	pci_endpoint_test_bar_writel(test, bar, addr, data);
> > > > +
> > > > +	wait_for_completion_timeout(&test->irq_raised, msecs_to_jiffies(1000));
> > > > +
> > > > +	status = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_STATUS);
> > > > +	if (status & STATUS_DOORBELL_SUCCESS)
> > > > +		return true;
> > > > +
> > > > +	return false;
> > > > +}
> > > > +
> > > >  static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
> > > >  				    unsigned long arg)
> > > >  {
> > > > @@ -760,6 +802,9 @@ static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
> > > >  	case PCITEST_CLEAR_IRQ:
> > > >  		ret = pci_endpoint_test_clear_irq(test);
> > > >  		break;
> > > > +	case PCITEST_DOORBELL:
> > > > +		ret = pci_endpoint_test_doorbell(test);
> > > > +		break;
> > > >  	}
> > > >  
> > > >  ret:
> > > > @@ -887,6 +932,9 @@ static int pci_endpoint_test_probe(struct pci_dev *pdev,
> > > >  	misc_device->parent = &pdev->dev;
> > > >  	misc_device->fops = &pci_endpoint_test_fops;
> > > >  
> > > > +	test->version = FIELD_GET(PCI_MAGIC_VERSION_MASK,
> > > > +				  pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_MAGIC));
> > > > +
> > > >  	err = misc_register(misc_device);
> > > >  	if (err) {
> > > >  		dev_err(dev, "Failed to register device\n");
> > > > diff --git a/include/uapi/linux/pcitest.h b/include/uapi/linux/pcitest.h
> > > > index f9c1af8d141b4..479ca1aa3ae0b 100644
> > > > --- a/include/uapi/linux/pcitest.h
> > > > +++ b/include/uapi/linux/pcitest.h
> > > > @@ -20,6 +20,7 @@
> > > >  #define PCITEST_SET_IRQTYPE	_IOW('P', 0x8, int)
> > > >  #define PCITEST_GET_IRQTYPE	_IO('P', 0x9)
> > > >  #define PCITEST_CLEAR_IRQ	_IO('P', 0x10)
> > > > +#define PCITEST_DOORBELL	_IO('P', 0x11)
> > > >  
> > > >  #define PCITEST_FLAGS_USE_DMA	0x00000001
> > > >  
> > > > -- 
> > > > 2.34.1
> > > > 
> > > 
> > > -- 
> > > மணிவண்ணன் சதாசிவம்
> > 
> > -- 
> > மணிவண்ணன் சதாசிவம்

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 4/5] misc: pci_endpoint_test: Add doorbell test case

[Frank Li]

On Fri, Oct 20, 2023 at 11:56:43PM +0530, Manivannan Sadhasivam wrote:
> On Fri, Oct 20, 2023 at 02:07:02PM -0400, Frank Li wrote:
> > On Fri, Oct 20, 2023 at 11:31:39PM +0530, Manivannan Sadhasivam wrote:
> > > On Fri, Oct 20, 2023 at 11:23:04PM +0530, Manivannan Sadhasivam wrote:
> > > > On Mon, Sep 11, 2023 at 06:09:19PM -0400, Frank Li wrote:
> > > > > Using bit 0..7 of magic as version number in pci_endpoint_test struct to
> > > > > support older driver versions. Save to 'version' field of struct
> > > > > pci_endpoint_test to prevent reading non-existent address.
> > > > > 
> > > > 
> > > > Since both drivers are in the kernel, I don't see a necessity to maintain
> > > > compatibility. Does it make sense to load drivers of previous kernel revision
> > > > with a new kernel?
> > > > 
> > > 
> > > Shoot... Sorry, I completely forgot that one is EP and another is host. Yes, we
> > > do need to maintain compatibility.
> > > 
> > > But can't we use the doorbell register contents to determine that?
> > 
> > Doorbell register is not exist at old EP driver. If old EP driver register
> > size is 64Byte,  doorbell register is 64 - 68.
> >
> 
> Yes, I know!
>  
> > Read unexisted, or unmapped space will cause kernel dump or other side
> > effects.
> > 
> 
> For sure it won't be unmapped as the BAR0 size is 512B. I thought we could infer
> something from the uninitialized registers. I need to think about other options.
> But changing the semantics of MAGIC register is a no-go.

@Mani:

Any other options? I plan respin these patches soon

Frank

> 
> - Mani
> 
> > Frank
> > 
> > > 
> > > - Mani
> > > 
> > > > > Add three registers: PCIE_ENDPOINT_TEST_DB_BAR, PCIE_ENDPOINT_TEST_DB_ADDR,
> > > > > PCIE_ENDPOINT_TEST_DB_DATA.
> > > > > 
> > > > 
> > > > This patch is not adding these registers and not this driver also. So this
> > > > statement is wrong.
> > > > 
> > > > > Write data from PCI_ENDPOINT_TEST_DB_DATA to address from
> > > > > PCI_ENDPOINT_TEST_DB_ADDR to trigger doorbell and wait for endpoint
> > > > > feedback.
> > > > > 
> > > > 
> > > > You can reuse a part of the commit description I suggested for previous patch.
> > > > 
> > > > Rest looks good to me.
> > > > 
> > > > - Mani
> > > > 
> > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > > ---
> > > > >  drivers/misc/pci_endpoint_test.c | 48 ++++++++++++++++++++++++++++++++
> > > > >  include/uapi/linux/pcitest.h     |  1 +
> > > > >  2 files changed, 49 insertions(+)
> > > > > 
> > > > > diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
> > > > > index ed4d0ef5e5c31..ed0b025132d17 100644
> > > > > --- a/drivers/misc/pci_endpoint_test.c
> > > > > +++ b/drivers/misc/pci_endpoint_test.c
> > > > > @@ -33,6 +33,8 @@
> > > > >  #define IRQ_TYPE_MSIX				2
> > > > >  
> > > > >  #define PCI_ENDPOINT_TEST_MAGIC			0x0
> > > > > +#define PCI_MAGIC_VERSION_MASK			GENMASK(7, 0)
> > > > > +#define PCI_ENDPOINT_TEST_V1			0x1
> > > > >  
> > > > >  #define PCI_ENDPOINT_TEST_COMMAND		0x4
> > > > >  #define COMMAND_RAISE_LEGACY_IRQ		BIT(0)
> > > > > @@ -52,6 +54,7 @@
> > > > >  #define STATUS_IRQ_RAISED			BIT(6)
> > > > >  #define STATUS_SRC_ADDR_INVALID			BIT(7)
> > > > >  #define STATUS_DST_ADDR_INVALID			BIT(8)
> > > > > +#define STATUS_DOORBELL_SUCCESS			BIT(9)
> > > > >  
> > > > >  #define PCI_ENDPOINT_TEST_LOWER_SRC_ADDR	0x0c
> > > > >  #define PCI_ENDPOINT_TEST_UPPER_SRC_ADDR	0x10
> > > > > @@ -66,7 +69,12 @@
> > > > >  #define PCI_ENDPOINT_TEST_IRQ_NUMBER		0x28
> > > > >  
> > > > >  #define PCI_ENDPOINT_TEST_FLAGS			0x2c
> > > > > +#define PCI_ENDPOINT_TEST_DB_BAR		0x30
> > > > > +#define PCI_ENDPOINT_TEST_DB_ADDR		0x34
> > > > > +#define PCI_ENDPOINT_TEST_DB_DATA		0x38
> > > > > +
> > > > >  #define FLAG_USE_DMA				BIT(0)
> > > > > +#define FLAG_SUPPORT_DOORBELL			BIT(1)
> > > > >  
> > > > >  #define PCI_DEVICE_ID_TI_AM654			0xb00c
> > > > >  #define PCI_DEVICE_ID_TI_J7200			0xb00f
> > > > > @@ -102,6 +110,7 @@ enum pci_barno {
> > > > >  	BAR_3,
> > > > >  	BAR_4,
> > > > >  	BAR_5,
> > > > > +	NO_BAR = -1,
> > > > >  };
> > > > >  
> > > > >  struct pci_endpoint_test {
> > > > > @@ -118,6 +127,7 @@ struct pci_endpoint_test {
> > > > >  	enum pci_barno test_reg_bar;
> > > > >  	size_t alignment;
> > > > >  	const char *name;
> > > > > +	u8 version;
> > > > >  };
> > > > >  
> > > > >  struct pci_endpoint_test_data {
> > > > > @@ -713,6 +723,38 @@ static bool pci_endpoint_test_set_irq(struct pci_endpoint_test *test,
> > > > >  	return false;
> > > > >  }
> > > > >  
> > > > > +static bool pci_endpoint_test_doorbell(struct pci_endpoint_test *test)
> > > > > +{
> > > > > +	enum pci_barno bar;
> > > > > +	u32 data, status;
> > > > > +	u32 addr;
> > > > > +
> > > > > +	if (test->version < PCI_ENDPOINT_TEST_V1)
> > > > > +		return false;
> > > > > +
> > > > > +	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
> > > > > +	if (bar == NO_BAR)
> > > > > +		return false;
> > > > > +
> > > > > +	data = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_DATA);
> > > > > +	addr = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_ADDR);
> > > > > +	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
> > > > > +
> > > > > +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_TYPE, irq_type);
> > > > > +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_NUMBER, 1);
> > > > > +
> > > > > +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_STATUS, 0);
> > > > > +	pci_endpoint_test_bar_writel(test, bar, addr, data);
> > > > > +
> > > > > +	wait_for_completion_timeout(&test->irq_raised, msecs_to_jiffies(1000));
> > > > > +
> > > > > +	status = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_STATUS);
> > > > > +	if (status & STATUS_DOORBELL_SUCCESS)
> > > > > +		return true;
> > > > > +
> > > > > +	return false;
> > > > > +}
> > > > > +
> > > > >  static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
> > > > >  				    unsigned long arg)
> > > > >  {
> > > > > @@ -760,6 +802,9 @@ static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
> > > > >  	case PCITEST_CLEAR_IRQ:
> > > > >  		ret = pci_endpoint_test_clear_irq(test);
> > > > >  		break;
> > > > > +	case PCITEST_DOORBELL:
> > > > > +		ret = pci_endpoint_test_doorbell(test);
> > > > > +		break;
> > > > >  	}
> > > > >  
> > > > >  ret:
> > > > > @@ -887,6 +932,9 @@ static int pci_endpoint_test_probe(struct pci_dev *pdev,
> > > > >  	misc_device->parent = &pdev->dev;
> > > > >  	misc_device->fops = &pci_endpoint_test_fops;
> > > > >  
> > > > > +	test->version = FIELD_GET(PCI_MAGIC_VERSION_MASK,
> > > > > +				  pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_MAGIC));
> > > > > +
> > > > >  	err = misc_register(misc_device);
> > > > >  	if (err) {
> > > > >  		dev_err(dev, "Failed to register device\n");
> > > > > diff --git a/include/uapi/linux/pcitest.h b/include/uapi/linux/pcitest.h
> > > > > index f9c1af8d141b4..479ca1aa3ae0b 100644
> > > > > --- a/include/uapi/linux/pcitest.h
> > > > > +++ b/include/uapi/linux/pcitest.h
> > > > > @@ -20,6 +20,7 @@
> > > > >  #define PCITEST_SET_IRQTYPE	_IOW('P', 0x8, int)
> > > > >  #define PCITEST_GET_IRQTYPE	_IO('P', 0x9)
> > > > >  #define PCITEST_CLEAR_IRQ	_IO('P', 0x10)
> > > > > +#define PCITEST_DOORBELL	_IO('P', 0x11)
> > > > >  
> > > > >  #define PCITEST_FLAGS_USE_DMA	0x00000001
> > > > >  
> > > > > -- 
> > > > > 2.34.1
> > > > > 
> > > > 
> > > > -- 
> > > > மணிவண்ணன் சதாசிவம்
> > > 
> > > -- 
> > > மணிவண்ணன் சதாசிவம்
> 
> -- 
> மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 4/5] misc: pci_endpoint_test: Add doorbell test case

[Manivannan Sadhasivam]

On Mon, Oct 30, 2023 at 02:33:33PM -0400, Frank Li wrote:
> On Fri, Oct 20, 2023 at 11:56:43PM +0530, Manivannan Sadhasivam wrote:
> > On Fri, Oct 20, 2023 at 02:07:02PM -0400, Frank Li wrote:
> > > On Fri, Oct 20, 2023 at 11:31:39PM +0530, Manivannan Sadhasivam wrote:
> > > > On Fri, Oct 20, 2023 at 11:23:04PM +0530, Manivannan Sadhasivam wrote:
> > > > > On Mon, Sep 11, 2023 at 06:09:19PM -0400, Frank Li wrote:
> > > > > > Using bit 0..7 of magic as version number in pci_endpoint_test struct to
> > > > > > support older driver versions. Save to 'version' field of struct
> > > > > > pci_endpoint_test to prevent reading non-existent address.
> > > > > > 
> > > > > 
> > > > > Since both drivers are in the kernel, I don't see a necessity to maintain
> > > > > compatibility. Does it make sense to load drivers of previous kernel revision
> > > > > with a new kernel?
> > > > > 
> > > > 
> > > > Shoot... Sorry, I completely forgot that one is EP and another is host. Yes, we
> > > > do need to maintain compatibility.
> > > > 
> > > > But can't we use the doorbell register contents to determine that?
> > > 
> > > Doorbell register is not exist at old EP driver. If old EP driver register
> > > size is 64Byte,  doorbell register is 64 - 68.
> > >
> > 
> > Yes, I know!
> >  
> > > Read unexisted, or unmapped space will cause kernel dump or other side
> > > effects.
> > > 
> > 
> > For sure it won't be unmapped as the BAR0 size is 512B. I thought we could infer
> > something from the uninitialized registers. I need to think about other options.
> > But changing the semantics of MAGIC register is a no-go.
> 
> @Mani:
> 
> Any other options? I plan respin these patches soon
> 

Nothing looks feasible to me. Kishon, any suggestion from your side?

In the worst case, I'd suggest to show a disclaimer that it is not compatible
with endpoints supporting older kernel revision and proceed. But let's wait for
Kishon.

- Mani

> Frank
> 
> > 
> > - Mani
> > 
> > > Frank
> > > 
> > > > 
> > > > - Mani
> > > > 
> > > > > > Add three registers: PCIE_ENDPOINT_TEST_DB_BAR, PCIE_ENDPOINT_TEST_DB_ADDR,
> > > > > > PCIE_ENDPOINT_TEST_DB_DATA.
> > > > > > 
> > > > > 
> > > > > This patch is not adding these registers and not this driver also. So this
> > > > > statement is wrong.
> > > > > 
> > > > > > Write data from PCI_ENDPOINT_TEST_DB_DATA to address from
> > > > > > PCI_ENDPOINT_TEST_DB_ADDR to trigger doorbell and wait for endpoint
> > > > > > feedback.
> > > > > > 
> > > > > 
> > > > > You can reuse a part of the commit description I suggested for previous patch.
> > > > > 
> > > > > Rest looks good to me.
> > > > > 
> > > > > - Mani
> > > > > 
> > > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > > > ---
> > > > > >  drivers/misc/pci_endpoint_test.c | 48 ++++++++++++++++++++++++++++++++
> > > > > >  include/uapi/linux/pcitest.h     |  1 +
> > > > > >  2 files changed, 49 insertions(+)
> > > > > > 
> > > > > > diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
> > > > > > index ed4d0ef5e5c31..ed0b025132d17 100644
> > > > > > --- a/drivers/misc/pci_endpoint_test.c
> > > > > > +++ b/drivers/misc/pci_endpoint_test.c
> > > > > > @@ -33,6 +33,8 @@
> > > > > >  #define IRQ_TYPE_MSIX				2
> > > > > >  
> > > > > >  #define PCI_ENDPOINT_TEST_MAGIC			0x0
> > > > > > +#define PCI_MAGIC_VERSION_MASK			GENMASK(7, 0)
> > > > > > +#define PCI_ENDPOINT_TEST_V1			0x1
> > > > > >  
> > > > > >  #define PCI_ENDPOINT_TEST_COMMAND		0x4
> > > > > >  #define COMMAND_RAISE_LEGACY_IRQ		BIT(0)
> > > > > > @@ -52,6 +54,7 @@
> > > > > >  #define STATUS_IRQ_RAISED			BIT(6)
> > > > > >  #define STATUS_SRC_ADDR_INVALID			BIT(7)
> > > > > >  #define STATUS_DST_ADDR_INVALID			BIT(8)
> > > > > > +#define STATUS_DOORBELL_SUCCESS			BIT(9)
> > > > > >  
> > > > > >  #define PCI_ENDPOINT_TEST_LOWER_SRC_ADDR	0x0c
> > > > > >  #define PCI_ENDPOINT_TEST_UPPER_SRC_ADDR	0x10
> > > > > > @@ -66,7 +69,12 @@
> > > > > >  #define PCI_ENDPOINT_TEST_IRQ_NUMBER		0x28
> > > > > >  
> > > > > >  #define PCI_ENDPOINT_TEST_FLAGS			0x2c
> > > > > > +#define PCI_ENDPOINT_TEST_DB_BAR		0x30
> > > > > > +#define PCI_ENDPOINT_TEST_DB_ADDR		0x34
> > > > > > +#define PCI_ENDPOINT_TEST_DB_DATA		0x38
> > > > > > +
> > > > > >  #define FLAG_USE_DMA				BIT(0)
> > > > > > +#define FLAG_SUPPORT_DOORBELL			BIT(1)
> > > > > >  
> > > > > >  #define PCI_DEVICE_ID_TI_AM654			0xb00c
> > > > > >  #define PCI_DEVICE_ID_TI_J7200			0xb00f
> > > > > > @@ -102,6 +110,7 @@ enum pci_barno {
> > > > > >  	BAR_3,
> > > > > >  	BAR_4,
> > > > > >  	BAR_5,
> > > > > > +	NO_BAR = -1,
> > > > > >  };
> > > > > >  
> > > > > >  struct pci_endpoint_test {
> > > > > > @@ -118,6 +127,7 @@ struct pci_endpoint_test {
> > > > > >  	enum pci_barno test_reg_bar;
> > > > > >  	size_t alignment;
> > > > > >  	const char *name;
> > > > > > +	u8 version;
> > > > > >  };
> > > > > >  
> > > > > >  struct pci_endpoint_test_data {
> > > > > > @@ -713,6 +723,38 @@ static bool pci_endpoint_test_set_irq(struct pci_endpoint_test *test,
> > > > > >  	return false;
> > > > > >  }
> > > > > >  
> > > > > > +static bool pci_endpoint_test_doorbell(struct pci_endpoint_test *test)
> > > > > > +{
> > > > > > +	enum pci_barno bar;
> > > > > > +	u32 data, status;
> > > > > > +	u32 addr;
> > > > > > +
> > > > > > +	if (test->version < PCI_ENDPOINT_TEST_V1)
> > > > > > +		return false;
> > > > > > +
> > > > > > +	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
> > > > > > +	if (bar == NO_BAR)
> > > > > > +		return false;
> > > > > > +
> > > > > > +	data = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_DATA);
> > > > > > +	addr = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_ADDR);
> > > > > > +	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
> > > > > > +
> > > > > > +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_TYPE, irq_type);
> > > > > > +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_NUMBER, 1);
> > > > > > +
> > > > > > +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_STATUS, 0);
> > > > > > +	pci_endpoint_test_bar_writel(test, bar, addr, data);
> > > > > > +
> > > > > > +	wait_for_completion_timeout(&test->irq_raised, msecs_to_jiffies(1000));
> > > > > > +
> > > > > > +	status = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_STATUS);
> > > > > > +	if (status & STATUS_DOORBELL_SUCCESS)
> > > > > > +		return true;
> > > > > > +
> > > > > > +	return false;
> > > > > > +}
> > > > > > +
> > > > > >  static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
> > > > > >  				    unsigned long arg)
> > > > > >  {
> > > > > > @@ -760,6 +802,9 @@ static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
> > > > > >  	case PCITEST_CLEAR_IRQ:
> > > > > >  		ret = pci_endpoint_test_clear_irq(test);
> > > > > >  		break;
> > > > > > +	case PCITEST_DOORBELL:
> > > > > > +		ret = pci_endpoint_test_doorbell(test);
> > > > > > +		break;
> > > > > >  	}
> > > > > >  
> > > > > >  ret:
> > > > > > @@ -887,6 +932,9 @@ static int pci_endpoint_test_probe(struct pci_dev *pdev,
> > > > > >  	misc_device->parent = &pdev->dev;
> > > > > >  	misc_device->fops = &pci_endpoint_test_fops;
> > > > > >  
> > > > > > +	test->version = FIELD_GET(PCI_MAGIC_VERSION_MASK,
> > > > > > +				  pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_MAGIC));
> > > > > > +
> > > > > >  	err = misc_register(misc_device);
> > > > > >  	if (err) {
> > > > > >  		dev_err(dev, "Failed to register device\n");
> > > > > > diff --git a/include/uapi/linux/pcitest.h b/include/uapi/linux/pcitest.h
> > > > > > index f9c1af8d141b4..479ca1aa3ae0b 100644
> > > > > > --- a/include/uapi/linux/pcitest.h
> > > > > > +++ b/include/uapi/linux/pcitest.h
> > > > > > @@ -20,6 +20,7 @@
> > > > > >  #define PCITEST_SET_IRQTYPE	_IOW('P', 0x8, int)
> > > > > >  #define PCITEST_GET_IRQTYPE	_IO('P', 0x9)
> > > > > >  #define PCITEST_CLEAR_IRQ	_IO('P', 0x10)
> > > > > > +#define PCITEST_DOORBELL	_IO('P', 0x11)
> > > > > >  
> > > > > >  #define PCITEST_FLAGS_USE_DMA	0x00000001
> > > > > >  
> > > > > > -- 
> > > > > > 2.34.1
> > > > > > 
> > > > > 
> > > > > -- 
> > > > > மணிவண்ணன் சதாசிவம்
> > > > 
> > > > -- 
> > > > மணிவண்ணன் சதாசிவம்
> > 
> > -- 
> > மணிவண்ணன் சதாசிவம்

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v2 4/5] misc: pci_endpoint_test: Add doorbell test case

[Kishon Vijay Abraham I]

Hi Frank,

On 9/12/2023 3:39 AM, Frank Li wrote:
> Using bit 0..7 of magic as version number in pci_endpoint_test struct to
> support older driver versions. Save to 'version' field of struct
> pci_endpoint_test to prevent reading non-existent address.
> 
> Add three registers: PCIE_ENDPOINT_TEST_DB_BAR, PCIE_ENDPOINT_TEST_DB_ADDR,
> PCIE_ENDPOINT_TEST_DB_DATA.
> 
> Write data from PCI_ENDPOINT_TEST_DB_DATA to address from
> PCI_ENDPOINT_TEST_DB_ADDR to trigger doorbell and wait for endpoint
> feedback.
> 
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---
>   drivers/misc/pci_endpoint_test.c | 48 ++++++++++++++++++++++++++++++++
>   include/uapi/linux/pcitest.h     |  1 +
>   2 files changed, 49 insertions(+)
> 
> diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c
> index ed4d0ef5e5c31..ed0b025132d17 100644
> --- a/drivers/misc/pci_endpoint_test.c
> +++ b/drivers/misc/pci_endpoint_test.c
> @@ -33,6 +33,8 @@
>   #define IRQ_TYPE_MSIX				2
>   
>   #define PCI_ENDPOINT_TEST_MAGIC			0x0
> +#define PCI_MAGIC_VERSION_MASK			GENMASK(7, 0)
> +#define PCI_ENDPOINT_TEST_V1			0x1
>   
>   #define PCI_ENDPOINT_TEST_COMMAND		0x4
>   #define COMMAND_RAISE_LEGACY_IRQ		BIT(0)
> @@ -52,6 +54,7 @@
>   #define STATUS_IRQ_RAISED			BIT(6)
>   #define STATUS_SRC_ADDR_INVALID			BIT(7)
>   #define STATUS_DST_ADDR_INVALID			BIT(8)
> +#define STATUS_DOORBELL_SUCCESS			BIT(9)
>   
>   #define PCI_ENDPOINT_TEST_LOWER_SRC_ADDR	0x0c
>   #define PCI_ENDPOINT_TEST_UPPER_SRC_ADDR	0x10
> @@ -66,7 +69,12 @@
>   #define PCI_ENDPOINT_TEST_IRQ_NUMBER		0x28
>   
>   #define PCI_ENDPOINT_TEST_FLAGS			0x2c
> +#define PCI_ENDPOINT_TEST_DB_BAR		0x30
> +#define PCI_ENDPOINT_TEST_DB_ADDR		0x34
> +#define PCI_ENDPOINT_TEST_DB_DATA		0x38
> +
>   #define FLAG_USE_DMA				BIT(0)
> +#define FLAG_SUPPORT_DOORBELL			BIT(1)
>   
>   #define PCI_DEVICE_ID_TI_AM654			0xb00c
>   #define PCI_DEVICE_ID_TI_J7200			0xb00f
> @@ -102,6 +110,7 @@ enum pci_barno {
>   	BAR_3,
>   	BAR_4,
>   	BAR_5,
> +	NO_BAR = -1,
>   };
>   
>   struct pci_endpoint_test {
> @@ -118,6 +127,7 @@ struct pci_endpoint_test {
>   	enum pci_barno test_reg_bar;
>   	size_t alignment;
>   	const char *name;
> +	u8 version;
>   };
>   
>   struct pci_endpoint_test_data {
> @@ -713,6 +723,38 @@ static bool pci_endpoint_test_set_irq(struct pci_endpoint_test *test,
>   	return false;
>   }
>   
> +static bool pci_endpoint_test_doorbell(struct pci_endpoint_test *test)
> +{
> +	enum pci_barno bar;
> +	u32 data, status;
> +	u32 addr;
> +
> +	if (test->version < PCI_ENDPOINT_TEST_V1)
> +		return false;
> +
> +	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
> +	if (bar == NO_BAR)
> +		return false;
> +
> +	data = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_DATA);
> +	addr = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_ADDR);
> +	bar = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_DB_BAR);
> +
> +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_TYPE, irq_type);
> +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_IRQ_NUMBER, 1);
> +
> +	pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_STATUS, 0);
> +	pci_endpoint_test_bar_writel(test, bar, addr, data);
> +
> +	wait_for_completion_timeout(&test->irq_raised, msecs_to_jiffies(1000));
> +
> +	status = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_STATUS);
> +	if (status & STATUS_DOORBELL_SUCCESS)
> +		return true;
> +
> +	return false;
> +}
> +
>   static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
>   				    unsigned long arg)
>   {
> @@ -760,6 +802,9 @@ static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd,
>   	case PCITEST_CLEAR_IRQ:
>   		ret = pci_endpoint_test_clear_irq(test);
>   		break;
> +	case PCITEST_DOORBELL:
> +		ret = pci_endpoint_test_doorbell(test);
> +		break;
>   	}
>   
>   ret:
> @@ -887,6 +932,9 @@ static int pci_endpoint_test_probe(struct pci_dev *pdev,
>   	misc_device->parent = &pdev->dev;
>   	misc_device->fops = &pci_endpoint_test_fops;
>   
> +	test->version = FIELD_GET(PCI_MAGIC_VERSION_MASK,
> +				  pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_MAGIC));

After running BAR test, the value in MAGIC register would get 
over-written and this is bound to break.

IIUC you are also not handling the case where BAR test would write to DB 
BARS.

IMO handling the above two is important than trying to find whether the 
endpoint device supports DB. In the worst case, DB interrupt would not 
be triggered which is okay IMO.

Thanks,
Kishon

> +
>   	err = misc_register(misc_device);
>   	if (err) {
>   		dev_err(dev, "Failed to register device\n");
> diff --git a/include/uapi/linux/pcitest.h b/include/uapi/linux/pcitest.h
> index f9c1af8d141b4..479ca1aa3ae0b 100644
> --- a/include/uapi/linux/pcitest.h
> +++ b/include/uapi/linux/pcitest.h
> @@ -20,6 +20,7 @@
>   #define PCITEST_SET_IRQTYPE	_IOW('P', 0x8, int)
>   #define PCITEST_GET_IRQTYPE	_IO('P', 0x9)
>   #define PCITEST_CLEAR_IRQ	_IO('P', 0x10)
> +#define PCITEST_DOORBELL	_IO('P', 0x11)
>   
>   #define PCITEST_FLAGS_USE_DMA	0x00000001
>