[PATCH v7 0/4] PCI EP driver support MSI doorbell from host

[PATCH v7 0/4] PCI EP driver support MSI doorbell from host

[Frank Li]

                  ┌───────┐          ┌──────────┐
                  │       │          │          │
┌─────────────┐   │       │          │ PCI Host │
│ MSI         │◄┐ │       │          │          │
│ Controller  │ │ │       │          │          │
└─────────────┘ └─┼───────┼──────────┼─Bar0     │
                  │ PCI   │          │ Bar1     │
                  │ Func  │          │ Bar2     │
                  │       │          │ Bar3     │
                  │       │          │ Bar4     │
                  │       ├─────────►│          │
                  └───────┘          └──────────┘

Many PCI controllers provided Endpoint functions.
Generally PCI endpoint is hardware, which is not running a rich OS,
like linux.

But Linux also supports endpoint functions.  PCI Host write BAR<n> space
like write to memory. The EP side can't know memory changed by the Host
driver. 

PCI Spec has not defined a standard method to do that.  Only define
MSI(x) to let EP notified RC status change. 

The basic idea is to trigger an IRQ when PCI RC writes to a memory
address. That's what MSI controller provided.  EP drivers just need to
request a platform MSI interrupt, struct MSI_msg *msg will pass down a
memory address and data.  EP driver will map such memory address to
one of PCI BAR<n>.  Host just writes such an address to trigger EP side
IRQ.

If system have gic-its, only need update PCI EP side driver. But i.MX
have not chip support gic-its yet. So we have to use MU to simulate a
MSI controller. Although only 4 MSI IRQs are simulated, it matched
vntb(pci-epf-vntb) network requirement.

After enable MSI, ping delay reduce < 1ms from ~8ms

IRQchip: imx mu worked as MSI controller: 
     let imx mu worked as MSI controllers. Although IP is not design
as MSI controller, we still can use it if limited IRQ number to 4.

pcie: endpoint: pci-epf-vntb: add endpoint MSI support
	 Based on ntb-next branch. https://github.com/jonmason/ntb/commits/ntb-next
	 Using MSI as door bell registers
	 This patch is totally independent on previous on. It can be
applied to ntb-next seperately.

i.MX EP function driver is upstreaming by Richard Zhu.
Some dts change missed at this patches. below is reference dts change

--- a/arch/arm64/boot/dts/freescale/imx8-ss-hsio.dtsi
+++ b/arch/arm64/boot/dts/freescale/imx8-ss-hsio.dtsi
@@ -160,5 +160,6 @@ pcieb_ep: pcie_ep@5f010000 {
                num-ib-windows = <6>;
                num-ob-windows = <6>;
                status = "disabled";
+               MSI-parent = <&lsio_mu12>;
        };

--- a/arch/arm64/boot/dts/freescale/imx8-ss-lsio.dtsi
+++ b/arch/arm64/boot/dts/freescale/imx8-ss-lsio.dtsi
@@ -172,6 +172,19 @@ lsio_mu6: mailbox@5d210000 {
                status = "disabled";
        };

+       lsio_mu12: mailbox@5d270000 {
+               compatible = "fsl,imx6sx-mu-MSI";
+               msi-controller;
+               interrupt-controller;
+               reg = <0x5d270000 0x10000>,     /* A side */
+                     <0x5d300000 0x10000>;     /* B side */
+               reg-names = "a", "b";
+               interrupts = <GIC_SPI 191 IRQ_TYPE_LEVEL_HIGH>;
+               power-domains = <&pd IMX_SC_R_MU_12A>,
+                               <&pd IMX_SC_R_MU_12B>;
+               power-domain-names = "a", "b";
+       };
+

Change Log
- Change from v6 to v7
  pcie: endpoint: add endpoint MSI support
  Fine tuning commit message
  Fixed issues, reviewed by Bjorn Helgaas

- Change from v5 to v6
  Fixed build error found by kernel test robot

- Change from v4 to v5
  Fixed dt-binding document
        add msi-cell
        add interrupt max number
	update naming reg-names and power-domain-names.
  Fixed irqchip-Add-IMX-MU-MSI-controller-driver.patch
        rework commit message
        remove some field in struct imx_mu_dcfg
	error handle when link power domain failure.
	add irq_domain_update_bus_token

- Change from v3 to v4
  Fixed dt-binding document according to Krzysztof Kozlowski's feedback
  Fixed irqchip-imx-mu-worked-as-msi-controller according to Marc Zyngier's
        comments.

	There are still two important points, which I am not sure.
	1. clean irq_set_affinity after platform_msi_create_irq_domain.
	   Some function, like platform_msi_write_msg() is static.
	   so I have to set MSI_FLAG_USE_DEF_CHIP_OPS flags, which will
	   set irq_set_affinity to default one.
	2. about comments

	> +	msi_data->msi_domain = platform_msi_create_irq_domain(
	> +				of_node_to_fwnode(msi_data->pdev->dev.of_node),
	> +				&imx_mu_msi_domain_info,
	> +				msi_data->parent);

	"And you don't get an error due to the fact that you use the same
	fwnode for both domains without overriding the domain bus token?"

 	I did not understand yet. 

  Fixed static check warning, reported by Dan Carpenter
	pcie: endpoint: pci-epf-vntb: add endpoint MSI support

- Change from v2 to v3
  Fixed dt-binding docment check failure
  Fixed typo a cover letter.
  Change according Bjorn's comments at patch 
	pcie: endpoint: pci-epf-vntb: add endpoint MSI support
	 

- from V1 to V2
  Fixed fsl,mu-msi.yaml's problem
  Fixed irq-imx-mu-msi.c problem according Marc Zyngier's feeback 
  Added a new patch to allow pass down .pm by IRQCHIP_PLATFORM_DRIVER_END

-- 
2.35.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v7 1/4] irqchip: allow pass down .pm field at IRQCHIP_PLATFORM_DRIVER_END

[Frank Li]

IRQCHIP_PLATFORM_DRIVER_* compilation define platform_driver
for irqchip. But can't set .pm field of platform_driver.
Added variadic macros to set .pm field or other field if need.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
 include/linux/irqchip.h | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/include/linux/irqchip.h b/include/linux/irqchip.h
index 3a091d0710ae1..d5e6024cb2a8c 100644
--- a/include/linux/irqchip.h
+++ b/include/linux/irqchip.h
@@ -44,7 +44,8 @@ static const struct of_device_id drv_name##_irqchip_match_table[] = {
 #define IRQCHIP_MATCH(compat, fn) { .compatible = compat,		\
 				    .data = typecheck_irq_init_cb(fn), },
 
-#define IRQCHIP_PLATFORM_DRIVER_END(drv_name)				\
+
+#define IRQCHIP_PLATFORM_DRIVER_END(drv_name, ...)			\
 	{},								\
 };									\
 MODULE_DEVICE_TABLE(of, drv_name##_irqchip_match_table);		\
@@ -56,6 +57,7 @@ static struct platform_driver drv_name##_driver = {			\
 		.owner = THIS_MODULE,					\
 		.of_match_table = drv_name##_irqchip_match_table,	\
 		.suppress_bind_attrs = true,				\
+		__VA_ARGS__						\
 	},								\
 };									\
 builtin_platform_driver(drv_name##_driver)
-- 
2.35.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v7 2/4] irqchip: Add IMX MU MSI controller driver

[Frank Li]

The MU block found in a number of Freescale/NXP SoCs supports generating
IRQs by writing data to a register

This enables the MU block to be used as a MSI controller, by leveraging
the platform-MSI API

Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
 drivers/irqchip/Kconfig          |   9 +
 drivers/irqchip/Makefile         |   1 +
 drivers/irqchip/irq-imx-mu-msi.c | 451 +++++++++++++++++++++++++++++++
 3 files changed, 461 insertions(+)
 create mode 100644 drivers/irqchip/irq-imx-mu-msi.c

diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
index 5e4e50122777d..e04c6521dce55 100644
--- a/drivers/irqchip/Kconfig
+++ b/drivers/irqchip/Kconfig
@@ -470,6 +470,15 @@ config IMX_INTMUX
 	help
 	  Support for the i.MX INTMUX interrupt multiplexer.
 
+config IMX_MU_MSI
+	bool "i.MX MU work as MSI controller"
+	default y if ARCH_MXC
+	select IRQ_DOMAIN
+	select IRQ_DOMAIN_HIERARCHY
+	select GENERIC_MSI_IRQ_DOMAIN
+	help
+	  MU work as MSI controller to do general doorbell
+
 config LS1X_IRQ
 	bool "Loongson-1 Interrupt Controller"
 	depends on MACH_LOONGSON32
diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
index 5d8e21d3dc6d8..870423746c783 100644
--- a/drivers/irqchip/Makefile
+++ b/drivers/irqchip/Makefile
@@ -98,6 +98,7 @@ obj-$(CONFIG_RISCV_INTC)		+= irq-riscv-intc.o
 obj-$(CONFIG_SIFIVE_PLIC)		+= irq-sifive-plic.o
 obj-$(CONFIG_IMX_IRQSTEER)		+= irq-imx-irqsteer.o
 obj-$(CONFIG_IMX_INTMUX)		+= irq-imx-intmux.o
+obj-$(CONFIG_IMX_MU_MSI)		+= irq-imx-mu-msi.o
 obj-$(CONFIG_MADERA_IRQ)		+= irq-madera.o
 obj-$(CONFIG_LS1X_IRQ)			+= irq-ls1x.o
 obj-$(CONFIG_TI_SCI_INTR_IRQCHIP)	+= irq-ti-sci-intr.o
diff --git a/drivers/irqchip/irq-imx-mu-msi.c b/drivers/irqchip/irq-imx-mu-msi.c
new file mode 100644
index 0000000000000..110e5df1d6aa8
--- /dev/null
+++ b/drivers/irqchip/irq-imx-mu-msi.c
@@ -0,0 +1,451 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Freescale MU worked as MSI controller
+ *
+ * Copyright (c) 2018 Pengutronix, Oleksij Rempel <o.rempel@pengutronix.de>
+ * Copyright 2022 NXP
+ *	Frank Li <Frank.Li@nxp.com>
+ *	Peng Fan <peng.fan@nxp.com>
+ *
+ * Based on drivers/mailbox/imx-mailbox.c
+ */
+#include <linux/clk.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/msi.h>
+#include <linux/interrupt.h>
+#include <linux/irq.h>
+#include <linux/irqchip/chained_irq.h>
+#include <linux/irqchip.h>
+#include <linux/irqdomain.h>
+#include <linux/of_irq.h>
+#include <linux/of_pci.h>
+#include <linux/of_platform.h>
+#include <linux/spinlock.h>
+#include <linux/dma-iommu.h>
+#include <linux/pm_runtime.h>
+#include <linux/pm_domain.h>
+
+
+#define IMX_MU_CHANS            4
+
+enum imx_mu_xcr {
+	IMX_MU_GIER,
+	IMX_MU_GCR,
+	IMX_MU_TCR,
+	IMX_MU_RCR,
+	IMX_MU_xCR_MAX,
+};
+
+enum imx_mu_xsr {
+	IMX_MU_SR,
+	IMX_MU_GSR,
+	IMX_MU_TSR,
+	IMX_MU_RSR,
+};
+
+enum imx_mu_type {
+	IMX_MU_V1 = BIT(0),
+	IMX_MU_V2 = BIT(1),
+	IMX_MU_V2_S4 = BIT(15),
+};
+
+/* Receive Interrupt Enable */
+#define IMX_MU_xCR_RIEn(data, x) ((data->cfg->type) & IMX_MU_V2 ? BIT(x) : BIT(24 + (3 - (x))))
+#define IMX_MU_xSR_RFn(data, x) ((data->cfg->type) & IMX_MU_V2 ? BIT(x) : BIT(24 + (3 - (x))))
+
+struct imx_mu_dcfg {
+	enum imx_mu_type type;
+	u32     xTR;            /* Transmit Register0 */
+	u32     xRR;            /* Receive Register0 */
+	u32     xSR[4];         /* Status Registers */
+	u32     xCR[4];         /* Control Registers */
+};
+
+struct imx_mu_msi {
+	spinlock_t			lock;
+	raw_spinlock_t			reglock;
+	struct irq_domain		*msi_domain;
+	void __iomem			*regs;
+	phys_addr_t			msiir_addr;
+	const struct imx_mu_dcfg	*cfg;
+	unsigned long			used;
+	struct clk			*clk;
+};
+
+static void imx_mu_write(struct imx_mu_msi *msi_data, u32 val, u32 offs)
+{
+	iowrite32(val, msi_data->regs + offs);
+}
+
+static u32 imx_mu_read(struct imx_mu_msi *msi_data, u32 offs)
+{
+	return ioread32(msi_data->regs + offs);
+}
+
+static u32 imx_mu_xcr_rmw(struct imx_mu_msi *msi_data, enum imx_mu_xcr type, u32 set, u32 clr)
+{
+	unsigned long flags;
+	u32 val;
+
+	raw_spin_lock_irqsave(&msi_data->reglock, flags);
+	val = imx_mu_read(msi_data, msi_data->cfg->xCR[type]);
+	val &= ~clr;
+	val |= set;
+	imx_mu_write(msi_data, val, msi_data->cfg->xCR[type]);
+	raw_spin_unlock_irqrestore(&msi_data->reglock, flags);
+
+	return val;
+}
+
+static void imx_mu_msi_parent_mask_irq(struct irq_data *data)
+{
+	struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
+
+	imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, 0, IMX_MU_xCR_RIEn(msi_data, data->hwirq));
+}
+
+static void imx_mu_msi_parent_unmask_irq(struct irq_data *data)
+{
+	struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
+
+	imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, IMX_MU_xCR_RIEn(msi_data, data->hwirq), 0);
+}
+
+static void imx_mu_msi_parent_ack_irq(struct irq_data *data)
+{
+	struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
+
+	imx_mu_read(msi_data, msi_data->cfg->xRR + data->hwirq * 4);
+}
+
+static struct irq_chip imx_mu_msi_irq_chip = {
+	.name = "MU-MSI",
+	.irq_ack = irq_chip_ack_parent,
+};
+
+static struct msi_domain_ops imx_mu_msi_irq_ops = {
+};
+
+static struct msi_domain_info imx_mu_msi_domain_info = {
+	.flags	= (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS),
+	.ops	= &imx_mu_msi_irq_ops,
+	.chip	= &imx_mu_msi_irq_chip,
+};
+
+static void imx_mu_msi_parent_compose_msg(struct irq_data *data,
+					  struct msi_msg *msg)
+{
+	struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
+	u64 addr = msi_data->msiir_addr + 4 * data->hwirq;
+
+	msg->address_hi = upper_32_bits(addr);
+	msg->address_lo = lower_32_bits(addr);
+	msg->data = data->hwirq;
+}
+
+static int imx_mu_msi_parent_set_affinity(struct irq_data *irq_data,
+				   const struct cpumask *mask, bool force)
+{
+	return -EINVAL;
+}
+
+static struct irq_chip imx_mu_msi_parent_chip = {
+	.name		= "MU",
+	.irq_mask	= imx_mu_msi_parent_mask_irq,
+	.irq_unmask	= imx_mu_msi_parent_unmask_irq,
+	.irq_ack	= imx_mu_msi_parent_ack_irq,
+	.irq_compose_msi_msg	= imx_mu_msi_parent_compose_msg,
+	.irq_set_affinity = imx_mu_msi_parent_set_affinity,
+};
+
+static int imx_mu_msi_domain_irq_alloc(struct irq_domain *domain,
+					unsigned int virq,
+					unsigned int nr_irqs,
+					void *args)
+{
+	struct imx_mu_msi *msi_data = domain->host_data;
+	unsigned long flags;
+	int pos, err = 0;
+
+	WARN_ON(nr_irqs != 1);
+
+	spin_lock_irqsave(&msi_data->lock, flags);
+	pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
+	if (pos < IMX_MU_CHANS)
+		__set_bit(pos, &msi_data->used);
+	else
+		err = -ENOSPC;
+	spin_unlock_irqrestore(&msi_data->lock, flags);
+
+	if (err)
+		return err;
+
+	irq_domain_set_info(domain, virq, pos,
+			    &imx_mu_msi_parent_chip, msi_data,
+			    handle_edge_irq, NULL, NULL);
+	return 0;
+}
+
+static void imx_mu_msi_domain_irq_free(struct irq_domain *domain,
+				       unsigned int virq, unsigned int nr_irqs)
+{
+	struct irq_data *d = irq_domain_get_irq_data(domain, virq);
+	struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(d);
+	unsigned long flags;
+
+	spin_lock_irqsave(&msi_data->lock, flags);
+	__clear_bit(d->hwirq, &msi_data->used);
+	spin_unlock_irqrestore(&msi_data->lock, flags);
+}
+
+static const struct irq_domain_ops imx_mu_msi_domain_ops = {
+	.alloc	= imx_mu_msi_domain_irq_alloc,
+	.free	= imx_mu_msi_domain_irq_free,
+};
+
+static void imx_mu_msi_irq_handler(struct irq_desc *desc)
+{
+	struct imx_mu_msi *msi_data = irq_desc_get_handler_data(desc);
+	struct irq_chip *chip = irq_desc_get_chip(desc);
+	u32 status;
+	int i;
+
+	status = imx_mu_read(msi_data, msi_data->cfg->xSR[IMX_MU_RSR]);
+
+	chained_irq_enter(chip, desc);
+	for (i = 0; i < IMX_MU_CHANS; i++) {
+		if (status & IMX_MU_xSR_RFn(msi_data, i))
+			generic_handle_domain_irq(msi_data->msi_domain, i);
+	}
+	chained_irq_exit(chip, desc);
+}
+
+static int imx_mu_msi_domains_init(struct imx_mu_msi *msi_data, struct device *dev)
+{
+	struct fwnode_handle *fwnodes = dev_fwnode(dev);
+	struct irq_domain *parent;
+
+	/* Initialize MSI domain parent */
+	parent = irq_domain_create_linear(fwnodes,
+					    IMX_MU_CHANS,
+					    &imx_mu_msi_domain_ops,
+					    msi_data);
+	if (!parent) {
+		dev_err(dev, "failed to create IRQ domain\n");
+		return -ENOMEM;
+	}
+
+	irq_domain_update_bus_token(parent, DOMAIN_BUS_NEXUS);
+
+	msi_data->msi_domain = platform_msi_create_irq_domain(
+				fwnodes,
+				&imx_mu_msi_domain_info,
+				parent);
+
+	if (!msi_data->msi_domain) {
+		dev_err(dev, "failed to create MSI domain\n");
+		irq_domain_remove(parent);
+		return -ENOMEM;
+	}
+
+	irq_domain_set_pm_device(msi_data->msi_domain, dev);
+
+	return 0;
+}
+
+/* Register offset of different version MU IP */
+static const struct imx_mu_dcfg imx_mu_cfg_imx6sx = {
+	.xTR    = 0x0,
+	.xRR    = 0x10,
+	.xSR    = {0x20, 0x20, 0x20, 0x20},
+	.xCR    = {0x24, 0x24, 0x24, 0x24},
+};
+
+static const struct imx_mu_dcfg imx_mu_cfg_imx7ulp = {
+	.xTR    = 0x20,
+	.xRR    = 0x40,
+	.xSR    = {0x60, 0x60, 0x60, 0x60},
+	.xCR    = {0x64, 0x64, 0x64, 0x64},
+};
+
+static const struct imx_mu_dcfg imx_mu_cfg_imx8ulp = {
+	.type   = IMX_MU_V2,
+	.xTR    = 0x200,
+	.xRR    = 0x280,
+	.xSR    = {0xC, 0x118, 0x124, 0x12C},
+	.xCR    = {0x110, 0x114, 0x120, 0x128},
+};
+
+static const struct imx_mu_dcfg imx_mu_cfg_imx8ulp_s4 = {
+
+	.type   = IMX_MU_V2 | IMX_MU_V2_S4,
+	.xTR    = 0x200,
+	.xRR    = 0x280,
+	.xSR    = {0xC, 0x118, 0x124, 0x12C},
+	.xCR    = {0x110, 0x114, 0x120, 0x128},
+};
+
+static int __init imx_mu_of_init(struct device_node *dn,
+				 struct device_node *parent,
+				 const struct imx_mu_dcfg *cfg
+				)
+{
+	struct platform_device *pdev = of_find_device_by_node(dn);
+	struct device_link *pd_link_a;
+	struct device_link *pd_link_b;
+	struct imx_mu_msi *msi_data;
+	struct resource *res;
+	struct device *pd_a;
+	struct device *pd_b;
+	struct device *dev;
+	int ret;
+	int irq;
+
+	if (!pdev)
+		return -ENODEV;
+
+	dev = &pdev->dev;
+
+	msi_data = devm_kzalloc(&pdev->dev, sizeof(*msi_data), GFP_KERNEL);
+	if (!msi_data)
+		return -ENOMEM;
+
+	msi_data->cfg = cfg;
+
+	msi_data->regs = devm_platform_ioremap_resource_byname(pdev, "processor a-facing");
+	if (IS_ERR(msi_data->regs)) {
+		dev_err(&pdev->dev, "failed to initialize 'regs'\n");
+		return PTR_ERR(msi_data->regs);
+	}
+
+	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "processor b-facing");
+	if (!res)
+		return -EIO;
+
+	msi_data->msiir_addr = res->start + msi_data->cfg->xTR;
+
+	irq = platform_get_irq(pdev, 0);
+	if (irq <= 0)
+		return -ENODEV;
+
+	platform_set_drvdata(pdev, msi_data);
+
+	msi_data->clk = devm_clk_get(dev, NULL);
+	if (IS_ERR(msi_data->clk)) {
+		if (PTR_ERR(msi_data->clk) != -ENOENT)
+			return PTR_ERR(msi_data->clk);
+
+		msi_data->clk = NULL;
+	}
+
+	pd_a = dev_pm_domain_attach_by_name(dev, "processor a-facing");
+	if (IS_ERR(pd_a))
+		return PTR_ERR(pd_a);
+
+	pd_b = dev_pm_domain_attach_by_name(dev, "processor b-facing");
+	if (IS_ERR(pd_b))
+		return PTR_ERR(pd_b);
+
+	pd_link_a = device_link_add(dev, pd_a,
+			DL_FLAG_STATELESS |
+			DL_FLAG_PM_RUNTIME |
+			DL_FLAG_RPM_ACTIVE);
+
+	if (!pd_link_a) {
+		dev_err(dev, "Failed to add device_link to mu a.\n");
+		goto err_pd_a;
+	}
+
+	pd_link_b = device_link_add(dev, pd_b,
+			DL_FLAG_STATELESS |
+			DL_FLAG_PM_RUNTIME |
+			DL_FLAG_RPM_ACTIVE);
+
+
+	if (!pd_link_b) {
+		dev_err(dev, "Failed to add device_link to mu a.\n");
+		goto err_pd_b;
+	}
+
+	ret = imx_mu_msi_domains_init(msi_data, dev);
+	if (ret)
+		goto err_dm_init;
+
+	irq_set_chained_handler_and_data(irq,
+					 imx_mu_msi_irq_handler,
+					 msi_data);
+
+	pm_runtime_enable(dev);
+
+	return 0;
+
+err_dm_init:
+	device_link_remove(dev,	pd_b);
+err_pd_b:
+	device_link_remove(dev, pd_a);
+err_pd_a:
+	return -EINVAL;
+}
+
+static int __maybe_unused imx_mu_runtime_suspend(struct device *dev)
+{
+	struct imx_mu_msi *priv = dev_get_drvdata(dev);
+
+	clk_disable_unprepare(priv->clk);
+
+	return 0;
+}
+
+static int __maybe_unused imx_mu_runtime_resume(struct device *dev)
+{
+	struct imx_mu_msi *priv = dev_get_drvdata(dev);
+	int ret;
+
+	ret = clk_prepare_enable(priv->clk);
+	if (ret)
+		dev_err(dev, "failed to enable clock\n");
+
+	return ret;
+}
+
+static const struct dev_pm_ops imx_mu_pm_ops = {
+	SET_RUNTIME_PM_OPS(imx_mu_runtime_suspend,
+			   imx_mu_runtime_resume, NULL)
+};
+
+static int __init imx_mu_imx7ulp_of_init(struct device_node *dn,
+					 struct device_node *parent)
+{
+	return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx7ulp);
+}
+
+static int __init imx_mu_imx6sx_of_init(struct device_node *dn,
+					struct device_node *parent)
+{
+	return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx6sx);
+}
+
+static int __init imx_mu_imx8ulp_of_init(struct device_node *dn,
+					 struct device_node *parent)
+{
+	return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx8ulp);
+}
+
+static int __init imx_mu_imx8ulp_s4_of_init(struct device_node *dn,
+					    struct device_node *parent)
+{
+	return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx8ulp_s4);
+}
+
+IRQCHIP_PLATFORM_DRIVER_BEGIN(imx_mu_msi)
+IRQCHIP_MATCH("fsl,imx7ulp-mu-msi", imx_mu_imx7ulp_of_init)
+IRQCHIP_MATCH("fsl,imx6sx-mu-msi", imx_mu_imx6sx_of_init)
+IRQCHIP_MATCH("fsl,imx8ulp-mu-msi", imx_mu_imx8ulp_of_init)
+IRQCHIP_MATCH("fsl,imx8ulp-mu-msi-s4", imx_mu_imx8ulp_s4_of_init)
+IRQCHIP_PLATFORM_DRIVER_END(imx_mu_msi, .pm = &imx_mu_pm_ops)
+
+
+MODULE_AUTHOR("Frank Li <Frank.Li@nxp.com>");
+MODULE_DESCRIPTION("Freescale MU MSI controller driver");
+MODULE_LICENSE("GPL");
-- 
2.35.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v7 3/4] dt-bindings: irqchip: imx mu work as msi controller

[Frank Li]

I.MX mu support generate irq by write a register. Provide msi controller
support so other driver such as PCI EP can use it by standard msi
interface as doorbell.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
 .../interrupt-controller/fsl,mu-msi.yaml      | 98 +++++++++++++++++++
 1 file changed, 98 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml

diff --git a/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
new file mode 100644
index 0000000000000..ac07b138e24c0
--- /dev/null
+++ b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
@@ -0,0 +1,98 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/interrupt-controller/fsl,mu-msi.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Freescale/NXP i.MX Messaging Unit (MU) work as msi controller
+
+maintainers:
+  - Frank Li <Frank.Li@nxp.com>
+
+description: |
+  The Messaging Unit module enables two processors within the SoC to
+  communicate and coordinate by passing messages (e.g. data, status
+  and control) through the MU interface. The MU also provides the ability
+  for one processor (A side) to signal the other processor (B side) using
+  interrupts.
+
+  Because the MU manages the messaging between processors, the MU uses
+  different clocks (from each side of the different peripheral buses).
+  Therefore, the MU must synchronize the accesses from one side to the
+  other. The MU accomplishes synchronization using two sets of matching
+  registers (Processor A-facing, Processor B-facing).
+
+  MU can work as msi interrupt controller to do doorbell
+
+allOf:
+  - $ref: /schemas/interrupt-controller/msi-controller.yaml#
+
+properties:
+  compatible:
+    enum:
+      - fsl,imx6sx-mu-msi
+      - fsl,imx7ulp-mu-msi
+      - fsl,imx8ulp-mu-msi
+      - fsl,imx8ulp-mu-msi-s4
+
+  reg:
+    items:
+      - description: a side register base address
+      - description: b side register base address
+
+  reg-names:
+    items:
+      - const: processor a-facing
+      - const: processor b-facing
+
+  interrupts:
+    description: a side interrupt number.
+    maxItems: 1
+
+  clocks:
+    maxItems: 1
+
+  power-domains:
+    items:
+      - description: a side power domain
+      - description: b side power domain
+
+  power-domain-names:
+    items:
+      - const: processor a-facing
+      - const: processor b-facing
+
+  interrupt-controller: true
+
+  msi-controller: true
+
+  "#msi-cells":
+    const: 0
+
+required:
+  - compatible
+  - reg
+  - interrupts
+  - interrupt-controller
+  - msi-controller
+
+additionalProperties: false
+
+examples:
+  - |
+    #include <dt-bindings/interrupt-controller/arm-gic.h>
+    #include <dt-bindings/firmware/imx/rsrc.h>
+
+    msi-controller@5d270000 {
+        compatible = "fsl,imx6sx-mu-msi";
+        msi-controller;
+        #msi-cells = <0>;
+        interrupt-controller;
+        reg = <0x5d270000 0x10000>,     /* A side */
+              <0x5d300000 0x10000>;     /* B side */
+        reg-names = "processor a-facing", "processor b-facing";
+        interrupts = <GIC_SPI 191 IRQ_TYPE_LEVEL_HIGH>;
+        power-domains = <&pd IMX_SC_R_MU_12A>,
+                        <&pd IMX_SC_R_MU_12B>;
+        power-domain-names = "processor a-facing", "processor b-facing";
+    };
-- 
2.35.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v7 3/4] dt-bindings: irqchip: imx mu work as msi controller

[Rob Herring]

On Mon, Aug 22, 2022 at 10:51:29AM -0500, Frank Li wrote:
> I.MX mu support generate irq by write a register. Provide msi controller
> support so other driver such as PCI EP can use it by standard msi
> interface as doorbell.
> 
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---
>  .../interrupt-controller/fsl,mu-msi.yaml      | 98 +++++++++++++++++++
>  1 file changed, 98 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
> 
> diff --git a/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
> new file mode 100644
> index 0000000000000..ac07b138e24c0
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
> @@ -0,0 +1,98 @@
> +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +%YAML 1.2
> +---
> +$id: http://devicetree.org/schemas/interrupt-controller/fsl,mu-msi.yaml#
> +$schema: http://devicetree.org/meta-schemas/core.yaml#
> +
> +title: Freescale/NXP i.MX Messaging Unit (MU) work as msi controller
> +
> +maintainers:
> +  - Frank Li <Frank.Li@nxp.com>
> +
> +description: |
> +  The Messaging Unit module enables two processors within the SoC to
> +  communicate and coordinate by passing messages (e.g. data, status
> +  and control) through the MU interface. The MU also provides the ability
> +  for one processor (A side) to signal the other processor (B side) using
> +  interrupts.
> +
> +  Because the MU manages the messaging between processors, the MU uses
> +  different clocks (from each side of the different peripheral buses).
> +  Therefore, the MU must synchronize the accesses from one side to the
> +  other. The MU accomplishes synchronization using two sets of matching
> +  registers (Processor A-facing, Processor B-facing).
> +
> +  MU can work as msi interrupt controller to do doorbell
> +
> +allOf:
> +  - $ref: /schemas/interrupt-controller/msi-controller.yaml#
> +
> +properties:
> +  compatible:
> +    enum:
> +      - fsl,imx6sx-mu-msi
> +      - fsl,imx7ulp-mu-msi
> +      - fsl,imx8ulp-mu-msi
> +      - fsl,imx8ulp-mu-msi-s4
> +
> +  reg:
> +    items:
> +      - description: a side register base address
> +      - description: b side register base address
> +
> +  reg-names:
> +    items:
> +      - const: processor a-facing
> +      - const: processor b-facing

Isn't 'a' and 'b' sufficient to distinguish? Personally, doesn't really 
look like a case that benefits from -names at all.

In any case, -names shouldn't have spaces.

> +
> +  interrupts:
> +    description: a side interrupt number.
> +    maxItems: 1
> +
> +  clocks:
> +    maxItems: 1
> +
> +  power-domains:
> +    items:
> +      - description: a side power domain
> +      - description: b side power domain
> +
> +  power-domain-names:
> +    items:
> +      - const: processor a-facing
> +      - const: processor b-facing

Same here.

> +
> +  interrupt-controller: true
> +
> +  msi-controller: true
> +
> +  "#msi-cells":
> +    const: 0
> +
> +required:
> +  - compatible
> +  - reg
> +  - interrupts
> +  - interrupt-controller
> +  - msi-controller

#msi-cells should be required.

> +
> +additionalProperties: false
> +
> +examples:
> +  - |
> +    #include <dt-bindings/interrupt-controller/arm-gic.h>
> +    #include <dt-bindings/firmware/imx/rsrc.h>
> +
> +    msi-controller@5d270000 {
> +        compatible = "fsl,imx6sx-mu-msi";
> +        msi-controller;
> +        #msi-cells = <0>;
> +        interrupt-controller;
> +        reg = <0x5d270000 0x10000>,     /* A side */
> +              <0x5d300000 0x10000>;     /* B side */
> +        reg-names = "processor a-facing", "processor b-facing";
> +        interrupts = <GIC_SPI 191 IRQ_TYPE_LEVEL_HIGH>;
> +        power-domains = <&pd IMX_SC_R_MU_12A>,
> +                        <&pd IMX_SC_R_MU_12B>;
> +        power-domain-names = "processor a-facing", "processor b-facing";
> +    };
> -- 
> 2.35.1
> 
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [EXT] Re: [PATCH v7 3/4] dt-bindings: irqchip: imx mu work as msi controller

[Frank Li]

> -----Original Message-----
> From: Rob Herring <robh@kernel.org>
> Sent: Thursday, August 25, 2022 4:22 PM
> To: Frank Li <frank.li@nxp.com>
> Cc: maz@kernel.org; tglx@linutronix.de; krzysztof.kozlowski+dt@linaro.org;
> shawnguo@kernel.org; s.hauer@pengutronix.de; kw@linux.com;
> bhelgaas@google.com; linux-kernel@vger.kernel.org;
> devicetree@vger.kernel.org; linux-arm-kernel@lists.infradead.org; linux-
> pci@vger.kernel.org; Peng Fan <peng.fan@nxp.com>; Aisheng Dong
> <aisheng.dong@nxp.com>; jdmason@kudzu.us; kernel@pengutronix.de;
> festevam@gmail.com; dl-linux-imx <linux-imx@nxp.com>; kishon@ti.com;
> lorenzo.pieralisi@arm.com; ntb@lists.linux.dev; lznuaa@gmail.com
> Subject: [EXT] Re: [PATCH v7 3/4] dt-bindings: irqchip: imx mu work as msi
> controller
> 
> Caution: EXT Email
> 
> On Mon, Aug 22, 2022 at 10:51:29AM -0500, Frank Li wrote:
> > I.MX mu support generate irq by write a register. Provide msi controller
> > support so other driver such as PCI EP can use it by standard msi
> > interface as doorbell.
> >
> > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > ---
> >  .../interrupt-controller/fsl,mu-msi.yaml      | 98 +++++++++++++++++++
> >  1 file changed, 98 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/interrupt-
> controller/fsl,mu-msi.yaml
> >
> > diff --git a/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> msi.yaml b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> msi.yaml
> > new file mode 100644
> > index 0000000000000..ac07b138e24c0
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> msi.yaml
> > @@ -0,0 +1,98 @@
> > +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> > +%YAML 1.2
> > +---
> > +$id:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
> ree.org%2Fschemas%2Finterrupt-controller%2Ffsl%2Cmu-
> msi.yaml%23&amp;data=05%7C01%7CFrank.Li%40nxp.com%7Cbff8f186128d
> 44209f4108da86dfc975%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0
> %7C637970592959950791%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%
> 7C%7C&amp;sdata=DHCOhmaJAhwb8Gl%2FEbPj32B6lR2zcIvyMY%2BTuPACb
> zI%3D&amp;reserved=0
> > +$schema:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
> ree.org%2Fmeta-
> schemas%2Fcore.yaml%23&amp;data=05%7C01%7CFrank.Li%40nxp.com%7
> Cbff8f186128d44209f4108da86dfc975%7C686ea1d3bc2b4c6fa92cd99c5c3016
> 35%7C0%7C0%7C637970592959950791%7CUnknown%7CTWFpbGZsb3d8eyJ
> WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> 7C3000%7C%7C%7C&amp;sdata=J4znEXyHnMyQOssSUsoxE2Mlhe2qCDC%2F
> 9WN6SKv69aM%3D&amp;reserved=0
> > +
> > +title: Freescale/NXP i.MX Messaging Unit (MU) work as msi controller
> > +
> > +maintainers:
> > +  - Frank Li <Frank.Li@nxp.com>
> > +
> > +description: |
> > +  The Messaging Unit module enables two processors within the SoC to
> > +  communicate and coordinate by passing messages (e.g. data, status
> > +  and control) through the MU interface. The MU also provides the ability
> > +  for one processor (A side) to signal the other processor (B side) using
> > +  interrupts.
> > +
> > +  Because the MU manages the messaging between processors, the MU
> uses
> > +  different clocks (from each side of the different peripheral buses).
> > +  Therefore, the MU must synchronize the accesses from one side to the
> > +  other. The MU accomplishes synchronization using two sets of matching
> > +  registers (Processor A-facing, Processor B-facing).
> > +
> > +  MU can work as msi interrupt controller to do doorbell
> > +
> > +allOf:
> > +  - $ref: /schemas/interrupt-controller/msi-controller.yaml#
> > +
> > +properties:
> > +  compatible:
> > +    enum:
> > +      - fsl,imx6sx-mu-msi
> > +      - fsl,imx7ulp-mu-msi
> > +      - fsl,imx8ulp-mu-msi
> > +      - fsl,imx8ulp-mu-msi-s4
> > +
> > +  reg:
> > +    items:
> > +      - description: a side register base address
> > +      - description: b side register base address
> > +
> > +  reg-names:
> > +    items:
> > +      - const: processor a-facing
> > +      - const: processor b-facing
> 
> Isn't 'a' and 'b' sufficient to distinguish? Personally, doesn't really
> look like a case that benefits from -names at all.
> 
> In any case, -names shouldn't have spaces.

I like "a" and "b".

But Marc Zyngier suggested use above name.
https://www.spinics.net/lists/linux-pci/msg128783.html

@Marc Zyngier

best regards
Frank Li

> 
> > +
> > +  interrupts:
> > +    description: a side interrupt number.
> > +    maxItems: 1
> > +
> > +  clocks:
> > +    maxItems: 1
> > +
> > +  power-domains:
> > +    items:
> > +      - description: a side power domain
> > +      - description: b side power domain
> > +
> > +  power-domain-names:
> > +    items:
> > +      - const: processor a-facing
> > +      - const: processor b-facing
> 
> Same here.
> 
> > +
> > +  interrupt-controller: true
> > +
> > +  msi-controller: true
> > +
> > +  "#msi-cells":
> > +    const: 0
> > +
> > +required:
> > +  - compatible
> > +  - reg
> > +  - interrupts
> > +  - interrupt-controller
> > +  - msi-controller
> 
> #msi-cells should be required.
> 
> > +
> > +additionalProperties: false
> > +
> > +examples:
> > +  - |
> > +    #include <dt-bindings/interrupt-controller/arm-gic.h>
> > +    #include <dt-bindings/firmware/imx/rsrc.h>
> > +
> > +    msi-controller@5d270000 {
> > +        compatible = "fsl,imx6sx-mu-msi";
> > +        msi-controller;
> > +        #msi-cells = <0>;
> > +        interrupt-controller;
> > +        reg = <0x5d270000 0x10000>,     /* A side */
> > +              <0x5d300000 0x10000>;     /* B side */
> > +        reg-names = "processor a-facing", "processor b-facing";
> > +        interrupts = <GIC_SPI 191 IRQ_TYPE_LEVEL_HIGH>;
> > +        power-domains = <&pd IMX_SC_R_MU_12A>,
> > +                        <&pd IMX_SC_R_MU_12B>;
> > +        power-domain-names = "processor a-facing", "processor b-facing";
> > +    };
> > --
> > 2.35.1
> >
> >

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [EXT] Re: [PATCH v7 3/4] dt-bindings: irqchip: imx mu work as msi controller

[Marc Zyngier]

On Thu, 25 Aug 2022 22:42:38 +0100,
Frank Li <frank.li@nxp.com> wrote:
> 
> 
> 
> > -----Original Message-----
> > From: Rob Herring <robh@kernel.org>
> > Sent: Thursday, August 25, 2022 4:22 PM
> > To: Frank Li <frank.li@nxp.com>
> > Cc: maz@kernel.org; tglx@linutronix.de; krzysztof.kozlowski+dt@linaro.org;
> > shawnguo@kernel.org; s.hauer@pengutronix.de; kw@linux.com;
> > bhelgaas@google.com; linux-kernel@vger.kernel.org;
> > devicetree@vger.kernel.org; linux-arm-kernel@lists.infradead.org; linux-
> > pci@vger.kernel.org; Peng Fan <peng.fan@nxp.com>; Aisheng Dong
> > <aisheng.dong@nxp.com>; jdmason@kudzu.us; kernel@pengutronix.de;
> > festevam@gmail.com; dl-linux-imx <linux-imx@nxp.com>; kishon@ti.com;
> > lorenzo.pieralisi@arm.com; ntb@lists.linux.dev; lznuaa@gmail.com
> > Subject: [EXT] Re: [PATCH v7 3/4] dt-bindings: irqchip: imx mu work as msi
> > controller
> > 
> > Caution: EXT Email
> > 
> > On Mon, Aug 22, 2022 at 10:51:29AM -0500, Frank Li wrote:
> > > I.MX mu support generate irq by write a register. Provide msi controller
> > > support so other driver such as PCI EP can use it by standard msi
> > > interface as doorbell.
> > >
> > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > ---
> > >  .../interrupt-controller/fsl,mu-msi.yaml      | 98 +++++++++++++++++++
> > >  1 file changed, 98 insertions(+)
> > >  create mode 100644 Documentation/devicetree/bindings/interrupt-
> > controller/fsl,mu-msi.yaml
> > >
> > > diff --git a/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> > msi.yaml b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> > msi.yaml
> > > new file mode 100644
> > > index 0000000000000..ac07b138e24c0
> > > --- /dev/null
> > > +++ b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> > msi.yaml
> > > @@ -0,0 +1,98 @@
> > > +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> > > +%YAML 1.2
> > > +---
> > > +$id:
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
> > ree.org%2Fschemas%2Finterrupt-controller%2Ffsl%2Cmu-
> > msi.yaml%23&amp;data=05%7C01%7CFrank.Li%40nxp.com%7Cbff8f186128d
> > 44209f4108da86dfc975%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0
> > %7C637970592959950791%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
> > AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%
> > 7C%7C&amp;sdata=DHCOhmaJAhwb8Gl%2FEbPj32B6lR2zcIvyMY%2BTuPACb
> > zI%3D&amp;reserved=0
> > > +$schema:
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
> > ree.org%2Fmeta-
> > schemas%2Fcore.yaml%23&amp;data=05%7C01%7CFrank.Li%40nxp.com%7
> > Cbff8f186128d44209f4108da86dfc975%7C686ea1d3bc2b4c6fa92cd99c5c3016
> > 35%7C0%7C0%7C637970592959950791%7CUnknown%7CTWFpbGZsb3d8eyJ
> > WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> > 7C3000%7C%7C%7C&amp;sdata=J4znEXyHnMyQOssSUsoxE2Mlhe2qCDC%2F
> > 9WN6SKv69aM%3D&amp;reserved=0
> > > +
> > > +title: Freescale/NXP i.MX Messaging Unit (MU) work as msi controller
> > > +
> > > +maintainers:
> > > +  - Frank Li <Frank.Li@nxp.com>
> > > +
> > > +description: |
> > > +  The Messaging Unit module enables two processors within the SoC to
> > > +  communicate and coordinate by passing messages (e.g. data, status
> > > +  and control) through the MU interface. The MU also provides the ability
> > > +  for one processor (A side) to signal the other processor (B side) using
> > > +  interrupts.
> > > +
> > > +  Because the MU manages the messaging between processors, the MU
> > uses
> > > +  different clocks (from each side of the different peripheral buses).
> > > +  Therefore, the MU must synchronize the accesses from one side to the
> > > +  other. The MU accomplishes synchronization using two sets of matching
> > > +  registers (Processor A-facing, Processor B-facing).
> > > +
> > > +  MU can work as msi interrupt controller to do doorbell
> > > +
> > > +allOf:
> > > +  - $ref: /schemas/interrupt-controller/msi-controller.yaml#
> > > +
> > > +properties:
> > > +  compatible:
> > > +    enum:
> > > +      - fsl,imx6sx-mu-msi
> > > +      - fsl,imx7ulp-mu-msi
> > > +      - fsl,imx8ulp-mu-msi
> > > +      - fsl,imx8ulp-mu-msi-s4
> > > +
> > > +  reg:
> > > +    items:
> > > +      - description: a side register base address
> > > +      - description: b side register base address
> > > +
> > > +  reg-names:
> > > +    items:
> > > +      - const: processor a-facing
> > > +      - const: processor b-facing
> > 
> > Isn't 'a' and 'b' sufficient to distinguish? Personally, doesn't really
> > look like a case that benefits from -names at all.
> > 
> > In any case, -names shouldn't have spaces.
> 
> I like "a" and "b".
> 
> But Marc Zyngier suggested use above name.
> https://www.spinics.net/lists/linux-pci/msg128783.html
> 
> @Marc Zyngier

And I stand by my initial request. "a" doesn't convey any sort of
useful information. Why not "I" and "II", while we're at it? Or
something even funkier?

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [EXT] Re: [PATCH v7 3/4] dt-bindings: irqchip: imx mu work as msi controller

[Frank Li]

> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Friday, August 26, 2022 1:35 PM
> To: Frank Li <frank.li@nxp.com>
> Cc: Rob Herring <robh@kernel.org>; tglx@linutronix.de;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com; linux-
> kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> jdmason@kudzu.us; kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx
> <linux-imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev; lznuaa@gmail.com
> Subject: Re: [EXT] Re: [PATCH v7 3/4] dt-bindings: irqchip: imx mu work as msi
> controller
> 
> Caution: EXT Email
> 
> On Thu, 25 Aug 2022 22:42:38 +0100,
> Frank Li <frank.li@nxp.com> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Rob Herring <robh@kernel.org>
> > > Sent: Thursday, August 25, 2022 4:22 PM
> > > To: Frank Li <frank.li@nxp.com>
> > > Cc: maz@kernel.org; tglx@linutronix.de; krzysztof.kozlowski+dt@linaro.org;
> > > shawnguo@kernel.org; s.hauer@pengutronix.de; kw@linux.com;
> > > bhelgaas@google.com; linux-kernel@vger.kernel.org;
> > > devicetree@vger.kernel.org; linux-arm-kernel@lists.infradead.org; linux-
> > > pci@vger.kernel.org; Peng Fan <peng.fan@nxp.com>; Aisheng Dong
> > > <aisheng.dong@nxp.com>; jdmason@kudzu.us; kernel@pengutronix.de;
> > > festevam@gmail.com; dl-linux-imx <linux-imx@nxp.com>; kishon@ti.com;
> > > lorenzo.pieralisi@arm.com; ntb@lists.linux.dev; lznuaa@gmail.com
> > > Subject: [EXT] Re: [PATCH v7 3/4] dt-bindings: irqchip: imx mu work as msi
> > > controller
> > >
> > > Caution: EXT Email
> > >
> > > On Mon, Aug 22, 2022 at 10:51:29AM -0500, Frank Li wrote:
> > > > I.MX mu support generate irq by write a register. Provide msi controller
> > > > support so other driver such as PCI EP can use it by standard msi
> > > > interface as doorbell.
> > > >
> > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > ---
> > > >  .../interrupt-controller/fsl,mu-msi.yaml      | 98 +++++++++++++++++++
> > > >  1 file changed, 98 insertions(+)
> > > >  create mode 100644 Documentation/devicetree/bindings/interrupt-
> > > controller/fsl,mu-msi.yaml
> > > >
> > > > diff --git a/Documentation/devicetree/bindings/interrupt-
> controller/fsl,mu-
> > > msi.yaml b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> > > msi.yaml
> > > > new file mode 100644
> > > > index 0000000000000..ac07b138e24c0
> > > > --- /dev/null
> > > > +++ b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> > > msi.yaml
> > > > @@ -0,0 +1,98 @@
> > > > +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> > > > +%YAML 1.2
> > > > +---
> > > > +$id:
> > >
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
> > > ree.org%2Fschemas%2Finterrupt-controller%2Ffsl%2Cmu-
> > > msi.yaml%23&amp;data=05%7C01%7CFrank.Li%40nxp.com%7Cbff8f186128d
> > > 44209f4108da86dfc975%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0
> > > %7C637970592959950791%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
> > > AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%
> > > 7C%7C&amp;sdata=DHCOhmaJAhwb8Gl%2FEbPj32B6lR2zcIvyMY%2BTuPACb
> > > zI%3D&amp;reserved=0
> > > > +$schema:
> > >
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
> > > ree.org%2Fmeta-
> > > schemas%2Fcore.yaml%23&amp;data=05%7C01%7CFrank.Li%40nxp.com%7
> > > Cbff8f186128d44209f4108da86dfc975%7C686ea1d3bc2b4c6fa92cd99c5c3016
> > > 35%7C0%7C0%7C637970592959950791%7CUnknown%7CTWFpbGZsb3d8eyJ
> > > WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> > > 7C3000%7C%7C%7C&amp;sdata=J4znEXyHnMyQOssSUsoxE2Mlhe2qCDC%2F
> > > 9WN6SKv69aM%3D&amp;reserved=0
> > > > +
> > > > +title: Freescale/NXP i.MX Messaging Unit (MU) work as msi controller
> > > > +
> > > > +maintainers:
> > > > +  - Frank Li <Frank.Li@nxp.com>
> > > > +
> > > > +description: |
> > > > +  The Messaging Unit module enables two processors within the SoC to
> > > > +  communicate and coordinate by passing messages (e.g. data, status
> > > > +  and control) through the MU interface. The MU also provides the ability
> > > > +  for one processor (A side) to signal the other processor (B side) using
> > > > +  interrupts.
> > > > +
> > > > +  Because the MU manages the messaging between processors, the MU
> > > uses
> > > > +  different clocks (from each side of the different peripheral buses).
> > > > +  Therefore, the MU must synchronize the accesses from one side to the
> > > > +  other. The MU accomplishes synchronization using two sets of matching
> > > > +  registers (Processor A-facing, Processor B-facing).
> > > > +
> > > > +  MU can work as msi interrupt controller to do doorbell
> > > > +
> > > > +allOf:
> > > > +  - $ref: /schemas/interrupt-controller/msi-controller.yaml#
> > > > +
> > > > +properties:
> > > > +  compatible:
> > > > +    enum:
> > > > +      - fsl,imx6sx-mu-msi
> > > > +      - fsl,imx7ulp-mu-msi
> > > > +      - fsl,imx8ulp-mu-msi
> > > > +      - fsl,imx8ulp-mu-msi-s4
> > > > +
> > > > +  reg:
> > > > +    items:
> > > > +      - description: a side register base address
> > > > +      - description: b side register base address
> > > > +
> > > > +  reg-names:
> > > > +    items:
> > > > +      - const: processor a-facing
> > > > +      - const: processor b-facing
> > >
> > > Isn't 'a' and 'b' sufficient to distinguish? Personally, doesn't really
> > > look like a case that benefits from -names at all.
> > >
> > > In any case, -names shouldn't have spaces.
> >
> > I like "a" and "b".
> >
> > But Marc Zyngier suggested use above name.
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.spi
> nics.net%2Flists%2Flinux-
> pci%2Fmsg128783.html&amp;data=05%7C01%7Cfrank.li%40nxp.com%7Cadd154d
> 4aeda4059c93408da8791ba1b%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0
> %7C637971357205475355%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwM
> DAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&a
> mp;sdata=vuaoWvu8BYcJ5NjOoUfFhlykBsW8vC2%2FbsrBmfx%2Bfz8%3D&amp;r
> eserved=0
> >
> > @Marc Zyngier
> 
> And I stand by my initial request. "a" doesn't convey any sort of
> useful information. Why not "I" and "II", while we're at it? Or
> something even funkier?

MU spec use term "a" and "b",  user have to map "I" an "II" to 
"a" and "b" when read MU spec and code. it is not straightforward.

I quote a part of spec. 
" The MU is connected as a peripheral under the Peripheral bus on both sides-on
the Processor A-side, the Processor A Peripheral Bus, and on the Processor B side,
the Processor B Peripheral Bus."

Rob Herring and Marc Zynginer:
I can change to any name, which you agree both. 

Some options:
1. "a", "b"
2. "a-side", "b-side"
3. "a-facing", "b-facing"
4. "I", "II"

> 
>         M.
> 
> --
> Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [EXT] Re: [PATCH v7 3/4] dt-bindings: irqchip: imx mu work as msi controller

[Marc Zyngier]

On Fri, 26 Aug 2022 19:59:44 +0100,
Frank Li <frank.li@nxp.com> wrote:
> 
> > And I stand by my initial request. "a" doesn't convey any sort of
> > useful information. Why not "I" and "II", while we're at it? Or
> > something even funkier?
> 
> MU spec use term "a" and "b",  user have to map "I" an "II" to 
> "a" and "b" when read MU spec and code. it is not straightforward.
> 
> I quote a part of spec. 
> " The MU is connected as a peripheral under the Peripheral bus on both sides-on
> the Processor A-side, the Processor A Peripheral Bus, and on the Processor B side,
> the Processor B Peripheral Bus."
> 
> Rob Herring and Marc Zynginer:
> I can change to any name, which you agree both. 
> 
> Some options:
> 1. "a", "b"
> 2. "a-side", "b-side"
> 3. "a-facing", "b-facing"
> 4. "I", "II"

Use the wording indicated in the spec: "processor-a-side", and
"processor-b-side". This is what I asked the first place.

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [EXT] Re: [PATCH v7 3/4] dt-bindings: irqchip: imx mu work as msi controller

[Frank Li]

> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Friday, August 26, 2022 4:45 PM
> To: Frank Li <frank.li@nxp.com>
> Cc: Rob Herring <robh@kernel.org>; tglx@linutronix.de;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com; linux-
> kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> jdmason@kudzu.us; kernel@pengutronix.de; festevam@gmail.com; dl-linux-
> imx <linux-imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev; lznuaa@gmail.com
> Subject: Re: [EXT] Re: [PATCH v7 3/4] dt-bindings: irqchip: imx mu work as
> msi controller
> 
> Caution: EXT Email
> 
> On Fri, 26 Aug 2022 19:59:44 +0100,
> Frank Li <frank.li@nxp.com> wrote:
> >
> > > And I stand by my initial request. "a" doesn't convey any sort of
> > > useful information. Why not "I" and "II", while we're at it? Or
> > > something even funkier?
> >
> > MU spec use term "a" and "b",  user have to map "I" an "II" to
> > "a" and "b" when read MU spec and code. it is not straightforward.
> >
> > I quote a part of spec.
> > " The MU is connected as a peripheral under the Peripheral bus on both
> sides-on
> > the Processor A-side, the Processor A Peripheral Bus, and on the Processor
> B side,
> > the Processor B Peripheral Bus."
> >
> > Rob Herring and Marc Zynginer:
> > I can change to any name, which you agree both.
> >
> > Some options:
> > 1. "a", "b"
> > 2. "a-side", "b-side"
> > 3. "a-facing", "b-facing"
> > 4. "I", "II"
> 
> Use the wording indicated in the spec: "processor-a-side", and
> "processor-b-side". This is what I asked the first place.

@Rob Herring:  Do you agree this name?

[Frank Li] 

> 
>         M.
> 
> --
> Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [EXT] Re: [PATCH v7 3/4] dt-bindings: irqchip: imx mu work as msi controller

[Frank Li]

> -----Original Message-----
> From: Frank Li <frank.li@nxp.com>
> Sent: Monday, August 29, 2022 9:48 AM
> To: Marc Zyngier <maz@kernel.org>; Rob Herring <robh@kernel.org>
> Cc: tglx@linutronix.de; krzysztof.kozlowski+dt@linaro.org;
> shawnguo@kernel.org; s.hauer@pengutronix.de; kw@linux.com;
> bhelgaas@google.com; linux-kernel@vger.kernel.org;
> devicetree@vger.kernel.org; linux-arm-kernel@lists.infradead.org; linux-
> pci@vger.kernel.org; Peng Fan <peng.fan@nxp.com>; Aisheng Dong
> <aisheng.dong@nxp.com>; jdmason@kudzu.us; kernel@pengutronix.de;
> festevam@gmail.com; dl-linux-imx <linux-imx@nxp.com>; kishon@ti.com;
> lorenzo.pieralisi@arm.com; ntb@lists.linux.dev; lznuaa@gmail.com
> Subject: RE: [EXT] Re: [PATCH v7 3/4] dt-bindings: irqchip: imx mu work as
> msi controller
> 
> Caution: EXT Email
> 
> > -----Original Message-----
> > From: Marc Zyngier <maz@kernel.org>
> > Sent: Friday, August 26, 2022 4:45 PM
> > To: Frank Li <frank.li@nxp.com>
> > Cc: Rob Herring <robh@kernel.org>; tglx@linutronix.de;
> > krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> > s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com; linux-
> > kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> > kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> > <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> > jdmason@kudzu.us; kernel@pengutronix.de; festevam@gmail.com; dl-
> linux-
> > imx <linux-imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > ntb@lists.linux.dev; lznuaa@gmail.com
> > Subject: Re: [EXT] Re: [PATCH v7 3/4] dt-bindings: irqchip: imx mu work as
> > msi controller
> >
> > Caution: EXT Email
> >
> > On Fri, 26 Aug 2022 19:59:44 +0100,
> > Frank Li <frank.li@nxp.com> wrote:
> > >
> > > > And I stand by my initial request. "a" doesn't convey any sort of
> > > > useful information. Why not "I" and "II", while we're at it? Or
> > > > something even funkier?
> > >
> > > MU spec use term "a" and "b",  user have to map "I" an "II" to
> > > "a" and "b" when read MU spec and code. it is not straightforward.
> > >
> > > I quote a part of spec.
> > > " The MU is connected as a peripheral under the Peripheral bus on both
> > sides-on
> > > the Processor A-side, the Processor A Peripheral Bus, and on the
> Processor
> > B side,
> > > the Processor B Peripheral Bus."
> > >
> > > Rob Herring and Marc Zynginer:
> > > I can change to any name, which you agree both.
> > >
> > > Some options:
> > > 1. "a", "b"
> > > 2. "a-side", "b-side"
> > > 3. "a-facing", "b-facing"
> > > 4. "I", "II"
> >
> > Use the wording indicated in the spec: "processor-a-side", and
> > "processor-b-side". This is what I asked the first place.
> 
> @Rob Herring:  Do you agree this name?

@Rob Herring:  How about "process-a-side"? 
If you agree,  I will resin these patches soon.

Best regards
Frank Li

> 
> [Frank Li]
> 
> >
> >         M.
> >
> > --
> > Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [EXT] Re: [PATCH v7 3/4] dt-bindings: irqchip: imx mu work as msi controller

[Rob Herring]

On Fri, Aug 26, 2022 at 4:44 PM Marc Zyngier <maz@kernel.org> wrote:
>
> On Fri, 26 Aug 2022 19:59:44 +0100,
> Frank Li <frank.li@nxp.com> wrote:
> >
> > > And I stand by my initial request. "a" doesn't convey any sort of
> > > useful information. Why not "I" and "II", while we're at it? Or
> > > something even funkier?
> >
> > MU spec use term "a" and "b",  user have to map "I" an "II" to
> > "a" and "b" when read MU spec and code. it is not straightforward.
> >
> > I quote a part of spec.
> > " The MU is connected as a peripheral under the Peripheral bus on both sides-on
> > the Processor A-side, the Processor A Peripheral Bus, and on the Processor B side,
> > the Processor B Peripheral Bus."
> >
> > Rob Herring and Marc Zynginer:
> > I can change to any name, which you agree both.
> >
> > Some options:
> > 1. "a", "b"
> > 2. "a-side", "b-side"
> > 3. "a-facing", "b-facing"
> > 4. "I", "II"
>
> Use the wording indicated in the spec: "processor-a-side", and
> "processor-b-side". This is what I asked the first place.

I would pick 2 (or nothing), but whatever... As long as there aren't spaces.

Rob

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v7 4/4] PCI: endpoint: Add NTB MSI support

[Frank Li]

                        ┌───────┐           ┌──────────┐
                        │       │           │          │
      ┌─────────────┐   │       │           │ PCI Host │
      │ MSI         │◄┐ │       │           │          │
      │ Controller  │ │ │       │ MSI Write │          │
      └─────────────┘ └─┼───────┼───────────┼─BAR0     │
                        │ PCI   │           │ BAR1     │
                        │ EP    │           │ BAR2     │
                        │       │           │ BAR3     │
                        │       │           │ BAR4     │
                        │       ├──────────►│          │
                        └───────┘           └──────────┘

Linux supports endpoint functions. PCI RC write BAR<n> space like write
to memory. The EP side can't know memory changed by the host driver.

PCI Spec has not defined a standard method to do that. Only define MSI(x)
to let EP notified RC status change.

The basic idea is to trigger an IRQ when PCI RC writes to a memory
address. That's what MSI controller provided. EP drivers just need to
request a platform MSI interrupt, struct msi_msg *msg will pass down a
memory address and data. EP driver will map such memory address to one of
PCI BAR<n>.  PCI RC writes to such memory to trigger EP side IRQ.

Add MSI support for pci-epf-vntb. Query if system has an MSI controller.
Set up doorbell address according to struct msi_msg.

So PCI RC can write this doorbell address to trigger EP side's IRQ.

If no MSI controller exists, fall back to software polling.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
 drivers/pci/endpoint/functions/pci-epf-vntb.c | 148 +++++++++++++++---
 1 file changed, 122 insertions(+), 26 deletions(-)

diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
index 1466dd1904175..f6e0b682da000 100644
--- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
@@ -44,6 +44,7 @@
 #include <linux/pci-epc.h>
 #include <linux/pci-epf.h>
 #include <linux/ntb.h>
+#include <linux/msi.h>
 
 static struct workqueue_struct *kpcintb_workqueue;
 
@@ -143,6 +144,8 @@ struct epf_ntb {
 	void __iomem *vpci_mw_addr[MAX_MW];
 
 	struct delayed_work cmd_handler;
+
+	int msi_virqbase;
 };
 
 #define to_epf_ntb(epf_group) container_of((epf_group), struct epf_ntb, group)
@@ -253,13 +256,15 @@ static void epf_ntb_cmd_handler(struct work_struct *work)
 
 	ntb = container_of(work, struct epf_ntb, cmd_handler.work);
 
-	for (i = 1; i < ntb->db_count; i++) {
-		if (readl(ntb->epf_db + i * 4)) {
-			if (readl(ntb->epf_db + i * 4))
-				ntb->db |= 1 << (i - 1);
+	if (!ntb->epf_db_phy) {
+		for (i = 1; i < ntb->db_count; i++) {
+			if (readl(ntb->epf_db + i * 4)) {
+				if (readl(ntb->epf_db + i * 4))
+					ntb->db |= 1 << (i - 1);
 
-			ntb_db_event(&ntb->ntb, i);
-			writel(0, ntb->epf_db + i * 4);
+				ntb_db_event(&ntb->ntb, i);
+				writel(0, ntb->epf_db + i * 4);
+			}
 		}
 	}
 
@@ -454,11 +459,9 @@ static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb)
 	ctrl->num_mws = ntb->num_mws;
 	ntb->spad_size = spad_size;
 
-	ctrl->db_entry_size = 4;
-
 	for (i = 0; i < ntb->db_count; i++) {
 		ntb->reg->db_data[i] = 1 + i;
-		ntb->reg->db_offset[i] = 0;
+		ntb->reg->db_offset[i] = 4 * i;
 	}
 
 	return 0;
@@ -509,6 +512,28 @@ static int epf_ntb_configure_interrupt(struct epf_ntb *ntb)
 	return 0;
 }
 
+static int epf_ntb_db_size(struct epf_ntb *ntb)
+{
+	const struct pci_epc_features *epc_features;
+	size_t size = 4 * ntb->db_count;
+	u32 align;
+
+	epc_features = pci_epc_get_features(ntb->epf->epc,
+					    ntb->epf->func_no,
+					    ntb->epf->vfunc_no);
+	align = epc_features->align;
+
+	if (size < 128)
+		size = 128;
+
+	if (align)
+		size = ALIGN(size, align);
+	else
+		size = roundup_pow_of_two(size);
+
+	return size;
+}
+
 /**
  * epf_ntb_db_bar_init() - Configure Doorbell window BARs
  * @ntb: NTB device that facilitates communication between HOST and vHOST
@@ -522,33 +547,32 @@ static int epf_ntb_db_bar_init(struct epf_ntb *ntb)
 	struct pci_epf_bar *epf_bar;
 	void __iomem *mw_addr;
 	enum pci_barno barno;
-	size_t size = 4 * ntb->db_count;
+	size_t size;
 
 	epc_features = pci_epc_get_features(ntb->epf->epc,
 					    ntb->epf->func_no,
 					    ntb->epf->vfunc_no);
 	align = epc_features->align;
-
-	if (size < 128)
-		size = 128;
-
-	if (align)
-		size = ALIGN(size, align);
-	else
-		size = roundup_pow_of_two(size);
+	size = epf_ntb_db_size(ntb);
 
 	barno = ntb->epf_ntb_bar[BAR_DB];
+	epf_bar = &ntb->epf->bar[barno];
 
-	mw_addr = pci_epf_alloc_space(ntb->epf, size, barno, align, 0);
-	if (!mw_addr) {
-		dev_err(dev, "Failed to allocate OB address\n");
-		return -ENOMEM;
+	if (ntb->epf_db_phy) {
+		mw_addr = NULL;
+		epf_bar->phys_addr = ntb->epf_db_phy;
+		epf_bar->barno = barno;
+		epf_bar->size = size;
+	} else {
+		mw_addr = pci_epf_alloc_space(ntb->epf, size, barno, align, 0);
+		if (!mw_addr) {
+			dev_err(dev, "Failed to allocate OB address\n");
+			return -ENOMEM;
+		}
 	}
 
 	ntb->epf_db = mw_addr;
 
-	epf_bar = &ntb->epf->bar[barno];
-
 	ret = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no, ntb->epf->vfunc_no, epf_bar);
 	if (ret) {
 		dev_err(dev, "Doorbell BAR set failed\n");
@@ -704,6 +728,77 @@ static int epf_ntb_init_epc_bar(struct epf_ntb *ntb)
 	return 0;
 }
 
+static void epf_ntb_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
+{
+	struct epf_ntb *ntb = dev_get_drvdata(desc->dev);
+	struct epf_ntb_ctrl *reg = ntb->reg;
+	int size = epf_ntb_db_size(ntb);
+	u64 addr;
+
+	addr = msg->address_hi;
+	addr <<= 32;
+	addr |= msg->address_lo;
+
+	reg->db_data[desc->msi_index] = msg->data;
+
+	if (desc->msi_index == 0)
+		ntb->epf_db_phy = round_down(addr, size);
+
+	reg->db_offset[desc->msi_index] = addr - ntb->epf_db_phy;
+}
+
+static irqreturn_t epf_ntb_interrupt_handler(int irq, void *data)
+{
+	struct epf_ntb *ntb = data;
+	int index;
+
+	index = irq - ntb->msi_virqbase;
+	ntb->db |= 1 << (index - 1);
+	ntb_db_event(&ntb->ntb, index);
+
+	return IRQ_HANDLED;
+}
+
+static void epf_ntb_epc_msi_init(struct epf_ntb *ntb)
+{
+	struct device *dev = &ntb->epf->dev;
+	struct irq_domain *domain;
+	int virq;
+	int ret;
+	int i;
+
+	domain = dev_get_msi_domain(ntb->epf->epc->dev.parent);
+	if (!domain)
+		return;
+
+	dev_set_msi_domain(dev, domain);
+
+	if (platform_msi_domain_alloc_irqs(&ntb->epf->dev,
+		ntb->db_count,
+		epf_ntb_write_msi_msg)) {
+		dev_info(dev, "Can't allocate MSI, fall back to poll mode\n");
+		return;
+	}
+
+	dev_info(dev, "vntb use MSI as doorbell\n");
+
+	for (i = 0; i < ntb->db_count; i++) {
+		virq = msi_get_virq(dev, i);
+		ret = devm_request_irq(dev, virq,
+			       epf_ntb_interrupt_handler, 0,
+			       "ntb", ntb);
+
+		if (ret) {
+			dev_err(dev, "devm_request_irq() failure, fall back to poll mode\n");
+			ntb->epf_db_phy = 0;
+			break;
+		}
+
+		if (!i)
+			ntb->msi_virqbase = virq;
+	}
+}
+
 /**
  * epf_ntb_epc_init() - Initialize NTB interface
  * @ntb: NTB device that facilitates communication between HOST and vHOST2
@@ -1299,14 +1394,15 @@ static int epf_ntb_bind(struct pci_epf *epf)
 		goto err_bar_alloc;
 	}
 
+	epf_set_drvdata(epf, ntb);
+	epf_ntb_epc_msi_init(ntb);
+
 	ret = epf_ntb_epc_init(ntb);
 	if (ret) {
 		dev_err(dev, "Failed to initialize EPC\n");
 		goto err_bar_alloc;
 	}
 
-	epf_set_drvdata(epf, ntb);
-
 	pci_space[0] = (ntb->vntb_pid << 16) | ntb->vntb_vid;
 	pci_vntb_table[0].vendor = ntb->vntb_vid;
 	pci_vntb_table[0].device = ntb->vntb_pid;
-- 
2.35.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v7 4/4] PCI: endpoint: Add NTB MSI support

[kernel test robot]

Hi Frank,

I love your patch! Yet something to improve:

[auto build test ERROR on jonmason-ntb/ntb-next]
[also build test ERROR on robh/for-next linus/master v6.0-rc2 next-20220822]
[cannot apply to tip/irq/core]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Frank-Li/PCI-EP-driver-support-MSI-doorbell-from-host/20220822-235323
base:   https://github.com/jonmason/ntb ntb-next
config: microblaze-randconfig-r005-20220821 (https://download.01.org/0day-ci/archive/20220823/202208230543.fAaLXJy7-lkp@intel.com/config)
compiler: microblaze-linux-gcc (GCC) 12.1.0
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # https://github.com/intel-lab-lkp/linux/commit/d98704aefa5b57814d7b9b1b40160df34977b2b6
        git remote add linux-review https://github.com/intel-lab-lkp/linux
        git fetch --no-tags linux-review Frank-Li/PCI-EP-driver-support-MSI-doorbell-from-host/20220822-235323
        git checkout d98704aefa5b57814d7b9b1b40160df34977b2b6
        # save the config file
        mkdir build_dir && cp config build_dir/.config
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=microblaze SHELL=/bin/bash drivers/pci/endpoint/functions/

If you fix the issue, kindly add following tag where applicable
Reported-by: kernel test robot <lkp@intel.com>

All errors (new ones prefixed by >>):

   drivers/pci/endpoint/functions/pci-epf-vntb.c: In function 'epf_ntb_epc_msi_init':
>> drivers/pci/endpoint/functions/pci-epf-vntb.c:786:13: error: implicit declaration of function 'platform_msi_domain_alloc_irqs' [-Werror=implicit-function-declaration]
     786 |         if (platform_msi_domain_alloc_irqs(&ntb->epf->dev,
         |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   cc1: some warnings being treated as errors


vim +/platform_msi_domain_alloc_irqs +786 drivers/pci/endpoint/functions/pci-epf-vntb.c

   771	
   772	static void epf_ntb_epc_msi_init(struct epf_ntb *ntb)
   773	{
   774		struct device *dev = &ntb->epf->dev;
   775		struct irq_domain *domain;
   776		int virq;
   777		int ret;
   778		int i;
   779	
   780		domain = dev_get_msi_domain(ntb->epf->epc->dev.parent);
   781		if (!domain)
   782			return;
   783	
   784		dev_set_msi_domain(dev, domain);
   785	
 > 786		if (platform_msi_domain_alloc_irqs(&ntb->epf->dev,
   787			ntb->db_count,
   788			epf_ntb_write_msi_msg)) {
   789			dev_info(dev, "Can't allocate MSI, fall back to poll mode\n");
   790			return;
   791		}
   792	
   793		dev_info(dev, "vntb use MSI as doorbell\n");
   794	
   795		for (i = 0; i < ntb->db_count; i++) {
   796			virq = msi_get_virq(dev, i);
   797			ret = devm_request_irq(dev, virq,
   798				       epf_ntb_interrupt_handler, 0,
   799				       "ntb", ntb);
   800	
   801			if (ret) {
   802				dev_err(dev, "devm_request_irq() failure, fall back to poll mode\n");
   803				ntb->epf_db_phy = 0;
   804				break;
   805			}
   806	
   807			if (!i)
   808				ntb->msi_virqbase = virq;
   809		}
   810	}
   811	

-- 
0-DAY CI Kernel Test Service
https://01.org/lkp

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v7 4/4] PCI: endpoint: Add NTB MSI support

[kernel test robot]

Hi Frank,

I love your patch! Perhaps something to improve:

[auto build test WARNING on jonmason-ntb/ntb-next]
[also build test WARNING on robh/for-next linus/master v6.0-rc2 next-20220822]
[cannot apply to tip/irq/core]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Frank-Li/PCI-EP-driver-support-MSI-doorbell-from-host/20220822-235323
base:   https://github.com/jonmason/ntb ntb-next
config: s390-randconfig-s053-20220821 (https://download.01.org/0day-ci/archive/20220823/202208230844.D25Fw8sg-lkp@intel.com/config)
compiler: s390-linux-gcc (GCC) 12.1.0
reproduce:
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # apt-get install sparse
        # sparse version: v0.6.4-39-gce1a6720-dirty
        # https://github.com/intel-lab-lkp/linux/commit/d98704aefa5b57814d7b9b1b40160df34977b2b6
        git remote add linux-review https://github.com/intel-lab-lkp/linux
        git fetch --no-tags linux-review Frank-Li/PCI-EP-driver-support-MSI-doorbell-from-host/20220822-235323
        git checkout d98704aefa5b57814d7b9b1b40160df34977b2b6
        # save the config file
        mkdir build_dir && cp config build_dir/.config
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' O=build_dir ARCH=s390 SHELL=/bin/bash drivers/pci/endpoint/functions/

If you fix the issue, kindly add following tag where applicable
Reported-by: kernel test robot <lkp@intel.com>

sparse warnings: (new ones prefixed by >>)
>> drivers/pci/endpoint/functions/pci-epf-vntb.c:567:25: sparse: sparse: incorrect type in assignment (different address spaces) @@     expected void [noderef] __iomem *[assigned] mw_addr @@     got void * @@
   drivers/pci/endpoint/functions/pci-epf-vntb.c:567:25: sparse:     expected void [noderef] __iomem *[assigned] mw_addr
   drivers/pci/endpoint/functions/pci-epf-vntb.c:567:25: sparse:     got void *
   drivers/pci/endpoint/functions/pci-epf-vntb.c:600:41: sparse: sparse: incorrect type in argument 2 (different address spaces) @@     expected void *addr @@     got void [noderef] __iomem *epf_db @@
   drivers/pci/endpoint/functions/pci-epf-vntb.c:600:41: sparse:     expected void *addr
   drivers/pci/endpoint/functions/pci-epf-vntb.c:600:41: sparse:     got void [noderef] __iomem *epf_db
   drivers/pci/endpoint/functions/pci-epf-vntb.c:1201:33: sparse: sparse: incorrect type in initializer (different address spaces) @@     expected void [noderef] __iomem *base @@     got struct epf_ntb_ctrl *reg @@
   drivers/pci/endpoint/functions/pci-epf-vntb.c:1201:33: sparse:     expected void [noderef] __iomem *base
   drivers/pci/endpoint/functions/pci-epf-vntb.c:1201:33: sparse:     got struct epf_ntb_ctrl *reg
   drivers/pci/endpoint/functions/pci-epf-vntb.c:1212:33: sparse: sparse: incorrect type in initializer (different address spaces) @@     expected void [noderef] __iomem *base @@     got struct epf_ntb_ctrl *reg @@
   drivers/pci/endpoint/functions/pci-epf-vntb.c:1212:33: sparse:     expected void [noderef] __iomem *base
   drivers/pci/endpoint/functions/pci-epf-vntb.c:1212:33: sparse:     got struct epf_ntb_ctrl *reg
   drivers/pci/endpoint/functions/pci-epf-vntb.c:1223:33: sparse: sparse: incorrect type in initializer (different address spaces) @@     expected void [noderef] __iomem *base @@     got struct epf_ntb_ctrl *reg @@
   drivers/pci/endpoint/functions/pci-epf-vntb.c:1223:33: sparse:     expected void [noderef] __iomem *base
   drivers/pci/endpoint/functions/pci-epf-vntb.c:1223:33: sparse:     got struct epf_ntb_ctrl *reg
   drivers/pci/endpoint/functions/pci-epf-vntb.c:1235:33: sparse: sparse: incorrect type in initializer (different address spaces) @@     expected void [noderef] __iomem *base @@     got struct epf_ntb_ctrl *reg @@
   drivers/pci/endpoint/functions/pci-epf-vntb.c:1235:33: sparse:     expected void [noderef] __iomem *base
   drivers/pci/endpoint/functions/pci-epf-vntb.c:1235:33: sparse:     got struct epf_ntb_ctrl *reg

vim +567 drivers/pci/endpoint/functions/pci-epf-vntb.c

d98704aefa5b578 Frank Li 2022-08-22  536  
e35f56bb03304ab Frank Li 2022-02-22  537  /**
e35f56bb03304ab Frank Li 2022-02-22  538   * epf_ntb_db_bar_init() - Configure Doorbell window BARs
e35f56bb03304ab Frank Li 2022-02-22  539   * @ntb: NTB device that facilitates communication between HOST and vHOST
e35f56bb03304ab Frank Li 2022-02-22  540   */
e35f56bb03304ab Frank Li 2022-02-22  541  static int epf_ntb_db_bar_init(struct epf_ntb *ntb)
e35f56bb03304ab Frank Li 2022-02-22  542  {
e35f56bb03304ab Frank Li 2022-02-22  543  	const struct pci_epc_features *epc_features;
e35f56bb03304ab Frank Li 2022-02-22  544  	u32 align;
e35f56bb03304ab Frank Li 2022-02-22  545  	struct device *dev = &ntb->epf->dev;
e35f56bb03304ab Frank Li 2022-02-22  546  	int ret;
e35f56bb03304ab Frank Li 2022-02-22  547  	struct pci_epf_bar *epf_bar;
e35f56bb03304ab Frank Li 2022-02-22  548  	void __iomem *mw_addr;
e35f56bb03304ab Frank Li 2022-02-22  549  	enum pci_barno barno;
d98704aefa5b578 Frank Li 2022-08-22  550  	size_t size;
e35f56bb03304ab Frank Li 2022-02-22  551  
e35f56bb03304ab Frank Li 2022-02-22  552  	epc_features = pci_epc_get_features(ntb->epf->epc,
e35f56bb03304ab Frank Li 2022-02-22  553  					    ntb->epf->func_no,
e35f56bb03304ab Frank Li 2022-02-22  554  					    ntb->epf->vfunc_no);
e35f56bb03304ab Frank Li 2022-02-22  555  	align = epc_features->align;
d98704aefa5b578 Frank Li 2022-08-22  556  	size = epf_ntb_db_size(ntb);
e35f56bb03304ab Frank Li 2022-02-22  557  
e35f56bb03304ab Frank Li 2022-02-22  558  	barno = ntb->epf_ntb_bar[BAR_DB];
d98704aefa5b578 Frank Li 2022-08-22  559  	epf_bar = &ntb->epf->bar[barno];
e35f56bb03304ab Frank Li 2022-02-22  560  
d98704aefa5b578 Frank Li 2022-08-22  561  	if (ntb->epf_db_phy) {
d98704aefa5b578 Frank Li 2022-08-22  562  		mw_addr = NULL;
d98704aefa5b578 Frank Li 2022-08-22  563  		epf_bar->phys_addr = ntb->epf_db_phy;
d98704aefa5b578 Frank Li 2022-08-22  564  		epf_bar->barno = barno;
d98704aefa5b578 Frank Li 2022-08-22  565  		epf_bar->size = size;
d98704aefa5b578 Frank Li 2022-08-22  566  	} else {
e35f56bb03304ab Frank Li 2022-02-22 @567  		mw_addr = pci_epf_alloc_space(ntb->epf, size, barno, align, 0);
e35f56bb03304ab Frank Li 2022-02-22  568  		if (!mw_addr) {
e35f56bb03304ab Frank Li 2022-02-22  569  			dev_err(dev, "Failed to allocate OB address\n");
e35f56bb03304ab Frank Li 2022-02-22  570  			return -ENOMEM;
e35f56bb03304ab Frank Li 2022-02-22  571  		}
d98704aefa5b578 Frank Li 2022-08-22  572  	}
e35f56bb03304ab Frank Li 2022-02-22  573  
e35f56bb03304ab Frank Li 2022-02-22  574  	ntb->epf_db = mw_addr;
e35f56bb03304ab Frank Li 2022-02-22  575  
e35f56bb03304ab Frank Li 2022-02-22  576  	ret = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no, ntb->epf->vfunc_no, epf_bar);
e35f56bb03304ab Frank Li 2022-02-22  577  	if (ret) {
e35f56bb03304ab Frank Li 2022-02-22  578  		dev_err(dev, "Doorbell BAR set failed\n");
e35f56bb03304ab Frank Li 2022-02-22  579  			goto err_alloc_peer_mem;
e35f56bb03304ab Frank Li 2022-02-22  580  	}
e35f56bb03304ab Frank Li 2022-02-22  581  	return ret;
e35f56bb03304ab Frank Li 2022-02-22  582  
e35f56bb03304ab Frank Li 2022-02-22  583  err_alloc_peer_mem:
e35f56bb03304ab Frank Li 2022-02-22  584  	pci_epc_mem_free_addr(ntb->epf->epc, epf_bar->phys_addr, mw_addr, epf_bar->size);
e35f56bb03304ab Frank Li 2022-02-22  585  	return -1;
e35f56bb03304ab Frank Li 2022-02-22  586  }
e35f56bb03304ab Frank Li 2022-02-22  587  

-- 
0-DAY CI Kernel Test Service
https://01.org/lkp

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel