* [PATCH v3 0/4] PCI EP driver support MSI doorbell from host
@ 2022-07-20 21:30 ` Frank Li
0 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-20 21:30 UTC (permalink / raw)
To: jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, peng.fan,
aisheng.dong, kernel, festevam, linux-imx, kishon,
lorenzo.pieralisi, ntb
┌───────┐ ┌──────────┐
│ │ │ │
┌─────────────┐ │ │ │ PCI Host │
│ MSI │◄┐ │ │ │ │
│ Controller │ │ │ │ │ │
└─────────────┘ └─┼───────┼──────────┼─Bar0 │
│ PCI │ │ Bar1 │
│ Func │ │ Bar2 │
│ │ │ Bar3 │
│ │ │ Bar4 │
│ ├─────────►│ │
└───────┘ └──────────┘
Many PCI controllers provided Endpoint functions.
Generally PCI endpoint is hardware, which is not running a rich OS,
like linux.
But Linux also supports endpoint functions. PCI Host write BAR<n> space
like write to memory. The EP side can't know memory changed by the Host
driver.
PCI Spec has not defined a standard method to do that. Only define
MSI(x) to let EP notified RC status change.
The basic idea is to trigger an IRQ when PCI RC writes to a memory
address. That's what MSI controller provided. EP drivers just need to
request a platform MSI interrupt, struct MSI_msg *msg will pass down a
memory address and data. EP driver will map such memory address to
one of PCI BAR<n>. Host just writes such an address to trigger EP side
IRQ.
If system have gic-its, only need update PCI EP side driver. But i.MX
have not chip support gic-its yet. So we have to use MU to simulate a
MSI controller. Although only 4 MSI IRQs are simulated, it matched
vntb(pci-epf-vntb) network requirement.
After enable MSI, ping delay reduce < 1ms from ~8ms
IRQchip: imx mu worked as MSI controller:
let imx mu worked as MSI controllers. Although IP is not design
as MSI controller, we still can use it if limited IRQ number to 4.
pcie: endpoint: pci-epf-vntb: add endpoint MSI support
Based on ntb-next branch. https://github.com/jonmason/ntb/commits/ntb-next
Using MSI as door bell registers
This patch is totally independent on previous on. It can be
applied to ntb-next seperately.
i.MX EP function driver is upstreaming by Richard Zhu.
Some dts change missed at this patches. below is reference dts change
--- a/arch/arm64/boot/dts/freescale/imx8-ss-hsio.dtsi
+++ b/arch/arm64/boot/dts/freescale/imx8-ss-hsio.dtsi
@@ -160,5 +160,6 @@ pcieb_ep: pcie_ep@5f010000 {
num-ib-windows = <6>;
num-ob-windows = <6>;
status = "disabled";
+ MSI-parent = <&lsio_mu12>;
};
--- a/arch/arm64/boot/dts/freescale/imx8-ss-lsio.dtsi
+++ b/arch/arm64/boot/dts/freescale/imx8-ss-lsio.dtsi
@@ -172,6 +172,19 @@ lsio_mu6: mailbox@5d210000 {
status = "disabled";
};
+ lsio_mu12: mailbox@5d270000 {
+ compatible = "fsl,imx6sx-mu-MSI";
+ msi-controller;
+ interrupt-controller;
+ reg = <0x5d270000 0x10000>, /* A side */
+ <0x5d300000 0x10000>; /* B side */
+ reg-names = "a", "b";
+ interrupts = <GIC_SPI 191 IRQ_TYPE_LEVEL_HIGH>;
+ power-domains = <&pd IMX_SC_R_MU_12A>,
+ <&pd IMX_SC_R_MU_12B>;
+ power-domain-names = "a", "b";
+ };
+
Change Log
- Change from v2 to v3
Fixed dt-binding docment check failure
Fixed typo a cover letter.
Change according Bjorn's comments at patch
pcie: endpoint: pci-epf-vntb: add endpoint MSI support
- from V1 to V2
Fixed fsl,mu-msi.yaml's problem
Fixed irq-imx-mu-msi.c problem according Marc Zyngier's feeback
Added a new patch to allow pass down .pm by IRQCHIP_PLATFORM_DRIVER_END
--
2.35.1
^ permalink raw reply [flat|nested] 50+ messages in thread
* [PATCH v3 0/4] PCI EP driver support MSI doorbell from host
@ 2022-07-20 21:30 ` Frank Li
0 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-20 21:30 UTC (permalink / raw)
To: jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, peng.fan,
aisheng.dong, kernel, festevam, linux-imx, kishon,
lorenzo.pieralisi, ntb
┌───────┐ ┌──────────┐
│ │ │ │
┌─────────────┐ │ │ │ PCI Host │
│ MSI │◄┐ │ │ │ │
│ Controller │ │ │ │ │ │
└─────────────┘ └─┼───────┼──────────┼─Bar0 │
│ PCI │ │ Bar1 │
│ Func │ │ Bar2 │
│ │ │ Bar3 │
│ │ │ Bar4 │
│ ├─────────►│ │
└───────┘ └──────────┘
Many PCI controllers provided Endpoint functions.
Generally PCI endpoint is hardware, which is not running a rich OS,
like linux.
But Linux also supports endpoint functions. PCI Host write BAR<n> space
like write to memory. The EP side can't know memory changed by the Host
driver.
PCI Spec has not defined a standard method to do that. Only define
MSI(x) to let EP notified RC status change.
The basic idea is to trigger an IRQ when PCI RC writes to a memory
address. That's what MSI controller provided. EP drivers just need to
request a platform MSI interrupt, struct MSI_msg *msg will pass down a
memory address and data. EP driver will map such memory address to
one of PCI BAR<n>. Host just writes such an address to trigger EP side
IRQ.
If system have gic-its, only need update PCI EP side driver. But i.MX
have not chip support gic-its yet. So we have to use MU to simulate a
MSI controller. Although only 4 MSI IRQs are simulated, it matched
vntb(pci-epf-vntb) network requirement.
After enable MSI, ping delay reduce < 1ms from ~8ms
IRQchip: imx mu worked as MSI controller:
let imx mu worked as MSI controllers. Although IP is not design
as MSI controller, we still can use it if limited IRQ number to 4.
pcie: endpoint: pci-epf-vntb: add endpoint MSI support
Based on ntb-next branch. https://github.com/jonmason/ntb/commits/ntb-next
Using MSI as door bell registers
This patch is totally independent on previous on. It can be
applied to ntb-next seperately.
i.MX EP function driver is upstreaming by Richard Zhu.
Some dts change missed at this patches. below is reference dts change
--- a/arch/arm64/boot/dts/freescale/imx8-ss-hsio.dtsi
+++ b/arch/arm64/boot/dts/freescale/imx8-ss-hsio.dtsi
@@ -160,5 +160,6 @@ pcieb_ep: pcie_ep@5f010000 {
num-ib-windows = <6>;
num-ob-windows = <6>;
status = "disabled";
+ MSI-parent = <&lsio_mu12>;
};
--- a/arch/arm64/boot/dts/freescale/imx8-ss-lsio.dtsi
+++ b/arch/arm64/boot/dts/freescale/imx8-ss-lsio.dtsi
@@ -172,6 +172,19 @@ lsio_mu6: mailbox@5d210000 {
status = "disabled";
};
+ lsio_mu12: mailbox@5d270000 {
+ compatible = "fsl,imx6sx-mu-MSI";
+ msi-controller;
+ interrupt-controller;
+ reg = <0x5d270000 0x10000>, /* A side */
+ <0x5d300000 0x10000>; /* B side */
+ reg-names = "a", "b";
+ interrupts = <GIC_SPI 191 IRQ_TYPE_LEVEL_HIGH>;
+ power-domains = <&pd IMX_SC_R_MU_12A>,
+ <&pd IMX_SC_R_MU_12B>;
+ power-domain-names = "a", "b";
+ };
+
Change Log
- Change from v2 to v3
Fixed dt-binding docment check failure
Fixed typo a cover letter.
Change according Bjorn's comments at patch
pcie: endpoint: pci-epf-vntb: add endpoint MSI support
- from V1 to V2
Fixed fsl,mu-msi.yaml's problem
Fixed irq-imx-mu-msi.c problem according Marc Zyngier's feeback
Added a new patch to allow pass down .pm by IRQCHIP_PLATFORM_DRIVER_END
--
2.35.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* [PATCH v3 1/4] irqchip: allow pass down .pm field at IRQCHIP_PLATFORM_DRIVER_END
2022-07-20 21:30 ` Frank Li
@ 2022-07-20 21:30 ` Frank Li
-1 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-20 21:30 UTC (permalink / raw)
To: jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, peng.fan,
aisheng.dong, kernel, festevam, linux-imx, kishon,
lorenzo.pieralisi, ntb
IRQCHIP_PLATFORM_DRIVER_* compilation define platform_driver
for irqchip. But can't set .pm field of platform_driver.
Added variadic macros to set .pm field or other field if need.
Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
include/linux/irqchip.h | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/include/linux/irqchip.h b/include/linux/irqchip.h
index 3a091d0710ae1..d5e6024cb2a8c 100644
--- a/include/linux/irqchip.h
+++ b/include/linux/irqchip.h
@@ -44,7 +44,8 @@ static const struct of_device_id drv_name##_irqchip_match_table[] = {
#define IRQCHIP_MATCH(compat, fn) { .compatible = compat, \
.data = typecheck_irq_init_cb(fn), },
-#define IRQCHIP_PLATFORM_DRIVER_END(drv_name) \
+
+#define IRQCHIP_PLATFORM_DRIVER_END(drv_name, ...) \
{}, \
}; \
MODULE_DEVICE_TABLE(of, drv_name##_irqchip_match_table); \
@@ -56,6 +57,7 @@ static struct platform_driver drv_name##_driver = { \
.owner = THIS_MODULE, \
.of_match_table = drv_name##_irqchip_match_table, \
.suppress_bind_attrs = true, \
+ __VA_ARGS__ \
}, \
}; \
builtin_platform_driver(drv_name##_driver)
--
2.35.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [PATCH v3 1/4] irqchip: allow pass down .pm field at IRQCHIP_PLATFORM_DRIVER_END
@ 2022-07-20 21:30 ` Frank Li
0 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-20 21:30 UTC (permalink / raw)
To: jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, peng.fan,
aisheng.dong, kernel, festevam, linux-imx, kishon,
lorenzo.pieralisi, ntb
IRQCHIP_PLATFORM_DRIVER_* compilation define platform_driver
for irqchip. But can't set .pm field of platform_driver.
Added variadic macros to set .pm field or other field if need.
Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
include/linux/irqchip.h | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/include/linux/irqchip.h b/include/linux/irqchip.h
index 3a091d0710ae1..d5e6024cb2a8c 100644
--- a/include/linux/irqchip.h
+++ b/include/linux/irqchip.h
@@ -44,7 +44,8 @@ static const struct of_device_id drv_name##_irqchip_match_table[] = {
#define IRQCHIP_MATCH(compat, fn) { .compatible = compat, \
.data = typecheck_irq_init_cb(fn), },
-#define IRQCHIP_PLATFORM_DRIVER_END(drv_name) \
+
+#define IRQCHIP_PLATFORM_DRIVER_END(drv_name, ...) \
{}, \
}; \
MODULE_DEVICE_TABLE(of, drv_name##_irqchip_match_table); \
@@ -56,6 +57,7 @@ static struct platform_driver drv_name##_driver = { \
.owner = THIS_MODULE, \
.of_match_table = drv_name##_irqchip_match_table, \
.suppress_bind_attrs = true, \
+ __VA_ARGS__ \
}, \
}; \
builtin_platform_driver(drv_name##_driver)
--
2.35.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [PATCH v3 2/4] irqchip: imx mu worked as msi controller
2022-07-20 21:30 ` Frank Li
@ 2022-07-20 21:30 ` Frank Li
-1 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-20 21:30 UTC (permalink / raw)
To: jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, peng.fan,
aisheng.dong, kernel, festevam, linux-imx, kishon,
lorenzo.pieralisi, ntb
MU support generate irq by write data to a register.
This patch make mu worked as msi controller.
So MU can do doorbell by using standard msi api.
Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
drivers/irqchip/Kconfig | 7 +
drivers/irqchip/Makefile | 1 +
drivers/irqchip/irq-imx-mu-msi.c | 462 +++++++++++++++++++++++++++++++
3 files changed, 470 insertions(+)
create mode 100644 drivers/irqchip/irq-imx-mu-msi.c
diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
index 5e4e50122777d..4599471d880c0 100644
--- a/drivers/irqchip/Kconfig
+++ b/drivers/irqchip/Kconfig
@@ -470,6 +470,13 @@ config IMX_INTMUX
help
Support for the i.MX INTMUX interrupt multiplexer.
+config IMX_MU_MSI
+ bool "i.MX MU work as MSI controller"
+ default y if ARCH_MXC
+ select IRQ_DOMAIN
+ help
+ MU work as MSI controller to do general doorbell
+
config LS1X_IRQ
bool "Loongson-1 Interrupt Controller"
depends on MACH_LOONGSON32
diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
index 5d8e21d3dc6d8..870423746c783 100644
--- a/drivers/irqchip/Makefile
+++ b/drivers/irqchip/Makefile
@@ -98,6 +98,7 @@ obj-$(CONFIG_RISCV_INTC) += irq-riscv-intc.o
obj-$(CONFIG_SIFIVE_PLIC) += irq-sifive-plic.o
obj-$(CONFIG_IMX_IRQSTEER) += irq-imx-irqsteer.o
obj-$(CONFIG_IMX_INTMUX) += irq-imx-intmux.o
+obj-$(CONFIG_IMX_MU_MSI) += irq-imx-mu-msi.o
obj-$(CONFIG_MADERA_IRQ) += irq-madera.o
obj-$(CONFIG_LS1X_IRQ) += irq-ls1x.o
obj-$(CONFIG_TI_SCI_INTR_IRQCHIP) += irq-ti-sci-intr.o
diff --git a/drivers/irqchip/irq-imx-mu-msi.c b/drivers/irqchip/irq-imx-mu-msi.c
new file mode 100644
index 0000000000000..8277dba857759
--- /dev/null
+++ b/drivers/irqchip/irq-imx-mu-msi.c
@@ -0,0 +1,462 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * NXP MU worked as MSI controller
+ *
+ * Copyright (c) 2018 Pengutronix, Oleksij Rempel <o.rempel@pengutronix.de>
+ * Copyright 2022 NXP
+ * Frank Li <Frank.Li@nxp.com>
+ * Peng Fan <peng.fan@nxp.com>
+ *
+ * Based on drivers/mailbox/imx-mailbox.c
+ */
+#include <linux/clk.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/msi.h>
+#include <linux/interrupt.h>
+#include <linux/irq.h>
+#include <linux/irqchip/chained_irq.h>
+#include <linux/irqchip.h>
+#include <linux/irqdomain.h>
+#include <linux/of_irq.h>
+#include <linux/of_pci.h>
+#include <linux/of_platform.h>
+#include <linux/spinlock.h>
+#include <linux/dma-iommu.h>
+#include <linux/pm_runtime.h>
+#include <linux/pm_domain.h>
+
+
+#define IMX_MU_CHANS 4
+
+enum imx_mu_xcr {
+ IMX_MU_GIER,
+ IMX_MU_GCR,
+ IMX_MU_TCR,
+ IMX_MU_RCR,
+ IMX_MU_xCR_MAX,
+};
+
+enum imx_mu_xsr {
+ IMX_MU_SR,
+ IMX_MU_GSR,
+ IMX_MU_TSR,
+ IMX_MU_RSR,
+};
+
+enum imx_mu_type {
+ IMX_MU_V1 = BIT(0),
+ IMX_MU_V2 = BIT(1),
+ IMX_MU_V2_S4 = BIT(15),
+};
+
+/* Receive Interrupt Enable */
+#define IMX_MU_xCR_RIEn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24 + (3 - (x))))
+#define IMX_MU_xSR_RFn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24 + (3 - (x))))
+
+struct imx_mu_dcfg {
+ enum imx_mu_type type;
+ u32 xTR; /* Transmit Register0 */
+ u32 xRR; /* Receive Register0 */
+ u32 xSR[4]; /* Status Registers */
+ u32 xCR[4]; /* Control Registers */
+};
+
+struct imx_mu_msi {
+ spinlock_t lock;
+ struct platform_device *pdev;
+ struct irq_domain *parent;
+ struct irq_domain *msi_domain;
+ void __iomem *regs;
+ phys_addr_t msiir_addr;
+ const struct imx_mu_dcfg *cfg;
+ unsigned long used;
+ u32 gic_irq;
+ struct clk *clk;
+ struct device *pd_a;
+ struct device *pd_b;
+ struct device_link *pd_link_a;
+ struct device_link *pd_link_b;
+};
+
+static void imx_mu_write(struct imx_mu_msi *msi_data, u32 val, u32 offs)
+{
+ iowrite32(val, msi_data->regs + offs);
+}
+
+static u32 imx_mu_read(struct imx_mu_msi *msi_data, u32 offs)
+{
+ return ioread32(msi_data->regs + offs);
+}
+
+static u32 imx_mu_xcr_rmw(struct imx_mu_msi *msi_data, enum imx_mu_xcr type, u32 set, u32 clr)
+{
+ unsigned long flags;
+ u32 val;
+
+ spin_lock_irqsave(&msi_data->lock, flags);
+ val = imx_mu_read(msi_data, msi_data->cfg->xCR[type]);
+ val &= ~clr;
+ val |= set;
+ imx_mu_write(msi_data, val, msi_data->cfg->xCR[type]);
+ spin_unlock_irqrestore(&msi_data->lock, flags);
+
+ return val;
+}
+
+static void imx_mu_msi_mask_irq(struct irq_data *data)
+{
+ struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data->parent_data);
+
+ imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, 0, IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq));
+}
+
+static void imx_mu_msi_unmask_irq(struct irq_data *data)
+{
+ struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data->parent_data);
+
+ imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq), 0);
+}
+
+static struct irq_chip imx_mu_msi_irq_chip = {
+ .name = "MU-MSI",
+ .irq_mask = imx_mu_msi_mask_irq,
+ .irq_unmask = imx_mu_msi_unmask_irq,
+};
+
+static struct msi_domain_ops its_pmsi_ops = {
+};
+
+static struct msi_domain_info imx_mu_msi_domain_info = {
+ .flags = (MSI_FLAG_USE_DEF_DOM_OPS |
+ MSI_FLAG_USE_DEF_CHIP_OPS |
+ MSI_FLAG_PCI_MSIX),
+ .ops = &its_pmsi_ops,
+ .chip = &imx_mu_msi_irq_chip,
+};
+
+static void imx_mu_msi_compose_msg(struct irq_data *data, struct msi_msg *msg)
+{
+ struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
+
+ msg->address_hi = upper_32_bits(msi_data->msiir_addr);
+ msg->address_lo = lower_32_bits(msi_data->msiir_addr + 4 * data->hwirq);
+ msg->data = data->hwirq;
+
+ iommu_dma_compose_msi_msg(irq_data_get_msi_desc(data), msg);
+}
+
+static int imx_mu_msi_set_affinity(struct irq_data *irq_data,
+ const struct cpumask *mask, bool force)
+
+{
+ return IRQ_SET_MASK_OK;
+}
+
+static struct irq_chip imx_mu_msi_parent_chip = {
+ .name = "MU",
+ .irq_compose_msi_msg = imx_mu_msi_compose_msg,
+ .irq_set_affinity = imx_mu_msi_set_affinity,
+};
+
+static int imx_mu_msi_domain_irq_alloc(struct irq_domain *domain,
+ unsigned int virq,
+ unsigned int nr_irqs,
+ void *args)
+{
+ struct imx_mu_msi *msi_data = domain->host_data;
+ msi_alloc_info_t *info = args;
+ int pos, err = 0;
+
+ WARN_ON(nr_irqs != 1);
+
+ spin_lock(&msi_data->lock);
+ pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
+ if (pos < IMX_MU_CHANS)
+ __set_bit(pos, &msi_data->used);
+ else
+ err = -ENOSPC;
+ spin_unlock(&msi_data->lock);
+
+ if (err)
+ return err;
+
+ err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr + pos * 4);
+ if (err)
+ return err;
+
+ irq_domain_set_info(domain, virq, pos,
+ &imx_mu_msi_parent_chip, msi_data,
+ handle_simple_irq, NULL, NULL);
+ return 0;
+}
+
+static void imx_mu_msi_domain_irq_free(struct irq_domain *domain,
+ unsigned int virq, unsigned int nr_irqs)
+{
+ struct irq_data *d = irq_domain_get_irq_data(domain, virq);
+ struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(d);
+
+ spin_lock(&msi_data->lock);
+ __clear_bit(d->hwirq, &msi_data->used);
+ spin_unlock(&msi_data->lock);
+}
+
+static const struct irq_domain_ops imx_mu_msi_domain_ops = {
+ .alloc = imx_mu_msi_domain_irq_alloc,
+ .free = imx_mu_msi_domain_irq_free,
+};
+
+static void imx_mu_msi_irq_handler(struct irq_desc *desc)
+{
+ struct imx_mu_msi *msi_data = irq_desc_get_handler_data(desc);
+ u32 status;
+ int i;
+
+ status = imx_mu_read(msi_data, msi_data->cfg->xSR[IMX_MU_RSR]);
+
+ chained_irq_enter(irq_desc_get_chip(desc), desc);
+ for (i = 0; i < IMX_MU_CHANS; i++) {
+ if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
+ imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
+ generic_handle_domain_irq(msi_data->parent, i);
+ }
+ }
+ chained_irq_exit(irq_desc_get_chip(desc), desc);
+}
+
+static int imx_mu_msi_domains_init(struct imx_mu_msi *msi_data)
+{
+ /* Initialize MSI domain parent */
+ msi_data->parent = irq_domain_add_linear(dev_of_node(&msi_data->pdev->dev),
+ IMX_MU_CHANS,
+ &imx_mu_msi_domain_ops,
+ msi_data);
+ if (!msi_data->parent) {
+ dev_err(&msi_data->pdev->dev, "failed to create IRQ domain\n");
+ return -ENOMEM;
+ }
+
+ msi_data->msi_domain = platform_msi_create_irq_domain(
+ of_node_to_fwnode(msi_data->pdev->dev.of_node),
+ &imx_mu_msi_domain_info,
+ msi_data->parent);
+
+ if (!msi_data->msi_domain) {
+ dev_err(&msi_data->pdev->dev, "failed to create MSI domain\n");
+ irq_domain_remove(msi_data->parent);
+ return -ENOMEM;
+ }
+
+ irq_domain_set_pm_device(msi_data->parent, &msi_data->pdev->dev);
+
+ return 0;
+}
+
+/* Register offset of different version MU IP */
+static const struct imx_mu_dcfg imx_mu_cfg_imx6sx = {
+ .xTR = 0x0,
+ .xRR = 0x10,
+ .xSR = {0x20, 0x20, 0x20, 0x20},
+ .xCR = {0x24, 0x24, 0x24, 0x24},
+};
+
+static const struct imx_mu_dcfg imx_mu_cfg_imx7ulp = {
+ .xTR = 0x20,
+ .xRR = 0x40,
+ .xSR = {0x60, 0x60, 0x60, 0x60},
+ .xCR = {0x64, 0x64, 0x64, 0x64},
+};
+
+static const struct imx_mu_dcfg imx_mu_cfg_imx8ulp = {
+ .type = IMX_MU_V2,
+ .xTR = 0x200,
+ .xRR = 0x280,
+ .xSR = {0xC, 0x118, 0x124, 0x12C},
+ .xCR = {0x110, 0x114, 0x120, 0x128},
+};
+
+static const struct imx_mu_dcfg imx_mu_cfg_imx8ulp_s4 = {
+
+ .type = IMX_MU_V2 | IMX_MU_V2_S4,
+ .xTR = 0x200,
+ .xRR = 0x280,
+ .xSR = {0xC, 0x118, 0x124, 0x12C},
+ .xCR = {0x110, 0x114, 0x120, 0x128},
+};
+
+static int __init imx_mu_of_init(struct device_node *dn,
+ struct device_node *parent,
+ const struct imx_mu_dcfg *cfg)
+{
+ struct platform_device *pdev = of_find_device_by_node(dn);
+ struct imx_mu_msi *msi_data, *priv;
+ struct resource *res;
+ struct device *dev;
+ int ret;
+
+ if (!pdev)
+ return -ENODEV;
+
+ dev = &pdev->dev;
+
+ priv = msi_data = devm_kzalloc(&pdev->dev, sizeof(*msi_data), GFP_KERNEL);
+ if (!msi_data)
+ return -ENOMEM;
+
+ msi_data->cfg = cfg;
+
+ msi_data->regs = devm_platform_ioremap_resource_byname(pdev, "a");
+ if (IS_ERR(msi_data->regs)) {
+ dev_err(&pdev->dev, "failed to initialize 'regs'\n");
+ return PTR_ERR(msi_data->regs);
+ }
+
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "b");
+ if (!res)
+ return -EIO;
+
+ msi_data->msiir_addr = res->start + msi_data->cfg->xTR;
+
+ msi_data->pdev = pdev;
+
+ msi_data->gic_irq = platform_get_irq(msi_data->pdev, 0);
+ if (msi_data->gic_irq <= 0)
+ return -ENODEV;
+
+ platform_set_drvdata(pdev, msi_data);
+
+ msi_data->clk = devm_clk_get(dev, NULL);
+ if (IS_ERR(msi_data->clk)) {
+ if (PTR_ERR(msi_data->clk) != -ENOENT)
+ return PTR_ERR(msi_data->clk);
+
+ msi_data->clk = NULL;
+ }
+
+ ret = clk_prepare_enable(msi_data->clk);
+ if (ret) {
+ dev_err(dev, "Failed to enable clock\n");
+ return ret;
+ }
+
+ priv->pd_a = dev_pm_domain_attach_by_name(dev, "a");
+ if (IS_ERR(priv->pd_a))
+ return PTR_ERR(priv->pd_a);
+
+ priv->pd_link_a = device_link_add(dev, priv->pd_a,
+ DL_FLAG_STATELESS |
+ DL_FLAG_PM_RUNTIME |
+ DL_FLAG_RPM_ACTIVE);
+
+ if (!priv->pd_link_a) {
+ dev_err(dev, "Failed to add device_link to mu a.\n");
+ return -EINVAL;
+ }
+
+ priv->pd_b = dev_pm_domain_attach_by_name(dev, "b");
+ if (IS_ERR(priv->pd_b))
+ return PTR_ERR(priv->pd_b);
+
+ priv->pd_link_b = device_link_add(dev, priv->pd_b,
+ DL_FLAG_STATELESS |
+ DL_FLAG_PM_RUNTIME |
+ DL_FLAG_RPM_ACTIVE);
+
+ if (!priv->pd_link_b) {
+ dev_err(dev, "Failed to add device_link to mu a.\n");
+ return -EINVAL;
+ }
+
+ ret = imx_mu_msi_domains_init(msi_data);
+ if (ret)
+ return ret;
+
+ irq_set_chained_handler_and_data(msi_data->gic_irq,
+ imx_mu_msi_irq_handler,
+ msi_data);
+
+ pm_runtime_enable(dev);
+
+ ret = pm_runtime_get_sync(dev);
+ if (ret < 0) {
+ pm_runtime_put_noidle(dev);
+ goto disable_runtime_pm;
+ }
+
+ ret = pm_runtime_put_sync(dev);
+ if (ret < 0)
+ goto disable_runtime_pm;
+
+ clk_disable_unprepare(msi_data->clk);
+
+ return 0;
+
+disable_runtime_pm:
+ pm_runtime_disable(dev);
+ clk_disable_unprepare(msi_data->clk);
+
+ return ret;
+}
+
+static int __maybe_unused imx_mu_runtime_suspend(struct device *dev)
+{
+ struct imx_mu_msi *priv = dev_get_drvdata(dev);
+
+ clk_disable_unprepare(priv->clk);
+
+ return 0;
+}
+
+static int __maybe_unused imx_mu_runtime_resume(struct device *dev)
+{
+ struct imx_mu_msi *priv = dev_get_drvdata(dev);
+ int ret;
+
+ ret = clk_prepare_enable(priv->clk);
+ if (ret)
+ dev_err(dev, "failed to enable clock\n");
+
+ return ret;
+}
+
+static const struct dev_pm_ops imx_mu_pm_ops = {
+ SET_RUNTIME_PM_OPS(imx_mu_runtime_suspend,
+ imx_mu_runtime_resume, NULL)
+};
+
+static int __init imx_mu_imx7ulp_of_init(struct device_node *dn,
+ struct device_node *parent)
+{
+ return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx7ulp);
+}
+
+static int __init imx_mu_imx6sx_of_init(struct device_node *dn,
+ struct device_node *parent)
+{
+ return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx6sx);
+}
+
+static int __init imx_mu_imx8ulp_of_init(struct device_node *dn,
+ struct device_node *parent)
+{
+ return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx8ulp);
+}
+
+static int __init imx_mu_imx8ulp_s4_of_init(struct device_node *dn,
+ struct device_node *parent)
+{
+ return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx8ulp_s4);
+}
+
+IRQCHIP_PLATFORM_DRIVER_BEGIN(imx_mu_msi)
+IRQCHIP_MATCH("fsl,imx7ulp-mu-msi", imx_mu_imx7ulp_of_init)
+IRQCHIP_MATCH("fsl,imx6sx-mu-msi", imx_mu_imx6sx_of_init)
+IRQCHIP_MATCH("fsl,imx8ulp-mu-msi", imx_mu_imx8ulp_of_init)
+IRQCHIP_MATCH("fsl,imx8ulp-mu-msi-s4", imx_mu_imx8ulp_s4_of_init)
+IRQCHIP_PLATFORM_DRIVER_END(imx_mu_msi, .pm = &imx_mu_pm_ops)
+
+
+MODULE_AUTHOR("Frank Li <Frank.Li@nxp.com>");
+MODULE_DESCRIPTION("Freescale MU work as MSI controller driver");
+MODULE_LICENSE("GPL");
--
2.35.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [PATCH v3 2/4] irqchip: imx mu worked as msi controller
@ 2022-07-20 21:30 ` Frank Li
0 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-20 21:30 UTC (permalink / raw)
To: jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, peng.fan,
aisheng.dong, kernel, festevam, linux-imx, kishon,
lorenzo.pieralisi, ntb
MU support generate irq by write data to a register.
This patch make mu worked as msi controller.
So MU can do doorbell by using standard msi api.
Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
drivers/irqchip/Kconfig | 7 +
drivers/irqchip/Makefile | 1 +
drivers/irqchip/irq-imx-mu-msi.c | 462 +++++++++++++++++++++++++++++++
3 files changed, 470 insertions(+)
create mode 100644 drivers/irqchip/irq-imx-mu-msi.c
diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
index 5e4e50122777d..4599471d880c0 100644
--- a/drivers/irqchip/Kconfig
+++ b/drivers/irqchip/Kconfig
@@ -470,6 +470,13 @@ config IMX_INTMUX
help
Support for the i.MX INTMUX interrupt multiplexer.
+config IMX_MU_MSI
+ bool "i.MX MU work as MSI controller"
+ default y if ARCH_MXC
+ select IRQ_DOMAIN
+ help
+ MU work as MSI controller to do general doorbell
+
config LS1X_IRQ
bool "Loongson-1 Interrupt Controller"
depends on MACH_LOONGSON32
diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
index 5d8e21d3dc6d8..870423746c783 100644
--- a/drivers/irqchip/Makefile
+++ b/drivers/irqchip/Makefile
@@ -98,6 +98,7 @@ obj-$(CONFIG_RISCV_INTC) += irq-riscv-intc.o
obj-$(CONFIG_SIFIVE_PLIC) += irq-sifive-plic.o
obj-$(CONFIG_IMX_IRQSTEER) += irq-imx-irqsteer.o
obj-$(CONFIG_IMX_INTMUX) += irq-imx-intmux.o
+obj-$(CONFIG_IMX_MU_MSI) += irq-imx-mu-msi.o
obj-$(CONFIG_MADERA_IRQ) += irq-madera.o
obj-$(CONFIG_LS1X_IRQ) += irq-ls1x.o
obj-$(CONFIG_TI_SCI_INTR_IRQCHIP) += irq-ti-sci-intr.o
diff --git a/drivers/irqchip/irq-imx-mu-msi.c b/drivers/irqchip/irq-imx-mu-msi.c
new file mode 100644
index 0000000000000..8277dba857759
--- /dev/null
+++ b/drivers/irqchip/irq-imx-mu-msi.c
@@ -0,0 +1,462 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * NXP MU worked as MSI controller
+ *
+ * Copyright (c) 2018 Pengutronix, Oleksij Rempel <o.rempel@pengutronix.de>
+ * Copyright 2022 NXP
+ * Frank Li <Frank.Li@nxp.com>
+ * Peng Fan <peng.fan@nxp.com>
+ *
+ * Based on drivers/mailbox/imx-mailbox.c
+ */
+#include <linux/clk.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/msi.h>
+#include <linux/interrupt.h>
+#include <linux/irq.h>
+#include <linux/irqchip/chained_irq.h>
+#include <linux/irqchip.h>
+#include <linux/irqdomain.h>
+#include <linux/of_irq.h>
+#include <linux/of_pci.h>
+#include <linux/of_platform.h>
+#include <linux/spinlock.h>
+#include <linux/dma-iommu.h>
+#include <linux/pm_runtime.h>
+#include <linux/pm_domain.h>
+
+
+#define IMX_MU_CHANS 4
+
+enum imx_mu_xcr {
+ IMX_MU_GIER,
+ IMX_MU_GCR,
+ IMX_MU_TCR,
+ IMX_MU_RCR,
+ IMX_MU_xCR_MAX,
+};
+
+enum imx_mu_xsr {
+ IMX_MU_SR,
+ IMX_MU_GSR,
+ IMX_MU_TSR,
+ IMX_MU_RSR,
+};
+
+enum imx_mu_type {
+ IMX_MU_V1 = BIT(0),
+ IMX_MU_V2 = BIT(1),
+ IMX_MU_V2_S4 = BIT(15),
+};
+
+/* Receive Interrupt Enable */
+#define IMX_MU_xCR_RIEn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24 + (3 - (x))))
+#define IMX_MU_xSR_RFn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24 + (3 - (x))))
+
+struct imx_mu_dcfg {
+ enum imx_mu_type type;
+ u32 xTR; /* Transmit Register0 */
+ u32 xRR; /* Receive Register0 */
+ u32 xSR[4]; /* Status Registers */
+ u32 xCR[4]; /* Control Registers */
+};
+
+struct imx_mu_msi {
+ spinlock_t lock;
+ struct platform_device *pdev;
+ struct irq_domain *parent;
+ struct irq_domain *msi_domain;
+ void __iomem *regs;
+ phys_addr_t msiir_addr;
+ const struct imx_mu_dcfg *cfg;
+ unsigned long used;
+ u32 gic_irq;
+ struct clk *clk;
+ struct device *pd_a;
+ struct device *pd_b;
+ struct device_link *pd_link_a;
+ struct device_link *pd_link_b;
+};
+
+static void imx_mu_write(struct imx_mu_msi *msi_data, u32 val, u32 offs)
+{
+ iowrite32(val, msi_data->regs + offs);
+}
+
+static u32 imx_mu_read(struct imx_mu_msi *msi_data, u32 offs)
+{
+ return ioread32(msi_data->regs + offs);
+}
+
+static u32 imx_mu_xcr_rmw(struct imx_mu_msi *msi_data, enum imx_mu_xcr type, u32 set, u32 clr)
+{
+ unsigned long flags;
+ u32 val;
+
+ spin_lock_irqsave(&msi_data->lock, flags);
+ val = imx_mu_read(msi_data, msi_data->cfg->xCR[type]);
+ val &= ~clr;
+ val |= set;
+ imx_mu_write(msi_data, val, msi_data->cfg->xCR[type]);
+ spin_unlock_irqrestore(&msi_data->lock, flags);
+
+ return val;
+}
+
+static void imx_mu_msi_mask_irq(struct irq_data *data)
+{
+ struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data->parent_data);
+
+ imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, 0, IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq));
+}
+
+static void imx_mu_msi_unmask_irq(struct irq_data *data)
+{
+ struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data->parent_data);
+
+ imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq), 0);
+}
+
+static struct irq_chip imx_mu_msi_irq_chip = {
+ .name = "MU-MSI",
+ .irq_mask = imx_mu_msi_mask_irq,
+ .irq_unmask = imx_mu_msi_unmask_irq,
+};
+
+static struct msi_domain_ops its_pmsi_ops = {
+};
+
+static struct msi_domain_info imx_mu_msi_domain_info = {
+ .flags = (MSI_FLAG_USE_DEF_DOM_OPS |
+ MSI_FLAG_USE_DEF_CHIP_OPS |
+ MSI_FLAG_PCI_MSIX),
+ .ops = &its_pmsi_ops,
+ .chip = &imx_mu_msi_irq_chip,
+};
+
+static void imx_mu_msi_compose_msg(struct irq_data *data, struct msi_msg *msg)
+{
+ struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
+
+ msg->address_hi = upper_32_bits(msi_data->msiir_addr);
+ msg->address_lo = lower_32_bits(msi_data->msiir_addr + 4 * data->hwirq);
+ msg->data = data->hwirq;
+
+ iommu_dma_compose_msi_msg(irq_data_get_msi_desc(data), msg);
+}
+
+static int imx_mu_msi_set_affinity(struct irq_data *irq_data,
+ const struct cpumask *mask, bool force)
+
+{
+ return IRQ_SET_MASK_OK;
+}
+
+static struct irq_chip imx_mu_msi_parent_chip = {
+ .name = "MU",
+ .irq_compose_msi_msg = imx_mu_msi_compose_msg,
+ .irq_set_affinity = imx_mu_msi_set_affinity,
+};
+
+static int imx_mu_msi_domain_irq_alloc(struct irq_domain *domain,
+ unsigned int virq,
+ unsigned int nr_irqs,
+ void *args)
+{
+ struct imx_mu_msi *msi_data = domain->host_data;
+ msi_alloc_info_t *info = args;
+ int pos, err = 0;
+
+ WARN_ON(nr_irqs != 1);
+
+ spin_lock(&msi_data->lock);
+ pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
+ if (pos < IMX_MU_CHANS)
+ __set_bit(pos, &msi_data->used);
+ else
+ err = -ENOSPC;
+ spin_unlock(&msi_data->lock);
+
+ if (err)
+ return err;
+
+ err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr + pos * 4);
+ if (err)
+ return err;
+
+ irq_domain_set_info(domain, virq, pos,
+ &imx_mu_msi_parent_chip, msi_data,
+ handle_simple_irq, NULL, NULL);
+ return 0;
+}
+
+static void imx_mu_msi_domain_irq_free(struct irq_domain *domain,
+ unsigned int virq, unsigned int nr_irqs)
+{
+ struct irq_data *d = irq_domain_get_irq_data(domain, virq);
+ struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(d);
+
+ spin_lock(&msi_data->lock);
+ __clear_bit(d->hwirq, &msi_data->used);
+ spin_unlock(&msi_data->lock);
+}
+
+static const struct irq_domain_ops imx_mu_msi_domain_ops = {
+ .alloc = imx_mu_msi_domain_irq_alloc,
+ .free = imx_mu_msi_domain_irq_free,
+};
+
+static void imx_mu_msi_irq_handler(struct irq_desc *desc)
+{
+ struct imx_mu_msi *msi_data = irq_desc_get_handler_data(desc);
+ u32 status;
+ int i;
+
+ status = imx_mu_read(msi_data, msi_data->cfg->xSR[IMX_MU_RSR]);
+
+ chained_irq_enter(irq_desc_get_chip(desc), desc);
+ for (i = 0; i < IMX_MU_CHANS; i++) {
+ if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
+ imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
+ generic_handle_domain_irq(msi_data->parent, i);
+ }
+ }
+ chained_irq_exit(irq_desc_get_chip(desc), desc);
+}
+
+static int imx_mu_msi_domains_init(struct imx_mu_msi *msi_data)
+{
+ /* Initialize MSI domain parent */
+ msi_data->parent = irq_domain_add_linear(dev_of_node(&msi_data->pdev->dev),
+ IMX_MU_CHANS,
+ &imx_mu_msi_domain_ops,
+ msi_data);
+ if (!msi_data->parent) {
+ dev_err(&msi_data->pdev->dev, "failed to create IRQ domain\n");
+ return -ENOMEM;
+ }
+
+ msi_data->msi_domain = platform_msi_create_irq_domain(
+ of_node_to_fwnode(msi_data->pdev->dev.of_node),
+ &imx_mu_msi_domain_info,
+ msi_data->parent);
+
+ if (!msi_data->msi_domain) {
+ dev_err(&msi_data->pdev->dev, "failed to create MSI domain\n");
+ irq_domain_remove(msi_data->parent);
+ return -ENOMEM;
+ }
+
+ irq_domain_set_pm_device(msi_data->parent, &msi_data->pdev->dev);
+
+ return 0;
+}
+
+/* Register offset of different version MU IP */
+static const struct imx_mu_dcfg imx_mu_cfg_imx6sx = {
+ .xTR = 0x0,
+ .xRR = 0x10,
+ .xSR = {0x20, 0x20, 0x20, 0x20},
+ .xCR = {0x24, 0x24, 0x24, 0x24},
+};
+
+static const struct imx_mu_dcfg imx_mu_cfg_imx7ulp = {
+ .xTR = 0x20,
+ .xRR = 0x40,
+ .xSR = {0x60, 0x60, 0x60, 0x60},
+ .xCR = {0x64, 0x64, 0x64, 0x64},
+};
+
+static const struct imx_mu_dcfg imx_mu_cfg_imx8ulp = {
+ .type = IMX_MU_V2,
+ .xTR = 0x200,
+ .xRR = 0x280,
+ .xSR = {0xC, 0x118, 0x124, 0x12C},
+ .xCR = {0x110, 0x114, 0x120, 0x128},
+};
+
+static const struct imx_mu_dcfg imx_mu_cfg_imx8ulp_s4 = {
+
+ .type = IMX_MU_V2 | IMX_MU_V2_S4,
+ .xTR = 0x200,
+ .xRR = 0x280,
+ .xSR = {0xC, 0x118, 0x124, 0x12C},
+ .xCR = {0x110, 0x114, 0x120, 0x128},
+};
+
+static int __init imx_mu_of_init(struct device_node *dn,
+ struct device_node *parent,
+ const struct imx_mu_dcfg *cfg)
+{
+ struct platform_device *pdev = of_find_device_by_node(dn);
+ struct imx_mu_msi *msi_data, *priv;
+ struct resource *res;
+ struct device *dev;
+ int ret;
+
+ if (!pdev)
+ return -ENODEV;
+
+ dev = &pdev->dev;
+
+ priv = msi_data = devm_kzalloc(&pdev->dev, sizeof(*msi_data), GFP_KERNEL);
+ if (!msi_data)
+ return -ENOMEM;
+
+ msi_data->cfg = cfg;
+
+ msi_data->regs = devm_platform_ioremap_resource_byname(pdev, "a");
+ if (IS_ERR(msi_data->regs)) {
+ dev_err(&pdev->dev, "failed to initialize 'regs'\n");
+ return PTR_ERR(msi_data->regs);
+ }
+
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "b");
+ if (!res)
+ return -EIO;
+
+ msi_data->msiir_addr = res->start + msi_data->cfg->xTR;
+
+ msi_data->pdev = pdev;
+
+ msi_data->gic_irq = platform_get_irq(msi_data->pdev, 0);
+ if (msi_data->gic_irq <= 0)
+ return -ENODEV;
+
+ platform_set_drvdata(pdev, msi_data);
+
+ msi_data->clk = devm_clk_get(dev, NULL);
+ if (IS_ERR(msi_data->clk)) {
+ if (PTR_ERR(msi_data->clk) != -ENOENT)
+ return PTR_ERR(msi_data->clk);
+
+ msi_data->clk = NULL;
+ }
+
+ ret = clk_prepare_enable(msi_data->clk);
+ if (ret) {
+ dev_err(dev, "Failed to enable clock\n");
+ return ret;
+ }
+
+ priv->pd_a = dev_pm_domain_attach_by_name(dev, "a");
+ if (IS_ERR(priv->pd_a))
+ return PTR_ERR(priv->pd_a);
+
+ priv->pd_link_a = device_link_add(dev, priv->pd_a,
+ DL_FLAG_STATELESS |
+ DL_FLAG_PM_RUNTIME |
+ DL_FLAG_RPM_ACTIVE);
+
+ if (!priv->pd_link_a) {
+ dev_err(dev, "Failed to add device_link to mu a.\n");
+ return -EINVAL;
+ }
+
+ priv->pd_b = dev_pm_domain_attach_by_name(dev, "b");
+ if (IS_ERR(priv->pd_b))
+ return PTR_ERR(priv->pd_b);
+
+ priv->pd_link_b = device_link_add(dev, priv->pd_b,
+ DL_FLAG_STATELESS |
+ DL_FLAG_PM_RUNTIME |
+ DL_FLAG_RPM_ACTIVE);
+
+ if (!priv->pd_link_b) {
+ dev_err(dev, "Failed to add device_link to mu a.\n");
+ return -EINVAL;
+ }
+
+ ret = imx_mu_msi_domains_init(msi_data);
+ if (ret)
+ return ret;
+
+ irq_set_chained_handler_and_data(msi_data->gic_irq,
+ imx_mu_msi_irq_handler,
+ msi_data);
+
+ pm_runtime_enable(dev);
+
+ ret = pm_runtime_get_sync(dev);
+ if (ret < 0) {
+ pm_runtime_put_noidle(dev);
+ goto disable_runtime_pm;
+ }
+
+ ret = pm_runtime_put_sync(dev);
+ if (ret < 0)
+ goto disable_runtime_pm;
+
+ clk_disable_unprepare(msi_data->clk);
+
+ return 0;
+
+disable_runtime_pm:
+ pm_runtime_disable(dev);
+ clk_disable_unprepare(msi_data->clk);
+
+ return ret;
+}
+
+static int __maybe_unused imx_mu_runtime_suspend(struct device *dev)
+{
+ struct imx_mu_msi *priv = dev_get_drvdata(dev);
+
+ clk_disable_unprepare(priv->clk);
+
+ return 0;
+}
+
+static int __maybe_unused imx_mu_runtime_resume(struct device *dev)
+{
+ struct imx_mu_msi *priv = dev_get_drvdata(dev);
+ int ret;
+
+ ret = clk_prepare_enable(priv->clk);
+ if (ret)
+ dev_err(dev, "failed to enable clock\n");
+
+ return ret;
+}
+
+static const struct dev_pm_ops imx_mu_pm_ops = {
+ SET_RUNTIME_PM_OPS(imx_mu_runtime_suspend,
+ imx_mu_runtime_resume, NULL)
+};
+
+static int __init imx_mu_imx7ulp_of_init(struct device_node *dn,
+ struct device_node *parent)
+{
+ return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx7ulp);
+}
+
+static int __init imx_mu_imx6sx_of_init(struct device_node *dn,
+ struct device_node *parent)
+{
+ return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx6sx);
+}
+
+static int __init imx_mu_imx8ulp_of_init(struct device_node *dn,
+ struct device_node *parent)
+{
+ return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx8ulp);
+}
+
+static int __init imx_mu_imx8ulp_s4_of_init(struct device_node *dn,
+ struct device_node *parent)
+{
+ return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx8ulp_s4);
+}
+
+IRQCHIP_PLATFORM_DRIVER_BEGIN(imx_mu_msi)
+IRQCHIP_MATCH("fsl,imx7ulp-mu-msi", imx_mu_imx7ulp_of_init)
+IRQCHIP_MATCH("fsl,imx6sx-mu-msi", imx_mu_imx6sx_of_init)
+IRQCHIP_MATCH("fsl,imx8ulp-mu-msi", imx_mu_imx8ulp_of_init)
+IRQCHIP_MATCH("fsl,imx8ulp-mu-msi-s4", imx_mu_imx8ulp_s4_of_init)
+IRQCHIP_PLATFORM_DRIVER_END(imx_mu_msi, .pm = &imx_mu_pm_ops)
+
+
+MODULE_AUTHOR("Frank Li <Frank.Li@nxp.com>");
+MODULE_DESCRIPTION("Freescale MU work as MSI controller driver");
+MODULE_LICENSE("GPL");
--
2.35.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
2022-07-20 21:30 ` Frank Li
@ 2022-07-20 21:30 ` Frank Li
-1 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-20 21:30 UTC (permalink / raw)
To: jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, peng.fan,
aisheng.dong, kernel, festevam, linux-imx, kishon,
lorenzo.pieralisi, ntb
imx mu support generate irq by write a register.
provide msi controller support so other driver
can use it by standard msi interface.
Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
.../interrupt-controller/fsl,mu-msi.yaml | 88 +++++++++++++++++++
1 file changed, 88 insertions(+)
create mode 100644 Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
diff --git a/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
new file mode 100644
index 0000000000000..e125294243af3
--- /dev/null
+++ b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
@@ -0,0 +1,88 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/interrupt-controller/fsl,mu-msi.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: NXP i.MX Messaging Unit (MU) work as msi controller
+
+maintainers:
+ - Frank Li <Frank.Li@nxp.com>
+
+description: |
+ The Messaging Unit module enables two processors within the SoC to
+ communicate and coordinate by passing messages (e.g. data, status
+ and control) through the MU interface. The MU also provides the ability
+ for one processor to signal the other processor using interrupts.
+
+ Because the MU manages the messaging between processors, the MU uses
+ different clocks (from each side of the different peripheral buses).
+ Therefore, the MU must synchronize the accesses from one side to the
+ other. The MU accomplishes synchronization using two sets of matching
+ registers (Processor A-facing, Processor B-facing).
+
+ MU can work as msi interrupt controller to do doorbell
+
+allOf:
+ - $ref: /schemas/interrupt-controller/msi-controller.yaml#
+
+properties:
+ compatible:
+ enum:
+ - fsl,imx6sx-mu-msi
+ - fsl,imx7ulp-mu-msi
+ - fsl,imx8ulp-mu-msi
+ - fsl,imx8ulp-mu-msi-s4
+
+ reg:
+ minItems: 2
+
+ reg-names:
+ items:
+ - const: a
+ - const: b
+
+ interrupts:
+ maxItems: 1
+
+ clocks:
+ maxItems: 1
+
+ power-domains:
+ maxItems: 2
+
+ power-domain-names:
+ items:
+ - const: a
+ - const: b
+
+ interrupt-controller: true
+
+ msi-controller: true
+
+required:
+ - compatible
+ - reg
+ - interrupts
+ - msi-controller
+ - interrupt-controller
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+ #include <dt-bindings/firmware/imx/rsrc.h>
+
+ lsio_mu12: msi-controller@5d270000 {
+ compatible = "fsl,imx6sx-mu-msi";
+ msi-controller;
+ interrupt-controller;
+ reg = <0x5d270000 0x10000>, /* A side */
+ <0x5d300000 0x10000>; /* B side */
+ reg-names = "a", "b";
+ interrupts = <GIC_SPI 191 IRQ_TYPE_LEVEL_HIGH>;
+ power-domains = <&pd IMX_SC_R_MU_12A>,
+ <&pd IMX_SC_R_MU_12B>;
+ power-domain-names = "a", "b";
+ };
--
2.35.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
@ 2022-07-20 21:30 ` Frank Li
0 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-20 21:30 UTC (permalink / raw)
To: jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, peng.fan,
aisheng.dong, kernel, festevam, linux-imx, kishon,
lorenzo.pieralisi, ntb
imx mu support generate irq by write a register.
provide msi controller support so other driver
can use it by standard msi interface.
Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
.../interrupt-controller/fsl,mu-msi.yaml | 88 +++++++++++++++++++
1 file changed, 88 insertions(+)
create mode 100644 Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
diff --git a/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
new file mode 100644
index 0000000000000..e125294243af3
--- /dev/null
+++ b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
@@ -0,0 +1,88 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/interrupt-controller/fsl,mu-msi.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: NXP i.MX Messaging Unit (MU) work as msi controller
+
+maintainers:
+ - Frank Li <Frank.Li@nxp.com>
+
+description: |
+ The Messaging Unit module enables two processors within the SoC to
+ communicate and coordinate by passing messages (e.g. data, status
+ and control) through the MU interface. The MU also provides the ability
+ for one processor to signal the other processor using interrupts.
+
+ Because the MU manages the messaging between processors, the MU uses
+ different clocks (from each side of the different peripheral buses).
+ Therefore, the MU must synchronize the accesses from one side to the
+ other. The MU accomplishes synchronization using two sets of matching
+ registers (Processor A-facing, Processor B-facing).
+
+ MU can work as msi interrupt controller to do doorbell
+
+allOf:
+ - $ref: /schemas/interrupt-controller/msi-controller.yaml#
+
+properties:
+ compatible:
+ enum:
+ - fsl,imx6sx-mu-msi
+ - fsl,imx7ulp-mu-msi
+ - fsl,imx8ulp-mu-msi
+ - fsl,imx8ulp-mu-msi-s4
+
+ reg:
+ minItems: 2
+
+ reg-names:
+ items:
+ - const: a
+ - const: b
+
+ interrupts:
+ maxItems: 1
+
+ clocks:
+ maxItems: 1
+
+ power-domains:
+ maxItems: 2
+
+ power-domain-names:
+ items:
+ - const: a
+ - const: b
+
+ interrupt-controller: true
+
+ msi-controller: true
+
+required:
+ - compatible
+ - reg
+ - interrupts
+ - msi-controller
+ - interrupt-controller
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+ #include <dt-bindings/firmware/imx/rsrc.h>
+
+ lsio_mu12: msi-controller@5d270000 {
+ compatible = "fsl,imx6sx-mu-msi";
+ msi-controller;
+ interrupt-controller;
+ reg = <0x5d270000 0x10000>, /* A side */
+ <0x5d300000 0x10000>; /* B side */
+ reg-names = "a", "b";
+ interrupts = <GIC_SPI 191 IRQ_TYPE_LEVEL_HIGH>;
+ power-domains = <&pd IMX_SC_R_MU_12A>,
+ <&pd IMX_SC_R_MU_12B>;
+ power-domain-names = "a", "b";
+ };
--
2.35.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [PATCH v3 4/4] pcie: endpoint: pci-epf-vntb: add endpoint MSI support
2022-07-20 21:30 ` Frank Li
@ 2022-07-20 21:30 ` Frank Li
-1 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-20 21:30 UTC (permalink / raw)
To: jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, peng.fan,
aisheng.dong, kernel, festevam, linux-imx, kishon,
lorenzo.pieralisi, ntb
┌───────┐ ┌──────────┐
│ │ │ │
┌─────────────┐ │ │ │ PCI Host │
│ MSI │◄┐ │ │ │ │
│ Controller │ │ │ │ │ │
└─────────────┘ └─┼───────┼──────────┼─BAR0 │
│ PCI │ │ BAR1 │
│ Func │ │ BAR2 │
│ │ │ BAR3 │
│ │ │ BAR4 │
│ ├─────────►│ │
└───────┘ └──────────┘
Linux supports endpoint functions. PCI Host write BAR<n> space like write
to memory. The EP side can't know memory changed by the host driver.
PCI Spec has not defined a standard method to do that. Only define MSI(x)
to let EP notified RC status change.
The basic idea is to trigger an IRQ when PCI RC writes to a memory
address. That's what MSI controller provided. EP drivers just need to
request a platform MSI interrupt, struct msi_msg *msg will pass down a
memory address and data. EP driver will map such memory address to one of
PCI BAR<n>. Host just writes such an address to trigger EP side irq.
Add MSI support for pci-epf-vntb. pci-epf-vntb driver query if system
have MSI controller. Setup doorbell address according to struct msi_msg.
So PCIe host can write this doorbell address to triger EP side's irq.
If no MSI controller exist, fall back to software polling.
Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
drivers/pci/endpoint/functions/pci-epf-vntb.c | 134 +++++++++++++++---
1 file changed, 112 insertions(+), 22 deletions(-)
diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
index 1466dd1904175..cf55fa402049a 100644
--- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
@@ -44,6 +44,7 @@
#include <linux/pci-epc.h>
#include <linux/pci-epf.h>
#include <linux/ntb.h>
+#include <linux/msi.h>
static struct workqueue_struct *kpcintb_workqueue;
@@ -143,6 +144,8 @@ struct epf_ntb {
void __iomem *vpci_mw_addr[MAX_MW];
struct delayed_work cmd_handler;
+
+ int msi_virqbase;
};
#define to_epf_ntb(epf_group) container_of((epf_group), struct epf_ntb, group)
@@ -253,7 +256,7 @@ static void epf_ntb_cmd_handler(struct work_struct *work)
ntb = container_of(work, struct epf_ntb, cmd_handler.work);
- for (i = 1; i < ntb->db_count; i++) {
+ for (i = 1; i < ntb->db_count && !ntb->epf_db_phy; i++) {
if (readl(ntb->epf_db + i * 4)) {
if (readl(ntb->epf_db + i * 4))
ntb->db |= 1 << (i - 1);
@@ -454,11 +457,9 @@ static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb)
ctrl->num_mws = ntb->num_mws;
ntb->spad_size = spad_size;
- ctrl->db_entry_size = 4;
-
for (i = 0; i < ntb->db_count; i++) {
ntb->reg->db_data[i] = 1 + i;
- ntb->reg->db_offset[i] = 0;
+ ntb->reg->db_offset[i] = 4 * i;
}
return 0;
@@ -509,6 +510,28 @@ static int epf_ntb_configure_interrupt(struct epf_ntb *ntb)
return 0;
}
+static int epf_ntb_db_size(struct epf_ntb *ntb)
+{
+ const struct pci_epc_features *epc_features;
+ size_t size = 4 * ntb->db_count;
+ u32 align;
+
+ epc_features = pci_epc_get_features(ntb->epf->epc,
+ ntb->epf->func_no,
+ ntb->epf->vfunc_no);
+ align = epc_features->align;
+
+ if (size < 128)
+ size = 128;
+
+ if (align)
+ size = ALIGN(size, align);
+ else
+ size = roundup_pow_of_two(size);
+
+ return size;
+}
+
/**
* epf_ntb_db_bar_init() - Configure Doorbell window BARs
* @ntb: NTB device that facilitates communication between HOST and vHOST
@@ -520,35 +543,33 @@ static int epf_ntb_db_bar_init(struct epf_ntb *ntb)
struct device *dev = &ntb->epf->dev;
int ret;
struct pci_epf_bar *epf_bar;
- void __iomem *mw_addr;
+ void __iomem *mw_addr = NULL;
enum pci_barno barno;
- size_t size = 4 * ntb->db_count;
+ size_t size;
epc_features = pci_epc_get_features(ntb->epf->epc,
ntb->epf->func_no,
ntb->epf->vfunc_no);
- align = epc_features->align;
- if (size < 128)
- size = 128;
-
- if (align)
- size = ALIGN(size, align);
- else
- size = roundup_pow_of_two(size);
+ size = epf_ntb_db_size(ntb);
barno = ntb->epf_ntb_bar[BAR_DB];
+ epf_bar = &ntb->epf->bar[barno];
- mw_addr = pci_epf_alloc_space(ntb->epf, size, barno, align, 0);
- if (!mw_addr) {
- dev_err(dev, "Failed to allocate OB address\n");
- return -ENOMEM;
+ if (!ntb->epf_db_phy) {
+ mw_addr = pci_epf_alloc_space(ntb->epf, size, barno, align, 0);
+ if (!mw_addr) {
+ dev_err(dev, "Failed to allocate OB address\n");
+ return -ENOMEM;
+ }
+ } else {
+ epf_bar->phys_addr = ntb->epf_db_phy;
+ epf_bar->barno = barno;
+ epf_bar->size = size;
}
ntb->epf_db = mw_addr;
- epf_bar = &ntb->epf->bar[barno];
-
ret = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no, ntb->epf->vfunc_no, epf_bar);
if (ret) {
dev_err(dev, "Doorbell BAR set failed\n");
@@ -704,6 +725,74 @@ static int epf_ntb_init_epc_bar(struct epf_ntb *ntb)
return 0;
}
+static void epf_ntb_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
+{
+ struct epf_ntb *ntb = dev_get_drvdata(desc->dev);
+ struct epf_ntb_ctrl *reg = ntb->reg;
+ int size = epf_ntb_db_size(ntb);
+ u64 addr;
+
+ addr = msg->address_hi;
+ addr <<= 32;
+ addr |= msg->address_lo;
+
+ reg->db_data[desc->msi_index] = msg->data;
+
+ if (desc->msi_index == 0)
+ ntb->epf_db_phy = round_down(addr, size);
+
+ reg->db_offset[desc->msi_index] = addr - ntb->epf_db_phy;
+}
+
+static irqreturn_t epf_ntb_interrupt_handler(int irq, void *data)
+{
+ struct epf_ntb *ntb = data;
+ int index;
+
+ index = irq - ntb->msi_virqbase;
+ ntb->db |= 1 << (index - 1);
+ ntb_db_event(&ntb->ntb, index);
+
+ return IRQ_HANDLED;
+}
+
+static void epf_ntb_epc_msi_init(struct epf_ntb *ntb)
+{
+ struct device *dev = &ntb->epf->dev;
+ struct irq_domain *domain;
+ int virq;
+ int ret;
+ int i;
+
+ domain = dev_get_msi_domain(ntb->epf->epc->dev.parent);
+ if (!domain)
+ return;
+
+ dev_set_msi_domain(dev, domain);
+
+ if (platform_msi_domain_alloc_irqs(&ntb->epf->dev,
+ ntb->db_count,
+ epf_ntb_write_msi_msg)) {
+ dev_info(dev, "Can't allocate MSI, fall back to poll mode\n");
+ return;
+ }
+
+ dev_info(dev, "vntb use MSI as doorbell\n");
+
+ for (i = 0; i < ntb->db_count; i++) {
+ virq = msi_get_virq(dev, i);
+ ret = devm_request_irq(dev, virq,
+ epf_ntb_interrupt_handler, 0,
+ "ntb", ntb);
+
+ if (ret)
+ dev_err(dev, "devm_request_irq() failure\n");
+
+ if (!i)
+ ntb->msi_virqbase = virq;
+ }
+}
+
/**
* epf_ntb_epc_init() - Initialize NTB interface
* @ntb: NTB device that facilitates communication between HOST and vHOST2
@@ -1299,14 +1388,15 @@ static int epf_ntb_bind(struct pci_epf *epf)
goto err_bar_alloc;
}
+ epf_set_drvdata(epf, ntb);
+ epf_ntb_epc_msi_init(ntb);
+
ret = epf_ntb_epc_init(ntb);
if (ret) {
dev_err(dev, "Failed to initialize EPC\n");
goto err_bar_alloc;
}
- epf_set_drvdata(epf, ntb);
-
pci_space[0] = (ntb->vntb_pid << 16) | ntb->vntb_vid;
pci_vntb_table[0].vendor = ntb->vntb_vid;
pci_vntb_table[0].device = ntb->vntb_pid;
--
2.35.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [PATCH v3 4/4] pcie: endpoint: pci-epf-vntb: add endpoint MSI support
@ 2022-07-20 21:30 ` Frank Li
0 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-20 21:30 UTC (permalink / raw)
To: jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, peng.fan,
aisheng.dong, kernel, festevam, linux-imx, kishon,
lorenzo.pieralisi, ntb
┌───────┐ ┌──────────┐
│ │ │ │
┌─────────────┐ │ │ │ PCI Host │
│ MSI │◄┐ │ │ │ │
│ Controller │ │ │ │ │ │
└─────────────┘ └─┼───────┼──────────┼─BAR0 │
│ PCI │ │ BAR1 │
│ Func │ │ BAR2 │
│ │ │ BAR3 │
│ │ │ BAR4 │
│ ├─────────►│ │
└───────┘ └──────────┘
Linux supports endpoint functions. PCI Host write BAR<n> space like write
to memory. The EP side can't know memory changed by the host driver.
PCI Spec has not defined a standard method to do that. Only define MSI(x)
to let EP notified RC status change.
The basic idea is to trigger an IRQ when PCI RC writes to a memory
address. That's what MSI controller provided. EP drivers just need to
request a platform MSI interrupt, struct msi_msg *msg will pass down a
memory address and data. EP driver will map such memory address to one of
PCI BAR<n>. Host just writes such an address to trigger EP side irq.
Add MSI support for pci-epf-vntb. pci-epf-vntb driver query if system
have MSI controller. Setup doorbell address according to struct msi_msg.
So PCIe host can write this doorbell address to triger EP side's irq.
If no MSI controller exist, fall back to software polling.
Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
drivers/pci/endpoint/functions/pci-epf-vntb.c | 134 +++++++++++++++---
1 file changed, 112 insertions(+), 22 deletions(-)
diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
index 1466dd1904175..cf55fa402049a 100644
--- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
@@ -44,6 +44,7 @@
#include <linux/pci-epc.h>
#include <linux/pci-epf.h>
#include <linux/ntb.h>
+#include <linux/msi.h>
static struct workqueue_struct *kpcintb_workqueue;
@@ -143,6 +144,8 @@ struct epf_ntb {
void __iomem *vpci_mw_addr[MAX_MW];
struct delayed_work cmd_handler;
+
+ int msi_virqbase;
};
#define to_epf_ntb(epf_group) container_of((epf_group), struct epf_ntb, group)
@@ -253,7 +256,7 @@ static void epf_ntb_cmd_handler(struct work_struct *work)
ntb = container_of(work, struct epf_ntb, cmd_handler.work);
- for (i = 1; i < ntb->db_count; i++) {
+ for (i = 1; i < ntb->db_count && !ntb->epf_db_phy; i++) {
if (readl(ntb->epf_db + i * 4)) {
if (readl(ntb->epf_db + i * 4))
ntb->db |= 1 << (i - 1);
@@ -454,11 +457,9 @@ static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb)
ctrl->num_mws = ntb->num_mws;
ntb->spad_size = spad_size;
- ctrl->db_entry_size = 4;
-
for (i = 0; i < ntb->db_count; i++) {
ntb->reg->db_data[i] = 1 + i;
- ntb->reg->db_offset[i] = 0;
+ ntb->reg->db_offset[i] = 4 * i;
}
return 0;
@@ -509,6 +510,28 @@ static int epf_ntb_configure_interrupt(struct epf_ntb *ntb)
return 0;
}
+static int epf_ntb_db_size(struct epf_ntb *ntb)
+{
+ const struct pci_epc_features *epc_features;
+ size_t size = 4 * ntb->db_count;
+ u32 align;
+
+ epc_features = pci_epc_get_features(ntb->epf->epc,
+ ntb->epf->func_no,
+ ntb->epf->vfunc_no);
+ align = epc_features->align;
+
+ if (size < 128)
+ size = 128;
+
+ if (align)
+ size = ALIGN(size, align);
+ else
+ size = roundup_pow_of_two(size);
+
+ return size;
+}
+
/**
* epf_ntb_db_bar_init() - Configure Doorbell window BARs
* @ntb: NTB device that facilitates communication between HOST and vHOST
@@ -520,35 +543,33 @@ static int epf_ntb_db_bar_init(struct epf_ntb *ntb)
struct device *dev = &ntb->epf->dev;
int ret;
struct pci_epf_bar *epf_bar;
- void __iomem *mw_addr;
+ void __iomem *mw_addr = NULL;
enum pci_barno barno;
- size_t size = 4 * ntb->db_count;
+ size_t size;
epc_features = pci_epc_get_features(ntb->epf->epc,
ntb->epf->func_no,
ntb->epf->vfunc_no);
- align = epc_features->align;
- if (size < 128)
- size = 128;
-
- if (align)
- size = ALIGN(size, align);
- else
- size = roundup_pow_of_two(size);
+ size = epf_ntb_db_size(ntb);
barno = ntb->epf_ntb_bar[BAR_DB];
+ epf_bar = &ntb->epf->bar[barno];
- mw_addr = pci_epf_alloc_space(ntb->epf, size, barno, align, 0);
- if (!mw_addr) {
- dev_err(dev, "Failed to allocate OB address\n");
- return -ENOMEM;
+ if (!ntb->epf_db_phy) {
+ mw_addr = pci_epf_alloc_space(ntb->epf, size, barno, align, 0);
+ if (!mw_addr) {
+ dev_err(dev, "Failed to allocate OB address\n");
+ return -ENOMEM;
+ }
+ } else {
+ epf_bar->phys_addr = ntb->epf_db_phy;
+ epf_bar->barno = barno;
+ epf_bar->size = size;
}
ntb->epf_db = mw_addr;
- epf_bar = &ntb->epf->bar[barno];
-
ret = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no, ntb->epf->vfunc_no, epf_bar);
if (ret) {
dev_err(dev, "Doorbell BAR set failed\n");
@@ -704,6 +725,74 @@ static int epf_ntb_init_epc_bar(struct epf_ntb *ntb)
return 0;
}
+static void epf_ntb_write_msi_msg(struct msi_desc *desc, struct msi_msg *msg)
+{
+ struct epf_ntb *ntb = dev_get_drvdata(desc->dev);
+ struct epf_ntb_ctrl *reg = ntb->reg;
+ int size = epf_ntb_db_size(ntb);
+ u64 addr;
+
+ addr = msg->address_hi;
+ addr <<= 32;
+ addr |= msg->address_lo;
+
+ reg->db_data[desc->msi_index] = msg->data;
+
+ if (desc->msi_index == 0)
+ ntb->epf_db_phy = round_down(addr, size);
+
+ reg->db_offset[desc->msi_index] = addr - ntb->epf_db_phy;
+}
+
+static irqreturn_t epf_ntb_interrupt_handler(int irq, void *data)
+{
+ struct epf_ntb *ntb = data;
+ int index;
+
+ index = irq - ntb->msi_virqbase;
+ ntb->db |= 1 << (index - 1);
+ ntb_db_event(&ntb->ntb, index);
+
+ return IRQ_HANDLED;
+}
+
+static void epf_ntb_epc_msi_init(struct epf_ntb *ntb)
+{
+ struct device *dev = &ntb->epf->dev;
+ struct irq_domain *domain;
+ int virq;
+ int ret;
+ int i;
+
+ domain = dev_get_msi_domain(ntb->epf->epc->dev.parent);
+ if (!domain)
+ return;
+
+ dev_set_msi_domain(dev, domain);
+
+ if (platform_msi_domain_alloc_irqs(&ntb->epf->dev,
+ ntb->db_count,
+ epf_ntb_write_msi_msg)) {
+ dev_info(dev, "Can't allocate MSI, fall back to poll mode\n");
+ return;
+ }
+
+ dev_info(dev, "vntb use MSI as doorbell\n");
+
+ for (i = 0; i < ntb->db_count; i++) {
+ virq = msi_get_virq(dev, i);
+ ret = devm_request_irq(dev, virq,
+ epf_ntb_interrupt_handler, 0,
+ "ntb", ntb);
+
+ if (ret)
+ dev_err(dev, "devm_request_irq() failure\n");
+
+ if (!i)
+ ntb->msi_virqbase = virq;
+ }
+}
+
/**
* epf_ntb_epc_init() - Initialize NTB interface
* @ntb: NTB device that facilitates communication between HOST and vHOST2
@@ -1299,14 +1388,15 @@ static int epf_ntb_bind(struct pci_epf *epf)
goto err_bar_alloc;
}
+ epf_set_drvdata(epf, ntb);
+ epf_ntb_epc_msi_init(ntb);
+
ret = epf_ntb_epc_init(ntb);
if (ret) {
dev_err(dev, "Failed to initialize EPC\n");
goto err_bar_alloc;
}
- epf_set_drvdata(epf, ntb);
-
pci_space[0] = (ntb->vntb_pid << 16) | ntb->vntb_vid;
pci_vntb_table[0].vendor = ntb->vntb_vid;
pci_vntb_table[0].device = ntb->vntb_pid;
--
2.35.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
2022-07-20 21:30 ` Frank Li
@ 2022-07-21 7:57 ` Marc Zyngier
-1 siblings, 0 replies; 50+ messages in thread
From: Marc Zyngier @ 2022-07-21 7:57 UTC (permalink / raw)
To: Frank Li
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, peng.fan, aisheng.dong, kernel, festevam, linux-imx,
kishon, lorenzo.pieralisi, ntb
On Wed, 20 Jul 2022 22:30:34 +0100,
Frank Li <Frank.Li@nxp.com> wrote:
>
> MU support generate irq by write data to a register.
> This patch make mu worked as msi controller.
> So MU can do doorbell by using standard msi api.
>
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---
> drivers/irqchip/Kconfig | 7 +
> drivers/irqchip/Makefile | 1 +
> drivers/irqchip/irq-imx-mu-msi.c | 462 +++++++++++++++++++++++++++++++
> 3 files changed, 470 insertions(+)
> create mode 100644 drivers/irqchip/irq-imx-mu-msi.c
>
> diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
> index 5e4e50122777d..4599471d880c0 100644
> --- a/drivers/irqchip/Kconfig
> +++ b/drivers/irqchip/Kconfig
> @@ -470,6 +470,13 @@ config IMX_INTMUX
> help
> Support for the i.MX INTMUX interrupt multiplexer.
>
> +config IMX_MU_MSI
> + bool "i.MX MU work as MSI controller"
> + default y if ARCH_MXC
> + select IRQ_DOMAIN
> + help
> + MU work as MSI controller to do general doorbell
> +
> config LS1X_IRQ
> bool "Loongson-1 Interrupt Controller"
> depends on MACH_LOONGSON32
> diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
> index 5d8e21d3dc6d8..870423746c783 100644
> --- a/drivers/irqchip/Makefile
> +++ b/drivers/irqchip/Makefile
> @@ -98,6 +98,7 @@ obj-$(CONFIG_RISCV_INTC) += irq-riscv-intc.o
> obj-$(CONFIG_SIFIVE_PLIC) += irq-sifive-plic.o
> obj-$(CONFIG_IMX_IRQSTEER) += irq-imx-irqsteer.o
> obj-$(CONFIG_IMX_INTMUX) += irq-imx-intmux.o
> +obj-$(CONFIG_IMX_MU_MSI) += irq-imx-mu-msi.o
> obj-$(CONFIG_MADERA_IRQ) += irq-madera.o
> obj-$(CONFIG_LS1X_IRQ) += irq-ls1x.o
> obj-$(CONFIG_TI_SCI_INTR_IRQCHIP) += irq-ti-sci-intr.o
> diff --git a/drivers/irqchip/irq-imx-mu-msi.c b/drivers/irqchip/irq-imx-mu-msi.c
> new file mode 100644
> index 0000000000000..8277dba857759
> --- /dev/null
> +++ b/drivers/irqchip/irq-imx-mu-msi.c
> @@ -0,0 +1,462 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * NXP MU worked as MSI controller
> + *
> + * Copyright (c) 2018 Pengutronix, Oleksij Rempel <o.rempel@pengutronix.de>
> + * Copyright 2022 NXP
> + * Frank Li <Frank.Li@nxp.com>
> + * Peng Fan <peng.fan@nxp.com>
> + *
> + * Based on drivers/mailbox/imx-mailbox.c
> + */
> +#include <linux/clk.h>
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/msi.h>
> +#include <linux/interrupt.h>
> +#include <linux/irq.h>
> +#include <linux/irqchip/chained_irq.h>
> +#include <linux/irqchip.h>
> +#include <linux/irqdomain.h>
> +#include <linux/of_irq.h>
> +#include <linux/of_pci.h>
> +#include <linux/of_platform.h>
> +#include <linux/spinlock.h>
> +#include <linux/dma-iommu.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/pm_domain.h>
> +
> +
> +#define IMX_MU_CHANS 4
> +
> +enum imx_mu_xcr {
> + IMX_MU_GIER,
> + IMX_MU_GCR,
> + IMX_MU_TCR,
> + IMX_MU_RCR,
> + IMX_MU_xCR_MAX,
> +};
> +
> +enum imx_mu_xsr {
> + IMX_MU_SR,
> + IMX_MU_GSR,
> + IMX_MU_TSR,
> + IMX_MU_RSR,
> +};
> +
> +enum imx_mu_type {
> + IMX_MU_V1 = BIT(0),
> + IMX_MU_V2 = BIT(1),
> + IMX_MU_V2_S4 = BIT(15),
> +};
> +
> +/* Receive Interrupt Enable */
> +#define IMX_MU_xCR_RIEn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24 + (3 - (x))))
> +#define IMX_MU_xSR_RFn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24 + (3 - (x))))
> +
> +struct imx_mu_dcfg {
> + enum imx_mu_type type;
> + u32 xTR; /* Transmit Register0 */
> + u32 xRR; /* Receive Register0 */
> + u32 xSR[4]; /* Status Registers */
> + u32 xCR[4]; /* Control Registers */
> +};
> +
> +struct imx_mu_msi {
> + spinlock_t lock;
> + struct platform_device *pdev;
> + struct irq_domain *parent;
> + struct irq_domain *msi_domain;
> + void __iomem *regs;
> + phys_addr_t msiir_addr;
> + const struct imx_mu_dcfg *cfg;
> + unsigned long used;
> + u32 gic_irq;
> + struct clk *clk;
> + struct device *pd_a;
> + struct device *pd_b;
> + struct device_link *pd_link_a;
> + struct device_link *pd_link_b;
> +};
> +
> +static void imx_mu_write(struct imx_mu_msi *msi_data, u32 val, u32 offs)
> +{
> + iowrite32(val, msi_data->regs + offs);
> +}
> +
> +static u32 imx_mu_read(struct imx_mu_msi *msi_data, u32 offs)
> +{
> + return ioread32(msi_data->regs + offs);
> +}
> +
> +static u32 imx_mu_xcr_rmw(struct imx_mu_msi *msi_data, enum imx_mu_xcr type, u32 set, u32 clr)
> +{
> + unsigned long flags;
> + u32 val;
> +
> + spin_lock_irqsave(&msi_data->lock, flags);
> + val = imx_mu_read(msi_data, msi_data->cfg->xCR[type]);
> + val &= ~clr;
> + val |= set;
> + imx_mu_write(msi_data, val, msi_data->cfg->xCR[type]);
> + spin_unlock_irqrestore(&msi_data->lock, flags);
> +
> + return val;
> +}
> +
> +static void imx_mu_msi_mask_irq(struct irq_data *data)
> +{
> + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data->parent_data);
Urgh... No. Please don't go poking into the *parent* stuff. Implement
the masking at the parent level, and use irq_chip_mask_parent() for
this level, unless you can explain why you can't do otherwise.
> +
> + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, 0, IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq));
How about making this readable and move the dereference inside the macro?
> +}
> +
> +static void imx_mu_msi_unmask_irq(struct irq_data *data)
> +{
> + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data->parent_data);
> +
> + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq), 0);
> +}
> +
> +static struct irq_chip imx_mu_msi_irq_chip = {
> + .name = "MU-MSI",
> + .irq_mask = imx_mu_msi_mask_irq,
> + .irq_unmask = imx_mu_msi_unmask_irq,
> +};
> +
> +static struct msi_domain_ops its_pmsi_ops = {
> +};
What does this have to do with the ITS?
> +
> +static struct msi_domain_info imx_mu_msi_domain_info = {
> + .flags = (MSI_FLAG_USE_DEF_DOM_OPS |
> + MSI_FLAG_USE_DEF_CHIP_OPS |
> + MSI_FLAG_PCI_MSIX),
What does PCI_MSIX mean in this context? I really wish you used
copy/paste a bit more carefully.
> + .ops = &its_pmsi_ops,
> + .chip = &imx_mu_msi_irq_chip,
> +};
> +
> +static void imx_mu_msi_compose_msg(struct irq_data *data, struct msi_msg *msg)
> +{
> + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
> +
> + msg->address_hi = upper_32_bits(msi_data->msiir_addr);
> + msg->address_lo = lower_32_bits(msi_data->msiir_addr + 4 * data->hwirq);
This is a horrible way of writing this. you should compute the address
first, and then apply the address split.
> + msg->data = data->hwirq;
> +
> + iommu_dma_compose_msi_msg(irq_data_get_msi_desc(data), msg);
> +}
> +
> +static int imx_mu_msi_set_affinity(struct irq_data *irq_data,
> + const struct cpumask *mask, bool force)
> +
> +{
> + return IRQ_SET_MASK_OK;
> +}
Err... What effect does this have if you don't do anything?
> +
> +static struct irq_chip imx_mu_msi_parent_chip = {
> + .name = "MU",
> + .irq_compose_msi_msg = imx_mu_msi_compose_msg,
> + .irq_set_affinity = imx_mu_msi_set_affinity,
> +};
> +
> +static int imx_mu_msi_domain_irq_alloc(struct irq_domain *domain,
> + unsigned int virq,
> + unsigned int nr_irqs,
> + void *args)
> +{
> + struct imx_mu_msi *msi_data = domain->host_data;
> + msi_alloc_info_t *info = args;
> + int pos, err = 0;
> +
> + WARN_ON(nr_irqs != 1);
> +
> + spin_lock(&msi_data->lock);
Interrupt fires after lock acquisition, handler masks the interrupt.
Deadlock.
> + pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
> + if (pos < IMX_MU_CHANS)
> + __set_bit(pos, &msi_data->used);
> + else
> + err = -ENOSPC;
> + spin_unlock(&msi_data->lock);
> +
> + if (err)
> + return err;
> +
> + err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr + pos * 4);
Does this HW even have an IOMMU?
> + if (err)
> + return err;
> +
> + irq_domain_set_info(domain, virq, pos,
> + &imx_mu_msi_parent_chip, msi_data,
> + handle_simple_irq, NULL, NULL);
This is an edge interrupt. Please handle it like one.
> + return 0;
> +}
> +
> +static void imx_mu_msi_domain_irq_free(struct irq_domain *domain,
> + unsigned int virq, unsigned int nr_irqs)
> +{
> + struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(d);
> +
> + spin_lock(&msi_data->lock);
Same problem.
> + __clear_bit(d->hwirq, &msi_data->used);
> + spin_unlock(&msi_data->lock);
> +}
> +
> +static const struct irq_domain_ops imx_mu_msi_domain_ops = {
> + .alloc = imx_mu_msi_domain_irq_alloc,
> + .free = imx_mu_msi_domain_irq_free,
> +};
> +
> +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> +{
> + struct imx_mu_msi *msi_data = irq_desc_get_handler_data(desc);
> + u32 status;
> + int i;
> +
> + status = imx_mu_read(msi_data, msi_data->cfg->xSR[IMX_MU_RSR]);
> +
> + chained_irq_enter(irq_desc_get_chip(desc), desc);
> + for (i = 0; i < IMX_MU_CHANS; i++) {
> + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> + generic_handle_domain_irq(msi_data->parent, i);
Why the parent? You must start at the top of the hierarchy.
> + }
> + }
> + chained_irq_exit(irq_desc_get_chip(desc), desc);
If your MSIs are a chained interrupt, why do you even provide an
affinity setting callback?
> +}
> +
> +static int imx_mu_msi_domains_init(struct imx_mu_msi *msi_data)
> +{
> + /* Initialize MSI domain parent */
> + msi_data->parent = irq_domain_add_linear(dev_of_node(&msi_data->pdev->dev),
> + IMX_MU_CHANS,
> + &imx_mu_msi_domain_ops,
> + msi_data);
Use irq_domain_create_linear() instead.
> + if (!msi_data->parent) {
> + dev_err(&msi_data->pdev->dev, "failed to create IRQ domain\n");
> + return -ENOMEM;
> + }
> +
> + msi_data->msi_domain = platform_msi_create_irq_domain(
> + of_node_to_fwnode(msi_data->pdev->dev.of_node),
> + &imx_mu_msi_domain_info,
> + msi_data->parent);
And you don't get an error due to the fact that you use the same
fwnode for both domains without overriding the domain bus token?
> +
> + if (!msi_data->msi_domain) {
> + dev_err(&msi_data->pdev->dev, "failed to create MSI domain\n");
> + irq_domain_remove(msi_data->parent);
> + return -ENOMEM;
> + }
> +
> + irq_domain_set_pm_device(msi_data->parent, &msi_data->pdev->dev);
I really wonder why you both implementing a MSI domain if you are
always bypassing it for absolutely everything... This completely
nullifies the effect of this call, as no interrupt request will ever
trigger the PM.
> +
> + return 0;
> +}
> +
> +/* Register offset of different version MU IP */
> +static const struct imx_mu_dcfg imx_mu_cfg_imx6sx = {
> + .xTR = 0x0,
> + .xRR = 0x10,
> + .xSR = {0x20, 0x20, 0x20, 0x20},
> + .xCR = {0x24, 0x24, 0x24, 0x24},
> +};
> +
> +static const struct imx_mu_dcfg imx_mu_cfg_imx7ulp = {
> + .xTR = 0x20,
> + .xRR = 0x40,
> + .xSR = {0x60, 0x60, 0x60, 0x60},
> + .xCR = {0x64, 0x64, 0x64, 0x64},
> +};
> +
> +static const struct imx_mu_dcfg imx_mu_cfg_imx8ulp = {
> + .type = IMX_MU_V2,
> + .xTR = 0x200,
> + .xRR = 0x280,
> + .xSR = {0xC, 0x118, 0x124, 0x12C},
> + .xCR = {0x110, 0x114, 0x120, 0x128},
> +};
> +
> +static const struct imx_mu_dcfg imx_mu_cfg_imx8ulp_s4 = {
> +
> + .type = IMX_MU_V2 | IMX_MU_V2_S4,
> + .xTR = 0x200,
> + .xRR = 0x280,
> + .xSR = {0xC, 0x118, 0x124, 0x12C},
> + .xCR = {0x110, 0x114, 0x120, 0x128},
> +};
> +
> +static int __init imx_mu_of_init(struct device_node *dn,
> + struct device_node *parent,
> + const struct imx_mu_dcfg *cfg)
> +{
> + struct platform_device *pdev = of_find_device_by_node(dn);
> + struct imx_mu_msi *msi_data, *priv;
> + struct resource *res;
> + struct device *dev;
> + int ret;
> +
> + if (!pdev)
> + return -ENODEV;
> +
> + dev = &pdev->dev;
> +
> + priv = msi_data = devm_kzalloc(&pdev->dev, sizeof(*msi_data), GFP_KERNEL);
> + if (!msi_data)
> + return -ENOMEM;
> +
> + msi_data->cfg = cfg;
> +
> + msi_data->regs = devm_platform_ioremap_resource_byname(pdev, "a");
> + if (IS_ERR(msi_data->regs)) {
> + dev_err(&pdev->dev, "failed to initialize 'regs'\n");
> + return PTR_ERR(msi_data->regs);
> + }
> +
> + res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "b");
> + if (!res)
> + return -EIO;
> +
> + msi_data->msiir_addr = res->start + msi_data->cfg->xTR;
> +
> + msi_data->pdev = pdev;
> +
> + msi_data->gic_irq = platform_get_irq(msi_data->pdev, 0);
> + if (msi_data->gic_irq <= 0)
> + return -ENODEV;
> +
> + platform_set_drvdata(pdev, msi_data);
> +
> + msi_data->clk = devm_clk_get(dev, NULL);
> + if (IS_ERR(msi_data->clk)) {
> + if (PTR_ERR(msi_data->clk) != -ENOENT)
> + return PTR_ERR(msi_data->clk);
> +
> + msi_data->clk = NULL;
> + }
> +
> + ret = clk_prepare_enable(msi_data->clk);
> + if (ret) {
> + dev_err(dev, "Failed to enable clock\n");
> + return ret;
> + }
> +
> + priv->pd_a = dev_pm_domain_attach_by_name(dev, "a");
> + if (IS_ERR(priv->pd_a))
> + return PTR_ERR(priv->pd_a);
> +
> + priv->pd_link_a = device_link_add(dev, priv->pd_a,
> + DL_FLAG_STATELESS |
> + DL_FLAG_PM_RUNTIME |
> + DL_FLAG_RPM_ACTIVE);
> +
> + if (!priv->pd_link_a) {
> + dev_err(dev, "Failed to add device_link to mu a.\n");
> + return -EINVAL;
> + }
> +
> + priv->pd_b = dev_pm_domain_attach_by_name(dev, "b");
> + if (IS_ERR(priv->pd_b))
> + return PTR_ERR(priv->pd_b);
> +
> + priv->pd_link_b = device_link_add(dev, priv->pd_b,
> + DL_FLAG_STATELESS |
> + DL_FLAG_PM_RUNTIME |
> + DL_FLAG_RPM_ACTIVE);
> +
> + if (!priv->pd_link_b) {
> + dev_err(dev, "Failed to add device_link to mu a.\n");
> + return -EINVAL;
> + }
> +
> + ret = imx_mu_msi_domains_init(msi_data);
> + if (ret)
> + return ret;
> +
> + irq_set_chained_handler_and_data(msi_data->gic_irq,
> + imx_mu_msi_irq_handler,
> + msi_data);
> +
> + pm_runtime_enable(dev);
> +
> + ret = pm_runtime_get_sync(dev);
> + if (ret < 0) {
> + pm_runtime_put_noidle(dev);
> + goto disable_runtime_pm;
> + }
> +
> + ret = pm_runtime_put_sync(dev);
> + if (ret < 0)
> + goto disable_runtime_pm;
> +
> + clk_disable_unprepare(msi_data->clk);
Why do you need to do all this dance?
> +
> + return 0;
> +
> +disable_runtime_pm:
> + pm_runtime_disable(dev);
> + clk_disable_unprepare(msi_data->clk);
> +
> + return ret;
> +}
> +
> +static int __maybe_unused imx_mu_runtime_suspend(struct device *dev)
> +{
> + struct imx_mu_msi *priv = dev_get_drvdata(dev);
> +
> + clk_disable_unprepare(priv->clk);
> +
> + return 0;
> +}
> +
> +static int __maybe_unused imx_mu_runtime_resume(struct device *dev)
> +{
> + struct imx_mu_msi *priv = dev_get_drvdata(dev);
> + int ret;
> +
> + ret = clk_prepare_enable(priv->clk);
> + if (ret)
> + dev_err(dev, "failed to enable clock\n");
> +
> + return ret;
> +}
> +
> +static const struct dev_pm_ops imx_mu_pm_ops = {
> + SET_RUNTIME_PM_OPS(imx_mu_runtime_suspend,
> + imx_mu_runtime_resume, NULL)
> +};
> +
> +static int __init imx_mu_imx7ulp_of_init(struct device_node *dn,
> + struct device_node *parent)
> +{
> + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx7ulp);
> +}
> +
> +static int __init imx_mu_imx6sx_of_init(struct device_node *dn,
> + struct device_node *parent)
> +{
> + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx6sx);
> +}
> +
> +static int __init imx_mu_imx8ulp_of_init(struct device_node *dn,
> + struct device_node *parent)
> +{
> + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx8ulp);
> +}
> +
> +static int __init imx_mu_imx8ulp_s4_of_init(struct device_node *dn,
> + struct device_node *parent)
> +{
> + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx8ulp_s4);
> +}
> +
> +IRQCHIP_PLATFORM_DRIVER_BEGIN(imx_mu_msi)
> +IRQCHIP_MATCH("fsl,imx7ulp-mu-msi", imx_mu_imx7ulp_of_init)
> +IRQCHIP_MATCH("fsl,imx6sx-mu-msi", imx_mu_imx6sx_of_init)
> +IRQCHIP_MATCH("fsl,imx8ulp-mu-msi", imx_mu_imx8ulp_of_init)
> +IRQCHIP_MATCH("fsl,imx8ulp-mu-msi-s4", imx_mu_imx8ulp_s4_of_init)
> +IRQCHIP_PLATFORM_DRIVER_END(imx_mu_msi, .pm = &imx_mu_pm_ops)
> +
> +
> +MODULE_AUTHOR("Frank Li <Frank.Li@nxp.com>");
> +MODULE_DESCRIPTION("Freescale MU work as MSI controller driver");
> +MODULE_LICENSE("GPL");
M.
--
Without deviation from the norm, progress is not possible.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
@ 2022-07-21 7:57 ` Marc Zyngier
0 siblings, 0 replies; 50+ messages in thread
From: Marc Zyngier @ 2022-07-21 7:57 UTC (permalink / raw)
To: Frank Li
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, peng.fan, aisheng.dong, kernel, festevam, linux-imx,
kishon, lorenzo.pieralisi, ntb
On Wed, 20 Jul 2022 22:30:34 +0100,
Frank Li <Frank.Li@nxp.com> wrote:
>
> MU support generate irq by write data to a register.
> This patch make mu worked as msi controller.
> So MU can do doorbell by using standard msi api.
>
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---
> drivers/irqchip/Kconfig | 7 +
> drivers/irqchip/Makefile | 1 +
> drivers/irqchip/irq-imx-mu-msi.c | 462 +++++++++++++++++++++++++++++++
> 3 files changed, 470 insertions(+)
> create mode 100644 drivers/irqchip/irq-imx-mu-msi.c
>
> diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
> index 5e4e50122777d..4599471d880c0 100644
> --- a/drivers/irqchip/Kconfig
> +++ b/drivers/irqchip/Kconfig
> @@ -470,6 +470,13 @@ config IMX_INTMUX
> help
> Support for the i.MX INTMUX interrupt multiplexer.
>
> +config IMX_MU_MSI
> + bool "i.MX MU work as MSI controller"
> + default y if ARCH_MXC
> + select IRQ_DOMAIN
> + help
> + MU work as MSI controller to do general doorbell
> +
> config LS1X_IRQ
> bool "Loongson-1 Interrupt Controller"
> depends on MACH_LOONGSON32
> diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
> index 5d8e21d3dc6d8..870423746c783 100644
> --- a/drivers/irqchip/Makefile
> +++ b/drivers/irqchip/Makefile
> @@ -98,6 +98,7 @@ obj-$(CONFIG_RISCV_INTC) += irq-riscv-intc.o
> obj-$(CONFIG_SIFIVE_PLIC) += irq-sifive-plic.o
> obj-$(CONFIG_IMX_IRQSTEER) += irq-imx-irqsteer.o
> obj-$(CONFIG_IMX_INTMUX) += irq-imx-intmux.o
> +obj-$(CONFIG_IMX_MU_MSI) += irq-imx-mu-msi.o
> obj-$(CONFIG_MADERA_IRQ) += irq-madera.o
> obj-$(CONFIG_LS1X_IRQ) += irq-ls1x.o
> obj-$(CONFIG_TI_SCI_INTR_IRQCHIP) += irq-ti-sci-intr.o
> diff --git a/drivers/irqchip/irq-imx-mu-msi.c b/drivers/irqchip/irq-imx-mu-msi.c
> new file mode 100644
> index 0000000000000..8277dba857759
> --- /dev/null
> +++ b/drivers/irqchip/irq-imx-mu-msi.c
> @@ -0,0 +1,462 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * NXP MU worked as MSI controller
> + *
> + * Copyright (c) 2018 Pengutronix, Oleksij Rempel <o.rempel@pengutronix.de>
> + * Copyright 2022 NXP
> + * Frank Li <Frank.Li@nxp.com>
> + * Peng Fan <peng.fan@nxp.com>
> + *
> + * Based on drivers/mailbox/imx-mailbox.c
> + */
> +#include <linux/clk.h>
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/msi.h>
> +#include <linux/interrupt.h>
> +#include <linux/irq.h>
> +#include <linux/irqchip/chained_irq.h>
> +#include <linux/irqchip.h>
> +#include <linux/irqdomain.h>
> +#include <linux/of_irq.h>
> +#include <linux/of_pci.h>
> +#include <linux/of_platform.h>
> +#include <linux/spinlock.h>
> +#include <linux/dma-iommu.h>
> +#include <linux/pm_runtime.h>
> +#include <linux/pm_domain.h>
> +
> +
> +#define IMX_MU_CHANS 4
> +
> +enum imx_mu_xcr {
> + IMX_MU_GIER,
> + IMX_MU_GCR,
> + IMX_MU_TCR,
> + IMX_MU_RCR,
> + IMX_MU_xCR_MAX,
> +};
> +
> +enum imx_mu_xsr {
> + IMX_MU_SR,
> + IMX_MU_GSR,
> + IMX_MU_TSR,
> + IMX_MU_RSR,
> +};
> +
> +enum imx_mu_type {
> + IMX_MU_V1 = BIT(0),
> + IMX_MU_V2 = BIT(1),
> + IMX_MU_V2_S4 = BIT(15),
> +};
> +
> +/* Receive Interrupt Enable */
> +#define IMX_MU_xCR_RIEn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24 + (3 - (x))))
> +#define IMX_MU_xSR_RFn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24 + (3 - (x))))
> +
> +struct imx_mu_dcfg {
> + enum imx_mu_type type;
> + u32 xTR; /* Transmit Register0 */
> + u32 xRR; /* Receive Register0 */
> + u32 xSR[4]; /* Status Registers */
> + u32 xCR[4]; /* Control Registers */
> +};
> +
> +struct imx_mu_msi {
> + spinlock_t lock;
> + struct platform_device *pdev;
> + struct irq_domain *parent;
> + struct irq_domain *msi_domain;
> + void __iomem *regs;
> + phys_addr_t msiir_addr;
> + const struct imx_mu_dcfg *cfg;
> + unsigned long used;
> + u32 gic_irq;
> + struct clk *clk;
> + struct device *pd_a;
> + struct device *pd_b;
> + struct device_link *pd_link_a;
> + struct device_link *pd_link_b;
> +};
> +
> +static void imx_mu_write(struct imx_mu_msi *msi_data, u32 val, u32 offs)
> +{
> + iowrite32(val, msi_data->regs + offs);
> +}
> +
> +static u32 imx_mu_read(struct imx_mu_msi *msi_data, u32 offs)
> +{
> + return ioread32(msi_data->regs + offs);
> +}
> +
> +static u32 imx_mu_xcr_rmw(struct imx_mu_msi *msi_data, enum imx_mu_xcr type, u32 set, u32 clr)
> +{
> + unsigned long flags;
> + u32 val;
> +
> + spin_lock_irqsave(&msi_data->lock, flags);
> + val = imx_mu_read(msi_data, msi_data->cfg->xCR[type]);
> + val &= ~clr;
> + val |= set;
> + imx_mu_write(msi_data, val, msi_data->cfg->xCR[type]);
> + spin_unlock_irqrestore(&msi_data->lock, flags);
> +
> + return val;
> +}
> +
> +static void imx_mu_msi_mask_irq(struct irq_data *data)
> +{
> + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data->parent_data);
Urgh... No. Please don't go poking into the *parent* stuff. Implement
the masking at the parent level, and use irq_chip_mask_parent() for
this level, unless you can explain why you can't do otherwise.
> +
> + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, 0, IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq));
How about making this readable and move the dereference inside the macro?
> +}
> +
> +static void imx_mu_msi_unmask_irq(struct irq_data *data)
> +{
> + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data->parent_data);
> +
> + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq), 0);
> +}
> +
> +static struct irq_chip imx_mu_msi_irq_chip = {
> + .name = "MU-MSI",
> + .irq_mask = imx_mu_msi_mask_irq,
> + .irq_unmask = imx_mu_msi_unmask_irq,
> +};
> +
> +static struct msi_domain_ops its_pmsi_ops = {
> +};
What does this have to do with the ITS?
> +
> +static struct msi_domain_info imx_mu_msi_domain_info = {
> + .flags = (MSI_FLAG_USE_DEF_DOM_OPS |
> + MSI_FLAG_USE_DEF_CHIP_OPS |
> + MSI_FLAG_PCI_MSIX),
What does PCI_MSIX mean in this context? I really wish you used
copy/paste a bit more carefully.
> + .ops = &its_pmsi_ops,
> + .chip = &imx_mu_msi_irq_chip,
> +};
> +
> +static void imx_mu_msi_compose_msg(struct irq_data *data, struct msi_msg *msg)
> +{
> + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
> +
> + msg->address_hi = upper_32_bits(msi_data->msiir_addr);
> + msg->address_lo = lower_32_bits(msi_data->msiir_addr + 4 * data->hwirq);
This is a horrible way of writing this. you should compute the address
first, and then apply the address split.
> + msg->data = data->hwirq;
> +
> + iommu_dma_compose_msi_msg(irq_data_get_msi_desc(data), msg);
> +}
> +
> +static int imx_mu_msi_set_affinity(struct irq_data *irq_data,
> + const struct cpumask *mask, bool force)
> +
> +{
> + return IRQ_SET_MASK_OK;
> +}
Err... What effect does this have if you don't do anything?
> +
> +static struct irq_chip imx_mu_msi_parent_chip = {
> + .name = "MU",
> + .irq_compose_msi_msg = imx_mu_msi_compose_msg,
> + .irq_set_affinity = imx_mu_msi_set_affinity,
> +};
> +
> +static int imx_mu_msi_domain_irq_alloc(struct irq_domain *domain,
> + unsigned int virq,
> + unsigned int nr_irqs,
> + void *args)
> +{
> + struct imx_mu_msi *msi_data = domain->host_data;
> + msi_alloc_info_t *info = args;
> + int pos, err = 0;
> +
> + WARN_ON(nr_irqs != 1);
> +
> + spin_lock(&msi_data->lock);
Interrupt fires after lock acquisition, handler masks the interrupt.
Deadlock.
> + pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
> + if (pos < IMX_MU_CHANS)
> + __set_bit(pos, &msi_data->used);
> + else
> + err = -ENOSPC;
> + spin_unlock(&msi_data->lock);
> +
> + if (err)
> + return err;
> +
> + err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr + pos * 4);
Does this HW even have an IOMMU?
> + if (err)
> + return err;
> +
> + irq_domain_set_info(domain, virq, pos,
> + &imx_mu_msi_parent_chip, msi_data,
> + handle_simple_irq, NULL, NULL);
This is an edge interrupt. Please handle it like one.
> + return 0;
> +}
> +
> +static void imx_mu_msi_domain_irq_free(struct irq_domain *domain,
> + unsigned int virq, unsigned int nr_irqs)
> +{
> + struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(d);
> +
> + spin_lock(&msi_data->lock);
Same problem.
> + __clear_bit(d->hwirq, &msi_data->used);
> + spin_unlock(&msi_data->lock);
> +}
> +
> +static const struct irq_domain_ops imx_mu_msi_domain_ops = {
> + .alloc = imx_mu_msi_domain_irq_alloc,
> + .free = imx_mu_msi_domain_irq_free,
> +};
> +
> +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> +{
> + struct imx_mu_msi *msi_data = irq_desc_get_handler_data(desc);
> + u32 status;
> + int i;
> +
> + status = imx_mu_read(msi_data, msi_data->cfg->xSR[IMX_MU_RSR]);
> +
> + chained_irq_enter(irq_desc_get_chip(desc), desc);
> + for (i = 0; i < IMX_MU_CHANS; i++) {
> + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> + generic_handle_domain_irq(msi_data->parent, i);
Why the parent? You must start at the top of the hierarchy.
> + }
> + }
> + chained_irq_exit(irq_desc_get_chip(desc), desc);
If your MSIs are a chained interrupt, why do you even provide an
affinity setting callback?
> +}
> +
> +static int imx_mu_msi_domains_init(struct imx_mu_msi *msi_data)
> +{
> + /* Initialize MSI domain parent */
> + msi_data->parent = irq_domain_add_linear(dev_of_node(&msi_data->pdev->dev),
> + IMX_MU_CHANS,
> + &imx_mu_msi_domain_ops,
> + msi_data);
Use irq_domain_create_linear() instead.
> + if (!msi_data->parent) {
> + dev_err(&msi_data->pdev->dev, "failed to create IRQ domain\n");
> + return -ENOMEM;
> + }
> +
> + msi_data->msi_domain = platform_msi_create_irq_domain(
> + of_node_to_fwnode(msi_data->pdev->dev.of_node),
> + &imx_mu_msi_domain_info,
> + msi_data->parent);
And you don't get an error due to the fact that you use the same
fwnode for both domains without overriding the domain bus token?
> +
> + if (!msi_data->msi_domain) {
> + dev_err(&msi_data->pdev->dev, "failed to create MSI domain\n");
> + irq_domain_remove(msi_data->parent);
> + return -ENOMEM;
> + }
> +
> + irq_domain_set_pm_device(msi_data->parent, &msi_data->pdev->dev);
I really wonder why you both implementing a MSI domain if you are
always bypassing it for absolutely everything... This completely
nullifies the effect of this call, as no interrupt request will ever
trigger the PM.
> +
> + return 0;
> +}
> +
> +/* Register offset of different version MU IP */
> +static const struct imx_mu_dcfg imx_mu_cfg_imx6sx = {
> + .xTR = 0x0,
> + .xRR = 0x10,
> + .xSR = {0x20, 0x20, 0x20, 0x20},
> + .xCR = {0x24, 0x24, 0x24, 0x24},
> +};
> +
> +static const struct imx_mu_dcfg imx_mu_cfg_imx7ulp = {
> + .xTR = 0x20,
> + .xRR = 0x40,
> + .xSR = {0x60, 0x60, 0x60, 0x60},
> + .xCR = {0x64, 0x64, 0x64, 0x64},
> +};
> +
> +static const struct imx_mu_dcfg imx_mu_cfg_imx8ulp = {
> + .type = IMX_MU_V2,
> + .xTR = 0x200,
> + .xRR = 0x280,
> + .xSR = {0xC, 0x118, 0x124, 0x12C},
> + .xCR = {0x110, 0x114, 0x120, 0x128},
> +};
> +
> +static const struct imx_mu_dcfg imx_mu_cfg_imx8ulp_s4 = {
> +
> + .type = IMX_MU_V2 | IMX_MU_V2_S4,
> + .xTR = 0x200,
> + .xRR = 0x280,
> + .xSR = {0xC, 0x118, 0x124, 0x12C},
> + .xCR = {0x110, 0x114, 0x120, 0x128},
> +};
> +
> +static int __init imx_mu_of_init(struct device_node *dn,
> + struct device_node *parent,
> + const struct imx_mu_dcfg *cfg)
> +{
> + struct platform_device *pdev = of_find_device_by_node(dn);
> + struct imx_mu_msi *msi_data, *priv;
> + struct resource *res;
> + struct device *dev;
> + int ret;
> +
> + if (!pdev)
> + return -ENODEV;
> +
> + dev = &pdev->dev;
> +
> + priv = msi_data = devm_kzalloc(&pdev->dev, sizeof(*msi_data), GFP_KERNEL);
> + if (!msi_data)
> + return -ENOMEM;
> +
> + msi_data->cfg = cfg;
> +
> + msi_data->regs = devm_platform_ioremap_resource_byname(pdev, "a");
> + if (IS_ERR(msi_data->regs)) {
> + dev_err(&pdev->dev, "failed to initialize 'regs'\n");
> + return PTR_ERR(msi_data->regs);
> + }
> +
> + res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "b");
> + if (!res)
> + return -EIO;
> +
> + msi_data->msiir_addr = res->start + msi_data->cfg->xTR;
> +
> + msi_data->pdev = pdev;
> +
> + msi_data->gic_irq = platform_get_irq(msi_data->pdev, 0);
> + if (msi_data->gic_irq <= 0)
> + return -ENODEV;
> +
> + platform_set_drvdata(pdev, msi_data);
> +
> + msi_data->clk = devm_clk_get(dev, NULL);
> + if (IS_ERR(msi_data->clk)) {
> + if (PTR_ERR(msi_data->clk) != -ENOENT)
> + return PTR_ERR(msi_data->clk);
> +
> + msi_data->clk = NULL;
> + }
> +
> + ret = clk_prepare_enable(msi_data->clk);
> + if (ret) {
> + dev_err(dev, "Failed to enable clock\n");
> + return ret;
> + }
> +
> + priv->pd_a = dev_pm_domain_attach_by_name(dev, "a");
> + if (IS_ERR(priv->pd_a))
> + return PTR_ERR(priv->pd_a);
> +
> + priv->pd_link_a = device_link_add(dev, priv->pd_a,
> + DL_FLAG_STATELESS |
> + DL_FLAG_PM_RUNTIME |
> + DL_FLAG_RPM_ACTIVE);
> +
> + if (!priv->pd_link_a) {
> + dev_err(dev, "Failed to add device_link to mu a.\n");
> + return -EINVAL;
> + }
> +
> + priv->pd_b = dev_pm_domain_attach_by_name(dev, "b");
> + if (IS_ERR(priv->pd_b))
> + return PTR_ERR(priv->pd_b);
> +
> + priv->pd_link_b = device_link_add(dev, priv->pd_b,
> + DL_FLAG_STATELESS |
> + DL_FLAG_PM_RUNTIME |
> + DL_FLAG_RPM_ACTIVE);
> +
> + if (!priv->pd_link_b) {
> + dev_err(dev, "Failed to add device_link to mu a.\n");
> + return -EINVAL;
> + }
> +
> + ret = imx_mu_msi_domains_init(msi_data);
> + if (ret)
> + return ret;
> +
> + irq_set_chained_handler_and_data(msi_data->gic_irq,
> + imx_mu_msi_irq_handler,
> + msi_data);
> +
> + pm_runtime_enable(dev);
> +
> + ret = pm_runtime_get_sync(dev);
> + if (ret < 0) {
> + pm_runtime_put_noidle(dev);
> + goto disable_runtime_pm;
> + }
> +
> + ret = pm_runtime_put_sync(dev);
> + if (ret < 0)
> + goto disable_runtime_pm;
> +
> + clk_disable_unprepare(msi_data->clk);
Why do you need to do all this dance?
> +
> + return 0;
> +
> +disable_runtime_pm:
> + pm_runtime_disable(dev);
> + clk_disable_unprepare(msi_data->clk);
> +
> + return ret;
> +}
> +
> +static int __maybe_unused imx_mu_runtime_suspend(struct device *dev)
> +{
> + struct imx_mu_msi *priv = dev_get_drvdata(dev);
> +
> + clk_disable_unprepare(priv->clk);
> +
> + return 0;
> +}
> +
> +static int __maybe_unused imx_mu_runtime_resume(struct device *dev)
> +{
> + struct imx_mu_msi *priv = dev_get_drvdata(dev);
> + int ret;
> +
> + ret = clk_prepare_enable(priv->clk);
> + if (ret)
> + dev_err(dev, "failed to enable clock\n");
> +
> + return ret;
> +}
> +
> +static const struct dev_pm_ops imx_mu_pm_ops = {
> + SET_RUNTIME_PM_OPS(imx_mu_runtime_suspend,
> + imx_mu_runtime_resume, NULL)
> +};
> +
> +static int __init imx_mu_imx7ulp_of_init(struct device_node *dn,
> + struct device_node *parent)
> +{
> + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx7ulp);
> +}
> +
> +static int __init imx_mu_imx6sx_of_init(struct device_node *dn,
> + struct device_node *parent)
> +{
> + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx6sx);
> +}
> +
> +static int __init imx_mu_imx8ulp_of_init(struct device_node *dn,
> + struct device_node *parent)
> +{
> + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx8ulp);
> +}
> +
> +static int __init imx_mu_imx8ulp_s4_of_init(struct device_node *dn,
> + struct device_node *parent)
> +{
> + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx8ulp_s4);
> +}
> +
> +IRQCHIP_PLATFORM_DRIVER_BEGIN(imx_mu_msi)
> +IRQCHIP_MATCH("fsl,imx7ulp-mu-msi", imx_mu_imx7ulp_of_init)
> +IRQCHIP_MATCH("fsl,imx6sx-mu-msi", imx_mu_imx6sx_of_init)
> +IRQCHIP_MATCH("fsl,imx8ulp-mu-msi", imx_mu_imx8ulp_of_init)
> +IRQCHIP_MATCH("fsl,imx8ulp-mu-msi-s4", imx_mu_imx8ulp_s4_of_init)
> +IRQCHIP_PLATFORM_DRIVER_END(imx_mu_msi, .pm = &imx_mu_pm_ops)
> +
> +
> +MODULE_AUTHOR("Frank Li <Frank.Li@nxp.com>");
> +MODULE_DESCRIPTION("Freescale MU work as MSI controller driver");
> +MODULE_LICENSE("GPL");
M.
--
Without deviation from the norm, progress is not possible.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
2022-07-21 7:57 ` Marc Zyngier
@ 2022-07-21 15:22 ` Frank Li
-1 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-21 15:22 UTC (permalink / raw)
To: Marc Zyngier
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Thursday, July 21, 2022 2:57 AM
> To: Frank Li <frank.li@nxp.com>
> Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
>
> Caution: EXT Email
>
> On Wed, 20 Jul 2022 22:30:34 +0100,
> Frank Li <Frank.Li@nxp.com> wrote:
> >
> > MU support generate irq by write data to a register.
> > This patch make mu worked as msi controller.
> > So MU can do doorbell by using standard msi api.
> >
> > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > ---
> > drivers/irqchip/Kconfig | 7 +
> > drivers/irqchip/Makefile | 1 +
> > drivers/irqchip/irq-imx-mu-msi.c | 462
> +++++++++++++++++++++++++++++++
> > 3 files changed, 470 insertions(+)
> > create mode 100644 drivers/irqchip/irq-imx-mu-msi.c
> >
> > diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
> > index 5e4e50122777d..4599471d880c0 100644
> > --- a/drivers/irqchip/Kconfig
> > +++ b/drivers/irqchip/Kconfig
> > @@ -470,6 +470,13 @@ config IMX_INTMUX
> > help
> > Support for the i.MX INTMUX interrupt multiplexer.
> >
> > +config IMX_MU_MSI
> > + bool "i.MX MU work as MSI controller"
> > + default y if ARCH_MXC
> > + select IRQ_DOMAIN
> > + help
> > + MU work as MSI controller to do general doorbell
> > +
> > config LS1X_IRQ
> > bool "Loongson-1 Interrupt Controller"
> > depends on MACH_LOONGSON32
> > diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
> > index 5d8e21d3dc6d8..870423746c783 100644
> > --- a/drivers/irqchip/Makefile
> > +++ b/drivers/irqchip/Makefile
> > @@ -98,6 +98,7 @@ obj-$(CONFIG_RISCV_INTC) += irq-riscv-intc.o
> > obj-$(CONFIG_SIFIVE_PLIC) += irq-sifive-plic.o
> > obj-$(CONFIG_IMX_IRQSTEER) += irq-imx-irqsteer.o
> > obj-$(CONFIG_IMX_INTMUX) += irq-imx-intmux.o
> > +obj-$(CONFIG_IMX_MU_MSI) += irq-imx-mu-msi.o
> > obj-$(CONFIG_MADERA_IRQ) += irq-madera.o
> > obj-$(CONFIG_LS1X_IRQ) += irq-ls1x.o
> > obj-$(CONFIG_TI_SCI_INTR_IRQCHIP) += irq-ti-sci-intr.o
> > diff --git a/drivers/irqchip/irq-imx-mu-msi.c b/drivers/irqchip/irq-imx-mu-
> msi.c
> > new file mode 100644
> > index 0000000000000..8277dba857759
> > --- /dev/null
> > +++ b/drivers/irqchip/irq-imx-mu-msi.c
> > @@ -0,0 +1,462 @@
> > +// SPDX-License-Identifier: GPL-2.0-only
> > +/*
> > + * NXP MU worked as MSI controller
> > + *
> > + * Copyright (c) 2018 Pengutronix, Oleksij Rempel
> <o.rempel@pengutronix.de>
> > + * Copyright 2022 NXP
> > + * Frank Li <Frank.Li@nxp.com>
> > + * Peng Fan <peng.fan@nxp.com>
> > + *
> > + * Based on drivers/mailbox/imx-mailbox.c
> > + */
> > +#include <linux/clk.h>
> > +#include <linux/kernel.h>
> > +#include <linux/module.h>
> > +#include <linux/msi.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/irq.h>
> > +#include <linux/irqchip/chained_irq.h>
> > +#include <linux/irqchip.h>
> > +#include <linux/irqdomain.h>
> > +#include <linux/of_irq.h>
> > +#include <linux/of_pci.h>
> > +#include <linux/of_platform.h>
> > +#include <linux/spinlock.h>
> > +#include <linux/dma-iommu.h>
> > +#include <linux/pm_runtime.h>
> > +#include <linux/pm_domain.h>
> > +
> > +
> > +#define IMX_MU_CHANS 4
> > +
> > +enum imx_mu_xcr {
> > + IMX_MU_GIER,
> > + IMX_MU_GCR,
> > + IMX_MU_TCR,
> > + IMX_MU_RCR,
> > + IMX_MU_xCR_MAX,
> > +};
> > +
> > +enum imx_mu_xsr {
> > + IMX_MU_SR,
> > + IMX_MU_GSR,
> > + IMX_MU_TSR,
> > + IMX_MU_RSR,
> > +};
> > +
> > +enum imx_mu_type {
> > + IMX_MU_V1 = BIT(0),
> > + IMX_MU_V2 = BIT(1),
> > + IMX_MU_V2_S4 = BIT(15),
> > +};
> > +
> > +/* Receive Interrupt Enable */
> > +#define IMX_MU_xCR_RIEn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24
> + (3 - (x))))
> > +#define IMX_MU_xSR_RFn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24 +
> (3 - (x))))
> > +
> > +struct imx_mu_dcfg {
> > + enum imx_mu_type type;
> > + u32 xTR; /* Transmit Register0 */
> > + u32 xRR; /* Receive Register0 */
> > + u32 xSR[4]; /* Status Registers */
> > + u32 xCR[4]; /* Control Registers */
> > +};
> > +
> > +struct imx_mu_msi {
> > + spinlock_t lock;
> > + struct platform_device *pdev;
> > + struct irq_domain *parent;
> > + struct irq_domain *msi_domain;
> > + void __iomem *regs;
> > + phys_addr_t msiir_addr;
> > + const struct imx_mu_dcfg *cfg;
> > + unsigned long used;
> > + u32 gic_irq;
> > + struct clk *clk;
> > + struct device *pd_a;
> > + struct device *pd_b;
> > + struct device_link *pd_link_a;
> > + struct device_link *pd_link_b;
> > +};
> > +
> > +static void imx_mu_write(struct imx_mu_msi *msi_data, u32 val, u32 offs)
> > +{
> > + iowrite32(val, msi_data->regs + offs);
> > +}
> > +
> > +static u32 imx_mu_read(struct imx_mu_msi *msi_data, u32 offs)
> > +{
> > + return ioread32(msi_data->regs + offs);
> > +}
> > +
> > +static u32 imx_mu_xcr_rmw(struct imx_mu_msi *msi_data, enum
> imx_mu_xcr type, u32 set, u32 clr)
> > +{
> > + unsigned long flags;
> > + u32 val;
> > +
> > + spin_lock_irqsave(&msi_data->lock, flags);
> > + val = imx_mu_read(msi_data, msi_data->cfg->xCR[type]);
> > + val &= ~clr;
> > + val |= set;
> > + imx_mu_write(msi_data, val, msi_data->cfg->xCR[type]);
> > + spin_unlock_irqrestore(&msi_data->lock, flags);
> > +
> > + return val;
> > +}
> > +
> > +static void imx_mu_msi_mask_irq(struct irq_data *data)
> > +{
> > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> >parent_data);
>
> Urgh... No. Please don't go poking into the *parent* stuff. Implement
> the masking at the parent level, and use irq_chip_mask_parent() for
> this level, unless you can explain why you can't do otherwise.
>
> > +
> > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, 0,
> IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq));
>
> How about making this readable and move the dereference inside the macro?
>
> > +}
> > +
> > +static void imx_mu_msi_unmask_irq(struct irq_data *data)
> > +{
> > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> >parent_data);
> > +
> > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR,
> IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq), 0);
> > +}
> > +
> > +static struct irq_chip imx_mu_msi_irq_chip = {
> > + .name = "MU-MSI",
> > + .irq_mask = imx_mu_msi_mask_irq,
> > + .irq_unmask = imx_mu_msi_unmask_irq,
> > +};
> > +
> > +static struct msi_domain_ops its_pmsi_ops = {
> > +};
>
> What does this have to do with the ITS?
Without this, it will be crashed as access 0 address.
>
> > +
> > +static struct msi_domain_info imx_mu_msi_domain_info = {
> > + .flags = (MSI_FLAG_USE_DEF_DOM_OPS |
> > + MSI_FLAG_USE_DEF_CHIP_OPS |
> > + MSI_FLAG_PCI_MSIX),
>
> What does PCI_MSIX mean in this context? I really wish you used
> copy/paste a bit more carefully.
>
> > + .ops = &its_pmsi_ops,
> > + .chip = &imx_mu_msi_irq_chip,
> > +};
> > +
> > +static void imx_mu_msi_compose_msg(struct irq_data *data, struct
> msi_msg *msg)
> > +{
> > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
> > +
> > + msg->address_hi = upper_32_bits(msi_data->msiir_addr);
> > + msg->address_lo = lower_32_bits(msi_data->msiir_addr + 4 * data-
> >hwirq);
>
> This is a horrible way of writing this. you should compute the address
> first, and then apply the address split.
>
> > + msg->data = data->hwirq;
> > +
> > + iommu_dma_compose_msi_msg(irq_data_get_msi_desc(data), msg);
> > +}
> > +
> > +static int imx_mu_msi_set_affinity(struct irq_data *irq_data,
> > + const struct cpumask *mask, bool force)
> > +
> > +{
> > + return IRQ_SET_MASK_OK;
> > +}
>
> Err... What effect does this have if you don't do anything?
[Frank Li] Without this, it will be crashed as access 0 address.
>
> > +
> > +static struct irq_chip imx_mu_msi_parent_chip = {
> > + .name = "MU",
> > + .irq_compose_msi_msg = imx_mu_msi_compose_msg,
> > + .irq_set_affinity = imx_mu_msi_set_affinity,
> > +};
> > +
> > +static int imx_mu_msi_domain_irq_alloc(struct irq_domain *domain,
> > + unsigned int virq,
> > + unsigned int nr_irqs,
> > + void *args)
> > +{
> > + struct imx_mu_msi *msi_data = domain->host_data;
> > + msi_alloc_info_t *info = args;
> > + int pos, err = 0;
> > +
> > + WARN_ON(nr_irqs != 1);
> > +
> > + spin_lock(&msi_data->lock);
>
> Interrupt fires after lock acquisition, handler masks the interrupt.
> Deadlock.
[Frank Li] you are right, it should be spin_lock_irqsave.
>
> > + pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
> > + if (pos < IMX_MU_CHANS)
> > + __set_bit(pos, &msi_data->used);
> > + else
> > + err = -ENOSPC;
> > + spin_unlock(&msi_data->lock);
> > +
> > + if (err)
> > + return err;
> > +
> > + err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr +
> pos * 4);
>
> Does this HW even have an IOMMU?
[Frank Li] we have a platform with iommu.
>
> > + if (err)
> > + return err;
> > +
> > + irq_domain_set_info(domain, virq, pos,
> > + &imx_mu_msi_parent_chip, msi_data,
> > + handle_simple_irq, NULL, NULL);
>
> This is an edge interrupt. Please handle it like one.
[Frank Li] Not sure what your means?
>
> > + return 0;
> > +}
> > +
> > +static void imx_mu_msi_domain_irq_free(struct irq_domain *domain,
> > + unsigned int virq, unsigned int nr_irqs)
> > +{
> > + struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(d);
> > +
> > + spin_lock(&msi_data->lock);
>
> Same problem.
>
> > + __clear_bit(d->hwirq, &msi_data->used);
> > + spin_unlock(&msi_data->lock);
> > +}
> > +
> > +static const struct irq_domain_ops imx_mu_msi_domain_ops = {
> > + .alloc = imx_mu_msi_domain_irq_alloc,
> > + .free = imx_mu_msi_domain_irq_free,
> > +};
> > +
> > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > +{
> > + struct imx_mu_msi *msi_data = irq_desc_get_handler_data(desc);
> > + u32 status;
> > + int i;
> > +
> > + status = imx_mu_read(msi_data, msi_data->cfg->xSR[IMX_MU_RSR]);
> > +
> > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > + generic_handle_domain_irq(msi_data->parent, i);
>
> Why the parent? You must start at the top of the hierarchy.
>
> > + }
> > + }
> > + chained_irq_exit(irq_desc_get_chip(desc), desc);
>
> If your MSIs are a chained interrupt, why do you even provide an
> affinity setting callback?
[Frank Li] it will be crash if no affinity setting callback.
>
> > +}
> > +
> > +static int imx_mu_msi_domains_init(struct imx_mu_msi *msi_data)
> > +{
> > + /* Initialize MSI domain parent */
> > + msi_data->parent = irq_domain_add_linear(dev_of_node(&msi_data-
> >pdev->dev),
> > + IMX_MU_CHANS,
> > + &imx_mu_msi_domain_ops,
> > + msi_data);
>
> Use irq_domain_create_linear() instead.
>
> > + if (!msi_data->parent) {
> > + dev_err(&msi_data->pdev->dev, "failed to create IRQ domain\n");
> > + return -ENOMEM;
> > + }
> > +
> > + msi_data->msi_domain = platform_msi_create_irq_domain(
> > + of_node_to_fwnode(msi_data->pdev->dev.of_node),
> > + &imx_mu_msi_domain_info,
> > + msi_data->parent);
>
> And you don't get an error due to the fact that you use the same
> fwnode for both domains without overriding the domain bus token?
>
> > +
> > + if (!msi_data->msi_domain) {
> > + dev_err(&msi_data->pdev->dev, "failed to create MSI domain\n");
> > + irq_domain_remove(msi_data->parent);
> > + return -ENOMEM;
> > + }
> > +
> > + irq_domain_set_pm_device(msi_data->parent, &msi_data->pdev-
> >dev);
>
> I really wonder why you both implementing a MSI domain if you are
> always bypassing it for absolutely everything... This completely
> nullifies the effect of this call, as no interrupt request will ever
> trigger the PM.
>
> > +
> > + return 0;
> > +}
> > +
> > +/* Register offset of different version MU IP */
> > +static const struct imx_mu_dcfg imx_mu_cfg_imx6sx = {
> > + .xTR = 0x0,
> > + .xRR = 0x10,
> > + .xSR = {0x20, 0x20, 0x20, 0x20},
> > + .xCR = {0x24, 0x24, 0x24, 0x24},
> > +};
> > +
> > +static const struct imx_mu_dcfg imx_mu_cfg_imx7ulp = {
> > + .xTR = 0x20,
> > + .xRR = 0x40,
> > + .xSR = {0x60, 0x60, 0x60, 0x60},
> > + .xCR = {0x64, 0x64, 0x64, 0x64},
> > +};
> > +
> > +static const struct imx_mu_dcfg imx_mu_cfg_imx8ulp = {
> > + .type = IMX_MU_V2,
> > + .xTR = 0x200,
> > + .xRR = 0x280,
> > + .xSR = {0xC, 0x118, 0x124, 0x12C},
> > + .xCR = {0x110, 0x114, 0x120, 0x128},
> > +};
> > +
> > +static const struct imx_mu_dcfg imx_mu_cfg_imx8ulp_s4 = {
> > +
> > + .type = IMX_MU_V2 | IMX_MU_V2_S4,
> > + .xTR = 0x200,
> > + .xRR = 0x280,
> > + .xSR = {0xC, 0x118, 0x124, 0x12C},
> > + .xCR = {0x110, 0x114, 0x120, 0x128},
> > +};
> > +
> > +static int __init imx_mu_of_init(struct device_node *dn,
> > + struct device_node *parent,
> > + const struct imx_mu_dcfg *cfg)
> > +{
> > + struct platform_device *pdev = of_find_device_by_node(dn);
> > + struct imx_mu_msi *msi_data, *priv;
> > + struct resource *res;
> > + struct device *dev;
> > + int ret;
> > +
> > + if (!pdev)
> > + return -ENODEV;
> > +
> > + dev = &pdev->dev;
> > +
> > + priv = msi_data = devm_kzalloc(&pdev->dev, sizeof(*msi_data),
> GFP_KERNEL);
> > + if (!msi_data)
> > + return -ENOMEM;
> > +
> > + msi_data->cfg = cfg;
> > +
> > + msi_data->regs = devm_platform_ioremap_resource_byname(pdev,
> "a");
> > + if (IS_ERR(msi_data->regs)) {
> > + dev_err(&pdev->dev, "failed to initialize 'regs'\n");
> > + return PTR_ERR(msi_data->regs);
> > + }
> > +
> > + res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "b");
> > + if (!res)
> > + return -EIO;
> > +
> > + msi_data->msiir_addr = res->start + msi_data->cfg->xTR;
> > +
> > + msi_data->pdev = pdev;
> > +
> > + msi_data->gic_irq = platform_get_irq(msi_data->pdev, 0);
> > + if (msi_data->gic_irq <= 0)
> > + return -ENODEV;
> > +
> > + platform_set_drvdata(pdev, msi_data);
> > +
> > + msi_data->clk = devm_clk_get(dev, NULL);
> > + if (IS_ERR(msi_data->clk)) {
> > + if (PTR_ERR(msi_data->clk) != -ENOENT)
> > + return PTR_ERR(msi_data->clk);
> > +
> > + msi_data->clk = NULL;
> > + }
> > +
> > + ret = clk_prepare_enable(msi_data->clk);
> > + if (ret) {
> > + dev_err(dev, "Failed to enable clock\n");
> > + return ret;
> > + }
> > +
> > + priv->pd_a = dev_pm_domain_attach_by_name(dev, "a");
> > + if (IS_ERR(priv->pd_a))
> > + return PTR_ERR(priv->pd_a);
> > +
> > + priv->pd_link_a = device_link_add(dev, priv->pd_a,
> > + DL_FLAG_STATELESS |
> > + DL_FLAG_PM_RUNTIME |
> > + DL_FLAG_RPM_ACTIVE);
> > +
> > + if (!priv->pd_link_a) {
> > + dev_err(dev, "Failed to add device_link to mu a.\n");
> > + return -EINVAL;
> > + }
> > +
> > + priv->pd_b = dev_pm_domain_attach_by_name(dev, "b");
> > + if (IS_ERR(priv->pd_b))
> > + return PTR_ERR(priv->pd_b);
> > +
> > + priv->pd_link_b = device_link_add(dev, priv->pd_b,
> > + DL_FLAG_STATELESS |
> > + DL_FLAG_PM_RUNTIME |
> > + DL_FLAG_RPM_ACTIVE);
> > +
> > + if (!priv->pd_link_b) {
> > + dev_err(dev, "Failed to add device_link to mu a.\n");
> > + return -EINVAL;
> > + }
> > +
> > + ret = imx_mu_msi_domains_init(msi_data);
> > + if (ret)
> > + return ret;
> > +
> > + irq_set_chained_handler_and_data(msi_data->gic_irq,
> > + imx_mu_msi_irq_handler,
> > + msi_data);
> > +
> > + pm_runtime_enable(dev);
> > +
> > + ret = pm_runtime_get_sync(dev);
> > + if (ret < 0) {
> > + pm_runtime_put_noidle(dev);
> > + goto disable_runtime_pm;
> > + }
> > +
> > + ret = pm_runtime_put_sync(dev);
> > + if (ret < 0)
> > + goto disable_runtime_pm;
> > +
> > + clk_disable_unprepare(msi_data->clk);
>
> Why do you need to do all this dance?
>
> > +
> > + return 0;
> > +
> > +disable_runtime_pm:
> > + pm_runtime_disable(dev);
> > + clk_disable_unprepare(msi_data->clk);
> > +
> > + return ret;
> > +}
> > +
> > +static int __maybe_unused imx_mu_runtime_suspend(struct device *dev)
> > +{
> > + struct imx_mu_msi *priv = dev_get_drvdata(dev);
> > +
> > + clk_disable_unprepare(priv->clk);
> > +
> > + return 0;
> > +}
> > +
> > +static int __maybe_unused imx_mu_runtime_resume(struct device *dev)
> > +{
> > + struct imx_mu_msi *priv = dev_get_drvdata(dev);
> > + int ret;
> > +
> > + ret = clk_prepare_enable(priv->clk);
> > + if (ret)
> > + dev_err(dev, "failed to enable clock\n");
> > +
> > + return ret;
> > +}
> > +
> > +static const struct dev_pm_ops imx_mu_pm_ops = {
> > + SET_RUNTIME_PM_OPS(imx_mu_runtime_suspend,
> > + imx_mu_runtime_resume, NULL)
> > +};
> > +
> > +static int __init imx_mu_imx7ulp_of_init(struct device_node *dn,
> > + struct device_node *parent)
> > +{
> > + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx7ulp);
> > +}
> > +
> > +static int __init imx_mu_imx6sx_of_init(struct device_node *dn,
> > + struct device_node *parent)
> > +{
> > + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx6sx);
> > +}
> > +
> > +static int __init imx_mu_imx8ulp_of_init(struct device_node *dn,
> > + struct device_node *parent)
> > +{
> > + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx8ulp);
> > +}
> > +
> > +static int __init imx_mu_imx8ulp_s4_of_init(struct device_node *dn,
> > + struct device_node *parent)
> > +{
> > + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx8ulp_s4);
> > +}
> > +
> > +IRQCHIP_PLATFORM_DRIVER_BEGIN(imx_mu_msi)
> > +IRQCHIP_MATCH("fsl,imx7ulp-mu-msi", imx_mu_imx7ulp_of_init)
> > +IRQCHIP_MATCH("fsl,imx6sx-mu-msi", imx_mu_imx6sx_of_init)
> > +IRQCHIP_MATCH("fsl,imx8ulp-mu-msi", imx_mu_imx8ulp_of_init)
> > +IRQCHIP_MATCH("fsl,imx8ulp-mu-msi-s4", imx_mu_imx8ulp_s4_of_init)
> > +IRQCHIP_PLATFORM_DRIVER_END(imx_mu_msi, .pm =
> &imx_mu_pm_ops)
> > +
> > +
> > +MODULE_AUTHOR("Frank Li <Frank.Li@nxp.com>");
> > +MODULE_DESCRIPTION("Freescale MU work as MSI controller driver");
> > +MODULE_LICENSE("GPL");
>
> M.
>
> --
> Without deviation from the norm, progress is not possible.
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
@ 2022-07-21 15:22 ` Frank Li
0 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-21 15:22 UTC (permalink / raw)
To: Marc Zyngier
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Thursday, July 21, 2022 2:57 AM
> To: Frank Li <frank.li@nxp.com>
> Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
>
> Caution: EXT Email
>
> On Wed, 20 Jul 2022 22:30:34 +0100,
> Frank Li <Frank.Li@nxp.com> wrote:
> >
> > MU support generate irq by write data to a register.
> > This patch make mu worked as msi controller.
> > So MU can do doorbell by using standard msi api.
> >
> > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > ---
> > drivers/irqchip/Kconfig | 7 +
> > drivers/irqchip/Makefile | 1 +
> > drivers/irqchip/irq-imx-mu-msi.c | 462
> +++++++++++++++++++++++++++++++
> > 3 files changed, 470 insertions(+)
> > create mode 100644 drivers/irqchip/irq-imx-mu-msi.c
> >
> > diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
> > index 5e4e50122777d..4599471d880c0 100644
> > --- a/drivers/irqchip/Kconfig
> > +++ b/drivers/irqchip/Kconfig
> > @@ -470,6 +470,13 @@ config IMX_INTMUX
> > help
> > Support for the i.MX INTMUX interrupt multiplexer.
> >
> > +config IMX_MU_MSI
> > + bool "i.MX MU work as MSI controller"
> > + default y if ARCH_MXC
> > + select IRQ_DOMAIN
> > + help
> > + MU work as MSI controller to do general doorbell
> > +
> > config LS1X_IRQ
> > bool "Loongson-1 Interrupt Controller"
> > depends on MACH_LOONGSON32
> > diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
> > index 5d8e21d3dc6d8..870423746c783 100644
> > --- a/drivers/irqchip/Makefile
> > +++ b/drivers/irqchip/Makefile
> > @@ -98,6 +98,7 @@ obj-$(CONFIG_RISCV_INTC) += irq-riscv-intc.o
> > obj-$(CONFIG_SIFIVE_PLIC) += irq-sifive-plic.o
> > obj-$(CONFIG_IMX_IRQSTEER) += irq-imx-irqsteer.o
> > obj-$(CONFIG_IMX_INTMUX) += irq-imx-intmux.o
> > +obj-$(CONFIG_IMX_MU_MSI) += irq-imx-mu-msi.o
> > obj-$(CONFIG_MADERA_IRQ) += irq-madera.o
> > obj-$(CONFIG_LS1X_IRQ) += irq-ls1x.o
> > obj-$(CONFIG_TI_SCI_INTR_IRQCHIP) += irq-ti-sci-intr.o
> > diff --git a/drivers/irqchip/irq-imx-mu-msi.c b/drivers/irqchip/irq-imx-mu-
> msi.c
> > new file mode 100644
> > index 0000000000000..8277dba857759
> > --- /dev/null
> > +++ b/drivers/irqchip/irq-imx-mu-msi.c
> > @@ -0,0 +1,462 @@
> > +// SPDX-License-Identifier: GPL-2.0-only
> > +/*
> > + * NXP MU worked as MSI controller
> > + *
> > + * Copyright (c) 2018 Pengutronix, Oleksij Rempel
> <o.rempel@pengutronix.de>
> > + * Copyright 2022 NXP
> > + * Frank Li <Frank.Li@nxp.com>
> > + * Peng Fan <peng.fan@nxp.com>
> > + *
> > + * Based on drivers/mailbox/imx-mailbox.c
> > + */
> > +#include <linux/clk.h>
> > +#include <linux/kernel.h>
> > +#include <linux/module.h>
> > +#include <linux/msi.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/irq.h>
> > +#include <linux/irqchip/chained_irq.h>
> > +#include <linux/irqchip.h>
> > +#include <linux/irqdomain.h>
> > +#include <linux/of_irq.h>
> > +#include <linux/of_pci.h>
> > +#include <linux/of_platform.h>
> > +#include <linux/spinlock.h>
> > +#include <linux/dma-iommu.h>
> > +#include <linux/pm_runtime.h>
> > +#include <linux/pm_domain.h>
> > +
> > +
> > +#define IMX_MU_CHANS 4
> > +
> > +enum imx_mu_xcr {
> > + IMX_MU_GIER,
> > + IMX_MU_GCR,
> > + IMX_MU_TCR,
> > + IMX_MU_RCR,
> > + IMX_MU_xCR_MAX,
> > +};
> > +
> > +enum imx_mu_xsr {
> > + IMX_MU_SR,
> > + IMX_MU_GSR,
> > + IMX_MU_TSR,
> > + IMX_MU_RSR,
> > +};
> > +
> > +enum imx_mu_type {
> > + IMX_MU_V1 = BIT(0),
> > + IMX_MU_V2 = BIT(1),
> > + IMX_MU_V2_S4 = BIT(15),
> > +};
> > +
> > +/* Receive Interrupt Enable */
> > +#define IMX_MU_xCR_RIEn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24
> + (3 - (x))))
> > +#define IMX_MU_xSR_RFn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24 +
> (3 - (x))))
> > +
> > +struct imx_mu_dcfg {
> > + enum imx_mu_type type;
> > + u32 xTR; /* Transmit Register0 */
> > + u32 xRR; /* Receive Register0 */
> > + u32 xSR[4]; /* Status Registers */
> > + u32 xCR[4]; /* Control Registers */
> > +};
> > +
> > +struct imx_mu_msi {
> > + spinlock_t lock;
> > + struct platform_device *pdev;
> > + struct irq_domain *parent;
> > + struct irq_domain *msi_domain;
> > + void __iomem *regs;
> > + phys_addr_t msiir_addr;
> > + const struct imx_mu_dcfg *cfg;
> > + unsigned long used;
> > + u32 gic_irq;
> > + struct clk *clk;
> > + struct device *pd_a;
> > + struct device *pd_b;
> > + struct device_link *pd_link_a;
> > + struct device_link *pd_link_b;
> > +};
> > +
> > +static void imx_mu_write(struct imx_mu_msi *msi_data, u32 val, u32 offs)
> > +{
> > + iowrite32(val, msi_data->regs + offs);
> > +}
> > +
> > +static u32 imx_mu_read(struct imx_mu_msi *msi_data, u32 offs)
> > +{
> > + return ioread32(msi_data->regs + offs);
> > +}
> > +
> > +static u32 imx_mu_xcr_rmw(struct imx_mu_msi *msi_data, enum
> imx_mu_xcr type, u32 set, u32 clr)
> > +{
> > + unsigned long flags;
> > + u32 val;
> > +
> > + spin_lock_irqsave(&msi_data->lock, flags);
> > + val = imx_mu_read(msi_data, msi_data->cfg->xCR[type]);
> > + val &= ~clr;
> > + val |= set;
> > + imx_mu_write(msi_data, val, msi_data->cfg->xCR[type]);
> > + spin_unlock_irqrestore(&msi_data->lock, flags);
> > +
> > + return val;
> > +}
> > +
> > +static void imx_mu_msi_mask_irq(struct irq_data *data)
> > +{
> > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> >parent_data);
>
> Urgh... No. Please don't go poking into the *parent* stuff. Implement
> the masking at the parent level, and use irq_chip_mask_parent() for
> this level, unless you can explain why you can't do otherwise.
>
> > +
> > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, 0,
> IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq));
>
> How about making this readable and move the dereference inside the macro?
>
> > +}
> > +
> > +static void imx_mu_msi_unmask_irq(struct irq_data *data)
> > +{
> > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> >parent_data);
> > +
> > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR,
> IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq), 0);
> > +}
> > +
> > +static struct irq_chip imx_mu_msi_irq_chip = {
> > + .name = "MU-MSI",
> > + .irq_mask = imx_mu_msi_mask_irq,
> > + .irq_unmask = imx_mu_msi_unmask_irq,
> > +};
> > +
> > +static struct msi_domain_ops its_pmsi_ops = {
> > +};
>
> What does this have to do with the ITS?
Without this, it will be crashed as access 0 address.
>
> > +
> > +static struct msi_domain_info imx_mu_msi_domain_info = {
> > + .flags = (MSI_FLAG_USE_DEF_DOM_OPS |
> > + MSI_FLAG_USE_DEF_CHIP_OPS |
> > + MSI_FLAG_PCI_MSIX),
>
> What does PCI_MSIX mean in this context? I really wish you used
> copy/paste a bit more carefully.
>
> > + .ops = &its_pmsi_ops,
> > + .chip = &imx_mu_msi_irq_chip,
> > +};
> > +
> > +static void imx_mu_msi_compose_msg(struct irq_data *data, struct
> msi_msg *msg)
> > +{
> > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
> > +
> > + msg->address_hi = upper_32_bits(msi_data->msiir_addr);
> > + msg->address_lo = lower_32_bits(msi_data->msiir_addr + 4 * data-
> >hwirq);
>
> This is a horrible way of writing this. you should compute the address
> first, and then apply the address split.
>
> > + msg->data = data->hwirq;
> > +
> > + iommu_dma_compose_msi_msg(irq_data_get_msi_desc(data), msg);
> > +}
> > +
> > +static int imx_mu_msi_set_affinity(struct irq_data *irq_data,
> > + const struct cpumask *mask, bool force)
> > +
> > +{
> > + return IRQ_SET_MASK_OK;
> > +}
>
> Err... What effect does this have if you don't do anything?
[Frank Li] Without this, it will be crashed as access 0 address.
>
> > +
> > +static struct irq_chip imx_mu_msi_parent_chip = {
> > + .name = "MU",
> > + .irq_compose_msi_msg = imx_mu_msi_compose_msg,
> > + .irq_set_affinity = imx_mu_msi_set_affinity,
> > +};
> > +
> > +static int imx_mu_msi_domain_irq_alloc(struct irq_domain *domain,
> > + unsigned int virq,
> > + unsigned int nr_irqs,
> > + void *args)
> > +{
> > + struct imx_mu_msi *msi_data = domain->host_data;
> > + msi_alloc_info_t *info = args;
> > + int pos, err = 0;
> > +
> > + WARN_ON(nr_irqs != 1);
> > +
> > + spin_lock(&msi_data->lock);
>
> Interrupt fires after lock acquisition, handler masks the interrupt.
> Deadlock.
[Frank Li] you are right, it should be spin_lock_irqsave.
>
> > + pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
> > + if (pos < IMX_MU_CHANS)
> > + __set_bit(pos, &msi_data->used);
> > + else
> > + err = -ENOSPC;
> > + spin_unlock(&msi_data->lock);
> > +
> > + if (err)
> > + return err;
> > +
> > + err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr +
> pos * 4);
>
> Does this HW even have an IOMMU?
[Frank Li] we have a platform with iommu.
>
> > + if (err)
> > + return err;
> > +
> > + irq_domain_set_info(domain, virq, pos,
> > + &imx_mu_msi_parent_chip, msi_data,
> > + handle_simple_irq, NULL, NULL);
>
> This is an edge interrupt. Please handle it like one.
[Frank Li] Not sure what your means?
>
> > + return 0;
> > +}
> > +
> > +static void imx_mu_msi_domain_irq_free(struct irq_domain *domain,
> > + unsigned int virq, unsigned int nr_irqs)
> > +{
> > + struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(d);
> > +
> > + spin_lock(&msi_data->lock);
>
> Same problem.
>
> > + __clear_bit(d->hwirq, &msi_data->used);
> > + spin_unlock(&msi_data->lock);
> > +}
> > +
> > +static const struct irq_domain_ops imx_mu_msi_domain_ops = {
> > + .alloc = imx_mu_msi_domain_irq_alloc,
> > + .free = imx_mu_msi_domain_irq_free,
> > +};
> > +
> > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > +{
> > + struct imx_mu_msi *msi_data = irq_desc_get_handler_data(desc);
> > + u32 status;
> > + int i;
> > +
> > + status = imx_mu_read(msi_data, msi_data->cfg->xSR[IMX_MU_RSR]);
> > +
> > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > + generic_handle_domain_irq(msi_data->parent, i);
>
> Why the parent? You must start at the top of the hierarchy.
>
> > + }
> > + }
> > + chained_irq_exit(irq_desc_get_chip(desc), desc);
>
> If your MSIs are a chained interrupt, why do you even provide an
> affinity setting callback?
[Frank Li] it will be crash if no affinity setting callback.
>
> > +}
> > +
> > +static int imx_mu_msi_domains_init(struct imx_mu_msi *msi_data)
> > +{
> > + /* Initialize MSI domain parent */
> > + msi_data->parent = irq_domain_add_linear(dev_of_node(&msi_data-
> >pdev->dev),
> > + IMX_MU_CHANS,
> > + &imx_mu_msi_domain_ops,
> > + msi_data);
>
> Use irq_domain_create_linear() instead.
>
> > + if (!msi_data->parent) {
> > + dev_err(&msi_data->pdev->dev, "failed to create IRQ domain\n");
> > + return -ENOMEM;
> > + }
> > +
> > + msi_data->msi_domain = platform_msi_create_irq_domain(
> > + of_node_to_fwnode(msi_data->pdev->dev.of_node),
> > + &imx_mu_msi_domain_info,
> > + msi_data->parent);
>
> And you don't get an error due to the fact that you use the same
> fwnode for both domains without overriding the domain bus token?
>
> > +
> > + if (!msi_data->msi_domain) {
> > + dev_err(&msi_data->pdev->dev, "failed to create MSI domain\n");
> > + irq_domain_remove(msi_data->parent);
> > + return -ENOMEM;
> > + }
> > +
> > + irq_domain_set_pm_device(msi_data->parent, &msi_data->pdev-
> >dev);
>
> I really wonder why you both implementing a MSI domain if you are
> always bypassing it for absolutely everything... This completely
> nullifies the effect of this call, as no interrupt request will ever
> trigger the PM.
>
> > +
> > + return 0;
> > +}
> > +
> > +/* Register offset of different version MU IP */
> > +static const struct imx_mu_dcfg imx_mu_cfg_imx6sx = {
> > + .xTR = 0x0,
> > + .xRR = 0x10,
> > + .xSR = {0x20, 0x20, 0x20, 0x20},
> > + .xCR = {0x24, 0x24, 0x24, 0x24},
> > +};
> > +
> > +static const struct imx_mu_dcfg imx_mu_cfg_imx7ulp = {
> > + .xTR = 0x20,
> > + .xRR = 0x40,
> > + .xSR = {0x60, 0x60, 0x60, 0x60},
> > + .xCR = {0x64, 0x64, 0x64, 0x64},
> > +};
> > +
> > +static const struct imx_mu_dcfg imx_mu_cfg_imx8ulp = {
> > + .type = IMX_MU_V2,
> > + .xTR = 0x200,
> > + .xRR = 0x280,
> > + .xSR = {0xC, 0x118, 0x124, 0x12C},
> > + .xCR = {0x110, 0x114, 0x120, 0x128},
> > +};
> > +
> > +static const struct imx_mu_dcfg imx_mu_cfg_imx8ulp_s4 = {
> > +
> > + .type = IMX_MU_V2 | IMX_MU_V2_S4,
> > + .xTR = 0x200,
> > + .xRR = 0x280,
> > + .xSR = {0xC, 0x118, 0x124, 0x12C},
> > + .xCR = {0x110, 0x114, 0x120, 0x128},
> > +};
> > +
> > +static int __init imx_mu_of_init(struct device_node *dn,
> > + struct device_node *parent,
> > + const struct imx_mu_dcfg *cfg)
> > +{
> > + struct platform_device *pdev = of_find_device_by_node(dn);
> > + struct imx_mu_msi *msi_data, *priv;
> > + struct resource *res;
> > + struct device *dev;
> > + int ret;
> > +
> > + if (!pdev)
> > + return -ENODEV;
> > +
> > + dev = &pdev->dev;
> > +
> > + priv = msi_data = devm_kzalloc(&pdev->dev, sizeof(*msi_data),
> GFP_KERNEL);
> > + if (!msi_data)
> > + return -ENOMEM;
> > +
> > + msi_data->cfg = cfg;
> > +
> > + msi_data->regs = devm_platform_ioremap_resource_byname(pdev,
> "a");
> > + if (IS_ERR(msi_data->regs)) {
> > + dev_err(&pdev->dev, "failed to initialize 'regs'\n");
> > + return PTR_ERR(msi_data->regs);
> > + }
> > +
> > + res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "b");
> > + if (!res)
> > + return -EIO;
> > +
> > + msi_data->msiir_addr = res->start + msi_data->cfg->xTR;
> > +
> > + msi_data->pdev = pdev;
> > +
> > + msi_data->gic_irq = platform_get_irq(msi_data->pdev, 0);
> > + if (msi_data->gic_irq <= 0)
> > + return -ENODEV;
> > +
> > + platform_set_drvdata(pdev, msi_data);
> > +
> > + msi_data->clk = devm_clk_get(dev, NULL);
> > + if (IS_ERR(msi_data->clk)) {
> > + if (PTR_ERR(msi_data->clk) != -ENOENT)
> > + return PTR_ERR(msi_data->clk);
> > +
> > + msi_data->clk = NULL;
> > + }
> > +
> > + ret = clk_prepare_enable(msi_data->clk);
> > + if (ret) {
> > + dev_err(dev, "Failed to enable clock\n");
> > + return ret;
> > + }
> > +
> > + priv->pd_a = dev_pm_domain_attach_by_name(dev, "a");
> > + if (IS_ERR(priv->pd_a))
> > + return PTR_ERR(priv->pd_a);
> > +
> > + priv->pd_link_a = device_link_add(dev, priv->pd_a,
> > + DL_FLAG_STATELESS |
> > + DL_FLAG_PM_RUNTIME |
> > + DL_FLAG_RPM_ACTIVE);
> > +
> > + if (!priv->pd_link_a) {
> > + dev_err(dev, "Failed to add device_link to mu a.\n");
> > + return -EINVAL;
> > + }
> > +
> > + priv->pd_b = dev_pm_domain_attach_by_name(dev, "b");
> > + if (IS_ERR(priv->pd_b))
> > + return PTR_ERR(priv->pd_b);
> > +
> > + priv->pd_link_b = device_link_add(dev, priv->pd_b,
> > + DL_FLAG_STATELESS |
> > + DL_FLAG_PM_RUNTIME |
> > + DL_FLAG_RPM_ACTIVE);
> > +
> > + if (!priv->pd_link_b) {
> > + dev_err(dev, "Failed to add device_link to mu a.\n");
> > + return -EINVAL;
> > + }
> > +
> > + ret = imx_mu_msi_domains_init(msi_data);
> > + if (ret)
> > + return ret;
> > +
> > + irq_set_chained_handler_and_data(msi_data->gic_irq,
> > + imx_mu_msi_irq_handler,
> > + msi_data);
> > +
> > + pm_runtime_enable(dev);
> > +
> > + ret = pm_runtime_get_sync(dev);
> > + if (ret < 0) {
> > + pm_runtime_put_noidle(dev);
> > + goto disable_runtime_pm;
> > + }
> > +
> > + ret = pm_runtime_put_sync(dev);
> > + if (ret < 0)
> > + goto disable_runtime_pm;
> > +
> > + clk_disable_unprepare(msi_data->clk);
>
> Why do you need to do all this dance?
>
> > +
> > + return 0;
> > +
> > +disable_runtime_pm:
> > + pm_runtime_disable(dev);
> > + clk_disable_unprepare(msi_data->clk);
> > +
> > + return ret;
> > +}
> > +
> > +static int __maybe_unused imx_mu_runtime_suspend(struct device *dev)
> > +{
> > + struct imx_mu_msi *priv = dev_get_drvdata(dev);
> > +
> > + clk_disable_unprepare(priv->clk);
> > +
> > + return 0;
> > +}
> > +
> > +static int __maybe_unused imx_mu_runtime_resume(struct device *dev)
> > +{
> > + struct imx_mu_msi *priv = dev_get_drvdata(dev);
> > + int ret;
> > +
> > + ret = clk_prepare_enable(priv->clk);
> > + if (ret)
> > + dev_err(dev, "failed to enable clock\n");
> > +
> > + return ret;
> > +}
> > +
> > +static const struct dev_pm_ops imx_mu_pm_ops = {
> > + SET_RUNTIME_PM_OPS(imx_mu_runtime_suspend,
> > + imx_mu_runtime_resume, NULL)
> > +};
> > +
> > +static int __init imx_mu_imx7ulp_of_init(struct device_node *dn,
> > + struct device_node *parent)
> > +{
> > + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx7ulp);
> > +}
> > +
> > +static int __init imx_mu_imx6sx_of_init(struct device_node *dn,
> > + struct device_node *parent)
> > +{
> > + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx6sx);
> > +}
> > +
> > +static int __init imx_mu_imx8ulp_of_init(struct device_node *dn,
> > + struct device_node *parent)
> > +{
> > + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx8ulp);
> > +}
> > +
> > +static int __init imx_mu_imx8ulp_s4_of_init(struct device_node *dn,
> > + struct device_node *parent)
> > +{
> > + return imx_mu_of_init(dn, parent, &imx_mu_cfg_imx8ulp_s4);
> > +}
> > +
> > +IRQCHIP_PLATFORM_DRIVER_BEGIN(imx_mu_msi)
> > +IRQCHIP_MATCH("fsl,imx7ulp-mu-msi", imx_mu_imx7ulp_of_init)
> > +IRQCHIP_MATCH("fsl,imx6sx-mu-msi", imx_mu_imx6sx_of_init)
> > +IRQCHIP_MATCH("fsl,imx8ulp-mu-msi", imx_mu_imx8ulp_of_init)
> > +IRQCHIP_MATCH("fsl,imx8ulp-mu-msi-s4", imx_mu_imx8ulp_s4_of_init)
> > +IRQCHIP_PLATFORM_DRIVER_END(imx_mu_msi, .pm =
> &imx_mu_pm_ops)
> > +
> > +
> > +MODULE_AUTHOR("Frank Li <Frank.Li@nxp.com>");
> > +MODULE_DESCRIPTION("Freescale MU work as MSI controller driver");
> > +MODULE_LICENSE("GPL");
>
> M.
>
> --
> Without deviation from the norm, progress is not possible.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
2022-07-21 15:22 ` Frank Li
@ 2022-07-21 15:28 ` Marc Zyngier
-1 siblings, 0 replies; 50+ messages in thread
From: Marc Zyngier @ 2022-07-21 15:28 UTC (permalink / raw)
To: Frank Li
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
On Thu, 21 Jul 2022 16:22:08 +0100,
Frank Li <frank.li@nxp.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Marc Zyngier <maz@kernel.org>
> > Sent: Thursday, July 21, 2022 2:57 AM
> > To: Frank Li <frank.li@nxp.com>
> > Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> > krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> > s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> > kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> > kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> > <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> > imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > ntb@lists.linux.dev
> > Subject: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
> >
> > Caution: EXT Email
> >
> > On Wed, 20 Jul 2022 22:30:34 +0100,
> > Frank Li <Frank.Li@nxp.com> wrote:
> > >
> > > MU support generate irq by write data to a register.
> > > This patch make mu worked as msi controller.
> > > So MU can do doorbell by using standard msi api.
> > >
> > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > ---
> > > drivers/irqchip/Kconfig | 7 +
> > > drivers/irqchip/Makefile | 1 +
> > > drivers/irqchip/irq-imx-mu-msi.c | 462
> > +++++++++++++++++++++++++++++++
> > > 3 files changed, 470 insertions(+)
> > > create mode 100644 drivers/irqchip/irq-imx-mu-msi.c
> > >
> > > diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
> > > index 5e4e50122777d..4599471d880c0 100644
> > > --- a/drivers/irqchip/Kconfig
> > > +++ b/drivers/irqchip/Kconfig
> > > @@ -470,6 +470,13 @@ config IMX_INTMUX
> > > help
> > > Support for the i.MX INTMUX interrupt multiplexer.
> > >
> > > +config IMX_MU_MSI
> > > + bool "i.MX MU work as MSI controller"
> > > + default y if ARCH_MXC
> > > + select IRQ_DOMAIN
> > > + help
> > > + MU work as MSI controller to do general doorbell
> > > +
> > > config LS1X_IRQ
> > > bool "Loongson-1 Interrupt Controller"
> > > depends on MACH_LOONGSON32
> > > diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
> > > index 5d8e21d3dc6d8..870423746c783 100644
> > > --- a/drivers/irqchip/Makefile
> > > +++ b/drivers/irqchip/Makefile
> > > @@ -98,6 +98,7 @@ obj-$(CONFIG_RISCV_INTC) += irq-riscv-intc.o
> > > obj-$(CONFIG_SIFIVE_PLIC) += irq-sifive-plic.o
> > > obj-$(CONFIG_IMX_IRQSTEER) += irq-imx-irqsteer.o
> > > obj-$(CONFIG_IMX_INTMUX) += irq-imx-intmux.o
> > > +obj-$(CONFIG_IMX_MU_MSI) += irq-imx-mu-msi.o
> > > obj-$(CONFIG_MADERA_IRQ) += irq-madera.o
> > > obj-$(CONFIG_LS1X_IRQ) += irq-ls1x.o
> > > obj-$(CONFIG_TI_SCI_INTR_IRQCHIP) += irq-ti-sci-intr.o
> > > diff --git a/drivers/irqchip/irq-imx-mu-msi.c b/drivers/irqchip/irq-imx-mu-
> > msi.c
> > > new file mode 100644
> > > index 0000000000000..8277dba857759
> > > --- /dev/null
> > > +++ b/drivers/irqchip/irq-imx-mu-msi.c
> > > @@ -0,0 +1,462 @@
> > > +// SPDX-License-Identifier: GPL-2.0-only
> > > +/*
> > > + * NXP MU worked as MSI controller
> > > + *
> > > + * Copyright (c) 2018 Pengutronix, Oleksij Rempel
> > <o.rempel@pengutronix.de>
> > > + * Copyright 2022 NXP
> > > + * Frank Li <Frank.Li@nxp.com>
> > > + * Peng Fan <peng.fan@nxp.com>
> > > + *
> > > + * Based on drivers/mailbox/imx-mailbox.c
> > > + */
> > > +#include <linux/clk.h>
> > > +#include <linux/kernel.h>
> > > +#include <linux/module.h>
> > > +#include <linux/msi.h>
> > > +#include <linux/interrupt.h>
> > > +#include <linux/irq.h>
> > > +#include <linux/irqchip/chained_irq.h>
> > > +#include <linux/irqchip.h>
> > > +#include <linux/irqdomain.h>
> > > +#include <linux/of_irq.h>
> > > +#include <linux/of_pci.h>
> > > +#include <linux/of_platform.h>
> > > +#include <linux/spinlock.h>
> > > +#include <linux/dma-iommu.h>
> > > +#include <linux/pm_runtime.h>
> > > +#include <linux/pm_domain.h>
> > > +
> > > +
> > > +#define IMX_MU_CHANS 4
> > > +
> > > +enum imx_mu_xcr {
> > > + IMX_MU_GIER,
> > > + IMX_MU_GCR,
> > > + IMX_MU_TCR,
> > > + IMX_MU_RCR,
> > > + IMX_MU_xCR_MAX,
> > > +};
> > > +
> > > +enum imx_mu_xsr {
> > > + IMX_MU_SR,
> > > + IMX_MU_GSR,
> > > + IMX_MU_TSR,
> > > + IMX_MU_RSR,
> > > +};
> > > +
> > > +enum imx_mu_type {
> > > + IMX_MU_V1 = BIT(0),
> > > + IMX_MU_V2 = BIT(1),
> > > + IMX_MU_V2_S4 = BIT(15),
> > > +};
> > > +
> > > +/* Receive Interrupt Enable */
> > > +#define IMX_MU_xCR_RIEn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24
> > + (3 - (x))))
> > > +#define IMX_MU_xSR_RFn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24 +
> > (3 - (x))))
> > > +
> > > +struct imx_mu_dcfg {
> > > + enum imx_mu_type type;
> > > + u32 xTR; /* Transmit Register0 */
> > > + u32 xRR; /* Receive Register0 */
> > > + u32 xSR[4]; /* Status Registers */
> > > + u32 xCR[4]; /* Control Registers */
> > > +};
> > > +
> > > +struct imx_mu_msi {
> > > + spinlock_t lock;
> > > + struct platform_device *pdev;
> > > + struct irq_domain *parent;
> > > + struct irq_domain *msi_domain;
> > > + void __iomem *regs;
> > > + phys_addr_t msiir_addr;
> > > + const struct imx_mu_dcfg *cfg;
> > > + unsigned long used;
> > > + u32 gic_irq;
> > > + struct clk *clk;
> > > + struct device *pd_a;
> > > + struct device *pd_b;
> > > + struct device_link *pd_link_a;
> > > + struct device_link *pd_link_b;
> > > +};
> > > +
> > > +static void imx_mu_write(struct imx_mu_msi *msi_data, u32 val, u32 offs)
> > > +{
> > > + iowrite32(val, msi_data->regs + offs);
> > > +}
> > > +
> > > +static u32 imx_mu_read(struct imx_mu_msi *msi_data, u32 offs)
> > > +{
> > > + return ioread32(msi_data->regs + offs);
> > > +}
> > > +
> > > +static u32 imx_mu_xcr_rmw(struct imx_mu_msi *msi_data, enum
> > imx_mu_xcr type, u32 set, u32 clr)
> > > +{
> > > + unsigned long flags;
> > > + u32 val;
> > > +
> > > + spin_lock_irqsave(&msi_data->lock, flags);
> > > + val = imx_mu_read(msi_data, msi_data->cfg->xCR[type]);
> > > + val &= ~clr;
> > > + val |= set;
> > > + imx_mu_write(msi_data, val, msi_data->cfg->xCR[type]);
> > > + spin_unlock_irqrestore(&msi_data->lock, flags);
> > > +
> > > + return val;
> > > +}
> > > +
> > > +static void imx_mu_msi_mask_irq(struct irq_data *data)
> > > +{
> > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> > >parent_data);
> >
> > Urgh... No. Please don't go poking into the *parent* stuff. Implement
> > the masking at the parent level, and use irq_chip_mask_parent() for
> > this level, unless you can explain why you can't do otherwise.
> >
> > > +
> > > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, 0,
> > IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq));
> >
> > How about making this readable and move the dereference inside the macro?
> >
> > > +}
> > > +
> > > +static void imx_mu_msi_unmask_irq(struct irq_data *data)
> > > +{
> > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> > >parent_data);
> > > +
> > > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR,
> > IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq), 0);
> > > +}
> > > +
> > > +static struct irq_chip imx_mu_msi_irq_chip = {
> > > + .name = "MU-MSI",
> > > + .irq_mask = imx_mu_msi_mask_irq,
> > > + .irq_unmask = imx_mu_msi_unmask_irq,
> > > +};
> > > +
> > > +static struct msi_domain_ops its_pmsi_ops = {
> > > +};
> >
> > What does this have to do with the ITS?
>
> Without this, it will be crashed as access 0 address.
Because the *name* of the structure has an influence on the behaviour
of the kernel?????
>
> >
> > > +
> > > +static struct msi_domain_info imx_mu_msi_domain_info = {
> > > + .flags = (MSI_FLAG_USE_DEF_DOM_OPS |
> > > + MSI_FLAG_USE_DEF_CHIP_OPS |
> > > + MSI_FLAG_PCI_MSIX),
> >
> > What does PCI_MSIX mean in this context? I really wish you used
> > copy/paste a bit more carefully.
> >
> > > + .ops = &its_pmsi_ops,
> > > + .chip = &imx_mu_msi_irq_chip,
> > > +};
> > > +
> > > +static void imx_mu_msi_compose_msg(struct irq_data *data, struct
> > msi_msg *msg)
> > > +{
> > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
> > > +
> > > + msg->address_hi = upper_32_bits(msi_data->msiir_addr);
> > > + msg->address_lo = lower_32_bits(msi_data->msiir_addr + 4 * data-
> > >hwirq);
> >
> > This is a horrible way of writing this. you should compute the address
> > first, and then apply the address split.
> >
> > > + msg->data = data->hwirq;
> > > +
> > > + iommu_dma_compose_msi_msg(irq_data_get_msi_desc(data), msg);
> > > +}
> > > +
> > > +static int imx_mu_msi_set_affinity(struct irq_data *irq_data,
> > > + const struct cpumask *mask, bool force)
> > > +
> > > +{
> > > + return IRQ_SET_MASK_OK;
> > > +}
> >
> > Err... What effect does this have if you don't do anything?
>
> [Frank Li] Without this, it will be crashed as access 0 address.
Then you should fix that bug, because what you have here makes
absolutely no sense.
>
> >
> > > +
> > > +static struct irq_chip imx_mu_msi_parent_chip = {
> > > + .name = "MU",
> > > + .irq_compose_msi_msg = imx_mu_msi_compose_msg,
> > > + .irq_set_affinity = imx_mu_msi_set_affinity,
> > > +};
> > > +
> > > +static int imx_mu_msi_domain_irq_alloc(struct irq_domain *domain,
> > > + unsigned int virq,
> > > + unsigned int nr_irqs,
> > > + void *args)
> > > +{
> > > + struct imx_mu_msi *msi_data = domain->host_data;
> > > + msi_alloc_info_t *info = args;
> > > + int pos, err = 0;
> > > +
> > > + WARN_ON(nr_irqs != 1);
> > > +
> > > + spin_lock(&msi_data->lock);
> >
> > Interrupt fires after lock acquisition, handler masks the interrupt.
> > Deadlock.
>
> [Frank Li] you are right, it should be spin_lock_irqsave.
>
> >
> > > + pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
> > > + if (pos < IMX_MU_CHANS)
> > > + __set_bit(pos, &msi_data->used);
> > > + else
> > > + err = -ENOSPC;
> > > + spin_unlock(&msi_data->lock);
> > > +
> > > + if (err)
> > > + return err;
> > > +
> > > + err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr +
> > pos * 4);
> >
> > Does this HW even have an IOMMU?
>
> [Frank Li] we have a platform with iommu.
>
> >
> > > + if (err)
> > > + return err;
> > > +
> > > + irq_domain_set_info(domain, virq, pos,
> > > + &imx_mu_msi_parent_chip, msi_data,
> > > + handle_simple_irq, NULL, NULL);
> >
> > This is an edge interrupt. Please handle it like one.
>
> [Frank Li] Not sure what your means?
A MSI is an edge interrupt. Use handle_edge_irq.
>
> >
> > > + return 0;
> > > +}
> > > +
> > > +static void imx_mu_msi_domain_irq_free(struct irq_domain *domain,
> > > + unsigned int virq, unsigned int nr_irqs)
> > > +{
> > > + struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(d);
> > > +
> > > + spin_lock(&msi_data->lock);
> >
> > Same problem.
> >
> > > + __clear_bit(d->hwirq, &msi_data->used);
> > > + spin_unlock(&msi_data->lock);
> > > +}
> > > +
> > > +static const struct irq_domain_ops imx_mu_msi_domain_ops = {
> > > + .alloc = imx_mu_msi_domain_irq_alloc,
> > > + .free = imx_mu_msi_domain_irq_free,
> > > +};
> > > +
> > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > +{
> > > + struct imx_mu_msi *msi_data = irq_desc_get_handler_data(desc);
> > > + u32 status;
> > > + int i;
> > > +
> > > + status = imx_mu_read(msi_data, msi_data->cfg->xSR[IMX_MU_RSR]);
> > > +
> > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > > + generic_handle_domain_irq(msi_data->parent, i);
> >
> > Why the parent? You must start at the top of the hierarchy.
> >
> > > + }
> > > + }
> > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> >
> > If your MSIs are a chained interrupt, why do you even provide an
> > affinity setting callback?
>
> [Frank Li] it will be crash if no affinity setting callback.
Then you have to fix your driver.
M.
--
Without deviation from the norm, progress is not possible.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
@ 2022-07-21 15:28 ` Marc Zyngier
0 siblings, 0 replies; 50+ messages in thread
From: Marc Zyngier @ 2022-07-21 15:28 UTC (permalink / raw)
To: Frank Li
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
On Thu, 21 Jul 2022 16:22:08 +0100,
Frank Li <frank.li@nxp.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Marc Zyngier <maz@kernel.org>
> > Sent: Thursday, July 21, 2022 2:57 AM
> > To: Frank Li <frank.li@nxp.com>
> > Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> > krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> > s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> > kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> > kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> > <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> > imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > ntb@lists.linux.dev
> > Subject: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
> >
> > Caution: EXT Email
> >
> > On Wed, 20 Jul 2022 22:30:34 +0100,
> > Frank Li <Frank.Li@nxp.com> wrote:
> > >
> > > MU support generate irq by write data to a register.
> > > This patch make mu worked as msi controller.
> > > So MU can do doorbell by using standard msi api.
> > >
> > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > ---
> > > drivers/irqchip/Kconfig | 7 +
> > > drivers/irqchip/Makefile | 1 +
> > > drivers/irqchip/irq-imx-mu-msi.c | 462
> > +++++++++++++++++++++++++++++++
> > > 3 files changed, 470 insertions(+)
> > > create mode 100644 drivers/irqchip/irq-imx-mu-msi.c
> > >
> > > diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
> > > index 5e4e50122777d..4599471d880c0 100644
> > > --- a/drivers/irqchip/Kconfig
> > > +++ b/drivers/irqchip/Kconfig
> > > @@ -470,6 +470,13 @@ config IMX_INTMUX
> > > help
> > > Support for the i.MX INTMUX interrupt multiplexer.
> > >
> > > +config IMX_MU_MSI
> > > + bool "i.MX MU work as MSI controller"
> > > + default y if ARCH_MXC
> > > + select IRQ_DOMAIN
> > > + help
> > > + MU work as MSI controller to do general doorbell
> > > +
> > > config LS1X_IRQ
> > > bool "Loongson-1 Interrupt Controller"
> > > depends on MACH_LOONGSON32
> > > diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
> > > index 5d8e21d3dc6d8..870423746c783 100644
> > > --- a/drivers/irqchip/Makefile
> > > +++ b/drivers/irqchip/Makefile
> > > @@ -98,6 +98,7 @@ obj-$(CONFIG_RISCV_INTC) += irq-riscv-intc.o
> > > obj-$(CONFIG_SIFIVE_PLIC) += irq-sifive-plic.o
> > > obj-$(CONFIG_IMX_IRQSTEER) += irq-imx-irqsteer.o
> > > obj-$(CONFIG_IMX_INTMUX) += irq-imx-intmux.o
> > > +obj-$(CONFIG_IMX_MU_MSI) += irq-imx-mu-msi.o
> > > obj-$(CONFIG_MADERA_IRQ) += irq-madera.o
> > > obj-$(CONFIG_LS1X_IRQ) += irq-ls1x.o
> > > obj-$(CONFIG_TI_SCI_INTR_IRQCHIP) += irq-ti-sci-intr.o
> > > diff --git a/drivers/irqchip/irq-imx-mu-msi.c b/drivers/irqchip/irq-imx-mu-
> > msi.c
> > > new file mode 100644
> > > index 0000000000000..8277dba857759
> > > --- /dev/null
> > > +++ b/drivers/irqchip/irq-imx-mu-msi.c
> > > @@ -0,0 +1,462 @@
> > > +// SPDX-License-Identifier: GPL-2.0-only
> > > +/*
> > > + * NXP MU worked as MSI controller
> > > + *
> > > + * Copyright (c) 2018 Pengutronix, Oleksij Rempel
> > <o.rempel@pengutronix.de>
> > > + * Copyright 2022 NXP
> > > + * Frank Li <Frank.Li@nxp.com>
> > > + * Peng Fan <peng.fan@nxp.com>
> > > + *
> > > + * Based on drivers/mailbox/imx-mailbox.c
> > > + */
> > > +#include <linux/clk.h>
> > > +#include <linux/kernel.h>
> > > +#include <linux/module.h>
> > > +#include <linux/msi.h>
> > > +#include <linux/interrupt.h>
> > > +#include <linux/irq.h>
> > > +#include <linux/irqchip/chained_irq.h>
> > > +#include <linux/irqchip.h>
> > > +#include <linux/irqdomain.h>
> > > +#include <linux/of_irq.h>
> > > +#include <linux/of_pci.h>
> > > +#include <linux/of_platform.h>
> > > +#include <linux/spinlock.h>
> > > +#include <linux/dma-iommu.h>
> > > +#include <linux/pm_runtime.h>
> > > +#include <linux/pm_domain.h>
> > > +
> > > +
> > > +#define IMX_MU_CHANS 4
> > > +
> > > +enum imx_mu_xcr {
> > > + IMX_MU_GIER,
> > > + IMX_MU_GCR,
> > > + IMX_MU_TCR,
> > > + IMX_MU_RCR,
> > > + IMX_MU_xCR_MAX,
> > > +};
> > > +
> > > +enum imx_mu_xsr {
> > > + IMX_MU_SR,
> > > + IMX_MU_GSR,
> > > + IMX_MU_TSR,
> > > + IMX_MU_RSR,
> > > +};
> > > +
> > > +enum imx_mu_type {
> > > + IMX_MU_V1 = BIT(0),
> > > + IMX_MU_V2 = BIT(1),
> > > + IMX_MU_V2_S4 = BIT(15),
> > > +};
> > > +
> > > +/* Receive Interrupt Enable */
> > > +#define IMX_MU_xCR_RIEn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24
> > + (3 - (x))))
> > > +#define IMX_MU_xSR_RFn(type, x) (type & IMX_MU_V2 ? BIT(x) : BIT(24 +
> > (3 - (x))))
> > > +
> > > +struct imx_mu_dcfg {
> > > + enum imx_mu_type type;
> > > + u32 xTR; /* Transmit Register0 */
> > > + u32 xRR; /* Receive Register0 */
> > > + u32 xSR[4]; /* Status Registers */
> > > + u32 xCR[4]; /* Control Registers */
> > > +};
> > > +
> > > +struct imx_mu_msi {
> > > + spinlock_t lock;
> > > + struct platform_device *pdev;
> > > + struct irq_domain *parent;
> > > + struct irq_domain *msi_domain;
> > > + void __iomem *regs;
> > > + phys_addr_t msiir_addr;
> > > + const struct imx_mu_dcfg *cfg;
> > > + unsigned long used;
> > > + u32 gic_irq;
> > > + struct clk *clk;
> > > + struct device *pd_a;
> > > + struct device *pd_b;
> > > + struct device_link *pd_link_a;
> > > + struct device_link *pd_link_b;
> > > +};
> > > +
> > > +static void imx_mu_write(struct imx_mu_msi *msi_data, u32 val, u32 offs)
> > > +{
> > > + iowrite32(val, msi_data->regs + offs);
> > > +}
> > > +
> > > +static u32 imx_mu_read(struct imx_mu_msi *msi_data, u32 offs)
> > > +{
> > > + return ioread32(msi_data->regs + offs);
> > > +}
> > > +
> > > +static u32 imx_mu_xcr_rmw(struct imx_mu_msi *msi_data, enum
> > imx_mu_xcr type, u32 set, u32 clr)
> > > +{
> > > + unsigned long flags;
> > > + u32 val;
> > > +
> > > + spin_lock_irqsave(&msi_data->lock, flags);
> > > + val = imx_mu_read(msi_data, msi_data->cfg->xCR[type]);
> > > + val &= ~clr;
> > > + val |= set;
> > > + imx_mu_write(msi_data, val, msi_data->cfg->xCR[type]);
> > > + spin_unlock_irqrestore(&msi_data->lock, flags);
> > > +
> > > + return val;
> > > +}
> > > +
> > > +static void imx_mu_msi_mask_irq(struct irq_data *data)
> > > +{
> > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> > >parent_data);
> >
> > Urgh... No. Please don't go poking into the *parent* stuff. Implement
> > the masking at the parent level, and use irq_chip_mask_parent() for
> > this level, unless you can explain why you can't do otherwise.
> >
> > > +
> > > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, 0,
> > IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq));
> >
> > How about making this readable and move the dereference inside the macro?
> >
> > > +}
> > > +
> > > +static void imx_mu_msi_unmask_irq(struct irq_data *data)
> > > +{
> > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> > >parent_data);
> > > +
> > > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR,
> > IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq), 0);
> > > +}
> > > +
> > > +static struct irq_chip imx_mu_msi_irq_chip = {
> > > + .name = "MU-MSI",
> > > + .irq_mask = imx_mu_msi_mask_irq,
> > > + .irq_unmask = imx_mu_msi_unmask_irq,
> > > +};
> > > +
> > > +static struct msi_domain_ops its_pmsi_ops = {
> > > +};
> >
> > What does this have to do with the ITS?
>
> Without this, it will be crashed as access 0 address.
Because the *name* of the structure has an influence on the behaviour
of the kernel?????
>
> >
> > > +
> > > +static struct msi_domain_info imx_mu_msi_domain_info = {
> > > + .flags = (MSI_FLAG_USE_DEF_DOM_OPS |
> > > + MSI_FLAG_USE_DEF_CHIP_OPS |
> > > + MSI_FLAG_PCI_MSIX),
> >
> > What does PCI_MSIX mean in this context? I really wish you used
> > copy/paste a bit more carefully.
> >
> > > + .ops = &its_pmsi_ops,
> > > + .chip = &imx_mu_msi_irq_chip,
> > > +};
> > > +
> > > +static void imx_mu_msi_compose_msg(struct irq_data *data, struct
> > msi_msg *msg)
> > > +{
> > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
> > > +
> > > + msg->address_hi = upper_32_bits(msi_data->msiir_addr);
> > > + msg->address_lo = lower_32_bits(msi_data->msiir_addr + 4 * data-
> > >hwirq);
> >
> > This is a horrible way of writing this. you should compute the address
> > first, and then apply the address split.
> >
> > > + msg->data = data->hwirq;
> > > +
> > > + iommu_dma_compose_msi_msg(irq_data_get_msi_desc(data), msg);
> > > +}
> > > +
> > > +static int imx_mu_msi_set_affinity(struct irq_data *irq_data,
> > > + const struct cpumask *mask, bool force)
> > > +
> > > +{
> > > + return IRQ_SET_MASK_OK;
> > > +}
> >
> > Err... What effect does this have if you don't do anything?
>
> [Frank Li] Without this, it will be crashed as access 0 address.
Then you should fix that bug, because what you have here makes
absolutely no sense.
>
> >
> > > +
> > > +static struct irq_chip imx_mu_msi_parent_chip = {
> > > + .name = "MU",
> > > + .irq_compose_msi_msg = imx_mu_msi_compose_msg,
> > > + .irq_set_affinity = imx_mu_msi_set_affinity,
> > > +};
> > > +
> > > +static int imx_mu_msi_domain_irq_alloc(struct irq_domain *domain,
> > > + unsigned int virq,
> > > + unsigned int nr_irqs,
> > > + void *args)
> > > +{
> > > + struct imx_mu_msi *msi_data = domain->host_data;
> > > + msi_alloc_info_t *info = args;
> > > + int pos, err = 0;
> > > +
> > > + WARN_ON(nr_irqs != 1);
> > > +
> > > + spin_lock(&msi_data->lock);
> >
> > Interrupt fires after lock acquisition, handler masks the interrupt.
> > Deadlock.
>
> [Frank Li] you are right, it should be spin_lock_irqsave.
>
> >
> > > + pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
> > > + if (pos < IMX_MU_CHANS)
> > > + __set_bit(pos, &msi_data->used);
> > > + else
> > > + err = -ENOSPC;
> > > + spin_unlock(&msi_data->lock);
> > > +
> > > + if (err)
> > > + return err;
> > > +
> > > + err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr +
> > pos * 4);
> >
> > Does this HW even have an IOMMU?
>
> [Frank Li] we have a platform with iommu.
>
> >
> > > + if (err)
> > > + return err;
> > > +
> > > + irq_domain_set_info(domain, virq, pos,
> > > + &imx_mu_msi_parent_chip, msi_data,
> > > + handle_simple_irq, NULL, NULL);
> >
> > This is an edge interrupt. Please handle it like one.
>
> [Frank Li] Not sure what your means?
A MSI is an edge interrupt. Use handle_edge_irq.
>
> >
> > > + return 0;
> > > +}
> > > +
> > > +static void imx_mu_msi_domain_irq_free(struct irq_domain *domain,
> > > + unsigned int virq, unsigned int nr_irqs)
> > > +{
> > > + struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(d);
> > > +
> > > + spin_lock(&msi_data->lock);
> >
> > Same problem.
> >
> > > + __clear_bit(d->hwirq, &msi_data->used);
> > > + spin_unlock(&msi_data->lock);
> > > +}
> > > +
> > > +static const struct irq_domain_ops imx_mu_msi_domain_ops = {
> > > + .alloc = imx_mu_msi_domain_irq_alloc,
> > > + .free = imx_mu_msi_domain_irq_free,
> > > +};
> > > +
> > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > +{
> > > + struct imx_mu_msi *msi_data = irq_desc_get_handler_data(desc);
> > > + u32 status;
> > > + int i;
> > > +
> > > + status = imx_mu_read(msi_data, msi_data->cfg->xSR[IMX_MU_RSR]);
> > > +
> > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > > + generic_handle_domain_irq(msi_data->parent, i);
> >
> > Why the parent? You must start at the top of the hierarchy.
> >
> > > + }
> > > + }
> > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> >
> > If your MSIs are a chained interrupt, why do you even provide an
> > affinity setting callback?
>
> [Frank Li] it will be crash if no affinity setting callback.
Then you have to fix your driver.
M.
--
Without deviation from the norm, progress is not possible.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
2022-07-21 15:28 ` Marc Zyngier
@ 2022-07-21 15:35 ` Frank Li
-1 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-21 15:35 UTC (permalink / raw)
To: Marc Zyngier
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Thursday, July 21, 2022 10:28 AM
> To: Frank Li <frank.li@nxp.com>
> Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
>
> Caution: EXT Email
>
> On Thu, 21 Jul 2022 16:22:08 +0100,
> Frank Li <frank.li@nxp.com> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Marc Zyngier <maz@kernel.org>
> > > Sent: Thursday, July 21, 2022 2:57 AM
> > > To: Frank Li <frank.li@nxp.com>
> > > Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> > > krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> > > s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> > > kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> > > kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> > > <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> > > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> > > imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > > ntb@lists.linux.dev
> > > Subject: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
> > >
> > > Caution: EXT Email
> > >
> > > On Wed, 20 Jul 2022 22:30:34 +0100,
> > > Frank Li <Frank.Li@nxp.com> wrote:
> > > >
> > > > MU support generate irq by write data to a register.
> > > > This patch make mu worked as msi controller.
> > > > So MU can do doorbell by using standard msi api.
> > > >
> > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > ---
> > > > drivers/irqchip/Kconfig | 7 +
> > > > drivers/irqchip/Makefile | 1 +
> > > > drivers/irqchip/irq-imx-mu-msi.c | 462
> > > +++++++++++++++++++++++++++++++
> > > > 3 files changed, 470 insertions(+)
> > > > create mode 100644 drivers/irqchip/irq-imx-mu-msi.c
> > > >
> > > > diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
> > > > index 5e4e50122777d..4599471d880c0 100644
> > > > --- a/drivers/irqchip/Kconfig
> > > > +++ b/drivers/irqchip/Kconfig
> > > > @@ -470,6 +470,13 @@ config IMX_INTMUX
> > > > help
> > > > Support for the i.MX INTMUX interrupt multiplexer.
> > > >
> > > > +config IMX_MU_MSI
> > > > + bool "i.MX MU work as MSI controller"
> > > > + default y if ARCH_MXC
> > > > + select IRQ_DOMAIN
> > > > + help
> > > > + MU work as MSI controller to do general doorbell
> > > > +
> > > > config LS1X_IRQ
> > > > bool "Loongson-1 Interrupt Controller"
> > > > depends on MACH_LOONGSON32
> > > > diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
> > > > index 5d8e21d3dc6d8..870423746c783 100644
> > > > --- a/drivers/irqchip/Makefile
> > > > +++ b/drivers/irqchip/Makefile
> > > > @@ -98,6 +98,7 @@ obj-$(CONFIG_RISCV_INTC) += irq-riscv-
> intc.o
> > > > obj-$(CONFIG_SIFIVE_PLIC) += irq-sifive-plic.o
> > > > obj-$(CONFIG_IMX_IRQSTEER) += irq-imx-irqsteer.o
> > > > obj-$(CONFIG_IMX_INTMUX) += irq-imx-intmux.o
> > > > +obj-$(CONFIG_IMX_MU_MSI) += irq-imx-mu-msi.o
> > > > obj-$(CONFIG_MADERA_IRQ) += irq-madera.o
> > > > obj-$(CONFIG_LS1X_IRQ) += irq-ls1x.o
> > > > obj-$(CONFIG_TI_SCI_INTR_IRQCHIP) += irq-ti-sci-intr.o
> > > > diff --git a/drivers/irqchip/irq-imx-mu-msi.c b/drivers/irqchip/irq-imx-
> mu-
> > > msi.c
> > > > new file mode 100644
> > > > index 0000000000000..8277dba857759
> > > > --- /dev/null
> > > > +++ b/drivers/irqchip/irq-imx-mu-msi.c
> > > > @@ -0,0 +1,462 @@
> > > > +// SPDX-License-Identifier: GPL-2.0-only
> > > > +/*
> > > > + * NXP MU worked as MSI controller
> > > > + *
> > > > + * Copyright (c) 2018 Pengutronix, Oleksij Rempel
> > > <o.rempel@pengutronix.de>
> > > > + * Copyright 2022 NXP
> > > > + * Frank Li <Frank.Li@nxp.com>
> > > > + * Peng Fan <peng.fan@nxp.com>
> > > > + *
> > > > + * Based on drivers/mailbox/imx-mailbox.c
> > > > + */
> > > > +#include <linux/clk.h>
> > > > +#include <linux/kernel.h>
> > > > +#include <linux/module.h>
> > > > +#include <linux/msi.h>
> > > > +#include <linux/interrupt.h>
> > > > +#include <linux/irq.h>
> > > > +#include <linux/irqchip/chained_irq.h>
> > > > +#include <linux/irqchip.h>
> > > > +#include <linux/irqdomain.h>
> > > > +#include <linux/of_irq.h>
> > > > +#include <linux/of_pci.h>
> > > > +#include <linux/of_platform.h>
> > > > +#include <linux/spinlock.h>
> > > > +#include <linux/dma-iommu.h>
> > > > +#include <linux/pm_runtime.h>
> > > > +#include <linux/pm_domain.h>
> > > > +
> > > > +
> > > > +#define IMX_MU_CHANS 4
> > > > +
> > > > +enum imx_mu_xcr {
> > > > + IMX_MU_GIER,
> > > > + IMX_MU_GCR,
> > > > + IMX_MU_TCR,
> > > > + IMX_MU_RCR,
> > > > + IMX_MU_xCR_MAX,
> > > > +};
> > > > +
> > > > +enum imx_mu_xsr {
> > > > + IMX_MU_SR,
> > > > + IMX_MU_GSR,
> > > > + IMX_MU_TSR,
> > > > + IMX_MU_RSR,
> > > > +};
> > > > +
> > > > +enum imx_mu_type {
> > > > + IMX_MU_V1 = BIT(0),
> > > > + IMX_MU_V2 = BIT(1),
> > > > + IMX_MU_V2_S4 = BIT(15),
> > > > +};
> > > > +
> > > > +/* Receive Interrupt Enable */
> > > > +#define IMX_MU_xCR_RIEn(type, x) (type & IMX_MU_V2 ? BIT(x) :
> BIT(24
> > > + (3 - (x))))
> > > > +#define IMX_MU_xSR_RFn(type, x) (type & IMX_MU_V2 ? BIT(x) :
> BIT(24 +
> > > (3 - (x))))
> > > > +
> > > > +struct imx_mu_dcfg {
> > > > + enum imx_mu_type type;
> > > > + u32 xTR; /* Transmit Register0 */
> > > > + u32 xRR; /* Receive Register0 */
> > > > + u32 xSR[4]; /* Status Registers */
> > > > + u32 xCR[4]; /* Control Registers */
> > > > +};
> > > > +
> > > > +struct imx_mu_msi {
> > > > + spinlock_t lock;
> > > > + struct platform_device *pdev;
> > > > + struct irq_domain *parent;
> > > > + struct irq_domain *msi_domain;
> > > > + void __iomem *regs;
> > > > + phys_addr_t msiir_addr;
> > > > + const struct imx_mu_dcfg *cfg;
> > > > + unsigned long used;
> > > > + u32 gic_irq;
> > > > + struct clk *clk;
> > > > + struct device *pd_a;
> > > > + struct device *pd_b;
> > > > + struct device_link *pd_link_a;
> > > > + struct device_link *pd_link_b;
> > > > +};
> > > > +
> > > > +static void imx_mu_write(struct imx_mu_msi *msi_data, u32 val, u32
> offs)
> > > > +{
> > > > + iowrite32(val, msi_data->regs + offs);
> > > > +}
> > > > +
> > > > +static u32 imx_mu_read(struct imx_mu_msi *msi_data, u32 offs)
> > > > +{
> > > > + return ioread32(msi_data->regs + offs);
> > > > +}
> > > > +
> > > > +static u32 imx_mu_xcr_rmw(struct imx_mu_msi *msi_data, enum
> > > imx_mu_xcr type, u32 set, u32 clr)
> > > > +{
> > > > + unsigned long flags;
> > > > + u32 val;
> > > > +
> > > > + spin_lock_irqsave(&msi_data->lock, flags);
> > > > + val = imx_mu_read(msi_data, msi_data->cfg->xCR[type]);
> > > > + val &= ~clr;
> > > > + val |= set;
> > > > + imx_mu_write(msi_data, val, msi_data->cfg->xCR[type]);
> > > > + spin_unlock_irqrestore(&msi_data->lock, flags);
> > > > +
> > > > + return val;
> > > > +}
> > > > +
> > > > +static void imx_mu_msi_mask_irq(struct irq_data *data)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> > > >parent_data);
> > >
> > > Urgh... No. Please don't go poking into the *parent* stuff. Implement
> > > the masking at the parent level, and use irq_chip_mask_parent() for
> > > this level, unless you can explain why you can't do otherwise.
> > >
> > > > +
> > > > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, 0,
> > > IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq));
> > >
> > > How about making this readable and move the dereference inside the
> macro?
> > >
> > > > +}
> > > > +
> > > > +static void imx_mu_msi_unmask_irq(struct irq_data *data)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> > > >parent_data);
> > > > +
> > > > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR,
> > > IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq), 0);
> > > > +}
> > > > +
> > > > +static struct irq_chip imx_mu_msi_irq_chip = {
> > > > + .name = "MU-MSI",
> > > > + .irq_mask = imx_mu_msi_mask_irq,
> > > > + .irq_unmask = imx_mu_msi_unmask_irq,
> > > > +};
> > > > +
> > > > +static struct msi_domain_ops its_pmsi_ops = {
> > > > +};
> > >
> > > What does this have to do with the ITS?
> >
> > Without this, it will be crashed as access 0 address.
>
> Because the *name* of the structure has an influence on the behaviour
> of the kernel?????
I understand your means now. The name "its_pmsi_ops" is wrong.
Not ask why empty structure here.
>
> >
> > >
> > > > +
> > > > +static struct msi_domain_info imx_mu_msi_domain_info = {
> > > > + .flags = (MSI_FLAG_USE_DEF_DOM_OPS |
> > > > + MSI_FLAG_USE_DEF_CHIP_OPS |
> > > > + MSI_FLAG_PCI_MSIX),
> > >
> > > What does PCI_MSIX mean in this context? I really wish you used
> > > copy/paste a bit more carefully.
> > >
> > > > + .ops = &its_pmsi_ops,
> > > > + .chip = &imx_mu_msi_irq_chip,
> > > > +};
> > > > +
> > > > +static void imx_mu_msi_compose_msg(struct irq_data *data, struct
> > > msi_msg *msg)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
> > > > +
> > > > + msg->address_hi = upper_32_bits(msi_data->msiir_addr);
> > > > + msg->address_lo = lower_32_bits(msi_data->msiir_addr + 4 * data-
> > > >hwirq);
> > >
> > > This is a horrible way of writing this. you should compute the address
> > > first, and then apply the address split.
> > >
> > > > + msg->data = data->hwirq;
> > > > +
> > > > + iommu_dma_compose_msi_msg(irq_data_get_msi_desc(data),
> msg);
> > > > +}
> > > > +
> > > > +static int imx_mu_msi_set_affinity(struct irq_data *irq_data,
> > > > + const struct cpumask *mask, bool force)
> > > > +
> > > > +{
> > > > + return IRQ_SET_MASK_OK;
> > > > +}
> > >
> > > Err... What effect does this have if you don't do anything?
> >
> > [Frank Li] Without this, it will be crashed as access 0 address.
>
> Then you should fix that bug, because what you have here makes
> absolutely no sense.
>
> >
> > >
> > > > +
> > > > +static struct irq_chip imx_mu_msi_parent_chip = {
> > > > + .name = "MU",
> > > > + .irq_compose_msi_msg = imx_mu_msi_compose_msg,
> > > > + .irq_set_affinity = imx_mu_msi_set_affinity,
> > > > +};
> > > > +
> > > > +static int imx_mu_msi_domain_irq_alloc(struct irq_domain *domain,
> > > > + unsigned int virq,
> > > > + unsigned int nr_irqs,
> > > > + void *args)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = domain->host_data;
> > > > + msi_alloc_info_t *info = args;
> > > > + int pos, err = 0;
> > > > +
> > > > + WARN_ON(nr_irqs != 1);
> > > > +
> > > > + spin_lock(&msi_data->lock);
> > >
> > > Interrupt fires after lock acquisition, handler masks the interrupt.
> > > Deadlock.
> >
> > [Frank Li] you are right, it should be spin_lock_irqsave.
> >
> > >
> > > > + pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
> > > > + if (pos < IMX_MU_CHANS)
> > > > + __set_bit(pos, &msi_data->used);
> > > > + else
> > > > + err = -ENOSPC;
> > > > + spin_unlock(&msi_data->lock);
> > > > +
> > > > + if (err)
> > > > + return err;
> > > > +
> > > > + err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr
> +
> > > pos * 4);
> > >
> > > Does this HW even have an IOMMU?
> >
> > [Frank Li] we have a platform with iommu.
> >
> > >
> > > > + if (err)
> > > > + return err;
> > > > +
> > > > + irq_domain_set_info(domain, virq, pos,
> > > > + &imx_mu_msi_parent_chip, msi_data,
> > > > + handle_simple_irq, NULL, NULL);
> > >
> > > This is an edge interrupt. Please handle it like one.
> >
> > [Frank Li] Not sure what your means?
>
> A MSI is an edge interrupt. Use handle_edge_irq.
>
> >
> > >
> > > > + return 0;
> > > > +}
> > > > +
> > > > +static void imx_mu_msi_domain_irq_free(struct irq_domain *domain,
> > > > + unsigned int virq, unsigned int nr_irqs)
> > > > +{
> > > > + struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(d);
> > > > +
> > > > + spin_lock(&msi_data->lock);
> > >
> > > Same problem.
> > >
> > > > + __clear_bit(d->hwirq, &msi_data->used);
> > > > + spin_unlock(&msi_data->lock);
> > > > +}
> > > > +
> > > > +static const struct irq_domain_ops imx_mu_msi_domain_ops = {
> > > > + .alloc = imx_mu_msi_domain_irq_alloc,
> > > > + .free = imx_mu_msi_domain_irq_free,
> > > > +};
> > > > +
> > > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_desc_get_handler_data(desc);
> > > > + u32 status;
> > > > + int i;
> > > > +
> > > > + status = imx_mu_read(msi_data, msi_data->cfg-
> >xSR[IMX_MU_RSR]);
> > > > +
> > > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > > > + generic_handle_domain_irq(msi_data->parent, i);
> > >
> > > Why the parent? You must start at the top of the hierarchy.
> > >
> > > > + }
> > > > + }
> > > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> > >
> > > If your MSIs are a chained interrupt, why do you even provide an
> > > affinity setting callback?
> >
> > [Frank Li] it will be crash if no affinity setting callback.
>
> Then you have to fix your driver.
>
> M.
>
> --
> Without deviation from the norm, progress is not possible.
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
@ 2022-07-21 15:35 ` Frank Li
0 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-21 15:35 UTC (permalink / raw)
To: Marc Zyngier
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Thursday, July 21, 2022 10:28 AM
> To: Frank Li <frank.li@nxp.com>
> Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
>
> Caution: EXT Email
>
> On Thu, 21 Jul 2022 16:22:08 +0100,
> Frank Li <frank.li@nxp.com> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Marc Zyngier <maz@kernel.org>
> > > Sent: Thursday, July 21, 2022 2:57 AM
> > > To: Frank Li <frank.li@nxp.com>
> > > Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> > > krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> > > s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> > > kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> > > kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> > > <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> > > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> > > imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > > ntb@lists.linux.dev
> > > Subject: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
> > >
> > > Caution: EXT Email
> > >
> > > On Wed, 20 Jul 2022 22:30:34 +0100,
> > > Frank Li <Frank.Li@nxp.com> wrote:
> > > >
> > > > MU support generate irq by write data to a register.
> > > > This patch make mu worked as msi controller.
> > > > So MU can do doorbell by using standard msi api.
> > > >
> > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > ---
> > > > drivers/irqchip/Kconfig | 7 +
> > > > drivers/irqchip/Makefile | 1 +
> > > > drivers/irqchip/irq-imx-mu-msi.c | 462
> > > +++++++++++++++++++++++++++++++
> > > > 3 files changed, 470 insertions(+)
> > > > create mode 100644 drivers/irqchip/irq-imx-mu-msi.c
> > > >
> > > > diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
> > > > index 5e4e50122777d..4599471d880c0 100644
> > > > --- a/drivers/irqchip/Kconfig
> > > > +++ b/drivers/irqchip/Kconfig
> > > > @@ -470,6 +470,13 @@ config IMX_INTMUX
> > > > help
> > > > Support for the i.MX INTMUX interrupt multiplexer.
> > > >
> > > > +config IMX_MU_MSI
> > > > + bool "i.MX MU work as MSI controller"
> > > > + default y if ARCH_MXC
> > > > + select IRQ_DOMAIN
> > > > + help
> > > > + MU work as MSI controller to do general doorbell
> > > > +
> > > > config LS1X_IRQ
> > > > bool "Loongson-1 Interrupt Controller"
> > > > depends on MACH_LOONGSON32
> > > > diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
> > > > index 5d8e21d3dc6d8..870423746c783 100644
> > > > --- a/drivers/irqchip/Makefile
> > > > +++ b/drivers/irqchip/Makefile
> > > > @@ -98,6 +98,7 @@ obj-$(CONFIG_RISCV_INTC) += irq-riscv-
> intc.o
> > > > obj-$(CONFIG_SIFIVE_PLIC) += irq-sifive-plic.o
> > > > obj-$(CONFIG_IMX_IRQSTEER) += irq-imx-irqsteer.o
> > > > obj-$(CONFIG_IMX_INTMUX) += irq-imx-intmux.o
> > > > +obj-$(CONFIG_IMX_MU_MSI) += irq-imx-mu-msi.o
> > > > obj-$(CONFIG_MADERA_IRQ) += irq-madera.o
> > > > obj-$(CONFIG_LS1X_IRQ) += irq-ls1x.o
> > > > obj-$(CONFIG_TI_SCI_INTR_IRQCHIP) += irq-ti-sci-intr.o
> > > > diff --git a/drivers/irqchip/irq-imx-mu-msi.c b/drivers/irqchip/irq-imx-
> mu-
> > > msi.c
> > > > new file mode 100644
> > > > index 0000000000000..8277dba857759
> > > > --- /dev/null
> > > > +++ b/drivers/irqchip/irq-imx-mu-msi.c
> > > > @@ -0,0 +1,462 @@
> > > > +// SPDX-License-Identifier: GPL-2.0-only
> > > > +/*
> > > > + * NXP MU worked as MSI controller
> > > > + *
> > > > + * Copyright (c) 2018 Pengutronix, Oleksij Rempel
> > > <o.rempel@pengutronix.de>
> > > > + * Copyright 2022 NXP
> > > > + * Frank Li <Frank.Li@nxp.com>
> > > > + * Peng Fan <peng.fan@nxp.com>
> > > > + *
> > > > + * Based on drivers/mailbox/imx-mailbox.c
> > > > + */
> > > > +#include <linux/clk.h>
> > > > +#include <linux/kernel.h>
> > > > +#include <linux/module.h>
> > > > +#include <linux/msi.h>
> > > > +#include <linux/interrupt.h>
> > > > +#include <linux/irq.h>
> > > > +#include <linux/irqchip/chained_irq.h>
> > > > +#include <linux/irqchip.h>
> > > > +#include <linux/irqdomain.h>
> > > > +#include <linux/of_irq.h>
> > > > +#include <linux/of_pci.h>
> > > > +#include <linux/of_platform.h>
> > > > +#include <linux/spinlock.h>
> > > > +#include <linux/dma-iommu.h>
> > > > +#include <linux/pm_runtime.h>
> > > > +#include <linux/pm_domain.h>
> > > > +
> > > > +
> > > > +#define IMX_MU_CHANS 4
> > > > +
> > > > +enum imx_mu_xcr {
> > > > + IMX_MU_GIER,
> > > > + IMX_MU_GCR,
> > > > + IMX_MU_TCR,
> > > > + IMX_MU_RCR,
> > > > + IMX_MU_xCR_MAX,
> > > > +};
> > > > +
> > > > +enum imx_mu_xsr {
> > > > + IMX_MU_SR,
> > > > + IMX_MU_GSR,
> > > > + IMX_MU_TSR,
> > > > + IMX_MU_RSR,
> > > > +};
> > > > +
> > > > +enum imx_mu_type {
> > > > + IMX_MU_V1 = BIT(0),
> > > > + IMX_MU_V2 = BIT(1),
> > > > + IMX_MU_V2_S4 = BIT(15),
> > > > +};
> > > > +
> > > > +/* Receive Interrupt Enable */
> > > > +#define IMX_MU_xCR_RIEn(type, x) (type & IMX_MU_V2 ? BIT(x) :
> BIT(24
> > > + (3 - (x))))
> > > > +#define IMX_MU_xSR_RFn(type, x) (type & IMX_MU_V2 ? BIT(x) :
> BIT(24 +
> > > (3 - (x))))
> > > > +
> > > > +struct imx_mu_dcfg {
> > > > + enum imx_mu_type type;
> > > > + u32 xTR; /* Transmit Register0 */
> > > > + u32 xRR; /* Receive Register0 */
> > > > + u32 xSR[4]; /* Status Registers */
> > > > + u32 xCR[4]; /* Control Registers */
> > > > +};
> > > > +
> > > > +struct imx_mu_msi {
> > > > + spinlock_t lock;
> > > > + struct platform_device *pdev;
> > > > + struct irq_domain *parent;
> > > > + struct irq_domain *msi_domain;
> > > > + void __iomem *regs;
> > > > + phys_addr_t msiir_addr;
> > > > + const struct imx_mu_dcfg *cfg;
> > > > + unsigned long used;
> > > > + u32 gic_irq;
> > > > + struct clk *clk;
> > > > + struct device *pd_a;
> > > > + struct device *pd_b;
> > > > + struct device_link *pd_link_a;
> > > > + struct device_link *pd_link_b;
> > > > +};
> > > > +
> > > > +static void imx_mu_write(struct imx_mu_msi *msi_data, u32 val, u32
> offs)
> > > > +{
> > > > + iowrite32(val, msi_data->regs + offs);
> > > > +}
> > > > +
> > > > +static u32 imx_mu_read(struct imx_mu_msi *msi_data, u32 offs)
> > > > +{
> > > > + return ioread32(msi_data->regs + offs);
> > > > +}
> > > > +
> > > > +static u32 imx_mu_xcr_rmw(struct imx_mu_msi *msi_data, enum
> > > imx_mu_xcr type, u32 set, u32 clr)
> > > > +{
> > > > + unsigned long flags;
> > > > + u32 val;
> > > > +
> > > > + spin_lock_irqsave(&msi_data->lock, flags);
> > > > + val = imx_mu_read(msi_data, msi_data->cfg->xCR[type]);
> > > > + val &= ~clr;
> > > > + val |= set;
> > > > + imx_mu_write(msi_data, val, msi_data->cfg->xCR[type]);
> > > > + spin_unlock_irqrestore(&msi_data->lock, flags);
> > > > +
> > > > + return val;
> > > > +}
> > > > +
> > > > +static void imx_mu_msi_mask_irq(struct irq_data *data)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> > > >parent_data);
> > >
> > > Urgh... No. Please don't go poking into the *parent* stuff. Implement
> > > the masking at the parent level, and use irq_chip_mask_parent() for
> > > this level, unless you can explain why you can't do otherwise.
> > >
> > > > +
> > > > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, 0,
> > > IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq));
> > >
> > > How about making this readable and move the dereference inside the
> macro?
> > >
> > > > +}
> > > > +
> > > > +static void imx_mu_msi_unmask_irq(struct irq_data *data)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> > > >parent_data);
> > > > +
> > > > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR,
> > > IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq), 0);
> > > > +}
> > > > +
> > > > +static struct irq_chip imx_mu_msi_irq_chip = {
> > > > + .name = "MU-MSI",
> > > > + .irq_mask = imx_mu_msi_mask_irq,
> > > > + .irq_unmask = imx_mu_msi_unmask_irq,
> > > > +};
> > > > +
> > > > +static struct msi_domain_ops its_pmsi_ops = {
> > > > +};
> > >
> > > What does this have to do with the ITS?
> >
> > Without this, it will be crashed as access 0 address.
>
> Because the *name* of the structure has an influence on the behaviour
> of the kernel?????
I understand your means now. The name "its_pmsi_ops" is wrong.
Not ask why empty structure here.
>
> >
> > >
> > > > +
> > > > +static struct msi_domain_info imx_mu_msi_domain_info = {
> > > > + .flags = (MSI_FLAG_USE_DEF_DOM_OPS |
> > > > + MSI_FLAG_USE_DEF_CHIP_OPS |
> > > > + MSI_FLAG_PCI_MSIX),
> > >
> > > What does PCI_MSIX mean in this context? I really wish you used
> > > copy/paste a bit more carefully.
> > >
> > > > + .ops = &its_pmsi_ops,
> > > > + .chip = &imx_mu_msi_irq_chip,
> > > > +};
> > > > +
> > > > +static void imx_mu_msi_compose_msg(struct irq_data *data, struct
> > > msi_msg *msg)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
> > > > +
> > > > + msg->address_hi = upper_32_bits(msi_data->msiir_addr);
> > > > + msg->address_lo = lower_32_bits(msi_data->msiir_addr + 4 * data-
> > > >hwirq);
> > >
> > > This is a horrible way of writing this. you should compute the address
> > > first, and then apply the address split.
> > >
> > > > + msg->data = data->hwirq;
> > > > +
> > > > + iommu_dma_compose_msi_msg(irq_data_get_msi_desc(data),
> msg);
> > > > +}
> > > > +
> > > > +static int imx_mu_msi_set_affinity(struct irq_data *irq_data,
> > > > + const struct cpumask *mask, bool force)
> > > > +
> > > > +{
> > > > + return IRQ_SET_MASK_OK;
> > > > +}
> > >
> > > Err... What effect does this have if you don't do anything?
> >
> > [Frank Li] Without this, it will be crashed as access 0 address.
>
> Then you should fix that bug, because what you have here makes
> absolutely no sense.
>
> >
> > >
> > > > +
> > > > +static struct irq_chip imx_mu_msi_parent_chip = {
> > > > + .name = "MU",
> > > > + .irq_compose_msi_msg = imx_mu_msi_compose_msg,
> > > > + .irq_set_affinity = imx_mu_msi_set_affinity,
> > > > +};
> > > > +
> > > > +static int imx_mu_msi_domain_irq_alloc(struct irq_domain *domain,
> > > > + unsigned int virq,
> > > > + unsigned int nr_irqs,
> > > > + void *args)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = domain->host_data;
> > > > + msi_alloc_info_t *info = args;
> > > > + int pos, err = 0;
> > > > +
> > > > + WARN_ON(nr_irqs != 1);
> > > > +
> > > > + spin_lock(&msi_data->lock);
> > >
> > > Interrupt fires after lock acquisition, handler masks the interrupt.
> > > Deadlock.
> >
> > [Frank Li] you are right, it should be spin_lock_irqsave.
> >
> > >
> > > > + pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
> > > > + if (pos < IMX_MU_CHANS)
> > > > + __set_bit(pos, &msi_data->used);
> > > > + else
> > > > + err = -ENOSPC;
> > > > + spin_unlock(&msi_data->lock);
> > > > +
> > > > + if (err)
> > > > + return err;
> > > > +
> > > > + err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr
> +
> > > pos * 4);
> > >
> > > Does this HW even have an IOMMU?
> >
> > [Frank Li] we have a platform with iommu.
> >
> > >
> > > > + if (err)
> > > > + return err;
> > > > +
> > > > + irq_domain_set_info(domain, virq, pos,
> > > > + &imx_mu_msi_parent_chip, msi_data,
> > > > + handle_simple_irq, NULL, NULL);
> > >
> > > This is an edge interrupt. Please handle it like one.
> >
> > [Frank Li] Not sure what your means?
>
> A MSI is an edge interrupt. Use handle_edge_irq.
>
> >
> > >
> > > > + return 0;
> > > > +}
> > > > +
> > > > +static void imx_mu_msi_domain_irq_free(struct irq_domain *domain,
> > > > + unsigned int virq, unsigned int nr_irqs)
> > > > +{
> > > > + struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(d);
> > > > +
> > > > + spin_lock(&msi_data->lock);
> > >
> > > Same problem.
> > >
> > > > + __clear_bit(d->hwirq, &msi_data->used);
> > > > + spin_unlock(&msi_data->lock);
> > > > +}
> > > > +
> > > > +static const struct irq_domain_ops imx_mu_msi_domain_ops = {
> > > > + .alloc = imx_mu_msi_domain_irq_alloc,
> > > > + .free = imx_mu_msi_domain_irq_free,
> > > > +};
> > > > +
> > > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_desc_get_handler_data(desc);
> > > > + u32 status;
> > > > + int i;
> > > > +
> > > > + status = imx_mu_read(msi_data, msi_data->cfg-
> >xSR[IMX_MU_RSR]);
> > > > +
> > > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > > > + generic_handle_domain_irq(msi_data->parent, i);
> > >
> > > Why the parent? You must start at the top of the hierarchy.
> > >
> > > > + }
> > > > + }
> > > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> > >
> > > If your MSIs are a chained interrupt, why do you even provide an
> > > affinity setting callback?
> >
> > [Frank Li] it will be crash if no affinity setting callback.
>
> Then you have to fix your driver.
>
> M.
>
> --
> Without deviation from the norm, progress is not possible.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
2022-07-21 15:22 ` Frank Li
@ 2022-07-22 7:33 ` Marc Zyngier
-1 siblings, 0 replies; 50+ messages in thread
From: Marc Zyngier @ 2022-07-22 7:33 UTC (permalink / raw)
To: Frank Li
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, devicetree, linux-arm-kernel, linux-pci,
Peng Fan, Aisheng Dong, kernel, festevam, dl-linux-imx, kishon,
lorenzo.pieralisi, ntb
On Thu, 21 Jul 2022 16:22:08 +0100,
Frank Li <frank.li@nxp.com> wrote:
>
> > > + pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
> > > + if (pos < IMX_MU_CHANS)
> > > + __set_bit(pos, &msi_data->used);
> > > + else
> > > + err = -ENOSPC;
> > > + spin_unlock(&msi_data->lock);
> > > +
> > > + if (err)
> > > + return err;
> > > +
> > > + err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr +
> > pos * 4);
> >
> > Does this HW even have an IOMMU?
>
> [Frank Li] we have a platform with iommu.
I really wonder whether you are taking me for a ride, or whether you
are completely misunderstanding what this IOMMU business is about.
This is a *CPU* writing to the device to generate an MSI. CPU
transactions cannot be translated by an IOMMU as they are not
(surprise!) IO devices. They are in control of their own translation,
contrary to devices that generate MSIs.
So what sort of translation do you expect this to setup? What StreamID
is getting used by the DMA framework? There is no answer to these
questions because they don't make any sense. None of it makes any
sense.
At best, you are simply copy-pasting things from various drivers
without understanding what they are all about. I suggest you stop
doing that and make use of your time working out the problem rather
than wasting the reviewers'.
M.
--
Without deviation from the norm, progress is not possible.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
@ 2022-07-22 7:33 ` Marc Zyngier
0 siblings, 0 replies; 50+ messages in thread
From: Marc Zyngier @ 2022-07-22 7:33 UTC (permalink / raw)
To: Frank Li
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, devicetree, linux-arm-kernel, linux-pci,
Peng Fan, Aisheng Dong, kernel, festevam, dl-linux-imx, kishon,
lorenzo.pieralisi, ntb
On Thu, 21 Jul 2022 16:22:08 +0100,
Frank Li <frank.li@nxp.com> wrote:
>
> > > + pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
> > > + if (pos < IMX_MU_CHANS)
> > > + __set_bit(pos, &msi_data->used);
> > > + else
> > > + err = -ENOSPC;
> > > + spin_unlock(&msi_data->lock);
> > > +
> > > + if (err)
> > > + return err;
> > > +
> > > + err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr +
> > pos * 4);
> >
> > Does this HW even have an IOMMU?
>
> [Frank Li] we have a platform with iommu.
I really wonder whether you are taking me for a ride, or whether you
are completely misunderstanding what this IOMMU business is about.
This is a *CPU* writing to the device to generate an MSI. CPU
transactions cannot be translated by an IOMMU as they are not
(surprise!) IO devices. They are in control of their own translation,
contrary to devices that generate MSIs.
So what sort of translation do you expect this to setup? What StreamID
is getting used by the DMA framework? There is no answer to these
questions because they don't make any sense. None of it makes any
sense.
At best, you are simply copy-pasting things from various drivers
without understanding what they are all about. I suggest you stop
doing that and make use of your time working out the problem rather
than wasting the reviewers'.
M.
--
Without deviation from the norm, progress is not possible.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
2022-07-22 7:33 ` Marc Zyngier
@ 2022-07-22 16:12 ` Frank Li
-1 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-22 16:12 UTC (permalink / raw)
To: Marc Zyngier
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, devicetree, linux-arm-kernel, linux-pci,
Peng Fan, Aisheng Dong, kernel, festevam, dl-linux-imx, kishon,
lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Friday, July 22, 2022 2:33 AM
> To: Frank Li <frank.li@nxp.com>
> Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> devicetree@vger.kernel.org; linux-arm-kernel@lists.infradead.org; linux-
> pci@vger.kernel.org; Peng Fan <peng.fan@nxp.com>; Aisheng Dong
> <aisheng.dong@nxp.com>; kernel@pengutronix.de; festevam@gmail.com;
> dl-linux-imx <linux-imx@nxp.com>; kishon@ti.com;
> lorenzo.pieralisi@arm.com; ntb@lists.linux.dev
> Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
>
> Caution: EXT Email
>
> On Thu, 21 Jul 2022 16:22:08 +0100,
> Frank Li <frank.li@nxp.com> wrote:
> >
> > > > + pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
> > > > + if (pos < IMX_MU_CHANS)
> > > > + __set_bit(pos, &msi_data->used);
> > > > + else
> > > > + err = -ENOSPC;
> > > > + spin_unlock(&msi_data->lock);
> > > > +
> > > > + if (err)
> > > > + return err;
> > > > +
> > > > + err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr
> +
> > > pos * 4);
> > >
> > > Does this HW even have an IOMMU?
> >
> > [Frank Li] we have a platform with iommu.
>
> I really wonder whether you are taking me for a ride, or whether you
> are completely misunderstanding what this IOMMU business is about.
>
> This is a *CPU* writing to the device to generate an MSI. CPU
> transactions cannot be translated by an IOMMU as they are not
> (surprise!) IO devices. They are in control of their own translation,
> contrary to devices that generate MSIs.
[Frank Li] I think that It is not *CPU* writing to the device to generate an MSI.
It is another bus master to write to generate IRQ. According to my understand,
Bus master (such as PCI EP devices) write to an address to trigger irq, OR
DMA setup DMA write descriptor at last one. Any address should be translate by
IOMMU if it is enabled. If not, this is empty function call.
>
> So what sort of translation do you expect this to setup? What StreamID
> is getting used by the DMA framework? There is no answer to these
> questions because they don't make any sense. None of it makes any
> sense.
>
> At best, you are simply copy-pasting things from various drivers
> without understanding what they are all about.
[Frank Li] it is first time to write irq chip driver, so I have to base on or reference
Other drivers to start development. Of course, there are some miss understand
During development. From functional call name, it make sense although I don't know
All detail behind that.
Any bus master access memory need go through IOMMU if enabled.
I have not tested it at IOMMU enabled platform yet. Some maintainer require
Added similar function call if a feature enabled when I do other upstream work.
So I kept as refer driver did. Maybe refer driver code is too old.
If there are problem, I can delete it. So far, it don't impact my test platform.
I can add fix patch if met problem at IOMMU enabled platform.
> I suggest you stop
> doing that and make use of your time working out the problem rather
> than wasting the reviewers'.
Thank you for your review and great comments. This driver function worked.
I know I made some stupid problem, it should be fixed before sent out.
I also want to make sure the overall idea worked firstly. So far no-one
Use MSI as doorbell from PCIe-EP function drivers yet.
>
> M.
>
> --
> Without deviation from the norm, progress is not possible.
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
@ 2022-07-22 16:12 ` Frank Li
0 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-22 16:12 UTC (permalink / raw)
To: Marc Zyngier
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, devicetree, linux-arm-kernel, linux-pci,
Peng Fan, Aisheng Dong, kernel, festevam, dl-linux-imx, kishon,
lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Friday, July 22, 2022 2:33 AM
> To: Frank Li <frank.li@nxp.com>
> Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> devicetree@vger.kernel.org; linux-arm-kernel@lists.infradead.org; linux-
> pci@vger.kernel.org; Peng Fan <peng.fan@nxp.com>; Aisheng Dong
> <aisheng.dong@nxp.com>; kernel@pengutronix.de; festevam@gmail.com;
> dl-linux-imx <linux-imx@nxp.com>; kishon@ti.com;
> lorenzo.pieralisi@arm.com; ntb@lists.linux.dev
> Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
>
> Caution: EXT Email
>
> On Thu, 21 Jul 2022 16:22:08 +0100,
> Frank Li <frank.li@nxp.com> wrote:
> >
> > > > + pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
> > > > + if (pos < IMX_MU_CHANS)
> > > > + __set_bit(pos, &msi_data->used);
> > > > + else
> > > > + err = -ENOSPC;
> > > > + spin_unlock(&msi_data->lock);
> > > > +
> > > > + if (err)
> > > > + return err;
> > > > +
> > > > + err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr
> +
> > > pos * 4);
> > >
> > > Does this HW even have an IOMMU?
> >
> > [Frank Li] we have a platform with iommu.
>
> I really wonder whether you are taking me for a ride, or whether you
> are completely misunderstanding what this IOMMU business is about.
>
> This is a *CPU* writing to the device to generate an MSI. CPU
> transactions cannot be translated by an IOMMU as they are not
> (surprise!) IO devices. They are in control of their own translation,
> contrary to devices that generate MSIs.
[Frank Li] I think that It is not *CPU* writing to the device to generate an MSI.
It is another bus master to write to generate IRQ. According to my understand,
Bus master (such as PCI EP devices) write to an address to trigger irq, OR
DMA setup DMA write descriptor at last one. Any address should be translate by
IOMMU if it is enabled. If not, this is empty function call.
>
> So what sort of translation do you expect this to setup? What StreamID
> is getting used by the DMA framework? There is no answer to these
> questions because they don't make any sense. None of it makes any
> sense.
>
> At best, you are simply copy-pasting things from various drivers
> without understanding what they are all about.
[Frank Li] it is first time to write irq chip driver, so I have to base on or reference
Other drivers to start development. Of course, there are some miss understand
During development. From functional call name, it make sense although I don't know
All detail behind that.
Any bus master access memory need go through IOMMU if enabled.
I have not tested it at IOMMU enabled platform yet. Some maintainer require
Added similar function call if a feature enabled when I do other upstream work.
So I kept as refer driver did. Maybe refer driver code is too old.
If there are problem, I can delete it. So far, it don't impact my test platform.
I can add fix patch if met problem at IOMMU enabled platform.
> I suggest you stop
> doing that and make use of your time working out the problem rather
> than wasting the reviewers'.
Thank you for your review and great comments. This driver function worked.
I know I made some stupid problem, it should be fixed before sent out.
I also want to make sure the overall idea worked firstly. So far no-one
Use MSI as doorbell from PCIe-EP function drivers yet.
>
> M.
>
> --
> Without deviation from the norm, progress is not possible.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
2022-07-20 21:30 ` Frank Li
@ 2022-07-23 18:50 ` Krzysztof Kozlowski
-1 siblings, 0 replies; 50+ messages in thread
From: Krzysztof Kozlowski @ 2022-07-23 18:50 UTC (permalink / raw)
To: Frank Li, jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt,
shawnguo, s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, peng.fan,
aisheng.dong, kernel, festevam, linux-imx, kishon,
lorenzo.pieralisi, ntb
On 20/07/2022 23:30, Frank Li wrote:
> imx mu support generate irq by write a register.
> provide msi controller support so other driver
> can use it by standard msi interface.
Please start sentences with capital letter. Unfortunately I don't
understand the sentences. Please describe shortly the hardware.
>
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---
> .../interrupt-controller/fsl,mu-msi.yaml | 88 +++++++++++++++++++
> 1 file changed, 88 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
>
> diff --git a/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
> new file mode 100644
> index 0000000000000..e125294243af3
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
> @@ -0,0 +1,88 @@
> +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +%YAML 1.2
> +---
> +$id: http://devicetree.org/schemas/interrupt-controller/fsl,mu-msi.yaml#
> +$schema: http://devicetree.org/meta-schemas/core.yaml#
> +
> +title: NXP i.MX Messaging Unit (MU) work as msi controller
> +
> +maintainers:
> + - Frank Li <Frank.Li@nxp.com>
> +
> +description: |
> + The Messaging Unit module enables two processors within the SoC to
> + communicate and coordinate by passing messages (e.g. data, status
> + and control) through the MU interface. The MU also provides the ability
> + for one processor to signal the other processor using interrupts.
> +
> + Because the MU manages the messaging between processors, the MU uses
> + different clocks (from each side of the different peripheral buses).
> + Therefore, the MU must synchronize the accesses from one side to the
> + other. The MU accomplishes synchronization using two sets of matching
> + registers (Processor A-facing, Processor B-facing).
> +
> + MU can work as msi interrupt controller to do doorbell
> +
> +allOf:
> + - $ref: /schemas/interrupt-controller/msi-controller.yaml#
> +
> +properties:
> + compatible:
> + enum:
> + - fsl,imx6sx-mu-msi
> + - fsl,imx7ulp-mu-msi
> + - fsl,imx8ulp-mu-msi
> + - fsl,imx8ulp-mu-msi-s4
> +
> + reg:
> + minItems: 2
Not minItems but maxItems in general, but anyway you need to actually
list and describe the items (and then skip min/max)
> +
> + reg-names:
> + items:
> + - const: a
> + - const: b
> +
> + interrupts:
> + maxItems: 1
> +
> + clocks:
> + maxItems: 1
> +
> + power-domains:
> + maxItems: 2
and here you correctly use maxItems, so why min in reg? Anyway, instead
you need to list and describe the items.
Actually I asked you this last time about interrupts, so you ignored
that comment.
> +
> + power-domain-names:
> + items:
> + - const: a
> + - const: b
> +
> + interrupt-controller: true
> +
> + msi-controller: true
> +
> +required:
> + - compatible
> + - reg
> + - interrupts
> + - msi-controller
> + - interrupt-controller
Why different order than used in properties?
Best regards,
Krzysztof
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
@ 2022-07-23 18:50 ` Krzysztof Kozlowski
0 siblings, 0 replies; 50+ messages in thread
From: Krzysztof Kozlowski @ 2022-07-23 18:50 UTC (permalink / raw)
To: Frank Li, jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt,
shawnguo, s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, peng.fan,
aisheng.dong, kernel, festevam, linux-imx, kishon,
lorenzo.pieralisi, ntb
On 20/07/2022 23:30, Frank Li wrote:
> imx mu support generate irq by write a register.
> provide msi controller support so other driver
> can use it by standard msi interface.
Please start sentences with capital letter. Unfortunately I don't
understand the sentences. Please describe shortly the hardware.
>
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---
> .../interrupt-controller/fsl,mu-msi.yaml | 88 +++++++++++++++++++
> 1 file changed, 88 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
>
> diff --git a/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
> new file mode 100644
> index 0000000000000..e125294243af3
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
> @@ -0,0 +1,88 @@
> +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +%YAML 1.2
> +---
> +$id: http://devicetree.org/schemas/interrupt-controller/fsl,mu-msi.yaml#
> +$schema: http://devicetree.org/meta-schemas/core.yaml#
> +
> +title: NXP i.MX Messaging Unit (MU) work as msi controller
> +
> +maintainers:
> + - Frank Li <Frank.Li@nxp.com>
> +
> +description: |
> + The Messaging Unit module enables two processors within the SoC to
> + communicate and coordinate by passing messages (e.g. data, status
> + and control) through the MU interface. The MU also provides the ability
> + for one processor to signal the other processor using interrupts.
> +
> + Because the MU manages the messaging between processors, the MU uses
> + different clocks (from each side of the different peripheral buses).
> + Therefore, the MU must synchronize the accesses from one side to the
> + other. The MU accomplishes synchronization using two sets of matching
> + registers (Processor A-facing, Processor B-facing).
> +
> + MU can work as msi interrupt controller to do doorbell
> +
> +allOf:
> + - $ref: /schemas/interrupt-controller/msi-controller.yaml#
> +
> +properties:
> + compatible:
> + enum:
> + - fsl,imx6sx-mu-msi
> + - fsl,imx7ulp-mu-msi
> + - fsl,imx8ulp-mu-msi
> + - fsl,imx8ulp-mu-msi-s4
> +
> + reg:
> + minItems: 2
Not minItems but maxItems in general, but anyway you need to actually
list and describe the items (and then skip min/max)
> +
> + reg-names:
> + items:
> + - const: a
> + - const: b
> +
> + interrupts:
> + maxItems: 1
> +
> + clocks:
> + maxItems: 1
> +
> + power-domains:
> + maxItems: 2
and here you correctly use maxItems, so why min in reg? Anyway, instead
you need to list and describe the items.
Actually I asked you this last time about interrupts, so you ignored
that comment.
> +
> + power-domain-names:
> + items:
> + - const: a
> + - const: b
> +
> + interrupt-controller: true
> +
> + msi-controller: true
> +
> +required:
> + - compatible
> + - reg
> + - interrupts
> + - msi-controller
> + - interrupt-controller
Why different order than used in properties?
Best regards,
Krzysztof
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
2022-07-23 18:50 ` Krzysztof Kozlowski
@ 2022-07-25 16:29 ` Frank Li
-1 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-25 16:29 UTC (permalink / raw)
To: Krzysztof Kozlowski, jdmason, maz, tglx, robh+dt,
krzysztof.kozlowski+dt, shawnguo, s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, Peng Fan,
Aisheng Dong, kernel, festevam, dl-linux-imx, kishon,
lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
> Sent: Saturday, July 23, 2022 1:50 PM
> To: Frank Li <frank.li@nxp.com>; jdmason@kudzu.us; maz@kernel.org;
> tglx@linutronix.de; robh+dt@kernel.org; krzysztof.kozlowski+dt@linaro.org;
> shawnguo@kernel.org; s.hauer@pengutronix.de; kw@linux.com;
> bhelgaas@google.com
> Cc: kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi
> controller
>
> Caution: EXT Email
>
> On 20/07/2022 23:30, Frank Li wrote:
> > imx mu support generate irq by write a register.
> > provide msi controller support so other driver
> > can use it by standard msi interface.
>
> Please start sentences with capital letter. Unfortunately I don't
> understand the sentences. Please describe shortly the hardware.
[Frank Li] MU have 4 registers and both side A and B. If write one of
Register, irq will be trigger at the other side.
For example, writle(a side reg1, 0). Then b side irq will be trigged.
>
>
> >
> > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > ---
> > .../interrupt-controller/fsl,mu-msi.yaml | 88 +++++++++++++++++++
> > 1 file changed, 88 insertions(+)
> > create mode 100644 Documentation/devicetree/bindings/interrupt-
> controller/fsl,mu-msi.yaml
> >
> > diff --git a/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> msi.yaml b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> msi.yaml
> > new file mode 100644
> > index 0000000000000..e125294243af3
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> msi.yaml
> > @@ -0,0 +1,88 @@
> > +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> > +%YAML 1.2
> > +---
> > +$id:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
> ree.org%2Fschemas%2Finterrupt-controller%2Ffsl%2Cmu-
> msi.yaml%23&data=05%7C01%7CFrank.Li%40nxp.com%7Cfcec12a0731c
> 454af5c308da6cdc2a0e%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0
> %7C637941990101591376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%
> 7C%7C&sdata=9h9nKyvsWaghry1hkpa5aaxVGYpx6xZRTxhN0S4uB50%3
> D&reserved=0
> > +$schema:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
> ree.org%2Fmeta-
> schemas%2Fcore.yaml%23&data=05%7C01%7CFrank.Li%40nxp.com%7
> Cfcec12a0731c454af5c308da6cdc2a0e%7C686ea1d3bc2b4c6fa92cd99c5c3016
> 35%7C0%7C0%7C637941990101591376%7CUnknown%7CTWFpbGZsb3d8eyJ
> WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> 7C3000%7C%7C%7C&sdata=wagM3hl8fpJSm%2Bibw6ENl5lNlQ9fVEHzS
> OlT%2Bjoridg%3D&reserved=0
> > +
> > +title: NXP i.MX Messaging Unit (MU) work as msi controller
> > +
> > +maintainers:
> > + - Frank Li <Frank.Li@nxp.com>
> > +
> > +description: |
> > + The Messaging Unit module enables two processors within the SoC to
> > + communicate and coordinate by passing messages (e.g. data, status
> > + and control) through the MU interface. The MU also provides the ability
> > + for one processor to signal the other processor using interrupts.
> > +
> > + Because the MU manages the messaging between processors, the MU
> uses
> > + different clocks (from each side of the different peripheral buses).
> > + Therefore, the MU must synchronize the accesses from one side to the
> > + other. The MU accomplishes synchronization using two sets of matching
> > + registers (Processor A-facing, Processor B-facing).
> > +
> > + MU can work as msi interrupt controller to do doorbell
> > +
> > +allOf:
> > + - $ref: /schemas/interrupt-controller/msi-controller.yaml#
> > +
> > +properties:
> > + compatible:
> > + enum:
> > + - fsl,imx6sx-mu-msi
> > + - fsl,imx7ulp-mu-msi
> > + - fsl,imx8ulp-mu-msi
> > + - fsl,imx8ulp-mu-msi-s4
> > +
> > + reg:
> > + minItems: 2
>
> Not minItems but maxItems in general, but anyway you need to actually
> list and describe the items (and then skip min/max)
[Frank Li]
I am not sure format. Any example?
Reg:
Items:
- description: a side register
- description: b side register
>
> > +
> > + reg-names:
> > + items:
> > + - const: a
> > + - const: b
> > +
> > + interrupts:
> > + maxItems: 1
> > +
> > + clocks:
> > + maxItems: 1
> > +
> > + power-domains:
> > + maxItems: 2
>
> and here you correctly use maxItems, so why min in reg? Anyway, instead
> you need to list and describe the items.
Does format is similar with reg?
>
> Actually I asked you this last time about interrupts, so you ignored
> that comment.
Sorry, which one. Is it below one?
---
> + interrupts:
> + minItems: 1
> + maxItems: 2
Instead describe the items.
---
>
> > +
> > + power-domain-names:
> > + items:
> > + - const: a
> > + - const: b
> > +
> > + interrupt-controller: true
> > +
> > + msi-controller: true
> > +
> > +required:
> > + - compatible
> > + - reg
> > + - interrupts
> > + - msi-controller
> > + - interrupt-controller
>
> Why different order than used in properties?
>
>
>
> Best regards,
> Krzysztof
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
@ 2022-07-25 16:29 ` Frank Li
0 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-25 16:29 UTC (permalink / raw)
To: Krzysztof Kozlowski, jdmason, maz, tglx, robh+dt,
krzysztof.kozlowski+dt, shawnguo, s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, Peng Fan,
Aisheng Dong, kernel, festevam, dl-linux-imx, kishon,
lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
> Sent: Saturday, July 23, 2022 1:50 PM
> To: Frank Li <frank.li@nxp.com>; jdmason@kudzu.us; maz@kernel.org;
> tglx@linutronix.de; robh+dt@kernel.org; krzysztof.kozlowski+dt@linaro.org;
> shawnguo@kernel.org; s.hauer@pengutronix.de; kw@linux.com;
> bhelgaas@google.com
> Cc: kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi
> controller
>
> Caution: EXT Email
>
> On 20/07/2022 23:30, Frank Li wrote:
> > imx mu support generate irq by write a register.
> > provide msi controller support so other driver
> > can use it by standard msi interface.
>
> Please start sentences with capital letter. Unfortunately I don't
> understand the sentences. Please describe shortly the hardware.
[Frank Li] MU have 4 registers and both side A and B. If write one of
Register, irq will be trigger at the other side.
For example, writle(a side reg1, 0). Then b side irq will be trigged.
>
>
> >
> > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > ---
> > .../interrupt-controller/fsl,mu-msi.yaml | 88 +++++++++++++++++++
> > 1 file changed, 88 insertions(+)
> > create mode 100644 Documentation/devicetree/bindings/interrupt-
> controller/fsl,mu-msi.yaml
> >
> > diff --git a/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> msi.yaml b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> msi.yaml
> > new file mode 100644
> > index 0000000000000..e125294243af3
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> msi.yaml
> > @@ -0,0 +1,88 @@
> > +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> > +%YAML 1.2
> > +---
> > +$id:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
> ree.org%2Fschemas%2Finterrupt-controller%2Ffsl%2Cmu-
> msi.yaml%23&data=05%7C01%7CFrank.Li%40nxp.com%7Cfcec12a0731c
> 454af5c308da6cdc2a0e%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0
> %7C637941990101591376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%
> 7C%7C&sdata=9h9nKyvsWaghry1hkpa5aaxVGYpx6xZRTxhN0S4uB50%3
> D&reserved=0
> > +$schema:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
> ree.org%2Fmeta-
> schemas%2Fcore.yaml%23&data=05%7C01%7CFrank.Li%40nxp.com%7
> Cfcec12a0731c454af5c308da6cdc2a0e%7C686ea1d3bc2b4c6fa92cd99c5c3016
> 35%7C0%7C0%7C637941990101591376%7CUnknown%7CTWFpbGZsb3d8eyJ
> WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> 7C3000%7C%7C%7C&sdata=wagM3hl8fpJSm%2Bibw6ENl5lNlQ9fVEHzS
> OlT%2Bjoridg%3D&reserved=0
> > +
> > +title: NXP i.MX Messaging Unit (MU) work as msi controller
> > +
> > +maintainers:
> > + - Frank Li <Frank.Li@nxp.com>
> > +
> > +description: |
> > + The Messaging Unit module enables two processors within the SoC to
> > + communicate and coordinate by passing messages (e.g. data, status
> > + and control) through the MU interface. The MU also provides the ability
> > + for one processor to signal the other processor using interrupts.
> > +
> > + Because the MU manages the messaging between processors, the MU
> uses
> > + different clocks (from each side of the different peripheral buses).
> > + Therefore, the MU must synchronize the accesses from one side to the
> > + other. The MU accomplishes synchronization using two sets of matching
> > + registers (Processor A-facing, Processor B-facing).
> > +
> > + MU can work as msi interrupt controller to do doorbell
> > +
> > +allOf:
> > + - $ref: /schemas/interrupt-controller/msi-controller.yaml#
> > +
> > +properties:
> > + compatible:
> > + enum:
> > + - fsl,imx6sx-mu-msi
> > + - fsl,imx7ulp-mu-msi
> > + - fsl,imx8ulp-mu-msi
> > + - fsl,imx8ulp-mu-msi-s4
> > +
> > + reg:
> > + minItems: 2
>
> Not minItems but maxItems in general, but anyway you need to actually
> list and describe the items (and then skip min/max)
[Frank Li]
I am not sure format. Any example?
Reg:
Items:
- description: a side register
- description: b side register
>
> > +
> > + reg-names:
> > + items:
> > + - const: a
> > + - const: b
> > +
> > + interrupts:
> > + maxItems: 1
> > +
> > + clocks:
> > + maxItems: 1
> > +
> > + power-domains:
> > + maxItems: 2
>
> and here you correctly use maxItems, so why min in reg? Anyway, instead
> you need to list and describe the items.
Does format is similar with reg?
>
> Actually I asked you this last time about interrupts, so you ignored
> that comment.
Sorry, which one. Is it below one?
---
> + interrupts:
> + minItems: 1
> + maxItems: 2
Instead describe the items.
---
>
> > +
> > + power-domain-names:
> > + items:
> > + - const: a
> > + - const: b
> > +
> > + interrupt-controller: true
> > +
> > + msi-controller: true
> > +
> > +required:
> > + - compatible
> > + - reg
> > + - interrupts
> > + - msi-controller
> > + - interrupt-controller
>
> Why different order than used in properties?
>
>
>
> Best regards,
> Krzysztof
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
2022-07-25 16:29 ` Frank Li
@ 2022-07-25 16:44 ` Krzysztof Kozlowski
-1 siblings, 0 replies; 50+ messages in thread
From: Krzysztof Kozlowski @ 2022-07-25 16:44 UTC (permalink / raw)
To: Frank Li, jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt,
shawnguo, s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, Peng Fan,
Aisheng Dong, kernel, festevam, dl-linux-imx, kishon,
lorenzo.pieralisi, ntb
On 25/07/2022 18:29, Frank Li wrote:
>
>
>> -----Original Message-----
>> From: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
>> Sent: Saturday, July 23, 2022 1:50 PM
>> To: Frank Li <frank.li@nxp.com>; jdmason@kudzu.us; maz@kernel.org;
>> tglx@linutronix.de; robh+dt@kernel.org; krzysztof.kozlowski+dt@linaro.org;
>> shawnguo@kernel.org; s.hauer@pengutronix.de; kw@linux.com;
>> bhelgaas@google.com
>> Cc: kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
>> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
>> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
>> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
>> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
>> ntb@lists.linux.dev
>> Subject: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi
>> controller
>>
>> Caution: EXT Email
>>
>> On 20/07/2022 23:30, Frank Li wrote:
>>> imx mu support generate irq by write a register.
>>> provide msi controller support so other driver
>>> can use it by standard msi interface.
>>
>> Please start sentences with capital letter. Unfortunately I don't
>> understand the sentences. Please describe shortly the hardware.
>
> [Frank Li] MU have 4 registers and both side A and B. If write one of
> Register, irq will be trigger at the other side.
>
> For example, writle(a side reg1, 0). Then b side irq will be trigged.
>
>>
>>
>>>
>>> Signed-off-by: Frank Li <Frank.Li@nxp.com>
>>> ---
>>> .../interrupt-controller/fsl,mu-msi.yaml | 88 +++++++++++++++++++
>>> 1 file changed, 88 insertions(+)
>>> create mode 100644 Documentation/devicetree/bindings/interrupt-
>> controller/fsl,mu-msi.yaml
>>>
>>> diff --git a/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
>> msi.yaml b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
>> msi.yaml
>>> new file mode 100644
>>> index 0000000000000..e125294243af3
>>> --- /dev/null
>>> +++ b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
>> msi.yaml
>>> @@ -0,0 +1,88 @@
>>> +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
>>> +%YAML 1.2
>>> +---
>>> +$id:
>> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
>> ree.org%2Fschemas%2Finterrupt-controller%2Ffsl%2Cmu-
>> msi.yaml%23&data=05%7C01%7CFrank.Li%40nxp.com%7Cfcec12a0731c
>> 454af5c308da6cdc2a0e%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0
>> %7C637941990101591376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
>> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%
>> 7C%7C&sdata=9h9nKyvsWaghry1hkpa5aaxVGYpx6xZRTxhN0S4uB50%3
>> D&reserved=0
>>> +$schema:
>> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
>> ree.org%2Fmeta-
>> schemas%2Fcore.yaml%23&data=05%7C01%7CFrank.Li%40nxp.com%7
>> Cfcec12a0731c454af5c308da6cdc2a0e%7C686ea1d3bc2b4c6fa92cd99c5c3016
>> 35%7C0%7C0%7C637941990101591376%7CUnknown%7CTWFpbGZsb3d8eyJ
>> WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
>> 7C3000%7C%7C%7C&sdata=wagM3hl8fpJSm%2Bibw6ENl5lNlQ9fVEHzS
>> OlT%2Bjoridg%3D&reserved=0
>>> +
>>> +title: NXP i.MX Messaging Unit (MU) work as msi controller
>>> +
>>> +maintainers:
>>> + - Frank Li <Frank.Li@nxp.com>
>>> +
>>> +description: |
>>> + The Messaging Unit module enables two processors within the SoC to
>>> + communicate and coordinate by passing messages (e.g. data, status
>>> + and control) through the MU interface. The MU also provides the ability
>>> + for one processor to signal the other processor using interrupts.
>>> +
>>> + Because the MU manages the messaging between processors, the MU
>> uses
>>> + different clocks (from each side of the different peripheral buses).
>>> + Therefore, the MU must synchronize the accesses from one side to the
>>> + other. The MU accomplishes synchronization using two sets of matching
>>> + registers (Processor A-facing, Processor B-facing).
>>> +
>>> + MU can work as msi interrupt controller to do doorbell
>>> +
>>> +allOf:
>>> + - $ref: /schemas/interrupt-controller/msi-controller.yaml#
>>> +
>>> +properties:
>>> + compatible:
>>> + enum:
>>> + - fsl,imx6sx-mu-msi
>>> + - fsl,imx7ulp-mu-msi
>>> + - fsl,imx8ulp-mu-msi
>>> + - fsl,imx8ulp-mu-msi-s4
>>> +
>>> + reg:
>>> + minItems: 2
>>
>> Not minItems but maxItems in general, but anyway you need to actually
>> list and describe the items (and then skip min/max)
> [Frank Li]
> I am not sure format. Any example?
>
> Reg:
> Items:
> - description: a side register
> - description: b side register
Yes, but then explain what is A and B in bindings description.
Why MU, which sits on A side needs to access other side (B) registers?
>>
>>> +
>>> + reg-names:
>>> + items:
>>> + - const: a
>>> + - const: b
>>> +
>>> + interrupts:
>>> + maxItems: 1
>>> +
>>> + clocks:
>>> + maxItems: 1
>>> +
>>> + power-domains:
>>> + maxItems: 2
>>
>> and here you correctly use maxItems, so why min in reg? Anyway, instead
>> you need to list and describe the items.
>
> Does format is similar with reg?
Yes.
>
>>
>> Actually I asked you this last time about interrupts, so you ignored
>> that comment.
>
> Sorry, which one. Is it below one?
>
> ---
>> + interrupts:
>> + minItems: 1
>> + maxItems: 2
>
> Instead describe the items.
Yes.
Best regards,
Krzysztof
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
@ 2022-07-25 16:44 ` Krzysztof Kozlowski
0 siblings, 0 replies; 50+ messages in thread
From: Krzysztof Kozlowski @ 2022-07-25 16:44 UTC (permalink / raw)
To: Frank Li, jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt,
shawnguo, s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, Peng Fan,
Aisheng Dong, kernel, festevam, dl-linux-imx, kishon,
lorenzo.pieralisi, ntb
On 25/07/2022 18:29, Frank Li wrote:
>
>
>> -----Original Message-----
>> From: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
>> Sent: Saturday, July 23, 2022 1:50 PM
>> To: Frank Li <frank.li@nxp.com>; jdmason@kudzu.us; maz@kernel.org;
>> tglx@linutronix.de; robh+dt@kernel.org; krzysztof.kozlowski+dt@linaro.org;
>> shawnguo@kernel.org; s.hauer@pengutronix.de; kw@linux.com;
>> bhelgaas@google.com
>> Cc: kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
>> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
>> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
>> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
>> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
>> ntb@lists.linux.dev
>> Subject: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi
>> controller
>>
>> Caution: EXT Email
>>
>> On 20/07/2022 23:30, Frank Li wrote:
>>> imx mu support generate irq by write a register.
>>> provide msi controller support so other driver
>>> can use it by standard msi interface.
>>
>> Please start sentences with capital letter. Unfortunately I don't
>> understand the sentences. Please describe shortly the hardware.
>
> [Frank Li] MU have 4 registers and both side A and B. If write one of
> Register, irq will be trigger at the other side.
>
> For example, writle(a side reg1, 0). Then b side irq will be trigged.
>
>>
>>
>>>
>>> Signed-off-by: Frank Li <Frank.Li@nxp.com>
>>> ---
>>> .../interrupt-controller/fsl,mu-msi.yaml | 88 +++++++++++++++++++
>>> 1 file changed, 88 insertions(+)
>>> create mode 100644 Documentation/devicetree/bindings/interrupt-
>> controller/fsl,mu-msi.yaml
>>>
>>> diff --git a/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
>> msi.yaml b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
>> msi.yaml
>>> new file mode 100644
>>> index 0000000000000..e125294243af3
>>> --- /dev/null
>>> +++ b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
>> msi.yaml
>>> @@ -0,0 +1,88 @@
>>> +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
>>> +%YAML 1.2
>>> +---
>>> +$id:
>> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
>> ree.org%2Fschemas%2Finterrupt-controller%2Ffsl%2Cmu-
>> msi.yaml%23&data=05%7C01%7CFrank.Li%40nxp.com%7Cfcec12a0731c
>> 454af5c308da6cdc2a0e%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0
>> %7C637941990101591376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
>> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%
>> 7C%7C&sdata=9h9nKyvsWaghry1hkpa5aaxVGYpx6xZRTxhN0S4uB50%3
>> D&reserved=0
>>> +$schema:
>> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
>> ree.org%2Fmeta-
>> schemas%2Fcore.yaml%23&data=05%7C01%7CFrank.Li%40nxp.com%7
>> Cfcec12a0731c454af5c308da6cdc2a0e%7C686ea1d3bc2b4c6fa92cd99c5c3016
>> 35%7C0%7C0%7C637941990101591376%7CUnknown%7CTWFpbGZsb3d8eyJ
>> WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
>> 7C3000%7C%7C%7C&sdata=wagM3hl8fpJSm%2Bibw6ENl5lNlQ9fVEHzS
>> OlT%2Bjoridg%3D&reserved=0
>>> +
>>> +title: NXP i.MX Messaging Unit (MU) work as msi controller
>>> +
>>> +maintainers:
>>> + - Frank Li <Frank.Li@nxp.com>
>>> +
>>> +description: |
>>> + The Messaging Unit module enables two processors within the SoC to
>>> + communicate and coordinate by passing messages (e.g. data, status
>>> + and control) through the MU interface. The MU also provides the ability
>>> + for one processor to signal the other processor using interrupts.
>>> +
>>> + Because the MU manages the messaging between processors, the MU
>> uses
>>> + different clocks (from each side of the different peripheral buses).
>>> + Therefore, the MU must synchronize the accesses from one side to the
>>> + other. The MU accomplishes synchronization using two sets of matching
>>> + registers (Processor A-facing, Processor B-facing).
>>> +
>>> + MU can work as msi interrupt controller to do doorbell
>>> +
>>> +allOf:
>>> + - $ref: /schemas/interrupt-controller/msi-controller.yaml#
>>> +
>>> +properties:
>>> + compatible:
>>> + enum:
>>> + - fsl,imx6sx-mu-msi
>>> + - fsl,imx7ulp-mu-msi
>>> + - fsl,imx8ulp-mu-msi
>>> + - fsl,imx8ulp-mu-msi-s4
>>> +
>>> + reg:
>>> + minItems: 2
>>
>> Not minItems but maxItems in general, but anyway you need to actually
>> list and describe the items (and then skip min/max)
> [Frank Li]
> I am not sure format. Any example?
>
> Reg:
> Items:
> - description: a side register
> - description: b side register
Yes, but then explain what is A and B in bindings description.
Why MU, which sits on A side needs to access other side (B) registers?
>>
>>> +
>>> + reg-names:
>>> + items:
>>> + - const: a
>>> + - const: b
>>> +
>>> + interrupts:
>>> + maxItems: 1
>>> +
>>> + clocks:
>>> + maxItems: 1
>>> +
>>> + power-domains:
>>> + maxItems: 2
>>
>> and here you correctly use maxItems, so why min in reg? Anyway, instead
>> you need to list and describe the items.
>
> Does format is similar with reg?
Yes.
>
>>
>> Actually I asked you this last time about interrupts, so you ignored
>> that comment.
>
> Sorry, which one. Is it below one?
>
> ---
>> + interrupts:
>> + minItems: 1
>> + maxItems: 2
>
> Instead describe the items.
Yes.
Best regards,
Krzysztof
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
2022-07-25 16:44 ` Krzysztof Kozlowski
@ 2022-07-25 16:55 ` Frank Li
-1 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-25 16:55 UTC (permalink / raw)
To: Krzysztof Kozlowski, jdmason, maz, tglx, robh+dt,
krzysztof.kozlowski+dt, shawnguo, s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, Peng Fan,
Aisheng Dong, kernel, festevam, dl-linux-imx, kishon,
lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
> Sent: Monday, July 25, 2022 11:45 AM
> To: Frank Li <frank.li@nxp.com>; jdmason@kudzu.us; maz@kernel.org;
> tglx@linutronix.de; robh+dt@kernel.org; krzysztof.kozlowski+dt@linaro.org;
> shawnguo@kernel.org; s.hauer@pengutronix.de; kw@linux.com;
> bhelgaas@google.com
> Cc: kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: Re: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as
> msi controller
>
> Caution: EXT Email
>
> On 25/07/2022 18:29, Frank Li wrote:
> >
> >
> >> -----Original Message-----
> >> From: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
> >> Sent: Saturday, July 23, 2022 1:50 PM
> >> To: Frank Li <frank.li@nxp.com>; jdmason@kudzu.us; maz@kernel.org;
> >> tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org;
> >> shawnguo@kernel.org; s.hauer@pengutronix.de; kw@linux.com;
> >> bhelgaas@google.com
> >> Cc: kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> >> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> >> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> >> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> >> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> >> ntb@lists.linux.dev
> >> Subject: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi
> >> controller
> >>
> >> Caution: EXT Email
> >>
> >> On 20/07/2022 23:30, Frank Li wrote:
> >>> imx mu support generate irq by write a register.
> >>> provide msi controller support so other driver
> >>> can use it by standard msi interface.
> >>
> >> Please start sentences with capital letter. Unfortunately I don't
> >> understand the sentences. Please describe shortly the hardware.
> >
> > [Frank Li] MU have 4 registers and both side A and B. If write one of
> > Register, irq will be trigger at the other side.
> >
> > For example, writle(a side reg1, 0). Then b side irq will be trigged.
>
>
>
> >
> >>
> >>
> >>>
> >>> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> >>> ---
> >>> .../interrupt-controller/fsl,mu-msi.yaml | 88 +++++++++++++++++++
> >>> 1 file changed, 88 insertions(+)
> >>> create mode 100644 Documentation/devicetree/bindings/interrupt-
> >> controller/fsl,mu-msi.yaml
> >>>
> >>> diff --git a/Documentation/devicetree/bindings/interrupt-
> controller/fsl,mu-
> >> msi.yaml b/Documentation/devicetree/bindings/interrupt-
> controller/fsl,mu-
> >> msi.yaml
> >>> new file mode 100644
> >>> index 0000000000000..e125294243af3
> >>> --- /dev/null
> >>> +++ b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> >> msi.yaml
> >>> @@ -0,0 +1,88 @@
> >>> +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> >>> +%YAML 1.2
> >>> +---
> >>> +$id:
> >>
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
> >> ree.org%2Fschemas%2Finterrupt-controller%2Ffsl%2Cmu-
> >>
> msi.yaml%23&data=05%7C01%7CFrank.Li%40nxp.com%7Cfcec12a0731c
> >>
> 454af5c308da6cdc2a0e%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0
> >> %7C637941990101591376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4
> wLj
> >>
> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%
> >>
> 7C%7C&sdata=9h9nKyvsWaghry1hkpa5aaxVGYpx6xZRTxhN0S4uB50%3
> >> D&reserved=0
> >>> +$schema:
> >>
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
> >> ree.org%2Fmeta-
> >>
> schemas%2Fcore.yaml%23&data=05%7C01%7CFrank.Li%40nxp.com%7
> >>
> Cfcec12a0731c454af5c308da6cdc2a0e%7C686ea1d3bc2b4c6fa92cd99c5c3016
> >>
> 35%7C0%7C0%7C637941990101591376%7CUnknown%7CTWFpbGZsb3d8eyJ
> >>
> WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> >>
> 7C3000%7C%7C%7C&sdata=wagM3hl8fpJSm%2Bibw6ENl5lNlQ9fVEHzS
> >> OlT%2Bjoridg%3D&reserved=0
> >>> +
> >>> +title: NXP i.MX Messaging Unit (MU) work as msi controller
> >>> +
> >>> +maintainers:
> >>> + - Frank Li <Frank.Li@nxp.com>
> >>> +
> >>> +description: |
> >>> + The Messaging Unit module enables two processors within the SoC to
> >>> + communicate and coordinate by passing messages (e.g. data, status
> >>> + and control) through the MU interface. The MU also provides the
> ability
> >>> + for one processor to signal the other processor using interrupts.
> >>> +
> >>> + Because the MU manages the messaging between processors, the MU
> >> uses
> >>> + different clocks (from each side of the different peripheral buses).
> >>> + Therefore, the MU must synchronize the accesses from one side to the
> >>> + other. The MU accomplishes synchronization using two sets of
> matching
> >>> + registers (Processor A-facing, Processor B-facing).
> >>> +
> >>> + MU can work as msi interrupt controller to do doorbell
> >>> +
> >>> +allOf:
> >>> + - $ref: /schemas/interrupt-controller/msi-controller.yaml#
> >>> +
> >>> +properties:
> >>> + compatible:
> >>> + enum:
> >>> + - fsl,imx6sx-mu-msi
> >>> + - fsl,imx7ulp-mu-msi
> >>> + - fsl,imx8ulp-mu-msi
> >>> + - fsl,imx8ulp-mu-msi-s4
> >>> +
> >>> + reg:
> >>> + minItems: 2
> >>
> >> Not minItems but maxItems in general, but anyway you need to actually
> >> list and describe the items (and then skip min/max)
> > [Frank Li]
> > I am not sure format. Any example?
> >
> > Reg:
> > Items:
> > - description: a side register
> > - description: b side register
>
> Yes, but then explain what is A and B in bindings description.
[Frank Li] How about "A(B) side base register address."
Any other description need?
>
> Why MU, which sits on A side needs to access other side (B) registers?
[Frank Li] MU work as MSI controller for PCI EP. So driver need provide
B side register to PCI EP by msi_msg. PCI EP driver use this address to set
PCI bar<n>. Then PCI host can write this address to trigger PCIe EP side irq
As doorbell.
MU MSI driver also need A side register
To get irq status. So MU MSI need both side registers.
>
> >>
> >>> +
> >>> + reg-names:
> >>> + items:
> >>> + - const: a
> >>> + - const: b
> >>> +
> >>> + interrupts:
> >>> + maxItems: 1
> >>> +
> >>> + clocks:
> >>> + maxItems: 1
> >>> +
> >>> + power-domains:
> >>> + maxItems: 2
> >>
> >> and here you correctly use maxItems, so why min in reg? Anyway, instead
> >> you need to list and describe the items.
> >
> > Does format is similar with reg?
>
> Yes.
>
> >
> >>
> >> Actually I asked you this last time about interrupts, so you ignored
> >> that comment.
> >
> > Sorry, which one. Is it below one?
> >
> > ---
> >> + interrupts:
> >> + minItems: 1
> >> + maxItems: 2
> >
> > Instead describe the items.
>
> Yes.
>
>
> Best regards,
> Krzysztof
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
@ 2022-07-25 16:55 ` Frank Li
0 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-25 16:55 UTC (permalink / raw)
To: Krzysztof Kozlowski, jdmason, maz, tglx, robh+dt,
krzysztof.kozlowski+dt, shawnguo, s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, Peng Fan,
Aisheng Dong, kernel, festevam, dl-linux-imx, kishon,
lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
> Sent: Monday, July 25, 2022 11:45 AM
> To: Frank Li <frank.li@nxp.com>; jdmason@kudzu.us; maz@kernel.org;
> tglx@linutronix.de; robh+dt@kernel.org; krzysztof.kozlowski+dt@linaro.org;
> shawnguo@kernel.org; s.hauer@pengutronix.de; kw@linux.com;
> bhelgaas@google.com
> Cc: kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: Re: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as
> msi controller
>
> Caution: EXT Email
>
> On 25/07/2022 18:29, Frank Li wrote:
> >
> >
> >> -----Original Message-----
> >> From: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
> >> Sent: Saturday, July 23, 2022 1:50 PM
> >> To: Frank Li <frank.li@nxp.com>; jdmason@kudzu.us; maz@kernel.org;
> >> tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org;
> >> shawnguo@kernel.org; s.hauer@pengutronix.de; kw@linux.com;
> >> bhelgaas@google.com
> >> Cc: kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> >> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> >> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> >> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> >> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> >> ntb@lists.linux.dev
> >> Subject: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi
> >> controller
> >>
> >> Caution: EXT Email
> >>
> >> On 20/07/2022 23:30, Frank Li wrote:
> >>> imx mu support generate irq by write a register.
> >>> provide msi controller support so other driver
> >>> can use it by standard msi interface.
> >>
> >> Please start sentences with capital letter. Unfortunately I don't
> >> understand the sentences. Please describe shortly the hardware.
> >
> > [Frank Li] MU have 4 registers and both side A and B. If write one of
> > Register, irq will be trigger at the other side.
> >
> > For example, writle(a side reg1, 0). Then b side irq will be trigged.
>
>
>
> >
> >>
> >>
> >>>
> >>> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> >>> ---
> >>> .../interrupt-controller/fsl,mu-msi.yaml | 88 +++++++++++++++++++
> >>> 1 file changed, 88 insertions(+)
> >>> create mode 100644 Documentation/devicetree/bindings/interrupt-
> >> controller/fsl,mu-msi.yaml
> >>>
> >>> diff --git a/Documentation/devicetree/bindings/interrupt-
> controller/fsl,mu-
> >> msi.yaml b/Documentation/devicetree/bindings/interrupt-
> controller/fsl,mu-
> >> msi.yaml
> >>> new file mode 100644
> >>> index 0000000000000..e125294243af3
> >>> --- /dev/null
> >>> +++ b/Documentation/devicetree/bindings/interrupt-controller/fsl,mu-
> >> msi.yaml
> >>> @@ -0,0 +1,88 @@
> >>> +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> >>> +%YAML 1.2
> >>> +---
> >>> +$id:
> >>
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
> >> ree.org%2Fschemas%2Finterrupt-controller%2Ffsl%2Cmu-
> >>
> msi.yaml%23&data=05%7C01%7CFrank.Li%40nxp.com%7Cfcec12a0731c
> >>
> 454af5c308da6cdc2a0e%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0
> >> %7C637941990101591376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4
> wLj
> >>
> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%
> >>
> 7C%7C&sdata=9h9nKyvsWaghry1hkpa5aaxVGYpx6xZRTxhN0S4uB50%3
> >> D&reserved=0
> >>> +$schema:
> >>
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdevicet
> >> ree.org%2Fmeta-
> >>
> schemas%2Fcore.yaml%23&data=05%7C01%7CFrank.Li%40nxp.com%7
> >>
> Cfcec12a0731c454af5c308da6cdc2a0e%7C686ea1d3bc2b4c6fa92cd99c5c3016
> >>
> 35%7C0%7C0%7C637941990101591376%7CUnknown%7CTWFpbGZsb3d8eyJ
> >>
> WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> >>
> 7C3000%7C%7C%7C&sdata=wagM3hl8fpJSm%2Bibw6ENl5lNlQ9fVEHzS
> >> OlT%2Bjoridg%3D&reserved=0
> >>> +
> >>> +title: NXP i.MX Messaging Unit (MU) work as msi controller
> >>> +
> >>> +maintainers:
> >>> + - Frank Li <Frank.Li@nxp.com>
> >>> +
> >>> +description: |
> >>> + The Messaging Unit module enables two processors within the SoC to
> >>> + communicate and coordinate by passing messages (e.g. data, status
> >>> + and control) through the MU interface. The MU also provides the
> ability
> >>> + for one processor to signal the other processor using interrupts.
> >>> +
> >>> + Because the MU manages the messaging between processors, the MU
> >> uses
> >>> + different clocks (from each side of the different peripheral buses).
> >>> + Therefore, the MU must synchronize the accesses from one side to the
> >>> + other. The MU accomplishes synchronization using two sets of
> matching
> >>> + registers (Processor A-facing, Processor B-facing).
> >>> +
> >>> + MU can work as msi interrupt controller to do doorbell
> >>> +
> >>> +allOf:
> >>> + - $ref: /schemas/interrupt-controller/msi-controller.yaml#
> >>> +
> >>> +properties:
> >>> + compatible:
> >>> + enum:
> >>> + - fsl,imx6sx-mu-msi
> >>> + - fsl,imx7ulp-mu-msi
> >>> + - fsl,imx8ulp-mu-msi
> >>> + - fsl,imx8ulp-mu-msi-s4
> >>> +
> >>> + reg:
> >>> + minItems: 2
> >>
> >> Not minItems but maxItems in general, but anyway you need to actually
> >> list and describe the items (and then skip min/max)
> > [Frank Li]
> > I am not sure format. Any example?
> >
> > Reg:
> > Items:
> > - description: a side register
> > - description: b side register
>
> Yes, but then explain what is A and B in bindings description.
[Frank Li] How about "A(B) side base register address."
Any other description need?
>
> Why MU, which sits on A side needs to access other side (B) registers?
[Frank Li] MU work as MSI controller for PCI EP. So driver need provide
B side register to PCI EP by msi_msg. PCI EP driver use this address to set
PCI bar<n>. Then PCI host can write this address to trigger PCIe EP side irq
As doorbell.
MU MSI driver also need A side register
To get irq status. So MU MSI need both side registers.
>
> >>
> >>> +
> >>> + reg-names:
> >>> + items:
> >>> + - const: a
> >>> + - const: b
> >>> +
> >>> + interrupts:
> >>> + maxItems: 1
> >>> +
> >>> + clocks:
> >>> + maxItems: 1
> >>> +
> >>> + power-domains:
> >>> + maxItems: 2
> >>
> >> and here you correctly use maxItems, so why min in reg? Anyway, instead
> >> you need to list and describe the items.
> >
> > Does format is similar with reg?
>
> Yes.
>
> >
> >>
> >> Actually I asked you this last time about interrupts, so you ignored
> >> that comment.
> >
> > Sorry, which one. Is it below one?
> >
> > ---
> >> + interrupts:
> >> + minItems: 1
> >> + maxItems: 2
> >
> > Instead describe the items.
>
> Yes.
>
>
> Best regards,
> Krzysztof
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
2022-07-25 16:55 ` Frank Li
@ 2022-07-25 20:28 ` Krzysztof Kozlowski
-1 siblings, 0 replies; 50+ messages in thread
From: Krzysztof Kozlowski @ 2022-07-25 20:28 UTC (permalink / raw)
To: Frank Li, jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt,
shawnguo, s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, Peng Fan,
Aisheng Dong, kernel, festevam, dl-linux-imx, kishon,
lorenzo.pieralisi, ntb
On 25/07/2022 18:55, Frank Li wrote:
>>>> Not minItems but maxItems in general, but anyway you need to actually
>>>> list and describe the items (and then skip min/max)
>>> [Frank Li]
>>> I am not sure format. Any example?
>>>
>>> Reg:
>>> Items:
>>> - description: a side register
>>> - description: b side register
>>
>> Yes, but then explain what is A and B in bindings description.
>
> [Frank Li] How about "A(B) side base register address."
> Any other description need?
In top-level description you have:
"The MU also provides the ability for one processor to signal the
other processor using interrupts."
so maybe:
"The MU also provides the ability for one processor (A side) to signal
the other processor (B side) using interrupts."
>
>>
>> Why MU, which sits on A side needs to access other side (B) registers?
>
> [Frank Li] MU work as MSI controller for PCI EP. So driver need provide
> B side register to PCI EP by msi_msg. PCI EP driver use this address to set
> PCI bar<n>. Then PCI host can write this address to trigger PCIe EP side irq
> As doorbell.
>
> MU MSI driver also need A side register
> To get irq status. So MU MSI need both side registers.
OK.
Best regards,
Krzysztof
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [EXT] Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
@ 2022-07-25 20:28 ` Krzysztof Kozlowski
0 siblings, 0 replies; 50+ messages in thread
From: Krzysztof Kozlowski @ 2022-07-25 20:28 UTC (permalink / raw)
To: Frank Li, jdmason, maz, tglx, robh+dt, krzysztof.kozlowski+dt,
shawnguo, s.hauer, kw, bhelgaas
Cc: kernel, devicetree, linux-arm-kernel, linux-pci, Peng Fan,
Aisheng Dong, kernel, festevam, dl-linux-imx, kishon,
lorenzo.pieralisi, ntb
On 25/07/2022 18:55, Frank Li wrote:
>>>> Not minItems but maxItems in general, but anyway you need to actually
>>>> list and describe the items (and then skip min/max)
>>> [Frank Li]
>>> I am not sure format. Any example?
>>>
>>> Reg:
>>> Items:
>>> - description: a side register
>>> - description: b side register
>>
>> Yes, but then explain what is A and B in bindings description.
>
> [Frank Li] How about "A(B) side base register address."
> Any other description need?
In top-level description you have:
"The MU also provides the ability for one processor to signal the
other processor using interrupts."
so maybe:
"The MU also provides the ability for one processor (A side) to signal
the other processor (B side) using interrupts."
>
>>
>> Why MU, which sits on A side needs to access other side (B) registers?
>
> [Frank Li] MU work as MSI controller for PCI EP. So driver need provide
> B side register to PCI EP by msi_msg. PCI EP driver use this address to set
> PCI bar<n>. Then PCI host can write this address to trigger PCIe EP side irq
> As doorbell.
>
> MU MSI driver also need A side register
> To get irq status. So MU MSI need both side registers.
OK.
Best regards,
Krzysztof
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
2022-07-21 15:28 ` Marc Zyngier
@ 2022-07-26 21:48 ` Frank Li
-1 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-26 21:48 UTC (permalink / raw)
To: Marc Zyngier
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Thursday, July 21, 2022 10:28 AM
> To: Frank Li <frank.li@nxp.com>
> Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
>
> Caution: EXT Email
>
> On Thu, 21 Jul 2022 16:22:08 +0100,
> Frank Li <frank.li@nxp.com> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Marc Zyngier <maz@kernel.org>
> > > Sent: Thursday, July 21, 2022 2:57 AM
> > > To: Frank Li <frank.li@nxp.com>
> > > Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> > > krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> > > s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> > > kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> > > kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> > > <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> > > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> > > imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > > ntb@lists.linux.dev
> > > Subject: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
> > >
> > > Caution: EXT Email
> > >
> > > On Wed, 20 Jul 2022 22:30:34 +0100,
> > > Frank Li <Frank.Li@nxp.com> wrote:
> > > >
> > > > MU support generate irq by write data to a register.
> > > > This patch make mu worked as msi controller.
> > > > So MU can do doorbell by using standard msi api.
> > > >
> > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > ---
> > > > drivers/irqchip/Kconfig | 7 +
> > > > drivers/irqchip/Makefile | 1 +
> > > > drivers/irqchip/irq-imx-mu-msi.c | 462
> > > +++++++++++++++++++++++++++++++
> > > > 3 files changed, 470 insertions(+)
> > > > create mode 100644 drivers/irqchip/irq-imx-mu-msi.c
> > > >
> > > > diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
> > > > index 5e4e50122777d..4599471d880c0 100644
> > > > --- a/drivers/irqchip/Kconfig
> > > > +++ b/drivers/irqchip/Kconfig
> > > > @@ -470,6 +470,13 @@ config IMX_INTMUX
> > > > help
> > > > Support for the i.MX INTMUX interrupt multiplexer.
> > > >
> > > > +config IMX_MU_MSI
> > > > + bool "i.MX MU work as MSI controller"
> > > > + default y if ARCH_MXC
> > > > + select IRQ_DOMAIN
> > > > + help
> > > > + MU work as MSI controller to do general doorbell
> > > > +
> > > > config LS1X_IRQ
> > > > bool "Loongson-1 Interrupt Controller"
> > > > depends on MACH_LOONGSON32
> > > > diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
> > > > index 5d8e21d3dc6d8..870423746c783 100644
> > > > --- a/drivers/irqchip/Makefile
> > > > +++ b/drivers/irqchip/Makefile
> > > > @@ -98,6 +98,7 @@ obj-$(CONFIG_RISCV_INTC) += irq-riscv-
> intc.o
> > > > obj-$(CONFIG_SIFIVE_PLIC) += irq-sifive-plic.o
> > > > obj-$(CONFIG_IMX_IRQSTEER) += irq-imx-irqsteer.o
> > > > obj-$(CONFIG_IMX_INTMUX) += irq-imx-intmux.o
> > > > +obj-$(CONFIG_IMX_MU_MSI) += irq-imx-mu-msi.o
> > > > obj-$(CONFIG_MADERA_IRQ) += irq-madera.o
> > > > obj-$(CONFIG_LS1X_IRQ) += irq-ls1x.o
> > > > obj-$(CONFIG_TI_SCI_INTR_IRQCHIP) += irq-ti-sci-intr.o
> > > > diff --git a/drivers/irqchip/irq-imx-mu-msi.c b/drivers/irqchip/irq-imx-
> mu-
> > > msi.c
> > > > new file mode 100644
> > > > index 0000000000000..8277dba857759
> > > > --- /dev/null
> > > > +++ b/drivers/irqchip/irq-imx-mu-msi.c
> > > > @@ -0,0 +1,462 @@
> > > > +// SPDX-License-Identifier: GPL-2.0-only
> > > > +/*
> > > > + * NXP MU worked as MSI controller
> > > > + *
> > > > + * Copyright (c) 2018 Pengutronix, Oleksij Rempel
> > > <o.rempel@pengutronix.de>
> > > > + * Copyright 2022 NXP
> > > > + * Frank Li <Frank.Li@nxp.com>
> > > > + * Peng Fan <peng.fan@nxp.com>
> > > > + *
> > > > + * Based on drivers/mailbox/imx-mailbox.c
> > > > + */
> > > > +#include <linux/clk.h>
> > > > +#include <linux/kernel.h>
> > > > +#include <linux/module.h>
> > > > +#include <linux/msi.h>
> > > > +#include <linux/interrupt.h>
> > > > +#include <linux/irq.h>
> > > > +#include <linux/irqchip/chained_irq.h>
> > > > +#include <linux/irqchip.h>
> > > > +#include <linux/irqdomain.h>
> > > > +#include <linux/of_irq.h>
> > > > +#include <linux/of_pci.h>
> > > > +#include <linux/of_platform.h>
> > > > +#include <linux/spinlock.h>
> > > > +#include <linux/dma-iommu.h>
> > > > +#include <linux/pm_runtime.h>
> > > > +#include <linux/pm_domain.h>
> > > > +
> > > > +
> > > > +#define IMX_MU_CHANS 4
> > > > +
> > > > +enum imx_mu_xcr {
> > > > + IMX_MU_GIER,
> > > > + IMX_MU_GCR,
> > > > + IMX_MU_TCR,
> > > > + IMX_MU_RCR,
> > > > + IMX_MU_xCR_MAX,
> > > > +};
> > > > +
> > > > +enum imx_mu_xsr {
> > > > + IMX_MU_SR,
> > > > + IMX_MU_GSR,
> > > > + IMX_MU_TSR,
> > > > + IMX_MU_RSR,
> > > > +};
> > > > +
> > > > +enum imx_mu_type {
> > > > + IMX_MU_V1 = BIT(0),
> > > > + IMX_MU_V2 = BIT(1),
> > > > + IMX_MU_V2_S4 = BIT(15),
> > > > +};
> > > > +
> > > > +/* Receive Interrupt Enable */
> > > > +#define IMX_MU_xCR_RIEn(type, x) (type & IMX_MU_V2 ? BIT(x) :
> BIT(24
> > > + (3 - (x))))
> > > > +#define IMX_MU_xSR_RFn(type, x) (type & IMX_MU_V2 ? BIT(x) :
> BIT(24 +
> > > (3 - (x))))
> > > > +
> > > > +struct imx_mu_dcfg {
> > > > + enum imx_mu_type type;
> > > > + u32 xTR; /* Transmit Register0 */
> > > > + u32 xRR; /* Receive Register0 */
> > > > + u32 xSR[4]; /* Status Registers */
> > > > + u32 xCR[4]; /* Control Registers */
> > > > +};
> > > > +
> > > > +struct imx_mu_msi {
> > > > + spinlock_t lock;
> > > > + struct platform_device *pdev;
> > > > + struct irq_domain *parent;
> > > > + struct irq_domain *msi_domain;
> > > > + void __iomem *regs;
> > > > + phys_addr_t msiir_addr;
> > > > + const struct imx_mu_dcfg *cfg;
> > > > + unsigned long used;
> > > > + u32 gic_irq;
> > > > + struct clk *clk;
> > > > + struct device *pd_a;
> > > > + struct device *pd_b;
> > > > + struct device_link *pd_link_a;
> > > > + struct device_link *pd_link_b;
> > > > +};
> > > > +
> > > > +static void imx_mu_write(struct imx_mu_msi *msi_data, u32 val, u32
> offs)
> > > > +{
> > > > + iowrite32(val, msi_data->regs + offs);
> > > > +}
> > > > +
> > > > +static u32 imx_mu_read(struct imx_mu_msi *msi_data, u32 offs)
> > > > +{
> > > > + return ioread32(msi_data->regs + offs);
> > > > +}
> > > > +
> > > > +static u32 imx_mu_xcr_rmw(struct imx_mu_msi *msi_data, enum
> > > imx_mu_xcr type, u32 set, u32 clr)
> > > > +{
> > > > + unsigned long flags;
> > > > + u32 val;
> > > > +
> > > > + spin_lock_irqsave(&msi_data->lock, flags);
> > > > + val = imx_mu_read(msi_data, msi_data->cfg->xCR[type]);
> > > > + val &= ~clr;
> > > > + val |= set;
> > > > + imx_mu_write(msi_data, val, msi_data->cfg->xCR[type]);
> > > > + spin_unlock_irqrestore(&msi_data->lock, flags);
> > > > +
> > > > + return val;
> > > > +}
> > > > +
> > > > +static void imx_mu_msi_mask_irq(struct irq_data *data)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> > > >parent_data);
> > >
> > > Urgh... No. Please don't go poking into the *parent* stuff. Implement
> > > the masking at the parent level, and use irq_chip_mask_parent() for
> > > this level, unless you can explain why you can't do otherwise.
> > >
> > > > +
> > > > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, 0,
> > > IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq));
> > >
> > > How about making this readable and move the dereference inside the
> macro?
> > >
> > > > +}
> > > > +
> > > > +static void imx_mu_msi_unmask_irq(struct irq_data *data)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> > > >parent_data);
> > > > +
> > > > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR,
> > > IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq), 0);
> > > > +}
> > > > +
> > > > +static struct irq_chip imx_mu_msi_irq_chip = {
> > > > + .name = "MU-MSI",
> > > > + .irq_mask = imx_mu_msi_mask_irq,
> > > > + .irq_unmask = imx_mu_msi_unmask_irq,
> > > > +};
> > > > +
> > > > +static struct msi_domain_ops its_pmsi_ops = {
> > > > +};
> > >
> > > What does this have to do with the ITS?
> >
> > Without this, it will be crashed as access 0 address.
>
> Because the *name* of the structure has an influence on the behaviour
> of the kernel?????
>
> >
> > >
> > > > +
> > > > +static struct msi_domain_info imx_mu_msi_domain_info = {
> > > > + .flags = (MSI_FLAG_USE_DEF_DOM_OPS |
> > > > + MSI_FLAG_USE_DEF_CHIP_OPS |
> > > > + MSI_FLAG_PCI_MSIX),
> > >
> > > What does PCI_MSIX mean in this context? I really wish you used
> > > copy/paste a bit more carefully.
> > >
> > > > + .ops = &its_pmsi_ops,
> > > > + .chip = &imx_mu_msi_irq_chip,
> > > > +};
> > > > +
> > > > +static void imx_mu_msi_compose_msg(struct irq_data *data, struct
> > > msi_msg *msg)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
> > > > +
> > > > + msg->address_hi = upper_32_bits(msi_data->msiir_addr);
> > > > + msg->address_lo = lower_32_bits(msi_data->msiir_addr + 4 * data-
> > > >hwirq);
> > >
> > > This is a horrible way of writing this. you should compute the address
> > > first, and then apply the address split.
> > >
> > > > + msg->data = data->hwirq;
> > > > +
> > > > + iommu_dma_compose_msi_msg(irq_data_get_msi_desc(data),
> msg);
> > > > +}
> > > > +
> > > > +static int imx_mu_msi_set_affinity(struct irq_data *irq_data,
> > > > + const struct cpumask *mask, bool force)
> > > > +
> > > > +{
> > > > + return IRQ_SET_MASK_OK;
> > > > +}
> > >
> > > Err... What effect does this have if you don't do anything?
> >
> > [Frank Li] Without this, it will be crashed as access 0 address.
>
> Then you should fix that bug, because what you have here makes
> absolutely no sense.
>
> >
> > >
> > > > +
> > > > +static struct irq_chip imx_mu_msi_parent_chip = {
> > > > + .name = "MU",
> > > > + .irq_compose_msi_msg = imx_mu_msi_compose_msg,
> > > > + .irq_set_affinity = imx_mu_msi_set_affinity,
> > > > +};
> > > > +
> > > > +static int imx_mu_msi_domain_irq_alloc(struct irq_domain *domain,
> > > > + unsigned int virq,
> > > > + unsigned int nr_irqs,
> > > > + void *args)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = domain->host_data;
> > > > + msi_alloc_info_t *info = args;
> > > > + int pos, err = 0;
> > > > +
> > > > + WARN_ON(nr_irqs != 1);
> > > > +
> > > > + spin_lock(&msi_data->lock);
> > >
> > > Interrupt fires after lock acquisition, handler masks the interrupt.
> > > Deadlock.
> >
> > [Frank Li] you are right, it should be spin_lock_irqsave.
> >
> > >
> > > > + pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
> > > > + if (pos < IMX_MU_CHANS)
> > > > + __set_bit(pos, &msi_data->used);
> > > > + else
> > > > + err = -ENOSPC;
> > > > + spin_unlock(&msi_data->lock);
> > > > +
> > > > + if (err)
> > > > + return err;
> > > > +
> > > > + err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr
> +
> > > pos * 4);
> > >
> > > Does this HW even have an IOMMU?
> >
> > [Frank Li] we have a platform with iommu.
> >
> > >
> > > > + if (err)
> > > > + return err;
> > > > +
> > > > + irq_domain_set_info(domain, virq, pos,
> > > > + &imx_mu_msi_parent_chip, msi_data,
> > > > + handle_simple_irq, NULL, NULL);
> > >
> > > This is an edge interrupt. Please handle it like one.
> >
> > [Frank Li] Not sure what your means?
>
> A MSI is an edge interrupt. Use handle_edge_irq.
>
> >
> > >
> > > > + return 0;
> > > > +}
> > > > +
> > > > +static void imx_mu_msi_domain_irq_free(struct irq_domain *domain,
> > > > + unsigned int virq, unsigned int nr_irqs)
> > > > +{
> > > > + struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(d);
> > > > +
> > > > + spin_lock(&msi_data->lock);
> > >
> > > Same problem.
> > >
> > > > + __clear_bit(d->hwirq, &msi_data->used);
> > > > + spin_unlock(&msi_data->lock);
> > > > +}
> > > > +
> > > > +static const struct irq_domain_ops imx_mu_msi_domain_ops = {
> > > > + .alloc = imx_mu_msi_domain_irq_alloc,
> > > > + .free = imx_mu_msi_domain_irq_free,
> > > > +};
> > > > +
> > > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_desc_get_handler_data(desc);
> > > > + u32 status;
> > > > + int i;
> > > > +
> > > > + status = imx_mu_read(msi_data, msi_data->cfg-
> >xSR[IMX_MU_RSR]);
> > > > +
> > > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > > > + generic_handle_domain_irq(msi_data->parent, i);
> > >
> > > Why the parent? You must start at the top of the hierarchy.
[Frank Li] Do you means that should be msi_data->msi_domain instead of msi_data->parent?
> > >
> > > > + }
> > > > + }
> > > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> > >
> > > If your MSIs are a chained interrupt, why do you even provide an
> > > affinity setting callback?
> >
> > [Frank Li] it will be crash if no affinity setting callback.
>
> Then you have to fix your driver.
[Frank Li] After debug, msi_domain_set_affinity() have not did null check for (parent->chip->irq_set_affinity).
I think impact by using dummy set_affinity is minimized.
int msi_domain_set_affinity(struct irq_data *irq_data,
const struct cpumask *mask, bool force)
{
struct irq_data *parent = irq_data->parent_data;
struct msi_msg msg[2] = { [1] = { }, };
int ret;
ret = parent->chip->irq_set_affinity(parent, mask, force);
if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) {
BUG_ON(irq_chip_compose_msi_msg(irq_data, msg));
msi_check_level(irq_data->domain, msg);
irq_chip_write_msi_msg(irq_data, msg);
}
return ret;
}
>
> M.
>
> --
> Without deviation from the norm, progress is not possible.
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
@ 2022-07-26 21:48 ` Frank Li
0 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-26 21:48 UTC (permalink / raw)
To: Marc Zyngier
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Thursday, July 21, 2022 10:28 AM
> To: Frank Li <frank.li@nxp.com>
> Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
>
> Caution: EXT Email
>
> On Thu, 21 Jul 2022 16:22:08 +0100,
> Frank Li <frank.li@nxp.com> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Marc Zyngier <maz@kernel.org>
> > > Sent: Thursday, July 21, 2022 2:57 AM
> > > To: Frank Li <frank.li@nxp.com>
> > > Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> > > krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> > > s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> > > kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> > > kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> > > <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> > > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> > > imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > > ntb@lists.linux.dev
> > > Subject: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
> > >
> > > Caution: EXT Email
> > >
> > > On Wed, 20 Jul 2022 22:30:34 +0100,
> > > Frank Li <Frank.Li@nxp.com> wrote:
> > > >
> > > > MU support generate irq by write data to a register.
> > > > This patch make mu worked as msi controller.
> > > > So MU can do doorbell by using standard msi api.
> > > >
> > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > ---
> > > > drivers/irqchip/Kconfig | 7 +
> > > > drivers/irqchip/Makefile | 1 +
> > > > drivers/irqchip/irq-imx-mu-msi.c | 462
> > > +++++++++++++++++++++++++++++++
> > > > 3 files changed, 470 insertions(+)
> > > > create mode 100644 drivers/irqchip/irq-imx-mu-msi.c
> > > >
> > > > diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
> > > > index 5e4e50122777d..4599471d880c0 100644
> > > > --- a/drivers/irqchip/Kconfig
> > > > +++ b/drivers/irqchip/Kconfig
> > > > @@ -470,6 +470,13 @@ config IMX_INTMUX
> > > > help
> > > > Support for the i.MX INTMUX interrupt multiplexer.
> > > >
> > > > +config IMX_MU_MSI
> > > > + bool "i.MX MU work as MSI controller"
> > > > + default y if ARCH_MXC
> > > > + select IRQ_DOMAIN
> > > > + help
> > > > + MU work as MSI controller to do general doorbell
> > > > +
> > > > config LS1X_IRQ
> > > > bool "Loongson-1 Interrupt Controller"
> > > > depends on MACH_LOONGSON32
> > > > diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
> > > > index 5d8e21d3dc6d8..870423746c783 100644
> > > > --- a/drivers/irqchip/Makefile
> > > > +++ b/drivers/irqchip/Makefile
> > > > @@ -98,6 +98,7 @@ obj-$(CONFIG_RISCV_INTC) += irq-riscv-
> intc.o
> > > > obj-$(CONFIG_SIFIVE_PLIC) += irq-sifive-plic.o
> > > > obj-$(CONFIG_IMX_IRQSTEER) += irq-imx-irqsteer.o
> > > > obj-$(CONFIG_IMX_INTMUX) += irq-imx-intmux.o
> > > > +obj-$(CONFIG_IMX_MU_MSI) += irq-imx-mu-msi.o
> > > > obj-$(CONFIG_MADERA_IRQ) += irq-madera.o
> > > > obj-$(CONFIG_LS1X_IRQ) += irq-ls1x.o
> > > > obj-$(CONFIG_TI_SCI_INTR_IRQCHIP) += irq-ti-sci-intr.o
> > > > diff --git a/drivers/irqchip/irq-imx-mu-msi.c b/drivers/irqchip/irq-imx-
> mu-
> > > msi.c
> > > > new file mode 100644
> > > > index 0000000000000..8277dba857759
> > > > --- /dev/null
> > > > +++ b/drivers/irqchip/irq-imx-mu-msi.c
> > > > @@ -0,0 +1,462 @@
> > > > +// SPDX-License-Identifier: GPL-2.0-only
> > > > +/*
> > > > + * NXP MU worked as MSI controller
> > > > + *
> > > > + * Copyright (c) 2018 Pengutronix, Oleksij Rempel
> > > <o.rempel@pengutronix.de>
> > > > + * Copyright 2022 NXP
> > > > + * Frank Li <Frank.Li@nxp.com>
> > > > + * Peng Fan <peng.fan@nxp.com>
> > > > + *
> > > > + * Based on drivers/mailbox/imx-mailbox.c
> > > > + */
> > > > +#include <linux/clk.h>
> > > > +#include <linux/kernel.h>
> > > > +#include <linux/module.h>
> > > > +#include <linux/msi.h>
> > > > +#include <linux/interrupt.h>
> > > > +#include <linux/irq.h>
> > > > +#include <linux/irqchip/chained_irq.h>
> > > > +#include <linux/irqchip.h>
> > > > +#include <linux/irqdomain.h>
> > > > +#include <linux/of_irq.h>
> > > > +#include <linux/of_pci.h>
> > > > +#include <linux/of_platform.h>
> > > > +#include <linux/spinlock.h>
> > > > +#include <linux/dma-iommu.h>
> > > > +#include <linux/pm_runtime.h>
> > > > +#include <linux/pm_domain.h>
> > > > +
> > > > +
> > > > +#define IMX_MU_CHANS 4
> > > > +
> > > > +enum imx_mu_xcr {
> > > > + IMX_MU_GIER,
> > > > + IMX_MU_GCR,
> > > > + IMX_MU_TCR,
> > > > + IMX_MU_RCR,
> > > > + IMX_MU_xCR_MAX,
> > > > +};
> > > > +
> > > > +enum imx_mu_xsr {
> > > > + IMX_MU_SR,
> > > > + IMX_MU_GSR,
> > > > + IMX_MU_TSR,
> > > > + IMX_MU_RSR,
> > > > +};
> > > > +
> > > > +enum imx_mu_type {
> > > > + IMX_MU_V1 = BIT(0),
> > > > + IMX_MU_V2 = BIT(1),
> > > > + IMX_MU_V2_S4 = BIT(15),
> > > > +};
> > > > +
> > > > +/* Receive Interrupt Enable */
> > > > +#define IMX_MU_xCR_RIEn(type, x) (type & IMX_MU_V2 ? BIT(x) :
> BIT(24
> > > + (3 - (x))))
> > > > +#define IMX_MU_xSR_RFn(type, x) (type & IMX_MU_V2 ? BIT(x) :
> BIT(24 +
> > > (3 - (x))))
> > > > +
> > > > +struct imx_mu_dcfg {
> > > > + enum imx_mu_type type;
> > > > + u32 xTR; /* Transmit Register0 */
> > > > + u32 xRR; /* Receive Register0 */
> > > > + u32 xSR[4]; /* Status Registers */
> > > > + u32 xCR[4]; /* Control Registers */
> > > > +};
> > > > +
> > > > +struct imx_mu_msi {
> > > > + spinlock_t lock;
> > > > + struct platform_device *pdev;
> > > > + struct irq_domain *parent;
> > > > + struct irq_domain *msi_domain;
> > > > + void __iomem *regs;
> > > > + phys_addr_t msiir_addr;
> > > > + const struct imx_mu_dcfg *cfg;
> > > > + unsigned long used;
> > > > + u32 gic_irq;
> > > > + struct clk *clk;
> > > > + struct device *pd_a;
> > > > + struct device *pd_b;
> > > > + struct device_link *pd_link_a;
> > > > + struct device_link *pd_link_b;
> > > > +};
> > > > +
> > > > +static void imx_mu_write(struct imx_mu_msi *msi_data, u32 val, u32
> offs)
> > > > +{
> > > > + iowrite32(val, msi_data->regs + offs);
> > > > +}
> > > > +
> > > > +static u32 imx_mu_read(struct imx_mu_msi *msi_data, u32 offs)
> > > > +{
> > > > + return ioread32(msi_data->regs + offs);
> > > > +}
> > > > +
> > > > +static u32 imx_mu_xcr_rmw(struct imx_mu_msi *msi_data, enum
> > > imx_mu_xcr type, u32 set, u32 clr)
> > > > +{
> > > > + unsigned long flags;
> > > > + u32 val;
> > > > +
> > > > + spin_lock_irqsave(&msi_data->lock, flags);
> > > > + val = imx_mu_read(msi_data, msi_data->cfg->xCR[type]);
> > > > + val &= ~clr;
> > > > + val |= set;
> > > > + imx_mu_write(msi_data, val, msi_data->cfg->xCR[type]);
> > > > + spin_unlock_irqrestore(&msi_data->lock, flags);
> > > > +
> > > > + return val;
> > > > +}
> > > > +
> > > > +static void imx_mu_msi_mask_irq(struct irq_data *data)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> > > >parent_data);
> > >
> > > Urgh... No. Please don't go poking into the *parent* stuff. Implement
> > > the masking at the parent level, and use irq_chip_mask_parent() for
> > > this level, unless you can explain why you can't do otherwise.
> > >
> > > > +
> > > > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR, 0,
> > > IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq));
> > >
> > > How about making this readable and move the dereference inside the
> macro?
> > >
> > > > +}
> > > > +
> > > > +static void imx_mu_msi_unmask_irq(struct irq_data *data)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data-
> > > >parent_data);
> > > > +
> > > > + imx_mu_xcr_rmw(msi_data, IMX_MU_RCR,
> > > IMX_MU_xCR_RIEn(msi_data->cfg->type, data->hwirq), 0);
> > > > +}
> > > > +
> > > > +static struct irq_chip imx_mu_msi_irq_chip = {
> > > > + .name = "MU-MSI",
> > > > + .irq_mask = imx_mu_msi_mask_irq,
> > > > + .irq_unmask = imx_mu_msi_unmask_irq,
> > > > +};
> > > > +
> > > > +static struct msi_domain_ops its_pmsi_ops = {
> > > > +};
> > >
> > > What does this have to do with the ITS?
> >
> > Without this, it will be crashed as access 0 address.
>
> Because the *name* of the structure has an influence on the behaviour
> of the kernel?????
>
> >
> > >
> > > > +
> > > > +static struct msi_domain_info imx_mu_msi_domain_info = {
> > > > + .flags = (MSI_FLAG_USE_DEF_DOM_OPS |
> > > > + MSI_FLAG_USE_DEF_CHIP_OPS |
> > > > + MSI_FLAG_PCI_MSIX),
> > >
> > > What does PCI_MSIX mean in this context? I really wish you used
> > > copy/paste a bit more carefully.
> > >
> > > > + .ops = &its_pmsi_ops,
> > > > + .chip = &imx_mu_msi_irq_chip,
> > > > +};
> > > > +
> > > > +static void imx_mu_msi_compose_msg(struct irq_data *data, struct
> > > msi_msg *msg)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(data);
> > > > +
> > > > + msg->address_hi = upper_32_bits(msi_data->msiir_addr);
> > > > + msg->address_lo = lower_32_bits(msi_data->msiir_addr + 4 * data-
> > > >hwirq);
> > >
> > > This is a horrible way of writing this. you should compute the address
> > > first, and then apply the address split.
> > >
> > > > + msg->data = data->hwirq;
> > > > +
> > > > + iommu_dma_compose_msi_msg(irq_data_get_msi_desc(data),
> msg);
> > > > +}
> > > > +
> > > > +static int imx_mu_msi_set_affinity(struct irq_data *irq_data,
> > > > + const struct cpumask *mask, bool force)
> > > > +
> > > > +{
> > > > + return IRQ_SET_MASK_OK;
> > > > +}
> > >
> > > Err... What effect does this have if you don't do anything?
> >
> > [Frank Li] Without this, it will be crashed as access 0 address.
>
> Then you should fix that bug, because what you have here makes
> absolutely no sense.
>
> >
> > >
> > > > +
> > > > +static struct irq_chip imx_mu_msi_parent_chip = {
> > > > + .name = "MU",
> > > > + .irq_compose_msi_msg = imx_mu_msi_compose_msg,
> > > > + .irq_set_affinity = imx_mu_msi_set_affinity,
> > > > +};
> > > > +
> > > > +static int imx_mu_msi_domain_irq_alloc(struct irq_domain *domain,
> > > > + unsigned int virq,
> > > > + unsigned int nr_irqs,
> > > > + void *args)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = domain->host_data;
> > > > + msi_alloc_info_t *info = args;
> > > > + int pos, err = 0;
> > > > +
> > > > + WARN_ON(nr_irqs != 1);
> > > > +
> > > > + spin_lock(&msi_data->lock);
> > >
> > > Interrupt fires after lock acquisition, handler masks the interrupt.
> > > Deadlock.
> >
> > [Frank Li] you are right, it should be spin_lock_irqsave.
> >
> > >
> > > > + pos = find_first_zero_bit(&msi_data->used, IMX_MU_CHANS);
> > > > + if (pos < IMX_MU_CHANS)
> > > > + __set_bit(pos, &msi_data->used);
> > > > + else
> > > > + err = -ENOSPC;
> > > > + spin_unlock(&msi_data->lock);
> > > > +
> > > > + if (err)
> > > > + return err;
> > > > +
> > > > + err = iommu_dma_prepare_msi(info->desc, msi_data->msiir_addr
> +
> > > pos * 4);
> > >
> > > Does this HW even have an IOMMU?
> >
> > [Frank Li] we have a platform with iommu.
> >
> > >
> > > > + if (err)
> > > > + return err;
> > > > +
> > > > + irq_domain_set_info(domain, virq, pos,
> > > > + &imx_mu_msi_parent_chip, msi_data,
> > > > + handle_simple_irq, NULL, NULL);
> > >
> > > This is an edge interrupt. Please handle it like one.
> >
> > [Frank Li] Not sure what your means?
>
> A MSI is an edge interrupt. Use handle_edge_irq.
>
> >
> > >
> > > > + return 0;
> > > > +}
> > > > +
> > > > +static void imx_mu_msi_domain_irq_free(struct irq_domain *domain,
> > > > + unsigned int virq, unsigned int nr_irqs)
> > > > +{
> > > > + struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> > > > + struct imx_mu_msi *msi_data = irq_data_get_irq_chip_data(d);
> > > > +
> > > > + spin_lock(&msi_data->lock);
> > >
> > > Same problem.
> > >
> > > > + __clear_bit(d->hwirq, &msi_data->used);
> > > > + spin_unlock(&msi_data->lock);
> > > > +}
> > > > +
> > > > +static const struct irq_domain_ops imx_mu_msi_domain_ops = {
> > > > + .alloc = imx_mu_msi_domain_irq_alloc,
> > > > + .free = imx_mu_msi_domain_irq_free,
> > > > +};
> > > > +
> > > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > > +{
> > > > + struct imx_mu_msi *msi_data = irq_desc_get_handler_data(desc);
> > > > + u32 status;
> > > > + int i;
> > > > +
> > > > + status = imx_mu_read(msi_data, msi_data->cfg-
> >xSR[IMX_MU_RSR]);
> > > > +
> > > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > > > + generic_handle_domain_irq(msi_data->parent, i);
> > >
> > > Why the parent? You must start at the top of the hierarchy.
[Frank Li] Do you means that should be msi_data->msi_domain instead of msi_data->parent?
> > >
> > > > + }
> > > > + }
> > > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> > >
> > > If your MSIs are a chained interrupt, why do you even provide an
> > > affinity setting callback?
> >
> > [Frank Li] it will be crash if no affinity setting callback.
>
> Then you have to fix your driver.
[Frank Li] After debug, msi_domain_set_affinity() have not did null check for (parent->chip->irq_set_affinity).
I think impact by using dummy set_affinity is minimized.
int msi_domain_set_affinity(struct irq_data *irq_data,
const struct cpumask *mask, bool force)
{
struct irq_data *parent = irq_data->parent_data;
struct msi_msg msg[2] = { [1] = { }, };
int ret;
ret = parent->chip->irq_set_affinity(parent, mask, force);
if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) {
BUG_ON(irq_chip_compose_msi_msg(irq_data, msg));
msi_check_level(irq_data->domain, msg);
irq_chip_write_msi_msg(irq_data, msg);
}
return ret;
}
>
> M.
>
> --
> Without deviation from the norm, progress is not possible.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
2022-07-26 21:48 ` Frank Li
@ 2022-07-27 8:02 ` Marc Zyngier
-1 siblings, 0 replies; 50+ messages in thread
From: Marc Zyngier @ 2022-07-27 8:02 UTC (permalink / raw)
To: Frank Li
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
On Tue, 26 Jul 2022 22:48:32 +0100,
Frank Li <frank.li@nxp.com> wrote:
>
> > > > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > > > +{
> > > > > + struct imx_mu_msi *msi_data = irq_desc_get_handler_data(desc);
> > > > > + u32 status;
> > > > > + int i;
> > > > > +
> > > > > + status = imx_mu_read(msi_data, msi_data->cfg-
> > >xSR[IMX_MU_RSR]);
> > > > > +
> > > > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > > > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > > > > + generic_handle_domain_irq(msi_data->parent, i);
> > > >
> > > > Why the parent? You must start at the top of the hierarchy.
>
> [Frank Li] Do you means that should be msi_data->msi_domain instead
> of msi_data->parent?
Indeed. you must *not* bypass the hierarchy, and the top level of the
hierarchy has to implement whatever is required by the interrupt flow.
>
> > > >
> > > > > + }
> > > > > + }
> > > > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> > > >
> > > > If your MSIs are a chained interrupt, why do you even provide an
> > > > affinity setting callback?
> > >
> > > [Frank Li] it will be crash if no affinity setting callback.
> >
> > Then you have to fix your driver.
>
> [Frank Li] After debug, msi_domain_set_affinity() have not did null check for (parent->chip->irq_set_affinity).
> I think impact by using dummy set_affinity is minimized.
>
> int msi_domain_set_affinity(struct irq_data *irq_data,
> const struct cpumask *mask, bool force)
> {
> struct irq_data *parent = irq_data->parent_data;
> struct msi_msg msg[2] = { [1] = { }, };
> int ret;
>
> ret = parent->chip->irq_set_affinity(parent, mask, force);
> if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) {
> BUG_ON(irq_chip_compose_msi_msg(irq_data, msg));
> msi_check_level(irq_data->domain, msg);
> irq_chip_write_msi_msg(irq_data, msg);
> }
>
> return ret;
> }
No. Changing the affinity of an interrupt must not affect the affinity
of another. Given that this is a chained handler, you *cannot* satisfy
this requirement. So you can't change the affinity at all.
N,
--
Without deviation from the norm, progress is not possible.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
@ 2022-07-27 8:02 ` Marc Zyngier
0 siblings, 0 replies; 50+ messages in thread
From: Marc Zyngier @ 2022-07-27 8:02 UTC (permalink / raw)
To: Frank Li
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
On Tue, 26 Jul 2022 22:48:32 +0100,
Frank Li <frank.li@nxp.com> wrote:
>
> > > > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > > > +{
> > > > > + struct imx_mu_msi *msi_data = irq_desc_get_handler_data(desc);
> > > > > + u32 status;
> > > > > + int i;
> > > > > +
> > > > > + status = imx_mu_read(msi_data, msi_data->cfg-
> > >xSR[IMX_MU_RSR]);
> > > > > +
> > > > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > > > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > > > > + generic_handle_domain_irq(msi_data->parent, i);
> > > >
> > > > Why the parent? You must start at the top of the hierarchy.
>
> [Frank Li] Do you means that should be msi_data->msi_domain instead
> of msi_data->parent?
Indeed. you must *not* bypass the hierarchy, and the top level of the
hierarchy has to implement whatever is required by the interrupt flow.
>
> > > >
> > > > > + }
> > > > > + }
> > > > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> > > >
> > > > If your MSIs are a chained interrupt, why do you even provide an
> > > > affinity setting callback?
> > >
> > > [Frank Li] it will be crash if no affinity setting callback.
> >
> > Then you have to fix your driver.
>
> [Frank Li] After debug, msi_domain_set_affinity() have not did null check for (parent->chip->irq_set_affinity).
> I think impact by using dummy set_affinity is minimized.
>
> int msi_domain_set_affinity(struct irq_data *irq_data,
> const struct cpumask *mask, bool force)
> {
> struct irq_data *parent = irq_data->parent_data;
> struct msi_msg msg[2] = { [1] = { }, };
> int ret;
>
> ret = parent->chip->irq_set_affinity(parent, mask, force);
> if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) {
> BUG_ON(irq_chip_compose_msi_msg(irq_data, msg));
> msi_check_level(irq_data->domain, msg);
> irq_chip_write_msi_msg(irq_data, msg);
> }
>
> return ret;
> }
No. Changing the affinity of an interrupt must not affect the affinity
of another. Given that this is a chained handler, you *cannot* satisfy
this requirement. So you can't change the affinity at all.
N,
--
Without deviation from the norm, progress is not possible.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
2022-07-27 8:02 ` Marc Zyngier
@ 2022-07-27 15:23 ` Frank Li
-1 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-27 15:23 UTC (permalink / raw)
To: Marc Zyngier
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Wednesday, July 27, 2022 3:03 AM
> To: Frank Li <frank.li@nxp.com>
> Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
>
> Caution: EXT Email
>
> On Tue, 26 Jul 2022 22:48:32 +0100,
> Frank Li <frank.li@nxp.com> wrote:
> >
> > > > > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > > > > +{
> > > > > > + struct imx_mu_msi *msi_data =
> irq_desc_get_handler_data(desc);
> > > > > > + u32 status;
> > > > > > + int i;
> > > > > > +
> > > > > > + status = imx_mu_read(msi_data, msi_data->cfg-
> > > >xSR[IMX_MU_RSR]);
> > > > > > +
> > > > > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > > > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > > > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > > > > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > > > > > + generic_handle_domain_irq(msi_data->parent, i);
> > > > >
> > > > > Why the parent? You must start at the top of the hierarchy.
> >
> > [Frank Li] Do you means that should be msi_data->msi_domain instead
> > of msi_data->parent?
>
> Indeed. you must *not* bypass the hierarchy, and the top level of the
> hierarchy has to implement whatever is required by the interrupt flow.
>
[Frank Li] I see, just want to confirm msi_data->msi_domain should be correct here?
It should be leaf of irq hierarchy tree.
> >
> > > > >
> > > > > > + }
> > > > > > + }
> > > > > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> > > > >
> > > > > If your MSIs are a chained interrupt, why do you even provide an
> > > > > affinity setting callback?
> > > >
> > > > [Frank Li] it will be crash if no affinity setting callback.
> > >
> > > Then you have to fix your driver.
> >
> > [Frank Li] After debug, msi_domain_set_affinity() have not did null check
> for (parent->chip->irq_set_affinity).
> > I think impact by using dummy set_affinity is minimized.
> >
> > int msi_domain_set_affinity(struct irq_data *irq_data,
> > const struct cpumask *mask, bool force)
> > {
> > struct irq_data *parent = irq_data->parent_data;
> > struct msi_msg msg[2] = { [1] = { }, };
> > int ret;
> >
> > ret = parent->chip->irq_set_affinity(parent, mask, force);
> > if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) {
> > BUG_ON(irq_chip_compose_msi_msg(irq_data, msg));
> > msi_check_level(irq_data->domain, msg);
> > irq_chip_write_msi_msg(irq_data, msg);
> > }
> >
> > return ret;
> > }
>
> No. Changing the affinity of an interrupt must not affect the affinity
> of another. Given that this is a chained handler, you *cannot* satisfy
> this requirement. So you can't change the affinity at all.
>
[Frank Li] I understand affinity can't be changed.
But system use set affinity to write msi msg.
The call stack as
[ 25.508229] epf_ntb_write_msi_msg+0x78/0x90
[ 25.512512] platform_msi_write_msg+0x2c/0x38
[ 25.516882] msi_domain_set_affinity+0xb0/0xc0
[ 25.521330] irq_do_set_affinity+0x174/0x220
[ 25.525604] irq_setup_affinity+0xe0/0x188
[ 25.529713] irq_startup+0x88/0x160
[ 25.533214] __setup_irq+0x6c8/0x768
I have not found good place to hook a function to write msi msg.
int irq_startup(struct irq_desc *desc, bool resend, bool force)
{
struct irq_data *d = irq_desc_get_irq_data(desc);
struct cpumask *aff = irq_data_get_affinity_mask(d);
int ret = 0;
desc->depth = 0;
if (irqd_is_started(d)) {
irq_enable(desc);
} else {
switch (__irq_startup_managed(desc, aff, force)) {
case IRQ_STARTUP_NORMAL:
if (d->chip->flags & IRQCHIP_AFFINITY_PRE_STARTUP)
irq_setup_affinity(desc);
ret = __irq_startup(desc);
if (!(d->chip->flags & IRQCHIP_AFFINITY_PRE_STARTUP))
irq_setup_affinity(desc);
break;
case IRQ_STARTUP_MANAGED:
irq_do_set_affinity(d, aff, false);
ret = __irq_startup(desc);
break;
case IRQ_STARTUP_ABORT:
irqd_set_managed_shutdown(d);
return 0;
}
}
if (resend)
check_irq_resend(desc, false);
return ret;
}
> N,
>
> --
> Without deviation from the norm, progress is not possible.
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
@ 2022-07-27 15:23 ` Frank Li
0 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-27 15:23 UTC (permalink / raw)
To: Marc Zyngier
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Wednesday, July 27, 2022 3:03 AM
> To: Frank Li <frank.li@nxp.com>
> Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
>
> Caution: EXT Email
>
> On Tue, 26 Jul 2022 22:48:32 +0100,
> Frank Li <frank.li@nxp.com> wrote:
> >
> > > > > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > > > > +{
> > > > > > + struct imx_mu_msi *msi_data =
> irq_desc_get_handler_data(desc);
> > > > > > + u32 status;
> > > > > > + int i;
> > > > > > +
> > > > > > + status = imx_mu_read(msi_data, msi_data->cfg-
> > > >xSR[IMX_MU_RSR]);
> > > > > > +
> > > > > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > > > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > > > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > > > > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > > > > > + generic_handle_domain_irq(msi_data->parent, i);
> > > > >
> > > > > Why the parent? You must start at the top of the hierarchy.
> >
> > [Frank Li] Do you means that should be msi_data->msi_domain instead
> > of msi_data->parent?
>
> Indeed. you must *not* bypass the hierarchy, and the top level of the
> hierarchy has to implement whatever is required by the interrupt flow.
>
[Frank Li] I see, just want to confirm msi_data->msi_domain should be correct here?
It should be leaf of irq hierarchy tree.
> >
> > > > >
> > > > > > + }
> > > > > > + }
> > > > > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> > > > >
> > > > > If your MSIs are a chained interrupt, why do you even provide an
> > > > > affinity setting callback?
> > > >
> > > > [Frank Li] it will be crash if no affinity setting callback.
> > >
> > > Then you have to fix your driver.
> >
> > [Frank Li] After debug, msi_domain_set_affinity() have not did null check
> for (parent->chip->irq_set_affinity).
> > I think impact by using dummy set_affinity is minimized.
> >
> > int msi_domain_set_affinity(struct irq_data *irq_data,
> > const struct cpumask *mask, bool force)
> > {
> > struct irq_data *parent = irq_data->parent_data;
> > struct msi_msg msg[2] = { [1] = { }, };
> > int ret;
> >
> > ret = parent->chip->irq_set_affinity(parent, mask, force);
> > if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) {
> > BUG_ON(irq_chip_compose_msi_msg(irq_data, msg));
> > msi_check_level(irq_data->domain, msg);
> > irq_chip_write_msi_msg(irq_data, msg);
> > }
> >
> > return ret;
> > }
>
> No. Changing the affinity of an interrupt must not affect the affinity
> of another. Given that this is a chained handler, you *cannot* satisfy
> this requirement. So you can't change the affinity at all.
>
[Frank Li] I understand affinity can't be changed.
But system use set affinity to write msi msg.
The call stack as
[ 25.508229] epf_ntb_write_msi_msg+0x78/0x90
[ 25.512512] platform_msi_write_msg+0x2c/0x38
[ 25.516882] msi_domain_set_affinity+0xb0/0xc0
[ 25.521330] irq_do_set_affinity+0x174/0x220
[ 25.525604] irq_setup_affinity+0xe0/0x188
[ 25.529713] irq_startup+0x88/0x160
[ 25.533214] __setup_irq+0x6c8/0x768
I have not found good place to hook a function to write msi msg.
int irq_startup(struct irq_desc *desc, bool resend, bool force)
{
struct irq_data *d = irq_desc_get_irq_data(desc);
struct cpumask *aff = irq_data_get_affinity_mask(d);
int ret = 0;
desc->depth = 0;
if (irqd_is_started(d)) {
irq_enable(desc);
} else {
switch (__irq_startup_managed(desc, aff, force)) {
case IRQ_STARTUP_NORMAL:
if (d->chip->flags & IRQCHIP_AFFINITY_PRE_STARTUP)
irq_setup_affinity(desc);
ret = __irq_startup(desc);
if (!(d->chip->flags & IRQCHIP_AFFINITY_PRE_STARTUP))
irq_setup_affinity(desc);
break;
case IRQ_STARTUP_MANAGED:
irq_do_set_affinity(d, aff, false);
ret = __irq_startup(desc);
break;
case IRQ_STARTUP_ABORT:
irqd_set_managed_shutdown(d);
return 0;
}
}
if (resend)
check_irq_resend(desc, false);
return ret;
}
> N,
>
> --
> Without deviation from the norm, progress is not possible.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
2022-07-27 15:23 ` Frank Li
@ 2022-07-27 15:34 ` Marc Zyngier
-1 siblings, 0 replies; 50+ messages in thread
From: Marc Zyngier @ 2022-07-27 15:34 UTC (permalink / raw)
To: Frank Li
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
On Wed, 27 Jul 2022 16:23:26 +0100,
Frank Li <frank.li@nxp.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Marc Zyngier <maz@kernel.org>
> > Sent: Wednesday, July 27, 2022 3:03 AM
> > To: Frank Li <frank.li@nxp.com>
> > Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> > krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> > s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> > kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> > kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> > <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> > imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > ntb@lists.linux.dev
> > Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> > controller
> >
> > Caution: EXT Email
> >
> > On Tue, 26 Jul 2022 22:48:32 +0100,
> > Frank Li <frank.li@nxp.com> wrote:
> > >
> > > > > > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > > > > > +{
> > > > > > > + struct imx_mu_msi *msi_data =
> > irq_desc_get_handler_data(desc);
> > > > > > > + u32 status;
> > > > > > > + int i;
> > > > > > > +
> > > > > > > + status = imx_mu_read(msi_data, msi_data->cfg-
> > > > >xSR[IMX_MU_RSR]);
> > > > > > > +
> > > > > > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > > > > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > > > > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > > > > > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > > > > > > + generic_handle_domain_irq(msi_data->parent, i);
> > > > > >
> > > > > > Why the parent? You must start at the top of the hierarchy.
> > >
> > > [Frank Li] Do you means that should be msi_data->msi_domain instead
> > > of msi_data->parent?
> >
> > Indeed. you must *not* bypass the hierarchy, and the top level of the
> > hierarchy has to implement whatever is required by the interrupt flow.
> >
>
> [Frank Li] I see, just want to confirm msi_data->msi_domain should
> be correct here? It should be leaf of irq hierarchy tree.
Yes.
>
> > >
> > > > > >
> > > > > > > + }
> > > > > > > + }
> > > > > > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> > > > > >
> > > > > > If your MSIs are a chained interrupt, why do you even provide an
> > > > > > affinity setting callback?
> > > > >
> > > > > [Frank Li] it will be crash if no affinity setting callback.
> > > >
> > > > Then you have to fix your driver.
> > >
> > > [Frank Li] After debug, msi_domain_set_affinity() have not did null check
> > for (parent->chip->irq_set_affinity).
> > > I think impact by using dummy set_affinity is minimized.
> > >
> > > int msi_domain_set_affinity(struct irq_data *irq_data,
> > > const struct cpumask *mask, bool force)
> > > {
> > > struct irq_data *parent = irq_data->parent_data;
> > > struct msi_msg msg[2] = { [1] = { }, };
> > > int ret;
> > >
> > > ret = parent->chip->irq_set_affinity(parent, mask, force);
> > > if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) {
> > > BUG_ON(irq_chip_compose_msi_msg(irq_data, msg));
> > > msi_check_level(irq_data->domain, msg);
> > > irq_chip_write_msi_msg(irq_data, msg);
> > > }
> > >
> > > return ret;
> > > }
> >
> > No. Changing the affinity of an interrupt must not affect the affinity
> > of another. Given that this is a chained handler, you *cannot* satisfy
> > this requirement. So you can't change the affinity at all.
> >
>
> [Frank Li] I understand affinity can't be changed.
> But system use set affinity to write msi msg.
>
> The call stack as
> [ 25.508229] epf_ntb_write_msi_msg+0x78/0x90
> [ 25.512512] platform_msi_write_msg+0x2c/0x38
> [ 25.516882] msi_domain_set_affinity+0xb0/0xc0
> [ 25.521330] irq_do_set_affinity+0x174/0x220
> [ 25.525604] irq_setup_affinity+0xe0/0x188
> [ 25.529713] irq_startup+0x88/0x160
> [ 25.533214] __setup_irq+0x6c8/0x768
>
> I have not found good place to hook a function to write msi msg.
It is called at MSI activation time (msi_domain_activate).
M.
--
Without deviation from the norm, progress is not possible.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
@ 2022-07-27 15:34 ` Marc Zyngier
0 siblings, 0 replies; 50+ messages in thread
From: Marc Zyngier @ 2022-07-27 15:34 UTC (permalink / raw)
To: Frank Li
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
On Wed, 27 Jul 2022 16:23:26 +0100,
Frank Li <frank.li@nxp.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Marc Zyngier <maz@kernel.org>
> > Sent: Wednesday, July 27, 2022 3:03 AM
> > To: Frank Li <frank.li@nxp.com>
> > Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> > krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> > s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> > kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> > kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> > <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> > imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > ntb@lists.linux.dev
> > Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> > controller
> >
> > Caution: EXT Email
> >
> > On Tue, 26 Jul 2022 22:48:32 +0100,
> > Frank Li <frank.li@nxp.com> wrote:
> > >
> > > > > > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > > > > > +{
> > > > > > > + struct imx_mu_msi *msi_data =
> > irq_desc_get_handler_data(desc);
> > > > > > > + u32 status;
> > > > > > > + int i;
> > > > > > > +
> > > > > > > + status = imx_mu_read(msi_data, msi_data->cfg-
> > > > >xSR[IMX_MU_RSR]);
> > > > > > > +
> > > > > > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > > > > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > > > > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > > > > > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > > > > > > + generic_handle_domain_irq(msi_data->parent, i);
> > > > > >
> > > > > > Why the parent? You must start at the top of the hierarchy.
> > >
> > > [Frank Li] Do you means that should be msi_data->msi_domain instead
> > > of msi_data->parent?
> >
> > Indeed. you must *not* bypass the hierarchy, and the top level of the
> > hierarchy has to implement whatever is required by the interrupt flow.
> >
>
> [Frank Li] I see, just want to confirm msi_data->msi_domain should
> be correct here? It should be leaf of irq hierarchy tree.
Yes.
>
> > >
> > > > > >
> > > > > > > + }
> > > > > > > + }
> > > > > > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> > > > > >
> > > > > > If your MSIs are a chained interrupt, why do you even provide an
> > > > > > affinity setting callback?
> > > > >
> > > > > [Frank Li] it will be crash if no affinity setting callback.
> > > >
> > > > Then you have to fix your driver.
> > >
> > > [Frank Li] After debug, msi_domain_set_affinity() have not did null check
> > for (parent->chip->irq_set_affinity).
> > > I think impact by using dummy set_affinity is minimized.
> > >
> > > int msi_domain_set_affinity(struct irq_data *irq_data,
> > > const struct cpumask *mask, bool force)
> > > {
> > > struct irq_data *parent = irq_data->parent_data;
> > > struct msi_msg msg[2] = { [1] = { }, };
> > > int ret;
> > >
> > > ret = parent->chip->irq_set_affinity(parent, mask, force);
> > > if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) {
> > > BUG_ON(irq_chip_compose_msi_msg(irq_data, msg));
> > > msi_check_level(irq_data->domain, msg);
> > > irq_chip_write_msi_msg(irq_data, msg);
> > > }
> > >
> > > return ret;
> > > }
> >
> > No. Changing the affinity of an interrupt must not affect the affinity
> > of another. Given that this is a chained handler, you *cannot* satisfy
> > this requirement. So you can't change the affinity at all.
> >
>
> [Frank Li] I understand affinity can't be changed.
> But system use set affinity to write msi msg.
>
> The call stack as
> [ 25.508229] epf_ntb_write_msi_msg+0x78/0x90
> [ 25.512512] platform_msi_write_msg+0x2c/0x38
> [ 25.516882] msi_domain_set_affinity+0xb0/0xc0
> [ 25.521330] irq_do_set_affinity+0x174/0x220
> [ 25.525604] irq_setup_affinity+0xe0/0x188
> [ 25.529713] irq_startup+0x88/0x160
> [ 25.533214] __setup_irq+0x6c8/0x768
>
> I have not found good place to hook a function to write msi msg.
It is called at MSI activation time (msi_domain_activate).
M.
--
Without deviation from the norm, progress is not possible.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
2022-07-27 15:34 ` Marc Zyngier
@ 2022-07-27 18:29 ` Frank Li
-1 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-27 18:29 UTC (permalink / raw)
To: Marc Zyngier
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Wednesday, July 27, 2022 10:35 AM
> To: Frank Li <frank.li@nxp.com>
> Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
>
> Caution: EXT Email
>
> On Wed, 27 Jul 2022 16:23:26 +0100,
> Frank Li <frank.li@nxp.com> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Marc Zyngier <maz@kernel.org>
> > > Sent: Wednesday, July 27, 2022 3:03 AM
> > > To: Frank Li <frank.li@nxp.com>
> > > Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> > > krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> > > s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> > > kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> > > kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> > > <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> > > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> > > imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > > ntb@lists.linux.dev
> > > Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> > > controller
> > >
> > > Caution: EXT Email
> > >
> > > On Tue, 26 Jul 2022 22:48:32 +0100,
> > > Frank Li <frank.li@nxp.com> wrote:
> > > >
> > > > > > > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > > > > > > +{
> > > > > > > > + struct imx_mu_msi *msi_data =
> > > irq_desc_get_handler_data(desc);
> > > > > > > > + u32 status;
> > > > > > > > + int i;
> > > > > > > > +
> > > > > > > > + status = imx_mu_read(msi_data, msi_data->cfg-
> > > > > >xSR[IMX_MU_RSR]);
> > > > > > > > +
> > > > > > > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > > > > > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > > > > > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > > > > > > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > > > > > > > + generic_handle_domain_irq(msi_data->parent, i);
> > > > > > >
> > > > > > > Why the parent? You must start at the top of the hierarchy.
> > > >
> > > > [Frank Li] Do you means that should be msi_data->msi_domain instead
> > > > of msi_data->parent?
> > >
> > > Indeed. you must *not* bypass the hierarchy, and the top level of the
> > > hierarchy has to implement whatever is required by the interrupt flow.
> > >
> >
> > [Frank Li] I see, just want to confirm msi_data->msi_domain should
> > be correct here? It should be leaf of irq hierarchy tree.
>
> Yes.
>
> >
> > > >
> > > > > > >
> > > > > > > > + }
> > > > > > > > + }
> > > > > > > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> > > > > > >
> > > > > > > If your MSIs are a chained interrupt, why do you even provide an
> > > > > > > affinity setting callback?
> > > > > >
> > > > > > [Frank Li] it will be crash if no affinity setting callback.
> > > > >
> > > > > Then you have to fix your driver.
> > > >
> > > > [Frank Li] After debug, msi_domain_set_affinity() have not did null
> check
> > > for (parent->chip->irq_set_affinity).
> > > > I think impact by using dummy set_affinity is minimized.
> > > >
> > > > int msi_domain_set_affinity(struct irq_data *irq_data,
> > > > const struct cpumask *mask, bool force)
> > > > {
> > > > struct irq_data *parent = irq_data->parent_data;
> > > > struct msi_msg msg[2] = { [1] = { }, };
> > > > int ret;
> > > >
> > > > ret = parent->chip->irq_set_affinity(parent, mask, force);
> > > > if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) {
> > > > BUG_ON(irq_chip_compose_msi_msg(irq_data, msg));
> > > > msi_check_level(irq_data->domain, msg);
> > > > irq_chip_write_msi_msg(irq_data, msg);
> > > > }
> > > >
> > > > return ret;
> > > > }
> > >
> > > No. Changing the affinity of an interrupt must not affect the affinity
> > > of another. Given that this is a chained handler, you *cannot* satisfy
> > > this requirement. So you can't change the affinity at all.
> > >
> >
> > [Frank Li] I understand affinity can't be changed.
> > But system use set affinity to write msi msg.
> >
> > The call stack as
> > [ 25.508229] epf_ntb_write_msi_msg+0x78/0x90
> > [ 25.512512] platform_msi_write_msg+0x2c/0x38
> > [ 25.516882] msi_domain_set_affinity+0xb0/0xc0
> > [ 25.521330] irq_do_set_affinity+0x174/0x220
> > [ 25.525604] irq_setup_affinity+0xe0/0x188
> > [ 25.529713] irq_startup+0x88/0x160
> > [ 25.533214] __setup_irq+0x6c8/0x768
> >
> > I have not found good place to hook a function to write msi msg.
>
> It is called at MSI activation time (msi_domain_activate).
Another issue: platform_msi_write_msg() is static function at platform-msi.c.
It access a local structure struct platform_msi_priv_data.
If I use MSI_FLAG_USE_DEF_CHIP_OPS flags, both msi_domain_set_affinity and msi_domain_set_affinity.
will be set at chip. So it will NULL point error happen if I don't set affinity function.
>
> M.
>
> --
> Without deviation from the norm, progress is not possible.
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
@ 2022-07-27 18:29 ` Frank Li
0 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-27 18:29 UTC (permalink / raw)
To: Marc Zyngier
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Wednesday, July 27, 2022 10:35 AM
> To: Frank Li <frank.li@nxp.com>
> Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
>
> Caution: EXT Email
>
> On Wed, 27 Jul 2022 16:23:26 +0100,
> Frank Li <frank.li@nxp.com> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Marc Zyngier <maz@kernel.org>
> > > Sent: Wednesday, July 27, 2022 3:03 AM
> > > To: Frank Li <frank.li@nxp.com>
> > > Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> > > krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> > > s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> > > kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> > > kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> > > <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> > > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> > > imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > > ntb@lists.linux.dev
> > > Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> > > controller
> > >
> > > Caution: EXT Email
> > >
> > > On Tue, 26 Jul 2022 22:48:32 +0100,
> > > Frank Li <frank.li@nxp.com> wrote:
> > > >
> > > > > > > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > > > > > > +{
> > > > > > > > + struct imx_mu_msi *msi_data =
> > > irq_desc_get_handler_data(desc);
> > > > > > > > + u32 status;
> > > > > > > > + int i;
> > > > > > > > +
> > > > > > > > + status = imx_mu_read(msi_data, msi_data->cfg-
> > > > > >xSR[IMX_MU_RSR]);
> > > > > > > > +
> > > > > > > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > > > > > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > > > > > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i)) {
> > > > > > > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i * 4);
> > > > > > > > + generic_handle_domain_irq(msi_data->parent, i);
> > > > > > >
> > > > > > > Why the parent? You must start at the top of the hierarchy.
> > > >
> > > > [Frank Li] Do you means that should be msi_data->msi_domain instead
> > > > of msi_data->parent?
> > >
> > > Indeed. you must *not* bypass the hierarchy, and the top level of the
> > > hierarchy has to implement whatever is required by the interrupt flow.
> > >
> >
> > [Frank Li] I see, just want to confirm msi_data->msi_domain should
> > be correct here? It should be leaf of irq hierarchy tree.
>
> Yes.
>
> >
> > > >
> > > > > > >
> > > > > > > > + }
> > > > > > > > + }
> > > > > > > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> > > > > > >
> > > > > > > If your MSIs are a chained interrupt, why do you even provide an
> > > > > > > affinity setting callback?
> > > > > >
> > > > > > [Frank Li] it will be crash if no affinity setting callback.
> > > > >
> > > > > Then you have to fix your driver.
> > > >
> > > > [Frank Li] After debug, msi_domain_set_affinity() have not did null
> check
> > > for (parent->chip->irq_set_affinity).
> > > > I think impact by using dummy set_affinity is minimized.
> > > >
> > > > int msi_domain_set_affinity(struct irq_data *irq_data,
> > > > const struct cpumask *mask, bool force)
> > > > {
> > > > struct irq_data *parent = irq_data->parent_data;
> > > > struct msi_msg msg[2] = { [1] = { }, };
> > > > int ret;
> > > >
> > > > ret = parent->chip->irq_set_affinity(parent, mask, force);
> > > > if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) {
> > > > BUG_ON(irq_chip_compose_msi_msg(irq_data, msg));
> > > > msi_check_level(irq_data->domain, msg);
> > > > irq_chip_write_msi_msg(irq_data, msg);
> > > > }
> > > >
> > > > return ret;
> > > > }
> > >
> > > No. Changing the affinity of an interrupt must not affect the affinity
> > > of another. Given that this is a chained handler, you *cannot* satisfy
> > > this requirement. So you can't change the affinity at all.
> > >
> >
> > [Frank Li] I understand affinity can't be changed.
> > But system use set affinity to write msi msg.
> >
> > The call stack as
> > [ 25.508229] epf_ntb_write_msi_msg+0x78/0x90
> > [ 25.512512] platform_msi_write_msg+0x2c/0x38
> > [ 25.516882] msi_domain_set_affinity+0xb0/0xc0
> > [ 25.521330] irq_do_set_affinity+0x174/0x220
> > [ 25.525604] irq_setup_affinity+0xe0/0x188
> > [ 25.529713] irq_startup+0x88/0x160
> > [ 25.533214] __setup_irq+0x6c8/0x768
> >
> > I have not found good place to hook a function to write msi msg.
>
> It is called at MSI activation time (msi_domain_activate).
Another issue: platform_msi_write_msg() is static function at platform-msi.c.
It access a local structure struct platform_msi_priv_data.
If I use MSI_FLAG_USE_DEF_CHIP_OPS flags, both msi_domain_set_affinity and msi_domain_set_affinity.
will be set at chip. So it will NULL point error happen if I don't set affinity function.
>
> M.
>
> --
> Without deviation from the norm, progress is not possible.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
2022-07-27 18:29 ` Frank Li
@ 2022-07-27 18:58 ` Frank Li
-1 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-27 18:58 UTC (permalink / raw)
To: Marc Zyngier
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Frank Li
> Sent: Wednesday, July 27, 2022 1:30 PM
> To: Marc Zyngier <maz@kernel.org>
> Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
>
>
>
> > -----Original Message-----
> > From: Marc Zyngier <maz@kernel.org>
> > Sent: Wednesday, July 27, 2022 10:35 AM
> > To: Frank Li <frank.li@nxp.com>
> > Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> > krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> > s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> > kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> > kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> > <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> > imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > ntb@lists.linux.dev
> > Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> > controller
> >
> > Caution: EXT Email
> >
> > On Wed, 27 Jul 2022 16:23:26 +0100,
> > Frank Li <frank.li@nxp.com> wrote:
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Marc Zyngier <maz@kernel.org>
> > > > Sent: Wednesday, July 27, 2022 3:03 AM
> > > > To: Frank Li <frank.li@nxp.com>
> > > > Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> > > > krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> > > > s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> > > > kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> > > > kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> > > > <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> > > > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> > > > imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > > > ntb@lists.linux.dev
> > > > Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> > > > controller
> > > >
> > > > Caution: EXT Email
> > > >
> > > > On Tue, 26 Jul 2022 22:48:32 +0100,
> > > > Frank Li <frank.li@nxp.com> wrote:
> > > > >
> > > > > > > > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > > > > > > > +{
> > > > > > > > > + struct imx_mu_msi *msi_data =
> > > > irq_desc_get_handler_data(desc);
> > > > > > > > > + u32 status;
> > > > > > > > > + int i;
> > > > > > > > > +
> > > > > > > > > + status = imx_mu_read(msi_data, msi_data->cfg-
> > > > > > >xSR[IMX_MU_RSR]);
> > > > > > > > > +
> > > > > > > > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > > > > > > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > > > > > > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i))
> {
> > > > > > > > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i *
> 4);
> > > > > > > > > + generic_handle_domain_irq(msi_data->parent, i);
> > > > > > > >
> > > > > > > > Why the parent? You must start at the top of the hierarchy.
> > > > >
> > > > > [Frank Li] Do you means that should be msi_data->msi_domain
> instead
> > > > > of msi_data->parent?
> > > >
> > > > Indeed. you must *not* bypass the hierarchy, and the top level of the
> > > > hierarchy has to implement whatever is required by the interrupt flow.
> > > >
> > >
> > > [Frank Li] I see, just want to confirm msi_data->msi_domain should
> > > be correct here? It should be leaf of irq hierarchy tree.
> >
> > Yes.
> >
> > >
> > > > >
> > > > > > > >
> > > > > > > > > + }
> > > > > > > > > + }
> > > > > > > > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> > > > > > > >
> > > > > > > > If your MSIs are a chained interrupt, why do you even provide an
> > > > > > > > affinity setting callback?
> > > > > > >
> > > > > > > [Frank Li] it will be crash if no affinity setting callback.
> > > > > >
> > > > > > Then you have to fix your driver.
> > > > >
> > > > > [Frank Li] After debug, msi_domain_set_affinity() have not did null
> > check
> > > > for (parent->chip->irq_set_affinity).
> > > > > I think impact by using dummy set_affinity is minimized.
> > > > >
> > > > > int msi_domain_set_affinity(struct irq_data *irq_data,
> > > > > const struct cpumask *mask, bool force)
> > > > > {
> > > > > struct irq_data *parent = irq_data->parent_data;
> > > > > struct msi_msg msg[2] = { [1] = { }, };
> > > > > int ret;
> > > > >
> > > > > ret = parent->chip->irq_set_affinity(parent, mask, force);
> > > > > if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) {
> > > > > BUG_ON(irq_chip_compose_msi_msg(irq_data, msg));
> > > > > msi_check_level(irq_data->domain, msg);
> > > > > irq_chip_write_msi_msg(irq_data, msg);
> > > > > }
> > > > >
> > > > > return ret;
> > > > > }
> > > >
> > > > No. Changing the affinity of an interrupt must not affect the affinity
> > > > of another. Given that this is a chained handler, you *cannot* satisfy
> > > > this requirement. So you can't change the affinity at all.
> > > >
> > >
> > > [Frank Li] I understand affinity can't be changed.
> > > But system use set affinity to write msi msg.
> > >
> > > The call stack as
> > > [ 25.508229] epf_ntb_write_msi_msg+0x78/0x90
> > > [ 25.512512] platform_msi_write_msg+0x2c/0x38
> > > [ 25.516882] msi_domain_set_affinity+0xb0/0xc0
> > > [ 25.521330] irq_do_set_affinity+0x174/0x220
> > > [ 25.525604] irq_setup_affinity+0xe0/0x188
> > > [ 25.529713] irq_startup+0x88/0x160
> > > [ 25.533214] __setup_irq+0x6c8/0x768
> > >
> > > I have not found good place to hook a function to write msi msg.
> >
> > It is called at MSI activation time (msi_domain_activate).
>
> Another issue: platform_msi_write_msg() is static function at platform-msi.c.
> It access a local structure struct platform_msi_priv_data.
>
> If I use MSI_FLAG_USE_DEF_CHIP_OPS flags, both msi_domain_set_affinity
> and msi_domain_set_affinity.
> will be set at chip. So it will NULL point error happen if I don't set affinity
> function.
>
[Frank Li] look like imx_mu_msi_irq_chip.irq_set_affinity = NULL; after call
platform_msi_create_irq_domain() can resolve this problem.
But it looks hack method. I think imx_mu_msi_irq_chip should keep unchanged
after call create platform_msi_create_irq_domain().
Do you have better solution or this way should be fine?
> >
> > M.
> >
> > --
> > Without deviation from the norm, progress is not possible.
^ permalink raw reply [flat|nested] 50+ messages in thread
* RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi controller
@ 2022-07-27 18:58 ` Frank Li
0 siblings, 0 replies; 50+ messages in thread
From: Frank Li @ 2022-07-27 18:58 UTC (permalink / raw)
To: Marc Zyngier
Cc: jdmason, tglx, robh+dt, krzysztof.kozlowski+dt, shawnguo,
s.hauer, kw, bhelgaas, kernel, devicetree, linux-arm-kernel,
linux-pci, Peng Fan, Aisheng Dong, kernel, festevam,
dl-linux-imx, kishon, lorenzo.pieralisi, ntb
> -----Original Message-----
> From: Frank Li
> Sent: Wednesday, July 27, 2022 1:30 PM
> To: Marc Zyngier <maz@kernel.org>
> Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> ntb@lists.linux.dev
> Subject: RE: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> controller
>
>
>
> > -----Original Message-----
> > From: Marc Zyngier <maz@kernel.org>
> > Sent: Wednesday, July 27, 2022 10:35 AM
> > To: Frank Li <frank.li@nxp.com>
> > Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> > krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> > s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> > kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> > kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> > <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> > imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > ntb@lists.linux.dev
> > Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> > controller
> >
> > Caution: EXT Email
> >
> > On Wed, 27 Jul 2022 16:23:26 +0100,
> > Frank Li <frank.li@nxp.com> wrote:
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Marc Zyngier <maz@kernel.org>
> > > > Sent: Wednesday, July 27, 2022 3:03 AM
> > > > To: Frank Li <frank.li@nxp.com>
> > > > Cc: jdmason@kudzu.us; tglx@linutronix.de; robh+dt@kernel.org;
> > > > krzysztof.kozlowski+dt@linaro.org; shawnguo@kernel.org;
> > > > s.hauer@pengutronix.de; kw@linux.com; bhelgaas@google.com;
> > > > kernel@vger.kernel.org; devicetree@vger.kernel.org; linux-arm-
> > > > kernel@lists.infradead.org; linux-pci@vger.kernel.org; Peng Fan
> > > > <peng.fan@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> > > > kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx <linux-
> > > > imx@nxp.com>; kishon@ti.com; lorenzo.pieralisi@arm.com;
> > > > ntb@lists.linux.dev
> > > > Subject: Re: [EXT] Re: [PATCH v3 2/4] irqchip: imx mu worked as msi
> > > > controller
> > > >
> > > > Caution: EXT Email
> > > >
> > > > On Tue, 26 Jul 2022 22:48:32 +0100,
> > > > Frank Li <frank.li@nxp.com> wrote:
> > > > >
> > > > > > > > > +static void imx_mu_msi_irq_handler(struct irq_desc *desc)
> > > > > > > > > +{
> > > > > > > > > + struct imx_mu_msi *msi_data =
> > > > irq_desc_get_handler_data(desc);
> > > > > > > > > + u32 status;
> > > > > > > > > + int i;
> > > > > > > > > +
> > > > > > > > > + status = imx_mu_read(msi_data, msi_data->cfg-
> > > > > > >xSR[IMX_MU_RSR]);
> > > > > > > > > +
> > > > > > > > > + chained_irq_enter(irq_desc_get_chip(desc), desc);
> > > > > > > > > + for (i = 0; i < IMX_MU_CHANS; i++) {
> > > > > > > > > + if (status & IMX_MU_xSR_RFn(msi_data->cfg->type, i))
> {
> > > > > > > > > + imx_mu_read(msi_data, msi_data->cfg->xRR + i *
> 4);
> > > > > > > > > + generic_handle_domain_irq(msi_data->parent, i);
> > > > > > > >
> > > > > > > > Why the parent? You must start at the top of the hierarchy.
> > > > >
> > > > > [Frank Li] Do you means that should be msi_data->msi_domain
> instead
> > > > > of msi_data->parent?
> > > >
> > > > Indeed. you must *not* bypass the hierarchy, and the top level of the
> > > > hierarchy has to implement whatever is required by the interrupt flow.
> > > >
> > >
> > > [Frank Li] I see, just want to confirm msi_data->msi_domain should
> > > be correct here? It should be leaf of irq hierarchy tree.
> >
> > Yes.
> >
> > >
> > > > >
> > > > > > > >
> > > > > > > > > + }
> > > > > > > > > + }
> > > > > > > > > + chained_irq_exit(irq_desc_get_chip(desc), desc);
> > > > > > > >
> > > > > > > > If your MSIs are a chained interrupt, why do you even provide an
> > > > > > > > affinity setting callback?
> > > > > > >
> > > > > > > [Frank Li] it will be crash if no affinity setting callback.
> > > > > >
> > > > > > Then you have to fix your driver.
> > > > >
> > > > > [Frank Li] After debug, msi_domain_set_affinity() have not did null
> > check
> > > > for (parent->chip->irq_set_affinity).
> > > > > I think impact by using dummy set_affinity is minimized.
> > > > >
> > > > > int msi_domain_set_affinity(struct irq_data *irq_data,
> > > > > const struct cpumask *mask, bool force)
> > > > > {
> > > > > struct irq_data *parent = irq_data->parent_data;
> > > > > struct msi_msg msg[2] = { [1] = { }, };
> > > > > int ret;
> > > > >
> > > > > ret = parent->chip->irq_set_affinity(parent, mask, force);
> > > > > if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) {
> > > > > BUG_ON(irq_chip_compose_msi_msg(irq_data, msg));
> > > > > msi_check_level(irq_data->domain, msg);
> > > > > irq_chip_write_msi_msg(irq_data, msg);
> > > > > }
> > > > >
> > > > > return ret;
> > > > > }
> > > >
> > > > No. Changing the affinity of an interrupt must not affect the affinity
> > > > of another. Given that this is a chained handler, you *cannot* satisfy
> > > > this requirement. So you can't change the affinity at all.
> > > >
> > >
> > > [Frank Li] I understand affinity can't be changed.
> > > But system use set affinity to write msi msg.
> > >
> > > The call stack as
> > > [ 25.508229] epf_ntb_write_msi_msg+0x78/0x90
> > > [ 25.512512] platform_msi_write_msg+0x2c/0x38
> > > [ 25.516882] msi_domain_set_affinity+0xb0/0xc0
> > > [ 25.521330] irq_do_set_affinity+0x174/0x220
> > > [ 25.525604] irq_setup_affinity+0xe0/0x188
> > > [ 25.529713] irq_startup+0x88/0x160
> > > [ 25.533214] __setup_irq+0x6c8/0x768
> > >
> > > I have not found good place to hook a function to write msi msg.
> >
> > It is called at MSI activation time (msi_domain_activate).
>
> Another issue: platform_msi_write_msg() is static function at platform-msi.c.
> It access a local structure struct platform_msi_priv_data.
>
> If I use MSI_FLAG_USE_DEF_CHIP_OPS flags, both msi_domain_set_affinity
> and msi_domain_set_affinity.
> will be set at chip. So it will NULL point error happen if I don't set affinity
> function.
>
[Frank Li] look like imx_mu_msi_irq_chip.irq_set_affinity = NULL; after call
platform_msi_create_irq_domain() can resolve this problem.
But it looks hack method. I think imx_mu_msi_irq_chip should keep unchanged
after call create platform_msi_create_irq_domain().
Do you have better solution or this way should be fine?
> >
> > M.
> >
> > --
> > Without deviation from the norm, progress is not possible.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
2022-07-20 21:30 ` Frank Li
@ 2022-08-10 14:01 ` Rob Herring
-1 siblings, 0 replies; 50+ messages in thread
From: Rob Herring @ 2022-08-10 14:01 UTC (permalink / raw)
To: Frank Li, Marc Zyngier
Cc: Jon Mason, Thomas Gleixner, Krzysztof Kozlowski, Shawn Guo,
Sascha Hauer, Krzysztof Wilczynski, Bjorn Helgaas,
linux-kernel@vger.kernel.org, devicetree, linux-arm-kernel, PCI,
Peng Fan, Dong Aisheng, Sascha Hauer, Fabio Estevam,
NXP Linux Team, Kishon Vijay Abraham I, Lorenzo Pieralisi, ntb
On Wed, Jul 20, 2022 at 3:31 PM Frank Li <Frank.Li@nxp.com> wrote:
>
> imx mu support generate irq by write a register.
> provide msi controller support so other driver
> can use it by standard msi interface.
>
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---
> .../interrupt-controller/fsl,mu-msi.yaml | 88 +++++++++++++++++++
> 1 file changed, 88 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
This is failing in linux-next now, but I'm wondering why it is there
given all the comments.
Error: Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.example.dts:31.41-42
syntax error
FATAL ERROR: Unable to parse input tree
make[1]: *** [scripts/Makefile.lib:396:
Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.example.dtb]
Error 1
Rob
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
@ 2022-08-10 14:01 ` Rob Herring
0 siblings, 0 replies; 50+ messages in thread
From: Rob Herring @ 2022-08-10 14:01 UTC (permalink / raw)
To: Frank Li, Marc Zyngier
Cc: Jon Mason, Thomas Gleixner, Krzysztof Kozlowski, Shawn Guo,
Sascha Hauer, Krzysztof Wilczynski, Bjorn Helgaas,
linux-kernel@vger.kernel.org, devicetree, linux-arm-kernel, PCI,
Peng Fan, Dong Aisheng, Sascha Hauer, Fabio Estevam,
NXP Linux Team, Kishon Vijay Abraham I, Lorenzo Pieralisi, ntb
On Wed, Jul 20, 2022 at 3:31 PM Frank Li <Frank.Li@nxp.com> wrote:
>
> imx mu support generate irq by write a register.
> provide msi controller support so other driver
> can use it by standard msi interface.
>
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---
> .../interrupt-controller/fsl,mu-msi.yaml | 88 +++++++++++++++++++
> 1 file changed, 88 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
This is failing in linux-next now, but I'm wondering why it is there
given all the comments.
Error: Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.example.dts:31.41-42
syntax error
FATAL ERROR: Unable to parse input tree
make[1]: *** [scripts/Makefile.lib:396:
Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.example.dtb]
Error 1
Rob
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
2022-08-10 14:01 ` Rob Herring
@ 2022-08-10 14:20 ` Marc Zyngier
-1 siblings, 0 replies; 50+ messages in thread
From: Marc Zyngier @ 2022-08-10 14:20 UTC (permalink / raw)
To: Rob Herring, Jon Mason
Cc: Frank Li, Thomas Gleixner, Krzysztof Kozlowski, Shawn Guo,
Sascha Hauer, Krzysztof Wilczynski, Bjorn Helgaas,
linux-kernel@vger.kernel.org, devicetree, linux-arm-kernel, PCI,
Peng Fan, Dong Aisheng, Sascha Hauer, Fabio Estevam,
NXP Linux Team, Kishon Vijay Abraham I, Lorenzo Pieralisi, ntb
On 2022-08-10 15:01, Rob Herring wrote:
> On Wed, Jul 20, 2022 at 3:31 PM Frank Li <Frank.Li@nxp.com> wrote:
>>
>> imx mu support generate irq by write a register.
>> provide msi controller support so other driver
>> can use it by standard msi interface.
>>
>> Signed-off-by: Frank Li <Frank.Li@nxp.com>
>> ---
>> .../interrupt-controller/fsl,mu-msi.yaml | 88
>> +++++++++++++++++++
>> 1 file changed, 88 insertions(+)
>> create mode 100644
>> Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
>
> This is failing in linux-next now, but I'm wondering why it is there
> given all the comments.
*BLINK*
Jon, please drop this and the corresponding irqchip changes
from -next. There is definitely not 6.1 material if the current
code is anything to go by. That'd be commits:
621b7555c914 ("dt-bindings: irqchip: imx mu work as msi controller")
893ae515ff8d ("irqchip: imx mu worked as msi controller")
Once it is ready, I'll take it via the irqchip tree. But only when
it is ready.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
@ 2022-08-10 14:20 ` Marc Zyngier
0 siblings, 0 replies; 50+ messages in thread
From: Marc Zyngier @ 2022-08-10 14:20 UTC (permalink / raw)
To: Rob Herring, Jon Mason
Cc: Frank Li, Thomas Gleixner, Krzysztof Kozlowski, Shawn Guo,
Sascha Hauer, Krzysztof Wilczynski, Bjorn Helgaas,
linux-kernel@vger.kernel.org, devicetree, linux-arm-kernel, PCI,
Peng Fan, Dong Aisheng, Sascha Hauer, Fabio Estevam,
NXP Linux Team, Kishon Vijay Abraham I, Lorenzo Pieralisi, ntb
On 2022-08-10 15:01, Rob Herring wrote:
> On Wed, Jul 20, 2022 at 3:31 PM Frank Li <Frank.Li@nxp.com> wrote:
>>
>> imx mu support generate irq by write a register.
>> provide msi controller support so other driver
>> can use it by standard msi interface.
>>
>> Signed-off-by: Frank Li <Frank.Li@nxp.com>
>> ---
>> .../interrupt-controller/fsl,mu-msi.yaml | 88
>> +++++++++++++++++++
>> 1 file changed, 88 insertions(+)
>> create mode 100644
>> Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
>
> This is failing in linux-next now, but I'm wondering why it is there
> given all the comments.
*BLINK*
Jon, please drop this and the corresponding irqchip changes
from -next. There is definitely not 6.1 material if the current
code is anything to go by. That'd be commits:
621b7555c914 ("dt-bindings: irqchip: imx mu work as msi controller")
893ae515ff8d ("irqchip: imx mu worked as msi controller")
Once it is ready, I'll take it via the irqchip tree. But only when
it is ready.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
2022-08-10 14:01 ` Rob Herring
@ 2022-08-10 14:32 ` Jon Mason
-1 siblings, 0 replies; 50+ messages in thread
From: Jon Mason @ 2022-08-10 14:32 UTC (permalink / raw)
To: Rob Herring
Cc: Frank Li, Marc Zyngier, Thomas Gleixner, Krzysztof Kozlowski,
Shawn Guo, Sascha Hauer, Krzysztof Wilczynski, Bjorn Helgaas,
linux-kernel@vger.kernel.org, devicetree, linux-arm-kernel, PCI,
Peng Fan, Dong Aisheng, Sascha Hauer, Fabio Estevam,
NXP Linux Team, Kishon Vijay Abraham I, Lorenzo Pieralisi, ntb
On Wed, Aug 10, 2022 at 08:01:29AM -0600, Rob Herring wrote:
> On Wed, Jul 20, 2022 at 3:31 PM Frank Li <Frank.Li@nxp.com> wrote:
> >
> > imx mu support generate irq by write a register.
> > provide msi controller support so other driver
> > can use it by standard msi interface.
> >
> > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > ---
> > .../interrupt-controller/fsl,mu-msi.yaml | 88 +++++++++++++++++++
> > 1 file changed, 88 insertions(+)
> > create mode 100644 Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
>
> This is failing in linux-next now, but I'm wondering why it is there
> given all the comments.
That was my fault. It is gone now. I was trying to get caught up
with patches, applied everything in my inbox to ntb-next, and was
pulling stuff out when it synced (cronjob backup of all git trees to
github).
Sorry,
Jon
>
> Error: Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.example.dts:31.41-42
> syntax error
> FATAL ERROR: Unable to parse input tree
> make[1]: *** [scripts/Makefile.lib:396:
> Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.example.dtb]
> Error 1
>
> Rob
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [PATCH v3 3/4] dt-bindings: irqchip: imx mu work as msi controller
@ 2022-08-10 14:32 ` Jon Mason
0 siblings, 0 replies; 50+ messages in thread
From: Jon Mason @ 2022-08-10 14:32 UTC (permalink / raw)
To: Rob Herring
Cc: Frank Li, Marc Zyngier, Thomas Gleixner, Krzysztof Kozlowski,
Shawn Guo, Sascha Hauer, Krzysztof Wilczynski, Bjorn Helgaas,
linux-kernel@vger.kernel.org, devicetree, linux-arm-kernel, PCI,
Peng Fan, Dong Aisheng, Sascha Hauer, Fabio Estevam,
NXP Linux Team, Kishon Vijay Abraham I, Lorenzo Pieralisi, ntb
On Wed, Aug 10, 2022 at 08:01:29AM -0600, Rob Herring wrote:
> On Wed, Jul 20, 2022 at 3:31 PM Frank Li <Frank.Li@nxp.com> wrote:
> >
> > imx mu support generate irq by write a register.
> > provide msi controller support so other driver
> > can use it by standard msi interface.
> >
> > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > ---
> > .../interrupt-controller/fsl,mu-msi.yaml | 88 +++++++++++++++++++
> > 1 file changed, 88 insertions(+)
> > create mode 100644 Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.yaml
>
> This is failing in linux-next now, but I'm wondering why it is there
> given all the comments.
That was my fault. It is gone now. I was trying to get caught up
with patches, applied everything in my inbox to ntb-next, and was
pulling stuff out when it synced (cronjob backup of all git trees to
github).
Sorry,
Jon
>
> Error: Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.example.dts:31.41-42
> syntax error
> FATAL ERROR: Unable to parse input tree
> make[1]: *** [scripts/Makefile.lib:396:
> Documentation/devicetree/bindings/interrupt-controller/fsl,mu-msi.example.dtb]
> Error 1
>
> Rob
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 50+ messages in thread
end of thread, other threads:[~2022-08-10 14:33 UTC | newest]
Thread overview: 50+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-07-20 21:30 [PATCH v3 0/4] PCI EP driver support MSI doorbell from host Frank Li
2022-07-20 21:30 ` Frank Li
2022-07-20 21:30 ` [PATCH v3 1/4] irqchip: allow pass down .pm field at IRQCHIP_PLATFORM_DRIVER_END Frank Li
2022-07-20 21:30 ` Frank Li
2022-07-20 21:30 ` [PATCH v3 2/4] irqchip: imx mu worked as msi controller Frank Li
2022-07-20 21:30 ` Frank Li
2022-07-21 7:57 ` Marc Zyngier
2022-07-21 7:57 ` Marc Zyngier
2022-07-21 15:22 ` [EXT] " Frank Li
2022-07-21 15:22 ` Frank Li
2022-07-21 15:28 ` Marc Zyngier
2022-07-21 15:28 ` Marc Zyngier
2022-07-21 15:35 ` Frank Li
2022-07-21 15:35 ` Frank Li
2022-07-26 21:48 ` Frank Li
2022-07-26 21:48 ` Frank Li
2022-07-27 8:02 ` Marc Zyngier
2022-07-27 8:02 ` Marc Zyngier
2022-07-27 15:23 ` Frank Li
2022-07-27 15:23 ` Frank Li
2022-07-27 15:34 ` Marc Zyngier
2022-07-27 15:34 ` Marc Zyngier
2022-07-27 18:29 ` Frank Li
2022-07-27 18:29 ` Frank Li
2022-07-27 18:58 ` Frank Li
2022-07-27 18:58 ` Frank Li
2022-07-22 7:33 ` Marc Zyngier
2022-07-22 7:33 ` Marc Zyngier
2022-07-22 16:12 ` Frank Li
2022-07-22 16:12 ` Frank Li
2022-07-20 21:30 ` [PATCH v3 3/4] dt-bindings: irqchip: imx mu work " Frank Li
2022-07-20 21:30 ` Frank Li
2022-07-23 18:50 ` Krzysztof Kozlowski
2022-07-23 18:50 ` Krzysztof Kozlowski
2022-07-25 16:29 ` [EXT] " Frank Li
2022-07-25 16:29 ` Frank Li
2022-07-25 16:44 ` Krzysztof Kozlowski
2022-07-25 16:44 ` Krzysztof Kozlowski
2022-07-25 16:55 ` Frank Li
2022-07-25 16:55 ` Frank Li
2022-07-25 20:28 ` Krzysztof Kozlowski
2022-07-25 20:28 ` Krzysztof Kozlowski
2022-08-10 14:01 ` Rob Herring
2022-08-10 14:01 ` Rob Herring
2022-08-10 14:20 ` Marc Zyngier
2022-08-10 14:20 ` Marc Zyngier
2022-08-10 14:32 ` Jon Mason
2022-08-10 14:32 ` Jon Mason
2022-07-20 21:30 ` [PATCH v3 4/4] pcie: endpoint: pci-epf-vntb: add endpoint MSI support Frank Li
2022-07-20 21:30 ` Frank Li
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.