[PATCH 0/6] PCI: mvebu: Add support for PME and AER interrupts

[PATCH 0/6] PCI: mvebu: Add support for PME and AER interrupts

[Pali Rohár]

mvebu PCIe PME and AER interrupts are reported via PCIe summary
interrupt. PCIe summary interrupt is reported via mvebu MPIC SoC error
summary interrupt. And MPIC SoC error summary interrupt is reported via
MPIC IRQ 4.

This patch series implements support for interrupts in MPIC SoC error
hierarchy in irq-armada-370-xp.c driver and support for interrupts in
mvebu PCIe hierarchy in pci-mvebu.c.

Finally PCIe PME and AER interrupts are routed to the correct PCIe Root
Port, which allows kernel PME and AER drivers to take care of them.

Tested on A385 board and kernel PME and AER drivers works correctly:

[    0.898482] pcieport 0000:00:01.0: PME: Signaling with IRQ 61
[    0.904422] pcieport 0000:00:01.0: AER: enabled with IRQ 61
[    0.910113] pcieport 0000:00:02.0: enabling device (0140 -> 0142)
[    0.916299] pcieport 0000:00:02.0: PME: Signaling with IRQ 62
[    0.922216] pcieport 0000:00:02.0: AER: enabled with IRQ 62
[    0.927917] pcieport 0000:00:03.0: enabling device (0140 -> 0142)
[    0.934090] pcieport 0000:00:03.0: PME: Signaling with IRQ 63
[    0.940006] pcieport 0000:00:03.0: AER: enabled with IRQ 63

This change finally allows to debug PCIe issues on A385 boards.

Pali Rohár (6):
  dt-bindings: irqchip: armada-370-xp: Update information about MPIC SoC
    Error
  irqchip/armada-370-xp: Implement SoC Error interrupts
  ARM: dts: armada-38x.dtsi: Add node for MPIC SoC Error IRQ controller
  dt-bindings: PCI: mvebu: Update information about summary interrupt
  PCI: mvebu: Implement support for interrupts on emulated bridge
  ARM: dts: armada-385.dtsi: Add definitions for PCIe summary interrupts

 .../marvell,armada-370-xp-mpic.txt            |   9 +
 .../devicetree/bindings/pci/mvebu-pci.txt     |   1 +
 arch/arm/boot/dts/armada-385.dtsi             |  20 +-
 arch/arm/boot/dts/armada-38x.dtsi             |   5 +
 drivers/irqchip/irq-armada-370-xp.c           | 213 +++++++++++++++++-
 drivers/pci/controller/pci-mvebu.c            | 208 +++++++++++++++--
 6 files changed, 426 insertions(+), 30 deletions(-)

-- 
2.20.1

[PATCH 1/6] dt-bindings: irqchip: armada-370-xp: Update information about MPIC SoC Error

[Pali Rohár]

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 .../interrupt-controller/marvell,armada-370-xp-mpic.txt  | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/Documentation/devicetree/bindings/interrupt-controller/marvell,armada-370-xp-mpic.txt b/Documentation/devicetree/bindings/interrupt-controller/marvell,armada-370-xp-mpic.txt
index 5fc03134a999..8cddbc16ddbd 100644
--- a/Documentation/devicetree/bindings/interrupt-controller/marvell,armada-370-xp-mpic.txt
+++ b/Documentation/devicetree/bindings/interrupt-controller/marvell,armada-370-xp-mpic.txt
@@ -24,6 +24,11 @@ Optional properties:
   connected as a slave to the Cortex-A9 GIC. The provided interrupt
   indicate to which GIC interrupt the MPIC output is connected.
 
+Optional subnodes:
+
+- interrupt-controller@20 with interrupt-controller property for
+  MPIC SoC Error IRQ controller
+
 Example:
 
         mpic: interrupt-controller@d0020000 {
@@ -35,4 +40,8 @@ Example:
               msi-controller;
               reg = <0xd0020a00 0x1d0>,
                     <0xd0021070 0x58>;
+              soc_err: interrupt-controller@20 {
+                    interrupt-controller;
+                    #interrupt-cells = <1>;
+              };
         };
-- 
2.20.1

Re: [PATCH 1/6] dt-bindings: irqchip: armada-370-xp: Update information about MPIC SoC Error

[Rob Herring]

On Fri, May 06, 2022 at 03:40:24PM +0200, Pali Rohár wrote:
> Signed-off-by: Pali Rohár <pali@kernel.org>

Why do we need/want this change?

> ---
>  .../interrupt-controller/marvell,armada-370-xp-mpic.txt  | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/Documentation/devicetree/bindings/interrupt-controller/marvell,armada-370-xp-mpic.txt b/Documentation/devicetree/bindings/interrupt-controller/marvell,armada-370-xp-mpic.txt
> index 5fc03134a999..8cddbc16ddbd 100644
> --- a/Documentation/devicetree/bindings/interrupt-controller/marvell,armada-370-xp-mpic.txt
> +++ b/Documentation/devicetree/bindings/interrupt-controller/marvell,armada-370-xp-mpic.txt
> @@ -24,6 +24,11 @@ Optional properties:
>    connected as a slave to the Cortex-A9 GIC. The provided interrupt
>    indicate to which GIC interrupt the MPIC output is connected.
>  
> +Optional subnodes:
> +
> +- interrupt-controller@20 with interrupt-controller property for
> +  MPIC SoC Error IRQ controller
> +
>  Example:
>  
>          mpic: interrupt-controller@d0020000 {
> @@ -35,4 +40,8 @@ Example:
>                msi-controller;
>                reg = <0xd0020a00 0x1d0>,
>                      <0xd0021070 0x58>;
> +              soc_err: interrupt-controller@20 {

unit address without 'reg' is an error.

> +                    interrupt-controller;
> +                    #interrupt-cells = <1>;
> +              };
>          };
> -- 
> 2.20.1
> 
> 

[PATCH 2/6] irqchip/armada-370-xp: Implement SoC Error interrupts

[Pali Rohár]

MPIC IRQ 4 is used as SoC Error Summary interrupt and provides access to
another hierarchy of SoC Error interrupts. Implement a new IRQ chip and
domain for accessing this IRQ hierarchy.

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 drivers/irqchip/irq-armada-370-xp.c | 213 +++++++++++++++++++++++++++-
 1 file changed, 210 insertions(+), 3 deletions(-)

diff --git a/drivers/irqchip/irq-armada-370-xp.c b/drivers/irqchip/irq-armada-370-xp.c
index ebd76ea1c69b..71578b65f5c8 100644
--- a/drivers/irqchip/irq-armada-370-xp.c
+++ b/drivers/irqchip/irq-armada-370-xp.c
@@ -117,6 +117,8 @@
 /* Registers relative to main_int_base */
 #define ARMADA_370_XP_INT_CONTROL		(0x00)
 #define ARMADA_370_XP_SW_TRIG_INT_OFFS		(0x04)
+#define ARMADA_370_XP_INT_SOC_ERR_0_CAUSE_OFFS	(0x20)
+#define ARMADA_370_XP_INT_SOC_ERR_1_CAUSE_OFFS	(0x24)
 #define ARMADA_370_XP_INT_SET_ENABLE_OFFS	(0x30)
 #define ARMADA_370_XP_INT_CLEAR_ENABLE_OFFS	(0x34)
 #define ARMADA_370_XP_INT_SOURCE_CTL(irq)	(0x100 + irq*4)
@@ -130,6 +132,8 @@
 #define ARMADA_370_XP_CPU_INTACK_OFFS		(0x44)
 #define ARMADA_370_XP_INT_SET_MASK_OFFS		(0x48)
 #define ARMADA_370_XP_INT_CLEAR_MASK_OFFS	(0x4C)
+#define ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF	(0x50)
+#define ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF	(0x54)
 #define ARMADA_370_XP_INT_FABRIC_MASK_OFFS	(0x54)
 #define ARMADA_370_XP_INT_CAUSE_PERF(cpu)	(1 << cpu)
 
@@ -146,6 +150,8 @@
 static void __iomem *per_cpu_int_base;
 static void __iomem *main_int_base;
 static struct irq_domain *armada_370_xp_mpic_domain;
+static struct irq_domain *armada_370_xp_soc_err_domain;
+static unsigned int soc_err_irq_num_regs;
 static u32 doorbell_mask_reg;
 static int parent_irq;
 #ifdef CONFIG_PCI_MSI
@@ -156,6 +162,8 @@ static DEFINE_MUTEX(msi_used_lock);
 static phys_addr_t msi_doorbell_addr;
 #endif
 
+static void armada_370_xp_soc_err_irq_unmask(struct irq_data *d);
+
 static inline bool is_percpu_irq(irq_hw_number_t irq)
 {
 	if (irq <= ARMADA_370_XP_MAX_PER_CPU_IRQS)
@@ -509,6 +517,27 @@ static void armada_xp_mpic_reenable_percpu(void)
 		armada_370_xp_irq_unmask(data);
 	}
 
+	/* Re-enable per-CPU SoC Error interrupts that were enabled before suspend */
+	for (irq = 0; irq < soc_err_irq_num_regs * 32; irq++) {
+		struct irq_data *data;
+		int virq;
+
+		virq = irq_linear_revmap(armada_370_xp_soc_err_domain, irq);
+		if (virq == 0)
+			continue;
+
+		data = irq_get_irq_data(virq);
+
+		if (!irq_percpu_is_enabled(virq))
+			continue;
+
+		armada_370_xp_soc_err_irq_unmask(data);
+	}
+
+	/* Unmask summary SoC Error Interrupt */
+	if (soc_err_irq_num_regs > 0)
+		writel(4, per_cpu_int_base + ARMADA_370_XP_INT_CLEAR_MASK_OFFS);
+
 	ipi_resume();
 }
 
@@ -546,8 +575,8 @@ static struct irq_chip armada_370_xp_irq_chip = {
 static int armada_370_xp_mpic_irq_map(struct irq_domain *h,
 				      unsigned int virq, irq_hw_number_t hw)
 {
-	/* IRQs 0 and 1 cannot be mapped, they are handled internally */
-	if (hw <= 1)
+	/* IRQs 0, 1 and 4 cannot be mapped, they are handled internally */
+	if (hw <= 1 || hw == 4)
 		return -EINVAL;
 
 	armada_370_xp_irq_mask(irq_get_irq_data(virq));
@@ -577,6 +606,99 @@ static const struct irq_domain_ops armada_370_xp_mpic_irq_ops = {
 	.xlate = irq_domain_xlate_onecell,
 };
 
+static DEFINE_RAW_SPINLOCK(armada_370_xp_soc_err_lock);
+
+static void armada_370_xp_soc_err_irq_mask(struct irq_data *d)
+{
+	irq_hw_number_t hwirq = irqd_to_hwirq(d);
+	u32 reg, mask;
+
+	reg = hwirq >= 32 ? ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF
+			  : ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF;
+
+	raw_spin_lock(&armada_370_xp_soc_err_lock);
+	mask = readl(per_cpu_int_base + reg);
+	mask &= ~BIT(hwirq % 32);
+	writel(mask, per_cpu_int_base + reg);
+	raw_spin_unlock(&armada_370_xp_soc_err_lock);
+}
+
+static void armada_370_xp_soc_err_irq_unmask(struct irq_data *d)
+{
+	irq_hw_number_t hwirq = irqd_to_hwirq(d);
+	u32 reg, mask;
+
+	reg = hwirq >= 32 ? ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF
+			  : ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF;
+
+	raw_spin_lock(&armada_370_xp_soc_err_lock);
+	mask = readl(per_cpu_int_base + reg);
+	mask |= BIT(hwirq % 32);
+	writel(mask, per_cpu_int_base + reg);
+	raw_spin_unlock(&armada_370_xp_soc_err_lock);
+}
+
+static int armada_370_xp_soc_err_irq_mask_on_cpu(void *par)
+{
+	struct irq_data *d = par;
+	armada_370_xp_soc_err_irq_mask(d);
+	return 0;
+}
+
+static int armada_370_xp_soc_err_irq_unmask_on_cpu(void *par)
+{
+	struct irq_data *d = par;
+	armada_370_xp_soc_err_irq_unmask(d);
+	return 0;
+}
+
+static int armada_xp_soc_err_irq_set_affinity(struct irq_data *d,
+					      const struct cpumask *mask,
+					      bool force)
+{
+	unsigned int cpu;
+
+	cpus_read_lock();
+
+	/* First disable IRQ on all cores */
+	for_each_online_cpu(cpu)
+		smp_call_on_cpu(cpu, armada_370_xp_soc_err_irq_mask_on_cpu, d, true);
+
+	/* Select a single core from the affinity mask which is online */
+	cpu = cpumask_any_and(mask, cpu_online_mask);
+	smp_call_on_cpu(cpu, armada_370_xp_soc_err_irq_unmask_on_cpu, d, true);
+
+	cpus_read_unlock();
+
+	irq_data_update_effective_affinity(d, cpumask_of(cpu));
+
+	return IRQ_SET_MASK_OK;
+}
+
+static struct irq_chip armada_370_xp_soc_err_irq_chip = {
+	.name = "MPIC SOC",
+	.irq_mask = armada_370_xp_soc_err_irq_mask,
+	.irq_unmask = armada_370_xp_soc_err_irq_unmask,
+	.irq_set_affinity = armada_xp_soc_err_irq_set_affinity,
+};
+
+static int armada_370_xp_soc_err_irq_map(struct irq_domain *h,
+					 unsigned int virq, irq_hw_number_t hw)
+{
+	armada_370_xp_soc_err_irq_mask(irq_get_irq_data(virq));
+	irq_set_status_flags(virq, IRQ_LEVEL);
+	irq_set_percpu_devid(virq);
+	irq_set_chip_and_handler(virq, &armada_370_xp_soc_err_irq_chip,
+				 handle_percpu_devid_irq);
+	irq_set_probe(virq);
+	return 0;
+}
+
+static const struct irq_domain_ops armada_370_xp_soc_err_irq_ops = {
+	.map = armada_370_xp_soc_err_irq_map,
+	.xlate = irq_domain_xlate_onecell,
+};
+
 #ifdef CONFIG_PCI_MSI
 static void armada_370_xp_handle_msi_irq(struct pt_regs *regs, bool is_chained)
 {
@@ -605,6 +727,32 @@ static void armada_370_xp_handle_msi_irq(struct pt_regs *regs, bool is_chained)
 static void armada_370_xp_handle_msi_irq(struct pt_regs *r, bool b) {}
 #endif
 
+static void armada_370_xp_handle_soc_err_irq(void)
+{
+	unsigned long status, bit;
+	u32 mask, cause;
+
+	if (soc_err_irq_num_regs < 1)
+		return;
+
+	mask = readl(per_cpu_int_base + ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF);
+	cause = readl(main_int_base + ARMADA_370_XP_INT_SOC_ERR_0_CAUSE_OFFS);
+	status = cause & mask;
+
+	for_each_set_bit(bit, &status, 32)
+		generic_handle_domain_irq(armada_370_xp_soc_err_domain, bit);
+
+	if (soc_err_irq_num_regs < 2)
+		return;
+
+	mask = readl(per_cpu_int_base + ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF);
+	cause = readl(main_int_base + ARMADA_370_XP_INT_SOC_ERR_1_CAUSE_OFFS);
+	status = cause & mask;
+
+	for_each_set_bit(bit, &status, 32)
+		generic_handle_domain_irq(armada_370_xp_soc_err_domain, bit + 32);
+}
+
 static void armada_370_xp_mpic_handle_cascade_irq(struct irq_desc *desc)
 {
 	struct irq_chip *chip = irq_desc_get_chip(desc);
@@ -630,6 +778,11 @@ static void armada_370_xp_mpic_handle_cascade_irq(struct irq_desc *desc)
 			continue;
 		}
 
+		if (irqn == 4) {
+			armada_370_xp_handle_soc_err_irq();
+			continue;
+		}
+
 		generic_handle_domain_irq(armada_370_xp_mpic_domain, irqn);
 	}
 
@@ -649,7 +802,7 @@ armada_370_xp_handle_irq(struct pt_regs *regs)
 		if (irqnr > 1022)
 			break;
 
-		if (irqnr > 1) {
+		if (irqnr > 1 && irqnr != 4) {
 			generic_handle_domain_irq(armada_370_xp_mpic_domain,
 						  irqnr);
 			continue;
@@ -659,6 +812,10 @@ armada_370_xp_handle_irq(struct pt_regs *regs)
 		if (irqnr == 1)
 			armada_370_xp_handle_msi_irq(regs, false);
 
+		/* SoC Error handling */
+		if (irqnr == 4)
+			armada_370_xp_handle_soc_err_irq();
+
 #ifdef CONFIG_SMP
 		/* IPI Handling */
 		if (irqnr == 0) {
@@ -722,6 +879,26 @@ static void armada_370_xp_mpic_resume(void)
 		}
 	}
 
+	/* Re-enable per-CPU SoC Error interrupts */
+	for (irq = 0; irq < soc_err_irq_num_regs * 32; irq++) {
+		struct irq_data *data;
+		int virq;
+
+		virq = irq_linear_revmap(armada_370_xp_soc_err_domain, irq);
+		if (virq == 0)
+			continue;
+
+		data = irq_get_irq_data(virq);
+
+		/*
+		 * Re-enable on the current CPU,
+		 * armada_xp_mpic_reenable_percpu() will take
+		 * care of secondary CPUs when they come up.
+		 */
+		if (irq_percpu_is_enabled(virq))
+			armada_370_xp_soc_err_irq_unmask(data);
+	}
+
 	/* Reconfigure doorbells for IPIs and MSIs */
 	writel(doorbell_mask_reg,
 	       per_cpu_int_base + ARMADA_370_XP_IN_DRBEL_MSK_OFFS);
@@ -730,6 +907,10 @@ static void armada_370_xp_mpic_resume(void)
 	if (doorbell_mask_reg & PCI_MSI_DOORBELL_MASK)
 		writel(1, per_cpu_int_base + ARMADA_370_XP_INT_CLEAR_MASK_OFFS);
 
+	/* Unmask summary SoC Error Interrupt */
+	if (soc_err_irq_num_regs > 0)
+		writel(4, per_cpu_int_base + ARMADA_370_XP_INT_CLEAR_MASK_OFFS);
+
 	ipi_resume();
 }
 
@@ -742,6 +923,7 @@ static int __init armada_370_xp_mpic_of_init(struct device_node *node,
 					     struct device_node *parent)
 {
 	struct resource main_int_res, per_cpu_int_res;
+	struct device_node *soc_err_node;
 	int nr_irqs, i;
 	u32 control;
 
@@ -775,12 +957,37 @@ static int __init armada_370_xp_mpic_of_init(struct device_node *node,
 	BUG_ON(!armada_370_xp_mpic_domain);
 	irq_domain_update_bus_token(armada_370_xp_mpic_domain, DOMAIN_BUS_WIRED);
 
+	soc_err_node = of_get_next_child(node, NULL);
+	if (!soc_err_node) {
+		pr_warn("Missing SoC Error Interrupt Controller node\n");
+		pr_warn("Extended interrupts are not supported\n");
+	} else {
+		pr_info("Registering MPIC SoC Error Interrupt Controller\n");
+		/*
+		 * Armada 370 and XP have only 32 SoC Error IRQs in one register
+		 * and other Armada platforms have 64 IRQs in two registers.
+		 */
+		soc_err_irq_num_regs =
+			of_machine_is_compatible("marvell,armada-370-xp") ? 1 : 2;
+		armada_370_xp_soc_err_domain =
+			irq_domain_add_hierarchy(armada_370_xp_mpic_domain, 0,
+						 soc_err_irq_num_regs * 32,
+						 soc_err_node,
+						 &armada_370_xp_soc_err_irq_ops,
+						 NULL);
+		BUG_ON(!armada_370_xp_soc_err_domain);
+	}
+
 	/* Setup for the boot CPU */
 	armada_xp_mpic_perf_init();
 	armada_xp_mpic_smp_cpu_init();
 
 	armada_370_xp_msi_init(node, main_int_res.start);
 
+	/* Unmask summary SoC Error Interrupt */
+	if (soc_err_irq_num_regs > 0)
+		writel(4, per_cpu_int_base + ARMADA_370_XP_INT_CLEAR_MASK_OFFS);
+
 	parent_irq = irq_of_parse_and_map(node, 0);
 	if (parent_irq <= 0) {
 		irq_set_default_host(armada_370_xp_mpic_domain);
-- 
2.20.1

Re: [PATCH 2/6] irqchip/armada-370-xp: Implement SoC Error interrupts

[Marc Zyngier]

On Fri, 06 May 2022 14:40:25 +0100,
Pali Rohár <pali@kernel.org> wrote:
> 
> MPIC IRQ 4 is used as SoC Error Summary interrupt and provides access to
> another hierarchy of SoC Error interrupts. Implement a new IRQ chip and
> domain for accessing this IRQ hierarchy.
> 
> Signed-off-by: Pali Rohár <pali@kernel.org>
> ---
>  drivers/irqchip/irq-armada-370-xp.c | 213 +++++++++++++++++++++++++++-
>  1 file changed, 210 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/irqchip/irq-armada-370-xp.c b/drivers/irqchip/irq-armada-370-xp.c
> index ebd76ea1c69b..71578b65f5c8 100644
> --- a/drivers/irqchip/irq-armada-370-xp.c
> +++ b/drivers/irqchip/irq-armada-370-xp.c
> @@ -117,6 +117,8 @@
>  /* Registers relative to main_int_base */
>  #define ARMADA_370_XP_INT_CONTROL		(0x00)
>  #define ARMADA_370_XP_SW_TRIG_INT_OFFS		(0x04)
> +#define ARMADA_370_XP_INT_SOC_ERR_0_CAUSE_OFFS	(0x20)
> +#define ARMADA_370_XP_INT_SOC_ERR_1_CAUSE_OFFS	(0x24)
>  #define ARMADA_370_XP_INT_SET_ENABLE_OFFS	(0x30)
>  #define ARMADA_370_XP_INT_CLEAR_ENABLE_OFFS	(0x34)
>  #define ARMADA_370_XP_INT_SOURCE_CTL(irq)	(0x100 + irq*4)
> @@ -130,6 +132,8 @@
>  #define ARMADA_370_XP_CPU_INTACK_OFFS		(0x44)
>  #define ARMADA_370_XP_INT_SET_MASK_OFFS		(0x48)
>  #define ARMADA_370_XP_INT_CLEAR_MASK_OFFS	(0x4C)
> +#define ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF	(0x50)
> +#define ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF	(0x54)
>  #define ARMADA_370_XP_INT_FABRIC_MASK_OFFS	(0x54)
>  #define ARMADA_370_XP_INT_CAUSE_PERF(cpu)	(1 << cpu)
>  
> @@ -146,6 +150,8 @@
>  static void __iomem *per_cpu_int_base;
>  static void __iomem *main_int_base;
>  static struct irq_domain *armada_370_xp_mpic_domain;
> +static struct irq_domain *armada_370_xp_soc_err_domain;
> +static unsigned int soc_err_irq_num_regs;
>  static u32 doorbell_mask_reg;
>  static int parent_irq;
>  #ifdef CONFIG_PCI_MSI
> @@ -156,6 +162,8 @@ static DEFINE_MUTEX(msi_used_lock);
>  static phys_addr_t msi_doorbell_addr;
>  #endif
>  
> +static void armada_370_xp_soc_err_irq_unmask(struct irq_data *d);
> +
>  static inline bool is_percpu_irq(irq_hw_number_t irq)
>  {
>  	if (irq <= ARMADA_370_XP_MAX_PER_CPU_IRQS)
> @@ -509,6 +517,27 @@ static void armada_xp_mpic_reenable_percpu(void)
>  		armada_370_xp_irq_unmask(data);
>  	}
>  
> +	/* Re-enable per-CPU SoC Error interrupts that were enabled before suspend */
> +	for (irq = 0; irq < soc_err_irq_num_regs * 32; irq++) {
> +		struct irq_data *data;
> +		int virq;
> +
> +		virq = irq_linear_revmap(armada_370_xp_soc_err_domain, irq);
> +		if (virq == 0)
> +			continue;
> +
> +		data = irq_get_irq_data(virq);
> +
> +		if (!irq_percpu_is_enabled(virq))
> +			continue;
> +
> +		armada_370_xp_soc_err_irq_unmask(data);
> +	}

So you do this loop and all these lookups, both here and in the resume
function (duplicated code!) just to be able to call the unmask
function?  This would be better served by two straight writes of the
mask register, which you'd conveniently save on suspend.

Yes, you have only duplicated the existing logic. But surely there is
something better to do.

> +
> +	/* Unmask summary SoC Error Interrupt */
> +	if (soc_err_irq_num_regs > 0)
> +		writel(4, per_cpu_int_base + ARMADA_370_XP_INT_CLEAR_MASK_OFFS);
> +
>  	ipi_resume();
>  }
>  
> @@ -546,8 +575,8 @@ static struct irq_chip armada_370_xp_irq_chip = {
>  static int armada_370_xp_mpic_irq_map(struct irq_domain *h,
>  				      unsigned int virq, irq_hw_number_t hw)
>  {
> -	/* IRQs 0 and 1 cannot be mapped, they are handled internally */
> -	if (hw <= 1)
> +	/* IRQs 0, 1 and 4 cannot be mapped, they are handled internally */
> +	if (hw <= 1 || hw == 4)
>  		return -EINVAL;
>  
>  	armada_370_xp_irq_mask(irq_get_irq_data(virq));
> @@ -577,6 +606,99 @@ static const struct irq_domain_ops armada_370_xp_mpic_irq_ops = {
>  	.xlate = irq_domain_xlate_onecell,
>  };
>  
> +static DEFINE_RAW_SPINLOCK(armada_370_xp_soc_err_lock);
> +
> +static void armada_370_xp_soc_err_irq_mask(struct irq_data *d)
> +{
> +	irq_hw_number_t hwirq = irqd_to_hwirq(d);
> +	u32 reg, mask;
> +
> +	reg = hwirq >= 32 ? ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF
> +			  : ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF;
> +
> +	raw_spin_lock(&armada_370_xp_soc_err_lock);
> +	mask = readl(per_cpu_int_base + reg);
> +	mask &= ~BIT(hwirq % 32);
> +	writel(mask, per_cpu_int_base + reg);
> +	raw_spin_unlock(&armada_370_xp_soc_err_lock);
> +}
> +
> +static void armada_370_xp_soc_err_irq_unmask(struct irq_data *d)
> +{
> +	irq_hw_number_t hwirq = irqd_to_hwirq(d);
> +	u32 reg, mask;
> +
> +	reg = hwirq >= 32 ? ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF
> +			  : ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF;
> +
> +	raw_spin_lock(&armada_370_xp_soc_err_lock);
> +	mask = readl(per_cpu_int_base + reg);
> +	mask |= BIT(hwirq % 32);
> +	writel(mask, per_cpu_int_base + reg);
> +	raw_spin_unlock(&armada_370_xp_soc_err_lock);
> +}
> +
> +static int armada_370_xp_soc_err_irq_mask_on_cpu(void *par)
> +{
> +	struct irq_data *d = par;
> +	armada_370_xp_soc_err_irq_mask(d);
> +	return 0;
> +}
> +
> +static int armada_370_xp_soc_err_irq_unmask_on_cpu(void *par)
> +{
> +	struct irq_data *d = par;
> +	armada_370_xp_soc_err_irq_unmask(d);
> +	return 0;
> +}
> +
> +static int armada_xp_soc_err_irq_set_affinity(struct irq_data *d,
> +					      const struct cpumask *mask,
> +					      bool force)
> +{
> +	unsigned int cpu;
> +
> +	cpus_read_lock();
> +
> +	/* First disable IRQ on all cores */
> +	for_each_online_cpu(cpu)
> +		smp_call_on_cpu(cpu, armada_370_xp_soc_err_irq_mask_on_cpu, d, true);
> +
> +	/* Select a single core from the affinity mask which is online */
> +	cpu = cpumask_any_and(mask, cpu_online_mask);
> +	smp_call_on_cpu(cpu, armada_370_xp_soc_err_irq_unmask_on_cpu, d, true);
> +
> +	cpus_read_unlock();
> +
> +	irq_data_update_effective_affinity(d, cpumask_of(cpu));
> +
> +	return IRQ_SET_MASK_OK;
> +}

Aren't these per-CPU interrupts anyway? What does it mean to set their
affinity? /me rolls eyes...

> +
> +static struct irq_chip armada_370_xp_soc_err_irq_chip = {
> +	.name = "MPIC SOC",
> +	.irq_mask = armada_370_xp_soc_err_irq_mask,
> +	.irq_unmask = armada_370_xp_soc_err_irq_unmask,
> +	.irq_set_affinity = armada_xp_soc_err_irq_set_affinity,
> +};
> +
> +static int armada_370_xp_soc_err_irq_map(struct irq_domain *h,
> +					 unsigned int virq, irq_hw_number_t hw)
> +{
> +	armada_370_xp_soc_err_irq_mask(irq_get_irq_data(virq));
> +	irq_set_status_flags(virq, IRQ_LEVEL);
> +	irq_set_percpu_devid(virq);
> +	irq_set_chip_and_handler(virq, &armada_370_xp_soc_err_irq_chip,
> +				 handle_percpu_devid_irq);
> +	irq_set_probe(virq);
> +	return 0;
> +}
> +
> +static const struct irq_domain_ops armada_370_xp_soc_err_irq_ops = {
> +	.map = armada_370_xp_soc_err_irq_map,
> +	.xlate = irq_domain_xlate_onecell,
> +};
> +
>  #ifdef CONFIG_PCI_MSI
>  static void armada_370_xp_handle_msi_irq(struct pt_regs *regs, bool is_chained)
>  {
> @@ -605,6 +727,32 @@ static void armada_370_xp_handle_msi_irq(struct pt_regs *regs, bool is_chained)
>  static void armada_370_xp_handle_msi_irq(struct pt_regs *r, bool b) {}
>  #endif
>  
> +static void armada_370_xp_handle_soc_err_irq(void)
> +{
> +	unsigned long status, bit;
> +	u32 mask, cause;
> +
> +	if (soc_err_irq_num_regs < 1)
> +		return;
> +
> +	mask = readl(per_cpu_int_base + ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF);
> +	cause = readl(main_int_base + ARMADA_370_XP_INT_SOC_ERR_0_CAUSE_OFFS);
> +	status = cause & mask;
> +
> +	for_each_set_bit(bit, &status, 32)
> +		generic_handle_domain_irq(armada_370_xp_soc_err_domain, bit);
> +
> +	if (soc_err_irq_num_regs < 2)
> +		return;
> +
> +	mask = readl(per_cpu_int_base + ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF);
> +	cause = readl(main_int_base + ARMADA_370_XP_INT_SOC_ERR_1_CAUSE_OFFS);
> +	status = cause & mask;
> +
> +	for_each_set_bit(bit, &status, 32)
> +		generic_handle_domain_irq(armada_370_xp_soc_err_domain, bit + 32);
> +}
> +
>  static void armada_370_xp_mpic_handle_cascade_irq(struct irq_desc *desc)
>  {
>  	struct irq_chip *chip = irq_desc_get_chip(desc);
> @@ -630,6 +778,11 @@ static void armada_370_xp_mpic_handle_cascade_irq(struct irq_desc *desc)
>  			continue;
>  		}
>  
> +		if (irqn == 4) {
> +			armada_370_xp_handle_soc_err_irq();
> +			continue;
> +		}
> +
>  		generic_handle_domain_irq(armada_370_xp_mpic_domain, irqn);
>  	}
>  
> @@ -649,7 +802,7 @@ armada_370_xp_handle_irq(struct pt_regs *regs)
>  		if (irqnr > 1022)
>  			break;
>  
> -		if (irqnr > 1) {
> +		if (irqnr > 1 && irqnr != 4) {
>  			generic_handle_domain_irq(armada_370_xp_mpic_domain,
>  						  irqnr);
>  			continue;
> @@ -659,6 +812,10 @@ armada_370_xp_handle_irq(struct pt_regs *regs)
>  		if (irqnr == 1)
>  			armada_370_xp_handle_msi_irq(regs, false);
>  
> +		/* SoC Error handling */
> +		if (irqnr == 4)
> +			armada_370_xp_handle_soc_err_irq();
> +
>  #ifdef CONFIG_SMP
>  		/* IPI Handling */
>  		if (irqnr == 0) {
> @@ -722,6 +879,26 @@ static void armada_370_xp_mpic_resume(void)
>  		}
>  	}
>  
> +	/* Re-enable per-CPU SoC Error interrupts */
> +	for (irq = 0; irq < soc_err_irq_num_regs * 32; irq++) {
> +		struct irq_data *data;
> +		int virq;
> +
> +		virq = irq_linear_revmap(armada_370_xp_soc_err_domain, irq);
> +		if (virq == 0)
> +			continue;
> +
> +		data = irq_get_irq_data(virq);
> +
> +		/*
> +		 * Re-enable on the current CPU,
> +		 * armada_xp_mpic_reenable_percpu() will take
> +		 * care of secondary CPUs when they come up.
> +		 */
> +		if (irq_percpu_is_enabled(virq))
> +			armada_370_xp_soc_err_irq_unmask(data);
> +	}

As I said above, this is duplicated code that should be replaced with
a simple write to the corresponding MMIO registers.

> +
>  	/* Reconfigure doorbells for IPIs and MSIs */
>  	writel(doorbell_mask_reg,
>  	       per_cpu_int_base + ARMADA_370_XP_IN_DRBEL_MSK_OFFS);
> @@ -730,6 +907,10 @@ static void armada_370_xp_mpic_resume(void)
>  	if (doorbell_mask_reg & PCI_MSI_DOORBELL_MASK)
>  		writel(1, per_cpu_int_base + ARMADA_370_XP_INT_CLEAR_MASK_OFFS);
>  
> +	/* Unmask summary SoC Error Interrupt */
> +	if (soc_err_irq_num_regs > 0)
> +		writel(4, per_cpu_int_base + ARMADA_370_XP_INT_CLEAR_MASK_OFFS);

Magic value?

Also, writing to this register tends to indicate that the whole thing
should really be a chained irqchip... Maybe that's overkill in this
instance, but the whole thing is rather oddly architected.

> +
>  	ipi_resume();
>  }
>  
> @@ -742,6 +923,7 @@ static int __init armada_370_xp_mpic_of_init(struct device_node *node,
>  					     struct device_node *parent)
>  {
>  	struct resource main_int_res, per_cpu_int_res;
> +	struct device_node *soc_err_node;
>  	int nr_irqs, i;
>  	u32 control;
>  
> @@ -775,12 +957,37 @@ static int __init armada_370_xp_mpic_of_init(struct device_node *node,
>  	BUG_ON(!armada_370_xp_mpic_domain);
>  	irq_domain_update_bus_token(armada_370_xp_mpic_domain, DOMAIN_BUS_WIRED);
>  
> +	soc_err_node = of_get_next_child(node, NULL);
> +	if (!soc_err_node) {
> +		pr_warn("Missing SoC Error Interrupt Controller node\n");
> +		pr_warn("Extended interrupts are not supported\n");
> +	} else {
> +		pr_info("Registering MPIC SoC Error Interrupt Controller\n");
> +		/*
> +		 * Armada 370 and XP have only 32 SoC Error IRQs in one register
> +		 * and other Armada platforms have 64 IRQs in two registers.
> +		 */
> +		soc_err_irq_num_regs =
> +			of_machine_is_compatible("marvell,armada-370-xp") ? 1 : 2;

Don't you have an actual compatible string for the interrupt
controller?  It seems odd to rely on the SoC name.

> +		armada_370_xp_soc_err_domain =
> +			irq_domain_add_hierarchy(armada_370_xp_mpic_domain, 0,
> +						 soc_err_irq_num_regs * 32,
> +						 soc_err_node,
> +						 &armada_370_xp_soc_err_irq_ops,
> +						 NULL);
> +		BUG_ON(!armada_370_xp_soc_err_domain);
> +	}
> +
>  	/* Setup for the boot CPU */
>  	armada_xp_mpic_perf_init();
>  	armada_xp_mpic_smp_cpu_init();
>  
>  	armada_370_xp_msi_init(node, main_int_res.start);
>  
> +	/* Unmask summary SoC Error Interrupt */
> +	if (soc_err_irq_num_regs > 0)
> +		writel(4, per_cpu_int_base + ARMADA_370_XP_INT_CLEAR_MASK_OFFS);
> +

Magic value, duplicated this time?

>  	parent_irq = irq_of_parse_and_map(node, 0);
>  	if (parent_irq <= 0) {
>  		irq_set_default_host(armada_370_xp_mpic_domain);
> -- 
> 2.20.1
> 
> 

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 2/6] irqchip/armada-370-xp: Implement SoC Error interrupts

[Pali Rohár]

On Friday 06 May 2022 19:19:46 Marc Zyngier wrote:
> On Fri, 06 May 2022 14:40:25 +0100,
> Pali Rohár <pali@kernel.org> wrote:
> > 
> > MPIC IRQ 4 is used as SoC Error Summary interrupt and provides access to
> > another hierarchy of SoC Error interrupts. Implement a new IRQ chip and
> > domain for accessing this IRQ hierarchy.
> > 
> > Signed-off-by: Pali Rohár <pali@kernel.org>
> > ---
> >  drivers/irqchip/irq-armada-370-xp.c | 213 +++++++++++++++++++++++++++-
> >  1 file changed, 210 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/irqchip/irq-armada-370-xp.c b/drivers/irqchip/irq-armada-370-xp.c
> > index ebd76ea1c69b..71578b65f5c8 100644
> > --- a/drivers/irqchip/irq-armada-370-xp.c
> > +++ b/drivers/irqchip/irq-armada-370-xp.c
> > @@ -117,6 +117,8 @@
> >  /* Registers relative to main_int_base */
> >  #define ARMADA_370_XP_INT_CONTROL		(0x00)
> >  #define ARMADA_370_XP_SW_TRIG_INT_OFFS		(0x04)
> > +#define ARMADA_370_XP_INT_SOC_ERR_0_CAUSE_OFFS	(0x20)
> > +#define ARMADA_370_XP_INT_SOC_ERR_1_CAUSE_OFFS	(0x24)
> >  #define ARMADA_370_XP_INT_SET_ENABLE_OFFS	(0x30)
> >  #define ARMADA_370_XP_INT_CLEAR_ENABLE_OFFS	(0x34)
> >  #define ARMADA_370_XP_INT_SOURCE_CTL(irq)	(0x100 + irq*4)
> > @@ -130,6 +132,8 @@
> >  #define ARMADA_370_XP_CPU_INTACK_OFFS		(0x44)
> >  #define ARMADA_370_XP_INT_SET_MASK_OFFS		(0x48)
> >  #define ARMADA_370_XP_INT_CLEAR_MASK_OFFS	(0x4C)
> > +#define ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF	(0x50)
> > +#define ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF	(0x54)
> >  #define ARMADA_370_XP_INT_FABRIC_MASK_OFFS	(0x54)
> >  #define ARMADA_370_XP_INT_CAUSE_PERF(cpu)	(1 << cpu)
> >  
> > @@ -146,6 +150,8 @@
> >  static void __iomem *per_cpu_int_base;
> >  static void __iomem *main_int_base;
> >  static struct irq_domain *armada_370_xp_mpic_domain;
> > +static struct irq_domain *armada_370_xp_soc_err_domain;
> > +static unsigned int soc_err_irq_num_regs;
> >  static u32 doorbell_mask_reg;
> >  static int parent_irq;
> >  #ifdef CONFIG_PCI_MSI
> > @@ -156,6 +162,8 @@ static DEFINE_MUTEX(msi_used_lock);
> >  static phys_addr_t msi_doorbell_addr;
> >  #endif
> >  
> > +static void armada_370_xp_soc_err_irq_unmask(struct irq_data *d);
> > +
> >  static inline bool is_percpu_irq(irq_hw_number_t irq)
> >  {
> >  	if (irq <= ARMADA_370_XP_MAX_PER_CPU_IRQS)
> > @@ -509,6 +517,27 @@ static void armada_xp_mpic_reenable_percpu(void)
> >  		armada_370_xp_irq_unmask(data);
> >  	}
> >  
> > +	/* Re-enable per-CPU SoC Error interrupts that were enabled before suspend */
> > +	for (irq = 0; irq < soc_err_irq_num_regs * 32; irq++) {
> > +		struct irq_data *data;
> > +		int virq;
> > +
> > +		virq = irq_linear_revmap(armada_370_xp_soc_err_domain, irq);
> > +		if (virq == 0)
> > +			continue;
> > +
> > +		data = irq_get_irq_data(virq);
> > +
> > +		if (!irq_percpu_is_enabled(virq))
> > +			continue;
> > +
> > +		armada_370_xp_soc_err_irq_unmask(data);
> > +	}
> 
> So you do this loop and all these lookups, both here and in the resume
> function (duplicated code!) just to be able to call the unmask
> function?  This would be better served by two straight writes of the
> mask register, which you'd conveniently save on suspend.
> 
> Yes, you have only duplicated the existing logic. But surely there is
> something better to do.

Yes, I just used existing logic.

I'm not rewriting driver or doing big refactor of it, as this is not in
the scope of the PCIe AER interrupt support.

> > +
> > +	/* Unmask summary SoC Error Interrupt */
> > +	if (soc_err_irq_num_regs > 0)
> > +		writel(4, per_cpu_int_base + ARMADA_370_XP_INT_CLEAR_MASK_OFFS);
> > +
> >  	ipi_resume();
> >  }
> >  
> > @@ -546,8 +575,8 @@ static struct irq_chip armada_370_xp_irq_chip = {
> >  static int armada_370_xp_mpic_irq_map(struct irq_domain *h,
> >  				      unsigned int virq, irq_hw_number_t hw)
> >  {
> > -	/* IRQs 0 and 1 cannot be mapped, they are handled internally */
> > -	if (hw <= 1)
> > +	/* IRQs 0, 1 and 4 cannot be mapped, they are handled internally */
> > +	if (hw <= 1 || hw == 4)
> >  		return -EINVAL;
> >  
> >  	armada_370_xp_irq_mask(irq_get_irq_data(virq));
> > @@ -577,6 +606,99 @@ static const struct irq_domain_ops armada_370_xp_mpic_irq_ops = {
> >  	.xlate = irq_domain_xlate_onecell,
> >  };
> >  
> > +static DEFINE_RAW_SPINLOCK(armada_370_xp_soc_err_lock);
> > +
> > +static void armada_370_xp_soc_err_irq_mask(struct irq_data *d)
> > +{
> > +	irq_hw_number_t hwirq = irqd_to_hwirq(d);
> > +	u32 reg, mask;
> > +
> > +	reg = hwirq >= 32 ? ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF
> > +			  : ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF;
> > +
> > +	raw_spin_lock(&armada_370_xp_soc_err_lock);
> > +	mask = readl(per_cpu_int_base + reg);
> > +	mask &= ~BIT(hwirq % 32);
> > +	writel(mask, per_cpu_int_base + reg);
> > +	raw_spin_unlock(&armada_370_xp_soc_err_lock);
> > +}
> > +
> > +static void armada_370_xp_soc_err_irq_unmask(struct irq_data *d)
> > +{
> > +	irq_hw_number_t hwirq = irqd_to_hwirq(d);
> > +	u32 reg, mask;
> > +
> > +	reg = hwirq >= 32 ? ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF
> > +			  : ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF;
> > +
> > +	raw_spin_lock(&armada_370_xp_soc_err_lock);
> > +	mask = readl(per_cpu_int_base + reg);
> > +	mask |= BIT(hwirq % 32);
> > +	writel(mask, per_cpu_int_base + reg);
> > +	raw_spin_unlock(&armada_370_xp_soc_err_lock);
> > +}
> > +
> > +static int armada_370_xp_soc_err_irq_mask_on_cpu(void *par)
> > +{
> > +	struct irq_data *d = par;
> > +	armada_370_xp_soc_err_irq_mask(d);
> > +	return 0;
> > +}
> > +
> > +static int armada_370_xp_soc_err_irq_unmask_on_cpu(void *par)
> > +{
> > +	struct irq_data *d = par;
> > +	armada_370_xp_soc_err_irq_unmask(d);
> > +	return 0;
> > +}
> > +
> > +static int armada_xp_soc_err_irq_set_affinity(struct irq_data *d,
> > +					      const struct cpumask *mask,
> > +					      bool force)
> > +{
> > +	unsigned int cpu;
> > +
> > +	cpus_read_lock();
> > +
> > +	/* First disable IRQ on all cores */
> > +	for_each_online_cpu(cpu)
> > +		smp_call_on_cpu(cpu, armada_370_xp_soc_err_irq_mask_on_cpu, d, true);
> > +
> > +	/* Select a single core from the affinity mask which is online */
> > +	cpu = cpumask_any_and(mask, cpu_online_mask);
> > +	smp_call_on_cpu(cpu, armada_370_xp_soc_err_irq_unmask_on_cpu, d, true);
> > +
> > +	cpus_read_unlock();
> > +
> > +	irq_data_update_effective_affinity(d, cpumask_of(cpu));
> > +
> > +	return IRQ_SET_MASK_OK;
> > +}
> 
> Aren't these per-CPU interrupts anyway? What does it mean to set their
> affinity? /me rolls eyes...

Yes, they are per-CPU interrupts. But to mask or unmask particular
interrupt for specific CPU is possible only from that CPU. CPU 0 just
cannot move interrupt from CPU 0 to CPU 1. CPU 0 can only mask that
interrupt and CPU 1 has to unmask it.

> > +
> > +static struct irq_chip armada_370_xp_soc_err_irq_chip = {
> > +	.name = "MPIC SOC",
> > +	.irq_mask = armada_370_xp_soc_err_irq_mask,
> > +	.irq_unmask = armada_370_xp_soc_err_irq_unmask,
> > +	.irq_set_affinity = armada_xp_soc_err_irq_set_affinity,
> > +};
> > +
> > +static int armada_370_xp_soc_err_irq_map(struct irq_domain *h,
> > +					 unsigned int virq, irq_hw_number_t hw)
> > +{
> > +	armada_370_xp_soc_err_irq_mask(irq_get_irq_data(virq));
> > +	irq_set_status_flags(virq, IRQ_LEVEL);
> > +	irq_set_percpu_devid(virq);
> > +	irq_set_chip_and_handler(virq, &armada_370_xp_soc_err_irq_chip,
> > +				 handle_percpu_devid_irq);
> > +	irq_set_probe(virq);
> > +	return 0;
> > +}
> > +
> > +static const struct irq_domain_ops armada_370_xp_soc_err_irq_ops = {
> > +	.map = armada_370_xp_soc_err_irq_map,
> > +	.xlate = irq_domain_xlate_onecell,
> > +};
> > +
> >  #ifdef CONFIG_PCI_MSI
> >  static void armada_370_xp_handle_msi_irq(struct pt_regs *regs, bool is_chained)
> >  {
> > @@ -605,6 +727,32 @@ static void armada_370_xp_handle_msi_irq(struct pt_regs *regs, bool is_chained)
> >  static void armada_370_xp_handle_msi_irq(struct pt_regs *r, bool b) {}
> >  #endif
> >  
> > +static void armada_370_xp_handle_soc_err_irq(void)
> > +{
> > +	unsigned long status, bit;
> > +	u32 mask, cause;
> > +
> > +	if (soc_err_irq_num_regs < 1)
> > +		return;
> > +
> > +	mask = readl(per_cpu_int_base + ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF);
> > +	cause = readl(main_int_base + ARMADA_370_XP_INT_SOC_ERR_0_CAUSE_OFFS);
> > +	status = cause & mask;
> > +
> > +	for_each_set_bit(bit, &status, 32)
> > +		generic_handle_domain_irq(armada_370_xp_soc_err_domain, bit);
> > +
> > +	if (soc_err_irq_num_regs < 2)
> > +		return;
> > +
> > +	mask = readl(per_cpu_int_base + ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF);
> > +	cause = readl(main_int_base + ARMADA_370_XP_INT_SOC_ERR_1_CAUSE_OFFS);
> > +	status = cause & mask;
> > +
> > +	for_each_set_bit(bit, &status, 32)
> > +		generic_handle_domain_irq(armada_370_xp_soc_err_domain, bit + 32);
> > +}
> > +
> >  static void armada_370_xp_mpic_handle_cascade_irq(struct irq_desc *desc)
> >  {
> >  	struct irq_chip *chip = irq_desc_get_chip(desc);
> > @@ -630,6 +778,11 @@ static void armada_370_xp_mpic_handle_cascade_irq(struct irq_desc *desc)
> >  			continue;
> >  		}
> >  
> > +		if (irqn == 4) {
> > +			armada_370_xp_handle_soc_err_irq();
> > +			continue;
> > +		}
> > +
> >  		generic_handle_domain_irq(armada_370_xp_mpic_domain, irqn);
> >  	}
> >  
> > @@ -649,7 +802,7 @@ armada_370_xp_handle_irq(struct pt_regs *regs)
> >  		if (irqnr > 1022)
> >  			break;
> >  
> > -		if (irqnr > 1) {
> > +		if (irqnr > 1 && irqnr != 4) {
> >  			generic_handle_domain_irq(armada_370_xp_mpic_domain,
> >  						  irqnr);
> >  			continue;
> > @@ -659,6 +812,10 @@ armada_370_xp_handle_irq(struct pt_regs *regs)
> >  		if (irqnr == 1)
> >  			armada_370_xp_handle_msi_irq(regs, false);
> >  
> > +		/* SoC Error handling */
> > +		if (irqnr == 4)
> > +			armada_370_xp_handle_soc_err_irq();
> > +
> >  #ifdef CONFIG_SMP
> >  		/* IPI Handling */
> >  		if (irqnr == 0) {
> > @@ -722,6 +879,26 @@ static void armada_370_xp_mpic_resume(void)
> >  		}
> >  	}
> >  
> > +	/* Re-enable per-CPU SoC Error interrupts */
> > +	for (irq = 0; irq < soc_err_irq_num_regs * 32; irq++) {
> > +		struct irq_data *data;
> > +		int virq;
> > +
> > +		virq = irq_linear_revmap(armada_370_xp_soc_err_domain, irq);
> > +		if (virq == 0)
> > +			continue;
> > +
> > +		data = irq_get_irq_data(virq);
> > +
> > +		/*
> > +		 * Re-enable on the current CPU,
> > +		 * armada_xp_mpic_reenable_percpu() will take
> > +		 * care of secondary CPUs when they come up.
> > +		 */
> > +		if (irq_percpu_is_enabled(virq))
> > +			armada_370_xp_soc_err_irq_unmask(data);
> > +	}
> 
> As I said above, this is duplicated code that should be replaced with
> a simple write to the corresponding MMIO registers.
> 
> > +
> >  	/* Reconfigure doorbells for IPIs and MSIs */
> >  	writel(doorbell_mask_reg,
> >  	       per_cpu_int_base + ARMADA_370_XP_IN_DRBEL_MSK_OFFS);
> > @@ -730,6 +907,10 @@ static void armada_370_xp_mpic_resume(void)
> >  	if (doorbell_mask_reg & PCI_MSI_DOORBELL_MASK)
> >  		writel(1, per_cpu_int_base + ARMADA_370_XP_INT_CLEAR_MASK_OFFS);
> >  
> > +	/* Unmask summary SoC Error Interrupt */
> > +	if (soc_err_irq_num_regs > 0)
> > +		writel(4, per_cpu_int_base + ARMADA_370_XP_INT_CLEAR_MASK_OFFS);
> 
> Magic value?
> 
> Also, writing to this register tends to indicate that the whole thing
> should really be a chained irqchip... Maybe that's overkill in this
> instance, but the whole thing is rather oddly architected.

I used exactly same code style like it is used for MSI doorbell
interrupts.

Yes, it looks like chained irqchip, but it is overkill.

> > +
> >  	ipi_resume();
> >  }
> >  
> > @@ -742,6 +923,7 @@ static int __init armada_370_xp_mpic_of_init(struct device_node *node,
> >  					     struct device_node *parent)
> >  {
> >  	struct resource main_int_res, per_cpu_int_res;
> > +	struct device_node *soc_err_node;
> >  	int nr_irqs, i;
> >  	u32 control;
> >  
> > @@ -775,12 +957,37 @@ static int __init armada_370_xp_mpic_of_init(struct device_node *node,
> >  	BUG_ON(!armada_370_xp_mpic_domain);
> >  	irq_domain_update_bus_token(armada_370_xp_mpic_domain, DOMAIN_BUS_WIRED);
> >  
> > +	soc_err_node = of_get_next_child(node, NULL);
> > +	if (!soc_err_node) {
> > +		pr_warn("Missing SoC Error Interrupt Controller node\n");
> > +		pr_warn("Extended interrupts are not supported\n");
> > +	} else {
> > +		pr_info("Registering MPIC SoC Error Interrupt Controller\n");
> > +		/*
> > +		 * Armada 370 and XP have only 32 SoC Error IRQs in one register
> > +		 * and other Armada platforms have 64 IRQs in two registers.
> > +		 */
> > +		soc_err_irq_num_regs =
> > +			of_machine_is_compatible("marvell,armada-370-xp") ? 1 : 2;
> 
> Don't you have an actual compatible string for the interrupt
> controller?  It seems odd to rely on the SoC name.

Compatible string is same for all those 32-bit Armada SoCs. So it cannot
be used to distinguish between XP and 385. That is why I used
of_machine_is_compatible.

> > +		armada_370_xp_soc_err_domain =
> > +			irq_domain_add_hierarchy(armada_370_xp_mpic_domain, 0,
> > +						 soc_err_irq_num_regs * 32,
> > +						 soc_err_node,
> > +						 &armada_370_xp_soc_err_irq_ops,
> > +						 NULL);
> > +		BUG_ON(!armada_370_xp_soc_err_domain);
> > +	}
> > +
> >  	/* Setup for the boot CPU */
> >  	armada_xp_mpic_perf_init();
> >  	armada_xp_mpic_smp_cpu_init();
> >  
> >  	armada_370_xp_msi_init(node, main_int_res.start);
> >  
> > +	/* Unmask summary SoC Error Interrupt */
> > +	if (soc_err_irq_num_regs > 0)
> > +		writel(4, per_cpu_int_base + ARMADA_370_XP_INT_CLEAR_MASK_OFFS);
> > +
> 
> Magic value, duplicated this time?
> 
> >  	parent_irq = irq_of_parse_and_map(node, 0);
> >  	if (parent_irq <= 0) {
> >  		irq_set_default_host(armada_370_xp_mpic_domain);
> > -- 
> > 2.20.1
> > 
> > 
> 
> 	M.
> 
> -- 
> Without deviation from the norm, progress is not possible.

Re: [PATCH 2/6] irqchip/armada-370-xp: Implement SoC Error interrupts

[Marc Zyngier]

On Fri, 06 May 2022 19:30:51 +0100,
Pali Rohár <pali@kernel.org> wrote:
> 
> On Friday 06 May 2022 19:19:46 Marc Zyngier wrote:
> > On Fri, 06 May 2022 14:40:25 +0100,
> > Pali Rohár <pali@kernel.org> wrote:
> > > 
> > > MPIC IRQ 4 is used as SoC Error Summary interrupt and provides access to
> > > another hierarchy of SoC Error interrupts. Implement a new IRQ chip and
> > > domain for accessing this IRQ hierarchy.
> > > 
> > > Signed-off-by: Pali Rohár <pali@kernel.org>
> > > ---
> > >  drivers/irqchip/irq-armada-370-xp.c | 213 +++++++++++++++++++++++++++-
> > >  1 file changed, 210 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/drivers/irqchip/irq-armada-370-xp.c b/drivers/irqchip/irq-armada-370-xp.c
> > > index ebd76ea1c69b..71578b65f5c8 100644
> > > --- a/drivers/irqchip/irq-armada-370-xp.c
> > > +++ b/drivers/irqchip/irq-armada-370-xp.c
> > > @@ -117,6 +117,8 @@
> > >  /* Registers relative to main_int_base */
> > >  #define ARMADA_370_XP_INT_CONTROL		(0x00)
> > >  #define ARMADA_370_XP_SW_TRIG_INT_OFFS		(0x04)
> > > +#define ARMADA_370_XP_INT_SOC_ERR_0_CAUSE_OFFS	(0x20)
> > > +#define ARMADA_370_XP_INT_SOC_ERR_1_CAUSE_OFFS	(0x24)
> > >  #define ARMADA_370_XP_INT_SET_ENABLE_OFFS	(0x30)
> > >  #define ARMADA_370_XP_INT_CLEAR_ENABLE_OFFS	(0x34)
> > >  #define ARMADA_370_XP_INT_SOURCE_CTL(irq)	(0x100 + irq*4)
> > > @@ -130,6 +132,8 @@
> > >  #define ARMADA_370_XP_CPU_INTACK_OFFS		(0x44)
> > >  #define ARMADA_370_XP_INT_SET_MASK_OFFS		(0x48)
> > >  #define ARMADA_370_XP_INT_CLEAR_MASK_OFFS	(0x4C)
> > > +#define ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF	(0x50)
> > > +#define ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF	(0x54)
> > >  #define ARMADA_370_XP_INT_FABRIC_MASK_OFFS	(0x54)
> > >  #define ARMADA_370_XP_INT_CAUSE_PERF(cpu)	(1 << cpu)
> > >  
> > > @@ -146,6 +150,8 @@
> > >  static void __iomem *per_cpu_int_base;
> > >  static void __iomem *main_int_base;
> > >  static struct irq_domain *armada_370_xp_mpic_domain;
> > > +static struct irq_domain *armada_370_xp_soc_err_domain;
> > > +static unsigned int soc_err_irq_num_regs;
> > >  static u32 doorbell_mask_reg;
> > >  static int parent_irq;
> > >  #ifdef CONFIG_PCI_MSI
> > > @@ -156,6 +162,8 @@ static DEFINE_MUTEX(msi_used_lock);
> > >  static phys_addr_t msi_doorbell_addr;
> > >  #endif
> > >  
> > > +static void armada_370_xp_soc_err_irq_unmask(struct irq_data *d);
> > > +
> > >  static inline bool is_percpu_irq(irq_hw_number_t irq)
> > >  {
> > >  	if (irq <= ARMADA_370_XP_MAX_PER_CPU_IRQS)
> > > @@ -509,6 +517,27 @@ static void armada_xp_mpic_reenable_percpu(void)
> > >  		armada_370_xp_irq_unmask(data);
> > >  	}
> > >  
> > > +	/* Re-enable per-CPU SoC Error interrupts that were enabled before suspend */
> > > +	for (irq = 0; irq < soc_err_irq_num_regs * 32; irq++) {
> > > +		struct irq_data *data;
> > > +		int virq;
> > > +
> > > +		virq = irq_linear_revmap(armada_370_xp_soc_err_domain, irq);
> > > +		if (virq == 0)
> > > +			continue;
> > > +
> > > +		data = irq_get_irq_data(virq);
> > > +
> > > +		if (!irq_percpu_is_enabled(virq))
> > > +			continue;
> > > +
> > > +		armada_370_xp_soc_err_irq_unmask(data);
> > > +	}
> > 
> > So you do this loop and all these lookups, both here and in the resume
> > function (duplicated code!) just to be able to call the unmask
> > function?  This would be better served by two straight writes of the
> > mask register, which you'd conveniently save on suspend.
> > 
> > Yes, you have only duplicated the existing logic. But surely there is
> > something better to do.
> 
> Yes, I just used existing logic.
> 
> I'm not rewriting driver or doing big refactor of it, as this is not in
> the scope of the PCIe AER interrupt support.

Fair enough. By the same logic, I'm not taking any change to the
driver until it is put in a better shape. Your call.

> > > +
> > > +	/* Unmask summary SoC Error Interrupt */
> > > +	if (soc_err_irq_num_regs > 0)
> > > +		writel(4, per_cpu_int_base + ARMADA_370_XP_INT_CLEAR_MASK_OFFS);
> > > +
> > >  	ipi_resume();
> > >  }
> > >  
> > > @@ -546,8 +575,8 @@ static struct irq_chip armada_370_xp_irq_chip = {
> > >  static int armada_370_xp_mpic_irq_map(struct irq_domain *h,
> > >  				      unsigned int virq, irq_hw_number_t hw)
> > >  {
> > > -	/* IRQs 0 and 1 cannot be mapped, they are handled internally */
> > > -	if (hw <= 1)
> > > +	/* IRQs 0, 1 and 4 cannot be mapped, they are handled internally */
> > > +	if (hw <= 1 || hw == 4)
> > >  		return -EINVAL;
> > >  
> > >  	armada_370_xp_irq_mask(irq_get_irq_data(virq));
> > > @@ -577,6 +606,99 @@ static const struct irq_domain_ops armada_370_xp_mpic_irq_ops = {
> > >  	.xlate = irq_domain_xlate_onecell,
> > >  };
> > >  
> > > +static DEFINE_RAW_SPINLOCK(armada_370_xp_soc_err_lock);
> > > +
> > > +static void armada_370_xp_soc_err_irq_mask(struct irq_data *d)
> > > +{
> > > +	irq_hw_number_t hwirq = irqd_to_hwirq(d);
> > > +	u32 reg, mask;
> > > +
> > > +	reg = hwirq >= 32 ? ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF
> > > +			  : ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF;
> > > +
> > > +	raw_spin_lock(&armada_370_xp_soc_err_lock);
> > > +	mask = readl(per_cpu_int_base + reg);
> > > +	mask &= ~BIT(hwirq % 32);
> > > +	writel(mask, per_cpu_int_base + reg);
> > > +	raw_spin_unlock(&armada_370_xp_soc_err_lock);
> > > +}
> > > +
> > > +static void armada_370_xp_soc_err_irq_unmask(struct irq_data *d)
> > > +{
> > > +	irq_hw_number_t hwirq = irqd_to_hwirq(d);
> > > +	u32 reg, mask;
> > > +
> > > +	reg = hwirq >= 32 ? ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF
> > > +			  : ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF;
> > > +
> > > +	raw_spin_lock(&armada_370_xp_soc_err_lock);
> > > +	mask = readl(per_cpu_int_base + reg);
> > > +	mask |= BIT(hwirq % 32);
> > > +	writel(mask, per_cpu_int_base + reg);
> > > +	raw_spin_unlock(&armada_370_xp_soc_err_lock);
> > > +}
> > > +
> > > +static int armada_370_xp_soc_err_irq_mask_on_cpu(void *par)
> > > +{
> > > +	struct irq_data *d = par;
> > > +	armada_370_xp_soc_err_irq_mask(d);
> > > +	return 0;
> > > +}
> > > +
> > > +static int armada_370_xp_soc_err_irq_unmask_on_cpu(void *par)
> > > +{
> > > +	struct irq_data *d = par;
> > > +	armada_370_xp_soc_err_irq_unmask(d);
> > > +	return 0;
> > > +}
> > > +
> > > +static int armada_xp_soc_err_irq_set_affinity(struct irq_data *d,
> > > +					      const struct cpumask *mask,
> > > +					      bool force)
> > > +{
> > > +	unsigned int cpu;
> > > +
> > > +	cpus_read_lock();
> > > +
> > > +	/* First disable IRQ on all cores */
> > > +	for_each_online_cpu(cpu)
> > > +		smp_call_on_cpu(cpu, armada_370_xp_soc_err_irq_mask_on_cpu, d, true);
> > > +
> > > +	/* Select a single core from the affinity mask which is online */
> > > +	cpu = cpumask_any_and(mask, cpu_online_mask);
> > > +	smp_call_on_cpu(cpu, armada_370_xp_soc_err_irq_unmask_on_cpu, d, true);
> > > +
> > > +	cpus_read_unlock();
> > > +
> > > +	irq_data_update_effective_affinity(d, cpumask_of(cpu));
> > > +
> > > +	return IRQ_SET_MASK_OK;
> > > +}
> > 
> > Aren't these per-CPU interrupts anyway? What does it mean to set their
> > affinity? /me rolls eyes...
> 
> Yes, they are per-CPU interrupts. But to mask or unmask particular
> interrupt for specific CPU is possible only from that CPU. CPU 0 just
> cannot move interrupt from CPU 0 to CPU 1. CPU 0 can only mask that
> interrupt and CPU 1 has to unmask it.

And that's no different form other per-CPU interrupts that have the
exact same requirements. NAK to this sort of hacks.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 2/6] irqchip/armada-370-xp: Implement SoC Error interrupts

[Pali Rohár]

On Friday 06 May 2022 19:47:25 Marc Zyngier wrote:
> On Fri, 06 May 2022 19:30:51 +0100,
> Pali Rohár <pali@kernel.org> wrote:
> > 
> > On Friday 06 May 2022 19:19:46 Marc Zyngier wrote:
> > > On Fri, 06 May 2022 14:40:25 +0100,
> > > Pali Rohár <pali@kernel.org> wrote:
> > > > 
> > > > MPIC IRQ 4 is used as SoC Error Summary interrupt and provides access to
> > > > another hierarchy of SoC Error interrupts. Implement a new IRQ chip and
> > > > domain for accessing this IRQ hierarchy.
> > > > 
> > > > Signed-off-by: Pali Rohár <pali@kernel.org>
> > > > ---
> > > >  drivers/irqchip/irq-armada-370-xp.c | 213 +++++++++++++++++++++++++++-
> > > >  1 file changed, 210 insertions(+), 3 deletions(-)
> > > > 
> > > > diff --git a/drivers/irqchip/irq-armada-370-xp.c b/drivers/irqchip/irq-armada-370-xp.c
> > > > index ebd76ea1c69b..71578b65f5c8 100644
> > > > --- a/drivers/irqchip/irq-armada-370-xp.c
> > > > +++ b/drivers/irqchip/irq-armada-370-xp.c
> > > > @@ -117,6 +117,8 @@
> > > >  /* Registers relative to main_int_base */
> > > >  #define ARMADA_370_XP_INT_CONTROL		(0x00)
> > > >  #define ARMADA_370_XP_SW_TRIG_INT_OFFS		(0x04)
> > > > +#define ARMADA_370_XP_INT_SOC_ERR_0_CAUSE_OFFS	(0x20)
> > > > +#define ARMADA_370_XP_INT_SOC_ERR_1_CAUSE_OFFS	(0x24)
> > > >  #define ARMADA_370_XP_INT_SET_ENABLE_OFFS	(0x30)
> > > >  #define ARMADA_370_XP_INT_CLEAR_ENABLE_OFFS	(0x34)
> > > >  #define ARMADA_370_XP_INT_SOURCE_CTL(irq)	(0x100 + irq*4)
> > > > @@ -130,6 +132,8 @@
> > > >  #define ARMADA_370_XP_CPU_INTACK_OFFS		(0x44)
> > > >  #define ARMADA_370_XP_INT_SET_MASK_OFFS		(0x48)
> > > >  #define ARMADA_370_XP_INT_CLEAR_MASK_OFFS	(0x4C)
> > > > +#define ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF	(0x50)
> > > > +#define ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF	(0x54)
> > > >  #define ARMADA_370_XP_INT_FABRIC_MASK_OFFS	(0x54)
> > > >  #define ARMADA_370_XP_INT_CAUSE_PERF(cpu)	(1 << cpu)
> > > >  
> > > > @@ -146,6 +150,8 @@
> > > >  static void __iomem *per_cpu_int_base;
> > > >  static void __iomem *main_int_base;
> > > >  static struct irq_domain *armada_370_xp_mpic_domain;
> > > > +static struct irq_domain *armada_370_xp_soc_err_domain;
> > > > +static unsigned int soc_err_irq_num_regs;
> > > >  static u32 doorbell_mask_reg;
> > > >  static int parent_irq;
> > > >  #ifdef CONFIG_PCI_MSI
> > > > @@ -156,6 +162,8 @@ static DEFINE_MUTEX(msi_used_lock);
> > > >  static phys_addr_t msi_doorbell_addr;
> > > >  #endif
> > > >  
> > > > +static void armada_370_xp_soc_err_irq_unmask(struct irq_data *d);
> > > > +
> > > >  static inline bool is_percpu_irq(irq_hw_number_t irq)
> > > >  {
> > > >  	if (irq <= ARMADA_370_XP_MAX_PER_CPU_IRQS)
> > > > @@ -509,6 +517,27 @@ static void armada_xp_mpic_reenable_percpu(void)
> > > >  		armada_370_xp_irq_unmask(data);
> > > >  	}
> > > >  
> > > > +	/* Re-enable per-CPU SoC Error interrupts that were enabled before suspend */
> > > > +	for (irq = 0; irq < soc_err_irq_num_regs * 32; irq++) {
> > > > +		struct irq_data *data;
> > > > +		int virq;
> > > > +
> > > > +		virq = irq_linear_revmap(armada_370_xp_soc_err_domain, irq);
> > > > +		if (virq == 0)
> > > > +			continue;
> > > > +
> > > > +		data = irq_get_irq_data(virq);
> > > > +
> > > > +		if (!irq_percpu_is_enabled(virq))
> > > > +			continue;
> > > > +
> > > > +		armada_370_xp_soc_err_irq_unmask(data);
> > > > +	}
> > > 
> > > So you do this loop and all these lookups, both here and in the resume
> > > function (duplicated code!) just to be able to call the unmask
> > > function?  This would be better served by two straight writes of the
> > > mask register, which you'd conveniently save on suspend.
> > > 
> > > Yes, you have only duplicated the existing logic. But surely there is
> > > something better to do.
> > 
> > Yes, I just used existing logic.
> > 
> > I'm not rewriting driver or doing big refactor of it, as this is not in
> > the scope of the PCIe AER interrupt support.
> 
> Fair enough. By the same logic, I'm not taking any change to the
> driver until it is put in a better shape. Your call.

If you are maintainer of this code then it is expected from _you_ to
move the current code into _better shape_ as you wrote and expect. And
then show us exactly, how new changes in this driver should look like,
in examples.

> > > > +
> > > > +	/* Unmask summary SoC Error Interrupt */
> > > > +	if (soc_err_irq_num_regs > 0)
> > > > +		writel(4, per_cpu_int_base + ARMADA_370_XP_INT_CLEAR_MASK_OFFS);
> > > > +
> > > >  	ipi_resume();
> > > >  }
> > > >  
> > > > @@ -546,8 +575,8 @@ static struct irq_chip armada_370_xp_irq_chip = {
> > > >  static int armada_370_xp_mpic_irq_map(struct irq_domain *h,
> > > >  				      unsigned int virq, irq_hw_number_t hw)
> > > >  {
> > > > -	/* IRQs 0 and 1 cannot be mapped, they are handled internally */
> > > > -	if (hw <= 1)
> > > > +	/* IRQs 0, 1 and 4 cannot be mapped, they are handled internally */
> > > > +	if (hw <= 1 || hw == 4)
> > > >  		return -EINVAL;
> > > >  
> > > >  	armada_370_xp_irq_mask(irq_get_irq_data(virq));
> > > > @@ -577,6 +606,99 @@ static const struct irq_domain_ops armada_370_xp_mpic_irq_ops = {
> > > >  	.xlate = irq_domain_xlate_onecell,
> > > >  };
> > > >  
> > > > +static DEFINE_RAW_SPINLOCK(armada_370_xp_soc_err_lock);
> > > > +
> > > > +static void armada_370_xp_soc_err_irq_mask(struct irq_data *d)
> > > > +{
> > > > +	irq_hw_number_t hwirq = irqd_to_hwirq(d);
> > > > +	u32 reg, mask;
> > > > +
> > > > +	reg = hwirq >= 32 ? ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF
> > > > +			  : ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF;
> > > > +
> > > > +	raw_spin_lock(&armada_370_xp_soc_err_lock);
> > > > +	mask = readl(per_cpu_int_base + reg);
> > > > +	mask &= ~BIT(hwirq % 32);
> > > > +	writel(mask, per_cpu_int_base + reg);
> > > > +	raw_spin_unlock(&armada_370_xp_soc_err_lock);
> > > > +}
> > > > +
> > > > +static void armada_370_xp_soc_err_irq_unmask(struct irq_data *d)
> > > > +{
> > > > +	irq_hw_number_t hwirq = irqd_to_hwirq(d);
> > > > +	u32 reg, mask;
> > > > +
> > > > +	reg = hwirq >= 32 ? ARMADA_370_XP_INT_SOC_ERR_1_MASK_OFF
> > > > +			  : ARMADA_370_XP_INT_SOC_ERR_0_MASK_OFF;
> > > > +
> > > > +	raw_spin_lock(&armada_370_xp_soc_err_lock);
> > > > +	mask = readl(per_cpu_int_base + reg);
> > > > +	mask |= BIT(hwirq % 32);
> > > > +	writel(mask, per_cpu_int_base + reg);
> > > > +	raw_spin_unlock(&armada_370_xp_soc_err_lock);
> > > > +}
> > > > +
> > > > +static int armada_370_xp_soc_err_irq_mask_on_cpu(void *par)
> > > > +{
> > > > +	struct irq_data *d = par;
> > > > +	armada_370_xp_soc_err_irq_mask(d);
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static int armada_370_xp_soc_err_irq_unmask_on_cpu(void *par)
> > > > +{
> > > > +	struct irq_data *d = par;
> > > > +	armada_370_xp_soc_err_irq_unmask(d);
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static int armada_xp_soc_err_irq_set_affinity(struct irq_data *d,
> > > > +					      const struct cpumask *mask,
> > > > +					      bool force)
> > > > +{
> > > > +	unsigned int cpu;
> > > > +
> > > > +	cpus_read_lock();
> > > > +
> > > > +	/* First disable IRQ on all cores */
> > > > +	for_each_online_cpu(cpu)
> > > > +		smp_call_on_cpu(cpu, armada_370_xp_soc_err_irq_mask_on_cpu, d, true);
> > > > +
> > > > +	/* Select a single core from the affinity mask which is online */
> > > > +	cpu = cpumask_any_and(mask, cpu_online_mask);
> > > > +	smp_call_on_cpu(cpu, armada_370_xp_soc_err_irq_unmask_on_cpu, d, true);
> > > > +
> > > > +	cpus_read_unlock();
> > > > +
> > > > +	irq_data_update_effective_affinity(d, cpumask_of(cpu));
> > > > +
> > > > +	return IRQ_SET_MASK_OK;
> > > > +}
> > > 
> > > Aren't these per-CPU interrupts anyway? What does it mean to set their
> > > affinity? /me rolls eyes...
> > 
> > Yes, they are per-CPU interrupts. But to mask or unmask particular
> > interrupt for specific CPU is possible only from that CPU. CPU 0 just
> > cannot move interrupt from CPU 0 to CPU 1. CPU 0 can only mask that
> > interrupt and CPU 1 has to unmask it.
> 
> And that's no different form other per-CPU interrupts that have the
> exact same requirements. NAK to this sort of hacks.

You forgot to mention in your previous email how to do it, right? So we
are waiting...

Re: [PATCH 2/6] irqchip/armada-370-xp: Implement SoC Error interrupts

[Marc Zyngier]

On Fri, 06 May 2022 19:55:46 +0100,
Pali Rohár <pali@kernel.org> wrote:
> 
> On Friday 06 May 2022 19:47:25 Marc Zyngier wrote:
> > On Fri, 06 May 2022 19:30:51 +0100,
> > Pali Rohár <pali@kernel.org> wrote:
> > > 
> > > On Friday 06 May 2022 19:19:46 Marc Zyngier wrote:
> > > > On Fri, 06 May 2022 14:40:25 +0100,
> > > > Pali Rohár <pali@kernel.org> wrote:
> > > > > 
> > > > > +static void armada_370_xp_soc_err_irq_unmask(struct irq_data *d);
> > > > > +
> > > > >  static inline bool is_percpu_irq(irq_hw_number_t irq)
> > > > >  {
> > > > >  	if (irq <= ARMADA_370_XP_MAX_PER_CPU_IRQS)
> > > > > @@ -509,6 +517,27 @@ static void armada_xp_mpic_reenable_percpu(void)
> > > > >  		armada_370_xp_irq_unmask(data);
> > > > >  	}
> > > > >  
> > > > > +	/* Re-enable per-CPU SoC Error interrupts that were enabled before suspend */
> > > > > +	for (irq = 0; irq < soc_err_irq_num_regs * 32; irq++) {
> > > > > +		struct irq_data *data;
> > > > > +		int virq;
> > > > > +
> > > > > +		virq = irq_linear_revmap(armada_370_xp_soc_err_domain, irq);
> > > > > +		if (virq == 0)
> > > > > +			continue;
> > > > > +
> > > > > +		data = irq_get_irq_data(virq);
> > > > > +
> > > > > +		if (!irq_percpu_is_enabled(virq))
> > > > > +			continue;
> > > > > +
> > > > > +		armada_370_xp_soc_err_irq_unmask(data);
> > > > > +	}
> > > > 
> > > > So you do this loop and all these lookups, both here and in the resume
> > > > function (duplicated code!) just to be able to call the unmask
> > > > function?  This would be better served by two straight writes of the
> > > > mask register, which you'd conveniently save on suspend.
> > > > 
> > > > Yes, you have only duplicated the existing logic. But surely there is
> > > > something better to do.
> > > 
> > > Yes, I just used existing logic.
> > > 
> > > I'm not rewriting driver or doing big refactor of it, as this is not in
> > > the scope of the PCIe AER interrupt support.
> > 
> > Fair enough. By the same logic, I'm not taking any change to the
> > driver until it is put in a better shape. Your call.
> 
> If you are maintainer of this code then it is expected from _you_ to
> move the current code into _better shape_ as you wrote and expect. And
> then show us exactly, how new changes in this driver should look like,
> in examples.

Sorry, but that's not how this works. You are the one willing to
change a sub-par piece of code, you get to make it better. You
obviously have the means (the HW) and the incentive (these patches).
But you don't get to make something even more unmaintainable because
you're unwilling to do some extra work.

If you're unhappy with my position, that's fine. I suggest you take it
with Thomas, and maybe even Linus. As I suggested before, you can also
post a patch removing me as the irqchip maintainer. I'm sure that will
spark an interesting discussion.

> > > > > +static int armada_xp_soc_err_irq_set_affinity(struct irq_data *d,
> > > > > +					      const struct cpumask *mask,
> > > > > +					      bool force)
> > > > > +{
> > > > > +	unsigned int cpu;
> > > > > +
> > > > > +	cpus_read_lock();
> > > > > +
> > > > > +	/* First disable IRQ on all cores */
> > > > > +	for_each_online_cpu(cpu)
> > > > > +		smp_call_on_cpu(cpu, armada_370_xp_soc_err_irq_mask_on_cpu, d, true);
> > > > > +
> > > > > +	/* Select a single core from the affinity mask which is online */
> > > > > +	cpu = cpumask_any_and(mask, cpu_online_mask);
> > > > > +	smp_call_on_cpu(cpu, armada_370_xp_soc_err_irq_unmask_on_cpu, d, true);
> > > > > +
> > > > > +	cpus_read_unlock();
> > > > > +
> > > > > +	irq_data_update_effective_affinity(d, cpumask_of(cpu));
> > > > > +
> > > > > +	return IRQ_SET_MASK_OK;
> > > > > +}
> > > > 
> > > > Aren't these per-CPU interrupts anyway? What does it mean to set their
> > > > affinity? /me rolls eyes...
> > > 
> > > Yes, they are per-CPU interrupts. But to mask or unmask particular
> > > interrupt for specific CPU is possible only from that CPU. CPU 0 just
> > > cannot move interrupt from CPU 0 to CPU 1. CPU 0 can only mask that
> > > interrupt and CPU 1 has to unmask it.
> > 
> > And that's no different form other per-CPU interrupts that have the
> > exact same requirements. NAK to this sort of hacks.
> 
> You forgot to mention in your previous email how to do it, right? So we
> are waiting...

I didn't forget. I explained that it should be handled just like any
other per-CPU interrupt. There is plenty of example of how to do that
in the tree (timers, for example), and if you had even looked at it,
you'd have seen that your approach most probably results in an
arbitrary pointer dereference on anything but CPU0 because the
requesting driver knows nothing about per-CPU interrupts.

But you're obviously trying to make a very different point here. I'll
let you play that game for as long as you want, no skin off my nose.
Maybe in the future, you'll be more interested in actively
collaborating on the kernel code instead of throwing your toys out of
the pram.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 2/6] irqchip/armada-370-xp: Implement SoC Error interrupts

[Pali Rohár]

On Saturday 07 May 2022 10:01:52 Marc Zyngier wrote:
> On Fri, 06 May 2022 19:55:46 +0100,
> Pali Rohár <pali@kernel.org> wrote:
> > 
> > On Friday 06 May 2022 19:47:25 Marc Zyngier wrote:
> > > On Fri, 06 May 2022 19:30:51 +0100,
> > > Pali Rohár <pali@kernel.org> wrote:
> > > > 
> > > > On Friday 06 May 2022 19:19:46 Marc Zyngier wrote:
> > > > > On Fri, 06 May 2022 14:40:25 +0100,
> > > > > Pali Rohár <pali@kernel.org> wrote:
> > > > > > 
> > > > > > +static void armada_370_xp_soc_err_irq_unmask(struct irq_data *d);
> > > > > > +
> > > > > >  static inline bool is_percpu_irq(irq_hw_number_t irq)
> > > > > >  {
> > > > > >  	if (irq <= ARMADA_370_XP_MAX_PER_CPU_IRQS)
> > > > > > @@ -509,6 +517,27 @@ static void armada_xp_mpic_reenable_percpu(void)
> > > > > >  		armada_370_xp_irq_unmask(data);
> > > > > >  	}
> > > > > >  
> > > > > > +	/* Re-enable per-CPU SoC Error interrupts that were enabled before suspend */
> > > > > > +	for (irq = 0; irq < soc_err_irq_num_regs * 32; irq++) {
> > > > > > +		struct irq_data *data;
> > > > > > +		int virq;
> > > > > > +
> > > > > > +		virq = irq_linear_revmap(armada_370_xp_soc_err_domain, irq);
> > > > > > +		if (virq == 0)
> > > > > > +			continue;
> > > > > > +
> > > > > > +		data = irq_get_irq_data(virq);
> > > > > > +
> > > > > > +		if (!irq_percpu_is_enabled(virq))
> > > > > > +			continue;
> > > > > > +
> > > > > > +		armada_370_xp_soc_err_irq_unmask(data);
> > > > > > +	}
> > > > > 
> > > > > So you do this loop and all these lookups, both here and in the resume
> > > > > function (duplicated code!) just to be able to call the unmask
> > > > > function?  This would be better served by two straight writes of the
> > > > > mask register, which you'd conveniently save on suspend.
> > > > > 
> > > > > Yes, you have only duplicated the existing logic. But surely there is
> > > > > something better to do.
> > > > 
> > > > Yes, I just used existing logic.
> > > > 
> > > > I'm not rewriting driver or doing big refactor of it, as this is not in
> > > > the scope of the PCIe AER interrupt support.
> > > 
> > > Fair enough. By the same logic, I'm not taking any change to the
> > > driver until it is put in a better shape. Your call.
> > 
> > If you are maintainer of this code then it is expected from _you_ to
> > move the current code into _better shape_ as you wrote and expect. And
> > then show us exactly, how new changes in this driver should look like,
> > in examples.
> 
> Sorry, but that's not how this works. You are the one willing to
> change a sub-par piece of code, you get to make it better. You
> obviously have the means (the HW) and the incentive (these patches).
> But you don't get to make something even more unmaintainable because
> you're unwilling to do some extra work.
> 
> If you're unhappy with my position, that's fine. I suggest you take it
> with Thomas, and maybe even Linus. As I suggested before, you can also
> post a patch removing me as the irqchip maintainer. I'm sure that will
> spark an interesting discussion.

You have already suggested it in email [1] but apparently you are _not_
maintainer of mvebu pci controller. get_maintainer.pl for part about
which you have talked in [1] says:

$ ./scripts/get_maintainer.pl -f drivers/pci/controller/pci-aardvark.c
Thomas Petazzoni <thomas.petazzoni@bootlin.com> (maintainer:PCI DRIVER FOR AARDVARK (Marvell Armada 3700))
"Pali Rohár" <pali@kernel.org> (maintainer:PCI DRIVER FOR AARDVARK (Marvell Armada 3700))
Lorenzo Pieralisi <lorenzo.pieralisi@arm.com> (supporter:PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS)
Rob Herring <robh@kernel.org> (reviewer:PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS)
"Krzysztof Wilczyński" <kw@linux.com> (reviewer:PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS)
Bjorn Helgaas <bhelgaas@google.com> (supporter:PCI SUBSYSTEM)
linux-pci@vger.kernel.org (open list:PCI DRIVER FOR AARDVARK (Marvell Armada 3700))
linux-arm-kernel@lists.infradead.org (moderated list:PCI DRIVER FOR AARDVARK (Marvell Armada 3700))
linux-kernel@vger.kernel.org (open list)

So I do not have to remove anything, you are _not_ on that list.
On the other hand, Thomas Petazzoni is on this list...

> > > > > > +static int armada_xp_soc_err_irq_set_affinity(struct irq_data *d,
> > > > > > +					      const struct cpumask *mask,
> > > > > > +					      bool force)
> > > > > > +{
> > > > > > +	unsigned int cpu;
> > > > > > +
> > > > > > +	cpus_read_lock();
> > > > > > +
> > > > > > +	/* First disable IRQ on all cores */
> > > > > > +	for_each_online_cpu(cpu)
> > > > > > +		smp_call_on_cpu(cpu, armada_370_xp_soc_err_irq_mask_on_cpu, d, true);
> > > > > > +
> > > > > > +	/* Select a single core from the affinity mask which is online */
> > > > > > +	cpu = cpumask_any_and(mask, cpu_online_mask);
> > > > > > +	smp_call_on_cpu(cpu, armada_370_xp_soc_err_irq_unmask_on_cpu, d, true);
> > > > > > +
> > > > > > +	cpus_read_unlock();
> > > > > > +
> > > > > > +	irq_data_update_effective_affinity(d, cpumask_of(cpu));
> > > > > > +
> > > > > > +	return IRQ_SET_MASK_OK;
> > > > > > +}
> > > > > 
> > > > > Aren't these per-CPU interrupts anyway? What does it mean to set their
> > > > > affinity? /me rolls eyes...
> > > > 
> > > > Yes, they are per-CPU interrupts. But to mask or unmask particular
> > > > interrupt for specific CPU is possible only from that CPU. CPU 0 just
> > > > cannot move interrupt from CPU 0 to CPU 1. CPU 0 can only mask that
> > > > interrupt and CPU 1 has to unmask it.
> > > 
> > > And that's no different form other per-CPU interrupts that have the
> > > exact same requirements. NAK to this sort of hacks.
> > 
> > You forgot to mention in your previous email how to do it, right? So we
> > are waiting...
> 
> I didn't forget. I explained that it should be handled just like any
> other per-CPU interrupt. There is plenty of example of how to do that
> in the tree (timers, for example), and if you had even looked at it,
> you'd have seen that your approach most probably results in an
> arbitrary pointer dereference on anything but CPU0 because the
> requesting driver knows nothing about per-CPU interrupts.
> 
> But you're obviously trying to make a very different point here. I'll
> let you play that game for as long as you want, no skin off my nose.
> Maybe in the future, you'll be more interested in actively
> collaborating on the kernel code instead of throwing your toys out of
> the pram.
> 
> Thanks,

The only _toy_ here is your broken mvebu board which your ego was unable
to fix, and you have put it into recycling pile [2] and since than for
months you are trying to reject every change or improvement in mvebu
drivers and trying to find out a way how to remove all mvebu code, like
if you were not able to fix your toy, then broke it also to all other
people. You have already expressed this, but I'm not going to search
emails more and find these your statements.

Sorry, I'm stopping here. This is just a prove that you are not
qualified in reviewing mvebu code.

[1] - https://lore.kernel.org/linux-pci/87mtk3tzum.wl-maz@kernel.org/
[2] - https://lore.kernel.org/linux-pci/87pmx1zjjt.wl-maz@kernel.org/

Re: [PATCH 2/6] irqchip/armada-370-xp: Implement SoC Error interrupts

[Marc Zyngier]

On Sat, 07 May 2022 10:20:54 +0100,
Pali Rohár <pali@kernel.org> wrote:
> 
> On Saturday 07 May 2022 10:01:52 Marc Zyngier wrote:
> > On Fri, 06 May 2022 19:55:46 +0100,
> > Pali Rohár <pali@kernel.org> wrote:
> > > 
> > > On Friday 06 May 2022 19:47:25 Marc Zyngier wrote:
> > > > On Fri, 06 May 2022 19:30:51 +0100,
> > > > Pali Rohár <pali@kernel.org> wrote:
> > > > > 
> > > > > On Friday 06 May 2022 19:19:46 Marc Zyngier wrote:
> > > > > > On Fri, 06 May 2022 14:40:25 +0100,
> > > > > > Pali Rohár <pali@kernel.org> wrote:
> > > > > > > 
> > > > > > > +static void armada_370_xp_soc_err_irq_unmask(struct irq_data *d);
> > > > > > > +
> > > > > > >  static inline bool is_percpu_irq(irq_hw_number_t irq)
> > > > > > >  {
> > > > > > >  	if (irq <= ARMADA_370_XP_MAX_PER_CPU_IRQS)
> > > > > > > @@ -509,6 +517,27 @@ static void armada_xp_mpic_reenable_percpu(void)
> > > > > > >  		armada_370_xp_irq_unmask(data);
> > > > > > >  	}
> > > > > > >  
> > > > > > > +	/* Re-enable per-CPU SoC Error interrupts that were enabled before suspend */
> > > > > > > +	for (irq = 0; irq < soc_err_irq_num_regs * 32; irq++) {
> > > > > > > +		struct irq_data *data;
> > > > > > > +		int virq;
> > > > > > > +
> > > > > > > +		virq = irq_linear_revmap(armada_370_xp_soc_err_domain, irq);
> > > > > > > +		if (virq == 0)
> > > > > > > +			continue;
> > > > > > > +
> > > > > > > +		data = irq_get_irq_data(virq);
> > > > > > > +
> > > > > > > +		if (!irq_percpu_is_enabled(virq))
> > > > > > > +			continue;
> > > > > > > +
> > > > > > > +		armada_370_xp_soc_err_irq_unmask(data);
> > > > > > > +	}
> > > > > > 
> > > > > > So you do this loop and all these lookups, both here and in the resume
> > > > > > function (duplicated code!) just to be able to call the unmask
> > > > > > function?  This would be better served by two straight writes of the
> > > > > > mask register, which you'd conveniently save on suspend.
> > > > > > 
> > > > > > Yes, you have only duplicated the existing logic. But surely there is
> > > > > > something better to do.
> > > > > 
> > > > > Yes, I just used existing logic.
> > > > > 
> > > > > I'm not rewriting driver or doing big refactor of it, as this is not in
> > > > > the scope of the PCIe AER interrupt support.
> > > > 
> > > > Fair enough. By the same logic, I'm not taking any change to the
> > > > driver until it is put in a better shape. Your call.
> > > 
> > > If you are maintainer of this code then it is expected from _you_ to
> > > move the current code into _better shape_ as you wrote and expect. And
> > > then show us exactly, how new changes in this driver should look like,
> > > in examples.
> > 
> > Sorry, but that's not how this works. You are the one willing to
> > change a sub-par piece of code, you get to make it better. You
> > obviously have the means (the HW) and the incentive (these patches).
> > But you don't get to make something even more unmaintainable because
> > you're unwilling to do some extra work.
> > 
> > If you're unhappy with my position, that's fine. I suggest you take it
> > with Thomas, and maybe even Linus. As I suggested before, you can also
> > post a patch removing me as the irqchip maintainer. I'm sure that will
> > spark an interesting discussion.
> 
> You have already suggested it in email [1] but apparently you are _not_
> maintainer of mvebu pci controller. get_maintainer.pl for part about
> which you have talked in [1] says:
> 
> $ ./scripts/get_maintainer.pl -f drivers/pci/controller/pci-aardvark.c

Remind me which file this patch is touching?

> The only _toy_ here is your broken mvebu board which your ego was unable
> to fix, and you have put it into recycling pile [2] and since than for
> months you are trying to reject every change or improvement in mvebu
> drivers and trying to find out a way how to remove all mvebu code, like
> if you were not able to fix your toy, then broke it also to all other
> people. You have already expressed this, but I'm not going to search
> emails more and find these your statements.

At this stage, this is pure paranoia. Do you think I am so emotionally
attached to HW purity that I would plot the annihilation of some ugly
platform?

> Sorry, I'm stopping here. This is just a prove that you are not
> qualified in reviewing mvebu code.

Happy not to have to review this code. Just stop Cc'ing me on your
patches, and don't expect me to merge any IRQ related patches coming
from you.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 2/6] irqchip/armada-370-xp: Implement SoC Error interrupts

[Pali Rohár]

On Saturday 07 May 2022 10:42:49 Marc Zyngier wrote:
> On Sat, 07 May 2022 10:20:54 +0100,
> Pali Rohár <pali@kernel.org> wrote:
> > 
> > On Saturday 07 May 2022 10:01:52 Marc Zyngier wrote:
> > > On Fri, 06 May 2022 19:55:46 +0100,
> > > Pali Rohár <pali@kernel.org> wrote:
> > > > 
> > > > On Friday 06 May 2022 19:47:25 Marc Zyngier wrote:
> > > > > On Fri, 06 May 2022 19:30:51 +0100,
> > > > > Pali Rohár <pali@kernel.org> wrote:
> > > > > > 
> > > > > > On Friday 06 May 2022 19:19:46 Marc Zyngier wrote:
> > > > > > > On Fri, 06 May 2022 14:40:25 +0100,
> > > > > > > Pali Rohár <pali@kernel.org> wrote:
> > > > > > > > 
> > > > > > > > +static void armada_370_xp_soc_err_irq_unmask(struct irq_data *d);
> > > > > > > > +
> > > > > > > >  static inline bool is_percpu_irq(irq_hw_number_t irq)
> > > > > > > >  {
> > > > > > > >  	if (irq <= ARMADA_370_XP_MAX_PER_CPU_IRQS)
> > > > > > > > @@ -509,6 +517,27 @@ static void armada_xp_mpic_reenable_percpu(void)
> > > > > > > >  		armada_370_xp_irq_unmask(data);
> > > > > > > >  	}
> > > > > > > >  
> > > > > > > > +	/* Re-enable per-CPU SoC Error interrupts that were enabled before suspend */
> > > > > > > > +	for (irq = 0; irq < soc_err_irq_num_regs * 32; irq++) {
> > > > > > > > +		struct irq_data *data;
> > > > > > > > +		int virq;
> > > > > > > > +
> > > > > > > > +		virq = irq_linear_revmap(armada_370_xp_soc_err_domain, irq);
> > > > > > > > +		if (virq == 0)
> > > > > > > > +			continue;
> > > > > > > > +
> > > > > > > > +		data = irq_get_irq_data(virq);
> > > > > > > > +
> > > > > > > > +		if (!irq_percpu_is_enabled(virq))
> > > > > > > > +			continue;
> > > > > > > > +
> > > > > > > > +		armada_370_xp_soc_err_irq_unmask(data);
> > > > > > > > +	}
> > > > > > > 
> > > > > > > So you do this loop and all these lookups, both here and in the resume
> > > > > > > function (duplicated code!) just to be able to call the unmask
> > > > > > > function?  This would be better served by two straight writes of the
> > > > > > > mask register, which you'd conveniently save on suspend.
> > > > > > > 
> > > > > > > Yes, you have only duplicated the existing logic. But surely there is
> > > > > > > something better to do.
> > > > > > 
> > > > > > Yes, I just used existing logic.
> > > > > > 
> > > > > > I'm not rewriting driver or doing big refactor of it, as this is not in
> > > > > > the scope of the PCIe AER interrupt support.
> > > > > 
> > > > > Fair enough. By the same logic, I'm not taking any change to the
> > > > > driver until it is put in a better shape. Your call.
> > > > 
> > > > If you are maintainer of this code then it is expected from _you_ to
> > > > move the current code into _better shape_ as you wrote and expect. And
> > > > then show us exactly, how new changes in this driver should look like,
> > > > in examples.
> > > 
> > > Sorry, but that's not how this works. You are the one willing to
> > > change a sub-par piece of code, you get to make it better. You
> > > obviously have the means (the HW) and the incentive (these patches).
> > > But you don't get to make something even more unmaintainable because
> > > you're unwilling to do some extra work.
> > > 
> > > If you're unhappy with my position, that's fine. I suggest you take it
> > > with Thomas, and maybe even Linus. As I suggested before, you can also
> > > post a patch removing me as the irqchip maintainer. I'm sure that will
> > > spark an interesting discussion.
> > 
> > You have already suggested it in email [1] but apparently you are _not_
> > maintainer of mvebu pci controller. get_maintainer.pl for part about
> > which you have talked in [1] says:
> > 
> > $ ./scripts/get_maintainer.pl -f drivers/pci/controller/pci-aardvark.c
> 
> Remind me which file this patch is touching?

So read again what you have presented in the past, in the email to which
you have referenced. I sent link to that your email in previous email.

Or you absolutely incompetent and I should have remind also previous
email to which you wrote your reaction?

> > The only _toy_ here is your broken mvebu board which your ego was unable
> > to fix, and you have put it into recycling pile [2] and since than for
> > months you are trying to reject every change or improvement in mvebu
> > drivers and trying to find out a way how to remove all mvebu code, like
> > if you were not able to fix your toy, then broke it also to all other
> > people. You have already expressed this, but I'm not going to search
> > emails more and find these your statements.
> 
> At this stage, this is pure paranoia.

No, just pure reality of your behavior of what you are doing and what
you are saying.

> Do you think I am so emotionally
> attached to HW purity that I would plot the annihilation of some ugly
> platform?

I do not think. You personally, have presented this statement, and I'm
just reminding it to you like you have asked for it.

> > Sorry, I'm stopping here. This is just a prove that you are not
> > qualified in reviewing mvebu code.
> 
> Happy not to have to review this code.

You are doing it for more than one year. Are you happy with it? Seem
absolutely.

> Just stop Cc'ing me on your patches

As there no progress from your side, nor change of your behavior from
more than one year, I'm accepting this offer.

This is my last email to you and I'm stopping right now to read your
emails.

I'm not obligated to remind you everything what you are asking just
because you are lazy to find you what you have wrote in the past.

> and don't expect me to merge any IRQ related patches coming
> from you.
> 
> 	M.
> 
> -- 
> Without deviation from the norm, progress is not possible.

Re: [PATCH 2/6] irqchip/armada-370-xp: Implement SoC Error interrupts

[Rob Herring]

On Sat, May 07, 2022 at 11:20:54AM +0200, Pali Rohár wrote:
> On Saturday 07 May 2022 10:01:52 Marc Zyngier wrote:
> > On Fri, 06 May 2022 19:55:46 +0100,
> > Pali Rohár <pali@kernel.org> wrote:
> > > 
> > > On Friday 06 May 2022 19:47:25 Marc Zyngier wrote:
> > > > On Fri, 06 May 2022 19:30:51 +0100,
> > > > Pali Rohár <pali@kernel.org> wrote:
> > > > > 
> > > > > On Friday 06 May 2022 19:19:46 Marc Zyngier wrote:
> > > > > > On Fri, 06 May 2022 14:40:25 +0100,
> > > > > > Pali Rohár <pali@kernel.org> wrote:
> > > > > > > 
> > > > > > > +static void armada_370_xp_soc_err_irq_unmask(struct irq_data *d);
> > > > > > > +
> > > > > > >  static inline bool is_percpu_irq(irq_hw_number_t irq)
> > > > > > >  {
> > > > > > >  	if (irq <= ARMADA_370_XP_MAX_PER_CPU_IRQS)
> > > > > > > @@ -509,6 +517,27 @@ static void armada_xp_mpic_reenable_percpu(void)
> > > > > > >  		armada_370_xp_irq_unmask(data);
> > > > > > >  	}
> > > > > > >  
> > > > > > > +	/* Re-enable per-CPU SoC Error interrupts that were enabled before suspend */
> > > > > > > +	for (irq = 0; irq < soc_err_irq_num_regs * 32; irq++) {
> > > > > > > +		struct irq_data *data;
> > > > > > > +		int virq;
> > > > > > > +
> > > > > > > +		virq = irq_linear_revmap(armada_370_xp_soc_err_domain, irq);
> > > > > > > +		if (virq == 0)
> > > > > > > +			continue;
> > > > > > > +
> > > > > > > +		data = irq_get_irq_data(virq);
> > > > > > > +
> > > > > > > +		if (!irq_percpu_is_enabled(virq))
> > > > > > > +			continue;
> > > > > > > +
> > > > > > > +		armada_370_xp_soc_err_irq_unmask(data);
> > > > > > > +	}
> > > > > > 
> > > > > > So you do this loop and all these lookups, both here and in the resume
> > > > > > function (duplicated code!) just to be able to call the unmask
> > > > > > function?  This would be better served by two straight writes of the
> > > > > > mask register, which you'd conveniently save on suspend.
> > > > > > 
> > > > > > Yes, you have only duplicated the existing logic. But surely there is
> > > > > > something better to do.
> > > > > 
> > > > > Yes, I just used existing logic.
> > > > > 
> > > > > I'm not rewriting driver or doing big refactor of it, as this is not in
> > > > > the scope of the PCIe AER interrupt support.
> > > > 
> > > > Fair enough. By the same logic, I'm not taking any change to the
> > > > driver until it is put in a better shape. Your call.
> > > 
> > > If you are maintainer of this code then it is expected from _you_ to
> > > move the current code into _better shape_ as you wrote and expect. And
> > > then show us exactly, how new changes in this driver should look like,
> > > in examples.
> > 
> > Sorry, but that's not how this works. You are the one willing to
> > change a sub-par piece of code, you get to make it better. You
> > obviously have the means (the HW) and the incentive (these patches).
> > But you don't get to make something even more unmaintainable because
> > you're unwilling to do some extra work.
> > 
> > If you're unhappy with my position, that's fine. I suggest you take it
> > with Thomas, and maybe even Linus. As I suggested before, you can also
> > post a patch removing me as the irqchip maintainer. I'm sure that will
> > spark an interesting discussion.
> 
> You have already suggested it in email [1] but apparently you are _not_
> maintainer of mvebu pci controller. get_maintainer.pl for part about
> which you have talked in [1] says:
> 
> $ ./scripts/get_maintainer.pl -f drivers/pci/controller/pci-aardvark.c
> Thomas Petazzoni <thomas.petazzoni@bootlin.com> (maintainer:PCI DRIVER FOR AARDVARK (Marvell Armada 3700))
> "Pali Rohár" <pali@kernel.org> (maintainer:PCI DRIVER FOR AARDVARK (Marvell Armada 3700))
> Lorenzo Pieralisi <lorenzo.pieralisi@arm.com> (supporter:PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS)
> Rob Herring <robh@kernel.org> (reviewer:PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS)

Please just refactor the code in question. You've wasted more time 
arguing about it than it would take to do. Having done a bit of PCI 
refactoring, I can tell you hardly anyone else does. I can barely even 
get comments/acks on refactoring until I break platforms (which happens 
a lot). Maintainers have no other leverage other than what Marc pointed 
out.

In any case, I think there's no way the PCI maintainers will take this 
as-is at this point.

Rob

Re: [PATCH 2/6] irqchip/armada-370-xp: Implement SoC Error interrupts

[Thomas Gleixner]

Pali,

On Fri, May 06 2022 at 20:55, Pali Rohár wrote:
> On Friday 06 May 2022 19:47:25 Marc Zyngier wrote:
>> > I'm not rewriting driver or doing big refactor of it, as this is not in
>> > the scope of the PCIe AER interrupt support.
>>
>> Fair enough. By the same logic, I'm not taking any change to the
>> driver until it is put in a better shape. Your call.
>
> If you are maintainer of this code then it is expected from _you_ to
> move the current code into _better shape_ as you wrote and expect. And
> then show us exactly, how new changes in this driver should look like,
> in examples.

this is not how kernel development works.

Maintainers are not the servants who mop up the mess which random people
dump into the tree. They are gatekeepers and one of their duties is to
prevent that mess is created or existing mess is proliferated.

You are asking the maintainer to take your changes, deal with the
fallout and maintain them for a long time free of charge. So it's a very
reasonable request from a maintainer to ask for refactoring of existing
code before adding new functionality to it.

With such a request the refactoring becomes scope of your work, whether
you and your manager like it or not. If you don't want to do that extra
work, then don't expect maintainers to care about your fancy new
features.

Marc gave you very reasonable and consice directions how the code should
be reworked. He spent a lot of time explaining it to you. Again, free of
charge. Now you expect him to do your homework free of charge, so you
can get your feature merged? Nice try.

Thanks,

        Thomas

[PATCH 3/6] ARM: dts: armada-38x.dtsi: Add node for MPIC SoC Error IRQ controller

[Pali Rohár]

It is child of the MPIC IRQ controller.

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 arch/arm/boot/dts/armada-38x.dtsi | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/arm/boot/dts/armada-38x.dtsi b/arch/arm/boot/dts/armada-38x.dtsi
index 9b1a24cc5e91..54051f0b1ad1 100644
--- a/arch/arm/boot/dts/armada-38x.dtsi
+++ b/arch/arm/boot/dts/armada-38x.dtsi
@@ -398,6 +398,11 @@
 				interrupt-controller;
 				msi-controller;
 				interrupts = <GIC_PPI 15 IRQ_TYPE_LEVEL_HIGH>;
+
+				soc_err: interrupt-controller@20 {
+					interrupt-controller;
+					#interrupt-cells = <1>;
+				};
 			};
 
 			timer: timer@20300 {
-- 
2.20.1

[PATCH 4/6] dt-bindings: PCI: mvebu: Update information about summary interrupt

[Pali Rohár]

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 Documentation/devicetree/bindings/pci/mvebu-pci.txt | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Documentation/devicetree/bindings/pci/mvebu-pci.txt b/Documentation/devicetree/bindings/pci/mvebu-pci.txt
index 6d022a9d36ee..ef33558210d0 100644
--- a/Documentation/devicetree/bindings/pci/mvebu-pci.txt
+++ b/Documentation/devicetree/bindings/pci/mvebu-pci.txt
@@ -83,6 +83,7 @@ and the following optional properties:
   specified will default to 100ms, as required by the PCIe specification.
 - interrupt-names: list of interrupt names, supported are:
    - "intx" - interrupt line triggered by one of the legacy interrupt
+   - "summary" - interrupt line triggered by any event, including "intx"
 - interrupts or interrupts-extended: List of the interrupt sources which
   corresponding to the "interrupt-names". If non-empty then also additional
   'interrupt-controller' subnode must be defined.
-- 
2.20.1

[PATCH 5/6] PCI: mvebu: Implement support for interrupts on emulated bridge

[Pali Rohár]

This adds support for PME and ERR interrupts reported by emulated bridge
(for PME and AER kernel drivers) via new Root Port irq chip as these
interrupts from PCIe Root Ports are handled by mvebu hardware completely
separately from INTx and MSI interrupts send by real PCIe devices.

With this change, kernel PME and AER drivers start working as they can
acquire required interrupt lines (provided by mvebu rp virtual irq chip).

Note that for this support, device tree files has to be properly adjusted
to provide "interrupts" or "interrupts-extended" property with summary
interrupt source and "interrupt-names" property with "summary" string.

If device tree files do not provide these properties then driver would work
as before and would not provide interrupts on emulated bridge, like before.

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 drivers/pci/controller/pci-mvebu.c | 208 ++++++++++++++++++++++++++---
 1 file changed, 189 insertions(+), 19 deletions(-)

diff --git a/drivers/pci/controller/pci-mvebu.c b/drivers/pci/controller/pci-mvebu.c
index 3c48b15e3948..74923202fe95 100644
--- a/drivers/pci/controller/pci-mvebu.c
+++ b/drivers/pci/controller/pci-mvebu.c
@@ -56,8 +56,16 @@
 #define PCIE_CONF_DATA_OFF	0x18fc
 #define PCIE_INT_CAUSE_OFF	0x1900
 #define PCIE_INT_UNMASK_OFF	0x1910
+#define  PCIE_INT_DET_COR		BIT(8)
+#define  PCIE_INT_DET_NONFATAL		BIT(9)
+#define  PCIE_INT_DET_FATAL		BIT(10)
+#define  PCIE_INT_ERR_FATAL		BIT(16)
+#define  PCIE_INT_ERR_NONFATAL		BIT(17)
+#define  PCIE_INT_ERR_COR		BIT(18)
 #define  PCIE_INT_INTX(i)		BIT(24+i)
 #define  PCIE_INT_PM_PME		BIT(28)
+#define  PCIE_INT_DET_MASK		(PCIE_INT_DET_COR | PCIE_INT_DET_NONFATAL | PCIE_INT_DET_FATAL)
+#define  PCIE_INT_ERR_MASK		(PCIE_INT_ERR_FATAL | PCIE_INT_ERR_NONFATAL | PCIE_INT_ERR_COR)
 #define  PCIE_INT_ALL_MASK		GENMASK(31, 0)
 #define PCIE_CTRL_OFF		0x1a00
 #define  PCIE_CTRL_X1_MODE		0x0001
@@ -120,9 +128,12 @@ struct mvebu_pcie_port {
 	struct resource regs;
 	u8 slot_power_limit_value;
 	u8 slot_power_limit_scale;
+	struct irq_domain *rp_irq_domain;
 	struct irq_domain *intx_irq_domain;
 	raw_spinlock_t irq_lock;
+	int summary_irq;
 	int intx_irq;
+	bool pme_pending;
 };
 
 static inline void mvebu_writel(struct mvebu_pcie_port *port, u32 val, u32 reg)
@@ -321,12 +332,27 @@ static void mvebu_pcie_setup_hw(struct mvebu_pcie_port *port)
 	/* Clear all interrupt causes. */
 	mvebu_writel(port, ~PCIE_INT_ALL_MASK, PCIE_INT_CAUSE_OFF);
 
-	/* Check if "intx" interrupt was specified in DT. */
-	if (port->intx_irq > 0)
+	/*
+	 * Unmask all error interrupts which are internally generated.
+	 * They cannot be disabled by SERR# Enable bit in PCI Command register,
+	 * see Figure 6-3: Pseudo Logic Diagram for Error Message Controls in
+	 * PCIe base specification.
+	 * Internally generated mvebu interrupts are reported via mvebu summary
+	 * interrupt which requires "summary" interrupt to be specified in DT.
+	 */
+	if (port->summary_irq > 0) {
+		unmask = mvebu_readl(port, PCIE_INT_UNMASK_OFF);
+		unmask |= PCIE_INT_DET_MASK;
+		mvebu_writel(port, unmask, PCIE_INT_UNMASK_OFF);
+	}
+
+	/* Check if "summary" or "intx" interrupt was specified in DT. */
+	if (port->summary_irq > 0 || port->intx_irq > 0)
 		return;
 
 	/*
-	 * Fallback code when "intx" interrupt was not specified in DT:
+	 * Fallback code when neither "summary" interrupt, nor "intx"
+	 * interrupt was specified in DT:
 	 * Unmask all legacy INTx interrupts as driver does not provide a way
 	 * for masking and unmasking of individual legacy INTx interrupts.
 	 * Legacy INTx are reported via one shared GIC source and therefore
@@ -603,11 +629,16 @@ mvebu_pci_bridge_emul_base_conf_read(struct pci_bridge_emul *bridge,
 	case PCI_INTERRUPT_LINE: {
 		/*
 		 * From the whole 32bit register we support reading from HW only
-		 * one bit: PCI_BRIDGE_CTL_BUS_RESET.
+		 * two bits: PCI_BRIDGE_CTL_BUS_RESET and PCI_BRIDGE_CTL_SERR.
 		 * Other bits are retrieved only from emulated config buffer.
 		 */
 		__le32 *cfgspace = (__le32 *)&bridge->conf;
 		u32 val = le32_to_cpu(cfgspace[PCI_INTERRUPT_LINE / 4]);
+		if ((mvebu_readl(port, PCIE_INT_UNMASK_OFF) &
+		      PCIE_INT_ERR_MASK) == PCIE_INT_ERR_MASK)
+			val |= PCI_BRIDGE_CTL_SERR << 16;
+		else
+			val &= ~(PCI_BRIDGE_CTL_SERR << 16);
 		if (mvebu_readl(port, PCIE_CTRL_OFF) & PCIE_CTRL_MASTER_HOT_RESET)
 			val |= PCI_BRIDGE_CTL_BUS_RESET << 16;
 		else
@@ -675,6 +706,11 @@ mvebu_pci_bridge_emul_pcie_conf_read(struct pci_bridge_emul *bridge,
 		break;
 	}
 
+	case PCI_EXP_RTCTL:
+		*value = (mvebu_readl(port, PCIE_INT_UNMASK_OFF) &
+			  PCIE_INT_PM_PME) ? PCI_EXP_RTCTL_PMEIE : 0;
+		break;
+
 	case PCI_EXP_RTSTA:
 		*value = mvebu_readl(port, PCIE_RC_RTSTA);
 		break;
@@ -780,6 +816,14 @@ mvebu_pci_bridge_emul_base_conf_write(struct pci_bridge_emul *bridge,
 		break;
 
 	case PCI_INTERRUPT_LINE:
+		if (mask & (PCI_BRIDGE_CTL_SERR << 16)) {
+			u32 unmask = mvebu_readl(port, PCIE_INT_UNMASK_OFF);
+			if (new & (PCI_BRIDGE_CTL_SERR << 16))
+				unmask |= PCIE_INT_ERR_MASK;
+			else
+				unmask &= ~PCIE_INT_ERR_MASK;
+			mvebu_writel(port, unmask, PCIE_INT_UNMASK_OFF);
+		}
 		if (mask & (PCI_BRIDGE_CTL_BUS_RESET << 16)) {
 			u32 ctrl = mvebu_readl(port, PCIE_CTRL_OFF);
 			if (new & (PCI_BRIDGE_CTL_BUS_RESET << 16))
@@ -838,10 +882,25 @@ mvebu_pci_bridge_emul_pcie_conf_write(struct pci_bridge_emul *bridge,
 		 * PME Status bit in Root Status Register (PCIE_RC_RTSTA)
 		 * is read-only and can be cleared only by writing 0b to the
 		 * Interrupt Cause RW0C register (PCIE_INT_CAUSE_OFF). So
-		 * clear PME via Interrupt Cause.
+		 * clear PME via Interrupt Cause and also set port->pme_pending
+		 * variable to false value to start processing PME interrupts
+		 * in interrupt handler again.
 		 */
-		if (new & PCI_EXP_RTSTA_PME)
+		if (new & PCI_EXP_RTSTA_PME) {
 			mvebu_writel(port, ~PCIE_INT_PM_PME, PCIE_INT_CAUSE_OFF);
+			port->pme_pending = false;
+		}
+		break;
+
+	case PCI_EXP_RTCTL:
+		if (mask & PCI_EXP_RTCTL_PMEIE) {
+			u32 unmask = mvebu_readl(port, PCIE_INT_UNMASK_OFF);
+			if (new & PCI_EXP_RTCTL_PMEIE)
+				unmask |= PCIE_INT_PM_PME;
+			else
+				unmask &= ~PCIE_INT_PM_PME;
+			mvebu_writel(port, unmask, PCIE_INT_UNMASK_OFF);
+		}
 		break;
 
 	case PCI_EXP_DEVCTL2:
@@ -924,6 +983,14 @@ static int mvebu_pci_bridge_emul_init(struct mvebu_pcie_port *port)
 		bridge_flags |= PCI_BRIDGE_EMUL_NO_IO_FORWARD;
 	}
 
+	/*
+	 * Interrupts on emulated bridge are supported only when "summary"
+	 * interrupt was specified in DT. Without it emulated bridge cannot
+	 * emulate interrupts.
+	 */
+	if (port->summary_irq > 0)
+		bridge->conf.intpin = PCI_INTERRUPT_INTA;
+
 	/*
 	 * Older mvebu hardware provides PCIe Capability structure only in
 	 * version 1. New hardware provides it in version 2.
@@ -1071,6 +1138,26 @@ static const struct irq_domain_ops mvebu_pcie_intx_irq_domain_ops = {
 	.xlate = irq_domain_xlate_onecell,
 };
 
+static struct irq_chip rp_irq_chip = {
+	.name = "mvebu-rp",
+};
+
+static int mvebu_pcie_rp_irq_map(struct irq_domain *h,
+				   unsigned int virq, irq_hw_number_t hwirq)
+{
+	struct mvebu_pcie_port *port = h->host_data;
+
+	irq_set_chip_and_handler(virq, &rp_irq_chip, handle_simple_irq);
+	irq_set_chip_data(virq, port);
+
+	return 0;
+}
+
+static const struct irq_domain_ops mvebu_pcie_rp_irq_domain_ops = {
+	.map = mvebu_pcie_rp_irq_map,
+	.xlate = irq_domain_xlate_onecell,
+};
+
 static int mvebu_pcie_init_irq_domain(struct mvebu_pcie_port *port)
 {
 	struct device *dev = &port->pcie->pdev->dev;
@@ -1093,6 +1180,22 @@ static int mvebu_pcie_init_irq_domain(struct mvebu_pcie_port *port)
 		return -ENOMEM;
 	}
 
+	/*
+	 * When "summary" interrupt was not specified in DT then there is no support
+	 * for interrupts on emulated root bridge. So skip following initialization.
+	 */
+	if (port->summary_irq <= 0)
+		return 0;
+
+	port->rp_irq_domain = irq_domain_add_linear(NULL, 1,
+						      &mvebu_pcie_rp_irq_domain_ops,
+						      port);
+	if (!port->rp_irq_domain) {
+		irq_domain_remove(port->intx_irq_domain);
+		dev_err(dev, "Failed to add Root Port IRQ domain for %s\n", port->name);
+		return -ENOMEM;
+	}
+
 	return 0;
 }
 
@@ -1110,6 +1213,34 @@ static void mvebu_pcie_irq_handler(struct irq_desc *desc)
 	unmask = mvebu_readl(port, PCIE_INT_UNMASK_OFF);
 	status = cause & unmask;
 
+	/* Process PME interrupt */
+	if ((status & PCIE_INT_PM_PME) && !port->pme_pending) {
+		/*
+		 * Do not clear PME interrupt bit in Cause Register as it
+		 * invalidates also content of Root Status Register. Instead
+		 * set port->pme_pending variable to true to indicate that
+		 * next time PME interrupt should be ignored until variable
+		 * is back to the false value.
+		 */
+		port->pme_pending = true;
+		if (generic_handle_domain_irq(port->rp_irq_domain, 0) == -EINVAL)
+			dev_err_ratelimited(dev, "unhandled PME IRQ\n");
+	}
+
+	/* Process ERR interrupt */
+	if (status & PCIE_INT_ERR_MASK) {
+		mvebu_writel(port, ~PCIE_INT_ERR_MASK, PCIE_INT_CAUSE_OFF);
+		if (generic_handle_domain_irq(port->rp_irq_domain, 0) == -EINVAL)
+			dev_err_ratelimited(dev, "unhandled ERR IRQ\n");
+	}
+
+	/* Process local ERR interrupt */
+	if (status & PCIE_INT_DET_MASK) {
+		mvebu_writel(port, ~PCIE_INT_DET_MASK, PCIE_INT_CAUSE_OFF);
+		if (generic_handle_domain_irq(port->rp_irq_domain, 0) == -EINVAL)
+			dev_err_ratelimited(dev, "unhandled ERR IRQ\n");
+	}
+
 	/* Process legacy INTx interrupts */
 	for (i = 0; i < PCI_NUM_INTX; i++) {
 		if (!(status & PCIE_INT_INTX(i)))
@@ -1124,9 +1255,29 @@ static void mvebu_pcie_irq_handler(struct irq_desc *desc)
 
 static int mvebu_pcie_map_irq(const struct pci_dev *dev, u8 slot, u8 pin)
 {
-	/* Interrupt support on mvebu emulated bridges is not implemented yet */
-	if (dev->bus->number == 0)
-		return 0; /* Proper return code 0 == NO_IRQ */
+	struct mvebu_pcie_port *port;
+	struct mvebu_pcie *pcie;
+
+	if (dev->bus->number == 0) {
+		/*
+		 * Each emulated root bridge for every mvebu port has its own
+		 * Root Port irq chip and irq domain. Argument pin is the INTx
+		 * pin (1=INTA, 2=INTB, 3=INTC, 4=INTD) and hwirq for function
+		 * irq_create_mapping() is indexed from zero.
+		 */
+		pcie = dev->bus->sysdata;
+		port = mvebu_pcie_find_port(pcie, dev->bus, PCI_DEVFN(slot, 0));
+		if (!port)
+			return 0; /* Proper return code 0 == NO_IRQ */
+		/*
+		 * port->rp_irq_domain is available only when "summary"
+		 * interrupt was specified in DT. When is not available then
+		 * interrupts for emulated root bridge are not provided.
+		 */
+		if (port->summary_irq <= 0)
+			return 0; /* Proper return code 0 == NO_IRQ */
+		return irq_create_mapping(port->rp_irq_domain, pin - 1);
+	}
 
 	return of_irq_parse_and_map_pci(dev, slot, pin);
 }
@@ -1321,17 +1472,32 @@ static int mvebu_pcie_parse_port(struct mvebu_pcie *pcie,
 	}
 
 	/*
-	 * Old DT bindings do not contain "intx" interrupt
+	 * Old DT bindings do not contain "summary" interrupt
 	 * so do not fail probing driver when interrupt does not exist.
 	 */
-	port->intx_irq = of_irq_get_byname(child, "intx");
-	if (port->intx_irq == -EPROBE_DEFER) {
-		ret = port->intx_irq;
+	port->summary_irq = of_irq_get_byname(child, "summary");
+	if (port->summary_irq == -EPROBE_DEFER) {
+		ret = port->summary_irq;
 		goto err;
-	}
-	if (port->intx_irq <= 0) {
-		dev_warn(dev, "%s: legacy INTx interrupts cannot be masked individually, "
-			      "%pOF does not contain intx interrupt\n",
+	} else if (port->summary_irq <= 0) {
+		/*
+		 * When "summary" interrupt (which includes also intx) is
+		 * unsupported then fallback to dedicated "intx" interrupt.
+		 * Old DT bindings do not contain "intx" interrupt so do not
+		 * fail probing driver when interrupt does not exist.
+		 */
+		port->intx_irq = of_irq_get_byname(child, "intx");
+		if (port->intx_irq == -EPROBE_DEFER) {
+			ret = port->intx_irq;
+			goto err;
+		}
+		if (port->intx_irq <= 0) {
+			dev_warn(dev, "%s: legacy INTx interrupts cannot be masked individually, "
+				      "%pOF does not contain intx interrupt\n",
+				 port->name, child);
+		}
+		dev_warn(dev, "%s: interrupts on Root Port are unsupported, "
+			      "%pOF does not contain summary interrupt\n",
 			 port->name, child);
 	}
 
@@ -1540,7 +1706,8 @@ static int mvebu_pcie_probe(struct platform_device *pdev)
 
 	for (i = 0; i < pcie->nports; i++) {
 		struct mvebu_pcie_port *port = &pcie->ports[i];
-		int irq = port->intx_irq;
+		int irq = (port->summary_irq > 0) ? port->summary_irq :
+			  (port->intx_irq > 0) ? port->intx_irq : 0;
 
 		child = port->dn;
 		if (!child)
@@ -1688,7 +1855,8 @@ static int mvebu_pcie_remove(struct platform_device *pdev)
 
 	for (i = 0; i < pcie->nports; i++) {
 		struct mvebu_pcie_port *port = &pcie->ports[i];
-		int irq = port->intx_irq;
+		int irq = (port->summary_irq > 0) ? port->summary_irq :
+			  (port->intx_irq > 0) ? port->intx_irq : 0;
 
 		if (!port->base)
 			continue;
@@ -1710,6 +1878,8 @@ static int mvebu_pcie_remove(struct platform_device *pdev)
 		/* Remove IRQ domains. */
 		if (port->intx_irq_domain)
 			irq_domain_remove(port->intx_irq_domain);
+		if (port->rp_irq_domain)
+			irq_domain_remove(port->rp_irq_domain);
 
 		/* Free config space for emulated root bridge. */
 		pci_bridge_emul_cleanup(&port->bridge);
-- 
2.20.1

[PATCH 6/6] ARM: dts: armada-385.dtsi: Add definitions for PCIe summary interrupts

[Pali Rohár]

PCIe summary interrupt is reported by MPIC SoC Error IRQ controller.

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 arch/arm/boot/dts/armada-385.dtsi | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/arch/arm/boot/dts/armada-385.dtsi b/arch/arm/boot/dts/armada-385.dtsi
index 83392b92dae2..b8219c13c93b 100644
--- a/arch/arm/boot/dts/armada-385.dtsi
+++ b/arch/arm/boot/dts/armada-385.dtsi
@@ -69,8 +69,9 @@
 				reg = <0x0800 0 0 0 0>;
 				#address-cells = <3>;
 				#size-cells = <2>;
-				interrupt-names = "intx";
-				interrupts-extended = <&gic GIC_SPI 29 IRQ_TYPE_LEVEL_HIGH>;
+				interrupt-names = "intx", "summary";
+				interrupts-extended = <&gic GIC_SPI 29 IRQ_TYPE_LEVEL_HIGH>,
+						      <&soc_err 4>;
 				#interrupt-cells = <1>;
 				ranges = <0x82000000 0 0 0x82000000 0x1 0 1 0
 					  0x81000000 0 0 0x81000000 0x1 0 1 0>;
@@ -97,8 +98,9 @@
 				reg = <0x1000 0 0 0 0>;
 				#address-cells = <3>;
 				#size-cells = <2>;
-				interrupt-names = "intx";
-				interrupts-extended = <&gic GIC_SPI 33 IRQ_TYPE_LEVEL_HIGH>;
+				interrupt-names = "intx", "summary";
+				interrupts-extended = <&gic GIC_SPI 33 IRQ_TYPE_LEVEL_HIGH>,
+						      <&soc_err 5>;
 				#interrupt-cells = <1>;
 				ranges = <0x82000000 0 0 0x82000000 0x2 0 1 0
 					  0x81000000 0 0 0x81000000 0x2 0 1 0>;
@@ -125,8 +127,9 @@
 				reg = <0x1800 0 0 0 0>;
 				#address-cells = <3>;
 				#size-cells = <2>;
-				interrupt-names = "intx";
-				interrupts-extended = <&gic GIC_SPI 70 IRQ_TYPE_LEVEL_HIGH>;
+				interrupt-names = "intx", "summary";
+				interrupts-extended = <&gic GIC_SPI 70 IRQ_TYPE_LEVEL_HIGH>,
+						      <&soc_err 15>;
 				#interrupt-cells = <1>;
 				ranges = <0x82000000 0 0 0x82000000 0x3 0 1 0
 					  0x81000000 0 0 0x81000000 0x3 0 1 0>;
@@ -156,8 +159,9 @@
 				reg = <0x2000 0 0 0 0>;
 				#address-cells = <3>;
 				#size-cells = <2>;
-				interrupt-names = "intx";
-				interrupts-extended = <&gic GIC_SPI 71 IRQ_TYPE_LEVEL_HIGH>;
+				interrupt-names = "intx", "summary";
+				interrupts-extended = <&gic GIC_SPI 71 IRQ_TYPE_LEVEL_HIGH>,
+						      <&soc_err 16>;
 				#interrupt-cells = <1>;
 				ranges = <0x82000000 0 0 0x82000000 0x4 0 1 0
 					  0x81000000 0 0 0x81000000 0x4 0 1 0>;
-- 
2.20.1

Re: [PATCH 0/6] PCI: mvebu: Add support for PME and AER interrupts

[Pali Rohár]

On Friday 06 May 2022 15:40:23 Pali Rohár wrote:
> mvebu PCIe PME and AER interrupts are reported via PCIe summary
> interrupt. PCIe summary interrupt is reported via mvebu MPIC SoC error
> summary interrupt. And MPIC SoC error summary interrupt is reported via
> MPIC IRQ 4.
> 
> This patch series implements support for interrupts in MPIC SoC error
> hierarchy in irq-armada-370-xp.c driver and support for interrupts in
> mvebu PCIe hierarchy in pci-mvebu.c.
> 
> Finally PCIe PME and AER interrupts are routed to the correct PCIe Root
> Port, which allows kernel PME and AER drivers to take care of them.
> 
> Tested on A385 board and kernel PME and AER drivers works correctly:
> 
> [    0.898482] pcieport 0000:00:01.0: PME: Signaling with IRQ 61
> [    0.904422] pcieport 0000:00:01.0: AER: enabled with IRQ 61
> [    0.910113] pcieport 0000:00:02.0: enabling device (0140 -> 0142)
> [    0.916299] pcieport 0000:00:02.0: PME: Signaling with IRQ 62
> [    0.922216] pcieport 0000:00:02.0: AER: enabled with IRQ 62
> [    0.927917] pcieport 0000:00:03.0: enabling device (0140 -> 0142)
> [    0.934090] pcieport 0000:00:03.0: PME: Signaling with IRQ 63
> [    0.940006] pcieport 0000:00:03.0: AER: enabled with IRQ 63
> 
> This change finally allows to debug PCIe issues on A385 boards.

FYI I tested that AER errors are now really handled by kernel AER driver:

[ 2733.258661] pcieport 0000:00:02.0: AER: Multiple Uncorrected (Non-Fatal) error received: 0000:02:00.0
[ 2733.258661] pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:01:00.0
[ 2733.258682] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 2733.267932] ath10k_pci 0000:02:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
[ 2733.275956] pcieport 0000:00:01.0:   device [11ab:6820] error status/mask=00000001/00002000
[ 2733.285547] ath10k_pci 0000:02:00.0:   device [168c:003c] error status/mask=00100000/00000000
[ 2733.296876] pcieport 0000:00:01.0:    [ 0] RxErr                  (First)
[ 2733.305245] ath10k_pci 0000:02:00.0:    [20] UnsupReq               (First)
[ 2733.305251] ath10k_pci 0000:02:00.0: AER:   TLP Header: 30000000 02080030 00000000 00000000
[ 2733.305282] ath10k_pci 0000:02:00.0: AER: can't recover (no error_detected callback)
[ 2733.313816] nvme 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Transmitter ID)
[ 2733.320671] pcieport 0000:00:02.0: AER: device recovery failed
[ 2733.327609] nvme 0000:01:00.0:   device [1e0f:0001] error status/mask=00001041/00002000
[ 2733.367127] nvme 0000:01:00.0:    [ 0] RxErr                  (First)
[ 2733.373591] nvme 0000:01:00.0:    [ 6] BadTLP                
[ 2733.379358] nvme 0000:01:00.0:    [12] Timeout               
[ 2733.385120] nvme 0000:01:00.0: AER:   Error of this Agent is reported first

> Pali Rohár (6):
>   dt-bindings: irqchip: armada-370-xp: Update information about MPIC SoC
>     Error
>   irqchip/armada-370-xp: Implement SoC Error interrupts
>   ARM: dts: armada-38x.dtsi: Add node for MPIC SoC Error IRQ controller
>   dt-bindings: PCI: mvebu: Update information about summary interrupt
>   PCI: mvebu: Implement support for interrupts on emulated bridge
>   ARM: dts: armada-385.dtsi: Add definitions for PCIe summary interrupts
> 
>  .../marvell,armada-370-xp-mpic.txt            |   9 +
>  .../devicetree/bindings/pci/mvebu-pci.txt     |   1 +
>  arch/arm/boot/dts/armada-385.dtsi             |  20 +-
>  arch/arm/boot/dts/armada-38x.dtsi             |   5 +
>  drivers/irqchip/irq-armada-370-xp.c           | 213 +++++++++++++++++-
>  drivers/pci/controller/pci-mvebu.c            | 208 +++++++++++++++--
>  6 files changed, 426 insertions(+), 30 deletions(-)
> 
> -- 
> 2.20.1
>