[RFC PATCH v1 0/4] arm/arm64: fix a migrating irq bug when hotplug cpu

[RFC PATCH v1 0/4] arm/arm64: fix a migrating irq bug when hotplug cpu

[Yang Yingliang]

Hi All,

There is a bug:

When cpu is disabled, all irqs will be migratged to another cpu.
In some cases, a new affinity is different, it needed to be coppied
to irq's affinity. But if the type of irq is LPI, it's affinity will
not be coppied because of irq_set_affinity's return value.



As Marc and Will suggested, I refactor the arm/arm64 migrating interrupts
code and fix the migrating irq bug while cpu is offline.

I'm trying let the core code do the migrating interrupts matter. kernel/irq/migration.c
depends on CONFIG_GENERIC_PENDING_IRQ, so I make it selected by CONFIG_SMP and
CONFIG_HOTPLUG_CPU and rename it to CONFIG_GENERIC_IRQ_MIGRATION for more general.
When CONFIG_GENERIC_IRQ_MIGRATION is enabled, an interrupt whose state_use_accessors
is not set with IRQD_MOVE_PCNTXT won't be migrated immediately in irq_set_affinity_locked().
So introduce irq_settings_set_move_pcntxt() helper to set the state in gic_irq_domain_map().

With the above preparation, move the migrating interrupts code into kernel/irq/migration.c
and fix the bug by using irq_do_set_affinity().

Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Cc: Hanjun Guo <hanjun.guo@linaro.org>

Yang Yingliang (4):
  genirq: Introduce irq_settings_set_move_pcntxt() helper
  irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT
  genirq: rename config GENERIC_PENDING_IRQ to GENERIC_IRQ_MIGRATION
  arm/arm64: fix a migrating irq bug when hotplug cpu

 arch/arc/Kconfig               |  2 +-
 arch/arm/Kconfig               |  1 +
 arch/arm/include/asm/irq.h     |  1 -
 arch/arm/kernel/irq.c          | 62 ------------------------------------------
 arch/arm64/Kconfig             |  1 +
 arch/arm64/include/asm/irq.h   |  1 -
 arch/arm64/kernel/irq.c        | 62 ------------------------------------------
 arch/hexagon/Kconfig           |  2 +-
 arch/ia64/Kconfig              |  2 +-
 arch/tile/Kconfig              |  2 +-
 arch/x86/Kconfig               |  2 +-
 arch/x86/kernel/apic/io_apic.c |  2 +-
 drivers/irqchip/irq-gic-v3.c   |  2 ++
 include/linux/irq.h            |  5 +++-
 include/linux/irqdesc.h        |  2 +-
 kernel/irq/Kconfig             |  4 +--
 kernel/irq/Makefile            |  2 +-
 kernel/irq/irqdesc.c           | 18 ++++++++++--
 kernel/irq/manage.c            |  2 +-
 kernel/irq/migration.c         | 62 ++++++++++++++++++++++++++++++++++++++++++
 kernel/irq/proc.c              |  2 +-
 kernel/irq/settings.h          |  5 ++++
 22 files changed, 102 insertions(+), 142 deletions(-)

-- 
2.5.0

[RFC PATCH v1 0/4] arm/arm64: fix a migrating irq bug when hotplug cpu

[Yang Yingliang]

Hi All,

There is a bug:

When cpu is disabled, all irqs will be migratged to another cpu.
In some cases, a new affinity is different, it needed to be coppied
to irq's affinity. But if the type of irq is LPI, it's affinity will
not be coppied because of irq_set_affinity's return value.



As Marc and Will suggested, I refactor the arm/arm64 migrating interrupts
code and fix the migrating irq bug while cpu is offline.

I'm trying let the core code do the migrating interrupts matter. kernel/irq/migration.c
depends on CONFIG_GENERIC_PENDING_IRQ, so I make it selected by CONFIG_SMP and
CONFIG_HOTPLUG_CPU and rename it to CONFIG_GENERIC_IRQ_MIGRATION for more general.
When CONFIG_GENERIC_IRQ_MIGRATION is enabled, an interrupt whose state_use_accessors
is not set with IRQD_MOVE_PCNTXT won't be migrated immediately in irq_set_affinity_locked().
So introduce irq_settings_set_move_pcntxt() helper to set the state in gic_irq_domain_map().

With the above preparation, move the migrating interrupts code into kernel/irq/migration.c
and fix the bug by using irq_do_set_affinity().

Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Cc: Hanjun Guo <hanjun.guo@linaro.org>

Yang Yingliang (4):
  genirq: Introduce irq_settings_set_move_pcntxt() helper
  irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT
  genirq: rename config GENERIC_PENDING_IRQ to GENERIC_IRQ_MIGRATION
  arm/arm64: fix a migrating irq bug when hotplug cpu

 arch/arc/Kconfig               |  2 +-
 arch/arm/Kconfig               |  1 +
 arch/arm/include/asm/irq.h     |  1 -
 arch/arm/kernel/irq.c          | 62 ------------------------------------------
 arch/arm64/Kconfig             |  1 +
 arch/arm64/include/asm/irq.h   |  1 -
 arch/arm64/kernel/irq.c        | 62 ------------------------------------------
 arch/hexagon/Kconfig           |  2 +-
 arch/ia64/Kconfig              |  2 +-
 arch/tile/Kconfig              |  2 +-
 arch/x86/Kconfig               |  2 +-
 arch/x86/kernel/apic/io_apic.c |  2 +-
 drivers/irqchip/irq-gic-v3.c   |  2 ++
 include/linux/irq.h            |  5 +++-
 include/linux/irqdesc.h        |  2 +-
 kernel/irq/Kconfig             |  4 +--
 kernel/irq/Makefile            |  2 +-
 kernel/irq/irqdesc.c           | 18 ++++++++++--
 kernel/irq/manage.c            |  2 +-
 kernel/irq/migration.c         | 62 ++++++++++++++++++++++++++++++++++++++++++
 kernel/irq/proc.c              |  2 +-
 kernel/irq/settings.h          |  5 ++++
 22 files changed, 102 insertions(+), 142 deletions(-)

-- 
2.5.0

[RFC PATCH v1 1/4] genirq: Introduce irq_settings_set_move_pcntxt() helper

[Yang Yingliang]

It's expected to use the helper when interrupt can be moved
in process.

Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Cc: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
---
 include/linux/irq.h   |  1 +
 kernel/irq/irqdesc.c  | 12 ++++++++++++
 kernel/irq/settings.h |  5 +++++
 3 files changed, 18 insertions(+)

diff --git a/include/linux/irq.h b/include/linux/irq.h
index 6f8b340..f4ecfb9 100644
--- a/include/linux/irq.h
+++ b/include/linux/irq.h
@@ -514,6 +514,7 @@ static inline void irq_set_chip_and_handler(unsigned int irq, struct irq_chip *c
 }
 
 extern int irq_set_percpu_devid(unsigned int irq);
+extern int irq_set_move_pcntxt(unsigned int irq);
 
 extern void
 __irq_set_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained,
diff --git a/kernel/irq/irqdesc.c b/kernel/irq/irqdesc.c
index 0a2a4b6..c036a1a 100644
--- a/kernel/irq/irqdesc.c
+++ b/kernel/irq/irqdesc.c
@@ -580,6 +580,18 @@ int irq_set_percpu_devid(unsigned int irq)
 	return 0;
 }
 
+int irq_set_move_pcntxt(unsigned int irq)
+{
+	struct irq_desc *desc = irq_to_desc(irq);
+
+	if (!desc)
+		return -EINVAL;
+
+	irq_settings_set_move_pcntxt(desc);
+
+	return 0;
+}
+
 void kstat_incr_irq_this_cpu(unsigned int irq)
 {
 	kstat_incr_irqs_this_cpu(irq_to_desc(irq));
diff --git a/kernel/irq/settings.h b/kernel/irq/settings.h
index 3320b84..ea58bc2 100644
--- a/kernel/irq/settings.h
+++ b/kernel/irq/settings.h
@@ -135,6 +135,11 @@ static inline void irq_settings_set_noprobe(struct irq_desc *desc)
 	desc->status_use_accessors |= _IRQ_NOPROBE;
 }
 
+static inline void irq_settings_set_move_pcntxt(struct irq_desc *desc)
+{
+	desc->status_use_accessors |= _IRQ_MOVE_PCNTXT;
+}
+
 static inline bool irq_settings_can_move_pcntxt(struct irq_desc *desc)
 {
 	return desc->status_use_accessors & _IRQ_MOVE_PCNTXT;
-- 
2.5.0

[RFC PATCH v1 1/4] genirq: Introduce irq_settings_set_move_pcntxt() helper

[Yang Yingliang]

It's expected to use the helper when interrupt can be moved
in process.

Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Cc: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
---
 include/linux/irq.h   |  1 +
 kernel/irq/irqdesc.c  | 12 ++++++++++++
 kernel/irq/settings.h |  5 +++++
 3 files changed, 18 insertions(+)

diff --git a/include/linux/irq.h b/include/linux/irq.h
index 6f8b340..f4ecfb9 100644
--- a/include/linux/irq.h
+++ b/include/linux/irq.h
@@ -514,6 +514,7 @@ static inline void irq_set_chip_and_handler(unsigned int irq, struct irq_chip *c
 }
 
 extern int irq_set_percpu_devid(unsigned int irq);
+extern int irq_set_move_pcntxt(unsigned int irq);
 
 extern void
 __irq_set_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained,
diff --git a/kernel/irq/irqdesc.c b/kernel/irq/irqdesc.c
index 0a2a4b6..c036a1a 100644
--- a/kernel/irq/irqdesc.c
+++ b/kernel/irq/irqdesc.c
@@ -580,6 +580,18 @@ int irq_set_percpu_devid(unsigned int irq)
 	return 0;
 }
 
+int irq_set_move_pcntxt(unsigned int irq)
+{
+	struct irq_desc *desc = irq_to_desc(irq);
+
+	if (!desc)
+		return -EINVAL;
+
+	irq_settings_set_move_pcntxt(desc);
+
+	return 0;
+}
+
 void kstat_incr_irq_this_cpu(unsigned int irq)
 {
 	kstat_incr_irqs_this_cpu(irq_to_desc(irq));
diff --git a/kernel/irq/settings.h b/kernel/irq/settings.h
index 3320b84..ea58bc2 100644
--- a/kernel/irq/settings.h
+++ b/kernel/irq/settings.h
@@ -135,6 +135,11 @@ static inline void irq_settings_set_noprobe(struct irq_desc *desc)
 	desc->status_use_accessors |= _IRQ_NOPROBE;
 }
 
+static inline void irq_settings_set_move_pcntxt(struct irq_desc *desc)
+{
+	desc->status_use_accessors |= _IRQ_MOVE_PCNTXT;
+}
+
 static inline bool irq_settings_can_move_pcntxt(struct irq_desc *desc)
 {
 	return desc->status_use_accessors & _IRQ_MOVE_PCNTXT;
-- 
2.5.0

[RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Yang Yingliang]

Use irq_settings_set_move_pcntxt() helper irqs status with
_IRQ_MOVE_PCNTXT. So that it can do set affinity when calling
irq_set_affinity_locked().

Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Cc: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
---
 drivers/irqchip/irq-gic-v3.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index e406bc5..9108387 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -688,6 +688,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
 		irq_domain_set_info(d, irq, hw, &gic_chip, d->host_data,
 				    handle_fasteoi_irq, NULL, NULL);
 		set_irq_flags(irq, IRQF_VALID | IRQF_PROBE);
+		irq_set_move_pcntxt(irq);
 	}
 	/* LPIs */
 	if (hw >= 8192 && hw < GIC_ID_NR) {
@@ -696,6 +697,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
 		irq_domain_set_info(d, irq, hw, &gic_chip, d->host_data,
 				    handle_fasteoi_irq, NULL, NULL);
 		set_irq_flags(irq, IRQF_VALID);
+		irq_set_move_pcntxt(irq);
 	}
 
 	return 0;
-- 
2.5.0

[RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Yang Yingliang]

Use irq_settings_set_move_pcntxt() helper irqs status with
_IRQ_MOVE_PCNTXT. So that it can do set affinity when calling
irq_set_affinity_locked().

Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Cc: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
---
 drivers/irqchip/irq-gic-v3.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index e406bc5..9108387 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -688,6 +688,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
 		irq_domain_set_info(d, irq, hw, &gic_chip, d->host_data,
 				    handle_fasteoi_irq, NULL, NULL);
 		set_irq_flags(irq, IRQF_VALID | IRQF_PROBE);
+		irq_set_move_pcntxt(irq);
 	}
 	/* LPIs */
 	if (hw >= 8192 && hw < GIC_ID_NR) {
@@ -696,6 +697,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
 		irq_domain_set_info(d, irq, hw, &gic_chip, d->host_data,
 				    handle_fasteoi_irq, NULL, NULL);
 		set_irq_flags(irq, IRQF_VALID);
+		irq_set_move_pcntxt(irq);
 	}
 
 	return 0;
-- 
2.5.0

[RFC PATCH v1 3/4] genirq: rename config GENERIC_PENDING_IRQ to GENERIC_IRQ_MIGRATION

[Yang Yingliang]

Make the config name more general for moving other migration
interrupts code into kernel/irq/migration.c

Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Cc: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
---
 arch/arc/Kconfig               | 2 +-
 arch/hexagon/Kconfig           | 2 +-
 arch/ia64/Kconfig              | 2 +-
 arch/tile/Kconfig              | 2 +-
 arch/x86/Kconfig               | 2 +-
 arch/x86/kernel/apic/io_apic.c | 2 +-
 include/linux/irq.h            | 4 +++-
 include/linux/irqdesc.h        | 2 +-
 kernel/irq/Kconfig             | 4 ++--
 kernel/irq/Makefile            | 2 +-
 kernel/irq/irqdesc.c           | 6 +++---
 kernel/irq/manage.c            | 2 +-
 kernel/irq/proc.c              | 2 +-
 13 files changed, 18 insertions(+), 16 deletions(-)

diff --git a/arch/arc/Kconfig b/arch/arc/Kconfig
index 78c0621..5d11976 100644
--- a/arch/arc/Kconfig
+++ b/arch/arc/Kconfig
@@ -19,7 +19,7 @@ config ARC
 	select GENERIC_FIND_FIRST_BIT
 	# for now, we don't need GENERIC_IRQ_PROBE, CONFIG_GENERIC_IRQ_CHIP
 	select GENERIC_IRQ_SHOW
-	select GENERIC_PENDING_IRQ if SMP
+	select GENERIC_IRQ_MIGRATION if SMP
 	select GENERIC_SMP_IDLE_THREAD
 	select HAVE_ARCH_KGDB
 	select HAVE_ARCH_TRACEHOOK
diff --git a/arch/hexagon/Kconfig b/arch/hexagon/Kconfig
index 4dc89d1..ffee613 100644
--- a/arch/hexagon/Kconfig
+++ b/arch/hexagon/Kconfig
@@ -11,7 +11,7 @@ config HEXAGON
 	# select ARCH_WANT_OPTIONAL_GPIOLIB
 	# select ARCH_REQUIRE_GPIOLIB
 	# select HAVE_CLK
-	# select GENERIC_PENDING_IRQ if SMP
+	# select GENERIC_IRQ_MIGRATION if SMP
 	select GENERIC_ATOMIC64
 	select HAVE_PERF_EVENTS
 	# GENERIC_ALLOCATOR is used by dma_alloc_coherent()
diff --git a/arch/ia64/Kconfig b/arch/ia64/Kconfig
index 42a91a7..6e7fb9b 100644
--- a/arch/ia64/Kconfig
+++ b/arch/ia64/Kconfig
@@ -36,7 +36,7 @@ config IA64
 	select VIRT_TO_BUS
 	select ARCH_DISCARD_MEMBLOCK
 	select GENERIC_IRQ_PROBE
-	select GENERIC_PENDING_IRQ if SMP
+	select GENERIC_IRQ_MIGRATION if SMP
 	select GENERIC_IRQ_SHOW
 	select GENERIC_IRQ_LEGACY
 	select ARCH_WANT_OPTIONAL_GPIOLIB
diff --git a/arch/tile/Kconfig b/arch/tile/Kconfig
index 9def1f5..cb47e6c 100644
--- a/arch/tile/Kconfig
+++ b/arch/tile/Kconfig
@@ -13,7 +13,7 @@ config TILE
 	select CC_OPTIMIZE_FOR_SIZE
 	select HAVE_DEBUG_KMEMLEAK
 	select GENERIC_IRQ_PROBE
-	select GENERIC_PENDING_IRQ if SMP
+	select GENERIC_IRQ_MIGRATION if SMP
 	select GENERIC_IRQ_SHOW
 	select HAVE_DEBUG_BUGVERBOSE
 	select VIRT_TO_BUS
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 48f7433..c869f75 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -65,7 +65,7 @@ config X86
 	select GENERIC_IOMAP
 	select GENERIC_IRQ_PROBE
 	select GENERIC_IRQ_SHOW
-	select GENERIC_PENDING_IRQ		if SMP
+	select GENERIC_IRQ_MIGRATION		if SMP
 	select GENERIC_SMP_IDLE_THREAD
 	select GENERIC_STRNCPY_FROM_USER
 	select GENERIC_STRNLEN_USER
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 38a76f8..a1203d5 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -1684,7 +1684,7 @@ static unsigned int startup_ioapic_irq(struct irq_data *data)
 
 atomic_t irq_mis_count;
 
-#ifdef CONFIG_GENERIC_PENDING_IRQ
+#ifdef CONFIG_GENERIC_IRQ_MIGRATION
 static bool io_apic_level_ack_pending(struct mp_chip_data *data)
 {
 	struct irq_pin_list *entry;
diff --git a/include/linux/irq.h b/include/linux/irq.h
index f4ecfb9..3461809 100644
--- a/include/linux/irq.h
+++ b/include/linux/irq.h
@@ -438,12 +438,14 @@ extern int irq_set_affinity_locked(struct irq_data *data,
 				   const struct cpumask *cpumask, bool force);
 extern int irq_set_vcpu_affinity(unsigned int irq, void *vcpu_info);
 
-#if defined(CONFIG_SMP) && defined(CONFIG_GENERIC_PENDING_IRQ)
+#if defined(CONFIG_SMP) && defined(CONFIG_GENERIC_IRQ_MIGRATION)
 void irq_move_irq(struct irq_data *data);
 void irq_move_masked_irq(struct irq_data *data);
+void migrate_irqs(void);
 #else
 static inline void irq_move_irq(struct irq_data *data) { }
 static inline void irq_move_masked_irq(struct irq_data *data) { }
+static inline void migrate_irqs(void) { }
 #endif
 
 extern int no_irq_affinity;
diff --git a/include/linux/irqdesc.h b/include/linux/irqdesc.h
index 5acfa26..1ec727d 100644
--- a/include/linux/irqdesc.h
+++ b/include/linux/irqdesc.h
@@ -66,7 +66,7 @@ struct irq_desc {
 #ifdef CONFIG_SMP
 	const struct cpumask	*affinity_hint;
 	struct irq_affinity_notify *affinity_notify;
-#ifdef CONFIG_GENERIC_PENDING_IRQ
+#ifdef CONFIG_GENERIC_IRQ_MIGRATION
 	cpumask_var_t		pending_mask;
 #endif
 #endif
diff --git a/kernel/irq/Kconfig b/kernel/irq/Kconfig
index 9a76e3b..4350358 100644
--- a/kernel/irq/Kconfig
+++ b/kernel/irq/Kconfig
@@ -26,8 +26,8 @@ config GENERIC_IRQ_SHOW_LEVEL
 config GENERIC_IRQ_LEGACY_ALLOC_HWIRQ
        bool
 
-# Support for delayed migration from interrupt context
-config GENERIC_PENDING_IRQ
+# Support for generic irq migration
+config GENERIC_IRQ_MIGRATION
 	bool
 
 # Alpha specific irq affinity mechanism
diff --git a/kernel/irq/Makefile b/kernel/irq/Makefile
index d121235..bdd31b7 100644
--- a/kernel/irq/Makefile
+++ b/kernel/irq/Makefile
@@ -4,6 +4,6 @@ obj-$(CONFIG_GENERIC_IRQ_CHIP) += generic-chip.o
 obj-$(CONFIG_GENERIC_IRQ_PROBE) += autoprobe.o
 obj-$(CONFIG_IRQ_DOMAIN) += irqdomain.o
 obj-$(CONFIG_PROC_FS) += proc.o
-obj-$(CONFIG_GENERIC_PENDING_IRQ) += migration.o
+obj-$(CONFIG_GENERIC_IRQ_MIGRATION) += migration.o
 obj-$(CONFIG_PM_SLEEP) += pm.o
 obj-$(CONFIG_GENERIC_MSI_IRQ) += msi.o
diff --git a/kernel/irq/irqdesc.c b/kernel/irq/irqdesc.c
index c036a1a..f5a8930 100644
--- a/kernel/irq/irqdesc.c
+++ b/kernel/irq/irqdesc.c
@@ -41,7 +41,7 @@ static int alloc_masks(struct irq_desc *desc, gfp_t gfp, int node)
 	if (!zalloc_cpumask_var_node(&desc->irq_data.affinity, gfp, node))
 		return -ENOMEM;
 
-#ifdef CONFIG_GENERIC_PENDING_IRQ
+#ifdef CONFIG_GENERIC_IRQ_MIGRATION
 	if (!zalloc_cpumask_var_node(&desc->pending_mask, gfp, node)) {
 		free_cpumask_var(desc->irq_data.affinity);
 		return -ENOMEM;
@@ -54,7 +54,7 @@ static void desc_smp_init(struct irq_desc *desc, int node)
 {
 	desc->irq_data.node = node;
 	cpumask_copy(desc->irq_data.affinity, irq_default_affinity);
-#ifdef CONFIG_GENERIC_PENDING_IRQ
+#ifdef CONFIG_GENERIC_IRQ_MIGRATION
 	cpumask_clear(desc->pending_mask);
 #endif
 }
@@ -118,7 +118,7 @@ static void delete_irq_desc(unsigned int irq)
 #ifdef CONFIG_SMP
 static void free_masks(struct irq_desc *desc)
 {
-#ifdef CONFIG_GENERIC_PENDING_IRQ
+#ifdef CONFIG_GENERIC_IRQ_MIGRATION
 	free_cpumask_var(desc->pending_mask);
 #endif
 	free_cpumask_var(desc->irq_data.affinity);
diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
index ad1b064..6207deb 100644
--- a/kernel/irq/manage.c
+++ b/kernel/irq/manage.c
@@ -153,7 +153,7 @@ void irq_set_thread_affinity(struct irq_desc *desc)
 	}
 }
 
-#ifdef CONFIG_GENERIC_PENDING_IRQ
+#ifdef CONFIG_GENERIC_IRQ_MIGRATION
 static inline bool irq_can_move_pcntxt(struct irq_data *data)
 {
 	return irqd_can_move_in_process_context(data);
diff --git a/kernel/irq/proc.c b/kernel/irq/proc.c
index 0e97c14..c070fa6 100644
--- a/kernel/irq/proc.c
+++ b/kernel/irq/proc.c
@@ -41,7 +41,7 @@ static int show_irq_affinity(int type, struct seq_file *m, void *v)
 	struct irq_desc *desc = irq_to_desc((long)m->private);
 	const struct cpumask *mask = desc->irq_data.affinity;
 
-#ifdef CONFIG_GENERIC_PENDING_IRQ
+#ifdef CONFIG_GENERIC_IRQ_MIGRATION
 	if (irqd_is_setaffinity_pending(&desc->irq_data))
 		mask = desc->pending_mask;
 #endif
-- 
2.5.0

[RFC PATCH v1 3/4] genirq: rename config GENERIC_PENDING_IRQ to GENERIC_IRQ_MIGRATION

[Yang Yingliang]

Make the config name more general for moving other migration
interrupts code into kernel/irq/migration.c

Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Cc: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
---
 arch/arc/Kconfig               | 2 +-
 arch/hexagon/Kconfig           | 2 +-
 arch/ia64/Kconfig              | 2 +-
 arch/tile/Kconfig              | 2 +-
 arch/x86/Kconfig               | 2 +-
 arch/x86/kernel/apic/io_apic.c | 2 +-
 include/linux/irq.h            | 4 +++-
 include/linux/irqdesc.h        | 2 +-
 kernel/irq/Kconfig             | 4 ++--
 kernel/irq/Makefile            | 2 +-
 kernel/irq/irqdesc.c           | 6 +++---
 kernel/irq/manage.c            | 2 +-
 kernel/irq/proc.c              | 2 +-
 13 files changed, 18 insertions(+), 16 deletions(-)

diff --git a/arch/arc/Kconfig b/arch/arc/Kconfig
index 78c0621..5d11976 100644
--- a/arch/arc/Kconfig
+++ b/arch/arc/Kconfig
@@ -19,7 +19,7 @@ config ARC
 	select GENERIC_FIND_FIRST_BIT
 	# for now, we don't need GENERIC_IRQ_PROBE, CONFIG_GENERIC_IRQ_CHIP
 	select GENERIC_IRQ_SHOW
-	select GENERIC_PENDING_IRQ if SMP
+	select GENERIC_IRQ_MIGRATION if SMP
 	select GENERIC_SMP_IDLE_THREAD
 	select HAVE_ARCH_KGDB
 	select HAVE_ARCH_TRACEHOOK
diff --git a/arch/hexagon/Kconfig b/arch/hexagon/Kconfig
index 4dc89d1..ffee613 100644
--- a/arch/hexagon/Kconfig
+++ b/arch/hexagon/Kconfig
@@ -11,7 +11,7 @@ config HEXAGON
 	# select ARCH_WANT_OPTIONAL_GPIOLIB
 	# select ARCH_REQUIRE_GPIOLIB
 	# select HAVE_CLK
-	# select GENERIC_PENDING_IRQ if SMP
+	# select GENERIC_IRQ_MIGRATION if SMP
 	select GENERIC_ATOMIC64
 	select HAVE_PERF_EVENTS
 	# GENERIC_ALLOCATOR is used by dma_alloc_coherent()
diff --git a/arch/ia64/Kconfig b/arch/ia64/Kconfig
index 42a91a7..6e7fb9b 100644
--- a/arch/ia64/Kconfig
+++ b/arch/ia64/Kconfig
@@ -36,7 +36,7 @@ config IA64
 	select VIRT_TO_BUS
 	select ARCH_DISCARD_MEMBLOCK
 	select GENERIC_IRQ_PROBE
-	select GENERIC_PENDING_IRQ if SMP
+	select GENERIC_IRQ_MIGRATION if SMP
 	select GENERIC_IRQ_SHOW
 	select GENERIC_IRQ_LEGACY
 	select ARCH_WANT_OPTIONAL_GPIOLIB
diff --git a/arch/tile/Kconfig b/arch/tile/Kconfig
index 9def1f5..cb47e6c 100644
--- a/arch/tile/Kconfig
+++ b/arch/tile/Kconfig
@@ -13,7 +13,7 @@ config TILE
 	select CC_OPTIMIZE_FOR_SIZE
 	select HAVE_DEBUG_KMEMLEAK
 	select GENERIC_IRQ_PROBE
-	select GENERIC_PENDING_IRQ if SMP
+	select GENERIC_IRQ_MIGRATION if SMP
 	select GENERIC_IRQ_SHOW
 	select HAVE_DEBUG_BUGVERBOSE
 	select VIRT_TO_BUS
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 48f7433..c869f75 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -65,7 +65,7 @@ config X86
 	select GENERIC_IOMAP
 	select GENERIC_IRQ_PROBE
 	select GENERIC_IRQ_SHOW
-	select GENERIC_PENDING_IRQ		if SMP
+	select GENERIC_IRQ_MIGRATION		if SMP
 	select GENERIC_SMP_IDLE_THREAD
 	select GENERIC_STRNCPY_FROM_USER
 	select GENERIC_STRNLEN_USER
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 38a76f8..a1203d5 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -1684,7 +1684,7 @@ static unsigned int startup_ioapic_irq(struct irq_data *data)
 
 atomic_t irq_mis_count;
 
-#ifdef CONFIG_GENERIC_PENDING_IRQ
+#ifdef CONFIG_GENERIC_IRQ_MIGRATION
 static bool io_apic_level_ack_pending(struct mp_chip_data *data)
 {
 	struct irq_pin_list *entry;
diff --git a/include/linux/irq.h b/include/linux/irq.h
index f4ecfb9..3461809 100644
--- a/include/linux/irq.h
+++ b/include/linux/irq.h
@@ -438,12 +438,14 @@ extern int irq_set_affinity_locked(struct irq_data *data,
 				   const struct cpumask *cpumask, bool force);
 extern int irq_set_vcpu_affinity(unsigned int irq, void *vcpu_info);
 
-#if defined(CONFIG_SMP) && defined(CONFIG_GENERIC_PENDING_IRQ)
+#if defined(CONFIG_SMP) && defined(CONFIG_GENERIC_IRQ_MIGRATION)
 void irq_move_irq(struct irq_data *data);
 void irq_move_masked_irq(struct irq_data *data);
+void migrate_irqs(void);
 #else
 static inline void irq_move_irq(struct irq_data *data) { }
 static inline void irq_move_masked_irq(struct irq_data *data) { }
+static inline void migrate_irqs(void) { }
 #endif
 
 extern int no_irq_affinity;
diff --git a/include/linux/irqdesc.h b/include/linux/irqdesc.h
index 5acfa26..1ec727d 100644
--- a/include/linux/irqdesc.h
+++ b/include/linux/irqdesc.h
@@ -66,7 +66,7 @@ struct irq_desc {
 #ifdef CONFIG_SMP
 	const struct cpumask	*affinity_hint;
 	struct irq_affinity_notify *affinity_notify;
-#ifdef CONFIG_GENERIC_PENDING_IRQ
+#ifdef CONFIG_GENERIC_IRQ_MIGRATION
 	cpumask_var_t		pending_mask;
 #endif
 #endif
diff --git a/kernel/irq/Kconfig b/kernel/irq/Kconfig
index 9a76e3b..4350358 100644
--- a/kernel/irq/Kconfig
+++ b/kernel/irq/Kconfig
@@ -26,8 +26,8 @@ config GENERIC_IRQ_SHOW_LEVEL
 config GENERIC_IRQ_LEGACY_ALLOC_HWIRQ
        bool
 
-# Support for delayed migration from interrupt context
-config GENERIC_PENDING_IRQ
+# Support for generic irq migration
+config GENERIC_IRQ_MIGRATION
 	bool
 
 # Alpha specific irq affinity mechanism
diff --git a/kernel/irq/Makefile b/kernel/irq/Makefile
index d121235..bdd31b7 100644
--- a/kernel/irq/Makefile
+++ b/kernel/irq/Makefile
@@ -4,6 +4,6 @@ obj-$(CONFIG_GENERIC_IRQ_CHIP) += generic-chip.o
 obj-$(CONFIG_GENERIC_IRQ_PROBE) += autoprobe.o
 obj-$(CONFIG_IRQ_DOMAIN) += irqdomain.o
 obj-$(CONFIG_PROC_FS) += proc.o
-obj-$(CONFIG_GENERIC_PENDING_IRQ) += migration.o
+obj-$(CONFIG_GENERIC_IRQ_MIGRATION) += migration.o
 obj-$(CONFIG_PM_SLEEP) += pm.o
 obj-$(CONFIG_GENERIC_MSI_IRQ) += msi.o
diff --git a/kernel/irq/irqdesc.c b/kernel/irq/irqdesc.c
index c036a1a..f5a8930 100644
--- a/kernel/irq/irqdesc.c
+++ b/kernel/irq/irqdesc.c
@@ -41,7 +41,7 @@ static int alloc_masks(struct irq_desc *desc, gfp_t gfp, int node)
 	if (!zalloc_cpumask_var_node(&desc->irq_data.affinity, gfp, node))
 		return -ENOMEM;
 
-#ifdef CONFIG_GENERIC_PENDING_IRQ
+#ifdef CONFIG_GENERIC_IRQ_MIGRATION
 	if (!zalloc_cpumask_var_node(&desc->pending_mask, gfp, node)) {
 		free_cpumask_var(desc->irq_data.affinity);
 		return -ENOMEM;
@@ -54,7 +54,7 @@ static void desc_smp_init(struct irq_desc *desc, int node)
 {
 	desc->irq_data.node = node;
 	cpumask_copy(desc->irq_data.affinity, irq_default_affinity);
-#ifdef CONFIG_GENERIC_PENDING_IRQ
+#ifdef CONFIG_GENERIC_IRQ_MIGRATION
 	cpumask_clear(desc->pending_mask);
 #endif
 }
@@ -118,7 +118,7 @@ static void delete_irq_desc(unsigned int irq)
 #ifdef CONFIG_SMP
 static void free_masks(struct irq_desc *desc)
 {
-#ifdef CONFIG_GENERIC_PENDING_IRQ
+#ifdef CONFIG_GENERIC_IRQ_MIGRATION
 	free_cpumask_var(desc->pending_mask);
 #endif
 	free_cpumask_var(desc->irq_data.affinity);
diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
index ad1b064..6207deb 100644
--- a/kernel/irq/manage.c
+++ b/kernel/irq/manage.c
@@ -153,7 +153,7 @@ void irq_set_thread_affinity(struct irq_desc *desc)
 	}
 }
 
-#ifdef CONFIG_GENERIC_PENDING_IRQ
+#ifdef CONFIG_GENERIC_IRQ_MIGRATION
 static inline bool irq_can_move_pcntxt(struct irq_data *data)
 {
 	return irqd_can_move_in_process_context(data);
diff --git a/kernel/irq/proc.c b/kernel/irq/proc.c
index 0e97c14..c070fa6 100644
--- a/kernel/irq/proc.c
+++ b/kernel/irq/proc.c
@@ -41,7 +41,7 @@ static int show_irq_affinity(int type, struct seq_file *m, void *v)
 	struct irq_desc *desc = irq_to_desc((long)m->private);
 	const struct cpumask *mask = desc->irq_data.affinity;
 
-#ifdef CONFIG_GENERIC_PENDING_IRQ
+#ifdef CONFIG_GENERIC_IRQ_MIGRATION
 	if (irqd_is_setaffinity_pending(&desc->irq_data))
 		mask = desc->pending_mask;
 #endif
-- 
2.5.0

[RFC PATCH v1 4/4] arm/arm64: fix a migrating irq bug when hotplug cpu

[Yang Yingliang]

When cpu is disabled, all irqs will be migratged to another cpu.
In some cases, a new affinity is different, it needed to be coppied
to irq's affinity. But if the type of irq is LPI, it's affinity will
not be coppied because of irq_set_affinity's return value. Fix it by
using irq_do_set_affinity.

And migrating interrupts is a core code matter, so move the code to
kernel/irq/migration.c and select CONFIG_GENERIC_IRQ_MIGRATION when
CONFIG_HOTPLUG_CPU and CONFIG_SMP is enabled.

Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Cc: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
---
 arch/arm/Kconfig             |  1 +
 arch/arm/include/asm/irq.h   |  1 -
 arch/arm/kernel/irq.c        | 62 --------------------------------------------
 arch/arm64/Kconfig           |  1 +
 arch/arm64/include/asm/irq.h |  1 -
 arch/arm64/kernel/irq.c      | 62 --------------------------------------------
 kernel/irq/migration.c       | 62 ++++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 64 insertions(+), 126 deletions(-)

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 41cbb4a..ebc8a33 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -22,6 +22,7 @@ config ARM
 	select GENERIC_CLOCKEVENTS_BROADCAST if SMP
 	select GENERIC_IDLE_POLL_SETUP
 	select GENERIC_IRQ_PROBE
+	select GENERIC_IRQ_MIGRATION if SMP && HOTPLUG_CPU
 	select GENERIC_IRQ_SHOW
 	select GENERIC_IRQ_SHOW_LEVEL
 	select GENERIC_PCI_IOMAP
diff --git a/arch/arm/include/asm/irq.h b/arch/arm/include/asm/irq.h
index 53c15de..d17fc900 100644
--- a/arch/arm/include/asm/irq.h
+++ b/arch/arm/include/asm/irq.h
@@ -24,7 +24,6 @@
 #ifndef __ASSEMBLY__
 struct irqaction;
 struct pt_regs;
-extern void migrate_irqs(void);
 
 extern void asm_do_IRQ(unsigned int, struct pt_regs *);
 void handle_IRQ(unsigned int, struct pt_regs *);
diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c
index baf8ede..2efdb40 100644
--- a/arch/arm/kernel/irq.c
+++ b/arch/arm/kernel/irq.c
@@ -31,7 +31,6 @@
 #include <linux/smp.h>
 #include <linux/init.h>
 #include <linux/seq_file.h>
-#include <linux/ratelimit.h>
 #include <linux/errno.h>
 #include <linux/list.h>
 #include <linux/kallsyms.h>
@@ -135,64 +134,3 @@ int __init arch_probe_nr_irqs(void)
 	return nr_irqs;
 }
 #endif
-
-#ifdef CONFIG_HOTPLUG_CPU
-static bool migrate_one_irq(struct irq_desc *desc)
-{
-	struct irq_data *d = irq_desc_get_irq_data(desc);
-	const struct cpumask *affinity = irq_data_get_affinity_mask(d);
-	struct irq_chip *c;
-	bool ret = false;
-
-	/*
-	 * If this is a per-CPU interrupt, or the affinity does not
-	 * include this CPU, then we have nothing to do.
-	 */
-	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
-		return false;
-
-	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
-		affinity = cpu_online_mask;
-		ret = true;
-	}
-
-	c = irq_data_get_irq_chip(d);
-	if (!c->irq_set_affinity)
-		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
-	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
-		cpumask_copy(irq_data_get_affinity_mask(d), affinity);
-
-	return ret;
-}
-
-/*
- * The current CPU has been marked offline.  Migrate IRQs off this CPU.
- * If the affinity settings do not allow other CPUs, force them onto any
- * available CPU.
- *
- * Note: we must iterate over all IRQs, whether they have an attached
- * action structure or not, as we need to get chained interrupts too.
- */
-void migrate_irqs(void)
-{
-	unsigned int i;
-	struct irq_desc *desc;
-	unsigned long flags;
-
-	local_irq_save(flags);
-
-	for_each_irq_desc(i, desc) {
-		bool affinity_broken;
-
-		raw_spin_lock(&desc->lock);
-		affinity_broken = migrate_one_irq(desc);
-		raw_spin_unlock(&desc->lock);
-
-		if (affinity_broken)
-			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
-				i, smp_processor_id());
-	}
-
-	local_irq_restore(flags);
-}
-#endif /* CONFIG_HOTPLUG_CPU */
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b7b9cea..6ffe411 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -32,6 +32,7 @@ config ARM64
 	select GENERIC_CPU_AUTOPROBE
 	select GENERIC_EARLY_IOREMAP
 	select GENERIC_IRQ_PROBE
+	select GENERIC_IRQ_MIGRATION if SMP && HOTPLUG_CPU
 	select GENERIC_IRQ_SHOW
 	select GENERIC_IRQ_SHOW_LEVEL
 	select GENERIC_PCI_IOMAP
diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h
index bbb251b..0916929 100644
--- a/arch/arm64/include/asm/irq.h
+++ b/arch/arm64/include/asm/irq.h
@@ -7,7 +7,6 @@
 
 struct pt_regs;
 
-extern void migrate_irqs(void);
 extern void set_handle_irq(void (*handle_irq)(struct pt_regs *));
 
 static inline void acpi_irq_init(void)
diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
index 463fa2e..04ac1f6 100644
--- a/arch/arm64/kernel/irq.c
+++ b/arch/arm64/kernel/irq.c
@@ -27,7 +27,6 @@
 #include <linux/init.h>
 #include <linux/irqchip.h>
 #include <linux/seq_file.h>
-#include <linux/ratelimit.h>
 
 unsigned long irq_err_count;
 
@@ -56,64 +55,3 @@ void __init init_IRQ(void)
 	if (!handle_arch_irq)
 		panic("No interrupt controller found.");
 }
-
-#ifdef CONFIG_HOTPLUG_CPU
-static bool migrate_one_irq(struct irq_desc *desc)
-{
-	struct irq_data *d = irq_desc_get_irq_data(desc);
-	const struct cpumask *affinity = irq_data_get_affinity_mask(d);
-	struct irq_chip *c;
-	bool ret = false;
-
-	/*
-	 * If this is a per-CPU interrupt, or the affinity does not
-	 * include this CPU, then we have nothing to do.
-	 */
-	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
-		return false;
-
-	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
-		affinity = cpu_online_mask;
-		ret = true;
-	}
-
-	c = irq_data_get_irq_chip(d);
-	if (!c->irq_set_affinity)
-		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
-	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
-		cpumask_copy(irq_data_get_affinity_mask(d), affinity);
-
-	return ret;
-}
-
-/*
- * The current CPU has been marked offline.  Migrate IRQs off this CPU.
- * If the affinity settings do not allow other CPUs, force them onto any
- * available CPU.
- *
- * Note: we must iterate over all IRQs, whether they have an attached
- * action structure or not, as we need to get chained interrupts too.
- */
-void migrate_irqs(void)
-{
-	unsigned int i;
-	struct irq_desc *desc;
-	unsigned long flags;
-
-	local_irq_save(flags);
-
-	for_each_irq_desc(i, desc) {
-		bool affinity_broken;
-
-		raw_spin_lock(&desc->lock);
-		affinity_broken = migrate_one_irq(desc);
-		raw_spin_unlock(&desc->lock);
-
-		if (affinity_broken)
-			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
-					    i, smp_processor_id());
-	}
-
-	local_irq_restore(flags);
-}
-#endif /* CONFIG_HOTPLUG_CPU */
diff --git a/kernel/irq/migration.c b/kernel/irq/migration.c
index 37ddb7b..5801c79 100644
--- a/kernel/irq/migration.c
+++ b/kernel/irq/migration.c
@@ -1,6 +1,7 @@
 
 #include <linux/irq.h>
 #include <linux/interrupt.h>
+#include <linux/ratelimit.h>
 
 #include "internals.h"
 
@@ -77,3 +78,64 @@ void irq_move_irq(struct irq_data *idata)
 	if (!masked)
 		idata->chip->irq_unmask(idata);
 }
+
+#ifdef CONFIG_HOTPLUG_CPU
+static bool migrate_one_irq(struct irq_desc *desc)
+{
+	struct irq_data *d = irq_desc_get_irq_data(desc);
+	const struct cpumask *affinity = d->affinity;
+	struct irq_chip *c;
+	bool ret = false;
+
+	/*
+	 * If this is a per-CPU interrupt, or the affinity does not
+	 * include this CPU, then we have nothing to do.
+	 */
+	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
+		return false;
+
+	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
+		affinity = cpu_online_mask;
+		ret = true;
+	}
+
+	c = irq_data_get_irq_chip(d);
+	if (!c->irq_set_affinity)
+		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
+	else
+		irq_do_set_affinity(d, affinity, false);
+
+	return ret;
+}
+
+/*
+ * The current CPU has been marked offline.  Migrate IRQs off this CPU.
+ * If the affinity settings do not allow other CPUs, force them onto any
+ * available CPU.
+ *
+ * Note: we must iterate over all IRQs, whether they have an attached
+ * action structure or not, as we need to get chained interrupts too.
+ */
+void migrate_irqs(void)
+{
+	unsigned int i;
+	struct irq_desc *desc;
+	unsigned long flags;
+
+	local_irq_save(flags);
+
+	for_each_irq_desc(i, desc) {
+		bool affinity_broken;
+
+		raw_spin_lock(&desc->lock);
+		affinity_broken = migrate_one_irq(desc);
+		raw_spin_unlock(&desc->lock);
+
+		if (affinity_broken)
+			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
+					    i, smp_processor_id());
+	}
+
+	local_irq_restore(flags);
+}
+#endif /* CONFIG_HOTPLUG_CPU */
-- 
2.5.0

[RFC PATCH v1 4/4] arm/arm64: fix a migrating irq bug when hotplug cpu

[Yang Yingliang]

When cpu is disabled, all irqs will be migratged to another cpu.
In some cases, a new affinity is different, it needed to be coppied
to irq's affinity. But if the type of irq is LPI, it's affinity will
not be coppied because of irq_set_affinity's return value. Fix it by
using irq_do_set_affinity.

And migrating interrupts is a core code matter, so move the code to
kernel/irq/migration.c and select CONFIG_GENERIC_IRQ_MIGRATION when
CONFIG_HOTPLUG_CPU and CONFIG_SMP is enabled.

Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Cc: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
---
 arch/arm/Kconfig             |  1 +
 arch/arm/include/asm/irq.h   |  1 -
 arch/arm/kernel/irq.c        | 62 --------------------------------------------
 arch/arm64/Kconfig           |  1 +
 arch/arm64/include/asm/irq.h |  1 -
 arch/arm64/kernel/irq.c      | 62 --------------------------------------------
 kernel/irq/migration.c       | 62 ++++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 64 insertions(+), 126 deletions(-)

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 41cbb4a..ebc8a33 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -22,6 +22,7 @@ config ARM
 	select GENERIC_CLOCKEVENTS_BROADCAST if SMP
 	select GENERIC_IDLE_POLL_SETUP
 	select GENERIC_IRQ_PROBE
+	select GENERIC_IRQ_MIGRATION if SMP && HOTPLUG_CPU
 	select GENERIC_IRQ_SHOW
 	select GENERIC_IRQ_SHOW_LEVEL
 	select GENERIC_PCI_IOMAP
diff --git a/arch/arm/include/asm/irq.h b/arch/arm/include/asm/irq.h
index 53c15de..d17fc900 100644
--- a/arch/arm/include/asm/irq.h
+++ b/arch/arm/include/asm/irq.h
@@ -24,7 +24,6 @@
 #ifndef __ASSEMBLY__
 struct irqaction;
 struct pt_regs;
-extern void migrate_irqs(void);
 
 extern void asm_do_IRQ(unsigned int, struct pt_regs *);
 void handle_IRQ(unsigned int, struct pt_regs *);
diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c
index baf8ede..2efdb40 100644
--- a/arch/arm/kernel/irq.c
+++ b/arch/arm/kernel/irq.c
@@ -31,7 +31,6 @@
 #include <linux/smp.h>
 #include <linux/init.h>
 #include <linux/seq_file.h>
-#include <linux/ratelimit.h>
 #include <linux/errno.h>
 #include <linux/list.h>
 #include <linux/kallsyms.h>
@@ -135,64 +134,3 @@ int __init arch_probe_nr_irqs(void)
 	return nr_irqs;
 }
 #endif
-
-#ifdef CONFIG_HOTPLUG_CPU
-static bool migrate_one_irq(struct irq_desc *desc)
-{
-	struct irq_data *d = irq_desc_get_irq_data(desc);
-	const struct cpumask *affinity = irq_data_get_affinity_mask(d);
-	struct irq_chip *c;
-	bool ret = false;
-
-	/*
-	 * If this is a per-CPU interrupt, or the affinity does not
-	 * include this CPU, then we have nothing to do.
-	 */
-	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
-		return false;
-
-	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
-		affinity = cpu_online_mask;
-		ret = true;
-	}
-
-	c = irq_data_get_irq_chip(d);
-	if (!c->irq_set_affinity)
-		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
-	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
-		cpumask_copy(irq_data_get_affinity_mask(d), affinity);
-
-	return ret;
-}
-
-/*
- * The current CPU has been marked offline.  Migrate IRQs off this CPU.
- * If the affinity settings do not allow other CPUs, force them onto any
- * available CPU.
- *
- * Note: we must iterate over all IRQs, whether they have an attached
- * action structure or not, as we need to get chained interrupts too.
- */
-void migrate_irqs(void)
-{
-	unsigned int i;
-	struct irq_desc *desc;
-	unsigned long flags;
-
-	local_irq_save(flags);
-
-	for_each_irq_desc(i, desc) {
-		bool affinity_broken;
-
-		raw_spin_lock(&desc->lock);
-		affinity_broken = migrate_one_irq(desc);
-		raw_spin_unlock(&desc->lock);
-
-		if (affinity_broken)
-			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
-				i, smp_processor_id());
-	}
-
-	local_irq_restore(flags);
-}
-#endif /* CONFIG_HOTPLUG_CPU */
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b7b9cea..6ffe411 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -32,6 +32,7 @@ config ARM64
 	select GENERIC_CPU_AUTOPROBE
 	select GENERIC_EARLY_IOREMAP
 	select GENERIC_IRQ_PROBE
+	select GENERIC_IRQ_MIGRATION if SMP && HOTPLUG_CPU
 	select GENERIC_IRQ_SHOW
 	select GENERIC_IRQ_SHOW_LEVEL
 	select GENERIC_PCI_IOMAP
diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h
index bbb251b..0916929 100644
--- a/arch/arm64/include/asm/irq.h
+++ b/arch/arm64/include/asm/irq.h
@@ -7,7 +7,6 @@
 
 struct pt_regs;
 
-extern void migrate_irqs(void);
 extern void set_handle_irq(void (*handle_irq)(struct pt_regs *));
 
 static inline void acpi_irq_init(void)
diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
index 463fa2e..04ac1f6 100644
--- a/arch/arm64/kernel/irq.c
+++ b/arch/arm64/kernel/irq.c
@@ -27,7 +27,6 @@
 #include <linux/init.h>
 #include <linux/irqchip.h>
 #include <linux/seq_file.h>
-#include <linux/ratelimit.h>
 
 unsigned long irq_err_count;
 
@@ -56,64 +55,3 @@ void __init init_IRQ(void)
 	if (!handle_arch_irq)
 		panic("No interrupt controller found.");
 }
-
-#ifdef CONFIG_HOTPLUG_CPU
-static bool migrate_one_irq(struct irq_desc *desc)
-{
-	struct irq_data *d = irq_desc_get_irq_data(desc);
-	const struct cpumask *affinity = irq_data_get_affinity_mask(d);
-	struct irq_chip *c;
-	bool ret = false;
-
-	/*
-	 * If this is a per-CPU interrupt, or the affinity does not
-	 * include this CPU, then we have nothing to do.
-	 */
-	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
-		return false;
-
-	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
-		affinity = cpu_online_mask;
-		ret = true;
-	}
-
-	c = irq_data_get_irq_chip(d);
-	if (!c->irq_set_affinity)
-		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
-	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
-		cpumask_copy(irq_data_get_affinity_mask(d), affinity);
-
-	return ret;
-}
-
-/*
- * The current CPU has been marked offline.  Migrate IRQs off this CPU.
- * If the affinity settings do not allow other CPUs, force them onto any
- * available CPU.
- *
- * Note: we must iterate over all IRQs, whether they have an attached
- * action structure or not, as we need to get chained interrupts too.
- */
-void migrate_irqs(void)
-{
-	unsigned int i;
-	struct irq_desc *desc;
-	unsigned long flags;
-
-	local_irq_save(flags);
-
-	for_each_irq_desc(i, desc) {
-		bool affinity_broken;
-
-		raw_spin_lock(&desc->lock);
-		affinity_broken = migrate_one_irq(desc);
-		raw_spin_unlock(&desc->lock);
-
-		if (affinity_broken)
-			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
-					    i, smp_processor_id());
-	}
-
-	local_irq_restore(flags);
-}
-#endif /* CONFIG_HOTPLUG_CPU */
diff --git a/kernel/irq/migration.c b/kernel/irq/migration.c
index 37ddb7b..5801c79 100644
--- a/kernel/irq/migration.c
+++ b/kernel/irq/migration.c
@@ -1,6 +1,7 @@
 
 #include <linux/irq.h>
 #include <linux/interrupt.h>
+#include <linux/ratelimit.h>
 
 #include "internals.h"
 
@@ -77,3 +78,64 @@ void irq_move_irq(struct irq_data *idata)
 	if (!masked)
 		idata->chip->irq_unmask(idata);
 }
+
+#ifdef CONFIG_HOTPLUG_CPU
+static bool migrate_one_irq(struct irq_desc *desc)
+{
+	struct irq_data *d = irq_desc_get_irq_data(desc);
+	const struct cpumask *affinity = d->affinity;
+	struct irq_chip *c;
+	bool ret = false;
+
+	/*
+	 * If this is a per-CPU interrupt, or the affinity does not
+	 * include this CPU, then we have nothing to do.
+	 */
+	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
+		return false;
+
+	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
+		affinity = cpu_online_mask;
+		ret = true;
+	}
+
+	c = irq_data_get_irq_chip(d);
+	if (!c->irq_set_affinity)
+		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
+	else
+		irq_do_set_affinity(d, affinity, false);
+
+	return ret;
+}
+
+/*
+ * The current CPU has been marked offline.  Migrate IRQs off this CPU.
+ * If the affinity settings do not allow other CPUs, force them onto any
+ * available CPU.
+ *
+ * Note: we must iterate over all IRQs, whether they have an attached
+ * action structure or not, as we need to get chained interrupts too.
+ */
+void migrate_irqs(void)
+{
+	unsigned int i;
+	struct irq_desc *desc;
+	unsigned long flags;
+
+	local_irq_save(flags);
+
+	for_each_irq_desc(i, desc) {
+		bool affinity_broken;
+
+		raw_spin_lock(&desc->lock);
+		affinity_broken = migrate_one_irq(desc);
+		raw_spin_unlock(&desc->lock);
+
+		if (affinity_broken)
+			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
+					    i, smp_processor_id());
+	}
+
+	local_irq_restore(flags);
+}
+#endif /* CONFIG_HOTPLUG_CPU */
-- 
2.5.0

Re: [RFC PATCH v1 4/4] arm/arm64: fix a migrating irq bug when hotplug cpu

[Jiang Liu]

On 2015/9/6 12:23, Yang Yingliang wrote:
> When cpu is disabled, all irqs will be migratged to another cpu.
> In some cases, a new affinity is different, it needed to be coppied
> to irq's affinity. But if the type of irq is LPI, it's affinity will
> not be coppied because of irq_set_affinity's return value. Fix it by
> using irq_do_set_affinity.
> 
> And migrating interrupts is a core code matter, so move the code to
> kernel/irq/migration.c and select CONFIG_GENERIC_IRQ_MIGRATION when
> CONFIG_HOTPLUG_CPU and CONFIG_SMP is enabled.
> 
> Cc: Jiang Liu <jiang.liu@linux.intel.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Marc Zyngier <marc.zyngier@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
> Cc: Hanjun Guo <hanjun.guo@linaro.org>
> Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
> ---
>  arch/arm/Kconfig             |  1 +
>  arch/arm/include/asm/irq.h   |  1 -
>  arch/arm/kernel/irq.c        | 62 --------------------------------------------
>  arch/arm64/Kconfig           |  1 +
>  arch/arm64/include/asm/irq.h |  1 -
>  arch/arm64/kernel/irq.c      | 62 --------------------------------------------
>  kernel/irq/migration.c       | 62 ++++++++++++++++++++++++++++++++++++++++++++
>  7 files changed, 64 insertions(+), 126 deletions(-)
> 
> diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
> index 41cbb4a..ebc8a33 100644
> --- a/arch/arm/Kconfig
> +++ b/arch/arm/Kconfig
> @@ -22,6 +22,7 @@ config ARM
>  	select GENERIC_CLOCKEVENTS_BROADCAST if SMP
>  	select GENERIC_IDLE_POLL_SETUP
>  	select GENERIC_IRQ_PROBE
> +	select GENERIC_IRQ_MIGRATION if SMP && HOTPLUG_CPU
>  	select GENERIC_IRQ_SHOW
>  	select GENERIC_IRQ_SHOW_LEVEL
>  	select GENERIC_PCI_IOMAP
> diff --git a/arch/arm/include/asm/irq.h b/arch/arm/include/asm/irq.h
> index 53c15de..d17fc900 100644
> --- a/arch/arm/include/asm/irq.h
> +++ b/arch/arm/include/asm/irq.h
> @@ -24,7 +24,6 @@
>  #ifndef __ASSEMBLY__
>  struct irqaction;
>  struct pt_regs;
> -extern void migrate_irqs(void);
>  
>  extern void asm_do_IRQ(unsigned int, struct pt_regs *);
>  void handle_IRQ(unsigned int, struct pt_regs *);
> diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c
> index baf8ede..2efdb40 100644
> --- a/arch/arm/kernel/irq.c
> +++ b/arch/arm/kernel/irq.c
> @@ -31,7 +31,6 @@
>  #include <linux/smp.h>
>  #include <linux/init.h>
>  #include <linux/seq_file.h>
> -#include <linux/ratelimit.h>
>  #include <linux/errno.h>
>  #include <linux/list.h>
>  #include <linux/kallsyms.h>
> @@ -135,64 +134,3 @@ int __init arch_probe_nr_irqs(void)
>  	return nr_irqs;
>  }
>  #endif
> -
> -#ifdef CONFIG_HOTPLUG_CPU
> -static bool migrate_one_irq(struct irq_desc *desc)
> -{
> -	struct irq_data *d = irq_desc_get_irq_data(desc);
> -	const struct cpumask *affinity = irq_data_get_affinity_mask(d);
> -	struct irq_chip *c;
> -	bool ret = false;
> -
> -	/*
> -	 * If this is a per-CPU interrupt, or the affinity does not
> -	 * include this CPU, then we have nothing to do.
> -	 */
> -	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
> -		return false;
> -
> -	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
> -		affinity = cpu_online_mask;
> -		ret = true;
> -	}
> -
> -	c = irq_data_get_irq_chip(d);
> -	if (!c->irq_set_affinity)
> -		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
> -	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
> -		cpumask_copy(irq_data_get_affinity_mask(d), affinity);
> -
> -	return ret;
> -}
> -
> -/*
> - * The current CPU has been marked offline.  Migrate IRQs off this CPU.
> - * If the affinity settings do not allow other CPUs, force them onto any
> - * available CPU.
> - *
> - * Note: we must iterate over all IRQs, whether they have an attached
> - * action structure or not, as we need to get chained interrupts too.
> - */
> -void migrate_irqs(void)
> -{
> -	unsigned int i;
> -	struct irq_desc *desc;
> -	unsigned long flags;
> -
> -	local_irq_save(flags);
> -
> -	for_each_irq_desc(i, desc) {
> -		bool affinity_broken;
> -
> -		raw_spin_lock(&desc->lock);
> -		affinity_broken = migrate_one_irq(desc);
> -		raw_spin_unlock(&desc->lock);
> -
> -		if (affinity_broken)
> -			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
> -				i, smp_processor_id());
> -	}
> -
> -	local_irq_restore(flags);
> -}
> -#endif /* CONFIG_HOTPLUG_CPU */
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index b7b9cea..6ffe411 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -32,6 +32,7 @@ config ARM64
>  	select GENERIC_CPU_AUTOPROBE
>  	select GENERIC_EARLY_IOREMAP
>  	select GENERIC_IRQ_PROBE
> +	select GENERIC_IRQ_MIGRATION if SMP && HOTPLUG_CPU
>  	select GENERIC_IRQ_SHOW
>  	select GENERIC_IRQ_SHOW_LEVEL
>  	select GENERIC_PCI_IOMAP
> diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h
> index bbb251b..0916929 100644
> --- a/arch/arm64/include/asm/irq.h
> +++ b/arch/arm64/include/asm/irq.h
> @@ -7,7 +7,6 @@
>  
>  struct pt_regs;
>  
> -extern void migrate_irqs(void);
>  extern void set_handle_irq(void (*handle_irq)(struct pt_regs *));
>  
>  static inline void acpi_irq_init(void)
> diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
> index 463fa2e..04ac1f6 100644
> --- a/arch/arm64/kernel/irq.c
> +++ b/arch/arm64/kernel/irq.c
> @@ -27,7 +27,6 @@
>  #include <linux/init.h>
>  #include <linux/irqchip.h>
>  #include <linux/seq_file.h>
> -#include <linux/ratelimit.h>
>  
>  unsigned long irq_err_count;
>  
> @@ -56,64 +55,3 @@ void __init init_IRQ(void)
>  	if (!handle_arch_irq)
>  		panic("No interrupt controller found.");
>  }
> -
> -#ifdef CONFIG_HOTPLUG_CPU
> -static bool migrate_one_irq(struct irq_desc *desc)
> -{
> -	struct irq_data *d = irq_desc_get_irq_data(desc);
> -	const struct cpumask *affinity = irq_data_get_affinity_mask(d);
> -	struct irq_chip *c;
> -	bool ret = false;
> -
> -	/*
> -	 * If this is a per-CPU interrupt, or the affinity does not
> -	 * include this CPU, then we have nothing to do.
> -	 */
> -	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
> -		return false;
> -
> -	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
> -		affinity = cpu_online_mask;
> -		ret = true;
> -	}
> -
> -	c = irq_data_get_irq_chip(d);
> -	if (!c->irq_set_affinity)
> -		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
> -	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
> -		cpumask_copy(irq_data_get_affinity_mask(d), affinity);
> -
> -	return ret;
> -}
> -
> -/*
> - * The current CPU has been marked offline.  Migrate IRQs off this CPU.
> - * If the affinity settings do not allow other CPUs, force them onto any
> - * available CPU.
> - *
> - * Note: we must iterate over all IRQs, whether they have an attached
> - * action structure or not, as we need to get chained interrupts too.
> - */
> -void migrate_irqs(void)
> -{
> -	unsigned int i;
> -	struct irq_desc *desc;
> -	unsigned long flags;
> -
> -	local_irq_save(flags);
> -
> -	for_each_irq_desc(i, desc) {
> -		bool affinity_broken;
> -
> -		raw_spin_lock(&desc->lock);
> -		affinity_broken = migrate_one_irq(desc);
> -		raw_spin_unlock(&desc->lock);
> -
> -		if (affinity_broken)
> -			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
> -					    i, smp_processor_id());
> -	}
> -
> -	local_irq_restore(flags);
> -}
> -#endif /* CONFIG_HOTPLUG_CPU */
> diff --git a/kernel/irq/migration.c b/kernel/irq/migration.c
> index 37ddb7b..5801c79 100644
> --- a/kernel/irq/migration.c
> +++ b/kernel/irq/migration.c
> @@ -1,6 +1,7 @@
>  
>  #include <linux/irq.h>
>  #include <linux/interrupt.h>
> +#include <linux/ratelimit.h>
>  
>  #include "internals.h"
>  
> @@ -77,3 +78,64 @@ void irq_move_irq(struct irq_data *idata)
>  	if (!masked)
>  		idata->chip->irq_unmask(idata);
>  }
> +
> +#ifdef CONFIG_HOTPLUG_CPU
> +static bool migrate_one_irq(struct irq_desc *desc)
> +{
> +	struct irq_data *d = irq_desc_get_irq_data(desc);
> +	const struct cpumask *affinity = d->affinity;
> +	struct irq_chip *c;
> +	bool ret = false;
> +
> +	/*
> +	 * If this is a per-CPU interrupt, or the affinity does not
> +	 * include this CPU, then we have nothing to do.
> +	 */
> +	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
> +		return false;
> +
> +	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
> +		affinity = cpu_online_mask;
> +		ret = true;
> +	}
> +
> +	c = irq_data_get_irq_chip(d);
> +	if (!c->irq_set_affinity)
> +		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
How about pr_warn here? It may cause serious drawback if this happens.

> +	else
> +		irq_do_set_affinity(d, affinity, false);
Should we check return value here?

> +
> +	return ret;
> +}
> +
> +/*
> + * The current CPU has been marked offline.  Migrate IRQs off this CPU.
> + * If the affinity settings do not allow other CPUs, force them onto any
> + * available CPU.
> + *
> + * Note: we must iterate over all IRQs, whether they have an attached
> + * action structure or not, as we need to get chained interrupts too.
> + */
> +void migrate_irqs(void)
> +{
> +	unsigned int i;
> +	struct irq_desc *desc;
> +	unsigned long flags;
> +
> +	local_irq_save(flags);
> +
> +	for_each_irq_desc(i, desc) {
Should we use for_each_active_irq() here to iterate over active
irqs only?

> +		bool affinity_broken;
> +
> +		raw_spin_lock(&desc->lock);
> +		affinity_broken = migrate_one_irq(desc);
> +		raw_spin_unlock(&desc->lock);
> +
> +		if (affinity_broken)
> +			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
> +					    i, smp_processor_id());
> +	}
> +
> +	local_irq_restore(flags);
> +}
> +#endif /* CONFIG_HOTPLUG_CPU */
> 

[RFC PATCH v1 4/4] arm/arm64: fix a migrating irq bug when hotplug cpu

[Jiang Liu]

On 2015/9/6 12:23, Yang Yingliang wrote:
> When cpu is disabled, all irqs will be migratged to another cpu.
> In some cases, a new affinity is different, it needed to be coppied
> to irq's affinity. But if the type of irq is LPI, it's affinity will
> not be coppied because of irq_set_affinity's return value. Fix it by
> using irq_do_set_affinity.
> 
> And migrating interrupts is a core code matter, so move the code to
> kernel/irq/migration.c and select CONFIG_GENERIC_IRQ_MIGRATION when
> CONFIG_HOTPLUG_CPU and CONFIG_SMP is enabled.
> 
> Cc: Jiang Liu <jiang.liu@linux.intel.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Marc Zyngier <marc.zyngier@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
> Cc: Hanjun Guo <hanjun.guo@linaro.org>
> Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
> ---
>  arch/arm/Kconfig             |  1 +
>  arch/arm/include/asm/irq.h   |  1 -
>  arch/arm/kernel/irq.c        | 62 --------------------------------------------
>  arch/arm64/Kconfig           |  1 +
>  arch/arm64/include/asm/irq.h |  1 -
>  arch/arm64/kernel/irq.c      | 62 --------------------------------------------
>  kernel/irq/migration.c       | 62 ++++++++++++++++++++++++++++++++++++++++++++
>  7 files changed, 64 insertions(+), 126 deletions(-)
> 
> diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
> index 41cbb4a..ebc8a33 100644
> --- a/arch/arm/Kconfig
> +++ b/arch/arm/Kconfig
> @@ -22,6 +22,7 @@ config ARM
>  	select GENERIC_CLOCKEVENTS_BROADCAST if SMP
>  	select GENERIC_IDLE_POLL_SETUP
>  	select GENERIC_IRQ_PROBE
> +	select GENERIC_IRQ_MIGRATION if SMP && HOTPLUG_CPU
>  	select GENERIC_IRQ_SHOW
>  	select GENERIC_IRQ_SHOW_LEVEL
>  	select GENERIC_PCI_IOMAP
> diff --git a/arch/arm/include/asm/irq.h b/arch/arm/include/asm/irq.h
> index 53c15de..d17fc900 100644
> --- a/arch/arm/include/asm/irq.h
> +++ b/arch/arm/include/asm/irq.h
> @@ -24,7 +24,6 @@
>  #ifndef __ASSEMBLY__
>  struct irqaction;
>  struct pt_regs;
> -extern void migrate_irqs(void);
>  
>  extern void asm_do_IRQ(unsigned int, struct pt_regs *);
>  void handle_IRQ(unsigned int, struct pt_regs *);
> diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c
> index baf8ede..2efdb40 100644
> --- a/arch/arm/kernel/irq.c
> +++ b/arch/arm/kernel/irq.c
> @@ -31,7 +31,6 @@
>  #include <linux/smp.h>
>  #include <linux/init.h>
>  #include <linux/seq_file.h>
> -#include <linux/ratelimit.h>
>  #include <linux/errno.h>
>  #include <linux/list.h>
>  #include <linux/kallsyms.h>
> @@ -135,64 +134,3 @@ int __init arch_probe_nr_irqs(void)
>  	return nr_irqs;
>  }
>  #endif
> -
> -#ifdef CONFIG_HOTPLUG_CPU
> -static bool migrate_one_irq(struct irq_desc *desc)
> -{
> -	struct irq_data *d = irq_desc_get_irq_data(desc);
> -	const struct cpumask *affinity = irq_data_get_affinity_mask(d);
> -	struct irq_chip *c;
> -	bool ret = false;
> -
> -	/*
> -	 * If this is a per-CPU interrupt, or the affinity does not
> -	 * include this CPU, then we have nothing to do.
> -	 */
> -	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
> -		return false;
> -
> -	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
> -		affinity = cpu_online_mask;
> -		ret = true;
> -	}
> -
> -	c = irq_data_get_irq_chip(d);
> -	if (!c->irq_set_affinity)
> -		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
> -	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
> -		cpumask_copy(irq_data_get_affinity_mask(d), affinity);
> -
> -	return ret;
> -}
> -
> -/*
> - * The current CPU has been marked offline.  Migrate IRQs off this CPU.
> - * If the affinity settings do not allow other CPUs, force them onto any
> - * available CPU.
> - *
> - * Note: we must iterate over all IRQs, whether they have an attached
> - * action structure or not, as we need to get chained interrupts too.
> - */
> -void migrate_irqs(void)
> -{
> -	unsigned int i;
> -	struct irq_desc *desc;
> -	unsigned long flags;
> -
> -	local_irq_save(flags);
> -
> -	for_each_irq_desc(i, desc) {
> -		bool affinity_broken;
> -
> -		raw_spin_lock(&desc->lock);
> -		affinity_broken = migrate_one_irq(desc);
> -		raw_spin_unlock(&desc->lock);
> -
> -		if (affinity_broken)
> -			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
> -				i, smp_processor_id());
> -	}
> -
> -	local_irq_restore(flags);
> -}
> -#endif /* CONFIG_HOTPLUG_CPU */
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index b7b9cea..6ffe411 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -32,6 +32,7 @@ config ARM64
>  	select GENERIC_CPU_AUTOPROBE
>  	select GENERIC_EARLY_IOREMAP
>  	select GENERIC_IRQ_PROBE
> +	select GENERIC_IRQ_MIGRATION if SMP && HOTPLUG_CPU
>  	select GENERIC_IRQ_SHOW
>  	select GENERIC_IRQ_SHOW_LEVEL
>  	select GENERIC_PCI_IOMAP
> diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h
> index bbb251b..0916929 100644
> --- a/arch/arm64/include/asm/irq.h
> +++ b/arch/arm64/include/asm/irq.h
> @@ -7,7 +7,6 @@
>  
>  struct pt_regs;
>  
> -extern void migrate_irqs(void);
>  extern void set_handle_irq(void (*handle_irq)(struct pt_regs *));
>  
>  static inline void acpi_irq_init(void)
> diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
> index 463fa2e..04ac1f6 100644
> --- a/arch/arm64/kernel/irq.c
> +++ b/arch/arm64/kernel/irq.c
> @@ -27,7 +27,6 @@
>  #include <linux/init.h>
>  #include <linux/irqchip.h>
>  #include <linux/seq_file.h>
> -#include <linux/ratelimit.h>
>  
>  unsigned long irq_err_count;
>  
> @@ -56,64 +55,3 @@ void __init init_IRQ(void)
>  	if (!handle_arch_irq)
>  		panic("No interrupt controller found.");
>  }
> -
> -#ifdef CONFIG_HOTPLUG_CPU
> -static bool migrate_one_irq(struct irq_desc *desc)
> -{
> -	struct irq_data *d = irq_desc_get_irq_data(desc);
> -	const struct cpumask *affinity = irq_data_get_affinity_mask(d);
> -	struct irq_chip *c;
> -	bool ret = false;
> -
> -	/*
> -	 * If this is a per-CPU interrupt, or the affinity does not
> -	 * include this CPU, then we have nothing to do.
> -	 */
> -	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
> -		return false;
> -
> -	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
> -		affinity = cpu_online_mask;
> -		ret = true;
> -	}
> -
> -	c = irq_data_get_irq_chip(d);
> -	if (!c->irq_set_affinity)
> -		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
> -	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
> -		cpumask_copy(irq_data_get_affinity_mask(d), affinity);
> -
> -	return ret;
> -}
> -
> -/*
> - * The current CPU has been marked offline.  Migrate IRQs off this CPU.
> - * If the affinity settings do not allow other CPUs, force them onto any
> - * available CPU.
> - *
> - * Note: we must iterate over all IRQs, whether they have an attached
> - * action structure or not, as we need to get chained interrupts too.
> - */
> -void migrate_irqs(void)
> -{
> -	unsigned int i;
> -	struct irq_desc *desc;
> -	unsigned long flags;
> -
> -	local_irq_save(flags);
> -
> -	for_each_irq_desc(i, desc) {
> -		bool affinity_broken;
> -
> -		raw_spin_lock(&desc->lock);
> -		affinity_broken = migrate_one_irq(desc);
> -		raw_spin_unlock(&desc->lock);
> -
> -		if (affinity_broken)
> -			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
> -					    i, smp_processor_id());
> -	}
> -
> -	local_irq_restore(flags);
> -}
> -#endif /* CONFIG_HOTPLUG_CPU */
> diff --git a/kernel/irq/migration.c b/kernel/irq/migration.c
> index 37ddb7b..5801c79 100644
> --- a/kernel/irq/migration.c
> +++ b/kernel/irq/migration.c
> @@ -1,6 +1,7 @@
>  
>  #include <linux/irq.h>
>  #include <linux/interrupt.h>
> +#include <linux/ratelimit.h>
>  
>  #include "internals.h"
>  
> @@ -77,3 +78,64 @@ void irq_move_irq(struct irq_data *idata)
>  	if (!masked)
>  		idata->chip->irq_unmask(idata);
>  }
> +
> +#ifdef CONFIG_HOTPLUG_CPU
> +static bool migrate_one_irq(struct irq_desc *desc)
> +{
> +	struct irq_data *d = irq_desc_get_irq_data(desc);
> +	const struct cpumask *affinity = d->affinity;
> +	struct irq_chip *c;
> +	bool ret = false;
> +
> +	/*
> +	 * If this is a per-CPU interrupt, or the affinity does not
> +	 * include this CPU, then we have nothing to do.
> +	 */
> +	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
> +		return false;
> +
> +	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
> +		affinity = cpu_online_mask;
> +		ret = true;
> +	}
> +
> +	c = irq_data_get_irq_chip(d);
> +	if (!c->irq_set_affinity)
> +		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
How about pr_warn here? It may cause serious drawback if this happens.

> +	else
> +		irq_do_set_affinity(d, affinity, false);
Should we check return value here?

> +
> +	return ret;
> +}
> +
> +/*
> + * The current CPU has been marked offline.  Migrate IRQs off this CPU.
> + * If the affinity settings do not allow other CPUs, force them onto any
> + * available CPU.
> + *
> + * Note: we must iterate over all IRQs, whether they have an attached
> + * action structure or not, as we need to get chained interrupts too.
> + */
> +void migrate_irqs(void)
> +{
> +	unsigned int i;
> +	struct irq_desc *desc;
> +	unsigned long flags;
> +
> +	local_irq_save(flags);
> +
> +	for_each_irq_desc(i, desc) {
Should we use for_each_active_irq() here to iterate over active
irqs only?

> +		bool affinity_broken;
> +
> +		raw_spin_lock(&desc->lock);
> +		affinity_broken = migrate_one_irq(desc);
> +		raw_spin_unlock(&desc->lock);
> +
> +		if (affinity_broken)
> +			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
> +					    i, smp_processor_id());
> +	}
> +
> +	local_irq_restore(flags);
> +}
> +#endif /* CONFIG_HOTPLUG_CPU */
> 

Re: [RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Jiang Liu]

On 2015/9/6 12:23, Yang Yingliang wrote:
> Use irq_settings_set_move_pcntxt() helper irqs status with
> _IRQ_MOVE_PCNTXT. So that it can do set affinity when calling
> irq_set_affinity_locked().
Hi Yingliang,
	We could only set _IRQ_MOVE_PCNTCT flag to enable migrating
IRQ in process context if your hardware platform supports atomically
change IRQ configuration. Not sure whether that's true for GICv3.
If GICv3 doesn't support atomically change irq configuration, this
change may cause trouble.
Thanks!
Gerry

> 
> Cc: Jiang Liu <jiang.liu@linux.intel.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Marc Zyngier <marc.zyngier@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
> Cc: Hanjun Guo <hanjun.guo@linaro.org>
> Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
> ---
>  drivers/irqchip/irq-gic-v3.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
> index e406bc5..9108387 100644
> --- a/drivers/irqchip/irq-gic-v3.c
> +++ b/drivers/irqchip/irq-gic-v3.c
> @@ -688,6 +688,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
>  		irq_domain_set_info(d, irq, hw, &gic_chip, d->host_data,
>  				    handle_fasteoi_irq, NULL, NULL);
>  		set_irq_flags(irq, IRQF_VALID | IRQF_PROBE);
> +		irq_set_move_pcntxt(irq);
>  	}
>  	/* LPIs */
>  	if (hw >= 8192 && hw < GIC_ID_NR) {
> @@ -696,6 +697,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
>  		irq_domain_set_info(d, irq, hw, &gic_chip, d->host_data,
>  				    handle_fasteoi_irq, NULL, NULL);
>  		set_irq_flags(irq, IRQF_VALID);
> +		irq_set_move_pcntxt(irq);
>  	}
>  
>  	return 0;
> 

[RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Jiang Liu]

On 2015/9/6 12:23, Yang Yingliang wrote:
> Use irq_settings_set_move_pcntxt() helper irqs status with
> _IRQ_MOVE_PCNTXT. So that it can do set affinity when calling
> irq_set_affinity_locked().
Hi Yingliang,
	We could only set _IRQ_MOVE_PCNTCT flag to enable migrating
IRQ in process context if your hardware platform supports atomically
change IRQ configuration. Not sure whether that's true for GICv3.
If GICv3 doesn't support atomically change irq configuration, this
change may cause trouble.
Thanks!
Gerry

> 
> Cc: Jiang Liu <jiang.liu@linux.intel.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Marc Zyngier <marc.zyngier@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
> Cc: Hanjun Guo <hanjun.guo@linaro.org>
> Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
> ---
>  drivers/irqchip/irq-gic-v3.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
> index e406bc5..9108387 100644
> --- a/drivers/irqchip/irq-gic-v3.c
> +++ b/drivers/irqchip/irq-gic-v3.c
> @@ -688,6 +688,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
>  		irq_domain_set_info(d, irq, hw, &gic_chip, d->host_data,
>  				    handle_fasteoi_irq, NULL, NULL);
>  		set_irq_flags(irq, IRQF_VALID | IRQF_PROBE);
> +		irq_set_move_pcntxt(irq);
>  	}
>  	/* LPIs */
>  	if (hw >= 8192 && hw < GIC_ID_NR) {
> @@ -696,6 +697,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
>  		irq_domain_set_info(d, irq, hw, &gic_chip, d->host_data,
>  				    handle_fasteoi_irq, NULL, NULL);
>  		set_irq_flags(irq, IRQF_VALID);
> +		irq_set_move_pcntxt(irq);
>  	}
>  
>  	return 0;
> 

Re: [RFC PATCH v1 0/4] arm/arm64: fix a migrating irq bug when hotplug cpu

[Jiang Liu]

On 2015/9/6 12:23, Yang Yingliang wrote:
> Hi All,
> 
> There is a bug:
> 
> When cpu is disabled, all irqs will be migratged to another cpu.
> In some cases, a new affinity is different, it needed to be coppied
> to irq's affinity. But if the type of irq is LPI, it's affinity will
> not be coppied because of irq_set_affinity's return value.
> 
> 
> 
> As Marc and Will suggested, I refactor the arm/arm64 migrating interrupts
> code and fix the migrating irq bug while cpu is offline.
> 
> I'm trying let the core code do the migrating interrupts matter. kernel/irq/migration.c
> depends on CONFIG_GENERIC_PENDING_IRQ, so I make it selected by CONFIG_SMP and
> CONFIG_HOTPLUG_CPU and rename it to CONFIG_GENERIC_IRQ_MIGRATION for more general.
> When CONFIG_GENERIC_IRQ_MIGRATION is enabled, an interrupt whose state_use_accessors
> is not set with IRQD_MOVE_PCNTXT won't be migrated immediately in irq_set_affinity_locked().
> So introduce irq_settings_set_move_pcntxt() helper to set the state in gic_irq_domain_map().
> 
> With the above preparation, move the migrating interrupts code into kernel/irq/migration.c
> and fix the bug by using irq_do_set_affinity().
Hi Yingliang,
	As we are going to move migrate_irqs() to generic kernel
code, and powerpc, metag, xtensa, sh, ia64 mn10300 also defines
 migrate_irqs() too. It would be great if we could consolidate
all these.
	And as we are going to refine these code, there's another
issue need attention. On x86, we need to allocate a CPU vector
if an irq is directed to a CPU. So there's possibility that
we run out of CPU vectors after CPU hot-removal. So we have a
mechanism to detect whether we will run out of CPU vector
after removing a CPU, and reject CPU hot-removal if that will
happen.
	So the key point is, if we a need to allocate some sort
of resource on the target CPUs for an irq, we need two steps
when removing a CPU
1) check whether resources are available after removing the CPU,
   and reject CPU removal request if we ran out of resource
2) fix irqs after hot-removing the CPU.
Thanks!
Gerry

> 
> Cc: Jiang Liu <jiang.liu@linux.intel.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Marc Zyngier <marc.zyngier@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
> Cc: Hanjun Guo <hanjun.guo@linaro.org>
> 
> Yang Yingliang (4):
>   genirq: Introduce irq_settings_set_move_pcntxt() helper
>   irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT
>   genirq: rename config GENERIC_PENDING_IRQ to GENERIC_IRQ_MIGRATION
>   arm/arm64: fix a migrating irq bug when hotplug cpu
> 
>  arch/arc/Kconfig               |  2 +-
>  arch/arm/Kconfig               |  1 +
>  arch/arm/include/asm/irq.h     |  1 -
>  arch/arm/kernel/irq.c          | 62 ------------------------------------------
>  arch/arm64/Kconfig             |  1 +
>  arch/arm64/include/asm/irq.h   |  1 -
>  arch/arm64/kernel/irq.c        | 62 ------------------------------------------
>  arch/hexagon/Kconfig           |  2 +-
>  arch/ia64/Kconfig              |  2 +-
>  arch/tile/Kconfig              |  2 +-
>  arch/x86/Kconfig               |  2 +-
>  arch/x86/kernel/apic/io_apic.c |  2 +-
>  drivers/irqchip/irq-gic-v3.c   |  2 ++
>  include/linux/irq.h            |  5 +++-
>  include/linux/irqdesc.h        |  2 +-
>  kernel/irq/Kconfig             |  4 +--
>  kernel/irq/Makefile            |  2 +-
>  kernel/irq/irqdesc.c           | 18 ++++++++++--
>  kernel/irq/manage.c            |  2 +-
>  kernel/irq/migration.c         | 62 ++++++++++++++++++++++++++++++++++++++++++
>  kernel/irq/proc.c              |  2 +-
>  kernel/irq/settings.h          |  5 ++++
>  22 files changed, 102 insertions(+), 142 deletions(-)
> 

[RFC PATCH v1 0/4] arm/arm64: fix a migrating irq bug when hotplug cpu

[Jiang Liu]

On 2015/9/6 12:23, Yang Yingliang wrote:
> Hi All,
> 
> There is a bug:
> 
> When cpu is disabled, all irqs will be migratged to another cpu.
> In some cases, a new affinity is different, it needed to be coppied
> to irq's affinity. But if the type of irq is LPI, it's affinity will
> not be coppied because of irq_set_affinity's return value.
> 
> 
> 
> As Marc and Will suggested, I refactor the arm/arm64 migrating interrupts
> code and fix the migrating irq bug while cpu is offline.
> 
> I'm trying let the core code do the migrating interrupts matter. kernel/irq/migration.c
> depends on CONFIG_GENERIC_PENDING_IRQ, so I make it selected by CONFIG_SMP and
> CONFIG_HOTPLUG_CPU and rename it to CONFIG_GENERIC_IRQ_MIGRATION for more general.
> When CONFIG_GENERIC_IRQ_MIGRATION is enabled, an interrupt whose state_use_accessors
> is not set with IRQD_MOVE_PCNTXT won't be migrated immediately in irq_set_affinity_locked().
> So introduce irq_settings_set_move_pcntxt() helper to set the state in gic_irq_domain_map().
> 
> With the above preparation, move the migrating interrupts code into kernel/irq/migration.c
> and fix the bug by using irq_do_set_affinity().
Hi Yingliang,
	As we are going to move migrate_irqs() to generic kernel
code, and powerpc, metag, xtensa, sh, ia64 mn10300 also defines
 migrate_irqs() too. It would be great if we could consolidate
all these.
	And as we are going to refine these code, there's another
issue need attention. On x86, we need to allocate a CPU vector
if an irq is directed to a CPU. So there's possibility that
we run out of CPU vectors after CPU hot-removal. So we have a
mechanism to detect whether we will run out of CPU vector
after removing a CPU, and reject CPU hot-removal if that will
happen.
	So the key point is, if we a need to allocate some sort
of resource on the target CPUs for an irq, we need two steps
when removing a CPU
1) check whether resources are available after removing the CPU,
   and reject CPU removal request if we ran out of resource
2) fix irqs after hot-removing the CPU.
Thanks!
Gerry

> 
> Cc: Jiang Liu <jiang.liu@linux.intel.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Marc Zyngier <marc.zyngier@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
> Cc: Hanjun Guo <hanjun.guo@linaro.org>
> 
> Yang Yingliang (4):
>   genirq: Introduce irq_settings_set_move_pcntxt() helper
>   irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT
>   genirq: rename config GENERIC_PENDING_IRQ to GENERIC_IRQ_MIGRATION
>   arm/arm64: fix a migrating irq bug when hotplug cpu
> 
>  arch/arc/Kconfig               |  2 +-
>  arch/arm/Kconfig               |  1 +
>  arch/arm/include/asm/irq.h     |  1 -
>  arch/arm/kernel/irq.c          | 62 ------------------------------------------
>  arch/arm64/Kconfig             |  1 +
>  arch/arm64/include/asm/irq.h   |  1 -
>  arch/arm64/kernel/irq.c        | 62 ------------------------------------------
>  arch/hexagon/Kconfig           |  2 +-
>  arch/ia64/Kconfig              |  2 +-
>  arch/tile/Kconfig              |  2 +-
>  arch/x86/Kconfig               |  2 +-
>  arch/x86/kernel/apic/io_apic.c |  2 +-
>  drivers/irqchip/irq-gic-v3.c   |  2 ++
>  include/linux/irq.h            |  5 +++-
>  include/linux/irqdesc.h        |  2 +-
>  kernel/irq/Kconfig             |  4 +--
>  kernel/irq/Makefile            |  2 +-
>  kernel/irq/irqdesc.c           | 18 ++++++++++--
>  kernel/irq/manage.c            |  2 +-
>  kernel/irq/migration.c         | 62 ++++++++++++++++++++++++++++++++++++++++++
>  kernel/irq/proc.c              |  2 +-
>  kernel/irq/settings.h          |  5 ++++
>  22 files changed, 102 insertions(+), 142 deletions(-)
> 

Re: [RFC PATCH v1 1/4] genirq: Introduce irq_settings_set_move_pcntxt() helper

[Thomas Gleixner]

On Sun, 6 Sep 2015, Yang Yingliang wrote:

> It's expected to use the helper when interrupt can be moved
> in process.

What's wrong with using the existing helper?

       irq_set_status_flags(irq, IRQ_MOVE_PCNTXT);

Thanks,

	tglx

[RFC PATCH v1 1/4] genirq: Introduce irq_settings_set_move_pcntxt() helper

[Thomas Gleixner]

On Sun, 6 Sep 2015, Yang Yingliang wrote:

> It's expected to use the helper when interrupt can be moved
> in process.

What's wrong with using the existing helper?

       irq_set_status_flags(irq, IRQ_MOVE_PCNTXT);

Thanks,

	tglx

Re: [RFC PATCH v1 1/4] genirq: Introduce irq_settings_set_move_pcntxt() helper

[Yang Yingliang]

On 2015/9/7 6:08, Thomas Gleixner wrote:
> On Sun, 6 Sep 2015, Yang Yingliang wrote:
>
>> It's expected to use the helper when interrupt can be moved
>> in process.
>
> What's wrong with using the existing helper?
>
>         irq_set_status_flags(irq, IRQ_MOVE_PCNTXT);

It looks OK. I will use it next version.

Thanks,
Yang

>
> Thanks,
>
> 	tglx
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>
>

[RFC PATCH v1 1/4] genirq: Introduce irq_settings_set_move_pcntxt() helper

[Yang Yingliang]

On 2015/9/7 6:08, Thomas Gleixner wrote:
> On Sun, 6 Sep 2015, Yang Yingliang wrote:
>
>> It's expected to use the helper when interrupt can be moved
>> in process.
>
> What's wrong with using the existing helper?
>
>         irq_set_status_flags(irq, IRQ_MOVE_PCNTXT);

It looks OK. I will use it next version.

Thanks,
Yang

>
> Thanks,
>
> 	tglx
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>
>

Re: [RFC PATCH v1 0/4] arm/arm64: fix a migrating irq bug when hotplug cpu

[Jiang Liu]

On 2015/9/6 12:23, Yang Yingliang wrote:
> Hi All,
> 
> There is a bug:
> 
> When cpu is disabled, all irqs will be migratged to another cpu.
> In some cases, a new affinity is different, it needed to be coppied
> to irq's affinity. But if the type of irq is LPI, it's affinity will
> not be coppied because of irq_set_affinity's return value.
> 
> 
> 
> As Marc and Will suggested, I refactor the arm/arm64 migrating interrupts
> code and fix the migrating irq bug while cpu is offline.
> 
> I'm trying let the core code do the migrating interrupts matter. kernel/irq/migration.c
> depends on CONFIG_GENERIC_PENDING_IRQ, so I make it selected by CONFIG_SMP and
> CONFIG_HOTPLUG_CPU and rename it to CONFIG_GENERIC_IRQ_MIGRATION for more general.
> When CONFIG_GENERIC_IRQ_MIGRATION is enabled, an interrupt whose state_use_accessors
> is not set with IRQD_MOVE_PCNTXT won't be migrated immediately in irq_set_affinity_locked().
> So introduce irq_settings_set_move_pcntxt() helper to set the state in gic_irq_domain_map().
> 
> With the above preparation, move the migrating interrupts code into kernel/irq/migration.c
> and fix the bug by using irq_do_set_affinity().
Hi Yingliang,
	As we are going to move migrate_irqs() to generic kernel
code, and powerpc, metag, xtensa, sh, ia64 mn10300 also defines
 migrate_irqs() too. It would be great if we could consolidate
all these.
	And as we are going to refine these code, there's another
issue need attention. On x86, we need to allocate a CPU vector
if an irq is directed to a CPU. So there's possibility that
we run out of CPU vectors after CPU hot-removal. So we have a
mechanism to detect whether we will run out of CPU vector
after removing a CPU, and reject CPU hot-removal if that will
happen.
	So the key point is, if we a need to allocate some sort
of resource on the target CPUs for an irq, we need two steps
when removing a CPU
1) check whether resources are available after removing the CPU,
   and reject CPU removal request if we ran out of resource
2) fix irqs affinity after removing the CPU.
Thanks!
Gerry

> 
> Cc: Jiang Liu <jiang.liu@linux.intel.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Marc Zyngier <marc.zyngier@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
> Cc: Hanjun Guo <hanjun.guo@linaro.org>
> 
> Yang Yingliang (4):
>   genirq: Introduce irq_settings_set_move_pcntxt() helper
>   irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT
>   genirq: rename config GENERIC_PENDING_IRQ to GENERIC_IRQ_MIGRATION
>   arm/arm64: fix a migrating irq bug when hotplug cpu
> 
>  arch/arc/Kconfig               |  2 +-
>  arch/arm/Kconfig               |  1 +
>  arch/arm/include/asm/irq.h     |  1 -
>  arch/arm/kernel/irq.c          | 62 ------------------------------------------
>  arch/arm64/Kconfig             |  1 +
>  arch/arm64/include/asm/irq.h   |  1 -
>  arch/arm64/kernel/irq.c        | 62 ------------------------------------------
>  arch/hexagon/Kconfig           |  2 +-
>  arch/ia64/Kconfig              |  2 +-
>  arch/tile/Kconfig              |  2 +-
>  arch/x86/Kconfig               |  2 +-
>  arch/x86/kernel/apic/io_apic.c |  2 +-
>  drivers/irqchip/irq-gic-v3.c   |  2 ++
>  include/linux/irq.h            |  5 +++-
>  include/linux/irqdesc.h        |  2 +-
>  kernel/irq/Kconfig             |  4 +--
>  kernel/irq/Makefile            |  2 +-
>  kernel/irq/irqdesc.c           | 18 ++++++++++--
>  kernel/irq/manage.c            |  2 +-
>  kernel/irq/migration.c         | 62 ++++++++++++++++++++++++++++++++++++++++++
>  kernel/irq/proc.c              |  2 +-
>  kernel/irq/settings.h          |  5 ++++
>  22 files changed, 102 insertions(+), 142 deletions(-)
> 

[RFC PATCH v1 0/4] arm/arm64: fix a migrating irq bug when hotplug cpu

[Jiang Liu]

On 2015/9/6 12:23, Yang Yingliang wrote:
> Hi All,
> 
> There is a bug:
> 
> When cpu is disabled, all irqs will be migratged to another cpu.
> In some cases, a new affinity is different, it needed to be coppied
> to irq's affinity. But if the type of irq is LPI, it's affinity will
> not be coppied because of irq_set_affinity's return value.
> 
> 
> 
> As Marc and Will suggested, I refactor the arm/arm64 migrating interrupts
> code and fix the migrating irq bug while cpu is offline.
> 
> I'm trying let the core code do the migrating interrupts matter. kernel/irq/migration.c
> depends on CONFIG_GENERIC_PENDING_IRQ, so I make it selected by CONFIG_SMP and
> CONFIG_HOTPLUG_CPU and rename it to CONFIG_GENERIC_IRQ_MIGRATION for more general.
> When CONFIG_GENERIC_IRQ_MIGRATION is enabled, an interrupt whose state_use_accessors
> is not set with IRQD_MOVE_PCNTXT won't be migrated immediately in irq_set_affinity_locked().
> So introduce irq_settings_set_move_pcntxt() helper to set the state in gic_irq_domain_map().
> 
> With the above preparation, move the migrating interrupts code into kernel/irq/migration.c
> and fix the bug by using irq_do_set_affinity().
Hi Yingliang,
	As we are going to move migrate_irqs() to generic kernel
code, and powerpc, metag, xtensa, sh, ia64 mn10300 also defines
 migrate_irqs() too. It would be great if we could consolidate
all these.
	And as we are going to refine these code, there's another
issue need attention. On x86, we need to allocate a CPU vector
if an irq is directed to a CPU. So there's possibility that
we run out of CPU vectors after CPU hot-removal. So we have a
mechanism to detect whether we will run out of CPU vector
after removing a CPU, and reject CPU hot-removal if that will
happen.
	So the key point is, if we a need to allocate some sort
of resource on the target CPUs for an irq, we need two steps
when removing a CPU
1) check whether resources are available after removing the CPU,
   and reject CPU removal request if we ran out of resource
2) fix irqs affinity after removing the CPU.
Thanks!
Gerry

> 
> Cc: Jiang Liu <jiang.liu@linux.intel.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Marc Zyngier <marc.zyngier@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
> Cc: Hanjun Guo <hanjun.guo@linaro.org>
> 
> Yang Yingliang (4):
>   genirq: Introduce irq_settings_set_move_pcntxt() helper
>   irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT
>   genirq: rename config GENERIC_PENDING_IRQ to GENERIC_IRQ_MIGRATION
>   arm/arm64: fix a migrating irq bug when hotplug cpu
> 
>  arch/arc/Kconfig               |  2 +-
>  arch/arm/Kconfig               |  1 +
>  arch/arm/include/asm/irq.h     |  1 -
>  arch/arm/kernel/irq.c          | 62 ------------------------------------------
>  arch/arm64/Kconfig             |  1 +
>  arch/arm64/include/asm/irq.h   |  1 -
>  arch/arm64/kernel/irq.c        | 62 ------------------------------------------
>  arch/hexagon/Kconfig           |  2 +-
>  arch/ia64/Kconfig              |  2 +-
>  arch/tile/Kconfig              |  2 +-
>  arch/x86/Kconfig               |  2 +-
>  arch/x86/kernel/apic/io_apic.c |  2 +-
>  drivers/irqchip/irq-gic-v3.c   |  2 ++
>  include/linux/irq.h            |  5 +++-
>  include/linux/irqdesc.h        |  2 +-
>  kernel/irq/Kconfig             |  4 +--
>  kernel/irq/Makefile            |  2 +-
>  kernel/irq/irqdesc.c           | 18 ++++++++++--
>  kernel/irq/manage.c            |  2 +-
>  kernel/irq/migration.c         | 62 ++++++++++++++++++++++++++++++++++++++++++
>  kernel/irq/proc.c              |  2 +-
>  kernel/irq/settings.h          |  5 ++++
>  22 files changed, 102 insertions(+), 142 deletions(-)
> 

Re: [RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Yang Yingliang]

On 2015/9/6 13:56, Jiang Liu wrote:
> On 2015/9/6 12:23, Yang Yingliang wrote:
>> Use irq_settings_set_move_pcntxt() helper irqs status with
>> _IRQ_MOVE_PCNTXT. So that it can do set affinity when calling
>> irq_set_affinity_locked().
> Hi Yingliang,
> 	We could only set _IRQ_MOVE_PCNTCT flag to enable migrating
> IRQ in process context if your hardware platform supports atomically
> change IRQ configuration. Not sure whether that's true for GICv3.
> If GICv3 doesn't support atomically change irq configuration, this
> change may cause trouble.
> Thanks!
> Gerry

I am not certain sure if GICv3 supports it. But current code consider it 
as true default without CONFIG_GENERIC_PENDING_IRQ enable on arm.
Does Marc have any opinion ?

Thanks
Yang
>
>>
>> Cc: Jiang Liu <jiang.liu@linux.intel.com>
>> Cc: Thomas Gleixner <tglx@linutronix.de>
>> Cc: Marc Zyngier <marc.zyngier@arm.com>
>> Cc: Mark Rutland <mark.rutland@arm.com>
>> Cc: Will Deacon <will.deacon@arm.com>
>> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
>> Cc: Hanjun Guo <hanjun.guo@linaro.org>
>> Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
>> ---
>>   drivers/irqchip/irq-gic-v3.c | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
>> index e406bc5..9108387 100644
>> --- a/drivers/irqchip/irq-gic-v3.c
>> +++ b/drivers/irqchip/irq-gic-v3.c
>> @@ -688,6 +688,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
>>   		irq_domain_set_info(d, irq, hw, &gic_chip, d->host_data,
>>   				    handle_fasteoi_irq, NULL, NULL);
>>   		set_irq_flags(irq, IRQF_VALID | IRQF_PROBE);
>> +		irq_set_move_pcntxt(irq);
>>   	}
>>   	/* LPIs */
>>   	if (hw >= 8192 && hw < GIC_ID_NR) {
>> @@ -696,6 +697,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
>>   		irq_domain_set_info(d, irq, hw, &gic_chip, d->host_data,
>>   				    handle_fasteoi_irq, NULL, NULL);
>>   		set_irq_flags(irq, IRQF_VALID);
>> +		irq_set_move_pcntxt(irq);
>>   	}
>>
>>   	return 0;
>>
>
> .
>

[RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Yang Yingliang]

On 2015/9/6 13:56, Jiang Liu wrote:
> On 2015/9/6 12:23, Yang Yingliang wrote:
>> Use irq_settings_set_move_pcntxt() helper irqs status with
>> _IRQ_MOVE_PCNTXT. So that it can do set affinity when calling
>> irq_set_affinity_locked().
> Hi Yingliang,
> 	We could only set _IRQ_MOVE_PCNTCT flag to enable migrating
> IRQ in process context if your hardware platform supports atomically
> change IRQ configuration. Not sure whether that's true for GICv3.
> If GICv3 doesn't support atomically change irq configuration, this
> change may cause trouble.
> Thanks!
> Gerry

I am not certain sure if GICv3 supports it. But current code consider it 
as true default without CONFIG_GENERIC_PENDING_IRQ enable on arm.
Does Marc have any opinion ?

Thanks
Yang
>
>>
>> Cc: Jiang Liu <jiang.liu@linux.intel.com>
>> Cc: Thomas Gleixner <tglx@linutronix.de>
>> Cc: Marc Zyngier <marc.zyngier@arm.com>
>> Cc: Mark Rutland <mark.rutland@arm.com>
>> Cc: Will Deacon <will.deacon@arm.com>
>> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
>> Cc: Hanjun Guo <hanjun.guo@linaro.org>
>> Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
>> ---
>>   drivers/irqchip/irq-gic-v3.c | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
>> index e406bc5..9108387 100644
>> --- a/drivers/irqchip/irq-gic-v3.c
>> +++ b/drivers/irqchip/irq-gic-v3.c
>> @@ -688,6 +688,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
>>   		irq_domain_set_info(d, irq, hw, &gic_chip, d->host_data,
>>   				    handle_fasteoi_irq, NULL, NULL);
>>   		set_irq_flags(irq, IRQF_VALID | IRQF_PROBE);
>> +		irq_set_move_pcntxt(irq);
>>   	}
>>   	/* LPIs */
>>   	if (hw >= 8192 && hw < GIC_ID_NR) {
>> @@ -696,6 +697,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
>>   		irq_domain_set_info(d, irq, hw, &gic_chip, d->host_data,
>>   				    handle_fasteoi_irq, NULL, NULL);
>>   		set_irq_flags(irq, IRQF_VALID);
>> +		irq_set_move_pcntxt(irq);
>>   	}
>>
>>   	return 0;
>>
>
> .
>

Re: [RFC PATCH v1 4/4] arm/arm64: fix a migrating irq bug when hotplug cpu

[Yang Yingliang]

On 2015/9/6 13:55, Jiang Liu wrote:
>
>
> On 2015/9/6 12:23, Yang Yingliang wrote:
>> When cpu is disabled, all irqs will be migratged to another cpu.
>> In some cases, a new affinity is different, it needed to be coppied
>> to irq's affinity. But if the type of irq is LPI, it's affinity will
>> not be coppied because of irq_set_affinity's return value. Fix it by
>> using irq_do_set_affinity.
>>
>> And migrating interrupts is a core code matter, so move the code to
>> kernel/irq/migration.c and select CONFIG_GENERIC_IRQ_MIGRATION when
>> CONFIG_HOTPLUG_CPU and CONFIG_SMP is enabled.
>>
>> Cc: Jiang Liu <jiang.liu@linux.intel.com>
>> Cc: Thomas Gleixner <tglx@linutronix.de>
>> Cc: Marc Zyngier <marc.zyngier@arm.com>
>> Cc: Mark Rutland <mark.rutland@arm.com>
>> Cc: Will Deacon <will.deacon@arm.com>
>> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
>> Cc: Hanjun Guo <hanjun.guo@linaro.org>
>> Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
>> ---
>>   arch/arm/Kconfig             |  1 +
>>   arch/arm/include/asm/irq.h   |  1 -
>>   arch/arm/kernel/irq.c        | 62 --------------------------------------------
>>   arch/arm64/Kconfig           |  1 +
>>   arch/arm64/include/asm/irq.h |  1 -
>>   arch/arm64/kernel/irq.c      | 62 --------------------------------------------
>>   kernel/irq/migration.c       | 62 ++++++++++++++++++++++++++++++++++++++++++++
>>   7 files changed, 64 insertions(+), 126 deletions(-)
>>
>> diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
>> index 41cbb4a..ebc8a33 100644
>> --- a/arch/arm/Kconfig
>> +++ b/arch/arm/Kconfig
>> @@ -22,6 +22,7 @@ config ARM
>>   	select GENERIC_CLOCKEVENTS_BROADCAST if SMP
>>   	select GENERIC_IDLE_POLL_SETUP
>>   	select GENERIC_IRQ_PROBE
>> +	select GENERIC_IRQ_MIGRATION if SMP && HOTPLUG_CPU
>>   	select GENERIC_IRQ_SHOW
>>   	select GENERIC_IRQ_SHOW_LEVEL
>>   	select GENERIC_PCI_IOMAP
>> diff --git a/arch/arm/include/asm/irq.h b/arch/arm/include/asm/irq.h
>> index 53c15de..d17fc900 100644
>> --- a/arch/arm/include/asm/irq.h
>> +++ b/arch/arm/include/asm/irq.h
>> @@ -24,7 +24,6 @@
>>   #ifndef __ASSEMBLY__
>>   struct irqaction;
>>   struct pt_regs;
>> -extern void migrate_irqs(void);
>>
>>   extern void asm_do_IRQ(unsigned int, struct pt_regs *);
>>   void handle_IRQ(unsigned int, struct pt_regs *);
>> diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c
>> index baf8ede..2efdb40 100644
>> --- a/arch/arm/kernel/irq.c
>> +++ b/arch/arm/kernel/irq.c
>> @@ -31,7 +31,6 @@
>>   #include <linux/smp.h>
>>   #include <linux/init.h>
>>   #include <linux/seq_file.h>
>> -#include <linux/ratelimit.h>
>>   #include <linux/errno.h>
>>   #include <linux/list.h>
>>   #include <linux/kallsyms.h>
>> @@ -135,64 +134,3 @@ int __init arch_probe_nr_irqs(void)
>>   	return nr_irqs;
>>   }
>>   #endif
>> -
>> -#ifdef CONFIG_HOTPLUG_CPU
>> -static bool migrate_one_irq(struct irq_desc *desc)
>> -{
>> -	struct irq_data *d = irq_desc_get_irq_data(desc);
>> -	const struct cpumask *affinity = irq_data_get_affinity_mask(d);
>> -	struct irq_chip *c;
>> -	bool ret = false;
>> -
>> -	/*
>> -	 * If this is a per-CPU interrupt, or the affinity does not
>> -	 * include this CPU, then we have nothing to do.
>> -	 */
>> -	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
>> -		return false;
>> -
>> -	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
>> -		affinity = cpu_online_mask;
>> -		ret = true;
>> -	}
>> -
>> -	c = irq_data_get_irq_chip(d);
>> -	if (!c->irq_set_affinity)
>> -		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
>> -	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
>> -		cpumask_copy(irq_data_get_affinity_mask(d), affinity);
>> -
>> -	return ret;
>> -}
>> -
>> -/*
>> - * The current CPU has been marked offline.  Migrate IRQs off this CPU.
>> - * If the affinity settings do not allow other CPUs, force them onto any
>> - * available CPU.
>> - *
>> - * Note: we must iterate over all IRQs, whether they have an attached
>> - * action structure or not, as we need to get chained interrupts too.
>> - */
>> -void migrate_irqs(void)
>> -{
>> -	unsigned int i;
>> -	struct irq_desc *desc;
>> -	unsigned long flags;
>> -
>> -	local_irq_save(flags);
>> -
>> -	for_each_irq_desc(i, desc) {
>> -		bool affinity_broken;
>> -
>> -		raw_spin_lock(&desc->lock);
>> -		affinity_broken = migrate_one_irq(desc);
>> -		raw_spin_unlock(&desc->lock);
>> -
>> -		if (affinity_broken)
>> -			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
>> -				i, smp_processor_id());
>> -	}
>> -
>> -	local_irq_restore(flags);
>> -}
>> -#endif /* CONFIG_HOTPLUG_CPU */
>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>> index b7b9cea..6ffe411 100644
>> --- a/arch/arm64/Kconfig
>> +++ b/arch/arm64/Kconfig
>> @@ -32,6 +32,7 @@ config ARM64
>>   	select GENERIC_CPU_AUTOPROBE
>>   	select GENERIC_EARLY_IOREMAP
>>   	select GENERIC_IRQ_PROBE
>> +	select GENERIC_IRQ_MIGRATION if SMP && HOTPLUG_CPU
>>   	select GENERIC_IRQ_SHOW
>>   	select GENERIC_IRQ_SHOW_LEVEL
>>   	select GENERIC_PCI_IOMAP
>> diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h
>> index bbb251b..0916929 100644
>> --- a/arch/arm64/include/asm/irq.h
>> +++ b/arch/arm64/include/asm/irq.h
>> @@ -7,7 +7,6 @@
>>
>>   struct pt_regs;
>>
>> -extern void migrate_irqs(void);
>>   extern void set_handle_irq(void (*handle_irq)(struct pt_regs *));
>>
>>   static inline void acpi_irq_init(void)
>> diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
>> index 463fa2e..04ac1f6 100644
>> --- a/arch/arm64/kernel/irq.c
>> +++ b/arch/arm64/kernel/irq.c
>> @@ -27,7 +27,6 @@
>>   #include <linux/init.h>
>>   #include <linux/irqchip.h>
>>   #include <linux/seq_file.h>
>> -#include <linux/ratelimit.h>
>>
>>   unsigned long irq_err_count;
>>
>> @@ -56,64 +55,3 @@ void __init init_IRQ(void)
>>   	if (!handle_arch_irq)
>>   		panic("No interrupt controller found.");
>>   }
>> -
>> -#ifdef CONFIG_HOTPLUG_CPU
>> -static bool migrate_one_irq(struct irq_desc *desc)
>> -{
>> -	struct irq_data *d = irq_desc_get_irq_data(desc);
>> -	const struct cpumask *affinity = irq_data_get_affinity_mask(d);
>> -	struct irq_chip *c;
>> -	bool ret = false;
>> -
>> -	/*
>> -	 * If this is a per-CPU interrupt, or the affinity does not
>> -	 * include this CPU, then we have nothing to do.
>> -	 */
>> -	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
>> -		return false;
>> -
>> -	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
>> -		affinity = cpu_online_mask;
>> -		ret = true;
>> -	}
>> -
>> -	c = irq_data_get_irq_chip(d);
>> -	if (!c->irq_set_affinity)
>> -		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
>> -	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
>> -		cpumask_copy(irq_data_get_affinity_mask(d), affinity);
>> -
>> -	return ret;
>> -}
>> -
>> -/*
>> - * The current CPU has been marked offline.  Migrate IRQs off this CPU.
>> - * If the affinity settings do not allow other CPUs, force them onto any
>> - * available CPU.
>> - *
>> - * Note: we must iterate over all IRQs, whether they have an attached
>> - * action structure or not, as we need to get chained interrupts too.
>> - */
>> -void migrate_irqs(void)
>> -{
>> -	unsigned int i;
>> -	struct irq_desc *desc;
>> -	unsigned long flags;
>> -
>> -	local_irq_save(flags);
>> -
>> -	for_each_irq_desc(i, desc) {
>> -		bool affinity_broken;
>> -
>> -		raw_spin_lock(&desc->lock);
>> -		affinity_broken = migrate_one_irq(desc);
>> -		raw_spin_unlock(&desc->lock);
>> -
>> -		if (affinity_broken)
>> -			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
>> -					    i, smp_processor_id());
>> -	}
>> -
>> -	local_irq_restore(flags);
>> -}
>> -#endif /* CONFIG_HOTPLUG_CPU */
>> diff --git a/kernel/irq/migration.c b/kernel/irq/migration.c
>> index 37ddb7b..5801c79 100644
>> --- a/kernel/irq/migration.c
>> +++ b/kernel/irq/migration.c
>> @@ -1,6 +1,7 @@
>>
>>   #include <linux/irq.h>
>>   #include <linux/interrupt.h>
>> +#include <linux/ratelimit.h>
>>
>>   #include "internals.h"
>>
>> @@ -77,3 +78,64 @@ void irq_move_irq(struct irq_data *idata)
>>   	if (!masked)
>>   		idata->chip->irq_unmask(idata);
>>   }
>> +
>> +#ifdef CONFIG_HOTPLUG_CPU
>> +static bool migrate_one_irq(struct irq_desc *desc)
>> +{
>> +	struct irq_data *d = irq_desc_get_irq_data(desc);
>> +	const struct cpumask *affinity = d->affinity;
>> +	struct irq_chip *c;
>> +	bool ret = false;
>> +
>> +	/*
>> +	 * If this is a per-CPU interrupt, or the affinity does not
>> +	 * include this CPU, then we have nothing to do.
>> +	 */
>> +	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
>> +		return false;
>> +
>> +	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
>> +		affinity = cpu_online_mask;
>> +		ret = true;
>> +	}
>> +
>> +	c = irq_data_get_irq_chip(d);
>> +	if (!c->irq_set_affinity)
>> +		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
> How about pr_warn here? It may cause serious drawback if this happens.
I think so, I will change it next version.
>
>> +	else
>> +		irq_do_set_affinity(d, affinity, false);
> Should we check return value here?
>
The return value is not used, so we don't check it.
Maybe I could add some warning by checking it.
>> +
>> +	return ret;
>> +}
>> +
>> +/*
>> + * The current CPU has been marked offline.  Migrate IRQs off this CPU.
>> + * If the affinity settings do not allow other CPUs, force them onto any
>> + * available CPU.
>> + *
>> + * Note: we must iterate over all IRQs, whether they have an attached
>> + * action structure or not, as we need to get chained interrupts too.
>> + */
>> +void migrate_irqs(void)
>> +{
>> +	unsigned int i;
>> +	struct irq_desc *desc;
>> +	unsigned long flags;
>> +
>> +	local_irq_save(flags);
>> +
>> +	for_each_irq_desc(i, desc) {
> Should we use for_each_active_irq() here to iterate over active
> irqs only?
>
It looks good. I will use it next version.
>> +		bool affinity_broken;
>> +
>> +		raw_spin_lock(&desc->lock);
>> +		affinity_broken = migrate_one_irq(desc);
>> +		raw_spin_unlock(&desc->lock);
>> +
>> +		if (affinity_broken)
>> +			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
>> +					    i, smp_processor_id());
>> +	}
>> +
>> +	local_irq_restore(flags);
>> +}
>> +#endif /* CONFIG_HOTPLUG_CPU */
>>
>
> .
>

[RFC PATCH v1 4/4] arm/arm64: fix a migrating irq bug when hotplug cpu

[Yang Yingliang]

On 2015/9/6 13:55, Jiang Liu wrote:
>
>
> On 2015/9/6 12:23, Yang Yingliang wrote:
>> When cpu is disabled, all irqs will be migratged to another cpu.
>> In some cases, a new affinity is different, it needed to be coppied
>> to irq's affinity. But if the type of irq is LPI, it's affinity will
>> not be coppied because of irq_set_affinity's return value. Fix it by
>> using irq_do_set_affinity.
>>
>> And migrating interrupts is a core code matter, so move the code to
>> kernel/irq/migration.c and select CONFIG_GENERIC_IRQ_MIGRATION when
>> CONFIG_HOTPLUG_CPU and CONFIG_SMP is enabled.
>>
>> Cc: Jiang Liu <jiang.liu@linux.intel.com>
>> Cc: Thomas Gleixner <tglx@linutronix.de>
>> Cc: Marc Zyngier <marc.zyngier@arm.com>
>> Cc: Mark Rutland <mark.rutland@arm.com>
>> Cc: Will Deacon <will.deacon@arm.com>
>> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
>> Cc: Hanjun Guo <hanjun.guo@linaro.org>
>> Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
>> ---
>>   arch/arm/Kconfig             |  1 +
>>   arch/arm/include/asm/irq.h   |  1 -
>>   arch/arm/kernel/irq.c        | 62 --------------------------------------------
>>   arch/arm64/Kconfig           |  1 +
>>   arch/arm64/include/asm/irq.h |  1 -
>>   arch/arm64/kernel/irq.c      | 62 --------------------------------------------
>>   kernel/irq/migration.c       | 62 ++++++++++++++++++++++++++++++++++++++++++++
>>   7 files changed, 64 insertions(+), 126 deletions(-)
>>
>> diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
>> index 41cbb4a..ebc8a33 100644
>> --- a/arch/arm/Kconfig
>> +++ b/arch/arm/Kconfig
>> @@ -22,6 +22,7 @@ config ARM
>>   	select GENERIC_CLOCKEVENTS_BROADCAST if SMP
>>   	select GENERIC_IDLE_POLL_SETUP
>>   	select GENERIC_IRQ_PROBE
>> +	select GENERIC_IRQ_MIGRATION if SMP && HOTPLUG_CPU
>>   	select GENERIC_IRQ_SHOW
>>   	select GENERIC_IRQ_SHOW_LEVEL
>>   	select GENERIC_PCI_IOMAP
>> diff --git a/arch/arm/include/asm/irq.h b/arch/arm/include/asm/irq.h
>> index 53c15de..d17fc900 100644
>> --- a/arch/arm/include/asm/irq.h
>> +++ b/arch/arm/include/asm/irq.h
>> @@ -24,7 +24,6 @@
>>   #ifndef __ASSEMBLY__
>>   struct irqaction;
>>   struct pt_regs;
>> -extern void migrate_irqs(void);
>>
>>   extern void asm_do_IRQ(unsigned int, struct pt_regs *);
>>   void handle_IRQ(unsigned int, struct pt_regs *);
>> diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c
>> index baf8ede..2efdb40 100644
>> --- a/arch/arm/kernel/irq.c
>> +++ b/arch/arm/kernel/irq.c
>> @@ -31,7 +31,6 @@
>>   #include <linux/smp.h>
>>   #include <linux/init.h>
>>   #include <linux/seq_file.h>
>> -#include <linux/ratelimit.h>
>>   #include <linux/errno.h>
>>   #include <linux/list.h>
>>   #include <linux/kallsyms.h>
>> @@ -135,64 +134,3 @@ int __init arch_probe_nr_irqs(void)
>>   	return nr_irqs;
>>   }
>>   #endif
>> -
>> -#ifdef CONFIG_HOTPLUG_CPU
>> -static bool migrate_one_irq(struct irq_desc *desc)
>> -{
>> -	struct irq_data *d = irq_desc_get_irq_data(desc);
>> -	const struct cpumask *affinity = irq_data_get_affinity_mask(d);
>> -	struct irq_chip *c;
>> -	bool ret = false;
>> -
>> -	/*
>> -	 * If this is a per-CPU interrupt, or the affinity does not
>> -	 * include this CPU, then we have nothing to do.
>> -	 */
>> -	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
>> -		return false;
>> -
>> -	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
>> -		affinity = cpu_online_mask;
>> -		ret = true;
>> -	}
>> -
>> -	c = irq_data_get_irq_chip(d);
>> -	if (!c->irq_set_affinity)
>> -		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
>> -	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
>> -		cpumask_copy(irq_data_get_affinity_mask(d), affinity);
>> -
>> -	return ret;
>> -}
>> -
>> -/*
>> - * The current CPU has been marked offline.  Migrate IRQs off this CPU.
>> - * If the affinity settings do not allow other CPUs, force them onto any
>> - * available CPU.
>> - *
>> - * Note: we must iterate over all IRQs, whether they have an attached
>> - * action structure or not, as we need to get chained interrupts too.
>> - */
>> -void migrate_irqs(void)
>> -{
>> -	unsigned int i;
>> -	struct irq_desc *desc;
>> -	unsigned long flags;
>> -
>> -	local_irq_save(flags);
>> -
>> -	for_each_irq_desc(i, desc) {
>> -		bool affinity_broken;
>> -
>> -		raw_spin_lock(&desc->lock);
>> -		affinity_broken = migrate_one_irq(desc);
>> -		raw_spin_unlock(&desc->lock);
>> -
>> -		if (affinity_broken)
>> -			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
>> -				i, smp_processor_id());
>> -	}
>> -
>> -	local_irq_restore(flags);
>> -}
>> -#endif /* CONFIG_HOTPLUG_CPU */
>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>> index b7b9cea..6ffe411 100644
>> --- a/arch/arm64/Kconfig
>> +++ b/arch/arm64/Kconfig
>> @@ -32,6 +32,7 @@ config ARM64
>>   	select GENERIC_CPU_AUTOPROBE
>>   	select GENERIC_EARLY_IOREMAP
>>   	select GENERIC_IRQ_PROBE
>> +	select GENERIC_IRQ_MIGRATION if SMP && HOTPLUG_CPU
>>   	select GENERIC_IRQ_SHOW
>>   	select GENERIC_IRQ_SHOW_LEVEL
>>   	select GENERIC_PCI_IOMAP
>> diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h
>> index bbb251b..0916929 100644
>> --- a/arch/arm64/include/asm/irq.h
>> +++ b/arch/arm64/include/asm/irq.h
>> @@ -7,7 +7,6 @@
>>
>>   struct pt_regs;
>>
>> -extern void migrate_irqs(void);
>>   extern void set_handle_irq(void (*handle_irq)(struct pt_regs *));
>>
>>   static inline void acpi_irq_init(void)
>> diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
>> index 463fa2e..04ac1f6 100644
>> --- a/arch/arm64/kernel/irq.c
>> +++ b/arch/arm64/kernel/irq.c
>> @@ -27,7 +27,6 @@
>>   #include <linux/init.h>
>>   #include <linux/irqchip.h>
>>   #include <linux/seq_file.h>
>> -#include <linux/ratelimit.h>
>>
>>   unsigned long irq_err_count;
>>
>> @@ -56,64 +55,3 @@ void __init init_IRQ(void)
>>   	if (!handle_arch_irq)
>>   		panic("No interrupt controller found.");
>>   }
>> -
>> -#ifdef CONFIG_HOTPLUG_CPU
>> -static bool migrate_one_irq(struct irq_desc *desc)
>> -{
>> -	struct irq_data *d = irq_desc_get_irq_data(desc);
>> -	const struct cpumask *affinity = irq_data_get_affinity_mask(d);
>> -	struct irq_chip *c;
>> -	bool ret = false;
>> -
>> -	/*
>> -	 * If this is a per-CPU interrupt, or the affinity does not
>> -	 * include this CPU, then we have nothing to do.
>> -	 */
>> -	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
>> -		return false;
>> -
>> -	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
>> -		affinity = cpu_online_mask;
>> -		ret = true;
>> -	}
>> -
>> -	c = irq_data_get_irq_chip(d);
>> -	if (!c->irq_set_affinity)
>> -		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
>> -	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
>> -		cpumask_copy(irq_data_get_affinity_mask(d), affinity);
>> -
>> -	return ret;
>> -}
>> -
>> -/*
>> - * The current CPU has been marked offline.  Migrate IRQs off this CPU.
>> - * If the affinity settings do not allow other CPUs, force them onto any
>> - * available CPU.
>> - *
>> - * Note: we must iterate over all IRQs, whether they have an attached
>> - * action structure or not, as we need to get chained interrupts too.
>> - */
>> -void migrate_irqs(void)
>> -{
>> -	unsigned int i;
>> -	struct irq_desc *desc;
>> -	unsigned long flags;
>> -
>> -	local_irq_save(flags);
>> -
>> -	for_each_irq_desc(i, desc) {
>> -		bool affinity_broken;
>> -
>> -		raw_spin_lock(&desc->lock);
>> -		affinity_broken = migrate_one_irq(desc);
>> -		raw_spin_unlock(&desc->lock);
>> -
>> -		if (affinity_broken)
>> -			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
>> -					    i, smp_processor_id());
>> -	}
>> -
>> -	local_irq_restore(flags);
>> -}
>> -#endif /* CONFIG_HOTPLUG_CPU */
>> diff --git a/kernel/irq/migration.c b/kernel/irq/migration.c
>> index 37ddb7b..5801c79 100644
>> --- a/kernel/irq/migration.c
>> +++ b/kernel/irq/migration.c
>> @@ -1,6 +1,7 @@
>>
>>   #include <linux/irq.h>
>>   #include <linux/interrupt.h>
>> +#include <linux/ratelimit.h>
>>
>>   #include "internals.h"
>>
>> @@ -77,3 +78,64 @@ void irq_move_irq(struct irq_data *idata)
>>   	if (!masked)
>>   		idata->chip->irq_unmask(idata);
>>   }
>> +
>> +#ifdef CONFIG_HOTPLUG_CPU
>> +static bool migrate_one_irq(struct irq_desc *desc)
>> +{
>> +	struct irq_data *d = irq_desc_get_irq_data(desc);
>> +	const struct cpumask *affinity = d->affinity;
>> +	struct irq_chip *c;
>> +	bool ret = false;
>> +
>> +	/*
>> +	 * If this is a per-CPU interrupt, or the affinity does not
>> +	 * include this CPU, then we have nothing to do.
>> +	 */
>> +	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
>> +		return false;
>> +
>> +	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
>> +		affinity = cpu_online_mask;
>> +		ret = true;
>> +	}
>> +
>> +	c = irq_data_get_irq_chip(d);
>> +	if (!c->irq_set_affinity)
>> +		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
> How about pr_warn here? It may cause serious drawback if this happens.
I think so, I will change it next version.
>
>> +	else
>> +		irq_do_set_affinity(d, affinity, false);
> Should we check return value here?
>
The return value is not used, so we don't check it.
Maybe I could add some warning by checking it.
>> +
>> +	return ret;
>> +}
>> +
>> +/*
>> + * The current CPU has been marked offline.  Migrate IRQs off this CPU.
>> + * If the affinity settings do not allow other CPUs, force them onto any
>> + * available CPU.
>> + *
>> + * Note: we must iterate over all IRQs, whether they have an attached
>> + * action structure or not, as we need to get chained interrupts too.
>> + */
>> +void migrate_irqs(void)
>> +{
>> +	unsigned int i;
>> +	struct irq_desc *desc;
>> +	unsigned long flags;
>> +
>> +	local_irq_save(flags);
>> +
>> +	for_each_irq_desc(i, desc) {
> Should we use for_each_active_irq() here to iterate over active
> irqs only?
>
It looks good. I will use it next version.
>> +		bool affinity_broken;
>> +
>> +		raw_spin_lock(&desc->lock);
>> +		affinity_broken = migrate_one_irq(desc);
>> +		raw_spin_unlock(&desc->lock);
>> +
>> +		if (affinity_broken)
>> +			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
>> +					    i, smp_processor_id());
>> +	}
>> +
>> +	local_irq_restore(flags);
>> +}
>> +#endif /* CONFIG_HOTPLUG_CPU */
>>
>
> .
>

Re: [RFC PATCH v1 0/4] arm/arm64: fix a migrating irq bug when hotplug cpu

[Yang Yingliang]

On 2015/9/6 16:07, Jiang Liu wrote:
> On 2015/9/6 12:23, Yang Yingliang wrote:
>> Hi All,
>>
>> There is a bug:
>>
>> When cpu is disabled, all irqs will be migratged to another cpu.
>> In some cases, a new affinity is different, it needed to be coppied
>> to irq's affinity. But if the type of irq is LPI, it's affinity will
>> not be coppied because of irq_set_affinity's return value.
>>
>>
>>
>> As Marc and Will suggested, I refactor the arm/arm64 migrating interrupts
>> code and fix the migrating irq bug while cpu is offline.
>>
>> I'm trying let the core code do the migrating interrupts matter. kernel/irq/migration.c
>> depends on CONFIG_GENERIC_PENDING_IRQ, so I make it selected by CONFIG_SMP and
>> CONFIG_HOTPLUG_CPU and rename it to CONFIG_GENERIC_IRQ_MIGRATION for more general.
>> When CONFIG_GENERIC_IRQ_MIGRATION is enabled, an interrupt whose state_use_accessors
>> is not set with IRQD_MOVE_PCNTXT won't be migrated immediately in irq_set_affinity_locked().
>> So introduce irq_settings_set_move_pcntxt() helper to set the state in gic_irq_domain_map().
>>
>> With the above preparation, move the migrating interrupts code into kernel/irq/migration.c
>> and fix the bug by using irq_do_set_affinity().
> Hi Yingliang,
> 	As we are going to move migrate_irqs() to generic kernel
> code, and powerpc, metag, xtensa, sh, ia64 mn10300 also defines
>   migrate_irqs() too. It would be great if we could consolidate
> all these.
> 	And as we are going to refine these code, there's another
> issue need attention. On x86, we need to allocate a CPU vector
> if an irq is directed to a CPU. So there's possibility that
> we run out of CPU vectors after CPU hot-removal. So we have a
> mechanism to detect whether we will run out of CPU vector
> after removing a CPU, and reject CPU hot-removal if that will
> happen.
> 	So the key point is, if we a need to allocate some sort
> of resource on the target CPUs for an irq, we need two steps
> when removing a CPU
> 1) check whether resources are available after removing the CPU,
>     and reject CPU removal request if we ran out of resource
> 2) fix irqs after hot-removing the CPU.
> Thanks!
> Gerry
>

On arm, as I know, it doesn't need extra resource for an irq.
I am not sure other platform need this way besides x86.

I think we could consolidate all migrate_irqs() later. I am not
sure if it's good to do so big changing and modify other arch code in
a patchset that supposed to fix a bug of arm.

[RFC PATCH v1 0/4] arm/arm64: fix a migrating irq bug when hotplug cpu

[Yang Yingliang]

On 2015/9/6 16:07, Jiang Liu wrote:
> On 2015/9/6 12:23, Yang Yingliang wrote:
>> Hi All,
>>
>> There is a bug:
>>
>> When cpu is disabled, all irqs will be migratged to another cpu.
>> In some cases, a new affinity is different, it needed to be coppied
>> to irq's affinity. But if the type of irq is LPI, it's affinity will
>> not be coppied because of irq_set_affinity's return value.
>>
>>
>>
>> As Marc and Will suggested, I refactor the arm/arm64 migrating interrupts
>> code and fix the migrating irq bug while cpu is offline.
>>
>> I'm trying let the core code do the migrating interrupts matter. kernel/irq/migration.c
>> depends on CONFIG_GENERIC_PENDING_IRQ, so I make it selected by CONFIG_SMP and
>> CONFIG_HOTPLUG_CPU and rename it to CONFIG_GENERIC_IRQ_MIGRATION for more general.
>> When CONFIG_GENERIC_IRQ_MIGRATION is enabled, an interrupt whose state_use_accessors
>> is not set with IRQD_MOVE_PCNTXT won't be migrated immediately in irq_set_affinity_locked().
>> So introduce irq_settings_set_move_pcntxt() helper to set the state in gic_irq_domain_map().
>>
>> With the above preparation, move the migrating interrupts code into kernel/irq/migration.c
>> and fix the bug by using irq_do_set_affinity().
> Hi Yingliang,
> 	As we are going to move migrate_irqs() to generic kernel
> code, and powerpc, metag, xtensa, sh, ia64 mn10300 also defines
>   migrate_irqs() too. It would be great if we could consolidate
> all these.
> 	And as we are going to refine these code, there's another
> issue need attention. On x86, we need to allocate a CPU vector
> if an irq is directed to a CPU. So there's possibility that
> we run out of CPU vectors after CPU hot-removal. So we have a
> mechanism to detect whether we will run out of CPU vector
> after removing a CPU, and reject CPU hot-removal if that will
> happen.
> 	So the key point is, if we a need to allocate some sort
> of resource on the target CPUs for an irq, we need two steps
> when removing a CPU
> 1) check whether resources are available after removing the CPU,
>     and reject CPU removal request if we ran out of resource
> 2) fix irqs after hot-removing the CPU.
> Thanks!
> Gerry
>

On arm, as I know, it doesn't need extra resource for an irq.
I am not sure other platform need this way besides x86.

I think we could consolidate all migrate_irqs() later. I am not
sure if it's good to do so big changing and modify other arch code in
a patchset that supposed to fix a bug of arm.

Re: [RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Marc Zyngier]

On 06/09/15 06:56, Jiang Liu wrote:
> On 2015/9/6 12:23, Yang Yingliang wrote:
>> Use irq_settings_set_move_pcntxt() helper irqs status with
>> _IRQ_MOVE_PCNTXT. So that it can do set affinity when calling
>> irq_set_affinity_locked().
> Hi Yingliang,
> 	We could only set _IRQ_MOVE_PCNTCT flag to enable migrating
> IRQ in process context if your hardware platform supports atomically
> change IRQ configuration. Not sure whether that's true for GICv3.
> If GICv3 doesn't support atomically change irq configuration, this
> change may cause trouble.

I think it boils down to what exactly "process context" means here. If
this means "we do not need to mask the interrupt" while moving it, then
it should be fine (the GIC architecture guarantees that a pending
interrupt will be migrated).

Is there any other requirement for this flag?

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

[RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Marc Zyngier]

On 06/09/15 06:56, Jiang Liu wrote:
> On 2015/9/6 12:23, Yang Yingliang wrote:
>> Use irq_settings_set_move_pcntxt() helper irqs status with
>> _IRQ_MOVE_PCNTXT. So that it can do set affinity when calling
>> irq_set_affinity_locked().
> Hi Yingliang,
> 	We could only set _IRQ_MOVE_PCNTCT flag to enable migrating
> IRQ in process context if your hardware platform supports atomically
> change IRQ configuration. Not sure whether that's true for GICv3.
> If GICv3 doesn't support atomically change irq configuration, this
> change may cause trouble.

I think it boils down to what exactly "process context" means here. If
this means "we do not need to mask the interrupt" while moving it, then
it should be fine (the GIC architecture guarantees that a pending
interrupt will be migrated).

Is there any other requirement for this flag?

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Thomas Gleixner]

On Mon, 7 Sep 2015, Marc Zyngier wrote:
> On 06/09/15 06:56, Jiang Liu wrote:
> > On 2015/9/6 12:23, Yang Yingliang wrote:
> >> Use irq_settings_set_move_pcntxt() helper irqs status with
> >> _IRQ_MOVE_PCNTXT. So that it can do set affinity when calling
> >> irq_set_affinity_locked().
> > Hi Yingliang,
> > 	We could only set _IRQ_MOVE_PCNTCT flag to enable migrating
> > IRQ in process context if your hardware platform supports atomically
> > change IRQ configuration. Not sure whether that's true for GICv3.
> > If GICv3 doesn't support atomically change irq configuration, this
> > change may cause trouble.
> 
> I think it boils down to what exactly "process context" means here. If
> this means "we do not need to mask the interrupt" while moving it, then
> it should be fine (the GIC architecture guarantees that a pending
> interrupt will be migrated).
> 
> Is there any other requirement for this flag?

The history of this flag is as follows:

On x86 interrupts can only be safely migrated while the interrupt is
handled. With the introduction of IRQ remapping this requirement
changed. Remapped interrupts can be migrated in any context.

If you look at irq_set_affinity_locked()

   if (irq_can_move_pcntxt(data) {
      irq_do_set_affinity(data,...)
        chip->irq_set_affinity(data,...);
   } else {
      irqd_set_move_pending(data);
   }

So if IRQ_MOVE_PCNTXT is not set, we handle the migration of the
interrupt from next the interrupt. If it's set set_affinity() is
called right away.

All architectures which do not select GENERIC_PENDING_IRQ are using
the direct method.

Thanks,

	tglx

[RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Thomas Gleixner]

On Mon, 7 Sep 2015, Marc Zyngier wrote:
> On 06/09/15 06:56, Jiang Liu wrote:
> > On 2015/9/6 12:23, Yang Yingliang wrote:
> >> Use irq_settings_set_move_pcntxt() helper irqs status with
> >> _IRQ_MOVE_PCNTXT. So that it can do set affinity when calling
> >> irq_set_affinity_locked().
> > Hi Yingliang,
> > 	We could only set _IRQ_MOVE_PCNTCT flag to enable migrating
> > IRQ in process context if your hardware platform supports atomically
> > change IRQ configuration. Not sure whether that's true for GICv3.
> > If GICv3 doesn't support atomically change irq configuration, this
> > change may cause trouble.
> 
> I think it boils down to what exactly "process context" means here. If
> this means "we do not need to mask the interrupt" while moving it, then
> it should be fine (the GIC architecture guarantees that a pending
> interrupt will be migrated).
> 
> Is there any other requirement for this flag?

The history of this flag is as follows:

On x86 interrupts can only be safely migrated while the interrupt is
handled. With the introduction of IRQ remapping this requirement
changed. Remapped interrupts can be migrated in any context.

If you look at irq_set_affinity_locked()

   if (irq_can_move_pcntxt(data) {
      irq_do_set_affinity(data,...)
        chip->irq_set_affinity(data,...);
   } else {
      irqd_set_move_pending(data);
   }

So if IRQ_MOVE_PCNTXT is not set, we handle the migration of the
interrupt from next the interrupt. If it's set set_affinity() is
called right away.

All architectures which do not select GENERIC_PENDING_IRQ are using
the direct method.

Thanks,

	tglx

Re: [RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Marc Zyngier]

Hi Thomas,

On 07/09/15 14:24, Thomas Gleixner wrote:
> On Mon, 7 Sep 2015, Marc Zyngier wrote:
>> On 06/09/15 06:56, Jiang Liu wrote:
>>> On 2015/9/6 12:23, Yang Yingliang wrote:
>>>> Use irq_settings_set_move_pcntxt() helper irqs status with
>>>> _IRQ_MOVE_PCNTXT. So that it can do set affinity when calling
>>>> irq_set_affinity_locked().
>>> Hi Yingliang,
>>> 	We could only set _IRQ_MOVE_PCNTCT flag to enable migrating
>>> IRQ in process context if your hardware platform supports atomically
>>> change IRQ configuration. Not sure whether that's true for GICv3.
>>> If GICv3 doesn't support atomically change irq configuration, this
>>> change may cause trouble.
>>
>> I think it boils down to what exactly "process context" means here. If
>> this means "we do not need to mask the interrupt" while moving it, then
>> it should be fine (the GIC architecture guarantees that a pending
>> interrupt will be migrated).
>>
>> Is there any other requirement for this flag?
> 
> The history of this flag is as follows:
> 
> On x86 interrupts can only be safely migrated while the interrupt is
> handled.

Woa! That's creative! :-) I suppose this doesn't work very well with CPU
hotplug though...

> With the introduction of IRQ remapping this requirement
> changed. Remapped interrupts can be migrated in any context.
> 
> If you look at irq_set_affinity_locked()
> 
>    if (irq_can_move_pcntxt(data) {
>       irq_do_set_affinity(data,...)
>         chip->irq_set_affinity(data,...);
>    } else {
>       irqd_set_move_pending(data);
>    }
> 
> So if IRQ_MOVE_PCNTXT is not set, we handle the migration of the
> interrupt from next the interrupt. If it's set set_affinity() is
> called right away.

OK, that is now starting to make more sense.

> All architectures which do not select GENERIC_PENDING_IRQ are using
> the direct method.

Right. On ARM, only the direct method makes sense so far (we have no
constraint such as the one you describe above).

So I wonder why we bother introducing the IRQ_MOVE_PCNTXT flag on ARM at
all. Is that just because migration.c is only compiled when
GENERIC_PENDING_IRQ is set?

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

[RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Marc Zyngier]

Hi Thomas,

On 07/09/15 14:24, Thomas Gleixner wrote:
> On Mon, 7 Sep 2015, Marc Zyngier wrote:
>> On 06/09/15 06:56, Jiang Liu wrote:
>>> On 2015/9/6 12:23, Yang Yingliang wrote:
>>>> Use irq_settings_set_move_pcntxt() helper irqs status with
>>>> _IRQ_MOVE_PCNTXT. So that it can do set affinity when calling
>>>> irq_set_affinity_locked().
>>> Hi Yingliang,
>>> 	We could only set _IRQ_MOVE_PCNTCT flag to enable migrating
>>> IRQ in process context if your hardware platform supports atomically
>>> change IRQ configuration. Not sure whether that's true for GICv3.
>>> If GICv3 doesn't support atomically change irq configuration, this
>>> change may cause trouble.
>>
>> I think it boils down to what exactly "process context" means here. If
>> this means "we do not need to mask the interrupt" while moving it, then
>> it should be fine (the GIC architecture guarantees that a pending
>> interrupt will be migrated).
>>
>> Is there any other requirement for this flag?
> 
> The history of this flag is as follows:
> 
> On x86 interrupts can only be safely migrated while the interrupt is
> handled.

Woa! That's creative! :-) I suppose this doesn't work very well with CPU
hotplug though...

> With the introduction of IRQ remapping this requirement
> changed. Remapped interrupts can be migrated in any context.
> 
> If you look at irq_set_affinity_locked()
> 
>    if (irq_can_move_pcntxt(data) {
>       irq_do_set_affinity(data,...)
>         chip->irq_set_affinity(data,...);
>    } else {
>       irqd_set_move_pending(data);
>    }
> 
> So if IRQ_MOVE_PCNTXT is not set, we handle the migration of the
> interrupt from next the interrupt. If it's set set_affinity() is
> called right away.

OK, that is now starting to make more sense.

> All architectures which do not select GENERIC_PENDING_IRQ are using
> the direct method.

Right. On ARM, only the direct method makes sense so far (we have no
constraint such as the one you describe above).

So I wonder why we bother introducing the IRQ_MOVE_PCNTXT flag on ARM at
all. Is that just because migration.c is only compiled when
GENERIC_PENDING_IRQ is set?

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Thomas Gleixner]

On Mon, 7 Sep 2015, Marc Zyngier wrote:
> On 07/09/15 14:24, Thomas Gleixner wrote:
> > The history of this flag is as follows:
> > 
> > On x86 interrupts can only be safely migrated while the interrupt is
> > handled.
> 
> Woa! That's creative! :-) I suppose this doesn't work very well with CPU
> hotplug though...

Go figure ....
 
> So I wonder why we bother introducing the IRQ_MOVE_PCNTXT flag on ARM at
> all. Is that just because migration.c is only compiled when
> GENERIC_PENDING_IRQ is set?

Looks like. We can distangle that, if this code needs to be reusable.

Thanks,

	tglx

[RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Thomas Gleixner]

On Mon, 7 Sep 2015, Marc Zyngier wrote:
> On 07/09/15 14:24, Thomas Gleixner wrote:
> > The history of this flag is as follows:
> > 
> > On x86 interrupts can only be safely migrated while the interrupt is
> > handled.
> 
> Woa! That's creative! :-) I suppose this doesn't work very well with CPU
> hotplug though...

Go figure ....
 
> So I wonder why we bother introducing the IRQ_MOVE_PCNTXT flag on ARM at
> all. Is that just because migration.c is only compiled when
> GENERIC_PENDING_IRQ is set?

Looks like. We can distangle that, if this code needs to be reusable.

Thanks,

	tglx

Re: [RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Jiang Liu]

On 2015/9/7 22:56, Marc Zyngier wrote:
> Hi Thomas,
> 
> On 07/09/15 14:24, Thomas Gleixner wrote:
>> On Mon, 7 Sep 2015, Marc Zyngier wrote:
>>> On 06/09/15 06:56, Jiang Liu wrote:
>>>> On 2015/9/6 12:23, Yang Yingliang wrote:
>>>>> Use irq_settings_set_move_pcntxt() helper irqs status with
>>>>> _IRQ_MOVE_PCNTXT. So that it can do set affinity when calling
>>>>> irq_set_affinity_locked().
>>>> Hi Yingliang,
>>>> 	We could only set _IRQ_MOVE_PCNTCT flag to enable migrating
>>>> IRQ in process context if your hardware platform supports atomically
>>>> change IRQ configuration. Not sure whether that's true for GICv3.
>>>> If GICv3 doesn't support atomically change irq configuration, this
>>>> change may cause trouble.
>>>
>>> I think it boils down to what exactly "process context" means here. If
>>> this means "we do not need to mask the interrupt" while moving it, then
>>> it should be fine (the GIC architecture guarantees that a pending
>>> interrupt will be migrated).
>>>
>>> Is there any other requirement for this flag?
>>
>> The history of this flag is as follows:
>>
>> On x86 interrupts can only be safely migrated while the interrupt is
>> handled.
> 
> Woa! That's creative! :-) I suppose this doesn't work very well with CPU
> hotplug though...
X86 has special handling of this case when hot-removing a CPU.
Basically, it does:
1) mask an irq
2) migrate irq to other cpus with set_affinity
3) redirect(retrigger) irq to other CPUs if it's pending on the CPU to
be removed.
Thanks!
Gerry

> 
>> With the introduction of IRQ remapping this requirement
>> changed. Remapped interrupts can be migrated in any context.
>>
>> If you look at irq_set_affinity_locked()
>>
>>    if (irq_can_move_pcntxt(data) {
>>       irq_do_set_affinity(data,...)
>>         chip->irq_set_affinity(data,...);
>>    } else {
>>       irqd_set_move_pending(data);
>>    }
>>
>> So if IRQ_MOVE_PCNTXT is not set, we handle the migration of the
>> interrupt from next the interrupt. If it's set set_affinity() is
>> called right away.
> 
> OK, that is now starting to make more sense.
> 
>> All architectures which do not select GENERIC_PENDING_IRQ are using
>> the direct method.
> 
> Right. On ARM, only the direct method makes sense so far (we have no
> constraint such as the one you describe above).
> 
> So I wonder why we bother introducing the IRQ_MOVE_PCNTXT flag on ARM at
> all. Is that just because migration.c is only compiled when
> GENERIC_PENDING_IRQ is set?
> 
> Thanks,
> 
> 	M.
> 

[RFC PATCH v1 2/4] irqchip: GICv3: set non-percpu irqs status with _IRQ_MOVE_PCNTXT

[Jiang Liu]

On 2015/9/7 22:56, Marc Zyngier wrote:
> Hi Thomas,
> 
> On 07/09/15 14:24, Thomas Gleixner wrote:
>> On Mon, 7 Sep 2015, Marc Zyngier wrote:
>>> On 06/09/15 06:56, Jiang Liu wrote:
>>>> On 2015/9/6 12:23, Yang Yingliang wrote:
>>>>> Use irq_settings_set_move_pcntxt() helper irqs status with
>>>>> _IRQ_MOVE_PCNTXT. So that it can do set affinity when calling
>>>>> irq_set_affinity_locked().
>>>> Hi Yingliang,
>>>> 	We could only set _IRQ_MOVE_PCNTCT flag to enable migrating
>>>> IRQ in process context if your hardware platform supports atomically
>>>> change IRQ configuration. Not sure whether that's true for GICv3.
>>>> If GICv3 doesn't support atomically change irq configuration, this
>>>> change may cause trouble.
>>>
>>> I think it boils down to what exactly "process context" means here. If
>>> this means "we do not need to mask the interrupt" while moving it, then
>>> it should be fine (the GIC architecture guarantees that a pending
>>> interrupt will be migrated).
>>>
>>> Is there any other requirement for this flag?
>>
>> The history of this flag is as follows:
>>
>> On x86 interrupts can only be safely migrated while the interrupt is
>> handled.
> 
> Woa! That's creative! :-) I suppose this doesn't work very well with CPU
> hotplug though...
X86 has special handling of this case when hot-removing a CPU.
Basically, it does:
1) mask an irq
2) migrate irq to other cpus with set_affinity
3) redirect(retrigger) irq to other CPUs if it's pending on the CPU to
be removed.
Thanks!
Gerry

> 
>> With the introduction of IRQ remapping this requirement
>> changed. Remapped interrupts can be migrated in any context.
>>
>> If you look at irq_set_affinity_locked()
>>
>>    if (irq_can_move_pcntxt(data) {
>>       irq_do_set_affinity(data,...)
>>         chip->irq_set_affinity(data,...);
>>    } else {
>>       irqd_set_move_pending(data);
>>    }
>>
>> So if IRQ_MOVE_PCNTXT is not set, we handle the migration of the
>> interrupt from next the interrupt. If it's set set_affinity() is
>> called right away.
> 
> OK, that is now starting to make more sense.
> 
>> All architectures which do not select GENERIC_PENDING_IRQ are using
>> the direct method.
> 
> Right. On ARM, only the direct method makes sense so far (we have no
> constraint such as the one you describe above).
> 
> So I wonder why we bother introducing the IRQ_MOVE_PCNTXT flag on ARM at
> all. Is that just because migration.c is only compiled when
> GENERIC_PENDING_IRQ is set?
> 
> Thanks,
> 
> 	M.
>