* [PATCH 4.2-rc1 v21 0/6] irq/arm: Implement arch_trigger_all_cpu_backtrace
@ 2015-07-06 13:13 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2015-07-06 13:13 UTC (permalink / raw)
To: linux-arm-kernel
This patchset modifies the GIC driver to allow it, on supported
platforms, to route IPI interrupts to FIQ. It then uses this
feature to implement arch_trigger_all_cpu_backtrace for arm.
In order to neatly deliver the changes for the arm we also
rearrange some of the existing x86 NMI code to make it architecture
neutral.
The patches have been runtime tested on both a system capable of
supporting FIQ (Freescale i.MX6) and one that cannot (Qualcomm
Snapdragon 600). In addition older versions of this patchset
have been tested on STiH416 and vexpress-a9. The changes to the x86
logic were tested using qemu.
v21:
* Change the way SGIs are raised to try to increase robustness starting
secondary cores. This is a theoretic fix for a regression reported
by Mark Rutland on vexpress-tc2 but it also allows us to remove
igroup0_shadow entirely since it is no longer needed.
* Fix a couple of variable names and add comments to describe the
hardware behavior better (Mark Rutland).
* Improved MULTI_IRQ_HANDLER support by clearing FIQs using
handle_arch_irq (Marc Zygnier).
* Fix gic_cpu_if_down() to ensure group 1 interrupts are disabled
then the interface is brought down.
For changes in v20 and earlier see:
http://thread.gmane.org/gmane.linux.kernel/1928465
Daniel Thompson (6):
irqchip: gic: Optimize locking in gic_raise_softirq
irqchip: gic: Make gic_raise_softirq FIQ-safe
irqchip: gic: Introduce plumbing for IPI FIQ
printk: Simple implementation for NMI backtracing
x86/nmi: Use common printk functions
ARM: Add support for on-demand backtrace of other CPUs
arch/arm/Kconfig | 1 +
arch/arm/include/asm/hardirq.h | 2 +-
arch/arm/include/asm/irq.h | 5 +
arch/arm/include/asm/smp.h | 3 +
arch/arm/kernel/smp.c | 82 +++++++++++++++
arch/arm/kernel/traps.c | 13 ++-
arch/x86/Kconfig | 1 +
arch/x86/kernel/apic/hw_nmi.c | 104 ++-----------------
drivers/irqchip/irq-gic.c | 220 +++++++++++++++++++++++++++++++++++++---
include/linux/irqchip/arm-gic.h | 6 ++
include/linux/printk.h | 20 ++++
init/Kconfig | 3 +
kernel/printk/Makefile | 1 +
kernel/printk/nmi_backtrace.c | 147 +++++++++++++++++++++++++++
14 files changed, 495 insertions(+), 113 deletions(-)
create mode 100644 kernel/printk/nmi_backtrace.c
--
2.4.3
^ permalink raw reply [flat|nested] 36+ messages in thread
* [PATCH 4.2-rc1 v21 1/6] irqchip: gic: Optimize locking in gic_raise_softirq
2015-07-06 13:13 ` Daniel Thompson
@ 2015-07-06 13:13 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2015-07-06 13:13 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper
Cc: Daniel Thompson, Russell King, Will Deacon, Catalin Marinas,
Marc Zyngier, Stephen Boyd, John Stultz, Steven Rostedt,
linux-kernel, linux-arm-kernel, patches, linaro-kernel,
Sumit Semwal, Dirk Behme, Daniel Drake, Dmitry Pervushin,
Tim Sander
Currently gic_raise_softirq() is locked using irq_controller_lock.
This lock is primarily used to make register read-modify-write sequences
atomic but gic_raise_softirq() uses it instead to ensure that the
big.LITTLE migration logic can figure out when it is safe to migrate
interrupts between physical cores.
This is sub-optimal in closely related ways:
1. No locking at all is required on systems where the b.L switcher is
not configured.
2. Finer grain locking can be used on systems where the b.L switcher is
present.
This patch resolves both of the above by introducing a separate finer
grain lock and providing conditionally compiled inlines to lock/unlock
it.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
---
drivers/irqchip/irq-gic.c | 36 +++++++++++++++++++++++++++++++++---
1 file changed, 33 insertions(+), 3 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index 8d7e1c8b6d56..9ed278902d64 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -75,6 +75,27 @@ struct gic_chip_data {
static DEFINE_RAW_SPINLOCK(irq_controller_lock);
/*
+ * This lock is used by the big.LITTLE migration code to ensure no IPIs
+ * can be pended on the old core after the map has been updated.
+ */
+#ifdef CONFIG_BL_SWITCHER
+static DEFINE_RAW_SPINLOCK(cpu_map_migration_lock);
+
+static inline void gic_migration_lock(unsigned long *flags)
+{
+ raw_spin_lock_irqsave(&cpu_map_migration_lock, *flags);
+}
+
+static inline void gic_migration_unlock(unsigned long flags)
+{
+ raw_spin_unlock_irqrestore(&cpu_map_migration_lock, flags);
+}
+#else
+static inline void gic_migration_lock(unsigned long *flags) {}
+static inline void gic_migration_unlock(unsigned long flags) {}
+#endif
+
+/*
* The GIC mapping of CPU interfaces does not necessarily match
* the logical CPU numbering. Let's use a mapping as returned
* by the GIC itself.
@@ -627,7 +648,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
int cpu;
unsigned long flags, map = 0;
- raw_spin_lock_irqsave(&irq_controller_lock, flags);
+ gic_migration_lock(&flags);
/* Convert our logical CPU mask into a physical one. */
for_each_cpu(cpu, mask)
@@ -642,7 +663,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
/* this always happens on GIC0 */
writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
- raw_spin_unlock_irqrestore(&irq_controller_lock, flags);
+ gic_migration_unlock(flags);
}
#endif
@@ -713,8 +734,17 @@ void gic_migrate_target(unsigned int new_cpu_id)
raw_spin_lock(&irq_controller_lock);
- /* Update the target interface for this logical CPU */
+ /*
+ * Update the target interface for this logical CPU
+ *
+ * From the point we release the cpu_map_migration_lock any new
+ * SGIs will be pended on the new cpu which makes the set of SGIs
+ * pending on the old cpu static. That means we can defer the
+ * migration until after we have released the irq_controller_lock.
+ */
+ raw_spin_lock(&cpu_map_migration_lock);
gic_cpu_map[cpu] = 1 << new_cpu_id;
+ raw_spin_unlock(&cpu_map_migration_lock);
/*
* Find all the peripheral interrupts targetting the current
--
2.4.3
^ permalink raw reply related [flat|nested] 36+ messages in thread* [PATCH 4.2-rc1 v21 1/6] irqchip: gic: Optimize locking in gic_raise_softirq
@ 2015-07-06 13:13 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2015-07-06 13:13 UTC (permalink / raw)
To: linux-arm-kernel
Currently gic_raise_softirq() is locked using irq_controller_lock.
This lock is primarily used to make register read-modify-write sequences
atomic but gic_raise_softirq() uses it instead to ensure that the
big.LITTLE migration logic can figure out when it is safe to migrate
interrupts between physical cores.
This is sub-optimal in closely related ways:
1. No locking at all is required on systems where the b.L switcher is
not configured.
2. Finer grain locking can be used on systems where the b.L switcher is
present.
This patch resolves both of the above by introducing a separate finer
grain lock and providing conditionally compiled inlines to lock/unlock
it.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
---
drivers/irqchip/irq-gic.c | 36 +++++++++++++++++++++++++++++++++---
1 file changed, 33 insertions(+), 3 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index 8d7e1c8b6d56..9ed278902d64 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -75,6 +75,27 @@ struct gic_chip_data {
static DEFINE_RAW_SPINLOCK(irq_controller_lock);
/*
+ * This lock is used by the big.LITTLE migration code to ensure no IPIs
+ * can be pended on the old core after the map has been updated.
+ */
+#ifdef CONFIG_BL_SWITCHER
+static DEFINE_RAW_SPINLOCK(cpu_map_migration_lock);
+
+static inline void gic_migration_lock(unsigned long *flags)
+{
+ raw_spin_lock_irqsave(&cpu_map_migration_lock, *flags);
+}
+
+static inline void gic_migration_unlock(unsigned long flags)
+{
+ raw_spin_unlock_irqrestore(&cpu_map_migration_lock, flags);
+}
+#else
+static inline void gic_migration_lock(unsigned long *flags) {}
+static inline void gic_migration_unlock(unsigned long flags) {}
+#endif
+
+/*
* The GIC mapping of CPU interfaces does not necessarily match
* the logical CPU numbering. Let's use a mapping as returned
* by the GIC itself.
@@ -627,7 +648,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
int cpu;
unsigned long flags, map = 0;
- raw_spin_lock_irqsave(&irq_controller_lock, flags);
+ gic_migration_lock(&flags);
/* Convert our logical CPU mask into a physical one. */
for_each_cpu(cpu, mask)
@@ -642,7 +663,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
/* this always happens on GIC0 */
writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
- raw_spin_unlock_irqrestore(&irq_controller_lock, flags);
+ gic_migration_unlock(flags);
}
#endif
@@ -713,8 +734,17 @@ void gic_migrate_target(unsigned int new_cpu_id)
raw_spin_lock(&irq_controller_lock);
- /* Update the target interface for this logical CPU */
+ /*
+ * Update the target interface for this logical CPU
+ *
+ * From the point we release the cpu_map_migration_lock any new
+ * SGIs will be pended on the new cpu which makes the set of SGIs
+ * pending on the old cpu static. That means we can defer the
+ * migration until after we have released the irq_controller_lock.
+ */
+ raw_spin_lock(&cpu_map_migration_lock);
gic_cpu_map[cpu] = 1 << new_cpu_id;
+ raw_spin_unlock(&cpu_map_migration_lock);
/*
* Find all the peripheral interrupts targetting the current
--
2.4.3
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 4.2-rc1 v21 2/6] irqchip: gic: Make gic_raise_softirq FIQ-safe
2015-07-06 13:13 ` Daniel Thompson
@ 2015-07-06 13:13 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2015-07-06 13:13 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper
Cc: Daniel Thompson, Russell King, Will Deacon, Catalin Marinas,
Marc Zyngier, Stephen Boyd, John Stultz, Steven Rostedt,
linux-kernel, linux-arm-kernel, patches, linaro-kernel,
Sumit Semwal, Dirk Behme, Daniel Drake, Dmitry Pervushin,
Tim Sander
It is currently possible for FIQ handlers to re-enter gic_raise_softirq()
and lock up.
gic_raise_softirq()
lock(x);
-~-> FIQ
handle_fiq()
gic_raise_softirq()
lock(x); <-- Lockup
arch/arm/ uses IPIs to implement arch_irq_work_raise(), thus this issue
renders it difficult for FIQ handlers to safely defer work to less
restrictive calling contexts.
This patch fixes the problem by converting the cpu_map_migration_lock
into a rwlock making it safe to re-enter the function.
Note that having made it safe to re-enter gic_raise_softirq() we no
longer need to mask interrupts during gic_raise_softirq() because the
b.L migration is always performed from task context.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
---
drivers/irqchip/irq-gic.c | 38 +++++++++++++++++++++++++-------------
1 file changed, 25 insertions(+), 13 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index 9ed278902d64..97b227cf3076 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -77,22 +77,25 @@ static DEFINE_RAW_SPINLOCK(irq_controller_lock);
/*
* This lock is used by the big.LITTLE migration code to ensure no IPIs
* can be pended on the old core after the map has been updated.
+ *
+ * This lock may be locked for reading from both IRQ and FIQ handlers
+ * and therefore must not be locked for writing when these are enabled.
*/
#ifdef CONFIG_BL_SWITCHER
-static DEFINE_RAW_SPINLOCK(cpu_map_migration_lock);
+static DEFINE_RWLOCK(cpu_map_migration_lock);
-static inline void gic_migration_lock(unsigned long *flags)
+static inline void gic_migration_lock(void)
{
- raw_spin_lock_irqsave(&cpu_map_migration_lock, *flags);
+ read_lock(&cpu_map_migration_lock);
}
-static inline void gic_migration_unlock(unsigned long flags)
+static inline void gic_migration_unlock(void)
{
- raw_spin_unlock_irqrestore(&cpu_map_migration_lock, flags);
+ read_unlock(&cpu_map_migration_lock);
}
#else
-static inline void gic_migration_lock(unsigned long *flags) {}
-static inline void gic_migration_unlock(unsigned long flags) {}
+static inline void gic_migration_lock(void) {}
+static inline void gic_migration_unlock(void) {}
#endif
/*
@@ -643,12 +646,20 @@ static void __init gic_pm_init(struct gic_chip_data *gic)
#endif
#ifdef CONFIG_SMP
+/*
+ * Raise the specified IPI on all cpus set in mask.
+ *
+ * This function is safe to call from all calling contexts, including
+ * FIQ handlers. It relies on gic_migration_lock() being multiply acquirable
+ * to avoid deadlocks when the function is re-entered at different
+ * exception levels.
+ */
static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
{
int cpu;
- unsigned long flags, map = 0;
+ unsigned long map = 0;
- gic_migration_lock(&flags);
+ gic_migration_lock();
/* Convert our logical CPU mask into a physical one. */
for_each_cpu(cpu, mask)
@@ -663,7 +674,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
/* this always happens on GIC0 */
writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
- gic_migration_unlock(flags);
+ gic_migration_unlock();
}
#endif
@@ -711,7 +722,8 @@ int gic_get_cpu_id(unsigned int cpu)
* Migrate all peripheral interrupts with a target matching the current CPU
* to the interface corresponding to @new_cpu_id. The CPU interface mapping
* is also updated. Targets to other CPU interfaces are unchanged.
- * This must be called with IRQs locally disabled.
+ * This must be called from a task context and with IRQ and FIQ locally
+ * disabled.
*/
void gic_migrate_target(unsigned int new_cpu_id)
{
@@ -742,9 +754,9 @@ void gic_migrate_target(unsigned int new_cpu_id)
* pending on the old cpu static. That means we can defer the
* migration until after we have released the irq_controller_lock.
*/
- raw_spin_lock(&cpu_map_migration_lock);
+ write_lock(&cpu_map_migration_lock);
gic_cpu_map[cpu] = 1 << new_cpu_id;
- raw_spin_unlock(&cpu_map_migration_lock);
+ write_unlock(&cpu_map_migration_lock);
/*
* Find all the peripheral interrupts targetting the current
--
2.4.3
^ permalink raw reply related [flat|nested] 36+ messages in thread* [PATCH 4.2-rc1 v21 2/6] irqchip: gic: Make gic_raise_softirq FIQ-safe
@ 2015-07-06 13:13 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2015-07-06 13:13 UTC (permalink / raw)
To: linux-arm-kernel
It is currently possible for FIQ handlers to re-enter gic_raise_softirq()
and lock up.
gic_raise_softirq()
lock(x);
-~-> FIQ
handle_fiq()
gic_raise_softirq()
lock(x); <-- Lockup
arch/arm/ uses IPIs to implement arch_irq_work_raise(), thus this issue
renders it difficult for FIQ handlers to safely defer work to less
restrictive calling contexts.
This patch fixes the problem by converting the cpu_map_migration_lock
into a rwlock making it safe to re-enter the function.
Note that having made it safe to re-enter gic_raise_softirq() we no
longer need to mask interrupts during gic_raise_softirq() because the
b.L migration is always performed from task context.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
---
drivers/irqchip/irq-gic.c | 38 +++++++++++++++++++++++++-------------
1 file changed, 25 insertions(+), 13 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index 9ed278902d64..97b227cf3076 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -77,22 +77,25 @@ static DEFINE_RAW_SPINLOCK(irq_controller_lock);
/*
* This lock is used by the big.LITTLE migration code to ensure no IPIs
* can be pended on the old core after the map has been updated.
+ *
+ * This lock may be locked for reading from both IRQ and FIQ handlers
+ * and therefore must not be locked for writing when these are enabled.
*/
#ifdef CONFIG_BL_SWITCHER
-static DEFINE_RAW_SPINLOCK(cpu_map_migration_lock);
+static DEFINE_RWLOCK(cpu_map_migration_lock);
-static inline void gic_migration_lock(unsigned long *flags)
+static inline void gic_migration_lock(void)
{
- raw_spin_lock_irqsave(&cpu_map_migration_lock, *flags);
+ read_lock(&cpu_map_migration_lock);
}
-static inline void gic_migration_unlock(unsigned long flags)
+static inline void gic_migration_unlock(void)
{
- raw_spin_unlock_irqrestore(&cpu_map_migration_lock, flags);
+ read_unlock(&cpu_map_migration_lock);
}
#else
-static inline void gic_migration_lock(unsigned long *flags) {}
-static inline void gic_migration_unlock(unsigned long flags) {}
+static inline void gic_migration_lock(void) {}
+static inline void gic_migration_unlock(void) {}
#endif
/*
@@ -643,12 +646,20 @@ static void __init gic_pm_init(struct gic_chip_data *gic)
#endif
#ifdef CONFIG_SMP
+/*
+ * Raise the specified IPI on all cpus set in mask.
+ *
+ * This function is safe to call from all calling contexts, including
+ * FIQ handlers. It relies on gic_migration_lock() being multiply acquirable
+ * to avoid deadlocks when the function is re-entered at different
+ * exception levels.
+ */
static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
{
int cpu;
- unsigned long flags, map = 0;
+ unsigned long map = 0;
- gic_migration_lock(&flags);
+ gic_migration_lock();
/* Convert our logical CPU mask into a physical one. */
for_each_cpu(cpu, mask)
@@ -663,7 +674,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
/* this always happens on GIC0 */
writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
- gic_migration_unlock(flags);
+ gic_migration_unlock();
}
#endif
@@ -711,7 +722,8 @@ int gic_get_cpu_id(unsigned int cpu)
* Migrate all peripheral interrupts with a target matching the current CPU
* to the interface corresponding to @new_cpu_id. The CPU interface mapping
* is also updated. Targets to other CPU interfaces are unchanged.
- * This must be called with IRQs locally disabled.
+ * This must be called from a task context and with IRQ and FIQ locally
+ * disabled.
*/
void gic_migrate_target(unsigned int new_cpu_id)
{
@@ -742,9 +754,9 @@ void gic_migrate_target(unsigned int new_cpu_id)
* pending on the old cpu static. That means we can defer the
* migration until after we have released the irq_controller_lock.
*/
- raw_spin_lock(&cpu_map_migration_lock);
+ write_lock(&cpu_map_migration_lock);
gic_cpu_map[cpu] = 1 << new_cpu_id;
- raw_spin_unlock(&cpu_map_migration_lock);
+ write_unlock(&cpu_map_migration_lock);
/*
* Find all the peripheral interrupts targetting the current
--
2.4.3
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 4.2-rc1 v21 3/6] irqchip: gic: Introduce plumbing for IPI FIQ
2015-07-06 13:13 ` Daniel Thompson
@ 2015-07-06 13:13 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2015-07-06 13:13 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper
Cc: Daniel Thompson, Russell King, Will Deacon, Catalin Marinas,
Marc Zyngier, Stephen Boyd, John Stultz, Steven Rostedt,
linux-kernel, linux-arm-kernel, patches, linaro-kernel,
Sumit Semwal, Dirk Behme, Daniel Drake, Dmitry Pervushin,
Tim Sander
Currently it is not possible to exploit FIQ for systems with a GIC, even
on systems are otherwise capable of it. This patch makes it possible
for IPIs to be delivered using FIQ.
To do so it modifies the register state so that normal interrupts are
placed in group 1 and specific IPIs are placed into group 0. It also
configures the controller to raise group 0 interrupts using the FIQ
signal. Finally it provides a means for architecture code to define
which IPIs shall use FIQ and to acknowledge any IPIs that are raised.
All GIC hardware except GICv1-without-TrustZone support provides a means
to group exceptions into group 0 and group 1 but the hardware
functionality is unavailable to the kernel when a secure monitor is
present because access to the grouping registers are prohibited outside
"secure world". However when grouping is not available (or in the case
of early GICv1 implementations is very hard to configure) the code to
change groups does not deploy and all IPIs will be raised via IRQ.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Tested-by: Jon Medhurst <tixy@linaro.org>
---
arch/arm/kernel/traps.c | 9 ++-
drivers/irqchip/irq-gic.c | 168 +++++++++++++++++++++++++++++++++++++---
include/linux/irqchip/arm-gic.h | 6 ++
3 files changed, 171 insertions(+), 12 deletions(-)
diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c
index d358226236f2..5634823a39cf 100644
--- a/arch/arm/kernel/traps.c
+++ b/arch/arm/kernel/traps.c
@@ -479,7 +479,14 @@ asmlinkage void __exception_irq_entry handle_fiq_as_nmi(struct pt_regs *regs)
nmi_enter();
- /* nop. FIQ handlers for special arch/arm features can be added here. */
+ /*
+ * Either the interrupt controller supports FIQ, meaning it will
+ * do the right thing with this call, or we will end up treating a
+ * spurious FIQ (which is normally fatal) as though it were an IRQ
+ * which, although it risks deadlock, still gives us a sporting
+ * chance of surviving long enough to log errors.
+ */
+ handle_arch_irq(regs);
nmi_exit();
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index 97b227cf3076..77d14beb0cc8 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -41,6 +41,7 @@
#include <linux/irqchip/chained_irq.h>
#include <linux/irqchip/arm-gic.h>
#include <linux/irqchip/arm-gic-acpi.h>
+#include <linux/ratelimit.h>
#include <asm/cputype.h>
#include <asm/irq.h>
@@ -50,6 +51,10 @@
#include "irq-gic-common.h"
#include "irqchip.h"
+#ifndef SMP_IPI_FIQ_MASK
+#define SMP_IPI_FIQ_MASK 0
+#endif
+
union gic_base {
void __iomem *common_base;
void __percpu * __iomem *percpu_base;
@@ -67,6 +72,7 @@ struct gic_chip_data {
#endif
struct irq_domain *domain;
unsigned int gic_irqs;
+ bool sgi_with_nsatt;
#ifdef CONFIG_GIC_NON_BANKED
void __iomem *(*get_base)(union gic_base *);
#endif
@@ -285,12 +291,39 @@ static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
}
#endif
+/*
+ * Fully acknowledge (both ack and eoi) any outstanding FIQ-based IPI,
+ * otherwise do nothing.
+ */
+void gic_handle_fiq(struct pt_regs *regs)
+{
+ struct gic_chip_data *gic = &gic_data[0];
+ void __iomem *cpu_base = gic_data_cpu_base(gic);
+ unsigned long irqstat, irqnr;
+
+ while ((1u << readl_relaxed(cpu_base + GIC_CPU_HIGHPRI)) &
+ SMP_IPI_FIQ_MASK) {
+ irqstat = readl_relaxed(cpu_base + GIC_CPU_INTACK);
+ writel_relaxed(irqstat, cpu_base + GIC_CPU_EOI);
+
+ irqnr = irqstat & GICC_IAR_INT_ID_MASK;
+ WARN_RATELIMIT(irqnr > 16,
+ "Unexpected irqnr %lu (bad prioritization?)\n",
+ irqnr);
+ }
+}
+
static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
{
u32 irqstat, irqnr;
struct gic_chip_data *gic = &gic_data[0];
void __iomem *cpu_base = gic_data_cpu_base(gic);
+ if (in_nmi()) {
+ gic_handle_fiq(regs);
+ return;
+ }
+
do {
irqstat = readl_relaxed(cpu_base + GIC_CPU_INTACK);
irqnr = irqstat & GICC_IAR_INT_ID_MASK;
@@ -351,6 +384,55 @@ static struct irq_chip gic_chip = {
.flags = IRQCHIP_SET_TYPE_MASKED,
};
+/*
+ * Shift an interrupt between Group 0 and Group 1.
+ *
+ * In addition to changing the group we also modify the priority to
+ * match what "ARM strongly recommends" for a system where no Group 1
+ * interrupt must ever preempt a Group 0 interrupt.
+ *
+ * If is safe to call this function on systems which do not support
+ * grouping (it will have no effect).
+ */
+static void gic_set_group_irq(struct gic_chip_data *gic, unsigned int hwirq,
+ int group)
+{
+ void __iomem *base = gic_data_dist_base(gic);
+ unsigned int grp_reg = hwirq / 32 * 4;
+ u32 grp_mask = BIT(hwirq % 32);
+ u32 grp_val;
+
+ unsigned int pri_reg = (hwirq / 4) * 4;
+ u32 pri_mask = BIT(7 + ((hwirq % 4) * 8));
+ u32 pri_val;
+
+ /*
+ * Systems which do not support grouping will have not have
+ * the EnableGrp1 bit set.
+ */
+ if (!(GICD_ENABLE_GRP1 & readl_relaxed(base + GIC_DIST_CTRL)))
+ return;
+
+ raw_spin_lock(&irq_controller_lock);
+
+ grp_val = readl_relaxed(base + GIC_DIST_IGROUP + grp_reg);
+ pri_val = readl_relaxed(base + GIC_DIST_PRI + pri_reg);
+
+ if (group) {
+ grp_val |= grp_mask;
+ pri_val |= pri_mask;
+ } else {
+ grp_val &= ~grp_mask;
+ pri_val &= ~pri_mask;
+ }
+
+ writel_relaxed(grp_val, base + GIC_DIST_IGROUP + grp_reg);
+ writel_relaxed(pri_val, base + GIC_DIST_PRI + pri_reg);
+
+ raw_spin_unlock(&irq_controller_lock);
+}
+
+
void __init gic_cascade_irq(unsigned int gic_nr, unsigned int irq)
{
if (gic_nr >= MAX_GIC_NR)
@@ -382,15 +464,24 @@ static u8 gic_get_cpumask(struct gic_chip_data *gic)
static void gic_cpu_if_up(void)
{
void __iomem *cpu_base = gic_data_cpu_base(&gic_data[0]);
- u32 bypass = 0;
+ void __iomem *dist_base = gic_data_dist_base(&gic_data[0]);
+ u32 ctrl = 0;
/*
- * Preserve bypass disable bits to be written back later
- */
- bypass = readl(cpu_base + GIC_CPU_CTRL);
- bypass &= GICC_DIS_BYPASS_MASK;
+ * Preserve bypass disable bits to be written back later
+ */
+ ctrl = readl(cpu_base + GIC_CPU_CTRL);
+ ctrl &= GICC_DIS_BYPASS_MASK;
+
+ /*
+ * If EnableGrp1 is set in the distributor then enable group 1
+ * support for this CPU (and route group 0 interrupts to FIQ).
+ */
+ if (GICD_ENABLE_GRP1 & readl_relaxed(dist_base + GIC_DIST_CTRL))
+ ctrl |= GICC_COMMON_BPR | GICC_FIQ_EN | GICC_ACK_CTL |
+ GICC_ENABLE_GRP1;
- writel_relaxed(bypass | GICC_ENABLE, cpu_base + GIC_CPU_CTRL);
+ writel_relaxed(ctrl | GICC_ENABLE, cpu_base + GIC_CPU_CTRL);
}
@@ -414,7 +505,34 @@ static void __init gic_dist_init(struct gic_chip_data *gic)
gic_dist_config(base, gic_irqs, NULL);
- writel_relaxed(GICD_ENABLE, base + GIC_DIST_CTRL);
+ /*
+ * Set EnableGrp1/EnableGrp0 (bit 1 and 0) or EnableGrp (bit 0 only,
+ * bit 1 ignored) depending on current mode.
+ */
+ writel_relaxed(GICD_ENABLE_GRP1 | GICD_ENABLE, base + GIC_DIST_CTRL);
+
+ /*
+ * Some GICv1 devices (even those with security extensions) do not
+ * implement EnableGrp1 meaning some parts of the above write might
+ * be ignored. We will only enable FIQ support if the bit can be set.
+ */
+ if (GICD_ENABLE_GRP1 & readl_relaxed(base + GIC_DIST_CTRL)) {
+ /*
+ * Set all global interrupts to be group 1 (signalled with
+ * IRQ).
+ */
+ for (i = 32; i < gic_irqs; i += 32)
+ writel_relaxed(0xffffffff,
+ base + GIC_DIST_IGROUP + i * 4 / 32);
+
+ /*
+ * If the GIC supports the security extension then SGIs
+ * will be filtered based on the value of NSATT. If the
+ * GIC has this support then enable NSATT support.
+ */
+ if (GICD_SECURITY_EXTN & readl_relaxed(base + GIC_DIST_CTR))
+ gic->sgi_with_nsatt = true;
+ }
}
static void gic_cpu_init(struct gic_chip_data *gic)
@@ -423,6 +541,7 @@ static void gic_cpu_init(struct gic_chip_data *gic)
void __iomem *base = gic_data_cpu_base(gic);
unsigned int cpu_mask, cpu = smp_processor_id();
int i;
+ unsigned long ipi_fiq_mask, fiq;
/*
* Get what the GIC says our CPU mask is.
@@ -441,6 +560,23 @@ static void gic_cpu_init(struct gic_chip_data *gic)
gic_cpu_config(dist_base, NULL);
+ /*
+ * If the distributor is configured to support interrupt grouping
+ * then set any PPI and SGI interrupts not set in SMP_IPI_FIQ_MASK
+ * to be group1 and ensure any remaining group 0 interrupts have
+ * the right priority.
+ *
+ * Note that IGROUP[0] is banked, meaning that although we are
+ * writing to a distributor register we are actually performing
+ * part of the per-cpu initialization.
+ */
+ if (GICD_ENABLE_GRP1 & readl_relaxed(dist_base + GIC_DIST_CTRL)) {
+ ipi_fiq_mask = SMP_IPI_FIQ_MASK;
+ writel_relaxed(~ipi_fiq_mask, dist_base + GIC_DIST_IGROUP + 0);
+ for_each_set_bit(fiq, &ipi_fiq_mask, 16)
+ gic_set_group_irq(gic, fiq, 0);
+ }
+
writel_relaxed(GICC_INT_PRI_THRESHOLD, base + GIC_CPU_PRIMASK);
gic_cpu_if_up();
}
@@ -451,7 +587,8 @@ void gic_cpu_if_down(void)
u32 val = 0;
val = readl(cpu_base + GIC_CPU_CTRL);
- val &= ~GICC_ENABLE;
+ val &= ~(GICC_COMMON_BPR | GICC_FIQ_EN | GICC_ACK_CTL |
+ GICC_ENABLE_GRP1 | GICC_ENABLE);
writel_relaxed(val, cpu_base + GIC_CPU_CTRL);
}
@@ -530,7 +667,8 @@ static void gic_dist_restore(unsigned int gic_nr)
writel_relaxed(gic_data[gic_nr].saved_spi_enable[i],
dist_base + GIC_DIST_ENABLE_SET + i * 4);
- writel_relaxed(GICD_ENABLE, dist_base + GIC_DIST_CTRL);
+ writel_relaxed(GICD_ENABLE_GRP1 | GICD_ENABLE,
+ dist_base + GIC_DIST_CTRL);
}
static void gic_cpu_save(unsigned int gic_nr)
@@ -658,6 +796,8 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
{
int cpu;
unsigned long map = 0;
+ unsigned long softint;
+ void __iomem *dist_base;
gic_migration_lock();
@@ -665,14 +805,20 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
for_each_cpu(cpu, mask)
map |= gic_cpu_map[cpu];
+ /* This always happens on GIC0 */
+ dist_base = gic_data_dist_base(&gic_data[0]);
+
/*
* Ensure that stores to Normal memory are visible to the
* other CPUs before they observe us issuing the IPI.
*/
dmb(ishst);
- /* this always happens on GIC0 */
- writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
+ softint = map << 16 | irq;
+
+ writel_relaxed(softint, dist_base + GIC_DIST_SOFTINT);
+ if (gic_data[0].sgi_with_nsatt)
+ writel_relaxed(softint | 0x8000, dist_base + GIC_DIST_SOFTINT);
gic_migration_unlock();
}
diff --git a/include/linux/irqchip/arm-gic.h b/include/linux/irqchip/arm-gic.h
index 9de976b4f9a7..249554388142 100644
--- a/include/linux/irqchip/arm-gic.h
+++ b/include/linux/irqchip/arm-gic.h
@@ -22,6 +22,10 @@
#define GIC_CPU_IDENT 0xfc
#define GICC_ENABLE 0x1
+#define GICC_ENABLE_GRP1 0x2
+#define GICC_ACK_CTL 0x4
+#define GICC_FIQ_EN 0x8
+#define GICC_COMMON_BPR 0x10
#define GICC_INT_PRI_THRESHOLD 0xf0
#define GICC_IAR_INT_ID_MASK 0x3ff
#define GICC_INT_SPURIOUS 1023
@@ -44,7 +48,9 @@
#define GIC_DIST_SGI_PENDING_SET 0xf20
#define GICD_ENABLE 0x1
+#define GICD_ENABLE_GRP1 0x2
#define GICD_DISABLE 0x0
+#define GICD_SECURITY_EXTN 0x400
#define GICD_INT_ACTLOW_LVLTRIG 0x0
#define GICD_INT_EN_CLR_X32 0xffffffff
#define GICD_INT_EN_SET_SGI 0x0000ffff
--
2.4.3
^ permalink raw reply related [flat|nested] 36+ messages in thread* [PATCH 4.2-rc1 v21 3/6] irqchip: gic: Introduce plumbing for IPI FIQ
@ 2015-07-06 13:13 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2015-07-06 13:13 UTC (permalink / raw)
To: linux-arm-kernel
Currently it is not possible to exploit FIQ for systems with a GIC, even
on systems are otherwise capable of it. This patch makes it possible
for IPIs to be delivered using FIQ.
To do so it modifies the register state so that normal interrupts are
placed in group 1 and specific IPIs are placed into group 0. It also
configures the controller to raise group 0 interrupts using the FIQ
signal. Finally it provides a means for architecture code to define
which IPIs shall use FIQ and to acknowledge any IPIs that are raised.
All GIC hardware except GICv1-without-TrustZone support provides a means
to group exceptions into group 0 and group 1 but the hardware
functionality is unavailable to the kernel when a secure monitor is
present because access to the grouping registers are prohibited outside
"secure world". However when grouping is not available (or in the case
of early GICv1 implementations is very hard to configure) the code to
change groups does not deploy and all IPIs will be raised via IRQ.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Tested-by: Jon Medhurst <tixy@linaro.org>
---
arch/arm/kernel/traps.c | 9 ++-
drivers/irqchip/irq-gic.c | 168 +++++++++++++++++++++++++++++++++++++---
include/linux/irqchip/arm-gic.h | 6 ++
3 files changed, 171 insertions(+), 12 deletions(-)
diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c
index d358226236f2..5634823a39cf 100644
--- a/arch/arm/kernel/traps.c
+++ b/arch/arm/kernel/traps.c
@@ -479,7 +479,14 @@ asmlinkage void __exception_irq_entry handle_fiq_as_nmi(struct pt_regs *regs)
nmi_enter();
- /* nop. FIQ handlers for special arch/arm features can be added here. */
+ /*
+ * Either the interrupt controller supports FIQ, meaning it will
+ * do the right thing with this call, or we will end up treating a
+ * spurious FIQ (which is normally fatal) as though it were an IRQ
+ * which, although it risks deadlock, still gives us a sporting
+ * chance of surviving long enough to log errors.
+ */
+ handle_arch_irq(regs);
nmi_exit();
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index 97b227cf3076..77d14beb0cc8 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -41,6 +41,7 @@
#include <linux/irqchip/chained_irq.h>
#include <linux/irqchip/arm-gic.h>
#include <linux/irqchip/arm-gic-acpi.h>
+#include <linux/ratelimit.h>
#include <asm/cputype.h>
#include <asm/irq.h>
@@ -50,6 +51,10 @@
#include "irq-gic-common.h"
#include "irqchip.h"
+#ifndef SMP_IPI_FIQ_MASK
+#define SMP_IPI_FIQ_MASK 0
+#endif
+
union gic_base {
void __iomem *common_base;
void __percpu * __iomem *percpu_base;
@@ -67,6 +72,7 @@ struct gic_chip_data {
#endif
struct irq_domain *domain;
unsigned int gic_irqs;
+ bool sgi_with_nsatt;
#ifdef CONFIG_GIC_NON_BANKED
void __iomem *(*get_base)(union gic_base *);
#endif
@@ -285,12 +291,39 @@ static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
}
#endif
+/*
+ * Fully acknowledge (both ack and eoi) any outstanding FIQ-based IPI,
+ * otherwise do nothing.
+ */
+void gic_handle_fiq(struct pt_regs *regs)
+{
+ struct gic_chip_data *gic = &gic_data[0];
+ void __iomem *cpu_base = gic_data_cpu_base(gic);
+ unsigned long irqstat, irqnr;
+
+ while ((1u << readl_relaxed(cpu_base + GIC_CPU_HIGHPRI)) &
+ SMP_IPI_FIQ_MASK) {
+ irqstat = readl_relaxed(cpu_base + GIC_CPU_INTACK);
+ writel_relaxed(irqstat, cpu_base + GIC_CPU_EOI);
+
+ irqnr = irqstat & GICC_IAR_INT_ID_MASK;
+ WARN_RATELIMIT(irqnr > 16,
+ "Unexpected irqnr %lu (bad prioritization?)\n",
+ irqnr);
+ }
+}
+
static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
{
u32 irqstat, irqnr;
struct gic_chip_data *gic = &gic_data[0];
void __iomem *cpu_base = gic_data_cpu_base(gic);
+ if (in_nmi()) {
+ gic_handle_fiq(regs);
+ return;
+ }
+
do {
irqstat = readl_relaxed(cpu_base + GIC_CPU_INTACK);
irqnr = irqstat & GICC_IAR_INT_ID_MASK;
@@ -351,6 +384,55 @@ static struct irq_chip gic_chip = {
.flags = IRQCHIP_SET_TYPE_MASKED,
};
+/*
+ * Shift an interrupt between Group 0 and Group 1.
+ *
+ * In addition to changing the group we also modify the priority to
+ * match what "ARM strongly recommends" for a system where no Group 1
+ * interrupt must ever preempt a Group 0 interrupt.
+ *
+ * If is safe to call this function on systems which do not support
+ * grouping (it will have no effect).
+ */
+static void gic_set_group_irq(struct gic_chip_data *gic, unsigned int hwirq,
+ int group)
+{
+ void __iomem *base = gic_data_dist_base(gic);
+ unsigned int grp_reg = hwirq / 32 * 4;
+ u32 grp_mask = BIT(hwirq % 32);
+ u32 grp_val;
+
+ unsigned int pri_reg = (hwirq / 4) * 4;
+ u32 pri_mask = BIT(7 + ((hwirq % 4) * 8));
+ u32 pri_val;
+
+ /*
+ * Systems which do not support grouping will have not have
+ * the EnableGrp1 bit set.
+ */
+ if (!(GICD_ENABLE_GRP1 & readl_relaxed(base + GIC_DIST_CTRL)))
+ return;
+
+ raw_spin_lock(&irq_controller_lock);
+
+ grp_val = readl_relaxed(base + GIC_DIST_IGROUP + grp_reg);
+ pri_val = readl_relaxed(base + GIC_DIST_PRI + pri_reg);
+
+ if (group) {
+ grp_val |= grp_mask;
+ pri_val |= pri_mask;
+ } else {
+ grp_val &= ~grp_mask;
+ pri_val &= ~pri_mask;
+ }
+
+ writel_relaxed(grp_val, base + GIC_DIST_IGROUP + grp_reg);
+ writel_relaxed(pri_val, base + GIC_DIST_PRI + pri_reg);
+
+ raw_spin_unlock(&irq_controller_lock);
+}
+
+
void __init gic_cascade_irq(unsigned int gic_nr, unsigned int irq)
{
if (gic_nr >= MAX_GIC_NR)
@@ -382,15 +464,24 @@ static u8 gic_get_cpumask(struct gic_chip_data *gic)
static void gic_cpu_if_up(void)
{
void __iomem *cpu_base = gic_data_cpu_base(&gic_data[0]);
- u32 bypass = 0;
+ void __iomem *dist_base = gic_data_dist_base(&gic_data[0]);
+ u32 ctrl = 0;
/*
- * Preserve bypass disable bits to be written back later
- */
- bypass = readl(cpu_base + GIC_CPU_CTRL);
- bypass &= GICC_DIS_BYPASS_MASK;
+ * Preserve bypass disable bits to be written back later
+ */
+ ctrl = readl(cpu_base + GIC_CPU_CTRL);
+ ctrl &= GICC_DIS_BYPASS_MASK;
+
+ /*
+ * If EnableGrp1 is set in the distributor then enable group 1
+ * support for this CPU (and route group 0 interrupts to FIQ).
+ */
+ if (GICD_ENABLE_GRP1 & readl_relaxed(dist_base + GIC_DIST_CTRL))
+ ctrl |= GICC_COMMON_BPR | GICC_FIQ_EN | GICC_ACK_CTL |
+ GICC_ENABLE_GRP1;
- writel_relaxed(bypass | GICC_ENABLE, cpu_base + GIC_CPU_CTRL);
+ writel_relaxed(ctrl | GICC_ENABLE, cpu_base + GIC_CPU_CTRL);
}
@@ -414,7 +505,34 @@ static void __init gic_dist_init(struct gic_chip_data *gic)
gic_dist_config(base, gic_irqs, NULL);
- writel_relaxed(GICD_ENABLE, base + GIC_DIST_CTRL);
+ /*
+ * Set EnableGrp1/EnableGrp0 (bit 1 and 0) or EnableGrp (bit 0 only,
+ * bit 1 ignored) depending on current mode.
+ */
+ writel_relaxed(GICD_ENABLE_GRP1 | GICD_ENABLE, base + GIC_DIST_CTRL);
+
+ /*
+ * Some GICv1 devices (even those with security extensions) do not
+ * implement EnableGrp1 meaning some parts of the above write might
+ * be ignored. We will only enable FIQ support if the bit can be set.
+ */
+ if (GICD_ENABLE_GRP1 & readl_relaxed(base + GIC_DIST_CTRL)) {
+ /*
+ * Set all global interrupts to be group 1 (signalled with
+ * IRQ).
+ */
+ for (i = 32; i < gic_irqs; i += 32)
+ writel_relaxed(0xffffffff,
+ base + GIC_DIST_IGROUP + i * 4 / 32);
+
+ /*
+ * If the GIC supports the security extension then SGIs
+ * will be filtered based on the value of NSATT. If the
+ * GIC has this support then enable NSATT support.
+ */
+ if (GICD_SECURITY_EXTN & readl_relaxed(base + GIC_DIST_CTR))
+ gic->sgi_with_nsatt = true;
+ }
}
static void gic_cpu_init(struct gic_chip_data *gic)
@@ -423,6 +541,7 @@ static void gic_cpu_init(struct gic_chip_data *gic)
void __iomem *base = gic_data_cpu_base(gic);
unsigned int cpu_mask, cpu = smp_processor_id();
int i;
+ unsigned long ipi_fiq_mask, fiq;
/*
* Get what the GIC says our CPU mask is.
@@ -441,6 +560,23 @@ static void gic_cpu_init(struct gic_chip_data *gic)
gic_cpu_config(dist_base, NULL);
+ /*
+ * If the distributor is configured to support interrupt grouping
+ * then set any PPI and SGI interrupts not set in SMP_IPI_FIQ_MASK
+ * to be group1 and ensure any remaining group 0 interrupts have
+ * the right priority.
+ *
+ * Note that IGROUP[0] is banked, meaning that although we are
+ * writing to a distributor register we are actually performing
+ * part of the per-cpu initialization.
+ */
+ if (GICD_ENABLE_GRP1 & readl_relaxed(dist_base + GIC_DIST_CTRL)) {
+ ipi_fiq_mask = SMP_IPI_FIQ_MASK;
+ writel_relaxed(~ipi_fiq_mask, dist_base + GIC_DIST_IGROUP + 0);
+ for_each_set_bit(fiq, &ipi_fiq_mask, 16)
+ gic_set_group_irq(gic, fiq, 0);
+ }
+
writel_relaxed(GICC_INT_PRI_THRESHOLD, base + GIC_CPU_PRIMASK);
gic_cpu_if_up();
}
@@ -451,7 +587,8 @@ void gic_cpu_if_down(void)
u32 val = 0;
val = readl(cpu_base + GIC_CPU_CTRL);
- val &= ~GICC_ENABLE;
+ val &= ~(GICC_COMMON_BPR | GICC_FIQ_EN | GICC_ACK_CTL |
+ GICC_ENABLE_GRP1 | GICC_ENABLE);
writel_relaxed(val, cpu_base + GIC_CPU_CTRL);
}
@@ -530,7 +667,8 @@ static void gic_dist_restore(unsigned int gic_nr)
writel_relaxed(gic_data[gic_nr].saved_spi_enable[i],
dist_base + GIC_DIST_ENABLE_SET + i * 4);
- writel_relaxed(GICD_ENABLE, dist_base + GIC_DIST_CTRL);
+ writel_relaxed(GICD_ENABLE_GRP1 | GICD_ENABLE,
+ dist_base + GIC_DIST_CTRL);
}
static void gic_cpu_save(unsigned int gic_nr)
@@ -658,6 +796,8 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
{
int cpu;
unsigned long map = 0;
+ unsigned long softint;
+ void __iomem *dist_base;
gic_migration_lock();
@@ -665,14 +805,20 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
for_each_cpu(cpu, mask)
map |= gic_cpu_map[cpu];
+ /* This always happens on GIC0 */
+ dist_base = gic_data_dist_base(&gic_data[0]);
+
/*
* Ensure that stores to Normal memory are visible to the
* other CPUs before they observe us issuing the IPI.
*/
dmb(ishst);
- /* this always happens on GIC0 */
- writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
+ softint = map << 16 | irq;
+
+ writel_relaxed(softint, dist_base + GIC_DIST_SOFTINT);
+ if (gic_data[0].sgi_with_nsatt)
+ writel_relaxed(softint | 0x8000, dist_base + GIC_DIST_SOFTINT);
gic_migration_unlock();
}
diff --git a/include/linux/irqchip/arm-gic.h b/include/linux/irqchip/arm-gic.h
index 9de976b4f9a7..249554388142 100644
--- a/include/linux/irqchip/arm-gic.h
+++ b/include/linux/irqchip/arm-gic.h
@@ -22,6 +22,10 @@
#define GIC_CPU_IDENT 0xfc
#define GICC_ENABLE 0x1
+#define GICC_ENABLE_GRP1 0x2
+#define GICC_ACK_CTL 0x4
+#define GICC_FIQ_EN 0x8
+#define GICC_COMMON_BPR 0x10
#define GICC_INT_PRI_THRESHOLD 0xf0
#define GICC_IAR_INT_ID_MASK 0x3ff
#define GICC_INT_SPURIOUS 1023
@@ -44,7 +48,9 @@
#define GIC_DIST_SGI_PENDING_SET 0xf20
#define GICD_ENABLE 0x1
+#define GICD_ENABLE_GRP1 0x2
#define GICD_DISABLE 0x0
+#define GICD_SECURITY_EXTN 0x400
#define GICD_INT_ACTLOW_LVLTRIG 0x0
#define GICD_INT_EN_CLR_X32 0xffffffff
#define GICD_INT_EN_SET_SGI 0x0000ffff
--
2.4.3
^ permalink raw reply related [flat|nested] 36+ messages in thread* Re: [PATCH 4.2-rc1 v21 3/6] irqchip: gic: Introduce plumbing for IPI FIQ
2015-07-06 13:13 ` Daniel Thompson
@ 2015-07-07 10:05 ` Holger Schurig
-1 siblings, 0 replies; 36+ messages in thread
From: Holger Schurig @ 2015-07-07 10:05 UTC (permalink / raw)
To: Daniel Thompson
Cc: Thomas Gleixner, Jason Cooper, linaro-kernel, Russell King,
patches, Marc Zyngier, Stephen Boyd, Will Deacon, open list,
Steven Rostedt, Daniel Drake, Dmitry Pervushin, Dirk Behme,
John Stultz, Tim Sander, Catalin Marinas, Sumit Semwal,
linux-arm-kernel
> Currently it is not possible to exploit FIQ for systems with a GIC, even
> on systems are otherwise capable of it. This patch makes it possible
> for IPIs to be delivered using FIQ.
I wonder if gic_set_group_irq() can easily be married with mxc_set_irq_fiq().
The driver sound/soc/fsl/imx-pcm-fiq.c uses mxc_set_irq_fiq() to
connect an (assembly) FIQ handler with a hardware IRQ. However, the
underlying implementation of mxc_set_irq_fiq() is only supported for
TZIC and AVIC (see arch/arm/mach-imx/irq-common.c).
On my i.MX6, which doesn't have an AVIC/TZIC but a GIC I could use a
method to turn a normal IRQ into a FIQ :-)
^ permalink raw reply [flat|nested] 36+ messages in thread
* [PATCH 4.2-rc1 v21 3/6] irqchip: gic: Introduce plumbing for IPI FIQ
@ 2015-07-07 10:05 ` Holger Schurig
0 siblings, 0 replies; 36+ messages in thread
From: Holger Schurig @ 2015-07-07 10:05 UTC (permalink / raw)
To: linux-arm-kernel
> Currently it is not possible to exploit FIQ for systems with a GIC, even
> on systems are otherwise capable of it. This patch makes it possible
> for IPIs to be delivered using FIQ.
I wonder if gic_set_group_irq() can easily be married with mxc_set_irq_fiq().
The driver sound/soc/fsl/imx-pcm-fiq.c uses mxc_set_irq_fiq() to
connect an (assembly) FIQ handler with a hardware IRQ. However, the
underlying implementation of mxc_set_irq_fiq() is only supported for
TZIC and AVIC (see arch/arm/mach-imx/irq-common.c).
On my i.MX6, which doesn't have an AVIC/TZIC but a GIC I could use a
method to turn a normal IRQ into a FIQ :-)
^ permalink raw reply [flat|nested] 36+ messages in thread
* [PATCH 4.2-rc1 v21 4/6] printk: Simple implementation for NMI backtracing
2015-07-06 13:13 ` Daniel Thompson
@ 2015-07-06 13:13 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2015-07-06 13:13 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper
Cc: Daniel Thompson, Russell King, Will Deacon, Catalin Marinas,
Marc Zyngier, Stephen Boyd, John Stultz, Steven Rostedt,
linux-kernel, linux-arm-kernel, patches, linaro-kernel,
Sumit Semwal, Dirk Behme, Daniel Drake, Dmitry Pervushin,
Tim Sander
Currently there is a quite a pile of code sitting in
arch/x86/kernel/apic/hw_nmi.c to support safe all-cpu backtracing from NMI.
The code is inaccessible to backtrace implementations for other
architectures, which is a shame because they would probably like to be
safe too.
Copy this code into printk, reworking it a little as we do so to make
it easier to exploit as library code.
We'll port the x86 NMI backtrace logic to it in a later patch.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
---
include/linux/printk.h | 20 ++++++
init/Kconfig | 3 +
kernel/printk/Makefile | 1 +
kernel/printk/nmi_backtrace.c | 147 ++++++++++++++++++++++++++++++++++++++++++
4 files changed, 171 insertions(+)
create mode 100644 kernel/printk/nmi_backtrace.c
diff --git a/include/linux/printk.h b/include/linux/printk.h
index 58b1fec40d37..07da23b10f57 100644
--- a/include/linux/printk.h
+++ b/include/linux/printk.h
@@ -230,6 +230,26 @@ static inline void show_regs_print_info(const char *log_lvl)
}
#endif
+#ifdef CONFIG_PRINTK_NMI_BACKTRACE
+/*
+ * printk_nmi_backtrace_prepare/complete are called to prepare the
+ * system for some or all cores to issue trace from NMI.
+ * printk_nmi_backtrace_complete will print buffered output and cannot
+ * (safely) be called from NMI.
+ */
+extern int printk_nmi_backtrace_prepare(void);
+extern void printk_nmi_backtrace_complete(void);
+
+/*
+ * printk_nmi_backtrace_this_cpu_begin/end are used divert/restore printk
+ * on this cpu. The result is the output of printk() (by this CPU) will be
+ * stored in temporary buffers for later printing by
+ * printk_nmi_backtrace_complete.
+ */
+extern void printk_nmi_backtrace_this_cpu_begin(void);
+extern void printk_nmi_backtrace_this_cpu_end(void);
+#endif
+
extern asmlinkage void dump_stack(void) __cold;
#ifndef pr_fmt
diff --git a/init/Kconfig b/init/Kconfig
index af09b4fb43d2..132e4f05253e 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1439,6 +1439,9 @@ config PRINTK
very difficult to diagnose system problems, saying N here is
strongly discouraged.
+config PRINTK_NMI_BACKTRACE
+ bool
+
config BUG
bool "BUG() support" if EXPERT
default y
diff --git a/kernel/printk/Makefile b/kernel/printk/Makefile
index 85405bdcf2b3..1849b001384a 100644
--- a/kernel/printk/Makefile
+++ b/kernel/printk/Makefile
@@ -1,2 +1,3 @@
obj-y = printk.o
+obj-$(CONFIG_PRINTK_NMI_BACKTRACE) += nmi_backtrace.o
obj-$(CONFIG_A11Y_BRAILLE_CONSOLE) += braille.o
diff --git a/kernel/printk/nmi_backtrace.c b/kernel/printk/nmi_backtrace.c
new file mode 100644
index 000000000000..f24761262756
--- /dev/null
+++ b/kernel/printk/nmi_backtrace.c
@@ -0,0 +1,147 @@
+#include <linux/kernel.h>
+#include <linux/seq_buf.h>
+
+#define NMI_BUF_SIZE 4096
+
+struct nmi_seq_buf {
+ unsigned char buffer[NMI_BUF_SIZE];
+ struct seq_buf seq;
+};
+
+/* Safe printing in NMI context */
+static DEFINE_PER_CPU(struct nmi_seq_buf, nmi_print_seq);
+
+static DEFINE_PER_CPU(printk_func_t, nmi_print_saved_print_func);
+
+/* "in progress" flag of NMI printing */
+static unsigned long nmi_print_flag;
+
+static int __init printk_nmi_backtrace_init(void)
+{
+ struct nmi_seq_buf *s;
+ int cpu;
+
+ for_each_possible_cpu(cpu) {
+ s = &per_cpu(nmi_print_seq, cpu);
+ seq_buf_init(&s->seq, s->buffer, NMI_BUF_SIZE);
+ }
+
+ return 0;
+}
+pure_initcall(printk_nmi_backtrace_init);
+
+/*
+ * It is not safe to call printk() directly from NMI handlers.
+ * It may be fine if the NMI detected a lock up and we have no choice
+ * but to do so, but doing a NMI on all other CPUs to get a back trace
+ * can be done with a sysrq-l. We don't want that to lock up, which
+ * can happen if the NMI interrupts a printk in progress.
+ *
+ * Instead, we redirect the vprintk() to this nmi_vprintk() that writes
+ * the content into a per cpu seq_buf buffer. Then when the NMIs are
+ * all done, we can safely dump the contents of the seq_buf to a printk()
+ * from a non NMI context.
+ *
+ * This is not a generic printk() implementation and must be used with
+ * great care. In particular there is a static limit on the quantity of
+ * data that may be emitted during NMI, only one client can be active at
+ * one time (arbitrated by the return value of printk_nmi_begin() and
+ * it is required that something at task or interrupt context be scheduled
+ * to issue the output.
+ */
+static int nmi_vprintk(const char *fmt, va_list args)
+{
+ struct nmi_seq_buf *s = this_cpu_ptr(&nmi_print_seq);
+ unsigned int len = seq_buf_used(&s->seq);
+
+ seq_buf_vprintf(&s->seq, fmt, args);
+ return seq_buf_used(&s->seq) - len;
+}
+
+/*
+ * Reserve the NMI printk mechanism. Return an error if some other component
+ * is already using it.
+ */
+int printk_nmi_backtrace_prepare(void)
+{
+ if (test_and_set_bit(0, &nmi_print_flag)) {
+ /*
+ * If something is already using the NMI print facility we
+ * can't allow a second one...
+ */
+ return -EBUSY;
+ }
+
+ return 0;
+}
+
+static void print_seq_line(struct nmi_seq_buf *s, int start, int end)
+{
+ const char *buf = s->buffer + start;
+
+ printk("%.*s", (end - start) + 1, buf);
+}
+
+void printk_nmi_backtrace_complete(void)
+{
+ struct nmi_seq_buf *s;
+ int len, cpu, i, last_i;
+
+ /*
+ * Now that all the NMIs have triggered, we can dump out their
+ * back traces safely to the console.
+ */
+ for_each_possible_cpu(cpu) {
+ s = &per_cpu(nmi_print_seq, cpu);
+ last_i = 0;
+
+ len = seq_buf_used(&s->seq);
+ if (!len)
+ continue;
+
+ /* Print line by line. */
+ for (i = 0; i < len; i++) {
+ if (s->buffer[i] == '\n') {
+ print_seq_line(s, last_i, i);
+ last_i = i + 1;
+ }
+ }
+ /* Check if there was a partial line. */
+ if (last_i < len) {
+ print_seq_line(s, last_i, len - 1);
+ pr_cont("\n");
+ }
+
+ /* Wipe out the buffer ready for the next time around. */
+ seq_buf_clear(&s->seq);
+ }
+
+ clear_bit(0, &nmi_print_flag);
+}
+
+void printk_nmi_backtrace_this_cpu_begin(void)
+{
+ /*
+ * Detect double-begins and report them. This code is unsafe (because
+ * it will print from NMI) but things are pretty badly damaged if the
+ * NMI re-enters and is somehow granted permission to use NMI printk,
+ * so how much worse can it get? Also since this code interferes with
+ * the operation of printk it is unlikely that any consequential
+ * failures will be able to log anything making this our last
+ * opportunity to tell anyone that something is wrong.
+ */
+ if (this_cpu_read(nmi_print_saved_print_func)) {
+ this_cpu_write(printk_func,
+ this_cpu_read(nmi_print_saved_print_func));
+ BUG();
+ }
+
+ this_cpu_write(nmi_print_saved_print_func, this_cpu_read(printk_func));
+ this_cpu_write(printk_func, nmi_vprintk);
+}
+
+void printk_nmi_backtrace_this_cpu_end(void)
+{
+ this_cpu_write(printk_func, this_cpu_read(nmi_print_saved_print_func));
+ this_cpu_write(nmi_print_saved_print_func, NULL);
+}
--
2.4.3
^ permalink raw reply related [flat|nested] 36+ messages in thread* [PATCH 4.2-rc1 v21 4/6] printk: Simple implementation for NMI backtracing
@ 2015-07-06 13:13 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2015-07-06 13:13 UTC (permalink / raw)
To: linux-arm-kernel
Currently there is a quite a pile of code sitting in
arch/x86/kernel/apic/hw_nmi.c to support safe all-cpu backtracing from NMI.
The code is inaccessible to backtrace implementations for other
architectures, which is a shame because they would probably like to be
safe too.
Copy this code into printk, reworking it a little as we do so to make
it easier to exploit as library code.
We'll port the x86 NMI backtrace logic to it in a later patch.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
---
include/linux/printk.h | 20 ++++++
init/Kconfig | 3 +
kernel/printk/Makefile | 1 +
kernel/printk/nmi_backtrace.c | 147 ++++++++++++++++++++++++++++++++++++++++++
4 files changed, 171 insertions(+)
create mode 100644 kernel/printk/nmi_backtrace.c
diff --git a/include/linux/printk.h b/include/linux/printk.h
index 58b1fec40d37..07da23b10f57 100644
--- a/include/linux/printk.h
+++ b/include/linux/printk.h
@@ -230,6 +230,26 @@ static inline void show_regs_print_info(const char *log_lvl)
}
#endif
+#ifdef CONFIG_PRINTK_NMI_BACKTRACE
+/*
+ * printk_nmi_backtrace_prepare/complete are called to prepare the
+ * system for some or all cores to issue trace from NMI.
+ * printk_nmi_backtrace_complete will print buffered output and cannot
+ * (safely) be called from NMI.
+ */
+extern int printk_nmi_backtrace_prepare(void);
+extern void printk_nmi_backtrace_complete(void);
+
+/*
+ * printk_nmi_backtrace_this_cpu_begin/end are used divert/restore printk
+ * on this cpu. The result is the output of printk() (by this CPU) will be
+ * stored in temporary buffers for later printing by
+ * printk_nmi_backtrace_complete.
+ */
+extern void printk_nmi_backtrace_this_cpu_begin(void);
+extern void printk_nmi_backtrace_this_cpu_end(void);
+#endif
+
extern asmlinkage void dump_stack(void) __cold;
#ifndef pr_fmt
diff --git a/init/Kconfig b/init/Kconfig
index af09b4fb43d2..132e4f05253e 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1439,6 +1439,9 @@ config PRINTK
very difficult to diagnose system problems, saying N here is
strongly discouraged.
+config PRINTK_NMI_BACKTRACE
+ bool
+
config BUG
bool "BUG() support" if EXPERT
default y
diff --git a/kernel/printk/Makefile b/kernel/printk/Makefile
index 85405bdcf2b3..1849b001384a 100644
--- a/kernel/printk/Makefile
+++ b/kernel/printk/Makefile
@@ -1,2 +1,3 @@
obj-y = printk.o
+obj-$(CONFIG_PRINTK_NMI_BACKTRACE) += nmi_backtrace.o
obj-$(CONFIG_A11Y_BRAILLE_CONSOLE) += braille.o
diff --git a/kernel/printk/nmi_backtrace.c b/kernel/printk/nmi_backtrace.c
new file mode 100644
index 000000000000..f24761262756
--- /dev/null
+++ b/kernel/printk/nmi_backtrace.c
@@ -0,0 +1,147 @@
+#include <linux/kernel.h>
+#include <linux/seq_buf.h>
+
+#define NMI_BUF_SIZE 4096
+
+struct nmi_seq_buf {
+ unsigned char buffer[NMI_BUF_SIZE];
+ struct seq_buf seq;
+};
+
+/* Safe printing in NMI context */
+static DEFINE_PER_CPU(struct nmi_seq_buf, nmi_print_seq);
+
+static DEFINE_PER_CPU(printk_func_t, nmi_print_saved_print_func);
+
+/* "in progress" flag of NMI printing */
+static unsigned long nmi_print_flag;
+
+static int __init printk_nmi_backtrace_init(void)
+{
+ struct nmi_seq_buf *s;
+ int cpu;
+
+ for_each_possible_cpu(cpu) {
+ s = &per_cpu(nmi_print_seq, cpu);
+ seq_buf_init(&s->seq, s->buffer, NMI_BUF_SIZE);
+ }
+
+ return 0;
+}
+pure_initcall(printk_nmi_backtrace_init);
+
+/*
+ * It is not safe to call printk() directly from NMI handlers.
+ * It may be fine if the NMI detected a lock up and we have no choice
+ * but to do so, but doing a NMI on all other CPUs to get a back trace
+ * can be done with a sysrq-l. We don't want that to lock up, which
+ * can happen if the NMI interrupts a printk in progress.
+ *
+ * Instead, we redirect the vprintk() to this nmi_vprintk() that writes
+ * the content into a per cpu seq_buf buffer. Then when the NMIs are
+ * all done, we can safely dump the contents of the seq_buf to a printk()
+ * from a non NMI context.
+ *
+ * This is not a generic printk() implementation and must be used with
+ * great care. In particular there is a static limit on the quantity of
+ * data that may be emitted during NMI, only one client can be active at
+ * one time (arbitrated by the return value of printk_nmi_begin() and
+ * it is required that something at task or interrupt context be scheduled
+ * to issue the output.
+ */
+static int nmi_vprintk(const char *fmt, va_list args)
+{
+ struct nmi_seq_buf *s = this_cpu_ptr(&nmi_print_seq);
+ unsigned int len = seq_buf_used(&s->seq);
+
+ seq_buf_vprintf(&s->seq, fmt, args);
+ return seq_buf_used(&s->seq) - len;
+}
+
+/*
+ * Reserve the NMI printk mechanism. Return an error if some other component
+ * is already using it.
+ */
+int printk_nmi_backtrace_prepare(void)
+{
+ if (test_and_set_bit(0, &nmi_print_flag)) {
+ /*
+ * If something is already using the NMI print facility we
+ * can't allow a second one...
+ */
+ return -EBUSY;
+ }
+
+ return 0;
+}
+
+static void print_seq_line(struct nmi_seq_buf *s, int start, int end)
+{
+ const char *buf = s->buffer + start;
+
+ printk("%.*s", (end - start) + 1, buf);
+}
+
+void printk_nmi_backtrace_complete(void)
+{
+ struct nmi_seq_buf *s;
+ int len, cpu, i, last_i;
+
+ /*
+ * Now that all the NMIs have triggered, we can dump out their
+ * back traces safely to the console.
+ */
+ for_each_possible_cpu(cpu) {
+ s = &per_cpu(nmi_print_seq, cpu);
+ last_i = 0;
+
+ len = seq_buf_used(&s->seq);
+ if (!len)
+ continue;
+
+ /* Print line by line. */
+ for (i = 0; i < len; i++) {
+ if (s->buffer[i] == '\n') {
+ print_seq_line(s, last_i, i);
+ last_i = i + 1;
+ }
+ }
+ /* Check if there was a partial line. */
+ if (last_i < len) {
+ print_seq_line(s, last_i, len - 1);
+ pr_cont("\n");
+ }
+
+ /* Wipe out the buffer ready for the next time around. */
+ seq_buf_clear(&s->seq);
+ }
+
+ clear_bit(0, &nmi_print_flag);
+}
+
+void printk_nmi_backtrace_this_cpu_begin(void)
+{
+ /*
+ * Detect double-begins and report them. This code is unsafe (because
+ * it will print from NMI) but things are pretty badly damaged if the
+ * NMI re-enters and is somehow granted permission to use NMI printk,
+ * so how much worse can it get? Also since this code interferes with
+ * the operation of printk it is unlikely that any consequential
+ * failures will be able to log anything making this our last
+ * opportunity to tell anyone that something is wrong.
+ */
+ if (this_cpu_read(nmi_print_saved_print_func)) {
+ this_cpu_write(printk_func,
+ this_cpu_read(nmi_print_saved_print_func));
+ BUG();
+ }
+
+ this_cpu_write(nmi_print_saved_print_func, this_cpu_read(printk_func));
+ this_cpu_write(printk_func, nmi_vprintk);
+}
+
+void printk_nmi_backtrace_this_cpu_end(void)
+{
+ this_cpu_write(printk_func, this_cpu_read(nmi_print_saved_print_func));
+ this_cpu_write(nmi_print_saved_print_func, NULL);
+}
--
2.4.3
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 4.2-rc1 v21 5/6] x86/nmi: Use common printk functions
2015-07-06 13:13 ` Daniel Thompson
@ 2015-07-06 13:13 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2015-07-06 13:13 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper
Cc: Daniel Thompson, Russell King, Will Deacon, Catalin Marinas,
Marc Zyngier, Stephen Boyd, John Stultz, Steven Rostedt,
linux-kernel, linux-arm-kernel, patches, linaro-kernel,
Sumit Semwal, Dirk Behme, Daniel Drake, Dmitry Pervushin,
Tim Sander, H. Peter Anvin, x86
Much of the code sitting in arch/x86/kernel/apic/hw_nmi.c to support safe
all-cpu backtracing from NMI has been copied to printk.c to make it
accessible to other architectures.
Port the x86 NMI backtrace to the generic code.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
---
arch/x86/Kconfig | 1 +
arch/x86/kernel/apic/hw_nmi.c | 104 ++++--------------------------------------
2 files changed, 10 insertions(+), 95 deletions(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 55bced17dc95..efc3510a3930 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -149,6 +149,7 @@ config X86
select VIRT_TO_BUS
select X86_DEV_DMA_OPS if X86_64
select X86_FEATURE_NAMES if PROC_FS
+ select PRINTK_NMI_BACKTRACE if X86_LOCAL_APIC
config INSTRUCTION_DECODER
def_bool y
diff --git a/arch/x86/kernel/apic/hw_nmi.c b/arch/x86/kernel/apic/hw_nmi.c
index 6873ab925d00..7a682beac3a0 100644
--- a/arch/x86/kernel/apic/hw_nmi.c
+++ b/arch/x86/kernel/apic/hw_nmi.c
@@ -30,40 +30,17 @@ u64 hw_nmi_get_sample_period(int watchdog_thresh)
#ifdef arch_trigger_all_cpu_backtrace
/* For reliability, we're prepared to waste bits here. */
static DECLARE_BITMAP(backtrace_mask, NR_CPUS) __read_mostly;
-static cpumask_t printtrace_mask;
-
-#define NMI_BUF_SIZE 4096
-
-struct nmi_seq_buf {
- unsigned char buffer[NMI_BUF_SIZE];
- struct seq_buf seq;
-};
-
-/* Safe printing in NMI context */
-static DEFINE_PER_CPU(struct nmi_seq_buf, nmi_print_seq);
-
-/* "in progress" flag of arch_trigger_all_cpu_backtrace */
-static unsigned long backtrace_flag;
-
-static void print_seq_line(struct nmi_seq_buf *s, int start, int end)
-{
- const char *buf = s->buffer + start;
-
- printk("%.*s", (end - start) + 1, buf);
-}
void arch_trigger_all_cpu_backtrace(bool include_self)
{
- struct nmi_seq_buf *s;
- int len;
- int cpu;
- int i;
+ int err, i;
int this_cpu = get_cpu();
- if (test_and_set_bit(0, &backtrace_flag)) {
+ err = printk_nmi_backtrace_prepare();
+ if (err) {
/*
- * If there is already a trigger_all_cpu_backtrace() in progress
- * (backtrace_flag == 1), don't output double cpu dump infos.
+ * If there is already an nmi printk sequence in
+ * progress then just give up...
*/
put_cpu();
return;
@@ -73,16 +50,6 @@ void arch_trigger_all_cpu_backtrace(bool include_self)
if (!include_self)
cpumask_clear_cpu(this_cpu, to_cpumask(backtrace_mask));
- cpumask_copy(&printtrace_mask, to_cpumask(backtrace_mask));
- /*
- * Set up per_cpu seq_buf buffers that the NMIs running on the other
- * CPUs will write to.
- */
- for_each_cpu(cpu, to_cpumask(backtrace_mask)) {
- s = &per_cpu(nmi_print_seq, cpu);
- seq_buf_init(&s->seq, s->buffer, NMI_BUF_SIZE);
- }
-
if (!cpumask_empty(to_cpumask(backtrace_mask))) {
pr_info("sending NMI to %s CPUs:\n",
(include_self ? "all" : "other"));
@@ -97,73 +64,20 @@ void arch_trigger_all_cpu_backtrace(bool include_self)
touch_softlockup_watchdog();
}
- /*
- * Now that all the NMIs have triggered, we can dump out their
- * back traces safely to the console.
- */
- for_each_cpu(cpu, &printtrace_mask) {
- int last_i = 0;
-
- s = &per_cpu(nmi_print_seq, cpu);
- len = seq_buf_used(&s->seq);
- if (!len)
- continue;
-
- /* Print line by line. */
- for (i = 0; i < len; i++) {
- if (s->buffer[i] == '\n') {
- print_seq_line(s, last_i, i);
- last_i = i + 1;
- }
- }
- /* Check if there was a partial line. */
- if (last_i < len) {
- print_seq_line(s, last_i, len - 1);
- pr_cont("\n");
- }
- }
-
- clear_bit(0, &backtrace_flag);
- smp_mb__after_atomic();
+ printk_nmi_backtrace_complete();
put_cpu();
}
-/*
- * It is not safe to call printk() directly from NMI handlers.
- * It may be fine if the NMI detected a lock up and we have no choice
- * but to do so, but doing a NMI on all other CPUs to get a back trace
- * can be done with a sysrq-l. We don't want that to lock up, which
- * can happen if the NMI interrupts a printk in progress.
- *
- * Instead, we redirect the vprintk() to this nmi_vprintk() that writes
- * the content into a per cpu seq_buf buffer. Then when the NMIs are
- * all done, we can safely dump the contents of the seq_buf to a printk()
- * from a non NMI context.
- */
-static int nmi_vprintk(const char *fmt, va_list args)
-{
- struct nmi_seq_buf *s = this_cpu_ptr(&nmi_print_seq);
- unsigned int len = seq_buf_used(&s->seq);
-
- seq_buf_vprintf(&s->seq, fmt, args);
- return seq_buf_used(&s->seq) - len;
-}
-
static int
arch_trigger_all_cpu_backtrace_handler(unsigned int cmd, struct pt_regs *regs)
{
- int cpu;
-
- cpu = smp_processor_id();
+ int cpu = smp_processor_id();
if (cpumask_test_cpu(cpu, to_cpumask(backtrace_mask))) {
- printk_func_t printk_func_save = this_cpu_read(printk_func);
-
- /* Replace printk to write into the NMI seq */
- this_cpu_write(printk_func, nmi_vprintk);
+ printk_nmi_backtrace_this_cpu_begin();
printk(KERN_WARNING "NMI backtrace for cpu %d\n", cpu);
show_regs(regs);
- this_cpu_write(printk_func, printk_func_save);
+ printk_nmi_backtrace_this_cpu_end();
cpumask_clear_cpu(cpu, to_cpumask(backtrace_mask));
return NMI_HANDLED;
--
2.4.3
^ permalink raw reply related [flat|nested] 36+ messages in thread* [PATCH 4.2-rc1 v21 5/6] x86/nmi: Use common printk functions
@ 2015-07-06 13:13 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2015-07-06 13:13 UTC (permalink / raw)
To: linux-arm-kernel
Much of the code sitting in arch/x86/kernel/apic/hw_nmi.c to support safe
all-cpu backtracing from NMI has been copied to printk.c to make it
accessible to other architectures.
Port the x86 NMI backtrace to the generic code.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86 at kernel.org
---
arch/x86/Kconfig | 1 +
arch/x86/kernel/apic/hw_nmi.c | 104 ++++--------------------------------------
2 files changed, 10 insertions(+), 95 deletions(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 55bced17dc95..efc3510a3930 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -149,6 +149,7 @@ config X86
select VIRT_TO_BUS
select X86_DEV_DMA_OPS if X86_64
select X86_FEATURE_NAMES if PROC_FS
+ select PRINTK_NMI_BACKTRACE if X86_LOCAL_APIC
config INSTRUCTION_DECODER
def_bool y
diff --git a/arch/x86/kernel/apic/hw_nmi.c b/arch/x86/kernel/apic/hw_nmi.c
index 6873ab925d00..7a682beac3a0 100644
--- a/arch/x86/kernel/apic/hw_nmi.c
+++ b/arch/x86/kernel/apic/hw_nmi.c
@@ -30,40 +30,17 @@ u64 hw_nmi_get_sample_period(int watchdog_thresh)
#ifdef arch_trigger_all_cpu_backtrace
/* For reliability, we're prepared to waste bits here. */
static DECLARE_BITMAP(backtrace_mask, NR_CPUS) __read_mostly;
-static cpumask_t printtrace_mask;
-
-#define NMI_BUF_SIZE 4096
-
-struct nmi_seq_buf {
- unsigned char buffer[NMI_BUF_SIZE];
- struct seq_buf seq;
-};
-
-/* Safe printing in NMI context */
-static DEFINE_PER_CPU(struct nmi_seq_buf, nmi_print_seq);
-
-/* "in progress" flag of arch_trigger_all_cpu_backtrace */
-static unsigned long backtrace_flag;
-
-static void print_seq_line(struct nmi_seq_buf *s, int start, int end)
-{
- const char *buf = s->buffer + start;
-
- printk("%.*s", (end - start) + 1, buf);
-}
void arch_trigger_all_cpu_backtrace(bool include_self)
{
- struct nmi_seq_buf *s;
- int len;
- int cpu;
- int i;
+ int err, i;
int this_cpu = get_cpu();
- if (test_and_set_bit(0, &backtrace_flag)) {
+ err = printk_nmi_backtrace_prepare();
+ if (err) {
/*
- * If there is already a trigger_all_cpu_backtrace() in progress
- * (backtrace_flag == 1), don't output double cpu dump infos.
+ * If there is already an nmi printk sequence in
+ * progress then just give up...
*/
put_cpu();
return;
@@ -73,16 +50,6 @@ void arch_trigger_all_cpu_backtrace(bool include_self)
if (!include_self)
cpumask_clear_cpu(this_cpu, to_cpumask(backtrace_mask));
- cpumask_copy(&printtrace_mask, to_cpumask(backtrace_mask));
- /*
- * Set up per_cpu seq_buf buffers that the NMIs running on the other
- * CPUs will write to.
- */
- for_each_cpu(cpu, to_cpumask(backtrace_mask)) {
- s = &per_cpu(nmi_print_seq, cpu);
- seq_buf_init(&s->seq, s->buffer, NMI_BUF_SIZE);
- }
-
if (!cpumask_empty(to_cpumask(backtrace_mask))) {
pr_info("sending NMI to %s CPUs:\n",
(include_self ? "all" : "other"));
@@ -97,73 +64,20 @@ void arch_trigger_all_cpu_backtrace(bool include_self)
touch_softlockup_watchdog();
}
- /*
- * Now that all the NMIs have triggered, we can dump out their
- * back traces safely to the console.
- */
- for_each_cpu(cpu, &printtrace_mask) {
- int last_i = 0;
-
- s = &per_cpu(nmi_print_seq, cpu);
- len = seq_buf_used(&s->seq);
- if (!len)
- continue;
-
- /* Print line by line. */
- for (i = 0; i < len; i++) {
- if (s->buffer[i] == '\n') {
- print_seq_line(s, last_i, i);
- last_i = i + 1;
- }
- }
- /* Check if there was a partial line. */
- if (last_i < len) {
- print_seq_line(s, last_i, len - 1);
- pr_cont("\n");
- }
- }
-
- clear_bit(0, &backtrace_flag);
- smp_mb__after_atomic();
+ printk_nmi_backtrace_complete();
put_cpu();
}
-/*
- * It is not safe to call printk() directly from NMI handlers.
- * It may be fine if the NMI detected a lock up and we have no choice
- * but to do so, but doing a NMI on all other CPUs to get a back trace
- * can be done with a sysrq-l. We don't want that to lock up, which
- * can happen if the NMI interrupts a printk in progress.
- *
- * Instead, we redirect the vprintk() to this nmi_vprintk() that writes
- * the content into a per cpu seq_buf buffer. Then when the NMIs are
- * all done, we can safely dump the contents of the seq_buf to a printk()
- * from a non NMI context.
- */
-static int nmi_vprintk(const char *fmt, va_list args)
-{
- struct nmi_seq_buf *s = this_cpu_ptr(&nmi_print_seq);
- unsigned int len = seq_buf_used(&s->seq);
-
- seq_buf_vprintf(&s->seq, fmt, args);
- return seq_buf_used(&s->seq) - len;
-}
-
static int
arch_trigger_all_cpu_backtrace_handler(unsigned int cmd, struct pt_regs *regs)
{
- int cpu;
-
- cpu = smp_processor_id();
+ int cpu = smp_processor_id();
if (cpumask_test_cpu(cpu, to_cpumask(backtrace_mask))) {
- printk_func_t printk_func_save = this_cpu_read(printk_func);
-
- /* Replace printk to write into the NMI seq */
- this_cpu_write(printk_func, nmi_vprintk);
+ printk_nmi_backtrace_this_cpu_begin();
printk(KERN_WARNING "NMI backtrace for cpu %d\n", cpu);
show_regs(regs);
- this_cpu_write(printk_func, printk_func_save);
+ printk_nmi_backtrace_this_cpu_end();
cpumask_clear_cpu(cpu, to_cpumask(backtrace_mask));
return NMI_HANDLED;
--
2.4.3
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 4.2-rc1 v21 6/6] ARM: Add support for on-demand backtrace of other CPUs
2015-07-06 13:13 ` Daniel Thompson
@ 2015-07-06 13:13 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2015-07-06 13:13 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper
Cc: Daniel Thompson, Russell King, Will Deacon, Catalin Marinas,
Marc Zyngier, Stephen Boyd, John Stultz, Steven Rostedt,
linux-kernel, linux-arm-kernel, patches, linaro-kernel,
Sumit Semwal, Dirk Behme, Daniel Drake, Dmitry Pervushin,
Tim Sander
Replicate the x86 code to trigger a backtrace using an NMI and hook
it up to IPI on ARM.
The code differs slightly from the code on x86 because, on ARM, we do
now know at compile time whether a platform is capable of supporting FIQ.
We must avoid using an IPI to request a backtrace from the CPU on which
the backtrace was requested if interrupts are disabled and fall back to
generating it directly.
In addition the implementation of arch_trigger_all_cpu_backtrace() the
patch also includes a few small items of plumbing that must be hooked
up for the new code to work.
Credit:
Russell King provided the initial prototype implementing this feature
for ARM. Today the patch has been reworked and, mostly, rewriten to
keep it aligned with x86. However this patch does still include some
code from Russell's original prototype.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Steven Rostedt <rostedt@goodmis.org>
---
arch/arm/Kconfig | 1 +
arch/arm/include/asm/hardirq.h | 2 +-
arch/arm/include/asm/irq.h | 5 +++
arch/arm/include/asm/smp.h | 3 ++
arch/arm/kernel/smp.c | 82 ++++++++++++++++++++++++++++++++++++++++++
arch/arm/kernel/traps.c | 4 +++
6 files changed, 96 insertions(+), 1 deletion(-)
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index a750c1425c3a..8bf16a7438b7 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -79,6 +79,7 @@ config ARM
select OLD_SIGACTION
select OLD_SIGSUSPEND3
select PERF_USE_VMALLOC
+ select PRINTK_NMI_BACKTRACE
select RTC_LIB
select SYS_SUPPORTS_APM_EMULATION
# Above selects are sorted alphabetically; please add new ones
diff --git a/arch/arm/include/asm/hardirq.h b/arch/arm/include/asm/hardirq.h
index fe3ea776dc34..5df33e30ae1b 100644
--- a/arch/arm/include/asm/hardirq.h
+++ b/arch/arm/include/asm/hardirq.h
@@ -5,7 +5,7 @@
#include <linux/threads.h>
#include <asm/irq.h>
-#define NR_IPI 8
+#define NR_IPI 9
typedef struct {
unsigned int __softirq_pending;
diff --git a/arch/arm/include/asm/irq.h b/arch/arm/include/asm/irq.h
index 53c15dec7af6..be1d07d59ee9 100644
--- a/arch/arm/include/asm/irq.h
+++ b/arch/arm/include/asm/irq.h
@@ -35,6 +35,11 @@ extern void (*handle_arch_irq)(struct pt_regs *);
extern void set_handle_irq(void (*handle_irq)(struct pt_regs *));
#endif
+#ifdef CONFIG_SMP
+extern void arch_trigger_all_cpu_backtrace(bool);
+#define arch_trigger_all_cpu_backtrace(x) arch_trigger_all_cpu_backtrace(x)
+#endif
+
#endif
#endif
diff --git a/arch/arm/include/asm/smp.h b/arch/arm/include/asm/smp.h
index 2f3ac1ba6fb4..652dde6a304c 100644
--- a/arch/arm/include/asm/smp.h
+++ b/arch/arm/include/asm/smp.h
@@ -18,6 +18,8 @@
# error "<asm/smp.h> included in non-SMP build"
#endif
+#define SMP_IPI_FIQ_MASK 0x0100
+
#define raw_smp_processor_id() (current_thread_info()->cpu)
struct seq_file;
@@ -80,6 +82,7 @@ extern void arch_send_call_function_single_ipi(int cpu);
extern void arch_send_call_function_ipi_mask(const struct cpumask *mask);
extern void arch_send_wakeup_ipi_mask(const struct cpumask *mask);
+extern void ipi_cpu_backtrace(struct pt_regs *regs);
extern int register_ipi_completion(struct completion *completion, int cpu);
struct smp_operations {
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index 90dfbedfbfb8..b28658989d73 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -26,6 +26,7 @@
#include <linux/completion.h>
#include <linux/cpufreq.h>
#include <linux/irq_work.h>
+#include <linux/seq_buf.h>
#include <linux/atomic.h>
#include <asm/smp.h>
@@ -72,6 +73,7 @@ enum ipi_msg_type {
IPI_CPU_STOP,
IPI_IRQ_WORK,
IPI_COMPLETION,
+ IPI_CPU_BACKTRACE,
};
static DECLARE_COMPLETION(cpu_running);
@@ -463,6 +465,7 @@ static const char *ipi_types[NR_IPI] __tracepoint_string = {
S(IPI_CPU_STOP, "CPU stop interrupts"),
S(IPI_IRQ_WORK, "IRQ work interrupts"),
S(IPI_COMPLETION, "completion interrupts"),
+ S(IPI_CPU_BACKTRACE, "backtrace interrupts"),
};
static void smp_cross_call(const struct cpumask *target, unsigned int ipinr)
@@ -577,6 +580,8 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
unsigned int cpu = smp_processor_id();
struct pt_regs *old_regs = set_irq_regs(regs);
+ BUILD_BUG_ON(SMP_IPI_FIQ_MASK != BIT(IPI_CPU_BACKTRACE));
+
if ((unsigned)ipinr < NR_IPI) {
trace_ipi_entry(ipi_types[ipinr]);
__inc_irq_stat(cpu, ipi_irqs[ipinr]);
@@ -630,6 +635,12 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
irq_exit();
break;
+ case IPI_CPU_BACKTRACE:
+ irq_enter();
+ ipi_cpu_backtrace(regs);
+ irq_exit();
+ break;
+
default:
pr_crit("CPU%u: Unknown IPI message 0x%x\n",
cpu, ipinr);
@@ -724,3 +735,74 @@ static int __init register_cpufreq_notifier(void)
core_initcall(register_cpufreq_notifier);
#endif
+
+/* For reliability, we're prepared to waste bits here. */
+static DECLARE_BITMAP(backtrace_mask, NR_CPUS) __read_mostly;
+
+void arch_trigger_all_cpu_backtrace(bool include_self)
+{
+ int err, i;
+ int this_cpu = get_cpu();
+
+ err = printk_nmi_backtrace_prepare();
+ if (err) {
+ /*
+ * If there is already an nmi printk sequence in
+ * progress then just give up...
+ */
+ put_cpu();
+ return;
+ }
+
+ cpumask_copy(to_cpumask(backtrace_mask), cpu_online_mask);
+
+ /*
+ * If irqs are disabled on the current processor and
+ * IPI_CPU_BACKTRACE is delivered using IRQ then we aren't be able to
+ * react to IPI_CPU_BACKTRACE until we leave this function. This
+ * would force us to get stuck and, eventually, timeout. We avoid
+ * the timeout (and the resulting failure to print useful information)
+ * by calling the backtrace logic directly whenever irqs are disabled.
+ */
+ if (include_self && irqs_disabled()) {
+ ipi_cpu_backtrace(in_interrupt() ? get_irq_regs() : NULL);
+ include_self = false;
+ }
+
+ if (!include_self)
+ cpumask_clear_cpu(this_cpu, to_cpumask(backtrace_mask));
+
+ if (!cpumask_empty(to_cpumask(backtrace_mask))) {
+ pr_info("Sending FIQ to %s CPUs:\n",
+ (include_self ? "all" : "other"));
+ smp_cross_call(to_cpumask(backtrace_mask), IPI_CPU_BACKTRACE);
+ }
+
+ /* Wait for up to 10 seconds for all CPUs to do the backtrace */
+ for (i = 0; i < 10 * 1000; i++) {
+ if (cpumask_empty(to_cpumask(backtrace_mask)))
+ break;
+ mdelay(1);
+ touch_softlockup_watchdog();
+ }
+
+ printk_nmi_backtrace_complete();
+ put_cpu();
+}
+
+void ipi_cpu_backtrace(struct pt_regs *regs)
+{
+ int cpu = smp_processor_id();
+
+ if (cpumask_test_cpu(cpu, to_cpumask(backtrace_mask))) {
+ printk_nmi_backtrace_this_cpu_begin();
+ pr_warn("FIQ backtrace for cpu %d\n", cpu);
+ if (regs != NULL)
+ show_regs(regs);
+ else
+ dump_stack();
+ printk_nmi_backtrace_this_cpu_end();
+
+ cpumask_clear_cpu(cpu, to_cpumask(backtrace_mask));
+ }
+}
diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c
index 5634823a39cf..c5fe42f345a9 100644
--- a/arch/arm/kernel/traps.c
+++ b/arch/arm/kernel/traps.c
@@ -488,6 +488,10 @@ asmlinkage void __exception_irq_entry handle_fiq_as_nmi(struct pt_regs *regs)
*/
handle_arch_irq(regs);
+#ifdef CONFIG_SMP
+ ipi_cpu_backtrace(regs);
+#endif
+
nmi_exit();
set_irq_regs(old_regs);
--
2.4.3
^ permalink raw reply related [flat|nested] 36+ messages in thread* [PATCH 4.2-rc1 v21 6/6] ARM: Add support for on-demand backtrace of other CPUs
@ 2015-07-06 13:13 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2015-07-06 13:13 UTC (permalink / raw)
To: linux-arm-kernel
Replicate the x86 code to trigger a backtrace using an NMI and hook
it up to IPI on ARM.
The code differs slightly from the code on x86 because, on ARM, we do
now know at compile time whether a platform is capable of supporting FIQ.
We must avoid using an IPI to request a backtrace from the CPU on which
the backtrace was requested if interrupts are disabled and fall back to
generating it directly.
In addition the implementation of arch_trigger_all_cpu_backtrace() the
patch also includes a few small items of plumbing that must be hooked
up for the new code to work.
Credit:
Russell King provided the initial prototype implementing this feature
for ARM. Today the patch has been reworked and, mostly, rewriten to
keep it aligned with x86. However this patch does still include some
code from Russell's original prototype.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Steven Rostedt <rostedt@goodmis.org>
---
arch/arm/Kconfig | 1 +
arch/arm/include/asm/hardirq.h | 2 +-
arch/arm/include/asm/irq.h | 5 +++
arch/arm/include/asm/smp.h | 3 ++
arch/arm/kernel/smp.c | 82 ++++++++++++++++++++++++++++++++++++++++++
arch/arm/kernel/traps.c | 4 +++
6 files changed, 96 insertions(+), 1 deletion(-)
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index a750c1425c3a..8bf16a7438b7 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -79,6 +79,7 @@ config ARM
select OLD_SIGACTION
select OLD_SIGSUSPEND3
select PERF_USE_VMALLOC
+ select PRINTK_NMI_BACKTRACE
select RTC_LIB
select SYS_SUPPORTS_APM_EMULATION
# Above selects are sorted alphabetically; please add new ones
diff --git a/arch/arm/include/asm/hardirq.h b/arch/arm/include/asm/hardirq.h
index fe3ea776dc34..5df33e30ae1b 100644
--- a/arch/arm/include/asm/hardirq.h
+++ b/arch/arm/include/asm/hardirq.h
@@ -5,7 +5,7 @@
#include <linux/threads.h>
#include <asm/irq.h>
-#define NR_IPI 8
+#define NR_IPI 9
typedef struct {
unsigned int __softirq_pending;
diff --git a/arch/arm/include/asm/irq.h b/arch/arm/include/asm/irq.h
index 53c15dec7af6..be1d07d59ee9 100644
--- a/arch/arm/include/asm/irq.h
+++ b/arch/arm/include/asm/irq.h
@@ -35,6 +35,11 @@ extern void (*handle_arch_irq)(struct pt_regs *);
extern void set_handle_irq(void (*handle_irq)(struct pt_regs *));
#endif
+#ifdef CONFIG_SMP
+extern void arch_trigger_all_cpu_backtrace(bool);
+#define arch_trigger_all_cpu_backtrace(x) arch_trigger_all_cpu_backtrace(x)
+#endif
+
#endif
#endif
diff --git a/arch/arm/include/asm/smp.h b/arch/arm/include/asm/smp.h
index 2f3ac1ba6fb4..652dde6a304c 100644
--- a/arch/arm/include/asm/smp.h
+++ b/arch/arm/include/asm/smp.h
@@ -18,6 +18,8 @@
# error "<asm/smp.h> included in non-SMP build"
#endif
+#define SMP_IPI_FIQ_MASK 0x0100
+
#define raw_smp_processor_id() (current_thread_info()->cpu)
struct seq_file;
@@ -80,6 +82,7 @@ extern void arch_send_call_function_single_ipi(int cpu);
extern void arch_send_call_function_ipi_mask(const struct cpumask *mask);
extern void arch_send_wakeup_ipi_mask(const struct cpumask *mask);
+extern void ipi_cpu_backtrace(struct pt_regs *regs);
extern int register_ipi_completion(struct completion *completion, int cpu);
struct smp_operations {
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index 90dfbedfbfb8..b28658989d73 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -26,6 +26,7 @@
#include <linux/completion.h>
#include <linux/cpufreq.h>
#include <linux/irq_work.h>
+#include <linux/seq_buf.h>
#include <linux/atomic.h>
#include <asm/smp.h>
@@ -72,6 +73,7 @@ enum ipi_msg_type {
IPI_CPU_STOP,
IPI_IRQ_WORK,
IPI_COMPLETION,
+ IPI_CPU_BACKTRACE,
};
static DECLARE_COMPLETION(cpu_running);
@@ -463,6 +465,7 @@ static const char *ipi_types[NR_IPI] __tracepoint_string = {
S(IPI_CPU_STOP, "CPU stop interrupts"),
S(IPI_IRQ_WORK, "IRQ work interrupts"),
S(IPI_COMPLETION, "completion interrupts"),
+ S(IPI_CPU_BACKTRACE, "backtrace interrupts"),
};
static void smp_cross_call(const struct cpumask *target, unsigned int ipinr)
@@ -577,6 +580,8 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
unsigned int cpu = smp_processor_id();
struct pt_regs *old_regs = set_irq_regs(regs);
+ BUILD_BUG_ON(SMP_IPI_FIQ_MASK != BIT(IPI_CPU_BACKTRACE));
+
if ((unsigned)ipinr < NR_IPI) {
trace_ipi_entry(ipi_types[ipinr]);
__inc_irq_stat(cpu, ipi_irqs[ipinr]);
@@ -630,6 +635,12 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
irq_exit();
break;
+ case IPI_CPU_BACKTRACE:
+ irq_enter();
+ ipi_cpu_backtrace(regs);
+ irq_exit();
+ break;
+
default:
pr_crit("CPU%u: Unknown IPI message 0x%x\n",
cpu, ipinr);
@@ -724,3 +735,74 @@ static int __init register_cpufreq_notifier(void)
core_initcall(register_cpufreq_notifier);
#endif
+
+/* For reliability, we're prepared to waste bits here. */
+static DECLARE_BITMAP(backtrace_mask, NR_CPUS) __read_mostly;
+
+void arch_trigger_all_cpu_backtrace(bool include_self)
+{
+ int err, i;
+ int this_cpu = get_cpu();
+
+ err = printk_nmi_backtrace_prepare();
+ if (err) {
+ /*
+ * If there is already an nmi printk sequence in
+ * progress then just give up...
+ */
+ put_cpu();
+ return;
+ }
+
+ cpumask_copy(to_cpumask(backtrace_mask), cpu_online_mask);
+
+ /*
+ * If irqs are disabled on the current processor and
+ * IPI_CPU_BACKTRACE is delivered using IRQ then we aren't be able to
+ * react to IPI_CPU_BACKTRACE until we leave this function. This
+ * would force us to get stuck and, eventually, timeout. We avoid
+ * the timeout (and the resulting failure to print useful information)
+ * by calling the backtrace logic directly whenever irqs are disabled.
+ */
+ if (include_self && irqs_disabled()) {
+ ipi_cpu_backtrace(in_interrupt() ? get_irq_regs() : NULL);
+ include_self = false;
+ }
+
+ if (!include_self)
+ cpumask_clear_cpu(this_cpu, to_cpumask(backtrace_mask));
+
+ if (!cpumask_empty(to_cpumask(backtrace_mask))) {
+ pr_info("Sending FIQ to %s CPUs:\n",
+ (include_self ? "all" : "other"));
+ smp_cross_call(to_cpumask(backtrace_mask), IPI_CPU_BACKTRACE);
+ }
+
+ /* Wait for up to 10 seconds for all CPUs to do the backtrace */
+ for (i = 0; i < 10 * 1000; i++) {
+ if (cpumask_empty(to_cpumask(backtrace_mask)))
+ break;
+ mdelay(1);
+ touch_softlockup_watchdog();
+ }
+
+ printk_nmi_backtrace_complete();
+ put_cpu();
+}
+
+void ipi_cpu_backtrace(struct pt_regs *regs)
+{
+ int cpu = smp_processor_id();
+
+ if (cpumask_test_cpu(cpu, to_cpumask(backtrace_mask))) {
+ printk_nmi_backtrace_this_cpu_begin();
+ pr_warn("FIQ backtrace for cpu %d\n", cpu);
+ if (regs != NULL)
+ show_regs(regs);
+ else
+ dump_stack();
+ printk_nmi_backtrace_this_cpu_end();
+
+ cpumask_clear_cpu(cpu, to_cpumask(backtrace_mask));
+ }
+}
diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c
index 5634823a39cf..c5fe42f345a9 100644
--- a/arch/arm/kernel/traps.c
+++ b/arch/arm/kernel/traps.c
@@ -488,6 +488,10 @@ asmlinkage void __exception_irq_entry handle_fiq_as_nmi(struct pt_regs *regs)
*/
handle_arch_irq(regs);
+#ifdef CONFIG_SMP
+ ipi_cpu_backtrace(regs);
+#endif
+
nmi_exit();
set_irq_regs(old_regs);
--
2.4.3
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 4.7-rc3 v23 0/4] irq/arm: Use FIQ for NMI backtrace (when possible)
2015-07-06 13:13 ` Daniel Thompson
@ 2016-06-30 14:57 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-06-30 14:57 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper, Russell King, Marc Zyngier
Cc: Daniel Thompson, Will Deacon, Catalin Marinas, Stephen Boyd,
John Stultz, Steven Rostedt, linux-kernel, linux-arm-kernel,
patches, linaro-kernel, Sumit Semwal, Dirk Behme, Daniel Drake,
Dmitry Pervushin, Tim Sander, Petr Mladek, Lucas Stach
This patchset modifies the GIC driver to allow it, on supported
platforms, to route IPI interrupts to FIQ. It then uses this
feature to allow the NMI backtrace code on arm to be implemented
using FIQ.
The patches have been runtime tested on the following systems, covering
both arm and arm64 systems and those with and without FIQ support:
* Freescale i.MX6 (arm, gicv1, supports FIQ)
* qemu-system-arm -M vexpress-a15 -cpu cortex-a15 (arm, gicv2, supports
FIQ)
* Qualcomm Snapdragon 600 (arm, gicv2, does not support FIQ)
* Hisilicon 6220 (arm64, gicv2, does not support FIQ)
* qemu-system-arm -M vexpress-a9 -cpu cortex-a9 (arm, gicv1, does not
support FIQ)
v23:
* Fixed build on systems without CONFIG_MULTI_IRQ_HANDLER (0-day test
robot)
* Rebased on v4.7-rc3 and added the Acked-by:s from v22 and v23.
* Remove the double register write when raising an SGI by created local
shadow of the SGI group bits (Marc Zyngier)
* Fixed an out-by-one error in one of the WARNings (Marc Zygnier)
* Added logic to cache whether or not the GIC support interrupt grouping
(Marc Zygnier)
* Added a comment to explain an unexpected #ifdef CONFIG_ARM and applied
the magic nit comb (Marc Zygnier)
v22:
* Rebase on v4.4-rc5 to adopt the new NMI backtrace code from Russell
King.
* Polished a few comments and reorganised the patches very slightly
(shifted a couple of arm changes to patch 4).
* Fixed bug in the way gic_handle_fiq() checks whether it is safe for
it to read IAR.
v21:
* Change the way SGIs are raised to try to increase robustness starting
secondary cores. This is a theoretic fix for a regression reported
by Mark Rutland on vexpress-tc2 but it also allows us to remove
igroup0_shadow entirely since it is no longer needed.
* Fix a couple of variable names and add comments to describe the
hardware behavior better (Mark Rutland).
* Improved MULTI_IRQ_HANDLER support by clearing FIQs using
handle_arch_irq (Marc Zygnier).
* Fix gic_cpu_if_down() to ensure group 1 interrupts are disabled
when the interface is brought down.
For changes in v20 and earlier see:
http://thread.gmane.org/gmane.linux.kernel/1928465
Daniel Thompson (4):
irqchip: gic: Optimize locking in gic_raise_softirq
irqchip: gic: Make gic_raise_softirq FIQ-safe
irqchip: gic: Introduce plumbing for IPI FIQ
ARM: Allow IPI_CPU_BACKTRACE to exploit FIQ
arch/arm/include/asm/smp.h | 9 ++
arch/arm/kernel/smp.c | 6 +
arch/arm/kernel/traps.c | 11 +-
drivers/irqchip/irq-gic.c | 254 ++++++++++++++++++++++++++++++++++++----
include/linux/irqchip/arm-gic.h | 6 +
5 files changed, 265 insertions(+), 21 deletions(-)
--
2.5.5
^ permalink raw reply [flat|nested] 36+ messages in thread
* [PATCH 4.7-rc3 v23 0/4] irq/arm: Use FIQ for NMI backtrace (when possible)
@ 2016-06-30 14:57 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-06-30 14:57 UTC (permalink / raw)
To: linux-arm-kernel
This patchset modifies the GIC driver to allow it, on supported
platforms, to route IPI interrupts to FIQ. It then uses this
feature to allow the NMI backtrace code on arm to be implemented
using FIQ.
The patches have been runtime tested on the following systems, covering
both arm and arm64 systems and those with and without FIQ support:
* Freescale i.MX6 (arm, gicv1, supports FIQ)
* qemu-system-arm -M vexpress-a15 -cpu cortex-a15 (arm, gicv2, supports
FIQ)
* Qualcomm Snapdragon 600 (arm, gicv2, does not support FIQ)
* Hisilicon 6220 (arm64, gicv2, does not support FIQ)
* qemu-system-arm -M vexpress-a9 -cpu cortex-a9 (arm, gicv1, does not
support FIQ)
v23:
* Fixed build on systems without CONFIG_MULTI_IRQ_HANDLER (0-day test
robot)
* Rebased on v4.7-rc3 and added the Acked-by:s from v22 and v23.
* Remove the double register write when raising an SGI by created local
shadow of the SGI group bits (Marc Zyngier)
* Fixed an out-by-one error in one of the WARNings (Marc Zygnier)
* Added logic to cache whether or not the GIC support interrupt grouping
(Marc Zygnier)
* Added a comment to explain an unexpected #ifdef CONFIG_ARM and applied
the magic nit comb (Marc Zygnier)
v22:
* Rebase on v4.4-rc5 to adopt the new NMI backtrace code from Russell
King.
* Polished a few comments and reorganised the patches very slightly
(shifted a couple of arm changes to patch 4).
* Fixed bug in the way gic_handle_fiq() checks whether it is safe for
it to read IAR.
v21:
* Change the way SGIs are raised to try to increase robustness starting
secondary cores. This is a theoretic fix for a regression reported
by Mark Rutland on vexpress-tc2 but it also allows us to remove
igroup0_shadow entirely since it is no longer needed.
* Fix a couple of variable names and add comments to describe the
hardware behavior better (Mark Rutland).
* Improved MULTI_IRQ_HANDLER support by clearing FIQs using
handle_arch_irq (Marc Zygnier).
* Fix gic_cpu_if_down() to ensure group 1 interrupts are disabled
when the interface is brought down.
For changes in v20 and earlier see:
http://thread.gmane.org/gmane.linux.kernel/1928465
Daniel Thompson (4):
irqchip: gic: Optimize locking in gic_raise_softirq
irqchip: gic: Make gic_raise_softirq FIQ-safe
irqchip: gic: Introduce plumbing for IPI FIQ
ARM: Allow IPI_CPU_BACKTRACE to exploit FIQ
arch/arm/include/asm/smp.h | 9 ++
arch/arm/kernel/smp.c | 6 +
arch/arm/kernel/traps.c | 11 +-
drivers/irqchip/irq-gic.c | 254 ++++++++++++++++++++++++++++++++++++----
include/linux/irqchip/arm-gic.h | 6 +
5 files changed, 265 insertions(+), 21 deletions(-)
--
2.5.5
^ permalink raw reply [flat|nested] 36+ messages in thread
* [PATCH 4.7-rc3 v23 1/4] irqchip: gic: Optimize locking in gic_raise_softirq
2016-06-30 14:57 ` Daniel Thompson
@ 2016-06-30 14:57 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-06-30 14:57 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper, Russell King, Marc Zyngier
Cc: Daniel Thompson, Will Deacon, Catalin Marinas, Stephen Boyd,
John Stultz, Steven Rostedt, linux-kernel, linux-arm-kernel,
patches, linaro-kernel, Sumit Semwal, Dirk Behme, Daniel Drake,
Dmitry Pervushin, Tim Sander, Petr Mladek, Lucas Stach
Currently gic_raise_softirq() is locked using irq_controller_lock.
This lock is primarily used to make register read-modify-write sequences
atomic but gic_raise_softirq() uses it instead to ensure that the
big.LITTLE migration logic can figure out when it is safe to migrate
interrupts between physical cores.
This is sub-optimal in closely related ways:
1. No locking at all is required on systems where the b.L switcher is
not configured.
2. Finer grain locking can be used on systems where the b.L switcher is
present.
This patch resolves both of the above by introducing a separate finer
grain lock and providing conditionally compiled inlines to lock/unlock
it.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
---
drivers/irqchip/irq-gic.c | 36 +++++++++++++++++++++++++++++++++---
1 file changed, 33 insertions(+), 3 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index fbc4ae2afd29..2c14eb047359 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -94,6 +94,27 @@ struct gic_chip_data {
static DEFINE_RAW_SPINLOCK(irq_controller_lock);
/*
+ * This lock is used by the big.LITTLE migration code to ensure no IPIs
+ * can be pended on the old core after the map has been updated.
+ */
+#ifdef CONFIG_BL_SWITCHER
+static DEFINE_RAW_SPINLOCK(cpu_map_migration_lock);
+
+static inline void gic_migration_lock(unsigned long *flags)
+{
+ raw_spin_lock_irqsave(&cpu_map_migration_lock, *flags);
+}
+
+static inline void gic_migration_unlock(unsigned long flags)
+{
+ raw_spin_unlock_irqrestore(&cpu_map_migration_lock, flags);
+}
+#else
+static inline void gic_migration_lock(unsigned long *flags) {}
+static inline void gic_migration_unlock(unsigned long flags) {}
+#endif
+
+/*
* The GIC mapping of CPU interfaces does not necessarily match
* the logical CPU numbering. Let's use a mapping as returned
* by the GIC itself.
@@ -769,7 +790,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
int cpu;
unsigned long flags, map = 0;
- raw_spin_lock_irqsave(&irq_controller_lock, flags);
+ gic_migration_lock(&flags);
/* Convert our logical CPU mask into a physical one. */
for_each_cpu(cpu, mask)
@@ -784,7 +805,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
/* this always happens on GIC0 */
writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
- raw_spin_unlock_irqrestore(&irq_controller_lock, flags);
+ gic_migration_unlock(flags);
}
#endif
@@ -854,8 +875,17 @@ void gic_migrate_target(unsigned int new_cpu_id)
raw_spin_lock(&irq_controller_lock);
- /* Update the target interface for this logical CPU */
+ /*
+ * Update the target interface for this logical CPU
+ *
+ * From the point we release the cpu_map_migration_lock any new
+ * SGIs will be pended on the new cpu which makes the set of SGIs
+ * pending on the old cpu static. That means we can defer the
+ * migration until after we have released the irq_controller_lock.
+ */
+ raw_spin_lock(&cpu_map_migration_lock);
gic_cpu_map[cpu] = 1 << new_cpu_id;
+ raw_spin_unlock(&cpu_map_migration_lock);
/*
* Find all the peripheral interrupts targetting the current
--
2.5.5
^ permalink raw reply related [flat|nested] 36+ messages in thread* [PATCH 4.7-rc3 v23 1/4] irqchip: gic: Optimize locking in gic_raise_softirq
@ 2016-06-30 14:57 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-06-30 14:57 UTC (permalink / raw)
To: linux-arm-kernel
Currently gic_raise_softirq() is locked using irq_controller_lock.
This lock is primarily used to make register read-modify-write sequences
atomic but gic_raise_softirq() uses it instead to ensure that the
big.LITTLE migration logic can figure out when it is safe to migrate
interrupts between physical cores.
This is sub-optimal in closely related ways:
1. No locking at all is required on systems where the b.L switcher is
not configured.
2. Finer grain locking can be used on systems where the b.L switcher is
present.
This patch resolves both of the above by introducing a separate finer
grain lock and providing conditionally compiled inlines to lock/unlock
it.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
---
drivers/irqchip/irq-gic.c | 36 +++++++++++++++++++++++++++++++++---
1 file changed, 33 insertions(+), 3 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index fbc4ae2afd29..2c14eb047359 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -94,6 +94,27 @@ struct gic_chip_data {
static DEFINE_RAW_SPINLOCK(irq_controller_lock);
/*
+ * This lock is used by the big.LITTLE migration code to ensure no IPIs
+ * can be pended on the old core after the map has been updated.
+ */
+#ifdef CONFIG_BL_SWITCHER
+static DEFINE_RAW_SPINLOCK(cpu_map_migration_lock);
+
+static inline void gic_migration_lock(unsigned long *flags)
+{
+ raw_spin_lock_irqsave(&cpu_map_migration_lock, *flags);
+}
+
+static inline void gic_migration_unlock(unsigned long flags)
+{
+ raw_spin_unlock_irqrestore(&cpu_map_migration_lock, flags);
+}
+#else
+static inline void gic_migration_lock(unsigned long *flags) {}
+static inline void gic_migration_unlock(unsigned long flags) {}
+#endif
+
+/*
* The GIC mapping of CPU interfaces does not necessarily match
* the logical CPU numbering. Let's use a mapping as returned
* by the GIC itself.
@@ -769,7 +790,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
int cpu;
unsigned long flags, map = 0;
- raw_spin_lock_irqsave(&irq_controller_lock, flags);
+ gic_migration_lock(&flags);
/* Convert our logical CPU mask into a physical one. */
for_each_cpu(cpu, mask)
@@ -784,7 +805,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
/* this always happens on GIC0 */
writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
- raw_spin_unlock_irqrestore(&irq_controller_lock, flags);
+ gic_migration_unlock(flags);
}
#endif
@@ -854,8 +875,17 @@ void gic_migrate_target(unsigned int new_cpu_id)
raw_spin_lock(&irq_controller_lock);
- /* Update the target interface for this logical CPU */
+ /*
+ * Update the target interface for this logical CPU
+ *
+ * From the point we release the cpu_map_migration_lock any new
+ * SGIs will be pended on the new cpu which makes the set of SGIs
+ * pending on the old cpu static. That means we can defer the
+ * migration until after we have released the irq_controller_lock.
+ */
+ raw_spin_lock(&cpu_map_migration_lock);
gic_cpu_map[cpu] = 1 << new_cpu_id;
+ raw_spin_unlock(&cpu_map_migration_lock);
/*
* Find all the peripheral interrupts targetting the current
--
2.5.5
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 4.7-rc3 v23 2/4] irqchip: gic: Make gic_raise_softirq FIQ-safe
2016-06-30 14:57 ` Daniel Thompson
@ 2016-06-30 14:57 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-06-30 14:57 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper, Russell King, Marc Zyngier
Cc: Daniel Thompson, Will Deacon, Catalin Marinas, Stephen Boyd,
John Stultz, Steven Rostedt, linux-kernel, linux-arm-kernel,
patches, linaro-kernel, Sumit Semwal, Dirk Behme, Daniel Drake,
Dmitry Pervushin, Tim Sander, Petr Mladek, Lucas Stach
It is currently possible for FIQ handlers to re-enter gic_raise_softirq()
and lock up.
gic_raise_softirq()
lock(x);
-~-> FIQ
handle_fiq()
gic_raise_softirq()
lock(x); <-- Lockup
arch/arm/ uses IPIs to implement arch_irq_work_raise(), thus this issue
renders it difficult for FIQ handlers to safely defer work to less
restrictive calling contexts.
This patch fixes the problem by converting the cpu_map_migration_lock
into a rwlock making it safe to re-enter the function.
Note that having made it safe to re-enter gic_raise_softirq() we no
longer need to mask interrupts during gic_raise_softirq() because the
b.L migration is always performed from task context.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
---
drivers/irqchip/irq-gic.c | 38 +++++++++++++++++++++++++-------------
1 file changed, 25 insertions(+), 13 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index 2c14eb047359..416f352ea6fc 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -96,22 +96,25 @@ static DEFINE_RAW_SPINLOCK(irq_controller_lock);
/*
* This lock is used by the big.LITTLE migration code to ensure no IPIs
* can be pended on the old core after the map has been updated.
+ *
+ * This lock may be locked for reading from both IRQ and FIQ handlers
+ * and therefore must not be locked for writing when these are enabled.
*/
#ifdef CONFIG_BL_SWITCHER
-static DEFINE_RAW_SPINLOCK(cpu_map_migration_lock);
+static DEFINE_RWLOCK(cpu_map_migration_lock);
-static inline void gic_migration_lock(unsigned long *flags)
+static inline void gic_migration_lock(void)
{
- raw_spin_lock_irqsave(&cpu_map_migration_lock, *flags);
+ read_lock(&cpu_map_migration_lock);
}
-static inline void gic_migration_unlock(unsigned long flags)
+static inline void gic_migration_unlock(void)
{
- raw_spin_unlock_irqrestore(&cpu_map_migration_lock, flags);
+ read_unlock(&cpu_map_migration_lock);
}
#else
-static inline void gic_migration_lock(unsigned long *flags) {}
-static inline void gic_migration_unlock(unsigned long flags) {}
+static inline void gic_migration_lock(void) {}
+static inline void gic_migration_unlock(void) {}
#endif
/*
@@ -785,12 +788,20 @@ static int __init gic_pm_init(struct gic_chip_data *gic)
#endif
#ifdef CONFIG_SMP
+/*
+ * Raise the specified IPI on all cpus set in mask.
+ *
+ * This function is safe to call from all calling contexts, including
+ * FIQ handlers. It relies on gic_migration_lock() being multiply acquirable
+ * to avoid deadlocks when the function is re-entered at different
+ * exception levels.
+ */
static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
{
int cpu;
- unsigned long flags, map = 0;
+ unsigned long map = 0;
- gic_migration_lock(&flags);
+ gic_migration_lock();
/* Convert our logical CPU mask into a physical one. */
for_each_cpu(cpu, mask)
@@ -805,7 +816,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
/* this always happens on GIC0 */
writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
- gic_migration_unlock(flags);
+ gic_migration_unlock();
}
#endif
@@ -853,7 +864,8 @@ int gic_get_cpu_id(unsigned int cpu)
* Migrate all peripheral interrupts with a target matching the current CPU
* to the interface corresponding to @new_cpu_id. The CPU interface mapping
* is also updated. Targets to other CPU interfaces are unchanged.
- * This must be called with IRQs locally disabled.
+ * This must be called from a task context and with IRQ and FIQ locally
+ * disabled.
*/
void gic_migrate_target(unsigned int new_cpu_id)
{
@@ -883,9 +895,9 @@ void gic_migrate_target(unsigned int new_cpu_id)
* pending on the old cpu static. That means we can defer the
* migration until after we have released the irq_controller_lock.
*/
- raw_spin_lock(&cpu_map_migration_lock);
+ write_lock(&cpu_map_migration_lock);
gic_cpu_map[cpu] = 1 << new_cpu_id;
- raw_spin_unlock(&cpu_map_migration_lock);
+ write_unlock(&cpu_map_migration_lock);
/*
* Find all the peripheral interrupts targetting the current
--
2.5.5
^ permalink raw reply related [flat|nested] 36+ messages in thread* [PATCH 4.7-rc3 v23 2/4] irqchip: gic: Make gic_raise_softirq FIQ-safe
@ 2016-06-30 14:57 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-06-30 14:57 UTC (permalink / raw)
To: linux-arm-kernel
It is currently possible for FIQ handlers to re-enter gic_raise_softirq()
and lock up.
gic_raise_softirq()
lock(x);
-~-> FIQ
handle_fiq()
gic_raise_softirq()
lock(x); <-- Lockup
arch/arm/ uses IPIs to implement arch_irq_work_raise(), thus this issue
renders it difficult for FIQ handlers to safely defer work to less
restrictive calling contexts.
This patch fixes the problem by converting the cpu_map_migration_lock
into a rwlock making it safe to re-enter the function.
Note that having made it safe to re-enter gic_raise_softirq() we no
longer need to mask interrupts during gic_raise_softirq() because the
b.L migration is always performed from task context.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
---
drivers/irqchip/irq-gic.c | 38 +++++++++++++++++++++++++-------------
1 file changed, 25 insertions(+), 13 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index 2c14eb047359..416f352ea6fc 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -96,22 +96,25 @@ static DEFINE_RAW_SPINLOCK(irq_controller_lock);
/*
* This lock is used by the big.LITTLE migration code to ensure no IPIs
* can be pended on the old core after the map has been updated.
+ *
+ * This lock may be locked for reading from both IRQ and FIQ handlers
+ * and therefore must not be locked for writing when these are enabled.
*/
#ifdef CONFIG_BL_SWITCHER
-static DEFINE_RAW_SPINLOCK(cpu_map_migration_lock);
+static DEFINE_RWLOCK(cpu_map_migration_lock);
-static inline void gic_migration_lock(unsigned long *flags)
+static inline void gic_migration_lock(void)
{
- raw_spin_lock_irqsave(&cpu_map_migration_lock, *flags);
+ read_lock(&cpu_map_migration_lock);
}
-static inline void gic_migration_unlock(unsigned long flags)
+static inline void gic_migration_unlock(void)
{
- raw_spin_unlock_irqrestore(&cpu_map_migration_lock, flags);
+ read_unlock(&cpu_map_migration_lock);
}
#else
-static inline void gic_migration_lock(unsigned long *flags) {}
-static inline void gic_migration_unlock(unsigned long flags) {}
+static inline void gic_migration_lock(void) {}
+static inline void gic_migration_unlock(void) {}
#endif
/*
@@ -785,12 +788,20 @@ static int __init gic_pm_init(struct gic_chip_data *gic)
#endif
#ifdef CONFIG_SMP
+/*
+ * Raise the specified IPI on all cpus set in mask.
+ *
+ * This function is safe to call from all calling contexts, including
+ * FIQ handlers. It relies on gic_migration_lock() being multiply acquirable
+ * to avoid deadlocks when the function is re-entered at different
+ * exception levels.
+ */
static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
{
int cpu;
- unsigned long flags, map = 0;
+ unsigned long map = 0;
- gic_migration_lock(&flags);
+ gic_migration_lock();
/* Convert our logical CPU mask into a physical one. */
for_each_cpu(cpu, mask)
@@ -805,7 +816,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
/* this always happens on GIC0 */
writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
- gic_migration_unlock(flags);
+ gic_migration_unlock();
}
#endif
@@ -853,7 +864,8 @@ int gic_get_cpu_id(unsigned int cpu)
* Migrate all peripheral interrupts with a target matching the current CPU
* to the interface corresponding to @new_cpu_id. The CPU interface mapping
* is also updated. Targets to other CPU interfaces are unchanged.
- * This must be called with IRQs locally disabled.
+ * This must be called from a task context and with IRQ and FIQ locally
+ * disabled.
*/
void gic_migrate_target(unsigned int new_cpu_id)
{
@@ -883,9 +895,9 @@ void gic_migrate_target(unsigned int new_cpu_id)
* pending on the old cpu static. That means we can defer the
* migration until after we have released the irq_controller_lock.
*/
- raw_spin_lock(&cpu_map_migration_lock);
+ write_lock(&cpu_map_migration_lock);
gic_cpu_map[cpu] = 1 << new_cpu_id;
- raw_spin_unlock(&cpu_map_migration_lock);
+ write_unlock(&cpu_map_migration_lock);
/*
* Find all the peripheral interrupts targetting the current
--
2.5.5
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 4.7-rc3 v23 3/4] irqchip: gic: Introduce plumbing for IPI FIQ
2016-06-30 14:57 ` Daniel Thompson
@ 2016-06-30 14:57 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-06-30 14:57 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper, Russell King, Marc Zyngier
Cc: Daniel Thompson, Will Deacon, Catalin Marinas, Stephen Boyd,
John Stultz, Steven Rostedt, linux-kernel, linux-arm-kernel,
patches, linaro-kernel, Sumit Semwal, Dirk Behme, Daniel Drake,
Dmitry Pervushin, Tim Sander, Petr Mladek, Lucas Stach,
Jon Medhurst
Currently it is not possible to exploit FIQ for systems with a GIC, even
on systems that are capable of it. This patch makes it possible
for IPIs to be delivered using FIQ.
To do so it modifies the register state so that normal interrupts are
placed in group 1 and specific IPIs are placed into group 0. It also
configures the controller to raise group 0 interrupts using the FIQ
signal. Finally it provides a means for architecture code to define
which IPIs shall use FIQ and to acknowledge any IPIs that are raised.
All GIC hardware except GICv1-without-TrustZone provides a means
to group exceptions into group 0 and group 1 but the hardware
functionality is unavailable to the kernel when a secure monitor is
present because access to the grouping registers are prohibited outside
secure world. When grouping is not available (or on early GICv1
implementations where it is present but tricky to enable) the code to
change groups does not deploy and all IPIs will be raised via IRQ.
Previous versions of this patch were tested-by Jon Medhurst
<tixy@linaro.org> (thanks!). However I have removed the Tested-by:
for this release because the changes are to great to carry it
over.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Jon Medhurst <tixy@linaro.org>
---
drivers/irqchip/irq-gic.c | 202 +++++++++++++++++++++++++++++++++++++---
include/linux/irqchip/arm-gic.h | 6 ++
2 files changed, 193 insertions(+), 15 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index 416f352ea6fc..d18407ca7808 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -41,6 +41,7 @@
#include <linux/irqchip.h>
#include <linux/irqchip/chained_irq.h>
#include <linux/irqchip/arm-gic.h>
+#include <linux/ratelimit.h>
#include <asm/cputype.h>
#include <asm/irq.h>
@@ -63,6 +64,10 @@ static void gic_check_cpu_features(void)
#define gic_check_cpu_features() do { } while(0)
#endif
+#ifndef SMP_IPI_FIQ_MASK
+#define SMP_IPI_FIQ_MASK 0
+#endif
+
union gic_base {
void __iomem *common_base;
void __percpu * __iomem *percpu_base;
@@ -86,6 +91,9 @@ struct gic_chip_data {
#endif
struct irq_domain *domain;
unsigned int gic_irqs;
+ bool has_grouping_support;
+ bool needs_sgi_with_nsatt;
+ u16 __percpu *sgi_with_nsatt_mask;
#ifdef CONFIG_GIC_NON_BANKED
void __iomem *(*get_base)(union gic_base *);
#endif
@@ -352,12 +360,59 @@ static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
}
#endif
+/*
+ * Fully acknowledge (ack, eoi and deactivate) any outstanding FIQ-based IPI,
+ * otherwise do nothing.
+ */
+static void __maybe_unused gic_handle_fiq(struct pt_regs *regs)
+{
+ struct gic_chip_data *gic = &gic_data[0];
+ void __iomem *cpu_base = gic_data_cpu_base(gic);
+ u32 hppstat, hppnr, irqstat, irqnr;
+
+ do {
+ hppstat = readl_relaxed(cpu_base + GIC_CPU_HIGHPRI);
+ hppnr = hppstat & GICC_IAR_INT_ID_MASK;
+ if (!(hppnr < 16 && BIT(hppnr) & SMP_IPI_FIQ_MASK))
+ break;
+
+ irqstat = readl_relaxed(cpu_base + GIC_CPU_INTACK);
+ irqnr = irqstat & GICC_IAR_INT_ID_MASK;
+
+ writel_relaxed(irqstat, cpu_base + GIC_CPU_EOI);
+ if (static_key_true(&supports_deactivate))
+ writel_relaxed(irqstat, cpu_base + GIC_CPU_DEACTIVATE);
+
+ if (WARN_RATELIMIT(irqnr > 15,
+ "Unexpected irqnr %u (bad prioritization?)\n",
+ irqnr))
+ continue;
+#ifdef CONFIG_SMP
+ handle_IPI(irqnr, regs);
+#endif
+ } while (1);
+}
+
static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
{
u32 irqstat, irqnr;
struct gic_chip_data *gic = &gic_data[0];
void __iomem *cpu_base = gic_data_cpu_base(gic);
+#ifdef CONFIG_ARM
+ /*
+ * ARMv8 added new architectural features that allow NMI to be
+ * emulated without resorting to FIQ. For that reason we can
+ * skip this check on 64-bit systems, it would be harmless on
+ * these systems but it would also be pointless because in_nmi()
+ * could never be true here.
+ */
+ if (in_nmi()) {
+ gic_handle_fiq(regs);
+ return;
+ }
+#endif
+
do {
irqstat = readl_relaxed(cpu_base + GIC_CPU_INTACK);
irqnr = irqstat & GICC_IAR_INT_ID_MASK;
@@ -428,6 +483,54 @@ static struct irq_chip gic_chip = {
IRQCHIP_MASK_ON_SUSPEND,
};
+/*
+ * Shift an interrupt between Group 0 and Group 1.
+ *
+ * In addition to changing the group we also modify the priority to
+ * match what "ARM strongly recommends" for a system where no Group 1
+ * interrupt must ever preempt a Group 0 interrupt.
+ *
+ * It is safe to call this function on systems which do not support
+ * grouping (it will have no effect).
+ */
+static void gic_set_group_irq(struct gic_chip_data *gic, unsigned int hwirq,
+ int group)
+{
+ void __iomem *base = gic_data_dist_base(gic);
+ unsigned int grp_reg = hwirq / 32 * 4;
+ u32 grp_mask = BIT(hwirq % 32);
+ u32 grp_val, pri_val;
+
+ if (!gic->has_grouping_support)
+ return;
+
+ raw_spin_lock(&irq_controller_lock);
+
+ grp_val = readl_relaxed(base + GIC_DIST_IGROUP + grp_reg);
+ pri_val = readb_relaxed(base + GIC_DIST_PRI + hwirq);
+
+ if (group) {
+ grp_val |= grp_mask;
+ pri_val |= BIT(7);
+ } else {
+ grp_val &= ~grp_mask;
+ pri_val &= ~BIT(7);
+ }
+
+ writel_relaxed(grp_val, base + GIC_DIST_IGROUP + grp_reg);
+ writeb_relaxed(pri_val, base + GIC_DIST_PRI + hwirq);
+
+ if (hwirq < 16 && gic->needs_sgi_with_nsatt) {
+ if (group)
+ raw_cpu_or(*gic->sgi_with_nsatt_mask, (u16)BIT(hwirq));
+ else
+ raw_cpu_and(*gic->sgi_with_nsatt_mask,
+ (u16) ~BIT(hwirq));
+ }
+
+ raw_spin_unlock(&irq_controller_lock);
+}
+
void __init gic_cascade_irq(unsigned int gic_nr, unsigned int irq)
{
BUG_ON(gic_nr >= CONFIG_ARM_GIC_MAX_NR);
@@ -457,19 +560,22 @@ static u8 gic_get_cpumask(struct gic_chip_data *gic)
static void gic_cpu_if_up(struct gic_chip_data *gic)
{
void __iomem *cpu_base = gic_data_cpu_base(gic);
- u32 bypass = 0;
- u32 mode = 0;
-
- if (gic == &gic_data[0] && static_key_true(&supports_deactivate))
- mode = GIC_CPU_CTRL_EOImodeNS;
+ u32 ctrl = 0;
/*
- * Preserve bypass disable bits to be written back later
- */
- bypass = readl(cpu_base + GIC_CPU_CTRL);
- bypass &= GICC_DIS_BYPASS_MASK;
+ * Preserve bypass disable bits to be written back later
+ */
+ ctrl = readl(cpu_base + GIC_CPU_CTRL);
+ ctrl &= GICC_DIS_BYPASS_MASK;
+
+ if (gic->has_grouping_support)
+ ctrl |= GICC_COMMON_BPR | GICC_FIQ_EN | GICC_ACK_CTL |
+ GICC_ENABLE_GRP1;
+
+ if (gic == &gic_data[0] && static_key_true(&supports_deactivate))
+ ctrl |= GIC_CPU_CTRL_EOImodeNS;
- writel_relaxed(bypass | mode | GICC_ENABLE, cpu_base + GIC_CPU_CTRL);
+ writel_relaxed(ctrl | GICC_ENABLE, cpu_base + GIC_CPU_CTRL);
}
@@ -493,7 +599,34 @@ static void __init gic_dist_init(struct gic_chip_data *gic)
gic_dist_config(base, gic_irqs, NULL);
- writel_relaxed(GICD_ENABLE, base + GIC_DIST_CTRL);
+ /*
+ * Set EnableGrp1/EnableGrp0 (bit 1 and 0) or EnableGrp (bit 0 only,
+ * bit 1 ignored) depending on current security mode.
+ */
+ writel_relaxed(GICD_ENABLE_GRP1 | GICD_ENABLE, base + GIC_DIST_CTRL);
+
+ /*
+ * Some GICv1 devices (even those with security extensions) do not
+ * implement EnableGrp1 meaning some parts of the above write may
+ * be ignored. We will only enable FIQ support if the bit can be set.
+ */
+ if (readl_relaxed(base + GIC_DIST_CTRL) & GICD_ENABLE_GRP1) {
+ /* Cache whether we support grouping */
+ gic->has_grouping_support = true;
+
+ /* Place all SPIs in group 1 (signal with IRQ). */
+ for (i = 32; i < gic_irqs; i += 32)
+ writel_relaxed(0xffffffff,
+ base + GIC_DIST_IGROUP + i * 4 / 32);
+
+ /*
+ * If the GIC supports the security extension then SGIs
+ * will be filtered based on the value of NSATT. If the
+ * GIC has this support then enable NSATT support.
+ */
+ if (readl_relaxed(base + GIC_DIST_CTR) & GICD_SECURITY_EXTN)
+ gic->needs_sgi_with_nsatt = true;
+ }
}
static int gic_cpu_init(struct gic_chip_data *gic)
@@ -502,6 +635,8 @@ static int gic_cpu_init(struct gic_chip_data *gic)
void __iomem *base = gic_data_cpu_base(gic);
unsigned int cpu_mask, cpu = smp_processor_id();
int i;
+ unsigned long ipi_fiq_mask;
+ unsigned int fiq;
/*
* Setting up the CPU map is only relevant for the primary GIC
@@ -530,6 +665,26 @@ static int gic_cpu_init(struct gic_chip_data *gic)
gic_cpu_config(dist_base, NULL);
+ /*
+ * If the distributor is configured to support interrupt grouping
+ * then set all SGI and PPI interrupts to group 1 and then,
+ * based on SMP_IPI_FIQ_MASK, return the FIQ based IPIs back to
+ * group 0 (updating meta-data and prioritization at the same
+ * time).
+ *
+ * Note that IGROUP[0] is banked, meaning that although we are
+ * writing to a distributor register we are actually performing
+ * part of the per-cpu initialization.
+ */
+ if (gic->has_grouping_support) {
+ writel_relaxed(0xffffffff, dist_base + GIC_DIST_IGROUP + 0);
+ __this_cpu_write(*gic->sgi_with_nsatt_mask, 0xffff);
+
+ ipi_fiq_mask = SMP_IPI_FIQ_MASK;
+ for_each_set_bit(fiq, &ipi_fiq_mask, 16)
+ gic_set_group_irq(gic, fiq, 0);
+ }
+
writel_relaxed(GICC_INT_PRI_THRESHOLD, base + GIC_CPU_PRIMASK);
gic_cpu_if_up(gic);
@@ -546,7 +701,8 @@ int gic_cpu_if_down(unsigned int gic_nr)
cpu_base = gic_data_cpu_base(&gic_data[gic_nr]);
val = readl(cpu_base + GIC_CPU_CTRL);
- val &= ~GICC_ENABLE;
+ val &= ~(GICC_COMMON_BPR | GICC_FIQ_EN | GICC_ACK_CTL |
+ GICC_ENABLE_GRP1 | GICC_ENABLE);
writel_relaxed(val, cpu_base + GIC_CPU_CTRL);
return 0;
@@ -641,7 +797,8 @@ static void gic_dist_restore(struct gic_chip_data *gic)
dist_base + GIC_DIST_ACTIVE_SET + i * 4);
}
- writel_relaxed(GICD_ENABLE, dist_base + GIC_DIST_CTRL);
+ writel_relaxed(GICD_ENABLE_GRP1 | GICD_ENABLE,
+ dist_base + GIC_DIST_CTRL);
}
static void gic_cpu_save(struct gic_chip_data *gic)
@@ -800,6 +957,8 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
{
int cpu;
unsigned long map = 0;
+ unsigned long softint;
+ void __iomem *dist_base;
gic_migration_lock();
@@ -807,14 +966,19 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
for_each_cpu(cpu, mask)
map |= gic_cpu_map[cpu];
+ /* This always happens on GIC0 */
+ dist_base = gic_data_dist_base(&gic_data[0]);
+
/*
* Ensure that stores to Normal memory are visible to the
* other CPUs before they observe us issuing the IPI.
*/
dmb(ishst);
- /* this always happens on GIC0 */
- writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
+ softint = map << 16 | irq;
+ if (this_cpu_read(*gic_data[0].sgi_with_nsatt_mask) & BIT(irq))
+ softint |= 0x8000;
+ writel_relaxed(softint, dist_base + GIC_DIST_SOFTINT);
gic_migration_unlock();
}
@@ -1197,6 +1361,12 @@ static int __init __gic_init_bases(struct gic_chip_data *gic, int irq_start,
pr_info("GIC: Using split EOI/Deactivate mode\n");
}
+ gic->sgi_with_nsatt_mask = alloc_percpu(u16);
+ if (WARN_ON(!gic->sgi_with_nsatt_mask)) {
+ ret = -ENOMEM;
+ goto error;
+ }
+
gic_dist_init(gic);
ret = gic_cpu_init(gic);
if (ret)
@@ -1209,6 +1379,8 @@ static int __init __gic_init_bases(struct gic_chip_data *gic, int irq_start,
return 0;
error:
+ free_percpu(gic->sgi_with_nsatt_mask);
+
if (IS_ENABLED(CONFIG_GIC_NON_BANKED) && gic->percpu_offset) {
free_percpu(gic->dist_base.percpu_base);
free_percpu(gic->cpu_base.percpu_base);
diff --git a/include/linux/irqchip/arm-gic.h b/include/linux/irqchip/arm-gic.h
index fd051855539b..6a16fac82933 100644
--- a/include/linux/irqchip/arm-gic.h
+++ b/include/linux/irqchip/arm-gic.h
@@ -23,6 +23,10 @@
#define GIC_CPU_DEACTIVATE 0x1000
#define GICC_ENABLE 0x1
+#define GICC_ENABLE_GRP1 0x2
+#define GICC_ACK_CTL 0x4
+#define GICC_FIQ_EN 0x8
+#define GICC_COMMON_BPR 0x10
#define GICC_INT_PRI_THRESHOLD 0xf0
#define GIC_CPU_CTRL_EOImodeNS (1 << 9)
@@ -49,7 +53,9 @@
#define GIC_DIST_SGI_PENDING_SET 0xf20
#define GICD_ENABLE 0x1
+#define GICD_ENABLE_GRP1 0x2
#define GICD_DISABLE 0x0
+#define GICD_SECURITY_EXTN 0x400
#define GICD_INT_ACTLOW_LVLTRIG 0x0
#define GICD_INT_EN_CLR_X32 0xffffffff
#define GICD_INT_EN_SET_SGI 0x0000ffff
--
2.5.5
^ permalink raw reply related [flat|nested] 36+ messages in thread* [PATCH 4.7-rc3 v23 3/4] irqchip: gic: Introduce plumbing for IPI FIQ
@ 2016-06-30 14:57 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-06-30 14:57 UTC (permalink / raw)
To: linux-arm-kernel
Currently it is not possible to exploit FIQ for systems with a GIC, even
on systems that are capable of it. This patch makes it possible
for IPIs to be delivered using FIQ.
To do so it modifies the register state so that normal interrupts are
placed in group 1 and specific IPIs are placed into group 0. It also
configures the controller to raise group 0 interrupts using the FIQ
signal. Finally it provides a means for architecture code to define
which IPIs shall use FIQ and to acknowledge any IPIs that are raised.
All GIC hardware except GICv1-without-TrustZone provides a means
to group exceptions into group 0 and group 1 but the hardware
functionality is unavailable to the kernel when a secure monitor is
present because access to the grouping registers are prohibited outside
secure world. When grouping is not available (or on early GICv1
implementations where it is present but tricky to enable) the code to
change groups does not deploy and all IPIs will be raised via IRQ.
Previous versions of this patch were tested-by Jon Medhurst
<tixy@linaro.org> (thanks!). However I have removed the Tested-by:
for this release because the changes are to great to carry it
over.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Jon Medhurst <tixy@linaro.org>
---
drivers/irqchip/irq-gic.c | 202 +++++++++++++++++++++++++++++++++++++---
include/linux/irqchip/arm-gic.h | 6 ++
2 files changed, 193 insertions(+), 15 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index 416f352ea6fc..d18407ca7808 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -41,6 +41,7 @@
#include <linux/irqchip.h>
#include <linux/irqchip/chained_irq.h>
#include <linux/irqchip/arm-gic.h>
+#include <linux/ratelimit.h>
#include <asm/cputype.h>
#include <asm/irq.h>
@@ -63,6 +64,10 @@ static void gic_check_cpu_features(void)
#define gic_check_cpu_features() do { } while(0)
#endif
+#ifndef SMP_IPI_FIQ_MASK
+#define SMP_IPI_FIQ_MASK 0
+#endif
+
union gic_base {
void __iomem *common_base;
void __percpu * __iomem *percpu_base;
@@ -86,6 +91,9 @@ struct gic_chip_data {
#endif
struct irq_domain *domain;
unsigned int gic_irqs;
+ bool has_grouping_support;
+ bool needs_sgi_with_nsatt;
+ u16 __percpu *sgi_with_nsatt_mask;
#ifdef CONFIG_GIC_NON_BANKED
void __iomem *(*get_base)(union gic_base *);
#endif
@@ -352,12 +360,59 @@ static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
}
#endif
+/*
+ * Fully acknowledge (ack, eoi and deactivate) any outstanding FIQ-based IPI,
+ * otherwise do nothing.
+ */
+static void __maybe_unused gic_handle_fiq(struct pt_regs *regs)
+{
+ struct gic_chip_data *gic = &gic_data[0];
+ void __iomem *cpu_base = gic_data_cpu_base(gic);
+ u32 hppstat, hppnr, irqstat, irqnr;
+
+ do {
+ hppstat = readl_relaxed(cpu_base + GIC_CPU_HIGHPRI);
+ hppnr = hppstat & GICC_IAR_INT_ID_MASK;
+ if (!(hppnr < 16 && BIT(hppnr) & SMP_IPI_FIQ_MASK))
+ break;
+
+ irqstat = readl_relaxed(cpu_base + GIC_CPU_INTACK);
+ irqnr = irqstat & GICC_IAR_INT_ID_MASK;
+
+ writel_relaxed(irqstat, cpu_base + GIC_CPU_EOI);
+ if (static_key_true(&supports_deactivate))
+ writel_relaxed(irqstat, cpu_base + GIC_CPU_DEACTIVATE);
+
+ if (WARN_RATELIMIT(irqnr > 15,
+ "Unexpected irqnr %u (bad prioritization?)\n",
+ irqnr))
+ continue;
+#ifdef CONFIG_SMP
+ handle_IPI(irqnr, regs);
+#endif
+ } while (1);
+}
+
static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
{
u32 irqstat, irqnr;
struct gic_chip_data *gic = &gic_data[0];
void __iomem *cpu_base = gic_data_cpu_base(gic);
+#ifdef CONFIG_ARM
+ /*
+ * ARMv8 added new architectural features that allow NMI to be
+ * emulated without resorting to FIQ. For that reason we can
+ * skip this check on 64-bit systems, it would be harmless on
+ * these systems but it would also be pointless because in_nmi()
+ * could never be true here.
+ */
+ if (in_nmi()) {
+ gic_handle_fiq(regs);
+ return;
+ }
+#endif
+
do {
irqstat = readl_relaxed(cpu_base + GIC_CPU_INTACK);
irqnr = irqstat & GICC_IAR_INT_ID_MASK;
@@ -428,6 +483,54 @@ static struct irq_chip gic_chip = {
IRQCHIP_MASK_ON_SUSPEND,
};
+/*
+ * Shift an interrupt between Group 0 and Group 1.
+ *
+ * In addition to changing the group we also modify the priority to
+ * match what "ARM strongly recommends" for a system where no Group 1
+ * interrupt must ever preempt a Group 0 interrupt.
+ *
+ * It is safe to call this function on systems which do not support
+ * grouping (it will have no effect).
+ */
+static void gic_set_group_irq(struct gic_chip_data *gic, unsigned int hwirq,
+ int group)
+{
+ void __iomem *base = gic_data_dist_base(gic);
+ unsigned int grp_reg = hwirq / 32 * 4;
+ u32 grp_mask = BIT(hwirq % 32);
+ u32 grp_val, pri_val;
+
+ if (!gic->has_grouping_support)
+ return;
+
+ raw_spin_lock(&irq_controller_lock);
+
+ grp_val = readl_relaxed(base + GIC_DIST_IGROUP + grp_reg);
+ pri_val = readb_relaxed(base + GIC_DIST_PRI + hwirq);
+
+ if (group) {
+ grp_val |= grp_mask;
+ pri_val |= BIT(7);
+ } else {
+ grp_val &= ~grp_mask;
+ pri_val &= ~BIT(7);
+ }
+
+ writel_relaxed(grp_val, base + GIC_DIST_IGROUP + grp_reg);
+ writeb_relaxed(pri_val, base + GIC_DIST_PRI + hwirq);
+
+ if (hwirq < 16 && gic->needs_sgi_with_nsatt) {
+ if (group)
+ raw_cpu_or(*gic->sgi_with_nsatt_mask, (u16)BIT(hwirq));
+ else
+ raw_cpu_and(*gic->sgi_with_nsatt_mask,
+ (u16) ~BIT(hwirq));
+ }
+
+ raw_spin_unlock(&irq_controller_lock);
+}
+
void __init gic_cascade_irq(unsigned int gic_nr, unsigned int irq)
{
BUG_ON(gic_nr >= CONFIG_ARM_GIC_MAX_NR);
@@ -457,19 +560,22 @@ static u8 gic_get_cpumask(struct gic_chip_data *gic)
static void gic_cpu_if_up(struct gic_chip_data *gic)
{
void __iomem *cpu_base = gic_data_cpu_base(gic);
- u32 bypass = 0;
- u32 mode = 0;
-
- if (gic == &gic_data[0] && static_key_true(&supports_deactivate))
- mode = GIC_CPU_CTRL_EOImodeNS;
+ u32 ctrl = 0;
/*
- * Preserve bypass disable bits to be written back later
- */
- bypass = readl(cpu_base + GIC_CPU_CTRL);
- bypass &= GICC_DIS_BYPASS_MASK;
+ * Preserve bypass disable bits to be written back later
+ */
+ ctrl = readl(cpu_base + GIC_CPU_CTRL);
+ ctrl &= GICC_DIS_BYPASS_MASK;
+
+ if (gic->has_grouping_support)
+ ctrl |= GICC_COMMON_BPR | GICC_FIQ_EN | GICC_ACK_CTL |
+ GICC_ENABLE_GRP1;
+
+ if (gic == &gic_data[0] && static_key_true(&supports_deactivate))
+ ctrl |= GIC_CPU_CTRL_EOImodeNS;
- writel_relaxed(bypass | mode | GICC_ENABLE, cpu_base + GIC_CPU_CTRL);
+ writel_relaxed(ctrl | GICC_ENABLE, cpu_base + GIC_CPU_CTRL);
}
@@ -493,7 +599,34 @@ static void __init gic_dist_init(struct gic_chip_data *gic)
gic_dist_config(base, gic_irqs, NULL);
- writel_relaxed(GICD_ENABLE, base + GIC_DIST_CTRL);
+ /*
+ * Set EnableGrp1/EnableGrp0 (bit 1 and 0) or EnableGrp (bit 0 only,
+ * bit 1 ignored) depending on current security mode.
+ */
+ writel_relaxed(GICD_ENABLE_GRP1 | GICD_ENABLE, base + GIC_DIST_CTRL);
+
+ /*
+ * Some GICv1 devices (even those with security extensions) do not
+ * implement EnableGrp1 meaning some parts of the above write may
+ * be ignored. We will only enable FIQ support if the bit can be set.
+ */
+ if (readl_relaxed(base + GIC_DIST_CTRL) & GICD_ENABLE_GRP1) {
+ /* Cache whether we support grouping */
+ gic->has_grouping_support = true;
+
+ /* Place all SPIs in group 1 (signal with IRQ). */
+ for (i = 32; i < gic_irqs; i += 32)
+ writel_relaxed(0xffffffff,
+ base + GIC_DIST_IGROUP + i * 4 / 32);
+
+ /*
+ * If the GIC supports the security extension then SGIs
+ * will be filtered based on the value of NSATT. If the
+ * GIC has this support then enable NSATT support.
+ */
+ if (readl_relaxed(base + GIC_DIST_CTR) & GICD_SECURITY_EXTN)
+ gic->needs_sgi_with_nsatt = true;
+ }
}
static int gic_cpu_init(struct gic_chip_data *gic)
@@ -502,6 +635,8 @@ static int gic_cpu_init(struct gic_chip_data *gic)
void __iomem *base = gic_data_cpu_base(gic);
unsigned int cpu_mask, cpu = smp_processor_id();
int i;
+ unsigned long ipi_fiq_mask;
+ unsigned int fiq;
/*
* Setting up the CPU map is only relevant for the primary GIC
@@ -530,6 +665,26 @@ static int gic_cpu_init(struct gic_chip_data *gic)
gic_cpu_config(dist_base, NULL);
+ /*
+ * If the distributor is configured to support interrupt grouping
+ * then set all SGI and PPI interrupts to group 1 and then,
+ * based on SMP_IPI_FIQ_MASK, return the FIQ based IPIs back to
+ * group 0 (updating meta-data and prioritization at the same
+ * time).
+ *
+ * Note that IGROUP[0] is banked, meaning that although we are
+ * writing to a distributor register we are actually performing
+ * part of the per-cpu initialization.
+ */
+ if (gic->has_grouping_support) {
+ writel_relaxed(0xffffffff, dist_base + GIC_DIST_IGROUP + 0);
+ __this_cpu_write(*gic->sgi_with_nsatt_mask, 0xffff);
+
+ ipi_fiq_mask = SMP_IPI_FIQ_MASK;
+ for_each_set_bit(fiq, &ipi_fiq_mask, 16)
+ gic_set_group_irq(gic, fiq, 0);
+ }
+
writel_relaxed(GICC_INT_PRI_THRESHOLD, base + GIC_CPU_PRIMASK);
gic_cpu_if_up(gic);
@@ -546,7 +701,8 @@ int gic_cpu_if_down(unsigned int gic_nr)
cpu_base = gic_data_cpu_base(&gic_data[gic_nr]);
val = readl(cpu_base + GIC_CPU_CTRL);
- val &= ~GICC_ENABLE;
+ val &= ~(GICC_COMMON_BPR | GICC_FIQ_EN | GICC_ACK_CTL |
+ GICC_ENABLE_GRP1 | GICC_ENABLE);
writel_relaxed(val, cpu_base + GIC_CPU_CTRL);
return 0;
@@ -641,7 +797,8 @@ static void gic_dist_restore(struct gic_chip_data *gic)
dist_base + GIC_DIST_ACTIVE_SET + i * 4);
}
- writel_relaxed(GICD_ENABLE, dist_base + GIC_DIST_CTRL);
+ writel_relaxed(GICD_ENABLE_GRP1 | GICD_ENABLE,
+ dist_base + GIC_DIST_CTRL);
}
static void gic_cpu_save(struct gic_chip_data *gic)
@@ -800,6 +957,8 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
{
int cpu;
unsigned long map = 0;
+ unsigned long softint;
+ void __iomem *dist_base;
gic_migration_lock();
@@ -807,14 +966,19 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
for_each_cpu(cpu, mask)
map |= gic_cpu_map[cpu];
+ /* This always happens on GIC0 */
+ dist_base = gic_data_dist_base(&gic_data[0]);
+
/*
* Ensure that stores to Normal memory are visible to the
* other CPUs before they observe us issuing the IPI.
*/
dmb(ishst);
- /* this always happens on GIC0 */
- writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
+ softint = map << 16 | irq;
+ if (this_cpu_read(*gic_data[0].sgi_with_nsatt_mask) & BIT(irq))
+ softint |= 0x8000;
+ writel_relaxed(softint, dist_base + GIC_DIST_SOFTINT);
gic_migration_unlock();
}
@@ -1197,6 +1361,12 @@ static int __init __gic_init_bases(struct gic_chip_data *gic, int irq_start,
pr_info("GIC: Using split EOI/Deactivate mode\n");
}
+ gic->sgi_with_nsatt_mask = alloc_percpu(u16);
+ if (WARN_ON(!gic->sgi_with_nsatt_mask)) {
+ ret = -ENOMEM;
+ goto error;
+ }
+
gic_dist_init(gic);
ret = gic_cpu_init(gic);
if (ret)
@@ -1209,6 +1379,8 @@ static int __init __gic_init_bases(struct gic_chip_data *gic, int irq_start,
return 0;
error:
+ free_percpu(gic->sgi_with_nsatt_mask);
+
if (IS_ENABLED(CONFIG_GIC_NON_BANKED) && gic->percpu_offset) {
free_percpu(gic->dist_base.percpu_base);
free_percpu(gic->cpu_base.percpu_base);
diff --git a/include/linux/irqchip/arm-gic.h b/include/linux/irqchip/arm-gic.h
index fd051855539b..6a16fac82933 100644
--- a/include/linux/irqchip/arm-gic.h
+++ b/include/linux/irqchip/arm-gic.h
@@ -23,6 +23,10 @@
#define GIC_CPU_DEACTIVATE 0x1000
#define GICC_ENABLE 0x1
+#define GICC_ENABLE_GRP1 0x2
+#define GICC_ACK_CTL 0x4
+#define GICC_FIQ_EN 0x8
+#define GICC_COMMON_BPR 0x10
#define GICC_INT_PRI_THRESHOLD 0xf0
#define GIC_CPU_CTRL_EOImodeNS (1 << 9)
@@ -49,7 +53,9 @@
#define GIC_DIST_SGI_PENDING_SET 0xf20
#define GICD_ENABLE 0x1
+#define GICD_ENABLE_GRP1 0x2
#define GICD_DISABLE 0x0
+#define GICD_SECURITY_EXTN 0x400
#define GICD_INT_ACTLOW_LVLTRIG 0x0
#define GICD_INT_EN_CLR_X32 0xffffffff
#define GICD_INT_EN_SET_SGI 0x0000ffff
--
2.5.5
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 4.7-rc3 v23 4/4] ARM: Allow IPI_CPU_BACKTRACE to exploit FIQ
2016-06-30 14:57 ` Daniel Thompson
@ 2016-06-30 14:57 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-06-30 14:57 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper, Russell King, Marc Zyngier
Cc: Daniel Thompson, Will Deacon, Catalin Marinas, Stephen Boyd,
John Stultz, Steven Rostedt, linux-kernel, linux-arm-kernel,
patches, linaro-kernel, Sumit Semwal, Dirk Behme, Daniel Drake,
Dmitry Pervushin, Tim Sander, Petr Mladek, Lucas Stach
The GIC (v1 & v2) driver allows its implementation of handle_arch_irq()
to be called from the FIQ handler but currently the ARM code is not
able to exploit this.
Extend handle_fiq_as_nmi() to call handle_arch_irq(). This will affect
all interrupt controllers, including ones that do not support FIQ. This
is OK because a spurious FIQ is normally fatal. Handling a spurious FIQ
like a normal interrupt does risk deadlock but does give us a chance
of surviving long enough to get an error message out.
We also extend the SMP code to indicate to irq drivers which IPIs they
should seek to implement using FIQ.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
---
arch/arm/include/asm/smp.h | 9 +++++++++
arch/arm/kernel/smp.c | 6 ++++++
arch/arm/kernel/traps.c | 11 ++++++++++-
3 files changed, 25 insertions(+), 1 deletion(-)
diff --git a/arch/arm/include/asm/smp.h b/arch/arm/include/asm/smp.h
index 3d6dc8b460e4..daf869cff02e 100644
--- a/arch/arm/include/asm/smp.h
+++ b/arch/arm/include/asm/smp.h
@@ -18,6 +18,15 @@
# error "<asm/smp.h> included in non-SMP build"
#endif
+/*
+ * Identify which IPIs are safe for the irqchip to handle using FIQ.
+ *
+ * This information is advisory. The interrupt controller may not be capable
+ * of routing these IPIs to FIQ and the kernel will continue to work if they
+ * are routed to IRQ as normal.
+ */
+#define SMP_IPI_FIQ_MASK 0x80
+
#define raw_smp_processor_id() (current_thread_info()->cpu)
struct seq_file;
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index df90bc59bfce..c054db0a7ac0 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -644,6 +644,11 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
break;
case IPI_CPU_BACKTRACE:
+ if (in_nmi()) {
+ nmi_cpu_backtrace(regs);
+ break;
+ }
+
printk_nmi_enter();
irq_enter();
nmi_cpu_backtrace(regs);
@@ -757,6 +762,7 @@ static void raise_nmi(cpumask_t *mask)
if (cpumask_test_cpu(smp_processor_id(), mask) && irqs_disabled())
nmi_cpu_backtrace(NULL);
+ BUILD_BUG_ON(SMP_IPI_FIQ_MASK != BIT(IPI_CPU_BACKTRACE));
smp_cross_call(mask, IPI_CPU_BACKTRACE);
}
diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c
index bc698383e822..8f6173cd0a54 100644
--- a/arch/arm/kernel/traps.c
+++ b/arch/arm/kernel/traps.c
@@ -479,7 +479,16 @@ asmlinkage void __exception_irq_entry handle_fiq_as_nmi(struct pt_regs *regs)
nmi_enter();
- /* nop. FIQ handlers for special arch/arm features can be added here. */
+ /*
+ * Either the interrupt controller supports FIQ, meaning it will
+ * do the right thing with this call, or we will end up treating a
+ * spurious FIQ (which is normally fatal) as though it were an IRQ
+ * which, although it risks deadlock, still gives us a sporting
+ * chance of surviving long enough to log errors.
+ */
+#ifdef CONFIG_MULTI_IRQ_HANDLER
+ handle_arch_irq(regs);
+#endif
nmi_exit();
--
2.5.5
^ permalink raw reply related [flat|nested] 36+ messages in thread* [PATCH 4.7-rc3 v23 4/4] ARM: Allow IPI_CPU_BACKTRACE to exploit FIQ
@ 2016-06-30 14:57 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-06-30 14:57 UTC (permalink / raw)
To: linux-arm-kernel
The GIC (v1 & v2) driver allows its implementation of handle_arch_irq()
to be called from the FIQ handler but currently the ARM code is not
able to exploit this.
Extend handle_fiq_as_nmi() to call handle_arch_irq(). This will affect
all interrupt controllers, including ones that do not support FIQ. This
is OK because a spurious FIQ is normally fatal. Handling a spurious FIQ
like a normal interrupt does risk deadlock but does give us a chance
of surviving long enough to get an error message out.
We also extend the SMP code to indicate to irq drivers which IPIs they
should seek to implement using FIQ.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
---
arch/arm/include/asm/smp.h | 9 +++++++++
arch/arm/kernel/smp.c | 6 ++++++
arch/arm/kernel/traps.c | 11 ++++++++++-
3 files changed, 25 insertions(+), 1 deletion(-)
diff --git a/arch/arm/include/asm/smp.h b/arch/arm/include/asm/smp.h
index 3d6dc8b460e4..daf869cff02e 100644
--- a/arch/arm/include/asm/smp.h
+++ b/arch/arm/include/asm/smp.h
@@ -18,6 +18,15 @@
# error "<asm/smp.h> included in non-SMP build"
#endif
+/*
+ * Identify which IPIs are safe for the irqchip to handle using FIQ.
+ *
+ * This information is advisory. The interrupt controller may not be capable
+ * of routing these IPIs to FIQ and the kernel will continue to work if they
+ * are routed to IRQ as normal.
+ */
+#define SMP_IPI_FIQ_MASK 0x80
+
#define raw_smp_processor_id() (current_thread_info()->cpu)
struct seq_file;
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index df90bc59bfce..c054db0a7ac0 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -644,6 +644,11 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
break;
case IPI_CPU_BACKTRACE:
+ if (in_nmi()) {
+ nmi_cpu_backtrace(regs);
+ break;
+ }
+
printk_nmi_enter();
irq_enter();
nmi_cpu_backtrace(regs);
@@ -757,6 +762,7 @@ static void raise_nmi(cpumask_t *mask)
if (cpumask_test_cpu(smp_processor_id(), mask) && irqs_disabled())
nmi_cpu_backtrace(NULL);
+ BUILD_BUG_ON(SMP_IPI_FIQ_MASK != BIT(IPI_CPU_BACKTRACE));
smp_cross_call(mask, IPI_CPU_BACKTRACE);
}
diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c
index bc698383e822..8f6173cd0a54 100644
--- a/arch/arm/kernel/traps.c
+++ b/arch/arm/kernel/traps.c
@@ -479,7 +479,16 @@ asmlinkage void __exception_irq_entry handle_fiq_as_nmi(struct pt_regs *regs)
nmi_enter();
- /* nop. FIQ handlers for special arch/arm features can be added here. */
+ /*
+ * Either the interrupt controller supports FIQ, meaning it will
+ * do the right thing with this call, or we will end up treating a
+ * spurious FIQ (which is normally fatal) as though it were an IRQ
+ * which, although it risks deadlock, still gives us a sporting
+ * chance of surviving long enough to log errors.
+ */
+#ifdef CONFIG_MULTI_IRQ_HANDLER
+ handle_arch_irq(regs);
+#endif
nmi_exit();
--
2.5.5
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 4.8-rc1 v23 0/4] irq/arm: Use FIQ for NMI backtrace (when possible)
2015-07-06 13:13 ` Daniel Thompson
@ 2016-08-11 11:55 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-08-11 11:55 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper, Russell King, Marc Zyngier
Cc: Daniel Thompson, Will Deacon, Catalin Marinas, Stephen Boyd,
John Stultz, Steven Rostedt, linux-kernel, linux-arm-kernel,
patches, linaro-kernel, Sumit Semwal, Dirk Behme, Daniel Drake,
Dmitry Pervushin, Tim Sander, Petr Mladek, Lucas Stach
This patchset modifies the GIC driver to allow it, on supported
platforms, to route IPI interrupts to FIQ. It then uses this
feature to allow the NMI backtrace code on arm to be implemented
using FIQ.
Other than the rebase the patchset is unchanged since previous posting.
The patches have been runtime tested on the following systems, covering
both arm and arm64 systems and those with and without FIQ support:
* Freescale i.MX6 (arm, gicv1, supports FIQ)
* qemu-system-arm -M vexpress-a9 -cpu cortex-a9 (arm, gicv1, does not
support FIQ)
* qemu-system-arm -M vexpress-a15 -cpu cortex-a15 (arm, gicv2, supports
FIQ)
* Qualcomm Snapdragon 600 (arm, gicv2, does not support FIQ)
* Hisilicon 6220 (arm64, gicv2, does not support FIQ)
v23:
* Fixed build on systems without CONFIG_MULTI_IRQ_HANDLER (0-day test
robot)
* Rebased on v4.7-rc3 and added the Acked-by:s from v22 and v23.
* Remove the double register write when raising an SGI by created local
shadow of the SGI group bits (Marc Zyngier)
* Fixed an out-by-one error in one of the WARNings (Marc Zygnier)
* Added logic to cache whether or not the GIC support interrupt grouping
(Marc Zygnier)
* Added a comment to explain an unexpected #ifdef CONFIG_ARM and applied
the magic nit comb (Marc Zygnier)
v22:
* Rebase on v4.4-rc5 to adopt the new NMI backtrace code from Russell
King.
* Polished a few comments and reorganised the patches very slightly
(shifted a couple of arm changes to patch 4).
* Fixed bug in the way gic_handle_fiq() checks whether it is safe for
it to read IAR.
v21:
* Change the way SGIs are raised to try to increase robustness starting
secondary cores. This is a theoretic fix for a regression reported
by Mark Rutland on vexpress-tc2 but it also allows us to remove
igroup0_shadow entirely since it is no longer needed.
* Fix a couple of variable names and add comments to describe the
hardware behavior better (Mark Rutland).
* Improved MULTI_IRQ_HANDLER support by clearing FIQs using
handle_arch_irq (Marc Zygnier).
* Fix gic_cpu_if_down() to ensure group 1 interrupts are disabled
when the interface is brought down.
For changes in v20 and earlier see:
http://thread.gmane.org/gmane.linux.kernel/1928465
Daniel Thompson (4):
irqchip: gic: Optimize locking in gic_raise_softirq
irqchip: gic: Make gic_raise_softirq FIQ-safe
irqchip: gic: Introduce plumbing for IPI FIQ
ARM: Allow IPI_CPU_BACKTRACE to exploit FIQ
arch/arm/include/asm/smp.h | 9 ++
arch/arm/kernel/smp.c | 6 +
arch/arm/kernel/traps.c | 11 +-
drivers/irqchip/irq-gic.c | 252 ++++++++++++++++++++++++++++++++++++----
include/linux/irqchip/arm-gic.h | 6 +
5 files changed, 263 insertions(+), 21 deletions(-)
--
2.7.4
^ permalink raw reply [flat|nested] 36+ messages in thread
* [PATCH 4.8-rc1 v23 0/4] irq/arm: Use FIQ for NMI backtrace (when possible)
@ 2016-08-11 11:55 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-08-11 11:55 UTC (permalink / raw)
To: linux-arm-kernel
This patchset modifies the GIC driver to allow it, on supported
platforms, to route IPI interrupts to FIQ. It then uses this
feature to allow the NMI backtrace code on arm to be implemented
using FIQ.
Other than the rebase the patchset is unchanged since previous posting.
The patches have been runtime tested on the following systems, covering
both arm and arm64 systems and those with and without FIQ support:
* Freescale i.MX6 (arm, gicv1, supports FIQ)
* qemu-system-arm -M vexpress-a9 -cpu cortex-a9 (arm, gicv1, does not
support FIQ)
* qemu-system-arm -M vexpress-a15 -cpu cortex-a15 (arm, gicv2, supports
FIQ)
* Qualcomm Snapdragon 600 (arm, gicv2, does not support FIQ)
* Hisilicon 6220 (arm64, gicv2, does not support FIQ)
v23:
* Fixed build on systems without CONFIG_MULTI_IRQ_HANDLER (0-day test
robot)
* Rebased on v4.7-rc3 and added the Acked-by:s from v22 and v23.
* Remove the double register write when raising an SGI by created local
shadow of the SGI group bits (Marc Zyngier)
* Fixed an out-by-one error in one of the WARNings (Marc Zygnier)
* Added logic to cache whether or not the GIC support interrupt grouping
(Marc Zygnier)
* Added a comment to explain an unexpected #ifdef CONFIG_ARM and applied
the magic nit comb (Marc Zygnier)
v22:
* Rebase on v4.4-rc5 to adopt the new NMI backtrace code from Russell
King.
* Polished a few comments and reorganised the patches very slightly
(shifted a couple of arm changes to patch 4).
* Fixed bug in the way gic_handle_fiq() checks whether it is safe for
it to read IAR.
v21:
* Change the way SGIs are raised to try to increase robustness starting
secondary cores. This is a theoretic fix for a regression reported
by Mark Rutland on vexpress-tc2 but it also allows us to remove
igroup0_shadow entirely since it is no longer needed.
* Fix a couple of variable names and add comments to describe the
hardware behavior better (Mark Rutland).
* Improved MULTI_IRQ_HANDLER support by clearing FIQs using
handle_arch_irq (Marc Zygnier).
* Fix gic_cpu_if_down() to ensure group 1 interrupts are disabled
when the interface is brought down.
For changes in v20 and earlier see:
http://thread.gmane.org/gmane.linux.kernel/1928465
Daniel Thompson (4):
irqchip: gic: Optimize locking in gic_raise_softirq
irqchip: gic: Make gic_raise_softirq FIQ-safe
irqchip: gic: Introduce plumbing for IPI FIQ
ARM: Allow IPI_CPU_BACKTRACE to exploit FIQ
arch/arm/include/asm/smp.h | 9 ++
arch/arm/kernel/smp.c | 6 +
arch/arm/kernel/traps.c | 11 +-
drivers/irqchip/irq-gic.c | 252 ++++++++++++++++++++++++++++++++++++----
include/linux/irqchip/arm-gic.h | 6 +
5 files changed, 263 insertions(+), 21 deletions(-)
--
2.7.4
^ permalink raw reply [flat|nested] 36+ messages in thread
* [PATCH 4.8-rc1 v23 1/4] irqchip: gic: Optimize locking in gic_raise_softirq
2016-08-11 11:55 ` Daniel Thompson
@ 2016-08-11 11:55 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-08-11 11:55 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper, Russell King, Marc Zyngier
Cc: Daniel Thompson, Will Deacon, Catalin Marinas, Stephen Boyd,
John Stultz, Steven Rostedt, linux-kernel, linux-arm-kernel,
patches, linaro-kernel, Sumit Semwal, Dirk Behme, Daniel Drake,
Dmitry Pervushin, Tim Sander, Petr Mladek, Lucas Stach
Currently gic_raise_softirq() is locked using irq_controller_lock.
This lock is primarily used to make register read-modify-write sequences
atomic but gic_raise_softirq() uses it instead to ensure that the
big.LITTLE migration logic can figure out when it is safe to migrate
interrupts between physical cores.
This is sub-optimal in closely related ways:
1. No locking at all is required on systems where the b.L switcher is
not configured.
2. Finer grain locking can be used on systems where the b.L switcher is
present.
This patch resolves both of the above by introducing a separate finer
grain lock and providing conditionally compiled inlines to lock/unlock
it.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
---
drivers/irqchip/irq-gic.c | 36 +++++++++++++++++++++++++++++++++---
1 file changed, 33 insertions(+), 3 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index c2cab572c511..f987e1acc914 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -94,6 +94,27 @@ struct gic_chip_data {
static DEFINE_RAW_SPINLOCK(irq_controller_lock);
/*
+ * This lock is used by the big.LITTLE migration code to ensure no IPIs
+ * can be pended on the old core after the map has been updated.
+ */
+#ifdef CONFIG_BL_SWITCHER
+static DEFINE_RAW_SPINLOCK(cpu_map_migration_lock);
+
+static inline void gic_migration_lock(unsigned long *flags)
+{
+ raw_spin_lock_irqsave(&cpu_map_migration_lock, *flags);
+}
+
+static inline void gic_migration_unlock(unsigned long flags)
+{
+ raw_spin_unlock_irqrestore(&cpu_map_migration_lock, flags);
+}
+#else
+static inline void gic_migration_lock(unsigned long *flags) {}
+static inline void gic_migration_unlock(unsigned long flags) {}
+#endif
+
+/*
* The GIC mapping of CPU interfaces does not necessarily match
* the logical CPU numbering. Let's use a mapping as returned
* by the GIC itself.
@@ -769,7 +790,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
int cpu;
unsigned long flags, map = 0;
- raw_spin_lock_irqsave(&irq_controller_lock, flags);
+ gic_migration_lock(&flags);
/* Convert our logical CPU mask into a physical one. */
for_each_cpu(cpu, mask)
@@ -784,7 +805,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
/* this always happens on GIC0 */
writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
- raw_spin_unlock_irqrestore(&irq_controller_lock, flags);
+ gic_migration_unlock(flags);
}
#endif
@@ -854,8 +875,17 @@ void gic_migrate_target(unsigned int new_cpu_id)
raw_spin_lock(&irq_controller_lock);
- /* Update the target interface for this logical CPU */
+ /*
+ * Update the target interface for this logical CPU
+ *
+ * From the point we release the cpu_map_migration_lock any new
+ * SGIs will be pended on the new cpu which makes the set of SGIs
+ * pending on the old cpu static. That means we can defer the
+ * migration until after we have released the irq_controller_lock.
+ */
+ raw_spin_lock(&cpu_map_migration_lock);
gic_cpu_map[cpu] = 1 << new_cpu_id;
+ raw_spin_unlock(&cpu_map_migration_lock);
/*
* Find all the peripheral interrupts targetting the current
--
2.7.4
^ permalink raw reply related [flat|nested] 36+ messages in thread* [PATCH 4.8-rc1 v23 1/4] irqchip: gic: Optimize locking in gic_raise_softirq
@ 2016-08-11 11:55 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-08-11 11:55 UTC (permalink / raw)
To: linux-arm-kernel
Currently gic_raise_softirq() is locked using irq_controller_lock.
This lock is primarily used to make register read-modify-write sequences
atomic but gic_raise_softirq() uses it instead to ensure that the
big.LITTLE migration logic can figure out when it is safe to migrate
interrupts between physical cores.
This is sub-optimal in closely related ways:
1. No locking at all is required on systems where the b.L switcher is
not configured.
2. Finer grain locking can be used on systems where the b.L switcher is
present.
This patch resolves both of the above by introducing a separate finer
grain lock and providing conditionally compiled inlines to lock/unlock
it.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
---
drivers/irqchip/irq-gic.c | 36 +++++++++++++++++++++++++++++++++---
1 file changed, 33 insertions(+), 3 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index c2cab572c511..f987e1acc914 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -94,6 +94,27 @@ struct gic_chip_data {
static DEFINE_RAW_SPINLOCK(irq_controller_lock);
/*
+ * This lock is used by the big.LITTLE migration code to ensure no IPIs
+ * can be pended on the old core after the map has been updated.
+ */
+#ifdef CONFIG_BL_SWITCHER
+static DEFINE_RAW_SPINLOCK(cpu_map_migration_lock);
+
+static inline void gic_migration_lock(unsigned long *flags)
+{
+ raw_spin_lock_irqsave(&cpu_map_migration_lock, *flags);
+}
+
+static inline void gic_migration_unlock(unsigned long flags)
+{
+ raw_spin_unlock_irqrestore(&cpu_map_migration_lock, flags);
+}
+#else
+static inline void gic_migration_lock(unsigned long *flags) {}
+static inline void gic_migration_unlock(unsigned long flags) {}
+#endif
+
+/*
* The GIC mapping of CPU interfaces does not necessarily match
* the logical CPU numbering. Let's use a mapping as returned
* by the GIC itself.
@@ -769,7 +790,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
int cpu;
unsigned long flags, map = 0;
- raw_spin_lock_irqsave(&irq_controller_lock, flags);
+ gic_migration_lock(&flags);
/* Convert our logical CPU mask into a physical one. */
for_each_cpu(cpu, mask)
@@ -784,7 +805,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
/* this always happens on GIC0 */
writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
- raw_spin_unlock_irqrestore(&irq_controller_lock, flags);
+ gic_migration_unlock(flags);
}
#endif
@@ -854,8 +875,17 @@ void gic_migrate_target(unsigned int new_cpu_id)
raw_spin_lock(&irq_controller_lock);
- /* Update the target interface for this logical CPU */
+ /*
+ * Update the target interface for this logical CPU
+ *
+ * From the point we release the cpu_map_migration_lock any new
+ * SGIs will be pended on the new cpu which makes the set of SGIs
+ * pending on the old cpu static. That means we can defer the
+ * migration until after we have released the irq_controller_lock.
+ */
+ raw_spin_lock(&cpu_map_migration_lock);
gic_cpu_map[cpu] = 1 << new_cpu_id;
+ raw_spin_unlock(&cpu_map_migration_lock);
/*
* Find all the peripheral interrupts targetting the current
--
2.7.4
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 4.8-rc1 v23 2/4] irqchip: gic: Make gic_raise_softirq FIQ-safe
2016-08-11 11:55 ` Daniel Thompson
@ 2016-08-11 11:55 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-08-11 11:55 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper, Russell King, Marc Zyngier
Cc: Daniel Thompson, Will Deacon, Catalin Marinas, Stephen Boyd,
John Stultz, Steven Rostedt, linux-kernel, linux-arm-kernel,
patches, linaro-kernel, Sumit Semwal, Dirk Behme, Daniel Drake,
Dmitry Pervushin, Tim Sander, Petr Mladek, Lucas Stach
It is currently possible for FIQ handlers to re-enter gic_raise_softirq()
and lock up.
gic_raise_softirq()
lock(x);
-~-> FIQ
handle_fiq()
gic_raise_softirq()
lock(x); <-- Lockup
arch/arm/ uses IPIs to implement arch_irq_work_raise(), thus this issue
renders it difficult for FIQ handlers to safely defer work to less
restrictive calling contexts.
This patch fixes the problem by converting the cpu_map_migration_lock
into a rwlock making it safe to re-enter the function.
Note that having made it safe to re-enter gic_raise_softirq() we no
longer need to mask interrupts during gic_raise_softirq() because the
b.L migration is always performed from task context.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
---
drivers/irqchip/irq-gic.c | 38 +++++++++++++++++++++++++-------------
1 file changed, 25 insertions(+), 13 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index f987e1acc914..4bd423d7b01a 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -96,22 +96,25 @@ static DEFINE_RAW_SPINLOCK(irq_controller_lock);
/*
* This lock is used by the big.LITTLE migration code to ensure no IPIs
* can be pended on the old core after the map has been updated.
+ *
+ * This lock may be locked for reading from both IRQ and FIQ handlers
+ * and therefore must not be locked for writing when these are enabled.
*/
#ifdef CONFIG_BL_SWITCHER
-static DEFINE_RAW_SPINLOCK(cpu_map_migration_lock);
+static DEFINE_RWLOCK(cpu_map_migration_lock);
-static inline void gic_migration_lock(unsigned long *flags)
+static inline void gic_migration_lock(void)
{
- raw_spin_lock_irqsave(&cpu_map_migration_lock, *flags);
+ read_lock(&cpu_map_migration_lock);
}
-static inline void gic_migration_unlock(unsigned long flags)
+static inline void gic_migration_unlock(void)
{
- raw_spin_unlock_irqrestore(&cpu_map_migration_lock, flags);
+ read_unlock(&cpu_map_migration_lock);
}
#else
-static inline void gic_migration_lock(unsigned long *flags) {}
-static inline void gic_migration_unlock(unsigned long flags) {}
+static inline void gic_migration_lock(void) {}
+static inline void gic_migration_unlock(void) {}
#endif
/*
@@ -785,12 +788,20 @@ static int gic_pm_init(struct gic_chip_data *gic)
#endif
#ifdef CONFIG_SMP
+/*
+ * Raise the specified IPI on all cpus set in mask.
+ *
+ * This function is safe to call from all calling contexts, including
+ * FIQ handlers. It relies on gic_migration_lock() being multiply acquirable
+ * to avoid deadlocks when the function is re-entered at different
+ * exception levels.
+ */
static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
{
int cpu;
- unsigned long flags, map = 0;
+ unsigned long map = 0;
- gic_migration_lock(&flags);
+ gic_migration_lock();
/* Convert our logical CPU mask into a physical one. */
for_each_cpu(cpu, mask)
@@ -805,7 +816,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
/* this always happens on GIC0 */
writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
- gic_migration_unlock(flags);
+ gic_migration_unlock();
}
#endif
@@ -853,7 +864,8 @@ int gic_get_cpu_id(unsigned int cpu)
* Migrate all peripheral interrupts with a target matching the current CPU
* to the interface corresponding to @new_cpu_id. The CPU interface mapping
* is also updated. Targets to other CPU interfaces are unchanged.
- * This must be called with IRQs locally disabled.
+ * This must be called from a task context and with IRQ and FIQ locally
+ * disabled.
*/
void gic_migrate_target(unsigned int new_cpu_id)
{
@@ -883,9 +895,9 @@ void gic_migrate_target(unsigned int new_cpu_id)
* pending on the old cpu static. That means we can defer the
* migration until after we have released the irq_controller_lock.
*/
- raw_spin_lock(&cpu_map_migration_lock);
+ write_lock(&cpu_map_migration_lock);
gic_cpu_map[cpu] = 1 << new_cpu_id;
- raw_spin_unlock(&cpu_map_migration_lock);
+ write_unlock(&cpu_map_migration_lock);
/*
* Find all the peripheral interrupts targetting the current
--
2.7.4
^ permalink raw reply related [flat|nested] 36+ messages in thread* [PATCH 4.8-rc1 v23 2/4] irqchip: gic: Make gic_raise_softirq FIQ-safe
@ 2016-08-11 11:55 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-08-11 11:55 UTC (permalink / raw)
To: linux-arm-kernel
It is currently possible for FIQ handlers to re-enter gic_raise_softirq()
and lock up.
gic_raise_softirq()
lock(x);
-~-> FIQ
handle_fiq()
gic_raise_softirq()
lock(x); <-- Lockup
arch/arm/ uses IPIs to implement arch_irq_work_raise(), thus this issue
renders it difficult for FIQ handlers to safely defer work to less
restrictive calling contexts.
This patch fixes the problem by converting the cpu_map_migration_lock
into a rwlock making it safe to re-enter the function.
Note that having made it safe to re-enter gic_raise_softirq() we no
longer need to mask interrupts during gic_raise_softirq() because the
b.L migration is always performed from task context.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
---
drivers/irqchip/irq-gic.c | 38 +++++++++++++++++++++++++-------------
1 file changed, 25 insertions(+), 13 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index f987e1acc914..4bd423d7b01a 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -96,22 +96,25 @@ static DEFINE_RAW_SPINLOCK(irq_controller_lock);
/*
* This lock is used by the big.LITTLE migration code to ensure no IPIs
* can be pended on the old core after the map has been updated.
+ *
+ * This lock may be locked for reading from both IRQ and FIQ handlers
+ * and therefore must not be locked for writing when these are enabled.
*/
#ifdef CONFIG_BL_SWITCHER
-static DEFINE_RAW_SPINLOCK(cpu_map_migration_lock);
+static DEFINE_RWLOCK(cpu_map_migration_lock);
-static inline void gic_migration_lock(unsigned long *flags)
+static inline void gic_migration_lock(void)
{
- raw_spin_lock_irqsave(&cpu_map_migration_lock, *flags);
+ read_lock(&cpu_map_migration_lock);
}
-static inline void gic_migration_unlock(unsigned long flags)
+static inline void gic_migration_unlock(void)
{
- raw_spin_unlock_irqrestore(&cpu_map_migration_lock, flags);
+ read_unlock(&cpu_map_migration_lock);
}
#else
-static inline void gic_migration_lock(unsigned long *flags) {}
-static inline void gic_migration_unlock(unsigned long flags) {}
+static inline void gic_migration_lock(void) {}
+static inline void gic_migration_unlock(void) {}
#endif
/*
@@ -785,12 +788,20 @@ static int gic_pm_init(struct gic_chip_data *gic)
#endif
#ifdef CONFIG_SMP
+/*
+ * Raise the specified IPI on all cpus set in mask.
+ *
+ * This function is safe to call from all calling contexts, including
+ * FIQ handlers. It relies on gic_migration_lock() being multiply acquirable
+ * to avoid deadlocks when the function is re-entered at different
+ * exception levels.
+ */
static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
{
int cpu;
- unsigned long flags, map = 0;
+ unsigned long map = 0;
- gic_migration_lock(&flags);
+ gic_migration_lock();
/* Convert our logical CPU mask into a physical one. */
for_each_cpu(cpu, mask)
@@ -805,7 +816,7 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
/* this always happens on GIC0 */
writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
- gic_migration_unlock(flags);
+ gic_migration_unlock();
}
#endif
@@ -853,7 +864,8 @@ int gic_get_cpu_id(unsigned int cpu)
* Migrate all peripheral interrupts with a target matching the current CPU
* to the interface corresponding to @new_cpu_id. The CPU interface mapping
* is also updated. Targets to other CPU interfaces are unchanged.
- * This must be called with IRQs locally disabled.
+ * This must be called from a task context and with IRQ and FIQ locally
+ * disabled.
*/
void gic_migrate_target(unsigned int new_cpu_id)
{
@@ -883,9 +895,9 @@ void gic_migrate_target(unsigned int new_cpu_id)
* pending on the old cpu static. That means we can defer the
* migration until after we have released the irq_controller_lock.
*/
- raw_spin_lock(&cpu_map_migration_lock);
+ write_lock(&cpu_map_migration_lock);
gic_cpu_map[cpu] = 1 << new_cpu_id;
- raw_spin_unlock(&cpu_map_migration_lock);
+ write_unlock(&cpu_map_migration_lock);
/*
* Find all the peripheral interrupts targetting the current
--
2.7.4
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 4.8-rc1 v23 3/4] irqchip: gic: Introduce plumbing for IPI FIQ
2016-08-11 11:55 ` Daniel Thompson
@ 2016-08-11 11:55 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-08-11 11:55 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper, Russell King, Marc Zyngier
Cc: Daniel Thompson, Will Deacon, Catalin Marinas, Stephen Boyd,
John Stultz, Steven Rostedt, linux-kernel, linux-arm-kernel,
patches, linaro-kernel, Sumit Semwal, Dirk Behme, Daniel Drake,
Dmitry Pervushin, Tim Sander, Petr Mladek, Lucas Stach,
Jon Medhurst
Currently it is not possible to exploit FIQ for systems with a GIC, even
on systems that are capable of it. This patch makes it possible
for IPIs to be delivered using FIQ.
To do so it modifies the register state so that normal interrupts are
placed in group 1 and specific IPIs are placed into group 0. It also
configures the controller to raise group 0 interrupts using the FIQ
signal. Finally it provides a means for architecture code to define
which IPIs shall use FIQ and to acknowledge any IPIs that are raised.
All GIC hardware except GICv1-without-TrustZone provides a means
to group exceptions into group 0 and group 1 but the hardware
functionality is unavailable to the kernel when a secure monitor is
present because access to the grouping registers are prohibited outside
secure world. When grouping is not available (or on early GICv1
implementations where it is present but tricky to enable) the code to
change groups does not deploy and all IPIs will be raised via IRQ.
Previous versions of this patch were tested-by Jon Medhurst
<tixy@linaro.org> (thanks!). However I have removed the Tested-by:
for this release because the changes are to great to carry it
over.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Jon Medhurst <tixy@linaro.org>
---
drivers/irqchip/irq-gic.c | 200 +++++++++++++++++++++++++++++++++++++---
include/linux/irqchip/arm-gic.h | 6 ++
2 files changed, 191 insertions(+), 15 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index 4bd423d7b01a..8827593feae3 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -41,6 +41,7 @@
#include <linux/irqchip.h>
#include <linux/irqchip/chained_irq.h>
#include <linux/irqchip/arm-gic.h>
+#include <linux/ratelimit.h>
#include <asm/cputype.h>
#include <asm/irq.h>
@@ -63,6 +64,10 @@ static void gic_check_cpu_features(void)
#define gic_check_cpu_features() do { } while(0)
#endif
+#ifndef SMP_IPI_FIQ_MASK
+#define SMP_IPI_FIQ_MASK 0
+#endif
+
union gic_base {
void __iomem *common_base;
void __percpu * __iomem *percpu_base;
@@ -86,6 +91,9 @@ struct gic_chip_data {
#endif
struct irq_domain *domain;
unsigned int gic_irqs;
+ bool has_grouping_support;
+ bool needs_sgi_with_nsatt;
+ u16 __percpu *sgi_with_nsatt_mask;
#ifdef CONFIG_GIC_NON_BANKED
void __iomem *(*get_base)(union gic_base *);
#endif
@@ -352,12 +360,59 @@ static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
}
#endif
+/*
+ * Fully acknowledge (ack, eoi and deactivate) any outstanding FIQ-based IPI,
+ * otherwise do nothing.
+ */
+static void __maybe_unused gic_handle_fiq(struct pt_regs *regs)
+{
+ struct gic_chip_data *gic = &gic_data[0];
+ void __iomem *cpu_base = gic_data_cpu_base(gic);
+ u32 hppstat, hppnr, irqstat, irqnr;
+
+ do {
+ hppstat = readl_relaxed(cpu_base + GIC_CPU_HIGHPRI);
+ hppnr = hppstat & GICC_IAR_INT_ID_MASK;
+ if (!(hppnr < 16 && BIT(hppnr) & SMP_IPI_FIQ_MASK))
+ break;
+
+ irqstat = readl_relaxed(cpu_base + GIC_CPU_INTACK);
+ irqnr = irqstat & GICC_IAR_INT_ID_MASK;
+
+ writel_relaxed(irqstat, cpu_base + GIC_CPU_EOI);
+ if (static_key_true(&supports_deactivate))
+ writel_relaxed(irqstat, cpu_base + GIC_CPU_DEACTIVATE);
+
+ if (WARN_RATELIMIT(irqnr > 15,
+ "Unexpected irqnr %u (bad prioritization?)\n",
+ irqnr))
+ continue;
+#ifdef CONFIG_SMP
+ handle_IPI(irqnr, regs);
+#endif
+ } while (1);
+}
+
static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
{
u32 irqstat, irqnr;
struct gic_chip_data *gic = &gic_data[0];
void __iomem *cpu_base = gic_data_cpu_base(gic);
+#ifdef CONFIG_ARM
+ /*
+ * ARMv8 added new architectural features that allow NMI to be
+ * emulated without resorting to FIQ. For that reason we can
+ * skip this check on 64-bit systems, it would be harmless on
+ * these systems but it would also be pointless because in_nmi()
+ * could never be true here.
+ */
+ if (in_nmi()) {
+ gic_handle_fiq(regs);
+ return;
+ }
+#endif
+
do {
irqstat = readl_relaxed(cpu_base + GIC_CPU_INTACK);
irqnr = irqstat & GICC_IAR_INT_ID_MASK;
@@ -428,6 +483,54 @@ static struct irq_chip gic_chip = {
IRQCHIP_MASK_ON_SUSPEND,
};
+/*
+ * Shift an interrupt between Group 0 and Group 1.
+ *
+ * In addition to changing the group we also modify the priority to
+ * match what "ARM strongly recommends" for a system where no Group 1
+ * interrupt must ever preempt a Group 0 interrupt.
+ *
+ * It is safe to call this function on systems which do not support
+ * grouping (it will have no effect).
+ */
+static void gic_set_group_irq(struct gic_chip_data *gic, unsigned int hwirq,
+ int group)
+{
+ void __iomem *base = gic_data_dist_base(gic);
+ unsigned int grp_reg = hwirq / 32 * 4;
+ u32 grp_mask = BIT(hwirq % 32);
+ u32 grp_val, pri_val;
+
+ if (!gic->has_grouping_support)
+ return;
+
+ raw_spin_lock(&irq_controller_lock);
+
+ grp_val = readl_relaxed(base + GIC_DIST_IGROUP + grp_reg);
+ pri_val = readb_relaxed(base + GIC_DIST_PRI + hwirq);
+
+ if (group) {
+ grp_val |= grp_mask;
+ pri_val |= BIT(7);
+ } else {
+ grp_val &= ~grp_mask;
+ pri_val &= ~BIT(7);
+ }
+
+ writel_relaxed(grp_val, base + GIC_DIST_IGROUP + grp_reg);
+ writeb_relaxed(pri_val, base + GIC_DIST_PRI + hwirq);
+
+ if (hwirq < 16 && gic->needs_sgi_with_nsatt) {
+ if (group)
+ raw_cpu_or(*gic->sgi_with_nsatt_mask, (u16)BIT(hwirq));
+ else
+ raw_cpu_and(*gic->sgi_with_nsatt_mask,
+ (u16) ~BIT(hwirq));
+ }
+
+ raw_spin_unlock(&irq_controller_lock);
+}
+
void __init gic_cascade_irq(unsigned int gic_nr, unsigned int irq)
{
BUG_ON(gic_nr >= CONFIG_ARM_GIC_MAX_NR);
@@ -457,19 +560,22 @@ static u8 gic_get_cpumask(struct gic_chip_data *gic)
static void gic_cpu_if_up(struct gic_chip_data *gic)
{
void __iomem *cpu_base = gic_data_cpu_base(gic);
- u32 bypass = 0;
- u32 mode = 0;
-
- if (gic == &gic_data[0] && static_key_true(&supports_deactivate))
- mode = GIC_CPU_CTRL_EOImodeNS;
+ u32 ctrl = 0;
/*
- * Preserve bypass disable bits to be written back later
- */
- bypass = readl(cpu_base + GIC_CPU_CTRL);
- bypass &= GICC_DIS_BYPASS_MASK;
+ * Preserve bypass disable bits to be written back later
+ */
+ ctrl = readl(cpu_base + GIC_CPU_CTRL);
+ ctrl &= GICC_DIS_BYPASS_MASK;
- writel_relaxed(bypass | mode | GICC_ENABLE, cpu_base + GIC_CPU_CTRL);
+ if (gic->has_grouping_support)
+ ctrl |= GICC_COMMON_BPR | GICC_FIQ_EN | GICC_ACK_CTL |
+ GICC_ENABLE_GRP1;
+
+ if (gic == &gic_data[0] && static_key_true(&supports_deactivate))
+ ctrl |= GIC_CPU_CTRL_EOImodeNS;
+
+ writel_relaxed(ctrl | GICC_ENABLE, cpu_base + GIC_CPU_CTRL);
}
@@ -493,7 +599,34 @@ static void gic_dist_init(struct gic_chip_data *gic)
gic_dist_config(base, gic_irqs, NULL);
- writel_relaxed(GICD_ENABLE, base + GIC_DIST_CTRL);
+ /*
+ * Set EnableGrp1/EnableGrp0 (bit 1 and 0) or EnableGrp (bit 0 only,
+ * bit 1 ignored) depending on current security mode.
+ */
+ writel_relaxed(GICD_ENABLE_GRP1 | GICD_ENABLE, base + GIC_DIST_CTRL);
+
+ /*
+ * Some GICv1 devices (even those with security extensions) do not
+ * implement EnableGrp1 meaning some parts of the above write may
+ * be ignored. We will only enable FIQ support if the bit can be set.
+ */
+ if (readl_relaxed(base + GIC_DIST_CTRL) & GICD_ENABLE_GRP1) {
+ /* Cache whether we support grouping */
+ gic->has_grouping_support = true;
+
+ /* Place all SPIs in group 1 (signal with IRQ). */
+ for (i = 32; i < gic_irqs; i += 32)
+ writel_relaxed(0xffffffff,
+ base + GIC_DIST_IGROUP + i * 4 / 32);
+
+ /*
+ * If the GIC supports the security extension then SGIs
+ * will be filtered based on the value of NSATT. If the
+ * GIC has this support then enable NSATT support.
+ */
+ if (readl_relaxed(base + GIC_DIST_CTR) & GICD_SECURITY_EXTN)
+ gic->needs_sgi_with_nsatt = true;
+ }
}
static int gic_cpu_init(struct gic_chip_data *gic)
@@ -502,6 +635,8 @@ static int gic_cpu_init(struct gic_chip_data *gic)
void __iomem *base = gic_data_cpu_base(gic);
unsigned int cpu_mask, cpu = smp_processor_id();
int i;
+ unsigned long ipi_fiq_mask;
+ unsigned int fiq;
/*
* Setting up the CPU map is only relevant for the primary GIC
@@ -530,6 +665,26 @@ static int gic_cpu_init(struct gic_chip_data *gic)
gic_cpu_config(dist_base, NULL);
+ /*
+ * If the distributor is configured to support interrupt grouping
+ * then set all SGI and PPI interrupts to group 1 and then,
+ * based on SMP_IPI_FIQ_MASK, return the FIQ based IPIs back to
+ * group 0 (updating meta-data and prioritization at the same
+ * time).
+ *
+ * Note that IGROUP[0] is banked, meaning that although we are
+ * writing to a distributor register we are actually performing
+ * part of the per-cpu initialization.
+ */
+ if (gic->has_grouping_support) {
+ writel_relaxed(0xffffffff, dist_base + GIC_DIST_IGROUP + 0);
+ __this_cpu_write(*gic->sgi_with_nsatt_mask, 0xffff);
+
+ ipi_fiq_mask = SMP_IPI_FIQ_MASK;
+ for_each_set_bit(fiq, &ipi_fiq_mask, 16)
+ gic_set_group_irq(gic, fiq, 0);
+ }
+
writel_relaxed(GICC_INT_PRI_THRESHOLD, base + GIC_CPU_PRIMASK);
gic_cpu_if_up(gic);
@@ -546,7 +701,8 @@ int gic_cpu_if_down(unsigned int gic_nr)
cpu_base = gic_data_cpu_base(&gic_data[gic_nr]);
val = readl(cpu_base + GIC_CPU_CTRL);
- val &= ~GICC_ENABLE;
+ val &= ~(GICC_COMMON_BPR | GICC_FIQ_EN | GICC_ACK_CTL |
+ GICC_ENABLE_GRP1 | GICC_ENABLE);
writel_relaxed(val, cpu_base + GIC_CPU_CTRL);
return 0;
@@ -641,7 +797,8 @@ void gic_dist_restore(struct gic_chip_data *gic)
dist_base + GIC_DIST_ACTIVE_SET + i * 4);
}
- writel_relaxed(GICD_ENABLE, dist_base + GIC_DIST_CTRL);
+ writel_relaxed(GICD_ENABLE_GRP1 | GICD_ENABLE,
+ dist_base + GIC_DIST_CTRL);
}
void gic_cpu_save(struct gic_chip_data *gic)
@@ -800,6 +957,8 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
{
int cpu;
unsigned long map = 0;
+ unsigned long softint;
+ void __iomem *dist_base;
gic_migration_lock();
@@ -807,14 +966,19 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
for_each_cpu(cpu, mask)
map |= gic_cpu_map[cpu];
+ /* This always happens on GIC0 */
+ dist_base = gic_data_dist_base(&gic_data[0]);
+
/*
* Ensure that stores to Normal memory are visible to the
* other CPUs before they observe us issuing the IPI.
*/
dmb(ishst);
- /* this always happens on GIC0 */
- writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
+ softint = map << 16 | irq;
+ if (this_cpu_read(*gic_data[0].sgi_with_nsatt_mask) & BIT(irq))
+ softint |= 0x8000;
+ writel_relaxed(softint, dist_base + GIC_DIST_SOFTINT);
gic_migration_unlock();
}
@@ -1166,6 +1330,12 @@ static int gic_init_bases(struct gic_chip_data *gic, int irq_start,
goto error;
}
+ gic->sgi_with_nsatt_mask = alloc_percpu(u16);
+ if (WARN_ON(!gic->sgi_with_nsatt_mask)) {
+ ret = -ENOMEM;
+ goto error;
+ }
+
gic_dist_init(gic);
ret = gic_cpu_init(gic);
if (ret)
diff --git a/include/linux/irqchip/arm-gic.h b/include/linux/irqchip/arm-gic.h
index eafc965b3eb8..86dae600aef1 100644
--- a/include/linux/irqchip/arm-gic.h
+++ b/include/linux/irqchip/arm-gic.h
@@ -23,6 +23,10 @@
#define GIC_CPU_DEACTIVATE 0x1000
#define GICC_ENABLE 0x1
+#define GICC_ENABLE_GRP1 0x2
+#define GICC_ACK_CTL 0x4
+#define GICC_FIQ_EN 0x8
+#define GICC_COMMON_BPR 0x10
#define GICC_INT_PRI_THRESHOLD 0xf0
#define GIC_CPU_CTRL_EOImodeNS (1 << 9)
@@ -49,7 +53,9 @@
#define GIC_DIST_SGI_PENDING_SET 0xf20
#define GICD_ENABLE 0x1
+#define GICD_ENABLE_GRP1 0x2
#define GICD_DISABLE 0x0
+#define GICD_SECURITY_EXTN 0x400
#define GICD_INT_ACTLOW_LVLTRIG 0x0
#define GICD_INT_EN_CLR_X32 0xffffffff
#define GICD_INT_EN_SET_SGI 0x0000ffff
--
2.7.4
^ permalink raw reply related [flat|nested] 36+ messages in thread* [PATCH 4.8-rc1 v23 3/4] irqchip: gic: Introduce plumbing for IPI FIQ
@ 2016-08-11 11:55 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-08-11 11:55 UTC (permalink / raw)
To: linux-arm-kernel
Currently it is not possible to exploit FIQ for systems with a GIC, even
on systems that are capable of it. This patch makes it possible
for IPIs to be delivered using FIQ.
To do so it modifies the register state so that normal interrupts are
placed in group 1 and specific IPIs are placed into group 0. It also
configures the controller to raise group 0 interrupts using the FIQ
signal. Finally it provides a means for architecture code to define
which IPIs shall use FIQ and to acknowledge any IPIs that are raised.
All GIC hardware except GICv1-without-TrustZone provides a means
to group exceptions into group 0 and group 1 but the hardware
functionality is unavailable to the kernel when a secure monitor is
present because access to the grouping registers are prohibited outside
secure world. When grouping is not available (or on early GICv1
implementations where it is present but tricky to enable) the code to
change groups does not deploy and all IPIs will be raised via IRQ.
Previous versions of this patch were tested-by Jon Medhurst
<tixy@linaro.org> (thanks!). However I have removed the Tested-by:
for this release because the changes are to great to carry it
over.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Jon Medhurst <tixy@linaro.org>
---
drivers/irqchip/irq-gic.c | 200 +++++++++++++++++++++++++++++++++++++---
include/linux/irqchip/arm-gic.h | 6 ++
2 files changed, 191 insertions(+), 15 deletions(-)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index 4bd423d7b01a..8827593feae3 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -41,6 +41,7 @@
#include <linux/irqchip.h>
#include <linux/irqchip/chained_irq.h>
#include <linux/irqchip/arm-gic.h>
+#include <linux/ratelimit.h>
#include <asm/cputype.h>
#include <asm/irq.h>
@@ -63,6 +64,10 @@ static void gic_check_cpu_features(void)
#define gic_check_cpu_features() do { } while(0)
#endif
+#ifndef SMP_IPI_FIQ_MASK
+#define SMP_IPI_FIQ_MASK 0
+#endif
+
union gic_base {
void __iomem *common_base;
void __percpu * __iomem *percpu_base;
@@ -86,6 +91,9 @@ struct gic_chip_data {
#endif
struct irq_domain *domain;
unsigned int gic_irqs;
+ bool has_grouping_support;
+ bool needs_sgi_with_nsatt;
+ u16 __percpu *sgi_with_nsatt_mask;
#ifdef CONFIG_GIC_NON_BANKED
void __iomem *(*get_base)(union gic_base *);
#endif
@@ -352,12 +360,59 @@ static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
}
#endif
+/*
+ * Fully acknowledge (ack, eoi and deactivate) any outstanding FIQ-based IPI,
+ * otherwise do nothing.
+ */
+static void __maybe_unused gic_handle_fiq(struct pt_regs *regs)
+{
+ struct gic_chip_data *gic = &gic_data[0];
+ void __iomem *cpu_base = gic_data_cpu_base(gic);
+ u32 hppstat, hppnr, irqstat, irqnr;
+
+ do {
+ hppstat = readl_relaxed(cpu_base + GIC_CPU_HIGHPRI);
+ hppnr = hppstat & GICC_IAR_INT_ID_MASK;
+ if (!(hppnr < 16 && BIT(hppnr) & SMP_IPI_FIQ_MASK))
+ break;
+
+ irqstat = readl_relaxed(cpu_base + GIC_CPU_INTACK);
+ irqnr = irqstat & GICC_IAR_INT_ID_MASK;
+
+ writel_relaxed(irqstat, cpu_base + GIC_CPU_EOI);
+ if (static_key_true(&supports_deactivate))
+ writel_relaxed(irqstat, cpu_base + GIC_CPU_DEACTIVATE);
+
+ if (WARN_RATELIMIT(irqnr > 15,
+ "Unexpected irqnr %u (bad prioritization?)\n",
+ irqnr))
+ continue;
+#ifdef CONFIG_SMP
+ handle_IPI(irqnr, regs);
+#endif
+ } while (1);
+}
+
static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
{
u32 irqstat, irqnr;
struct gic_chip_data *gic = &gic_data[0];
void __iomem *cpu_base = gic_data_cpu_base(gic);
+#ifdef CONFIG_ARM
+ /*
+ * ARMv8 added new architectural features that allow NMI to be
+ * emulated without resorting to FIQ. For that reason we can
+ * skip this check on 64-bit systems, it would be harmless on
+ * these systems but it would also be pointless because in_nmi()
+ * could never be true here.
+ */
+ if (in_nmi()) {
+ gic_handle_fiq(regs);
+ return;
+ }
+#endif
+
do {
irqstat = readl_relaxed(cpu_base + GIC_CPU_INTACK);
irqnr = irqstat & GICC_IAR_INT_ID_MASK;
@@ -428,6 +483,54 @@ static struct irq_chip gic_chip = {
IRQCHIP_MASK_ON_SUSPEND,
};
+/*
+ * Shift an interrupt between Group 0 and Group 1.
+ *
+ * In addition to changing the group we also modify the priority to
+ * match what "ARM strongly recommends" for a system where no Group 1
+ * interrupt must ever preempt a Group 0 interrupt.
+ *
+ * It is safe to call this function on systems which do not support
+ * grouping (it will have no effect).
+ */
+static void gic_set_group_irq(struct gic_chip_data *gic, unsigned int hwirq,
+ int group)
+{
+ void __iomem *base = gic_data_dist_base(gic);
+ unsigned int grp_reg = hwirq / 32 * 4;
+ u32 grp_mask = BIT(hwirq % 32);
+ u32 grp_val, pri_val;
+
+ if (!gic->has_grouping_support)
+ return;
+
+ raw_spin_lock(&irq_controller_lock);
+
+ grp_val = readl_relaxed(base + GIC_DIST_IGROUP + grp_reg);
+ pri_val = readb_relaxed(base + GIC_DIST_PRI + hwirq);
+
+ if (group) {
+ grp_val |= grp_mask;
+ pri_val |= BIT(7);
+ } else {
+ grp_val &= ~grp_mask;
+ pri_val &= ~BIT(7);
+ }
+
+ writel_relaxed(grp_val, base + GIC_DIST_IGROUP + grp_reg);
+ writeb_relaxed(pri_val, base + GIC_DIST_PRI + hwirq);
+
+ if (hwirq < 16 && gic->needs_sgi_with_nsatt) {
+ if (group)
+ raw_cpu_or(*gic->sgi_with_nsatt_mask, (u16)BIT(hwirq));
+ else
+ raw_cpu_and(*gic->sgi_with_nsatt_mask,
+ (u16) ~BIT(hwirq));
+ }
+
+ raw_spin_unlock(&irq_controller_lock);
+}
+
void __init gic_cascade_irq(unsigned int gic_nr, unsigned int irq)
{
BUG_ON(gic_nr >= CONFIG_ARM_GIC_MAX_NR);
@@ -457,19 +560,22 @@ static u8 gic_get_cpumask(struct gic_chip_data *gic)
static void gic_cpu_if_up(struct gic_chip_data *gic)
{
void __iomem *cpu_base = gic_data_cpu_base(gic);
- u32 bypass = 0;
- u32 mode = 0;
-
- if (gic == &gic_data[0] && static_key_true(&supports_deactivate))
- mode = GIC_CPU_CTRL_EOImodeNS;
+ u32 ctrl = 0;
/*
- * Preserve bypass disable bits to be written back later
- */
- bypass = readl(cpu_base + GIC_CPU_CTRL);
- bypass &= GICC_DIS_BYPASS_MASK;
+ * Preserve bypass disable bits to be written back later
+ */
+ ctrl = readl(cpu_base + GIC_CPU_CTRL);
+ ctrl &= GICC_DIS_BYPASS_MASK;
- writel_relaxed(bypass | mode | GICC_ENABLE, cpu_base + GIC_CPU_CTRL);
+ if (gic->has_grouping_support)
+ ctrl |= GICC_COMMON_BPR | GICC_FIQ_EN | GICC_ACK_CTL |
+ GICC_ENABLE_GRP1;
+
+ if (gic == &gic_data[0] && static_key_true(&supports_deactivate))
+ ctrl |= GIC_CPU_CTRL_EOImodeNS;
+
+ writel_relaxed(ctrl | GICC_ENABLE, cpu_base + GIC_CPU_CTRL);
}
@@ -493,7 +599,34 @@ static void gic_dist_init(struct gic_chip_data *gic)
gic_dist_config(base, gic_irqs, NULL);
- writel_relaxed(GICD_ENABLE, base + GIC_DIST_CTRL);
+ /*
+ * Set EnableGrp1/EnableGrp0 (bit 1 and 0) or EnableGrp (bit 0 only,
+ * bit 1 ignored) depending on current security mode.
+ */
+ writel_relaxed(GICD_ENABLE_GRP1 | GICD_ENABLE, base + GIC_DIST_CTRL);
+
+ /*
+ * Some GICv1 devices (even those with security extensions) do not
+ * implement EnableGrp1 meaning some parts of the above write may
+ * be ignored. We will only enable FIQ support if the bit can be set.
+ */
+ if (readl_relaxed(base + GIC_DIST_CTRL) & GICD_ENABLE_GRP1) {
+ /* Cache whether we support grouping */
+ gic->has_grouping_support = true;
+
+ /* Place all SPIs in group 1 (signal with IRQ). */
+ for (i = 32; i < gic_irqs; i += 32)
+ writel_relaxed(0xffffffff,
+ base + GIC_DIST_IGROUP + i * 4 / 32);
+
+ /*
+ * If the GIC supports the security extension then SGIs
+ * will be filtered based on the value of NSATT. If the
+ * GIC has this support then enable NSATT support.
+ */
+ if (readl_relaxed(base + GIC_DIST_CTR) & GICD_SECURITY_EXTN)
+ gic->needs_sgi_with_nsatt = true;
+ }
}
static int gic_cpu_init(struct gic_chip_data *gic)
@@ -502,6 +635,8 @@ static int gic_cpu_init(struct gic_chip_data *gic)
void __iomem *base = gic_data_cpu_base(gic);
unsigned int cpu_mask, cpu = smp_processor_id();
int i;
+ unsigned long ipi_fiq_mask;
+ unsigned int fiq;
/*
* Setting up the CPU map is only relevant for the primary GIC
@@ -530,6 +665,26 @@ static int gic_cpu_init(struct gic_chip_data *gic)
gic_cpu_config(dist_base, NULL);
+ /*
+ * If the distributor is configured to support interrupt grouping
+ * then set all SGI and PPI interrupts to group 1 and then,
+ * based on SMP_IPI_FIQ_MASK, return the FIQ based IPIs back to
+ * group 0 (updating meta-data and prioritization at the same
+ * time).
+ *
+ * Note that IGROUP[0] is banked, meaning that although we are
+ * writing to a distributor register we are actually performing
+ * part of the per-cpu initialization.
+ */
+ if (gic->has_grouping_support) {
+ writel_relaxed(0xffffffff, dist_base + GIC_DIST_IGROUP + 0);
+ __this_cpu_write(*gic->sgi_with_nsatt_mask, 0xffff);
+
+ ipi_fiq_mask = SMP_IPI_FIQ_MASK;
+ for_each_set_bit(fiq, &ipi_fiq_mask, 16)
+ gic_set_group_irq(gic, fiq, 0);
+ }
+
writel_relaxed(GICC_INT_PRI_THRESHOLD, base + GIC_CPU_PRIMASK);
gic_cpu_if_up(gic);
@@ -546,7 +701,8 @@ int gic_cpu_if_down(unsigned int gic_nr)
cpu_base = gic_data_cpu_base(&gic_data[gic_nr]);
val = readl(cpu_base + GIC_CPU_CTRL);
- val &= ~GICC_ENABLE;
+ val &= ~(GICC_COMMON_BPR | GICC_FIQ_EN | GICC_ACK_CTL |
+ GICC_ENABLE_GRP1 | GICC_ENABLE);
writel_relaxed(val, cpu_base + GIC_CPU_CTRL);
return 0;
@@ -641,7 +797,8 @@ void gic_dist_restore(struct gic_chip_data *gic)
dist_base + GIC_DIST_ACTIVE_SET + i * 4);
}
- writel_relaxed(GICD_ENABLE, dist_base + GIC_DIST_CTRL);
+ writel_relaxed(GICD_ENABLE_GRP1 | GICD_ENABLE,
+ dist_base + GIC_DIST_CTRL);
}
void gic_cpu_save(struct gic_chip_data *gic)
@@ -800,6 +957,8 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
{
int cpu;
unsigned long map = 0;
+ unsigned long softint;
+ void __iomem *dist_base;
gic_migration_lock();
@@ -807,14 +966,19 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
for_each_cpu(cpu, mask)
map |= gic_cpu_map[cpu];
+ /* This always happens on GIC0 */
+ dist_base = gic_data_dist_base(&gic_data[0]);
+
/*
* Ensure that stores to Normal memory are visible to the
* other CPUs before they observe us issuing the IPI.
*/
dmb(ishst);
- /* this always happens on GIC0 */
- writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT);
+ softint = map << 16 | irq;
+ if (this_cpu_read(*gic_data[0].sgi_with_nsatt_mask) & BIT(irq))
+ softint |= 0x8000;
+ writel_relaxed(softint, dist_base + GIC_DIST_SOFTINT);
gic_migration_unlock();
}
@@ -1166,6 +1330,12 @@ static int gic_init_bases(struct gic_chip_data *gic, int irq_start,
goto error;
}
+ gic->sgi_with_nsatt_mask = alloc_percpu(u16);
+ if (WARN_ON(!gic->sgi_with_nsatt_mask)) {
+ ret = -ENOMEM;
+ goto error;
+ }
+
gic_dist_init(gic);
ret = gic_cpu_init(gic);
if (ret)
diff --git a/include/linux/irqchip/arm-gic.h b/include/linux/irqchip/arm-gic.h
index eafc965b3eb8..86dae600aef1 100644
--- a/include/linux/irqchip/arm-gic.h
+++ b/include/linux/irqchip/arm-gic.h
@@ -23,6 +23,10 @@
#define GIC_CPU_DEACTIVATE 0x1000
#define GICC_ENABLE 0x1
+#define GICC_ENABLE_GRP1 0x2
+#define GICC_ACK_CTL 0x4
+#define GICC_FIQ_EN 0x8
+#define GICC_COMMON_BPR 0x10
#define GICC_INT_PRI_THRESHOLD 0xf0
#define GIC_CPU_CTRL_EOImodeNS (1 << 9)
@@ -49,7 +53,9 @@
#define GIC_DIST_SGI_PENDING_SET 0xf20
#define GICD_ENABLE 0x1
+#define GICD_ENABLE_GRP1 0x2
#define GICD_DISABLE 0x0
+#define GICD_SECURITY_EXTN 0x400
#define GICD_INT_ACTLOW_LVLTRIG 0x0
#define GICD_INT_EN_CLR_X32 0xffffffff
#define GICD_INT_EN_SET_SGI 0x0000ffff
--
2.7.4
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 4.8-rc1 v23 4/4] ARM: Allow IPI_CPU_BACKTRACE to exploit FIQ
2016-08-11 11:55 ` Daniel Thompson
@ 2016-08-11 11:55 ` Daniel Thompson
-1 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-08-11 11:55 UTC (permalink / raw)
To: Thomas Gleixner, Jason Cooper, Russell King, Marc Zyngier
Cc: Daniel Thompson, Will Deacon, Catalin Marinas, Stephen Boyd,
John Stultz, Steven Rostedt, linux-kernel, linux-arm-kernel,
patches, linaro-kernel, Sumit Semwal, Dirk Behme, Daniel Drake,
Dmitry Pervushin, Tim Sander, Petr Mladek, Lucas Stach
The GIC (v1 & v2) driver allows its implementation of handle_arch_irq()
to be called from the FIQ handler but currently the ARM code is not
able to exploit this.
Extend handle_fiq_as_nmi() to call handle_arch_irq(). This will affect
all interrupt controllers, including ones that do not support FIQ. This
is OK because a spurious FIQ is normally fatal. Handling a spurious FIQ
like a normal interrupt does risk deadlock but does give us a chance
of surviving long enough to get an error message out.
We also extend the SMP code to indicate to irq drivers which IPIs they
should seek to implement using FIQ.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
---
arch/arm/include/asm/smp.h | 9 +++++++++
arch/arm/kernel/smp.c | 6 ++++++
arch/arm/kernel/traps.c | 11 ++++++++++-
3 files changed, 25 insertions(+), 1 deletion(-)
diff --git a/arch/arm/include/asm/smp.h b/arch/arm/include/asm/smp.h
index 3d6dc8b460e4..daf869cff02e 100644
--- a/arch/arm/include/asm/smp.h
+++ b/arch/arm/include/asm/smp.h
@@ -18,6 +18,15 @@
# error "<asm/smp.h> included in non-SMP build"
#endif
+/*
+ * Identify which IPIs are safe for the irqchip to handle using FIQ.
+ *
+ * This information is advisory. The interrupt controller may not be capable
+ * of routing these IPIs to FIQ and the kernel will continue to work if they
+ * are routed to IRQ as normal.
+ */
+#define SMP_IPI_FIQ_MASK 0x80
+
#define raw_smp_processor_id() (current_thread_info()->cpu)
struct seq_file;
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index 861521606c6d..5e955ad80a1e 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -644,6 +644,11 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
break;
case IPI_CPU_BACKTRACE:
+ if (in_nmi()) {
+ nmi_cpu_backtrace(regs);
+ break;
+ }
+
printk_nmi_enter();
irq_enter();
nmi_cpu_backtrace(regs);
@@ -757,6 +762,7 @@ static void raise_nmi(cpumask_t *mask)
if (cpumask_test_cpu(smp_processor_id(), mask) && irqs_disabled())
nmi_cpu_backtrace(NULL);
+ BUILD_BUG_ON(SMP_IPI_FIQ_MASK != BIT(IPI_CPU_BACKTRACE));
smp_cross_call(mask, IPI_CPU_BACKTRACE);
}
diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c
index bc698383e822..8f6173cd0a54 100644
--- a/arch/arm/kernel/traps.c
+++ b/arch/arm/kernel/traps.c
@@ -479,7 +479,16 @@ asmlinkage void __exception_irq_entry handle_fiq_as_nmi(struct pt_regs *regs)
nmi_enter();
- /* nop. FIQ handlers for special arch/arm features can be added here. */
+ /*
+ * Either the interrupt controller supports FIQ, meaning it will
+ * do the right thing with this call, or we will end up treating a
+ * spurious FIQ (which is normally fatal) as though it were an IRQ
+ * which, although it risks deadlock, still gives us a sporting
+ * chance of surviving long enough to log errors.
+ */
+#ifdef CONFIG_MULTI_IRQ_HANDLER
+ handle_arch_irq(regs);
+#endif
nmi_exit();
--
2.7.4
^ permalink raw reply related [flat|nested] 36+ messages in thread* [PATCH 4.8-rc1 v23 4/4] ARM: Allow IPI_CPU_BACKTRACE to exploit FIQ
@ 2016-08-11 11:55 ` Daniel Thompson
0 siblings, 0 replies; 36+ messages in thread
From: Daniel Thompson @ 2016-08-11 11:55 UTC (permalink / raw)
To: linux-arm-kernel
The GIC (v1 & v2) driver allows its implementation of handle_arch_irq()
to be called from the FIQ handler but currently the ARM code is not
able to exploit this.
Extend handle_fiq_as_nmi() to call handle_arch_irq(). This will affect
all interrupt controllers, including ones that do not support FIQ. This
is OK because a spurious FIQ is normally fatal. Handling a spurious FIQ
like a normal interrupt does risk deadlock but does give us a chance
of surviving long enough to get an error message out.
We also extend the SMP code to indicate to irq drivers which IPIs they
should seek to implement using FIQ.
Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
---
arch/arm/include/asm/smp.h | 9 +++++++++
arch/arm/kernel/smp.c | 6 ++++++
arch/arm/kernel/traps.c | 11 ++++++++++-
3 files changed, 25 insertions(+), 1 deletion(-)
diff --git a/arch/arm/include/asm/smp.h b/arch/arm/include/asm/smp.h
index 3d6dc8b460e4..daf869cff02e 100644
--- a/arch/arm/include/asm/smp.h
+++ b/arch/arm/include/asm/smp.h
@@ -18,6 +18,15 @@
# error "<asm/smp.h> included in non-SMP build"
#endif
+/*
+ * Identify which IPIs are safe for the irqchip to handle using FIQ.
+ *
+ * This information is advisory. The interrupt controller may not be capable
+ * of routing these IPIs to FIQ and the kernel will continue to work if they
+ * are routed to IRQ as normal.
+ */
+#define SMP_IPI_FIQ_MASK 0x80
+
#define raw_smp_processor_id() (current_thread_info()->cpu)
struct seq_file;
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index 861521606c6d..5e955ad80a1e 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -644,6 +644,11 @@ void handle_IPI(int ipinr, struct pt_regs *regs)
break;
case IPI_CPU_BACKTRACE:
+ if (in_nmi()) {
+ nmi_cpu_backtrace(regs);
+ break;
+ }
+
printk_nmi_enter();
irq_enter();
nmi_cpu_backtrace(regs);
@@ -757,6 +762,7 @@ static void raise_nmi(cpumask_t *mask)
if (cpumask_test_cpu(smp_processor_id(), mask) && irqs_disabled())
nmi_cpu_backtrace(NULL);
+ BUILD_BUG_ON(SMP_IPI_FIQ_MASK != BIT(IPI_CPU_BACKTRACE));
smp_cross_call(mask, IPI_CPU_BACKTRACE);
}
diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c
index bc698383e822..8f6173cd0a54 100644
--- a/arch/arm/kernel/traps.c
+++ b/arch/arm/kernel/traps.c
@@ -479,7 +479,16 @@ asmlinkage void __exception_irq_entry handle_fiq_as_nmi(struct pt_regs *regs)
nmi_enter();
- /* nop. FIQ handlers for special arch/arm features can be added here. */
+ /*
+ * Either the interrupt controller supports FIQ, meaning it will
+ * do the right thing with this call, or we will end up treating a
+ * spurious FIQ (which is normally fatal) as though it were an IRQ
+ * which, although it risks deadlock, still gives us a sporting
+ * chance of surviving long enough to log errors.
+ */
+#ifdef CONFIG_MULTI_IRQ_HANDLER
+ handle_arch_irq(regs);
+#endif
nmi_exit();
--
2.7.4
^ permalink raw reply related [flat|nested] 36+ messages in thread