* [PATCH 01/10] ARM: fix incorrect use of get_kernel_nofault()
2021-09-21 9:53 [PATCH 00/10] ARM: add support for IRQ stacks Ard Biesheuvel
@ 2021-09-21 9:53 ` Ard Biesheuvel
2021-09-21 10:39 ` Arnd Bergmann
2021-09-21 14:02 ` Russell King (Oracle)
2021-09-21 9:54 ` [PATCH 02/10] ARM: remove some dead code Ard Biesheuvel
` (8 subsequent siblings)
9 siblings, 2 replies; 14+ messages in thread
From: Ard Biesheuvel @ 2021-09-21 9:53 UTC (permalink / raw)
To: linux
Cc: linux-arm-kernel, Ard Biesheuvel, Arnd Bergmann, Kees Cook,
Keith Packard, Linus Walleij
Commit 344179fc7ef4 ("ARM: 9106/1: traps: use get_kernel_nofault instead
of set_fs()") replaced an occurrence of __get_user() with
get_kernel_nofault(), but inverted the sense of the conditional in the
process, resulting in no values to be printed at all.
I.e., every exception stack now looks like this:
Exception stack(0xc18d1fb0 to 0xc18d1ff8)
1fa0: ???????? ???????? ???????? ????????
1fc0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ????????
1fe0: ???????? ???????? ???????? ???????? ???????? ????????
which is rather unhelpful.
Fixes: 344179fc7ef4 ("ARM: 9106/1: traps: use get_kernel_nofault instead of set_fs()")
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
arch/arm/kernel/traps.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c
index 4a7edc6e848f..195dff58bafc 100644
--- a/arch/arm/kernel/traps.c
+++ b/arch/arm/kernel/traps.c
@@ -136,7 +136,7 @@ static void dump_mem(const char *lvl, const char *str, unsigned long bottom,
for (p = first, i = 0; i < 8 && p < top; i++, p += 4) {
if (p >= bottom && p < top) {
unsigned long val;
- if (get_kernel_nofault(val, (unsigned long *)p))
+ if (!get_kernel_nofault(val, (unsigned long *)p))
sprintf(str + i * 9, " %08lx", val);
else
sprintf(str + i * 9, " ????????");
--
2.30.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH 01/10] ARM: fix incorrect use of get_kernel_nofault()
2021-09-21 9:53 ` [PATCH 01/10] ARM: fix incorrect use of get_kernel_nofault() Ard Biesheuvel
@ 2021-09-21 10:39 ` Arnd Bergmann
2021-09-21 14:02 ` Russell King (Oracle)
1 sibling, 0 replies; 14+ messages in thread
From: Arnd Bergmann @ 2021-09-21 10:39 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Russell King - ARM Linux, Linux ARM, Arnd Bergmann, Kees Cook,
Keith Packard, Linus Walleij
On Tue, Sep 21, 2021 at 11:53 AM Ard Biesheuvel <ardb@kernel.org> wrote:
>
> Commit 344179fc7ef4 ("ARM: 9106/1: traps: use get_kernel_nofault instead
> of set_fs()") replaced an occurrence of __get_user() with
> get_kernel_nofault(), but inverted the sense of the conditional in the
> process, resulting in no values to be printed at all.
>
> I.e., every exception stack now looks like this:
>
> Exception stack(0xc18d1fb0 to 0xc18d1ff8)
> 1fa0: ???????? ???????? ???????? ????????
> 1fc0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ????????
> 1fe0: ???????? ???????? ???????? ???????? ???????? ????????
>
> which is rather unhelpful.
>
> Fixes: 344179fc7ef4 ("ARM: 9106/1: traps: use get_kernel_nofault instead of set_fs()")
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Thanks for catching this!
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH 01/10] ARM: fix incorrect use of get_kernel_nofault()
2021-09-21 9:53 ` [PATCH 01/10] ARM: fix incorrect use of get_kernel_nofault() Ard Biesheuvel
2021-09-21 10:39 ` Arnd Bergmann
@ 2021-09-21 14:02 ` Russell King (Oracle)
2021-09-21 14:33 ` Ard Biesheuvel
1 sibling, 1 reply; 14+ messages in thread
From: Russell King (Oracle) @ 2021-09-21 14:02 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: linux-arm-kernel, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij
On Tue, Sep 21, 2021 at 11:53:59AM +0200, Ard Biesheuvel wrote:
> Commit 344179fc7ef4 ("ARM: 9106/1: traps: use get_kernel_nofault instead
> of set_fs()") replaced an occurrence of __get_user() with
> get_kernel_nofault(), but inverted the sense of the conditional in the
> process, resulting in no values to be printed at all.
>
> I.e., every exception stack now looks like this:
>
> Exception stack(0xc18d1fb0 to 0xc18d1ff8)
> 1fa0: ???????? ???????? ???????? ????????
> 1fc0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ????????
> 1fe0: ???????? ???????? ???????? ???????? ???????? ????????
>
> which is rather unhelpful.
>
> Fixes: 344179fc7ef4 ("ARM: 9106/1: traps: use get_kernel_nofault instead of set_fs()")
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
This is definitely a bug fix to one of the previous series, and should
be independent of the rest of the series.
Please put it in the patch system, thanks.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH 01/10] ARM: fix incorrect use of get_kernel_nofault()
2021-09-21 14:02 ` Russell King (Oracle)
@ 2021-09-21 14:33 ` Ard Biesheuvel
0 siblings, 0 replies; 14+ messages in thread
From: Ard Biesheuvel @ 2021-09-21 14:33 UTC (permalink / raw)
To: Russell King (Oracle)
Cc: Linux ARM, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij
On Tue, 21 Sept 2021 at 16:02, Russell King (Oracle)
<linux@armlinux.org.uk> wrote:
>
> On Tue, Sep 21, 2021 at 11:53:59AM +0200, Ard Biesheuvel wrote:
> > Commit 344179fc7ef4 ("ARM: 9106/1: traps: use get_kernel_nofault instead
> > of set_fs()") replaced an occurrence of __get_user() with
> > get_kernel_nofault(), but inverted the sense of the conditional in the
> > process, resulting in no values to be printed at all.
> >
> > I.e., every exception stack now looks like this:
> >
> > Exception stack(0xc18d1fb0 to 0xc18d1ff8)
> > 1fa0: ???????? ???????? ???????? ????????
> > 1fc0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ????????
> > 1fe0: ???????? ???????? ???????? ???????? ???????? ????????
> >
> > which is rather unhelpful.
> >
> > Fixes: 344179fc7ef4 ("ARM: 9106/1: traps: use get_kernel_nofault instead of set_fs()")
> > Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
>
> This is definitely a bug fix to one of the previous series, and should
> be independent of the rest of the series.
>
> Please put it in the patch system, thanks.
>
Queued up as 9125/1
--
Ard.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 02/10] ARM: remove some dead code
2021-09-21 9:53 [PATCH 00/10] ARM: add support for IRQ stacks Ard Biesheuvel
2021-09-21 9:53 ` [PATCH 01/10] ARM: fix incorrect use of get_kernel_nofault() Ard Biesheuvel
@ 2021-09-21 9:54 ` Ard Biesheuvel
2021-09-21 9:54 ` [PATCH 03/10] ARM: assembler: introduce bl_r and bl_m macros Ard Biesheuvel
` (7 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Ard Biesheuvel @ 2021-09-21 9:54 UTC (permalink / raw)
To: linux
Cc: linux-arm-kernel, Ard Biesheuvel, Arnd Bergmann, Kees Cook,
Keith Packard, Linus Walleij
This code appears to be no longer used so let's get rid of it.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
arch/arm/include/asm/entry-macro-multi.S | 24 --------------------
arch/arm/include/asm/smp.h | 5 ----
arch/arm/kernel/smp.c | 5 ----
3 files changed, 34 deletions(-)
diff --git a/arch/arm/include/asm/entry-macro-multi.S b/arch/arm/include/asm/entry-macro-multi.S
index dfc6bfa43012..24486dad9e19 100644
--- a/arch/arm/include/asm/entry-macro-multi.S
+++ b/arch/arm/include/asm/entry-macro-multi.S
@@ -13,28 +13,4 @@
@
badrne lr, 1b
bne asm_do_IRQ
-
-#ifdef CONFIG_SMP
- /*
- * XXX
- *
- * this macro assumes that irqstat (r2) and base (r6) are
- * preserved from get_irqnr_and_base above
- */
- ALT_SMP(test_for_ipi r0, r2, r6, lr)
- ALT_UP_B(9997f)
- movne r1, sp
- badrne lr, 1b
- bne do_IPI
-#endif
-9997:
- .endm
-
- .macro arch_irq_handler, symbol_name
- .align 5
- .global \symbol_name
-\symbol_name:
- mov r8, lr
- arch_irq_handler_default
- ret r8
.endm
diff --git a/arch/arm/include/asm/smp.h b/arch/arm/include/asm/smp.h
index f16cbbd5cda4..7c1c90d9f582 100644
--- a/arch/arm/include/asm/smp.h
+++ b/arch/arm/include/asm/smp.h
@@ -24,11 +24,6 @@ struct seq_file;
*/
extern void show_ipi_list(struct seq_file *, int);
-/*
- * Called from assembly code, this handles an IPI.
- */
-asmlinkage void do_IPI(int ipinr, struct pt_regs *regs);
-
/*
* Called from C code, this handles an IPI.
*/
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index cde5b6d8bac5..9c55ca915ba4 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -631,11 +631,6 @@ static void ipi_complete(unsigned int cpu)
/*
* Main handler for inter-processor interrupts
*/
-asmlinkage void __exception_irq_entry do_IPI(int ipinr, struct pt_regs *regs)
-{
- handle_IPI(ipinr, regs);
-}
-
static void do_handle_IPI(int ipinr)
{
unsigned int cpu = smp_processor_id();
--
2.30.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 03/10] ARM: assembler: introduce bl_r and bl_m macros
2021-09-21 9:53 [PATCH 00/10] ARM: add support for IRQ stacks Ard Biesheuvel
2021-09-21 9:53 ` [PATCH 01/10] ARM: fix incorrect use of get_kernel_nofault() Ard Biesheuvel
2021-09-21 9:54 ` [PATCH 02/10] ARM: remove some dead code Ard Biesheuvel
@ 2021-09-21 9:54 ` Ard Biesheuvel
2021-09-21 9:54 ` [PATCH 04/10] ARM: optimize indirect call to handle_arch_irq for v7 cores Ard Biesheuvel
` (6 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Ard Biesheuvel @ 2021-09-21 9:54 UTC (permalink / raw)
To: linux
Cc: linux-arm-kernel, Ard Biesheuvel, Arnd Bergmann, Kees Cook,
Keith Packard, Linus Walleij
Add some macros that abstract the difference between the ways indirect
calls are performed on older and newer ARM architecture revisions.
The main difference is to prefer blx instructions over explicit LR
assignments when possible, as these tend to confuse the prediction logic
in out-of-order cores when speculating across a function return.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
arch/arm/include/asm/assembler.h | 37 ++++++++++++++++++++
1 file changed, 37 insertions(+)
diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h
index 7d23d4bb2168..abb8202ef0da 100644
--- a/arch/arm/include/asm/assembler.h
+++ b/arch/arm/include/asm/assembler.h
@@ -624,4 +624,41 @@ THUMB( orr \reg , \reg , #PSR_T_BIT )
.endif
.endm
+ /*
+ * bl_r - branch and link to register
+ *
+ * @dst: target to branch to
+ * @c: conditional opcode suffix
+ */
+ .macro bl_r, dst:req, c
+ .if __LINUX_ARM_ARCH__ < 6
+ mov\c lr, pc
+ mov\c pc, \dst
+ .else
+ blx\c \dst
+ .endif
+ .endm
+
+ /*
+ * bl_m - branch and link to address held in memory
+ *
+ * @dst: memory reference to load the branch target from
+ *
+ * This uses the same syntax as ldr instructions, including things like
+ * register writeback (where it makes sense). For example,
+ *
+ * bl_m [ip, #<offset>]
+ *
+ * Note that @dst should not reference lr as it may be clobbered early.
+ */
+ .macro bl_m, dst:vararg
+ .if __LINUX_ARM_ARCH__ < 6
+ mov lr, pc
+ ldr pc, \dst
+ .else
+ ldr lr, \dst
+ blx lr
+ .endif
+ .endm
+
#endif /* __ASM_ASSEMBLER_H__ */
--
2.30.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 04/10] ARM: optimize indirect call to handle_arch_irq for v7 cores
2021-09-21 9:53 [PATCH 00/10] ARM: add support for IRQ stacks Ard Biesheuvel
` (2 preceding siblings ...)
2021-09-21 9:54 ` [PATCH 03/10] ARM: assembler: introduce bl_r and bl_m macros Ard Biesheuvel
@ 2021-09-21 9:54 ` Ard Biesheuvel
2021-09-21 9:54 ` [PATCH 05/10] ARM: unwind: support unwinding across multiple stacks Ard Biesheuvel
` (5 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Ard Biesheuvel @ 2021-09-21 9:54 UTC (permalink / raw)
To: linux
Cc: linux-arm-kernel, Ard Biesheuvel, Arnd Bergmann, Kees Cook,
Keith Packard, Linus Walleij
The current irq_handler macro uses a literal load followed by an
explicit assignment of lr before an indirect call, both of which are
sub-optimal on recent ARM cores. Replace it by a mov_l call, which will
evaluate to a movw/movt pair on v7 cores, avoiding one of the two loads,
followed by a call to a newly introduced 'bl_m' macro, which will
evaluate to a blx instruction on cores that support it. (Setting lr
explicitly rather than via a bl or blx instruction confuses the return
address prediction that modern out-of-order cores use to speculatively
perform the function return.)
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
arch/arm/kernel/entry-armv.S | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S
index a54b5044d406..21896f702447 100644
--- a/arch/arm/kernel/entry-armv.S
+++ b/arch/arm/kernel/entry-armv.S
@@ -38,14 +38,12 @@
*/
.macro irq_handler
#ifdef CONFIG_GENERIC_IRQ_MULTI_HANDLER
- ldr r1, =handle_arch_irq
+ mov_l r1, handle_arch_irq
mov r0, sp
- badr lr, 9997f
- ldr pc, [r1]
+ bl_m [r1]
#else
arch_irq_handler_default
#endif
-9997:
.endm
.macro pabt_helper
--
2.30.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 05/10] ARM: unwind: support unwinding across multiple stacks
2021-09-21 9:53 [PATCH 00/10] ARM: add support for IRQ stacks Ard Biesheuvel
` (3 preceding siblings ...)
2021-09-21 9:54 ` [PATCH 04/10] ARM: optimize indirect call to handle_arch_irq for v7 cores Ard Biesheuvel
@ 2021-09-21 9:54 ` Ard Biesheuvel
2021-09-21 9:54 ` [PATCH 06/10] ARM: export dump_mem() to other objects Ard Biesheuvel
` (4 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Ard Biesheuvel @ 2021-09-21 9:54 UTC (permalink / raw)
To: linux
Cc: linux-arm-kernel, Ard Biesheuvel, Arnd Bergmann, Kees Cook,
Keith Packard, Linus Walleij
Implement support in the unwinder for dealing with multiple stacks.
This will be needed once we add support for IRQ stacks, or for the
overflow stack used by the vmap'ed stacks code.
This involves tracking the unwind opcodes that either update the virtual
stack pointer from another virtual register, or perform an explicit
subtract on the virtual stack pointer, and updating the low and high
bounds that we use to sanitize the stack pointer accordingly.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
arch/arm/kernel/unwind.c | 25 +++++++++++++-------
1 file changed, 16 insertions(+), 9 deletions(-)
diff --git a/arch/arm/kernel/unwind.c b/arch/arm/kernel/unwind.c
index 59fdf257bf8b..9cb9af3fc433 100644
--- a/arch/arm/kernel/unwind.c
+++ b/arch/arm/kernel/unwind.c
@@ -52,6 +52,7 @@ EXPORT_SYMBOL(__aeabi_unwind_cpp_pr2);
struct unwind_ctrl_block {
unsigned long vrs[16]; /* virtual register set */
const unsigned long *insn; /* pointer to the current instructions word */
+ unsigned long sp_low; /* lowest value of sp allowed */
unsigned long sp_high; /* highest value of sp allowed */
/*
* 1 : check for stack overflow for each register pop.
@@ -256,8 +257,12 @@ static int unwind_exec_pop_subset_r4_to_r13(struct unwind_ctrl_block *ctrl,
mask >>= 1;
reg++;
}
- if (!load_sp)
+ if (!load_sp) {
ctrl->vrs[SP] = (unsigned long)vsp;
+ } else {
+ ctrl->sp_low = ctrl->vrs[SP];
+ ctrl->sp_high = ALIGN(ctrl->sp_low, THREAD_SIZE);
+ }
return URC_OK;
}
@@ -313,9 +318,10 @@ static int unwind_exec_insn(struct unwind_ctrl_block *ctrl)
if ((insn & 0xc0) == 0x00)
ctrl->vrs[SP] += ((insn & 0x3f) << 2) + 4;
- else if ((insn & 0xc0) == 0x40)
+ else if ((insn & 0xc0) == 0x40) {
ctrl->vrs[SP] -= ((insn & 0x3f) << 2) + 4;
- else if ((insn & 0xf0) == 0x80) {
+ ctrl->sp_low = ctrl->vrs[SP];
+ } else if ((insn & 0xf0) == 0x80) {
unsigned long mask;
insn = (insn << 8) | unwind_get_byte(ctrl);
@@ -330,9 +336,11 @@ static int unwind_exec_insn(struct unwind_ctrl_block *ctrl)
if (ret)
goto error;
} else if ((insn & 0xf0) == 0x90 &&
- (insn & 0x0d) != 0x0d)
+ (insn & 0x0d) != 0x0d) {
ctrl->vrs[SP] = ctrl->vrs[insn & 0x0f];
- else if ((insn & 0xf0) == 0xa0) {
+ ctrl->sp_low = ctrl->vrs[SP];
+ ctrl->sp_high = ALIGN(ctrl->sp_low, THREAD_SIZE);
+ } else if ((insn & 0xf0) == 0xa0) {
ret = unwind_exec_pop_r4_to_rN(ctrl, insn);
if (ret)
goto error;
@@ -375,13 +383,12 @@ static int unwind_exec_insn(struct unwind_ctrl_block *ctrl)
*/
int unwind_frame(struct stackframe *frame)
{
- unsigned long low;
const struct unwind_idx *idx;
struct unwind_ctrl_block ctrl;
/* store the highest address on the stack to avoid crossing it*/
- low = frame->sp;
- ctrl.sp_high = ALIGN(low, THREAD_SIZE);
+ ctrl.sp_low = frame->sp;
+ ctrl.sp_high = ALIGN(ctrl.sp_low, THREAD_SIZE);
pr_debug("%s(pc = %08lx lr = %08lx sp = %08lx)\n", __func__,
frame->pc, frame->lr, frame->sp);
@@ -437,7 +444,7 @@ int unwind_frame(struct stackframe *frame)
urc = unwind_exec_insn(&ctrl);
if (urc < 0)
return urc;
- if (ctrl.vrs[SP] < low || ctrl.vrs[SP] >= ctrl.sp_high)
+ if (ctrl.vrs[SP] < ctrl.sp_low || ctrl.vrs[SP] > ctrl.sp_high)
return -URC_FAILURE;
}
--
2.30.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 06/10] ARM: export dump_mem() to other objects
2021-09-21 9:53 [PATCH 00/10] ARM: add support for IRQ stacks Ard Biesheuvel
` (4 preceding siblings ...)
2021-09-21 9:54 ` [PATCH 05/10] ARM: unwind: support unwinding across multiple stacks Ard Biesheuvel
@ 2021-09-21 9:54 ` Ard Biesheuvel
2021-09-21 9:54 ` [PATCH 07/10] ARM: unwind: dump exception stack from calling frame Ard Biesheuvel
` (3 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Ard Biesheuvel @ 2021-09-21 9:54 UTC (permalink / raw)
To: linux
Cc: linux-arm-kernel, Ard Biesheuvel, Arnd Bergmann, Kees Cook,
Keith Packard, Linus Walleij
The unwind info based stack unwinder will make its own call to
dump_mem() to dump the exception stack, so give it external linkage.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
arch/arm/include/asm/stacktrace.h | 2 ++
arch/arm/kernel/traps.c | 7 +++----
2 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/arch/arm/include/asm/stacktrace.h b/arch/arm/include/asm/stacktrace.h
index 2d76a2e29f05..e86f55155500 100644
--- a/arch/arm/include/asm/stacktrace.h
+++ b/arch/arm/include/asm/stacktrace.h
@@ -27,5 +27,7 @@ void arm_get_current_stackframe(struct pt_regs *regs, struct stackframe *frame)
extern int unwind_frame(struct stackframe *frame);
extern void walk_stackframe(struct stackframe *frame,
int (*fn)(struct stackframe *, void *), void *data);
+extern void dump_mem(const char *lvl, const char *str, unsigned long bottom,
+ unsigned long top);
#endif /* __ASM_STACKTRACE_H */
diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c
index 195dff58bafc..e4698605bb72 100644
--- a/arch/arm/kernel/traps.c
+++ b/arch/arm/kernel/traps.c
@@ -35,6 +35,7 @@
#include <asm/ptrace.h>
#include <asm/unwind.h>
#include <asm/tls.h>
+#include <asm/stacktrace.h>
#include <asm/system_misc.h>
#include <asm/opcodes.h>
@@ -60,8 +61,6 @@ static int __init user_debug_setup(char *str)
__setup("user_debug=", user_debug_setup);
#endif
-static void dump_mem(const char *, const char *, unsigned long, unsigned long);
-
void dump_backtrace_entry(unsigned long where, unsigned long from,
unsigned long frame, const char *loglvl)
{
@@ -118,8 +117,8 @@ static int verify_stack(unsigned long sp)
/*
* Dump out the contents of some memory nicely...
*/
-static void dump_mem(const char *lvl, const char *str, unsigned long bottom,
- unsigned long top)
+void dump_mem(const char *lvl, const char *str, unsigned long bottom,
+ unsigned long top)
{
unsigned long first;
int i;
--
2.30.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 07/10] ARM: unwind: dump exception stack from calling frame
2021-09-21 9:53 [PATCH 00/10] ARM: add support for IRQ stacks Ard Biesheuvel
` (5 preceding siblings ...)
2021-09-21 9:54 ` [PATCH 06/10] ARM: export dump_mem() to other objects Ard Biesheuvel
@ 2021-09-21 9:54 ` Ard Biesheuvel
2021-09-21 9:54 ` [PATCH 08/10] ARM: implement IRQ stacks Ard Biesheuvel
` (2 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Ard Biesheuvel @ 2021-09-21 9:54 UTC (permalink / raw)
To: linux
Cc: linux-arm-kernel, Ard Biesheuvel, Arnd Bergmann, Kees Cook,
Keith Packard, Linus Walleij
The existing code that dumps the contents of the pt_regs structure
passed to __entry routines does so while unwinding the callee frame, and
dereferences the stack pointer as a struct pt_regs*. This will no longer
work when we enable support for IRQ or overflow stacks, because the
struct pt_regs may live on the task stack, while we are executing from
another stack.
The unwinder has access to this information, but only while unwinding
the calling frame. So let's combine the exception stack dumping code
with the handling of the calling frame as well. By printing it before
dumping the caller/callee addresses, the output order is preserved.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
arch/arm/include/asm/stacktrace.h | 11 +++++++++++
arch/arm/kernel/traps.c | 3 ++-
arch/arm/kernel/unwind.c | 8 +++++++-
3 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/arch/arm/include/asm/stacktrace.h b/arch/arm/include/asm/stacktrace.h
index e86f55155500..8dcad8156561 100644
--- a/arch/arm/include/asm/stacktrace.h
+++ b/arch/arm/include/asm/stacktrace.h
@@ -13,6 +13,17 @@ struct stackframe {
unsigned long sp;
unsigned long lr;
unsigned long pc;
+
+#ifdef CONFIG_ARM_UNWIND
+ /*
+ * This field is used to track the stack pointer value when calling
+ * __entry routines. This is needed when IRQ stacks and overflow stacks
+ * are used, because in that case, the struct pt_regs passed to these
+ * __entry routines may be at the top of the task stack, while we are
+ * executing from another stack.
+ */
+ unsigned long sp_low;
+#endif
};
static __always_inline
diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c
index e4698605bb72..89be21ec3b52 100644
--- a/arch/arm/kernel/traps.c
+++ b/arch/arm/kernel/traps.c
@@ -74,7 +74,8 @@ void dump_backtrace_entry(unsigned long where, unsigned long from,
loglvl, where, from);
#endif
- if (in_entry_text(from) && end <= ALIGN(frame, THREAD_SIZE))
+ if (!IS_ENABLED(CONFIG_UNWINDER_ARM) &&
+ in_entry_text(from) && end <= ALIGN(frame, THREAD_SIZE))
dump_mem(loglvl, "Exception stack", frame + 4, end);
}
diff --git a/arch/arm/kernel/unwind.c b/arch/arm/kernel/unwind.c
index 9cb9af3fc433..b7a6141c342f 100644
--- a/arch/arm/kernel/unwind.c
+++ b/arch/arm/kernel/unwind.c
@@ -29,6 +29,7 @@
#include <linux/spinlock.h>
#include <linux/list.h>
+#include <asm/sections.h>
#include <asm/stacktrace.h>
#include <asm/traps.h>
#include <asm/unwind.h>
@@ -459,6 +460,7 @@ int unwind_frame(struct stackframe *frame)
frame->sp = ctrl.vrs[SP];
frame->lr = ctrl.vrs[LR];
frame->pc = ctrl.vrs[PC];
+ frame->sp_low = ctrl.sp_low;
return URC_OK;
}
@@ -502,7 +504,11 @@ void unwind_backtrace(struct pt_regs *regs, struct task_struct *tsk,
urc = unwind_frame(&frame);
if (urc < 0)
break;
- dump_backtrace_entry(where, frame.pc, frame.sp - 4, loglvl);
+ if (in_entry_text(where))
+ dump_mem(loglvl, "Exception stack", frame.sp_low,
+ frame.sp_low + sizeof(struct pt_regs));
+
+ dump_backtrace_entry(where, frame.pc, 0, loglvl);
}
}
--
2.30.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 08/10] ARM: implement IRQ stacks
2021-09-21 9:53 [PATCH 00/10] ARM: add support for IRQ stacks Ard Biesheuvel
` (6 preceding siblings ...)
2021-09-21 9:54 ` [PATCH 07/10] ARM: unwind: dump exception stack from calling frame Ard Biesheuvel
@ 2021-09-21 9:54 ` Ard Biesheuvel
2021-09-21 9:54 ` [PATCH 09/10] ARM: call_with_stack: add unwind support Ard Biesheuvel
2021-09-21 9:54 ` [PATCH 10/10] ARM: run softirqs on the per-CPU IRQ stack Ard Biesheuvel
9 siblings, 0 replies; 14+ messages in thread
From: Ard Biesheuvel @ 2021-09-21 9:54 UTC (permalink / raw)
To: linux
Cc: linux-arm-kernel, Ard Biesheuvel, Arnd Bergmann, Kees Cook,
Keith Packard, Linus Walleij
Now that we no longer rely on the stack pointer to access the current
task struct or thread info, we can implement support for IRQ stacks
cleanly as well.
Define a per-CPU IRQ stack and switch to this stack when taking an IRQ,
provided that we were not already using that stack in the interrupted
context. This is never the case for IRQs taken from user space, but ones
taken while running in the kernel could fire while one taken from user
space has not completed yet.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
arch/arm/Kconfig | 4 ++
arch/arm/kernel/entry-armv.S | 40 ++++++++++++++++++--
arch/arm/kernel/irq.c | 23 +++++++++++
arch/arm/kernel/traps.c | 15 +++++++-
arch/arm/lib/backtrace-clang.S | 8 ++++
arch/arm/lib/backtrace.S | 8 ++++
6 files changed, 94 insertions(+), 4 deletions(-)
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 4f61c9789e7f..a8c1db0736f3 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -1163,6 +1163,10 @@ config CURRENT_POINTER_IN_TPIDRURO
def_bool y
depends on SMP && CPU_32v6K && !CPU_V6
+config IRQSTACKS
+ def_bool y
+ depends on GENERIC_IRQ_MULTI_HANDLER && THREAD_INFO_IN_TASK
+
config ARM_CPU_TOPOLOGY
bool "Support cpu topology definition"
depends on SMP && CPU_V7
diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S
index 21896f702447..c1b1b125f647 100644
--- a/arch/arm/kernel/entry-armv.S
+++ b/arch/arm/kernel/entry-armv.S
@@ -36,11 +36,45 @@
/*
* Interrupt handling.
*/
- .macro irq_handler
+ .macro irq_handler, from_user:req
#ifdef CONFIG_GENERIC_IRQ_MULTI_HANDLER
mov_l r1, handle_arch_irq
mov r0, sp
+#ifdef CONFIG_IRQSTACKS
+ mov_l r2, irq_stack_ptr @ Take base address
+ mrc p15, 0, r3, c13, c0, 4 @ Get CPU offset
+ mov r7, sp @ Preserve original SP
+ ldr sp, [r2, r3] @ Load SP from per-CPU var
+ .if \from_user == 0
+ @
+ @ If we took the interrupt while running in the kernel, we may already
+ @ be using the IRQ stack, so revert to the original value in that case.
+ @
+ subs r3, sp, r7 @ SP above bottom of IRQ stack?
+ rsbscs r3, r3, #THREAD_SIZE @ ... and below the top?
+ movcs sp, r7 @ If so, revert to incoming SP
+
+ @
+ @ Inform the unwinder where the next frame lives
+ @
+#if defined(CONFIG_UNWINDER_FRAME_POINTER) && defined(CONFIG_CC_IS_GCC)
+ movcc ip, r7 @ Store the old SP in the frame record.
+ movcc lr, pc @ Make LR point into .entry.text so
+ @ that we will get a dump of the
+ @ exception stack for this frame.
+ stmdbcc sp!, {fp, ip, lr, pc}
+ addcc fp, sp, #12
+#elif defined(CONFIG_UNWINDER_ARM)
+ARM( mov fp, r7 )
+ARM( .setfp fp, sp )
+THUMB( .setfp r7, sp )
+#endif
+ .endif
+#endif
bl_m [r1]
+#ifdef CONFIG_IRQSTACKS
+ mov sp, r7 @ Restore original SP
+#endif
#else
arch_irq_handler_default
#endif
@@ -200,7 +234,7 @@ ENDPROC(__dabt_svc)
.align 5
__irq_svc:
svc_entry
- irq_handler
+ irq_handler from_user=0
#ifdef CONFIG_PREEMPTION
ldr r8, [tsk, #TI_PREEMPT] @ get preempt count
@@ -427,7 +461,7 @@ ENDPROC(__dabt_usr)
__irq_usr:
usr_entry
kuser_cmpxchg_check
- irq_handler
+ irq_handler from_user=1
get_thread_info tsk
mov why, #0
b ret_to_user_from_irq
diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c
index 20ab1e607522..58af2adb1583 100644
--- a/arch/arm/kernel/irq.c
+++ b/arch/arm/kernel/irq.c
@@ -43,6 +43,25 @@
unsigned long irq_err_count;
+#ifdef CONFIG_IRQSTACKS
+
+asmlinkage DEFINE_PER_CPU_READ_MOSTLY(u8 *, irq_stack_ptr);
+
+static void __init init_irq_stacks(void)
+{
+ u8 *stack;
+ int cpu;
+
+ for_each_possible_cpu(cpu) {
+ stack = (u8 *)__get_free_pages(GFP_KERNEL, THREAD_SIZE_ORDER);
+ if (WARN_ON(!stack))
+ break;
+ per_cpu(irq_stack_ptr, cpu) = &stack[THREAD_SIZE];
+ }
+}
+
+#endif
+
int arch_show_interrupts(struct seq_file *p, int prec)
{
#ifdef CONFIG_FIQ
@@ -99,6 +118,10 @@ void __init init_IRQ(void)
{
int ret;
+#ifdef CONFIG_IRQSTACKS
+ init_irq_stacks();
+#endif
+
if (IS_ENABLED(CONFIG_OF) && !machine_desc->init_irq)
irqchip_init();
else
diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c
index 89be21ec3b52..b42c446cec9a 100644
--- a/arch/arm/kernel/traps.c
+++ b/arch/arm/kernel/traps.c
@@ -66,6 +66,19 @@ void dump_backtrace_entry(unsigned long where, unsigned long from,
{
unsigned long end = frame + 4 + sizeof(struct pt_regs);
+ if (IS_ENABLED(CONFIG_UNWINDER_FRAME_POINTER) &&
+ IS_ENABLED(CONFIG_CC_IS_GCC) &&
+ end > ALIGN(frame, THREAD_SIZE)) {
+ /*
+ * If we are walking past the end of the stack, it may be due
+ * to the fact that we are on an IRQ or overflow stack. In this
+ * case, we can load the address of the other stack from the
+ * frame record.
+ */
+ frame = ((unsigned long *)frame)[-2] - 4;
+ end = frame + 4 + sizeof(struct pt_regs);
+ }
+
#ifdef CONFIG_KALLSYMS
printk("%s[<%08lx>] (%ps) from [<%08lx>] (%pS)\n",
loglvl, where, (void *)where, from, (void *)from);
@@ -278,7 +291,7 @@ static int __die(const char *str, int err, struct pt_regs *regs)
if (!user_mode(regs) || in_interrupt()) {
dump_mem(KERN_EMERG, "Stack: ", regs->ARM_sp,
- THREAD_SIZE + (unsigned long)task_stack_page(tsk));
+ ALIGN(regs->ARM_sp, THREAD_SIZE));
dump_backtrace(regs, tsk, KERN_EMERG);
dump_instr(KERN_EMERG, regs);
}
diff --git a/arch/arm/lib/backtrace-clang.S b/arch/arm/lib/backtrace-clang.S
index 5b2cdb1003e3..9fefb391bb39 100644
--- a/arch/arm/lib/backtrace-clang.S
+++ b/arch/arm/lib/backtrace-clang.S
@@ -197,6 +197,14 @@ finished_setup:
cmp sv_fp, frame @ next frame must be
mov frame, sv_fp @ above the current frame
+#ifdef CONFIG_IRQSTACKS
+ @
+ @ Kernel stacks may be discontiguous in memory. If the next
+ @ frame is below the previous frame, accept it as long as it
+ @ lives in kernel memory.
+ @
+ cmpls sv_fp, #PAGE_OFFSET
+#endif
bhi for_each_frame
1006: adr r0, .Lbad
diff --git a/arch/arm/lib/backtrace.S b/arch/arm/lib/backtrace.S
index e8408f22d4dc..293a2716bd20 100644
--- a/arch/arm/lib/backtrace.S
+++ b/arch/arm/lib/backtrace.S
@@ -98,6 +98,14 @@ for_each_frame: tst frame, mask @ Check for address exceptions
cmp sv_fp, frame @ next frame must be
mov frame, sv_fp @ above the current frame
+#ifdef CONFIG_IRQSTACKS
+ @
+ @ Kernel stacks may be discontiguous in memory. If the next
+ @ frame is below the previous frame, accept it as long as it
+ @ lives in kernel memory.
+ @
+ cmpls sv_fp, #PAGE_OFFSET
+#endif
bhi for_each_frame
1006: adr r0, .Lbad
--
2.30.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 09/10] ARM: call_with_stack: add unwind support
2021-09-21 9:53 [PATCH 00/10] ARM: add support for IRQ stacks Ard Biesheuvel
` (7 preceding siblings ...)
2021-09-21 9:54 ` [PATCH 08/10] ARM: implement IRQ stacks Ard Biesheuvel
@ 2021-09-21 9:54 ` Ard Biesheuvel
2021-09-21 9:54 ` [PATCH 10/10] ARM: run softirqs on the per-CPU IRQ stack Ard Biesheuvel
9 siblings, 0 replies; 14+ messages in thread
From: Ard Biesheuvel @ 2021-09-21 9:54 UTC (permalink / raw)
To: linux
Cc: linux-arm-kernel, Ard Biesheuvel, Arnd Bergmann, Kees Cook,
Keith Packard, Linus Walleij
Restructure the code and add the unwind annotations so that both the
frame pointer unwinder as well as the ELF unwind info based unwinder
will be able to follow the call stack through call_with_stack().
Note that the former cannot support GCC and Clang at the same time, as
they use a different idiom for the prologue/epilogue. So the code uses
the GCC idiom, adding full frame pointer based unwind support for GCC
while preserving the existing behavior of the Clang version, which
simply omits call_with_stack() from its call stack.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
arch/arm/lib/call_with_stack.S | 44 +++++++++++++++++---
1 file changed, 38 insertions(+), 6 deletions(-)
diff --git a/arch/arm/lib/call_with_stack.S b/arch/arm/lib/call_with_stack.S
index 28b0341ae786..133dffa2404a 100644
--- a/arch/arm/lib/call_with_stack.S
+++ b/arch/arm/lib/call_with_stack.S
@@ -8,25 +8,57 @@
#include <linux/linkage.h>
#include <asm/assembler.h>
+#include <asm/unwind.h>
/*
* void call_with_stack(void (*fn)(void *), void *arg, void *sp)
*
* Change the stack to that pointed at by sp, then invoke fn(arg) with
* the new stack.
+ *
+ * The sequence below follows the APCS frame convention for frame pointer
+ * unwinding, and implements the unwinder annotations needed by the EABI
+ * unwinder.
+ */
+
+#if defined(CONFIG_THUMB2_KERNEL) || \
+ (defined(CONFIG_UNWINDER_FRAME_POINTER) && defined(CONFIG_CC_IS_CLANG))
+/*
+ * Thumb-2 builds must use R7 as the frame pointer due to the way our unwind
+ * info based unwinder is constructed.
+ *
+ * The code below uses the GCC idiom for managing the frame pointer in the
+ * function prologue and epilogue, which Clang does not support. So the best we
+ * can do here is not touch the frame pointer at all: this will simply omit
+ * this frame when unwinding the call stack. So use R7 in this case as well,
+ * and leave R11 unmodified.
*/
+ fpreg .req r7
+#else
+ fpreg .req fp
+#endif
+
ENTRY(call_with_stack)
- str sp, [r2, #-4]!
- str lr, [r2, #-4]!
+UNWIND( .fnstart )
+UNWIND( .movsp ip )
+ mov ip, sp
+
+UNWIND( .pad #4 )
+UNWIND( .save {fpreg, ip, lr} )
+THUMB( sub sp, #4 )
+ push {fpreg, ip, lr ARM(, pc)}
+
+UNWIND( .setfp fpreg, ip, #-4 )
+ sub fpreg, ip, #4
mov sp, r2
mov r2, r0
mov r0, r1
- badr lr, 1f
- ret r2
+ bl_r r2
-1: ldr lr, [sp]
- ldr sp, [sp, #4]
+ ldmdb fpreg, {fpreg, ip, lr}
+ mov sp, ip
ret lr
+UNWIND( .fnend )
ENDPROC(call_with_stack)
--
2.30.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 10/10] ARM: run softirqs on the per-CPU IRQ stack
2021-09-21 9:53 [PATCH 00/10] ARM: add support for IRQ stacks Ard Biesheuvel
` (8 preceding siblings ...)
2021-09-21 9:54 ` [PATCH 09/10] ARM: call_with_stack: add unwind support Ard Biesheuvel
@ 2021-09-21 9:54 ` Ard Biesheuvel
9 siblings, 0 replies; 14+ messages in thread
From: Ard Biesheuvel @ 2021-09-21 9:54 UTC (permalink / raw)
To: linux
Cc: linux-arm-kernel, Ard Biesheuvel, Arnd Bergmann, Kees Cook,
Keith Packard, Linus Walleij
Now that we have enabled IRQ stacks, any softIRQs that are handled over
the back of a hard IRQ will run from the IRQ stack as well. However, any
synchronous softirq processing that happens when re-enabling softIRQs
from task context will still execute on that task's stack.
So let's wire up the existing infrastructure to run these softIRQs from
the IRQ stack as well.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
arch/arm/Kconfig | 2 ++
arch/arm/kernel/irq.c | 8 ++++++++
2 files changed, 10 insertions(+)
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index a8c1db0736f3..d46b243e1b26 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -1166,6 +1166,8 @@ config CURRENT_POINTER_IN_TPIDRURO
config IRQSTACKS
def_bool y
depends on GENERIC_IRQ_MULTI_HANDLER && THREAD_INFO_IN_TASK
+ select HAVE_IRQ_EXIT_ON_IRQ_STACK
+ select HAVE_SOFTIRQ_ON_OWN_STACK
config ARM_CPU_TOPOLOGY
bool "Support cpu topology definition"
diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c
index 58af2adb1583..feb07f703a98 100644
--- a/arch/arm/kernel/irq.c
+++ b/arch/arm/kernel/irq.c
@@ -41,6 +41,8 @@
#include <asm/mach/irq.h>
#include <asm/mach/time.h>
+#include "reboot.h"
+
unsigned long irq_err_count;
#ifdef CONFIG_IRQSTACKS
@@ -60,6 +62,12 @@ static void __init init_irq_stacks(void)
}
}
+void do_softirq_own_stack(void)
+{
+ call_with_stack((void *)__do_softirq, NULL,
+ __this_cpu_read(irq_stack_ptr));
+}
+
#endif
int arch_show_interrupts(struct seq_file *p, int prec)
--
2.30.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 14+ messages in thread