* [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP
@ 2021-11-30 12:58 Ard Biesheuvel
2021-11-30 12:58 ` [PATCH v2 01/12] ARM: riscpc: use GENERIC_IRQ_MULTI_HANDLER Ard Biesheuvel
` (12 more replies)
0 siblings, 13 replies; 23+ messages in thread
From: Ard Biesheuvel @ 2021-11-30 12:58 UTC (permalink / raw)
To: linux-arm-kernel, linux
Cc: Ard Biesheuvel, Nicolas Pitre, Arnd Bergmann, Kees Cook,
Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren
Enable the use of the TLS register to hold the 'current' pointer for all
configurations that can support it, including non-SMP ones that target
v6k or later CPUs, and multi-platform SMP ones that also support v6
based UP systems.
The remaining configurations are all strictly UP, which means we can
switch to a global variable to hold the current pointer. By doing this,
we can enable THREAD_INFO_IN_TASK, which moves thread info off the
stack, protecting it from overflows. It also permits us to enable IRQ
stacks and vmap'ed stacks for UP configurations as well.
Supporting v6 cores without SMP extensions in SMP configurations (e.g.,
omap2plus_defconfig or imx_v6_v7_defconfig) makes this a bit tricky, and
this is a feature we may consider dropping entirely in the future. But
for the time being, we can support this mode as well.
The accesses to the global variable holding 'current' are constructed in
a way that ensures that no literal pool accesses (and associated D-cache
misses) are needed unless the access is from a module and module PLTs
are enabled. This means that accessing 'current' is just as costly as
before, as it used to require some arithmetic involving the stack
pointer and a load from the thread_info::task field.
However, accessing thread_info itself now also involves a load, although
it should be noted that all thread_info and current accesses now go via
the same variable, which is therefore expected to be hot in the caches
at all times.
Changes since RFC/v1:
- add five preparatory patches that move RiscPC, IOP32x and Footbridge
to GENERIC_IRQ_MULTI_HANDLER so that even these ancient platforms can
benefit from the IRQ stacks changes for UP that this series proposes
(contributed by Arnd)
- fix various issues related to SMP+v6 corner cases that were caught by
kernelci testing;
- add acks from Nico and Linus (thanks!)
Cc: Russell King <linux@armlinux.org.uk>
Cc: Nicolas Pitre <nico@fluxnic.net>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Kees Cook <keescook@chromium.org>
Cc: Keith Packard <keithpac@amazon.com>
Cc: Linus Walleij <linus.walleij@linaro.org>
Cc: Nick Desaulniers <ndesaulniers@google.com>
Cc: Tony Lindgren <tony@atomide.com>
Ard Biesheuvel (7):
ARM: entry: preserve thread_info pointer in switch_to
ARM: module: implement support for PC-relative group relocations
ARM: assembler: add optimized ldr/str macros to load variables from
memory
ARM: percpu: add SMP_ON_UP support
ARM: use TLS register for 'current' on !SMP as well
ARM: smp: defer TPIDRURO update for SMP v6 configurations too
ARM: implement THREAD_INFO_IN_TASK for uniprocessor systems
Arnd Bergmann (5):
ARM: riscpc: use GENERIC_IRQ_MULTI_HANDLER
ARM: footbridge: use GENERIC_IRQ_MULTI_HANDLER
ARM: iop32x: offset IRQ numbers by 1
ARM: iop32x: use GENERIC_IRQ_MULTI_HANDLER
ARM: remove old-style irq entry
arch/arm/Kconfig | 22 +--
arch/arm/include/asm/assembler.h | 185 ++++++++++++++++----
arch/arm/include/asm/current.h | 37 ++--
arch/arm/include/asm/elf.h | 3 +
arch/arm/include/asm/entry-macro-multi.S | 16 --
arch/arm/include/asm/hardware/entry-macro-iomd.S | 131 --------------
arch/arm/include/asm/insn.h | 24 +++
arch/arm/include/asm/irq.h | 1 -
arch/arm/include/asm/mach/arch.h | 2 -
arch/arm/include/asm/percpu.h | 25 ++-
arch/arm/include/asm/switch_to.h | 3 +-
arch/arm/include/asm/thread_info.h | 27 ---
arch/arm/include/asm/tls.h | 13 +-
arch/arm/kernel/asm-offsets.c | 3 -
arch/arm/kernel/entry-armv.S | 48 ++---
arch/arm/kernel/entry-common.S | 10 +-
arch/arm/kernel/entry-header.S | 13 +-
arch/arm/kernel/head-common.S | 4 +-
arch/arm/kernel/irq.c | 17 --
arch/arm/kernel/module.c | 63 +++++++
arch/arm/kernel/process.c | 7 +-
arch/arm/kernel/sleep.S | 4 +-
arch/arm/kernel/smp.c | 11 ++
arch/arm/kernel/traps.c | 4 +
arch/arm/mach-footbridge/common.c | 87 +++++++++
arch/arm/mach-footbridge/include/mach/entry-macro.S | 107 -----------
arch/arm/mach-iop32x/cp6.c | 10 +-
arch/arm/mach-iop32x/include/mach/entry-macro.S | 31 ----
arch/arm/mach-iop32x/include/mach/irqs.h | 2 +-
arch/arm/mach-iop32x/iop3xx.h | 1 +
arch/arm/mach-iop32x/irq.c | 23 +++
arch/arm/mach-iop32x/irqs.h | 60 ++++---
arch/arm/mach-rpc/fiq.S | 5 +-
arch/arm/mach-rpc/include/mach/entry-macro.S | 13 --
arch/arm/mach-rpc/irq.c | 145 +++++++++++++++
arch/arm/mm/Kconfig | 1 +
36 files changed, 656 insertions(+), 502 deletions(-)
delete mode 100644 arch/arm/include/asm/entry-macro-multi.S
delete mode 100644 arch/arm/include/asm/hardware/entry-macro-iomd.S
delete mode 100644 arch/arm/mach-footbridge/include/mach/entry-macro.S
delete mode 100644 arch/arm/mach-iop32x/include/mach/entry-macro.S
delete mode 100644 arch/arm/mach-rpc/include/mach/entry-macro.S
--
2.30.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v2 01/12] ARM: riscpc: use GENERIC_IRQ_MULTI_HANDLER 2021-11-30 12:58 [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Ard Biesheuvel @ 2021-11-30 12:58 ` Ard Biesheuvel 2021-11-30 12:58 ` [PATCH v2 02/12] ARM: footbridge: " Ard Biesheuvel ` (11 subsequent siblings) 12 siblings, 0 replies; 23+ messages in thread From: Ard Biesheuvel @ 2021-11-30 12:58 UTC (permalink / raw) To: linux-arm-kernel, linux Cc: Ard Biesheuvel, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren From: Arnd Bergmann <arnd@arndb.de> This is one of the last platforms using the old entry path. While this code path is spread over a few files, it is fairly straightforward to convert it into an equivalent C version, leaving the existing algorithm and all the priority handling the same. Unlike most irqchip drivers, this means reading the status register(s) in a loop and always handling the highest-priority irq first. The IOMD_IRQREQC and IOMD_IRQREQD registers are not actaully used here, but I left the code in place for the time being, to keep the conversion as direct as possible. It could be removed in a cleanup on top. Signed-off-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- arch/arm/Kconfig | 1 + arch/arm/include/asm/hardware/entry-macro-iomd.S | 131 ------------------ arch/arm/mach-rpc/fiq.S | 5 +- arch/arm/mach-rpc/include/mach/entry-macro.S | 13 -- arch/arm/mach-rpc/irq.c | 145 ++++++++++++++++++++ 5 files changed, 149 insertions(+), 146 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index e2ab72f2bf4a..25f1868e5703 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -444,6 +444,7 @@ config ARCH_RPC select ARM_HAS_SG_CHAIN select CPU_SA110 select FIQ + select GENERIC_IRQ_MULTI_HANDLER select HAVE_PATA_PLATFORM select ISA_DMA_API select LEGACY_TIMER_TICK diff --git a/arch/arm/include/asm/hardware/entry-macro-iomd.S b/arch/arm/include/asm/hardware/entry-macro-iomd.S deleted file mode 100644 index f7692731e514..000000000000 --- a/arch/arm/include/asm/hardware/entry-macro-iomd.S +++ /dev/null @@ -1,131 +0,0 @@ -/* - * arch/arm/include/asm/hardware/entry-macro-iomd.S - * - * Low-level IRQ helper macros for IOC/IOMD based platforms - * - * This file is licensed under the terms of the GNU General Public - * License version 2. This program is licensed "as is" without any - * warranty of any kind, whether express or implied. - */ - -/* IOC / IOMD based hardware */ -#include <asm/hardware/iomd.h> - - .macro get_irqnr_and_base, irqnr, irqstat, base, tmp - ldrb \irqstat, [\base, #IOMD_IRQREQB] @ get high priority first - ldr \tmp, =irq_prio_h - teq \irqstat, #0 -#ifdef IOMD_BASE - ldrbeq \irqstat, [\base, #IOMD_DMAREQ] @ get dma - addeq \tmp, \tmp, #256 @ irq_prio_h table size - teqeq \irqstat, #0 - bne 2406f -#endif - ldrbeq \irqstat, [\base, #IOMD_IRQREQA] @ get low priority - addeq \tmp, \tmp, #256 @ irq_prio_d table size - teqeq \irqstat, #0 -#ifdef IOMD_IRQREQC - ldrbeq \irqstat, [\base, #IOMD_IRQREQC] - addeq \tmp, \tmp, #256 @ irq_prio_l table size - teqeq \irqstat, #0 -#endif -#ifdef IOMD_IRQREQD - ldrbeq \irqstat, [\base, #IOMD_IRQREQD] - addeq \tmp, \tmp, #256 @ irq_prio_lc table size - teqeq \irqstat, #0 -#endif -2406: ldrbne \irqnr, [\tmp, \irqstat] @ get IRQ number - .endm - -/* - * Interrupt table (incorporates priority). Please note that we - * rely on the order of these tables (see above code). - */ - .align 5 -irq_prio_h: .byte 0, 8, 9, 8,10,10,10,10,11,11,11,11,10,10,10,10 - .byte 12, 8, 9, 8,10,10,10,10,11,11,11,11,10,10,10,10 - .byte 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10 - .byte 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10 - .byte 14,14,14,14,10,10,10,10,11,11,11,11,10,10,10,10 - .byte 14,14,14,14,10,10,10,10,11,11,11,11,10,10,10,10 - .byte 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10 - .byte 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10 - .byte 15,15,15,15,10,10,10,10,11,11,11,11,10,10,10,10 - .byte 15,15,15,15,10,10,10,10,11,11,11,11,10,10,10,10 - .byte 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10 - .byte 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10 - .byte 15,15,15,15,10,10,10,10,11,11,11,11,10,10,10,10 - .byte 15,15,15,15,10,10,10,10,11,11,11,11,10,10,10,10 - .byte 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10 - .byte 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10 -#ifdef IOMD_BASE -irq_prio_d: .byte 0,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 - .byte 20,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 - .byte 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 - .byte 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 - .byte 22,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 - .byte 22,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 - .byte 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 - .byte 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 - .byte 23,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 - .byte 23,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 - .byte 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 - .byte 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 - .byte 22,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 - .byte 22,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 - .byte 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 - .byte 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16 -#endif -irq_prio_l: .byte 0, 0, 1, 0, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3 - .byte 4, 0, 1, 0, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3 - .byte 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5 - .byte 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5 - .byte 6, 6, 6, 6, 6, 6, 6, 6, 3, 3, 3, 3, 3, 3, 3, 3 - .byte 6, 6, 6, 6, 6, 6, 6, 6, 3, 3, 3, 3, 3, 3, 3, 3 - .byte 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5 - .byte 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5 - .byte 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7 - .byte 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7 - .byte 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7 - .byte 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7 - .byte 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7 - .byte 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7 - .byte 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7 - .byte 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7 -#ifdef IOMD_IRQREQC -irq_prio_lc: .byte 24,24,25,24,26,26,26,26,27,27,27,27,27,27,27,27 - .byte 28,24,25,24,26,26,26,26,27,27,27,27,27,27,27,27 - .byte 29,29,29,29,29,29,29,29,29,29,29,29,29,29,29,29 - .byte 29,29,29,29,29,29,29,29,29,29,29,29,29,29,29,29 - .byte 30,30,30,30,30,30,30,30,27,27,27,27,27,27,27,27 - .byte 30,30,30,30,30,30,30,30,27,27,27,27,27,27,27,27 - .byte 29,29,29,29,29,29,29,29,29,29,29,29,29,29,29,29 - .byte 29,29,29,29,29,29,29,29,29,29,29,29,29,29,29,29 - .byte 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31 - .byte 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31 - .byte 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31 - .byte 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31 - .byte 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31 - .byte 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31 - .byte 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31 - .byte 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31 -#endif -#ifdef IOMD_IRQREQD -irq_prio_ld: .byte 40,40,41,40,42,42,42,42,43,43,43,43,43,43,43,43 - .byte 44,40,41,40,42,42,42,42,43,43,43,43,43,43,43,43 - .byte 45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45 - .byte 45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45 - .byte 46,46,46,46,46,46,46,46,43,43,43,43,43,43,43,43 - .byte 46,46,46,46,46,46,46,46,43,43,43,43,43,43,43,43 - .byte 45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45 - .byte 45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45 - .byte 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47 - .byte 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47 - .byte 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47 - .byte 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47 - .byte 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47 - .byte 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47 - .byte 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47 - .byte 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47 -#endif - diff --git a/arch/arm/mach-rpc/fiq.S b/arch/arm/mach-rpc/fiq.S index 0de83e9b0b39..087bdf4bc093 100644 --- a/arch/arm/mach-rpc/fiq.S +++ b/arch/arm/mach-rpc/fiq.S @@ -2,10 +2,11 @@ #include <linux/linkage.h> #include <asm/assembler.h> #include <mach/hardware.h> -#include <mach/entry-macro.S> - .text + .equ ioc_base_high, IOC_BASE & 0xff000000 + .equ ioc_base_low, IOC_BASE & 0x00ff0000 + .text .global rpc_default_fiq_end ENTRY(rpc_default_fiq_start) mov r12, #ioc_base_high diff --git a/arch/arm/mach-rpc/include/mach/entry-macro.S b/arch/arm/mach-rpc/include/mach/entry-macro.S deleted file mode 100644 index a6d1a9f4bb79..000000000000 --- a/arch/arm/mach-rpc/include/mach/entry-macro.S +++ /dev/null @@ -1,13 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#include <mach/hardware.h> -#include <asm/hardware/entry-macro-iomd.S> - - .equ ioc_base_high, IOC_BASE & 0xff000000 - .equ ioc_base_low, IOC_BASE & 0x00ff0000 - - .macro get_irqnr_preamble, base, tmp - mov \base, #ioc_base_high @ point at IOC - .if ioc_base_low - orr \base, \base, #ioc_base_low - .endif - .endm diff --git a/arch/arm/mach-rpc/irq.c b/arch/arm/mach-rpc/irq.c index 803aeb126f0e..ff473746f6a3 100644 --- a/arch/arm/mach-rpc/irq.c +++ b/arch/arm/mach-rpc/irq.c @@ -14,6 +14,149 @@ #define CLR 0x04 #define MASK 0x08 +static const u8 irq_prio_h[256] = { + 0, 8, 9, 8,10,10,10,10,11,11,11,11,10,10,10,10, + 12, 8, 9, 8,10,10,10,10,11,11,11,11,10,10,10,10, + 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10, + 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10, + 14,14,14,14,10,10,10,10,11,11,11,11,10,10,10,10, + 14,14,14,14,10,10,10,10,11,11,11,11,10,10,10,10, + 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10, + 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10, + 15,15,15,15,10,10,10,10,11,11,11,11,10,10,10,10, + 15,15,15,15,10,10,10,10,11,11,11,11,10,10,10,10, + 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10, + 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10, + 15,15,15,15,10,10,10,10,11,11,11,11,10,10,10,10, + 15,15,15,15,10,10,10,10,11,11,11,11,10,10,10,10, + 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10, + 13,13,13,13,10,10,10,10,11,11,11,11,10,10,10,10, +}; + +static const u8 irq_prio_d[256] = { + 0,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, + 20,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, + 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, + 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, + 22,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, + 22,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, + 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, + 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, + 23,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, + 23,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, + 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, + 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, + 22,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, + 22,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, + 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, + 21,16,17,16,18,16,17,16,19,16,17,16,18,16,17,16, +}; + +static const u8 irq_prio_l[256] = { + 0, 0, 1, 0, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, + 4, 0, 1, 0, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, + 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, + 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, + 6, 6, 6, 6, 6, 6, 6, 6, 3, 3, 3, 3, 3, 3, 3, 3, + 6, 6, 6, 6, 6, 6, 6, 6, 3, 3, 3, 3, 3, 3, 3, 3, + 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, + 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, + 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, + 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, + 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, + 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, + 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, + 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, + 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, + 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, +}; + +static const u8 irq_prio_lc[256] = { + 24,24,25,24,26,26,26,26,27,27,27,27,27,27,27,27, + 28,24,25,24,26,26,26,26,27,27,27,27,27,27,27,27, + 29,29,29,29,29,29,29,29,29,29,29,29,29,29,29,29, + 29,29,29,29,29,29,29,29,29,29,29,29,29,29,29,29, + 30,30,30,30,30,30,30,30,27,27,27,27,27,27,27,27, + 30,30,30,30,30,30,30,30,27,27,27,27,27,27,27,27, + 29,29,29,29,29,29,29,29,29,29,29,29,29,29,29,29, + 29,29,29,29,29,29,29,29,29,29,29,29,29,29,29,29, + 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31, + 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31, + 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31, + 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31, + 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31, + 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31, + 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31, + 31,31,31,31,31,31,31,31,31,31,31,31,31,31,31,31, +}; + +static const u8 irq_prio_ld[256] = { + 40,40,41,40,42,42,42,42,43,43,43,43,43,43,43,43, + 44,40,41,40,42,42,42,42,43,43,43,43,43,43,43,43, + 45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45, + 45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45, + 46,46,46,46,46,46,46,46,43,43,43,43,43,43,43,43, + 46,46,46,46,46,46,46,46,43,43,43,43,43,43,43,43, + 45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45, + 45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45, + 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47, + 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47, + 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47, + 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47, + 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47, + 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47, + 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47, + 47,47,47,47,47,47,47,47,47,47,47,47,47,47,47,47, +}; + +static int iomd_get_irq_nr(void) +{ + int irq; + u8 reg; + + /* get highest priority first */ + reg = readb(IOC_BASE + IOMD_IRQREQB); + irq = irq_prio_h[reg]; + if (irq) + return irq; + + /* get DMA */ + reg = readb(IOC_BASE + IOMD_DMAREQ); + irq = irq_prio_d[reg]; + if (irq) + return irq; + + /* get low priority */ + reg = readb(IOC_BASE + IOMD_IRQREQA); + irq = irq_prio_l[reg]; + if (irq) + return irq; +#ifdef IOMD_IRQREQC + reg = readb(IOC_BASE + IOMD_IRQREQC); + irq = irq_prio_lc[reg]; + if (irq) + return irq; +#endif +#ifdef IOMD_IRQREQD + reg = readb(IOC_BASE + IOMD_IRQREQD); + irq = irq_prio_ld[reg]; + if (irq) + return irq; +#endif + return 0; +} + +static void iomd_handle_irq(struct pt_regs *regs) +{ + int irq; + + do { + irq = iomd_get_irq_nr(); + if (irq) + generic_handle_irq(irq); + } while (irq); +} + static void __iomem *iomd_get_base(struct irq_data *d) { void *cd = irq_data_get_irq_chip_data(d); @@ -82,6 +225,8 @@ void __init rpc_init_irq(void) set_fiq_handler(&rpc_default_fiq_start, &rpc_default_fiq_end - &rpc_default_fiq_start); + set_handle_irq(iomd_handle_irq); + for (irq = 0; irq < NR_IRQS; irq++) { clr = IRQ_NOREQUEST; set = 0; -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 02/12] ARM: footbridge: use GENERIC_IRQ_MULTI_HANDLER 2021-11-30 12:58 [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Ard Biesheuvel 2021-11-30 12:58 ` [PATCH v2 01/12] ARM: riscpc: use GENERIC_IRQ_MULTI_HANDLER Ard Biesheuvel @ 2021-11-30 12:58 ` Ard Biesheuvel 2021-12-04 23:42 ` Linus Walleij 2021-11-30 12:58 ` [PATCH v2 03/12] ARM: iop32x: offset IRQ numbers by 1 Ard Biesheuvel ` (10 subsequent siblings) 12 siblings, 1 reply; 23+ messages in thread From: Ard Biesheuvel @ 2021-11-30 12:58 UTC (permalink / raw) To: linux-arm-kernel, linux Cc: Ard Biesheuvel, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren From: Arnd Bergmann <arnd@arndb.de> Footbridge still uses the classic IRQ entry path in assembler, but this is easily converted into an equivalent C version. In this case, the correlation between IRQ numbers and bits in the status register is non-obvious, and the priorities are handled by manually checking each bit in a static order, re-reading the status register after each handled event. I moved the code into the new file and edited the syntax without changing this sequence to keep the behavior as close as possible to what it traditionally did. Signed-off-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- arch/arm/Kconfig | 1 + arch/arm/mach-footbridge/common.c | 87 ++++++++++++++++ arch/arm/mach-footbridge/include/mach/entry-macro.S | 107 -------------------- 3 files changed, 88 insertions(+), 107 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 25f1868e5703..a0cc9ca66ae0 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -362,6 +362,7 @@ config ARCH_FOOTBRIDGE select FOOTBRIDGE select NEED_MACH_IO_H if !MMU select NEED_MACH_MEMORY_H + select GENERIC_IRQ_MULTI_HANDLER help Support for systems based on the DC21285 companion chip ("FootBridge"), such as the Simtec CATS and the Rebel NetWinder. diff --git a/arch/arm/mach-footbridge/common.c b/arch/arm/mach-footbridge/common.c index eee095f0e2f6..90d9d6fe48a3 100644 --- a/arch/arm/mach-footbridge/common.c +++ b/arch/arm/mach-footbridge/common.c @@ -27,6 +27,91 @@ #include "common.h" +#include <mach/hardware.h> +#include <mach/irqs.h> +#include <asm/hardware/dec21285.h> + +static int dc21285_get_irq(void) +{ + void __iomem *irqstatus = (void __iomem *)CSR_IRQ_STATUS; + u32 mask = readl(irqstatus); + + if (mask & IRQ_MASK_SDRAMPARITY) + return IRQ_SDRAMPARITY; + + if (mask & IRQ_MASK_UART_RX) + return IRQ_CONRX; + + if (mask & IRQ_MASK_DMA1) + return IRQ_DMA1; + + if (mask & IRQ_MASK_DMA2) + return IRQ_DMA2; + + if (mask & IRQ_MASK_IN0) + return IRQ_IN0; + + if (mask & IRQ_MASK_IN1) + return IRQ_IN1; + + if (mask & IRQ_MASK_IN2) + return IRQ_IN2; + + if (mask & IRQ_MASK_IN3) + return IRQ_IN3; + + if (mask & IRQ_MASK_PCI) + return IRQ_PCI; + + if (mask & IRQ_MASK_DOORBELLHOST) + return IRQ_DOORBELLHOST; + + if (mask & IRQ_MASK_I2OINPOST) + return IRQ_I2OINPOST; + + if (mask & IRQ_MASK_TIMER1) + return IRQ_TIMER1; + + if (mask & IRQ_MASK_TIMER2) + return IRQ_TIMER2; + + if (mask & IRQ_MASK_TIMER3) + return IRQ_TIMER3; + + if (mask & IRQ_MASK_UART_TX) + return IRQ_CONTX; + + if (mask & IRQ_MASK_PCI_ABORT) + return IRQ_PCI_ABORT; + + if (mask & IRQ_MASK_PCI_SERR) + return IRQ_PCI_SERR; + + if (mask & IRQ_MASK_DISCARD_TIMER) + return IRQ_DISCARD_TIMER; + + if (mask & IRQ_MASK_PCI_DPERR) + return IRQ_PCI_DPERR; + + if (mask & IRQ_MASK_PCI_PERR) + return IRQ_PCI_PERR; + + return 0; +} + +void dc21285_handle_irq(struct pt_regs *regs) +{ + int irq; + do { + irq = dc21285_get_irq(); + if (!irq) + break; + + generic_handle_irq(irq); + } while (1); +} + + unsigned int mem_fclk_21285 = 50000000; EXPORT_SYMBOL(mem_fclk_21285); @@ -108,6 +193,8 @@ static void __init __fb_init_irq(void) void __init footbridge_init_irq(void) { + set_handle_irq(dc21285_handle_irq); + __fb_init_irq(); if (!footbridge_cfn_mode()) diff --git a/arch/arm/mach-footbridge/include/mach/entry-macro.S b/arch/arm/mach-footbridge/include/mach/entry-macro.S deleted file mode 100644 index dabbd5c54a78..000000000000 --- a/arch/arm/mach-footbridge/include/mach/entry-macro.S +++ /dev/null @@ -1,107 +0,0 @@ -/* - * arch/arm/mach-footbridge/include/mach/entry-macro.S - * - * Low-level IRQ helper macros for footbridge-based platforms - * - * This file is licensed under the terms of the GNU General Public - * License version 2. This program is licensed "as is" without any - * warranty of any kind, whether express or implied. - */ -#include <mach/hardware.h> -#include <mach/irqs.h> -#include <asm/hardware/dec21285.h> - - .equ dc21285_high, ARMCSR_BASE & 0xff000000 - .equ dc21285_low, ARMCSR_BASE & 0x00ffffff - - .macro get_irqnr_preamble, base, tmp - mov \base, #dc21285_high - .if dc21285_low - orr \base, \base, #dc21285_low - .endif - .endm - - .macro get_irqnr_and_base, irqnr, irqstat, base, tmp - ldr \irqstat, [\base, #0x180] @ get interrupts - - mov \irqnr, #IRQ_SDRAMPARITY - tst \irqstat, #IRQ_MASK_SDRAMPARITY - bne 1001f - - tst \irqstat, #IRQ_MASK_UART_RX - movne \irqnr, #IRQ_CONRX - bne 1001f - - tst \irqstat, #IRQ_MASK_DMA1 - movne \irqnr, #IRQ_DMA1 - bne 1001f - - tst \irqstat, #IRQ_MASK_DMA2 - movne \irqnr, #IRQ_DMA2 - bne 1001f - - tst \irqstat, #IRQ_MASK_IN0 - movne \irqnr, #IRQ_IN0 - bne 1001f - - tst \irqstat, #IRQ_MASK_IN1 - movne \irqnr, #IRQ_IN1 - bne 1001f - - tst \irqstat, #IRQ_MASK_IN2 - movne \irqnr, #IRQ_IN2 - bne 1001f - - tst \irqstat, #IRQ_MASK_IN3 - movne \irqnr, #IRQ_IN3 - bne 1001f - - tst \irqstat, #IRQ_MASK_PCI - movne \irqnr, #IRQ_PCI - bne 1001f - - tst \irqstat, #IRQ_MASK_DOORBELLHOST - movne \irqnr, #IRQ_DOORBELLHOST - bne 1001f - - tst \irqstat, #IRQ_MASK_I2OINPOST - movne \irqnr, #IRQ_I2OINPOST - bne 1001f - - tst \irqstat, #IRQ_MASK_TIMER1 - movne \irqnr, #IRQ_TIMER1 - bne 1001f - - tst \irqstat, #IRQ_MASK_TIMER2 - movne \irqnr, #IRQ_TIMER2 - bne 1001f - - tst \irqstat, #IRQ_MASK_TIMER3 - movne \irqnr, #IRQ_TIMER3 - bne 1001f - - tst \irqstat, #IRQ_MASK_UART_TX - movne \irqnr, #IRQ_CONTX - bne 1001f - - tst \irqstat, #IRQ_MASK_PCI_ABORT - movne \irqnr, #IRQ_PCI_ABORT - bne 1001f - - tst \irqstat, #IRQ_MASK_PCI_SERR - movne \irqnr, #IRQ_PCI_SERR - bne 1001f - - tst \irqstat, #IRQ_MASK_DISCARD_TIMER - movne \irqnr, #IRQ_DISCARD_TIMER - bne 1001f - - tst \irqstat, #IRQ_MASK_PCI_DPERR - movne \irqnr, #IRQ_PCI_DPERR - bne 1001f - - tst \irqstat, #IRQ_MASK_PCI_PERR - movne \irqnr, #IRQ_PCI_PERR -1001: - .endm - -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH v2 02/12] ARM: footbridge: use GENERIC_IRQ_MULTI_HANDLER 2021-11-30 12:58 ` [PATCH v2 02/12] ARM: footbridge: " Ard Biesheuvel @ 2021-12-04 23:42 ` Linus Walleij 0 siblings, 0 replies; 23+ messages in thread From: Linus Walleij @ 2021-12-04 23:42 UTC (permalink / raw) To: Ard Biesheuvel Cc: linux-arm-kernel, linux, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Nick Desaulniers, Tony Lindgren On Tue, Nov 30, 2021 at 1:59 PM Ard Biesheuvel <ardb@kernel.org> wrote: > From: Arnd Bergmann <arnd@arndb.de> > > Footbridge still uses the classic IRQ entry path in assembler, > but this is easily converted into an equivalent C version. > > In this case, the correlation between IRQ numbers and bits in > the status register is non-obvious, and the priorities are > handled by manually checking each bit in a static order, > re-reading the status register after each handled event. > > I moved the code into the new file and edited the syntax without > changing this sequence to keep the behavior as close as possible > to what it traditionally did. > > Signed-off-by: Arnd Bergmann <arnd@arndb.de> > Signed-off-by: Ard Biesheuvel <ardb@kernel.org> This generally looks good to me, if I had time to go into my office I could also test it but not right now because of external factors. Reviewed-by: Linus Walleij <linus.walleij@linaro.org> Yours, Linus Walleij _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 03/12] ARM: iop32x: offset IRQ numbers by 1 2021-11-30 12:58 [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Ard Biesheuvel 2021-11-30 12:58 ` [PATCH v2 01/12] ARM: riscpc: use GENERIC_IRQ_MULTI_HANDLER Ard Biesheuvel 2021-11-30 12:58 ` [PATCH v2 02/12] ARM: footbridge: " Ard Biesheuvel @ 2021-11-30 12:58 ` Ard Biesheuvel 2021-12-04 23:43 ` Linus Walleij 2021-11-30 12:58 ` [PATCH v2 04/12] ARM: iop32x: use GENERIC_IRQ_MULTI_HANDLER Ard Biesheuvel ` (9 subsequent siblings) 12 siblings, 1 reply; 23+ messages in thread From: Ard Biesheuvel @ 2021-11-30 12:58 UTC (permalink / raw) To: linux-arm-kernel, linux Cc: Ard Biesheuvel, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren From: Arnd Bergmann <arnd@arndb.de> iop32x is one of the last platforms to use IRQ 0, and this has apparently stopped working in a 2014 cleanup without anyone noticing. This interrupt is used for the DMA engine, so most likely this has not actually worked in the past 7 years, but it's also not essential for using this board. I'm splitting out this change from my GENERIC_IRQ_MULTI_HANDLER conversion so it can be backported if anyone cares. Fixes: a71b092a9c68 ("ARM: Convert handle_IRQ to use __handle_domain_irq") Signed-off-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- arch/arm/mach-iop32x/include/mach/entry-macro.S | 2 +- arch/arm/mach-iop32x/include/mach/irqs.h | 2 +- arch/arm/mach-iop32x/irqs.h | 60 +++++++++++--------- 3 files changed, 34 insertions(+), 30 deletions(-) diff --git a/arch/arm/mach-iop32x/include/mach/entry-macro.S b/arch/arm/mach-iop32x/include/mach/entry-macro.S index 8e6766d4621e..341e5d9a6616 100644 --- a/arch/arm/mach-iop32x/include/mach/entry-macro.S +++ b/arch/arm/mach-iop32x/include/mach/entry-macro.S @@ -20,7 +20,7 @@ mrc p6, 0, \irqstat, c8, c0, 0 @ Read IINTSRC cmp \irqstat, #0 clzne \irqnr, \irqstat - rsbne \irqnr, \irqnr, #31 + rsbne \irqnr, \irqnr, #32 .endm .macro arch_ret_to_user, tmp1, tmp2 diff --git a/arch/arm/mach-iop32x/include/mach/irqs.h b/arch/arm/mach-iop32x/include/mach/irqs.h index c4e78df428e8..e09ae5f48aec 100644 --- a/arch/arm/mach-iop32x/include/mach/irqs.h +++ b/arch/arm/mach-iop32x/include/mach/irqs.h @@ -9,6 +9,6 @@ #ifndef __IRQS_H #define __IRQS_H -#define NR_IRQS 32 +#define NR_IRQS 33 #endif diff --git a/arch/arm/mach-iop32x/irqs.h b/arch/arm/mach-iop32x/irqs.h index 69858e4e905d..e1dfc8b4e7d7 100644 --- a/arch/arm/mach-iop32x/irqs.h +++ b/arch/arm/mach-iop32x/irqs.h @@ -7,36 +7,40 @@ #ifndef __IOP32X_IRQS_H #define __IOP32X_IRQS_H +/* Interrupts in Linux start at 1, hardware starts at 0 */ + +#define IOP_IRQ(x) ((x) + 1) + /* * IOP80321 chipset interrupts */ -#define IRQ_IOP32X_DMA0_EOT 0 -#define IRQ_IOP32X_DMA0_EOC 1 -#define IRQ_IOP32X_DMA1_EOT 2 -#define IRQ_IOP32X_DMA1_EOC 3 -#define IRQ_IOP32X_AA_EOT 6 -#define IRQ_IOP32X_AA_EOC 7 -#define IRQ_IOP32X_CORE_PMON 8 -#define IRQ_IOP32X_TIMER0 9 -#define IRQ_IOP32X_TIMER1 10 -#define IRQ_IOP32X_I2C_0 11 -#define IRQ_IOP32X_I2C_1 12 -#define IRQ_IOP32X_MESSAGING 13 -#define IRQ_IOP32X_ATU_BIST 14 -#define IRQ_IOP32X_PERFMON 15 -#define IRQ_IOP32X_CORE_PMU 16 -#define IRQ_IOP32X_BIU_ERR 17 -#define IRQ_IOP32X_ATU_ERR 18 -#define IRQ_IOP32X_MCU_ERR 19 -#define IRQ_IOP32X_DMA0_ERR 20 -#define IRQ_IOP32X_DMA1_ERR 21 -#define IRQ_IOP32X_AA_ERR 23 -#define IRQ_IOP32X_MSG_ERR 24 -#define IRQ_IOP32X_SSP 25 -#define IRQ_IOP32X_XINT0 27 -#define IRQ_IOP32X_XINT1 28 -#define IRQ_IOP32X_XINT2 29 -#define IRQ_IOP32X_XINT3 30 -#define IRQ_IOP32X_HPI 31 +#define IRQ_IOP32X_DMA0_EOT IOP_IRQ(0) +#define IRQ_IOP32X_DMA0_EOC IOP_IRQ(1) +#define IRQ_IOP32X_DMA1_EOT IOP_IRQ(2) +#define IRQ_IOP32X_DMA1_EOC IOP_IRQ(3) +#define IRQ_IOP32X_AA_EOT IOP_IRQ(6) +#define IRQ_IOP32X_AA_EOC IOP_IRQ(7) +#define IRQ_IOP32X_CORE_PMON IOP_IRQ(8) +#define IRQ_IOP32X_TIMER0 IOP_IRQ(9) +#define IRQ_IOP32X_TIMER1 IOP_IRQ(10) +#define IRQ_IOP32X_I2C_0 IOP_IRQ(11) +#define IRQ_IOP32X_I2C_1 IOP_IRQ(12) +#define IRQ_IOP32X_MESSAGING IOP_IRQ(13) +#define IRQ_IOP32X_ATU_BIST IOP_IRQ(14) +#define IRQ_IOP32X_PERFMON IOP_IRQ(15) +#define IRQ_IOP32X_CORE_PMU IOP_IRQ(16) +#define IRQ_IOP32X_BIU_ERR IOP_IRQ(17) +#define IRQ_IOP32X_ATU_ERR IOP_IRQ(18) +#define IRQ_IOP32X_MCU_ERR IOP_IRQ(19) +#define IRQ_IOP32X_DMA0_ERR IOP_IRQ(20) +#define IRQ_IOP32X_DMA1_ERR IOP_IRQ(21) +#define IRQ_IOP32X_AA_ERR IOP_IRQ(23) +#define IRQ_IOP32X_MSG_ERR IOP_IRQ(24) +#define IRQ_IOP32X_SSP IOP_IRQ(25) +#define IRQ_IOP32X_XINT0 IOP_IRQ(27) +#define IRQ_IOP32X_XINT1 IOP_IRQ(28) +#define IRQ_IOP32X_XINT2 IOP_IRQ(29) +#define IRQ_IOP32X_XINT3 IOP_IRQ(30) +#define IRQ_IOP32X_HPI IOP_IRQ(31) #endif -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH v2 03/12] ARM: iop32x: offset IRQ numbers by 1 2021-11-30 12:58 ` [PATCH v2 03/12] ARM: iop32x: offset IRQ numbers by 1 Ard Biesheuvel @ 2021-12-04 23:43 ` Linus Walleij 0 siblings, 0 replies; 23+ messages in thread From: Linus Walleij @ 2021-12-04 23:43 UTC (permalink / raw) To: Ard Biesheuvel Cc: linux-arm-kernel, linux, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Nick Desaulniers, Tony Lindgren On Tue, Nov 30, 2021 at 1:59 PM Ard Biesheuvel <ardb@kernel.org> wrote: > From: Arnd Bergmann <arnd@arndb.de> > > iop32x is one of the last platforms to use IRQ 0, and this has apparently > stopped working in a 2014 cleanup without anyone noticing. This interrupt > is used for the DMA engine, so most likely this has not actually worked > in the past 7 years, but it's also not essential for using this board. > > I'm splitting out this change from my GENERIC_IRQ_MULTI_HANDLER > conversion so it can be backported if anyone cares. > > Fixes: a71b092a9c68 ("ARM: Convert handle_IRQ to use __handle_domain_irq") > Signed-off-by: Arnd Bergmann <arnd@arndb.de> > Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Good riddance. Reviewed-by: Linus Walleij <linus.walleij@linaro.org> Yours, Linus Walleij _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 04/12] ARM: iop32x: use GENERIC_IRQ_MULTI_HANDLER 2021-11-30 12:58 [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Ard Biesheuvel ` (2 preceding siblings ...) 2021-11-30 12:58 ` [PATCH v2 03/12] ARM: iop32x: offset IRQ numbers by 1 Ard Biesheuvel @ 2021-11-30 12:58 ` Ard Biesheuvel 2021-11-30 13:29 ` Arnd Bergmann 2021-11-30 12:58 ` [PATCH v2 05/12] ARM: remove old-style irq entry Ard Biesheuvel ` (8 subsequent siblings) 12 siblings, 1 reply; 23+ messages in thread From: Ard Biesheuvel @ 2021-11-30 12:58 UTC (permalink / raw) To: linux-arm-kernel, linux Cc: Ard Biesheuvel, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren From: Arnd Bergmann <arnd@arndb.de> iop32x uses the entry-macro.S file for both the IRQ entry and for hooking into the arch_ret_to_user code path. This is done because the cp6 registers have to be enabled before accessing any of the interrupt controller registers but have to be disabled when running in user space. There is also a lazy-enable logic in cp6.c, but during a hardirq, we know it has to be enabled. Both the cp6-enable code and the code to read the IRQ status can be lifted into the normal generic_handle_arch_irq() path, but the cp6-disable code has to remain in the user return code. As nothing other than iop32x uses this hook, just open-code it there with an ifdef for the platform that can eventually be removed when iop32x has reached the end of its life. The cp6-enable path in the IRQ entry has an extra cp_wait barrier that the trap version does not have, but it is harmless to do it in both cases to simplify the logic here at the cost of a few extra cycles for the trap. I'm leaving the logic completely unchanged, including the way this platform starts its IRQs at zero, but add a note that this can cause problems. Signed-off-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- arch/arm/Kconfig | 5 +--- arch/arm/kernel/entry-common.S | 10 ++++--- arch/arm/mach-iop32x/cp6.c | 10 ++++++- arch/arm/mach-iop32x/include/mach/entry-macro.S | 31 -------------------- arch/arm/mach-iop32x/iop3xx.h | 1 + arch/arm/mach-iop32x/irq.c | 23 +++++++++++++++ 6 files changed, 40 insertions(+), 40 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index a0cc9ca66ae0..d9ba6961b295 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -227,9 +227,6 @@ config GENERIC_ISA_DMA config FIQ bool -config NEED_RET_TO_USER - bool - config ARCH_MTD_XIP bool @@ -371,9 +368,9 @@ config ARCH_IOP32X bool "IOP32x-based" depends on MMU select CPU_XSCALE + select GENERIC_IRQ_MULTI_HANDLER select GPIO_IOP select GPIOLIB - select NEED_RET_TO_USER select FORCE_PCI select PLAT_IOP help diff --git a/arch/arm/kernel/entry-common.S b/arch/arm/kernel/entry-common.S index ac86c34682bb..67a89e598f9e 100644 --- a/arch/arm/kernel/entry-common.S +++ b/arch/arm/kernel/entry-common.S @@ -16,12 +16,14 @@ .equ NR_syscalls, __NR_syscalls -#ifdef CONFIG_NEED_RET_TO_USER -#include <mach/entry-macro.S> -#else .macro arch_ret_to_user, tmp1, tmp2 - .endm +#ifdef CONFIG_ARCH_IOP32X + mrc p15, 0, \tmp1, c15, c1, 0 + ands \tmp2, \tmp1, #(1 << 6) + bicne \tmp1, \tmp1, #(1 << 6) + mcrne p15, 0, \tmp1, c15, c1, 0 @ Disable cp6 access #endif + .endm #include "entry-header.S" diff --git a/arch/arm/mach-iop32x/cp6.c b/arch/arm/mach-iop32x/cp6.c index ec74b07fb7e3..095f84673ccc 100644 --- a/arch/arm/mach-iop32x/cp6.c +++ b/arch/arm/mach-iop32x/cp6.c @@ -7,7 +7,7 @@ #include <asm/traps.h> #include <asm/ptrace.h> -static int cp6_trap(struct pt_regs *regs, unsigned int instr) +void iop_enable_cp6(void) { u32 temp; @@ -16,7 +16,15 @@ static int cp6_trap(struct pt_regs *regs, unsigned int instr) "mrc p15, 0, %0, c15, c1, 0\n\t" "orr %0, %0, #(1 << 6)\n\t" "mcr p15, 0, %0, c15, c1, 0\n\t" + "mrc p15, 0, %0, c15, c1, 0\n\t" + "mov %0, %0\n\t" + "sub pc, pc, #4 @ cp_wait\n\t" : "=r"(temp)); +} + +static int cp6_trap(struct pt_regs *regs, unsigned int instr) +{ + iop_enable_cp6(); return 0; } diff --git a/arch/arm/mach-iop32x/include/mach/entry-macro.S b/arch/arm/mach-iop32x/include/mach/entry-macro.S deleted file mode 100644 index 341e5d9a6616..000000000000 --- a/arch/arm/mach-iop32x/include/mach/entry-macro.S +++ /dev/null @@ -1,31 +0,0 @@ -/* - * arch/arm/mach-iop32x/include/mach/entry-macro.S - * - * Low-level IRQ helper macros for IOP32x-based platforms - * - * This file is licensed under the terms of the GNU General Public - * License version 2. This program is licensed "as is" without any - * warranty of any kind, whether express or implied. - */ - .macro get_irqnr_preamble, base, tmp - mrc p15, 0, \tmp, c15, c1, 0 - orr \tmp, \tmp, #(1 << 6) - mcr p15, 0, \tmp, c15, c1, 0 @ Enable cp6 access - mrc p15, 0, \tmp, c15, c1, 0 - mov \tmp, \tmp - sub pc, pc, #4 @ cp_wait - .endm - - .macro get_irqnr_and_base, irqnr, irqstat, base, tmp - mrc p6, 0, \irqstat, c8, c0, 0 @ Read IINTSRC - cmp \irqstat, #0 - clzne \irqnr, \irqstat - rsbne \irqnr, \irqnr, #32 - .endm - - .macro arch_ret_to_user, tmp1, tmp2 - mrc p15, 0, \tmp1, c15, c1, 0 - ands \tmp2, \tmp1, #(1 << 6) - bicne \tmp1, \tmp1, #(1 << 6) - mcrne p15, 0, \tmp1, c15, c1, 0 @ Disable cp6 access - .endm diff --git a/arch/arm/mach-iop32x/iop3xx.h b/arch/arm/mach-iop32x/iop3xx.h index 46b4b34a4ad2..a6ec7ebadb35 100644 --- a/arch/arm/mach-iop32x/iop3xx.h +++ b/arch/arm/mach-iop32x/iop3xx.h @@ -225,6 +225,7 @@ extern int iop3xx_get_init_atu(void); #include <linux/reboot.h> void iop3xx_map_io(void); +void iop_enable_cp6(void); void iop_init_cp6_handler(void); void iop_init_time(unsigned long tickrate); void iop3xx_restart(enum reboot_mode, const char *); diff --git a/arch/arm/mach-iop32x/irq.c b/arch/arm/mach-iop32x/irq.c index 2d48bf1398c1..9d6f7ed45ada 100644 --- a/arch/arm/mach-iop32x/irq.c +++ b/arch/arm/mach-iop32x/irq.c @@ -29,6 +29,15 @@ static void intstr_write(u32 val) asm volatile("mcr p6, 0, %0, c4, c0, 0" : : "r" (val)); } +static u32 iintsrc_read(void) +{ + int irq; + + asm volatile("mrc p6, 0, %0, c8, c0, 0" : "=r" (irq)); + + return irq; +} + static void iop32x_irq_mask(struct irq_data *d) { @@ -50,11 +59,25 @@ struct irq_chip ext_chip = { .irq_unmask = iop32x_irq_unmask, }; +void iop_handle_irq(struct pt_regs *regs) +{ + u32 mask; + + iop_enable_cp6(); + + do { + mask = iintsrc_read(); + if (mask) + generic_handle_irq(fls(mask)); + } while (mask); +} + void __init iop32x_init_irq(void) { int i; iop_init_cp6_handler(); + set_handle_irq(iop_handle_irq); intctl_write(0); intstr_write(0); -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH v2 04/12] ARM: iop32x: use GENERIC_IRQ_MULTI_HANDLER 2021-11-30 12:58 ` [PATCH v2 04/12] ARM: iop32x: use GENERIC_IRQ_MULTI_HANDLER Ard Biesheuvel @ 2021-11-30 13:29 ` Arnd Bergmann 0 siblings, 0 replies; 23+ messages in thread From: Arnd Bergmann @ 2021-11-30 13:29 UTC (permalink / raw) To: Ard Biesheuvel Cc: Linux ARM, Russell King - ARM Linux, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren On Tue, Nov 30, 2021 at 1:58 PM Ard Biesheuvel <ardb@kernel.org> wrote: > > From: Arnd Bergmann <arnd@arndb.de> > The cp6-enable path in the IRQ entry has an extra cp_wait barrier that > the trap version does not have, but it is harmless to do it in both > cases to simplify the logic here at the cost of a few extra cycles > for the trap. > > I'm leaving the logic completely unchanged, including the way this > platform starts its IRQs at zero, but add a note that this can > cause problems. (Replying to my own patch here) The last paragraph is now obsolete, please remove it. Arnd _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 05/12] ARM: remove old-style irq entry 2021-11-30 12:58 [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Ard Biesheuvel ` (3 preceding siblings ...) 2021-11-30 12:58 ` [PATCH v2 04/12] ARM: iop32x: use GENERIC_IRQ_MULTI_HANDLER Ard Biesheuvel @ 2021-11-30 12:58 ` Ard Biesheuvel 2021-12-04 23:45 ` Linus Walleij 2021-11-30 12:58 ` [PATCH v2 06/12] ARM: entry: preserve thread_info pointer in switch_to Ard Biesheuvel ` (7 subsequent siblings) 12 siblings, 1 reply; 23+ messages in thread From: Ard Biesheuvel @ 2021-11-30 12:58 UTC (permalink / raw) To: linux-arm-kernel, linux Cc: Ard Biesheuvel, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren From: Arnd Bergmann <arnd@arndb.de> The last user of arch_irq_handler_default is gone now, so the entry-macro-multi.S file and all references to mach/entry-macro.S can be removed, as well as the asm_do_IRQ() entrypoint into the interrupt handling routines implemented in C. Note: The ARMv7-M entry still uses its own top-level IRQ entry, calling nvic_handle_irq() from assembly. This could be changed to go through generic_handle_arch_irq() as well, but it's unclear to me if there are any benefits. Signed-off-by: Arnd Bergmann <arnd@arndb.de> [ardb: keep irq_handler macro as it carries all the IRQ stack handling] Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- arch/arm/Kconfig | 14 ++------------ arch/arm/include/asm/entry-macro-multi.S | 16 ---------------- arch/arm/include/asm/irq.h | 1 - arch/arm/include/asm/mach/arch.h | 2 -- arch/arm/kernel/entry-armv.S | 8 -------- arch/arm/kernel/irq.c | 17 ----------------- 6 files changed, 2 insertions(+), 56 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index d9ba6961b295..b0e403076227 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -58,6 +58,7 @@ config ARM select GENERIC_CPU_AUTOPROBE select GENERIC_EARLY_IOREMAP select GENERIC_IDLE_POLL_SETUP + select GENERIC_IRQ_MULTI_HANDLER if MMU select GENERIC_IRQ_PROBE select GENERIC_IRQ_SHOW select GENERIC_IRQ_SHOW_LEVEL @@ -320,7 +321,6 @@ config ARCH_MULTIPLATFORM select AUTO_ZRELADDR select TIMER_OF select COMMON_CLK - select GENERIC_IRQ_MULTI_HANDLER select HAVE_PCI select PCI_DOMAINS_GENERIC if PCI select SPARSE_IRQ @@ -344,7 +344,6 @@ config ARCH_EP93XX select ARM_AMBA imply ARM_PATCH_PHYS_VIRT select ARM_VIC - select GENERIC_IRQ_MULTI_HANDLER select AUTO_ZRELADDR select CLKSRC_MMIO select CPU_ARM920T @@ -359,7 +358,6 @@ config ARCH_FOOTBRIDGE select FOOTBRIDGE select NEED_MACH_IO_H if !MMU select NEED_MACH_MEMORY_H - select GENERIC_IRQ_MULTI_HANDLER help Support for systems based on the DC21285 companion chip ("FootBridge"), such as the Simtec CATS and the Rebel NetWinder. @@ -368,7 +366,6 @@ config ARCH_IOP32X bool "IOP32x-based" depends on MMU select CPU_XSCALE - select GENERIC_IRQ_MULTI_HANDLER select GPIO_IOP select GPIOLIB select FORCE_PCI @@ -384,7 +381,6 @@ config ARCH_IXP4XX select ARCH_SUPPORTS_BIG_ENDIAN select CPU_XSCALE select DMABOUNCE if PCI - select GENERIC_IRQ_MULTI_HANDLER select GPIO_IXP4XX select GPIOLIB select HAVE_PCI @@ -400,7 +396,6 @@ config ARCH_IXP4XX config ARCH_DOVE bool "Marvell Dove" select CPU_PJ4 - select GENERIC_IRQ_MULTI_HANDLER select GPIOLIB select HAVE_PCI select MVEBU_MBUS @@ -423,7 +418,6 @@ config ARCH_PXA select CLKSRC_MMIO select TIMER_OF select CPU_XSCALE if !CPU_XSC3 - select GENERIC_IRQ_MULTI_HANDLER select GPIO_PXA select GPIOLIB select IRQ_DOMAIN @@ -442,7 +436,6 @@ config ARCH_RPC select ARM_HAS_SG_CHAIN select CPU_SA110 select FIQ - select GENERIC_IRQ_MULTI_HANDLER select HAVE_PATA_PLATFORM select ISA_DMA_API select LEGACY_TIMER_TICK @@ -463,7 +456,6 @@ config ARCH_SA1100 select COMMON_CLK select CPU_FREQ select CPU_SA1100 - select GENERIC_IRQ_MULTI_HANDLER select GPIOLIB select IRQ_DOMAIN select ISA @@ -478,7 +470,6 @@ config ARCH_S3C24XX select CLKSRC_SAMSUNG_PWM select GPIO_SAMSUNG select GPIOLIB - select GENERIC_IRQ_MULTI_HANDLER select HAVE_S3C2410_I2C if I2C select NEED_MACH_IO_H select S3C2410_WATCHDOG @@ -497,7 +488,6 @@ config ARCH_OMAP1 select ARCH_OMAP select CLKSRC_MMIO select GENERIC_IRQ_CHIP - select GENERIC_IRQ_MULTI_HANDLER select GPIOLIB select HAVE_LEGACY_CLK select IRQ_DOMAIN @@ -1168,7 +1158,7 @@ config CURRENT_POINTER_IN_TPIDRURO config IRQSTACKS def_bool y - depends on GENERIC_IRQ_MULTI_HANDLER && THREAD_INFO_IN_TASK + depends on THREAD_INFO_IN_TASK select HAVE_IRQ_EXIT_ON_IRQ_STACK select HAVE_SOFTIRQ_ON_OWN_STACK diff --git a/arch/arm/include/asm/entry-macro-multi.S b/arch/arm/include/asm/entry-macro-multi.S deleted file mode 100644 index 24486dad9e19..000000000000 --- a/arch/arm/include/asm/entry-macro-multi.S +++ /dev/null @@ -1,16 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#include <asm/assembler.h> - -/* - * Interrupt handling. Preserves r7, r8, r9 - */ - .macro arch_irq_handler_default - get_irqnr_preamble r6, lr -1: get_irqnr_and_base r0, r2, r6, lr - movne r1, sp - @ - @ routine called with r0 = irq number, r1 = struct pt_regs * - @ - badrne lr, 1b - bne asm_do_IRQ - .endm diff --git a/arch/arm/include/asm/irq.h b/arch/arm/include/asm/irq.h index 1cbcc462b07e..a7c2337b0c7d 100644 --- a/arch/arm/include/asm/irq.h +++ b/arch/arm/include/asm/irq.h @@ -26,7 +26,6 @@ struct irqaction; struct pt_regs; -extern void asm_do_IRQ(unsigned int, struct pt_regs *); void handle_IRQ(unsigned int, struct pt_regs *); void init_IRQ(void); diff --git a/arch/arm/include/asm/mach/arch.h b/arch/arm/include/asm/mach/arch.h index eec0c0bda766..9349e7a82c9c 100644 --- a/arch/arm/include/asm/mach/arch.h +++ b/arch/arm/include/asm/mach/arch.h @@ -56,9 +56,7 @@ struct machine_desc { void (*init_time)(void); void (*init_machine)(void); void (*init_late)(void); -#ifdef CONFIG_GENERIC_IRQ_MULTI_HANDLER void (*handle_irq)(struct pt_regs *); -#endif void (*restart)(enum reboot_mode, const char *); }; diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 5fb7465d14d9..9744d087ee9f 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -19,9 +19,6 @@ #include <asm/glue-df.h> #include <asm/glue-pf.h> #include <asm/vfpmacros.h> -#ifndef CONFIG_GENERIC_IRQ_MULTI_HANDLER -#include <mach/entry-macro.S> -#endif #include <asm/thread_notify.h> #include <asm/unwind.h> #include <asm/unistd.h> @@ -30,14 +27,12 @@ #include <asm/uaccess-asm.h> #include "entry-header.S" -#include <asm/entry-macro-multi.S> #include <asm/probes.h> /* * Interrupt handling. */ .macro irq_handler, from_user:req -#ifdef CONFIG_GENERIC_IRQ_MULTI_HANDLER mov r0, sp #ifdef CONFIG_IRQSTACKS mov_l r2, irq_stack_ptr @ Take base address @@ -92,9 +87,6 @@ UNWIND( .setfp fpreg, sp ) mov sp, r9 @ Restore original SP #endif // CONFIG_UNWINDER_ARM #endif // CONFIG_IRQSTACKS -#else - arch_irq_handler_default -#endif .endm .macro pabt_helper diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c index 5deb40f39999..5c6f8d11a3ce 100644 --- a/arch/arm/kernel/irq.c +++ b/arch/arm/kernel/irq.c @@ -120,23 +120,6 @@ void handle_IRQ(unsigned int irq, struct pt_regs *regs) ack_bad_irq(irq); } -/* - * asm_do_IRQ is the interface to be used from assembly code. - */ -asmlinkage void __exception_irq_entry -asm_do_IRQ(unsigned int irq, struct pt_regs *regs) -{ - struct pt_regs *old_regs; - - irq_enter(); - old_regs = set_irq_regs(regs); - - handle_IRQ(irq, regs); - - set_irq_regs(old_regs); - irq_exit(); -} - void __init init_IRQ(void) { int ret; -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH v2 05/12] ARM: remove old-style irq entry 2021-11-30 12:58 ` [PATCH v2 05/12] ARM: remove old-style irq entry Ard Biesheuvel @ 2021-12-04 23:45 ` Linus Walleij 0 siblings, 0 replies; 23+ messages in thread From: Linus Walleij @ 2021-12-04 23:45 UTC (permalink / raw) To: Ard Biesheuvel Cc: linux-arm-kernel, linux, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Nick Desaulniers, Tony Lindgren On Tue, Nov 30, 2021 at 1:59 PM Ard Biesheuvel <ardb@kernel.org> wrote: > From: Arnd Bergmann <arnd@arndb.de> > > The last user of arch_irq_handler_default is gone now, so the > entry-macro-multi.S file and all references to mach/entry-macro.S can > be removed, as well as the asm_do_IRQ() entrypoint into the interrupt > handling routines implemented in C. > > Note: The ARMv7-M entry still uses its own top-level IRQ entry, calling > nvic_handle_irq() from assembly. This could be changed to go through > generic_handle_arch_irq() as well, but it's unclear to me if there are > any benefits. > > Signed-off-by: Arnd Bergmann <arnd@arndb.de> > [ardb: keep irq_handler macro as it carries all the IRQ stack handling] > Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Reviewed-by: Linus Walleij <linus.walleij@linaro.org> Yours, Linus Walleij _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 06/12] ARM: entry: preserve thread_info pointer in switch_to 2021-11-30 12:58 [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Ard Biesheuvel ` (4 preceding siblings ...) 2021-11-30 12:58 ` [PATCH v2 05/12] ARM: remove old-style irq entry Ard Biesheuvel @ 2021-11-30 12:58 ` Ard Biesheuvel 2021-11-30 12:58 ` [PATCH v2 07/12] ARM: module: implement support for PC-relative group relocations Ard Biesheuvel ` (6 subsequent siblings) 12 siblings, 0 replies; 23+ messages in thread From: Ard Biesheuvel @ 2021-11-30 12:58 UTC (permalink / raw) To: linux-arm-kernel, linux Cc: Ard Biesheuvel, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren Tweak the UP stack protector handling code so that the thread info pointer is preserved in R7 until set_current is called. This is needed for a subsequent patch that implements THREAD_INFO_IN_TASK and set_current for UP as well. This also means we will prefer the per-task protector on UP systems that implement the thread ID registers, so tweak the preprocessor conditionals to reflect this. Acked-by: Linus Walleij <linus.walleij@linaro.org> Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- arch/arm/kernel/entry-armv.S | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 9744d087ee9f..1a6cf711a3b4 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -806,16 +806,16 @@ ENTRY(__switch_to) ldr r6, [r2, #TI_CPU_DOMAIN] #endif switch_tls r1, r4, r5, r3, r7 -#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) - ldr r7, [r2, #TI_TASK] +#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) && \ + !defined(CONFIG_STACKPROTECTOR_PER_TASK) + ldr r9, [r2, #TI_TASK] ldr r8, =__stack_chk_guard .if (TSK_STACK_CANARY > IMM12_MASK) - add r7, r7, #TSK_STACK_CANARY & ~IMM12_MASK + add r9, r9, #TSK_STACK_CANARY & ~IMM12_MASK .endif - ldr r7, [r7, #TSK_STACK_CANARY & IMM12_MASK] -#elif defined(CONFIG_CURRENT_POINTER_IN_TPIDRURO) - mov r7, r2 @ Preserve 'next' + ldr r9, [r9, #TSK_STACK_CANARY & IMM12_MASK] #endif + mov r7, r2 @ Preserve 'next' #ifdef CONFIG_CPU_USE_DOMAINS mcr p15, 0, r6, c3, c0, 0 @ Set domain register #endif @@ -824,8 +824,9 @@ ENTRY(__switch_to) ldr r0, =thread_notify_head mov r1, #THREAD_NOTIFY_SWITCH bl atomic_notifier_call_chain -#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) - str r7, [r8] +#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) && \ + !defined(CONFIG_STACKPROTECTOR_PER_TASK) + str r9, [r8] #endif mov r0, r5 #if !defined(CONFIG_THUMB2_KERNEL) && !defined(CONFIG_VMAP_STACK) -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 07/12] ARM: module: implement support for PC-relative group relocations 2021-11-30 12:58 [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Ard Biesheuvel ` (5 preceding siblings ...) 2021-11-30 12:58 ` [PATCH v2 06/12] ARM: entry: preserve thread_info pointer in switch_to Ard Biesheuvel @ 2021-11-30 12:58 ` Ard Biesheuvel 2021-11-30 12:58 ` [PATCH v2 08/12] ARM: assembler: add optimized ldr/str macros to load variables from memory Ard Biesheuvel ` (5 subsequent siblings) 12 siblings, 0 replies; 23+ messages in thread From: Ard Biesheuvel @ 2021-11-30 12:58 UTC (permalink / raw) To: linux-arm-kernel, linux Cc: Ard Biesheuvel, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren Add support for the R_ARM_ALU_PC_Gn_NC and R_ARM_LDR_PC_G2 group relocations so we can use them in modules. These will be used to load the current task pointer from a global variable without having to rely on a literal pool entry to carry the address of this variable, which would be slightly less efficient. Acked-by: Linus Walleij <linus.walleij@linaro.org> Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- arch/arm/include/asm/elf.h | 3 + arch/arm/kernel/module.c | 63 ++++++++++++++++++++ 2 files changed, 66 insertions(+) diff --git a/arch/arm/include/asm/elf.h b/arch/arm/include/asm/elf.h index b8102a6ddf16..d68101655b74 100644 --- a/arch/arm/include/asm/elf.h +++ b/arch/arm/include/asm/elf.h @@ -61,6 +61,9 @@ typedef struct user_fp elf_fpregset_t; #define R_ARM_MOVT_ABS 44 #define R_ARM_MOVW_PREL_NC 45 #define R_ARM_MOVT_PREL 46 +#define R_ARM_ALU_PC_G0_NC 57 +#define R_ARM_ALU_PC_G1_NC 59 +#define R_ARM_LDR_PC_G2 63 #define R_ARM_THM_CALL 10 #define R_ARM_THM_JUMP24 30 diff --git a/arch/arm/kernel/module.c b/arch/arm/kernel/module.c index beac45e89ba6..22bdce7a7f95 100644 --- a/arch/arm/kernel/module.c +++ b/arch/arm/kernel/module.c @@ -68,6 +68,20 @@ bool module_exit_section(const char *name) strstarts(name, ".ARM.exidx.exit"); } +static u32 get_group_rem(u32 group, u32 *offset) +{ + u32 val = *offset; + u32 shift; + do { + shift = val ? (31 - __fls(val)) & ~1 : 32; + *offset = val; + if (!val) + break; + val &= 0xffffff >> shift; + } while (group--); + return shift; +} + int apply_relocate(Elf32_Shdr *sechdrs, const char *strtab, unsigned int symindex, unsigned int relindex, struct module *module) @@ -82,6 +96,7 @@ apply_relocate(Elf32_Shdr *sechdrs, const char *strtab, unsigned int symindex, unsigned long loc; Elf32_Sym *sym; const char *symname; + u32 shift, group = 1; s32 offset; u32 tmp; #ifdef CONFIG_THUMB2_KERNEL @@ -212,6 +227,54 @@ apply_relocate(Elf32_Shdr *sechdrs, const char *strtab, unsigned int symindex, *(u32 *)loc = __opcode_to_mem_arm(tmp); break; + case R_ARM_ALU_PC_G0_NC: + group = 0; + fallthrough; + case R_ARM_ALU_PC_G1_NC: + tmp = __mem_to_opcode_arm(*(u32 *)loc); + offset = ror32(tmp & 0xff, (tmp & 0xf00) >> 7); + if (tmp & BIT(22)) + offset = -offset; + offset += sym->st_value - loc; + if (offset < 0) { + offset = -offset; + tmp = (tmp & ~BIT(23)) | BIT(22); // SUB opcode + } else { + tmp = (tmp & ~BIT(22)) | BIT(23); // ADD opcode + } + + shift = get_group_rem(group, &offset); + if (shift < 24) { + offset >>= 24 - shift; + offset |= (shift + 8) << 7; + } + *(u32 *)loc = __opcode_to_mem_arm((tmp & ~0xfff) | offset); + break; + + case R_ARM_LDR_PC_G2: + tmp = __mem_to_opcode_arm(*(u32 *)loc); + offset = tmp & 0xfff; + if (~tmp & BIT(23)) // U bit cleared? + offset = -offset; + offset += sym->st_value - loc; + if (offset < 0) { + offset = -offset; + tmp &= ~BIT(23); // clear U bit + } else { + tmp |= BIT(23); // set U bit + } + get_group_rem(2, &offset); + + if (offset > 0xfff) { + pr_err("%s: section %u reloc %u sym '%s': relocation %u out of range (%#lx -> %#x)\n", + module->name, relindex, i, symname, + ELF32_R_TYPE(rel->r_info), loc, + sym->st_value); + return -ENOEXEC; + } + *(u32 *)loc = __opcode_to_mem_arm((tmp & ~0xfff) | offset); + break; + #ifdef CONFIG_THUMB2_KERNEL case R_ARM_THM_CALL: case R_ARM_THM_JUMP24: -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 08/12] ARM: assembler: add optimized ldr/str macros to load variables from memory 2021-11-30 12:58 [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Ard Biesheuvel ` (6 preceding siblings ...) 2021-11-30 12:58 ` [PATCH v2 07/12] ARM: module: implement support for PC-relative group relocations Ard Biesheuvel @ 2021-11-30 12:58 ` Ard Biesheuvel 2021-11-30 12:58 ` [PATCH v2 09/12] ARM: percpu: add SMP_ON_UP support Ard Biesheuvel ` (4 subsequent siblings) 12 siblings, 0 replies; 23+ messages in thread From: Ard Biesheuvel @ 2021-11-30 12:58 UTC (permalink / raw) To: linux-arm-kernel, linux Cc: Ard Biesheuvel, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren We will be adding variable loads to various hot paths, so it makes sense to add a helper macro that can load variables from asm code without the use of literal pool entries. On v7 or later, we can simply use MOVW/MOVT pairs, but on earlier cores, this requires a bit of hackery to emit a instruction sequence that implements this using a sequence of ADD/LDR instructions. Acked-by: Linus Walleij <linus.walleij@linaro.org> Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- arch/arm/include/asm/assembler.h | 45 ++++++++++++++++++-- arch/arm/kernel/entry-armv.S | 2 +- arch/arm/kernel/entry-header.S | 2 +- 3 files changed, 43 insertions(+), 6 deletions(-) diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h index 1b9d4df331aa..2095638b7140 100644 --- a/arch/arm/include/asm/assembler.h +++ b/arch/arm/include/asm/assembler.h @@ -568,12 +568,12 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) /* * mov_l - move a constant value or [relocated] address into a register */ - .macro mov_l, dst:req, imm:req + .macro mov_l, dst:req, imm:req, cond .if __LINUX_ARM_ARCH__ < 7 - ldr \dst, =\imm + ldr\cond \dst, =\imm .else - movw \dst, #:lower16:\imm - movt \dst, #:upper16:\imm + movw\cond \dst, #:lower16:\imm + movt\cond \dst, #:upper16:\imm .endif .endm @@ -611,6 +611,43 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) __adldst_l str, \src, \sym, \tmp, \cond .endm + .macro __ldst_va, op, reg, tmp, sym, cond +#if __LINUX_ARM_ARCH__ >= 7 || \ + (defined(MODULE) && defined(CONFIG_ARM_MODULE_PLTS)) || \ + (defined(CONFIG_LD_IS_LLD) && CONFIG_LLD_VERSION < 140000) + mov_l \tmp, \sym, \cond + \op\cond \reg, [\tmp] +#else + /* + * Avoid a literal load, by emitting a sequence of ADD/LDR instructions + * with the appropriate relocations. The combined sequence has a range + * of -/+ 256 MiB, which should be sufficient for the core kernel and + * for modules loaded into the module region. + */ + .globl \sym + .reloc .L0_\@, R_ARM_ALU_PC_G0_NC, \sym + .reloc .L1_\@, R_ARM_ALU_PC_G1_NC, \sym + .reloc .L2_\@, R_ARM_LDR_PC_G2, \sym +.L0_\@: sub\cond \tmp, pc, #8 +.L1_\@: sub\cond \tmp, \tmp, #4 +.L2_\@: \op\cond \reg, [\tmp, #0] +#endif + .endm + + /* + * ldr_va - load a 32-bit word from the virtual address of \sym + */ + .macro ldr_va, rd:req, sym:req, cond + __ldst_va ldr, \rd, \rd, \sym, \cond + .endm + + /* + * str_va - store a 32-bit word to the virtual address of \sym + */ + .macro str_va, rn:req, sym:req, tmp:req, cond + __ldst_va str, \rn, \tmp, \sym, \cond + .endm + /* * rev_l - byte-swap a 32-bit value * diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 1a6cf711a3b4..7f7ac963445c 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -53,7 +53,7 @@ UNWIND( .setfp fpreg, sp ) subs r2, sp, r0 @ SP above bottom of IRQ stack? rsbscs r2, r2, #THREAD_SIZE @ ... and below the top? #ifdef CONFIG_VMAP_STACK - ldr_l r2, high_memory, cc @ End of the linear region + ldr_va r2, high_memory, cc @ End of the linear region cmpcc r2, r0 @ Stack pointer was below it? #endif movcs sp, r0 @ If so, revert to incoming SP diff --git a/arch/arm/kernel/entry-header.S b/arch/arm/kernel/entry-header.S index 81df2a3561ca..268f7f4c5c05 100644 --- a/arch/arm/kernel/entry-header.S +++ b/arch/arm/kernel/entry-header.S @@ -445,7 +445,7 @@ THUMB( it ne ) @ in such cases so just carry on. @ str ip, [r0, #12] @ Stash IP on the mode stack - ldr_l ip, high_memory @ Start of VMALLOC space + ldr_va ip, high_memory @ Start of VMALLOC space ARM( cmp sp, ip ) @ SP in vmalloc space? THUMB( cmp r1, ip ) THUMB( itt lo ) -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 09/12] ARM: percpu: add SMP_ON_UP support 2021-11-30 12:58 [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Ard Biesheuvel ` (7 preceding siblings ...) 2021-11-30 12:58 ` [PATCH v2 08/12] ARM: assembler: add optimized ldr/str macros to load variables from memory Ard Biesheuvel @ 2021-11-30 12:58 ` Ard Biesheuvel 2021-11-30 15:12 ` Russell King (Oracle) 2021-11-30 12:58 ` [PATCH v2 10/12] ARM: use TLS register for 'current' on !SMP as well Ard Biesheuvel ` (3 subsequent siblings) 12 siblings, 1 reply; 23+ messages in thread From: Ard Biesheuvel @ 2021-11-30 12:58 UTC (permalink / raw) To: linux-arm-kernel, linux Cc: Ard Biesheuvel, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren Permit the use of the TPIDRPRW system register for carrying the per-CPU offset in generic SMP configurations that also target non-SMP capable ARMv6 cores. This uses the SMP_ON_UP code patching framework to turn all TPIDRPRW accesses into reads/writes of entry #0 in the __per_cpu_offset array. While at it, switch over some existing direct TPIDRPRW accesses in asm code to invocations of a new helper that is patched in the same way when necessary. Note that CPU_V6+SMP without SMP_ON_UP results in a kernel that does not boot on v6 CPUs without SMP extensions, so add this dependency to Kconfig as well. Acked-by: Linus Walleij <linus.walleij@linaro.org> Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- arch/arm/include/asm/assembler.h | 59 +++++++++++++++++++- arch/arm/include/asm/insn.h | 24 ++++++++ arch/arm/include/asm/percpu.h | 25 ++++++++- arch/arm/kernel/entry-armv.S | 16 +----- arch/arm/kernel/sleep.S | 4 +- arch/arm/mm/Kconfig | 1 + 6 files changed, 107 insertions(+), 22 deletions(-) diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h index 2095638b7140..f9b3dd0e9ef5 100644 --- a/arch/arm/include/asm/assembler.h +++ b/arch/arm/include/asm/assembler.h @@ -220,9 +220,7 @@ THUMB( fpreg .req r7 ) .macro reload_current, t1:req, t2:req #ifdef CONFIG_CURRENT_POINTER_IN_TPIDRURO - adr_l \t1, __entry_task @ get __entry_task base address - mrc p15, 0, \t2, c13, c0, 4 @ get per-CPU offset - ldr \t1, [\t1, \t2] @ load variable + ldr_this_cpu \t1, __entry_task, \t1, \t2 mcr p15, 0, \t1, c13, c0, 3 @ store in TPIDRURO #endif .endm @@ -312,6 +310,26 @@ THUMB( fpreg .req r7 ) #define ALT_UP_B(label) b label #endif + /* + * this_cpu_offset - load the per-CPU offset of this CPU into + * register 'rd' + */ + .macro this_cpu_offset, rd:req +#ifdef CONFIG_SMP +ALT_SMP(mrc p15, 0, \rd, c13, c0, 4) +#ifdef CONFIG_CPU_V6 +ALT_UP_B(.L1_\@) +.L0_\@: + .subsection 1 +.L1_\@: ldr_va \rd, __per_cpu_offset + b .L0_\@ + .previous +#endif +#else + mov \rd, #0 +#endif + .endm + /* * Instruction barrier */ @@ -648,6 +666,41 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) __ldst_va str, \rn, \tmp, \sym, \cond .endm + /* + * ldr_this_cpu_armv6 - Load a 32-bit word from the per-CPU variable 'sym', + * without using a temp register. Supported in ARM mode + * only. + */ + .macro ldr_this_cpu_armv6, rd:req, sym:req + this_cpu_offset \rd + .globl \sym + .reloc .L0_\@, R_ARM_ALU_PC_G0_NC, \sym + .reloc .L1_\@, R_ARM_ALU_PC_G1_NC, \sym + .reloc .L2_\@, R_ARM_LDR_PC_G2, \sym + add \rd, \rd, pc +.L0_\@: sub \rd, \rd, #4 +.L1_\@: sub \rd, \rd, #0 +.L2_\@: ldr \rd, [\rd, #4] + .endm + + /* + * ldr_this_cpu - Load a 32-bit word from the per-CPU variable 'sym' + * into register 'rd', which may be the stack pointer, + * using 't1' and 't2' as general temp registers. These + * are permitted to overlap with 'rd' if != sp + */ + .macro ldr_this_cpu, rd:req, sym:req, t1:req, t2:req +#if __LINUX_ARM_ARCH__ >= 7 || \ + (defined(MODULE) && defined(CONFIG_ARM_MODULE_PLTS)) || \ + (defined(CONFIG_LD_IS_LLD) && CONFIG_LLD_VERSION < 140000) + this_cpu_offset \t1 + mov_l \t2, \sym + ldr \rd, [\t1, \t2] +#else + ldr_this_cpu_armv6 \rd, \sym +#endif + .endm + /* * rev_l - byte-swap a 32-bit value * diff --git a/arch/arm/include/asm/insn.h b/arch/arm/include/asm/insn.h index 5475cbf9fb6b..a160ed3ea427 100644 --- a/arch/arm/include/asm/insn.h +++ b/arch/arm/include/asm/insn.h @@ -2,6 +2,30 @@ #ifndef __ASM_ARM_INSN_H #define __ASM_ARM_INSN_H +#include <linux/types.h> + +/* + * Avoid a literal load by emitting a sequence of ADD/LDR instructions with the + * appropriate relocations. The combined sequence has a range of -/+ 256 MiB, + * which should be sufficient for the core kernel as well as modules loaded + * into the module region. (Not supported by LLD before release 14) + */ +#if !(defined(MODULE) && defined(CONFIG_ARM_MODULE_PLTS)) && \ + !(defined(CONFIG_LD_IS_LLD) && CONFIG_LLD_VERSION < 140000) +#define LOAD_SYM_ARMV6(reg, sym) \ + " .globl " #sym " \n\t" \ + " .reloc 10f, R_ARM_ALU_PC_G0_NC, " #sym " \n\t" \ + " .reloc 11f, R_ARM_ALU_PC_G1_NC, " #sym " \n\t" \ + " .reloc 12f, R_ARM_LDR_PC_G2, " #sym " \n\t" \ + "10: sub " #reg ", pc, #8 \n\t" \ + "11: sub " #reg ", " #reg ", #4 \n\t" \ + "12: ldr " #reg ", [" #reg ", #0] \n\t" +#else +#define LOAD_SYM_ARMV6(reg, sym) \ + " ldr " #reg ", =" #sym " \n\t" \ + " ldr " #reg ", [" #reg "] \n\t" +#endif + static inline unsigned long arm_gen_nop(void) { diff --git a/arch/arm/include/asm/percpu.h b/arch/arm/include/asm/percpu.h index e2fcb3cfd3de..ff8f0829e041 100644 --- a/arch/arm/include/asm/percpu.h +++ b/arch/arm/include/asm/percpu.h @@ -5,15 +5,22 @@ #ifndef _ASM_ARM_PERCPU_H_ #define _ASM_ARM_PERCPU_H_ +#include <asm/insn.h> + register unsigned long current_stack_pointer asm ("sp"); /* * Same as asm-generic/percpu.h, except that we store the per cpu offset * in the TPIDRPRW. TPIDRPRW only exists on V6K and V7 */ -#if defined(CONFIG_SMP) && !defined(CONFIG_CPU_V6) +#ifdef CONFIG_SMP +extern unsigned int smp_on_up; + static inline void set_my_cpu_offset(unsigned long off) { + if (IS_ENABLED(CONFIG_CPU_V6) && !smp_on_up) + return; + /* Set TPIDRPRW */ asm volatile("mcr p15, 0, %0, c13, c0, 4" : : "r" (off) : "memory"); } @@ -27,8 +34,20 @@ static inline unsigned long __my_cpu_offset(void) * We want to allow caching the value, so avoid using volatile and * instead use a fake stack read to hazard against barrier(). */ - asm("mrc p15, 0, %0, c13, c0, 4" : "=r" (off) - : "Q" (*(const unsigned long *)current_stack_pointer)); + asm("0: mrc p15, 0, %0, c13, c0, 4 \n\t" +#ifdef CONFIG_CPU_V6 + "1: \n\t" + " .subsection 1 \n\t" + "2: " LOAD_SYM_ARMV6(%0, __per_cpu_offset) " \n\t" + " b 1b \n\t" + " .previous \n\t" + " .pushsection \".alt.smp.init\", \"a\" \n\t" + " .long 0b - . \n\t" + " b . + (2b - 0b) \n\t" + " .popsection \n\t" +#endif + : "=r" (off) + : "Q" (*(const unsigned long *)current_stack_pointer)); return off; } diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 7f7ac963445c..43d917f0d9a9 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -35,15 +35,14 @@ .macro irq_handler, from_user:req mov r0, sp #ifdef CONFIG_IRQSTACKS - mov_l r2, irq_stack_ptr @ Take base address - mrc p15, 0, r3, c13, c0, 4 @ Get CPU offset #ifdef CONFIG_UNWINDER_ARM mov fpreg, sp @ Preserve original SP #else mov r8, fp @ Preserve original FP mov r9, sp @ Preserve original SP #endif - ldr sp, [r2, r3] @ Load SP from per-CPU var + ldr_this_cpu sp, irq_stack_ptr, r2, r3 + .if \from_user == 0 UNWIND( .setfp fpreg, sp ) @ @@ -876,16 +875,7 @@ __bad_stack: THUMB( bx pc ) THUMB( nop ) THUMB( .arm ) - mrc p15, 0, ip, c13, c0, 4 @ Get per-CPU offset - - .globl overflow_stack_ptr - .reloc 0f, R_ARM_ALU_PC_G0_NC, overflow_stack_ptr - .reloc 1f, R_ARM_ALU_PC_G1_NC, overflow_stack_ptr - .reloc 2f, R_ARM_LDR_PC_G2, overflow_stack_ptr - add ip, ip, pc -0: add ip, ip, #-4 -1: add ip, ip, #0 -2: ldr ip, [ip, #4] + ldr_this_cpu_armv6 ip, overflow_stack_ptr str sp, [ip, #-4]! @ Preserve original SP value mov sp, ip @ Switch to overflow stack diff --git a/arch/arm/kernel/sleep.S b/arch/arm/kernel/sleep.S index 803b51e5cba0..f909baf17912 100644 --- a/arch/arm/kernel/sleep.S +++ b/arch/arm/kernel/sleep.S @@ -71,9 +71,7 @@ ENTRY(__cpu_suspend) @ Run the suspend code from the overflow stack so we don't have to rely @ on vmalloc-to-phys conversions anywhere in the arch suspend code. @ The original SP value captured in R5 will be restored on the way out. - mov_l r6, overflow_stack_ptr @ Base pointer - mrc p15, 0, r7, c13, c0, 4 @ Get per-CPU offset - ldr sp, [r6, r7] @ Address of this CPU's overflow stack + ldr_this_cpu sp, overflow_stack_ptr, r6, r7 #endif add r4, r4, #12 @ Space for pgd, virt sp, phys resume fn sub sp, sp, r4 @ allocate CPU state on stack diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig index 58afba346729..a91ff22c6c2e 100644 --- a/arch/arm/mm/Kconfig +++ b/arch/arm/mm/Kconfig @@ -386,6 +386,7 @@ config CPU_V6 select CPU_PABRT_V6 select CPU_THUMB_CAPABLE select CPU_TLB_V6 if MMU + select SMP_ON_UP if SMP # ARMv6k config CPU_V6K -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH v2 09/12] ARM: percpu: add SMP_ON_UP support 2021-11-30 12:58 ` [PATCH v2 09/12] ARM: percpu: add SMP_ON_UP support Ard Biesheuvel @ 2021-11-30 15:12 ` Russell King (Oracle) 2021-11-30 15:45 ` Ard Biesheuvel 0 siblings, 1 reply; 23+ messages in thread From: Russell King (Oracle) @ 2021-11-30 15:12 UTC (permalink / raw) To: Ard Biesheuvel Cc: linux-arm-kernel, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren On Tue, Nov 30, 2021 at 01:58:58PM +0100, Ard Biesheuvel wrote: > diff --git a/arch/arm/include/asm/percpu.h b/arch/arm/include/asm/percpu.h > index e2fcb3cfd3de..ff8f0829e041 100644 > --- a/arch/arm/include/asm/percpu.h > +++ b/arch/arm/include/asm/percpu.h > @@ -5,15 +5,22 @@ > #ifndef _ASM_ARM_PERCPU_H_ > #define _ASM_ARM_PERCPU_H_ > > +#include <asm/insn.h> > + > register unsigned long current_stack_pointer asm ("sp"); > > /* > * Same as asm-generic/percpu.h, except that we store the per cpu offset > * in the TPIDRPRW. TPIDRPRW only exists on V6K and V7 > */ > -#if defined(CONFIG_SMP) && !defined(CONFIG_CPU_V6) > +#ifdef CONFIG_SMP > +extern unsigned int smp_on_up; I would much rather we kept this hidden from global view by placing it inside the set_my_cpu_offset() body. -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last! _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 09/12] ARM: percpu: add SMP_ON_UP support 2021-11-30 15:12 ` Russell King (Oracle) @ 2021-11-30 15:45 ` Ard Biesheuvel 0 siblings, 0 replies; 23+ messages in thread From: Ard Biesheuvel @ 2021-11-30 15:45 UTC (permalink / raw) To: Russell King (Oracle) Cc: Linux ARM, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren On Tue, 30 Nov 2021 at 16:12, Russell King (Oracle) <linux@armlinux.org.uk> wrote: > > On Tue, Nov 30, 2021 at 01:58:58PM +0100, Ard Biesheuvel wrote: > > diff --git a/arch/arm/include/asm/percpu.h b/arch/arm/include/asm/percpu.h > > index e2fcb3cfd3de..ff8f0829e041 100644 > > --- a/arch/arm/include/asm/percpu.h > > +++ b/arch/arm/include/asm/percpu.h > > @@ -5,15 +5,22 @@ > > #ifndef _ASM_ARM_PERCPU_H_ > > #define _ASM_ARM_PERCPU_H_ > > > > +#include <asm/insn.h> > > + > > register unsigned long current_stack_pointer asm ("sp"); > > > > /* > > * Same as asm-generic/percpu.h, except that we store the per cpu offset > > * in the TPIDRPRW. TPIDRPRW only exists on V6K and V7 > > */ > > -#if defined(CONFIG_SMP) && !defined(CONFIG_CPU_V6) > > +#ifdef CONFIG_SMP > > +extern unsigned int smp_on_up; > > I would much rather we kept this hidden from global view by placing it > inside the set_my_cpu_offset() body. > Good point, I'll change that. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 10/12] ARM: use TLS register for 'current' on !SMP as well 2021-11-30 12:58 [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Ard Biesheuvel ` (8 preceding siblings ...) 2021-11-30 12:58 ` [PATCH v2 09/12] ARM: percpu: add SMP_ON_UP support Ard Biesheuvel @ 2021-11-30 12:58 ` Ard Biesheuvel 2021-11-30 14:05 ` Arnd Bergmann 2021-11-30 12:59 ` [PATCH v2 11/12] ARM: smp: defer TPIDRURO update for SMP v6 configurations too Ard Biesheuvel ` (2 subsequent siblings) 12 siblings, 1 reply; 23+ messages in thread From: Ard Biesheuvel @ 2021-11-30 12:58 UTC (permalink / raw) To: linux-arm-kernel, linux Cc: Ard Biesheuvel, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren Enable the use of the TLS register to hold the 'current' pointer also on non-SMP configurations that target v6k or later CPUs. This will permit the use of THREAD_INFO_IN_TASK as well as IRQ stacks and vmap'ed stacks for such configurations. Acked-by: Linus Walleij <linus.walleij@linaro.org> Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- arch/arm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index b0e403076227..5ad2151c43dd 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -1154,7 +1154,7 @@ config SMP_ON_UP config CURRENT_POINTER_IN_TPIDRURO def_bool y - depends on SMP && CPU_32v6K && !CPU_V6 + depends on CPU_32v6K && !CPU_V6 config IRQSTACKS def_bool y -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH v2 10/12] ARM: use TLS register for 'current' on !SMP as well 2021-11-30 12:58 ` [PATCH v2 10/12] ARM: use TLS register for 'current' on !SMP as well Ard Biesheuvel @ 2021-11-30 14:05 ` Arnd Bergmann 0 siblings, 0 replies; 23+ messages in thread From: Arnd Bergmann @ 2021-11-30 14:05 UTC (permalink / raw) To: Ard Biesheuvel Cc: Linux ARM, Russell King - ARM Linux, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren On Tue, Nov 30, 2021 at 1:58 PM Ard Biesheuvel <ardb@kernel.org> wrote: > > Enable the use of the TLS register to hold the 'current' pointer also on > non-SMP configurations that target v6k or later CPUs. This will permit > the use of THREAD_INFO_IN_TASK as well as IRQ stacks and vmap'ed stacks > for such configurations. > > Acked-by: Linus Walleij <linus.walleij@linaro.org> > Acked-by: Nicolas Pitre <nico@fluxnic.net> > Signed-off-by: Ard Biesheuvel <ardb@kernel.org> > --- > arch/arm/Kconfig | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig > index b0e403076227..5ad2151c43dd 100644 > --- a/arch/arm/Kconfig > +++ b/arch/arm/Kconfig > @@ -1154,7 +1154,7 @@ config SMP_ON_UP > > config CURRENT_POINTER_IN_TPIDRURO > def_bool y > - depends on SMP && CPU_32v6K && !CPU_V6 > + depends on CPU_32v6K && !CPU_V6 Acked-by: Arnd Bergmann <arnd@arndb.de> _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 11/12] ARM: smp: defer TPIDRURO update for SMP v6 configurations too 2021-11-30 12:58 [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Ard Biesheuvel ` (9 preceding siblings ...) 2021-11-30 12:58 ` [PATCH v2 10/12] ARM: use TLS register for 'current' on !SMP as well Ard Biesheuvel @ 2021-11-30 12:59 ` Ard Biesheuvel 2021-11-30 12:59 ` [PATCH v2 12/12] ARM: implement THREAD_INFO_IN_TASK for uniprocessor systems Ard Biesheuvel 2021-11-30 14:08 ` [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Arnd Bergmann 12 siblings, 0 replies; 23+ messages in thread From: Ard Biesheuvel @ 2021-11-30 12:59 UTC (permalink / raw) To: linux-arm-kernel, linux Cc: Ard Biesheuvel, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren Defer TPIDURO updates for user space until exit also for CPU_V6+SMP configurations so that we can decide at runtime whether to use it to carry the current pointer, provided that we are running on a CPU that actually implements this register. This is needed for THREAD_INFO_IN_TASK support for UP systems, which requires that all SMP capable systems use the TPIDRURO based access to 'current' as the only remaining alternative will be a global variable which only works on UP. Acked-by: Linus Walleij <linus.walleij@linaro.org> Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- arch/arm/include/asm/tls.h | 13 +++++++------ arch/arm/kernel/entry-header.S | 11 ++++++++++- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/arch/arm/include/asm/tls.h b/arch/arm/include/asm/tls.h index c3296499176c..d712c170c095 100644 --- a/arch/arm/include/asm/tls.h +++ b/arch/arm/include/asm/tls.h @@ -18,13 +18,14 @@ .endm .macro switch_tls_v6, base, tp, tpuser, tmp1, tmp2 - ldr \tmp1, =elf_hwcap - ldr \tmp1, [\tmp1, #0] + ldr_va \tmp1, elf_hwcap mov \tmp2, #0xffff0fff tst \tmp1, #HWCAP_TLS @ hardware TLS available? streq \tp, [\tmp2, #-15] @ set TLS value at 0xffff0ff0 mrcne p15, 0, \tmp2, c13, c0, 2 @ get the user r/w register +#ifndef CONFIG_SMP mcrne p15, 0, \tp, c13, c0, 3 @ yes, set TLS register +#endif mcrne p15, 0, \tpuser, c13, c0, 2 @ set user r/w register strne \tmp2, [\base, #TI_TP_VALUE + 4] @ save it .endm @@ -43,7 +44,7 @@ #elif defined(CONFIG_CPU_V6) #define tls_emu 0 #define has_tls_reg (elf_hwcap & HWCAP_TLS) -#define defer_tls_reg_update 0 +#define defer_tls_reg_update IS_ENABLED(CONFIG_SMP) #define switch_tls switch_tls_v6 #elif defined(CONFIG_CPU_32v6K) #define tls_emu 0 @@ -81,11 +82,11 @@ static inline void set_tls(unsigned long val) */ barrier(); - if (!tls_emu && !defer_tls_reg_update) { - if (has_tls_reg) { + if (!tls_emu) { + if (has_tls_reg && !defer_tls_reg_update) { asm("mcr p15, 0, %0, c13, c0, 3" : : "r" (val)); - } else { + } else if (!has_tls_reg) { #ifdef CONFIG_KUSER_HELPERS /* * User space must never try to access this diff --git a/arch/arm/kernel/entry-header.S b/arch/arm/kernel/entry-header.S index 268f7f4c5c05..cb82ff5adec1 100644 --- a/arch/arm/kernel/entry-header.S +++ b/arch/arm/kernel/entry-header.S @@ -292,12 +292,21 @@ .macro restore_user_regs, fast = 0, offset = 0 -#if defined(CONFIG_CPU_32v6K) && !defined(CONFIG_CPU_V6) +#if defined(CONFIG_CPU_32v6K) || defined(CONFIG_SMP) +#if defined(CONFIG_CPU_V6) && defined(CONFIG_SMP) +ALT_SMP(b .L1_\@ ) +ALT_UP( nop ) + ldr_va r1, elf_hwcap + tst r1, #HWCAP_TLS @ hardware TLS available? + beq .L2_\@ +.L1_\@: +#endif @ The TLS register update is deferred until return to user space so we @ can use it for other things while running in the kernel get_thread_info r1 ldr r1, [r1, #TI_TP_VALUE] mcr p15, 0, r1, c13, c0, 3 @ set TLS register +.L2_\@: #endif uaccess_enable r1, isb=0 -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 12/12] ARM: implement THREAD_INFO_IN_TASK for uniprocessor systems 2021-11-30 12:58 [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Ard Biesheuvel ` (10 preceding siblings ...) 2021-11-30 12:59 ` [PATCH v2 11/12] ARM: smp: defer TPIDRURO update for SMP v6 configurations too Ard Biesheuvel @ 2021-11-30 12:59 ` Ard Biesheuvel 2021-11-30 13:43 ` Arnd Bergmann 2021-11-30 14:08 ` [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Arnd Bergmann 12 siblings, 1 reply; 23+ messages in thread From: Ard Biesheuvel @ 2021-11-30 12:59 UTC (permalink / raw) To: linux-arm-kernel, linux Cc: Ard Biesheuvel, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren On UP systems, only a single task can be 'current' at the same time, which means we can use a global variable to track it. This means we can also enable THREAD_INFO_IN_TASK for those systems, as in that case, thread_info is accessed via current rather than the other way around, removing the need to store thread_info at the base of the task stack. This, in turn, permits us to enable IRQ stacks and vmap'ed stacks on UP systems as well. To partially mitigate the performance overhead of this arrangement, use a ADD/ADD/LDR sequence with the appropriate PC-relative group relocations to load the value of current when needed. This means that accessing current will still only require a single load as before, avoiding the need for a literal to carry the address of the global variable in each function. However, accessing thread_info will now require this load as well. Acked-by: Linus Walleij <linus.walleij@linaro.org> Acked-by: Nicolas Pitre <nico@fluxnic.net> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- arch/arm/Kconfig | 7 +- arch/arm/include/asm/assembler.h | 83 +++++++++++++------- arch/arm/include/asm/current.h | 37 +++++---- arch/arm/include/asm/switch_to.h | 3 +- arch/arm/include/asm/thread_info.h | 27 ------- arch/arm/kernel/asm-offsets.c | 3 - arch/arm/kernel/entry-armv.S | 11 +-- arch/arm/kernel/head-common.S | 4 +- arch/arm/kernel/process.c | 7 +- arch/arm/kernel/smp.c | 11 +++ arch/arm/kernel/traps.c | 4 + 11 files changed, 109 insertions(+), 88 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 5ad2151c43dd..359a3b85c8b3 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -127,8 +127,8 @@ config ARM select PERF_USE_VMALLOC select RTC_LIB select SYS_SUPPORTS_APM_EMULATION - select THREAD_INFO_IN_TASK if CURRENT_POINTER_IN_TPIDRURO - select HAVE_ARCH_VMAP_STACK if MMU && THREAD_INFO_IN_TASK && (!LD_IS_LLD || LLD_VERSION >= 140000) + select THREAD_INFO_IN_TASK + select HAVE_ARCH_VMAP_STACK if MMU && (!LD_IS_LLD || LLD_VERSION >= 140000) select TRACE_IRQFLAGS_SUPPORT if !CPU_V7M # Above selects are sorted alphabetically; please add new ones # according to that. Thanks. @@ -1158,7 +1158,6 @@ config CURRENT_POINTER_IN_TPIDRURO config IRQSTACKS def_bool y - depends on THREAD_INFO_IN_TASK select HAVE_IRQ_EXIT_ON_IRQ_STACK select HAVE_SOFTIRQ_ON_OWN_STACK @@ -1608,7 +1607,7 @@ config CC_HAVE_STACKPROTECTOR_TLS config STACKPROTECTOR_PER_TASK bool "Use a unique stack canary value for each task" - depends on STACKPROTECTOR && THREAD_INFO_IN_TASK && !XIP_DEFLATED_DATA + depends on STACKPROTECTOR && CURRENT_POINTER_IN_TPIDRURO && !XIP_DEFLATED_DATA depends on GCC_PLUGINS || CC_HAVE_STACKPROTECTOR_TLS select GCC_PLUGIN_ARM_SSP_PER_TASK if !CC_HAVE_STACKPROTECTOR_TLS default y diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h index f9b3dd0e9ef5..59d7b9e81934 100644 --- a/arch/arm/include/asm/assembler.h +++ b/arch/arm/include/asm/assembler.h @@ -203,41 +203,12 @@ THUMB( fpreg .req r7 ) .endm .endr - .macro get_current, rd -#ifdef CONFIG_CURRENT_POINTER_IN_TPIDRURO - mrc p15, 0, \rd, c13, c0, 3 @ get TPIDRURO register -#else - get_thread_info \rd - ldr \rd, [\rd, #TI_TASK] -#endif - .endm - - .macro set_current, rn -#ifdef CONFIG_CURRENT_POINTER_IN_TPIDRURO - mcr p15, 0, \rn, c13, c0, 3 @ set TPIDRURO register -#endif - .endm - - .macro reload_current, t1:req, t2:req -#ifdef CONFIG_CURRENT_POINTER_IN_TPIDRURO - ldr_this_cpu \t1, __entry_task, \t1, \t2 - mcr p15, 0, \t1, c13, c0, 3 @ store in TPIDRURO -#endif - .endm - /* * Get current thread_info. */ .macro get_thread_info, rd -#ifdef CONFIG_THREAD_INFO_IN_TASK /* thread_info is the first member of struct task_struct */ get_current \rd -#else - ARM( mov \rd, sp, lsr #THREAD_SIZE_ORDER + PAGE_SHIFT ) - THUMB( mov \rd, sp ) - THUMB( lsr \rd, \rd, #THREAD_SIZE_ORDER + PAGE_SHIFT ) - mov \rd, \rd, lsl #THREAD_SIZE_ORDER + PAGE_SHIFT -#endif .endm /* @@ -330,6 +301,60 @@ ALT_UP_B(.L1_\@) #endif .endm + /* + * set_current - store the task pointer of this CPU's current task + */ + .macro set_current, rn:req, tmp:req +#if defined(CONFIG_CURRENT_POINTER_IN_TPIDRURO) || defined(CONFIG_SMP) +9998: mcr p15, 0, \rn, c13, c0, 3 @ set TPIDRURO register +#ifdef CONFIG_CPU_V6 +ALT_UP_B(.L0_\@) + .subsection 1 +.L0_\@: str_va \rn, __current, \tmp + b .L1_\@ + .previous +.L1_\@: +#endif +#else + str_va \rn, __current, \tmp +#endif + .endm + + /* + * get_current - load the task pointer of this CPU's current task + */ + .macro get_current, rd:req +#if defined(CONFIG_CURRENT_POINTER_IN_TPIDRURO) || defined(CONFIG_SMP) +9998: mrc p15, 0, \rd, c13, c0, 3 @ get TPIDRURO register +#ifdef CONFIG_CPU_V6 +ALT_UP_B(.L0_\@) + .subsection 1 +.L0_\@: ldr_va \rd, __current + b .L1_\@ + .previous +.L1_\@: +#endif +#else + ldr_va \rd, __current +#endif + .endm + + /* + * reload_current - reload the task pointer of this CPU's current task + * into the TLS register + */ + .macro reload_current, t1:req, t2:req +#if defined(CONFIG_CURRENT_POINTER_IN_TPIDRURO) || defined(CONFIG_SMP) +#ifdef CONFIG_CPU_V6 +ALT_SMP(nop) +ALT_UP_B(.L0_\@) +#endif + ldr_this_cpu \t1, __entry_task, \t1, \t2 + mcr p15, 0, \t1, c13, c0, 3 @ store in TPIDRURO +.L0_\@: +#endif + .endm + /* * Instruction barrier */ diff --git a/arch/arm/include/asm/current.h b/arch/arm/include/asm/current.h index 6bf0aad672c3..69ecf4c6c725 100644 --- a/arch/arm/include/asm/current.h +++ b/arch/arm/include/asm/current.h @@ -8,25 +8,18 @@ #define _ASM_ARM_CURRENT_H #ifndef __ASSEMBLY__ +#include <asm/insn.h> struct task_struct; -static inline void set_current(struct task_struct *cur) -{ - if (!IS_ENABLED(CONFIG_CURRENT_POINTER_IN_TPIDRURO)) - return; - - /* Set TPIDRURO */ - asm("mcr p15, 0, %0, c13, c0, 3" :: "r"(cur) : "memory"); -} +extern struct task_struct *__current; -#ifdef CONFIG_CURRENT_POINTER_IN_TPIDRURO - -static inline struct task_struct *get_current(void) +static inline __attribute_const__ struct task_struct *get_current(void) { struct task_struct *cur; #if __has_builtin(__builtin_thread_pointer) && \ + defined(CONFIG_CURRENT_POINTER_IN_TPIDRURO) && \ !(defined(CONFIG_THUMB2_KERNEL) && \ defined(CONFIG_CC_IS_CLANG) && CONFIG_CLANG_VERSION < 130001) /* @@ -39,16 +32,30 @@ static inline struct task_struct *get_current(void) * https://github.com/ClangBuiltLinux/linux/issues/1485 */ cur = __builtin_thread_pointer(); +#elif defined(CONFIG_CURRENT_POINTER_IN_TPIDRURO) || defined(CONFIG_SMP) + asm("0: mrc p15, 0, %0, c13, c0, 3 \n\t" +#ifdef CONFIG_CPU_V6 + "1: \n\t" + " .subsection 1 \n\t" + "2: " LOAD_SYM_ARMV6(%0, __current) " \n\t" + " b 1b \n\t" + " .previous \n\t" + " .pushsection \".alt.smp.init\", \"a\" \n\t" + " .long 0b - . \n\t" + " b . + (2b - 0b) \n\t" + " .popsection \n\t" +#endif + : "=r"(cur)); +#elif __LINUX_ARM_ARCH__>=7 || \ + (defined(MODULE) && defined(CONFIG_ARM_MODULE_PLTS)) + cur = __current; #else - asm("mrc p15, 0, %0, c13, c0, 3" : "=r"(cur)); + asm(LOAD_SYM_ARMV6(%0, __current) : "=r"(cur)); #endif return cur; } #define current get_current() -#else -#include <asm-generic/current.h> -#endif /* CONFIG_CURRENT_POINTER_IN_TPIDRURO */ #endif /* __ASSEMBLY__ */ diff --git a/arch/arm/include/asm/switch_to.h b/arch/arm/include/asm/switch_to.h index b55c7b2755e4..a482c99934ff 100644 --- a/arch/arm/include/asm/switch_to.h +++ b/arch/arm/include/asm/switch_to.h @@ -40,7 +40,8 @@ static inline void set_ti_cpu(struct task_struct *p) do { \ __complete_pending_tlbi(); \ set_ti_cpu(next); \ - if (IS_ENABLED(CONFIG_CURRENT_POINTER_IN_TPIDRURO)) \ + if (IS_ENABLED(CONFIG_CURRENT_POINTER_IN_TPIDRURO) || \ + IS_ENABLED(CONFIG_SMP)) \ __this_cpu_write(__entry_task, next); \ last = __switch_to(prev,task_thread_info(prev), task_thread_info(next)); \ } while (0) diff --git a/arch/arm/include/asm/thread_info.h b/arch/arm/include/asm/thread_info.h index 004b89d86224..aecc403b2880 100644 --- a/arch/arm/include/asm/thread_info.h +++ b/arch/arm/include/asm/thread_info.h @@ -62,9 +62,6 @@ struct cpu_context_save { struct thread_info { unsigned long flags; /* low level flags */ int preempt_count; /* 0 => preemptable, <0 => bug */ -#ifndef CONFIG_THREAD_INFO_IN_TASK - struct task_struct *task; /* main task structure */ -#endif __u32 cpu; /* cpu */ __u32 cpu_domain; /* cpu domain */ struct cpu_context_save cpu_context; /* cpu context */ @@ -80,39 +77,15 @@ struct thread_info { #define INIT_THREAD_INFO(tsk) \ { \ - INIT_THREAD_INFO_TASK(tsk) \ .flags = 0, \ .preempt_count = INIT_PREEMPT_COUNT, \ } -#ifdef CONFIG_THREAD_INFO_IN_TASK -#define INIT_THREAD_INFO_TASK(tsk) - static inline struct task_struct *thread_task(struct thread_info* ti) { return (struct task_struct *)ti; } -#else -#define INIT_THREAD_INFO_TASK(tsk) .task = &(tsk), - -static inline struct task_struct *thread_task(struct thread_info* ti) -{ - return ti->task; -} - -/* - * how to get the thread information struct from C - */ -static inline struct thread_info *current_thread_info(void) __attribute_const__; - -static inline struct thread_info *current_thread_info(void) -{ - return (struct thread_info *) - (current_stack_pointer & ~(THREAD_SIZE - 1)); -} -#endif - #define thread_saved_pc(tsk) \ ((unsigned long)(task_thread_info(tsk)->cpu_context.pc)) #define thread_saved_sp(tsk) \ diff --git a/arch/arm/kernel/asm-offsets.c b/arch/arm/kernel/asm-offsets.c index 645845e4982a..2c8d76fd7c66 100644 --- a/arch/arm/kernel/asm-offsets.c +++ b/arch/arm/kernel/asm-offsets.c @@ -43,9 +43,6 @@ int main(void) BLANK(); DEFINE(TI_FLAGS, offsetof(struct thread_info, flags)); DEFINE(TI_PREEMPT, offsetof(struct thread_info, preempt_count)); -#ifndef CONFIG_THREAD_INFO_IN_TASK - DEFINE(TI_TASK, offsetof(struct thread_info, task)); -#endif DEFINE(TI_CPU, offsetof(struct thread_info, cpu)); DEFINE(TI_CPU_DOMAIN, offsetof(struct thread_info, cpu_domain)); DEFINE(TI_CPU_SAVE, offsetof(struct thread_info, cpu_context)); diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 43d917f0d9a9..b58bda51e4b8 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -807,12 +807,13 @@ ENTRY(__switch_to) switch_tls r1, r4, r5, r3, r7 #if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) && \ !defined(CONFIG_STACKPROTECTOR_PER_TASK) - ldr r9, [r2, #TI_TASK] ldr r8, =__stack_chk_guard .if (TSK_STACK_CANARY > IMM12_MASK) - add r9, r9, #TSK_STACK_CANARY & ~IMM12_MASK - .endif + add r9, r2, #TSK_STACK_CANARY & ~IMM12_MASK ldr r9, [r9, #TSK_STACK_CANARY & IMM12_MASK] + .else + ldr r9, [r2, #TSK_STACK_CANARY & IMM12_MASK] + .endif #endif mov r7, r2 @ Preserve 'next' #ifdef CONFIG_CPU_USE_DOMAINS @@ -829,7 +830,7 @@ ENTRY(__switch_to) #endif mov r0, r5 #if !defined(CONFIG_THUMB2_KERNEL) && !defined(CONFIG_VMAP_STACK) - set_current r7 + set_current r7, r8 ldmia r4, {r4 - sl, fp, sp, pc} @ Load all regs saved previously #else mov r1, r7 @@ -851,7 +852,7 @@ ENTRY(__switch_to) @ switches us to another stack, with few other side effects. In order @ to prevent this distinction from causing any inconsistencies, let's @ keep the 'set_current' call as close as we can to the update of SP. - set_current r1 + set_current r1, r2 mov sp, ip ret lr #endif diff --git a/arch/arm/kernel/head-common.S b/arch/arm/kernel/head-common.S index da18e0a17dc2..42cae73fcc19 100644 --- a/arch/arm/kernel/head-common.S +++ b/arch/arm/kernel/head-common.S @@ -105,10 +105,8 @@ __mmap_switched: mov r1, #0 bl __memset @ clear .bss -#ifdef CONFIG_CURRENT_POINTER_IN_TPIDRURO adr_l r0, init_task @ get swapper task_struct - set_current r0 -#endif + set_current r0, r1 ldmia r4, {r0, r1, r2, r3} str r9, [r0] @ Save processor ID diff --git a/arch/arm/kernel/process.c b/arch/arm/kernel/process.c index d47159f3791c..0617af11377f 100644 --- a/arch/arm/kernel/process.c +++ b/arch/arm/kernel/process.c @@ -36,7 +36,7 @@ #include "signal.h" -#ifdef CONFIG_CURRENT_POINTER_IN_TPIDRURO +#if defined(CONFIG_CURRENT_POINTER_IN_TPIDRURO) || defined(CONFIG_SMP) DEFINE_PER_CPU(struct task_struct *, __entry_task); #endif @@ -46,6 +46,11 @@ unsigned long __stack_chk_guard __read_mostly; EXPORT_SYMBOL(__stack_chk_guard); #endif +#ifndef CONFIG_CURRENT_POINTER_IN_TPIDRURO +asmlinkage struct task_struct *__current; +EXPORT_SYMBOL(__current); +#endif + static const char *processor_modes[] __maybe_unused = { "USER_26", "FIQ_26" , "IRQ_26" , "SVC_26" , "UK4_26" , "UK5_26" , "UK6_26" , "UK7_26" , "UK8_26" , "UK9_26" , "UK10_26", "UK11_26", "UK12_26", "UK13_26", "UK14_26", "UK15_26", diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c index 9c55ca915ba4..951559e5bea3 100644 --- a/arch/arm/kernel/smp.c +++ b/arch/arm/kernel/smp.c @@ -403,6 +403,17 @@ static void smp_store_cpu_info(unsigned int cpuid) check_cpu_icache_size(cpuid); } +static void set_current(struct task_struct *cur) +{ + if (!IS_ENABLED(CONFIG_CURRENT_POINTER_IN_TPIDRURO) && !is_smp()) { + __current = cur; + return; + } + + /* Set TPIDRURO */ + asm("mcr p15, 0, %0, c13, c0, 3" :: "r"(cur) : "memory"); +} + /* * This is the secondary CPU boot entry. We're using this CPUs * idle thread stack, but a set of temporary page tables. diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c index b28a705c49cb..3f38357efc46 100644 --- a/arch/arm/kernel/traps.c +++ b/arch/arm/kernel/traps.c @@ -865,7 +865,9 @@ early_initcall(allocate_overflow_stacks); asmlinkage void handle_bad_stack(struct pt_regs *regs) { unsigned long tsk_stk = (unsigned long)current->stack; +#ifdef CONFIG_IRQSTACKS unsigned long irq_stk = (unsigned long)this_cpu_read(irq_stack_ptr); +#endif unsigned long ovf_stk = (unsigned long)this_cpu_read(overflow_stack_ptr); console_verbose(); @@ -873,8 +875,10 @@ asmlinkage void handle_bad_stack(struct pt_regs *regs) pr_emerg("Task stack: [0x%08lx..0x%08lx]\n", tsk_stk, tsk_stk + THREAD_SIZE); +#ifdef CONFIG_IRQSTACKS pr_emerg("IRQ stack: [0x%08lx..0x%08lx]\n", irq_stk - THREAD_SIZE, irq_stk); +#endif pr_emerg("Overflow stack: [0x%08lx..0x%08lx]\n", ovf_stk - OVERFLOW_STACK_SIZE, ovf_stk); -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH v2 12/12] ARM: implement THREAD_INFO_IN_TASK for uniprocessor systems 2021-11-30 12:59 ` [PATCH v2 12/12] ARM: implement THREAD_INFO_IN_TASK for uniprocessor systems Ard Biesheuvel @ 2021-11-30 13:43 ` Arnd Bergmann 2021-11-30 13:54 ` Ard Biesheuvel 0 siblings, 1 reply; 23+ messages in thread From: Arnd Bergmann @ 2021-11-30 13:43 UTC (permalink / raw) To: Ard Biesheuvel Cc: Linux ARM, Russell King - ARM Linux, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren On Tue, Nov 30, 2021 at 1:59 PM Ard Biesheuvel <ardb@kernel.org> wrote: > config IRQSTACKS > def_bool y > - depends on THREAD_INFO_IN_TASK > select HAVE_IRQ_EXIT_ON_IRQ_STACK > select HAVE_SOFTIRQ_ON_OWN_STACK Doesn't this still need a 'depends on MMU' or the corresponding patch to entry-v7m.S? I could only find your patch to entry-armv.S and it looks like you need the same thing for both. Arnd _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 12/12] ARM: implement THREAD_INFO_IN_TASK for uniprocessor systems 2021-11-30 13:43 ` Arnd Bergmann @ 2021-11-30 13:54 ` Ard Biesheuvel 0 siblings, 0 replies; 23+ messages in thread From: Ard Biesheuvel @ 2021-11-30 13:54 UTC (permalink / raw) To: Arnd Bergmann Cc: Linux ARM, Russell King - ARM Linux, Nicolas Pitre, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren On Tue, 30 Nov 2021 at 14:43, Arnd Bergmann <arnd@arndb.de> wrote: > > On Tue, Nov 30, 2021 at 1:59 PM Ard Biesheuvel <ardb@kernel.org> wrote: > > > config IRQSTACKS > > def_bool y > > - depends on THREAD_INFO_IN_TASK > > select HAVE_IRQ_EXIT_ON_IRQ_STACK > > select HAVE_SOFTIRQ_ON_OWN_STACK > > Doesn't this still need a 'depends on MMU' or the corresponding patch to > entry-v7m.S? I could only find your patch to entry-armv.S and it looks like > you need the same thing for both. > Yes, I suppose you're right. I'll add the 'depends on MMU' here. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP 2021-11-30 12:58 [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Ard Biesheuvel ` (11 preceding siblings ...) 2021-11-30 12:59 ` [PATCH v2 12/12] ARM: implement THREAD_INFO_IN_TASK for uniprocessor systems Ard Biesheuvel @ 2021-11-30 14:08 ` Arnd Bergmann 12 siblings, 0 replies; 23+ messages in thread From: Arnd Bergmann @ 2021-11-30 14:08 UTC (permalink / raw) To: Ard Biesheuvel Cc: Linux ARM, Russell King - ARM Linux, Nicolas Pitre, Arnd Bergmann, Kees Cook, Keith Packard, Linus Walleij, Nick Desaulniers, Tony Lindgren On Tue, Nov 30, 2021 at 1:58 PM Ard Biesheuvel <ardb@kernel.org> wrote: > > Enable the use of the TLS register to hold the 'current' pointer for all > configurations that can support it, including non-SMP ones that target > v6k or later CPUs, and multi-platform SMP ones that also support v6 > based UP systems. > > The remaining configurations are all strictly UP, which means we can > switch to a global variable to hold the current pointer. By doing this, > we can enable THREAD_INFO_IN_TASK, which moves thread info off the > stack, protecting it from overflows. It also permits us to enable IRQ > stacks and vmap'ed stacks for UP configurations as well. > > Supporting v6 cores without SMP extensions in SMP configurations (e.g., > omap2plus_defconfig or imx_v6_v7_defconfig) makes this a bit tricky, and > this is a feature we may consider dropping entirely in the future. But > for the time being, we can support this mode as well. > > The accesses to the global variable holding 'current' are constructed in > a way that ensures that no literal pool accesses (and associated D-cache > misses) are needed unless the access is from a module and module PLTs > are enabled. This means that accessing 'current' is just as costly as > before, as it used to require some arithmetic involving the stack > pointer and a load from the thread_info::task field. > > However, accessing thread_info itself now also involves a load, although > it should be noted that all thread_info and current accesses now go via > the same variable, which is therefore expected to be hot in the caches > at all times. Most of your assembler changes are beyond my understanding of that code, but everything that I do understand moves the code in the right direction, this looks really good overall. I'll add this to my randconfig compile test setup to see if that finds any corner cases you may have missed. Arnd _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2021-12-04 23:46 UTC | newest] Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-11-30 12:58 [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Ard Biesheuvel 2021-11-30 12:58 ` [PATCH v2 01/12] ARM: riscpc: use GENERIC_IRQ_MULTI_HANDLER Ard Biesheuvel 2021-11-30 12:58 ` [PATCH v2 02/12] ARM: footbridge: " Ard Biesheuvel 2021-12-04 23:42 ` Linus Walleij 2021-11-30 12:58 ` [PATCH v2 03/12] ARM: iop32x: offset IRQ numbers by 1 Ard Biesheuvel 2021-12-04 23:43 ` Linus Walleij 2021-11-30 12:58 ` [PATCH v2 04/12] ARM: iop32x: use GENERIC_IRQ_MULTI_HANDLER Ard Biesheuvel 2021-11-30 13:29 ` Arnd Bergmann 2021-11-30 12:58 ` [PATCH v2 05/12] ARM: remove old-style irq entry Ard Biesheuvel 2021-12-04 23:45 ` Linus Walleij 2021-11-30 12:58 ` [PATCH v2 06/12] ARM: entry: preserve thread_info pointer in switch_to Ard Biesheuvel 2021-11-30 12:58 ` [PATCH v2 07/12] ARM: module: implement support for PC-relative group relocations Ard Biesheuvel 2021-11-30 12:58 ` [PATCH v2 08/12] ARM: assembler: add optimized ldr/str macros to load variables from memory Ard Biesheuvel 2021-11-30 12:58 ` [PATCH v2 09/12] ARM: percpu: add SMP_ON_UP support Ard Biesheuvel 2021-11-30 15:12 ` Russell King (Oracle) 2021-11-30 15:45 ` Ard Biesheuvel 2021-11-30 12:58 ` [PATCH v2 10/12] ARM: use TLS register for 'current' on !SMP as well Ard Biesheuvel 2021-11-30 14:05 ` Arnd Bergmann 2021-11-30 12:59 ` [PATCH v2 11/12] ARM: smp: defer TPIDRURO update for SMP v6 configurations too Ard Biesheuvel 2021-11-30 12:59 ` [PATCH v2 12/12] ARM: implement THREAD_INFO_IN_TASK for uniprocessor systems Ard Biesheuvel 2021-11-30 13:43 ` Arnd Bergmann 2021-11-30 13:54 ` Ard Biesheuvel 2021-11-30 14:08 ` [PATCH v2 00/12] ARM: enable IRQ stacks and vmap'ed stacks for UP Arnd Bergmann
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.