* [PATCH 1/6] arm64: Implement stack trace termination record
2021-05-20 11:50 [PATCH 0/6] arm64: boot cleanups Mark Rutland
@ 2021-05-20 11:50 ` Mark Rutland
2021-05-20 11:50 ` [PATCH 2/6] arm64: assembler: add set_this_cpu_offset Mark Rutland
` (6 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Mark Rutland @ 2021-05-20 11:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: ardb, broonie, catalin.marinas, james.morse, madvenka,
mark.rutland, maz, suzuki.poulose, will
From: "Madhavan T. Venkataraman" <madvenka@linux.microsoft.com>
Reliable stacktracing requires that we identify when a stacktrace is
terminated early. We can do this by ensuring all tasks have a final
frame record at a known location on their task stack, and checking
that this is the final frame record in the chain.
We'd like to use task_pt_regs(task)->stackframe as the final frame
record, as this is already setup upon exception entry from EL0. For
kernel tasks we need to consistently reserve the pt_regs and point x29
at this, which we can do with small changes to __primary_switched,
__secondary_switched, and copy_process().
Since the final frame record must be at a specific location, we must
create the final frame record in __primary_switched and
__secondary_switched rather than leaving this to start_kernel and
secondary_start_kernel. Thus, __primary_switched and
__secondary_switched will now show up in stacktraces for the idle tasks.
Since the final frame record is now identified by its location rather
than by its contents, we identify it at the start of unwind_frame(),
before we read any values from it.
External debuggers may terminate the stack trace when FP == 0. In the
pt_regs->stackframe, the PC is 0 as well. So, stack traces taken in the
debugger may print an extra record 0x0 at the end. While this is not
pretty, this does not do any harm. This is a small price to pay for
having reliable stack trace termination in the kernel. That said, gdb
does not show the extra record probably because it uses DWARF and not
frame pointers for stack traces.
Signed-off-by: Madhavan T. Venkataraman <madvenka@linux.microsoft.com>
Reviewed-by: Mark Brown <broonie@kernel.org>
[Mark: rebase, use ASM_BUG(), update comments, update commit message]
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
---
arch/arm64/kernel/entry.S | 2 +-
arch/arm64/kernel/head.S | 25 +++++++++++++++++++------
arch/arm64/kernel/process.c | 5 +++++
arch/arm64/kernel/stacktrace.c | 16 +++++++---------
4 files changed, 32 insertions(+), 16 deletions(-)
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index 3513984a88bd..294f24e16fee 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -285,7 +285,7 @@ alternative_else_nop_endif
stp lr, x21, [sp, #S_LR]
/*
- * For exceptions from EL0, create a terminal frame record.
+ * For exceptions from EL0, create a final frame record.
* For exceptions from EL1, create a synthetic frame record so the
* interrupted code shows up in the backtrace.
*/
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 96873dfa67fd..cc2d45d54838 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -16,6 +16,7 @@
#include <asm/asm_pointer_auth.h>
#include <asm/assembler.h>
#include <asm/boot.h>
+#include <asm/bug.h>
#include <asm/ptrace.h>
#include <asm/asm-offsets.h>
#include <asm/cache.h>
@@ -393,6 +394,18 @@ SYM_FUNC_START_LOCAL(__create_page_tables)
ret x28
SYM_FUNC_END(__create_page_tables)
+ /*
+ * Create a final frame record at task_pt_regs(current)->stackframe, so
+ * that the unwinder can identify the final frame record of any task by
+ * its location in the task stack. We reserve the entire pt_regs space
+ * for consistency with user tasks and kthreads.
+ */
+ .macro setup_final_frame
+ sub sp, sp, #PT_REGS_SIZE
+ stp xzr, xzr, [sp, #S_STACKFRAME]
+ add x29, sp, #S_STACKFRAME
+ .endm
+
/*
* The following fragment of code is executed with the MMU enabled.
*
@@ -447,9 +460,9 @@ SYM_FUNC_START_LOCAL(__primary_switched)
#endif
bl switch_to_vhe // Prefer VHE if possible
add sp, sp, #16
- mov x29, #0
- mov x30, #0
- b start_kernel
+ setup_final_frame
+ bl start_kernel
+ ASM_BUG()
SYM_FUNC_END(__primary_switched)
.pushsection ".rodata", "a"
@@ -639,14 +652,14 @@ SYM_FUNC_START_LOCAL(__secondary_switched)
cbz x2, __secondary_too_slow
msr sp_el0, x2
scs_load x2, x3
- mov x29, #0
- mov x30, #0
+ setup_final_frame
#ifdef CONFIG_ARM64_PTR_AUTH
ptrauth_keys_init_cpu x2, x3, x4, x5
#endif
- b secondary_start_kernel
+ bl secondary_start_kernel
+ ASM_BUG()
SYM_FUNC_END(__secondary_switched)
SYM_FUNC_START_LOCAL(__secondary_too_slow)
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index b4bb67f17a2c..8928fba54e4b 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -435,6 +435,11 @@ int copy_thread(unsigned long clone_flags, unsigned long stack_start,
}
p->thread.cpu_context.pc = (unsigned long)ret_from_fork;
p->thread.cpu_context.sp = (unsigned long)childregs;
+ /*
+ * For the benefit of the unwinder, set up childregs->stackframe
+ * as the final frame for the new task.
+ */
+ p->thread.cpu_context.fp = (unsigned long)childregs->stackframe;
ptrace_hw_copy_thread(p);
diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c
index de07147a7926..36cf05d5eb9e 100644
--- a/arch/arm64/kernel/stacktrace.c
+++ b/arch/arm64/kernel/stacktrace.c
@@ -68,12 +68,16 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame)
unsigned long fp = frame->fp;
struct stack_info info;
- if (fp & 0xf)
- return -EINVAL;
-
if (!tsk)
tsk = current;
+ /* Final frame; nothing to unwind */
+ if (fp == (unsigned long)task_pt_regs(tsk)->stackframe)
+ return -ENOENT;
+
+ if (fp & 0xf)
+ return -EINVAL;
+
if (!on_accessible_stack(tsk, fp, &info))
return -EINVAL;
@@ -128,12 +132,6 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame)
frame->pc = ptrauth_strip_insn_pac(frame->pc);
- /*
- * This is a terminal record, so we have finished unwinding.
- */
- if (!frame->fp && !frame->pc)
- return -ENOENT;
-
return 0;
}
NOKPROBE_SYMBOL(unwind_frame);
--
2.11.0
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 2/6] arm64: assembler: add set_this_cpu_offset
2021-05-20 11:50 [PATCH 0/6] arm64: boot cleanups Mark Rutland
2021-05-20 11:50 ` [PATCH 1/6] arm64: Implement stack trace termination record Mark Rutland
@ 2021-05-20 11:50 ` Mark Rutland
2021-05-20 11:50 ` [PATCH 3/6] arm64: smp: remove pointless secondary_data maintenance Mark Rutland
` (5 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Mark Rutland @ 2021-05-20 11:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: ardb, broonie, catalin.marinas, james.morse, madvenka,
mark.rutland, maz, suzuki.poulose, will
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Suzuki Poulose <suzuki.poulose@arm.com>
Cc: Will Deacon <will@kernel.org>
---
arch/arm64/include/asm/assembler.h | 18 +++++++++++++-----
arch/arm64/mm/proc.S | 12 ++----------
2 files changed, 15 insertions(+), 15 deletions(-)
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 8418c1bd8f04..f0188903557f 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -232,17 +232,25 @@ lr .req x30 // link register
* @dst: destination register
*/
#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
- .macro this_cpu_offset, dst
+ .macro get_this_cpu_offset, dst
mrs \dst, tpidr_el2
.endm
#else
- .macro this_cpu_offset, dst
+ .macro get_this_cpu_offset, dst
alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
mrs \dst, tpidr_el1
alternative_else
mrs \dst, tpidr_el2
alternative_endif
.endm
+
+ .macro set_this_cpu_offset, src
+alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
+ msr tpidr_el1, \src
+alternative_else
+ msr tpidr_el2, \src
+alternative_endif
+ .endm
#endif
/*
@@ -253,7 +261,7 @@ alternative_endif
.macro adr_this_cpu, dst, sym, tmp
adrp \tmp, \sym
add \dst, \tmp, #:lo12:\sym
- this_cpu_offset \tmp
+ get_this_cpu_offset \tmp
add \dst, \dst, \tmp
.endm
@@ -264,7 +272,7 @@ alternative_endif
*/
.macro ldr_this_cpu dst, sym, tmp
adr_l \dst, \sym
- this_cpu_offset \tmp
+ get_this_cpu_offset \tmp
ldr \dst, [\dst, \tmp]
.endm
@@ -745,7 +753,7 @@ USER(\label, ic ivau, \tmp2) // invalidate I line PoU
cbz \tmp, \lbl
#endif
adr_l \tmp, irq_stat + IRQ_CPUSTAT_SOFTIRQ_PENDING
- this_cpu_offset \tmp2
+ get_this_cpu_offset \tmp2
ldr w\tmp, [\tmp, \tmp2]
cbnz w\tmp, \lbl // yield on pending softirq in task context
.Lnoyield_\@:
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 0a48191534ff..bb64ca6544a7 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -83,11 +83,7 @@ SYM_FUNC_START(cpu_do_suspend)
mrs x9, mdscr_el1
mrs x10, oslsr_el1
mrs x11, sctlr_el1
-alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
- mrs x12, tpidr_el1
-alternative_else
- mrs x12, tpidr_el2
-alternative_endif
+ get_this_cpu_offset x12
mrs x13, sp_el0
stp x2, x3, [x0]
stp x4, x5, [x0, #16]
@@ -145,11 +141,7 @@ SYM_FUNC_START(cpu_do_resume)
msr mdscr_el1, x10
msr sctlr_el1, x12
-alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
- msr tpidr_el1, x13
-alternative_else
- msr tpidr_el2, x13
-alternative_endif
+ set_this_cpu_offset x13
msr sp_el0, x14
/*
* Restore oslsr_el1 by writing oslar_el1
--
2.11.0
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 3/6] arm64: smp: remove pointless secondary_data maintenance
2021-05-20 11:50 [PATCH 0/6] arm64: boot cleanups Mark Rutland
2021-05-20 11:50 ` [PATCH 1/6] arm64: Implement stack trace termination record Mark Rutland
2021-05-20 11:50 ` [PATCH 2/6] arm64: assembler: add set_this_cpu_offset Mark Rutland
@ 2021-05-20 11:50 ` Mark Rutland
2021-05-20 11:50 ` [PATCH 4/6] arm64: smp: remove stack from secondary_data Mark Rutland
` (4 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Mark Rutland @ 2021-05-20 11:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: ardb, broonie, catalin.marinas, james.morse, madvenka,
mark.rutland, maz, suzuki.poulose, will
All reads and writes of secondary_data occur with the MMU on, using
coherent attributes, so there's no need to perform any cache maintenance
for this.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Suzuki Poulose <suzuki.poulose@arm.com>
Cc: Will Deacon <will@kernel.org>
---
arch/arm64/kernel/smp.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index dcd7041b2b07..92e83e8bac94 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -122,7 +122,6 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
secondary_data.task = idle;
secondary_data.stack = task_stack_page(idle) + THREAD_SIZE;
update_cpu_boot_status(CPU_MMU_OFF);
- __flush_dcache_area(&secondary_data, sizeof(secondary_data));
/* Now bring the CPU into our world */
ret = boot_secondary(cpu, idle);
@@ -143,7 +142,6 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
pr_crit("CPU%u: failed to come online\n", cpu);
secondary_data.task = NULL;
secondary_data.stack = NULL;
- __flush_dcache_area(&secondary_data, sizeof(secondary_data));
status = READ_ONCE(secondary_data.status);
if (status == CPU_MMU_OFF)
status = READ_ONCE(__early_cpu_boot_status);
--
2.11.0
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 4/6] arm64: smp: remove stack from secondary_data
2021-05-20 11:50 [PATCH 0/6] arm64: boot cleanups Mark Rutland
` (2 preceding siblings ...)
2021-05-20 11:50 ` [PATCH 3/6] arm64: smp: remove pointless secondary_data maintenance Mark Rutland
@ 2021-05-20 11:50 ` Mark Rutland
2021-05-20 11:50 ` [PATCH 5/6] arm64: smp: unify task and sp setup Mark Rutland
` (3 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Mark Rutland @ 2021-05-20 11:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: ardb, broonie, catalin.marinas, james.morse, madvenka,
mark.rutland, maz, suzuki.poulose, will
When we boot a secondary CPU, we pass it a task and a stack to use. As
the stack is always the task's stack, which can be derived from the
task, let's have the secondary CPU derive this itself and avoid passing
redundant information.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Suzuki Poulose <suzuki.poulose@arm.com>
Cc: Will Deacon <will@kernel.org>
---
arch/arm64/include/asm/smp.h | 2 --
arch/arm64/kernel/asm-offsets.c | 1 -
arch/arm64/kernel/head.S | 7 ++++---
arch/arm64/kernel/smp.c | 2 --
4 files changed, 4 insertions(+), 8 deletions(-)
diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h
index 0e357757c0cc..fc55f5a57a06 100644
--- a/arch/arm64/include/asm/smp.h
+++ b/arch/arm64/include/asm/smp.h
@@ -73,12 +73,10 @@ asmlinkage void secondary_start_kernel(void);
/*
* Initial data for bringing up a secondary CPU.
- * @stack - sp for the secondary CPU
* @status - Result passed back from the secondary CPU to
* indicate failure.
*/
struct secondary_data {
- void *stack;
struct task_struct *task;
long status;
};
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 0cb34ccb6e73..4a5e204c33af 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -99,7 +99,6 @@ int main(void)
DEFINE(SOFTIRQ_SHIFT, SOFTIRQ_SHIFT);
DEFINE(IRQ_CPUSTAT_SOFTIRQ_PENDING, offsetof(irq_cpustat_t, __softirq_pending));
BLANK();
- DEFINE(CPU_BOOT_STACK, offsetof(struct secondary_data, stack));
DEFINE(CPU_BOOT_TASK, offsetof(struct secondary_data, task));
BLANK();
DEFINE(FTR_OVR_VAL_OFFSET, offsetof(struct arm64_ftr_override, val));
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index cc2d45d54838..9be95e11367d 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -645,11 +645,12 @@ SYM_FUNC_START_LOCAL(__secondary_switched)
isb
adr_l x0, secondary_data
- ldr x1, [x0, #CPU_BOOT_STACK] // get secondary_data.stack
- cbz x1, __secondary_too_slow
- mov sp, x1
ldr x2, [x0, #CPU_BOOT_TASK]
cbz x2, __secondary_too_slow
+
+ ldr x1, [x2, #TSK_STACK]
+ add sp, x1, #THREAD_SIZE
+
msr sp_el0, x2
scs_load x2, x3
setup_final_frame
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 92e83e8bac94..73625cc39574 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -120,7 +120,6 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
* page tables.
*/
secondary_data.task = idle;
- secondary_data.stack = task_stack_page(idle) + THREAD_SIZE;
update_cpu_boot_status(CPU_MMU_OFF);
/* Now bring the CPU into our world */
@@ -141,7 +140,6 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
pr_crit("CPU%u: failed to come online\n", cpu);
secondary_data.task = NULL;
- secondary_data.stack = NULL;
status = READ_ONCE(secondary_data.status);
if (status == CPU_MMU_OFF)
status = READ_ONCE(__early_cpu_boot_status);
--
2.11.0
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 5/6] arm64: smp: unify task and sp setup
2021-05-20 11:50 [PATCH 0/6] arm64: boot cleanups Mark Rutland
` (3 preceding siblings ...)
2021-05-20 11:50 ` [PATCH 4/6] arm64: smp: remove stack from secondary_data Mark Rutland
@ 2021-05-20 11:50 ` Mark Rutland
2021-05-20 11:50 ` [PATCH 6/6] arm64: smp: initialize cpu offset earlier Mark Rutland
` (2 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Mark Rutland @ 2021-05-20 11:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: ardb, broonie, catalin.marinas, james.morse, madvenka,
mark.rutland, maz, suzuki.poulose, will
Once we enable the MMU, we have to initialize:
* SP_EL0 to point at the active task
* SP to point at the active task's stack
* SCS_SP to point at the active task's shadow stack
For all tasks (including init_task), this information can be derived
from the task's task_struct.
Let's unify __primary_switched and __secondary_switched to consistently
acquire this information from the relevant task_struct. At the same
time, let's fold this together with initializing a task's final frame.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Suzuki Poulose <suzuki.poulose@arm.com>
Cc: Will Deacon <will@kernel.org>
---
arch/arm64/kernel/head.S | 33 +++++++++++++++------------------
1 file changed, 15 insertions(+), 18 deletions(-)
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 9be95e11367d..e83b2899dce5 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -395,15 +395,24 @@ SYM_FUNC_START_LOCAL(__create_page_tables)
SYM_FUNC_END(__create_page_tables)
/*
+ * Initialize CPU registers with task-specific and cpu-specific context.
+ *
* Create a final frame record at task_pt_regs(current)->stackframe, so
* that the unwinder can identify the final frame record of any task by
* its location in the task stack. We reserve the entire pt_regs space
* for consistency with user tasks and kthreads.
*/
- .macro setup_final_frame
+ .macro init_cpu_task tsk, tmp
+ msr sp_el0, \tsk
+
+ ldr \tmp, [\tsk, #TSK_STACK]
+ add sp, \tmp, #THREAD_SIZE
sub sp, sp, #PT_REGS_SIZE
+
stp xzr, xzr, [sp, #S_STACKFRAME]
add x29, sp, #S_STACKFRAME
+
+ scs_load \tsk, \tmp
.endm
/*
@@ -412,22 +421,16 @@ SYM_FUNC_END(__create_page_tables)
* x0 = __PHYS_OFFSET
*/
SYM_FUNC_START_LOCAL(__primary_switched)
- adrp x4, init_thread_union
- add sp, x4, #THREAD_SIZE
- adr_l x5, init_task
- msr sp_el0, x5 // Save thread_info
+ adr_l x4, init_task
+ init_cpu_task x4, x5
adr_l x8, vectors // load VBAR_EL1 with virtual
msr vbar_el1, x8 // vector table address
isb
- stp xzr, x30, [sp, #-16]!
+ stp x29, x30, [sp, #-16]!
mov x29, sp
-#ifdef CONFIG_SHADOW_CALL_STACK
- adr_l scs_sp, init_shadow_call_stack // Set shadow call stack
-#endif
-
str_l x21, __fdt_pointer, x5 // Save FDT pointer
ldr_l x4, kimage_vaddr // Save the offset between
@@ -459,8 +462,7 @@ SYM_FUNC_START_LOCAL(__primary_switched)
0:
#endif
bl switch_to_vhe // Prefer VHE if possible
- add sp, sp, #16
- setup_final_frame
+ ldp x29, x30, [sp], #16
bl start_kernel
ASM_BUG()
SYM_FUNC_END(__primary_switched)
@@ -648,12 +650,7 @@ SYM_FUNC_START_LOCAL(__secondary_switched)
ldr x2, [x0, #CPU_BOOT_TASK]
cbz x2, __secondary_too_slow
- ldr x1, [x2, #TSK_STACK]
- add sp, x1, #THREAD_SIZE
-
- msr sp_el0, x2
- scs_load x2, x3
- setup_final_frame
+ init_cpu_task x2, x1
#ifdef CONFIG_ARM64_PTR_AUTH
ptrauth_keys_init_cpu x2, x3, x4, x5
--
2.11.0
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 6/6] arm64: smp: initialize cpu offset earlier
2021-05-20 11:50 [PATCH 0/6] arm64: boot cleanups Mark Rutland
` (4 preceding siblings ...)
2021-05-20 11:50 ` [PATCH 5/6] arm64: smp: unify task and sp setup Mark Rutland
@ 2021-05-20 11:50 ` Mark Rutland
2021-05-20 14:46 ` [PATCH 0/6] arm64: boot cleanups Ard Biesheuvel
2021-05-26 22:16 ` Will Deacon
7 siblings, 0 replies; 10+ messages in thread
From: Mark Rutland @ 2021-05-20 11:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: ardb, broonie, catalin.marinas, james.morse, madvenka,
mark.rutland, maz, suzuki.poulose, will
Now that we have a consistent place to initialize CPU context registers
early in the boot path, let's also initialize the per-cpu offset here.
This makes the primary and secondary boot paths more consistent, and
allows for the use of per-cpu operations earlier, which will be
necessary for instrumentation with KCSAN.
Note that smp_prepare_boot_cpu() still needs to re-initialize CPU0's
offset as immediately prior to this the per-cpu areas may be
reallocated, and hence the boot-time offset may be stale. A comment is
added to make this clear.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Suzuki Poulose <suzuki.poulose@arm.com>
Cc: Will Deacon <will@kernel.org>
---
arch/arm64/kernel/asm-offsets.c | 1 +
arch/arm64/kernel/head.S | 17 +++++++++++------
arch/arm64/kernel/setup.c | 6 ------
arch/arm64/kernel/smp.c | 10 ++++++----
4 files changed, 18 insertions(+), 16 deletions(-)
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 4a5e204c33af..bd0fc23d8719 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -27,6 +27,7 @@
int main(void)
{
DEFINE(TSK_ACTIVE_MM, offsetof(struct task_struct, active_mm));
+ DEFINE(TSK_CPU, offsetof(struct task_struct, cpu));
BLANK();
DEFINE(TSK_TI_FLAGS, offsetof(struct task_struct, thread_info.flags));
DEFINE(TSK_TI_PREEMPT, offsetof(struct task_struct, thread_info.preempt_count));
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index e83b2899dce5..070ed53c049d 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -402,17 +402,22 @@ SYM_FUNC_END(__create_page_tables)
* its location in the task stack. We reserve the entire pt_regs space
* for consistency with user tasks and kthreads.
*/
- .macro init_cpu_task tsk, tmp
+ .macro init_cpu_task tsk, tmp1, tmp2
msr sp_el0, \tsk
- ldr \tmp, [\tsk, #TSK_STACK]
- add sp, \tmp, #THREAD_SIZE
+ ldr \tmp1, [\tsk, #TSK_STACK]
+ add sp, \tmp1, #THREAD_SIZE
sub sp, sp, #PT_REGS_SIZE
stp xzr, xzr, [sp, #S_STACKFRAME]
add x29, sp, #S_STACKFRAME
- scs_load \tsk, \tmp
+ scs_load \tsk, \tmp1
+
+ adr_l \tmp1, __per_cpu_offset
+ ldr w\tmp2, [\tsk, #TSK_CPU]
+ ldr \tmp1, [\tmp1, \tmp2, lsl #3]
+ set_this_cpu_offset \tmp1
.endm
/*
@@ -422,7 +427,7 @@ SYM_FUNC_END(__create_page_tables)
*/
SYM_FUNC_START_LOCAL(__primary_switched)
adr_l x4, init_task
- init_cpu_task x4, x5
+ init_cpu_task x4, x5, x6
adr_l x8, vectors // load VBAR_EL1 with virtual
msr vbar_el1, x8 // vector table address
@@ -650,7 +655,7 @@ SYM_FUNC_START_LOCAL(__secondary_switched)
ldr x2, [x0, #CPU_BOOT_TASK]
cbz x2, __secondary_too_slow
- init_cpu_task x2, x1
+ init_cpu_task x2, x1, x3
#ifdef CONFIG_ARM64_PTR_AUTH
ptrauth_keys_init_cpu x2, x3, x4, x5
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 61845c0821d9..b7a35a03e9b9 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -87,12 +87,6 @@ void __init smp_setup_processor_id(void)
u64 mpidr = read_cpuid_mpidr() & MPIDR_HWID_BITMASK;
set_cpu_logical_map(0, mpidr);
- /*
- * clear __my_cpu_offset on boot CPU to avoid hang caused by
- * using percpu variable early, for example, lockdep will
- * access percpu variable inside lock_release
- */
- set_my_cpu_offset(0);
pr_info("Booting Linux on physical CPU 0x%010lx [0x%08x]\n",
(unsigned long)mpidr, read_cpuid_id());
}
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 73625cc39574..2fe8fab886e2 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -198,10 +198,7 @@ asmlinkage notrace void secondary_start_kernel(void)
u64 mpidr = read_cpuid_mpidr() & MPIDR_HWID_BITMASK;
struct mm_struct *mm = &init_mm;
const struct cpu_operations *ops;
- unsigned int cpu;
-
- cpu = task_cpu(current);
- set_my_cpu_offset(per_cpu_offset(cpu));
+ unsigned int cpu = smp_processor_id();
/*
* All kernel threads share the same mm context; grab a
@@ -448,6 +445,11 @@ void __init smp_cpus_done(unsigned int max_cpus)
void __init smp_prepare_boot_cpu(void)
{
+ /*
+ * The runtime per-cpu areas have been allocated by
+ * setup_per_cpu_areas(), and CPU0's boot time per-cpu area will be
+ * freed shortly, so we must move over to the runtime per-cpu area.
+ */
set_my_cpu_offset(per_cpu_offset(smp_processor_id()));
cpuinfo_store_boot_cpu();
--
2.11.0
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [PATCH 0/6] arm64: boot cleanups
2021-05-20 11:50 [PATCH 0/6] arm64: boot cleanups Mark Rutland
` (5 preceding siblings ...)
2021-05-20 11:50 ` [PATCH 6/6] arm64: smp: initialize cpu offset earlier Mark Rutland
@ 2021-05-20 14:46 ` Ard Biesheuvel
2021-05-26 22:16 ` Will Deacon
7 siblings, 0 replies; 10+ messages in thread
From: Ard Biesheuvel @ 2021-05-20 14:46 UTC (permalink / raw)
To: Mark Rutland
Cc: Linux ARM, Mark Brown, Catalin Marinas, James Morse,
Madhavan T. Venkataraman, Marc Zyngier, Suzuki K Poulose,
Will Deacon
On Thu, 20 May 2021 at 13:50, Mark Rutland <mark.rutland@arm.com> wrote:
>
> This series (based on v5.13-rc1) reworks the way we initialize some state at
> boot time, simplifying and unifying the logic for primary and secondary CPUs.
> This allows us to initalize the per-cpu offsets earlier (which will help to
> enable KCSAN), and reduces the data we need to pass to a secondary. In future,
> this should allow us to transfer the secondary data atomically and make the
> secondary boot paths more robust to arbitrarily long delays.
>
> I've based this on Mahdavan's stacktrace termination patch [1] (duplicated here
> unchanged), since it made sense to combine the unwind initialization along with
> the other CPU state, and otherwise there would be non-trivial merge conflicts.
>
> I've given the series some boot testing with a variety of configurations,
> checking that stacktraces work correctly, etc.
>
> The series can be found on my arm64/boot/rework branch on kernel.org:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/boot/rework
> git://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git arm64/boot/rework
>
> Thanks,
> Mark.
>
> [1] https://lore.kernel.org/r/20210510110026.18061-1-mark.rutland@arm.com
>
> Madhavan T. Venkataraman (1):
> arm64: Implement stack trace termination record
>
> Mark Rutland (5):
> arm64: assembler: add set_this_cpu_offset
> arm64: smp: remove pointless secondary_data maintenance
> arm64: smp: remove stack from secondary_data
> arm64: smp: unify task and sp setup
> arm64: smp: initialize cpu offset earlier
>
For the series
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Note that this will conflict with Fuad's cache maintenance cleanup series.
> arch/arm64/include/asm/assembler.h | 18 ++++++++----
> arch/arm64/include/asm/smp.h | 2 --
> arch/arm64/kernel/asm-offsets.c | 2 +-
> arch/arm64/kernel/entry.S | 2 +-
> arch/arm64/kernel/head.S | 58 ++++++++++++++++++++++++--------------
> arch/arm64/kernel/process.c | 5 ++++
> arch/arm64/kernel/setup.c | 6 ----
> arch/arm64/kernel/smp.c | 14 ++++-----
> arch/arm64/kernel/stacktrace.c | 16 +++++------
> arch/arm64/mm/proc.S | 12 ++------
> 10 files changed, 72 insertions(+), 63 deletions(-)
>
> --
> 2.11.0
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH 0/6] arm64: boot cleanups
2021-05-20 11:50 [PATCH 0/6] arm64: boot cleanups Mark Rutland
` (6 preceding siblings ...)
2021-05-20 14:46 ` [PATCH 0/6] arm64: boot cleanups Ard Biesheuvel
@ 2021-05-26 22:16 ` Will Deacon
2021-05-27 9:33 ` Mark Rutland
7 siblings, 1 reply; 10+ messages in thread
From: Will Deacon @ 2021-05-26 22:16 UTC (permalink / raw)
To: Mark Rutland
Cc: linux-arm-kernel, ardb, broonie, catalin.marinas, james.morse,
madvenka, maz, suzuki.poulose
On Thu, May 20, 2021 at 12:50:25PM +0100, Mark Rutland wrote:
> This series (based on v5.13-rc1) reworks the way we initialize some state at
> boot time, simplifying and unifying the logic for primary and secondary CPUs.
> This allows us to initalize the per-cpu offsets earlier (which will help to
> enable KCSAN), and reduces the data we need to pass to a secondary. In future,
> this should allow us to transfer the secondary data atomically and make the
> secondary boot paths more robust to arbitrarily long delays.
>
> I've based this on Mahdavan's stacktrace termination patch [1] (duplicated here
> unchanged), since it made sense to combine the unwind initialization along with
> the other CPU state, and otherwise there would be non-trivial merge conflicts.
>
> I've given the series some boot testing with a variety of configurations,
> checking that stacktraces work correctly, etc.
Thanks. I've already got the stack trace patch queued on
for-next/stacktrace, so I've merged that branch into a new for-next/boot
branch and put the remainder of the patches on top of that.
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH 0/6] arm64: boot cleanups
2021-05-26 22:16 ` Will Deacon
@ 2021-05-27 9:33 ` Mark Rutland
0 siblings, 0 replies; 10+ messages in thread
From: Mark Rutland @ 2021-05-27 9:33 UTC (permalink / raw)
To: Will Deacon
Cc: linux-arm-kernel, ardb, broonie, catalin.marinas, james.morse,
madvenka, maz, suzuki.poulose
On Wed, May 26, 2021 at 11:16:42PM +0100, Will Deacon wrote:
> On Thu, May 20, 2021 at 12:50:25PM +0100, Mark Rutland wrote:
> > This series (based on v5.13-rc1) reworks the way we initialize some state at
> > boot time, simplifying and unifying the logic for primary and secondary CPUs.
> > This allows us to initalize the per-cpu offsets earlier (which will help to
> > enable KCSAN), and reduces the data we need to pass to a secondary. In future,
> > this should allow us to transfer the secondary data atomically and make the
> > secondary boot paths more robust to arbitrarily long delays.
> >
> > I've based this on Mahdavan's stacktrace termination patch [1] (duplicated here
> > unchanged), since it made sense to combine the unwind initialization along with
> > the other CPU state, and otherwise there would be non-trivial merge conflicts.
> >
> > I've given the series some boot testing with a variety of configurations,
> > checking that stacktraces work correctly, etc.
>
> Thanks. I've already got the stack trace patch queued on
> for-next/stacktrace, so I've merged that branch into a new for-next/boot
> branch and put the remainder of the patches on top of that.
Thanks!
I took a look and I see that you also handled the conflict with the
cache rework (where this series deltetes some maintenance); the result
looks good to me.
Mark.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 10+ messages in thread