From: Mark Rutland <mark.rutland@arm.com> To: linux-arm-kernel@lists.infradead.org Cc: ardb@kernel.org, catalin.marinas@arm.com, frederic@kernel.org, juri.lelli@redhat.com, linux-kernel@vger.kernel.org, mark.rutland@arm.com, mingo@redhat.com, peterz@infradead.org, will@kernel.org Subject: [PATCH 6/6] arm64: support PREEMPT_DYNAMIC Date: Tue, 9 Nov 2021 17:24:08 +0000 [thread overview] Message-ID: <20211109172408.49641-7-mark.rutland@arm.com> (raw) In-Reply-To: <20211109172408.49641-1-mark.rutland@arm.com> This patch enables support for PREEMPT_DYNAMIC on arm64, using static keys to disable the out-of-line forms of the preemption functions. This side-steps a number of pain points highlighted with static calls, and I expect the overhead to be similar (and hopefully lower in some cases) than would be the case for non-inlined static calls, by virtue of effectively inlining the trampolines into the callees. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Cc: Ard Biesheuvel <ardb@kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Frederic Weisbecker <frederic@kernel.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: Juri Lelli <juri.lelli@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Will Deacon <will@kernel.org> --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/preempt.h | 16 ++++++++++++++-- arch/arm64/kernel/entry-common.c | 9 +++++++++ 3 files changed, 24 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index c4207cf9bb17..928b710465ce 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -191,6 +191,7 @@ config ARM64 select HAVE_PERF_EVENTS select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP + select HAVE_PREEMPT_DYNAMIC_KEY select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_POSIX_CPU_TIMERS_TASK_WORK select HAVE_FUNCTION_ARG_ACCESS_API diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/preempt.h index e83f0982b99c..73623f719d7e 100644 --- a/arch/arm64/include/asm/preempt.h +++ b/arch/arm64/include/asm/preempt.h @@ -80,10 +80,22 @@ static inline bool should_resched(int preempt_offset) } #ifdef CONFIG_PREEMPTION +#ifdef CONFIG_PREEMPT_DYNAMIC + +DECLARE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched); +void dynamic_preempt_schedule(void); +#define __preempt_schedule() dynamic_preempt_schedule() +void dynamic_preempt_schedule_notrace(void); +#define __preempt_schedule_notrace() dynamic_preempt_schedule_notrace() + +#else + void preempt_schedule(void); -#define __preempt_schedule() preempt_schedule() +#define __preempt_schedule() preempt_schedule() void preempt_schedule_notrace(void); -#define __preempt_schedule_notrace() preempt_schedule_notrace() +#define __preempt_schedule_notrace() preempt_schedule_notrace() + +#endif #endif /* CONFIG_PREEMPTION */ #endif /* __ASM_PREEMPT_H */ diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index f7408edf8571..f096678dd580 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -220,10 +220,19 @@ static void noinstr arm64_exit_el1_dbg(struct pt_regs *regs) lockdep_hardirqs_on(CALLER_ADDR0); } +#ifdef CONFIG_PREEMPT_DYNAMIC +DEFINE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched); +#endif + static void __sched arm64_preempt_schedule_irq(void) { lockdep_assert_irqs_disabled(); +#ifdef CONFIG_PREEMPT_DYNAMIC + if (!static_branch_unlikely(&sk_dynamic_irqentry_exit_cond_resched)) + return; +#endif + /* * DAIF.DA are cleared at the start of IRQ/FIQ handling, and when GIC * priority masking is used the GIC irqchip driver will clear DAIF.IF -- 2.11.0
WARNING: multiple messages have this Message-ID (diff)
From: Mark Rutland <mark.rutland@arm.com> To: linux-arm-kernel@lists.infradead.org Cc: ardb@kernel.org, catalin.marinas@arm.com, frederic@kernel.org, juri.lelli@redhat.com, linux-kernel@vger.kernel.org, mark.rutland@arm.com, mingo@redhat.com, peterz@infradead.org, will@kernel.org Subject: [PATCH 6/6] arm64: support PREEMPT_DYNAMIC Date: Tue, 9 Nov 2021 17:24:08 +0000 [thread overview] Message-ID: <20211109172408.49641-7-mark.rutland@arm.com> (raw) In-Reply-To: <20211109172408.49641-1-mark.rutland@arm.com> This patch enables support for PREEMPT_DYNAMIC on arm64, using static keys to disable the out-of-line forms of the preemption functions. This side-steps a number of pain points highlighted with static calls, and I expect the overhead to be similar (and hopefully lower in some cases) than would be the case for non-inlined static calls, by virtue of effectively inlining the trampolines into the callees. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Cc: Ard Biesheuvel <ardb@kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Frederic Weisbecker <frederic@kernel.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: Juri Lelli <juri.lelli@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Will Deacon <will@kernel.org> --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/preempt.h | 16 ++++++++++++++-- arch/arm64/kernel/entry-common.c | 9 +++++++++ 3 files changed, 24 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index c4207cf9bb17..928b710465ce 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -191,6 +191,7 @@ config ARM64 select HAVE_PERF_EVENTS select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP + select HAVE_PREEMPT_DYNAMIC_KEY select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_POSIX_CPU_TIMERS_TASK_WORK select HAVE_FUNCTION_ARG_ACCESS_API diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/preempt.h index e83f0982b99c..73623f719d7e 100644 --- a/arch/arm64/include/asm/preempt.h +++ b/arch/arm64/include/asm/preempt.h @@ -80,10 +80,22 @@ static inline bool should_resched(int preempt_offset) } #ifdef CONFIG_PREEMPTION +#ifdef CONFIG_PREEMPT_DYNAMIC + +DECLARE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched); +void dynamic_preempt_schedule(void); +#define __preempt_schedule() dynamic_preempt_schedule() +void dynamic_preempt_schedule_notrace(void); +#define __preempt_schedule_notrace() dynamic_preempt_schedule_notrace() + +#else + void preempt_schedule(void); -#define __preempt_schedule() preempt_schedule() +#define __preempt_schedule() preempt_schedule() void preempt_schedule_notrace(void); -#define __preempt_schedule_notrace() preempt_schedule_notrace() +#define __preempt_schedule_notrace() preempt_schedule_notrace() + +#endif #endif /* CONFIG_PREEMPTION */ #endif /* __ASM_PREEMPT_H */ diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index f7408edf8571..f096678dd580 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -220,10 +220,19 @@ static void noinstr arm64_exit_el1_dbg(struct pt_regs *regs) lockdep_hardirqs_on(CALLER_ADDR0); } +#ifdef CONFIG_PREEMPT_DYNAMIC +DEFINE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched); +#endif + static void __sched arm64_preempt_schedule_irq(void) { lockdep_assert_irqs_disabled(); +#ifdef CONFIG_PREEMPT_DYNAMIC + if (!static_branch_unlikely(&sk_dynamic_irqentry_exit_cond_resched)) + return; +#endif + /* * DAIF.DA are cleared at the start of IRQ/FIQ handling, and when GIC * priority masking is used the GIC irqchip driver will clear DAIF.IF -- 2.11.0 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2021-11-09 17:24 UTC|newest] Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-11-09 17:24 [PATCH 0/6] arm64 / sched/preempt: support PREEMPT_DYNAMIC with static keys Mark Rutland 2021-11-09 17:24 ` Mark Rutland 2021-11-09 17:24 ` [PATCH 1/6] sched/preempt: move PREEMPT_DYNAMIC logic later Mark Rutland 2021-11-09 17:24 ` Mark Rutland 2021-11-09 17:24 ` [PATCH 2/6] sched/preempt: refactor sched_dynamic_update() Mark Rutland 2021-11-09 17:24 ` Mark Rutland 2021-12-10 15:13 ` Frederic Weisbecker 2021-12-10 15:13 ` Frederic Weisbecker 2022-02-02 15:13 ` Mark Rutland 2022-02-02 15:13 ` Mark Rutland 2022-02-02 16:01 ` Frederic Weisbecker 2022-02-02 16:01 ` Frederic Weisbecker 2022-02-02 18:08 ` Mark Rutland 2022-02-02 18:08 ` Mark Rutland 2022-02-03 11:52 ` Frederic Weisbecker 2022-02-03 11:52 ` Frederic Weisbecker 2021-11-09 17:24 ` [PATCH 3/6] sched/preempt: simplify irqentry_exit_cond_resched() callers Mark Rutland 2021-11-09 17:24 ` Mark Rutland 2021-11-09 17:24 ` [PATCH 4/6] sched/preempt: decouple HAVE_PREEMPT_DYNAMIC from GENERIC_ENTRY Mark Rutland 2021-11-09 17:24 ` Mark Rutland 2021-11-09 17:24 ` [PATCH 5/6] sched/preempt: add PREEMPT_DYNAMIC using static keys Mark Rutland 2021-11-09 17:24 ` Mark Rutland 2021-12-13 22:05 ` Frederic Weisbecker 2021-12-13 22:05 ` Frederic Weisbecker 2022-02-02 15:29 ` Mark Rutland 2022-02-02 15:29 ` Mark Rutland 2022-02-03 22:40 ` Peter Zijlstra 2022-02-03 22:40 ` Peter Zijlstra 2022-02-02 23:21 ` Frederic Weisbecker 2022-02-02 23:21 ` Frederic Weisbecker 2022-02-03 9:51 ` Mark Rutland 2022-02-03 9:51 ` Mark Rutland 2022-02-03 11:34 ` Frederic Weisbecker 2022-02-03 11:34 ` Frederic Weisbecker 2022-02-03 12:27 ` Mark Rutland 2022-02-03 12:27 ` Mark Rutland 2021-11-09 17:24 ` Mark Rutland [this message] 2021-11-09 17:24 ` [PATCH 6/6] arm64: support PREEMPT_DYNAMIC Mark Rutland
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20211109172408.49641-7-mark.rutland@arm.com \ --to=mark.rutland@arm.com \ --cc=ardb@kernel.org \ --cc=catalin.marinas@arm.com \ --cc=frederic@kernel.org \ --cc=juri.lelli@redhat.com \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=mingo@redhat.com \ --cc=peterz@infradead.org \ --cc=will@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.