* [PATCH 0/4] MIPS: Restartable sequences (rseq) support @ 2018-06-14 23:52 ` Paul Burton 0 siblings, 0 replies; 18+ messages in thread From: Paul Burton @ 2018-06-14 23:52 UTC (permalink / raw) To: linux-mips Cc: Peter Zijlstra, James Hogan, linux-kernel, Mathieu Desnoyers, Boqun Feng, Paul E . McKenney, Ralf Baechle, Paul Burton This series implements MIPS support for restartable sequences, hooks up the rseq syscall & implements MIPS support in the rseq selftests. Applies atop Linus' master as of 2837461dbe6f ("Merge tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi"). Thanks, Paul Paul Burton (4): MIPS: Add support for restartable sequences MIPS: Add syscall detection for restartable sequences MIPS: Wire up the restartable sequences (rseq) syscall rseq/selftests: Implement MIPS support arch/mips/Kconfig | 1 + arch/mips/include/uapi/asm/unistd.h | 15 +- arch/mips/kernel/entry.S | 8 + arch/mips/kernel/scall32-o32.S | 1 + arch/mips/kernel/scall64-64.S | 1 + arch/mips/kernel/scall64-n32.S | 1 + arch/mips/kernel/scall64-o32.S | 1 + arch/mips/kernel/signal.c | 3 + tools/testing/selftests/rseq/param_test.c | 24 + tools/testing/selftests/rseq/rseq-mips.h | 725 ++++++++++++++++++++++ tools/testing/selftests/rseq/rseq.h | 2 + 11 files changed, 776 insertions(+), 6 deletions(-) create mode 100644 tools/testing/selftests/rseq/rseq-mips.h -- 2.17.1 ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 0/4] MIPS: Restartable sequences (rseq) support @ 2018-06-14 23:52 ` Paul Burton 0 siblings, 0 replies; 18+ messages in thread From: Paul Burton @ 2018-06-14 23:52 UTC (permalink / raw) To: linux-mips Cc: Peter Zijlstra, James Hogan, linux-kernel, Mathieu Desnoyers, Boqun Feng, Paul E . McKenney, Ralf Baechle, Paul Burton This series implements MIPS support for restartable sequences, hooks up the rseq syscall & implements MIPS support in the rseq selftests. Applies atop Linus' master as of 2837461dbe6f ("Merge tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi"). Thanks, Paul Paul Burton (4): MIPS: Add support for restartable sequences MIPS: Add syscall detection for restartable sequences MIPS: Wire up the restartable sequences (rseq) syscall rseq/selftests: Implement MIPS support arch/mips/Kconfig | 1 + arch/mips/include/uapi/asm/unistd.h | 15 +- arch/mips/kernel/entry.S | 8 + arch/mips/kernel/scall32-o32.S | 1 + arch/mips/kernel/scall64-64.S | 1 + arch/mips/kernel/scall64-n32.S | 1 + arch/mips/kernel/scall64-o32.S | 1 + arch/mips/kernel/signal.c | 3 + tools/testing/selftests/rseq/param_test.c | 24 + tools/testing/selftests/rseq/rseq-mips.h | 725 ++++++++++++++++++++++ tools/testing/selftests/rseq/rseq.h | 2 + 11 files changed, 776 insertions(+), 6 deletions(-) create mode 100644 tools/testing/selftests/rseq/rseq-mips.h -- 2.17.1 ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 1/4] MIPS: Add support for restartable sequences @ 2018-06-14 23:52 ` Paul Burton 0 siblings, 0 replies; 18+ messages in thread From: Paul Burton @ 2018-06-14 23:52 UTC (permalink / raw) To: linux-mips Cc: Peter Zijlstra, James Hogan, linux-kernel, Mathieu Desnoyers, Boqun Feng, Paul E . McKenney, Ralf Baechle, Paul Burton Implement support for restartable sequences on MIPS, which requires 3 simple things: - Call rseq_handle_notify_resume() on return to userspace if TIF_NOTIFY_RESUME is set. - Call rseq_signal_deliver() to fixup the pre-signal stack frame when a signal is delivered whilst executing a restartable sequence critical section. - Select CONFIG_HAVE_RSEQ. Signed-off-by: Paul Burton <paul.burton@mips.com> Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: linux-mips@linux-mips.org Cc: linux-kernel@vger.kernel.org --- arch/mips/Kconfig | 1 + arch/mips/kernel/signal.c | 3 +++ 2 files changed, 4 insertions(+) diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig index fe98e459a416..afe2b0c867ac 100644 --- a/arch/mips/Kconfig +++ b/arch/mips/Kconfig @@ -66,6 +66,7 @@ config MIPS select HAVE_OPROFILE select HAVE_PERF_EVENTS select HAVE_REGS_AND_STACK_ACCESS_API + select HAVE_RSEQ select HAVE_SYSCALL_TRACEPOINTS select HAVE_VIRT_CPU_ACCOUNTING_GEN if 64BIT || !SMP select IRQ_FORCED_THREADING diff --git a/arch/mips/kernel/signal.c b/arch/mips/kernel/signal.c index 9e224469c788..00f2535d2226 100644 --- a/arch/mips/kernel/signal.c +++ b/arch/mips/kernel/signal.c @@ -801,6 +801,8 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs) regs->regs[0] = 0; /* Don't deal with this again. */ } + rseq_signal_deliver(regs); + if (sig_uses_siginfo(&ksig->ka, abi)) ret = abi->setup_rt_frame(vdso + abi->vdso->off_rt_sigreturn, ksig, regs, oldset); @@ -868,6 +870,7 @@ asmlinkage void do_notify_resume(struct pt_regs *regs, void *unused, if (thread_info_flags & _TIF_NOTIFY_RESUME) { clear_thread_flag(TIF_NOTIFY_RESUME); tracehook_notify_resume(regs); + rseq_handle_notify_resume(regs); } user_enter(); -- 2.17.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 1/4] MIPS: Add support for restartable sequences @ 2018-06-14 23:52 ` Paul Burton 0 siblings, 0 replies; 18+ messages in thread From: Paul Burton @ 2018-06-14 23:52 UTC (permalink / raw) To: linux-mips Cc: Peter Zijlstra, James Hogan, linux-kernel, Mathieu Desnoyers, Boqun Feng, Paul E . McKenney, Ralf Baechle, Paul Burton Implement support for restartable sequences on MIPS, which requires 3 simple things: - Call rseq_handle_notify_resume() on return to userspace if TIF_NOTIFY_RESUME is set. - Call rseq_signal_deliver() to fixup the pre-signal stack frame when a signal is delivered whilst executing a restartable sequence critical section. - Select CONFIG_HAVE_RSEQ. Signed-off-by: Paul Burton <paul.burton@mips.com> Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: linux-mips@linux-mips.org Cc: linux-kernel@vger.kernel.org --- arch/mips/Kconfig | 1 + arch/mips/kernel/signal.c | 3 +++ 2 files changed, 4 insertions(+) diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig index fe98e459a416..afe2b0c867ac 100644 --- a/arch/mips/Kconfig +++ b/arch/mips/Kconfig @@ -66,6 +66,7 @@ config MIPS select HAVE_OPROFILE select HAVE_PERF_EVENTS select HAVE_REGS_AND_STACK_ACCESS_API + select HAVE_RSEQ select HAVE_SYSCALL_TRACEPOINTS select HAVE_VIRT_CPU_ACCOUNTING_GEN if 64BIT || !SMP select IRQ_FORCED_THREADING diff --git a/arch/mips/kernel/signal.c b/arch/mips/kernel/signal.c index 9e224469c788..00f2535d2226 100644 --- a/arch/mips/kernel/signal.c +++ b/arch/mips/kernel/signal.c @@ -801,6 +801,8 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs) regs->regs[0] = 0; /* Don't deal with this again. */ } + rseq_signal_deliver(regs); + if (sig_uses_siginfo(&ksig->ka, abi)) ret = abi->setup_rt_frame(vdso + abi->vdso->off_rt_sigreturn, ksig, regs, oldset); @@ -868,6 +870,7 @@ asmlinkage void do_notify_resume(struct pt_regs *regs, void *unused, if (thread_info_flags & _TIF_NOTIFY_RESUME) { clear_thread_flag(TIF_NOTIFY_RESUME); tracehook_notify_resume(regs); + rseq_handle_notify_resume(regs); } user_enter(); -- 2.17.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 2/4] MIPS: Add syscall detection for restartable sequences @ 2018-06-14 23:52 ` Paul Burton 0 siblings, 0 replies; 18+ messages in thread From: Paul Burton @ 2018-06-14 23:52 UTC (permalink / raw) To: linux-mips Cc: Peter Zijlstra, James Hogan, linux-kernel, Mathieu Desnoyers, Boqun Feng, Paul E . McKenney, Ralf Baechle, Paul Burton Syscalls are not allowed inside restartable sequences, so add a call to rseq_syscall() at the very beginning of the system call exit path when CONFIG_DEBUG_RSEQ=y. This will help us to detect whether there is a syscall issued erroneously inside a restartable sequence. Signed-off-by: Paul Burton <paul.burton@mips.com> Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: linux-mips@linux-mips.org Cc: linux-kernel@vger.kernel.org --- arch/mips/kernel/entry.S | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/mips/kernel/entry.S b/arch/mips/kernel/entry.S index 38a302919e6b..d7de8adcfcc8 100644 --- a/arch/mips/kernel/entry.S +++ b/arch/mips/kernel/entry.S @@ -79,6 +79,10 @@ FEXPORT(ret_from_fork) jal schedule_tail # a0 = struct task_struct *prev FEXPORT(syscall_exit) +#ifdef CONFIG_DEBUG_RSEQ + move a0, sp + jal rseq_syscall +#endif local_irq_disable # make sure need_resched and # signals dont change between # sampling and return @@ -141,6 +145,10 @@ work_notifysig: # deal with pending signals and j resume_userspace_check FEXPORT(syscall_exit_partial) +#ifdef CONFIG_DEBUG_RSEQ + move a0, sp + jal rseq_syscall +#endif local_irq_disable # make sure need_resched doesn't # change between and return LONG_L a2, TI_FLAGS($28) # current->work -- 2.17.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 2/4] MIPS: Add syscall detection for restartable sequences @ 2018-06-14 23:52 ` Paul Burton 0 siblings, 0 replies; 18+ messages in thread From: Paul Burton @ 2018-06-14 23:52 UTC (permalink / raw) To: linux-mips Cc: Peter Zijlstra, James Hogan, linux-kernel, Mathieu Desnoyers, Boqun Feng, Paul E . McKenney, Ralf Baechle, Paul Burton Syscalls are not allowed inside restartable sequences, so add a call to rseq_syscall() at the very beginning of the system call exit path when CONFIG_DEBUG_RSEQ=y. This will help us to detect whether there is a syscall issued erroneously inside a restartable sequence. Signed-off-by: Paul Burton <paul.burton@mips.com> Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: linux-mips@linux-mips.org Cc: linux-kernel@vger.kernel.org --- arch/mips/kernel/entry.S | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/mips/kernel/entry.S b/arch/mips/kernel/entry.S index 38a302919e6b..d7de8adcfcc8 100644 --- a/arch/mips/kernel/entry.S +++ b/arch/mips/kernel/entry.S @@ -79,6 +79,10 @@ FEXPORT(ret_from_fork) jal schedule_tail # a0 = struct task_struct *prev FEXPORT(syscall_exit) +#ifdef CONFIG_DEBUG_RSEQ + move a0, sp + jal rseq_syscall +#endif local_irq_disable # make sure need_resched and # signals dont change between # sampling and return @@ -141,6 +145,10 @@ work_notifysig: # deal with pending signals and j resume_userspace_check FEXPORT(syscall_exit_partial) +#ifdef CONFIG_DEBUG_RSEQ + move a0, sp + jal rseq_syscall +#endif local_irq_disable # make sure need_resched doesn't # change between and return LONG_L a2, TI_FLAGS($28) # current->work -- 2.17.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH 2/4] MIPS: Add syscall detection for restartable sequences 2018-06-14 23:52 ` Paul Burton (?) @ 2018-06-15 17:41 ` Mathieu Desnoyers 2018-06-15 18:43 ` Paul Burton -1 siblings, 1 reply; 18+ messages in thread From: Mathieu Desnoyers @ 2018-06-15 17:41 UTC (permalink / raw) To: Paul Burton Cc: linux-mips, Peter Zijlstra, James Hogan, linux-kernel, Boqun Feng, Paul E. McKenney, Ralf Baechle ----- On Jun 14, 2018, at 7:52 PM, Paul Burton paul.burton@mips.com wrote: > Syscalls are not allowed inside restartable sequences, so add a call to > rseq_syscall() at the very beginning of the system call exit path when > CONFIG_DEBUG_RSEQ=y. This will help us to detect whether there is a > syscall issued erroneously inside a restartable sequence. > > Signed-off-by: Paul Burton <paul.burton@mips.com> > Cc: James Hogan <jhogan@kernel.org> > Cc: Ralf Baechle <ralf@linux-mips.org> > Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> > Cc: Peter Zijlstra <peterz@infradead.org> > Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> > Cc: Boqun Feng <boqun.feng@gmail.com> > Cc: linux-mips@linux-mips.org > Cc: linux-kernel@vger.kernel.org > --- > > arch/mips/kernel/entry.S | 8 ++++++++ > 1 file changed, 8 insertions(+) > > diff --git a/arch/mips/kernel/entry.S b/arch/mips/kernel/entry.S > index 38a302919e6b..d7de8adcfcc8 100644 > --- a/arch/mips/kernel/entry.S > +++ b/arch/mips/kernel/entry.S > @@ -79,6 +79,10 @@ FEXPORT(ret_from_fork) > jal schedule_tail # a0 = struct task_struct *prev > > FEXPORT(syscall_exit) > +#ifdef CONFIG_DEBUG_RSEQ > + move a0, sp > + jal rseq_syscall > +#endif > local_irq_disable # make sure need_resched and > # signals dont change between > # sampling and return > @@ -141,6 +145,10 @@ work_notifysig: # deal with pending signals and > j resume_userspace_check > > FEXPORT(syscall_exit_partial) > +#ifdef CONFIG_DEBUG_RSEQ > + move a0, sp > + jal rseq_syscall > +#endif > local_irq_disable # make sure need_resched doesn't > # change between and return > LONG_L a2, TI_FLAGS($28) # current->work Just to double-check: you did test with CONFIG_DEBUG_RSEQ=y, right ? Are there any live registers that need to be saved before calling rseq_syscall ? Thanks, Mathieu -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 2/4] MIPS: Add syscall detection for restartable sequences 2018-06-15 17:41 ` Mathieu Desnoyers @ 2018-06-15 18:43 ` Paul Burton 2018-06-16 20:28 ` Mathieu Desnoyers 0 siblings, 1 reply; 18+ messages in thread From: Paul Burton @ 2018-06-15 18:43 UTC (permalink / raw) To: Mathieu Desnoyers Cc: linux-mips, Peter Zijlstra, James Hogan, linux-kernel, Boqun Feng, Paul E. McKenney, Ralf Baechle Hi Mathieu, On Fri, Jun 15, 2018 at 01:41:25PM -0400, Mathieu Desnoyers wrote: > > diff --git a/arch/mips/kernel/entry.S b/arch/mips/kernel/entry.S > > index 38a302919e6b..d7de8adcfcc8 100644 > > --- a/arch/mips/kernel/entry.S > > +++ b/arch/mips/kernel/entry.S > > @@ -79,6 +79,10 @@ FEXPORT(ret_from_fork) > > jal schedule_tail # a0 = struct task_struct *prev > > > > FEXPORT(syscall_exit) > > +#ifdef CONFIG_DEBUG_RSEQ > > + move a0, sp > > + jal rseq_syscall > > +#endif > > local_irq_disable # make sure need_resched and > > # signals dont change between > > # sampling and return > > @@ -141,6 +145,10 @@ work_notifysig: # deal with pending signals and > > j resume_userspace_check > > > > FEXPORT(syscall_exit_partial) > > +#ifdef CONFIG_DEBUG_RSEQ > > + move a0, sp > > + jal rseq_syscall > > +#endif > > local_irq_disable # make sure need_resched doesn't > > # change between and return > > LONG_L a2, TI_FLAGS($28) # current->work > > Just to double-check: you did test with CONFIG_DEBUG_RSEQ=y, right ? Yes, I did. Although I only ran the selftests, which I don't believe would actually trigger the SIGSEGV condition. Side-note: maybe it'd be useful to have a test that does intentionally perform a syscall within a restartable sequence & checks that it actually receives a SIGSEGV?. > Are there any live registers that need to be saved before calling > rseq_syscall ? No - we just need gp/$28 & sp/$29, and the calling convention means rseq_syscall() should return with those unmodified. Everything else that we or userland care about is about to be loaded from the stack anyway. Thanks, Paul ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 2/4] MIPS: Add syscall detection for restartable sequences 2018-06-15 18:43 ` Paul Burton @ 2018-06-16 20:28 ` Mathieu Desnoyers 0 siblings, 0 replies; 18+ messages in thread From: Mathieu Desnoyers @ 2018-06-16 20:28 UTC (permalink / raw) To: Paul Burton Cc: linux-mips, Peter Zijlstra, James Hogan, linux-kernel, Boqun Feng, Paul E. McKenney, Ralf Baechle ----- On Jun 15, 2018, at 2:43 PM, Paul Burton paul.burton@mips.com wrote: > Hi Mathieu, > > On Fri, Jun 15, 2018 at 01:41:25PM -0400, Mathieu Desnoyers wrote: >> > diff --git a/arch/mips/kernel/entry.S b/arch/mips/kernel/entry.S >> > index 38a302919e6b..d7de8adcfcc8 100644 >> > --- a/arch/mips/kernel/entry.S >> > +++ b/arch/mips/kernel/entry.S >> > @@ -79,6 +79,10 @@ FEXPORT(ret_from_fork) >> > jal schedule_tail # a0 = struct task_struct *prev >> > >> > FEXPORT(syscall_exit) >> > +#ifdef CONFIG_DEBUG_RSEQ >> > + move a0, sp >> > + jal rseq_syscall >> > +#endif >> > local_irq_disable # make sure need_resched and >> > # signals dont change between >> > # sampling and return >> > @@ -141,6 +145,10 @@ work_notifysig: # deal with pending signals and >> > j resume_userspace_check >> > >> > FEXPORT(syscall_exit_partial) >> > +#ifdef CONFIG_DEBUG_RSEQ >> > + move a0, sp >> > + jal rseq_syscall >> > +#endif >> > local_irq_disable # make sure need_resched doesn't >> > # change between and return >> > LONG_L a2, TI_FLAGS($28) # current->work >> >> Just to double-check: you did test with CONFIG_DEBUG_RSEQ=y, right ? > > Yes, I did. Although I only ran the selftests, which I don't believe > would actually trigger the SIGSEGV condition. Yeah, I typically hand-craft a critical section that generate a system call in order to trigger this. It's hackish however, and only triggers the SIGSEGV on kernels with CONFIG_DEBUG_RSEQ=y. > > Side-note: maybe it'd be useful to have a test that does intentionally > perform a syscall within a restartable sequence & checks that it > actually receives a SIGSEGV?. We'd have to craft asm code for each architecture issuing a system call in a rseq c.s. to test this. And we'd need to make this test only runs on a kernel with CONFIG_DEBUG_RSEQ=y. I'm not convinced yet it's worth the effort to cleanly integrate this in selftests, but I'm very open to the idea. > >> Are there any live registers that need to be saved before calling >> rseq_syscall ? > > No - we just need gp/$28 & sp/$29, and the calling convention means > rseq_syscall() should return with those unmodified. Everything else that > we or userland care about is about to be loaded from the stack anyway. Sounds good! Thanks, Mathieu > > Thanks, > Paul -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 3/4] MIPS: Wire up the restartable sequences (rseq) syscall @ 2018-06-14 23:52 ` Paul Burton 0 siblings, 0 replies; 18+ messages in thread From: Paul Burton @ 2018-06-14 23:52 UTC (permalink / raw) To: linux-mips Cc: Peter Zijlstra, James Hogan, linux-kernel, Mathieu Desnoyers, Boqun Feng, Paul E . McKenney, Ralf Baechle, Paul Burton Wire up the restartable sequences (rseq) syscall for MIPS. This was introduced by commit d7822b1e24f2 ("rseq: Introduce restartable sequences system call") & MIPS now supports the prerequisites. Signed-off-by: Paul Burton <paul.burton@mips.com> Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: linux-mips@linux-mips.org Cc: linux-kernel@vger.kernel.org --- arch/mips/include/uapi/asm/unistd.h | 15 +++++++++------ arch/mips/kernel/scall32-o32.S | 1 + arch/mips/kernel/scall64-64.S | 1 + arch/mips/kernel/scall64-n32.S | 1 + arch/mips/kernel/scall64-o32.S | 1 + 5 files changed, 13 insertions(+), 6 deletions(-) diff --git a/arch/mips/include/uapi/asm/unistd.h b/arch/mips/include/uapi/asm/unistd.h index bb05e9916a5f..170bf0b5b250 100644 --- a/arch/mips/include/uapi/asm/unistd.h +++ b/arch/mips/include/uapi/asm/unistd.h @@ -388,17 +388,18 @@ #define __NR_pkey_alloc (__NR_Linux + 364) #define __NR_pkey_free (__NR_Linux + 365) #define __NR_statx (__NR_Linux + 366) +#define __NR_rseq (__NR_Linux + 367) /* * Offset of the last Linux o32 flavoured syscall */ -#define __NR_Linux_syscalls 366 +#define __NR_Linux_syscalls 367 #endif /* _MIPS_SIM == _MIPS_SIM_ABI32 */ #define __NR_O32_Linux 4000 -#define __NR_O32_Linux_syscalls 366 +#define __NR_O32_Linux_syscalls 367 #if _MIPS_SIM == _MIPS_SIM_ABI64 @@ -733,16 +734,17 @@ #define __NR_pkey_alloc (__NR_Linux + 324) #define __NR_pkey_free (__NR_Linux + 325) #define __NR_statx (__NR_Linux + 326) +#define __NR_rseq (__NR_Linux + 327) /* * Offset of the last Linux 64-bit flavoured syscall */ -#define __NR_Linux_syscalls 326 +#define __NR_Linux_syscalls 327 #endif /* _MIPS_SIM == _MIPS_SIM_ABI64 */ #define __NR_64_Linux 5000 -#define __NR_64_Linux_syscalls 326 +#define __NR_64_Linux_syscalls 327 #if _MIPS_SIM == _MIPS_SIM_NABI32 @@ -1081,15 +1083,16 @@ #define __NR_pkey_alloc (__NR_Linux + 328) #define __NR_pkey_free (__NR_Linux + 329) #define __NR_statx (__NR_Linux + 330) +#define __NR_rseq (__NR_Linux + 331) /* * Offset of the last N32 flavoured syscall */ -#define __NR_Linux_syscalls 330 +#define __NR_Linux_syscalls 331 #endif /* _MIPS_SIM == _MIPS_SIM_NABI32 */ #define __NR_N32_Linux 6000 -#define __NR_N32_Linux_syscalls 330 +#define __NR_N32_Linux_syscalls 331 #endif /* _UAPI_ASM_UNISTD_H */ diff --git a/arch/mips/kernel/scall32-o32.S b/arch/mips/kernel/scall32-o32.S index a9a7d78803cd..842ff1612893 100644 --- a/arch/mips/kernel/scall32-o32.S +++ b/arch/mips/kernel/scall32-o32.S @@ -590,3 +590,4 @@ EXPORT(sys_call_table) PTR sys_pkey_alloc PTR sys_pkey_free /* 4365 */ PTR sys_statx + PTR sys_rseq diff --git a/arch/mips/kernel/scall64-64.S b/arch/mips/kernel/scall64-64.S index 65d5aeeb9bdb..558830d1e5ba 100644 --- a/arch/mips/kernel/scall64-64.S +++ b/arch/mips/kernel/scall64-64.S @@ -439,4 +439,5 @@ EXPORT(sys_call_table) PTR sys_pkey_alloc PTR sys_pkey_free /* 5325 */ PTR sys_statx + PTR sys_rseq .size sys_call_table,.-sys_call_table diff --git a/arch/mips/kernel/scall64-n32.S b/arch/mips/kernel/scall64-n32.S index cbf190ef9e8a..293f0b0119f3 100644 --- a/arch/mips/kernel/scall64-n32.S +++ b/arch/mips/kernel/scall64-n32.S @@ -434,4 +434,5 @@ EXPORT(sysn32_call_table) PTR sys_pkey_alloc PTR sys_pkey_free PTR sys_statx /* 6330 */ + PTR sys_rseq .size sysn32_call_table,.-sysn32_call_table diff --git a/arch/mips/kernel/scall64-o32.S b/arch/mips/kernel/scall64-o32.S index 9ebe3e2403b1..f13a08de8078 100644 --- a/arch/mips/kernel/scall64-o32.S +++ b/arch/mips/kernel/scall64-o32.S @@ -583,4 +583,5 @@ EXPORT(sys32_call_table) PTR sys_pkey_alloc PTR sys_pkey_free /* 4365 */ PTR sys_statx + PTR sys_rseq .size sys32_call_table,.-sys32_call_table -- 2.17.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 3/4] MIPS: Wire up the restartable sequences (rseq) syscall @ 2018-06-14 23:52 ` Paul Burton 0 siblings, 0 replies; 18+ messages in thread From: Paul Burton @ 2018-06-14 23:52 UTC (permalink / raw) To: linux-mips Cc: Peter Zijlstra, James Hogan, linux-kernel, Mathieu Desnoyers, Boqun Feng, Paul E . McKenney, Ralf Baechle, Paul Burton Wire up the restartable sequences (rseq) syscall for MIPS. This was introduced by commit d7822b1e24f2 ("rseq: Introduce restartable sequences system call") & MIPS now supports the prerequisites. Signed-off-by: Paul Burton <paul.burton@mips.com> Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: linux-mips@linux-mips.org Cc: linux-kernel@vger.kernel.org --- arch/mips/include/uapi/asm/unistd.h | 15 +++++++++------ arch/mips/kernel/scall32-o32.S | 1 + arch/mips/kernel/scall64-64.S | 1 + arch/mips/kernel/scall64-n32.S | 1 + arch/mips/kernel/scall64-o32.S | 1 + 5 files changed, 13 insertions(+), 6 deletions(-) diff --git a/arch/mips/include/uapi/asm/unistd.h b/arch/mips/include/uapi/asm/unistd.h index bb05e9916a5f..170bf0b5b250 100644 --- a/arch/mips/include/uapi/asm/unistd.h +++ b/arch/mips/include/uapi/asm/unistd.h @@ -388,17 +388,18 @@ #define __NR_pkey_alloc (__NR_Linux + 364) #define __NR_pkey_free (__NR_Linux + 365) #define __NR_statx (__NR_Linux + 366) +#define __NR_rseq (__NR_Linux + 367) /* * Offset of the last Linux o32 flavoured syscall */ -#define __NR_Linux_syscalls 366 +#define __NR_Linux_syscalls 367 #endif /* _MIPS_SIM == _MIPS_SIM_ABI32 */ #define __NR_O32_Linux 4000 -#define __NR_O32_Linux_syscalls 366 +#define __NR_O32_Linux_syscalls 367 #if _MIPS_SIM == _MIPS_SIM_ABI64 @@ -733,16 +734,17 @@ #define __NR_pkey_alloc (__NR_Linux + 324) #define __NR_pkey_free (__NR_Linux + 325) #define __NR_statx (__NR_Linux + 326) +#define __NR_rseq (__NR_Linux + 327) /* * Offset of the last Linux 64-bit flavoured syscall */ -#define __NR_Linux_syscalls 326 +#define __NR_Linux_syscalls 327 #endif /* _MIPS_SIM == _MIPS_SIM_ABI64 */ #define __NR_64_Linux 5000 -#define __NR_64_Linux_syscalls 326 +#define __NR_64_Linux_syscalls 327 #if _MIPS_SIM == _MIPS_SIM_NABI32 @@ -1081,15 +1083,16 @@ #define __NR_pkey_alloc (__NR_Linux + 328) #define __NR_pkey_free (__NR_Linux + 329) #define __NR_statx (__NR_Linux + 330) +#define __NR_rseq (__NR_Linux + 331) /* * Offset of the last N32 flavoured syscall */ -#define __NR_Linux_syscalls 330 +#define __NR_Linux_syscalls 331 #endif /* _MIPS_SIM == _MIPS_SIM_NABI32 */ #define __NR_N32_Linux 6000 -#define __NR_N32_Linux_syscalls 330 +#define __NR_N32_Linux_syscalls 331 #endif /* _UAPI_ASM_UNISTD_H */ diff --git a/arch/mips/kernel/scall32-o32.S b/arch/mips/kernel/scall32-o32.S index a9a7d78803cd..842ff1612893 100644 --- a/arch/mips/kernel/scall32-o32.S +++ b/arch/mips/kernel/scall32-o32.S @@ -590,3 +590,4 @@ EXPORT(sys_call_table) PTR sys_pkey_alloc PTR sys_pkey_free /* 4365 */ PTR sys_statx + PTR sys_rseq diff --git a/arch/mips/kernel/scall64-64.S b/arch/mips/kernel/scall64-64.S index 65d5aeeb9bdb..558830d1e5ba 100644 --- a/arch/mips/kernel/scall64-64.S +++ b/arch/mips/kernel/scall64-64.S @@ -439,4 +439,5 @@ EXPORT(sys_call_table) PTR sys_pkey_alloc PTR sys_pkey_free /* 5325 */ PTR sys_statx + PTR sys_rseq .size sys_call_table,.-sys_call_table diff --git a/arch/mips/kernel/scall64-n32.S b/arch/mips/kernel/scall64-n32.S index cbf190ef9e8a..293f0b0119f3 100644 --- a/arch/mips/kernel/scall64-n32.S +++ b/arch/mips/kernel/scall64-n32.S @@ -434,4 +434,5 @@ EXPORT(sysn32_call_table) PTR sys_pkey_alloc PTR sys_pkey_free PTR sys_statx /* 6330 */ + PTR sys_rseq .size sysn32_call_table,.-sysn32_call_table diff --git a/arch/mips/kernel/scall64-o32.S b/arch/mips/kernel/scall64-o32.S index 9ebe3e2403b1..f13a08de8078 100644 --- a/arch/mips/kernel/scall64-o32.S +++ b/arch/mips/kernel/scall64-o32.S @@ -583,4 +583,5 @@ EXPORT(sys32_call_table) PTR sys_pkey_alloc PTR sys_pkey_free /* 4365 */ PTR sys_statx + PTR sys_rseq .size sys32_call_table,.-sys32_call_table -- 2.17.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 4/4] rseq/selftests: Implement MIPS support @ 2018-06-14 23:52 ` Paul Burton 0 siblings, 0 replies; 18+ messages in thread From: Paul Burton @ 2018-06-14 23:52 UTC (permalink / raw) To: linux-mips Cc: Peter Zijlstra, James Hogan, linux-kernel, Mathieu Desnoyers, Boqun Feng, Paul E . McKenney, Ralf Baechle, Paul Burton Implement support for both MIPS32 & MIPS64 in the rseq selftests, in order to sanity check the recently enabled rseq syscall. The tests all pass on a MIPS Boston development board running either a MIPS32r2 interAptiv CPU & a MIPS64r6 I6500 CPU, both of which were configured with 2 cores each of which have 2 hardware threads (VP(E)s) - ie. 4 CPUs. Signed-off-by: Paul Burton <paul.burton@mips.com> Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: linux-mips@linux-mips.org Cc: linux-kernel@vger.kernel.org --- tools/testing/selftests/rseq/param_test.c | 24 + tools/testing/selftests/rseq/rseq-mips.h | 725 ++++++++++++++++++++++ tools/testing/selftests/rseq/rseq.h | 2 + 3 files changed, 751 insertions(+) create mode 100644 tools/testing/selftests/rseq/rseq-mips.h diff --git a/tools/testing/selftests/rseq/param_test.c b/tools/testing/selftests/rseq/param_test.c index 6a9f602a8718..615252331813 100644 --- a/tools/testing/selftests/rseq/param_test.c +++ b/tools/testing/selftests/rseq/param_test.c @@ -137,6 +137,30 @@ unsigned int yield_mod_cnt, nr_abort; "subic. %%" INJECT_ASM_REG ", %%" INJECT_ASM_REG ", 1\n\t" \ "bne 222b\n\t" \ "333:\n\t" + +#elif defined(__mips__) + +#define RSEQ_INJECT_INPUT \ + , [loop_cnt_1]"m"(loop_cnt[1]) \ + , [loop_cnt_2]"m"(loop_cnt[2]) \ + , [loop_cnt_3]"m"(loop_cnt[3]) \ + , [loop_cnt_4]"m"(loop_cnt[4]) \ + , [loop_cnt_5]"m"(loop_cnt[5]) \ + , [loop_cnt_6]"m"(loop_cnt[6]) + +#define INJECT_ASM_REG "$5" + +#define RSEQ_INJECT_CLOBBER \ + , INJECT_ASM_REG + +#define RSEQ_INJECT_ASM(n) \ + "lw " INJECT_ASM_REG ", %[loop_cnt_" #n "]\n\t" \ + "beqz " INJECT_ASM_REG ", 333f\n\t" \ + "222:\n\t" \ + "addiu " INJECT_ASM_REG ", -1\n\t" \ + "bnez " INJECT_ASM_REG ", 222b\n\t" \ + "333:\n\t" + #else #error unsupported target #endif diff --git a/tools/testing/selftests/rseq/rseq-mips.h b/tools/testing/selftests/rseq/rseq-mips.h new file mode 100644 index 000000000000..63131352fbdd --- /dev/null +++ b/tools/testing/selftests/rseq/rseq-mips.h @@ -0,0 +1,725 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * Author: Paul Burton <paul.burton@mips.com> + * (C) Copyright 2018 MIPS Tech LLC + * + * Based on rseq-arm.h: + * (C) Copyright 2016-2018 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com> + */ + +#define RSEQ_SIG 0x53053053 + +#define rseq_smp_mb() __asm__ __volatile__ ("sync" ::: "memory") +#define rseq_smp_rmb() rseq_smp_mb() +#define rseq_smp_wmb() rseq_smp_mb() + +#define rseq_smp_load_acquire(p) \ +__extension__ ({ \ + __typeof(*p) ____p1 = RSEQ_READ_ONCE(*p); \ + rseq_smp_mb(); \ + ____p1; \ +}) + +#define rseq_smp_acquire__after_ctrl_dep() rseq_smp_rmb() + +#define rseq_smp_store_release(p, v) \ +do { \ + rseq_smp_mb(); \ + RSEQ_WRITE_ONCE(*p, v); \ +} while (0) + +#ifdef RSEQ_SKIP_FASTPATH +#include "rseq-skip.h" +#else /* !RSEQ_SKIP_FASTPATH */ + +#if _MIPS_SZLONG == 64 +# define LONG ".dword" +# define LONG_LA "dla" +# define LONG_L "ld" +# define LONG_S "sd" +# define LONG_ADDI "daddiu" +# define U32_U64_PAD(x) x +#elif _MIPS_SZLONG == 32 +# define LONG ".word" +# define LONG_LA "la" +# define LONG_L "lw" +# define LONG_S "sw" +# define LONG_ADDI "addiu" +# ifdef __BIG_ENDIAN +# define U32_U64_PAD(x) "0x0, " x +# else +# define U32_U64_PAD(x) x ", 0x0" +# endif +#else +# error unsupported _MIPS_SZLONG +#endif + +#define __RSEQ_ASM_DEFINE_TABLE(version, flags, start_ip, \ + post_commit_offset, abort_ip) \ + ".pushsection __rseq_table, \"aw\"\n\t" \ + ".balign 32\n\t" \ + ".word " __rseq_str(version) ", " __rseq_str(flags) "\n\t" \ + LONG " " U32_U64_PAD(__rseq_str(start_ip)) "\n\t" \ + LONG " " U32_U64_PAD(__rseq_str(post_commit_offset)) "\n\t" \ + LONG " " U32_U64_PAD(__rseq_str(abort_ip)) "\n\t" \ + ".popsection\n\t" + +#define RSEQ_ASM_DEFINE_TABLE(start_ip, post_commit_ip, abort_ip) \ + __RSEQ_ASM_DEFINE_TABLE(0x0, 0x0, start_ip, \ + (post_commit_ip - start_ip), abort_ip) + +#define RSEQ_ASM_STORE_RSEQ_CS(label, cs_label, rseq_cs) \ + RSEQ_INJECT_ASM(1) \ + LONG_LA " $4, " __rseq_str(cs_label) "\n\t" \ + LONG_S " $4, %[" __rseq_str(rseq_cs) "]\n\t" \ + __rseq_str(label) ":\n\t" + +#define RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, label) \ + RSEQ_INJECT_ASM(2) \ + "lw $4, %[" __rseq_str(current_cpu_id) "]\n\t" \ + "bne $4, %[" __rseq_str(cpu_id) "], " __rseq_str(label) "\n\t" + +#define __RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, \ + abort_label, version, flags, \ + start_ip, post_commit_offset, abort_ip) \ + ".balign 32\n\t" \ + __rseq_str(table_label) ":\n\t" \ + ".word " __rseq_str(version) ", " __rseq_str(flags) "\n\t" \ + LONG " " U32_U64_PAD(__rseq_str(start_ip)) "\n\t" \ + LONG " " U32_U64_PAD(__rseq_str(post_commit_offset)) "\n\t" \ + LONG " " U32_U64_PAD(__rseq_str(abort_ip)) "\n\t" \ + ".word " __rseq_str(RSEQ_SIG) "\n\t" \ + __rseq_str(label) ":\n\t" \ + teardown \ + "b %l[" __rseq_str(abort_label) "]\n\t" + +#define RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, abort_label, \ + start_ip, post_commit_ip, abort_ip) \ + __RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, \ + abort_label, 0x0, 0x0, start_ip, \ + (post_commit_ip - start_ip), abort_ip) + +#define RSEQ_ASM_DEFINE_CMPFAIL(label, teardown, cmpfail_label) \ + __rseq_str(label) ":\n\t" \ + teardown \ + "b %l[" __rseq_str(cmpfail_label) "]\n\t" + +#define rseq_workaround_gcc_asm_size_guess() __asm__ __volatile__("") + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[error2]\n\t" +#endif + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(5) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_workaround_gcc_asm_size_guess(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, + off_t voffp, intptr_t *load, int cpu) +{ + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "beq $4, %[expectnot], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_L " $4, %[v]\n\t" + "beq $4, %[expectnot], %l[error2]\n\t" +#endif + LONG_S " $4, %[load]\n\t" + LONG_ADDI " $4, %[voffp]\n\t" + LONG_L " $4, 0($4)\n\t" + /* final store */ + LONG_S " $4, %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(5) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + /* final store input */ + [v] "m" (*v), + [expectnot] "r" (expectnot), + [voffp] "Ir" (voffp), + [load] "m" (*load) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_workaround_gcc_asm_size_guess(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_addv(intptr_t *v, intptr_t count, int cpu) +{ + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) +#endif + LONG_L " $4, %[v]\n\t" + LONG_ADDI " $4, %[count]\n\t" + /* final store */ + LONG_S " $4, %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(4) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [count] "Ir" (count) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort +#ifdef RSEQ_COMPARE_TWICE + , error1 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[error2]\n\t" +#endif + /* try store */ + LONG_S " %[newv2], %[v2]\n\t" + RSEQ_INJECT_ASM(5) + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + /* try store input */ + [v2] "m" (*v2), + [newv2] "r" (newv2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_workaround_gcc_asm_size_guess(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[error2]\n\t" +#endif + /* try store */ + LONG_S " %[newv2], %[v2]\n\t" + RSEQ_INJECT_ASM(5) + "sync\n\t" /* full sync provides store-release */ + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + /* try store input */ + [v2] "m" (*v2), + [newv2] "r" (newv2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_workaround_gcc_asm_size_guess(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t expect2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) + LONG_L " $4, %[v2]\n\t" + "bne $4, %[expect2], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[error2]\n\t" + LONG_L " $4, %[v2]\n\t" + "bne $4, %[expect2], %l[error3]\n\t" +#endif + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + /* cmp2 input */ + [v2] "m" (*v2), + [expect2] "r" (expect2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2, error3 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_workaround_gcc_asm_size_guess(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("1st expected value comparison failed"); +error3: + rseq_bug("2nd expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + uintptr_t rseq_scratch[3]; + + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + LONG_S " %[src], %[rseq_scratch0]\n\t" + LONG_S " %[dst], %[rseq_scratch1]\n\t" + LONG_S " %[len], %[rseq_scratch2]\n\t" + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], 5f\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], 7f\n\t" +#endif + /* try memcpy */ + "beqz %[len], 333f\n\t" \ + "222:\n\t" \ + "lb $4, 0(%[src])\n\t" \ + "sb $4, 0(%[dst])\n\t" \ + LONG_ADDI " %[src], 1\n\t" \ + LONG_ADDI " %[dst], 1\n\t" \ + LONG_ADDI " %[len], -1\n\t" \ + "bnez %[len], 222b\n\t" \ + "333:\n\t" \ + RSEQ_INJECT_ASM(5) + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t" + "b 8f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + abort, 1b, 2b, 4f) + RSEQ_ASM_DEFINE_CMPFAIL(5, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + cmpfail) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_CMPFAIL(6, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + error1) + RSEQ_ASM_DEFINE_CMPFAIL(7, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + error2) +#endif + "8:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv), + /* try memcpy input */ + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len), + [rseq_scratch0] "m" (rseq_scratch[0]), + [rseq_scratch1] "m" (rseq_scratch[1]), + [rseq_scratch2] "m" (rseq_scratch[2]) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_workaround_gcc_asm_size_guess(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_workaround_gcc_asm_size_guess(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_workaround_gcc_asm_size_guess(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + uintptr_t rseq_scratch[3]; + + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + LONG_S " %[src], %[rseq_scratch0]\n\t" + LONG_S " %[dst], %[rseq_scratch1]\n\t" + LONG_S " %[len], %[rseq_scratch2]\n\t" + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], 5f\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], 7f\n\t" +#endif + /* try memcpy */ + "beqz %[len], 333f\n\t" \ + "222:\n\t" \ + "lb $4, 0(%[src])\n\t" \ + "sb $4, 0(%[dst])\n\t" \ + LONG_ADDI " %[src], 1\n\t" \ + LONG_ADDI " %[dst], 1\n\t" \ + LONG_ADDI " %[len], -1\n\t" \ + "bnez %[len], 222b\n\t" \ + "333:\n\t" \ + RSEQ_INJECT_ASM(5) + "sync\n\t" /* full sync provides store-release */ + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t" + "b 8f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + abort, 1b, 2b, 4f) + RSEQ_ASM_DEFINE_CMPFAIL(5, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + cmpfail) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_CMPFAIL(6, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + error1) + RSEQ_ASM_DEFINE_CMPFAIL(7, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + error2) +#endif + "8:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv), + /* try memcpy input */ + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len), + [rseq_scratch0] "m" (rseq_scratch[0]), + [rseq_scratch1] "m" (rseq_scratch[1]), + [rseq_scratch2] "m" (rseq_scratch[2]) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_workaround_gcc_asm_size_guess(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_workaround_gcc_asm_size_guess(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_workaround_gcc_asm_size_guess(); + rseq_bug("expected value comparison failed"); +#endif +} + +#endif /* !RSEQ_SKIP_FASTPATH */ diff --git a/tools/testing/selftests/rseq/rseq.h b/tools/testing/selftests/rseq/rseq.h index 0a808575cbc4..a4684112676c 100644 --- a/tools/testing/selftests/rseq/rseq.h +++ b/tools/testing/selftests/rseq/rseq.h @@ -73,6 +73,8 @@ extern __thread volatile struct rseq __rseq_abi; #include <rseq-arm.h> #elif defined(__PPC__) #include <rseq-ppc.h> +#elif defined(__mips__) +#include <rseq-mips.h> #else #error unsupported target #endif -- 2.17.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 4/4] rseq/selftests: Implement MIPS support @ 2018-06-14 23:52 ` Paul Burton 0 siblings, 0 replies; 18+ messages in thread From: Paul Burton @ 2018-06-14 23:52 UTC (permalink / raw) To: linux-mips Cc: Peter Zijlstra, James Hogan, linux-kernel, Mathieu Desnoyers, Boqun Feng, Paul E . McKenney, Ralf Baechle, Paul Burton Implement support for both MIPS32 & MIPS64 in the rseq selftests, in order to sanity check the recently enabled rseq syscall. The tests all pass on a MIPS Boston development board running either a MIPS32r2 interAptiv CPU & a MIPS64r6 I6500 CPU, both of which were configured with 2 cores each of which have 2 hardware threads (VP(E)s) - ie. 4 CPUs. Signed-off-by: Paul Burton <paul.burton@mips.com> Cc: James Hogan <jhogan@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: linux-mips@linux-mips.org Cc: linux-kernel@vger.kernel.org --- tools/testing/selftests/rseq/param_test.c | 24 + tools/testing/selftests/rseq/rseq-mips.h | 725 ++++++++++++++++++++++ tools/testing/selftests/rseq/rseq.h | 2 + 3 files changed, 751 insertions(+) create mode 100644 tools/testing/selftests/rseq/rseq-mips.h diff --git a/tools/testing/selftests/rseq/param_test.c b/tools/testing/selftests/rseq/param_test.c index 6a9f602a8718..615252331813 100644 --- a/tools/testing/selftests/rseq/param_test.c +++ b/tools/testing/selftests/rseq/param_test.c @@ -137,6 +137,30 @@ unsigned int yield_mod_cnt, nr_abort; "subic. %%" INJECT_ASM_REG ", %%" INJECT_ASM_REG ", 1\n\t" \ "bne 222b\n\t" \ "333:\n\t" + +#elif defined(__mips__) + +#define RSEQ_INJECT_INPUT \ + , [loop_cnt_1]"m"(loop_cnt[1]) \ + , [loop_cnt_2]"m"(loop_cnt[2]) \ + , [loop_cnt_3]"m"(loop_cnt[3]) \ + , [loop_cnt_4]"m"(loop_cnt[4]) \ + , [loop_cnt_5]"m"(loop_cnt[5]) \ + , [loop_cnt_6]"m"(loop_cnt[6]) + +#define INJECT_ASM_REG "$5" + +#define RSEQ_INJECT_CLOBBER \ + , INJECT_ASM_REG + +#define RSEQ_INJECT_ASM(n) \ + "lw " INJECT_ASM_REG ", %[loop_cnt_" #n "]\n\t" \ + "beqz " INJECT_ASM_REG ", 333f\n\t" \ + "222:\n\t" \ + "addiu " INJECT_ASM_REG ", -1\n\t" \ + "bnez " INJECT_ASM_REG ", 222b\n\t" \ + "333:\n\t" + #else #error unsupported target #endif diff --git a/tools/testing/selftests/rseq/rseq-mips.h b/tools/testing/selftests/rseq/rseq-mips.h new file mode 100644 index 000000000000..63131352fbdd --- /dev/null +++ b/tools/testing/selftests/rseq/rseq-mips.h @@ -0,0 +1,725 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * Author: Paul Burton <paul.burton@mips.com> + * (C) Copyright 2018 MIPS Tech LLC + * + * Based on rseq-arm.h: + * (C) Copyright 2016-2018 - Mathieu Desnoyers <mathieu.desnoyers@efficios.com> + */ + +#define RSEQ_SIG 0x53053053 + +#define rseq_smp_mb() __asm__ __volatile__ ("sync" ::: "memory") +#define rseq_smp_rmb() rseq_smp_mb() +#define rseq_smp_wmb() rseq_smp_mb() + +#define rseq_smp_load_acquire(p) \ +__extension__ ({ \ + __typeof(*p) ____p1 = RSEQ_READ_ONCE(*p); \ + rseq_smp_mb(); \ + ____p1; \ +}) + +#define rseq_smp_acquire__after_ctrl_dep() rseq_smp_rmb() + +#define rseq_smp_store_release(p, v) \ +do { \ + rseq_smp_mb(); \ + RSEQ_WRITE_ONCE(*p, v); \ +} while (0) + +#ifdef RSEQ_SKIP_FASTPATH +#include "rseq-skip.h" +#else /* !RSEQ_SKIP_FASTPATH */ + +#if _MIPS_SZLONG == 64 +# define LONG ".dword" +# define LONG_LA "dla" +# define LONG_L "ld" +# define LONG_S "sd" +# define LONG_ADDI "daddiu" +# define U32_U64_PAD(x) x +#elif _MIPS_SZLONG == 32 +# define LONG ".word" +# define LONG_LA "la" +# define LONG_L "lw" +# define LONG_S "sw" +# define LONG_ADDI "addiu" +# ifdef __BIG_ENDIAN +# define U32_U64_PAD(x) "0x0, " x +# else +# define U32_U64_PAD(x) x ", 0x0" +# endif +#else +# error unsupported _MIPS_SZLONG +#endif + +#define __RSEQ_ASM_DEFINE_TABLE(version, flags, start_ip, \ + post_commit_offset, abort_ip) \ + ".pushsection __rseq_table, \"aw\"\n\t" \ + ".balign 32\n\t" \ + ".word " __rseq_str(version) ", " __rseq_str(flags) "\n\t" \ + LONG " " U32_U64_PAD(__rseq_str(start_ip)) "\n\t" \ + LONG " " U32_U64_PAD(__rseq_str(post_commit_offset)) "\n\t" \ + LONG " " U32_U64_PAD(__rseq_str(abort_ip)) "\n\t" \ + ".popsection\n\t" + +#define RSEQ_ASM_DEFINE_TABLE(start_ip, post_commit_ip, abort_ip) \ + __RSEQ_ASM_DEFINE_TABLE(0x0, 0x0, start_ip, \ + (post_commit_ip - start_ip), abort_ip) + +#define RSEQ_ASM_STORE_RSEQ_CS(label, cs_label, rseq_cs) \ + RSEQ_INJECT_ASM(1) \ + LONG_LA " $4, " __rseq_str(cs_label) "\n\t" \ + LONG_S " $4, %[" __rseq_str(rseq_cs) "]\n\t" \ + __rseq_str(label) ":\n\t" + +#define RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, label) \ + RSEQ_INJECT_ASM(2) \ + "lw $4, %[" __rseq_str(current_cpu_id) "]\n\t" \ + "bne $4, %[" __rseq_str(cpu_id) "], " __rseq_str(label) "\n\t" + +#define __RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, \ + abort_label, version, flags, \ + start_ip, post_commit_offset, abort_ip) \ + ".balign 32\n\t" \ + __rseq_str(table_label) ":\n\t" \ + ".word " __rseq_str(version) ", " __rseq_str(flags) "\n\t" \ + LONG " " U32_U64_PAD(__rseq_str(start_ip)) "\n\t" \ + LONG " " U32_U64_PAD(__rseq_str(post_commit_offset)) "\n\t" \ + LONG " " U32_U64_PAD(__rseq_str(abort_ip)) "\n\t" \ + ".word " __rseq_str(RSEQ_SIG) "\n\t" \ + __rseq_str(label) ":\n\t" \ + teardown \ + "b %l[" __rseq_str(abort_label) "]\n\t" + +#define RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, abort_label, \ + start_ip, post_commit_ip, abort_ip) \ + __RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, \ + abort_label, 0x0, 0x0, start_ip, \ + (post_commit_ip - start_ip), abort_ip) + +#define RSEQ_ASM_DEFINE_CMPFAIL(label, teardown, cmpfail_label) \ + __rseq_str(label) ":\n\t" \ + teardown \ + "b %l[" __rseq_str(cmpfail_label) "]\n\t" + +#define rseq_workaround_gcc_asm_size_guess() __asm__ __volatile__("") + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[error2]\n\t" +#endif + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(5) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_workaround_gcc_asm_size_guess(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, + off_t voffp, intptr_t *load, int cpu) +{ + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "beq $4, %[expectnot], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_L " $4, %[v]\n\t" + "beq $4, %[expectnot], %l[error2]\n\t" +#endif + LONG_S " $4, %[load]\n\t" + LONG_ADDI " $4, %[voffp]\n\t" + LONG_L " $4, 0($4)\n\t" + /* final store */ + LONG_S " $4, %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(5) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + /* final store input */ + [v] "m" (*v), + [expectnot] "r" (expectnot), + [voffp] "Ir" (voffp), + [load] "m" (*load) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_workaround_gcc_asm_size_guess(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_addv(intptr_t *v, intptr_t count, int cpu) +{ + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) +#endif + LONG_L " $4, %[v]\n\t" + LONG_ADDI " $4, %[count]\n\t" + /* final store */ + LONG_S " $4, %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(4) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [count] "Ir" (count) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort +#ifdef RSEQ_COMPARE_TWICE + , error1 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[error2]\n\t" +#endif + /* try store */ + LONG_S " %[newv2], %[v2]\n\t" + RSEQ_INJECT_ASM(5) + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + /* try store input */ + [v2] "m" (*v2), + [newv2] "r" (newv2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_workaround_gcc_asm_size_guess(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[error2]\n\t" +#endif + /* try store */ + LONG_S " %[newv2], %[v2]\n\t" + RSEQ_INJECT_ASM(5) + "sync\n\t" /* full sync provides store-release */ + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + /* try store input */ + [v2] "m" (*v2), + [newv2] "r" (newv2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_workaround_gcc_asm_size_guess(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t expect2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) + LONG_L " $4, %[v2]\n\t" + "bne $4, %[expect2], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[error2]\n\t" + LONG_L " $4, %[v2]\n\t" + "bne $4, %[expect2], %l[error3]\n\t" +#endif + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + /* cmp2 input */ + [v2] "m" (*v2), + [expect2] "r" (expect2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2, error3 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_workaround_gcc_asm_size_guess(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("1st expected value comparison failed"); +error3: + rseq_bug("2nd expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + uintptr_t rseq_scratch[3]; + + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + LONG_S " %[src], %[rseq_scratch0]\n\t" + LONG_S " %[dst], %[rseq_scratch1]\n\t" + LONG_S " %[len], %[rseq_scratch2]\n\t" + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], 5f\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], 7f\n\t" +#endif + /* try memcpy */ + "beqz %[len], 333f\n\t" \ + "222:\n\t" \ + "lb $4, 0(%[src])\n\t" \ + "sb $4, 0(%[dst])\n\t" \ + LONG_ADDI " %[src], 1\n\t" \ + LONG_ADDI " %[dst], 1\n\t" \ + LONG_ADDI " %[len], -1\n\t" \ + "bnez %[len], 222b\n\t" \ + "333:\n\t" \ + RSEQ_INJECT_ASM(5) + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t" + "b 8f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + abort, 1b, 2b, 4f) + RSEQ_ASM_DEFINE_CMPFAIL(5, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + cmpfail) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_CMPFAIL(6, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + error1) + RSEQ_ASM_DEFINE_CMPFAIL(7, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + error2) +#endif + "8:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv), + /* try memcpy input */ + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len), + [rseq_scratch0] "m" (rseq_scratch[0]), + [rseq_scratch1] "m" (rseq_scratch[1]), + [rseq_scratch2] "m" (rseq_scratch[2]) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_workaround_gcc_asm_size_guess(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_workaround_gcc_asm_size_guess(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_workaround_gcc_asm_size_guess(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + uintptr_t rseq_scratch[3]; + + RSEQ_INJECT_C(9) + + rseq_workaround_gcc_asm_size_guess(); + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */ + LONG_S " %[src], %[rseq_scratch0]\n\t" + LONG_S " %[dst], %[rseq_scratch1]\n\t" + LONG_S " %[len], %[rseq_scratch2]\n\t" + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], 5f\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], 7f\n\t" +#endif + /* try memcpy */ + "beqz %[len], 333f\n\t" \ + "222:\n\t" \ + "lb $4, 0(%[src])\n\t" \ + "sb $4, 0(%[dst])\n\t" \ + LONG_ADDI " %[src], 1\n\t" \ + LONG_ADDI " %[dst], 1\n\t" \ + LONG_ADDI " %[len], -1\n\t" \ + "bnez %[len], 222b\n\t" \ + "333:\n\t" \ + RSEQ_INJECT_ASM(5) + "sync\n\t" /* full sync provides store-release */ + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t" + "b 8f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + abort, 1b, 2b, 4f) + RSEQ_ASM_DEFINE_CMPFAIL(5, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + cmpfail) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_CMPFAIL(6, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + error1) + RSEQ_ASM_DEFINE_CMPFAIL(7, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + error2) +#endif + "8:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv), + /* try memcpy input */ + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len), + [rseq_scratch0] "m" (rseq_scratch[0]), + [rseq_scratch1] "m" (rseq_scratch[1]), + [rseq_scratch2] "m" (rseq_scratch[2]) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_workaround_gcc_asm_size_guess(); + return 0; +abort: + rseq_workaround_gcc_asm_size_guess(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_workaround_gcc_asm_size_guess(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_workaround_gcc_asm_size_guess(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_workaround_gcc_asm_size_guess(); + rseq_bug("expected value comparison failed"); +#endif +} + +#endif /* !RSEQ_SKIP_FASTPATH */ diff --git a/tools/testing/selftests/rseq/rseq.h b/tools/testing/selftests/rseq/rseq.h index 0a808575cbc4..a4684112676c 100644 --- a/tools/testing/selftests/rseq/rseq.h +++ b/tools/testing/selftests/rseq/rseq.h @@ -73,6 +73,8 @@ extern __thread volatile struct rseq __rseq_abi; #include <rseq-arm.h> #elif defined(__PPC__) #include <rseq-ppc.h> +#elif defined(__mips__) +#include <rseq-mips.h> #else #error unsupported target #endif -- 2.17.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH 4/4] rseq/selftests: Implement MIPS support 2018-06-14 23:52 ` Paul Burton (?) @ 2018-06-15 10:58 ` James Hogan 2018-06-15 17:37 ` Mathieu Desnoyers 2018-06-15 18:16 ` Paul Burton -1 siblings, 2 replies; 18+ messages in thread From: James Hogan @ 2018-06-15 10:58 UTC (permalink / raw) To: Paul Burton Cc: linux-mips, Peter Zijlstra, linux-kernel, Mathieu Desnoyers, Boqun Feng, Paul E . McKenney, Ralf Baechle [-- Attachment #1: Type: text/plain, Size: 631 bytes --] On Thu, Jun 14, 2018 at 04:52:10PM -0700, Paul Burton wrote: > +#define __RSEQ_ASM_DEFINE_TABLE(version, flags, start_ip, \ Nit: technically all these \'s are on 81st column... > +#define __RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, \ > + abort_label, version, flags, \ > + start_ip, post_commit_offset, abort_ip) \ > + ".balign 32\n\t" \ ARM doesn't do this for DEFINE_ABORT. Is it intentional that we do for MIPS? Otherwise this whole series looks reasonable to me, so feel free to add my rb on the whole series if you do apply youself: Reviewed-by: James Hogan <jhogan@kernel.org> Thanks James [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 228 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 4/4] rseq/selftests: Implement MIPS support 2018-06-15 10:58 ` James Hogan @ 2018-06-15 17:37 ` Mathieu Desnoyers 2018-06-15 18:16 ` Paul Burton 1 sibling, 0 replies; 18+ messages in thread From: Mathieu Desnoyers @ 2018-06-15 17:37 UTC (permalink / raw) To: James Hogan Cc: Paul Burton, linux-mips, Peter Zijlstra, linux-kernel, Boqun Feng, Paul E. McKenney, Ralf Baechle ----- On Jun 15, 2018, at 6:58 AM, James Hogan jhogan@kernel.org wrote: > On Thu, Jun 14, 2018 at 04:52:10PM -0700, Paul Burton wrote: >> +#define __RSEQ_ASM_DEFINE_TABLE(version, flags, start_ip, \ > > Nit: technically all these \'s are on 81st column... > >> +#define __RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, \ >> + abort_label, version, flags, \ >> + start_ip, post_commit_offset, abort_ip) \ >> + ".balign 32\n\t" \ > > ARM doesn't do this for DEFINE_ABORT. Is it intentional that we do for > MIPS? Given that include/uapi/linux/rseq.h declares struct rseq_cs as __attribute__((aligned(4 * sizeof(__u64)))), and considering this comment: /* * struct rseq_cs is aligned on 4 * 8 bytes to ensure it is always * contained within a single cache-line. It is usually declared as * link-time constant data. */ The .balign 32 is the right thing to do here. I will add a .balign 32 to ARM selftests code as well. Thanks, Mathieu > > Thanks > James -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 4/4] rseq/selftests: Implement MIPS support @ 2018-06-15 18:16 ` Paul Burton 0 siblings, 0 replies; 18+ messages in thread From: Paul Burton @ 2018-06-15 18:16 UTC (permalink / raw) To: James Hogan Cc: linux-mips, Peter Zijlstra, linux-kernel, Mathieu Desnoyers, Boqun Feng, Paul E . McKenney, Ralf Baechle On Fri, Jun 15, 2018 at 11:58:10AM +0100, James Hogan wrote: > On Thu, Jun 14, 2018 at 04:52:10PM -0700, Paul Burton wrote: > > +#define __RSEQ_ASM_DEFINE_TABLE(version, flags, start_ip, \ > > Nit: technically all these \'s are on 81st column... True... I'll replace the runs of tabs with a single space. > > +#define __RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, \ > > + abort_label, version, flags, \ > > + start_ip, post_commit_offset, abort_ip) \ > > + ".balign 32\n\t" \ > > ARM doesn't do this for DEFINE_ABORT. Is it intentional that we do for > MIPS? We need to align this structure at least in the MIPS64 case because the .dword directive seems to lead to extra padding if we don't, and that messes up the offsets of the fields. For example here's an extract from basic_percpu_ops_test built for MIPS64r6el without the .balign directive, starting from the RSEQ_ASM_STORE_RSEQ_CS macro in rseq_cmpeqv_storev(): 120001298: df848068 ld a0,-32664(gp) 12000129c: fc640008 sd a0,8(v1) 1200012a0: 8c640004 lw a0,4(v1) 1200012a4: 14820011 bne a0,v0,1200012ec <.L4^B1> 1200012a8: 00000000 nop 1200012ac: dca40000 ld a0,0(a1) 1200012b0: 14870015 bne a0,a3,120001308 <.L5> 1200012b4: 00000000 nop 1200012b8: fca60000 sd a2,0(a1) 1200012bc: 1000000d b 1200012f4 <.L5^B1> 1200012c0: 00000000 nop 1200012c4: 00000000 nop 1200012c8: 00000000 nop 1200012cc: 00000000 nop 1200012d0: 200012a0 bovc zero,zero,120005d54 <__FRAME_END__+0x3e28> 1200012d4: 00000001 0x1 1200012d8: 0000001c 0x1c 1200012dc: 00000000 nop 1200012e0: 200012ec bovc zero,zero,120005e94 <__FRAME_END__+0x3f68> 1200012e4: 00000001 0x1 1200012e8: 53053053 0x53053053 ... 120012118: 200012c4 bovc zero,zero,120016c2c <_end+0x49bc> 12001211c: 00000001 0x1 And _gp, which the gp register contains the address of: $ nm tools/testing/selftests/rseq/basic_percpu_ops_test | grep gp 000000012001a0b0 d _gp The ld instruction @120001298 is what the "dla $4, 3f" expanded to, so it's loading the address of the table which we're going to write to rseq_cs. It loads from gp-32664, ie. 0x12001a0b0-32664, ie. 0x120012118, so the table address loaded is 0x1200012c4. If we take that as the start of the struct rseq_cs then we get: 1200012c4: __u32 version = 0x0 1200012c8: __u32 flags = 0x0 1200012cc: __u64 start_ip = 200012a000000000 Where start_ip is both not naturally aligned, so might be slow to access or involve T&E, and more importantly doesn't have the intended value. What happened is that gas inserted 4 bytes of padding at 1200012cc to naturally align the .dword directive for start_ip, and that throws us off. Using the .balign directive avoids this, and I went with 32 bytes because that's what struct rseq_cs is declared with in linux/rseq.h. The ARM code has probably gotten away with it because it's 32 bit only, so isn't emitting any 64 bit values (though if it did I don't know what an ARM64 assembler would do with regards to alignment & padding anyway). > Otherwise this whole series looks reasonable to me, so feel free to add > my rb on the whole series if you do apply youself: > > Reviewed-by: James Hogan <jhogan@kernel.org> Thanks James :) Paul ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 4/4] rseq/selftests: Implement MIPS support @ 2018-06-15 18:16 ` Paul Burton 0 siblings, 0 replies; 18+ messages in thread From: Paul Burton @ 2018-06-15 18:16 UTC (permalink / raw) To: James Hogan Cc: linux-mips, Peter Zijlstra, linux-kernel, Mathieu Desnoyers, Boqun Feng, Paul E . McKenney, Ralf Baechle On Fri, Jun 15, 2018 at 11:58:10AM +0100, James Hogan wrote: > On Thu, Jun 14, 2018 at 04:52:10PM -0700, Paul Burton wrote: > > +#define __RSEQ_ASM_DEFINE_TABLE(version, flags, start_ip, \ > > Nit: technically all these \'s are on 81st column... True... I'll replace the runs of tabs with a single space. > > +#define __RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, \ > > + abort_label, version, flags, \ > > + start_ip, post_commit_offset, abort_ip) \ > > + ".balign 32\n\t" \ > > ARM doesn't do this for DEFINE_ABORT. Is it intentional that we do for > MIPS? We need to align this structure at least in the MIPS64 case because the .dword directive seems to lead to extra padding if we don't, and that messes up the offsets of the fields. For example here's an extract from basic_percpu_ops_test built for MIPS64r6el without the .balign directive, starting from the RSEQ_ASM_STORE_RSEQ_CS macro in rseq_cmpeqv_storev(): 120001298: df848068 ld a0,-32664(gp) 12000129c: fc640008 sd a0,8(v1) 1200012a0: 8c640004 lw a0,4(v1) 1200012a4: 14820011 bne a0,v0,1200012ec <.L4^B1> 1200012a8: 00000000 nop 1200012ac: dca40000 ld a0,0(a1) 1200012b0: 14870015 bne a0,a3,120001308 <.L5> 1200012b4: 00000000 nop 1200012b8: fca60000 sd a2,0(a1) 1200012bc: 1000000d b 1200012f4 <.L5^B1> 1200012c0: 00000000 nop 1200012c4: 00000000 nop 1200012c8: 00000000 nop 1200012cc: 00000000 nop 1200012d0: 200012a0 bovc zero,zero,120005d54 <__FRAME_END__+0x3e28> 1200012d4: 00000001 0x1 1200012d8: 0000001c 0x1c 1200012dc: 00000000 nop 1200012e0: 200012ec bovc zero,zero,120005e94 <__FRAME_END__+0x3f68> 1200012e4: 00000001 0x1 1200012e8: 53053053 0x53053053 ... 120012118: 200012c4 bovc zero,zero,120016c2c <_end+0x49bc> 12001211c: 00000001 0x1 And _gp, which the gp register contains the address of: $ nm tools/testing/selftests/rseq/basic_percpu_ops_test | grep gp 000000012001a0b0 d _gp The ld instruction @120001298 is what the "dla $4, 3f" expanded to, so it's loading the address of the table which we're going to write to rseq_cs. It loads from gp-32664, ie. 0x12001a0b0-32664, ie. 0x120012118, so the table address loaded is 0x1200012c4. If we take that as the start of the struct rseq_cs then we get: 1200012c4: __u32 version = 0x0 1200012c8: __u32 flags = 0x0 1200012cc: __u64 start_ip = 200012a000000000 Where start_ip is both not naturally aligned, so might be slow to access or involve T&E, and more importantly doesn't have the intended value. What happened is that gas inserted 4 bytes of padding at 1200012cc to naturally align the .dword directive for start_ip, and that throws us off. Using the .balign directive avoids this, and I went with 32 bytes because that's what struct rseq_cs is declared with in linux/rseq.h. The ARM code has probably gotten away with it because it's 32 bit only, so isn't emitting any 64 bit values (though if it did I don't know what an ARM64 assembler would do with regards to alignment & padding anyway). > Otherwise this whole series looks reasonable to me, so feel free to add > my rb on the whole series if you do apply youself: > > Reviewed-by: James Hogan <jhogan@kernel.org> Thanks James :) Paul ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 0/4] MIPS: Restartable sequences (rseq) support 2018-06-14 23:52 ` Paul Burton ` (4 preceding siblings ...) (?) @ 2018-06-16 20:24 ` Mathieu Desnoyers -1 siblings, 0 replies; 18+ messages in thread From: Mathieu Desnoyers @ 2018-06-16 20:24 UTC (permalink / raw) To: Paul Burton Cc: linux-mips, Peter Zijlstra, James Hogan, linux-kernel, Boqun Feng, Paul E. McKenney, Ralf Baechle ----- On Jun 14, 2018, at 7:52 PM, Paul Burton paul.burton@mips.com wrote: > This series implements MIPS support for restartable sequences, hooks up > the rseq syscall & implements MIPS support in the rseq selftests. > > Applies atop Linus' master as of 2837461dbe6f ("Merge tag 'scsi-fixes' > of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi"). > For the entire series: Reviewed-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Unless you ask otherwise, I will expect this patchset to be picked up through the MIPS maintainer tree. Thanks! Mathieu > Thanks, > Paul > > Paul Burton (4): > MIPS: Add support for restartable sequences > MIPS: Add syscall detection for restartable sequences > MIPS: Wire up the restartable sequences (rseq) syscall > rseq/selftests: Implement MIPS support > > arch/mips/Kconfig | 1 + > arch/mips/include/uapi/asm/unistd.h | 15 +- > arch/mips/kernel/entry.S | 8 + > arch/mips/kernel/scall32-o32.S | 1 + > arch/mips/kernel/scall64-64.S | 1 + > arch/mips/kernel/scall64-n32.S | 1 + > arch/mips/kernel/scall64-o32.S | 1 + > arch/mips/kernel/signal.c | 3 + > tools/testing/selftests/rseq/param_test.c | 24 + > tools/testing/selftests/rseq/rseq-mips.h | 725 ++++++++++++++++++++++ > tools/testing/selftests/rseq/rseq.h | 2 + > 11 files changed, 776 insertions(+), 6 deletions(-) > create mode 100644 tools/testing/selftests/rseq/rseq-mips.h > > -- > 2.17.1 -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2018-06-16 20:28 UTC | newest] Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-06-14 23:52 [PATCH 0/4] MIPS: Restartable sequences (rseq) support Paul Burton 2018-06-14 23:52 ` Paul Burton 2018-06-14 23:52 ` [PATCH 1/4] MIPS: Add support for restartable sequences Paul Burton 2018-06-14 23:52 ` Paul Burton 2018-06-14 23:52 ` [PATCH 2/4] MIPS: Add syscall detection " Paul Burton 2018-06-14 23:52 ` Paul Burton 2018-06-15 17:41 ` Mathieu Desnoyers 2018-06-15 18:43 ` Paul Burton 2018-06-16 20:28 ` Mathieu Desnoyers 2018-06-14 23:52 ` [PATCH 3/4] MIPS: Wire up the restartable sequences (rseq) syscall Paul Burton 2018-06-14 23:52 ` Paul Burton 2018-06-14 23:52 ` [PATCH 4/4] rseq/selftests: Implement MIPS support Paul Burton 2018-06-14 23:52 ` Paul Burton 2018-06-15 10:58 ` James Hogan 2018-06-15 17:37 ` Mathieu Desnoyers 2018-06-15 18:16 ` Paul Burton 2018-06-15 18:16 ` Paul Burton 2018-06-16 20:24 ` [PATCH 0/4] MIPS: Restartable sequences (rseq) support Mathieu Desnoyers
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.