* [PATCH V2 0/3] riscv: add support for restartable sequence @ 2020-03-09 5:59 Vincent Chen 2020-03-09 5:59 ` [PATCH V2 1/3] riscv: add required functions to enable HAVE_REGS_AND_STACK_ACCESS_API Vincent Chen ` (2 more replies) 0 siblings, 3 replies; 9+ messages in thread From: Vincent Chen @ 2020-03-09 5:59 UTC (permalink / raw) To: paul.walmsley, palmer, mathieu.desnoyers Cc: Vincent Chen, linux-riscv, linux-kselftest Add RSEQ, restartable sequence, support and related selftest to RISCV. The Kconfig option HAVE_REGS_AND_STACK_ACCESS_API is also required by RSEQ because RSEQ will modify the content of pt_regs.sepc through instruction_pointer_set() during the fixup procedure. In order to select the config HAVE_REGS_AND_STACK_ACCESS_API, the missing APIs for accessing pt_regs are also added in this patch set. The relevant RSEQ tests in kselftest require the Binutils patch "RISC-V: Fix linker problems with TLS copy relocs" to avoid placing PREINIT_ARRAY and TLS variable of librseq.so at the same address. https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=3e7bd7f24146f162565edf878840449f36a8d974 A segmental fault will happen if binutils misses this patch. Patrick Stählin (1): riscv: add required functions to enable HAVE_REGS_AND_STACK_ACCESS_API Vincent Chen (2): riscv: Add support for restartable sequence rseq/selftests: Add support for riscv Changes since v1: 1. Use the correct register name to access pt_regs arch/riscv/Kconfig | 2 + arch/riscv/include/asm/ptrace.h | 29 +- arch/riscv/kernel/entry.S | 4 + arch/riscv/kernel/ptrace.c | 99 +++++ arch/riscv/kernel/signal.c | 2 + tools/testing/selftests/rseq/param_test.c | 23 ++ tools/testing/selftests/rseq/rseq-riscv.h | 622 ++++++++++++++++++++++++++++++ tools/testing/selftests/rseq/rseq.h | 2 + 8 files changed, 782 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/rseq/rseq-riscv.h -- 2.7.4 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH V2 1/3] riscv: add required functions to enable HAVE_REGS_AND_STACK_ACCESS_API 2020-03-09 5:59 [PATCH V2 0/3] riscv: add support for restartable sequence Vincent Chen @ 2020-03-09 5:59 ` Vincent Chen 2020-03-09 5:59 ` [PATCH V2 2/3] riscv: Add support for restartable sequence Vincent Chen 2020-03-09 5:59 ` [PATCH V2 3/3] rseq/selftests: Add support for riscv Vincent Chen 2 siblings, 0 replies; 9+ messages in thread From: Vincent Chen @ 2020-03-09 5:59 UTC (permalink / raw) To: paul.walmsley, palmer, mathieu.desnoyers Cc: Vincent Chen, linux-riscv, Patrick Stählin, linux-kselftest From: Patrick Stählin <me@packi.ch> In order to select HAVE_REGS_AND_STACK_ACCESS_API, adding the APIs required by kprobes to access pt_regs and stack entries to the RISC-V ports. Signed-off-by: Patrick Stählin <me@packi.ch> Signed-off-by: Vincent Chen <vincent.chen@sifive.com> --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/ptrace.h | 29 +++++++++++- arch/riscv/kernel/ptrace.c | 99 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 128 insertions(+), 1 deletion(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 73f029eae0cc..fa98b279257e 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -66,6 +66,7 @@ config RISCV select ARCH_HAS_GCOV_PROFILE_ALL select HAVE_COPY_THREAD_TLS select HAVE_ARCH_KASAN if MMU && 64BIT + select HAVE_REGS_AND_STACK_ACCESS_API config ARCH_MMAP_RND_BITS_MIN default 18 if 64BIT diff --git a/arch/riscv/include/asm/ptrace.h b/arch/riscv/include/asm/ptrace.h index ee49f80c9533..9fc685659eb1 100644 --- a/arch/riscv/include/asm/ptrace.h +++ b/arch/riscv/include/asm/ptrace.h @@ -8,6 +8,7 @@ #include <uapi/asm/ptrace.h> #include <asm/csr.h> +#include <linux/compiler.h> #ifndef __ASSEMBLY__ @@ -59,7 +60,7 @@ struct pt_regs { #endif #define user_mode(regs) (((regs)->status & SR_PP) == 0) - +#define MAX_REG_OFFSET offsetof(struct pt_regs, orig_a0) /* Helpers for working with the instruction pointer */ static inline unsigned long instruction_pointer(struct pt_regs *regs) @@ -79,6 +80,12 @@ static inline unsigned long user_stack_pointer(struct pt_regs *regs) { return regs->sp; } + +static inline unsigned long kernel_stack_pointer(struct pt_regs *regs) +{ + return regs->sp; +} + static inline void user_stack_pointer_set(struct pt_regs *regs, unsigned long val) { @@ -101,6 +108,26 @@ static inline unsigned long regs_return_value(struct pt_regs *regs) return regs->a0; } +int regs_query_register_offset(const char *name); + +unsigned long regs_get_kernel_stack_nth(struct pt_regs *regs, unsigned int n); +/** + * regs_get_register() - get register value from its offset + * @regs: pt_regs from which register value is gotten + * @offset: offset of the register. + * + * regs_get_register returns the value of a register whose offset from @regs. + * The @offset is the offset of the register in struct pt_regs. + * If @offset is bigger than MAX_REG_OFFSET, this returns 0. + */ +static inline unsigned long regs_get_register(struct pt_regs *regs, + unsigned int offset) +{ + if (unlikely(offset > MAX_REG_OFFSET)) + return 0; + + return *(unsigned long *)((unsigned long)regs + offset); +} #endif /* __ASSEMBLY__ */ #endif /* _ASM_RISCV_PTRACE_H */ diff --git a/arch/riscv/kernel/ptrace.c b/arch/riscv/kernel/ptrace.c index 407464201b91..bdfee5eff777 100644 --- a/arch/riscv/kernel/ptrace.c +++ b/arch/riscv/kernel/ptrace.c @@ -125,6 +125,105 @@ const struct user_regset_view *task_user_regset_view(struct task_struct *task) return &riscv_user_native_view; } +struct pt_regs_offset { + const char *name; + int offset; +}; + +#define REG_OFFSET_NAME(r) {.name = #r, .offset = offsetof(struct pt_regs, r)} +#define REG_OFFSET_END {.name = NULL, .offset = 0} + +static const struct pt_regs_offset regoffset_table[] = { + REG_OFFSET_NAME(epc), + REG_OFFSET_NAME(ra), + REG_OFFSET_NAME(sp), + REG_OFFSET_NAME(gp), + REG_OFFSET_NAME(tp), + REG_OFFSET_NAME(t0), + REG_OFFSET_NAME(t1), + REG_OFFSET_NAME(t2), + REG_OFFSET_NAME(s0), + REG_OFFSET_NAME(s1), + REG_OFFSET_NAME(a0), + REG_OFFSET_NAME(a1), + REG_OFFSET_NAME(a2), + REG_OFFSET_NAME(a3), + REG_OFFSET_NAME(a4), + REG_OFFSET_NAME(a5), + REG_OFFSET_NAME(a6), + REG_OFFSET_NAME(a7), + REG_OFFSET_NAME(s2), + REG_OFFSET_NAME(s3), + REG_OFFSET_NAME(s4), + REG_OFFSET_NAME(s5), + REG_OFFSET_NAME(s6), + REG_OFFSET_NAME(s7), + REG_OFFSET_NAME(s8), + REG_OFFSET_NAME(s9), + REG_OFFSET_NAME(s10), + REG_OFFSET_NAME(s11), + REG_OFFSET_NAME(t3), + REG_OFFSET_NAME(t4), + REG_OFFSET_NAME(t5), + REG_OFFSET_NAME(t6), + REG_OFFSET_NAME(status), + REG_OFFSET_NAME(badaddr), + REG_OFFSET_NAME(cause), + REG_OFFSET_NAME(orig_a0), + REG_OFFSET_END, +}; + +/** + * regs_query_register_offset() - query register offset from its name + * @name: the name of a register + * + * regs_query_register_offset() returns the offset of a register in struct + * pt_regs from its name. If the name is invalid, this returns -EINVAL; + */ +int regs_query_register_offset(const char *name) +{ + const struct pt_regs_offset *roff; + + for (roff = regoffset_table; roff->name; roff++) + if (!strcmp(roff->name, name)) + return roff->offset; + return -EINVAL; +} + +/** + * regs_within_kernel_stack() - check the address in the stack + * @regs: pt_regs which contains kernel stack pointer. + * @addr: address which is checked. + * + * regs_within_kernel_stack() checks @addr is within the kernel stack page(s). + * If @addr is within the kernel stack, it returns true. If not, returns false. + */ +static bool regs_within_kernel_stack(struct pt_regs *regs, unsigned long addr) +{ + return (addr & ~(THREAD_SIZE - 1)) == + (kernel_stack_pointer(regs) & ~(THREAD_SIZE - 1)); +} + +/** + * regs_get_kernel_stack_nth() - get Nth entry of the stack + * @regs: pt_regs which contains kernel stack pointer. + * @n: stack entry number. + * + * regs_get_kernel_stack_nth() returns @n th entry of the kernel stack which + * is specified by @regs. If the @n th entry is NOT in the kernel stack, + * this returns 0. + */ +unsigned long regs_get_kernel_stack_nth(struct pt_regs *regs, unsigned int n) +{ + unsigned long *addr = (unsigned long *)kernel_stack_pointer(regs); + + addr += n; + if (regs_within_kernel_stack(regs, (unsigned long)addr)) + return *addr; + else + return 0; +} + void ptrace_disable(struct task_struct *child) { clear_tsk_thread_flag(child, TIF_SYSCALL_TRACE); -- 2.7.4 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH V2 2/3] riscv: Add support for restartable sequence 2020-03-09 5:59 [PATCH V2 0/3] riscv: add support for restartable sequence Vincent Chen 2020-03-09 5:59 ` [PATCH V2 1/3] riscv: add required functions to enable HAVE_REGS_AND_STACK_ACCESS_API Vincent Chen @ 2020-03-09 5:59 ` Vincent Chen 2021-07-19 14:43 ` Mathieu Desnoyers 2020-03-09 5:59 ` [PATCH V2 3/3] rseq/selftests: Add support for riscv Vincent Chen 2 siblings, 1 reply; 9+ messages in thread From: Vincent Chen @ 2020-03-09 5:59 UTC (permalink / raw) To: paul.walmsley, palmer, mathieu.desnoyers Cc: Vincent Chen, linux-riscv, linux-kselftest Add calls to rseq_signal_deliver(), rseq_handle_notify_resume() and rseq_syscall() to introduce RSEQ support. 1. Call the rseq_handle_notify_resume() function on return to userspace if TIF_NOTIFY_RESUME thread flag is set. 2. Call the rseq_signal_deliver() function to fixup on the pre-signal frame when a signal is delivered on top of a restartable sequence critical section. 3. Check that system calls are not invoked from within rseq critical sections by invoking rseq_signal() from ret_from_syscall(). With CONFIG_DEBUG_RSEQ, such behavior results in termination of the process with SIGSEGV. Signed-off-by: Vincent Chen <vincent.chen@sifive.com> --- arch/riscv/Kconfig | 1 + arch/riscv/kernel/entry.S | 4 ++++ arch/riscv/kernel/signal.c | 2 ++ 3 files changed, 7 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index fa98b279257e..ded32979d33d 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -67,6 +67,7 @@ config RISCV select HAVE_COPY_THREAD_TLS select HAVE_ARCH_KASAN if MMU && 64BIT select HAVE_REGS_AND_STACK_ACCESS_API + select HAVE_RSEQ config ARCH_MMAP_RND_BITS_MIN default 18 if 64BIT diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index bad4d85b5e91..89d3713b0aef 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -214,6 +214,10 @@ ENTRY(handle_exception) handle_syscall: /* save the initial A0 value (needed in signal handlers) */ REG_S a0, PT_ORIG_A0(sp) +#ifdef CONFIG_RSEQ_DEBUG + move a0, sp + call rseq_syscall +#endif /* * Advance SEPC to avoid executing the original * scall instruction on sret diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index 17ba190e84a5..d939c5de41c4 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -234,6 +234,7 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs) sigset_t *oldset = sigmask_to_save(); int ret; + rseq_signal_deliver(ksig, regs); /* Are we from a system call? */ if (regs->cause == EXC_SYSCALL) { /* Avoid additional syscall restarting via ret_from_exception */ @@ -316,5 +317,6 @@ asmlinkage __visible void do_notify_resume(struct pt_regs *regs, if (thread_info_flags & _TIF_NOTIFY_RESUME) { clear_thread_flag(TIF_NOTIFY_RESUME); tracehook_notify_resume(regs); + rseq_handle_notify_resume(NULL, regs); } } -- 2.7.4 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH V2 2/3] riscv: Add support for restartable sequence 2020-03-09 5:59 ` [PATCH V2 2/3] riscv: Add support for restartable sequence Vincent Chen @ 2021-07-19 14:43 ` Mathieu Desnoyers 2021-07-21 3:19 ` Vincent Chen 0 siblings, 1 reply; 9+ messages in thread From: Mathieu Desnoyers @ 2021-07-19 14:43 UTC (permalink / raw) To: Vincent Chen, Guo Ren Cc: Paul Walmsley, Palmer Dabbelt, linux-kselftest, linux-riscv, linux-kernel, Alexander Viro, Linus Torvalds ----- On Mar 9, 2020, at 1:59 AM, Vincent Chen vincent.chen@sifive.com wrote: [...] > --- a/arch/riscv/kernel/signal.c > +++ b/arch/riscv/kernel/signal.c > @@ -234,6 +234,7 @@ static void handle_signal(struct ksignal *ksig, struct > pt_regs *regs) > sigset_t *oldset = sigmask_to_save(); > int ret; > > + rseq_signal_deliver(ksig, regs); > /* Are we from a system call? */ > if (regs->cause == EXC_SYSCALL) { [...] As Al Viro pointed out on IRC, the rseq_signal_deliver() should go after syscall restart handling, similarly to what is done on every other supported architecture. Note that there is already an upstream commit derived on this non-upstream patch: commit 9866d141a097 ("csky: Add support for restartable sequence") which is broken in the same way. I'm not sure why I was never CC'd on the csky patch. Considering that nobody bothered to implement the rseq selftests for csky, I don't see how any of this is tested. I would favor a revert of that commit until the testing glue is contributed. Unfortunately, the csky commit has been upstream since v5.7. Thanks, Mathieu -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH V2 2/3] riscv: Add support for restartable sequence 2021-07-19 14:43 ` Mathieu Desnoyers @ 2021-07-21 3:19 ` Vincent Chen 0 siblings, 0 replies; 9+ messages in thread From: Vincent Chen @ 2021-07-21 3:19 UTC (permalink / raw) To: Mathieu Desnoyers Cc: Guo Ren, Paul Walmsley, Palmer Dabbelt, linux-kselftest, linux-riscv, linux-kernel, Alexander Viro, Linus Torvalds On Mon, Jul 19, 2021 at 10:43 PM Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote: > > ----- On Mar 9, 2020, at 1:59 AM, Vincent Chen vincent.chen@sifive.com wrote: > [...] > > --- a/arch/riscv/kernel/signal.c > > +++ b/arch/riscv/kernel/signal.c > > @@ -234,6 +234,7 @@ static void handle_signal(struct ksignal *ksig, struct > > pt_regs *regs) > > sigset_t *oldset = sigmask_to_save(); > > int ret; > > > > + rseq_signal_deliver(ksig, regs); > > /* Are we from a system call? */ > > if (regs->cause == EXC_SYSCALL) { > > [...] > > As Al Viro pointed out on IRC, the rseq_signal_deliver() should go after syscall > restart handling, similarly to what is done on every other supported architecture. Thanks for the notification. I will adjust the porting and try to send the patch again for review. > > Note that there is already an upstream commit derived on this non-upstream patch: > > commit 9866d141a097 ("csky: Add support for restartable sequence") > > which is broken in the same way. > > I'm not sure why I was never CC'd on the csky patch. Considering that nobody > bothered to implement the rseq selftests for csky, I don't see how any of > this is tested. I would favor a revert of that commit until the testing glue > is contributed. Unfortunately, the csky commit has been upstream since v5.7. > > Thanks, > > Mathieu > > > -- > Mathieu Desnoyers > EfficiOS Inc. > http://www.efficios.com _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH V2 3/3] rseq/selftests: Add support for riscv 2020-03-09 5:59 [PATCH V2 0/3] riscv: add support for restartable sequence Vincent Chen 2020-03-09 5:59 ` [PATCH V2 1/3] riscv: add required functions to enable HAVE_REGS_AND_STACK_ACCESS_API Vincent Chen 2020-03-09 5:59 ` [PATCH V2 2/3] riscv: Add support for restartable sequence Vincent Chen @ 2020-03-09 5:59 ` Vincent Chen 2020-03-26 15:49 ` Palmer Dabbelt 2 siblings, 1 reply; 9+ messages in thread From: Vincent Chen @ 2020-03-09 5:59 UTC (permalink / raw) To: paul.walmsley, palmer, mathieu.desnoyers Cc: Vincent Chen, linux-riscv, linux-kselftest Add support for risc-v in the rseq selftests, which covers both 64-bit and 32-bit ISA with little endian mode. Signed-off-by: Vincent Chen <vincent.chen@sifive.com> --- tools/testing/selftests/rseq/param_test.c | 23 ++ tools/testing/selftests/rseq/rseq-riscv.h | 622 ++++++++++++++++++++++++++++++ tools/testing/selftests/rseq/rseq.h | 2 + 3 files changed, 647 insertions(+) create mode 100644 tools/testing/selftests/rseq/rseq-riscv.h diff --git a/tools/testing/selftests/rseq/param_test.c b/tools/testing/selftests/rseq/param_test.c index e8a657a5f48a..4032dfaa3af2 100644 --- a/tools/testing/selftests/rseq/param_test.c +++ b/tools/testing/selftests/rseq/param_test.c @@ -205,6 +205,29 @@ unsigned int yield_mod_cnt, nr_abort; "addiu " INJECT_ASM_REG ", -1\n\t" \ "bnez " INJECT_ASM_REG ", 222b\n\t" \ "333:\n\t" +#elif defined(__riscv) + +#define RSEQ_INJECT_INPUT \ + , [loop_cnt_1]"m"(loop_cnt[1]) \ + , [loop_cnt_2]"m"(loop_cnt[2]) \ + , [loop_cnt_3]"m"(loop_cnt[3]) \ + , [loop_cnt_4]"m"(loop_cnt[4]) \ + , [loop_cnt_5]"m"(loop_cnt[5]) \ + , [loop_cnt_6]"m"(loop_cnt[6]) + +#define INJECT_ASM_REG "t1" + +#define RSEQ_INJECT_CLOBBER \ + , INJECT_ASM_REG + +#define RSEQ_INJECT_ASM(n) \ + "lw " INJECT_ASM_REG ", %[loop_cnt_" #n "]\n\t" \ + "beqz " INJECT_ASM_REG ", 333f\n\t" \ + "222:\n\t" \ + "addi " INJECT_ASM_REG "," INJECT_ASM_REG ", -1\n\t" \ + "bnez " INJECT_ASM_REG ", 222b\n\t" \ + "333:\n\t" + #else #error unsupported target diff --git a/tools/testing/selftests/rseq/rseq-riscv.h b/tools/testing/selftests/rseq/rseq-riscv.h new file mode 100644 index 000000000000..56b47db4a9a4 --- /dev/null +++ b/tools/testing/selftests/rseq/rseq-riscv.h @@ -0,0 +1,622 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * Select the instruction "csrw mhartid, x0" as the RSEQ_SIG. Unlike + * other architecture, the ebreak instruction has no immediate field for + * distinguishing purposes. Hence, ebreak is not suitable as RSEQ_SIG. + * "csrw mhartid, x0" can also satisfy the RSEQ requirement because it + * is an uncommon instruction and will raise an illegal instruction + * exception when executed in all modes. + */ + +#if __ORDER_LITTLE_ENDIAN__ == 1234 +#define RSEQ_SIG 0xf1401073 /* csrr mhartid, x0 */ +#else +#error "Currently, RSEQ only supports Little-Endian version" +#endif + +#if __riscv_xlen == 64 +#define __REG_SEL(a, b) a +#elif __riscv_xlen == 32 +#define __REG_SEL(a, b) b +#endif + +#define REG_L __REG_SEL("ld ", "lw ") +#define REG_S __REG_SEL("sd ", "sw ") + +#define RISCV_FENCE(p, s) \ + __asm__ __volatile__ ("fence " #p "," #s : : : "memory") +#define rseq_smp_mb() RISCV_FENCE(rw, rw) +#define rseq_smp_rmb() RISCV_FENCE(r, r) +#define rseq_smp_wmb() RISCV_FENCE(w, w) +#define RSEQ_ASM_TMP_REG_1 "t6" +#define RSEQ_ASM_TMP_REG_2 "t5" +#define RSEQ_ASM_TMP_REG_3 "t4" +#define RSEQ_ASM_TMP_REG_4 "t3" + +#define rseq_smp_load_acquire(p) \ +__extension__ ({ \ + __typeof(*p) ____p1 = RSEQ_READ_ONCE(*p); \ + RISCV_FENCE(r, rw) \ + ____p1; \ +}) + +#define rseq_smp_acquire__after_ctrl_dep() rseq_smp_rmb() + +#define rseq_smp_store_release(p, v) \ +do { \ + RISCV_FENCE(rw, w); \ + RSEQ_WRITE_ONCE(*p, v); \ +} while (0) + + +#ifdef RSEQ_SKIP_FASTPATH +#include "rseq-skip.h" +#else /* !RSEQ_SKIP_FASTPATH */ + +#define __RSEQ_ASM_DEFINE_TABLE(label, version, flags, start_ip, \ + post_commit_offset, abort_ip) \ + ".pushsection __rseq_cs, \"aw\"\n" \ + ".balign 32\n" \ + __rseq_str(label) ":\n" \ + ".long " __rseq_str(version) ", " __rseq_str(flags) "\n" \ + ".quad " __rseq_str(start_ip) ", " \ + __rseq_str(post_commit_offset) ", " \ + __rseq_str(abort_ip) "\n" \ + ".popsection\n\t" \ + ".pushsection __rseq_cs_ptr_array, \"aw\"\n" \ + ".quad " __rseq_str(label) "b\n" \ + ".popsection\n" + +#define RSEQ_ASM_DEFINE_TABLE(label, start_ip, post_commit_ip, abort_ip) \ + __RSEQ_ASM_DEFINE_TABLE(label, 0x0, 0x0, start_ip, \ + (post_commit_ip - start_ip), abort_ip) + + +/* + * Exit points of a rseq critical section consist of all instructions outside + * of the critical section where a critical section can either branch to or + * reach through the normal course of its execution. The abort IP and the + * post-commit IP are already part of the __rseq_cs section and should not be + * explicitly defined as additional exit points. Knowing all exit points is + * useful to assist debuggers stepping over the critical section. + */ +#define RSEQ_ASM_DEFINE_EXIT_POINT(start_ip, exit_ip) \ + ".pushsection __rseq_exit_point_array, \"aw\"\n" \ + ".quad " __rseq_str(start_ip) ", " __rseq_str(exit_ip) "\n" \ + ".popsection\n" + +#define RSEQ_ASM_STORE_RSEQ_CS(label, cs_label, rseq_cs) \ + RSEQ_INJECT_ASM(1) \ + "la "RSEQ_ASM_TMP_REG_1 ", " __rseq_str(cs_label) "\n" \ + REG_S RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(rseq_cs) "]\n" \ + __rseq_str(label) ":\n" + +#define RSEQ_ASM_DEFINE_ABORT(label, abort_label) \ + "j 222f\n" \ + ".balign 4\n" \ + ".long " __rseq_str(RSEQ_SIG) "\n" \ + __rseq_str(label) ":\n" \ + "j %l[" __rseq_str(abort_label) "]\n" \ + "222:\n" + +#define RSEQ_ASM_OP_STORE(value, var) \ + REG_S "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n" + +#define RSEQ_ASM_OP_CMPEQ(var, expect, label) \ + REG_L RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \ + "bne "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(expect) "] ," \ + __rseq_str(label) "\n" + +#define RSEQ_ASM_OP_CMPEQ32(var, expect, label) \ + "lw "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \ + "bne "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(expect) "] ," \ + __rseq_str(label) "\n" + +#define RSEQ_ASM_OP_CMPNE(var, expect, label) \ + REG_L RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \ + "beq "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(expect) "] ," \ + __rseq_str(label) "\n" + +#define RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, label) \ + RSEQ_INJECT_ASM(2) \ + RSEQ_ASM_OP_CMPEQ32(current_cpu_id, cpu_id, label) + +#define RSEQ_ASM_OP_R_LOAD(var) \ + REG_L RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" + +#define RSEQ_ASM_OP_R_STORE(var) \ + REG_S RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" + +#define RSEQ_ASM_OP_R_LOAD_OFF(offset) \ + "add "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(offset) "], " \ + RSEQ_ASM_TMP_REG_1 "\n" \ + REG_L RSEQ_ASM_TMP_REG_1 ", (" RSEQ_ASM_TMP_REG_1 ")\n" + +#define RSEQ_ASM_OP_R_ADD(count) \ + "add "RSEQ_ASM_TMP_REG_1 ", " RSEQ_ASM_TMP_REG_1 \ + ", %[" __rseq_str(count) "]\n" + +#define RSEQ_ASM_OP_FINAL_STORE(value, var, post_commit_label) \ + RSEQ_ASM_OP_STORE(value, var) \ + __rseq_str(post_commit_label) ":\n" + +#define RSEQ_ASM_OP_FINAL_STORE_RELEASE(value, var, post_commit_label) \ + "fence rw, w\n" \ + RSEQ_ASM_OP_STORE(value, var) \ + __rseq_str(post_commit_label) ":\n" + +#define RSEQ_ASM_OP_R_FINAL_STORE(var, post_commit_label) \ + REG_S RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \ + __rseq_str(post_commit_label) ":\n" + +#define RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) \ + "beqz %[" __rseq_str(len) "], 333f\n" \ + "mv " RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(len) "]\n" \ + "mv " RSEQ_ASM_TMP_REG_2 ", %[" __rseq_str(src) "]\n" \ + "mv " RSEQ_ASM_TMP_REG_3 ", %[" __rseq_str(dst) "]\n" \ + "222:\n" \ + "lb " RSEQ_ASM_TMP_REG_4 ", 0(" RSEQ_ASM_TMP_REG_2 ")\n" \ + "sb " RSEQ_ASM_TMP_REG_4 ", 0(" RSEQ_ASM_TMP_REG_3 ")\n" \ + "addi " RSEQ_ASM_TMP_REG_1 ", " RSEQ_ASM_TMP_REG_1 ", -1\n" \ + "addi " RSEQ_ASM_TMP_REG_2 ", " RSEQ_ASM_TMP_REG_2 ", 1\n" \ + "addi " RSEQ_ASM_TMP_REG_3 ", " RSEQ_ASM_TMP_REG_3 ", 1\n" \ + "bnez " RSEQ_ASM_TMP_REG_1 ", 222b\n" \ + "333:\n" + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, + off_t voffp, intptr_t *load, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPNE(v, expectnot, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPNE(v, expectnot, %l[error2]) +#endif + RSEQ_ASM_OP_R_LOAD(v) + RSEQ_ASM_OP_R_STORE(load) + RSEQ_ASM_OP_R_LOAD_OFF(voffp) + RSEQ_ASM_OP_R_FINAL_STORE(v, 3) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [expectnot] "r" (expectnot), + [load] "m" (*load), + [voffp] "r" (voffp) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_addv(intptr_t *v, intptr_t count, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) +#endif + RSEQ_ASM_OP_R_LOAD(v) + RSEQ_ASM_OP_R_ADD(count) + RSEQ_ASM_OP_R_FINAL_STORE(v, 3) + RSEQ_INJECT_ASM(4) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [count] "r" (count) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + : abort +#ifdef RSEQ_COMPARE_TWICE + , error1 +#endif + ); + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + RSEQ_ASM_OP_STORE(newv2, v2) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [expect] "r" (expect), + [v] "m" (*v), + [newv] "r" (newv), + [v2] "m" (*v2), + [newv2] "r" (newv2) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + RSEQ_ASM_OP_STORE(newv2, v2) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [expect] "r" (expect), + [v] "m" (*v), + [newv] "r" (newv), + [v2] "m" (*v2), + [newv2] "r" (newv2) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t expect2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error3]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) + RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[cmpfail]) + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) + RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[error3]) +#endif + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [expect] "r" (expect), + [v2] "m" (*v2), + [expect2] "r" (expect2), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2, error3 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +error3: + rseq_bug("2nd expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [expect] "r" (expect), + [v] "m" (*v), + [newv] "r" (newv), + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1, RSEQ_ASM_TMP_REG_2, + RSEQ_ASM_TMP_REG_3, RSEQ_ASM_TMP_REG_4 + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [expect] "r" (expect), + [v] "m" (*v), + [newv] "r" (newv), + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1, RSEQ_ASM_TMP_REG_2, + RSEQ_ASM_TMP_REG_3, RSEQ_ASM_TMP_REG_4 + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +#endif /* !RSEQ_SKIP_FASTPATH */ diff --git a/tools/testing/selftests/rseq/rseq.h b/tools/testing/selftests/rseq/rseq.h index 3f63eb362b92..72efb6d3d84e 100644 --- a/tools/testing/selftests/rseq/rseq.h +++ b/tools/testing/selftests/rseq/rseq.h @@ -79,6 +79,8 @@ extern int __rseq_handled; #include <rseq-mips.h> #elif defined(__s390__) #include <rseq-s390.h> +#elif defined(__riscv) +#include <rseq-riscv.h> #else #error unsupported target #endif -- 2.7.4 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH V2 3/3] rseq/selftests: Add support for riscv 2020-03-09 5:59 ` [PATCH V2 3/3] rseq/selftests: Add support for riscv Vincent Chen @ 2020-03-26 15:49 ` Palmer Dabbelt 2020-03-26 16:17 ` Mathieu Desnoyers 0 siblings, 1 reply; 9+ messages in thread From: Palmer Dabbelt @ 2020-03-26 15:49 UTC (permalink / raw) To: vincent.chen Cc: vincent.chen, linux-riscv, mathieu.desnoyers, linux-kselftest, Paul Walmsley On Sun, 08 Mar 2020 22:59:52 PDT (-0700), vincent.chen@sifive.com wrote: > Add support for risc-v in the rseq selftests, which covers both > 64-bit and 32-bit ISA with little endian mode. > > Signed-off-by: Vincent Chen <vincent.chen@sifive.com> > --- > tools/testing/selftests/rseq/param_test.c | 23 ++ > tools/testing/selftests/rseq/rseq-riscv.h | 622 ++++++++++++++++++++++++++++++ > tools/testing/selftests/rseq/rseq.h | 2 + > 3 files changed, 647 insertions(+) > create mode 100644 tools/testing/selftests/rseq/rseq-riscv.h There are a ton of checkpatch errors in here. > > diff --git a/tools/testing/selftests/rseq/param_test.c b/tools/testing/selftests/rseq/param_test.c > index e8a657a5f48a..4032dfaa3af2 100644 > --- a/tools/testing/selftests/rseq/param_test.c > +++ b/tools/testing/selftests/rseq/param_test.c > @@ -205,6 +205,29 @@ unsigned int yield_mod_cnt, nr_abort; > "addiu " INJECT_ASM_REG ", -1\n\t" \ > "bnez " INJECT_ASM_REG ", 222b\n\t" \ > "333:\n\t" > +#elif defined(__riscv) > + > +#define RSEQ_INJECT_INPUT \ > + , [loop_cnt_1]"m"(loop_cnt[1]) \ > + , [loop_cnt_2]"m"(loop_cnt[2]) \ > + , [loop_cnt_3]"m"(loop_cnt[3]) \ > + , [loop_cnt_4]"m"(loop_cnt[4]) \ > + , [loop_cnt_5]"m"(loop_cnt[5]) \ > + , [loop_cnt_6]"m"(loop_cnt[6]) > + > +#define INJECT_ASM_REG "t1" > + > +#define RSEQ_INJECT_CLOBBER \ > + , INJECT_ASM_REG > + > +#define RSEQ_INJECT_ASM(n) \ > + "lw " INJECT_ASM_REG ", %[loop_cnt_" #n "]\n\t" \ > + "beqz " INJECT_ASM_REG ", 333f\n\t" \ > + "222:\n\t" \ > + "addi " INJECT_ASM_REG "," INJECT_ASM_REG ", -1\n\t" \ > + "bnez " INJECT_ASM_REG ", 222b\n\t" \ > + "333:\n\t" > + > > #else > #error unsupported target > diff --git a/tools/testing/selftests/rseq/rseq-riscv.h b/tools/testing/selftests/rseq/rseq-riscv.h > new file mode 100644 > index 000000000000..56b47db4a9a4 > --- /dev/null > +++ b/tools/testing/selftests/rseq/rseq-riscv.h > @@ -0,0 +1,622 @@ > +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ > +/* > + * Select the instruction "csrw mhartid, x0" as the RSEQ_SIG. Unlike > + * other architecture, the ebreak instruction has no immediate field for > + * distinguishing purposes. Hence, ebreak is not suitable as RSEQ_SIG. > + * "csrw mhartid, x0" can also satisfy the RSEQ requirement because it > + * is an uncommon instruction and will raise an illegal instruction > + * exception when executed in all modes. > + */ > + > +#if __ORDER_LITTLE_ENDIAN__ == 1234 > +#define RSEQ_SIG 0xf1401073 /* csrr mhartid, x0 */ > +#else > +#error "Currently, RSEQ only supports Little-Endian version" > +#endif > + > +#if __riscv_xlen == 64 > +#define __REG_SEL(a, b) a > +#elif __riscv_xlen == 32 > +#define __REG_SEL(a, b) b > +#endif > + > +#define REG_L __REG_SEL("ld ", "lw ") > +#define REG_S __REG_SEL("sd ", "sw ") > + > +#define RISCV_FENCE(p, s) \ > + __asm__ __volatile__ ("fence " #p "," #s : : : "memory") > +#define rseq_smp_mb() RISCV_FENCE(rw, rw) > +#define rseq_smp_rmb() RISCV_FENCE(r, r) > +#define rseq_smp_wmb() RISCV_FENCE(w, w) > +#define RSEQ_ASM_TMP_REG_1 "t6" > +#define RSEQ_ASM_TMP_REG_2 "t5" > +#define RSEQ_ASM_TMP_REG_3 "t4" > +#define RSEQ_ASM_TMP_REG_4 "t3" > + > +#define rseq_smp_load_acquire(p) \ > +__extension__ ({ \ > + __typeof(*p) ____p1 = RSEQ_READ_ONCE(*p); \ > + RISCV_FENCE(r, rw) \ > + ____p1; \ > +}) > + > +#define rseq_smp_acquire__after_ctrl_dep() rseq_smp_rmb() > + > +#define rseq_smp_store_release(p, v) \ > +do { \ > + RISCV_FENCE(rw, w); \ > + RSEQ_WRITE_ONCE(*p, v); \ > +} while (0) > + > + > +#ifdef RSEQ_SKIP_FASTPATH > +#include "rseq-skip.h" > +#else /* !RSEQ_SKIP_FASTPATH */ > + > +#define __RSEQ_ASM_DEFINE_TABLE(label, version, flags, start_ip, \ > + post_commit_offset, abort_ip) \ > + ".pushsection __rseq_cs, \"aw\"\n" \ > + ".balign 32\n" \ > + __rseq_str(label) ":\n" \ > + ".long " __rseq_str(version) ", " __rseq_str(flags) "\n" \ > + ".quad " __rseq_str(start_ip) ", " \ > + __rseq_str(post_commit_offset) ", " \ > + __rseq_str(abort_ip) "\n" \ > + ".popsection\n\t" \ > + ".pushsection __rseq_cs_ptr_array, \"aw\"\n" \ > + ".quad " __rseq_str(label) "b\n" \ > + ".popsection\n" > + > +#define RSEQ_ASM_DEFINE_TABLE(label, start_ip, post_commit_ip, abort_ip) \ > + __RSEQ_ASM_DEFINE_TABLE(label, 0x0, 0x0, start_ip, \ > + (post_commit_ip - start_ip), abort_ip) > + > + > +/* > + * Exit points of a rseq critical section consist of all instructions outside > + * of the critical section where a critical section can either branch to or > + * reach through the normal course of its execution. The abort IP and the > + * post-commit IP are already part of the __rseq_cs section and should not be > + * explicitly defined as additional exit points. Knowing all exit points is > + * useful to assist debuggers stepping over the critical section. > + */ > +#define RSEQ_ASM_DEFINE_EXIT_POINT(start_ip, exit_ip) \ > + ".pushsection __rseq_exit_point_array, \"aw\"\n" \ > + ".quad " __rseq_str(start_ip) ", " __rseq_str(exit_ip) "\n" \ > + ".popsection\n" > + > +#define RSEQ_ASM_STORE_RSEQ_CS(label, cs_label, rseq_cs) \ > + RSEQ_INJECT_ASM(1) \ > + "la "RSEQ_ASM_TMP_REG_1 ", " __rseq_str(cs_label) "\n" \ > + REG_S RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(rseq_cs) "]\n" \ > + __rseq_str(label) ":\n" > + > +#define RSEQ_ASM_DEFINE_ABORT(label, abort_label) \ > + "j 222f\n" \ > + ".balign 4\n" \ > + ".long " __rseq_str(RSEQ_SIG) "\n" \ > + __rseq_str(label) ":\n" \ > + "j %l[" __rseq_str(abort_label) "]\n" \ > + "222:\n" > + > +#define RSEQ_ASM_OP_STORE(value, var) \ > + REG_S "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n" > + > +#define RSEQ_ASM_OP_CMPEQ(var, expect, label) \ > + REG_L RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \ > + "bne "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(expect) "] ," \ > + __rseq_str(label) "\n" > + > +#define RSEQ_ASM_OP_CMPEQ32(var, expect, label) \ > + "lw "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \ > + "bne "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(expect) "] ," \ > + __rseq_str(label) "\n" > + > +#define RSEQ_ASM_OP_CMPNE(var, expect, label) \ > + REG_L RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \ > + "beq "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(expect) "] ," \ > + __rseq_str(label) "\n" > + > +#define RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, label) \ > + RSEQ_INJECT_ASM(2) \ > + RSEQ_ASM_OP_CMPEQ32(current_cpu_id, cpu_id, label) > + > +#define RSEQ_ASM_OP_R_LOAD(var) \ > + REG_L RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" > + > +#define RSEQ_ASM_OP_R_STORE(var) \ > + REG_S RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" > + > +#define RSEQ_ASM_OP_R_LOAD_OFF(offset) \ > + "add "RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(offset) "], " \ > + RSEQ_ASM_TMP_REG_1 "\n" \ > + REG_L RSEQ_ASM_TMP_REG_1 ", (" RSEQ_ASM_TMP_REG_1 ")\n" > + > +#define RSEQ_ASM_OP_R_ADD(count) \ > + "add "RSEQ_ASM_TMP_REG_1 ", " RSEQ_ASM_TMP_REG_1 \ > + ", %[" __rseq_str(count) "]\n" > + > +#define RSEQ_ASM_OP_FINAL_STORE(value, var, post_commit_label) \ > + RSEQ_ASM_OP_STORE(value, var) \ > + __rseq_str(post_commit_label) ":\n" > + > +#define RSEQ_ASM_OP_FINAL_STORE_RELEASE(value, var, post_commit_label) \ > + "fence rw, w\n" \ > + RSEQ_ASM_OP_STORE(value, var) \ > + __rseq_str(post_commit_label) ":\n" > + > +#define RSEQ_ASM_OP_R_FINAL_STORE(var, post_commit_label) \ > + REG_S RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(var) "]\n" \ > + __rseq_str(post_commit_label) ":\n" > + > +#define RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) \ > + "beqz %[" __rseq_str(len) "], 333f\n" \ > + "mv " RSEQ_ASM_TMP_REG_1 ", %[" __rseq_str(len) "]\n" \ > + "mv " RSEQ_ASM_TMP_REG_2 ", %[" __rseq_str(src) "]\n" \ > + "mv " RSEQ_ASM_TMP_REG_3 ", %[" __rseq_str(dst) "]\n" \ > + "222:\n" \ > + "lb " RSEQ_ASM_TMP_REG_4 ", 0(" RSEQ_ASM_TMP_REG_2 ")\n" \ > + "sb " RSEQ_ASM_TMP_REG_4 ", 0(" RSEQ_ASM_TMP_REG_3 ")\n" \ > + "addi " RSEQ_ASM_TMP_REG_1 ", " RSEQ_ASM_TMP_REG_1 ", -1\n" \ > + "addi " RSEQ_ASM_TMP_REG_2 ", " RSEQ_ASM_TMP_REG_2 ", 1\n" \ > + "addi " RSEQ_ASM_TMP_REG_3 ", " RSEQ_ASM_TMP_REG_3 ", 1\n" \ > + "bnez " RSEQ_ASM_TMP_REG_1 ", 222b\n" \ > + "333:\n" > + > +static inline __attribute__((always_inline)) > +int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu) > +{ > + RSEQ_INJECT_C(9) > + > + __asm__ __volatile__ goto ( > + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) > +#endif > + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) > + RSEQ_INJECT_ASM(3) > + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) > + RSEQ_INJECT_ASM(4) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) > + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) > +#endif > + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) > + RSEQ_INJECT_ASM(5) > + RSEQ_ASM_DEFINE_ABORT(4, abort) > + : /* gcc asm goto does not allow outputs */ > + : [cpu_id] "r" (cpu), > + [current_cpu_id] "m" (__rseq_abi.cpu_id), > + [rseq_cs] "m" (__rseq_abi.rseq_cs), > + [v] "m" (*v), > + [expect] "r" (expect), > + [newv] "r" (newv) > + RSEQ_INJECT_INPUT > + : "memory", RSEQ_ASM_TMP_REG_1 > + : abort, cmpfail > +#ifdef RSEQ_COMPARE_TWICE > + , error1, error2 > +#endif > + ); > + > + return 0; > +abort: > + RSEQ_INJECT_FAILED > + return -1; > +cmpfail: > + return 1; > +#ifdef RSEQ_COMPARE_TWICE > +error1: > + rseq_bug("cpu_id comparison failed"); > +error2: > + rseq_bug("expected value comparison failed"); > +#endif > +} > + > +static inline __attribute__((always_inline)) > +int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, > + off_t voffp, intptr_t *load, int cpu) > +{ > + RSEQ_INJECT_C(9) > + > + __asm__ __volatile__ goto ( > + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) > +#endif > + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) > + RSEQ_INJECT_ASM(3) > + RSEQ_ASM_OP_CMPNE(v, expectnot, %l[cmpfail]) > + RSEQ_INJECT_ASM(4) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) > + RSEQ_ASM_OP_CMPNE(v, expectnot, %l[error2]) > +#endif > + RSEQ_ASM_OP_R_LOAD(v) > + RSEQ_ASM_OP_R_STORE(load) > + RSEQ_ASM_OP_R_LOAD_OFF(voffp) > + RSEQ_ASM_OP_R_FINAL_STORE(v, 3) > + RSEQ_INJECT_ASM(5) > + RSEQ_ASM_DEFINE_ABORT(4, abort) > + : /* gcc asm goto does not allow outputs */ > + : [cpu_id] "r" (cpu), > + [current_cpu_id] "m" (__rseq_abi.cpu_id), > + [rseq_cs] "m" (__rseq_abi.rseq_cs), > + [v] "m" (*v), > + [expectnot] "r" (expectnot), > + [load] "m" (*load), > + [voffp] "r" (voffp) > + RSEQ_INJECT_INPUT > + : "memory", RSEQ_ASM_TMP_REG_1 > + : abort, cmpfail > +#ifdef RSEQ_COMPARE_TWICE > + , error1, error2 > +#endif > + ); > + return 0; > +abort: > + RSEQ_INJECT_FAILED > + return -1; > +cmpfail: > + return 1; > +#ifdef RSEQ_COMPARE_TWICE > +error1: > + rseq_bug("cpu_id comparison failed"); > +error2: > + rseq_bug("expected value comparison failed"); > +#endif > +} > + > +static inline __attribute__((always_inline)) > +int rseq_addv(intptr_t *v, intptr_t count, int cpu) > +{ > + RSEQ_INJECT_C(9) > + > + __asm__ __volatile__ goto ( > + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) > +#endif > + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) > + RSEQ_INJECT_ASM(3) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) > +#endif > + RSEQ_ASM_OP_R_LOAD(v) > + RSEQ_ASM_OP_R_ADD(count) > + RSEQ_ASM_OP_R_FINAL_STORE(v, 3) > + RSEQ_INJECT_ASM(4) > + RSEQ_ASM_DEFINE_ABORT(4, abort) > + : /* gcc asm goto does not allow outputs */ > + : [cpu_id] "r" (cpu), > + [current_cpu_id] "m" (__rseq_abi.cpu_id), > + [rseq_cs] "m" (__rseq_abi.rseq_cs), > + [v] "m" (*v), > + [count] "r" (count) > + RSEQ_INJECT_INPUT > + : "memory", RSEQ_ASM_TMP_REG_1 > + : abort > +#ifdef RSEQ_COMPARE_TWICE > + , error1 > +#endif > + ); > + return 0; > +abort: > + RSEQ_INJECT_FAILED > + return -1; > +#ifdef RSEQ_COMPARE_TWICE > +error1: > + rseq_bug("cpu_id comparison failed"); > +#endif > +} > + > +static inline __attribute__((always_inline)) > +int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, > + intptr_t *v2, intptr_t newv2, > + intptr_t newv, int cpu) > +{ > + RSEQ_INJECT_C(9) > + > + __asm__ __volatile__ goto ( > + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) > +#endif > + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) > + RSEQ_INJECT_ASM(3) > + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) > + RSEQ_INJECT_ASM(4) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) > + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) > +#endif > + RSEQ_ASM_OP_STORE(newv2, v2) > + RSEQ_INJECT_ASM(5) > + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) > + RSEQ_INJECT_ASM(6) > + RSEQ_ASM_DEFINE_ABORT(4, abort) > + : /* gcc asm goto does not allow outputs */ > + : [cpu_id] "r" (cpu), > + [current_cpu_id] "m" (__rseq_abi.cpu_id), > + [rseq_cs] "m" (__rseq_abi.rseq_cs), > + [expect] "r" (expect), > + [v] "m" (*v), > + [newv] "r" (newv), > + [v2] "m" (*v2), > + [newv2] "r" (newv2) > + RSEQ_INJECT_INPUT > + : "memory", RSEQ_ASM_TMP_REG_1 > + : abort, cmpfail > +#ifdef RSEQ_COMPARE_TWICE > + , error1, error2 > +#endif > + ); > + > + return 0; > +abort: > + RSEQ_INJECT_FAILED > + return -1; > +cmpfail: > + return 1; > +#ifdef RSEQ_COMPARE_TWICE > +error1: > + rseq_bug("cpu_id comparison failed"); > +error2: > + rseq_bug("expected value comparison failed"); > +#endif > +} > + > +static inline __attribute__((always_inline)) > +int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, > + intptr_t *v2, intptr_t newv2, > + intptr_t newv, int cpu) > +{ > + RSEQ_INJECT_C(9) > + > + __asm__ __volatile__ goto ( > + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) > +#endif > + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) > + RSEQ_INJECT_ASM(3) > + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) > + RSEQ_INJECT_ASM(4) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) > + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) > +#endif > + RSEQ_ASM_OP_STORE(newv2, v2) > + RSEQ_INJECT_ASM(5) > + RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3) > + RSEQ_INJECT_ASM(6) > + RSEQ_ASM_DEFINE_ABORT(4, abort) > + : /* gcc asm goto does not allow outputs */ > + : [cpu_id] "r" (cpu), > + [current_cpu_id] "m" (__rseq_abi.cpu_id), > + [rseq_cs] "m" (__rseq_abi.rseq_cs), > + [expect] "r" (expect), > + [v] "m" (*v), > + [newv] "r" (newv), > + [v2] "m" (*v2), > + [newv2] "r" (newv2) > + RSEQ_INJECT_INPUT > + : "memory", RSEQ_ASM_TMP_REG_1 > + : abort, cmpfail > +#ifdef RSEQ_COMPARE_TWICE > + , error1, error2 > +#endif > + ); > + > + return 0; > +abort: > + RSEQ_INJECT_FAILED > + return -1; > +cmpfail: > + return 1; > +#ifdef RSEQ_COMPARE_TWICE > +error1: > + rseq_bug("cpu_id comparison failed"); > +error2: > + rseq_bug("expected value comparison failed"); > +#endif > +} > + > +static inline __attribute__((always_inline)) > +int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, > + intptr_t *v2, intptr_t expect2, > + intptr_t newv, int cpu) > +{ > + RSEQ_INJECT_C(9) > + > + __asm__ __volatile__ goto ( > + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error3]) > +#endif > + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) > + RSEQ_INJECT_ASM(3) > + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) > + RSEQ_INJECT_ASM(4) > + RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[cmpfail]) > + RSEQ_INJECT_ASM(5) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) > + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) > + RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[error3]) > +#endif > + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) > + RSEQ_INJECT_ASM(6) > + RSEQ_ASM_DEFINE_ABORT(4, abort) > + : /* gcc asm goto does not allow outputs */ > + : [cpu_id] "r" (cpu), > + [current_cpu_id] "m" (__rseq_abi.cpu_id), > + [rseq_cs] "m" (__rseq_abi.rseq_cs), > + [v] "m" (*v), > + [expect] "r" (expect), > + [v2] "m" (*v2), > + [expect2] "r" (expect2), > + [newv] "r" (newv) > + RSEQ_INJECT_INPUT > + : "memory", RSEQ_ASM_TMP_REG_1 > + : abort, cmpfail > +#ifdef RSEQ_COMPARE_TWICE > + , error1, error2, error3 > +#endif > + ); > + > + return 0; > +abort: > + RSEQ_INJECT_FAILED > + return -1; > +cmpfail: > + return 1; > +#ifdef RSEQ_COMPARE_TWICE > +error1: > + rseq_bug("cpu_id comparison failed"); > +error2: > + rseq_bug("expected value comparison failed"); > +error3: > + rseq_bug("2nd expected value comparison failed"); > +#endif > +} > + > +static inline __attribute__((always_inline)) > +int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, > + void *dst, void *src, size_t len, > + intptr_t newv, int cpu) > +{ > + RSEQ_INJECT_C(9) > + __asm__ __volatile__ goto ( > + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) > +#endif > + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) > + RSEQ_INJECT_ASM(3) > + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) > + RSEQ_INJECT_ASM(4) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) > + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) > +#endif > + RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) > + RSEQ_INJECT_ASM(5) > + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) > + RSEQ_INJECT_ASM(6) > + RSEQ_ASM_DEFINE_ABORT(4, abort) > + : /* gcc asm goto does not allow outputs */ > + : [cpu_id] "r" (cpu), > + [current_cpu_id] "m" (__rseq_abi.cpu_id), > + [rseq_cs] "m" (__rseq_abi.rseq_cs), > + [expect] "r" (expect), > + [v] "m" (*v), > + [newv] "r" (newv), > + [dst] "r" (dst), > + [src] "r" (src), > + [len] "r" (len) > + RSEQ_INJECT_INPUT > + : "memory", RSEQ_ASM_TMP_REG_1, RSEQ_ASM_TMP_REG_2, > + RSEQ_ASM_TMP_REG_3, RSEQ_ASM_TMP_REG_4 > + : abort, cmpfail > +#ifdef RSEQ_COMPARE_TWICE > + , error1, error2 > +#endif > + ); > + > + return 0; > +abort: > + RSEQ_INJECT_FAILED > + return -1; > +cmpfail: > + return 1; > +#ifdef RSEQ_COMPARE_TWICE > +error1: > + rseq_bug("cpu_id comparison failed"); > +error2: > + rseq_bug("expected value comparison failed"); > +#endif > +} > + > +static inline __attribute__((always_inline)) > +int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, > + void *dst, void *src, size_t len, > + intptr_t newv, int cpu) > +{ > + RSEQ_INJECT_C(9) > + > + __asm__ __volatile__ goto ( > + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) > + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) > +#endif > + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) > + RSEQ_INJECT_ASM(3) > + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) > + RSEQ_INJECT_ASM(4) > +#ifdef RSEQ_COMPARE_TWICE > + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) > + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) > +#endif > + RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) > + RSEQ_INJECT_ASM(5) > + RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3) > + RSEQ_INJECT_ASM(6) > + RSEQ_ASM_DEFINE_ABORT(4, abort) > + : /* gcc asm goto does not allow outputs */ > + : [cpu_id] "r" (cpu), > + [current_cpu_id] "m" (__rseq_abi.cpu_id), > + [rseq_cs] "m" (__rseq_abi.rseq_cs), > + [expect] "r" (expect), > + [v] "m" (*v), > + [newv] "r" (newv), > + [dst] "r" (dst), > + [src] "r" (src), > + [len] "r" (len) > + RSEQ_INJECT_INPUT > + : "memory", RSEQ_ASM_TMP_REG_1, RSEQ_ASM_TMP_REG_2, > + RSEQ_ASM_TMP_REG_3, RSEQ_ASM_TMP_REG_4 > + : abort, cmpfail > +#ifdef RSEQ_COMPARE_TWICE > + , error1, error2 > +#endif > + ); > + > + return 0; > +abort: > + RSEQ_INJECT_FAILED > + return -1; > +cmpfail: > + return 1; > +#ifdef RSEQ_COMPARE_TWICE > +error1: > + rseq_bug("cpu_id comparison failed"); > +error2: > + rseq_bug("expected value comparison failed"); > +#endif > +} > + > +#endif /* !RSEQ_SKIP_FASTPATH */ > diff --git a/tools/testing/selftests/rseq/rseq.h b/tools/testing/selftests/rseq/rseq.h > index 3f63eb362b92..72efb6d3d84e 100644 > --- a/tools/testing/selftests/rseq/rseq.h > +++ b/tools/testing/selftests/rseq/rseq.h > @@ -79,6 +79,8 @@ extern int __rseq_handled; > #include <rseq-mips.h> > #elif defined(__s390__) > #include <rseq-s390.h> > +#elif defined(__riscv) > +#include <rseq-riscv.h> > #else > #error unsupported target > #endif ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH V2 3/3] rseq/selftests: Add support for riscv 2020-03-26 15:49 ` Palmer Dabbelt @ 2020-03-26 16:17 ` Mathieu Desnoyers 2020-03-27 8:33 ` Vincent Chen 0 siblings, 1 reply; 9+ messages in thread From: Mathieu Desnoyers @ 2020-03-26 16:17 UTC (permalink / raw) To: Palmer Dabbelt; +Cc: Vincent Chen, linux-riscv, linux-kselftest, Paul Walmsley ----- On Mar 26, 2020, at 11:49 AM, Palmer Dabbelt palmer@dabbelt.com wrote: > On Sun, 08 Mar 2020 22:59:52 PDT (-0700), vincent.chen@sifive.com wrote: >> Add support for risc-v in the rseq selftests, which covers both >> 64-bit and 32-bit ISA with little endian mode. >> >> Signed-off-by: Vincent Chen <vincent.chen@sifive.com> >> --- >> tools/testing/selftests/rseq/param_test.c | 23 ++ >> tools/testing/selftests/rseq/rseq-riscv.h | 622 ++++++++++++++++++++++++++++++ >> tools/testing/selftests/rseq/rseq.h | 2 + >> 3 files changed, 647 insertions(+) >> create mode 100644 tools/testing/selftests/rseq/rseq-riscv.h > > There are a ton of checkpatch errors in here. Is it just my mail client or the main issue is: ERROR: DOS line endings ? As far as other issues are concerned, I know there are a few checkpatch false-positives that trigger for my rseq-{$ARCH}.h header, from which rseq-riscv.h is derived, because it has issues with extensive use of inline assembly. Thanks, Mathieu -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH V2 3/3] rseq/selftests: Add support for riscv 2020-03-26 16:17 ` Mathieu Desnoyers @ 2020-03-27 8:33 ` Vincent Chen 0 siblings, 0 replies; 9+ messages in thread From: Vincent Chen @ 2020-03-27 8:33 UTC (permalink / raw) To: Mathieu Desnoyers Cc: linux-riscv, Palmer Dabbelt, linux-kselftest, Paul Walmsley On Fri, Mar 27, 2020 at 12:17 AM Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote: > > ----- On Mar 26, 2020, at 11:49 AM, Palmer Dabbelt palmer@dabbelt.com wrote: > > > On Sun, 08 Mar 2020 22:59:52 PDT (-0700), vincent.chen@sifive.com wrote: > >> Add support for risc-v in the rseq selftests, which covers both > >> 64-bit and 32-bit ISA with little endian mode. > >> > >> Signed-off-by: Vincent Chen <vincent.chen@sifive.com> > >> --- > >> tools/testing/selftests/rseq/param_test.c | 23 ++ > >> tools/testing/selftests/rseq/rseq-riscv.h | 622 ++++++++++++++++++++++++++++++ > >> tools/testing/selftests/rseq/rseq.h | 2 + > >> 3 files changed, 647 insertions(+) > >> create mode 100644 tools/testing/selftests/rseq/rseq-riscv.h > > > > There are a ton of checkpatch errors in here. > > Is it just my mail client or the main issue is: > > ERROR: DOS line endings > > ? I am not sure, but I did not run into this error in my environment. > > As far as other issues are concerned, I know there are a few checkpatch > false-positives that trigger for my rseq-{$ARCH}.h header, from which rseq-riscv.h > is derived, because it has issues with extensive use of inline assembly. > > Thanks, > > Mathieu > Thank Mathieu for your explanation. The errors reported by checkpatch.pl can be categorized into two cases. The first one is "need consistent spacing around %". such as ERROR: need consistent spacing around '%' (ctx:WxV) #628: FILE: tools/testing/selftests/rseq/rseq-riscv.h:572: + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) where RSEQ_ASM_DEFINE_EXIT_POINT is defined as below #define RSEQ_ASM_DEFINE_EXIT_POINT(start_ip, exit_ip) \ ".pushsection __rseq_exit_point_array, \"aw\"\n" \ ".quad " __rseq_str(start_ip) ", " __rseq_str(exit_ip) "\n" \ ".popsection\n" These errors were mainly found in the rseq-riscv.h. As Mathieu mentioned, the RSEQ_ASM_DEFINE_EXIT_POINT macro is used in the inline assembly, which the second argument %l[error2] indicates the error2 is a label and it locates outside of the inline assembly. To obey the syntax, I cannot add a space after % to fix this bug. The second kind of error is "Macros with complex values should be enclosed in parentheses" such as ERROR: Macros with complex values should be enclosed in parentheses #27: FILE: tools/testing/selftests/rseq/param_test.c:210: +#define RSEQ_INJECT_INPUT \ + , [loop_cnt_1]"m"(loop_cnt[1]) \ + , [loop_cnt_2]"m"(loop_cnt[2]) \ + , [loop_cnt_3]"m"(loop_cnt[3]) \ + , [loop_cnt_4]"m"(loop_cnt[4]) \ + , [loop_cnt_5]"m"(loop_cnt[5]) \ + , [loop_cnt_6]"m"(loop_cnt[6]) In this case, it was a input operand list of inline assembly, so I could not add parentheses to enclose them. Except for these two kinds of error, there are two erros could be solved by adding parentheses. I also checked it and I think it would be safe. So, I mimic the implementations of other architecture without using parenthese. If needed, I think can add the parenthese to solve these two errors. Thanks ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-07-21 3:20 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-03-09 5:59 [PATCH V2 0/3] riscv: add support for restartable sequence Vincent Chen 2020-03-09 5:59 ` [PATCH V2 1/3] riscv: add required functions to enable HAVE_REGS_AND_STACK_ACCESS_API Vincent Chen 2020-03-09 5:59 ` [PATCH V2 2/3] riscv: Add support for restartable sequence Vincent Chen 2021-07-19 14:43 ` Mathieu Desnoyers 2021-07-21 3:19 ` Vincent Chen 2020-03-09 5:59 ` [PATCH V2 3/3] rseq/selftests: Add support for riscv Vincent Chen 2020-03-26 15:49 ` Palmer Dabbelt 2020-03-26 16:17 ` Mathieu Desnoyers 2020-03-27 8:33 ` Vincent Chen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).