Re: [RFC PATCH for 4.16 02/21] rseq: Introduce restartable sequences system call (v12)

[Mathieu Desnoyers <mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/w@public.gmane.org>]

----- On Dec 14, 2017, at 1:50 PM, Chris Lameter cl-vYTEC60ixJUAvxtiuMwx3w@public.gmane.org wrote:

> On Thu, 14 Dec 2017, Mathieu Desnoyers wrote:
> 
>> > I think the proper way to think about gs and fs on x86 is as base
>> > registers. They are essentially values in registers added to the address
>> > generated in an instruction. As such the approach is transferable to other
>> > processor architecture. Many support base register and base register
>> > relative processing. If a processor can do RMV instructions base register
>> > relative then you have something similar.
>>
>> How would you do it on ARM32 ?
> 
> Actually you do not really need RMV instructions. The data is cpu specific
> so within a restartable sequence you would have exclusive access right?

Yep.

> 
> F.e. a increment would be
> 
> 1. Load base register relative
> 2. add 1
> 3. Store base register relative

Actually, for the increment case, rseq headers provide a "add" API,
which uses a "add" instruction on x86. On arm 32, it does indeed:

RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
"ldr r0, %[v]\n\t"
"add r0, %[count]\n\t"
/* final store */
"str r0, %[v]\n\t"

> 
> The main overhead would be the registeration of the sequence.

Registering a rseq is a single store to a TLS (in user-space),
which really isn't that expensive.

If we port this concept to kernel-space (as I start to understand
would be your wish), then a simple pointer store to the current
task_struct would suffice.

> 
> The advantage on x86 is that you do not need a restartable sequence
> since a single lockless RMV instruction can do this (this_cpu_inc f.e.)

Indeed, on x86, for the specific case of counter increment, the single-instruction
"add" or "inc" with a segment-selector prefix can save setting up the rseq
(a pointer store), and offsetting from a base using the cpu number.

If your wish is to do this at kernel level, where we have full control over
the gs segment, this makes sense. I'm worried that applying this to user-space
might create conflicts wrt who owns that segment selector register wrt
pre-existing applications.

> 
>> One benefit of your proposal is to lessen the number of retired instructions,
>> but if we take the IPC into account, it is slower than rseq in my benchmark.
>> What
>> benefits do you expect from using segment selectors and non-lock-prefixed atomic
>> instructions on the fast-path ?
> 
> Ultimately I wish fast increments like done by this_cpu_inc() could be
> implemented in an efficient way on non x86 platforms that do not have
> cheap instructions like that.

My understanding is that your focus is mainly on kernel code, right ? Or is
your aim to port this_cpu_inc() to userspace as well ?

Indeed, the concepts behind rseq could be ported to kernel code eventually.
The immediate gain is much higher by exposing this to user-space though,
given that there is no good way to perform per-cpu operations efficiently
at all there, whereas kernel code can always disable preemption.

> 
> If cmpxchg local is slower than a group of instructions to do the same
> then there is an obvious question to the cpu architects why we would need
> the instruction at all (aside from the fact that we do not need a
> restartable sequence for these instructions).

I'm not a specialist in CPU instruction scheduling, so I won't speculate
on this topic. ;-)

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com