Re: [PATCH v4 1/5] getcpu_cache system call: cache CPU number of running thread

[Rasmus Villemoes <linux@rasmusvillemoes.dk>]

On Wed, Feb 24 2016, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:

>
>        Typically, a library or application will keep the cpu  number
>        cache  in  a  thread-local  storage variable, or other memory
>        areas belonging to each thread. It is recommended to  perform
>        a  volatile  read of the cpu number cache to prevent the com‐
>        piler from doing load tearing. An alternative approach is  to
>        read  the  cpu  number cache from inline assembly in a single
>        instruction.
>
>        Each thread is responsible for registering its own cpu number
>        cache.   Only  one  cpu  cache  address can be registered per
>        thread.
>
>        The symbol  __getcpu_cache_tls  is  recommended  to  be  used
>        across  libraries  and  applications  wishing  to  register a
>        thread-local getcpu_cache. The  attribute  "weak"  is  recom‐
>        mended  when  declaring this variable in libraries.  Applica‐
>        tions can choose to define their own version of  this  symbol
>        without the weak attribute as a performance improvement.
>
>        In  a  typical usage scenario, the thread registering the cpu
>        number cache will be performing reads from that cache. It  is
>        however  also allowed to read the cpu number cache from other
>        threads. The cpu number cache updates performed by the kernel
>        provide single-copy atomicity semantics, which guarantee that
>        other threads performing single-copy atomic reads of the  cpu
>        number cache will always observe a consistent value.
>
>        Memory registered as cpu number cache should never be deallo‐
>        cated before the thread which registered it  exits:  specifi‐
>        cally, it should not be freed, and the library containing the
>        registered thread-local storage should not be dlclose'd.

Maybe spell out the consequence if this is violated - since the SIGSEGV
only happens on migration, it may take a while to strike.

Random thoughts: The current implementation ensures that getcpu_cache is
"idempotent" from within a single thread - once set, it can never get
unset nor set to some other pointer. I think that can be useful, since
it means a library can reliably use the TLS variable itself (initialized
with some negative number) as an indicator of whether
getcpu_cache(GETCPU_CACHE_SET) has been called. So if a single test on a
fast path where the library would need to load __getcpu_cache_tls anyway
is acceptable, it can avoid requiring some library init function to be
called in each thread - which can sometimes be hard to arrange. Is this
something we want to guarantee - that is, will we never implement
GETCPU_CACHE_UNSET or a "force" flag to _SET? Either way, I think we
should spend a few words on it to avoid the current behaviour becoming
accidental ABI.

In another thread:

> However, there are other use-cases for having a fast mechanism for
> reading the current CPU number, besides restartable sequences.  For
> instance, it can be used by glibc to implement a faster sched_getcpu.

Will glibc do that? It may be a little contentious for glibc to claim a
unique resource such as task_struct::cpu_cache for itself, even if
everybody is supposed to use the same symbol. Hm, maybe one could say
that if an application does define the symbol __getcpu_cache_tls (which
is techically in the implementation namespace), that gives glibc (and
any other library) license to do getcpu_cache(SET, &&__getcpu_cache_tls)
(pseudo-code, of course). If a library initializes its own weak version
with -2 it can check whether the application defined
__getcpu_cache_tls. Ok, I'm probably overthinking this...

Rasmus