From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752706AbcBYXc0 (ORCPT ); Thu, 25 Feb 2016 18:32:26 -0500 Received: from mail-wm0-f48.google.com ([74.125.82.48]:35816 "EHLO mail-wm0-f48.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750984AbcBYXcX convert rfc822-to-8bit (ORCPT ); Thu, 25 Feb 2016 18:32:23 -0500 From: Rasmus Villemoes To: Mathieu Desnoyers Cc: Thomas Gleixner , Andrew Morton , Russell King , Ingo Molnar , "H. Peter Anvin" , linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, Paul Turner , Andrew Hunter , Peter Zijlstra , Andy Lutomirski , Andi Kleen , Dave Watson , Chris Lameter , Ben Maurer , Steven Rostedt , "Paul E. McKenney" , Josh Triplett , Linus Torvalds , Catalin Marinas , Will Deacon , Michael Kerrisk Subject: Re: [PATCH v4 1/5] getcpu_cache system call: cache CPU number of running thread Organization: D03 References: <1456270120-7560-1-git-send-email-mathieu.desnoyers@efficios.com> <1456270120-7560-2-git-send-email-mathieu.desnoyers@efficios.com> X-Hashcash: 1:20:160225:catalin.marinas@arm.com::+LFTthGn15CtwtUw:000000000000000000000000000000000000000JXx X-Hashcash: 1:20:160225:cl@linux.com::fT08pq6OQPWz5uKY:000000IYF X-Hashcash: 1:20:160225:linux-api@vger.kernel.org::C0mkfR4V0NOOT46w:0000000000000000000000000000000000000LLS X-Hashcash: 1:20:160225:mathieu.desnoyers@efficios.com::r23DqqXgCistkVGb:00000000000000000000000000000000pMl X-Hashcash: 1:20:160225:tglx@linutronix.de::fsWD4qUFShgRbGko:00000000000000000000000000000000000000000001G+4 X-Hashcash: 1:20:160225:will.deacon@arm.com::H7kpI7ynr7a0wuPB:0000000000000000000000000000000000000000000OeU X-Hashcash: 1:20:160225:rostedt@goodmis.org::7Ua38lXLOIaUNMya:0000000000000000000000000000000000000000000pal X-Hashcash: 1:20:160225:andi@firstfloor.org::47/Et63YUjG/bJZF:00000000000000000000000000000000000000000018QH X-Hashcash: 1:20:160225:paulmck@linux.vnet.ibm.com::SXRZO+vL66o7V/PF:0000000000000000000000000000000000015DB X-Hashcash: 1:20:160225:ahh@google.com::2RW2Fvx0/Jidq7aW:0001Ra0 X-Hashcash: 1:20:160225:linux@arm.linux.org.uk::ABYPM3fJeBBxbGrP:0000000000000000000000000000000000000001uVv X-Hashcash: 1:20:160225:josh@joshtriplett.org::aFA68sv4W+wiihuB:00000000000000000000000000000000000000001G92 X-Hashcash: 1:20:160225:davejwatson@fb.com::TMoUMXuo9yVSVWh8:00000000000000000000000000000000000000000001kU5 X-Hashcash: 1:20:160225:linux-kernel@vger.kernel.org::0HYvglOB8lqx5tJ8:0000000000000000000000000000000001zoU X-Hashcash: 1:20:160225:pjt@google.com::86X+IRcgEqH2uiJM:00025WL X-Hashcash: 1:20:160225:bmaurer@fb.com::9AACWQmGR/8qfBCk:0002a7p X-Hashcash: 1:20:160225:akpm@linux-foundation.org::xSnnLlbQAZNR5wAQ:0000000000000000000000000000000000003DbX X-Hashcash: 1:20:160225:mingo@redhat.com::sGQzVkjkTCv31GbC:04g3/ X-Hashcash: 1:20:160225:hpa@zytor.com::8ic8zougCI/L5NUz:00004rWS X-Hashcash: 1:20:160225:peterz@infradead.org::Bm+06Xt9evfukRHr:000000000000000000000000000000000000000004FSb X-Hashcash: 1:20:160225:luto@amacapital.net::mIHrbw2PVGeApWMy:0000000000000000000000000000000000000000007gAH X-Hashcash: 1:20:160225:mtk.manpages@gmail.com::jkavy0d6I6pZoPqM:000000000000000000000000000000000000000BoaZ X-Hashcash: 1:20:160225:torvalds@linux-foundation.org::FyroZrF5rg1I2PRH:00000000000000000000000000000000IY9e Date: Fri, 26 Feb 2016 00:32:19 +0100 In-Reply-To: (Thomas Gleixner's message of "Wed, 24 Feb 2016 12:11:15 +0100 (CET)") Message-ID: <87egc0l62k.fsf@rasmusvillemoes.dk> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Feb 24 2016, Mathieu Desnoyers wrote: > > Typically, a library or application will keep the cpu number > cache in a thread-local storage variable, or other memory > areas belonging to each thread. It is recommended to perform > a volatile read of the cpu number cache to prevent the com‐ > piler from doing load tearing. An alternative approach is to > read the cpu number cache from inline assembly in a single > instruction. > > Each thread is responsible for registering its own cpu number > cache. Only one cpu cache address can be registered per > thread. > > The symbol __getcpu_cache_tls is recommended to be used > across libraries and applications wishing to register a > thread-local getcpu_cache. The attribute "weak" is recom‐ > mended when declaring this variable in libraries. Applica‐ > tions can choose to define their own version of this symbol > without the weak attribute as a performance improvement. > > In a typical usage scenario, the thread registering the cpu > number cache will be performing reads from that cache. It is > however also allowed to read the cpu number cache from other > threads. The cpu number cache updates performed by the kernel > provide single-copy atomicity semantics, which guarantee that > other threads performing single-copy atomic reads of the cpu > number cache will always observe a consistent value. > > Memory registered as cpu number cache should never be deallo‐ > cated before the thread which registered it exits: specifi‐ > cally, it should not be freed, and the library containing the > registered thread-local storage should not be dlclose'd. Maybe spell out the consequence if this is violated - since the SIGSEGV only happens on migration, it may take a while to strike. Random thoughts: The current implementation ensures that getcpu_cache is "idempotent" from within a single thread - once set, it can never get unset nor set to some other pointer. I think that can be useful, since it means a library can reliably use the TLS variable itself (initialized with some negative number) as an indicator of whether getcpu_cache(GETCPU_CACHE_SET) has been called. So if a single test on a fast path where the library would need to load __getcpu_cache_tls anyway is acceptable, it can avoid requiring some library init function to be called in each thread - which can sometimes be hard to arrange. Is this something we want to guarantee - that is, will we never implement GETCPU_CACHE_UNSET or a "force" flag to _SET? Either way, I think we should spend a few words on it to avoid the current behaviour becoming accidental ABI. In another thread: > However, there are other use-cases for having a fast mechanism for > reading the current CPU number, besides restartable sequences. For > instance, it can be used by glibc to implement a faster sched_getcpu. Will glibc do that? It may be a little contentious for glibc to claim a unique resource such as task_struct::cpu_cache for itself, even if everybody is supposed to use the same symbol. Hm, maybe one could say that if an application does define the symbol __getcpu_cache_tls (which is techically in the implementation namespace), that gives glibc (and any other library) license to do getcpu_cache(SET, &&__getcpu_cache_tls) (pseudo-code, of course). If a library initializes its own weak version with -2 it can check whether the application defined __getcpu_cache_tls. Ok, I'm probably overthinking this... Rasmus