Re: [RFC PATCH v1] pin_on_cpu: Introduce thread CPU pinning system call

From: "H. Peter Anvin" <hpa@zytor.com>
To: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
	Chris Lameter <cl@linux.com>
Cc: Jann Horn <jannh@google.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Thomas Gleixner <tglx@linutronix.de>,
	linux-kernel <linux-kernel@vger.kernel.org>,
	Joel Fernandes <joelaf@google.com>,
	Ingo Molnar <mingo@redhat.com>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Dave Watson <davejwatson@fb.com>,
	Will Deacon <will.deacon@arm.com>, shuah <shuah@kernel.org>,
	Andi Kleen <andi@firstfloor.org>,
	linux-kselftest <linux-kselftest@vger.kernel.org>,
	Russell King <linux@arm.linux.org.uk>,
	Michael Kerrisk <mtk.manpages@gmail.com>,
	Paul <paulmck@linux.vnet.ibm.com>, Paul Turner <pjt@google.com>,
	Boqun Feng <boqun.feng@gmail.com>,
	Josh Triplett <josh@joshtriplett.org>,
	rostedt <rostedt@goodmis.org>, Ben Maurer <bmaurer@fb.com>,
	linux-api <linux-api@vger.kernel.org>,
	Andy Lutomirski <luto@amacapital.net>
Subject: Re: [RFC PATCH v1] pin_on_cpu: Introduce thread CPU pinning system call
Date: Wed, 22 Jan 2020 23:53:54 -0800	[thread overview]
Message-ID: <ead7a565-9a23-a7d7-904d-c4860f63952a@zytor.com> (raw)
In-Reply-To: <1648013936.596672.1579655468604.JavaMail.zimbra@efficios.com>

On 2020-01-21 17:11, Mathieu Desnoyers wrote:
> ----- On Jan 21, 2020, at 4:44 PM, Chris Lameter cl@linux.com wrote:
> 
>> These scenarios are all pretty complex and will be difficult to understand
>> for the user of these APIs.
>>
>> I think the easiest solution (and most comprehensible) is for the user
>> space process that does per cpu operations to get some sort of signal. If
>> its not able to handle that then terminate it. The code makes a basic
>> assumption after all that the process is running on a specific cpu. If
>> this is no longer the case then its better to abort if the process cannot
>> handle moving to a different processor.
> 
> The point of pin_on_cpu() is to allow threads to access per-cpu data
> structures belonging to a given CPU even if they cannot run on that
> CPU (because it is offline).
> 
> I am not sure what scenario your signal delivery proposal aims to cover.
> 
> Just to try to put this into the context of a specific scenario to see
> if I understand your point, is the following what you have in mind ?
> 
> 1. Thread A issues pin_on_cpu(5),
> 2. Thread B issues sched_setaffinity removing cpu 5 from thread A's
>    affinity mask,
> 3. Noticing that it would generate an invalid combination, rather than
>    failing sched_setaffinity, it would send a SIGSEGV (or other) signal
>    to thread A.
> 
> Or so you have something entirely different in mind ?
> 

I would agree that this seems like the only sane option, or you will be in a
world of hurt because of conflicting semantics. It is not just offlining, but
what happens if a policy manager calls sched_setaffinity() on another thread
-- and now the universe breaks because a library is updated to use this new
system call which collides with the expectations of the policy manager.

There doesn't seem to be any way to get this to be a local event which doesn't
break assumptions elsewhere in the system without making this an abort event
of some type. However, signals are painful in their own right, mostly because
of the lack of any infrastructure for allocating signals to libraries in user
space. I was actually thinking about exactly that issue just this weekend.

	-hpa