Re: [RFC PATCH v2 18/20] context_tracking,x86: Defer kernel text patching IPIs

From: Joel Fernandes <joel@joelfernandes.org>
To: Peter Zijlstra <peterz@infradead.org>
Cc: "Valentin Schneider" <vschneid@redhat.com>,
	linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org,
	linux-doc@vger.kernel.org, kvm@vger.kernel.org,
	linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org,
	rcu@vger.kernel.org, linux-kselftest@vger.kernel.org,
	"Nicolas Saenz Julienne" <nsaenzju@redhat.com>,
	"Steven Rostedt" <rostedt@goodmis.org>,
	"Masami Hiramatsu" <mhiramat@kernel.org>,
	"Jonathan Corbet" <corbet@lwn.net>,
	"Thomas Gleixner" <tglx@linutronix.de>,
	"Ingo Molnar" <mingo@redhat.com>,
	"Borislav Petkov" <bp@alien8.de>,
	"Dave Hansen" <dave.hansen@linux.intel.com>,
	"H. Peter Anvin" <hpa@zytor.com>,
	"Paolo Bonzini" <pbonzini@redhat.com>,
	"Wanpeng Li" <wanpengli@tencent.com>,
	"Vitaly Kuznetsov" <vkuznets@redhat.com>,
	"Andy Lutomirski" <luto@kernel.org>,
	"Frederic Weisbecker" <frederic@kernel.org>,
	"Paul E. McKenney" <paulmck@kernel.org>,
	"Neeraj Upadhyay" <quic_neeraju@quicinc.com>,
	"Josh Triplett" <josh@joshtriplett.org>,
	"Boqun Feng" <boqun.feng@gmail.com>,
	"Mathieu Desnoyers" <mathieu.desnoyers@efficios.com>,
	"Lai Jiangshan" <jiangshanlai@gmail.com>,
	Zqiang <qiang.zhang1211@gmail.com>,
	"Andrew Morton" <akpm@linux-foundation.org>,
	"Uladzislau Rezki" <urezki@gmail.com>,
	"Christoph Hellwig" <hch@infradead.org>,
	"Lorenzo Stoakes" <lstoakes@gmail.com>,
	"Josh Poimboeuf" <jpoimboe@kernel.org>,
	"Jason Baron" <jbaron@akamai.com>,
	"Kees Cook" <keescook@chromium.org>,
	"Sami Tolvanen" <samitolvanen@google.com>,
	"Ard Biesheuvel" <ardb@kernel.org>,
	"Nicholas Piggin" <npiggin@gmail.com>,
	"Juerg Haefliger" <juerg.haefliger@canonical.com>,
	"Nicolas Saenz Julienne" <nsaenz@kernel.org>,
	"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
	"Nadav Amit" <namit@vmware.com>,
	"Dan Carpenter" <error27@gmail.com>,
	"Chuang Wang" <nashuiliang@gmail.com>,
	"Yang Jihong" <yangjihong1@huawei.com>,
	"Petr Mladek" <pmladek@suse.com>,
	"Jason A. Donenfeld" <Jason@zx2c4.com>,
	"Song Liu" <song@kernel.org>,
	"Julian Pidancet" <julian.pidancet@oracle.com>,
	"Tom Lendacky" <thomas.lendacky@amd.com>,
	"Dionna Glaze" <dionnaglaze@google.com>,
	"Thomas Weißschuh" <linux@weissschuh.net>,
	"Juri Lelli" <juri.lelli@redhat.com>,
	"Daniel Bristot de Oliveira" <bristot@redhat.com>,
	"Marcelo Tosatti" <mtosatti@redhat.com>,
	"Yair Podemsky" <ypodemsk@redhat.com>
Subject: Re: [RFC PATCH v2 18/20] context_tracking,x86: Defer kernel text patching IPIs
Date: Tue, 25 Jul 2023 13:47:09 -0400	[thread overview]
Message-ID: <c72c1089-f5ac-9ed2-3412-cdb310cf5b51@joelfernandes.org> (raw)
In-Reply-To: <20230725133936.GM3765278@hirez.programming.kicks-ass.net>

On 7/25/23 09:39, Peter Zijlstra wrote:
> On Tue, Jul 25, 2023 at 06:49:45AM -0400, Joel Fernandes wrote:
>> Interesting series Valentin. Some high-level question/comments on this one:
>>
>>> On Jul 20, 2023, at 12:34 PM, Valentin Schneider <vschneid@redhat.com> wrote:
>>>
>>> text_poke_bp_batch() sends IPIs to all online CPUs to synchronize
>>> them vs the newly patched instruction. CPUs that are executing in userspace
>>> do not need this synchronization to happen immediately, and this is
>>> actually harmful interference for NOHZ_FULL CPUs.
>>
>> Does the amount of harm not correspond to practical frequency of text_poke? 
>> How often does instruction patching really happen? If it is very infrequent
>> then I am not sure if it is that harmful.
> 
> Well, it can happen quite a bit, also from things people would not
> typically 'expect' it.
> 
> For instance, the moment you create the first per-task perf event we
> frob some jump-labels (and again some second after the last one goes
> away).
> 
> The same for a bunch of runtime network configurations.

Ok cool. I guess I still have memories of that old ARM device I had
where modifications to kernel text was forbidden by hardware (was a
security feature). That was making kprobes unusable...

>>> As the synchronization IPIs are sent using a blocking call, returning from
>>> text_poke_bp_batch() implies all CPUs will observe the patched
>>> instruction(s), and this should be preserved even if the IPI is deferred.
>>> In other words, to safely defer this synchronization, any kernel
>>> instruction leading to the execution of the deferred instruction
>>> sync (ct_work_flush()) must *not* be mutable (patchable) at runtime.
>>
>> If it is not infrequent, then are you handling the case where userland
>> spends multiple seconds before entering the kernel, and all this while
>> the blocking call waits? Perhaps in such situation you want the real IPI
>> to be sent out instead of the deferred one?
> 
> Please re-read what Valentin wrote -- nobody is waiting on anything.

Makes sense. To be fair I received his email 3 minutes before yours ;-).
But thank you both for clarifying!

 - Joel