Live-Patching Archive on lore.kernel.org
 help / color / Atom feed
* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
       [not found]                   ` <20191015135634.GK2328@hirez.programming.kicks-ass.net>
@ 2019-10-15 14:13                     ` Miroslav Benes
  2019-10-15 15:06                       ` Joe Lawrence
  0 siblings, 1 reply; 24+ messages in thread
From: Miroslav Benes @ 2019-10-15 14:13 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Jessica Yu, Steven Rostedt, x86, linux-kernel, mhiramat, bristot,
	jbaron, torvalds, tglx, mingo, namit, hpa, luto, ard.biesheuvel,
	jpoimboe, live-patching

[ live-patching ML CCed ]

On Tue, 15 Oct 2019, Peter Zijlstra wrote:

> On Tue, Oct 15, 2019 at 03:07:40PM +0200, Jessica Yu wrote:
> 
> > > The fact that it is executable; also the fact that you do it right after
> > > we mark it ro. Flipping the memory protections back and forth is just
> > > really poor everything.
> > > 
> > > Once this ftrace thing is sorted, we'll change x86 to _refuse_ to make
> > > executable (kernel) memory writable.
> > 
> > Not sure if relevant, but just thought I'd clarify: IIRC,
> > klp_module_coming() is not poking the coming module, but it calls
> > module_enable_ro() on itself (the livepatch module) so it can apply
> > relocations and such on the new code, which lives inside the livepatch
> > module, and it needs to possibly do this numerous times over the
> > lifetime of the patch module for any coming module it is responsible
> > for patching (i.e., call module_enable_ro() on the patch module, not
> > necessarily the coming module). So I am not be sure why
> > klp_module_coming() should be moved before complete_formation(). I
> > hope I'm remembering the details correctly, livepatch folks feel free
> > to chime in if I'm incorrect here.
> 
> You mean it does module_disable_ro() ? That would be broken and it needs
> to be fixed. Can some livepatch person explain what it does and why?

Yes, it does. klp_module_coming() calls module_disable_ro() on all 
patching modules which patch the coming module in order to call 
apply_relocate_add(). New (patching) code for a module can be relocated 
only when the relevant module is loaded.

Miroslav

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-15 14:13                     ` [PATCH v3 5/6] x86/ftrace: Use text_poke() Miroslav Benes
@ 2019-10-15 15:06                       ` Joe Lawrence
  2019-10-15 15:31                         ` Jessica Yu
  2019-10-16  6:51                         ` Miroslav Benes
  0 siblings, 2 replies; 24+ messages in thread
From: Joe Lawrence @ 2019-10-15 15:06 UTC (permalink / raw)
  To: Miroslav Benes, Peter Zijlstra
  Cc: Jessica Yu, Steven Rostedt, x86, linux-kernel, mhiramat, bristot,
	jbaron, torvalds, tglx, mingo, namit, hpa, luto, ard.biesheuvel,
	jpoimboe, live-patching

On 10/15/19 10:13 AM, Miroslav Benes wrote:
> Yes, it does. klp_module_coming() calls module_disable_ro() on all
> patching modules which patch the coming module in order to call
> apply_relocate_add(). New (patching) code for a module can be relocated
> only when the relevant module is loaded.

FWIW, would the LPC blue-sky2 model (ie, Steve's suggestion @ plumber's 
where livepatches only patch a single object and updates are kept on 
disk to handle coming module updates as they are loaded) eliminate those 
outstanding relocations and the need to perform this late permission 
flipping?

-- Joe

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-15 15:06                       ` Joe Lawrence
@ 2019-10-15 15:31                         ` Jessica Yu
  2019-10-15 22:17                           ` Joe Lawrence
  2019-10-16  6:51                         ` Miroslav Benes
  1 sibling, 1 reply; 24+ messages in thread
From: Jessica Yu @ 2019-10-15 15:31 UTC (permalink / raw)
  To: Joe Lawrence
  Cc: Miroslav Benes, Peter Zijlstra, Steven Rostedt, x86,
	linux-kernel, mhiramat, bristot, jbaron, torvalds, tglx, mingo,
	namit, hpa, luto, ard.biesheuvel, jpoimboe, live-patching

+++ Joe Lawrence [15/10/19 11:06 -0400]:
>On 10/15/19 10:13 AM, Miroslav Benes wrote:
>>Yes, it does. klp_module_coming() calls module_disable_ro() on all
>>patching modules which patch the coming module in order to call
>>apply_relocate_add(). New (patching) code for a module can be relocated
>>only when the relevant module is loaded.
>
>FWIW, would the LPC blue-sky2 model (ie, Steve's suggestion @ 
>plumber's where livepatches only patch a single object and updates are 
>kept on disk to handle coming module updates as they are loaded) 
>eliminate those outstanding relocations and the need to perform this 
>late permission flipping?

I wasn't at Plumbers sadly, was this idea documented/talked about in
detail somewhere? (recording, slides, etherpad, etc?). What do you
mean by updates are kept on disk? Maybe someone can explain it more
in detail? :)


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-15 15:31                         ` Jessica Yu
@ 2019-10-15 22:17                           ` Joe Lawrence
  2019-10-15 22:27                             ` Steven Rostedt
  0 siblings, 1 reply; 24+ messages in thread
From: Joe Lawrence @ 2019-10-15 22:17 UTC (permalink / raw)
  To: Jessica Yu
  Cc: Miroslav Benes, Peter Zijlstra, Steven Rostedt, x86,
	linux-kernel, mhiramat, bristot, jbaron, torvalds, tglx, mingo,
	namit, hpa, luto, ard.biesheuvel, jpoimboe, live-patching

On 10/15/19 11:31 AM, Jessica Yu wrote:
> +++ Joe Lawrence [15/10/19 11:06 -0400]:
>> On 10/15/19 10:13 AM, Miroslav Benes wrote:
>>> Yes, it does. klp_module_coming() calls module_disable_ro() on all
>>> patching modules which patch the coming module in order to call
>>> apply_relocate_add(). New (patching) code for a module can be relocated
>>> only when the relevant module is loaded.
>>
>> FWIW, would the LPC blue-sky2 model (ie, Steve's suggestion @
>> plumber's where livepatches only patch a single object and updates are
>> kept on disk to handle coming module updates as they are loaded)
>> eliminate those outstanding relocations and the need to perform this
>> late permission flipping?
> 
> I wasn't at Plumbers sadly, was this idea documented/talked about in
> detail somewhere? (recording, slides, etherpad, etc?). What do you
> mean by updates are kept on disk? Maybe someone can explain it more
> in detail? :)
> 

Livepatching folks -- I don't have the LPC summary link (etherpad?) that 
Jiri put together.  Does someone have that handy for Jessica?

Thanks,

-- Joe

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-15 22:17                           ` Joe Lawrence
@ 2019-10-15 22:27                             ` Steven Rostedt
  2019-10-16  7:42                               ` Peter Zijlstra
  2019-10-16  7:49                               ` Peter Zijlstra
  0 siblings, 2 replies; 24+ messages in thread
From: Steven Rostedt @ 2019-10-15 22:27 UTC (permalink / raw)
  To: Joe Lawrence
  Cc: Jessica Yu, Miroslav Benes, Peter Zijlstra, x86, linux-kernel,
	mhiramat, bristot, jbaron, torvalds, tglx, mingo, namit, hpa,
	luto, ard.biesheuvel, jpoimboe, live-patching

On Tue, 15 Oct 2019 18:17:43 -0400
Joe Lawrence <joe.lawrence@redhat.com> wrote:

> 
> Livepatching folks -- I don't have the LPC summary link (etherpad?) that 
> Jiri put together.  Does someone have that handy for Jessica?

Yes, and I'll be posting this on the LPC web site soon. But here's the
write up that Jiri sent me (with the etherpad link at the end):

Live patching miniconference covered 8 topics overall.

(1) First session, titled "What happened in kernel live patching over the last
year" was led by Miroslav Benes.  It was quite a natural followup to where we
ended at the LPC 2018 miniconf, summarizing which of the points that have been
agreed on back then have already been fully implemented, where obstacles have
been enounctered, etc.

The most prominent feature that has been merged during past year was "atomic
replace", which allows for easier stacking of patches. This is especially
useful for distros, as it naturally aligns with the way patches are being
distributed by them.
Another big step forward since LPC 2018 miniconf was addition of livepatching
selftests, which already tremendously helped in various cases, as it e.g.
helped to track down quite a few issues during development of reliable
stacktraces on s390. Proposal has been made that all major KLP features in the
future should be accompanied by accompanying selftest, which the audience
agreed on.
One of the last year's discussion topics / pain points were GCC optimizations
which are not compatible with livepatching. GCC upstream now has
-flive-patching option, which disables all those interfering optimizations.

(2) Second session, titled "Rethinking late module patching" was led by Miroslav
Benes again.
The problem statement is: in case when there is a patch loaded for module that
is yet to be loaded, it has to be patched before it starts executing. The
current solution relies on hooks in the module loader, and module is patched
when its being linked.  It gets a bit nasty with the arch-specifics of the
module loader handling all the relocations, patching of alternatives, etc. One
of the issues is that all the paravirt / jump label patching has to be done
after relocations are resolved, this is getting a bit fragile and not well
maintainable.
Miroslav sketched out the possible solutions:

	- livepatch would immediately load all the modules for which it has
	  patch via dependency; half-loading modules (not promoting to final
	  LIVE state)
	- splitting the currently one big monolithic livepatch to a per-object
	  structure; might cause issues with consistency model
	- "blue sky" idea from Joe Lawrence: livepatch loaded modules,
	  binary-patch .ko on disk, blacklist vulnerable version

Miroslav proposed to actually stick to the current solution, and improve
selftests coverage for all the considered-fragile arch-specific module linking
code hooks. The discussion then mostly focused, based on proposals from several
attendees (most prominently Steven Rostedt and Amit Shah), on expanding on the
"blue sky" idea.
The final proposal converged to having a separate .ko for livepatches that's
installed on the disk along with the module.  This addresses the module
signature issue (as signature does not actually change), as well as module
removal case (the case where a module was previously loaded while a livepatch
is applied, and then later unloaded and reloaded).  The slight downside is that
this will require changes to the module loader to also look for livepatches
when loading a module.  When unloading the module, the livepatch module will
also need to be unloaded.  Steven approved of this approach over his previous
suggestion.

(3) Third session, titled "Source-based livepatch creation tooling", was led by
Nicolai Stange.
The primary objective of the session was basing on the source-based creation
of livepatches, while avoiding the tedious (and error-prone task) of copying
a lot of kernel code around (from the source tree to the livepatch). Nicolai
spent par of last year writing a klp-ccp (KLP Copy and Paste) utility, which
automates a big chunk of the process.
Nicolai then presented the still open issues with the tool and with the process
around it, most promonent ones being:

	- obtaining original GCC commandline that was used to build the
	  original kernel
	- externalizability of static functions; we need to know whether GCC
	  emitted static function into the patched object

Miroslav proposed to extend existing IPA dumping capabiity of GCC to emit also
the information about dead code elimination; DWARF information is guaranteed
not to be reliable when it comes to IPA optimizations.

(4) Fourth session, titled "Objtool on power -- update", was led by Kamalesh
Babulal.
Kamalesh reported that as a followup to last year's miniconference, the objtool
support for powerpc actually came to life. It hasn't yet been posted upstream,
but is currently available on github [1].
Kamalesh further reported, that decoder has basic functionality (stack
operations + validation, branches, unreachable code, switch table (through gcc
plugin), conditional branches, prologue sequences). It turns out that stack
validation on powerpc is easier than on x86, as the ABI is much more strict
there; which leaves the validation phase to mostly focus on hand-written
assembly.
The next steps are basing on arm64 objtool code which already abstracted out
the arch-specific bits, and further optimizations can be stacked on top of that
(switch table detection, more testing, different gcc versions).

(5) Fifth session, titled "Do we need a Livepatch Developers Guide?", was led
by Joe Lawrence.
Joe postulated, that Current in-kernel documentation provides very good
documentation for individual features the infrastructure provides to the
livepatch author, but Joe further suggested to also include something along the
lines of what they currently have for kpatch, which takes a more general look
from the point of view of livepatch developer.

Proposals that have been brought up for discussion:
    - FAQ
    - collecting already existing CVE fixes and ammend them with a lot of
      commentary
    - creating a livepatch blog on people.kernel.org

Mark Brown asked for documenting what architectures need to implement in order
to support livepatching.
Amit Shah asked if the 'kpatch' and 'kpatch-build' script/program be renamed to
'livepatch'-friendly names so that kernel sources can also reference them for
the user docs part of it.
Both Mark's and Amit's remarks have been considered very valid and useful, and
agreement was reached that they will be taken care of.

(6) Sixth session, titled "API for state changes made by callbacks", was led
by Petr Mladek.

Petr described his proposal for API for changing, updating and disabling
patches (by callbacks). Example where this was needed: L1TF fix, which needed
to change PTE semantics (particular bits). This can't be done before all the
code understands this new PTE format/semantics. Therefore pre-patch and
post-patch callbacks had to do the actual modifications to all the existing
PTEs. What is also currently missing is tracking compatibilities / dependencies
between individual livepatches.
Petr's proposal (v2) is already on ML.
struct klp_state is being introduced which tracks the actual states of the
patch. klp_is_patch_compatible() checks the compatibility of the current states
to the states that the new livepatch is going to bring.
No principal issues / objections have been raised, and it's appreciated by the
patch author(s), so v3 will be submitted and pre-merge bikeshedding will start.

(7) Seventh session, titled "klp-convert and livepatch relocations", was led
by Joe Lawrence.

Joe started the session with problem statement: accessing non exported / static
symbols from inside the patch module. One possible workardound is manually via
kallsyms. Second workaround is klp-convert, which actually creates proper
relocations inside the livepatch module from the symbol database during the
final .ko link.
Currently module loader looks for special livepatch relocations and resolves
those during runtime; kernel support for these relocations have so far been
added for x86 only. Special livepatch relocations are supported and processed
also on other architectures. Special quirks/sections are not yet supported.
Plus klp-convert would still be needed even with late module patching update.
vmlinux or modules could have ambiguous static symbols.

It turns out that the features / bugs below have to be resolved before we
can claim the klp-convert support for relocation complete:
    - handle all the corner cases (jump labels, static keys, ...) properly and
      have a good regression tests in place
    - one day we might (or might not) add support for out-of-tree modules which
      need klp-convert
    - BFD bug 24456 (multiple relocations to the same .text section)

(8) Eight sesstion, titled "Making livepatching infrastructure better", was led
by Kamalesh Babulal.


The primary goal of the discussion as presented by Kamalesh was simple: how to
improve our testing coverage.  Currently we have sample modules + kselftests.
We seem to be currently missing specific unit cases and tests for corner cases.
What Kamalesh would also like to see would be more stress testing oriented
tests for the infrastructure. We should make sure that projects like kernelCI
are running with CONFIG_LIVEPATCH=y.
Another thing Kamalesh currently sees as missing are failure test cases too. 
It should be checked with sosreport and supportconfig guys whether those
diagnostic tools do provide necessary coverage of (at least) livepatching sysfs
state. This is especially a task for distro people to figure out.
Nicolai proposed as one of the testcases identity patching, as that should
reveal issues directly in the infrastructure.

(9) Last, ninth session, titled "Open sourcing live patching services", was led
by Alice Ferrazzi.
This session followed up on previous suggestion of having public repository for
livepatches against LTS kernel.
Alice reported on improviement of elivepatch since last year as having moved
everything to docker.
Alice proposed to more share livepatch sources; SUSE does publish those [2][3],
but it's important to mention that livepatches are very closely tied to
particular kernel version.

[1] https://github.com/kamalesh-babulal/linux/tree/objtool-v1
[2] On https://kernel.suse.com/
[3] Example source-based SUSE's livepatch is at https://kernel.suse.com/cgit/kernel-livepatch/tree/uname_patch/kgr_patch_uname.c?id=d4e00de0b0a3f858fec4e83640f12e1f17298667

Eherpad: https://etherpad.net/p/LPC2019_Live_Patching

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-15 15:06                       ` Joe Lawrence
  2019-10-15 15:31                         ` Jessica Yu
@ 2019-10-16  6:51                         ` Miroslav Benes
  2019-10-16  9:23                           ` Peter Zijlstra
  2019-10-16 12:39                           ` Peter Zijlstra
  1 sibling, 2 replies; 24+ messages in thread
From: Miroslav Benes @ 2019-10-16  6:51 UTC (permalink / raw)
  To: Joe Lawrence
  Cc: Peter Zijlstra, Jessica Yu, Steven Rostedt, x86, linux-kernel,
	mhiramat, bristot, jbaron, torvalds, tglx, mingo, namit, hpa,
	luto, ard.biesheuvel, jpoimboe, live-patching

On Tue, 15 Oct 2019, Joe Lawrence wrote:

> On 10/15/19 10:13 AM, Miroslav Benes wrote:
> > Yes, it does. klp_module_coming() calls module_disable_ro() on all
> > patching modules which patch the coming module in order to call
> > apply_relocate_add(). New (patching) code for a module can be relocated
> > only when the relevant module is loaded.
> 
> FWIW, would the LPC blue-sky2 model (ie, Steve's suggestion @ plumber's where
> livepatches only patch a single object and updates are kept on disk to handle
> coming module updates as they are loaded) eliminate those outstanding
> relocations and the need to perform this late permission flipping?

Yes, it should, but we don't have to wait for it. PeterZ proposed a 
different solution to this specific issue in 
https://lore.kernel.org/lkml/20191015141111.GP2359@hirez.programming.kicks-ass.net/

It should not be a problem to create a live patch module like that and the 
code in kernel/livepatch/ is almost ready. Something like 
module_section_disable_ro(mod, section) (and similar for X protection) 
should be enough. Module reloads would still require juggling with the 
protections, but I think it is all feasible.

Miroslav

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-15 22:27                             ` Steven Rostedt
@ 2019-10-16  7:42                               ` Peter Zijlstra
  2019-10-16 10:15                                 ` Miroslav Benes
  2019-10-21 15:05                                 ` Josh Poimboeuf
  2019-10-16  7:49                               ` Peter Zijlstra
  1 sibling, 2 replies; 24+ messages in thread
From: Peter Zijlstra @ 2019-10-16  7:42 UTC (permalink / raw)
  To: Steven Rostedt
  Cc: Joe Lawrence, Jessica Yu, Miroslav Benes, x86, linux-kernel,
	mhiramat, bristot, jbaron, torvalds, tglx, mingo, namit, hpa,
	luto, ard.biesheuvel, jpoimboe, live-patching

> which are not compatible with livepatching. GCC upstream now has
> -flive-patching option, which disables all those interfering optimizations.

Which, IIRC, has a significant performance impact and should thus really
not be used...

If distros ship that crap, I'm going to laugh at them the next time they
want a single digit performance improvement because *important*.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-15 22:27                             ` Steven Rostedt
  2019-10-16  7:42                               ` Peter Zijlstra
@ 2019-10-16  7:49                               ` Peter Zijlstra
  2019-10-16 10:20                                 ` Miroslav Benes
  1 sibling, 1 reply; 24+ messages in thread
From: Peter Zijlstra @ 2019-10-16  7:49 UTC (permalink / raw)
  To: Steven Rostedt
  Cc: Joe Lawrence, Jessica Yu, Miroslav Benes, x86, linux-kernel,
	mhiramat, bristot, jbaron, torvalds, tglx, mingo, namit, hpa,
	luto, ard.biesheuvel, jpoimboe, live-patching

On Tue, Oct 15, 2019 at 06:27:05PM -0400, Steven Rostedt wrote:

> (7) Seventh session, titled "klp-convert and livepatch relocations", was led
> by Joe Lawrence.
> 
> Joe started the session with problem statement: accessing non exported / static
> symbols from inside the patch module. One possible workardound is manually via
> kallsyms. Second workaround is klp-convert, which actually creates proper
> relocations inside the livepatch module from the symbol database during the
> final .ko link.
> Currently module loader looks for special livepatch relocations and resolves
> those during runtime; kernel support for these relocations have so far been
> added for x86 only. Special livepatch relocations are supported and processed
> also on other architectures. Special quirks/sections are not yet supported.
> Plus klp-convert would still be needed even with late module patching update.
> vmlinux or modules could have ambiguous static symbols.
> 
> It turns out that the features / bugs below have to be resolved before we
> can claim the klp-convert support for relocation complete:
>     - handle all the corner cases (jump labels, static keys, ...) properly and
>       have a good regression tests in place

I suppose all the patches in this series-of-series here will make life
harder for KLP, static_call() and 2 byte jumps etc..

>     - one day we might (or might not) add support for out-of-tree modules which
>       need klp-convert
>     - BFD bug 24456 (multiple relocations to the same .text section)



^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-16  6:51                         ` Miroslav Benes
@ 2019-10-16  9:23                           ` Peter Zijlstra
  2019-10-16  9:36                             ` Jessica Yu
  2019-10-16 12:39                           ` Peter Zijlstra
  1 sibling, 1 reply; 24+ messages in thread
From: Peter Zijlstra @ 2019-10-16  9:23 UTC (permalink / raw)
  To: Miroslav Benes
  Cc: Joe Lawrence, Jessica Yu, Steven Rostedt, x86, linux-kernel,
	mhiramat, bristot, jbaron, torvalds, tglx, mingo, namit, hpa,
	luto, ard.biesheuvel, jpoimboe, live-patching

On Wed, Oct 16, 2019 at 08:51:27AM +0200, Miroslav Benes wrote:
> On Tue, 15 Oct 2019, Joe Lawrence wrote:
> 
> > On 10/15/19 10:13 AM, Miroslav Benes wrote:
> > > Yes, it does. klp_module_coming() calls module_disable_ro() on all
> > > patching modules which patch the coming module in order to call
> > > apply_relocate_add(). New (patching) code for a module can be relocated
> > > only when the relevant module is loaded.
> > 
> > FWIW, would the LPC blue-sky2 model (ie, Steve's suggestion @ plumber's where
> > livepatches only patch a single object and updates are kept on disk to handle
> > coming module updates as they are loaded) eliminate those outstanding
> > relocations and the need to perform this late permission flipping?
> 
> Yes, it should, but we don't have to wait for it. PeterZ proposed a 
> different solution to this specific issue in 
> https://lore.kernel.org/lkml/20191015141111.GP2359@hirez.programming.kicks-ass.net/
> 
> It should not be a problem to create a live patch module like that and the 
> code in kernel/livepatch/ is almost ready. Something like 
> module_section_disable_ro(mod, section) (and similar for X protection) 
> should be enough. Module reloads would still require juggling with the 
> protections, but I think it is all feasible.

Just had a browse around the module code, and while the section info is
in load_info, it doesn't get retained for active modules.

So I suppose I'll go add that for KLP enabled things.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-16  9:23                           ` Peter Zijlstra
@ 2019-10-16  9:36                             ` Jessica Yu
  2019-10-16  9:51                               ` Peter Zijlstra
  0 siblings, 1 reply; 24+ messages in thread
From: Jessica Yu @ 2019-10-16  9:36 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Miroslav Benes, Joe Lawrence, Steven Rostedt, x86, linux-kernel,
	mhiramat, bristot, jbaron, torvalds, tglx, mingo, namit, hpa,
	luto, ard.biesheuvel, jpoimboe, live-patching

+++ Peter Zijlstra [16/10/19 11:23 +0200]:
>On Wed, Oct 16, 2019 at 08:51:27AM +0200, Miroslav Benes wrote:
>> On Tue, 15 Oct 2019, Joe Lawrence wrote:
>>
>> > On 10/15/19 10:13 AM, Miroslav Benes wrote:
>> > > Yes, it does. klp_module_coming() calls module_disable_ro() on all
>> > > patching modules which patch the coming module in order to call
>> > > apply_relocate_add(). New (patching) code for a module can be relocated
>> > > only when the relevant module is loaded.
>> >
>> > FWIW, would the LPC blue-sky2 model (ie, Steve's suggestion @ plumber's where
>> > livepatches only patch a single object and updates are kept on disk to handle
>> > coming module updates as they are loaded) eliminate those outstanding
>> > relocations and the need to perform this late permission flipping?
>>
>> Yes, it should, but we don't have to wait for it. PeterZ proposed a
>> different solution to this specific issue in
>> https://lore.kernel.org/lkml/20191015141111.GP2359@hirez.programming.kicks-ass.net/
>>
>> It should not be a problem to create a live patch module like that and the
>> code in kernel/livepatch/ is almost ready. Something like
>> module_section_disable_ro(mod, section) (and similar for X protection)
>> should be enough. Module reloads would still require juggling with the
>> protections, but I think it is all feasible.
>
>Just had a browse around the module code, and while the section info is
>in load_info, it doesn't get retained for active modules.
>
>So I suppose I'll go add that for KLP enabled things.

Elf section info does get saved for livepatch modules though, see
mod->klp_info. And wouldn't this only be needed for livepatch modules?

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-16  9:36                             ` Jessica Yu
@ 2019-10-16  9:51                               ` Peter Zijlstra
  0 siblings, 0 replies; 24+ messages in thread
From: Peter Zijlstra @ 2019-10-16  9:51 UTC (permalink / raw)
  To: Jessica Yu
  Cc: Miroslav Benes, Joe Lawrence, Steven Rostedt, x86, linux-kernel,
	mhiramat, bristot, jbaron, torvalds, tglx, mingo, namit, hpa,
	luto, ard.biesheuvel, jpoimboe, live-patching

On Wed, Oct 16, 2019 at 11:36:10AM +0200, Jessica Yu wrote:
> +++ Peter Zijlstra [16/10/19 11:23 +0200]:
> > On Wed, Oct 16, 2019 at 08:51:27AM +0200, Miroslav Benes wrote:
> > > On Tue, 15 Oct 2019, Joe Lawrence wrote:
> > > 
> > > > On 10/15/19 10:13 AM, Miroslav Benes wrote:
> > > > > Yes, it does. klp_module_coming() calls module_disable_ro() on all
> > > > > patching modules which patch the coming module in order to call
> > > > > apply_relocate_add(). New (patching) code for a module can be relocated
> > > > > only when the relevant module is loaded.
> > > >
> > > > FWIW, would the LPC blue-sky2 model (ie, Steve's suggestion @ plumber's where
> > > > livepatches only patch a single object and updates are kept on disk to handle
> > > > coming module updates as they are loaded) eliminate those outstanding
> > > > relocations and the need to perform this late permission flipping?
> > > 
> > > Yes, it should, but we don't have to wait for it. PeterZ proposed a
> > > different solution to this specific issue in
> > > https://lore.kernel.org/lkml/20191015141111.GP2359@hirez.programming.kicks-ass.net/
> > > 
> > > It should not be a problem to create a live patch module like that and the
> > > code in kernel/livepatch/ is almost ready. Something like
> > > module_section_disable_ro(mod, section) (and similar for X protection)
> > > should be enough. Module reloads would still require juggling with the
> > > protections, but I think it is all feasible.
> > 
> > Just had a browse around the module code, and while the section info is
> > in load_info, it doesn't get retained for active modules.
> > 
> > So I suppose I'll go add that for KLP enabled things.
> 
> Elf section info does get saved for livepatch modules though, see
> mod->klp_info. And wouldn't this only be needed for livepatch modules?

Right, I just found that, but it is x86 only for some mysterious reason.
And yes, it's KLP only.

I was thikning of adding a KLP only list of {name,addr,size} sections
that start with ".klp" in layout_sections(). Would that not work across
architectures?

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-16  7:42                               ` Peter Zijlstra
@ 2019-10-16 10:15                                 ` Miroslav Benes
  2019-10-21 15:05                                 ` Josh Poimboeuf
  1 sibling, 0 replies; 24+ messages in thread
From: Miroslav Benes @ 2019-10-16 10:15 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Steven Rostedt, Joe Lawrence, Jessica Yu, x86, linux-kernel,
	mhiramat, bristot, jbaron, torvalds, tglx, mingo, namit, hpa,
	luto, ard.biesheuvel, jpoimboe, live-patching

On Wed, 16 Oct 2019, Peter Zijlstra wrote:

> > which are not compatible with livepatching. GCC upstream now has
> > -flive-patching option, which disables all those interfering optimizations.
> 
> Which, IIRC, has a significant performance impact and should thus really
> not be used...

Fortunately, the impact is negligible.

Miroslav

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-16  7:49                               ` Peter Zijlstra
@ 2019-10-16 10:20                                 ` Miroslav Benes
  2019-10-16 13:29                                   ` Miroslav Benes
  0 siblings, 1 reply; 24+ messages in thread
From: Miroslav Benes @ 2019-10-16 10:20 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Steven Rostedt, Joe Lawrence, Jessica Yu, x86, linux-kernel,
	mhiramat, bristot, jbaron, torvalds, tglx, mingo, namit, hpa,
	luto, ard.biesheuvel, jpoimboe, live-patching, pmladek

On Wed, 16 Oct 2019, Peter Zijlstra wrote:

> On Tue, Oct 15, 2019 at 06:27:05PM -0400, Steven Rostedt wrote:
> 
> > (7) Seventh session, titled "klp-convert and livepatch relocations", was led
> > by Joe Lawrence.
> > 
> > Joe started the session with problem statement: accessing non exported / static
> > symbols from inside the patch module. One possible workardound is manually via
> > kallsyms. Second workaround is klp-convert, which actually creates proper
> > relocations inside the livepatch module from the symbol database during the
> > final .ko link.
> > Currently module loader looks for special livepatch relocations and resolves
> > those during runtime; kernel support for these relocations have so far been
> > added for x86 only. Special livepatch relocations are supported and processed
> > also on other architectures. Special quirks/sections are not yet supported.
> > Plus klp-convert would still be needed even with late module patching update.
> > vmlinux or modules could have ambiguous static symbols.
> > 
> > It turns out that the features / bugs below have to be resolved before we
> > can claim the klp-convert support for relocation complete:
> >     - handle all the corner cases (jump labels, static keys, ...) properly and
> >       have a good regression tests in place
> 
> I suppose all the patches in this series-of-series here will make life
> harder for KLP, static_call() and 2 byte jumps etc..

Yes, I think so. We'll have to deal with that once it lands. That is why 
we want to get rid of all this arch-specific code in livepatch and 
reinvent the late module patching. So it is perhaps better to start 
working on it sooner than later. Adding Petr, who hesitantly signed up for 
the task...

Miroslav

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-16  6:51                         ` Miroslav Benes
  2019-10-16  9:23                           ` Peter Zijlstra
@ 2019-10-16 12:39                           ` Peter Zijlstra
  2019-10-22  8:45                             ` Miroslav Benes
  1 sibling, 1 reply; 24+ messages in thread
From: Peter Zijlstra @ 2019-10-16 12:39 UTC (permalink / raw)
  To: Miroslav Benes
  Cc: Joe Lawrence, Jessica Yu, Steven Rostedt, x86, linux-kernel,
	mhiramat, bristot, jbaron, torvalds, tglx, mingo, namit, hpa,
	luto, ard.biesheuvel, jpoimboe, live-patching

On Wed, Oct 16, 2019 at 08:51:27AM +0200, Miroslav Benes wrote:
> On Tue, 15 Oct 2019, Joe Lawrence wrote:
> 
> > On 10/15/19 10:13 AM, Miroslav Benes wrote:
> > > Yes, it does. klp_module_coming() calls module_disable_ro() on all
> > > patching modules which patch the coming module in order to call
> > > apply_relocate_add(). New (patching) code for a module can be relocated
> > > only when the relevant module is loaded.
> > 
> > FWIW, would the LPC blue-sky2 model (ie, Steve's suggestion @ plumber's where
> > livepatches only patch a single object and updates are kept on disk to handle
> > coming module updates as they are loaded) eliminate those outstanding
> > relocations and the need to perform this late permission flipping?
> 
> Yes, it should, but we don't have to wait for it. PeterZ proposed a 
> different solution to this specific issue in 
> https://lore.kernel.org/lkml/20191015141111.GP2359@hirez.programming.kicks-ass.net/
> 
> It should not be a problem to create a live patch module like that and the 
> code in kernel/livepatch/ is almost ready. Something like 
> module_section_disable_ro(mod, section) (and similar for X protection) 
> should be enough. Module reloads would still require juggling with the 
> protections, but I think it is all feasible.

Something a little like so.. completely fresh of the keyboard.

---
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -853,6 +853,18 @@ static inline void module_enable_ro(cons
 static inline void module_disable_ro(const struct module *mod) { }
 #endif
 
+#if defined(CONFIG_STRICT_MODULE_RWX) && defined(CONFIG_LIVEPATCH)
+extern void module_section_disable_ro(struct module *mod, const char *sec);
+extern void module_section_enable_ro(struct module *mod, const char *sec);
+extern void module_section_disable_x(struct module *mod, const char *sec);
+extern void module_section_enable_x(struct module *mod, const char *sec);
+#else
+static inline void module_section_disable_ro(struct module *mod, const char *sec) { }
+static inline void module_section_enable_ro(struct module *mod, const char *sec) { }
+static inline void module_section_disable_x(struct module *mod, const char *sec) { }
+static inline void module_section_enable_x(struct module *mod, const char *sec) { }
+#endif
+
 #ifdef CONFIG_GENERIC_BUG
 void module_bug_finalize(const Elf_Ehdr *, const Elf_Shdr *,
 			 struct module *);
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -2107,6 +2107,54 @@ static void free_module_elf(struct modul
 	kfree(mod->klp_info->secstrings);
 	kfree(mod->klp_info);
 }
+
+#ifdef CONFIG_STRICT_MODULE_RWX
+
+static void __frob_section(struct Elf_Shdr *sec, int (*set_memory)(unsigned long start, int num_pages))
+{
+	BUG_ON((unsigned long)sec->sh_addr & (PAGE_SIZE-1));
+	BUG_ON((unsigned long)sec->sh_size & (PAGE_SIZE-1));
+	set_memory((unsigned long)sec->sh_addr, sec->sh_size >> PAGE_SHIFT);
+}
+
+static void frob_section(struct module *mod, const char *section,
+			 int (*set_memory)(unsigned long start, int num_pages))
+{
+	struct klp_modinfo *info = mod->klp_info;
+	const char *secname;
+	Elf_Shdr *s;
+
+	for (s = info->sechdrs; s < info->sechdrs + info->hdr.e_shnum; s++) {
+		secname = mod->klp_info->secstrings + s->sh_name;
+		if (strcmp(secname, section))
+			continue;
+
+		__frob_section(s, set_memory);
+	}
+}
+
+void module_section_disable_ro(struct module *mod, const char *section)
+{
+	frob_section(mod, section, set_memory_rw);
+}
+
+void module_section_enable_ro(struct module *mod, const char *section)
+{
+	frob_section(mod, section, set_memory_ro);
+}
+
+void module_section_disable_x(struct module *mod, const char *section)
+{
+	frob_section(mod, section, set_memory_nx);
+}
+
+void module_section_enable_x(struct module *mod, const char *section)
+{
+	frob_section(mod, section, set_memory_x);
+}
+
+#endif /* ONFIG_STRICT_MODULE_RWX */
+
 #else /* !CONFIG_LIVEPATCH */
 static int copy_module_elf(struct module *mod, struct load_info *info)
 {

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-16 10:20                                 ` Miroslav Benes
@ 2019-10-16 13:29                                   ` Miroslav Benes
  2019-10-18 13:03                                     ` Jessica Yu
  0 siblings, 1 reply; 24+ messages in thread
From: Miroslav Benes @ 2019-10-16 13:29 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Steven Rostedt, Joe Lawrence, Jessica Yu, x86, linux-kernel,
	mhiramat, bristot, jbaron, torvalds, tglx, mingo, namit, hpa,
	luto, ard.biesheuvel, jpoimboe, live-patching, pmladek

On Wed, 16 Oct 2019, Miroslav Benes wrote:

> On Wed, 16 Oct 2019, Peter Zijlstra wrote:
> 
> > On Tue, Oct 15, 2019 at 06:27:05PM -0400, Steven Rostedt wrote:
> > 
> > > (7) Seventh session, titled "klp-convert and livepatch relocations", was led
> > > by Joe Lawrence.
> > > 
> > > Joe started the session with problem statement: accessing non exported / static
> > > symbols from inside the patch module. One possible workardound is manually via
> > > kallsyms. Second workaround is klp-convert, which actually creates proper
> > > relocations inside the livepatch module from the symbol database during the
> > > final .ko link.
> > > Currently module loader looks for special livepatch relocations and resolves
> > > those during runtime; kernel support for these relocations have so far been
> > > added for x86 only. Special livepatch relocations are supported and processed
> > > also on other architectures. Special quirks/sections are not yet supported.
> > > Plus klp-convert would still be needed even with late module patching update.
> > > vmlinux or modules could have ambiguous static symbols.
> > > 
> > > It turns out that the features / bugs below have to be resolved before we
> > > can claim the klp-convert support for relocation complete:
> > >     - handle all the corner cases (jump labels, static keys, ...) properly and
> > >       have a good regression tests in place
> > 
> > I suppose all the patches in this series-of-series here will make life
> > harder for KLP, static_call() and 2 byte jumps etc..
> 
> Yes, I think so. We'll have to deal with that once it lands. That is why 
> we want to get rid of all this arch-specific code in livepatch and 
> reinvent the late module patching. So it is perhaps better to start 
> working on it sooner than later. Adding Petr, who hesitantly signed up for 
> the task...

Thinking about it more... crazy idea. I think we could leverage these new 
ELF .text per vmlinux/module sections for the reinvention I was talking 
about. If we teach module loader to relocate (and apply alternatives and 
so on, everything in arch-specific module_finalize()) not the whole module 
in case of live patch modules, but separate ELF .text sections, it could 
solve the issue with late module patching we have. It is a variation on 
Steven's idea. When live patch module is loaded, only its section for 
present modules would be processed. Then whenever a to-be-patched module 
is loaded, its .text section in all present patch module would be 
processed.

The upside is that almost no work would be required on patch modules 
creation side. The downside is that klp_modinfo must stay. Module loader 
needs to be hacked a lot in both cases. So it remains to be seen which 
idea is easier to implement.

Jessica, do you think it would be feasible?

Petr, Joe, Josh, am I missing something or would it work?

Miroslav

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-16 13:29                                   ` Miroslav Benes
@ 2019-10-18 13:03                                     ` Jessica Yu
  2019-10-18 13:40                                       ` Petr Mladek
  2019-10-22  8:27                                       ` Miroslav Benes
  0 siblings, 2 replies; 24+ messages in thread
From: Jessica Yu @ 2019-10-18 13:03 UTC (permalink / raw)
  To: Miroslav Benes
  Cc: Peter Zijlstra, Steven Rostedt, Joe Lawrence, x86, linux-kernel,
	mhiramat, bristot, jbaron, torvalds, tglx, mingo, namit, hpa,
	luto, ard.biesheuvel, jpoimboe, live-patching, pmladek

+++ Miroslav Benes [16/10/19 15:29 +0200]:
>On Wed, 16 Oct 2019, Miroslav Benes wrote:
>
>> On Wed, 16 Oct 2019, Peter Zijlstra wrote:
>>
>> > On Tue, Oct 15, 2019 at 06:27:05PM -0400, Steven Rostedt wrote:
>> >
>> > > (7) Seventh session, titled "klp-convert and livepatch relocations", was led
>> > > by Joe Lawrence.
>> > >
>> > > Joe started the session with problem statement: accessing non exported / static
>> > > symbols from inside the patch module. One possible workardound is manually via
>> > > kallsyms. Second workaround is klp-convert, which actually creates proper
>> > > relocations inside the livepatch module from the symbol database during the
>> > > final .ko link.
>> > > Currently module loader looks for special livepatch relocations and resolves
>> > > those during runtime; kernel support for these relocations have so far been
>> > > added for x86 only. Special livepatch relocations are supported and processed
>> > > also on other architectures. Special quirks/sections are not yet supported.
>> > > Plus klp-convert would still be needed even with late module patching update.
>> > > vmlinux or modules could have ambiguous static symbols.
>> > >
>> > > It turns out that the features / bugs below have to be resolved before we
>> > > can claim the klp-convert support for relocation complete:
>> > >     - handle all the corner cases (jump labels, static keys, ...) properly and
>> > >       have a good regression tests in place
>> >
>> > I suppose all the patches in this series-of-series here will make life
>> > harder for KLP, static_call() and 2 byte jumps etc..
>>
>> Yes, I think so. We'll have to deal with that once it lands. That is why
>> we want to get rid of all this arch-specific code in livepatch and
>> reinvent the late module patching. So it is perhaps better to start
>> working on it sooner than later. Adding Petr, who hesitantly signed up for
>> the task...
>
>Thinking about it more... crazy idea. I think we could leverage these new
>ELF .text per vmlinux/module sections for the reinvention I was talking
>about. If we teach module loader to relocate (and apply alternatives and
>so on, everything in arch-specific module_finalize()) not the whole module
>in case of live patch modules, but separate ELF .text sections, it could
>solve the issue with late module patching we have. It is a variation on
>Steven's idea. When live patch module is loaded, only its section for
>present modules would be processed. Then whenever a to-be-patched module
>is loaded, its .text section in all present patch module would be
>processed.
>
>The upside is that almost no work would be required on patch modules
>creation side. The downside is that klp_modinfo must stay. Module loader
>needs to be hacked a lot in both cases. So it remains to be seen which
>idea is easier to implement.
>
>Jessica, do you think it would be feasible?

I think that does sound feasible. I'm trying to visualize how that
would look. I guess there would need to be various livepatching hooks
called during the different stages (apply_relocate_add(),
module_finalize(), module_enable_ro/x()).

So maybe something like the following?

When a livepatch module loads:
    apply_relocate_add()
        klp hook: apply .klp.rela.$objname relocations *only* for
        already loaded modules
    module_finalize()
        klp hook: apply .klp.arch.$objname changes for already loaded modules
    module_enable_ro()
        klp hook: only enable ro/x for .klp.text.$objname for already
        loaded modules

When a to-be-patched module loads:
    apply_relocate_add()
        klp hook: for each patch module that patches the coming
        module, apply .klp.rela.$objname relocations for this object
    module_finalize()
        klp hook: for each patch module that patches the coming
        module, apply .klp.arch.$objname changes for this object
    module_enable_ro()
        klp hook: for each patch module, apply ro/x permissions for
        .klp.text.$objname for this object

Then, in klp_module_coming, we only need to do the callbacks and
enable the patch, and get rid of the module_disable_ro->apply
relocs->module_enable_ro block.

Does that sound like what you had in mind or am I totally off?

Thanks!

Jessica


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-18 13:03                                     ` Jessica Yu
@ 2019-10-18 13:40                                       ` Petr Mladek
  2019-10-21 14:14                                         ` Jessica Yu
  2019-10-21 15:31                                         ` Josh Poimboeuf
  2019-10-22  8:27                                       ` Miroslav Benes
  1 sibling, 2 replies; 24+ messages in thread
From: Petr Mladek @ 2019-10-18 13:40 UTC (permalink / raw)
  To: Jessica Yu
  Cc: Miroslav Benes, Peter Zijlstra, Steven Rostedt, Joe Lawrence,
	x86, linux-kernel, mhiramat, bristot, jbaron, torvalds, tglx,
	mingo, namit, hpa, luto, ard.biesheuvel, jpoimboe, live-patching

On Fri 2019-10-18 15:03:42, Jessica Yu wrote:
> +++ Miroslav Benes [16/10/19 15:29 +0200]:
> > On Wed, 16 Oct 2019, Miroslav Benes wrote:
> > Thinking about it more... crazy idea. I think we could leverage these new
> > ELF .text per vmlinux/module sections for the reinvention I was talking
> > about. If we teach module loader to relocate (and apply alternatives and
> > so on, everything in arch-specific module_finalize()) not the whole module
> > in case of live patch modules, but separate ELF .text sections, it could
> > solve the issue with late module patching we have. It is a variation on
> > Steven's idea. When live patch module is loaded, only its section for
> > present modules would be processed. Then whenever a to-be-patched module
> > is loaded, its .text section in all present patch module would be
> > processed.
> > 
> > The upside is that almost no work would be required on patch modules
> > creation side. The downside is that klp_modinfo must stay. Module loader
> > needs to be hacked a lot in both cases. So it remains to be seen which
> > idea is easier to implement.
> > 
> > Jessica, do you think it would be feasible?
> 
> I think that does sound feasible. I'm trying to visualize how that
> would look. I guess there would need to be various livepatching hooks
> called during the different stages (apply_relocate_add(),
> module_finalize(), module_enable_ro/x()).
> 
> So maybe something like the following?
> 
> When a livepatch module loads:
>    apply_relocate_add()
>        klp hook: apply .klp.rela.$objname relocations *only* for
>        already loaded modules
>    module_finalize()
>        klp hook: apply .klp.arch.$objname changes for already loaded modules
>    module_enable_ro()
>        klp hook: only enable ro/x for .klp.text.$objname for already
>        loaded modules

Just for record. We should also set ro for the not-yet used
.klp.text.$objname at this stage so that it can't be modified
easily "by accident".


> When a to-be-patched module loads:
>    apply_relocate_add()
>        klp hook: for each patch module that patches the coming
>        module, apply .klp.rela.$objname relocations for this object
>    module_finalize()
>        klp hook: for each patch module that patches the coming
>        module, apply .klp.arch.$objname changes for this object
>    module_enable_ro()
>        klp hook: for each patch module, apply ro/x permissions for
>        .klp.text.$objname for this object
> 
> Then, in klp_module_coming, we only need to do the callbacks and
> enable the patch, and get rid of the module_disable_ro->apply
> relocs->module_enable_ro block.
> 
> Does that sound like what you had in mind or am I totally off?

Makes sense to me.

Well, I wonder if it is really any better from what we have now.
We would still need special delayed handling for the module-specific
elf sections. Also we still would not need to clear the modifications
in these sections when the livepatched object gets unloaded.

I am afraid that the real difference might come when we split
the livepatch into per-livepatched object modules. This would
move the complexity to another parts of the code ;-) I am
unable to say what approach is easier and more safe to maintain
at the moment.

Best Regards,
Petr

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-18 13:40                                       ` Petr Mladek
@ 2019-10-21 14:14                                         ` Jessica Yu
  2019-10-21 15:31                                         ` Josh Poimboeuf
  1 sibling, 0 replies; 24+ messages in thread
From: Jessica Yu @ 2019-10-21 14:14 UTC (permalink / raw)
  To: Petr Mladek
  Cc: Miroslav Benes, Peter Zijlstra, Steven Rostedt, Joe Lawrence,
	x86, linux-kernel, mhiramat, bristot, jbaron, torvalds, tglx,
	mingo, namit, hpa, luto, ard.biesheuvel, jpoimboe, live-patching

+++ Petr Mladek [18/10/19 15:40 +0200]:
>On Fri 2019-10-18 15:03:42, Jessica Yu wrote:
>> +++ Miroslav Benes [16/10/19 15:29 +0200]:
>> > On Wed, 16 Oct 2019, Miroslav Benes wrote:
>> > Thinking about it more... crazy idea. I think we could leverage these new
>> > ELF .text per vmlinux/module sections for the reinvention I was talking
>> > about. If we teach module loader to relocate (and apply alternatives and
>> > so on, everything in arch-specific module_finalize()) not the whole module
>> > in case of live patch modules, but separate ELF .text sections, it could
>> > solve the issue with late module patching we have. It is a variation on
>> > Steven's idea. When live patch module is loaded, only its section for
>> > present modules would be processed. Then whenever a to-be-patched module
>> > is loaded, its .text section in all present patch module would be
>> > processed.
>> >
>> > The upside is that almost no work would be required on patch modules
>> > creation side. The downside is that klp_modinfo must stay. Module loader
>> > needs to be hacked a lot in both cases. So it remains to be seen which
>> > idea is easier to implement.
>> >
>> > Jessica, do you think it would be feasible?
>>
>> I think that does sound feasible. I'm trying to visualize how that
>> would look. I guess there would need to be various livepatching hooks
>> called during the different stages (apply_relocate_add(),
>> module_finalize(), module_enable_ro/x()).
>>
>> So maybe something like the following?
>>
>> When a livepatch module loads:
>>    apply_relocate_add()
>>        klp hook: apply .klp.rela.$objname relocations *only* for
>>        already loaded modules
>>    module_finalize()
>>        klp hook: apply .klp.arch.$objname changes for already loaded modules
>>    module_enable_ro()
>>        klp hook: only enable ro/x for .klp.text.$objname for already
>>        loaded modules
>
>Just for record. We should also set ro for the not-yet used
>.klp.text.$objname at this stage so that it can't be modified
>easily "by accident".

If we also set ro protection already for .klp.text.$objname for
not-yet loaded modules, I think this would unfortunately mean we would
still have to do the protection flipping for late module patching that
Peter was trying to avoid, right?

That is, we *still* end up having to do the whole module_disable_ro()
-> apply_relocate_add() -> module_finalize() -> module_enable_ro()
thing for late module patching, except now we've moved that work to
the module loader instead of in klp_module_coming.. It sounds just as
complicated as the current way :/

However, I think this complaint would not apply if livepatch switches
to the one patch module per module model..


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-16  7:42                               ` Peter Zijlstra
  2019-10-16 10:15                                 ` Miroslav Benes
@ 2019-10-21 15:05                                 ` Josh Poimboeuf
  1 sibling, 0 replies; 24+ messages in thread
From: Josh Poimboeuf @ 2019-10-21 15:05 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Steven Rostedt, Joe Lawrence, Jessica Yu, Miroslav Benes, x86,
	linux-kernel, mhiramat, bristot, jbaron, torvalds, tglx, mingo,
	namit, hpa, luto, ard.biesheuvel, live-patching

On Wed, Oct 16, 2019 at 09:42:17AM +0200, Peter Zijlstra wrote:
> > which are not compatible with livepatching. GCC upstream now has
> > -flive-patching option, which disables all those interfering optimizations.
> 
> Which, IIRC, has a significant performance impact and should thus really
> not be used...
> 
> If distros ship that crap, I'm going to laugh at them the next time they
> want a single digit performance improvement because *important*.

I have a crazy plan to try to use objtool to detect function changes at
a binary level, which would hopefully allow us to drop this flag.

But regardless, I wonder if we enabled this flag prematurely.  We still
don't have a reasonable way to use it for creating source-based live
patches upstream, and it should really be optional for CONFIG_LIVEPATCH,
since kpatch-build doesn't need it.

-- 
Josh


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-18 13:40                                       ` Petr Mladek
  2019-10-21 14:14                                         ` Jessica Yu
@ 2019-10-21 15:31                                         ` Josh Poimboeuf
  1 sibling, 0 replies; 24+ messages in thread
From: Josh Poimboeuf @ 2019-10-21 15:31 UTC (permalink / raw)
  To: Petr Mladek
  Cc: Jessica Yu, Miroslav Benes, Peter Zijlstra, Steven Rostedt,
	Joe Lawrence, x86, linux-kernel, mhiramat, bristot, jbaron,
	torvalds, tglx, mingo, namit, hpa, luto, ard.biesheuvel,
	live-patching

On Fri, Oct 18, 2019 at 03:40:58PM +0200, Petr Mladek wrote:
> On Fri 2019-10-18 15:03:42, Jessica Yu wrote:
> > +++ Miroslav Benes [16/10/19 15:29 +0200]:
> > > On Wed, 16 Oct 2019, Miroslav Benes wrote:
> > > Thinking about it more... crazy idea. I think we could leverage these new
> > > ELF .text per vmlinux/module sections for the reinvention I was talking
> > > about. If we teach module loader to relocate (and apply alternatives and
> > > so on, everything in arch-specific module_finalize()) not the whole module
> > > in case of live patch modules, but separate ELF .text sections, it could
> > > solve the issue with late module patching we have. It is a variation on
> > > Steven's idea. When live patch module is loaded, only its section for
> > > present modules would be processed. Then whenever a to-be-patched module
> > > is loaded, its .text section in all present patch module would be
> > > processed.
> > > 
> > > The upside is that almost no work would be required on patch modules
> > > creation side. The downside is that klp_modinfo must stay. Module loader
> > > needs to be hacked a lot in both cases. So it remains to be seen which
> > > idea is easier to implement.
> > > 
> > > Jessica, do you think it would be feasible?
> > 
> > I think that does sound feasible. I'm trying to visualize how that
> > would look. I guess there would need to be various livepatching hooks
> > called during the different stages (apply_relocate_add(),
> > module_finalize(), module_enable_ro/x()).
> > 
> > So maybe something like the following?
> > 
> > When a livepatch module loads:
> >    apply_relocate_add()
> >        klp hook: apply .klp.rela.$objname relocations *only* for
> >        already loaded modules
> >    module_finalize()
> >        klp hook: apply .klp.arch.$objname changes for already loaded modules
> >    module_enable_ro()
> >        klp hook: only enable ro/x for .klp.text.$objname for already
> >        loaded modules
> 
> Just for record. We should also set ro for the not-yet used
> .klp.text.$objname at this stage so that it can't be modified
> easily "by accident".
> 
> 
> > When a to-be-patched module loads:
> >    apply_relocate_add()
> >        klp hook: for each patch module that patches the coming
> >        module, apply .klp.rela.$objname relocations for this object
> >    module_finalize()
> >        klp hook: for each patch module that patches the coming
> >        module, apply .klp.arch.$objname changes for this object
> >    module_enable_ro()
> >        klp hook: for each patch module, apply ro/x permissions for
> >        .klp.text.$objname for this object
> > 
> > Then, in klp_module_coming, we only need to do the callbacks and
> > enable the patch, and get rid of the module_disable_ro->apply
> > relocs->module_enable_ro block.
> > 
> > Does that sound like what you had in mind or am I totally off?
> 
> Makes sense to me.
> 
> Well, I wonder if it is really any better from what we have now.

AFAICT, this would still have a lot of the same problems we have today.
It has a lot of complexity.  It needs arch-specific livepatch code and
sections, and introduces special cases in the module code.

I'd much prefer the proposal from LPC to have per-module live patches.
It's simpler and has less things that can go wrong IMO.

-- 
Josh


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-18 13:03                                     ` Jessica Yu
  2019-10-18 13:40                                       ` Petr Mladek
@ 2019-10-22  8:27                                       ` Miroslav Benes
  2019-10-22 14:31                                         ` Josh Poimboeuf
  1 sibling, 1 reply; 24+ messages in thread
From: Miroslav Benes @ 2019-10-22  8:27 UTC (permalink / raw)
  To: Jessica Yu
  Cc: Peter Zijlstra, Steven Rostedt, Joe Lawrence, x86, linux-kernel,
	mhiramat, bristot, jbaron, torvalds, tglx, mingo, namit, hpa,
	luto, ard.biesheuvel, jpoimboe, live-patching, pmladek

On Fri, 18 Oct 2019, Jessica Yu wrote:

> +++ Miroslav Benes [16/10/19 15:29 +0200]:
> >On Wed, 16 Oct 2019, Miroslav Benes wrote:
> >
> >> On Wed, 16 Oct 2019, Peter Zijlstra wrote:
> >>
> >> > On Tue, Oct 15, 2019 at 06:27:05PM -0400, Steven Rostedt wrote:
> >> >
> >> > > (7) Seventh session, titled "klp-convert and livepatch relocations",
> >> > > was led
> >> > > by Joe Lawrence.
> >> > >
> >> > > Joe started the session with problem statement: accessing non exported
> >> > > / static
> >> > > symbols from inside the patch module. One possible workardound is
> >> > > manually via
> >> > > kallsyms. Second workaround is klp-convert, which actually creates
> >> > > proper
> >> > > relocations inside the livepatch module from the symbol database during
> >> > > the
> >> > > final .ko link.
> >> > > Currently module loader looks for special livepatch relocations and
> >> > > resolves
> >> > > those during runtime; kernel support for these relocations have so far
> >> > > been
> >> > > added for x86 only. Special livepatch relocations are supported and
> >> > > processed
> >> > > also on other architectures. Special quirks/sections are not yet
> >> > > supported.
> >> > > Plus klp-convert would still be needed even with late module patching
> >> > > update.
> >> > > vmlinux or modules could have ambiguous static symbols.
> >> > >
> >> > > It turns out that the features / bugs below have to be resolved before
> >> > > we
> >> > > can claim the klp-convert support for relocation complete:
> >> > >     - handle all the corner cases (jump labels, static keys, ...)
> >> > >     properly and
> >> > >       have a good regression tests in place
> >> >
> >> > I suppose all the patches in this series-of-series here will make life
> >> > harder for KLP, static_call() and 2 byte jumps etc..
> >>
> >> Yes, I think so. We'll have to deal with that once it lands. That is why
> >> we want to get rid of all this arch-specific code in livepatch and
> >> reinvent the late module patching. So it is perhaps better to start
> >> working on it sooner than later. Adding Petr, who hesitantly signed up for
> >> the task...
> >
> >Thinking about it more... crazy idea. I think we could leverage these new
> >ELF .text per vmlinux/module sections for the reinvention I was talking
> >about. If we teach module loader to relocate (and apply alternatives and
> >so on, everything in arch-specific module_finalize()) not the whole module
> >in case of live patch modules, but separate ELF .text sections, it could
> >solve the issue with late module patching we have. It is a variation on
> >Steven's idea. When live patch module is loaded, only its section for
> >present modules would be processed. Then whenever a to-be-patched module
> >is loaded, its .text section in all present patch module would be
> >processed.
> >
> >The upside is that almost no work would be required on patch modules
> >creation side. The downside is that klp_modinfo must stay. Module loader
> >needs to be hacked a lot in both cases. So it remains to be seen which
> >idea is easier to implement.
> >
> >Jessica, do you think it would be feasible?
> 
> I think that does sound feasible. I'm trying to visualize how that
> would look. I guess there would need to be various livepatching hooks
> called during the different stages (apply_relocate_add(),
> module_finalize(), module_enable_ro/x()).
> 
> So maybe something like the following?
> 
> When a livepatch module loads:
>    apply_relocate_add()
>        klp hook: apply .klp.rela.$objname relocations *only* for
>        already loaded modules
>    module_finalize()
>        klp hook: apply .klp.arch.$objname changes for already loaded modules
>    module_enable_ro()
>        klp hook: only enable ro/x for .klp.text.$objname for already
>        loaded modules
> 
> When a to-be-patched module loads:
>    apply_relocate_add()
>        klp hook: for each patch module that patches the coming
>        module, apply .klp.rela.$objname relocations for this object
>    module_finalize()
>        klp hook: for each patch module that patches the coming
>        module, apply .klp.arch.$objname changes for this object
>    module_enable_ro()
>        klp hook: for each patch module, apply ro/x permissions for
>        .klp.text.$objname for this object
> 
> Then, in klp_module_coming, we only need to do the callbacks and
> enable the patch, and get rid of the module_disable_ro->apply
> relocs->module_enable_ro block.
> 
> Does that sound like what you had in mind or am I totally off?

Sort of. What I had in mind was that we could get rid of all special .klp 
ELF section if module loader guarantees that only sections for loaded 
modules are processed. Then .klp.rela.$objname is not needed and proper 
.rela.text.$objname (or whatever its text section is named) should be 
sufficient. The same for the rest (.klp.arch).

Only then it would be useful.

Miroslav

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-16 12:39                           ` Peter Zijlstra
@ 2019-10-22  8:45                             ` Miroslav Benes
  0 siblings, 0 replies; 24+ messages in thread
From: Miroslav Benes @ 2019-10-22  8:45 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Joe Lawrence, Jessica Yu, Steven Rostedt, x86, linux-kernel,
	mhiramat, bristot, jbaron, torvalds, tglx, mingo, namit, hpa,
	luto, ard.biesheuvel, jpoimboe, live-patching

On Wed, 16 Oct 2019, Peter Zijlstra wrote:

> On Wed, Oct 16, 2019 at 08:51:27AM +0200, Miroslav Benes wrote:
> > On Tue, 15 Oct 2019, Joe Lawrence wrote:
> > 
> > > On 10/15/19 10:13 AM, Miroslav Benes wrote:
> > > > Yes, it does. klp_module_coming() calls module_disable_ro() on all
> > > > patching modules which patch the coming module in order to call
> > > > apply_relocate_add(). New (patching) code for a module can be relocated
> > > > only when the relevant module is loaded.
> > > 
> > > FWIW, would the LPC blue-sky2 model (ie, Steve's suggestion @ plumber's where
> > > livepatches only patch a single object and updates are kept on disk to handle
> > > coming module updates as they are loaded) eliminate those outstanding
> > > relocations and the need to perform this late permission flipping?
> > 
> > Yes, it should, but we don't have to wait for it. PeterZ proposed a 
> > different solution to this specific issue in 
> > https://lore.kernel.org/lkml/20191015141111.GP2359@hirez.programming.kicks-ass.net/
> > 
> > It should not be a problem to create a live patch module like that and the 
> > code in kernel/livepatch/ is almost ready. Something like 
> > module_section_disable_ro(mod, section) (and similar for X protection) 
> > should be enough. Module reloads would still require juggling with the 
> > protections, but I think it is all feasible.
> 
> Something a little like so.. completely fresh of the keyboard.

Yes, but I noticed you found different and better way through text_poke() 
(I was not aware that text_poke() works around the protections).

Miroslav
 
> ---
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
> @@ -853,6 +853,18 @@ static inline void module_enable_ro(cons
>  static inline void module_disable_ro(const struct module *mod) { }
>  #endif
>  
> +#if defined(CONFIG_STRICT_MODULE_RWX) && defined(CONFIG_LIVEPATCH)
> +extern void module_section_disable_ro(struct module *mod, const char *sec);
> +extern void module_section_enable_ro(struct module *mod, const char *sec);
> +extern void module_section_disable_x(struct module *mod, const char *sec);
> +extern void module_section_enable_x(struct module *mod, const char *sec);
> +#else
> +static inline void module_section_disable_ro(struct module *mod, const char *sec) { }
> +static inline void module_section_enable_ro(struct module *mod, const char *sec) { }
> +static inline void module_section_disable_x(struct module *mod, const char *sec) { }
> +static inline void module_section_enable_x(struct module *mod, const char *sec) { }
> +#endif
> +
>  #ifdef CONFIG_GENERIC_BUG
>  void module_bug_finalize(const Elf_Ehdr *, const Elf_Shdr *,
>  			 struct module *);
> --- a/kernel/module.c
> +++ b/kernel/module.c
> @@ -2107,6 +2107,54 @@ static void free_module_elf(struct modul
>  	kfree(mod->klp_info->secstrings);
>  	kfree(mod->klp_info);
>  }
> +
> +#ifdef CONFIG_STRICT_MODULE_RWX
> +
> +static void __frob_section(struct Elf_Shdr *sec, int (*set_memory)(unsigned long start, int num_pages))
> +{
> +	BUG_ON((unsigned long)sec->sh_addr & (PAGE_SIZE-1));
> +	BUG_ON((unsigned long)sec->sh_size & (PAGE_SIZE-1));
> +	set_memory((unsigned long)sec->sh_addr, sec->sh_size >> PAGE_SHIFT);
> +}
> +
> +static void frob_section(struct module *mod, const char *section,
> +			 int (*set_memory)(unsigned long start, int num_pages))
> +{
> +	struct klp_modinfo *info = mod->klp_info;
> +	const char *secname;
> +	Elf_Shdr *s;
> +
> +	for (s = info->sechdrs; s < info->sechdrs + info->hdr.e_shnum; s++) {
> +		secname = mod->klp_info->secstrings + s->sh_name;
> +		if (strcmp(secname, section))
> +			continue;
> +
> +		__frob_section(s, set_memory);
> +	}
> +}
> +
> +void module_section_disable_ro(struct module *mod, const char *section)
> +{
> +	frob_section(mod, section, set_memory_rw);
> +}
> +
> +void module_section_enable_ro(struct module *mod, const char *section)
> +{
> +	frob_section(mod, section, set_memory_ro);
> +}
> +
> +void module_section_disable_x(struct module *mod, const char *section)
> +{
> +	frob_section(mod, section, set_memory_nx);
> +}
> +
> +void module_section_enable_x(struct module *mod, const char *section)
> +{
> +	frob_section(mod, section, set_memory_x);
> +}
> +
> +#endif /* ONFIG_STRICT_MODULE_RWX */
> +
>  #else /* !CONFIG_LIVEPATCH */
>  static int copy_module_elf(struct module *mod, struct load_info *info)
>  {
> 


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-22  8:27                                       ` Miroslav Benes
@ 2019-10-22 14:31                                         ` Josh Poimboeuf
  2019-10-23  9:04                                           ` Miroslav Benes
  0 siblings, 1 reply; 24+ messages in thread
From: Josh Poimboeuf @ 2019-10-22 14:31 UTC (permalink / raw)
  To: Miroslav Benes
  Cc: Jessica Yu, Peter Zijlstra, Steven Rostedt, Joe Lawrence, x86,
	linux-kernel, mhiramat, bristot, jbaron, torvalds, tglx, mingo,
	namit, hpa, luto, ard.biesheuvel, live-patching, pmladek

On Tue, Oct 22, 2019 at 10:27:49AM +0200, Miroslav Benes wrote:
> > Does that sound like what you had in mind or am I totally off?
> 
> Sort of. What I had in mind was that we could get rid of all special .klp 
> ELF section if module loader guarantees that only sections for loaded 
> modules are processed. Then .klp.rela.$objname is not needed and proper 
> .rela.text.$objname (or whatever its text section is named) should be 
> sufficient. The same for the rest (.klp.arch).

If I understand correctly, using kvm as an example to-be-patched module,
we'd have:

  .text.kvm
  .rela.text.kvm
  .altinstructions.kvm
  .rela.altinstructions.kvm
  __jump_table.kvm
  .rela__jump_table.kvm

etc.  i.e. any "special" sections would need to be renamed.

Is that right?

But also I think *any* sections which need relocations would need to be
renamed, for example:

  .rodata.kvm
  .rela.rodata.kvm
  .orc_unwind_ip.kvm
  .rela.orc_unwind_ip.kvm


It's an interesting idea.

We'd have to be careful about ordering issues.  For example, there are
module-specific jump labels stored in mod->jump_entries.  Right now
that's just a pointer to the module's __jump_table section.  With late
module patching, when kvm is loaded we'd have to insert the klp module's
__jump_table.kvm entries into kvm's mod->jump_entries list somehow.

Presumably we'd also have that issue for other sections.  Handling that
_might_ be as simple as just hacking up find_module_sections() to
re-allocate sections and append "patched sections" to them.

But then you still have to worry about when to apply the relocations.
If you apply them before patching the sections, then relative
relocations would have the wrong values.  If you apply them after, then
you have to figure out where the appended relocations are.

And if we allow unpatching then we'd presumably have to be able to
remove entries from the module specific section lists.

So I get the feeling a lot of complexity would creep in.  Even just
thinking about it requires more mental gymnastics than the
one-patch-per-module idea, so I view that as a bad sign.

-- 
Josh


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH v3 5/6] x86/ftrace: Use text_poke()
  2019-10-22 14:31                                         ` Josh Poimboeuf
@ 2019-10-23  9:04                                           ` Miroslav Benes
  0 siblings, 0 replies; 24+ messages in thread
From: Miroslav Benes @ 2019-10-23  9:04 UTC (permalink / raw)
  To: Josh Poimboeuf
  Cc: Jessica Yu, Peter Zijlstra, Steven Rostedt, Joe Lawrence, x86,
	linux-kernel, mhiramat, bristot, jbaron, torvalds, tglx, mingo,
	namit, hpa, luto, ard.biesheuvel, live-patching, pmladek

On Tue, 22 Oct 2019, Josh Poimboeuf wrote:

> On Tue, Oct 22, 2019 at 10:27:49AM +0200, Miroslav Benes wrote:
> > > Does that sound like what you had in mind or am I totally off?
> > 
> > Sort of. What I had in mind was that we could get rid of all special .klp 
> > ELF section if module loader guarantees that only sections for loaded 
> > modules are processed. Then .klp.rela.$objname is not needed and proper 
> > .rela.text.$objname (or whatever its text section is named) should be 
> > sufficient. The same for the rest (.klp.arch).
> 
> If I understand correctly, using kvm as an example to-be-patched module,
> we'd have:
> 
>   .text.kvm
>   .rela.text.kvm
>   .altinstructions.kvm
>   .rela.altinstructions.kvm
>   __jump_table.kvm
>   .rela__jump_table.kvm
> 
> etc.  i.e. any "special" sections would need to be renamed.
> 
> Is that right?

Yes.
 
> But also I think *any* sections which need relocations would need to be
> renamed, for example:
> 
>   .rodata.kvm
>   .rela.rodata.kvm
>   .orc_unwind_ip.kvm
>   .rela.orc_unwind_ip.kvm

Correct.
 
> It's an interesting idea.
> 
> We'd have to be careful about ordering issues.  For example, there are
> module-specific jump labels stored in mod->jump_entries.  Right now
> that's just a pointer to the module's __jump_table section.  With late
> module patching, when kvm is loaded we'd have to insert the klp module's
> __jump_table.kvm entries into kvm's mod->jump_entries list somehow.

Yes.
 
> Presumably we'd also have that issue for other sections.  Handling that
> _might_ be as simple as just hacking up find_module_sections() to
> re-allocate sections and append "patched sections" to them.
>
> But then you still have to worry about when to apply the relocations.
> If you apply them before patching the sections, then relative
> relocations would have the wrong values.  If you apply them after, then
> you have to figure out where the appended relocations are.

Ah, right. That is a valid remark.
 
> And if we allow unpatching then we'd presumably have to be able to
> remove entries from the module specific section lists.

Correct.

> So I get the feeling a lot of complexity would creep in.  Even just
> thinking about it requires more mental gymnastics than the
> one-patch-per-module idea, so I view that as a bad sign.

Yes, the devil is in the details. It would be better if the approach 
helped even someone/something else in the kernel. Without it, it is 
probably better to stick to Steven's proposal and handle the complexity 
elsewhere.

Thanks
Miroslav

^ permalink raw reply	[flat|nested] 24+ messages in thread

end of thread, back to index

Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <20191007081945.10951536.8@infradead.org>
     [not found] ` <20191008104335.6fcd78c9@gandalf.local.home>
     [not found]   ` <20191009224135.2dcf7767@oasis.local.home>
     [not found]     ` <20191010092054.GR2311@hirez.programming.kicks-ass.net>
     [not found]       ` <20191010091956.48fbcf42@gandalf.local.home>
     [not found]         ` <20191010140513.GT2311@hirez.programming.kicks-ass.net>
     [not found]           ` <20191010115449.22044b53@gandalf.local.home>
     [not found]             ` <20191010172819.GS2328@hirez.programming.kicks-ass.net>
     [not found]               ` <20191011125903.GN2359@hirez.programming.kicks-ass.net>
     [not found]                 ` <20191015130739.GA23565@linux-8ccs>
     [not found]                   ` <20191015135634.GK2328@hirez.programming.kicks-ass.net>
2019-10-15 14:13                     ` [PATCH v3 5/6] x86/ftrace: Use text_poke() Miroslav Benes
2019-10-15 15:06                       ` Joe Lawrence
2019-10-15 15:31                         ` Jessica Yu
2019-10-15 22:17                           ` Joe Lawrence
2019-10-15 22:27                             ` Steven Rostedt
2019-10-16  7:42                               ` Peter Zijlstra
2019-10-16 10:15                                 ` Miroslav Benes
2019-10-21 15:05                                 ` Josh Poimboeuf
2019-10-16  7:49                               ` Peter Zijlstra
2019-10-16 10:20                                 ` Miroslav Benes
2019-10-16 13:29                                   ` Miroslav Benes
2019-10-18 13:03                                     ` Jessica Yu
2019-10-18 13:40                                       ` Petr Mladek
2019-10-21 14:14                                         ` Jessica Yu
2019-10-21 15:31                                         ` Josh Poimboeuf
2019-10-22  8:27                                       ` Miroslav Benes
2019-10-22 14:31                                         ` Josh Poimboeuf
2019-10-23  9:04                                           ` Miroslav Benes
2019-10-16  6:51                         ` Miroslav Benes
2019-10-16  9:23                           ` Peter Zijlstra
2019-10-16  9:36                             ` Jessica Yu
2019-10-16  9:51                               ` Peter Zijlstra
2019-10-16 12:39                           ` Peter Zijlstra
2019-10-22  8:45                             ` Miroslav Benes

Live-Patching Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/live-patching/0 live-patching/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 live-patching live-patching/ https://lore.kernel.org/live-patching \
		live-patching@vger.kernel.org
	public-inbox-index live-patching

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.live-patching


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git