Live Patching Microconference at Linux Plumbers

Live Patching Microconference at Linux Plumbers

[Miroslav Benes]

Hi,

we would like to organize Live Patching Microconference at Linux Plumbers 
2023. The conference will take place in Richmond, VA, USA. 13-15 November. 
https://lpc.events/. The call for proposals has been opened so it is time 
to start the preparation on our side.

You can find the proposal below. Comments are welcome. The list of topics 
is open, so feel free to add more. I tried to add key people to discuss 
the topics, but the list is not exhaustive. I would like to submit the 
proposal soonish even though the deadline is on 1 June. I assume that we 
can update the topics later. My plan is to also organize a proper Call for 
Topics after the submission and advertise it also on LKML.

Last but not least it would be nice to have a co-runner of the show. Josh, 
Joe, any volunteer? :)

Thank you
Miroslav


Proposal
--------
The Live Patching microconference at Linux Plumbers 2023 aims to gather
stakeholders and interested parties to discuss proposed features and
outstanding issues in live patching.

Live patching is a critical tool for maintaining system uptime and
security by enabling fixes to be applied to running systems without the
need for a reboot. The development of the infrastructure is an ongoing
effort and while many problems have been resolved and features
implemented, there are still open questions, some with already submitted
patch sets, which need to be discussed.

Live Patching microconferences at the previous Linux Plumbers
conferences proved to be useful in this regard and helped us to find
final solutions or at least promising directions to push the development
forward. It includes for example a support for several architectures
(ppc64le and s390x were added after x86_64), a late module patching and
module dependencies and user space live patching.

Currently proposed topics follow. The list is open though and more will
be added during the regular Call for Topics.

  - klp-convert (as means to fix CET IBT limitations) and its 
    upstreamability
  - shadow variables, global state transition
  - kselftests and the future direction of development
  - arm64 live patching

Key people

  - Josh Poimboeuf <jpoimboe@kernel.org>
  - Jiri Kosina <jikos@kernel.org>
  - Miroslav Benes <mbenes@suse.cz>
  - Petr Mladek <pmladek@suse.com>
  - Joe Lawrence <joe.lawrence@redhat.com>
  - Nicolai Stange <nstange@suse.de>
  - Marcos Paulo de Souza <mpdesouza@suse.de>
  - Mark Rutland <mark.rutland@arm.com>
  - Mark Brown <broonie@kernel.org>

We encourage all attendees to actively participate in the
microconference by sharing their ideas, experiences, and insights.

Re: Live Patching Microconference at Linux Plumbers

[Joe Lawrence]

On 3/29/23 08:05, Miroslav Benes wrote:
> Last but not least it would be nice to have a co-runner of the show. Josh, 
> Joe, any volunteer? 😄

Sure, lmk whatever help you need to co-run.

-- 
Joe

Re: Live Patching Microconference at Linux Plumbers

[Miroslav Benes]

Hi,

> You can find the proposal below. Comments are welcome. The list of topics 
> is open, so feel free to add more. I tried to add key people to discuss 
> the topics, but the list is not exhaustive. I would like to submit the 
> proposal soonish even though the deadline is on 1 June. I assume that we 
> can update the topics later. My plan is to also organize a proper Call for 
> Topics after the submission and advertise it also on LKML.

submitted now.

> Last but not least it would be nice to have a co-runner of the show. Josh, 
> Joe, any volunteer? :)

Joe is the co-runner. Thank you!

Miroslav

Re: Live Patching Microconference at Linux Plumbers

[Mark Rutland]

On Wed, Mar 29, 2023 at 02:05:43PM +0200, Miroslav Benes wrote:
> Hi,
> 
> we would like to organize Live Patching Microconference at Linux Plumbers 
> 2023. The conference will take place in Richmond, VA, USA. 13-15 November. 
> https://lpc.events/. The call for proposals has been opened so it is time 
> to start the preparation on our side.
> 
> You can find the proposal below. Comments are welcome. The list of topics 
> is open, so feel free to add more. I tried to add key people to discuss 
> the topics, but the list is not exhaustive. I would like to submit the 
> proposal soonish even though the deadline is on 1 June. I assume that we 
> can update the topics later. My plan is to also organize a proper Call for 
> Topics after the submission and advertise it also on LKML.
> 
> Last but not least it would be nice to have a co-runner of the show. Josh, 
> Joe, any volunteer? :)
> 
> Thank you
> Miroslav
> 
> 
> Proposal
> --------
> The Live Patching microconference at Linux Plumbers 2023 aims to gather
> stakeholders and interested parties to discuss proposed features and
> outstanding issues in live patching.
> 
> Live patching is a critical tool for maintaining system uptime and
> security by enabling fixes to be applied to running systems without the
> need for a reboot. The development of the infrastructure is an ongoing
> effort and while many problems have been resolved and features
> implemented, there are still open questions, some with already submitted
> patch sets, which need to be discussed.
> 
> Live Patching microconferences at the previous Linux Plumbers
> conferences proved to be useful in this regard and helped us to find
> final solutions or at least promising directions to push the development
> forward. It includes for example a support for several architectures
> (ppc64le and s390x were added after x86_64), a late module patching and
> module dependencies and user space live patching.
> 
> Currently proposed topics follow. The list is open though and more will
> be added during the regular Call for Topics.
> 
>   - klp-convert (as means to fix CET IBT limitations) and its 
>     upstreamability
>   - shadow variables, global state transition
>   - kselftests and the future direction of development
>   - arm64 live patching

I'm happy to talk about the kernel-side of arm64 live patching; it'd be good to
get in contact with anyone looking at the arm64 userspace side (e.g.
klp-convert).

I have some topics which overlap between live-patching and general toolchain
bits and pieces, and I'm not sure if they'd be best suited here or in a
toolchain track, e.g.

* How to avoid/minimize the need to reverse-engineer control flow for things
  like ORC generation.

  On the arm64 side we're pretty averse to doing this to generate metadata for
  unwinding (and we might not need to), but there are things objtool does today
  that requires awareness of control-flow (e.g. forward-edge checks for noinstr
  safety).

  Hopefully without a flamewar about DWARF...

* Better compiler support for noinstr and similar properties.

  For example, noinstr functions are currently all noinline, and we can't
  inline a noinstr function into a noinstr function, leading to a painful mix
  of noinstr and __always_inline. Having a mechanism to allow noinstr code to
  be inlined into other noinstr code would be nice.

  Likewise, whether we could somehow get compile-time warnings about unintended
  calls from instrumentable code from noinstr code.

* How is this going to work with rust?

  It's not clear to me whether/how things like ftrace, RELIABLE_STACKTRACE, and
  live-patching are going to work with rust. We probably need to start looking
  soon.

I've Cc'd Nick, Jose, and Miguel, in case they have thoughts.

Mark.

> 
> Key people
> 
>   - Josh Poimboeuf <jpoimboe@kernel.org>
>   - Jiri Kosina <jikos@kernel.org>
>   - Miroslav Benes <mbenes@suse.cz>
>   - Petr Mladek <pmladek@suse.com>
>   - Joe Lawrence <joe.lawrence@redhat.com>
>   - Nicolai Stange <nstange@suse.de>
>   - Marcos Paulo de Souza <mpdesouza@suse.de>
>   - Mark Rutland <mark.rutland@arm.com>
>   - Mark Brown <broonie@kernel.org>
> 
> We encourage all attendees to actively participate in the
> microconference by sharing their ideas, experiences, and insights.
> 

Re: Live Patching Microconference at Linux Plumbers

[Jose E. Marchesi]

> On Wed, Mar 29, 2023 at 02:05:43PM +0200, Miroslav Benes wrote:
>> Hi,
>> 
>> we would like to organize Live Patching Microconference at Linux Plumbers 
>> 2023. The conference will take place in Richmond, VA, USA. 13-15 November. 
>> https://lpc.events/. The call for proposals has been opened so it is time 
>> to start the preparation on our side.
>> 
>> You can find the proposal below. Comments are welcome. The list of topics 
>> is open, so feel free to add more. I tried to add key people to discuss 
>> the topics, but the list is not exhaustive. I would like to submit the 
>> proposal soonish even though the deadline is on 1 June. I assume that we 
>> can update the topics later. My plan is to also organize a proper Call for 
>> Topics after the submission and advertise it also on LKML.
>> 
>> Last but not least it would be nice to have a co-runner of the show. Josh, 
>> Joe, any volunteer? :)
>> 
>> Thank you
>> Miroslav
>> 
>> 
>> Proposal
>> --------
>> The Live Patching microconference at Linux Plumbers 2023 aims to gather
>> stakeholders and interested parties to discuss proposed features and
>> outstanding issues in live patching.
>> 
>> Live patching is a critical tool for maintaining system uptime and
>> security by enabling fixes to be applied to running systems without the
>> need for a reboot. The development of the infrastructure is an ongoing
>> effort and while many problems have been resolved and features
>> implemented, there are still open questions, some with already submitted
>> patch sets, which need to be discussed.
>> 
>> Live Patching microconferences at the previous Linux Plumbers
>> conferences proved to be useful in this regard and helped us to find
>> final solutions or at least promising directions to push the development
>> forward. It includes for example a support for several architectures
>> (ppc64le and s390x were added after x86_64), a late module patching and
>> module dependencies and user space live patching.
>> 
>> Currently proposed topics follow. The list is open though and more will
>> be added during the regular Call for Topics.
>> 
>>   - klp-convert (as means to fix CET IBT limitations) and its 
>>     upstreamability
>>   - shadow variables, global state transition
>>   - kselftests and the future direction of development
>>   - arm64 live patching
>
> I'm happy to talk about the kernel-side of arm64 live patching; it'd be good to
> get in contact with anyone looking at the arm64 userspace side (e.g.
> klp-convert).
>
> I have some topics which overlap between live-patching and general toolchain
> bits and pieces, and I'm not sure if they'd be best suited here or in a
> toolchain track, e.g.
>
> * How to avoid/minimize the need to reverse-engineer control flow for things
>   like ORC generation.
>
>   On the arm64 side we're pretty averse to doing this to generate metadata for
>   unwinding (and we might not need to), but there are things objtool does today
>   that requires awareness of control-flow (e.g. forward-edge checks for noinstr
>   safety).
>
>   Hopefully without a flamewar about DWARF...
>
> * Better compiler support for noinstr and similar properties.
>
>   For example, noinstr functions are currently all noinline, and we can't
>   inline a noinstr function into a noinstr function, leading to a painful mix
>   of noinstr and __always_inline. Having a mechanism to allow noinstr code to
>   be inlined into other noinstr code would be nice.
>
>   Likewise, whether we could somehow get compile-time warnings about unintended
>   calls from instrumentable code from noinstr code.
>
> * How is this going to work with rust?
>
>   It's not clear to me whether/how things like ftrace, RELIABLE_STACKTRACE, and
>   live-patching are going to work with rust. We probably need to start looking
>   soon.
>
> I've Cc'd Nick, Jose, and Miguel, in case they have thoughts.

We have indeed submitted a proposal for a Toolchains MC for Plumbers.

I think the two first topics mentioned above (CFG in ELF and handling of
noinstr functions) would definitely fit well in the Toolchains MC.

As for Rust... we have the Rust GCC support and that would fit in the MC
as well.  We can surely invite some of the hackers working in the
front-end.  But maybe it would be better to have that discussion in a
Rust MC, if there is gonna be one (Miguel?).

For starters, I would make sure that the involved MCs (Live Patching,
Toolchains, and an eventual Rust MC) do not overlap in the schedule.
Then we could have these discussions in either microconferences.

>> 
>> Key people
>> 
>>   - Josh Poimboeuf <jpoimboe@kernel.org>
>>   - Jiri Kosina <jikos@kernel.org>
>>   - Miroslav Benes <mbenes@suse.cz>
>>   - Petr Mladek <pmladek@suse.com>
>>   - Joe Lawrence <joe.lawrence@redhat.com>
>>   - Nicolai Stange <nstange@suse.de>
>>   - Marcos Paulo de Souza <mpdesouza@suse.de>
>>   - Mark Rutland <mark.rutland@arm.com>
>>   - Mark Brown <broonie@kernel.org>
>> 
>> We encourage all attendees to actively participate in the
>> microconference by sharing their ideas, experiences, and insights.
>> 

Re: Live Patching Microconference at Linux Plumbers

[Miguel Ojeda]

On Fri, Apr 14, 2023 at 2:54 PM Jose E. Marchesi
<jose.marchesi@oracle.com> wrote:
>
> As for Rust... we have the Rust GCC support and that would fit in the MC
> as well.  We can surely invite some of the hackers working in the
> front-end.  But maybe it would be better to have that discussion in a
> Rust MC, if there is gonna be one (Miguel?).

It is likely we will submit a Rust MC proposal, yeah.

Last year we had both Rust GCC and rustc_codegen_gcc presenting in the
Rust MC (and Kangrejos too), and I would be grateful to have them
again, but if it is best for everybody otherwise, we can change things
of course.

(Is it already known how tight timing will be this year for MCs?)

> For starters, I would make sure that the involved MCs (Live Patching,
> Toolchains, and an eventual Rust MC) do not overlap in the schedule.
> Then we could have these discussions in either microconferences.

Yeah, it sounds good to me.

Cheers,
Miguel

Re: Live Patching Microconference at Linux Plumbers

[Miroslav Benes]

> > I'm happy to talk about the kernel-side of arm64 live patching; it'd be good to
> > get in contact with anyone looking at the arm64 userspace side (e.g.
> > klp-convert).
> >
> > I have some topics which overlap between live-patching and general toolchain
> > bits and pieces, and I'm not sure if they'd be best suited here or in a
> > toolchain track, e.g.
> >
> > * How to avoid/minimize the need to reverse-engineer control flow for things
> >   like ORC generation.
> >
> >   On the arm64 side we're pretty averse to doing this to generate metadata for
> >   unwinding (and we might not need to), but there are things objtool does today
> >   that requires awareness of control-flow (e.g. forward-edge checks for noinstr
> >   safety).
> >
> >   Hopefully without a flamewar about DWARF...
> >
> > * Better compiler support for noinstr and similar properties.
> >
> >   For example, noinstr functions are currently all noinline, and we can't
> >   inline a noinstr function into a noinstr function, leading to a painful mix
> >   of noinstr and __always_inline. Having a mechanism to allow noinstr code to
> >   be inlined into other noinstr code would be nice.
> >
> >   Likewise, whether we could somehow get compile-time warnings about unintended
> >   calls from instrumentable code from noinstr code.
> >
> > * How is this going to work with rust?
> >
> >   It's not clear to me whether/how things like ftrace, RELIABLE_STACKTRACE, and
> >   live-patching are going to work with rust. We probably need to start looking
> >   soon.
> >
> > I've Cc'd Nick, Jose, and Miguel, in case they have thoughts.
> 
> We have indeed submitted a proposal for a Toolchains MC for Plumbers.

Great.
 
> I think the two first topics mentioned above (CFG in ELF and handling of
> noinstr functions) would definitely fit well in the Toolchains MC.

Yes.

> As for Rust... we have the Rust GCC support and that would fit in the MC
> as well.  We can surely invite some of the hackers working in the
> front-end.  But maybe it would be better to have that discussion in a
> Rust MC, if there is gonna be one (Miguel?).

I would definitely welcome a session which Mark proposed. Be it at Live 
Patching MC or Rust MC. I know next to nothing about how Rust is wired 
into the kernel and what impact it might have on us. The sooner we 
understand possible issues, the better.

> For starters, I would make sure that the involved MCs (Live Patching,
> Toolchains, and an eventual Rust MC) do not overlap in the schedule.
> Then we could have these discussions in either microconferences.

Agreed.

Miroslav

Re: Live Patching Microconference at Linux Plumbers

[Jose E. Marchesi]

> On Fri, Apr 14, 2023 at 2:54 PM Jose E. Marchesi
> <jose.marchesi@oracle.com> wrote:
>>
>> As for Rust... we have the Rust GCC support and that would fit in the MC
>> as well.  We can surely invite some of the hackers working in the
>> front-end.  But maybe it would be better to have that discussion in a
>> Rust MC, if there is gonna be one (Miguel?).
>
> It is likely we will submit a Rust MC proposal, yeah.
>
> Last year we had both Rust GCC and rustc_codegen_gcc presenting in the
> Rust MC (and Kangrejos too), and I would be grateful to have them
> again, but if it is best for everybody otherwise, we can change things
> of course.

I think it makes sense to discuss these toolchain-related topics in the
Rust MC.  Infrastructure tends to be traversal :)

> (Is it already known how tight timing will be this year for MCs?)
>
>> For starters, I would make sure that the involved MCs (Live Patching,
>> Toolchains, and an eventual Rust MC) do not overlap in the schedule.
>> Then we could have these discussions in either microconferences.
>
> Yeah, it sounds good to me.
>
> Cheers,
> Miguel

Re: Live Patching Microconference at Linux Plumbers

[Josh Poimboeuf]

On Fri, Apr 14, 2023 at 10:53:03AM +0100, Mark Rutland wrote:
> > Currently proposed topics follow. The list is open though and more will
> > be added during the regular Call for Topics.
> > 
> >   - klp-convert (as means to fix CET IBT limitations) and its 
> >     upstreamability
> >   - shadow variables, global state transition
> >   - kselftests and the future direction of development
> >   - arm64 live patching
> 
> I'm happy to talk about the kernel-side of arm64 live patching; it'd be good to
> get in contact with anyone looking at the arm64 userspace side (e.g.
> klp-convert).

klp-convert is still considered experimental.  FYI here's a pull request
which adds arm64 support to kpatch-build:

  https://github.com/dynup/kpatch/pull/1302

> I have some topics which overlap between live-patching and general toolchain
> bits and pieces, and I'm not sure if they'd be best suited here or in a
> toolchain track, e.g.
> 
> * How to avoid/minimize the need to reverse-engineer control flow for things
>   like ORC generation.
> 
>   On the arm64 side we're pretty averse to doing this to generate metadata for
>   unwinding (and we might not need to), but there are things objtool does today
>   that requires awareness of control-flow (e.g. forward-edge checks for noinstr
>   safety).
> 
>   Hopefully without a flamewar about DWARF...

If objtool is going to be doing control-flow anyway then it could just
validate DWARF/SFrame.  Then everybody's happy?

> * Better compiler support for noinstr and similar properties.
> 
>   For example, noinstr functions are currently all noinline, and we can't
>   inline a noinstr function into a noinstr function, leading to a painful mix
>   of noinstr and __always_inline. Having a mechanism to allow noinstr code to
>   be inlined into other noinstr code would be nice.

Can you elaborate?  Why can't noinstr inline noinstr?  (that's a
mouthful)

Is it because of potential cloning caused by IPA optimizations?

>   Likewise, whether we could somehow get compile-time warnings about unintended
>   calls from instrumentable code from noinstr code.
> 
> * How is this going to work with rust?
> 
>   It's not clear to me whether/how things like ftrace, RELIABLE_STACKTRACE, and
>   live-patching are going to work with rust.

Not to mention how objtool will react to compiled rust code (has it
already been tried?)

-- 
Josh

Re: Live Patching Microconference at Linux Plumbers

[Miguel Ojeda]

On Fri, Apr 14, 2023 at 7:12 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>
> Not to mention how objtool will react to compiled rust code (has it
> already been tried?)

Rust uses LLVM, so it should be generally fine -- at least some of the
checks appear to work. For instance, I can trigger:

      RUSTC L rust/kernel.o
    rust/kernel.o: warning: objtool: .text+0x0: unreachable instruction

      RUSTC [M] samples/rust/rust_minimal.o
    samples/rust/rust_minimal.o: warning: objtool:
_R..._6kernel6Module4init+0x172: unreachable instruction

via a random instruction in the middle of nowhere in the former (with
`global_asm!`) and a jumped-over instruction in the latter (with
`asm!`).

Moreover, we were already getting warnings when rethunk / x86 IBT is
enabled (since we got `objtool` called for `vmlinux.o`), e.g.

    vmlinux.o: warning: objtool: .rodata+0x18c58: data relocation to
!ENDBR: _R...IsWhitespaceEEB4_+0x0

    vmlinux.o: warning: objtool: _R...into_foreign+0x5: 'naked' return
found in RETHUNK build

I can send the patch to run it for all Rust object files via
`$(cmd_objtool)`, unless you think it is a bad idea.

Having said that, tooling may indeed have issues, e.g. Arnaldo (Cc'd)
is improving `pahole` to avoid assumptions around struct layout like
field reordering (which Rust does by default).

Cheers,
Miguel

Re: Live Patching Microconference at Linux Plumbers

[Peter Zijlstra]

On Fri, Apr 14, 2023 at 10:12:55AM -0700, Josh Poimboeuf wrote:

> > * How to avoid/minimize the need to reverse-engineer control flow for things
> >   like ORC generation.
> > 
> >   On the arm64 side we're pretty averse to doing this to generate metadata for
> >   unwinding (and we might not need to), but there are things objtool does today
> >   that requires awareness of control-flow (e.g. forward-edge checks for noinstr
> >   safety).
> > 
> >   Hopefully without a flamewar about DWARF...
> 
> If objtool is going to be doing control-flow anyway then it could just
> validate DWARF/SFrame.  Then everybody's happy?

Right; so per another recent thread somewhere; you can't rely on
DWARF/Sframe or any other compiler generated thing simply because it
doesn't cover .S files and inline asm -- and this being a kernel, we've
got quite a bit of that.

At best it could use DWARF to help reconstruct code flow and then
validate Sframe for the bits that got sframe.

FWIW, manually annotated asm without validation is an utter fail -- x86
tried that and it never works.

FWIW2, building with DWARF info on is significantly slower / bigger.

> > * Better compiler support for noinstr and similar properties.
> > 
> >   For example, noinstr functions are currently all noinline, and we can't
> >   inline a noinstr function into a noinstr function, leading to a painful mix
> >   of noinstr and __always_inline. Having a mechanism to allow noinstr code to
> >   be inlined into other noinstr code would be nice.
> 
> Can you elaborate?  Why can't noinstr inline noinstr?  (that's a
> mouthful)
> 
> Is it because of potential cloning caused by IPA optimizations?

It is because noinstr includes an explicit noinline. This is to avoid
the cmpiler from inlining noinstr functions in regular functions --
which for obvious raisins must never happen.

The compiler does not take the whole __sectio() attribute as a strong
enough clue for inlining boundaries.

Re: Live Patching Microconference at Linux Plumbers

[Arnaldo Carvalho de Melo]

On April 14, 2023 4:04:36 PM GMT-03:00, Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> wrote:
>On Fri, Apr 14, 2023 at 7:12 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>>
>> Not to mention how objtool will react to compiled rust code (has it
>> already been tried?)
>
>Rust uses LLVM, so it should be generally fine -- at least some of the
>checks appear to work. For instance, I can trigger:
>
>      RUSTC L rust/kernel.o
>    rust/kernel.o: warning: objtool: .text+0x0: unreachable instruction
>
>      RUSTC [M] samples/rust/rust_minimal.o
>    samples/rust/rust_minimal.o: warning: objtool:
>_R..._6kernel6Module4init+0x172: unreachable instruction
>
>via a random instruction in the middle of nowhere in the former (with
>`global_asm!`) and a jumped-over instruction in the latter (with
>`asm!`).
>
>Moreover, we were already getting warnings when rethunk / x86 IBT is
>enabled (since we got `objtool` called for `vmlinux.o`), e.g.
>
>    vmlinux.o: warning: objtool: .rodata+0x18c58: data relocation to
>!ENDBR: _R...IsWhitespaceEEB4_+0x0
>
>    vmlinux.o: warning: objtool: _R...into_foreign+0x5: 'naked' return
>found in RETHUNK build
>
>I can send the patch to run it for all Rust object files via
>`$(cmd_objtool)`, unless you think it is a bad idea.
>
>Having said that, tooling may indeed have issues, e.g. Arnaldo (Cc'd)
>is improving `pahole` to avoid assumptions around struct layout like
>field reordering (which Rust does by default).

That part should be dealt with the recently released pahole 1.25:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=c4eb1897d1f3841d291ee39dc969c4212750cf2c
https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=1231b6b9b4d88e0084bef4254eb1a05eb9935c99

But there are more issues, I'll resume work on Rust soon, after a detour on Go, and I think the changes made for Go make the DWARF loading more robust and may even have helped with Rust already:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=31bc0d7410572f6e03e3ed9da7c8c6f0d8df23c8

- Arnaldo

>
>Cheers,
>Miguel

Re: Live Patching Microconference at Linux Plumbers

[Miguel Ojeda]

On Fri, Apr 14, 2023 at 10:22 PM Arnaldo Carvalho de Melo
<arnaldo.melo@gmail.com> wrote:
>
> That part should be dealt with the recently released pahole 1.25:
>
> https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=c4eb1897d1f3841d291ee39dc969c4212750cf2c
> https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=1231b6b9b4d88e0084bef4254eb1a05eb9935c99
>
> But there are more issues, I'll resume work on Rust soon, after a detour on Go, and I think the changes made for Go make the DWARF loading more robust and may even have helped with Rust already:
>
> https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=31bc0d7410572f6e03e3ed9da7c8c6f0d8df23c8

Thanks Arnaldo! On my side, I still have to come back to `pahole` (for
the test cases).

Cheers,
Miguel

Re: Live Patching Microconference at Linux Plumbers

[Josh Poimboeuf]

On Fri, Apr 14, 2023 at 09:04:36PM +0200, Miguel Ojeda wrote:
> On Fri, Apr 14, 2023 at 7:12 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> >
> > Not to mention how objtool will react to compiled rust code (has it
> > already been tried?)
> 
> Rust uses LLVM, so it should be generally fine -- at least some of the
> checks appear to work. For instance, I can trigger:
> 
>       RUSTC L rust/kernel.o
>     rust/kernel.o: warning: objtool: .text+0x0: unreachable instruction
> 
>       RUSTC [M] samples/rust/rust_minimal.o
>     samples/rust/rust_minimal.o: warning: objtool:
> _R..._6kernel6Module4init+0x172: unreachable instruction
> 
> via a random instruction in the middle of nowhere in the former (with
> `global_asm!`) and a jumped-over instruction in the latter (with
> `asm!`).
> 
> Moreover, we were already getting warnings when rethunk / x86 IBT is
> enabled (since we got `objtool` called for `vmlinux.o`), e.g.
> 
>     vmlinux.o: warning: objtool: .rodata+0x18c58: data relocation to
> !ENDBR: _R...IsWhitespaceEEB4_+0x0
> 
>     vmlinux.o: warning: objtool: _R...into_foreign+0x5: 'naked' return
> found in RETHUNK build
> 
> I can send the patch to run it for all Rust object files via
> `$(cmd_objtool)`, unless you think it is a bad idea.

That's good to hear that it seems to just work.  Feel free to send a
patch to enable it for Rust objects, though I may need your help if we
see any odd warnings ;-)

-- 
Josh

Re: Live Patching Microconference at Linux Plumbers

[Josh Poimboeuf]

On Fri, Apr 14, 2023 at 09:30:13PM +0200, Peter Zijlstra wrote:
> On Fri, Apr 14, 2023 at 10:12:55AM -0700, Josh Poimboeuf wrote:
> 
> > > * How to avoid/minimize the need to reverse-engineer control flow for things
> > >   like ORC generation.
> > > 
> > >   On the arm64 side we're pretty averse to doing this to generate metadata for
> > >   unwinding (and we might not need to), but there are things objtool does today
> > >   that requires awareness of control-flow (e.g. forward-edge checks for noinstr
> > >   safety).
> > > 
> > >   Hopefully without a flamewar about DWARF...
> > 
> > If objtool is going to be doing control-flow anyway then it could just
> > validate DWARF/SFrame.  Then everybody's happy?
> 
> Right; so per another recent thread somewhere; you can't rely on
> DWARF/Sframe or any other compiler generated thing simply because it
> doesn't cover .S files and inline asm -- and this being a kernel, we've
> got quite a bit of that.
> 
> At best it could use DWARF to help reconstruct code flow and then
> validate Sframe for the bits that got sframe.

I wasn't (necessarily) suggesting that objtool use DWARF as an input to
help it construct the control-flow graph (CFG).

Instead, the idea is for objtool to continue to reverse-engineer the CFG
as it does today (albeit with a little help from the compiler in
specific problematic areas, e.g. noreturns and jump tables).

Then, it could use that independently-developed CFG to read the
compiler-generated metadata (DWARF/SFrame/whatever), and report any
warnings if the DWARF doesn't match objtool's CFG.

In other words, DWARF validation becomes just another optional objtool
feature, similar to its other unwinding-related features like frame
pointer validation and ORC generation.

That way, regardless of which philosophy [1] you subscribe to, if
something is amiss with reliable unwinding, the end result is the same:
a warning.

Then a human can look at the warning and investigate whether it's
objtool or DWARF/whatever that needs to be fixed.

[1] "reverse engineering is risky" vs "reverse engineering is reliable"

-- 
Josh

Re: Live Patching Microconference at Linux Plumbers

[Peter Zijlstra]

On Fri, Apr 14, 2023 at 09:39:49PM -0700, Josh Poimboeuf wrote:

> Instead, the idea is for objtool to continue to reverse-engineer the CFG
> as it does today (albeit with a little help from the compiler in
> specific problematic areas, e.g. noreturns and jump tables).

Yes, that would be best. I was merely suggesting that as a stop gap
measure -- until compilers can get us this -- we might perhaps
optionally extract this from DWARF.

But yeah, I'm not a great fan of using DWARF as input for this either.

Re: Live Patching Microconference at Linux Plumbers

[Mark Rutland]

On Fri, Apr 14, 2023 at 10:12:55AM -0700, Josh Poimboeuf wrote:
> On Fri, Apr 14, 2023 at 10:53:03AM +0100, Mark Rutland wrote:
> > > Currently proposed topics follow. The list is open though and more will
> > > be added during the regular Call for Topics.
> > > 
> > >   - klp-convert (as means to fix CET IBT limitations) and its 
> > >     upstreamability
> > >   - shadow variables, global state transition
> > >   - kselftests and the future direction of development
> > >   - arm64 live patching
> > 
> > I'm happy to talk about the kernel-side of arm64 live patching; it'd be good to
> > get in contact with anyone looking at the arm64 userspace side (e.g.
> > klp-convert).
> 
> klp-convert is still considered experimental.  FYI here's a pull request
> which adds arm64 support to kpatch-build:
> 
>   https://github.com/dynup/kpatch/pull/1302

Ah; I'm clearly not familiar with the userspace side at all!

> > I have some topics which overlap between live-patching and general toolchain
> > bits and pieces, and I'm not sure if they'd be best suited here or in a
> > toolchain track, e.g.
> > 
> > * How to avoid/minimize the need to reverse-engineer control flow for things
> >   like ORC generation.
> > 
> >   On the arm64 side we're pretty averse to doing this to generate metadata for
> >   unwinding (and we might not need to), but there are things objtool does today
> >   that requires awareness of control-flow (e.g. forward-edge checks for noinstr
> >   safety).
> > 
> >   Hopefully without a flamewar about DWARF...
> 
> If objtool is going to be doing control-flow anyway then it could just
> validate DWARF/SFrame.  Then everybody's happy?
> 
> > * Better compiler support for noinstr and similar properties.
> > 
> >   For example, noinstr functions are currently all noinline, and we can't
> >   inline a noinstr function into a noinstr function, leading to a painful mix
> >   of noinstr and __always_inline. Having a mechanism to allow noinstr code to
> >   be inlined into other noinstr code would be nice.
> 
> Can you elaborate?  Why can't noinstr inline noinstr?  (that's a
> mouthful)
> 
> Is it because of potential cloning caused by IPA optimizations?

Today, noinstr is marks as noinline:

| /* Section for code which can't be instrumented at all */
| #define __noinstr_section(section)                                      \
|         noinline notrace __attribute((__section__(section)))            \
|         __no_kcsan __no_sanitize_address __no_profile __no_sanitize_coverage \
|         __no_sanitize_memory
| 
| #define noinstr __noinstr_section(".noinstr.text")

My understnading is that without that, a noinstr function *could* get inlined
into an instrumentable function and get instrumented (e.g. in the case of a
static noinstr function with a single caller).

> >   Likewise, whether we could somehow get compile-time warnings about unintended
> >   calls from instrumentable code from noinstr code.
> > 
> > * How is this going to work with rust?
> > 
> >   It's not clear to me whether/how things like ftrace, RELIABLE_STACKTRACE, and
> >   live-patching are going to work with rust.
> 
> Not to mention how objtool will react to compiled rust code (has it
> already been tried?)

I have no idea :)

Thanks,
Mark

Re: Live Patching Microconference at Linux Plumbers

[James Morris]

On Fri, 14 Apr 2023, Mark Rutland wrote:

> I'm happy to talk about the kernel-side of arm64 live patching; it'd be good to
> get in contact with anyone looking at the arm64 userspace side (e.g.
> klp-convert).
> 
> I have some topics which overlap between live-patching and general toolchain
> bits and pieces, and I'm not sure if they'd be best suited here or in a
> toolchain track, e.g.
> 

> I've Cc'd Nick, Jose, and Miguel, in case they have thoughts.


Adding Madhavan, who is implementing the kernel code.



-- 
James Morris
<jmorris@namei.org>

Re: Live Patching Microconference at Linux Plumbers

[Joe Lawrence]

On 4/14/23 05:53, Mark Rutland wrote:
> 
> * How is this going to work with rust?
> 
>   It's not clear to me whether/how things like ftrace, RELIABLE_STACKTRACE, and
>   live-patching are going to work with rust. We probably need to start looking
>   soon.
> 

[ cc += Steve ]

For me, any explanation of kernel livepatching to another kernel dev
usually starts with ftrace, handlers, function granularity, etc.
Thinking about livepatching + rust, I can only imagine there will be a
lot of known and unknown gotchas with respect to data scoping, stacks,
relocations, etc... but I would still work my way up from learning more
about how / if Rust code will be trace-able and what that roadmap may be.

Any thoughts on that Steve?  I see that the "Kernel Testing &
Dependability" microconf has Rust on their proposal, are there any other
planned talks re: ftrace / rust?

Regards,
-- 
Joe

Re: Live Patching Microconference at Linux Plumbers

[Steven Rostedt]

On Wed, 3 May 2023 16:05:15 -0400
Joe Lawrence <joe.lawrence@redhat.com> wrote:

> On 4/14/23 05:53, Mark Rutland wrote:
> > 
> > * How is this going to work with rust?
> > 
> >   It's not clear to me whether/how things like ftrace, RELIABLE_STACKTRACE, and
> >   live-patching are going to work with rust. We probably need to start looking
> >   soon.
> >   
> 
> [ cc += Steve ]
> 
> For me, any explanation of kernel livepatching to another kernel dev
> usually starts with ftrace, handlers, function granularity, etc.
> Thinking about livepatching + rust, I can only imagine there will be a
> lot of known and unknown gotchas with respect to data scoping, stacks,
> relocations, etc... but I would still work my way up from learning more
> about how / if Rust code will be trace-able and what that roadmap may be.
> 
> Any thoughts on that Steve?  I see that the "Kernel Testing &
> Dependability" microconf has Rust on their proposal, are there any other
> planned talks re: ftrace / rust?
>

Thanks for bringing this up. ftrace on rust has been on the back of my
mind, and yeah, we should start looking into it. I should push for a
tracing MC, we haven't had one in a few years.

-- Steve

Re: Live Patching Microconference at Linux Plumbers

[Joe Lawrence]

On 3/29/23 08:05, Miroslav Benes wrote:
> Hi,
> 
> we would like to organize Live Patching Microconference at Linux Plumbers 
> 2023. The conference will take place in Richmond, VA, USA. 13-15 November. 
> https://lpc.events/. The call for proposals has been opened so it is time 
> to start the preparation on our side.
> 
> You can find the proposal below. Comments are welcome. The list of topics 
> is open, so feel free to add more. I tried to add key people to discuss 
> the topics, but the list is not exhaustive. I would like to submit the 
> proposal soonish even though the deadline is on 1 June. I assume that we 
> can update the topics later. My plan is to also organize a proper Call for 
> Topics after the submission and advertise it also on LKML.
> 
> Last but not least it would be nice to have a co-runner of the show. Josh, 
> Joe, any volunteer? :)
> 
> Thank you
> Miroslav
> 
> 
> Proposal
> --------
> The Live Patching microconference at Linux Plumbers 2023 aims to gather
> stakeholders and interested parties to discuss proposed features and
> outstanding issues in live patching.
> 
> Live patching is a critical tool for maintaining system uptime and
> security by enabling fixes to be applied to running systems without the
> need for a reboot. The development of the infrastructure is an ongoing
> effort and while many problems have been resolved and features
> implemented, there are still open questions, some with already submitted
> patch sets, which need to be discussed.
> 
> Live Patching microconferences at the previous Linux Plumbers
> conferences proved to be useful in this regard and helped us to find
> final solutions or at least promising directions to push the development
> forward. It includes for example a support for several architectures
> (ppc64le and s390x were added after x86_64), a late module patching and
> module dependencies and user space live patching.
> 
> Currently proposed topics follow. The list is open though and more will
> be added during the regular Call for Topics.
> 
>   - klp-convert (as means to fix CET IBT limitations) and its 
>     upstreamability
>   - shadow variables, global state transition
>   - kselftests and the future direction of development
>   - arm64 live patching
> 
> Key people
> 
>   - Josh Poimboeuf <jpoimboe@kernel.org>
>   - Jiri Kosina <jikos@kernel.org>
>   - Miroslav Benes <mbenes@suse.cz>
>   - Petr Mladek <pmladek@suse.com>
>   - Joe Lawrence <joe.lawrence@redhat.com>
>   - Nicolai Stange <nstange@suse.de>
>   - Marcos Paulo de Souza <mpdesouza@suse.de>
>   - Mark Rutland <mark.rutland@arm.com>
>   - Mark Brown <broonie@kernel.org>
> 
> We encourage all attendees to actively participate in the
> microconference by sharing their ideas, experiences, and insights.
> 

Hi folks,

Gentle bump and call for any late updates to the Live Patching
Microconference Proposal.  I believe we have about a week left to edit
the proposal for consideration at this year's LPC.

See [1] for more info on LPC 2023 and [2] for the currently proposed
microconferences.

[1] https://lpc.events/event/17/
[2[ https://lpc.events/event/17/page/200-proposed-microconferences

-- 
Joe