announcing LLpatch: arch-independent live-patch creation

announcing LLpatch: arch-independent live-patch creation

[Peter Swain]

We have a new userspace live-patch creation tool, LLpatch, paralleling
kpatch-build, but without requiring its arch-specific code for ELF
analysis and manipulation.

We considered extending kpatch-build to a new target architecture
(arm64), cluttering its code with details of another architecture’s
quirky instruction sequences & relocation modes, and suspected there
might be a better way.


The LLVM suite already knows these details, and offers llvm-diff, for
comparing generated code at the LLVM-IR (internal representation)
level, which has access to much more of the code’s _intent_ than
kpatch’s create-diff-object is able to infer from ELF-level
differences.


Building on this, LLpatch adds namespace analysis, further
dead/duplicate code elimination, and creation of patch modules
compatible with kernel’s livepatch API.

Arm64 is supported - testing against a livepatch-capable v5.12 arm64
kernel, using the preliminary reliable-stacktrace work from
madvenka@linux.microsoft.com, LLpatch modules for x86 and arm64 behave
identically to the x86 kpatch-build modules, without requiring any
additional arch-specific code.

On x86, where both tools are available, LLpatch produces smaller patch
modules than kpatch, and already correctly handles most of the kpatch
test cases, without any arch-specific code. This suggests it can work
with any clang-supported kernel architecture.


Work is ongoing, collaboration is welcome.


See https://github.com/google/LLpatch for further details on the
technology and its benefits.


Yonghyun Hwang (yonghyun@google.com freeaion@gmail.com)
Bill Wendling (morbo@google.com isanbard@gmail.com)
Pete Swain (swine@google.com swine@pobox.com)

Re: announcing LLpatch: arch-independent live-patch creation

[Madhavan T. Venkataraman]

On 8/26/21 5:34 PM, Peter Swain wrote:
> We have a new userspace live-patch creation tool, LLpatch, paralleling
> kpatch-build, but without requiring its arch-specific code for ELF
> analysis and manipulation.
> 
> We considered extending kpatch-build to a new target architecture
> (arm64), cluttering its code with details of another architecture’s
> quirky instruction sequences & relocation modes, and suspected there
> might be a better way.
> 
> 
> The LLVM suite already knows these details, and offers llvm-diff, for
> comparing generated code at the LLVM-IR (internal representation)
> level, which has access to much more of the code’s _intent_ than
> kpatch’s create-diff-object is able to infer from ELF-level
> differences.
> 
> 
> Building on this, LLpatch adds namespace analysis, further
> dead/duplicate code elimination, and creation of patch modules
> compatible with kernel’s livepatch API.
> 
> Arm64 is supported - testing against a livepatch-capable v5.12 arm64
> kernel, using the preliminary reliable-stacktrace work from
> madvenka@linux.microsoft.com, LLpatch modules for x86 and arm64 behave
> identically to the x86 kpatch-build modules, without requiring any
> additional arch-specific code.
> 
> On x86, where both tools are available, LLpatch produces smaller patch
> modules than kpatch, and already correctly handles most of the kpatch
> test cases, without any arch-specific code. This suggests it can work
> with any clang-supported kernel architecture.
> 
> 
> Work is ongoing, collaboration is welcome.
> 
> 
> See https://github.com/google/LLpatch for further details on the
> technology and its benefits.
> 
> 
> Yonghyun Hwang (yonghyun@google.com freeaion@gmail.com)
> Bill Wendling (morbo@google.com isanbard@gmail.com)
> Pete Swain (swine@google.com swine@pobox.com)
> 

This is great.

I have implemented an alternative method in objtool to do stack
validation for livepatch purposes. I have successfully built a livepatch
kernel and tested it. I have run all the livepatch tests in the
linux kernel sources successfully.

But I needed kpatch (or something similar) to do more testing. From Josh,
I came to know that a port to ARM64 exists for kpatch. But I was not sure
how well ARM64 was supported.

Since your tool already works on ARM64, I could really use your tool for
my testing. I will study it and contact you with any questions I might
have or any help that I might need. If everything works, I can give
you a "Tested-by".

Thanks.

Madhavan

Re: announcing LLpatch: arch-independent live-patch creation

[Evgenii Shatokhin]

Hi,

On 27.08.2021 01:34, Peter Swain wrote:
> We have a new userspace live-patch creation tool, LLpatch, paralleling
> kpatch-build, but without requiring its arch-specific code for ELF
> analysis and manipulation.
> 
> We considered extending kpatch-build to a new target architecture
> (arm64), cluttering its code with details of another architecture’s
> quirky instruction sequences & relocation modes, and suspected there
> might be a better way.
> 
> 
> The LLVM suite already knows these details, and offers llvm-diff, for
> comparing generated code at the LLVM-IR (internal representation)
> level, which has access to much more of the code’s _intent_ than
> kpatch’s create-diff-object is able to infer from ELF-level
> differences.
> 

Interesting.

LLpatch requires a pre-built kernel tree ("repository"), right?

Does that mean that the kernel should be built with clang first?
Or, perhaps, clang is only used when building the patch itself, while 
the kernel can be built with GCC or other compiler used by the given 
Linux distro?

> 
> Building on this, LLpatch adds namespace analysis, further
> dead/duplicate code elimination, and creation of patch modules
> compatible with kernel’s livepatch API.
> 
> Arm64 is supported - testing against a livepatch-capable v5.12 arm64
> kernel, using the preliminary reliable-stacktrace work from
> madvenka@linux.microsoft.com, LLpatch modules for x86 and arm64 behave
> identically to the x86 kpatch-build modules, without requiring any
> additional arch-specific code.
> 
> On x86, where both tools are available, LLpatch produces smaller patch
> modules than kpatch, and already correctly handles most of the kpatch
> test cases, without any arch-specific code. This suggests it can work
> with any clang-supported kernel architecture.
> 
> 
> Work is ongoing, collaboration is welcome.
> 
> 
> See https://github.com/google/LLpatch for further details on the
> technology and its benefits.
> 
> 
> Yonghyun Hwang (yonghyun@google.com freeaion@gmail.com)
> Bill Wendling (morbo@google.com isanbard@gmail.com)
> Pete Swain (swine@google.com swine@pobox.com)
> .
> 

Regards,
Evgenii

Re: announcing LLpatch: arch-independent live-patch creation

[Peter Swain]

(apologies for re-post, having trouble enforcing no-HTML mode in gmail)

On Fri, Aug 27, 2021 at 9:08 AM Evgenii Shatokhin <eshatokhin@virtuozzo.com>
wrote:
> LLpatch requires a pre-built kernel tree ("repository"), right?
Yes.
It doesn't rebuild existing *.o as a pre-image of the patched code, as
kpatch does.
But LLpatch does use the dependency analysis the previous make left in
*.o.cmd, although the same information could also be extracted by "make -n
-W changed_file ..."

> Does that mean that the kernel should be built with clang first?
> Or, perhaps, clang is only used when building the patch itself, while
> the kernel can be built with GCC or other compiler used by the given
> Linux distro?
We haven't explored this deeply, as all our kernels are clang-built.
In principle this should work with gcc-built kernels, as long as the
particular change doesn't intersect with some feature which is expressed
differently between the gcc/clang worlds, such as some ELF section names.
But as there are so many such potential incompatibilities, we do not
recommend this.
As a precondition for LLpatch-patchable kernels, I would recommend moving
to clang-built base kernels

Re: announcing LLpatch: arch-independent live-patch creation

[Evgenii Shatokhin]

On 27.08.2021 21:33, Peter Swain wrote:
> (apologies for re-post, having trouble enforcing no-HTML mode in gmail)
> 
> On Fri, Aug 27, 2021 at 9:08 AM Evgenii Shatokhin <eshatokhin@virtuozzo.com>
> wrote:
>> LLpatch requires a pre-built kernel tree ("repository"), right?
> Yes.
> It doesn't rebuild existing *.o as a pre-image of the patched code, as
> kpatch does.
> But LLpatch does use the dependency analysis the previous make left in
> *.o.cmd, although the same information could also be extracted by "make -n
> -W changed_file ..."
> 
>> Does that mean that the kernel should be built with clang first?
>> Or, perhaps, clang is only used when building the patch itself, while
>> the kernel can be built with GCC or other compiler used by the given
>> Linux distro?
> We haven't explored this deeply, as all our kernels are clang-built.
> In principle this should work with gcc-built kernels, as long as the
> particular change doesn't intersect with some feature which is expressed
> differently between the gcc/clang worlds, such as some ELF section names.
> But as there are so many such potential incompatibilities, we do not
> recommend this.

I see. Yes, it is safer to use the same toolchain both for the kernel 
and for the patch.

> As a precondition for LLpatch-patchable kernels, I would recommend moving
> to clang-built base kernels

This is not always an option ;-) For example, the kernels in our 
(Virtuozzo) Linux distros are based on those from RHEL. We try to change 
only what we really need to, otherwise we lose the benefits of a 
well-tested code base. So, the compiler and even its version are 
basically fixed for us.

Thanks, anyway.
> .
> 

Regards,
Evgenii