ppc64le STRICT_MODULE_RWX and livepatch apply_relocate_add() crashes

ppc64le STRICT_MODULE_RWX and livepatch apply_relocate_add() crashes

[Joe Lawrence]

Starting with 5.14 kernels, I can reliably reproduce a crash [1] on
ppc64le when loading livepatches containing late klp-relocations [2].
These are relocations, specific to livepatching, that are resolved not
when a livepatch module is loaded, but only when a livepatch-target
module is loaded.

There was previously related work by Josh and Peter [3] to simplify a
lot of x86 and s390x code (at the time, the only two arches to
HAVE_LIVEPATCH and STRICT_MODULE_RWX) as part of disallowing writable
executable mappings.

Now that Power has STRICT_MODULE_RWX, I think we will need to consider
this architecture as well.

The crash was originally spotted by the external kpatch-build tool [4]
when building its integration tests on rhel-9-beta.  It can also be
reproduced by the endless-WIP klp-convert patchset [5], which brings
klp-relocation creation from kpatch-build to the upstream build.

I further verified:
  - turning STRICT_MODULE_RWX off resulted in no crash
  - alternatively, reverting the following commits resulted in no crash:

    d556e1be3332 ("livepatch: Remove module_disable_ro() usage")
    0d9fbf78fefb ("module: Remove module_disable_ro()")

I haven't started looking at a fix yet, but in the case of the x86 code
update, its apply_relocate_add() implementation was modified to use a
common text_poke() function to allowed us to drop
module_{en,dis}ble_ro() games by the livepatching code.

I can take a closer look this week, but thought I'd send out a report
in case this may be a known todo for STRICT_MODULE_RWX on Power.

-- Joe


[1] crashing kernel log

[   84.837986] ===== TEST: klp-convert symbols =====
[   84.858937] % modprobe test_klp_convert_mod
[   84.879040] % modprobe test_klp_convert1
[   84.908056] BUG: Unable to handle kernel data access on write at 0xc0080000018402f0
[   84.908067] Faulting instruction address: 0xc000000000056b58
[   84.908072] Oops: Kernel access of bad area, sig: 11 [#1]
[   84.908077] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
[   84.908082] Modules linked in: test_klp_convert1(K+) test_klp_convert_mod bonding tls rfkill pseries_rng drm fuse drm_panel_orientation_quirks xfs libcrc32c sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp vmx_crypto dm_mirror dm_region_hash dm_log dm_mod [last unloaded: test_klp_atomic_replace]
[   84.908114] CPU: 1 PID: 4205 Comm: modprobe Kdump: loaded Tainted: G              K   5.14.0+ #2
[   84.908121] NIP:  c000000000056b58 LR: c000000000056b1c CTR: 0000000000000009
[   84.908127] REGS: c00000005dce3480 TRAP: 0300   Tainted: G              K    (5.14.0+)
[   84.908132] MSR:  800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  CR: 24224484  XER: 00000000
[   84.908147] CFAR: c000000000056a68 DAR: c0080000018402f0 DSISR: 0a000000 IRQMASK: 0 
GPR00: c000000000056b1c c00000005dce3720 c000000002a2af00 0000000000000000 
GPR04: c0080000018402f0 396b00003d620000 e98b0020f8410018 00000000ffffffff 
GPR08: 4e8004207d8903a6 0000000080000000 c0080000018382f0 000000000000000d 
GPR12: 0000000000004000 c000000007fcf480 c00000004d7e2000 c0080000018706d8 
GPR16: c008000001850228 c00000004d7e2c00 00000000ffffffff c0000000010d6248 
GPR20: c00000000298c1c8 c008000001860380 c0080000018706f0 aaaaaaaaaaaaaaab 
GPR24: c00000004d7e2b40 c008000001870000 c00800000184005c 000000000000008c 
GPR28: c008000001860380 c008000000770008 c00000004d7e2000 c0080000018402f0 
[   84.908209] NIP [c000000000056b58] create_stub+0x78/0x240
[   84.908217] LR [c000000000056b1c] create_stub+0x3c/0x240
[   84.908223] Call Trace:
[   84.908225] [c00000005dce3720] [c00000004d7e2b40] 0xc00000004d7e2b40 (unreliable)
[   84.908232] [c00000005dce37a0] [c000000000056e0c] stub_for_addr+0xec/0x120
[   84.908240] [c00000005dce37d0] [c000000000057f14] apply_relocate_add+0x814/0x9a0
[   84.908247] [c00000005dce38d0] [c00000000021ca38] klp_apply_section_relocs+0x208/0x2d0
[   84.908255] [c00000005dce39c0] [c00000000021cb90] klp_init_object_loaded+0x90/0x1d0
[   84.908262] [c00000005dce3a50] [c00000000021d2dc] klp_enable_patch+0x32c/0x540
[   84.908269] [c00000005dce3b10] [c008000001840030] test_klp_convert_init+0x28/0x48 [test_klp_convert1]
[   84.908277] [c00000005dce3b30] [c000000000012230] do_one_initcall+0x60/0x2c0
[   84.908284] [c00000005dce3c00] [c00000000026012c] do_init_module+0x7c/0x3b0
[   84.908290] [c00000005dce3c90] [c000000000262b74] __do_sys_finit_module+0xd4/0x160
[   84.908296] [c00000005dce3db0] [c000000000030664] system_call_exception+0x144/0x280
[   84.908303] [c00000005dce3e10] [c00000000000bff0] system_call_vectored_common+0xf0/0x280
[   84.908310] --- interrupt: 3000 at 0x7fffa06d6b9c
[   84.908315] NIP:  00007fffa06d6b9c LR: 0000000000000000 CTR: 0000000000000000
[   84.908320] REGS: c00000005dce3e80 TRAP: 3000   Tainted: G              K    (5.14.0+)
[   84.908325] MSR:  800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE>  CR: 28224244  XER: 00000000
[   84.908340] IRQMASK: 0 
GPR00: 0000000000000161 00007fffc4f74ad0 00007fffa07d7100 0000000000000005 
GPR04: 000000012a926ca0 0000000000000000 0000000000000005 0000000000000000 
GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 
GPR12: 0000000000000000 00007fffa0f9c380 0000000000000020 0000000000000000 
GPR16: 00000100010a1de0 0000000000000000 000000012a927d50 00000100010a02f8 
GPR20: 0000000000000001 0000000000000908 00000100010a2020 00000100010a19b0 
GPR24: 0000000000000000 0000000000000000 00000100010a2040 00000100010a03f0 
GPR28: 00000100010a1e00 000000012a926ca0 0000000000040000 00000100010a19b0 
[   84.908399] NIP [00007fffa06d6b9c] 0x7fffa06d6b9c
[   84.908403] LR [0000000000000000] 0x0
[   84.908406] --- interrupt: 3000
[   84.908410] Instruction dump:
[   84.908413] 3d02ffb2 395f8000 3d208000 3ce0ffff 38c68d70 39088d84 79290020 60e7ffff 
[   84.908423] e8a60014 e8c80008 e9080010 78e70020 <f8bf0000> f8df0008 f91f0010 811c0224 
[   84.908435] ---[ end trace 961b4b817da4a53b ]---

[2] https://www.kernel.org/doc/html/latest/livepatch/module-elf-format.html
[3] https://lore.kernel.org/lkml/cover.1588173720.git.jpoimboe@redhat.com/
[4] https://github.com/dynup/kpatch/issues/1228
[5] https://github.com/joe-lawrence/linux/tree/klp-convert-v5-expanded-v5.14-rebase1

Re: ppc64le STRICT_MODULE_RWX and livepatch apply_relocate_add() crashes

[Russell Currey]

On Sun, 2021-10-31 at 22:43 -0400, Joe Lawrence wrote:
> Starting with 5.14 kernels, I can reliably reproduce a crash [1] on
> ppc64le when loading livepatches containing late klp-relocations [2].
> These are relocations, specific to livepatching, that are resolved not
> when a livepatch module is loaded, but only when a livepatch-target
> module is loaded.

Hey Joe, thanks for the report.

> I haven't started looking at a fix yet, but in the case of the x86 code
> update, its apply_relocate_add() implementation was modified to use a
> common text_poke() function to allowed us to drop
> module_{en,dis}ble_ro() games by the livepatching code.

It should be a similar fix for Power, our patch_instruction() uses a
text poke area but apply_relocate_add() doesn't use it and does its own
raw patching instead.

> I can take a closer look this week, but thought I'd send out a report
> in case this may be a known todo for STRICT_MODULE_RWX on Power.

I'm looking into this now, will update when there's progress.  I
personally wasn't aware but Jordan flagged this as an issue back in
August [0].  Are the selftests in the klp-convert tree sufficient for
testing?  I'm not especially familiar with livepatching & haven't used
the userspace tools.

- Russell

[0] https://github.com/linuxppc/issues/issues/375

> 
> -- Joe

Re: ppc64le STRICT_MODULE_RWX and livepatch apply_relocate_add() crashes

[Joe Lawrence]

On 11/1/21 5:20 AM, Russell Currey wrote:
> I'm looking into this now, will update when there's progress.  I
> personally wasn't aware but Jordan flagged this as an issue back in
> August [0].  Are the selftests in the klp-convert tree sufficient for
> testing?  I'm not especially familiar with livepatching & haven't used
> the userspace tools.
> 

Hi Russell, thanks for taking a look.

Testing with that klp-convert tree is probably the quickest and easiest
way to verify the late relocations.

I'm happy to setup and test additional tools (ie, kpatch-build) with any
potential changes as I know they take longer to config and run.

Thanks,

-- 
Joe

Re: ppc64le STRICT_MODULE_RWX and livepatch apply_relocate_add() crashes

[Suraj Jitindar Singh]

Hi Russell,

On Mon, 2021-11-01 at 19:20 +1000, Russell Currey wrote:
> On Sun, 2021-10-31 at 22:43 -0400, Joe Lawrence wrote:
> > Starting with 5.14 kernels, I can reliably reproduce a crash [1] on
> > ppc64le when loading livepatches containing late klp-relocations
> > [2].
> > These are relocations, specific to livepatching, that are resolved
> > not
> > when a livepatch module is loaded, but only when a livepatch-target
> > module is loaded.
> 
> Hey Joe, thanks for the report.
> 
> > I haven't started looking at a fix yet, but in the case of the x86
> > code
> > update, its apply_relocate_add() implementation was modified to use
> > a
> > common text_poke() function to allowed us to drop
> > module_{en,dis}ble_ro() games by the livepatching code.
> 
> It should be a similar fix for Power, our patch_instruction() uses a
> text poke area but apply_relocate_add() doesn't use it and does its
> own
> raw patching instead.
> 
> > I can take a closer look this week, but thought I'd send out a
> > report
> > in case this may be a known todo for STRICT_MODULE_RWX on Power.
> 
> I'm looking into this now, will update when there's progress.  I
> personally wasn't aware but Jordan flagged this as an issue back in
> August [0].  Are the selftests in the klp-convert tree sufficient for
> testing?  I'm not especially familiar with livepatching & haven't
> used
> the userspace tools.
> 

You can test this by livepatching any module since this only occurs
when writing relocations for modules since the vmlinux relocations are
written earlier before the module text is mapped read-only.

- Suraj

> - Russell
> 
> [0] https://github.com/linuxppc/issues/issues/375
> 
> > 
> > -- Joe
> 
> 

Re: ppc64le STRICT_MODULE_RWX and livepatch apply_relocate_add() crashes

[Christophe Leroy]

Le 01/11/2021 à 14:48, Joe Lawrence a écrit :
> On 11/1/21 5:20 AM, Russell Currey wrote:
>> I'm looking into this now, will update when there's progress.  I
>> personally wasn't aware but Jordan flagged this as an issue back in
>> August [0].  Are the selftests in the klp-convert tree sufficient for
>> testing?  I'm not especially familiar with livepatching & haven't used
>> the userspace tools.
>>
> 
> Hi Russell, thanks for taking a look.
> 
> Testing with that klp-convert tree is probably the quickest and easiest
> way to verify the late relocations.
> 
> I'm happy to setup and test additional tools (ie, kpatch-build) with any
> potential changes as I know they take longer to config and run.
> 

Hello Joe,

I'm implementing LIVEPATCH on PPC32 and I wanted to test with 
STRICT_MODULE_RWX enabled so I took your branch as suggested, but I'm 
getting the following errors on build. What shall I do ?

   CALL    scripts/checksyscalls.sh
   CALL    scripts/atomic/check-atomics.sh
   CHK     include/generated/compile.h
   KLP     lib/livepatch/test_klp_convert1.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length 
beyond nr_entries

klp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79: 
lib/livepatch/test_klp_convert1.ko] Error 255
   KLP     lib/livepatch/test_klp_convert2.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length 
beyond nr_entries

klp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79: 
lib/livepatch/test_klp_convert2.ko] Error 255
   KLP     lib/livepatch/test_klp_convert_sections.ko
klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length 
beyond nr_entries

klp-convert: Unable to load user-provided sympos
make[2]: *** [scripts/Makefile.modfinal:79: 
lib/livepatch/test_klp_convert_sections.ko] Error 255
make[2]: Target '__modfinal' not remade because of errors.
make[1]: *** [scripts/Makefile.modpost:145: __modpost] Error 2
make: *** [Makefile:1770: modules] Error 2

Christophe

Re: ppc64le STRICT_MODULE_RWX and livepatch apply_relocate_add() crashes

[Joe Lawrence]

On 12/13/21 2:42 AM, Christophe Leroy wrote:
> 
> Hello Joe,
> 
> I'm implementing LIVEPATCH on PPC32 and I wanted to test with 
> STRICT_MODULE_RWX enabled so I took your branch as suggested, but I'm 
> getting the following errors on build. What shall I do ?
> 
>    CALL    scripts/checksyscalls.sh
>    CALL    scripts/atomic/check-atomics.sh
>    CHK     include/generated/compile.h
>    KLP     lib/livepatch/test_klp_convert1.ko
> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length 
> beyond nr_entries
> 
> klp-convert: Unable to load user-provided sympos
> make[2]: *** [scripts/Makefile.modfinal:79: 
> lib/livepatch/test_klp_convert1.ko] Error 255
>    KLP     lib/livepatch/test_klp_convert2.ko
> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length 
> beyond nr_entries
> 
> klp-convert: Unable to load user-provided sympos
> make[2]: *** [scripts/Makefile.modfinal:79: 
> lib/livepatch/test_klp_convert2.ko] Error 255
>    KLP     lib/livepatch/test_klp_convert_sections.ko
> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length 
> beyond nr_entries
> 
> klp-convert: Unable to load user-provided sympos
> make[2]: *** [scripts/Makefile.modfinal:79: 
> lib/livepatch/test_klp_convert_sections.ko] Error 255
> make[2]: Target '__modfinal' not remade because of errors.
> make[1]: *** [scripts/Makefile.modpost:145: __modpost] Error 2
> make: *** [Makefile:1770: modules] Error 2
> 

Hi Christophe,

Interesting failure mode.  That's klp-convert complaining that it found
more relocations in a .klp.module_relocs.<objname> section than
expected, i.e. nr_entries = sec->size / sizeof(struct klp_module_reloc).

A few possibilities: the ELF sec->size was incorrectly set/read by
build/libelf (I doubt that).  Or maybe the layout/size of struct
klp_module_reloc is not consistent between kernel and userspace (I'm
more suspicious of this).

Can you post a copy of the build's symbols.klp and
lib/livepatch/test_klp_convert1.tmp.ko somewhere?  I should be able to
start debug with those files.

Thanks,
-- 
Joe

Re: ppc64le STRICT_MODULE_RWX and livepatch apply_relocate_add() crashes

[Christophe Leroy]

Le 13/12/2021 à 15:47, Joe Lawrence a écrit :
> On 12/13/21 2:42 AM, Christophe Leroy wrote:
>>
>> Hello Joe,
>>
>> I'm implementing LIVEPATCH on PPC32 and I wanted to test with
>> STRICT_MODULE_RWX enabled so I took your branch as suggested, but I'm
>> getting the following errors on build. What shall I do ?
>>
>>     CALL    scripts/checksyscalls.sh
>>     CALL    scripts/atomic/check-atomics.sh
>>     CHK     include/generated/compile.h
>>     KLP     lib/livepatch/test_klp_convert1.ko
>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>> beyond nr_entries
>>
>> klp-convert: Unable to load user-provided sympos
>> make[2]: *** [scripts/Makefile.modfinal:79:
>> lib/livepatch/test_klp_convert1.ko] Error 255
>>     KLP     lib/livepatch/test_klp_convert2.ko
>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>> beyond nr_entries
>>
>> klp-convert: Unable to load user-provided sympos
>> make[2]: *** [scripts/Makefile.modfinal:79:
>> lib/livepatch/test_klp_convert2.ko] Error 255
>>     KLP     lib/livepatch/test_klp_convert_sections.ko
>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>> beyond nr_entries
>>
>> klp-convert: Unable to load user-provided sympos
>> make[2]: *** [scripts/Makefile.modfinal:79:
>> lib/livepatch/test_klp_convert_sections.ko] Error 255
>> make[2]: Target '__modfinal' not remade because of errors.
>> make[1]: *** [scripts/Makefile.modpost:145: __modpost] Error 2
>> make: *** [Makefile:1770: modules] Error 2
>>
> 
> Hi Christophe,
> 
> Interesting failure mode.  That's klp-convert complaining that it found
> more relocations in a .klp.module_relocs.<objname> section than
> expected, i.e. nr_entries = sec->size / sizeof(struct klp_module_reloc).
> 
> A few possibilities: the ELF sec->size was incorrectly set/read by
> build/libelf (I doubt that).  Or maybe the layout/size of struct
> klp_module_reloc is not consistent between kernel and userspace (I'm
> more suspicious of this).
> 
> Can you post a copy of the build's symbols.klp and
> lib/livepatch/test_klp_convert1.tmp.ko somewhere?  I should be able to
> start debug with those files.
> 

I sent you both files off list.

It looks like klp-convert doesn't use the correct size. It finds a 
struct of size 12 hence 3 entries for a section of size 40.

On PPC32 the struct has size 8 (void * is 4 and int is 4).

But I'm cross-building from x86_64 where the struct is 8 + 4 = 12.

Can it be the reason ?

Christophe

Re: ppc64le STRICT_MODULE_RWX and livepatch apply_relocate_add() crashes

[Joe Lawrence]

On 12/13/21 11:36 AM, Christophe Leroy wrote:
> 
> 
> Le 13/12/2021 à 15:47, Joe Lawrence a écrit :
>> On 12/13/21 2:42 AM, Christophe Leroy wrote:
>>>
>>> Hello Joe,
>>>
>>> I'm implementing LIVEPATCH on PPC32 and I wanted to test with
>>> STRICT_MODULE_RWX enabled so I took your branch as suggested, but I'm
>>> getting the following errors on build. What shall I do ?
>>>
>>>     CALL    scripts/checksyscalls.sh
>>>     CALL    scripts/atomic/check-atomics.sh
>>>     CHK     include/generated/compile.h
>>>     KLP     lib/livepatch/test_klp_convert1.ko
>>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>>> beyond nr_entries
>>>
>>> klp-convert: Unable to load user-provided sympos
>>> make[2]: *** [scripts/Makefile.modfinal:79:
>>> lib/livepatch/test_klp_convert1.ko] Error 255
>>>     KLP     lib/livepatch/test_klp_convert2.ko
>>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>>> beyond nr_entries
>>>
>>> klp-convert: Unable to load user-provided sympos
>>> make[2]: *** [scripts/Makefile.modfinal:79:
>>> lib/livepatch/test_klp_convert2.ko] Error 255
>>>     KLP     lib/livepatch/test_klp_convert_sections.ko
>>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>>> beyond nr_entries
>>>
>>> klp-convert: Unable to load user-provided sympos
>>> make[2]: *** [scripts/Makefile.modfinal:79:
>>> lib/livepatch/test_klp_convert_sections.ko] Error 255
>>> make[2]: Target '__modfinal' not remade because of errors.
>>> make[1]: *** [scripts/Makefile.modpost:145: __modpost] Error 2
>>> make: *** [Makefile:1770: modules] Error 2
>>>
>>
>> Hi Christophe,
>>
>> Interesting failure mode.  That's klp-convert complaining that it found
>> more relocations in a .klp.module_relocs.<objname> section than
>> expected, i.e. nr_entries = sec->size / sizeof(struct klp_module_reloc).
>>
>> A few possibilities: the ELF sec->size was incorrectly set/read by
>> build/libelf (I doubt that).  Or maybe the layout/size of struct
>> klp_module_reloc is not consistent between kernel and userspace (I'm
>> more suspicious of this).
>>
>> Can you post a copy of the build's symbols.klp and
>> lib/livepatch/test_klp_convert1.tmp.ko somewhere?  I should be able to
>> start debug with those files.
>>
> 
> I sent you both files off list.
> 
> It looks like klp-convert doesn't use the correct size. It finds a 
> struct of size 12 hence 3 entries for a section of size 40.
> 
> On PPC32 the struct has size 8 (void * is 4 and int is 4).
> 
> But I'm cross-building from x86_64 where the struct is 8 + 4 = 12.
> 
> Can it be the reason ?
> 

I'm pretty sure that is it.  I haven't had much runtime with klp-convert
and cross-building (I've only found one big/little endian bug with
x86_64->s390x) and was going to ask you how you were testing :)

Do you know if there are other kernel build tools that deal with similar
situations?  This seems like a tricky job for the userspace build tool
to determine non-native target struct layout.

In the meantime, hacking in:

 struct klp_module_reloc {
-       void *sym;
+       uint32_t sym;
        unsigned int sympos;
 } __packed;

gets me generating an output .ko file, but the readelf output doesn't
look right.

I'll add this to the patchset TODO list, but may not get to it for a
while -- is there any chance the above hack works or could you test a
local non-cross build?

Thanks,

-- 
Joe

Re: ppc64le STRICT_MODULE_RWX and livepatch apply_relocate_add() crashes

[Christophe Leroy]

Le 13/12/2021 à 18:26, Joe Lawrence a écrit :
> On 12/13/21 11:36 AM, Christophe Leroy wrote:
>>
>>
>> Le 13/12/2021 à 15:47, Joe Lawrence a écrit :
>>> On 12/13/21 2:42 AM, Christophe Leroy wrote:
>>>>
>>>> Hello Joe,
>>>>
>>>> I'm implementing LIVEPATCH on PPC32 and I wanted to test with
>>>> STRICT_MODULE_RWX enabled so I took your branch as suggested, but I'm
>>>> getting the following errors on build. What shall I do ?
>>>>
>>>>      CALL    scripts/checksyscalls.sh
>>>>      CALL    scripts/atomic/check-atomics.sh
>>>>      CHK     include/generated/compile.h
>>>>      KLP     lib/livepatch/test_klp_convert1.ko
>>>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>>>> beyond nr_entries
>>>>
>>>> klp-convert: Unable to load user-provided sympos
>>>> make[2]: *** [scripts/Makefile.modfinal:79:
>>>> lib/livepatch/test_klp_convert1.ko] Error 255
>>>>      KLP     lib/livepatch/test_klp_convert2.ko
>>>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>>>> beyond nr_entries
>>>>
>>>> klp-convert: Unable to load user-provided sympos
>>>> make[2]: *** [scripts/Makefile.modfinal:79:
>>>> lib/livepatch/test_klp_convert2.ko] Error 255
>>>>      KLP     lib/livepatch/test_klp_convert_sections.ko
>>>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>>>> beyond nr_entries
>>>>
>>>> klp-convert: Unable to load user-provided sympos
>>>> make[2]: *** [scripts/Makefile.modfinal:79:
>>>> lib/livepatch/test_klp_convert_sections.ko] Error 255
>>>> make[2]: Target '__modfinal' not remade because of errors.
>>>> make[1]: *** [scripts/Makefile.modpost:145: __modpost] Error 2
>>>> make: *** [Makefile:1770: modules] Error 2
>>>>
>>>
>>> Hi Christophe,
>>>
>>> Interesting failure mode.  That's klp-convert complaining that it found
>>> more relocations in a .klp.module_relocs.<objname> section than
>>> expected, i.e. nr_entries = sec->size / sizeof(struct klp_module_reloc).
>>>
>>> A few possibilities: the ELF sec->size was incorrectly set/read by
>>> build/libelf (I doubt that).  Or maybe the layout/size of struct
>>> klp_module_reloc is not consistent between kernel and userspace (I'm
>>> more suspicious of this).
>>>
>>> Can you post a copy of the build's symbols.klp and
>>> lib/livepatch/test_klp_convert1.tmp.ko somewhere?  I should be able to
>>> start debug with those files.
>>>
>>
>> I sent you both files off list.
>>
>> It looks like klp-convert doesn't use the correct size. It finds a
>> struct of size 12 hence 3 entries for a section of size 40.
>>
>> On PPC32 the struct has size 8 (void * is 4 and int is 4).
>>
>> But I'm cross-building from x86_64 where the struct is 8 + 4 = 12.
>>
>> Can it be the reason ?
>>
> 
> I'm pretty sure that is it.  I haven't had much runtime with klp-convert
> and cross-building (I've only found one big/little endian bug with
> x86_64->s390x) and was going to ask you how you were testing :)
> 
> Do you know if there are other kernel build tools that deal with similar
> situations?  This seems like a tricky job for the userspace build tool
> to determine non-native target struct layout.
> 
> In the meantime, hacking in:
> 
>   struct klp_module_reloc {
> -       void *sym;
> +       uint32_t sym;
>          unsigned int sympos;
>   } __packed;
> 
> gets me generating an output .ko file, but the readelf output doesn't
> look right.
> 
> I'll add this to the patchset TODO list, but may not get to it for a
> while -- is there any chance the above hack works or could you test a
> local non-cross build?
> 

No I have no way to do a non-cross build. My target is an embedded board 
with slow CPU and little memory.

I tested with your hack, I get:

root@vgoip:~# insmod /lib/modules/test_klp_convert1.ko
insmod: can't insert '/lib/modules/test_klp_convert1.ko': unknown symbol 
in module, or unknown parameter
root@vgoip:~# insmod /lib/modules/test_klp_livepatch.ko
insmod: can't insert '/lib/modules/test_klp_livepatch.ko': unknown 
symbol in module, or unknown parameter


I agree with you readelf shows something went wrong with relocations.

Christophe

Re: ppc64le STRICT_MODULE_RWX and livepatch apply_relocate_add() crashes

[Joe Lawrence]

On 12/14/21 7:44 AM, Christophe Leroy wrote:
> 
> 
> Le 13/12/2021 à 18:26, Joe Lawrence a écrit :
>> On 12/13/21 11:36 AM, Christophe Leroy wrote:
>>>
>>>
>>> Le 13/12/2021 à 15:47, Joe Lawrence a écrit :
>>>> On 12/13/21 2:42 AM, Christophe Leroy wrote:
>>>>>
>>>>> Hello Joe,
>>>>>
>>>>> I'm implementing LIVEPATCH on PPC32 and I wanted to test with
>>>>> STRICT_MODULE_RWX enabled so I took your branch as suggested, but I'm
>>>>> getting the following errors on build. What shall I do ?
>>>>>
>>>>>      CALL    scripts/checksyscalls.sh
>>>>>      CALL    scripts/atomic/check-atomics.sh
>>>>>      CHK     include/generated/compile.h
>>>>>      KLP     lib/livepatch/test_klp_convert1.ko
>>>>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>>>>> beyond nr_entries
>>>>>
>>>>> klp-convert: Unable to load user-provided sympos
>>>>> make[2]: *** [scripts/Makefile.modfinal:79:
>>>>> lib/livepatch/test_klp_convert1.ko] Error 255
>>>>>      KLP     lib/livepatch/test_klp_convert2.ko
>>>>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>>>>> beyond nr_entries
>>>>>
>>>>> klp-convert: Unable to load user-provided sympos
>>>>> make[2]: *** [scripts/Makefile.modfinal:79:
>>>>> lib/livepatch/test_klp_convert2.ko] Error 255
>>>>>      KLP     lib/livepatch/test_klp_convert_sections.ko
>>>>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>>>>> beyond nr_entries
>>>>>
>>>>> klp-convert: Unable to load user-provided sympos
>>>>> make[2]: *** [scripts/Makefile.modfinal:79:
>>>>> lib/livepatch/test_klp_convert_sections.ko] Error 255
>>>>> make[2]: Target '__modfinal' not remade because of errors.
>>>>> make[1]: *** [scripts/Makefile.modpost:145: __modpost] Error 2
>>>>> make: *** [Makefile:1770: modules] Error 2
>>>>>
>>>>
>>>> Hi Christophe,
>>>>
>>>> Interesting failure mode.  That's klp-convert complaining that it found
>>>> more relocations in a .klp.module_relocs.<objname> section than
>>>> expected, i.e. nr_entries = sec->size / sizeof(struct klp_module_reloc).
>>>>
>>>> A few possibilities: the ELF sec->size was incorrectly set/read by
>>>> build/libelf (I doubt that).  Or maybe the layout/size of struct
>>>> klp_module_reloc is not consistent between kernel and userspace (I'm
>>>> more suspicious of this).
>>>>
>>>> Can you post a copy of the build's symbols.klp and
>>>> lib/livepatch/test_klp_convert1.tmp.ko somewhere?  I should be able to
>>>> start debug with those files.
>>>>
>>>
>>> I sent you both files off list.
>>>
>>> It looks like klp-convert doesn't use the correct size. It finds a
>>> struct of size 12 hence 3 entries for a section of size 40.
>>>
>>> On PPC32 the struct has size 8 (void * is 4 and int is 4).
>>>
>>> But I'm cross-building from x86_64 where the struct is 8 + 4 = 12.
>>>
>>> Can it be the reason ?
>>>
>>
>> I'm pretty sure that is it.  I haven't had much runtime with klp-convert
>> and cross-building (I've only found one big/little endian bug with
>> x86_64->s390x) and was going to ask you how you were testing :)
>>
>> Do you know if there are other kernel build tools that deal with similar
>> situations?  This seems like a tricky job for the userspace build tool
>> to determine non-native target struct layout.
>>
>> In the meantime, hacking in:
>>
>>   struct klp_module_reloc {
>> -       void *sym;
>> +       uint32_t sym;
>>          unsigned int sympos;
>>   } __packed;
>>
>> gets me generating an output .ko file, but the readelf output doesn't
>> look right.
>>
>> I'll add this to the patchset TODO list, but may not get to it for a
>> while -- is there any chance the above hack works or could you test a
>> local non-cross build?
>>
> 
> No I have no way to do a non-cross build. My target is an embedded board 
> with slow CPU and little memory.
> 
> I tested with your hack, I get:
> 
> root@vgoip:~# insmod /lib/modules/test_klp_convert1.ko
> insmod: can't insert '/lib/modules/test_klp_convert1.ko': unknown symbol 
> in module, or unknown parameter
> root@vgoip:~# insmod /lib/modules/test_klp_livepatch.ko
> insmod: can't insert '/lib/modules/test_klp_livepatch.ko': unknown 
> symbol in module, or unknown parameter
> 
> 
> I agree with you readelf shows something went wrong with relocations.
> 

Thanks for trying that.  Can you point me to the cross-compiler suite
that you are using for build and readelf?  Kernel .config would be handy
too and I can try to reproduce locally for debugging.

Thanks,

-- 
Joe

Re: ppc64le STRICT_MODULE_RWX and livepatch apply_relocate_add() crashes

[Christophe Leroy]

[-- Attachment #1: Type: text/plain, Size: 5223 bytes --]



Le 14/12/2021 à 14:00, Joe Lawrence a écrit :
> On 12/14/21 7:44 AM, Christophe Leroy wrote:
>>
>>
>> Le 13/12/2021 à 18:26, Joe Lawrence a écrit :
>>> On 12/13/21 11:36 AM, Christophe Leroy wrote:
>>>>
>>>>
>>>> Le 13/12/2021 à 15:47, Joe Lawrence a écrit :
>>>>> On 12/13/21 2:42 AM, Christophe Leroy wrote:
>>>>>>
>>>>>> Hello Joe,
>>>>>>
>>>>>> I'm implementing LIVEPATCH on PPC32 and I wanted to test with
>>>>>> STRICT_MODULE_RWX enabled so I took your branch as suggested, but I'm
>>>>>> getting the following errors on build. What shall I do ?
>>>>>>
>>>>>>       CALL    scripts/checksyscalls.sh
>>>>>>       CALL    scripts/atomic/check-atomics.sh
>>>>>>       CHK     include/generated/compile.h
>>>>>>       KLP     lib/livepatch/test_klp_convert1.ko
>>>>>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>>>>>> beyond nr_entries
>>>>>>
>>>>>> klp-convert: Unable to load user-provided sympos
>>>>>> make[2]: *** [scripts/Makefile.modfinal:79:
>>>>>> lib/livepatch/test_klp_convert1.ko] Error 255
>>>>>>       KLP     lib/livepatch/test_klp_convert2.ko
>>>>>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>>>>>> beyond nr_entries
>>>>>>
>>>>>> klp-convert: Unable to load user-provided sympos
>>>>>> make[2]: *** [scripts/Makefile.modfinal:79:
>>>>>> lib/livepatch/test_klp_convert2.ko] Error 255
>>>>>>       KLP     lib/livepatch/test_klp_convert_sections.ko
>>>>>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>>>>>> beyond nr_entries
>>>>>>
>>>>>> klp-convert: Unable to load user-provided sympos
>>>>>> make[2]: *** [scripts/Makefile.modfinal:79:
>>>>>> lib/livepatch/test_klp_convert_sections.ko] Error 255
>>>>>> make[2]: Target '__modfinal' not remade because of errors.
>>>>>> make[1]: *** [scripts/Makefile.modpost:145: __modpost] Error 2
>>>>>> make: *** [Makefile:1770: modules] Error 2
>>>>>>
>>>>>
>>>>> Hi Christophe,
>>>>>
>>>>> Interesting failure mode.  That's klp-convert complaining that it found
>>>>> more relocations in a .klp.module_relocs.<objname> section than
>>>>> expected, i.e. nr_entries = sec->size / sizeof(struct klp_module_reloc).
>>>>>
>>>>> A few possibilities: the ELF sec->size was incorrectly set/read by
>>>>> build/libelf (I doubt that).  Or maybe the layout/size of struct
>>>>> klp_module_reloc is not consistent between kernel and userspace (I'm
>>>>> more suspicious of this).
>>>>>
>>>>> Can you post a copy of the build's symbols.klp and
>>>>> lib/livepatch/test_klp_convert1.tmp.ko somewhere?  I should be able to
>>>>> start debug with those files.
>>>>>
>>>>
>>>> I sent you both files off list.
>>>>
>>>> It looks like klp-convert doesn't use the correct size. It finds a
>>>> struct of size 12 hence 3 entries for a section of size 40.
>>>>
>>>> On PPC32 the struct has size 8 (void * is 4 and int is 4).
>>>>
>>>> But I'm cross-building from x86_64 where the struct is 8 + 4 = 12.
>>>>
>>>> Can it be the reason ?
>>>>
>>>
>>> I'm pretty sure that is it.  I haven't had much runtime with klp-convert
>>> and cross-building (I've only found one big/little endian bug with
>>> x86_64->s390x) and was going to ask you how you were testing :)
>>>
>>> Do you know if there are other kernel build tools that deal with similar
>>> situations?  This seems like a tricky job for the userspace build tool
>>> to determine non-native target struct layout.
>>>
>>> In the meantime, hacking in:
>>>
>>>    struct klp_module_reloc {
>>> -       void *sym;
>>> +       uint32_t sym;
>>>           unsigned int sympos;
>>>    } __packed;
>>>
>>> gets me generating an output .ko file, but the readelf output doesn't
>>> look right.
>>>
>>> I'll add this to the patchset TODO list, but may not get to it for a
>>> while -- is there any chance the above hack works or could you test a
>>> local non-cross build?
>>>
>>
>> No I have no way to do a non-cross build. My target is an embedded board
>> with slow CPU and little memory.
>>
>> I tested with your hack, I get:
>>
>> root@vgoip:~# insmod /lib/modules/test_klp_convert1.ko
>> insmod: can't insert '/lib/modules/test_klp_convert1.ko': unknown symbol
>> in module, or unknown parameter
>> root@vgoip:~# insmod /lib/modules/test_klp_livepatch.ko
>> insmod: can't insert '/lib/modules/test_klp_livepatch.ko': unknown
>> symbol in module, or unknown parameter
>>
>>
>> I agree with you readelf shows something went wrong with relocations.
>>
> 
> Thanks for trying that.  Can you point me to the cross-compiler suite
> that you are using for build and readelf?  Kernel .config would be handy
> too and I can try to reproduce locally for debugging.
> 

Livepatch on PPC32 is WIP, branch 'livepatch' on 
https://github.com/chleroy/linux.git.

You can use branch 'joe' on https://github.com/chleroy/linux.git which 
is a merge of your branch klp-convert-v5-expanded-v5.16-rebase1 with my 
livepatch branch.

Crosstool: 
https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/11.1.0/x86_64-gcc-11.1.0-nolibc-powerpc64-linux.tar.xz

defconfig is attached

Thanks for your help
Christophe

[-- Attachment #2: defconfig --]
[-- Type: text/plain, Size: 1974 bytes --]

# CONFIG_SWAP is not set
CONFIG_SYSVIPC=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_BPF_JIT=y
CONFIG_VIRT_CPU_ACCOUNTING_NATIVE=y
CONFIG_LOG_BUF_SHIFT=14
CONFIG_EXPERT=y
# CONFIG_ELF_CORE is not set
# CONFIG_BASE_FULL is not set
# CONFIG_FUTEX is not set
CONFIG_KALLSYMS_ALL=y
CONFIG_PERF_EVENTS=y
# CONFIG_VM_EVENT_COUNTERS is not set
CONFIG_PPC_8xx=y
CONFIG_8xx_GPIO=y
CONFIG_SMC_UCODE_PATCH=y
CONFIG_PIN_TLB=y
CONFIG_GEN_RTC=y
CONFIG_HZ_100=y
CONFIG_MATH_EMULATION=y
CONFIG_PPC_16K_PAGES=y
CONFIG_ADVANCED_OPTIONS=y
CONFIG_LIVEPATCH=y
# CONFIG_SECCOMP is not set
CONFIG_MODULES=y
CONFIG_PARTITION_ADVANCED=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_PNP=y
CONFIG_SYN_COOKIES=y
# CONFIG_FW_LOADER is not set
CONFIG_MTD=y
CONFIG_MTD_BLOCK=y
CONFIG_MTD_JEDECPROBE=y
CONFIG_MTD_CFI_ADV_OPTIONS=y
CONFIG_MTD_CFI_GEOMETRY=y
# CONFIG_MTD_MAP_BANK_WIDTH_1 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_2 is not set
# CONFIG_MTD_CFI_I1 is not set
# CONFIG_MTD_CFI_I2 is not set
CONFIG_MTD_CFI_I4=y
CONFIG_MTD_CFI_AMDSTD=y
CONFIG_MTD_PHYSMAP=y
CONFIG_MTD_PHYSMAP_OF=y
# CONFIG_BLK_DEV is not set
CONFIG_NETDEVICES=y
CONFIG_FS_ENET=y
# CONFIG_FS_ENET_HAS_SCC is not set
CONFIG_DAVICOM_PHY=y
# CONFIG_INPUT is not set
# CONFIG_SERIO is not set
# CONFIG_VT is not set
# CONFIG_LEGACY_PTYS is not set
CONFIG_SERIAL_CPM=y
CONFIG_SERIAL_CPM_CONSOLE=y
CONFIG_SPI=y
CONFIG_SPI_FSL_SPI=y
# CONFIG_HWMON is not set
CONFIG_WATCHDOG=y
CONFIG_8xxx_WDT=y
# CONFIG_USB_SUPPORT is not set
# CONFIG_DNOTIFY is not set
CONFIG_TMPFS=y
CONFIG_CRAMFS=y
CONFIG_NFS_FS=y
CONFIG_ROOT_NFS=y
CONFIG_CRYPTO=y
CONFIG_CRYPTO_DEV_TALITOS=y
CONFIG_CRC32_SLICEBY4=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DEBUG_INFO=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_VM_PGTABLE=y
CONFIG_DETECT_HUNG_TASK=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FTRACE_STARTUP_TEST=y
# CONFIG_EVENT_TRACE_STARTUP_TEST is not set
CONFIG_BDI_SWITCH=y
CONFIG_PPC_EARLY_DEBUG=y
CONFIG_TEST_LIVEPATCH=m