Re: [PATCH] x86/livepatch: Fix crash with !CONFIG_DEBUG_SET_MODULE_RONX

[Miroslav Benes <mbenes@suse.cz>]

On Mon, 2 Nov 2015, Josh Poimboeuf wrote:

> When loading a patch module on a kernel with
> !CONFIG_DEBUG_SET_MODULE_RONX, the following crash occurs:
> 
>   [  205.988776] livepatch: enabling patch 'kpatch_meminfo_string'
>   [  205.989829] BUG: unable to handle kernel paging request at ffffffffa08d2fc0
>   [  205.989863] IP: [<ffffffff8154fecb>] do_init_module+0x8c/0x1ba
>   [  205.989888] PGD 1a10067 PUD 1a11063 PMD 7bcde067 PTE 3740e161
>   [  205.989915] Oops: 0003 [#1] SMP
>   [  205.990187] CPU: 2 PID: 14570 Comm: insmod Tainted: G           O  K 4.1.12
>   [  205.990214] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.1-20150318_183358- 04/01/2014
>   [  205.990249] task: ffff8800374aaa90 ti: ffff8800794b8000 task.ti: ffff8800794b8000
>   [  205.990276] RIP: 0010:[<ffffffff8154fecb>]  [<ffffffff8154fecb>] do_init_module+0x8c/0x1ba
>   [  205.990307] RSP: 0018:ffff8800794bbd58  EFLAGS: 00010246
>   [  205.990327] RAX: 0000000000000000 RBX: ffffffffa08d2fc0 RCX: 0000000000000000
>   [  205.990356] RDX: 01ffff8000000080 RSI: 0000000000000000 RDI: ffffffff81a54b40
>   [  205.990382] RBP: ffff88007b4c4d80 R08: 0000000000000007 R09: 0000000000000000
>   [  205.990408] R10: 0000000000000008 R11: ffffea0001f18840 R12: 0000000000000000
>   [  205.990433] R13: 0000000000000001 R14: ffffffffa08d2fc0 R15: ffff88007bd0bc40
>   [  205.990459] FS:  00007f1128fbc700(0000) GS:ffff88007fc80000(0000) knlGS:0000000000000000
>   [  205.990488] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>   [  205.990509] CR2: ffffffffa08d2fc0 CR3: 000000002606e000 CR4: 00000000001406e0
>   [  205.990536] Stack:
>   [  205.990545]  ffff8800794bbec8 0000000000000001 ffffffffa08d3010 ffffffff810ecea9
>   [  205.990576]  ffffffff810e8e40 000000000005f360 ffff88007bd0bc50 ffffffffa08d3240
>   [  205.990608]  ffffffffa08d52c0 ffffffffa08d3210 ffff8800794bbed8 ffff8800794bbf1c
>   [  205.990639] Call Trace:
>   [  205.990651]  [<ffffffff810ecea9>] ? load_module+0x1e59/0x23a0
>   [  205.990672]  [<ffffffff810e8e40>] ? store_uevent+0x40/0x40
>   [  205.990693]  [<ffffffff810e99b5>] ? copy_module_from_fd.isra.49+0xb5/0x140
>   [  205.990718]  [<ffffffff810ed5bd>] ? SyS_finit_module+0x7d/0xa0
>   [  205.990741]  [<ffffffff81556832>] ? system_call_fastpath+0x16/0x75
>   [  205.990763] Code: f9 00 00 00 74 23 49 c7 c0 92 e1 60 81 48 8d 53 18 89 c1 4c 89 c6 48 c7 c7 f0 85 7d 81 31 c0 e8 71 fa ff ff e8 58 0e 00 00 31 f6 <c7> 03 00 00 00 00 48 89 da 48 c7 c7 20 c7 a5 81 e8 d0 ec b3 ff
>   [  205.990916] RIP  [<ffffffff8154fecb>] do_init_module+0x8c/0x1ba
>   [  205.990940]  RSP <ffff8800794bbd58>
>   [  205.990953] CR2: ffffffffa08d2fc0
> 
> With !CONFIG_DEBUG_SET_MODULE_RONX, module text and rodata pages are
> writable, and the debug_align() macro allows the module struct to share
> a page with executable text.  When klp_write_module_reloc() calls
> set_memory_ro() on the page, it effectively turns the module struct into
> a read-only structure, resulting in a page fault when load_module() does
> "mod->state = MODULE_STATE_LIVE".
> 
> Reported-by: Cyril B. <cbay@alwaysdata.com>
> Tested-by: Cyril B. <cbay@alwaysdata.com>
> Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
> ---

Oh, this reminds me I had to solve this issue in kgraft while playing with 
relocations. I wanted to fix it in klp as well and forgot about it so it 
is great to see it done.

Anyway...

>  arch/x86/kernel/livepatch.c | 9 +++++----
>  1 file changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/x86/kernel/livepatch.c b/arch/x86/kernel/livepatch.c
> index ff3c3101d..d1d35cc 100644
> --- a/arch/x86/kernel/livepatch.c
> +++ b/arch/x86/kernel/livepatch.c
> @@ -42,7 +42,6 @@ int klp_write_module_reloc(struct module *mod, unsigned long type,
>  	bool readonly;
>  	unsigned long val;
>  	unsigned long core = (unsigned long)mod->module_core;
> -	unsigned long core_ro_size = mod->core_ro_size;
>  	unsigned long core_size = mod->core_size;
>  
>  	switch (type) {
> @@ -70,10 +69,12 @@ int klp_write_module_reloc(struct module *mod, unsigned long type,
>  		/* loc does not point to any symbol inside the module */
>  		return -EINVAL;
>  
> -	if (loc < core + core_ro_size)
> +	readonly = false;
> +
> +#ifdef CONFIG_DEBUG_SET_MODULE_RONX
> +	if (loc < core + mod->core_ro_size)
>  		readonly = true;
> -	else
> -		readonly = false;
> +#endif
>  

Wouldn't it be better to use static inline function defined somewhere in 
our header file which would do the job for CONFIG_DEBUG_SET_MODULE_RONX 
set and nothing for the opposite? I think it is better than to have ifdef 
here. Probably just a matter of taste.

Secondly, why do we call set_memory_rw/ro here for each relocation? 
Wouldn't it be possible to do it once for all the relevant pages of the 
patched module? I mean we could call set_memory_rw in 
klp_write_object_relocations from kernel/livepatch/core.c just before the 
for loop and set_memory_ro on the way out. With the static inline function 
it could look like this

+#ifdef CONFIG_DEBUG_SET_MODULE_RONX
+static inline void set_module_text(void *start, void *end, int (*set)(unsigned long start,
+       int num_pages))
+{
+       unsigned long begin_pfn = PFN_DOWN((unsigned long)start);
+       unsigned long end_pfn = PFN_DOWN((unsigned long)end);
+
+       if (end_pfn > begin_pfn)
+               set(begin_pfn << PAGE_SHIFT, end_pfn - begin_pfn);
+}
+#else
+static inline void set_module_text(void *start, void *end, int (*set)(unsigned long start,
+       int num_pages)) { }
+#endif

somewhere in include/linux/livepatch.h. And in 
klp_write_object_relocations() there could be 

+       set_module_text(mod->module_core, mod->module_core +
+               mod->core_text_size, set_memory_rw);

just before the for loop which goes through all the relocations

and 

+       set_module_text(mod->module_core, mod->module_core +
+               mod->core_text_size, set_memory_ro);

on the way out. 

We would also get rid of readonly variable.

At least this is how I solved it. Take it as an idea. I don't know if it 
is even better and I certainly do not insist on it. Your approach is ok.

Thanks,
Miroslav