* [RFC PATCH v2 1/3] livepatch: Clear relocation targets on a module removal
2019-09-05 12:45 [RFC PATCH v2 0/3] livepatch: Clear relocation targets on a module removal Miroslav Benes
@ 2019-09-05 12:45 ` Miroslav Benes
2019-10-02 13:22 ` Petr Mladek
2019-10-02 18:18 ` Josh Poimboeuf
2019-09-05 12:45 ` [RFC PATCH v2 2/3] livepatch: Unify functions for writing and clearing object relocations Miroslav Benes
` (2 subsequent siblings)
3 siblings, 2 replies; 12+ messages in thread
From: Miroslav Benes @ 2019-09-05 12:45 UTC (permalink / raw)
To: jikos, jpoimboe, pmladek
Cc: joe.lawrence, nstange, live-patching, linux-kernel, Miroslav Benes
Josh reported a bug:
When the object to be patched is a module, and that module is
rmmod'ed and reloaded, it fails to load with:
module: x86/modules: Skipping invalid relocation target, existing value is nonzero for type 2, loc 00000000ba0302e9, val ffffffffa03e293c
livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'
The livepatch module has a relocation which references a symbol
in the _previous_ loading of nfsd. When apply_relocate_add()
tries to replace the old relocation with a new one, it sees that
the previous one is nonzero and it errors out.
On ppc64le, we have a similar issue:
module_64: livepatch_nfsd: Expected nop after call, got e8410018 at e_show+0x60/0x548 [livepatch_nfsd]
livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'
He also proposed three different solutions. We could remove the error
check in apply_relocate_add() introduced by commit eda9cec4c9a1
("x86/module: Detect and skip invalid relocations"). However the check
is useful for detecting corrupted modules.
We could also deny the patched modules to be removed. If it proved to be
a major drawback for users, we could still implement a different
approach. The solution would also complicate the existing code a lot.
We thus decided to reverse the relocation patching (clear all relocation
targets on x86_64, or return back nops on powerpc). The solution is not
universal and is too much arch-specific, but it may prove to be simpler
in the end.
Reported-by: Josh Poimboeuf <jpoimboe@redhat.com>
Signed-off-by: Miroslav Benes <mbenes@suse.cz>
---
arch/powerpc/kernel/module_64.c | 45 +++++++++++++++++++++++++++++++++
arch/s390/kernel/module.c | 8 ++++++
arch/x86/kernel/module.c | 43 +++++++++++++++++++++++++++++++
include/linux/moduleloader.h | 7 +++++
kernel/livepatch/core.c | 45 +++++++++++++++++++++++++++++++++
5 files changed, 148 insertions(+)
diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c
index a93b10c48000..e461d456e447 100644
--- a/arch/powerpc/kernel/module_64.c
+++ b/arch/powerpc/kernel/module_64.c
@@ -741,6 +741,51 @@ int apply_relocate_add(Elf64_Shdr *sechdrs,
return 0;
}
+#ifdef CONFIG_LIVEPATCH
+void clear_relocate_add(Elf64_Shdr *sechdrs,
+ const char *strtab,
+ unsigned int symindex,
+ unsigned int relsec,
+ struct module *me)
+{
+ unsigned int i;
+ Elf64_Rela *rela = (void *)sechdrs[relsec].sh_addr;
+ Elf64_Sym *sym;
+ unsigned long *location;
+ const char *symname;
+ u32 *instruction;
+
+ pr_debug("Applying ADD relocate section %u to %u\n", relsec,
+ sechdrs[relsec].sh_info);
+
+ for (i = 0; i < sechdrs[relsec].sh_size / sizeof(*rela); i++) {
+ location = (void *)sechdrs[sechdrs[relsec].sh_info].sh_addr
+ + rela[i].r_offset;
+ sym = (Elf64_Sym *)sechdrs[symindex].sh_addr
+ + ELF64_R_SYM(rela[i].r_info);
+ symname = me->core_kallsyms.strtab
+ + sym->st_name;
+
+ if (ELF64_R_TYPE(rela[i].r_info) != R_PPC_REL24)
+ continue;
+
+ if (sym->st_shndx != SHN_UNDEF &&
+ sym->st_shndx != SHN_LIVEPATCH)
+ continue;
+
+ instruction = (u32 *)location;
+ if (is_mprofile_mcount_callsite(symname, instruction))
+ continue;
+
+ if (!instr_is_relative_link_branch(*instruction))
+ continue;
+
+ instruction += 1;
+ *instruction = PPC_INST_NOP;
+ }
+}
+#endif
+
#ifdef CONFIG_DYNAMIC_FTRACE
#ifdef CONFIG_MPROFILE_KERNEL
diff --git a/arch/s390/kernel/module.c b/arch/s390/kernel/module.c
index 31889db609e9..56867d052010 100644
--- a/arch/s390/kernel/module.c
+++ b/arch/s390/kernel/module.c
@@ -437,6 +437,14 @@ int apply_relocate_add(Elf_Shdr *sechdrs, const char *strtab,
return 0;
}
+#ifdef CONFIG_LIVEPATCH
+void clear_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
+ unsigned int symindex, unsigned int relsec,
+ struct module *me)
+{
+}
+#endif
+
int module_finalize(const Elf_Ehdr *hdr,
const Elf_Shdr *sechdrs,
struct module *me)
diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c
index d5c72cb877b3..b07d71f125e6 100644
--- a/arch/x86/kernel/module.c
+++ b/arch/x86/kernel/module.c
@@ -215,6 +215,49 @@ int apply_relocate_add(Elf64_Shdr *sechdrs,
me->name);
return -ENOEXEC;
}
+
+#ifdef CONFIG_LIVEPATCH
+void clear_relocate_add(Elf64_Shdr *sechdrs,
+ const char *strtab,
+ unsigned int symindex,
+ unsigned int relsec,
+ struct module *me)
+{
+ unsigned int i;
+ Elf64_Rela *rel = (void *)sechdrs[relsec].sh_addr;
+ void *loc;
+
+ DEBUGP("Clearing relocate section %u to %u\n",
+ relsec, sechdrs[relsec].sh_info);
+ for (i = 0; i < sechdrs[relsec].sh_size / sizeof(*rel); i++) {
+ loc = (void *)sechdrs[sechdrs[relsec].sh_info].sh_addr
+ + rel[i].r_offset;
+
+ switch (ELF64_R_TYPE(rel[i].r_info)) {
+ case R_X86_64_NONE:
+ break;
+ case R_X86_64_64:
+ *(u64 *)loc = 0;
+ break;
+ case R_X86_64_32:
+ *(u32 *)loc = 0;
+ break;
+ case R_X86_64_32S:
+ *(s32 *)loc = 0;
+ break;
+ case R_X86_64_PC32:
+ case R_X86_64_PLT32:
+ *(u32 *)loc = 0;
+ break;
+ case R_X86_64_PC64:
+ *(u64 *)loc = 0;
+ break;
+ default:
+ break;
+ }
+ }
+}
+#endif
#endif
int module_finalize(const Elf_Ehdr *hdr,
diff --git a/include/linux/moduleloader.h b/include/linux/moduleloader.h
index 31013c2effd3..f1e52692db5f 100644
--- a/include/linux/moduleloader.h
+++ b/include/linux/moduleloader.h
@@ -62,6 +62,13 @@ int apply_relocate_add(Elf_Shdr *sechdrs,
unsigned int symindex,
unsigned int relsec,
struct module *mod);
+#ifdef CONFIG_LIVEPATCH
+void clear_relocate_add(Elf64_Shdr *sechdrs,
+ const char *strtab,
+ unsigned int symindex,
+ unsigned int relsec,
+ struct module *me);
+#endif
#else
static inline int apply_relocate_add(Elf_Shdr *sechdrs,
const char *strtab,
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index ab4a4606d19b..f0b380d2a17a 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -295,6 +295,45 @@ static int klp_write_object_relocations(struct module *pmod,
return ret;
}
+static void klp_clear_object_relocations(struct module *pmod,
+ struct klp_object *obj)
+{
+ int i, cnt;
+ const char *objname, *secname;
+ char sec_objname[MODULE_NAME_LEN];
+ Elf_Shdr *sec;
+
+ objname = klp_is_module(obj) ? obj->name : "vmlinux";
+
+ /* For each klp relocation section */
+ for (i = 1; i < pmod->klp_info->hdr.e_shnum; i++) {
+ sec = pmod->klp_info->sechdrs + i;
+ secname = pmod->klp_info->secstrings + sec->sh_name;
+ if (!(sec->sh_flags & SHF_RELA_LIVEPATCH))
+ continue;
+
+ /*
+ * Format: .klp.rela.sec_objname.section_name
+ * See comment in klp_resolve_symbols() for an explanation
+ * of the selected field width value.
+ */
+ secname = pmod->klp_info->secstrings + sec->sh_name;
+ cnt = sscanf(secname, ".klp.rela.%55[^.]", sec_objname);
+ if (cnt != 1) {
+ pr_err("section %s has an incorrectly formatted name\n",
+ secname);
+ continue;
+ }
+
+ if (strcmp(objname, sec_objname))
+ continue;
+
+ clear_relocate_add(pmod->klp_info->sechdrs,
+ pmod->core_kallsyms.strtab,
+ pmod->klp_info->symndx, i, pmod);
+ }
+}
+
/*
* Sysfs Interface
*
@@ -1100,6 +1139,12 @@ static void klp_cleanup_module_patches_limited(struct module *mod,
klp_post_unpatch_callback(obj);
+ mutex_lock(&text_mutex);
+ module_disable_ro(patch->mod);
+ klp_clear_object_relocations(patch->mod, obj);
+ module_enable_ro(patch->mod, true);
+ mutex_unlock(&text_mutex);
+
klp_free_object_loaded(obj);
break;
}
--
2.23.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [RFC PATCH v2 1/3] livepatch: Clear relocation targets on a module removal
2019-09-05 12:45 ` [RFC PATCH v2 1/3] " Miroslav Benes
@ 2019-10-02 13:22 ` Petr Mladek
2019-10-03 8:55 ` Miroslav Benes
2019-10-02 18:18 ` Josh Poimboeuf
1 sibling, 1 reply; 12+ messages in thread
From: Petr Mladek @ 2019-10-02 13:22 UTC (permalink / raw)
To: Miroslav Benes
Cc: jikos, jpoimboe, joe.lawrence, nstange, live-patching, linux-kernel
On Thu 2019-09-05 14:45:12, Miroslav Benes wrote:
> We thus decided to reverse the relocation patching (clear all relocation
> targets on x86_64, or return back nops on powerpc). The solution is not
> universal and is too much arch-specific, but it may prove to be simpler
> in the end.
>
> diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c
> index a93b10c48000..e461d456e447 100644
> --- a/arch/powerpc/kernel/module_64.c
> +++ b/arch/powerpc/kernel/module_64.c
> @@ -741,6 +741,51 @@ int apply_relocate_add(Elf64_Shdr *sechdrs,
> return 0;
> }
>
> +#ifdef CONFIG_LIVEPATCH
> +void clear_relocate_add(Elf64_Shdr *sechdrs,
> + const char *strtab,
> + unsigned int symindex,
> + unsigned int relsec,
> + struct module *me)
> +{
> + unsigned int i;
> + Elf64_Rela *rela = (void *)sechdrs[relsec].sh_addr;
> + Elf64_Sym *sym;
> + unsigned long *location;
> + const char *symname;
> + u32 *instruction;
> +
> + pr_debug("Applying ADD relocate section %u to %u\n", relsec,
s/Applying/Clearing/
> + sechdrs[relsec].sh_info);
> +
> + for (i = 0; i < sechdrs[relsec].sh_size / sizeof(*rela); i++) {
> + location = (void *)sechdrs[sechdrs[relsec].sh_info].sh_addr
> + + rela[i].r_offset;
> + sym = (Elf64_Sym *)sechdrs[symindex].sh_addr
> + + ELF64_R_SYM(rela[i].r_info);
> + symname = me->core_kallsyms.strtab
> + + sym->st_name;
> +
> + if (ELF64_R_TYPE(rela[i].r_info) != R_PPC_REL24)
> + continue;
I expected that the code below would reverse the operations
in apply_relocate_add() for case R_PPC_REL24. But it is not
obvious for me.
It might be because I am not familiar with the code. Or would
it deserve some comments?
> +
> + if (sym->st_shndx != SHN_UNDEF &&
> + sym->st_shndx != SHN_LIVEPATCH)
> + continue;
> +
> + instruction = (u32 *)location;
> + if (is_mprofile_mcount_callsite(symname, instruction))
> + continue;
> +
> + if (!instr_is_relative_link_branch(*instruction))
> + continue;
> +
> + instruction += 1;
> + *instruction = PPC_INST_NOP;
> + }
> +}
> +#endif
> +
> #ifdef CONFIG_DYNAMIC_FTRACE
>
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index ab4a4606d19b..f0b380d2a17a 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -295,6 +295,45 @@ static int klp_write_object_relocations(struct module *pmod,
> return ret;
> }
>
> +static void klp_clear_object_relocations(struct module *pmod,
> + struct klp_object *obj)
> +{
> + int i, cnt;
> + const char *objname, *secname;
> + char sec_objname[MODULE_NAME_LEN];
> + Elf_Shdr *sec;
> +
> + objname = klp_is_module(obj) ? obj->name : "vmlinux";
> +
> + /* For each klp relocation section */
> + for (i = 1; i < pmod->klp_info->hdr.e_shnum; i++) {
> + sec = pmod->klp_info->sechdrs + i;
> + secname = pmod->klp_info->secstrings + sec->sh_name;
> + if (!(sec->sh_flags & SHF_RELA_LIVEPATCH))
> + continue;
> +
> + /*
> + * Format: .klp.rela.sec_objname.section_name
> + * See comment in klp_resolve_symbols() for an explanation
> + * of the selected field width value.
> + */
> + secname = pmod->klp_info->secstrings + sec->sh_name;
> + cnt = sscanf(secname, ".klp.rela.%55[^.]", sec_objname);
> + if (cnt != 1) {
> + pr_err("section %s has an incorrectly formatted name\n",
> + secname);
> + continue;
> + }
> +
> + if (strcmp(objname, sec_objname))
> + continue;
> +
It would make the review easier when the order of 1st and 2nd
patch was swaped. I mean that I would not need to check twice
that the two functions actually share the same code.
> + clear_relocate_add(pmod->klp_info->sechdrs,
> + pmod->core_kallsyms.strtab,
> + pmod->klp_info->symndx, i, pmod);
> + }
> +}
> +
> /*
> * Sysfs Interface
> *
I was not able to check correctness of the ppc and s390 parts.
Otherwise, it looks good to me.
Best Regards,
Petr
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [RFC PATCH v2 1/3] livepatch: Clear relocation targets on a module removal
2019-10-02 13:22 ` Petr Mladek
@ 2019-10-03 8:55 ` Miroslav Benes
0 siblings, 0 replies; 12+ messages in thread
From: Miroslav Benes @ 2019-10-03 8:55 UTC (permalink / raw)
To: Petr Mladek
Cc: jikos, jpoimboe, joe.lawrence, nstange, live-patching, linux-kernel
On Wed, 2 Oct 2019, Petr Mladek wrote:
> On Thu 2019-09-05 14:45:12, Miroslav Benes wrote:
> > We thus decided to reverse the relocation patching (clear all relocation
> > targets on x86_64, or return back nops on powerpc). The solution is not
> > universal and is too much arch-specific, but it may prove to be simpler
> > in the end.
> >
> > diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c
> > index a93b10c48000..e461d456e447 100644
> > --- a/arch/powerpc/kernel/module_64.c
> > +++ b/arch/powerpc/kernel/module_64.c
> > @@ -741,6 +741,51 @@ int apply_relocate_add(Elf64_Shdr *sechdrs,
> > return 0;
> > }
> >
> > +#ifdef CONFIG_LIVEPATCH
> > +void clear_relocate_add(Elf64_Shdr *sechdrs,
> > + const char *strtab,
> > + unsigned int symindex,
> > + unsigned int relsec,
> > + struct module *me)
> > +{
> > + unsigned int i;
> > + Elf64_Rela *rela = (void *)sechdrs[relsec].sh_addr;
> > + Elf64_Sym *sym;
> > + unsigned long *location;
> > + const char *symname;
> > + u32 *instruction;
> > +
> > + pr_debug("Applying ADD relocate section %u to %u\n", relsec,
>
> s/Applying/Clearing/
Ugh. Thanks for noticing.
> > + sechdrs[relsec].sh_info);
> > +
> > + for (i = 0; i < sechdrs[relsec].sh_size / sizeof(*rela); i++) {
> > + location = (void *)sechdrs[sechdrs[relsec].sh_info].sh_addr
> > + + rela[i].r_offset;
> > + sym = (Elf64_Sym *)sechdrs[symindex].sh_addr
> > + + ELF64_R_SYM(rela[i].r_info);
> > + symname = me->core_kallsyms.strtab
> > + + sym->st_name;
> > +
> > + if (ELF64_R_TYPE(rela[i].r_info) != R_PPC_REL24)
> > + continue;
>
> I expected that the code below would reverse the operations
> in apply_relocate_add() for case R_PPC_REL24. But it is not
> obvious for me.
It should, but it is not obvious. See restore_r2(). We only need to
replace PPC_INST_LD_TOC instruction with PPC_INST_NOP and that's it.
> It might be because I am not familiar with the code. Or would
> it deserve some comments?
Maybe.
> > +
> > + if (sym->st_shndx != SHN_UNDEF &&
> > + sym->st_shndx != SHN_LIVEPATCH)
> > + continue;
> > +
> > + instruction = (u32 *)location;
> > + if (is_mprofile_mcount_callsite(symname, instruction))
> > + continue;
> > +
> > + if (!instr_is_relative_link_branch(*instruction))
> > + continue;
> > +
> > + instruction += 1;
> > + *instruction = PPC_INST_NOP;
> > + }
> > +}
> > +#endif
> > +
> > #ifdef CONFIG_DYNAMIC_FTRACE
> >
> > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > index ab4a4606d19b..f0b380d2a17a 100644
> > --- a/kernel/livepatch/core.c
> > +++ b/kernel/livepatch/core.c
> > @@ -295,6 +295,45 @@ static int klp_write_object_relocations(struct module *pmod,
> > return ret;
> > }
> >
> > +static void klp_clear_object_relocations(struct module *pmod,
> > + struct klp_object *obj)
> > +{
> > + int i, cnt;
> > + const char *objname, *secname;
> > + char sec_objname[MODULE_NAME_LEN];
> > + Elf_Shdr *sec;
> > +
> > + objname = klp_is_module(obj) ? obj->name : "vmlinux";
> > +
> > + /* For each klp relocation section */
> > + for (i = 1; i < pmod->klp_info->hdr.e_shnum; i++) {
> > + sec = pmod->klp_info->sechdrs + i;
> > + secname = pmod->klp_info->secstrings + sec->sh_name;
> > + if (!(sec->sh_flags & SHF_RELA_LIVEPATCH))
> > + continue;
> > +
> > + /*
> > + * Format: .klp.rela.sec_objname.section_name
> > + * See comment in klp_resolve_symbols() for an explanation
> > + * of the selected field width value.
> > + */
> > + secname = pmod->klp_info->secstrings + sec->sh_name;
> > + cnt = sscanf(secname, ".klp.rela.%55[^.]", sec_objname);
> > + if (cnt != 1) {
> > + pr_err("section %s has an incorrectly formatted name\n",
> > + secname);
> > + continue;
> > + }
> > +
> > + if (strcmp(objname, sec_objname))
> > + continue;
> > +
>
> It would make the review easier when the order of 1st and 2nd
> patch was swaped. I mean that I would not need to check twice
> that the two functions actually share the same code.
Ok.
> > + clear_relocate_add(pmod->klp_info->sechdrs,
> > + pmod->core_kallsyms.strtab,
> > + pmod->klp_info->symndx, i, pmod);
> > + }
> > +}
> > +
> > /*
> > * Sysfs Interface
> > *
>
> I was not able to check correctness of the ppc and s390 parts.
> Otherwise, it looks good to me.
Thanks
Miroslav
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [RFC PATCH v2 1/3] livepatch: Clear relocation targets on a module removal
2019-09-05 12:45 ` [RFC PATCH v2 1/3] " Miroslav Benes
2019-10-02 13:22 ` Petr Mladek
@ 2019-10-02 18:18 ` Josh Poimboeuf
2019-10-03 9:17 ` Miroslav Benes
1 sibling, 1 reply; 12+ messages in thread
From: Josh Poimboeuf @ 2019-10-02 18:18 UTC (permalink / raw)
To: Miroslav Benes
Cc: jikos, pmladek, joe.lawrence, nstange, live-patching, linux-kernel
On Thu, Sep 05, 2019 at 02:45:12PM +0200, Miroslav Benes wrote:
> Josh reported a bug:
>
> When the object to be patched is a module, and that module is
> rmmod'ed and reloaded, it fails to load with:
>
> module: x86/modules: Skipping invalid relocation target, existing value is nonzero for type 2, loc 00000000ba0302e9, val ffffffffa03e293c
> livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
> livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'
>
> The livepatch module has a relocation which references a symbol
> in the _previous_ loading of nfsd. When apply_relocate_add()
> tries to replace the old relocation with a new one, it sees that
> the previous one is nonzero and it errors out.
>
> On ppc64le, we have a similar issue:
>
> module_64: livepatch_nfsd: Expected nop after call, got e8410018 at e_show+0x60/0x548 [livepatch_nfsd]
> livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
> livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'
>
> He also proposed three different solutions. We could remove the error
> check in apply_relocate_add() introduced by commit eda9cec4c9a1
> ("x86/module: Detect and skip invalid relocations"). However the check
> is useful for detecting corrupted modules.
>
> We could also deny the patched modules to be removed. If it proved to be
> a major drawback for users, we could still implement a different
> approach. The solution would also complicate the existing code a lot.
>
> We thus decided to reverse the relocation patching (clear all relocation
> targets on x86_64, or return back nops on powerpc). The solution is not
> universal and is too much arch-specific, but it may prove to be simpler
> in the end.
>
> Reported-by: Josh Poimboeuf <jpoimboe@redhat.com>
> Signed-off-by: Miroslav Benes <mbenes@suse.cz>
Since we decided to fix late module patching at LPC, the commit message
and clear_relocate_add() should both probably clarify that these
functions are hacks which are relatively temporary, until we fix the
root cause.
But this patch gives me a bad feeling :-/ Not that I have a better
idea.
Has anybody seen this problem in the real world? If not, maybe we'd be
better off just pretending the problem doesn't exist for now.
--
Josh
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [RFC PATCH v2 1/3] livepatch: Clear relocation targets on a module removal
2019-10-02 18:18 ` Josh Poimboeuf
@ 2019-10-03 9:17 ` Miroslav Benes
0 siblings, 0 replies; 12+ messages in thread
From: Miroslav Benes @ 2019-10-03 9:17 UTC (permalink / raw)
To: Josh Poimboeuf
Cc: jikos, pmladek, joe.lawrence, nstange, live-patching, linux-kernel
On Wed, 2 Oct 2019, Josh Poimboeuf wrote:
> On Thu, Sep 05, 2019 at 02:45:12PM +0200, Miroslav Benes wrote:
> > Josh reported a bug:
> >
> > When the object to be patched is a module, and that module is
> > rmmod'ed and reloaded, it fails to load with:
> >
> > module: x86/modules: Skipping invalid relocation target, existing value is nonzero for type 2, loc 00000000ba0302e9, val ffffffffa03e293c
> > livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
> > livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'
> >
> > The livepatch module has a relocation which references a symbol
> > in the _previous_ loading of nfsd. When apply_relocate_add()
> > tries to replace the old relocation with a new one, it sees that
> > the previous one is nonzero and it errors out.
> >
> > On ppc64le, we have a similar issue:
> >
> > module_64: livepatch_nfsd: Expected nop after call, got e8410018 at e_show+0x60/0x548 [livepatch_nfsd]
> > livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
> > livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'
> >
> > He also proposed three different solutions. We could remove the error
> > check in apply_relocate_add() introduced by commit eda9cec4c9a1
> > ("x86/module: Detect and skip invalid relocations"). However the check
> > is useful for detecting corrupted modules.
> >
> > We could also deny the patched modules to be removed. If it proved to be
> > a major drawback for users, we could still implement a different
> > approach. The solution would also complicate the existing code a lot.
> >
> > We thus decided to reverse the relocation patching (clear all relocation
> > targets on x86_64, or return back nops on powerpc). The solution is not
> > universal and is too much arch-specific, but it may prove to be simpler
> > in the end.
> >
> > Reported-by: Josh Poimboeuf <jpoimboe@redhat.com>
> > Signed-off-by: Miroslav Benes <mbenes@suse.cz>
>
> Since we decided to fix late module patching at LPC, the commit message
> and clear_relocate_add() should both probably clarify that these
> functions are hacks which are relatively temporary, until we fix the
> root cause.
It was the plan, but thanks for pointing it out explicitly. I could
forget.
> But this patch gives me a bad feeling :-/ Not that I have a better
> idea.
I know what you are talking about.
> Has anybody seen this problem in the real world? If not, maybe we'd be
> better off just pretending the problem doesn't exist for now.
I don't think so. You reported the issue originally and I guess it
happened during the testing. Then there is a report from Huawei, but it
suggests testing environment too. Reloading modules seems artificial to
me.
So I agree, we can pretend the issue does not exist and wait for the real
solution.
Miroslav
^ permalink raw reply [flat|nested] 12+ messages in thread
* [RFC PATCH v2 2/3] livepatch: Unify functions for writing and clearing object relocations
2019-09-05 12:45 [RFC PATCH v2 0/3] livepatch: Clear relocation targets on a module removal Miroslav Benes
2019-09-05 12:45 ` [RFC PATCH v2 1/3] " Miroslav Benes
@ 2019-09-05 12:45 ` Miroslav Benes
2019-10-02 13:35 ` Petr Mladek
2019-09-05 12:45 ` [RFC PATCH v2 3/3] livepatch: Clean up klp_update_object_relocations() return paths Miroslav Benes
2019-10-01 12:30 ` [RFC PATCH v2 0/3] livepatch: Clear relocation targets on a module removal Miroslav Benes
3 siblings, 1 reply; 12+ messages in thread
From: Miroslav Benes @ 2019-09-05 12:45 UTC (permalink / raw)
To: jikos, jpoimboe, pmladek
Cc: joe.lawrence, nstange, live-patching, linux-kernel, Miroslav Benes
Functions klp_write_object_relocations() and
klp_clear_object_relocations() share a lot of code. Take the code out to
a common function and provide the specific actions through callbacks.
Signed-off-by: Miroslav Benes <mbenes@suse.cz>
---
kernel/livepatch/core.c | 84 +++++++++++++++++------------------------
1 file changed, 34 insertions(+), 50 deletions(-)
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index f0b380d2a17a..023c9333c276 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -245,8 +245,11 @@ static int klp_resolve_symbols(Elf_Shdr *relasec, struct module *pmod)
return 0;
}
-static int klp_write_object_relocations(struct module *pmod,
- struct klp_object *obj)
+typedef int (reloc_update_fn_t)(struct module *, unsigned int);
+
+static int klp_update_object_relocations(struct module *pmod,
+ struct klp_object *obj,
+ reloc_update_fn_t reloc_update_fn)
{
int i, cnt, ret = 0;
const char *objname, *secname;
@@ -281,13 +284,7 @@ static int klp_write_object_relocations(struct module *pmod,
if (strcmp(objname, sec_objname))
continue;
- ret = klp_resolve_symbols(sec, pmod);
- if (ret)
- break;
-
- ret = apply_relocate_add(pmod->klp_info->sechdrs,
- pmod->core_kallsyms.strtab,
- pmod->klp_info->symndx, i, pmod);
+ ret = reloc_update_fn(pmod, i);
if (ret)
break;
}
@@ -295,45 +292,6 @@ static int klp_write_object_relocations(struct module *pmod,
return ret;
}
-static void klp_clear_object_relocations(struct module *pmod,
- struct klp_object *obj)
-{
- int i, cnt;
- const char *objname, *secname;
- char sec_objname[MODULE_NAME_LEN];
- Elf_Shdr *sec;
-
- objname = klp_is_module(obj) ? obj->name : "vmlinux";
-
- /* For each klp relocation section */
- for (i = 1; i < pmod->klp_info->hdr.e_shnum; i++) {
- sec = pmod->klp_info->sechdrs + i;
- secname = pmod->klp_info->secstrings + sec->sh_name;
- if (!(sec->sh_flags & SHF_RELA_LIVEPATCH))
- continue;
-
- /*
- * Format: .klp.rela.sec_objname.section_name
- * See comment in klp_resolve_symbols() for an explanation
- * of the selected field width value.
- */
- secname = pmod->klp_info->secstrings + sec->sh_name;
- cnt = sscanf(secname, ".klp.rela.%55[^.]", sec_objname);
- if (cnt != 1) {
- pr_err("section %s has an incorrectly formatted name\n",
- secname);
- continue;
- }
-
- if (strcmp(objname, sec_objname))
- continue;
-
- clear_relocate_add(pmod->klp_info->sechdrs,
- pmod->core_kallsyms.strtab,
- pmod->klp_info->symndx, i, pmod);
- }
-}
-
/*
* Sysfs Interface
*
@@ -751,6 +709,21 @@ void __weak arch_klp_init_object_loaded(struct klp_patch *patch,
{
}
+static int klp_write_relocations(struct module *pmod,
+ unsigned int relsec)
+{
+ int ret;
+
+ ret = klp_resolve_symbols(pmod->klp_info->sechdrs + relsec, pmod);
+ if (ret)
+ return ret;
+
+ ret = apply_relocate_add(pmod->klp_info->sechdrs,
+ pmod->core_kallsyms.strtab,
+ pmod->klp_info->symndx, relsec, pmod);
+ return ret;
+}
+
/* parts of the initialization that is done only when the object is loaded */
static int klp_init_object_loaded(struct klp_patch *patch,
struct klp_object *obj)
@@ -761,7 +734,8 @@ static int klp_init_object_loaded(struct klp_patch *patch,
mutex_lock(&text_mutex);
module_disable_ro(patch->mod);
- ret = klp_write_object_relocations(patch->mod, obj);
+ ret = klp_update_object_relocations(patch->mod, obj,
+ klp_write_relocations);
if (ret) {
module_enable_ro(patch->mod, true);
mutex_unlock(&text_mutex);
@@ -1111,6 +1085,15 @@ void klp_discard_nops(struct klp_patch *new_patch)
klp_free_objects_dynamic(klp_transition_patch);
}
+static int klp_clear_relocations(struct module *pmod,
+ unsigned int relsec)
+{
+ clear_relocate_add(pmod->klp_info->sechdrs,
+ pmod->core_kallsyms.strtab,
+ pmod->klp_info->symndx, relsec, pmod);
+ return 0;
+}
+
/*
* Remove parts of patches that touch a given kernel module. The list of
* patches processed might be limited. When limit is NULL, all patches
@@ -1141,7 +1124,8 @@ static void klp_cleanup_module_patches_limited(struct module *mod,
mutex_lock(&text_mutex);
module_disable_ro(patch->mod);
- klp_clear_object_relocations(patch->mod, obj);
+ klp_update_object_relocations(patch->mod, obj,
+ klp_clear_relocations);
module_enable_ro(patch->mod, true);
mutex_unlock(&text_mutex);
--
2.23.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [RFC PATCH v2 3/3] livepatch: Clean up klp_update_object_relocations() return paths
2019-09-05 12:45 [RFC PATCH v2 0/3] livepatch: Clear relocation targets on a module removal Miroslav Benes
2019-09-05 12:45 ` [RFC PATCH v2 1/3] " Miroslav Benes
2019-09-05 12:45 ` [RFC PATCH v2 2/3] livepatch: Unify functions for writing and clearing object relocations Miroslav Benes
@ 2019-09-05 12:45 ` Miroslav Benes
2019-10-02 13:46 ` Petr Mladek
2019-10-01 12:30 ` [RFC PATCH v2 0/3] livepatch: Clear relocation targets on a module removal Miroslav Benes
3 siblings, 1 reply; 12+ messages in thread
From: Miroslav Benes @ 2019-09-05 12:45 UTC (permalink / raw)
To: jikos, jpoimboe, pmladek
Cc: joe.lawrence, nstange, live-patching, linux-kernel, Miroslav Benes
Signed-off-by: Miroslav Benes <mbenes@suse.cz>
---
kernel/livepatch/core.c | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 023c9333c276..73ddddd5add5 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -251,7 +251,7 @@ static int klp_update_object_relocations(struct module *pmod,
struct klp_object *obj,
reloc_update_fn_t reloc_update_fn)
{
- int i, cnt, ret = 0;
+ int i, cnt, ret;
const char *objname, *secname;
char sec_objname[MODULE_NAME_LEN];
Elf_Shdr *sec;
@@ -277,8 +277,7 @@ static int klp_update_object_relocations(struct module *pmod,
if (cnt != 1) {
pr_err("section %s has an incorrectly formatted name\n",
secname);
- ret = -EINVAL;
- break;
+ return -EINVAL;
}
if (strcmp(objname, sec_objname))
@@ -286,10 +285,10 @@ static int klp_update_object_relocations(struct module *pmod,
ret = reloc_update_fn(pmod, i);
if (ret)
- break;
+ return ret;
}
- return ret;
+ return 0;
}
/*
--
2.23.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [RFC PATCH v2 3/3] livepatch: Clean up klp_update_object_relocations() return paths
2019-09-05 12:45 ` [RFC PATCH v2 3/3] livepatch: Clean up klp_update_object_relocations() return paths Miroslav Benes
@ 2019-10-02 13:46 ` Petr Mladek
2019-10-03 9:08 ` Miroslav Benes
0 siblings, 1 reply; 12+ messages in thread
From: Petr Mladek @ 2019-10-02 13:46 UTC (permalink / raw)
To: Miroslav Benes
Cc: jikos, jpoimboe, joe.lawrence, nstange, live-patching, linux-kernel
On Thu 2019-09-05 14:45:14, Miroslav Benes wrote:
> Signed-off-by: Miroslav Benes <mbenes@suse.cz>
This might depend on personal preferences. What was the motivation
for this patch, please? Did it just follow some common
style in this source file?
To make it clear. I have no real preference. I just want to avoid
some back and forth changes of the code depending on who touches
it at the moment.
I would prefer to either remove this patch or explain the motivation
in the commit message. Beside that
Reviewed-by: Petr Mladek <pmladek@suse.com>
Best Regards,
Petr
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [RFC PATCH v2 3/3] livepatch: Clean up klp_update_object_relocations() return paths
2019-10-02 13:46 ` Petr Mladek
@ 2019-10-03 9:08 ` Miroslav Benes
0 siblings, 0 replies; 12+ messages in thread
From: Miroslav Benes @ 2019-10-03 9:08 UTC (permalink / raw)
To: Petr Mladek
Cc: jikos, jpoimboe, joe.lawrence, nstange, live-patching, linux-kernel
On Wed, 2 Oct 2019, Petr Mladek wrote:
> On Thu 2019-09-05 14:45:14, Miroslav Benes wrote:
> > Signed-off-by: Miroslav Benes <mbenes@suse.cz>
>
> This might depend on personal preferences.
True.
> What was the motivation
> for this patch, please? Did it just follow some common
> style in this source file?
We had it like this once, so it is only going back to the original code.
And yes, I think it is better.
Commit b56b36ee6751 ("livepatch: Cleanup module page permission changes")
changed it due to the error handling. Commit 255e732c61db ("livepatch: use
arch_klp_init_object_loaded() to finish arch-specific tasks") removed the
reason for the change but did not cleanup the rest.
> To make it clear. I have no real preference. I just want to avoid
> some back and forth changes of the code depending on who touches
> it at the moment.
I have no real preference either. I noticed something I did not like while
touching the code and that's it.
> I would prefer to either remove this patch or explain the motivation
> in the commit message. Beside that
>
> Reviewed-by: Petr Mladek <pmladek@suse.com>
Ok, thanks.
Miroslav
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [RFC PATCH v2 0/3] livepatch: Clear relocation targets on a module removal
2019-09-05 12:45 [RFC PATCH v2 0/3] livepatch: Clear relocation targets on a module removal Miroslav Benes
` (2 preceding siblings ...)
2019-09-05 12:45 ` [RFC PATCH v2 3/3] livepatch: Clean up klp_update_object_relocations() return paths Miroslav Benes
@ 2019-10-01 12:30 ` Miroslav Benes
3 siblings, 0 replies; 12+ messages in thread
From: Miroslav Benes @ 2019-10-01 12:30 UTC (permalink / raw)
To: jikos, jpoimboe, pmladek
Cc: joe.lawrence, nstange, live-patching, linux-kernel
On Thu, 5 Sep 2019, Miroslav Benes wrote:
> Updated version with Petr's feedback. It looks a bit different and
> better now (I would say). Not that it should be considered before we
> decide what to do with late module patching, but I finished it before
> the discussion started and someone could be interested.
>
> v1: http://lore.kernel.org/r/20190719122840.15353-1-mbenes@suse.cz
>
> Tested on x86_64, ppc64le and s390x. Cross-compiled on arm64 to verify
> that nothing is broken.
>
> [1] 20180602161151.apuhs2dygsexmcg2@treble
> [2] 1561019068-132672-1-git-send-email-cj.chengjian@huawei.com
> [3] 20180607092949.1706-1-mbenes@suse.cz
>
> Miroslav Benes (3):
> livepatch: Clear relocation targets on a module removal
> livepatch: Unify functions for writing and clearing object relocations
> livepatch: Clean up klp_update_object_relocations() return paths
>
> arch/powerpc/kernel/module_64.c | 45 +++++++++++++++++++++++++
> arch/s390/kernel/module.c | 8 +++++
> arch/x86/kernel/module.c | 43 ++++++++++++++++++++++++
> include/linux/moduleloader.h | 7 ++++
> kernel/livepatch/core.c | 58 ++++++++++++++++++++++++---------
> 5 files changed, 146 insertions(+), 15 deletions(-)
Ping.
If I remember correctly, we decided to have this as a temporary solution
before better late module patching is implemented. Feedback is welcome.
I'll then resend with arch maintainters CCed.
Thanks
Miroslav
^ permalink raw reply [flat|nested] 12+ messages in thread