[RFC PATCH v2 0/3] livepatch: Clear relocation targets on a module removal

[RFC PATCH v2 0/3] livepatch: Clear relocation targets on a module removal

[Miroslav Benes]

Updated version with Petr's feedback. It looks a bit different and
better now (I would say). Not that it should be considered before we
decide what to do with late module patching, but I finished it before
the discussion started and someone could be interested.

v1: http://lore.kernel.org/r/20190719122840.15353-1-mbenes@suse.cz

Tested on x86_64, ppc64le and s390x. Cross-compiled on arm64 to verify
that nothing is broken.

[1] 20180602161151.apuhs2dygsexmcg2@treble
[2] 1561019068-132672-1-git-send-email-cj.chengjian@huawei.com
[3] 20180607092949.1706-1-mbenes@suse.cz

Miroslav Benes (3):
  livepatch: Clear relocation targets on a module removal
  livepatch: Unify functions for writing and clearing object relocations
  livepatch: Clean up klp_update_object_relocations() return paths

 arch/powerpc/kernel/module_64.c | 45 +++++++++++++++++++++++++
 arch/s390/kernel/module.c       |  8 +++++
 arch/x86/kernel/module.c        | 43 ++++++++++++++++++++++++
 include/linux/moduleloader.h    |  7 ++++
 kernel/livepatch/core.c         | 58 ++++++++++++++++++++++++---------
 5 files changed, 146 insertions(+), 15 deletions(-)

-- 
2.23.0

[RFC PATCH v2 1/3] livepatch: Clear relocation targets on a module removal

[Miroslav Benes]

Josh reported a bug:

  When the object to be patched is a module, and that module is
  rmmod'ed and reloaded, it fails to load with:

  module: x86/modules: Skipping invalid relocation target, existing value is nonzero for type 2, loc 00000000ba0302e9, val ffffffffa03e293c
  livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
  livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'

  The livepatch module has a relocation which references a symbol
  in the _previous_ loading of nfsd. When apply_relocate_add()
  tries to replace the old relocation with a new one, it sees that
  the previous one is nonzero and it errors out.

  On ppc64le, we have a similar issue:

  module_64: livepatch_nfsd: Expected nop after call, got e8410018 at e_show+0x60/0x548 [livepatch_nfsd]
  livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
  livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'

He also proposed three different solutions. We could remove the error
check in apply_relocate_add() introduced by commit eda9cec4c9a1
("x86/module: Detect and skip invalid relocations"). However the check
is useful for detecting corrupted modules.

We could also deny the patched modules to be removed. If it proved to be
a major drawback for users, we could still implement a different
approach. The solution would also complicate the existing code a lot.

We thus decided to reverse the relocation patching (clear all relocation
targets on x86_64, or return back nops on powerpc). The solution is not
universal and is too much arch-specific, but it may prove to be simpler
in the end.

Reported-by: Josh Poimboeuf <jpoimboe@redhat.com>
Signed-off-by: Miroslav Benes <mbenes@suse.cz>
---
 arch/powerpc/kernel/module_64.c | 45 +++++++++++++++++++++++++++++++++
 arch/s390/kernel/module.c       |  8 ++++++
 arch/x86/kernel/module.c        | 43 +++++++++++++++++++++++++++++++
 include/linux/moduleloader.h    |  7 +++++
 kernel/livepatch/core.c         | 45 +++++++++++++++++++++++++++++++++
 5 files changed, 148 insertions(+)

diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c
index a93b10c48000..e461d456e447 100644
--- a/arch/powerpc/kernel/module_64.c
+++ b/arch/powerpc/kernel/module_64.c
@@ -741,6 +741,51 @@ int apply_relocate_add(Elf64_Shdr *sechdrs,
 	return 0;
 }
 
+#ifdef CONFIG_LIVEPATCH
+void clear_relocate_add(Elf64_Shdr *sechdrs,
+		       const char *strtab,
+		       unsigned int symindex,
+		       unsigned int relsec,
+		       struct module *me)
+{
+	unsigned int i;
+	Elf64_Rela *rela = (void *)sechdrs[relsec].sh_addr;
+	Elf64_Sym *sym;
+	unsigned long *location;
+	const char *symname;
+	u32 *instruction;
+
+	pr_debug("Applying ADD relocate section %u to %u\n", relsec,
+	       sechdrs[relsec].sh_info);
+
+	for (i = 0; i < sechdrs[relsec].sh_size / sizeof(*rela); i++) {
+		location = (void *)sechdrs[sechdrs[relsec].sh_info].sh_addr
+			+ rela[i].r_offset;
+		sym = (Elf64_Sym *)sechdrs[symindex].sh_addr
+			+ ELF64_R_SYM(rela[i].r_info);
+		symname = me->core_kallsyms.strtab
+			+ sym->st_name;
+
+		if (ELF64_R_TYPE(rela[i].r_info) != R_PPC_REL24)
+			continue;
+
+		if (sym->st_shndx != SHN_UNDEF &&
+		    sym->st_shndx != SHN_LIVEPATCH)
+			continue;
+
+		instruction = (u32 *)location;
+		if (is_mprofile_mcount_callsite(symname, instruction))
+			continue;
+
+		if (!instr_is_relative_link_branch(*instruction))
+			continue;
+
+		instruction += 1;
+		*instruction = PPC_INST_NOP;
+	}
+}
+#endif
+
 #ifdef CONFIG_DYNAMIC_FTRACE
 
 #ifdef CONFIG_MPROFILE_KERNEL
diff --git a/arch/s390/kernel/module.c b/arch/s390/kernel/module.c
index 31889db609e9..56867d052010 100644
--- a/arch/s390/kernel/module.c
+++ b/arch/s390/kernel/module.c
@@ -437,6 +437,14 @@ int apply_relocate_add(Elf_Shdr *sechdrs, const char *strtab,
 	return 0;
 }
 
+#ifdef CONFIG_LIVEPATCH
+void clear_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
+			unsigned int symindex, unsigned int relsec,
+			struct module *me)
+{
+}
+#endif
+
 int module_finalize(const Elf_Ehdr *hdr,
 		    const Elf_Shdr *sechdrs,
 		    struct module *me)
diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c
index d5c72cb877b3..b07d71f125e6 100644
--- a/arch/x86/kernel/module.c
+++ b/arch/x86/kernel/module.c
@@ -215,6 +215,49 @@ int apply_relocate_add(Elf64_Shdr *sechdrs,
 	       me->name);
 	return -ENOEXEC;
 }
+
+#ifdef CONFIG_LIVEPATCH
+void clear_relocate_add(Elf64_Shdr *sechdrs,
+		   const char *strtab,
+		   unsigned int symindex,
+		   unsigned int relsec,
+		   struct module *me)
+{
+	unsigned int i;
+	Elf64_Rela *rel = (void *)sechdrs[relsec].sh_addr;
+	void *loc;
+
+	DEBUGP("Clearing relocate section %u to %u\n",
+	       relsec, sechdrs[relsec].sh_info);
+	for (i = 0; i < sechdrs[relsec].sh_size / sizeof(*rel); i++) {
+		loc = (void *)sechdrs[sechdrs[relsec].sh_info].sh_addr
+			+ rel[i].r_offset;
+
+		switch (ELF64_R_TYPE(rel[i].r_info)) {
+		case R_X86_64_NONE:
+			break;
+		case R_X86_64_64:
+			*(u64 *)loc = 0;
+			break;
+		case R_X86_64_32:
+			*(u32 *)loc = 0;
+			break;
+		case R_X86_64_32S:
+			*(s32 *)loc = 0;
+			break;
+		case R_X86_64_PC32:
+		case R_X86_64_PLT32:
+			*(u32 *)loc = 0;
+			break;
+		case R_X86_64_PC64:
+			*(u64 *)loc = 0;
+			break;
+		default:
+			break;
+		}
+	}
+}
+#endif
 #endif
 
 int module_finalize(const Elf_Ehdr *hdr,
diff --git a/include/linux/moduleloader.h b/include/linux/moduleloader.h
index 31013c2effd3..f1e52692db5f 100644
--- a/include/linux/moduleloader.h
+++ b/include/linux/moduleloader.h
@@ -62,6 +62,13 @@ int apply_relocate_add(Elf_Shdr *sechdrs,
 		       unsigned int symindex,
 		       unsigned int relsec,
 		       struct module *mod);
+#ifdef CONFIG_LIVEPATCH
+void clear_relocate_add(Elf64_Shdr *sechdrs,
+		   const char *strtab,
+		   unsigned int symindex,
+		   unsigned int relsec,
+		   struct module *me);
+#endif
 #else
 static inline int apply_relocate_add(Elf_Shdr *sechdrs,
 				     const char *strtab,
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index ab4a4606d19b..f0b380d2a17a 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -295,6 +295,45 @@ static int klp_write_object_relocations(struct module *pmod,
 	return ret;
 }
 
+static void klp_clear_object_relocations(struct module *pmod,
+					struct klp_object *obj)
+{
+	int i, cnt;
+	const char *objname, *secname;
+	char sec_objname[MODULE_NAME_LEN];
+	Elf_Shdr *sec;
+
+	objname = klp_is_module(obj) ? obj->name : "vmlinux";
+
+	/* For each klp relocation section */
+	for (i = 1; i < pmod->klp_info->hdr.e_shnum; i++) {
+		sec = pmod->klp_info->sechdrs + i;
+		secname = pmod->klp_info->secstrings + sec->sh_name;
+		if (!(sec->sh_flags & SHF_RELA_LIVEPATCH))
+			continue;
+
+		/*
+		 * Format: .klp.rela.sec_objname.section_name
+		 * See comment in klp_resolve_symbols() for an explanation
+		 * of the selected field width value.
+		 */
+		secname = pmod->klp_info->secstrings + sec->sh_name;
+		cnt = sscanf(secname, ".klp.rela.%55[^.]", sec_objname);
+		if (cnt != 1) {
+			pr_err("section %s has an incorrectly formatted name\n",
+			       secname);
+			continue;
+		}
+
+		if (strcmp(objname, sec_objname))
+			continue;
+
+		clear_relocate_add(pmod->klp_info->sechdrs,
+				   pmod->core_kallsyms.strtab,
+				   pmod->klp_info->symndx, i, pmod);
+	}
+}
+
 /*
  * Sysfs Interface
  *
@@ -1100,6 +1139,12 @@ static void klp_cleanup_module_patches_limited(struct module *mod,
 
 			klp_post_unpatch_callback(obj);
 
+			mutex_lock(&text_mutex);
+			module_disable_ro(patch->mod);
+			klp_clear_object_relocations(patch->mod, obj);
+			module_enable_ro(patch->mod, true);
+			mutex_unlock(&text_mutex);
+
 			klp_free_object_loaded(obj);
 			break;
 		}
-- 
2.23.0

Re: [RFC PATCH v2 1/3] livepatch: Clear relocation targets on a module removal

[Petr Mladek]

On Thu 2019-09-05 14:45:12, Miroslav Benes wrote:
> We thus decided to reverse the relocation patching (clear all relocation
> targets on x86_64, or return back nops on powerpc). The solution is not
> universal and is too much arch-specific, but it may prove to be simpler
> in the end.
> 
> diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c
> index a93b10c48000..e461d456e447 100644
> --- a/arch/powerpc/kernel/module_64.c
> +++ b/arch/powerpc/kernel/module_64.c
> @@ -741,6 +741,51 @@ int apply_relocate_add(Elf64_Shdr *sechdrs,
>  	return 0;
>  }
>  
> +#ifdef CONFIG_LIVEPATCH
> +void clear_relocate_add(Elf64_Shdr *sechdrs,
> +		       const char *strtab,
> +		       unsigned int symindex,
> +		       unsigned int relsec,
> +		       struct module *me)
> +{
> +	unsigned int i;
> +	Elf64_Rela *rela = (void *)sechdrs[relsec].sh_addr;
> +	Elf64_Sym *sym;
> +	unsigned long *location;
> +	const char *symname;
> +	u32 *instruction;
> +
> +	pr_debug("Applying ADD relocate section %u to %u\n", relsec,

s/Applying/Clearing/

> +	       sechdrs[relsec].sh_info);
> +
> +	for (i = 0; i < sechdrs[relsec].sh_size / sizeof(*rela); i++) {
> +		location = (void *)sechdrs[sechdrs[relsec].sh_info].sh_addr
> +			+ rela[i].r_offset;
> +		sym = (Elf64_Sym *)sechdrs[symindex].sh_addr
> +			+ ELF64_R_SYM(rela[i].r_info);
> +		symname = me->core_kallsyms.strtab
> +			+ sym->st_name;
> +
> +		if (ELF64_R_TYPE(rela[i].r_info) != R_PPC_REL24)
> +			continue;

I expected that the code below would reverse the operations
in apply_relocate_add() for case R_PPC_REL24. But it is not
obvious for me.

It might be because I am not familiar with the code. Or would
it deserve some comments?

> +
> +		if (sym->st_shndx != SHN_UNDEF &&
> +		    sym->st_shndx != SHN_LIVEPATCH)
> +			continue;
> +
> +		instruction = (u32 *)location;
> +		if (is_mprofile_mcount_callsite(symname, instruction))
> +			continue;
> +
> +		if (!instr_is_relative_link_branch(*instruction))
> +			continue;
> +
> +		instruction += 1;
> +		*instruction = PPC_INST_NOP;
> +	}
> +}
> +#endif
> +
>  #ifdef CONFIG_DYNAMIC_FTRACE
>  
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index ab4a4606d19b..f0b380d2a17a 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -295,6 +295,45 @@ static int klp_write_object_relocations(struct module *pmod,
>  	return ret;
>  }
>  
> +static void klp_clear_object_relocations(struct module *pmod,
> +					struct klp_object *obj)
> +{
> +	int i, cnt;
> +	const char *objname, *secname;
> +	char sec_objname[MODULE_NAME_LEN];
> +	Elf_Shdr *sec;
> +
> +	objname = klp_is_module(obj) ? obj->name : "vmlinux";
> +
> +	/* For each klp relocation section */
> +	for (i = 1; i < pmod->klp_info->hdr.e_shnum; i++) {
> +		sec = pmod->klp_info->sechdrs + i;
> +		secname = pmod->klp_info->secstrings + sec->sh_name;
> +		if (!(sec->sh_flags & SHF_RELA_LIVEPATCH))
> +			continue;
> +
> +		/*
> +		 * Format: .klp.rela.sec_objname.section_name
> +		 * See comment in klp_resolve_symbols() for an explanation
> +		 * of the selected field width value.
> +		 */
> +		secname = pmod->klp_info->secstrings + sec->sh_name;
> +		cnt = sscanf(secname, ".klp.rela.%55[^.]", sec_objname);
> +		if (cnt != 1) {
> +			pr_err("section %s has an incorrectly formatted name\n",
> +			       secname);
> +			continue;
> +		}
> +
> +		if (strcmp(objname, sec_objname))
> +			continue;
> +

It would make the review easier when the order of 1st and 2nd
patch was swaped. I mean that I would not need to check twice
that the two functions actually share the same code.

> +		clear_relocate_add(pmod->klp_info->sechdrs,
> +				   pmod->core_kallsyms.strtab,
> +				   pmod->klp_info->symndx, i, pmod);
> +	}
> +}
> +
>  /*
>   * Sysfs Interface
>   *

I was not able to check correctness of the ppc and s390 parts.
Otherwise, it looks good to me.

Best Regards,
Petr

Re: [RFC PATCH v2 1/3] livepatch: Clear relocation targets on a module removal

[Miroslav Benes]

On Wed, 2 Oct 2019, Petr Mladek wrote:

> On Thu 2019-09-05 14:45:12, Miroslav Benes wrote:
> > We thus decided to reverse the relocation patching (clear all relocation
> > targets on x86_64, or return back nops on powerpc). The solution is not
> > universal and is too much arch-specific, but it may prove to be simpler
> > in the end.
> > 
> > diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c
> > index a93b10c48000..e461d456e447 100644
> > --- a/arch/powerpc/kernel/module_64.c
> > +++ b/arch/powerpc/kernel/module_64.c
> > @@ -741,6 +741,51 @@ int apply_relocate_add(Elf64_Shdr *sechdrs,
> >  	return 0;
> >  }
> >  
> > +#ifdef CONFIG_LIVEPATCH
> > +void clear_relocate_add(Elf64_Shdr *sechdrs,
> > +		       const char *strtab,
> > +		       unsigned int symindex,
> > +		       unsigned int relsec,
> > +		       struct module *me)
> > +{
> > +	unsigned int i;
> > +	Elf64_Rela *rela = (void *)sechdrs[relsec].sh_addr;
> > +	Elf64_Sym *sym;
> > +	unsigned long *location;
> > +	const char *symname;
> > +	u32 *instruction;
> > +
> > +	pr_debug("Applying ADD relocate section %u to %u\n", relsec,
> 
> s/Applying/Clearing/

Ugh. Thanks for noticing.
 
> > +	       sechdrs[relsec].sh_info);
> > +
> > +	for (i = 0; i < sechdrs[relsec].sh_size / sizeof(*rela); i++) {
> > +		location = (void *)sechdrs[sechdrs[relsec].sh_info].sh_addr
> > +			+ rela[i].r_offset;
> > +		sym = (Elf64_Sym *)sechdrs[symindex].sh_addr
> > +			+ ELF64_R_SYM(rela[i].r_info);
> > +		symname = me->core_kallsyms.strtab
> > +			+ sym->st_name;
> > +
> > +		if (ELF64_R_TYPE(rela[i].r_info) != R_PPC_REL24)
> > +			continue;
> 
> I expected that the code below would reverse the operations
> in apply_relocate_add() for case R_PPC_REL24. But it is not
> obvious for me.

It should, but it is not obvious. See restore_r2(). We only need to 
replace PPC_INST_LD_TOC instruction with PPC_INST_NOP and that's it.

> It might be because I am not familiar with the code. Or would
> it deserve some comments?

Maybe.

> > +
> > +		if (sym->st_shndx != SHN_UNDEF &&
> > +		    sym->st_shndx != SHN_LIVEPATCH)
> > +			continue;
> > +
> > +		instruction = (u32 *)location;
> > +		if (is_mprofile_mcount_callsite(symname, instruction))
> > +			continue;
> > +
> > +		if (!instr_is_relative_link_branch(*instruction))
> > +			continue;
> > +
> > +		instruction += 1;
> > +		*instruction = PPC_INST_NOP;
> > +	}
> > +}
> > +#endif
> > +
> >  #ifdef CONFIG_DYNAMIC_FTRACE
> >  
> > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > index ab4a4606d19b..f0b380d2a17a 100644
> > --- a/kernel/livepatch/core.c
> > +++ b/kernel/livepatch/core.c
> > @@ -295,6 +295,45 @@ static int klp_write_object_relocations(struct module *pmod,
> >  	return ret;
> >  }
> >  
> > +static void klp_clear_object_relocations(struct module *pmod,
> > +					struct klp_object *obj)
> > +{
> > +	int i, cnt;
> > +	const char *objname, *secname;
> > +	char sec_objname[MODULE_NAME_LEN];
> > +	Elf_Shdr *sec;
> > +
> > +	objname = klp_is_module(obj) ? obj->name : "vmlinux";
> > +
> > +	/* For each klp relocation section */
> > +	for (i = 1; i < pmod->klp_info->hdr.e_shnum; i++) {
> > +		sec = pmod->klp_info->sechdrs + i;
> > +		secname = pmod->klp_info->secstrings + sec->sh_name;
> > +		if (!(sec->sh_flags & SHF_RELA_LIVEPATCH))
> > +			continue;
> > +
> > +		/*
> > +		 * Format: .klp.rela.sec_objname.section_name
> > +		 * See comment in klp_resolve_symbols() for an explanation
> > +		 * of the selected field width value.
> > +		 */
> > +		secname = pmod->klp_info->secstrings + sec->sh_name;
> > +		cnt = sscanf(secname, ".klp.rela.%55[^.]", sec_objname);
> > +		if (cnt != 1) {
> > +			pr_err("section %s has an incorrectly formatted name\n",
> > +			       secname);
> > +			continue;
> > +		}
> > +
> > +		if (strcmp(objname, sec_objname))
> > +			continue;
> > +
> 
> It would make the review easier when the order of 1st and 2nd
> patch was swaped. I mean that I would not need to check twice
> that the two functions actually share the same code.

Ok.
 
> > +		clear_relocate_add(pmod->klp_info->sechdrs,
> > +				   pmod->core_kallsyms.strtab,
> > +				   pmod->klp_info->symndx, i, pmod);
> > +	}
> > +}
> > +
> >  /*
> >   * Sysfs Interface
> >   *
> 
> I was not able to check correctness of the ppc and s390 parts.
> Otherwise, it looks good to me.

Thanks
Miroslav

Re: [RFC PATCH v2 1/3] livepatch: Clear relocation targets on a module removal

[Josh Poimboeuf]

On Thu, Sep 05, 2019 at 02:45:12PM +0200, Miroslav Benes wrote:
> Josh reported a bug:
> 
>   When the object to be patched is a module, and that module is
>   rmmod'ed and reloaded, it fails to load with:
> 
>   module: x86/modules: Skipping invalid relocation target, existing value is nonzero for type 2, loc 00000000ba0302e9, val ffffffffa03e293c
>   livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
>   livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'
> 
>   The livepatch module has a relocation which references a symbol
>   in the _previous_ loading of nfsd. When apply_relocate_add()
>   tries to replace the old relocation with a new one, it sees that
>   the previous one is nonzero and it errors out.
> 
>   On ppc64le, we have a similar issue:
> 
>   module_64: livepatch_nfsd: Expected nop after call, got e8410018 at e_show+0x60/0x548 [livepatch_nfsd]
>   livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
>   livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'
> 
> He also proposed three different solutions. We could remove the error
> check in apply_relocate_add() introduced by commit eda9cec4c9a1
> ("x86/module: Detect and skip invalid relocations"). However the check
> is useful for detecting corrupted modules.
> 
> We could also deny the patched modules to be removed. If it proved to be
> a major drawback for users, we could still implement a different
> approach. The solution would also complicate the existing code a lot.
> 
> We thus decided to reverse the relocation patching (clear all relocation
> targets on x86_64, or return back nops on powerpc). The solution is not
> universal and is too much arch-specific, but it may prove to be simpler
> in the end.
> 
> Reported-by: Josh Poimboeuf <jpoimboe@redhat.com>
> Signed-off-by: Miroslav Benes <mbenes@suse.cz>

Since we decided to fix late module patching at LPC, the commit message
and clear_relocate_add() should both probably clarify that these
functions are hacks which are relatively temporary, until we fix the
root cause.

But this patch gives me a bad feeling :-/  Not that I have a better
idea.

Has anybody seen this problem in the real world?  If not, maybe we'd be
better off just pretending the problem doesn't exist for now.

-- 
Josh

Re: [RFC PATCH v2 1/3] livepatch: Clear relocation targets on a module removal

[Miroslav Benes]

On Wed, 2 Oct 2019, Josh Poimboeuf wrote:

> On Thu, Sep 05, 2019 at 02:45:12PM +0200, Miroslav Benes wrote:
> > Josh reported a bug:
> > 
> >   When the object to be patched is a module, and that module is
> >   rmmod'ed and reloaded, it fails to load with:
> > 
> >   module: x86/modules: Skipping invalid relocation target, existing value is nonzero for type 2, loc 00000000ba0302e9, val ffffffffa03e293c
> >   livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
> >   livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'
> > 
> >   The livepatch module has a relocation which references a symbol
> >   in the _previous_ loading of nfsd. When apply_relocate_add()
> >   tries to replace the old relocation with a new one, it sees that
> >   the previous one is nonzero and it errors out.
> > 
> >   On ppc64le, we have a similar issue:
> > 
> >   module_64: livepatch_nfsd: Expected nop after call, got e8410018 at e_show+0x60/0x548 [livepatch_nfsd]
> >   livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
> >   livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'
> > 
> > He also proposed three different solutions. We could remove the error
> > check in apply_relocate_add() introduced by commit eda9cec4c9a1
> > ("x86/module: Detect and skip invalid relocations"). However the check
> > is useful for detecting corrupted modules.
> > 
> > We could also deny the patched modules to be removed. If it proved to be
> > a major drawback for users, we could still implement a different
> > approach. The solution would also complicate the existing code a lot.
> > 
> > We thus decided to reverse the relocation patching (clear all relocation
> > targets on x86_64, or return back nops on powerpc). The solution is not
> > universal and is too much arch-specific, but it may prove to be simpler
> > in the end.
> > 
> > Reported-by: Josh Poimboeuf <jpoimboe@redhat.com>
> > Signed-off-by: Miroslav Benes <mbenes@suse.cz>
> 
> Since we decided to fix late module patching at LPC, the commit message
> and clear_relocate_add() should both probably clarify that these
> functions are hacks which are relatively temporary, until we fix the
> root cause.

It was the plan, but thanks for pointing it out explicitly. I could 
forget.
 
> But this patch gives me a bad feeling :-/  Not that I have a better
> idea.

I know what you are talking about.

> Has anybody seen this problem in the real world?  If not, maybe we'd be
> better off just pretending the problem doesn't exist for now.

I don't think so. You reported the issue originally and I guess it 
happened during the testing. Then there is a report from Huawei, but it 
suggests testing environment too. Reloading modules seems artificial to 
me.

So I agree, we can pretend the issue does not exist and wait for the real 
solution.

Miroslav

[RFC PATCH v2 2/3] livepatch: Unify functions for writing and clearing object relocations

[Miroslav Benes]

Functions klp_write_object_relocations() and
klp_clear_object_relocations() share a lot of code. Take the code out to
a common function and provide the specific actions through callbacks.

Signed-off-by: Miroslav Benes <mbenes@suse.cz>
---
 kernel/livepatch/core.c | 84 +++++++++++++++++------------------------
 1 file changed, 34 insertions(+), 50 deletions(-)

diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index f0b380d2a17a..023c9333c276 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -245,8 +245,11 @@ static int klp_resolve_symbols(Elf_Shdr *relasec, struct module *pmod)
 	return 0;
 }
 
-static int klp_write_object_relocations(struct module *pmod,
-					struct klp_object *obj)
+typedef int (reloc_update_fn_t)(struct module *, unsigned int);
+
+static int klp_update_object_relocations(struct module *pmod,
+					 struct klp_object *obj,
+					 reloc_update_fn_t reloc_update_fn)
 {
 	int i, cnt, ret = 0;
 	const char *objname, *secname;
@@ -281,13 +284,7 @@ static int klp_write_object_relocations(struct module *pmod,
 		if (strcmp(objname, sec_objname))
 			continue;
 
-		ret = klp_resolve_symbols(sec, pmod);
-		if (ret)
-			break;
-
-		ret = apply_relocate_add(pmod->klp_info->sechdrs,
-					 pmod->core_kallsyms.strtab,
-					 pmod->klp_info->symndx, i, pmod);
+		ret = reloc_update_fn(pmod, i);
 		if (ret)
 			break;
 	}
@@ -295,45 +292,6 @@ static int klp_write_object_relocations(struct module *pmod,
 	return ret;
 }
 
-static void klp_clear_object_relocations(struct module *pmod,
-					struct klp_object *obj)
-{
-	int i, cnt;
-	const char *objname, *secname;
-	char sec_objname[MODULE_NAME_LEN];
-	Elf_Shdr *sec;
-
-	objname = klp_is_module(obj) ? obj->name : "vmlinux";
-
-	/* For each klp relocation section */
-	for (i = 1; i < pmod->klp_info->hdr.e_shnum; i++) {
-		sec = pmod->klp_info->sechdrs + i;
-		secname = pmod->klp_info->secstrings + sec->sh_name;
-		if (!(sec->sh_flags & SHF_RELA_LIVEPATCH))
-			continue;
-
-		/*
-		 * Format: .klp.rela.sec_objname.section_name
-		 * See comment in klp_resolve_symbols() for an explanation
-		 * of the selected field width value.
-		 */
-		secname = pmod->klp_info->secstrings + sec->sh_name;
-		cnt = sscanf(secname, ".klp.rela.%55[^.]", sec_objname);
-		if (cnt != 1) {
-			pr_err("section %s has an incorrectly formatted name\n",
-			       secname);
-			continue;
-		}
-
-		if (strcmp(objname, sec_objname))
-			continue;
-
-		clear_relocate_add(pmod->klp_info->sechdrs,
-				   pmod->core_kallsyms.strtab,
-				   pmod->klp_info->symndx, i, pmod);
-	}
-}
-
 /*
  * Sysfs Interface
  *
@@ -751,6 +709,21 @@ void __weak arch_klp_init_object_loaded(struct klp_patch *patch,
 {
 }
 
+static int klp_write_relocations(struct module *pmod,
+				 unsigned int relsec)
+{
+	int ret;
+
+	ret = klp_resolve_symbols(pmod->klp_info->sechdrs + relsec, pmod);
+	if (ret)
+		return ret;
+
+	ret = apply_relocate_add(pmod->klp_info->sechdrs,
+				 pmod->core_kallsyms.strtab,
+				 pmod->klp_info->symndx, relsec, pmod);
+	return ret;
+}
+
 /* parts of the initialization that is done only when the object is loaded */
 static int klp_init_object_loaded(struct klp_patch *patch,
 				  struct klp_object *obj)
@@ -761,7 +734,8 @@ static int klp_init_object_loaded(struct klp_patch *patch,
 	mutex_lock(&text_mutex);
 
 	module_disable_ro(patch->mod);
-	ret = klp_write_object_relocations(patch->mod, obj);
+	ret = klp_update_object_relocations(patch->mod, obj,
+					    klp_write_relocations);
 	if (ret) {
 		module_enable_ro(patch->mod, true);
 		mutex_unlock(&text_mutex);
@@ -1111,6 +1085,15 @@ void klp_discard_nops(struct klp_patch *new_patch)
 	klp_free_objects_dynamic(klp_transition_patch);
 }
 
+static int klp_clear_relocations(struct module *pmod,
+				 unsigned int relsec)
+{
+	clear_relocate_add(pmod->klp_info->sechdrs,
+			   pmod->core_kallsyms.strtab,
+			   pmod->klp_info->symndx, relsec, pmod);
+	return 0;
+}
+
 /*
  * Remove parts of patches that touch a given kernel module. The list of
  * patches processed might be limited. When limit is NULL, all patches
@@ -1141,7 +1124,8 @@ static void klp_cleanup_module_patches_limited(struct module *mod,
 
 			mutex_lock(&text_mutex);
 			module_disable_ro(patch->mod);
-			klp_clear_object_relocations(patch->mod, obj);
+			klp_update_object_relocations(patch->mod, obj,
+						      klp_clear_relocations);
 			module_enable_ro(patch->mod, true);
 			mutex_unlock(&text_mutex);
 
-- 
2.23.0

Re: [RFC PATCH v2 2/3] livepatch: Unify functions for writing and clearing object relocations

[Petr Mladek]

On Thu 2019-09-05 14:45:13, Miroslav Benes wrote:
> Functions klp_write_object_relocations() and
> klp_clear_object_relocations() share a lot of code. Take the code out to
> a common function and provide the specific actions through callbacks.
> 
> Signed-off-by: Miroslav Benes <mbenes@suse.cz>

I would prefer when the callback is introduced before the 1st patch.
I do not resist on it. This patch looks good, so:

Reviewed-by: Petr Mladek <pmladek@suse.com>

Best Regards,
Petr

[RFC PATCH v2 3/3] livepatch: Clean up klp_update_object_relocations() return paths

[Miroslav Benes]

Signed-off-by: Miroslav Benes <mbenes@suse.cz>
---
 kernel/livepatch/core.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 023c9333c276..73ddddd5add5 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -251,7 +251,7 @@ static int klp_update_object_relocations(struct module *pmod,
 					 struct klp_object *obj,
 					 reloc_update_fn_t reloc_update_fn)
 {
-	int i, cnt, ret = 0;
+	int i, cnt, ret;
 	const char *objname, *secname;
 	char sec_objname[MODULE_NAME_LEN];
 	Elf_Shdr *sec;
@@ -277,8 +277,7 @@ static int klp_update_object_relocations(struct module *pmod,
 		if (cnt != 1) {
 			pr_err("section %s has an incorrectly formatted name\n",
 			       secname);
-			ret = -EINVAL;
-			break;
+			return -EINVAL;
 		}
 
 		if (strcmp(objname, sec_objname))
@@ -286,10 +285,10 @@ static int klp_update_object_relocations(struct module *pmod,
 
 		ret = reloc_update_fn(pmod, i);
 		if (ret)
-			break;
+			return ret;
 	}
 
-	return ret;
+	return 0;
 }
 
 /*
-- 
2.23.0

Re: [RFC PATCH v2 3/3] livepatch: Clean up klp_update_object_relocations() return paths

[Petr Mladek]

On Thu 2019-09-05 14:45:14, Miroslav Benes wrote:
> Signed-off-by: Miroslav Benes <mbenes@suse.cz>

This might depend on personal preferences. What was the motivation
for this patch, please? Did it just follow some common
style in this source file?

To make it clear. I have no real preference. I just want to avoid
some back and forth changes of the code depending on who touches
it at the moment.

I would prefer to either remove this patch or explain the motivation
in the commit message. Beside that

Reviewed-by: Petr Mladek <pmladek@suse.com>

Best Regards,
Petr

Re: [RFC PATCH v2 3/3] livepatch: Clean up klp_update_object_relocations() return paths

[Miroslav Benes]

On Wed, 2 Oct 2019, Petr Mladek wrote:

> On Thu 2019-09-05 14:45:14, Miroslav Benes wrote:
> > Signed-off-by: Miroslav Benes <mbenes@suse.cz>
> 
> This might depend on personal preferences.

True.

> What was the motivation
> for this patch, please? Did it just follow some common
> style in this source file?

We had it like this once, so it is only going back to the original code. 
And yes, I think it is better.

Commit b56b36ee6751 ("livepatch: Cleanup module page permission changes") 
changed it due to the error handling. Commit 255e732c61db ("livepatch: use 
arch_klp_init_object_loaded() to finish arch-specific tasks") removed the 
reason for the change but did not cleanup the rest.

> To make it clear. I have no real preference. I just want to avoid
> some back and forth changes of the code depending on who touches
> it at the moment.

I have no real preference either. I noticed something I did not like while 
touching the code and that's it.

> I would prefer to either remove this patch or explain the motivation
> in the commit message. Beside that
> 
> Reviewed-by: Petr Mladek <pmladek@suse.com>

Ok, thanks.
Miroslav

Re: [RFC PATCH v2 0/3] livepatch: Clear relocation targets on a module removal

[Miroslav Benes]

On Thu, 5 Sep 2019, Miroslav Benes wrote:

> Updated version with Petr's feedback. It looks a bit different and
> better now (I would say). Not that it should be considered before we
> decide what to do with late module patching, but I finished it before
> the discussion started and someone could be interested.
> 
> v1: http://lore.kernel.org/r/20190719122840.15353-1-mbenes@suse.cz
> 
> Tested on x86_64, ppc64le and s390x. Cross-compiled on arm64 to verify
> that nothing is broken.
> 
> [1] 20180602161151.apuhs2dygsexmcg2@treble
> [2] 1561019068-132672-1-git-send-email-cj.chengjian@huawei.com
> [3] 20180607092949.1706-1-mbenes@suse.cz
> 
> Miroslav Benes (3):
>   livepatch: Clear relocation targets on a module removal
>   livepatch: Unify functions for writing and clearing object relocations
>   livepatch: Clean up klp_update_object_relocations() return paths
> 
>  arch/powerpc/kernel/module_64.c | 45 +++++++++++++++++++++++++
>  arch/s390/kernel/module.c       |  8 +++++
>  arch/x86/kernel/module.c        | 43 ++++++++++++++++++++++++
>  include/linux/moduleloader.h    |  7 ++++
>  kernel/livepatch/core.c         | 58 ++++++++++++++++++++++++---------
>  5 files changed, 146 insertions(+), 15 deletions(-)

Ping.

If I remember correctly, we decided to have this as a temporary solution 
before better late module patching is implemented. Feedback is welcome.
I'll then resend with arch maintainters CCed.

Thanks
Miroslav