[RFC PATCH 0/5] Arch-independent livepatch

[RFC PATCH 0/5] Arch-independent livepatch

[Jessica Yu]

This patchset removes livepatch's need for architecture-specific relocation
code by leveraging existing code in the module loader to perform
arch-dependent work. Specifically, instead of duplicating code and
re-implementing what the apply_relocate_add() function in the module loader
already does in livepatch's klp_write_module_reloc(), we reuse
apply_relocate_add() to write relocations. The hope is that this will make
livepatch more easily portable to other architectures and greatly reduce
the amount of arch-specific code required to port livepatch to a particular
architecture.

Background: Why does livepatch need to write its own relocations?
==
A typical livepatch module contains patched versions of functions that can
reference non-exported global symbols and non-included local symbols.
Relocations referencing these types of symbols cannot be left in as-is
since the kernel module loader cannot resolve them and will therefore
reject the livepatch module. Furthermore, we cannot apply relocations that
affect modules not loaded yet at run time (e.g. a patch to a driver). The
current kpatch build system therefore solves this problem by embedding
special "dynrela" (dynamic reloc) sections in the resulting patch module
elf output. Using these dynrela sections, livepatch can correctly resolve
symbols while taking into account its scope and what module the symbol
belongs to, and then manually apply the dynamic relocations.

Motivation: Why is having arch-dependent relocation code a problem?
==
The original motivation for this patchset stems from the increasing
roadblocks encountered while attempting to port livepatch to s390.
Specifically, there were problems dealing with s390 PLT and GOT relocation
types (R_390_{PLT,GOT}), which are handled differently from x86's
relocation types (which are much simpler to deal with, and a single
livepatch function (klp_write_module_reloc()) has been sufficient enough).
These s390 reloc types cannot be handled by simply performing a calculation
(as in the x86 case). For s390 modules with PLT/GOT relocations, the kernel
module loader allocates and fills in PLT+GOT table entries for every symbol
referenced by a PLT/GOT reloc in module core memory. So the problem of
porting livepatch to s390 became much more complicated than simply writing
an s390-specific klp_write_module_reloc() function. How can livepatch
handle these relocation types if the s390 module loader needs to allocate
and fill PLT/GOT entries ahead of time? The potential solutions were: 1)
have livepatch possibly allocate and maintain its own PLT/GOT tables for
every patch module (requiring even more arch-specific code), 2) modify the
s390 module loader heavily to accommodate livepatch modules (i.e. allocate
all the needed PLT/GOT entries for livepatch in advance but refrain from
applying relocations for to-be-patched modules), or 3) eliminate this
potential mess by leveraging module loader code to do all the relocation
work, letting livepatch off the hook completely. Solution #3 is what this
patchset implements.

How does this patchset remedy these problems?
==
Reusing the module loader code to perform livepatch relocations means that
livepatch no longer needs arch-specific reloc code and the aforementioned
problems with s390 PLT/GOT reloc types disappear (because we let the module
loader do all the relocation work for us). It will enable livepatch to be
more easily ported to other architectures.

Summary of proposed changes
==
This patch series enables livepatch to use the module loader's
apply_relocate_add() function to resolve livepatch relocations (i.e. what
used to be dynrelas). apply_relocate_add() requires access to a patch
module's section headers, symbol table, reloc section indices, etc., and all
of these are accessible through the load_info struct used in the module
loader. Therefore we persist this struct for livepatch modules and it is
made available through module->info.

The ELF-related changes enable livepatch to patch modules that are not
loaded yet. In order to use apply_relocate_add(), we need real SHT_RELA
sections to pass in. A complication here is that relocations for
not-yet-loaded modules should not be applied when the patch module loads;
they should only be applied once the target module is loaded. Thus kpatch
build scripts were modified to output a livepatch module that contains
special __klp_rela sections that correspond to the modules being patched.
They are marked with a special SHF_RELA_LIVEPATCH section flag to indicate
to the module loader that it should ignore that reloc section and that
livepatch will handle them. The SHN_LIVEPATCH shndx marks symbols that will
have to be resolved once their respective target module loads. So, the
module loader ignores these symbols (and does not attempt to resolve them).
Finally, the STB_LIVEPATCH_EXT symbol bind marks the scope of certain
livepatch symbols, so that livepatch can find the symbol in the right
place. These ELF constants were selected from OS-specific ranges according
to the definitions from glibc.

Jessica Yu (5):
  elf: add livepatch-specific elf constants
  module: save load_info for livepatch modules
  livepatch: reuse module loader code to write relocations
  samples: livepatch: init reloc list and mark as klp module
  livepatch: x86: remove unused relocation code

 arch/x86/kernel/Makefile             |   1 -
 arch/x86/kernel/livepatch.c          |  91 ------------------------------
 include/linux/livepatch.h            |  11 +++-
 include/linux/module.h               |  31 ++++++++++
 include/uapi/linux/elf.h             |   3 +
 kernel/livepatch/core.c              | 106 ++++++++++++++++++++++++-----------
 kernel/module.c                      |  36 +++++++-----
 samples/livepatch/livepatch-sample.c |   2 +
 8 files changed, 139 insertions(+), 142 deletions(-)
 delete mode 100644 arch/x86/kernel/livepatch.c

-- 
2.4.3

[RFC PATCH 0/5] Arch-independent livepatch

[Jessica Yu]

This patchset removes livepatch's need for architecture-specific relocation
code by leveraging existing code in the module loader to perform
arch-dependent work. Specifically, instead of duplicating code and
re-implementing what the apply_relocate_add() function in the module loader
already does in livepatch's klp_write_module_reloc(), we reuse
apply_relocate_add() to write relocations. The hope is that this will make
livepatch more easily portable to other architectures and greatly reduce
the amount of arch-specific code required to port livepatch to a particular
architecture.

Background: Why does livepatch need to write its own relocations?
==
A typical livepatch module contains patched versions of functions that can
reference non-exported global symbols and non-included local symbols.
Relocations referencing these types of symbols cannot be left in as-is
since the kernel module loader cannot resolve them and will therefore
reject the livepatch module. Furthermore, we cannot apply relocations that
affect modules not loaded yet at run time (e.g. a patch to a driver). The
current kpatch build system therefore solves this problem by embedding
special "dynrela" (dynamic reloc) sections in the resulting patch module
elf output. Using these dynrela sections, livepatch can correctly resolve
symbols while taking into account its scope and what module the symbol
belongs to, and then manually apply the dynamic relocations.

Motivation: Why is having arch-dependent relocation code a problem?
==
The original motivation for this patchset stems from the increasing
roadblocks encountered while attempting to port livepatch to s390.
Specifically, there were problems dealing with s390 PLT and GOT relocation
types (R_390_{PLT,GOT}), which are handled differently from x86's
relocation types (which are much simpler to deal with, and a single
livepatch function (klp_write_module_reloc()) has been sufficient enough).
These s390 reloc types cannot be handled by simply performing a calculation
(as in the x86 case). For s390 modules with PLT/GOT relocations, the kernel
module loader allocates and fills in PLT+GOT table entries for every symbol
referenced by a PLT/GOT reloc in module core memory. So the problem of
porting livepatch to s390 became much more complicated than simply writing
an s390-specific klp_write_module_reloc() function. How can livepatch
handle these relocation types if the s390 module loader needs to allocate
and fill PLT/GOT entries ahead of time? The potential solutions were: 1)
have livepatch possibly allocate and maintain its own PLT/GOT tables for
every patch module (requiring even more arch-specific code), 2) modify the
s390 module loader heavily to accommodate livepatch modules (i.e. allocate
all the needed PLT/GOT entries for livepatch in advance but refrain from
applying relocations for to-be-patched modules), or 3) eliminate this
potential mess by leveraging module loader code to do all the relocation
work, letting livepatch off the hook completely. Solution #3 is what this
patchset implements.

How does this patchset remedy these problems?
==
Reusing the module loader code to perform livepatch relocations means that
livepatch no longer needs arch-specific reloc code and the aforementioned
problems with s390 PLT/GOT reloc types disappear (because we let the module
loader do all the relocation work for us). It will enable livepatch to be
more easily ported to other architectures.

Summary of proposed changes
==
This patch series enables livepatch to use the module loader's
apply_relocate_add() function to resolve livepatch relocations (i.e. what
used to be dynrelas). apply_relocate_add() requires access to a patch
module's section headers, symbol table, reloc section indices, etc., and all
of these are accessible through the load_info struct used in the module
loader. Therefore we persist this struct for livepatch modules and it is
made available through module->info.

The ELF-related changes enable livepatch to patch modules that are not
loaded yet. In order to use apply_relocate_add(), we need real SHT_RELA
sections to pass in. A complication here is that relocations for
not-yet-loaded modules should not be applied when the patch module loads;
they should only be applied once the target module is loaded. Thus kpatch
build scripts were modified to output a livepatch module that contains
special __klp_rela sections that correspond to the modules being patched.
They are marked with a special SHF_RELA_LIVEPATCH section flag to indicate
to the module loader that it should ignore that reloc section and that
livepatch will handle them. The SHN_LIVEPATCH shndx marks symbols that will
have to be resolved once their respective target module loads. So, the
module loader ignores these symbols (and does not attempt to resolve them).
Finally, the STB_LIVEPATCH_EXT symbol bind marks the scope of certain
livepatch symbols, so that livepatch can find the symbol in the right
place. These ELF constants were selected from OS-specific ranges according
to the definitions from glibc.

Jessica Yu (5):
  elf: add livepatch-specific elf constants
  module: save load_info for livepatch modules
  livepatch: reuse module loader code to write relocations
  samples: livepatch: init reloc list and mark as klp module
  livepatch: x86: remove unused relocation code

 arch/x86/kernel/Makefile             |   1 -
 arch/x86/kernel/livepatch.c          |  91 ------------------------------
 include/linux/livepatch.h            |  11 +++-
 include/linux/module.h               |  31 ++++++++++
 include/uapi/linux/elf.h             |   3 +
 kernel/livepatch/core.c              | 106 ++++++++++++++++++++++++-----------
 kernel/module.c                      |  36 +++++++-----
 samples/livepatch/livepatch-sample.c |   2 +
 8 files changed, 139 insertions(+), 142 deletions(-)
 delete mode 100644 arch/x86/kernel/livepatch.c

-- 
2.4.3

[RFC PATCH 1/5] elf: add livepatch-specific elf constants

[Jessica Yu]

Add livepatch elf reloc section flag, livepatch symbol bind
and section index

Signed-off-by: Jessica Yu <jeyu@redhat.com>
---
 include/uapi/linux/elf.h | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h
index 71e1d0e..967ce1b 100644
--- a/include/uapi/linux/elf.h
+++ b/include/uapi/linux/elf.h
@@ -118,6 +118,7 @@ typedef __s64	Elf64_Sxword;
 #define STB_LOCAL  0
 #define STB_GLOBAL 1
 #define STB_WEAK   2
+#define STB_LIVEPATCH_EXT 11
 
 #define STT_NOTYPE  0
 #define STT_OBJECT  1
@@ -286,6 +287,7 @@ typedef struct elf64_phdr {
 #define SHF_ALLOC	0x2
 #define SHF_EXECINSTR	0x4
 #define SHF_MASKPROC	0xf0000000
+#define SHF_RELA_LIVEPATCH  0x4000000
 
 /* special section indexes */
 #define SHN_UNDEF	0
@@ -295,6 +297,7 @@ typedef struct elf64_phdr {
 #define SHN_ABS		0xfff1
 #define SHN_COMMON	0xfff2
 #define SHN_HIRESERVE	0xffff
+#define SHN_LIVEPATCH 0xff21
  
 typedef struct elf32_shdr {
   Elf32_Word	sh_name;
-- 
2.4.3

[RFC PATCH 2/5] module: save load_info for livepatch modules

[Jessica Yu]

In livepatch modules, preserve section, symbol, string information from
the load_info struct in the module loader. This information is used to
patch modules that are not loaded in memory yet; specifically it is used
to resolve remaining symbols and write relocations when the target
module loads.

Signed-off-by: Jessica Yu <jeyu@redhat.com>
---
 include/linux/module.h  | 25 +++++++++++++++++++++++++
 kernel/livepatch/core.c | 17 +++++++++++++++++
 kernel/module.c         | 36 ++++++++++++++++++++++--------------
 3 files changed, 64 insertions(+), 14 deletions(-)

diff --git a/include/linux/module.h b/include/linux/module.h
index 3a19c79..c8680b1 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -36,6 +36,20 @@ struct modversion_info {
 	char name[MODULE_NAME_LEN];
 };
 
+struct load_info {
+	Elf_Ehdr *hdr;
+	unsigned long len;
+	Elf_Shdr *sechdrs;
+	char *secstrings, *strtab;
+	unsigned long symoffs, stroffs;
+	struct _ddebug *debug;
+	unsigned int num_debug;
+	bool sig_ok;
+	struct {
+		unsigned int sym, str, mod, vers, info, pcpu;
+	} index;
+};
+
 struct module;
 
 struct module_kobject {
@@ -462,6 +476,8 @@ struct module {
 
 #ifdef CONFIG_LIVEPATCH
 	bool klp_alive;
+	/* save info to patch to-be-loaded modules */
+	struct load_info *info;
 #endif
 
 #ifdef CONFIG_MODULE_UNLOAD
@@ -635,6 +651,15 @@ static inline bool module_requested_async_probing(struct module *module)
 	return module && module->async_probe_requested;
 }
 
+#ifdef CONFIG_LIVEPATCH
+extern void klp_prepare_patch_module(struct module *mod,
+				     struct load_info *info);
+extern int
+apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
+		   unsigned int symindex, unsigned int relsec,
+		   struct module *me);
+#endif
+
 #else /* !CONFIG_MODULES... */
 
 /* Given an address, look for it in the exception tables. */
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 6e53441..087a8c7 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
 	.priority = INT_MIN+1, /* called late but before ftrace notifier */
 };
 
+/*
+ * Save necessary information from info in order to be able to
+ * patch modules that might be loaded later
+ */
+void klp_prepare_patch_module(struct module *mod, struct load_info *info)
+{
+	Elf_Shdr *symsect;
+
+	symsect = info->sechdrs + info->index.sym;
+	/* update sh_addr to point to symtab */
+	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
+
+	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
+	memcpy(mod->info, info, sizeof(*info));
+
+}
+
 static int __init klp_init(void)
 {
 	int ret;
diff --git a/kernel/module.c b/kernel/module.c
index 8f051a1..8ae3ca5 100644
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -318,20 +318,6 @@ int unregister_module_notifier(struct notifier_block *nb)
 }
 EXPORT_SYMBOL(unregister_module_notifier);
 
-struct load_info {
-	Elf_Ehdr *hdr;
-	unsigned long len;
-	Elf_Shdr *sechdrs;
-	char *secstrings, *strtab;
-	unsigned long symoffs, stroffs;
-	struct _ddebug *debug;
-	unsigned int num_debug;
-	bool sig_ok;
-	struct {
-		unsigned int sym, str, mod, vers, info, pcpu;
-	} index;
-};
-
 /* We require a truly strong try_module_get(): 0 means failure due to
    ongoing or failed initialization etc. */
 static inline int strong_try_module_get(struct module *mod)
@@ -2137,6 +2123,11 @@ static int simplify_symbols(struct module *mod, const struct load_info *info)
 			       (long)sym[i].st_value);
 			break;
 
+#ifdef CONFIG_LIVEPATCH
+		case SHN_LIVEPATCH:
+			break;
+#endif
+
 		case SHN_UNDEF:
 			ksym = resolve_symbol_wait(mod, info, name);
 			/* Ok if resolved.  */
@@ -2185,6 +2176,11 @@ static int apply_relocations(struct module *mod, const struct load_info *info)
 		if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC))
 			continue;
 
+#ifdef CONFIG_LIVEPATCH
+		if (info->sechdrs[i].sh_flags & SHF_RELA_LIVEPATCH)
+			continue;
+#endif
+
 		if (info->sechdrs[i].sh_type == SHT_REL)
 			err = apply_relocate(info->sechdrs, info->strtab,
 					     info->index.sym, i, mod);
@@ -3530,8 +3526,20 @@ static int load_module(struct load_info *info, const char __user *uargs,
 	if (err < 0)
 		goto bug_cleanup;
 
+#ifdef CONFIG_LIVEPATCH
+	/*
+	 * Save sechdrs, indices, and other data from info
+	 * in order to patch to-be-loaded modules.
+	 * Do not call free_copy() for livepatch modules.
+	 */
+	if (get_modinfo((struct load_info *)info, "livepatch"))
+		klp_prepare_patch_module(mod, info);
+	else
+		free_copy(info);
+#else
 	/* Get rid of temporary copy. */
 	free_copy(info);
+#endif
 
 	/* Done! */
 	trace_module_load(mod);
-- 
2.4.3

[RFC PATCH 3/5] livepatch: reuse module loader code to write relocations

[Jessica Yu]

Reuse module loader code to write relocations, thereby eliminating the
need for architecture specific code in livepatch. Namely, we reuse
apply_relocate_add() in the module loader to write relocs instead of
duplicating functionality in livepatch's klp_write_module_reloc(). To
apply relocation sections, remaining SHN_LIVEPATCH symbols referenced by
relocs are resolved and then apply_relocate_add() is called to apply
those relocations.

Signed-off-by: Jessica Yu <jeyu@redhat.com>
---
 include/linux/livepatch.h | 11 ++++--
 include/linux/module.h    |  6 ++++
 kernel/livepatch/core.c   | 89 +++++++++++++++++++++++++++++------------------
 3 files changed, 70 insertions(+), 36 deletions(-)

diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
index 31db7a0..601e892 100644
--- a/include/linux/livepatch.h
+++ b/include/linux/livepatch.h
@@ -85,7 +85,7 @@ struct klp_reloc {
 /**
  * struct klp_object - kernel object structure for live patching
  * @name:	module name (or NULL for vmlinux)
- * @relocs:	relocation entries to be applied at load time
+ * @reloc_secs:	relocation sections to be applied at load time
  * @funcs:	function entries for functions to be patched in the object
  * @kobj:	kobject for sysfs resources
  * @mod:	kernel module associated with the patched object
@@ -95,7 +95,7 @@ struct klp_reloc {
 struct klp_object {
 	/* external */
 	const char *name;
-	struct klp_reloc *relocs;
+	struct list_head reloc_secs;
 	struct klp_func *funcs;
 
 	/* internal */
@@ -129,6 +129,13 @@ struct klp_patch {
 #define klp_for_each_func(obj, func) \
 	for (func = obj->funcs; func->old_name; func++)
 
+struct klp_reloc_sec {
+	unsigned int index;
+	char *name;
+	char *objname;
+	struct list_head list;
+};
+
 int klp_register_patch(struct klp_patch *);
 int klp_unregister_patch(struct klp_patch *);
 int klp_enable_patch(struct klp_patch *);
diff --git a/include/linux/module.h b/include/linux/module.h
index c8680b1..3c34eb8 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -793,9 +793,15 @@ extern int module_sysfs_initialized;
 #ifdef CONFIG_DEBUG_SET_MODULE_RONX
 extern void set_all_modules_text_rw(void);
 extern void set_all_modules_text_ro(void);
+extern void
+set_page_attributes(void *start, void *end,
+		    int (*set)(unsigned long start, int num_pages));
 #else
 static inline void set_all_modules_text_rw(void) { }
 static inline void set_all_modules_text_ro(void) { }
+static inline void
+set_page_attributes(void *start, void *end,
+		    int (*set)(unsigned long start, int num_pages)) { }
 #endif
 
 #ifdef CONFIG_GENERIC_BUG
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 087a8c7..26c419f 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -28,6 +28,8 @@
 #include <linux/list.h>
 #include <linux/kallsyms.h>
 #include <linux/livepatch.h>
+#include <linux/elf.h>
+#include <asm/cacheflush.h>
 
 /**
  * struct klp_ops - structure for tracking registered ftrace ops structs
@@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module *pmod, const char *name,
 }
 
 static int klp_write_object_relocations(struct module *pmod,
-					struct klp_object *obj)
+					struct klp_object *obj,
+					struct klp_patch *patch)
 {
-	int ret;
-	struct klp_reloc *reloc;
+	int relindex, num_relas;
+	int i, ret = 0;
+	unsigned long addr;
+	unsigned int bind;
+	char *symname;
+	struct klp_reloc_sec *reloc_sec;
+	struct load_info *info;
+	Elf_Rela *rela;
+	Elf_Sym *sym, *symtab;
+	Elf_Shdr *symsect;
 
 	if (WARN_ON(!klp_is_object_loaded(obj)))
 		return -EINVAL;
 
-	if (WARN_ON(!obj->relocs))
-		return -EINVAL;
-
-	for (reloc = obj->relocs; reloc->name; reloc++) {
-		if (!klp_is_module(obj)) {
-			ret = klp_verify_vmlinux_symbol(reloc->name,
-							reloc->val);
-			if (ret)
-				return ret;
-		} else {
-			/* module, reloc->val needs to be discovered */
-			if (reloc->external)
-				ret = klp_find_external_symbol(pmod,
-							       reloc->name,
-							       &reloc->val);
-			else
-				ret = klp_find_object_symbol(obj->mod->name,
-							     reloc->name,
-							     &reloc->val);
-			if (ret)
-				return ret;
-		}
-		ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc,
-					     reloc->val + reloc->addend);
-		if (ret) {
-			pr_err("relocation failed for symbol '%s' at 0x%016lx (%d)\n",
-			       reloc->name, reloc->val, ret);
-			return ret;
+	info = pmod->info;
+	symsect = info->sechdrs + info->index.sym;
+	symtab = (void *)info->hdr + symsect->sh_offset;
+
+	/* For each __klp_rela section for this object */
+	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
+		relindex = reloc_sec->index;
+		num_relas = info->sechdrs[relindex].sh_size / sizeof(Elf_Rela);
+		rela = (Elf_Rela *) info->sechdrs[relindex].sh_addr;
+
+		/* For each rela in this __klp_rela section */
+		for (i = 0; i < num_relas; i++, rela++) {
+			sym = symtab + ELF_R_SYM(rela->r_info);
+			symname = info->strtab + sym->st_name;
+			bind = ELF_ST_BIND(sym->st_info);
+
+			if (sym->st_shndx == SHN_LIVEPATCH) {
+				if (bind == STB_LIVEPATCH_EXT)
+					ret = klp_find_external_symbol(pmod, symname, &addr);
+				else
+					ret = klp_find_object_symbol(obj->name, symname, &addr);
+				if (ret)
+					return ret;
+				sym->st_value = addr;
+			}
 		}
+		ret = apply_relocate_add(info->sechdrs, info->strtab,
+					 info->index.sym, relindex, pmod);
 	}
 
-	return 0;
+	return ret;
 }
 
 static void notrace klp_ftrace_handler(unsigned long ip,
@@ -741,12 +751,23 @@ static int klp_init_object_loaded(struct klp_patch *patch,
 				  struct klp_object *obj)
 {
 	struct klp_func *func;
+	struct module *pmod;
 	int ret;
 
-	if (obj->relocs) {
-		ret = klp_write_object_relocations(patch->mod, obj);
+	pmod = patch->mod;
+
+	if (!list_empty(&obj->reloc_secs)) {
+		set_page_attributes(pmod->module_core,
+				    pmod->module_core + pmod->core_text_size,
+				    set_memory_rw);
+
+		ret = klp_write_object_relocations(pmod, obj, patch);
 		if (ret)
 			return ret;
+
+		set_page_attributes(pmod->module_core,
+				    pmod->module_core + pmod->core_text_size,
+				    set_memory_ro);
 	}
 
 	klp_for_each_func(obj, func) {
-- 
2.4.3

[RFC PATCH 4/5] samples: livepatch: init reloc list and mark as klp module

[Jessica Yu]

Intialize the list of relocation sections in the sample
klp_object (even if the list will be empty in this case).
Also mark module as a livepatch module so that the module
loader can appropriately initialize it.

Signed-off-by: Jessica Yu <jeyu@redhat.com>
---
 samples/livepatch/livepatch-sample.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/samples/livepatch/livepatch-sample.c b/samples/livepatch/livepatch-sample.c
index fb8c861..2ef9345 100644
--- a/samples/livepatch/livepatch-sample.c
+++ b/samples/livepatch/livepatch-sample.c
@@ -57,6 +57,7 @@ static struct klp_object objs[] = {
 	{
 		/* name being NULL means vmlinux */
 		.funcs = funcs,
+		.reloc_secs = LIST_HEAD_INIT(objs[0].reloc_secs)
 	}, { }
 };
 
@@ -89,3 +90,4 @@ static void livepatch_exit(void)
 module_init(livepatch_init);
 module_exit(livepatch_exit);
 MODULE_LICENSE("GPL");
+MODULE_INFO(livepatch, "Y");
-- 
2.4.3

[RFC PATCH 5/5] livepatch: x86: remove unused relocation code

[Jessica Yu]

Architecture-specific relocation code no longer needed, since symbol
resolution and relocation work will be offloaded to module loader.

Signed-off-by: Jessica Yu <jeyu@redhat.com>
---
 arch/x86/kernel/Makefile    |  1 -
 arch/x86/kernel/livepatch.c | 91 ---------------------------------------------
 2 files changed, 92 deletions(-)
 delete mode 100644 arch/x86/kernel/livepatch.c

diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index b1b78ff..c5e9a5c 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -67,7 +67,6 @@ obj-$(CONFIG_X86_MPPARSE)	+= mpparse.o
 obj-y				+= apic/
 obj-$(CONFIG_X86_REBOOTFIXUPS)	+= reboot_fixups_32.o
 obj-$(CONFIG_DYNAMIC_FTRACE)	+= ftrace.o
-obj-$(CONFIG_LIVEPATCH)		+= livepatch.o
 obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += ftrace.o
 obj-$(CONFIG_FTRACE_SYSCALLS)	+= ftrace.o
 obj-$(CONFIG_X86_TSC)		+= trace_clock.o
diff --git a/arch/x86/kernel/livepatch.c b/arch/x86/kernel/livepatch.c
deleted file mode 100644
index d1d35cc..0000000
--- a/arch/x86/kernel/livepatch.c
+++ /dev/null
@@ -1,91 +0,0 @@
-/*
- * livepatch.c - x86-specific Kernel Live Patching Core
- *
- * Copyright (C) 2014 Seth Jennings <sjenning@redhat.com>
- * Copyright (C) 2014 SUSE
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * as published by the Free Software Foundation; either version 2
- * of the License, or (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, see <http://www.gnu.org/licenses/>.
- */
-
-#include <linux/module.h>
-#include <linux/uaccess.h>
-#include <asm/cacheflush.h>
-#include <asm/page_types.h>
-#include <asm/elf.h>
-#include <asm/livepatch.h>
-
-/**
- * klp_write_module_reloc() - write a relocation in a module
- * @mod:	module in which the section to be modified is found
- * @type:	ELF relocation type (see asm/elf.h)
- * @loc:	address that the relocation should be written to
- * @value:	relocation value (sym address + addend)
- *
- * This function writes a relocation to the specified location for
- * a particular module.
- */
-int klp_write_module_reloc(struct module *mod, unsigned long type,
-			   unsigned long loc, unsigned long value)
-{
-	int ret, numpages, size = 4;
-	bool readonly;
-	unsigned long val;
-	unsigned long core = (unsigned long)mod->module_core;
-	unsigned long core_size = mod->core_size;
-
-	switch (type) {
-	case R_X86_64_NONE:
-		return 0;
-	case R_X86_64_64:
-		val = value;
-		size = 8;
-		break;
-	case R_X86_64_32:
-		val = (u32)value;
-		break;
-	case R_X86_64_32S:
-		val = (s32)value;
-		break;
-	case R_X86_64_PC32:
-		val = (u32)(value - loc);
-		break;
-	default:
-		/* unsupported relocation type */
-		return -EINVAL;
-	}
-
-	if (loc < core || loc >= core + core_size)
-		/* loc does not point to any symbol inside the module */
-		return -EINVAL;
-
-	readonly = false;
-
-#ifdef CONFIG_DEBUG_SET_MODULE_RONX
-	if (loc < core + mod->core_ro_size)
-		readonly = true;
-#endif
-
-	/* determine if the relocation spans a page boundary */
-	numpages = ((loc & PAGE_MASK) == ((loc + size) & PAGE_MASK)) ? 1 : 2;
-
-	if (readonly)
-		set_memory_rw(loc & PAGE_MASK, numpages);
-
-	ret = probe_kernel_write((void *)loc, &val, size);
-
-	if (readonly)
-		set_memory_ro(loc & PAGE_MASK, numpages);
-
-	return ret;
-}
-- 
2.4.3

Re: [RFC PATCH 3/5] livepatch: reuse module loader code to write relocations

[Jiri Slaby]

On 11/10/2015, 05:45 AM, Jessica Yu wrote:
> @@ -95,7 +95,7 @@ struct klp_reloc {
>  struct klp_object {
>  	/* external */
>  	const char *name;
> -	struct klp_reloc *relocs;
> +	struct list_head reloc_secs;

Too early to proper review the code, but you seem you never call
INIT_LIST_HEAD for this guy.

thanks,
-- 
js
suse labs

Re: [RFC PATCH 4/5] samples: livepatch: init reloc list and mark as klp module

[Jiri Slaby]

On 11/10/2015, 05:45 AM, Jessica Yu wrote:
> Intialize the list of relocation sections in the sample
> klp_object (even if the list will be empty in this case).
> Also mark module as a livepatch module so that the module
> loader can appropriately initialize it.
> 
> Signed-off-by: Jessica Yu <jeyu@redhat.com>
> ---
>  samples/livepatch/livepatch-sample.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/samples/livepatch/livepatch-sample.c b/samples/livepatch/livepatch-sample.c
> index fb8c861..2ef9345 100644
> --- a/samples/livepatch/livepatch-sample.c
> +++ b/samples/livepatch/livepatch-sample.c
> @@ -57,6 +57,7 @@ static struct klp_object objs[] = {
>  	{
>  		/* name being NULL means vmlinux */
>  		.funcs = funcs,
> +		.reloc_secs = LIST_HEAD_INIT(objs[0].reloc_secs)

And I see now. This is not the best place for it though. I would put
INIT_LIST_HEAD in the core code instead.

thanks,
-- 
js
suse labs

Re: [RFC PATCH 4/5] samples: livepatch: init reloc list and mark as klp module

[Jiri Slaby]

On 11/10/2015, 05:45 AM, Jessica Yu wrote:
> Intialize the list of relocation sections in the sample
> klp_object (even if the list will be empty in this case).
> Also mark module as a livepatch module so that the module
> loader can appropriately initialize it.
> 
> Signed-off-by: Jessica Yu <jeyu-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> ---
>  samples/livepatch/livepatch-sample.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/samples/livepatch/livepatch-sample.c b/samples/livepatch/livepatch-sample.c
> index fb8c861..2ef9345 100644
> --- a/samples/livepatch/livepatch-sample.c
> +++ b/samples/livepatch/livepatch-sample.c
> @@ -57,6 +57,7 @@ static struct klp_object objs[] = {
>  	{
>  		/* name being NULL means vmlinux */
>  		.funcs = funcs,
> +		.reloc_secs = LIST_HEAD_INIT(objs[0].reloc_secs)

And I see now. This is not the best place for it though. I would put
INIT_LIST_HEAD in the core code instead.

thanks,
-- 
js
suse labs

Re: [RFC PATCH 4/5] samples: livepatch: init reloc list and mark as klp module

[Josh Poimboeuf]

On Tue, Nov 10, 2015 at 09:15:54AM +0100, Jiri Slaby wrote:
> On 11/10/2015, 05:45 AM, Jessica Yu wrote:
> > Intialize the list of relocation sections in the sample
> > klp_object (even if the list will be empty in this case).
> > Also mark module as a livepatch module so that the module
> > loader can appropriately initialize it.
> > 
> > Signed-off-by: Jessica Yu <jeyu@redhat.com>
> > ---
> >  samples/livepatch/livepatch-sample.c | 2 ++
> >  1 file changed, 2 insertions(+)
> > 
> > diff --git a/samples/livepatch/livepatch-sample.c b/samples/livepatch/livepatch-sample.c
> > index fb8c861..2ef9345 100644
> > --- a/samples/livepatch/livepatch-sample.c
> > +++ b/samples/livepatch/livepatch-sample.c
> > @@ -57,6 +57,7 @@ static struct klp_object objs[] = {
> >  	{
> >  		/* name being NULL means vmlinux */
> >  		.funcs = funcs,
> > +		.reloc_secs = LIST_HEAD_INIT(objs[0].reloc_secs)
> 
> And I see now. This is not the best place for it though. I would put
> INIT_LIST_HEAD in the core code instead.

Maybe the reloc_secs list should instead be an array of klp_reloc_sec
structs, with the last entry being 0, similar to how we do other "lists"
in the klp interface (e.g., klp_patch.objs, klp_object.funcs).

-- 
Josh

Re: samples: livepatch: init reloc list and mark as klp module

[Jessica Yu]

+++ Josh Poimboeuf [10/11/15 07:50 -0600]:
>On Tue, Nov 10, 2015 at 09:15:54AM +0100, Jiri Slaby wrote:
>> On 11/10/2015, 05:45 AM, Jessica Yu wrote:
>> > Intialize the list of relocation sections in the sample
>> > klp_object (even if the list will be empty in this case).
>> > Also mark module as a livepatch module so that the module
>> > loader can appropriately initialize it.
>> >
>> > Signed-off-by: Jessica Yu <jeyu@redhat.com>
>> > ---
>> >  samples/livepatch/livepatch-sample.c | 2 ++
>> >  1 file changed, 2 insertions(+)
>> >
>> > diff --git a/samples/livepatch/livepatch-sample.c b/samples/livepatch/livepatch-sample.c
>> > index fb8c861..2ef9345 100644
>> > --- a/samples/livepatch/livepatch-sample.c
>> > +++ b/samples/livepatch/livepatch-sample.c
>> > @@ -57,6 +57,7 @@ static struct klp_object objs[] = {
>> >  	{
>> >  		/* name being NULL means vmlinux */
>> >  		.funcs = funcs,
>> > +		.reloc_secs = LIST_HEAD_INIT(objs[0].reloc_secs)
>>
>> And I see now. This is not the best place for it though. I would put
>> INIT_LIST_HEAD in the core code instead.
>
>Maybe the reloc_secs list should instead be an array of klp_reloc_sec
>structs, with the last entry being 0, similar to how we do other "lists"
>in the klp interface (e.g., klp_patch.objs, klp_object.funcs).

Yeah, I admit the asymmetry is a bit awkward. Plus we can add in a
klp_for_each_reloc_sec to keep the api consistent. I'll go ahead and
change this to follow the existing klp "list" format.

Jessica

Re: samples: livepatch: init reloc list and mark as klp module

[Jessica Yu]

+++ Josh Poimboeuf [10/11/15 07:50 -0600]:
>On Tue, Nov 10, 2015 at 09:15:54AM +0100, Jiri Slaby wrote:
>> On 11/10/2015, 05:45 AM, Jessica Yu wrote:
>> > Intialize the list of relocation sections in the sample
>> > klp_object (even if the list will be empty in this case).
>> > Also mark module as a livepatch module so that the module
>> > loader can appropriately initialize it.
>> >
>> > Signed-off-by: Jessica Yu <jeyu-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>> > ---
>> >  samples/livepatch/livepatch-sample.c | 2 ++
>> >  1 file changed, 2 insertions(+)
>> >
>> > diff --git a/samples/livepatch/livepatch-sample.c b/samples/livepatch/livepatch-sample.c
>> > index fb8c861..2ef9345 100644
>> > --- a/samples/livepatch/livepatch-sample.c
>> > +++ b/samples/livepatch/livepatch-sample.c
>> > @@ -57,6 +57,7 @@ static struct klp_object objs[] = {
>> >  	{
>> >  		/* name being NULL means vmlinux */
>> >  		.funcs = funcs,
>> > +		.reloc_secs = LIST_HEAD_INIT(objs[0].reloc_secs)
>>
>> And I see now. This is not the best place for it though. I would put
>> INIT_LIST_HEAD in the core code instead.
>
>Maybe the reloc_secs list should instead be an array of klp_reloc_sec
>structs, with the last entry being 0, similar to how we do other "lists"
>in the klp interface (e.g., klp_patch.objs, klp_object.funcs).

Yeah, I admit the asymmetry is a bit awkward. Plus we can add in a
klp_for_each_reloc_sec to keep the api consistent. I'll go ahead and
change this to follow the existing klp "list" format.

Jessica

Re: [RFC PATCH 2/5] module: save load_info for livepatch modules

[Minfei Huang]

On 11/09/15 at 11:45pm, Jessica Yu wrote:
> In livepatch modules, preserve section, symbol, string information from
> the load_info struct in the module loader. This information is used to
> patch modules that are not loaded in memory yet; specifically it is used
> to resolve remaining symbols and write relocations when the target
> module loads.
> 
> Signed-off-by: Jessica Yu <jeyu@redhat.com>
> ---
>  include/linux/module.h  | 25 +++++++++++++++++++++++++
>  kernel/livepatch/core.c | 17 +++++++++++++++++
>  kernel/module.c         | 36 ++++++++++++++++++++++--------------
>  3 files changed, 64 insertions(+), 14 deletions(-)
> 
> diff --git a/include/linux/module.h b/include/linux/module.h
> index 3a19c79..c8680b1 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
> @@ -36,6 +36,20 @@ struct modversion_info {
>  	char name[MODULE_NAME_LEN];
>  };
>  
> +struct load_info {
> +	Elf_Ehdr *hdr;
> +	unsigned long len;
> +	Elf_Shdr *sechdrs;
> +	char *secstrings, *strtab;
> +	unsigned long symoffs, stroffs;
> +	struct _ddebug *debug;
> +	unsigned int num_debug;
> +	bool sig_ok;
> +	struct {
> +		unsigned int sym, str, mod, vers, info, pcpu;
> +	} index;
> +};
> +
>  struct module;
>  
>  struct module_kobject {
> @@ -462,6 +476,8 @@ struct module {
>  
>  #ifdef CONFIG_LIVEPATCH
>  	bool klp_alive;
> +	/* save info to patch to-be-loaded modules */
> +	struct load_info *info;
>  #endif
>  
>  #ifdef CONFIG_MODULE_UNLOAD
> @@ -635,6 +651,15 @@ static inline bool module_requested_async_probing(struct module *module)
>  	return module && module->async_probe_requested;
>  }
>  
> +#ifdef CONFIG_LIVEPATCH
> +extern void klp_prepare_patch_module(struct module *mod,
> +				     struct load_info *info);
> +extern int
> +apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
> +		   unsigned int symindex, unsigned int relsec,
> +		   struct module *me);
> +#endif
> +
>  #else /* !CONFIG_MODULES... */
>  
>  /* Given an address, look for it in the exception tables. */
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index 6e53441..087a8c7 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>  	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>  };
>  
> +/*
> + * Save necessary information from info in order to be able to
> + * patch modules that might be loaded later
> + */
> +void klp_prepare_patch_module(struct module *mod, struct load_info *info)
> +{
> +	Elf_Shdr *symsect;
> +
> +	symsect = info->sechdrs + info->index.sym;
> +	/* update sh_addr to point to symtab */
> +	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
> +
> +	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);

We should test the value of mod->info, since kernel may fail to allocate
the memory.

Thanks
Minfei

> +	memcpy(mod->info, info, sizeof(*info));
> +
> +}
> +
>  static int __init klp_init(void)
>  {
>  	int ret;
> diff --git a/kernel/module.c b/kernel/module.c
> index 8f051a1..8ae3ca5 100644
> --- a/kernel/module.c
> +++ b/kernel/module.c
> @@ -318,20 +318,6 @@ int unregister_module_notifier(struct notifier_block *nb)
>  }
>  EXPORT_SYMBOL(unregister_module_notifier);
>  
> -struct load_info {
> -	Elf_Ehdr *hdr;
> -	unsigned long len;
> -	Elf_Shdr *sechdrs;
> -	char *secstrings, *strtab;
> -	unsigned long symoffs, stroffs;
> -	struct _ddebug *debug;
> -	unsigned int num_debug;
> -	bool sig_ok;
> -	struct {
> -		unsigned int sym, str, mod, vers, info, pcpu;
> -	} index;
> -};
> -
>  /* We require a truly strong try_module_get(): 0 means failure due to
>     ongoing or failed initialization etc. */
>  static inline int strong_try_module_get(struct module *mod)
> @@ -2137,6 +2123,11 @@ static int simplify_symbols(struct module *mod, const struct load_info *info)
>  			       (long)sym[i].st_value);
>  			break;
>  
> +#ifdef CONFIG_LIVEPATCH
> +		case SHN_LIVEPATCH:
> +			break;
> +#endif
> +
>  		case SHN_UNDEF:
>  			ksym = resolve_symbol_wait(mod, info, name);
>  			/* Ok if resolved.  */
> @@ -2185,6 +2176,11 @@ static int apply_relocations(struct module *mod, const struct load_info *info)
>  		if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC))
>  			continue;
>  
> +#ifdef CONFIG_LIVEPATCH
> +		if (info->sechdrs[i].sh_flags & SHF_RELA_LIVEPATCH)
> +			continue;
> +#endif
> +
>  		if (info->sechdrs[i].sh_type == SHT_REL)
>  			err = apply_relocate(info->sechdrs, info->strtab,
>  					     info->index.sym, i, mod);
> @@ -3530,8 +3526,20 @@ static int load_module(struct load_info *info, const char __user *uargs,
>  	if (err < 0)
>  		goto bug_cleanup;
>  
> +#ifdef CONFIG_LIVEPATCH
> +	/*
> +	 * Save sechdrs, indices, and other data from info
> +	 * in order to patch to-be-loaded modules.
> +	 * Do not call free_copy() for livepatch modules.
> +	 */
> +	if (get_modinfo((struct load_info *)info, "livepatch"))
> +		klp_prepare_patch_module(mod, info);
> +	else
> +		free_copy(info);
> +#else
>  	/* Get rid of temporary copy. */
>  	free_copy(info);
> +#endif
>  
>  	/* Done! */
>  	trace_module_load(mod);
> -- 
> 2.4.3
> 
> --
> To unsubscribe from this list: send the line "unsubscribe live-patching" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 1/5] elf: add livepatch-specific elf constants

[Petr Mladek]

On Mon 2015-11-09 23:45:51, Jessica Yu wrote:
> Add livepatch elf reloc section flag, livepatch symbol bind
> and section index

Please, add here more detailed description from the initial mail.
It would be helpful to have it in the git history.

> Signed-off-by: Jessica Yu <jeyu@redhat.com>
> ---
>  include/uapi/linux/elf.h | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h
> index 71e1d0e..967ce1b 100644
> --- a/include/uapi/linux/elf.h
> +++ b/include/uapi/linux/elf.h
> @@ -118,6 +118,7 @@ typedef __s64	Elf64_Sxword;
>  #define STB_LOCAL  0
>  #define STB_GLOBAL 1
>  #define STB_WEAK   2
> +#define STB_LIVEPATCH_EXT 11
>  
>  #define STT_NOTYPE  0
>  #define STT_OBJECT  1
> @@ -286,6 +287,7 @@ typedef struct elf64_phdr {
>  #define SHF_ALLOC	0x2
>  #define SHF_EXECINSTR	0x4
>  #define SHF_MASKPROC	0xf0000000
> +#define SHF_RELA_LIVEPATCH  0x4000000

I would make the names ordered by the numeric values.

>  /* special section indexes */
>  #define SHN_UNDEF	0
> @@ -295,6 +297,7 @@ typedef struct elf64_phdr {
>  #define SHN_ABS		0xfff1
>  #define SHN_COMMON	0xfff2
>  #define SHN_HIRESERVE	0xffff
> +#define SHN_LIVEPATCH 0xff21

Same here.

Best Regards,
Petr

>  typedef struct elf32_shdr {
>    Elf32_Word	sh_name;
> -- 
> 2.4.3
> 
> --
> To unsubscribe from this list: send the line "unsubscribe live-patching" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] Arch-independent livepatch

[Miroslav Benes]

On Mon, 9 Nov 2015, Jessica Yu wrote:

> This patchset removes livepatch's need for architecture-specific relocation
> code by leveraging existing code in the module loader to perform
> arch-dependent work. Specifically, instead of duplicating code and
> re-implementing what the apply_relocate_add() function in the module loader
> already does in livepatch's klp_write_module_reloc(), we reuse
> apply_relocate_add() to write relocations. The hope is that this will make
> livepatch more easily portable to other architectures and greatly reduce
> the amount of arch-specific code required to port livepatch to a particular
> architecture.

Hi,

thanks for the patch set. I started going through it but it is gonna take 
some time. Nevertheless I've already found few things which I need to 
clarify. See respective patches.

> Background: Why does livepatch need to write its own relocations?
> ==
> A typical livepatch module contains patched versions of functions that can
> reference non-exported global symbols and non-included local symbols.
> Relocations referencing these types of symbols cannot be left in as-is
> since the kernel module loader cannot resolve them and will therefore
> reject the livepatch module. Furthermore, we cannot apply relocations that
> affect modules not loaded yet at run time (e.g. a patch to a driver). The
> current kpatch build system therefore solves this problem by embedding
> special "dynrela" (dynamic reloc) sections in the resulting patch module
> elf output. Using these dynrela sections, livepatch can correctly resolve
> symbols while taking into account its scope and what module the symbol
> belongs to, and then manually apply the dynamic relocations.

I'll only add that we solve the problem with kallsyms calls in kGraft. It 
can get really cumbersome from time to time, so this work would simplify 
our effort as well.

Miroslav

Re: [RFC PATCH 2/5] module: save load_info for livepatch modules

[Miroslav Benes]

On Mon, 9 Nov 2015, Jessica Yu wrote:

> diff --git a/include/linux/module.h b/include/linux/module.h
> index 3a19c79..c8680b1 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h

[...]

> +#ifdef CONFIG_LIVEPATCH
> +extern void klp_prepare_patch_module(struct module *mod,
> +				     struct load_info *info);
> +extern int
> +apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
> +		   unsigned int symindex, unsigned int relsec,
> +		   struct module *me);
> +#endif
> +
>  #else /* !CONFIG_MODULES... */

apply_relocate_add() is already in include/linux/moduleloader.h (guarded 
by CONFIG_MODULES_USE_ELF_RELA), so maybe we can just include that where 
we need it. As for the klp_prepare_patch_module() wouldn't it be better to 
have it in our livepatch.h and include that in kernel/module.c?

>  /* Given an address, look for it in the exception tables. */
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index 6e53441..087a8c7 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>  	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>  };
>  
> +/*
> + * Save necessary information from info in order to be able to
> + * patch modules that might be loaded later
> + */
> +void klp_prepare_patch_module(struct module *mod, struct load_info *info)
> +{
> +	Elf_Shdr *symsect;
> +
> +	symsect = info->sechdrs + info->index.sym;
> +	/* update sh_addr to point to symtab */
> +	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
> +
> +	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
> +	memcpy(mod->info, info, sizeof(*info));
> +
> +}

What about arch-specific 'struct mod_arch_specific'? We need to preserve 
it somewhere as well for s390x and other non-x86 architectures.

> +#ifdef CONFIG_LIVEPATCH
> +	/*
> +	 * Save sechdrs, indices, and other data from info
> +	 * in order to patch to-be-loaded modules.
> +	 * Do not call free_copy() for livepatch modules.
> +	 */
> +	if (get_modinfo((struct load_info *)info, "livepatch"))
> +		klp_prepare_patch_module(mod, info);
> +	else
> +		free_copy(info);
> +#else
>  	/* Get rid of temporary copy. */
>  	free_copy(info);
> +#endif

Maybe I am missing something but isn't it necessary to call vfree() on 
info somewhere in the end?

Regards,
Miroslav

Re: [RFC PATCH 3/5] livepatch: reuse module loader code to write relocations

[Miroslav Benes]

On Mon, 9 Nov 2015, Jessica Yu wrote:

> diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
> index 31db7a0..601e892 100644
> --- a/include/linux/livepatch.h
> +++ b/include/linux/livepatch.h
> @@ -85,7 +85,7 @@ struct klp_reloc {
>  /**
>   * struct klp_object - kernel object structure for live patching
>   * @name:	module name (or NULL for vmlinux)
> - * @relocs:	relocation entries to be applied at load time
> + * @reloc_secs:	relocation sections to be applied at load time
>   * @funcs:	function entries for functions to be patched in the object
>   * @kobj:	kobject for sysfs resources
>   * @mod:	kernel module associated with the patched object
> @@ -95,7 +95,7 @@ struct klp_reloc {
>  struct klp_object {
>  	/* external */
>  	const char *name;
> -	struct klp_reloc *relocs;
> +	struct list_head reloc_secs;
>  	struct klp_func *funcs;

So I guess we don't need klp_reloc anymore. If true, we should really 
start thinking about proper documentation because there are going to be 
plenty of assumptions about a patch module and we need to have it written 
somewhere. Especially how the relocation sections look like.

>  	/* internal */
> @@ -129,6 +129,13 @@ struct klp_patch {
>  #define klp_for_each_func(obj, func) \
>  	for (func = obj->funcs; func->old_name; func++)
>  
> +struct klp_reloc_sec {
> +	unsigned int index;
> +	char *name;
> +	char *objname;
> +	struct list_head list;
> +};

Description of the structure and its members is missing.

> diff --git a/include/linux/module.h b/include/linux/module.h
> index c8680b1..3c34eb8 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
> @@ -793,9 +793,15 @@ extern int module_sysfs_initialized;
>  #ifdef CONFIG_DEBUG_SET_MODULE_RONX
>  extern void set_all_modules_text_rw(void);
>  extern void set_all_modules_text_ro(void);
> +extern void
> +set_page_attributes(void *start, void *end,
> +		    int (*set)(unsigned long start, int num_pages));
>  #else
>  static inline void set_all_modules_text_rw(void) { }
>  static inline void set_all_modules_text_ro(void) { }
> +static inline void
> +set_page_attributes(void *start, void *end,
> +		    int (*set)(unsigned long start, int num_pages)) { }
>  #endif

This would be solved after Rusty's and Josh's patches get merged, right?

> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index 087a8c7..26c419f 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -28,6 +28,8 @@
>  #include <linux/list.h>
>  #include <linux/kallsyms.h>
>  #include <linux/livepatch.h>
> +#include <linux/elf.h>
> +#include <asm/cacheflush.h>
>  
>  /**
>   * struct klp_ops - structure for tracking registered ftrace ops structs
> @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module *pmod, const char *name,
>  }
>  
>  static int klp_write_object_relocations(struct module *pmod,
> -					struct klp_object *obj)
> +					struct klp_object *obj,
> +					struct klp_patch *patch)
>  {
> -	int ret;
> -	struct klp_reloc *reloc;
> +	int relindex, num_relas;
> +	int i, ret = 0;
> +	unsigned long addr;
> +	unsigned int bind;
> +	char *symname;
> +	struct klp_reloc_sec *reloc_sec;
> +	struct load_info *info;
> +	Elf_Rela *rela;
> +	Elf_Sym *sym, *symtab;
> +	Elf_Shdr *symsect;
>  
>  	if (WARN_ON(!klp_is_object_loaded(obj)))
>  		return -EINVAL;
>  
> -	if (WARN_ON(!obj->relocs))
> -		return -EINVAL;
> -
> -	for (reloc = obj->relocs; reloc->name; reloc++) {
> -		if (!klp_is_module(obj)) {
> -			ret = klp_verify_vmlinux_symbol(reloc->name,
> -							reloc->val);
> -			if (ret)
> -				return ret;
> -		} else {
> -			/* module, reloc->val needs to be discovered */
> -			if (reloc->external)
> -				ret = klp_find_external_symbol(pmod,
> -							       reloc->name,
> -							       &reloc->val);
> -			else
> -				ret = klp_find_object_symbol(obj->mod->name,
> -							     reloc->name,
> -							     &reloc->val);
> -			if (ret)
> -				return ret;
> -		}
> -		ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc,
> -					     reloc->val + reloc->addend);
> -		if (ret) {
> -			pr_err("relocation failed for symbol '%s' at 0x%016lx (%d)\n",
> -			       reloc->name, reloc->val, ret);
> -			return ret;
> +	info = pmod->info;
> +	symsect = info->sechdrs + info->index.sym;
> +	symtab = (void *)info->hdr + symsect->sh_offset;
> +
> +	/* For each __klp_rela section for this object */
> +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
> +		relindex = reloc_sec->index;
> +		num_relas = info->sechdrs[relindex].sh_size / sizeof(Elf_Rela);
> +		rela = (Elf_Rela *) info->sechdrs[relindex].sh_addr;
> +
> +		/* For each rela in this __klp_rela section */
> +		for (i = 0; i < num_relas; i++, rela++) {
> +			sym = symtab + ELF_R_SYM(rela->r_info);
> +			symname = info->strtab + sym->st_name;
> +			bind = ELF_ST_BIND(sym->st_info);
> +
> +			if (sym->st_shndx == SHN_LIVEPATCH) {
> +				if (bind == STB_LIVEPATCH_EXT)
> +					ret = klp_find_external_symbol(pmod, symname, &addr);
> +				else
> +					ret = klp_find_object_symbol(obj->name, symname, &addr);
> +				if (ret)
> +					return ret;
> +				sym->st_value = addr;
> +			}
>  		}
> +		ret = apply_relocate_add(info->sechdrs, info->strtab,
> +					 info->index.sym, relindex, pmod);
>  	}
>  
> -	return 0;
> +	return ret;
>  }

Looking at this... do we even need reloc_secs in klp_object? Question is 
whether we need more than one dynrela section for an object. If not then 
the binding between klp_reloc_sec and an object is the only relevant thing 
in the structure, be it index or objname. So we can replace the 
list of structures with just the index in klp_object, or get rid of it 
completely and rely on the name of dynrela section be something like 
__klp_rela_{objname}.

You see, we go through elf sections here which were preserved by module 
loader. We even have relevant sections marked with SHF_RELA_LIVEPATCH. So 
maybe all the stuff around klp_reloc_sec is not necessary.

Thoughts?

> @@ -741,12 +751,23 @@ static int klp_init_object_loaded(struct klp_patch *patch,
>  				  struct klp_object *obj)
>  {
>  	struct klp_func *func;
> +	struct module *pmod;
>  	int ret;
>  
> -	if (obj->relocs) {
> -		ret = klp_write_object_relocations(patch->mod, obj);
> +	pmod = patch->mod;
> +
> +	if (!list_empty(&obj->reloc_secs)) {
> +		set_page_attributes(pmod->module_core,
> +				    pmod->module_core + pmod->core_text_size,
> +				    set_memory_rw);
> +
> +		ret = klp_write_object_relocations(pmod, obj, patch);
>  		if (ret)
>  			return ret;
> +
> +		set_page_attributes(pmod->module_core,
> +				    pmod->module_core + pmod->core_text_size,
> +				    set_memory_ro);
>  	}

And this would get solved with different patches as well. I think the 
calls to set_page_attributes() should be hidden in 
klp_write_object_relocations() as it is in Josh's patch IIRC.

Thanks,
Miroslav

Re: [RFC PATCH 2/5] module: save load_info for livepatch modules

[Petr Mladek]

On Mon 2015-11-09 23:45:52, Jessica Yu wrote:
> In livepatch modules, preserve section, symbol, string information from
> the load_info struct in the module loader. This information is used to
> patch modules that are not loaded in memory yet; specifically it is used
> to resolve remaining symbols and write relocations when the target
> module loads.
> 
> Signed-off-by: Jessica Yu <jeyu@redhat.com>
> ---
>  include/linux/module.h  | 25 +++++++++++++++++++++++++
>  kernel/livepatch/core.c | 17 +++++++++++++++++
>  kernel/module.c         | 36 ++++++++++++++++++++++--------------
>  3 files changed, 64 insertions(+), 14 deletions(-)
> 
> diff --git a/include/linux/module.h b/include/linux/module.h
> index 3a19c79..c8680b1 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
[...]
> @@ -635,6 +651,15 @@ static inline bool module_requested_async_probing(struct module *module)
>  	return module && module->async_probe_requested;
>  }
>  
> +#ifdef CONFIG_LIVEPATCH
> +extern void klp_prepare_patch_module(struct module *mod,
> +				     struct load_info *info);
> +extern int
> +apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
> +		   unsigned int symindex, unsigned int relsec,
> +		   struct module *me);
> +#endif

This function is already declared in moduleloader.h.
It is implemted only when CONFIG_MODULES_USE_ELF_RELA is defined.

I guess that we want to include moduleloader.h in livepatch.

> +
>  #else /* !CONFIG_MODULES... */
>  
>  /* Given an address, look for it in the exception tables. */
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index 6e53441..087a8c7 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>  	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>  };
>  
> +/*
> + * Save necessary information from info in order to be able to
> + * patch modules that might be loaded later
> + */
> +void klp_prepare_patch_module(struct module *mod, struct load_info *info)
> +{
> +	Elf_Shdr *symsect;
> +
> +	symsect = info->sechdrs + info->index.sym;
> +	/* update sh_addr to point to symtab */
> +	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;

Is livepatch the only user of this value? By other words, is this safe?

> +	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
> +	memcpy(mod->info, info, sizeof(*info));
> +
> +}

It is strange that this funtion is defined in livepatch/core.c
but declared in module.h. I would move the definition to
module.c.

>  static int __init klp_init(void)
>  {
>  	int ret;
> diff --git a/kernel/module.c b/kernel/module.c
> index 8f051a1..8ae3ca5 100644
> --- a/kernel/module.c
> +++ b/kernel/module.c
> @@ -318,20 +318,6 @@ int unregister_module_notifier(struct notifier_block *nb)
>  }
>  EXPORT_SYMBOL(unregister_module_notifier);
>  
> -struct load_info {
> -	Elf_Ehdr *hdr;
> -	unsigned long len;
> -	Elf_Shdr *sechdrs;
> -	char *secstrings, *strtab;
> -	unsigned long symoffs, stroffs;
> -	struct _ddebug *debug;
> -	unsigned int num_debug;
> -	bool sig_ok;
> -	struct {
> -		unsigned int sym, str, mod, vers, info, pcpu;
> -	} index;
> -};
> -
>  /* We require a truly strong try_module_get(): 0 means failure due to
>     ongoing or failed initialization etc. */
>  static inline int strong_try_module_get(struct module *mod)
> @@ -2137,6 +2123,11 @@ static int simplify_symbols(struct module *mod, const struct load_info *info)
>  			       (long)sym[i].st_value);
>  			break;
>  
> +#ifdef CONFIG_LIVEPATCH
> +		case SHN_LIVEPATCH:
> +			break;
> +#endif

IMHO, even a kernel compiled without CONFIG_LIVEPATCH should handle livepatch
modules with grace. It means to reject loading.

>  		case SHN_UNDEF:
>  			ksym = resolve_symbol_wait(mod, info, name);
>  			/* Ok if resolved.  */
> @@ -2185,6 +2176,11 @@ static int apply_relocations(struct module *mod, const struct load_info *info)
>  		if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC))
>  			continue;
>  
> +#ifdef CONFIG_LIVEPATCH
> +		if (info->sechdrs[i].sh_flags & SHF_RELA_LIVEPATCH)
> +			continue;
> +#endif
> +
>  		if (info->sechdrs[i].sh_type == SHT_REL)
>  			err = apply_relocate(info->sechdrs, info->strtab,
>  					     info->index.sym, i, mod);
> @@ -3530,8 +3526,20 @@ static int load_module(struct load_info *info, const char __user *uargs,
>  	if (err < 0)
>  		goto bug_cleanup;
>  
> +#ifdef CONFIG_LIVEPATCH
> +	/*
> +	 * Save sechdrs, indices, and other data from info
> +	 * in order to patch to-be-loaded modules.
> +	 * Do not call free_copy() for livepatch modules.
> +	 */
> +	if (get_modinfo((struct load_info *)info, "livepatch"))
> +		klp_prepare_patch_module(mod, info);
> +	else
> +		free_copy(info);
> +#else

I would move this #else one line above and get rid of the
double free_copy(info); But it is a matter of taste.


>  	/* Get rid of temporary copy. */
>  	free_copy(info);
> +#endif

Best Regards,
Petr

Re: [RFC PATCH 2/5] module: save load_info for livepatch modules

[Petr Mladek]

On Mon 2015-11-09 23:45:52, Jessica Yu wrote:
> In livepatch modules, preserve section, symbol, string information from
> the load_info struct in the module loader. This information is used to
> patch modules that are not loaded in memory yet; specifically it is used
> to resolve remaining symbols and write relocations when the target
> module loads.
> 
> Signed-off-by: Jessica Yu <jeyu-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> ---
>  include/linux/module.h  | 25 +++++++++++++++++++++++++
>  kernel/livepatch/core.c | 17 +++++++++++++++++
>  kernel/module.c         | 36 ++++++++++++++++++++++--------------
>  3 files changed, 64 insertions(+), 14 deletions(-)
> 
> diff --git a/include/linux/module.h b/include/linux/module.h
> index 3a19c79..c8680b1 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
[...]
> @@ -635,6 +651,15 @@ static inline bool module_requested_async_probing(struct module *module)
>  	return module && module->async_probe_requested;
>  }
>  
> +#ifdef CONFIG_LIVEPATCH
> +extern void klp_prepare_patch_module(struct module *mod,
> +				     struct load_info *info);
> +extern int
> +apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
> +		   unsigned int symindex, unsigned int relsec,
> +		   struct module *me);
> +#endif

This function is already declared in moduleloader.h.
It is implemted only when CONFIG_MODULES_USE_ELF_RELA is defined.

I guess that we want to include moduleloader.h in livepatch.

> +
>  #else /* !CONFIG_MODULES... */
>  
>  /* Given an address, look for it in the exception tables. */
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index 6e53441..087a8c7 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>  	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>  };
>  
> +/*
> + * Save necessary information from info in order to be able to
> + * patch modules that might be loaded later
> + */
> +void klp_prepare_patch_module(struct module *mod, struct load_info *info)
> +{
> +	Elf_Shdr *symsect;
> +
> +	symsect = info->sechdrs + info->index.sym;
> +	/* update sh_addr to point to symtab */
> +	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;

Is livepatch the only user of this value? By other words, is this safe?

> +	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
> +	memcpy(mod->info, info, sizeof(*info));
> +
> +}

It is strange that this funtion is defined in livepatch/core.c
but declared in module.h. I would move the definition to
module.c.

>  static int __init klp_init(void)
>  {
>  	int ret;
> diff --git a/kernel/module.c b/kernel/module.c
> index 8f051a1..8ae3ca5 100644
> --- a/kernel/module.c
> +++ b/kernel/module.c
> @@ -318,20 +318,6 @@ int unregister_module_notifier(struct notifier_block *nb)
>  }
>  EXPORT_SYMBOL(unregister_module_notifier);
>  
> -struct load_info {
> -	Elf_Ehdr *hdr;
> -	unsigned long len;
> -	Elf_Shdr *sechdrs;
> -	char *secstrings, *strtab;
> -	unsigned long symoffs, stroffs;
> -	struct _ddebug *debug;
> -	unsigned int num_debug;
> -	bool sig_ok;
> -	struct {
> -		unsigned int sym, str, mod, vers, info, pcpu;
> -	} index;
> -};
> -
>  /* We require a truly strong try_module_get(): 0 means failure due to
>     ongoing or failed initialization etc. */
>  static inline int strong_try_module_get(struct module *mod)
> @@ -2137,6 +2123,11 @@ static int simplify_symbols(struct module *mod, const struct load_info *info)
>  			       (long)sym[i].st_value);
>  			break;
>  
> +#ifdef CONFIG_LIVEPATCH
> +		case SHN_LIVEPATCH:
> +			break;
> +#endif

IMHO, even a kernel compiled without CONFIG_LIVEPATCH should handle livepatch
modules with grace. It means to reject loading.

>  		case SHN_UNDEF:
>  			ksym = resolve_symbol_wait(mod, info, name);
>  			/* Ok if resolved.  */
> @@ -2185,6 +2176,11 @@ static int apply_relocations(struct module *mod, const struct load_info *info)
>  		if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC))
>  			continue;
>  
> +#ifdef CONFIG_LIVEPATCH
> +		if (info->sechdrs[i].sh_flags & SHF_RELA_LIVEPATCH)
> +			continue;
> +#endif
> +
>  		if (info->sechdrs[i].sh_type == SHT_REL)
>  			err = apply_relocate(info->sechdrs, info->strtab,
>  					     info->index.sym, i, mod);
> @@ -3530,8 +3526,20 @@ static int load_module(struct load_info *info, const char __user *uargs,
>  	if (err < 0)
>  		goto bug_cleanup;
>  
> +#ifdef CONFIG_LIVEPATCH
> +	/*
> +	 * Save sechdrs, indices, and other data from info
> +	 * in order to patch to-be-loaded modules.
> +	 * Do not call free_copy() for livepatch modules.
> +	 */
> +	if (get_modinfo((struct load_info *)info, "livepatch"))
> +		klp_prepare_patch_module(mod, info);
> +	else
> +		free_copy(info);
> +#else

I would move this #else one line above and get rid of the
double free_copy(info); But it is a matter of taste.


>  	/* Get rid of temporary copy. */
>  	free_copy(info);
> +#endif

Best Regards,
Petr

Re: [RFC PATCH 3/5] livepatch: reuse module loader code to write relocations

[Petr Mladek]

On Mon 2015-11-09 23:45:53, Jessica Yu wrote:
> Reuse module loader code to write relocations, thereby eliminating the
> need for architecture specific code in livepatch. Namely, we reuse
> apply_relocate_add() in the module loader to write relocs instead of
> duplicating functionality in livepatch's klp_write_module_reloc(). To
> apply relocation sections, remaining SHN_LIVEPATCH symbols referenced by
> relocs are resolved and then apply_relocate_add() is called to apply
> those relocations.
> 
> Signed-off-by: Jessica Yu <jeyu@redhat.com>
> ---
>  include/linux/livepatch.h | 11 ++++--
>  include/linux/module.h    |  6 ++++
>  kernel/livepatch/core.c   | 89 +++++++++++++++++++++++++++++------------------
>  3 files changed, 70 insertions(+), 36 deletions(-)
> 
> index 087a8c7..26c419f 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -28,6 +28,8 @@
>  #include <linux/list.h>
>  #include <linux/kallsyms.h>
>  #include <linux/livepatch.h>
> +#include <linux/elf.h>
> +#include <asm/cacheflush.h>
>  
>  /**
>   * struct klp_ops - structure for tracking registered ftrace ops structs
> @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module *pmod, const char *name,
>  }
>  
>  static int klp_write_object_relocations(struct module *pmod,
> -					struct klp_object *obj)
> +					struct klp_object *obj,
> +					struct klp_patch *patch)
>  {
[...]

> +
> +	/* For each __klp_rela section for this object */
> +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {

Who, when, and how will define reloc_secs, please?

I guess that it is just an optimization. It helps to avoid going
through all elf sections of all livepatch modules. Therefore, I think
that we might fill this when the livepatch module is loaded. But
I do not see the code for this.

Best Regards,
Petr

Re: [RFC PATCH 4/5] samples: livepatch: init reloc list and mark as klp module

[Petr Mladek]

On Mon 2015-11-09 23:45:54, Jessica Yu wrote:
> Intialize the list of relocation sections in the sample
> klp_object (even if the list will be empty in this case).
> Also mark module as a livepatch module so that the module
> loader can appropriately initialize it.
> 
> Signed-off-by: Jessica Yu <jeyu@redhat.com>
> ---
>  samples/livepatch/livepatch-sample.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/samples/livepatch/livepatch-sample.c b/samples/livepatch/livepatch-sample.c
> index fb8c861..2ef9345 100644
> --- a/samples/livepatch/livepatch-sample.c
> +++ b/samples/livepatch/livepatch-sample.c
> @@ -89,3 +90,4 @@ static void livepatch_exit(void)
>  module_init(livepatch_init);
>  module_exit(livepatch_exit);
>  MODULE_LICENSE("GPL");
> +MODULE_INFO(livepatch, "Y");

This looks a bit error prone. I wonder if we could detect this
information another way. For example, by a check for the
livepatch-related elf sections. If it is missing,
we do not need to preserve struct load_info even
when it is a livepatch.

Best Regards,
Petr

Re: [RFC PATCH 4/5] samples: livepatch: init reloc list and mark as klp module

[Petr Mladek]

On Mon 2015-11-09 23:45:54, Jessica Yu wrote:
> Intialize the list of relocation sections in the sample
> klp_object (even if the list will be empty in this case).
> Also mark module as a livepatch module so that the module
> loader can appropriately initialize it.
> 
> Signed-off-by: Jessica Yu <jeyu-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> ---
>  samples/livepatch/livepatch-sample.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/samples/livepatch/livepatch-sample.c b/samples/livepatch/livepatch-sample.c
> index fb8c861..2ef9345 100644
> --- a/samples/livepatch/livepatch-sample.c
> +++ b/samples/livepatch/livepatch-sample.c
> @@ -89,3 +90,4 @@ static void livepatch_exit(void)
>  module_init(livepatch_init);
>  module_exit(livepatch_exit);
>  MODULE_LICENSE("GPL");
> +MODULE_INFO(livepatch, "Y");

This looks a bit error prone. I wonder if we could detect this
information another way. For example, by a check for the
livepatch-related elf sections. If it is missing,
we do not need to preserve struct load_info even
when it is a livepatch.

Best Regards,
Petr

Re: [RFC PATCH 5/5] livepatch: x86: remove unused relocation code

[Petr Mladek]

On Mon 2015-11-09 23:45:55, Jessica Yu wrote:
> Architecture-specific relocation code no longer needed, since symbol
> resolution and relocation work will be offloaded to module loader.
                                 ^^^^
				 has been

I would personally merge this with the 3rd patch. It will be more
clear what code has been replaced there. But it is a matter of taste.

Best Regards,
Petr

Re: [RFC PATCH 0/5] Arch-independent livepatch

[Josh Poimboeuf]

On Wed, Nov 11, 2015 at 03:00:44PM +0100, Miroslav Benes wrote:
> > Background: Why does livepatch need to write its own relocations?
> > ==
> > A typical livepatch module contains patched versions of functions that can
> > reference non-exported global symbols and non-included local symbols.
> > Relocations referencing these types of symbols cannot be left in as-is
> > since the kernel module loader cannot resolve them and will therefore
> > reject the livepatch module. Furthermore, we cannot apply relocations that
> > affect modules not loaded yet at run time (e.g. a patch to a driver). The
> > current kpatch build system therefore solves this problem by embedding
> > special "dynrela" (dynamic reloc) sections in the resulting patch module
> > elf output. Using these dynrela sections, livepatch can correctly resolve
> > symbols while taking into account its scope and what module the symbol
> > belongs to, and then manually apply the dynamic relocations.
> 
> I'll only add that we solve the problem with kallsyms calls in kGraft. It 
> can get really cumbersome from time to time, so this work would simplify 
> our effort as well.

I haven't yet reviewed the code in-depth, but overall I think this patch
set is a good idea.  It simplifies livepatch itself as well as the patch
creation process (for both kGraft and kpatch-build), so it's a win-win.

-- 
Josh

Re: livepatch: reuse module loader code to write relocations

[Jessica Yu]

+++ Petr Mladek [11/11/15 16:22 +0100]:
>On Mon 2015-11-09 23:45:53, Jessica Yu wrote:
>> Reuse module loader code to write relocations, thereby eliminating the
>> need for architecture specific code in livepatch. Namely, we reuse
>> apply_relocate_add() in the module loader to write relocs instead of
>> duplicating functionality in livepatch's klp_write_module_reloc(). To
>> apply relocation sections, remaining SHN_LIVEPATCH symbols referenced by
>> relocs are resolved and then apply_relocate_add() is called to apply
>> those relocations.
>>
>> Signed-off-by: Jessica Yu <jeyu@redhat.com>
>> ---
>>  include/linux/livepatch.h | 11 ++++--
>>  include/linux/module.h    |  6 ++++
>>  kernel/livepatch/core.c   | 89 +++++++++++++++++++++++++++++------------------
>>  3 files changed, 70 insertions(+), 36 deletions(-)
>>
>> index 087a8c7..26c419f 100644
>> --- a/kernel/livepatch/core.c
>> +++ b/kernel/livepatch/core.c
>> @@ -28,6 +28,8 @@
>>  #include <linux/list.h>
>>  #include <linux/kallsyms.h>
>>  #include <linux/livepatch.h>
>> +#include <linux/elf.h>
>> +#include <asm/cacheflush.h>
>>
>>  /**
>>   * struct klp_ops - structure for tracking registered ftrace ops structs
>> @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module *pmod, const char *name,
>>  }
>>
>>  static int klp_write_object_relocations(struct module *pmod,
>> -					struct klp_object *obj)
>> +					struct klp_object *obj,
>> +					struct klp_patch *patch)
>>  {
>[...]
>
>> +
>> +	/* For each __klp_rela section for this object */
>> +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
>
>Who, when, and how will define reloc_secs, please?
>
>I guess that it is just an optimization. It helps to avoid going
>through all elf sections of all livepatch modules. Therefore, I think
>that we might fill this when the livepatch module is loaded. But
>I do not see the code for this.

Thanks for bringing this up, I admit that from this patchset it is
unclear where and how the reloc_secs list is built. 

Basically, the patch module code is expected to build the reloc_secs
list for each object that is being patched. For example in kpatch, the
patch module generates this list in patch_init(). Like you guessed, it
does go through all the elf sections of the patch module to find the
reloc sections marked SHF_RELA_LIVEPATCH. We are able to access these
sections through module->info, which is set up for livepatch modules
before the module loader calls do_init_module() (and hence before
patch_init(), See patch 2/5).
See below for an example of how the reloc_secs list might be built: 
https://github.com/flaming-toast/kpatch/blob/no_dynrela_redux/kmod/patch/livepatch-patch-hook.c#L213

Once the patch module has built this list, it is good to go for use in
livepatch core. All livepatch has to do then is to iterate though the
list, resolve any outstanding symbols, and call apply_relocate_add(),
as shown in this patch.

Miroslav mentioned in another email that we should start thinking
about including documentation about this, including the expected patch
module format. So perhaps v2 should include some documentation about
this whole process somewhere.

Please let me know if anything else is unclear.

Thanks,

Jessica

Re: livepatch: reuse module loader code to write relocations

[Jessica Yu]

+++ Petr Mladek [11/11/15 16:22 +0100]:
>On Mon 2015-11-09 23:45:53, Jessica Yu wrote:
>> Reuse module loader code to write relocations, thereby eliminating the
>> need for architecture specific code in livepatch. Namely, we reuse
>> apply_relocate_add() in the module loader to write relocs instead of
>> duplicating functionality in livepatch's klp_write_module_reloc(). To
>> apply relocation sections, remaining SHN_LIVEPATCH symbols referenced by
>> relocs are resolved and then apply_relocate_add() is called to apply
>> those relocations.
>>
>> Signed-off-by: Jessica Yu <jeyu-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>> ---
>>  include/linux/livepatch.h | 11 ++++--
>>  include/linux/module.h    |  6 ++++
>>  kernel/livepatch/core.c   | 89 +++++++++++++++++++++++++++++------------------
>>  3 files changed, 70 insertions(+), 36 deletions(-)
>>
>> index 087a8c7..26c419f 100644
>> --- a/kernel/livepatch/core.c
>> +++ b/kernel/livepatch/core.c
>> @@ -28,6 +28,8 @@
>>  #include <linux/list.h>
>>  #include <linux/kallsyms.h>
>>  #include <linux/livepatch.h>
>> +#include <linux/elf.h>
>> +#include <asm/cacheflush.h>
>>
>>  /**
>>   * struct klp_ops - structure for tracking registered ftrace ops structs
>> @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module *pmod, const char *name,
>>  }
>>
>>  static int klp_write_object_relocations(struct module *pmod,
>> -					struct klp_object *obj)
>> +					struct klp_object *obj,
>> +					struct klp_patch *patch)
>>  {
>[...]
>
>> +
>> +	/* For each __klp_rela section for this object */
>> +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
>
>Who, when, and how will define reloc_secs, please?
>
>I guess that it is just an optimization. It helps to avoid going
>through all elf sections of all livepatch modules. Therefore, I think
>that we might fill this when the livepatch module is loaded. But
>I do not see the code for this.

Thanks for bringing this up, I admit that from this patchset it is
unclear where and how the reloc_secs list is built. 

Basically, the patch module code is expected to build the reloc_secs
list for each object that is being patched. For example in kpatch, the
patch module generates this list in patch_init(). Like you guessed, it
does go through all the elf sections of the patch module to find the
reloc sections marked SHF_RELA_LIVEPATCH. We are able to access these
sections through module->info, which is set up for livepatch modules
before the module loader calls do_init_module() (and hence before
patch_init(), See patch 2/5).
See below for an example of how the reloc_secs list might be built: 
https://github.com/flaming-toast/kpatch/blob/no_dynrela_redux/kmod/patch/livepatch-patch-hook.c#L213

Once the patch module has built this list, it is good to go for use in
livepatch core. All livepatch has to do then is to iterate though the
list, resolve any outstanding symbols, and call apply_relocate_add(),
as shown in this patch.

Miroslav mentioned in another email that we should start thinking
about including documentation about this, including the expected patch
module format. So perhaps v2 should include some documentation about
this whole process somewhere.

Please let me know if anything else is unclear.

Thanks,

Jessica

Re: livepatch: reuse module loader code to write relocations

[Jessica Yu]

+++ Miroslav Benes [11/11/15 15:30 +0100]:
>On Mon, 9 Nov 2015, Jessica Yu wrote:
>
>> diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
>> index 31db7a0..601e892 100644
>> --- a/include/linux/livepatch.h
>> +++ b/include/linux/livepatch.h
>> @@ -85,7 +85,7 @@ struct klp_reloc {
>>  /**
>>   * struct klp_object - kernel object structure for live patching
>>   * @name:	module name (or NULL for vmlinux)
>> - * @relocs:	relocation entries to be applied at load time
>> + * @reloc_secs:	relocation sections to be applied at load time
>>   * @funcs:	function entries for functions to be patched in the object
>>   * @kobj:	kobject for sysfs resources
>>   * @mod:	kernel module associated with the patched object
>> @@ -95,7 +95,7 @@ struct klp_reloc {
>>  struct klp_object {
>>  	/* external */
>>  	const char *name;
>> -	struct klp_reloc *relocs;
>> +	struct list_head reloc_secs;
>>  	struct klp_func *funcs;
>
>So I guess we don't need klp_reloc anymore.

Yes, that's correct. I am noticing just now that I forgot to remove
the klp_reloc struct definition from livepatch.h. That change will be
reflected in v2...

>If true, we should really
>start thinking about proper documentation because there are going to be
>plenty of assumptions about a patch module and we need to have it written
>somewhere. Especially how the relocation sections look like.

Agreed. As a first step the patch module format can perhaps be
documented somewhere. Perhaps it's time we create
Documentation/livepatch/? :-)

>>  	/* internal */
>> @@ -129,6 +129,13 @@ struct klp_patch {
>>  #define klp_for_each_func(obj, func) \
>>  	for (func = obj->funcs; func->old_name; func++)
>>
>> +struct klp_reloc_sec {
>> +	unsigned int index;
>> +	char *name;
>> +	char *objname;
>> +	struct list_head list;
>> +};
>
>Description of the structure and its members is missing.
>
>> diff --git a/include/linux/module.h b/include/linux/module.h
>> index c8680b1..3c34eb8 100644
>> --- a/include/linux/module.h
>> +++ b/include/linux/module.h
>> @@ -793,9 +793,15 @@ extern int module_sysfs_initialized;
>>  #ifdef CONFIG_DEBUG_SET_MODULE_RONX
>>  extern void set_all_modules_text_rw(void);
>>  extern void set_all_modules_text_ro(void);
>> +extern void
>> +set_page_attributes(void *start, void *end,
>> +		    int (*set)(unsigned long start, int num_pages));
>>  #else
>>  static inline void set_all_modules_text_rw(void) { }
>>  static inline void set_all_modules_text_ro(void) { }
>> +static inline void
>> +set_page_attributes(void *start, void *end,
>> +		    int (*set)(unsigned long start, int num_pages)) { }
>>  #endif
>
>This would be solved after Rusty's and Josh's patches get merged, right?

Yes, correct. When Josh and Rusty's patches get merged, I'll base
subsequent versions of this patchset on them.

>> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>> index 087a8c7..26c419f 100644
>> --- a/kernel/livepatch/core.c
>> +++ b/kernel/livepatch/core.c
>> @@ -28,6 +28,8 @@
>>  #include <linux/list.h>
>>  #include <linux/kallsyms.h>
>>  #include <linux/livepatch.h>
>> +#include <linux/elf.h>
>> +#include <asm/cacheflush.h>
>>
>>  /**
>>   * struct klp_ops - structure for tracking registered ftrace ops structs
>> @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module *pmod, const char *name,
>>  }
>>
>>  static int klp_write_object_relocations(struct module *pmod,
>> -					struct klp_object *obj)
>> +					struct klp_object *obj,
>> +					struct klp_patch *patch)
>>  {
>> -	int ret;
>> -	struct klp_reloc *reloc;
>> +	int relindex, num_relas;
>> +	int i, ret = 0;
>> +	unsigned long addr;
>> +	unsigned int bind;
>> +	char *symname;
>> +	struct klp_reloc_sec *reloc_sec;
>> +	struct load_info *info;
>> +	Elf_Rela *rela;
>> +	Elf_Sym *sym, *symtab;
>> +	Elf_Shdr *symsect;
>>
>>  	if (WARN_ON(!klp_is_object_loaded(obj)))
>>  		return -EINVAL;
>>
>> -	if (WARN_ON(!obj->relocs))
>> -		return -EINVAL;
>> -
>> -	for (reloc = obj->relocs; reloc->name; reloc++) {
>> -		if (!klp_is_module(obj)) {
>> -			ret = klp_verify_vmlinux_symbol(reloc->name,
>> -							reloc->val);
>> -			if (ret)
>> -				return ret;
>> -		} else {
>> -			/* module, reloc->val needs to be discovered */
>> -			if (reloc->external)
>> -				ret = klp_find_external_symbol(pmod,
>> -							       reloc->name,
>> -							       &reloc->val);
>> -			else
>> -				ret = klp_find_object_symbol(obj->mod->name,
>> -							     reloc->name,
>> -							     &reloc->val);
>> -			if (ret)
>> -				return ret;
>> -		}
>> -		ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc,
>> -					     reloc->val + reloc->addend);
>> -		if (ret) {
>> -			pr_err("relocation failed for symbol '%s' at 0x%016lx (%d)\n",
>> -			       reloc->name, reloc->val, ret);
>> -			return ret;
>> +	info = pmod->info;
>> +	symsect = info->sechdrs + info->index.sym;
>> +	symtab = (void *)info->hdr + symsect->sh_offset;
>> +
>> +	/* For each __klp_rela section for this object */
>> +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
>> +		relindex = reloc_sec->index;
>> +		num_relas = info->sechdrs[relindex].sh_size / sizeof(Elf_Rela);
>> +		rela = (Elf_Rela *) info->sechdrs[relindex].sh_addr;
>> +
>> +		/* For each rela in this __klp_rela section */
>> +		for (i = 0; i < num_relas; i++, rela++) {
>> +			sym = symtab + ELF_R_SYM(rela->r_info);
>> +			symname = info->strtab + sym->st_name;
>> +			bind = ELF_ST_BIND(sym->st_info);
>> +
>> +			if (sym->st_shndx == SHN_LIVEPATCH) {
>> +				if (bind == STB_LIVEPATCH_EXT)
>> +					ret = klp_find_external_symbol(pmod, symname, &addr);
>> +				else
>> +					ret = klp_find_object_symbol(obj->name, symname, &addr);
>> +				if (ret)
>> +					return ret;
>> +				sym->st_value = addr;
>> +			}
>>  		}
>> +		ret = apply_relocate_add(info->sechdrs, info->strtab,
>> +					 info->index.sym, relindex, pmod);
>>  	}
>>
>> -	return 0;
>> +	return ret;
>>  }
>
>Looking at this... do we even need reloc_secs in klp_object? Question is
>whether we need more than one dynrela section for an object. If not then
>the binding between klp_reloc_sec and an object is the only relevant thing
>in the structure, be it index or objname. So we can replace the
>list of structures with just the index in klp_object, or get rid of it
>completely and rely on the name of dynrela section be something like
>__klp_rela_{objname}.

Hm, you bring up a good point. I think theoretically yes, it is
possible to just have one klp_reloc_sec for each object and therefore
a list is not required (I have not checked yet how difficult it would
be to implement this on the kpatch-build side of things).  However,
considering the final format of the patch module, I think it is
semantically clearer to leave it as a list, and for each object to
possibly have more than one __klp_rela section.

For example, say we are patching two functions in ext4. In my
resulting kpatch module I will have two __klp_rela_ext4 sections, and
they might look like this when we run readelf --sections:

[34] __klp_rela_ext4.text.ext4_attr_store RELA ...
[35] __klp_rela_ext4.text.ext4_attr_show RELA ...

Then these two klp rela sections end up as two elements in the
reloc_secs list for the ext4 patch object. I think this way, we can
better tell which rela is being applied to what function. Might be
easier to understand what's happening from the developer's point of
view.

>You see, we go through elf sections here which were preserved by module
>loader. We even have relevant sections marked with SHF_RELA_LIVEPATCH. So
>maybe all the stuff around klp_reloc_sec is not necessary.
>
>Thoughts?

Ah, so this is where descriptive comments and documentation might have
been useful :-) So I think we will still need to keep the
klp_reloc_sec struct to help the patch module initialize. Though the
name and objname fields aren't used in this patchset, they are used in
the kpatch patch module code [1], where we iterate through each elf
section, find the ones marked with SHF_RELA_LIVEPATCH, set the
klp_reloc_sec's objname (which we find from the "name" field,
formatted as __klp_rela_{objname}.text..). Once we have the objname
set, we can then find the object to attach the reloc_sec to (i.e. add
it to its list of reloc_secs).

Hope that clears some things up.


[1] https://github.com/flaming-toast/kpatch/blob/no_dynrela_redux/kmod/patch/livepatch-patch-hook.c#L213

>> @@ -741,12 +751,23 @@ static int klp_init_object_loaded(struct klp_patch *patch,
>>  				  struct klp_object *obj)
>>  {
>>  	struct klp_func *func;
>> +	struct module *pmod;
>>  	int ret;
>>
>> -	if (obj->relocs) {
>> -		ret = klp_write_object_relocations(patch->mod, obj);
>> +	pmod = patch->mod;
>> +
>> +	if (!list_empty(&obj->reloc_secs)) {
>> +		set_page_attributes(pmod->module_core,
>> +				    pmod->module_core + pmod->core_text_size,
>> +				    set_memory_rw);
>> +
>> +		ret = klp_write_object_relocations(pmod, obj, patch);
>>  		if (ret)
>>  			return ret;
>> +
>> +		set_page_attributes(pmod->module_core,
>> +				    pmod->module_core + pmod->core_text_size,
>> +				    set_memory_ro);
>>  	}
>
>And this would get solved with different patches as well. I think the
>calls to set_page_attributes() should be hidden in
>klp_write_object_relocations() as it is in Josh's patch IIRC.

Yes, when those patches are merged I will switch over to Josh's
functions for setting/unsetting memory ro.

Thanks,
Jessica

Re: livepatch: reuse module loader code to write relocations

[Jessica Yu]

+++ Miroslav Benes [11/11/15 15:30 +0100]:
>On Mon, 9 Nov 2015, Jessica Yu wrote:
>
>> diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
>> index 31db7a0..601e892 100644
>> --- a/include/linux/livepatch.h
>> +++ b/include/linux/livepatch.h
>> @@ -85,7 +85,7 @@ struct klp_reloc {
>>  /**
>>   * struct klp_object - kernel object structure for live patching
>>   * @name:	module name (or NULL for vmlinux)
>> - * @relocs:	relocation entries to be applied at load time
>> + * @reloc_secs:	relocation sections to be applied at load time
>>   * @funcs:	function entries for functions to be patched in the object
>>   * @kobj:	kobject for sysfs resources
>>   * @mod:	kernel module associated with the patched object
>> @@ -95,7 +95,7 @@ struct klp_reloc {
>>  struct klp_object {
>>  	/* external */
>>  	const char *name;
>> -	struct klp_reloc *relocs;
>> +	struct list_head reloc_secs;
>>  	struct klp_func *funcs;
>
>So I guess we don't need klp_reloc anymore.

Yes, that's correct. I am noticing just now that I forgot to remove
the klp_reloc struct definition from livepatch.h. That change will be
reflected in v2...

>If true, we should really
>start thinking about proper documentation because there are going to be
>plenty of assumptions about a patch module and we need to have it written
>somewhere. Especially how the relocation sections look like.

Agreed. As a first step the patch module format can perhaps be
documented somewhere. Perhaps it's time we create
Documentation/livepatch/? :-)

>>  	/* internal */
>> @@ -129,6 +129,13 @@ struct klp_patch {
>>  #define klp_for_each_func(obj, func) \
>>  	for (func = obj->funcs; func->old_name; func++)
>>
>> +struct klp_reloc_sec {
>> +	unsigned int index;
>> +	char *name;
>> +	char *objname;
>> +	struct list_head list;
>> +};
>
>Description of the structure and its members is missing.
>
>> diff --git a/include/linux/module.h b/include/linux/module.h
>> index c8680b1..3c34eb8 100644
>> --- a/include/linux/module.h
>> +++ b/include/linux/module.h
>> @@ -793,9 +793,15 @@ extern int module_sysfs_initialized;
>>  #ifdef CONFIG_DEBUG_SET_MODULE_RONX
>>  extern void set_all_modules_text_rw(void);
>>  extern void set_all_modules_text_ro(void);
>> +extern void
>> +set_page_attributes(void *start, void *end,
>> +		    int (*set)(unsigned long start, int num_pages));
>>  #else
>>  static inline void set_all_modules_text_rw(void) { }
>>  static inline void set_all_modules_text_ro(void) { }
>> +static inline void
>> +set_page_attributes(void *start, void *end,
>> +		    int (*set)(unsigned long start, int num_pages)) { }
>>  #endif
>
>This would be solved after Rusty's and Josh's patches get merged, right?

Yes, correct. When Josh and Rusty's patches get merged, I'll base
subsequent versions of this patchset on them.

>> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>> index 087a8c7..26c419f 100644
>> --- a/kernel/livepatch/core.c
>> +++ b/kernel/livepatch/core.c
>> @@ -28,6 +28,8 @@
>>  #include <linux/list.h>
>>  #include <linux/kallsyms.h>
>>  #include <linux/livepatch.h>
>> +#include <linux/elf.h>
>> +#include <asm/cacheflush.h>
>>
>>  /**
>>   * struct klp_ops - structure for tracking registered ftrace ops structs
>> @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module *pmod, const char *name,
>>  }
>>
>>  static int klp_write_object_relocations(struct module *pmod,
>> -					struct klp_object *obj)
>> +					struct klp_object *obj,
>> +					struct klp_patch *patch)
>>  {
>> -	int ret;
>> -	struct klp_reloc *reloc;
>> +	int relindex, num_relas;
>> +	int i, ret = 0;
>> +	unsigned long addr;
>> +	unsigned int bind;
>> +	char *symname;
>> +	struct klp_reloc_sec *reloc_sec;
>> +	struct load_info *info;
>> +	Elf_Rela *rela;
>> +	Elf_Sym *sym, *symtab;
>> +	Elf_Shdr *symsect;
>>
>>  	if (WARN_ON(!klp_is_object_loaded(obj)))
>>  		return -EINVAL;
>>
>> -	if (WARN_ON(!obj->relocs))
>> -		return -EINVAL;
>> -
>> -	for (reloc = obj->relocs; reloc->name; reloc++) {
>> -		if (!klp_is_module(obj)) {
>> -			ret = klp_verify_vmlinux_symbol(reloc->name,
>> -							reloc->val);
>> -			if (ret)
>> -				return ret;
>> -		} else {
>> -			/* module, reloc->val needs to be discovered */
>> -			if (reloc->external)
>> -				ret = klp_find_external_symbol(pmod,
>> -							       reloc->name,
>> -							       &reloc->val);
>> -			else
>> -				ret = klp_find_object_symbol(obj->mod->name,
>> -							     reloc->name,
>> -							     &reloc->val);
>> -			if (ret)
>> -				return ret;
>> -		}
>> -		ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc,
>> -					     reloc->val + reloc->addend);
>> -		if (ret) {
>> -			pr_err("relocation failed for symbol '%s' at 0x%016lx (%d)\n",
>> -			       reloc->name, reloc->val, ret);
>> -			return ret;
>> +	info = pmod->info;
>> +	symsect = info->sechdrs + info->index.sym;
>> +	symtab = (void *)info->hdr + symsect->sh_offset;
>> +
>> +	/* For each __klp_rela section for this object */
>> +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
>> +		relindex = reloc_sec->index;
>> +		num_relas = info->sechdrs[relindex].sh_size / sizeof(Elf_Rela);
>> +		rela = (Elf_Rela *) info->sechdrs[relindex].sh_addr;
>> +
>> +		/* For each rela in this __klp_rela section */
>> +		for (i = 0; i < num_relas; i++, rela++) {
>> +			sym = symtab + ELF_R_SYM(rela->r_info);
>> +			symname = info->strtab + sym->st_name;
>> +			bind = ELF_ST_BIND(sym->st_info);
>> +
>> +			if (sym->st_shndx == SHN_LIVEPATCH) {
>> +				if (bind == STB_LIVEPATCH_EXT)
>> +					ret = klp_find_external_symbol(pmod, symname, &addr);
>> +				else
>> +					ret = klp_find_object_symbol(obj->name, symname, &addr);
>> +				if (ret)
>> +					return ret;
>> +				sym->st_value = addr;
>> +			}
>>  		}
>> +		ret = apply_relocate_add(info->sechdrs, info->strtab,
>> +					 info->index.sym, relindex, pmod);
>>  	}
>>
>> -	return 0;
>> +	return ret;
>>  }
>
>Looking at this... do we even need reloc_secs in klp_object? Question is
>whether we need more than one dynrela section for an object. If not then
>the binding between klp_reloc_sec and an object is the only relevant thing
>in the structure, be it index or objname. So we can replace the
>list of structures with just the index in klp_object, or get rid of it
>completely and rely on the name of dynrela section be something like
>__klp_rela_{objname}.

Hm, you bring up a good point. I think theoretically yes, it is
possible to just have one klp_reloc_sec for each object and therefore
a list is not required (I have not checked yet how difficult it would
be to implement this on the kpatch-build side of things).  However,
considering the final format of the patch module, I think it is
semantically clearer to leave it as a list, and for each object to
possibly have more than one __klp_rela section.

For example, say we are patching two functions in ext4. In my
resulting kpatch module I will have two __klp_rela_ext4 sections, and
they might look like this when we run readelf --sections:

[34] __klp_rela_ext4.text.ext4_attr_store RELA ...
[35] __klp_rela_ext4.text.ext4_attr_show RELA ...

Then these two klp rela sections end up as two elements in the
reloc_secs list for the ext4 patch object. I think this way, we can
better tell which rela is being applied to what function. Might be
easier to understand what's happening from the developer's point of
view.

>You see, we go through elf sections here which were preserved by module
>loader. We even have relevant sections marked with SHF_RELA_LIVEPATCH. So
>maybe all the stuff around klp_reloc_sec is not necessary.
>
>Thoughts?

Ah, so this is where descriptive comments and documentation might have
been useful :-) So I think we will still need to keep the
klp_reloc_sec struct to help the patch module initialize. Though the
name and objname fields aren't used in this patchset, they are used in
the kpatch patch module code [1], where we iterate through each elf
section, find the ones marked with SHF_RELA_LIVEPATCH, set the
klp_reloc_sec's objname (which we find from the "name" field,
formatted as __klp_rela_{objname}.text..). Once we have the objname
set, we can then find the object to attach the reloc_sec to (i.e. add
it to its list of reloc_secs).

Hope that clears some things up.


[1] https://github.com/flaming-toast/kpatch/blob/no_dynrela_redux/kmod/patch/livepatch-patch-hook.c#L213

>> @@ -741,12 +751,23 @@ static int klp_init_object_loaded(struct klp_patch *patch,
>>  				  struct klp_object *obj)
>>  {
>>  	struct klp_func *func;
>> +	struct module *pmod;
>>  	int ret;
>>
>> -	if (obj->relocs) {
>> -		ret = klp_write_object_relocations(patch->mod, obj);
>> +	pmod = patch->mod;
>> +
>> +	if (!list_empty(&obj->reloc_secs)) {
>> +		set_page_attributes(pmod->module_core,
>> +				    pmod->module_core + pmod->core_text_size,
>> +				    set_memory_rw);
>> +
>> +		ret = klp_write_object_relocations(pmod, obj, patch);
>>  		if (ret)
>>  			return ret;
>> +
>> +		set_page_attributes(pmod->module_core,
>> +				    pmod->module_core + pmod->core_text_size,
>> +				    set_memory_ro);
>>  	}
>
>And this would get solved with different patches as well. I think the
>calls to set_page_attributes() should be hidden in
>klp_write_object_relocations() as it is in Josh's patch IIRC.

Yes, when those patches are merged I will switch over to Josh's
functions for setting/unsetting memory ro.

Thanks,
Jessica

Re: module: save load_info for livepatch modules

[Jessica Yu]

+++ Petr Mladek [11/11/15 15:31 +0100]:
>On Mon 2015-11-09 23:45:52, Jessica Yu wrote:
>> In livepatch modules, preserve section, symbol, string information from
>> the load_info struct in the module loader. This information is used to
>> patch modules that are not loaded in memory yet; specifically it is used
>> to resolve remaining symbols and write relocations when the target
>> module loads.
>>
>> Signed-off-by: Jessica Yu <jeyu@redhat.com>
>> ---
>>  include/linux/module.h  | 25 +++++++++++++++++++++++++
>>  kernel/livepatch/core.c | 17 +++++++++++++++++
>>  kernel/module.c         | 36 ++++++++++++++++++++++--------------
>>  3 files changed, 64 insertions(+), 14 deletions(-)
>>
>> diff --git a/include/linux/module.h b/include/linux/module.h
>> index 3a19c79..c8680b1 100644
>> --- a/include/linux/module.h
>> +++ b/include/linux/module.h
>[...]
>> @@ -635,6 +651,15 @@ static inline bool module_requested_async_probing(struct module *module)
>>  	return module && module->async_probe_requested;
>>  }
>>
>> +#ifdef CONFIG_LIVEPATCH
>> +extern void klp_prepare_patch_module(struct module *mod,
>> +				     struct load_info *info);
>> +extern int
>> +apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
>> +		   unsigned int symindex, unsigned int relsec,
>> +		   struct module *me);
>> +#endif
>
>This function is already declared in moduleloader.h.
>It is implemted only when CONFIG_MODULES_USE_ELF_RELA is defined.
>
>I guess that we want to include moduleloader.h in livepatch.
>
>> +
>>  #else /* !CONFIG_MODULES... */
>>
>>  /* Given an address, look for it in the exception tables. */
>> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>> index 6e53441..087a8c7 100644
>> --- a/kernel/livepatch/core.c
>> +++ b/kernel/livepatch/core.c
>> @@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>>  	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>>  };
>>
>> +/*
>> + * Save necessary information from info in order to be able to
>> + * patch modules that might be loaded later
>> + */
>> +void klp_prepare_patch_module(struct module *mod, struct load_info *info)
>> +{
>> +	Elf_Shdr *symsect;
>> +
>> +	symsect = info->sechdrs + info->index.sym;
>> +	/* update sh_addr to point to symtab */
>> +	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
>
>Is livepatch the only user of this value? By other words, is this safe?

I think it is safe to say yes. klp_prepare_patch_module() is only
called at the very end of load_module(), right before
do_init_module(). Normally, at that point, info->hdr will have already
been freed by free_copy() along with the elf section information
associated with it. But if we have a livepatch module, we don't free.
So we should be the very last user, and there should be nobody
utilizing the memory associated with the load_info struct anymore at
that point.

>> +	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
>> +	memcpy(mod->info, info, sizeof(*info));
>> +
>> +}
>
>It is strange that this funtion is defined in livepatch/core.c
>but declared in module.h. I would move the definition to
>module.c.

Right, I was trying to keep all the livepatch-related functions
together in livepatch/core.c. but I can move it to module.c if it
makes more sense/Rusty doesn't object to it :-)

>>  static int __init klp_init(void)
>>  {
>>  	int ret;
>> diff --git a/kernel/module.c b/kernel/module.c
>> index 8f051a1..8ae3ca5 100644
>> --- a/kernel/module.c
>> +++ b/kernel/module.c
>> @@ -318,20 +318,6 @@ int unregister_module_notifier(struct notifier_block *nb)
>>  }
>>  EXPORT_SYMBOL(unregister_module_notifier);
>>
>> -struct load_info {
>> -	Elf_Ehdr *hdr;
>> -	unsigned long len;
>> -	Elf_Shdr *sechdrs;
>> -	char *secstrings, *strtab;
>> -	unsigned long symoffs, stroffs;
>> -	struct _ddebug *debug;
>> -	unsigned int num_debug;
>> -	bool sig_ok;
>> -	struct {
>> -		unsigned int sym, str, mod, vers, info, pcpu;
>> -	} index;
>> -};
>> -
>>  /* We require a truly strong try_module_get(): 0 means failure due to
>>     ongoing or failed initialization etc. */
>>  static inline int strong_try_module_get(struct module *mod)
>> @@ -2137,6 +2123,11 @@ static int simplify_symbols(struct module *mod, const struct load_info *info)
>>  			       (long)sym[i].st_value);
>>  			break;
>>
>> +#ifdef CONFIG_LIVEPATCH
>> +		case SHN_LIVEPATCH:
>> +			break;
>> +#endif
>
>IMHO, even a kernel compiled without CONFIG_LIVEPATCH should handle livepatch
>modules with grace. It means to reject loading.

I think even right now, without considering this patchset, we don't
reject modules "gracefully" when we load a livepatch module without
CONFIG_LIVEPATCH. The module loader will complain and reject the
livepatch module, saying something like "Unknown symbol
klp_register_patch." This behavior is the same with or without
this patch series applied. If we want to add a bit more logic to
gracefully reject patch modules, perhaps that should be a different
patch altogether, as I think it is unrelated to the goal of this one :-)

>>  		case SHN_UNDEF:
>>  			ksym = resolve_symbol_wait(mod, info, name);
>>  			/* Ok if resolved.  */
>> @@ -2185,6 +2176,11 @@ static int apply_relocations(struct module *mod, const struct load_info *info)
>>  		if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC))
>>  			continue;
>>
>> +#ifdef CONFIG_LIVEPATCH
>> +		if (info->sechdrs[i].sh_flags & SHF_RELA_LIVEPATCH)
>> +			continue;
>> +#endif
>> +
>>  		if (info->sechdrs[i].sh_type == SHT_REL)
>>  			err = apply_relocate(info->sechdrs, info->strtab,
>>  					     info->index.sym, i, mod);
>> @@ -3530,8 +3526,20 @@ static int load_module(struct load_info *info, const char __user *uargs,
>>  	if (err < 0)
>>  		goto bug_cleanup;
>>
>> +#ifdef CONFIG_LIVEPATCH
>> +	/*
>> +	 * Save sechdrs, indices, and other data from info
>> +	 * in order to patch to-be-loaded modules.
>> +	 * Do not call free_copy() for livepatch modules.
>> +	 */
>> +	if (get_modinfo((struct load_info *)info, "livepatch"))
>> +		klp_prepare_patch_module(mod, info);
>> +	else
>> +		free_copy(info);
>> +#else
>
>I would move this #else one line above and get rid of the
>double free_copy(info); But it is a matter of taste.

Maybe I'm missing something, but I think we do need the double
free_copy(), because in the CONFIG_LIVEPATCH case, we still want to
call free_copy() for non-livepatch modules. And we want to avoid
calling free_copy() for livepatch modules (hence the extra else).

>>  	/* Get rid of temporary copy. */
>>  	free_copy(info);
>> +#endif
>

Thanks for the comments,
Jessica

Re: module: save load_info for livepatch modules

[Jessica Yu]

+++ Petr Mladek [11/11/15 15:31 +0100]:
>On Mon 2015-11-09 23:45:52, Jessica Yu wrote:
>> In livepatch modules, preserve section, symbol, string information from
>> the load_info struct in the module loader. This information is used to
>> patch modules that are not loaded in memory yet; specifically it is used
>> to resolve remaining symbols and write relocations when the target
>> module loads.
>>
>> Signed-off-by: Jessica Yu <jeyu-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>> ---
>>  include/linux/module.h  | 25 +++++++++++++++++++++++++
>>  kernel/livepatch/core.c | 17 +++++++++++++++++
>>  kernel/module.c         | 36 ++++++++++++++++++++++--------------
>>  3 files changed, 64 insertions(+), 14 deletions(-)
>>
>> diff --git a/include/linux/module.h b/include/linux/module.h
>> index 3a19c79..c8680b1 100644
>> --- a/include/linux/module.h
>> +++ b/include/linux/module.h
>[...]
>> @@ -635,6 +651,15 @@ static inline bool module_requested_async_probing(struct module *module)
>>  	return module && module->async_probe_requested;
>>  }
>>
>> +#ifdef CONFIG_LIVEPATCH
>> +extern void klp_prepare_patch_module(struct module *mod,
>> +				     struct load_info *info);
>> +extern int
>> +apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
>> +		   unsigned int symindex, unsigned int relsec,
>> +		   struct module *me);
>> +#endif
>
>This function is already declared in moduleloader.h.
>It is implemted only when CONFIG_MODULES_USE_ELF_RELA is defined.
>
>I guess that we want to include moduleloader.h in livepatch.
>
>> +
>>  #else /* !CONFIG_MODULES... */
>>
>>  /* Given an address, look for it in the exception tables. */
>> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>> index 6e53441..087a8c7 100644
>> --- a/kernel/livepatch/core.c
>> +++ b/kernel/livepatch/core.c
>> @@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>>  	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>>  };
>>
>> +/*
>> + * Save necessary information from info in order to be able to
>> + * patch modules that might be loaded later
>> + */
>> +void klp_prepare_patch_module(struct module *mod, struct load_info *info)
>> +{
>> +	Elf_Shdr *symsect;
>> +
>> +	symsect = info->sechdrs + info->index.sym;
>> +	/* update sh_addr to point to symtab */
>> +	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
>
>Is livepatch the only user of this value? By other words, is this safe?

I think it is safe to say yes. klp_prepare_patch_module() is only
called at the very end of load_module(), right before
do_init_module(). Normally, at that point, info->hdr will have already
been freed by free_copy() along with the elf section information
associated with it. But if we have a livepatch module, we don't free.
So we should be the very last user, and there should be nobody
utilizing the memory associated with the load_info struct anymore at
that point.

>> +	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
>> +	memcpy(mod->info, info, sizeof(*info));
>> +
>> +}
>
>It is strange that this funtion is defined in livepatch/core.c
>but declared in module.h. I would move the definition to
>module.c.

Right, I was trying to keep all the livepatch-related functions
together in livepatch/core.c. but I can move it to module.c if it
makes more sense/Rusty doesn't object to it :-)

>>  static int __init klp_init(void)
>>  {
>>  	int ret;
>> diff --git a/kernel/module.c b/kernel/module.c
>> index 8f051a1..8ae3ca5 100644
>> --- a/kernel/module.c
>> +++ b/kernel/module.c
>> @@ -318,20 +318,6 @@ int unregister_module_notifier(struct notifier_block *nb)
>>  }
>>  EXPORT_SYMBOL(unregister_module_notifier);
>>
>> -struct load_info {
>> -	Elf_Ehdr *hdr;
>> -	unsigned long len;
>> -	Elf_Shdr *sechdrs;
>> -	char *secstrings, *strtab;
>> -	unsigned long symoffs, stroffs;
>> -	struct _ddebug *debug;
>> -	unsigned int num_debug;
>> -	bool sig_ok;
>> -	struct {
>> -		unsigned int sym, str, mod, vers, info, pcpu;
>> -	} index;
>> -};
>> -
>>  /* We require a truly strong try_module_get(): 0 means failure due to
>>     ongoing or failed initialization etc. */
>>  static inline int strong_try_module_get(struct module *mod)
>> @@ -2137,6 +2123,11 @@ static int simplify_symbols(struct module *mod, const struct load_info *info)
>>  			       (long)sym[i].st_value);
>>  			break;
>>
>> +#ifdef CONFIG_LIVEPATCH
>> +		case SHN_LIVEPATCH:
>> +			break;
>> +#endif
>
>IMHO, even a kernel compiled without CONFIG_LIVEPATCH should handle livepatch
>modules with grace. It means to reject loading.

I think even right now, without considering this patchset, we don't
reject modules "gracefully" when we load a livepatch module without
CONFIG_LIVEPATCH. The module loader will complain and reject the
livepatch module, saying something like "Unknown symbol
klp_register_patch." This behavior is the same with or without
this patch series applied. If we want to add a bit more logic to
gracefully reject patch modules, perhaps that should be a different
patch altogether, as I think it is unrelated to the goal of this one :-)

>>  		case SHN_UNDEF:
>>  			ksym = resolve_symbol_wait(mod, info, name);
>>  			/* Ok if resolved.  */
>> @@ -2185,6 +2176,11 @@ static int apply_relocations(struct module *mod, const struct load_info *info)
>>  		if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC))
>>  			continue;
>>
>> +#ifdef CONFIG_LIVEPATCH
>> +		if (info->sechdrs[i].sh_flags & SHF_RELA_LIVEPATCH)
>> +			continue;
>> +#endif
>> +
>>  		if (info->sechdrs[i].sh_type == SHT_REL)
>>  			err = apply_relocate(info->sechdrs, info->strtab,
>>  					     info->index.sym, i, mod);
>> @@ -3530,8 +3526,20 @@ static int load_module(struct load_info *info, const char __user *uargs,
>>  	if (err < 0)
>>  		goto bug_cleanup;
>>
>> +#ifdef CONFIG_LIVEPATCH
>> +	/*
>> +	 * Save sechdrs, indices, and other data from info
>> +	 * in order to patch to-be-loaded modules.
>> +	 * Do not call free_copy() for livepatch modules.
>> +	 */
>> +	if (get_modinfo((struct load_info *)info, "livepatch"))
>> +		klp_prepare_patch_module(mod, info);
>> +	else
>> +		free_copy(info);
>> +#else
>
>I would move this #else one line above and get rid of the
>double free_copy(info); But it is a matter of taste.

Maybe I'm missing something, but I think we do need the double
free_copy(), because in the CONFIG_LIVEPATCH case, we still want to
call free_copy() for non-livepatch modules. And we want to avoid
calling free_copy() for livepatch modules (hence the extra else).

>>  	/* Get rid of temporary copy. */
>>  	free_copy(info);
>> +#endif
>

Thanks for the comments,
Jessica

Re: module: save load_info for livepatch modules

[Jessica Yu]

+++ Miroslav Benes [11/11/15 15:17 +0100]:
>On Mon, 9 Nov 2015, Jessica Yu wrote:
>
>> diff --git a/include/linux/module.h b/include/linux/module.h
>> index 3a19c79..c8680b1 100644
>> --- a/include/linux/module.h
>> +++ b/include/linux/module.h
>
>[...]
>
>> +#ifdef CONFIG_LIVEPATCH
>> +extern void klp_prepare_patch_module(struct module *mod,
>> +				     struct load_info *info);
>> +extern int
>> +apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
>> +		   unsigned int symindex, unsigned int relsec,
>> +		   struct module *me);
>> +#endif
>> +
>>  #else /* !CONFIG_MODULES... */
>
>apply_relocate_add() is already in include/linux/moduleloader.h (guarded
>by CONFIG_MODULES_USE_ELF_RELA), so maybe we can just include that where
>we need it. As for the klp_prepare_patch_module() wouldn't it be better to
>have it in our livepatch.h and include that in kernel/module.c?

Yeah, Petr pointed this out as well :-) I will just include
moduleloader.h for the apply_relocate_add() declaration.

It also looks like we have some disagreement over where to put
klp_prepare_patch_module(), either in livepatch/core.c (and add the
function declaration in livepatch.h, and have module.c include
livepatch.h) or in kernel/module.c, keeping the
klp_prepare_patch_module() declaration in module.h. Maybe Rusty can
provide some input.

>>  /* Given an address, look for it in the exception tables. */
>> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>> index 6e53441..087a8c7 100644
>> --- a/kernel/livepatch/core.c
>> +++ b/kernel/livepatch/core.c
>> @@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>>  	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>>  };
>>
>> +/*
>> + * Save necessary information from info in order to be able to
>> + * patch modules that might be loaded later
>> + */
>> +void klp_prepare_patch_module(struct module *mod, struct load_info *info)
>> +{
>> +	Elf_Shdr *symsect;
>> +
>> +	symsect = info->sechdrs + info->index.sym;
>> +	/* update sh_addr to point to symtab */
>> +	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
>> +
>> +	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
>> +	memcpy(mod->info, info, sizeof(*info));
>> +
>> +}
>
>What about arch-specific 'struct mod_arch_specific'? We need to preserve
>it somewhere as well for s390x and other non-x86 architectures.

Ah! Thank you for catching this, I overlooked this important detail.
Yes, we do need to save the arch-specific struct. We would be in
trouble for s390 relocs if we didn't. I am trying to think of a way to
save this information for s390, since s390's module_finalize() frees
mod->arch.syminfo, which we definitely need in order for the call to
apply_relocate_add() to work. Maybe we can add an extra call right
before module_finalize() that will do some livepatch-specific
processing and copy this information (this would be in
post_relocation() in kernel/module.c). Perhaps this patchset cannot be
entirely free of arch-specific code after all :-( Still thinking.

>> +#ifdef CONFIG_LIVEPATCH
>> +	/*
>> +	 * Save sechdrs, indices, and other data from info
>> +	 * in order to patch to-be-loaded modules.
>> +	 * Do not call free_copy() for livepatch modules.
>> +	 */
>> +	if (get_modinfo((struct load_info *)info, "livepatch"))
>> +		klp_prepare_patch_module(mod, info);
>> +	else
>> +		free_copy(info);
>> +#else
>>  	/* Get rid of temporary copy. */
>>  	free_copy(info);
>> +#endif
>
>Maybe I am missing something but isn't it necessary to call vfree() on
>info somewhere in the end?

So free_copy() will call vfree(info->hdr), except in livepatch modules
we want to keep all the elf section information stored there, so we
avoid calling free_copy(), As for the info struct itself, if you look
at the init_module and finit_module syscall definitions in
kernel/module.c, you will see that info is actually a local function
variable, simply passed in to the call to load_module(), and will be
automatically deallocated when the syscall returns. :-) No need to
explicitly free info.

Thanks for the comments,
Jessica

Re: samples: livepatch: init reloc list and mark as klp module

[Jessica Yu]

+++ Petr Mladek [11/11/15 16:42 +0100]:
>On Mon 2015-11-09 23:45:54, Jessica Yu wrote:
>> Intialize the list of relocation sections in the sample
>> klp_object (even if the list will be empty in this case).
>> Also mark module as a livepatch module so that the module
>> loader can appropriately initialize it.
>>
>> Signed-off-by: Jessica Yu <jeyu@redhat.com>
>> ---
>>  samples/livepatch/livepatch-sample.c | 2 ++
>>  1 file changed, 2 insertions(+)
>>
>> diff --git a/samples/livepatch/livepatch-sample.c b/samples/livepatch/livepatch-sample.c
>> index fb8c861..2ef9345 100644
>> --- a/samples/livepatch/livepatch-sample.c
>> +++ b/samples/livepatch/livepatch-sample.c
>> @@ -89,3 +90,4 @@ static void livepatch_exit(void)
>>  module_init(livepatch_init);
>>  module_exit(livepatch_exit);
>>  MODULE_LICENSE("GPL");
>> +MODULE_INFO(livepatch, "Y");
>
>This looks a bit error prone. I wonder if we could detect this
>information another way. For example, by a check for the
>livepatch-related elf sections. If it is missing,
>we do not need to preserve struct load_info even
>when it is a livepatch.

Yeah, I agree that it is unnecessary for a livepatch module without
reloc secs to keep a copy of the load_info struct. My justification
for using MODULE_INFO is that I was trying to be consistent with the
way how other module "characteristics" are checked in the module
loader. For example, if the module came from the staging tree, the
module loader simply checks get_modinfo(info, "staging")). If the
module is a livepatch module, we check get_modinfo(info,
"livepatch")). I also thought that it might be useful additional
information for the user to be able to issue the modinfo command on a
module to see if it's a livepatch module or not (but maybe this
information won't be so useful after all, that's quite subjective).
But if we want to do a more thorough check, we could, like you said,
check for the livepatch-related elf sections before copying load_info.

Thanks,
Jessica

Re: samples: livepatch: init reloc list and mark as klp module

[Jessica Yu]

+++ Petr Mladek [11/11/15 16:42 +0100]:
>On Mon 2015-11-09 23:45:54, Jessica Yu wrote:
>> Intialize the list of relocation sections in the sample
>> klp_object (even if the list will be empty in this case).
>> Also mark module as a livepatch module so that the module
>> loader can appropriately initialize it.
>>
>> Signed-off-by: Jessica Yu <jeyu-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>> ---
>>  samples/livepatch/livepatch-sample.c | 2 ++
>>  1 file changed, 2 insertions(+)
>>
>> diff --git a/samples/livepatch/livepatch-sample.c b/samples/livepatch/livepatch-sample.c
>> index fb8c861..2ef9345 100644
>> --- a/samples/livepatch/livepatch-sample.c
>> +++ b/samples/livepatch/livepatch-sample.c
>> @@ -89,3 +90,4 @@ static void livepatch_exit(void)
>>  module_init(livepatch_init);
>>  module_exit(livepatch_exit);
>>  MODULE_LICENSE("GPL");
>> +MODULE_INFO(livepatch, "Y");
>
>This looks a bit error prone. I wonder if we could detect this
>information another way. For example, by a check for the
>livepatch-related elf sections. If it is missing,
>we do not need to preserve struct load_info even
>when it is a livepatch.

Yeah, I agree that it is unnecessary for a livepatch module without
reloc secs to keep a copy of the load_info struct. My justification
for using MODULE_INFO is that I was trying to be consistent with the
way how other module "characteristics" are checked in the module
loader. For example, if the module came from the staging tree, the
module loader simply checks get_modinfo(info, "staging")). If the
module is a livepatch module, we check get_modinfo(info,
"livepatch")). I also thought that it might be useful additional
information for the user to be able to issue the modinfo command on a
module to see if it's a livepatch module or not (but maybe this
information won't be so useful after all, that's quite subjective).
But if we want to do a more thorough check, we could, like you said,
check for the livepatch-related elf sections before copying load_info.

Thanks,
Jessica

Re: livepatch: reuse module loader code to write relocations

[Petr Mladek]

On Wed 2015-11-11 13:27:18, Jessica Yu wrote:
> Basically, the patch module code is expected to build the reloc_secs
> list for each object that is being patched. For example in kpatch, the
> patch module generates this list in patch_init(). Like you guessed, it
> does go through all the elf sections of the patch module to find the
> reloc sections marked SHF_RELA_LIVEPATCH. We are able to access these
> sections through module->info, which is set up for livepatch modules
> before the module loader calls do_init_module() (and hence before
> patch_init(), See patch 2/5).
> See below for an example of how the reloc_secs list might be built: https://github.com/flaming-toast/kpatch/blob/no_dynrela_redux/kmod/patch/livepatch-patch-hook.c#L213

Thanks a lot for the link and explanation. I think that it would be nice to
incorporate this into the patchset. I would put it into a separate
function, e.g. klp_reloc_secs_init(). It can be called either from
klp_register_patch() or from klp_init_patch().

> Once the patch module has built this list, it is good to go for use in
> livepatch core. All livepatch has to do then is to iterate though the
> list, resolve any outstanding symbols, and call apply_relocate_add(),
> as shown in this patch.
>
> Miroslav mentioned in another email that we should start thinking
> about including documentation about this, including the expected patch
> module format. So perhaps v2 should include some documentation about
> this whole process somewhere.

That would be cool.

Thank you,
Petr

Re: livepatch: reuse module loader code to write relocations

[Petr Mladek]

On Wed 2015-11-11 13:27:18, Jessica Yu wrote:
> Basically, the patch module code is expected to build the reloc_secs
> list for each object that is being patched. For example in kpatch, the
> patch module generates this list in patch_init(). Like you guessed, it
> does go through all the elf sections of the patch module to find the
> reloc sections marked SHF_RELA_LIVEPATCH. We are able to access these
> sections through module->info, which is set up for livepatch modules
> before the module loader calls do_init_module() (and hence before
> patch_init(), See patch 2/5).
> See below for an example of how the reloc_secs list might be built: https://github.com/flaming-toast/kpatch/blob/no_dynrela_redux/kmod/patch/livepatch-patch-hook.c#L213

Thanks a lot for the link and explanation. I think that it would be nice to
incorporate this into the patchset. I would put it into a separate
function, e.g. klp_reloc_secs_init(). It can be called either from
klp_register_patch() or from klp_init_patch().

> Once the patch module has built this list, it is good to go for use in
> livepatch core. All livepatch has to do then is to iterate though the
> list, resolve any outstanding symbols, and call apply_relocate_add(),
> as shown in this patch.
>
> Miroslav mentioned in another email that we should start thinking
> about including documentation about this, including the expected patch
> module format. So perhaps v2 should include some documentation about
> this whole process somewhere.

That would be cool.

Thank you,
Petr

Re: module: save load_info for livepatch modules

[Petr Mladek]

On Wed 2015-11-11 23:44:08, Jessica Yu wrote:
> +++ Petr Mladek [11/11/15 15:31 +0100]:
> >On Mon 2015-11-09 23:45:52, Jessica Yu wrote:
> >>diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> >>index 6e53441..087a8c7 100644
> >>--- a/kernel/livepatch/core.c
> >>+++ b/kernel/livepatch/core.c
> >>@@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
> >> 	.priority = INT_MIN+1, /* called late but before ftrace notifier */
> >> };
> >>
> >>+/*
> >>+ * Save necessary information from info in order to be able to
> >>+ * patch modules that might be loaded later
> >>+ */
> >>+void klp_prepare_patch_module(struct module *mod, struct load_info *info)
> >>+{
> >>+	Elf_Shdr *symsect;
> >>+
> >>+	symsect = info->sechdrs + info->index.sym;
> >>+	/* update sh_addr to point to symtab */
> >>+	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
> >
> >Is livepatch the only user of this value? By other words, is this safe?
> 
> I think it is safe to say yes. klp_prepare_patch_module() is only
> called at the very end of load_module(), right before
> do_init_module(). Normally, at that point, info->hdr will have already
> been freed by free_copy() along with the elf section information
> associated with it. But if we have a livepatch module, we don't free.
> So we should be the very last user, and there should be nobody
> utilizing the memory associated with the load_info struct anymore at
> that point.

I see. It looks safe at this point. But still I wonder if it would be
possible to calculate this later in the livepatch code. It will allow
to potentially use the info structure also by other subsystem.

BTW: Where is "sh_addr" value used, please? I see it used only
in the third patch as info->sechdrs[relindex].sh_addr. But it is
an array. I am not sure if it is the same variable.


> >>+	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
> >>+	memcpy(mod->info, info, sizeof(*info));
> >>+
> >>+}
> >
> >It is strange that this funtion is defined in livepatch/core.c
> >but declared in module.h. I would move the definition to
> >module.c.
> 
> Right, I was trying to keep all the livepatch-related functions
> together in livepatch/core.c. but I can move it to module.c if it
> makes more sense/Rusty doesn't object to it :-)

Sure. I think that we could use some generic name, e.g. copy_module_info().

> >> static int __init klp_init(void)
> >> {
> >> 	int ret;
> >>diff --git a/kernel/module.c b/kernel/module.c
> >>index 8f051a1..8ae3ca5 100644
> >>--- a/kernel/module.c
> >>+++ b/kernel/module.c
> >>@@ -2137,6 +2123,11 @@ static int simplify_symbols(struct module *mod, const struct load_info *info)
> >> 			       (long)sym[i].st_value);
> >> 			break;
> >>
> >>+#ifdef CONFIG_LIVEPATCH
> >>+		case SHN_LIVEPATCH:
> >>+			break;
> >>+#endif
> >
> >IMHO, even a kernel compiled without CONFIG_LIVEPATCH should handle livepatch
> >modules with grace. It means to reject loading.
> 
> I think even right now, without considering this patchset, we don't
> reject modules "gracefully" when we load a livepatch module without
> CONFIG_LIVEPATCH. The module loader will complain and reject the
> livepatch module, saying something like "Unknown symbol
> klp_register_patch." This behavior is the same with or without
> this patch series applied. If we want to add a bit more logic to
> gracefully reject patch modules, perhaps that should be a different
> patch altogether, as I think it is unrelated to the goal of this one :-)

Yup, the module load would fail anyway because of the missing symbol.
But I think that we should fail on the first error occurence.

In each case, IMHO, we should not do the "default:" action for this
section even when complied without CONFIG_LIVEPATCH.


> >> 		case SHN_UNDEF:
> >> 			ksym = resolve_symbol_wait(mod, info, name);
> >> 			/* Ok if resolved.  */
> >>@@ -2185,6 +2176,11 @@ static int apply_relocations(struct module *mod, const struct load_info *info)
> >> 		if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC))
> >> 			continue;
> >>
> >>+#ifdef CONFIG_LIVEPATCH
> >>+		if (info->sechdrs[i].sh_flags & SHF_RELA_LIVEPATCH)
> >>+			continue;
> >>+#endif

I guess that if we do not trigger the error above, and do
not have the check here, we will try to call apply_relocate() below.
I guess that it will fail. If we are lucky it will print "Unknown
relocation". I think that we could do better.

> >>+
> >> 		if (info->sechdrs[i].sh_type == SHT_REL)
> >> 			err = apply_relocate(info->sechdrs, info->strtab,
> >> 					     info->index.sym, i, mod);
> >>@@ -3530,8 +3526,20 @@ static int load_module(struct load_info *info, const char __user *uargs,
> >> 	if (err < 0)
> >> 		goto bug_cleanup;
> >>
> >>+#ifdef CONFIG_LIVEPATCH
> >>+	/*
> >>+	 * Save sechdrs, indices, and other data from info
> >>+	 * in order to patch to-be-loaded modules.
> >>+	 * Do not call free_copy() for livepatch modules.
> >>+	 */
> >>+	if (get_modinfo((struct load_info *)info, "livepatch"))
> >>+		klp_prepare_patch_module(mod, info);
> >>+	else
> >>+		free_copy(info);
> >>+#else
> >
> >I would move this #else one line above and get rid of the
> >double free_copy(info); But it is a matter of taste.
> 
> Maybe I'm missing something, but I think we do need the double
> free_copy(), because in the CONFIG_LIVEPATCH case, we still want to
> call free_copy() for non-livepatch modules. And we want to avoid
> calling free_copy() for livepatch modules (hence the extra else).

Ah, this was just a cosmetic change. I meant to use something like:

#ifdef CONFIG_LIVEPATCH
	/*
	 * Save sechdrs, indices, and other data from info
	 * in order to patch to-be-loaded modules.
	 * Do not call free_copy() for livepatch modules.
	 */
	if (get_modinfo((struct load_info *)info, "livepatch"))
		klp_prepare_patch_module(mod, info);
	else
#endif
		/* Get rid of temporary copy. */
		free_copy(info);


It is a matter of taste. Maybe, your variant was better in the end.

Thank you,
Petr

Re: module: save load_info for livepatch modules

[Petr Mladek]

On Wed 2015-11-11 23:44:08, Jessica Yu wrote:
> +++ Petr Mladek [11/11/15 15:31 +0100]:
> >On Mon 2015-11-09 23:45:52, Jessica Yu wrote:
> >>diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> >>index 6e53441..087a8c7 100644
> >>--- a/kernel/livepatch/core.c
> >>+++ b/kernel/livepatch/core.c
> >>@@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
> >> 	.priority = INT_MIN+1, /* called late but before ftrace notifier */
> >> };
> >>
> >>+/*
> >>+ * Save necessary information from info in order to be able to
> >>+ * patch modules that might be loaded later
> >>+ */
> >>+void klp_prepare_patch_module(struct module *mod, struct load_info *info)
> >>+{
> >>+	Elf_Shdr *symsect;
> >>+
> >>+	symsect = info->sechdrs + info->index.sym;
> >>+	/* update sh_addr to point to symtab */
> >>+	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
> >
> >Is livepatch the only user of this value? By other words, is this safe?
> 
> I think it is safe to say yes. klp_prepare_patch_module() is only
> called at the very end of load_module(), right before
> do_init_module(). Normally, at that point, info->hdr will have already
> been freed by free_copy() along with the elf section information
> associated with it. But if we have a livepatch module, we don't free.
> So we should be the very last user, and there should be nobody
> utilizing the memory associated with the load_info struct anymore at
> that point.

I see. It looks safe at this point. But still I wonder if it would be
possible to calculate this later in the livepatch code. It will allow
to potentially use the info structure also by other subsystem.

BTW: Where is "sh_addr" value used, please? I see it used only
in the third patch as info->sechdrs[relindex].sh_addr. But it is
an array. I am not sure if it is the same variable.


> >>+	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
> >>+	memcpy(mod->info, info, sizeof(*info));
> >>+
> >>+}
> >
> >It is strange that this funtion is defined in livepatch/core.c
> >but declared in module.h. I would move the definition to
> >module.c.
> 
> Right, I was trying to keep all the livepatch-related functions
> together in livepatch/core.c. but I can move it to module.c if it
> makes more sense/Rusty doesn't object to it :-)

Sure. I think that we could use some generic name, e.g. copy_module_info().

> >> static int __init klp_init(void)
> >> {
> >> 	int ret;
> >>diff --git a/kernel/module.c b/kernel/module.c
> >>index 8f051a1..8ae3ca5 100644
> >>--- a/kernel/module.c
> >>+++ b/kernel/module.c
> >>@@ -2137,6 +2123,11 @@ static int simplify_symbols(struct module *mod, const struct load_info *info)
> >> 			       (long)sym[i].st_value);
> >> 			break;
> >>
> >>+#ifdef CONFIG_LIVEPATCH
> >>+		case SHN_LIVEPATCH:
> >>+			break;
> >>+#endif
> >
> >IMHO, even a kernel compiled without CONFIG_LIVEPATCH should handle livepatch
> >modules with grace. It means to reject loading.
> 
> I think even right now, without considering this patchset, we don't
> reject modules "gracefully" when we load a livepatch module without
> CONFIG_LIVEPATCH. The module loader will complain and reject the
> livepatch module, saying something like "Unknown symbol
> klp_register_patch." This behavior is the same with or without
> this patch series applied. If we want to add a bit more logic to
> gracefully reject patch modules, perhaps that should be a different
> patch altogether, as I think it is unrelated to the goal of this one :-)

Yup, the module load would fail anyway because of the missing symbol.
But I think that we should fail on the first error occurence.

In each case, IMHO, we should not do the "default:" action for this
section even when complied without CONFIG_LIVEPATCH.


> >> 		case SHN_UNDEF:
> >> 			ksym = resolve_symbol_wait(mod, info, name);
> >> 			/* Ok if resolved.  */
> >>@@ -2185,6 +2176,11 @@ static int apply_relocations(struct module *mod, const struct load_info *info)
> >> 		if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC))
> >> 			continue;
> >>
> >>+#ifdef CONFIG_LIVEPATCH
> >>+		if (info->sechdrs[i].sh_flags & SHF_RELA_LIVEPATCH)
> >>+			continue;
> >>+#endif

I guess that if we do not trigger the error above, and do
not have the check here, we will try to call apply_relocate() below.
I guess that it will fail. If we are lucky it will print "Unknown
relocation". I think that we could do better.

> >>+
> >> 		if (info->sechdrs[i].sh_type == SHT_REL)
> >> 			err = apply_relocate(info->sechdrs, info->strtab,
> >> 					     info->index.sym, i, mod);
> >>@@ -3530,8 +3526,20 @@ static int load_module(struct load_info *info, const char __user *uargs,
> >> 	if (err < 0)
> >> 		goto bug_cleanup;
> >>
> >>+#ifdef CONFIG_LIVEPATCH
> >>+	/*
> >>+	 * Save sechdrs, indices, and other data from info
> >>+	 * in order to patch to-be-loaded modules.
> >>+	 * Do not call free_copy() for livepatch modules.
> >>+	 */
> >>+	if (get_modinfo((struct load_info *)info, "livepatch"))
> >>+		klp_prepare_patch_module(mod, info);
> >>+	else
> >>+		free_copy(info);
> >>+#else
> >
> >I would move this #else one line above and get rid of the
> >double free_copy(info); But it is a matter of taste.
> 
> Maybe I'm missing something, but I think we do need the double
> free_copy(), because in the CONFIG_LIVEPATCH case, we still want to
> call free_copy() for non-livepatch modules. And we want to avoid
> calling free_copy() for livepatch modules (hence the extra else).

Ah, this was just a cosmetic change. I meant to use something like:

#ifdef CONFIG_LIVEPATCH
	/*
	 * Save sechdrs, indices, and other data from info
	 * in order to patch to-be-loaded modules.
	 * Do not call free_copy() for livepatch modules.
	 */
	if (get_modinfo((struct load_info *)info, "livepatch"))
		klp_prepare_patch_module(mod, info);
	else
#endif
		/* Get rid of temporary copy. */
		free_copy(info);


It is a matter of taste. Maybe, your variant was better in the end.

Thank you,
Petr

Re: module: save load_info for livepatch modules

[Petr Mladek]

On Thu 2015-11-12 00:33:12, Jessica Yu wrote:
> +++ Miroslav Benes [11/11/15 15:17 +0100]:
> >On Mon, 9 Nov 2015, Jessica Yu wrote:
> >
> >>diff --git a/include/linux/module.h b/include/linux/module.h
> >>index 3a19c79..c8680b1 100644
> >>--- a/include/linux/module.h
> >>+++ b/include/linux/module.h
> >
> >[...]
> >
> >>+#ifdef CONFIG_LIVEPATCH
> >>+extern void klp_prepare_patch_module(struct module *mod,
> >>+				     struct load_info *info);
> >>+extern int
> >>+apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
> >>+		   unsigned int symindex, unsigned int relsec,
> >>+		   struct module *me);
> >>+#endif
> >>+
> >> #else /* !CONFIG_MODULES... */
> >
> >apply_relocate_add() is already in include/linux/moduleloader.h (guarded
> >by CONFIG_MODULES_USE_ELF_RELA), so maybe we can just include that where
> >we need it. As for the klp_prepare_patch_module() wouldn't it be better to
> >have it in our livepatch.h and include that in kernel/module.c?
> 
> Yeah, Petr pointed this out as well :-) I will just include
> moduleloader.h for the apply_relocate_add() declaration.
> 
> It also looks like we have some disagreement over where to put
> klp_prepare_patch_module(), either in livepatch/core.c (and add the
> function declaration in livepatch.h, and have module.c include
> livepatch.h) or in kernel/module.c, keeping the
> klp_prepare_patch_module() declaration in module.h. Maybe Rusty can
> provide some input.
> 
> >> /* Given an address, look for it in the exception tables. */
> >>diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> >>index 6e53441..087a8c7 100644
> >>--- a/kernel/livepatch/core.c
> >>+++ b/kernel/livepatch/core.c
> >>@@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
> >> 	.priority = INT_MIN+1, /* called late but before ftrace notifier */
> >> };
> >>
> >>+/*
> >>+ * Save necessary information from info in order to be able to
> >>+ * patch modules that might be loaded later
> >>+ */
> >>+void klp_prepare_patch_module(struct module *mod, struct load_info *info)
> >>+{
> >>+	Elf_Shdr *symsect;
> >>+
> >>+	symsect = info->sechdrs + info->index.sym;
> >>+	/* update sh_addr to point to symtab */
> >>+	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
> >>+
> >>+	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
> >>+	memcpy(mod->info, info, sizeof(*info));
> >>+
> >>+}
> >
> >What about arch-specific 'struct mod_arch_specific'? We need to preserve
> >it somewhere as well for s390x and other non-x86 architectures.
> 
> Ah! Thank you for catching this, I overlooked this important detail.
> Yes, we do need to save the arch-specific struct. We would be in
> trouble for s390 relocs if we didn't. I am trying to think of a way to
> save this information for s390, since s390's module_finalize() frees
> mod->arch.syminfo, which we definitely need in order for the call to
> apply_relocate_add() to work. Maybe we can add an extra call right
> before module_finalize() that will do some livepatch-specific
> processing and copy this information (this would be in
> post_relocation() in kernel/module.c). Perhaps this patchset cannot be
> entirely free of arch-specific code after all :-( Still thinking.

I think about adding a flag somewhere, e.g. mod->preserve_relocs.
It might be set in simplify_symbols() when SHN_LIVEPATCH is found.
It might be checked when freeing the needed structures in both
the generic and arch-specific code.

> >>+#ifdef CONFIG_LIVEPATCH
> >>+	/*
> >>+	 * Save sechdrs, indices, and other data from info
> >>+	 * in order to patch to-be-loaded modules.
> >>+	 * Do not call free_copy() for livepatch modules.
> >>+	 */
> >>+	if (get_modinfo((struct load_info *)info, "livepatch"))
> >>+		klp_prepare_patch_module(mod, info);
> >>+	else
> >>+		free_copy(info);
> >>+#else
> >> 	/* Get rid of temporary copy. */
> >> 	free_copy(info);
> >>+#endif
> >
> >Maybe I am missing something but isn't it necessary to call vfree() on
> >info somewhere in the end?
> 
> So free_copy() will call vfree(info->hdr), except in livepatch modules
> we want to keep all the elf section information stored there, so we
> avoid calling free_copy(), As for the info struct itself, if you look
> at the init_module and finit_module syscall definitions in
> kernel/module.c, you will see that info is actually a local function
> variable, simply passed in to the call to load_module(), and will be
> automatically deallocated when the syscall returns. :-) No need to
> explicitly free info.

We still have to free the copied or preserved structures when
the module is unloaded.

Thank you,
Petr

Re: samples: livepatch: init reloc list and mark as klp module

[Miroslav Benes]

On Thu, 12 Nov 2015, Jessica Yu wrote:

> +++ Petr Mladek [11/11/15 16:42 +0100]:
> > On Mon 2015-11-09 23:45:54, Jessica Yu wrote:
> > > Intialize the list of relocation sections in the sample
> > > klp_object (even if the list will be empty in this case).
> > > Also mark module as a livepatch module so that the module
> > > loader can appropriately initialize it.
> > > 
> > > Signed-off-by: Jessica Yu <jeyu@redhat.com>
> > > ---
> > >  samples/livepatch/livepatch-sample.c | 2 ++
> > >  1 file changed, 2 insertions(+)
> > > 
> > > diff --git a/samples/livepatch/livepatch-sample.c
> > > b/samples/livepatch/livepatch-sample.c
> > > index fb8c861..2ef9345 100644
> > > --- a/samples/livepatch/livepatch-sample.c
> > > +++ b/samples/livepatch/livepatch-sample.c
> > > @@ -89,3 +90,4 @@ static void livepatch_exit(void)
> > >  module_init(livepatch_init);
> > >  module_exit(livepatch_exit);
> > >  MODULE_LICENSE("GPL");
> > > +MODULE_INFO(livepatch, "Y");
> > 
> > This looks a bit error prone. I wonder if we could detect this
> > information another way. For example, by a check for the
> > livepatch-related elf sections. If it is missing,
> > we do not need to preserve struct load_info even
> > when it is a livepatch.
> 
> Yeah, I agree that it is unnecessary for a livepatch module without
> reloc secs to keep a copy of the load_info struct. My justification
> for using MODULE_INFO is that I was trying to be consistent with the
> way how other module "characteristics" are checked in the module
> loader. For example, if the module came from the staging tree, the
> module loader simply checks get_modinfo(info, "staging")). If the
> module is a livepatch module, we check get_modinfo(info,
> "livepatch")). I also thought that it might be useful additional
> information for the user to be able to issue the modinfo command on a
> module to see if it's a livepatch module or not (but maybe this
> information won't be so useful after all, that's quite subjective).

Yup, in my opinion this is a good way to do it. We already impose quite a 
lot on a patch module and this does not make a big difference. Easy 
identification of a patch module is good bonus as well.

> But if we want to do a more thorough check, we could, like you said,
> check for the livepatch-related elf sections before copying load_info.

I wouldn't do that. It could be even more error prone.

I'd like to think that we can live with load_info struct even for patch 
modules which do not use relocations. Don't know.

Miroslav

Re: samples: livepatch: init reloc list and mark as klp module

[Miroslav Benes]

On Thu, 12 Nov 2015, Jessica Yu wrote:

> +++ Petr Mladek [11/11/15 16:42 +0100]:
> > On Mon 2015-11-09 23:45:54, Jessica Yu wrote:
> > > Intialize the list of relocation sections in the sample
> > > klp_object (even if the list will be empty in this case).
> > > Also mark module as a livepatch module so that the module
> > > loader can appropriately initialize it.
> > > 
> > > Signed-off-by: Jessica Yu <jeyu-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > > ---
> > >  samples/livepatch/livepatch-sample.c | 2 ++
> > >  1 file changed, 2 insertions(+)
> > > 
> > > diff --git a/samples/livepatch/livepatch-sample.c
> > > b/samples/livepatch/livepatch-sample.c
> > > index fb8c861..2ef9345 100644
> > > --- a/samples/livepatch/livepatch-sample.c
> > > +++ b/samples/livepatch/livepatch-sample.c
> > > @@ -89,3 +90,4 @@ static void livepatch_exit(void)
> > >  module_init(livepatch_init);
> > >  module_exit(livepatch_exit);
> > >  MODULE_LICENSE("GPL");
> > > +MODULE_INFO(livepatch, "Y");
> > 
> > This looks a bit error prone. I wonder if we could detect this
> > information another way. For example, by a check for the
> > livepatch-related elf sections. If it is missing,
> > we do not need to preserve struct load_info even
> > when it is a livepatch.
> 
> Yeah, I agree that it is unnecessary for a livepatch module without
> reloc secs to keep a copy of the load_info struct. My justification
> for using MODULE_INFO is that I was trying to be consistent with the
> way how other module "characteristics" are checked in the module
> loader. For example, if the module came from the staging tree, the
> module loader simply checks get_modinfo(info, "staging")). If the
> module is a livepatch module, we check get_modinfo(info,
> "livepatch")). I also thought that it might be useful additional
> information for the user to be able to issue the modinfo command on a
> module to see if it's a livepatch module or not (but maybe this
> information won't be so useful after all, that's quite subjective).

Yup, in my opinion this is a good way to do it. We already impose quite a 
lot on a patch module and this does not make a big difference. Easy 
identification of a patch module is good bonus as well.

> But if we want to do a more thorough check, we could, like you said,
> check for the livepatch-related elf sections before copying load_info.

I wouldn't do that. It could be even more error prone.

I'd like to think that we can live with load_info struct even for patch 
modules which do not use relocations. Don't know.

Miroslav

Re: module: save load_info for livepatch modules

[Miroslav Benes]

On Thu, 12 Nov 2015, Petr Mladek wrote:

> On Thu 2015-11-12 00:33:12, Jessica Yu wrote:
> > +++ Miroslav Benes [11/11/15 15:17 +0100]:
> > >On Mon, 9 Nov 2015, Jessica Yu wrote:
> > >
> > >>diff --git a/include/linux/module.h b/include/linux/module.h
> > >>index 3a19c79..c8680b1 100644
> > >>--- a/include/linux/module.h
> > >>+++ b/include/linux/module.h
> > >
> > >[...]
> > >
> > >>+#ifdef CONFIG_LIVEPATCH
> > >>+extern void klp_prepare_patch_module(struct module *mod,
> > >>+				     struct load_info *info);
> > >>+extern int
> > >>+apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
> > >>+		   unsigned int symindex, unsigned int relsec,
> > >>+		   struct module *me);
> > >>+#endif
> > >>+
> > >> #else /* !CONFIG_MODULES... */
> > >
> > >apply_relocate_add() is already in include/linux/moduleloader.h (guarded
> > >by CONFIG_MODULES_USE_ELF_RELA), so maybe we can just include that where
> > >we need it. As for the klp_prepare_patch_module() wouldn't it be better to
> > >have it in our livepatch.h and include that in kernel/module.c?
> > 
> > Yeah, Petr pointed this out as well :-) I will just include
> > moduleloader.h for the apply_relocate_add() declaration.
> > 
> > It also looks like we have some disagreement over where to put
> > klp_prepare_patch_module(), either in livepatch/core.c (and add the
> > function declaration in livepatch.h, and have module.c include
> > livepatch.h) or in kernel/module.c, keeping the
> > klp_prepare_patch_module() declaration in module.h. Maybe Rusty can
> > provide some input.

Yes, there are several ways how to do it. Maybe the best would be some 
generic way in kernel/module.c. I am not sure if there will be another 
user of this in the future but nevertheless. It would also allow us to 
somehow solve the issues mentioned below. Thus, klp_prepare_patch_module 
is inappropriate name and it should be for example just preserve_load_info 
(or more general if needed) and it should be in kernel/module.c.

> > >> /* Given an address, look for it in the exception tables. */
> > >>diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > >>index 6e53441..087a8c7 100644
> > >>--- a/kernel/livepatch/core.c
> > >>+++ b/kernel/livepatch/core.c
> > >>@@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
> > >> 	.priority = INT_MIN+1, /* called late but before ftrace notifier */
> > >> };
> > >>
> > >>+/*
> > >>+ * Save necessary information from info in order to be able to
> > >>+ * patch modules that might be loaded later
> > >>+ */
> > >>+void klp_prepare_patch_module(struct module *mod, struct load_info *info)
> > >>+{
> > >>+	Elf_Shdr *symsect;
> > >>+
> > >>+	symsect = info->sechdrs + info->index.sym;
> > >>+	/* update sh_addr to point to symtab */
> > >>+	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
> > >>+
> > >>+	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
> > >>+	memcpy(mod->info, info, sizeof(*info));
> > >>+
> > >>+}
> > >
> > >What about arch-specific 'struct mod_arch_specific'? We need to preserve
> > >it somewhere as well for s390x and other non-x86 architectures.
> > 
> > Ah! Thank you for catching this, I overlooked this important detail.
> > Yes, we do need to save the arch-specific struct. We would be in
> > trouble for s390 relocs if we didn't. I am trying to think of a way to
> > save this information for s390, since s390's module_finalize() frees
> > mod->arch.syminfo, which we definitely need in order for the call to
> > apply_relocate_add() to work. Maybe we can add an extra call right
> > before module_finalize() that will do some livepatch-specific
> > processing and copy this information (this would be in
> > post_relocation() in kernel/module.c). Perhaps this patchset cannot be
> > entirely free of arch-specific code after all :-( Still thinking.

Well, mod_arch_specific is defined as each architecture needs. So for x86 
it is empty. It is arch-agnostic in this way and we can deal with it as 
"a black box". We just need it not to be freed in module_finalize. And...

> I think about adding a flag somewhere, e.g. mod->preserve_relocs.
> It might be set in simplify_symbols() when SHN_LIVEPATCH is found.
> It might be checked when freeing the needed structures in both
> the generic and arch-specific code.

...that is the reason why some sort of flag seems to be necessary. It 
could be set when livepatch is set in modinfo. We would use it for 
preserving both load_info and mod_arch_specific struct (in some form) and 
for...

> > >>+#ifdef CONFIG_LIVEPATCH
> > >>+	/*
> > >>+	 * Save sechdrs, indices, and other data from info
> > >>+	 * in order to patch to-be-loaded modules.
> > >>+	 * Do not call free_copy() for livepatch modules.
> > >>+	 */
> > >>+	if (get_modinfo((struct load_info *)info, "livepatch"))
> > >>+		klp_prepare_patch_module(mod, info);
> > >>+	else
> > >>+		free_copy(info);
> > >>+#else
> > >> 	/* Get rid of temporary copy. */
> > >> 	free_copy(info);
> > >>+#endif
> > >
> > >Maybe I am missing something but isn't it necessary to call vfree() on
> > >info somewhere in the end?
> > 
> > So free_copy() will call vfree(info->hdr), except in livepatch modules
> > we want to keep all the elf section information stored there, so we
> > avoid calling free_copy(), As for the info struct itself, if you look
> > at the init_module and finit_module syscall definitions in
> > kernel/module.c, you will see that info is actually a local function
> > variable, simply passed in to the call to load_module(), and will be
> > automatically deallocated when the syscall returns. :-) No need to
> > explicitly free info.
> 
> We still have to free the copied or preserved structures when
> the module is unloaded.

...freeing what we allocated. We need to free info->hdr somewhere if not 
here and also mod_arch_specific struct where the patch module is removed. 
This would unfortunately lead to changes in arch-specific code in 
module.c. For example in arch/s390/kernel/module.c there is vfree call on 
part of mod_arch_specific in module_finalize. We would call it only if the 
flag mentioned above is not set and at the same time we would need to call 
it when the patch module is being removed.

Hm, this is (again) getting complicated and ugly. Is there someone who can 
simplify things? Josh? :)

Miroslav

Re: module: save load_info for livepatch modules

[Miroslav Benes]

On Thu, 12 Nov 2015, Petr Mladek wrote:

> On Thu 2015-11-12 00:33:12, Jessica Yu wrote:
> > +++ Miroslav Benes [11/11/15 15:17 +0100]:
> > >On Mon, 9 Nov 2015, Jessica Yu wrote:
> > >
> > >>diff --git a/include/linux/module.h b/include/linux/module.h
> > >>index 3a19c79..c8680b1 100644
> > >>--- a/include/linux/module.h
> > >>+++ b/include/linux/module.h
> > >
> > >[...]
> > >
> > >>+#ifdef CONFIG_LIVEPATCH
> > >>+extern void klp_prepare_patch_module(struct module *mod,
> > >>+				     struct load_info *info);
> > >>+extern int
> > >>+apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
> > >>+		   unsigned int symindex, unsigned int relsec,
> > >>+		   struct module *me);
> > >>+#endif
> > >>+
> > >> #else /* !CONFIG_MODULES... */
> > >
> > >apply_relocate_add() is already in include/linux/moduleloader.h (guarded
> > >by CONFIG_MODULES_USE_ELF_RELA), so maybe we can just include that where
> > >we need it. As for the klp_prepare_patch_module() wouldn't it be better to
> > >have it in our livepatch.h and include that in kernel/module.c?
> > 
> > Yeah, Petr pointed this out as well :-) I will just include
> > moduleloader.h for the apply_relocate_add() declaration.
> > 
> > It also looks like we have some disagreement over where to put
> > klp_prepare_patch_module(), either in livepatch/core.c (and add the
> > function declaration in livepatch.h, and have module.c include
> > livepatch.h) or in kernel/module.c, keeping the
> > klp_prepare_patch_module() declaration in module.h. Maybe Rusty can
> > provide some input.

Yes, there are several ways how to do it. Maybe the best would be some 
generic way in kernel/module.c. I am not sure if there will be another 
user of this in the future but nevertheless. It would also allow us to 
somehow solve the issues mentioned below. Thus, klp_prepare_patch_module 
is inappropriate name and it should be for example just preserve_load_info 
(or more general if needed) and it should be in kernel/module.c.

> > >> /* Given an address, look for it in the exception tables. */
> > >>diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > >>index 6e53441..087a8c7 100644
> > >>--- a/kernel/livepatch/core.c
> > >>+++ b/kernel/livepatch/core.c
> > >>@@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
> > >> 	.priority = INT_MIN+1, /* called late but before ftrace notifier */
> > >> };
> > >>
> > >>+/*
> > >>+ * Save necessary information from info in order to be able to
> > >>+ * patch modules that might be loaded later
> > >>+ */
> > >>+void klp_prepare_patch_module(struct module *mod, struct load_info *info)
> > >>+{
> > >>+	Elf_Shdr *symsect;
> > >>+
> > >>+	symsect = info->sechdrs + info->index.sym;
> > >>+	/* update sh_addr to point to symtab */
> > >>+	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
> > >>+
> > >>+	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
> > >>+	memcpy(mod->info, info, sizeof(*info));
> > >>+
> > >>+}
> > >
> > >What about arch-specific 'struct mod_arch_specific'? We need to preserve
> > >it somewhere as well for s390x and other non-x86 architectures.
> > 
> > Ah! Thank you for catching this, I overlooked this important detail.
> > Yes, we do need to save the arch-specific struct. We would be in
> > trouble for s390 relocs if we didn't. I am trying to think of a way to
> > save this information for s390, since s390's module_finalize() frees
> > mod->arch.syminfo, which we definitely need in order for the call to
> > apply_relocate_add() to work. Maybe we can add an extra call right
> > before module_finalize() that will do some livepatch-specific
> > processing and copy this information (this would be in
> > post_relocation() in kernel/module.c). Perhaps this patchset cannot be
> > entirely free of arch-specific code after all :-( Still thinking.

Well, mod_arch_specific is defined as each architecture needs. So for x86 
it is empty. It is arch-agnostic in this way and we can deal with it as 
"a black box". We just need it not to be freed in module_finalize. And...

> I think about adding a flag somewhere, e.g. mod->preserve_relocs.
> It might be set in simplify_symbols() when SHN_LIVEPATCH is found.
> It might be checked when freeing the needed structures in both
> the generic and arch-specific code.

...that is the reason why some sort of flag seems to be necessary. It 
could be set when livepatch is set in modinfo. We would use it for 
preserving both load_info and mod_arch_specific struct (in some form) and 
for...

> > >>+#ifdef CONFIG_LIVEPATCH
> > >>+	/*
> > >>+	 * Save sechdrs, indices, and other data from info
> > >>+	 * in order to patch to-be-loaded modules.
> > >>+	 * Do not call free_copy() for livepatch modules.
> > >>+	 */
> > >>+	if (get_modinfo((struct load_info *)info, "livepatch"))
> > >>+		klp_prepare_patch_module(mod, info);
> > >>+	else
> > >>+		free_copy(info);
> > >>+#else
> > >> 	/* Get rid of temporary copy. */
> > >> 	free_copy(info);
> > >>+#endif
> > >
> > >Maybe I am missing something but isn't it necessary to call vfree() on
> > >info somewhere in the end?
> > 
> > So free_copy() will call vfree(info->hdr), except in livepatch modules
> > we want to keep all the elf section information stored there, so we
> > avoid calling free_copy(), As for the info struct itself, if you look
> > at the init_module and finit_module syscall definitions in
> > kernel/module.c, you will see that info is actually a local function
> > variable, simply passed in to the call to load_module(), and will be
> > automatically deallocated when the syscall returns. :-) No need to
> > explicitly free info.
> 
> We still have to free the copied or preserved structures when
> the module is unloaded.

...freeing what we allocated. We need to free info->hdr somewhere if not 
here and also mod_arch_specific struct where the patch module is removed. 
This would unfortunately lead to changes in arch-specific code in 
module.c. For example in arch/s390/kernel/module.c there is vfree call on 
part of mod_arch_specific in module_finalize. We would call it only if the 
flag mentioned above is not set and at the same time we would need to call 
it when the patch module is being removed.

Hm, this is (again) getting complicated and ugly. Is there someone who can 
simplify things? Josh? :)

Miroslav

Re: module: save load_info for livepatch modules

[Miroslav Benes]

On Thu, 12 Nov 2015, Petr Mladek wrote:

> On Wed 2015-11-11 23:44:08, Jessica Yu wrote:
> > +++ Petr Mladek [11/11/15 15:31 +0100]:
> > >On Mon 2015-11-09 23:45:52, Jessica Yu wrote:
> > >>diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > >>index 6e53441..087a8c7 100644
> > >>--- a/kernel/livepatch/core.c
> > >>+++ b/kernel/livepatch/core.c
> > >>@@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
> > >> 	.priority = INT_MIN+1, /* called late but before ftrace notifier */
> > >> };
> > >>
> > >>+/*
> > >>+ * Save necessary information from info in order to be able to
> > >>+ * patch modules that might be loaded later
> > >>+ */
> > >>+void klp_prepare_patch_module(struct module *mod, struct load_info *info)
> > >>+{
> > >>+	Elf_Shdr *symsect;
> > >>+
> > >>+	symsect = info->sechdrs + info->index.sym;
> > >>+	/* update sh_addr to point to symtab */
> > >>+	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
> > >
> > >Is livepatch the only user of this value? By other words, is this safe?
> > 
> > I think it is safe to say yes. klp_prepare_patch_module() is only
> > called at the very end of load_module(), right before
> > do_init_module(). Normally, at that point, info->hdr will have already
> > been freed by free_copy() along with the elf section information
> > associated with it. But if we have a livepatch module, we don't free.
> > So we should be the very last user, and there should be nobody
> > utilizing the memory associated with the load_info struct anymore at
> > that point.
> 
> I see. It looks safe at this point. But still I wonder if it would be
> possible to calculate this later in the livepatch code. It will allow
> to potentially use the info structure also by other subsystem.
> 
> BTW: Where is "sh_addr" value used, please? I see it used only
> in the third patch as info->sechdrs[relindex].sh_addr. But it is
> an array. I am not sure if it is the same variable.

Jessica, why do we need to update sh_addr for symtab? It is not clear to 
me.

Miroslav

Re: module: save load_info for livepatch modules

[Miroslav Benes]

On Thu, 12 Nov 2015, Petr Mladek wrote:

> On Wed 2015-11-11 23:44:08, Jessica Yu wrote:
> > +++ Petr Mladek [11/11/15 15:31 +0100]:
> > >On Mon 2015-11-09 23:45:52, Jessica Yu wrote:
> > >>diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > >>index 6e53441..087a8c7 100644
> > >>--- a/kernel/livepatch/core.c
> > >>+++ b/kernel/livepatch/core.c
> > >>@@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
> > >> 	.priority = INT_MIN+1, /* called late but before ftrace notifier */
> > >> };
> > >>
> > >>+/*
> > >>+ * Save necessary information from info in order to be able to
> > >>+ * patch modules that might be loaded later
> > >>+ */
> > >>+void klp_prepare_patch_module(struct module *mod, struct load_info *info)
> > >>+{
> > >>+	Elf_Shdr *symsect;
> > >>+
> > >>+	symsect = info->sechdrs + info->index.sym;
> > >>+	/* update sh_addr to point to symtab */
> > >>+	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
> > >
> > >Is livepatch the only user of this value? By other words, is this safe?
> > 
> > I think it is safe to say yes. klp_prepare_patch_module() is only
> > called at the very end of load_module(), right before
> > do_init_module(). Normally, at that point, info->hdr will have already
> > been freed by free_copy() along with the elf section information
> > associated with it. But if we have a livepatch module, we don't free.
> > So we should be the very last user, and there should be nobody
> > utilizing the memory associated with the load_info struct anymore at
> > that point.
> 
> I see. It looks safe at this point. But still I wonder if it would be
> possible to calculate this later in the livepatch code. It will allow
> to potentially use the info structure also by other subsystem.
> 
> BTW: Where is "sh_addr" value used, please? I see it used only
> in the third patch as info->sechdrs[relindex].sh_addr. But it is
> an array. I am not sure if it is the same variable.

Jessica, why do we need to update sh_addr for symtab? It is not clear to 
me.

Miroslav

Re: module: save load_info for livepatch modules

[Petr Mladek]

On Thu 2015-11-12 14:22:28, Miroslav Benes wrote:
> On Thu, 12 Nov 2015, Petr Mladek wrote:
> > > >Maybe I am missing something but isn't it necessary to call vfree() on
> > > >info somewhere in the end?
> > > 
> > > So free_copy() will call vfree(info->hdr), except in livepatch modules
> > > we want to keep all the elf section information stored there, so we
> > > avoid calling free_copy(), As for the info struct itself, if you look
> > > at the init_module and finit_module syscall definitions in
> > > kernel/module.c, you will see that info is actually a local function
> > > variable, simply passed in to the call to load_module(), and will be
> > > automatically deallocated when the syscall returns. :-) No need to
> > > explicitly free info.
> > 
> > We still have to free the copied or preserved structures when
> > the module is unloaded.
> 
> ...freeing what we allocated. We need to free info->hdr somewhere if not 
> here and also mod_arch_specific struct where the patch module is removed. 
> This would unfortunately lead to changes in arch-specific code in 
> module.c. For example in arch/s390/kernel/module.c there is vfree call on 
> part of mod_arch_specific in module_finalize. We would call it only if the 
> flag mentioned above is not set and at the same time we would need to call 
> it when the patch module is being removed.

Sigh, I am afraid that the flag is not enough. IMHO, we need to split
the load finalizing functions into two pieces. One will be always
called when the module load is finalized. The other part will free
the load_info. It will be called either when the load is finalized or
when the module is unloaded, depending on if we want to preserve
the load_info.

Sigh, it is getting complicated. But let's see how it looks in reality.

Best Regards,
Petr

Re: module: save load_info for livepatch modules

[Petr Mladek]

On Thu 2015-11-12 14:22:28, Miroslav Benes wrote:
> On Thu, 12 Nov 2015, Petr Mladek wrote:
> > > >Maybe I am missing something but isn't it necessary to call vfree() on
> > > >info somewhere in the end?
> > > 
> > > So free_copy() will call vfree(info->hdr), except in livepatch modules
> > > we want to keep all the elf section information stored there, so we
> > > avoid calling free_copy(), As for the info struct itself, if you look
> > > at the init_module and finit_module syscall definitions in
> > > kernel/module.c, you will see that info is actually a local function
> > > variable, simply passed in to the call to load_module(), and will be
> > > automatically deallocated when the syscall returns. :-) No need to
> > > explicitly free info.
> > 
> > We still have to free the copied or preserved structures when
> > the module is unloaded.
> 
> ...freeing what we allocated. We need to free info->hdr somewhere if not 
> here and also mod_arch_specific struct where the patch module is removed. 
> This would unfortunately lead to changes in arch-specific code in 
> module.c. For example in arch/s390/kernel/module.c there is vfree call on 
> part of mod_arch_specific in module_finalize. We would call it only if the 
> flag mentioned above is not set and at the same time we would need to call 
> it when the patch module is being removed.

Sigh, I am afraid that the flag is not enough. IMHO, we need to split
the load finalizing functions into two pieces. One will be always
called when the module load is finalized. The other part will free
the load_info. It will be called either when the load is finalized or
when the module is unloaded, depending on if we want to preserve
the load_info.

Sigh, it is getting complicated. But let's see how it looks in reality.

Best Regards,
Petr

Re: livepatch: reuse module loader code to write relocations

[Miroslav Benes]

On Wed, 11 Nov 2015, Jessica Yu wrote:

> +++ Miroslav Benes [11/11/15 15:30 +0100]:
> > On Mon, 9 Nov 2015, Jessica Yu wrote:
> >
> > So I guess we don't need klp_reloc anymore.
> 
> Yes, that's correct. I am noticing just now that I forgot to remove
> the klp_reloc struct definition from livepatch.h. That change will be
> reflected in v2...
> 
> > If true, we should really
> > start thinking about proper documentation because there are going to be
> > plenty of assumptions about a patch module and we need to have it written
> > somewhere. Especially how the relocation sections look like.
> 
> Agreed. As a first step the patch module format can perhaps be
> documented somewhere. Perhaps it's time we create
> Documentation/livepatch/? :-)

Yes, I think so.

> > > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > > index 087a8c7..26c419f 100644
> > > --- a/kernel/livepatch/core.c
> > > +++ b/kernel/livepatch/core.c
> > > @@ -28,6 +28,8 @@
> > >  #include <linux/list.h>
> > >  #include <linux/kallsyms.h>
> > >  #include <linux/livepatch.h>
> > > +#include <linux/elf.h>
> > > +#include <asm/cacheflush.h>
> > > 
> > >  /**
> > >   * struct klp_ops - structure for tracking registered ftrace ops structs
> > > @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module
> > > *pmod, const char *name,
> > >  }
> > > 
> > >  static int klp_write_object_relocations(struct module *pmod,
> > > -					struct klp_object *obj)
> > > +					struct klp_object *obj,
> > > +					struct klp_patch *patch)
> > >  {
> > > -	int ret;
> > > -	struct klp_reloc *reloc;
> > > +	int relindex, num_relas;
> > > +	int i, ret = 0;
> > > +	unsigned long addr;
> > > +	unsigned int bind;
> > > +	char *symname;
> > > +	struct klp_reloc_sec *reloc_sec;
> > > +	struct load_info *info;
> > > +	Elf_Rela *rela;
> > > +	Elf_Sym *sym, *symtab;
> > > +	Elf_Shdr *symsect;
> > > 
> > >  	if (WARN_ON(!klp_is_object_loaded(obj)))
> > >  		return -EINVAL;
> > > 
> > > -	if (WARN_ON(!obj->relocs))
> > > -		return -EINVAL;
> > > -
> > > -	for (reloc = obj->relocs; reloc->name; reloc++) {
> > > -		if (!klp_is_module(obj)) {
> > > -			ret = klp_verify_vmlinux_symbol(reloc->name,
> > > -							reloc->val);
> > > -			if (ret)
> > > -				return ret;
> > > -		} else {
> > > -			/* module, reloc->val needs to be discovered */
> > > -			if (reloc->external)
> > > -				ret = klp_find_external_symbol(pmod,
> > > -							       reloc->name,
> > > -							       &reloc->val);
> > > -			else
> > > -				ret = klp_find_object_symbol(obj->mod->name,
> > > -							     reloc->name,
> > > -							     &reloc->val);
> > > -			if (ret)
> > > -				return ret;
> > > -		}
> > > -		ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc,
> > > -					     reloc->val + reloc->addend);
> > > -		if (ret) {
> > > -			pr_err("relocation failed for symbol '%s' at 0x%016lx
> > > (%d)\n",
> > > -			       reloc->name, reloc->val, ret);
> > > -			return ret;
> > > +	info = pmod->info;
> > > +	symsect = info->sechdrs + info->index.sym;
> > > +	symtab = (void *)info->hdr + symsect->sh_offset;
> > > +
> > > +	/* For each __klp_rela section for this object */
> > > +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
> > > +		relindex = reloc_sec->index;
> > > +		num_relas = info->sechdrs[relindex].sh_size /
> > > sizeof(Elf_Rela);
> > > +		rela = (Elf_Rela *) info->sechdrs[relindex].sh_addr;
> > > +
> > > +		/* For each rela in this __klp_rela section */
> > > +		for (i = 0; i < num_relas; i++, rela++) {
> > > +			sym = symtab + ELF_R_SYM(rela->r_info);
> > > +			symname = info->strtab + sym->st_name;
> > > +			bind = ELF_ST_BIND(sym->st_info);
> > > +
> > > +			if (sym->st_shndx == SHN_LIVEPATCH) {
> > > +				if (bind == STB_LIVEPATCH_EXT)
> > > +					ret = klp_find_external_symbol(pmod,
> > > symname, &addr);
> > > +				else
> > > +					ret =
> > > klp_find_object_symbol(obj->name, symname, &addr);
> > > +				if (ret)
> > > +					return ret;
> > > +				sym->st_value = addr;
> > > +			}
> > >  		}
> > > +		ret = apply_relocate_add(info->sechdrs, info->strtab,
> > > +					 info->index.sym, relindex, pmod);
> > >  	}
> > > 
> > > -	return 0;
> > > +	return ret;
> > >  }
> > 
> > Looking at this... do we even need reloc_secs in klp_object? Question is
> > whether we need more than one dynrela section for an object. If not then
> > the binding between klp_reloc_sec and an object is the only relevant thing
> > in the structure, be it index or objname. So we can replace the
> > list of structures with just the index in klp_object, or get rid of it
> > completely and rely on the name of dynrela section be something like
> > __klp_rela_{objname}.
> 
> Hm, you bring up a good point. I think theoretically yes, it is
> possible to just have one klp_reloc_sec for each object and therefore
> a list is not required (I have not checked yet how difficult it would
> be to implement this on the kpatch-build side of things).  However,
> considering the final format of the patch module, I think it is
> semantically clearer to leave it as a list, and for each object to
> possibly have more than one __klp_rela section.
> 
> For example, say we are patching two functions in ext4. In my
> resulting kpatch module I will have two __klp_rela_ext4 sections, and
> they might look like this when we run readelf --sections:
> 
> [34] __klp_rela_ext4.text.ext4_attr_store RELA ...
> [35] __klp_rela_ext4.text.ext4_attr_show RELA ...
> 
> Then these two klp rela sections end up as two elements in the
> reloc_secs list for the ext4 patch object. I think this way, we can
> better tell which rela is being applied to what function. Might be
> easier to understand what's happening from the developer's point of
> view.
> 
> > You see, we go through elf sections here which were preserved by module
> > loader. We even have relevant sections marked with SHF_RELA_LIVEPATCH. So
> > maybe all the stuff around klp_reloc_sec is not necessary.
> > 
> > Thoughts?
> 
> Ah, so this is where descriptive comments and documentation might have
> been useful :-) So I think we will still need to keep the
> klp_reloc_sec struct to help the patch module initialize. Though the
> name and objname fields aren't used in this patchset, they are used in
> the kpatch patch module code [1], where we iterate through each elf
> section, find the ones marked with SHF_RELA_LIVEPATCH, set the
> klp_reloc_sec's objname (which we find from the "name" field,
> formatted as __klp_rela_{objname}.text..). Once we have the objname
> set, we can then find the object to attach the reloc_sec to (i.e. add
> it to its list of reloc_secs).
> 
> Hope that clears some things up.

Ok, I'll try to explain myself and it is gonna be long. I'll try to 
describe how we deal with dynrelas in klp today, how you use it in kpatch 
(and this is the place where my knowledge can be wrong or obsolete), what 
you propose and what I'd like to propose.

1. First let's look on what we have now.

We have a patch module in which there is a section with all dynrelas 
needed to be resolved (it was like this in kpatch some time ago and maybe 
it is different now so just have a patience, I'll get to it). In the init 
function of the module kpatch builds all the relevant info from dynrela 
section. It goes through it, creates an array of klp_reloc for each object 
and includes each dynrela record with an objname to the array of 
klp_object with that objname. Later when we need to apply relocations for 
patched object we just go through the list (array) of its dynrelas in 
klp_object and call our arch-specific klp_write_module_reloc().

Now this was probably changed in kpatch and you do not have one dynrela 
section but one dynrela section for each patched function. Is that 
correct? (and can you tell us what the reason for the change was? It might 
be crucial because I might be missing something.). Which leads us to your 
proposal...

2. So we have several dynrela section for one patched object. During init 
function in a patch module kpatch once again builds needed structures from 
these sections. Now they are called klp_reloc_sec and contain different 
kind of info. There is no val, loc and such, only name of the symbol, 
objname and index to dynrela section in ELF. So when you need to write 
relocations for the patched object you go through all relevant dynrela 
sections (because they are stored in the klp_object), all dynrela records 
in each section and you resolve the undefined symbols. All needed info is 
stored in ELF. Then you just call apply_relocate_add().

3. I propose to go one step further. I think we don't need klp_reloc_sec 
if there is only one dynrela section for patched object (and I currently 
cannot see why this is not possible. It is possible even with one dynrela 
section for whole patch module, that is for all patched objects.).

In my proposal there would be nothing done in init function of the patched 
module (albeit some optimization mentioned later). When you call 
klp_write_object_relocations you would go through all ELF section and find 
the relevant one for the object (it is marked with SHF_RELA_LIVEPATCH and 
objname is in the name of the section. It is the same thing you do in 2. 
in the init function.). Then you go through all dynrela records in the 
section, you do the same things which you do in the proposed patch above, 
and call apply_relocate_add.

Now it would be crazy to go through all sections each time 
klp_write_object_relocations is called (maybe it does not even matter but 
nevertheless). So klp_object could have an index to its ELF dynrela 
section. This index can be retrieved in the init function the same way you 
do all the stuff with klp_reloc_sec.

If you insisted on more than one dynrela section for a patched object we 
could have an array of indices there. Or whatever.

It is almost the same as your proposal but in my opinion somewhat nicer. 
We just use the info stored in ELF directly without unnecessary middle 
layer (== klp_reloc_sec).

Does it make sense? I hope it does. Would it work?

Miroslav

Re: livepatch: reuse module loader code to write relocations

[Miroslav Benes]

On Wed, 11 Nov 2015, Jessica Yu wrote:

> +++ Miroslav Benes [11/11/15 15:30 +0100]:
> > On Mon, 9 Nov 2015, Jessica Yu wrote:
> >
> > So I guess we don't need klp_reloc anymore.
> 
> Yes, that's correct. I am noticing just now that I forgot to remove
> the klp_reloc struct definition from livepatch.h. That change will be
> reflected in v2...
> 
> > If true, we should really
> > start thinking about proper documentation because there are going to be
> > plenty of assumptions about a patch module and we need to have it written
> > somewhere. Especially how the relocation sections look like.
> 
> Agreed. As a first step the patch module format can perhaps be
> documented somewhere. Perhaps it's time we create
> Documentation/livepatch/? :-)

Yes, I think so.

> > > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > > index 087a8c7..26c419f 100644
> > > --- a/kernel/livepatch/core.c
> > > +++ b/kernel/livepatch/core.c
> > > @@ -28,6 +28,8 @@
> > >  #include <linux/list.h>
> > >  #include <linux/kallsyms.h>
> > >  #include <linux/livepatch.h>
> > > +#include <linux/elf.h>
> > > +#include <asm/cacheflush.h>
> > > 
> > >  /**
> > >   * struct klp_ops - structure for tracking registered ftrace ops structs
> > > @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module
> > > *pmod, const char *name,
> > >  }
> > > 
> > >  static int klp_write_object_relocations(struct module *pmod,
> > > -					struct klp_object *obj)
> > > +					struct klp_object *obj,
> > > +					struct klp_patch *patch)
> > >  {
> > > -	int ret;
> > > -	struct klp_reloc *reloc;
> > > +	int relindex, num_relas;
> > > +	int i, ret = 0;
> > > +	unsigned long addr;
> > > +	unsigned int bind;
> > > +	char *symname;
> > > +	struct klp_reloc_sec *reloc_sec;
> > > +	struct load_info *info;
> > > +	Elf_Rela *rela;
> > > +	Elf_Sym *sym, *symtab;
> > > +	Elf_Shdr *symsect;
> > > 
> > >  	if (WARN_ON(!klp_is_object_loaded(obj)))
> > >  		return -EINVAL;
> > > 
> > > -	if (WARN_ON(!obj->relocs))
> > > -		return -EINVAL;
> > > -
> > > -	for (reloc = obj->relocs; reloc->name; reloc++) {
> > > -		if (!klp_is_module(obj)) {
> > > -			ret = klp_verify_vmlinux_symbol(reloc->name,
> > > -							reloc->val);
> > > -			if (ret)
> > > -				return ret;
> > > -		} else {
> > > -			/* module, reloc->val needs to be discovered */
> > > -			if (reloc->external)
> > > -				ret = klp_find_external_symbol(pmod,
> > > -							       reloc->name,
> > > -							       &reloc->val);
> > > -			else
> > > -				ret = klp_find_object_symbol(obj->mod->name,
> > > -							     reloc->name,
> > > -							     &reloc->val);
> > > -			if (ret)
> > > -				return ret;
> > > -		}
> > > -		ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc,
> > > -					     reloc->val + reloc->addend);
> > > -		if (ret) {
> > > -			pr_err("relocation failed for symbol '%s' at 0x%016lx
> > > (%d)\n",
> > > -			       reloc->name, reloc->val, ret);
> > > -			return ret;
> > > +	info = pmod->info;
> > > +	symsect = info->sechdrs + info->index.sym;
> > > +	symtab = (void *)info->hdr + symsect->sh_offset;
> > > +
> > > +	/* For each __klp_rela section for this object */
> > > +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
> > > +		relindex = reloc_sec->index;
> > > +		num_relas = info->sechdrs[relindex].sh_size /
> > > sizeof(Elf_Rela);
> > > +		rela = (Elf_Rela *) info->sechdrs[relindex].sh_addr;
> > > +
> > > +		/* For each rela in this __klp_rela section */
> > > +		for (i = 0; i < num_relas; i++, rela++) {
> > > +			sym = symtab + ELF_R_SYM(rela->r_info);
> > > +			symname = info->strtab + sym->st_name;
> > > +			bind = ELF_ST_BIND(sym->st_info);
> > > +
> > > +			if (sym->st_shndx == SHN_LIVEPATCH) {
> > > +				if (bind == STB_LIVEPATCH_EXT)
> > > +					ret = klp_find_external_symbol(pmod,
> > > symname, &addr);
> > > +				else
> > > +					ret =
> > > klp_find_object_symbol(obj->name, symname, &addr);
> > > +				if (ret)
> > > +					return ret;
> > > +				sym->st_value = addr;
> > > +			}
> > >  		}
> > > +		ret = apply_relocate_add(info->sechdrs, info->strtab,
> > > +					 info->index.sym, relindex, pmod);
> > >  	}
> > > 
> > > -	return 0;
> > > +	return ret;
> > >  }
> > 
> > Looking at this... do we even need reloc_secs in klp_object? Question is
> > whether we need more than one dynrela section for an object. If not then
> > the binding between klp_reloc_sec and an object is the only relevant thing
> > in the structure, be it index or objname. So we can replace the
> > list of structures with just the index in klp_object, or get rid of it
> > completely and rely on the name of dynrela section be something like
> > __klp_rela_{objname}.
> 
> Hm, you bring up a good point. I think theoretically yes, it is
> possible to just have one klp_reloc_sec for each object and therefore
> a list is not required (I have not checked yet how difficult it would
> be to implement this on the kpatch-build side of things).  However,
> considering the final format of the patch module, I think it is
> semantically clearer to leave it as a list, and for each object to
> possibly have more than one __klp_rela section.
> 
> For example, say we are patching two functions in ext4. In my
> resulting kpatch module I will have two __klp_rela_ext4 sections, and
> they might look like this when we run readelf --sections:
> 
> [34] __klp_rela_ext4.text.ext4_attr_store RELA ...
> [35] __klp_rela_ext4.text.ext4_attr_show RELA ...
> 
> Then these two klp rela sections end up as two elements in the
> reloc_secs list for the ext4 patch object. I think this way, we can
> better tell which rela is being applied to what function. Might be
> easier to understand what's happening from the developer's point of
> view.
> 
> > You see, we go through elf sections here which were preserved by module
> > loader. We even have relevant sections marked with SHF_RELA_LIVEPATCH. So
> > maybe all the stuff around klp_reloc_sec is not necessary.
> > 
> > Thoughts?
> 
> Ah, so this is where descriptive comments and documentation might have
> been useful :-) So I think we will still need to keep the
> klp_reloc_sec struct to help the patch module initialize. Though the
> name and objname fields aren't used in this patchset, they are used in
> the kpatch patch module code [1], where we iterate through each elf
> section, find the ones marked with SHF_RELA_LIVEPATCH, set the
> klp_reloc_sec's objname (which we find from the "name" field,
> formatted as __klp_rela_{objname}.text..). Once we have the objname
> set, we can then find the object to attach the reloc_sec to (i.e. add
> it to its list of reloc_secs).
> 
> Hope that clears some things up.

Ok, I'll try to explain myself and it is gonna be long. I'll try to 
describe how we deal with dynrelas in klp today, how you use it in kpatch 
(and this is the place where my knowledge can be wrong or obsolete), what 
you propose and what I'd like to propose.

1. First let's look on what we have now.

We have a patch module in which there is a section with all dynrelas 
needed to be resolved (it was like this in kpatch some time ago and maybe 
it is different now so just have a patience, I'll get to it). In the init 
function of the module kpatch builds all the relevant info from dynrela 
section. It goes through it, creates an array of klp_reloc for each object 
and includes each dynrela record with an objname to the array of 
klp_object with that objname. Later when we need to apply relocations for 
patched object we just go through the list (array) of its dynrelas in 
klp_object and call our arch-specific klp_write_module_reloc().

Now this was probably changed in kpatch and you do not have one dynrela 
section but one dynrela section for each patched function. Is that 
correct? (and can you tell us what the reason for the change was? It might 
be crucial because I might be missing something.). Which leads us to your 
proposal...

2. So we have several dynrela section for one patched object. During init 
function in a patch module kpatch once again builds needed structures from 
these sections. Now they are called klp_reloc_sec and contain different 
kind of info. There is no val, loc and such, only name of the symbol, 
objname and index to dynrela section in ELF. So when you need to write 
relocations for the patched object you go through all relevant dynrela 
sections (because they are stored in the klp_object), all dynrela records 
in each section and you resolve the undefined symbols. All needed info is 
stored in ELF. Then you just call apply_relocate_add().

3. I propose to go one step further. I think we don't need klp_reloc_sec 
if there is only one dynrela section for patched object (and I currently 
cannot see why this is not possible. It is possible even with one dynrela 
section for whole patch module, that is for all patched objects.).

In my proposal there would be nothing done in init function of the patched 
module (albeit some optimization mentioned later). When you call 
klp_write_object_relocations you would go through all ELF section and find 
the relevant one for the object (it is marked with SHF_RELA_LIVEPATCH and 
objname is in the name of the section. It is the same thing you do in 2. 
in the init function.). Then you go through all dynrela records in the 
section, you do the same things which you do in the proposed patch above, 
and call apply_relocate_add.

Now it would be crazy to go through all sections each time 
klp_write_object_relocations is called (maybe it does not even matter but 
nevertheless). So klp_object could have an index to its ELF dynrela 
section. This index can be retrieved in the init function the same way you 
do all the stuff with klp_reloc_sec.

If you insisted on more than one dynrela section for a patched object we 
could have an array of indices there. Or whatever.

It is almost the same as your proposal but in my opinion somewhat nicer. 
We just use the info stored in ELF directly without unnecessary middle 
layer (== klp_reloc_sec).

Does it make sense? I hope it does. Would it work?

Miroslav

Re: [RFC PATCH 1/5] elf: add livepatch-specific elf constants

[Josh Poimboeuf]

On Mon, Nov 09, 2015 at 11:45:51PM -0500, Jessica Yu wrote:
> Add livepatch elf reloc section flag, livepatch symbol bind
> and section index
> 
> Signed-off-by: Jessica Yu <jeyu@redhat.com>
> ---
>  include/uapi/linux/elf.h | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h
> index 71e1d0e..967ce1b 100644
> --- a/include/uapi/linux/elf.h
> +++ b/include/uapi/linux/elf.h
> @@ -118,6 +118,7 @@ typedef __s64	Elf64_Sxword;
>  #define STB_LOCAL  0
>  #define STB_GLOBAL 1
>  #define STB_WEAK   2
> +#define STB_LIVEPATCH_EXT 11
>  
>  #define STT_NOTYPE  0
>  #define STT_OBJECT  1
> @@ -286,6 +287,7 @@ typedef struct elf64_phdr {
>  #define SHF_ALLOC	0x2
>  #define SHF_EXECINSTR	0x4
>  #define SHF_MASKPROC	0xf0000000
> +#define SHF_RELA_LIVEPATCH  0x4000000
>  
>  /* special section indexes */
>  #define SHN_UNDEF	0
> @@ -295,6 +297,7 @@ typedef struct elf64_phdr {
>  #define SHN_ABS		0xfff1
>  #define SHN_COMMON	0xfff2
>  #define SHN_HIRESERVE	0xffff
> +#define SHN_LIVEPATCH 0xff21
>   
>  typedef struct elf32_shdr {
>    Elf32_Word	sh_name;

For all of these it would be good to re-indent the surrounding macros so
their values are all vertically aligned with the new LIVEPATCH ones.

-- 
Josh

Re: [RFC PATCH 1/5] elf: add livepatch-specific elf constants

[Josh Poimboeuf]

On Mon, Nov 09, 2015 at 11:45:51PM -0500, Jessica Yu wrote:
> Add livepatch elf reloc section flag, livepatch symbol bind
> and section index
> 
> Signed-off-by: Jessica Yu <jeyu@redhat.com>
> ---
>  include/uapi/linux/elf.h | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h
> index 71e1d0e..967ce1b 100644
> --- a/include/uapi/linux/elf.h
> +++ b/include/uapi/linux/elf.h
> @@ -118,6 +118,7 @@ typedef __s64	Elf64_Sxword;
>  #define STB_LOCAL  0
>  #define STB_GLOBAL 1
>  #define STB_WEAK   2
> +#define STB_LIVEPATCH_EXT 11
>  
>  #define STT_NOTYPE  0
>  #define STT_OBJECT  1
> @@ -286,6 +287,7 @@ typedef struct elf64_phdr {
>  #define SHF_ALLOC	0x2
>  #define SHF_EXECINSTR	0x4
>  #define SHF_MASKPROC	0xf0000000
> +#define SHF_RELA_LIVEPATCH  0x4000000

Writing the value with leading zeros (0x04000000) would it more
readable.

Also the OS-specific range mask (SHF_MASKOS) is 0x0ff00000.  Any reason
you went with 0x04000000 as opposed to the first value in the range
(0x00100000)?  I don't see anybody else using that value.

>  /* special section indexes */
>  #define SHN_UNDEF	0
> @@ -295,6 +297,7 @@ typedef struct elf64_phdr {
>  #define SHN_ABS		0xfff1
>  #define SHN_COMMON	0xfff2
>  #define SHN_HIRESERVE	0xffff
> +#define SHN_LIVEPATCH 0xff21

Similar question here, why not use 0xff20 (SHN_LOOS)?

-- 
Josh

Re: module: save load_info for livepatch modules

[Josh Poimboeuf]

On Thu, Nov 12, 2015 at 04:03:45PM +0100, Petr Mladek wrote:
> On Thu 2015-11-12 14:22:28, Miroslav Benes wrote:
> > On Thu, 12 Nov 2015, Petr Mladek wrote:
> > > > >Maybe I am missing something but isn't it necessary to call vfree() on
> > > > >info somewhere in the end?
> > > > 
> > > > So free_copy() will call vfree(info->hdr), except in livepatch modules
> > > > we want to keep all the elf section information stored there, so we
> > > > avoid calling free_copy(), As for the info struct itself, if you look
> > > > at the init_module and finit_module syscall definitions in
> > > > kernel/module.c, you will see that info is actually a local function
> > > > variable, simply passed in to the call to load_module(), and will be
> > > > automatically deallocated when the syscall returns. :-) No need to
> > > > explicitly free info.
> > > 
> > > We still have to free the copied or preserved structures when
> > > the module is unloaded.
> > 
> > ...freeing what we allocated. We need to free info->hdr somewhere if not 
> > here and also mod_arch_specific struct where the patch module is removed. 
> > This would unfortunately lead to changes in arch-specific code in 
> > module.c. For example in arch/s390/kernel/module.c there is vfree call on 
> > part of mod_arch_specific in module_finalize. We would call it only if the 
> > flag mentioned above is not set and at the same time we would need to call 
> > it when the patch module is being removed.
> 
> Sigh, I am afraid that the flag is not enough. IMHO, we need to split
> the load finalizing functions into two pieces. One will be always
> called when the module load is finalized. The other part will free
> the load_info. It will be called either when the load is finalized or
> when the module is unloaded, depending on if we want to preserve
> the load_info.
> 
> Sigh, it is getting complicated. But let's see how it looks in reality.

At the other end of the spectrum, we could do the simplest thing
possible: _always_ save the data (even if CONFIG_LIVEPATCH is disabled).

(gdb) print sizeof(*info)
$3 = 96
(gdb) p sizeof(*info->hdr)
$4 = 64
s390 mod_arch_syminfo struct: 24 bytes by my reckoning.

So between info, info->hdr, and s390 mod_arch_syminfo, we're talking
about 184 bytes on s390 and 160 bytes on x86_64.  That seems like
peanuts compared to the size of a typical module.  The benefit is that
the code would be simpler because we don't have any special cases and
the structs would automatically get freed with the module struct when
the module gets unloaded.

-- 
Josh

Re: module: save load_info for livepatch modules

[Josh Poimboeuf]

On Thu, Nov 12, 2015 at 04:03:45PM +0100, Petr Mladek wrote:
> On Thu 2015-11-12 14:22:28, Miroslav Benes wrote:
> > On Thu, 12 Nov 2015, Petr Mladek wrote:
> > > > >Maybe I am missing something but isn't it necessary to call vfree() on
> > > > >info somewhere in the end?
> > > > 
> > > > So free_copy() will call vfree(info->hdr), except in livepatch modules
> > > > we want to keep all the elf section information stored there, so we
> > > > avoid calling free_copy(), As for the info struct itself, if you look
> > > > at the init_module and finit_module syscall definitions in
> > > > kernel/module.c, you will see that info is actually a local function
> > > > variable, simply passed in to the call to load_module(), and will be
> > > > automatically deallocated when the syscall returns. :-) No need to
> > > > explicitly free info.
> > > 
> > > We still have to free the copied or preserved structures when
> > > the module is unloaded.
> > 
> > ...freeing what we allocated. We need to free info->hdr somewhere if not 
> > here and also mod_arch_specific struct where the patch module is removed. 
> > This would unfortunately lead to changes in arch-specific code in 
> > module.c. For example in arch/s390/kernel/module.c there is vfree call on 
> > part of mod_arch_specific in module_finalize. We would call it only if the 
> > flag mentioned above is not set and at the same time we would need to call 
> > it when the patch module is being removed.
> 
> Sigh, I am afraid that the flag is not enough. IMHO, we need to split
> the load finalizing functions into two pieces. One will be always
> called when the module load is finalized. The other part will free
> the load_info. It will be called either when the load is finalized or
> when the module is unloaded, depending on if we want to preserve
> the load_info.
> 
> Sigh, it is getting complicated. But let's see how it looks in reality.

At the other end of the spectrum, we could do the simplest thing
possible: _always_ save the data (even if CONFIG_LIVEPATCH is disabled).

(gdb) print sizeof(*info)
$3 = 96
(gdb) p sizeof(*info->hdr)
$4 = 64
s390 mod_arch_syminfo struct: 24 bytes by my reckoning.

So between info, info->hdr, and s390 mod_arch_syminfo, we're talking
about 184 bytes on s390 and 160 bytes on x86_64.  That seems like
peanuts compared to the size of a typical module.  The benefit is that
the code would be simpler because we don't have any special cases and
the structs would automatically get freed with the module struct when
the module gets unloaded.

-- 
Josh

Re: [RFC PATCH 2/5] module: save load_info for livepatch modules

[Josh Poimboeuf]

On Mon, Nov 09, 2015 at 11:45:52PM -0500, Jessica Yu wrote:
> In livepatch modules, preserve section, symbol, string information from
> the load_info struct in the module loader. This information is used to
> patch modules that are not loaded in memory yet; specifically it is used
> to resolve remaining symbols and write relocations when the target
> module loads.
> 
> Signed-off-by: Jessica Yu <jeyu@redhat.com>
> ---
>  include/linux/module.h  | 25 +++++++++++++++++++++++++
>  kernel/livepatch/core.c | 17 +++++++++++++++++
>  kernel/module.c         | 36 ++++++++++++++++++++++--------------
>  3 files changed, 64 insertions(+), 14 deletions(-)
> 
> diff --git a/include/linux/module.h b/include/linux/module.h
> index 3a19c79..c8680b1 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
> @@ -36,6 +36,20 @@ struct modversion_info {
>  	char name[MODULE_NAME_LEN];
>  };
>  
> +struct load_info {
> +	Elf_Ehdr *hdr;
> +	unsigned long len;
> +	Elf_Shdr *sechdrs;
> +	char *secstrings, *strtab;
> +	unsigned long symoffs, stroffs;
> +	struct _ddebug *debug;
> +	unsigned int num_debug;
> +	bool sig_ok;
> +	struct {
> +		unsigned int sym, str, mod, vers, info, pcpu;
> +	} index;
> +};
> +
>  struct module;
>  
>  struct module_kobject {
> @@ -462,6 +476,8 @@ struct module {
>  
>  #ifdef CONFIG_LIVEPATCH
>  	bool klp_alive;
> +	/* save info to patch to-be-loaded modules */
> +	struct load_info *info;
>  #endif
>  
>  #ifdef CONFIG_MODULE_UNLOAD
> @@ -635,6 +651,15 @@ static inline bool module_requested_async_probing(struct module *module)
>  	return module && module->async_probe_requested;
>  }
>  
> +#ifdef CONFIG_LIVEPATCH
> +extern void klp_prepare_patch_module(struct module *mod,
> +				     struct load_info *info);
> +extern int
> +apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
> +		   unsigned int symindex, unsigned int relsec,
> +		   struct module *me);
> +#endif
> +
>  #else /* !CONFIG_MODULES... */
>  
>  /* Given an address, look for it in the exception tables. */
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index 6e53441..087a8c7 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>  	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>  };
>  
> +/*
> + * Save necessary information from info in order to be able to
> + * patch modules that might be loaded later
> + */
> +void klp_prepare_patch_module(struct module *mod, struct load_info *info)
> +{
> +	Elf_Shdr *symsect;
> +
> +	symsect = info->sechdrs + info->index.sym;
> +	/* update sh_addr to point to symtab */
> +	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
> +
> +	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
> +	memcpy(mod->info, info, sizeof(*info));
> +
> +}
> +
>  static int __init klp_init(void)
>  {
>  	int ret;
> diff --git a/kernel/module.c b/kernel/module.c
> index 8f051a1..8ae3ca5 100644
> --- a/kernel/module.c
> +++ b/kernel/module.c
> @@ -318,20 +318,6 @@ int unregister_module_notifier(struct notifier_block *nb)
>  }
>  EXPORT_SYMBOL(unregister_module_notifier);
>  
> -struct load_info {
> -	Elf_Ehdr *hdr;
> -	unsigned long len;
> -	Elf_Shdr *sechdrs;
> -	char *secstrings, *strtab;
> -	unsigned long symoffs, stroffs;
> -	struct _ddebug *debug;
> -	unsigned int num_debug;
> -	bool sig_ok;
> -	struct {
> -		unsigned int sym, str, mod, vers, info, pcpu;
> -	} index;
> -};
> -
>  /* We require a truly strong try_module_get(): 0 means failure due to
>     ongoing or failed initialization etc. */
>  static inline int strong_try_module_get(struct module *mod)
> @@ -2137,6 +2123,11 @@ static int simplify_symbols(struct module *mod, const struct load_info *info)
>  			       (long)sym[i].st_value);
>  			break;
>  
> +#ifdef CONFIG_LIVEPATCH
> +		case SHN_LIVEPATCH:
> +			break;
> +#endif
> +

The kernel coding style generally frowns on ifdefs inside functions.
IMO it's fine to just remove the ifdef completely here.

>  		case SHN_UNDEF:
>  			ksym = resolve_symbol_wait(mod, info, name);
>  			/* Ok if resolved.  */
> @@ -2185,6 +2176,11 @@ static int apply_relocations(struct module *mod, const struct load_info *info)
>  		if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC))
>  			continue;
>  
> +#ifdef CONFIG_LIVEPATCH
> +		if (info->sechdrs[i].sh_flags & SHF_RELA_LIVEPATCH)
> +			continue;
> +#endif
> +

Same here, I think the ifdef can be removed.

>  		if (info->sechdrs[i].sh_type == SHT_REL)
>  			err = apply_relocate(info->sechdrs, info->strtab,
>  					     info->index.sym, i, mod);
> @@ -3530,8 +3526,20 @@ static int load_module(struct load_info *info, const char __user *uargs,
>  	if (err < 0)
>  		goto bug_cleanup;
>  
> +#ifdef CONFIG_LIVEPATCH
> +	/*
> +	 * Save sechdrs, indices, and other data from info
> +	 * in order to patch to-be-loaded modules.
> +	 * Do not call free_copy() for livepatch modules.
> +	 */
> +	if (get_modinfo((struct load_info *)info, "livepatch"))
> +		klp_prepare_patch_module(mod, info);
> +	else
> +		free_copy(info);
> +#else
>  	/* Get rid of temporary copy. */
>  	free_copy(info);
> +#endif

Same here, IMO.  (And as Miroslav suggested, klp_prepare_patch_module()
can be renamed to something else (save_info()?) which exists even in the
!CONFIG_LIVEPATCH case.

-- 
Josh

Re: [RFC PATCH 2/5] module: save load_info for livepatch modules

[Josh Poimboeuf]

On Mon, Nov 09, 2015 at 11:45:52PM -0500, Jessica Yu wrote:
> In livepatch modules, preserve section, symbol, string information from
> the load_info struct in the module loader. This information is used to
> patch modules that are not loaded in memory yet; specifically it is used
> to resolve remaining symbols and write relocations when the target
> module loads.
> 
> Signed-off-by: Jessica Yu <jeyu-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> ---
>  include/linux/module.h  | 25 +++++++++++++++++++++++++
>  kernel/livepatch/core.c | 17 +++++++++++++++++
>  kernel/module.c         | 36 ++++++++++++++++++++++--------------
>  3 files changed, 64 insertions(+), 14 deletions(-)
> 
> diff --git a/include/linux/module.h b/include/linux/module.h
> index 3a19c79..c8680b1 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
> @@ -36,6 +36,20 @@ struct modversion_info {
>  	char name[MODULE_NAME_LEN];
>  };
>  
> +struct load_info {
> +	Elf_Ehdr *hdr;
> +	unsigned long len;
> +	Elf_Shdr *sechdrs;
> +	char *secstrings, *strtab;
> +	unsigned long symoffs, stroffs;
> +	struct _ddebug *debug;
> +	unsigned int num_debug;
> +	bool sig_ok;
> +	struct {
> +		unsigned int sym, str, mod, vers, info, pcpu;
> +	} index;
> +};
> +
>  struct module;
>  
>  struct module_kobject {
> @@ -462,6 +476,8 @@ struct module {
>  
>  #ifdef CONFIG_LIVEPATCH
>  	bool klp_alive;
> +	/* save info to patch to-be-loaded modules */
> +	struct load_info *info;
>  #endif
>  
>  #ifdef CONFIG_MODULE_UNLOAD
> @@ -635,6 +651,15 @@ static inline bool module_requested_async_probing(struct module *module)
>  	return module && module->async_probe_requested;
>  }
>  
> +#ifdef CONFIG_LIVEPATCH
> +extern void klp_prepare_patch_module(struct module *mod,
> +				     struct load_info *info);
> +extern int
> +apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
> +		   unsigned int symindex, unsigned int relsec,
> +		   struct module *me);
> +#endif
> +
>  #else /* !CONFIG_MODULES... */
>  
>  /* Given an address, look for it in the exception tables. */
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index 6e53441..087a8c7 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>  	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>  };
>  
> +/*
> + * Save necessary information from info in order to be able to
> + * patch modules that might be loaded later
> + */
> +void klp_prepare_patch_module(struct module *mod, struct load_info *info)
> +{
> +	Elf_Shdr *symsect;
> +
> +	symsect = info->sechdrs + info->index.sym;
> +	/* update sh_addr to point to symtab */
> +	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
> +
> +	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
> +	memcpy(mod->info, info, sizeof(*info));
> +
> +}
> +
>  static int __init klp_init(void)
>  {
>  	int ret;
> diff --git a/kernel/module.c b/kernel/module.c
> index 8f051a1..8ae3ca5 100644
> --- a/kernel/module.c
> +++ b/kernel/module.c
> @@ -318,20 +318,6 @@ int unregister_module_notifier(struct notifier_block *nb)
>  }
>  EXPORT_SYMBOL(unregister_module_notifier);
>  
> -struct load_info {
> -	Elf_Ehdr *hdr;
> -	unsigned long len;
> -	Elf_Shdr *sechdrs;
> -	char *secstrings, *strtab;
> -	unsigned long symoffs, stroffs;
> -	struct _ddebug *debug;
> -	unsigned int num_debug;
> -	bool sig_ok;
> -	struct {
> -		unsigned int sym, str, mod, vers, info, pcpu;
> -	} index;
> -};
> -
>  /* We require a truly strong try_module_get(): 0 means failure due to
>     ongoing or failed initialization etc. */
>  static inline int strong_try_module_get(struct module *mod)
> @@ -2137,6 +2123,11 @@ static int simplify_symbols(struct module *mod, const struct load_info *info)
>  			       (long)sym[i].st_value);
>  			break;
>  
> +#ifdef CONFIG_LIVEPATCH
> +		case SHN_LIVEPATCH:
> +			break;
> +#endif
> +

The kernel coding style generally frowns on ifdefs inside functions.
IMO it's fine to just remove the ifdef completely here.

>  		case SHN_UNDEF:
>  			ksym = resolve_symbol_wait(mod, info, name);
>  			/* Ok if resolved.  */
> @@ -2185,6 +2176,11 @@ static int apply_relocations(struct module *mod, const struct load_info *info)
>  		if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC))
>  			continue;
>  
> +#ifdef CONFIG_LIVEPATCH
> +		if (info->sechdrs[i].sh_flags & SHF_RELA_LIVEPATCH)
> +			continue;
> +#endif
> +

Same here, I think the ifdef can be removed.

>  		if (info->sechdrs[i].sh_type == SHT_REL)
>  			err = apply_relocate(info->sechdrs, info->strtab,
>  					     info->index.sym, i, mod);
> @@ -3530,8 +3526,20 @@ static int load_module(struct load_info *info, const char __user *uargs,
>  	if (err < 0)
>  		goto bug_cleanup;
>  
> +#ifdef CONFIG_LIVEPATCH
> +	/*
> +	 * Save sechdrs, indices, and other data from info
> +	 * in order to patch to-be-loaded modules.
> +	 * Do not call free_copy() for livepatch modules.
> +	 */
> +	if (get_modinfo((struct load_info *)info, "livepatch"))
> +		klp_prepare_patch_module(mod, info);
> +	else
> +		free_copy(info);
> +#else
>  	/* Get rid of temporary copy. */
>  	free_copy(info);
> +#endif

Same here, IMO.  (And as Miroslav suggested, klp_prepare_patch_module()
can be renamed to something else (save_info()?) which exists even in the
!CONFIG_LIVEPATCH case.

-- 
Josh

Re: [RFC PATCH 2/5] module: save load_info for livepatch modules

[Josh Poimboeuf]

On Mon, Nov 09, 2015 at 11:45:52PM -0500, Jessica Yu wrote:
> In livepatch modules, preserve section, symbol, string information from
> the load_info struct in the module loader. This information is used to
> patch modules that are not loaded in memory yet; specifically it is used
> to resolve remaining symbols and write relocations when the target
> module loads.
> 
> Signed-off-by: Jessica Yu <jeyu@redhat.com>
> ---
>  include/linux/module.h  | 25 +++++++++++++++++++++++++
>  kernel/livepatch/core.c | 17 +++++++++++++++++
>  kernel/module.c         | 36 ++++++++++++++++++++++--------------
>  3 files changed, 64 insertions(+), 14 deletions(-)
> 
> diff --git a/include/linux/module.h b/include/linux/module.h
> index 3a19c79..c8680b1 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
> @@ -36,6 +36,20 @@ struct modversion_info {
>  	char name[MODULE_NAME_LEN];
>  };
>  
> +struct load_info {
> +	Elf_Ehdr *hdr;
> +	unsigned long len;
> +	Elf_Shdr *sechdrs;
> +	char *secstrings, *strtab;
> +	unsigned long symoffs, stroffs;
> +	struct _ddebug *debug;
> +	unsigned int num_debug;
> +	bool sig_ok;
> +	struct {
> +		unsigned int sym, str, mod, vers, info, pcpu;
> +	} index;
> +};
> +
>  struct module;
>  
>  struct module_kobject {
> @@ -462,6 +476,8 @@ struct module {
>  
>  #ifdef CONFIG_LIVEPATCH
>  	bool klp_alive;
> +	/* save info to patch to-be-loaded modules */
> +	struct load_info *info;
>  #endif
>  
>  #ifdef CONFIG_MODULE_UNLOAD
> @@ -635,6 +651,15 @@ static inline bool module_requested_async_probing(struct module *module)
>  	return module && module->async_probe_requested;
>  }
>  
> +#ifdef CONFIG_LIVEPATCH
> +extern void klp_prepare_patch_module(struct module *mod,
> +				     struct load_info *info);
> +extern int
> +apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
> +		   unsigned int symindex, unsigned int relsec,
> +		   struct module *me);
> +#endif
> +
>  #else /* !CONFIG_MODULES... */
>  
>  /* Given an address, look for it in the exception tables. */
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index 6e53441..087a8c7 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>  	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>  };
>  
> +/*
> + * Save necessary information from info in order to be able to
> + * patch modules that might be loaded later
> + */
> +void klp_prepare_patch_module(struct module *mod, struct load_info *info)
> +{
> +	Elf_Shdr *symsect;
> +
> +	symsect = info->sechdrs + info->index.sym;
> +	/* update sh_addr to point to symtab */
> +	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
> +
> +	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
> +	memcpy(mod->info, info, sizeof(*info));
> +
> +}
> +
>  static int __init klp_init(void)
>  {
>  	int ret;
> diff --git a/kernel/module.c b/kernel/module.c
> index 8f051a1..8ae3ca5 100644
> --- a/kernel/module.c
> +++ b/kernel/module.c
> @@ -318,20 +318,6 @@ int unregister_module_notifier(struct notifier_block *nb)
>  }
>  EXPORT_SYMBOL(unregister_module_notifier);
>  
> -struct load_info {
> -	Elf_Ehdr *hdr;
> -	unsigned long len;
> -	Elf_Shdr *sechdrs;
> -	char *secstrings, *strtab;
> -	unsigned long symoffs, stroffs;
> -	struct _ddebug *debug;
> -	unsigned int num_debug;
> -	bool sig_ok;
> -	struct {
> -		unsigned int sym, str, mod, vers, info, pcpu;
> -	} index;
> -};
> -
>  /* We require a truly strong try_module_get(): 0 means failure due to
>     ongoing or failed initialization etc. */
>  static inline int strong_try_module_get(struct module *mod)
> @@ -2137,6 +2123,11 @@ static int simplify_symbols(struct module *mod, const struct load_info *info)
>  			       (long)sym[i].st_value);
>  			break;
>  
> +#ifdef CONFIG_LIVEPATCH
> +		case SHN_LIVEPATCH:
> +			break;
> +#endif
> +

It would be good to add a short comment here stating that livepatch
symbols are resolved later by the klp code.

>  		case SHN_UNDEF:
>  			ksym = resolve_symbol_wait(mod, info, name);
>  			/* Ok if resolved.  */
> @@ -2185,6 +2176,11 @@ static int apply_relocations(struct module *mod, const struct load_info *info)
>  		if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC))
>  			continue;
>  
> +#ifdef CONFIG_LIVEPATCH
> +		if (info->sechdrs[i].sh_flags & SHF_RELA_LIVEPATCH)
> +			continue;
> +#endif
> +

Same here, a short comment stating that livepatch relocations are
applied later by klp code.

>  		if (info->sechdrs[i].sh_type == SHT_REL)
>  			err = apply_relocate(info->sechdrs, info->strtab,
>  					     info->index.sym, i, mod);
> @@ -3530,8 +3526,20 @@ static int load_module(struct load_info *info, const char __user *uargs,
>  	if (err < 0)
>  		goto bug_cleanup;
>  
> +#ifdef CONFIG_LIVEPATCH
> +	/*
> +	 * Save sechdrs, indices, and other data from info
> +	 * in order to patch to-be-loaded modules.
> +	 * Do not call free_copy() for livepatch modules.
> +	 */
> +	if (get_modinfo((struct load_info *)info, "livepatch"))
> +		klp_prepare_patch_module(mod, info);
> +	else
> +		free_copy(info);
> +#else
>  	/* Get rid of temporary copy. */
>  	free_copy(info);
> +#endif
>  
>  	/* Done! */
>  	trace_module_load(mod);
> -- 
> 2.4.3
> 

-- 
Josh

Re: [RFC PATCH 2/5] module: save load_info for livepatch modules

[Josh Poimboeuf]

On Mon, Nov 09, 2015 at 11:45:52PM -0500, Jessica Yu wrote:
> In livepatch modules, preserve section, symbol, string information from
> the load_info struct in the module loader. This information is used to
> patch modules that are not loaded in memory yet; specifically it is used
> to resolve remaining symbols and write relocations when the target
> module loads.
> 
> Signed-off-by: Jessica Yu <jeyu-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> ---
>  include/linux/module.h  | 25 +++++++++++++++++++++++++
>  kernel/livepatch/core.c | 17 +++++++++++++++++
>  kernel/module.c         | 36 ++++++++++++++++++++++--------------
>  3 files changed, 64 insertions(+), 14 deletions(-)
> 
> diff --git a/include/linux/module.h b/include/linux/module.h
> index 3a19c79..c8680b1 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
> @@ -36,6 +36,20 @@ struct modversion_info {
>  	char name[MODULE_NAME_LEN];
>  };
>  
> +struct load_info {
> +	Elf_Ehdr *hdr;
> +	unsigned long len;
> +	Elf_Shdr *sechdrs;
> +	char *secstrings, *strtab;
> +	unsigned long symoffs, stroffs;
> +	struct _ddebug *debug;
> +	unsigned int num_debug;
> +	bool sig_ok;
> +	struct {
> +		unsigned int sym, str, mod, vers, info, pcpu;
> +	} index;
> +};
> +
>  struct module;
>  
>  struct module_kobject {
> @@ -462,6 +476,8 @@ struct module {
>  
>  #ifdef CONFIG_LIVEPATCH
>  	bool klp_alive;
> +	/* save info to patch to-be-loaded modules */
> +	struct load_info *info;
>  #endif
>  
>  #ifdef CONFIG_MODULE_UNLOAD
> @@ -635,6 +651,15 @@ static inline bool module_requested_async_probing(struct module *module)
>  	return module && module->async_probe_requested;
>  }
>  
> +#ifdef CONFIG_LIVEPATCH
> +extern void klp_prepare_patch_module(struct module *mod,
> +				     struct load_info *info);
> +extern int
> +apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
> +		   unsigned int symindex, unsigned int relsec,
> +		   struct module *me);
> +#endif
> +
>  #else /* !CONFIG_MODULES... */
>  
>  /* Given an address, look for it in the exception tables. */
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index 6e53441..087a8c7 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>  	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>  };
>  
> +/*
> + * Save necessary information from info in order to be able to
> + * patch modules that might be loaded later
> + */
> +void klp_prepare_patch_module(struct module *mod, struct load_info *info)
> +{
> +	Elf_Shdr *symsect;
> +
> +	symsect = info->sechdrs + info->index.sym;
> +	/* update sh_addr to point to symtab */
> +	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
> +
> +	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
> +	memcpy(mod->info, info, sizeof(*info));
> +
> +}
> +
>  static int __init klp_init(void)
>  {
>  	int ret;
> diff --git a/kernel/module.c b/kernel/module.c
> index 8f051a1..8ae3ca5 100644
> --- a/kernel/module.c
> +++ b/kernel/module.c
> @@ -318,20 +318,6 @@ int unregister_module_notifier(struct notifier_block *nb)
>  }
>  EXPORT_SYMBOL(unregister_module_notifier);
>  
> -struct load_info {
> -	Elf_Ehdr *hdr;
> -	unsigned long len;
> -	Elf_Shdr *sechdrs;
> -	char *secstrings, *strtab;
> -	unsigned long symoffs, stroffs;
> -	struct _ddebug *debug;
> -	unsigned int num_debug;
> -	bool sig_ok;
> -	struct {
> -		unsigned int sym, str, mod, vers, info, pcpu;
> -	} index;
> -};
> -
>  /* We require a truly strong try_module_get(): 0 means failure due to
>     ongoing or failed initialization etc. */
>  static inline int strong_try_module_get(struct module *mod)
> @@ -2137,6 +2123,11 @@ static int simplify_symbols(struct module *mod, const struct load_info *info)
>  			       (long)sym[i].st_value);
>  			break;
>  
> +#ifdef CONFIG_LIVEPATCH
> +		case SHN_LIVEPATCH:
> +			break;
> +#endif
> +

It would be good to add a short comment here stating that livepatch
symbols are resolved later by the klp code.

>  		case SHN_UNDEF:
>  			ksym = resolve_symbol_wait(mod, info, name);
>  			/* Ok if resolved.  */
> @@ -2185,6 +2176,11 @@ static int apply_relocations(struct module *mod, const struct load_info *info)
>  		if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC))
>  			continue;
>  
> +#ifdef CONFIG_LIVEPATCH
> +		if (info->sechdrs[i].sh_flags & SHF_RELA_LIVEPATCH)
> +			continue;
> +#endif
> +

Same here, a short comment stating that livepatch relocations are
applied later by klp code.

>  		if (info->sechdrs[i].sh_type == SHT_REL)
>  			err = apply_relocate(info->sechdrs, info->strtab,
>  					     info->index.sym, i, mod);
> @@ -3530,8 +3526,20 @@ static int load_module(struct load_info *info, const char __user *uargs,
>  	if (err < 0)
>  		goto bug_cleanup;
>  
> +#ifdef CONFIG_LIVEPATCH
> +	/*
> +	 * Save sechdrs, indices, and other data from info
> +	 * in order to patch to-be-loaded modules.
> +	 * Do not call free_copy() for livepatch modules.
> +	 */
> +	if (get_modinfo((struct load_info *)info, "livepatch"))
> +		klp_prepare_patch_module(mod, info);
> +	else
> +		free_copy(info);
> +#else
>  	/* Get rid of temporary copy. */
>  	free_copy(info);
> +#endif
>  
>  	/* Done! */
>  	trace_module_load(mod);
> -- 
> 2.4.3
> 

-- 
Josh

Re: livepatch: reuse module loader code to write relocations

[Josh Poimboeuf]

On Thu, Nov 12, 2015 at 04:27:01PM +0100, Miroslav Benes wrote:
> On Wed, 11 Nov 2015, Jessica Yu wrote:
> 
> > +++ Miroslav Benes [11/11/15 15:30 +0100]:
> > > On Mon, 9 Nov 2015, Jessica Yu wrote:
> > >
> > > So I guess we don't need klp_reloc anymore.
> > 
> > Yes, that's correct. I am noticing just now that I forgot to remove
> > the klp_reloc struct definition from livepatch.h. That change will be
> > reflected in v2...
> > 
> > > If true, we should really
> > > start thinking about proper documentation because there are going to be
> > > plenty of assumptions about a patch module and we need to have it written
> > > somewhere. Especially how the relocation sections look like.
> > 
> > Agreed. As a first step the patch module format can perhaps be
> > documented somewhere. Perhaps it's time we create
> > Documentation/livepatch/? :-)
> 
> Yes, I think so.
> 
> > > > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > > > index 087a8c7..26c419f 100644
> > > > --- a/kernel/livepatch/core.c
> > > > +++ b/kernel/livepatch/core.c
> > > > @@ -28,6 +28,8 @@
> > > >  #include <linux/list.h>
> > > >  #include <linux/kallsyms.h>
> > > >  #include <linux/livepatch.h>
> > > > +#include <linux/elf.h>
> > > > +#include <asm/cacheflush.h>
> > > > 
> > > >  /**
> > > >   * struct klp_ops - structure for tracking registered ftrace ops structs
> > > > @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module
> > > > *pmod, const char *name,
> > > >  }
> > > > 
> > > >  static int klp_write_object_relocations(struct module *pmod,
> > > > -					struct klp_object *obj)
> > > > +					struct klp_object *obj,
> > > > +					struct klp_patch *patch)
> > > >  {
> > > > -	int ret;
> > > > -	struct klp_reloc *reloc;
> > > > +	int relindex, num_relas;
> > > > +	int i, ret = 0;
> > > > +	unsigned long addr;
> > > > +	unsigned int bind;
> > > > +	char *symname;
> > > > +	struct klp_reloc_sec *reloc_sec;
> > > > +	struct load_info *info;
> > > > +	Elf_Rela *rela;
> > > > +	Elf_Sym *sym, *symtab;
> > > > +	Elf_Shdr *symsect;
> > > > 
> > > >  	if (WARN_ON(!klp_is_object_loaded(obj)))
> > > >  		return -EINVAL;
> > > > 
> > > > -	if (WARN_ON(!obj->relocs))
> > > > -		return -EINVAL;
> > > > -
> > > > -	for (reloc = obj->relocs; reloc->name; reloc++) {
> > > > -		if (!klp_is_module(obj)) {
> > > > -			ret = klp_verify_vmlinux_symbol(reloc->name,
> > > > -							reloc->val);
> > > > -			if (ret)
> > > > -				return ret;
> > > > -		} else {
> > > > -			/* module, reloc->val needs to be discovered */
> > > > -			if (reloc->external)
> > > > -				ret = klp_find_external_symbol(pmod,
> > > > -							       reloc->name,
> > > > -							       &reloc->val);
> > > > -			else
> > > > -				ret = klp_find_object_symbol(obj->mod->name,
> > > > -							     reloc->name,
> > > > -							     &reloc->val);
> > > > -			if (ret)
> > > > -				return ret;
> > > > -		}
> > > > -		ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc,
> > > > -					     reloc->val + reloc->addend);
> > > > -		if (ret) {
> > > > -			pr_err("relocation failed for symbol '%s' at 0x%016lx
> > > > (%d)\n",
> > > > -			       reloc->name, reloc->val, ret);
> > > > -			return ret;
> > > > +	info = pmod->info;
> > > > +	symsect = info->sechdrs + info->index.sym;
> > > > +	symtab = (void *)info->hdr + symsect->sh_offset;
> > > > +
> > > > +	/* For each __klp_rela section for this object */
> > > > +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
> > > > +		relindex = reloc_sec->index;
> > > > +		num_relas = info->sechdrs[relindex].sh_size /
> > > > sizeof(Elf_Rela);
> > > > +		rela = (Elf_Rela *) info->sechdrs[relindex].sh_addr;
> > > > +
> > > > +		/* For each rela in this __klp_rela section */
> > > > +		for (i = 0; i < num_relas; i++, rela++) {
> > > > +			sym = symtab + ELF_R_SYM(rela->r_info);
> > > > +			symname = info->strtab + sym->st_name;
> > > > +			bind = ELF_ST_BIND(sym->st_info);
> > > > +
> > > > +			if (sym->st_shndx == SHN_LIVEPATCH) {
> > > > +				if (bind == STB_LIVEPATCH_EXT)
> > > > +					ret = klp_find_external_symbol(pmod,
> > > > symname, &addr);
> > > > +				else
> > > > +					ret =
> > > > klp_find_object_symbol(obj->name, symname, &addr);
> > > > +				if (ret)
> > > > +					return ret;
> > > > +				sym->st_value = addr;
> > > > +			}
> > > >  		}
> > > > +		ret = apply_relocate_add(info->sechdrs, info->strtab,
> > > > +					 info->index.sym, relindex, pmod);
> > > >  	}
> > > > 
> > > > -	return 0;
> > > > +	return ret;
> > > >  }
> > > 
> > > Looking at this... do we even need reloc_secs in klp_object? Question is
> > > whether we need more than one dynrela section for an object. If not then
> > > the binding between klp_reloc_sec and an object is the only relevant thing
> > > in the structure, be it index or objname. So we can replace the
> > > list of structures with just the index in klp_object, or get rid of it
> > > completely and rely on the name of dynrela section be something like
> > > __klp_rela_{objname}.
> > 
> > Hm, you bring up a good point. I think theoretically yes, it is
> > possible to just have one klp_reloc_sec for each object and therefore
> > a list is not required (I have not checked yet how difficult it would
> > be to implement this on the kpatch-build side of things).  However,
> > considering the final format of the patch module, I think it is
> > semantically clearer to leave it as a list, and for each object to
> > possibly have more than one __klp_rela section.
> > 
> > For example, say we are patching two functions in ext4. In my
> > resulting kpatch module I will have two __klp_rela_ext4 sections, and
> > they might look like this when we run readelf --sections:
> > 
> > [34] __klp_rela_ext4.text.ext4_attr_store RELA ...
> > [35] __klp_rela_ext4.text.ext4_attr_show RELA ...
> > 
> > Then these two klp rela sections end up as two elements in the
> > reloc_secs list for the ext4 patch object. I think this way, we can
> > better tell which rela is being applied to what function. Might be
> > easier to understand what's happening from the developer's point of
> > view.
> > 
> > > You see, we go through elf sections here which were preserved by module
> > > loader. We even have relevant sections marked with SHF_RELA_LIVEPATCH. So
> > > maybe all the stuff around klp_reloc_sec is not necessary.
> > > 
> > > Thoughts?
> > 
> > Ah, so this is where descriptive comments and documentation might have
> > been useful :-) So I think we will still need to keep the
> > klp_reloc_sec struct to help the patch module initialize. Though the
> > name and objname fields aren't used in this patchset, they are used in
> > the kpatch patch module code [1], where we iterate through each elf
> > section, find the ones marked with SHF_RELA_LIVEPATCH, set the
> > klp_reloc_sec's objname (which we find from the "name" field,
> > formatted as __klp_rela_{objname}.text..). Once we have the objname
> > set, we can then find the object to attach the reloc_sec to (i.e. add
> > it to its list of reloc_secs).
> > 
> > Hope that clears some things up.
> 
> Ok, I'll try to explain myself and it is gonna be long. I'll try to 
> describe how we deal with dynrelas in klp today, how you use it in kpatch 
> (and this is the place where my knowledge can be wrong or obsolete), what 
> you propose and what I'd like to propose.
> 
> 1. First let's look on what we have now.
> 
> We have a patch module in which there is a section with all dynrelas 
> needed to be resolved (it was like this in kpatch some time ago and maybe 
> it is different now so just have a patience, I'll get to it). In the init 
> function of the module kpatch builds all the relevant info from dynrela 
> section. It goes through it, creates an array of klp_reloc for each object 
> and includes each dynrela record with an objname to the array of 
> klp_object with that objname. Later when we need to apply relocations for 
> patched object we just go through the list (array) of its dynrelas in 
> klp_object and call our arch-specific klp_write_module_reloc().
> 
> Now this was probably changed in kpatch and you do not have one dynrela 
> section but one dynrela section for each patched function. Is that 
> correct? (and can you tell us what the reason for the change was? It might 
> be crucial because I might be missing something.). Which leads us to your 
> proposal...
> 
> 2. So we have several dynrela section for one patched object. During init 
> function in a patch module kpatch once again builds needed structures from 
> these sections. Now they are called klp_reloc_sec and contain different 
> kind of info. There is no val, loc and such, only name of the symbol, 
> objname and index to dynrela section in ELF. So when you need to write 
> relocations for the patched object you go through all relevant dynrela 
> sections (because they are stored in the klp_object), all dynrela records 
> in each section and you resolve the undefined symbols. All needed info is 
> stored in ELF. Then you just call apply_relocate_add().
> 
> 3. I propose to go one step further. I think we don't need klp_reloc_sec 
> if there is only one dynrela section for patched object (and I currently 
> cannot see why this is not possible. It is possible even with one dynrela 
> section for whole patch module, that is for all patched objects.).

I think I agree that we don't need klp_reloc_sec, and that klp rela
sections can presumably be discovered by iterating over the sections.

But I don't think it matters much whether we have one klp rela section
per object, or multiple rela sections per object.  Either way, can't we
find them when we iterate over the sections?

For example, for one rela section per object, it could be named:

__klp_rela.objname

Or for multiple rela sections per object, they could be named:

__klp_rela.objname.func1
__klp_rela.objname.func2

Either way, when iterating over the sections, we could just look for
"__klp_rela.objname".

All that said, I think I would vote for one rela section per object,
just because it seems simpler.

> 
> In my proposal there would be nothing done in init function of the patched 
> module (albeit some optimization mentioned later). When you call 
> klp_write_object_relocations you would go through all ELF section and find 
> the relevant one for the object (it is marked with SHF_RELA_LIVEPATCH and 
> objname is in the name of the section. It is the same thing you do in 2. 
> in the init function.). Then you go through all dynrela records in the 
> section, you do the same things which you do in the proposed patch above, 
> and call apply_relocate_add.
> 
> Now it would be crazy to go through all sections each time 
> klp_write_object_relocations is called (maybe it does not even matter but 

Why would that be crazy?  To me it seems perfectly logical :-)  It
doesn't seem like a very expensive operation to me.

> nevertheless). So klp_object could have an index to its ELF dynrela 
> section. This index can be retrieved in the init function the same way you 
> do all the stuff with klp_reloc_sec.
> 
> If you insisted on more than one dynrela section for a patched object we 
> could have an array of indices there. Or whatever.
> 
> It is almost the same as your proposal but in my opinion somewhat nicer. 
> We just use the info stored in ELF directly without unnecessary middle 
> layer (== klp_reloc_sec).
> 
> Does it make sense? I hope it does. Would it work?

-- 
Josh

Re: [RFC PATCH 3/5] livepatch: reuse module loader code to write relocations

[Josh Poimboeuf]

On Mon, Nov 09, 2015 at 11:45:53PM -0500, Jessica Yu wrote:
> Reuse module loader code to write relocations, thereby eliminating the
> need for architecture specific code in livepatch. Namely, we reuse
> apply_relocate_add() in the module loader to write relocs instead of
> duplicating functionality in livepatch's klp_write_module_reloc(). To
> apply relocation sections, remaining SHN_LIVEPATCH symbols referenced by
> relocs are resolved and then apply_relocate_add() is called to apply
> those relocations.
> 
> Signed-off-by: Jessica Yu <jeyu@redhat.com>
> ---
>  include/linux/livepatch.h | 11 ++++--
>  include/linux/module.h    |  6 ++++
>  kernel/livepatch/core.c   | 89 +++++++++++++++++++++++++++++------------------
>  3 files changed, 70 insertions(+), 36 deletions(-)
> 
> diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
> index 31db7a0..601e892 100644
> --- a/include/linux/livepatch.h
> +++ b/include/linux/livepatch.h
> @@ -85,7 +85,7 @@ struct klp_reloc {
>  /**
>   * struct klp_object - kernel object structure for live patching
>   * @name:	module name (or NULL for vmlinux)
> - * @relocs:	relocation entries to be applied at load time
> + * @reloc_secs:	relocation sections to be applied at load time
>   * @funcs:	function entries for functions to be patched in the object
>   * @kobj:	kobject for sysfs resources
>   * @mod:	kernel module associated with the patched object
> @@ -95,7 +95,7 @@ struct klp_reloc {
>  struct klp_object {
>  	/* external */
>  	const char *name;
> -	struct klp_reloc *relocs;
> +	struct list_head reloc_secs;
>  	struct klp_func *funcs;
>  
>  	/* internal */
> @@ -129,6 +129,13 @@ struct klp_patch {
>  #define klp_for_each_func(obj, func) \
>  	for (func = obj->funcs; func->old_name; func++)
>  
> +struct klp_reloc_sec {
> +	unsigned int index;
> +	char *name;
> +	char *objname;
> +	struct list_head list;
> +};
> +
>  int klp_register_patch(struct klp_patch *);
>  int klp_unregister_patch(struct klp_patch *);
>  int klp_enable_patch(struct klp_patch *);
> diff --git a/include/linux/module.h b/include/linux/module.h
> index c8680b1..3c34eb8 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
> @@ -793,9 +793,15 @@ extern int module_sysfs_initialized;
>  #ifdef CONFIG_DEBUG_SET_MODULE_RONX
>  extern void set_all_modules_text_rw(void);
>  extern void set_all_modules_text_ro(void);
> +extern void
> +set_page_attributes(void *start, void *end,
> +		    int (*set)(unsigned long start, int num_pages));
>  #else
>  static inline void set_all_modules_text_rw(void) { }
>  static inline void set_all_modules_text_ro(void) { }
> +static inline void
> +set_page_attributes(void *start, void *end,
> +		    int (*set)(unsigned long start, int num_pages)) { }
>  #endif
>  
>  #ifdef CONFIG_GENERIC_BUG
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index 087a8c7..26c419f 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -28,6 +28,8 @@
>  #include <linux/list.h>
>  #include <linux/kallsyms.h>
>  #include <linux/livepatch.h>
> +#include <linux/elf.h>
> +#include <asm/cacheflush.h>
>  
>  /**
>   * struct klp_ops - structure for tracking registered ftrace ops structs
> @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module *pmod, const char *name,
>  }
>  
>  static int klp_write_object_relocations(struct module *pmod,
> -					struct klp_object *obj)
> +					struct klp_object *obj,
> +					struct klp_patch *patch)
>  {
> -	int ret;
> -	struct klp_reloc *reloc;
> +	int relindex, num_relas;
> +	int i, ret = 0;
> +	unsigned long addr;
> +	unsigned int bind;
> +	char *symname;
> +	struct klp_reloc_sec *reloc_sec;
> +	struct load_info *info;
> +	Elf_Rela *rela;
> +	Elf_Sym *sym, *symtab;
> +	Elf_Shdr *symsect;
>  
>  	if (WARN_ON(!klp_is_object_loaded(obj)))
>  		return -EINVAL;
>  
> -	if (WARN_ON(!obj->relocs))
> -		return -EINVAL;
> -
> -	for (reloc = obj->relocs; reloc->name; reloc++) {
> -		if (!klp_is_module(obj)) {
> -			ret = klp_verify_vmlinux_symbol(reloc->name,
> -							reloc->val);
> -			if (ret)
> -				return ret;
> -		} else {
> -			/* module, reloc->val needs to be discovered */
> -			if (reloc->external)
> -				ret = klp_find_external_symbol(pmod,
> -							       reloc->name,
> -							       &reloc->val);
> -			else
> -				ret = klp_find_object_symbol(obj->mod->name,
> -							     reloc->name,
> -							     &reloc->val);
> -			if (ret)
> -				return ret;
> -		}
> -		ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc,
> -					     reloc->val + reloc->addend);
> -		if (ret) {
> -			pr_err("relocation failed for symbol '%s' at 0x%016lx (%d)\n",
> -			       reloc->name, reloc->val, ret);
> -			return ret;
> +	info = pmod->info;
> +	symsect = info->sechdrs + info->index.sym;
> +	symtab = (void *)info->hdr + symsect->sh_offset;
> +
> +	/* For each __klp_rela section for this object */
> +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
> +		relindex = reloc_sec->index;
> +		num_relas = info->sechdrs[relindex].sh_size / sizeof(Elf_Rela);
> +		rela = (Elf_Rela *) info->sechdrs[relindex].sh_addr;
> +
> +		/* For each rela in this __klp_rela section */
> +		for (i = 0; i < num_relas; i++, rela++) {
> +			sym = symtab + ELF_R_SYM(rela->r_info);
> +			symname = info->strtab + sym->st_name;
> +			bind = ELF_ST_BIND(sym->st_info);
> +
> +			if (sym->st_shndx == SHN_LIVEPATCH) {
> +				if (bind == STB_LIVEPATCH_EXT)
> +					ret = klp_find_external_symbol(pmod, symname, &addr);
> +				else
> +					ret = klp_find_object_symbol(obj->name, symname, &addr);
> +				if (ret)
> +					return ret;
> +				sym->st_value = addr;
> +			}

This is missing an important piece, I think.  There's no way to
disambiguate duplicate symbols.  Before, there was reloc->val, which
contained the symbol's address.  There's nothing like that here.

I think we could use sym->st_value for that purpose.  After Chris's
patch set gets merged, it could contain the sympos.

>  		}
> +		ret = apply_relocate_add(info->sechdrs, info->strtab,
> +					 info->index.sym, relindex, pmod);
>  	}
>  
> -	return 0;
> +	return ret;
>  }
>  
>  static void notrace klp_ftrace_handler(unsigned long ip,
> @@ -741,12 +751,23 @@ static int klp_init_object_loaded(struct klp_patch *patch,
>  				  struct klp_object *obj)
>  {
>  	struct klp_func *func;
> +	struct module *pmod;
>  	int ret;
>  
> -	if (obj->relocs) {
> -		ret = klp_write_object_relocations(patch->mod, obj);
> +	pmod = patch->mod;
> +
> +	if (!list_empty(&obj->reloc_secs)) {
> +		set_page_attributes(pmod->module_core,
> +				    pmod->module_core + pmod->core_text_size,
> +				    set_memory_rw);
> +
> +		ret = klp_write_object_relocations(pmod, obj, patch);
>  		if (ret)
>  			return ret;
> +
> +		set_page_attributes(pmod->module_core,
> +				    pmod->module_core + pmod->core_text_size,
> +				    set_memory_ro);
>  	}
>  
>  	klp_for_each_func(obj, func) {
> -- 
> 2.4.3
> 

-- 
Josh

Re: [RFC PATCH 3/5] livepatch: reuse module loader code to write relocations

[Josh Poimboeuf]

On Mon, Nov 09, 2015 at 11:45:53PM -0500, Jessica Yu wrote:
> Reuse module loader code to write relocations, thereby eliminating the
> need for architecture specific code in livepatch. Namely, we reuse
> apply_relocate_add() in the module loader to write relocs instead of
> duplicating functionality in livepatch's klp_write_module_reloc(). To
> apply relocation sections, remaining SHN_LIVEPATCH symbols referenced by
> relocs are resolved and then apply_relocate_add() is called to apply
> those relocations.
> 
> Signed-off-by: Jessica Yu <jeyu-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> ---
>  include/linux/livepatch.h | 11 ++++--
>  include/linux/module.h    |  6 ++++
>  kernel/livepatch/core.c   | 89 +++++++++++++++++++++++++++++------------------
>  3 files changed, 70 insertions(+), 36 deletions(-)
> 
> diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
> index 31db7a0..601e892 100644
> --- a/include/linux/livepatch.h
> +++ b/include/linux/livepatch.h
> @@ -85,7 +85,7 @@ struct klp_reloc {
>  /**
>   * struct klp_object - kernel object structure for live patching
>   * @name:	module name (or NULL for vmlinux)
> - * @relocs:	relocation entries to be applied at load time
> + * @reloc_secs:	relocation sections to be applied at load time
>   * @funcs:	function entries for functions to be patched in the object
>   * @kobj:	kobject for sysfs resources
>   * @mod:	kernel module associated with the patched object
> @@ -95,7 +95,7 @@ struct klp_reloc {
>  struct klp_object {
>  	/* external */
>  	const char *name;
> -	struct klp_reloc *relocs;
> +	struct list_head reloc_secs;
>  	struct klp_func *funcs;
>  
>  	/* internal */
> @@ -129,6 +129,13 @@ struct klp_patch {
>  #define klp_for_each_func(obj, func) \
>  	for (func = obj->funcs; func->old_name; func++)
>  
> +struct klp_reloc_sec {
> +	unsigned int index;
> +	char *name;
> +	char *objname;
> +	struct list_head list;
> +};
> +
>  int klp_register_patch(struct klp_patch *);
>  int klp_unregister_patch(struct klp_patch *);
>  int klp_enable_patch(struct klp_patch *);
> diff --git a/include/linux/module.h b/include/linux/module.h
> index c8680b1..3c34eb8 100644
> --- a/include/linux/module.h
> +++ b/include/linux/module.h
> @@ -793,9 +793,15 @@ extern int module_sysfs_initialized;
>  #ifdef CONFIG_DEBUG_SET_MODULE_RONX
>  extern void set_all_modules_text_rw(void);
>  extern void set_all_modules_text_ro(void);
> +extern void
> +set_page_attributes(void *start, void *end,
> +		    int (*set)(unsigned long start, int num_pages));
>  #else
>  static inline void set_all_modules_text_rw(void) { }
>  static inline void set_all_modules_text_ro(void) { }
> +static inline void
> +set_page_attributes(void *start, void *end,
> +		    int (*set)(unsigned long start, int num_pages)) { }
>  #endif
>  
>  #ifdef CONFIG_GENERIC_BUG
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index 087a8c7..26c419f 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -28,6 +28,8 @@
>  #include <linux/list.h>
>  #include <linux/kallsyms.h>
>  #include <linux/livepatch.h>
> +#include <linux/elf.h>
> +#include <asm/cacheflush.h>
>  
>  /**
>   * struct klp_ops - structure for tracking registered ftrace ops structs
> @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module *pmod, const char *name,
>  }
>  
>  static int klp_write_object_relocations(struct module *pmod,
> -					struct klp_object *obj)
> +					struct klp_object *obj,
> +					struct klp_patch *patch)
>  {
> -	int ret;
> -	struct klp_reloc *reloc;
> +	int relindex, num_relas;
> +	int i, ret = 0;
> +	unsigned long addr;
> +	unsigned int bind;
> +	char *symname;
> +	struct klp_reloc_sec *reloc_sec;
> +	struct load_info *info;
> +	Elf_Rela *rela;
> +	Elf_Sym *sym, *symtab;
> +	Elf_Shdr *symsect;
>  
>  	if (WARN_ON(!klp_is_object_loaded(obj)))
>  		return -EINVAL;
>  
> -	if (WARN_ON(!obj->relocs))
> -		return -EINVAL;
> -
> -	for (reloc = obj->relocs; reloc->name; reloc++) {
> -		if (!klp_is_module(obj)) {
> -			ret = klp_verify_vmlinux_symbol(reloc->name,
> -							reloc->val);
> -			if (ret)
> -				return ret;
> -		} else {
> -			/* module, reloc->val needs to be discovered */
> -			if (reloc->external)
> -				ret = klp_find_external_symbol(pmod,
> -							       reloc->name,
> -							       &reloc->val);
> -			else
> -				ret = klp_find_object_symbol(obj->mod->name,
> -							     reloc->name,
> -							     &reloc->val);
> -			if (ret)
> -				return ret;
> -		}
> -		ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc,
> -					     reloc->val + reloc->addend);
> -		if (ret) {
> -			pr_err("relocation failed for symbol '%s' at 0x%016lx (%d)\n",
> -			       reloc->name, reloc->val, ret);
> -			return ret;
> +	info = pmod->info;
> +	symsect = info->sechdrs + info->index.sym;
> +	symtab = (void *)info->hdr + symsect->sh_offset;
> +
> +	/* For each __klp_rela section for this object */
> +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
> +		relindex = reloc_sec->index;
> +		num_relas = info->sechdrs[relindex].sh_size / sizeof(Elf_Rela);
> +		rela = (Elf_Rela *) info->sechdrs[relindex].sh_addr;
> +
> +		/* For each rela in this __klp_rela section */
> +		for (i = 0; i < num_relas; i++, rela++) {
> +			sym = symtab + ELF_R_SYM(rela->r_info);
> +			symname = info->strtab + sym->st_name;
> +			bind = ELF_ST_BIND(sym->st_info);
> +
> +			if (sym->st_shndx == SHN_LIVEPATCH) {
> +				if (bind == STB_LIVEPATCH_EXT)
> +					ret = klp_find_external_symbol(pmod, symname, &addr);
> +				else
> +					ret = klp_find_object_symbol(obj->name, symname, &addr);
> +				if (ret)
> +					return ret;
> +				sym->st_value = addr;
> +			}

This is missing an important piece, I think.  There's no way to
disambiguate duplicate symbols.  Before, there was reloc->val, which
contained the symbol's address.  There's nothing like that here.

I think we could use sym->st_value for that purpose.  After Chris's
patch set gets merged, it could contain the sympos.

>  		}
> +		ret = apply_relocate_add(info->sechdrs, info->strtab,
> +					 info->index.sym, relindex, pmod);
>  	}
>  
> -	return 0;
> +	return ret;
>  }
>  
>  static void notrace klp_ftrace_handler(unsigned long ip,
> @@ -741,12 +751,23 @@ static int klp_init_object_loaded(struct klp_patch *patch,
>  				  struct klp_object *obj)
>  {
>  	struct klp_func *func;
> +	struct module *pmod;
>  	int ret;
>  
> -	if (obj->relocs) {
> -		ret = klp_write_object_relocations(patch->mod, obj);
> +	pmod = patch->mod;
> +
> +	if (!list_empty(&obj->reloc_secs)) {
> +		set_page_attributes(pmod->module_core,
> +				    pmod->module_core + pmod->core_text_size,
> +				    set_memory_rw);
> +
> +		ret = klp_write_object_relocations(pmod, obj, patch);
>  		if (ret)
>  			return ret;
> +
> +		set_page_attributes(pmod->module_core,
> +				    pmod->module_core + pmod->core_text_size,
> +				    set_memory_ro);
>  	}
>  
>  	klp_for_each_func(obj, func) {
> -- 
> 2.4.3
> 

-- 
Josh

Re: [RFC PATCH 5/5] livepatch: x86: remove unused relocation code

[Josh Poimboeuf]

On Wed, Nov 11, 2015 at 04:48:07PM +0100, Petr Mladek wrote:
> On Mon 2015-11-09 23:45:55, Jessica Yu wrote:
> > Architecture-specific relocation code no longer needed, since symbol
> > resolution and relocation work will be offloaded to module loader.
>                                  ^^^^
> 				 has been
> 
> I would personally merge this with the 3rd patch. It will be more
> clear what code has been replaced there. But it is a matter of taste.

Ditto.

-- 
Josh

Re: livepatch: reuse module loader code to write relocations

[Jessica Yu]

+++ Miroslav Benes [12/11/15 16:27 +0100]:
>On Wed, 11 Nov 2015, Jessica Yu wrote:
>
>> +++ Miroslav Benes [11/11/15 15:30 +0100]:
>> > On Mon, 9 Nov 2015, Jessica Yu wrote:
>> >
>> > So I guess we don't need klp_reloc anymore.
>>
>> Yes, that's correct. I am noticing just now that I forgot to remove
>> the klp_reloc struct definition from livepatch.h. That change will be
>> reflected in v2...
>>
>> > If true, we should really
>> > start thinking about proper documentation because there are going to be
>> > plenty of assumptions about a patch module and we need to have it written
>> > somewhere. Especially how the relocation sections look like.
>>
>> Agreed. As a first step the patch module format can perhaps be
>> documented somewhere. Perhaps it's time we create
>> Documentation/livepatch/? :-)
>
>Yes, I think so.
>
>> > > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>> > > index 087a8c7..26c419f 100644
>> > > --- a/kernel/livepatch/core.c
>> > > +++ b/kernel/livepatch/core.c
>> > > @@ -28,6 +28,8 @@
>> > >  #include <linux/list.h>
>> > >  #include <linux/kallsyms.h>
>> > >  #include <linux/livepatch.h>
>> > > +#include <linux/elf.h>
>> > > +#include <asm/cacheflush.h>
>> > >
>> > >  /**
>> > >   * struct klp_ops - structure for tracking registered ftrace ops structs
>> > > @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module
>> > > *pmod, const char *name,
>> > >  }
>> > >
>> > >  static int klp_write_object_relocations(struct module *pmod,
>> > > -					struct klp_object *obj)
>> > > +					struct klp_object *obj,
>> > > +					struct klp_patch *patch)
>> > >  {
>> > > -	int ret;
>> > > -	struct klp_reloc *reloc;
>> > > +	int relindex, num_relas;
>> > > +	int i, ret = 0;
>> > > +	unsigned long addr;
>> > > +	unsigned int bind;
>> > > +	char *symname;
>> > > +	struct klp_reloc_sec *reloc_sec;
>> > > +	struct load_info *info;
>> > > +	Elf_Rela *rela;
>> > > +	Elf_Sym *sym, *symtab;
>> > > +	Elf_Shdr *symsect;
>> > >
>> > >  	if (WARN_ON(!klp_is_object_loaded(obj)))
>> > >  		return -EINVAL;
>> > >
>> > > -	if (WARN_ON(!obj->relocs))
>> > > -		return -EINVAL;
>> > > -
>> > > -	for (reloc = obj->relocs; reloc->name; reloc++) {
>> > > -		if (!klp_is_module(obj)) {
>> > > -			ret = klp_verify_vmlinux_symbol(reloc->name,
>> > > -							reloc->val);
>> > > -			if (ret)
>> > > -				return ret;
>> > > -		} else {
>> > > -			/* module, reloc->val needs to be discovered */
>> > > -			if (reloc->external)
>> > > -				ret = klp_find_external_symbol(pmod,
>> > > -							       reloc->name,
>> > > -							       &reloc->val);
>> > > -			else
>> > > -				ret = klp_find_object_symbol(obj->mod->name,
>> > > -							     reloc->name,
>> > > -							     &reloc->val);
>> > > -			if (ret)
>> > > -				return ret;
>> > > -		}
>> > > -		ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc,
>> > > -					     reloc->val + reloc->addend);
>> > > -		if (ret) {
>> > > -			pr_err("relocation failed for symbol '%s' at 0x%016lx
>> > > (%d)\n",
>> > > -			       reloc->name, reloc->val, ret);
>> > > -			return ret;
>> > > +	info = pmod->info;
>> > > +	symsect = info->sechdrs + info->index.sym;
>> > > +	symtab = (void *)info->hdr + symsect->sh_offset;
>> > > +
>> > > +	/* For each __klp_rela section for this object */
>> > > +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
>> > > +		relindex = reloc_sec->index;
>> > > +		num_relas = info->sechdrs[relindex].sh_size /
>> > > sizeof(Elf_Rela);
>> > > +		rela = (Elf_Rela *) info->sechdrs[relindex].sh_addr;
>> > > +
>> > > +		/* For each rela in this __klp_rela section */
>> > > +		for (i = 0; i < num_relas; i++, rela++) {
>> > > +			sym = symtab + ELF_R_SYM(rela->r_info);
>> > > +			symname = info->strtab + sym->st_name;
>> > > +			bind = ELF_ST_BIND(sym->st_info);
>> > > +
>> > > +			if (sym->st_shndx == SHN_LIVEPATCH) {
>> > > +				if (bind == STB_LIVEPATCH_EXT)
>> > > +					ret = klp_find_external_symbol(pmod,
>> > > symname, &addr);
>> > > +				else
>> > > +					ret =
>> > > klp_find_object_symbol(obj->name, symname, &addr);
>> > > +				if (ret)
>> > > +					return ret;
>> > > +				sym->st_value = addr;
>> > > +			}
>> > >  		}
>> > > +		ret = apply_relocate_add(info->sechdrs, info->strtab,
>> > > +					 info->index.sym, relindex, pmod);
>> > >  	}
>> > >
>> > > -	return 0;
>> > > +	return ret;
>> > >  }
>> >
>> > Looking at this... do we even need reloc_secs in klp_object? Question is
>> > whether we need more than one dynrela section for an object. If not then
>> > the binding between klp_reloc_sec and an object is the only relevant thing
>> > in the structure, be it index or objname. So we can replace the
>> > list of structures with just the index in klp_object, or get rid of it
>> > completely and rely on the name of dynrela section be something like
>> > __klp_rela_{objname}.
>>
>> Hm, you bring up a good point. I think theoretically yes, it is
>> possible to just have one klp_reloc_sec for each object and therefore
>> a list is not required (I have not checked yet how difficult it would
>> be to implement this on the kpatch-build side of things).  However,
>> considering the final format of the patch module, I think it is
>> semantically clearer to leave it as a list, and for each object to
>> possibly have more than one __klp_rela section.
>>
>> For example, say we are patching two functions in ext4. In my
>> resulting kpatch module I will have two __klp_rela_ext4 sections, and
>> they might look like this when we run readelf --sections:
>>
>> [34] __klp_rela_ext4.text.ext4_attr_store RELA ...
>> [35] __klp_rela_ext4.text.ext4_attr_show RELA ...
>>
>> Then these two klp rela sections end up as two elements in the
>> reloc_secs list for the ext4 patch object. I think this way, we can
>> better tell which rela is being applied to what function. Might be
>> easier to understand what's happening from the developer's point of
>> view.
>>
>> > You see, we go through elf sections here which were preserved by module
>> > loader. We even have relevant sections marked with SHF_RELA_LIVEPATCH. So
>> > maybe all the stuff around klp_reloc_sec is not necessary.
>> >
>> > Thoughts?
>>
>> Ah, so this is where descriptive comments and documentation might have
>> been useful :-) So I think we will still need to keep the
>> klp_reloc_sec struct to help the patch module initialize. Though the
>> name and objname fields aren't used in this patchset, they are used in
>> the kpatch patch module code [1], where we iterate through each elf
>> section, find the ones marked with SHF_RELA_LIVEPATCH, set the
>> klp_reloc_sec's objname (which we find from the "name" field,
>> formatted as __klp_rela_{objname}.text..). Once we have the objname
>> set, we can then find the object to attach the reloc_sec to (i.e. add
>> it to its list of reloc_secs).
>>
>> Hope that clears some things up.
>
>Ok, I'll try to explain myself and it is gonna be long. I'll try to
>describe how we deal with dynrelas in klp today, how you use it in kpatch
>(and this is the place where my knowledge can be wrong or obsolete), what
>you propose and what I'd like to propose.
>
>1. First let's look on what we have now.
>
>We have a patch module in which there is a section with all dynrelas
>needed to be resolved (it was like this in kpatch some time ago and maybe
>it is different now so just have a patience, I'll get to it). In the init
>function of the module kpatch builds all the relevant info from dynrela
>section. It goes through it, creates an array of klp_reloc for each object
>and includes each dynrela record with an objname to the array of
>klp_object with that objname. Later when we need to apply relocations for
>patched object we just go through the list (array) of its dynrelas in
>klp_object and call our arch-specific klp_write_module_reloc().

Sounds correct to me.

>Now this was probably changed in kpatch and you do not have one dynrela
>section but one dynrela section for each patched function. Is that
>correct? (and can you tell us what the reason for the change was? It might
>be crucial because I might be missing something.). Which leads us to your
>proposal...

Your original assumption was correct; current kpatch has one big
.kpatch.dynrelas section, and each dynrela entry within that single
section gets sorted to the correct object as you described above. The
one dynrela section per patched function only came with this patchset,
but for no particular reason other than to make the kpatch-build code
for generating the patch module slightly less complicated. But I
haven't checked how big of a change it would be to do
one-dynrela-section per object, perhaps (and I hope) it will be an
easy change.

>2. So we have several dynrela section for one patched object. During init
>function in a patch module kpatch once again builds needed structures from
>these sections. Now they are called klp_reloc_sec and contain different
>kind of info. There is no val, loc and such, only name of the symbol,
>objname and index to dynrela section in ELF. So when you need to write
>relocations for the patched object you go through all relevant dynrela
>sections (because they are stored in the klp_object), all dynrela records
>in each section and you resolve the undefined symbols. All needed info is
>stored in ELF. Then you just call apply_relocate_add().
>
>3. I propose to go one step further. I think we don't need klp_reloc_sec
>if there is only one dynrela section for patched object (and I currently
>cannot see why this is not possible. It is possible even with one dynrela
>section for whole patch module, that is for all patched objects.).

I think the furthest we can go in terms of simplifying klp rela secs
is to have one __klp_rela section per object. We can't smush all the
__klp_rela_objname sections into one big __klp_rela section since we
could be patching some objects that won't be loaded yet, and
apply_relocate_add() needs to work with real SHT_RELA sections + their
corresponding section indices (i.e., we cannot call
apply_relocate_add() with that single, combined klp rela section).

So, I think I would be OK with one __klp_rela section per object.

>In my proposal there would be nothing done in init function of the patched
>module (albeit some optimization mentioned later). When you call
>klp_write_object_relocations you would go through all ELF section and find
>the relevant one for the object (it is marked with SHF_RELA_LIVEPATCH and
>objname is in the name of the section. It is the same thing you do in 2.
>in the init function.). Then you go through all dynrela records in the
>section, you do the same things which you do in the proposed patch above,
>and call apply_relocate_add.

I'm not sure I like having klp_write_object_relocations() repeatedly
perform a loop through all the elf sections when we can pre-process
this information in the patch module's init function, and "cache" the
relevant klp section indices before passing things off to
klp_write_object_relocations(). So how about this: we do the
__klp_rela sec sorting in the init function of the patched module. The
patch module would iterate through its own elf sections, and when it
encounters a __klp_rela section, it looks at its objname, find the
corresponding klp_object, and save the section index of the __klp_rela
section in that klp_object. Then in klp_write_object_relocations, we
just have to look at the saved section index for the corresponding
object and access the klp rela section through that index, do the same
processing in this patch and call apply_relocate_add().

>Now it would be crazy to go through all sections each time
>klp_write_object_relocations is called (maybe it does not even matter but
>nevertheless). So klp_object could have an index to its ELF dynrela
>section. This index can be retrieved in the init function the same way you
>do all the stuff with klp_reloc_sec.

Ah, exactly what I said above :-D

>If you insisted on more than one dynrela section for a patched object we
>could have an array of indices there. Or whatever.
>
>It is almost the same as your proposal but in my opinion somewhat nicer.
>We just use the info stored in ELF directly without unnecessary middle
>layer (== klp_reloc_sec).
>
>Does it make sense? I hope it does. Would it work?

It does make sense, and I think we can make it work without
klp_reloc_sec. Thanks for the explanations.

Jessica

Re: livepatch: reuse module loader code to write relocations

[Jessica Yu]

+++ Miroslav Benes [12/11/15 16:27 +0100]:
>On Wed, 11 Nov 2015, Jessica Yu wrote:
>
>> +++ Miroslav Benes [11/11/15 15:30 +0100]:
>> > On Mon, 9 Nov 2015, Jessica Yu wrote:
>> >
>> > So I guess we don't need klp_reloc anymore.
>>
>> Yes, that's correct. I am noticing just now that I forgot to remove
>> the klp_reloc struct definition from livepatch.h. That change will be
>> reflected in v2...
>>
>> > If true, we should really
>> > start thinking about proper documentation because there are going to be
>> > plenty of assumptions about a patch module and we need to have it written
>> > somewhere. Especially how the relocation sections look like.
>>
>> Agreed. As a first step the patch module format can perhaps be
>> documented somewhere. Perhaps it's time we create
>> Documentation/livepatch/? :-)
>
>Yes, I think so.
>
>> > > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>> > > index 087a8c7..26c419f 100644
>> > > --- a/kernel/livepatch/core.c
>> > > +++ b/kernel/livepatch/core.c
>> > > @@ -28,6 +28,8 @@
>> > >  #include <linux/list.h>
>> > >  #include <linux/kallsyms.h>
>> > >  #include <linux/livepatch.h>
>> > > +#include <linux/elf.h>
>> > > +#include <asm/cacheflush.h>
>> > >
>> > >  /**
>> > >   * struct klp_ops - structure for tracking registered ftrace ops structs
>> > > @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module
>> > > *pmod, const char *name,
>> > >  }
>> > >
>> > >  static int klp_write_object_relocations(struct module *pmod,
>> > > -					struct klp_object *obj)
>> > > +					struct klp_object *obj,
>> > > +					struct klp_patch *patch)
>> > >  {
>> > > -	int ret;
>> > > -	struct klp_reloc *reloc;
>> > > +	int relindex, num_relas;
>> > > +	int i, ret = 0;
>> > > +	unsigned long addr;
>> > > +	unsigned int bind;
>> > > +	char *symname;
>> > > +	struct klp_reloc_sec *reloc_sec;
>> > > +	struct load_info *info;
>> > > +	Elf_Rela *rela;
>> > > +	Elf_Sym *sym, *symtab;
>> > > +	Elf_Shdr *symsect;
>> > >
>> > >  	if (WARN_ON(!klp_is_object_loaded(obj)))
>> > >  		return -EINVAL;
>> > >
>> > > -	if (WARN_ON(!obj->relocs))
>> > > -		return -EINVAL;
>> > > -
>> > > -	for (reloc = obj->relocs; reloc->name; reloc++) {
>> > > -		if (!klp_is_module(obj)) {
>> > > -			ret = klp_verify_vmlinux_symbol(reloc->name,
>> > > -							reloc->val);
>> > > -			if (ret)
>> > > -				return ret;
>> > > -		} else {
>> > > -			/* module, reloc->val needs to be discovered */
>> > > -			if (reloc->external)
>> > > -				ret = klp_find_external_symbol(pmod,
>> > > -							       reloc->name,
>> > > -							       &reloc->val);
>> > > -			else
>> > > -				ret = klp_find_object_symbol(obj->mod->name,
>> > > -							     reloc->name,
>> > > -							     &reloc->val);
>> > > -			if (ret)
>> > > -				return ret;
>> > > -		}
>> > > -		ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc,
>> > > -					     reloc->val + reloc->addend);
>> > > -		if (ret) {
>> > > -			pr_err("relocation failed for symbol '%s' at 0x%016lx
>> > > (%d)\n",
>> > > -			       reloc->name, reloc->val, ret);
>> > > -			return ret;
>> > > +	info = pmod->info;
>> > > +	symsect = info->sechdrs + info->index.sym;
>> > > +	symtab = (void *)info->hdr + symsect->sh_offset;
>> > > +
>> > > +	/* For each __klp_rela section for this object */
>> > > +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
>> > > +		relindex = reloc_sec->index;
>> > > +		num_relas = info->sechdrs[relindex].sh_size /
>> > > sizeof(Elf_Rela);
>> > > +		rela = (Elf_Rela *) info->sechdrs[relindex].sh_addr;
>> > > +
>> > > +		/* For each rela in this __klp_rela section */
>> > > +		for (i = 0; i < num_relas; i++, rela++) {
>> > > +			sym = symtab + ELF_R_SYM(rela->r_info);
>> > > +			symname = info->strtab + sym->st_name;
>> > > +			bind = ELF_ST_BIND(sym->st_info);
>> > > +
>> > > +			if (sym->st_shndx == SHN_LIVEPATCH) {
>> > > +				if (bind == STB_LIVEPATCH_EXT)
>> > > +					ret = klp_find_external_symbol(pmod,
>> > > symname, &addr);
>> > > +				else
>> > > +					ret =
>> > > klp_find_object_symbol(obj->name, symname, &addr);
>> > > +				if (ret)
>> > > +					return ret;
>> > > +				sym->st_value = addr;
>> > > +			}
>> > >  		}
>> > > +		ret = apply_relocate_add(info->sechdrs, info->strtab,
>> > > +					 info->index.sym, relindex, pmod);
>> > >  	}
>> > >
>> > > -	return 0;
>> > > +	return ret;
>> > >  }
>> >
>> > Looking at this... do we even need reloc_secs in klp_object? Question is
>> > whether we need more than one dynrela section for an object. If not then
>> > the binding between klp_reloc_sec and an object is the only relevant thing
>> > in the structure, be it index or objname. So we can replace the
>> > list of structures with just the index in klp_object, or get rid of it
>> > completely and rely on the name of dynrela section be something like
>> > __klp_rela_{objname}.
>>
>> Hm, you bring up a good point. I think theoretically yes, it is
>> possible to just have one klp_reloc_sec for each object and therefore
>> a list is not required (I have not checked yet how difficult it would
>> be to implement this on the kpatch-build side of things).  However,
>> considering the final format of the patch module, I think it is
>> semantically clearer to leave it as a list, and for each object to
>> possibly have more than one __klp_rela section.
>>
>> For example, say we are patching two functions in ext4. In my
>> resulting kpatch module I will have two __klp_rela_ext4 sections, and
>> they might look like this when we run readelf --sections:
>>
>> [34] __klp_rela_ext4.text.ext4_attr_store RELA ...
>> [35] __klp_rela_ext4.text.ext4_attr_show RELA ...
>>
>> Then these two klp rela sections end up as two elements in the
>> reloc_secs list for the ext4 patch object. I think this way, we can
>> better tell which rela is being applied to what function. Might be
>> easier to understand what's happening from the developer's point of
>> view.
>>
>> > You see, we go through elf sections here which were preserved by module
>> > loader. We even have relevant sections marked with SHF_RELA_LIVEPATCH. So
>> > maybe all the stuff around klp_reloc_sec is not necessary.
>> >
>> > Thoughts?
>>
>> Ah, so this is where descriptive comments and documentation might have
>> been useful :-) So I think we will still need to keep the
>> klp_reloc_sec struct to help the patch module initialize. Though the
>> name and objname fields aren't used in this patchset, they are used in
>> the kpatch patch module code [1], where we iterate through each elf
>> section, find the ones marked with SHF_RELA_LIVEPATCH, set the
>> klp_reloc_sec's objname (which we find from the "name" field,
>> formatted as __klp_rela_{objname}.text..). Once we have the objname
>> set, we can then find the object to attach the reloc_sec to (i.e. add
>> it to its list of reloc_secs).
>>
>> Hope that clears some things up.
>
>Ok, I'll try to explain myself and it is gonna be long. I'll try to
>describe how we deal with dynrelas in klp today, how you use it in kpatch
>(and this is the place where my knowledge can be wrong or obsolete), what
>you propose and what I'd like to propose.
>
>1. First let's look on what we have now.
>
>We have a patch module in which there is a section with all dynrelas
>needed to be resolved (it was like this in kpatch some time ago and maybe
>it is different now so just have a patience, I'll get to it). In the init
>function of the module kpatch builds all the relevant info from dynrela
>section. It goes through it, creates an array of klp_reloc for each object
>and includes each dynrela record with an objname to the array of
>klp_object with that objname. Later when we need to apply relocations for
>patched object we just go through the list (array) of its dynrelas in
>klp_object and call our arch-specific klp_write_module_reloc().

Sounds correct to me.

>Now this was probably changed in kpatch and you do not have one dynrela
>section but one dynrela section for each patched function. Is that
>correct? (and can you tell us what the reason for the change was? It might
>be crucial because I might be missing something.). Which leads us to your
>proposal...

Your original assumption was correct; current kpatch has one big
.kpatch.dynrelas section, and each dynrela entry within that single
section gets sorted to the correct object as you described above. The
one dynrela section per patched function only came with this patchset,
but for no particular reason other than to make the kpatch-build code
for generating the patch module slightly less complicated. But I
haven't checked how big of a change it would be to do
one-dynrela-section per object, perhaps (and I hope) it will be an
easy change.

>2. So we have several dynrela section for one patched object. During init
>function in a patch module kpatch once again builds needed structures from
>these sections. Now they are called klp_reloc_sec and contain different
>kind of info. There is no val, loc and such, only name of the symbol,
>objname and index to dynrela section in ELF. So when you need to write
>relocations for the patched object you go through all relevant dynrela
>sections (because they are stored in the klp_object), all dynrela records
>in each section and you resolve the undefined symbols. All needed info is
>stored in ELF. Then you just call apply_relocate_add().
>
>3. I propose to go one step further. I think we don't need klp_reloc_sec
>if there is only one dynrela section for patched object (and I currently
>cannot see why this is not possible. It is possible even with one dynrela
>section for whole patch module, that is for all patched objects.).

I think the furthest we can go in terms of simplifying klp rela secs
is to have one __klp_rela section per object. We can't smush all the
__klp_rela_objname sections into one big __klp_rela section since we
could be patching some objects that won't be loaded yet, and
apply_relocate_add() needs to work with real SHT_RELA sections + their
corresponding section indices (i.e., we cannot call
apply_relocate_add() with that single, combined klp rela section).

So, I think I would be OK with one __klp_rela section per object.

>In my proposal there would be nothing done in init function of the patched
>module (albeit some optimization mentioned later). When you call
>klp_write_object_relocations you would go through all ELF section and find
>the relevant one for the object (it is marked with SHF_RELA_LIVEPATCH and
>objname is in the name of the section. It is the same thing you do in 2.
>in the init function.). Then you go through all dynrela records in the
>section, you do the same things which you do in the proposed patch above,
>and call apply_relocate_add.

I'm not sure I like having klp_write_object_relocations() repeatedly
perform a loop through all the elf sections when we can pre-process
this information in the patch module's init function, and "cache" the
relevant klp section indices before passing things off to
klp_write_object_relocations(). So how about this: we do the
__klp_rela sec sorting in the init function of the patched module. The
patch module would iterate through its own elf sections, and when it
encounters a __klp_rela section, it looks at its objname, find the
corresponding klp_object, and save the section index of the __klp_rela
section in that klp_object. Then in klp_write_object_relocations, we
just have to look at the saved section index for the corresponding
object and access the klp rela section through that index, do the same
processing in this patch and call apply_relocate_add().

>Now it would be crazy to go through all sections each time
>klp_write_object_relocations is called (maybe it does not even matter but
>nevertheless). So klp_object could have an index to its ELF dynrela
>section. This index can be retrieved in the init function the same way you
>do all the stuff with klp_reloc_sec.

Ah, exactly what I said above :-D

>If you insisted on more than one dynrela section for a patched object we
>could have an array of indices there. Or whatever.
>
>It is almost the same as your proposal but in my opinion somewhat nicer.
>We just use the info stored in ELF directly without unnecessary middle
>layer (== klp_reloc_sec).
>
>Does it make sense? I hope it does. Would it work?

It does make sense, and I think we can make it work without
klp_reloc_sec. Thanks for the explanations.

Jessica

Re: livepatch: reuse module loader code to write relocations

[Jessica Yu]

+++ Josh Poimboeuf [12/11/15 11:40 -0600]:
>On Thu, Nov 12, 2015 at 04:27:01PM +0100, Miroslav Benes wrote:
>> On Wed, 11 Nov 2015, Jessica Yu wrote:
>>
>> > +++ Miroslav Benes [11/11/15 15:30 +0100]:
>> > > On Mon, 9 Nov 2015, Jessica Yu wrote:
>> > >
>> > > So I guess we don't need klp_reloc anymore.
>> >
>> > Yes, that's correct. I am noticing just now that I forgot to remove
>> > the klp_reloc struct definition from livepatch.h. That change will be
>> > reflected in v2...
>> >
>> > > If true, we should really
>> > > start thinking about proper documentation because there are going to be
>> > > plenty of assumptions about a patch module and we need to have it written
>> > > somewhere. Especially how the relocation sections look like.
>> >
>> > Agreed. As a first step the patch module format can perhaps be
>> > documented somewhere. Perhaps it's time we create
>> > Documentation/livepatch/? :-)
>>
>> Yes, I think so.
>>
>> > > > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>> > > > index 087a8c7..26c419f 100644
>> > > > --- a/kernel/livepatch/core.c
>> > > > +++ b/kernel/livepatch/core.c
>> > > > @@ -28,6 +28,8 @@
>> > > >  #include <linux/list.h>
>> > > >  #include <linux/kallsyms.h>
>> > > >  #include <linux/livepatch.h>
>> > > > +#include <linux/elf.h>
>> > > > +#include <asm/cacheflush.h>
>> > > >
>> > > >  /**
>> > > >   * struct klp_ops - structure for tracking registered ftrace ops structs
>> > > > @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module
>> > > > *pmod, const char *name,
>> > > >  }
>> > > >
>> > > >  static int klp_write_object_relocations(struct module *pmod,
>> > > > -					struct klp_object *obj)
>> > > > +					struct klp_object *obj,
>> > > > +					struct klp_patch *patch)
>> > > >  {
>> > > > -	int ret;
>> > > > -	struct klp_reloc *reloc;
>> > > > +	int relindex, num_relas;
>> > > > +	int i, ret = 0;
>> > > > +	unsigned long addr;
>> > > > +	unsigned int bind;
>> > > > +	char *symname;
>> > > > +	struct klp_reloc_sec *reloc_sec;
>> > > > +	struct load_info *info;
>> > > > +	Elf_Rela *rela;
>> > > > +	Elf_Sym *sym, *symtab;
>> > > > +	Elf_Shdr *symsect;
>> > > >
>> > > >  	if (WARN_ON(!klp_is_object_loaded(obj)))
>> > > >  		return -EINVAL;
>> > > >
>> > > > -	if (WARN_ON(!obj->relocs))
>> > > > -		return -EINVAL;
>> > > > -
>> > > > -	for (reloc = obj->relocs; reloc->name; reloc++) {
>> > > > -		if (!klp_is_module(obj)) {
>> > > > -			ret = klp_verify_vmlinux_symbol(reloc->name,
>> > > > -							reloc->val);
>> > > > -			if (ret)
>> > > > -				return ret;
>> > > > -		} else {
>> > > > -			/* module, reloc->val needs to be discovered */
>> > > > -			if (reloc->external)
>> > > > -				ret = klp_find_external_symbol(pmod,
>> > > > -							       reloc->name,
>> > > > -							       &reloc->val);
>> > > > -			else
>> > > > -				ret = klp_find_object_symbol(obj->mod->name,
>> > > > -							     reloc->name,
>> > > > -							     &reloc->val);
>> > > > -			if (ret)
>> > > > -				return ret;
>> > > > -		}
>> > > > -		ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc,
>> > > > -					     reloc->val + reloc->addend);
>> > > > -		if (ret) {
>> > > > -			pr_err("relocation failed for symbol '%s' at 0x%016lx
>> > > > (%d)\n",
>> > > > -			       reloc->name, reloc->val, ret);
>> > > > -			return ret;
>> > > > +	info = pmod->info;
>> > > > +	symsect = info->sechdrs + info->index.sym;
>> > > > +	symtab = (void *)info->hdr + symsect->sh_offset;
>> > > > +
>> > > > +	/* For each __klp_rela section for this object */
>> > > > +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
>> > > > +		relindex = reloc_sec->index;
>> > > > +		num_relas = info->sechdrs[relindex].sh_size /
>> > > > sizeof(Elf_Rela);
>> > > > +		rela = (Elf_Rela *) info->sechdrs[relindex].sh_addr;
>> > > > +
>> > > > +		/* For each rela in this __klp_rela section */
>> > > > +		for (i = 0; i < num_relas; i++, rela++) {
>> > > > +			sym = symtab + ELF_R_SYM(rela->r_info);
>> > > > +			symname = info->strtab + sym->st_name;
>> > > > +			bind = ELF_ST_BIND(sym->st_info);
>> > > > +
>> > > > +			if (sym->st_shndx == SHN_LIVEPATCH) {
>> > > > +				if (bind == STB_LIVEPATCH_EXT)
>> > > > +					ret = klp_find_external_symbol(pmod,
>> > > > symname, &addr);
>> > > > +				else
>> > > > +					ret =
>> > > > klp_find_object_symbol(obj->name, symname, &addr);
>> > > > +				if (ret)
>> > > > +					return ret;
>> > > > +				sym->st_value = addr;
>> > > > +			}
>> > > >  		}
>> > > > +		ret = apply_relocate_add(info->sechdrs, info->strtab,
>> > > > +					 info->index.sym, relindex, pmod);
>> > > >  	}
>> > > >
>> > > > -	return 0;
>> > > > +	return ret;
>> > > >  }
>> > >
>> > > Looking at this... do we even need reloc_secs in klp_object? Question is
>> > > whether we need more than one dynrela section for an object. If not then
>> > > the binding between klp_reloc_sec and an object is the only relevant thing
>> > > in the structure, be it index or objname. So we can replace the
>> > > list of structures with just the index in klp_object, or get rid of it
>> > > completely and rely on the name of dynrela section be something like
>> > > __klp_rela_{objname}.
>> >
>> > Hm, you bring up a good point. I think theoretically yes, it is
>> > possible to just have one klp_reloc_sec for each object and therefore
>> > a list is not required (I have not checked yet how difficult it would
>> > be to implement this on the kpatch-build side of things).  However,
>> > considering the final format of the patch module, I think it is
>> > semantically clearer to leave it as a list, and for each object to
>> > possibly have more than one __klp_rela section.
>> >
>> > For example, say we are patching two functions in ext4. In my
>> > resulting kpatch module I will have two __klp_rela_ext4 sections, and
>> > they might look like this when we run readelf --sections:
>> >
>> > [34] __klp_rela_ext4.text.ext4_attr_store RELA ...
>> > [35] __klp_rela_ext4.text.ext4_attr_show RELA ...
>> >
>> > Then these two klp rela sections end up as two elements in the
>> > reloc_secs list for the ext4 patch object. I think this way, we can
>> > better tell which rela is being applied to what function. Might be
>> > easier to understand what's happening from the developer's point of
>> > view.
>> >
>> > > You see, we go through elf sections here which were preserved by module
>> > > loader. We even have relevant sections marked with SHF_RELA_LIVEPATCH. So
>> > > maybe all the stuff around klp_reloc_sec is not necessary.
>> > >
>> > > Thoughts?
>> >
>> > Ah, so this is where descriptive comments and documentation might have
>> > been useful :-) So I think we will still need to keep the
>> > klp_reloc_sec struct to help the patch module initialize. Though the
>> > name and objname fields aren't used in this patchset, they are used in
>> > the kpatch patch module code [1], where we iterate through each elf
>> > section, find the ones marked with SHF_RELA_LIVEPATCH, set the
>> > klp_reloc_sec's objname (which we find from the "name" field,
>> > formatted as __klp_rela_{objname}.text..). Once we have the objname
>> > set, we can then find the object to attach the reloc_sec to (i.e. add
>> > it to its list of reloc_secs).
>> >
>> > Hope that clears some things up.
>>
>> Ok, I'll try to explain myself and it is gonna be long. I'll try to
>> describe how we deal with dynrelas in klp today, how you use it in kpatch
>> (and this is the place where my knowledge can be wrong or obsolete), what
>> you propose and what I'd like to propose.
>>
>> 1. First let's look on what we have now.
>>
>> We have a patch module in which there is a section with all dynrelas
>> needed to be resolved (it was like this in kpatch some time ago and maybe
>> it is different now so just have a patience, I'll get to it). In the init
>> function of the module kpatch builds all the relevant info from dynrela
>> section. It goes through it, creates an array of klp_reloc for each object
>> and includes each dynrela record with an objname to the array of
>> klp_object with that objname. Later when we need to apply relocations for
>> patched object we just go through the list (array) of its dynrelas in
>> klp_object and call our arch-specific klp_write_module_reloc().
>>
>> Now this was probably changed in kpatch and you do not have one dynrela
>> section but one dynrela section for each patched function. Is that
>> correct? (and can you tell us what the reason for the change was? It might
>> be crucial because I might be missing something.). Which leads us to your
>> proposal...
>>
>> 2. So we have several dynrela section for one patched object. During init
>> function in a patch module kpatch once again builds needed structures from
>> these sections. Now they are called klp_reloc_sec and contain different
>> kind of info. There is no val, loc and such, only name of the symbol,
>> objname and index to dynrela section in ELF. So when you need to write
>> relocations for the patched object you go through all relevant dynrela
>> sections (because they are stored in the klp_object), all dynrela records
>> in each section and you resolve the undefined symbols. All needed info is
>> stored in ELF. Then you just call apply_relocate_add().
>>
>> 3. I propose to go one step further. I think we don't need klp_reloc_sec
>> if there is only one dynrela section for patched object (and I currently
>> cannot see why this is not possible. It is possible even with one dynrela
>> section for whole patch module, that is for all patched objects.).
>
>I think I agree that we don't need klp_reloc_sec, and that klp rela
>sections can presumably be discovered by iterating over the sections.
>
>But I don't think it matters much whether we have one klp rela section
>per object, or multiple rela sections per object.  Either way, can't we
>find them when we iterate over the sections?
>
>For example, for one rela section per object, it could be named:
>
>__klp_rela.objname
>
>Or for multiple rela sections per object, they could be named:
>
>__klp_rela.objname.func1
>__klp_rela.objname.func2
>
>Either way, when iterating over the sections, we could just look for
>"__klp_rela.objname".
>
>All that said, I think I would vote for one rela section per object,
>just because it seems simpler.

Looking into this more, I think we do need one __klp_rela section per
function being patched.  Each rela section is linked to the section to
which the relocations apply via the rela section's sh_info field. In
SHT_RELA sections, the sh_info field contains the section index to
which the relocs apply. We cannot have one single combined rela
section per object as the call to apply_relocate_add() simply won't
work, because we would have relocs that apply to different functions
(and hence different sections).

So I guess instead of a single field in klp_object specifying the
__klp_rela section index, we could probably just have an array of
section indices.

>>
>> In my proposal there would be nothing done in init function of the patched
>> module (albeit some optimization mentioned later). When you call
>> klp_write_object_relocations you would go through all ELF section and find
>> the relevant one for the object (it is marked with SHF_RELA_LIVEPATCH and
>> objname is in the name of the section. It is the same thing you do in 2.
>> in the init function.). Then you go through all dynrela records in the
>> section, you do the same things which you do in the proposed patch above,
>> and call apply_relocate_add.
>>
>> Now it would be crazy to go through all sections each time
>> klp_write_object_relocations is called (maybe it does not even matter but
>
>Why would that be crazy?  To me it seems perfectly logical :-)  It
>doesn't seem like a very expensive operation to me.
>
>> nevertheless). So klp_object could have an index to its ELF dynrela
>> section. This index can be retrieved in the init function the same way you
>> do all the stuff with klp_reloc_sec.
>>
>> If you insisted on more than one dynrela section for a patched object we
>> could have an array of indices there. Or whatever.
>>
>> It is almost the same as your proposal but in my opinion somewhat nicer.
>> We just use the info stored in ELF directly without unnecessary middle
>> layer (== klp_reloc_sec).
>>
>> Does it make sense? I hope it does. Would it work?
>
>-- 
>Josh

Re: livepatch: reuse module loader code to write relocations

[Jessica Yu]

+++ Josh Poimboeuf [12/11/15 11:40 -0600]:
>On Thu, Nov 12, 2015 at 04:27:01PM +0100, Miroslav Benes wrote:
>> On Wed, 11 Nov 2015, Jessica Yu wrote:
>>
>> > +++ Miroslav Benes [11/11/15 15:30 +0100]:
>> > > On Mon, 9 Nov 2015, Jessica Yu wrote:
>> > >
>> > > So I guess we don't need klp_reloc anymore.
>> >
>> > Yes, that's correct. I am noticing just now that I forgot to remove
>> > the klp_reloc struct definition from livepatch.h. That change will be
>> > reflected in v2...
>> >
>> > > If true, we should really
>> > > start thinking about proper documentation because there are going to be
>> > > plenty of assumptions about a patch module and we need to have it written
>> > > somewhere. Especially how the relocation sections look like.
>> >
>> > Agreed. As a first step the patch module format can perhaps be
>> > documented somewhere. Perhaps it's time we create
>> > Documentation/livepatch/? :-)
>>
>> Yes, I think so.
>>
>> > > > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>> > > > index 087a8c7..26c419f 100644
>> > > > --- a/kernel/livepatch/core.c
>> > > > +++ b/kernel/livepatch/core.c
>> > > > @@ -28,6 +28,8 @@
>> > > >  #include <linux/list.h>
>> > > >  #include <linux/kallsyms.h>
>> > > >  #include <linux/livepatch.h>
>> > > > +#include <linux/elf.h>
>> > > > +#include <asm/cacheflush.h>
>> > > >
>> > > >  /**
>> > > >   * struct klp_ops - structure for tracking registered ftrace ops structs
>> > > > @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module
>> > > > *pmod, const char *name,
>> > > >  }
>> > > >
>> > > >  static int klp_write_object_relocations(struct module *pmod,
>> > > > -					struct klp_object *obj)
>> > > > +					struct klp_object *obj,
>> > > > +					struct klp_patch *patch)
>> > > >  {
>> > > > -	int ret;
>> > > > -	struct klp_reloc *reloc;
>> > > > +	int relindex, num_relas;
>> > > > +	int i, ret = 0;
>> > > > +	unsigned long addr;
>> > > > +	unsigned int bind;
>> > > > +	char *symname;
>> > > > +	struct klp_reloc_sec *reloc_sec;
>> > > > +	struct load_info *info;
>> > > > +	Elf_Rela *rela;
>> > > > +	Elf_Sym *sym, *symtab;
>> > > > +	Elf_Shdr *symsect;
>> > > >
>> > > >  	if (WARN_ON(!klp_is_object_loaded(obj)))
>> > > >  		return -EINVAL;
>> > > >
>> > > > -	if (WARN_ON(!obj->relocs))
>> > > > -		return -EINVAL;
>> > > > -
>> > > > -	for (reloc = obj->relocs; reloc->name; reloc++) {
>> > > > -		if (!klp_is_module(obj)) {
>> > > > -			ret = klp_verify_vmlinux_symbol(reloc->name,
>> > > > -							reloc->val);
>> > > > -			if (ret)
>> > > > -				return ret;
>> > > > -		} else {
>> > > > -			/* module, reloc->val needs to be discovered */
>> > > > -			if (reloc->external)
>> > > > -				ret = klp_find_external_symbol(pmod,
>> > > > -							       reloc->name,
>> > > > -							       &reloc->val);
>> > > > -			else
>> > > > -				ret = klp_find_object_symbol(obj->mod->name,
>> > > > -							     reloc->name,
>> > > > -							     &reloc->val);
>> > > > -			if (ret)
>> > > > -				return ret;
>> > > > -		}
>> > > > -		ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc,
>> > > > -					     reloc->val + reloc->addend);
>> > > > -		if (ret) {
>> > > > -			pr_err("relocation failed for symbol '%s' at 0x%016lx
>> > > > (%d)\n",
>> > > > -			       reloc->name, reloc->val, ret);
>> > > > -			return ret;
>> > > > +	info = pmod->info;
>> > > > +	symsect = info->sechdrs + info->index.sym;
>> > > > +	symtab = (void *)info->hdr + symsect->sh_offset;
>> > > > +
>> > > > +	/* For each __klp_rela section for this object */
>> > > > +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
>> > > > +		relindex = reloc_sec->index;
>> > > > +		num_relas = info->sechdrs[relindex].sh_size /
>> > > > sizeof(Elf_Rela);
>> > > > +		rela = (Elf_Rela *) info->sechdrs[relindex].sh_addr;
>> > > > +
>> > > > +		/* For each rela in this __klp_rela section */
>> > > > +		for (i = 0; i < num_relas; i++, rela++) {
>> > > > +			sym = symtab + ELF_R_SYM(rela->r_info);
>> > > > +			symname = info->strtab + sym->st_name;
>> > > > +			bind = ELF_ST_BIND(sym->st_info);
>> > > > +
>> > > > +			if (sym->st_shndx == SHN_LIVEPATCH) {
>> > > > +				if (bind == STB_LIVEPATCH_EXT)
>> > > > +					ret = klp_find_external_symbol(pmod,
>> > > > symname, &addr);
>> > > > +				else
>> > > > +					ret =
>> > > > klp_find_object_symbol(obj->name, symname, &addr);
>> > > > +				if (ret)
>> > > > +					return ret;
>> > > > +				sym->st_value = addr;
>> > > > +			}
>> > > >  		}
>> > > > +		ret = apply_relocate_add(info->sechdrs, info->strtab,
>> > > > +					 info->index.sym, relindex, pmod);
>> > > >  	}
>> > > >
>> > > > -	return 0;
>> > > > +	return ret;
>> > > >  }
>> > >
>> > > Looking at this... do we even need reloc_secs in klp_object? Question is
>> > > whether we need more than one dynrela section for an object. If not then
>> > > the binding between klp_reloc_sec and an object is the only relevant thing
>> > > in the structure, be it index or objname. So we can replace the
>> > > list of structures with just the index in klp_object, or get rid of it
>> > > completely and rely on the name of dynrela section be something like
>> > > __klp_rela_{objname}.
>> >
>> > Hm, you bring up a good point. I think theoretically yes, it is
>> > possible to just have one klp_reloc_sec for each object and therefore
>> > a list is not required (I have not checked yet how difficult it would
>> > be to implement this on the kpatch-build side of things).  However,
>> > considering the final format of the patch module, I think it is
>> > semantically clearer to leave it as a list, and for each object to
>> > possibly have more than one __klp_rela section.
>> >
>> > For example, say we are patching two functions in ext4. In my
>> > resulting kpatch module I will have two __klp_rela_ext4 sections, and
>> > they might look like this when we run readelf --sections:
>> >
>> > [34] __klp_rela_ext4.text.ext4_attr_store RELA ...
>> > [35] __klp_rela_ext4.text.ext4_attr_show RELA ...
>> >
>> > Then these two klp rela sections end up as two elements in the
>> > reloc_secs list for the ext4 patch object. I think this way, we can
>> > better tell which rela is being applied to what function. Might be
>> > easier to understand what's happening from the developer's point of
>> > view.
>> >
>> > > You see, we go through elf sections here which were preserved by module
>> > > loader. We even have relevant sections marked with SHF_RELA_LIVEPATCH. So
>> > > maybe all the stuff around klp_reloc_sec is not necessary.
>> > >
>> > > Thoughts?
>> >
>> > Ah, so this is where descriptive comments and documentation might have
>> > been useful :-) So I think we will still need to keep the
>> > klp_reloc_sec struct to help the patch module initialize. Though the
>> > name and objname fields aren't used in this patchset, they are used in
>> > the kpatch patch module code [1], where we iterate through each elf
>> > section, find the ones marked with SHF_RELA_LIVEPATCH, set the
>> > klp_reloc_sec's objname (which we find from the "name" field,
>> > formatted as __klp_rela_{objname}.text..). Once we have the objname
>> > set, we can then find the object to attach the reloc_sec to (i.e. add
>> > it to its list of reloc_secs).
>> >
>> > Hope that clears some things up.
>>
>> Ok, I'll try to explain myself and it is gonna be long. I'll try to
>> describe how we deal with dynrelas in klp today, how you use it in kpatch
>> (and this is the place where my knowledge can be wrong or obsolete), what
>> you propose and what I'd like to propose.
>>
>> 1. First let's look on what we have now.
>>
>> We have a patch module in which there is a section with all dynrelas
>> needed to be resolved (it was like this in kpatch some time ago and maybe
>> it is different now so just have a patience, I'll get to it). In the init
>> function of the module kpatch builds all the relevant info from dynrela
>> section. It goes through it, creates an array of klp_reloc for each object
>> and includes each dynrela record with an objname to the array of
>> klp_object with that objname. Later when we need to apply relocations for
>> patched object we just go through the list (array) of its dynrelas in
>> klp_object and call our arch-specific klp_write_module_reloc().
>>
>> Now this was probably changed in kpatch and you do not have one dynrela
>> section but one dynrela section for each patched function. Is that
>> correct? (and can you tell us what the reason for the change was? It might
>> be crucial because I might be missing something.). Which leads us to your
>> proposal...
>>
>> 2. So we have several dynrela section for one patched object. During init
>> function in a patch module kpatch once again builds needed structures from
>> these sections. Now they are called klp_reloc_sec and contain different
>> kind of info. There is no val, loc and such, only name of the symbol,
>> objname and index to dynrela section in ELF. So when you need to write
>> relocations for the patched object you go through all relevant dynrela
>> sections (because they are stored in the klp_object), all dynrela records
>> in each section and you resolve the undefined symbols. All needed info is
>> stored in ELF. Then you just call apply_relocate_add().
>>
>> 3. I propose to go one step further. I think we don't need klp_reloc_sec
>> if there is only one dynrela section for patched object (and I currently
>> cannot see why this is not possible. It is possible even with one dynrela
>> section for whole patch module, that is for all patched objects.).
>
>I think I agree that we don't need klp_reloc_sec, and that klp rela
>sections can presumably be discovered by iterating over the sections.
>
>But I don't think it matters much whether we have one klp rela section
>per object, or multiple rela sections per object.  Either way, can't we
>find them when we iterate over the sections?
>
>For example, for one rela section per object, it could be named:
>
>__klp_rela.objname
>
>Or for multiple rela sections per object, they could be named:
>
>__klp_rela.objname.func1
>__klp_rela.objname.func2
>
>Either way, when iterating over the sections, we could just look for
>"__klp_rela.objname".
>
>All that said, I think I would vote for one rela section per object,
>just because it seems simpler.

Looking into this more, I think we do need one __klp_rela section per
function being patched.  Each rela section is linked to the section to
which the relocations apply via the rela section's sh_info field. In
SHT_RELA sections, the sh_info field contains the section index to
which the relocs apply. We cannot have one single combined rela
section per object as the call to apply_relocate_add() simply won't
work, because we would have relocs that apply to different functions
(and hence different sections).

So I guess instead of a single field in klp_object specifying the
__klp_rela section index, we could probably just have an array of
section indices.

>>
>> In my proposal there would be nothing done in init function of the patched
>> module (albeit some optimization mentioned later). When you call
>> klp_write_object_relocations you would go through all ELF section and find
>> the relevant one for the object (it is marked with SHF_RELA_LIVEPATCH and
>> objname is in the name of the section. It is the same thing you do in 2.
>> in the init function.). Then you go through all dynrela records in the
>> section, you do the same things which you do in the proposed patch above,
>> and call apply_relocate_add.
>>
>> Now it would be crazy to go through all sections each time
>> klp_write_object_relocations is called (maybe it does not even matter but
>
>Why would that be crazy?  To me it seems perfectly logical :-)  It
>doesn't seem like a very expensive operation to me.
>
>> nevertheless). So klp_object could have an index to its ELF dynrela
>> section. This index can be retrieved in the init function the same way you
>> do all the stuff with klp_reloc_sec.
>>
>> If you insisted on more than one dynrela section for a patched object we
>> could have an array of indices there. Or whatever.
>>
>> It is almost the same as your proposal but in my opinion somewhat nicer.
>> We just use the info stored in ELF directly without unnecessary middle
>> layer (== klp_reloc_sec).
>>
>> Does it make sense? I hope it does. Would it work?
>
>-- 
>Josh

Re: livepatch: reuse module loader code to write relocations

[Josh Poimboeuf]

On Thu, Nov 12, 2015 at 03:22:44PM -0500, Jessica Yu wrote:
> Looking into this more, I think we do need one __klp_rela section per
> function being patched.  Each rela section is linked to the section to
> which the relocations apply via the rela section's sh_info field. In
> SHT_RELA sections, the sh_info field contains the section index to
> which the relocs apply. We cannot have one single combined rela
> section per object as the call to apply_relocate_add() simply won't
> work, because we would have relocs that apply to different functions
> (and hence different sections).
> 
> So I guess instead of a single field in klp_object specifying the
> __klp_rela section index, we could probably just have an array of
> section indices.

Ok, makes sense, sounds like we need multiple klp relas per object.

I still don't quite understand the benefit of caching the klp_rela
section indices.  What problem does it solve?  It seems simpler to just
iterate over all the sections in klp_write_object_relocations().

-- 
Josh

Re: livepatch: reuse module loader code to write relocations

[Josh Poimboeuf]

On Thu, Nov 12, 2015 at 03:22:44PM -0500, Jessica Yu wrote:
> Looking into this more, I think we do need one __klp_rela section per
> function being patched.  Each rela section is linked to the section to
> which the relocations apply via the rela section's sh_info field. In
> SHT_RELA sections, the sh_info field contains the section index to
> which the relocs apply. We cannot have one single combined rela
> section per object as the call to apply_relocate_add() simply won't
> work, because we would have relocs that apply to different functions
> (and hence different sections).
> 
> So I guess instead of a single field in klp_object specifying the
> __klp_rela section index, we could probably just have an array of
> section indices.

Ok, makes sense, sounds like we need multiple klp relas per object.

I still don't quite understand the benefit of caching the klp_rela
section indices.  What problem does it solve?  It seems simpler to just
iterate over all the sections in klp_write_object_relocations().

-- 
Josh

Re: livepatch: reuse module loader code to write relocations

[Jessica Yu]

+++ Jessica Yu [12/11/15 14:14 -0500]:
>+++ Miroslav Benes [12/11/15 16:27 +0100]:
>>On Wed, 11 Nov 2015, Jessica Yu wrote:
>>
>>>+++ Miroslav Benes [11/11/15 15:30 +0100]:
>>>> On Mon, 9 Nov 2015, Jessica Yu wrote:
>>>>
>>>> So I guess we don't need klp_reloc anymore.
>>>
>>>Yes, that's correct. I am noticing just now that I forgot to remove
>>>the klp_reloc struct definition from livepatch.h. That change will be
>>>reflected in v2...
>>>
>>>> If true, we should really
>>>> start thinking about proper documentation because there are going to be
>>>> plenty of assumptions about a patch module and we need to have it written
>>>> somewhere. Especially how the relocation sections look like.
>>>
>>>Agreed. As a first step the patch module format can perhaps be
>>>documented somewhere. Perhaps it's time we create
>>>Documentation/livepatch/? :-)
>>
>>Yes, I think so.
>>
>>>> > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>>>> > index 087a8c7..26c419f 100644
>>>> > --- a/kernel/livepatch/core.c
>>>> > +++ b/kernel/livepatch/core.c
>>>> > @@ -28,6 +28,8 @@
>>>> >  #include <linux/list.h>
>>>> >  #include <linux/kallsyms.h>
>>>> >  #include <linux/livepatch.h>
>>>> > +#include <linux/elf.h>
>>>> > +#include <asm/cacheflush.h>
>>>> >
>>>> >  /**
>>>> >   * struct klp_ops - structure for tracking registered ftrace ops structs
>>>> > @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module
>>>> > *pmod, const char *name,
>>>> >  }
>>>> >
>>>> >  static int klp_write_object_relocations(struct module *pmod,
>>>> > -					struct klp_object *obj)
>>>> > +					struct klp_object *obj,
>>>> > +					struct klp_patch *patch)
>>>> >  {
>>>> > -	int ret;
>>>> > -	struct klp_reloc *reloc;
>>>> > +	int relindex, num_relas;
>>>> > +	int i, ret = 0;
>>>> > +	unsigned long addr;
>>>> > +	unsigned int bind;
>>>> > +	char *symname;
>>>> > +	struct klp_reloc_sec *reloc_sec;
>>>> > +	struct load_info *info;
>>>> > +	Elf_Rela *rela;
>>>> > +	Elf_Sym *sym, *symtab;
>>>> > +	Elf_Shdr *symsect;
>>>> >
>>>> >  	if (WARN_ON(!klp_is_object_loaded(obj)))
>>>> >  		return -EINVAL;
>>>> >
>>>> > -	if (WARN_ON(!obj->relocs))
>>>> > -		return -EINVAL;
>>>> > -
>>>> > -	for (reloc = obj->relocs; reloc->name; reloc++) {
>>>> > -		if (!klp_is_module(obj)) {
>>>> > -			ret = klp_verify_vmlinux_symbol(reloc->name,
>>>> > -							reloc->val);
>>>> > -			if (ret)
>>>> > -				return ret;
>>>> > -		} else {
>>>> > -			/* module, reloc->val needs to be discovered */
>>>> > -			if (reloc->external)
>>>> > -				ret = klp_find_external_symbol(pmod,
>>>> > -							       reloc->name,
>>>> > -							       &reloc->val);
>>>> > -			else
>>>> > -				ret = klp_find_object_symbol(obj->mod->name,
>>>> > -							     reloc->name,
>>>> > -							     &reloc->val);
>>>> > -			if (ret)
>>>> > -				return ret;
>>>> > -		}
>>>> > -		ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc,
>>>> > -					     reloc->val + reloc->addend);
>>>> > -		if (ret) {
>>>> > -			pr_err("relocation failed for symbol '%s' at 0x%016lx
>>>> > (%d)\n",
>>>> > -			       reloc->name, reloc->val, ret);
>>>> > -			return ret;
>>>> > +	info = pmod->info;
>>>> > +	symsect = info->sechdrs + info->index.sym;
>>>> > +	symtab = (void *)info->hdr + symsect->sh_offset;
>>>> > +
>>>> > +	/* For each __klp_rela section for this object */
>>>> > +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
>>>> > +		relindex = reloc_sec->index;
>>>> > +		num_relas = info->sechdrs[relindex].sh_size /
>>>> > sizeof(Elf_Rela);
>>>> > +		rela = (Elf_Rela *) info->sechdrs[relindex].sh_addr;
>>>> > +
>>>> > +		/* For each rela in this __klp_rela section */
>>>> > +		for (i = 0; i < num_relas; i++, rela++) {
>>>> > +			sym = symtab + ELF_R_SYM(rela->r_info);
>>>> > +			symname = info->strtab + sym->st_name;
>>>> > +			bind = ELF_ST_BIND(sym->st_info);
>>>> > +
>>>> > +			if (sym->st_shndx == SHN_LIVEPATCH) {
>>>> > +				if (bind == STB_LIVEPATCH_EXT)
>>>> > +					ret = klp_find_external_symbol(pmod,
>>>> > symname, &addr);
>>>> > +				else
>>>> > +					ret =
>>>> > klp_find_object_symbol(obj->name, symname, &addr);
>>>> > +				if (ret)
>>>> > +					return ret;
>>>> > +				sym->st_value = addr;
>>>> > +			}
>>>> >  		}
>>>> > +		ret = apply_relocate_add(info->sechdrs, info->strtab,
>>>> > +					 info->index.sym, relindex, pmod);
>>>> >  	}
>>>> >
>>>> > -	return 0;
>>>> > +	return ret;
>>>> >  }
>>>>
>>>> Looking at this... do we even need reloc_secs in klp_object? Question is
>>>> whether we need more than one dynrela section for an object. If not then
>>>> the binding between klp_reloc_sec and an object is the only relevant thing
>>>> in the structure, be it index or objname. So we can replace the
>>>> list of structures with just the index in klp_object, or get rid of it
>>>> completely and rely on the name of dynrela section be something like
>>>> __klp_rela_{objname}.
>>>
>>>Hm, you bring up a good point. I think theoretically yes, it is
>>>possible to just have one klp_reloc_sec for each object and therefore
>>>a list is not required (I have not checked yet how difficult it would
>>>be to implement this on the kpatch-build side of things).  However,
>>>considering the final format of the patch module, I think it is
>>>semantically clearer to leave it as a list, and for each object to
>>>possibly have more than one __klp_rela section.
>>>
>>>For example, say we are patching two functions in ext4. In my
>>>resulting kpatch module I will have two __klp_rela_ext4 sections, and
>>>they might look like this when we run readelf --sections:
>>>
>>>[34] __klp_rela_ext4.text.ext4_attr_store RELA ...
>>>[35] __klp_rela_ext4.text.ext4_attr_show RELA ...
>>>
>>>Then these two klp rela sections end up as two elements in the
>>>reloc_secs list for the ext4 patch object. I think this way, we can
>>>better tell which rela is being applied to what function. Might be
>>>easier to understand what's happening from the developer's point of
>>>view.
>>>
>>>> You see, we go through elf sections here which were preserved by module
>>>> loader. We even have relevant sections marked with SHF_RELA_LIVEPATCH. So
>>>> maybe all the stuff around klp_reloc_sec is not necessary.
>>>>
>>>> Thoughts?
>>>
>>>Ah, so this is where descriptive comments and documentation might have
>>>been useful :-) So I think we will still need to keep the
>>>klp_reloc_sec struct to help the patch module initialize. Though the
>>>name and objname fields aren't used in this patchset, they are used in
>>>the kpatch patch module code [1], where we iterate through each elf
>>>section, find the ones marked with SHF_RELA_LIVEPATCH, set the
>>>klp_reloc_sec's objname (which we find from the "name" field,
>>>formatted as __klp_rela_{objname}.text..). Once we have the objname
>>>set, we can then find the object to attach the reloc_sec to (i.e. add
>>>it to its list of reloc_secs).
>>>
>>>Hope that clears some things up.
>>
>>Ok, I'll try to explain myself and it is gonna be long. I'll try to
>>describe how we deal with dynrelas in klp today, how you use it in kpatch
>>(and this is the place where my knowledge can be wrong or obsolete), what
>>you propose and what I'd like to propose.
>>
>>1. First let's look on what we have now.
>>
>>We have a patch module in which there is a section with all dynrelas
>>needed to be resolved (it was like this in kpatch some time ago and maybe
>>it is different now so just have a patience, I'll get to it). In the init
>>function of the module kpatch builds all the relevant info from dynrela
>>section. It goes through it, creates an array of klp_reloc for each object
>>and includes each dynrela record with an objname to the array of
>>klp_object with that objname. Later when we need to apply relocations for
>>patched object we just go through the list (array) of its dynrelas in
>>klp_object and call our arch-specific klp_write_module_reloc().
>
>Sounds correct to me.
>
>>Now this was probably changed in kpatch and you do not have one dynrela
>>section but one dynrela section for each patched function. Is that
>>correct? (and can you tell us what the reason for the change was? It might
>>be crucial because I might be missing something.). Which leads us to your
>>proposal...
>
>Your original assumption was correct; current kpatch has one big
>.kpatch.dynrelas section, and each dynrela entry within that single
>section gets sorted to the correct object as you described above. The
>one dynrela section per patched function only came with this patchset,
>but for no particular reason other than to make the kpatch-build code
>for generating the patch module slightly less complicated. But I
>haven't checked how big of a change it would be to do
>one-dynrela-section per object, perhaps (and I hope) it will be an
>easy change.
>
>>2. So we have several dynrela section for one patched object. During init
>>function in a patch module kpatch once again builds needed structures from
>>these sections. Now they are called klp_reloc_sec and contain different
>>kind of info. There is no val, loc and such, only name of the symbol,
>>objname and index to dynrela section in ELF. So when you need to write
>>relocations for the patched object you go through all relevant dynrela
>>sections (because they are stored in the klp_object), all dynrela records
>>in each section and you resolve the undefined symbols. All needed info is
>>stored in ELF. Then you just call apply_relocate_add().
>>
>>3. I propose to go one step further. I think we don't need klp_reloc_sec
>>if there is only one dynrela section for patched object (and I currently
>>cannot see why this is not possible. It is possible even with one dynrela
>>section for whole patch module, that is for all patched objects.).
>
>I think the furthest we can go in terms of simplifying klp rela secs
>is to have one __klp_rela section per object. We can't smush all the
>__klp_rela_objname sections into one big __klp_rela section since we
>could be patching some objects that won't be loaded yet, and
>apply_relocate_add() needs to work with real SHT_RELA sections + their
>corresponding section indices (i.e., we cannot call
>apply_relocate_add() with that single, combined klp rela section).
>
>So, I think I would be OK with one __klp_rela section per object.

I just found a problem with this one __klp_rela section per object
approach, so I have to retract what I said above. I no longer think it
works, and the reason has to do with how apply_relocate_add() uses the
sh_info field of each relocation section to figure out to which
section the reloc section applies. Each relocation section references
the section to which it modifies, so this is a 1-1 correspondance we
cannot break. Thus we cannot simply combine all relocations into a
single __klp_rela section, because they would apply to multiple
different sections. See my reply to Josh.

This isn't a big problem anyway, we can just use an array of section
indices and still get rid of klp_reloc_sec. Plus the current patchset
already expects/implements multiple __klp_rela sections per object.

>>In my proposal there would be nothing done in init function of the patched
>>module (albeit some optimization mentioned later). When you call
>>klp_write_object_relocations you would go through all ELF section and find
>>the relevant one for the object (it is marked with SHF_RELA_LIVEPATCH and
>>objname is in the name of the section. It is the same thing you do in 2.
>>in the init function.). Then you go through all dynrela records in the
>>section, you do the same things which you do in the proposed patch above,
>>and call apply_relocate_add.
>
>I'm not sure I like having klp_write_object_relocations() repeatedly
>perform a loop through all the elf sections when we can pre-process
>this information in the patch module's init function, and "cache" the
>relevant klp section indices before passing things off to
>klp_write_object_relocations(). So how about this: we do the
>__klp_rela sec sorting in the init function of the patched module. The
>patch module would iterate through its own elf sections, and when it
>encounters a __klp_rela section, it looks at its objname, find the
>corresponding klp_object, and save the section index of the __klp_rela
>section in that klp_object. Then in klp_write_object_relocations, we
>just have to look at the saved section index for the corresponding
>object and access the klp rela section through that index, do the same
>processing in this patch and call apply_relocate_add().

s/index/indices/, so instead of saving a single section index, just
keep track of all the indices of the __klp_rela_objname sections in
each klp_object.

>>Now it would be crazy to go through all sections each time
>>klp_write_object_relocations is called (maybe it does not even matter but
>>nevertheless). So klp_object could have an index to its ELF dynrela
>>section. This index can be retrieved in the init function the same way you
>>do all the stuff with klp_reloc_sec.
>
>Ah, exactly what I said above :-D
>
>>If you insisted on more than one dynrela section for a patched object we
>>could have an array of indices there. Or whatever.
>>
>>It is almost the same as your proposal but in my opinion somewhat nicer.
>>We just use the info stored in ELF directly without unnecessary middle
>>layer (== klp_reloc_sec).
>>
>>Does it make sense? I hope it does. Would it work?
>
>It does make sense, and I think we can make it work without
>klp_reloc_sec. Thanks for the explanations.

Re: livepatch: reuse module loader code to write relocations

[Jessica Yu]

+++ Jessica Yu [12/11/15 14:14 -0500]:
>+++ Miroslav Benes [12/11/15 16:27 +0100]:
>>On Wed, 11 Nov 2015, Jessica Yu wrote:
>>
>>>+++ Miroslav Benes [11/11/15 15:30 +0100]:
>>>> On Mon, 9 Nov 2015, Jessica Yu wrote:
>>>>
>>>> So I guess we don't need klp_reloc anymore.
>>>
>>>Yes, that's correct. I am noticing just now that I forgot to remove
>>>the klp_reloc struct definition from livepatch.h. That change will be
>>>reflected in v2...
>>>
>>>> If true, we should really
>>>> start thinking about proper documentation because there are going to be
>>>> plenty of assumptions about a patch module and we need to have it written
>>>> somewhere. Especially how the relocation sections look like.
>>>
>>>Agreed. As a first step the patch module format can perhaps be
>>>documented somewhere. Perhaps it's time we create
>>>Documentation/livepatch/? :-)
>>
>>Yes, I think so.
>>
>>>> > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>>>> > index 087a8c7..26c419f 100644
>>>> > --- a/kernel/livepatch/core.c
>>>> > +++ b/kernel/livepatch/core.c
>>>> > @@ -28,6 +28,8 @@
>>>> >  #include <linux/list.h>
>>>> >  #include <linux/kallsyms.h>
>>>> >  #include <linux/livepatch.h>
>>>> > +#include <linux/elf.h>
>>>> > +#include <asm/cacheflush.h>
>>>> >
>>>> >  /**
>>>> >   * struct klp_ops - structure for tracking registered ftrace ops structs
>>>> > @@ -281,46 +283,54 @@ static int klp_find_external_symbol(struct module
>>>> > *pmod, const char *name,
>>>> >  }
>>>> >
>>>> >  static int klp_write_object_relocations(struct module *pmod,
>>>> > -					struct klp_object *obj)
>>>> > +					struct klp_object *obj,
>>>> > +					struct klp_patch *patch)
>>>> >  {
>>>> > -	int ret;
>>>> > -	struct klp_reloc *reloc;
>>>> > +	int relindex, num_relas;
>>>> > +	int i, ret = 0;
>>>> > +	unsigned long addr;
>>>> > +	unsigned int bind;
>>>> > +	char *symname;
>>>> > +	struct klp_reloc_sec *reloc_sec;
>>>> > +	struct load_info *info;
>>>> > +	Elf_Rela *rela;
>>>> > +	Elf_Sym *sym, *symtab;
>>>> > +	Elf_Shdr *symsect;
>>>> >
>>>> >  	if (WARN_ON(!klp_is_object_loaded(obj)))
>>>> >  		return -EINVAL;
>>>> >
>>>> > -	if (WARN_ON(!obj->relocs))
>>>> > -		return -EINVAL;
>>>> > -
>>>> > -	for (reloc = obj->relocs; reloc->name; reloc++) {
>>>> > -		if (!klp_is_module(obj)) {
>>>> > -			ret = klp_verify_vmlinux_symbol(reloc->name,
>>>> > -							reloc->val);
>>>> > -			if (ret)
>>>> > -				return ret;
>>>> > -		} else {
>>>> > -			/* module, reloc->val needs to be discovered */
>>>> > -			if (reloc->external)
>>>> > -				ret = klp_find_external_symbol(pmod,
>>>> > -							       reloc->name,
>>>> > -							       &reloc->val);
>>>> > -			else
>>>> > -				ret = klp_find_object_symbol(obj->mod->name,
>>>> > -							     reloc->name,
>>>> > -							     &reloc->val);
>>>> > -			if (ret)
>>>> > -				return ret;
>>>> > -		}
>>>> > -		ret = klp_write_module_reloc(pmod, reloc->type, reloc->loc,
>>>> > -					     reloc->val + reloc->addend);
>>>> > -		if (ret) {
>>>> > -			pr_err("relocation failed for symbol '%s' at 0x%016lx
>>>> > (%d)\n",
>>>> > -			       reloc->name, reloc->val, ret);
>>>> > -			return ret;
>>>> > +	info = pmod->info;
>>>> > +	symsect = info->sechdrs + info->index.sym;
>>>> > +	symtab = (void *)info->hdr + symsect->sh_offset;
>>>> > +
>>>> > +	/* For each __klp_rela section for this object */
>>>> > +	list_for_each_entry(reloc_sec, &obj->reloc_secs, list) {
>>>> > +		relindex = reloc_sec->index;
>>>> > +		num_relas = info->sechdrs[relindex].sh_size /
>>>> > sizeof(Elf_Rela);
>>>> > +		rela = (Elf_Rela *) info->sechdrs[relindex].sh_addr;
>>>> > +
>>>> > +		/* For each rela in this __klp_rela section */
>>>> > +		for (i = 0; i < num_relas; i++, rela++) {
>>>> > +			sym = symtab + ELF_R_SYM(rela->r_info);
>>>> > +			symname = info->strtab + sym->st_name;
>>>> > +			bind = ELF_ST_BIND(sym->st_info);
>>>> > +
>>>> > +			if (sym->st_shndx == SHN_LIVEPATCH) {
>>>> > +				if (bind == STB_LIVEPATCH_EXT)
>>>> > +					ret = klp_find_external_symbol(pmod,
>>>> > symname, &addr);
>>>> > +				else
>>>> > +					ret =
>>>> > klp_find_object_symbol(obj->name, symname, &addr);
>>>> > +				if (ret)
>>>> > +					return ret;
>>>> > +				sym->st_value = addr;
>>>> > +			}
>>>> >  		}
>>>> > +		ret = apply_relocate_add(info->sechdrs, info->strtab,
>>>> > +					 info->index.sym, relindex, pmod);
>>>> >  	}
>>>> >
>>>> > -	return 0;
>>>> > +	return ret;
>>>> >  }
>>>>
>>>> Looking at this... do we even need reloc_secs in klp_object? Question is
>>>> whether we need more than one dynrela section for an object. If not then
>>>> the binding between klp_reloc_sec and an object is the only relevant thing
>>>> in the structure, be it index or objname. So we can replace the
>>>> list of structures with just the index in klp_object, or get rid of it
>>>> completely and rely on the name of dynrela section be something like
>>>> __klp_rela_{objname}.
>>>
>>>Hm, you bring up a good point. I think theoretically yes, it is
>>>possible to just have one klp_reloc_sec for each object and therefore
>>>a list is not required (I have not checked yet how difficult it would
>>>be to implement this on the kpatch-build side of things).  However,
>>>considering the final format of the patch module, I think it is
>>>semantically clearer to leave it as a list, and for each object to
>>>possibly have more than one __klp_rela section.
>>>
>>>For example, say we are patching two functions in ext4. In my
>>>resulting kpatch module I will have two __klp_rela_ext4 sections, and
>>>they might look like this when we run readelf --sections:
>>>
>>>[34] __klp_rela_ext4.text.ext4_attr_store RELA ...
>>>[35] __klp_rela_ext4.text.ext4_attr_show RELA ...
>>>
>>>Then these two klp rela sections end up as two elements in the
>>>reloc_secs list for the ext4 patch object. I think this way, we can
>>>better tell which rela is being applied to what function. Might be
>>>easier to understand what's happening from the developer's point of
>>>view.
>>>
>>>> You see, we go through elf sections here which were preserved by module
>>>> loader. We even have relevant sections marked with SHF_RELA_LIVEPATCH. So
>>>> maybe all the stuff around klp_reloc_sec is not necessary.
>>>>
>>>> Thoughts?
>>>
>>>Ah, so this is where descriptive comments and documentation might have
>>>been useful :-) So I think we will still need to keep the
>>>klp_reloc_sec struct to help the patch module initialize. Though the
>>>name and objname fields aren't used in this patchset, they are used in
>>>the kpatch patch module code [1], where we iterate through each elf
>>>section, find the ones marked with SHF_RELA_LIVEPATCH, set the
>>>klp_reloc_sec's objname (which we find from the "name" field,
>>>formatted as __klp_rela_{objname}.text..). Once we have the objname
>>>set, we can then find the object to attach the reloc_sec to (i.e. add
>>>it to its list of reloc_secs).
>>>
>>>Hope that clears some things up.
>>
>>Ok, I'll try to explain myself and it is gonna be long. I'll try to
>>describe how we deal with dynrelas in klp today, how you use it in kpatch
>>(and this is the place where my knowledge can be wrong or obsolete), what
>>you propose and what I'd like to propose.
>>
>>1. First let's look on what we have now.
>>
>>We have a patch module in which there is a section with all dynrelas
>>needed to be resolved (it was like this in kpatch some time ago and maybe
>>it is different now so just have a patience, I'll get to it). In the init
>>function of the module kpatch builds all the relevant info from dynrela
>>section. It goes through it, creates an array of klp_reloc for each object
>>and includes each dynrela record with an objname to the array of
>>klp_object with that objname. Later when we need to apply relocations for
>>patched object we just go through the list (array) of its dynrelas in
>>klp_object and call our arch-specific klp_write_module_reloc().
>
>Sounds correct to me.
>
>>Now this was probably changed in kpatch and you do not have one dynrela
>>section but one dynrela section for each patched function. Is that
>>correct? (and can you tell us what the reason for the change was? It might
>>be crucial because I might be missing something.). Which leads us to your
>>proposal...
>
>Your original assumption was correct; current kpatch has one big
>.kpatch.dynrelas section, and each dynrela entry within that single
>section gets sorted to the correct object as you described above. The
>one dynrela section per patched function only came with this patchset,
>but for no particular reason other than to make the kpatch-build code
>for generating the patch module slightly less complicated. But I
>haven't checked how big of a change it would be to do
>one-dynrela-section per object, perhaps (and I hope) it will be an
>easy change.
>
>>2. So we have several dynrela section for one patched object. During init
>>function in a patch module kpatch once again builds needed structures from
>>these sections. Now they are called klp_reloc_sec and contain different
>>kind of info. There is no val, loc and such, only name of the symbol,
>>objname and index to dynrela section in ELF. So when you need to write
>>relocations for the patched object you go through all relevant dynrela
>>sections (because they are stored in the klp_object), all dynrela records
>>in each section and you resolve the undefined symbols. All needed info is
>>stored in ELF. Then you just call apply_relocate_add().
>>
>>3. I propose to go one step further. I think we don't need klp_reloc_sec
>>if there is only one dynrela section for patched object (and I currently
>>cannot see why this is not possible. It is possible even with one dynrela
>>section for whole patch module, that is for all patched objects.).
>
>I think the furthest we can go in terms of simplifying klp rela secs
>is to have one __klp_rela section per object. We can't smush all the
>__klp_rela_objname sections into one big __klp_rela section since we
>could be patching some objects that won't be loaded yet, and
>apply_relocate_add() needs to work with real SHT_RELA sections + their
>corresponding section indices (i.e., we cannot call
>apply_relocate_add() with that single, combined klp rela section).
>
>So, I think I would be OK with one __klp_rela section per object.

I just found a problem with this one __klp_rela section per object
approach, so I have to retract what I said above. I no longer think it
works, and the reason has to do with how apply_relocate_add() uses the
sh_info field of each relocation section to figure out to which
section the reloc section applies. Each relocation section references
the section to which it modifies, so this is a 1-1 correspondance we
cannot break. Thus we cannot simply combine all relocations into a
single __klp_rela section, because they would apply to multiple
different sections. See my reply to Josh.

This isn't a big problem anyway, we can just use an array of section
indices and still get rid of klp_reloc_sec. Plus the current patchset
already expects/implements multiple __klp_rela sections per object.

>>In my proposal there would be nothing done in init function of the patched
>>module (albeit some optimization mentioned later). When you call
>>klp_write_object_relocations you would go through all ELF section and find
>>the relevant one for the object (it is marked with SHF_RELA_LIVEPATCH and
>>objname is in the name of the section. It is the same thing you do in 2.
>>in the init function.). Then you go through all dynrela records in the
>>section, you do the same things which you do in the proposed patch above,
>>and call apply_relocate_add.
>
>I'm not sure I like having klp_write_object_relocations() repeatedly
>perform a loop through all the elf sections when we can pre-process
>this information in the patch module's init function, and "cache" the
>relevant klp section indices before passing things off to
>klp_write_object_relocations(). So how about this: we do the
>__klp_rela sec sorting in the init function of the patched module. The
>patch module would iterate through its own elf sections, and when it
>encounters a __klp_rela section, it looks at its objname, find the
>corresponding klp_object, and save the section index of the __klp_rela
>section in that klp_object. Then in klp_write_object_relocations, we
>just have to look at the saved section index for the corresponding
>object and access the klp rela section through that index, do the same
>processing in this patch and call apply_relocate_add().

s/index/indices/, so instead of saving a single section index, just
keep track of all the indices of the __klp_rela_objname sections in
each klp_object.

>>Now it would be crazy to go through all sections each time
>>klp_write_object_relocations is called (maybe it does not even matter but
>>nevertheless). So klp_object could have an index to its ELF dynrela
>>section. This index can be retrieved in the init function the same way you
>>do all the stuff with klp_reloc_sec.
>
>Ah, exactly what I said above :-D
>
>>If you insisted on more than one dynrela section for a patched object we
>>could have an array of indices there. Or whatever.
>>
>>It is almost the same as your proposal but in my opinion somewhat nicer.
>>We just use the info stored in ELF directly without unnecessary middle
>>layer (== klp_reloc_sec).
>>
>>Does it make sense? I hope it does. Would it work?
>
>It does make sense, and I think we can make it work without
>klp_reloc_sec. Thanks for the explanations.

Re: module: save load_info for livepatch modules

[Jessica Yu]

+++ Josh Poimboeuf [12/11/15 11:05 -0600]:
>On Thu, Nov 12, 2015 at 04:03:45PM +0100, Petr Mladek wrote:
>> On Thu 2015-11-12 14:22:28, Miroslav Benes wrote:
>> > On Thu, 12 Nov 2015, Petr Mladek wrote:
>> > > > >Maybe I am missing something but isn't it necessary to call vfree() on
>> > > > >info somewhere in the end?
>> > > >
>> > > > So free_copy() will call vfree(info->hdr), except in livepatch modules
>> > > > we want to keep all the elf section information stored there, so we
>> > > > avoid calling free_copy(), As for the info struct itself, if you look
>> > > > at the init_module and finit_module syscall definitions in
>> > > > kernel/module.c, you will see that info is actually a local function
>> > > > variable, simply passed in to the call to load_module(), and will be
>> > > > automatically deallocated when the syscall returns. :-) No need to
>> > > > explicitly free info.
>> > >
>> > > We still have to free the copied or preserved structures when
>> > > the module is unloaded.
>> >
>> > ...freeing what we allocated. We need to free info->hdr somewhere if not
>> > here and also mod_arch_specific struct where the patch module is removed.
>> > This would unfortunately lead to changes in arch-specific code in
>> > module.c. For example in arch/s390/kernel/module.c there is vfree call on
>> > part of mod_arch_specific in module_finalize. We would call it only if the
>> > flag mentioned above is not set and at the same time we would need to call
>> > it when the patch module is being removed.
>>
>> Sigh, I am afraid that the flag is not enough. IMHO, we need to split
>> the load finalizing functions into two pieces. One will be always
>> called when the module load is finalized. The other part will free
>> the load_info. It will be called either when the load is finalized or
>> when the module is unloaded, depending on if we want to preserve
>> the load_info.
>>
>> Sigh, it is getting complicated. But let's see how it looks in reality.
>
>At the other end of the spectrum, we could do the simplest thing
>possible: _always_ save the data (even if CONFIG_LIVEPATCH is disabled).
>
>(gdb) print sizeof(*info)
>$3 = 96
>(gdb) p sizeof(*info->hdr)
>$4 = 64
>s390 mod_arch_syminfo struct: 24 bytes by my reckoning.
>
>So between info, info->hdr, and s390 mod_arch_syminfo, we're talking
>about 184 bytes on s390 and 160 bytes on x86_64.  That seems like
>peanuts compared to the size of a typical module.  The benefit is that
>the code would be simpler because we don't have any special cases and
>the structs would automatically get freed with the module struct when
>the module gets unloaded.

I think I agree with Josh on this one (except, I would always save
load_info if it is a livepatch module, instead of for every module in the
!CONFIG_LIVEPATCH case, and we can just check modinfo to see if it is
a livepatch module).

If the tradeoff here is between simplicity and readibility of code vs.
saving some extra space (and by the looks of it, not a lot), I think I
would choose having clear code over saving some bytes of memory. Hard
coding checks and edge cases imo might cause confusion and trouble
down the road.

Re: module: save load_info for livepatch modules

[Jessica Yu]

+++ Josh Poimboeuf [12/11/15 11:05 -0600]:
>On Thu, Nov 12, 2015 at 04:03:45PM +0100, Petr Mladek wrote:
>> On Thu 2015-11-12 14:22:28, Miroslav Benes wrote:
>> > On Thu, 12 Nov 2015, Petr Mladek wrote:
>> > > > >Maybe I am missing something but isn't it necessary to call vfree() on
>> > > > >info somewhere in the end?
>> > > >
>> > > > So free_copy() will call vfree(info->hdr), except in livepatch modules
>> > > > we want to keep all the elf section information stored there, so we
>> > > > avoid calling free_copy(), As for the info struct itself, if you look
>> > > > at the init_module and finit_module syscall definitions in
>> > > > kernel/module.c, you will see that info is actually a local function
>> > > > variable, simply passed in to the call to load_module(), and will be
>> > > > automatically deallocated when the syscall returns. :-) No need to
>> > > > explicitly free info.
>> > >
>> > > We still have to free the copied or preserved structures when
>> > > the module is unloaded.
>> >
>> > ...freeing what we allocated. We need to free info->hdr somewhere if not
>> > here and also mod_arch_specific struct where the patch module is removed.
>> > This would unfortunately lead to changes in arch-specific code in
>> > module.c. For example in arch/s390/kernel/module.c there is vfree call on
>> > part of mod_arch_specific in module_finalize. We would call it only if the
>> > flag mentioned above is not set and at the same time we would need to call
>> > it when the patch module is being removed.
>>
>> Sigh, I am afraid that the flag is not enough. IMHO, we need to split
>> the load finalizing functions into two pieces. One will be always
>> called when the module load is finalized. The other part will free
>> the load_info. It will be called either when the load is finalized or
>> when the module is unloaded, depending on if we want to preserve
>> the load_info.
>>
>> Sigh, it is getting complicated. But let's see how it looks in reality.
>
>At the other end of the spectrum, we could do the simplest thing
>possible: _always_ save the data (even if CONFIG_LIVEPATCH is disabled).
>
>(gdb) print sizeof(*info)
>$3 = 96
>(gdb) p sizeof(*info->hdr)
>$4 = 64
>s390 mod_arch_syminfo struct: 24 bytes by my reckoning.
>
>So between info, info->hdr, and s390 mod_arch_syminfo, we're talking
>about 184 bytes on s390 and 160 bytes on x86_64.  That seems like
>peanuts compared to the size of a typical module.  The benefit is that
>the code would be simpler because we don't have any special cases and
>the structs would automatically get freed with the module struct when
>the module gets unloaded.

I think I agree with Josh on this one (except, I would always save
load_info if it is a livepatch module, instead of for every module in the
!CONFIG_LIVEPATCH case, and we can just check modinfo to see if it is
a livepatch module).

If the tradeoff here is between simplicity and readibility of code vs.
saving some extra space (and by the looks of it, not a lot), I think I
would choose having clear code over saving some bytes of memory. Hard
coding checks and edge cases imo might cause confusion and trouble
down the road.

Re: module: save load_info for livepatch modules

[Jessica Yu]

+++ Miroslav Benes [12/11/15 14:22 +0100]:
>On Thu, 12 Nov 2015, Petr Mladek wrote:
>
>> On Thu 2015-11-12 00:33:12, Jessica Yu wrote:
>> > +++ Miroslav Benes [11/11/15 15:17 +0100]:
>> > >On Mon, 9 Nov 2015, Jessica Yu wrote:
>> > >
>> > >>diff --git a/include/linux/module.h b/include/linux/module.h
>> > >>index 3a19c79..c8680b1 100644
>> > >>--- a/include/linux/module.h
>> > >>+++ b/include/linux/module.h
>> > >
>> > >[...]
>> > >
>> > >>+#ifdef CONFIG_LIVEPATCH
>> > >>+extern void klp_prepare_patch_module(struct module *mod,
>> > >>+				     struct load_info *info);
>> > >>+extern int
>> > >>+apply_relocate_add(Elf64_Shdr *sechdrs, const char *strtab,
>> > >>+		   unsigned int symindex, unsigned int relsec,
>> > >>+		   struct module *me);
>> > >>+#endif
>> > >>+
>> > >> #else /* !CONFIG_MODULES... */
>> > >
>> > >apply_relocate_add() is already in include/linux/moduleloader.h (guarded
>> > >by CONFIG_MODULES_USE_ELF_RELA), so maybe we can just include that where
>> > >we need it. As for the klp_prepare_patch_module() wouldn't it be better to
>> > >have it in our livepatch.h and include that in kernel/module.c?
>> >
>> > Yeah, Petr pointed this out as well :-) I will just include
>> > moduleloader.h for the apply_relocate_add() declaration.
>> >
>> > It also looks like we have some disagreement over where to put
>> > klp_prepare_patch_module(), either in livepatch/core.c (and add the
>> > function declaration in livepatch.h, and have module.c include
>> > livepatch.h) or in kernel/module.c, keeping the
>> > klp_prepare_patch_module() declaration in module.h. Maybe Rusty can
>> > provide some input.
>
>Yes, there are several ways how to do it. Maybe the best would be some
>generic way in kernel/module.c. I am not sure if there will be another
>user of this in the future but nevertheless. It would also allow us to
>somehow solve the issues mentioned below. Thus, klp_prepare_patch_module
>is inappropriate name and it should be for example just preserve_load_info
>(or more general if needed) and it should be in kernel/module.c.

A more generic way sounds good. I think Petr is leaning towards this
too, i.e. have a generic function named copy_module_info() in
module.c, instead of klp_prepare_patch_module().

>> > >> /* Given an address, look for it in the exception tables. */
>> > >>diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>> > >>index 6e53441..087a8c7 100644
>> > >>--- a/kernel/livepatch/core.c
>> > >>+++ b/kernel/livepatch/core.c
>> > >>@@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>> > >> 	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>> > >> };
>> > >>
>> > >>+/*
>> > >>+ * Save necessary information from info in order to be able to
>> > >>+ * patch modules that might be loaded later
>> > >>+ */
>> > >>+void klp_prepare_patch_module(struct module *mod, struct load_info *info)
>> > >>+{
>> > >>+	Elf_Shdr *symsect;
>> > >>+
>> > >>+	symsect = info->sechdrs + info->index.sym;
>> > >>+	/* update sh_addr to point to symtab */
>> > >>+	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
>> > >>+
>> > >>+	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
>> > >>+	memcpy(mod->info, info, sizeof(*info));
>> > >>+
>> > >>+}
>> > >
>> > >What about arch-specific 'struct mod_arch_specific'? We need to preserve
>> > >it somewhere as well for s390x and other non-x86 architectures.
>> >
>> > Ah! Thank you for catching this, I overlooked this important detail.
>> > Yes, we do need to save the arch-specific struct. We would be in
>> > trouble for s390 relocs if we didn't. I am trying to think of a way to
>> > save this information for s390, since s390's module_finalize() frees
>> > mod->arch.syminfo, which we definitely need in order for the call to
>> > apply_relocate_add() to work. Maybe we can add an extra call right
>> > before module_finalize() that will do some livepatch-specific
>> > processing and copy this information (this would be in
>> > post_relocation() in kernel/module.c). Perhaps this patchset cannot be
>> > entirely free of arch-specific code after all :-( Still thinking.
>
>Well, mod_arch_specific is defined as each architecture needs. So for x86
>it is empty. It is arch-agnostic in this way and we can deal with it as
>"a black box". We just need it not to be freed in module_finalize. And...
>
>> I think about adding a flag somewhere, e.g. mod->preserve_relocs.
>> It might be set in simplify_symbols() when SHN_LIVEPATCH is found.
>> It might be checked when freeing the needed structures in both
>> the generic and arch-specific code.
>
>...that is the reason why some sort of flag seems to be necessary. It
>could be set when livepatch is set in modinfo. We would use it for
>preserving both load_info and mod_arch_specific struct (in some form) and
>for...
>
>> > >>+#ifdef CONFIG_LIVEPATCH
>> > >>+	/*
>> > >>+	 * Save sechdrs, indices, and other data from info
>> > >>+	 * in order to patch to-be-loaded modules.
>> > >>+	 * Do not call free_copy() for livepatch modules.
>> > >>+	 */
>> > >>+	if (get_modinfo((struct load_info *)info, "livepatch"))
>> > >>+		klp_prepare_patch_module(mod, info);
>> > >>+	else
>> > >>+		free_copy(info);
>> > >>+#else
>> > >> 	/* Get rid of temporary copy. */
>> > >> 	free_copy(info);
>> > >>+#endif
>> > >
>> > >Maybe I am missing something but isn't it necessary to call vfree() on
>> > >info somewhere in the end?
>> >
>> > So free_copy() will call vfree(info->hdr), except in livepatch modules
>> > we want to keep all the elf section information stored there, so we
>> > avoid calling free_copy(), As for the info struct itself, if you look
>> > at the init_module and finit_module syscall definitions in
>> > kernel/module.c, you will see that info is actually a local function
>> > variable, simply passed in to the call to load_module(), and will be
>> > automatically deallocated when the syscall returns. :-) No need to
>> > explicitly free info.
>>
>> We still have to free the copied or preserved structures when
>> the module is unloaded.
>
>...freeing what we allocated. We need to free info->hdr somewhere if not
>here and also mod_arch_specific struct where the patch module is removed.

Right, I intended to free the preserved/copied structures in patch_exit():
https://github.com/flaming-toast/kpatch/blob/no_dynrela_redux/kmod/patch/livepatch-patch-hook.c#L322

But now that I'm thinking about it, perhaps it is better and clearer
to have the freeing be done in free_module()?

>This would unfortunately lead to changes in arch-specific code in
>module.c. For example in arch/s390/kernel/module.c there is vfree call on
>part of mod_arch_specific in module_finalize. We would call it only if the
>flag mentioned above is not set and at the same time we would need to call
>it when the patch module is being removed.

Yup..this is what I meant when I said I was concerned that this
patchset might end up needing arch-specific code. :-\

Hard coding a flag check doesn't seem very portable or modular (here,
it would be a specific case to s390). If we do require arch code, how
about using a small arch-specific livepatch function to do the
copying, maybe call it klp_copy_arch_info()?

Actually, if we're going the generic route, we can just call it
copy_arch_info(). Maybe we can put the call and definition in
arch/../kernel/module.c, and it will copy the mod_arch_specific struct
(plus do whatever else that's needed). In the case of s390, we need to
additionally copy the mod_arch_syminfo array. Then, we can just leave
the vfree alone.

So in this scheme, I'd imagine we'd have copy_module_info() +
copy_arch_info(), called from the module loader if the module is a
livepatch module. However I am not yet entirely sure where to put the
call to copy_arch_info(), maybe within module_finalize()? 
Then, we could have the corresponding free_module_info() and
free_arch_info() functions, maybe called from free_module() instead of
patch_exit(). Does this sound too complicated? Would it work?

Re: module: save load_info for livepatch modules

[Jessica Yu]

+++ Miroslav Benes [12/11/15 15:19 +0100]:
>On Thu, 12 Nov 2015, Petr Mladek wrote:
>
>> On Wed 2015-11-11 23:44:08, Jessica Yu wrote:
>> > +++ Petr Mladek [11/11/15 15:31 +0100]:
>> > >On Mon 2015-11-09 23:45:52, Jessica Yu wrote:
>> > >>diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>> > >>index 6e53441..087a8c7 100644
>> > >>--- a/kernel/livepatch/core.c
>> > >>+++ b/kernel/livepatch/core.c
>> > >>@@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>> > >> 	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>> > >> };
>> > >>
>> > >>+/*
>> > >>+ * Save necessary information from info in order to be able to
>> > >>+ * patch modules that might be loaded later
>> > >>+ */
>> > >>+void klp_prepare_patch_module(struct module *mod, struct load_info *info)
>> > >>+{
>> > >>+	Elf_Shdr *symsect;
>> > >>+
>> > >>+	symsect = info->sechdrs + info->index.sym;
>> > >>+	/* update sh_addr to point to symtab */
>> > >>+	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
>> > >
>> > >Is livepatch the only user of this value? By other words, is this safe?
>> >
>> > I think it is safe to say yes. klp_prepare_patch_module() is only
>> > called at the very end of load_module(), right before
>> > do_init_module(). Normally, at that point, info->hdr will have already
>> > been freed by free_copy() along with the elf section information
>> > associated with it. But if we have a livepatch module, we don't free.
>> > So we should be the very last user, and there should be nobody
>> > utilizing the memory associated with the load_info struct anymore at
>> > that point.
>>
>> I see. It looks safe at this point. But still I wonder if it would be
>> possible to calculate this later in the livepatch code. It will allow
>> to potentially use the info structure also by other subsystem.
>>
>> BTW: Where is "sh_addr" value used, please? I see it used only
>> in the third patch as info->sechdrs[relindex].sh_addr. But it is
>> an array. I am not sure if it is the same variable.
>
>Jessica, why do we need to update sh_addr for symtab? It is not clear to
>me.

Ah, I definitely need to make that comment a lot more informative.
Will make sure to add that in v2. 

So, the sh_addr field tells us where a certain section is in memory.
Here, we need to update the symbol table section's sh_addr because if
we don't, it will eventually point to freed module init memory, which
is freed in do_init_module(). Let me explain what happens.

At the beginning of load_module(), the sh_addr fields of each section
initially point to the vmalloc'd memory within info->hdr (which is
allocated in copy_module_from_{fd,user}() in module.c). The sh_addr's
are first assigned in rewrite_section_headers(), called from
setup_load_info(). These sh_addr's initially just point to an offset
within info->hdr depending on each section's sh_offset.

However, in move_module(), where we layout and allocate the memory
where the module will finally reside, these sh_addr's will get
reassigned. For the symtab section's sh_addr, it gets reassigned to
module init memory. (In layout_symtab(), you'll see that the symtab
section gets marked with INIT_OFFSET_MASK, which indicates that it
will get an address in init memory when the sh_addr's get reassigned
in move_module()). Thus the symbol table that simplify_symbols() uses
is actually in init memory, and will be freed later in
do_init_module().

info->hdr is just a temporary holding place for module elf section
data in memory. Normally, we would get rid of info->hdr and free the
memory associated with it at the end of the module loading process
(via free_copy()). But in this patchset, we save all the original elf
section information because we need it (along with the original
symtab) in order to make the call to apply_relocate_add(). If you look
at apply_relocate_add() for x86, s390, etc you'll see that it expects
a symbol table at the symbol section's sh_addr field (basically, an
array of Elf_Sym's). This is why we fix up the sh_addr of the symtab
section to point back to the memory associated with info->hdr (and not
module init memory). I hope that makes sense.

Thanks,
Jessica

Re: module: save load_info for livepatch modules

[Jessica Yu]

+++ Miroslav Benes [12/11/15 15:19 +0100]:
>On Thu, 12 Nov 2015, Petr Mladek wrote:
>
>> On Wed 2015-11-11 23:44:08, Jessica Yu wrote:
>> > +++ Petr Mladek [11/11/15 15:31 +0100]:
>> > >On Mon 2015-11-09 23:45:52, Jessica Yu wrote:
>> > >>diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>> > >>index 6e53441..087a8c7 100644
>> > >>--- a/kernel/livepatch/core.c
>> > >>+++ b/kernel/livepatch/core.c
>> > >>@@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>> > >> 	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>> > >> };
>> > >>
>> > >>+/*
>> > >>+ * Save necessary information from info in order to be able to
>> > >>+ * patch modules that might be loaded later
>> > >>+ */
>> > >>+void klp_prepare_patch_module(struct module *mod, struct load_info *info)
>> > >>+{
>> > >>+	Elf_Shdr *symsect;
>> > >>+
>> > >>+	symsect = info->sechdrs + info->index.sym;
>> > >>+	/* update sh_addr to point to symtab */
>> > >>+	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
>> > >
>> > >Is livepatch the only user of this value? By other words, is this safe?
>> >
>> > I think it is safe to say yes. klp_prepare_patch_module() is only
>> > called at the very end of load_module(), right before
>> > do_init_module(). Normally, at that point, info->hdr will have already
>> > been freed by free_copy() along with the elf section information
>> > associated with it. But if we have a livepatch module, we don't free.
>> > So we should be the very last user, and there should be nobody
>> > utilizing the memory associated with the load_info struct anymore at
>> > that point.
>>
>> I see. It looks safe at this point. But still I wonder if it would be
>> possible to calculate this later in the livepatch code. It will allow
>> to potentially use the info structure also by other subsystem.
>>
>> BTW: Where is "sh_addr" value used, please? I see it used only
>> in the third patch as info->sechdrs[relindex].sh_addr. But it is
>> an array. I am not sure if it is the same variable.
>
>Jessica, why do we need to update sh_addr for symtab? It is not clear to
>me.

Ah, I definitely need to make that comment a lot more informative.
Will make sure to add that in v2. 

So, the sh_addr field tells us where a certain section is in memory.
Here, we need to update the symbol table section's sh_addr because if
we don't, it will eventually point to freed module init memory, which
is freed in do_init_module(). Let me explain what happens.

At the beginning of load_module(), the sh_addr fields of each section
initially point to the vmalloc'd memory within info->hdr (which is
allocated in copy_module_from_{fd,user}() in module.c). The sh_addr's
are first assigned in rewrite_section_headers(), called from
setup_load_info(). These sh_addr's initially just point to an offset
within info->hdr depending on each section's sh_offset.

However, in move_module(), where we layout and allocate the memory
where the module will finally reside, these sh_addr's will get
reassigned. For the symtab section's sh_addr, it gets reassigned to
module init memory. (In layout_symtab(), you'll see that the symtab
section gets marked with INIT_OFFSET_MASK, which indicates that it
will get an address in init memory when the sh_addr's get reassigned
in move_module()). Thus the symbol table that simplify_symbols() uses
is actually in init memory, and will be freed later in
do_init_module().

info->hdr is just a temporary holding place for module elf section
data in memory. Normally, we would get rid of info->hdr and free the
memory associated with it at the end of the module loading process
(via free_copy()). But in this patchset, we save all the original elf
section information because we need it (along with the original
symtab) in order to make the call to apply_relocate_add(). If you look
at apply_relocate_add() for x86, s390, etc you'll see that it expects
a symbol table at the symbol section's sh_addr field (basically, an
array of Elf_Sym's). This is why we fix up the sh_addr of the symtab
section to point back to the memory associated with info->hdr (and not
module init memory). I hope that makes sense.

Thanks,
Jessica

Re: elf: add livepatch-specific elf constants

[Jessica Yu]

+++ Josh Poimboeuf [12/11/15 09:45 -0600]:
>On Mon, Nov 09, 2015 at 11:45:51PM -0500, Jessica Yu wrote:
>> Add livepatch elf reloc section flag, livepatch symbol bind
>> and section index
>>
>> Signed-off-by: Jessica Yu <jeyu@redhat.com>
>> ---
>>  include/uapi/linux/elf.h | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h
>> index 71e1d0e..967ce1b 100644
>> --- a/include/uapi/linux/elf.h
>> +++ b/include/uapi/linux/elf.h
>> @@ -118,6 +118,7 @@ typedef __s64	Elf64_Sxword;
>>  #define STB_LOCAL  0
>>  #define STB_GLOBAL 1
>>  #define STB_WEAK   2
>> +#define STB_LIVEPATCH_EXT 11
>>
>>  #define STT_NOTYPE  0
>>  #define STT_OBJECT  1
>> @@ -286,6 +287,7 @@ typedef struct elf64_phdr {
>>  #define SHF_ALLOC	0x2
>>  #define SHF_EXECINSTR	0x4
>>  #define SHF_MASKPROC	0xf0000000
>> +#define SHF_RELA_LIVEPATCH  0x4000000
>
>Writing the value with leading zeros (0x04000000) would it more
>readable.
>
>Also the OS-specific range mask (SHF_MASKOS) is 0x0ff00000.  Any reason
>you went with 0x04000000 as opposed to the first value in the range
>(0x00100000)?  I don't see anybody else using that value.

I don't have any particular reason, I think I just picked any value
and ran with it. I'll just change it to the first value in the range
since that makes more sense.

>>  /* special section indexes */
>>  #define SHN_UNDEF	0
>> @@ -295,6 +297,7 @@ typedef struct elf64_phdr {
>>  #define SHN_ABS		0xfff1
>>  #define SHN_COMMON	0xfff2
>>  #define SHN_HIRESERVE	0xffff
>> +#define SHN_LIVEPATCH 0xff21
>
>Similar question here, why not use 0xff20 (SHN_LOOS)?
>
>-- 
>Josh

Re: livepatch: reuse module loader code to write relocations

[Jessica Yu]

+++ Josh Poimboeuf [12/11/15 14:32 -0600]:
>On Thu, Nov 12, 2015 at 03:22:44PM -0500, Jessica Yu wrote:
>> Looking into this more, I think we do need one __klp_rela section per
>> function being patched.  Each rela section is linked to the section to
>> which the relocations apply via the rela section's sh_info field. In
>> SHT_RELA sections, the sh_info field contains the section index to
>> which the relocs apply. We cannot have one single combined rela
>> section per object as the call to apply_relocate_add() simply won't
>> work, because we would have relocs that apply to different functions
>> (and hence different sections).
>>
>> So I guess instead of a single field in klp_object specifying the
>> __klp_rela section index, we could probably just have an array of
>> section indices.
>
>Ok, makes sense, sounds like we need multiple klp relas per object.
>
>I still don't quite understand the benefit of caching the klp_rela
>section indices.  What problem does it solve?  It seems simpler to just
>iterate over all the sections in klp_write_object_relocations().

I was trying to accomodate Miroslav's dislike of klp_reloc_sec :-)
But now that we are sure that we need multiple __klp_rela sections
per object, I think an array of indices might still be somewhat
complicated (we need to figure out its size and everything, a list
here would be easier). I think it might be fine to leave things the
way they are here.

Jessica

Re: livepatch: reuse module loader code to write relocations

[Jessica Yu]

+++ Josh Poimboeuf [12/11/15 14:32 -0600]:
>On Thu, Nov 12, 2015 at 03:22:44PM -0500, Jessica Yu wrote:
>> Looking into this more, I think we do need one __klp_rela section per
>> function being patched.  Each rela section is linked to the section to
>> which the relocations apply via the rela section's sh_info field. In
>> SHT_RELA sections, the sh_info field contains the section index to
>> which the relocs apply. We cannot have one single combined rela
>> section per object as the call to apply_relocate_add() simply won't
>> work, because we would have relocs that apply to different functions
>> (and hence different sections).
>>
>> So I guess instead of a single field in klp_object specifying the
>> __klp_rela section index, we could probably just have an array of
>> section indices.
>
>Ok, makes sense, sounds like we need multiple klp relas per object.
>
>I still don't quite understand the benefit of caching the klp_rela
>section indices.  What problem does it solve?  It seems simpler to just
>iterate over all the sections in klp_write_object_relocations().

I was trying to accomodate Miroslav's dislike of klp_reloc_sec :-)
But now that we are sure that we need multiple __klp_rela sections
per object, I think an array of indices might still be somewhat
complicated (we need to figure out its size and everything, a list
here would be easier). I think it might be fine to leave things the
way they are here.

Jessica

Re: module: save load_info for livepatch modules

[Jessica Yu]

+++ Petr Mladek [12/11/15 11:05 +0100]:
>On Wed 2015-11-11 23:44:08, Jessica Yu wrote:
>> +++ Petr Mladek [11/11/15 15:31 +0100]:
>> >On Mon 2015-11-09 23:45:52, Jessica Yu wrote:
>> >>diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>> >>index 6e53441..087a8c7 100644
>> >>--- a/kernel/livepatch/core.c
>> >>+++ b/kernel/livepatch/core.c
>> >>@@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>> >> 	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>> >> };
>> >>
>> >>+/*
>> >>+ * Save necessary information from info in order to be able to
>> >>+ * patch modules that might be loaded later
>> >>+ */
>> >>+void klp_prepare_patch_module(struct module *mod, struct load_info *info)
>> >>+{
>> >>+	Elf_Shdr *symsect;
>> >>+
>> >>+	symsect = info->sechdrs + info->index.sym;
>> >>+	/* update sh_addr to point to symtab */
>> >>+	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
>> >
>> >Is livepatch the only user of this value? By other words, is this safe?
>>
>> I think it is safe to say yes. klp_prepare_patch_module() is only
>> called at the very end of load_module(), right before
>> do_init_module(). Normally, at that point, info->hdr will have already
>> been freed by free_copy() along with the elf section information
>> associated with it. But if we have a livepatch module, we don't free.
>> So we should be the very last user, and there should be nobody
>> utilizing the memory associated with the load_info struct anymore at
>> that point.
>
>I see. It looks safe at this point. But still I wonder if it would be
>possible to calculate this later in the livepatch code. It will allow
>to potentially use the info structure also by other subsystem.

We can technically reassign sh_addr later in livepatch somewhere, yes,
I'd have to think more about where it'd make the most sense to do
this. Maybe in patch_init? It just seemed at the time a bit clearer to
do it in klp_prepare_patch_module() (soon to be called
copy_module_info() probably).

>BTW: Where is "sh_addr" value used, please? I see it used only
>in the third patch as info->sechdrs[relindex].sh_addr. But it is
>an array. I am not sure if it is the same variable.

I will add a more informative comment in the code, see my reply to
Miroslav.

>
>> >>+	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
>> >>+	memcpy(mod->info, info, sizeof(*info));
>> >>+
>> >>+}
>> >
>> >It is strange that this funtion is defined in livepatch/core.c
>> >but declared in module.h. I would move the definition to
>> >module.c.
>>
>> Right, I was trying to keep all the livepatch-related functions
>> together in livepatch/core.c. but I can move it to module.c if it
>> makes more sense/Rusty doesn't object to it :-)
>
>Sure. I think that we could use some generic name, e.g. copy_module_info().
>
>> >> static int __init klp_init(void)
>> >> {
>> >> 	int ret;
>> >>diff --git a/kernel/module.c b/kernel/module.c
>> >>index 8f051a1..8ae3ca5 100644
>> >>--- a/kernel/module.c
>> >>+++ b/kernel/module.c
>> >>@@ -2137,6 +2123,11 @@ static int simplify_symbols(struct module *mod, const struct load_info *info)
>> >> 			       (long)sym[i].st_value);
>> >> 			break;
>> >>
>> >>+#ifdef CONFIG_LIVEPATCH
>> >>+		case SHN_LIVEPATCH:
>> >>+			break;
>> >>+#endif
>> >
>> >IMHO, even a kernel compiled without CONFIG_LIVEPATCH should handle livepatch
>> >modules with grace. It means to reject loading.
>>
>> I think even right now, without considering this patchset, we don't
>> reject modules "gracefully" when we load a livepatch module without
>> CONFIG_LIVEPATCH. The module loader will complain and reject the
>> livepatch module, saying something like "Unknown symbol
>> klp_register_patch." This behavior is the same with or without
>> this patch series applied. If we want to add a bit more logic to
>> gracefully reject patch modules, perhaps that should be a different
>> patch altogether, as I think it is unrelated to the goal of this one :-)
>
>Yup, the module load would fail anyway because of the missing symbol.
>But I think that we should fail on the first error occurence.
>
>In each case, IMHO, we should not do the "default:" action for this
>section even when complied without CONFIG_LIVEPATCH.

See comment below -- 

>
>> >> 		case SHN_UNDEF:
>> >> 			ksym = resolve_symbol_wait(mod, info, name);
>> >> 			/* Ok if resolved.  */
>> >>@@ -2185,6 +2176,11 @@ static int apply_relocations(struct module *mod, const struct load_info *info)
>> >> 		if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC))
>> >> 			continue;
>> >>
>> >>+#ifdef CONFIG_LIVEPATCH
>> >>+		if (info->sechdrs[i].sh_flags & SHF_RELA_LIVEPATCH)
>> >>+			continue;
>> >>+#endif
>
>I guess that if we do not trigger the error above, and do
>not have the check here, we will try to call apply_relocate() below.
>I guess that it will fail. If we are lucky it will print "Unknown
>relocation". I think that we could do better.

For the loading of livepatch modules in !CONFIG_LIVEPATCH kernels, we
should probably gracefully reject it in the beginning of load_module()
(so that MODULE_INFO flag might come in handy here after all). If it's
a livepatch module && !CONFIG_LIVEPATCH, reject it. Then we wouldn't
even call apply_relocations() here, we wouldn't run into the
possibility of this check falling through, nor would
simplify_symbols() be even called.

>> >>+
>> >> 		if (info->sechdrs[i].sh_type == SHT_REL)
>> >> 			err = apply_relocate(info->sechdrs, info->strtab,
>> >> 					     info->index.sym, i, mod);
>> >>@@ -3530,8 +3526,20 @@ static int load_module(struct load_info *info, const char __user *uargs,
>> >> 	if (err < 0)
>> >> 		goto bug_cleanup;
>> >>
>> >>+#ifdef CONFIG_LIVEPATCH
>> >>+	/*
>> >>+	 * Save sechdrs, indices, and other data from info
>> >>+	 * in order to patch to-be-loaded modules.
>> >>+	 * Do not call free_copy() for livepatch modules.
>> >>+	 */
>> >>+	if (get_modinfo((struct load_info *)info, "livepatch"))
>> >>+		klp_prepare_patch_module(mod, info);
>> >>+	else
>> >>+		free_copy(info);
>> >>+#else
>> >
>> >I would move this #else one line above and get rid of the
>> >double free_copy(info); But it is a matter of taste.
>>
>> Maybe I'm missing something, but I think we do need the double
>> free_copy(), because in the CONFIG_LIVEPATCH case, we still want to
>> call free_copy() for non-livepatch modules. And we want to avoid
>> calling free_copy() for livepatch modules (hence the extra else).
>
>Ah, this was just a cosmetic change. I meant to use something like:
>
>#ifdef CONFIG_LIVEPATCH
>	/*
>	 * Save sechdrs, indices, and other data from info
>	 * in order to patch to-be-loaded modules.
>	 * Do not call free_copy() for livepatch modules.
>	 */
>	if (get_modinfo((struct load_info *)info, "livepatch"))
>		klp_prepare_patch_module(mod, info);
>	else
>#endif
>		/* Get rid of temporary copy. */
>		free_copy(info);
>

Oh OK, so that's what you meant. :-)

Thanks,
Jessica

Re: module: save load_info for livepatch modules

[Jessica Yu]

+++ Petr Mladek [12/11/15 11:05 +0100]:
>On Wed 2015-11-11 23:44:08, Jessica Yu wrote:
>> +++ Petr Mladek [11/11/15 15:31 +0100]:
>> >On Mon 2015-11-09 23:45:52, Jessica Yu wrote:
>> >>diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
>> >>index 6e53441..087a8c7 100644
>> >>--- a/kernel/livepatch/core.c
>> >>+++ b/kernel/livepatch/core.c
>> >>@@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
>> >> 	.priority = INT_MIN+1, /* called late but before ftrace notifier */
>> >> };
>> >>
>> >>+/*
>> >>+ * Save necessary information from info in order to be able to
>> >>+ * patch modules that might be loaded later
>> >>+ */
>> >>+void klp_prepare_patch_module(struct module *mod, struct load_info *info)
>> >>+{
>> >>+	Elf_Shdr *symsect;
>> >>+
>> >>+	symsect = info->sechdrs + info->index.sym;
>> >>+	/* update sh_addr to point to symtab */
>> >>+	symsect->sh_addr = (unsigned long)info->hdr + symsect->sh_offset;
>> >
>> >Is livepatch the only user of this value? By other words, is this safe?
>>
>> I think it is safe to say yes. klp_prepare_patch_module() is only
>> called at the very end of load_module(), right before
>> do_init_module(). Normally, at that point, info->hdr will have already
>> been freed by free_copy() along with the elf section information
>> associated with it. But if we have a livepatch module, we don't free.
>> So we should be the very last user, and there should be nobody
>> utilizing the memory associated with the load_info struct anymore at
>> that point.
>
>I see. It looks safe at this point. But still I wonder if it would be
>possible to calculate this later in the livepatch code. It will allow
>to potentially use the info structure also by other subsystem.

We can technically reassign sh_addr later in livepatch somewhere, yes,
I'd have to think more about where it'd make the most sense to do
this. Maybe in patch_init? It just seemed at the time a bit clearer to
do it in klp_prepare_patch_module() (soon to be called
copy_module_info() probably).

>BTW: Where is "sh_addr" value used, please? I see it used only
>in the third patch as info->sechdrs[relindex].sh_addr. But it is
>an array. I am not sure if it is the same variable.

I will add a more informative comment in the code, see my reply to
Miroslav.

>
>> >>+	mod->info = kzalloc(sizeof(*info), GFP_KERNEL);
>> >>+	memcpy(mod->info, info, sizeof(*info));
>> >>+
>> >>+}
>> >
>> >It is strange that this funtion is defined in livepatch/core.c
>> >but declared in module.h. I would move the definition to
>> >module.c.
>>
>> Right, I was trying to keep all the livepatch-related functions
>> together in livepatch/core.c. but I can move it to module.c if it
>> makes more sense/Rusty doesn't object to it :-)
>
>Sure. I think that we could use some generic name, e.g. copy_module_info().
>
>> >> static int __init klp_init(void)
>> >> {
>> >> 	int ret;
>> >>diff --git a/kernel/module.c b/kernel/module.c
>> >>index 8f051a1..8ae3ca5 100644
>> >>--- a/kernel/module.c
>> >>+++ b/kernel/module.c
>> >>@@ -2137,6 +2123,11 @@ static int simplify_symbols(struct module *mod, const struct load_info *info)
>> >> 			       (long)sym[i].st_value);
>> >> 			break;
>> >>
>> >>+#ifdef CONFIG_LIVEPATCH
>> >>+		case SHN_LIVEPATCH:
>> >>+			break;
>> >>+#endif
>> >
>> >IMHO, even a kernel compiled without CONFIG_LIVEPATCH should handle livepatch
>> >modules with grace. It means to reject loading.
>>
>> I think even right now, without considering this patchset, we don't
>> reject modules "gracefully" when we load a livepatch module without
>> CONFIG_LIVEPATCH. The module loader will complain and reject the
>> livepatch module, saying something like "Unknown symbol
>> klp_register_patch." This behavior is the same with or without
>> this patch series applied. If we want to add a bit more logic to
>> gracefully reject patch modules, perhaps that should be a different
>> patch altogether, as I think it is unrelated to the goal of this one :-)
>
>Yup, the module load would fail anyway because of the missing symbol.
>But I think that we should fail on the first error occurence.
>
>In each case, IMHO, we should not do the "default:" action for this
>section even when complied without CONFIG_LIVEPATCH.

See comment below -- 

>
>> >> 		case SHN_UNDEF:
>> >> 			ksym = resolve_symbol_wait(mod, info, name);
>> >> 			/* Ok if resolved.  */
>> >>@@ -2185,6 +2176,11 @@ static int apply_relocations(struct module *mod, const struct load_info *info)
>> >> 		if (!(info->sechdrs[infosec].sh_flags & SHF_ALLOC))
>> >> 			continue;
>> >>
>> >>+#ifdef CONFIG_LIVEPATCH
>> >>+		if (info->sechdrs[i].sh_flags & SHF_RELA_LIVEPATCH)
>> >>+			continue;
>> >>+#endif
>
>I guess that if we do not trigger the error above, and do
>not have the check here, we will try to call apply_relocate() below.
>I guess that it will fail. If we are lucky it will print "Unknown
>relocation". I think that we could do better.

For the loading of livepatch modules in !CONFIG_LIVEPATCH kernels, we
should probably gracefully reject it in the beginning of load_module()
(so that MODULE_INFO flag might come in handy here after all). If it's
a livepatch module && !CONFIG_LIVEPATCH, reject it. Then we wouldn't
even call apply_relocations() here, we wouldn't run into the
possibility of this check falling through, nor would
simplify_symbols() be even called.

>> >>+
>> >> 		if (info->sechdrs[i].sh_type == SHT_REL)
>> >> 			err = apply_relocate(info->sechdrs, info->strtab,
>> >> 					     info->index.sym, i, mod);
>> >>@@ -3530,8 +3526,20 @@ static int load_module(struct load_info *info, const char __user *uargs,
>> >> 	if (err < 0)
>> >> 		goto bug_cleanup;
>> >>
>> >>+#ifdef CONFIG_LIVEPATCH
>> >>+	/*
>> >>+	 * Save sechdrs, indices, and other data from info
>> >>+	 * in order to patch to-be-loaded modules.
>> >>+	 * Do not call free_copy() for livepatch modules.
>> >>+	 */
>> >>+	if (get_modinfo((struct load_info *)info, "livepatch"))
>> >>+		klp_prepare_patch_module(mod, info);
>> >>+	else
>> >>+		free_copy(info);
>> >>+#else
>> >
>> >I would move this #else one line above and get rid of the
>> >double free_copy(info); But it is a matter of taste.
>>
>> Maybe I'm missing something, but I think we do need the double
>> free_copy(), because in the CONFIG_LIVEPATCH case, we still want to
>> call free_copy() for non-livepatch modules. And we want to avoid
>> calling free_copy() for livepatch modules (hence the extra else).
>
>Ah, this was just a cosmetic change. I meant to use something like:
>
>#ifdef CONFIG_LIVEPATCH
>	/*
>	 * Save sechdrs, indices, and other data from info
>	 * in order to patch to-be-loaded modules.
>	 * Do not call free_copy() for livepatch modules.
>	 */
>	if (get_modinfo((struct load_info *)info, "livepatch"))
>		klp_prepare_patch_module(mod, info);
>	else
>#endif
>		/* Get rid of temporary copy. */
>		free_copy(info);
>

Oh OK, so that's what you meant. :-)

Thanks,
Jessica

Re: module: save load_info for livepatch modules

[Miroslav Benes]

On Thu, 12 Nov 2015, Jessica Yu wrote:

> +++ Josh Poimboeuf [12/11/15 11:05 -0600]:
> > On Thu, Nov 12, 2015 at 04:03:45PM +0100, Petr Mladek wrote:
> > > On Thu 2015-11-12 14:22:28, Miroslav Benes wrote:
> > > > On Thu, 12 Nov 2015, Petr Mladek wrote:
> > > > > > >Maybe I am missing something but isn't it necessary to call vfree()
> > > on
> > > > > > >info somewhere in the end?
> > > > > >
> > > > > > So free_copy() will call vfree(info->hdr), except in livepatch
> > > modules
> > > > > > we want to keep all the elf section information stored there, so we
> > > > > > avoid calling free_copy(), As for the info struct itself, if you
> > > look
> > > > > > at the init_module and finit_module syscall definitions in
> > > > > > kernel/module.c, you will see that info is actually a local function
> > > > > > variable, simply passed in to the call to load_module(), and will be
> > > > > > automatically deallocated when the syscall returns. :-) No need to
> > > > > > explicitly free info.
> > > > >
> > > > > We still have to free the copied or preserved structures when
> > > > > the module is unloaded.
> > > >
> > > > ...freeing what we allocated. We need to free info->hdr somewhere if not
> > > > here and also mod_arch_specific struct where the patch module is
> > > removed.
> > > > This would unfortunately lead to changes in arch-specific code in
> > > > module.c. For example in arch/s390/kernel/module.c there is vfree call
> > > on
> > > > part of mod_arch_specific in module_finalize. We would call it only if
> > > the
> > > > flag mentioned above is not set and at the same time we would need to
> > > call
> > > > it when the patch module is being removed.
> > > 
> > > Sigh, I am afraid that the flag is not enough. IMHO, we need to split
> > > the load finalizing functions into two pieces. One will be always
> > > called when the module load is finalized. The other part will free
> > > the load_info. It will be called either when the load is finalized or
> > > when the module is unloaded, depending on if we want to preserve
> > > the load_info.
> > > 
> > > Sigh, it is getting complicated. But let's see how it looks in reality.
> > 
> > At the other end of the spectrum, we could do the simplest thing
> > possible: _always_ save the data (even if CONFIG_LIVEPATCH is disabled).
> > 
> > (gdb) print sizeof(*info)
> > $3 = 96
> > (gdb) p sizeof(*info->hdr)
> > $4 = 64
> > s390 mod_arch_syminfo struct: 24 bytes by my reckoning.
> > 
> > So between info, info->hdr, and s390 mod_arch_syminfo, we're talking
> > about 184 bytes on s390 and 160 bytes on x86_64.  That seems like
> > peanuts compared to the size of a typical module.  The benefit is that
> > the code would be simpler because we don't have any special cases and
> > the structs would automatically get freed with the module struct when
> > the module gets unloaded.

Agreed. mod_arch_specific contains more things on certain architectures, 
but compared to the size of a module it is still not much.

> 
> I think I agree with Josh on this one (except, I would always save
> load_info if it is a livepatch module, instead of for every module in the
> !CONFIG_LIVEPATCH case, and we can just check modinfo to see if it is
> a livepatch module).
> 
> If the tradeoff here is between simplicity and readibility of code vs.
> saving some extra space (and by the looks of it, not a lot), I think I
> would choose having clear code over saving some bytes of memory. Hard
> coding checks and edge cases imo might cause confusion and trouble
> down the road.

I agree this seems like the best approach. So if we preserve 
mod_arch_syminfo (in case of s390) we should free it not in 
module_finalize, but somewhere in free_module... where 
module_arch_cleanup() is called... and also module_arch_freeing_init() is 
called there too. And what you find there for s390 is 

	vfree(mod->arch.syminfo);
	mod->arch.syminfo = NULL;

Well, it does nothing here, because mod->arch.syminfo is already NULL. It 
was freed in module_finalize. So we can even remove this code from 
module_finalize and all should be fine. At least for s390.

As for load_info, I don't have a strong opinion whether to keep it for all 
modules or for livepatch modules only.

Miroslav

Re: module: save load_info for livepatch modules

[Miroslav Benes]

On Thu, 12 Nov 2015, Jessica Yu wrote:

> +++ Josh Poimboeuf [12/11/15 11:05 -0600]:
> > On Thu, Nov 12, 2015 at 04:03:45PM +0100, Petr Mladek wrote:
> > > On Thu 2015-11-12 14:22:28, Miroslav Benes wrote:
> > > > On Thu, 12 Nov 2015, Petr Mladek wrote:
> > > > > > >Maybe I am missing something but isn't it necessary to call vfree()
> > > on
> > > > > > >info somewhere in the end?
> > > > > >
> > > > > > So free_copy() will call vfree(info->hdr), except in livepatch
> > > modules
> > > > > > we want to keep all the elf section information stored there, so we
> > > > > > avoid calling free_copy(), As for the info struct itself, if you
> > > look
> > > > > > at the init_module and finit_module syscall definitions in
> > > > > > kernel/module.c, you will see that info is actually a local function
> > > > > > variable, simply passed in to the call to load_module(), and will be
> > > > > > automatically deallocated when the syscall returns. :-) No need to
> > > > > > explicitly free info.
> > > > >
> > > > > We still have to free the copied or preserved structures when
> > > > > the module is unloaded.
> > > >
> > > > ...freeing what we allocated. We need to free info->hdr somewhere if not
> > > > here and also mod_arch_specific struct where the patch module is
> > > removed.
> > > > This would unfortunately lead to changes in arch-specific code in
> > > > module.c. For example in arch/s390/kernel/module.c there is vfree call
> > > on
> > > > part of mod_arch_specific in module_finalize. We would call it only if
> > > the
> > > > flag mentioned above is not set and at the same time we would need to
> > > call
> > > > it when the patch module is being removed.
> > > 
> > > Sigh, I am afraid that the flag is not enough. IMHO, we need to split
> > > the load finalizing functions into two pieces. One will be always
> > > called when the module load is finalized. The other part will free
> > > the load_info. It will be called either when the load is finalized or
> > > when the module is unloaded, depending on if we want to preserve
> > > the load_info.
> > > 
> > > Sigh, it is getting complicated. But let's see how it looks in reality.
> > 
> > At the other end of the spectrum, we could do the simplest thing
> > possible: _always_ save the data (even if CONFIG_LIVEPATCH is disabled).
> > 
> > (gdb) print sizeof(*info)
> > $3 = 96
> > (gdb) p sizeof(*info->hdr)
> > $4 = 64
> > s390 mod_arch_syminfo struct: 24 bytes by my reckoning.
> > 
> > So between info, info->hdr, and s390 mod_arch_syminfo, we're talking
> > about 184 bytes on s390 and 160 bytes on x86_64.  That seems like
> > peanuts compared to the size of a typical module.  The benefit is that
> > the code would be simpler because we don't have any special cases and
> > the structs would automatically get freed with the module struct when
> > the module gets unloaded.

Agreed. mod_arch_specific contains more things on certain architectures, 
but compared to the size of a module it is still not much.

> 
> I think I agree with Josh on this one (except, I would always save
> load_info if it is a livepatch module, instead of for every module in the
> !CONFIG_LIVEPATCH case, and we can just check modinfo to see if it is
> a livepatch module).
> 
> If the tradeoff here is between simplicity and readibility of code vs.
> saving some extra space (and by the looks of it, not a lot), I think I
> would choose having clear code over saving some bytes of memory. Hard
> coding checks and edge cases imo might cause confusion and trouble
> down the road.

I agree this seems like the best approach. So if we preserve 
mod_arch_syminfo (in case of s390) we should free it not in 
module_finalize, but somewhere in free_module... where 
module_arch_cleanup() is called... and also module_arch_freeing_init() is 
called there too. And what you find there for s390 is 

	vfree(mod->arch.syminfo);
	mod->arch.syminfo = NULL;

Well, it does nothing here, because mod->arch.syminfo is already NULL. It 
was freed in module_finalize. So we can even remove this code from 
module_finalize and all should be fine. At least for s390.

As for load_info, I don't have a strong opinion whether to keep it for all 
modules or for livepatch modules only.

Miroslav

Re: module: save load_info for livepatch modules

[Miroslav Benes]

On Fri, 13 Nov 2015, Miroslav Benes wrote:

> As for load_info, I don't have a strong opinion whether to keep it for all 
> modules or for livepatch modules only.

I have. We cannot keep it, even for livepatch modules...

In info->hdr there is a temporary copy of the whole module (see 
init_module syscall and the first parts of load_module). In load_module 
a final struct module * is created with parts of info->hdr copied (I'll 
get to that later). So if we saved info->hdr for later purposes we would 
just have two copies of the same module in the memory. The original one 
with !SHF_ALLOC sections and everything in vmalloc area, and the new
final copy with SHF_ALLOC sections only. This is not good.

If this is correct (and I think it is after some staring into the code) we 
need to do something different. We should build the info we need for 
delayed relocations from the final copy (or refactor the existing 
module code).

The second problem... dynrela sections need to be marked with SHF_ALLOC 
flag, right? Perhaps it would be better not to do it and copy also 
SHF_RELA_LIVEPATCH sections. It is equivalent but not hidden somewhere 
else (in userspace "kpatch-build" tool).

Miroslav

Re: module: save load_info for livepatch modules

[Miroslav Benes]

On Fri, 13 Nov 2015, Miroslav Benes wrote:

> As for load_info, I don't have a strong opinion whether to keep it for all 
> modules or for livepatch modules only.

I have. We cannot keep it, even for livepatch modules...

In info->hdr there is a temporary copy of the whole module (see 
init_module syscall and the first parts of load_module). In load_module 
a final struct module * is created with parts of info->hdr copied (I'll 
get to that later). So if we saved info->hdr for later purposes we would 
just have two copies of the same module in the memory. The original one 
with !SHF_ALLOC sections and everything in vmalloc area, and the new
final copy with SHF_ALLOC sections only. This is not good.

If this is correct (and I think it is after some staring into the code) we 
need to do something different. We should build the info we need for 
delayed relocations from the final copy (or refactor the existing 
module code).

The second problem... dynrela sections need to be marked with SHF_ALLOC 
flag, right? Perhaps it would be better not to do it and copy also 
SHF_RELA_LIVEPATCH sections. It is equivalent but not hidden somewhere 
else (in userspace "kpatch-build" tool).

Miroslav

Re: module: save load_info for livepatch modules

[Miroslav Benes]

On Fri, 13 Nov 2015, Miroslav Benes wrote:

> I agree this seems like the best approach. So if we preserve 
> mod_arch_syminfo (in case of s390) we should free it not in 
> module_finalize, but somewhere in free_module... where 
> module_arch_cleanup() is called... and also module_arch_freeing_init() is 
> called there too. And what you find there for s390 is 
> 
> 	vfree(mod->arch.syminfo);
> 	mod->arch.syminfo = NULL;
> 
> Well, it does nothing here, because mod->arch.syminfo is already NULL. It 
> was freed in module_finalize. So we can even remove this code from 
> module_finalize and all should be fine. At least for s390.

Which is not true because module_arch_freeing_init is also called from 
do_init_module, called from load_module. So we should move it to 
module_arch_cleanup.

That code is like a maze without Ariadne's thread.

Miroslav

Re: module: save load_info for livepatch modules

[Miroslav Benes]

On Fri, 13 Nov 2015, Miroslav Benes wrote:

> I agree this seems like the best approach. So if we preserve 
> mod_arch_syminfo (in case of s390) we should free it not in 
> module_finalize, but somewhere in free_module... where 
> module_arch_cleanup() is called... and also module_arch_freeing_init() is 
> called there too. And what you find there for s390 is 
> 
> 	vfree(mod->arch.syminfo);
> 	mod->arch.syminfo = NULL;
> 
> Well, it does nothing here, because mod->arch.syminfo is already NULL. It 
> was freed in module_finalize. So we can even remove this code from 
> module_finalize and all should be fine. At least for s390.

Which is not true because module_arch_freeing_init is also called from 
do_init_module, called from load_module. So we should move it to 
module_arch_cleanup.

That code is like a maze without Ariadne's thread.

Miroslav

Re: module: save load_info for livepatch modules

[Miroslav Benes]

On Fri, 13 Nov 2015, Jessica Yu wrote:

> +++ Miroslav Benes [12/11/15 15:19 +0100]:
> > On Thu, 12 Nov 2015, Petr Mladek wrote:
> > 
> > > On Wed 2015-11-11 23:44:08, Jessica Yu wrote:
> > > > +++ Petr Mladek [11/11/15 15:31 +0100]:
> > > > >On Mon 2015-11-09 23:45:52, Jessica Yu wrote:
> > > > >>diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > > > >>index 6e53441..087a8c7 100644
> > > > >>--- a/kernel/livepatch/core.c
> > > > >>+++ b/kernel/livepatch/core.c
> > > > >>@@ -1001,6 +1001,23 @@ static struct notifier_block klp_module_nb = {
> > > > >> 	.priority = INT_MIN+1, /* called late but before ftrace
> > > notifier */
> > > > >> };
> > > > >>
> > > > >>+/*
> > > > >>+ * Save necessary information from info in order to be able to
> > > > >>+ * patch modules that might be loaded later
> > > > >>+ */
> > > > >>+void klp_prepare_patch_module(struct module *mod, struct load_info
> > > *info)
> > > > >>+{
> > > > >>+	Elf_Shdr *symsect;
> > > > >>+
> > > > >>+	symsect = info->sechdrs + info->index.sym;
> > > > >>+	/* update sh_addr to point to symtab */
> > > > >>+	symsect->sh_addr = (unsigned long)info->hdr +
> > > symsect->sh_offset;
> > > > >
> > > > >Is livepatch the only user of this value? By other words, is this safe?
> > > >
> > > > I think it is safe to say yes. klp_prepare_patch_module() is only
> > > > called at the very end of load_module(), right before
> > > > do_init_module(). Normally, at that point, info->hdr will have already
> > > > been freed by free_copy() along with the elf section information
> > > > associated with it. But if we have a livepatch module, we don't free.
> > > > So we should be the very last user, and there should be nobody
> > > > utilizing the memory associated with the load_info struct anymore at
> > > > that point.
> > > 
> > > I see. It looks safe at this point. But still I wonder if it would be
> > > possible to calculate this later in the livepatch code. It will allow
> > > to potentially use the info structure also by other subsystem.
> > > 
> > > BTW: Where is "sh_addr" value used, please? I see it used only
> > > in the third patch as info->sechdrs[relindex].sh_addr. But it is
> > > an array. I am not sure if it is the same variable.
> > 
> > Jessica, why do we need to update sh_addr for symtab? It is not clear to
> > me.
> 
> Ah, I definitely need to make that comment a lot more informative.
> Will make sure to add that in v2. 
> So, the sh_addr field tells us where a certain section is in memory.
> Here, we need to update the symbol table section's sh_addr because if
> we don't, it will eventually point to freed module init memory, which
> is freed in do_init_module(). Let me explain what happens.
> 
> At the beginning of load_module(), the sh_addr fields of each section
> initially point to the vmalloc'd memory within info->hdr (which is
> allocated in copy_module_from_{fd,user}() in module.c). The sh_addr's
> are first assigned in rewrite_section_headers(), called from
> setup_load_info(). These sh_addr's initially just point to an offset
> within info->hdr depending on each section's sh_offset.
> 
> However, in move_module(), where we layout and allocate the memory
> where the module will finally reside, these sh_addr's will get
> reassigned. For the symtab section's sh_addr, it gets reassigned to
> module init memory. (In layout_symtab(), you'll see that the symtab
> section gets marked with INIT_OFFSET_MASK, which indicates that it
> will get an address in init memory when the sh_addr's get reassigned
> in move_module()). Thus the symbol table that simplify_symbols() uses
> is actually in init memory, and will be freed later in
> do_init_module().
> 
> info->hdr is just a temporary holding place for module elf section
> data in memory. Normally, we would get rid of info->hdr and free the
> memory associated with it at the end of the module loading process
> (via free_copy()). But in this patchset, we save all the original elf
> section information because we need it (along with the original
> symtab) in order to make the call to apply_relocate_add(). If you look
> at apply_relocate_add() for x86, s390, etc you'll see that it expects
> a symbol table at the symbol section's sh_addr field (basically, an
> array of Elf_Sym's). This is why we fix up the sh_addr of the symtab
> section to point back to the memory associated with info->hdr (and not
> module init memory). I hope that makes sense.

Great explanation. Thanks.

Only info->hdr makes me worried. See my other mail.

Miroslav

Re: livepatch: reuse module loader code to write relocations

[Miroslav Benes]

On Thu, 12 Nov 2015, Josh Poimboeuf wrote:

> On Thu, Nov 12, 2015 at 03:22:44PM -0500, Jessica Yu wrote:
> > Looking into this more, I think we do need one __klp_rela section per
> > function being patched.  Each rela section is linked to the section to
> > which the relocations apply via the rela section's sh_info field. In
> > SHT_RELA sections, the sh_info field contains the section index to
> > which the relocs apply. We cannot have one single combined rela
> > section per object as the call to apply_relocate_add() simply won't
> > work, because we would have relocs that apply to different functions
> > (and hence different sections).
> > 
> > So I guess instead of a single field in klp_object specifying the
> > __klp_rela section index, we could probably just have an array of
> > section indices.
> 
> Ok, makes sense, sounds like we need multiple klp relas per object.

Ok, it seems so.

> I still don't quite understand the benefit of caching the klp_rela
> section indices.  What problem does it solve?  It seems simpler to just
> iterate over all the sections in klp_write_object_relocations().

It was just my need to be efficient and I think it would have made sense 
with only one dynrela section per object. An array of indices is "ugly" so 
I am all for iteration over all the sections in 
klp_write_object_relocations().

Miroslav

Re: module: save load_info for livepatch modules

[Jessica Yu]

+++ Miroslav Benes [13/11/15 13:46 +0100]:
>On Fri, 13 Nov 2015, Miroslav Benes wrote:
>
>> As for load_info, I don't have a strong opinion whether to keep it for all
>> modules or for livepatch modules only.
>
>I have. We cannot keep it, even for livepatch modules...
>
>In info->hdr there is a temporary copy of the whole module (see
>init_module syscall and the first parts of load_module). In load_module
>a final struct module * is created with parts of info->hdr copied (I'll
>get to that later). So if we saved info->hdr for later purposes we would
>just have two copies of the same module in the memory. The original one
>with !SHF_ALLOC sections and everything in vmalloc area, and the new
>final copy with SHF_ALLOC sections only. This is not good.
>
>If this is correct (and I think it is after some staring into the code) we
>need to do something different. We should build the info we need for
>delayed relocations from the final copy (or refactor the existing
>module code).
>
>The second problem... dynrela sections need to be marked with SHF_ALLOC
>flag, right? Perhaps it would be better not to do it and copy also
>SHF_RELA_LIVEPATCH sections. It is equivalent but not hidden somewhere
>else (in userspace "kpatch-build" tool).

Hm, OK. I understand your concern about leaving a redundant copy of
the module in memory and I agree that we need to do better. I think I
have a solution.

I'm looking at exactly what components we need to make the calls to
apply_relocate_add() work. It's quite simple, I think we only need to
keep the following:

1. A copy of the module's elf section headers i.e. info->sechdrs.
This should be easy to copy. memcpy [info->hdr->e_shnum *
sizeof(Elf_Shdr)] bytes from info->sechdrs. We can maybe put
this in a new field called module->sechdrs.

2. A copy of each __klp_rela section.
If we don't keep info, the current code will discard/not copy the rela
sections over to module core memory since they are !SHF_ALLOC. In
kpatch-build, it is very easy to simply |= the SHF_ALLOC flag with
each __klp_rela section and they will automatically get copied over to
module core memory, and their sh_addr's automatically get reassigned
as well. Thus the klp rela sections will be accessible at
sechdrs[index_of_klpsec].sh_addr. I think this is the easiest solution.

3. A copy of the symbol table. 
Notice that module already has a "symtab" field. In kernels configured
with CONFIG_KALLSYMS, it points to a trimmed down symtab (the
mod->core_symtab) in module core memory. This symtab is not normally
complete; only "core" symbols are kept in it. See add_kallsyms()
(called in post_relocations()) for how core symbols are copied into
this symtab. Then, after the symbols have been copied, module->symtab
is reassigned to point to this core_symtab in do_init_module(). Since
CONFIG_LIVEPATCH requires CONFIG_KALLSYMS, I think we can assume that
mod->symtab will be pointing to mod->core_symtab at the end of the
module load process, since mod->symtab gets assigned to core_symtab in
do_init_module() if CONFIG_KALLSYMS is set.

So for livepatch, what we can do is make sure every symbol in a
livepatch module gets copied into this core symtab. It is important we
keep every symbol since apply_relocate_add() will be using the
original symbol indices. We can implement this by adding a check in
add_kallsyms() to see if we're dealing with a livepatch module. If
yes, just copy all the symbols over.

Then, we will also update Elf_Shdr corresponding to the symbol table
section (sechdrs[symindex].sh_addr) to make sure its sh_addr points to
mod->symtab, so apply_relocate_add() will be able to use it.

4. A copy of mod_arch_specific
I think we discussed this in another email somewhere, but we need to
keep a copy if this somewhere as well. 

So to summarize, keep a copy of sechdrs in module->sechdrs, keep a
copy of mod_arch_specific, mark klp rela sections with SHF_ALLOC,
re-use module->symtab by making sure every symbol gets considered a
"core" symbol and gets copied over. And of course any memory we
allocate (sechdrs, arch stuff) we will free in perhaps free_module()
somewhere.

I haven't implemented it yet but I think it will work, and we don't
need to keep load_info in this scheme. What do you think?

Thanks,
Jessica

Re: module: save load_info for livepatch modules

[Jessica Yu]

+++ Miroslav Benes [13/11/15 13:46 +0100]:
>On Fri, 13 Nov 2015, Miroslav Benes wrote:
>
>> As for load_info, I don't have a strong opinion whether to keep it for all
>> modules or for livepatch modules only.
>
>I have. We cannot keep it, even for livepatch modules...
>
>In info->hdr there is a temporary copy of the whole module (see
>init_module syscall and the first parts of load_module). In load_module
>a final struct module * is created with parts of info->hdr copied (I'll
>get to that later). So if we saved info->hdr for later purposes we would
>just have two copies of the same module in the memory. The original one
>with !SHF_ALLOC sections and everything in vmalloc area, and the new
>final copy with SHF_ALLOC sections only. This is not good.
>
>If this is correct (and I think it is after some staring into the code) we
>need to do something different. We should build the info we need for
>delayed relocations from the final copy (or refactor the existing
>module code).
>
>The second problem... dynrela sections need to be marked with SHF_ALLOC
>flag, right? Perhaps it would be better not to do it and copy also
>SHF_RELA_LIVEPATCH sections. It is equivalent but not hidden somewhere
>else (in userspace "kpatch-build" tool).

Hm, OK. I understand your concern about leaving a redundant copy of
the module in memory and I agree that we need to do better. I think I
have a solution.

I'm looking at exactly what components we need to make the calls to
apply_relocate_add() work. It's quite simple, I think we only need to
keep the following:

1. A copy of the module's elf section headers i.e. info->sechdrs.
This should be easy to copy. memcpy [info->hdr->e_shnum *
sizeof(Elf_Shdr)] bytes from info->sechdrs. We can maybe put
this in a new field called module->sechdrs.

2. A copy of each __klp_rela section.
If we don't keep info, the current code will discard/not copy the rela
sections over to module core memory since they are !SHF_ALLOC. In
kpatch-build, it is very easy to simply |= the SHF_ALLOC flag with
each __klp_rela section and they will automatically get copied over to
module core memory, and their sh_addr's automatically get reassigned
as well. Thus the klp rela sections will be accessible at
sechdrs[index_of_klpsec].sh_addr. I think this is the easiest solution.

3. A copy of the symbol table. 
Notice that module already has a "symtab" field. In kernels configured
with CONFIG_KALLSYMS, it points to a trimmed down symtab (the
mod->core_symtab) in module core memory. This symtab is not normally
complete; only "core" symbols are kept in it. See add_kallsyms()
(called in post_relocations()) for how core symbols are copied into
this symtab. Then, after the symbols have been copied, module->symtab
is reassigned to point to this core_symtab in do_init_module(). Since
CONFIG_LIVEPATCH requires CONFIG_KALLSYMS, I think we can assume that
mod->symtab will be pointing to mod->core_symtab at the end of the
module load process, since mod->symtab gets assigned to core_symtab in
do_init_module() if CONFIG_KALLSYMS is set.

So for livepatch, what we can do is make sure every symbol in a
livepatch module gets copied into this core symtab. It is important we
keep every symbol since apply_relocate_add() will be using the
original symbol indices. We can implement this by adding a check in
add_kallsyms() to see if we're dealing with a livepatch module. If
yes, just copy all the symbols over.

Then, we will also update Elf_Shdr corresponding to the symbol table
section (sechdrs[symindex].sh_addr) to make sure its sh_addr points to
mod->symtab, so apply_relocate_add() will be able to use it.

4. A copy of mod_arch_specific
I think we discussed this in another email somewhere, but we need to
keep a copy if this somewhere as well. 

So to summarize, keep a copy of sechdrs in module->sechdrs, keep a
copy of mod_arch_specific, mark klp rela sections with SHF_ALLOC,
re-use module->symtab by making sure every symbol gets considered a
"core" symbol and gets copied over. And of course any memory we
allocate (sechdrs, arch stuff) we will free in perhaps free_module()
somewhere.

I haven't implemented it yet but I think it will work, and we don't
need to keep load_info in this scheme. What do you think?

Thanks,
Jessica

Re: module: save load_info for livepatch modules

[Jessica Yu]

+++ Miroslav Benes [13/11/15 13:56 +0100]:
>On Fri, 13 Nov 2015, Miroslav Benes wrote:
>
>> I agree this seems like the best approach. So if we preserve
>> mod_arch_syminfo (in case of s390) we should free it not in
>> module_finalize, but somewhere in free_module... where
>> module_arch_cleanup() is called... and also module_arch_freeing_init() is
>> called there too. And what you find there for s390 is
>>
>> 	vfree(mod->arch.syminfo);
>> 	mod->arch.syminfo = NULL;
>>
>> Well, it does nothing here, because mod->arch.syminfo is already NULL. It
>> was freed in module_finalize. So we can even remove this code from
>> module_finalize and all should be fine. At least for s390.
>
>Which is not true because module_arch_freeing_init is also called from
>do_init_module, called from load_module. So we should move it to
>module_arch_cleanup.
>
>That code is like a maze without Ariadne's thread.

Heh, I agree with that sentiment.

I am slightly confused about the s390 code, and whether the authors
originally intended for that double vfree() to happen in both
module_finalize() and module_arch_freeing_init() (called from
do_init_module). Seems like a mistake. If module load succeeds,
do_init_module calls module_arch_freeing_init(). And if load_module
fails halfway through, both module_deallocate() and free_module() will
also call module_arch_freeing_init(). I feel like that vfree should
only happen once in module_arch_freeing_init() and not in
module_finalize(). If we can remove the double vfree() code from
module_finalize(), we can copy the mod_arch_specific safely before the
call to do_init_module().

Jessica

Re: module: save load_info for livepatch modules

[Jessica Yu]

+++ Miroslav Benes [13/11/15 13:56 +0100]:
>On Fri, 13 Nov 2015, Miroslav Benes wrote:
>
>> I agree this seems like the best approach. So if we preserve
>> mod_arch_syminfo (in case of s390) we should free it not in
>> module_finalize, but somewhere in free_module... where
>> module_arch_cleanup() is called... and also module_arch_freeing_init() is
>> called there too. And what you find there for s390 is
>>
>> 	vfree(mod->arch.syminfo);
>> 	mod->arch.syminfo = NULL;
>>
>> Well, it does nothing here, because mod->arch.syminfo is already NULL. It
>> was freed in module_finalize. So we can even remove this code from
>> module_finalize and all should be fine. At least for s390.
>
>Which is not true because module_arch_freeing_init is also called from
>do_init_module, called from load_module. So we should move it to
>module_arch_cleanup.
>
>That code is like a maze without Ariadne's thread.

Heh, I agree with that sentiment.

I am slightly confused about the s390 code, and whether the authors
originally intended for that double vfree() to happen in both
module_finalize() and module_arch_freeing_init() (called from
do_init_module). Seems like a mistake. If module load succeeds,
do_init_module calls module_arch_freeing_init(). And if load_module
fails halfway through, both module_deallocate() and free_module() will
also call module_arch_freeing_init(). I feel like that vfree should
only happen once in module_arch_freeing_init() and not in
module_finalize(). If we can remove the double vfree() code from
module_finalize(), we can copy the mod_arch_specific safely before the
call to do_init_module().

Jessica

Re: module: save load_info for livepatch modules

[Miroslav Benes]

On Fri, 13 Nov 2015, Jessica Yu wrote:

> +++ Miroslav Benes [13/11/15 13:46 +0100]:
> > On Fri, 13 Nov 2015, Miroslav Benes wrote:
> > 
> > > As for load_info, I don't have a strong opinion whether to keep it for all
> > > modules or for livepatch modules only.
> > 
> > I have. We cannot keep it, even for livepatch modules...
> > 
> > In info->hdr there is a temporary copy of the whole module (see
> > init_module syscall and the first parts of load_module). In load_module
> > a final struct module * is created with parts of info->hdr copied (I'll
> > get to that later). So if we saved info->hdr for later purposes we would
> > just have two copies of the same module in the memory. The original one
> > with !SHF_ALLOC sections and everything in vmalloc area, and the new
> > final copy with SHF_ALLOC sections only. This is not good.
> > 
> > If this is correct (and I think it is after some staring into the code) we
> > need to do something different. We should build the info we need for
> > delayed relocations from the final copy (or refactor the existing
> > module code).
> > 
> > The second problem... dynrela sections need to be marked with SHF_ALLOC
> > flag, right? Perhaps it would be better not to do it and copy also
> > SHF_RELA_LIVEPATCH sections. It is equivalent but not hidden somewhere
> > else (in userspace "kpatch-build" tool).
> 
> Hm, OK. I understand your concern about leaving a redundant copy of
> the module in memory and I agree that we need to do better. I think I
> have a solution.
> 
> I'm looking at exactly what components we need to make the calls to
> apply_relocate_add() work. It's quite simple, I think we only need to
> keep the following:
> 
> 1. A copy of the module's elf section headers i.e. info->sechdrs.
> This should be easy to copy. memcpy [info->hdr->e_shnum *
> sizeof(Elf_Shdr)] bytes from info->sechdrs. We can maybe put
> this in a new field called module->sechdrs.

Yes.

> 2. A copy of each __klp_rela section.
> If we don't keep info, the current code will discard/not copy the rela
> sections over to module core memory since they are !SHF_ALLOC. In
> kpatch-build, it is very easy to simply |= the SHF_ALLOC flag with
> each __klp_rela section and they will automatically get copied over to
> module core memory, and their sh_addr's automatically get reassigned
> as well. Thus the klp rela sections will be accessible at
> sechdrs[index_of_klpsec].sh_addr. I think this is the easiest solution.

I agree.

> 3. A copy of the symbol table. Notice that module already has a "symtab"
> field. In kernels configured
> with CONFIG_KALLSYMS, it points to a trimmed down symtab (the
> mod->core_symtab) in module core memory. This symtab is not normally
> complete; only "core" symbols are kept in it. See add_kallsyms()
> (called in post_relocations()) for how core symbols are copied into
> this symtab. Then, after the symbols have been copied, module->symtab
> is reassigned to point to this core_symtab in do_init_module(). Since
> CONFIG_LIVEPATCH requires CONFIG_KALLSYMS, I think we can assume that
> mod->symtab will be pointing to mod->core_symtab at the end of the
> module load process, since mod->symtab gets assigned to core_symtab in
> do_init_module() if CONFIG_KALLSYMS is set.
> 
> So for livepatch, what we can do is make sure every symbol in a
> livepatch module gets copied into this core symtab. It is important we
> keep every symbol since apply_relocate_add() will be using the
> original symbol indices. We can implement this by adding a check in
> add_kallsyms() to see if we're dealing with a livepatch module. If
> yes, just copy all the symbols over.
> 
> Then, we will also update Elf_Shdr corresponding to the symbol table
> section (sechdrs[symindex].sh_addr) to make sure its sh_addr points to
> mod->symtab, so apply_relocate_add() will be able to use it.

It seems like the way (while looking at the code). I think we should do 
the same for strtab. It is also the parameter of apply_relocate_add and 
although it is not directly used in x86 code, it is (albeit for error 
handling) for s390 case. It corresponds to symtab. Also only the core 
symbols are preserved.

> 4. A copy of mod_arch_specific
> I think we discussed this in another email somewhere, but we need to
> keep a copy if this somewhere as well. 
> So to summarize, keep a copy of sechdrs in module->sechdrs, keep a
> copy of mod_arch_specific, mark klp rela sections with SHF_ALLOC,
> re-use module->symtab by making sure every symbol gets considered a
> "core" symbol and gets copied over. And of course any memory we
> allocate (sechdrs, arch stuff) we will free in perhaps free_module()
> somewhere.
> 
> I haven't implemented it yet but I think it will work, and we don't
> need to keep load_info in this scheme. What do you think?

I think it should be work. Great. I think this is the way to go. The only 
thing which needs to be solved is where all needed info should be stored. 
Whether directly in struct module (for example module->sechdrs you 
mentioned) or somewhere in our livepatching code only (klp_patch?). As we 
have already discussed it depends if such functionality could be useful 
also for someone else. I am not sure if we came to a final decision, but 
we inclined to make it general, didn't we?

Thanks,
Miroslav

Re: module: save load_info for livepatch modules

[Miroslav Benes]

On Fri, 13 Nov 2015, Jessica Yu wrote:

> +++ Miroslav Benes [13/11/15 13:46 +0100]:
> > On Fri, 13 Nov 2015, Miroslav Benes wrote:
> > 
> > > As for load_info, I don't have a strong opinion whether to keep it for all
> > > modules or for livepatch modules only.
> > 
> > I have. We cannot keep it, even for livepatch modules...
> > 
> > In info->hdr there is a temporary copy of the whole module (see
> > init_module syscall and the first parts of load_module). In load_module
> > a final struct module * is created with parts of info->hdr copied (I'll
> > get to that later). So if we saved info->hdr for later purposes we would
> > just have two copies of the same module in the memory. The original one
> > with !SHF_ALLOC sections and everything in vmalloc area, and the new
> > final copy with SHF_ALLOC sections only. This is not good.
> > 
> > If this is correct (and I think it is after some staring into the code) we
> > need to do something different. We should build the info we need for
> > delayed relocations from the final copy (or refactor the existing
> > module code).
> > 
> > The second problem... dynrela sections need to be marked with SHF_ALLOC
> > flag, right? Perhaps it would be better not to do it and copy also
> > SHF_RELA_LIVEPATCH sections. It is equivalent but not hidden somewhere
> > else (in userspace "kpatch-build" tool).
> 
> Hm, OK. I understand your concern about leaving a redundant copy of
> the module in memory and I agree that we need to do better. I think I
> have a solution.
> 
> I'm looking at exactly what components we need to make the calls to
> apply_relocate_add() work. It's quite simple, I think we only need to
> keep the following:
> 
> 1. A copy of the module's elf section headers i.e. info->sechdrs.
> This should be easy to copy. memcpy [info->hdr->e_shnum *
> sizeof(Elf_Shdr)] bytes from info->sechdrs. We can maybe put
> this in a new field called module->sechdrs.

Yes.

> 2. A copy of each __klp_rela section.
> If we don't keep info, the current code will discard/not copy the rela
> sections over to module core memory since they are !SHF_ALLOC. In
> kpatch-build, it is very easy to simply |= the SHF_ALLOC flag with
> each __klp_rela section and they will automatically get copied over to
> module core memory, and their sh_addr's automatically get reassigned
> as well. Thus the klp rela sections will be accessible at
> sechdrs[index_of_klpsec].sh_addr. I think this is the easiest solution.

I agree.

> 3. A copy of the symbol table. Notice that module already has a "symtab"
> field. In kernels configured
> with CONFIG_KALLSYMS, it points to a trimmed down symtab (the
> mod->core_symtab) in module core memory. This symtab is not normally
> complete; only "core" symbols are kept in it. See add_kallsyms()
> (called in post_relocations()) for how core symbols are copied into
> this symtab. Then, after the symbols have been copied, module->symtab
> is reassigned to point to this core_symtab in do_init_module(). Since
> CONFIG_LIVEPATCH requires CONFIG_KALLSYMS, I think we can assume that
> mod->symtab will be pointing to mod->core_symtab at the end of the
> module load process, since mod->symtab gets assigned to core_symtab in
> do_init_module() if CONFIG_KALLSYMS is set.
> 
> So for livepatch, what we can do is make sure every symbol in a
> livepatch module gets copied into this core symtab. It is important we
> keep every symbol since apply_relocate_add() will be using the
> original symbol indices. We can implement this by adding a check in
> add_kallsyms() to see if we're dealing with a livepatch module. If
> yes, just copy all the symbols over.
> 
> Then, we will also update Elf_Shdr corresponding to the symbol table
> section (sechdrs[symindex].sh_addr) to make sure its sh_addr points to
> mod->symtab, so apply_relocate_add() will be able to use it.

It seems like the way (while looking at the code). I think we should do 
the same for strtab. It is also the parameter of apply_relocate_add and 
although it is not directly used in x86 code, it is (albeit for error 
handling) for s390 case. It corresponds to symtab. Also only the core 
symbols are preserved.

> 4. A copy of mod_arch_specific
> I think we discussed this in another email somewhere, but we need to
> keep a copy if this somewhere as well. 
> So to summarize, keep a copy of sechdrs in module->sechdrs, keep a
> copy of mod_arch_specific, mark klp rela sections with SHF_ALLOC,
> re-use module->symtab by making sure every symbol gets considered a
> "core" symbol and gets copied over. And of course any memory we
> allocate (sechdrs, arch stuff) we will free in perhaps free_module()
> somewhere.
> 
> I haven't implemented it yet but I think it will work, and we don't
> need to keep load_info in this scheme. What do you think?

I think it should be work. Great. I think this is the way to go. The only 
thing which needs to be solved is where all needed info should be stored. 
Whether directly in struct module (for example module->sechdrs you 
mentioned) or somewhere in our livepatching code only (klp_patch?). As we 
have already discussed it depends if such functionality could be useful 
also for someone else. I am not sure if we came to a final decision, but 
we inclined to make it general, didn't we?

Thanks,
Miroslav

Re: module: save load_info for livepatch modules

[Miroslav Benes]

On Fri, 13 Nov 2015, Jessica Yu wrote:

> +++ Miroslav Benes [13/11/15 13:56 +0100]:
> > On Fri, 13 Nov 2015, Miroslav Benes wrote:
> > 
> > > I agree this seems like the best approach. So if we preserve
> > > mod_arch_syminfo (in case of s390) we should free it not in
> > > module_finalize, but somewhere in free_module... where
> > > module_arch_cleanup() is called... and also module_arch_freeing_init() is
> > > called there too. And what you find there for s390 is
> > > 
> > > 	vfree(mod->arch.syminfo);
> > > 	mod->arch.syminfo = NULL;
> > > 
> > > Well, it does nothing here, because mod->arch.syminfo is already NULL. It
> > > was freed in module_finalize. So we can even remove this code from
> > > module_finalize and all should be fine. At least for s390.
> > 
> > Which is not true because module_arch_freeing_init is also called from
> > do_init_module, called from load_module. So we should move it to
> > module_arch_cleanup.
> > 
> > That code is like a maze without Ariadne's thread.
> 
> Heh, I agree with that sentiment.
> 
> I am slightly confused about the s390 code, and whether the authors
> originally intended for that double vfree() to happen in both
> module_finalize() and module_arch_freeing_init() (called from
> do_init_module). Seems like a mistake. If module load succeeds,
> do_init_module calls module_arch_freeing_init(). And if load_module
> fails halfway through, both module_deallocate() and free_module() will
> also call module_arch_freeing_init(). 

It seems like the authors are aware of this. At least the authors of 
general module code :). See comment in free_module()

 /* This may be NULL, but that's OK */

It is ok, because vfree checks if the pointer is NULL. 

> I feel like that vfree should
> only happen once in module_arch_freeing_init() and not in
> module_finalize(). If we can remove the double vfree() code from
> module_finalize(), we can copy the mod_arch_specific safely before the
> call to do_init_module().

Yes, I think so.

Miroslav

Re: module: save load_info for livepatch modules

[Miroslav Benes]

On Fri, 13 Nov 2015, Jessica Yu wrote:

> +++ Miroslav Benes [13/11/15 13:56 +0100]:
> > On Fri, 13 Nov 2015, Miroslav Benes wrote:
> > 
> > > I agree this seems like the best approach. So if we preserve
> > > mod_arch_syminfo (in case of s390) we should free it not in
> > > module_finalize, but somewhere in free_module... where
> > > module_arch_cleanup() is called... and also module_arch_freeing_init() is
> > > called there too. And what you find there for s390 is
> > > 
> > > 	vfree(mod->arch.syminfo);
> > > 	mod->arch.syminfo = NULL;
> > > 
> > > Well, it does nothing here, because mod->arch.syminfo is already NULL. It
> > > was freed in module_finalize. So we can even remove this code from
> > > module_finalize and all should be fine. At least for s390.
> > 
> > Which is not true because module_arch_freeing_init is also called from
> > do_init_module, called from load_module. So we should move it to
> > module_arch_cleanup.
> > 
> > That code is like a maze without Ariadne's thread.
> 
> Heh, I agree with that sentiment.
> 
> I am slightly confused about the s390 code, and whether the authors
> originally intended for that double vfree() to happen in both
> module_finalize() and module_arch_freeing_init() (called from
> do_init_module). Seems like a mistake. If module load succeeds,
> do_init_module calls module_arch_freeing_init(). And if load_module
> fails halfway through, both module_deallocate() and free_module() will
> also call module_arch_freeing_init(). 

It seems like the authors are aware of this. At least the authors of 
general module code :). See comment in free_module()

 /* This may be NULL, but that's OK */

It is ok, because vfree checks if the pointer is NULL. 

> I feel like that vfree should
> only happen once in module_arch_freeing_init() and not in
> module_finalize(). If we can remove the double vfree() code from
> module_finalize(), we can copy the mod_arch_specific safely before the
> call to do_init_module().

Yes, I think so.

Miroslav