[PATCH v10 00/10] livepatch: Atomic replace feature

[PATCH v10 00/10] livepatch: Atomic replace feature

[Petr Mladek]

The atomic replace allows to create cumulative patches. They
are useful when you maintain many livepatches and want to remove
one that is lower on the stack. In addition it is very useful when
more patches touch the same function and there are dependencies
between them.


Changes against v9:

  + Fixed check of valid NOPs for already loaded objects,
    regression introduced in v9 [Joe, Mirek]
  + Allow to replace even disabled patches [Evgenii]

Changes against v8:

  + Fixed handling of statically defined struct klp_object
    with empty array of functions [Joe, Mirek]
  + Removed redundant func->new_func assignment for NOPs [Mirek]
  + Improved some wording [Mirek]

Changes against v7:

  + Fixed handling of NOPs for not-yet-loaded modules
  + Made klp_replaced_patches list static [Mirek]
  + Made klp_free_object() public later [Mirek]
  + Fixed several reported typos [Mirek, Joe]
  + Updated documentation according to the feedback [Joe]
  + Added some Acks [Mirek]

Changes against v6:

  + used list_move when disabling replaced patches [Jason]
  + renamed KLP_FUNC_ORIGINAL -> KLP_FUNC_STATIC [Mirek]
  + used klp_is_func_type() in klp_unpatch_object() [Mirek]
  + moved static definition of klp_get_or_add_object() [Mirek]
  + updated comment about synchronization in forced mode [Mirek]
  + added user documentation
  + fixed several typos


Jason Baron (5):
  livepatch: Use lists to manage patches, objects and functions
  livepatch: Initial support for dynamic structures
  livepatch: Allow to unpatch only functions of the given type
  livepatch: Support separate list for replaced patches.
  livepatch: Add atomic replace

Petr Mladek (5):
  livepatch: Free only structures with initialized kobject
  livepatch: Correctly handle atomic replace for not yet loaded modules
  livepatch: Improve dynamic struct klp_object detection and
    manipulation
  livepatch: Allow to replace even disabled patches
  livepatch: Atomic replace and cumulative patches documentation

 Documentation/livepatch/cumulative-patches.txt |  83 +++++
 include/linux/livepatch.h                      |  65 +++-
 kernel/livepatch/core.c                        | 422 ++++++++++++++++++++++---
 kernel/livepatch/core.h                        |   4 +
 kernel/livepatch/patch.c                       |  31 +-
 kernel/livepatch/patch.h                       |   4 +-
 kernel/livepatch/transition.c                  |  41 ++-
 7 files changed, 598 insertions(+), 52 deletions(-)
 create mode 100644 Documentation/livepatch/cumulative-patches.txt

-- 
2.13.6

[PATCH v10 01/10] livepatch: Use lists to manage patches, objects and functions

[Petr Mladek]

From: Jason Baron <jbaron@akamai.com>

Currently klp_patch contains a pointer to a statically allocated array of
struct klp_object and struct klp_objects contains a pointer to a statically
allocated array of klp_func. In order to allow for the dynamic allocation
of objects and functions, link klp_patch, klp_object, and klp_func together
via linked lists. This allows us to more easily allocate new objects and
functions, while having the iterator be a simple linked list walk.

The static structures are added to the lists early. It allows to add
the dynamically allocated objects before klp_init_object() and
klp_init_func() calls. Therefore it reduces the further changes
to the code.

Also klp_init_*_list() functions are split because they will
be used when adding the dynamically allocated structures.

This patch does not change the existing behavior.

Signed-off-by: Jason Baron <jbaron@akamai.com>
[pmladek@suse.com: Initialize lists before init calls]
Signed-off-by: Petr Mladek <pmladek@suse.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Jessica Yu <jeyu@kernel.org>
Cc: Jiri Kosina <jikos@kernel.org>
Acked-by: Miroslav Benes <mbenes@suse.cz>
---
 include/linux/livepatch.h | 19 +++++++++++++++++--
 kernel/livepatch/core.c   | 27 +++++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 2 deletions(-)

diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
index 4754f01c1abb..e5db2ba7e2a5 100644
--- a/include/linux/livepatch.h
+++ b/include/linux/livepatch.h
@@ -24,6 +24,7 @@
 #include <linux/module.h>
 #include <linux/ftrace.h>
 #include <linux/completion.h>
+#include <linux/list.h>
 
 #if IS_ENABLED(CONFIG_LIVEPATCH)
 
@@ -43,6 +44,7 @@
  * @old_addr:	the address of the function being patched
  * @kobj:	kobject for sysfs resources
  * @stack_node:	list node for klp_ops func_stack list
+ * @func_entry:	links struct klp_func to struct klp_object
  * @old_size:	size of the old function
  * @new_size:	size of the new function
  * @patched:	the func has been added to the klp_ops list
@@ -80,6 +82,7 @@ struct klp_func {
 	unsigned long old_addr;
 	struct kobject kobj;
 	struct list_head stack_node;
+	struct list_head func_entry;
 	unsigned long old_size, new_size;
 	bool patched;
 	bool transition;
@@ -117,6 +120,8 @@ struct klp_callbacks {
  * @kobj:	kobject for sysfs resources
  * @mod:	kernel module associated with the patched object
  *		(NULL for vmlinux)
+ * @func_list:	head of list for struct klp_func
+ * @obj_entry:	links struct klp_object to struct klp_patch
  * @patched:	the object's funcs have been added to the klp_ops list
  */
 struct klp_object {
@@ -127,6 +132,8 @@ struct klp_object {
 
 	/* internal */
 	struct kobject kobj;
+	struct list_head func_list;
+	struct list_head obj_entry;
 	struct module *mod;
 	bool patched;
 };
@@ -137,6 +144,7 @@ struct klp_object {
  * @objs:	object entries for kernel objects to be patched
  * @list:	list node for global list of registered patches
  * @kobj:	kobject for sysfs resources
+ * @obj_list:	head of list for struct klp_object
  * @enabled:	the patch is enabled (but operation may be incomplete)
  * @finish:	for waiting till it is safe to remove the patch module
  */
@@ -148,18 +156,25 @@ struct klp_patch {
 	/* internal */
 	struct list_head list;
 	struct kobject kobj;
+	struct list_head obj_list;
 	bool enabled;
 	struct completion finish;
 };
 
-#define klp_for_each_object(patch, obj) \
+#define klp_for_each_object_static(patch, obj) \
 	for (obj = patch->objs; obj->funcs || obj->name; obj++)
 
-#define klp_for_each_func(obj, func) \
+#define klp_for_each_object(patch, obj)	\
+	list_for_each_entry(obj, &patch->obj_list, obj_entry)
+
+#define klp_for_each_func_static(obj, func) \
 	for (func = obj->funcs; \
 	     func->old_name || func->new_func || func->old_sympos; \
 	     func++)
 
+#define klp_for_each_func(obj, func)	\
+	list_for_each_entry(func, &obj->func_list, func_entry)
+
 int klp_register_patch(struct klp_patch *);
 int klp_unregister_patch(struct klp_patch *);
 int klp_enable_patch(struct klp_patch *);
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 3a4656fb7047..1d525f4a270a 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -49,6 +49,32 @@ static LIST_HEAD(klp_patches);
 
 static struct kobject *klp_root_kobj;
 
+static void klp_init_func_list(struct klp_object *obj, struct klp_func *func)
+{
+	list_add(&func->func_entry, &obj->func_list);
+}
+
+static void klp_init_object_list(struct klp_patch *patch,
+				 struct klp_object *obj)
+{
+	struct klp_func *func;
+
+	list_add(&obj->obj_entry, &patch->obj_list);
+
+	INIT_LIST_HEAD(&obj->func_list);
+	klp_for_each_func_static(obj, func)
+		klp_init_func_list(obj, func);
+}
+
+static void klp_init_patch_list(struct klp_patch *patch)
+{
+	struct klp_object *obj;
+
+	INIT_LIST_HEAD(&patch->obj_list);
+	klp_for_each_object_static(patch, obj)
+		klp_init_object_list(patch, obj);
+}
+
 static bool klp_is_module(struct klp_object *obj)
 {
 	return obj->name;
@@ -794,6 +820,7 @@ static int klp_init_patch(struct klp_patch *patch)
 
 	patch->enabled = false;
 	init_completion(&patch->finish);
+	klp_init_patch_list(patch);
 
 	ret = kobject_init_and_add(&patch->kobj, &klp_ktype_patch,
 				   klp_root_kobj, "%s", patch->mod->name);
-- 
2.13.6

[PATCH v10 02/10] livepatch: Free only structures with initialized kobject

[Petr Mladek]

We are going to add a feature called atomic replace. It will allow to
create a patch that would replace all already registered patches.
For this, we will need to dynamically create funcs and objects
for functions that are no longer patched.

We will want to reuse the existing init() and free() functions. Up to now,
the free() functions checked a limit and were called only for structures
with initialized kobject. But we will want to call them also for structures
that were allocated but where the kobject was not initialized yet.

This patch removes the limit. It calls klp_free*() functions for all
structures. But only the ones with initialized kobject are freed.
The handling of un-initialized structures will be added later with
the support for dynamic structures.

This patch does not change the existing behavior.

Signed-off-by: Petr Mladek <pmladek@suse.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Jessica Yu <jeyu@kernel.org>
Cc: Jiri Kosina <jikos@kernel.org>
Cc: Jason Baron <jbaron@akamai.com>
Acked-by: Miroslav Benes <mbenes@suse.cz>
---
 kernel/livepatch/core.c | 44 ++++++++++++++++++--------------------------
 1 file changed, 18 insertions(+), 26 deletions(-)

diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 1d525f4a270a..69bde95e76f8 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -653,17 +653,15 @@ static struct kobj_type klp_ktype_func = {
 	.sysfs_ops = &kobj_sysfs_ops,
 };
 
-/*
- * Free all functions' kobjects in the array up to some limit. When limit is
- * NULL, all kobjects are freed.
- */
-static void klp_free_funcs_limited(struct klp_object *obj,
-				   struct klp_func *limit)
+/* Free all funcs that have the kobject initialized. */
+static void klp_free_funcs(struct klp_object *obj)
 {
 	struct klp_func *func;
 
-	for (func = obj->funcs; func->old_name && func != limit; func++)
-		kobject_put(&func->kobj);
+	klp_for_each_func(obj, func) {
+		if (func->kobj.state_initialized)
+			kobject_put(&func->kobj);
+	}
 }
 
 /* Clean up when a patched object is unloaded */
@@ -677,24 +675,23 @@ static void klp_free_object_loaded(struct klp_object *obj)
 		func->old_addr = 0;
 }
 
-/*
- * Free all objects' kobjects in the array up to some limit. When limit is
- * NULL, all kobjects are freed.
- */
-static void klp_free_objects_limited(struct klp_patch *patch,
-				     struct klp_object *limit)
+/* Free all funcs and objects that have the kobject initialized. */
+static void klp_free_objects(struct klp_patch *patch)
 {
 	struct klp_object *obj;
 
-	for (obj = patch->objs; obj->funcs && obj != limit; obj++) {
-		klp_free_funcs_limited(obj, NULL);
-		kobject_put(&obj->kobj);
+	klp_for_each_object(patch, obj) {
+		klp_free_funcs(obj);
+
+		if (obj->kobj.state_initialized)
+			kobject_put(&obj->kobj);
 	}
 }
 
 static void klp_free_patch(struct klp_patch *patch)
 {
-	klp_free_objects_limited(patch, NULL);
+	klp_free_objects(patch);
+
 	if (!list_empty(&patch->list))
 		list_del(&patch->list);
 }
@@ -791,21 +788,16 @@ static int klp_init_object(struct klp_patch *patch, struct klp_object *obj)
 	klp_for_each_func(obj, func) {
 		ret = klp_init_func(obj, func);
 		if (ret)
-			goto free;
+			return ret;
 	}
 
 	if (klp_is_object_loaded(obj)) {
 		ret = klp_init_object_loaded(patch, obj);
 		if (ret)
-			goto free;
+			return ret;
 	}
 
 	return 0;
-
-free:
-	klp_free_funcs_limited(obj, func);
-	kobject_put(&obj->kobj);
-	return ret;
 }
 
 static int klp_init_patch(struct klp_patch *patch)
@@ -842,7 +834,7 @@ static int klp_init_patch(struct klp_patch *patch)
 	return 0;
 
 free:
-	klp_free_objects_limited(patch, obj);
+	klp_free_objects(patch);
 
 	mutex_unlock(&klp_mutex);
 
-- 
2.13.6

[PATCH v10 03/10] livepatch: Initial support for dynamic structures

[Petr Mladek]

From: Jason Baron <jbaron@akamai.com>

We are going to add a feature called atomic replace. It will allow to
create a patch that would replace all already registered patches.
For this, we will need to dynamically create funcs and objects
for functions that are no longer patched.

This patch adds basic framework to handle such dynamic structures.

It adds enum klp_func_type that allows to distinguish the dynamically
allocated funcs' structures. Note that objects' structures do not have
a clear type. Namely the static objects' structures might list both static
and dynamic funcs' structures.

The function type is then used to limit klp_free() functions. We will
want to free the dynamic structures separately when they are no longer
needed. At the same time, we also want to make our life easier,
and introduce _any_ type that will allow to process all existing
structures in one go.

We need to be careful here. First, objects' structures must be freed
only when all listed funcs' structures are freed. Also we must avoid
double free. Both problems are solved by removing the freed structures
from the list.

Also note that klp_free*() functions are called also in klp_init_patch()
error path when only some kobjects have been initialized. The other
dynamic structures must be freed immediately by calling the respective
klp_free_*_dynamic() functions.

Finally, the dynamic objects' structures are generic. The respective
klp_allocate_object_dynamic() and klp_free_object_dynamic() can
be implemented here. On the other hand, klp_free_func_dynamic()
is empty. It must be updated when a particular dynamic
klp_func_type is introduced.

Signed-off-by: Jason Baron <jbaron@akamai.com>
[pmladek@suse.com: Converted into a generic API]
Signed-off-by: Petr Mladek <pmladek@suse.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Jessica Yu <jeyu@kernel.org>
Cc: Jiri Kosina <jikos@kernel.org>
Acked-by: Miroslav Benes <mbenes@suse.cz>
---
 include/linux/livepatch.h |  37 +++++++++++-
 kernel/livepatch/core.c   | 141 +++++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 163 insertions(+), 15 deletions(-)

diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
index e5db2ba7e2a5..7fb76e7d2693 100644
--- a/include/linux/livepatch.h
+++ b/include/linux/livepatch.h
@@ -35,12 +35,22 @@
 #define KLP_UNPATCHED	 0
 #define KLP_PATCHED	 1
 
+/*
+ * Function type is used to distinguish dynamically allocated structures
+ * and limit some operations.
+ */
+enum klp_func_type {
+	KLP_FUNC_ANY = -1,	/* Substitute any type */
+	KLP_FUNC_STATIC = 0,    /* Original statically defined structure */
+};
+
 /**
  * struct klp_func - function structure for live patching
  * @old_name:	name of the function to be patched
  * @new_func:	pointer to the patched function code
  * @old_sympos: a hint indicating which symbol position the old function
  *		can be found (optional)
+ * @ftype:	distinguish static and dynamic structures
  * @old_addr:	the address of the function being patched
  * @kobj:	kobject for sysfs resources
  * @stack_node:	list node for klp_ops func_stack list
@@ -79,6 +89,7 @@ struct klp_func {
 	unsigned long old_sympos;
 
 	/* internal */
+	enum klp_func_type ftype;
 	unsigned long old_addr;
 	struct kobject kobj;
 	struct list_head stack_node;
@@ -164,17 +175,41 @@ struct klp_patch {
 #define klp_for_each_object_static(patch, obj) \
 	for (obj = patch->objs; obj->funcs || obj->name; obj++)
 
+#define klp_for_each_object_safe(patch, obj, tmp_obj)		\
+	list_for_each_entry_safe(obj, tmp_obj, &patch->obj_list, obj_entry)
+
 #define klp_for_each_object(patch, obj)	\
 	list_for_each_entry(obj, &patch->obj_list, obj_entry)
 
+/* Support also dynamically allocated struct klp_object */
 #define klp_for_each_func_static(obj, func) \
 	for (func = obj->funcs; \
-	     func->old_name || func->new_func || func->old_sympos; \
+	     func && (func->old_name || func->new_func || func->old_sympos); \
 	     func++)
 
+#define klp_for_each_func_safe(obj, func, tmp_func)			\
+	list_for_each_entry_safe(func, tmp_func, &obj->func_list, func_entry)
+
 #define klp_for_each_func(obj, func)	\
 	list_for_each_entry(func, &obj->func_list, func_entry)
 
+static inline bool klp_is_object_dynamic(struct klp_object *obj)
+{
+	return !obj->funcs;
+}
+
+static inline bool klp_is_func_dynamic(struct klp_func *func)
+{
+	WARN_ON_ONCE(func->ftype == KLP_FUNC_ANY);
+	return func->ftype != KLP_FUNC_STATIC;
+}
+
+static inline bool klp_is_func_type(struct klp_func *func,
+				    enum klp_func_type ftype)
+{
+	return ftype == KLP_FUNC_ANY || ftype == func->ftype;
+}
+
 int klp_register_patch(struct klp_patch *);
 int klp_unregister_patch(struct klp_patch *);
 int klp_enable_patch(struct klp_patch *);
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 69bde95e76f8..a47c26147c17 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -124,6 +124,26 @@ static bool klp_initialized(void)
 	return !!klp_root_kobj;
 }
 
+static struct klp_object *klp_find_object(struct klp_patch *patch,
+					  struct klp_object *old_obj)
+{
+	struct klp_object *obj;
+	bool mod = klp_is_module(old_obj);
+
+	klp_for_each_object(patch, obj) {
+		if (mod) {
+			if (klp_is_module(obj) &&
+			    strcmp(old_obj->name, obj->name) == 0) {
+				return obj;
+			}
+		} else if (!klp_is_module(obj)) {
+			return obj;
+		}
+	}
+
+	return NULL;
+}
+
 struct klp_find_arg {
 	const char *objname;
 	const char *name;
@@ -621,6 +641,66 @@ static struct attribute *klp_patch_attrs[] = {
 	NULL
 };
 
+/*
+ * Dynamically allocated objects and functions.
+ */
+static void klp_free_func_dynamic(struct klp_func *func)
+{
+}
+
+static void klp_free_object_dynamic(struct klp_object *obj)
+{
+	kfree(obj->name);
+	kfree(obj);
+}
+
+static struct klp_object *klp_alloc_object_dynamic(const char *name)
+{
+	struct klp_object *obj;
+
+	obj = kzalloc(sizeof(*obj), GFP_KERNEL);
+	if (!obj)
+		return ERR_PTR(-ENOMEM);
+
+	if (name) {
+		obj->name = kstrdup(name, GFP_KERNEL);
+		if (!obj->name) {
+			kfree(obj);
+			return ERR_PTR(-ENOMEM);
+		}
+	}
+
+	return obj;
+}
+
+static struct klp_object *klp_get_or_add_object(struct klp_patch *patch,
+						struct klp_object *old_obj)
+{
+	struct klp_object *obj;
+
+	obj = klp_find_object(patch, old_obj);
+	if (obj)
+		return obj;
+
+	obj = klp_alloc_object_dynamic(old_obj->name);
+	if (IS_ERR(obj))
+		return obj;
+
+	klp_init_object_list(patch, obj);
+	return obj;
+}
+
+/*
+ * Patch release framework must support the following scenarios:
+ *
+ *   + Asynchonous release is used when kobjects are initialized.
+ *
+ *   + Direct release is used in error paths for structures that
+ *     have not had kobj initialized yet.
+ *
+ *   + Allow to release dynamic structures of the given type when
+ *     they are not longer needed.
+ */
 static void klp_kobj_release_patch(struct kobject *kobj)
 {
 	struct klp_patch *patch;
@@ -637,6 +717,12 @@ static struct kobj_type klp_ktype_patch = {
 
 static void klp_kobj_release_object(struct kobject *kobj)
 {
+	struct klp_object *obj;
+
+	obj = container_of(kobj, struct klp_object, kobj);
+
+	if (klp_is_object_dynamic(obj))
+		klp_free_object_dynamic(obj);
 }
 
 static struct kobj_type klp_ktype_object = {
@@ -646,6 +732,12 @@ static struct kobj_type klp_ktype_object = {
 
 static void klp_kobj_release_func(struct kobject *kobj)
 {
+	struct klp_func *func;
+
+	func = container_of(kobj, struct klp_func, kobj);
+
+	if (klp_is_func_dynamic(func))
+		klp_free_func_dynamic(func);
 }
 
 static struct kobj_type klp_ktype_func = {
@@ -653,14 +745,26 @@ static struct kobj_type klp_ktype_func = {
 	.sysfs_ops = &kobj_sysfs_ops,
 };
 
-/* Free all funcs that have the kobject initialized. */
-static void klp_free_funcs(struct klp_object *obj)
+/*
+ * Free all funcs of the given ftype. Use the kobject when it has already
+ * been initialized. Otherwise, do it directly.
+ */
+static void klp_free_funcs(struct klp_object *obj,
+			   enum klp_func_type ftype)
 {
-	struct klp_func *func;
+	struct klp_func *func, *tmp_func;
+
+	klp_for_each_func_safe(obj, func, tmp_func) {
+		if (!klp_is_func_type(func, ftype))
+			continue;
+
+		/* Avoid double free and allow to detect empty objects. */
+		list_del(&func->func_entry);
 
-	klp_for_each_func(obj, func) {
 		if (func->kobj.state_initialized)
 			kobject_put(&func->kobj);
+		else if (klp_is_func_dynamic(func))
+			klp_free_func_dynamic(func);
 	}
 }
 
@@ -675,22 +779,34 @@ static void klp_free_object_loaded(struct klp_object *obj)
 		func->old_addr = 0;
 }
 
-/* Free all funcs and objects that have the kobject initialized. */
-static void klp_free_objects(struct klp_patch *patch)
+/*
+ * Free all linked funcs of the given ftype. Then free empty objects.
+ * Use the kobject when it has already been initialized. Otherwise,
+ * do it directly.
+ */
+static void klp_free_objects(struct klp_patch *patch, enum klp_func_type ftype)
 {
-	struct klp_object *obj;
+	struct klp_object *obj, *tmp_obj;
 
-	klp_for_each_object(patch, obj) {
-		klp_free_funcs(obj);
+	klp_for_each_object_safe(patch, obj, tmp_obj) {
+		klp_free_funcs(obj, ftype);
+
+		if (!list_empty(&obj->func_list))
+			continue;
+
+		/* Avoid freeing the object twice. */
+		list_del(&obj->obj_entry);
 
 		if (obj->kobj.state_initialized)
 			kobject_put(&obj->kobj);
+		else if (klp_is_object_dynamic(obj))
+			klp_free_object_dynamic(obj);
 	}
 }
 
 static void klp_free_patch(struct klp_patch *patch)
 {
-	klp_free_objects(patch);
+	klp_free_objects(patch, KLP_FUNC_ANY);
 
 	if (!list_empty(&patch->list))
 		list_del(&patch->list);
@@ -771,9 +887,6 @@ static int klp_init_object(struct klp_patch *patch, struct klp_object *obj)
 	int ret;
 	const char *name;
 
-	if (!obj->funcs)
-		return -EINVAL;
-
 	obj->patched = false;
 	obj->mod = NULL;
 
@@ -834,7 +947,7 @@ static int klp_init_patch(struct klp_patch *patch)
 	return 0;
 
 free:
-	klp_free_objects(patch);
+	klp_free_objects(patch, KLP_FUNC_ANY);
 
 	mutex_unlock(&klp_mutex);
 
-- 
2.13.6

[PATCH v10 04/10] livepatch: Allow to unpatch only functions of the given type

[Petr Mladek]

From: Jason Baron <jbaron@akamai.com>

We are going to add a feature called atomic replace. It will allow to
create a patch that would replace all already registered patches.
For this, we will need to dynamically create funcs and objects
for functions that are no longer patched.

The dynamically allocated objects will not longer be needed once
the patch is applied.

This patch allows to unpatch functions of the given type. It might
cause that the obj->patched flag is true even when some listed
functions are not longer patched. This is fine as long as the
unpatched funcs' structures are removed right after. It will
be the case. Anyway, it is safe. In the worst case, it will
not be possible to enable the disabled functions.

Signed-off-by: Jason Baron <jbaron@akamai.com>
[pmladek@suse.com: Split and modified to use the generic ftype]
Signed-off-by: Petr Mladek <pmladek@suse.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Jessica Yu <jeyu@kernel.org>
Cc: Jiri Kosina <jikos@kernel.org>
Acked-by: Miroslav Benes <mbenes@suse.cz>
---
 kernel/livepatch/core.c       |  2 +-
 kernel/livepatch/patch.c      | 26 +++++++++++++++++++-------
 kernel/livepatch/patch.h      |  4 ++--
 kernel/livepatch/transition.c |  2 +-
 4 files changed, 23 insertions(+), 11 deletions(-)

diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index a47c26147c17..ab1f6a371fc8 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -1060,7 +1060,7 @@ static void klp_cleanup_module_patches_limited(struct module *mod,
 
 				pr_notice("reverting patch '%s' on unloading module '%s'\n",
 					  patch->mod->name, obj->mod->name);
-				klp_unpatch_object(obj);
+				klp_unpatch_object(obj, KLP_FUNC_ANY);
 
 				klp_post_unpatch_callback(obj);
 			}
diff --git a/kernel/livepatch/patch.c b/kernel/livepatch/patch.c
index 82d584225dc6..54b3c379bb0f 100644
--- a/kernel/livepatch/patch.c
+++ b/kernel/livepatch/patch.c
@@ -236,15 +236,27 @@ static int klp_patch_func(struct klp_func *func)
 	return ret;
 }
 
-void klp_unpatch_object(struct klp_object *obj)
+/*
+ * It keeps obj->patched flag true when any listed function is still patched.
+ * The caller is responsible for removing the unpatched functions to
+ * make the flag clean again.
+ */
+void klp_unpatch_object(struct klp_object *obj, enum klp_func_type ftype)
 {
 	struct klp_func *func;
+	bool patched = false;
 
-	klp_for_each_func(obj, func)
-		if (func->patched)
+	klp_for_each_func(obj, func) {
+		if (!func->patched)
+			continue;
+
+		if (klp_is_func_type(func, ftype))
 			klp_unpatch_func(func);
+		else
+			patched = true;
+	}
 
-	obj->patched = false;
+	obj->patched = patched;
 }
 
 int klp_patch_object(struct klp_object *obj)
@@ -258,7 +270,7 @@ int klp_patch_object(struct klp_object *obj)
 	klp_for_each_func(obj, func) {
 		ret = klp_patch_func(func);
 		if (ret) {
-			klp_unpatch_object(obj);
+			klp_unpatch_object(obj, KLP_FUNC_ANY);
 			return ret;
 		}
 	}
@@ -267,11 +279,11 @@ int klp_patch_object(struct klp_object *obj)
 	return 0;
 }
 
-void klp_unpatch_objects(struct klp_patch *patch)
+void klp_unpatch_objects(struct klp_patch *patch, enum klp_func_type ftype)
 {
 	struct klp_object *obj;
 
 	klp_for_each_object(patch, obj)
 		if (obj->patched)
-			klp_unpatch_object(obj);
+			klp_unpatch_object(obj, ftype);
 }
diff --git a/kernel/livepatch/patch.h b/kernel/livepatch/patch.h
index e72d8250d04b..885f644add4c 100644
--- a/kernel/livepatch/patch.h
+++ b/kernel/livepatch/patch.h
@@ -28,7 +28,7 @@ struct klp_ops {
 struct klp_ops *klp_find_ops(unsigned long old_addr);
 
 int klp_patch_object(struct klp_object *obj);
-void klp_unpatch_object(struct klp_object *obj);
-void klp_unpatch_objects(struct klp_patch *patch);
+void klp_unpatch_object(struct klp_object *obj, enum klp_func_type ftype);
+void klp_unpatch_objects(struct klp_patch *patch, enum klp_func_type ftype);
 
 #endif /* _LIVEPATCH_PATCH_H */
diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
index 7c6631e693bc..6917100fbe79 100644
--- a/kernel/livepatch/transition.c
+++ b/kernel/livepatch/transition.c
@@ -92,7 +92,7 @@ static void klp_complete_transition(void)
 		 * All tasks have transitioned to KLP_UNPATCHED so we can now
 		 * remove the new functions from the func_stack.
 		 */
-		klp_unpatch_objects(klp_transition_patch);
+		klp_unpatch_objects(klp_transition_patch, KLP_FUNC_ANY);
 
 		/*
 		 * Make sure klp_ftrace_handler() can no longer see functions
-- 
2.13.6

[PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Petr Mladek]

From: Jason Baron <jbaron@akamai.com>

We are going to add a feature called atomic replace. It will allow to
create a patch that would replace all already registered patches.

The replaced patches will stay registered because they are typically
unregistered by some package uninstall scripts. But we will remove
these patches from @klp_patches list to keep the enabled patch
on the bottom of the stack. Otherwise, we would need to implement
rather complex logic for moving the patches on the stack. Also
it would complicate implementation of the atomic replace feature.
It is not worth it.

As a result, we will have patches that are registered but that
are no longer usable. Let's get prepared for this and use
a better descriptive name for klp_is_patch_registered() function.

Also create separate list for the replaced patches and allow to
unregister them. Alternative solution would be to add a flag
into struct klp_patch. Note that patch->kobj.state_initialized
is not safe because it can be cleared outside klp_mutex.

This patch does not change the existing behavior.

Signed-off-by: Jason Baron <jbaron@akamai.com>
[pmladek@suse.com: Split and renamed klp_is_patch_usable()]
Signed-off-by: Petr Mladek <pmladek@suse.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Jessica Yu <jeyu@kernel.org>
Cc: Jiri Kosina <jikos@kernel.org>
Acked-by: Miroslav Benes <mbenes@suse.cz>
---
 kernel/livepatch/core.c | 30 ++++++++++++++++++++++++------
 1 file changed, 24 insertions(+), 6 deletions(-)

diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index ab1f6a371fc8..fd0296859ff4 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -47,6 +47,13 @@ DEFINE_MUTEX(klp_mutex);
 
 static LIST_HEAD(klp_patches);
 
+/*
+ * List of 'replaced' patches that have been replaced by a patch that has the
+ * 'replace' bit set. When they are added to this list, they are disabled and
+ * can not be re-enabled, but they can be unregistered().
+ */
+static LIST_HEAD(klp_replaced_patches);
+
 static struct kobject *klp_root_kobj;
 
 static void klp_init_func_list(struct klp_object *obj, struct klp_func *func)
@@ -108,17 +115,28 @@ static void klp_find_object_module(struct klp_object *obj)
 	mutex_unlock(&module_mutex);
 }
 
-static bool klp_is_patch_registered(struct klp_patch *patch)
+static bool klp_is_patch_in_list(struct klp_patch *patch,
+				 struct list_head *head)
 {
 	struct klp_patch *mypatch;
 
-	list_for_each_entry(mypatch, &klp_patches, list)
+	list_for_each_entry(mypatch, head, list)
 		if (mypatch == patch)
 			return true;
 
 	return false;
 }
 
+static bool klp_is_patch_usable(struct klp_patch *patch)
+{
+	return klp_is_patch_in_list(patch, &klp_patches);
+}
+
+static bool klp_is_patch_replaced(struct klp_patch *patch)
+{
+	return klp_is_patch_in_list(patch, &klp_replaced_patches);
+}
+
 static bool klp_initialized(void)
 {
 	return !!klp_root_kobj;
@@ -375,7 +393,7 @@ int klp_disable_patch(struct klp_patch *patch)
 
 	mutex_lock(&klp_mutex);
 
-	if (!klp_is_patch_registered(patch)) {
+	if (!klp_is_patch_usable(patch)) {
 		ret = -EINVAL;
 		goto err;
 	}
@@ -475,7 +493,7 @@ int klp_enable_patch(struct klp_patch *patch)
 
 	mutex_lock(&klp_mutex);
 
-	if (!klp_is_patch_registered(patch)) {
+	if (!klp_is_patch_usable(patch)) {
 		ret = -EINVAL;
 		goto err;
 	}
@@ -516,7 +534,7 @@ static ssize_t enabled_store(struct kobject *kobj, struct kobj_attribute *attr,
 
 	mutex_lock(&klp_mutex);
 
-	if (!klp_is_patch_registered(patch)) {
+	if (!klp_is_patch_usable(patch)) {
 		/*
 		 * Module with the patch could either disappear meanwhile or is
 		 * not properly initialized yet.
@@ -971,7 +989,7 @@ int klp_unregister_patch(struct klp_patch *patch)
 
 	mutex_lock(&klp_mutex);
 
-	if (!klp_is_patch_registered(patch)) {
+	if (!klp_is_patch_usable(patch) && !klp_is_patch_replaced(patch)) {
 		ret = -EINVAL;
 		goto err;
 	}
-- 
2.13.6

[PATCH v10 06/10] livepatch: Add atomic replace

[Petr Mladek]

From: Jason Baron <jbaron@akamai.com>

Sometimes we would like to revert a particular fix. Currently, this
is not easy because we want to keep all other fixes active and we
could revert only the last applied patch.

One solution would be to apply new patch that implemented all
the reverted functions like in the original code. It would work
as expected but there will be unnecessary redirections. In addition,
it would also require knowing which functions need to be reverted at
build time.

Another problem is when there are many patches that touch the same
functions. There might be dependencies between patches that are
not enforced on the kernel side. Also it might be pretty hard to
actually prepare the patch and ensure compatibility with
the other patches.

A better solution would be to create cumulative patch and say that
it replaces all older ones.

This patch adds a new "replace" flag to struct klp_patch. When it is
enabled, a set of 'nop' klp_func will be dynamically created for all
functions that are already being patched but that will no longer be
modified by the new patch. They are temporarily used as a new target
during the patch transition.

There are used several simplifications:

  + nops' structures are generated already when the patch is registered.
    All registered patches are taken into account, even the disabled ones.
    As a result, there might be more nops than are really needed when
    the patch is enabled and some disabled patches were removed before.
    But we are on the safe side and it simplifies the implementation.
    Especially we could reuse the existing init() functions. Also freeing
    is easier because the same nops are created and removed only once.

    Alternative solution would be to create nops when the patch is enabled.
    But then any reusing of the init() functions and error paths would be
    complicated. Also it would increase the risk of errors because of
    late kobject initialization. Finally, it would need tricky waiting
    for freed kobjects when finalizing a reverted enable transaction.

  + The replaced patches are removed from the stack and cannot longer
    be enabled directly. Otherwise, we would need to implement a more
    complex logic of handling the stack of patches. It might be hard
    to come with a reasonable semantic.

    A fallback is to remove (rmmod) the replaced patches and register
    (insmod) them again.

  + Nops are handled like normal function patches. It reduces changes
    in the existing code.

    It would be possible to copy internal values when they are allocated
    and make short cuts in init() functions. It would be possible to use
    the fact that old_func and new_func point to the same function and
    do not init new_func and new_size at all. It would be possible to
    detect nop func in ftrace handler and just leave. But all these would
    just complicate the code and maintenance.

  + The callbacks from the replaced patches are not called. It would be
    pretty hard to define a reasonable semantic and implement it.

    It might even be counter-productive. The new patch is cumulative.
    It is supposed to include most of the changes from older patches.
    In most cases, it will not want to call pre_unpatch() post_unpatch()
    callbacks from the replaced patches. It would disable/break things
    for no good reasons. Also it should be easier to handle various
    scenarios in a single script in the new patch than think about
    interactions caused by running many scripts from older patches.
    No to say that the old scripts even would not expect to be called
    in this situation.

Signed-off-by: Jason Baron <jbaron@akamai.com>
[pmladek@suse.com: Split, reuse existing code, simplified]
Signed-off-by: Petr Mladek <pmladek@suse.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Jessica Yu <jeyu@kernel.org>
Cc: Jiri Kosina <jikos@kernel.org>
Acked-by: Miroslav Benes <mbenes@suse.cz>
---
 include/linux/livepatch.h     |   3 +
 kernel/livepatch/core.c       | 162 +++++++++++++++++++++++++++++++++++++++++-
 kernel/livepatch/core.h       |   4 ++
 kernel/livepatch/transition.c |  39 ++++++++++
 4 files changed, 206 insertions(+), 2 deletions(-)

diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
index 7fb76e7d2693..ed598d849029 100644
--- a/include/linux/livepatch.h
+++ b/include/linux/livepatch.h
@@ -42,6 +42,7 @@
 enum klp_func_type {
 	KLP_FUNC_ANY = -1,	/* Substitute any type */
 	KLP_FUNC_STATIC = 0,    /* Original statically defined structure */
+	KLP_FUNC_NOP,		/* Dynamically allocated NOP function patch */
 };
 
 /**
@@ -153,6 +154,7 @@ struct klp_object {
  * struct klp_patch - patch structure for live patching
  * @mod:	reference to the live patch module
  * @objs:	object entries for kernel objects to be patched
+ * @replace:	replace all already registered patches
  * @list:	list node for global list of registered patches
  * @kobj:	kobject for sysfs resources
  * @obj_list:	head of list for struct klp_object
@@ -163,6 +165,7 @@ struct klp_patch {
 	/* external */
 	struct module *mod;
 	struct klp_object *objs;
+	bool replace;
 
 	/* internal */
 	struct list_head list;
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index fd0296859ff4..ad508a86b2f9 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -142,6 +142,21 @@ static bool klp_initialized(void)
 	return !!klp_root_kobj;
 }
 
+static struct klp_func *klp_find_func(struct klp_object *obj,
+				      struct klp_func *old_func)
+{
+	struct klp_func *func;
+
+	klp_for_each_func(obj, func) {
+		if ((strcmp(old_func->old_name, func->old_name) == 0) &&
+		    (old_func->old_sympos == func->old_sympos)) {
+			return func;
+		}
+	}
+
+	return NULL;
+}
+
 static struct klp_object *klp_find_object(struct klp_patch *patch,
 					  struct klp_object *old_obj)
 {
@@ -342,6 +357,39 @@ static int klp_write_object_relocations(struct module *pmod,
 	return ret;
 }
 
+/*
+ * This function removes replaced patches from both func_stack
+ * and klp_patches stack.
+ *
+ * We could be pretty aggressive here. It is called in situation
+ * when these structures are no longer accessible. All functions
+ * are redirected using the klp_transition_patch. They use either
+ * a new code or they are in the original code because of the special
+ * nop function patches.
+ */
+void klp_throw_away_replaced_patches(struct klp_patch *new_patch,
+				     bool keep_module)
+{
+	struct klp_patch *old_patch, *tmp_patch;
+
+	list_for_each_entry_safe(old_patch, tmp_patch, &klp_patches, list) {
+		if (old_patch == new_patch)
+			return;
+
+		klp_unpatch_objects(old_patch, KLP_FUNC_ANY);
+		old_patch->enabled = false;
+
+		/*
+		 * Replaced patches could not get re-enabled to keep
+		 * the code sane.
+		 */
+		list_move(&old_patch->list, &klp_replaced_patches);
+
+		if (!keep_module)
+			module_put(old_patch->mod);
+	}
+}
+
 static int __klp_disable_patch(struct klp_patch *patch)
 {
 	struct klp_object *obj;
@@ -537,7 +585,7 @@ static ssize_t enabled_store(struct kobject *kobj, struct kobj_attribute *attr,
 	if (!klp_is_patch_usable(patch)) {
 		/*
 		 * Module with the patch could either disappear meanwhile or is
-		 * not properly initialized yet.
+		 * not properly initialized yet or the patch was just replaced.
 		 */
 		ret = -EINVAL;
 		goto err;
@@ -662,8 +710,16 @@ static struct attribute *klp_patch_attrs[] = {
 /*
  * Dynamically allocated objects and functions.
  */
+static void klp_free_func_nop(struct klp_func *func)
+{
+	kfree(func->old_name);
+	kfree(func);
+}
+
 static void klp_free_func_dynamic(struct klp_func *func)
 {
+	if (func->ftype == KLP_FUNC_NOP)
+		klp_free_func_nop(func);
 }
 
 static void klp_free_object_dynamic(struct klp_object *obj)
@@ -708,6 +764,102 @@ static struct klp_object *klp_get_or_add_object(struct klp_patch *patch,
 	return obj;
 }
 
+static struct klp_func *klp_alloc_func_nop(struct klp_func *old_func,
+					   struct klp_object *obj)
+{
+	struct klp_func *func;
+
+	func = kzalloc(sizeof(*func), GFP_KERNEL);
+	if (!func)
+		return ERR_PTR(-ENOMEM);
+
+	if (old_func->old_name) {
+		func->old_name = kstrdup(old_func->old_name, GFP_KERNEL);
+		if (!func->old_name) {
+			kfree(func);
+			return ERR_PTR(-ENOMEM);
+		}
+	}
+	func->old_sympos = old_func->old_sympos;
+	/* NOP func is the same as using the original implementation. */
+	func->new_func = (void *)old_func->old_addr;
+	func->ftype = KLP_FUNC_NOP;
+
+	return func;
+}
+
+static int klp_add_func_nop(struct klp_object *obj,
+			    struct klp_func *old_func)
+{
+	struct klp_func *func;
+
+	func = klp_find_func(obj, old_func);
+
+	if (func)
+		return 0;
+
+	func = klp_alloc_func_nop(old_func, obj);
+	if (IS_ERR(func))
+		return PTR_ERR(func);
+
+	klp_init_func_list(obj, func);
+
+	return 0;
+}
+
+static int klp_add_object_nops(struct klp_patch *patch,
+			       struct klp_object *old_obj)
+{
+	struct klp_object *obj;
+	struct klp_func *old_func;
+	int err = 0;
+
+	obj = klp_get_or_add_object(patch, old_obj);
+	if (IS_ERR(obj))
+		return PTR_ERR(obj);
+
+	klp_for_each_func(old_obj, old_func) {
+		err = klp_add_func_nop(obj, old_func);
+		if (err)
+			return err;
+	}
+
+	return 0;
+}
+
+/*
+ * Add 'nop' functions which simply return to the caller to run
+ * the original function. The 'nop' functions are added to a
+ * patch to facilitate a 'replace' mode
+ *
+ * The nops are generated for all patches on the stack when
+ * the new patch is initialized. It is safe even though some
+ * older patches might get disabled and removed before the
+ * new one is enabled. In the worst case, there might be nops
+ * which will not be really needed. But it does not harm and
+ * simplifies the implementation a lot. Especially we could
+ * use the init functions as is.
+ */
+static int klp_add_nops(struct klp_patch *patch)
+{
+	struct klp_patch *old_patch;
+	struct klp_object *old_obj;
+	int err = 0;
+
+	if (WARN_ON(!patch->replace))
+		return -EINVAL;
+
+	list_for_each_entry(old_patch, &klp_patches, list) {
+		klp_for_each_object(old_patch, old_obj) {
+			err = klp_add_object_nops(patch, old_obj);
+			if (err)
+				return err;
+		}
+	}
+
+	return 0;
+}
+
 /*
  * Patch release framework must support the following scenarios:
  *
@@ -802,7 +954,7 @@ static void klp_free_object_loaded(struct klp_object *obj)
  * Use the kobject when it has already been initialized. Otherwise,
  * do it directly.
  */
-static void klp_free_objects(struct klp_patch *patch, enum klp_func_type ftype)
+void klp_free_objects(struct klp_patch *patch, enum klp_func_type ftype)
 {
 	struct klp_object *obj, *tmp_obj;
 
@@ -952,6 +1104,12 @@ static int klp_init_patch(struct klp_patch *patch)
 		return ret;
 	}
 
+	if (patch->replace) {
+		ret = klp_add_nops(patch);
+		if (ret)
+			goto free;
+	}
+
 	klp_for_each_object(patch, obj) {
 		ret = klp_init_object(patch, obj);
 		if (ret)
diff --git a/kernel/livepatch/core.h b/kernel/livepatch/core.h
index 48a83d4364cf..43184a5318d8 100644
--- a/kernel/livepatch/core.h
+++ b/kernel/livepatch/core.h
@@ -6,6 +6,10 @@
 
 extern struct mutex klp_mutex;
 
+void klp_throw_away_replaced_patches(struct klp_patch *new_patch,
+				     bool keep_module);
+void klp_free_objects(struct klp_patch *patch, enum klp_func_type ftype);
+
 static inline bool klp_is_object_loaded(struct klp_object *obj)
 {
 	return !obj->name || obj->mod;
diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
index 6917100fbe79..d6af190865d2 100644
--- a/kernel/livepatch/transition.c
+++ b/kernel/livepatch/transition.c
@@ -87,6 +87,36 @@ static void klp_complete_transition(void)
 		 klp_transition_patch->mod->name,
 		 klp_target_state == KLP_PATCHED ? "patching" : "unpatching");
 
+	/*
+	 * For replace patches, we disable all previous patches, and replace
+	 * the dynamic no-op functions by removing the ftrace hook.
+	 */
+	if (klp_transition_patch->replace && klp_target_state == KLP_PATCHED) {
+		/*
+		 * Make sure that no ftrace handler accesses any older patch
+		 * on the stack.  This might happen when the user forced the
+		 * transaction while some running tasks were still falling
+		 * back to the old code.  There might even still be ftrace
+		 * handlers that have not seen the last patch on the stack yet.
+		 *
+		 * It probably is not necessary because of the rcu-safe access.
+		 * But better be safe than sorry.
+		 */
+		if (klp_forced)
+			klp_synchronize_transition();
+
+		klp_throw_away_replaced_patches(klp_transition_patch,
+						klp_forced);
+
+		/*
+		 * There is no need to synchronize the transition after removing
+		 * nops. They must be the last on the func_stack. Ftrace
+		 * gurantees that nobody will stay in the trampoline after
+		 * the ftrace handler is unregistered.
+		 */
+		klp_unpatch_objects(klp_transition_patch, KLP_FUNC_NOP);
+	}
+
 	if (klp_target_state == KLP_UNPATCHED) {
 		/*
 		 * All tasks have transitioned to KLP_UNPATCHED so we can now
@@ -143,6 +173,15 @@ static void klp_complete_transition(void)
 	if (!klp_forced && klp_target_state == KLP_UNPATCHED)
 		module_put(klp_transition_patch->mod);
 
+	/*
+	 * We do not need to wait until the objects are really freed.
+	 * The patch must be on the bottom of the stack. Therefore it
+	 * will never replace anything else. The only important thing
+	 * is that we wait when the patch is being unregistered.
+	 */
+	if (klp_transition_patch->replace && klp_target_state == KLP_PATCHED)
+		klp_free_objects(klp_transition_patch, KLP_FUNC_NOP);
+
 	klp_target_state = KLP_UNDEFINED;
 	klp_transition_patch = NULL;
 }
-- 
2.13.6

[PATCH v10 07/10] livepatch: Correctly handle atomic replace for not yet loaded modules

[Petr Mladek]

The atomic replace feature uses dynamically allocated struct klp_func to
handle functions that will no longer be patched. These structures are
of the type KLP_FUNC_NOP. They cause the ftrace handler to jump to
the original code. But the address of the original code is not known
until the patched module is loaded.

This patch allows the late initialization. Also it adds a sanity check
into the ftrace handler.

Alternative solution would be not to set the address at all. The ftrace
handler could just return to the original code when NOP struct klp_func
is used. But this would require another changes. For example, in the stack
checking. Note that NOP structures might be available even when the patch
is being disabled. This would happen when the patch enable transition is
reverted.

Signed-off-by: Petr Mladek <pmladek@suse.com>
---
 kernel/livepatch/core.c  | 24 ++++++++++++++++++++----
 kernel/livepatch/patch.c |  5 +++++
 2 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index ad508a86b2f9..67aa4ec9e087 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -781,8 +781,10 @@ static struct klp_func *klp_alloc_func_nop(struct klp_func *old_func,
 		}
 	}
 	func->old_sympos = old_func->old_sympos;
-	/* NOP func is the same as using the original implementation. */
-	func->new_func = (void *)old_func->old_addr;
+	/*
+	 * func->new_func is same as func->old_addr. These addresses are
+	 * set when the object is loaded, see klp_init_object_loaded().
+	 */
 	func->ftype = KLP_FUNC_NOP;
 
 	return func;
@@ -945,8 +947,12 @@ static void klp_free_object_loaded(struct klp_object *obj)
 
 	obj->mod = NULL;
 
-	klp_for_each_func(obj, func)
+	klp_for_each_func(obj, func) {
 		func->old_addr = 0;
+
+		if (klp_is_func_type(func, KLP_FUNC_NOP))
+			func->new_func = NULL;
+	}
 }
 
 /*
@@ -984,7 +990,14 @@ static void klp_free_patch(struct klp_patch *patch)
 
 static int klp_init_func(struct klp_object *obj, struct klp_func *func)
 {
-	if (!func->old_name || !func->new_func)
+	if (!func->old_name)
+		return -EINVAL;
+
+	/*
+	 * NOPs get the address later. The the patched module must be loaded,
+	 * see klp_init_object_loaded().
+	 */
+	if (!func->new_func && !klp_is_func_type(func, KLP_FUNC_NOP))
 		return -EINVAL;
 
 	INIT_LIST_HEAD(&func->stack_node);
@@ -1039,6 +1052,9 @@ static int klp_init_object_loaded(struct klp_patch *patch,
 			return -ENOENT;
 		}
 
+		if (klp_is_func_type(func, KLP_FUNC_NOP))
+			func->new_func = (void *)func->old_addr;
+
 		ret = kallsyms_lookup_size_offset((unsigned long)func->new_func,
 						  &func->new_size, NULL);
 		if (!ret) {
diff --git a/kernel/livepatch/patch.c b/kernel/livepatch/patch.c
index 54b3c379bb0f..1f5c3eea9ee1 100644
--- a/kernel/livepatch/patch.c
+++ b/kernel/livepatch/patch.c
@@ -118,7 +118,12 @@ static void notrace klp_ftrace_handler(unsigned long ip,
 		}
 	}
 
+	/* Survive ugly mistakes, for example, when handling NOPs. */
+	if (WARN_ON_ONCE(!func->new_func))
+		goto unlock;
+
 	klp_arch_set_pc(regs, (unsigned long)func->new_func);
+
 unlock:
 	preempt_enable_notrace();
 }
-- 
2.13.6

[PATCH v10 08/10] livepatch: Improve dynamic struct klp_object detection and manipulation

[Petr Mladek]

The check for dynamically allocated objects was too optimistic. There
might exist livepatches that modify an object only with callbacks. They
are statically defined and have "funcs" array empty.

A solution would be to check also the callback pointers in
klp_is_object_dynamic(). But it still might be error prone.

Note that we must avoid calling klp_free_object_dynamic() even for useless
structures that were defined statically.

Therefore this patch takes a more safe approach. It adds an extra flag
into struct klp_object. The type is different from a similar flag on
the func level. It is because one object structure might point to func
structures of different types. In general, only two states make sense
on the object level.

This fixed the problem _how_ the structures were freed. But there
was also a bug _when_ this happened. For this we added a check
to keep statically defined structures until the statically defined
function structures are being freed.

Signed-off-by: Petr Mladek <pmladek@suse.com>
---
 include/linux/livepatch.h |  8 +++++++-
 kernel/livepatch/core.c   | 10 ++++++++++
 2 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
index ed598d849029..7222b801d63a 100644
--- a/include/linux/livepatch.h
+++ b/include/linux/livepatch.h
@@ -45,6 +45,11 @@ enum klp_func_type {
 	KLP_FUNC_NOP,		/* Dynamically allocated NOP function patch */
 };
 
+enum klp_object_type {
+	KLP_OBJECT_STATIC = 0,  /* Original statically defined structure */
+	KLP_OBJECT_DYNAMIC,	/* Dynamically allocated structure. */
+};
+
 /**
  * struct klp_func - function structure for live patching
  * @old_name:	name of the function to be patched
@@ -143,6 +148,7 @@ struct klp_object {
 	struct klp_callbacks callbacks;
 
 	/* internal */
+	enum klp_object_type otype;
 	struct kobject kobj;
 	struct list_head func_list;
 	struct list_head obj_entry;
@@ -198,7 +204,7 @@ struct klp_patch {
 
 static inline bool klp_is_object_dynamic(struct klp_object *obj)
 {
-	return !obj->funcs;
+	return obj->otype == KLP_OBJECT_DYNAMIC;
 }
 
 static inline bool klp_is_func_dynamic(struct klp_func *func)
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 67aa4ec9e087..73ce3f93e0bc 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -743,6 +743,7 @@ static struct klp_object *klp_alloc_object_dynamic(const char *name)
 			return ERR_PTR(-ENOMEM);
 		}
 	}
+	obj->otype = KLP_OBJECT_DYNAMIC;
 
 	return obj;
 }
@@ -970,6 +971,15 @@ void klp_free_objects(struct klp_patch *patch, enum klp_func_type ftype)
 		if (!list_empty(&obj->func_list))
 			continue;
 
+		/*
+		 * Keep objects from the original patch initialized until
+		 * the entire patch is being freed.
+		 */
+		if (!klp_is_object_dynamic(obj) &&
+		    ftype != KLP_FUNC_STATIC &&
+		    ftype != KLP_FUNC_ANY)
+			continue;
+
 		/* Avoid freeing the object twice. */
 		list_del(&obj->obj_entry);
 
-- 
2.13.6

[PATCH v10 09/10] livepatch: Allow to replace even disabled patches

[Petr Mladek]

Patches without the replace flag might depend on each other. It makes
sense to enforce the order in which they are enabled and disabled.

The situation is different when the patch replaces all existing ones.
It should make the life easier for both: patch producers and users.
Such a patch should be ready to replace basically any older patch.
It should work well even in situations when the previous patches
were not installed or when they were disabled from some reasons.

The code is almost ready for this:

  + klp_add_nops() takes into account even disabled patches. In the worst
    case, we might enable some NOPs that are not really needed.

  + klp_throw_away_replaced_patches() removes all patches down the stack.

We only need to make sure that the livepatch module is put only when
it was gotten (enabled) before.

Also we need to stop enforcing the stack order for the patches
with the replace flag. Instead, we need to make sure that they
are still usable (not replaced).

Signed-off-by: Petr Mladek <pmladek@suse.com>
---
 kernel/livepatch/core.c | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 73ce3f93e0bc..b098dc10d4d5 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -376,17 +376,19 @@ void klp_throw_away_replaced_patches(struct klp_patch *new_patch,
 		if (old_patch == new_patch)
 			return;
 
-		klp_unpatch_objects(old_patch, KLP_FUNC_ANY);
-		old_patch->enabled = false;
+		if (old_patch->enabled) {
+			klp_unpatch_objects(old_patch, KLP_FUNC_ANY);
+			old_patch->enabled = false;
+
+			if (!keep_module)
+				module_put(old_patch->mod);
+		}
 
 		/*
 		 * Replaced patches could not get re-enabled to keep
 		 * the code sane.
 		 */
 		list_move(&old_patch->list, &klp_replaced_patches);
-
-		if (!keep_module)
-			module_put(old_patch->mod);
 	}
 }
 
@@ -470,8 +472,16 @@ static int __klp_enable_patch(struct klp_patch *patch)
 	if (WARN_ON(patch->enabled))
 		return -EINVAL;
 
-	/* enforce stacking: only the first disabled patch can be enabled */
-	if (patch->list.prev != &klp_patches &&
+	if (!klp_is_patch_usable(patch))
+		return -EINVAL;
+
+	/*
+	 * Enforce stacking: only the first disabled patch can be enabled.
+	 * This is not required for patches with the replace flags. They
+	 * override even disabled patches that were registered earlier.
+	 */
+	if (!patch->replace &&
+	    patch->list.prev != &klp_patches &&
 	    !list_prev_entry(patch, list)->enabled)
 		return -EBUSY;
 
-- 
2.13.6

[PATCH v10 10/10] livepatch: Atomic replace and cumulative patches documentation

[Petr Mladek]

User documentation for the atomic replace feature. It makes it easier
to maintain livepatches using so-called cumulative patches.

Signed-off-by: Petr Mladek <pmladek@suse.com>
Acked-by: Miroslav Benes <mbenes@suse.cz>
---
 Documentation/livepatch/cumulative-patches.txt | 83 ++++++++++++++++++++++++++
 1 file changed, 83 insertions(+)
 create mode 100644 Documentation/livepatch/cumulative-patches.txt

diff --git a/Documentation/livepatch/cumulative-patches.txt b/Documentation/livepatch/cumulative-patches.txt
new file mode 100644
index 000000000000..c041fc1bd259
--- /dev/null
+++ b/Documentation/livepatch/cumulative-patches.txt
@@ -0,0 +1,83 @@
+===================================
+Atomic Replace & Cumulative Patches
+===================================
+
+There might be dependencies between livepatches. If multiple patches need
+to do different changes to the same function(s) then we need to define
+an order in which the patches will be installed. And function implementations
+from any newer livepatch must be done on top of the older ones.
+
+This might become a maintenance nightmare. Especially if anyone would want
+to remove a patch that is in the middle of the stack.
+
+An elegant solution comes with the feature called "Atomic Replace". It allows
+to create so called "Cumulative Patches". They include all wanted changes
+from all older livepatches and completely replace them in one transition.
+
+Usage
+-----
+
+The atomic replace can be enabled by setting "replace" flag in struct klp_patch,
+for example:
+
+	static struct klp_patch patch = {
+		.mod = THIS_MODULE,
+		.objs = objs,
+		.replace = true,
+	};
+
+Such a patch is added on top of the livepatch stack when registered. It can
+be enabled even when some earlier patches have not been enabled yet.
+
+All processes are then migrated to use the code only from the new patch.
+Once the transition is finished, all older patches are removed from the stack
+of patches. Even the older not-enabled patches mentioned above.
+
+Ftrace handlers are transparently removed from functions that are no
+longer modified by the new cumulative patch.
+
+As a result, the livepatch author might maintain sources only for one
+cumulative patch. It helps to keep the patch consistent while adding or
+removing various fixes or features.
+
+
+Limitations:
+------------
+
+  + Replaced patches can no longer be enabled. But if the transition
+    to the cumulative patch was not forced, the kernel modules with
+    the older livepatches can be removed and eventually added again.
+
+    A good practice is to set .replace flag in any released livepatch.
+    Then re-adding an older livepatch is equivalent to downgrading
+    to that patch. This is safe as long as the livepatches do _not_ do
+    extra modifications in (un)patching callbacks or in the module_init()
+    or module_exit() functions, see below.
+
+
+  + Only the (un)patching callbacks from the _new_ cumulative livepatch are
+    executed. Any callbacks from the replaced patches are ignored.
+
+    By other words, the cumulative patch is responsible for doing any actions
+    that are necessary to properly replace any older patch.
+
+    As a result, it might be dangerous to replace newer cumulative patches by
+    older ones. The old livepatches might not provide the necessary callbacks.
+
+    This might be seen as a limitation in some scenarios. But it makes the life
+    easier in many others. Only the new cumulative livepatch knows what
+    fixes/features are added/removed and what special actions are necessary
+    for a smooth transition.
+
+    In each case, it would be a nightmare to think about the order of
+    the various callbacks and their interactions if the callbacks from all
+    enabled patches were called.
+
+
+  + There is no special handling of shadow variables. Livepatch authors
+    must create their own rules how to pass them from one cumulative
+    patch to the other. Especially they should not blindly remove them
+    in module_exit() functions.
+
+    A good practice might be to remove shadow variables in the post-unpatch
+    callback. It is called only when the livepatch is properly disabled.
-- 
2.13.6

Re: [PATCH v10 00/10] livepatch: Atomic replace feature

[Joe Lawrence]

On 03/07/2018 03:20 AM, Petr Mladek wrote:
> The atomic replace allows to create cumulative patches. They
> are useful when you maintain many livepatches and want to remove
> one that is lower on the stack. In addition it is very useful when
> more patches touch the same function and there are dependencies
> between them.
> 
> 
> Changes against v9:
> 
>   + Fixed check of valid NOPs for already loaded objects,
>     regression introduced in v9 [Joe, Mirek]
>   + Allow to replace even disabled patches [Evgenii]
> 
> Changes against v8:
> 
>   + Fixed handling of statically defined struct klp_object
>     with empty array of functions [Joe, Mirek]
>   + Removed redundant func->new_func assignment for NOPs [Mirek]
>   + Improved some wording [Mirek]
>
> [ ... snip ... ]

Hi Petr,

I tried updating the test cases I was adding in "[PATCH v0 0/3]
additional cumulative livepatch doc/samples" and although one of the
cases is better than before, I'm running into a new issue:  an expected
pre-unpatch callback is not executed (its obj->patched is false).

Here's the updated test case:

Test 11
-------

- load livepatch
- load second livepatch (atomic replace)     <- callbacks ok
- disable second livepatch                   <- pre-unpatch skipped
- unload livepatch
- unload second livepatch

  % insmod samples/livepatch/livepatch-callbacks-demo.ko
  [ 2306.806046] livepatch: enabling patch 'livepatch_callbacks_demo'
  [ 2306.806048] livepatch: 'livepatch_callbacks_demo': initializing patching transition
  [ 2306.806083] livepatch_callbacks_demo: pre_patch_callback: vmlinux
  [ 2306.806083] livepatch: 'livepatch_callbacks_demo': starting patching transition
  [ 2307.743170] livepatch: 'livepatch_callbacks_demo': completing patching transition
  [ 2307.743317] livepatch_callbacks_demo: post_patch_callback: vmlinux
  [ 2307.743319] livepatch: 'livepatch_callbacks_demo': patching complet

  % insmod samples/livepatch/livepatch-callbacks-demo2.ko replace=1
  [ 2316.161804] livepatch: enabling patch 'livepatch_callbacks_demo2'
  [ 2316.161807] livepatch: 'livepatch_callbacks_demo2': initializing patching transition
  [ 2316.161842] livepatch_callbacks_demo2: pre_patch_callback: vmlinux
  [ 2316.161843] livepatch: 'livepatch_callbacks_demo2': starting patching transition
  [ 2317.727141] livepatch: 'livepatch_callbacks_demo2': completing patching transition
  [ 2317.727254] livepatch_callbacks_demo2: post_patch_callback: vmlinux
  [ 2317.727255] livepatch: 'livepatch_callbacks_demo2': patching complete

  % echo 0 > /sys/kernel/livepatch/livepatch_callbacks_demo2/enabled
  [ 2328.995854] livepatch: 'livepatch_callbacks_demo2': initializing unpatching transition
  [ 2328.995898] livepatch: 'livepatch_callbacks_demo2': starting unpatching transition
  [ 2330.719234] livepatch: 'livepatch_callbacks_demo2': completing unpatching transition
  [ 2330.719597] livepatch_callbacks_demo2: post_unpatch_callback: vmlinux
  [ 2330.719599] livepatch: 'livepatch_callbacks_demo2': unpatching complete

  % rmmod samples/livepatch/livepatch-callbacks-demo2.ko
  % rmmod samples/livepatch/livepatch-callbacks-demo.ko

Running against v10, callbacks seem to be good up until I disable an
atomic replace patch.  My understanding is that the original patch's
unpatch callbacks should be skipped (as they were).  I was surprised to
see that atomic replacement patch only ran it's post-unpatch callback.

Unfortunately I'm running out of time to further debug today, but
thought I would share these results.  I can dig in more tomorrow.

Regards,

-- Joe

Re: [PATCH v10 00/10] livepatch: Atomic replace feature

[Petr Mladek]

On Wed 2018-03-07 16:55:53, Joe Lawrence wrote:
> On 03/07/2018 03:20 AM, Petr Mladek wrote:
> > The atomic replace allows to create cumulative patches. They
> > are useful when you maintain many livepatches and want to remove
> > one that is lower on the stack. In addition it is very useful when
> > more patches touch the same function and there are dependencies
> > between them.
> > 
> > 
> > Changes against v9:
> > 
> >   + Fixed check of valid NOPs for already loaded objects,
> >     regression introduced in v9 [Joe, Mirek]
> >   + Allow to replace even disabled patches [Evgenii]
> > 
> > Changes against v8:
> > 
> >   + Fixed handling of statically defined struct klp_object
> >     with empty array of functions [Joe, Mirek]
> >   + Removed redundant func->new_func assignment for NOPs [Mirek]
> >   + Improved some wording [Mirek]
> >
> > [ ... snip ... ]
> 
> Hi Petr,
> 
> I tried updating the test cases I was adding in "[PATCH v0 0/3]
> additional cumulative livepatch doc/samples" and although one of the
> cases is better than before, I'm running into a new issue:  an expected
> pre-unpatch callback is not executed (its obj->patched is false).
> 
> Here's the updated test case:
> 
> Test 11
> -------
> 
> - load livepatch
> - load second livepatch (atomic replace)     <- callbacks ok
> - disable second livepatch                   <- pre-unpatch skipped
> - unload livepatch
> - unload second livepatch
> 
>   % insmod samples/livepatch/livepatch-callbacks-demo.ko
>   [ 2306.806046] livepatch: enabling patch 'livepatch_callbacks_demo'
>   [ 2306.806048] livepatch: 'livepatch_callbacks_demo': initializing patching transition
>   [ 2306.806083] livepatch_callbacks_demo: pre_patch_callback: vmlinux
>   [ 2306.806083] livepatch: 'livepatch_callbacks_demo': starting patching transition
>   [ 2307.743170] livepatch: 'livepatch_callbacks_demo': completing patching transition
>   [ 2307.743317] livepatch_callbacks_demo: post_patch_callback: vmlinux
>   [ 2307.743319] livepatch: 'livepatch_callbacks_demo': patching complet
> 
>   % insmod samples/livepatch/livepatch-callbacks-demo2.ko replace=1
>   [ 2316.161804] livepatch: enabling patch 'livepatch_callbacks_demo2'
>   [ 2316.161807] livepatch: 'livepatch_callbacks_demo2': initializing patching transition
>   [ 2316.161842] livepatch_callbacks_demo2: pre_patch_callback: vmlinux
>   [ 2316.161843] livepatch: 'livepatch_callbacks_demo2': starting patching transition
>   [ 2317.727141] livepatch: 'livepatch_callbacks_demo2': completing patching transition
>   [ 2317.727254] livepatch_callbacks_demo2: post_patch_callback: vmlinux
>   [ 2317.727255] livepatch: 'livepatch_callbacks_demo2': patching complete
> 
>   % echo 0 > /sys/kernel/livepatch/livepatch_callbacks_demo2/enabled
>   [ 2328.995854] livepatch: 'livepatch_callbacks_demo2': initializing unpatching transition
>   [ 2328.995898] livepatch: 'livepatch_callbacks_demo2': starting unpatching transition
>   [ 2330.719234] livepatch: 'livepatch_callbacks_demo2': completing unpatching transition
>   [ 2330.719597] livepatch_callbacks_demo2: post_unpatch_callback: vmlinux
>   [ 2330.719599] livepatch: 'livepatch_callbacks_demo2': unpatching complete
> 
>   % rmmod samples/livepatch/livepatch-callbacks-demo2.ko
>   % rmmod samples/livepatch/livepatch-callbacks-demo.ko
> 
> Running against v10, callbacks seem to be good up until I disable an
> atomic replace patch.  My understanding is that the original patch's
> unpatch callbacks should be skipped (as they were).  I was surprised to
> see that atomic replacement patch only ran it's post-unpatch callback.

Great catch!

I guess that it is caused by the heuristic used in
klp_unpatch_object() to decide whether the object is patched
or not.

We need to change the state only when manipulating the
statically defined functions.

Thanks a lot for so extensive testing!!!

Best Regards,
Petr

Re: [PATCH v10 00/10] livepatch: Atomic replace feature

[Joe Lawrence]

On 03/08/2018 10:01 AM, Petr Mladek wrote:
> On Wed 2018-03-07 16:55:53, Joe Lawrence wrote:
>> Running against v10, callbacks seem to be good up until I disable an
>> atomic replace patch.  My understanding is that the original patch's
>> unpatch callbacks should be skipped (as they were).  I was surprised to
>> see that atomic replacement patch only ran it's post-unpatch callback.
> 
> Great catch!
> 
> I guess that it is caused by the heuristic used in
> klp_unpatch_object() to decide whether the object is patched
> or not.
> 
> We need to change the state only when manipulating the
> statically defined functions.
> 
> Thanks a lot for so extensive testing!!!

Sorry for not getting to this series sooner.  I'm trying to refit these
tests into a kselftest so we can more easily reuse them.  Still hacking
away at it, but I'll post when something soon to start a livepatch
selftests conversation :)

-- Joe

Re: [PATCH v10 00/10] livepatch: Atomic replace feature

[Joe Lawrence]

Hi Petr,

These are the callback tests that I hacked up into a livepatch
kselftest.  (Basically I copied a bunch of the sample modules and
verified the expected dmesg output as I had listed in in the
Documentation/livepatch/callbacks.txt file.)  The script is still a
little rough and maybe this isn't the direction we want to go for proper
kselftests, but perhaps it saves you some time/sanity for verifying this
patchset.

Hope this helps,

-- Joe

-- >8 -- >8 -- >8 -- >8 --

>From 0364430c53e12e21923bed20cb651374b4cf9ba9 Mon Sep 17 00:00:00 2001
From: Joe Lawrence <joe.lawrence@redhat.com>
Date: Tue, 6 Mar 2018 17:32:25 -0500
Subject: WIP - livepatch kselftest

CONFIG_TEST_LIVEPATCH=m
% make -C tools/testing/selftests TARGETS=livepatch run_tests

Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
---
 lib/Kconfig.debug                                |  11 +
 lib/Makefile                                     |   9 +
 lib/test_klp_callbacks_busy.c                    |  58 ++
 lib/test_klp_callbacks_demo.c                    | 205 +++++++
 lib/test_klp_callbacks_demo2.c                   | 149 +++++
 lib/test_klp_callbacks_mod.c                     |  39 ++
 tools/testing/selftests/Makefile                 |   1 +
 tools/testing/selftests/livepatch/Makefile       |   5 +
 tools/testing/selftests/livepatch/config         |   1 +
 tools/testing/selftests/livepatch/livepatch-test | 658 +++++++++++++++++++++++
 10 files changed, 1136 insertions(+)
 create mode 100644 lib/test_klp_callbacks_busy.c
 create mode 100644 lib/test_klp_callbacks_demo.c
 create mode 100644 lib/test_klp_callbacks_demo2.c
 create mode 100644 lib/test_klp_callbacks_mod.c
 create mode 100644 tools/testing/selftests/livepatch/Makefile
 create mode 100644 tools/testing/selftests/livepatch/config
 create mode 100755 tools/testing/selftests/livepatch/livepatch-test

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 64155e310a9f..cd2a2d25314e 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1932,6 +1932,17 @@ config TEST_DEBUG_VIRTUAL
 
 	  If unsure, say N.
 
+config TEST_LIVEPATCH
+	tristate "Test livepatching"
+	default n
+	depends on LIVEPATCH
+	help
+	  Test various kernel livepatching features for correctness.
+	  The tests will load test modules that will be livepatched
+	  in various scenarios.
+
+	  If unsure, say N.
+
 endif # RUNTIME_TESTING_MENU
 
 config MEMTEST
diff --git a/lib/Makefile b/lib/Makefile
index a90d4fcd748f..919de0acf1a8 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -67,6 +67,15 @@ obj-$(CONFIG_TEST_PARMAN) += test_parman.o
 obj-$(CONFIG_TEST_KMOD) += test_kmod.o
 obj-$(CONFIG_TEST_DEBUG_VIRTUAL) += test_debug_virtual.o
 
+obj-$(CONFIG_TEST_LIVEPATCH) += test_klp_callbacks_demo.o \
+				test_klp_callbacks_demo2.o \
+				test_klp_callbacks_busy.o \
+				test_klp_callbacks_mod.o
+CFLAGS_test_klp_callbacks_demo.o	+= $(CC_FLAGS_FTRACE)
+CFLAGS_test_klp_callbacks_demo2.o	+= $(CC_FLAGS_FTRACE)
+CFLAGS_test_klp_callbacks_busy.o	+= $(CC_FLAGS_FTRACE)
+CFLAGS_test_klp_callbacks_mod.o		+= $(CC_FLAGS_FTRACE)
+
 ifeq ($(CONFIG_DEBUG_KOBJECT),y)
 CFLAGS_kobject.o += -DDEBUG
 CFLAGS_kobject_uevent.o += -DDEBUG
diff --git a/lib/test_klp_callbacks_busy.c b/lib/test_klp_callbacks_busy.c
new file mode 100644
index 000000000000..f76a7e6bed00
--- /dev/null
+++ b/lib/test_klp_callbacks_busy.c
@@ -0,0 +1,58 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2018 Joe Lawrence <joe.lawrence@redhat.com>
+
+/*
+ * livepatch-callbacks-busymod.c - (un)patching callbacks demo support module
+ *
+ *
+ * Purpose
+ * -------
+ *
+ * Simple module to demonstrate livepatch (un)patching callbacks.
+ *
+ *
+ * Usage
+ * -----
+ *
+ * This module is not intended to be standalone.  See the "Usage"
+ * section of livepatch-callbacks-mod.c.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/workqueue.h>
+#include <linux/delay.h>
+
+static int sleep_secs;
+module_param(sleep_secs, int, 0644);
+MODULE_PARM_DESC(sleep_secs, "sleep_secs (default=0)");
+
+static void busymod_work_func(struct work_struct *work);
+static DECLARE_DELAYED_WORK(work, busymod_work_func);
+
+static void busymod_work_func(struct work_struct *work)
+{
+	pr_info("%s, sleeping %d seconds ...\n", __func__, sleep_secs);
+	msleep(sleep_secs * 1000);
+	pr_info("%s exit\n", __func__);
+}
+
+static int livepatch_callbacks_mod_init(void)
+{
+	pr_info("%s\n", __func__);
+	schedule_delayed_work(&work,
+		msecs_to_jiffies(1000 * 0));
+	return 0;
+}
+
+static void livepatch_callbacks_mod_exit(void)
+{
+	cancel_delayed_work_sync(&work);
+	pr_info("%s\n", __func__);
+}
+
+module_init(livepatch_callbacks_mod_init);
+module_exit(livepatch_callbacks_mod_exit);
+MODULE_LICENSE("GPL");
diff --git a/lib/test_klp_callbacks_demo.c b/lib/test_klp_callbacks_demo.c
new file mode 100644
index 000000000000..dfbedef232a5
--- /dev/null
+++ b/lib/test_klp_callbacks_demo.c
@@ -0,0 +1,205 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2018 Joe Lawrence <joe.lawrence@redhat.com>
+
+/*
+ * livepatch-callbacks-demo.c - (un)patching callbacks livepatch demo
+ *
+ *
+ * Purpose
+ * -------
+ *
+ * Demonstration of registering livepatch (un)patching callbacks.
+ *
+ *
+ * Usage
+ * -----
+ *
+ * Step 1 - load the simple module
+ *
+ *   insmod samples/livepatch/livepatch-callbacks-mod.ko
+ *
+ *
+ * Step 2 - load the demonstration livepatch (with callbacks)
+ *
+ *   insmod samples/livepatch/livepatch-callbacks-demo.ko
+ *
+ *
+ * Step 3 - cleanup
+ *
+ *   echo 0 > /sys/kernel/livepatch/livepatch_callbacks_demo/enabled
+ *   rmmod livepatch_callbacks_demo
+ *   rmmod livepatch_callbacks_mod
+ *
+ * Watch dmesg output to see livepatch enablement, callback execution
+ * and patching operations for both vmlinux and module targets.
+ *
+ * NOTE: swap the insmod order of livepatch-callbacks-mod.ko and
+ *       livepatch-callbacks-demo.ko to observe what happens when a
+ *       target module is loaded after a livepatch with callbacks.
+ *
+ * NOTE: 'pre_patch_ret' is a module parameter that sets the pre-patch
+ *       callback return status.  Try setting up a non-zero status
+ *       such as -19 (-ENODEV):
+ *
+ *       # Load demo livepatch, vmlinux is patched
+ *       insmod samples/livepatch/livepatch-callbacks-demo.ko
+ *
+ *       # Setup next pre-patch callback to return -ENODEV
+ *       echo -19 > /sys/module/livepatch_callbacks_demo/parameters/pre_patch_ret
+ *
+ *       # Module loader refuses to load the target module
+ *       insmod samples/livepatch/livepatch-callbacks-mod.ko
+ *       insmod: ERROR: could not insert module samples/livepatch/livepatch-callbacks-mod.ko: No such device
+ *
+ * NOTE: There is a second target module,
+ *       livepatch-callbacks-busymod.ko, available for experimenting
+ *       with livepatch (un)patch callbacks.  This module contains
+ *       a 'sleep_secs' parameter that parks the module on one of the
+ *       functions that the livepatch demo module wants to patch.
+ *       Modifying this value and tweaking the order of module loads can
+ *       effectively demonstrate stalled patch transitions:
+ *
+ *       # Load a target module, let it park on 'busymod_work_func' for
+ *       # thirty seconds
+ *       insmod samples/livepatch/livepatch-callbacks-busymod.ko sleep_secs=30
+ *
+ *       # Meanwhile load the livepatch
+ *       insmod samples/livepatch/livepatch-callbacks-demo.ko
+ *
+ *       # ... then load and unload another target module while the
+ *       # transition is in progress
+ *       insmod samples/livepatch/livepatch-callbacks-mod.ko
+ *       rmmod samples/livepatch/livepatch-callbacks-mod.ko
+ *
+ *       # Finally cleanup
+ *       echo 0 > /sys/kernel/livepatch/livepatch_callbacks_demo/enabled
+ *       rmmod samples/livepatch/livepatch-callbacks-demo.ko
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/livepatch.h>
+
+static int pre_patch_ret;
+module_param(pre_patch_ret, int, 0644);
+MODULE_PARM_DESC(pre_patch_ret, "pre_patch_ret (default=0)");
+
+static const char *const module_state[] = {
+	[MODULE_STATE_LIVE]	= "[MODULE_STATE_LIVE] Normal state",
+	[MODULE_STATE_COMING]	= "[MODULE_STATE_COMING] Full formed, running module_init",
+	[MODULE_STATE_GOING]	= "[MODULE_STATE_GOING] Going away",
+	[MODULE_STATE_UNFORMED]	= "[MODULE_STATE_UNFORMED] Still setting it up",
+};
+
+static void callback_info(const char *callback, struct klp_object *obj)
+{
+	if (obj->mod)
+		pr_info("%s: %s -> %s\n", callback, obj->mod->name,
+			module_state[obj->mod->state]);
+	else
+		pr_info("%s: vmlinux\n", callback);
+}
+
+/* Executed on object patching (ie, patch enablement) */
+static int pre_patch_callback(struct klp_object *obj)
+{
+	callback_info(__func__, obj);
+	return pre_patch_ret;
+}
+
+/* Executed on object unpatching (ie, patch disablement) */
+static void post_patch_callback(struct klp_object *obj)
+{
+	callback_info(__func__, obj);
+}
+
+/* Executed on object unpatching (ie, patch disablement) */
+static void pre_unpatch_callback(struct klp_object *obj)
+{
+	callback_info(__func__, obj);
+}
+
+/* Executed on object unpatching (ie, patch disablement) */
+static void post_unpatch_callback(struct klp_object *obj)
+{
+	callback_info(__func__, obj);
+}
+
+static void patched_work_func(struct work_struct *work)
+{
+	pr_info("%s\n", __func__);
+}
+
+static struct klp_func no_funcs[] = {
+	{ }
+};
+
+static struct klp_func busymod_funcs[] = {
+	{
+		.old_name = "busymod_work_func",
+		.new_func = patched_work_func,
+	}, { }
+};
+
+static struct klp_object objs[] = {
+	{
+		.name = NULL,	/* vmlinux */
+		.funcs = no_funcs,
+		.callbacks = {
+			.pre_patch = pre_patch_callback,
+			.post_patch = post_patch_callback,
+			.pre_unpatch = pre_unpatch_callback,
+			.post_unpatch = post_unpatch_callback,
+		},
+	},	{
+		.name = "test_klp_callbacks_mod",
+		.funcs = no_funcs,
+		.callbacks = {
+			.pre_patch = pre_patch_callback,
+			.post_patch = post_patch_callback,
+			.pre_unpatch = pre_unpatch_callback,
+			.post_unpatch = post_unpatch_callback,
+		},
+	},	{
+		.name = "test_klp_callbacks_busy",
+		.funcs = busymod_funcs,
+		.callbacks = {
+			.pre_patch = pre_patch_callback,
+			.post_patch = post_patch_callback,
+			.pre_unpatch = pre_unpatch_callback,
+			.post_unpatch = post_unpatch_callback,
+		},
+	}, { }
+};
+
+static struct klp_patch patch = {
+	.mod = THIS_MODULE,
+	.objs = objs,
+};
+
+static int livepatch_callbacks_demo_init(void)
+{
+	int ret;
+
+	ret = klp_register_patch(&patch);
+	if (ret)
+		return ret;
+	ret = klp_enable_patch(&patch);
+	if (ret) {
+		WARN_ON(klp_unregister_patch(&patch));
+		return ret;
+	}
+	return 0;
+}
+
+static void livepatch_callbacks_demo_exit(void)
+{
+	WARN_ON(klp_unregister_patch(&patch));
+}
+
+module_init(livepatch_callbacks_demo_init);
+module_exit(livepatch_callbacks_demo_exit);
+MODULE_LICENSE("GPL");
+MODULE_INFO(livepatch, "Y");
diff --git a/lib/test_klp_callbacks_demo2.c b/lib/test_klp_callbacks_demo2.c
new file mode 100644
index 000000000000..8283c77bf780
--- /dev/null
+++ b/lib/test_klp_callbacks_demo2.c
@@ -0,0 +1,149 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2018 Joe Lawrence <joe.lawrence@redhat.com>
+
+/*
+ * livepatch-callbacks-demo2.c - (un)patching callbacks livepatch demo
+ *
+ *
+ * Purpose
+ * -------
+ *
+ * Demonstration of registering livepatch (un)patching callbacks and
+ * their behavior in atomic replace patches.
+ *
+ *
+ * Usage
+ * -----
+ *
+ * Step 1 - load two livepatch callback demos (default behavior)
+ *
+ *   insmod samples/livepatch/livepatch-callbacks-demo.ko
+ *   insmod samples/livepatch/livepatch-callbacks-demo2.ko replace=0
+ *   echo 0 > /sys/kernel/livepatch/livepatch_callbacks_demo2/enabled
+ *   echo 0 > /sys/kernel/livepatch/livepatch_callbacks_demo/enabled
+ *
+ * Watch dmesg output to see pre and post (un)patch callbacks made for
+ * both livepatch-callbacks-demo and livepatch-callbacks-demo2.
+ *
+ * Remove the modules to prepare for the next step:
+ *
+ *   rmmod samples/livepatch/livepatch-callbacks-demo2.ko
+ *   rmmod samples/livepatch/livepatch-callbacks-demo.ko
+ *
+ * Step 1 - load two livepatch callback demos (atomic replace behavior)
+ *
+ *   insmod samples/livepatch/livepatch-callbacks-demo.ko
+ *   insmod samples/livepatch/livepatch-callbacks-demo2.ko replace=1
+ *   echo 0 > /sys/kernel/livepatch/livepatch_callbacks_demo2/enabled
+ *   echo 0 > /sys/kernel/livepatch/livepatch_callbacks_demo/enabled
+ *
+ * Check dmesg output again and notice that when an atomic replace
+ * patch is loaded, only its pre and post unpatch callbacks are
+ * executed.
+ *
+ * Final cleanup:
+ *
+ *   rmmod samples/livepatch/livepatch-callbacks-demo2.ko
+ *   rmmod samples/livepatch/livepatch-callbacks-demo.ko
+ *
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/livepatch.h>
+
+static int replace;
+module_param(replace, int, 0644);
+MODULE_PARM_DESC(replace, "replace (default=0)");
+
+static const char *const module_state[] = {
+	[MODULE_STATE_LIVE]	= "[MODULE_STATE_LIVE] Normal state",
+	[MODULE_STATE_COMING]	= "[MODULE_STATE_COMING] Full formed, running module_init",
+	[MODULE_STATE_GOING]	= "[MODULE_STATE_GOING] Going away",
+	[MODULE_STATE_UNFORMED]	= "[MODULE_STATE_UNFORMED] Still setting it up",
+};
+
+static void callback_info(const char *callback, struct klp_object *obj)
+{
+	if (obj->mod)
+		pr_info("%s: %s -> %s\n", callback, obj->mod->name,
+			module_state[obj->mod->state]);
+	else
+		pr_info("%s: vmlinux\n", callback);
+}
+
+/* Executed on object patching (ie, patch enablement) */
+static int pre_patch_callback(struct klp_object *obj)
+{
+	callback_info(__func__, obj);
+	return 0;
+}
+
+/* Executed on object unpatching (ie, patch disablement) */
+static void post_patch_callback(struct klp_object *obj)
+{
+	callback_info(__func__, obj);
+}
+
+/* Executed on object unpatching (ie, patch disablement) */
+static void pre_unpatch_callback(struct klp_object *obj)
+{
+	callback_info(__func__, obj);
+}
+
+/* Executed on object unpatching (ie, patch disablement) */
+static void post_unpatch_callback(struct klp_object *obj)
+{
+	callback_info(__func__, obj);
+}
+
+static struct klp_func no_funcs[] = {
+	{ }
+};
+
+static struct klp_object objs[] = {
+	{
+		.name = NULL,	/* vmlinux */
+		.funcs = no_funcs,
+		.callbacks = {
+			.pre_patch = pre_patch_callback,
+			.post_patch = post_patch_callback,
+			.pre_unpatch = pre_unpatch_callback,
+			.post_unpatch = post_unpatch_callback,
+		},
+	}, { }
+};
+
+static struct klp_patch patch = {
+	.mod = THIS_MODULE,
+	.objs = objs,
+};
+
+static int livepatch_callbacks_demo2_init(void)
+{
+	int ret;
+
+	patch.replace = replace;
+
+	ret = klp_register_patch(&patch);
+	if (ret)
+		return ret;
+	ret = klp_enable_patch(&patch);
+	if (ret) {
+		WARN_ON(klp_unregister_patch(&patch));
+		return ret;
+	}
+	return 0;
+}
+
+static void livepatch_callbacks_demo2_exit(void)
+{
+	WARN_ON(klp_unregister_patch(&patch));
+}
+
+module_init(livepatch_callbacks_demo2_init);
+module_exit(livepatch_callbacks_demo2_exit);
+MODULE_LICENSE("GPL");
+MODULE_INFO(livepatch, "Y");
diff --git a/lib/test_klp_callbacks_mod.c b/lib/test_klp_callbacks_mod.c
new file mode 100644
index 000000000000..b6953d8d0c37
--- /dev/null
+++ b/lib/test_klp_callbacks_mod.c
@@ -0,0 +1,39 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2018 Joe Lawrence <joe.lawrence@redhat.com>
+
+/*
+ * livepatch-callbacks-mod.c - (un)patching callbacks demo support module
+ *
+ *
+ * Purpose
+ * -------
+ *
+ * Simple module to demonstrate livepatch (un)patching callbacks.
+ *
+ *
+ * Usage
+ * -----
+ *
+ * This module is not intended to be standalone.  See the "Usage"
+ * section of livepatch-callbacks-demo.c.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+
+static int livepatch_callbacks_mod_init(void)
+{
+	pr_info("%s\n", __func__);
+	return 0;
+}
+
+static void livepatch_callbacks_mod_exit(void)
+{
+	pr_info("%s\n", __func__);
+}
+
+module_init(livepatch_callbacks_mod_init);
+module_exit(livepatch_callbacks_mod_exit);
+MODULE_LICENSE("GPL");
diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 7442dfb73b7f..5dea4632a297 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -15,6 +15,7 @@ TARGETS += intel_pstate
 TARGETS += ipc
 TARGETS += kcmp
 TARGETS += lib
+TARGETS += livepatching
 TARGETS += membarrier
 TARGETS += memfd
 TARGETS += memory-hotplug
diff --git a/tools/testing/selftests/livepatch/Makefile b/tools/testing/selftests/livepatch/Makefile
new file mode 100644
index 000000000000..d2cb02b95be7
--- /dev/null
+++ b/tools/testing/selftests/livepatch/Makefile
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+
+TEST_GEN_PROGS := livepatch-test
+
+include ../lib.mk
diff --git a/tools/testing/selftests/livepatch/config b/tools/testing/selftests/livepatch/config
new file mode 100644
index 000000000000..d16e7bc0033d
--- /dev/null
+++ b/tools/testing/selftests/livepatch/config
@@ -0,0 +1 @@
+CONFIG_TEST_LIVEPATCH=y
diff --git a/tools/testing/selftests/livepatch/livepatch-test b/tools/testing/selftests/livepatch/livepatch-test
new file mode 100755
index 000000000000..798317bf69f6
--- /dev/null
+++ b/tools/testing/selftests/livepatch/livepatch-test
@@ -0,0 +1,658 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2018 Joe Lawrence <joe.lawrence@redhat.com>
+
+MAX_RETRIES=30
+RETRY_INTERVAL=2	# seconds
+BETWEEN_TESTS=20	# seconds
+
+MOD_LIVEPATCH=test_klp_callbacks_demo
+MOD_LIVEPATCH2=test_klp_callbacks_demo2
+MOD_TARGET=test_klp_callbacks_mod
+MOD_TARGET_BUSY=test_klp_callbacks_busy
+
+# die() - game over, man
+function die() {
+	echo "ERROR: $1" >&2
+	exit 1
+}
+
+# set_dynamic_debug() - setup kernel dynamic debug
+#	TODO - push and pop this config?
+function set_dynamic_debug() {
+	cat << EOF > /sys/kernel/debug/dynamic_debug/control
+file kernel/livepatch/* +p
+func klp_try_switch_task -p
+EOF
+}
+
+# wait_for_transition(modname)
+#	modname - livepatch module name
+wait_for_transition() {
+	local mod="$1"; shift
+
+	# Wait for livepatch transition  ...
+	local i=0
+	while [[ $(cat /sys/kernel/livepatch/"$mod"/transition) != "0" ]]; do
+		i=$((i+1))
+		if [[ $i = "$MAX_RETRIES" ]]; then
+			die "failed to complete transition for module $mod"
+		fi
+		sleep $RETRY_INTERVAL
+	done
+}
+
+# load_mod(modname, params) - load a kernel module
+#	modname - module name to load
+#       params  - module parameters to pass to modprobe
+function load_mod() {
+	local mod="$1"; shift
+	local args="$*"
+
+	local msg="% modprobe $mod $args"
+	echo "${msg%% }" > /dev/kmsg
+	ret=$(modprobe "$mod" "$args" 2>&1)
+	if [[ "$ret" != "" ]]; then
+		echo "$ret" > /dev/kmsg
+		die "$ret"
+	fi
+
+	# Wait for module in sysfs ...
+	local i=0
+	while [ ! -e /sys/module/"$mod" ]; do
+		i=$((i+1))
+		if [[ $i = "$MAX_RETRIES" ]]; then
+			die "failed to load module $mod"
+		fi
+		sleep $RETRY_INTERVAL
+	done
+
+	# Wait for livepatch ...
+	if [[ $(modinfo "$mod" | awk '/^livepatch:/{print $NF}') == "Y" ]]; then
+
+		# Wait for livepatch in sysfs ...
+		local i=0
+		while [ ! -e /sys/kernel/livepatch/"$mod" ]; do
+			i=$((i+1))
+			if [[ $i = "$MAX_RETRIES" ]]; then
+				die "failed to load module $mod (sysfs)"
+			fi
+			sleep $RETRY_INTERVAL
+		done
+	fi
+}
+
+# load_failing_mod(modname, params) - load a kernel module, expect to fail
+#	modname - module name to load
+#       params  - module parameters to pass to modprobe
+function load_failing_mod() {
+	local mod="$1"; shift
+	local args="$*"
+
+	local msg="% modprobe $mod $args"
+	echo "${msg%% }" > /dev/kmsg
+	ret=$(modprobe "$mod" "$args" 2>&1)
+	if [[ "$ret" == "" ]]; then
+		echo "$mod unexpectedly loaded" > /dev/kmsg
+		die "$mod unexpectedly loaded"
+	fi
+	echo "$ret" > /dev/kmsg
+}
+
+# unload_mod(modname) - unload a kernel module
+#	modname - module name to unload
+function unload_mod() {
+	local mod="$1"
+
+	# Wait for module reference count to clear ...
+	local i=0
+	while [[ $(cat /sys/module/"$mod"/refcnt) != "0" ]]; do
+		i=$((i+1))
+		if [[ $i = "$MAX_RETRIES" ]]; then
+			die "failed to unload module $mod (refcnt)"
+		fi
+		sleep $RETRY_INTERVAL
+	done
+
+	echo "% rmmod $mod" > /dev/kmsg
+	ret=$(rmmod "$mod" 2>&1)
+	if [[ "$ret" != "" ]]; then
+		echo "$ret" > /dev/kmsg
+		die "$ret"
+	fi
+
+	# Wait for module in sysfs ...
+	local i=0
+	while [ -e /sys/module/"$mod" ]; do
+		i=$((i+1))
+		if [[ $i = "$MAX_RETRIES" ]]; then
+			die "failed to unload module $mod (/sys/module)"
+		fi
+		sleep $RETRY_INTERVAL
+	done
+
+	# Wait for livepatch sysfs if applicable ...
+	if [[ $(modinfo "$mod" | awk '/^livepatch:/{print $NF}') == "Y" ]]; then
+
+		local i=0
+		while [ -e /sys/kernel/livepatch/"$mod" ]; do
+			i=$((i+1))
+			if [[ $i = "$MAX_RETRIES" ]]; then
+				die "failed to unload module $mod (/sys/livepatch)"
+			fi
+			sleep $RETRY_INTERVAL
+		done
+	fi
+}
+
+# display_lp(modname) - disable a livepatch
+#	modname - module name to unload
+function disable_lp() {
+	local mod="$1"
+
+        echo "% echo 0 > /sys/kernel/livepatch/$mod/enabled" > /dev/kmsg
+        echo 0 > /sys/kernel/livepatch/"$mod"/enabled
+
+	# Wait for livepatch enable to clear ...
+	local i=0
+	while [[ $(cat /sys/kernel/livepatch/"$mod"/enabled) != "0" ]]; do
+		i=$((i+1))
+		if [[ $i = "$MAX_RETRIES" ]]; then
+			die "failed to disable livepatch $mod"
+		fi
+		sleep $RETRY_INTERVAL
+	done
+}
+
+# set_pre_patch_ret(modname, pre_patch_ret)
+#	modname - module name to set
+#	pre_patch_ret - new pre_patch_ret value
+function set_pre_patch_ret {
+	local mod="$1"; shift
+        local ret="$1"
+
+        echo "% echo $1 > /sys/module/$mod/parameters/pre_patch_ret" > /dev/kmsg
+        echo "$1" > /sys/module/"$mod"/parameters/pre_patch_ret
+
+	local i=0
+	while [[ $(cat /sys/module/"$mod"/parameters/pre_patch_ret) != "$1" ]]; do
+		i=$((i+1))
+		if [[ $i = "$MAX_RETRIES" ]]; then
+			die "failed to set pre_patch_ret parameter for $mod module"
+		fi
+		sleep $RETRY_INTERVAL
+	done
+}
+
+# filter_dmesg() - print a filtered dmesg
+#	TODO - better filter, out of order msgs, etc?
+function check_result {
+	local expect="$*"
+	local result=$(dmesg | grep -v 'tainting' | grep -e 'livepatch:' -e 'test_klp' | sed 's/^\[[ 0-9.]*\] //')
+
+	if [[ "$expect" == "$result" ]] ; then
+		echo "PASS"
+	else
+		echo -e "FAIL\n\n$(diff -upr --label expected --label result <(echo "$expect") <(echo "$result"))"
+		die "livepatch kselftest(s) failed"
+	fi
+}
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+set_dynamic_debug
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+echo -n "TEST1 ... "
+dmesg -C
+
+load_mod $MOD_TARGET
+load_mod $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+disable_lp $MOD_LIVEPATCH
+unload_mod $MOD_LIVEPATCH
+unload_mod $MOD_TARGET
+
+check_result "% modprobe $MOD_TARGET
+$MOD_TARGET: livepatch_callbacks_mod_init
+% modprobe $MOD_LIVEPATCH
+livepatch: enabling patch '$MOD_LIVEPATCH'
+livepatch: '$MOD_LIVEPATCH': initializing patching transition
+$MOD_LIVEPATCH: pre_patch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
+$MOD_LIVEPATCH: pre_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting patching transition
+livepatch: '$MOD_LIVEPATCH': completing patching transition
+$MOD_LIVEPATCH: post_patch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
+$MOD_LIVEPATCH: post_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': patching complete
+% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled
+livepatch: '$MOD_LIVEPATCH': initializing unpatching transition
+$MOD_LIVEPATCH: pre_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
+$MOD_LIVEPATCH: pre_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting unpatching transition
+livepatch: '$MOD_LIVEPATCH': completing unpatching transition
+$MOD_LIVEPATCH: post_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
+$MOD_LIVEPATCH: post_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': unpatching complete
+% rmmod $MOD_LIVEPATCH
+% rmmod $MOD_TARGET
+$MOD_TARGET: livepatch_callbacks_mod_exit"
+
+sleep $BETWEEN_TESTS
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+echo -n "TEST2 ... "
+dmesg -C
+
+load_mod $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+load_mod $MOD_TARGET
+disable_lp $MOD_LIVEPATCH
+unload_mod $MOD_LIVEPATCH
+unload_mod $MOD_TARGET
+
+check_result "% modprobe $MOD_LIVEPATCH
+livepatch: enabling patch '$MOD_LIVEPATCH'
+livepatch: '$MOD_LIVEPATCH': initializing patching transition
+$MOD_LIVEPATCH: pre_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting patching transition
+livepatch: '$MOD_LIVEPATCH': completing patching transition
+$MOD_LIVEPATCH: post_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': patching complete
+% modprobe $MOD_TARGET
+livepatch: applying patch '$MOD_LIVEPATCH' to loading module '$MOD_TARGET'
+$MOD_LIVEPATCH: pre_patch_callback: $MOD_TARGET -> [MODULE_STATE_COMING] Full formed, running module_init
+$MOD_LIVEPATCH: post_patch_callback: $MOD_TARGET -> [MODULE_STATE_COMING] Full formed, running module_init
+$MOD_TARGET: livepatch_callbacks_mod_init
+% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled
+livepatch: '$MOD_LIVEPATCH': initializing unpatching transition
+$MOD_LIVEPATCH: pre_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
+$MOD_LIVEPATCH: pre_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting unpatching transition
+livepatch: '$MOD_LIVEPATCH': completing unpatching transition
+$MOD_LIVEPATCH: post_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
+$MOD_LIVEPATCH: post_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': unpatching complete
+% rmmod $MOD_LIVEPATCH
+% rmmod $MOD_TARGET
+$MOD_TARGET: livepatch_callbacks_mod_exit"
+
+sleep $BETWEEN_TESTS
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+echo -n "TEST3 ... "
+dmesg -C
+
+load_mod $MOD_TARGET
+load_mod $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+unload_mod $MOD_TARGET
+disable_lp $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+unload_mod $MOD_LIVEPATCH
+
+check_result "% modprobe $MOD_TARGET
+$MOD_TARGET: livepatch_callbacks_mod_init
+% modprobe $MOD_LIVEPATCH
+livepatch: enabling patch '$MOD_LIVEPATCH'
+livepatch: '$MOD_LIVEPATCH': initializing patching transition
+$MOD_LIVEPATCH: pre_patch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
+$MOD_LIVEPATCH: pre_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting patching transition
+livepatch: '$MOD_LIVEPATCH': completing patching transition
+$MOD_LIVEPATCH: post_patch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
+$MOD_LIVEPATCH: post_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': patching complete
+% rmmod $MOD_TARGET
+$MOD_TARGET: livepatch_callbacks_mod_exit
+$MOD_LIVEPATCH: pre_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_GOING] Going away
+livepatch: reverting patch '$MOD_LIVEPATCH' on unloading module '$MOD_TARGET'
+$MOD_LIVEPATCH: post_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_GOING] Going away
+% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled
+livepatch: '$MOD_LIVEPATCH': initializing unpatching transition
+$MOD_LIVEPATCH: pre_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting unpatching transition
+livepatch: '$MOD_LIVEPATCH': completing unpatching transition
+$MOD_LIVEPATCH: post_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': unpatching complete
+% rmmod $MOD_LIVEPATCH"
+
+sleep $BETWEEN_TESTS
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+echo -n "TEST4 ... "
+dmesg -C
+
+load_mod $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+load_mod $MOD_TARGET
+unload_mod $MOD_TARGET
+disable_lp $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+unload_mod $MOD_LIVEPATCH
+
+check_result "% modprobe $MOD_LIVEPATCH
+livepatch: enabling patch '$MOD_LIVEPATCH'
+livepatch: '$MOD_LIVEPATCH': initializing patching transition
+$MOD_LIVEPATCH: pre_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting patching transition
+livepatch: '$MOD_LIVEPATCH': completing patching transition
+$MOD_LIVEPATCH: post_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': patching complete
+% modprobe $MOD_TARGET
+livepatch: applying patch '$MOD_LIVEPATCH' to loading module '$MOD_TARGET'
+$MOD_LIVEPATCH: pre_patch_callback: $MOD_TARGET -> [MODULE_STATE_COMING] Full formed, running module_init
+$MOD_LIVEPATCH: post_patch_callback: $MOD_TARGET -> [MODULE_STATE_COMING] Full formed, running module_init
+$MOD_TARGET: livepatch_callbacks_mod_init
+% rmmod $MOD_TARGET
+$MOD_TARGET: livepatch_callbacks_mod_exit
+$MOD_LIVEPATCH: pre_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_GOING] Going away
+livepatch: reverting patch '$MOD_LIVEPATCH' on unloading module '$MOD_TARGET'
+$MOD_LIVEPATCH: post_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_GOING] Going away
+% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled
+livepatch: '$MOD_LIVEPATCH': initializing unpatching transition
+$MOD_LIVEPATCH: pre_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting unpatching transition
+livepatch: '$MOD_LIVEPATCH': completing unpatching transition
+$MOD_LIVEPATCH: post_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': unpatching complete
+% rmmod $MOD_LIVEPATCH"
+
+sleep $BETWEEN_TESTS
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+echo -n "TEST5 ... "
+dmesg -C
+
+load_mod $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+disable_lp $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+unload_mod $MOD_LIVEPATCH
+
+check_result "% modprobe $MOD_LIVEPATCH
+livepatch: enabling patch '$MOD_LIVEPATCH'
+livepatch: '$MOD_LIVEPATCH': initializing patching transition
+$MOD_LIVEPATCH: pre_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting patching transition
+livepatch: '$MOD_LIVEPATCH': completing patching transition
+$MOD_LIVEPATCH: post_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': patching complete
+% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled
+livepatch: '$MOD_LIVEPATCH': initializing unpatching transition
+$MOD_LIVEPATCH: pre_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting unpatching transition
+livepatch: '$MOD_LIVEPATCH': completing unpatching transition
+$MOD_LIVEPATCH: post_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': unpatching complete
+% rmmod $MOD_LIVEPATCH"
+
+sleep $BETWEEN_TESTS
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+echo -n "TEST6 ... "
+dmesg -C
+
+load_mod $MOD_TARGET
+load_failing_mod $MOD_LIVEPATCH pre_patch_ret=-19
+unload_mod $MOD_TARGET
+
+check_result "% modprobe $MOD_TARGET
+$MOD_TARGET: livepatch_callbacks_mod_init
+% modprobe $MOD_LIVEPATCH pre_patch_ret=-19
+livepatch: enabling patch '$MOD_LIVEPATCH'
+livepatch: '$MOD_LIVEPATCH': initializing patching transition
+$MOD_LIVEPATCH: pre_patch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
+livepatch: pre-patch callback failed for object '$MOD_TARGET'
+livepatch: failed to enable patch '$MOD_LIVEPATCH'
+livepatch: '$MOD_LIVEPATCH': canceling patching transition, going to unpatch
+livepatch: '$MOD_LIVEPATCH': completing unpatching transition
+livepatch: '$MOD_LIVEPATCH': unpatching complete
+modprobe: ERROR: could not insert '$MOD_LIVEPATCH': No such device
+% rmmod $MOD_TARGET
+$MOD_TARGET: livepatch_callbacks_mod_exit"
+
+sleep $BETWEEN_TESTS
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+echo -n "TEST7 ... "
+dmesg -C
+
+load_mod $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+set_pre_patch_ret $MOD_LIVEPATCH -19
+load_failing_mod $MOD_TARGET
+disable_lp $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+unload_mod $MOD_LIVEPATCH
+
+check_result "% modprobe $MOD_LIVEPATCH
+livepatch: enabling patch '$MOD_LIVEPATCH'
+livepatch: '$MOD_LIVEPATCH': initializing patching transition
+$MOD_LIVEPATCH: pre_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting patching transition
+livepatch: '$MOD_LIVEPATCH': completing patching transition
+$MOD_LIVEPATCH: post_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': patching complete
+% echo -19 > /sys/module/$MOD_LIVEPATCH/parameters/pre_patch_ret
+% modprobe $MOD_TARGET
+livepatch: applying patch '$MOD_LIVEPATCH' to loading module '$MOD_TARGET'
+$MOD_LIVEPATCH: pre_patch_callback: $MOD_TARGET -> [MODULE_STATE_COMING] Full formed, running module_init
+livepatch: pre-patch callback failed for object '$MOD_TARGET'
+livepatch: patch '$MOD_LIVEPATCH' failed for module '$MOD_TARGET', refusing to load module '$MOD_TARGET'
+modprobe: ERROR: could not insert '$MOD_TARGET': No such device
+% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled
+livepatch: '$MOD_LIVEPATCH': initializing unpatching transition
+$MOD_LIVEPATCH: pre_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting unpatching transition
+livepatch: '$MOD_LIVEPATCH': completing unpatching transition
+$MOD_LIVEPATCH: post_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': unpatching complete
+% rmmod $MOD_LIVEPATCH"
+
+sleep $BETWEEN_TESTS
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+echo -n "TEST8 ... "
+dmesg -C
+
+load_mod $MOD_TARGET_BUSY sleep_secs=0
+# give $MOD_TARGET_BUSY::busymod_work_func() a chance to run
+sleep 5
+load_mod $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+load_mod $MOD_TARGET
+unload_mod $MOD_TARGET
+disable_lp $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+unload_mod $MOD_LIVEPATCH
+unload_mod $MOD_TARGET_BUSY
+
+check_result "% modprobe $MOD_TARGET_BUSY sleep_secs=0
+$MOD_TARGET_BUSY: livepatch_callbacks_mod_init
+$MOD_TARGET_BUSY: busymod_work_func, sleeping 0 seconds ...
+$MOD_TARGET_BUSY: busymod_work_func exit
+% modprobe $MOD_LIVEPATCH
+livepatch: enabling patch '$MOD_LIVEPATCH'
+livepatch: '$MOD_LIVEPATCH': initializing patching transition
+$MOD_LIVEPATCH: pre_patch_callback: $MOD_TARGET_BUSY -> [MODULE_STATE_LIVE] Normal state
+$MOD_LIVEPATCH: pre_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting patching transition
+livepatch: '$MOD_LIVEPATCH': completing patching transition
+$MOD_LIVEPATCH: post_patch_callback: $MOD_TARGET_BUSY -> [MODULE_STATE_LIVE] Normal state
+$MOD_LIVEPATCH: post_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': patching complete
+% modprobe $MOD_TARGET
+livepatch: applying patch '$MOD_LIVEPATCH' to loading module '$MOD_TARGET'
+$MOD_LIVEPATCH: pre_patch_callback: $MOD_TARGET -> [MODULE_STATE_COMING] Full formed, running module_init
+$MOD_LIVEPATCH: post_patch_callback: $MOD_TARGET -> [MODULE_STATE_COMING] Full formed, running module_init
+$MOD_TARGET: livepatch_callbacks_mod_init
+% rmmod $MOD_TARGET
+$MOD_TARGET: livepatch_callbacks_mod_exit
+$MOD_LIVEPATCH: pre_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_GOING] Going away
+livepatch: reverting patch '$MOD_LIVEPATCH' on unloading module '$MOD_TARGET'
+$MOD_LIVEPATCH: post_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_GOING] Going away
+% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled
+livepatch: '$MOD_LIVEPATCH': initializing unpatching transition
+$MOD_LIVEPATCH: pre_unpatch_callback: $MOD_TARGET_BUSY -> [MODULE_STATE_LIVE] Normal state
+$MOD_LIVEPATCH: pre_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting unpatching transition
+livepatch: '$MOD_LIVEPATCH': completing unpatching transition
+$MOD_LIVEPATCH: post_unpatch_callback: $MOD_TARGET_BUSY -> [MODULE_STATE_LIVE] Normal state
+$MOD_LIVEPATCH: post_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': unpatching complete
+% rmmod $MOD_LIVEPATCH
+% rmmod $MOD_TARGET_BUSY
+$MOD_TARGET_BUSY: livepatch_callbacks_mod_exit"
+
+sleep $BETWEEN_TESTS
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+echo -n "TEST9 ... "
+dmesg -C
+
+load_mod $MOD_TARGET_BUSY sleep_secs=60
+load_mod $MOD_LIVEPATCH
+# Don't wait for transition, load $MOD_TARGET while the transition
+# is still stalled in $MOD_TARGET_BUSY::busymod_work_func()
+sleep 5
+load_mod $MOD_TARGET
+unload_mod $MOD_TARGET
+disable_lp $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+unload_mod $MOD_LIVEPATCH
+unload_mod $MOD_TARGET_BUSY
+
+check_result "% modprobe $MOD_TARGET_BUSY sleep_secs=60
+$MOD_TARGET_BUSY: livepatch_callbacks_mod_init
+$MOD_TARGET_BUSY: busymod_work_func, sleeping 60 seconds ...
+% modprobe $MOD_LIVEPATCH
+livepatch: enabling patch '$MOD_LIVEPATCH'
+livepatch: '$MOD_LIVEPATCH': initializing patching transition
+$MOD_LIVEPATCH: pre_patch_callback: $MOD_TARGET_BUSY -> [MODULE_STATE_LIVE] Normal state
+$MOD_LIVEPATCH: pre_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting patching transition
+% modprobe $MOD_TARGET
+livepatch: applying patch '$MOD_LIVEPATCH' to loading module '$MOD_TARGET'
+$MOD_LIVEPATCH: pre_patch_callback: $MOD_TARGET -> [MODULE_STATE_COMING] Full formed, running module_init
+$MOD_TARGET: livepatch_callbacks_mod_init
+% rmmod $MOD_TARGET
+$MOD_TARGET: livepatch_callbacks_mod_exit
+livepatch: reverting patch '$MOD_LIVEPATCH' on unloading module '$MOD_TARGET'
+$MOD_LIVEPATCH: post_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_GOING] Going away
+% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled
+livepatch: '$MOD_LIVEPATCH': reversing transition from patching to unpatching
+livepatch: '$MOD_LIVEPATCH': starting unpatching transition
+livepatch: '$MOD_LIVEPATCH': completing unpatching transition
+$MOD_LIVEPATCH: post_unpatch_callback: $MOD_TARGET_BUSY -> [MODULE_STATE_LIVE] Normal state
+$MOD_LIVEPATCH: post_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': unpatching complete
+% rmmod $MOD_LIVEPATCH
+% rmmod $MOD_TARGET_BUSY
+$MOD_TARGET_BUSY: busymod_work_func exit
+$MOD_TARGET_BUSY: livepatch_callbacks_mod_exit"
+
+sleep $BETWEEN_TESTS
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+echo -n "TEST10 ... "
+dmesg -C
+
+load_mod $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+load_mod $MOD_LIVEPATCH2
+wait_for_transition $MOD_LIVEPATCH2
+disable_lp $MOD_LIVEPATCH2
+wait_for_transition $MOD_LIVEPATCH2
+disable_lp $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+unload_mod $MOD_LIVEPATCH2
+unload_mod $MOD_LIVEPATCH
+
+check_result "% modprobe $MOD_LIVEPATCH
+livepatch: enabling patch '$MOD_LIVEPATCH'
+livepatch: '$MOD_LIVEPATCH': initializing patching transition
+$MOD_LIVEPATCH: pre_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting patching transition
+livepatch: '$MOD_LIVEPATCH': completing patching transition
+$MOD_LIVEPATCH: post_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': patching complete
+% modprobe $MOD_LIVEPATCH2
+livepatch: enabling patch '$MOD_LIVEPATCH2'
+livepatch: '$MOD_LIVEPATCH2': initializing patching transition
+$MOD_LIVEPATCH2: pre_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH2': starting patching transition
+livepatch: '$MOD_LIVEPATCH2': completing patching transition
+$MOD_LIVEPATCH2: post_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH2': patching complete
+% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH2/enabled
+livepatch: '$MOD_LIVEPATCH2': initializing unpatching transition
+$MOD_LIVEPATCH2: pre_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH2': starting unpatching transition
+livepatch: '$MOD_LIVEPATCH2': completing unpatching transition
+$MOD_LIVEPATCH2: post_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH2': unpatching complete
+% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled
+livepatch: '$MOD_LIVEPATCH': initializing unpatching transition
+$MOD_LIVEPATCH: pre_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting unpatching transition
+livepatch: '$MOD_LIVEPATCH': completing unpatching transition
+$MOD_LIVEPATCH: post_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': unpatching complete
+% rmmod $MOD_LIVEPATCH2
+% rmmod $MOD_LIVEPATCH"
+
+sleep $BETWEEN_TESTS
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+echo -n "TEST11 ... "
+dmesg -C
+
+load_mod $MOD_LIVEPATCH
+wait_for_transition $MOD_LIVEPATCH
+load_mod $MOD_LIVEPATCH2 replace=1
+wait_for_transition $MOD_LIVEPATCH2
+disable_lp $MOD_LIVEPATCH2
+wait_for_transition $MOD_LIVEPATCH2
+unload_mod $MOD_LIVEPATCH2
+unload_mod $MOD_LIVEPATCH
+
+check_result "% modprobe $MOD_LIVEPATCH
+livepatch: enabling patch '$MOD_LIVEPATCH'
+livepatch: '$MOD_LIVEPATCH': initializing patching transition
+$MOD_LIVEPATCH: pre_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': starting patching transition
+livepatch: '$MOD_LIVEPATCH': completing patching transition
+$MOD_LIVEPATCH: post_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH': patching complete
+% modprobe $MOD_LIVEPATCH2 replace=1
+livepatch: enabling patch '$MOD_LIVEPATCH2'
+livepatch: '$MOD_LIVEPATCH2': initializing patching transition
+$MOD_LIVEPATCH2: pre_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH2': starting patching transition
+livepatch: '$MOD_LIVEPATCH2': completing patching transition
+$MOD_LIVEPATCH2: post_patch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH2': patching complete
+% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH2/enabled
+livepatch: '$MOD_LIVEPATCH2': initializing unpatching transition
+$MOD_LIVEPATCH2: pre_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH2': starting unpatching transition
+livepatch: '$MOD_LIVEPATCH2': completing unpatching transition
+$MOD_LIVEPATCH2: post_unpatch_callback: vmlinux
+livepatch: '$MOD_LIVEPATCH2': unpatching complete
+% rmmod $MOD_LIVEPATCH2
+% rmmod $MOD_LIVEPATCH"
+
+echo "livepatch kselftest(s) success"
+exit 0
-- 
1.8.3.1

Re: [PATCH v10 07/10] livepatch: Correctly handle atomic replace for not yet loaded modules

[Petr Mladek]

On Wed 2018-03-07 09:20:36, Petr Mladek wrote:
> The atomic replace feature uses dynamically allocated struct klp_func to
> handle functions that will no longer be patched. These structures are
> of the type KLP_FUNC_NOP. They cause the ftrace handler to jump to
> the original code. But the address of the original code is not known
> until the patched module is loaded.
> 
> This patch allows the late initialization. Also it adds a sanity check
> into the ftrace handler.
> 
> diff --git a/kernel/livepatch/patch.c b/kernel/livepatch/patch.c
> index 54b3c379bb0f..1f5c3eea9ee1 100644
> --- a/kernel/livepatch/patch.c
> +++ b/kernel/livepatch/patch.c
> @@ -118,7 +118,12 @@ static void notrace klp_ftrace_handler(unsigned long ip,
>  		}
>  	}
>  
> +	/* Survive ugly mistakes, for example, when handling NOPs. */
> +	if (WARN_ON_ONCE(!func->new_func))
> +		goto unlock;

JFYI, this is not enough. We really have to skip klp_arch_set_pc()
for NOPs. Otherwise, we end up in an infinite loop. NOPs cause that
we go back to the beginning of the original function, enter
the ftrace handler again, ...

I am going to fix this in v11.

> +
>  	klp_arch_set_pc(regs, (unsigned long)func->new_func);
> +
>  unlock:
>  	preempt_enable_notrace();

Kudos to Mirek for testing and hitting the problem.

Best Regards,
Petr

Re: [PATCH v10 01/10] livepatch: Use lists to manage patches, objects and functions

[Josh Poimboeuf]

On Wed, Mar 07, 2018 at 09:20:30AM +0100, Petr Mladek wrote:
> From: Jason Baron <jbaron@akamai.com>
> 
> Currently klp_patch contains a pointer to a statically allocated array of
> struct klp_object and struct klp_objects contains a pointer to a statically
> allocated array of klp_func. In order to allow for the dynamic allocation
> of objects and functions, link klp_patch, klp_object, and klp_func together
> via linked lists. This allows us to more easily allocate new objects and
> functions, while having the iterator be a simple linked list walk.
> 
> The static structures are added to the lists early. It allows to add
> the dynamically allocated objects before klp_init_object() and
> klp_init_func() calls. Therefore it reduces the further changes
> to the code.
> 
> Also klp_init_*_list() functions are split because they will
> be used when adding the dynamically allocated structures.
> 
> This patch does not change the existing behavior.
> 
> Signed-off-by: Jason Baron <jbaron@akamai.com>
> [pmladek@suse.com: Initialize lists before init calls]
> Signed-off-by: Petr Mladek <pmladek@suse.com>
> Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> Cc: Jessica Yu <jeyu@kernel.org>
> Cc: Jiri Kosina <jikos@kernel.org>
> Acked-by: Miroslav Benes <mbenes@suse.cz>
> ---
>  include/linux/livepatch.h | 19 +++++++++++++++++--
>  kernel/livepatch/core.c   | 27 +++++++++++++++++++++++++++
>  2 files changed, 44 insertions(+), 2 deletions(-)
> 
> diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
> index 4754f01c1abb..e5db2ba7e2a5 100644
> --- a/include/linux/livepatch.h
> +++ b/include/linux/livepatch.h
> @@ -24,6 +24,7 @@
>  #include <linux/module.h>
>  #include <linux/ftrace.h>
>  #include <linux/completion.h>
> +#include <linux/list.h>
>  
>  #if IS_ENABLED(CONFIG_LIVEPATCH)
>  
> @@ -43,6 +44,7 @@
>   * @old_addr:	the address of the function being patched
>   * @kobj:	kobject for sysfs resources
>   * @stack_node:	list node for klp_ops func_stack list
> + * @func_entry:	links struct klp_func to struct klp_object

For consistency with 'stack_node', I think the field should have 'node'
in the name.  Also, putting 'func' in the name is redundant because the
struct already has 'func' in its name.  IMO, just 'node' would be good.

I also found the description to be confusing.  How about something
similar to the 'stack_node' description:

    * @node:		list node for klp_object func list

>   * @old_size:	size of the old function
>   * @new_size:	size of the new function
>   * @patched:	the func has been added to the klp_ops list
> @@ -80,6 +82,7 @@ struct klp_func {
>  	unsigned long old_addr;
>  	struct kobject kobj;
>  	struct list_head stack_node;
> +	struct list_head func_entry;
>  	unsigned long old_size, new_size;
>  	bool patched;
>  	bool transition;
> @@ -117,6 +120,8 @@ struct klp_callbacks {
>   * @kobj:	kobject for sysfs resources
>   * @mod:	kernel module associated with the patched object
>   *		(NULL for vmlinux)
> + * @func_list:	head of list for struct klp_func
> + * @obj_entry:	links struct klp_object to struct klp_patch

Similar comments here.

>   * @patched:	the object's funcs have been added to the klp_ops list
>   */
>  struct klp_object {
> @@ -127,6 +132,8 @@ struct klp_object {
>  
>  	/* internal */
>  	struct kobject kobj;
> +	struct list_head func_list;
> +	struct list_head obj_entry;
>  	struct module *mod;
>  	bool patched;
>  };
> @@ -137,6 +144,7 @@ struct klp_object {
>   * @objs:	object entries for kernel objects to be patched
>   * @list:	list node for global list of registered patches
>   * @kobj:	kobject for sysfs resources
> + * @obj_list:	head of list for struct klp_object
>   * @enabled:	the patch is enabled (but operation may be incomplete)
>   * @finish:	for waiting till it is safe to remove the patch module
>   */
> @@ -148,18 +156,25 @@ struct klp_patch {
>  	/* internal */
>  	struct list_head list;
>  	struct kobject kobj;
> +	struct list_head obj_list;
>  	bool enabled;
>  	struct completion finish;
>  };
>  
> -#define klp_for_each_object(patch, obj) \
> +#define klp_for_each_object_static(patch, obj) \
>  	for (obj = patch->objs; obj->funcs || obj->name; obj++)
>  
> -#define klp_for_each_func(obj, func) \
> +#define klp_for_each_object(patch, obj)	\
> +	list_for_each_entry(obj, &patch->obj_list, obj_entry)
> +
> +#define klp_for_each_func_static(obj, func) \
>  	for (func = obj->funcs; \
>  	     func->old_name || func->new_func || func->old_sympos; \
>  	     func++)
>  
> +#define klp_for_each_func(obj, func)	\
> +	list_for_each_entry(func, &obj->func_list, func_entry)
> +
>  int klp_register_patch(struct klp_patch *);
>  int klp_unregister_patch(struct klp_patch *);
>  int klp_enable_patch(struct klp_patch *);
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index 3a4656fb7047..1d525f4a270a 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -49,6 +49,32 @@ static LIST_HEAD(klp_patches);
>  
>  static struct kobject *klp_root_kobj;
>  
> +static void klp_init_func_list(struct klp_object *obj, struct klp_func *func)
> +{
> +	list_add(&func->func_entry, &obj->func_list);
> +}

This function is confusingly named.  It *adds* to the func list instead
of initializing it.

Also, I think wrapping a single list_add() in a function makes the
calling code harder to read and understand.  I would suggest getting rid
of the function altogether and just inlining the list_add().

I also found the code harder to read in several other places, for
similar reasons (several small single-use functions).  I call it
functionitis :-)

> +
> +static void klp_init_object_list(struct klp_patch *patch,
> +				 struct klp_object *obj)
> +{
> +	struct klp_func *func;
> +
> +	list_add(&obj->obj_entry, &patch->obj_list);
> +
> +	INIT_LIST_HEAD(&obj->func_list);
> +	klp_for_each_func_static(obj, func)
> +		klp_init_func_list(obj, func);
> +}
> +
> +static void klp_init_patch_list(struct klp_patch *patch)
> +{
> +	struct klp_object *obj;
> +
> +	INIT_LIST_HEAD(&patch->obj_list);
> +	klp_for_each_object_static(patch, obj)
> +		klp_init_object_list(patch, obj);
> +}

klp_init_patch_list() can be a confusing name.  To me it sounds like a
list of patches.  But really it's a list of objects.

The same goes for klp_init_object_list().

What if we just call it klp_init_lists(), and move
klp_init_object_list() inline?

I know a later patch also adds a call to klp_init_object_list() from
klp_get_or_add_object(), but I think it should be inlined there as well,
because all that function needs to do is the list_add() and the
INIT_LIST_HEAD() -- the object is freshly allocated so it doesn't need
to copy any funcs to the list.  IMO it's more readable and more explicit
to repeat the few minor and obvious lines of code than to hide them in a
wrapper function which also does other things for other callers.

All this will help with some of the functionitis-related readability
issues, IMO.

> +
>  static bool klp_is_module(struct klp_object *obj)
>  {
>  	return obj->name;
> @@ -794,6 +820,7 @@ static int klp_init_patch(struct klp_patch *patch)
>  
>  	patch->enabled = false;
>  	init_completion(&patch->finish);
> +	klp_init_patch_list(patch);
>  
>  	ret = kobject_init_and_add(&patch->kobj, &klp_ktype_patch,
>  				   klp_root_kobj, "%s", patch->mod->name);
> -- 
> 2.13.6
> 

-- 
Josh

Re: [PATCH v10 02/10] livepatch: Free only structures with initialized kobject

[Josh Poimboeuf]

On Wed, Mar 07, 2018 at 09:20:31AM +0100, Petr Mladek wrote:
> We are going to add a feature called atomic replace. It will allow to
> create a patch that would replace all already registered patches.
> For this, we will need to dynamically create funcs and objects
> for functions that are no longer patched.
> 
> We will want to reuse the existing init() and free() functions. Up to now,
> the free() functions checked a limit and were called only for structures
> with initialized kobject. But we will want to call them also for structures
> that were allocated but where the kobject was not initialized yet.
> 
> This patch removes the limit. It calls klp_free*() functions for all
> structures. But only the ones with initialized kobject are freed.
> The handling of un-initialized structures will be added later with
> the support for dynamic structures.
> 
> This patch does not change the existing behavior.
> 
> Signed-off-by: Petr Mladek <pmladek@suse.com>
> Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> Cc: Jessica Yu <jeyu@kernel.org>
> Cc: Jiri Kosina <jikos@kernel.org>
> Cc: Jason Baron <jbaron@akamai.com>
> Acked-by: Miroslav Benes <mbenes@suse.cz>
> ---
>  kernel/livepatch/core.c | 44 ++++++++++++++++++--------------------------
>  1 file changed, 18 insertions(+), 26 deletions(-)
> 
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index 1d525f4a270a..69bde95e76f8 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -653,17 +653,15 @@ static struct kobj_type klp_ktype_func = {
>  	.sysfs_ops = &kobj_sysfs_ops,
>  };
>  
> -/*
> - * Free all functions' kobjects in the array up to some limit. When limit is
> - * NULL, all kobjects are freed.
> - */
> -static void klp_free_funcs_limited(struct klp_object *obj,
> -				   struct klp_func *limit)
> +/* Free all funcs that have the kobject initialized. */
> +static void klp_free_funcs(struct klp_object *obj)
>  {
>  	struct klp_func *func;
>  
> -	for (func = obj->funcs; func->old_name && func != limit; func++)
> -		kobject_put(&func->kobj);
> +	klp_for_each_func(obj, func) {
> +		if (func->kobj.state_initialized)
> +			kobject_put(&func->kobj);
> +	}
>  }

Now that this function only has a single caller, personally I think it
would become more readable if klp_free_funcs() were inlined into its
caller (klp_free_objects()).  At the very least it should be moved to be
right above it.

>  
>  /* Clean up when a patched object is unloaded */
> @@ -677,24 +675,23 @@ static void klp_free_object_loaded(struct klp_object *obj)
>  		func->old_addr = 0;
>  }
>  
> -/*
> - * Free all objects' kobjects in the array up to some limit. When limit is
> - * NULL, all kobjects are freed.
> - */
> -static void klp_free_objects_limited(struct klp_patch *patch,
> -				     struct klp_object *limit)
> +/* Free all funcs and objects that have the kobject initialized. */
> +static void klp_free_objects(struct klp_patch *patch)
>  {
>  	struct klp_object *obj;
>  
> -	for (obj = patch->objs; obj->funcs && obj != limit; obj++) {
> -		klp_free_funcs_limited(obj, NULL);
> -		kobject_put(&obj->kobj);
> +	klp_for_each_object(patch, obj) {
> +		klp_free_funcs(obj);
> +
> +		if (obj->kobj.state_initialized)
> +			kobject_put(&obj->kobj);
>  	}
>  }
>  
>  static void klp_free_patch(struct klp_patch *patch)
>  {
> -	klp_free_objects_limited(patch, NULL);
> +	klp_free_objects(patch);
> +
>  	if (!list_empty(&patch->list))
>  		list_del(&patch->list);
>  }
> @@ -791,21 +788,16 @@ static int klp_init_object(struct klp_patch *patch, struct klp_object *obj)
>  	klp_for_each_func(obj, func) {
>  		ret = klp_init_func(obj, func);
>  		if (ret)
> -			goto free;
> +			return ret;
>  	}
>  
>  	if (klp_is_object_loaded(obj)) {
>  		ret = klp_init_object_loaded(patch, obj);
>  		if (ret)
> -			goto free;
> +			return ret;
>  	}
>  
>  	return 0;
> -
> -free:
> -	klp_free_funcs_limited(obj, func);
> -	kobject_put(&obj->kobj);
> -	return ret;
>  }
>  
>  static int klp_init_patch(struct klp_patch *patch)
> @@ -842,7 +834,7 @@ static int klp_init_patch(struct klp_patch *patch)
>  	return 0;
>  
>  free:
> -	klp_free_objects_limited(patch, obj);
> +	klp_free_objects(patch);
>  
>  	mutex_unlock(&klp_mutex);
>  
> -- 
> 2.13.6
> 

-- 
Josh

Re: [PATCH v10 03/10] livepatch: Initial support for dynamic structures

[Josh Poimboeuf]

On Wed, Mar 07, 2018 at 09:20:32AM +0100, Petr Mladek wrote:
> From: Jason Baron <jbaron@akamai.com>
> 
> We are going to add a feature called atomic replace. It will allow to
> create a patch that would replace all already registered patches.
> For this, we will need to dynamically create funcs and objects
> for functions that are no longer patched.
> 
> This patch adds basic framework to handle such dynamic structures.
> 
> It adds enum klp_func_type that allows to distinguish the dynamically
> allocated funcs' structures. Note that objects' structures do not have
> a clear type. Namely the static objects' structures might list both static
> and dynamic funcs' structures.
> 
> The function type is then used to limit klp_free() functions. We will
> want to free the dynamic structures separately when they are no longer
> needed. At the same time, we also want to make our life easier,
> and introduce _any_ type that will allow to process all existing
> structures in one go.
> 
> We need to be careful here. First, objects' structures must be freed
> only when all listed funcs' structures are freed. Also we must avoid
> double free. Both problems are solved by removing the freed structures
> from the list.
> 
> Also note that klp_free*() functions are called also in klp_init_patch()
> error path when only some kobjects have been initialized. The other
> dynamic structures must be freed immediately by calling the respective
> klp_free_*_dynamic() functions.
> 
> Finally, the dynamic objects' structures are generic. The respective
> klp_allocate_object_dynamic() and klp_free_object_dynamic() can
> be implemented here. On the other hand, klp_free_func_dynamic()
> is empty. It must be updated when a particular dynamic
> klp_func_type is introduced.
> 
> Signed-off-by: Jason Baron <jbaron@akamai.com>
> [pmladek@suse.com: Converted into a generic API]
> Signed-off-by: Petr Mladek <pmladek@suse.com>
> Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> Cc: Jessica Yu <jeyu@kernel.org>
> Cc: Jiri Kosina <jikos@kernel.org>
> Acked-by: Miroslav Benes <mbenes@suse.cz>

I appreciate the split out patches, but I feel like they're split up
*too* much.  I found that it was hard to make sense of patches 3-5
without looking at patch 6.  So I'd suggest combining 3-6 into a single
patch.

Also, since patches 7-8 seem to be bug fixes for patch 6, they should be
squashed in, so we don't break bisectability unnecessarily.

> ---
>  include/linux/livepatch.h |  37 +++++++++++-
>  kernel/livepatch/core.c   | 141 +++++++++++++++++++++++++++++++++++++++++-----
>  2 files changed, 163 insertions(+), 15 deletions(-)
> 
> diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
> index e5db2ba7e2a5..7fb76e7d2693 100644
> --- a/include/linux/livepatch.h
> +++ b/include/linux/livepatch.h
> @@ -35,12 +35,22 @@
>  #define KLP_UNPATCHED	 0
>  #define KLP_PATCHED	 1
>  
> +/*
> + * Function type is used to distinguish dynamically allocated structures
> + * and limit some operations.
> + */
> +enum klp_func_type {
> +	KLP_FUNC_ANY = -1,	/* Substitute any type */
> +	KLP_FUNC_STATIC = 0,    /* Original statically defined structure */
> +};
> +
>  /**
>   * struct klp_func - function structure for live patching
>   * @old_name:	name of the function to be patched
>   * @new_func:	pointer to the patched function code
>   * @old_sympos: a hint indicating which symbol position the old function
>   *		can be found (optional)
> + * @ftype:	distinguish static and dynamic structures

The "f" in ftype is redundant because it's already in a klp_func struct.

Also, I wonder if a 'dynamic' bool would be cleaner / more legible than
the enum.  Instead of e.g.

  klp_free_objects(patch, KLP_FUNC_ANY);
  klp_free_objects(patch, KLP_FUNC_NOP);

it could be

  klp_free_objects(patch)
  klp_free_objects_dynamic(patch);

with the klp_free_objects*() functions calling a __klp_free_objects()
helper which takes a bool as an argument.

There would need to be other changes, so I'm not sure it would be
better, but it could be worth trying out and seeing if it's cleaner.

>   * @old_addr:	the address of the function being patched
>   * @kobj:	kobject for sysfs resources
>   * @stack_node:	list node for klp_ops func_stack list
> @@ -79,6 +89,7 @@ struct klp_func {
>  	unsigned long old_sympos;
>  
>  	/* internal */
> +	enum klp_func_type ftype;
>  	unsigned long old_addr;
>  	struct kobject kobj;
>  	struct list_head stack_node;
> @@ -164,17 +175,41 @@ struct klp_patch {
>  #define klp_for_each_object_static(patch, obj) \
>  	for (obj = patch->objs; obj->funcs || obj->name; obj++)
>  
> +#define klp_for_each_object_safe(patch, obj, tmp_obj)		\
> +	list_for_each_entry_safe(obj, tmp_obj, &patch->obj_list, obj_entry)
> +
>  #define klp_for_each_object(patch, obj)	\
>  	list_for_each_entry(obj, &patch->obj_list, obj_entry)
>  
> +/* Support also dynamically allocated struct klp_object */
>  #define klp_for_each_func_static(obj, func) \
>  	for (func = obj->funcs; \
> -	     func->old_name || func->new_func || func->old_sympos; \
> +	     func && (func->old_name || func->new_func || func->old_sympos); \
>  	     func++)
>  
> +#define klp_for_each_func_safe(obj, func, tmp_func)			\
> +	list_for_each_entry_safe(func, tmp_func, &obj->func_list, func_entry)
> +
>  #define klp_for_each_func(obj, func)	\
>  	list_for_each_entry(func, &obj->func_list, func_entry)
>  
> +static inline bool klp_is_object_dynamic(struct klp_object *obj)
> +{
> +	return !obj->funcs;
> +}
> +
> +static inline bool klp_is_func_dynamic(struct klp_func *func)
> +{
> +	WARN_ON_ONCE(func->ftype == KLP_FUNC_ANY);

This seems like a silly warning.  Surely such a condition wouldn't get
past code review :-)

> +	return func->ftype != KLP_FUNC_STATIC;
> +}

I think this would be clearer:

	return func->ftype == KLP_FUNC_NOP;

and perhaps KLP_FUNC_NOP should be renamed to KLP_FUNC_DYNAMIC?

	return func->ftype == KLP_FUNC_DYNAMIC;

(though I realize this patch doesn't have KLP_FUNC_NOP yet)

> +
> +static inline bool klp_is_func_type(struct klp_func *func,
> +				    enum klp_func_type ftype)
> +{
> +	return ftype == KLP_FUNC_ANY || ftype == func->ftype;
> +}
> +
>  int klp_register_patch(struct klp_patch *);
>  int klp_unregister_patch(struct klp_patch *);
>  int klp_enable_patch(struct klp_patch *);
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index 69bde95e76f8..a47c26147c17 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -124,6 +124,26 @@ static bool klp_initialized(void)
>  	return !!klp_root_kobj;
>  }
>  
> +static struct klp_object *klp_find_object(struct klp_patch *patch,
> +					  struct klp_object *old_obj)
> +{
> +	struct klp_object *obj;
> +	bool mod = klp_is_module(old_obj);
> +
> +	klp_for_each_object(patch, obj) {
> +		if (mod) {

The 'mod' variable is a bit confusing here, as it's not clear that it
refers to the old object.  For clarity, maybe just do the
klp_is_module(old_obj) check here.

> +			if (klp_is_module(obj) &&
> +			    strcmp(old_obj->name, obj->name) == 0) {
> +				return obj;
> +			}
> +		} else if (!klp_is_module(obj)) {
> +			return obj;
> +		}
> +	}
> +
> +	return NULL;
> +}
> +
>  struct klp_find_arg {
>  	const char *objname;
>  	const char *name;
> @@ -621,6 +641,66 @@ static struct attribute *klp_patch_attrs[] = {
>  	NULL
>  };
>  
> +/*
> + * Dynamically allocated objects and functions.
> + */
> +static void klp_free_func_dynamic(struct klp_func *func)
> +{
> +}
> +
> +static void klp_free_object_dynamic(struct klp_object *obj)
> +{
> +	kfree(obj->name);
> +	kfree(obj);
> +}
> +
> +static struct klp_object *klp_alloc_object_dynamic(const char *name)
> +{
> +	struct klp_object *obj;
> +
> +	obj = kzalloc(sizeof(*obj), GFP_KERNEL);
> +	if (!obj)
> +		return ERR_PTR(-ENOMEM);
> +
> +	if (name) {
> +		obj->name = kstrdup(name, GFP_KERNEL);
> +		if (!obj->name) {
> +			kfree(obj);
> +			return ERR_PTR(-ENOMEM);
> +		}
> +	}
> +
> +	return obj;
> +}
> +
> +static struct klp_object *klp_get_or_add_object(struct klp_patch *patch,
> +						struct klp_object *old_obj)
> +{
> +	struct klp_object *obj;
> +
> +	obj = klp_find_object(patch, old_obj);
> +	if (obj)
> +		return obj;
> +
> +	obj = klp_alloc_object_dynamic(old_obj->name);
> +	if (IS_ERR(obj))
> +		return obj;
> +
> +	klp_init_object_list(patch, obj);
> +	return obj;
> +}
> +
> +/*
> + * Patch release framework must support the following scenarios:
> + *
> + *   + Asynchonous release is used when kobjects are initialized.
> + *
> + *   + Direct release is used in error paths for structures that
> + *     have not had kobj initialized yet.
> + *
> + *   + Allow to release dynamic structures of the given type when
> + *     they are not longer needed.
> + */
>  static void klp_kobj_release_patch(struct kobject *kobj)
>  {
>  	struct klp_patch *patch;
> @@ -637,6 +717,12 @@ static struct kobj_type klp_ktype_patch = {
>  
>  static void klp_kobj_release_object(struct kobject *kobj)
>  {
> +	struct klp_object *obj;
> +
> +	obj = container_of(kobj, struct klp_object, kobj);
> +
> +	if (klp_is_object_dynamic(obj))
> +		klp_free_object_dynamic(obj);
>  }
>  
>  static struct kobj_type klp_ktype_object = {
> @@ -646,6 +732,12 @@ static struct kobj_type klp_ktype_object = {
>  
>  static void klp_kobj_release_func(struct kobject *kobj)
>  {
> +	struct klp_func *func;
> +
> +	func = container_of(kobj, struct klp_func, kobj);
> +
> +	if (klp_is_func_dynamic(func))
> +		klp_free_func_dynamic(func);

It's a bit difficult to review the code here in this patch, when dynamic
funcs don't exist yet.  Again it seems to me the patches are split up
too much.

>  }
>  
>  static struct kobj_type klp_ktype_func = {
> @@ -653,14 +745,26 @@ static struct kobj_type klp_ktype_func = {
>  	.sysfs_ops = &kobj_sysfs_ops,
>  };
>  
> -/* Free all funcs that have the kobject initialized. */
> -static void klp_free_funcs(struct klp_object *obj)
> +/*
> + * Free all funcs of the given ftype. Use the kobject when it has already
> + * been initialized. Otherwise, do it directly.
> + */
> +static void klp_free_funcs(struct klp_object *obj,
> +			   enum klp_func_type ftype)
>  {
> -	struct klp_func *func;
> +	struct klp_func *func, *tmp_func;
> +
> +	klp_for_each_func_safe(obj, func, tmp_func) {
> +		if (!klp_is_func_type(func, ftype))
> +			continue;
> +
> +		/* Avoid double free and allow to detect empty objects. */
> +		list_del(&func->func_entry);
>  
> -	klp_for_each_func(obj, func) {
>  		if (func->kobj.state_initialized)
>  			kobject_put(&func->kobj);
> +		else if (klp_is_func_dynamic(func))
> +			klp_free_func_dynamic(func);

Here it's not obvious why the else condition is needed, since dynamic
objects also have kobjects.  I *think* it's only needed for the case
where initialization fails before the kobject has been added.  A comment
would help.

>  	}
>  }
>  
> @@ -675,22 +779,34 @@ static void klp_free_object_loaded(struct klp_object *obj)
>  		func->old_addr = 0;
>  }
>  
> -/* Free all funcs and objects that have the kobject initialized. */
> -static void klp_free_objects(struct klp_patch *patch)
> +/*
> + * Free all linked funcs of the given ftype. Then free empty objects.
> + * Use the kobject when it has already been initialized. Otherwise,
> + * do it directly.
> + */
> +static void klp_free_objects(struct klp_patch *patch, enum klp_func_type ftype)
>  {
> -	struct klp_object *obj;
> +	struct klp_object *obj, *tmp_obj;
>  
> -	klp_for_each_object(patch, obj) {
> -		klp_free_funcs(obj);
> +	klp_for_each_object_safe(patch, obj, tmp_obj) {
> +		klp_free_funcs(obj, ftype);
> +
> +		if (!list_empty(&obj->func_list))
> +			continue;
> +
> +		/* Avoid freeing the object twice. */
> +		list_del(&obj->obj_entry);
>  
>  		if (obj->kobj.state_initialized)
>  			kobject_put(&obj->kobj);
> +		else if (klp_is_object_dynamic(obj))
> +			klp_free_object_dynamic(obj);

A comment is also needed for this else condition.

>  	}
>  }
>  
>  static void klp_free_patch(struct klp_patch *patch)
>  {
> -	klp_free_objects(patch);
> +	klp_free_objects(patch, KLP_FUNC_ANY);
>  
>  	if (!list_empty(&patch->list))
>  		list_del(&patch->list);
> @@ -771,9 +887,6 @@ static int klp_init_object(struct klp_patch *patch, struct klp_object *obj)
>  	int ret;
>  	const char *name;
>  
> -	if (!obj->funcs)
> -		return -EINVAL;
> -
>  	obj->patched = false;
>  	obj->mod = NULL;
>  
> @@ -834,7 +947,7 @@ static int klp_init_patch(struct klp_patch *patch)
>  	return 0;
>  
>  free:
> -	klp_free_objects(patch);
> +	klp_free_objects(patch, KLP_FUNC_ANY);
>  
>  	mutex_unlock(&klp_mutex);
>  
> -- 
> 2.13.6
> 

-- 
Josh

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Josh Poimboeuf]

On Wed, Mar 07, 2018 at 09:20:34AM +0100, Petr Mladek wrote:
> From: Jason Baron <jbaron@akamai.com>
> 
> We are going to add a feature called atomic replace. It will allow to
> create a patch that would replace all already registered patches.
> 
> The replaced patches will stay registered because they are typically
> unregistered by some package uninstall scripts. But we will remove
> these patches from @klp_patches list to keep the enabled patch
> on the bottom of the stack. Otherwise, we would need to implement
> rather complex logic for moving the patches on the stack. Also
> it would complicate implementation of the atomic replace feature.
> It is not worth it.
> 
> As a result, we will have patches that are registered but that
> are no longer usable. Let's get prepared for this and use
> a better descriptive name for klp_is_patch_registered() function.
> 
> Also create separate list for the replaced patches and allow to
> unregister them. Alternative solution would be to add a flag
> into struct klp_patch. Note that patch->kobj.state_initialized
> is not safe because it can be cleared outside klp_mutex.
> 
> This patch does not change the existing behavior.
> 
> Signed-off-by: Jason Baron <jbaron@akamai.com>
> [pmladek@suse.com: Split and renamed klp_is_patch_usable()]
> Signed-off-by: Petr Mladek <pmladek@suse.com>
> Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> Cc: Jessica Yu <jeyu@kernel.org>
> Cc: Jiri Kosina <jikos@kernel.org>
> Acked-by: Miroslav Benes <mbenes@suse.cz>
> ---
>  kernel/livepatch/core.c | 30 ++++++++++++++++++++++++------
>  1 file changed, 24 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index ab1f6a371fc8..fd0296859ff4 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -47,6 +47,13 @@ DEFINE_MUTEX(klp_mutex);
>  
>  static LIST_HEAD(klp_patches);
>  
> +/*
> + * List of 'replaced' patches that have been replaced by a patch that has the
> + * 'replace' bit set. When they are added to this list, they are disabled and
> + * can not be re-enabled, but they can be unregistered().
> + */
> +static LIST_HEAD(klp_replaced_patches);

Can someone remind me why we're permanently disabling replaced patches?
I seem to remember being involved in that decision, but at least with
this latest version of the patches, it seems like it would be simpler to
just let 'replace' patches be rolled back to the previous state when
they're unpatched.  Then we don't need two lists of patches, the nops
can become more permanent, the replaced patches remain "enabled" but
inert, and the unpatching behavior is less surprising to the user, more
like a normal rollback.

Along those lines, I'd also propose that we constrain our existing patch
stacking even further.  Right now we allow a new patch to be registered
on top of a disabled patch, though we don't allow the new patch to be
enabled until the previous patch gets enabled.  I'd propose we no longer
allow that condition.  We should instead enforce that all existing
patches are *enabled* before allowing a new patch to be registered on
top.  That way the patch stacking is even more sane, and there are less
"unusual" conditions to worry about.  We have enough of those already.
Each additional bit of flexibility has a maintenance cost, and this one
isn't worth it IMO.

The downside of the above proposals is that now you can't remove an old
patch after it's been replaced, but IMO that's a small price to pay for
code sanity.  Every additional bit of flexibility has a maintenance
cost, and this one isn't worth it IMO.  Just force the user to leave the
old inert patches loaded.  They shouldn't take up much memory anyway,
and they might come in handy should a revert be needed.

> +
>  static struct kobject *klp_root_kobj;
>  
>  static void klp_init_func_list(struct klp_object *obj, struct klp_func *func)
> @@ -108,17 +115,28 @@ static void klp_find_object_module(struct klp_object *obj)
>  	mutex_unlock(&module_mutex);
>  }
>  
> -static bool klp_is_patch_registered(struct klp_patch *patch)
> +static bool klp_is_patch_in_list(struct klp_patch *patch,
> +				 struct list_head *head)
>  {
>  	struct klp_patch *mypatch;
>  
> -	list_for_each_entry(mypatch, &klp_patches, list)
> +	list_for_each_entry(mypatch, head, list)
>  		if (mypatch == patch)
>  			return true;
>  
>  	return false;
>  }
>  
> +static bool klp_is_patch_usable(struct klp_patch *patch)
> +{
> +	return klp_is_patch_in_list(patch, &klp_patches);
> +}
> +
> +static bool klp_is_patch_replaced(struct klp_patch *patch)
> +{
> +	return klp_is_patch_in_list(patch, &klp_replaced_patches);
> +}
> +
>  static bool klp_initialized(void)
>  {
>  	return !!klp_root_kobj;
> @@ -375,7 +393,7 @@ int klp_disable_patch(struct klp_patch *patch)
>  
>  	mutex_lock(&klp_mutex);
>  
> -	if (!klp_is_patch_registered(patch)) {
> +	if (!klp_is_patch_usable(patch)) {

This is another case where a wrapper function hurts readability.  What
does "usable" even mean to the reader of this code?  Every time I see
it, I have to do a double take.  I think getting rid of the wrapper in
favor of just doing

	if (!klp_is_patch_in_list(patch, &klp_patches))

would be much more obvious.

>  		ret = -EINVAL;
>  		goto err;
>  	}
> @@ -475,7 +493,7 @@ int klp_enable_patch(struct klp_patch *patch)
>  
>  	mutex_lock(&klp_mutex);
>  
> -	if (!klp_is_patch_registered(patch)) {
> +	if (!klp_is_patch_usable(patch)) {
>  		ret = -EINVAL;
>  		goto err;
>  	}
> @@ -516,7 +534,7 @@ static ssize_t enabled_store(struct kobject *kobj, struct kobj_attribute *attr,
>  
>  	mutex_lock(&klp_mutex);
>  
> -	if (!klp_is_patch_registered(patch)) {
> +	if (!klp_is_patch_usable(patch)) {
>  		/*
>  		 * Module with the patch could either disappear meanwhile or is
>  		 * not properly initialized yet.
> @@ -971,7 +989,7 @@ int klp_unregister_patch(struct klp_patch *patch)
>  
>  	mutex_lock(&klp_mutex);
>  
> -	if (!klp_is_patch_registered(patch)) {
> +	if (!klp_is_patch_usable(patch) && !klp_is_patch_replaced(patch)) {
>  		ret = -EINVAL;
>  		goto err;
>  	}
> -- 
> 2.13.6
> 

-- 
Josh

Re: [PATCH v10 06/10] livepatch: Add atomic replace

[Josh Poimboeuf]

On Wed, Mar 07, 2018 at 09:20:35AM +0100, Petr Mladek wrote:
> From: Jason Baron <jbaron@akamai.com>
> 
> Sometimes we would like to revert a particular fix. Currently, this
> is not easy because we want to keep all other fixes active and we
> could revert only the last applied patch.
> 
> One solution would be to apply new patch that implemented all
> the reverted functions like in the original code. It would work
> as expected but there will be unnecessary redirections. In addition,
> it would also require knowing which functions need to be reverted at
> build time.
> 
> Another problem is when there are many patches that touch the same
> functions. There might be dependencies between patches that are
> not enforced on the kernel side. Also it might be pretty hard to
> actually prepare the patch and ensure compatibility with
> the other patches.
> 
> A better solution would be to create cumulative patch and say that
> it replaces all older ones.
> 
> This patch adds a new "replace" flag to struct klp_patch. When it is
> enabled, a set of 'nop' klp_func will be dynamically created for all
> functions that are already being patched but that will no longer be
> modified by the new patch. They are temporarily used as a new target
> during the patch transition.
> 
> There are used several simplifications:
> 
>   + nops' structures are generated already when the patch is registered.
>     All registered patches are taken into account, even the disabled ones.
>     As a result, there might be more nops than are really needed when
>     the patch is enabled and some disabled patches were removed before.
>     But we are on the safe side and it simplifies the implementation.
>     Especially we could reuse the existing init() functions. Also freeing
>     is easier because the same nops are created and removed only once.

This quirk would go away with my patch stacking proposal to disallow
registering on top of a disabled patch.

>     Alternative solution would be to create nops when the patch is enabled.
>     But then any reusing of the init() functions and error paths would be
>     complicated. Also it would increase the risk of errors because of
>     late kobject initialization. Finally, it would need tricky waiting
>     for freed kobjects when finalizing a reverted enable transaction.
> 
>   + The replaced patches are removed from the stack and cannot longer
>     be enabled directly. Otherwise, we would need to implement a more
>     complex logic of handling the stack of patches. It might be hard
>     to come with a reasonable semantic.
> 
>     A fallback is to remove (rmmod) the replaced patches and register
>     (insmod) them again.

Hopefully this quirk can also go away.

>   + Nops are handled like normal function patches. It reduces changes
>     in the existing code.
> 
>     It would be possible to copy internal values when they are allocated
>     and make short cuts in init() functions. It would be possible to use
>     the fact that old_func and new_func point to the same function and
>     do not init new_func and new_size at all. It would be possible to
>     detect nop func in ftrace handler and just leave. But all these would
>     just complicate the code and maintenance.
> 
>   + The callbacks from the replaced patches are not called. It would be
>     pretty hard to define a reasonable semantic and implement it.
> 
>     It might even be counter-productive. The new patch is cumulative.
>     It is supposed to include most of the changes from older patches.
>     In most cases, it will not want to call pre_unpatch() post_unpatch()
>     callbacks from the replaced patches. It would disable/break things
>     for no good reasons. Also it should be easier to handle various
>     scenarios in a single script in the new patch than think about
>     interactions caused by running many scripts from older patches.
>     No to say that the old scripts even would not expect to be called
>     in this situation.
> 
> Signed-off-by: Jason Baron <jbaron@akamai.com>
> [pmladek@suse.com: Split, reuse existing code, simplified]
> Signed-off-by: Petr Mladek <pmladek@suse.com>
> Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> Cc: Jessica Yu <jeyu@kernel.org>
> Cc: Jiri Kosina <jikos@kernel.org>
> Acked-by: Miroslav Benes <mbenes@suse.cz>
> ---
>  include/linux/livepatch.h     |   3 +
>  kernel/livepatch/core.c       | 162 +++++++++++++++++++++++++++++++++++++++++-
>  kernel/livepatch/core.h       |   4 ++
>  kernel/livepatch/transition.c |  39 ++++++++++
>  4 files changed, 206 insertions(+), 2 deletions(-)
> 
> diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
> index 7fb76e7d2693..ed598d849029 100644
> --- a/include/linux/livepatch.h
> +++ b/include/linux/livepatch.h
> @@ -42,6 +42,7 @@
>  enum klp_func_type {
>  	KLP_FUNC_ANY = -1,	/* Substitute any type */
>  	KLP_FUNC_STATIC = 0,    /* Original statically defined structure */
> +	KLP_FUNC_NOP,		/* Dynamically allocated NOP function patch */
>  };
>  
>  /**
> @@ -153,6 +154,7 @@ struct klp_object {
>   * struct klp_patch - patch structure for live patching
>   * @mod:	reference to the live patch module
>   * @objs:	object entries for kernel objects to be patched
> + * @replace:	replace all already registered patches
>   * @list:	list node for global list of registered patches
>   * @kobj:	kobject for sysfs resources
>   * @obj_list:	head of list for struct klp_object
> @@ -163,6 +165,7 @@ struct klp_patch {
>  	/* external */
>  	struct module *mod;
>  	struct klp_object *objs;
> +	bool replace;
>  
>  	/* internal */
>  	struct list_head list;
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index fd0296859ff4..ad508a86b2f9 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -142,6 +142,21 @@ static bool klp_initialized(void)
>  	return !!klp_root_kobj;
>  }
>  
> +static struct klp_func *klp_find_func(struct klp_object *obj,
> +				      struct klp_func *old_func)
> +{
> +	struct klp_func *func;
> +
> +	klp_for_each_func(obj, func) {
> +		if ((strcmp(old_func->old_name, func->old_name) == 0) &&
> +		    (old_func->old_sympos == func->old_sympos)) {
> +			return func;
> +		}
> +	}
> +
> +	return NULL;
> +}
> +
>  static struct klp_object *klp_find_object(struct klp_patch *patch,
>  					  struct klp_object *old_obj)
>  {
> @@ -342,6 +357,39 @@ static int klp_write_object_relocations(struct module *pmod,
>  	return ret;
>  }
>  
> +/*
> + * This function removes replaced patches from both func_stack
> + * and klp_patches stack.
> + *
> + * We could be pretty aggressive here. It is called in situation
> + * when these structures are no longer accessible. All functions

"in the situation where"

> + * are redirected using the klp_transition_patch. They use either
> + * a new code or they are in the original code because of the special
> + * nop function patches.
> + */
> +void klp_throw_away_replaced_patches(struct klp_patch *new_patch,
> +				     bool keep_module)
> +{
> +	struct klp_patch *old_patch, *tmp_patch;
> +
> +	list_for_each_entry_safe(old_patch, tmp_patch, &klp_patches, list) {
> +		if (old_patch == new_patch)
> +			return;
> +
> +		klp_unpatch_objects(old_patch, KLP_FUNC_ANY);
> +		old_patch->enabled = false;
> +
> +		/*
> +		 * Replaced patches could not get re-enabled to keep
> +		 * the code sane.
> +		 */
> +		list_move(&old_patch->list, &klp_replaced_patches);
> +
> +		if (!keep_module)
> +			module_put(old_patch->mod);
> +	}
> +}

As I said before, hopefully this function can go away.  But if not, I
would s/throw_away/discard/ to be clearer.

> +
>  static int __klp_disable_patch(struct klp_patch *patch)
>  {
>  	struct klp_object *obj;
> @@ -537,7 +585,7 @@ static ssize_t enabled_store(struct kobject *kobj, struct kobj_attribute *attr,
>  	if (!klp_is_patch_usable(patch)) {
>  		/*
>  		 * Module with the patch could either disappear meanwhile or is
> -		 * not properly initialized yet.
> +		 * not properly initialized yet or the patch was just replaced.
>  		 */
>  		ret = -EINVAL;
>  		goto err;
> @@ -662,8 +710,16 @@ static struct attribute *klp_patch_attrs[] = {
>  /*
>   * Dynamically allocated objects and functions.
>   */
> +static void klp_free_func_nop(struct klp_func *func)
> +{
> +	kfree(func->old_name);
> +	kfree(func);
> +}
> +
>  static void klp_free_func_dynamic(struct klp_func *func)
>  {
> +	if (func->ftype == KLP_FUNC_NOP)
> +		klp_free_func_nop(func);
>  }

The conditional check isn't needed, because its callers already do the
check.  In fact, to reduce functionitis I would get rid of this function
and rename klp_free_func_nop() to klp_free_func_dynamic().

>  
>  static void klp_free_object_dynamic(struct klp_object *obj)
> @@ -708,6 +764,102 @@ static struct klp_object *klp_get_or_add_object(struct klp_patch *patch,
>  	return obj;
>  }
>  
> +static struct klp_func *klp_alloc_func_nop(struct klp_func *old_func,
> +					   struct klp_object *obj)
> +{
> +	struct klp_func *func;
> +
> +	func = kzalloc(sizeof(*func), GFP_KERNEL);
> +	if (!func)
> +		return ERR_PTR(-ENOMEM);

I don't see a need for ERR_PTR, here or elsewhere.  Just return NULL,
and the caller should know that means -ENOMEM.

> +
> +	if (old_func->old_name) {
> +		func->old_name = kstrdup(old_func->old_name, GFP_KERNEL);
> +		if (!func->old_name) {
> +			kfree(func);
> +			return ERR_PTR(-ENOMEM);
> +		}
> +	}
> +	func->old_sympos = old_func->old_sympos;
> +	/* NOP func is the same as using the original implementation. */
> +	func->new_func = (void *)old_func->old_addr;
> +	func->ftype = KLP_FUNC_NOP;
> +
> +	return func;
> +}
> +
> +static int klp_add_func_nop(struct klp_object *obj,
> +			    struct klp_func *old_func)
> +{
> +	struct klp_func *func;
> +
> +	func = klp_find_func(obj, old_func);
> +
> +	if (func)
> +		return 0;
> +
> +	func = klp_alloc_func_nop(old_func, obj);
> +	if (IS_ERR(func))
> +		return PTR_ERR(func);
> +
> +	klp_init_func_list(obj, func);
> +
> +	return 0;
> +}
> +
> +static int klp_add_object_nops(struct klp_patch *patch,
> +			       struct klp_object *old_obj)
> +{
> +	struct klp_object *obj;
> +	struct klp_func *old_func;
> +	int err = 0;
> +
> +	obj = klp_get_or_add_object(patch, old_obj);
> +	if (IS_ERR(obj))
> +		return PTR_ERR(obj);
> +
> +	klp_for_each_func(old_obj, old_func) {
> +		err = klp_add_func_nop(obj, old_func);
> +		if (err)
> +			return err;
> +	}

The functionality seems correct, but it's a bit confusing that it's
calling klp_add_func_nop() for *all* funcs.  I think it would be clearer
to do the klp_find_func() check here in klp_add_object_nops() instead of
in klp_add_func_nop().

And also that might make klp_add_func_nop() small enough that you can
just inline it.

> +
> +	return 0;
> +}
> +
> +/*
> + * Add 'nop' functions which simply return to the caller to run
> + * the original function. The 'nop' functions are added to a
> + * patch to facilitate a 'replace' mode

Missing period at the end of the sentence.

> + *
> + * The nops are generated for all patches on the stack when
> + * the new patch is initialized. It is safe even though some
> + * older patches might get disabled and removed before the
> + * new one is enabled. In the worst case, there might be nops
> + * which will not be really needed. But it does not harm and
> + * simplifies the implementation a lot. Especially we could
> + * use the init functions as is.

Hopefully most of this comment can be removed, if we do my proposal to
further constrain patch stacking.

> + */
> +static int klp_add_nops(struct klp_patch *patch)
> +{
> +	struct klp_patch *old_patch;
> +	struct klp_object *old_obj;
> +	int err = 0;
> +
> +	if (WARN_ON(!patch->replace))
> +		return -EINVAL;

IMO, this is another one of those overly paranoid warnings that isn't
really needed.  Why would we call klp_add_nops() for a non-replace
patch?

> +
> +	list_for_each_entry(old_patch, &klp_patches, list) {
> +		klp_for_each_object(old_patch, old_obj) {
> +			err = klp_add_object_nops(patch, old_obj);
> +			if (err)
> +				return err;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
>  /*
>   * Patch release framework must support the following scenarios:
>   *
> @@ -802,7 +954,7 @@ static void klp_free_object_loaded(struct klp_object *obj)
>   * Use the kobject when it has already been initialized. Otherwise,
>   * do it directly.
>   */
> -static void klp_free_objects(struct klp_patch *patch, enum klp_func_type ftype)
> +void klp_free_objects(struct klp_patch *patch, enum klp_func_type ftype)
>  {
>  	struct klp_object *obj, *tmp_obj;
>  
> @@ -952,6 +1104,12 @@ static int klp_init_patch(struct klp_patch *patch)
>  		return ret;
>  	}
>  
> +	if (patch->replace) {
> +		ret = klp_add_nops(patch);
> +		if (ret)
> +			goto free;
> +	}
> +
>  	klp_for_each_object(patch, obj) {
>  		ret = klp_init_object(patch, obj);
>  		if (ret)
> diff --git a/kernel/livepatch/core.h b/kernel/livepatch/core.h
> index 48a83d4364cf..43184a5318d8 100644
> --- a/kernel/livepatch/core.h
> +++ b/kernel/livepatch/core.h
> @@ -6,6 +6,10 @@
>  
>  extern struct mutex klp_mutex;
>  
> +void klp_throw_away_replaced_patches(struct klp_patch *new_patch,
> +				     bool keep_module);
> +void klp_free_objects(struct klp_patch *patch, enum klp_func_type ftype);
> +
>  static inline bool klp_is_object_loaded(struct klp_object *obj)
>  {
>  	return !obj->name || obj->mod;
> diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
> index 6917100fbe79..d6af190865d2 100644
> --- a/kernel/livepatch/transition.c
> +++ b/kernel/livepatch/transition.c
> @@ -87,6 +87,36 @@ static void klp_complete_transition(void)
>  		 klp_transition_patch->mod->name,
>  		 klp_target_state == KLP_PATCHED ? "patching" : "unpatching");
>  
> +	/*
> +	 * For replace patches, we disable all previous patches, and replace
> +	 * the dynamic no-op functions by removing the ftrace hook.
> +	 */
> +	if (klp_transition_patch->replace && klp_target_state == KLP_PATCHED) {
> +		/*
> +		 * Make sure that no ftrace handler accesses any older patch
> +		 * on the stack.  This might happen when the user forced the
> +		 * transaction while some running tasks were still falling
> +		 * back to the old code.  There might even still be ftrace
> +		 * handlers that have not seen the last patch on the stack yet.
> +		 *
> +		 * It probably is not necessary because of the rcu-safe access.
> +		 * But better be safe than sorry.
> +		 */
> +		if (klp_forced)
> +			klp_synchronize_transition();

I don't like this.  Hopefully we can get just rid of it, if we also get
rid of the concept of "throwing away" patches like I proposed.

> +
> +		klp_throw_away_replaced_patches(klp_transition_patch,
> +						klp_forced);
> +
> +		/*
> +		 * There is no need to synchronize the transition after removing
> +		 * nops. They must be the last on the func_stack. Ftrace
> +		 * gurantees that nobody will stay in the trampoline after

"guarantees"

> +		 * the ftrace handler is unregistered.
> +		 */
> +		klp_unpatch_objects(klp_transition_patch, KLP_FUNC_NOP);
> +	}
> +
>  	if (klp_target_state == KLP_UNPATCHED) {
>  		/*
>  		 * All tasks have transitioned to KLP_UNPATCHED so we can now
> @@ -143,6 +173,15 @@ static void klp_complete_transition(void)
>  	if (!klp_forced && klp_target_state == KLP_UNPATCHED)
>  		module_put(klp_transition_patch->mod);
>  
> +	/*
> +	 * We do not need to wait until the objects are really freed.
> +	 * The patch must be on the bottom of the stack. Therefore it
> +	 * will never replace anything else. The only important thing
> +	 * is that we wait when the patch is being unregistered.
> +	 */
> +	if (klp_transition_patch->replace && klp_target_state == KLP_PATCHED)
> +		klp_free_objects(klp_transition_patch, KLP_FUNC_NOP);
> +

This makes me a bit nervous.  What happens if the patch is enabled, then
disabled, then enabled again?  Then klp_free_objects() wouldn't do
anything, because the ops would already be freed.  I guess it would be
fine, as klp_free_objects() is smart enough to not do anything in that
case.  However I wonder if there are other cases which make assumptions
based on patch->replace, which might no longer be relevant if the patch
is re-enabled.  I also wonder if it makes sense to set patch->replace to
false here, since it would no longer be replacing anything if it were
disabled or re-enabled.

Either way, I guess freeing the nops is optional here, since they would
also get freed on patch unregister?

Anyway, all my points here would be moot if we made the nops more
permanent and allowedd the 'replace' patch to be rolled back to the
previous patch.  Which seems like it would make a lot of cases simpler,
as there would be fewer special cases associated with atomic replace,
and it would act more like a normal patch.

>  	klp_target_state = KLP_UNDEFINED;
>  	klp_transition_patch = NULL;
>  }
> -- 
> 2.13.6
> 

-- 
Josh

Re: [PATCH v10 02/10] livepatch: Free only structures with initialized kobject

[Petr Mladek]

On Tue 2018-03-13 17:38:58, Josh Poimboeuf wrote:
> On Wed, Mar 07, 2018 at 09:20:31AM +0100, Petr Mladek wrote:
> > We are going to add a feature called atomic replace. It will allow to
> > create a patch that would replace all already registered patches.
> > For this, we will need to dynamically create funcs and objects
> > for functions that are no longer patched.
> > 
> > We will want to reuse the existing init() and free() functions. Up to now,
> > the free() functions checked a limit and were called only for structures
> > with initialized kobject. But we will want to call them also for structures
> > that were allocated but where the kobject was not initialized yet.
> > 
> > This patch removes the limit. It calls klp_free*() functions for all
> > structures. But only the ones with initialized kobject are freed.
> > The handling of un-initialized structures will be added later with
> > the support for dynamic structures.
> > 
> > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > index 1d525f4a270a..69bde95e76f8 100644
> > --- a/kernel/livepatch/core.c
> > +++ b/kernel/livepatch/core.c
> > @@ -653,17 +653,15 @@ static struct kobj_type klp_ktype_func = {
> >  	.sysfs_ops = &kobj_sysfs_ops,
> >  };
> >  
> > -/*
> > - * Free all functions' kobjects in the array up to some limit. When limit is
> > - * NULL, all kobjects are freed.
> > - */
> > -static void klp_free_funcs_limited(struct klp_object *obj,
> > -				   struct klp_func *limit)
> > +/* Free all funcs that have the kobject initialized. */
> > +static void klp_free_funcs(struct klp_object *obj)
> >  {
> >  	struct klp_func *func;
> >  
> > -	for (func = obj->funcs; func->old_name && func != limit; func++)
> > -		kobject_put(&func->kobj);
> > +	klp_for_each_func(obj, func) {
> > +		if (func->kobj.state_initialized)
> > +			kobject_put(&func->kobj);
> > +	}
> >  }
> 
> Now that this function only has a single caller, personally I think it
> would become more readable if klp_free_funcs() were inlined into its
> caller (klp_free_objects()).  At the very least it should be moved to be
> right above it.

I would keep it as is. The function will get later more complicated by
adding support for dynamically allocated structures.

I still have to think about squashing the patches. I'll reply
on this in the other mail where you opened this question.

Best regards,
Petr

PS: Thank you for review. I took some pills against functionitis and
already applied all changes suggested for the 1st patch.

Re: [PATCH v10 03/10] livepatch: Initial support for dynamic structures

[Petr Mladek]

On Tue 2018-03-13 17:44:17, Josh Poimboeuf wrote:
> On Wed, Mar 07, 2018 at 09:20:32AM +0100, Petr Mladek wrote:
> > From: Jason Baron <jbaron@akamai.com>
> > 
> > We are going to add a feature called atomic replace. It will allow to
> > create a patch that would replace all already registered patches.
> > For this, we will need to dynamically create funcs and objects
> > for functions that are no longer patched.
> > 
> > This patch adds basic framework to handle such dynamic structures.
> > 
> > It adds enum klp_func_type that allows to distinguish the dynamically
> > allocated funcs' structures. Note that objects' structures do not have
> > a clear type. Namely the static objects' structures might list both static
> > and dynamic funcs' structures.
> > 
> > The function type is then used to limit klp_free() functions. We will
> > want to free the dynamic structures separately when they are no longer
> > needed. At the same time, we also want to make our life easier,
> > and introduce _any_ type that will allow to process all existing
> > structures in one go.
> > 
> > We need to be careful here. First, objects' structures must be freed
> > only when all listed funcs' structures are freed. Also we must avoid
> > double free. Both problems are solved by removing the freed structures
> > from the list.
> > 
> > Also note that klp_free*() functions are called also in klp_init_patch()
> > error path when only some kobjects have been initialized. The other
> > dynamic structures must be freed immediately by calling the respective
> > klp_free_*_dynamic() functions.
> > 
> > Finally, the dynamic objects' structures are generic. The respective
> > klp_allocate_object_dynamic() and klp_free_object_dynamic() can
> > be implemented here. On the other hand, klp_free_func_dynamic()
> > is empty. It must be updated when a particular dynamic
> > klp_func_type is introduced.
> > 
> > Signed-off-by: Jason Baron <jbaron@akamai.com>
> > [pmladek@suse.com: Converted into a generic API]
> > Signed-off-by: Petr Mladek <pmladek@suse.com>
> > Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> > Cc: Jessica Yu <jeyu@kernel.org>
> > Cc: Jiri Kosina <jikos@kernel.org>
> > Acked-by: Miroslav Benes <mbenes@suse.cz>
> 
> I appreciate the split out patches, but I feel like they're split up
> *too* much.  I found that it was hard to make sense of patches 3-5
> without looking at patch 6.  So I'd suggest combining 3-6 into a single
> patch.

I have squashed the patchset to 5 patches as of this writing. Well,
the main motivation was that it was much easier to do the other
suggested changes this way.

Otherwise I do not have a strong opinion. I think that preparatory
patches are useful if they help to split a huge change into some
logical steps. Everything usually makes much more sense in 2nd
review... ;-)


> Also, since patches 7-8 seem to be bug fixes for patch 6, they should be
> squashed in, so we don't break bisectability unnecessarily.

All the bugs were visible only with patches using the atomic replace.
Therefore they did not affect the bisectability. I guess that they
helped people who reviewed the earlier revisions. Anyway, they will
be squashed in v11.

 
> > ---
> >  include/linux/livepatch.h |  37 +++++++++++-
> >  kernel/livepatch/core.c   | 141 +++++++++++++++++++++++++++++++++++++++++-----
> >  2 files changed, 163 insertions(+), 15 deletions(-)
> > 
> > diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
> > index e5db2ba7e2a5..7fb76e7d2693 100644
> > --- a/include/linux/livepatch.h
> > +++ b/include/linux/livepatch.h
> > @@ -35,12 +35,22 @@
> >  #define KLP_UNPATCHED	 0
> >  #define KLP_PATCHED	 1
> >  
> > +/*
> > + * Function type is used to distinguish dynamically allocated structures
> > + * and limit some operations.
> > + */
> > +enum klp_func_type {
> > +	KLP_FUNC_ANY = -1,	/* Substitute any type */
> > +	KLP_FUNC_STATIC = 0,    /* Original statically defined structure */
> > +};
> > +
> >  /**
> >   * struct klp_func - function structure for live patching
> >   * @old_name:	name of the function to be patched
> >   * @new_func:	pointer to the patched function code
> >   * @old_sympos: a hint indicating which symbol position the old function
> >   *		can be found (optional)
> > + * @ftype:	distinguish static and dynamic structures
> 
> The "f" in ftype is redundant because it's already in a klp_func struct.

The type item will be gone in v11.


> Also, I wonder if a 'dynamic' bool would be cleaner / more legible than
> the enum.  Instead of e.g.
> 
>   klp_free_objects(patch, KLP_FUNC_ANY);
>   klp_free_objects(patch, KLP_FUNC_NOP);
> 
> it could be
> 
>   klp_free_objects(patch)
>   klp_free_objects_dynamic(patch);
> 
> with the klp_free_objects*() functions calling a __klp_free_objects()
> helper which takes a bool as an argument.
>
> There would need to be other changes, so I'm not sure it would be
> better, but it could be worth trying out and seeing if it's cleaner.

I gave it a chance. Somewhere it helped, somewhere it made it worse.
The overall result looks better to me, so I will use it in v11.

The main challenge was that I wanted to use "bool nop" in struct klp_func
and "bool dynamic" in struct klp_object. Then I needed to pass a
common flag to handle only these structures to klp_free_objects(),
klp_free_funcs(), klp_unpatch_objects(), klp_unpatch_func().
I solved this by using the inverse logic, for example, by passing
"bool free_all" to the free() functions.


> >   * @old_addr:	the address of the function being patched
> >   * @kobj:	kobject for sysfs resources
> >   * @stack_node:	list node for klp_ops func_stack list
> > @@ -164,17 +175,41 @@ struct klp_patch {
[...]
> > +static inline bool klp_is_func_dynamic(struct klp_func *func)
> > +{
> > +	WARN_ON_ONCE(func->ftype == KLP_FUNC_ANY);
> 
> This seems like a silly warning.  Surely such a condition wouldn't get
> past code review :-)
>
> > +	return func->ftype != KLP_FUNC_STATIC;
> > +}
> 
> I think this would be clearer:
> 
> 	return func->ftype == KLP_FUNC_NOP;
> 
> and perhaps KLP_FUNC_NOP should be renamed to KLP_FUNC_DYNAMIC?
> 
> 	return func->ftype == KLP_FUNC_DYNAMIC;
> 
> (though I realize this patch doesn't have KLP_FUNC_NOP yet)

All these helpers are gone in v11. They are not needed with
the booleans.


> > +
> > +static inline bool klp_is_func_type(struct klp_func *func,
> > +				    enum klp_func_type ftype)
> > +{
> > +	return ftype == KLP_FUNC_ANY || ftype == func->ftype;
> > +}
> > +
> >  int klp_register_patch(struct klp_patch *);
> >  int klp_unregister_patch(struct klp_patch *);
> >  int klp_enable_patch(struct klp_patch *);

I followed also the other suggestions from this mail.


Best Regardsm
Petr

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Petr Mladek]

On Tue 2018-03-13 17:46:13, Josh Poimboeuf wrote:
> On Wed, Mar 07, 2018 at 09:20:34AM +0100, Petr Mladek wrote:
> > From: Jason Baron <jbaron@akamai.com>
> > 
> > We are going to add a feature called atomic replace. It will allow to
> > create a patch that would replace all already registered patches.
> > 
> > The replaced patches will stay registered because they are typically
> > unregistered by some package uninstall scripts. But we will remove
> > these patches from @klp_patches list to keep the enabled patch
> > on the bottom of the stack. Otherwise, we would need to implement
> > rather complex logic for moving the patches on the stack. Also
> > it would complicate implementation of the atomic replace feature.
> > It is not worth it.
> > 
> > As a result, we will have patches that are registered but that
> > are no longer usable. Let's get prepared for this and use
> > a better descriptive name for klp_is_patch_registered() function.
> > 
> > Also create separate list for the replaced patches and allow to
> > unregister them. Alternative solution would be to add a flag
> > into struct klp_patch. Note that patch->kobj.state_initialized
> > is not safe because it can be cleared outside klp_mutex.
> > ---
> >  kernel/livepatch/core.c | 30 ++++++++++++++++++++++++------
> >  1 file changed, 24 insertions(+), 6 deletions(-)
> > 
> > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > index ab1f6a371fc8..fd0296859ff4 100644
> > --- a/kernel/livepatch/core.c
> > +++ b/kernel/livepatch/core.c
> > @@ -47,6 +47,13 @@ DEFINE_MUTEX(klp_mutex);
> >  
> >  static LIST_HEAD(klp_patches);
> >  
> > +/*
> > + * List of 'replaced' patches that have been replaced by a patch that has the
> > + * 'replace' bit set. When they are added to this list, they are disabled and
> > + * can not be re-enabled, but they can be unregistered().
> > + */
> > +static LIST_HEAD(klp_replaced_patches);
> 
> Can someone remind me why we're permanently disabling replaced patches?
> I seem to remember being involved in that decision, but at least with
> this latest version of the patches, it seems like it would be simpler to
> just let 'replace' patches be rolled back to the previous state when
> they're unpatched.  Then we don't need two lists of patches, the nops
> can become more permanent, the replaced patches remain "enabled" but
> inert, and the unpatching behavior is less surprising to the user, more
> like a normal rollback.

Yes, keeping the patches might make some things easier. But it might
also bring some problems and it would make the feature less useful.
The following arguments come to my mind:

1. The feature should help to keep the system in a consistent and
   well defined state. It should not depend on what patches were
   installed before.

   We do not want to force people to install every single livepatch
   before. It should be enough to install the last one. But then
   we might get different amount of NOPs depending on what was
   installed before.


2. The feature should allow to unpatch some functions while keeping
   the others patched.

   The ftrace handler might cause some unwanted slowdown or other
   problems. The performance might get restored only when we remove
   the NOPs when they are not longer necessary.


3. The handling of callbacks is already problematic. We run only
   the ones from the last patch to make things easier.

   We would need to come with something more complicated if we
   want to support rollback to "random" patches on the stack.
   And support for random patches is fundamental at least
   from my point of view.


> Along those lines, I'd also propose that we constrain our existing patch
> stacking even further.  Right now we allow a new patch to be registered
> on top of a disabled patch, though we don't allow the new patch to be
> enabled until the previous patch gets enabled.  I'd propose we no longer
> allow that condition.  We should instead enforce that all existing
> patches are *enabled* before allowing a new patch to be registered on
> top.  That way the patch stacking is even more sane, and there are less
> "unusual" conditions to worry about.  We have enough of those already.
> Each additional bit of flexibility has a maintenance cost, and this one
> isn't worth it IMO.

Again, this might make some things easier but it might also bring
problems.

For example, we would need to solve the situation when the last
patch is disabled and cannot be removed because the transition
was forced. This might be more common after removing the immediate
feature.

Also it might be less user friendly. White the atomic replace could
make things easier for both developers and users.


> The downside of the above proposals is that now you can't remove an old
> patch after it's been replaced, but IMO that's a small price to pay for
> code sanity.  Every additional bit of flexibility has a maintenance
> cost, and this one isn't worth it IMO.  Just force the user to leave the
> old inert patches loaded.  They shouldn't take up much memory anyway,
> and they might come in handy should a revert be needed.

I actually think about exactly the opposite way. IMHO, the atomic replace
and cumulative patches allow to handle livepatches more securely.
Also most situations might be solved by just going forward. If we support
only replace mode, we could get rid of the stack, disable feature,
and possibly make the code more straightforward.

To make it clear. I do not have any plans to work on this. It is
just an idea.


> > +
> >  static struct kobject *klp_root_kobj;
> >  
> >  static void klp_init_func_list(struct klp_object *obj, struct klp_func *func)

[...]

> > @@ -375,7 +393,7 @@ int klp_disable_patch(struct klp_patch *patch)
> >  
> >  	mutex_lock(&klp_mutex);
> >  
> > -	if (!klp_is_patch_registered(patch)) {
> > +	if (!klp_is_patch_usable(patch)) {
> 
> This is another case where a wrapper function hurts readability.  What
> does "usable" even mean to the reader of this code?  Every time I see
> it, I have to do a double take.  I think getting rid of the wrapper in
> favor of just doing
>
> 	if (!klp_is_patch_in_list(patch, &klp_patches))

IMHO, this is only a matter of taste. In the long term, the name
"is_patch_usable" might be more descriptive than the information that
the patch is on some  list.

Anyway, I reworked this for v11. I removed klp_replaced_patches list.
Instead, I added "bool registered" into struct klp_patch. Also
I renamed klp_is_patch_registered() to klp_is_patch_on_stack().

Then you could check is_patch_on_stack() in the enable/disable
functions. And you could check patch->registered in
klp_unregister_patch(). I believe that it is more clear.


Best Regards,
Petr

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Josh Poimboeuf]

On Mon, Mar 19, 2018 at 04:02:07PM +0100, Petr Mladek wrote:
> > Can someone remind me why we're permanently disabling replaced patches?
> > I seem to remember being involved in that decision, but at least with
> > this latest version of the patches, it seems like it would be simpler to
> > just let 'replace' patches be rolled back to the previous state when
> > they're unpatched.  Then we don't need two lists of patches, the nops
> > can become more permanent, the replaced patches remain "enabled" but
> > inert, and the unpatching behavior is less surprising to the user, more
> > like a normal rollback.
> 
> Yes, keeping the patches might make some things easier. But it might
> also bring some problems and it would make the feature less useful.
> The following arguments come to my mind:
> 
> 1. The feature should help to keep the system in a consistent and
>    well defined state. It should not depend on what patches were
>    installed before.

But the nops already accomplish that.  If they didn't, then this patch
set has a major problem.

>    We do not want to force people to install every single livepatch
>    before. It should be enough to install the last one.

Of course...

>    But then we might get different amount of NOPs depending on what
>    was installed before.

The same is true of this patch set as it is today.  The number of NOPs
used in the patching process will differ based on what patches were
previously applied.  This is unavoidable.  My proposal is to let the
NOPs hang around after the patching process.  Either way we must rely on
them *during* the patching process as well.

> 2. The feature should allow to unpatch some functions while keeping
>    the others patched.
> 
>    The ftrace handler might cause some unwanted slowdown or other
>    problems. The performance might get restored only when we remove
>    the NOPs when they are not longer necessary.

I'd say simplicity and maintainability of the code is more important
than an (imagined) performance issue.  The NOPs should be pretty fast
anyway.

Not to mention that my proposal would make the behavior less surprising
and more user friendly (reverting a 'replace' patch restores it to its
previous state).

> 3. The handling of callbacks is already problematic. We run only
>    the ones from the last patch to make things easier.
> 
>    We would need to come with something more complicated if we
>    want to support rollback to "random" patches on the stack.
>    And support for random patches is fundamental at least
>    from my point of view.

Can you elaborate on what you mean by random patches and why it would
require something more complicated from the callbacks?

> > Along those lines, I'd also propose that we constrain our existing patch
> > stacking even further.  Right now we allow a new patch to be registered
> > on top of a disabled patch, though we don't allow the new patch to be
> > enabled until the previous patch gets enabled.  I'd propose we no longer
> > allow that condition.  We should instead enforce that all existing
> > patches are *enabled* before allowing a new patch to be registered on
> > top.  That way the patch stacking is even more sane, and there are less
> > "unusual" conditions to worry about.  We have enough of those already.
> > Each additional bit of flexibility has a maintenance cost, and this one
> > isn't worth it IMO.
> 
> Again, this might make some things easier but it might also bring
> problems.
> 
> For example, we would need to solve the situation when the last
> patch is disabled and cannot be removed because the transition
> was forced. This might be more common after removing the immediate
> feature.

I would stop worrying about forced patches so much :-)

Forced patches already come with a disclaimer, and we can't bend over
backwards for them.  In such a rare case, the admin can just re-enable
the forced patch before loading the 'replace' patch.

> Also it might be less user friendly.

I don't know, does anybody really care about this case (patching on top
of a disabled patch)?  It just adds to the crazy matrix of possible
scenarios we have to keep in our heads, which means more bugs, for very
little (hypothetical) gain.

> White the atomic replace could make things easier for both developers
> and users.

I agree that atomic replace is a useful feature and I'm not arguing
against it, so maybe I missed your point?

> > The downside of the above proposals is that now you can't remove an old
> > patch after it's been replaced, but IMO that's a small price to pay for
> > code sanity.  Every additional bit of flexibility has a maintenance
> > cost, and this one isn't worth it IMO.  Just force the user to leave the
> > old inert patches loaded.  They shouldn't take up much memory anyway,
> > and they might come in handy should a revert be needed.
> 
> I actually think about exactly the opposite way. IMHO, the atomic replace
> and cumulative patches allow to handle livepatches more securely.
> Also most situations might be solved by just going forward. If we support
> only replace mode, we could get rid of the stack, disable feature,
> and possibly make the code more straightforward.
> 
> To make it clear. I do not have any plans to work on this. It is
> just an idea.

I'm all for simplifying, and I would be strongly tempted to ACK such a
proposal in a heartbeat, though I'm not sure whether it would support
everybody else's use cases.

(But, with today's code, we *do* support the disabling of patches.  So
given that constraint, my proposal results in simpler code.)

-- 
Josh

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Petr Mladek]

On Mon 2018-03-19 16:43:24, Josh Poimboeuf wrote:
> On Mon, Mar 19, 2018 at 04:02:07PM +0100, Petr Mladek wrote:
> > > Can someone remind me why we're permanently disabling replaced patches?
> > > I seem to remember being involved in that decision, but at least with
> > > this latest version of the patches, it seems like it would be simpler to
> > > just let 'replace' patches be rolled back to the previous state when
> > > they're unpatched.  Then we don't need two lists of patches, the nops
> > > can become more permanent, the replaced patches remain "enabled" but
> > > inert, and the unpatching behavior is less surprising to the user, more
> > > like a normal rollback.
> > 
> > Yes, keeping the patches might make some things easier. But it might
> > also bring some problems and it would make the feature less useful.
> > The following arguments come to my mind:
> > 
> > 1. The feature should help to keep the system in a consistent and
> >    well defined state. It should not depend on what patches were
> >    installed before.
> 
> But the nops already accomplish that.  If they didn't, then this patch
> set has a major problem.

The nops are enough to keep the functionality but they might harm
the performance.

Livepatching is about preventing bugs without reboot. I could simply
imagine that ftrace on a hot patch might cause performance
problems on some workloads. And I would like to have a way out in
this case.

Anyway, I am reworking the patchset so that it implements your
approach first. The possibility to remove NOPs and replaced
livepatches is done via a followup patch. This might help
to discuss if the changes are worth it or not.


> > 2. The feature should allow to unpatch some functions while keeping
> >    the others patched.
> > 
> >    The ftrace handler might cause some unwanted slowdown or other
> >    problems. The performance might get restored only when we remove
> >    the NOPs when they are not longer necessary.
> 
> I'd say simplicity and maintainability of the code is more important
> than an (imagined) performance issue.  The NOPs should be pretty fast
> anyway.
> 
> Not to mention that my proposal would make the behavior less surprising
> and more user friendly (reverting a 'replace' patch restores it to its
> previous state).

If the "disable" way works as expected, see below.

Also it is less surprising only if people understand the stack of
patches. If they are familiar only with replace patches then it is
normal for them that the patches get replaced. It is then like
a package version update.


> > 3. The handling of callbacks is already problematic. We run only
> >    the ones from the last patch to make things easier.
> > 
> >    We would need to come with something more complicated if we
> >    want to support rollback to "random" patches on the stack.
> >    And support for random patches is fundamental at least
> >    from my point of view.
> 
> Can you elaborate on what you mean by random patches and why it would
> require something more complicated from the callbacks?

Let's say that we will use atomic replace for cumulative
patches. Then every new patch knows what earlier patches did.
It just did not know which of them was already installed. Therefore
it needs to detect what callbacks were already called. The callbacks
usually create or change something. So there should be something to check.
Therefore the way forward should be rather straightforward.

The way back is more problematic. The callbacks in the new cumulative
patch would need to store information about the previous state and
be able to restore it when the patch gets disabled. It might more
or less double the callbacks code and testing scenarios.


> > > Along those lines, I'd also propose that we constrain our existing patch
> > > stacking even further.  Right now we allow a new patch to be registered
> > > on top of a disabled patch, though we don't allow the new patch to be
> > > enabled until the previous patch gets enabled.  I'd propose we no longer
> > > allow that condition.  We should instead enforce that all existing
> > > patches are *enabled* before allowing a new patch to be registered on
> > > top.  That way the patch stacking is even more sane, and there are less
> > > "unusual" conditions to worry about.  We have enough of those already.
> > > Each additional bit of flexibility has a maintenance cost, and this one
> > > isn't worth it IMO.
> > 
> > Again, this might make some things easier but it might also bring
> > problems.
> > 
> > For example, we would need to solve the situation when the last
> > patch is disabled and cannot be removed because the transition
> > was forced. This might be more common after removing the immediate
> > feature.
> 
> I would stop worrying about forced patches so much :-)

I have already seen blocked transition several times. It is true that
it was with kGraft. But we just do not have enough real life experience
with the upstream livepatch code.


> Forced patches already come with a disclaimer, and we can't bend over
> backwards for them.  In such a rare case, the admin can just re-enable
> the forced patch before loading the 'replace' patch.


> > Also it might be less user friendly.
> 
> I don't know, does anybody really care about this case (patching on top
> of a disabled patch)?  It just adds to the crazy matrix of possible
> scenarios we have to keep in our heads, which means more bugs, for very
> little (hypothetical) gain.

It depends if the we remove the replaced patches or not. If they are
removed then replacing disabled patches is rather trivial from both
coding and understanding side.

I am going to add this as a separate patch as well. Let's discuss
it with the code.


> > White the atomic replace could make things easier for both developers
> > and users.
> 
> I agree that atomic replace is a useful feature and I'm not arguing
> against it, so maybe I missed your point?

Your suggestion allows easier code but it reduces the advantages of
the atomic replace feature. We would achieve almost the same results
with a normal livepatch where the functions behave like in
the original code.

Also removing replaced patches can be seen as a clean up after
each patch. It might be more code but the target system might be
easier to debug. Also we do not need to mind about various
disable scenarios.

Best Regards,
Petr

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Evgenii Shatokhin]

On 20.03.2018 15:25, Petr Mladek wrote:
> On Mon 2018-03-19 16:43:24, Josh Poimboeuf wrote:
>> On Mon, Mar 19, 2018 at 04:02:07PM +0100, Petr Mladek wrote:
>>>> Can someone remind me why we're permanently disabling replaced patches?
>>>> I seem to remember being involved in that decision, but at least with
>>>> this latest version of the patches, it seems like it would be simpler to
>>>> just let 'replace' patches be rolled back to the previous state when
>>>> they're unpatched.  Then we don't need two lists of patches, the nops
>>>> can become more permanent, the replaced patches remain "enabled" but
>>>> inert, and the unpatching behavior is less surprising to the user, more
>>>> like a normal rollback.
>>>
>>> Yes, keeping the patches might make some things easier. But it might
>>> also bring some problems and it would make the feature less useful.
>>> The following arguments come to my mind:
>>>
>>> 1. The feature should help to keep the system in a consistent and
>>>     well defined state. It should not depend on what patches were
>>>     installed before.
>>
>> But the nops already accomplish that.  If they didn't, then this patch
>> set has a major problem.
> 
> The nops are enough to keep the functionality but they might harm
> the performance.
> 
> Livepatching is about preventing bugs without reboot. I could simply
> imagine that ftrace on a hot patch might cause performance
> problems on some workloads. And I would like to have a way out in
> this case.
> 
> Anyway, I am reworking the patchset so that it implements your
> approach first. The possibility to remove NOPs and replaced
> livepatches is done via a followup patch. This might help
> to discuss if the changes are worth it or not.
> 
> 
>>> 2. The feature should allow to unpatch some functions while keeping
>>>     the others patched.
>>>
>>>     The ftrace handler might cause some unwanted slowdown or other
>>>     problems. The performance might get restored only when we remove
>>>     the NOPs when they are not longer necessary.
>>
>> I'd say simplicity and maintainability of the code is more important
>> than an (imagined) performance issue.  The NOPs should be pretty fast
>> anyway.
>>
>> Not to mention that my proposal would make the behavior less surprising
>> and more user friendly (reverting a 'replace' patch restores it to its
>> previous state).
> 
> If the "disable" way works as expected, see below.
> 
> Also it is less surprising only if people understand the stack of
> patches. If they are familiar only with replace patches then it is
> normal for them that the patches get replaced. It is then like
> a package version update.
> 
> 
>>> 3. The handling of callbacks is already problematic. We run only
>>>     the ones from the last patch to make things easier.
>>>
>>>     We would need to come with something more complicated if we
>>>     want to support rollback to "random" patches on the stack.
>>>     And support for random patches is fundamental at least
>>>     from my point of view.
>>
>> Can you elaborate on what you mean by random patches and why it would
>> require something more complicated from the callbacks?
> 
> Let's say that we will use atomic replace for cumulative
> patches. Then every new patch knows what earlier patches did.
> It just did not know which of them was already installed. Therefore
> it needs to detect what callbacks were already called. The callbacks
> usually create or change something. So there should be something to check.
> Therefore the way forward should be rather straightforward.
> 
> The way back is more problematic. The callbacks in the new cumulative
> patch would need to store information about the previous state and
> be able to restore it when the patch gets disabled. It might more
> or less double the callbacks code and testing scenarios.
> 
> 
>>>> Along those lines, I'd also propose that we constrain our existing patch
>>>> stacking even further.  Right now we allow a new patch to be registered
>>>> on top of a disabled patch, though we don't allow the new patch to be
>>>> enabled until the previous patch gets enabled.  I'd propose we no longer
>>>> allow that condition.  We should instead enforce that all existing
>>>> patches are *enabled* before allowing a new patch to be registered on
>>>> top.  That way the patch stacking is even more sane, and there are less
>>>> "unusual" conditions to worry about.  We have enough of those already.
>>>> Each additional bit of flexibility has a maintenance cost, and this one
>>>> isn't worth it IMO.
>>>
>>> Again, this might make some things easier but it might also bring
>>> problems.
>>>
>>> For example, we would need to solve the situation when the last
>>> patch is disabled and cannot be removed because the transition
>>> was forced. This might be more common after removing the immediate
>>> feature.
>>
>> I would stop worrying about forced patches so much :-)
> 
> I have already seen blocked transition several times. It is true that
> it was with kGraft. But we just do not have enough real life experience
> with the upstream livepatch code.
> 
> 
>> Forced patches already come with a disclaimer, and we can't bend over
>> backwards for them.  In such a rare case, the admin can just re-enable
>> the forced patch before loading the 'replace' patch.
> 
> 
>>> Also it might be less user friendly.
>>
>> I don't know, does anybody really care about this case (patching on top
>> of a disabled patch)?  It just adds to the crazy matrix of possible
>> scenarios we have to keep in our heads, which means more bugs, for very
>> little (hypothetical) gain.
> 
> It depends if the we remove the replaced patches or not. If they are
> removed then replacing disabled patches is rather trivial from both
> coding and understanding side.
> 
> I am going to add this as a separate patch as well. Let's discuss
> it with the code.
> 
> 
>>> White the atomic replace could make things easier for both developers
>>> and users.
>>
>> I agree that atomic replace is a useful feature and I'm not arguing
>> against it, so maybe I missed your point?
> 
> Your suggestion allows easier code but it reduces the advantages of
> the atomic replace feature. We would achieve almost the same results
> with a normal livepatch where the functions behave like in
> the original code.
> 
> Also removing replaced patches can be seen as a clean up after
> each patch. It might be more code but the target system might be
> easier to debug. Also we do not need to mind about various
> disable scenarios.

+1

In our case (ReadyKernel), we mostly use cumulative patches with 
occasional "normal" patches as hot fixes and such.

After the new cumulative patch has been loaded and enabled, all the 
previously loaded patches are unloaded. Yes, it is seen as a cleanup.

So, most of the time, a single binary patch is loaded and active. It 
helps debugging if issues show up and it also makes it easier to explain 
to the customers which fixes are in effect on their machines.

Just my 2 cents.

> 
> Best Regards,
> Petr
> .
> 

Regards,
Evgenii

Re: [PATCH v10 00/10] livepatch: Atomic replace feature

[Miroslav Benes]

On Mon, 12 Mar 2018, Joe Lawrence wrote:

> These are the callback tests that I hacked up into a livepatch
> kselftest.  (Basically I copied a bunch of the sample modules and
> verified the expected dmesg output as I had listed in in the
> Documentation/livepatch/callbacks.txt file.)  The script is still a
> little rough and maybe this isn't the direction we want to go for proper
> kselftests, but perhaps it saves you some time/sanity for verifying this
> patchset.
> 
> Hope this helps,

Hi Joe,

it helps a lot. Thanks for implementing it.

Miroslav

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Miroslav Benes]

> > I don't know, does anybody really care about this case (patching on top
> > of a disabled patch)?  It just adds to the crazy matrix of possible
> > scenarios we have to keep in our heads, which means more bugs, for very
> > little (hypothetical) gain.
> 
> It depends if the we remove the replaced patches or not. If they are
> removed then replacing disabled patches is rather trivial from both
> coding and understanding side.

I agree. Since we already have the code, there is no point not to have the 
feature. It is not that complicated after all.
 
> I am going to add this as a separate patch as well. Let's discuss
> it with the code.
> 
> 
> > > White the atomic replace could make things easier for both developers
> > > and users.
> > 
> > I agree that atomic replace is a useful feature and I'm not arguing
> > against it, so maybe I missed your point?
> 
> Your suggestion allows easier code but it reduces the advantages of
> the atomic replace feature. We would achieve almost the same results
> with a normal livepatch where the functions behave like in
> the original code.
> 
> Also removing replaced patches can be seen as a clean up after
> each patch. It might be more code but the target system might be
> easier to debug. Also we do not need to mind about various
> disable scenarios.

I agree with this as well.

Yes, it was a bit painful to review, but I was quite content with the 
result. I don't want to go halfway and be stuck with NOPs when it is 
not complicated to remove them completely after the transition. It'd be 
odd in my opinion.

Regards,
Miroslav

Re: [PATCH v10 06/10] livepatch: Add atomic replace

[Petr Mladek]

On Tue 2018-03-13 17:48:04, Josh Poimboeuf wrote:
> On Wed, Mar 07, 2018 at 09:20:35AM +0100, Petr Mladek wrote:
> > From: Jason Baron <jbaron@akamai.com>
> > 
> > Sometimes we would like to revert a particular fix. Currently, this
> > is not easy because we want to keep all other fixes active and we
> > could revert only the last applied patch.
> > 
> > One solution would be to apply new patch that implemented all
> > the reverted functions like in the original code. It would work
> > as expected but there will be unnecessary redirections. In addition,
> > it would also require knowing which functions need to be reverted at
> > build time.
> > 
> > Another problem is when there are many patches that touch the same
> > functions. There might be dependencies between patches that are
> > not enforced on the kernel side. Also it might be pretty hard to
> > actually prepare the patch and ensure compatibility with
> > the other patches.
> > 
> > A better solution would be to create cumulative patch and say that
> > it replaces all older ones.
> > 
> > This patch adds a new "replace" flag to struct klp_patch. When it is
> > enabled, a set of 'nop' klp_func will be dynamically created for all
> > functions that are already being patched but that will no longer be
> > modified by the new patch. They are temporarily used as a new target
> > during the patch transition.
> > 
> > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > index fd0296859ff4..ad508a86b2f9 100644
> > --- a/kernel/livepatch/core.c
> > +++ b/kernel/livepatch/core.c
> > +static int klp_add_nops(struct klp_patch *patch)
> > +{
> > +	struct klp_patch *old_patch;
> > +	struct klp_object *old_obj;
> > +	int err = 0;
> > +
> > +	if (WARN_ON(!patch->replace))
> > +		return -EINVAL;
> 
> IMO, this is another one of those overly paranoid warnings that isn't
> really needed.  Why would we call klp_add_nops() for a non-replace
> patch?

Just to be sure. What is the difference, for example, against the following
checks in __klp_enable_patch() from your point of view, please?

	if (klp_transition_patch)
		return -EBUSY;

	if (WARN_ON(patch->enabled))
		return -EINVAL;

One difference is that klp_enable_patch() is exported symbol. One the
other hand, livepatch code developers could do mistakes as well.
Adding nops sounds like an innoncent operation after all ;-)


> > +	list_for_each_entry(old_patch, &klp_patches, list) {
> > +		klp_for_each_object(old_patch, old_obj) {
> > +			err = klp_add_object_nops(patch, old_obj);
> > +			if (err)
> > +				return err;
> > +		}
> > +	}
> > +
> > +	return 0;
> > +}

[...]

> > diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
> > index 6917100fbe79..d6af190865d2 100644
> > --- a/kernel/livepatch/transition.c
> > +++ b/kernel/livepatch/transition.c
> > @@ -87,6 +87,36 @@ static void klp_complete_transition(void)
> >  		 klp_transition_patch->mod->name,
> >  		 klp_target_state == KLP_PATCHED ? "patching" : "unpatching");
> >  
> > +	/*
> > +	 * For replace patches, we disable all previous patches, and replace
> > +	 * the dynamic no-op functions by removing the ftrace hook.
> > +	 */
> > +	if (klp_transition_patch->replace && klp_target_state == KLP_PATCHED) {
> > +		/*
> > +		 * Make sure that no ftrace handler accesses any older patch
> > +		 * on the stack.  This might happen when the user forced the
> > +		 * transaction while some running tasks were still falling
> > +		 * back to the old code.  There might even still be ftrace
> > +		 * handlers that have not seen the last patch on the stack yet.
> > +		 *
> > +		 * It probably is not necessary because of the rcu-safe access.
> > +		 * But better be safe than sorry.
> > +		 */
> > +		if (klp_forced)
> > +			klp_synchronize_transition();
> 
> I don't like this.  Hopefully we can get just rid of it, if we also get
> rid of the concept of "throwing away" patches like I proposed.

What exactly you do not like about it, please?

It is not needed if all processes were migrated using the consistency
model, definitely.

If the transition has been forced then the barrier should be needed from
similar reasons as the barrier after klp_unpatch_objects() below.
We basically want to be sure what ftrace handlers see on the stack.

Will it help, when I remove the last paragraph where the formulation
is quite uncertain?


> > +
> > +		klp_throw_away_replaced_patches(klp_transition_patch,
> > +						klp_forced);
> > +
> > +		/*
> > +		 * There is no need to synchronize the transition after removing
> > +		 * nops. They must be the last on the func_stack. Ftrace
> > +		 * gurantees that nobody will stay in the trampoline after
> 
> "guarantees"
> 
> > +		 * the ftrace handler is unregistered.
> > +		 */
> > +		klp_unpatch_objects(klp_transition_patch, KLP_FUNC_NOP);
> > +	}
> > +
> >  	if (klp_target_state == KLP_UNPATCHED) {
> >  		/*
> >  		 * All tasks have transitioned to KLP_UNPATCHED so we can now
> > @@ -143,6 +173,15 @@ static void klp_complete_transition(void)
> >  	if (!klp_forced && klp_target_state == KLP_UNPATCHED)
> >  		module_put(klp_transition_patch->mod);
> >  
> > +	/*
> > +	 * We do not need to wait until the objects are really freed.
> > +	 * The patch must be on the bottom of the stack. Therefore it
> > +	 * will never replace anything else. The only important thing
> > +	 * is that we wait when the patch is being unregistered.
> > +	 */
> > +	if (klp_transition_patch->replace && klp_target_state == KLP_PATCHED)
> > +		klp_free_objects(klp_transition_patch, KLP_FUNC_NOP);
> > +
> 
> This makes me a bit nervous.  What happens if the patch is enabled, then
> disabled, then enabled again?  Then klp_free_objects() wouldn't do
> anything, because the ops would already be freed.

They are not necessary when all replaced patches are removed from
the stack. There will be no livepatch if this one gets disabled.


> Either way, I guess freeing the nops is optional here, since they would
> also get freed on patch unregister?

If we keep NOPs here, they will be necessary also by the next
cumulative patch. By other words, if we want to get rid of the ftrace
handler when it is not needed, we need to remove the NOPs as well.


> Anyway, all my points here would be moot if we made the nops more
> permanent and allowedd the 'replace' patch to be rolled back to the
> previous patch.

Yes, for v11, I moved all these changes to the followup patch
that removes the replaced patches.

Best Regards,
Petr

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Josh Poimboeuf]

On Tue, Mar 20, 2018 at 01:25:38PM +0100, Petr Mladek wrote:
> On Mon 2018-03-19 16:43:24, Josh Poimboeuf wrote:
> > On Mon, Mar 19, 2018 at 04:02:07PM +0100, Petr Mladek wrote:
> > > > Can someone remind me why we're permanently disabling replaced patches?
> > > > I seem to remember being involved in that decision, but at least with
> > > > this latest version of the patches, it seems like it would be simpler to
> > > > just let 'replace' patches be rolled back to the previous state when
> > > > they're unpatched.  Then we don't need two lists of patches, the nops
> > > > can become more permanent, the replaced patches remain "enabled" but
> > > > inert, and the unpatching behavior is less surprising to the user, more
> > > > like a normal rollback.
> > > 
> > > Yes, keeping the patches might make some things easier. But it might
> > > also bring some problems and it would make the feature less useful.
> > > The following arguments come to my mind:
> > > 
> > > 1. The feature should help to keep the system in a consistent and
> > >    well defined state. It should not depend on what patches were
> > >    installed before.
> > 
> > But the nops already accomplish that.  If they didn't, then this patch
> > set has a major problem.
> 
> The nops are enough to keep the functionality but they might harm
> the performance.
> 
> Livepatching is about preventing bugs without reboot. I could simply
> imagine that ftrace on a hot patch might cause performance
> problems on some workloads. And I would like to have a way out in
> this case.

But this is still an imaginary performance issue.  Premature
optimization and all that...

> Anyway, I am reworking the patchset so that it implements your
> approach first. The possibility to remove NOPs and replaced
> livepatches is done via a followup patch. This might help
> to discuss if the changes are worth it or not.

Thanks, that will definitely help.

> > > 2. The feature should allow to unpatch some functions while keeping
> > >    the others patched.
> > > 
> > >    The ftrace handler might cause some unwanted slowdown or other
> > >    problems. The performance might get restored only when we remove
> > >    the NOPs when they are not longer necessary.
> > 
> > I'd say simplicity and maintainability of the code is more important
> > than an (imagined) performance issue.  The NOPs should be pretty fast
> > anyway.
> > 
> > Not to mention that my proposal would make the behavior less surprising
> > and more user friendly (reverting a 'replace' patch restores it to its
> > previous state).
> 
> If the "disable" way works as expected, see below.
> 
> Also it is less surprising only if people understand the stack of
> patches. If they are familiar only with replace patches then it is
> normal for them that the patches get replaced. It is then like
> a package version update.

Fair enough.  It does seem analagous to package management in that way.

> > > 3. The handling of callbacks is already problematic. We run only
> > >    the ones from the last patch to make things easier.
> > > 
> > >    We would need to come with something more complicated if we
> > >    want to support rollback to "random" patches on the stack.
> > >    And support for random patches is fundamental at least
> > >    from my point of view.
> > 
> > Can you elaborate on what you mean by random patches and why it would
> > require something more complicated from the callbacks?
> 
> Let's say that we will use atomic replace for cumulative
> patches. Then every new patch knows what earlier patches did.
> It just did not know which of them was already installed. Therefore
> it needs to detect what callbacks were already called. The callbacks
> usually create or change something. So there should be something to check.
> Therefore the way forward should be rather straightforward.
> 
> The way back is more problematic. The callbacks in the new cumulative
> patch would need to store information about the previous state and
> be able to restore it when the patch gets disabled. It might more
> or less double the callbacks code and testing scenarios.

Makes sense.

> > > > Along those lines, I'd also propose that we constrain our existing patch
> > > > stacking even further.  Right now we allow a new patch to be registered
> > > > on top of a disabled patch, though we don't allow the new patch to be
> > > > enabled until the previous patch gets enabled.  I'd propose we no longer
> > > > allow that condition.  We should instead enforce that all existing
> > > > patches are *enabled* before allowing a new patch to be registered on
> > > > top.  That way the patch stacking is even more sane, and there are less
> > > > "unusual" conditions to worry about.  We have enough of those already.
> > > > Each additional bit of flexibility has a maintenance cost, and this one
> > > > isn't worth it IMO.
> > > 
> > > Again, this might make some things easier but it might also bring
> > > problems.
> > > 
> > > For example, we would need to solve the situation when the last
> > > patch is disabled and cannot be removed because the transition
> > > was forced. This might be more common after removing the immediate
> > > feature.
> > 
> > I would stop worrying about forced patches so much :-)
> 
> I have already seen blocked transition several times. It is true that
> it was with kGraft. But we just do not have enough real life experience
> with the upstream livepatch code.

But we're talking about patching on top of a *disabled* patch.  Forced
or not, why would the patch be disabled in the first place?

We were just recently discussing the possibility of not allowing the
disabling of patches at all.  If we're not going that far, let's at
least further restrict it, for the sanity of our code, so we don't have
to worry about a nasty mismatched stack of enabled/disabled/enabled/etc,
at least for the cases where 'replace' patches aren't used.

> > Forced patches already come with a disclaimer, and we can't bend over
> > backwards for them.  In such a rare case, the admin can just re-enable
> > the forced patch before loading the 'replace' patch.
> 
> 
> > > Also it might be less user friendly.
> > 
> > I don't know, does anybody really care about this case (patching on top
> > of a disabled patch)?  It just adds to the crazy matrix of possible
> > scenarios we have to keep in our heads, which means more bugs, for very
> > little (hypothetical) gain.
> 
> It depends if the we remove the replaced patches or not. If they are
> removed then replacing disabled patches is rather trivial from both
> coding and understanding side.
> 
> I am going to add this as a separate patch as well. Let's discuss
> it with the code.

Thanks.  I'm hoping this will be a nice improvement, regardless of the
nops thing.

> > > White the atomic replace could make things easier for both developers
> > > and users.
> > 
> > I agree that atomic replace is a useful feature and I'm not arguing
> > against it, so maybe I missed your point?
> 
> Your suggestion allows easier code but it reduces the advantages of
> the atomic replace feature. We would achieve almost the same results
> with a normal livepatch where the functions behave like in
> the original code.
> 
> Also removing replaced patches can be seen as a clean up after
> each patch. It might be more code but the target system might be
> easier to debug. Also we do not need to mind about various
> disable scenarios.

After these discussions I realize that there are several motivations for
this patch set:

1) Atomically reverting some functions in a previous patch while
   upgrading other functions: accomplished by inserting nops for
   reverted functions.  Could also be accomplished by tooling which
   patches functions with duplicates of the originals.

2) Improving performance implications of #1: accomplished by removing
   the nops and old patch modules.

3) Decreasing user confusion about stacking order and what patches are
   currently in effect: accomplished by permanently disabling and
   removing the replaced patches.  Could also be accomplished by a
   better sysfs interface and tooling.

4) Improving debugging?  How?  (It seems to me that removing the
   replaced patches could make debugging more difficult, as it erases
   the patching history (similar to a git rebase))

It would be good to document all those.

Anyway, I need to think about it some more, but I may be warming up to
the idea of permanently disabling the replaced patches.  Your package
management analogy helped me to understand it better.  Especially since
we'll be delivering these patches in packages, so the patch state would
mirror the patch state.  It might be good to describe that analogy in
the documentation as well.

Still, I do think splitting up the "controversial" bits into separate
patches is a good idea, if possible.  Thanks for doing that.

-- 
Josh

Re: [PATCH v10 06/10] livepatch: Add atomic replace

[Josh Poimboeuf]

On Tue, Mar 20, 2018 at 03:35:01PM +0100, Petr Mladek wrote:
> On Tue 2018-03-13 17:48:04, Josh Poimboeuf wrote:
> > On Wed, Mar 07, 2018 at 09:20:35AM +0100, Petr Mladek wrote:
> > > From: Jason Baron <jbaron@akamai.com>
> > > 
> > > Sometimes we would like to revert a particular fix. Currently, this
> > > is not easy because we want to keep all other fixes active and we
> > > could revert only the last applied patch.
> > > 
> > > One solution would be to apply new patch that implemented all
> > > the reverted functions like in the original code. It would work
> > > as expected but there will be unnecessary redirections. In addition,
> > > it would also require knowing which functions need to be reverted at
> > > build time.
> > > 
> > > Another problem is when there are many patches that touch the same
> > > functions. There might be dependencies between patches that are
> > > not enforced on the kernel side. Also it might be pretty hard to
> > > actually prepare the patch and ensure compatibility with
> > > the other patches.
> > > 
> > > A better solution would be to create cumulative patch and say that
> > > it replaces all older ones.
> > > 
> > > This patch adds a new "replace" flag to struct klp_patch. When it is
> > > enabled, a set of 'nop' klp_func will be dynamically created for all
> > > functions that are already being patched but that will no longer be
> > > modified by the new patch. They are temporarily used as a new target
> > > during the patch transition.
> > > 
> > > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > > index fd0296859ff4..ad508a86b2f9 100644
> > > --- a/kernel/livepatch/core.c
> > > +++ b/kernel/livepatch/core.c
> > > +static int klp_add_nops(struct klp_patch *patch)
> > > +{
> > > +	struct klp_patch *old_patch;
> > > +	struct klp_object *old_obj;
> > > +	int err = 0;
> > > +
> > > +	if (WARN_ON(!patch->replace))
> > > +		return -EINVAL;
> > 
> > IMO, this is another one of those overly paranoid warnings that isn't
> > really needed.  Why would we call klp_add_nops() for a non-replace
> > patch?
> 
> Just to be sure. What is the difference, for example, against the following
> checks in __klp_enable_patch() from your point of view, please?
> 
> 	if (klp_transition_patch)
> 		return -EBUSY;
> 
> 	if (WARN_ON(patch->enabled))
> 		return -EINVAL;
> 
> One difference is that klp_enable_patch() is exported symbol. One the
> other hand, livepatch code developers could do mistakes as well.
> Adding nops sounds like an innoncent operation after all ;-)

But klp_enable_patch() being an exported symbol is an important
difference.  It catches a patch author abusing the interface.  Which is
much more likely than one of us accidentally calling klp_add_nops().
Have you not noticed how thorough our code reviews are? ;-)

Anyway, I suppose it's a harmless check and I don't feel very strongly
about it, it just seems unnecessary.

> > > +	list_for_each_entry(old_patch, &klp_patches, list) {
> > > +		klp_for_each_object(old_patch, old_obj) {
> > > +			err = klp_add_object_nops(patch, old_obj);
> > > +			if (err)
> > > +				return err;
> > > +		}
> > > +	}
> > > +
> > > +	return 0;
> > > +}
> 
> [...]
> 
> > > diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
> > > index 6917100fbe79..d6af190865d2 100644
> > > --- a/kernel/livepatch/transition.c
> > > +++ b/kernel/livepatch/transition.c
> > > @@ -87,6 +87,36 @@ static void klp_complete_transition(void)
> > >  		 klp_transition_patch->mod->name,
> > >  		 klp_target_state == KLP_PATCHED ? "patching" : "unpatching");
> > >  
> > > +	/*
> > > +	 * For replace patches, we disable all previous patches, and replace
> > > +	 * the dynamic no-op functions by removing the ftrace hook.
> > > +	 */
> > > +	if (klp_transition_patch->replace && klp_target_state == KLP_PATCHED) {
> > > +		/*
> > > +		 * Make sure that no ftrace handler accesses any older patch
> > > +		 * on the stack.  This might happen when the user forced the
> > > +		 * transaction while some running tasks were still falling
> > > +		 * back to the old code.  There might even still be ftrace
> > > +		 * handlers that have not seen the last patch on the stack yet.
> > > +		 *
> > > +		 * It probably is not necessary because of the rcu-safe access.
> > > +		 * But better be safe than sorry.
> > > +		 */
> > > +		if (klp_forced)
> > > +			klp_synchronize_transition();
> > 
> > I don't like this.  Hopefully we can get just rid of it, if we also get
> > rid of the concept of "throwing away" patches like I proposed.
> 
> What exactly you do not like about it, please?
> 
> It is not needed if all processes were migrated using the consistency
> model, definitely.
> 
> If the transition has been forced then the barrier should be needed from
> similar reasons as the barrier after klp_unpatch_objects() below.
> We basically want to be sure what ftrace handlers see on the stack.
> 
> Will it help, when I remove the last paragraph where the formulation
> is quite uncertain?

Well, the last paragraph doesn't inspire a lot of confidence ;-)  It
sounds like voodoo.

Also the comment just seems very confusing to me:

- What specifically is it protecting against, e.g., _why_ should no
  ftrace handler access any old patch on the stack, and when shouldn't
  it do so?  Is the barrier needed before func->transition is cleared,
  or what?

- Why is it located where it is?  i.e., why does it come before
  klp_throw_away_replaced_patches() instead of after, unlike the
  unpatching barrier?  What exactly does it need to come before, and
  what exactly does it need to come after?  Can it be combined with the
  other klp_synchronize_transition() for KLP_PATCHED?

- The comment doesn't describe what it has to do with 'replace'.

- Why is it only needed for the 'force' case?

- Does RCU make it safe, or doesn't it?  If yes, why is this needed?  If
  no, why not?

- The wording is imprecise.  Technically, the ftrace handler accesses
  funcs, not patches.  Also, "ftrace handler" and "stack" are ambiguous,
  "klp_ftrace_handler()" and "ops->func_stack" would be more clear.

> > > +
> > > +		klp_throw_away_replaced_patches(klp_transition_patch,
> > > +						klp_forced);
> > > +
> > > +		/*
> > > +		 * There is no need to synchronize the transition after removing
> > > +		 * nops. They must be the last on the func_stack. Ftrace
> > > +		 * gurantees that nobody will stay in the trampoline after
> > 
> > "guarantees"
> > 
> > > +		 * the ftrace handler is unregistered.
> > > +		 */
> > > +		klp_unpatch_objects(klp_transition_patch, KLP_FUNC_NOP);
> > > +	}
> > > +
> > >  	if (klp_target_state == KLP_UNPATCHED) {
> > >  		/*
> > >  		 * All tasks have transitioned to KLP_UNPATCHED so we can now
> > > @@ -143,6 +173,15 @@ static void klp_complete_transition(void)
> > >  	if (!klp_forced && klp_target_state == KLP_UNPATCHED)
> > >  		module_put(klp_transition_patch->mod);
> > >  
> > > +	/*
> > > +	 * We do not need to wait until the objects are really freed.
> > > +	 * The patch must be on the bottom of the stack. Therefore it
> > > +	 * will never replace anything else. The only important thing
> > > +	 * is that we wait when the patch is being unregistered.
> > > +	 */
> > > +	if (klp_transition_patch->replace && klp_target_state == KLP_PATCHED)
> > > +		klp_free_objects(klp_transition_patch, KLP_FUNC_NOP);
> > > +
> > 
> > This makes me a bit nervous.  What happens if the patch is enabled, then
> > disabled, then enabled again?  Then klp_free_objects() wouldn't do
> > anything, because the ops would already be freed.
> 
> They are not necessary when all replaced patches are removed from
> the stack. There will be no livepatch if this one gets disabled.

My point was that if you enable, then disable, then enable again,
klp_free_objects() will get called again, and it will do nothing the
second time around.

Maybe that's safe in this instance, but in general, it's easy to forget
the re-enable case when adding special cases for 'patch->replace'.  I
get the feeling that it would be safer to just clear 'patch->replace'
after this step to avoid such scenarios.  After all, when re-enabling a
'replace' patch, it's no longer replacing anything (assuming here that a
replace patch will permanently disable all previous patches).

-- 
Josh

Re: [PATCH v10 06/10] livepatch: Add atomic replace

[Petr Mladek]

On Tue 2018-03-20 16:26:19, Josh Poimboeuf wrote:
> On Tue, Mar 20, 2018 at 03:35:01PM +0100, Petr Mladek wrote:
> > On Tue 2018-03-13 17:48:04, Josh Poimboeuf wrote:
> > > On Wed, Mar 07, 2018 at 09:20:35AM +0100, Petr Mladek wrote:
> > > > This patch adds a new "replace" flag to struct klp_patch. When it is
> > > > enabled, a set of 'nop' klp_func will be dynamically created for all
> > > > functions that are already being patched but that will no longer be
> > > > modified by the new patch. They are temporarily used as a new target
> > > > during the patch transition.
> > > > 
> > > > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > > > index fd0296859ff4..ad508a86b2f9 100644
> > > > --- a/kernel/livepatch/core.c
> > > > +++ b/kernel/livepatch/core.c
> > > > +static int klp_add_nops(struct klp_patch *patch)
> > > > +{
> > > > +	struct klp_patch *old_patch;
> > > > +	struct klp_object *old_obj;
> > > > +	int err = 0;
> > > > +
> > > > +	if (WARN_ON(!patch->replace))
> > > > +		return -EINVAL;
> > > 
> > > IMO, this is another one of those overly paranoid warnings that isn't
> > > really needed.  Why would we call klp_add_nops() for a non-replace
> > > patch?
> > 
> > Just to be sure. What is the difference, for example, against the following
> > checks in __klp_enable_patch() from your point of view, please?
> > 
> > 	if (klp_transition_patch)
> > 		return -EBUSY;
> > 
> > 	if (WARN_ON(patch->enabled))
> > 		return -EINVAL;
> > 
> > One difference is that klp_enable_patch() is exported symbol. One the
> > other hand, livepatch code developers could do mistakes as well.
> > Adding nops sounds like an innoncent operation after all ;-)
> 
> But klp_enable_patch() being an exported symbol is an important
> difference.  It catches a patch author abusing the interface.  Which is
> much more likely than one of us accidentally calling klp_add_nops().
> Have you not noticed how thorough our code reviews are? ;-)
> 
> Anyway, I suppose it's a harmless check and I don't feel very strongly
> about it, it just seems unnecessary.

I have removed the check.


> > > > diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
> > > > index 6917100fbe79..d6af190865d2 100644
> > > > --- a/kernel/livepatch/transition.c
> > > > +++ b/kernel/livepatch/transition.c
> > > > @@ -87,6 +87,36 @@ static void klp_complete_transition(void)
> > > >  		 klp_transition_patch->mod->name,
> > > >  		 klp_target_state == KLP_PATCHED ? "patching" : "unpatching");
> > > >  
> > > > +	/*
> > > > +	 * For replace patches, we disable all previous patches, and replace
> > > > +	 * the dynamic no-op functions by removing the ftrace hook.
> > > > +	 */
> > > > +	if (klp_transition_patch->replace && klp_target_state == KLP_PATCHED) {
> > > > +		/*
> > > > +		 * Make sure that no ftrace handler accesses any older patch
> > > > +		 * on the stack.  This might happen when the user forced the
> > > > +		 * transaction while some running tasks were still falling
> > > > +		 * back to the old code.  There might even still be ftrace
> > > > +		 * handlers that have not seen the last patch on the stack yet.
> > > > +		 *
> > > > +		 * It probably is not necessary because of the rcu-safe access.
> > > > +		 * But better be safe than sorry.
> > > > +		 */
> > > > +		if (klp_forced)
> > > > +			klp_synchronize_transition();
> > > 
> > > I don't like this.  Hopefully we can get just rid of it, if we also get
> > > rid of the concept of "throwing away" patches like I proposed.
> > 
> > What exactly you do not like about it, please?
> > 
> > It is not needed if all processes were migrated using the consistency
> > model, definitely.
> > 
> > If the transition has been forced then the barrier should be needed from
> > similar reasons as the barrier after klp_unpatch_objects() below.
> > We basically want to be sure what ftrace handlers see on the stack.
> > 
> > Will it help, when I remove the last paragraph where the formulation
> > is quite uncertain?
> 
> Well, the last paragraph doesn't inspire a lot of confidence ;-)  It
> sounds like voodoo.

Races are never easy area. You are right that I was not completely
confident and wanted to be on the safe side. Your questions helped
me to realize that the synchronization is not neeeded.


> Also the comment just seems very confusing to me:
> 
> - What specifically is it protecting against, e.g., _why_ should no
>   ftrace handler access any old patch on the stack, and when shouldn't
>   it do so?  Is the barrier needed before func->transition is cleared,
>   or what?

I was afraid of invalid memory accessed in klp_ftrace_handler().
I simply underestimated the power of RCU. I am not that familiar
with it. Also the following is a bit non-standard:

	func = list_entry_rcu(func->stack_node.next,
			      struct klp_func, stack_node);

Anyway, you made me to check it. All looks safe after all.

> - Does RCU make it safe, or doesn't it?  If yes, why is this needed?  If
>   no, why not?

Yes. I removed the synchronization. Instead, I explained the situation
in a comment above klp_discard_replaced_patches().


 
> > > > +
> > > > +		klp_throw_away_replaced_patches(klp_transition_patch,
> > > > +						klp_forced);
> > > > +
> > > > +		/*
> > > > +		 * There is no need to synchronize the transition after removing
> > > > +		 * nops. They must be the last on the func_stack. Ftrace
> > > > +		 * gurantees that nobody will stay in the trampoline after
> > > 
> > > "guarantees"
> > > 
> > > > +		 * the ftrace handler is unregistered.
> > > > +		 */
> > > > +		klp_unpatch_objects(klp_transition_patch, KLP_FUNC_NOP);
> > > > +	}
> > > > +
> > > >  	if (klp_target_state == KLP_UNPATCHED) {
> > > >  		/*
> > > >  		 * All tasks have transitioned to KLP_UNPATCHED so we can now
> > > > @@ -143,6 +173,15 @@ static void klp_complete_transition(void)
> > > >  	if (!klp_forced && klp_target_state == KLP_UNPATCHED)
> > > >  		module_put(klp_transition_patch->mod);
> > > >  
> > > > +	/*
> > > > +	 * We do not need to wait until the objects are really freed.
> > > > +	 * The patch must be on the bottom of the stack. Therefore it
> > > > +	 * will never replace anything else. The only important thing
> > > > +	 * is that we wait when the patch is being unregistered.
> > > > +	 */
> > > > +	if (klp_transition_patch->replace && klp_target_state == KLP_PATCHED)
> > > > +		klp_free_objects(klp_transition_patch, KLP_FUNC_NOP);
> > > > +
> > > 
> > > This makes me a bit nervous.  What happens if the patch is enabled, then
> > > disabled, then enabled again?  Then klp_free_objects() wouldn't do
> > > anything, because the ops would already be freed.
> > 
> > They are not necessary when all replaced patches are removed from
> > the stack. There will be no livepatch if this one gets disabled.
> 
> My point was that if you enable, then disable, then enable again,
> klp_free_objects() will get called again, and it will do nothing the
> second time around.

> Maybe that's safe in this instance, but in general, it's easy to forget
> the re-enable case when adding special cases for 'patch->replace'.

Yes, it is safe.


> I get the feeling that it would be safer to just clear 'patch->replace'
> after this step to avoid such scenarios.  After all, when re-enabling a
> 'replace' patch, it's no longer replacing anything (assuming here that a
> replace patch will permanently disable all previous patches).

I am not completely comfortable with touching item that is set by the
author of the patch. On the other hand, I do not see how it could
harm. I have just added the line to disable the flag.

Best Regards,
Petr

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Petr Mladek]

On Tue 2018-03-20 15:15:02, Josh Poimboeuf wrote:
> On Tue, Mar 20, 2018 at 01:25:38PM +0100, Petr Mladek wrote:
> > On Mon 2018-03-19 16:43:24, Josh Poimboeuf wrote:
> > > On Mon, Mar 19, 2018 at 04:02:07PM +0100, Petr Mladek wrote:
> > > > > Along those lines, I'd also propose that we constrain our existing patch
> > > > > stacking even further.  Right now we allow a new patch to be registered
> > > > > on top of a disabled patch, though we don't allow the new patch to be
> > > > > enabled until the previous patch gets enabled.  I'd propose we no longer
> > > > > allow that condition.  We should instead enforce that all existing
> > > > > patches are *enabled* before allowing a new patch to be registered on
> > > > > top.  That way the patch stacking is even more sane, and there are less
> > > > > "unusual" conditions to worry about.  We have enough of those already.
> > > > > Each additional bit of flexibility has a maintenance cost, and this one
> > > > > isn't worth it IMO.
> > > > 
> > > > Again, this might make some things easier but it might also bring
> > > > problems.
> > > > 
> > > > For example, we would need to solve the situation when the last
> > > > patch is disabled and cannot be removed because the transition
> > > > was forced. This might be more common after removing the immediate
> > > > feature.
> > > 
> > > I would stop worrying about forced patches so much :-)
> > 
> > I have already seen blocked transition several times. It is true that
> > it was with kGraft. But we just do not have enough real life experience
> > with the upstream livepatch code.
> 
> But we're talking about patching on top of a *disabled* patch.  Forced
> or not, why would the patch be disabled in the first place?

For example, it might be disabled because the transition stalled for
too long and the user reverted it. Or just because it is possible
to disable it.


> We were just recently discussing the possibility of not allowing the
> disabling of patches at all.  If we're not going that far, let's at
> least further restrict it, for the sanity of our code, so we don't have
> to worry about a nasty mismatched stack of enabled/disabled/enabled/etc,
> at least for the cases where 'replace' patches aren't used.

I am not completely sure where all these fears come from. From my
point of view, this change is pretty safe and trivial thanks to NOPs
and overall design. It would be a shame to do not have it. But I
might be blind after spending so much time with the feature.

BTW: It seems that I am more paranoid when being in the reviewer role.

Anyway, I am getting close to send v11. This change is done by a separate
and last code-related patch. So, it can be omitted rather easily.


> After these discussions I realize that there are several motivations for
> this patch set:
> 
> 1) Atomically reverting some functions in a previous patch while
>    upgrading other functions: accomplished by inserting nops for
>    reverted functions.  Could also be accomplished by tooling which
>    patches functions with duplicates of the originals.
> 
> 2) Improving performance implications of #1: accomplished by removing
>    the nops and old patch modules.
> 
> 3) Decreasing user confusion about stacking order and what patches are
>    currently in effect: accomplished by permanently disabling and
>    removing the replaced patches.  Could also be accomplished by a
>    better sysfs interface and tooling.
> 
> 4) Improving debugging?  How?  (It seems to me that removing the
>    replaced patches could make debugging more difficult, as it erases
>    the patching history (similar to a git rebase))

This explains why you might be against replacing disabled patches. It could
confuse the history.

On the other hand, it might also hide bugs and create mysterious ones.
If we remove code that should not longer be used, the system should
crash immediately if the removed code/data is accessed by mistake.
IMHO, this should help to catch bugs early and in more clear situation.

The history can be seen in the logs or in users reports. I know
that both sources are not 100% reliable but...

IMHO, it is case by case. I could imagine situations where the clean
up might help and situations where it might cause loosing information.
It is hard to judge. I still vote for the clean up ;-)


> It would be good to document all those.
> 
> Anyway, I need to think about it some more, but I may be warming up to
> the idea of permanently disabling the replaced patches.  Your package
> management analogy helped me to understand it better.  Especially since
> we'll be delivering these patches in packages, so the patch state would
> mirror the patch state.  It might be good to describe that analogy in
> the documentation as well.

I have updated the documentation a bit. I did not have the energy
to completely rewrite it though. Anyway, we still have to agree
on the code first.

Best Regards,
Petr

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Josh Poimboeuf]

On Fri, Mar 23, 2018 at 10:45:07AM +0100, Petr Mladek wrote:
> On Tue 2018-03-20 15:15:02, Josh Poimboeuf wrote:
> > On Tue, Mar 20, 2018 at 01:25:38PM +0100, Petr Mladek wrote:
> > > On Mon 2018-03-19 16:43:24, Josh Poimboeuf wrote:
> > > > On Mon, Mar 19, 2018 at 04:02:07PM +0100, Petr Mladek wrote:
> > > > > > Along those lines, I'd also propose that we constrain our existing patch
> > > > > > stacking even further.  Right now we allow a new patch to be registered
> > > > > > on top of a disabled patch, though we don't allow the new patch to be
> > > > > > enabled until the previous patch gets enabled.  I'd propose we no longer
> > > > > > allow that condition.  We should instead enforce that all existing
> > > > > > patches are *enabled* before allowing a new patch to be registered on
> > > > > > top.  That way the patch stacking is even more sane, and there are less
> > > > > > "unusual" conditions to worry about.  We have enough of those already.
> > > > > > Each additional bit of flexibility has a maintenance cost, and this one
> > > > > > isn't worth it IMO.
> > > > > 
> > > > > Again, this might make some things easier but it might also bring
> > > > > problems.
> > > > > 
> > > > > For example, we would need to solve the situation when the last
> > > > > patch is disabled and cannot be removed because the transition
> > > > > was forced. This might be more common after removing the immediate
> > > > > feature.
> > > > 
> > > > I would stop worrying about forced patches so much :-)
> > > 
> > > I have already seen blocked transition several times. It is true that
> > > it was with kGraft. But we just do not have enough real life experience
> > > with the upstream livepatch code.
> > 
> > But we're talking about patching on top of a *disabled* patch.  Forced
> > or not, why would the patch be disabled in the first place?
> 
> For example, it might be disabled because the transition stalled for
> too long and the user reverted it. Or just because it is possible
> to disable it.

If they haven't previously forced any patches, and they reverted the
topmost patch because it stalled, they can easily unload the patch.

If they *have* previously forced a patch, they can force enable the
topmost patch as well, or if that's not safe they can reboot (that's
what you get for forcing a patch...)

> > We were just recently discussing the possibility of not allowing the
> > disabling of patches at all.  If we're not going that far, let's at
> > least further restrict it, for the sanity of our code, so we don't have
> > to worry about a nasty mismatched stack of enabled/disabled/enabled/etc,
> > at least for the cases where 'replace' patches aren't used.
> 
> I am not completely sure where all these fears come from. From my
> point of view, this change is pretty safe and trivial thanks to NOPs
> and overall design. It would be a shame to do not have it. But I
> might be blind after spending so much time with the feature.

I think you're missing my point.  This isn't about your patch set, per
se.  It's really about our existing code.  Today, our patch stack
doesn't follow real stack semantics, because patches in the middle might
be disabled.  I see that as a problem.

If 'replace' were the only mode, then we wouldn't even need a patch
stack because it wouldn't really matter much whether the previous patch
is enabled or disabled.  I think this is in agreement with the point
you're making.

But we still support non-replace patches.  My feeling is that we should
either do a true stack, or no stack at all.  The in-between thing is
going to be confusing, not only for us, but for patch authors and end
users.

In fact, *no stack* might be better, now that we have replace.
Cumulative patches can use replace, so a stack is no longer needed for
them.  And for non cumulative patches, a stack may not make sense, since
the patches should be independent anyway.

But I don't know how big of a deal it is.

> BTW: It seems that I am more paranoid when being in the reviewer role.
> 
> Anyway, I am getting close to send v11. This change is done by a separate
> and last code-related patch. So, it can be omitted rather easily.

Thanks a lot for splitting it up.  It looks much easier to review now.

I still would really like to see the selftests merged first (or at the
same time).

Unfortunately I'll be out next week, so I won't be able to give it a
proper review until I get back.  Maybe I will come back in a more
ACK-conducive mood ;-)

-- 
Josh

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Petr Mladek]

On Fri 2018-03-23 17:44:10, Josh Poimboeuf wrote:
> On Fri, Mar 23, 2018 at 10:45:07AM +0100, Petr Mladek wrote:
> > On Tue 2018-03-20 15:15:02, Josh Poimboeuf wrote:
> > > On Tue, Mar 20, 2018 at 01:25:38PM +0100, Petr Mladek wrote:
> > > > On Mon 2018-03-19 16:43:24, Josh Poimboeuf wrote:
> > > > > On Mon, Mar 19, 2018 at 04:02:07PM +0100, Petr Mladek wrote:
> > > > > > > Along those lines, I'd also propose that we constrain our existing patch
> > > > > > > stacking even further.  Right now we allow a new patch to be registered
> > > > > > > on top of a disabled patch, though we don't allow the new patch to be
> > > > > > > enabled until the previous patch gets enabled.  I'd propose we no longer
> > > > > > > allow that condition.  We should instead enforce that all existing
> > > > > > > patches are *enabled* before allowing a new patch to be registered on
> > > > > > > top.  That way the patch stacking is even more sane, and there are less
> > > > > > > "unusual" conditions to worry about.  We have enough of those already.
> > > > > > > Each additional bit of flexibility has a maintenance cost, and this one
> > > > > > > isn't worth it IMO.
> > > > > > 
> > > > > > Again, this might make some things easier but it might also bring
> > > > > > problems.
> > > > > > 
> > > > > > For example, we would need to solve the situation when the last
> > > > > > patch is disabled and cannot be removed because the transition
> > > > > > was forced. This might be more common after removing the immediate
> > > > > > feature.
> > > > > 
> > > > > I would stop worrying about forced patches so much :-)
> > > > 
> > > > I have already seen blocked transition several times. It is true that
> > > > it was with kGraft. But we just do not have enough real life experience
> > > > with the upstream livepatch code.
> > > 
> > > But we're talking about patching on top of a *disabled* patch.  Forced
> > > or not, why would the patch be disabled in the first place?
> > 
> > For example, it might be disabled because the transition stalled for
> > too long and the user reverted it. Or just because it is possible
> > to disable it.
> 
> If they haven't previously forced any patches, and they reverted the
> topmost patch because it stalled, they can easily unload the patch.
> 
> If they *have* previously forced a patch, they can force enable the
> topmost patch as well, or if that's not safe they can reboot (that's
> what you get for forcing a patch...)

IMHO, the reboot is the very last option for people that are using
livepatching.


> > > We were just recently discussing the possibility of not allowing the
> > > disabling of patches at all.  If we're not going that far, let's at
> > > least further restrict it, for the sanity of our code, so we don't have
> > > to worry about a nasty mismatched stack of enabled/disabled/enabled/etc,
> > > at least for the cases where 'replace' patches aren't used.
> > 
> > I am not completely sure where all these fears come from. From my
> > point of view, this change is pretty safe and trivial thanks to NOPs
> > and overall design. It would be a shame to do not have it. But I
> > might be blind after spending so much time with the feature.
> 
> I think you're missing my point.  This isn't about your patch set, per
> se.  It's really about our existing code.  Today, our patch stack
> doesn't follow real stack semantics, because patches in the middle might
> be disabled.  I see that as a problem.

This would be true if we keep the replaced patches on the stack. But
if we remove the replaced patches then there never will be disabled
patches in the middle.

OK, there might be disabled patches in the middle during the
transition. But this is the situation where we basically could
not manipulate the stack.


> If 'replace' were the only mode, then we wouldn't even need a patch
> stack because it wouldn't really matter much whether the previous patch
> is enabled or disabled.  I think this is in agreement with the point
> you're making.
> 
> But we still support non-replace patches.  My feeling is that we should
> either do a true stack, or no stack at all.  The in-between thing is
> going to be confusing, not only for us, but for patch authors and end
> users.

I see it like two different modes. We either have a stack of patches
that depend on each other. Or we have replace patches that are
standalone and we allow a smooth transfer from one to another one.

Anyway, for us, it is much more important the removal of replaced
patches. We could probably live without the possibility to replace
disabled patches.


> > Anyway, I am getting close to send v11. This change is done by a separate
> > and last code-related patch. So, it can be omitted rather easily.
> 
> Thanks a lot for splitting it up.  It looks much easier to review now.
>
> I still would really like to see the selftests merged first (or at the
> same time).

The selftest passes with the patchset. I think that it is more or
less ready to be merged. It is possible that it might need some
improvements in the future but it looks like a good start.


> Unfortunately I'll be out next week, so I won't be able to give it a
> proper review until I get back.  Maybe I will come back in a more
> ACK-conducive mood ;-)

Thanks for review and info.

Best Regards,
Petr

Re: [PATCH v10 00/10] livepatch: Atomic replace feature

[Petr Mladek]

On Mon 2018-03-12 14:57:04, Joe Lawrence wrote:
> Hi Petr,
> 
> These are the callback tests that I hacked up into a livepatch
> kselftest.  (Basically I copied a bunch of the sample modules and
> verified the expected dmesg output as I had listed in in the
> Documentation/livepatch/callbacks.txt file.)  The script is still a
> little rough and maybe this isn't the direction we want to go for proper
> kselftests, but perhaps it saves you some time/sanity for verifying this
> patchset.



> Hope this helps,
> 
> -- Joe
> 
> -- >8 -- >8 -- >8 -- >8 --
> 
> >From 0364430c53e12e21923bed20cb651374b4cf9ba9 Mon Sep 17 00:00:00 2001
> From: Joe Lawrence <joe.lawrence@redhat.com>
> Date: Tue, 6 Mar 2018 17:32:25 -0500
> Subject: WIP - livepatch kselftest
> 
> CONFIG_TEST_LIVEPATCH=m
> % make -C tools/testing/selftests TARGETS=livepatch run_tests
> 
> Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
> ---
>  lib/Kconfig.debug                                |  11 +
>  lib/Makefile                                     |   9 +
>  lib/test_klp_callbacks_busy.c                    |  58 ++
>  lib/test_klp_callbacks_demo.c                    | 205 +++++++
>  lib/test_klp_callbacks_demo2.c                   | 149 +++++
>  lib/test_klp_callbacks_mod.c                     |  39 ++

I would suggest to put it into lib/livepatch/ subdirectory. I hope
that we will add more tests in the long term.


> diff --git a/tools/testing/selftests/livepatch/livepatch-test b/tools/testing/selftests/livepatch/livepatch-test
> new file mode 100755
> index 000000000000..798317bf69f6
> --- /dev/null
> +++ b/tools/testing/selftests/livepatch/livepatch-test

s/livepatch-test/livepatch-test.sh/  ?

It seems to be common to add .sh suffix for bash script names.


> @@ -0,0 +1,658 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (C) 2018 Joe Lawrence <joe.lawrence@redhat.com>
> +
> +MAX_RETRIES=30
> +RETRY_INTERVAL=2	# seconds
> +BETWEEN_TESTS=20	# seconds

These 20 seconds kept me in a tense (waiting for the final result)
for a very long time ;-) Is there any particular reason for such
a long delay?

I wonder if we need a delay at all or if let's say 2 seconds might
be enough.


> +echo -n "TEST1 ... "
> +dmesg -C
> +
> +load_mod $MOD_TARGET
> +load_mod $MOD_LIVEPATCH
> +wait_for_transition $MOD_LIVEPATCH
> +disable_lp $MOD_LIVEPATCH
> +unload_mod $MOD_LIVEPATCH
> +unload_mod $MOD_TARGET
> +
> +check_result "% modprobe $MOD_TARGET
> +$MOD_TARGET: livepatch_callbacks_mod_init
> +% modprobe $MOD_LIVEPATCH
> +livepatch: enabling patch '$MOD_LIVEPATCH'
> +livepatch: '$MOD_LIVEPATCH': initializing patching transition
> +$MOD_LIVEPATCH: pre_patch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
> +$MOD_LIVEPATCH: pre_patch_callback: vmlinux
> +livepatch: '$MOD_LIVEPATCH': starting patching transition
> +livepatch: '$MOD_LIVEPATCH': completing patching transition
> +$MOD_LIVEPATCH: post_patch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
> +$MOD_LIVEPATCH: post_patch_callback: vmlinux
> +livepatch: '$MOD_LIVEPATCH': patching complete
> +% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled
> +livepatch: '$MOD_LIVEPATCH': initializing unpatching transition
> +$MOD_LIVEPATCH: pre_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
> +$MOD_LIVEPATCH: pre_unpatch_callback: vmlinux
> +livepatch: '$MOD_LIVEPATCH': starting unpatching transition
> +livepatch: '$MOD_LIVEPATCH': completing unpatching transition
> +$MOD_LIVEPATCH: post_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
> +$MOD_LIVEPATCH: post_unpatch_callback: vmlinux
> +livepatch: '$MOD_LIVEPATCH': unpatching complete
> +% rmmod $MOD_LIVEPATCH
> +% rmmod $MOD_TARGET
> +$MOD_TARGET: livepatch_callbacks_mod_exit"

I was a bit surprised when seeing this way of checking results.
But on the other hand, it looks pretty effective, especially for
the callbacks. And the 3rd look, any patched function might write
something into the log when called.

I like it. Let's see how it works in the long term. But I am rather
positive.

Thanks a lot for working on it.

Best Regards,
Petr

Re: [PATCH v10 00/10] livepatch: Atomic replace feature

[Joe Lawrence]

On 03/26/2018 06:56 AM, Petr Mladek wrote:
> On Mon 2018-03-12 14:57:04, Joe Lawrence wrote:
>> Hi Petr,
>>
>> These are the callback tests that I hacked up into a livepatch
>> kselftest.  (Basically I copied a bunch of the sample modules and
>> verified the expected dmesg output as I had listed in in the
>> Documentation/livepatch/callbacks.txt file.)  The script is still a
>> little rough and maybe this isn't the direction we want to go for proper
>> kselftests, but perhaps it saves you some time/sanity for verifying this
>> patchset.
> 
> 
> 
>> Hope this helps,
>>
>> -- Joe
>>
>> -- >8 -- >8 -- >8 -- >8 --
>>
>> >From 0364430c53e12e21923bed20cb651374b4cf9ba9 Mon Sep 17 00:00:00 2001
>> From: Joe Lawrence <joe.lawrence@redhat.com>
>> Date: Tue, 6 Mar 2018 17:32:25 -0500
>> Subject: WIP - livepatch kselftest
>>
>> CONFIG_TEST_LIVEPATCH=m
>> % make -C tools/testing/selftests TARGETS=livepatch run_tests
>>
>> Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
>> ---
>>  lib/Kconfig.debug                                |  11 +
>>  lib/Makefile                                     |   9 +
>>  lib/test_klp_callbacks_busy.c                    |  58 ++
>>  lib/test_klp_callbacks_demo.c                    | 205 +++++++
>>  lib/test_klp_callbacks_demo2.c                   | 149 +++++
>>  lib/test_klp_callbacks_mod.c                     |  39 ++
> 
> I would suggest to put it into lib/livepatch/ subdirectory. I hope
> that we will add more tests in the long term.

Good idea, we should encourage more tests in a livepatch/ subdir and not
clutter up the lib/ directory.

>> diff --git a/tools/testing/selftests/livepatch/livepatch-test b/tools/testing/selftests/livepatch/livepatch-test
>> new file mode 100755
>> index 000000000000..798317bf69f6
>> --- /dev/null
>> +++ b/tools/testing/selftests/livepatch/livepatch-test
> 
> s/livepatch-test/livepatch-test.sh/  ?
> 
> It seems to be common to add .sh suffix for bash script names.

I'll rename with the suffix.

>> @@ -0,0 +1,658 @@
>> +#!/bin/bash
>> +# SPDX-License-Identifier: GPL-2.0
>> +# Copyright (C) 2018 Joe Lawrence <joe.lawrence@redhat.com>
>> +
>> +MAX_RETRIES=30
>> +RETRY_INTERVAL=2	# seconds
>> +BETWEEN_TESTS=20	# seconds
> 
> These 20 seconds kept me in a tense (waiting for the final result)
> for a very long time ;-) Is there any particular reason for such
> a long delay?

It certainly builds suspense :)

> I wonder if we need a delay at all or if let's say 2 seconds might
> be enough.

I removed the delays completely and the tests ran successfully.   What
might be better than a between test delay would be some kind of
initial-condition verification, ie, make sure that the test starts/ends
with none of the livepatch test modules loaded.

For the test cases which load multiple livepatches, is there an easy way
to determine the patch stack order from userspace?  I think that would
be helpful when trying to remove all of them.

>> +echo -n "TEST1 ... "
>> +dmesg -C
>> +
>> +load_mod $MOD_TARGET
>> +load_mod $MOD_LIVEPATCH
>> +wait_for_transition $MOD_LIVEPATCH
>> +disable_lp $MOD_LIVEPATCH
>> +unload_mod $MOD_LIVEPATCH
>> +unload_mod $MOD_TARGET
>> +
>> +check_result "% modprobe $MOD_TARGET
>> +$MOD_TARGET: livepatch_callbacks_mod_init
>> +% modprobe $MOD_LIVEPATCH
>> +livepatch: enabling patch '$MOD_LIVEPATCH'
>> +livepatch: '$MOD_LIVEPATCH': initializing patching transition
>> +$MOD_LIVEPATCH: pre_patch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
>> +$MOD_LIVEPATCH: pre_patch_callback: vmlinux
>> +livepatch: '$MOD_LIVEPATCH': starting patching transition
>> +livepatch: '$MOD_LIVEPATCH': completing patching transition
>> +$MOD_LIVEPATCH: post_patch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
>> +$MOD_LIVEPATCH: post_patch_callback: vmlinux
>> +livepatch: '$MOD_LIVEPATCH': patching complete
>> +% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled
>> +livepatch: '$MOD_LIVEPATCH': initializing unpatching transition
>> +$MOD_LIVEPATCH: pre_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
>> +$MOD_LIVEPATCH: pre_unpatch_callback: vmlinux
>> +livepatch: '$MOD_LIVEPATCH': starting unpatching transition
>> +livepatch: '$MOD_LIVEPATCH': completing unpatching transition
>> +$MOD_LIVEPATCH: post_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
>> +$MOD_LIVEPATCH: post_unpatch_callback: vmlinux
>> +livepatch: '$MOD_LIVEPATCH': unpatching complete
>> +% rmmod $MOD_LIVEPATCH
>> +% rmmod $MOD_TARGET
>> +$MOD_TARGET: livepatch_callbacks_mod_exit"
> 
> I was a bit surprised when seeing this way of checking results.
> But on the other hand, it looks pretty effective, especially for
> the callbacks. And the 3rd look, any patched function might write
> something into the log when called.

This was a quickly scripted version of what I was manually verifying
with the sample example livepatches.  I don't know if it will scale, but
it was pretty easy to add tests this way.

I wonder though if better dmesg filters will be required as the
livepatch core adds more debug msgs?

> I like it. Let's see how it works in the long term. But I am rather
> positive.
> 
> Thanks a lot for working on it.

Thanks for taking a look and running the tests.  I'll make some of your
suggested changes and send it up for a proper review soon.

-- Joe

Re: [PATCH v10 00/10] livepatch: Atomic replace feature

[Petr Mladek]

On Mon 2018-03-26 14:12:03, Joe Lawrence wrote:
> On 03/26/2018 06:56 AM, Petr Mladek wrote:
> > On Mon 2018-03-12 14:57:04, Joe Lawrence wrote:
> >> diff --git a/tools/testing/selftests/livepatch/livepatch-test b/tools/testing/selftests/livepatch/livepatch-test
> >> new file mode 100755
> >> index 000000000000..798317bf69f6
> >> --- /dev/null
> >> +++ b/tools/testing/selftests/livepatch/livepatch-test
> >> @@ -0,0 +1,658 @@
> >> +#!/bin/bash
> >> +# SPDX-License-Identifier: GPL-2.0
> >> +# Copyright (C) 2018 Joe Lawrence <joe.lawrence@redhat.com>
> >> +
> >> +MAX_RETRIES=30
> >> +RETRY_INTERVAL=2	# seconds
> >> +BETWEEN_TESTS=20	# seconds
> > 
> > These 20 seconds kept me in a tense (waiting for the final result)
> > for a very long time ;-) Is there any particular reason for such
> > a long delay?
> 
> It certainly builds suspense :)
> 
> > I wonder if we need a delay at all or if let's say 2 seconds might
> > be enough.
> 
> I removed the delays completely and the tests ran successfully.   What
> might be better than a between test delay would be some kind of
> initial-condition verification, ie, make sure that the test starts/ends
> with none of the livepatch test modules loaded.

We could check is /sys/kernel/livepatch directory is empty.

Also we could run modinfo on all modules printed by lsmod
and check for the livepatch flag.


> For the test cases which load multiple livepatches, is there an easy way
> to determine the patch stack order from userspace?  I think that would
> be helpful when trying to remove all of them.

I am not aware about any easy way. Only the following hacks come
to my mind:

One possibility would be to use the creation time of the directories
under /sys/kernel/livepatch.

Or I wonder if the output from lsmod is sorted by the order in which
the modules were loaded.


> >> +echo -n "TEST1 ... "
> >> +dmesg -C
> >> +
> >> +load_mod $MOD_TARGET
> >> +load_mod $MOD_LIVEPATCH
> >> +wait_for_transition $MOD_LIVEPATCH
> >> +disable_lp $MOD_LIVEPATCH
> >> +unload_mod $MOD_LIVEPATCH
> >> +unload_mod $MOD_TARGET
> >> +
> >> +check_result "% modprobe $MOD_TARGET
> >> +$MOD_TARGET: livepatch_callbacks_mod_init
> >> +% modprobe $MOD_LIVEPATCH
> >> +livepatch: enabling patch '$MOD_LIVEPATCH'
> >> +livepatch: '$MOD_LIVEPATCH': initializing patching transition
> >> +$MOD_LIVEPATCH: pre_patch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
> >> +$MOD_LIVEPATCH: pre_patch_callback: vmlinux
> >> +livepatch: '$MOD_LIVEPATCH': starting patching transition
> >> +livepatch: '$MOD_LIVEPATCH': completing patching transition
> >> +$MOD_LIVEPATCH: post_patch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
> >> +$MOD_LIVEPATCH: post_patch_callback: vmlinux
> >> +livepatch: '$MOD_LIVEPATCH': patching complete
> >> +% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled
> >> +livepatch: '$MOD_LIVEPATCH': initializing unpatching transition
> >> +$MOD_LIVEPATCH: pre_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
> >> +$MOD_LIVEPATCH: pre_unpatch_callback: vmlinux
> >> +livepatch: '$MOD_LIVEPATCH': starting unpatching transition
> >> +livepatch: '$MOD_LIVEPATCH': completing unpatching transition
> >> +$MOD_LIVEPATCH: post_unpatch_callback: $MOD_TARGET -> [MODULE_STATE_LIVE] Normal state
> >> +$MOD_LIVEPATCH: post_unpatch_callback: vmlinux
> >> +livepatch: '$MOD_LIVEPATCH': unpatching complete
> >> +% rmmod $MOD_LIVEPATCH
> >> +% rmmod $MOD_TARGET
> >> +$MOD_TARGET: livepatch_callbacks_mod_exit"
> > 
> > I was a bit surprised when seeing this way of checking results.
> > But on the other hand, it looks pretty effective, especially for
> > the callbacks. And the 3rd look, any patched function might write
> > something into the log when called.
> 
> This was a quickly scripted version of what I was manually verifying
> with the sample example livepatches.  I don't know if it will scale, but
> it was pretty easy to add tests this way.
> 
> I wonder though if better dmesg filters will be required as the
> livepatch core adds more debug msgs?

Let's see. Most of the messages seem to be from the test modules
itself, so it should not be that bad.


> > I like it. Let's see how it works in the long term. But I am rather
> > positive.
> > 
> > Thanks a lot for working on it.
> 
> Thanks for taking a look and running the tests.  I'll make some of your
> suggested changes and send it up for a proper review soon.

Great.

Best Regards,
Petr

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Josh Poimboeuf]

On Mon, Mar 26, 2018 at 12:11:07PM +0200, Petr Mladek wrote:
> On Fri 2018-03-23 17:44:10, Josh Poimboeuf wrote:
> > On Fri, Mar 23, 2018 at 10:45:07AM +0100, Petr Mladek wrote:
> > > On Tue 2018-03-20 15:15:02, Josh Poimboeuf wrote:
> > > > On Tue, Mar 20, 2018 at 01:25:38PM +0100, Petr Mladek wrote:
> > > > > On Mon 2018-03-19 16:43:24, Josh Poimboeuf wrote:
> > > > > > On Mon, Mar 19, 2018 at 04:02:07PM +0100, Petr Mladek wrote:
> > > > > > > > Along those lines, I'd also propose that we constrain our existing patch
> > > > > > > > stacking even further.  Right now we allow a new patch to be registered
> > > > > > > > on top of a disabled patch, though we don't allow the new patch to be
> > > > > > > > enabled until the previous patch gets enabled.  I'd propose we no longer
> > > > > > > > allow that condition.  We should instead enforce that all existing
> > > > > > > > patches are *enabled* before allowing a new patch to be registered on
> > > > > > > > top.  That way the patch stacking is even more sane, and there are less
> > > > > > > > "unusual" conditions to worry about.  We have enough of those already.
> > > > > > > > Each additional bit of flexibility has a maintenance cost, and this one
> > > > > > > > isn't worth it IMO.
> > > > > > > 
> > > > > > > Again, this might make some things easier but it might also bring
> > > > > > > problems.
> > > > > > > 
> > > > > > > For example, we would need to solve the situation when the last
> > > > > > > patch is disabled and cannot be removed because the transition
> > > > > > > was forced. This might be more common after removing the immediate
> > > > > > > feature.
> > > > > > 
> > > > > > I would stop worrying about forced patches so much :-)
> > > > > 
> > > > > I have already seen blocked transition several times. It is true that
> > > > > it was with kGraft. But we just do not have enough real life experience
> > > > > with the upstream livepatch code.
> > > > 
> > > > But we're talking about patching on top of a *disabled* patch.  Forced
> > > > or not, why would the patch be disabled in the first place?
> > > 
> > > For example, it might be disabled because the transition stalled for
> > > too long and the user reverted it. Or just because it is possible
> > > to disable it.
> > 
> > If they haven't previously forced any patches, and they reverted the
> > topmost patch because it stalled, they can easily unload the patch.
> > 
> > If they *have* previously forced a patch, they can force enable the
> > topmost patch as well, or if that's not safe they can reboot (that's
> > what you get for forcing a patch...)
> 
> IMHO, the reboot is the very last option for people that are using
> livepatching.

... but it may be a natural consequence of forcing a patch.

> > > > We were just recently discussing the possibility of not allowing the
> > > > disabling of patches at all.  If we're not going that far, let's at
> > > > least further restrict it, for the sanity of our code, so we don't have
> > > > to worry about a nasty mismatched stack of enabled/disabled/enabled/etc,
> > > > at least for the cases where 'replace' patches aren't used.
> > > 
> > > I am not completely sure where all these fears come from. From my
> > > point of view, this change is pretty safe and trivial thanks to NOPs
> > > and overall design. It would be a shame to do not have it. But I
> > > might be blind after spending so much time with the feature.
> > 
> > I think you're missing my point.  This isn't about your patch set, per
> > se.  It's really about our existing code.  Today, our patch stack
> > doesn't follow real stack semantics, because patches in the middle might
> > be disabled.  I see that as a problem.
> 
> This would be true if we keep the replaced patches on the stack. But
> if we remove the replaced patches then there never will be disabled
> patches in the middle.
> 
> OK, there might be disabled patches in the middle during the
> transition. But this is the situation where we basically could
> not manipulate the stack.

No, please read it again.  I wasn't talking about replaced patches.

> > If 'replace' were the only mode, then we wouldn't even need a patch
> > stack because it wouldn't really matter much whether the previous patch
> > is enabled or disabled.  I think this is in agreement with the point
> > you're making.
> > 
> > But we still support non-replace patches.  My feeling is that we should
> > either do a true stack, or no stack at all.  The in-between thing is
> > going to be confusing, not only for us, but for patch authors and end
> > users.
> 
> I see it like two different modes. We either have a stack of patches
> that depend on each other.

But if they depend on each other, they can use 'replace' and a stack
isn't needed.

And If they *don't* depend on each other, then the stack is overly
restrictive, for no good reason.

Either way, why do we still need a stack?

> Or we have replace patches that are
> standalone and we allow a smooth transfer from one to another one.
> 
> Anyway, for us, it is much more important the removal of replaced
> patches. We could probably live without the possibility to replace
> disabled patches.

I think replacing disabled patches is ok, *if* we get rid of the
illusion of a stack.  The current stack isn't really a stack, it's just
awkward.

-- 
Josh

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Petr Mladek]

On Fri 2018-04-06 14:50:49, Josh Poimboeuf wrote:
> On Mon, Mar 26, 2018 at 12:11:07PM +0200, Petr Mladek wrote:
> > On Fri 2018-03-23 17:44:10, Josh Poimboeuf wrote:
> > > On Fri, Mar 23, 2018 at 10:45:07AM +0100, Petr Mladek wrote:
> > > > On Tue 2018-03-20 15:15:02, Josh Poimboeuf wrote:
> > > > > On Tue, Mar 20, 2018 at 01:25:38PM +0100, Petr Mladek wrote:
> > > > > > On Mon 2018-03-19 16:43:24, Josh Poimboeuf wrote:
> > > > > > > On Mon, Mar 19, 2018 at 04:02:07PM +0100, Petr Mladek wrote:
> > > > > > > > > Along those lines, I'd also propose that we constrain our existing patch
> > > > > > > > > stacking even further.  Right now we allow a new patch to be registered
> > > > > > > > > on top of a disabled patch, though we don't allow the new patch to be
> > > > > > > > > enabled until the previous patch gets enabled.  I'd propose we no longer
> > > > > > > > > allow that condition.  We should instead enforce that all existing
> > > > > > > > > patches are *enabled* before allowing a new patch to be registered on
> > > > > > > > > top.  That way the patch stacking is even more sane, and there are less
> > > > > > > > > "unusual" conditions to worry about.  We have enough of those already.
> > > > > > > > > Each additional bit of flexibility has a maintenance cost, and this one
> > > > > > > > > isn't worth it IMO.
> > > > > > > > 
> > > > > > > > Again, this might make some things easier but it might also bring
> > > > > > > > problems.
> > > > > > > > 
> > > > > > > > For example, we would need to solve the situation when the last
> > > > > > > > patch is disabled and cannot be removed because the transition
> > > > > > > > was forced. This might be more common after removing the immediate
> > > > > > > > feature.
> > > > > > > 
> > > > > > > I would stop worrying about forced patches so much :-)
> > > > > > 
> > > > > > I have already seen blocked transition several times. It is true that
> > > > > > it was with kGraft. But we just do not have enough real life experience
> > > > > > with the upstream livepatch code.
> > > > > 
> > > > > But we're talking about patching on top of a *disabled* patch.  Forced
> > > > > or not, why would the patch be disabled in the first place?
> > > > 
> > > > For example, it might be disabled because the transition stalled for
> > > > too long and the user reverted it. Or just because it is possible
> > > > to disable it.
> > > 
> > > If they haven't previously forced any patches, and they reverted the
> > > topmost patch because it stalled, they can easily unload the patch.
> > > 
> > > If they *have* previously forced a patch, they can force enable the
> > > topmost patch as well, or if that's not safe they can reboot (that's
> > > what you get for forcing a patch...)
> > 
> > IMHO, the reboot is the very last option for people that are using
> > livepatching.
> 
> ... but it may be a natural consequence of forcing a patch.
> 
> > > > > We were just recently discussing the possibility of not allowing the
> > > > > disabling of patches at all.  If we're not going that far, let's at
> > > > > least further restrict it, for the sanity of our code, so we don't have
> > > > > to worry about a nasty mismatched stack of enabled/disabled/enabled/etc,
> > > > > at least for the cases where 'replace' patches aren't used.
> > > > 
> > > > I am not completely sure where all these fears come from. From my
> > > > point of view, this change is pretty safe and trivial thanks to NOPs
> > > > and overall design. It would be a shame to do not have it. But I
> > > > might be blind after spending so much time with the feature.
> > > 
> > > I think you're missing my point.  This isn't about your patch set, per
> > > se.  It's really about our existing code.  Today, our patch stack
> > > doesn't follow real stack semantics, because patches in the middle might
> > > be disabled.  I see that as a problem.
> 
> No, please read it again.  I wasn't talking about replaced patches.

I was confused by wording "in the middle". It suggested that there
might had been enabled patches on the top and the bottom of the stack
and some disabled patches in between at the same time (or vice versa).
This was not true.

Do I understand it correctly that you do not like that the patches
on the stack might be in two states (disabled/enabled). This might
be translated that you do not like the state when the patch is
registered and disabled.

I wonder if the problem is in the "stack" abstraction. Would it help
if we call it "sorted list" or whatever more suitable?

Another possibility would be to get rid of the enable/disable states.
I mean that the patch will be automatically enabled during
registration and removed during unregistration. Or we could rename
the operation do add/remove or anything else. In fact, this is how
it worked in kGraft.

AFAIK, the enable/disabled state made more sense for immediate
patches that could not be unloaded. We still need to keep the patches
when the transaction is forced. The question is if we need to keep
the sysfs entries for loaded but unused patches.


> > > If 'replace' were the only mode, then we wouldn't even need a patch
> > > stack because it wouldn't really matter much whether the previous patch
> > > is enabled or disabled.  I think this is in agreement with the point
> > > you're making.
> > > 
> > > But we still support non-replace patches.  My feeling is that we should
> > > either do a true stack, or no stack at all.  The in-between thing is
> > > going to be confusing, not only for us, but for patch authors and end
> > > users.
> > 
> > I see it like two different modes. We either have a stack of patches
> > that depend on each other.
> 
> But if they depend on each other, they can use 'replace' and a stack
> isn't needed.

Yes but see below.


> And If they *don't* depend on each other, then the stack is overly
> restrictive, for no good reason.
> 
> Either way, why do we still need a stack?

Good question. I suggest to agree on some terms first:

   + Independent patches make unrelated changes. Any new function
     must not rely on changes done by any other patch.

   + Dependent patches mean that a later patch depend on changes
     done by an earlier patch. For example, a new function might
     use function added by an earlier patch.

   + Each cumulative patch include all necessary changes. I would say
     that it is self-containing and independent. Except that they should
     be able to continue using changes made by earlier patches (shadow
     variables, callbacks).


Then we could say:

   + The stack helps to enforce dependencies between dependent
     patches. But there is needed also some external solution
     that forces loading the patches in the right order.

   + The "replace" feature is useful for cumulative patches.
     It allows to remove existing changes and remove unused stuff.
     But cumulative patches might be done even _without_ the atomic
     replace.

   + Cumulative patches _with_ "replace" flag might be in theory
     installed in random order because they handle not-longer patched
     functions. Well, some incompatibility might be caused by shadow
     variables and callbacks. Therefore it still might make sense
     to install them in the right order.

     Cumulative patches _without_ "replace" flag must be installed
     in the right order because they do not handle not-longer patched
     functions.

     Anyway, for cumulative patches is important the external
     ordering (loading modules) and not the stack.


Back to your question:

The stack is needed for dependent non-cumulative patches.

The cumulative patches with "replace" flag seems the be
the most safe and most flexible solution. The question is
if we want to force all users to use this mode.


> > Or we have replace patches that are
> > standalone and we allow a smooth transfer from one to another one.
> > 
> > Anyway, for us, it is much more important the removal of replaced
> > patches. We could probably live without the possibility to replace
> > disabled patches.
> 
> I think replacing disabled patches is ok, *if* we get rid of the
> illusion of a stack.  The current stack isn't really a stack, it's just
> awkward.

I personally do not have problems with it. As I said, I see this as
two different modes how the life patches are distributed. The stack
is needed for dependent patches. The cumulative patches with
"replace" flag are self-contained and independent. They might
replace anything.

Well, it would make sense to reduce the amount of possible
situations and use cases. The question is what is acceptable
to others and if it needs to be done as part of this patch set.

Best Regards,
Petr

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Miroslav Benes]

> > > > I think you're missing my point.  This isn't about your patch set, per
> > > > se.  It's really about our existing code.  Today, our patch stack
> > > > doesn't follow real stack semantics, because patches in the middle might
> > > > be disabled.  I see that as a problem.
> > 
> > No, please read it again.  I wasn't talking about replaced patches.
> 
> I was confused by wording "in the middle". It suggested that there
> might had been enabled patches on the top and the bottom of the stack
> and some disabled patches in between at the same time (or vice versa).
> This was not true.
> 
> Do I understand it correctly that you do not like that the patches
> on the stack might be in two states (disabled/enabled). This might
> be translated that you do not like the state when the patch is
> registered and disabled.
> 
> I wonder if the problem is in the "stack" abstraction. Would it help
> if we call it "sorted list" or whatever more suitable?
> 
> Another possibility would be to get rid of the enable/disable states.
> I mean that the patch will be automatically enabled during
> registration and removed during unregistration. Or we could rename
> the operation do add/remove or anything else. In fact, this is how
> it worked in kGraft.

I've already wondered couple of times why we had separate enable/disable. 
If there is someone who knows, remind me, please. I wouldn't be against a 
simplification here.

On the other hand, it is kind of nice to keep the registration and 
enablement separate. It is more flexible if someone needs it.

Anyway, we should solve it together with the stacking. It is tightly 
connected.
 
> AFAIK, the enable/disabled state made more sense for immediate
> patches that could not be unloaded. We still need to keep the patches
> when the transaction is forced. The question is if we need to keep
> the sysfs entries for loaded but unused patches.
> 
> 
> > > > If 'replace' were the only mode, then we wouldn't even need a patch
> > > > stack because it wouldn't really matter much whether the previous patch
> > > > is enabled or disabled.  I think this is in agreement with the point
> > > > you're making.
> > > > 
> > > > But we still support non-replace patches.  My feeling is that we should
> > > > either do a true stack, or no stack at all.  The in-between thing is
> > > > going to be confusing, not only for us, but for patch authors and end
> > > > users.
> > > 
> > > I see it like two different modes. We either have a stack of patches
> > > that depend on each other.
> > 
> > But if they depend on each other, they can use 'replace' and a stack
> > isn't needed.
> 
> Yes but see below.
> 
> 
> > And If they *don't* depend on each other, then the stack is overly
> > restrictive, for no good reason.
> > 
> > Either way, why do we still need a stack?
> 
> Good question. I suggest to agree on some terms first:
> 
>    + Independent patches make unrelated changes. Any new function
>      must not rely on changes done by any other patch.
> 
>    + Dependent patches mean that a later patch depend on changes
>      done by an earlier patch. For example, a new function might
>      use function added by an earlier patch.
> 
>    + Each cumulative patch include all necessary changes. I would say
>      that it is self-containing and independent. Except that they should
>      be able to continue using changes made by earlier patches (shadow
>      variables, callbacks).
> 
> 
> Then we could say:
> 
>    + The stack helps to enforce dependencies between dependent
>      patches. But there is needed also some external solution
>      that forces loading the patches in the right order.
> 
>    + The "replace" feature is useful for cumulative patches.
>      It allows to remove existing changes and remove unused stuff.
>      But cumulative patches might be done even _without_ the atomic
>      replace.
> 
>    + Cumulative patches _with_ "replace" flag might be in theory
>      installed in random order because they handle not-longer patched
>      functions. Well, some incompatibility might be caused by shadow
>      variables and callbacks. Therefore it still might make sense
>      to install them in the right order.
> 
>      Cumulative patches _without_ "replace" flag must be installed
>      in the right order because they do not handle not-longer patched
>      functions.
> 
>      Anyway, for cumulative patches is important the external
>      ordering (loading modules) and not the stack.
> 
> 
> Back to your question:
> 
> The stack is needed for dependent non-cumulative patches.
> 
> The cumulative patches with "replace" flag seems the be
> the most safe and most flexible solution. The question is
> if we want to force all users to use this mode.
> 
> 
> > > Or we have replace patches that are
> > > standalone and we allow a smooth transfer from one to another one.
> > > 
> > > Anyway, for us, it is much more important the removal of replaced
> > > patches. We could probably live without the possibility to replace
> > > disabled patches.
> > 
> > I think replacing disabled patches is ok, *if* we get rid of the
> > illusion of a stack.  The current stack isn't really a stack, it's just
> > awkward.
> 
> I personally do not have problems with it. As I said, I see this as
> two different modes how the life patches are distributed. The stack
> is needed for dependent patches. The cumulative patches with
> "replace" flag are self-contained and independent. They might
> replace anything.

I agree here. Practically we use only cumulative replacement patches at 
SUSE. So with that in mind I don't care about the stacking much. But, it 
may make sense for someone else. Evgenii mentioned they used it for 
hotfixes. Therefore I'm reluctant to remove it completely.
 
> Well, it would make sense to reduce the amount of possible
> situations and use cases. The question is what is acceptable
> to others and if it needs to be done as part of this patch set.

Yes. Input from actual users would be tremendously useful here.

Miroslav

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Evgenii Shatokhin]

On 10.04.2018 16:21, Miroslav Benes wrote:
> 
>>>>> I think you're missing my point.  This isn't about your patch set, per
>>>>> se.  It's really about our existing code.  Today, our patch stack
>>>>> doesn't follow real stack semantics, because patches in the middle might
>>>>> be disabled.  I see that as a problem.
>>>
>>> No, please read it again.  I wasn't talking about replaced patches.
>>
>> I was confused by wording "in the middle". It suggested that there
>> might had been enabled patches on the top and the bottom of the stack
>> and some disabled patches in between at the same time (or vice versa).
>> This was not true.
>>
>> Do I understand it correctly that you do not like that the patches
>> on the stack might be in two states (disabled/enabled). This might
>> be translated that you do not like the state when the patch is
>> registered and disabled.
>>
>> I wonder if the problem is in the "stack" abstraction. Would it help
>> if we call it "sorted list" or whatever more suitable?
>>
>> Another possibility would be to get rid of the enable/disable states.
>> I mean that the patch will be automatically enabled during
>> registration and removed during unregistration. Or we could rename
>> the operation do add/remove or anything else. In fact, this is how
>> it worked in kGraft.
> 
> I've already wondered couple of times why we had separate enable/disable.
> If there is someone who knows, remind me, please. I wouldn't be against a
> simplification here.
> 
> On the other hand, it is kind of nice to keep the registration and
> enablement separate. It is more flexible if someone needs it.
> 
> Anyway, we should solve it together with the stacking. It is tightly
> connected.
>   
>> AFAIK, the enable/disabled state made more sense for immediate
>> patches that could not be unloaded. We still need to keep the patches
>> when the transaction is forced. The question is if we need to keep
>> the sysfs entries for loaded but unused patches.
>>
>>
>>>>> If 'replace' were the only mode, then we wouldn't even need a patch
>>>>> stack because it wouldn't really matter much whether the previous patch
>>>>> is enabled or disabled.  I think this is in agreement with the point
>>>>> you're making.
>>>>>
>>>>> But we still support non-replace patches.  My feeling is that we should
>>>>> either do a true stack, or no stack at all.  The in-between thing is
>>>>> going to be confusing, not only for us, but for patch authors and end
>>>>> users.
>>>>
>>>> I see it like two different modes. We either have a stack of patches
>>>> that depend on each other.
>>>
>>> But if they depend on each other, they can use 'replace' and a stack
>>> isn't needed.
>>
>> Yes but see below.
>>
>>
>>> And If they *don't* depend on each other, then the stack is overly
>>> restrictive, for no good reason.
>>>
>>> Either way, why do we still need a stack?
>>
>> Good question. I suggest to agree on some terms first:
>>
>>     + Independent patches make unrelated changes. Any new function
>>       must not rely on changes done by any other patch.
>>
>>     + Dependent patches mean that a later patch depend on changes
>>       done by an earlier patch. For example, a new function might
>>       use function added by an earlier patch.
>>
>>     + Each cumulative patch include all necessary changes. I would say
>>       that it is self-containing and independent. Except that they should
>>       be able to continue using changes made by earlier patches (shadow
>>       variables, callbacks).
>>
>>
>> Then we could say:
>>
>>     + The stack helps to enforce dependencies between dependent
>>       patches. But there is needed also some external solution
>>       that forces loading the patches in the right order.
>>
>>     + The "replace" feature is useful for cumulative patches.
>>       It allows to remove existing changes and remove unused stuff.
>>       But cumulative patches might be done even _without_ the atomic
>>       replace.
>>
>>     + Cumulative patches _with_ "replace" flag might be in theory
>>       installed in random order because they handle not-longer patched
>>       functions. Well, some incompatibility might be caused by shadow
>>       variables and callbacks. Therefore it still might make sense
>>       to install them in the right order.
>>
>>       Cumulative patches _without_ "replace" flag must be installed
>>       in the right order because they do not handle not-longer patched
>>       functions.
>>
>>       Anyway, for cumulative patches is important the external
>>       ordering (loading modules) and not the stack.
>>
>>
>> Back to your question:
>>
>> The stack is needed for dependent non-cumulative patches.
>>
>> The cumulative patches with "replace" flag seems the be
>> the most safe and most flexible solution. The question is
>> if we want to force all users to use this mode.
>>
>>
>>>> Or we have replace patches that are
>>>> standalone and we allow a smooth transfer from one to another one.
>>>>
>>>> Anyway, for us, it is much more important the removal of replaced
>>>> patches. We could probably live without the possibility to replace
>>>> disabled patches.
>>>
>>> I think replacing disabled patches is ok, *if* we get rid of the
>>> illusion of a stack.  The current stack isn't really a stack, it's just
>>> awkward.
>>
>> I personally do not have problems with it. As I said, I see this as
>> two different modes how the life patches are distributed. The stack
>> is needed for dependent patches. The cumulative patches with
>> "replace" flag are self-contained and independent. They might
>> replace anything.
> 
> I agree here. Practically we use only cumulative replacement patches at
> SUSE. So with that in mind I don't care about the stacking much. But, it
> may make sense for someone else. Evgenii mentioned they used it for
> hotfixes. Therefore I'm reluctant to remove it completely.

Well, it was convenient in some cases to provide a hot fix for a given 
bug on top of our official cumulative patch. So far, such fixes were 
only used on a few of the customers' machines (where they were needed 
ASAP). It just made it easier to see where is the common set of fixes 
and where is the customer-specific addition.

I think, we can use cumulative patches in such cases too without much 
additional effort. For example, we can encode the distinction (base set 
of fixes + addition) in the module name or somewhere else.

So, I think, it is fine for us, if stacking support is removed. 
Especially if that makes the implementation of livepatch less complex 
and more reliable.

>   
>> Well, it would make sense to reduce the amount of possible
>> situations and use cases. The question is what is acceptable
>> to others and if it needs to be done as part of this patch set.
> 
> Yes. Input from actual users would be tremendously useful here.
> 
> Miroslav
> .
> 

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Josh Poimboeuf]

On Tue, Apr 10, 2018 at 10:34:55AM +0200, Petr Mladek wrote:
> > > > > > We were just recently discussing the possibility of not allowing the
> > > > > > disabling of patches at all.  If we're not going that far, let's at
> > > > > > least further restrict it, for the sanity of our code, so we don't have
> > > > > > to worry about a nasty mismatched stack of enabled/disabled/enabled/etc,
> > > > > > at least for the cases where 'replace' patches aren't used.
> > > > > 
> > > > > I am not completely sure where all these fears come from. From my
> > > > > point of view, this change is pretty safe and trivial thanks to NOPs
> > > > > and overall design. It would be a shame to do not have it. But I
> > > > > might be blind after spending so much time with the feature.
> > > > 
> > > > I think you're missing my point.  This isn't about your patch set, per
> > > > se.  It's really about our existing code.  Today, our patch stack
> > > > doesn't follow real stack semantics, because patches in the middle might
> > > > be disabled.  I see that as a problem.
> > 
> > No, please read it again.  I wasn't talking about replaced patches.
> 
> I was confused by wording "in the middle". It suggested that there
> might had been enabled patches on the top and the bottom of the stack
> and some disabled patches in between at the same time (or vice versa).
> This was not true.

That *was* what I meant.  Consider the following sequence of events:

- Register patch 1
- Enable patch 1
- Register patch 2
- Enable patch 2
- Disable patch 2
- Register patch 3
- Enable patch 3

Notice that patch 2 (in the middle) is disabled, whereas patch 1 (on the
bottom) and patch 3 (on the top) are enabled.

> Do I understand it correctly that you do not like that the patches
> on the stack might be in two states (disabled/enabled). This might
> be translated that you do not like the state when the patch is
> registered and disabled.

No, that wasn't really what I meant, but I have often wondered whether
we need such a distinction.

> I wonder if the problem is in the "stack" abstraction. Would it help
> if we call it "sorted list" or whatever more suitable?

But I don't even see a reason to have a sorted list (more on that
below).

> Another possibility would be to get rid of the enable/disable states.
> I mean that the patch will be automatically enabled during
> registration and removed during unregistration.

I don't see how disabling during unregistration would be possible, since
the unregister is called from the patch module exit function, which
can only be called *after* the patch is disabled.

However, we could unregister immediately after disabling (i.e., in
enabled_store context).

> Or we could rename the operation do add/remove or anything else. In
> fact, this is how it worked in kGraft.

I'm not sure what renaming would solve, unless you mean to combine
registration and enablement into a single concept?  Sounds good to me.

> AFAIK, the enable/disabled state made more sense for immediate
> patches that could not be unloaded. We still need to keep the patches
> when the transaction is forced. The question is if we need to keep
> the sysfs entries for loaded but unused patches.

I think we wouldn't need the sysfs entries.  Just disable/unregister
forced patches like normal, except skipping the module_put().

> > Either way, why do we still need a stack?
> 
> Good question. I suggest to agree on some terms first:
> 
>    + Independent patches make unrelated changes. Any new function
>      must not rely on changes done by any other patch.
> 
>    + Dependent patches mean that a later patch depend on changes
>      done by an earlier patch. For example, a new function might
>      use function added by an earlier patch.
> 
>    + Each cumulative patch include all necessary changes. I would say
>      that it is self-containing and independent. Except that they should
>      be able to continue using changes made by earlier patches (shadow
>      variables, callbacks).
> 
> 
> Then we could say:
> 
>    + The stack helps to enforce dependencies between dependent
>      patches. But there is needed also some external solution
>      that forces loading the patches in the right order.
> 
>    + The "replace" feature is useful for cumulative patches.
>      It allows to remove existing changes and remove unused stuff.
>      But cumulative patches might be done even _without_ the atomic
>      replace.
> 
>    + Cumulative patches _with_ "replace" flag might be in theory
>      installed in random order because they handle not-longer patched
>      functions. Well, some incompatibility might be caused by shadow
>      variables and callbacks. Therefore it still might make sense
>      to install them in the right order.
> 
>      Cumulative patches _without_ "replace" flag must be installed
>      in the right order because they do not handle not-longer patched
>      functions.
> 
>      Anyway, for cumulative patches is important the external
>      ordering (loading modules) and not the stack.
> 
> 
> Back to your question:
> 
> The stack is needed for dependent non-cumulative patches.
> 
> The cumulative patches with "replace" flag seems the be
> the most safe and most flexible solution. The question is
> if we want to force all users to use this mode.

If they have dependencies between modules, they can either a) enforce it
with tooling, or they can instead b) use 'replace'.

But let's get the module load order enforcement out of the kernel.
There's no real need for the kernel to do it, and we're not even doing a
good job at it.

> > > Or we have replace patches that are
> > > standalone and we allow a smooth transfer from one to another one.
> > > 
> > > Anyway, for us, it is much more important the removal of replaced
> > > patches. We could probably live without the possibility to replace
> > > disabled patches.
> > 
> > I think replacing disabled patches is ok, *if* we get rid of the
> > illusion of a stack.  The current stack isn't really a stack, it's just
> > awkward.
> 
> I personally do not have problems with it. As I said, I see this as
> two different modes how the life patches are distributed. The stack
> is needed for dependent patches. The cumulative patches with
> "replace" flag are self-contained and independent. They might
> replace anything.
> 
> Well, it would make sense to reduce the amount of possible
> situations and use cases.

A big +1.

> The question is what is acceptable to others

If there are any objections, this is their chance to speak up :-)

> and if it needs to be done as part of this patch set.

Maybe so, for at least a few reasons:

- This patch set makes the 'stack' obsolete, so it makes sense to remove
  the 'stack' with it.

- This patch set will already affect tooling, let's make tooling's life
  easier by making all the related changes at the same time.
  
  (Though I'm not quite convinced on this point, would removing the
  stack affect tooling at all?  If we also combined registration and
  enablement into a single concept, then it definitely would.)

-- 
Josh

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Josh Poimboeuf]

> > I agree here. Practically we use only cumulative replacement patches at
> > SUSE. So with that in mind I don't care about the stacking much. But, it
> > may make sense for someone else. Evgenii mentioned they used it for
> > hotfixes. Therefore I'm reluctant to remove it completely.
> 
> Well, it was convenient in some cases to provide a hot fix for a given bug
> on top of our official cumulative patch. So far, such fixes were only used
> on a few of the customers' machines (where they were needed ASAP). It just
> made it easier to see where is the common set of fixes and where is the
> customer-specific addition.
> 
> I think, we can use cumulative patches in such cases too without much
> additional effort. For example, we can encode the distinction (base set of
> fixes + addition) in the module name or somewhere else.
> 
> So, I think, it is fine for us, if stacking support is removed. Especially
> if that makes the implementation of livepatch less complex and more
> reliable.

Just to clarify, I think we are just proposing the removal of the
enforcement of the stacking order.  We will still allow multiple
non-replace patches to be applied.  We just won't enforce which patches
can be disabled/removed at any given time.

So I think your old way of doing things (individual unrelated patches on
top of a cumulative patch) would still work.

-- 
Josh

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Miroslav Benes]

On Tue, 10 Apr 2018, Josh Poimboeuf wrote:

> > > I agree here. Practically we use only cumulative replacement patches at
> > > SUSE. So with that in mind I don't care about the stacking much. But, it
> > > may make sense for someone else. Evgenii mentioned they used it for
> > > hotfixes. Therefore I'm reluctant to remove it completely.
> > 
> > Well, it was convenient in some cases to provide a hot fix for a given bug
> > on top of our official cumulative patch. So far, such fixes were only used
> > on a few of the customers' machines (where they were needed ASAP). It just
> > made it easier to see where is the common set of fixes and where is the
> > customer-specific addition.
> > 
> > I think, we can use cumulative patches in such cases too without much
> > additional effort. For example, we can encode the distinction (base set of
> > fixes + addition) in the module name or somewhere else.
> > 
> > So, I think, it is fine for us, if stacking support is removed. Especially
> > if that makes the implementation of livepatch less complex and more
> > reliable.
> 
> Just to clarify, I think we are just proposing the removal of the
> enforcement of the stacking order.  We will still allow multiple
> non-replace patches to be applied.  We just won't enforce which patches
> can be disabled/removed at any given time.

Heh, so I misunderstood. I thought you were talking about the removal of 
the stacking. Now it makes more sense.
 
> So I think your old way of doing things (individual unrelated patches on
> top of a cumulative patch) would still work.

Yes. On the other hand the user needs to be even more careful, so I'd 
expect an update of documentation with the removal :).

Miroslav

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Miroslav Benes]

On Tue, 10 Apr 2018, Josh Poimboeuf wrote:

> On Tue, Apr 10, 2018 at 10:34:55AM +0200, Petr Mladek wrote:
> > > > > > > We were just recently discussing the possibility of not allowing the
> > > > > > > disabling of patches at all.  If we're not going that far, let's at
> > > > > > > least further restrict it, for the sanity of our code, so we don't have
> > > > > > > to worry about a nasty mismatched stack of enabled/disabled/enabled/etc,
> > > > > > > at least for the cases where 'replace' patches aren't used.
> > > > > > 
> > > > > > I am not completely sure where all these fears come from. From my
> > > > > > point of view, this change is pretty safe and trivial thanks to NOPs
> > > > > > and overall design. It would be a shame to do not have it. But I
> > > > > > might be blind after spending so much time with the feature.
> > > > > 
> > > > > I think you're missing my point.  This isn't about your patch set, per
> > > > > se.  It's really about our existing code.  Today, our patch stack
> > > > > doesn't follow real stack semantics, because patches in the middle might
> > > > > be disabled.  I see that as a problem.
> > > 
> > > No, please read it again.  I wasn't talking about replaced patches.
> > 
> > I was confused by wording "in the middle". It suggested that there
> > might had been enabled patches on the top and the bottom of the stack
> > and some disabled patches in between at the same time (or vice versa).
> > This was not true.
> 
> That *was* what I meant.  Consider the following sequence of events:
> 
> - Register patch 1
> - Enable patch 1
> - Register patch 2
> - Enable patch 2
> - Disable patch 2
> - Register patch 3
> - Enable patch 3
> 
> Notice that patch 2 (in the middle) is disabled, whereas patch 1 (on the
> bottom) and patch 3 (on the top) are enabled.

This should not be possible at all.

__klp_enable_patch:

        if (patch->list.prev != &klp_patches &&
            !list_prev_entry(patch, list)->enabled)
                return -EBUSY;

When patch 3 is enabled, list_prev_entry() returns patch 2 and its 
->enabled is false.
 
> > Do I understand it correctly that you do not like that the patches
> > on the stack might be in two states (disabled/enabled). This might
> > be translated that you do not like the state when the patch is
> > registered and disabled.
> 
> No, that wasn't really what I meant, but I have often wondered whether
> we need such a distinction.
> 
> > I wonder if the problem is in the "stack" abstraction. Would it help
> > if we call it "sorted list" or whatever more suitable?
> 
> But I don't even see a reason to have a sorted list (more on that
> below).
> 
> > Another possibility would be to get rid of the enable/disable states.
> > I mean that the patch will be automatically enabled during
> > registration and removed during unregistration.
> 
> I don't see how disabling during unregistration would be possible, since
> the unregister is called from the patch module exit function, which
> can only be called *after* the patch is disabled.
> 
> However, we could unregister immediately after disabling (i.e., in
> enabled_store context).

I think this is what Petr meant. So there would be nothing in the patch 
module exit function. Well, not exactly. We'd need to remove sysfs dir and 
maybe something more.

> > Or we could rename the operation do add/remove or anything else. In
> > fact, this is how it worked in kGraft.
> 
> I'm not sure what renaming would solve, unless you mean to combine
> registration and enablement into a single concept?  Sounds good to me.
> 
> > AFAIK, the enable/disabled state made more sense for immediate
> > patches that could not be unloaded. We still need to keep the patches
> > when the transaction is forced. The question is if we need to keep
> > the sysfs entries for loaded but unused patches.
> 
> I think we wouldn't need the sysfs entries.  Just disable/unregister
> forced patches like normal, except skipping the module_put().
> 
> > > Either way, why do we still need a stack?
> > 
> > Good question. I suggest to agree on some terms first:
> > 
> >    + Independent patches make unrelated changes. Any new function
> >      must not rely on changes done by any other patch.
> > 
> >    + Dependent patches mean that a later patch depend on changes
> >      done by an earlier patch. For example, a new function might
> >      use function added by an earlier patch.
> > 
> >    + Each cumulative patch include all necessary changes. I would say
> >      that it is self-containing and independent. Except that they should
> >      be able to continue using changes made by earlier patches (shadow
> >      variables, callbacks).
> > 
> > 
> > Then we could say:
> > 
> >    + The stack helps to enforce dependencies between dependent
> >      patches. But there is needed also some external solution
> >      that forces loading the patches in the right order.
> > 
> >    + The "replace" feature is useful for cumulative patches.
> >      It allows to remove existing changes and remove unused stuff.
> >      But cumulative patches might be done even _without_ the atomic
> >      replace.
> > 
> >    + Cumulative patches _with_ "replace" flag might be in theory
> >      installed in random order because they handle not-longer patched
> >      functions. Well, some incompatibility might be caused by shadow
> >      variables and callbacks. Therefore it still might make sense
> >      to install them in the right order.
> > 
> >      Cumulative patches _without_ "replace" flag must be installed
> >      in the right order because they do not handle not-longer patched
> >      functions.
> > 
> >      Anyway, for cumulative patches is important the external
> >      ordering (loading modules) and not the stack.
> > 
> > 
> > Back to your question:
> > 
> > The stack is needed for dependent non-cumulative patches.
> > 
> > The cumulative patches with "replace" flag seems the be
> > the most safe and most flexible solution. The question is
> > if we want to force all users to use this mode.
> 
> If they have dependencies between modules, they can either a) enforce it
> with tooling, or they can instead b) use 'replace'.
> 
> But let's get the module load order enforcement out of the kernel.
> There's no real need for the kernel to do it, and we're not even doing a
> good job at it.
> 
> > > > Or we have replace patches that are
> > > > standalone and we allow a smooth transfer from one to another one.
> > > > 
> > > > Anyway, for us, it is much more important the removal of replaced
> > > > patches. We could probably live without the possibility to replace
> > > > disabled patches.
> > > 
> > > I think replacing disabled patches is ok, *if* we get rid of the
> > > illusion of a stack.  The current stack isn't really a stack, it's just
> > > awkward.
> > 
> > I personally do not have problems with it. As I said, I see this as
> > two different modes how the life patches are distributed. The stack
> > is needed for dependent patches. The cumulative patches with
> > "replace" flag are self-contained and independent. They might
> > replace anything.
> > 
> > Well, it would make sense to reduce the amount of possible
> > situations and use cases.
> 
> A big +1.
> 
> > The question is what is acceptable to others
> 
> If there are any objections, this is their chance to speak up :-)
> 
> > and if it needs to be done as part of this patch set.
> 
> Maybe so, for at least a few reasons:
> 
> - This patch set makes the 'stack' obsolete, so it makes sense to remove
>   the 'stack' with it.

Not necessarily. I like Petr's rebase explanation here.

Miroslav
 
> - This patch set will already affect tooling, let's make tooling's life
>   easier by making all the related changes at the same time.
>   
>   (Though I'm not quite convinced on this point, would removing the
>   stack affect tooling at all?  If we also combined registration and
>   enablement into a single concept, then it definitely would.)
> 
> -- 
> Josh
> 

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Josh Poimboeuf]

On Wed, Apr 11, 2018 at 10:07:31AM +0200, Miroslav Benes wrote:
> > > I was confused by wording "in the middle". It suggested that there
> > > might had been enabled patches on the top and the bottom of the stack
> > > and some disabled patches in between at the same time (or vice versa).
> > > This was not true.
> > 
> > That *was* what I meant.  Consider the following sequence of events:
> > 
> > - Register patch 1
> > - Enable patch 1
> > - Register patch 2
> > - Enable patch 2
> > - Disable patch 2
> > - Register patch 3
> > - Enable patch 3
> > 
> > Notice that patch 2 (in the middle) is disabled, whereas patch 1 (on the
> > bottom) and patch 3 (on the top) are enabled.
> 
> This should not be possible at all.
> 
> __klp_enable_patch:
> 
>         if (patch->list.prev != &klp_patches &&
>             !list_prev_entry(patch, list)->enabled)
>                 return -EBUSY;
> 
> When patch 3 is enabled, list_prev_entry() returns patch 2 and its 
> ->enabled is false.

Hm, you're right.  I'm not sure how I got that idea...

I still agree with my original conclusion that enforcing stack order no
longer makes sense though.

> > > Another possibility would be to get rid of the enable/disable states.
> > > I mean that the patch will be automatically enabled during
> > > registration and removed during unregistration.
> > 
> > I don't see how disabling during unregistration would be possible, since
> > the unregister is called from the patch module exit function, which
> > can only be called *after* the patch is disabled.
> > 
> > However, we could unregister immediately after disabling (i.e., in
> > enabled_store context).
> 
> I think this is what Petr meant. So there would be nothing in the patch 
> module exit function. Well, not exactly. We'd need to remove sysfs dir and 
> maybe something more.

Sounds good to me, though aren't the livepatch sysfs entries removed by
klp during unregister?

> > > The question is what is acceptable to others
> > 
> > If there are any objections, this is their chance to speak up :-)
> > 
> > > and if it needs to be done as part of this patch set.
> > 
> > Maybe so, for at least a few reasons:
> > 
> > - This patch set makes the 'stack' obsolete, so it makes sense to remove
> >   the 'stack' with it.
> 
> Not necessarily. I like Petr's rebase explanation here.

I'm not sure what you mean.  IIRC, his rebase explanation referred to
how we handle 'replace' patches, for which there is no stacking (as I
meant the term: enforcement of stack order for registration and
enablement).

-- 
Josh

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Miroslav Benes]

On Wed, 11 Apr 2018, Josh Poimboeuf wrote:

> On Wed, Apr 11, 2018 at 10:07:31AM +0200, Miroslav Benes wrote:
> > > > I was confused by wording "in the middle". It suggested that there
> > > > might had been enabled patches on the top and the bottom of the stack
> > > > and some disabled patches in between at the same time (or vice versa).
> > > > This was not true.
> > > 
> > > That *was* what I meant.  Consider the following sequence of events:
> > > 
> > > - Register patch 1
> > > - Enable patch 1
> > > - Register patch 2
> > > - Enable patch 2
> > > - Disable patch 2
> > > - Register patch 3
> > > - Enable patch 3
> > > 
> > > Notice that patch 2 (in the middle) is disabled, whereas patch 1 (on the
> > > bottom) and patch 3 (on the top) are enabled.
> > 
> > This should not be possible at all.
> > 
> > __klp_enable_patch:
> > 
> >         if (patch->list.prev != &klp_patches &&
> >             !list_prev_entry(patch, list)->enabled)
> >                 return -EBUSY;
> > 
> > When patch 3 is enabled, list_prev_entry() returns patch 2 and its 
> > ->enabled is false.
> 
> Hm, you're right.  I'm not sure how I got that idea...
> 
> I still agree with my original conclusion that enforcing stack order no
> longer makes sense though.

Frankly I cannot say. I have got no opinion on this, so if there is a 
patch to remove it, I won't oppose it (probably). I just think it is 
connected to the atomic replace patch set.
 
> > > > Another possibility would be to get rid of the enable/disable states.
> > > > I mean that the patch will be automatically enabled during
> > > > registration and removed during unregistration.
> > > 
> > > I don't see how disabling during unregistration would be possible, since
> > > the unregister is called from the patch module exit function, which
> > > can only be called *after* the patch is disabled.
> > > 
> > > However, we could unregister immediately after disabling (i.e., in
> > > enabled_store context).
> > 
> > I think this is what Petr meant. So there would be nothing in the patch 
> > module exit function. Well, not exactly. We'd need to remove sysfs dir and 
> > maybe something more.
> 
> Sounds good to me, though aren't the livepatch sysfs entries removed by
> klp during unregister?

Yes. I was thinking we may take something out of disable+unregister step 
and leave it in the exit function, but maybe there is nothing like that.

> > > > The question is what is acceptable to others
> > > 
> > > If there are any objections, this is their chance to speak up :-)
> > > 
> > > > and if it needs to be done as part of this patch set.
> > > 
> > > Maybe so, for at least a few reasons:
> > > 
> > > - This patch set makes the 'stack' obsolete, so it makes sense to remove
> > >   the 'stack' with it.
> > 
> > Not necessarily. I like Petr's rebase explanation here.
> 
> I'm not sure what you mean.  IIRC, his rebase explanation referred to
> how we handle 'replace' patches, for which there is no stacking (as I
> meant the term: enforcement of stack order for registration and
> enablement).

See above. I don't completely agree with the obsoleteness part. I don't 
see it that way. But it is moot. I'm not against the enforcement removal.

Regards,
Miroslav

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Petr Mladek]

On Wed 2018-04-11 07:32:14, Josh Poimboeuf wrote:
> On Wed, Apr 11, 2018 at 10:07:31AM +0200, Miroslav Benes wrote:
> > > > I was confused by wording "in the middle". It suggested that there
> > > > might had been enabled patches on the top and the bottom of the stack
> > > > and some disabled patches in between at the same time (or vice versa).
> > > > This was not true.
> > > 
> > > That *was* what I meant.  Consider the following sequence of events:
> > > 
> > > - Register patch 1
> > > - Enable patch 1
> > > - Register patch 2
> > > - Enable patch 2
> > > - Disable patch 2
> > > - Register patch 3
> > > - Enable patch 3
> > > 
> > > Notice that patch 2 (in the middle) is disabled, whereas patch 1 (on the
> > > bottom) and patch 3 (on the top) are enabled.
> > 
> > This should not be possible at all.
> > 
> > __klp_enable_patch:
> > 
> >         if (patch->list.prev != &klp_patches &&
> >             !list_prev_entry(patch, list)->enabled)
> >                 return -EBUSY;
> > 
> > When patch 3 is enabled, list_prev_entry() returns patch 2 and its 
> > ->enabled is false.
> 
> Hm, you're right.  I'm not sure how I got that idea...

It is great that we finally understand each other.

> I still agree with my original conclusion that enforcing stack order no
> longer makes sense though.

The question is what we will get if we remove the stack. Will it
really make the code easier and livepatching more safe?

First, note that stack is the main trick used by the ftrace handler.
It gets the patch from top of the stack and use the new_addr from
that patch.

If we remove the stack, we still need to handle 3 possibilities.
The handler will need to continue either with original_code,
old_patch or new_patch.

Now, just imagine the code. We will need variables orig_addr,
old_addr, new_addr or so which might be confusing. It will be
even more confusion if we do revert/disable. Also new_addr will
become old_addr if we install yet another patch.

We had exactly this in kGraft and it was a mess. I said "wow,
that is genius" when I saw the stack approach in the upstream
code proposal.


Second, unrelated patches must never patch the same functions.
Otherwise we would not be able to define which implementation
should be used. This is especially important when a patch is
removed and we need to fallback either to another patch or
original code. Yes, it makes perfect sense. But it needs code
that will check it, refuse loading the patch, ... It is not
complicated. But it is rather additional code than
simplification. I might make livepatching more safe
but probably not simplify the code.


> > > > Another possibility would be to get rid of the enable/disable states.
> > > > I mean that the patch will be automatically enabled during
> > > > registration and removed during unregistration.
> > > 
> > > I don't see how disabling during unregistration would be possible, since
> > > the unregister is called from the patch module exit function, which
> > > can only be called *after* the patch is disabled.
> > > 
> > > However, we could unregister immediately after disabling (i.e., in
> > > enabled_store context).
> > 
> > I think this is what Petr meant. So there would be nothing in the patch 
> > module exit function. Well, not exactly. We'd need to remove sysfs dir and 
> > maybe something more.
> 
> Sounds good to me, though aren't the livepatch sysfs entries removed by
> klp during unregister?

This is why I asked in my earlier mail if we need to keep sysfs
entries for unused patches. We could remove them when the patch
gets disabled (transition finishes). Then we do not need to
do anything in module_exit().

The patch module can be removed only when the transition is not
forced and we call module_put().

Best Regards,
Petr

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Josh Poimboeuf]

On Wed, Apr 11, 2018 at 04:17:11PM +0200, Petr Mladek wrote:
> > I still agree with my original conclusion that enforcing stack order no
> > longer makes sense though.
> 
> The question is what we will get if we remove the stack. Will it
> really make the code easier and livepatching more safe?
> 
> First, note that stack is the main trick used by the ftrace handler.
> It gets the patch from top of the stack and use the new_addr from
> that patch.
> 
> If we remove the stack, we still need to handle 3 possibilities.
> The handler will need to continue either with original_code,
> old_patch or new_patch.
> 
> Now, just imagine the code. We will need variables orig_addr,
> old_addr, new_addr or so which might be confusing. It will be
> even more confusion if we do revert/disable. Also new_addr will
> become old_addr if we install yet another patch.
> 
> We had exactly this in kGraft and it was a mess. I said "wow,
> that is genius" when I saw the stack approach in the upstream
> code proposal.

You're confusing the func stack with the patch stack.

My proposal is to get rid of the patch stack.

We can keep the func stack.  It will be needed anyway for the 'replace'
case, where the stack may be of size 2.

> Second, unrelated patches must never patch the same functions.
> Otherwise we would not be able to define which implementation
> should be used. This is especially important when a patch is
> removed and we need to fallback either to another patch or
> original code. Yes, it makes perfect sense. But it needs code
> that will check it, refuse loading the patch, ... It is not
> complicated. But it is rather additional code than
> simplification. I might make livepatching more safe
> but probably not simplify the code.

We don't need to enforce that.  The func stack can stay.  If somebody
wants to patch the same function multiple times (without using
'replace'), that's inadvisable, but it's also their business.  They're
responsible for the tooling to ensure the patch stack order is sane.

> > > > > Another possibility would be to get rid of the enable/disable states.
> > > > > I mean that the patch will be automatically enabled during
> > > > > registration and removed during unregistration.
> > > > 
> > > > I don't see how disabling during unregistration would be possible, since
> > > > the unregister is called from the patch module exit function, which
> > > > can only be called *after* the patch is disabled.
> > > > 
> > > > However, we could unregister immediately after disabling (i.e., in
> > > > enabled_store context).
> > > 
> > > I think this is what Petr meant. So there would be nothing in the patch 
> > > module exit function. Well, not exactly. We'd need to remove sysfs dir and 
> > > maybe something more.
> > 
> > Sounds good to me, though aren't the livepatch sysfs entries removed by
> > klp during unregister?
> 
> This is why I asked in my earlier mail if we need to keep sysfs
> entries for unused patches.

If they are permanently disabled then I think the sysfs entries should
be removed.

> We could remove them when the patch gets disabled (transition
> finishes). Then we do not need to do anything in module_exit().

Agreed, though what happens if the transition finishes while still
running in the context of the write to the sysfs entry?  Can we remove a
sysfs entry while executing code associated with it?

-- 
Josh

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Petr Mladek]

On Wed 2018-04-11 10:48:52, Josh Poimboeuf wrote:
> On Wed, Apr 11, 2018 at 04:17:11PM +0200, Petr Mladek wrote:
> > > I still agree with my original conclusion that enforcing stack order no
> > > longer makes sense though.
> > 
> > The question is what we will get if we remove the stack. Will it
> > really make the code easier and livepatching more safe?
> > 
> > First, note that stack is the main trick used by the ftrace handler.
> > It gets the patch from top of the stack and use the new_addr from
> > that patch.
> > 
> > If we remove the stack, we still need to handle 3 possibilities.
> > The handler will need to continue either with original_code,
> > old_patch or new_patch.
> > 
> > Now, just imagine the code. We will need variables orig_addr,
> > old_addr, new_addr or so which might be confusing. It will be
> > even more confusion if we do revert/disable. Also new_addr will
> > become old_addr if we install yet another patch.
> > 
> > We had exactly this in kGraft and it was a mess. I said "wow,
> > that is genius" when I saw the stack approach in the upstream
> > code proposal.
> 
> You're confusing the func stack with the patch stack.
>
> My proposal is to get rid of the patch stack.

The are related from my POV. The patches have the same ordering
in both patch and function stacks. The patch ordering is checked
when the patches are enabled and disabled.


> We can keep the func stack.  It will be needed anyway for the 'replace'
> case, where the stack may be of size 2.

OK, you want to keep the func stack because it is useful from the
implementation point of view.


> > Second, unrelated patches must never patch the same functions.
> > Otherwise we would not be able to define which implementation
> > should be used. This is especially important when a patch is
> > removed and we need to fallback either to another patch or
> > original code. Yes, it makes perfect sense. But it needs code
> > that will check it, refuse loading the patch, ... It is not
> > complicated. But it is rather additional code than
> > simplification. I might make livepatching more safe
> > but probably not simplify the code.
> 
> We don't need to enforce that.  The func stack can stay.  If somebody
> wants to patch the same function multiple times (without using
> 'replace'), that's inadvisable, but it's also their business.  They're
> responsible for the tooling to ensure the patch stack order is sane.


While it might make sense to ignore the patch stack (ordering) for
the enable operation. Do we really want to ignore it when disabling
a patch.

By other words, do we want to allow disabling a patch that is in
the middle of the stack, only partly in use? Does not this allow
some other crazy scenarios? Is it really the user business?
Will it make our life easier?

This will not happen if we refuse to load non-replace patches
that touch an already patches fucntion. Then the patch stack
might become only implementation detail. It will not define
the ordering any longer.


> > > > > > Another possibility would be to get rid of the enable/disable states.
> > > > > > I mean that the patch will be automatically enabled during
> > > > > > registration and removed during unregistration.
> > > > > 
> > > > > I don't see how disabling during unregistration would be possible, since
> > > > > the unregister is called from the patch module exit function, which
> > > > > can only be called *after* the patch is disabled.
> > > > > 
> > > > > However, we could unregister immediately after disabling (i.e., in
> > > > > enabled_store context).
> > > > 
> > > > I think this is what Petr meant. So there would be nothing in the patch 
> > > > module exit function. Well, not exactly. We'd need to remove sysfs dir and 
> > > > maybe something more.
> > > 
> > > Sounds good to me, though aren't the livepatch sysfs entries removed by
> > > klp during unregister?
> > 
> > This is why I asked in my earlier mail if we need to keep sysfs
> > entries for unused patches.
> 
> If they are permanently disabled then I think the sysfs entries should
> be removed.
> 
> > We could remove them when the patch gets disabled (transition
> > finishes). Then we do not need to do anything in module_exit().
> 
> Agreed, though what happens if the transition finishes while still
> running in the context of the write to the sysfs entry?  Can we remove a
> sysfs entry while executing code associated with it?

I would need to test it to be sure. But I believe that it will be
possible to remove sysfs entry from its own callback. All this code
heavily uses reference counters and callbacks called when the count
reaches zero.

OK. What about the following solution?

   + Enable patch directly from klp_register_patch()
   + Unregister the patch directly from klp_complete_transition()
     when the patch is disabled.
   + Put the module later when all sysfs entries are removed
     in klp_unregister_patch().

As a result:

   + module_init() will need to call only klp_register_patch()
   + module_exit() will do nothing
   + the module can be removed only when it is not longer needed


Some other ideas:

   + rename /sys/kernel/livepatch/<patch>/enable -> unregister
     allow to write into /sys/kernel/livepatch/<patch>/transition

     + echo 1 >unregistrer  to disable&unregister the patch
     + echo 0 >transition   to revert the running transition


The question is what to do with the stack of patches. It will have
no meaning for the enable operation because it will be done
automatically. But what about the disable/unregistrer operation?


Anyway, this looks like a revolution. Do we want to do all these
changes, including atomic replace, in a single patchset?

In each case, the atomic replace must be a pre-requisite. It will
become the only way to handle dependent patches.

Best Regards,
Petr

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Josh Poimboeuf]

On Mon, Apr 16, 2018 at 04:58:11PM +0200, Petr Mladek wrote:
> On Wed 2018-04-11 10:48:52, Josh Poimboeuf wrote:
> > On Wed, Apr 11, 2018 at 04:17:11PM +0200, Petr Mladek wrote:
> > > > I still agree with my original conclusion that enforcing stack order no
> > > > longer makes sense though.
> > > 
> > > The question is what we will get if we remove the stack. Will it
> > > really make the code easier and livepatching more safe?
> > > 
> > > First, note that stack is the main trick used by the ftrace handler.
> > > It gets the patch from top of the stack and use the new_addr from
> > > that patch.
> > > 
> > > If we remove the stack, we still need to handle 3 possibilities.
> > > The handler will need to continue either with original_code,
> > > old_patch or new_patch.
> > > 
> > > Now, just imagine the code. We will need variables orig_addr,
> > > old_addr, new_addr or so which might be confusing. It will be
> > > even more confusion if we do revert/disable. Also new_addr will
> > > become old_addr if we install yet another patch.
> > > 
> > > We had exactly this in kGraft and it was a mess. I said "wow,
> > > that is genius" when I saw the stack approach in the upstream
> > > code proposal.
> > 
> > You're confusing the func stack with the patch stack.
> >
> > My proposal is to get rid of the patch stack.
> 
> The are related from my POV. The patches have the same ordering
> in both patch and function stacks. The patch ordering is checked
> when the patches are enabled and disabled.

True.

> > We can keep the func stack.  It will be needed anyway for the 'replace'
> > case, where the stack may be of size 2.
> 
> OK, you want to keep the func stack because it is useful from the
> implementation point of view.
> 
> 
> > > Second, unrelated patches must never patch the same functions.
> > > Otherwise we would not be able to define which implementation
> > > should be used. This is especially important when a patch is
> > > removed and we need to fallback either to another patch or
> > > original code. Yes, it makes perfect sense. But it needs code
> > > that will check it, refuse loading the patch, ... It is not
> > > complicated. But it is rather additional code than
> > > simplification. I might make livepatching more safe
> > > but probably not simplify the code.
> > 
> > We don't need to enforce that.  The func stack can stay.  If somebody
> > wants to patch the same function multiple times (without using
> > 'replace'), that's inadvisable, but it's also their business.  They're
> > responsible for the tooling to ensure the patch stack order is sane.
> 
> 
> While it might make sense to ignore the patch stack (ordering) for
> the enable operation. Do we really want to ignore it when disabling
> a patch.
> 
> By other words, do we want to allow disabling a patch that is in
> the middle of the stack, only partly in use? Does not this allow
> some other crazy scenarios? Is it really the user business?
> Will it make our life easier?

If there's no longer a patch stack, then there's no concept of a middle.
We would expect the patches to be independent of one another, and so
disabling any of them independently would be harmless.

If any of the patches share a func, and the user disables one in the
"middle", it's not our job to support that.  The vendor / patch author
should prevent such cases from occurring with tooling, packaging,
documentation, etc.  Or they can just use 'replace'.

We can already have similar unexpected situations today.  For example,
what if patch B is a cumulative superset of patch A, but the user
mistakenly loads patch A (without replace) *after* loading patch B?
Then some unforeseen craziness could ensue.

We can't control all such scenarios (and that's ok), but we shouldn't
pretend that we support them.

> This will not happen if we refuse to load non-replace patches
> that touch an already patches fucntion. Then the patch stack
> might become only implementation detail. It will not define
> the ordering any longer.

I think this would only be a partial solution.  Patches can have
implicit interdependencies, even if they don't patch the same function.
Also it doesn't solve the problem when patches are loaded in the wrong
order.  We have to trust vendors and admins to do the right thing.

> > > > > > > Another possibility would be to get rid of the enable/disable states.
> > > > > > > I mean that the patch will be automatically enabled during
> > > > > > > registration and removed during unregistration.
> > > > > > 
> > > > > > I don't see how disabling during unregistration would be possible, since
> > > > > > the unregister is called from the patch module exit function, which
> > > > > > can only be called *after* the patch is disabled.
> > > > > > 
> > > > > > However, we could unregister immediately after disabling (i.e., in
> > > > > > enabled_store context).
> > > > > 
> > > > > I think this is what Petr meant. So there would be nothing in the patch 
> > > > > module exit function. Well, not exactly. We'd need to remove sysfs dir and 
> > > > > maybe something more.
> > > > 
> > > > Sounds good to me, though aren't the livepatch sysfs entries removed by
> > > > klp during unregister?
> > > 
> > > This is why I asked in my earlier mail if we need to keep sysfs
> > > entries for unused patches.
> > 
> > If they are permanently disabled then I think the sysfs entries should
> > be removed.
> > 
> > > We could remove them when the patch gets disabled (transition
> > > finishes). Then we do not need to do anything in module_exit().
> > 
> > Agreed, though what happens if the transition finishes while still
> > running in the context of the write to the sysfs entry?  Can we remove a
> > sysfs entry while executing code associated with it?
> 
> I would need to test it to be sure. But I believe that it will be
> possible to remove sysfs entry from its own callback. All this code
> heavily uses reference counters and callbacks called when the count
> reaches zero.
> 
> OK. What about the following solution?
> 
>    + Enable patch directly from klp_register_patch()
>    + Unregister the patch directly from klp_complete_transition()
>      when the patch is disabled.
>    + Put the module later when all sysfs entries are removed
>      in klp_unregister_patch().
> 
> As a result:
> 
>    + module_init() will need to call only klp_register_patch()
>    + module_exit() will do nothing
>    + the module can be removed only when it is not longer needed

Sounds good to me.

> Some other ideas:
> 
>    + rename /sys/kernel/livepatch/<patch>/enable -> unregister
>      allow to write into /sys/kernel/livepatch/<patch>/transition
> 
>      + echo 1 >unregistrer  to disable&unregister the patch
>      + echo 0 >transition   to revert the running transition

Why not keep the existing sysfs interfaces?  So

  echo 0 > enable

would disable (and eventually unregister) the patch.

> The question is what to do with the stack of patches. It will have
> no meaning for the enable operation because it will be done
> automatically. But what about the disable/unregistrer operation?

Assuming we got rid of the patch stack, would we even need to keep a
global list of patches anymore?

> Anyway, this looks like a revolution. Do we want to do all these
> changes, including atomic replace, in a single patchset?

At least for the bits which affect external tooling interfaces, it would
be nice to group them all together.

-- 
Josh

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Miroslav Benes]

> > > > Second, unrelated patches must never patch the same functions.
> > > > Otherwise we would not be able to define which implementation
> > > > should be used. This is especially important when a patch is
> > > > removed and we need to fallback either to another patch or
> > > > original code. Yes, it makes perfect sense. But it needs code
> > > > that will check it, refuse loading the patch, ... It is not
> > > > complicated. But it is rather additional code than
> > > > simplification. I might make livepatching more safe
> > > > but probably not simplify the code.
> > > 
> > > We don't need to enforce that.  The func stack can stay.  If somebody
> > > wants to patch the same function multiple times (without using
> > > 'replace'), that's inadvisable, but it's also their business.  They're
> > > responsible for the tooling to ensure the patch stack order is sane.
> > 
> > 
> > While it might make sense to ignore the patch stack (ordering) for
> > the enable operation. Do we really want to ignore it when disabling
> > a patch.
> > 
> > By other words, do we want to allow disabling a patch that is in
> > the middle of the stack, only partly in use? Does not this allow
> > some other crazy scenarios? Is it really the user business?
> > Will it make our life easier?
> 
> If there's no longer a patch stack, then there's no concept of a middle.
> We would expect the patches to be independent of one another, and so
> disabling any of them independently would be harmless.
> 
> If any of the patches share a func, and the user disables one in the
> "middle", it's not our job to support that.  The vendor / patch author
> should prevent such cases from occurring with tooling, packaging,
> documentation, etc.  Or they can just use 'replace'.
> 
> We can already have similar unexpected situations today.  For example,
> what if patch B is a cumulative superset of patch A, but the user
> mistakenly loads patch A (without replace) *after* loading patch B?
> Then some unforeseen craziness could ensue.
> 
> We can't control all such scenarios (and that's ok), but we shouldn't
> pretend that we support them.

FWIW I agree with the above. Provided we keep the func stack.

[...]

> > The question is what to do with the stack of patches. It will have
> > no meaning for the enable operation because it will be done
> > automatically. But what about the disable/unregistrer operation?
> 
> Assuming we got rid of the patch stack, would we even need to keep a
> global list of patches anymore?

There are places we walk through the list of patches (klp_add_nops() in 
this patch set, klp_module_coming()), so probably yes.

Miroslav

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Petr Mladek]

On Mon 2018-04-16 14:04:25, Josh Poimboeuf wrote:
> On Mon, Apr 16, 2018 at 04:58:11PM +0200, Petr Mladek wrote:
> > On Wed 2018-04-11 10:48:52, Josh Poimboeuf wrote:
> > > On Wed, Apr 11, 2018 at 04:17:11PM +0200, Petr Mladek wrote:
> > > > Second, unrelated patches must never patch the same functions.
> > > > Otherwise we would not be able to define which implementation
> > > > should be used. This is especially important when a patch is
> > > > removed and we need to fallback either to another patch or
> > > > original code. Yes, it makes perfect sense. But it needs code
> > > > that will check it, refuse loading the patch, ... It is not
> > > > complicated. But it is rather additional code than
> > > > simplification. I might make livepatching more safe
> > > > but probably not simplify the code.
> > > 
> > > We don't need to enforce that.  The func stack can stay.  If somebody
> > > wants to patch the same function multiple times (without using
> > > 'replace'), that's inadvisable, but it's also their business.  They're
> > > responsible for the tooling to ensure the patch stack order is sane.
> > 
> > 
> > While it might make sense to ignore the patch stack (ordering) for
> > the enable operation. Do we really want to ignore it when disabling
> > a patch.
> > 
> > By other words, do we want to allow disabling a patch that is in
> > the middle of the stack, only partly in use? Does not this allow
> > some other crazy scenarios? Is it really the user business?
> > Will it make our life easier?
> 
> If there's no longer a patch stack, then there's no concept of a middle.
> We would expect the patches to be independent of one another, and so
> disabling any of them independently would be harmless.
>
> If any of the patches share a func, and the user disables one in the
> "middle", it's not our job to support that.  The vendor / patch author
> should prevent such cases from occurring with tooling, packaging,
> documentation, etc.  Or they can just use 'replace'.
> 
> We can already have similar unexpected situations today.  For example,
> what if patch B is a cumulative superset of patch A, but the user
> mistakenly loads patch A (without replace) *after* loading patch B?
> Then some unforeseen craziness could ensue.
> 
> We can't control all such scenarios (and that's ok), but we shouldn't
> pretend that we support them.

I recall some earlier discussion. The aim was to make livepatching
more safe. AFAIK, it was more related to the helper tooling around,
like "automatic" generation of the special relocation entries, ...
Anyway, I feel that the above gets somehow against it.

The patch stack does not solve everything. But it makes it harder
to do wrong things.


> > This will not happen if we refuse to load non-replace patches
> > that touch an already patches fucntion. Then the patch stack
> > might become only implementation detail. It will not define
> > the ordering any longer.
> 
> I think this would only be a partial solution.  Patches can have
> implicit interdependencies, even if they don't patch the same function.
> Also it doesn't solve the problem when patches are loaded in the wrong
> order.  We have to trust vendors and admins to do the right thing.

I would still like to add this check if we remove the stack.
Is it too restrictive? Maybe. But it would help to prevent
creating some ugly mistakes. By other words, we will force
people using proper cumulative patches instead of
dangerous semi-cumulative Frankenstains.

At least, it would reduce the number of scenarios that
we might meet and eventually need to debug.


> > > > > > > > Another possibility would be to get rid of the enable/disable states.
> > > > > > > > I mean that the patch will be automatically enabled during
> > > > > > > > registration and removed during unregistration.
> > 
> > OK. What about the following solution?
> > 
> >    + Enable patch directly from klp_register_patch()
> >    + Unregister the patch directly from klp_complete_transition()
> >      when the patch is disabled.
> >    + Put the module later when all sysfs entries are removed
> >      in klp_unregister_patch().
> > 
> > As a result:
> > 
> >    + module_init() will need to call only klp_register_patch()
> >    + module_exit() will do nothing
> >    + the module can be removed only when it is not longer needed
> 
> Sounds good to me.

It should be consistent with sysfs interface, see below.


> > Some other ideas:
> > 
> >    + rename /sys/kernel/livepatch/<patch>/enable -> unregister
> >      allow to write into /sys/kernel/livepatch/<patch>/transition
> > 
> >      + echo 1 >unregistrer  to disable&unregister the patch
> >      + echo 0 >transition   to revert the running transition
> 
> Why not keep the existing sysfs interfaces?  So
> 
>   echo 0 > enable
> 
> would disable (and eventually unregister) the patch.

First, it would be a bit inconsistent. The patch would be added by calling
"register" function and removed by writing into "enabled" file.

Second, if we remove sysfs entries for disabled patches, it would
make the meaning of "enabled" file a bit useless. I mean that
the patch will always be enabled when the sysfs directory exists
(when ignoring transition states).


BTW: I have just today got a feedback that the user interface is not
ideal. In particular, it is not easy to detect what process is blocking
the transition. It is because the target value in
/proc/<pid>/patch_state depends on the value in
/sys/kernel/livepatch/<patch>/enable (direction of the transition).


Anyway, the first problem might be solved by doing patch registration
at the beginning of klp_enable_patch(). Then it will match the
"enabled" sysfs file.

I would actually feel more comfortable if we do not change the
sysfs interface.

Well, any better idea for a simplified interface is appreciated.


> > The question is what to do with the stack of patches. It will have
> > no meaning for the enable operation because it will be done
> > automatically. But what about the disable/unregistrer operation?
> 
> Assuming we got rid of the patch stack, would we even need to keep a
> global list of patches anymore?

As Mirek said, a list still might be useful in some situations
(nops generation, conflicts check). But it would become an
implementation detail. The ordering would not be important
any longer.


> > Anyway, this looks like a revolution. Do we want to do all these
> > changes, including atomic replace, in a single patchset?
> 
> At least for the bits which affect external tooling interfaces, it would
> be nice to group them all together.

OK, I'll try to cook up something once we settle on the above
questions.

Best Regards,
Petr

Re: [PATCH v10 05/10] livepatch: Support separate list for replaced patches.

[Josh Poimboeuf]

On Tue, Apr 17, 2018 at 05:37:46PM +0200, Petr Mladek wrote:
> On Mon 2018-04-16 14:04:25, Josh Poimboeuf wrote:
> > On Mon, Apr 16, 2018 at 04:58:11PM +0200, Petr Mladek wrote:
> > > On Wed 2018-04-11 10:48:52, Josh Poimboeuf wrote:
> > > > On Wed, Apr 11, 2018 at 04:17:11PM +0200, Petr Mladek wrote:
> > > > > Second, unrelated patches must never patch the same functions.
> > > > > Otherwise we would not be able to define which implementation
> > > > > should be used. This is especially important when a patch is
> > > > > removed and we need to fallback either to another patch or
> > > > > original code. Yes, it makes perfect sense. But it needs code
> > > > > that will check it, refuse loading the patch, ... It is not
> > > > > complicated. But it is rather additional code than
> > > > > simplification. I might make livepatching more safe
> > > > > but probably not simplify the code.
> > > > 
> > > > We don't need to enforce that.  The func stack can stay.  If somebody
> > > > wants to patch the same function multiple times (without using
> > > > 'replace'), that's inadvisable, but it's also their business.  They're
> > > > responsible for the tooling to ensure the patch stack order is sane.
> > > 
> > > 
> > > While it might make sense to ignore the patch stack (ordering) for
> > > the enable operation. Do we really want to ignore it when disabling
> > > a patch.
> > > 
> > > By other words, do we want to allow disabling a patch that is in
> > > the middle of the stack, only partly in use? Does not this allow
> > > some other crazy scenarios? Is it really the user business?
> > > Will it make our life easier?
> > 
> > If there's no longer a patch stack, then there's no concept of a middle.
> > We would expect the patches to be independent of one another, and so
> > disabling any of them independently would be harmless.
> >
> > If any of the patches share a func, and the user disables one in the
> > "middle", it's not our job to support that.  The vendor / patch author
> > should prevent such cases from occurring with tooling, packaging,
> > documentation, etc.  Or they can just use 'replace'.
> > 
> > We can already have similar unexpected situations today.  For example,
> > what if patch B is a cumulative superset of patch A, but the user
> > mistakenly loads patch A (without replace) *after* loading patch B?
> > Then some unforeseen craziness could ensue.
> > 
> > We can't control all such scenarios (and that's ok), but we shouldn't
> > pretend that we support them.
> 
> I recall some earlier discussion. The aim was to make livepatching
> more safe. AFAIK, it was more related to the helper tooling around,
> like "automatic" generation of the special relocation entries, ...
> Anyway, I feel that the above gets somehow against it.
> 
> The patch stack does not solve everything. But it makes it harder
> to do wrong things.

It also makes it harder to do _right_ things.  For example, disabling a
patch in the "middle" should be allowable if there are no patch
dependencies.

> > > This will not happen if we refuse to load non-replace patches
> > > that touch an already patches fucntion. Then the patch stack
> > > might become only implementation detail. It will not define
> > > the ordering any longer.
> > 
> > I think this would only be a partial solution.  Patches can have
> > implicit interdependencies, even if they don't patch the same function.
> > Also it doesn't solve the problem when patches are loaded in the wrong
> > order.  We have to trust vendors and admins to do the right thing.
> 
> I would still like to add this check if we remove the stack.
> Is it too restrictive? Maybe. But it would help to prevent
> creating some ugly mistakes. By other words, we will force
> people using proper cumulative patches instead of
> dangerous semi-cumulative Frankenstains.
> 
> At least, it would reduce the number of scenarios that
> we might meet and eventually need to debug.

But the non-replace cumulative model can be perfectly safe, if the
packaging/tooling supports the right workflow.

I think we're talking about theoretical situations anyway.  I'd rather
not add any code for theoretical problems.

If somebody reports a bug due to dangerous real world usage, then we can
decide how to handle it at that point -- though I suspect my response
will be "don't do that!"

> > > > > > > > > Another possibility would be to get rid of the enable/disable states.
> > > > > > > > > I mean that the patch will be automatically enabled during
> > > > > > > > > registration and removed during unregistration.
> > > 
> > > OK. What about the following solution?
> > > 
> > >    + Enable patch directly from klp_register_patch()
> > >    + Unregister the patch directly from klp_complete_transition()
> > >      when the patch is disabled.
> > >    + Put the module later when all sysfs entries are removed
> > >      in klp_unregister_patch().
> > > 
> > > As a result:
> > > 
> > >    + module_init() will need to call only klp_register_patch()
> > >    + module_exit() will do nothing
> > >    + the module can be removed only when it is not longer needed
> > 
> > Sounds good to me.
> 
> It should be consistent with sysfs interface, see below.
> 
> 
> > > Some other ideas:
> > > 
> > >    + rename /sys/kernel/livepatch/<patch>/enable -> unregister
> > >      allow to write into /sys/kernel/livepatch/<patch>/transition
> > > 
> > >      + echo 1 >unregistrer  to disable&unregister the patch
> > >      + echo 0 >transition   to revert the running transition
> > 
> > Why not keep the existing sysfs interfaces?  So
> > 
> >   echo 0 > enable
> > 
> > would disable (and eventually unregister) the patch.
> 
> First, it would be a bit inconsistent. The patch would be added by calling
> "register" function and removed by writing into "enabled" file.

True.  Although from the end user (and tooling) standpoint, it looks the
same as before, since the register/unregister and enable are done
automatically by the patch module.  The only user-visible difference is
that the sysfs directory disappears and the patch module can't be
re-enabled.

> Second, if we remove sysfs entries for disabled patches, it would
> make the meaning of "enabled" file a bit useless. I mean that
> the patch will always be enabled when the sysfs directory exists
> (when ignoring transition states).

Though as you mentioned, it does still indicate which direction the
transition is going.

> BTW: I have just today got a feedback that the user interface is not
> ideal. In particular, it is not easy to detect what process is blocking
> the transition. It is because the target value in
> /proc/<pid>/patch_state depends on the value in
> /sys/kernel/livepatch/<patch>/enable (direction of the transition).

But hopefully tooling can hide the intricacies of the interface.

> Anyway, the first problem might be solved by doing patch registration
> at the beginning of klp_enable_patch(). Then it will match the
> "enabled" sysfs file.
> 
> I would actually feel more comfortable if we do not change the
> sysfs interface.
> 
> Well, any better idea for a simplified interface is appreciated.

We can just keep the current interface, which I think will make tooling
happy since it will function basically the same as before.

> > > The question is what to do with the stack of patches. It will have
> > > no meaning for the enable operation because it will be done
> > > automatically. But what about the disable/unregistrer operation?
> > 
> > Assuming we got rid of the patch stack, would we even need to keep a
> > global list of patches anymore?
> 
> As Mirek said, a list still might be useful in some situations
> (nops generation, conflicts check). But it would become an
> implementation detail. The ordering would not be important
> any longer.

Ok.

-- 
Josh

Re: [PATCH v10 00/10] livepatch: Atomic replace feature

[Evgenii Shatokhin]

Hi,

On 07.03.2018 11:20, Petr Mladek wrote:
> The atomic replace allows to create cumulative patches. They
> are useful when you maintain many livepatches and want to remove
> one that is lower on the stack. In addition it is very useful when
> more patches touch the same function and there are dependencies
> between them.

Could you tell me what is the current status of this series? Looks like 
I lost track of it.

Is there anything else that should be done here before this series could 
be merged?

> 
> 
> Changes against v9:
> 
>    + Fixed check of valid NOPs for already loaded objects,
>      regression introduced in v9 [Joe, Mirek]
>    + Allow to replace even disabled patches [Evgenii]
> 
> Changes against v8:
> 
>    + Fixed handling of statically defined struct klp_object
>      with empty array of functions [Joe, Mirek]
>    + Removed redundant func->new_func assignment for NOPs [Mirek]
>    + Improved some wording [Mirek]
> 
> Changes against v7:
> 
>    + Fixed handling of NOPs for not-yet-loaded modules
>    + Made klp_replaced_patches list static [Mirek]
>    + Made klp_free_object() public later [Mirek]
>    + Fixed several reported typos [Mirek, Joe]
>    + Updated documentation according to the feedback [Joe]
>    + Added some Acks [Mirek]
> 
> Changes against v6:
> 
>    + used list_move when disabling replaced patches [Jason]
>    + renamed KLP_FUNC_ORIGINAL -> KLP_FUNC_STATIC [Mirek]
>    + used klp_is_func_type() in klp_unpatch_object() [Mirek]
>    + moved static definition of klp_get_or_add_object() [Mirek]
>    + updated comment about synchronization in forced mode [Mirek]
>    + added user documentation
>    + fixed several typos
> 
> 
> Jason Baron (5):
>    livepatch: Use lists to manage patches, objects and functions
>    livepatch: Initial support for dynamic structures
>    livepatch: Allow to unpatch only functions of the given type
>    livepatch: Support separate list for replaced patches.
>    livepatch: Add atomic replace
> 
> Petr Mladek (5):
>    livepatch: Free only structures with initialized kobject
>    livepatch: Correctly handle atomic replace for not yet loaded modules
>    livepatch: Improve dynamic struct klp_object detection and
>      manipulation
>    livepatch: Allow to replace even disabled patches
>    livepatch: Atomic replace and cumulative patches documentation
> 
>   Documentation/livepatch/cumulative-patches.txt |  83 +++++
>   include/linux/livepatch.h                      |  65 +++-
>   kernel/livepatch/core.c                        | 422 ++++++++++++++++++++++---
>   kernel/livepatch/core.h                        |   4 +
>   kernel/livepatch/patch.c                       |  31 +-
>   kernel/livepatch/patch.h                       |   4 +-
>   kernel/livepatch/transition.c                  |  41 ++-
>   7 files changed, 598 insertions(+), 52 deletions(-)
>   create mode 100644 Documentation/livepatch/cumulative-patches.txt
> 

Regards,
Evgenii

Re: [PATCH v10 00/10] livepatch: Atomic replace feature

[Petr Mladek]

On Fri 2018-08-17 13:17:27, Evgenii Shatokhin wrote:
> Hi,
> 
> On 07.03.2018 11:20, Petr Mladek wrote:
> > The atomic replace allows to create cumulative patches. They
> > are useful when you maintain many livepatches and want to remove
> > one that is lower on the stack. In addition it is very useful when
> > more patches touch the same function and there are dependencies
> > between them.
> 
> Could you tell me what is the current status of this series? Looks like I
> lost track of it.
> 
> Is there anything else that should be done here before this series could be
> merged?

I have almost ready the rework suggested by Josh. I still need to
port the new tests and send.

Unfortunately I had to put it temporary on hold because of other
important tasks.

I am about to leave for vacation (one week off). I will try to
find some cycles, finish and send it once I am back.

Best Regards,
Petr

Re: [PATCH v10 00/10] livepatch: Atomic replace feature

[Evgenii Shatokhin]

On 17.08.2018 17:53, Petr Mladek wrote:
> On Fri 2018-08-17 13:17:27, Evgenii Shatokhin wrote:
>> Hi,
>>
>> On 07.03.2018 11:20, Petr Mladek wrote:
>>> The atomic replace allows to create cumulative patches. They
>>> are useful when you maintain many livepatches and want to remove
>>> one that is lower on the stack. In addition it is very useful when
>>> more patches touch the same function and there are dependencies
>>> between them.
>>
>> Could you tell me what is the current status of this series? Looks like I
>> lost track of it.
>>
>> Is there anything else that should be done here before this series could be
>> merged?
> 
> I have almost ready the rework suggested by Josh. I still need to
> port the new tests and send.
> 
> Unfortunately I had to put it temporary on hold because of other
> important tasks.
> 
> I am about to leave for vacation (one week off). I will try to
> find some cycles, finish and send it once I am back.

That would be great! Thanks.

> 
> Best Regards,
> Petr
> .
>