[PATCH v7 0/5] of: change overlay apply input data from unflattened

[PATCH v7 0/5] of: change overlay apply input data from unflattened

[frowand.list]

From: Frank Rowand <frank.rowand@sony.com>

Move duplicating and unflattening of an overlay flattened devicetree
(FDT) into the overlay application code.  To accomplish this,
of_overlay_apply() is replaced by of_overlay_fdt_apply().

The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
code, which is thus responsible for freeing the duplicate FDT.  The
caller of of_overlay_fdt_apply() remains responsible for freeing the
original FDT.

The unflattened devicetree now belongs to devicetree code, which is
thus responsible for freeing the unflattened devicetree.

These ownership changes prevent early freeing of the duplicated FDT
or the unflattened devicetree, which could result in use after free
errors.

These changes led to migrating some unittest overlay data into
their own devicetree source files, and then converting most of
them to use sugar syntax instead of hand coding fragments.

Changes from v6:
  - patches 2-5 were previously patches 1-4
  - patch 1/5 previously submitted separately as:
      "x86: devicetree: fix config option around x86_flattree_get_config()"
      https://lkml.org/lkml/2018/3/2/1148

Changes from v5:
  - Add __init to of_unittest_apply_revert_overlay_check(),
    of_unittest_overlay_5(), and of_unittest_overlay_11() to fix
    compile warnings reported by Geert

Changes from v4:
  (all in patch 1/4)
  - move of_overlay_fdt_apply() prototype from of_private.h to of.h
  - of_overlay_apply(): add kfree(fdt) if resolve phandles or
    initialize changeset fail
  - of_overlay_fdt_apply(): add parameter size of overlay_fdt and use
    it for some additional validation of the overlay fdt
  - update unittest.c with additional parameter to of_overlay_fdt_apply()

Changes from v3:
  - patch 1/4: OF_OVERLAY: add select OF_FLATTREE

Changes from v2:
  - improve error messages in patch 4/4, as suggested by Geert

Changes from v1:
  - rebase on v4.16-rc1
  - update documentation
  - split out error message to a separate patch


Frank Rowand (5):
  x86: devicetree: fix config option around x86_flattree_get_config()
  of: change overlay apply input data from unflattened to FDT
  of: Documentation: of_overlay_apply() replaced by
    of_overlay_fdt_apply()
  of: convert unittest overlay devicetree source to sugar syntax
  of: improve reporting invalid overlay target path

 Documentation/devicetree/overlay-notes.txt       |   4 +-
 arch/x86/kernel/devicetree.c                     |   2 +-
 drivers/of/Kconfig                               |   1 +
 drivers/of/overlay.c                             | 134 ++++++++--
 drivers/of/resolver.c                            |   6 -
 drivers/of/unittest-data/Makefile                |  28 ++-
 drivers/of/unittest-data/overlay.dts             | 101 ++++----
 drivers/of/unittest-data/overlay_0.dts           |  14 ++
 drivers/of/unittest-data/overlay_1.dts           |  14 ++
 drivers/of/unittest-data/overlay_10.dts          |  27 ++
 drivers/of/unittest-data/overlay_11.dts          |  28 +++
 drivers/of/unittest-data/overlay_12.dts          |  14 ++
 drivers/of/unittest-data/overlay_13.dts          |  14 ++
 drivers/of/unittest-data/overlay_15.dts          |  30 +++
 drivers/of/unittest-data/overlay_2.dts           |   9 +
 drivers/of/unittest-data/overlay_3.dts           |   9 +
 drivers/of/unittest-data/overlay_4.dts           |  18 ++
 drivers/of/unittest-data/overlay_5.dts           |   9 +
 drivers/of/unittest-data/overlay_6.dts           |  10 +
 drivers/of/unittest-data/overlay_7.dts           |  10 +
 drivers/of/unittest-data/overlay_8.dts           |  10 +
 drivers/of/unittest-data/overlay_9.dts           |  10 +
 drivers/of/unittest-data/overlay_bad_phandle.dts |  23 +-
 drivers/of/unittest-data/overlay_bad_symbol.dts  |  25 +-
 drivers/of/unittest-data/tests-overlay.dtsi      | 217 +---------------
 drivers/of/unittest.c                            | 300 +++++++++++------------
 include/linux/of.h                               |   6 +-
 27 files changed, 586 insertions(+), 487 deletions(-)
 create mode 100644 drivers/of/unittest-data/overlay_0.dts
 create mode 100644 drivers/of/unittest-data/overlay_1.dts
 create mode 100644 drivers/of/unittest-data/overlay_10.dts
 create mode 100644 drivers/of/unittest-data/overlay_11.dts
 create mode 100644 drivers/of/unittest-data/overlay_12.dts
 create mode 100644 drivers/of/unittest-data/overlay_13.dts
 create mode 100644 drivers/of/unittest-data/overlay_15.dts
 create mode 100644 drivers/of/unittest-data/overlay_2.dts
 create mode 100644 drivers/of/unittest-data/overlay_3.dts
 create mode 100644 drivers/of/unittest-data/overlay_4.dts
 create mode 100644 drivers/of/unittest-data/overlay_5.dts
 create mode 100644 drivers/of/unittest-data/overlay_6.dts
 create mode 100644 drivers/of/unittest-data/overlay_7.dts
 create mode 100644 drivers/of/unittest-data/overlay_8.dts
 create mode 100644 drivers/of/unittest-data/overlay_9.dts

-- 
Frank Rowand <frank.rowand@sony.com>

[PATCH v7 1/5] x86: devicetree: fix config option around x86_flattree_get_config()

[frowand.list]

From: Frank Rowand <frank.rowand@sony.com>

x86_flattree_get_config() is incorrectly protected by
ifdef CONFIG_OF_FLATTREE.  It uses of_get_flat_dt_size(), which
only exists if CONFIG_OF_EARLY_FLATTREE.  This issue has not
been exposed previously because OF_FLATTREE did not occur unless
it was selected by OF_EARLY_FLATTREE.  A devicetree overlay change
is selecting OF_FLATTREE directly instead of indirectly enabling
it by selecting OF_EARLY_FLATTREE.

This problem was exposed by a randconfig generated by the kbuild
test robot, where Platform OLPC was enabled.  OLPC selects
OF_PROMTREE instead of OF_EARLY_FLATREE.  The only other x86
platform that selects OF is X86_INTEL_CE, which does select
OF_EARLY_FLATTREE.

Signed-off-by: Frank Rowand <frank.rowand@sony.com>
---

x86 Maintainers, please ack this patch for Rob to accept

This patch previously submitted separately as:

  "x86: devicetree: fix config option around x86_flattree_get_config()"
  https://lkml.org/lkml/2018/3/2/1148

so this is a jump from version 1 to version 6.

No changes from version 1.


 arch/x86/kernel/devicetree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kernel/devicetree.c b/arch/x86/kernel/devicetree.c
index 25de5f6ca997..45416826f6ee 100644
--- a/arch/x86/kernel/devicetree.c
+++ b/arch/x86/kernel/devicetree.c
@@ -259,7 +259,7 @@ static void __init dtb_apic_setup(void)
 	dtb_ioapic_setup();
 }
 
-#ifdef CONFIG_OF_FLATTREE
+#ifdef CONFIG_OF_EARLY_FLATTREE
 static void __init x86_flattree_get_config(void)
 {
 	u32 size, map_len;
-- 
Frank Rowand <frank.rowand@sony.com>

Re: [PATCH v7 1/5] x86: devicetree: fix config option around x86_flattree_get_config()

[Frank Rowand]

On 03/03/18 16:17, frowand.list@gmail.com wrote:
> From: Frank Rowand <frank.rowand@sony.com>
> 
> x86_flattree_get_config() is incorrectly protected by
> ifdef CONFIG_OF_FLATTREE.  It uses of_get_flat_dt_size(), which
> only exists if CONFIG_OF_EARLY_FLATTREE.  This issue has not
> been exposed previously because OF_FLATTREE did not occur unless
> it was selected by OF_EARLY_FLATTREE.  A devicetree overlay change
> is selecting OF_FLATTREE directly instead of indirectly enabling
> it by selecting OF_EARLY_FLATTREE.
> 
> This problem was exposed by a randconfig generated by the kbuild
> test robot, where Platform OLPC was enabled.  OLPC selects
> OF_PROMTREE instead of OF_EARLY_FLATREE.  The only other x86
> platform that selects OF is X86_INTEL_CE, which does select
> OF_EARLY_FLATTREE.
> 
> Signed-off-by: Frank Rowand <frank.rowand@sony.com>
> ---
> 
> x86 Maintainers, please ack this patch for Rob to accept
> 
> This patch previously submitted separately as:
> 
>   "x86: devicetree: fix config option around x86_flattree_get_config()"
>   https://lkml.org/lkml/2018/3/2/1148
> 
> so this is a jump from version 1 to version 6.

                                      ^^^^^^^^^
                                      version 7

> 
> No changes from version 1.
> 
> 
>  arch/x86/kernel/devicetree.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kernel/devicetree.c b/arch/x86/kernel/devicetree.c
> index 25de5f6ca997..45416826f6ee 100644
> --- a/arch/x86/kernel/devicetree.c
> +++ b/arch/x86/kernel/devicetree.c
> @@ -259,7 +259,7 @@ static void __init dtb_apic_setup(void)
>  	dtb_ioapic_setup();
>  }
>  
> -#ifdef CONFIG_OF_FLATTREE
> +#ifdef CONFIG_OF_EARLY_FLATTREE
>  static void __init x86_flattree_get_config(void)
>  {
>  	u32 size, map_len;
> 

Re: [PATCH v7 1/5] x86: devicetree: fix config option around x86_flattree_get_config()

[Thomas Gleixner]

On Sat, 3 Mar 2018, Frank Rowand wrote:

> On 03/03/18 16:17, frowand.list@gmail.com wrote:
> > From: Frank Rowand <frank.rowand@sony.com>
> > 
> > x86_flattree_get_config() is incorrectly protected by
> > ifdef CONFIG_OF_FLATTREE.  It uses of_get_flat_dt_size(), which
> > only exists if CONFIG_OF_EARLY_FLATTREE.  This issue has not
> > been exposed previously because OF_FLATTREE did not occur unless
> > it was selected by OF_EARLY_FLATTREE.  A devicetree overlay change
> > is selecting OF_FLATTREE directly instead of indirectly enabling
> > it by selecting OF_EARLY_FLATTREE.
> > 
> > This problem was exposed by a randconfig generated by the kbuild
> > test robot, where Platform OLPC was enabled.  OLPC selects
> > OF_PROMTREE instead of OF_EARLY_FLATREE.  The only other x86
> > platform that selects OF is X86_INTEL_CE, which does select
> > OF_EARLY_FLATTREE.
> > 
> > Signed-off-by: Frank Rowand <frank.rowand@sony.com>
> > ---
> > 
> > x86 Maintainers, please ack this patch for Rob to accept

Acked-by: Thomas Gleixner <tglx@linutronix.de>

[PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[frowand.list]

From: Frank Rowand <frank.rowand@sony.com>

Move duplicating and unflattening of an overlay flattened devicetree
(FDT) into the overlay application code.  To accomplish this,
of_overlay_apply() is replaced by of_overlay_fdt_apply().

The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
code, which is thus responsible for freeing the duplicate FDT.  The
caller of of_overlay_fdt_apply() remains responsible for freeing the
original FDT.

The unflattened devicetree now belongs to devicetree code, which is
thus responsible for freeing the unflattened devicetree.

These ownership changes prevent early freeing of the duplicated FDT
or the unflattened devicetree, which could result in use after free
errors.

of_overlay_fdt_apply() is a private function for the anticipated
overlay loader.

Update unittest.c to use of_overlay_fdt_apply() instead of
of_overlay_apply().

Move overlay fragments from artificial locations in
drivers/of/unittest-data/tests-overlay.dtsi into one devicetree
source file per overlay.  This led to changes in
drivers/of/unitest-data/Makefile and drivers/of/unitest.c.

  - Add overlay directives to the overlay devicetree source files so
    that dtc will compile them as true overlays into one FDT data
    chunk per overlay.

  - Set CFLAGS for drivers/of/unittest-data/testcases.dts so that
    symbols will be generated for overlay resolution of overlays
    that are no longer artificially contained in testcases.dts

  - Unflatten and apply each unittest overlay FDT using
    of_overlay_fdt_apply().

  - Enable the of_resolve_phandles() check for whether the unflattened
    overlay is detached.  This check was previously disabled because the
    overlays from tests-overlay.dtsi were not unflattened into detached
    trees.

  - Other changes to unittest.c infrastructure to manage multiple test
    FDTs built into the kernel image (access by name instead of
    arbitrary number).

  - of_unittest_overlay_high_level(): previously unused code to add
    properties from the overlay_base devicetree to the live tree
    was triggered by the restructuring of tests-overlay.dtsi and thus
    testcases.dts.  This exposed two bugs: (1) the need to dup a
    property before adding it, and (2) property 'name' is
    auto-generated in the unflatten code and thus will be a duplicate
    in the __symbols__ node - do not treat this duplicate as an error.

Signed-off-by: Frank Rowand <frank.rowand@sony.com>
---

There are checkpatch warnings.  I have reviewed them and feel they
can be ignored.  They are "line over 80 characters" for either
pre-existing long lines, or lines in devicetree source files.

Rob, I have added making unittest a loadable module to my todo list.

Changes from v5:
  - Add __init to of_unittest_apply_revert_overlay_check(),
    of_unittest_overlay_5(), and of_unittest_overlay_11() to fix
    compile warnings reported by Geert

Changes from v4:
  - move of_overlay_fdt_apply() prototype from of_private.h to of.h
  - of_overlay_apply(): add kfree(fdt) if resolve phandles or
    initialize changeset fail
  - of_overlay_fdt_apply(): add parameter size of overlay_fdt and use
    it for some additional validation of the overlay fdt
  - update unittest.c with additional parameter to of_overlay_fdt_apply()

Changes from v3:
  - OF_OVERLAY: add select OF_FLATTREE

Changes from v1:
  - rebase on v4.16-rc1

 drivers/of/Kconfig                          |   1 +
 drivers/of/overlay.c                        | 112 ++++++++++-
 drivers/of/resolver.c                       |   6 -
 drivers/of/unittest-data/Makefile           |  28 ++-
 drivers/of/unittest-data/overlay_0.dts      |  14 ++
 drivers/of/unittest-data/overlay_1.dts      |  14 ++
 drivers/of/unittest-data/overlay_10.dts     |  34 ++++
 drivers/of/unittest-data/overlay_11.dts     |  34 ++++
 drivers/of/unittest-data/overlay_12.dts     |  14 ++
 drivers/of/unittest-data/overlay_13.dts     |  14 ++
 drivers/of/unittest-data/overlay_15.dts     |  35 ++++
 drivers/of/unittest-data/overlay_2.dts      |  14 ++
 drivers/of/unittest-data/overlay_3.dts      |  14 ++
 drivers/of/unittest-data/overlay_4.dts      |  23 +++
 drivers/of/unittest-data/overlay_5.dts      |  14 ++
 drivers/of/unittest-data/overlay_6.dts      |  15 ++
 drivers/of/unittest-data/overlay_7.dts      |  15 ++
 drivers/of/unittest-data/overlay_8.dts      |  15 ++
 drivers/of/unittest-data/overlay_9.dts      |  15 ++
 drivers/of/unittest-data/tests-overlay.dtsi | 213 --------------------
 drivers/of/unittest.c                       | 300 ++++++++++++++--------------
 include/linux/of.h                          |   6 +-
 22 files changed, 562 insertions(+), 388 deletions(-)
 create mode 100644 drivers/of/unittest-data/overlay_0.dts
 create mode 100644 drivers/of/unittest-data/overlay_1.dts
 create mode 100644 drivers/of/unittest-data/overlay_10.dts
 create mode 100644 drivers/of/unittest-data/overlay_11.dts
 create mode 100644 drivers/of/unittest-data/overlay_12.dts
 create mode 100644 drivers/of/unittest-data/overlay_13.dts
 create mode 100644 drivers/of/unittest-data/overlay_15.dts
 create mode 100644 drivers/of/unittest-data/overlay_2.dts
 create mode 100644 drivers/of/unittest-data/overlay_3.dts
 create mode 100644 drivers/of/unittest-data/overlay_4.dts
 create mode 100644 drivers/of/unittest-data/overlay_5.dts
 create mode 100644 drivers/of/unittest-data/overlay_6.dts
 create mode 100644 drivers/of/unittest-data/overlay_7.dts
 create mode 100644 drivers/of/unittest-data/overlay_8.dts
 create mode 100644 drivers/of/unittest-data/overlay_9.dts

diff --git a/drivers/of/Kconfig b/drivers/of/Kconfig
index 783e0870bd22..ad3fcad4d75b 100644
--- a/drivers/of/Kconfig
+++ b/drivers/of/Kconfig
@@ -92,6 +92,7 @@ config OF_RESOLVE
 config OF_OVERLAY
 	bool "Device Tree overlays"
 	select OF_DYNAMIC
+	select OF_FLATTREE
 	select OF_RESOLVE
 	help
 	  Overlays are a method to dynamically modify part of the kernel's
diff --git a/drivers/of/overlay.c b/drivers/of/overlay.c
index 3397d7642958..e3d7f69a8333 100644
--- a/drivers/of/overlay.c
+++ b/drivers/of/overlay.c
@@ -12,10 +12,12 @@
 #include <linux/module.h>
 #include <linux/of.h>
 #include <linux/of_device.h>
+#include <linux/of_fdt.h>
 #include <linux/string.h>
 #include <linux/ctype.h>
 #include <linux/errno.h>
 #include <linux/slab.h>
+#include <linux/libfdt.h>
 #include <linux/err.h>
 #include <linux/idr.h>
 
@@ -33,7 +35,9 @@ struct fragment {
 
 /**
  * struct overlay_changeset
+ * @id:			changeset identifier
  * @ovcs_list:		list on which we are located
+ * @fdt:		FDT that was unflattened to create @overlay_tree
  * @overlay_tree:	expanded device tree that contains the fragment nodes
  * @count:		count of fragment structures
  * @fragments:		fragment nodes in the overlay expanded device tree
@@ -43,6 +47,7 @@ struct fragment {
 struct overlay_changeset {
 	int id;
 	struct list_head ovcs_list;
+	const void *fdt;
 	struct device_node *overlay_tree;
 	int count;
 	struct fragment *fragments;
@@ -503,7 +508,8 @@ static struct device_node *find_target_node(struct device_node *info_node)
 
 /**
  * init_overlay_changeset() - initialize overlay changeset from overlay tree
- * @ovcs	Overlay changeset to build
+ * @ovcs:	Overlay changeset to build
+ * @fdt:	the FDT that was unflattened to create @tree
  * @tree:	Contains all the overlay fragments and overlay fixup nodes
  *
  * Initialize @ovcs.  Populate @ovcs->fragments with node information from
@@ -514,7 +520,7 @@ static struct device_node *find_target_node(struct device_node *info_node)
  * detected in @tree, or -ENOSPC if idr_alloc() error.
  */
 static int init_overlay_changeset(struct overlay_changeset *ovcs,
-		struct device_node *tree)
+		const void *fdt, struct device_node *tree)
 {
 	struct device_node *node, *overlay_node;
 	struct fragment *fragment;
@@ -535,6 +541,7 @@ static int init_overlay_changeset(struct overlay_changeset *ovcs,
 		pr_debug("%s() tree is not root\n", __func__);
 
 	ovcs->overlay_tree = tree;
+	ovcs->fdt = fdt;
 
 	INIT_LIST_HEAD(&ovcs->ovcs_list);
 
@@ -606,6 +613,7 @@ static int init_overlay_changeset(struct overlay_changeset *ovcs,
 	}
 
 	if (!cnt) {
+		pr_err("no fragments or symbols in overlay\n");
 		ret = -EINVAL;
 		goto err_free_fragments;
 	}
@@ -642,11 +650,24 @@ static void free_overlay_changeset(struct overlay_changeset *ovcs)
 	}
 	kfree(ovcs->fragments);
 
+	/*
+	 * TODO
+	 *
+	 * would like to: kfree(ovcs->overlay_tree);
+	 * but can not since drivers may have pointers into this data
+	 *
+	 * would like to: kfree(ovcs->fdt);
+	 * but can not since drivers may have pointers into this data
+	 */
+
 	kfree(ovcs);
 }
 
-/**
+/*
+ * internal documentation
+ *
  * of_overlay_apply() - Create and apply an overlay changeset
+ * @fdt:	the FDT that was unflattened to create @tree
  * @tree:	Expanded overlay device tree
  * @ovcs_id:	Pointer to overlay changeset id
  *
@@ -685,21 +706,29 @@ static void free_overlay_changeset(struct overlay_changeset *ovcs)
  * id is returned to *ovcs_id.
  */
 
-int of_overlay_apply(struct device_node *tree, int *ovcs_id)
+static int of_overlay_apply(const void *fdt, struct device_node *tree,
+		int *ovcs_id)
 {
 	struct overlay_changeset *ovcs;
 	int ret = 0, ret_revert, ret_tmp;
 
-	*ovcs_id = 0;
+	/*
+	 * As of this point, fdt and tree belong to the overlay changeset.
+	 * overlay changeset code is responsible for freeing them.
+	 */
 
 	if (devicetree_corrupt()) {
 		pr_err("devicetree state suspect, refuse to apply overlay\n");
+		kfree(fdt);
+		kfree(tree);
 		ret = -EBUSY;
 		goto out;
 	}
 
 	ovcs = kzalloc(sizeof(*ovcs), GFP_KERNEL);
 	if (!ovcs) {
+		kfree(fdt);
+		kfree(tree);
 		ret = -ENOMEM;
 		goto out;
 	}
@@ -709,12 +738,17 @@ int of_overlay_apply(struct device_node *tree, int *ovcs_id)
 
 	ret = of_resolve_phandles(tree);
 	if (ret)
-		goto err_free_overlay_changeset;
+		goto err_free_tree;
 
-	ret = init_overlay_changeset(ovcs, tree);
+	ret = init_overlay_changeset(ovcs, fdt, tree);
 	if (ret)
-		goto err_free_overlay_changeset;
+		goto err_free_tree;
 
+	/*
+	 * after overlay_notify(), ovcs->overlay_tree related pointers may have
+	 * leaked to drivers, so can not kfree() tree, aka ovcs->overlay_tree;
+	 * and can not free fdt, aka ovcs->fdt
+	 */
 	ret = overlay_notify(ovcs, OF_OVERLAY_PRE_APPLY);
 	if (ret) {
 		pr_err("overlay changeset pre-apply notify error %d\n", ret);
@@ -754,6 +788,10 @@ int of_overlay_apply(struct device_node *tree, int *ovcs_id)
 
 	goto out_unlock;
 
+err_free_tree:
+	kfree(fdt);
+	kfree(tree);
+
 err_free_overlay_changeset:
 	free_overlay_changeset(ovcs);
 
@@ -766,7 +804,63 @@ int of_overlay_apply(struct device_node *tree, int *ovcs_id)
 
 	return ret;
 }
-EXPORT_SYMBOL_GPL(of_overlay_apply);
+
+int of_overlay_fdt_apply(const void *overlay_fdt, u32 overlay_fdt_size,
+			 int *ovcs_id)
+{
+	const void *new_fdt;
+	int ret;
+	u32 size;
+	struct device_node *overlay_root;
+
+	*ovcs_id = 0;
+	ret = 0;
+
+	if (overlay_fdt_size < sizeof(struct fdt_header) ||
+	    fdt_check_header(overlay_fdt)) {
+		pr_err("Invalid overlay_fdt header\n");
+		return -EINVAL;
+	}
+
+	size = fdt_totalsize(overlay_fdt);
+	if (overlay_fdt_size < size)
+		return -EINVAL;
+
+	/*
+	 * Must create permanent copy of FDT because of_fdt_unflatten_tree()
+	 * will create pointers to the passed in FDT in the unflattened tree.
+	 */
+	new_fdt = kmemdup(overlay_fdt, size, GFP_KERNEL);
+	if (!new_fdt)
+		return -ENOMEM;
+
+	of_fdt_unflatten_tree(new_fdt, NULL, &overlay_root);
+	if (!overlay_root) {
+		pr_err("unable to unflatten overlay_fdt\n");
+		ret = -EINVAL;
+		goto out_free_new_fdt;
+	}
+
+	ret = of_overlay_apply(new_fdt, overlay_root, ovcs_id);
+	if (ret < 0) {
+		/*
+		 * new_fdt and overlay_root now belong to the overlay
+		 * changeset.
+		 * overlay changeset code is responsible for freeing them.
+		 */
+		goto out;
+	}
+
+	return 0;
+
+
+out_free_new_fdt:
+	kfree(new_fdt);
+
+out:
+	return ret;
+}
+EXPORT_SYMBOL_GPL(of_overlay_fdt_apply);
 
 /*
  * Find @np in @tree.
diff --git a/drivers/of/resolver.c b/drivers/of/resolver.c
index 740d19bde601..b2f645187213 100644
--- a/drivers/of/resolver.c
+++ b/drivers/of/resolver.c
@@ -269,17 +269,11 @@ int of_resolve_phandles(struct device_node *overlay)
 		goto out;
 	}
 
-#if 0
-	Temporarily disable check so that old style overlay unittests
-	do not fail when of_resolve_phandles() is moved into
-	of_overlay_apply().
-
 	if (!of_node_check_flag(overlay, OF_DETACHED)) {
 		pr_err("overlay not detached\n");
 		err = -EINVAL;
 		goto out;
 	}
-#endif
 
 	phandle_delta = live_tree_max_phandle() + 1;
 	adjust_overlay_phandles(overlay, phandle_delta);
diff --git a/drivers/of/unittest-data/Makefile b/drivers/of/unittest-data/Makefile
index df697976740a..8fd0ea4b92b0 100644
--- a/drivers/of/unittest-data/Makefile
+++ b/drivers/of/unittest-data/Makefile
@@ -1,8 +1,22 @@
 # SPDX-License-Identifier: GPL-2.0
-DTC_FLAGS_testcases := -Wno-interrupts_property
 obj-y += testcases.dtb.o
 
 obj-$(CONFIG_OF_OVERLAY) += overlay.dtb.o \
+			    overlay_0.dtb.o \
+			    overlay_1.dtb.o \
+			    overlay_2.dtb.o \
+			    overlay_3.dtb.o \
+			    overlay_4.dtb.o \
+			    overlay_5.dtb.o \
+			    overlay_6.dtb.o \
+			    overlay_7.dtb.o \
+			    overlay_8.dtb.o \
+			    overlay_9.dtb.o \
+			    overlay_10.dtb.o \
+			    overlay_11.dtb.o \
+			    overlay_12.dtb.o \
+			    overlay_13.dtb.o \
+			    overlay_15.dtb.o \
 			    overlay_bad_phandle.dtb.o \
 			    overlay_bad_symbol.dtb.o \
 			    overlay_base.dtb.o
@@ -10,10 +24,14 @@ obj-$(CONFIG_OF_OVERLAY) += overlay.dtb.o \
 targets += $(foreach suffix, dtb dtb.S, $(patsubst %.dtb.o,%.$(suffix),$(obj-y)))
 
 # enable creation of __symbols__ node
-DTC_FLAGS_overlay := -@
-DTC_FLAGS_overlay_bad_phandle := -@
-DTC_FLAGS_overlay_bad_symbol := -@
-DTC_FLAGS_overlay_base := -@
+DTC_FLAGS_overlay += -@
+DTC_FLAGS_overlay_bad_phandle += -@
+DTC_FLAGS_overlay_bad_symbol += -@
+DTC_FLAGS_overlay_base += -@
+DTC_FLAGS_testcases += -@
+
+# suppress warnings about intentional errors
+DTC_FLAGS_testcases += -Wno-interrupts_property
 
 .PRECIOUS: \
 	$(obj)/%.dtb.S \
diff --git a/drivers/of/unittest-data/overlay_0.dts b/drivers/of/unittest-data/overlay_0.dts
new file mode 100644
index 000000000000..ac0f9e0fe65f
--- /dev/null
+++ b/drivers/of/unittest-data/overlay_0.dts
@@ -0,0 +1,14 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/ {
+	/* overlay_0 - enable using absolute target path */
+
+	fragment@0 {
+		target-path = "/testcase-data/overlay-node/test-bus/test-unittest0";
+		__overlay__ {
+			status = "okay";
+		};
+	};
+};
diff --git a/drivers/of/unittest-data/overlay_1.dts b/drivers/of/unittest-data/overlay_1.dts
new file mode 100644
index 000000000000..e92a626e2948
--- /dev/null
+++ b/drivers/of/unittest-data/overlay_1.dts
@@ -0,0 +1,14 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/ {
+	/* overlay_1 - disable using absolute target path */
+
+	fragment@0 {
+		target-path = "/testcase-data/overlay-node/test-bus/test-unittest1";
+		__overlay__ {
+			status = "disabled";
+		};
+	};
+};
diff --git a/drivers/of/unittest-data/overlay_10.dts b/drivers/of/unittest-data/overlay_10.dts
new file mode 100644
index 000000000000..445925a10cd3
--- /dev/null
+++ b/drivers/of/unittest-data/overlay_10.dts
@@ -0,0 +1,34 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/ {
+	/* overlay_10 */
+	/* overlays 8, 9, 10, 11 application and removal in bad sequence */
+
+	fragment@0 {
+		target-path = "/testcase-data/overlay-node/test-bus";
+		__overlay__ {
+
+			/* suppress DTC warning */
+			#address-cells = <1>;
+			#size-cells = <0>;
+
+			test-unittest10 {
+				compatible = "unittest";
+				status = "okay";
+				reg = <10>;
+
+				#address-cells = <1>;
+				#size-cells = <0>;
+
+				test-unittest101 {
+					compatible = "unittest";
+					status = "okay";
+					reg = <1>;
+				};
+
+			};
+		};
+	};
+};
diff --git a/drivers/of/unittest-data/overlay_11.dts b/drivers/of/unittest-data/overlay_11.dts
new file mode 100644
index 000000000000..c1d14f34359e
--- /dev/null
+++ b/drivers/of/unittest-data/overlay_11.dts
@@ -0,0 +1,34 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/ {
+	/* overlay_11 */
+	/* overlays 8, 9, 10, 11 application and removal in bad sequence */
+
+	fragment@0 {
+		target-path = "/testcase-data/overlay-node/test-bus";
+		__overlay__ {
+
+			/* suppress DTC warning */
+			#address-cells = <1>;
+			#size-cells = <0>;
+
+			test-unittest11 {
+				compatible = "unittest";
+				status = "okay";
+				reg = <11>;
+
+				#address-cells = <1>;
+				#size-cells = <0>;
+
+				test-unittest111 {
+					compatible = "unittest";
+					status = "okay";
+					reg = <1>;
+				};
+
+			};
+		};
+	};
+};
diff --git a/drivers/of/unittest-data/overlay_12.dts b/drivers/of/unittest-data/overlay_12.dts
new file mode 100644
index 000000000000..ca3441e2cbec
--- /dev/null
+++ b/drivers/of/unittest-data/overlay_12.dts
@@ -0,0 +1,14 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/ {
+	/* overlay_12 - enable using absolute target path (i2c) */
+
+	fragment@0 {
+		target-path = "/testcase-data/overlay-node/test-bus/i2c-test-bus/test-unittest12";
+		__overlay__ {
+			status = "okay";
+		};
+	};
+};
diff --git a/drivers/of/unittest-data/overlay_13.dts b/drivers/of/unittest-data/overlay_13.dts
new file mode 100644
index 000000000000..3c30dec63894
--- /dev/null
+++ b/drivers/of/unittest-data/overlay_13.dts
@@ -0,0 +1,14 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/ {
+	/* overlay_13 - disable using absolute target path (i2c) */
+
+	fragment@0 {
+		target-path = "/testcase-data/overlay-node/test-bus/i2c-test-bus/test-unittest13";
+		__overlay__ {
+			status = "disabled";
+		};
+	};
+};
diff --git a/drivers/of/unittest-data/overlay_15.dts b/drivers/of/unittest-data/overlay_15.dts
new file mode 100644
index 000000000000..44e44c62b739
--- /dev/null
+++ b/drivers/of/unittest-data/overlay_15.dts
@@ -0,0 +1,35 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/ {
+	/* overlay_15 - mux overlay */
+
+	fragment@0 {
+		target-path = "/testcase-data/overlay-node/test-bus/i2c-test-bus";
+		__overlay__ {
+			#address-cells = <1>;
+			#size-cells = <0>;
+			test-unittest15 {
+				reg = <11>;
+				compatible = "unittest-i2c-mux";
+				status = "okay";
+
+				#address-cells = <1>;
+				#size-cells = <0>;
+
+				i2c@0 {
+					#address-cells = <1>;
+					#size-cells = <0>;
+					reg = <0>;
+
+					test-mux-dev {
+						reg = <32>;
+						compatible = "unittest-i2c-dev";
+						status = "okay";
+					};
+				};
+			};
+		};
+	};
+};
diff --git a/drivers/of/unittest-data/overlay_2.dts b/drivers/of/unittest-data/overlay_2.dts
new file mode 100644
index 000000000000..cf1e4245b7ce
--- /dev/null
+++ b/drivers/of/unittest-data/overlay_2.dts
@@ -0,0 +1,14 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/ {
+	/* overlay_2 -  enable using label */
+
+	fragment@0 {
+		target = <&unittest2>;
+		__overlay__ {
+			status = "okay";
+		};
+	};
+};
diff --git a/drivers/of/unittest-data/overlay_3.dts b/drivers/of/unittest-data/overlay_3.dts
new file mode 100644
index 000000000000..158dc44fc20a
--- /dev/null
+++ b/drivers/of/unittest-data/overlay_3.dts
@@ -0,0 +1,14 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/ {
+	/* overlay_3 - disable using label */
+
+	fragment@0 {
+		target = <&unittest3>;
+		__overlay__ {
+			status = "disabled";
+		};
+	};
+};
diff --git a/drivers/of/unittest-data/overlay_4.dts b/drivers/of/unittest-data/overlay_4.dts
new file mode 100644
index 000000000000..b4a2e6c6b016
--- /dev/null
+++ b/drivers/of/unittest-data/overlay_4.dts
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/ {
+	/* overlay_4 - test insertion of a full node */
+
+	fragment@0 {
+		target = <&unittestbus>;
+		__overlay__ {
+
+			/* suppress DTC warning */
+			#address-cells = <1>;
+			#size-cells = <0>;
+
+			test-unittest4 {
+				compatible = "unittest";
+				status = "okay";
+				reg = <4>;
+			};
+		};
+	};
+};
diff --git a/drivers/of/unittest-data/overlay_5.dts b/drivers/of/unittest-data/overlay_5.dts
new file mode 100644
index 000000000000..02ad25c1f19c
--- /dev/null
+++ b/drivers/of/unittest-data/overlay_5.dts
@@ -0,0 +1,14 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/ {
+	/* overlay_5 - test overlay apply revert */
+
+	fragment@0 {
+		target-path = "/testcase-data/overlay-node/test-bus/test-unittest5";
+		__overlay__ {
+			status = "okay";
+		};
+	};
+};
diff --git a/drivers/of/unittest-data/overlay_6.dts b/drivers/of/unittest-data/overlay_6.dts
new file mode 100644
index 000000000000..a14e965f5497
--- /dev/null
+++ b/drivers/of/unittest-data/overlay_6.dts
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/ {
+	/* overlay_6 */
+	/* overlays 6, 7 application and removal in sequence */
+
+	fragment@0 {
+		target-path = "/testcase-data/overlay-node/test-bus/test-unittest6";
+		__overlay__ {
+			status = "okay";
+		};
+	};
+};
diff --git a/drivers/of/unittest-data/overlay_7.dts b/drivers/of/unittest-data/overlay_7.dts
new file mode 100644
index 000000000000..4bd7e423209c
--- /dev/null
+++ b/drivers/of/unittest-data/overlay_7.dts
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/ {
+	/* overlay_7 */
+	/* overlays 6, 7 application and removal in sequence */
+
+	fragment@0 {
+		target-path = "/testcase-data/overlay-node/test-bus/test-unittest7";
+		__overlay__ {
+			status = "okay";
+		};
+	};
+};
diff --git a/drivers/of/unittest-data/overlay_8.dts b/drivers/of/unittest-data/overlay_8.dts
new file mode 100644
index 000000000000..5b21c53945a9
--- /dev/null
+++ b/drivers/of/unittest-data/overlay_8.dts
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/ {
+	/* overlay_8 */
+	/* overlays 8, 9, 10, 11 application and removal in bad sequence */
+
+	fragment@0 {
+		target-path = "/testcase-data/overlay-node/test-bus/test-unittest8";
+		__overlay__ {
+			status = "okay";
+		};
+	};
+};
diff --git a/drivers/of/unittest-data/overlay_9.dts b/drivers/of/unittest-data/overlay_9.dts
new file mode 100644
index 000000000000..20ff055a5349
--- /dev/null
+++ b/drivers/of/unittest-data/overlay_9.dts
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/ {
+	/* overlay_9 */
+	/* overlays 8, 9, 10, 11 application and removal in bad sequence */
+
+	fragment@0 {
+		target-path = "/testcase-data/overlay-node/test-bus/test-unittest8";
+		__overlay__ {
+			property-foo = "bar";
+		};
+	};
+};
diff --git a/drivers/of/unittest-data/tests-overlay.dtsi b/drivers/of/unittest-data/tests-overlay.dtsi
index 7b8001ab9f3a..fa2fb43bccac 100644
--- a/drivers/of/unittest-data/tests-overlay.dtsi
+++ b/drivers/of/unittest-data/tests-overlay.dtsi
@@ -113,218 +113,5 @@
 				};
 			};
 		};
-
-		/* test enable using absolute target path */
-		overlay0 {
-			fragment@0 {
-				target-path = "/testcase-data/overlay-node/test-bus/test-unittest0";
-				__overlay__ {
-					status = "okay";
-				};
-			};
-		};
-
-		/* test disable using absolute target path */
-		overlay1 {
-			fragment@0 {
-				target-path = "/testcase-data/overlay-node/test-bus/test-unittest1";
-				__overlay__ {
-					status = "disabled";
-				};
-			};
-		};
-
-		/* test enable using label */
-		overlay2 {
-			fragment@0 {
-				target = <&unittest2>;
-				__overlay__ {
-					status = "okay";
-				};
-			};
-		};
-
-		/* test disable using label */
-		overlay3 {
-			fragment@0 {
-				target = <&unittest3>;
-				__overlay__ {
-					status = "disabled";
-				};
-			};
-		};
-
-		/* test insertion of a full node */
-		overlay4 {
-			fragment@0 {
-				target = <&unittestbus>;
-				__overlay__ {
-
-					/* suppress DTC warning */
-					#address-cells = <1>;
-					#size-cells = <0>;
-
-					test-unittest4 {
-						compatible = "unittest";
-						status = "okay";
-						reg = <4>;
-					};
-				};
-			};
-		};
-
-		/* test overlay apply revert */
-		overlay5 {
-			fragment@0 {
-				target-path = "/testcase-data/overlay-node/test-bus/test-unittest5";
-				__overlay__ {
-					status = "okay";
-				};
-			};
-		};
-
-		/* test overlays application and removal in sequence */
-		overlay6 {
-			fragment@0 {
-				target-path = "/testcase-data/overlay-node/test-bus/test-unittest6";
-				__overlay__ {
-					status = "okay";
-				};
-			};
-		};
-		overlay7 {
-			fragment@0 {
-				target-path = "/testcase-data/overlay-node/test-bus/test-unittest7";
-				__overlay__ {
-					status = "okay";
-				};
-			};
-		};
-
-		/* test overlays application and removal in bad sequence */
-		overlay8 {
-			fragment@0 {
-				target-path = "/testcase-data/overlay-node/test-bus/test-unittest8";
-				__overlay__ {
-					status = "okay";
-				};
-			};
-		};
-		overlay9 {
-			fragment@0 {
-				target-path = "/testcase-data/overlay-node/test-bus/test-unittest8";
-				__overlay__ {
-					property-foo = "bar";
-				};
-			};
-		};
-
-		overlay10 {
-			fragment@0 {
-				target-path = "/testcase-data/overlay-node/test-bus";
-				__overlay__ {
-
-					/* suppress DTC warning */
-					#address-cells = <1>;
-					#size-cells = <0>;
-
-					test-unittest10 {
-						compatible = "unittest";
-						status = "okay";
-						reg = <10>;
-
-						#address-cells = <1>;
-						#size-cells = <0>;
-
-						test-unittest101 {
-							compatible = "unittest";
-							status = "okay";
-							reg = <1>;
-						};
-
-					};
-				};
-			};
-		};
-
-		overlay11 {
-			fragment@0 {
-				target-path = "/testcase-data/overlay-node/test-bus";
-				__overlay__ {
-
-					/* suppress DTC warning */
-					#address-cells = <1>;
-					#size-cells = <0>;
-
-					test-unittest11 {
-						compatible = "unittest";
-						status = "okay";
-						reg = <11>;
-
-						#address-cells = <1>;
-						#size-cells = <0>;
-
-						test-unittest111 {
-							compatible = "unittest";
-							status = "okay";
-							reg = <1>;
-						};
-
-					};
-				};
-			};
-		};
-
-		/* test enable using absolute target path (i2c) */
-		overlay12 {
-			fragment@0 {
-				target-path = "/testcase-data/overlay-node/test-bus/i2c-test-bus/test-unittest12";
-				__overlay__ {
-					status = "okay";
-				};
-			};
-		};
-
-		/* test disable using absolute target path (i2c) */
-		overlay13 {
-			fragment@0 {
-				target-path = "/testcase-data/overlay-node/test-bus/i2c-test-bus/test-unittest13";
-				__overlay__ {
-					status = "disabled";
-				};
-			};
-		};
-
-		/* test mux overlay */
-		overlay15 {
-			fragment@0 {
-				target-path = "/testcase-data/overlay-node/test-bus/i2c-test-bus";
-				__overlay__ {
-					#address-cells = <1>;
-					#size-cells = <0>;
-					test-unittest15 {
-						reg = <11>;
-						compatible = "unittest-i2c-mux";
-						status = "okay";
-
-						#address-cells = <1>;
-						#size-cells = <0>;
-
-						i2c@0 {
-							#address-cells = <1>;
-							#size-cells = <0>;
-							reg = <0>;
-
-							test-mux-dev {
-								reg = <32>;
-								compatible = "unittest-i2c-dev";
-								status = "okay";
-							};
-						};
-					};
-				};
-			};
-		};
-
 	};
 };
diff --git a/drivers/of/unittest.c b/drivers/of/unittest.c
index 7a9abaae874d..a23b54780c7d 100644
--- a/drivers/of/unittest.c
+++ b/drivers/of/unittest.c
@@ -45,6 +45,8 @@
 	failed; \
 })
 
+static int __init overlay_data_apply(const char *overlay_name, int *overlay_id);
+
 static void __init of_unittest_find_node_by_name(void)
 {
 	struct device_node *np;
@@ -997,8 +999,7 @@ static int __init unittest_data_add(void)
 	}
 
 	/*
-	 * This lock normally encloses of_overlay_apply() as well as
-	 * of_resolve_phandles().
+	 * This lock normally encloses of_resolve_phandles()
 	 */
 	of_overlay_mutex_lock();
 
@@ -1191,12 +1192,12 @@ static int of_unittest_device_exists(int unittest_nr, enum overlay_type ovtype)
 	return 0;
 }
 
-static const char *overlay_path(int nr)
+static const char *overlay_name_from_nr(int nr)
 {
 	static char buf[256];
 
 	snprintf(buf, sizeof(buf) - 1,
-		"/testcase-data/overlay%d", nr);
+		"overlay_%d", nr);
 	buf[sizeof(buf) - 1] = '\0';
 
 	return buf;
@@ -1263,25 +1264,19 @@ static void of_unittest_destroy_tracked_overlays(void)
 	} while (defers > 0);
 }
 
-static int of_unittest_apply_overlay(int overlay_nr, int unittest_nr,
+static int __init of_unittest_apply_overlay(int overlay_nr, int unittest_nr,
 		int *overlay_id)
 {
 	struct device_node *np = NULL;
+	const char *overlay_name;
 	int ret;
 
-	np = of_find_node_by_path(overlay_path(overlay_nr));
-	if (np == NULL) {
-		unittest(0, "could not find overlay node @\"%s\"\n",
-				overlay_path(overlay_nr));
-		ret = -EINVAL;
-		goto out;
-	}
+	overlay_name = overlay_name_from_nr(overlay_nr);
 
-	*overlay_id = 0;
-	ret = of_overlay_apply(np, overlay_id);
-	if (ret < 0) {
-		unittest(0, "could not create overlay from \"%s\"\n",
-				overlay_path(overlay_nr));
+	ret = overlay_data_apply(overlay_name, overlay_id);
+	if (!ret) {
+		unittest(0, "could not apply overlay \"%s\"\n",
+				overlay_name);
 		goto out;
 	}
 	of_unittest_track_overlay(*overlay_id);
@@ -1295,15 +1290,16 @@ static int of_unittest_apply_overlay(int overlay_nr, int unittest_nr,
 }
 
 /* apply an overlay while checking before and after states */
-static int of_unittest_apply_overlay_check(int overlay_nr, int unittest_nr,
-		int before, int after, enum overlay_type ovtype)
+static int __init of_unittest_apply_overlay_check(int overlay_nr,
+		int unittest_nr, int before, int after,
+		enum overlay_type ovtype)
 {
 	int ret, ovcs_id;
 
 	/* unittest device must not be in before state */
 	if (of_unittest_device_exists(unittest_nr, ovtype) != before) {
-		unittest(0, "overlay @\"%s\" with device @\"%s\" %s\n",
-				overlay_path(overlay_nr),
+		unittest(0, "%s with device @\"%s\" %s\n",
+				overlay_name_from_nr(overlay_nr),
 				unittest_path(unittest_nr, ovtype),
 				!before ? "enabled" : "disabled");
 		return -EINVAL;
@@ -1318,8 +1314,8 @@ static int of_unittest_apply_overlay_check(int overlay_nr, int unittest_nr,
 
 	/* unittest device must be to set to after state */
 	if (of_unittest_device_exists(unittest_nr, ovtype) != after) {
-		unittest(0, "overlay @\"%s\" failed to create @\"%s\" %s\n",
-				overlay_path(overlay_nr),
+		unittest(0, "%s failed to create @\"%s\" %s\n",
+				overlay_name_from_nr(overlay_nr),
 				unittest_path(unittest_nr, ovtype),
 				!after ? "enabled" : "disabled");
 		return -EINVAL;
@@ -1329,7 +1325,7 @@ static int of_unittest_apply_overlay_check(int overlay_nr, int unittest_nr,
 }
 
 /* apply an overlay and then revert it while checking before, after states */
-static int of_unittest_apply_revert_overlay_check(int overlay_nr,
+static int __init of_unittest_apply_revert_overlay_check(int overlay_nr,
 		int unittest_nr, int before, int after,
 		enum overlay_type ovtype)
 {
@@ -1337,8 +1333,8 @@ static int of_unittest_apply_revert_overlay_check(int overlay_nr,
 
 	/* unittest device must be in before state */
 	if (of_unittest_device_exists(unittest_nr, ovtype) != before) {
-		unittest(0, "overlay @\"%s\" with device @\"%s\" %s\n",
-				overlay_path(overlay_nr),
+		unittest(0, "%s with device @\"%s\" %s\n",
+				overlay_name_from_nr(overlay_nr),
 				unittest_path(unittest_nr, ovtype),
 				!before ? "enabled" : "disabled");
 		return -EINVAL;
@@ -1354,8 +1350,8 @@ static int of_unittest_apply_revert_overlay_check(int overlay_nr,
 
 	/* unittest device must be in after state */
 	if (of_unittest_device_exists(unittest_nr, ovtype) != after) {
-		unittest(0, "overlay @\"%s\" failed to create @\"%s\" %s\n",
-				overlay_path(overlay_nr),
+		unittest(0, "%s failed to create @\"%s\" %s\n",
+				overlay_name_from_nr(overlay_nr),
 				unittest_path(unittest_nr, ovtype),
 				!after ? "enabled" : "disabled");
 		return -EINVAL;
@@ -1363,16 +1359,16 @@ static int of_unittest_apply_revert_overlay_check(int overlay_nr,
 
 	ret = of_overlay_remove(&ovcs_id);
 	if (ret != 0) {
-		unittest(0, "overlay @\"%s\" failed to be destroyed @\"%s\"\n",
-				overlay_path(overlay_nr),
+		unittest(0, "%s failed to be destroyed @\"%s\"\n",
+				overlay_name_from_nr(overlay_nr),
 				unittest_path(unittest_nr, ovtype));
 		return ret;
 	}
 
 	/* unittest device must be again in before state */
 	if (of_unittest_device_exists(unittest_nr, PDEV_OVERLAY) != before) {
-		unittest(0, "overlay @\"%s\" with device @\"%s\" %s\n",
-				overlay_path(overlay_nr),
+		unittest(0, "%s with device @\"%s\" %s\n",
+				overlay_name_from_nr(overlay_nr),
 				unittest_path(unittest_nr, ovtype),
 				!before ? "enabled" : "disabled");
 		return -EINVAL;
@@ -1382,7 +1378,7 @@ static int of_unittest_apply_revert_overlay_check(int overlay_nr,
 }
 
 /* test activation of device */
-static void of_unittest_overlay_0(void)
+static void __init of_unittest_overlay_0(void)
 {
 	int ret;
 
@@ -1395,7 +1391,7 @@ static void of_unittest_overlay_0(void)
 }
 
 /* test deactivation of device */
-static void of_unittest_overlay_1(void)
+static void __init of_unittest_overlay_1(void)
 {
 	int ret;
 
@@ -1408,7 +1404,7 @@ static void of_unittest_overlay_1(void)
 }
 
 /* test activation of device */
-static void of_unittest_overlay_2(void)
+static void __init of_unittest_overlay_2(void)
 {
 	int ret;
 
@@ -1421,7 +1417,7 @@ static void of_unittest_overlay_2(void)
 }
 
 /* test deactivation of device */
-static void of_unittest_overlay_3(void)
+static void __init of_unittest_overlay_3(void)
 {
 	int ret;
 
@@ -1434,7 +1430,7 @@ static void of_unittest_overlay_3(void)
 }
 
 /* test activation of a full device node */
-static void of_unittest_overlay_4(void)
+static void __init of_unittest_overlay_4(void)
 {
 	int ret;
 
@@ -1447,7 +1443,7 @@ static void of_unittest_overlay_4(void)
 }
 
 /* test overlay apply/revert sequence */
-static void of_unittest_overlay_5(void)
+static void __init of_unittest_overlay_5(void)
 {
 	int ret;
 
@@ -1460,19 +1456,19 @@ static void of_unittest_overlay_5(void)
 }
 
 /* test overlay application in sequence */
-static void of_unittest_overlay_6(void)
+static void __init of_unittest_overlay_6(void)
 {
-	struct device_node *np;
 	int ret, i, ov_id[2], ovcs_id;
 	int overlay_nr = 6, unittest_nr = 6;
 	int before = 0, after = 1;
+	const char *overlay_name;
 
 	/* unittest device must be in before state */
 	for (i = 0; i < 2; i++) {
 		if (of_unittest_device_exists(unittest_nr + i, PDEV_OVERLAY)
 				!= before) {
-			unittest(0, "overlay @\"%s\" with device @\"%s\" %s\n",
-					overlay_path(overlay_nr + i),
+			unittest(0, "%s with device @\"%s\" %s\n",
+					overlay_name_from_nr(overlay_nr + i),
 					unittest_path(unittest_nr + i,
 						PDEV_OVERLAY),
 					!before ? "enabled" : "disabled");
@@ -1483,18 +1479,12 @@ static void of_unittest_overlay_6(void)
 	/* apply the overlays */
 	for (i = 0; i < 2; i++) {
 
-		np = of_find_node_by_path(overlay_path(overlay_nr + i));
-		if (np == NULL) {
-			unittest(0, "could not find overlay node @\"%s\"\n",
-					overlay_path(overlay_nr + i));
-			return;
-		}
+		overlay_name = overlay_name_from_nr(overlay_nr + i);
 
-		ovcs_id = 0;
-		ret = of_overlay_apply(np, &ovcs_id);
-		if (ret < 0)  {
-			unittest(0, "could not create overlay from \"%s\"\n",
-					overlay_path(overlay_nr + i));
+		ret = overlay_data_apply(overlay_name, &ovcs_id);
+		if (!ret)  {
+			unittest(0, "could not apply overlay \"%s\"\n",
+					overlay_name);
 			return;
 		}
 		ov_id[i] = ovcs_id;
@@ -1506,7 +1496,7 @@ static void of_unittest_overlay_6(void)
 		if (of_unittest_device_exists(unittest_nr + i, PDEV_OVERLAY)
 				!= after) {
 			unittest(0, "overlay @\"%s\" failed @\"%s\" %s\n",
-					overlay_path(overlay_nr + i),
+					overlay_name_from_nr(overlay_nr + i),
 					unittest_path(unittest_nr + i,
 						PDEV_OVERLAY),
 					!after ? "enabled" : "disabled");
@@ -1518,8 +1508,8 @@ static void of_unittest_overlay_6(void)
 		ovcs_id = ov_id[i];
 		ret = of_overlay_remove(&ovcs_id);
 		if (ret != 0) {
-			unittest(0, "overlay @\"%s\" failed destroy @\"%s\"\n",
-					overlay_path(overlay_nr + i),
+			unittest(0, "%s failed destroy @\"%s\"\n",
+					overlay_name_from_nr(overlay_nr + i),
 					unittest_path(unittest_nr + i,
 						PDEV_OVERLAY));
 			return;
@@ -1531,8 +1521,8 @@ static void of_unittest_overlay_6(void)
 		/* unittest device must be again in before state */
 		if (of_unittest_device_exists(unittest_nr + i, PDEV_OVERLAY)
 				!= before) {
-			unittest(0, "overlay @\"%s\" with device @\"%s\" %s\n",
-					overlay_path(overlay_nr + i),
+			unittest(0, "%s with device @\"%s\" %s\n",
+					overlay_name_from_nr(overlay_nr + i),
 					unittest_path(unittest_nr + i,
 						PDEV_OVERLAY),
 					!before ? "enabled" : "disabled");
@@ -1544,29 +1534,23 @@ static void of_unittest_overlay_6(void)
 }
 
 /* test overlay application in sequence */
-static void of_unittest_overlay_8(void)
+static void __init of_unittest_overlay_8(void)
 {
-	struct device_node *np;
 	int ret, i, ov_id[2], ovcs_id;
 	int overlay_nr = 8, unittest_nr = 8;
+	const char *overlay_name;
 
 	/* we don't care about device state in this test */
 
 	/* apply the overlays */
 	for (i = 0; i < 2; i++) {
 
-		np = of_find_node_by_path(overlay_path(overlay_nr + i));
-		if (np == NULL) {
-			unittest(0, "could not find overlay node @\"%s\"\n",
-					overlay_path(overlay_nr + i));
-			return;
-		}
+		overlay_name = overlay_name_from_nr(overlay_nr + i);
 
-		ovcs_id = 0;
-		ret = of_overlay_apply(np, &ovcs_id);
+		ret = overlay_data_apply(overlay_name, &ovcs_id);
 		if (ret < 0)  {
-			unittest(0, "could not create overlay from \"%s\"\n",
-					overlay_path(overlay_nr + i));
+			unittest(0, "could not apply overlay \"%s\"\n",
+					overlay_name);
 			return;
 		}
 		ov_id[i] = ovcs_id;
@@ -1577,8 +1561,8 @@ static void of_unittest_overlay_8(void)
 	ovcs_id = ov_id[0];
 	ret = of_overlay_remove(&ovcs_id);
 	if (ret == 0) {
-		unittest(0, "overlay @\"%s\" was destroyed @\"%s\"\n",
-				overlay_path(overlay_nr + 0),
+		unittest(0, "%s was destroyed @\"%s\"\n",
+				overlay_name_from_nr(overlay_nr + 0),
 				unittest_path(unittest_nr,
 					PDEV_OVERLAY));
 		return;
@@ -1589,8 +1573,8 @@ static void of_unittest_overlay_8(void)
 		ovcs_id = ov_id[i];
 		ret = of_overlay_remove(&ovcs_id);
 		if (ret != 0) {
-			unittest(0, "overlay @\"%s\" not destroyed @\"%s\"\n",
-					overlay_path(overlay_nr + i),
+			unittest(0, "%s not destroyed @\"%s\"\n",
+					overlay_name_from_nr(overlay_nr + i),
 					unittest_path(unittest_nr,
 						PDEV_OVERLAY));
 			return;
@@ -1602,7 +1586,7 @@ static void of_unittest_overlay_8(void)
 }
 
 /* test insertion of a bus with parent devices */
-static void of_unittest_overlay_10(void)
+static void __init of_unittest_overlay_10(void)
 {
 	int ret;
 	char *child_path;
@@ -1625,7 +1609,7 @@ static void of_unittest_overlay_10(void)
 }
 
 /* test insertion of a bus with parent devices (and revert) */
-static void of_unittest_overlay_11(void)
+static void __init of_unittest_overlay_11(void)
 {
 	int ret;
 
@@ -1891,7 +1875,7 @@ static void of_unittest_overlay_i2c_cleanup(void)
 	i2c_del_driver(&unittest_i2c_dev_driver);
 }
 
-static void of_unittest_overlay_i2c_12(void)
+static void __init of_unittest_overlay_i2c_12(void)
 {
 	int ret;
 
@@ -1904,7 +1888,7 @@ static void of_unittest_overlay_i2c_12(void)
 }
 
 /* test deactivation of device */
-static void of_unittest_overlay_i2c_13(void)
+static void __init of_unittest_overlay_i2c_13(void)
 {
 	int ret;
 
@@ -1921,7 +1905,7 @@ static void of_unittest_overlay_i2c_14(void)
 {
 }
 
-static void of_unittest_overlay_i2c_15(void)
+static void __init of_unittest_overlay_i2c_15(void)
 {
 	int ret;
 
@@ -2023,23 +2007,38 @@ static inline void __init of_unittest_overlay(void) { }
 	extern uint8_t __dtb_##name##_begin[]; \
 	extern uint8_t __dtb_##name##_end[]
 
-#define OVERLAY_INFO(name, expected) \
-{	.dtb_begin	 = __dtb_##name##_begin, \
-	.dtb_end	 = __dtb_##name##_end, \
-	.expected_result = expected, \
+#define OVERLAY_INFO(overlay_name, expected)             \
+{	.dtb_begin       = __dtb_##overlay_name##_begin, \
+	.dtb_end         = __dtb_##overlay_name##_end,   \
+	.expected_result = expected,                     \
+	.name            = #overlay_name,                \
 }
 
 struct overlay_info {
-	uint8_t		   *dtb_begin;
-	uint8_t		   *dtb_end;
-	void		   *data;
-	struct device_node *np_overlay;
-	int		   expected_result;
-	int		   overlay_id;
+	uint8_t		*dtb_begin;
+	uint8_t		*dtb_end;
+	int		expected_result;
+	int		overlay_id;
+	char		*name;
 };
 
 OVERLAY_INFO_EXTERN(overlay_base);
 OVERLAY_INFO_EXTERN(overlay);
+OVERLAY_INFO_EXTERN(overlay_0);
+OVERLAY_INFO_EXTERN(overlay_1);
+OVERLAY_INFO_EXTERN(overlay_2);
+OVERLAY_INFO_EXTERN(overlay_3);
+OVERLAY_INFO_EXTERN(overlay_4);
+OVERLAY_INFO_EXTERN(overlay_5);
+OVERLAY_INFO_EXTERN(overlay_6);
+OVERLAY_INFO_EXTERN(overlay_7);
+OVERLAY_INFO_EXTERN(overlay_8);
+OVERLAY_INFO_EXTERN(overlay_9);
+OVERLAY_INFO_EXTERN(overlay_10);
+OVERLAY_INFO_EXTERN(overlay_11);
+OVERLAY_INFO_EXTERN(overlay_12);
+OVERLAY_INFO_EXTERN(overlay_13);
+OVERLAY_INFO_EXTERN(overlay_15);
 OVERLAY_INFO_EXTERN(overlay_bad_phandle);
 OVERLAY_INFO_EXTERN(overlay_bad_symbol);
 
@@ -2047,6 +2046,21 @@ struct overlay_info {
 static struct overlay_info overlays[] = {
 	OVERLAY_INFO(overlay_base, -9999),
 	OVERLAY_INFO(overlay, 0),
+	OVERLAY_INFO(overlay_0, 0),
+	OVERLAY_INFO(overlay_1, 0),
+	OVERLAY_INFO(overlay_2, 0),
+	OVERLAY_INFO(overlay_3, 0),
+	OVERLAY_INFO(overlay_4, 0),
+	OVERLAY_INFO(overlay_5, 0),
+	OVERLAY_INFO(overlay_6, 0),
+	OVERLAY_INFO(overlay_7, 0),
+	OVERLAY_INFO(overlay_8, 0),
+	OVERLAY_INFO(overlay_9, 0),
+	OVERLAY_INFO(overlay_10, 0),
+	OVERLAY_INFO(overlay_11, 0),
+	OVERLAY_INFO(overlay_12, 0),
+	OVERLAY_INFO(overlay_13, 0),
+	OVERLAY_INFO(overlay_15, 0),
 	OVERLAY_INFO(overlay_bad_phandle, -EINVAL),
 	OVERLAY_INFO(overlay_bad_symbol, -EINVAL),
 	{}
@@ -2077,6 +2091,7 @@ void __init unittest_unflatten_overlay_base(void)
 {
 	struct overlay_info *info;
 	u32 data_size;
+	void *new_fdt;
 	u32 size;
 
 	info = &overlays[0];
@@ -2098,17 +2113,16 @@ void __init unittest_unflatten_overlay_base(void)
 		return;
 	}
 
-	info->data = dt_alloc_memory(size, roundup_pow_of_two(FDT_V17_SIZE));
-	if (!info->data) {
+	new_fdt = dt_alloc_memory(size, roundup_pow_of_two(FDT_V17_SIZE));
+	if (!new_fdt) {
 		pr_err("alloc for dtb 'overlay_base' failed");
 		return;
 	}
 
-	memcpy(info->data, info->dtb_begin, size);
+	memcpy(new_fdt, info->dtb_begin, size);
 
-	__unflatten_device_tree(info->data, NULL, &info->np_overlay,
+	__unflatten_device_tree(new_fdt, NULL, &overlay_base_root,
 				dt_alloc_memory, true);
-	overlay_base_root = info->np_overlay;
 }
 
 /*
@@ -2122,73 +2136,44 @@ void __init unittest_unflatten_overlay_base(void)
  *
  * Return 0 on unexpected error.
  */
-static int __init overlay_data_add(int onum)
+static int __init overlay_data_apply(const char *overlay_name, int *overlay_id)
 {
 	struct overlay_info *info;
+	int found = 0;
 	int k;
 	int ret;
 	u32 size;
-	u32 size_from_header;
 
-	for (k = 0, info = overlays; info; info++, k++) {
-		if (k == onum)
+	for (k = 0, info = overlays; info && info->name; info++, k++) {
+		if (!strcmp(overlay_name, info->name)) {
+			found = 1;
 			break;
+		}
 	}
-	if (onum > k)
+	if (!found) {
+		pr_err("no overlay data for %s\n", overlay_name);
 		return 0;
+	}
 
 	size = info->dtb_end - info->dtb_begin;
 	if (!size) {
-		pr_err("no overlay to attach, %d\n", onum);
+		pr_err("no overlay data for %s\n", overlay_name);
 		ret = 0;
 	}
 
-	size_from_header = fdt_totalsize(info->dtb_begin);
-	if (size_from_header != size) {
-		pr_err("overlay header totalsize != actual size, %d", onum);
-		return 0;
-	}
-
-	/*
-	 * Must create permanent copy of FDT because of_fdt_unflatten_tree()
-	 * will create pointers to the passed in FDT in the EDT.
-	 */
-	info->data = kmemdup(info->dtb_begin, size, GFP_KERNEL);
-	if (!info->data) {
-		pr_err("unable to allocate memory for data, %d\n", onum);
-		return 0;
-	}
-
-	of_fdt_unflatten_tree(info->data, NULL, &info->np_overlay);
-	if (!info->np_overlay) {
-		pr_err("unable to unflatten overlay, %d\n", onum);
-		ret = 0;
-		goto out_free_data;
-	}
-
-	info->overlay_id = 0;
-	ret = of_overlay_apply(info->np_overlay, &info->overlay_id);
-	if (ret < 0) {
-		pr_err("of_overlay_apply() (ret=%d), %d\n", ret, onum);
-		goto out_free_np_overlay;
-	}
-
-	pr_debug("__dtb_overlay_begin applied, overlay id %d\n", ret);
-
-	goto out;
-
-out_free_np_overlay:
-	/*
-	 * info->np_overlay is the unflattened device tree
-	 * It has not been spliced into the live tree.
-	 */
-
-	/* todo: function to free unflattened device tree */
+	ret = of_overlay_fdt_apply(info->dtb_begin, size, &info->overlay_id);
+	if (overlay_id)
+		*overlay_id = info->overlay_id;
+	if (ret < 0)
+		goto out;
 
-out_free_data:
-	kfree(info->data);
+	pr_debug("%s applied\n", overlay_name);
 
 out:
+	if (ret != info->expected_result)
+		pr_err("of_overlay_fdt_apply() expected %d, ret=%d, %s\n",
+		       info->expected_result, ret, overlay_name);
+
 	return (ret == info->expected_result);
 }
 
@@ -2290,18 +2275,29 @@ static __init void of_unittest_overlay_high_level(void)
 		__of_attach_node_sysfs(np);
 
 	if (of_symbols) {
+		struct property *new_prop;
 		for_each_property_of_node(overlay_base_symbols, prop) {
-			ret = __of_add_property(of_symbols, prop);
+
+			new_prop = __of_prop_dup(prop, GFP_KERNEL);
+			if (!new_prop) {
+				unittest(0, "__of_prop_dup() of '%s' from overlay_base node __symbols__",
+					 prop->name);
+				goto err_unlock;
+			}
+			ret = __of_add_property(of_symbols, new_prop);
 			if (ret) {
-				unittest(0,
-					 "duplicate property '%s' in overlay_base node __symbols__",
+				if (!strcmp(new_prop->name, "name")) {
+					/* auto-generated by unflatten */
+					ret = 0;
+					continue;
+				}
+				unittest(0, "duplicate property '%s' in overlay_base node __symbols__",
 					 prop->name);
 				goto err_unlock;
 			}
-			ret = __of_add_property_sysfs(of_symbols, prop);
+			ret = __of_add_property_sysfs(of_symbols, new_prop);
 			if (ret) {
-				unittest(0,
-					 "unable to add property '%s' in overlay_base node __symbols__ to sysfs",
+				unittest(0, "unable to add property '%s' in overlay_base node __symbols__ to sysfs",
 					 prop->name);
 				goto err_unlock;
 			}
@@ -2313,13 +2309,13 @@ static __init void of_unittest_overlay_high_level(void)
 
 	/* now do the normal overlay usage test */
 
-	unittest(overlay_data_add(1),
+	unittest(overlay_data_apply("overlay", NULL),
 		 "Adding overlay 'overlay' failed\n");
 
-	unittest(overlay_data_add(2),
+	unittest(overlay_data_apply("overlay_bad_phandle", NULL),
 		 "Adding overlay 'overlay_bad_phandle' failed\n");
 
-	unittest(overlay_data_add(3),
+	unittest(overlay_data_apply("overlay_bad_symbol", NULL),
 		 "Adding overlay 'overlay_bad_symbol' failed\n");
 
 	return;
diff --git a/include/linux/of.h b/include/linux/of.h
index da1ee95241c1..ebf22dd0860c 100644
--- a/include/linux/of.h
+++ b/include/linux/of.h
@@ -1359,8 +1359,8 @@ struct of_overlay_notify_data {
 
 #ifdef CONFIG_OF_OVERLAY
 
-/* ID based overlays; the API for external users */
-int of_overlay_apply(struct device_node *tree, int *ovcs_id);
+int of_overlay_fdt_apply(const void *overlay_fdt, u32 overlay_fdt_size,
+			 int *ovcs_id);
 int of_overlay_remove(int *ovcs_id);
 int of_overlay_remove_all(void);
 
@@ -1369,7 +1369,7 @@ struct of_overlay_notify_data {
 
 #else
 
-static inline int of_overlay_apply(struct device_node *tree, int *ovcs_id)
+static inline int of_overlay_fdt_apply(void *overlay_fdt, int *ovcs_id)
 {
 	return -ENOTSUPP;
 }
-- 
Frank Rowand <frank.rowand@sony.com>

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Jan Kiszka]

Hi Frank,

On 2018-03-04 01:17, frowand.list@gmail.com wrote:
> From: Frank Rowand <frank.rowand@sony.com>
> 
> Move duplicating and unflattening of an overlay flattened devicetree
> (FDT) into the overlay application code.  To accomplish this,
> of_overlay_apply() is replaced by of_overlay_fdt_apply().
> 
> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
> code, which is thus responsible for freeing the duplicate FDT.  The
> caller of of_overlay_fdt_apply() remains responsible for freeing the
> original FDT.
> 
> The unflattened devicetree now belongs to devicetree code, which is
> thus responsible for freeing the unflattened devicetree.
> 
> These ownership changes prevent early freeing of the duplicated FDT
> or the unflattened devicetree, which could result in use after free
> errors.
> 
> of_overlay_fdt_apply() is a private function for the anticipated
> overlay loader.

We are using of_fdt_unflatten_tree + of_overlay_apply in the
(out-of-tree) Jailhouse loader driver in order to register a virtual
device during hypervisor activation with Linux. The DT overlay is
created from a a template but modified prior to application to account
for runtime-specific parameters. See [1] for the current implementation.

I'm now wondering how to model that scenario best with the new API.
Given that the loader lost ownership of the unflattened tree but the
modification API exist only for the that DT state, I'm not yet seeing a
clear solution. Should we apply the template in disabled form (status =
"disabled"), modify it, and then activate it while it is already applied?

Thanks,
Jan

[1]
https://github.com/siemens/jailhouse/blob/11ef401a9064644bf7faf28e35c97f27ececd81d/driver/pci.c#L280

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Rob Herring]

On Wed, Apr 4, 2018 at 5:35 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
> Hi Frank,
>
> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>> From: Frank Rowand <frank.rowand@sony.com>
>>
>> Move duplicating and unflattening of an overlay flattened devicetree
>> (FDT) into the overlay application code.  To accomplish this,
>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>
>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>> code, which is thus responsible for freeing the duplicate FDT.  The
>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>> original FDT.
>>
>> The unflattened devicetree now belongs to devicetree code, which is
>> thus responsible for freeing the unflattened devicetree.
>>
>> These ownership changes prevent early freeing of the duplicated FDT
>> or the unflattened devicetree, which could result in use after free
>> errors.
>>
>> of_overlay_fdt_apply() is a private function for the anticipated
>> overlay loader.
>
> We are using of_fdt_unflatten_tree + of_overlay_apply in the
> (out-of-tree) Jailhouse loader driver in order to register a virtual
> device during hypervisor activation with Linux. The DT overlay is
> created from a a template but modified prior to application to account
> for runtime-specific parameters. See [1] for the current implementation.
>
> I'm now wondering how to model that scenario best with the new API.
> Given that the loader lost ownership of the unflattened tree but the
> modification API exist only for the that DT state, I'm not yet seeing a
> clear solution. Should we apply the template in disabled form (status =
> "disabled"), modify it, and then activate it while it is already applied?

No. I don't think that will work.

The of_overlay_apply() function is still there, but static. We can
export it again if the need arises.

Another option is there is a notifier callback OF_OVERLAY_PRE_APPLY,
but I'm not sure we want to make that be the normal interface to make
modifications.

Rob

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Jan Kiszka]

On 2018-04-05 02:55, Rob Herring wrote:
> On Wed, Apr 4, 2018 at 5:35 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
>> Hi Frank,
>>
>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>> From: Frank Rowand <frank.rowand@sony.com>
>>>
>>> Move duplicating and unflattening of an overlay flattened devicetree
>>> (FDT) into the overlay application code.  To accomplish this,
>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>
>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>> original FDT.
>>>
>>> The unflattened devicetree now belongs to devicetree code, which is
>>> thus responsible for freeing the unflattened devicetree.
>>>
>>> These ownership changes prevent early freeing of the duplicated FDT
>>> or the unflattened devicetree, which could result in use after free
>>> errors.
>>>
>>> of_overlay_fdt_apply() is a private function for the anticipated
>>> overlay loader.
>>
>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>> device during hypervisor activation with Linux. The DT overlay is
>> created from a a template but modified prior to application to account
>> for runtime-specific parameters. See [1] for the current implementation.
>>
>> I'm now wondering how to model that scenario best with the new API.
>> Given that the loader lost ownership of the unflattened tree but the
>> modification API exist only for the that DT state, I'm not yet seeing a
>> clear solution. Should we apply the template in disabled form (status =
>> "disabled"), modify it, and then activate it while it is already applied?
> 
> No. I don't think that will work.
> 
> The of_overlay_apply() function is still there, but static. We can
> export it again if the need arises.

That would be the simplest solution from our perspective, but I'm not
sure if that is in the original spirit of this change.

> 
> Another option is there is a notifier callback OF_OVERLAY_PRE_APPLY,
> but I'm not sure we want to make that be the normal interface to make
> modifications.

And would calling modification functions from that callback be legal at all?

Jan

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

On 04/05/18 00:22, Jan Kiszka wrote:
> On 2018-04-05 02:55, Rob Herring wrote:
>> On Wed, Apr 4, 2018 at 5:35 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
>>> Hi Frank,
>>>
>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>
>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>> (FDT) into the overlay application code.  To accomplish this,
>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>
>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>> original FDT.
>>>>
>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>> thus responsible for freeing the unflattened devicetree.
>>>>
>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>> or the unflattened devicetree, which could result in use after free
>>>> errors.
>>>>
>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>> overlay loader.
>>>
>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>> device during hypervisor activation with Linux. The DT overlay is
>>> created from a a template but modified prior to application to account
>>> for runtime-specific parameters. See [1] for the current implementation.
>>>
>>> I'm now wondering how to model that scenario best with the new API.
>>> Given that the loader lost ownership of the unflattened tree but the
>>> modification API exist only for the that DT state, I'm not yet seeing a
>>> clear solution. Should we apply the template in disabled form (status =
>>> "disabled"), modify it, and then activate it while it is already applied?
>>
>> No. I don't think that will work.
>>
>> The of_overlay_apply() function is still there, but static. We can
>> export it again if the need arises.
> 
> That would be the simplest solution from our perspective, but I'm not
> sure if that is in the original spirit of this change.

For short term out of tree usage, exporting of_overlay_apply() is ok.

Yes, for in-tree, exporting it again defeats the attempted process to
solve the overlay issues to make them acceptable in main line.


>>
>> Another option is there is a notifier callback OF_OVERLAY_PRE_APPLY,
>> but I'm not sure we want to make that be the normal interface to make
>> modifications.
> 
> And would calling modification functions from that callback be legal at all?

It might work in some specific cases, but the result is undefined.


> 
> Jan
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Rob Herring]

On Thu, Apr 5, 2018 at 2:16 PM, Frank Rowand <frowand.list@gmail.com> wrote:
> On 04/05/18 00:22, Jan Kiszka wrote:
>> On 2018-04-05 02:55, Rob Herring wrote:
>>> On Wed, Apr 4, 2018 at 5:35 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
>>>> Hi Frank,
>>>>
>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>
>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>
>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>> original FDT.
>>>>>
>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>
>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>> or the unflattened devicetree, which could result in use after free
>>>>> errors.
>>>>>
>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>> overlay loader.
>>>>
>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>> device during hypervisor activation with Linux. The DT overlay is
>>>> created from a a template but modified prior to application to account
>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>
>>>> I'm now wondering how to model that scenario best with the new API.
>>>> Given that the loader lost ownership of the unflattened tree but the
>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>> clear solution. Should we apply the template in disabled form (status =
>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>
>>> No. I don't think that will work.
>>>
>>> The of_overlay_apply() function is still there, but static. We can
>>> export it again if the need arises.
>>
>> That would be the simplest solution from our perspective, but I'm not
>> sure if that is in the original spirit of this change.
>
> For short term out of tree usage, exporting of_overlay_apply() is ok.

Meaning the out of tree users can go export it themselves.

> Yes, for in-tree, exporting it again defeats the attempted process to
> solve the overlay issues to make them acceptable in main line.

The purpose of providing a function to apply an overlay in one step is
to handle the common case and avoid open coding the sequence
everywhere. That doesn't mean everyone will fall into the common case.

Of course, it also doesn't mean we have to revert back to the previous
way to handle users that need to tweak the unflattened tree before
applying.

Rob

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

On 04/05/18 12:26, Rob Herring wrote:
> On Thu, Apr 5, 2018 at 2:16 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>> On 04/05/18 00:22, Jan Kiszka wrote:
>>> On 2018-04-05 02:55, Rob Herring wrote:
>>>> On Wed, Apr 4, 2018 at 5:35 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
>>>>> Hi Frank,
>>>>>
>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>
>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>
>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>> original FDT.
>>>>>>
>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>
>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>> errors.
>>>>>>
>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>> overlay loader.
>>>>>
>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>> created from a a template but modified prior to application to account
>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>
>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>
>>>> No. I don't think that will work.
>>>>
>>>> The of_overlay_apply() function is still there, but static. We can
>>>> export it again if the need arises.
>>>
>>> That would be the simplest solution from our perspective, but I'm not
>>> sure if that is in the original spirit of this change.
>>
>> For short term out of tree usage, exporting of_overlay_apply() is ok.
> 
> Meaning the out of tree users can go export it themselves.
> 
>> Yes, for in-tree, exporting it again defeats the attempted process to
>> solve the overlay issues to make them acceptable in main line.
> 
> The purpose of providing a function to apply an overlay in one step is
> to handle the common case and avoid open coding the sequence
> everywhere. That doesn't mean everyone will fall into the common case.

That was one of the intents behind the change.

The other intent is that with lifetime of the overlay FDT and overlay
expanded tree visible outside the devicetree core, we can not free the
overlay FDT or the overlay expanded tree because we need to protect
against use after free errors.  The open coded examples of applying
overlays commonly have issues with freeing the overlay FDT and/or the
overlay expanded tree, which leads to the potential for use after free
errors.


> Of course, it also doesn't mean we have to revert back to the previous
> way to handle users that need to tweak the unflattened tree before
> applying.
> 
> Rob
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

On 04/04/18 17:55, Rob Herring wrote:
> On Wed, Apr 4, 2018 at 5:35 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
>> Hi Frank,
>>
>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>> From: Frank Rowand <frank.rowand@sony.com>
>>>
>>> Move duplicating and unflattening of an overlay flattened devicetree
>>> (FDT) into the overlay application code.  To accomplish this,
>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>
>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>> original FDT.
>>>
>>> The unflattened devicetree now belongs to devicetree code, which is
>>> thus responsible for freeing the unflattened devicetree.
>>>
>>> These ownership changes prevent early freeing of the duplicated FDT
>>> or the unflattened devicetree, which could result in use after free
>>> errors.
>>>
>>> of_overlay_fdt_apply() is a private function for the anticipated
>>> overlay loader.
>>
>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>> device during hypervisor activation with Linux. The DT overlay is
>> created from a a template but modified prior to application to account
>> for runtime-specific parameters. See [1] for the current implementation.
>>
>> I'm now wondering how to model that scenario best with the new API.
>> Given that the loader lost ownership of the unflattened tree but the
>> modification API exist only for the that DT state, I'm not yet seeing a
>> clear solution. Should we apply the template in disabled form (status =
>> "disabled"), modify it, and then activate it while it is already applied?
> 
> No. I don't think that will work.

I don't think it will work either.  The drivers won't be probed if the
nodes are disabled.


> The of_overlay_apply() function is still there, but static. We can
> export it again if the need arises.

I really don't want to export it again.  sigh.


> Another option is there is a notifier callback OF_OVERLAY_PRE_APPLY,
> but I'm not sure we want to make that be the normal interface to make
> modifications.

This callback is on my list of yet more ways that pointers into the
(overlay) FDT are exposed.  Which means another solution to the
underlying problem (which was an FPGA problem) needs to be created.
I'd prefer that use of this mechanism not be expanded in the mean time.

Even if that were not the case, using the OF_OVERLAY_PRE_APPLY notifier
to modify the tree is problematic because the notifier(s) is called
after the overlay FDT is in final form

-Frank

> 
> Rob
> .
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Jan Kiszka]

On 2018-04-05 21:13, Frank Rowand wrote:
> On 04/04/18 17:55, Rob Herring wrote:
>> On Wed, Apr 4, 2018 at 5:35 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
>>> Hi Frank,
>>>
>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>
>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>> (FDT) into the overlay application code.  To accomplish this,
>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>
>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>> original FDT.
>>>>
>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>> thus responsible for freeing the unflattened devicetree.
>>>>
>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>> or the unflattened devicetree, which could result in use after free
>>>> errors.
>>>>
>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>> overlay loader.
>>>
>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>> device during hypervisor activation with Linux. The DT overlay is
>>> created from a a template but modified prior to application to account
>>> for runtime-specific parameters. See [1] for the current implementation.
>>>
>>> I'm now wondering how to model that scenario best with the new API.
>>> Given that the loader lost ownership of the unflattened tree but the
>>> modification API exist only for the that DT state, I'm not yet seeing a
>>> clear solution. Should we apply the template in disabled form (status =
>>> "disabled"), modify it, and then activate it while it is already applied?
>>
>> No. I don't think that will work.
> 
> I don't think it will work either.  The drivers won't be probed if the
> nodes are disabled.
> 

That would be the purpose of registering it disabled. The question is:
Would it be probed at the point when we enabled it while being applied?
If not, strike this.

Jan

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

Hi Jan,

On 04/04/18 15:35, Jan Kiszka wrote:
> Hi Frank,
> 
> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>> From: Frank Rowand <frank.rowand@sony.com>
>>
>> Move duplicating and unflattening of an overlay flattened devicetree
>> (FDT) into the overlay application code.  To accomplish this,
>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>
>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>> code, which is thus responsible for freeing the duplicate FDT.  The
>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>> original FDT.
>>
>> The unflattened devicetree now belongs to devicetree code, which is
>> thus responsible for freeing the unflattened devicetree.
>>
>> These ownership changes prevent early freeing of the duplicated FDT
>> or the unflattened devicetree, which could result in use after free
>> errors.
>>
>> of_overlay_fdt_apply() is a private function for the anticipated
>> overlay loader.
> 
> We are using of_fdt_unflatten_tree + of_overlay_apply in the
> (out-of-tree) Jailhouse loader driver in order to register a virtual
> device during hypervisor activation with Linux. The DT overlay is
> created from a a template but modified prior to application to account
> for runtime-specific parameters. See [1] for the current implementation.
> 
> I'm now wondering how to model that scenario best with the new API.
> Given that the loader lost ownership of the unflattened tree but the
> modification API exist only for the that DT state, I'm not yet seeing a
> clear solution. Should we apply the template in disabled form (status =
> "disabled"), modify it, and then activate it while it is already applied?

Thank you for the pointer to the driver - that makes it much easier to
understand the use case and consider solutions.

If you can make the changes directly on the FDT instead of on the
expanded devicetree, then you could move to the new API.

Looking at the driver, I see one potential issue with that approach.
The property "interrupt-map" is added directly to the changeset
instead of being an existing property in the overlay.  Is it possible
to have this property in the overlay when needed?

I'll also reply to other comments in this thread.

-Frank

> 
> Thanks,
> Jan
> 
> [1]
> https://github.com/siemens/jailhouse/blob/11ef401a9064644bf7faf28e35c97f27ececd81d/driver/pci.c#L280
> .
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Jan Kiszka]

On 2018-04-05 20:59, Frank Rowand wrote:
> Hi Jan,
> 
> On 04/04/18 15:35, Jan Kiszka wrote:
>> Hi Frank,
>>
>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>> From: Frank Rowand <frank.rowand@sony.com>
>>>
>>> Move duplicating and unflattening of an overlay flattened devicetree
>>> (FDT) into the overlay application code.  To accomplish this,
>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>
>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>> original FDT.
>>>
>>> The unflattened devicetree now belongs to devicetree code, which is
>>> thus responsible for freeing the unflattened devicetree.
>>>
>>> These ownership changes prevent early freeing of the duplicated FDT
>>> or the unflattened devicetree, which could result in use after free
>>> errors.
>>>
>>> of_overlay_fdt_apply() is a private function for the anticipated
>>> overlay loader.
>>
>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>> device during hypervisor activation with Linux. The DT overlay is
>> created from a a template but modified prior to application to account
>> for runtime-specific parameters. See [1] for the current implementation.
>>
>> I'm now wondering how to model that scenario best with the new API.
>> Given that the loader lost ownership of the unflattened tree but the
>> modification API exist only for the that DT state, I'm not yet seeing a
>> clear solution. Should we apply the template in disabled form (status =
>> "disabled"), modify it, and then activate it while it is already applied?
> 
> Thank you for the pointer to the driver - that makes it much easier to
> understand the use case and consider solutions.
> 
> If you can make the changes directly on the FDT instead of on the
> expanded devicetree, then you could move to the new API.

Are there some examples/references on how to edit FDTs in-place in the
kernel? I'd like to avoid writing the n-th FDT parser/generator.

> 
> Looking at the driver, I see one potential issue with that approach.
> The property "interrupt-map" is added directly to the changeset
> instead of being an existing property in the overlay.  Is it possible
> to have this property in the overlay when needed?

Well, the size of that property has a runtime dependency on the gic's
#address-cells. If that is easy to account for depends a bit on the
available FDT manipulation services. Or it would take multiple templates
to handle the different cases (0, 1, or 2 IIRC).

> 
> I'll also reply to other comments in this thread.

TIA!

Jan

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

On 04/05/18 12:13, Jan Kiszka wrote:
> On 2018-04-05 20:59, Frank Rowand wrote:
>> Hi Jan,
>>
>> On 04/04/18 15:35, Jan Kiszka wrote:
>>> Hi Frank,
>>>
>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>
>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>> (FDT) into the overlay application code.  To accomplish this,
>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>
>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>> original FDT.
>>>>
>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>> thus responsible for freeing the unflattened devicetree.
>>>>
>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>> or the unflattened devicetree, which could result in use after free
>>>> errors.
>>>>
>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>> overlay loader.
>>>
>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>> device during hypervisor activation with Linux. The DT overlay is
>>> created from a a template but modified prior to application to account
>>> for runtime-specific parameters. See [1] for the current implementation.
>>>
>>> I'm now wondering how to model that scenario best with the new API.
>>> Given that the loader lost ownership of the unflattened tree but the
>>> modification API exist only for the that DT state, I'm not yet seeing a
>>> clear solution. Should we apply the template in disabled form (status =
>>> "disabled"), modify it, and then activate it while it is already applied?
>>
>> Thank you for the pointer to the driver - that makes it much easier to
>> understand the use case and consider solutions.
>>
>> If you can make the changes directly on the FDT instead of on the
>> expanded devicetree, then you could move to the new API.
> 
> Are there some examples/references on how to edit FDTs in-place in the
> kernel? I'd like to avoid writing the n-th FDT parser/generator.

I don't know of any existing in-kernel edits of the FDT (but they might
exist).  The functions to access an FDT are in libfdt, which is in
scripts/dtc/libfdt/.


>>
>> Looking at the driver, I see one potential issue with that approach.
>> The property "interrupt-map" is added directly to the changeset
>> instead of being an existing property in the overlay.  Is it possible
>> to have this property in the overlay when needed?
> 
> Well, the size of that property has a runtime dependency on the gic's
> #address-cells. If that is easy to account for depends a bit on the
> available FDT manipulation services. Or it would take multiple templates
> to handle the different cases (0, 1, or 2 IIRC).

If I understand create_vpci_of_overlay() correctly, it is assuming a
fixed size of 4 cells.  Line 314 is: for (n = 0, cell = 0; n < 4; n++) {

Off the top of my head, it is theoretically possible to create a property
that can contain the largest possible size for the property, then shrink
the size by inserting NOPs at the end of the property to shrink it.


>>
>> I'll also reply to other comments in this thread.
> 
> TIA!
> 
> Jan
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Jan Kiszka]

On 2018-04-05 21:28, Frank Rowand wrote:
> On 04/05/18 12:13, Jan Kiszka wrote:
>> On 2018-04-05 20:59, Frank Rowand wrote:
>>> Hi Jan,
>>>
>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>> Hi Frank,
>>>>
>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>
>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>
>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>> original FDT.
>>>>>
>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>
>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>> or the unflattened devicetree, which could result in use after free
>>>>> errors.
>>>>>
>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>> overlay loader.
>>>>
>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>> device during hypervisor activation with Linux. The DT overlay is
>>>> created from a a template but modified prior to application to account
>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>
>>>> I'm now wondering how to model that scenario best with the new API.
>>>> Given that the loader lost ownership of the unflattened tree but the
>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>> clear solution. Should we apply the template in disabled form (status =
>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>
>>> Thank you for the pointer to the driver - that makes it much easier to
>>> understand the use case and consider solutions.
>>>
>>> If you can make the changes directly on the FDT instead of on the
>>> expanded devicetree, then you could move to the new API.
>>
>> Are there some examples/references on how to edit FDTs in-place in the
>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
> 
> I don't know of any existing in-kernel edits of the FDT (but they might
> exist).  The functions to access an FDT are in libfdt, which is in
> scripts/dtc/libfdt/.
> 

Ah, libfdt is available for kernel drivers as well. That looks like a
viable path on first sight. I'll try that and come back in case it does
not solve all issues.

> 
>>>
>>> Looking at the driver, I see one potential issue with that approach.
>>> The property "interrupt-map" is added directly to the changeset
>>> instead of being an existing property in the overlay.  Is it possible
>>> to have this property in the overlay when needed?
>>
>> Well, the size of that property has a runtime dependency on the gic's
>> #address-cells. If that is easy to account for depends a bit on the
>> available FDT manipulation services. Or it would take multiple templates
>> to handle the different cases (0, 1, or 2 IIRC).
> 
> If I understand create_vpci_of_overlay() correctly, it is assuming a
> fixed size of 4 cells.  Line 314 is: for (n = 0, cell = 0; n < 4; n++) {
> 
> Off the top of my head, it is theoretically possible to create a property
> that can contain the largest possible size for the property, then shrink
> the size by inserting NOPs at the end of the property to shrink it.

Well, I even find fdt_appendprop which sounds like we could keep adding
that property on the fly.

How does memory management work with libfdt? Do I have to ensure that
the fdt is already backed by an area large enough also for it modified form?

Thanks,
Jan

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

On 04/05/18 12:38, Jan Kiszka wrote:
> On 2018-04-05 21:28, Frank Rowand wrote:
>> On 04/05/18 12:13, Jan Kiszka wrote:
>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>> Hi Jan,
>>>>
>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>> Hi Frank,
>>>>>
>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>
>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>
>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>> original FDT.
>>>>>>
>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>
>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>> errors.
>>>>>>
>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>> overlay loader.
>>>>>
>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>> created from a a template but modified prior to application to account
>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>
>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>
>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>> understand the use case and consider solutions.
>>>>
>>>> If you can make the changes directly on the FDT instead of on the
>>>> expanded devicetree, then you could move to the new API.
>>>
>>> Are there some examples/references on how to edit FDTs in-place in the
>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>
>> I don't know of any existing in-kernel edits of the FDT (but they might
>> exist).  The functions to access an FDT are in libfdt, which is in
>> scripts/dtc/libfdt/.
>>
> 
> Ah, libfdt is available for kernel drivers as well. That looks like a
> viable path on first sight. I'll try that and come back in case it does
> not solve all issues.
> 
>>
>>>>
>>>> Looking at the driver, I see one potential issue with that approach.
>>>> The property "interrupt-map" is added directly to the changeset
>>>> instead of being an existing property in the overlay.  Is it possible
>>>> to have this property in the overlay when needed?
>>>
>>> Well, the size of that property has a runtime dependency on the gic's
>>> #address-cells. If that is easy to account for depends a bit on the
>>> available FDT manipulation services. Or it would take multiple templates
>>> to handle the different cases (0, 1, or 2 IIRC).
>>
>> If I understand create_vpci_of_overlay() correctly, it is assuming a
>> fixed size of 4 cells.  Line 314 is: for (n = 0, cell = 0; n < 4; n++) {
>>
>> Off the top of my head, it is theoretically possible to create a property
>> that can contain the largest possible size for the property, then shrink
>> the size by inserting NOPs at the end of the property to shrink it.
> 
> Well, I even find fdt_appendprop which sounds like we could keep adding
> that property on the fly.
> 
> How does memory management work with libfdt? Do I have to ensure that
> the fdt is already backed by an area large enough also for it modified form?

I have not looked at those functions, so I don't know.


> 
> Thanks,
> Jan
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Rob Herring]

On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
> On 04/05/18 12:13, Jan Kiszka wrote:
>> On 2018-04-05 20:59, Frank Rowand wrote:
>>> Hi Jan,
>>>
>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>> Hi Frank,
>>>>
>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>
>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>
>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>> original FDT.
>>>>>
>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>
>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>> or the unflattened devicetree, which could result in use after free
>>>>> errors.
>>>>>
>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>> overlay loader.
>>>>
>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>> device during hypervisor activation with Linux. The DT overlay is
>>>> created from a a template but modified prior to application to account
>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>
>>>> I'm now wondering how to model that scenario best with the new API.
>>>> Given that the loader lost ownership of the unflattened tree but the
>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>> clear solution. Should we apply the template in disabled form (status =
>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>
>>> Thank you for the pointer to the driver - that makes it much easier to
>>> understand the use case and consider solutions.
>>>
>>> If you can make the changes directly on the FDT instead of on the
>>> expanded devicetree, then you could move to the new API.
>>
>> Are there some examples/references on how to edit FDTs in-place in the
>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>
> I don't know of any existing in-kernel edits of the FDT (but they might
> exist).  The functions to access an FDT are in libfdt, which is in
> scripts/dtc/libfdt/.

Let's please not go down that route of doing FDT modifications. There
is little reason to other than for early boot changes. And it is much
easier to work on unflattened trees.

Rob

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Jan Kiszka]

On 2018-04-05 23:12, Rob Herring wrote:
> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>> On 04/05/18 12:13, Jan Kiszka wrote:
>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>> Hi Jan,
>>>>
>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>> Hi Frank,
>>>>>
>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>
>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>
>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>> original FDT.
>>>>>>
>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>
>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>> errors.
>>>>>>
>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>> overlay loader.
>>>>>
>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>> created from a a template but modified prior to application to account
>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>
>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>
>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>> understand the use case and consider solutions.
>>>>
>>>> If you can make the changes directly on the FDT instead of on the
>>>> expanded devicetree, then you could move to the new API.
>>>
>>> Are there some examples/references on how to edit FDTs in-place in the
>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>
>> I don't know of any existing in-kernel edits of the FDT (but they might
>> exist).  The functions to access an FDT are in libfdt, which is in
>> scripts/dtc/libfdt/.
> 
> Let's please not go down that route of doing FDT modifications. There
> is little reason to other than for early boot changes. And it is much
> easier to work on unflattened trees.

I just briefly looked into libfdt, and it would have meant building it
into the module as there are no library functions exported by the kernel
either. Another reason to drop that.

What's apparently working now is the pattern I initially suggested:
Register template with status = "disabled" as overlay, then prepare and
apply changeset that contains all needed modifications and sets the
status to "ok". I might be leaking additional resources, but to find
that out, I will now finally have to resolve clean unbinding of the
generic PCI host controller [1] first.

Thanks,
Jan

[1] http://lkml.iu.edu/hypermail/linux/kernel/1606.3/00072.html

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Jan Kiszka]

On 2018-04-11 07:42, Jan Kiszka wrote:
> On 2018-04-05 23:12, Rob Herring wrote:
>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>> Hi Jan,
>>>>>
>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>> Hi Frank,
>>>>>>
>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>
>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>
>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>> original FDT.
>>>>>>>
>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>
>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>> errors.
>>>>>>>
>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>> overlay loader.
>>>>>>
>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>> created from a a template but modified prior to application to account
>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>
>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>
>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>> understand the use case and consider solutions.
>>>>>
>>>>> If you can make the changes directly on the FDT instead of on the
>>>>> expanded devicetree, then you could move to the new API.
>>>>
>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>
>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>> exist).  The functions to access an FDT are in libfdt, which is in
>>> scripts/dtc/libfdt/.
>>
>> Let's please not go down that route of doing FDT modifications. There
>> is little reason to other than for early boot changes. And it is much
>> easier to work on unflattened trees.
> 
> I just briefly looked into libfdt, and it would have meant building it
> into the module as there are no library functions exported by the kernel
> either. Another reason to drop that.
> 
> What's apparently working now is the pattern I initially suggested:
> Register template with status = "disabled" as overlay, then prepare and
> apply changeset that contains all needed modifications and sets the
> status to "ok". I might be leaking additional resources, but to find
> that out, I will now finally have to resolve clean unbinding of the
> generic PCI host controller [1] first.

static void free_overlay_changeset(struct overlay_changeset *ovcs)
{
	[...]
	/*
	 * TODO
	 *
	 * would like to: kfree(ovcs->overlay_tree);
	 * but can not since drivers may have pointers into this data
	 *
	 * would like to: kfree(ovcs->fdt);
	 * but can not since drivers may have pointers into this data
	 */

	kfree(ovcs);
}

What's this? I have kmemleak now jumping at me over this. Who is suppose
to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
to those objects. I would say that's a regression of the new API.

Jan

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

Hi Jan,

+ Alan Tull for fpga perspective

On 04/22/18 03:30, Jan Kiszka wrote:
> On 2018-04-11 07:42, Jan Kiszka wrote:
>> On 2018-04-05 23:12, Rob Herring wrote:
>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>> Hi Jan,
>>>>>>
>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>> Hi Frank,
>>>>>>>
>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>
>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>
>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>> original FDT.
>>>>>>>>
>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>
>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>> errors.
>>>>>>>>
>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>> overlay loader.
>>>>>>>
>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>> created from a a template but modified prior to application to account
>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>
>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>
>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>> understand the use case and consider solutions.
>>>>>>
>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>> expanded devicetree, then you could move to the new API.
>>>>>
>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>
>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>> scripts/dtc/libfdt/.
>>>
>>> Let's please not go down that route of doing FDT modifications. There
>>> is little reason to other than for early boot changes. And it is much
>>> easier to work on unflattened trees.
>>
>> I just briefly looked into libfdt, and it would have meant building it
>> into the module as there are no library functions exported by the kernel
>> either. Another reason to drop that.
>>
>> What's apparently working now is the pattern I initially suggested:
>> Register template with status = "disabled" as overlay, then prepare and
>> apply changeset that contains all needed modifications and sets the
>> status to "ok". I might be leaking additional resources, but to find
>> that out, I will now finally have to resolve clean unbinding of the
>> generic PCI host controller [1] first.
> 
> static void free_overlay_changeset(struct overlay_changeset *ovcs)
> {
> 	[...]
> 	/*
> 	 * TODO
> 	 *
> 	 * would like to: kfree(ovcs->overlay_tree);
> 	 * but can not since drivers may have pointers into this data
> 	 *
> 	 * would like to: kfree(ovcs->fdt);
> 	 * but can not since drivers may have pointers into this data
> 	 */
> 
> 	kfree(ovcs);
> }
> 
> What's this? I have kmemleak now jumping at me over this. Who is suppose
> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
> to those objects. I would say that's a regression of the new API.

The problem already existed but it was hidden.  We have never been able to
kfree() these object because we do not know if there are any pointers into
these objects.  The new API makes the problem visible to kmemleak.

The reason that we do not know if there are any pointers into these objects
is that devicetree access APIs return pointers into the devicetree internal
data structures (that is, into the overlay unflattened devicetree).  If we
want to be able to do the kfree()s, we could change the devicetree access
APIs.

The reason that pointers into the overlay flattened tree (ovcs->fdt) are
also exposed is that the overlay unflattened devicetree property values
are pointers into the overlay fdt.

** This paragraph becomes academic (and not needed) if the fix in the next
paragraph can be implemented. **
I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
(I would want to read through the code again to make sure I'm not missing
any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
was modified so that property values were copied into newly allocated memory
and the live tree property pointers were set to the copy instead of to
the value in the fdt, then I _think_ the fdt could be freed in
of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
frees a devicetree would also have to be aware of this change -- I'm not
sure if that leads to ugly complications or if it is easy.  The other
question to consider is whether to make the same change to
of_fdt_unflatten_tree() when it is called in early boot to unflatten
the base devicetree.  Doing so would increase the memory usage of the
live tree (we would not be able to free the base fdt after unflattening
it because we make the fdt visible in /sys/firmware/fdt -- though
_maybe_ that could be conditioned on CONFIG_KEXEC).

But all of the complexity of that fix is _only_ because of_overlay_apply()
and of_overlay_remove() call overlay_notify(), passing in the overlay
unflattened devicetree (which has pointers into the overlay fdt). Pointers
into the overlay unflattened devicetree are then passed to the notifiers.
(Again, I may be missing some other place that the overlay unflattened
devicetree is made visible to other code -- a more thorough reading of
the code is needed.) If the notifiers could be modified to accept the
changeset list instead of of pointers to the fragments in the overlay
unflattened devicetree then there would be no possibility of the notifiers
keeping a pointer into the overlay fdt. I do not know if this is a
practical change for the notifiers -- there are no callers of
of_overlay_notifier_register() in the mainline kernel source. My
recollection is that the overlay notifiers were added for the fpga
subsystem.

Why is overlay_notify() the only issue related to unknown users having
pointers into the overlay fdt?  The answer is that the overlay code
does not directly expose the overlay unflattened devicetree (and thus
indirectly the overlay fdt) to the live devicetree -- when the
overlay code creates the overlay changeset, it copies from the
overlay unflattened devicetree and overlay fdt and only exposes
pointers to the copies.

And hopefully the issues with the overlay unflattened devicetree can
be resolved in the same way as for the overlay fdt.

-Frank

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Jan Kiszka]

On 2018-04-24 00:38, Frank Rowand wrote:
> Hi Jan,
> 
> + Alan Tull for fpga perspective
> 
> On 04/22/18 03:30, Jan Kiszka wrote:
>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>> Hi Jan,
>>>>>>>
>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>> Hi Frank,
>>>>>>>>
>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>
>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>
>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>> original FDT.
>>>>>>>>>
>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>
>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>> errors.
>>>>>>>>>
>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>> overlay loader.
>>>>>>>>
>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>
>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>
>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>> understand the use case and consider solutions.
>>>>>>>
>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>
>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>
>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>> scripts/dtc/libfdt/.
>>>>
>>>> Let's please not go down that route of doing FDT modifications. There
>>>> is little reason to other than for early boot changes. And it is much
>>>> easier to work on unflattened trees.
>>>
>>> I just briefly looked into libfdt, and it would have meant building it
>>> into the module as there are no library functions exported by the kernel
>>> either. Another reason to drop that.
>>>
>>> What's apparently working now is the pattern I initially suggested:
>>> Register template with status = "disabled" as overlay, then prepare and
>>> apply changeset that contains all needed modifications and sets the
>>> status to "ok". I might be leaking additional resources, but to find
>>> that out, I will now finally have to resolve clean unbinding of the
>>> generic PCI host controller [1] first.
>>
>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>> {
>> 	[...]
>> 	/*
>> 	 * TODO
>> 	 *
>> 	 * would like to: kfree(ovcs->overlay_tree);
>> 	 * but can not since drivers may have pointers into this data
>> 	 *
>> 	 * would like to: kfree(ovcs->fdt);
>> 	 * but can not since drivers may have pointers into this data
>> 	 */
>>
>> 	kfree(ovcs);
>> }
>>
>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>> to those objects. I would say that's a regression of the new API.
> 
> The problem already existed but it was hidden.  We have never been able to
> kfree() these object because we do not know if there are any pointers into
> these objects.  The new API makes the problem visible to kmemleak.

My old code didn't have the problem because there was no one steeling
pointers to my overlay, and I was able to safely release all the
resources that I or the core on my behalf allocated. In fact, I recently
even dropped the duplication the fdt prior to unflattening it because I
got its lifecycle under control (and both kmemleak as well as kasan
confirmed this). I still consider this intentional leak a regression of
the new API.

> 
> The reason that we do not know if there are any pointers into these objects
> is that devicetree access APIs return pointers into the devicetree internal
> data structures (that is, into the overlay unflattened devicetree).  If we
> want to be able to do the kfree()s, we could change the devicetree access
> APIs.
> 
> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
> also exposed is that the overlay unflattened devicetree property values
> are pointers into the overlay fdt.
> 
> ** This paragraph becomes academic (and not needed) if the fix in the next
> paragraph can be implemented. **
> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
> (I would want to read through the code again to make sure I'm not missing
> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
> was modified so that property values were copied into newly allocated memory
> and the live tree property pointers were set to the copy instead of to
> the value in the fdt, then I _think_ the fdt could be freed in
> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that

I don't see yet how more duplicating of objects would help. Then we
would not leak the fdt or the unflattened tree on overlay destruction
but that duplicates, no?

> frees a devicetree would also have to be aware of this change -- I'm not
> sure if that leads to ugly complications or if it is easy.  The other
> question to consider is whether to make the same change to
> of_fdt_unflatten_tree() when it is called in early boot to unflatten
> the base devicetree.  Doing so would increase the memory usage of the
> live tree (we would not be able to free the base fdt after unflattening
> it because we make the fdt visible in /sys/firmware/fdt -- though
> _maybe_ that could be conditioned on CONFIG_KEXEC).
> 
> But all of the complexity of that fix is _only_ because of_overlay_apply()
> and of_overlay_remove() call overlay_notify(), passing in the overlay
> unflattened devicetree (which has pointers into the overlay fdt). Pointers
> into the overlay unflattened devicetree are then passed to the notifiers.
> (Again, I may be missing some other place that the overlay unflattened
> devicetree is made visible to other code -- a more thorough reading of
> the code is needed.) If the notifiers could be modified to accept the
> changeset list instead of of pointers to the fragments in the overlay
> unflattened devicetree then there would be no possibility of the notifiers
> keeping a pointer into the overlay fdt. I do not know if this is a

But then again the convention has to be that those changeset pointers
must not be kept - because the changeset is history after of_overlay_remove.

> practical change for the notifiers -- there are no callers of
> of_overlay_notifier_register() in the mainline kernel source. My
> recollection is that the overlay notifiers were added for the fpga
> subsystem.

We have drivers/fpga/of-fpga-region.c in-tree, and that does not seem to
store any pointers to objects, rather consumes them in-place. And I
would consider it fair to impose such a limitation on the notifier
interface.

> 
> Why is overlay_notify() the only issue related to unknown users having
> pointers into the overlay fdt?  The answer is that the overlay code
> does not directly expose the overlay unflattened devicetree (and thus
> indirectly the overlay fdt) to the live devicetree -- when the
> overlay code creates the overlay changeset, it copies from the
> overlay unflattened devicetree and overlay fdt and only exposes
> pointers to the copies.
> 
> And hopefully the issues with the overlay unflattened devicetree can
> be resolved in the same way as for the overlay fdt.

As noted above, I don't see there is a technical solution to this issue
but it's rather a matter of convention: no overlay notifier callback is
allowed to keep references to the passed tree content (unless it
reference-counts some tree nodes) beyond the execution of the callback.
With that in place, we can safely drop the backing memory IMHO.

Jan

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Alan Tull]

On Tue, Apr 24, 2018 at 12:29 AM, Jan Kiszka <jan.kiszka@web.de> wrote:
> On 2018-04-24 00:38, Frank Rowand wrote:
>> Hi Jan,
>>
>> + Alan Tull for fpga perspective
>>
>> On 04/22/18 03:30, Jan Kiszka wrote:
>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>> Hi Jan,
>>>>>>>>
>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>> Hi Frank,
>>>>>>>>>
>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>
>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>
>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>> original FDT.
>>>>>>>>>>
>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>
>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>> errors.
>>>>>>>>>>
>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>> overlay loader.
>>>>>>>>>
>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>
>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>
>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>> understand the use case and consider solutions.
>>>>>>>>
>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>
>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>
>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>> scripts/dtc/libfdt/.
>>>>>
>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>> is little reason to other than for early boot changes. And it is much
>>>>> easier to work on unflattened trees.
>>>>
>>>> I just briefly looked into libfdt, and it would have meant building it
>>>> into the module as there are no library functions exported by the kernel
>>>> either. Another reason to drop that.
>>>>
>>>> What's apparently working now is the pattern I initially suggested:
>>>> Register template with status = "disabled" as overlay, then prepare and
>>>> apply changeset that contains all needed modifications and sets the
>>>> status to "ok". I might be leaking additional resources, but to find
>>>> that out, I will now finally have to resolve clean unbinding of the
>>>> generic PCI host controller [1] first.
>>>
>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>> {
>>>      [...]
>>>      /*
>>>       * TODO
>>>       *
>>>       * would like to: kfree(ovcs->overlay_tree);
>>>       * but can not since drivers may have pointers into this data
>>>       *
>>>       * would like to: kfree(ovcs->fdt);
>>>       * but can not since drivers may have pointers into this data
>>>       */
>>>
>>>      kfree(ovcs);
>>> }
>>>
>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>> to those objects. I would say that's a regression of the new API.
>>
>> The problem already existed but it was hidden.  We have never been able to
>> kfree() these object because we do not know if there are any pointers into
>> these objects.  The new API makes the problem visible to kmemleak.
>
> My old code didn't have the problem because there was no one steeling
> pointers to my overlay, and I was able to safely release all the
> resources that I or the core on my behalf allocated. In fact, I recently
> even dropped the duplication the fdt prior to unflattening it because I
> got its lifecycle under control (and both kmemleak as well as kasan
> confirmed this). I still consider this intentional leak a regression of
> the new API.
>
>>
>> The reason that we do not know if there are any pointers into these objects
>> is that devicetree access APIs return pointers into the devicetree internal
>> data structures (that is, into the overlay unflattened devicetree).  If we
>> want to be able to do the kfree()s, we could change the devicetree access
>> APIs.
>>
>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>> also exposed is that the overlay unflattened devicetree property values
>> are pointers into the overlay fdt.
>>
>> ** This paragraph becomes academic (and not needed) if the fix in the next
>> paragraph can be implemented. **
>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>> (I would want to read through the code again to make sure I'm not missing
>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>> was modified so that property values were copied into newly allocated memory
>> and the live tree property pointers were set to the copy instead of to
>> the value in the fdt, then I _think_ the fdt could be freed in
>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>
> I don't see yet how more duplicating of objects would help. Then we
> would not leak the fdt or the unflattened tree on overlay destruction
> but that duplicates, no?
>
>> frees a devicetree would also have to be aware of this change -- I'm not
>> sure if that leads to ugly complications or if it is easy.  The other
>> question to consider is whether to make the same change to
>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>> the base devicetree.  Doing so would increase the memory usage of the
>> live tree (we would not be able to free the base fdt after unflattening
>> it because we make the fdt visible in /sys/firmware/fdt -- though
>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>
>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>> into the overlay unflattened devicetree are then passed to the notifiers.
>> (Again, I may be missing some other place that the overlay unflattened
>> devicetree is made visible to other code -- a more thorough reading of
>> the code is needed.) If the notifiers could be modified to accept the
>> changeset list instead of of pointers to the fragments in the overlay
>> unflattened devicetree then there would be no possibility of the notifiers
>> keeping a pointer into the overlay fdt. I do not know if this is a
>
> But then again the convention has to be that those changeset pointers
> must not be kept - because the changeset is history after of_overlay_remove.
>
>> practical change for the notifiers -- there are no callers of
>> of_overlay_notifier_register() in the mainline kernel source. My
>> recollection is that the overlay notifiers were added for the fpga
>> subsystem.

That's right.

>
> We have drivers/fpga/of-fpga-region.c in-tree, and that does not seem to
> store any pointers to objects, rather consumes them in-place. And I
> would consider it fair to impose such a limitation on the notifier
> interface.

The FPGA code was written assuming that overlays could be removed.

>
>>
>> Why is overlay_notify() the only issue related to unknown users having
>> pointers into the overlay fdt?  The answer is that the overlay code
>> does not directly expose the overlay unflattened devicetree (and thus
>> indirectly the overlay fdt) to the live devicetree -- when the
>> overlay code creates the overlay changeset, it copies from the
>> overlay unflattened devicetree and overlay fdt and only exposes
>> pointers to the copies.
>>
>> And hopefully the issues with the overlay unflattened devicetree can
>> be resolved in the same way as for the overlay fdt.
>
> As noted above, I don't see there is a technical solution to this issue
> but it's rather a matter of convention: no overlay notifier callback is
> allowed to keep references to the passed tree content (unless it
> reference-counts some tree nodes) beyond the execution of the callback.
> With that in place, we can safely drop the backing memory IMHO.
>
> Jan

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Alan Tull]

On Tue, Apr 24, 2018 at 11:08 AM, Alan Tull <atull@kernel.org> wrote:
> On Tue, Apr 24, 2018 at 12:29 AM, Jan Kiszka <jan.kiszka@web.de> wrote:
>> On 2018-04-24 00:38, Frank Rowand wrote:
>>> Hi Jan,
>>>
>>> + Alan Tull for fpga perspective
>>>
>>> On 04/22/18 03:30, Jan Kiszka wrote:
>>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>>> Hi Jan,
>>>>>>>>>
>>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>>> Hi Frank,
>>>>>>>>>>
>>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>>
>>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>>
>>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>>> original FDT.
>>>>>>>>>>>
>>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>>
>>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>>> errors.
>>>>>>>>>>>
>>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>>> overlay loader.
>>>>>>>>>>
>>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>>
>>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>>
>>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>>> understand the use case and consider solutions.
>>>>>>>>>
>>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>>
>>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>>
>>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>>> scripts/dtc/libfdt/.
>>>>>>
>>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>>> is little reason to other than for early boot changes. And it is much
>>>>>> easier to work on unflattened trees.
>>>>>
>>>>> I just briefly looked into libfdt, and it would have meant building it
>>>>> into the module as there are no library functions exported by the kernel
>>>>> either. Another reason to drop that.
>>>>>
>>>>> What's apparently working now is the pattern I initially suggested:
>>>>> Register template with status = "disabled" as overlay, then prepare and
>>>>> apply changeset that contains all needed modifications and sets the
>>>>> status to "ok". I might be leaking additional resources, but to find
>>>>> that out, I will now finally have to resolve clean unbinding of the
>>>>> generic PCI host controller [1] first.
>>>>
>>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>>> {
>>>>      [...]
>>>>      /*
>>>>       * TODO
>>>>       *
>>>>       * would like to: kfree(ovcs->overlay_tree);
>>>>       * but can not since drivers may have pointers into this data
>>>>       *
>>>>       * would like to: kfree(ovcs->fdt);
>>>>       * but can not since drivers may have pointers into this data
>>>>       */
>>>>
>>>>      kfree(ovcs);
>>>> }
>>>>
>>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>>> to those objects. I would say that's a regression of the new API.
>>>
>>> The problem already existed but it was hidden.  We have never been able to
>>> kfree() these object because we do not know if there are any pointers into
>>> these objects.  The new API makes the problem visible to kmemleak.
>>
>> My old code didn't have the problem because there was no one steeling
>> pointers to my overlay, and I was able to safely release all the
>> resources that I or the core on my behalf allocated. In fact, I recently
>> even dropped the duplication the fdt prior to unflattening it because I
>> got its lifecycle under control (and both kmemleak as well as kasan
>> confirmed this). I still consider this intentional leak a regression of
>> the new API.
>>
>>>
>>> The reason that we do not know if there are any pointers into these objects
>>> is that devicetree access APIs return pointers into the devicetree internal
>>> data structures (that is, into the overlay unflattened devicetree).  If we
>>> want to be able to do the kfree()s, we could change the devicetree access
>>> APIs.
>>>
>>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>>> also exposed is that the overlay unflattened devicetree property values
>>> are pointers into the overlay fdt.
>>>
>>> ** This paragraph becomes academic (and not needed) if the fix in the next
>>> paragraph can be implemented. **
>>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>>> (I would want to read through the code again to make sure I'm not missing
>>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>>> was modified so that property values were copied into newly allocated memory
>>> and the live tree property pointers were set to the copy instead of to
>>> the value in the fdt, then I _think_ the fdt could be freed in
>>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>>
>> I don't see yet how more duplicating of objects would help. Then we
>> would not leak the fdt or the unflattened tree on overlay destruction
>> but that duplicates, no?
>>
>>> frees a devicetree would also have to be aware of this change -- I'm not
>>> sure if that leads to ugly complications or if it is easy.  The other
>>> question to consider is whether to make the same change to
>>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>>> the base devicetree.  Doing so would increase the memory usage of the
>>> live tree (we would not be able to free the base fdt after unflattening
>>> it because we make the fdt visible in /sys/firmware/fdt -- though
>>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>>
>>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>>> into the overlay unflattened devicetree are then passed to the notifiers.
>>> (Again, I may be missing some other place that the overlay unflattened
>>> devicetree is made visible to other code -- a more thorough reading of
>>> the code is needed.) If the notifiers could be modified to accept the
>>> changeset list instead of of pointers to the fragments in the overlay
>>> unflattened devicetree then there would be no possibility of the notifiers
>>> keeping a pointer into the overlay fdt. I do not know if this is a
>>
>> But then again the convention has to be that those changeset pointers
>> must not be kept - because the changeset is history after of_overlay_remove.
>>
>>> practical change for the notifiers -- there are no callers of
>>> of_overlay_notifier_register() in the mainline kernel source. My
>>> recollection is that the overlay notifiers were added for the fpga
>>> subsystem.
>
> That's right.
>
>>
>> We have drivers/fpga/of-fpga-region.c in-tree, and that does not seem to
>> store any pointers to objects, rather consumes them in-place. And I
>> would consider it fair to impose such a limitation on the notifier
>> interface.
>
> The FPGA code was written assuming that overlays could be removed.

To be more specific, drivers/fpga/of-fpga-region.c currently saves a
pointer to the overlay and uses it only during the pre-apply
notification.

>
>>
>>>
>>> Why is overlay_notify() the only issue related to unknown users having
>>> pointers into the overlay fdt?  The answer is that the overlay code
>>> does not directly expose the overlay unflattened devicetree (and thus
>>> indirectly the overlay fdt) to the live devicetree -- when the
>>> overlay code creates the overlay changeset, it copies from the
>>> overlay unflattened devicetree and overlay fdt and only exposes
>>> pointers to the copies.
>>>
>>> And hopefully the issues with the overlay unflattened devicetree can
>>> be resolved in the same way as for the overlay fdt.
>>
>> As noted above, I don't see there is a technical solution to this issue
>> but it's rather a matter of convention: no overlay notifier callback is
>> allowed to keep references to the passed tree content (unless it
>> reference-counts some tree nodes) beyond the execution of the callback.
>> With that in place, we can safely drop the backing memory IMHO.
>>
>> Jan

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Jan Kiszka]

On 2018-04-24 20:26, Alan Tull wrote:
> On Tue, Apr 24, 2018 at 11:08 AM, Alan Tull <atull@kernel.org> wrote:
>> On Tue, Apr 24, 2018 at 12:29 AM, Jan Kiszka <jan.kiszka@web.de> wrote:
>>>
>>> We have drivers/fpga/of-fpga-region.c in-tree, and that does not seem to
>>> store any pointers to objects, rather consumes them in-place. And I
>>> would consider it fair to impose such a limitation on the notifier
>>> interface.
>>
>> The FPGA code was written assuming that overlays could be removed.
> 
> To be more specific, drivers/fpga/of-fpga-region.c currently saves a
> pointer to the overlay and uses it only during the pre-apply
> notification.

Even more exactly: You are saving a reference during pre-apply and
freeing that one again on the corresponding post-remove. That is what I
would have expected as normal usage of the notification API as well.

Jan

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

On 04/23/18 22:29, Jan Kiszka wrote:
> On 2018-04-24 00:38, Frank Rowand wrote:
>> Hi Jan,
>>
>> + Alan Tull for fpga perspective
>>
>> On 04/22/18 03:30, Jan Kiszka wrote:
>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>> Hi Jan,
>>>>>>>>
>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>> Hi Frank,
>>>>>>>>>
>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>
>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>
>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>> original FDT.
>>>>>>>>>>
>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>
>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>> errors.
>>>>>>>>>>
>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>> overlay loader.
>>>>>>>>>
>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>
>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>
>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>> understand the use case and consider solutions.
>>>>>>>>
>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>
>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>
>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>> scripts/dtc/libfdt/.
>>>>>
>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>> is little reason to other than for early boot changes. And it is much
>>>>> easier to work on unflattened trees.
>>>>
>>>> I just briefly looked into libfdt, and it would have meant building it
>>>> into the module as there are no library functions exported by the kernel
>>>> either. Another reason to drop that.
>>>>
>>>> What's apparently working now is the pattern I initially suggested:
>>>> Register template with status = "disabled" as overlay, then prepare and
>>>> apply changeset that contains all needed modifications and sets the
>>>> status to "ok". I might be leaking additional resources, but to find
>>>> that out, I will now finally have to resolve clean unbinding of the
>>>> generic PCI host controller [1] first.
>>>
>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>> {
>>> 	[...]
>>> 	/*
>>> 	 * TODO
>>> 	 *
>>> 	 * would like to: kfree(ovcs->overlay_tree);
>>> 	 * but can not since drivers may have pointers into this data
>>> 	 *
>>> 	 * would like to: kfree(ovcs->fdt);
>>> 	 * but can not since drivers may have pointers into this data
>>> 	 */
>>>
>>> 	kfree(ovcs);
>>> }
>>>
>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>> to those objects. I would say that's a regression of the new API.
>>
>> The problem already existed but it was hidden.  We have never been able to
>> kfree() these object because we do not know if there are any pointers into
>> these objects.  The new API makes the problem visible to kmemleak.
> 
> My old code didn't have the problem because there was no one steeling
> pointers to my overlay, and I was able to safely release all the
> resources that I or the core on my behalf allocated. In fact, I recently
> even dropped the duplication the fdt prior to unflattening it because I
> got its lifecycle under control (and both kmemleak as well as kasan
> confirmed this). I still consider this intentional leak a regression of
> the new API.

The API has to work for any user, not just your clean code.


>> The reason that we do not know if there are any pointers into these objects
>> is that devicetree access APIs return pointers into the devicetree internal
>> data structures (that is, into the overlay unflattened devicetree).  If we
>> want to be able to do the kfree()s, we could change the devicetree access
>> APIs.
>>
>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>> also exposed is that the overlay unflattened devicetree property values
>> are pointers into the overlay fdt.
>>
>> ** This paragraph becomes academic (and not needed) if the fix in the next
>> paragraph can be implemented. **
>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>> (I would want to read through the code again to make sure I'm not missing
>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>> was modified so that property values were copied into newly allocated memory
>> and the live tree property pointers were set to the copy instead of to
>> the value in the fdt, then I _think_ the fdt could be freed in
>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
> 
> I don't see yet how more duplicating of objects would help. Then we
> would not leak the fdt or the unflattened tree on overlay destruction
> but that duplicates, no?

Yes, we would leak the duplicates.  That is exactly what the existing
overlay remove code does.  My long term goal is to remove that leakage.
But that leakage can not be resolved until we can guarantee that there
are no pointers held to those duplicates.

I don't like adding this additional copy - I would much prefer to change
the overlay notify code as proposed below.


>> frees a devicetree would also have to be aware of this change -- I'm not
>> sure if that leads to ugly complications or if it is easy.  The other
>> question to consider is whether to make the same change to
>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>> the base devicetree.  Doing so would increase the memory usage of the
>> live tree (we would not be able to free the base fdt after unflattening
>> it because we make the fdt visible in /sys/firmware/fdt -- though
>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>
>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>> into the overlay unflattened devicetree are then passed to the notifiers.
>> (Again, I may be missing some other place that the overlay unflattened
>> devicetree is made visible to other code -- a more thorough reading of
>> the code is needed.) If the notifiers could be modified to accept the
>> changeset list instead of of pointers to the fragments in the overlay
>> unflattened devicetree then there would be no possibility of the notifiers
>> keeping a pointer into the overlay fdt. I do not know if this is a
> 
> But then again the convention has to be that those changeset pointers
> must not be kept - because the changeset is history after of_overlay_remove.

I don't trust convention.  The result is fragile code.


>> practical change for the notifiers -- there are no callers of
>> of_overlay_notifier_register() in the mainline kernel source. My
>> recollection is that the overlay notifiers were added for the fpga
>> subsystem.
> 
> We have drivers/fpga/of-fpga-region.c in-tree, and that does not seem to

Thanks for finding that.  For some reason my 'git grep' did not find
that.  (I'll blame fat fingers or something...)


> store any pointers to objects, rather consumes them in-place. And I
> would consider it fair to impose such a limitation on the notifier
> interface.

How do you enforce that limitation?


>> Why is overlay_notify() the only issue related to unknown users having
>> pointers into the overlay fdt?  The answer is that the overlay code
>> does not directly expose the overlay unflattened devicetree (and thus
>> indirectly the overlay fdt) to the live devicetree -- when the
>> overlay code creates the overlay changeset, it copies from the
>> overlay unflattened devicetree and overlay fdt and only exposes
>> pointers to the copies.
>>
>> And hopefully the issues with the overlay unflattened devicetree can
>> be resolved in the same way as for the overlay fdt.
> 
> As noted above, I don't see there is a technical solution to this issue
> but it's rather a matter of convention: no overlay notifier callback is
> allowed to keep references to the passed tree content (unless it
> reference-counts some tree nodes) beyond the execution of the callback.
> With that in place, we can safely drop the backing memory IMHO.
> 
> Jan
> .
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Jan Kiszka]

On 2018-04-24 23:15, Frank Rowand wrote:
> On 04/23/18 22:29, Jan Kiszka wrote:
>> On 2018-04-24 00:38, Frank Rowand wrote:
>>> Hi Jan,
>>>
>>> + Alan Tull for fpga perspective
>>>
>>> On 04/22/18 03:30, Jan Kiszka wrote:
>>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>>> Hi Jan,
>>>>>>>>>
>>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>>> Hi Frank,
>>>>>>>>>>
>>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>>
>>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>>
>>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>>> original FDT.
>>>>>>>>>>>
>>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>>
>>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>>> errors.
>>>>>>>>>>>
>>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>>> overlay loader.
>>>>>>>>>>
>>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>>
>>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>>
>>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>>> understand the use case and consider solutions.
>>>>>>>>>
>>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>>
>>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>>
>>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>>> scripts/dtc/libfdt/.
>>>>>>
>>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>>> is little reason to other than for early boot changes. And it is much
>>>>>> easier to work on unflattened trees.
>>>>>
>>>>> I just briefly looked into libfdt, and it would have meant building it
>>>>> into the module as there are no library functions exported by the kernel
>>>>> either. Another reason to drop that.
>>>>>
>>>>> What's apparently working now is the pattern I initially suggested:
>>>>> Register template with status = "disabled" as overlay, then prepare and
>>>>> apply changeset that contains all needed modifications and sets the
>>>>> status to "ok". I might be leaking additional resources, but to find
>>>>> that out, I will now finally have to resolve clean unbinding of the
>>>>> generic PCI host controller [1] first.
>>>>
>>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>>> {
>>>> 	[...]
>>>> 	/*
>>>> 	 * TODO
>>>> 	 *
>>>> 	 * would like to: kfree(ovcs->overlay_tree);
>>>> 	 * but can not since drivers may have pointers into this data
>>>> 	 *
>>>> 	 * would like to: kfree(ovcs->fdt);
>>>> 	 * but can not since drivers may have pointers into this data
>>>> 	 */
>>>>
>>>> 	kfree(ovcs);
>>>> }
>>>>
>>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>>> to those objects. I would say that's a regression of the new API.
>>>
>>> The problem already existed but it was hidden.  We have never been able to
>>> kfree() these object because we do not know if there are any pointers into
>>> these objects.  The new API makes the problem visible to kmemleak.
>>
>> My old code didn't have the problem because there was no one steeling
>> pointers to my overlay, and I was able to safely release all the
>> resources that I or the core on my behalf allocated. In fact, I recently
>> even dropped the duplication the fdt prior to unflattening it because I
>> got its lifecycle under control (and both kmemleak as well as kasan
>> confirmed this). I still consider this intentional leak a regression of
>> the new API.
> 
> The API has to work for any user, not just your clean code.
> 

Please point us to code that does have a real problem. Is there any nice
vendor tree, in addition to the known users? Or are we only speculating
about how people might be (mis)using the API?

> 
>>> The reason that we do not know if there are any pointers into these objects
>>> is that devicetree access APIs return pointers into the devicetree internal
>>> data structures (that is, into the overlay unflattened devicetree).  If we
>>> want to be able to do the kfree()s, we could change the devicetree access
>>> APIs.
>>>
>>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>>> also exposed is that the overlay unflattened devicetree property values
>>> are pointers into the overlay fdt.
>>>
>>> ** This paragraph becomes academic (and not needed) if the fix in the next
>>> paragraph can be implemented. **
>>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>>> (I would want to read through the code again to make sure I'm not missing
>>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>>> was modified so that property values were copied into newly allocated memory
>>> and the live tree property pointers were set to the copy instead of to
>>> the value in the fdt, then I _think_ the fdt could be freed in
>>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>>
>> I don't see yet how more duplicating of objects would help. Then we
>> would not leak the fdt or the unflattened tree on overlay destruction
>> but that duplicates, no?
> 
> Yes, we would leak the duplicates.  That is exactly what the existing
> overlay remove code does.  My long term goal is to remove that leakage.
> But that leakage can not be resolved until we can guarantee that there
> are no pointers held to those duplicates.
> 
> I don't like adding this additional copy - I would much prefer to change
> the overlay notify code as proposed below.
> 

Replacing one leak with another is no solution. And if it's additionally
enforcing an API change, I would call it counterproductive.

> 
>>> frees a devicetree would also have to be aware of this change -- I'm not
>>> sure if that leads to ugly complications or if it is easy.  The other
>>> question to consider is whether to make the same change to
>>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>>> the base devicetree.  Doing so would increase the memory usage of the
>>> live tree (we would not be able to free the base fdt after unflattening
>>> it because we make the fdt visible in /sys/firmware/fdt -- though
>>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>>
>>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>>> into the overlay unflattened devicetree are then passed to the notifiers.
>>> (Again, I may be missing some other place that the overlay unflattened
>>> devicetree is made visible to other code -- a more thorough reading of
>>> the code is needed.) If the notifiers could be modified to accept the
>>> changeset list instead of of pointers to the fragments in the overlay
>>> unflattened devicetree then there would be no possibility of the notifiers
>>> keeping a pointer into the overlay fdt. I do not know if this is a
>>
>> But then again the convention has to be that those changeset pointers
>> must not be kept - because the changeset is history after of_overlay_remove.
> 
> I don't trust convention.  The result is fragile code.
> 

Look, we are all programming in C here. There is no implicit reference
counting, no garbage collecting, not strong typing, you-name-it. That
doesn't leave you with many sharper weapons than well documented
conventions.

I'm rather concerned that you are over-designing an API that, due to its
nature, cannot be made foolproof.

> 
>>> practical change for the notifiers -- there are no callers of
>>> of_overlay_notifier_register() in the mainline kernel source. My
>>> recollection is that the overlay notifiers were added for the fpga
>>> subsystem.
>>
>> We have drivers/fpga/of-fpga-region.c in-tree, and that does not seem to
> 
> Thanks for finding that.  For some reason my 'git grep' did not find
> that.  (I'll blame fat fingers or something...)
> 
> 
>> store any pointers to objects, rather consumes them in-place. And I
>> would consider it fair to impose such a limitation on the notifier
>> interface.
> 
> How do you enforce that limitation?
> 

Primarily, code review. Of course, we can't help out-of-tree adventurers
this way but, well, they prefer to travel alone anyway.

And then there are also tools like kasan that can be very helpful
revealing object lifecycle issues early.

Jan

> 
>>> Why is overlay_notify() the only issue related to unknown users having
>>> pointers into the overlay fdt?  The answer is that the overlay code
>>> does not directly expose the overlay unflattened devicetree (and thus
>>> indirectly the overlay fdt) to the live devicetree -- when the
>>> overlay code creates the overlay changeset, it copies from the
>>> overlay unflattened devicetree and overlay fdt and only exposes
>>> pointers to the copies.
>>>
>>> And hopefully the issues with the overlay unflattened devicetree can
>>> be resolved in the same way as for the overlay fdt.
>>
>> As noted above, I don't see there is a technical solution to this issue
>> but it's rather a matter of convention: no overlay notifier callback is
>> allowed to keep references to the passed tree content (unless it
>> reference-counts some tree nodes) beyond the execution of the callback.
>> With that in place, we can safely drop the backing memory IMHO.
>>
>> Jan
>> .
>>
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

On 04/24/18 22:22, Jan Kiszka wrote:
> On 2018-04-24 23:15, Frank Rowand wrote:
>> On 04/23/18 22:29, Jan Kiszka wrote:
>>> On 2018-04-24 00:38, Frank Rowand wrote:
>>>> Hi Jan,
>>>>
>>>> + Alan Tull for fpga perspective
>>>>
>>>> On 04/22/18 03:30, Jan Kiszka wrote:
>>>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>>>> Hi Jan,
>>>>>>>>>>
>>>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>
>>>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>>>
>>>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>>>
>>>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>>>> original FDT.
>>>>>>>>>>>>
>>>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>>>
>>>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>>>> errors.
>>>>>>>>>>>>
>>>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>>>> overlay loader.
>>>>>>>>>>>
>>>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>>>
>>>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>>>
>>>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>>>> understand the use case and consider solutions.
>>>>>>>>>>
>>>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>>>
>>>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>>>
>>>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>>>> scripts/dtc/libfdt/.
>>>>>>>
>>>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>>>> is little reason to other than for early boot changes. And it is much
>>>>>>> easier to work on unflattened trees.
>>>>>>
>>>>>> I just briefly looked into libfdt, and it would have meant building it
>>>>>> into the module as there are no library functions exported by the kernel
>>>>>> either. Another reason to drop that.
>>>>>>
>>>>>> What's apparently working now is the pattern I initially suggested:
>>>>>> Register template with status = "disabled" as overlay, then prepare and
>>>>>> apply changeset that contains all needed modifications and sets the
>>>>>> status to "ok". I might be leaking additional resources, but to find
>>>>>> that out, I will now finally have to resolve clean unbinding of the
>>>>>> generic PCI host controller [1] first.
>>>>>
>>>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>>>> {
>>>>> 	[...]
>>>>> 	/*
>>>>> 	 * TODO
>>>>> 	 *
>>>>> 	 * would like to: kfree(ovcs->overlay_tree);
>>>>> 	 * but can not since drivers may have pointers into this data
>>>>> 	 *
>>>>> 	 * would like to: kfree(ovcs->fdt);
>>>>> 	 * but can not since drivers may have pointers into this data
>>>>> 	 */
>>>>>
>>>>> 	kfree(ovcs);
>>>>> }
>>>>>
>>>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>>>> to those objects. I would say that's a regression of the new API.
>>>>
>>>> The problem already existed but it was hidden.  We have never been able to
>>>> kfree() these object because we do not know if there are any pointers into
>>>> these objects.  The new API makes the problem visible to kmemleak.
>>>
>>> My old code didn't have the problem because there was no one steeling
>>> pointers to my overlay, and I was able to safely release all the
>>> resources that I or the core on my behalf allocated. In fact, I recently
>>> even dropped the duplication the fdt prior to unflattening it because I
>>> got its lifecycle under control (and both kmemleak as well as kasan
>>> confirmed this). I still consider this intentional leak a regression of
>>> the new API.
>>
>> The API has to work for any user, not just your clean code.
>>
> 
> Please point us to code that does have a real problem. Is there any nice
> vendor tree, in addition to the known users? Or are we only speculating
> about how people might be (mis)using the API?

No, I will not even attempt to find such code.

The underlying problem is that the devicetree access API returns pointers
into the devicetree.  And we have no way of knowing whether any driver or
subsystem has a live pointer into the overlay data in the devicetree when
we remove an overlay.  The devicetree access API did not anticipate and
account for overlays.  As overlay related code has been added, this is an
issue that has not yet been fixed.


>>
>>>> The reason that we do not know if there are any pointers into these objects
>>>> is that devicetree access APIs return pointers into the devicetree internal
>>>> data structures (that is, into the overlay unflattened devicetree).  If we
>>>> want to be able to do the kfree()s, we could change the devicetree access
>>>> APIs.
>>>>
>>>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>>>> also exposed is that the overlay unflattened devicetree property values
>>>> are pointers into the overlay fdt.
>>>>
>>>> ** This paragraph becomes academic (and not needed) if the fix in the next
>>>> paragraph can be implemented. **
>>>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>>>> (I would want to read through the code again to make sure I'm not missing
>>>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>>>> was modified so that property values were copied into newly allocated memory
>>>> and the live tree property pointers were set to the copy instead of to
>>>> the value in the fdt, then I _think_ the fdt could be freed in
>>>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>>>
>>> I don't see yet how more duplicating of objects would help. Then we
>>> would not leak the fdt or the unflattened tree on overlay destruction
>>> but that duplicates, no?
>>
>> Yes, we would leak the duplicates.  That is exactly what the existing
>> overlay remove code does.  My long term goal is to remove that leakage.
>> But that leakage can not be resolved until we can guarantee that there
>> are no pointers held to those duplicates.
>>
>> I don't like adding this additional copy - I would much prefer to change
>> the overlay notify code as proposed below.
>>
> 
> Replacing one leak with another is no solution.

I agree.  I do not see it as a viable solution.


> And if it's additionally
> enforcing an API change, I would call it counterproductive.
> 
>>
>>>> frees a devicetree would also have to be aware of this change -- I'm not
>>>> sure if that leads to ugly complications or if it is easy.  The other
>>>> question to consider is whether to make the same change to
>>>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>>>> the base devicetree.  Doing so would increase the memory usage of the
>>>> live tree (we would not be able to free the base fdt after unflattening
>>>> it because we make the fdt visible in /sys/firmware/fdt -- though
>>>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>>>
>>>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>>>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>>>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>>>> into the overlay unflattened devicetree are then passed to the notifiers.
>>>> (Again, I may be missing some other place that the overlay unflattened
>>>> devicetree is made visible to other code -- a more thorough reading of
>>>> the code is needed.) If the notifiers could be modified to accept the
>>>> changeset list instead of of pointers to the fragments in the overlay
>>>> unflattened devicetree then there would be no possibility of the notifiers
>>>> keeping a pointer into the overlay fdt. I do not know if this is a
>>>
>>> But then again the convention has to be that those changeset pointers
>>> must not be kept - because the changeset is history after of_overlay_remove.
>>
>> I don't trust convention.  The result is fragile code.
>>
> 
> Look, we are all programming in C here. There is no implicit reference
> counting, no garbage collecting, not strong typing, you-name-it. That
> doesn't leave you with many sharper weapons than well documented
> conventions.

Nope.  We can (hopefully) modify the devicetree access API so that it
does not return pointers into the devicetree.  For property values,
this is "easy", though at a cost.  Where the API currently returns
a pointer to a property (or property value), copy that data into
newly allocated memory and return a pointer to that newly allocated
memory -- the caller is now responsible for the new memory and there
is no stray pointer into the devicetree.  One other place that pointers
into the devicetree are exposed are the tree traversal APIs.  In theory
it should be possible to create an API that uses opaque "iterators"
(I'm probably mis-using that word) so that location in the tree while
traversing is not exposed in the form of a pointer into the devicetree.

-Frank

> I'm rather concerned that you are over-designing an API that, due to its
> nature, cannot be made foolproof.
> 
>>
>>>> practical change for the notifiers -- there are no callers of
>>>> of_overlay_notifier_register() in the mainline kernel source. My
>>>> recollection is that the overlay notifiers were added for the fpga
>>>> subsystem.
>>>
>>> We have drivers/fpga/of-fpga-region.c in-tree, and that does not seem to
>>
>> Thanks for finding that.  For some reason my 'git grep' did not find
>> that.  (I'll blame fat fingers or something...)
>>
>>
>>> store any pointers to objects, rather consumes them in-place. And I
>>> would consider it fair to impose such a limitation on the notifier
>>> interface.
>>
>> How do you enforce that limitation?
>>
> 
> Primarily, code review. Of course, we can't help out-of-tree adventurers
> this way but, well, they prefer to travel alone anyway.
> 
> And then there are also tools like kasan that can be very helpful
> revealing object lifecycle issues early.
> 
> Jan
> 
>>
>>>> Why is overlay_notify() the only issue related to unknown users having
>>>> pointers into the overlay fdt?  The answer is that the overlay code
>>>> does not directly expose the overlay unflattened devicetree (and thus
>>>> indirectly the overlay fdt) to the live devicetree -- when the
>>>> overlay code creates the overlay changeset, it copies from the
>>>> overlay unflattened devicetree and overlay fdt and only exposes
>>>> pointers to the copies.
>>>>
>>>> And hopefully the issues with the overlay unflattened devicetree can
>>>> be resolved in the same way as for the overlay fdt.
>>>
>>> As noted above, I don't see there is a technical solution to this issue
>>> but it's rather a matter of convention: no overlay notifier callback is
>>> allowed to keep references to the passed tree content (unless it
>>> reference-counts some tree nodes) beyond the execution of the callback.
>>> With that in place, we can safely drop the backing memory IMHO.
>>>
>>> Jan
>>> .
>>>
>>
> 
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

On 04/25/18 11:56, Frank Rowand wrote:
> On 04/24/18 22:22, Jan Kiszka wrote:
>> On 2018-04-24 23:15, Frank Rowand wrote:
>>> On 04/23/18 22:29, Jan Kiszka wrote:
>>>> On 2018-04-24 00:38, Frank Rowand wrote:
>>>>> Hi Jan,
>>>>>
>>>>> + Alan Tull for fpga perspective
>>>>>
>>>>> On 04/22/18 03:30, Jan Kiszka wrote:
>>>>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>>>>> Hi Jan,
>>>>>>>>>>>
>>>>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>
>>>>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>>>>
>>>>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>>>>> original FDT.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>>>>
>>>>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>>>>> errors.
>>>>>>>>>>>>>
>>>>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>>>>> overlay loader.
>>>>>>>>>>>>
>>>>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>>>>
>>>>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>>>>> understand the use case and consider solutions.
>>>>>>>>>>>
>>>>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>>>>
>>>>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>>>>
>>>>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>>>>> scripts/dtc/libfdt/.
>>>>>>>>
>>>>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>>>>> is little reason to other than for early boot changes. And it is much
>>>>>>>> easier to work on unflattened trees.
>>>>>>>
>>>>>>> I just briefly looked into libfdt, and it would have meant building it
>>>>>>> into the module as there are no library functions exported by the kernel
>>>>>>> either. Another reason to drop that.
>>>>>>>
>>>>>>> What's apparently working now is the pattern I initially suggested:
>>>>>>> Register template with status = "disabled" as overlay, then prepare and
>>>>>>> apply changeset that contains all needed modifications and sets the
>>>>>>> status to "ok". I might be leaking additional resources, but to find
>>>>>>> that out, I will now finally have to resolve clean unbinding of the
>>>>>>> generic PCI host controller [1] first.
>>>>>>
>>>>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>>>>> {
>>>>>> 	[...]
>>>>>> 	/*
>>>>>> 	 * TODO
>>>>>> 	 *
>>>>>> 	 * would like to: kfree(ovcs->overlay_tree);
>>>>>> 	 * but can not since drivers may have pointers into this data
>>>>>> 	 *
>>>>>> 	 * would like to: kfree(ovcs->fdt);
>>>>>> 	 * but can not since drivers may have pointers into this data
>>>>>> 	 */
>>>>>>
>>>>>> 	kfree(ovcs);
>>>>>> }
>>>>>>
>>>>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>>>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>>>>> to those objects. I would say that's a regression of the new API.
>>>>>
>>>>> The problem already existed but it was hidden.  We have never been able to
>>>>> kfree() these object because we do not know if there are any pointers into
>>>>> these objects.  The new API makes the problem visible to kmemleak.
>>>>
>>>> My old code didn't have the problem because there was no one steeling
>>>> pointers to my overlay, and I was able to safely release all the
>>>> resources that I or the core on my behalf allocated. In fact, I recently
>>>> even dropped the duplication the fdt prior to unflattening it because I
>>>> got its lifecycle under control (and both kmemleak as well as kasan
>>>> confirmed this). I still consider this intentional leak a regression of
>>>> the new API.
>>>
>>> The API has to work for any user, not just your clean code.
>>>
>>
>> Please point us to code that does have a real problem. Is there any nice
>> vendor tree, in addition to the known users? Or are we only speculating
>> about how people might be (mis)using the API?
> 
> No, I will not even attempt to find such code.
> 
> The underlying problem is that the devicetree access API returns pointers
> into the devicetree.  And we have no way of knowing whether any driver or
> subsystem has a live pointer into the overlay data in the devicetree when
> we remove an overlay.  The devicetree access API did not anticipate and
> account for overlays.  As overlay related code has been added, this is an
> issue that has not yet been fixed.
> 
> 
>>>
>>>>> The reason that we do not know if there are any pointers into these objects
>>>>> is that devicetree access APIs return pointers into the devicetree internal
>>>>> data structures (that is, into the overlay unflattened devicetree).  If we
>>>>> want to be able to do the kfree()s, we could change the devicetree access
>>>>> APIs.
>>>>>
>>>>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>>>>> also exposed is that the overlay unflattened devicetree property values
>>>>> are pointers into the overlay fdt.
>>>>>
>>>>> ** This paragraph becomes academic (and not needed) if the fix in the next
>>>>> paragraph can be implemented. **
>>>>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>>>>> (I would want to read through the code again to make sure I'm not missing
>>>>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>>>>> was modified so that property values were copied into newly allocated memory
>>>>> and the live tree property pointers were set to the copy instead of to
>>>>> the value in the fdt, then I _think_ the fdt could be freed in
>>>>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>>>>
>>>> I don't see yet how more duplicating of objects would help. Then we
>>>> would not leak the fdt or the unflattened tree on overlay destruction
>>>> but that duplicates, no?
>>>
>>> Yes, we would leak the duplicates.  That is exactly what the existing
>>> overlay remove code does.  My long term goal is to remove that leakage.
>>> But that leakage can not be resolved until we can guarantee that there
>>> are no pointers held to those duplicates.
>>>
>>> I don't like adding this additional copy - I would much prefer to change
>>> the overlay notify code as proposed below.
>>>
>>
>> Replacing one leak with another is no solution.
> 
> I agree.  I do not see it as a viable solution.
> 
> 
>> And if it's additionally
>> enforcing an API change, I would call it counterproductive.
>>
>>>
>>>>> frees a devicetree would also have to be aware of this change -- I'm not
>>>>> sure if that leads to ugly complications or if it is easy.  The other
>>>>> question to consider is whether to make the same change to
>>>>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>>>>> the base devicetree.  Doing so would increase the memory usage of the
>>>>> live tree (we would not be able to free the base fdt after unflattening
>>>>> it because we make the fdt visible in /sys/firmware/fdt -- though
>>>>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>>>>
>>>>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>>>>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>>>>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>>>>> into the overlay unflattened devicetree are then passed to the notifiers.
>>>>> (Again, I may be missing some other place that the overlay unflattened
>>>>> devicetree is made visible to other code -- a more thorough reading of
>>>>> the code is needed.) If the notifiers could be modified to accept the
>>>>> changeset list instead of of pointers to the fragments in the overlay
>>>>> unflattened devicetree then there would be no possibility of the notifiers
>>>>> keeping a pointer into the overlay fdt. I do not know if this is a
>>>>
>>>> But then again the convention has to be that those changeset pointers
>>>> must not be kept - because the changeset is history after of_overlay_remove.
>>>
>>> I don't trust convention.  The result is fragile code.
>>>
>>
>> Look, we are all programming in C here. There is no implicit reference
>> counting, no garbage collecting, not strong typing, you-name-it. That
>> doesn't leave you with many sharper weapons than well documented
>> conventions.
> 
> Nope.  We can (hopefully) modify the devicetree access API so that it
> does not return pointers into the devicetree.  For property values,
> this is "easy", though at a cost.  Where the API currently returns
> a pointer to a property (or property value), copy that data into
> newly allocated memory and return a pointer to that newly allocated

Or the data could be copied to memory designated by a pointer that the
caller passed in.  I was not intending this paragraph to be an actual
thought out design for an API, it is just an attempt to describe what
needs to change.


> memory -- the caller is now responsible for the new memory and there
> is no stray pointer into the devicetree.  One other place that pointers
> into the devicetree are exposed are the tree traversal APIs.  In theory
> it should be possible to create an API that uses opaque "iterators"
> (I'm probably mis-using that word) so that location in the tree while
> traversing is not exposed in the form of a pointer into the devicetree.
> 
> -Frank
> 
>> I'm rather concerned that you are over-designing an API that, due to its
>> nature, cannot be made foolproof.
>>
>>>
>>>>> practical change for the notifiers -- there are no callers of
>>>>> of_overlay_notifier_register() in the mainline kernel source. My
>>>>> recollection is that the overlay notifiers were added for the fpga
>>>>> subsystem.
>>>>
>>>> We have drivers/fpga/of-fpga-region.c in-tree, and that does not seem to
>>>
>>> Thanks for finding that.  For some reason my 'git grep' did not find
>>> that.  (I'll blame fat fingers or something...)
>>>
>>>
>>>> store any pointers to objects, rather consumes them in-place. And I
>>>> would consider it fair to impose such a limitation on the notifier
>>>> interface.
>>>
>>> How do you enforce that limitation?
>>>
>>
>> Primarily, code review. Of course, we can't help out-of-tree adventurers
>> this way but, well, they prefer to travel alone anyway.
>>
>> And then there are also tools like kasan that can be very helpful
>> revealing object lifecycle issues early.
>>
>> Jan
>>
>>>
>>>>> Why is overlay_notify() the only issue related to unknown users having
>>>>> pointers into the overlay fdt?  The answer is that the overlay code
>>>>> does not directly expose the overlay unflattened devicetree (and thus
>>>>> indirectly the overlay fdt) to the live devicetree -- when the
>>>>> overlay code creates the overlay changeset, it copies from the
>>>>> overlay unflattened devicetree and overlay fdt and only exposes
>>>>> pointers to the copies.
>>>>>
>>>>> And hopefully the issues with the overlay unflattened devicetree can
>>>>> be resolved in the same way as for the overlay fdt.
>>>>
>>>> As noted above, I don't see there is a technical solution to this issue
>>>> but it's rather a matter of convention: no overlay notifier callback is
>>>> allowed to keep references to the passed tree content (unless it
>>>> reference-counts some tree nodes) beyond the execution of the callback.
>>>> With that in place, we can safely drop the backing memory IMHO.
>>>>
>>>> Jan
>>>> .
>>>>
>>>
>>
>>
> 
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Jan Kiszka]

On 2018-04-25 21:02, Frank Rowand wrote:
> On 04/25/18 11:56, Frank Rowand wrote:
>> On 04/24/18 22:22, Jan Kiszka wrote:
>>> On 2018-04-24 23:15, Frank Rowand wrote:
>>>> On 04/23/18 22:29, Jan Kiszka wrote:
>>>>> On 2018-04-24 00:38, Frank Rowand wrote:
>>>>>> Hi Jan,
>>>>>>
>>>>>> + Alan Tull for fpga perspective
>>>>>>
>>>>>> On 04/22/18 03:30, Jan Kiszka wrote:
>>>>>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>>>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>>>>>> Hi Jan,
>>>>>>>>>>>>
>>>>>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>>>>>> original FDT.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>>>>>> errors.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>>>>>> overlay loader.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>>>>>> understand the use case and consider solutions.
>>>>>>>>>>>>
>>>>>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>>>>>
>>>>>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>>>>>
>>>>>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>>>>>> scripts/dtc/libfdt/.
>>>>>>>>>
>>>>>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>>>>>> is little reason to other than for early boot changes. And it is much
>>>>>>>>> easier to work on unflattened trees.
>>>>>>>>
>>>>>>>> I just briefly looked into libfdt, and it would have meant building it
>>>>>>>> into the module as there are no library functions exported by the kernel
>>>>>>>> either. Another reason to drop that.
>>>>>>>>
>>>>>>>> What's apparently working now is the pattern I initially suggested:
>>>>>>>> Register template with status = "disabled" as overlay, then prepare and
>>>>>>>> apply changeset that contains all needed modifications and sets the
>>>>>>>> status to "ok". I might be leaking additional resources, but to find
>>>>>>>> that out, I will now finally have to resolve clean unbinding of the
>>>>>>>> generic PCI host controller [1] first.
>>>>>>>
>>>>>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>>>>>> {
>>>>>>> 	[...]
>>>>>>> 	/*
>>>>>>> 	 * TODO
>>>>>>> 	 *
>>>>>>> 	 * would like to: kfree(ovcs->overlay_tree);
>>>>>>> 	 * but can not since drivers may have pointers into this data
>>>>>>> 	 *
>>>>>>> 	 * would like to: kfree(ovcs->fdt);
>>>>>>> 	 * but can not since drivers may have pointers into this data
>>>>>>> 	 */
>>>>>>>
>>>>>>> 	kfree(ovcs);
>>>>>>> }
>>>>>>>
>>>>>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>>>>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>>>>>> to those objects. I would say that's a regression of the new API.
>>>>>>
>>>>>> The problem already existed but it was hidden.  We have never been able to
>>>>>> kfree() these object because we do not know if there are any pointers into
>>>>>> these objects.  The new API makes the problem visible to kmemleak.
>>>>>
>>>>> My old code didn't have the problem because there was no one steeling
>>>>> pointers to my overlay, and I was able to safely release all the
>>>>> resources that I or the core on my behalf allocated. In fact, I recently
>>>>> even dropped the duplication the fdt prior to unflattening it because I
>>>>> got its lifecycle under control (and both kmemleak as well as kasan
>>>>> confirmed this). I still consider this intentional leak a regression of
>>>>> the new API.
>>>>
>>>> The API has to work for any user, not just your clean code.
>>>>
>>>
>>> Please point us to code that does have a real problem. Is there any nice
>>> vendor tree, in addition to the known users? Or are we only speculating
>>> about how people might be (mis)using the API?
>>
>> No, I will not even attempt to find such code.
>>
>> The underlying problem is that the devicetree access API returns pointers
>> into the devicetree.  And we have no way of knowing whether any driver or
>> subsystem has a live pointer into the overlay data in the devicetree when
>> we remove an overlay.  The devicetree access API did not anticipate and
>> account for overlays.  As overlay related code has been added, this is an
>> issue that has not yet been fixed.
>>

Yes, it returns pointer - which is not that uncommon for other
subsystems as well. And for that reason, it would be valuable to have
some concrete broken users as example at hand, also to validate solution
designs against it. If things are so fundamentally broken, that should
not be hard to find.

>>
>>>>
>>>>>> The reason that we do not know if there are any pointers into these objects
>>>>>> is that devicetree access APIs return pointers into the devicetree internal
>>>>>> data structures (that is, into the overlay unflattened devicetree).  If we
>>>>>> want to be able to do the kfree()s, we could change the devicetree access
>>>>>> APIs.
>>>>>>
>>>>>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>>>>>> also exposed is that the overlay unflattened devicetree property values
>>>>>> are pointers into the overlay fdt.
>>>>>>
>>>>>> ** This paragraph becomes academic (and not needed) if the fix in the next
>>>>>> paragraph can be implemented. **
>>>>>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>>>>>> (I would want to read through the code again to make sure I'm not missing
>>>>>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>>>>>> was modified so that property values were copied into newly allocated memory
>>>>>> and the live tree property pointers were set to the copy instead of to
>>>>>> the value in the fdt, then I _think_ the fdt could be freed in
>>>>>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>>>>>
>>>>> I don't see yet how more duplicating of objects would help. Then we
>>>>> would not leak the fdt or the unflattened tree on overlay destruction
>>>>> but that duplicates, no?
>>>>
>>>> Yes, we would leak the duplicates.  That is exactly what the existing
>>>> overlay remove code does.  My long term goal is to remove that leakage.
>>>> But that leakage can not be resolved until we can guarantee that there
>>>> are no pointers held to those duplicates.
>>>>
>>>> I don't like adding this additional copy - I would much prefer to change
>>>> the overlay notify code as proposed below.
>>>>
>>>
>>> Replacing one leak with another is no solution.
>>
>> I agree.  I do not see it as a viable solution.
>>
>>
>>> And if it's additionally
>>> enforcing an API change, I would call it counterproductive.
>>>
>>>>
>>>>>> frees a devicetree would also have to be aware of this change -- I'm not
>>>>>> sure if that leads to ugly complications or if it is easy.  The other
>>>>>> question to consider is whether to make the same change to
>>>>>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>>>>>> the base devicetree.  Doing so would increase the memory usage of the
>>>>>> live tree (we would not be able to free the base fdt after unflattening
>>>>>> it because we make the fdt visible in /sys/firmware/fdt -- though
>>>>>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>>>>>
>>>>>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>>>>>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>>>>>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>>>>>> into the overlay unflattened devicetree are then passed to the notifiers.
>>>>>> (Again, I may be missing some other place that the overlay unflattened
>>>>>> devicetree is made visible to other code -- a more thorough reading of
>>>>>> the code is needed.) If the notifiers could be modified to accept the
>>>>>> changeset list instead of of pointers to the fragments in the overlay
>>>>>> unflattened devicetree then there would be no possibility of the notifiers
>>>>>> keeping a pointer into the overlay fdt. I do not know if this is a
>>>>>
>>>>> But then again the convention has to be that those changeset pointers
>>>>> must not be kept - because the changeset is history after of_overlay_remove.
>>>>
>>>> I don't trust convention.  The result is fragile code.
>>>>
>>>
>>> Look, we are all programming in C here. There is no implicit reference
>>> counting, no garbage collecting, not strong typing, you-name-it. That
>>> doesn't leave you with many sharper weapons than well documented
>>> conventions.
>>
>> Nope.  We can (hopefully) modify the devicetree access API so that it
>> does not return pointers into the devicetree.  For property values,
>> this is "easy", though at a cost.  Where the API currently returns
>> a pointer to a property (or property value), copy that data into
>> newly allocated memory and return a pointer to that newly allocated
> 
> Or the data could be copied to memory designated by a pointer that the
> caller passed in.  I was not intending this paragraph to be an actual
> thought out design for an API, it is just an attempt to describe what
> needs to change.
> 
> 
>> memory -- the caller is now responsible for the new memory and there
>> is no stray pointer into the devicetree.  One other place that pointers
>> into the devicetree are exposed are the tree traversal APIs.  In theory
>> it should be possible to create an API that uses opaque "iterators"
>> (I'm probably mis-using that word) so that location in the tree while
>> traversing is not exposed in the form of a pointer into the devicetree.
>>
>> -Frank

Property pointers and the associated content are currently valid as long
as the caller holds a reference counter to the node they come from. So
the problem is lacking reference counting by the callers? I would still
consider it over-design to punish all users by a kmalloc plus local
kfree for some users not properly doing the required of_node_get().

What other pointers are we talking about?

Jan

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Geert Uytterhoeven]

Hi Jan,

On Wed, Apr 25, 2018 at 9:40 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
> What other pointers are we talking about?

There's also the issue that some data has been allocated using kmalloc(),
while others hasn't. The "other" pointers point to e.g. early bootmem or
memblock.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Jan Kiszka]

On 2018-04-25 21:53, Geert Uytterhoeven wrote:
> Hi Jan,
> 
> On Wed, Apr 25, 2018 at 9:40 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
>> What other pointers are we talking about?
> 
> There's also the issue that some data has been allocated using kmalloc(),
> while others hasn't. The "other" pointers point to e.g. early bootmem or
> memblock.

Should not be an issue for overlays or of_changesets, should it?

Yes, I've learned the hard way that it is a very bad idea to build
changesets with properties that were not dynamically allocated. Maybe
that is documented somewhere, how the ownership of those objects change
after apply and how they may get handled later on. But now as I know,
it's kind of consistent and rather easy to account for.

Jan

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

On 04/25/18 12:40, Jan Kiszka wrote:
> On 2018-04-25 21:02, Frank Rowand wrote:
>> On 04/25/18 11:56, Frank Rowand wrote:
>>> On 04/24/18 22:22, Jan Kiszka wrote:
>>>> On 2018-04-24 23:15, Frank Rowand wrote:
>>>>> On 04/23/18 22:29, Jan Kiszka wrote:
>>>>>> On 2018-04-24 00:38, Frank Rowand wrote:
>>>>>>> Hi Jan,
>>>>>>>
>>>>>>> + Alan Tull for fpga perspective
>>>>>>>
>>>>>>> On 04/22/18 03:30, Jan Kiszka wrote:
>>>>>>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>>>>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>>>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>>>>>>> Hi Jan,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>>>>>>> original FDT.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>>>>>>> errors.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>>>>>>> overlay loader.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>>>>>>> understand the use case and consider solutions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>>>>>>
>>>>>>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>>>>>>
>>>>>>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>>>>>>> scripts/dtc/libfdt/.
>>>>>>>>>>
>>>>>>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>>>>>>> is little reason to other than for early boot changes. And it is much
>>>>>>>>>> easier to work on unflattened trees.
>>>>>>>>>
>>>>>>>>> I just briefly looked into libfdt, and it would have meant building it
>>>>>>>>> into the module as there are no library functions exported by the kernel
>>>>>>>>> either. Another reason to drop that.
>>>>>>>>>
>>>>>>>>> What's apparently working now is the pattern I initially suggested:
>>>>>>>>> Register template with status = "disabled" as overlay, then prepare and
>>>>>>>>> apply changeset that contains all needed modifications and sets the
>>>>>>>>> status to "ok". I might be leaking additional resources, but to find
>>>>>>>>> that out, I will now finally have to resolve clean unbinding of the
>>>>>>>>> generic PCI host controller [1] first.
>>>>>>>>
>>>>>>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>>>>>>> {
>>>>>>>> 	[...]
>>>>>>>> 	/*
>>>>>>>> 	 * TODO
>>>>>>>> 	 *
>>>>>>>> 	 * would like to: kfree(ovcs->overlay_tree);
>>>>>>>> 	 * but can not since drivers may have pointers into this data
>>>>>>>> 	 *
>>>>>>>> 	 * would like to: kfree(ovcs->fdt);
>>>>>>>> 	 * but can not since drivers may have pointers into this data
>>>>>>>> 	 */
>>>>>>>>
>>>>>>>> 	kfree(ovcs);
>>>>>>>> }
>>>>>>>>
>>>>>>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>>>>>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>>>>>>> to those objects. I would say that's a regression of the new API.
>>>>>>>
>>>>>>> The problem already existed but it was hidden.  We have never been able to
>>>>>>> kfree() these object because we do not know if there are any pointers into
>>>>>>> these objects.  The new API makes the problem visible to kmemleak.
>>>>>>
>>>>>> My old code didn't have the problem because there was no one steeling
>>>>>> pointers to my overlay, and I was able to safely release all the
>>>>>> resources that I or the core on my behalf allocated. In fact, I recently
>>>>>> even dropped the duplication the fdt prior to unflattening it because I
>>>>>> got its lifecycle under control (and both kmemleak as well as kasan
>>>>>> confirmed this). I still consider this intentional leak a regression of
>>>>>> the new API.
>>>>>
>>>>> The API has to work for any user, not just your clean code.
>>>>>
>>>>
>>>> Please point us to code that does have a real problem. Is there any nice
>>>> vendor tree, in addition to the known users? Or are we only speculating
>>>> about how people might be (mis)using the API?
>>>
>>> No, I will not even attempt to find such code.
>>>
>>> The underlying problem is that the devicetree access API returns pointers
>>> into the devicetree.  And we have no way of knowing whether any driver or
>>> subsystem has a live pointer into the overlay data in the devicetree when
>>> we remove an overlay.  The devicetree access API did not anticipate and
>>> account for overlays.  As overlay related code has been added, this is an
>>> issue that has not yet been fixed.
>>>
> 
> Yes, it returns pointer - which is not that uncommon for other
> subsystems as well. And for that reason, it would be valuable to have
> some concrete broken users as example at hand, also to validate solution
> designs against it. If things are so fundamentally broken, that should
> not be hard to find.

We essentially do not support generic overlays in the mainline kernel.  Overlays
are a feature that is under development.  (Yes, there is a driver that directly
uses overlays - if I had been a devicetree maintainer at the time that usage
was added I would have objected).  Acceptance of an overlay manager has been
refused several times, as least partly because the underlying overlay features
are not yet complete enough.

The fundamental breakage is not exposed if we do not remove overlays and thus
do not free the underlying memory.


>>>
>>>>>
>>>>>>> The reason that we do not know if there are any pointers into these objects
>>>>>>> is that devicetree access APIs return pointers into the devicetree internal
>>>>>>> data structures (that is, into the overlay unflattened devicetree).  If we
>>>>>>> want to be able to do the kfree()s, we could change the devicetree access
>>>>>>> APIs.
>>>>>>>
>>>>>>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>>>>>>> also exposed is that the overlay unflattened devicetree property values
>>>>>>> are pointers into the overlay fdt.
>>>>>>>
>>>>>>> ** This paragraph becomes academic (and not needed) if the fix in the next
>>>>>>> paragraph can be implemented. **
>>>>>>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>>>>>>> (I would want to read through the code again to make sure I'm not missing
>>>>>>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>>>>>>> was modified so that property values were copied into newly allocated memory
>>>>>>> and the live tree property pointers were set to the copy instead of to
>>>>>>> the value in the fdt, then I _think_ the fdt could be freed in
>>>>>>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>>>>>>
>>>>>> I don't see yet how more duplicating of objects would help. Then we
>>>>>> would not leak the fdt or the unflattened tree on overlay destruction
>>>>>> but that duplicates, no?
>>>>>
>>>>> Yes, we would leak the duplicates.  That is exactly what the existing
>>>>> overlay remove code does.  My long term goal is to remove that leakage.
>>>>> But that leakage can not be resolved until we can guarantee that there
>>>>> are no pointers held to those duplicates.
>>>>>
>>>>> I don't like adding this additional copy - I would much prefer to change
>>>>> the overlay notify code as proposed below.
>>>>>
>>>>
>>>> Replacing one leak with another is no solution.
>>>
>>> I agree.  I do not see it as a viable solution.
>>>
>>>
>>>> And if it's additionally
>>>> enforcing an API change, I would call it counterproductive.
>>>>
>>>>>
>>>>>>> frees a devicetree would also have to be aware of this change -- I'm not
>>>>>>> sure if that leads to ugly complications or if it is easy.  The other
>>>>>>> question to consider is whether to make the same change to
>>>>>>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>>>>>>> the base devicetree.  Doing so would increase the memory usage of the
>>>>>>> live tree (we would not be able to free the base fdt after unflattening
>>>>>>> it because we make the fdt visible in /sys/firmware/fdt -- though
>>>>>>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>>>>>>
>>>>>>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>>>>>>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>>>>>>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>>>>>>> into the overlay unflattened devicetree are then passed to the notifiers.
>>>>>>> (Again, I may be missing some other place that the overlay unflattened
>>>>>>> devicetree is made visible to other code -- a more thorough reading of
>>>>>>> the code is needed.) If the notifiers could be modified to accept the
>>>>>>> changeset list instead of of pointers to the fragments in the overlay
>>>>>>> unflattened devicetree then there would be no possibility of the notifiers
>>>>>>> keeping a pointer into the overlay fdt. I do not know if this is a
>>>>>>
>>>>>> But then again the convention has to be that those changeset pointers
>>>>>> must not be kept - because the changeset is history after of_overlay_remove.
>>>>>
>>>>> I don't trust convention.  The result is fragile code.
>>>>>
>>>>
>>>> Look, we are all programming in C here. There is no implicit reference
>>>> counting, no garbage collecting, not strong typing, you-name-it. That
>>>> doesn't leave you with many sharper weapons than well documented
>>>> conventions.
>>>
>>> Nope.  We can (hopefully) modify the devicetree access API so that it
>>> does not return pointers into the devicetree.  For property values,
>>> this is "easy", though at a cost.  Where the API currently returns
>>> a pointer to a property (or property value), copy that data into
>>> newly allocated memory and return a pointer to that newly allocated
>>
>> Or the data could be copied to memory designated by a pointer that the
>> caller passed in.  I was not intending this paragraph to be an actual
>> thought out design for an API, it is just an attempt to describe what
>> needs to change.
>>
>>
>>> memory -- the caller is now responsible for the new memory and there
>>> is no stray pointer into the devicetree.  One other place that pointers
>>> into the devicetree are exposed are the tree traversal APIs.  In theory
>>> it should be possible to create an API that uses opaque "iterators"
>>> (I'm probably mis-using that word) so that location in the tree while
>>> traversing is not exposed in the form of a pointer into the devicetree.
>>>
>>> -Frank
> 
> Property pointers and the associated content are currently valid as long
> as the caller holds a reference counter to the node they come from. So
> the problem is lacking reference counting by the callers? I would still
> consider it over-design to punish all users by a kmalloc plus local
> kfree for some users not properly doing the required of_node_get().

Properties do not have their own reference counter (as you note in the
comment that a caller hold a reference counter to the node that contains
the property).  But there is no documentation or expectation that a
caller of a function that returns a property pointer needs to hold the
node reference counter.  For example, of_find_property() and
of_property_read_string().

We know that node reference counters are often incorrect.


> What other pointers are we talking about?
> 
> Jan
> .
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

Hi Alan,

On 04/23/18 15:38, Frank Rowand wrote:
> Hi Jan,
> 
> + Alan Tull for fpga perspective
> 
> On 04/22/18 03:30, Jan Kiszka wrote:
>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>> Hi Jan,
>>>>>>>
>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>> Hi Frank,
>>>>>>>>
>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>
>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>
>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>> original FDT.
>>>>>>>>>
>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>
>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>> errors.
>>>>>>>>>
>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>> overlay loader.
>>>>>>>>
>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>
>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>
>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>> understand the use case and consider solutions.
>>>>>>>
>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>
>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>
>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>> scripts/dtc/libfdt/.
>>>>
>>>> Let's please not go down that route of doing FDT modifications. There
>>>> is little reason to other than for early boot changes. And it is much
>>>> easier to work on unflattened trees.
>>>
>>> I just briefly looked into libfdt, and it would have meant building it
>>> into the module as there are no library functions exported by the kernel
>>> either. Another reason to drop that.
>>>
>>> What's apparently working now is the pattern I initially suggested:
>>> Register template with status = "disabled" as overlay, then prepare and
>>> apply changeset that contains all needed modifications and sets the
>>> status to "ok". I might be leaking additional resources, but to find
>>> that out, I will now finally have to resolve clean unbinding of the
>>> generic PCI host controller [1] first.
>>
>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>> {
>> 	[...]
>> 	/*
>> 	 * TODO
>> 	 *
>> 	 * would like to: kfree(ovcs->overlay_tree);
>> 	 * but can not since drivers may have pointers into this data
>> 	 *
>> 	 * would like to: kfree(ovcs->fdt);
>> 	 * but can not since drivers may have pointers into this data
>> 	 */
>>
>> 	kfree(ovcs);
>> }
>>
>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>> to those objects. I would say that's a regression of the new API.
> 
> The problem already existed but it was hidden.  We have never been able to
> kfree() these object because we do not know if there are any pointers into
> these objects.  The new API makes the problem visible to kmemleak.
> 
> The reason that we do not know if there are any pointers into these objects
> is that devicetree access APIs return pointers into the devicetree internal
> data structures (that is, into the overlay unflattened devicetree).  If we
> want to be able to do the kfree()s, we could change the devicetree access
> APIs.
> 
> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
> also exposed is that the overlay unflattened devicetree property values
> are pointers into the overlay fdt.
> 
> ** This paragraph becomes academic (and not needed) if the fix in the next
> paragraph can be implemented. **
> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
> (I would want to read through the code again to make sure I'm not missing
> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
> was modified so that property values were copied into newly allocated memory
> and the live tree property pointers were set to the copy instead of to
> the value in the fdt, then I _think_ the fdt could be freed in
> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
> frees a devicetree would also have to be aware of this change -- I'm not
> sure if that leads to ugly complications or if it is easy.  The other
> question to consider is whether to make the same change to
> of_fdt_unflatten_tree() when it is called in early boot to unflatten
> the base devicetree.  Doing so would increase the memory usage of the
> live tree (we would not be able to free the base fdt after unflattening
> it because we make the fdt visible in /sys/firmware/fdt -- though
> _maybe_ that could be conditioned on CONFIG_KEXEC).

Question added below this paragraph.


> But all of the complexity of that fix is _only_ because of_overlay_apply()
> and of_overlay_remove() call overlay_notify(), passing in the overlay
> unflattened devicetree (which has pointers into the overlay fdt). Pointers
> into the overlay unflattened devicetree are then passed to the notifiers.
> (Again, I may be missing some other place that the overlay unflattened
> devicetree is made visible to other code -- a more thorough reading of
> the code is needed.) If the notifiers could be modified to accept the
> changeset list instead of of pointers to the fragments in the overlay
> unflattened devicetree then there would be no possibility of the notifiers
> keeping a pointer into the overlay fdt. I do not know if this is a
> practical change for the notifiers -- there are no callers of
> of_overlay_notifier_register() in the mainline kernel source. My
> recollection is that the overlay notifiers were added for the fpga
> subsystem.

Can the fpga notifiers be changed to have the changeset as an input
instead of having the overlay devicetree fragment and target as an
input?

The changeset lists nodes and properties to be added, but does not
expose any pointers to the overlay fdt or the overlay unflattened
devicetree.  This guarantees no leakage of pointers into the overlay
fdt or the overlay unflattened devicetree.  The changeset contains
pointers to copies of data, but those copies are never freed (and
thus they are yet another existing memory leak).

-Frank

> Why is overlay_notify() the only issue related to unknown users having
> pointers into the overlay fdt?  The answer is that the overlay code
> does not directly expose the overlay unflattened devicetree (and thus
> indirectly the overlay fdt) to the live devicetree -- when the
> overlay code creates the overlay changeset, it copies from the
> overlay unflattened devicetree and overlay fdt and only exposes
> pointers to the copies.
> 
> And hopefully the issues with the overlay unflattened devicetree can
> be resolved in the same way as for the overlay fdt.
> 
> -Frank
> 
> 
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Jan Kiszka]

On 2018-04-24 22:56, Frank Rowand wrote:
> Hi Alan,
> 
> On 04/23/18 15:38, Frank Rowand wrote:
>> Hi Jan,
>>
>> + Alan Tull for fpga perspective
>>
>> On 04/22/18 03:30, Jan Kiszka wrote:
>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>> Hi Jan,
>>>>>>>>
>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>> Hi Frank,
>>>>>>>>>
>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>
>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>
>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>> original FDT.
>>>>>>>>>>
>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>
>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>> errors.
>>>>>>>>>>
>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>> overlay loader.
>>>>>>>>>
>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>
>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>
>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>> understand the use case and consider solutions.
>>>>>>>>
>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>
>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>
>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>> scripts/dtc/libfdt/.
>>>>>
>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>> is little reason to other than for early boot changes. And it is much
>>>>> easier to work on unflattened trees.
>>>>
>>>> I just briefly looked into libfdt, and it would have meant building it
>>>> into the module as there are no library functions exported by the kernel
>>>> either. Another reason to drop that.
>>>>
>>>> What's apparently working now is the pattern I initially suggested:
>>>> Register template with status = "disabled" as overlay, then prepare and
>>>> apply changeset that contains all needed modifications and sets the
>>>> status to "ok". I might be leaking additional resources, but to find
>>>> that out, I will now finally have to resolve clean unbinding of the
>>>> generic PCI host controller [1] first.
>>>
>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>> {
>>> 	[...]
>>> 	/*
>>> 	 * TODO
>>> 	 *
>>> 	 * would like to: kfree(ovcs->overlay_tree);
>>> 	 * but can not since drivers may have pointers into this data
>>> 	 *
>>> 	 * would like to: kfree(ovcs->fdt);
>>> 	 * but can not since drivers may have pointers into this data
>>> 	 */
>>>
>>> 	kfree(ovcs);
>>> }
>>>
>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>> to those objects. I would say that's a regression of the new API.
>>
>> The problem already existed but it was hidden.  We have never been able to
>> kfree() these object because we do not know if there are any pointers into
>> these objects.  The new API makes the problem visible to kmemleak.
>>
>> The reason that we do not know if there are any pointers into these objects
>> is that devicetree access APIs return pointers into the devicetree internal
>> data structures (that is, into the overlay unflattened devicetree).  If we
>> want to be able to do the kfree()s, we could change the devicetree access
>> APIs.
>>
>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>> also exposed is that the overlay unflattened devicetree property values
>> are pointers into the overlay fdt.
>>
>> ** This paragraph becomes academic (and not needed) if the fix in the next
>> paragraph can be implemented. **
>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>> (I would want to read through the code again to make sure I'm not missing
>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>> was modified so that property values were copied into newly allocated memory
>> and the live tree property pointers were set to the copy instead of to
>> the value in the fdt, then I _think_ the fdt could be freed in
>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>> frees a devicetree would also have to be aware of this change -- I'm not
>> sure if that leads to ugly complications or if it is easy.  The other
>> question to consider is whether to make the same change to
>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>> the base devicetree.  Doing so would increase the memory usage of the
>> live tree (we would not be able to free the base fdt after unflattening
>> it because we make the fdt visible in /sys/firmware/fdt -- though
>> _maybe_ that could be conditioned on CONFIG_KEXEC).
> 
> Question added below this paragraph.
> 
> 
>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>> into the overlay unflattened devicetree are then passed to the notifiers.
>> (Again, I may be missing some other place that the overlay unflattened
>> devicetree is made visible to other code -- a more thorough reading of
>> the code is needed.) If the notifiers could be modified to accept the
>> changeset list instead of of pointers to the fragments in the overlay
>> unflattened devicetree then there would be no possibility of the notifiers
>> keeping a pointer into the overlay fdt. I do not know if this is a
>> practical change for the notifiers -- there are no callers of
>> of_overlay_notifier_register() in the mainline kernel source. My
>> recollection is that the overlay notifiers were added for the fpga
>> subsystem.
> 
> Can the fpga notifiers be changed to have the changeset as an input
> instead of having the overlay devicetree fragment and target as an
> input?
> 
> The changeset lists nodes and properties to be added, but does not
> expose any pointers to the overlay fdt or the overlay unflattened
> devicetree.  This guarantees no leakage of pointers into the overlay
> fdt or the overlay unflattened devicetree.  The changeset contains
> pointers to copies of data, but those copies are never freed (and
> thus they are yet another existing memory leak).

Also they are freed, of course: When the last reference to the node they
point to reaches 0 (e.g. triggered by of_changeset_destroy), that node
goes away and takes down remaining dead properties. I've ran through
this already. And I also made sure that my code is not triggering such
kind of leaks as well.

I agree that object management in dynamic OF code is hairy and easy to
get wrong, e.g. by passing in static objects that suddenly change
ownership, and the new owner expects dynamically allocated objects...
But that is not cured by leaking memory.

Jan

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

On 04/24/18 22:23, Jan Kiszka wrote:
> On 2018-04-24 22:56, Frank Rowand wrote:
>> Hi Alan,
>>
>> On 04/23/18 15:38, Frank Rowand wrote:
>>> Hi Jan,
>>>
>>> + Alan Tull for fpga perspective
>>>
>>> On 04/22/18 03:30, Jan Kiszka wrote:
>>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>>> Hi Jan,
>>>>>>>>>
>>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>>> Hi Frank,
>>>>>>>>>>
>>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>>
>>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>>
>>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>>> original FDT.
>>>>>>>>>>>
>>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>>
>>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>>> errors.
>>>>>>>>>>>
>>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>>> overlay loader.
>>>>>>>>>>
>>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>>
>>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>>
>>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>>> understand the use case and consider solutions.
>>>>>>>>>
>>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>>
>>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>>
>>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>>> scripts/dtc/libfdt/.
>>>>>>
>>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>>> is little reason to other than for early boot changes. And it is much
>>>>>> easier to work on unflattened trees.
>>>>>
>>>>> I just briefly looked into libfdt, and it would have meant building it
>>>>> into the module as there are no library functions exported by the kernel
>>>>> either. Another reason to drop that.
>>>>>
>>>>> What's apparently working now is the pattern I initially suggested:
>>>>> Register template with status = "disabled" as overlay, then prepare and
>>>>> apply changeset that contains all needed modifications and sets the
>>>>> status to "ok". I might be leaking additional resources, but to find
>>>>> that out, I will now finally have to resolve clean unbinding of the
>>>>> generic PCI host controller [1] first.
>>>>
>>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>>> {
>>>> 	[...]
>>>> 	/*
>>>> 	 * TODO
>>>> 	 *
>>>> 	 * would like to: kfree(ovcs->overlay_tree);
>>>> 	 * but can not since drivers may have pointers into this data
>>>> 	 *
>>>> 	 * would like to: kfree(ovcs->fdt);
>>>> 	 * but can not since drivers may have pointers into this data
>>>> 	 */
>>>>
>>>> 	kfree(ovcs);
>>>> }
>>>>
>>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>>> to those objects. I would say that's a regression of the new API.
>>>
>>> The problem already existed but it was hidden.  We have never been able to
>>> kfree() these object because we do not know if there are any pointers into
>>> these objects.  The new API makes the problem visible to kmemleak.
>>>
>>> The reason that we do not know if there are any pointers into these objects
>>> is that devicetree access APIs return pointers into the devicetree internal
>>> data structures (that is, into the overlay unflattened devicetree).  If we
>>> want to be able to do the kfree()s, we could change the devicetree access
>>> APIs.
>>>
>>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>>> also exposed is that the overlay unflattened devicetree property values
>>> are pointers into the overlay fdt.
>>>
>>> ** This paragraph becomes academic (and not needed) if the fix in the next
>>> paragraph can be implemented. **
>>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>>> (I would want to read through the code again to make sure I'm not missing
>>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>>> was modified so that property values were copied into newly allocated memory
>>> and the live tree property pointers were set to the copy instead of to
>>> the value in the fdt, then I _think_ the fdt could be freed in
>>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>>> frees a devicetree would also have to be aware of this change -- I'm not
>>> sure if that leads to ugly complications or if it is easy.  The other
>>> question to consider is whether to make the same change to
>>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>>> the base devicetree.  Doing so would increase the memory usage of the
>>> live tree (we would not be able to free the base fdt after unflattening
>>> it because we make the fdt visible in /sys/firmware/fdt -- though
>>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>
>> Question added below this paragraph.
>>
>>
>>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>>> into the overlay unflattened devicetree are then passed to the notifiers.
>>> (Again, I may be missing some other place that the overlay unflattened
>>> devicetree is made visible to other code -- a more thorough reading of
>>> the code is needed.) If the notifiers could be modified to accept the
>>> changeset list instead of of pointers to the fragments in the overlay
>>> unflattened devicetree then there would be no possibility of the notifiers
>>> keeping a pointer into the overlay fdt. I do not know if this is a
>>> practical change for the notifiers -- there are no callers of
>>> of_overlay_notifier_register() in the mainline kernel source. My
>>> recollection is that the overlay notifiers were added for the fpga
>>> subsystem.
>>
>> Can the fpga notifiers be changed to have the changeset as an input
>> instead of having the overlay devicetree fragment and target as an
>> input?
>>
>> The changeset lists nodes and properties to be added, but does not
>> expose any pointers to the overlay fdt or the overlay unflattened
>> devicetree.  This guarantees no leakage of pointers into the overlay
>> fdt or the overlay unflattened devicetree.  The changeset contains
>> pointers to copies of data, but those copies are never freed (and
>> thus they are yet another existing memory leak).
> 
> Also they are freed, of course: When the last reference to the node they
> point to reaches 0 (e.g. triggered by of_changeset_destroy), that node
> goes away and takes down remaining dead properties. I've ran through
> this already. And I also made sure that my code is not triggering such
> kind of leaks as well.

mea culpa.  I go around in circles while trying to remember all the
overlay related issues.  I needed to go back and read the code to
refresh my memory.  Thanks for the prod to re-read the code.

Yes, of_changeset_destroy() will lead to the kfree() of the node and
it's properties _if_ the node reference count is correct.  So what I
said about a memory leak was incorrect in a perfect world (and my
memory was wrong).  However, this is not a perfect world and we know
that the reference count on devicetree nodes is often incorrect due
to bugs in common infrastructure and drivers.  This issue will not
be resolved until we pull all reference count manipulation into the
devicetree core.  The net result is that we should not expect
overlay removal to correctly free all memory that was allocated
when applying the overlay.

I _think_ (but did not spend the time to verify) that there is a small
corner case memory leak even if the reference count on devicetree
nodes is correct.  If an overlay adds a property to an existing node
then removing the overlay will not kfree() the property, and it
will remain on the deadprops list.  There are some places that
properties are removed from deadprops, but I don't think they fully
resolve the issue.  Again, this is a corner case, and I am willing
to document it as a limitation until it gets fixed.

Then returning to me going around in circles... This thread led me to
think that since since the overlay apply code copied data into never
freed memory (false premise, as you pointed out) that we did not
have to worry about drivers retaining pointers into overlay data
after the overlay had been freed (with the one remaining exposure
being via the overlay notifiers, which _might_ be easily resolved,
pending Alan's analysis) -- this would have been great news for
removing an issue for general use of overlays.

But now we are back to the long-standing problem that we have no way
of knowing whether there are any live pointers to the memory that is
freed by of_changeset_destroy().  And I am not aware of any solution
to this problem other than changing the devicetree access API so that
it never returns any pointer into the live devicetree.

The practical impact of all of this, is if we can change the overlay
notifier parameters to include the overlay changeset instead of
the overlay devicetree, then I think that of_overlay_apply() will
be able to kfree() the overlay fdt and overlay devicetree.  And
if not of_overlay_apply(), then free_overlay_changeset().


> I agree that object management in dynamic OF code is hairy and easy to
> get wrong, e.g. by passing in static objects that suddenly change
> ownership, and the new owner expects dynamically allocated objects...
> But that is not cured by leaking memory.
> 
> Jan
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Jan Kiszka]

On 2018-04-25 20:40, Frank Rowand wrote:
> On 04/24/18 22:23, Jan Kiszka wrote:
>> On 2018-04-24 22:56, Frank Rowand wrote:
>>> Hi Alan,
>>>
>>> On 04/23/18 15:38, Frank Rowand wrote:
>>>> Hi Jan,
>>>>
>>>> + Alan Tull for fpga perspective
>>>>
>>>> On 04/22/18 03:30, Jan Kiszka wrote:
>>>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>>>> Hi Jan,
>>>>>>>>>>
>>>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>
>>>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>>>
>>>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>>>
>>>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>>>> original FDT.
>>>>>>>>>>>>
>>>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>>>
>>>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>>>> errors.
>>>>>>>>>>>>
>>>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>>>> overlay loader.
>>>>>>>>>>>
>>>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>>>
>>>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>>>
>>>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>>>> understand the use case and consider solutions.
>>>>>>>>>>
>>>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>>>
>>>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>>>
>>>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>>>> scripts/dtc/libfdt/.
>>>>>>>
>>>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>>>> is little reason to other than for early boot changes. And it is much
>>>>>>> easier to work on unflattened trees.
>>>>>>
>>>>>> I just briefly looked into libfdt, and it would have meant building it
>>>>>> into the module as there are no library functions exported by the kernel
>>>>>> either. Another reason to drop that.
>>>>>>
>>>>>> What's apparently working now is the pattern I initially suggested:
>>>>>> Register template with status = "disabled" as overlay, then prepare and
>>>>>> apply changeset that contains all needed modifications and sets the
>>>>>> status to "ok". I might be leaking additional resources, but to find
>>>>>> that out, I will now finally have to resolve clean unbinding of the
>>>>>> generic PCI host controller [1] first.
>>>>>
>>>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>>>> {
>>>>> 	[...]
>>>>> 	/*
>>>>> 	 * TODO
>>>>> 	 *
>>>>> 	 * would like to: kfree(ovcs->overlay_tree);
>>>>> 	 * but can not since drivers may have pointers into this data
>>>>> 	 *
>>>>> 	 * would like to: kfree(ovcs->fdt);
>>>>> 	 * but can not since drivers may have pointers into this data
>>>>> 	 */
>>>>>
>>>>> 	kfree(ovcs);
>>>>> }
>>>>>
>>>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>>>> to those objects. I would say that's a regression of the new API.
>>>>
>>>> The problem already existed but it was hidden.  We have never been able to
>>>> kfree() these object because we do not know if there are any pointers into
>>>> these objects.  The new API makes the problem visible to kmemleak.
>>>>
>>>> The reason that we do not know if there are any pointers into these objects
>>>> is that devicetree access APIs return pointers into the devicetree internal
>>>> data structures (that is, into the overlay unflattened devicetree).  If we
>>>> want to be able to do the kfree()s, we could change the devicetree access
>>>> APIs.
>>>>
>>>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>>>> also exposed is that the overlay unflattened devicetree property values
>>>> are pointers into the overlay fdt.
>>>>
>>>> ** This paragraph becomes academic (and not needed) if the fix in the next
>>>> paragraph can be implemented. **
>>>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>>>> (I would want to read through the code again to make sure I'm not missing
>>>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>>>> was modified so that property values were copied into newly allocated memory
>>>> and the live tree property pointers were set to the copy instead of to
>>>> the value in the fdt, then I _think_ the fdt could be freed in
>>>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>>>> frees a devicetree would also have to be aware of this change -- I'm not
>>>> sure if that leads to ugly complications or if it is easy.  The other
>>>> question to consider is whether to make the same change to
>>>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>>>> the base devicetree.  Doing so would increase the memory usage of the
>>>> live tree (we would not be able to free the base fdt after unflattening
>>>> it because we make the fdt visible in /sys/firmware/fdt -- though
>>>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>>
>>> Question added below this paragraph.
>>>
>>>
>>>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>>>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>>>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>>>> into the overlay unflattened devicetree are then passed to the notifiers.
>>>> (Again, I may be missing some other place that the overlay unflattened
>>>> devicetree is made visible to other code -- a more thorough reading of
>>>> the code is needed.) If the notifiers could be modified to accept the
>>>> changeset list instead of of pointers to the fragments in the overlay
>>>> unflattened devicetree then there would be no possibility of the notifiers
>>>> keeping a pointer into the overlay fdt. I do not know if this is a
>>>> practical change for the notifiers -- there are no callers of
>>>> of_overlay_notifier_register() in the mainline kernel source. My
>>>> recollection is that the overlay notifiers were added for the fpga
>>>> subsystem.
>>>
>>> Can the fpga notifiers be changed to have the changeset as an input
>>> instead of having the overlay devicetree fragment and target as an
>>> input?
>>>
>>> The changeset lists nodes and properties to be added, but does not
>>> expose any pointers to the overlay fdt or the overlay unflattened
>>> devicetree.  This guarantees no leakage of pointers into the overlay
>>> fdt or the overlay unflattened devicetree.  The changeset contains
>>> pointers to copies of data, but those copies are never freed (and
>>> thus they are yet another existing memory leak).
>>
>> Also they are freed, of course: When the last reference to the node they
>> point to reaches 0 (e.g. triggered by of_changeset_destroy), that node
>> goes away and takes down remaining dead properties. I've ran through
>> this already. And I also made sure that my code is not triggering such
>> kind of leaks as well.
> 
> mea culpa.  I go around in circles while trying to remember all the
> overlay related issues.  I needed to go back and read the code to
> refresh my memory.  Thanks for the prod to re-read the code.
> 
> Yes, of_changeset_destroy() will lead to the kfree() of the node and
> it's properties _if_ the node reference count is correct.  So what I
> said about a memory leak was incorrect in a perfect world (and my
> memory was wrong).  However, this is not a perfect world and we know
> that the reference count on devicetree nodes is often incorrect due
> to bugs in common infrastructure and drivers.  This issue will not
> be resolved until we pull all reference count manipulation into the
> devicetree core.

I don't get this yet. When I want some value from life tree, I do a node
search, get a pointer and the core incremented its reference, can query
the node and its properties, and when I'm done, I call of_node_put and
forget about all pointers I got. What would you do differently?

>  The net result is that we should not expect
> overlay removal to correctly free all memory that was allocated
> when applying the overlay.

Depends on the overlay. If you do not modify existing nodes but only add
new ones, it is fair to expect complete removal.

> 
> I _think_ (but did not spend the time to verify) that there is a small
> corner case memory leak even if the reference count on devicetree
> nodes is correct.  If an overlay adds a property to an existing node
> then removing the overlay will not kfree() the property, and it
> will remain on the deadprops list.  There are some places that
> properties are removed from deadprops, but I don't think they fully
> resolve the issue.  Again, this is a corner case, and I am willing
> to document it as a limitation until it gets fixed.

I ran into this the other day: If you modify an existing property, the
old value will be put into deadprops and only be freed when the node is
freed. It may come back from deadprops if a changeset comes around with
the very same property object for another modification.

But that means: if your overlay just adds nodes, all of them, including
their deadprops from potential changes on top, will go away on overlay
removal.

BTW, here is my new code that exploits this to be leak-free:
https://github.com/siemens/jailhouse/blob/156a93fcc02585d78d4418d3e6761cd72a65b359/driver/pci.c#L296

> 
> Then returning to me going around in circles... This thread led me to
> think that since since the overlay apply code copied data into never
> freed memory (false premise, as you pointed out) that we did not
> have to worry about drivers retaining pointers into overlay data
> after the overlay had been freed (with the one remaining exposure
> being via the overlay notifiers, which _might_ be easily resolved,
> pending Alan's analysis) -- this would have been great news for
> removing an issue for general use of overlays.
> 
> But now we are back to the long-standing problem that we have no way
> of knowing whether there are any live pointers to the memory that is
> freed by of_changeset_destroy().  And I am not aware of any solution
> to this problem other than changing the devicetree access API so that
> it never returns any pointer into the live devicetree.

I don't agree yet with this drastic measure until you can point me to
code that pulls and stores pointers to arbitrary devicetree content
without that node reference counting. The pattern we otherwise see all
around it you get a pointer (or a set of them) along with the duty to
explicitly drop it again by some put() operation.

> 
> The practical impact of all of this, is if we can change the overlay
> notifier parameters to include the overlay changeset instead of
> the overlay devicetree, then I think that of_overlay_apply() will
> be able to kfree() the overlay fdt and overlay devicetree.  And
> if not of_overlay_apply(), then free_overlay_changeset().

Isn't that just s/node/changeset/ without any other semantic changes? If
the receiver of the changeset reference does not take care of lifecycle
management for that object either, we are back at square #1. A changeset
is just a gate to the nodes and properties that are currently passed
directly.

Jan

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Alan Tull]

On Wed, Apr 25, 2018 at 3:07 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
> On 2018-04-25 20:40, Frank Rowand wrote:
>> On 04/24/18 22:23, Jan Kiszka wrote:
>>> On 2018-04-24 22:56, Frank Rowand wrote:
>>>> Hi Alan,
>>>>
>>>> On 04/23/18 15:38, Frank Rowand wrote:
>>>>> Hi Jan,
>>>>>
>>>>> + Alan Tull for fpga perspective
>>>>>
>>>>> On 04/22/18 03:30, Jan Kiszka wrote:
>>>>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>>>>> Hi Jan,
>>>>>>>>>>>
>>>>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>
>>>>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>>>>
>>>>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>>>>> original FDT.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>>>>
>>>>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>>>>> errors.
>>>>>>>>>>>>>
>>>>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>>>>> overlay loader.
>>>>>>>>>>>>
>>>>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>>>>
>>>>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>>>>> understand the use case and consider solutions.
>>>>>>>>>>>
>>>>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>>>>
>>>>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>>>>
>>>>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>>>>> scripts/dtc/libfdt/.
>>>>>>>>
>>>>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>>>>> is little reason to other than for early boot changes. And it is much
>>>>>>>> easier to work on unflattened trees.
>>>>>>>
>>>>>>> I just briefly looked into libfdt, and it would have meant building it
>>>>>>> into the module as there are no library functions exported by the kernel
>>>>>>> either. Another reason to drop that.
>>>>>>>
>>>>>>> What's apparently working now is the pattern I initially suggested:
>>>>>>> Register template with status = "disabled" as overlay, then prepare and
>>>>>>> apply changeset that contains all needed modifications and sets the
>>>>>>> status to "ok". I might be leaking additional resources, but to find
>>>>>>> that out, I will now finally have to resolve clean unbinding of the
>>>>>>> generic PCI host controller [1] first.
>>>>>>
>>>>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>>>>> {
>>>>>>   [...]
>>>>>>   /*
>>>>>>    * TODO
>>>>>>    *
>>>>>>    * would like to: kfree(ovcs->overlay_tree);
>>>>>>    * but can not since drivers may have pointers into this data
>>>>>>    *
>>>>>>    * would like to: kfree(ovcs->fdt);
>>>>>>    * but can not since drivers may have pointers into this data
>>>>>>    */
>>>>>>
>>>>>>   kfree(ovcs);
>>>>>> }
>>>>>>
>>>>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>>>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>>>>> to those objects. I would say that's a regression of the new API.
>>>>>
>>>>> The problem already existed but it was hidden.  We have never been able to
>>>>> kfree() these object because we do not know if there are any pointers into
>>>>> these objects.  The new API makes the problem visible to kmemleak.
>>>>>
>>>>> The reason that we do not know if there are any pointers into these objects
>>>>> is that devicetree access APIs return pointers into the devicetree internal
>>>>> data structures (that is, into the overlay unflattened devicetree).  If we
>>>>> want to be able to do the kfree()s, we could change the devicetree access
>>>>> APIs.
>>>>>
>>>>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>>>>> also exposed is that the overlay unflattened devicetree property values
>>>>> are pointers into the overlay fdt.
>>>>>
>>>>> ** This paragraph becomes academic (and not needed) if the fix in the next
>>>>> paragraph can be implemented. **
>>>>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>>>>> (I would want to read through the code again to make sure I'm not missing
>>>>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>>>>> was modified so that property values were copied into newly allocated memory
>>>>> and the live tree property pointers were set to the copy instead of to
>>>>> the value in the fdt, then I _think_ the fdt could be freed in
>>>>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>>>>> frees a devicetree would also have to be aware of this change -- I'm not
>>>>> sure if that leads to ugly complications or if it is easy.  The other
>>>>> question to consider is whether to make the same change to
>>>>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>>>>> the base devicetree.  Doing so would increase the memory usage of the
>>>>> live tree (we would not be able to free the base fdt after unflattening
>>>>> it because we make the fdt visible in /sys/firmware/fdt -- though
>>>>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>>>
>>>> Question added below this paragraph.
>>>>
>>>>
>>>>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>>>>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>>>>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>>>>> into the overlay unflattened devicetree are then passed to the notifiers.
>>>>> (Again, I may be missing some other place that the overlay unflattened
>>>>> devicetree is made visible to other code -- a more thorough reading of
>>>>> the code is needed.) If the notifiers could be modified to accept the
>>>>> changeset list instead of of pointers to the fragments in the overlay
>>>>> unflattened devicetree then there would be no possibility of the notifiers
>>>>> keeping a pointer into the overlay fdt. I do not know if this is a
>>>>> practical change for the notifiers -- there are no callers of
>>>>> of_overlay_notifier_register() in the mainline kernel source. My
>>>>> recollection is that the overlay notifiers were added for the fpga
>>>>> subsystem.
>>>>
>>>> Can the fpga notifiers be changed to have the changeset as an input
>>>> instead of having the overlay devicetree fragment and target as an
>>>> input?
>>>>
>>>> The changeset lists nodes and properties to be added, but does not
>>>> expose any pointers to the overlay fdt or the overlay unflattened
>>>> devicetree.  This guarantees no leakage of pointers into the overlay
>>>> fdt or the overlay unflattened devicetree.  The changeset contains
>>>> pointers to copies of data, but those copies are never freed (and
>>>> thus they are yet another existing memory leak).
>>>
>>> Also they are freed, of course: When the last reference to the node they
>>> point to reaches 0 (e.g. triggered by of_changeset_destroy), that node
>>> goes away and takes down remaining dead properties. I've ran through
>>> this already. And I also made sure that my code is not triggering such
>>> kind of leaks as well.
>>
>> mea culpa.  I go around in circles while trying to remember all the
>> overlay related issues.  I needed to go back and read the code to
>> refresh my memory.  Thanks for the prod to re-read the code.
>>
>> Yes, of_changeset_destroy() will lead to the kfree() of the node and
>> it's properties _if_ the node reference count is correct.  So what I
>> said about a memory leak was incorrect in a perfect world (and my
>> memory was wrong).  However, this is not a perfect world and we know
>> that the reference count on devicetree nodes is often incorrect due
>> to bugs in common infrastructure and drivers.  This issue will not
>> be resolved until we pull all reference count manipulation into the
>> devicetree core.
>
> I don't get this yet. When I want some value from life tree, I do a node
> search, get a pointer and the core incremented its reference, can query
> the node and its properties, and when I'm done, I call of_node_put and
> forget about all pointers I got. What would you do differently?
>
>>  The net result is that we should not expect
>> overlay removal to correctly free all memory that was allocated
>> when applying the overlay.
>
> Depends on the overlay. If you do not modify existing nodes but only add
> new ones, it is fair to expect complete removal.
>
>>
>> I _think_ (but did not spend the time to verify) that there is a small
>> corner case memory leak even if the reference count on devicetree
>> nodes is correct.  If an overlay adds a property to an existing node
>> then removing the overlay will not kfree() the property, and it
>> will remain on the deadprops list.  There are some places that
>> properties are removed from deadprops, but I don't think they fully
>> resolve the issue.  Again, this is a corner case, and I am willing
>> to document it as a limitation until it gets fixed.

This doesn't solve all of your concern, but it gets me wondering
whether overlay_notify should add a of_node_get(fragment->overlay)
before doing the  blocking_notifier_call_chain and a of_node_put
afterwards.

>
> I ran into this the other day: If you modify an existing property, the
> old value will be put into deadprops and only be freed when the node is
> freed. It may come back from deadprops if a changeset comes around with
> the very same property object for another modification.
>
> But that means: if your overlay just adds nodes, all of them, including
> their deadprops from potential changes on top, will go away on overlay
> removal.
>
> BTW, here is my new code that exploits this to be leak-free:
> https://github.com/siemens/jailhouse/blob/156a93fcc02585d78d4418d3e6761cd72a65b359/driver/pci.c#L296
>
>>
>> Then returning to me going around in circles... This thread led me to
>> think that since since the overlay apply code copied data into never
>> freed memory (false premise, as you pointed out) that we did not
>> have to worry about drivers retaining pointers into overlay data
>> after the overlay had been freed (with the one remaining exposure
>> being via the overlay notifiers, which _might_ be easily resolved,
>> pending Alan's analysis) -- this would have been great news for
>> removing an issue for general use of overlays.
>>
>> But now we are back to the long-standing problem that we have no way
>> of knowing whether there are any live pointers to the memory that is
>> freed by of_changeset_destroy().  And I am not aware of any solution
>> to this problem other than changing the devicetree access API so that
>> it never returns any pointer into the live devicetree.
>
> I don't agree yet with this drastic measure until you can point me to
> code that pulls and stores pointers to arbitrary devicetree content
> without that node reference counting. The pattern we otherwise see all
> around it you get a pointer (or a set of them) along with the duty to
> explicitly drop it again by some put() operation.
>
>>
>> The practical impact of all of this, is if we can change the overlay
>> notifier parameters to include the overlay changeset instead of
>> the overlay devicetree, then I think that of_overlay_apply() will
>> be able to kfree() the overlay fdt and overlay devicetree.  And
>> if not of_overlay_apply(), then free_overlay_changeset().
>
> Isn't that just s/node/changeset/ without any other semantic changes? If
> the receiver of the changeset reference does not take care of lifecycle
> management for that object either, we are back at square #1. A changeset
> is just a gate to the nodes and properties that are currently passed
> directly.
>
> Jan

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

Hi Jan, Alan,

On 04/25/18 13:07, Jan Kiszka wrote:
> On 2018-04-25 20:40, Frank Rowand wrote:
>> On 04/24/18 22:23, Jan Kiszka wrote:
>>> On 2018-04-24 22:56, Frank Rowand wrote:
>>>> Hi Alan,
>>>>
>>>> On 04/23/18 15:38, Frank Rowand wrote:
>>>>> Hi Jan,
>>>>>
>>>>> + Alan Tull for fpga perspective
>>>>>
>>>>> On 04/22/18 03:30, Jan Kiszka wrote:
>>>>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>>>>> Hi Jan,
>>>>>>>>>>>
>>>>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>
>>>>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>>>>
>>>>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>>>>> original FDT.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>>>>
>>>>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>>>>> errors.
>>>>>>>>>>>>>
>>>>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>>>>> overlay loader.
>>>>>>>>>>>>
>>>>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>>>>
>>>>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>>>>> understand the use case and consider solutions.
>>>>>>>>>>>
>>>>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>>>>
>>>>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>>>>
>>>>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>>>>> scripts/dtc/libfdt/.
>>>>>>>>
>>>>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>>>>> is little reason to other than for early boot changes. And it is much
>>>>>>>> easier to work on unflattened trees.
>>>>>>>
>>>>>>> I just briefly looked into libfdt, and it would have meant building it
>>>>>>> into the module as there are no library functions exported by the kernel
>>>>>>> either. Another reason to drop that.
>>>>>>>
>>>>>>> What's apparently working now is the pattern I initially suggested:
>>>>>>> Register template with status = "disabled" as overlay, then prepare and
>>>>>>> apply changeset that contains all needed modifications and sets the
>>>>>>> status to "ok". I might be leaking additional resources, but to find
>>>>>>> that out, I will now finally have to resolve clean unbinding of the
>>>>>>> generic PCI host controller [1] first.
>>>>>>
>>>>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>>>>> {
>>>>>> 	[...]
>>>>>> 	/*
>>>>>> 	 * TODO
>>>>>> 	 *
>>>>>> 	 * would like to: kfree(ovcs->overlay_tree);
>>>>>> 	 * but can not since drivers may have pointers into this data
>>>>>> 	 *
>>>>>> 	 * would like to: kfree(ovcs->fdt);
>>>>>> 	 * but can not since drivers may have pointers into this data
>>>>>> 	 */
>>>>>>
>>>>>> 	kfree(ovcs);
>>>>>> }
>>>>>>
>>>>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>>>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>>>>> to those objects. I would say that's a regression of the new API.
>>>>>
>>>>> The problem already existed but it was hidden.  We have never been able to
>>>>> kfree() these object because we do not know if there are any pointers into
>>>>> these objects.  The new API makes the problem visible to kmemleak.
>>>>>
>>>>> The reason that we do not know if there are any pointers into these objects
>>>>> is that devicetree access APIs return pointers into the devicetree internal
>>>>> data structures (that is, into the overlay unflattened devicetree).  If we
>>>>> want to be able to do the kfree()s, we could change the devicetree access
>>>>> APIs.
>>>>>
>>>>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>>>>> also exposed is that the overlay unflattened devicetree property values
>>>>> are pointers into the overlay fdt.
>>>>>
>>>>> ** This paragraph becomes academic (and not needed) if the fix in the next
>>>>> paragraph can be implemented. **
>>>>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>>>>> (I would want to read through the code again to make sure I'm not missing
>>>>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>>>>> was modified so that property values were copied into newly allocated memory
>>>>> and the live tree property pointers were set to the copy instead of to
>>>>> the value in the fdt, then I _think_ the fdt could be freed in
>>>>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>>>>> frees a devicetree would also have to be aware of this change -- I'm not
>>>>> sure if that leads to ugly complications or if it is easy.  The other
>>>>> question to consider is whether to make the same change to
>>>>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>>>>> the base devicetree.  Doing so would increase the memory usage of the
>>>>> live tree (we would not be able to free the base fdt after unflattening
>>>>> it because we make the fdt visible in /sys/firmware/fdt -- though
>>>>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>>>
>>>> Question added below this paragraph.
>>>>
>>>>
>>>>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>>>>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>>>>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>>>>> into the overlay unflattened devicetree are then passed to the notifiers.
>>>>> (Again, I may be missing some other place that the overlay unflattened
>>>>> devicetree is made visible to other code -- a more thorough reading of
>>>>> the code is needed.) If the notifiers could be modified to accept the
>>>>> changeset list instead of of pointers to the fragments in the overlay
>>>>> unflattened devicetree then there would be no possibility of the notifiers
>>>>> keeping a pointer into the overlay fdt. I do not know if this is a
>>>>> practical change for the notifiers -- there are no callers of
>>>>> of_overlay_notifier_register() in the mainline kernel source. My
>>>>> recollection is that the overlay notifiers were added for the fpga
>>>>> subsystem.
>>>>
>>>> Can the fpga notifiers be changed to have the changeset as an input
>>>> instead of having the overlay devicetree fragment and target as an
>>>> input?
>>>>
>>>> The changeset lists nodes and properties to be added, but does not
>>>> expose any pointers to the overlay fdt or the overlay unflattened
>>>> devicetree.  This guarantees no leakage of pointers into the overlay
>>>> fdt or the overlay unflattened devicetree.  The changeset contains
>>>> pointers to copies of data, but those copies are never freed (and
>>>> thus they are yet another existing memory leak).
>>>
>>> Also they are freed, of course: When the last reference to the node they
>>> point to reaches 0 (e.g. triggered by of_changeset_destroy), that node
>>> goes away and takes down remaining dead properties. I've ran through
>>> this already. And I also made sure that my code is not triggering such
>>> kind of leaks as well.
>>
>> mea culpa.  I go around in circles while trying to remember all the
>> overlay related issues.  I needed to go back and read the code to
>> refresh my memory.  Thanks for the prod to re-read the code.
>>
>> Yes, of_changeset_destroy() will lead to the kfree() of the node and
>> it's properties _if_ the node reference count is correct.  So what I
>> said about a memory leak was incorrect in a perfect world (and my
>> memory was wrong).  However, this is not a perfect world and we know
>> that the reference count on devicetree nodes is often incorrect due
>> to bugs in common infrastructure and drivers.  This issue will not
>> be resolved until we pull all reference count manipulation into the
>> devicetree core.
> 
> I don't get this yet. When I want some value from life tree, I do a node
> search, get a pointer and the core incremented its reference, can query
> the node and its properties, and when I'm done, I call of_node_put and
> forget about all pointers I got. What would you do differently?

I don't know what causes the refcount errors.  I started looking at the
issue a couple of years ago, but did not finish and the issue did not
remain high enough of my todo list.

I added a printk() to of_node_get() and of_node_put(), then post
processed the console log to see where the reference count was being
modified.  I also had a version of the patch that also printed the call
stack so I could better investigate where the gets and puts were invoked
from.

Here is an example from a random 4.11-rc1 boot (the columns are
address of the device_node, minimum reference count seen, maximum
reference count seen, final reference count after the system
was booted, and full path of the node):

0xeefe759c    2   23   23  /
0xeefeaec8    2    4    3  /adsp-pil
0xeefe78a0    2    3    3  /aliases
0xeefe7734    2    3    3  /chosen
0xeefea9fc    2    4    4  /clocks
0xeefeac48    2    3    3  /clocks/sleep_clk
0xeefeaa98    2    3    3  /clocks/xo_board
0xeefea8f4    2    4    3  /cpu-pmu
0xeefe86f8    2   10   10  /cpus
0xeefe8830    2    4    4  /cpus/cpu@0
0xeefe8a6c    2    4    4  /cpus/cpu@1
0xeefe8ca8    2    4    4  /cpus/cpu@2
0xeefe8ee4    2    4    4  /cpus/cpu@3
0xeefe92cc    2    2    2  /cpus/idle-states
0xeefe9378    2    6    6  /cpus/idle-states/spc
0xeefe9120    2    2    2  /cpus/l2-cache
0xeefec7e4    2    4    2  /firmware
0xeefec888    2    4    3  /firmware/scm
0xeefe79dc    2    2    2  /memory
0xeefe7ae0    2   11   11  /reserved-memory
0xeefe8224    2    3    2  /reserved-memory/smem@fa00000
0xeeff5010    5    7    6  /smd
0xeeff50dc    2    4    3  /smd/adsp
0xeeff5218    2    4    3  /smd/modem
0xeeff5354    3    5    4  /smd/rpm
0xeeff548c    4    6    5  /smd/rpm/rpm_requests
0xeeff55a8    3   12    3  /smd/rpm/rpm_requests/pm8841-regulators
0xeeff56a4    2    4    4  /smd/rpm/rpm_requests/pm8841-regulators/s1
0xeeff57c8    2    4    4  /smd/rpm/rpm_requests/pm8841-regulators/s2
0xeeff5954    2    4    4  /smd/rpm/rpm_requests/pm8841-regulators/s3
0xeeff5a78    2    4    4  /smd/rpm/rpm_requests/pm8841-regulators/s4
0xeeff5b9c    2    4    4  /smd/rpm/rpm_requests/pm8841-regulators/s5
0xeeff5c58    2    4    4  /smd/rpm/rpm_requests/pm8841-regulators/s6
0xeeff5d14    2    4    4  /smd/rpm/rpm_requests/pm8841-regulators/s7
0xeeff5dd0    2    3    3  /smd/rpm/rpm_requests/pm8841-regulators/s8
0xeeff5e8c    3   37    3  /smd/rpm/rpm_requests/pm8941-regulators
0xeeff8a18    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/5vs1
0xeeff8adc    2    3    3  /smd/rpm/rpm_requests/pm8941-regulators/5vs2
0xeeff678c    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l1
0xeeff7308    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l10
0xeeff7460    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l11
0xeeff7584    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l12
0xeeff7710    2    7    7  /smd/rpm/rpm_requests/pm8941-regulators/l13
0xeeff78d0    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l14
0xeeff79f4    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l15
0xeeff7b18    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l16
0xeeff7c3c    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l17
0xeeff7d60    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l18
0xeeff7e84    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l19
0xeeff6918    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l2
0xeeff7fdc    2    7    7  /smd/rpm/rpm_requests/pm8941-regulators/l20
0xeeff8204    2    7    7  /smd/rpm/rpm_requests/pm8941-regulators/l21
0xeeff83c4    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l22
0xeeff84e8    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l23
0xeeff860c    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l24
0xeeff6a3c    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l3
0xeeff6b60    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l4
0xeeff6c84    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l5
0xeeff6da8    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l6
0xeeff6f68    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l7
0xeeff70c0    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l8
0xeeff71e4    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/l9
0xeeff87cc    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/lvs1
0xeeff8890    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/lvs2
0xeeff8954    2    4    4  /smd/rpm/rpm_requests/pm8941-regulators/lvs3
0xeeff60c0    2    8    8  /smd/rpm/rpm_requests/pm8941-regulators/s1
0xeeff62b4    2   10   10  /smd/rpm/rpm_requests/pm8941-regulators/s2
0xeeff6474    2   11   11  /smd/rpm/rpm_requests/pm8941-regulators/s3
0xeeff6668    2    5    5  /smd/rpm/rpm_requests/pm8941-regulators/s4
0xeefeb0d8    2    4    3  /smem
0xeefeb244    2    6    5  /smp2p-adsp
0xeefeb5dc    2   12   12  /smp2p-adsp/slave-kernel
0xeefeb798    2    6    5  /smp2p-modem
0xeefebc20    2    6    5  /smp2p-wcnss
0xeefec0a8    2    8    7  /smsm
0xeefec9c4    2   37   36  /soc
0xeefee954    2    4    3  /soc/clock-controller@f9088000
0xeefeeae4    2    4    3  /soc/clock-controller@f9098000
0xeefeec74    2    4    3  /soc/clock-controller@f90a8000
0xeefeee04    2    4    3  /soc/clock-controller@f90b8000
0xeefef0a8    2    5    5  /soc/clock-controller@fc400000
0xeefef450    2    4    4  /soc/clock-controller@fd8c0000
0xeeff471c    2    4    3  /soc/dma-controller@f9944000
0xeeff194c    2    2    2  /soc/i2c@f9924000
0xeeff1b90    2    2    2  /soc/i2c@f9964000
0xeeff1dd4    2    5    5  /soc/i2c@f9967000
0xeeff211c    2    3    3  /soc/i2c@f9967000/eeprom@52
0xeefecb2c    2   75   74  /soc/interrupt-controller@f9000000
0xeefef7c0    2    4    3  /soc/memory@fc428000
0xeeff0578    2   10   10  /soc/pinctrl@fd510000
0xeeff07f8    3    4    3  /soc/pinctrl@fd510000/i2c11
0xeeff0910    2    2    2  /soc/pinctrl@fd510000/i2c11/mux
0xeeff0f70    4    5    4  /soc/pinctrl@fd510000/sdhc1-pin-active
0xeeff10a0    2    2    2  /soc/pinctrl@fd510000/sdhc1-pin-active/clk
0xeeff11f8    2    2    2  /soc/pinctrl@fd510000/sdhc1-pin-active/cmd-data
0xeeff135c    2    3    2  /soc/pinctrl@fd510000/sdhc2-cd-pin-active
0xeeff1560    4    5    4  /soc/pinctrl@fd510000/sdhc2-pin-active
0xeeff1690    2    2    2  /soc/pinctrl@fd510000/sdhc2-pin-active/clk
0xeeff17e8    2    2    2  /soc/pinctrl@fd510000/sdhc2-pin-active/cmd-data
0xeeff0a28    6    6    6  /soc/pinctrl@fd510000/spi8_default
0xeefee790    2    4    3  /soc/power-controller@f9012000
0xeefee150    2    4    3  /soc/power-controller@f9089000
0xeefee2e0    2    4    3  /soc/power-controller@f9099000
0xeefee470    2    4    3  /soc/power-controller@f90a9000
0xeefee600    2    4    3  /soc/power-controller@f90b9000
0xeefecea8    2    7    7  /soc/qfprom@fc4bc000
0xeefed174    2    2    2  /soc/qfprom@fc4bc000/backup@440
0xeefed024    2    2    2  /soc/qfprom@fc4bc000/calib@d0
0xeefeef94    2    4    3  /soc/restart@fc4ab000
0xeeff0404    2    4    3  /soc/rng@f9bff000
0xeefefd04    2    4    4  /soc/sdhci@f9824900
0xeeff0084    2    4    4  /soc/sdhci@f98a4900
0xeefef93c    2    2    2  /soc/serial@f991d000
0xeefefb20    2    6    5  /soc/serial@f991e000
0xeeff229c    2  238  237  /soc/spmi@fc4cf000
0xeeff25e8    2    7    5  /soc/spmi@fc4cf000/pm8841@4
0xeeff2768    2    4    4  /soc/spmi@fc4cf000/pm8841@4/mpps@a000
0xeeff2928    2    4    3  /soc/spmi@fc4cf000/pm8841@4/temp-alarm@2400
0xeeff2a88    2    3    2  /soc/spmi@fc4cf000/pm8841@5
0xeeff2c08    2   14   12  /soc/spmi@fc4cf000/pm8941@0
0xeeff30d0    2    4    2  /soc/spmi@fc4cf000/pm8941@0/charger@1000
0xeeff42b0    2    2    2  /soc/spmi@fc4cf000/pm8941@0/coincell@2800
0xeeff3260    3    5    5  /soc/spmi@fc4cf000/pm8941@0/gpios@c000
0xeeff3488    2    3    2  /soc/spmi@fc4cf000/pm8941@0/gpios@c000/boost-bypass
0xeeff4124    2    4    3  /soc/spmi@fc4cf000/pm8941@0/iadc@3600
0xeeff3628    2    4    4  /soc/spmi@fc4cf000/pm8941@0/mpps@a000
0xeeff2f10    2    4    3  /soc/spmi@fc4cf000/pm8941@0/pwrkey@800
0xeeff2d88    2    4    3  /soc/spmi@fc4cf000/pm8941@0/rtc@6000
0xeeff37e8    2    4    3  /soc/spmi@fc4cf000/pm8941@0/temp-alarm@2400
0xeeff39e4    7    9    8  /soc/spmi@fc4cf000/pm8941@0/vadc@3100
0xeeff4410    3    4    3  /soc/spmi@fc4cf000/pm8941@1
0xeeff4590    2    2    2  /soc/spmi@fc4cf000/pm8941@1/wled@d800
0xeefecd2c    2    5    5  /soc/syscon@f9011000
0xeefef2d4    2    4    3  /soc/syscon@fd484000
0xeefef614    2    4    3  /soc/tcsr-mutex
0xeefed2c4    2    4    3  /soc/thermal-sensor@fc4a8000
0xeefed4ec    2   11   10  /soc/timer@f9020000
0xeefed6cc    2    3    3  /soc/timer@f9020000/frame@f9021000
0xeeff49ac    2    2    2  /soc/usb-phy@f9a55000
0xeeff4e00    2    2    2  /soc/usb@f9a55000
0xeefe9558    2    6    6  /thermal-zones
0xeefe9604    3    3    3  /thermal-zones/cpu-thermal0
0xeefe9758    4    4    4  /thermal-zones/cpu-thermal0/trips
0xeefe9810    2    3    3  /thermal-zones/cpu-thermal0/trips/trip0
0xeefe9968    2    3    3  /thermal-zones/cpu-thermal0/trips/trip1
0xeefe9ac0    3    3    3  /thermal-zones/cpu-thermal1
0xeefe9c14    4    4    4  /thermal-zones/cpu-thermal1/trips
0xeefe9ccc    2    3    3  /thermal-zones/cpu-thermal1/trips/trip0
0xeefe9e24    2    3    3  /thermal-zones/cpu-thermal1/trips/trip1
0xeefe9f7c    3    3    3  /thermal-zones/cpu-thermal2
0xeefea0d0    4    4    4  /thermal-zones/cpu-thermal2/trips
0xeefea188    2    3    3  /thermal-zones/cpu-thermal2/trips/trip0
0xeefea2e0    2    3    3  /thermal-zones/cpu-thermal2/trips/trip1
0xeefea438    3    3    3  /thermal-zones/cpu-thermal3
0xeefea58c    4    4    4  /thermal-zones/cpu-thermal3/trips
0xeefea644    2    3    3  /thermal-zones/cpu-thermal3/trips/trip0
0xeefea79c    2    3    3  /thermal-zones/cpu-thermal3/trips/trip1
0xeefead90    2    4    3  /timer
0xeeff8ba0    2    4    4  /vreg-boost
0xeeff8e4c    2    4    3  /vreg-vph-pwr

> 
>>  The net result is that we should not expect
>> overlay removal to correctly free all memory that was allocated
>> when applying the overlay.
> 
> Depends on the overlay. If you do not modify existing nodes but only add
> new ones, it is fair to expect complete removal.

Only if reference counts are correct.  I assert that reference counts are
often not correct.


>>
>> I _think_ (but did not spend the time to verify) that there is a small
>> corner case memory leak even if the reference count on devicetree
>> nodes is correct.  If an overlay adds a property to an existing node
>> then removing the overlay will not kfree() the property, and it
>> will remain on the deadprops list.  There are some places that
>> properties are removed from deadprops, but I don't think they fully
>> resolve the issue.  Again, this is a corner case, and I am willing
>> to document it as a limitation until it gets fixed.
> 
> I ran into this the other day: If you modify an existing property, the
> old value will be put into deadprops and only be freed when the node is
> freed. It may come back from deadprops if a changeset comes around with
> the very same property object for another modification.
> 
> But that means: if your overlay just adds nodes, all of them, including
> their deadprops from potential changes on top, will go away on overlay
> removal.
> 
> BTW, here is my new code that exploits this to be leak-free:
> https://github.com/siemens/jailhouse/blob/156a93fcc02585d78d4418d3e6761cd72a65b359/driver/pci.c#L296
> 
>>
>> Then returning to me going around in circles... This thread led me to
>> think that since since the overlay apply code copied data into never
>> freed memory (false premise, as you pointed out) that we did not
>> have to worry about drivers retaining pointers into overlay data
>> after the overlay had been freed (with the one remaining exposure
>> being via the overlay notifiers, which _might_ be easily resolved,
>> pending Alan's analysis) -- this would have been great news for
>> removing an issue for general use of overlays.
>>
>> But now we are back to the long-standing problem that we have no way
>> of knowing whether there are any live pointers to the memory that is
>> freed by of_changeset_destroy().  And I am not aware of any solution
>> to this problem other than changing the devicetree access API so that
>> it never returns any pointer into the live devicetree.
> 
> I don't agree yet with this drastic measure until you can point me to
> code that pulls and stores pointers to arbitrary devicetree content
> without that node reference counting. The pattern we otherwise see all
> around it you get a pointer (or a set of them) along with the duty to
> explicitly drop it again by some put() operation.

And we have long history of these rules not being followed.  Evidence
is the patches we get to correct incorrect of_node_get() / of_node_put()
usage and the random node reference count report I included above.

That is also conflating the issue of reference counts being correct
and whether pointers are held after a reference count is decremented.
I am not going to audit the kernel and future patches to ensure that.


>>
>> The practical impact of all of this, is if we can change the overlay
>> notifier parameters to include the overlay changeset instead of
>> the overlay devicetree, then I think that of_overlay_apply() will
>> be able to kfree() the overlay fdt and overlay devicetree.  And
>> if not of_overlay_apply(), then free_overlay_changeset().
> 
> Isn't that just s/node/changeset/ without any other semantic changes? If
> the receiver of the changeset reference does not take care of lifecycle
> management for that object either, we are back at square #1. A changeset
> is just a gate to the nodes and properties that are currently passed
> directly.

Sigh.  Yes.  Thanks for pointing that out.

So my thought of changing the arguments of the overlay notifier was wrong
and we should just stick with the current calling convention for the
overlay notifiers.  But we need to document the "no retaining pointers
into the overlay devicetree" for the overlay notifiers.

Thank you Alan for looking into the alternative approach.

OK, being pragmatic about not letting failure to achieve perfection stand
in the way of achieving good enough -- I am willing to accept the review
burden of overlay notifiers and that review will be an acceptable defense
against errors for the overlay notifiers.  Although my long-term goal is
to change to devicetree access APIs to not return pointers into the
devicetree (in this case, the overlay devicetree, not the live devicetree)
to preclude the possibility of pointer leakage.

I will repeat this conditional acceptance in reply to the proposed patch
that you sent.

-Frank

> 
> Jan
> .
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Alan Tull]

On Tue, Apr 24, 2018 at 3:56 PM, Frank Rowand <frowand.list@gmail.com> wrote:
> Hi Alan,
>
> On 04/23/18 15:38, Frank Rowand wrote:
>> Hi Jan,
>>
>> + Alan Tull for fpga perspective
>>
>> On 04/22/18 03:30, Jan Kiszka wrote:
>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>> Hi Jan,
>>>>>>>>
>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>> Hi Frank,
>>>>>>>>>
>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>
>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>
>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>> original FDT.
>>>>>>>>>>
>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>
>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>> errors.
>>>>>>>>>>
>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>> overlay loader.
>>>>>>>>>
>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>
>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>
>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>> understand the use case and consider solutions.
>>>>>>>>
>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>
>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>
>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>> scripts/dtc/libfdt/.
>>>>>
>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>> is little reason to other than for early boot changes. And it is much
>>>>> easier to work on unflattened trees.
>>>>
>>>> I just briefly looked into libfdt, and it would have meant building it
>>>> into the module as there are no library functions exported by the kernel
>>>> either. Another reason to drop that.
>>>>
>>>> What's apparently working now is the pattern I initially suggested:
>>>> Register template with status = "disabled" as overlay, then prepare and
>>>> apply changeset that contains all needed modifications and sets the
>>>> status to "ok". I might be leaking additional resources, but to find
>>>> that out, I will now finally have to resolve clean unbinding of the
>>>> generic PCI host controller [1] first.
>>>
>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>> {
>>>      [...]
>>>      /*
>>>       * TODO
>>>       *
>>>       * would like to: kfree(ovcs->overlay_tree);
>>>       * but can not since drivers may have pointers into this data
>>>       *
>>>       * would like to: kfree(ovcs->fdt);
>>>       * but can not since drivers may have pointers into this data
>>>       */
>>>
>>>      kfree(ovcs);
>>> }
>>>
>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>> to those objects. I would say that's a regression of the new API.
>>
>> The problem already existed but it was hidden.  We have never been able to
>> kfree() these object because we do not know if there are any pointers into
>> these objects.  The new API makes the problem visible to kmemleak.
>>
>> The reason that we do not know if there are any pointers into these objects
>> is that devicetree access APIs return pointers into the devicetree internal
>> data structures (that is, into the overlay unflattened devicetree).  If we
>> want to be able to do the kfree()s, we could change the devicetree access
>> APIs.
>>
>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>> also exposed is that the overlay unflattened devicetree property values
>> are pointers into the overlay fdt.
>>
>> ** This paragraph becomes academic (and not needed) if the fix in the next
>> paragraph can be implemented. **
>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>> (I would want to read through the code again to make sure I'm not missing
>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>> was modified so that property values were copied into newly allocated memory
>> and the live tree property pointers were set to the copy instead of to
>> the value in the fdt, then I _think_ the fdt could be freed in
>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>> frees a devicetree would also have to be aware of this change -- I'm not
>> sure if that leads to ugly complications or if it is easy.  The other
>> question to consider is whether to make the same change to
>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>> the base devicetree.  Doing so would increase the memory usage of the
>> live tree (we would not be able to free the base fdt after unflattening
>> it because we make the fdt visible in /sys/firmware/fdt -- though
>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>
> Question added below this paragraph.
>
>
>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>> into the overlay unflattened devicetree are then passed to the notifiers.
>> (Again, I may be missing some other place that the overlay unflattened
>> devicetree is made visible to other code -- a more thorough reading of
>> the code is needed.) If the notifiers could be modified to accept the
>> changeset list instead of of pointers to the fragments in the overlay
>> unflattened devicetree then there would be no possibility of the notifiers
>> keeping a pointer into the overlay fdt. I do not know if this is a
>> practical change for the notifiers -- there are no callers of
>> of_overlay_notifier_register() in the mainline kernel source. My
>> recollection is that the overlay notifiers were added for the fpga
>> subsystem.
>
> Can the fpga notifiers be changed to have the changeset as an input
> instead of having the overlay devicetree fragment and target as an
> input?

I'll look into it.  Just to be clear, are you suggesting passing
struct overlay_changeset instead in the notifier?

struct overlay_changeset and struct fragment would have to be moved to a header.

>
> The changeset lists nodes and properties to be added, but does not
> expose any pointers to the overlay fdt or the overlay unflattened
> devicetree.  This guarantees no leakage of pointers into the overlay
> fdt or the overlay unflattened devicetree.  The changeset contains
> pointers to copies of data, but those copies are never freed (and
> thus they are yet another existing memory leak).
>
> -Frank
>
>> Why is overlay_notify() the only issue related to unknown users having
>> pointers into the overlay fdt?  The answer is that the overlay code
>> does not directly expose the overlay unflattened devicetree (and thus
>> indirectly the overlay fdt) to the live devicetree -- when the
>> overlay code creates the overlay changeset, it copies from the
>> overlay unflattened devicetree and overlay fdt and only exposes
>> pointers to the copies.
>>
>> And hopefully the issues with the overlay unflattened devicetree can
>> be resolved in the same way as for the overlay fdt.
>>
>> -Frank
>>
>>
>>
>

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

On 04/25/18 07:59, Alan Tull wrote:
> On Tue, Apr 24, 2018 at 3:56 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>> Hi Alan,
>>
>> On 04/23/18 15:38, Frank Rowand wrote:
>>> Hi Jan,
>>>
>>> + Alan Tull for fpga perspective
>>>
>>> On 04/22/18 03:30, Jan Kiszka wrote:
>>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>>> Hi Jan,
>>>>>>>>>
>>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>>> Hi Frank,
>>>>>>>>>>
>>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>>
>>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>>
>>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>>> original FDT.
>>>>>>>>>>>
>>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>>
>>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>>> errors.
>>>>>>>>>>>
>>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>>> overlay loader.
>>>>>>>>>>
>>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>>
>>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>>
>>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>>> understand the use case and consider solutions.
>>>>>>>>>
>>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>>
>>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>>
>>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>>> scripts/dtc/libfdt/.
>>>>>>
>>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>>> is little reason to other than for early boot changes. And it is much
>>>>>> easier to work on unflattened trees.
>>>>>
>>>>> I just briefly looked into libfdt, and it would have meant building it
>>>>> into the module as there are no library functions exported by the kernel
>>>>> either. Another reason to drop that.
>>>>>
>>>>> What's apparently working now is the pattern I initially suggested:
>>>>> Register template with status = "disabled" as overlay, then prepare and
>>>>> apply changeset that contains all needed modifications and sets the
>>>>> status to "ok". I might be leaking additional resources, but to find
>>>>> that out, I will now finally have to resolve clean unbinding of the
>>>>> generic PCI host controller [1] first.
>>>>
>>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>>> {
>>>>      [...]
>>>>      /*
>>>>       * TODO
>>>>       *
>>>>       * would like to: kfree(ovcs->overlay_tree);
>>>>       * but can not since drivers may have pointers into this data
>>>>       *
>>>>       * would like to: kfree(ovcs->fdt);
>>>>       * but can not since drivers may have pointers into this data
>>>>       */
>>>>
>>>>      kfree(ovcs);
>>>> }
>>>>
>>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>>> to those objects. I would say that's a regression of the new API.
>>>
>>> The problem already existed but it was hidden.  We have never been able to
>>> kfree() these object because we do not know if there are any pointers into
>>> these objects.  The new API makes the problem visible to kmemleak.
>>>
>>> The reason that we do not know if there are any pointers into these objects
>>> is that devicetree access APIs return pointers into the devicetree internal
>>> data structures (that is, into the overlay unflattened devicetree).  If we
>>> want to be able to do the kfree()s, we could change the devicetree access
>>> APIs.
>>>
>>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>>> also exposed is that the overlay unflattened devicetree property values
>>> are pointers into the overlay fdt.
>>>
>>> ** This paragraph becomes academic (and not needed) if the fix in the next
>>> paragraph can be implemented. **
>>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>>> (I would want to read through the code again to make sure I'm not missing
>>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>>> was modified so that property values were copied into newly allocated memory
>>> and the live tree property pointers were set to the copy instead of to
>>> the value in the fdt, then I _think_ the fdt could be freed in
>>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>>> frees a devicetree would also have to be aware of this change -- I'm not
>>> sure if that leads to ugly complications or if it is easy.  The other
>>> question to consider is whether to make the same change to
>>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>>> the base devicetree.  Doing so would increase the memory usage of the
>>> live tree (we would not be able to free the base fdt after unflattening
>>> it because we make the fdt visible in /sys/firmware/fdt -- though
>>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>
>> Question added below this paragraph.
>>
>>
>>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>>> into the overlay unflattened devicetree are then passed to the notifiers.
>>> (Again, I may be missing some other place that the overlay unflattened
>>> devicetree is made visible to other code -- a more thorough reading of
>>> the code is needed.) If the notifiers could be modified to accept the
>>> changeset list instead of of pointers to the fragments in the overlay
>>> unflattened devicetree then there would be no possibility of the notifiers
>>> keeping a pointer into the overlay fdt. I do not know if this is a
>>> practical change for the notifiers -- there are no callers of
>>> of_overlay_notifier_register() in the mainline kernel source. My
>>> recollection is that the overlay notifiers were added for the fpga
>>> subsystem.
>>
>> Can the fpga notifiers be changed to have the changeset as an input
>> instead of having the overlay devicetree fragment and target as an
>> input?
> 
> I'll look into it.  Just to be clear, are you suggesting passing
> struct overlay_changeset instead in the notifier?

Ah, poor phrasing on my part.  I meant a "struct of_changeset", as is
passed into __of_changeset_apply_entries(), which is called from
of_overlay_apply().  This means that the call to overlay_notify()
would have to move down a few lines to just after calling
build_changeset().


> struct overlay_changeset and struct fragment would have to be moved to a header.
> 
>>
>> The changeset lists nodes and properties to be added, but does not
>> expose any pointers to the overlay fdt or the overlay unflattened
>> devicetree.  This guarantees no leakage of pointers into the overlay
>> fdt or the overlay unflattened devicetree.  The changeset contains
>> pointers to copies of data, but those copies are never freed (and
>> thus they are yet another existing memory leak).
>>
>> -Frank
>>
>>> Why is overlay_notify() the only issue related to unknown users having
>>> pointers into the overlay fdt?  The answer is that the overlay code
>>> does not directly expose the overlay unflattened devicetree (and thus
>>> indirectly the overlay fdt) to the live devicetree -- when the
>>> overlay code creates the overlay changeset, it copies from the
>>> overlay unflattened devicetree and overlay fdt and only exposes
>>> pointers to the copies.
>>>
>>> And hopefully the issues with the overlay unflattened devicetree can
>>> be resolved in the same way as for the overlay fdt.
>>>
>>> -Frank
>>>
>>>
>>>
>>
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Alan Tull]

On Wed, Apr 25, 2018 at 12:41 PM, Frank Rowand <frowand.list@gmail.com> wrote:
> On 04/25/18 07:59, Alan Tull wrote:
>> On Tue, Apr 24, 2018 at 3:56 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>> Hi Alan,
>>>
>>> On 04/23/18 15:38, Frank Rowand wrote:
>>>> Hi Jan,
>>>>
>>>> + Alan Tull for fpga perspective
>>>>
>>>> On 04/22/18 03:30, Jan Kiszka wrote:
>>>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>>>> Hi Jan,
>>>>>>>>>>
>>>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>
>>>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>>>
>>>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>>>
>>>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>>>> original FDT.
>>>>>>>>>>>>
>>>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>>>
>>>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>>>> errors.
>>>>>>>>>>>>
>>>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>>>> overlay loader.
>>>>>>>>>>>
>>>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>>>
>>>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>>>
>>>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>>>> understand the use case and consider solutions.
>>>>>>>>>>
>>>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>>>
>>>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>>>
>>>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>>>> scripts/dtc/libfdt/.
>>>>>>>
>>>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>>>> is little reason to other than for early boot changes. And it is much
>>>>>>> easier to work on unflattened trees.
>>>>>>
>>>>>> I just briefly looked into libfdt, and it would have meant building it
>>>>>> into the module as there are no library functions exported by the kernel
>>>>>> either. Another reason to drop that.
>>>>>>
>>>>>> What's apparently working now is the pattern I initially suggested:
>>>>>> Register template with status = "disabled" as overlay, then prepare and
>>>>>> apply changeset that contains all needed modifications and sets the
>>>>>> status to "ok". I might be leaking additional resources, but to find
>>>>>> that out, I will now finally have to resolve clean unbinding of the
>>>>>> generic PCI host controller [1] first.
>>>>>
>>>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>>>> {
>>>>>      [...]
>>>>>      /*
>>>>>       * TODO
>>>>>       *
>>>>>       * would like to: kfree(ovcs->overlay_tree);
>>>>>       * but can not since drivers may have pointers into this data
>>>>>       *
>>>>>       * would like to: kfree(ovcs->fdt);
>>>>>       * but can not since drivers may have pointers into this data
>>>>>       */
>>>>>
>>>>>      kfree(ovcs);
>>>>> }
>>>>>
>>>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>>>> to those objects. I would say that's a regression of the new API.
>>>>
>>>> The problem already existed but it was hidden.  We have never been able to
>>>> kfree() these object because we do not know if there are any pointers into
>>>> these objects.  The new API makes the problem visible to kmemleak.
>>>>
>>>> The reason that we do not know if there are any pointers into these objects
>>>> is that devicetree access APIs return pointers into the devicetree internal
>>>> data structures (that is, into the overlay unflattened devicetree).  If we
>>>> want to be able to do the kfree()s, we could change the devicetree access
>>>> APIs.
>>>>
>>>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>>>> also exposed is that the overlay unflattened devicetree property values
>>>> are pointers into the overlay fdt.
>>>>
>>>> ** This paragraph becomes academic (and not needed) if the fix in the next
>>>> paragraph can be implemented. **
>>>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>>>> (I would want to read through the code again to make sure I'm not missing
>>>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>>>> was modified so that property values were copied into newly allocated memory
>>>> and the live tree property pointers were set to the copy instead of to
>>>> the value in the fdt, then I _think_ the fdt could be freed in
>>>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>>>> frees a devicetree would also have to be aware of this change -- I'm not
>>>> sure if that leads to ugly complications or if it is easy.  The other
>>>> question to consider is whether to make the same change to
>>>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>>>> the base devicetree.  Doing so would increase the memory usage of the
>>>> live tree (we would not be able to free the base fdt after unflattening
>>>> it because we make the fdt visible in /sys/firmware/fdt -- though
>>>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>>
>>> Question added below this paragraph.
>>>
>>>
>>>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>>>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>>>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>>>> into the overlay unflattened devicetree are then passed to the notifiers.
>>>> (Again, I may be missing some other place that the overlay unflattened
>>>> devicetree is made visible to other code -- a more thorough reading of
>>>> the code is needed.) If the notifiers could be modified to accept the
>>>> changeset list instead of of pointers to the fragments in the overlay
>>>> unflattened devicetree then there would be no possibility of the notifiers
>>>> keeping a pointer into the overlay fdt. I do not know if this is a
>>>> practical change for the notifiers -- there are no callers of
>>>> of_overlay_notifier_register() in the mainline kernel source. My
>>>> recollection is that the overlay notifiers were added for the fpga
>>>> subsystem.
>>>
>>> Can the fpga notifiers be changed to have the changeset as an input
>>> instead of having the overlay devicetree fragment and target as an
>>> input?
>>
>> I'll look into it.  Just to be clear, are you suggesting passing
>> struct overlay_changeset instead in the notifier?
>
> Ah, poor phrasing on my part.  I meant a "struct of_changeset", as is
> passed into __of_changeset_apply_entries(), which is called from
> of_overlay_apply().  This means that the call to overlay_notify()
> would have to move down a few lines to just after calling
> build_changeset().

Ah yes, I thought it was looking too easy. :)  I had it working with
notify data passing the struct overlay_changeset, I was about to send
you the patch.

The FPGA code really wants the data as fragments, so it will know
first of all what the target is.  Passing of_changeset would mean that
the code receiving the notification would be essentially be tasked
reassembling the changeset into fragments. Perhaps it could be done,
but it could easily be broken by changes to overlay.c and it would be
ugly.   That breaks the exact thing that I added overlay notifications
for.  I really need to see for each fragment what the target is, and
all the properties together.

>
>
>> struct overlay_changeset and struct fragment would have to be moved to a header.
>>
>>>
>>> The changeset lists nodes and properties to be added, but does not
>>> expose any pointers to the overlay fdt or the overlay unflattened
>>> devicetree.  This guarantees no leakage of pointers into the overlay
>>> fdt or the overlay unflattened devicetree.  The changeset contains
>>> pointers to copies of data, but those copies are never freed (and
>>> thus they are yet another existing memory leak).
>>>
>>> -Frank
>>>
>>>> Why is overlay_notify() the only issue related to unknown users having
>>>> pointers into the overlay fdt?  The answer is that the overlay code
>>>> does not directly expose the overlay unflattened devicetree (and thus
>>>> indirectly the overlay fdt) to the live devicetree -- when the
>>>> overlay code creates the overlay changeset, it copies from the
>>>> overlay unflattened devicetree and overlay fdt and only exposes
>>>> pointers to the copies.
>>>>
>>>> And hopefully the issues with the overlay unflattened devicetree can
>>>> be resolved in the same way as for the overlay fdt.
>>>>
>>>> -Frank
>>>>
>>>>
>>>>
>>>
>>
>

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

Hi Alan,

On 04/25/18 11:19, Alan Tull wrote:
> On Wed, Apr 25, 2018 at 12:41 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>> On 04/25/18 07:59, Alan Tull wrote:
>>> On Tue, Apr 24, 2018 at 3:56 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>> Hi Alan,
>>>>
>>>> On 04/23/18 15:38, Frank Rowand wrote:
>>>>> Hi Jan,
>>>>>
>>>>> + Alan Tull for fpga perspective
>>>>>
>>>>> On 04/22/18 03:30, Jan Kiszka wrote:
>>>>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>>>>> Hi Jan,
>>>>>>>>>>>
>>>>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>
>>>>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>>>>
>>>>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>>>>> original FDT.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>>>>
>>>>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>>>>> errors.
>>>>>>>>>>>>>
>>>>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>>>>> overlay loader.
>>>>>>>>>>>>
>>>>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>>>>
>>>>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>>>>> understand the use case and consider solutions.
>>>>>>>>>>>
>>>>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>>>>
>>>>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>>>>
>>>>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>>>>> scripts/dtc/libfdt/.
>>>>>>>>
>>>>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>>>>> is little reason to other than for early boot changes. And it is much
>>>>>>>> easier to work on unflattened trees.
>>>>>>>
>>>>>>> I just briefly looked into libfdt, and it would have meant building it
>>>>>>> into the module as there are no library functions exported by the kernel
>>>>>>> either. Another reason to drop that.
>>>>>>>
>>>>>>> What's apparently working now is the pattern I initially suggested:
>>>>>>> Register template with status = "disabled" as overlay, then prepare and
>>>>>>> apply changeset that contains all needed modifications and sets the
>>>>>>> status to "ok". I might be leaking additional resources, but to find
>>>>>>> that out, I will now finally have to resolve clean unbinding of the
>>>>>>> generic PCI host controller [1] first.
>>>>>>
>>>>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>>>>> {
>>>>>>      [...]
>>>>>>      /*
>>>>>>       * TODO
>>>>>>       *
>>>>>>       * would like to: kfree(ovcs->overlay_tree);
>>>>>>       * but can not since drivers may have pointers into this data
>>>>>>       *
>>>>>>       * would like to: kfree(ovcs->fdt);
>>>>>>       * but can not since drivers may have pointers into this data
>>>>>>       */
>>>>>>
>>>>>>      kfree(ovcs);
>>>>>> }
>>>>>>
>>>>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>>>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>>>>> to those objects. I would say that's a regression of the new API.
>>>>>
>>>>> The problem already existed but it was hidden.  We have never been able to
>>>>> kfree() these object because we do not know if there are any pointers into
>>>>> these objects.  The new API makes the problem visible to kmemleak.
>>>>>
>>>>> The reason that we do not know if there are any pointers into these objects
>>>>> is that devicetree access APIs return pointers into the devicetree internal
>>>>> data structures (that is, into the overlay unflattened devicetree).  If we
>>>>> want to be able to do the kfree()s, we could change the devicetree access
>>>>> APIs.
>>>>>
>>>>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>>>>> also exposed is that the overlay unflattened devicetree property values
>>>>> are pointers into the overlay fdt.
>>>>>
>>>>> ** This paragraph becomes academic (and not needed) if the fix in the next
>>>>> paragraph can be implemented. **
>>>>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>>>>> (I would want to read through the code again to make sure I'm not missing
>>>>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>>>>> was modified so that property values were copied into newly allocated memory
>>>>> and the live tree property pointers were set to the copy instead of to
>>>>> the value in the fdt, then I _think_ the fdt could be freed in
>>>>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>>>>> frees a devicetree would also have to be aware of this change -- I'm not
>>>>> sure if that leads to ugly complications or if it is easy.  The other
>>>>> question to consider is whether to make the same change to
>>>>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>>>>> the base devicetree.  Doing so would increase the memory usage of the
>>>>> live tree (we would not be able to free the base fdt after unflattening
>>>>> it because we make the fdt visible in /sys/firmware/fdt -- though
>>>>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>>>
>>>> Question added below this paragraph.
>>>>
>>>>
>>>>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>>>>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>>>>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>>>>> into the overlay unflattened devicetree are then passed to the notifiers.
>>>>> (Again, I may be missing some other place that the overlay unflattened
>>>>> devicetree is made visible to other code -- a more thorough reading of
>>>>> the code is needed.) If the notifiers could be modified to accept the
>>>>> changeset list instead of of pointers to the fragments in the overlay
>>>>> unflattened devicetree then there would be no possibility of the notifiers
>>>>> keeping a pointer into the overlay fdt. I do not know if this is a
>>>>> practical change for the notifiers -- there are no callers of
>>>>> of_overlay_notifier_register() in the mainline kernel source. My
>>>>> recollection is that the overlay notifiers were added for the fpga
>>>>> subsystem.
>>>>
>>>> Can the fpga notifiers be changed to have the changeset as an input
>>>> instead of having the overlay devicetree fragment and target as an
>>>> input?
>>>
>>> I'll look into it.  Just to be clear, are you suggesting passing
>>> struct overlay_changeset instead in the notifier?
>>
>> Ah, poor phrasing on my part.  I meant a "struct of_changeset", as is
>> passed into __of_changeset_apply_entries(), which is called from
>> of_overlay_apply().  This means that the call to overlay_notify()
>> would have to move down a few lines to just after calling
>> build_changeset().
> 
> Ah yes, I thought it was looking too easy. :)  I had it working with
> notify data passing the struct overlay_changeset, I was about to send
> you the patch.
> 
> The FPGA code really wants the data as fragments, so it will know
> first of all what the target is.  Passing of_changeset would mean that
> the code receiving the notification would be essentially be tasked
> reassembling the changeset into fragments. Perhaps it could be done,
> but it could easily be broken by changes to overlay.c and it would be
> ugly.   That breaks the exact thing that I added overlay notifications
> for.  I really need to see for each fragment what the target is, and
> all the properties together.

approach I proposed here is not solving the underlying problem, but just
moving it to another place.  So I am not going to pursue this approach.

-Frank

> 
>>
>>
>>> struct overlay_changeset and struct fragment would have to be moved to a header.
>>>
>>>>
>>>> The changeset lists nodes and properties to be added, but does not
>>>> expose any pointers to the overlay fdt or the overlay unflattened
>>>> devicetree.  This guarantees no leakage of pointers into the overlay
>>>> fdt or the overlay unflattened devicetree.  The changeset contains
>>>> pointers to copies of data, but those copies are never freed (and
>>>> thus they are yet another existing memory leak).
>>>>
>>>> -Frank
>>>>
>>>>> Why is overlay_notify() the only issue related to unknown users having
>>>>> pointers into the overlay fdt?  The answer is that the overlay code
>>>>> does not directly expose the overlay unflattened devicetree (and thus
>>>>> indirectly the overlay fdt) to the live devicetree -- when the
>>>>> overlay code creates the overlay changeset, it copies from the
>>>>> overlay unflattened devicetree and overlay fdt and only exposes
>>>>> pointers to the copies.
>>>>>
>>>>> And hopefully the issues with the overlay unflattened devicetree can
>>>>> be resolved in the same way as for the overlay fdt.
>>>>>
>>>>> -Frank
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
> 

Re: [PATCH v7 2/5] of: change overlay apply input data from unflattened to FDT

[Frank Rowand]

On 04/25/18 17:07, Frank Rowand wrote:
> Hi Alan,
> 
> On 04/25/18 11:19, Alan Tull wrote:
>> On Wed, Apr 25, 2018 at 12:41 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>> On 04/25/18 07:59, Alan Tull wrote:
>>>> On Tue, Apr 24, 2018 at 3:56 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>> Hi Alan,
>>>>>
>>>>> On 04/23/18 15:38, Frank Rowand wrote:
>>>>>> Hi Jan,
>>>>>>
>>>>>> + Alan Tull for fpga perspective
>>>>>>
>>>>>> On 04/22/18 03:30, Jan Kiszka wrote:
>>>>>>> On 2018-04-11 07:42, Jan Kiszka wrote:
>>>>>>>> On 2018-04-05 23:12, Rob Herring wrote:
>>>>>>>>> On Thu, Apr 5, 2018 at 2:28 PM, Frank Rowand <frowand.list@gmail.com> wrote:
>>>>>>>>>> On 04/05/18 12:13, Jan Kiszka wrote:
>>>>>>>>>>> On 2018-04-05 20:59, Frank Rowand wrote:
>>>>>>>>>>>> Hi Jan,
>>>>>>>>>>>>
>>>>>>>>>>>> On 04/04/18 15:35, Jan Kiszka wrote:
>>>>>>>>>>>>> Hi Frank,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2018-03-04 01:17, frowand.list@gmail.com wrote:
>>>>>>>>>>>>>> From: Frank Rowand <frank.rowand@sony.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Move duplicating and unflattening of an overlay flattened devicetree
>>>>>>>>>>>>>> (FDT) into the overlay application code.  To accomplish this,
>>>>>>>>>>>>>> of_overlay_apply() is replaced by of_overlay_fdt_apply().
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The copy of the FDT (aka "duplicate FDT") now belongs to devicetree
>>>>>>>>>>>>>> code, which is thus responsible for freeing the duplicate FDT.  The
>>>>>>>>>>>>>> caller of of_overlay_fdt_apply() remains responsible for freeing the
>>>>>>>>>>>>>> original FDT.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The unflattened devicetree now belongs to devicetree code, which is
>>>>>>>>>>>>>> thus responsible for freeing the unflattened devicetree.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> These ownership changes prevent early freeing of the duplicated FDT
>>>>>>>>>>>>>> or the unflattened devicetree, which could result in use after free
>>>>>>>>>>>>>> errors.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> of_overlay_fdt_apply() is a private function for the anticipated
>>>>>>>>>>>>>> overlay loader.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We are using of_fdt_unflatten_tree + of_overlay_apply in the
>>>>>>>>>>>>> (out-of-tree) Jailhouse loader driver in order to register a virtual
>>>>>>>>>>>>> device during hypervisor activation with Linux. The DT overlay is
>>>>>>>>>>>>> created from a a template but modified prior to application to account
>>>>>>>>>>>>> for runtime-specific parameters. See [1] for the current implementation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm now wondering how to model that scenario best with the new API.
>>>>>>>>>>>>> Given that the loader lost ownership of the unflattened tree but the
>>>>>>>>>>>>> modification API exist only for the that DT state, I'm not yet seeing a
>>>>>>>>>>>>> clear solution. Should we apply the template in disabled form (status =
>>>>>>>>>>>>> "disabled"), modify it, and then activate it while it is already applied?
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you for the pointer to the driver - that makes it much easier to
>>>>>>>>>>>> understand the use case and consider solutions.
>>>>>>>>>>>>
>>>>>>>>>>>> If you can make the changes directly on the FDT instead of on the
>>>>>>>>>>>> expanded devicetree, then you could move to the new API.
>>>>>>>>>>>
>>>>>>>>>>> Are there some examples/references on how to edit FDTs in-place in the
>>>>>>>>>>> kernel? I'd like to avoid writing the n-th FDT parser/generator.
>>>>>>>>>>
>>>>>>>>>> I don't know of any existing in-kernel edits of the FDT (but they might
>>>>>>>>>> exist).  The functions to access an FDT are in libfdt, which is in
>>>>>>>>>> scripts/dtc/libfdt/.
>>>>>>>>>
>>>>>>>>> Let's please not go down that route of doing FDT modifications. There
>>>>>>>>> is little reason to other than for early boot changes. And it is much
>>>>>>>>> easier to work on unflattened trees.
>>>>>>>>
>>>>>>>> I just briefly looked into libfdt, and it would have meant building it
>>>>>>>> into the module as there are no library functions exported by the kernel
>>>>>>>> either. Another reason to drop that.
>>>>>>>>
>>>>>>>> What's apparently working now is the pattern I initially suggested:
>>>>>>>> Register template with status = "disabled" as overlay, then prepare and
>>>>>>>> apply changeset that contains all needed modifications and sets the
>>>>>>>> status to "ok". I might be leaking additional resources, but to find
>>>>>>>> that out, I will now finally have to resolve clean unbinding of the
>>>>>>>> generic PCI host controller [1] first.
>>>>>>>
>>>>>>> static void free_overlay_changeset(struct overlay_changeset *ovcs)
>>>>>>> {
>>>>>>>      [...]
>>>>>>>      /*
>>>>>>>       * TODO
>>>>>>>       *
>>>>>>>       * would like to: kfree(ovcs->overlay_tree);
>>>>>>>       * but can not since drivers may have pointers into this data
>>>>>>>       *
>>>>>>>       * would like to: kfree(ovcs->fdt);
>>>>>>>       * but can not since drivers may have pointers into this data
>>>>>>>       */
>>>>>>>
>>>>>>>      kfree(ovcs);
>>>>>>> }
>>>>>>>
>>>>>>> What's this? I have kmemleak now jumping at me over this. Who is suppose
>>>>>>> to plug these leaks? The caller of of_overlay_fdt_apply has no pointers
>>>>>>> to those objects. I would say that's a regression of the new API.
>>>>>>
>>>>>> The problem already existed but it was hidden.  We have never been able to
>>>>>> kfree() these object because we do not know if there are any pointers into
>>>>>> these objects.  The new API makes the problem visible to kmemleak.
>>>>>>
>>>>>> The reason that we do not know if there are any pointers into these objects
>>>>>> is that devicetree access APIs return pointers into the devicetree internal
>>>>>> data structures (that is, into the overlay unflattened devicetree).  If we
>>>>>> want to be able to do the kfree()s, we could change the devicetree access
>>>>>> APIs.
>>>>>>
>>>>>> The reason that pointers into the overlay flattened tree (ovcs->fdt) are
>>>>>> also exposed is that the overlay unflattened devicetree property values
>>>>>> are pointers into the overlay fdt.
>>>>>>
>>>>>> ** This paragraph becomes academic (and not needed) if the fix in the next
>>>>>> paragraph can be implemented. **
>>>>>> I _think_ that the fdt issue __for overlays__ can be fixed somewhat easily.
>>>>>> (I would want to read through the code again to make sure I'm not missing
>>>>>> any issues.)  If the of_fdt_unflatten_tree() called by of_overlay_fdt_apply()
>>>>>> was modified so that property values were copied into newly allocated memory
>>>>>> and the live tree property pointers were set to the copy instead of to
>>>>>> the value in the fdt, then I _think_ the fdt could be freed in
>>>>>> of_overlay_fdt_apply() after calling of_overlay_apply().  The code that
>>>>>> frees a devicetree would also have to be aware of this change -- I'm not
>>>>>> sure if that leads to ugly complications or if it is easy.  The other
>>>>>> question to consider is whether to make the same change to
>>>>>> of_fdt_unflatten_tree() when it is called in early boot to unflatten
>>>>>> the base devicetree.  Doing so would increase the memory usage of the
>>>>>> live tree (we would not be able to free the base fdt after unflattening
>>>>>> it because we make the fdt visible in /sys/firmware/fdt -- though
>>>>>> _maybe_ that could be conditioned on CONFIG_KEXEC).
>>>>>
>>>>> Question added below this paragraph.
>>>>>
>>>>>
>>>>>> But all of the complexity of that fix is _only_ because of_overlay_apply()
>>>>>> and of_overlay_remove() call overlay_notify(), passing in the overlay
>>>>>> unflattened devicetree (which has pointers into the overlay fdt). Pointers
>>>>>> into the overlay unflattened devicetree are then passed to the notifiers.
>>>>>> (Again, I may be missing some other place that the overlay unflattened
>>>>>> devicetree is made visible to other code -- a more thorough reading of
>>>>>> the code is needed.) If the notifiers could be modified to accept the
>>>>>> changeset list instead of of pointers to the fragments in the overlay
>>>>>> unflattened devicetree then there would be no possibility of the notifiers
>>>>>> keeping a pointer into the overlay fdt. I do not know if this is a
>>>>>> practical change for the notifiers -- there are no callers of
>>>>>> of_overlay_notifier_register() in the mainline kernel source. My
>>>>>> recollection is that the overlay notifiers were added for the fpga
>>>>>> subsystem.
>>>>>
>>>>> Can the fpga notifiers be changed to have the changeset as an input
>>>>> instead of having the overlay devicetree fragment and target as an
>>>>> input?
>>>>
>>>> I'll look into it.  Just to be clear, are you suggesting passing
>>>> struct overlay_changeset instead in the notifier?
>>>
>>> Ah, poor phrasing on my part.  I meant a "struct of_changeset", as is
>>> passed into __of_changeset_apply_entries(), which is called from
>>> of_overlay_apply().  This means that the call to overlay_notify()
>>> would have to move down a few lines to just after calling
>>> build_changeset().
>>
>> Ah yes, I thought it was looking too easy. :)  I had it working with
>> notify data passing the struct overlay_changeset, I was about to send
>> you the patch.
>>
>> The FPGA code really wants the data as fragments, so it will know
>> first of all what the target is.  Passing of_changeset would mean that
>> the code receiving the notification would be essentially be tasked
>> reassembling the changeset into fragments. Perhaps it could be done,
>> but it could easily be broken by changes to overlay.c and it would be
>> ugly.   That breaks the exact thing that I added overlay notifications
>> for.  I really need to see for each fragment what the target is, and
>> all the properties together.
> 

I inadvertently chopped off part of my reply.  I think is also meant to
say something like:

As Jan pointed out in another email, the

> approach I proposed here is not solving the underlying problem, but just
> moving it to another place.  So I am not going to pursue this approach.
> 
> -Frank
> 
>>
>>>
>>>
>>>> struct overlay_changeset and struct fragment would have to be moved to a header.
>>>>
>>>>>
>>>>> The changeset lists nodes and properties to be added, but does not
>>>>> expose any pointers to the overlay fdt or the overlay unflattened
>>>>> devicetree.  This guarantees no leakage of pointers into the overlay
>>>>> fdt or the overlay unflattened devicetree.  The changeset contains
>>>>> pointers to copies of data, but those copies are never freed (and
>>>>> thus they are yet another existing memory leak).
>>>>>
>>>>> -Frank
>>>>>
>>>>>> Why is overlay_notify() the only issue related to unknown users having
>>>>>> pointers into the overlay fdt?  The answer is that the overlay code
>>>>>> does not directly expose the overlay unflattened devicetree (and thus
>>>>>> indirectly the overlay fdt) to the live devicetree -- when the
>>>>>> overlay code creates the overlay changeset, it copies from the
>>>>>> overlay unflattened devicetree and overlay fdt and only exposes
>>>>>> pointers to the copies.
>>>>>>
>>>>>> And hopefully the issues with the overlay unflattened devicetree can
>>>>>> be resolved in the same way as for the overlay fdt.
>>>>>>
>>>>>> -Frank
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
> 
> 

[PATCH v7 3/5] of: Documentation: of_overlay_apply() replaced by of_overlay_fdt_apply()

[frowand.list]

From: Frank Rowand <frank.rowand@sony.com>

Signed-off-by: Frank Rowand <frank.rowand@sony.com>
---
 Documentation/devicetree/overlay-notes.txt | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Documentation/devicetree/overlay-notes.txt b/Documentation/devicetree/overlay-notes.txt
index c4aa0adf13ec..5175a24d387e 100644
--- a/Documentation/devicetree/overlay-notes.txt
+++ b/Documentation/devicetree/overlay-notes.txt
@@ -87,8 +87,8 @@ Overlay in-kernel API
 
 The API is quite easy to use.
 
-1. Call of_overlay_apply() to create and apply an overlay changeset. The return
-value is an error or a cookie identifying this overlay.
+1. Call of_overlay_fdt_apply() to create and apply an overlay changeset. The
+return value is an error or a cookie identifying this overlay.
 
 2. Call of_overlay_remove() to remove and cleanup the overlay changeset
 previously created via the call to of_overlay_apply(). Removal of an overlay
-- 
Frank Rowand <frank.rowand@sony.com>

[PATCH v7 4/5] of: convert unittest overlay devicetree source to sugar syntax

[frowand.list]

From: Frank Rowand <frank.rowand@sony.com>

The unittest-data overlays have been pulled into proper overlay
devicetree source files without changing their format.  The
next step is to convert them to use sugar syntax instead of
hand coding overlay fragments structure.

A few of the overlays can not be converted because they test
absolute target paths in the overlay fragment.  dtc does not
generate this type of target:
  overlay_0.dts
  overlay_1.dts
  overlay_12.dts
  overlay_13.dts

Two pre-existing unittest overlay devicetree source files are
also converted:
  overlay_bad_phandle.dts
  overlay_bad_symbol.dts

Signed-off-by: Frank Rowand <frank.rowand@sony.com>
---

There are checkpatch warnings.  I have reviewed them and feel they
can be ignored.  They are pre-existing warnings of un-documented
bindings of made up (fake) compatibles in devicetree source for
test overlays.

 drivers/of/unittest-data/overlay.dts             | 101 ++++++++++-------------
 drivers/of/unittest-data/overlay_10.dts          |  39 ++++-----
 drivers/of/unittest-data/overlay_11.dts          |  40 ++++-----
 drivers/of/unittest-data/overlay_15.dts          |  41 ++++-----
 drivers/of/unittest-data/overlay_2.dts           |  11 +--
 drivers/of/unittest-data/overlay_3.dts           |  11 +--
 drivers/of/unittest-data/overlay_4.dts           |  23 ++----
 drivers/of/unittest-data/overlay_5.dts           |  11 +--
 drivers/of/unittest-data/overlay_6.dts           |  13 +--
 drivers/of/unittest-data/overlay_7.dts           |  13 +--
 drivers/of/unittest-data/overlay_8.dts           |  13 +--
 drivers/of/unittest-data/overlay_9.dts           |  13 +--
 drivers/of/unittest-data/overlay_bad_phandle.dts |  23 ++----
 drivers/of/unittest-data/overlay_bad_symbol.dts  |  25 ++----
 drivers/of/unittest-data/tests-overlay.dtsi      |   4 +-
 15 files changed, 148 insertions(+), 233 deletions(-)

diff --git a/drivers/of/unittest-data/overlay.dts b/drivers/of/unittest-data/overlay.dts
index ab5e89b5e27e..3bbc59e922fe 100644
--- a/drivers/of/unittest-data/overlay.dts
+++ b/drivers/of/unittest-data/overlay.dts
@@ -2,76 +2,63 @@
 /dts-v1/;
 /plugin/;
 
-/ {
+&electric_1 {
 
-	fragment@0 {
-		target = <&electric_1>;
+	status = "okay";
 
-		__overlay__ {
-			status = "okay";
-
-			hvac_2: hvac-large-1 {
-				compatible = "ot,hvac-large";
-				heat-range = < 40 75 >;
-				cool-range = < 65 80 >;
-			};
-		};
+	hvac_2: hvac-large-1 {
+		compatible = "ot,hvac-large";
+		heat-range = < 40 75 >;
+		cool-range = < 65 80 >;
 	};
+};
 
-	fragment@1 {
-		target = <&rides_1>;
-
-		__overlay__ {
-			#address-cells = <1>;
-			#size-cells = <1>;
-			status = "okay";
-
-			ride@100 {
-				#address-cells = <1>;
-				#size-cells = <1>;
-
-				track@30 {
-					incline-up = < 48 32 16 >;
-				};
+&rides_1 {
 
-				track@40 {
-					incline-up = < 47 31 15 >;
-				};
-			};
+	#address-cells = <1>;
+	#size-cells = <1>;
+	status = "okay";
 
-			ride_200: ride@200 {
-				#address-cells = <1>;
-				#size-cells = <1>;
-				compatible = "ot,ferris-wheel";
-				reg = < 0x00000200 0x100 >;
-				hvac-provider = < &hvac_2 >;
-				hvac-thermostat = < 27 32 > ;
-				hvac-zones = < 12 5 >;
-				hvac-zone-names = "operator", "snack-bar";
-				spin-controller = < &spin_ctrl_1 3 >;
-				spin-rph = < 30 >;
-				gondolas = < 16 >;
-				gondola-capacity = < 6 >;
+	ride@100 {
+		#address-cells = <1>;
+		#size-cells = <1>;
 
-				ride_200_left: track@10 {
-					reg = < 0x00000010 0x10 >;
-				};
+		track@30 {
+			incline-up = < 48 32 16 >;
+		};
 
-				ride_200_right: track@20 {
-					reg = < 0x00000020 0x10 >;
-				};
-			};
+		track@40 {
+			incline-up = < 47 31 15 >;
 		};
 	};
 
-	fragment@2 {
-		target = <&lights_2>;
+	ride_200: ride@200 {
+		#address-cells = <1>;
+		#size-cells = <1>;
+		compatible = "ot,ferris-wheel";
+		reg = < 0x00000200 0x100 >;
+		hvac-provider = < &hvac_2 >;
+		hvac-thermostat = < 27 32 > ;
+		hvac-zones = < 12 5 >;
+		hvac-zone-names = "operator", "snack-bar";
+		spin-controller = < &spin_ctrl_1 3 >;
+		spin-rph = < 30 >;
+		gondolas = < 16 >;
+		gondola-capacity = < 6 >;
+
+		ride_200_left: track@10 {
+			reg = < 0x00000010 0x10 >;
+		};
 
-		__overlay__ {
-			status = "okay";
-			color = "purple", "white", "red", "green";
-			rate = < 3 256 >;
+		ride_200_right: track@20 {
+			reg = < 0x00000020 0x10 >;
 		};
 	};
+};
+
+&lights_2 {
 
+	status = "okay";
+	color = "purple", "white", "red", "green";
+	rate = < 3 256 >;
 };
diff --git a/drivers/of/unittest-data/overlay_10.dts b/drivers/of/unittest-data/overlay_10.dts
index 445925a10cd3..73993bf23bf8 100644
--- a/drivers/of/unittest-data/overlay_10.dts
+++ b/drivers/of/unittest-data/overlay_10.dts
@@ -2,33 +2,26 @@
 /dts-v1/;
 /plugin/;
 
-/ {
-	/* overlay_10 */
-	/* overlays 8, 9, 10, 11 application and removal in bad sequence */
+/* overlay_10 */
+/* overlays 8, 9, 10, 11 application and removal in bad sequence */
 
-	fragment@0 {
-		target-path = "/testcase-data/overlay-node/test-bus";
-		__overlay__ {
+&unittest_test_bus {
+	/* suppress DTC warning */
+	#address-cells = <1>;
+	#size-cells = <0>;
 
-			/* suppress DTC warning */
-			#address-cells = <1>;
-			#size-cells = <0>;
+	test-unittest10 {
+		compatible = "unittest";
+		status = "okay";
+		reg = <10>;
 
-			test-unittest10 {
-				compatible = "unittest";
-				status = "okay";
-				reg = <10>;
+		#address-cells = <1>;
+		#size-cells = <0>;
 
-				#address-cells = <1>;
-				#size-cells = <0>;
-
-				test-unittest101 {
-					compatible = "unittest";
-					status = "okay";
-					reg = <1>;
-				};
-
-			};
+		test-unittest101 {
+			compatible = "unittest";
+			status = "okay";
+			reg = <1>;
 		};
 	};
 };
diff --git a/drivers/of/unittest-data/overlay_11.dts b/drivers/of/unittest-data/overlay_11.dts
index c1d14f34359e..9a79b253a809 100644
--- a/drivers/of/unittest-data/overlay_11.dts
+++ b/drivers/of/unittest-data/overlay_11.dts
@@ -2,33 +2,27 @@
 /dts-v1/;
 /plugin/;
 
-/ {
-	/* overlay_11 */
-	/* overlays 8, 9, 10, 11 application and removal in bad sequence */
+/* overlay_11 */
+/* overlays 8, 9, 10, 11 application and removal in bad sequence */
 
-	fragment@0 {
-		target-path = "/testcase-data/overlay-node/test-bus";
-		__overlay__ {
+&unittest_test_bus {
+	/* suppress DTC warning */
+	#address-cells = <1>;
+	#size-cells = <0>;
 
-			/* suppress DTC warning */
-			#address-cells = <1>;
-			#size-cells = <0>;
+	test-unittest11 {
+		compatible = "unittest";
+		status = "okay";
+		reg = <11>;
 
-			test-unittest11 {
-				compatible = "unittest";
-				status = "okay";
-				reg = <11>;
+		#address-cells = <1>;
+		#size-cells = <0>;
 
-				#address-cells = <1>;
-				#size-cells = <0>;
-
-				test-unittest111 {
-					compatible = "unittest";
-					status = "okay";
-					reg = <1>;
-				};
-
-			};
+		test-unittest111 {
+			compatible = "unittest";
+			status = "okay";
+			reg = <1>;
 		};
+
 	};
 };
diff --git a/drivers/of/unittest-data/overlay_15.dts b/drivers/of/unittest-data/overlay_15.dts
index 44e44c62b739..b98f2514df4b 100644
--- a/drivers/of/unittest-data/overlay_15.dts
+++ b/drivers/of/unittest-data/overlay_15.dts
@@ -2,33 +2,28 @@
 /dts-v1/;
 /plugin/;
 
-/ {
-	/* overlay_15 - mux overlay */
+/* overlay_15 - mux overlay */
 
-	fragment@0 {
-		target-path = "/testcase-data/overlay-node/test-bus/i2c-test-bus";
-		__overlay__ {
-			#address-cells = <1>;
-			#size-cells = <0>;
-			test-unittest15 {
-				reg = <11>;
-				compatible = "unittest-i2c-mux";
-				status = "okay";
+&unittest_i2c_test_bus {
+	#address-cells = <1>;
+	#size-cells = <0>;
+	test-unittest15 {
+		reg = <11>;
+		compatible = "unittest-i2c-mux";
+		status = "okay";
 
-				#address-cells = <1>;
-				#size-cells = <0>;
+		#address-cells = <1>;
+		#size-cells = <0>;
 
-				i2c@0 {
-					#address-cells = <1>;
-					#size-cells = <0>;
-					reg = <0>;
+		i2c@0 {
+			#address-cells = <1>;
+			#size-cells = <0>;
+			reg = <0>;
 
-					test-mux-dev {
-						reg = <32>;
-						compatible = "unittest-i2c-dev";
-						status = "okay";
-					};
-				};
+			test-mux-dev {
+				reg = <32>;
+				compatible = "unittest-i2c-dev";
+				status = "okay";
 			};
 		};
 	};
diff --git a/drivers/of/unittest-data/overlay_2.dts b/drivers/of/unittest-data/overlay_2.dts
index cf1e4245b7ce..db8684ba89d9 100644
--- a/drivers/of/unittest-data/overlay_2.dts
+++ b/drivers/of/unittest-data/overlay_2.dts
@@ -2,13 +2,8 @@
 /dts-v1/;
 /plugin/;
 
-/ {
-	/* overlay_2 -  enable using label */
+/* overlay_2 -  enable using label */
 
-	fragment@0 {
-		target = <&unittest2>;
-		__overlay__ {
-			status = "okay";
-		};
-	};
+&unittest2 {
+	status = "okay";
 };
diff --git a/drivers/of/unittest-data/overlay_3.dts b/drivers/of/unittest-data/overlay_3.dts
index 158dc44fc20a..40f289e7c237 100644
--- a/drivers/of/unittest-data/overlay_3.dts
+++ b/drivers/of/unittest-data/overlay_3.dts
@@ -2,13 +2,8 @@
 /dts-v1/;
 /plugin/;
 
-/ {
-	/* overlay_3 - disable using label */
+/* overlay_3 - disable using label */
 
-	fragment@0 {
-		target = <&unittest3>;
-		__overlay__ {
-			status = "disabled";
-		};
-	};
+&unittest3 {
+	status = "disabled";
 };
diff --git a/drivers/of/unittest-data/overlay_4.dts b/drivers/of/unittest-data/overlay_4.dts
index b4a2e6c6b016..a8a77ddf9abe 100644
--- a/drivers/of/unittest-data/overlay_4.dts
+++ b/drivers/of/unittest-data/overlay_4.dts
@@ -2,22 +2,17 @@
 /dts-v1/;
 /plugin/;
 
-/ {
-	/* overlay_4 - test insertion of a full node */
+/* overlay_4 - test insertion of a full node */
 
-	fragment@0 {
-		target = <&unittestbus>;
-		__overlay__ {
+&unittest_test_bus {
 
-			/* suppress DTC warning */
-			#address-cells = <1>;
-			#size-cells = <0>;
+	/* suppress DTC warning */
+	#address-cells = <1>;
+	#size-cells = <0>;
 
-			test-unittest4 {
-				compatible = "unittest";
-				status = "okay";
-				reg = <4>;
-			};
-		};
+	test-unittest4 {
+		compatible = "unittest";
+		status = "okay";
+		reg = <4>;
 	};
 };
diff --git a/drivers/of/unittest-data/overlay_5.dts b/drivers/of/unittest-data/overlay_5.dts
index 02ad25c1f19c..706f5f1b737c 100644
--- a/drivers/of/unittest-data/overlay_5.dts
+++ b/drivers/of/unittest-data/overlay_5.dts
@@ -2,13 +2,8 @@
 /dts-v1/;
 /plugin/;
 
-/ {
-	/* overlay_5 - test overlay apply revert */
+/* overlay_5 - test overlay apply revert */
 
-	fragment@0 {
-		target-path = "/testcase-data/overlay-node/test-bus/test-unittest5";
-		__overlay__ {
-			status = "okay";
-		};
-	};
+&unittest5 {
+	status = "okay";
 };
diff --git a/drivers/of/unittest-data/overlay_6.dts b/drivers/of/unittest-data/overlay_6.dts
index a14e965f5497..21a7fa4ca45e 100644
--- a/drivers/of/unittest-data/overlay_6.dts
+++ b/drivers/of/unittest-data/overlay_6.dts
@@ -2,14 +2,9 @@
 /dts-v1/;
 /plugin/;
 
-/ {
-	/* overlay_6 */
-	/* overlays 6, 7 application and removal in sequence */
+/* overlay_6 */
+/* overlays 6, 7 application and removal in sequence */
 
-	fragment@0 {
-		target-path = "/testcase-data/overlay-node/test-bus/test-unittest6";
-		__overlay__ {
-			status = "okay";
-		};
-	};
+&unittest6 {
+	status = "okay";
 };
diff --git a/drivers/of/unittest-data/overlay_7.dts b/drivers/of/unittest-data/overlay_7.dts
index 4bd7e423209c..58ba1bb51b50 100644
--- a/drivers/of/unittest-data/overlay_7.dts
+++ b/drivers/of/unittest-data/overlay_7.dts
@@ -2,14 +2,9 @@
 /dts-v1/;
 /plugin/;
 
-/ {
-	/* overlay_7 */
-	/* overlays 6, 7 application and removal in sequence */
+/* overlay_7 */
+/* overlays 6, 7 application and removal in sequence */
 
-	fragment@0 {
-		target-path = "/testcase-data/overlay-node/test-bus/test-unittest7";
-		__overlay__ {
-			status = "okay";
-		};
-	};
+&unittest7 {
+	status = "okay";
 };
diff --git a/drivers/of/unittest-data/overlay_8.dts b/drivers/of/unittest-data/overlay_8.dts
index 5b21c53945a9..e9718d118e38 100644
--- a/drivers/of/unittest-data/overlay_8.dts
+++ b/drivers/of/unittest-data/overlay_8.dts
@@ -2,14 +2,9 @@
 /dts-v1/;
 /plugin/;
 
-/ {
-	/* overlay_8 */
-	/* overlays 8, 9, 10, 11 application and removal in bad sequence */
+/* overlay_8 */
+/* overlays 8, 9, 10, 11 application and removal in bad sequence */
 
-	fragment@0 {
-		target-path = "/testcase-data/overlay-node/test-bus/test-unittest8";
-		__overlay__ {
-			status = "okay";
-		};
-	};
+&unittest8 {
+	status = "okay";
 };
diff --git a/drivers/of/unittest-data/overlay_9.dts b/drivers/of/unittest-data/overlay_9.dts
index 20ff055a5349..b35e23edae50 100644
--- a/drivers/of/unittest-data/overlay_9.dts
+++ b/drivers/of/unittest-data/overlay_9.dts
@@ -2,14 +2,9 @@
 /dts-v1/;
 /plugin/;
 
-/ {
-	/* overlay_9 */
-	/* overlays 8, 9, 10, 11 application and removal in bad sequence */
+/* overlay_9 */
+/* overlays 8, 9, 10, 11 application and removal in bad sequence */
 
-	fragment@0 {
-		target-path = "/testcase-data/overlay-node/test-bus/test-unittest8";
-		__overlay__ {
-			property-foo = "bar";
-		};
-	};
+&unittest8 {
+	property-foo = "bar";
 };
diff --git a/drivers/of/unittest-data/overlay_bad_phandle.dts b/drivers/of/unittest-data/overlay_bad_phandle.dts
index 4d5b99723bad..83b797360318 100644
--- a/drivers/of/unittest-data/overlay_bad_phandle.dts
+++ b/drivers/of/unittest-data/overlay_bad_phandle.dts
@@ -2,20 +2,13 @@
 /dts-v1/;
 /plugin/;
 
-/ {
-
-	fragment@0 {
-		target = <&electric_1>;
-
-		__overlay__ {
-
-			// This label should cause an error when the overlay
-			// is applied.  There is already a phandle value
-			// in the base tree for motor-1.
-			spin_ctrl_1_conflict: motor-1 {
-				accelerate = < 3 >;
-				decelerate = < 5 >;
-			};
-		};
+&electric_1 {
+
+	// This label should cause an error when the overlay
+	// is applied.  There is already a phandle value
+	// in the base tree for motor-1.
+	spin_ctrl_1_conflict: motor-1 {
+		accelerate = < 3 >;
+		decelerate = < 5 >;
 	};
 };
diff --git a/drivers/of/unittest-data/overlay_bad_symbol.dts b/drivers/of/unittest-data/overlay_bad_symbol.dts
index 135052ee1517..98c6d1de144a 100644
--- a/drivers/of/unittest-data/overlay_bad_symbol.dts
+++ b/drivers/of/unittest-data/overlay_bad_symbol.dts
@@ -2,22 +2,15 @@
 /dts-v1/;
 /plugin/;
 
-/ {
+&electric_1 {
 
-	fragment@0 {
-		target = <&electric_1>;
-
-		__overlay__ {
-
-			// This label should cause an error when the overlay
-			// is applied.  There is already a symbol hvac_1
-			// in the base tree
-			hvac_1: hvac-medium-2 {
-				compatible = "ot,hvac-medium";
-				heat-range = < 50 75 >;
-				cool-range = < 60 80 >;
-			};
-
-		};
+	// This label should cause an error when the overlay
+	// is applied.  There is already a symbol hvac_1
+	// in the base tree
+	hvac_1: hvac-medium-2 {
+		compatible = "ot,hvac-medium";
+		heat-range = < 50 75 >;
+		cool-range = < 60 80 >;
 	};
+
 };
diff --git a/drivers/of/unittest-data/tests-overlay.dtsi b/drivers/of/unittest-data/tests-overlay.dtsi
index fa2fb43bccac..25cf397b8f6b 100644
--- a/drivers/of/unittest-data/tests-overlay.dtsi
+++ b/drivers/of/unittest-data/tests-overlay.dtsi
@@ -5,7 +5,7 @@
 		overlay-node {
 
 			/* test bus */
-			unittestbus: test-bus {
+			unittest_test_bus: test-bus {
 				compatible = "simple-bus";
 				#address-cells = <1>;
 				#size-cells = <0>;
@@ -70,7 +70,7 @@
 					reg = <8>;
 				};
 
-				i2c-test-bus {
+				unittest_i2c_test_bus: i2c-test-bus {
 					compatible = "unittest-i2c-bus";
 					status = "okay";
 					reg = <50>;
-- 
Frank Rowand <frank.rowand@sony.com>

[PATCH v7 5/5] of: improve reporting invalid overlay target path

[frowand.list]

From: Frank Rowand <frank.rowand@sony.com>

Errors while developing the patch to create of_overlay_fdt_apply()
exposed inadequate error messages to debug problems when overlay
devicetree fragment nodes contain an invalid target path.  Improve
the messages in find_target_node() to remedy this.

Signed-off-by: Frank Rowand <frank.rowand@sony.com>
---

changes from v2:
  - add fragment node name as suggested by Geert
  - existing error message printed short node name, thus not
    discriminating between fragments; change to full node name
  - existing error message printed node address, which is devicetree
    internal debugging, not useful in an overlay apply error message;
    remove node address from message

 drivers/of/overlay.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/drivers/of/overlay.c b/drivers/of/overlay.c
index e3d7f69a8333..b930e05d1215 100644
--- a/drivers/of/overlay.c
+++ b/drivers/of/overlay.c
@@ -488,20 +488,30 @@ static int build_changeset(struct overlay_changeset *ovcs)
  */
 static struct device_node *find_target_node(struct device_node *info_node)
 {
+	struct device_node *node;
 	const char *path;
 	u32 val;
 	int ret;
 
 	ret = of_property_read_u32(info_node, "target", &val);
-	if (!ret)
-		return of_find_node_by_phandle(val);
+	if (!ret) {
+		node = of_find_node_by_phandle(val);
+		if (!node)
+			pr_err("find target, node: %pOF, phandle 0x%x not found\n",
+			       info_node, val);
+		return node;
+	}
 
 	ret = of_property_read_string(info_node, "target-path", &path);
-	if (!ret)
-		return of_find_node_by_path(path);
+	if (!ret) {
+		node =  of_find_node_by_path(path);
+		if (!node)
+			pr_err("find target, node: %pOF, path '%s' not found\n",
+			       info_node, path);
+		return node;
+	}
 
-	pr_err("Failed to find target for node %p (%s)\n",
-		info_node, info_node->name);
+	pr_err("find target, node: %pOF, no target property\n", info_node);
 
 	return NULL;
 }
-- 
Frank Rowand <frank.rowand@sony.com>