[XEN][PATCH v7 00/19] dynamic node programming using overlay dtbo

[XEN][PATCH v7 00/19] dynamic node programming using overlay dtbo

[Vikram Garhwal]

Hi,
This patch series is for introducing dynamic programming i.e. add/remove the
devices during run time. Using "xl dt_overlay" a device can be added/removed
with dtbo.

For adding a node using dynamic programming:
    1. flatten device tree overlay node will be added to a fdt
    2. Updated fdt will be unflattened to a new dt_host_new
    3. Extract the newly added node information from dt_host_new
    4. Add the added node under correct parent in original dt_host.
    3. Map/Permit interrupt and iomem region as required.

For removing a node:
    1. Find the node with given path.
    2. Check if the node is used by any of domus. Removes the node only when
        it's not used by any domain.
    3. Removes IRQ permissions and MMIO access.
    5. Find the node in dt_host and delete the device node entry from dt_host.
    6. Free the overlay_tracker entry which means free dt_host_new also(created
in adding node step).

The main purpose of this series to address first part of the dynamic programming
i.e. making Xen aware of new device tree node which means updating the dt_host
with overlay node information. Here we are adding/removing node from dt_host,
and checking/set IOMMU and IRQ permission but never mapping them to any domain.
Right now, mapping/Un-mapping will happen only when a new domU is
created/destroyed using "xl create".

To map IOREQ and IOMMU during runtime, there will be another small series after
this one where we will do the actual IOMMU and IRQ mapping to a running domain
and will call unmap_mmio_regions() to remove the mapping.

Change Log:
 v5 -> v6:
    Add separate patch for memory allocation failure in __unflatten_device_tree().
    Move __unflatten_device_tree() function type changes to single patch.
    Add error propagation for failures in unflatten_dt_node.
    Change CONFIG_OVERLAY_DTB status to "ARM: Tech Preview".
    xen/smmu: Add remove_device callback for smmu_iommu ops:
        Added check to see if device is currently used.
    common/device_tree: Add rwlock for dt_host:
        Addressed feedback from Henry to rearrange code.
    xen/arm: Implement device tree node removal functionalities:
        Changed file name to dash format.
        Addressed Michal's comments.
    Rectified formatting related errors pointed by Michal.

 v4 -> v5:
    Split patch 01/16 to two patches. One with function type changes and another
        with changes inside unflatten_device_tree().
    Change dt_overlay xl command to dt-overlay.
    Protect overlay functionality with CONFIG(arm).
    Fix rwlock issues.
    Move include "device_tree.h" to c file where arch_cpu_init() is called and
        forward declare dt_device_node. This was done to avoid circular deps b/w
        device_tree.h and rwlock.h
    Address Michal's comment on coding style.

 v3 -> v4:
    Add support for adding node's children.
    Add rwlock to dt_host functions.
    Corrected fdt size issue when applying overlay into it.
    Add memory allocation fail handling for unflatten_device_tree().
    Changed xl overlay to xl dt_overlay.
    Correct commit messages.
    Addressed code issue from v3 review.

 v2 -> v3:
    Moved overlay functionalities to dt_overlay.c file.
    Renamed XEN_SYSCTL_overlay to XEN_SYSCTL_dt_overlay.
    Add dt_* prefix to overlay_add/remove_nodes.
    Added dtdevs_lock to protect iommu_add_dt_device().
    For iommu, moved spin_lock to caller.
    Address code issue from v2 review.

 v1 -> v2:
    Add support for multiple node addition/removal using dtbo.
    Replaced fpga-add and fpga-remove with one hypercall overlay_op.
    Moved common domain_build.c function to device.c
    Add OVERLAY_DTB configuration.
    Renamed overlay_get_target() to fdt_overlay_get_target().
    Split remove_device patch into two patches.
    Moved overlay_add/remove code to sysctl and changed it from domctl to sysctl.
    Added all overlay code under CONFIG_OVERLAY_DTB
    Renamed all tool domains fpga function to overlay
    Addressed code issues from v1 review.

Regards,
Vikram

Vikram Garhwal (19):
  common/device_tree: handle memory allocation failure in
    __unflatten_device_tree()
  common/device_tree.c: unflatten_device_tree() propagate errors
  xen/arm/device: Remove __init from function type
  common/device_tree: change __unflatten_device_tree() type
  xen/arm: Add CONFIG_OVERLAY_DTB
  libfdt: Keep fdt functions after init for CONFIG_OVERLAY_DTB.
  libfdt: overlay: change overlay_get_target()
  xen/device-tree: Add device_tree_find_node_by_path() to find nodes in
    device tree
  xen/iommu: Move spin_lock from iommu_dt_device_is_assigned to caller
  xen/iommu: protect iommu_add_dt_device() with dtdevs_lock
  xen/iommu: Introduce iommu_remove_dt_device()
  xen/smmu: Add remove_device callback for smmu_iommu ops
  asm/smp.h: Fix circular dependency for device_tree.h and rwlock.h
  common/device_tree: Add rwlock for dt_host
  xen/arm: Implement device tree node removal functionalities
  xen/arm: Implement device tree node addition functionalities
  tools/libs/ctrl: Implement new xc interfaces for dt overlay
  tools/libs/light: Implement new libxl functions for device tree
    overlay ops
  tools/xl: Add new xl command overlay for device tree overlay support

 CHANGELOG.md                            |   2 +
 SUPPORT.md                              |   6 +
 tools/include/libxl.h                   |  11 +
 tools/include/xenctrl.h                 |   5 +
 tools/libs/ctrl/Makefile.common         |   1 +
 tools/libs/ctrl/xc_dt_overlay.c         |  51 ++
 tools/libs/light/Makefile               |   3 +
 tools/libs/light/libxl_dt_overlay.c     |  71 ++
 tools/xl/xl.h                           |   1 +
 tools/xl/xl_cmdtable.c                  |   6 +
 tools/xl/xl_vmcontrol.c                 |  52 ++
 xen/arch/arm/Kconfig                    |   5 +
 xen/arch/arm/device.c                   | 149 ++++
 xen/arch/arm/domain_build.c             | 147 ----
 xen/arch/arm/include/asm/domain_build.h |   2 -
 xen/arch/arm/include/asm/setup.h        |   6 +
 xen/arch/arm/include/asm/smp.h          |   3 +-
 xen/arch/arm/smpboot.c                  |   1 +
 xen/arch/arm/sysctl.c                   |  16 +-
 xen/common/Makefile                     |   1 +
 xen/common/device_tree.c                |  52 +-
 xen/common/dt-overlay.c                 | 953 ++++++++++++++++++++++++
 xen/common/libfdt/Makefile              |   4 +
 xen/common/libfdt/fdt_overlay.c         |  29 +-
 xen/common/libfdt/version.lds           |   1 +
 xen/drivers/passthrough/arm/smmu.c      |  59 ++
 xen/drivers/passthrough/device_tree.c   |  85 ++-
 xen/include/public/sysctl.h             |  24 +
 xen/include/xen/device_tree.h           |  29 +-
 xen/include/xen/dt-overlay.h            |  59 ++
 xen/include/xen/iommu-private.h         |  27 +
 xen/include/xen/iommu.h                 |   2 +
 xen/include/xen/libfdt/libfdt.h         |  18 +
 33 files changed, 1687 insertions(+), 194 deletions(-)
 create mode 100644 tools/libs/ctrl/xc_dt_overlay.c
 create mode 100644 tools/libs/light/libxl_dt_overlay.c
 create mode 100644 xen/common/dt-overlay.c
 create mode 100644 xen/include/xen/dt-overlay.h
 create mode 100644 xen/include/xen/iommu-private.h

-- 
2.17.1

[XEN][PATCH v7 01/19] common/device_tree: handle memory allocation failure in __unflatten_device_tree()

[Vikram Garhwal]

Change __unflatten_device_tree() return type to integer so it can propagate
memory allocation failure. Add panic() in dt_unflatten_host_device_tree() for
memory allocation failure during boot.

Fixes: fb97eb614acf ("xen/arm: Create a hierarchical device tree")

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
Reviewed-by: Henry Wang <Henry.Wang@arm.com>
---
 xen/common/device_tree.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
index 8da1052911..dfdb10e488 100644
--- a/xen/common/device_tree.c
+++ b/xen/common/device_tree.c
@@ -2092,8 +2092,8 @@ static unsigned long __init unflatten_dt_node(const void *fdt,
  * @fdt: The fdt to expand
  * @mynodes: The device_node tree created by the call
  */
-static void __init __unflatten_device_tree(const void *fdt,
-                                           struct dt_device_node **mynodes)
+static int __init __unflatten_device_tree(const void *fdt,
+                                          struct dt_device_node **mynodes)
 {
     unsigned long start, mem, size;
     struct dt_device_node **allnextp = mynodes;
@@ -2114,6 +2114,8 @@ static void __init __unflatten_device_tree(const void *fdt,
 
     /* Allocate memory for the expanded device tree */
     mem = (unsigned long)_xmalloc (size + 4, __alignof__(struct dt_device_node));
+    if ( !mem )
+        return -ENOMEM;
 
     ((__be32 *)mem)[size / 4] = cpu_to_be32(0xdeadbeef);
 
@@ -2131,6 +2133,8 @@ static void __init __unflatten_device_tree(const void *fdt,
     *allnextp = NULL;
 
     dt_dprintk(" <- unflatten_device_tree()\n");
+
+    return 0;
 }
 
 static void dt_alias_add(struct dt_alias_prop *ap,
@@ -2215,7 +2219,11 @@ dt_find_interrupt_controller(const struct dt_device_match *matches)
 
 void __init dt_unflatten_host_device_tree(void)
 {
-    __unflatten_device_tree(device_tree_flattened, &dt_host);
+    int error = __unflatten_device_tree(device_tree_flattened, &dt_host);
+
+    if ( error )
+        panic("__unflatten_device_tree failed with error %d\n", error);
+
     dt_alias_scan();
 }
 
-- 
2.17.1

[XEN][PATCH v7 02/19] common/device_tree.c: unflatten_device_tree() propagate errors

[Vikram Garhwal]

This will be useful in dynamic node programming when new dt nodes are unflattend
during runtime. Invalid device tree node related errors should be propagated
back to the caller.

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
Reviewed-by: Henry Wang <Henry.Wang@arm.com>
---
 xen/common/device_tree.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
index dfdb10e488..117ccccb09 100644
--- a/xen/common/device_tree.c
+++ b/xen/common/device_tree.c
@@ -2108,6 +2108,9 @@ static int __init __unflatten_device_tree(const void *fdt,
     /* First pass, scan for size */
     start = ((unsigned long)fdt) + fdt_off_dt_struct(fdt);
     size = unflatten_dt_node(fdt, 0, &start, NULL, NULL, 0);
+    if ( !size )
+        return -EINVAL;
+
     size = (size | 3) + 1;
 
     dt_dprintk("  size is %#lx allocating...\n", size);
@@ -2125,11 +2128,19 @@ static int __init __unflatten_device_tree(const void *fdt,
     start = ((unsigned long)fdt) + fdt_off_dt_struct(fdt);
     unflatten_dt_node(fdt, mem, &start, NULL, &allnextp, 0);
     if ( be32_to_cpup((__be32 *)start) != FDT_END )
-        printk(XENLOG_WARNING "Weird tag at end of tree: %08x\n",
+    {
+        printk(XENLOG_ERR "Weird tag at end of tree: %08x\n",
                   *((u32 *)start));
+        return -EINVAL;
+    }
+
     if ( be32_to_cpu(((__be32 *)mem)[size / 4]) != 0xdeadbeef )
-        printk(XENLOG_WARNING "End of tree marker overwritten: %08x\n",
+    {
+        printk(XENLOG_ERR "End of tree marker overwritten: %08x\n",
                   be32_to_cpu(((__be32 *)mem)[size / 4]));
+        return -EINVAL;
+    }
+
     *allnextp = NULL;
 
     dt_dprintk(" <- unflatten_device_tree()\n");
-- 
2.17.1

[XEN][PATCH v7 03/19] xen/arm/device: Remove __init from function type

[Vikram Garhwal]

Remove __init from following function to access during runtime:
    1. map_irq_to_domain()
    2. handle_device_interrupts()
    3. map_range_to_domain()
    4. unflatten_dt_node()

Move map_irq_to_domain() prototype from domain_build.h to setup.h.

To avoid breaking the build, following changes are also done:
1. Move map_irq_to_domain(), handle_device_interrupts() and map_range_to_domain()
    to device.c. After removing __init type,  these functions are not specific
    to domain building, so moving them out of domain_build.c to device.c.
2. Remove static type from handle_device_interrupt().

Overall, these changes are done to support the dynamic programming of a nodes
where an overlay node will be added to fdt and unflattened node will be added to
dt_host. Furthermore, IRQ and mmio mapping will be done for the added node.

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>
---
 xen/arch/arm/device.c                   | 149 ++++++++++++++++++++++++
 xen/arch/arm/domain_build.c             | 147 -----------------------
 xen/arch/arm/include/asm/domain_build.h |   2 -
 xen/arch/arm/include/asm/setup.h        |   6 +
 xen/common/device_tree.c                |  12 +-
 5 files changed, 161 insertions(+), 155 deletions(-)

diff --git a/xen/arch/arm/device.c b/xen/arch/arm/device.c
index ca8539dee5..1652d765bd 100644
--- a/xen/arch/arm/device.c
+++ b/xen/arch/arm/device.c
@@ -9,8 +9,10 @@
  */
 
 #include <asm/device.h>
+#include <asm/setup.h>
 #include <xen/errno.h>
 #include <xen/init.h>
+#include <xen/iocap.h>
 #include <xen/lib.h>
 
 extern const struct device_desc _sdevice[], _edevice[];
@@ -75,6 +77,153 @@ enum device_class device_get_class(const struct dt_device_node *dev)
     return DEVICE_UNKNOWN;
 }
 
+int map_irq_to_domain(struct domain *d, unsigned int irq,
+                      bool need_mapping, const char *devname)
+{
+    int res;
+
+    res = irq_permit_access(d, irq);
+    if ( res )
+    {
+        printk(XENLOG_ERR "Unable to permit to %pd access to IRQ %u\n", d, irq);
+        return res;
+    }
+
+    if ( need_mapping )
+    {
+        /*
+         * Checking the return of vgic_reserve_virq is not
+         * necessary. It should not fail except when we try to map
+         * the IRQ twice. This can legitimately happen if the IRQ is shared
+         */
+        vgic_reserve_virq(d, irq);
+
+        res = route_irq_to_guest(d, irq, irq, devname);
+        if ( res < 0 )
+        {
+            printk(XENLOG_ERR "Unable to map IRQ%u to %pd\n", irq, d);
+            return res;
+        }
+    }
+
+    dt_dprintk("  - IRQ: %u\n", irq);
+    return 0;
+}
+
+int map_range_to_domain(const struct dt_device_node *dev,
+                        uint64_t addr, uint64_t len, void *data)
+{
+    struct map_range_data *mr_data = data;
+    struct domain *d = mr_data->d;
+    int res;
+
+    if ( (addr != (paddr_t)addr) || (((paddr_t)~0 - addr) < len) )
+    {
+        printk(XENLOG_ERR "%s: [0x%"PRIx64", 0x%"PRIx64"] exceeds the maximum allowed PA width (%u bits)",
+               dt_node_full_name(dev), addr, (addr + len), PADDR_BITS);
+        return -ERANGE;
+    }
+
+    /*
+     * reserved-memory regions are RAM carved out for a special purpose.
+     * They are not MMIO and therefore a domain should not be able to
+     * manage them via the IOMEM interface.
+     */
+    if ( strncasecmp(dt_node_full_name(dev), "/reserved-memory/",
+                     strlen("/reserved-memory/")) != 0 )
+    {
+        res = iomem_permit_access(d, paddr_to_pfn(addr),
+                paddr_to_pfn(PAGE_ALIGN(addr + len - 1)));
+        if ( res )
+        {
+            printk(XENLOG_ERR "Unable to permit to dom%d access to"
+                    " 0x%"PRIx64" - 0x%"PRIx64"\n",
+                    d->domain_id,
+                    addr & PAGE_MASK, PAGE_ALIGN(addr + len) - 1);
+            return res;
+        }
+    }
+
+    if ( !mr_data->skip_mapping )
+    {
+        res = map_regions_p2mt(d,
+                               gaddr_to_gfn(addr),
+                               PFN_UP(len),
+                               maddr_to_mfn(addr),
+                               mr_data->p2mt);
+
+        if ( res < 0 )
+        {
+            printk(XENLOG_ERR "Unable to map 0x%"PRIx64
+                   " - 0x%"PRIx64" in domain %d\n",
+                   addr & PAGE_MASK, PAGE_ALIGN(addr + len) - 1,
+                   d->domain_id);
+            return res;
+        }
+    }
+
+    dt_dprintk("  - MMIO: %010"PRIx64" - %010"PRIx64" P2MType=%x\n",
+               addr, addr + len, mr_data->p2mt);
+
+    return 0;
+}
+
+/*
+ * handle_device_interrupts retrieves the interrupts configuration from
+ * a device tree node and maps those interrupts to the target domain.
+ *
+ * Returns:
+ *   < 0 error
+ *   0   success
+ */
+int handle_device_interrupts(struct domain *d,
+                             struct dt_device_node *dev,
+                             bool need_mapping)
+{
+    unsigned int i, nirq;
+    int res;
+    struct dt_raw_irq rirq;
+
+    nirq = dt_number_of_irq(dev);
+
+    /* Give permission and map IRQs */
+    for ( i = 0; i < nirq; i++ )
+    {
+        res = dt_device_get_raw_irq(dev, i, &rirq);
+        if ( res )
+        {
+            printk(XENLOG_ERR "Unable to retrieve irq %u for %s\n",
+                   i, dt_node_full_name(dev));
+            return res;
+        }
+
+        /*
+         * Don't map IRQ that have no physical meaning
+         * ie: IRQ whose controller is not the GIC
+         */
+        if ( rirq.controller != dt_interrupt_controller )
+        {
+            dt_dprintk("irq %u not connected to primary controller. Connected to %s\n",
+                      i, dt_node_full_name(rirq.controller));
+            continue;
+        }
+
+        res = platform_get_irq(dev, i);
+        if ( res < 0 )
+        {
+            printk(XENLOG_ERR "Unable to get irq %u for %s\n",
+                   i, dt_node_full_name(dev));
+            return res;
+        }
+
+        res = map_irq_to_domain(d, res, need_mapping, dt_node_name(dev));
+        if ( res )
+            return res;
+    }
+
+    return 0;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 3f4558ade6..8bcd73dfcf 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -2271,39 +2271,6 @@ int __init make_chosen_node(const struct kernel_info *kinfo)
     return res;
 }
 
-int __init map_irq_to_domain(struct domain *d, unsigned int irq,
-                             bool need_mapping, const char *devname)
-{
-    int res;
-
-    res = irq_permit_access(d, irq);
-    if ( res )
-    {
-        printk(XENLOG_ERR "Unable to permit to %pd access to IRQ %u\n", d, irq);
-        return res;
-    }
-
-    if ( need_mapping )
-    {
-        /*
-         * Checking the return of vgic_reserve_virq is not
-         * necessary. It should not fail except when we try to map
-         * the IRQ twice. This can legitimately happen if the IRQ is shared
-         */
-        vgic_reserve_virq(d, irq);
-
-        res = route_irq_to_guest(d, irq, irq, devname);
-        if ( res < 0 )
-        {
-            printk(XENLOG_ERR "Unable to map IRQ%u to %pd\n", irq, d);
-            return res;
-        }
-    }
-
-    dt_dprintk("  - IRQ: %u\n", irq);
-    return 0;
-}
-
 static int __init map_dt_irq_to_domain(const struct dt_device_node *dev,
                                        const struct dt_irq *dt_irq,
                                        void *data)
@@ -2333,64 +2300,6 @@ static int __init map_dt_irq_to_domain(const struct dt_device_node *dev,
     return res;
 }
 
-int __init map_range_to_domain(const struct dt_device_node *dev,
-                               uint64_t addr, uint64_t len, void *data)
-{
-    struct map_range_data *mr_data = data;
-    struct domain *d = mr_data->d;
-    int res;
-
-    if ( (addr != (paddr_t)addr) || (((paddr_t)~0 - addr) < len) )
-    {
-        printk(XENLOG_ERR "%s: [0x%"PRIx64", 0x%"PRIx64"] exceeds the maximum allowed PA width (%u bits)",
-               dt_node_full_name(dev), addr, (addr + len), PADDR_BITS);
-        return -ERANGE;
-    }
-
-    /*
-     * reserved-memory regions are RAM carved out for a special purpose.
-     * They are not MMIO and therefore a domain should not be able to
-     * manage them via the IOMEM interface.
-     */
-    if ( strncasecmp(dt_node_full_name(dev), "/reserved-memory/",
-                     strlen("/reserved-memory/")) != 0 )
-    {
-        res = iomem_permit_access(d, paddr_to_pfn(addr),
-                paddr_to_pfn(PAGE_ALIGN(addr + len - 1)));
-        if ( res )
-        {
-            printk(XENLOG_ERR "Unable to permit to dom%d access to"
-                    " 0x%"PRIx64" - 0x%"PRIx64"\n",
-                    d->domain_id,
-                    addr & PAGE_MASK, PAGE_ALIGN(addr + len) - 1);
-            return res;
-        }
-    }
-
-    if ( !mr_data->skip_mapping )
-    {
-        res = map_regions_p2mt(d,
-                               gaddr_to_gfn(addr),
-                               PFN_UP(len),
-                               maddr_to_mfn(addr),
-                               mr_data->p2mt);
-
-        if ( res < 0 )
-        {
-            printk(XENLOG_ERR "Unable to map 0x%"PRIx64
-                   " - 0x%"PRIx64" in domain %d\n",
-                   addr & PAGE_MASK, PAGE_ALIGN(addr + len) - 1,
-                   d->domain_id);
-            return res;
-        }
-    }
-
-    dt_dprintk("  - MMIO: %010"PRIx64" - %010"PRIx64" P2MType=%x\n",
-               addr, addr + len, mr_data->p2mt);
-
-    return 0;
-}
-
 /*
  * For a node which describes a discoverable bus (such as a PCI bus)
  * then we may need to perform additional mappings in order to make
@@ -2418,62 +2327,6 @@ static int __init map_device_children(const struct dt_device_node *dev,
     return 0;
 }
 
-/*
- * handle_device_interrupts retrieves the interrupts configuration from
- * a device tree node and maps those interrupts to the target domain.
- *
- * Returns:
- *   < 0 error
- *   0   success
- */
-static int __init handle_device_interrupts(struct domain *d,
-                                           struct dt_device_node *dev,
-                                           bool need_mapping)
-{
-    unsigned int i, nirq;
-    int res;
-    struct dt_raw_irq rirq;
-
-    nirq = dt_number_of_irq(dev);
-
-    /* Give permission and map IRQs */
-    for ( i = 0; i < nirq; i++ )
-    {
-        res = dt_device_get_raw_irq(dev, i, &rirq);
-        if ( res )
-        {
-            printk(XENLOG_ERR "Unable to retrieve irq %u for %s\n",
-                   i, dt_node_full_name(dev));
-            return res;
-        }
-
-        /*
-         * Don't map IRQ that have no physical meaning
-         * ie: IRQ whose controller is not the GIC
-         */
-        if ( rirq.controller != dt_interrupt_controller )
-        {
-            dt_dprintk("irq %u not connected to primary controller. Connected to %s\n",
-                      i, dt_node_full_name(rirq.controller));
-            continue;
-        }
-
-        res = platform_get_irq(dev, i);
-        if ( res < 0 )
-        {
-            printk(XENLOG_ERR "Unable to get irq %u for %s\n",
-                   i, dt_node_full_name(dev));
-            return res;
-        }
-
-        res = map_irq_to_domain(d, res, need_mapping, dt_node_name(dev));
-        if ( res )
-            return res;
-    }
-
-    return 0;
-}
-
 /*
  * For a given device node:
  *  - Give permission to the guest to manage IRQ and MMIO range
diff --git a/xen/arch/arm/include/asm/domain_build.h b/xen/arch/arm/include/asm/domain_build.h
index 34ceddc995..b9329c9ee0 100644
--- a/xen/arch/arm/include/asm/domain_build.h
+++ b/xen/arch/arm/include/asm/domain_build.h
@@ -4,8 +4,6 @@
 #include <xen/sched.h>
 #include <asm/kernel.h>
 
-int map_irq_to_domain(struct domain *d, unsigned int irq,
-                      bool need_mapping, const char *devname);
 int make_chosen_node(const struct kernel_info *kinfo);
 void evtchn_allocate(struct domain *d);
 
diff --git a/xen/arch/arm/include/asm/setup.h b/xen/arch/arm/include/asm/setup.h
index 19dc637d55..f532332d6c 100644
--- a/xen/arch/arm/include/asm/setup.h
+++ b/xen/arch/arm/include/asm/setup.h
@@ -165,9 +165,15 @@ void device_tree_get_reg(const __be32 **cell, uint32_t address_cells,
 u32 device_tree_get_u32(const void *fdt, int node,
                         const char *prop_name, u32 dflt);
 
+int handle_device_interrupts(struct domain *d, struct dt_device_node *dev,
+                             bool need_mapping);
+
 int map_range_to_domain(const struct dt_device_node *dev,
                         uint64_t addr, uint64_t len, void *data);
 
+int map_irq_to_domain(struct domain *d, unsigned int irq,
+                      bool need_mapping, const char *devname);
+
 extern lpae_t boot_pgtable[XEN_PT_LPAE_ENTRIES];
 
 #ifdef CONFIG_ARM_64
diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
index 117ccccb09..bbdab07596 100644
--- a/xen/common/device_tree.c
+++ b/xen/common/device_tree.c
@@ -1847,12 +1847,12 @@ int dt_count_phandle_with_args(const struct dt_device_node *np,
  * @allnextpp: pointer to ->allnext from last allocated device_node
  * @fpsize: Size of the node path up at the current depth.
  */
-static unsigned long __init unflatten_dt_node(const void *fdt,
-                                              unsigned long mem,
-                                              unsigned long *p,
-                                              struct dt_device_node *dad,
-                                              struct dt_device_node ***allnextpp,
-                                              unsigned long fpsize)
+static unsigned long unflatten_dt_node(const void *fdt,
+                                       unsigned long mem,
+                                       unsigned long *p,
+                                       struct dt_device_node *dad,
+                                       struct dt_device_node ***allnextpp,
+                                       unsigned long fpsize)
 {
     struct dt_device_node *np;
     struct dt_property *pp, **prev_pp = NULL;
-- 
2.17.1

[XEN][PATCH v7 04/19] common/device_tree: change __unflatten_device_tree() type

[Vikram Garhwal]

Following changes are done to __unflatten_device_tree():
    1. __unflatten_device_tree() is renamed to unflatten_device_tree().
    2. Remove __init and static function type.

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
Reviewed-by: Henry Wang <Henry.Wang@arm.com>
---
 xen/common/device_tree.c      | 9 ++++-----
 xen/include/xen/device_tree.h | 5 +++++
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
index bbdab07596..16b4b4e946 100644
--- a/xen/common/device_tree.c
+++ b/xen/common/device_tree.c
@@ -2083,7 +2083,7 @@ static unsigned long unflatten_dt_node(const void *fdt,
 }
 
 /**
- * __unflatten_device_tree - create tree of device_nodes from flat blob
+ * unflatten_device_tree - create tree of device_nodes from flat blob
  *
  * unflattens a device-tree, creating the
  * tree of struct device_node. It also fills the "name" and "type"
@@ -2092,8 +2092,7 @@ static unsigned long unflatten_dt_node(const void *fdt,
  * @fdt: The fdt to expand
  * @mynodes: The device_node tree created by the call
  */
-static int __init __unflatten_device_tree(const void *fdt,
-                                          struct dt_device_node **mynodes)
+int unflatten_device_tree(const void *fdt, struct dt_device_node **mynodes)
 {
     unsigned long start, mem, size;
     struct dt_device_node **allnextp = mynodes;
@@ -2230,10 +2229,10 @@ dt_find_interrupt_controller(const struct dt_device_match *matches)
 
 void __init dt_unflatten_host_device_tree(void)
 {
-    int error = __unflatten_device_tree(device_tree_flattened, &dt_host);
+    int error = unflatten_device_tree(device_tree_flattened, &dt_host);
 
     if ( error )
-        panic("__unflatten_device_tree failed with error %d\n", error);
+        panic("unflatten_device_tree failed with error %d\n", error);
 
     dt_alias_scan();
 }
diff --git a/xen/include/xen/device_tree.h b/xen/include/xen/device_tree.h
index c2eada7489..2c35c0d391 100644
--- a/xen/include/xen/device_tree.h
+++ b/xen/include/xen/device_tree.h
@@ -178,6 +178,11 @@ int device_tree_for_each_node(const void *fdt, int node,
  */
 void dt_unflatten_host_device_tree(void);
 
+/**
+ * unflatten any device tree.
+ */
+int unflatten_device_tree(const void *fdt, struct dt_device_node **mynodes);
+
 /**
  * IRQ translation callback
  * TODO: For the moment we assume that we only have ONE
-- 
2.17.1

[XEN][PATCH v7 05/19] xen/arm: Add CONFIG_OVERLAY_DTB

[Vikram Garhwal]

Introduce a config option where the user can enable support for adding/removing
device tree nodes using a device tree binary overlay.

Update SUPPORT.md and CHANGELOG.md to state the Device Tree Overlays support for
Arm.

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>

---
Changes from v6:
    Add CHANGELOG and SUPPORT.md entries.
---
 CHANGELOG.md         | 2 ++
 SUPPORT.md           | 6 ++++++
 xen/arch/arm/Kconfig | 5 +++++
 3 files changed, 13 insertions(+)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5bfd3aa5c0..a137fce576 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -20,6 +20,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
    - Bus-lock detection, used by Xen to mitigate (by rate-limiting) the system
      wide impact of a guest misusing atomic instructions.
  - xl/libxl can customize SMBIOS strings for HVM guests.
+ - On Arm, support for dynamic addition/removal of Xen device tree nodes using
+   a device tree overlay binary(.dtbo).
 
 ## [4.17.0](https://xenbits.xen.org/gitweb/?p=xen.git;a=shortlog;h=RELEASE-4.17.0) - 2022-12-12
 
diff --git a/SUPPORT.md b/SUPPORT.md
index 6dbed9d5d0..6b27d43cc6 100644
--- a/SUPPORT.md
+++ b/SUPPORT.md
@@ -832,6 +832,12 @@ No support for QEMU backends in a 16K or 64K domain.
 
     Status: Supported
 
+### Device Tree Overlays
+
+Add/Remove device tree nodes using a device tree overlay binary(.dtbo).
+
+    Status, ARM: Experimental
+
 ### ARM: Guest ACPI support
 
     Status: Supported
diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index 239d3aed3c..1fe3d698a5 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -53,6 +53,11 @@ config HAS_ITS
         bool "GICv3 ITS MSI controller support (UNSUPPORTED)" if UNSUPPORTED
         depends on GICV3 && !NEW_VGIC && !ARM_32
 
+config OVERLAY_DTB
+	bool "DTB overlay support (UNSUPPORTED)" if UNSUPPORTED
+	help
+	  Dynamic addition/removal of Xen device tree nodes using a dtbo.
+
 config HVM
         def_bool y
 
-- 
2.17.1

[XEN][PATCH v7 06/19] libfdt: Keep fdt functions after init for CONFIG_OVERLAY_DTB.

[Vikram Garhwal]

This is done to access fdt library function which are required for adding device
tree overlay nodes for dynamic programming of nodes.

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
Acked-by: Julien Grall <jgrall@amazon.com>
---
 xen/common/libfdt/Makefile | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/xen/common/libfdt/Makefile b/xen/common/libfdt/Makefile
index 75aaefa2e3..d50487aa6e 100644
--- a/xen/common/libfdt/Makefile
+++ b/xen/common/libfdt/Makefile
@@ -1,7 +1,11 @@
 include $(src)/Makefile.libfdt
 
 SECTIONS := text data $(SPECIAL_DATA_SECTIONS)
+
+# For CONFIG_OVERLAY_DTB, libfdt functionalities will be needed during runtime.
+ifneq ($(CONFIG_OVERLAY_DTB),y)
 OBJCOPYFLAGS := $(foreach s,$(SECTIONS),--rename-section .$(s)=.init.$(s))
+endif
 
 obj-y += libfdt.o
 nocov-y += libfdt.o
-- 
2.17.1

[XEN][PATCH v7 07/19] libfdt: overlay: change overlay_get_target()

[Vikram Garhwal]

Rename overlay_get_target() to fdt_overlay_target_offset() and remove static
function type.

This is done to get the target path for the overlay nodes which is very useful
in many cases. For example, Xen hypervisor needs it when applying overlays
because Xen needs to do further processing of the overlay nodes, e.g. mapping of
resources(IRQs and IOMMUs) to other VMs, creation of SMMU pagetables, etc.

Signed-off-by: Vikram Garhwal <fnu.vikram@xilinx.com>
Message-Id: <1637204036-382159-2-git-send-email-fnu.vikram@xilinx.com>
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
Origin: git://git.kernel.org/pub/scm/utils/dtc/dtc.git 45f3d1a095dd

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>
Reviewed-by: Henry Wang <Henry.Wang@arm.com>
---
 xen/common/libfdt/fdt_overlay.c | 29 +++++++----------------------
 xen/common/libfdt/version.lds   |  1 +
 xen/include/xen/libfdt/libfdt.h | 18 ++++++++++++++++++
 3 files changed, 26 insertions(+), 22 deletions(-)

diff --git a/xen/common/libfdt/fdt_overlay.c b/xen/common/libfdt/fdt_overlay.c
index 7b95e2b639..acf0c4c2a6 100644
--- a/xen/common/libfdt/fdt_overlay.c
+++ b/xen/common/libfdt/fdt_overlay.c
@@ -41,37 +41,22 @@ static uint32_t overlay_get_target_phandle(const void *fdto, int fragment)
 	return fdt32_to_cpu(*val);
 }
 
-/**
- * overlay_get_target - retrieves the offset of a fragment's target
- * @fdt: Base device tree blob
- * @fdto: Device tree overlay blob
- * @fragment: node offset of the fragment in the overlay
- * @pathp: pointer which receives the path of the target (or NULL)
- *
- * overlay_get_target() retrieves the target offset in the base
- * device tree of a fragment, no matter how the actual targeting is
- * done (through a phandle or a path)
- *
- * returns:
- *      the targeted node offset in the base device tree
- *      Negative error code on error
- */
-static int overlay_get_target(const void *fdt, const void *fdto,
-			      int fragment, char const **pathp)
+int fdt_overlay_target_offset(const void *fdt, const void *fdto,
+			      int fragment_offset, char const **pathp)
 {
 	uint32_t phandle;
 	const char *path = NULL;
 	int path_len = 0, ret;
 
 	/* Try first to do a phandle based lookup */
-	phandle = overlay_get_target_phandle(fdto, fragment);
+	phandle = overlay_get_target_phandle(fdto, fragment_offset);
 	if (phandle == (uint32_t)-1)
 		return -FDT_ERR_BADPHANDLE;
 
 	/* no phandle, try path */
 	if (!phandle) {
 		/* And then a path based lookup */
-		path = fdt_getprop(fdto, fragment, "target-path", &path_len);
+		path = fdt_getprop(fdto, fragment_offset, "target-path", &path_len);
 		if (path)
 			ret = fdt_path_offset(fdt, path);
 		else
@@ -638,7 +623,7 @@ static int overlay_merge(void *fdt, void *fdto)
 		if (overlay < 0)
 			return overlay;
 
-		target = overlay_get_target(fdt, fdto, fragment, NULL);
+		target = fdt_overlay_target_offset(fdt, fdto, fragment, NULL);
 		if (target < 0)
 			return target;
 
@@ -781,7 +766,7 @@ static int overlay_symbol_update(void *fdt, void *fdto)
 			return -FDT_ERR_BADOVERLAY;
 
 		/* get the target of the fragment */
-		ret = overlay_get_target(fdt, fdto, fragment, &target_path);
+		ret = fdt_overlay_target_offset(fdt, fdto, fragment, &target_path);
 		if (ret < 0)
 			return ret;
 		target = ret;
@@ -803,7 +788,7 @@ static int overlay_symbol_update(void *fdt, void *fdto)
 
 		if (!target_path) {
 			/* again in case setprop_placeholder changed it */
-			ret = overlay_get_target(fdt, fdto, fragment, &target_path);
+			ret = fdt_overlay_target_offset(fdt, fdto, fragment, &target_path);
 			if (ret < 0)
 				return ret;
 			target = ret;
diff --git a/xen/common/libfdt/version.lds b/xen/common/libfdt/version.lds
index 7ab85f1d9d..cbce5d4a8b 100644
--- a/xen/common/libfdt/version.lds
+++ b/xen/common/libfdt/version.lds
@@ -77,6 +77,7 @@ LIBFDT_1.2 {
 		fdt_appendprop_addrrange;
 		fdt_setprop_inplace_namelen_partial;
 		fdt_create_with_flags;
+		fdt_overlay_target_offset;
 	local:
 		*;
 };
diff --git a/xen/include/xen/libfdt/libfdt.h b/xen/include/xen/libfdt/libfdt.h
index c71689e2be..fabddbee8c 100644
--- a/xen/include/xen/libfdt/libfdt.h
+++ b/xen/include/xen/libfdt/libfdt.h
@@ -2109,6 +2109,24 @@ int fdt_del_node(void *fdt, int nodeoffset);
  */
 int fdt_overlay_apply(void *fdt, void *fdto);
 
+/**
+ * fdt_overlay_target_offset - retrieves the offset of a fragment's target
+ * @fdt: Base device tree blob
+ * @fdto: Device tree overlay blob
+ * @fragment_offset: node offset of the fragment in the overlay
+ * @pathp: pointer which receives the path of the target (or NULL)
+ *
+ * fdt_overlay_target_offset() retrieves the target offset in the base
+ * device tree of a fragment, no matter how the actual targeting is
+ * done (through a phandle or a path)
+ *
+ * returns:
+ *      the targeted node offset in the base device tree
+ *      Negative error code on error
+ */
+int fdt_overlay_target_offset(const void *fdt, const void *fdto,
+			      int fragment_offset, char const **pathp);
+
 /**********************************************************************/
 /* Debugging / informational functions                                */
 /**********************************************************************/
-- 
2.17.1

[XEN][PATCH v7 08/19] xen/device-tree: Add device_tree_find_node_by_path() to find nodes in device tree

[Vikram Garhwal]

Add device_tree_find_node_by_path() to find a matching node with path for a
dt_device_node.

Reason behind this function:
    Each time overlay nodes are added using .dtbo, a new fdt(memcpy of
    device_tree_flattened) is created and updated with overlay nodes. This
    updated fdt is further unflattened to a dt_host_new. Next, we need to find
    the overlay nodes in dt_host_new, find the overlay node's parent in dt_host
    and add the nodes as child under their parent in the dt_host. Thus we need
    this function to search for node in different unflattened device trees.

Also, make dt_find_node_by_path() static inline.

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>

---
Changes from v6:
    Rename "dt_node" to "from"
---
 xen/common/device_tree.c      |  6 ++++--
 xen/include/xen/device_tree.h | 18 ++++++++++++++++--
 2 files changed, 20 insertions(+), 4 deletions(-)

diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
index 16b4b4e946..c5250a1644 100644
--- a/xen/common/device_tree.c
+++ b/xen/common/device_tree.c
@@ -358,11 +358,13 @@ struct dt_device_node *dt_find_node_by_type(struct dt_device_node *from,
     return np;
 }
 
-struct dt_device_node *dt_find_node_by_path(const char *path)
+struct dt_device_node *
+                    device_tree_find_node_by_path(struct dt_device_node *from,
+                                                  const char *path)
 {
     struct dt_device_node *np;
 
-    dt_for_each_device_node(dt_host, np)
+    dt_for_each_device_node(from, np)
         if ( np->full_name && (dt_node_cmp(np->full_name, path) == 0) )
             break;
 
diff --git a/xen/include/xen/device_tree.h b/xen/include/xen/device_tree.h
index 2c35c0d391..e239f7de26 100644
--- a/xen/include/xen/device_tree.h
+++ b/xen/include/xen/device_tree.h
@@ -561,13 +561,27 @@ struct dt_device_node *dt_find_node_by_type(struct dt_device_node *from,
 struct dt_device_node *dt_find_node_by_alias(const char *alias);
 
 /**
- * dt_find_node_by_path - Find a node matching a full DT path
+ * device_tree_find_node_by_path - Generic function to find a node matching the
+ * full DT path for any given unflatten device tree
+ * @from: The device tree node to start searching from
  * @path: The full path to match
  *
  * Returns a node pointer.
  */
-struct dt_device_node *dt_find_node_by_path(const char *path);
+struct dt_device_node *
+                    device_tree_find_node_by_path(struct dt_device_node *from,
+                                                  const char *path);
 
+/**
+ * dt_find_node_by_path - Find a node matching a full DT path in dt_host
+ * @path: The full path to match
+ *
+ * Returns a node pointer.
+ */
+static inline struct dt_device_node *dt_find_node_by_path(const char *path)
+{
+    return device_tree_find_node_by_path(dt_host, path);
+}
 
 /**
  * dt_find_node_by_gpath - Same as dt_find_node_by_path but retrieve the
-- 
2.17.1

[XEN][PATCH v7 09/19] xen/iommu: Move spin_lock from iommu_dt_device_is_assigned to caller

[Vikram Garhwal]

Rename iommu_dt_device_is_assigned() to iommu_dt_device_is_assigned_locked().
Remove static type so this can also be used by SMMU drivers to check if the
device is being used before removing.

Moving spin_lock to caller was done to prevent the concurrent access to
iommu_dt_device_is_assigned while doing add/remove/assign/deassign.

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>

---
Changes from v6:
    Created a private header and moved iommu_dt_device_is_assigned() to header.
---
 xen/drivers/passthrough/device_tree.c | 20 ++++++++++++++++----
 xen/include/xen/iommu-private.h       | 27 +++++++++++++++++++++++++++
 2 files changed, 43 insertions(+), 4 deletions(-)
 create mode 100644 xen/include/xen/iommu-private.h

diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
index 1c32d7b50c..52e370db01 100644
--- a/xen/drivers/passthrough/device_tree.c
+++ b/xen/drivers/passthrough/device_tree.c
@@ -18,6 +18,7 @@
 #include <xen/device_tree.h>
 #include <xen/guest_access.h>
 #include <xen/iommu.h>
+#include <xen/iommu-private.h>
 #include <xen/lib.h>
 #include <xen/sched.h>
 #include <xsm/xsm.h>
@@ -83,16 +84,14 @@ fail:
     return rc;
 }
 
-static bool_t iommu_dt_device_is_assigned(const struct dt_device_node *dev)
+bool_t iommu_dt_device_is_assigned_locked(const struct dt_device_node *dev)
 {
     bool_t assigned = 0;
 
     if ( !dt_device_is_protected(dev) )
         return 0;
 
-    spin_lock(&dtdevs_lock);
     assigned = !list_empty(&dev->domain_list);
-    spin_unlock(&dtdevs_lock);
 
     return assigned;
 }
@@ -213,27 +212,40 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
         if ( (d && d->is_dying) || domctl->u.assign_device.flags )
             break;
 
+        spin_lock(&dtdevs_lock);
+
         ret = dt_find_node_by_gpath(domctl->u.assign_device.u.dt.path,
                                     domctl->u.assign_device.u.dt.size,
                                     &dev);
         if ( ret )
+        {
+            spin_unlock(&dtdevs_lock);
             break;
+        }
 
         ret = xsm_assign_dtdevice(XSM_HOOK, d, dt_node_full_name(dev));
         if ( ret )
+        {
+            spin_unlock(&dtdevs_lock);
             break;
+        }
 
         if ( domctl->cmd == XEN_DOMCTL_test_assign_device )
         {
-            if ( iommu_dt_device_is_assigned(dev) )
+
+            if ( iommu_dt_device_is_assigned_locked(dev) )
             {
                 printk(XENLOG_G_ERR "%s already assigned.\n",
                        dt_node_full_name(dev));
                 ret = -EINVAL;
             }
+
+            spin_unlock(&dtdevs_lock);
             break;
         }
 
+        spin_unlock(&dtdevs_lock);
+
         if ( d == dom_io )
             return -EINVAL;
 
diff --git a/xen/include/xen/iommu-private.h b/xen/include/xen/iommu-private.h
new file mode 100644
index 0000000000..5615decaff
--- /dev/null
+++ b/xen/include/xen/iommu-private.h
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+ /*
+ * xen/iommu-private.h
+ *
+ *
+ * Copyright (C) 2023, Advanced Micro Devices, Inc. All Rights Reserved.
+ * Written by Vikram Garhwal <vikram.garhwal@amd.com>
+ *
+ */
+#ifndef __XEN_IOMMU_PRIVATE_H__
+#define __XEN_IOMMU_PRIVATE_H__
+
+#ifdef CONFIG_HAS_DEVICE_TREE
+#include <xen/device_tree.h>
+bool_t iommu_dt_device_is_assigned_locked(const struct dt_device_node *dev);
+#endif
+
+#endif /* __XEN_IOMMU_PRIVATE_H__ */
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
2.17.1

[XEN][PATCH v7 10/19] xen/iommu: protect iommu_add_dt_device() with dtdevs_lock

[Vikram Garhwal]

Protect iommu_add_dt_device() with dtdevs_lock to prevent concurrent access add.

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
Reviewed-by: Luca Fancellu <luca.fancellu@arm.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>
---
 xen/drivers/passthrough/device_tree.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
index 52e370db01..8cc413f867 100644
--- a/xen/drivers/passthrough/device_tree.c
+++ b/xen/drivers/passthrough/device_tree.c
@@ -146,6 +146,8 @@ int iommu_add_dt_device(struct dt_device_node *np)
     if ( dev_iommu_fwspec_get(dev) )
         return 0;
 
+    spin_lock(&dtdevs_lock);
+
     /*
      * According to the Documentation/devicetree/bindings/iommu/iommu.txt
      * from Linux.
@@ -158,7 +160,10 @@ int iommu_add_dt_device(struct dt_device_node *np)
          * these callback implemented.
          */
         if ( !ops->add_device || !ops->dt_xlate )
-            return -EINVAL;
+        {
+            rc = -EINVAL;
+            goto fail;
+        }
 
         if ( !dt_device_is_available(iommu_spec.np) )
             break;
@@ -189,6 +194,8 @@ int iommu_add_dt_device(struct dt_device_node *np)
     if ( rc < 0 )
         iommu_fwspec_free(dev);
 
+fail:
+    spin_unlock(&dtdevs_lock);
     return rc;
 }
 
-- 
2.17.1

[XEN][PATCH v7 11/19] xen/iommu: Introduce iommu_remove_dt_device()

[Vikram Garhwal]

Remove master device from the IOMMU. This will be helpful when removing the
overlay nodes using dynamic programming during run time.

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
---
 xen/drivers/passthrough/device_tree.c | 41 +++++++++++++++++++++++++++
 xen/include/xen/iommu.h               |  2 ++
 2 files changed, 43 insertions(+)

diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
index 8cc413f867..301a5bcd97 100644
--- a/xen/drivers/passthrough/device_tree.c
+++ b/xen/drivers/passthrough/device_tree.c
@@ -126,6 +126,47 @@ int iommu_release_dt_devices(struct domain *d)
     return 0;
 }
 
+int iommu_remove_dt_device(struct dt_device_node *np)
+{
+    const struct iommu_ops *ops = iommu_get_ops();
+    struct device *dev = dt_to_dev(np);
+    int rc;
+
+    if ( !ops )
+        return -EOPNOTSUPP;
+
+    spin_lock(&dtdevs_lock);
+
+    if ( iommu_dt_device_is_assigned_locked(np) )
+    {
+        rc = -EBUSY;
+        goto fail;
+    }
+
+    /*
+     * The driver which supports generic IOMMU DT bindings must have this
+     * callback implemented.
+     */
+    if ( !ops->remove_device )
+    {
+        rc = -EOPNOTSUPP;
+        goto fail;
+    }
+
+    /*
+     * Remove master device from the IOMMU if latter is present and available.
+     * The driver is responsible for removing is_protected flag.
+     */
+    rc = ops->remove_device(0, dev);
+
+    if ( !rc )
+        iommu_fwspec_free(dev);
+
+fail:
+    spin_unlock(&dtdevs_lock);
+    return rc;
+}
+
 int iommu_add_dt_device(struct dt_device_node *np)
 {
     const struct iommu_ops *ops = iommu_get_ops();
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index 405db59971..0d7924821b 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -229,6 +229,8 @@ int iommu_release_dt_devices(struct domain *d);
  */
 int iommu_add_dt_device(struct dt_device_node *np);
 
+int iommu_remove_dt_device(struct dt_device_node *np);
+
 int iommu_do_dt_domctl(struct xen_domctl *, struct domain *,
                        XEN_GUEST_HANDLE_PARAM(xen_domctl_t));
 
-- 
2.17.1

[XEN][PATCH v7 12/19] xen/smmu: Add remove_device callback for smmu_iommu ops

[Vikram Garhwal]

Add remove_device callback for removing the device entry from smmu-master using
following steps:
1. Find if SMMU master exists for the device node.
2. Check if device is currently in use.
3. Remove the SMMU master.

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
Reviewed-by: Luca Fancellu <luca.fancellu@arm.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>
---
 xen/drivers/passthrough/arm/smmu.c | 59 ++++++++++++++++++++++++++++++
 1 file changed, 59 insertions(+)

diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
index c37fa9af13..fdef6e7a7d 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -44,6 +44,7 @@
 #include <xen/irq.h>
 #include <xen/lib.h>
 #include <xen/list.h>
+#include <xen/iommu-private.h>
 #include <xen/mm.h>
 #include <xen/vmap.h>
 #include <xen/rbtree.h>
@@ -815,6 +816,19 @@ static int insert_smmu_master(struct arm_smmu_device *smmu,
 	return 0;
 }
 
+static int remove_smmu_master(struct arm_smmu_device *smmu,
+			      struct arm_smmu_master *master)
+{
+	if (!smmu->masters.rb_node) {
+		ASSERT_UNREACHABLE();
+		return -ENOENT;
+	}
+
+	rb_erase(&master->node, &smmu->masters);
+
+	return 0;
+}
+
 static int arm_smmu_dt_add_device_legacy(struct arm_smmu_device *smmu,
 					 struct device *dev,
 					 struct iommu_fwspec *fwspec)
@@ -852,6 +866,34 @@ static int arm_smmu_dt_add_device_legacy(struct arm_smmu_device *smmu,
 	return insert_smmu_master(smmu, master);
 }
 
+static int arm_smmu_dt_remove_device_legacy(struct arm_smmu_device *smmu,
+					 struct device *dev)
+{
+	struct arm_smmu_master *master;
+	struct device_node *dev_node = dev_get_dev_node(dev);
+	int ret;
+
+	master = find_smmu_master(smmu, dev_node);
+	if (master == NULL) {
+		dev_err(dev,
+			"No registrations found for master device %s\n",
+			dev_node->name);
+		return -EINVAL;
+	}
+
+	if (iommu_dt_device_is_assigned_locked(dev_to_dt(dev)))
+		return -EBUSY;
+
+	ret = remove_smmu_master(smmu, master);
+	if (ret)
+		return ret;
+
+	dev_node->is_protected = false;
+
+	kfree(master);
+	return 0;
+}
+
 static int register_smmu_master(struct arm_smmu_device *smmu,
 				struct device *dev,
 				struct of_phandle_args *masterspec)
@@ -875,6 +917,22 @@ static int register_smmu_master(struct arm_smmu_device *smmu,
 					     fwspec);
 }
 
+static int arm_smmu_dt_remove_device_generic(u8 devfn, struct device *dev)
+{
+	struct arm_smmu_device *smmu;
+	struct iommu_fwspec *fwspec;
+
+	fwspec = dev_iommu_fwspec_get(dev);
+	if (fwspec == NULL)
+		return -ENXIO;
+
+	smmu = find_smmu(fwspec->iommu_dev);
+	if (smmu == NULL)
+		return -ENXIO;
+
+	return arm_smmu_dt_remove_device_legacy(smmu, dev);
+}
+
 static int arm_smmu_dt_add_device_generic(u8 devfn, struct device *dev)
 {
 	struct arm_smmu_device *smmu;
@@ -2859,6 +2917,7 @@ static const struct iommu_ops arm_smmu_iommu_ops = {
     .init = arm_smmu_iommu_domain_init,
     .hwdom_init = arch_iommu_hwdom_init,
     .add_device = arm_smmu_dt_add_device_generic,
+    .remove_device = arm_smmu_dt_remove_device_generic,
     .teardown = arm_smmu_iommu_domain_teardown,
     .iotlb_flush = arm_smmu_iotlb_flush,
     .assign_device = arm_smmu_assign_dev,
-- 
2.17.1

[XEN][PATCH v7 13/19] asm/smp.h: Fix circular dependency for device_tree.h and rwlock.h

[Vikram Garhwal]

Dynamic programming ops will modify the dt_host and there might be other
function which are browsing the dt_host at the same time. To avoid the race
conditions, adding rwlock for browsing the dt_host. But adding rwlock in
device_tree.h causes following circular dependency:
    device_tree.h->rwlock.h->smp.h->asm/smp.h->device_tree.h

To fix this, removed the "#include <xen/device_tree.h> and forward declared
"struct dt_device_node".

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
Reviewed-by: Henry Wang <Henry.Wang@arm.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>
---
 xen/arch/arm/include/asm/smp.h | 3 ++-
 xen/arch/arm/smpboot.c         | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/include/asm/smp.h b/xen/arch/arm/include/asm/smp.h
index a37ca55bff..b12949ba8a 100644
--- a/xen/arch/arm/include/asm/smp.h
+++ b/xen/arch/arm/include/asm/smp.h
@@ -3,13 +3,14 @@
 
 #ifndef __ASSEMBLY__
 #include <xen/cpumask.h>
-#include <xen/device_tree.h>
 #include <asm/current.h>
 #endif
 
 DECLARE_PER_CPU(cpumask_var_t, cpu_sibling_mask);
 DECLARE_PER_CPU(cpumask_var_t, cpu_core_mask);
 
+struct dt_device_node;
+
 #define cpu_is_offline(cpu) unlikely(!cpu_online(cpu))
 
 #define smp_processor_id() get_processor_id()
diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index e107b86b7b..eeb76cd551 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -10,6 +10,7 @@
 #include <xen/cpu.h>
 #include <xen/cpumask.h>
 #include <xen/delay.h>
+#include <xen/device_tree.h>
 #include <xen/domain_page.h>
 #include <xen/errno.h>
 #include <xen/init.h>
-- 
2.17.1

[XEN][PATCH v7 14/19] common/device_tree: Add rwlock for dt_host

[Vikram Garhwal]

 Dynamic programming ops will modify the dt_host and there might be other
 function which are browsing the dt_host at the same time. To avoid the race
 conditions, adding rwlock for browsing the dt_host during runtime.

 Reason behind adding rwlock instead of spinlock:
    For now, dynamic programming is the sole modifier of dt_host in Xen during
        run time. All other access functions like iommu_release_dt_device() are
        just reading the dt_host during run-time. So, there is a need to protect
        others from browsing the dt_host while dynamic programming is modifying
        it. rwlock is better suitable for this task as spinlock won't be able to
        differentiate between read and write access.

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>

---
Changes from v6:
    Remove redundant "read_unlock(&dt_host->lock);" in the following case:
         XEN_DOMCTL_deassign_device
---
 xen/common/device_tree.c              |  4 ++++
 xen/drivers/passthrough/device_tree.c | 15 +++++++++++++++
 xen/include/xen/device_tree.h         |  6 ++++++
 3 files changed, 25 insertions(+)

diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
index c5250a1644..c8fcdf8fa1 100644
--- a/xen/common/device_tree.c
+++ b/xen/common/device_tree.c
@@ -2146,7 +2146,11 @@ int unflatten_device_tree(const void *fdt, struct dt_device_node **mynodes)
 
     dt_dprintk(" <- unflatten_device_tree()\n");
 
+    /* Init r/w lock for host device tree. */
+    rwlock_init(&dt_host->lock);
+
     return 0;
+
 }
 
 static void dt_alias_add(struct dt_alias_prop *ap,
diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
index 301a5bcd97..f4d9deb624 100644
--- a/xen/drivers/passthrough/device_tree.c
+++ b/xen/drivers/passthrough/device_tree.c
@@ -112,6 +112,8 @@ int iommu_release_dt_devices(struct domain *d)
     if ( !is_iommu_enabled(d) )
         return 0;
 
+    read_lock(&dt_host->lock);
+
     list_for_each_entry_safe(dev, _dev, &hd->dt_devices, domain_list)
     {
         rc = iommu_deassign_dt_device(d, dev);
@@ -119,10 +121,14 @@ int iommu_release_dt_devices(struct domain *d)
         {
             dprintk(XENLOG_ERR, "Failed to deassign %s in domain %u\n",
                     dt_node_full_name(dev), d->domain_id);
+
+            read_unlock(&dt_host->lock);
             return rc;
         }
     }
 
+    read_unlock(&dt_host->lock);
+
     return 0;
 }
 
@@ -246,6 +252,8 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
     int ret;
     struct dt_device_node *dev;
 
+    read_lock(&dt_host->lock);
+
     switch ( domctl->cmd )
     {
     case XEN_DOMCTL_assign_device:
@@ -295,7 +303,10 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
         spin_unlock(&dtdevs_lock);
 
         if ( d == dom_io )
+        {
+            read_unlock(&dt_host->lock);
             return -EINVAL;
+        }
 
         ret = iommu_add_dt_device(dev);
         if ( ret < 0 )
@@ -333,7 +344,10 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
             break;
 
         if ( d == dom_io )
+        {
+            read_unlock(&dt_host->lock);
             return -EINVAL;
+        }
 
         ret = iommu_deassign_dt_device(d, dev);
 
@@ -348,5 +362,6 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
         break;
     }
 
+    read_unlock(&dt_host->lock);
     return ret;
 }
diff --git a/xen/include/xen/device_tree.h b/xen/include/xen/device_tree.h
index e239f7de26..dee40d2ea3 100644
--- a/xen/include/xen/device_tree.h
+++ b/xen/include/xen/device_tree.h
@@ -18,6 +18,7 @@
 #include <xen/string.h>
 #include <xen/types.h>
 #include <xen/list.h>
+#include <xen/rwlock.h>
 
 #define DEVICE_TREE_MAX_DEPTH 16
 
@@ -106,6 +107,11 @@ struct dt_device_node {
     struct list_head domain_list;
 
     struct device dev;
+
+    /*
+     * Lock that protects r/w updates to unflattened device tree i.e. dt_host.
+     */
+    rwlock_t lock;
 };
 
 #define dt_to_dev(dt_node)  (&(dt_node)->dev)
-- 
2.17.1

[XEN][PATCH v7 15/19] xen/arm: Implement device tree node removal functionalities

[Vikram Garhwal]

Introduce sysctl XEN_SYSCTL_dt_overlay to remove device-tree nodes added using
device tree overlay.

xl dt-overlay remove file.dtbo:
    Removes all the nodes in a given dtbo.
    First, removes IRQ permissions and MMIO accesses. Next, it finds the nodes
    in dt_host and delete the device node entries from dt_host.

    The nodes get removed only if it is not used by any of dom0 or domio.

Also, added overlay_track struct to keep the track of added node through device
tree overlay. overlay_track has dt_host_new which is unflattened form of updated
fdt and name of overlay nodes. When a node is removed, we also free the memory
used by overlay_track for the particular overlay node.

Nested overlay removal is supported in sequential manner only i.e. if
overlay_child nests under overlay_parent, it is assumed that user first removes
overlay_child and then removes overlay_parent.

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>

---
Changes from v6:
    Add explicit padding for xen_system_dt_overlay{}
    Update license.
    Rearrange xfree in dt_sysctl()
    Update overlay_track struct comment with relevant message.
    Fix missing xen/errno.h for builds without CONFIG_OVERLAY_DTB cases.
    Fix header formatting.
---
 xen/arch/arm/sysctl.c        |  16 +-
 xen/common/Makefile          |   1 +
 xen/common/dt-overlay.c      | 420 +++++++++++++++++++++++++++++++++++
 xen/include/public/sysctl.h  |  24 ++
 xen/include/xen/dt-overlay.h |  59 +++++
 5 files changed, 519 insertions(+), 1 deletion(-)
 create mode 100644 xen/common/dt-overlay.c
 create mode 100644 xen/include/xen/dt-overlay.h

diff --git a/xen/arch/arm/sysctl.c b/xen/arch/arm/sysctl.c
index b0a78a8b10..8b813c970f 100644
--- a/xen/arch/arm/sysctl.c
+++ b/xen/arch/arm/sysctl.c
@@ -9,6 +9,7 @@
 
 #include <xen/types.h>
 #include <xen/lib.h>
+#include <xen/dt-overlay.h>
 #include <xen/errno.h>
 #include <xen/hypercall.h>
 #include <public/sysctl.h>
@@ -21,7 +22,20 @@ void arch_do_physinfo(struct xen_sysctl_physinfo *pi)
 long arch_do_sysctl(struct xen_sysctl *sysctl,
                     XEN_GUEST_HANDLE_PARAM(xen_sysctl_t) u_sysctl)
 {
-    return -ENOSYS;
+    long ret;
+
+    switch ( sysctl->cmd )
+    {
+    case XEN_SYSCTL_dt_overlay:
+        ret = dt_sysctl(&sysctl->u.dt_overlay);
+        break;
+
+    default:
+        ret = -ENOSYS;
+        break;
+    }
+
+    return ret;
 }
 
 /*
diff --git a/xen/common/Makefile b/xen/common/Makefile
index 46049eac35..e7e96b1087 100644
--- a/xen/common/Makefile
+++ b/xen/common/Makefile
@@ -8,6 +8,7 @@ obj-$(CONFIG_DEBUG_TRACE) += debugtrace.o
 obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o
 obj-$(CONFIG_IOREQ_SERVER) += dm.o
 obj-y += domain.o
+obj-$(CONFIG_OVERLAY_DTB) += dt-overlay.o
 obj-y += event_2l.o
 obj-y += event_channel.o
 obj-y += event_fifo.o
diff --git a/xen/common/dt-overlay.c b/xen/common/dt-overlay.c
new file mode 100644
index 0000000000..b2a7e441df
--- /dev/null
+++ b/xen/common/dt-overlay.c
@@ -0,0 +1,420 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * xen/common/dt-overlay.c
+ *
+ * Device tree overlay support in Xen.
+ *
+ * Copyright (C) 2023, Advanced Micro Devices, Inc. All Rights Reserved.
+ * Written by Vikram Garhwal <vikram.garhwal@amd.com>
+ *
+ */
+#include <asm/domain_build.h>
+#include <xen/dt-overlay.h>
+#include <xen/guest_access.h>
+#include <xen/iocap.h>
+#include <xen/xmalloc.h>
+
+static LIST_HEAD(overlay_tracker);
+static DEFINE_SPINLOCK(overlay_lock);
+
+/* Find last descendants of the device_node. */
+static struct dt_device_node *
+                find_last_descendants_node(struct dt_device_node *device_node)
+{
+    struct dt_device_node *child_node;
+
+    for ( child_node = device_node->child; child_node->sibling != NULL;
+          child_node = child_node->sibling );
+
+    /* If last child_node also have children. */
+    if ( child_node->child )
+        child_node = find_last_descendants_node(child_node);
+
+    return child_node;
+}
+
+static int dt_overlay_remove_node(struct dt_device_node *device_node)
+{
+    struct dt_device_node *np;
+    struct dt_device_node *parent_node;
+    struct dt_device_node *device_node_last_descendant = device_node->child;
+
+    parent_node = device_node->parent;
+
+    if ( parent_node == NULL )
+    {
+        dt_dprintk("%s's parent node not found\n", device_node->name);
+        return -EFAULT;
+    }
+
+    np = parent_node->child;
+
+    if ( np == NULL )
+    {
+        dt_dprintk("parent node %s's not found\n", parent_node->name);
+        return -EFAULT;
+    }
+
+    /* If node to be removed is only child node or first child. */
+    if ( !dt_node_cmp(np->full_name, device_node->full_name) )
+    {
+        parent_node->child = np->sibling;
+
+        /*
+         * Iterate over all child nodes of device_node. Given that we are
+         * removing parent node, we need to remove all it's descendants too.
+         */
+        if ( device_node_last_descendant )
+        {
+            device_node_last_descendant =
+                                        find_last_descendants_node(device_node);
+            parent_node->allnext = device_node_last_descendant->allnext;
+        }
+        else
+            parent_node->allnext = np->allnext;
+
+        return 0;
+    }
+
+    for ( np = parent_node->child; np->sibling != NULL; np = np->sibling )
+    {
+        if ( !dt_node_cmp(np->sibling->full_name, device_node->full_name) )
+        {
+            /* Found the node. Now we remove it. */
+            np->sibling = np->sibling->sibling;
+
+            if ( np->child )
+                np = find_last_descendants_node(np);
+
+            /*
+             * Iterate over all child nodes of device_node. Given that we are
+             * removing parent node, we need to remove all it's descendants too.
+             */
+            if ( device_node_last_descendant )
+                device_node_last_descendant =
+                                        find_last_descendants_node(device_node);
+
+            if ( device_node_last_descendant )
+                np->allnext = device_node_last_descendant->allnext;
+            else
+                np->allnext = np->allnext->allnext;
+
+            break;
+        }
+    }
+
+    return 0;
+}
+
+/* Basic sanity check for the dtbo tool stack provided to Xen. */
+static int check_overlay_fdt(const void *overlay_fdt, uint32_t overlay_fdt_size)
+{
+    if ( (fdt_totalsize(overlay_fdt) != overlay_fdt_size) ||
+          fdt_check_header(overlay_fdt) )
+    {
+        printk(XENLOG_ERR "The overlay FDT is not a valid Flat Device Tree\n");
+        return -EINVAL;
+    }
+
+    return 0;
+}
+
+/* Count number of nodes till one level of __overlay__ tag. */
+static unsigned int overlay_node_count(const void *overlay_fdt)
+{
+    unsigned int num_overlay_nodes = 0;
+    int fragment;
+
+    fdt_for_each_subnode(fragment, overlay_fdt, 0)
+    {
+        int subnode;
+        int overlay;
+
+        overlay = fdt_subnode_offset(overlay_fdt, fragment, "__overlay__");
+
+        /*
+         * overlay value can be < 0. But fdt_for_each_subnode() loop checks for
+         * overlay >= 0. So, no need for a overlay>=0 check here.
+         */
+        fdt_for_each_subnode(subnode, overlay_fdt, overlay)
+        {
+            num_overlay_nodes++;
+        }
+    }
+
+    return num_overlay_nodes;
+}
+
+static int handle_remove_irq_iommu(struct dt_device_node *device_node)
+{
+    int rc = 0;
+    struct domain *d = hardware_domain;
+    domid_t domid;
+    unsigned int naddr, len;
+    unsigned int i, nirq;
+
+    domid = dt_device_used_by(device_node);
+
+    dt_dprintk("Checking if node %s is used by any domain\n",
+               device_node->full_name);
+
+    /* Remove the node if only it's assigned to domain 0 or domain io. */
+    if ( domid != 0 && domid != DOMID_IO )
+    {
+        printk(XENLOG_ERR "Device %s is being used by domain %u. Removing nodes failed\n",
+               device_node->full_name, domid);
+        return -EINVAL;
+    }
+
+    dt_dprintk("Removing node: %s\n", device_node->full_name);
+
+    nirq = dt_number_of_irq(device_node);
+
+    /* Remove IRQ permission */
+    for ( i = 0; i < nirq; i++ )
+    {
+        rc = platform_get_irq(device_node, i);
+        if ( rc < 0 )
+        {
+            printk(XENLOG_ERR "Failed to get IRQ num for device node %s\n",
+                   device_node->full_name);
+            return -EINVAL;
+        }
+
+        if ( irq_access_permitted(d, rc) == false )
+        {
+            printk(XENLOG_ERR "IRQ %d is not routed to domain %u\n", rc,
+                   domid);
+            return -EINVAL;
+        }
+        /*
+         * TODO: We don't handle shared IRQs for now. So, it is assumed that
+         * the IRQs was not shared with another devices.
+         */
+        rc = irq_deny_access(d, rc);
+        if ( rc )
+        {
+            printk(XENLOG_ERR "unable to revoke access for irq %u for %s\n",
+                   i, device_node->full_name);
+            return rc;
+        }
+    }
+
+    /* Check if iommu property exists. */
+    if ( dt_get_property(device_node, "iommus", &len) )
+    {
+        rc = iommu_remove_dt_device(device_node);
+        if ( rc != 0 && rc != -ENXIO )
+            return rc;
+    }
+
+    naddr = dt_number_of_address(device_node);
+
+    /* Remove mmio access. */
+    for ( i = 0; i < naddr; i++ )
+    {
+        uint64_t addr, size;
+
+        rc = dt_device_get_address(device_node, i, &addr, &size);
+        if ( rc )
+        {
+            printk(XENLOG_ERR "Unable to retrieve address %u for %s\n",
+                   i, dt_node_full_name(device_node));
+            return rc;
+        }
+
+        rc = iomem_deny_access(d, paddr_to_pfn(addr),
+                               paddr_to_pfn(PAGE_ALIGN(addr + size - 1)));
+        if ( rc )
+        {
+            printk(XENLOG_ERR "Unable to remove dom%d access to"
+                   " 0x%"PRIx64" - 0x%"PRIx64"\n",
+                   d->domain_id,
+                   addr & PAGE_MASK, PAGE_ALIGN(addr + size) - 1);
+            return rc;
+        }
+
+    }
+
+    return rc;
+}
+
+/* Removes all descendants of the given node. */
+static int remove_all_descendant_nodes(struct dt_device_node *device_node)
+{
+    int rc = 0;
+    struct dt_device_node *child_node;
+
+    for ( child_node = device_node->child; child_node != NULL;
+         child_node = child_node->sibling )
+    {
+        if ( child_node->child )
+            remove_all_descendant_nodes(child_node);
+
+        rc = handle_remove_irq_iommu(child_node);
+        if ( rc )
+            return rc;
+    }
+
+    return rc;
+}
+
+/* Remove nodes from dt_host. */
+static int remove_nodes(const struct overlay_track *tracker)
+{
+    int rc = 0;
+    struct dt_device_node *overlay_node;
+    unsigned int j;
+
+    for ( j = 0; j < tracker->num_nodes; j++ )
+    {
+        overlay_node = (struct dt_device_node *)tracker->nodes_address[j];
+        if ( overlay_node == NULL )
+        {
+            printk(XENLOG_ERR "Device %s is not present in the tree. Removing nodes failed\n",
+                   overlay_node->full_name);
+            return -EINVAL;
+        }
+
+        rc = remove_all_descendant_nodes(overlay_node);
+
+        /* All children nodes are unmapped. Now remove the node itself. */
+        rc = handle_remove_irq_iommu(overlay_node);
+        if ( rc )
+            return rc;
+
+        read_lock(&dt_host->lock);
+
+        rc = dt_overlay_remove_node(overlay_node);
+        if ( rc )
+        {
+            read_unlock(&dt_host->lock);
+
+            return rc;
+        }
+
+        read_unlock(&dt_host->lock);
+    }
+
+    return rc;
+}
+
+/*
+ * First finds the device node to remove. Check if the device is being used by
+ * any dom and finally remove it from dt_host. IOMMU is already being taken care
+ * while destroying the domain.
+ */
+static long handle_remove_overlay_nodes(void *overlay_fdt,
+                                        uint32_t overlay_fdt_size)
+{
+    int rc;
+    struct overlay_track *entry, *temp, *track;
+    bool found_entry = false;
+
+    rc = check_overlay_fdt(overlay_fdt, overlay_fdt_size);
+    if ( rc )
+        return rc;
+
+    if ( overlay_node_count(overlay_fdt) == 0 )
+        return -EINVAL;
+
+    spin_lock(&overlay_lock);
+
+    /*
+     * First check if dtbo is correct i.e. it should one of the dtbo which was
+     * used when dynamically adding the node.
+     * Limitation: Cases with same node names but different property are not
+     * supported currently. We are relying on user to provide the same dtbo
+     * as it was used when adding the nodes.
+     */
+    list_for_each_entry_safe( entry, temp, &overlay_tracker, entry )
+    {
+        if ( memcmp(entry->overlay_fdt, overlay_fdt, overlay_fdt_size) == 0 )
+        {
+            track = entry;
+            found_entry = true;
+            break;
+        }
+    }
+
+    if ( found_entry == false )
+    {
+        rc = -EINVAL;
+
+        printk(XENLOG_ERR "Cannot find any matching tracker with input dtbo."
+               " Removing nodes is supported only for prior added dtbo.\n");
+        goto out;
+
+    }
+
+    rc = remove_nodes(entry);
+    if ( rc )
+    {
+        printk(XENLOG_ERR "Removing node failed\n");
+        goto out;
+    }
+
+    list_del(&entry->entry);
+
+    xfree(entry->dt_host_new);
+    xfree(entry->fdt);
+    xfree(entry->overlay_fdt);
+
+    xfree(entry->nodes_address);
+
+    xfree(entry);
+
+out:
+    spin_unlock(&overlay_lock);
+    return rc;
+}
+
+long dt_sysctl(struct xen_sysctl_dt_overlay *op)
+{
+    long ret;
+    void *overlay_fdt;
+
+    if ( op->overlay_fdt_size == 0 || op->overlay_fdt_size > KB(500) )
+        return -EINVAL;
+
+    if ( op->pad[0] || op->pad[1] || op->pad[2] )
+        return -EINVAL;
+
+    overlay_fdt = xmalloc_bytes(op->overlay_fdt_size);
+
+    if ( overlay_fdt == NULL )
+        return -ENOMEM;
+
+    ret = copy_from_guest(overlay_fdt, op->overlay_fdt, op->overlay_fdt_size);
+    if ( ret )
+    {
+        gprintk(XENLOG_ERR, "copy from guest failed\n");
+        xfree(overlay_fdt);
+
+        return -EFAULT;
+    }
+
+    switch ( op->overlay_op )
+    {
+    case XEN_SYSCTL_DT_OVERLAY_REMOVE:
+        ret = handle_remove_overlay_nodes(overlay_fdt, op->overlay_fdt_size);
+
+        break;
+
+    default:
+        break;
+    }
+
+    xfree(overlay_fdt);
+
+    return ret;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
index 2b24d6bfd0..ff54607617 100644
--- a/xen/include/public/sysctl.h
+++ b/xen/include/public/sysctl.h
@@ -1057,6 +1057,25 @@ typedef struct xen_sysctl_cpu_policy xen_sysctl_cpu_policy_t;
 DEFINE_XEN_GUEST_HANDLE(xen_sysctl_cpu_policy_t);
 #endif
 
+#if defined(__arm__) || defined (__aarch64__)
+/*
+ * XEN_SYSCTL_dt_overlay
+ * Performs addition/removal of device tree nodes under parent node using dtbo.
+ * This does in three steps:
+ *  - Adds/Removes the nodes from dt_host.
+ *  - Adds/Removes IRQ permission for the nodes.
+ *  - Adds/Removes MMIO accesses.
+ */
+struct xen_sysctl_dt_overlay {
+    XEN_GUEST_HANDLE_64(void) overlay_fdt;  /* IN: overlay fdt. */
+    uint32_t overlay_fdt_size;              /* IN: Overlay dtb size. */
+#define XEN_SYSCTL_DT_OVERLAY_ADD                   1
+#define XEN_SYSCTL_DT_OVERLAY_REMOVE                2
+    uint8_t overlay_op;                     /* IN: Add or remove. */
+    uint8_t pad[3];                         /* IN: Must be zero. */
+};
+#endif
+
 struct xen_sysctl {
     uint32_t cmd;
 #define XEN_SYSCTL_readconsole                    1
@@ -1087,6 +1106,7 @@ struct xen_sysctl {
 #define XEN_SYSCTL_livepatch_op                  27
 /* #define XEN_SYSCTL_set_parameter              28 */
 #define XEN_SYSCTL_get_cpu_policy                29
+#define XEN_SYSCTL_dt_overlay                    30
     uint32_t interface_version; /* XEN_SYSCTL_INTERFACE_VERSION */
     union {
         struct xen_sysctl_readconsole       readconsole;
@@ -1117,6 +1137,10 @@ struct xen_sysctl {
 #if defined(__i386__) || defined(__x86_64__)
         struct xen_sysctl_cpu_policy        cpu_policy;
 #endif
+
+#if defined(__arm__) || defined (__aarch64__)
+        struct xen_sysctl_dt_overlay        dt_overlay;
+#endif
         uint8_t                             pad[128];
     } u;
 };
diff --git a/xen/include/xen/dt-overlay.h b/xen/include/xen/dt-overlay.h
new file mode 100644
index 0000000000..43fa5a02a0
--- /dev/null
+++ b/xen/include/xen/dt-overlay.h
@@ -0,0 +1,59 @@
+ /* SPDX-License-Identifier: GPL-2.0-only */
+ /*
+ * xen/dt-overlay.h
+ *
+ * Device tree overlay support in Xen.
+ *
+ * Copyright (C) 2023, Advanced Micro Devices, Inc. All Rights Reserved.
+ * Written by Vikram Garhwal <vikram.garhwal@amd.com>
+ *
+ */
+#ifndef __XEN_DT_OVERLAY_H__
+#define __XEN_DT_OVERLAY_H__
+
+#include <xen/list.h>
+#include <xen/libfdt/libfdt.h>
+#include <xen/device_tree.h>
+#include <xen/rangeset.h>
+
+/*
+ * overlay_track describes information about added nodes through dtbo.
+ * @entry: List pointer.
+ * @dt_host_new: Pointer to the updated dt_host_new which is unflattened from
+    the 'updated fdt'.
+ * @fdt: Stores the fdt.
+ * @overlay_fdt: Stores a copy of input overlay_fdt.
+ * @nodes_address: Stores each overlay_node's address.
+ * @num_nodes: Total number of nodes in overlay dtb.
+ */
+struct overlay_track {
+    struct list_head entry;
+    struct dt_device_node *dt_host_new;
+    void *fdt;
+    void *overlay_fdt;
+    unsigned long *nodes_address;
+    unsigned int num_nodes;
+};
+
+struct xen_sysctl_dt_overlay;
+
+#ifdef CONFIG_OVERLAY_DTB
+long dt_sysctl(struct xen_sysctl_dt_overlay *op);
+#else
+#include <xen/errno.h>
+static inline long dt_sysctl(struct xen_sysctl_dt_overlay *op)
+{
+    return -EOPNOTSUPP;
+}
+#endif
+
+#endif /* __XEN_DT_OVERLAY_H__ */
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
2.17.1

[XEN][PATCH v7 16/19] xen/arm: Implement device tree node addition functionalities

[Vikram Garhwal]

Update sysctl XEN_SYSCTL_dt_overlay to enable support for dtbo nodes addition
using device tree overlay.

xl dt-overlay add file.dtbo:
    Each time overlay nodes are added using .dtbo, a new fdt(memcpy of
    device_tree_flattened) is created and updated with overlay nodes. This
    updated fdt is further unflattened to a dt_host_new. Next, it checks if any
    of the overlay nodes already exists in the dt_host. If overlay nodes doesn't
    exist then find the overlay nodes in dt_host_new, find the overlay node's
    parent in dt_host and add the nodes as child under their parent in the
    dt_host. The node is attached as the last node under target parent.

    Finally, add IRQs, add device to IOMMUs, set permissions and map MMIO for the
    overlay node.

When a node is added using overlay, a new entry is allocated in the
overlay_track to keep the track of memory allocation due to addition of overlay
node. This is helpful for freeing the memory allocated when a device tree node
is removed.

The main purpose of this to address first part of dynamic programming i.e.
making xen aware of new device tree node which means updating the dt_host with
overlay node information. Here we are adding/removing node from dt_host, and
checking/setting IOMMU and IRQ permission but never mapping them to any domain.
Right now, mapping/Un-mapping will happen only when a new domU is
created/destroyed using "xl create".

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>

---
Changes from v6:
    Fix comment style and add comment regarding false flag in irq mapping.
    Move malloc for nodes_full_path to handle_add_overlay_nodes.
    Move node_num define to start of overlay_get_nodes_info().
    Remove "domain *d" from handle_add_irq_iommu().
    Fix error handling for handle_add_irq_iommu().
    Split handle_add_overlay_nodes to two functions.
    Create a separate function for freeing nodes_full_path.
    Fix xfree for dt_sysctl.
---
 xen/common/dt-overlay.c | 533 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 533 insertions(+)

diff --git a/xen/common/dt-overlay.c b/xen/common/dt-overlay.c
index b2a7e441df..12b6b010ef 100644
--- a/xen/common/dt-overlay.c
+++ b/xen/common/dt-overlay.c
@@ -33,6 +33,25 @@ static struct dt_device_node *
     return child_node;
 }
 
+/*
+ * Returns next node to the input node. If node has children then return
+ * last descendant's next node.
+*/
+static struct dt_device_node *
+dt_find_next_node(struct dt_device_node *dt, const struct dt_device_node *node)
+{
+    struct dt_device_node *np;
+
+    dt_for_each_device_node(dt, np)
+        if ( np == node )
+            break;
+
+    if ( np->child )
+        np = find_last_descendants_node(np);
+
+    return np->allnext;
+}
+
 static int dt_overlay_remove_node(struct dt_device_node *device_node)
 {
     struct dt_device_node *np;
@@ -106,6 +125,78 @@ static int dt_overlay_remove_node(struct dt_device_node *device_node)
     return 0;
 }
 
+static int dt_overlay_add_node(struct dt_device_node *device_node,
+                               const char *parent_node_path)
+{
+    struct dt_device_node *parent_node;
+    struct dt_device_node *next_node;
+
+    parent_node = dt_find_node_by_path(parent_node_path);
+
+    if ( parent_node == NULL )
+    {
+        dt_dprintk("Parent node %s not found. Overlay node will not be added\n",
+                   parent_node_path);
+        return -EINVAL;
+    }
+
+    /* If parent has no child. */
+    if ( parent_node->child == NULL )
+    {
+        next_node = parent_node->allnext;
+        device_node->parent = parent_node;
+        parent_node->allnext = device_node;
+        parent_node->child = device_node;
+    }
+    else
+    {
+        struct dt_device_node *np;
+        /*
+         * If parent has at least one child node.
+         * Iterate to the last child node of parent.
+         */
+        for ( np = parent_node->child; np->sibling != NULL; np = np->sibling );
+
+        /* Iterate over all child nodes of np node. */
+        if ( np->child )
+        {
+            struct dt_device_node *np_last_descendant;
+
+            np_last_descendant = find_last_descendants_node(np);
+
+            next_node = np_last_descendant->allnext;
+            np_last_descendant->allnext = device_node;
+        }
+        else
+        {
+            next_node = np->allnext;
+            np->allnext = device_node;
+        }
+
+        device_node->parent = parent_node;
+        np->sibling = device_node;
+        np->sibling->sibling = NULL;
+    }
+
+    /* Iterate over all child nodes of device_node to add children too. */
+    if ( device_node->child )
+    {
+        struct dt_device_node *device_node_last_descendant;
+
+        device_node_last_descendant = find_last_descendants_node(device_node);
+
+        /* Plug next_node at the end of last children of device_node. */
+        device_node_last_descendant->allnext = next_node;
+    }
+    else
+    {
+        /* Now plug next_node at the end of device_node. */
+        device_node->allnext = next_node;
+    }
+
+    return 0;
+}
+
 /* Basic sanity check for the dtbo tool stack provided to Xen. */
 static int check_overlay_fdt(const void *overlay_fdt, uint32_t overlay_fdt_size)
 {
@@ -145,6 +236,76 @@ static unsigned int overlay_node_count(const void *overlay_fdt)
     return num_overlay_nodes;
 }
 
+/*
+ * overlay_get_nodes_info gets full name with path for all the nodes which
+ * are in one level of __overlay__ tag. This is useful when checking node for
+ * duplication i.e. dtbo tries to add nodes which already exists in device tree.
+ */
+static int overlay_get_nodes_info(const void *fdto, char **nodes_full_path)
+{
+    int fragment;
+    unsigned int node_num = 0;
+
+    fdt_for_each_subnode(fragment, fdto, 0)
+    {
+        int target;
+        int overlay;
+        int subnode;
+        const char *target_path;
+
+        target = fdt_overlay_target_offset(device_tree_flattened, fdto,
+                                           fragment, &target_path);
+        if ( target < 0 )
+            return target;
+
+        if ( target_path == NULL )
+            return -EINVAL;
+
+        overlay = fdt_subnode_offset(fdto, fragment, "__overlay__");
+
+        /*
+         * overlay value can be < 0. But fdt_for_each_subnode() loop checks for
+         * overlay >= 0. So, no need for a overlay>=0 check here.
+         */
+        fdt_for_each_subnode(subnode, fdto, overlay)
+        {
+            const char *node_name = NULL;
+            int node_name_len;
+            unsigned int target_path_len = strlen(target_path);
+            unsigned int node_full_name_len;
+
+            node_name = fdt_get_name(fdto, subnode, &node_name_len);
+
+            if ( node_name == NULL )
+                return node_name_len;
+
+            /*
+             * Magic number 2 is for adding '/' and '\0'. This is done to keep
+             * the node_full_path in the correct full node name format.
+             */
+            node_full_name_len = target_path_len + node_name_len + 2;
+
+            nodes_full_path[node_num] = xmalloc_bytes(node_full_name_len);
+
+            if ( nodes_full_path[node_num] == NULL )
+                return -ENOMEM;
+
+            memcpy(nodes_full_path[node_num], target_path, target_path_len);
+
+            nodes_full_path[node_num][target_path_len] = '/';
+
+            memcpy(nodes_full_path[node_num] + target_path_len + 1,
+                    node_name, node_name_len);
+
+            nodes_full_path[node_num][node_full_name_len - 1] = '\0';
+
+            node_num++;
+        }
+    }
+
+    return 0;
+}
+
 static int handle_remove_irq_iommu(struct dt_device_node *device_node)
 {
     int rc = 0;
@@ -369,6 +530,373 @@ out:
     return rc;
 }
 
+/*
+ * Handles IRQ and IOMMU mapping for the overlay_node and all descendants of the
+ * overlay_node.
+ */
+static int handle_add_irq_iommu(struct dt_device_node *overlay_node)
+{
+    int rc;
+    unsigned int naddr, i, len;
+    struct dt_device_node *np;
+
+    /*
+     * First let's handle the interrupts.
+     * For now, need_mapping is set to false which means it will only permit IRQ
+     * access to hardware_domain using irq_permit_access() but will never route
+     * as route_irq_to_guest() will not be called with false flag.
+     */
+    rc = handle_device_interrupts(hardware_domain, overlay_node, false);
+    if ( rc < 0 )
+    {
+        printk(XENLOG_ERR "Failed to retrieve interrupts configuration\n");
+        return rc;
+    }
+
+    /* Check if iommu property exists. */
+    if ( dt_get_property(overlay_node, "iommus", &len) )
+    {
+        /* Add device to IOMMUs. */
+        rc = iommu_add_dt_device(overlay_node);
+        if ( rc < 0 )
+        {
+            printk(XENLOG_ERR "Failed to add %s to the IOMMU\n",
+                   dt_node_full_name(overlay_node));
+            return rc;
+        }
+    }
+
+    /* Set permissions. */
+    naddr = dt_number_of_address(overlay_node);
+
+    dt_dprintk("%s naddr = %u\n", dt_node_full_name(overlay_node), naddr);
+
+    /* Give permission to map MMIOs */
+    for ( i = 0; i < naddr; i++ )
+    {
+        uint64_t addr, size;
+
+        /*
+         * For now, we skip_mapping which means it will only permit iomem access
+         * to hardware_domain using iomem_permit_access() but will never map as
+         * map_range_p2mt() will not be called.
+         */
+        struct map_range_data mr_data = { .d = hardware_domain,
+                                          .p2mt = p2m_mmio_direct_c,
+                                          .skip_mapping = true
+                                        };
+
+        rc = dt_device_get_address(overlay_node, i, &addr, &size);
+        if ( rc )
+        {
+            printk(XENLOG_ERR "Unable to retrieve address %u for %s\n",
+                   i, dt_node_full_name(overlay_node));
+            return rc;
+        }
+
+        rc = map_range_to_domain(overlay_node, addr, size, &mr_data);
+        if ( rc )
+            return rc;
+    }
+
+    /* Map IRQ and IOMMU for overlay_node's children. */
+    for ( np = overlay_node->child; np != NULL; np = np->sibling )
+    {
+        rc = handle_add_irq_iommu(np);
+        if ( rc )
+        {
+            printk(XENLOG_ERR "Adding IRQ and IOMMU failed\n");
+            return rc;
+        }
+    }
+
+    return rc;
+}
+
+static void free_nodes_full_path(int num_nodes, char **nodes_full_path)
+{
+    int i;
+
+    if ( nodes_full_path != NULL )
+    {
+        for ( i = 0; i < num_nodes && nodes_full_path[i] != NULL;
+              i++ )
+        {
+            xfree(nodes_full_path[i]);
+        }
+        xfree(nodes_full_path);
+    }
+
+    return;
+}
+
+static long add_nodes(struct overlay_track *tr, char **nodes_full_path)
+
+{
+    int j, rc;
+    struct dt_device_node *overlay_node;
+
+    for ( j = 0; j < tr->num_nodes; j++ )
+    {
+        struct dt_device_node *prev_node, *next_node;
+
+        dt_dprintk("Adding node: %s\n", nodes_full_path[j]);
+
+        /* Find the newly added node in tr->dt_host_new by it's full path. */
+        overlay_node = device_tree_find_node_by_path(tr->dt_host_new,
+                                                     nodes_full_path[j]);
+        if ( overlay_node == NULL )
+        {
+            /* Sanity check. But code will never come here. */
+            ASSERT_UNREACHABLE();
+            return -EFAULT;
+        }
+
+        /*
+         * Find previous and next node to overlay_node in dt_host_new. We will
+         * need these nodes to fix the dt_host_new mapping. When overlay_node is
+         * take out of dt_host_new tree and added to dt_host, link between
+         * previous node and next_node is broken. We will need to refresh
+         * dt_host_new with correct linking for any other overlay nodes
+         * extraction in future.
+         */
+        dt_for_each_device_node(tr->dt_host_new, prev_node)
+            if ( prev_node->allnext == overlay_node )
+                break;
+
+        next_node = dt_find_next_node(tr->dt_host_new, overlay_node);
+
+        read_lock(&dt_host->lock);
+
+        /* Add the node to dt_host. */
+        rc = dt_overlay_add_node(overlay_node, overlay_node->parent->full_name);
+        if ( rc )
+        {
+            read_unlock(&dt_host->lock);
+
+            /* Node not added in dt_host. */
+            return rc;
+        }
+
+        read_unlock(&dt_host->lock);
+
+        prev_node->allnext = next_node;
+
+        overlay_node = dt_find_node_by_path(overlay_node->full_name);
+        if ( overlay_node == NULL )
+        {
+            /* Sanity check. But code will never come here. */
+            ASSERT_UNREACHABLE();
+            return -EFAULT;
+        }
+
+        rc = handle_add_irq_iommu(overlay_node);
+        if ( rc )
+        {
+            printk(XENLOG_ERR "Adding IRQ and IOMMU failed\n");
+            return rc;
+        }
+
+        /* Keep overlay_node address in tracker. */
+        tr->nodes_address[j] = (unsigned long)overlay_node;
+    }
+
+    return 0;
+}
+/*
+ * Adds device tree nodes under target node.
+ * We use tr->dt_host_new to unflatten the updated device_tree_flattened. This
+ * is done to avoid the removal of device_tree generation, iomem regions mapping
+ * to hardware domain done by handle_node().
+ */
+static long handle_add_overlay_nodes(void *overlay_fdt,
+                                     uint32_t overlay_fdt_size)
+{
+    int rc, j;
+    struct dt_device_node *overlay_node;
+    struct overlay_track *tr = NULL;
+    char **nodes_full_path = NULL;
+    unsigned int new_fdt_size;
+
+    tr = xzalloc(struct overlay_track);
+    if ( tr == NULL )
+        return -ENOMEM;
+
+    new_fdt_size = fdt_totalsize(device_tree_flattened) +
+                                 fdt_totalsize(overlay_fdt);
+
+    tr->fdt = xzalloc_bytes(new_fdt_size);
+    if ( tr->fdt == NULL )
+    {
+        xfree(tr);
+        return -ENOMEM;
+    }
+
+    tr->num_nodes = overlay_node_count(overlay_fdt);
+    if ( tr->num_nodes == 0 )
+    {
+        xfree(tr->fdt);
+        xfree(tr);
+        return -ENOMEM;
+    }
+
+    tr->nodes_address = xzalloc_bytes(tr->num_nodes * sizeof(unsigned long));
+    if ( tr->nodes_address == NULL )
+    {
+        xfree(tr->fdt);
+        xfree(tr);
+        return -ENOMEM;
+    }
+
+    rc = check_overlay_fdt(overlay_fdt, overlay_fdt_size);
+    if ( rc )
+    {
+        xfree(tr->nodes_address);
+        xfree(tr->fdt);
+        xfree(tr);
+        return rc;
+    }
+
+    /*
+     * Keep a copy of overlay_fdt as fdt_overlay_apply will change the input
+     * overlay's content(magic) when applying overlay.
+     */
+    tr->overlay_fdt = xzalloc_bytes(overlay_fdt_size);
+    if ( tr->overlay_fdt == NULL )
+    {
+        xfree(tr->nodes_address);
+        xfree(tr->fdt);
+        xfree(tr);
+        return -ENOMEM;
+    }
+
+    memcpy(tr->overlay_fdt, overlay_fdt, overlay_fdt_size);
+
+    spin_lock(&overlay_lock);
+
+    memcpy(tr->fdt, device_tree_flattened,
+           fdt_totalsize(device_tree_flattened));
+
+    /* Open tr->fdt with more space to accommodate the overlay_fdt. */
+    rc = fdt_open_into(tr->fdt, tr->fdt, new_fdt_size);
+    if ( rc )
+    {
+        printk(XENLOG_ERR "Increasing fdt size to accommodate overlay_fdt failed with error %d\n",
+               rc);
+        goto err;
+    }
+
+    nodes_full_path = xzalloc_bytes(tr->num_nodes * sizeof(char *));
+    if ( nodes_full_path == NULL )
+    {
+        rc = -ENOMEM;
+        goto err;
+    }
+
+    /*
+     * overlay_get_nodes_info is called to get the node information from dtbo.
+     * This is done before fdt_overlay_apply() because the overlay apply will
+     * erase the magic of overlay_fdt.
+     */
+    rc = overlay_get_nodes_info(overlay_fdt, nodes_full_path);
+    if ( rc )
+    {
+        printk(XENLOG_ERR "Getting nodes information failed with error %d\n",
+               rc);
+        goto err;
+    }
+
+    rc = fdt_overlay_apply(tr->fdt, overlay_fdt);
+    if ( rc )
+    {
+        printk(XENLOG_ERR "Adding overlay node failed with error %d\n", rc);
+        goto err;
+    }
+
+    /*
+     * Check if any of the node already exists in dt_host. If node already exits
+     * we can return here as this overlay_fdt is not suitable for overlay ops.
+     */
+    for ( j = 0; j < tr->num_nodes; j++ )
+    {
+        overlay_node = dt_find_node_by_path(nodes_full_path[j]);
+        if ( overlay_node != NULL )
+        {
+            printk(XENLOG_ERR "node %s exists in device tree\n",
+                   nodes_full_path[j]);
+            rc = -EINVAL;
+            goto err;
+        }
+    }
+
+    /* Unflatten the tr->fdt into a new dt_host. */
+    rc = unflatten_device_tree(tr->fdt, &tr->dt_host_new);
+    if ( rc )
+    {
+        printk(XENLOG_ERR "unflatten_device_tree failed with error %d\n", rc);
+        goto err;
+    }
+
+    rc = add_nodes(tr, nodes_full_path);
+    if ( rc )
+    {
+        printk(XENLOG_ERR "Adding nodes failed. Removing the partially added nodes.\n");
+        goto remove_node;
+    }
+
+    INIT_LIST_HEAD(&tr->entry);
+    list_add_tail(&tr->entry, &overlay_tracker);
+
+    spin_unlock(&overlay_lock);
+
+    free_nodes_full_path(tr->num_nodes, nodes_full_path);
+
+    return rc;
+
+/*
+ * Failure case. We need to remove the nodes, free tracker(if tr exists) and
+ * tr->dt_host_new.
+ */
+remove_node:
+    tr->num_nodes = j;
+    rc = remove_nodes(tr);
+
+    if ( rc )
+    {
+        /*
+         * User needs to provide right overlay. Incorrect node information
+         * example parent node doesn't exist in dt_host etc can cause memory
+         * leaks as removing_nodes() will fail and this means nodes memory is
+         * not freed from tracker. Which may cause memory leaks. Ideally, these
+         * device tree related mistakes will be caught by fdt_overlay_apply()
+         * but given that we don't manage that code keeping this warning message
+         * is better here.
+         */
+        printk(XENLOG_ERR "Removing node failed.\n");
+        spin_unlock(&overlay_lock);
+
+        free_nodes_full_path(tr->num_nodes, nodes_full_path);
+
+        return rc;
+    }
+
+err:
+    spin_unlock(&overlay_lock);
+
+    if ( tr->dt_host_new )
+        xfree(tr->dt_host_new);
+
+    xfree(tr->overlay_fdt);
+    xfree(tr->nodes_address);
+    xfree(tr->fdt);
+
+    free_nodes_full_path(tr->num_nodes, nodes_full_path);
+
+    xfree(tr);
+
+    return rc;
+}
+
 long dt_sysctl(struct xen_sysctl_dt_overlay *op)
 {
     long ret;
@@ -396,6 +924,11 @@ long dt_sysctl(struct xen_sysctl_dt_overlay *op)
 
     switch ( op->overlay_op )
     {
+    case XEN_SYSCTL_DT_OVERLAY_ADD:
+        ret = handle_add_overlay_nodes(overlay_fdt, op->overlay_fdt_size);
+
+        break;
+
     case XEN_SYSCTL_DT_OVERLAY_REMOVE:
         ret = handle_remove_overlay_nodes(overlay_fdt, op->overlay_fdt_size);
 
-- 
2.17.1

[XEN][PATCH v7 17/19] tools/libs/ctrl: Implement new xc interfaces for dt overlay

[Vikram Garhwal]

xc_dt_overlay() sends the device tree binary overlay, size of .dtbo and overlay
operation type i.e. add or remove to xen.

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
---
 tools/include/xenctrl.h         |  5 ++++
 tools/libs/ctrl/Makefile.common |  1 +
 tools/libs/ctrl/xc_dt_overlay.c | 51 +++++++++++++++++++++++++++++++++
 3 files changed, 57 insertions(+)
 create mode 100644 tools/libs/ctrl/xc_dt_overlay.c

diff --git a/tools/include/xenctrl.h b/tools/include/xenctrl.h
index dba33d5d0f..411f7ef04b 100644
--- a/tools/include/xenctrl.h
+++ b/tools/include/xenctrl.h
@@ -2626,6 +2626,11 @@ int xc_livepatch_replace(xc_interface *xch, char *name, uint32_t timeout, uint32
 int xc_domain_cacheflush(xc_interface *xch, uint32_t domid,
                          xen_pfn_t start_pfn, xen_pfn_t nr_pfns);
 
+#if defined(__arm__) || defined(__aarch64__)
+int xc_dt_overlay(xc_interface *xch, void *overlay_fdt,
+                  uint32_t overlay_fdt_size, uint8_t overlay_op);
+#endif
+
 /* Compat shims */
 #include "xenctrl_compat.h"
 
diff --git a/tools/libs/ctrl/Makefile.common b/tools/libs/ctrl/Makefile.common
index 0a09c28fd3..247afbe5f9 100644
--- a/tools/libs/ctrl/Makefile.common
+++ b/tools/libs/ctrl/Makefile.common
@@ -24,6 +24,7 @@ OBJS-y       += xc_hcall_buf.o
 OBJS-y       += xc_foreign_memory.o
 OBJS-y       += xc_kexec.o
 OBJS-y       += xc_resource.o
+OBJS-$(CONFIG_ARM)  += xc_dt_overlay.o
 OBJS-$(CONFIG_X86) += xc_psr.o
 OBJS-$(CONFIG_X86) += xc_pagetab.o
 OBJS-$(CONFIG_Linux) += xc_linux.o
diff --git a/tools/libs/ctrl/xc_dt_overlay.c b/tools/libs/ctrl/xc_dt_overlay.c
new file mode 100644
index 0000000000..58283b9ef6
--- /dev/null
+++ b/tools/libs/ctrl/xc_dt_overlay.c
@@ -0,0 +1,51 @@
+/*
+ *
+ * Device Tree Overlay functions.
+ * Copyright (C) 2021 Xilinx Inc.
+ * Author Vikram Garhwal <fnu.vikram@xilinx.com>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation;
+ * version 2.1 of the License.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "xc_private.h"
+
+int xc_dt_overlay(xc_interface *xch, void *overlay_fdt,
+                  uint32_t overlay_fdt_size, uint8_t overlay_op)
+{
+    int err;
+    DECLARE_SYSCTL;
+
+    DECLARE_HYPERCALL_BOUNCE(overlay_fdt, overlay_fdt_size,
+                             XC_HYPERCALL_BUFFER_BOUNCE_IN);
+
+    if ( (err = xc_hypercall_bounce_pre(xch, overlay_fdt)) )
+        goto err;
+
+    sysctl.cmd = XEN_SYSCTL_dt_overlay;
+    sysctl.u.dt_overlay.overlay_op = overlay_op;
+    sysctl.u.dt_overlay.overlay_fdt_size = overlay_fdt_size;
+    sysctl.u.dt_overlay.pad[0]= 0;
+    sysctl.u.dt_overlay.pad[1]= 0;
+    sysctl.u.dt_overlay.pad[2]= 0;
+
+    set_xen_guest_handle(sysctl.u.dt_overlay.overlay_fdt, overlay_fdt);
+
+    if ( (err = do_sysctl(xch, &sysctl)) != 0 )
+        PERROR("%s failed", __func__);
+
+err:
+    xc_hypercall_bounce_post(xch, overlay_fdt);
+
+    return err;
+}
-- 
2.17.1

[XEN][PATCH v7 18/19] tools/libs/light: Implement new libxl functions for device tree overlay ops

[Vikram Garhwal]

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
Reviewed-by: Anthony PERARD <anthony.perard@citrix.com>
---
 tools/include/libxl.h               | 11 +++++
 tools/libs/light/Makefile           |  3 ++
 tools/libs/light/libxl_dt_overlay.c | 71 +++++++++++++++++++++++++++++
 3 files changed, 85 insertions(+)
 create mode 100644 tools/libs/light/libxl_dt_overlay.c

diff --git a/tools/include/libxl.h b/tools/include/libxl.h
index cfa1a19131..1c5e8abaae 100644
--- a/tools/include/libxl.h
+++ b/tools/include/libxl.h
@@ -250,6 +250,12 @@
  */
 #define LIBXL_HAVE_DEVICETREE_PASSTHROUGH 1
 
+#if defined(__arm__) || defined(__aarch64__)
+/**
+ * This means Device Tree Overlay is supported.
+ */
+#define LIBXL_HAVE_DT_OVERLAY 1
+#endif
 /*
  * libxl_domain_build_info has device_model_user to specify the user to
  * run the device model with. See docs/misc/qemu-deprivilege.txt.
@@ -2453,6 +2459,11 @@ libxl_device_pci *libxl_device_pci_list(libxl_ctx *ctx, uint32_t domid,
                                         int *num);
 void libxl_device_pci_list_free(libxl_device_pci* list, int num);
 
+#if defined(__arm__) || defined(__aarch64__)
+int libxl_dt_overlay(libxl_ctx *ctx, void *overlay,
+                     uint32_t overlay_size, uint8_t overlay_op);
+#endif
+
 /*
  * Turns the current process into a backend device service daemon
  * for a driver domain.
diff --git a/tools/libs/light/Makefile b/tools/libs/light/Makefile
index 5d7ff94b05..ba4c1b7933 100644
--- a/tools/libs/light/Makefile
+++ b/tools/libs/light/Makefile
@@ -112,6 +112,9 @@ OBJS-y += _libxl_types.o
 OBJS-y += libxl_flask.o
 OBJS-y += _libxl_types_internal.o
 
+# Device tree overlay is enabled only for ARM architecture.
+OBJS-$(CONFIG_ARM) += libxl_dt_overlay.o
+
 ifeq ($(CONFIG_LIBNL),y)
 CFLAGS_LIBXL += $(LIBNL3_CFLAGS)
 endif
diff --git a/tools/libs/light/libxl_dt_overlay.c b/tools/libs/light/libxl_dt_overlay.c
new file mode 100644
index 0000000000..a6c709a6dc
--- /dev/null
+++ b/tools/libs/light/libxl_dt_overlay.c
@@ -0,0 +1,71 @@
+/*
+ * Copyright (C) 2021 Xilinx Inc.
+ * Author Vikram Garhwal <fnu.vikram@xilinx.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+#include "libxl_internal.h"
+#include <libfdt.h>
+#include <xenctrl.h>
+
+static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size)
+{
+    int r;
+
+    if (fdt_magic(fdt) != FDT_MAGIC) {
+        LOG(ERROR, "Overlay FDT is not a valid Flat Device Tree");
+        return ERROR_FAIL;
+    }
+
+    r = fdt_check_header(fdt);
+    if (r) {
+        LOG(ERROR, "Failed to check the overlay FDT (%d)", r);
+        return ERROR_FAIL;
+    }
+
+    if (fdt_totalsize(fdt) > size) {
+        LOG(ERROR, "Overlay FDT totalsize is too big");
+        return ERROR_FAIL;
+    }
+
+    return 0;
+}
+
+int libxl_dt_overlay(libxl_ctx *ctx, void *overlay_dt, uint32_t overlay_dt_size,
+                     uint8_t overlay_op)
+{
+    int rc;
+    int r;
+    GC_INIT(ctx);
+
+    if (check_overlay_fdt(gc, overlay_dt, overlay_dt_size)) {
+        LOG(ERROR, "Overlay DTB check failed");
+        rc = ERROR_FAIL;
+        goto out;
+    } else {
+        LOG(DEBUG, "Overlay DTB check passed");
+        rc = 0;
+    }
+
+    r = xc_dt_overlay(ctx->xch, overlay_dt, overlay_dt_size, overlay_op);
+
+    if (r) {
+        LOG(ERROR, "%s: Adding/Removing overlay dtb failed.", __func__);
+        rc = ERROR_FAIL;
+    }
+
+out:
+    GC_FREE;
+    return rc;
+}
+
-- 
2.17.1

[XEN][PATCH v7 19/19] tools/xl: Add new xl command overlay for device tree overlay support

[Vikram Garhwal]

Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
Reviewed-by: Anthony PERARD <anthony.perard@citrix.com>
---
 tools/xl/xl.h           |  1 +
 tools/xl/xl_cmdtable.c  |  6 +++++
 tools/xl/xl_vmcontrol.c | 52 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 59 insertions(+)

diff --git a/tools/xl/xl.h b/tools/xl/xl.h
index 72538d6a81..a923daccd3 100644
--- a/tools/xl/xl.h
+++ b/tools/xl/xl.h
@@ -138,6 +138,7 @@ int main_shutdown(int argc, char **argv);
 int main_reboot(int argc, char **argv);
 int main_list(int argc, char **argv);
 int main_vm_list(int argc, char **argv);
+int main_dt_overlay(int argc, char **argv);
 int main_create(int argc, char **argv);
 int main_config_update(int argc, char **argv);
 int main_button_press(int argc, char **argv);
diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
index ccf4d83584..db0acff62a 100644
--- a/tools/xl/xl_cmdtable.c
+++ b/tools/xl/xl_cmdtable.c
@@ -630,6 +630,12 @@ const struct cmd_spec cmd_table[] = {
       "Issue a qemu monitor command to the device model of a domain",
       "<Domain> <Command>",
     },
+    { "dt-overlay",
+      &main_dt_overlay, 0, 1,
+      "Add/Remove a device tree overlay",
+      "add/remove <.dtbo>"
+      "-h print this help\n"
+    },
 };
 
 const int cmdtable_len = ARRAY_SIZE(cmd_table);
diff --git a/tools/xl/xl_vmcontrol.c b/tools/xl/xl_vmcontrol.c
index 5518c78dc6..de56e00d8b 100644
--- a/tools/xl/xl_vmcontrol.c
+++ b/tools/xl/xl_vmcontrol.c
@@ -1265,6 +1265,58 @@ int main_create(int argc, char **argv)
     return 0;
 }
 
+int main_dt_overlay(int argc, char **argv)
+{
+    const char *overlay_ops = NULL;
+    const char *overlay_config_file = NULL;
+    void *overlay_dtb = NULL;
+    int rc;
+    uint8_t op;
+    int overlay_dtb_size = 0;
+    const int overlay_add_op = 1;
+    const int overlay_remove_op = 2;
+
+    if (argc < 2) {
+        help("dt_overlay");
+        return EXIT_FAILURE;
+    }
+
+    overlay_ops = argv[1];
+    overlay_config_file = argv[2];
+
+    if (strcmp(overlay_ops, "add") == 0)
+        op = overlay_add_op;
+    else if (strcmp(overlay_ops, "remove") == 0)
+        op = overlay_remove_op;
+    else {
+        fprintf(stderr, "Invalid dt overlay operation\n");
+        return EXIT_FAILURE;
+    }
+
+    if (overlay_config_file) {
+        rc = libxl_read_file_contents(ctx, overlay_config_file,
+                                      &overlay_dtb, &overlay_dtb_size);
+
+        if (rc) {
+            fprintf(stderr, "failed to read the overlay device tree file %s\n",
+                    overlay_config_file);
+            free(overlay_dtb);
+            return ERROR_FAIL;
+        }
+    } else {
+        fprintf(stderr, "overlay dtbo file not provided\n");
+        return ERROR_FAIL;
+    }
+
+    rc = libxl_dt_overlay(ctx, overlay_dtb, overlay_dtb_size, op);
+
+    free(overlay_dtb);
+
+    if (rc)
+        return EXIT_FAILURE;
+
+    return rc;
+}
 /*
  * Local variables:
  * mode: C
-- 
2.17.1

RE: [XEN][PATCH v7 05/19] xen/arm: Add CONFIG_OVERLAY_DTB

[Henry Wang]

Hi Vikram,

> -----Original Message-----
> Subject: [XEN][PATCH v7 05/19] xen/arm: Add CONFIG_OVERLAY_DTB
> 
> Introduce a config option where the user can enable support for
> adding/removing
> device tree nodes using a device tree binary overlay.
> 
> Update SUPPORT.md and CHANGELOG.md to state the Device Tree Overlays
> support for
> Arm.
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>

Acked-by: Henry Wang <Henry.Wang@arm.com> # CHANGELOG

Kind regards,
Henry

RE: [XEN][PATCH v7 08/19] xen/device-tree: Add device_tree_find_node_by_path() to find nodes in device tree

[Henry Wang]

Hi Vikram,

> -----Original Message-----
> Subject: [XEN][PATCH v7 08/19] xen/device-tree: Add
> device_tree_find_node_by_path() to find nodes in device tree
> 
> Add device_tree_find_node_by_path() to find a matching node with path for
> a
> dt_device_node.
> 
> Reason behind this function:
>     Each time overlay nodes are added using .dtbo, a new fdt(memcpy of
>     device_tree_flattened) is created and updated with overlay nodes. This
>     updated fdt is further unflattened to a dt_host_new. Next, we need to find
>     the overlay nodes in dt_host_new, find the overlay node's parent in dt_host
>     and add the nodes as child under their parent in the dt_host. Thus we need
>     this function to search for node in different unflattened device trees.
> 
> Also, make dt_find_node_by_path() static inline.
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> 
> ---
> Changes from v6:
>     Rename "dt_node" to "from"
> ---
>  xen/common/device_tree.c      |  6 ++++--
>  xen/include/xen/device_tree.h | 18 ++++++++++++++++--
>  2 files changed, 20 insertions(+), 4 deletions(-)
> 
> diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
> index 16b4b4e946..c5250a1644 100644
> --- a/xen/common/device_tree.c
> +++ b/xen/common/device_tree.c
> @@ -358,11 +358,13 @@ struct dt_device_node
> *dt_find_node_by_type(struct dt_device_node *from,
>      return np;
>  }
> 
> -struct dt_device_node *dt_find_node_by_path(const char *path)
> +struct dt_device_node *
> +                    device_tree_find_node_by_path(struct dt_device_node *from,
> +                                                  const char *path)

NIT: I found that the indentation here is a bit strange to me. I personally would
write like:
struct dt_device_node *
device_tree_find_node_by_path(struct dt_device_node *from, char *path)

[...]

> -struct dt_device_node *dt_find_node_by_path(const char *path);
> +struct dt_device_node *
> +                    device_tree_find_node_by_path(struct dt_device_node *from,
> +                                                  const char *path);

Same here.

But anyway, the content of this patch looks good to me and I confirm you've
addressed the comment of mine and Michal's in v6, so:

Reviewed-by: Henry Wang <Henry.Wang@arm.com>

Kind regards,
Henry

RE: [XEN][PATCH v7 14/19] common/device_tree: Add rwlock for dt_host

[Henry Wang]

Hi Vikram,

> -----Original Message-----
> Subject: [XEN][PATCH v7 14/19] common/device_tree: Add rwlock for dt_host
> 
>  Dynamic programming ops will modify the dt_host and there might be other
>  function which are browsing the dt_host at the same time. To avoid the race
>  conditions, adding rwlock for browsing the dt_host during runtime.
> 
>  Reason behind adding rwlock instead of spinlock:
>     For now, dynamic programming is the sole modifier of dt_host in Xen
> during
>         run time. All other access functions like iommu_release_dt_device() are
>         just reading the dt_host during run-time. So, there is a need to protect
>         others from browsing the dt_host while dynamic programming is
> modifying
>         it. rwlock is better suitable for this task as spinlock won't be able to
>         differentiate between read and write access.
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>

Reviewed-by: Henry Wang <Henry.Wang@arm.com>

Kind regards,
Henry

Re: [XEN][PATCH v7 01/19] common/device_tree: handle memory allocation failure in __unflatten_device_tree()

[Michal Orzel]

On 02/06/2023 02:48, Vikram Garhwal wrote:
> Change __unflatten_device_tree() return type to integer so it can propagate
> memory allocation failure. Add panic() in dt_unflatten_host_device_tree() for
> memory allocation failure during boot.
> 
> Fixes: fb97eb614acf ("xen/arm: Create a hierarchical device tree")
> 
NIT: no blank line here

> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> Reviewed-by: Henry Wang <Henry.Wang@arm.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>

~Michal

Re: [XEN][PATCH v7 02/19] common/device_tree.c: unflatten_device_tree() propagate errors

[Michal Orzel]

Title: s/unflatten_device_tree/__unflatten_device_tree/ or you mean to propagate
errors from unflatten_dt_node?

On 02/06/2023 02:48, Vikram Garhwal wrote:
> This will be useful in dynamic node programming when new dt nodes are unflattend
> during runtime. Invalid device tree node related errors should be propagated
> back to the caller.
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> Reviewed-by: Henry Wang <Henry.Wang@arm.com>
> ---
>  xen/common/device_tree.c | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
> index dfdb10e488..117ccccb09 100644
> --- a/xen/common/device_tree.c
> +++ b/xen/common/device_tree.c
> @@ -2108,6 +2108,9 @@ static int __init __unflatten_device_tree(const void *fdt,
>      /* First pass, scan for size */
>      start = ((unsigned long)fdt) + fdt_off_dt_struct(fdt);
>      size = unflatten_dt_node(fdt, 0, &start, NULL, NULL, 0);
> +    if ( !size )
> +        return -EINVAL;
> +
>      size = (size | 3) + 1;
>  
>      dt_dprintk("  size is %#lx allocating...\n", size);
> @@ -2125,11 +2128,19 @@ static int __init __unflatten_device_tree(const void *fdt,
>      start = ((unsigned long)fdt) + fdt_off_dt_struct(fdt);
>      unflatten_dt_node(fdt, mem, &start, NULL, &allnextp, 0);
>      if ( be32_to_cpup((__be32 *)start) != FDT_END )
> -        printk(XENLOG_WARNING "Weird tag at end of tree: %08x\n",
> +    {
> +        printk(XENLOG_ERR "Weird tag at end of tree: %08x\n",
>                    *((u32 *)start));
> +        return -EINVAL;
What about memory that we allocated? Shouldn't it be freed in case of these two errors?
For now it is called from boot so we do panic but later in this series this could result
in a memory leak. Am I correct?

> +    }
> +
>      if ( be32_to_cpu(((__be32 *)mem)[size / 4]) != 0xdeadbeef )
> -        printk(XENLOG_WARNING "End of tree marker overwritten: %08x\n",
> +    {
> +        printk(XENLOG_ERR "End of tree marker overwritten: %08x\n",
>                    be32_to_cpu(((__be32 *)mem)[size / 4]));
> +        return -EINVAL;
> +    }
> +
>      *allnextp = NULL;
>  
>      dt_dprintk(" <- unflatten_device_tree()\n");

~Michal

Re: [XEN][PATCH v7 04/19] common/device_tree: change __unflatten_device_tree() type

[Michal Orzel]

On 02/06/2023 02:48, Vikram Garhwal wrote:
> Following changes are done to __unflatten_device_tree():
>     1. __unflatten_device_tree() is renamed to unflatten_device_tree().
>     2. Remove __init and static function type.
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> Reviewed-by: Henry Wang <Henry.Wang@arm.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>

~Michal

Re: [XEN][PATCH v7 05/19] xen/arm: Add CONFIG_OVERLAY_DTB

[Michal Orzel]

On 02/06/2023 02:48, Vikram Garhwal wrote:
> Introduce a config option where the user can enable support for adding/removing
> device tree nodes using a device tree binary overlay.
> 
> Update SUPPORT.md and CHANGELOG.md to state the Device Tree Overlays support for
> Arm.
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>

~Michal

Re: [XEN][PATCH v7 08/19] xen/device-tree: Add device_tree_find_node_by_path() to find nodes in device tree

[Michal Orzel]

On 02/06/2023 03:52, Henry Wang wrote:
> 
> 
> Hi Vikram,
> 
>> -----Original Message-----
>> Subject: [XEN][PATCH v7 08/19] xen/device-tree: Add
>> device_tree_find_node_by_path() to find nodes in device tree
>>
>> Add device_tree_find_node_by_path() to find a matching node with path for
>> a
>> dt_device_node.
>>
>> Reason behind this function:
>>     Each time overlay nodes are added using .dtbo, a new fdt(memcpy of
>>     device_tree_flattened) is created and updated with overlay nodes. This
>>     updated fdt is further unflattened to a dt_host_new. Next, we need to find
>>     the overlay nodes in dt_host_new, find the overlay node's parent in dt_host
>>     and add the nodes as child under their parent in the dt_host. Thus we need
>>     this function to search for node in different unflattened device trees.
>>
>> Also, make dt_find_node_by_path() static inline.
>>
>> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
>>
>> ---
>> Changes from v6:
>>     Rename "dt_node" to "from"
>> ---
>>  xen/common/device_tree.c      |  6 ++++--
>>  xen/include/xen/device_tree.h | 18 ++++++++++++++++--
>>  2 files changed, 20 insertions(+), 4 deletions(-)
>>
>> diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
>> index 16b4b4e946..c5250a1644 100644
>> --- a/xen/common/device_tree.c
>> +++ b/xen/common/device_tree.c
>> @@ -358,11 +358,13 @@ struct dt_device_node
>> *dt_find_node_by_type(struct dt_device_node *from,
>>      return np;
>>  }
>>
>> -struct dt_device_node *dt_find_node_by_path(const char *path)
>> +struct dt_device_node *
>> +                    device_tree_find_node_by_path(struct dt_device_node *from,
>> +                                                  const char *path)
> 
> NIT: I found that the indentation here is a bit strange to me. I personally would
> write like:
> struct dt_device_node *
> device_tree_find_node_by_path(struct dt_device_node *from, char *path)
+1

With the indentation fixed:
Reviewed-by: Michal Orzel <michal.orzel@amd.com>

~Michal

Re: [XEN][PATCH v7 09/19] xen/iommu: Move spin_lock from iommu_dt_device_is_assigned to caller

[Michal Orzel]

On 02/06/2023 02:48, Vikram Garhwal wrote:
> Rename iommu_dt_device_is_assigned() to iommu_dt_device_is_assigned_locked().
> Remove static type so this can also be used by SMMU drivers to check if the
> device is being used before removing.
> 
> Moving spin_lock to caller was done to prevent the concurrent access to
> iommu_dt_device_is_assigned while doing add/remove/assign/deassign.

Would be nice to explain the need of putting a prototype into a private header.

> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> 
> ---
> Changes from v6:
>     Created a private header and moved iommu_dt_device_is_assigned() to header.
> ---
>  xen/drivers/passthrough/device_tree.c | 20 ++++++++++++++++----
>  xen/include/xen/iommu-private.h       | 27 +++++++++++++++++++++++++++
>  2 files changed, 43 insertions(+), 4 deletions(-)
>  create mode 100644 xen/include/xen/iommu-private.h
> 
> diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
> index 1c32d7b50c..52e370db01 100644
> --- a/xen/drivers/passthrough/device_tree.c
> +++ b/xen/drivers/passthrough/device_tree.c
> @@ -18,6 +18,7 @@
>  #include <xen/device_tree.h>
>  #include <xen/guest_access.h>
>  #include <xen/iommu.h>
> +#include <xen/iommu-private.h>
>  #include <xen/lib.h>
>  #include <xen/sched.h>
>  #include <xsm/xsm.h>
> @@ -83,16 +84,14 @@ fail:
>      return rc;
>  }
>  
> -static bool_t iommu_dt_device_is_assigned(const struct dt_device_node *dev)
I would write a comment that this function expects to be called with dtdevs_lock locked.

> +bool_t iommu_dt_device_is_assigned_locked(const struct dt_device_node *dev)
>  {
>      bool_t assigned = 0;
>  
>      if ( !dt_device_is_protected(dev) )
>          return 0;
>  
> -    spin_lock(&dtdevs_lock);
>      assigned = !list_empty(&dev->domain_list);
> -    spin_unlock(&dtdevs_lock);
>  
>      return assigned;
>  }
> @@ -213,27 +212,40 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
>          if ( (d && d->is_dying) || domctl->u.assign_device.flags )
>              break;
>  
> +        spin_lock(&dtdevs_lock);
> +
>          ret = dt_find_node_by_gpath(domctl->u.assign_device.u.dt.path,
>                                      domctl->u.assign_device.u.dt.size,
>                                      &dev);
>          if ( ret )
> +        {
> +            spin_unlock(&dtdevs_lock);
>              break;
> +        }
>  
>          ret = xsm_assign_dtdevice(XSM_HOOK, d, dt_node_full_name(dev));
>          if ( ret )
> +        {
> +            spin_unlock(&dtdevs_lock);
>              break;
> +        }
>  
>          if ( domctl->cmd == XEN_DOMCTL_test_assign_device )
>          {
> -            if ( iommu_dt_device_is_assigned(dev) )
> +
> +            if ( iommu_dt_device_is_assigned_locked(dev) )
>              {
>                  printk(XENLOG_G_ERR "%s already assigned.\n",
>                         dt_node_full_name(dev));
>                  ret = -EINVAL;
>              }
> +
> +            spin_unlock(&dtdevs_lock);
>              break;
>          }
>  
> +        spin_unlock(&dtdevs_lock);
> +
>          if ( d == dom_io )
>              return -EINVAL;
>  
> diff --git a/xen/include/xen/iommu-private.h b/xen/include/xen/iommu-private.h
> new file mode 100644
> index 0000000000..5615decaff
> --- /dev/null
> +++ b/xen/include/xen/iommu-private.h
> @@ -0,0 +1,27 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> + /*
incorrect indentation (<< 1)

> + * xen/iommu-private.h
> + *
> + *
> + * Copyright (C) 2023, Advanced Micro Devices, Inc. All Rights Reserved.
> + * Written by Vikram Garhwal <vikram.garhwal@amd.com>
I'm not sure if placing the copyright is appropriate, given a single prototype for a function
not really implemented by you with just spinlocks removed. But I might be wrong.

> + *
> + */
> +#ifndef __XEN_IOMMU_PRIVATE_H__
> +#define __XEN_IOMMU_PRIVATE_H__
> +
> +#ifdef CONFIG_HAS_DEVICE_TREE
> +#include <xen/device_tree.h>
> +bool_t iommu_dt_device_is_assigned_locked(const struct dt_device_node *dev);
> +#endif
> +
> +#endif /* __XEN_IOMMU_PRIVATE_H__ */
empty line here please

> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */

~Michal

Re: [XEN][PATCH v7 12/19] xen/smmu: Add remove_device callback for smmu_iommu ops

[Michal Orzel]

On 02/06/2023 02:48, Vikram Garhwal wrote:
> Add remove_device callback for removing the device entry from smmu-master using
> following steps:
> 1. Find if SMMU master exists for the device node.
> 2. Check if device is currently in use.
> 3. Remove the SMMU master.
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> Reviewed-by: Luca Fancellu <luca.fancellu@arm.com>
> Reviewed-by: Michal Orzel <michal.orzel@amd.com>
> ---
>  xen/drivers/passthrough/arm/smmu.c | 59 ++++++++++++++++++++++++++++++
>  1 file changed, 59 insertions(+)
> 
> diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
> index c37fa9af13..fdef6e7a7d 100644
> --- a/xen/drivers/passthrough/arm/smmu.c
> +++ b/xen/drivers/passthrough/arm/smmu.c
> @@ -44,6 +44,7 @@
>  #include <xen/irq.h>
>  #include <xen/lib.h>
>  #include <xen/list.h>
> +#include <xen/iommu-private.h>
Please list the headers in alphabetical order ('i' before 'l').

~Michal

Re: [XEN][PATCH v7 05/19] xen/arm: Add CONFIG_OVERLAY_DTB

[Jan Beulich]

On 02.06.2023 02:48, Vikram Garhwal wrote:
> Introduce a config option where the user can enable support for adding/removing
> device tree nodes using a device tree binary overlay.
> 
> Update SUPPORT.md and CHANGELOG.md to state the Device Tree Overlays support for
> Arm.
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> 
> ---
> Changes from v6:
>     Add CHANGELOG and SUPPORT.md entries.
> ---
>  CHANGELOG.md         | 2 ++
>  SUPPORT.md           | 6 ++++++
>  xen/arch/arm/Kconfig | 5 +++++
>  3 files changed, 13 insertions(+)
> 
> diff --git a/CHANGELOG.md b/CHANGELOG.md
> index 5bfd3aa5c0..a137fce576 100644
> --- a/CHANGELOG.md
> +++ b/CHANGELOG.md
> @@ -20,6 +20,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
>     - Bus-lock detection, used by Xen to mitigate (by rate-limiting) the system
>       wide impact of a guest misusing atomic instructions.
>   - xl/libxl can customize SMBIOS strings for HVM guests.
> + - On Arm, support for dynamic addition/removal of Xen device tree nodes using
> +   a device tree overlay binary(.dtbo).

May I suggest "..., experimental support ..." here?

Jan

Re: [XEN][PATCH v7 06/19] libfdt: Keep fdt functions after init for CONFIG_OVERLAY_DTB.

[Jan Beulich]

On 02.06.2023 02:48, Vikram Garhwal wrote:
> --- a/xen/common/libfdt/Makefile
> +++ b/xen/common/libfdt/Makefile
> @@ -1,7 +1,11 @@
>  include $(src)/Makefile.libfdt
>  
>  SECTIONS := text data $(SPECIAL_DATA_SECTIONS)
> +
> +# For CONFIG_OVERLAY_DTB, libfdt functionalities will be needed during runtime.
> +ifneq ($(CONFIG_OVERLAY_DTB),y)
>  OBJCOPYFLAGS := $(foreach s,$(SECTIONS),--rename-section .$(s)=.init.$(s))
> +endif

I have to admit that I find this odd: While it may be the least intrusive
change, why do the objcopy step at all in this case?

Jan

Re: [XEN][PATCH v7 09/19] xen/iommu: Move spin_lock from iommu_dt_device_is_assigned to caller

[Jan Beulich]

On 02.06.2023 02:48, Vikram Garhwal wrote:
> --- a/xen/drivers/passthrough/device_tree.c
> +++ b/xen/drivers/passthrough/device_tree.c
> @@ -18,6 +18,7 @@
>  #include <xen/device_tree.h>
>  #include <xen/guest_access.h>
>  #include <xen/iommu.h>
> +#include <xen/iommu-private.h>
>  #include <xen/lib.h>
>  #include <xen/sched.h>
>  #include <xsm/xsm.h>
> @@ -83,16 +84,14 @@ fail:
>      return rc;
>  }
>  
> -static bool_t iommu_dt_device_is_assigned(const struct dt_device_node *dev)
> +bool_t iommu_dt_device_is_assigned_locked(const struct dt_device_node *dev)
>  {
>      bool_t assigned = 0;
>  
>      if ( !dt_device_is_protected(dev) )
>          return 0;
>  
> -    spin_lock(&dtdevs_lock);
>      assigned = !list_empty(&dev->domain_list);
> -    spin_unlock(&dtdevs_lock);
>  
>      return assigned;
>  }
> @@ -213,27 +212,40 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
>          if ( (d && d->is_dying) || domctl->u.assign_device.flags )
>              break;
>  
> +        spin_lock(&dtdevs_lock);
> +
>          ret = dt_find_node_by_gpath(domctl->u.assign_device.u.dt.path,
>                                      domctl->u.assign_device.u.dt.size,
>                                      &dev);
>          if ( ret )
> +        {
> +            spin_unlock(&dtdevs_lock);
>              break;
> +        }
>  
>          ret = xsm_assign_dtdevice(XSM_HOOK, d, dt_node_full_name(dev));
>          if ( ret )
> +        {
> +            spin_unlock(&dtdevs_lock);
>              break;
> +        }
>  
>          if ( domctl->cmd == XEN_DOMCTL_test_assign_device )
>          {
> -            if ( iommu_dt_device_is_assigned(dev) )
> +
> +            if ( iommu_dt_device_is_assigned_locked(dev) )
>              {
>                  printk(XENLOG_G_ERR "%s already assigned.\n",
>                         dt_node_full_name(dev));
>                  ret = -EINVAL;
>              }
> +
> +            spin_unlock(&dtdevs_lock);
>              break;
>          }
>  
> +        spin_unlock(&dtdevs_lock);
> +
>          if ( d == dom_io )
>              return -EINVAL;
>  

I'm having trouble seeing how this patch can be correct: How do present
callers of iommu_dt_device_is_assigned() continue to build? I can only
guess that a new wrapper of this name is introduced in an earlier or
later patch, but thus breaking bisectability (either here or, if the
wrapper is added in an earlier patch, there).

> --- /dev/null
> +++ b/xen/include/xen/iommu-private.h

I don't think private headers belong in this directory. From this patch
alone (including its description) it also doesn't become clear how (and
hence from where) this is intended to be used, which makes it hard to
suggest an alternative.

> @@ -0,0 +1,27 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> + /*

Nit: Stray blank.

> + * xen/iommu-private.h
> + *
> + *

Nit: No double empty comment lines please.

> + * Copyright (C) 2023, Advanced Micro Devices, Inc. All Rights Reserved.
> + * Written by Vikram Garhwal <vikram.garhwal@amd.com>
> + *
> + */
> +#ifndef __XEN_IOMMU_PRIVATE_H__
> +#define __XEN_IOMMU_PRIVATE_H__
> +
> +#ifdef CONFIG_HAS_DEVICE_TREE
> +#include <xen/device_tree.h>

Why (for both of the lines)? All you need is a forward declaration of
struct dt_device_node.

> +bool_t iommu_dt_device_is_assigned_locked(const struct dt_device_node *dev);

Just bool please.

Jan

Re: [XEN][PATCH v7 10/19] xen/iommu: protect iommu_add_dt_device() with dtdevs_lock

[Jan Beulich]

On 02.06.2023 02:48, Vikram Garhwal wrote:
> @@ -189,6 +194,8 @@ int iommu_add_dt_device(struct dt_device_node *np)
>      if ( rc < 0 )
>          iommu_fwspec_free(dev);
>  
> +fail:
> +    spin_unlock(&dtdevs_lock);

Nit: Labels indented by at least on blank please (see ./CODING_STYLE).

Jan

RE: [XEN][PATCH v7 05/19] xen/arm: Add CONFIG_OVERLAY_DTB

[Henry Wang]

Hi Jan and Vikram,

> -----Original Message-----
> Subject: Re: [XEN][PATCH v7 05/19] xen/arm: Add CONFIG_OVERLAY_DTB
> 
> > diff --git a/CHANGELOG.md b/CHANGELOG.md
> > index 5bfd3aa5c0..a137fce576 100644
> > --- a/CHANGELOG.md
> > +++ b/CHANGELOG.md
> > @@ -20,6 +20,8 @@ The format is based on [Keep a
> Changelog](https://keepachangelog.com/en/1.0.0/)
> >     - Bus-lock detection, used by Xen to mitigate (by rate-limiting) the system
> >       wide impact of a guest misusing atomic instructions.
> >   - xl/libxl can customize SMBIOS strings for HVM guests.
> > + - On Arm, support for dynamic addition/removal of Xen device tree nodes
> using
> > +   a device tree overlay binary(.dtbo).
> 
> May I suggest "..., experimental support ..." here?

Great point! I agree using "experimental support" is better here.

@Vikram: Just to be clear, if you agree and change the wording following Jan's
suggestion, you can keep my acked-by tag for the changelog :))

Kind regards,
Henry

> 
> Jan

Re: [XEN][PATCH v7 11/19] xen/iommu: Introduce iommu_remove_dt_device()

[Jan Beulich]

On 02.06.2023 02:48, Vikram Garhwal wrote:
> --- a/xen/drivers/passthrough/device_tree.c
> +++ b/xen/drivers/passthrough/device_tree.c
> @@ -126,6 +126,47 @@ int iommu_release_dt_devices(struct domain *d)
>      return 0;
>  }
>  
> +int iommu_remove_dt_device(struct dt_device_node *np)
> +{
> +    const struct iommu_ops *ops = iommu_get_ops();
> +    struct device *dev = dt_to_dev(np);
> +    int rc;
> +
> +    if ( !ops )
> +        return -EOPNOTSUPP;
> +
> +    spin_lock(&dtdevs_lock);
> +
> +    if ( iommu_dt_device_is_assigned_locked(np) )
> +    {
> +        rc = -EBUSY;
> +        goto fail;
> +    }
> +
> +    /*
> +     * The driver which supports generic IOMMU DT bindings must have this
> +     * callback implemented.
> +     */
> +    if ( !ops->remove_device )
> +    {
> +        rc = -EOPNOTSUPP;
> +        goto fail;
> +    }
> +
> +    /*
> +     * Remove master device from the IOMMU if latter is present and available.
> +     * The driver is responsible for removing is_protected flag.
> +     */
> +    rc = ops->remove_device(0, dev);
> +
> +    if ( !rc )
> +        iommu_fwspec_free(dev);
> +
> +fail:
> +    spin_unlock(&dtdevs_lock);
> +    return rc;

Same remark regarding label indentation - please address throughout the
series.

Jan

Re: [XEN][PATCH v7 09/19] xen/iommu: Move spin_lock from iommu_dt_device_is_assigned to caller

[Jan Beulich]

On 02.06.2023 11:19, Jan Beulich wrote:
> On 02.06.2023 02:48, Vikram Garhwal wrote:
>> --- a/xen/drivers/passthrough/device_tree.c
>> +++ b/xen/drivers/passthrough/device_tree.c
>> @@ -18,6 +18,7 @@
>>  #include <xen/device_tree.h>
>>  #include <xen/guest_access.h>
>>  #include <xen/iommu.h>
>> +#include <xen/iommu-private.h>
>>  #include <xen/lib.h>
>>  #include <xen/sched.h>
>>  #include <xsm/xsm.h>
>> @@ -83,16 +84,14 @@ fail:
>>      return rc;
>>  }
>>  
>> -static bool_t iommu_dt_device_is_assigned(const struct dt_device_node *dev)
>> +bool_t iommu_dt_device_is_assigned_locked(const struct dt_device_node *dev)
>>  {
>>      bool_t assigned = 0;
>>  
>>      if ( !dt_device_is_protected(dev) )
>>          return 0;
>>  
>> -    spin_lock(&dtdevs_lock);
>>      assigned = !list_empty(&dev->domain_list);
>> -    spin_unlock(&dtdevs_lock);
>>  
>>      return assigned;
>>  }
>> @@ -213,27 +212,40 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
>>          if ( (d && d->is_dying) || domctl->u.assign_device.flags )
>>              break;
>>  
>> +        spin_lock(&dtdevs_lock);
>> +
>>          ret = dt_find_node_by_gpath(domctl->u.assign_device.u.dt.path,
>>                                      domctl->u.assign_device.u.dt.size,
>>                                      &dev);
>>          if ( ret )
>> +        {
>> +            spin_unlock(&dtdevs_lock);
>>              break;
>> +        }
>>  
>>          ret = xsm_assign_dtdevice(XSM_HOOK, d, dt_node_full_name(dev));
>>          if ( ret )
>> +        {
>> +            spin_unlock(&dtdevs_lock);
>>              break;
>> +        }
>>  
>>          if ( domctl->cmd == XEN_DOMCTL_test_assign_device )
>>          {
>> -            if ( iommu_dt_device_is_assigned(dev) )
>> +
>> +            if ( iommu_dt_device_is_assigned_locked(dev) )
>>              {
>>                  printk(XENLOG_G_ERR "%s already assigned.\n",
>>                         dt_node_full_name(dev));
>>                  ret = -EINVAL;
>>              }
>> +
>> +            spin_unlock(&dtdevs_lock);
>>              break;
>>          }
>>  
>> +        spin_unlock(&dtdevs_lock);
>> +
>>          if ( d == dom_io )
>>              return -EINVAL;
>>  
> 
> I'm having trouble seeing how this patch can be correct: How do present
> callers of iommu_dt_device_is_assigned() continue to build? I can only
> guess that a new wrapper of this name is introduced in an earlier or
> later patch, but thus breaking bisectability (either here or, if the
> wrapper is added in an earlier patch, there).

Oh, I'm sorry - looks like I overlooked that the 2nd hunk alters the
(sole) caller. Somehow I was under the (wrong) impression that both
hunks fiddle with the same function.

Jan

Re: [XEN][PATCH v7 15/19] xen/arm: Implement device tree node removal functionalities

[Jan Beulich]

On 02.06.2023 02:48, Vikram Garhwal wrote:
> --- a/xen/include/public/sysctl.h
> +++ b/xen/include/public/sysctl.h
> @@ -1057,6 +1057,25 @@ typedef struct xen_sysctl_cpu_policy xen_sysctl_cpu_policy_t;
>  DEFINE_XEN_GUEST_HANDLE(xen_sysctl_cpu_policy_t);
>  #endif
>  
> +#if defined(__arm__) || defined (__aarch64__)
> +/*
> + * XEN_SYSCTL_dt_overlay
> + * Performs addition/removal of device tree nodes under parent node using dtbo.
> + * This does in three steps:
> + *  - Adds/Removes the nodes from dt_host.
> + *  - Adds/Removes IRQ permission for the nodes.
> + *  - Adds/Removes MMIO accesses.
> + */
> +struct xen_sysctl_dt_overlay {
> +    XEN_GUEST_HANDLE_64(void) overlay_fdt;  /* IN: overlay fdt. */

If the comment is correct, then const_void please.

Jan

Re: [XEN][PATCH v7 14/19] common/device_tree: Add rwlock for dt_host

[Michal Orzel]

Hi Vikram,

On 02/06/2023 02:48, Vikram Garhwal wrote:
>  Dynamic programming ops will modify the dt_host and there might be other
>  function which are browsing the dt_host at the same time. To avoid the race
>  conditions, adding rwlock for browsing the dt_host during runtime.
> 
>  Reason behind adding rwlock instead of spinlock:
>     For now, dynamic programming is the sole modifier of dt_host in Xen during
>         run time. All other access functions like iommu_release_dt_device() are
>         just reading the dt_host during run-time. So, there is a need to protect
>         others from browsing the dt_host while dynamic programming is modifying
>         it. rwlock is better suitable for this task as spinlock won't be able to
>         differentiate between read and write access.
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> 
> ---
> Changes from v6:
>     Remove redundant "read_unlock(&dt_host->lock);" in the following case:
>          XEN_DOMCTL_deassign_device
> ---
>  xen/common/device_tree.c              |  4 ++++
>  xen/drivers/passthrough/device_tree.c | 15 +++++++++++++++
>  xen/include/xen/device_tree.h         |  6 ++++++
>  3 files changed, 25 insertions(+)
> 
> diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
> index c5250a1644..c8fcdf8fa1 100644
> --- a/xen/common/device_tree.c
> +++ b/xen/common/device_tree.c
> @@ -2146,7 +2146,11 @@ int unflatten_device_tree(const void *fdt, struct dt_device_node **mynodes)
>  
>      dt_dprintk(" <- unflatten_device_tree()\n");
>  
> +    /* Init r/w lock for host device tree. */
> +    rwlock_init(&dt_host->lock);
unflatten_device_tree() is called for dt_host and will also be used when adding a new dt overlay meaning that
rwlock_init for dt_host will be called multiple times (even if mynodes != &dt_host) which is rather strange.

> +
>      return 0;
> +
>  }
>  
>  static void dt_alias_add(struct dt_alias_prop *ap,
> diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
> index 301a5bcd97..f4d9deb624 100644
> --- a/xen/drivers/passthrough/device_tree.c
> +++ b/xen/drivers/passthrough/device_tree.c
> @@ -112,6 +112,8 @@ int iommu_release_dt_devices(struct domain *d)
>      if ( !is_iommu_enabled(d) )
>          return 0;
>  
> +    read_lock(&dt_host->lock);
> +
>      list_for_each_entry_safe(dev, _dev, &hd->dt_devices, domain_list)
>      {
>          rc = iommu_deassign_dt_device(d, dev);
> @@ -119,10 +121,14 @@ int iommu_release_dt_devices(struct domain *d)
>          {
>              dprintk(XENLOG_ERR, "Failed to deassign %s in domain %u\n",
>                      dt_node_full_name(dev), d->domain_id);
> +
> +            read_unlock(&dt_host->lock);
>              return rc;
>          }
>      }
>  
> +    read_unlock(&dt_host->lock);
> +
>      return 0;
>  }
>  
> @@ -246,6 +252,8 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
>      int ret;
>      struct dt_device_node *dev;
>  
> +    read_lock(&dt_host->lock);
> +
>      switch ( domctl->cmd )
>      {
>      case XEN_DOMCTL_assign_device:
> @@ -295,7 +303,10 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
>          spin_unlock(&dtdevs_lock);
>  
>          if ( d == dom_io )
> +        {
> +            read_unlock(&dt_host->lock);
>              return -EINVAL;
> +        }
>  
>          ret = iommu_add_dt_device(dev);
>          if ( ret < 0 )
> @@ -333,7 +344,10 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
>              break;
>  
>          if ( d == dom_io )
> +        {
> +            read_unlock(&dt_host->lock);
>              return -EINVAL;
> +        }
>  
>          ret = iommu_deassign_dt_device(d, dev);
>  
> @@ -348,5 +362,6 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
>          break;
>      }
>  
> +    read_unlock(&dt_host->lock);
>      return ret;
>  }
> diff --git a/xen/include/xen/device_tree.h b/xen/include/xen/device_tree.h
> index e239f7de26..dee40d2ea3 100644
> --- a/xen/include/xen/device_tree.h
> +++ b/xen/include/xen/device_tree.h
> @@ -18,6 +18,7 @@
>  #include <xen/string.h>
>  #include <xen/types.h>
>  #include <xen/list.h>
> +#include <xen/rwlock.h>
Sort alphabetically (despite list.h not being placed correctly)

>  
>  #define DEVICE_TREE_MAX_DEPTH 16
>  
> @@ -106,6 +107,11 @@ struct dt_device_node {
>      struct list_head domain_list;
>  
>      struct device dev;
> +
> +    /*
> +     * Lock that protects r/w updates to unflattened device tree i.e. dt_host.
> +     */
No need for multiline comment style

> +    rwlock_t lock;
>  };
>  
>  #define dt_to_dev(dt_node)  (&(dt_node)->dev)

~Michal

Re: [XEN][PATCH v7 15/19] xen/arm: Implement device tree node removal functionalities

[Michal Orzel]

Hi Vikram,

On 02/06/2023 02:48, Vikram Garhwal wrote:
> Introduce sysctl XEN_SYSCTL_dt_overlay to remove device-tree nodes added using
> device tree overlay.
> 
> xl dt-overlay remove file.dtbo:
>     Removes all the nodes in a given dtbo.
>     First, removes IRQ permissions and MMIO accesses. Next, it finds the nodes
>     in dt_host and delete the device node entries from dt_host.
> 
>     The nodes get removed only if it is not used by any of dom0 or domio.
> 
> Also, added overlay_track struct to keep the track of added node through device
> tree overlay. overlay_track has dt_host_new which is unflattened form of updated
> fdt and name of overlay nodes. When a node is removed, we also free the memory
> used by overlay_track for the particular overlay node.
> 
> Nested overlay removal is supported in sequential manner only i.e. if
> overlay_child nests under overlay_parent, it is assumed that user first removes
> overlay_child and then removes overlay_parent.
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> 
> ---
> Changes from v6:
>     Add explicit padding for xen_system_dt_overlay{}
>     Update license.
>     Rearrange xfree in dt_sysctl()
>     Update overlay_track struct comment with relevant message.
>     Fix missing xen/errno.h for builds without CONFIG_OVERLAY_DTB cases.
>     Fix header formatting.
> ---
>  xen/arch/arm/sysctl.c        |  16 +-
>  xen/common/Makefile          |   1 +
>  xen/common/dt-overlay.c      | 420 +++++++++++++++++++++++++++++++++++
>  xen/include/public/sysctl.h  |  24 ++
>  xen/include/xen/dt-overlay.h |  59 +++++
>  5 files changed, 519 insertions(+), 1 deletion(-)
>  create mode 100644 xen/common/dt-overlay.c
>  create mode 100644 xen/include/xen/dt-overlay.h
> 
> diff --git a/xen/arch/arm/sysctl.c b/xen/arch/arm/sysctl.c
> index b0a78a8b10..8b813c970f 100644
> --- a/xen/arch/arm/sysctl.c
> +++ b/xen/arch/arm/sysctl.c
> @@ -9,6 +9,7 @@
>  
>  #include <xen/types.h>
>  #include <xen/lib.h>
> +#include <xen/dt-overlay.h>
>  #include <xen/errno.h>
>  #include <xen/hypercall.h>
>  #include <public/sysctl.h>
> @@ -21,7 +22,20 @@ void arch_do_physinfo(struct xen_sysctl_physinfo *pi)
>  long arch_do_sysctl(struct xen_sysctl *sysctl,
>                      XEN_GUEST_HANDLE_PARAM(xen_sysctl_t) u_sysctl)
>  {
> -    return -ENOSYS;
> +    long ret;
> +
> +    switch ( sysctl->cmd )
> +    {
> +    case XEN_SYSCTL_dt_overlay:
> +        ret = dt_sysctl(&sysctl->u.dt_overlay);
> +        break;
> +
> +    default:
> +        ret = -ENOSYS;
> +        break;
> +    }
> +
> +    return ret;
>  }
>  
>  /*
> diff --git a/xen/common/Makefile b/xen/common/Makefile
> index 46049eac35..e7e96b1087 100644
> --- a/xen/common/Makefile
> +++ b/xen/common/Makefile
> @@ -8,6 +8,7 @@ obj-$(CONFIG_DEBUG_TRACE) += debugtrace.o
>  obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o
>  obj-$(CONFIG_IOREQ_SERVER) += dm.o
>  obj-y += domain.o
> +obj-$(CONFIG_OVERLAY_DTB) += dt-overlay.o
>  obj-y += event_2l.o
>  obj-y += event_channel.o
>  obj-y += event_fifo.o
> diff --git a/xen/common/dt-overlay.c b/xen/common/dt-overlay.c
> new file mode 100644
> index 0000000000..b2a7e441df
> --- /dev/null
> +++ b/xen/common/dt-overlay.c
> @@ -0,0 +1,420 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * xen/common/dt-overlay.c
> + *
> + * Device tree overlay support in Xen.
> + *
> + * Copyright (C) 2023, Advanced Micro Devices, Inc. All Rights Reserved.
> + * Written by Vikram Garhwal <vikram.garhwal@amd.com>
> + *
> + */
> +#include <asm/domain_build.h>
> +#include <xen/dt-overlay.h>
> +#include <xen/guest_access.h>
> +#include <xen/iocap.h>
> +#include <xen/xmalloc.h>
> +
> +static LIST_HEAD(overlay_tracker);
> +static DEFINE_SPINLOCK(overlay_lock);
> +
> +/* Find last descendants of the device_node. */
> +static struct dt_device_node *
> +                find_last_descendants_node(struct dt_device_node *device_node)
I make this remark at every single revision. Please do the indentation correctly, i.e.:
static struct dt_device_node *
find_last_descendants_node(struct dt_device_node *device_node)

Also, I think device_node can be const.

> +{
> +    struct dt_device_node *child_node;
> +
> +    for ( child_node = device_node->child; child_node->sibling != NULL;
> +          child_node = child_node->sibling );
> +
> +    /* If last child_node also have children. */
> +    if ( child_node->child )
> +        child_node = find_last_descendants_node(child_node);
> +
> +    return child_node;
> +}
> +
> +static int dt_overlay_remove_node(struct dt_device_node *device_node)
> +{
> +    struct dt_device_node *np;
> +    struct dt_device_node *parent_node;
> +    struct dt_device_node *device_node_last_descendant = device_node->child;
> +
> +    parent_node = device_node->parent;
> +
> +    if ( parent_node == NULL )
> +    {
> +        dt_dprintk("%s's parent node not found\n", device_node->name);
> +        return -EFAULT;
> +    }
> +
> +    np = parent_node->child;
> +
> +    if ( np == NULL )
> +    {
> +        dt_dprintk("parent node %s's not found\n", parent_node->name);
> +        return -EFAULT;
> +    }
> +
> +    /* If node to be removed is only child node or first child. */
> +    if ( !dt_node_cmp(np->full_name, device_node->full_name) )
> +    {
> +        parent_node->child = np->sibling;
> +
> +        /*
> +         * Iterate over all child nodes of device_node. Given that we are
> +         * removing parent node, we need to remove all it's descendants too.
> +         */
> +        if ( device_node_last_descendant )
> +        {
> +            device_node_last_descendant =
> +                                        find_last_descendants_node(device_node);
> +            parent_node->allnext = device_node_last_descendant->allnext;
> +        }
> +        else
> +            parent_node->allnext = np->allnext;
> +
> +        return 0;
> +    }
> +
> +    for ( np = parent_node->child; np->sibling != NULL; np = np->sibling )
> +    {
> +        if ( !dt_node_cmp(np->sibling->full_name, device_node->full_name) )
> +        {
> +            /* Found the node. Now we remove it. */
> +            np->sibling = np->sibling->sibling;
> +
> +            if ( np->child )
> +                np = find_last_descendants_node(np);
> +
> +            /*
> +             * Iterate over all child nodes of device_node. Given that we are
> +             * removing parent node, we need to remove all it's descendants too.
> +             */
> +            if ( device_node_last_descendant )
> +                device_node_last_descendant =
> +                                        find_last_descendants_node(device_node);
> +
> +            if ( device_node_last_descendant )
> +                np->allnext = device_node_last_descendant->allnext;
> +            else
> +                np->allnext = np->allnext->allnext;
> +
> +            break;
> +        }
> +    }
> +
> +    return 0;
> +}
> +
> +/* Basic sanity check for the dtbo tool stack provided to Xen. */
> +static int check_overlay_fdt(const void *overlay_fdt, uint32_t overlay_fdt_size)
> +{
> +    if ( (fdt_totalsize(overlay_fdt) != overlay_fdt_size) ||
> +          fdt_check_header(overlay_fdt) )
> +    {
> +        printk(XENLOG_ERR "The overlay FDT is not a valid Flat Device Tree\n");
> +        return -EINVAL;
> +    }
> +
> +    return 0;
> +}
> +
> +/* Count number of nodes till one level of __overlay__ tag. */
> +static unsigned int overlay_node_count(const void *overlay_fdt)
> +{
> +    unsigned int num_overlay_nodes = 0;
> +    int fragment;
> +
> +    fdt_for_each_subnode(fragment, overlay_fdt, 0)
> +    {
> +        int subnode;
> +        int overlay;
> +
> +        overlay = fdt_subnode_offset(overlay_fdt, fragment, "__overlay__");
> +
> +        /*
> +         * overlay value can be < 0. But fdt_for_each_subnode() loop checks for
> +         * overlay >= 0. So, no need for a overlay>=0 check here.
> +         */
> +        fdt_for_each_subnode(subnode, overlay_fdt, overlay)
> +        {
> +            num_overlay_nodes++;
> +        }
> +    }
> +
> +    return num_overlay_nodes;
> +}
> +
> +static int handle_remove_irq_iommu(struct dt_device_node *device_node)
> +{
> +    int rc = 0;
> +    struct domain *d = hardware_domain;
> +    domid_t domid;
> +    unsigned int naddr, len;
> +    unsigned int i, nirq;
> +
> +    domid = dt_device_used_by(device_node);
> +
> +    dt_dprintk("Checking if node %s is used by any domain\n",
> +               device_node->full_name);
> +
> +    /* Remove the node if only it's assigned to domain 0 or domain io. */
> +    if ( domid != 0 && domid != DOMID_IO )
> +    {
> +        printk(XENLOG_ERR "Device %s is being used by domain %u. Removing nodes failed\n",
> +               device_node->full_name, domid);
> +        return -EINVAL;
> +    }
> +
> +    dt_dprintk("Removing node: %s\n", device_node->full_name);
> +
> +    nirq = dt_number_of_irq(device_node);
> +
> +    /* Remove IRQ permission */
> +    for ( i = 0; i < nirq; i++ )
> +    {
> +        rc = platform_get_irq(device_node, i);
> +        if ( rc < 0 )
> +        {
> +            printk(XENLOG_ERR "Failed to get IRQ num for device node %s\n",
> +                   device_node->full_name);
> +            return -EINVAL;
> +        }
> +
> +        if ( irq_access_permitted(d, rc) == false )
> +        {
> +            printk(XENLOG_ERR "IRQ %d is not routed to domain %u\n", rc,
> +                   domid);
> +            return -EINVAL;
> +        }
> +        /*
> +         * TODO: We don't handle shared IRQs for now. So, it is assumed that
> +         * the IRQs was not shared with another devices.
> +         */
> +        rc = irq_deny_access(d, rc);
> +        if ( rc )
> +        {
> +            printk(XENLOG_ERR "unable to revoke access for irq %u for %s\n",
> +                   i, device_node->full_name);
> +            return rc;
> +        }
> +    }
> +
> +    /* Check if iommu property exists. */
> +    if ( dt_get_property(device_node, "iommus", &len) )
> +    {
> +        rc = iommu_remove_dt_device(device_node);
> +        if ( rc != 0 && rc != -ENXIO )
> +            return rc;
> +    }
> +
> +    naddr = dt_number_of_address(device_node);
> +
> +    /* Remove mmio access. */
> +    for ( i = 0; i < naddr; i++ )
> +    {
> +        uint64_t addr, size;
Take a look at recent patches from Ayan that were merged. These shall be paddr_t and ...

> +
> +        rc = dt_device_get_address(device_node, i, &addr, &size);
... here you should use dt_device_get_paddr().

> +        if ( rc )
> +        {
> +            printk(XENLOG_ERR "Unable to retrieve address %u for %s\n",
> +                   i, dt_node_full_name(device_node));
> +            return rc;
> +        }
> +
> +        rc = iomem_deny_access(d, paddr_to_pfn(addr),
> +                               paddr_to_pfn(PAGE_ALIGN(addr + size - 1)));
> +        if ( rc )
> +        {
> +            printk(XENLOG_ERR "Unable to remove dom%d access to"
> +                   " 0x%"PRIx64" - 0x%"PRIx64"\n",
> +                   d->domain_id,
> +                   addr & PAGE_MASK, PAGE_ALIGN(addr + size) - 1);
> +            return rc;
> +        }
> +
> +    }
> +
> +    return rc;
> +}
> +
> +/* Removes all descendants of the given node. */
> +static int remove_all_descendant_nodes(struct dt_device_node *device_node)
device_node can be const I think

> +{
> +    int rc = 0;
> +    struct dt_device_node *child_node;
> +
> +    for ( child_node = device_node->child; child_node != NULL;
> +         child_node = child_node->sibling )
> +    {
> +        if ( child_node->child )
> +            remove_all_descendant_nodes(child_node);
> +
> +        rc = handle_remove_irq_iommu(child_node);
> +        if ( rc )
> +            return rc;
> +    }
> +
> +    return rc;
> +}
> +
> +/* Remove nodes from dt_host. */
> +static int remove_nodes(const struct overlay_track *tracker)
> +{
> +    int rc = 0;
> +    struct dt_device_node *overlay_node;
> +    unsigned int j;
> +
> +    for ( j = 0; j < tracker->num_nodes; j++ )
> +    {
> +        overlay_node = (struct dt_device_node *)tracker->nodes_address[j];
> +        if ( overlay_node == NULL )
> +        {
> +            printk(XENLOG_ERR "Device %s is not present in the tree. Removing nodes failed\n",
> +                   overlay_node->full_name);
> +            return -EINVAL;
> +        }
> +
> +        rc = remove_all_descendant_nodes(overlay_node);
Why are you not checking rc for an error?

> +
> +        /* All children nodes are unmapped. Now remove the node itself. */
> +        rc = handle_remove_irq_iommu(overlay_node);
> +        if ( rc )
> +            return rc;
> +
> +        read_lock(&dt_host->lock);
> +
> +        rc = dt_overlay_remove_node(overlay_node);
> +        if ( rc )
> +        {
> +            read_unlock(&dt_host->lock);
> +
remove this empty line

> +            return rc;
> +        }
> +
> +        read_unlock(&dt_host->lock);
> +    }
> +
> +    return rc;
> +}
> +
> +/*
> + * First finds the device node to remove. Check if the device is being used by
> + * any dom and finally remove it from dt_host. IOMMU is already being taken care
> + * while destroying the domain.
> + */
> +static long handle_remove_overlay_nodes(void *overlay_fdt,
overlay_fdt can be const I think

> +                                        uint32_t overlay_fdt_size)
> +{
> +    int rc;
> +    struct overlay_track *entry, *temp, *track;
> +    bool found_entry = false;
> +
> +    rc = check_overlay_fdt(overlay_fdt, overlay_fdt_size);
> +    if ( rc )
> +        return rc;
> +
> +    if ( overlay_node_count(overlay_fdt) == 0 )
> +        return -EINVAL;
> +
> +    spin_lock(&overlay_lock);
> +
> +    /*
> +     * First check if dtbo is correct i.e. it should one of the dtbo which was
> +     * used when dynamically adding the node.
> +     * Limitation: Cases with same node names but different property are not
> +     * supported currently. We are relying on user to provide the same dtbo
> +     * as it was used when adding the nodes.
> +     */
> +    list_for_each_entry_safe( entry, temp, &overlay_tracker, entry )
> +    {
> +        if ( memcmp(entry->overlay_fdt, overlay_fdt, overlay_fdt_size) == 0 )
> +        {
> +            track = entry;
> +            found_entry = true;
> +            break;
> +        }
> +    }
> +
> +    if ( found_entry == false )
> +    {
> +        rc = -EINVAL;
> +
> +        printk(XENLOG_ERR "Cannot find any matching tracker with input dtbo."
> +               " Removing nodes is supported only for prior added dtbo.\n");
> +        goto out;
> +
> +    }
> +
> +    rc = remove_nodes(entry);
> +    if ( rc )
> +    {
> +        printk(XENLOG_ERR "Removing node failed\n");
> +        goto out;
> +    }
> +
> +    list_del(&entry->entry);
> +
> +    xfree(entry->dt_host_new);
> +    xfree(entry->fdt);
> +    xfree(entry->overlay_fdt);
> +
> +    xfree(entry->nodes_address);
> +
> +    xfree(entry);
> +
> +out:
> +    spin_unlock(&overlay_lock);
> +    return rc;
> +}
> +
> +long dt_sysctl(struct xen_sysctl_dt_overlay *op)
> +{
> +    long ret;
> +    void *overlay_fdt;
> +
> +    if ( op->overlay_fdt_size == 0 || op->overlay_fdt_size > KB(500) )
It would be nice to enclose the expressions in parentheses

> +        return -EINVAL;
> +
> +    if ( op->pad[0] || op->pad[1] || op->pad[2] )
> +        return -EINVAL;
> +
> +    overlay_fdt = xmalloc_bytes(op->overlay_fdt_size);
> +
> +    if ( overlay_fdt == NULL )
> +        return -ENOMEM;
> +
> +    ret = copy_from_guest(overlay_fdt, op->overlay_fdt, op->overlay_fdt_size);
> +    if ( ret )
> +    {
> +        gprintk(XENLOG_ERR, "copy from guest failed\n");
> +        xfree(overlay_fdt);
> +
> +        return -EFAULT;
> +    }
> +
> +    switch ( op->overlay_op )
> +    {
> +    case XEN_SYSCTL_DT_OVERLAY_REMOVE:
> +        ret = handle_remove_overlay_nodes(overlay_fdt, op->overlay_fdt_size);
> +
remove empty line

> +        break;
> +
> +    default:
> +        break;
> +    }
> +
> +    xfree(overlay_fdt);
> +
> +    return ret;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
> index 2b24d6bfd0..ff54607617 100644
> --- a/xen/include/public/sysctl.h
> +++ b/xen/include/public/sysctl.h
> @@ -1057,6 +1057,25 @@ typedef struct xen_sysctl_cpu_policy xen_sysctl_cpu_policy_t;
>  DEFINE_XEN_GUEST_HANDLE(xen_sysctl_cpu_policy_t);
>  #endif
>  
> +#if defined(__arm__) || defined (__aarch64__)
> +/*
> + * XEN_SYSCTL_dt_overlay
> + * Performs addition/removal of device tree nodes under parent node using dtbo.
> + * This does in three steps:
> + *  - Adds/Removes the nodes from dt_host.
> + *  - Adds/Removes IRQ permission for the nodes.
> + *  - Adds/Removes MMIO accesses.
> + */
> +struct xen_sysctl_dt_overlay {
> +    XEN_GUEST_HANDLE_64(void) overlay_fdt;  /* IN: overlay fdt. */
> +    uint32_t overlay_fdt_size;              /* IN: Overlay dtb size. */
> +#define XEN_SYSCTL_DT_OVERLAY_ADD                   1
> +#define XEN_SYSCTL_DT_OVERLAY_REMOVE                2
> +    uint8_t overlay_op;                     /* IN: Add or remove. */
> +    uint8_t pad[3];                         /* IN: Must be zero. */
> +};
> +#endif
> +
>  struct xen_sysctl {
>      uint32_t cmd;
>  #define XEN_SYSCTL_readconsole                    1
> @@ -1087,6 +1106,7 @@ struct xen_sysctl {
>  #define XEN_SYSCTL_livepatch_op                  27
>  /* #define XEN_SYSCTL_set_parameter              28 */
>  #define XEN_SYSCTL_get_cpu_policy                29
> +#define XEN_SYSCTL_dt_overlay                    30
>      uint32_t interface_version; /* XEN_SYSCTL_INTERFACE_VERSION */
>      union {
>          struct xen_sysctl_readconsole       readconsole;
> @@ -1117,6 +1137,10 @@ struct xen_sysctl {
>  #if defined(__i386__) || defined(__x86_64__)
>          struct xen_sysctl_cpu_policy        cpu_policy;
>  #endif
> +
> +#if defined(__arm__) || defined (__aarch64__)
> +        struct xen_sysctl_dt_overlay        dt_overlay;
> +#endif
>          uint8_t                             pad[128];
>      } u;
>  };
> diff --git a/xen/include/xen/dt-overlay.h b/xen/include/xen/dt-overlay.h
> new file mode 100644
> index 0000000000..43fa5a02a0
> --- /dev/null
> +++ b/xen/include/xen/dt-overlay.h
> @@ -0,0 +1,59 @@
> + /* SPDX-License-Identifier: GPL-2.0-only */
> + /*
> + * xen/dt-overlay.h
> + *
> + * Device tree overlay support in Xen.
> + *
> + * Copyright (C) 2023, Advanced Micro Devices, Inc. All Rights Reserved.
> + * Written by Vikram Garhwal <vikram.garhwal@amd.com>
> + *
> + */
> +#ifndef __XEN_DT_OVERLAY_H__
> +#define __XEN_DT_OVERLAY_H__
> +
> +#include <xen/list.h>
> +#include <xen/libfdt/libfdt.h>
> +#include <xen/device_tree.h>
> +#include <xen/rangeset.h>
Sort alphabetically please

> +
> +/*
> + * overlay_track describes information about added nodes through dtbo.
> + * @entry: List pointer.
> + * @dt_host_new: Pointer to the updated dt_host_new which is unflattened from
> +    the 'updated fdt'.
> + * @fdt: Stores the fdt.
> + * @overlay_fdt: Stores a copy of input overlay_fdt.
> + * @nodes_address: Stores each overlay_node's address.
> + * @num_nodes: Total number of nodes in overlay dtb.
> + */
> +struct overlay_track {
> +    struct list_head entry;
> +    struct dt_device_node *dt_host_new;
> +    void *fdt;
> +    void *overlay_fdt;
> +    unsigned long *nodes_address;
> +    unsigned int num_nodes;
> +};
> +
> +struct xen_sysctl_dt_overlay;
> +
> +#ifdef CONFIG_OVERLAY_DTB
> +long dt_sysctl(struct xen_sysctl_dt_overlay *op);
> +#else
> +#include <xen/errno.h>
> +static inline long dt_sysctl(struct xen_sysctl_dt_overlay *op)
> +{
> +    return -EOPNOTSUPP;
> +}
> +#endif
> +
> +#endif /* __XEN_DT_OVERLAY_H__ */
empty line here

> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */

~Michal

Re: [XEN][PATCH v7 16/19] xen/arm: Implement device tree node addition functionalities

[Michal Orzel]

Hi Vikram,

On 02/06/2023 02:48, Vikram Garhwal wrote:
> Update sysctl XEN_SYSCTL_dt_overlay to enable support for dtbo nodes addition
> using device tree overlay.
> 
> xl dt-overlay add file.dtbo:
>     Each time overlay nodes are added using .dtbo, a new fdt(memcpy of
>     device_tree_flattened) is created and updated with overlay nodes. This
>     updated fdt is further unflattened to a dt_host_new. Next, it checks if any
>     of the overlay nodes already exists in the dt_host. If overlay nodes doesn't
>     exist then find the overlay nodes in dt_host_new, find the overlay node's
>     parent in dt_host and add the nodes as child under their parent in the
>     dt_host. The node is attached as the last node under target parent.
> 
>     Finally, add IRQs, add device to IOMMUs, set permissions and map MMIO for the
>     overlay node.
> 
> When a node is added using overlay, a new entry is allocated in the
> overlay_track to keep the track of memory allocation due to addition of overlay
> node. This is helpful for freeing the memory allocated when a device tree node
> is removed.
> 
> The main purpose of this to address first part of dynamic programming i.e.
> making xen aware of new device tree node which means updating the dt_host with
> overlay node information. Here we are adding/removing node from dt_host, and
> checking/setting IOMMU and IRQ permission but never mapping them to any domain.
> Right now, mapping/Un-mapping will happen only when a new domU is
> created/destroyed using "xl create".
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> 
> ---
> Changes from v6:
>     Fix comment style and add comment regarding false flag in irq mapping.
>     Move malloc for nodes_full_path to handle_add_overlay_nodes.
>     Move node_num define to start of overlay_get_nodes_info().
>     Remove "domain *d" from handle_add_irq_iommu().
>     Fix error handling for handle_add_irq_iommu().
>     Split handle_add_overlay_nodes to two functions.
>     Create a separate function for freeing nodes_full_path.
>     Fix xfree for dt_sysctl.
> ---
>  xen/common/dt-overlay.c | 533 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 533 insertions(+)
> 
> diff --git a/xen/common/dt-overlay.c b/xen/common/dt-overlay.c
> index b2a7e441df..12b6b010ef 100644
> --- a/xen/common/dt-overlay.c
> +++ b/xen/common/dt-overlay.c
> @@ -33,6 +33,25 @@ static struct dt_device_node *
>      return child_node;
>  }
>  
> +/*
> + * Returns next node to the input node. If node has children then return
> + * last descendant's next node.
> +*/
> +static struct dt_device_node *
> +dt_find_next_node(struct dt_device_node *dt, const struct dt_device_node *node)
> +{
> +    struct dt_device_node *np;
> +
> +    dt_for_each_device_node(dt, np)
> +        if ( np == node )
> +            break;
> +
> +    if ( np->child )
> +        np = find_last_descendants_node(np);
> +
> +    return np->allnext;
> +}
> +
>  static int dt_overlay_remove_node(struct dt_device_node *device_node)
>  {
>      struct dt_device_node *np;
> @@ -106,6 +125,78 @@ static int dt_overlay_remove_node(struct dt_device_node *device_node)
>      return 0;
>  }
>  
> +static int dt_overlay_add_node(struct dt_device_node *device_node,
> +                               const char *parent_node_path)
> +{
> +    struct dt_device_node *parent_node;
> +    struct dt_device_node *next_node;
> +
> +    parent_node = dt_find_node_by_path(parent_node_path);
> +
> +    if ( parent_node == NULL )
> +    {
> +        dt_dprintk("Parent node %s not found. Overlay node will not be added\n",
> +                   parent_node_path);
> +        return -EINVAL;
> +    }
> +
> +    /* If parent has no child. */
> +    if ( parent_node->child == NULL )
> +    {
> +        next_node = parent_node->allnext;
> +        device_node->parent = parent_node;
> +        parent_node->allnext = device_node;
> +        parent_node->child = device_node;
> +    }
> +    else
> +    {
> +        struct dt_device_node *np;
> +        /*
> +         * If parent has at least one child node.
> +         * Iterate to the last child node of parent.
> +         */
> +        for ( np = parent_node->child; np->sibling != NULL; np = np->sibling );
> +
> +        /* Iterate over all child nodes of np node. */
> +        if ( np->child )
> +        {
> +            struct dt_device_node *np_last_descendant;
> +
> +            np_last_descendant = find_last_descendants_node(np);
> +
> +            next_node = np_last_descendant->allnext;
> +            np_last_descendant->allnext = device_node;
> +        }
> +        else
> +        {
> +            next_node = np->allnext;
> +            np->allnext = device_node;
> +        }
> +
> +        device_node->parent = parent_node;
> +        np->sibling = device_node;
> +        np->sibling->sibling = NULL;
> +    }
> +
> +    /* Iterate over all child nodes of device_node to add children too. */
> +    if ( device_node->child )
> +    {
> +        struct dt_device_node *device_node_last_descendant;
> +
> +        device_node_last_descendant = find_last_descendants_node(device_node);
> +
> +        /* Plug next_node at the end of last children of device_node. */
> +        device_node_last_descendant->allnext = next_node;
> +    }
> +    else
> +    {
> +        /* Now plug next_node at the end of device_node. */
> +        device_node->allnext = next_node;
> +    }
> +
> +    return 0;
> +}
> +
>  /* Basic sanity check for the dtbo tool stack provided to Xen. */
>  static int check_overlay_fdt(const void *overlay_fdt, uint32_t overlay_fdt_size)
>  {
> @@ -145,6 +236,76 @@ static unsigned int overlay_node_count(const void *overlay_fdt)
>      return num_overlay_nodes;
>  }
>  
> +/*
> + * overlay_get_nodes_info gets full name with path for all the nodes which
> + * are in one level of __overlay__ tag. This is useful when checking node for
> + * duplication i.e. dtbo tries to add nodes which already exists in device tree.
> + */
> +static int overlay_get_nodes_info(const void *fdto, char **nodes_full_path)
> +{
> +    int fragment;
> +    unsigned int node_num = 0;
> +
> +    fdt_for_each_subnode(fragment, fdto, 0)
> +    {
> +        int target;
> +        int overlay;
> +        int subnode;
> +        const char *target_path;
> +
> +        target = fdt_overlay_target_offset(device_tree_flattened, fdto,
> +                                           fragment, &target_path);
> +        if ( target < 0 )
> +            return target;
> +
> +        if ( target_path == NULL )
> +            return -EINVAL;
> +
> +        overlay = fdt_subnode_offset(fdto, fragment, "__overlay__");
> +
> +        /*
> +         * overlay value can be < 0. But fdt_for_each_subnode() loop checks for
> +         * overlay >= 0. So, no need for a overlay>=0 check here.
> +         */
> +        fdt_for_each_subnode(subnode, fdto, overlay)
> +        {
> +            const char *node_name = NULL;
> +            int node_name_len;
> +            unsigned int target_path_len = strlen(target_path);
> +            unsigned int node_full_name_len;
> +
> +            node_name = fdt_get_name(fdto, subnode, &node_name_len);
> +
> +            if ( node_name == NULL )
> +                return node_name_len;
> +
> +            /*
> +             * Magic number 2 is for adding '/' and '\0'. This is done to keep
> +             * the node_full_path in the correct full node name format.
> +             */
> +            node_full_name_len = target_path_len + node_name_len + 2;
> +
> +            nodes_full_path[node_num] = xmalloc_bytes(node_full_name_len);
> +
> +            if ( nodes_full_path[node_num] == NULL )
> +                return -ENOMEM;
> +
> +            memcpy(nodes_full_path[node_num], target_path, target_path_len);
> +
> +            nodes_full_path[node_num][target_path_len] = '/';
> +
> +            memcpy(nodes_full_path[node_num] + target_path_len + 1,
> +                    node_name, node_name_len);
> +
> +            nodes_full_path[node_num][node_full_name_len - 1] = '\0';
> +
> +            node_num++;
> +        }
> +    }
> +
> +    return 0;
> +}
> +
>  static int handle_remove_irq_iommu(struct dt_device_node *device_node)
>  {
>      int rc = 0;
> @@ -369,6 +530,373 @@ out:
>      return rc;
>  }
>  
> +/*
> + * Handles IRQ and IOMMU mapping for the overlay_node and all descendants of the
> + * overlay_node.
> + */
> +static int handle_add_irq_iommu(struct dt_device_node *overlay_node)
> +{
> +    int rc;
> +    unsigned int naddr, i, len;
> +    struct dt_device_node *np;
> +
> +    /*
> +     * First let's handle the interrupts.
> +     * For now, need_mapping is set to false which means it will only permit IRQ
> +     * access to hardware_domain using irq_permit_access() but will never route
> +     * as route_irq_to_guest() will not be called with false flag.
> +     */
> +    rc = handle_device_interrupts(hardware_domain, overlay_node, false);
> +    if ( rc < 0 )
> +    {
> +        printk(XENLOG_ERR "Failed to retrieve interrupts configuration\n");
> +        return rc;
> +    }
> +
> +    /* Check if iommu property exists. */
> +    if ( dt_get_property(overlay_node, "iommus", &len) )
> +    {
> +        /* Add device to IOMMUs. */
> +        rc = iommu_add_dt_device(overlay_node);
> +        if ( rc < 0 )
> +        {
> +            printk(XENLOG_ERR "Failed to add %s to the IOMMU\n",
> +                   dt_node_full_name(overlay_node));
> +            return rc;
> +        }
> +    }
> +
> +    /* Set permissions. */
> +    naddr = dt_number_of_address(overlay_node);
> +
> +    dt_dprintk("%s naddr = %u\n", dt_node_full_name(overlay_node), naddr);
> +
> +    /* Give permission to map MMIOs */
> +    for ( i = 0; i < naddr; i++ )
> +    {
> +        uint64_t addr, size;
These shall be paddr_t and ...

> +
> +        /*
> +         * For now, we skip_mapping which means it will only permit iomem access
> +         * to hardware_domain using iomem_permit_access() but will never map as
> +         * map_range_p2mt() will not be called.
> +         */
> +        struct map_range_data mr_data = { .d = hardware_domain,
> +                                          .p2mt = p2m_mmio_direct_c,
> +                                          .skip_mapping = true
> +                                        };
> +
> +        rc = dt_device_get_address(overlay_node, i, &addr, &size);
... here you should use dt_device_get_paddr()

> +        if ( rc )
> +        {
> +            printk(XENLOG_ERR "Unable to retrieve address %u for %s\n",
> +                   i, dt_node_full_name(overlay_node));
> +            return rc;
> +        }
> +
> +        rc = map_range_to_domain(overlay_node, addr, size, &mr_data);
> +        if ( rc )
> +            return rc;
> +    }
> +
> +    /* Map IRQ and IOMMU for overlay_node's children. */
> +    for ( np = overlay_node->child; np != NULL; np = np->sibling )
> +    {
> +        rc = handle_add_irq_iommu(np);
> +        if ( rc )
> +        {
> +            printk(XENLOG_ERR "Adding IRQ and IOMMU failed\n");
> +            return rc;
> +        }
> +    }
> +
> +    return rc;
> +}
> +
> +static void free_nodes_full_path(int num_nodes, char **nodes_full_path)
num_nodes shall be unsigned

> +{
> +    int i;
unsigned

> +
> +    if ( nodes_full_path != NULL )
> +    {
> +        for ( i = 0; i < num_nodes && nodes_full_path[i] != NULL;
> +              i++ )
> +        {
> +            xfree(nodes_full_path[i]);
> +        }
> +        xfree(nodes_full_path);
> +    }
> +
> +    return;
Why explicit return from void function?
Also, you could get rid of the {} by doing:
    if ( nodes_full_path == NULL )
        return;

    for ( i = 0; i < num_nodes && nodes_full_path[i] != NULL; i++ )
        xfree(nodes_full_path[i]);

    xfree(nodes_full_path);

> +}
> +
> +static long add_nodes(struct overlay_track *tr, char **nodes_full_path)
> +
> +{
> +    int j, rc;
j shall be unsigned as you use it as an index to nodes_full_path and num_nodes of overlay_track is unsigned.

> +    struct dt_device_node *overlay_node;
> +
> +    for ( j = 0; j < tr->num_nodes; j++ )
> +    {
> +        struct dt_device_node *prev_node, *next_node;
> +
> +        dt_dprintk("Adding node: %s\n", nodes_full_path[j]);
> +
> +        /* Find the newly added node in tr->dt_host_new by it's full path. */
> +        overlay_node = device_tree_find_node_by_path(tr->dt_host_new,
> +                                                     nodes_full_path[j]);
> +        if ( overlay_node == NULL )
> +        {
> +            /* Sanity check. But code will never come here. */
> +            ASSERT_UNREACHABLE();
> +            return -EFAULT;
> +        }
> +
> +        /*
> +         * Find previous and next node to overlay_node in dt_host_new. We will
> +         * need these nodes to fix the dt_host_new mapping. When overlay_node is
> +         * take out of dt_host_new tree and added to dt_host, link between
> +         * previous node and next_node is broken. We will need to refresh
> +         * dt_host_new with correct linking for any other overlay nodes
> +         * extraction in future.
> +         */
> +        dt_for_each_device_node(tr->dt_host_new, prev_node)
> +            if ( prev_node->allnext == overlay_node )
> +                break;
> +
> +        next_node = dt_find_next_node(tr->dt_host_new, overlay_node);
> +
> +        read_lock(&dt_host->lock);
> +
> +        /* Add the node to dt_host. */
> +        rc = dt_overlay_add_node(overlay_node, overlay_node->parent->full_name);
> +        if ( rc )
> +        {
> +            read_unlock(&dt_host->lock);
> +
> +            /* Node not added in dt_host. */
> +            return rc;
> +        }
> +
> +        read_unlock(&dt_host->lock);
> +
> +        prev_node->allnext = next_node;
> +
> +        overlay_node = dt_find_node_by_path(overlay_node->full_name);
> +        if ( overlay_node == NULL )
> +        {
> +            /* Sanity check. But code will never come here. */
> +            ASSERT_UNREACHABLE();
> +            return -EFAULT;
> +        }
> +
> +        rc = handle_add_irq_iommu(overlay_node);
> +        if ( rc )
> +        {
> +            printk(XENLOG_ERR "Adding IRQ and IOMMU failed\n");
> +            return rc;
> +        }
> +
> +        /* Keep overlay_node address in tracker. */
> +        tr->nodes_address[j] = (unsigned long)overlay_node;
> +    }
> +
> +    return 0;
> +}
> +/*
> + * Adds device tree nodes under target node.
> + * We use tr->dt_host_new to unflatten the updated device_tree_flattened. This
> + * is done to avoid the removal of device_tree generation, iomem regions mapping
> + * to hardware domain done by handle_node().
> + */
> +static long handle_add_overlay_nodes(void *overlay_fdt,
> +                                     uint32_t overlay_fdt_size)
> +{
> +    int rc, j;
j shall be unsigned

> +    struct dt_device_node *overlay_node;
> +    struct overlay_track *tr = NULL;
> +    char **nodes_full_path = NULL;
> +    unsigned int new_fdt_size;
> +
> +    tr = xzalloc(struct overlay_track);
> +    if ( tr == NULL )
> +        return -ENOMEM;
> +
> +    new_fdt_size = fdt_totalsize(device_tree_flattened) +
> +                                 fdt_totalsize(overlay_fdt);
> +
> +    tr->fdt = xzalloc_bytes(new_fdt_size);
> +    if ( tr->fdt == NULL )
> +    {
> +        xfree(tr);
> +        return -ENOMEM;
> +    }
> +
> +    tr->num_nodes = overlay_node_count(overlay_fdt);
> +    if ( tr->num_nodes == 0 )
> +    {
> +        xfree(tr->fdt);
> +        xfree(tr);
> +        return -ENOMEM;
> +    }
> +
> +    tr->nodes_address = xzalloc_bytes(tr->num_nodes * sizeof(unsigned long));
> +    if ( tr->nodes_address == NULL )
> +    {
> +        xfree(tr->fdt);
> +        xfree(tr);
> +        return -ENOMEM;
> +    }
> +
> +    rc = check_overlay_fdt(overlay_fdt, overlay_fdt_size);
> +    if ( rc )
> +    {
> +        xfree(tr->nodes_address);
> +        xfree(tr->fdt);
> +        xfree(tr);
> +        return rc;
> +    }
> +
> +    /*
> +     * Keep a copy of overlay_fdt as fdt_overlay_apply will change the input
> +     * overlay's content(magic) when applying overlay.
> +     */
> +    tr->overlay_fdt = xzalloc_bytes(overlay_fdt_size);
> +    if ( tr->overlay_fdt == NULL )
> +    {
> +        xfree(tr->nodes_address);
> +        xfree(tr->fdt);
> +        xfree(tr);
> +        return -ENOMEM;
> +    }
> +
> +    memcpy(tr->overlay_fdt, overlay_fdt, overlay_fdt_size);
> +
> +    spin_lock(&overlay_lock);
> +
> +    memcpy(tr->fdt, device_tree_flattened,
> +           fdt_totalsize(device_tree_flattened));
> +
> +    /* Open tr->fdt with more space to accommodate the overlay_fdt. */
> +    rc = fdt_open_into(tr->fdt, tr->fdt, new_fdt_size);
> +    if ( rc )
> +    {
> +        printk(XENLOG_ERR "Increasing fdt size to accommodate overlay_fdt failed with error %d\n",
> +               rc);
> +        goto err;
> +    }
> +
> +    nodes_full_path = xzalloc_bytes(tr->num_nodes * sizeof(char *));
> +    if ( nodes_full_path == NULL )
> +    {
> +        rc = -ENOMEM;
> +        goto err;
> +    }
> +
> +    /*
> +     * overlay_get_nodes_info is called to get the node information from dtbo.
> +     * This is done before fdt_overlay_apply() because the overlay apply will
> +     * erase the magic of overlay_fdt.
> +     */
> +    rc = overlay_get_nodes_info(overlay_fdt, nodes_full_path);
> +    if ( rc )
> +    {
> +        printk(XENLOG_ERR "Getting nodes information failed with error %d\n",
> +               rc);
> +        goto err;
> +    }
> +
> +    rc = fdt_overlay_apply(tr->fdt, overlay_fdt);
> +    if ( rc )
> +    {
> +        printk(XENLOG_ERR "Adding overlay node failed with error %d\n", rc);
> +        goto err;
> +    }
> +
> +    /*
> +     * Check if any of the node already exists in dt_host. If node already exits
> +     * we can return here as this overlay_fdt is not suitable for overlay ops.
> +     */
> +    for ( j = 0; j < tr->num_nodes; j++ )
> +    {
> +        overlay_node = dt_find_node_by_path(nodes_full_path[j]);
> +        if ( overlay_node != NULL )
> +        {
> +            printk(XENLOG_ERR "node %s exists in device tree\n",
> +                   nodes_full_path[j]);
> +            rc = -EINVAL;
> +            goto err;
> +        }
> +    }
> +
> +    /* Unflatten the tr->fdt into a new dt_host. */
> +    rc = unflatten_device_tree(tr->fdt, &tr->dt_host_new);
> +    if ( rc )
> +    {
> +        printk(XENLOG_ERR "unflatten_device_tree failed with error %d\n", rc);
> +        goto err;
> +    }
> +
> +    rc = add_nodes(tr, nodes_full_path);
> +    if ( rc )
> +    {
> +        printk(XENLOG_ERR "Adding nodes failed. Removing the partially added nodes.\n");
> +        goto remove_node;
> +    }
> +
> +    INIT_LIST_HEAD(&tr->entry);
> +    list_add_tail(&tr->entry, &overlay_tracker);
> +
> +    spin_unlock(&overlay_lock);
> +
> +    free_nodes_full_path(tr->num_nodes, nodes_full_path);
> +
> +    return rc;
> +
> +/*
> + * Failure case. We need to remove the nodes, free tracker(if tr exists) and
> + * tr->dt_host_new.
> + */
> +remove_node:
> +    tr->num_nodes = j;
> +    rc = remove_nodes(tr);
> +
> +    if ( rc )
> +    {
> +        /*
> +         * User needs to provide right overlay. Incorrect node information
> +         * example parent node doesn't exist in dt_host etc can cause memory
> +         * leaks as removing_nodes() will fail and this means nodes memory is
> +         * not freed from tracker. Which may cause memory leaks. Ideally, these
> +         * device tree related mistakes will be caught by fdt_overlay_apply()
> +         * but given that we don't manage that code keeping this warning message
> +         * is better here.
> +         */
> +        printk(XENLOG_ERR "Removing node failed.\n");
> +        spin_unlock(&overlay_lock);
> +
> +        free_nodes_full_path(tr->num_nodes, nodes_full_path);
> +
> +        return rc;
> +    }
> +
> +err:
> +    spin_unlock(&overlay_lock);
> +
> +    if ( tr->dt_host_new )
> +        xfree(tr->dt_host_new);
> +
> +    xfree(tr->overlay_fdt);
> +    xfree(tr->nodes_address);
> +    xfree(tr->fdt);
> +
> +    free_nodes_full_path(tr->num_nodes, nodes_full_path);
This should be moved after freeing dt_host_new to keep reverse order

> +
> +    xfree(tr);
> +
> +    return rc;
> +}
> +
>  long dt_sysctl(struct xen_sysctl_dt_overlay *op)
>  {
>      long ret;
> @@ -396,6 +924,11 @@ long dt_sysctl(struct xen_sysctl_dt_overlay *op)
>  
>      switch ( op->overlay_op )
>      {
> +    case XEN_SYSCTL_DT_OVERLAY_ADD:
> +        ret = handle_add_overlay_nodes(overlay_fdt, op->overlay_fdt_size);
> +
remove empty line

> +        break;
> +
>      case XEN_SYSCTL_DT_OVERLAY_REMOVE:
>          ret = handle_remove_overlay_nodes(overlay_fdt, op->overlay_fdt_size);
>  

~Michal

Re: [XEN][PATCH v7 01/19] common/device_tree: handle memory allocation failure in __unflatten_device_tree()

[Julien Grall]

Hi,

On 02/06/2023 01:48, Vikram Garhwal wrote:
> Change __unflatten_device_tree() return type to integer so it can propagate
> memory allocation failure. Add panic() in dt_unflatten_host_device_tree() for
> memory allocation failure during boot.
> 
> Fixes: fb97eb614acf ("xen/arm: Create a hierarchical device tree")
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>

With Michal's remark handled (it can be done on commit):

Acked-by: Julien Grall <jgrall@amazon.com>

Cheers,

> Reviewed-by: Henry Wang <Henry.Wang@arm.com>
> ---
>   xen/common/device_tree.c | 14 +++++++++++---
>   1 file changed, 11 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
> index 8da1052911..dfdb10e488 100644
> --- a/xen/common/device_tree.c
> +++ b/xen/common/device_tree.c
> @@ -2092,8 +2092,8 @@ static unsigned long __init unflatten_dt_node(const void *fdt,
>    * @fdt: The fdt to expand
>    * @mynodes: The device_node tree created by the call
>    */
> -static void __init __unflatten_device_tree(const void *fdt,
> -                                           struct dt_device_node **mynodes)
> +static int __init __unflatten_device_tree(const void *fdt,
> +                                          struct dt_device_node **mynodes)
>   {
>       unsigned long start, mem, size;
>       struct dt_device_node **allnextp = mynodes;
> @@ -2114,6 +2114,8 @@ static void __init __unflatten_device_tree(const void *fdt,
>   
>       /* Allocate memory for the expanded device tree */
>       mem = (unsigned long)_xmalloc (size + 4, __alignof__(struct dt_device_node));
> +    if ( !mem )
> +        return -ENOMEM;
>   
>       ((__be32 *)mem)[size / 4] = cpu_to_be32(0xdeadbeef);
>   
> @@ -2131,6 +2133,8 @@ static void __init __unflatten_device_tree(const void *fdt,
>       *allnextp = NULL;
>   
>       dt_dprintk(" <- unflatten_device_tree()\n");
> +
> +    return 0;
>   }
>   
>   static void dt_alias_add(struct dt_alias_prop *ap,
> @@ -2215,7 +2219,11 @@ dt_find_interrupt_controller(const struct dt_device_match *matches)
>   
>   void __init dt_unflatten_host_device_tree(void)
>   {
> -    __unflatten_device_tree(device_tree_flattened, &dt_host);
> +    int error = __unflatten_device_tree(device_tree_flattened, &dt_host);
> +
> +    if ( error )
> +        panic("__unflatten_device_tree failed with error %d\n", error);
> +
>       dt_alias_scan();
>   }
>   

-- 
Julien Grall

Re: [XEN][PATCH v7 04/19] common/device_tree: change __unflatten_device_tree() type

[Julien Grall]

Hi,

Title:

'type' is a bit confusing here. How about "Export __unflatten_device_tre()"?

On 02/06/2023 01:48, Vikram Garhwal wrote:
> Following changes are done to __unflatten_device_tree():
>      1. __unflatten_device_tree() is renamed to unflatten_device_tree().
>      2. Remove __init and static function type.

As there is no external caller yet, please explain why you want to 
export the function.

Cheers,

> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> Reviewed-by: Henry Wang <Henry.Wang@arm.com>
> ---
>   xen/common/device_tree.c      | 9 ++++-----
>   xen/include/xen/device_tree.h | 5 +++++
>   2 files changed, 9 insertions(+), 5 deletions(-)
> 
> diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
> index bbdab07596..16b4b4e946 100644
> --- a/xen/common/device_tree.c
> +++ b/xen/common/device_tree.c
> @@ -2083,7 +2083,7 @@ static unsigned long unflatten_dt_node(const void *fdt,
>   }
>   
>   /**
> - * __unflatten_device_tree - create tree of device_nodes from flat blob
> + * unflatten_device_tree - create tree of device_nodes from flat blob
>    *
>    * unflattens a device-tree, creating the
>    * tree of struct device_node. It also fills the "name" and "type"
> @@ -2092,8 +2092,7 @@ static unsigned long unflatten_dt_node(const void *fdt,
>    * @fdt: The fdt to expand
>    * @mynodes: The device_node tree created by the call
>    */
> -static int __init __unflatten_device_tree(const void *fdt,
> -                                          struct dt_device_node **mynodes)
> +int unflatten_device_tree(const void *fdt, struct dt_device_node **mynodes)
>   {
>       unsigned long start, mem, size;
>       struct dt_device_node **allnextp = mynodes;
> @@ -2230,10 +2229,10 @@ dt_find_interrupt_controller(const struct dt_device_match *matches)
>   
>   void __init dt_unflatten_host_device_tree(void)
>   {
> -    int error = __unflatten_device_tree(device_tree_flattened, &dt_host);
> +    int error = unflatten_device_tree(device_tree_flattened, &dt_host);
>   
>       if ( error )
> -        panic("__unflatten_device_tree failed with error %d\n", error);
> +        panic("unflatten_device_tree failed with error %d\n", error);
>   
>       dt_alias_scan();

This function doesn't seem to be called in the case of the overlay 
device-tree. Does this mean that it will never contain any alias?

>   }
> diff --git a/xen/include/xen/device_tree.h b/xen/include/xen/device_tree.h
> index c2eada7489..2c35c0d391 100644
> --- a/xen/include/xen/device_tree.h
> +++ b/xen/include/xen/device_tree.h
> @@ -178,6 +178,11 @@ int device_tree_for_each_node(const void *fdt, int node,
>    */
>   void dt_unflatten_host_device_tree(void);
>   
> +/**
> + * unflatten any device tree.

Most of the exported function in device_tre.h have documentation. Can 
you do the same here?

> + */
> +int unflatten_device_tree(const void *fdt, struct dt_device_node **mynodes);

NIT: From an external interface perspective, do we actually need to pass 
an extra pointer? Can't we instead, return the pointer?

> +
>   /**
>    * IRQ translation callback
>    * TODO: For the moment we assume that we only have ONE

Cheers,

-- 
Julien Grall

Re: [XEN][PATCH v7 07/19] libfdt: overlay: change overlay_get_target()

[Julien Grall]

Hi,

On 02/06/2023 01:48, Vikram Garhwal wrote:
> Rename overlay_get_target() to fdt_overlay_target_offset() and remove static
> function type.
> 
> This is done to get the target path for the overlay nodes which is very useful
> in many cases. For example, Xen hypervisor needs it when applying overlays
> because Xen needs to do further processing of the overlay nodes, e.g. mapping of
> resources(IRQs and IOMMUs) to other VMs, creation of SMMU pagetables, etc.
> 
> Signed-off-by: Vikram Garhwal <fnu.vikram@xilinx.com>
> Message-Id: <1637204036-382159-2-git-send-email-fnu.vikram@xilinx.com>
> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
> Origin: git://git.kernel.org/pub/scm/utils/dtc/dtc.git 45f3d1a095dd
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> Reviewed-by: Michal Orzel <michal.orzel@amd.com>
> Reviewed-by: Henry Wang <Henry.Wang@arm.com>

Acked-by: Juline Grall <jgrall@amazon.com>

Cheers,

-- 
Julien Grall

Re: [XEN][PATCH v7 08/19] xen/device-tree: Add device_tree_find_node_by_path() to find nodes in device tree

[Julien Grall]

Hi,

On 02/06/2023 01:48, Vikram Garhwal wrote:
> Add device_tree_find_node_by_path() to find a matching node with path for a

AFAICT, the only difference in name between the new function and the 
existing one is "device_tree" vs "dt". The latter is just a shorthand of 
"device tree", so it feels to me the name are a bit too similar.

 From my understanding, the main difference between the two functions 
are that the current one is starting from root whereas the current one 
is starting from a given node. So how about "dt_find_node_by_path_from()"?

> dt_device_node.
> 
> Reason behind this function:
>      Each time overlay nodes are added using .dtbo, a new fdt(memcpy of
>      device_tree_flattened) is created and updated with overlay nodes. This
>      updated fdt is further unflattened to a dt_host_new. Next, we need to find
>      the overlay nodes in dt_host_new, find the overlay node's parent in dt_host
>      and add the nodes as child under their parent in the dt_host. Thus we need
>      this function to search for node in different unflattened device trees.
> 
> Also, make dt_find_node_by_path() static inline.
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> 
> ---
> Changes from v6:
>      Rename "dt_node" to "from"
> ---
>   xen/common/device_tree.c      |  6 ++++--
>   xen/include/xen/device_tree.h | 18 ++++++++++++++++--
>   2 files changed, 20 insertions(+), 4 deletions(-)
> 
> diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
> index 16b4b4e946..c5250a1644 100644
> --- a/xen/common/device_tree.c
> +++ b/xen/common/device_tree.c
> @@ -358,11 +358,13 @@ struct dt_device_node *dt_find_node_by_type(struct dt_device_node *from,
>       return np;
>   }
>   
> -struct dt_device_node *dt_find_node_by_path(const char *path)
> +struct dt_device_node *
> +                    device_tree_find_node_by_path(struct dt_device_node *from,
> +                                                  const char *path)
>   {
>       struct dt_device_node *np;
>   
> -    dt_for_each_device_node(dt_host, np)
> +    dt_for_each_device_node(from, np)
>           if ( np->full_name && (dt_node_cmp(np->full_name, path) == 0) )
>               break;
>   
> diff --git a/xen/include/xen/device_tree.h b/xen/include/xen/device_tree.h
> index 2c35c0d391..e239f7de26 100644
> --- a/xen/include/xen/device_tree.h
> +++ b/xen/include/xen/device_tree.h
> @@ -561,13 +561,27 @@ struct dt_device_node *dt_find_node_by_type(struct dt_device_node *from,
>   struct dt_device_node *dt_find_node_by_alias(const char *alias);
>   
>   /**
> - * dt_find_node_by_path - Find a node matching a full DT path
> + * device_tree_find_node_by_path - Generic function to find a node matching the
> + * full DT path for any given unflatten device tree
> + * @from: The device tree node to start searching from
>    * @path: The full path to match
>    *
>    * Returns a node pointer.
>    */
> -struct dt_device_node *dt_find_node_by_path(const char *path);
> +struct dt_device_node *
> +                    device_tree_find_node_by_path(struct dt_device_node *from,
> +                                                  const char *path);

The indentation looks slightly odd. In general, if the return type is on 
its own line, then the function name is not indented on the new line.

>   
> +/**
> + * dt_find_node_by_path - Find a node matching a full DT path in dt_host
> + * @path: The full path to match
> + *
> + * Returns a node pointer.
> + */
> +static inline struct dt_device_node *dt_find_node_by_path(const char *path)
> +{
> +    return device_tree_find_node_by_path(dt_host, path);
> +}
>   
>   /**
>    * dt_find_node_by_gpath - Same as dt_find_node_by_path but retrieve the

Cheers,

-- 
Julien Grall

Re: [XEN][PATCH v7 09/19] xen/iommu: Move spin_lock from iommu_dt_device_is_assigned to caller

[Julien Grall]

Hi,

On 02/06/2023 01:48, Vikram Garhwal wrote:
> Rename iommu_dt_device_is_assigned() to iommu_dt_device_is_assigned_locked().
> Remove static type so this can also be used by SMMU drivers to check if the
> device is being used before removing.
> 
> Moving spin_lock to caller was done to prevent the concurrent access to
> iommu_dt_device_is_assigned while doing add/remove/assign/deassign.

Can you explain if you are trying to resolve an existing bug, or this is 
something that will be necessary in a follow-up patch?

> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> 
> ---
> Changes from v6:
>      Created a private header and moved iommu_dt_device_is_assigned() to header.
> ---
>   xen/drivers/passthrough/device_tree.c | 20 ++++++++++++++++----
>   xen/include/xen/iommu-private.h       | 27 +++++++++++++++++++++++++++
>   2 files changed, 43 insertions(+), 4 deletions(-)
>   create mode 100644 xen/include/xen/iommu-private.h
> 
> diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
> index 1c32d7b50c..52e370db01 100644
> --- a/xen/drivers/passthrough/device_tree.c
> +++ b/xen/drivers/passthrough/device_tree.c
> @@ -18,6 +18,7 @@
>   #include <xen/device_tree.h>
>   #include <xen/guest_access.h>
>   #include <xen/iommu.h>
> +#include <xen/iommu-private.h>
>   #include <xen/lib.h>
>   #include <xen/sched.h>
>   #include <xsm/xsm.h>
> @@ -83,16 +84,14 @@ fail:
>       return rc;
>   }
>   
> -static bool_t iommu_dt_device_is_assigned(const struct dt_device_node *dev)
> +bool_t iommu_dt_device_is_assigned_locked(const struct dt_device_node *dev)
>   {

Please add an ASSERT() checking the lock is taken.

>       bool_t assigned = 0;
>   
>       if ( !dt_device_is_protected(dev) )
>           return 0;
>   
> -    spin_lock(&dtdevs_lock);
>       assigned = !list_empty(&dev->domain_list);
> -    spin_unlock(&dtdevs_lock);
>   
>       return assigned;
>   }
> @@ -213,27 +212,40 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
>           if ( (d && d->is_dying) || domctl->u.assign_device.flags )
>               break;
>   
> +        spin_lock(&dtdevs_lock);
> +

'dtdevs_lock' was intended to protect modification related to any IOMMU 
change. But here...

>           ret = dt_find_node_by_gpath(domctl->u.assign_device.u.dt.path,
>                                       domctl->u.assign_device.u.dt.size,
>                                       &dev);

... you also include "dt_find_node_by_gpath". Can you explain why and 
add a comment on top of 'dtdevs_lock' to explain what it is intended use?

>           if ( ret )
> +        {
> +            spin_unlock(&dtdevs_lock);
>               break;
> +        }
>   
>           ret = xsm_assign_dtdevice(XSM_HOOK, d, dt_node_full_name(dev));
>           if ( ret )
> +        {
> +            spin_unlock(&dtdevs_lock);
>               break;
> +        }
>   
>           if ( domctl->cmd == XEN_DOMCTL_test_assign_device )
>           {
> -            if ( iommu_dt_device_is_assigned(dev) )
> +
> +            if ( iommu_dt_device_is_assigned_locked(dev) )
>               {
>                   printk(XENLOG_G_ERR "%s already assigned.\n",
>                          dt_node_full_name(dev));
>                   ret = -EINVAL;
>               }
> +
> +            spin_unlock(&dtdevs_lock);
>               break;
>           }
>   
> +        spin_unlock(&dtdevs_lock);
> +
>           if ( d == dom_io )
>               return -EINVAL;
>   
> diff --git a/xen/include/xen/iommu-private.h b/xen/include/xen/iommu-private.h
> new file mode 100644
> index 0000000000..5615decaff
> --- /dev/null
> +++ b/xen/include/xen/iommu-private.h
> @@ -0,0 +1,27 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> + /*
> + * xen/iommu-private.h
> + *
> + *
> + * Copyright (C) 2023, Advanced Micro Devices, Inc. All Rights Reserved.
> + * Written by Vikram Garhwal <vikram.garhwal@amd.com>
> + *
> + */
> +#ifndef __XEN_IOMMU_PRIVATE_H__
> +#define __XEN_IOMMU_PRIVATE_H__
> +
> +#ifdef CONFIG_HAS_DEVICE_TREE
> +#include <xen/device_tree.h>
> +bool_t iommu_dt_device_is_assigned_locked(const struct dt_device_node *dev);
> +#endif
> +
> +#endif /* __XEN_IOMMU_PRIVATE_H__ */
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */

-- 
Julien Grall

Re: [XEN][PATCH v7 09/19] xen/iommu: Move spin_lock from iommu_dt_device_is_assigned to caller

[Julien Grall]

Hi,

On 02/06/2023 08:45, Michal Orzel wrote:
> 
> On 02/06/2023 02:48, Vikram Garhwal wrote:
>> diff --git a/xen/include/xen/iommu-private.h b/xen/include/xen/iommu-private.h
>> new file mode 100644
>> index 0000000000..5615decaff
>> --- /dev/null
>> +++ b/xen/include/xen/iommu-private.h
>> @@ -0,0 +1,27 @@
>> +/* SPDX-License-Identifier: GPL-2.0-only */
>> + /*
> incorrect indentation (<< 1)
> 
>> + * xen/iommu-private.h
>> + *
>> + *
>> + * Copyright (C) 2023, Advanced Micro Devices, Inc. All Rights Reserved.
>> + * Written by Vikram Garhwal <vikram.garhwal@amd.com>
> I'm not sure if placing the copyright is appropriate, given a single prototype for a function
> not really implemented by you with just spinlocks removed. But I might be wrong.

I agree. If you want to add a copyright then it should be the one from 
the original author.

But in this case, I don't think we should add an copyright as there is 
nothing really interesting in the header.

Cheers,

-- 
Julien Grall

Re: [XEN][PATCH v7 10/19] xen/iommu: protect iommu_add_dt_device() with dtdevs_lock

[Julien Grall]

Hi,

On 02/06/2023 01:48, Vikram Garhwal wrote:
> Protect iommu_add_dt_device() with dtdevs_lock to prevent concurrent access add.

The commit message is a bit light. What sort of concurrent access add 
are you talking about? Is it of the same node? Different node?

Also, is it a existing issue or something that will become one with a 
follow-up patch in your series?

Cheers,

-- 
Julien Grall

Re: [XEN][PATCH v7 11/19] xen/iommu: Introduce iommu_remove_dt_device()

[Julien Grall]

Hi,

On 02/06/2023 01:48, Vikram Garhwal wrote:
> Remove master device from the IOMMU. This will be helpful when removing the
> overlay nodes using dynamic programming during run time.
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> Reviewed-by: Michal Orzel <michal.orzel@amd.com>
> Acked-by: Jan Beulich <jbeulich@suse.com>
> ---
>   xen/drivers/passthrough/device_tree.c | 41 +++++++++++++++++++++++++++
>   xen/include/xen/iommu.h               |  2 ++
>   2 files changed, 43 insertions(+)
> 
> diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
> index 8cc413f867..301a5bcd97 100644
> --- a/xen/drivers/passthrough/device_tree.c
> +++ b/xen/drivers/passthrough/device_tree.c
> @@ -126,6 +126,47 @@ int iommu_release_dt_devices(struct domain *d)
>       return 0;
>   }
>   
> +int iommu_remove_dt_device(struct dt_device_node *np)
> +{
> +    const struct iommu_ops *ops = iommu_get_ops();
> +    struct device *dev = dt_to_dev(np);
> +    int rc;
> +

iommu_add_dt_device() checks if the IOMMU is enabled. I think you should 
do the same here as well and return 0 if it is disabled.

> +    if ( !ops )
> +        return -EOPNOTSUPP;
> +
> +    spin_lock(&dtdevs_lock);
> +
> +    if ( iommu_dt_device_is_assigned_locked(np) )
> +    {
> +        rc = -EBUSY;
> +        goto fail;
> +    }
> +
> +    /*
> +     * The driver which supports generic IOMMU DT bindings must have this
> +     * callback implemented.
> +     */

It is not clear to me why you want to mandate remove_device when using 
the generic IOMMU DT bindings.

But if this is really necessary, then I think the comment should be 
placed on top of the callback definition rather than in the caller.

> +    if ( !ops->remove_device )
> +    {
> +        rc = -EOPNOTSUPP;
> +        goto fail;
> +    }
> +
> +    /*
> +     * Remove master device from the IOMMU if latter is present and available.
> +     * The driver is responsible for removing is_protected flag.
> +     */
> +    rc = ops->remove_device(0, dev);
> +
> +    if ( !rc )
> +        iommu_fwspec_free(dev);
> +
> +fail:
> +    spin_unlock(&dtdevs_lock);
> +    return rc;
> +}
> +
>   int iommu_add_dt_device(struct dt_device_node *np)
>   {
>       const struct iommu_ops *ops = iommu_get_ops();
> diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
> index 405db59971..0d7924821b 100644
> --- a/xen/include/xen/iommu.h
> +++ b/xen/include/xen/iommu.h
> @@ -229,6 +229,8 @@ int iommu_release_dt_devices(struct domain *d);
>    */
>   int iommu_add_dt_device(struct dt_device_node *np);
>   
> +int iommu_remove_dt_device(struct dt_device_node *np);
> +
>   int iommu_do_dt_domctl(struct xen_domctl *, struct domain *,
>                          XEN_GUEST_HANDLE_PARAM(xen_domctl_t));
>   

Cheers,

-- 
Julien Grall

Re: [XEN][PATCH v7 13/19] asm/smp.h: Fix circular dependency for device_tree.h and rwlock.h

[Julien Grall]

Hi,

On 02/06/2023 01:48, Vikram Garhwal wrote:
> Dynamic programming ops will modify the dt_host and there might be other
> function which are browsing the dt_host at the same time. To avoid the race
> conditions, adding rwlock for browsing the dt_host.

Reading this sentence, it sounds like you are adding the rwlock in this 
patch. However, this is not the case. Also, the rwlock is not only for 
browsing but also add new node. So how about ", we will need to add a 
rwlock to protect access to the dt_host".

> But adding rwlock in
> device_tree.h causes following circular dependency:
>      device_tree.h->rwlock.h->smp.h->asm/smp.h->device_tree.h
> 
> To fix this, removed the "#include <xen/device_tree.h> and forward declared
> "struct dt_device_node".
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> Reviewed-by: Henry Wang <Henry.Wang@arm.com>
> Reviewed-by: Michal Orzel <michal.orzel@amd.com>
> ---
>   xen/arch/arm/include/asm/smp.h | 3 ++-
>   xen/arch/arm/smpboot.c         | 1 +
>   2 files changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/include/asm/smp.h b/xen/arch/arm/include/asm/smp.h
> index a37ca55bff..b12949ba8a 100644
> --- a/xen/arch/arm/include/asm/smp.h
> +++ b/xen/arch/arm/include/asm/smp.h
> @@ -3,13 +3,14 @@
>   
>   #ifndef __ASSEMBLY__
>   #include <xen/cpumask.h>
> -#include <xen/device_tree.h>
>   #include <asm/current.h>
>   #endif
>   
>   DECLARE_PER_CPU(cpumask_var_t, cpu_sibling_mask);
>   DECLARE_PER_CPU(cpumask_var_t, cpu_core_mask);
>   
> +struct dt_device_node;

Can you add the declaration just above arch_cpu_init()? This will make 
clearer why the forward declaration is necessary.

> +
>   #define cpu_is_offline(cpu) unlikely(!cpu_online(cpu))
>   
>   #define smp_processor_id() get_processor_id()
> diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
> index e107b86b7b..eeb76cd551 100644
> --- a/xen/arch/arm/smpboot.c
> +++ b/xen/arch/arm/smpboot.c
> @@ -10,6 +10,7 @@
>   #include <xen/cpu.h>
>   #include <xen/cpumask.h>
>   #include <xen/delay.h>
> +#include <xen/device_tree.h>
>   #include <xen/domain_page.h>
>   #include <xen/errno.h>
>   #include <xen/init.h>

Cheers,

-- 
Julien Grall

Re: [XEN][PATCH v7 14/19] common/device_tree: Add rwlock for dt_host

[Julien Grall]

Hi,

On 02/06/2023 01:48, Vikram Garhwal wrote:
>   Dynamic programming ops will modify the dt_host and there might be other
>   function which are browsing the dt_host at the same time. To avoid the race
>   conditions, adding rwlock for browsing the dt_host during runtime.
Please explain that writer will be added in a follow-up patch.

> 
>   Reason behind adding rwlock instead of spinlock:
>      For now, dynamic programming is the sole modifier of dt_host in Xen during
>          run time. All other access functions like iommu_release_dt_device() are
>          just reading the dt_host during run-time. So, there is a need to protect
>          others from browsing the dt_host while dynamic programming is modifying
>          it. rwlock is better suitable for this task as spinlock won't be able to
>          differentiate between read and write access.
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> 
> ---
> Changes from v6:
>      Remove redundant "read_unlock(&dt_host->lock);" in the following case:
>           XEN_DOMCTL_deassign_device
> ---
>   xen/common/device_tree.c              |  4 ++++
>   xen/drivers/passthrough/device_tree.c | 15 +++++++++++++++
>   xen/include/xen/device_tree.h         |  6 ++++++
>   3 files changed, 25 insertions(+)
> 
> diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
> index c5250a1644..c8fcdf8fa1 100644
> --- a/xen/common/device_tree.c
> +++ b/xen/common/device_tree.c
> @@ -2146,7 +2146,11 @@ int unflatten_device_tree(const void *fdt, struct dt_device_node **mynodes)
>   
>       dt_dprintk(" <- unflatten_device_tree()\n");
>   
> +    /* Init r/w lock for host device tree. */
> +    rwlock_init(&dt_host->lock);
> +
>       return 0;
> +
>   }
>   
>   static void dt_alias_add(struct dt_alias_prop *ap,
> diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
> index 301a5bcd97..f4d9deb624 100644
> --- a/xen/drivers/passthrough/device_tree.c
> +++ b/xen/drivers/passthrough/device_tree.c
> @@ -112,6 +112,8 @@ int iommu_release_dt_devices(struct domain *d)
>       if ( !is_iommu_enabled(d) )
>           return 0;
>   
> +    read_lock(&dt_host->lock);
> +
>       list_for_each_entry_safe(dev, _dev, &hd->dt_devices, domain_list)
>       {
>           rc = iommu_deassign_dt_device(d, dev);
> @@ -119,10 +121,14 @@ int iommu_release_dt_devices(struct domain *d)
>           {
>               dprintk(XENLOG_ERR, "Failed to deassign %s in domain %u\n",
>                       dt_node_full_name(dev), d->domain_id);
> +
> +            read_unlock(&dt_host->lock);
>               return rc;
>           }
>       }
>   
> +    read_unlock(&dt_host->lock);
> +
>       return 0;
>   }
>   
> @@ -246,6 +252,8 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
>       int ret;
>       struct dt_device_node *dev;
>   
> +    read_lock(&dt_host->lock);
> +
>       switch ( domctl->cmd )
>       {
>       case XEN_DOMCTL_assign_device:
> @@ -295,7 +303,10 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
>           spin_unlock(&dtdevs_lock);
>   
>           if ( d == dom_io )
> +        {
> +            read_unlock(&dt_host->lock);
>               return -EINVAL;
> +        }
>   
>           ret = iommu_add_dt_device(dev);
>           if ( ret < 0 )
> @@ -333,7 +344,10 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
>               break;
>   
>           if ( d == dom_io )
> +        {
> +            read_unlock(&dt_host->lock);
>               return -EINVAL;
> +        }
>   
>           ret = iommu_deassign_dt_device(d, dev);
>   
> @@ -348,5 +362,6 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
>           break;
>       }
>   
> +    read_unlock(&dt_host->lock);
>       return ret;
>   }
> diff --git a/xen/include/xen/device_tree.h b/xen/include/xen/device_tree.h
> index e239f7de26..dee40d2ea3 100644
> --- a/xen/include/xen/device_tree.h
> +++ b/xen/include/xen/device_tree.h
> @@ -18,6 +18,7 @@
>   #include <xen/string.h>
>   #include <xen/types.h>
>   #include <xen/list.h>
> +#include <xen/rwlock.h>
>   
>   #define DEVICE_TREE_MAX_DEPTH 16
>   
> @@ -106,6 +107,11 @@ struct dt_device_node {
>       struct list_head domain_list;
>   
>       struct device dev;
> +
> +    /*
> +     * Lock that protects r/w updates to unflattened device tree i.e. dt_host.
> +     */

 From the description, it sounds like the rwlock will only be used to 
protect the entire device-tree rather than a single node. So it doesn't 
seem to be sensible to increase each node structure (there are a lot) by 
12 bytes.

Can you outline your plan?

> +    rwlock_t lock;
>   };
>   
>   #define dt_to_dev(dt_node)  (&(dt_node)->dev)

Cheers,

-- 
Julien Grall

Re: [XEN][PATCH v7 15/19] xen/arm: Implement device tree node removal functionalities

[Julien Grall]

Hi Vikram,

On 02/06/2023 01:48, Vikram Garhwal wrote:
> Introduce sysctl XEN_SYSCTL_dt_overlay to remove device-tree nodes added using
> device tree overlay.
> 
> xl dt-overlay remove file.dtbo:
>      Removes all the nodes in a given dtbo.
>      First, removes IRQ permissions and MMIO accesses. Next, it finds the nodes
>      in dt_host and delete the device node entries from dt_host.
> 
>      The nodes get removed only if it is not used by any of dom0 or domio.
> 
> Also, added overlay_track struct to keep the track of added node through device
> tree overlay. overlay_track has dt_host_new which is unflattened form of updated
> fdt and name of overlay nodes. When a node is removed, we also free the memory
> used by overlay_track for the particular overlay node.
> 
> Nested overlay removal is supported in sequential manner only i.e. if
> overlay_child nests under overlay_parent, it is assumed that user first removes
> overlay_child and then removes overlay_parent.
> 
> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> 
> ---
> Changes from v6:
>      Add explicit padding for xen_system_dt_overlay{}
>      Update license.
>      Rearrange xfree in dt_sysctl()
>      Update overlay_track struct comment with relevant message.
>      Fix missing xen/errno.h for builds without CONFIG_OVERLAY_DTB cases.
>      Fix header formatting.
> ---
>   xen/arch/arm/sysctl.c        |  16 +-
>   xen/common/Makefile          |   1 +
>   xen/common/dt-overlay.c      | 420 +++++++++++++++++++++++++++++++++++

I think the new dt-overlay.c wants to be added under the section 
"Device-Tree" in MAINTAINERS.

>   xen/include/public/sysctl.h  |  24 ++
>   xen/include/xen/dt-overlay.h |  59 +++++
>   5 files changed, 519 insertions(+), 1 deletion(-)
>   create mode 100644 xen/common/dt-overlay.c
>   create mode 100644 xen/include/xen/dt-overlay.h
> 
> diff --git a/xen/arch/arm/sysctl.c b/xen/arch/arm/sysctl.c
> index b0a78a8b10..8b813c970f 100644
> --- a/xen/arch/arm/sysctl.c
> +++ b/xen/arch/arm/sysctl.c
> @@ -9,6 +9,7 @@
>   
>   #include <xen/types.h>
>   #include <xen/lib.h>
> +#include <xen/dt-overlay.h>
>   #include <xen/errno.h>
>   #include <xen/hypercall.h>
>   #include <public/sysctl.h>
> @@ -21,7 +22,20 @@ void arch_do_physinfo(struct xen_sysctl_physinfo *pi)
>   long arch_do_sysctl(struct xen_sysctl *sysctl,
>                       XEN_GUEST_HANDLE_PARAM(xen_sysctl_t) u_sysctl)
>   {
> -    return -ENOSYS;
> +    long ret;
> +
> +    switch ( sysctl->cmd )
> +    {
> +    case XEN_SYSCTL_dt_overlay:
> +        ret = dt_sysctl(&sysctl->u.dt_overlay);
> +        break;
> +
> +    default:
> +        ret = -ENOSYS;
> +        break;
> +    }
> +
> +    return ret;
>   }
>   
>   /*
> diff --git a/xen/common/Makefile b/xen/common/Makefile
> index 46049eac35..e7e96b1087 100644
> --- a/xen/common/Makefile
> +++ b/xen/common/Makefile
> @@ -8,6 +8,7 @@ obj-$(CONFIG_DEBUG_TRACE) += debugtrace.o
>   obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o
>   obj-$(CONFIG_IOREQ_SERVER) += dm.o
>   obj-y += domain.o
> +obj-$(CONFIG_OVERLAY_DTB) += dt-overlay.o
>   obj-y += event_2l.o
>   obj-y += event_channel.o
>   obj-y += event_fifo.o
> diff --git a/xen/common/dt-overlay.c b/xen/common/dt-overlay.c
> new file mode 100644
> index 0000000000..b2a7e441df
> --- /dev/null
> +++ b/xen/common/dt-overlay.c
> @@ -0,0 +1,420 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * xen/common/dt-overlay.c
> + *
> + * Device tree overlay support in Xen.
> + *
> + * Copyright (C) 2023, Advanced Micro Devices, Inc. All Rights Reserved.
> + * Written by Vikram Garhwal <vikram.garhwal@amd.com>
> + *
> + */
> +#include <asm/domain_build.h>
> +#include <xen/dt-overlay.h>
> +#include <xen/guest_access.h>
> +#include <xen/iocap.h>
> +#include <xen/xmalloc.h>
> +
> +static LIST_HEAD(overlay_tracker);
> +static DEFINE_SPINLOCK(overlay_lock);
> +
> +/* Find last descendants of the device_node. */
> +static struct dt_device_node *
> +                find_last_descendants_node(struct dt_device_node *device_node)
> +{
> +    struct dt_device_node *child_node;
> +
> +    for ( child_node = device_node->child; child_node->sibling != NULL;
> +          child_node = child_node->sibling );
> +
> +    /* If last child_node also have children. */
> +    if ( child_node->child )
> +        child_node = find_last_descendants_node(child_node);
> +
> +    return child_node;
> +}
> +
> +static int dt_overlay_remove_node(struct dt_device_node *device_node)
> +{
> +    struct dt_device_node *np;
> +    struct dt_device_node *parent_node;
> +    struct dt_device_node *device_node_last_descendant = device_node->child;
> +
> +    parent_node = device_node->parent;
> +
> +    if ( parent_node == NULL )

Is this trying to check if we are trying to remove "/"? If not, what's 
this for?

Either way, I would suggest to add a comment explaining what this check 
is for.

> +    {
> +        dt_dprintk("%s's parent node not found\n", device_node->name);
> +        return -EFAULT;
> +    }
> +
> +    np = parent_node->child;
> +
> +    if ( np == NULL )

Why would parent_node->child is NULL if we found the parent from 
device_node?

> +    {
> +        dt_dprintk("parent node %s's not found\n", parent_node->name);
> +        return -EFAULT;
> +    }
> +
> +    /* If node to be removed is only child node or first child. */
> +    if ( !dt_node_cmp(np->full_name, device_node->full_name) )
> +    {
> +        parent_node->child = np->sibling;
> +
> +        /*
> +         * Iterate over all child nodes of device_node.

I am not sure how this matches the code below. I think it might be 
better to explain why it is necessary to find the last descendant. 
AFAIU, you seem to rely on the fact that allnext will point to the next 
children and every node in that list up to the last descandant will be 
under device_node.

> Given that we are
> +         * removing parent node, we need to remove all it's descendants too.
> +         */
> +        if ( device_node_last_descendant )
> +        {
> +            device_node_last_descendant =
> +                                        find_last_descendants_node(device_node);

This indentation is not nice. Can we shorten the variable to 
last_descendant?

> +            parent_node->allnext = device_node_last_descendant->allnext;
> +        }
> +        else
> +            parent_node->allnext = np->allnext;
> +
> +        return 0;
> +    }
> +
> +    for ( np = parent_node->child; np->sibling != NULL; np = np->sibling )
> +    {
> +        if ( !dt_node_cmp(np->sibling->full_name, device_node->full_name) )
> +        {
> +            /* Found the node. Now we remove it. */
> +            np->sibling = np->sibling->sibling;
> +
> +            if ( np->child )
> +                np = find_last_descendants_node(np);
> +
> +            /*
> +             * Iterate over all child nodes of device_node. Given that we are
> +             * removing parent node, we need to remove all it's descendants too.
> +             */
> +            if ( device_node_last_descendant )
> +                device_node_last_descendant =
> +                                        find_last_descendants_node(device_node);
> +
> +            if ( device_node_last_descendant )
> +                np->allnext = device_node_last_descendant->allnext;
> +            else
> +                np->allnext = np->allnext->allnext;
> +
> +            break;
> +        }
> +    }
> +
> +    return 0;
> +}
> +
> +/* Basic sanity check for the dtbo tool stack provided to Xen. */
> +static int check_overlay_fdt(const void *overlay_fdt, uint32_t overlay_fdt_size)
> +{
> +    if ( (fdt_totalsize(overlay_fdt) != overlay_fdt_size) ||
> +          fdt_check_header(overlay_fdt) )
> +    {
> +        printk(XENLOG_ERR "The overlay FDT is not a valid Flat Device Tree\n");
> +        return -EINVAL;
> +    }
> +
> +    return 0;
> +}
> +
> +/* Count number of nodes till one level of __overlay__ tag. */
> +static unsigned int overlay_node_count(const void *overlay_fdt)
> +{
> +    unsigned int num_overlay_nodes = 0;
> +    int fragment;
> +
> +    fdt_for_each_subnode(fragment, overlay_fdt, 0)
> +    {
> +        int subnode;
> +        int overlay;
> +
> +        overlay = fdt_subnode_offset(overlay_fdt, fragment, "__overlay__");
> +
> +        /*
> +         * overlay value can be < 0. But fdt_for_each_subnode() loop checks for
> +         * overlay >= 0. So, no need for a overlay>=0 check here.
> +         */
> +        fdt_for_each_subnode(subnode, overlay_fdt, overlay)
> +        {
> +            num_overlay_nodes++;
> +        }
> +    }
> +
> +    return num_overlay_nodes;
> +}
> +
> +static int handle_remove_irq_iommu(struct dt_device_node *device_node)

How about remove_resources()? This would avoid to miss the fact you are 
removing MMIO mapping as well and make the name a bit more palatable.

> +{
> +    int rc = 0;
> +    struct domain *d = hardware_domain;
> +    domid_t domid;
> +    unsigned int naddr, len;
> +    unsigned int i, nirq;
> +
> +    domid = dt_device_used_by(device_node);

Looking at the caller, it is not clear to me which lock is preventing 
the device to be assigned whilst you remove it.

> +
> +    dt_dprintk("Checking if node %s is used by any domain\n",
> +               device_node->full_name);
> +
> +    /* Remove the node if only it's assigned to domain 0 or domain io. */

Hmmm... Above, you are using hardware_domain. This may be dom0. So 
shouldn't you check against hardware_domain->domain_id?

> +    if ( domid != 0 && domid != DOMID_IO )
> +    {
> +        printk(XENLOG_ERR "Device %s is being used by domain %u. Removing nodes failed\n",
> +               device_node->full_name, domid);
> +        return -EINVAL;
> +    }
> +
> +    dt_dprintk("Removing node: %s\n", device_node->full_name);
> +
> +    nirq = dt_number_of_irq(device_node); > +
> +    /* Remove IRQ permission */
> +    for ( i = 0; i < nirq; i++ )
> +    {
> +        rc = platform_get_irq(device_node, i);

As I mentioned in [1], I think that parsing the Device-Tree again when 
removing any interrupts/mappings is a bit odd as there are more possible 
failures and is more complex than necessary. I have proposed a way to do 
it with rangeset, but I can't find any reason why this wasn't done. Can 
you explain?

> +        if ( rc < 0 )
> +        {
> +            printk(XENLOG_ERR "Failed to get IRQ num for device node %s\n",
> +                   device_node->full_name);
> +            return -EINVAL;
> +        }
> +
> +        if ( irq_access_permitted(d, rc) == false )

We commonly use "!irq_acess_permitted()". But, as pointed out in [1] 
with this check, it means that we would not be able to call remove again 
on the same node if the removal was partially done.

You suggested that the admin would need to reboot the platform. But this 
decision doesn't seem to documented in the commit message nor in the 
tools documentation. Can this be done?

> +        {
> +            printk(XENLOG_ERR "IRQ %d is not routed to domain %u\n", rc,
> +                   domid);

I am probably missing something here. From my understanding, 
irq_access_permitted() only tells you if the domain is allowed to manage 
the interrupt. So you could have two domains (the hardware domain and 
the guest) with irq_access_permitted() set. In fact, the hardware domain 
should always have it permitted.

If you want to check for the routing, then you will have to look for the 
information in the 'irq->desc'.

> +            return -EINVAL;
> +        }
> +        /*
> +         * TODO: We don't handle shared IRQs for now. So, it is assumed that
> +         * the IRQs was not shared with another devices.
> +         */
> +        rc = irq_deny_access(d, rc);
> +        if ( rc )
> +        {
> +            printk(XENLOG_ERR "unable to revoke access for irq %u for %s\n",
> +                   i, device_node->full_name);
> +            return rc;
> +        }

You don't reverse the change in the routing. What would happen if the 
next overlay is updated to now pass the same device to a guest?

I would be OK if this is not handled in this series. But it should be 
marked as a TODO.

> +    }
> +
> +    /* Check if iommu property exists. */
> +    if ( dt_get_property(device_node, "iommus", &len) )
> +    {
> +        rc = iommu_remove_dt_device(device_node);
> +        if ( rc != 0 && rc != -ENXIO )

Checking iommu_remove_dt_device() (I didn't check the driver 
implementation), I can't find any use of ENXIO. So can you explain in 
which condition this is meant to be returned?

> +            return rc;
> +    }
> +
> +    naddr = dt_number_of_address(device_node);
> +
> +    /* Remove mmio access. */
> +    for ( i = 0; i < naddr; i++ )
> +    {
> +        uint64_t addr, size;
> +
> +        rc = dt_device_get_address(device_node, i, &addr, &size);
> +        if ( rc )
> +        {
> +            printk(XENLOG_ERR "Unable to retrieve address %u for %s\n",
> +                   i, dt_node_full_name(device_node));
> +            return rc;
> +        }
> +
> +        rc = iomem_deny_access(d, paddr_to_pfn(addr),
> +                               paddr_to_pfn(PAGE_ALIGN(addr + size - 1)));

As pointed out in [1], I expect the function remove to be the inverse of 
add. Yet, you are not removing the mapping in the P2M.

I still don't think it is correct to leave the mapping around. But if 
you think it is necessary then I think this should be documented.

> +        if ( rc )
> +        {
> +            printk(XENLOG_ERR "Unable to remove dom%d access to"
> +                   " 0x%"PRIx64" - 0x%"PRIx64"\n",
> +                   d->domain_id,
> +                   addr & PAGE_MASK, PAGE_ALIGN(addr + size) - 1);
> +            return rc;
> +        }
> +
> +    }
> +
> +    return rc;
> +}
> +
> +/* Removes all descendants of the given node. */
> +static int remove_all_descendant_nodes(struct dt_device_node *device_node)
> +{
> +    int rc = 0;
> +    struct dt_device_node *child_node;
> +
> +    for ( child_node = device_node->child; child_node != NULL;
> +         child_node = child_node->sibling )
> +    {
> +        if ( child_node->child )
> +            remove_all_descendant_nodes(child_node);

AFAICT, the function could return an error. Can you explain why this is 
not checked?

> +
> +        rc = handle_remove_irq_iommu(child_node);
> +        if ( rc )
> +            return rc;
> +    }
> +
> +    return rc;
> +}
> +
> +/* Remove nodes from dt_host. */
> +static int remove_nodes(const struct overlay_track *tracker)
> +{
> +    int rc = 0;
> +    struct dt_device_node *overlay_node;
> +    unsigned int j;
> +
> +    for ( j = 0; j < tracker->num_nodes; j++ )
> +    {
> +        overlay_node = (struct dt_device_node *)tracker->nodes_address[j];
> +        if ( overlay_node == NULL )
> +        {
> +            printk(XENLOG_ERR "Device %s is not present in the tree. Removing nodes failed\n",
> +                   overlay_node->full_name);
> +            return -EINVAL;
> +        }
> +
> +        rc = remove_all_descendant_nodes(overlay_node);
> +
> +        /* All children nodes are unmapped. Now remove the node itself. */
> +        rc = handle_remove_irq_iommu(overlay_node);
> +        if ( rc )
> +            return rc;
> +
> +        read_lock(&dt_host->lock);
Shouldn't this be a write_lock()?

> +
> +        rc = dt_overlay_remove_node(overlay_node);
> +        if ( rc )
> +        {
> +            read_unlock(&dt_host->lock);
> +
> +            return rc;
> +        }
> +
> +        read_unlock(&dt_host->lock);
> +    }
> +
> +    return rc;
> +}
> +
> +/*
> + * First finds the device node to remove. Check if the device is being used by
> + * any dom and finally remove it from dt_host. IOMMU is already being taken care
> + * while destroying the domain.
> + */
> +static long handle_remove_overlay_nodes(void *overlay_fdt,
> +                                        uint32_t overlay_fdt_size)
> +{
> +    int rc;
> +    struct overlay_track *entry, *temp, *track;
> +    bool found_entry = false;
> +
> +    rc = check_overlay_fdt(overlay_fdt, overlay_fdt_size);
> +    if ( rc )
> +        return rc;
> +
> +    if ( overlay_node_count(overlay_fdt) == 0 )
> +        return -EINVAL;

Why do you need to check the validity of the FDT when...

> +
> +    spin_lock(&overlay_lock);
> +
> +    /*
> +     * First check if dtbo is correct i.e. it should one of the dtbo which was
> +     * used when dynamically adding the node.
> +     * Limitation: Cases with same node names but different property are not
> +     * supported currently. We are relying on user to provide the same dtbo
> +     * as it was used when adding the nodes.
> +     */
> +    list_for_each_entry_safe( entry, temp, &overlay_tracker, entry )
> +    {
> +        if ( memcmp(entry->overlay_fdt, overlay_fdt, overlay_fdt_size) == 0 )

... you compare against what we stored in Xen? If we found the entry, 
then wouldn't it mean that the FDT is valid?

> +        {
> +            track = entry;
> +            found_entry = true;
> +            break;
> +        }
> +    }
> +
> +    if ( found_entry == false )

We commonly use !found_entry.

> +    {
> +        rc = -EINVAL;
> +
> +        printk(XENLOG_ERR "Cannot find any matching tracker with input dtbo."
> +               " Removing nodes is supported only for prior added dtbo.\n");
> +        goto out;
> +
> +    }
> +
> +    rc = remove_nodes(entry);
> +    if ( rc )
> +    {
> +        printk(XENLOG_ERR "Removing node failed\n");
> +        goto out;
> +    }
> +
> +    list_del(&entry->entry);
> +
> +    xfree(entry->dt_host_new);
> +    xfree(entry->fdt);
> +    xfree(entry->overlay_fdt);
> +
> +    xfree(entry->nodes_address);
> +
> +    xfree(entry);
> +
> +out:
> +    spin_unlock(&overlay_lock);
> +    return rc;
> +}
> +
> +long dt_sysctl(struct xen_sysctl_dt_overlay *op)

It would be best to name it dt_overlay_sysctl() is this is meant to 
handle only overlay operations. If we want to do more, then I think we 
want to name the sysctl SYSCTL_DEVICE_TREE to reflect the more general 
purpose (renaming the sysctl after the fact is a pain).

> +{
> +    long ret;
> +    void *overlay_fdt;
> +
> +    if ( op->overlay_fdt_size == 0 ||

Do you expect the rest of the code to misbehave if we pass 0?

> op->overlay_fdt_size > KB(500) )

How did you decide the value KB(500)?

> +        return -EINVAL;
> +
> +    if ( op->pad[0] || op->pad[1] || op->pad[2] )
> +        return -EINVAL;
> +
> +    overlay_fdt = xmalloc_bytes(op->overlay_fdt_size);
> +
> +    if ( overlay_fdt == NULL )
> +        return -ENOMEM;
> +
> +    ret = copy_from_guest(overlay_fdt, op->overlay_fdt, op->overlay_fdt_size);
> +    if ( ret )
> +    {
> +        gprintk(XENLOG_ERR, "copy from guest failed\n");
> +        xfree(overlay_fdt);
> +
> +        return -EFAULT;
> +    }
> +
> +    switch ( op->overlay_op )
> +    {
> +    case XEN_SYSCTL_DT_OVERLAY_REMOVE:
> +        ret = handle_remove_overlay_nodes(overlay_fdt, op->overlay_fdt_size);
> +
> +        break;
> +
> +    default:

I think you want to set ret to -EOPNOTSUPP here rather than returning 0. 
Also, it feels to me you want to first check the validity of the 
operation before checking for overlay_fdt_size and the allocation.

This would allow to re-use the fields 'overlay_fdt_size'/'overlay_fdt' 
for a different purpose on a new operation and also avoid allocating 
memory when we know it will fail.

The former would not be possible in your current approach because a Xen 
without the hypothetical new sysctl may return -EINVAL/-EFAULT rather 
than -EOPNOTSUPP. So it would be more difficult to differentiate it in 
the toolstack.

> +        break;
> +    }
> +
> +    xfree(overlay_fdt);
> +
> +    return ret;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
> index 2b24d6bfd0..ff54607617 100644
> --- a/xen/include/public/sysctl.h
> +++ b/xen/include/public/sysctl.h
> @@ -1057,6 +1057,25 @@ typedef struct xen_sysctl_cpu_policy xen_sysctl_cpu_policy_t;
>   DEFINE_XEN_GUEST_HANDLE(xen_sysctl_cpu_policy_t);
>   #endif
>   
> +#if defined(__arm__) || defined (__aarch64__)
> +/*
> + * XEN_SYSCTL_dt_overlay
> + * Performs addition/removal of device tree nodes under parent node using dtbo.
> + * This does in three steps:
> + *  - Adds/Removes the nodes from dt_host.
> + *  - Adds/Removes IRQ permission for the nodes.
> + *  - Adds/Removes MMIO accesses.
> + */
> +struct xen_sysctl_dt_overlay {
> +    XEN_GUEST_HANDLE_64(void) overlay_fdt;  /* IN: overlay fdt. */
> +    uint32_t overlay_fdt_size;              /* IN: Overlay dtb size. */
> +#define XEN_SYSCTL_DT_OVERLAY_ADD                   1
> +#define XEN_SYSCTL_DT_OVERLAY_REMOVE                2
> +    uint8_t overlay_op;                     /* IN: Add or remove. */
> +    uint8_t pad[3];                         /* IN: Must be zero. */
> +};
> +#endif
> +
>   struct xen_sysctl {
>       uint32_t cmd;
>   #define XEN_SYSCTL_readconsole                    1
> @@ -1087,6 +1106,7 @@ struct xen_sysctl {
>   #define XEN_SYSCTL_livepatch_op                  27
>   /* #define XEN_SYSCTL_set_parameter              28 */
>   #define XEN_SYSCTL_get_cpu_policy                29
> +#define XEN_SYSCTL_dt_overlay                    30
>       uint32_t interface_version; /* XEN_SYSCTL_INTERFACE_VERSION */
>       union {
>           struct xen_sysctl_readconsole       readconsole;
> @@ -1117,6 +1137,10 @@ struct xen_sysctl {
>   #if defined(__i386__) || defined(__x86_64__)
>           struct xen_sysctl_cpu_policy        cpu_policy;
>   #endif
> +
> +#if defined(__arm__) || defined (__aarch64__)
> +        struct xen_sysctl_dt_overlay        dt_overlay;
> +#endif
>           uint8_t                             pad[128];
>       } u;
>   };
> diff --git a/xen/include/xen/dt-overlay.h b/xen/include/xen/dt-overlay.h
> new file mode 100644
> index 0000000000..43fa5a02a0
> --- /dev/null
> +++ b/xen/include/xen/dt-overlay.h
> @@ -0,0 +1,59 @@
> + /* SPDX-License-Identifier: GPL-2.0-only */
> + /*
> + * xen/dt-overlay.h
> + *
> + * Device tree overlay support in Xen.
> + *
> + * Copyright (C) 2023, Advanced Micro Devices, Inc. All Rights Reserved.
> + * Written by Vikram Garhwal <vikram.garhwal@amd.com>
> + *
> + */
> +#ifndef __XEN_DT_OVERLAY_H__
> +#define __XEN_DT_OVERLAY_H__
> +
> +#include <xen/list.h>
> +#include <xen/libfdt/libfdt.h>

Nothing below is using libfdt.h and ...

> +#include <xen/device_tree.h>
> +#include <xen/rangeset.h>

.. rangeset.h. So why do you include them?

> +
> +/*
> + * overlay_track describes information about added nodes through dtbo.
> + * @entry: List pointer.
> + * @dt_host_new: Pointer to the updated dt_host_new which is unflattened from
> +    the 'updated fdt'.
> + * @fdt: Stores the fdt.
> + * @overlay_fdt: Stores a copy of input overlay_fdt.
> + * @nodes_address: Stores each overlay_node's address.
> + * @num_nodes: Total number of nodes in overlay dtb.
> + */
> +struct overlay_track {
> +    struct list_head entry;
> +    struct dt_device_node *dt_host_new;
> +    void *fdt;
> +    void *overlay_fdt;
> +    unsigned long *nodes_address;
> +    unsigned int num_nodes;
> +};
> +
> +struct xen_sysctl_dt_overlay;
> +
> +#ifdef CONFIG_OVERLAY_DTB
> +long dt_sysctl(struct xen_sysctl_dt_overlay *op);
> +#else
> +#include <xen/errno.h>
> +static inline long dt_sysctl(struct xen_sysctl_dt_overlay *op)
> +{
> +    return -EOPNOTSUPP;
> +}
> +#endif
> +
> +#endif /* __XEN_DT_OVERLAY_H__ */
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */

Cheers,

[1] 
https://lore.kernel.org/all/478d4c20-31b7-e98d-25c1-4b4e9afe7e0a@xen.org/

-- 
Julien Grall

Re: [XEN][PATCH v7 02/19] common/device_tree.c: unflatten_device_tree() propagate errors

[Vikram Garhwal]

Hi Michal,

On 6/2/23 12:14 AM, Michal Orzel wrote:
> Title: s/unflatten_device_tree/__unflatten_device_tree/ or you mean to propagate
> errors from unflatten_dt_node?
>
> On 02/06/2023 02:48, Vikram Garhwal wrote:
>> This will be useful in dynamic node programming when new dt nodes are unflattend
>> during runtime. Invalid device tree node related errors should be propagated
>> back to the caller.
>>
>> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
>> Reviewed-by: Henry Wang <Henry.Wang@arm.com>
>> ---
>>   xen/common/device_tree.c | 15 +++++++++++++--
>>   1 file changed, 13 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
>> index dfdb10e488..117ccccb09 100644
>> --- a/xen/common/device_tree.c
>> +++ b/xen/common/device_tree.c
>> @@ -2108,6 +2108,9 @@ static int __init __unflatten_device_tree(const void *fdt,
>>       /* First pass, scan for size */
>>       start = ((unsigned long)fdt) + fdt_off_dt_struct(fdt);
>>       size = unflatten_dt_node(fdt, 0, &start, NULL, NULL, 0);
>> +    if ( !size )
>> +        return -EINVAL;
>> +
>>       size = (size | 3) + 1;
>>   
>>       dt_dprintk("  size is %#lx allocating...\n", size);
>> @@ -2125,11 +2128,19 @@ static int __init __unflatten_device_tree(const void *fdt,
>>       start = ((unsigned long)fdt) + fdt_off_dt_struct(fdt);
>>       unflatten_dt_node(fdt, mem, &start, NULL, &allnextp, 0);
>>       if ( be32_to_cpup((__be32 *)start) != FDT_END )
>> -        printk(XENLOG_WARNING "Weird tag at end of tree: %08x\n",
>> +    {
>> +        printk(XENLOG_ERR "Weird tag at end of tree: %08x\n",
>>                     *((u32 *)start));
>> +        return -EINVAL;
> What about memory that we allocated? Shouldn't it be freed in case of these two errors?
> For now it is called from boot so we do panic but later in this series this could result
> in a memory leak. Am I correct?
Yeah, that's correct. Let me add the memory free handling in v8.

>> +    }
>> +
>>       if ( be32_to_cpu(((__be32 *)mem)[size / 4]) != 0xdeadbeef )
>> -        printk(XENLOG_WARNING "End of tree marker overwritten: %08x\n",
>> +    {
>> +        printk(XENLOG_ERR "End of tree marker overwritten: %08x\n",
>>                     be32_to_cpu(((__be32 *)mem)[size / 4]));
>> +        return -EINVAL;
>> +    }
>> +
>>       *allnextp = NULL;
>>   
>>       dt_dprintk(" <- unflatten_device_tree()\n");
> ~Michal

Re: [XEN][PATCH v7 04/19] common/device_tree: change __unflatten_device_tree() type

[Vikram Garhwal]

Hi Julien,
Will update the commit message regarding why we need to export this for 
dtbo programming.

On 6/5/23 12:04 PM, Julien Grall wrote:
> Hi,
>
> Title:
>
> 'type' is a bit confusing here. How about "Export 
> __unflatten_device_tre()"?
>
> On 02/06/2023 01:48, Vikram Garhwal wrote:
>> Following changes are done to __unflatten_device_tree():
>>      1. __unflatten_device_tree() is renamed to unflatten_device_tree().
>>      2. Remove __init and static function type.
>
> As there is no external caller yet, please explain why you want to 
> export the function.
>
> Cheers,
>
>>
>> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
>> Reviewed-by: Henry Wang <Henry.Wang@arm.com>
>> ---
>>   xen/common/device_tree.c      | 9 ++++-----
>>   xen/include/xen/device_tree.h | 5 +++++
>>   2 files changed, 9 insertions(+), 5 deletions(-)
>>
>> diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
>> index bbdab07596..16b4b4e946 100644
>> --- a/xen/common/device_tree.c
>> +++ b/xen/common/device_tree.c
>> @@ -2083,7 +2083,7 @@ static unsigned long unflatten_dt_node(const 
>> void *fdt,
>>   }
>>     /**
>> - * __unflatten_device_tree - create tree of device_nodes from flat blob
>> + * unflatten_device_tree - create tree of device_nodes from flat blob
>>    *
>>    * unflattens a device-tree, creating the
>>    * tree of struct device_node. It also fills the "name" and "type"
>> @@ -2092,8 +2092,7 @@ static unsigned long unflatten_dt_node(const 
>> void *fdt,
>>    * @fdt: The fdt to expand
>>    * @mynodes: The device_node tree created by the call
>>    */
>> -static int __init __unflatten_device_tree(const void *fdt,
>> -                                          struct dt_device_node 
>> **mynodes)
>> +int unflatten_device_tree(const void *fdt, struct dt_device_node 
>> **mynodes)
>>   {
>>       unsigned long start, mem, size;
>>       struct dt_device_node **allnextp = mynodes;
>> @@ -2230,10 +2229,10 @@ dt_find_interrupt_controller(const struct 
>> dt_device_match *matches)
>>     void __init dt_unflatten_host_device_tree(void)
>>   {
>> -    int error = __unflatten_device_tree(device_tree_flattened, 
>> &dt_host);
>> +    int error = unflatten_device_tree(device_tree_flattened, &dt_host);
>>         if ( error )
>> -        panic("__unflatten_device_tree failed with error %d\n", error);
>> +        panic("unflatten_device_tree failed with error %d\n", error);
>>         dt_alias_scan();
>
> This function doesn't seem to be called in the case of the overlay 
> device-tree. Does this mean that it will never contain any alias?
>
>>   }
>> diff --git a/xen/include/xen/device_tree.h 
>> b/xen/include/xen/device_tree.h
>> index c2eada7489..2c35c0d391 100644
>> --- a/xen/include/xen/device_tree.h
>> +++ b/xen/include/xen/device_tree.h
>> @@ -178,6 +178,11 @@ int device_tree_for_each_node(const void *fdt, 
>> int node,
>>    */
>>   void dt_unflatten_host_device_tree(void);
>>   +/**
>> + * unflatten any device tree.
>
> Most of the exported function in device_tre.h have documentation. Can 
> you do the same here?
>
>> + */
>> +int unflatten_device_tree(const void *fdt, struct dt_device_node 
>> **mynodes);
>
> NIT: From an external interface perspective, do we actually need to 
> pass an extra pointer? Can't we instead, return the pointer?
>
>> +
>>   /**
>>    * IRQ translation callback
>>    * TODO: For the moment we assume that we only have ONE
>
> Cheers,
>

Re: [XEN][PATCH v7 05/19] xen/arm: Add CONFIG_OVERLAY_DTB

[Vikram Garhwal]

Thanks, will update the Changelog.md with Jan's suggestions.

On 6/2/23 2:22 AM, Henry Wang wrote:
> Hi Jan and Vikram,
>
>> -----Original Message-----
>> Subject: Re: [XEN][PATCH v7 05/19] xen/arm: Add CONFIG_OVERLAY_DTB
>>
>>> diff --git a/CHANGELOG.md b/CHANGELOG.md
>>> index 5bfd3aa5c0..a137fce576 100644
>>> --- a/CHANGELOG.md
>>> +++ b/CHANGELOG.md
>>> @@ -20,6 +20,8 @@ The format is based on [Keep a
>> Changelog](https://keepachangelog.com/en/1.0.0/)
>>>      - Bus-lock detection, used by Xen to mitigate (by rate-limiting) the system
>>>        wide impact of a guest misusing atomic instructions.
>>>    - xl/libxl can customize SMBIOS strings for HVM guests.
>>> + - On Arm, support for dynamic addition/removal of Xen device tree nodes
>> using
>>> +   a device tree overlay binary(.dtbo).
>> May I suggest "..., experimental support ..." here?
> Great point! I agree using "experimental support" is better here.
>
> @Vikram: Just to be clear, if you agree and change the wording following Jan's
> suggestion, you can keep my acked-by tag for the changelog :))
>
> Kind regards,
> Henry
>
>> Jan

Re: [XEN][PATCH v7 08/19] xen/device-tree: Add device_tree_find_node_by_path() to find nodes in device tree

[Vikram Garhwal]

Hi,

On 6/5/23 12:12 PM, Julien Grall wrote:
> Hi,
>
> On 02/06/2023 01:48, Vikram Garhwal wrote:
>> Add device_tree_find_node_by_path() to find a matching node with path 
>> for a
>
> AFAICT, the only difference in name between the new function and the 
> existing one is "device_tree" vs "dt". The latter is just a shorthand 
> of "device tree", so it feels to me the name are a bit too similar.
>
> From my understanding, the main difference between the two functions 
> are that the current one is starting from root whereas the current one 
> is starting from a given node. So how about 
> "dt_find_node_by_path_from()"?
Thank you for the suggestion. This name was added in v3 as Luca Fancellu 
suggested to rename this function to "device_tree_find_node_by_path". I 
am okay with renaming it to dt_find_node_by_path_from().

Luca, Michal and Henry: Does the dt_find_node_by_path_from() name works 
for you?

Regards,
Vikram
>
>> dt_device_node.
>>
>> Reason behind this function:
>>      Each time overlay nodes are added using .dtbo, a new fdt(memcpy of
>>      device_tree_flattened) is created and updated with overlay 
>> nodes. This
>>      updated fdt is further unflattened to a dt_host_new. Next, we 
>> need to find
>>      the overlay nodes in dt_host_new, find the overlay node's parent 
>> in dt_host
>>      and add the nodes as child under their parent in the dt_host. 
>> Thus we need
>>      this function to search for node in different unflattened device 
>> trees.
>>
>> Also, make dt_find_node_by_path() static inline.
>>
>> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
>>
>> ---
>> Changes from v6:
>>      Rename "dt_node" to "from"
>> ---
>>   xen/common/device_tree.c      |  6 ++++--
>>   xen/include/xen/device_tree.h | 18 ++++++++++++++++--
>>   2 files changed, 20 insertions(+), 4 deletions(-)
>>
>> diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
>> index 16b4b4e946..c5250a1644 100644
>> --- a/xen/common/device_tree.c
>> +++ b/xen/common/device_tree.c
>> @@ -358,11 +358,13 @@ struct dt_device_node 
>> *dt_find_node_by_type(struct dt_device_node *from,
>>       return np;
>>   }
>>   -struct dt_device_node *dt_find_node_by_path(const char *path)
>> +struct dt_device_node *
>> +                    device_tree_find_node_by_path(struct 
>> dt_device_node *from,
>> +                                                  const char *path)
>>   {
>>       struct dt_device_node *np;
>>   -    dt_for_each_device_node(dt_host, np)
>> +    dt_for_each_device_node(from, np)
>>           if ( np->full_name && (dt_node_cmp(np->full_name, path) == 
>> 0) )
>>               break;
>>   diff --git a/xen/include/xen/device_tree.h 
>> b/xen/include/xen/device_tree.h
>> index 2c35c0d391..e239f7de26 100644
>> --- a/xen/include/xen/device_tree.h
>> +++ b/xen/include/xen/device_tree.h
>> @@ -561,13 +561,27 @@ struct dt_device_node 
>> *dt_find_node_by_type(struct dt_device_node *from,
>>   struct dt_device_node *dt_find_node_by_alias(const char *alias);
>>     /**
>> - * dt_find_node_by_path - Find a node matching a full DT path
>> + * device_tree_find_node_by_path - Generic function to find a node 
>> matching the
>> + * full DT path for any given unflatten device tree
>> + * @from: The device tree node to start searching from
>>    * @path: The full path to match
>>    *
>>    * Returns a node pointer.
>>    */
>> -struct dt_device_node *dt_find_node_by_path(const char *path);
>> +struct dt_device_node *
>> +                    device_tree_find_node_by_path(struct 
>> dt_device_node *from,
>> +                                                  const char *path);
>
> The indentation looks slightly odd. In general, if the return type is 
> on its own line, then the function name is not indented on the new line.
Will fix the indention.
>
>>   +/**
>> + * dt_find_node_by_path - Find a node matching a full DT path in 
>> dt_host
>> + * @path: The full path to match
>> + *
>> + * Returns a node pointer.
>> + */
>> +static inline struct dt_device_node *dt_find_node_by_path(const char 
>> *path)
>> +{
>> +    return device_tree_find_node_by_path(dt_host, path);
>> +}
>>     /**
>>    * dt_find_node_by_gpath - Same as dt_find_node_by_path but 
>> retrieve the
>
> Cheers,
>

Re: [XEN][PATCH v7 09/19] xen/iommu: Move spin_lock from iommu_dt_device_is_assigned to caller

[Vikram Garhwal]

Hi,

On 6/5/23 12:22 PM, Julien Grall wrote:
> Hi,
>
> On 02/06/2023 08:45, Michal Orzel wrote:
>>
>> On 02/06/2023 02:48, Vikram Garhwal wrote:
>>> diff --git a/xen/include/xen/iommu-private.h 
>>> b/xen/include/xen/iommu-private.h
>>> new file mode 100644
>>> index 0000000000..5615decaff
>>> --- /dev/null
>>> +++ b/xen/include/xen/iommu-private.h
>>> @@ -0,0 +1,27 @@
>>> +/* SPDX-License-Identifier: GPL-2.0-only */
>>> + /*
>> incorrect indentation (<< 1)
>>
>>> + * xen/iommu-private.h
>>> + *
>>> + *
>>> + * Copyright (C) 2023, Advanced Micro Devices, Inc. All Rights 
>>> Reserved.
>>> + * Written by Vikram Garhwal <vikram.garhwal@amd.com>
>> I'm not sure if placing the copyright is appropriate, given a single 
>> prototype for a function
>> not really implemented by you with just spinlocks removed. But I 
>> might be wrong.
>
> I agree. If you want to add a copyright then it should be the one from 
> the original author.
>
> But in this case, I don't think we should add an copyright as there is 
> nothing really interesting in the header.

Thanks for pointing this out and explaining the process. Will remove in 
next version.
>
> Cheers,
>

Re: [XEN][PATCH v7 08/19] xen/device-tree: Add device_tree_find_node_by_path() to find nodes in device tree

[Michal Orzel]

On 06/06/2023 22:29, Vikram Garhwal wrote:
> Hi,
> 
> On 6/5/23 12:12 PM, Julien Grall wrote:
>> Hi,
>>
>> On 02/06/2023 01:48, Vikram Garhwal wrote:
>>> Add device_tree_find_node_by_path() to find a matching node with path 
>>> for a
>>
>> AFAICT, the only difference in name between the new function and the 
>> existing one is "device_tree" vs "dt". The latter is just a shorthand 
>> of "device tree", so it feels to me the name are a bit too similar.
>>
>> From my understanding, the main difference between the two functions 
>> are that the current one is starting from root whereas the current one 
>> is starting from a given node. So how about 
>> "dt_find_node_by_path_from()"?
> Thank you for the suggestion. This name was added in v3 as Luca Fancellu 
> suggested to rename this function to "device_tree_find_node_by_path". I 
> am okay with renaming it to dt_find_node_by_path_from().
> 
> Luca, Michal and Henry: Does the dt_find_node_by_path_from() name works 
> for you?
Works for me.

~Michal

RE: [XEN][PATCH v7 08/19] xen/device-tree: Add device_tree_find_node_by_path() to find nodes in device tree

[Henry Wang]

Hi Vikram,

> -----Original Message-----
> >> is starting from a given node. So how about
> >> "dt_find_node_by_path_from()"?
> > Thank you for the suggestion. This name was added in v3 as Luca Fancellu
> > suggested to rename this function to "device_tree_find_node_by_path". I
> > am okay with renaming it to dt_find_node_by_path_from().
> >
> > Luca, Michal and Henry: Does the dt_find_node_by_path_from() name
> works
> > for you?
> Works for me.

+1

Kind regards,
Henry

> 
> ~Michal

Re: [XEN][PATCH v7 08/19] xen/device-tree: Add device_tree_find_node_by_path() to find nodes in device tree

[Luca Fancellu]

> On 6 Jun 2023, at 21:29, Vikram Garhwal <vikram.garhwal@amd.com> wrote:
> 
> Hi,
> 
> On 6/5/23 12:12 PM, Julien Grall wrote:
>> Hi,
>> 
>> On 02/06/2023 01:48, Vikram Garhwal wrote:
>>> Add device_tree_find_node_by_path() to find a matching node with path for a
>> 
>> AFAICT, the only difference in name between the new function and the existing one is "device_tree" vs "dt". The latter is just a shorthand of "device tree", so it feels to me the name are a bit too similar.
>> 
>> From my understanding, the main difference between the two functions are that the current one is starting from root whereas the current one is starting from a given node. So how about "dt_find_node_by_path_from()"?
> Thank you for the suggestion. This name was added in v3 as Luca Fancellu suggested to rename this function to "device_tree_find_node_by_path". I am okay with renaming it to dt_find_node_by_path_from().
> 
> Luca, Michal and Henry: Does the dt_find_node_by_path_from() name works for you?

Sure, go for it!

Cheers,
Luca

Re: [XEN][PATCH v7 17/19] tools/libs/ctrl: Implement new xc interfaces for dt overlay

[Anthony PERARD]

On Thu, Jun 01, 2023 at 05:48:22PM -0700, Vikram Garhwal wrote:
> --- /dev/null
> +++ b/tools/libs/ctrl/xc_dt_overlay.c
> +int xc_dt_overlay(xc_interface *xch, void *overlay_fdt,
> +                  uint32_t overlay_fdt_size, uint8_t overlay_op)
> +{
> +    int err;
> +    DECLARE_SYSCTL;
> +
> +    DECLARE_HYPERCALL_BOUNCE(overlay_fdt, overlay_fdt_size,
> +                             XC_HYPERCALL_BUFFER_BOUNCE_IN);
> +
> +    if ( (err = xc_hypercall_bounce_pre(xch, overlay_fdt)) )
> +        goto err;
> +
> +    sysctl.cmd = XEN_SYSCTL_dt_overlay;
> +    sysctl.u.dt_overlay.overlay_op = overlay_op;
> +    sysctl.u.dt_overlay.overlay_fdt_size = overlay_fdt_size;
> +    sysctl.u.dt_overlay.pad[0]= 0;
> +    sysctl.u.dt_overlay.pad[1]= 0;
> +    sysctl.u.dt_overlay.pad[2]= 0;

This is kind of silly to have to write explicitly 0 to each pads,
there's probably a better way.

I haven't find any example for "sysctl" but they are plenty for
"domctl", and I think this is maybe something that's wanted, so could
you replace "DECLARE_SYSCTL" by:

    struct xen_sysctl sysctl = {
        .cmd = XEN_SYSCTL_dt_overlay,
        .u.dt_overlay = {
            .overlay_op = overlay_op,
            .overlay_fdt_size = overlay_fdt_size,
        }
    };

With that, I don't think there would be need to set a value for "pad",
it would default to 0. And it maybe looks cleaner?

With that: Reviewed-by: Anthony PERARD <anthony.perard@citrix.com>

Thanks,

-- 
Anthony PERARD

Re: [XEN][PATCH v7 04/19] common/device_tree: change __unflatten_device_tree() type

[Vikram Garhwal]

On Tue, Jun 06, 2023 at 12:09:35PM -0700, Vikram Garhwal wrote:
> Hi Julien,
> Will update the commit message regarding why we need to export this for dtbo
> programming.
> 
> On 6/5/23 12:04 PM, Julien Grall wrote:
> > Hi,
> > 
> > Title:
> > 
> > 'type' is a bit confusing here. How about "Export
> > __unflatten_device_tre()"?
> > 
> > On 02/06/2023 01:48, Vikram Garhwal wrote:
> > > Following changes are done to __unflatten_device_tree():
> > >      1. __unflatten_device_tree() is renamed to unflatten_device_tree().
> > >      2. Remove __init and static function type.
> > 
> > As there is no external caller yet, please explain why you want to
> > export the function.
Update the commit message in v8.
> > 
> > Cheers,
> > 
> > > 
> > > Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> > > Reviewed-by: Henry Wang <Henry.Wang@arm.com>
> > > ---
> > >   xen/common/device_tree.c      | 9 ++++-----
> > >   xen/include/xen/device_tree.h | 5 +++++
> > >   2 files changed, 9 insertions(+), 5 deletions(-)
> > > 
> > > diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
> > > index bbdab07596..16b4b4e946 100644
> > > --- a/xen/common/device_tree.c
> > > +++ b/xen/common/device_tree.c
> > > @@ -2083,7 +2083,7 @@ static unsigned long unflatten_dt_node(const
> > > void *fdt,
> > >   }
> > >     /**
> > > - * __unflatten_device_tree - create tree of device_nodes from flat blob
> > > + * unflatten_device_tree - create tree of device_nodes from flat blob
> > >    *
> > >    * unflattens a device-tree, creating the
> > >    * tree of struct device_node. It also fills the "name" and "type"
> > > @@ -2092,8 +2092,7 @@ static unsigned long unflatten_dt_node(const
> > > void *fdt,
> > >    * @fdt: The fdt to expand
> > >    * @mynodes: The device_node tree created by the call
> > >    */
> > > -static int __init __unflatten_device_tree(const void *fdt,
> > > -                                          struct dt_device_node
> > > **mynodes)
> > > +int unflatten_device_tree(const void *fdt, struct dt_device_node
> > > **mynodes)
> > >   {
> > >       unsigned long start, mem, size;
> > >       struct dt_device_node **allnextp = mynodes;
> > > @@ -2230,10 +2229,10 @@ dt_find_interrupt_controller(const struct
> > > dt_device_match *matches)
> > >     void __init dt_unflatten_host_device_tree(void)
> > >   {
> > > -    int error = __unflatten_device_tree(device_tree_flattened,
> > > &dt_host);
> > > +    int error = unflatten_device_tree(device_tree_flattened, &dt_host);
> > >         if ( error )
> > > -        panic("__unflatten_device_tree failed with error %d\n", error);
> > > +        panic("unflatten_device_tree failed with error %d\n", error);
> > >         dt_alias_scan();
> > 
> > This function doesn't seem to be called in the case of the overlay
> > device-tree. Does this mean that it will never contain any alias?
> > 
I haven't seen any overlay example for FPGA use cases where alias are added.
I have added a TODO in patch 16/19 where we are calling unflatten_device_tree().
> > >   }
> > > diff --git a/xen/include/xen/device_tree.h
> > > b/xen/include/xen/device_tree.h
> > > index c2eada7489..2c35c0d391 100644
> > > --- a/xen/include/xen/device_tree.h
> > > +++ b/xen/include/xen/device_tree.h
> > > @@ -178,6 +178,11 @@ int device_tree_for_each_node(const void *fdt,
> > > int node,
> > >    */
> > >   void dt_unflatten_host_device_tree(void);
> > >   +/**
> > > + * unflatten any device tree.
> > 
> > Most of the exported function in device_tre.h have documentation. Can
> > you do the same here?
Done!
> > 
> > > + */
> > > +int unflatten_device_tree(const void *fdt, struct dt_device_node
> > > **mynodes);
> > 
> > NIT: From an external interface perspective, do we actually need to pass
> > an extra pointer? Can't we instead, return the pointer?
We will also need the error from the function. So, that's why i kept it as it is.
Please review v8. I will send it in few hours.
> > 
> > > +
> > >   /**
> > >    * IRQ translation callback
> > >    * TODO: For the moment we assume that we only have ONE
> > 
> > Cheers,
> > 
> 

Re: [XEN][PATCH v7 09/19] xen/iommu: Move spin_lock from iommu_dt_device_is_assigned to caller

[Vikram Garhwal]

On Mon, Jun 05, 2023 at 08:19:38PM +0100, Julien Grall wrote:
> Hi,
> 
> On 02/06/2023 01:48, Vikram Garhwal wrote:
> > Rename iommu_dt_device_is_assigned() to iommu_dt_device_is_assigned_locked().
> > Remove static type so this can also be used by SMMU drivers to check if the
> > device is being used before removing.
> > 
> > Moving spin_lock to caller was done to prevent the concurrent access to
> > iommu_dt_device_is_assigned while doing add/remove/assign/deassign.
> 
> Can you explain if you are trying to resolve an existing bug, or this is
> something that will be necessary in a follow-up patch?
Updated for v8.
> 
> > 
> > Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> > 
> > ---
> > Changes from v6:
> >      Created a private header and moved iommu_dt_device_is_assigned() to header.
> > ---
> >   xen/drivers/passthrough/device_tree.c | 20 ++++++++++++++++----
> >   xen/include/xen/iommu-private.h       | 27 +++++++++++++++++++++++++++
> >   2 files changed, 43 insertions(+), 4 deletions(-)
> >   create mode 100644 xen/include/xen/iommu-private.h
> > 
> > diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
> > index 1c32d7b50c..52e370db01 100644
> > --- a/xen/drivers/passthrough/device_tree.c
> > +++ b/xen/drivers/passthrough/device_tree.c
> > @@ -18,6 +18,7 @@
> >   #include <xen/device_tree.h>
> >   #include <xen/guest_access.h>
> >   #include <xen/iommu.h>
> > +#include <xen/iommu-private.h>
> >   #include <xen/lib.h>
> >   #include <xen/sched.h>
> >   #include <xsm/xsm.h>
> > @@ -83,16 +84,14 @@ fail:
> >       return rc;
> >   }
> > -static bool_t iommu_dt_device_is_assigned(const struct dt_device_node *dev)
> > +bool_t iommu_dt_device_is_assigned_locked(const struct dt_device_node *dev)
> >   {
> 
> Please add an ASSERT() checking the lock is taken.
> 
Done in v8.
> >       bool_t assigned = 0;
> >       if ( !dt_device_is_protected(dev) )
> >           return 0;
> > -    spin_lock(&dtdevs_lock);
> >       assigned = !list_empty(&dev->domain_list);
> > -    spin_unlock(&dtdevs_lock);
> >       return assigned;
> >   }
> > @@ -213,27 +212,40 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
> >           if ( (d && d->is_dying) || domctl->u.assign_device.flags )
> >               break;
> > +        spin_lock(&dtdevs_lock);
> > +
> 
> 'dtdevs_lock' was intended to protect modification related to any IOMMU
> change. But here...
> 
> >           ret = dt_find_node_by_gpath(domctl->u.assign_device.u.dt.path,
> >                                       domctl->u.assign_device.u.dt.size,
> >                                       &dev);
> 
> ... you also include "dt_find_node_by_gpath". Can you explain why and add a
> comment on top of 'dtdevs_lock' to explain what it is intended use?
I have added a comment in v8. There was a comment in v3:
"ensure that the "dev" doesn't disappear between the time we look it up". So,
i moved the lock here and for dt_host the lock is added in follow-up patch:
"common/device_tree: Add rwlock for dt_host". So, this all will happen with
dtdevs_lock and dt_host_lock.
> 
> >           if ( ret )
> > +        {
> > +            spin_unlock(&dtdevs_lock);
> >               break;
> > +        }
> >           ret = xsm_assign_dtdevice(XSM_HOOK, d, dt_node_full_name(dev));
> >           if ( ret )
> > +        {
> > +            spin_unlock(&dtdevs_lock);
> >               break;
> > +        }
> >           if ( domctl->cmd == XEN_DOMCTL_test_assign_device )
> >           {
> > -            if ( iommu_dt_device_is_assigned(dev) )
> > +
> > +            if ( iommu_dt_device_is_assigned_locked(dev) )
> >               {
> >                   printk(XENLOG_G_ERR "%s already assigned.\n",
> >                          dt_node_full_name(dev));
> >                   ret = -EINVAL;
> >               }
> > +
> > +            spin_unlock(&dtdevs_lock);
> >               break;
> >           }
> > +        spin_unlock(&dtdevs_lock);
> > +
> >           if ( d == dom_io )
> >               return -EINVAL;
> > diff --git a/xen/include/xen/iommu-private.h b/xen/include/xen/iommu-private.h
> > new file mode 100644
> > index 0000000000..5615decaff
> > --- /dev/null
> > +++ b/xen/include/xen/iommu-private.h
> > @@ -0,0 +1,27 @@
> > +/* SPDX-License-Identifier: GPL-2.0-only */
> > + /*
> > + * xen/iommu-private.h
> > + *
> > + *
> > + * Copyright (C) 2023, Advanced Micro Devices, Inc. All Rights Reserved.
> > + * Written by Vikram Garhwal <vikram.garhwal@amd.com>
> > + *
> > + */
> > +#ifndef __XEN_IOMMU_PRIVATE_H__
> > +#define __XEN_IOMMU_PRIVATE_H__
> > +
> > +#ifdef CONFIG_HAS_DEVICE_TREE
> > +#include <xen/device_tree.h>
> > +bool_t iommu_dt_device_is_assigned_locked(const struct dt_device_node *dev);
> > +#endif
> > +
> > +#endif /* __XEN_IOMMU_PRIVATE_H__ */
> > +/*
> > + * Local variables:
> > + * mode: C
> > + * c-file-style: "BSD"
> > + * c-basic-offset: 4
> > + * tab-width: 4
> > + * indent-tabs-mode: nil
> > + * End:
> > + */
> 
> -- 
> Julien Grall

Re: [XEN][PATCH v7 11/19] xen/iommu: Introduce iommu_remove_dt_device()

[Vikram Garhwal]

On Mon, Jun 05, 2023 at 08:37:03PM +0100, Julien Grall wrote:
> Hi,
> 
> On 02/06/2023 01:48, Vikram Garhwal wrote:
> > Remove master device from the IOMMU. This will be helpful when removing the
> > overlay nodes using dynamic programming during run time.
> > 
> > Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> > Reviewed-by: Michal Orzel <michal.orzel@amd.com>
> > Acked-by: Jan Beulich <jbeulich@suse.com>
> > ---
> >   xen/drivers/passthrough/device_tree.c | 41 +++++++++++++++++++++++++++
> >   xen/include/xen/iommu.h               |  2 ++
> >   2 files changed, 43 insertions(+)
> > 
> > diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
> > index 8cc413f867..301a5bcd97 100644
> > --- a/xen/drivers/passthrough/device_tree.c
> > +++ b/xen/drivers/passthrough/device_tree.c
> > @@ -126,6 +126,47 @@ int iommu_release_dt_devices(struct domain *d)
> >       return 0;
> >   }
> > +int iommu_remove_dt_device(struct dt_device_node *np)
> > +{
> > +    const struct iommu_ops *ops = iommu_get_ops();
> > +    struct device *dev = dt_to_dev(np);
> > +    int rc;
> > +
> 
> iommu_add_dt_device() checks if the IOMMU is enabled. I think you should do
> the same here as well and return 0 if it is disabled.
Added iommu_enabled check in v8.
> 
> > +    if ( !ops )
> > +        return -EOPNOTSUPP;
> > +
> > +    spin_lock(&dtdevs_lock);
> > +
> > +    if ( iommu_dt_device_is_assigned_locked(np) )
> > +    {
> > +        rc = -EBUSY;
> > +        goto fail;
> > +    }
> > +
> > +    /*
> > +     * The driver which supports generic IOMMU DT bindings must have this
> > +     * callback implemented.
> > +     */
> 
> It is not clear to me why you want to mandate remove_device when using the
> generic IOMMU DT bindings.
> 
> But if this is really necessary, then I think the comment should be placed
> on top of the callback definition rather than in the caller.
Added a comment on top of remove_generic in smmu.c
> 
> > +    if ( !ops->remove_device )
> > +    {
> > +        rc = -EOPNOTSUPP;
> > +        goto fail;
> > +    }
> > +
> > +    /*
> > +     * Remove master device from the IOMMU if latter is present and available.
> > +     * The driver is responsible for removing is_protected flag.
> > +     */
> > +    rc = ops->remove_device(0, dev);
> > +
> > +    if ( !rc )
> > +        iommu_fwspec_free(dev);
> > +
> > +fail:
> > +    spin_unlock(&dtdevs_lock);
> > +    return rc;
> > +}
> > +
> >   int iommu_add_dt_device(struct dt_device_node *np)
> >   {
> >       const struct iommu_ops *ops = iommu_get_ops();
> > diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
> > index 405db59971..0d7924821b 100644
> > --- a/xen/include/xen/iommu.h
> > +++ b/xen/include/xen/iommu.h
> > @@ -229,6 +229,8 @@ int iommu_release_dt_devices(struct domain *d);
> >    */
> >   int iommu_add_dt_device(struct dt_device_node *np);
> > +int iommu_remove_dt_device(struct dt_device_node *np);
> > +
> >   int iommu_do_dt_domctl(struct xen_domctl *, struct domain *,
> >                          XEN_GUEST_HANDLE_PARAM(xen_domctl_t));
> 
> Cheers,
> 
> -- 
> Julien Grall

Re: [XEN][PATCH v7 14/19] common/device_tree: Add rwlock for dt_host

[Vikram Garhwal]

On Mon, Jun 05, 2023 at 08:52:17PM +0100, Julien Grall wrote:
> Hi,
> 
> On 02/06/2023 01:48, Vikram Garhwal wrote:
> >   Dynamic programming ops will modify the dt_host and there might be other
> >   function which are browsing the dt_host at the same time. To avoid the race
> >   conditions, adding rwlock for browsing the dt_host during runtime.
> Please explain that writer will be added in a follow-up patch.
> 
> > 
> >   Reason behind adding rwlock instead of spinlock:
> >      For now, dynamic programming is the sole modifier of dt_host in Xen during
> >          run time. All other access functions like iommu_release_dt_device() are
> >          just reading the dt_host during run-time. So, there is a need to protect
> >          others from browsing the dt_host while dynamic programming is modifying
> >          it. rwlock is better suitable for this task as spinlock won't be able to
> >          differentiate between read and write access.
> > 
> > Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> > 
> > ---
> > Changes from v6:
> >      Remove redundant "read_unlock(&dt_host->lock);" in the following case:
> >           XEN_DOMCTL_deassign_device
> > ---
> >   xen/common/device_tree.c              |  4 ++++
> >   xen/drivers/passthrough/device_tree.c | 15 +++++++++++++++
> >   xen/include/xen/device_tree.h         |  6 ++++++
> >   3 files changed, 25 insertions(+)
> > 
> > diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
> > index c5250a1644..c8fcdf8fa1 100644
> > --- a/xen/common/device_tree.c
> > +++ b/xen/common/device_tree.c
> > @@ -2146,7 +2146,11 @@ int unflatten_device_tree(const void *fdt, struct dt_device_node **mynodes)
> >       dt_dprintk(" <- unflatten_device_tree()\n");
> > +    /* Init r/w lock for host device tree. */
> > +    rwlock_init(&dt_host->lock);
> > +
> >       return 0;
> > +
> >   }
> >   static void dt_alias_add(struct dt_alias_prop *ap,
> > diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
> > index 301a5bcd97..f4d9deb624 100644
> > --- a/xen/drivers/passthrough/device_tree.c
> > +++ b/xen/drivers/passthrough/device_tree.c
> > @@ -112,6 +112,8 @@ int iommu_release_dt_devices(struct domain *d)
> >       if ( !is_iommu_enabled(d) )
> >           return 0;
> > +    read_lock(&dt_host->lock);
> > +
> >       list_for_each_entry_safe(dev, _dev, &hd->dt_devices, domain_list)
> >       {
> >           rc = iommu_deassign_dt_device(d, dev);
> > @@ -119,10 +121,14 @@ int iommu_release_dt_devices(struct domain *d)
> >           {
> >               dprintk(XENLOG_ERR, "Failed to deassign %s in domain %u\n",
> >                       dt_node_full_name(dev), d->domain_id);
> > +
> > +            read_unlock(&dt_host->lock);
> >               return rc;
> >           }
> >       }
> > +    read_unlock(&dt_host->lock);
> > +
> >       return 0;
> >   }
> > @@ -246,6 +252,8 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
> >       int ret;
> >       struct dt_device_node *dev;
> > +    read_lock(&dt_host->lock);
> > +
> >       switch ( domctl->cmd )
> >       {
> >       case XEN_DOMCTL_assign_device:
> > @@ -295,7 +303,10 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
> >           spin_unlock(&dtdevs_lock);
> >           if ( d == dom_io )
> > +        {
> > +            read_unlock(&dt_host->lock);
> >               return -EINVAL;
> > +        }
> >           ret = iommu_add_dt_device(dev);
> >           if ( ret < 0 )
> > @@ -333,7 +344,10 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
> >               break;
> >           if ( d == dom_io )
> > +        {
> > +            read_unlock(&dt_host->lock);
> >               return -EINVAL;
> > +        }
> >           ret = iommu_deassign_dt_device(d, dev);
> > @@ -348,5 +362,6 @@ int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
> >           break;
> >       }
> > +    read_unlock(&dt_host->lock);
> >       return ret;
> >   }
> > diff --git a/xen/include/xen/device_tree.h b/xen/include/xen/device_tree.h
> > index e239f7de26..dee40d2ea3 100644
> > --- a/xen/include/xen/device_tree.h
> > +++ b/xen/include/xen/device_tree.h
> > @@ -18,6 +18,7 @@
> >   #include <xen/string.h>
> >   #include <xen/types.h>
> >   #include <xen/list.h>
> > +#include <xen/rwlock.h>
> >   #define DEVICE_TREE_MAX_DEPTH 16
> > @@ -106,6 +107,11 @@ struct dt_device_node {
> >       struct list_head domain_list;
> >       struct device dev;
> > +
> > +    /*
> > +     * Lock that protects r/w updates to unflattened device tree i.e. dt_host.
> > +     */
> 
> From the description, it sounds like the rwlock will only be used to protect
> the entire device-tree rather than a single node. So it doesn't seem to be
> sensible to increase each node structure (there are a lot) by 12 bytes.
> 
> Can you outline your plan?
Yeah, so intent is to protect the dt_host as whole instead of each node.
I moved it out of struct and kept a single lock for dt_host.
> 
> > +    rwlock_t lock;
> >   };
> >   #define dt_to_dev(dt_node)  (&(dt_node)->dev)
> 
> Cheers,
> 
> -- 
> Julien Grall

Re: [XEN][PATCH v7 15/19] xen/arm: Implement device tree node removal functionalities

[Vikram Garhwal]

On Mon, Jun 05, 2023 at 10:07:48PM +0100, Julien Grall wrote:
> Hi Vikram,
> 
> On 02/06/2023 01:48, Vikram Garhwal wrote:
> > Introduce sysctl XEN_SYSCTL_dt_overlay to remove device-tree nodes added using
> > device tree overlay.
> > 
> > xl dt-overlay remove file.dtbo:
> >      Removes all the nodes in a given dtbo.
> >      First, removes IRQ permissions and MMIO accesses. Next, it finds the nodes
> >      in dt_host and delete the device node entries from dt_host.
> > 
> >      The nodes get removed only if it is not used by any of dom0 or domio.
> > 
> > Also, added overlay_track struct to keep the track of added node through device
> > tree overlay. overlay_track has dt_host_new which is unflattened form of updated
> > fdt and name of overlay nodes. When a node is removed, we also free the memory
> > used by overlay_track for the particular overlay node.
> > 
> > Nested overlay removal is supported in sequential manner only i.e. if
> > overlay_child nests under overlay_parent, it is assumed that user first removes
> > overlay_child and then removes overlay_parent.
> > 
> > Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
> > 
> > ---
> > Changes from v6:
> >      Add explicit padding for xen_system_dt_overlay{}
> >      Update license.
> >      Rearrange xfree in dt_sysctl()
> >      Update overlay_track struct comment with relevant message.
> >      Fix missing xen/errno.h for builds without CONFIG_OVERLAY_DTB cases.
> >      Fix header formatting.
> > ---
> >   xen/arch/arm/sysctl.c        |  16 +-
> >   xen/common/Makefile          |   1 +
> >   xen/common/dt-overlay.c      | 420 +++++++++++++++++++++++++++++++++++
> 
> I think the new dt-overlay.c wants to be added under the section
> "Device-Tree" in MAINTAINERS.
Updated MAINTAINER in v8.
> 
> >   xen/include/public/sysctl.h  |  24 ++
> >   xen/include/xen/dt-overlay.h |  59 +++++
> >   5 files changed, 519 insertions(+), 1 deletion(-)
> >   create mode 100644 xen/common/dt-overlay.c
> >   create mode 100644 xen/include/xen/dt-overlay.h
> > 
> > diff --git a/xen/arch/arm/sysctl.c b/xen/arch/arm/sysctl.c
> > index b0a78a8b10..8b813c970f 100644
> > --- a/xen/arch/arm/sysctl.c
> > +++ b/xen/arch/arm/sysctl.c
> > @@ -9,6 +9,7 @@
> >   #include <xen/types.h>
> >   #include <xen/lib.h>
> > +#include <xen/dt-overlay.h>
> >   #include <xen/errno.h>
> >   #include <xen/hypercall.h>
> >   #include <public/sysctl.h>
> > @@ -21,7 +22,20 @@ void arch_do_physinfo(struct xen_sysctl_physinfo *pi)
> >   long arch_do_sysctl(struct xen_sysctl *sysctl,
> >                       XEN_GUEST_HANDLE_PARAM(xen_sysctl_t) u_sysctl)
> >   {
> > -    return -ENOSYS;
> > +    long ret;
> > +
> > +    switch ( sysctl->cmd )
> > +    {
> > +    case XEN_SYSCTL_dt_overlay:
> > +        ret = dt_sysctl(&sysctl->u.dt_overlay);
> > +        break;
> > +
> > +    default:
> > +        ret = -ENOSYS;
> > +        break;
> > +    }
> > +
> > +    return ret;
> >   }
> >   /*
> > diff --git a/xen/common/Makefile b/xen/common/Makefile
> > index 46049eac35..e7e96b1087 100644
> > --- a/xen/common/Makefile
> > +++ b/xen/common/Makefile
> > @@ -8,6 +8,7 @@ obj-$(CONFIG_DEBUG_TRACE) += debugtrace.o
> >   obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o
> >   obj-$(CONFIG_IOREQ_SERVER) += dm.o
> >   obj-y += domain.o
> > +obj-$(CONFIG_OVERLAY_DTB) += dt-overlay.o
> >   obj-y += event_2l.o
> >   obj-y += event_channel.o
> >   obj-y += event_fifo.o
> > diff --git a/xen/common/dt-overlay.c b/xen/common/dt-overlay.c
> > new file mode 100644
> > index 0000000000..b2a7e441df
> > --- /dev/null
> > +++ b/xen/common/dt-overlay.c
> > @@ -0,0 +1,420 @@
> > +/* SPDX-License-Identifier: GPL-2.0-only */
> > +/*
> > + * xen/common/dt-overlay.c
> > + *
> > + * Device tree overlay support in Xen.
> > + *
> > + * Copyright (C) 2023, Advanced Micro Devices, Inc. All Rights Reserved.
> > + * Written by Vikram Garhwal <vikram.garhwal@amd.com>
> > + *
> > + */
> > +#include <asm/domain_build.h>
> > +#include <xen/dt-overlay.h>
> > +#include <xen/guest_access.h>
> > +#include <xen/iocap.h>
> > +#include <xen/xmalloc.h>
> > +
> > +static LIST_HEAD(overlay_tracker);
> > +static DEFINE_SPINLOCK(overlay_lock);
> > +
> > +/* Find last descendants of the device_node. */
> > +static struct dt_device_node *
> > +                find_last_descendants_node(struct dt_device_node *device_node)
> > +{
> > +    struct dt_device_node *child_node;
> > +
> > +    for ( child_node = device_node->child; child_node->sibling != NULL;
> > +          child_node = child_node->sibling );
> > +
> > +    /* If last child_node also have children. */
> > +    if ( child_node->child )
> > +        child_node = find_last_descendants_node(child_node);
> > +
> > +    return child_node;
> > +}
> > +
> > +static int dt_overlay_remove_node(struct dt_device_node *device_node)
> > +{
> > +    struct dt_device_node *np;
> > +    struct dt_device_node *parent_node;
> > +    struct dt_device_node *device_node_last_descendant = device_node->child;
> > +
> > +    parent_node = device_node->parent;
> > +
> > +    if ( parent_node == NULL )
> 
> Is this trying to check if we are trying to remove "/"? If not, what's this
> for?
> 
> Either way, I would suggest to add a comment explaining what this check is
> for.
Yeah, the check is to make sure we don't remove the root node. Update the code
with comment
> 
> > +    {
> > +        dt_dprintk("%s's parent node not found\n", device_node->name);
> > +        return -EFAULT;
> > +    }
> > +
> > +    np = parent_node->child;
> > +
> > +    if ( np == NULL )
> 
> Why would parent_node->child is NULL if we found the parent from
> device_node?
This was a sanity check to make sure parent->child exist to avoid accessing a
NULL pointer.
> 
> > +    {
> > +        dt_dprintk("parent node %s's not found\n", parent_node->name);
> > +        return -EFAULT;
> > +    }
> > +
> > +    /* If node to be removed is only child node or first child. */
> > +    if ( !dt_node_cmp(np->full_name, device_node->full_name) )
> > +    {
> > +        parent_node->child = np->sibling;
> > +
> > +        /*
> > +         * Iterate over all child nodes of device_node.
> 
> I am not sure how this matches the code below. I think it might be better to
> explain why it is necessary to find the last descendant. AFAIU, you seem to
> rely on the fact that allnext will point to the next children and every node
> in that list up to the last descandant will be under device_node.
I added explanation as comment. Hopefully, it's better now.
> 
> > Given that we are
> > +         * removing parent node, we need to remove all it's descendants too.
> > +         */
> > +        if ( device_node_last_descendant )
> > +        {
> > +            device_node_last_descendant =
> > +                                        find_last_descendants_node(device_node);
> 
> This indentation is not nice. Can we shorten the variable to
> last_descendant?
Done
> 
> > +            parent_node->allnext = device_node_last_descendant->allnext;
> > +        }
> > +        else
> > +            parent_node->allnext = np->allnext;
> > +
> > +        return 0;
> > +    }
> > +
> > +    for ( np = parent_node->child; np->sibling != NULL; np = np->sibling )
> > +    {
> > +        if ( !dt_node_cmp(np->sibling->full_name, device_node->full_name) )
> > +        {
> > +            /* Found the node. Now we remove it. */
> > +            np->sibling = np->sibling->sibling;
> > +
> > +            if ( np->child )
> > +                np = find_last_descendants_node(np);
> > +
> > +            /*
> > +             * Iterate over all child nodes of device_node. Given that we are
> > +             * removing parent node, we need to remove all it's descendants too.
> > +             */
> > +            if ( device_node_last_descendant )
> > +                device_node_last_descendant =
> > +                                        find_last_descendants_node(device_node);
> > +
> > +            if ( device_node_last_descendant )
> > +                np->allnext = device_node_last_descendant->allnext;
> > +            else
> > +                np->allnext = np->allnext->allnext;
> > +
> > +            break;
> > +        }
> > +    }
> > +
> > +    return 0;
> > +}
> > +
> > +/* Basic sanity check for the dtbo tool stack provided to Xen. */
> > +static int check_overlay_fdt(const void *overlay_fdt, uint32_t overlay_fdt_size)
> > +{
> > +    if ( (fdt_totalsize(overlay_fdt) != overlay_fdt_size) ||
> > +          fdt_check_header(overlay_fdt) )
> > +    {
> > +        printk(XENLOG_ERR "The overlay FDT is not a valid Flat Device Tree\n");
> > +        return -EINVAL;
> > +    }
> > +
> > +    return 0;
> > +}
> > +
> > +/* Count number of nodes till one level of __overlay__ tag. */
> > +static unsigned int overlay_node_count(const void *overlay_fdt)
> > +{
> > +    unsigned int num_overlay_nodes = 0;
> > +    int fragment;
> > +
> > +    fdt_for_each_subnode(fragment, overlay_fdt, 0)
> > +    {
> > +        int subnode;
> > +        int overlay;
> > +
> > +        overlay = fdt_subnode_offset(overlay_fdt, fragment, "__overlay__");
> > +
> > +        /*
> > +         * overlay value can be < 0. But fdt_for_each_subnode() loop checks for
> > +         * overlay >= 0. So, no need for a overlay>=0 check here.
> > +         */
> > +        fdt_for_each_subnode(subnode, overlay_fdt, overlay)
> > +        {
> > +            num_overlay_nodes++;
> > +        }
> > +    }
> > +
> > +    return num_overlay_nodes;
> > +}
> > +
> > +static int handle_remove_irq_iommu(struct dt_device_node *device_node)
> 
> How about remove_resources()? This would avoid to miss the fact you are
> removing MMIO mapping as well and make the name a bit more palatable.
Changed in v8.
> 
> > +{
> > +    int rc = 0;
> > +    struct domain *d = hardware_domain;
> > +    domid_t domid;
> > +    unsigned int naddr, len;
> > +    unsigned int i, nirq;
> > +
> > +    domid = dt_device_used_by(device_node);
> 
> Looking at the caller, it is not clear to me which lock is preventing the
> device to be assigned whilst you remove it.
So, this is on user to make sure the domain is not using the device.
The part 2 of this series, where we assign/deassign nodes to a running VM, there
we have the check to make sure no one is assigning it when it's being removed.
For now, can we protect it with dtdevs_lock?
> 
> > +
> > +    dt_dprintk("Checking if node %s is used by any domain\n",
> > +               device_node->full_name);
> > +
> > +    /* Remove the node if only it's assigned to domain 0 or domain io. */
> 
> Hmmm... Above, you are using hardware_domain. This may be dom0. So shouldn't
> you check against hardware_domain->domain_id?
Changed this one
> 
> > +    if ( domid != 0 && domid != DOMID_IO )
> > +    {
> > +        printk(XENLOG_ERR "Device %s is being used by domain %u. Removing nodes failed\n",
> > +               device_node->full_name, domid);
> > +        return -EINVAL;
> > +    }
> > +
> > +    dt_dprintk("Removing node: %s\n", device_node->full_name);
> > +
> > +    nirq = dt_number_of_irq(device_node); > +
> > +    /* Remove IRQ permission */
> > +    for ( i = 0; i < nirq; i++ )
> > +    {
> > +        rc = platform_get_irq(device_node, i);
> 
> As I mentioned in [1], I think that parsing the Device-Tree again when
> removing any interrupts/mappings is a bit odd as there are more possible
> failures and is more complex than necessary. I have proposed a way to do it
> with rangeset, but I can't find any reason why this wasn't done. Can you
> explain?
IIUC, range sets can work if we have only one level of node i.e. no children.
I tried in previous version to use range but it's complicated to get info in
correct order using rangeset. Example, we have three nodes, node A, B and C. A
has three child A_a, A_b and A_c. While adding the nodes, we add A first then
A_a, A_b, A_c and finally B and C. And rangeset is updated in same order but 
when we remove node, first A_c is removed followed by A_b and A_a and then A.
So, this was the problem for me on how to keep track which interrupt belong to
which node.

> 
> > +        if ( rc < 0 )
> > +        {
> > +            printk(XENLOG_ERR "Failed to get IRQ num for device node %s\n",
> > +                   device_node->full_name);
> > +            return -EINVAL;
> > +        }
> > +
> > +        if ( irq_access_permitted(d, rc) == false )
> 
> We commonly use "!irq_acess_permitted()". But, as pointed out in [1] with
> this check, it means that we would not be able to call remove again on the
> same node if the removal was partially done.
> 
> You suggested that the admin would need to reboot the platform. But this
> decision doesn't seem to documented in the commit message nor in the tools
> documentation. Can this be done?
Added in commit message.
> 
> > +        {
> > +            printk(XENLOG_ERR "IRQ %d is not routed to domain %u\n", rc,
> > +                   domid);
> 
> I am probably missing something here. From my understanding,
> irq_access_permitted() only tells you if the domain is allowed to manage the
> interrupt. So you could have two domains (the hardware domain and the guest)
> with irq_access_permitted() set. In fact, the hardware domain should always
> have it permitted.
> 
> If you want to check for the routing, then you will have to look for the
> information in the 'irq->desc'.

Yeah, this was a redundant check. removed it.
> > +            return -EINVAL;
> > +        }
> > +        /*
> > +         * TODO: We don't handle shared IRQs for now. So, it is assumed that
> > +         * the IRQs was not shared with another devices.
> > +         */
> > +        rc = irq_deny_access(d, rc);
> > +        if ( rc )
> > +        {
> > +            printk(XENLOG_ERR "unable to revoke access for irq %u for %s\n",
> > +                   i, device_node->full_name);
> > +            return rc;
> > +        }
> 
> You don't reverse the change in the routing. What would happen if the next
> overlay is updated to now pass the same device to a guest?
> 
> I would be OK if this is not handled in this series. But it should be marked
> as a TODO.
So, i explained the reason behind this in v4:
    https://www.mail-archive.com/xen-devel@lists.xenproject.org/msg137490.html
Let me copy paste here if the link doesn't work:
"The main purpose of this series to address first part of dynamic programming
i.e. making Xen aware of new device tree node which means updating the dt_host
with overlay node information. Here we are adding/removing node from dt_host,
and checking/set IOMMU and IRQ permission but never mapping them to any domain.
Right now, mapping/Un-mapping will happen only when a new domU is
created/destroyed using "xl create".
To map IOREQ and IOMMU during runtime, there will be another small series after 
this one where we will do the actual IOMMU and IRQ mapping do a running domain
and will call  unmap_mmio_regions() to remove the mapping.

Here is the patch for that series:
https://github.com/Xilinx/xen/commit/76fcd5defc1c7c375cb99ac73a4d1138aa87d474
I will clean it and send once current series is done.

Also, If you see addition part of this series, we put "skip_mapping = true"
which means map_range_to_domain()->map_range_p2mt() is never called.
The only thing function which will be called is iomem_permit_access() while
adding the node and here we are calling opposite function to remove the access."

> 
> > +    }
> > +
> > +    /* Check if iommu property exists. */
> > +    if ( dt_get_property(device_node, "iommus", &len) )
> > +    {
> > +        rc = iommu_remove_dt_device(device_node);
> > +        if ( rc != 0 && rc != -ENXIO )
> 
> Checking iommu_remove_dt_device() (I didn't check the driver
> implementation), I can't find any use of ENXIO. So can you explain in which
> condition this is meant to be returned?
There were some cases it was returning ENXIO for valid overlay with wrong IOMMU
but i cannot find that example anymore. Removed the check for now.
> 
> > +            return rc;
> > +    }
> > +
> > +    naddr = dt_number_of_address(device_node);
> > +
> > +    /* Remove mmio access. */
> > +    for ( i = 0; i < naddr; i++ )
> > +    {
> > +        uint64_t addr, size;
> > +
> > +        rc = dt_device_get_address(device_node, i, &addr, &size);
> > +        if ( rc )
> > +        {
> > +            printk(XENLOG_ERR "Unable to retrieve address %u for %s\n",
> > +                   i, dt_node_full_name(device_node));
> > +            return rc;
> > +        }
> > +
> > +        rc = iomem_deny_access(d, paddr_to_pfn(addr),
> > +                               paddr_to_pfn(PAGE_ALIGN(addr + size - 1)));
> 
> As pointed out in [1], I expect the function remove to be the inverse of
> add. Yet, you are not removing the mapping in the P2M.
> 
> I still don't think it is correct to leave the mapping around. But if you
> think it is necessary then I think this should be documented.
> 
> > +        if ( rc )
> > +        {
> > +            printk(XENLOG_ERR "Unable to remove dom%d access to"
> > +                   " 0x%"PRIx64" - 0x%"PRIx64"\n",
> > +                   d->domain_id,
> > +                   addr & PAGE_MASK, PAGE_ALIGN(addr + size) - 1);
> > +            return rc;
> > +        }
> > +
> > +    }
> > +
> > +    return rc;
> > +}
> > +
> > +/* Removes all descendants of the given node. */
> > +static int remove_all_descendant_nodes(struct dt_device_node *device_node)
> > +{
> > +    int rc = 0;
> > +    struct dt_device_node *child_node;
> > +
> > +    for ( child_node = device_node->child; child_node != NULL;
> > +         child_node = child_node->sibling )
> > +    {
> > +        if ( child_node->child )
> > +            remove_all_descendant_nodes(child_node);
> 
> AFAICT, the function could return an error. Can you explain why this is not
> checked?
Added
> 
> > +
> > +        rc = handle_remove_irq_iommu(child_node);
> > +        if ( rc )
> > +            return rc;
> > +    }
> > +
> > +    return rc;
> > +}
> > +
> > +/* Remove nodes from dt_host. */
> > +static int remove_nodes(const struct overlay_track *tracker)
> > +{
> > +    int rc = 0;
> > +    struct dt_device_node *overlay_node;
> > +    unsigned int j;
> > +
> > +    for ( j = 0; j < tracker->num_nodes; j++ )
> > +    {
> > +        overlay_node = (struct dt_device_node *)tracker->nodes_address[j];
> > +        if ( overlay_node == NULL )
> > +        {
> > +            printk(XENLOG_ERR "Device %s is not present in the tree. Removing nodes failed\n",
> > +                   overlay_node->full_name);
> > +            return -EINVAL;
> > +        }
> > +
> > +        rc = remove_all_descendant_nodes(overlay_node);
> > +
> > +        /* All children nodes are unmapped. Now remove the node itself. */
> > +        rc = handle_remove_irq_iommu(overlay_node);
> > +        if ( rc )
> > +            return rc;
> > +
> > +        read_lock(&dt_host->lock);
> Shouldn't this be a write_lock()?
Changed
> 
> > +
> > +        rc = dt_overlay_remove_node(overlay_node);
> > +        if ( rc )
> > +        {
> > +            read_unlock(&dt_host->lock);
> > +
> > +            return rc;
> > +        }
> > +
> > +        read_unlock(&dt_host->lock);
> > +    }
> > +
> > +    return rc;
> > +}
> > +
> > +/*
> > + * First finds the device node to remove. Check if the device is being used by
> > + * any dom and finally remove it from dt_host. IOMMU is already being taken care
> > + * while destroying the domain.
> > + */
> > +static long handle_remove_overlay_nodes(void *overlay_fdt,
> > +                                        uint32_t overlay_fdt_size)
> > +{
> > +    int rc;
> > +    struct overlay_track *entry, *temp, *track;
> > +    bool found_entry = false;
> > +
> > +    rc = check_overlay_fdt(overlay_fdt, overlay_fdt_size);
> > +    if ( rc )
> > +        return rc;
> > +
> > +    if ( overlay_node_count(overlay_fdt) == 0 )
> > +        return -EINVAL;
> 
> Why do you need to check the validity of the FDT when...
> 
> > +
> > +    spin_lock(&overlay_lock);
> > +
> > +    /*
> > +     * First check if dtbo is correct i.e. it should one of the dtbo which was
> > +     * used when dynamically adding the node.
> > +     * Limitation: Cases with same node names but different property are not
> > +     * supported currently. We are relying on user to provide the same dtbo
> > +     * as it was used when adding the nodes.
> > +     */
> > +    list_for_each_entry_safe( entry, temp, &overlay_tracker, entry )
> > +    {
> > +        if ( memcmp(entry->overlay_fdt, overlay_fdt, overlay_fdt_size) == 0 )
> 
> ... you compare against what we stored in Xen? If we found the entry, then
> wouldn't it mean that the FDT is valid?
> 
Updated this one!
> > +        {
> > +            track = entry;
> > +            found_entry = true;
> > +            break;
> > +        }
> > +    }
> > +
> > +    if ( found_entry == false )
> 
> We commonly use !found_entry.
> 
> > +    {
> > +        rc = -EINVAL;
> > +
> > +        printk(XENLOG_ERR "Cannot find any matching tracker with input dtbo."
> > +               " Removing nodes is supported only for prior added dtbo.\n");
> > +        goto out;
> > +
> > +    }
> > +
> > +    rc = remove_nodes(entry);
> > +    if ( rc )
> > +    {
> > +        printk(XENLOG_ERR "Removing node failed\n");
> > +        goto out;
> > +    }
> > +
> > +    list_del(&entry->entry);
> > +
> > +    xfree(entry->dt_host_new);
> > +    xfree(entry->fdt);
> > +    xfree(entry->overlay_fdt);
> > +
> > +    xfree(entry->nodes_address);
> > +
> > +    xfree(entry);
> > +
> > +out:
> > +    spin_unlock(&overlay_lock);
> > +    return rc;
> > +}
> > +
> > +long dt_sysctl(struct xen_sysctl_dt_overlay *op)
> 
> It would be best to name it dt_overlay_sysctl() is this is meant to handle
> only overlay operations. If we want to do more, then I think we want to name
> the sysctl SYSCTL_DEVICE_TREE to reflect the more general purpose (renaming
> the sysctl after the fact is a pain).
> 
Changed it to dt_overlay_sysctl
> > +{
> > +    long ret;
> > +    void *overlay_fdt;
> > +
> > +    if ( op->overlay_fdt_size == 0 ||
> 
> Do you expect the rest of the code to misbehave if we pass 0?
> 
> > op->overlay_fdt_size > KB(500) )
> 
> How did you decide the value KB(500)?
That maximum overlay dtb used for FPGA examples was 128KB. We put the size to
500KB. I can increase/decrease it if 500KB doesn't make sense.
> 
> > +        return -EINVAL;
> > +
> > +    if ( op->pad[0] || op->pad[1] || op->pad[2] )
> > +        return -EINVAL;
> > +
> > +    overlay_fdt = xmalloc_bytes(op->overlay_fdt_size);
> > +
> > +    if ( overlay_fdt == NULL )
> > +        return -ENOMEM;
> > +
> > +    ret = copy_from_guest(overlay_fdt, op->overlay_fdt, op->overlay_fdt_size);
> > +    if ( ret )
> > +    {
> > +        gprintk(XENLOG_ERR, "copy from guest failed\n");
> > +        xfree(overlay_fdt);
> > +
> > +        return -EFAULT;
> > +    }
> > +
> > +    switch ( op->overlay_op )
> > +    {
> > +    case XEN_SYSCTL_DT_OVERLAY_REMOVE:
> > +        ret = handle_remove_overlay_nodes(overlay_fdt, op->overlay_fdt_size);
> > +
> > +        break;
> > +
> > +    default:
> 
> I think you want to set ret to -EOPNOTSUPP here rather than returning 0.
> Also, it feels to me you want to first check the validity of the operation
> before checking for overlay_fdt_size and the allocation.
> 
> This would allow to re-use the fields 'overlay_fdt_size'/'overlay_fdt' for a
> different purpose on a new operation and also avoid allocating memory when
> we know it will fail.
> 
> The former would not be possible in your current approach because a Xen
> without the hypothetical new sysctl may return -EINVAL/-EFAULT rather than
> -EOPNOTSUPP. So it would be more difficult to differentiate it in the
> toolstack.
> 
So, i re-organized the code now. First, validity of ops is checked, next
memory is allocated and add/remove ops are done.
> > +        break;
> > +    }
> > +
> > +    xfree(overlay_fdt);
> > +
> > +    return ret;
> > +}
> > +
> > +/*
> > + * Local variables:
> > + * mode: C
> > + * c-file-style: "BSD"
> > + * c-basic-offset: 4
> > + * indent-tabs-mode: nil
> > + * End:
> > + */
> > diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
> > index 2b24d6bfd0..ff54607617 100644
> > --- a/xen/include/public/sysctl.h
> > +++ b/xen/include/public/sysctl.h
> > @@ -1057,6 +1057,25 @@ typedef struct xen_sysctl_cpu_policy xen_sysctl_cpu_policy_t;
> >   DEFINE_XEN_GUEST_HANDLE(xen_sysctl_cpu_policy_t);
> >   #endif
> > +#if defined(__arm__) || defined (__aarch64__)
> > +/*
> > + * XEN_SYSCTL_dt_overlay
> > + * Performs addition/removal of device tree nodes under parent node using dtbo.
> > + * This does in three steps:
> > + *  - Adds/Removes the nodes from dt_host.
> > + *  - Adds/Removes IRQ permission for the nodes.
> > + *  - Adds/Removes MMIO accesses.
> > + */
> > +struct xen_sysctl_dt_overlay {
> > +    XEN_GUEST_HANDLE_64(void) overlay_fdt;  /* IN: overlay fdt. */
> > +    uint32_t overlay_fdt_size;              /* IN: Overlay dtb size. */
> > +#define XEN_SYSCTL_DT_OVERLAY_ADD                   1
> > +#define XEN_SYSCTL_DT_OVERLAY_REMOVE                2
> > +    uint8_t overlay_op;                     /* IN: Add or remove. */
> > +    uint8_t pad[3];                         /* IN: Must be zero. */
> > +};
> > +#endif
> > +
> >   struct xen_sysctl {
> >       uint32_t cmd;
> >   #define XEN_SYSCTL_readconsole                    1
> > @@ -1087,6 +1106,7 @@ struct xen_sysctl {
> >   #define XEN_SYSCTL_livepatch_op                  27
> >   /* #define XEN_SYSCTL_set_parameter              28 */
> >   #define XEN_SYSCTL_get_cpu_policy                29
> > +#define XEN_SYSCTL_dt_overlay                    30
> >       uint32_t interface_version; /* XEN_SYSCTL_INTERFACE_VERSION */
> >       union {
> >           struct xen_sysctl_readconsole       readconsole;
> > @@ -1117,6 +1137,10 @@ struct xen_sysctl {
> >   #if defined(__i386__) || defined(__x86_64__)
> >           struct xen_sysctl_cpu_policy        cpu_policy;
> >   #endif
> > +
> > +#if defined(__arm__) || defined (__aarch64__)
> > +        struct xen_sysctl_dt_overlay        dt_overlay;
> > +#endif
> >           uint8_t                             pad[128];
> >       } u;
> >   };
> > diff --git a/xen/include/xen/dt-overlay.h b/xen/include/xen/dt-overlay.h
> > new file mode 100644
> > index 0000000000..43fa5a02a0
> > --- /dev/null
> > +++ b/xen/include/xen/dt-overlay.h
> > @@ -0,0 +1,59 @@
> > + /* SPDX-License-Identifier: GPL-2.0-only */
> > + /*
> > + * xen/dt-overlay.h
> > + *
> > + * Device tree overlay support in Xen.
> > + *
> > + * Copyright (C) 2023, Advanced Micro Devices, Inc. All Rights Reserved.
> > + * Written by Vikram Garhwal <vikram.garhwal@amd.com>
> > + *
> > + */
> > +#ifndef __XEN_DT_OVERLAY_H__
> > +#define __XEN_DT_OVERLAY_H__
> > +
> > +#include <xen/list.h>
> > +#include <xen/libfdt/libfdt.h>
> 
> Nothing below is using libfdt.h and ...
Moved this to dt-overlay.c
> 
> > +#include <xen/device_tree.h>
> > +#include <xen/rangeset.h>
> 
> .. rangeset.h. So why do you include them?
> 
Remove this.

Thank you so much for reviewing this patch series!
> > +
> > +/*
> > + * overlay_track describes information about added nodes through dtbo.
> > + * @entry: List pointer.
> > + * @dt_host_new: Pointer to the updated dt_host_new which is unflattened from
> > +    the 'updated fdt'.
> > + * @fdt: Stores the fdt.
> > + * @overlay_fdt: Stores a copy of input overlay_fdt.
> > + * @nodes_address: Stores each overlay_node's address.
> > + * @num_nodes: Total number of nodes in overlay dtb.
> > + */
> > +struct overlay_track {
> > +    struct list_head entry;
> > +    struct dt_device_node *dt_host_new;
> > +    void *fdt;
> > +    void *overlay_fdt;
> > +    unsigned long *nodes_address;
> > +    unsigned int num_nodes;
> > +};
> > +
> > +struct xen_sysctl_dt_overlay;
> > +
> > +#ifdef CONFIG_OVERLAY_DTB
> > +long dt_sysctl(struct xen_sysctl_dt_overlay *op);
> > +#else
> > +#include <xen/errno.h>
> > +static inline long dt_sysctl(struct xen_sysctl_dt_overlay *op)
> > +{
> > +    return -EOPNOTSUPP;
> > +}
> > +#endif
> > +
> > +#endif /* __XEN_DT_OVERLAY_H__ */
> > +/*
> > + * Local variables:
> > + * mode: C
> > + * c-file-style: "BSD"
> > + * c-basic-offset: 4
> > + * tab-width: 4
> > + * indent-tabs-mode: nil
> > + * End:
> > + */
> 
> Cheers,
> 
> [1]
> https://lore.kernel.org/all/478d4c20-31b7-e98d-25c1-4b4e9afe7e0a@xen.org/
> 
> -- 
> Julien Grall

Re: [XEN][PATCH v7 04/19] common/device_tree: change __unflatten_device_tree() type

[Julien Grall]

Hi,

On 17/08/2023 00:49, Vikram Garhwal wrote:
> On Tue, Jun 06, 2023 at 12:09:35PM -0700, Vikram Garhwal wrote:
>> Hi Julien,
>> Will update the commit message regarding why we need to export this for dtbo
>> programming.
>>
>> On 6/5/23 12:04 PM, Julien Grall wrote:
>>> Hi,
>>>
>>> Title:
>>>
>>> 'type' is a bit confusing here. How about "Export
>>> __unflatten_device_tre()"?
>>>
>>> On 02/06/2023 01:48, Vikram Garhwal wrote:
>>>> Following changes are done to __unflatten_device_tree():
>>>>       1. __unflatten_device_tree() is renamed to unflatten_device_tree().
>>>>       2. Remove __init and static function type.
>>>
>>> As there is no external caller yet, please explain why you want to
>>> export the function.
> Update the commit message in v8.
>>>
>>> Cheers,
>>>
>>>>
>>>> Signed-off-by: Vikram Garhwal <vikram.garhwal@amd.com>
>>>> Reviewed-by: Henry Wang <Henry.Wang@arm.com>
>>>> ---
>>>>    xen/common/device_tree.c      | 9 ++++-----
>>>>    xen/include/xen/device_tree.h | 5 +++++
>>>>    2 files changed, 9 insertions(+), 5 deletions(-)
>>>>
>>>> diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
>>>> index bbdab07596..16b4b4e946 100644
>>>> --- a/xen/common/device_tree.c
>>>> +++ b/xen/common/device_tree.c
>>>> @@ -2083,7 +2083,7 @@ static unsigned long unflatten_dt_node(const
>>>> void *fdt,
>>>>    }
>>>>      /**
>>>> - * __unflatten_device_tree - create tree of device_nodes from flat blob
>>>> + * unflatten_device_tree - create tree of device_nodes from flat blob
>>>>     *
>>>>     * unflattens a device-tree, creating the
>>>>     * tree of struct device_node. It also fills the "name" and "type"
>>>> @@ -2092,8 +2092,7 @@ static unsigned long unflatten_dt_node(const
>>>> void *fdt,
>>>>     * @fdt: The fdt to expand
>>>>     * @mynodes: The device_node tree created by the call
>>>>     */
>>>> -static int __init __unflatten_device_tree(const void *fdt,
>>>> -                                          struct dt_device_node
>>>> **mynodes)
>>>> +int unflatten_device_tree(const void *fdt, struct dt_device_node
>>>> **mynodes)
>>>>    {
>>>>        unsigned long start, mem, size;
>>>>        struct dt_device_node **allnextp = mynodes;
>>>> @@ -2230,10 +2229,10 @@ dt_find_interrupt_controller(const struct
>>>> dt_device_match *matches)
>>>>      void __init dt_unflatten_host_device_tree(void)
>>>>    {
>>>> -    int error = __unflatten_device_tree(device_tree_flattened,
>>>> &dt_host);
>>>> +    int error = unflatten_device_tree(device_tree_flattened, &dt_host);
>>>>          if ( error )
>>>> -        panic("__unflatten_device_tree failed with error %d\n", error);
>>>> +        panic("unflatten_device_tree failed with error %d\n", error);
>>>>          dt_alias_scan();
>>>
>>> This function doesn't seem to be called in the case of the overlay
>>> device-tree. Does this mean that it will never contain any alias?
>>>
> I haven't seen any overlay example for FPGA use cases where alias are added.
> I have added a TODO in patch 16/19 where we are calling unflatten_device_tree().
>>>>    }
>>>> diff --git a/xen/include/xen/device_tree.h
>>>> b/xen/include/xen/device_tree.h
>>>> index c2eada7489..2c35c0d391 100644
>>>> --- a/xen/include/xen/device_tree.h
>>>> +++ b/xen/include/xen/device_tree.h
>>>> @@ -178,6 +178,11 @@ int device_tree_for_each_node(const void *fdt,
>>>> int node,
>>>>     */
>>>>    void dt_unflatten_host_device_tree(void);
>>>>    +/**
>>>> + * unflatten any device tree.
>>>
>>> Most of the exported function in device_tre.h have documentation. Can
>>> you do the same here?
> Done!
>>>
>>>> + */
>>>> +int unflatten_device_tree(const void *fdt, struct dt_device_node
>>>> **mynodes);
>>>
>>> NIT: From an external interface perspective, do we actually need to pass
>>> an extra pointer? Can't we instead, return the pointer?
> We will also need the error from the function. So, that's why i kept it as it is.

This can be achieved by using the ERR_PTR() infrastructure which I would 
rather prefer over passing an extra pointer here.

Cheers,

-- 
Julien Grall

Re: [XEN][PATCH v7 15/19] xen/arm: Implement device tree node removal functionalities

[Julien Grall]

Hi,

On 17/08/2023 01:31, Vikram Garhwal wrote:
> On Mon, Jun 05, 2023 at 10:07:48PM +0100, Julien Grall wrote:
>>> +{
>>> +    int rc = 0;
>>> +    struct domain *d = hardware_domain;
>>> +    domid_t domid;
>>> +    unsigned int naddr, len;
>>> +    unsigned int i, nirq;
>>> +
>>> +    domid = dt_device_used_by(device_node);
>>
>> Looking at the caller, it is not clear to me which lock is preventing the
>> device to be assigned whilst you remove it.
> So, this is on user to make sure the domain is not using the device.

It is not clear what you mean by user. Is it someone external to Xen? If 
not, which function is responsible? If yes, then we can't rely on an 
entity outside of Xen to do the right thing.

>>> +    if ( domid != 0 && domid != DOMID_IO )
>>> +    {
>>> +        printk(XENLOG_ERR "Device %s is being used by domain %u. Removing nodes failed\n",
>>> +               device_node->full_name, domid);
>>> +        return -EINVAL;
>>> +    }
>>> +
>>> +    dt_dprintk("Removing node: %s\n", device_node->full_name);
>>> +
>>> +    nirq = dt_number_of_irq(device_node); > +
>>> +    /* Remove IRQ permission */
>>> +    for ( i = 0; i < nirq; i++ )
>>> +    {
>>> +        rc = platform_get_irq(device_node, i);
>>
>> As I mentioned in [1], I think that parsing the Device-Tree again when
>> removing any interrupts/mappings is a bit odd as there are more possible
>> failures and is more complex than necessary. I have proposed a way to do it
>> with rangeset, but I can't find any reason why this wasn't done. Can you
>> explain?
> IIUC, range sets can work if we have only one level of node i.e. no children.
> I tried in previous version to use range but it's complicated to get info in
> correct order using rangeset. Example, we have three nodes, node A, B and C. A
> has three child A_a, A_b and A_c. While adding the nodes, we add A first then
> A_a, A_b, A_c and finally B and C. And rangeset is updated in same order but
> when we remove node, first A_c is removed followed by A_b and A_a and then A.
> So, this was the problem for me on how to keep track which interrupt belong to
> which node.

 From my understanding, all the nodes added together will have to be 
removed together. IOW, it is not possible to remove A_c in one hypercall 
by not A_b.

It is not clear to me why you need to know which interrupt belong to 
which node. On removal, it should be sufficient to revert all the 
permissions in one go and then all the nodes.

>>> +            return -EINVAL;
>>> +        }
>>> +        /*
>>> +         * TODO: We don't handle shared IRQs for now. So, it is assumed that
>>> +         * the IRQs was not shared with another devices.
>>> +         */
>>> +        rc = irq_deny_access(d, rc);
>>> +        if ( rc )
>>> +        {
>>> +            printk(XENLOG_ERR "unable to revoke access for irq %u for %s\n",
>>> +                   i, device_node->full_name);
>>> +            return rc;
>>> +        }
>>
>> You don't reverse the change in the routing. What would happen if the next
>> overlay is updated to now pass the same device to a guest?
>>
>> I would be OK if this is not handled in this series. But it should be marked
>> as a TODO.
> So, i explained the reason behind this in v4:

I couldn't find the appropriate summary in this series. Can you point me 
to it?

Cheers,

-- 
Julien Grall