[PATCH V3 00/10] Domain on Static Allocation

[PATCH V3 00/10] Domain on Static Allocation

[Penny Zheng]

Static Allocation refers to system or sub-system(domains) for which memory
areas are pre-defined by configuration using physical address ranges.
Those pre-defined memory, -- Static Memory, as parts of RAM reserved in the
beginning, shall never go to heap allocator or boot allocator for any use.

Domain on Static Allocation is supported through device tree property
`xen,static-mem` specifying reserved RAM banks as this domain's guest RAM.
The size of address-cells/size-cells must be defined in
"#xen,static-mem-address-cells" and "#xen,static-mem-size-cells".

This Patch Serie only talks about Domain on Static Allocation.

Looking into related [design link](
https://lists.xenproject.org/archives/html/xen-devel/2021-05/msg00882.html)
for more details.

The whole design is about Static Allocation and 1:1 direct-map, and this
Patch Serie only covers parts of it, which are Domain on Static Allocation.
Other features will be delievered through different patch series.
---
changes in v3:
- use "xen,static-mem" property to be compatible with System Sevice Tree
in the future
- introduce new helper device_tree_get_meminfo
- extract common codes for dealing with reserved memory stored in
bootinfo
- rename from "free_page" to "mark_page_free"
- remove non-trivial page_to_mfn conversion in "mark_page_free" due to
pdx compression, and let the MFN passed in
- let all switch-cases shared in "mark_page_free"
- change CONFIG_STATIC_ALLOCATION to CONFIG_STATIC_MEMORY
- put init_staticmem_pages in setup_mm
- rename assign_pages_nr to assign_pages
- alter the order of assign_pages parameters to help backporting
- change name from alloc_staticmem_pages/alloc_domstatic_pages to
acquire_staticmem_pages and acquire_domstatic_pages.
- remove hunks' #ifdef-ary by introducing PGC_reserved = 0
- remove DMA restriction
- "memory" property shall be mandatory
- rename allocate_static_bank_memory to append_static_memory_to_bank
- infer the next GFN from the bank information in append_static_memory_to_bank
- simplify the code of double loop in allocate_static_memory

Penny Zheng (10):
  xen/arm: introduce domain on Static Allocation
  xen/arm: introduce new helper device_tree_get_meminfo
  xen/arm: handle static memory in dt_unreserved_regions
  xen: introduce mark_page_free
  xen/arm: static memory initialization
  xen/arm: introduce PGC_reserved
  xen: re-define assign_pages and introduce assign_page
  xen/arm: introduce acquire_staticmem_pages and acquire_domstatic_pages
  xen/arm: check "xen,static-mem" property during domain construction
  xen/arm: introduce allocate_static_memory

 docs/misc/arm/device-tree/booting.txt |  40 +++++
 xen/arch/arm/Kconfig                  |   3 +
 xen/arch/arm/bootfdt.c                |  89 +++++++---
 xen/arch/arm/domain_build.c           | 170 +++++++++++++++++-
 xen/arch/arm/setup.c                  |  73 ++++++--
 xen/arch/x86/pv/dom0_build.c          |   2 +-
 xen/common/grant_table.c              |   2 +-
 xen/common/memory.c                   |   4 +-
 xen/common/page_alloc.c               | 245 ++++++++++++++++++++------
 xen/include/asm-arm/mm.h              |   3 +
 xen/include/asm-arm/setup.h           |   2 +
 xen/include/xen/mm.h                  |  15 ++
 12 files changed, 546 insertions(+), 102 deletions(-)

-- 
2.25.1

[PATCH V3 01/10] xen/arm: introduce domain on Static Allocation

[Penny Zheng]

Static Allocation refers to system or sub-system(domains) for which memory
areas are pre-defined by configuration using physical address ranges.
Those pre-defined memory, -- Static Memory, as parts of RAM reserved in the
beginning, shall never go to heap allocator or boot allocator for any use.

Domains on Static Allocation is supported through device tree property
`xen,static-mem` specifying reserved RAM banks as this domain's guest RAM.
By default, they shall be mapped to the fixed guest RAM address
`GUEST_RAM0_BASE`, `GUEST_RAM1_BASE`.

This patch introduces this new `xen,static-mem` feature, and also documents
and parses this new attribute at boot time and stores related info in
static_mem for later initialization.

Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
v3 changes:
- use "xen,static-mem" property to be compatible with System Sevice Tree
in the future
---
 docs/misc/arm/device-tree/booting.txt | 40 +++++++++++++++++++++
 xen/arch/arm/bootfdt.c                | 51 +++++++++++++++++++++++++++
 xen/include/asm-arm/setup.h           |  2 ++
 3 files changed, 93 insertions(+)

diff --git a/docs/misc/arm/device-tree/booting.txt b/docs/misc/arm/device-tree/booting.txt
index 5243bc7fd3..2a1ddca29b 100644
--- a/docs/misc/arm/device-tree/booting.txt
+++ b/docs/misc/arm/device-tree/booting.txt
@@ -268,3 +268,43 @@ The DTB fragment is loaded at 0xc000000 in the example above. It should
 follow the convention explained in docs/misc/arm/passthrough.txt. The
 DTB fragment will be added to the guest device tree, so that the guest
 kernel will be able to discover the device.
+
+
+Static Allocation
+=============
+
+Static Allocation refers to system or sub-system(domains) for which memory
+areas are pre-defined by configuration using physical address ranges.
+Those pre-defined memory, -- Static Memory, as parts of RAM reserved in the
+beginning, shall never go to heap allocator or boot allocator for any use.
+
+Domains on Static Allocation is supported through static memory property,
+defined under according /domUx in the name of "xen,static-mem", which are
+specifying physical RAM as this domain's guest RAM.
+The size of address-cells/size-cells must be defined in
+"#xen,static-mem-address-cells" and "#xen,static-mem-size-cells".
+
+On memory allocation, these pre-defined static memory ranges shall be
+firstly mapped to the fixed guest bank "GUEST_RAM0". Until it exhausts the
+`GUEST_RAM0_SIZE`, then it will seek to `GUEST_RAM1_BASE`, and so on.
+`GUEST_RAM0` may take up several pre-defined physical RAM regions.
+
+The dtb property should look like as follows:
+
+    / {
+        chosen {
+            domU1 {
+                compatible = "xen,domain";
+                #address-cells = <0x2>;
+                #size-cells = <0x2>;
+                cpus = <2>;
+                #xen,static-mem-address-cells = <0x1>;
+                #xen,static-mem-size-cells = <0x1>;
+                xen,static-mem = <0x30000000 0x20000000>;
+                ...
+            };
+        };
+    };
+
+DomU1 will have a static memory of 512MB reserved from the physical address
+0x30000000 to 0x50000000.
diff --git a/xen/arch/arm/bootfdt.c b/xen/arch/arm/bootfdt.c
index 476e32e0f5..d2714446e1 100644
--- a/xen/arch/arm/bootfdt.c
+++ b/xen/arch/arm/bootfdt.c
@@ -193,6 +193,55 @@ static int __init process_reserved_memory_node(const void *fdt, int node,
     return 0;
 }
 
+static int __init process_static_memory(const void *fdt, int node, void *data)
+{
+    int i = 0, banks;
+    const __be32 *cell;
+    paddr_t start, size;
+    u32 address_cells, size_cells, reg_cells;
+    struct meminfo *mem = data;
+    const struct fdt_property *prop;
+
+
+    address_cells = device_tree_get_u32(fdt, node,
+                                        "#xen,static-mem-address-cells", 0);
+    size_cells = device_tree_get_u32(fdt, node,
+                                     "#xen,static-mem-size-cells", 0);
+    if ( (address_cells == 0) || (size_cells == 0) )
+    {
+         printk("Missing \"#xen,static-mem-address-cell\" or "
+                 "\"#xen,static-mem-address-cell\".\n");
+         return -EINVAL;
+    }
+    reg_cells = address_cells + size_cells;
+
+    prop = fdt_get_property(fdt, node, "xen,static-mem", NULL);
+    /*
+     * Static memory shall belong to a specific domain, that is,
+     * its node `domUx` has compatible string "xen,domain".
+     */
+    if ( fdt_node_check_compatible(fdt, node, "xen,domain") != 0 )
+    {
+        printk("xen,static-mem property can only be located under /domUx node.\n");
+        return -EINVAL;
+    }
+
+    cell = (const __be32 *)prop->data;
+    banks = fdt32_to_cpu(prop->len) / (reg_cells * sizeof (u32));
+
+    for ( ; i < banks && mem->nr_banks < NR_MEM_BANKS; i++ )
+    {
+        device_tree_get_reg(&cell, address_cells, size_cells, &start, &size);
+        mem->bank[mem->nr_banks].start = start;
+        mem->bank[mem->nr_banks].size = size;
+        mem->nr_banks++;
+    }
+
+    if ( i < banks )
+        return -ENOSPC;
+    return 0;
+}
+
 static int __init process_reserved_memory(const void *fdt, int node,
                                           const char *name, int depth,
                                           u32 address_cells, u32 size_cells)
@@ -346,6 +395,8 @@ static int __init early_scan_node(const void *fdt,
         process_multiboot_node(fdt, node, name, address_cells, size_cells);
     else if ( depth == 1 && device_tree_node_matches(fdt, node, "chosen") )
         process_chosen_node(fdt, node, name, address_cells, size_cells);
+    else if ( depth == 2 && fdt_get_property(fdt, node, "xen,static-mem", NULL) )
+        process_static_memory(fdt, node, &bootinfo.static_mem);
 
     if ( rc < 0 )
         printk("fdt: node `%s': parsing failed\n", name);
diff --git a/xen/include/asm-arm/setup.h b/xen/include/asm-arm/setup.h
index c4b6af6029..e076329fc4 100644
--- a/xen/include/asm-arm/setup.h
+++ b/xen/include/asm-arm/setup.h
@@ -74,6 +74,8 @@ struct bootinfo {
 #ifdef CONFIG_ACPI
     struct meminfo acpi;
 #endif
+    /* Static Memory */
+    struct meminfo static_mem;
 };
 
 extern struct bootinfo bootinfo;
-- 
2.25.1

[PATCH V3 02/10] xen/arm: introduce new helper device_tree_get_meminfo

[Penny Zheng]

A few functions iterate over the device tree property to get memory info,
like "reg" or "xen,static-mem", so this commit creates a new helper
device_tree_get_meminfo to extract the
common codes.

Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
 xen/arch/arm/bootfdt.c | 104 +++++++++++++++++------------------------
 1 file changed, 42 insertions(+), 62 deletions(-)

diff --git a/xen/arch/arm/bootfdt.c b/xen/arch/arm/bootfdt.c
index d2714446e1..04210684c9 100644
--- a/xen/arch/arm/bootfdt.c
+++ b/xen/arch/arm/bootfdt.c
@@ -63,6 +63,44 @@ void __init device_tree_get_reg(const __be32 **cell, u32 address_cells,
     *size = dt_next_cell(size_cells, cell);
 }
 
+static int __init device_tree_get_meminfo(const void *fdt, int node,
+                                          const char *prop_name,
+                                          u32 address_cells, u32 size_cells,
+                                          void *data)
+{
+    const struct fdt_property *prop;
+    unsigned int i, banks;
+    const __be32 *cell;
+    u32 reg_cells = address_cells + size_cells;
+    paddr_t start, size;
+    struct meminfo *mem = data;
+
+    prop = fdt_get_property(fdt, node, prop_name, NULL);
+    if ( !prop )
+        return -ENOENT;
+
+    cell = (const __be32 *)prop->data;
+    banks = fdt32_to_cpu(prop->len) / (reg_cells * sizeof (u32));
+
+    for ( i = 0; i < banks && mem->nr_banks < NR_MEM_BANKS; i++ )
+    {
+        device_tree_get_reg(&cell, address_cells, size_cells, &start, &size);
+        /* Some DT may describe empty bank, ignore them */
+        if ( !size )
+            continue;
+        mem->bank[mem->nr_banks].start = start;
+        mem->bank[mem->nr_banks].size = size;
+        mem->nr_banks++;
+    }
+
+    if ( i < banks )
+    {
+        printk("Warning: Max number of supported memory regions reached.\n");
+        return -ENOSPC;
+    }
+    return 0;
+}
+
 u32 __init device_tree_get_u32(const void *fdt, int node,
                                const char *prop_name, u32 dflt)
 {
@@ -139,14 +177,6 @@ static int __init process_memory_node(const void *fdt, int node,
                                       u32 address_cells, u32 size_cells,
                                       void *data)
 {
-    const struct fdt_property *prop;
-    int i;
-    int banks;
-    const __be32 *cell;
-    paddr_t start, size;
-    u32 reg_cells = address_cells + size_cells;
-    struct meminfo *mem = data;
-
     if ( address_cells < 1 || size_cells < 1 )
     {
         printk("fdt: node `%s': invalid #address-cells or #size-cells",
@@ -154,27 +184,7 @@ static int __init process_memory_node(const void *fdt, int node,
         return -EINVAL;
     }
 
-    prop = fdt_get_property(fdt, node, "reg", NULL);
-    if ( !prop )
-        return -ENOENT;
-
-    cell = (const __be32 *)prop->data;
-    banks = fdt32_to_cpu(prop->len) / (reg_cells * sizeof (u32));
-
-    for ( i = 0; i < banks && mem->nr_banks < NR_MEM_BANKS; i++ )
-    {
-        device_tree_get_reg(&cell, address_cells, size_cells, &start, &size);
-        /* Some DT may describe empty bank, ignore them */
-        if ( !size )
-            continue;
-        mem->bank[mem->nr_banks].start = start;
-        mem->bank[mem->nr_banks].size = size;
-        mem->nr_banks++;
-    }
-
-    if ( i < banks )
-        return -ENOSPC;
-    return 0;
+    return device_tree_get_meminfo(fdt, node, "reg", address_cells, size_cells, data);
 }
 
 static int __init process_reserved_memory_node(const void *fdt, int node,
@@ -195,13 +205,7 @@ static int __init process_reserved_memory_node(const void *fdt, int node,
 
 static int __init process_static_memory(const void *fdt, int node, void *data)
 {
-    int i = 0, banks;
-    const __be32 *cell;
-    paddr_t start, size;
-    u32 address_cells, size_cells, reg_cells;
-    struct meminfo *mem = data;
-    const struct fdt_property *prop;
-
+    u32 address_cells, size_cells;
 
     address_cells = device_tree_get_u32(fdt, node,
                                         "#xen,static-mem-address-cells", 0);
@@ -213,33 +217,9 @@ static int __init process_static_memory(const void *fdt, int node, void *data)
                  "\"#xen,static-mem-address-cell\".\n");
          return -EINVAL;
     }
-    reg_cells = address_cells + size_cells;
-
-    prop = fdt_get_property(fdt, node, "xen,static-mem", NULL);
-    /*
-     * Static memory shall belong to a specific domain, that is,
-     * its node `domUx` has compatible string "xen,domain".
-     */
-    if ( fdt_node_check_compatible(fdt, node, "xen,domain") != 0 )
-    {
-        printk("xen,static-mem property can only be located under /domUx node.\n");
-        return -EINVAL;
-    }
-
-    cell = (const __be32 *)prop->data;
-    banks = fdt32_to_cpu(prop->len) / (reg_cells * sizeof (u32));
-
-    for ( ; i < banks && mem->nr_banks < NR_MEM_BANKS; i++ )
-    {
-        device_tree_get_reg(&cell, address_cells, size_cells, &start, &size);
-        mem->bank[mem->nr_banks].start = start;
-        mem->bank[mem->nr_banks].size = size;
-        mem->nr_banks++;
-    }
 
-    if ( i < banks )
-        return -ENOSPC;
-    return 0;
+    return device_tree_get_meminfo(fdt, node, "xen,static-mem", address_cells,
+                                   size_cells, data);
 }
 
 static int __init process_reserved_memory(const void *fdt, int node,
-- 
2.25.1

[PATCH V3 03/10] xen/arm: handle static memory in dt_unreserved_regions

[Penny Zheng]

static memory regions overlap with memory nodes. The
overlapping memory is reserved-memory and should be
handled accordingly:
dt_unreserved_regions should skip these regions the
same way they are already skipping mem-reserved regions.

Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
v3 changes:
- extract common codes for dealing with reserved memory stored in
bootinfo
---
 xen/arch/arm/setup.c | 47 ++++++++++++++++++++++++++++----------------
 1 file changed, 30 insertions(+), 17 deletions(-)

diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index 63a908e325..f569134317 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -200,6 +200,13 @@ static void __init dt_unreserved_regions(paddr_t s, paddr_t e,
                                          int first)
 {
     int i, nr = fdt_num_mem_rsv(device_tree_flattened);
+    /*
+     * There are two types of reserved memory stored in bootinfo, one defines
+     * in /reserved-memory node, the other refers to domain on static allocation
+     * through "xen,static-mem" property.
+     */
+    int nr_rsv_type = 2, t = 0, prev_nr;
+    struct meminfo *rsv_type[2] = {&bootinfo.reserved_mem, &bootinfo.static_mem};
 
     for ( i = first; i < nr ; i++ )
     {
@@ -219,26 +226,32 @@ static void __init dt_unreserved_regions(paddr_t s, paddr_t e,
         }
     }
 
-    /*
-     * i is the current bootmodule we are evaluating across all possible
-     * kinds.
-     *
-     * When retrieving the corresponding reserved-memory addresses
-     * below, we need to index the bootinfo.reserved_mem bank starting
-     * from 0, and only counting the reserved-memory modules. Hence,
-     * we need to use i - nr.
-     */
-    for ( ; i - nr < bootinfo.reserved_mem.nr_banks; i++ )
+    prev_nr = nr;
+    while ( t < nr_rsv_type )
     {
-        paddr_t r_s = bootinfo.reserved_mem.bank[i - nr].start;
-        paddr_t r_e = r_s + bootinfo.reserved_mem.bank[i - nr].size;
-
-        if ( s < r_e && r_s < e )
+        /*
+         * i is the current bootmodule we are evaluating across all possible
+         * kinds.
+         *
+         * When retrieving the corresponding reserved-memory addresses
+         * below, we need to index the reserved mem bank starting
+         * from 0, and only counting the reserved-memory modules. Hence,
+         * we need to use i - prev_nr.
+         */
+        i = i - prev_nr;
+        for ( ; i < rsv_type[t]->nr_banks; i++ )
         {
-            dt_unreserved_regions(r_e, e, cb, i + 1);
-            dt_unreserved_regions(s, r_s, cb, i + 1);
-            return;
+            paddr_t r_s = rsv_type[t]->bank[i].start;
+            paddr_t r_e = r_s + rsv_type[t]->bank[i].size;
+
+            if ( s < r_e && r_s < e )
+            {
+                dt_unreserved_regions(r_e, e, cb, i + 1);
+                dt_unreserved_regions(s, r_s, cb, i + 1);
+                return;
+            }
         }
+        prev_nr = rsv_type[t++]->nr_banks;
     }
 
     cb(s, e);
-- 
2.25.1

[PATCH V3 04/10] xen: introduce mark_page_free

[Penny Zheng]

This commit defines a new helper mark_page_free to extract common code,
like following the same cache/TLB coherency policy, between free_heap_pages
and the new function free_staticmem_pages, which will be introduced later.

Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
v3 changes:
- rename from "free_page" to "mark_page_free"
- remove non-trivial page_to_mfn conversion due to pdx compression, and
let the MFN passed in
- add an ASSERT() to double check both passed-in pg and MFN match up.
- let all switch-cases shared
---
 xen/common/page_alloc.c | 89 ++++++++++++++++++++++-------------------
 1 file changed, 48 insertions(+), 41 deletions(-)

diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index 958ba0cd92..95d9be169d 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -1376,6 +1376,53 @@ bool scrub_free_pages(void)
     return node_to_scrub(false) != NUMA_NO_NODE;
 }
 
+static void mark_page_free(struct page_info *pg, mfn_t mfn)
+{
+    ASSERT(mfn_x(mfn) == mfn_x(page_to_mfn(pg)));
+
+    /*
+     * Cannot assume that count_info == 0, as there are some corner cases
+     * where it isn't the case and yet it isn't a bug:
+     *  1. page_get_owner() is NULL
+     *  2. page_get_owner() is a domain that was never accessible by
+     *     its domid (e.g., failed to fully construct the domain).
+     *  3. page was never addressable by the guest (e.g., it's an
+     *     auto-translate-physmap guest and the page was never included
+     *     in its pseudophysical address space).
+     * In all the above cases there can be no guest mappings of this page.
+     */
+    switch ( pg->count_info & PGC_state )
+    {
+    case PGC_state_inuse:
+        BUG_ON(pg->count_info & PGC_broken);
+        pg->count_info = PGC_state_free;
+        break;
+
+    case PGC_state_offlining:
+        pg->count_info = (pg->count_info & PGC_broken) |
+                           PGC_state_offlined;
+        tainted = 1;
+        break;
+
+    default:
+        printk(XENLOG_ERR
+               "pg MFN %"PRI_mfn" c=%#lx o=%u v=%#lx t=%#x\n",
+               mfn_x(mfn),
+               pg->count_info, pg->v.free.order,
+               pg->u.free.val, pg->tlbflush_timestamp);
+        BUG();
+    }
+
+    /* If a page has no owner it will need no safety TLB flush. */
+    pg->u.free.need_tlbflush = (page_get_owner(pg) != NULL);
+    if ( pg->u.free.need_tlbflush )
+        page_set_tlbflush_timestamp(pg);
+
+    /* This page is not a guest frame any more. */
+    page_set_owner(pg, NULL); /* set_gpfn_from_mfn snoops pg owner */
+    set_gpfn_from_mfn(mfn_x(mfn), INVALID_M2P_ENTRY);
+}
+
 /* Free 2^@order set of pages. */
 static void free_heap_pages(
     struct page_info *pg, unsigned int order, bool need_scrub)
@@ -1392,47 +1439,7 @@ static void free_heap_pages(
 
     for ( i = 0; i < (1 << order); i++ )
     {
-        /*
-         * Cannot assume that count_info == 0, as there are some corner cases
-         * where it isn't the case and yet it isn't a bug:
-         *  1. page_get_owner() is NULL
-         *  2. page_get_owner() is a domain that was never accessible by
-         *     its domid (e.g., failed to fully construct the domain).
-         *  3. page was never addressable by the guest (e.g., it's an
-         *     auto-translate-physmap guest and the page was never included
-         *     in its pseudophysical address space).
-         * In all the above cases there can be no guest mappings of this page.
-         */
-        switch ( pg[i].count_info & PGC_state )
-        {
-        case PGC_state_inuse:
-            BUG_ON(pg[i].count_info & PGC_broken);
-            pg[i].count_info = PGC_state_free;
-            break;
-
-        case PGC_state_offlining:
-            pg[i].count_info = (pg[i].count_info & PGC_broken) |
-                               PGC_state_offlined;
-            tainted = 1;
-            break;
-
-        default:
-            printk(XENLOG_ERR
-                   "pg[%u] MFN %"PRI_mfn" c=%#lx o=%u v=%#lx t=%#x\n",
-                   i, mfn_x(mfn) + i,
-                   pg[i].count_info, pg[i].v.free.order,
-                   pg[i].u.free.val, pg[i].tlbflush_timestamp);
-            BUG();
-        }
-
-        /* If a page has no owner it will need no safety TLB flush. */
-        pg[i].u.free.need_tlbflush = (page_get_owner(&pg[i]) != NULL);
-        if ( pg[i].u.free.need_tlbflush )
-            page_set_tlbflush_timestamp(&pg[i]);
-
-        /* This page is not a guest frame any more. */
-        page_set_owner(&pg[i], NULL); /* set_gpfn_from_mfn snoops pg owner */
-        set_gpfn_from_mfn(mfn_x(mfn) + i, INVALID_M2P_ENTRY);
+        mark_page_free(&pg[i], mfn_add(mfn, i));
 
         if ( need_scrub )
         {
-- 
2.25.1

[PATCH V3 05/10] xen/arm: static memory initialization

[Penny Zheng]

This patch introduces static memory initialization, during system boot up.

The new function init_staticmem_pages is responsible for static memory
initialization.

Helper free_staticmem_pages is the equivalent of free_heap_pages, to free
nr_mfns pages of static memory.

This commit also introduces new CONFIG_STATIC_MEMORY to avoid bringing dead
codes in other archs.

Put asynchronous scrubbing for pages of static memory in TODO list.

Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
v3 change:
- include addition of CONFIG_STATIC_ALLOCATION in this commit, where it
is firstly used and also change the name to CONFIG_STATIC_MEMORY
- Fix TAB typo in Kconfig
- put init_staticmem_pages in setup_mm
---
 xen/arch/arm/Kconfig    |  3 +++
 xen/arch/arm/setup.c    | 26 ++++++++++++++++++++++++++
 xen/common/page_alloc.c | 20 ++++++++++++++++++++
 xen/include/xen/mm.h    |  6 ++++++
 4 files changed, 55 insertions(+)

diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index ecfa6822e4..b91e2efc4f 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -106,6 +106,9 @@ config TEE
 
 source "arch/arm/tee/Kconfig"
 
+config STATIC_MEMORY
+        def_bool y
+
 endmenu
 
 menu "ARM errata workaround via the alternative framework"
diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index f569134317..aec8bc8494 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -622,6 +622,28 @@ static void __init init_pdx(void)
     }
 }
 
+/* Static memory initialization */
+static void __init init_staticmem_pages(void)
+{
+    unsigned int bank;
+
+    /* TODO: Considering NUMA-support scenario. */
+    for ( bank = 0 ; bank < bootinfo.static_mem.nr_banks; bank++ )
+    {
+        paddr_t bank_start = bootinfo.static_mem.bank[bank].start;
+        paddr_t bank_size = bootinfo.static_mem.bank[bank].size;
+        paddr_t bank_end = bank_start + bank_size;
+
+        bank_start = round_pgup(bank_start);
+        bank_end = round_pgdown(bank_end);
+        if ( bank_end <= bank_start )
+            return;
+
+        free_staticmem_pages(maddr_to_page(bank_start),
+                            (bank_end - bank_start) >> PAGE_SHIFT, false);
+    }
+}
+
 #ifdef CONFIG_ARM_32
 static void __init setup_mm(void)
 {
@@ -749,6 +771,8 @@ static void __init setup_mm(void)
     /* Add xenheap memory that was not already added to the boot allocator. */
     init_xenheap_pages(mfn_to_maddr(xenheap_mfn_start),
                        mfn_to_maddr(xenheap_mfn_end));
+
+    init_staticmem_pages();
 }
 #else /* CONFIG_ARM_64 */
 static void __init setup_mm(void)
@@ -802,6 +826,8 @@ static void __init setup_mm(void)
 
     setup_frametable_mappings(ram_start, ram_end);
     max_page = PFN_DOWN(ram_end);
+
+    init_staticmem_pages();
 }
 #endif
 
diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index 95d9be169d..f394d6627a 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -1519,6 +1519,26 @@ static void free_heap_pages(
     spin_unlock(&heap_lock);
 }
 
+#ifdef CONFIG_STATIC_MEMORY
+/* Equivalent of free_heap_pages to free nr_mfns pages of static memory. */
+void __init free_staticmem_pages(struct page_info *pg, unsigned long nr_mfns,
+                                 bool need_scrub)
+{
+    mfn_t mfn = page_to_mfn(pg);
+    unsigned long i;
+
+    for ( i = 0; i < nr_mfns; i++ )
+    {
+        mark_page_free(&pg[i], mfn_add(mfn, i));
+
+        if ( need_scrub )
+        {
+            /* TODO: asynchronous scrubbing for pages of static memory. */
+            scrub_one_page(pg);
+        }
+    }
+}
+#endif
 
 /*
  * Following rules applied for page offline:
diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
index 667f9dac83..8e8fb5a615 100644
--- a/xen/include/xen/mm.h
+++ b/xen/include/xen/mm.h
@@ -85,6 +85,12 @@ bool scrub_free_pages(void);
 } while ( false )
 #define FREE_XENHEAP_PAGE(p) FREE_XENHEAP_PAGES(p, 0)
 
+#ifdef CONFIG_STATIC_MEMORY
+/* These functions are for static memory */
+void free_staticmem_pages(struct page_info *pg, unsigned long nr_mfns,
+                          bool need_scrub);
+#endif
+
 /* Map machine page range in Xen virtual address space. */
 int map_pages_to_xen(
     unsigned long virt,
-- 
2.25.1

[PATCH V3 06/10] xen/arm: introduce PGC_reserved

[Penny Zheng]

This patch introduces a new page flag PGC_reserved in order to differentiate
pages of static memory from those allocated from heap.

Mark pages of static memory PGC_reserved when initializing them.

Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
v3 change:
- introduce PGC_reserved when it is firstly used.
---
 xen/common/page_alloc.c  | 3 +++
 xen/include/asm-arm/mm.h | 3 +++
 2 files changed, 6 insertions(+)

diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index f394d6627a..15edaca227 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -1536,6 +1536,9 @@ void __init free_staticmem_pages(struct page_info *pg, unsigned long nr_mfns,
             /* TODO: asynchronous scrubbing for pages of static memory. */
             scrub_one_page(pg);
         }
+
+        /* In case initializing page of static memory, mark it PGC_reserved. */
+        pg[i].count_info |= PGC_reserved;
     }
 }
 #endif
diff --git a/xen/include/asm-arm/mm.h b/xen/include/asm-arm/mm.h
index ded74d29da..7b5e7b7f69 100644
--- a/xen/include/asm-arm/mm.h
+++ b/xen/include/asm-arm/mm.h
@@ -108,6 +108,9 @@ struct page_info
   /* Page is Xen heap? */
 #define _PGC_xen_heap     PG_shift(2)
 #define PGC_xen_heap      PG_mask(1, 2)
+  /* Page is reserved */
+#define _PGC_reserved     PG_shift(3)
+#define PGC_reserved      PG_mask(1, 3)
 /* ... */
 /* Page is broken? */
 #define _PGC_broken       PG_shift(7)
-- 
2.25.1

[PATCH V3 07/10] xen: re-define assign_pages and introduce assign_page

[Penny Zheng]

In order to deal with the trouble of count-to-order conversion when page number
is not in a power-of-two, this commit re-define assign_pages for nr pages and
assign_page for original page with a single order.

Backporting confusion could be helped by altering the order of assign_pages
parameters, such that the compiler would point out that adjustments at call
sites are needed.

Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
v3 change:
- rename assign_pages_nr to assign_pages
- alter the order of assign_pages parameters
---
 xen/arch/x86/pv/dom0_build.c |  2 +-
 xen/common/grant_table.c     |  2 +-
 xen/common/memory.c          |  4 ++--
 xen/common/page_alloc.c      | 21 +++++++++++++--------
 xen/include/xen/mm.h         |  6 ++++++
 5 files changed, 23 insertions(+), 12 deletions(-)

diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index af47615b22..476f8a2012 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -556,7 +556,7 @@ int __init dom0_construct_pv(struct domain *d,
         else
         {
             while ( count-- )
-                if ( assign_pages(d, mfn_to_page(_mfn(mfn++)), 0, 0) )
+                if ( assign_page(d, mfn_to_page(_mfn(mfn++)), 0, 0) )
                     BUG();
         }
         initrd->mod_end = 0;
diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c
index fab77ab9cc..10b23f7e09 100644
--- a/xen/common/grant_table.c
+++ b/xen/common/grant_table.c
@@ -2342,7 +2342,7 @@ gnttab_transfer(
          * is respected and speculative execution is blocked accordingly
          */
         if ( unlikely(!evaluate_nospec(okay)) ||
-            unlikely(assign_pages(e, page, 0, MEMF_no_refcount)) )
+            unlikely(assign_page(e, page, 0, MEMF_no_refcount)) )
         {
             bool drop_dom_ref;
 
diff --git a/xen/common/memory.c b/xen/common/memory.c
index e07bd9a5ea..8c7c9c8fe4 100644
--- a/xen/common/memory.c
+++ b/xen/common/memory.c
@@ -728,7 +728,7 @@ static long memory_exchange(XEN_GUEST_HANDLE_PARAM(xen_memory_exchange_t) arg)
         /* Assign each output page to the domain. */
         for ( j = 0; (page = page_list_remove_head(&out_chunk_list)); ++j )
         {
-            if ( assign_pages(d, page, exch.out.extent_order,
+            if ( assign_page(d, page, exch.out.extent_order,
                               MEMF_no_refcount) )
             {
                 unsigned long dec_count;
@@ -797,7 +797,7 @@ static long memory_exchange(XEN_GUEST_HANDLE_PARAM(xen_memory_exchange_t) arg)
      * cleared PGC_allocated.
      */
     while ( (page = page_list_remove_head(&in_chunk_list)) )
-        if ( assign_pages(d, page, 0, MEMF_no_refcount) )
+        if ( assign_page(d, page, 0, MEMF_no_refcount) )
         {
             BUG_ON(!d->is_dying);
             free_domheap_page(page);
diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index 15edaca227..3414873679 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -2283,8 +2283,8 @@ void init_domheap_pages(paddr_t ps, paddr_t pe)
 
 int assign_pages(
     struct domain *d,
+    unsigned long nr,
     struct page_info *pg,
-    unsigned int order,
     unsigned int memflags)
 {
     int rc = 0;
@@ -2304,7 +2304,7 @@ int assign_pages(
     {
         unsigned int extra_pages = 0;
 
-        for ( i = 0; i < (1ul << order); i++ )
+        for ( i = 0; i < nr; i++ )
         {
             ASSERT(!(pg[i].count_info & ~PGC_extra));
             if ( pg[i].count_info & PGC_extra )
@@ -2313,18 +2313,18 @@ int assign_pages(
 
         ASSERT(!extra_pages ||
                ((memflags & MEMF_no_refcount) &&
-                extra_pages == 1u << order));
+                extra_pages == nr));
     }
 #endif
 
     if ( pg[0].count_info & PGC_extra )
     {
-        d->extra_pages += 1u << order;
+        d->extra_pages += nr;
         memflags &= ~MEMF_no_refcount;
     }
     else if ( !(memflags & MEMF_no_refcount) )
     {
-        unsigned int tot_pages = domain_tot_pages(d) + (1 << order);
+        unsigned int tot_pages = domain_tot_pages(d) + nr;
 
         if ( unlikely(tot_pages > d->max_pages) )
         {
@@ -2336,10 +2336,10 @@ int assign_pages(
     }
 
     if ( !(memflags & MEMF_no_refcount) &&
-         unlikely(domain_adjust_tot_pages(d, 1 << order) == (1 << order)) )
+         unlikely(domain_adjust_tot_pages(d, nr) == nr) )
         get_knownalive_domain(d);
 
-    for ( i = 0; i < (1 << order); i++ )
+    for ( i = 0; i < nr; i++ )
     {
         ASSERT(page_get_owner(&pg[i]) == NULL);
         page_set_owner(&pg[i], d);
@@ -2354,6 +2354,11 @@ int assign_pages(
     return rc;
 }
 
+int assign_page(struct domain *d, struct page_info *pg, unsigned int order,
+                unsigned int memflags)
+{
+    return assign_pages(d, (1UL << order), pg, memflags);
+}
 
 struct page_info *alloc_domheap_pages(
     struct domain *d, unsigned int order, unsigned int memflags)
@@ -2396,7 +2401,7 @@ struct page_info *alloc_domheap_pages(
                 pg[i].count_info = PGC_extra;
             }
         }
-        if ( assign_pages(d, pg, order, memflags) )
+        if ( assign_page(d, pg, order, memflags) )
         {
             free_heap_pages(pg, order, memflags & MEMF_no_scrub);
             return NULL;
diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
index 8e8fb5a615..65ba1587ad 100644
--- a/xen/include/xen/mm.h
+++ b/xen/include/xen/mm.h
@@ -132,6 +132,12 @@ int query_page_offline(mfn_t mfn, uint32_t *status);
 void heap_init_late(void);
 
 int assign_pages(
+    struct domain *d,
+    unsigned long nr,
+    struct page_info *pg,
+    unsigned int memflags);
+
+int assign_page(
     struct domain *d,
     struct page_info *pg,
     unsigned int order,
-- 
2.25.1

[PATCH V3 08/10] xen/arm: introduce acquire_staticmem_pages and acquire_domstatic_pages

[Penny Zheng]

acquire_staticmem_pages aims to acquire nr_mfns contiguous pages of
static memory. And it is the equivalent of alloc_heap_pages for static
memory. Here only covers acquiring pre-configured static memory.

For each page, it shall check if the page is reserved(PGC_reserved)
and free. It shall also do a set of necessary initialization, which are
mostly the same ones in alloc_heap_pages, like, following the same
cache-coherency policy and turning page status into PGC_state_inuse, etc.

acquire_domstatic_pages is the equivalent of alloc_domheap_pages for
static memory, and it is to acquire nr_mfns contiguous pages of static memory
and assign them to one specific domain.

It uses acquire_staticmem_pages to acquire nr_mfns pre-configured pages of
static memory, then on success, it will use assign_pages to assign those pages
to one specific domain.

Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
v3 change:
- Assuming caller knows the static memory range is reserved (and free),
change name from alloc_staticmem_pages/alloc_domstatic_pages to
acquire_staticmem_pages and acquire_domstatic_pages.
- proper locking moved from the next commit to here.
- remove and refine extra verbosity log
- remove hunks' #ifdef-ary by introducing PGC_reserved = 0
- remove DMA restriction
---
 xen/common/page_alloc.c | 112 +++++++++++++++++++++++++++++++++++++++-
 xen/include/xen/mm.h    |   3 ++
 2 files changed, 113 insertions(+), 2 deletions(-)

diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index 3414873679..c9702533f4 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -151,6 +151,10 @@
 #define p2m_pod_offline_or_broken_replace(pg) BUG_ON(pg != NULL)
 #endif
 
+#ifndef CONFIG_STATIC_MEMORY
+#define PGC_reserved 0
+#endif
+
 /*
  * Comma-separated list of hexadecimal page numbers containing bad bytes.
  * e.g. 'badpage=0x3f45,0x8a321'.
@@ -1065,6 +1069,73 @@ static struct page_info *alloc_heap_pages(
     return pg;
 }
 
+#ifdef CONFIG_STATIC_MEMORY
+/*
+ * Acquire nr_mfns contiguous reserved pages, starting at #smfn, of
+ * static memory.
+ */
+static struct page_info *acquire_staticmem_pages(unsigned long nr_mfns,
+                                                 mfn_t smfn,
+                                                 unsigned int memflags)
+{
+    bool need_tlbflush = false;
+    uint32_t tlbflush_timestamp = 0;
+    unsigned long i;
+    struct page_info *pg;
+
+    /* For now, it only supports pre-configured static memory. */
+    if ( !mfn_valid(smfn) || !nr_mfns )
+        return NULL;
+
+    spin_lock(&heap_lock);
+
+    pg = mfn_to_page(smfn);
+
+    for ( i = 0; i < nr_mfns; i++ )
+    {
+        /*
+         * Reference count must continuously be zero for free pages
+         * of static memory(PGC_reserved).
+         */
+        if ( pg[i].count_info != (PGC_state_free | PGC_reserved) )
+        {
+            printk(XENLOG_ERR
+                   "pg[%lu] Static MFN %"PRI_mfn" c=%#lx t=%#x\n",
+                   i, mfn_x(page_to_mfn(pg + i)),
+                   pg[i].count_info, pg[i].tlbflush_timestamp);
+            BUG();
+        }
+
+        if ( !(memflags & MEMF_no_tlbflush) )
+            accumulate_tlbflush(&need_tlbflush, &pg[i],
+                                &tlbflush_timestamp);
+
+        /*
+         * Preserve flag PGC_reserved and change page state
+         * to PGC_state_inuse.
+         */
+        pg[i].count_info = (PGC_reserved | PGC_state_inuse);
+        /* Initialise fields which have other uses for free pages. */
+        pg[i].u.inuse.type_info = 0;
+        page_set_owner(&pg[i], NULL);
+
+        /*
+         * Ensure cache and RAM are consistent for platforms where the
+         * guest can control its own visibility of/through the cache.
+         */
+        flush_page_to_ram(mfn_x(page_to_mfn(&pg[i])),
+                            !(memflags & MEMF_no_icache_flush));
+    }
+
+    if ( need_tlbflush )
+        filtered_flush_tlb_mask(tlbflush_timestamp);
+
+    spin_unlock(&heap_lock);
+
+    return pg;
+}
+#endif
+
 /* Remove any offlined page in the buddy pointed to by head. */
 static int reserve_offlined_page(struct page_info *head)
 {
@@ -2306,7 +2377,7 @@ int assign_pages(
 
         for ( i = 0; i < nr; i++ )
         {
-            ASSERT(!(pg[i].count_info & ~PGC_extra));
+            ASSERT(!(pg[i].count_info & ~(PGC_extra | PGC_reserved)));
             if ( pg[i].count_info & PGC_extra )
                 extra_pages++;
         }
@@ -2345,7 +2416,8 @@ int assign_pages(
         page_set_owner(&pg[i], d);
         smp_wmb(); /* Domain pointer must be visible before updating refcnt. */
         pg[i].count_info =
-            (pg[i].count_info & PGC_extra) | PGC_allocated | 1;
+            (pg[i].count_info & (PGC_extra | PGC_reserved)) | PGC_allocated | 1;
+
         page_list_add_tail(&pg[i], page_to_list(d, &pg[i]));
     }
 
@@ -2411,6 +2483,42 @@ struct page_info *alloc_domheap_pages(
     return pg;
 }
 
+#ifdef CONFIG_STATIC_MEMORY
+/*
+ * Acquire nr_mfns contiguous pages, starting at #smfn, of static memory,
+ * then assign them to one specific domain #d.
+ */
+struct page_info *acquire_domstatic_pages(
+        struct domain *d, unsigned long nr_mfns, mfn_t smfn,
+        unsigned int memflags)
+{
+    struct page_info *pg = NULL;
+
+    ASSERT(!in_irq());
+
+    pg = acquire_staticmem_pages(nr_mfns, smfn, memflags);
+    if ( !pg )
+        return NULL;
+
+    /* Right now, MEMF_no_owner case is meaningless here. */
+    ASSERT(d);
+    if ( memflags & MEMF_no_refcount )
+    {
+        unsigned long i;
+
+        for ( i = 0; i < nr_mfns; i++ )
+            pg[i].count_info |= PGC_extra;
+    }
+    if ( assign_pages(d, nr_mfns, pg, memflags) )
+    {
+        free_staticmem_pages(pg, nr_mfns, memflags & MEMF_no_scrub);
+        return NULL;
+    }
+
+    return pg;
+}
+#endif
+
 void free_domheap_pages(struct page_info *pg, unsigned int order)
 {
     struct domain *d = page_get_owner(pg);
diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
index 65ba1587ad..69e3586d8a 100644
--- a/xen/include/xen/mm.h
+++ b/xen/include/xen/mm.h
@@ -89,6 +89,9 @@ bool scrub_free_pages(void);
 /* These functions are for static memory */
 void free_staticmem_pages(struct page_info *pg, unsigned long nr_mfns,
                           bool need_scrub);
+struct page_info *acquire_domstatic_pages(struct domain *d,
+                                          unsigned long nr_mfns, mfn_t smfn,
+                                          unsigned int memflags);
 #endif
 
 /* Map machine page range in Xen virtual address space. */
-- 
2.25.1

[PATCH V3 09/10] xen/arm: check "xen,static-mem" property during domain construction

[Penny Zheng]

This commit checks "xen,static-mem" device tree property in /domUx node,
to determine whether domain is on Static Allocation, when constructing
domain during boot-up.

Right now, the implementation of allocate_static_memory is missing, and
will be introduced later. It just BUG() out at the moment.

Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
v3 change:
- parse "xen,static-mem" in way of phandle back to property
- The return for dt_property_read_u32() shall be checked.
- "memory" property shall be mandatory
---
 xen/arch/arm/domain_build.c | 37 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 6c86d52781..cdb16f2086 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -2425,6 +2425,37 @@ static int __init construct_domU(struct domain *d,
     struct kernel_info kinfo = {};
     int rc;
     u64 mem;
+    const struct dt_property *static_mem_prop;
+    u32 static_mem_addr_cells, static_mem_size_cells;
+    bool static_mem = false;
+
+    /*
+     * Guest RAM could be static memory which will be specified through
+     * "xen,static-mem" property.
+     */
+    static_mem_prop = dt_find_property(node, "xen,static-mem", NULL);
+    if ( static_mem_prop )
+    {
+        static_mem = true;
+
+        if ( !dt_property_read_u32(node, "#xen,static-mem-address-cells",
+                                   &static_mem_addr_cells) )
+        {
+            printk("Error building DomU: cannot read "
+                   "\"#xen,static-mem-address-cells\" property\n");
+            return -EINVAL;
+        }
+
+        if ( !dt_property_read_u32(node, "#xen,static-mem-size-cells",
+                                   &static_mem_size_cells) )
+        {
+            printk("Error building DomU: cannot read "
+                   "\"#xen,static-mem-size-cells\" property\n");
+            return -EINVAL;
+        }
+
+        BUG_ON(static_mem_size_cells > 2 || static_mem_addr_cells > 2);
+    }
 
     rc = dt_property_read_u64(node, "memory", &mem);
     if ( !rc )
@@ -2452,7 +2483,11 @@ static int __init construct_domU(struct domain *d,
     /* type must be set before allocate memory */
     d->arch.type = kinfo.type;
 #endif
-    allocate_memory(d, &kinfo);
+    if ( !static_mem )
+        allocate_memory(d, &kinfo);
+    else
+        /* TODO: allocate_static_memory(...). */
+        BUG();
 
     rc = prepare_dtb_domU(d, &kinfo);
     if ( rc < 0 )
-- 
2.25.1

[PATCH V3 10/10] xen/arm: introduce allocate_static_memory

[Penny Zheng]

This commit introduces allocate_static_memory to allocate static memory as
guest RAM for Domain on Static Allocation.

It uses acquire_domstatic_pages to acquire pre-configured static memory
for this domain, and uses guest_physmap_add_page to set up P2M table.
These pre-defined static memory banks shall be firstly mapped to the
fixed guest RAM address `GUEST_RAM0_BASE`. And until it exhausts the
`GUEST_RAM0_SIZE`, it will seek to `GUEST_RAM1_BASE`, and so on.
`GUEST_RAM0` may take up several pre-defined physical RAM regions.

Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
v3 changes:
- parse "xen,static-mem" in way of phandle back to property.
- use unsigned int for index
- rename allocate_static_bank_memory to append_static_memory_to_bank
- infer the next GFN from the bank information
- simplify the code in double loop.
---
 xen/arch/arm/domain_build.c | 137 +++++++++++++++++++++++++++++++++++-
 1 file changed, 135 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index cdb16f2086..ed290ee31b 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -480,6 +480,139 @@ fail:
           (unsigned long)kinfo->unassigned_mem >> 10);
 }
 
+static bool __init append_static_memory_to_bank(struct domain *d,
+                                                struct membank *bank,
+                                                mfn_t smfn,
+                                                paddr_t size)
+{
+    int res;
+    paddr_t tot_size = size;
+    /* Infer next GFN. */
+    gfn_t sgfn = gaddr_to_gfn(bank->start + bank->size);
+
+    while ( tot_size > 0 )
+    {
+        unsigned int order = get_allocation_size(tot_size);
+
+        res = guest_physmap_add_page(d, sgfn, smfn, order);
+        if ( res )
+        {
+            dprintk(XENLOG_ERR, "Failed map pages to DOMU: %d", res);
+            return false;
+        }
+
+        smfn = mfn_add(smfn, 1UL << order);
+        tot_size -= (1UL << (PAGE_SHIFT + order));
+    }
+
+    bank->size = bank->size + size;
+    return true;
+}
+
+/* Allocate memory from static memory as RAM for one specific domain d. */
+static void __init allocate_static_memory(struct domain *d,
+                                          struct kernel_info *kinfo,
+                                          const struct dt_property *prop,
+                                          u32 addr_cells, u32 size_cells)
+{
+    unsigned int nr_banks, gbank, bank = 0;
+    const uint64_t rambase[] = GUEST_RAM_BANK_BASES;
+    const uint64_t ramsize[] = GUEST_RAM_BANK_SIZES;
+    const __be32 *cell;
+    u32 reg_cells = addr_cells + size_cells;
+    u64 tot_size = 0;
+    paddr_t pbase, psize, gsize;
+    mfn_t smfn;
+
+    /* Start with GUEST_RAM0. */
+    kinfo->mem.nr_banks = 0;
+    gbank = 0;
+    gsize = ramsize[gbank];
+    kinfo->mem.bank[gbank].start = rambase[gbank];
+
+    cell = (const __be32 *)prop->value;
+    nr_banks = (prop->length) / (reg_cells * sizeof (u32));
+    BUG_ON(nr_banks > NR_MEM_BANKS);
+
+    while ( bank < nr_banks )
+    {
+        device_tree_get_reg(&cell, addr_cells, size_cells, &pbase, &psize);
+        tot_size += psize;
+        smfn = maddr_to_mfn(pbase);
+
+        if ( !acquire_domstatic_pages(d, psize >> PAGE_SHIFT, smfn, 0) )
+        {
+            printk(XENLOG_ERR
+                    "%pd: cannot acquire static memory "
+                    "(0x%"PRIpaddr" - 0x%"PRIpaddr").\n",
+                    d, pbase, pbase + psize);
+            goto fail;
+        }
+
+        printk(XENLOG_INFO "%pd: STATIC BANK[%d] %#"PRIpaddr"-%#"PRIpaddr"\n",
+               d, bank, pbase, pbase + psize);
+
+        /*
+         * It shall be mapped to the fixed guest RAM address rambase[i],
+         * And until it exhausts the ramsize[i], it will seek to the next
+         * rambase[i+1].
+         */
+        while ( 1 )
+        {
+            /*
+             * The current physical bank is fully mapped.
+             * Handle the next physical bank.
+             */
+            if ( gsize >= psize )
+            {
+                if ( !append_static_memory_to_bank(d, &kinfo->mem.bank[gbank],
+                                                   smfn, psize) )
+                    goto fail;
+
+                gsize = gsize - psize;
+                bank++;
+                break;
+            }
+            /*
+             * Current guest bank memory is not enough to map.
+             * Check if we have another guest bank available.
+             * gbank refers guest memory bank index.
+             */
+            else if ( (gbank + 2) > GUEST_RAM_BANKS ) {
+                printk("Exhausted the number of guest bank\n");
+                goto fail;
+            }
+            else
+            {
+                if ( !append_static_memory_to_bank(d, &kinfo->mem.bank[gbank],
+                                                   smfn, gsize) )
+                    goto fail;
+
+                psize = psize - gsize;
+                smfn = mfn_add(smfn, gsize >> PAGE_SHIFT);
+                /* Update to the next guest bank. */
+                gbank++;
+                gsize = ramsize[gbank];
+                kinfo->mem.bank[gbank].start = rambase[gbank];
+            }
+        }
+    }
+
+    kinfo->mem.nr_banks = ++gbank;
+    kinfo->unassigned_mem -= tot_size;
+    if ( kinfo->unassigned_mem )
+        printk(XENLOG_ERR
+               "Size of \"memory\" property doesn't match up with the ones "
+               "defined in \"xen,static-mem\".\n");
+
+    return;
+
+fail:
+    panic("Failed to allocate requested static memory for domain %pd."
+          "Fix the VMs configurations.\n",
+          d);
+}
+
 static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
                                    const struct dt_device_node *node)
 {
@@ -2486,8 +2619,8 @@ static int __init construct_domU(struct domain *d,
     if ( !static_mem )
         allocate_memory(d, &kinfo);
     else
-        /* TODO: allocate_static_memory(...). */
-        BUG();
+        allocate_static_memory(d, &kinfo, static_mem_prop,
+                               static_mem_addr_cells, static_mem_size_cells);
 
     rc = prepare_dtb_domU(d, &kinfo);
     if ( rc < 0 )
-- 
2.25.1

Re: [PATCH V3 04/10] xen: introduce mark_page_free

[Jan Beulich]

On 15.07.2021 07:18, Penny Zheng wrote:
> This commit defines a new helper mark_page_free to extract common code,
> like following the same cache/TLB coherency policy, between free_heap_pages
> and the new function free_staticmem_pages, which will be introduced later.
> 
> Signed-off-by: Penny Zheng <penny.zheng@arm.com>

Acked-by: Jan Beulich <jbeulich@suse.com>
with ...

> --- a/xen/common/page_alloc.c
> +++ b/xen/common/page_alloc.c
> @@ -1376,6 +1376,53 @@ bool scrub_free_pages(void)
>      return node_to_scrub(false) != NUMA_NO_NODE;
>  }
>  
> +static void mark_page_free(struct page_info *pg, mfn_t mfn)
> +{
> +    ASSERT(mfn_x(mfn) == mfn_x(page_to_mfn(pg)));
> +
> +    /*
> +     * Cannot assume that count_info == 0, as there are some corner cases
> +     * where it isn't the case and yet it isn't a bug:
> +     *  1. page_get_owner() is NULL
> +     *  2. page_get_owner() is a domain that was never accessible by
> +     *     its domid (e.g., failed to fully construct the domain).
> +     *  3. page was never addressable by the guest (e.g., it's an
> +     *     auto-translate-physmap guest and the page was never included
> +     *     in its pseudophysical address space).
> +     * In all the above cases there can be no guest mappings of this page.
> +     */
> +    switch ( pg->count_info & PGC_state )
> +    {
> +    case PGC_state_inuse:
> +        BUG_ON(pg->count_info & PGC_broken);
> +        pg->count_info = PGC_state_free;
> +        break;
> +
> +    case PGC_state_offlining:
> +        pg->count_info = (pg->count_info & PGC_broken) |
> +                           PGC_state_offlined;

... indentation here not screwed up (it was correct originally).

> +        tainted = 1;
> +        break;
> +
> +    default:
> +        printk(XENLOG_ERR
> +               "pg MFN %"PRI_mfn" c=%#lx o=%u v=%#lx t=%#x\n",
> +               mfn_x(mfn),
> +               pg->count_info, pg->v.free.order,
> +               pg->u.free.val, pg->tlbflush_timestamp);

Just as a remark: Let's hope that the lost piece of information here
(the caller's induction variable) won't cause us trouble later on.

Jan

Re: [PATCH V3 05/10] xen/arm: static memory initialization

[Jan Beulich]

On 15.07.2021 07:18, Penny Zheng wrote:
> v3 change:
> - include addition of CONFIG_STATIC_ALLOCATION in this commit, where it
> is firstly used and also change the name to CONFIG_STATIC_MEMORY
> - Fix TAB typo in Kconfig

Not sure what this relates to, but ...

> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -106,6 +106,9 @@ config TEE
>  
>  source "arch/arm/tee/Kconfig"
>  
> +config STATIC_MEMORY
> +        def_bool y

... this is (wrongly) using spaces for indentation.

I also wonder about the placement: Shouldn't the option live in common
code, with Arm "select"ing it?

> --- a/xen/arch/arm/setup.c
> +++ b/xen/arch/arm/setup.c
> @@ -622,6 +622,28 @@ static void __init init_pdx(void)
>      }
>  }
>  
> +/* Static memory initialization */
> +static void __init init_staticmem_pages(void)
> +{
> +    unsigned int bank;
> +
> +    /* TODO: Considering NUMA-support scenario. */
> +    for ( bank = 0 ; bank < bootinfo.static_mem.nr_banks; bank++ )
> +    {
> +        paddr_t bank_start = bootinfo.static_mem.bank[bank].start;
> +        paddr_t bank_size = bootinfo.static_mem.bank[bank].size;
> +        paddr_t bank_end = bank_start + bank_size;
> +
> +        bank_start = round_pgup(bank_start);
> +        bank_end = round_pgdown(bank_end);
> +        if ( bank_end <= bank_start )
> +            return;
> +
> +        free_staticmem_pages(maddr_to_page(bank_start),
> +                            (bank_end - bank_start) >> PAGE_SHIFT, false);

Indentation (one too few spaces). Perhaps also consider to avoid
open-coding PFN_DOWN() here; in fact it and PFN_UP() could be used
in place of round_pg{down,up}() above.

Jan

Re: [PATCH V3 05/10] xen/arm: static memory initialization

[Jan Beulich]

On 15.07.2021 07:18, Penny Zheng wrote:
> --- a/xen/common/page_alloc.c
> +++ b/xen/common/page_alloc.c
> @@ -1519,6 +1519,26 @@ static void free_heap_pages(
>      spin_unlock(&heap_lock);
>  }
>  
> +#ifdef CONFIG_STATIC_MEMORY
> +/* Equivalent of free_heap_pages to free nr_mfns pages of static memory. */
> +void __init free_staticmem_pages(struct page_info *pg, unsigned long nr_mfns,
> +                                 bool need_scrub)
> +{
> +    mfn_t mfn = page_to_mfn(pg);
> +    unsigned long i;
> +
> +    for ( i = 0; i < nr_mfns; i++ )
> +    {
> +        mark_page_free(&pg[i], mfn_add(mfn, i));
> +
> +        if ( need_scrub )
> +        {
> +            /* TODO: asynchronous scrubbing for pages of static memory. */
> +            scrub_one_page(pg);
> +        }
> +    }
> +}
> +#endif

Btw, I think the lack of locking warrants extending the comment above
the function, to spell out what the implications are (in particular:
calls here need to happen early enough).

Jan

Re: [PATCH V3 07/10] xen: re-define assign_pages and introduce assign_page

[Jan Beulich]

On 15.07.2021 07:18, Penny Zheng wrote:
> In order to deal with the trouble of count-to-order conversion when page number
> is not in a power-of-two, this commit re-define assign_pages for nr pages and
> assign_page for original page with a single order.
> 
> Backporting confusion could be helped by altering the order of assign_pages
> parameters, such that the compiler would point out that adjustments at call
> sites are needed.

Back on the initial form of this patch Julien said:

"However, I think I would prefer if we introduce a new interface (maybe 
 assign_pages_nr()) rather than change the meaning of the field. This is 
 for two reasons:
    1) We limit the risk to make mistake when backporting a patch touch 
 assign_pages().
    2) Adding (1UL << order) for pretty much all the caller is not nice."

1) is taken care of here anyway (albeit see the remark below), and of the
callers only some would really need "1UL <<" added (others could simply
convert their literal 0 to literal 1). Julien - do you still think 2) is
pretty important to avoid at the, overall, 2 places that would be left?

> --- a/xen/arch/x86/pv/dom0_build.c
> +++ b/xen/arch/x86/pv/dom0_build.c
> @@ -556,7 +556,7 @@ int __init dom0_construct_pv(struct domain *d,
>          else
>          {
>              while ( count-- )
> -                if ( assign_pages(d, mfn_to_page(_mfn(mfn++)), 0, 0) )
> +                if ( assign_page(d, mfn_to_page(_mfn(mfn++)), 0, 0) )

I think in all cases where order-0 pages get passed, you'd rather want
to call assign_pages() directly (if, as per above, we'll keep both
functions in the first place).

> --- a/xen/common/page_alloc.c
> +++ b/xen/common/page_alloc.c
> @@ -2283,8 +2283,8 @@ void init_domheap_pages(paddr_t ps, paddr_t pe)
>  
>  int assign_pages(
>      struct domain *d,
> +    unsigned long nr,
>      struct page_info *pg,
> -    unsigned int order,
>      unsigned int memflags)
>  {

I'm afraid I consider putting "nr" ahead of "pg" unusual (considering
the rest of our code base). How about

int assign_pages(
    struct page_info *pg,
    unsigned long nr,
    struct domain *d,
    unsigned int memflags)
{

?

> @@ -2354,6 +2354,11 @@ int assign_pages(
>      return rc;
>  }
>  
> +int assign_page(struct domain *d, struct page_info *pg, unsigned int order,
> +                unsigned int memflags)
> +{
> +    return assign_pages(d, (1UL << order), pg, memflags);

May I ask that you omit the unnecessary parentheses?

Jan

Re: [PATCH V3 08/10] xen/arm: introduce acquire_staticmem_pages and acquire_domstatic_pages

[Jan Beulich]

On 15.07.2021 07:18, Penny Zheng wrote:
> @@ -1065,6 +1069,73 @@ static struct page_info *alloc_heap_pages(
>      return pg;
>  }
>  
> +#ifdef CONFIG_STATIC_MEMORY
> +/*
> + * Acquire nr_mfns contiguous reserved pages, starting at #smfn, of
> + * static memory.
> + */
> +static struct page_info *acquire_staticmem_pages(unsigned long nr_mfns,
> +                                                 mfn_t smfn,
> +                                                 unsigned int memflags)

This and the other function not being __init, and there neither being
any caller nor any freeing, a question is whether __init wants adding;
patch 10 suggests so. The lack of freeing in particular means that
domains created using static memory won't be restartable, requiring a
host reboot instead.

> +{
> +    bool need_tlbflush = false;
> +    uint32_t tlbflush_timestamp = 0;
> +    unsigned long i;
> +    struct page_info *pg;
> +
> +    /* For now, it only supports pre-configured static memory. */
> +    if ( !mfn_valid(smfn) || !nr_mfns )
> +        return NULL;
> +
> +    spin_lock(&heap_lock);
> +
> +    pg = mfn_to_page(smfn);
> +
> +    for ( i = 0; i < nr_mfns; i++ )
> +    {
> +        /*
> +         * Reference count must continuously be zero for free pages
> +         * of static memory(PGC_reserved).
> +         */
> +        if ( pg[i].count_info != (PGC_state_free | PGC_reserved) )
> +        {
> +            printk(XENLOG_ERR
> +                   "pg[%lu] Static MFN %"PRI_mfn" c=%#lx t=%#x\n",
> +                   i, mfn_x(page_to_mfn(pg + i)),
> +                   pg[i].count_info, pg[i].tlbflush_timestamp);
> +            BUG();
> +        }
> +
> +        if ( !(memflags & MEMF_no_tlbflush) )
> +            accumulate_tlbflush(&need_tlbflush, &pg[i],
> +                                &tlbflush_timestamp);
> +
> +        /*
> +         * Preserve flag PGC_reserved and change page state
> +         * to PGC_state_inuse.
> +         */
> +        pg[i].count_info = (PGC_reserved | PGC_state_inuse);
> +        /* Initialise fields which have other uses for free pages. */
> +        pg[i].u.inuse.type_info = 0;
> +        page_set_owner(&pg[i], NULL);
> +
> +        /*
> +         * Ensure cache and RAM are consistent for platforms where the
> +         * guest can control its own visibility of/through the cache.
> +         */
> +        flush_page_to_ram(mfn_x(page_to_mfn(&pg[i])),
> +                            !(memflags & MEMF_no_icache_flush));

Indentation.

> +    }
> +
> +    if ( need_tlbflush )
> +        filtered_flush_tlb_mask(tlbflush_timestamp);
> +
> +    spin_unlock(&heap_lock);

I'm pretty sure I did comment before on the flush_page_to_ram() being
inside the locked region here. While XSA-364 doesn't apply here because
you don't defer scrubbing (yet), the flushing may still take too long
to happen inside the locked region. Of course there's a dependency here
on when exactly the call(s) to this code will happen. In particular if
other DomU-s can already be running at that point, this may not be
tolerable by some of them. Yet if this is intentional and deemed not a
problem, then I'd have kind of expected the description to mention this
difference from alloc_heap_pages(), which you say this is an equivalent
of.

> @@ -2411,6 +2483,42 @@ struct page_info *alloc_domheap_pages(
>      return pg;
>  }
>  
> +#ifdef CONFIG_STATIC_MEMORY
> +/*
> + * Acquire nr_mfns contiguous pages, starting at #smfn, of static memory,
> + * then assign them to one specific domain #d.
> + */
> +struct page_info *acquire_domstatic_pages(
> +        struct domain *d, unsigned long nr_mfns, mfn_t smfn,
> +        unsigned int memflags)
> +{
> +    struct page_info *pg = NULL;
> +
> +    ASSERT(!in_irq());
> +
> +    pg = acquire_staticmem_pages(nr_mfns, smfn, memflags);
> +    if ( !pg )
> +        return NULL;
> +
> +    /* Right now, MEMF_no_owner case is meaningless here. */
> +    ASSERT(d);
> +    if ( memflags & MEMF_no_refcount )
> +    {
> +        unsigned long i;
> +
> +        for ( i = 0; i < nr_mfns; i++ )
> +            pg[i].count_info |= PGC_extra;
> +    }

With MEMF_no_owner now called out as meaningless here, what about
MEMF_no_refcount?

Jan

Re: [PATCH V3 08/10] xen/arm: introduce acquire_staticmem_pages and acquire_domstatic_pages

[Julien Grall]

Hi Jan,

On 19/07/2021 10:26, Jan Beulich wrote:
> On 15.07.2021 07:18, Penny Zheng wrote:
>> @@ -1065,6 +1069,73 @@ static struct page_info *alloc_heap_pages(
>>       return pg;
>>   }
>>   
>> +#ifdef CONFIG_STATIC_MEMORY
>> +/*
>> + * Acquire nr_mfns contiguous reserved pages, starting at #smfn, of
>> + * static memory.
>> + */
>> +static struct page_info *acquire_staticmem_pages(unsigned long nr_mfns,
>> +                                                 mfn_t smfn,
>> +                                                 unsigned int memflags)
> 
> This and the other function not being __init, and there neither being
> any caller nor any freeing, a question is whether __init wants adding;
> patch 10 suggests so. The lack of freeing in particular means that
> domains created using static memory won't be restartable, requiring a
> host reboot instead.

I am open to request an host reboot in case of an issue. Although, it 
would be good to have code in place for that.

Regardless the reboot part, I would still expect the domain to balloon 
in/out memory. This is pretty broken today because Xen would end up to 
give the memory to the heap allocator and the next allocation would be 
through the heap allocate.

For runtime "free/allocate", we may want to follow the same behavior as 
direct-mapped domain (i.e. GFN == MFN): the page will not be given back 
to any allocator and we only check if the page belongs to the domain on 
allocation.

So adding __init for acquire_staticmem_pages() is probably fine.

On a side node, on v2, I have requested to keep track of the list of 
missing pieces. @Penny, where can I find the list?

Cheers,

-- 
Julien Grall

RE: [PATCH V3 05/10] xen/arm: static memory initialization

[Penny Zheng]

Hi Jan

> -----Original Message-----
> From: Jan Beulich <jbeulich@suse.com>
> Sent: Monday, July 19, 2021 4:20 PM
> To: Penny Zheng <Penny.Zheng@arm.com>
> Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>; Wei Chen
> <Wei.Chen@arm.com>; nd <nd@arm.com>; xen-devel@lists.xenproject.org;
> sstabellini@kernel.org; julien@xen.org
> Subject: Re: [PATCH V3 05/10] xen/arm: static memory initialization
> 
> On 15.07.2021 07:18, Penny Zheng wrote:
> > v3 change:
> > - include addition of CONFIG_STATIC_ALLOCATION in this commit, where
> > it is firstly used and also change the name to CONFIG_STATIC_MEMORY
> > - Fix TAB typo in Kconfig
> 
> Not sure what this relates to, but ...
> 

Before I was wrongly using " set tabstop=4 " for Kconfig file, so...

> > --- a/xen/arch/arm/Kconfig
> > +++ b/xen/arch/arm/Kconfig
> > @@ -106,6 +106,9 @@ config TEE
> >
> >  source "arch/arm/tee/Kconfig"
> >
> > +config STATIC_MEMORY
> > +        def_bool y
> 
> ... this is (wrongly) using spaces for indentation.
> 
> I also wonder about the placement: Shouldn't the option live in common code,
> with Arm "select"ing it?
> 

Sure, I will place it in xen/common/Kconfig, and add "depends on ARM".

> > --- a/xen/arch/arm/setup.c
> > +++ b/xen/arch/arm/setup.c
> > @@ -622,6 +622,28 @@ static void __init init_pdx(void)
> >      }
> >  }
> >
> > +/* Static memory initialization */
> > +static void __init init_staticmem_pages(void) {
> > +    unsigned int bank;
> > +
> > +    /* TODO: Considering NUMA-support scenario. */
> > +    for ( bank = 0 ; bank < bootinfo.static_mem.nr_banks; bank++ )
> > +    {
> > +        paddr_t bank_start = bootinfo.static_mem.bank[bank].start;
> > +        paddr_t bank_size = bootinfo.static_mem.bank[bank].size;
> > +        paddr_t bank_end = bank_start + bank_size;
> > +
> > +        bank_start = round_pgup(bank_start);
> > +        bank_end = round_pgdown(bank_end);
> > +        if ( bank_end <= bank_start )
> > +            return;
> > +
> > +        free_staticmem_pages(maddr_to_page(bank_start),
> > +                            (bank_end - bank_start) >> PAGE_SHIFT,
> > + false);
> 
> Indentation (one too few spaces). Perhaps also consider to avoid open-coding
> PFN_DOWN() here; in fact it and PFN_UP() could be used in place of
> round_pg{down,up}() above.
> 

Sure. I will replace round_pg{down,up}() with PFN_DOWN()/PFN_UP().

> Jan

Cheers

Penny Zheng

RE: [PATCH V3 07/10] xen: re-define assign_pages and introduce assign_page

[Penny Zheng]

Hi Jan

> -----Original Message-----
> From: Jan Beulich <jbeulich@suse.com>
> Sent: Monday, July 19, 2021 4:41 PM
> To: Penny Zheng <Penny.Zheng@arm.com>; julien@xen.org
> Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>; Wei Chen
> <Wei.Chen@arm.com>; nd <nd@arm.com>; xen-devel@lists.xenproject.org;
> sstabellini@kernel.org
> Subject: Re: [PATCH V3 07/10] xen: re-define assign_pages and introduce
> assign_page
> 
> On 15.07.2021 07:18, Penny Zheng wrote:
> > In order to deal with the trouble of count-to-order conversion when
> > page number is not in a power-of-two, this commit re-define
> > assign_pages for nr pages and assign_page for original page with a single
> order.
> >
> > Backporting confusion could be helped by altering the order of
> > assign_pages parameters, such that the compiler would point out that
> > adjustments at call sites are needed.
> 
> Back on the initial form of this patch Julien said:
> 
> "However, I think I would prefer if we introduce a new interface (maybe
>  assign_pages_nr()) rather than change the meaning of the field. This is  for two
> reasons:
>     1) We limit the risk to make mistake when backporting a patch touch
> assign_pages().
>     2) Adding (1UL << order) for pretty much all the caller is not nice."
> 
> 1) is taken care of here anyway (albeit see the remark below), and of the
> callers only some would really need "1UL <<" added (others could simply
> convert their literal 0 to literal 1). Julien - do you still think 2) is pretty
> important to avoid at the, overall, 2 places that would be left?
> 
> > --- a/xen/arch/x86/pv/dom0_build.c
> > +++ b/xen/arch/x86/pv/dom0_build.c
> > @@ -556,7 +556,7 @@ int __init dom0_construct_pv(struct domain *d,
> >          else
> >          {
> >              while ( count-- )
> > -                if ( assign_pages(d, mfn_to_page(_mfn(mfn++)), 0, 0) )
> > +                if ( assign_page(d, mfn_to_page(_mfn(mfn++)), 0, 0) )
> 
> I think in all cases where order-0 pages get passed, you'd rather want to call
> assign_pages() directly (if, as per above, we'll keep both functions in the first
> place).
> 

Sure.  I'll use literal 1, it's more reasonable to me also.

> > --- a/xen/common/page_alloc.c
> > +++ b/xen/common/page_alloc.c
> > @@ -2283,8 +2283,8 @@ void init_domheap_pages(paddr_t ps, paddr_t pe)
> >
> >  int assign_pages(
> >      struct domain *d,
> > +    unsigned long nr,
> >      struct page_info *pg,
> > -    unsigned int order,
> >      unsigned int memflags)
> >  {
> 
> I'm afraid I consider putting "nr" ahead of "pg" unusual (considering the rest of
> our code base). How about
> 

> int assign_pages(
>     struct page_info *pg,
>     unsigned long nr,
>     struct domain *d,
>     unsigned int memflags)
> {
> 
> ?
> 

Sure. Thx for reconstructing.

> > @@ -2354,6 +2354,11 @@ int assign_pages(
> >      return rc;
> >  }
> >
> > +int assign_page(struct domain *d, struct page_info *pg, unsigned int order,
> > +                unsigned int memflags) {
> > +    return assign_pages(d, (1UL << order), pg, memflags);
> 
> May I ask that you omit the unnecessary parentheses?
> 

Oh, sorry. I'll remove it, loopy head sometimes...

> Jan

Cheers
Penny

RE: [PATCH V3 08/10] xen/arm: introduce acquire_staticmem_pages and acquire_domstatic_pages

[Penny Zheng]

Hi Jan

> -----Original Message-----
> From: Jan Beulich <jbeulich@suse.com>
> Sent: Monday, July 19, 2021 5:26 PM
> To: Penny Zheng <Penny.Zheng@arm.com>
> Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>; Wei Chen
> <Wei.Chen@arm.com>; nd <nd@arm.com>; xen-devel@lists.xenproject.org;
> sstabellini@kernel.org; julien@xen.org
> Subject: Re: [PATCH V3 08/10] xen/arm: introduce acquire_staticmem_pages
> and acquire_domstatic_pages
> 
> On 15.07.2021 07:18, Penny Zheng wrote:
> > @@ -1065,6 +1069,73 @@ static struct page_info *alloc_heap_pages(
> >      return pg;
> >  }
> >
> > +#ifdef CONFIG_STATIC_MEMORY
> > +/*
> > + * Acquire nr_mfns contiguous reserved pages, starting at #smfn, of
> > + * static memory.
> > + */
> > +static struct page_info *acquire_staticmem_pages(unsigned long nr_mfns,
> > +                                                 mfn_t smfn,
> > +                                                 unsigned int
> > +memflags)
> 
> This and the other function not being __init, and there neither being any caller
> nor any freeing, a question is whether __init wants adding; patch 10 suggests
> so. The lack of freeing in particular means that domains created using static
> memory won't be restartable, requiring a host reboot instead.
> 

Right now, all domain on static allocation get constructed through device tree, in
boot-up time.  "__init" is preferred.

> > +{
> > +    bool need_tlbflush = false;
> > +    uint32_t tlbflush_timestamp = 0;
> > +    unsigned long i;
> > +    struct page_info *pg;
> > +
> > +    /* For now, it only supports pre-configured static memory. */
> > +    if ( !mfn_valid(smfn) || !nr_mfns )
> > +        return NULL;
> > +
> > +    spin_lock(&heap_lock);
> > +
> > +    pg = mfn_to_page(smfn);
> > +
> > +    for ( i = 0; i < nr_mfns; i++ )
> > +    {
> > +        /*
> > +         * Reference count must continuously be zero for free pages
> > +         * of static memory(PGC_reserved).
> > +         */
> > +        if ( pg[i].count_info != (PGC_state_free | PGC_reserved) )
> > +        {
> > +            printk(XENLOG_ERR
> > +                   "pg[%lu] Static MFN %"PRI_mfn" c=%#lx t=%#x\n",
> > +                   i, mfn_x(page_to_mfn(pg + i)),
> > +                   pg[i].count_info, pg[i].tlbflush_timestamp);
> > +            BUG();
> > +        }
> > +
> > +        if ( !(memflags & MEMF_no_tlbflush) )
> > +            accumulate_tlbflush(&need_tlbflush, &pg[i],
> > +                                &tlbflush_timestamp);
> > +
> > +        /*
> > +         * Preserve flag PGC_reserved and change page state
> > +         * to PGC_state_inuse.
> > +         */
> > +        pg[i].count_info = (PGC_reserved | PGC_state_inuse);
> > +        /* Initialise fields which have other uses for free pages. */
> > +        pg[i].u.inuse.type_info = 0;
> > +        page_set_owner(&pg[i], NULL);
> > +
> > +        /*
> > +         * Ensure cache and RAM are consistent for platforms where the
> > +         * guest can control its own visibility of/through the cache.
> > +         */
> > +        flush_page_to_ram(mfn_x(page_to_mfn(&pg[i])),
> > +                            !(memflags & MEMF_no_icache_flush));
> 
> Indentation.
> 
> > +    }
> > +
> > +    if ( need_tlbflush )
> > +        filtered_flush_tlb_mask(tlbflush_timestamp);
> > +
> > +    spin_unlock(&heap_lock);
> 
> I'm pretty sure I did comment before on the flush_page_to_ram() being inside
> the locked region here. While XSA-364 doesn't apply here because you don't
> defer scrubbing (yet), the flushing may still take too long to happen inside the
> locked region. Of course there's a dependency here on when exactly the call(s)
> to this code will happen. In particular if other DomU-s can already be running
> at that point, this may not be tolerable by some of them. Yet if this is
> intentional and deemed not a problem, then I'd have kind of expected the
> description to mention this difference from alloc_heap_pages(), which you say
> this is an equivalent of.
> 

Sorry for missing the comments on the flush_page_to_ram() being inside the
locked region.

Taking a while reading the XSA-364, you're right, especially with asynchronous
scrubbing in the to-do list, putting cleaning cache outside of the locked region
is necessary here.

> > @@ -2411,6 +2483,42 @@ struct page_info *alloc_domheap_pages(
> >      return pg;
> >  }
> >
> > +#ifdef CONFIG_STATIC_MEMORY
> > +/*
> > + * Acquire nr_mfns contiguous pages, starting at #smfn, of static
> > +memory,
> > + * then assign them to one specific domain #d.
> > + */
> > +struct page_info *acquire_domstatic_pages(
> > +        struct domain *d, unsigned long nr_mfns, mfn_t smfn,
> > +        unsigned int memflags)
> > +{
> > +    struct page_info *pg = NULL;
> > +
> > +    ASSERT(!in_irq());
> > +
> > +    pg = acquire_staticmem_pages(nr_mfns, smfn, memflags);
> > +    if ( !pg )
> > +        return NULL;
> > +
> > +    /* Right now, MEMF_no_owner case is meaningless here. */
> > +    ASSERT(d);
> > +    if ( memflags & MEMF_no_refcount )
> > +    {
> > +        unsigned long i;
> > +
> > +        for ( i = 0; i < nr_mfns; i++ )
> > +            pg[i].count_info |= PGC_extra;
> > +    }
> 
> With MEMF_no_owner now called out as meaningless here, what about
> MEMF_no_refcount?
> 

I could not think of a case where "memflags & MEMF_no_refcount" is needed right now.

All acquired pages are for guest RAM right now, extra pages like grant table, etc, are not supported
on domain on static allocation, even more, any dom0-less domain.

I'll remove this part and put a few comments on it.

> Jan

Cheers
Penny

RE: [PATCH V3 08/10] xen/arm: introduce acquire_staticmem_pages and acquire_domstatic_pages

[Penny Zheng]

Hi Julien 

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: Monday, July 19, 2021 6:00 PM
> To: Jan Beulich <jbeulich@suse.com>; Penny Zheng <Penny.Zheng@arm.com>
> Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>; Wei Chen
> <Wei.Chen@arm.com>; nd <nd@arm.com>; xen-devel@lists.xenproject.org;
> sstabellini@kernel.org
> Subject: Re: [PATCH V3 08/10] xen/arm: introduce acquire_staticmem_pages
> and acquire_domstatic_pages
> 
> Hi Jan,
> 
> On 19/07/2021 10:26, Jan Beulich wrote:
> > On 15.07.2021 07:18, Penny Zheng wrote:
> >> @@ -1065,6 +1069,73 @@ static struct page_info *alloc_heap_pages(
> >>       return pg;
> >>   }
> >>
> >> +#ifdef CONFIG_STATIC_MEMORY
> >> +/*
> >> + * Acquire nr_mfns contiguous reserved pages, starting at #smfn, of
> >> + * static memory.
> >> + */
> >> +static struct page_info *acquire_staticmem_pages(unsigned long nr_mfns,
> >> +                                                 mfn_t smfn,
> >> +                                                 unsigned int
> >> +memflags)
> >
> > This and the other function not being __init, and there neither being
> > any caller nor any freeing, a question is whether __init wants adding;
> > patch 10 suggests so. The lack of freeing in particular means that
> > domains created using static memory won't be restartable, requiring a
> > host reboot instead.
> 
> I am open to request an host reboot in case of an issue. Although, it would be
> good to have code in place for that.
> 
> Regardless the reboot part, I would still expect the domain to balloon in/out
> memory. This is pretty broken today because Xen would end up to give the
> memory to the heap allocator and the next allocation would be through the
> heap allocate.
> 
> For runtime "free/allocate", we may want to follow the same behavior as
> direct-mapped domain (i.e. GFN == MFN): the page will not be given back to
> any allocator and we only check if the page belongs to the domain on
> allocation.
> 
> So adding __init for acquire_staticmem_pages() is probably fine.
> 
> On a side node, on v2, I have requested to keep track of the list of missing
> pieces. @Penny, where can I find the list?
> 

Oh, sorry... 

I thought you were requesting a new mail list issue to track all missing pieces in ARM...
And a second though here, you shall only mean the missing pieces for this patch serie.

I'll do a quick sum-up here and put in the Patch v4 cover letter:

TODO:
- reboot domain on static allocation.
- All memory-ops(hypercalls) regarding domain on static allocation to balloon in/out memory
- asynchronously scrubbing PGC_reserved pages
- consider domain on static allocation on NUMA-support scenario

> Cheers,
> 
> --
> Julien Grall

Cheers,

--
Penny Zheng

Re: [PATCH V3 05/10] xen/arm: static memory initialization

[Jan Beulich]

On 21.07.2021 05:07, Penny Zheng wrote:
>> -----Original Message-----
>> From: Jan Beulich <jbeulich@suse.com>
>> Sent: Monday, July 19, 2021 4:20 PM
>> To: Penny Zheng <Penny.Zheng@arm.com>
>> Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>; Wei Chen
>> <Wei.Chen@arm.com>; nd <nd@arm.com>; xen-devel@lists.xenproject.org;
>> sstabellini@kernel.org; julien@xen.org
>> Subject: Re: [PATCH V3 05/10] xen/arm: static memory initialization
>>
>> On 15.07.2021 07:18, Penny Zheng wrote:
>>> v3 change:
>>> - include addition of CONFIG_STATIC_ALLOCATION in this commit, where
>>> it is firstly used and also change the name to CONFIG_STATIC_MEMORY
>>> - Fix TAB typo in Kconfig
>>
>> Not sure what this relates to, but ...
>>
> 
> Before I was wrongly using " set tabstop=4 " for Kconfig file, so...
> 
>>> --- a/xen/arch/arm/Kconfig
>>> +++ b/xen/arch/arm/Kconfig
>>> @@ -106,6 +106,9 @@ config TEE
>>>
>>>  source "arch/arm/tee/Kconfig"
>>>
>>> +config STATIC_MEMORY
>>> +        def_bool y
>>
>> ... this is (wrongly) using spaces for indentation.
>>
>> I also wonder about the placement: Shouldn't the option live in common code,
>> with Arm "select"ing it?
>>
> 
> Sure, I will place it in xen/common/Kconfig, and add "depends on ARM".

But I didn't say to use "depends on", but "select". "depends on"
would be appropriate to use if the option had a prompt (i.e. if
one could also disable it on Arm).

Jan

RE: [PATCH V3 10/10] xen/arm: introduce allocate_static_memory

[Penny Zheng]

Hi Julien

> -----Original Message-----
> From: Penny Zheng <penny.zheng@arm.com>
> Sent: Thursday, July 15, 2021 1:18 PM
> To: xen-devel@lists.xenproject.org; sstabellini@kernel.org; julien@xen.org
> Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>; Penny Zheng
> <Penny.Zheng@arm.com>; Wei Chen <Wei.Chen@arm.com>;
> jbeulich@suse.com; nd <nd@arm.com>
> Subject: [PATCH V3 10/10] xen/arm: introduce allocate_static_memory
> 
> This commit introduces allocate_static_memory to allocate static memory as
> guest RAM for Domain on Static Allocation.
> 
> It uses acquire_domstatic_pages to acquire pre-configured static memory for
> this domain, and uses guest_physmap_add_page to set up P2M table.
> These pre-defined static memory banks shall be firstly mapped to the fixed
> guest RAM address `GUEST_RAM0_BASE`. And until it exhausts the
> `GUEST_RAM0_SIZE`, it will seek to `GUEST_RAM1_BASE`, and so on.
> `GUEST_RAM0` may take up several pre-defined physical RAM regions.
> 
> Signed-off-by: Penny Zheng <penny.zheng@arm.com>
> ---
> v3 changes:
> - parse "xen,static-mem" in way of phandle back to property.
> - use unsigned int for index
> - rename allocate_static_bank_memory to append_static_memory_to_bank
> - infer the next GFN from the bank information
> - simplify the code in double loop.
> ---
>  xen/arch/arm/domain_build.c | 137
> +++++++++++++++++++++++++++++++++++-
>  1 file changed, 135 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index cdb16f2086..ed290ee31b 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -480,6 +480,139 @@ fail:
>            (unsigned long)kinfo->unassigned_mem >> 10);  }
> 
> +static bool __init append_static_memory_to_bank(struct domain *d,
> +                                                struct membank *bank,
> +                                                mfn_t smfn,
> +                                                paddr_t size) {
> +    int res;
> +    paddr_t tot_size = size;
> +    /* Infer next GFN. */
> +    gfn_t sgfn = gaddr_to_gfn(bank->start + bank->size);
> +
> +    while ( tot_size > 0 )
> +    {
> +        unsigned int order = get_allocation_size(tot_size);
> +
> +        res = guest_physmap_add_page(d, sgfn, smfn, order);

When constructing Patch v4, and second thought on this commit:

Do you think that here we shall define a new function guest_physmap_add_pages for adding
nr pages p2m mapping, just like what we did for assign_pages(...).

Since right now guest_physmap_add_page is also limited to taking care of page with a single order, 
We had trouble of count-to-order conversion when page number is not in a power-of-two here too.

> +        if ( res )
> +        {
> +            dprintk(XENLOG_ERR, "Failed map pages to DOMU: %d", res);
> +            return false;
> +        }
> +
> +        smfn = mfn_add(smfn, 1UL << order);
> +        tot_size -= (1UL << (PAGE_SHIFT + order));
> +    }
> +
> +    bank->size = bank->size + size;
> +    return true;
> +}
> +
> +/* Allocate memory from static memory as RAM for one specific domain d.
> +*/ static void __init allocate_static_memory(struct domain *d,
> +                                          struct kernel_info *kinfo,
> +                                          const struct dt_property *prop,
> +                                          u32 addr_cells, u32
> +size_cells) {
> +    unsigned int nr_banks, gbank, bank = 0;
> +    const uint64_t rambase[] = GUEST_RAM_BANK_BASES;
> +    const uint64_t ramsize[] = GUEST_RAM_BANK_SIZES;
> +    const __be32 *cell;
> +    u32 reg_cells = addr_cells + size_cells;
> +    u64 tot_size = 0;
> +    paddr_t pbase, psize, gsize;
> +    mfn_t smfn;
> +
> +    /* Start with GUEST_RAM0. */
> +    kinfo->mem.nr_banks = 0;
> +    gbank = 0;
> +    gsize = ramsize[gbank];
> +    kinfo->mem.bank[gbank].start = rambase[gbank];
> +
> +    cell = (const __be32 *)prop->value;
> +    nr_banks = (prop->length) / (reg_cells * sizeof (u32));
> +    BUG_ON(nr_banks > NR_MEM_BANKS);
> +
> +    while ( bank < nr_banks )
> +    {
> +        device_tree_get_reg(&cell, addr_cells, size_cells, &pbase, &psize);
> +        tot_size += psize;
> +        smfn = maddr_to_mfn(pbase);
> +
> +        if ( !acquire_domstatic_pages(d, psize >> PAGE_SHIFT, smfn, 0) )
> +        {
> +            printk(XENLOG_ERR
> +                    "%pd: cannot acquire static memory "
> +                    "(0x%"PRIpaddr" - 0x%"PRIpaddr").\n",
> +                    d, pbase, pbase + psize);
> +            goto fail;
> +        }
> +
> +        printk(XENLOG_INFO "%pd: STATIC BANK[%d] %#"PRIpaddr"-
> %#"PRIpaddr"\n",
> +               d, bank, pbase, pbase + psize);
> +
> +        /*
> +         * It shall be mapped to the fixed guest RAM address rambase[i],
> +         * And until it exhausts the ramsize[i], it will seek to the next
> +         * rambase[i+1].
> +         */
> +        while ( 1 )
> +        {
> +            /*
> +             * The current physical bank is fully mapped.
> +             * Handle the next physical bank.
> +             */
> +            if ( gsize >= psize )
> +            {
> +                if ( !append_static_memory_to_bank(d, &kinfo->mem.bank[gbank],
> +                                                   smfn, psize) )
> +                    goto fail;
> +
> +                gsize = gsize - psize;
> +                bank++;
> +                break;
> +            }
> +            /*
> +             * Current guest bank memory is not enough to map.
> +             * Check if we have another guest bank available.
> +             * gbank refers guest memory bank index.
> +             */
> +            else if ( (gbank + 2) > GUEST_RAM_BANKS ) {
> +                printk("Exhausted the number of guest bank\n");
> +                goto fail;
> +            }
> +            else
> +            {
> +                if ( !append_static_memory_to_bank(d, &kinfo->mem.bank[gbank],
> +                                                   smfn, gsize) )
> +                    goto fail;
> +
> +                psize = psize - gsize;
> +                smfn = mfn_add(smfn, gsize >> PAGE_SHIFT);
> +                /* Update to the next guest bank. */
> +                gbank++;
> +                gsize = ramsize[gbank];
> +                kinfo->mem.bank[gbank].start = rambase[gbank];
> +            }
> +        }
> +    }
> +
> +    kinfo->mem.nr_banks = ++gbank;
> +    kinfo->unassigned_mem -= tot_size;
> +    if ( kinfo->unassigned_mem )
> +        printk(XENLOG_ERR
> +               "Size of \"memory\" property doesn't match up with the ones "
> +               "defined in \"xen,static-mem\".\n");
> +
> +    return;
> +
> +fail:
> +    panic("Failed to allocate requested static memory for domain %pd."
> +          "Fix the VMs configurations.\n",
> +          d);
> +}
> +
>  static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
>                                     const struct dt_device_node *node)  { @@ -2486,8 +2619,8
> @@ static int __init construct_domU(struct domain *d,
>      if ( !static_mem )
>          allocate_memory(d, &kinfo);
>      else
> -        /* TODO: allocate_static_memory(...). */
> -        BUG();
> +        allocate_static_memory(d, &kinfo, static_mem_prop,
> +                               static_mem_addr_cells,
> + static_mem_size_cells);
> 
>      rc = prepare_dtb_domU(d, &kinfo);
>      if ( rc < 0 )
> --
> 2.25.1

A lot thanks

Penny

RE: [PATCH V3 10/10] xen/arm: introduce allocate_static_memory

[Penny Zheng]

Hi Julien 

> -----Original Message-----
> From: Penny Zheng
> Sent: Tuesday, July 27, 2021 11:45 AM
> To: julien@xen.org
> Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>; Wei Chen
> <Wei.Chen@arm.com>; jbeulich@suse.com; xen-devel@lists.xenproject.org;
> sstabellini@kernel.org
> Subject: RE: [PATCH V3 10/10] xen/arm: introduce allocate_static_memory
> 
> Hi Julien
> 
> > -----Original Message-----
> > From: Penny Zheng <penny.zheng@arm.com>
> > Sent: Thursday, July 15, 2021 1:18 PM
> > To: xen-devel@lists.xenproject.org; sstabellini@kernel.org;
> > julien@xen.org
> > Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>; Penny Zheng
> > <Penny.Zheng@arm.com>; Wei Chen <Wei.Chen@arm.com>;
> jbeulich@suse.com;
> > nd <nd@arm.com>
> > Subject: [PATCH V3 10/10] xen/arm: introduce allocate_static_memory
> >
> > This commit introduces allocate_static_memory to allocate static
> > memory as guest RAM for Domain on Static Allocation.
> >
> > It uses acquire_domstatic_pages to acquire pre-configured static
> > memory for this domain, and uses guest_physmap_add_page to set up P2M
> table.
> > These pre-defined static memory banks shall be firstly mapped to the
> > fixed guest RAM address `GUEST_RAM0_BASE`. And until it exhausts the
> > `GUEST_RAM0_SIZE`, it will seek to `GUEST_RAM1_BASE`, and so on.
> > `GUEST_RAM0` may take up several pre-defined physical RAM regions.
> >
> > Signed-off-by: Penny Zheng <penny.zheng@arm.com>
> > ---
> > v3 changes:
> > - parse "xen,static-mem" in way of phandle back to property.
> > - use unsigned int for index
> > - rename allocate_static_bank_memory to append_static_memory_to_bank
> > - infer the next GFN from the bank information
> > - simplify the code in double loop.
> > ---
> >  xen/arch/arm/domain_build.c | 137
> > +++++++++++++++++++++++++++++++++++-
> >  1 file changed, 135 insertions(+), 2 deletions(-)
> >
> > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> > index cdb16f2086..ed290ee31b 100644
> > --- a/xen/arch/arm/domain_build.c
> > +++ b/xen/arch/arm/domain_build.c
> > @@ -480,6 +480,139 @@ fail:
> >            (unsigned long)kinfo->unassigned_mem >> 10);  }
> >
> > +static bool __init append_static_memory_to_bank(struct domain *d,
> > +                                                struct membank *bank,
> > +                                                mfn_t smfn,
> > +                                                paddr_t size) {
> > +    int res;
> > +    paddr_t tot_size = size;
> > +    /* Infer next GFN. */
> > +    gfn_t sgfn = gaddr_to_gfn(bank->start + bank->size);
> > +
> > +    while ( tot_size > 0 )
> > +    {
> > +        unsigned int order = get_allocation_size(tot_size);
> > +
> > +        res = guest_physmap_add_page(d, sgfn, smfn, order);
> 
> When constructing Patch v4, and second thought on this commit:
> 
> Do you think that here we shall define a new function
> guest_physmap_add_pages for adding nr pages p2m mapping, just like what
> we did for assign_pages(...).
> 
> Since right now guest_physmap_add_page is also limited to taking care of
> page with a single order, We had trouble of count-to-order conversion when
> page number is not in a power-of-two here too.

Extra info on second thought:

If doing above changes, draft diff is as follows. 

Interface of guest_physmap_add_entry is also needing changes, parameter
from "unsigned long page_order " to "unsigned long nr_pages". 
From the point view of ARM, page_order is always taken as 1 << page_order, so
IMO the transfer is seamless.

However, to be compatible, we shared the same interface "guest_physmap_add_entry" with x86,
sharing the same name and parameters but with different implementation.

Even now, we didn't using this interface on any common codes, so we could keep
above changes only on ARM. But it definitely is a hidden trouble in the future...

So hmmmm, Wdyt? 

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index d414c4feb9..07c9f73809 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -1376,10 +1376,10 @@ int map_dev_mmio_region(struct domain *d,
 int guest_physmap_add_entry(struct domain *d,
                             gfn_t gfn,
                             mfn_t mfn,
-                            unsigned long page_order,
+                            unsigned long nr_pages,
                             p2m_type_t t)
 {
-    return p2m_insert_mapping(d, gfn, (1 << page_order), mfn, t);
+    return p2m_insert_mapping(d, gfn, nr_pages, mfn, t);
 }

 int guest_physmap_remove_page(struct domain *d, gfn_t gfn, mfn_t mfn,

diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
index 4f8b3b0ec7..a869639fb7 100644
--- a/xen/include/asm-arm/p2m.h
+++ b/xen/include/asm-arm/p2m.h
@@ -301,7 +301,7 @@ int map_dev_mmio_region(struct domain *d,
 int guest_physmap_add_entry(struct domain *d,
                             gfn_t gfn,
                             mfn_t mfn,
-                            unsigned long page_order,
+                            unsigned long nr_pages,
                             p2m_type_t t);

 /* Untyped version for RAM only, for compatibility */
@@ -310,7 +310,15 @@ static inline int guest_physmap_add_page(struct domain *d,
                                          mfn_t mfn,
                                          unsigned int page_order)
 {
-    return guest_physmap_add_entry(d, gfn, mfn, page_order, p2m_ram_rw);
+    return guest_physmap_add_entry(d, gfn, mfn, (1UL << page_order), p2m_ram_rw);
+}
+
+static inline int guest_physmap_add_pages(struct domain *d,
+                                          gfn_t gfn,
+                                          mfn_t mfn,
+                                          unsigned long nr_pages)
+{
+    return guest_physmap_add_entry(d, gfn, mfn, nr_pages, p2m_ram_rw);
 }

 mfn_t gfn_to_mfn(struct domain *d, gfn_t gfn);


> > +        if ( res )
> > +        {
> > +            dprintk(XENLOG_ERR, "Failed map pages to DOMU: %d", res);
> > +            return false;
> > +        }
> > +
> > +        smfn = mfn_add(smfn, 1UL << order);
> > +        tot_size -= (1UL << (PAGE_SHIFT + order));
> > +    }
> > +
> > +    bank->size = bank->size + size;
> > +    return true;
> > +}
> > +
> > +/* Allocate memory from static memory as RAM for one specific domain d.
> > +*/ static void __init allocate_static_memory(struct domain *d,
> > +                                          struct kernel_info *kinfo,
> > +                                          const struct dt_property *prop,
> > +                                          u32 addr_cells, u32
> > +size_cells) {
> > +    unsigned int nr_banks, gbank, bank = 0;
> > +    const uint64_t rambase[] = GUEST_RAM_BANK_BASES;
> > +    const uint64_t ramsize[] = GUEST_RAM_BANK_SIZES;
> > +    const __be32 *cell;
> > +    u32 reg_cells = addr_cells + size_cells;
> > +    u64 tot_size = 0;
> > +    paddr_t pbase, psize, gsize;
> > +    mfn_t smfn;
> > +
> > +    /* Start with GUEST_RAM0. */
> > +    kinfo->mem.nr_banks = 0;
> > +    gbank = 0;
> > +    gsize = ramsize[gbank];
> > +    kinfo->mem.bank[gbank].start = rambase[gbank];
> > +
> > +    cell = (const __be32 *)prop->value;
> > +    nr_banks = (prop->length) / (reg_cells * sizeof (u32));
> > +    BUG_ON(nr_banks > NR_MEM_BANKS);
> > +
> > +    while ( bank < nr_banks )
> > +    {
> > +        device_tree_get_reg(&cell, addr_cells, size_cells, &pbase, &psize);
> > +        tot_size += psize;
> > +        smfn = maddr_to_mfn(pbase);
> > +
> > +        if ( !acquire_domstatic_pages(d, psize >> PAGE_SHIFT, smfn, 0) )
> > +        {
> > +            printk(XENLOG_ERR
> > +                    "%pd: cannot acquire static memory "
> > +                    "(0x%"PRIpaddr" - 0x%"PRIpaddr").\n",
> > +                    d, pbase, pbase + psize);
> > +            goto fail;
> > +        }
> > +
> > +        printk(XENLOG_INFO "%pd: STATIC BANK[%d] %#"PRIpaddr"-
> > %#"PRIpaddr"\n",
> > +               d, bank, pbase, pbase + psize);
> > +
> > +        /*
> > +         * It shall be mapped to the fixed guest RAM address rambase[i],
> > +         * And until it exhausts the ramsize[i], it will seek to the next
> > +         * rambase[i+1].
> > +         */
> > +        while ( 1 )
> > +        {
> > +            /*
> > +             * The current physical bank is fully mapped.
> > +             * Handle the next physical bank.
> > +             */
> > +            if ( gsize >= psize )
> > +            {
> > +                if ( !append_static_memory_to_bank(d, &kinfo->mem.bank[gbank],
> > +                                                   smfn, psize) )
> > +                    goto fail;
> > +
> > +                gsize = gsize - psize;
> > +                bank++;
> > +                break;
> > +            }
> > +            /*
> > +             * Current guest bank memory is not enough to map.
> > +             * Check if we have another guest bank available.
> > +             * gbank refers guest memory bank index.
> > +             */
> > +            else if ( (gbank + 2) > GUEST_RAM_BANKS ) {
> > +                printk("Exhausted the number of guest bank\n");
> > +                goto fail;
> > +            }
> > +            else
> > +            {
> > +                if ( !append_static_memory_to_bank(d, &kinfo->mem.bank[gbank],
> > +                                                   smfn, gsize) )
> > +                    goto fail;
> > +
> > +                psize = psize - gsize;
> > +                smfn = mfn_add(smfn, gsize >> PAGE_SHIFT);
> > +                /* Update to the next guest bank. */
> > +                gbank++;
> > +                gsize = ramsize[gbank];
> > +                kinfo->mem.bank[gbank].start = rambase[gbank];
> > +            }
> > +        }
> > +    }
> > +
> > +    kinfo->mem.nr_banks = ++gbank;
> > +    kinfo->unassigned_mem -= tot_size;
> > +    if ( kinfo->unassigned_mem )
> > +        printk(XENLOG_ERR
> > +               "Size of \"memory\" property doesn't match up with the ones "
> > +               "defined in \"xen,static-mem\".\n");
> > +
> > +    return;
> > +
> > +fail:
> > +    panic("Failed to allocate requested static memory for domain %pd."
> > +          "Fix the VMs configurations.\n",
> > +          d);
> > +}
> > +
> >  static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
> >                                     const struct dt_device_node *node)
> > { @@ -2486,8 +2619,8 @@ static int __init construct_domU(struct domain
> *d,
> >      if ( !static_mem )
> >          allocate_memory(d, &kinfo);
> >      else
> > -        /* TODO: allocate_static_memory(...). */
> > -        BUG();
> > +        allocate_static_memory(d, &kinfo, static_mem_prop,
> > +                               static_mem_addr_cells,
> > + static_mem_size_cells);
> >
> >      rc = prepare_dtb_domU(d, &kinfo);
> >      if ( rc < 0 )
> > --
> > 2.25.1
> 
> A lot thanks
> 
> Penny