[PATCH] Xen: Design doc for 1:1 direct-map and static allocation

[PATCH] Xen: Design doc for 1:1 direct-map and static allocation

[Penny Zheng]

Create one design doc for 1:1 direct-map and static allocation.
It is the first draft and aims to describe why and how we allocate
1:1 direct-map(guest physical == physical) domains, and why and how
we let domains on static allocation.

Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
 docs/designs/static_alloc_and_direct_map.md | 239 ++++++++++++++++++++
 1 file changed, 239 insertions(+)
 create mode 100644 docs/designs/static_alloc_and_direct_map.md

diff --git a/docs/designs/static_alloc_and_direct_map.md b/docs/designs/static_alloc_and_direct_map.md
new file mode 100644
index 0000000000..fdda162188
--- /dev/null
+++ b/docs/designs/static_alloc_and_direct_map.md
@@ -0,0 +1,239 @@
+# Preface
+
+The document is an early draft for 1:1 direct-map memory map
+(`guest physical == physical`) of domUs and Static Allocation.
+Since the implementation of these two features shares a lot, we would like
+to introduce both in one design.
+
+Right now, these two features are limited to ARM architecture.
+
+This design aims to describe why and how the guest would be created as 1:1
+direct-map domain, and why and what the static allocation is.
+
+This document is partly based on Stefano Stabellini's patch serie v1:
+[direct-map DomUs](
+https://lists.xenproject.org/archives/html/xen-devel/2020-04/msg00707.html).
+
+This is a first draft and some questions are still unanswered. When this is
+the case, it will be included under chapter `DISCUSSION`.
+
+# Introduction on Static Allocation
+
+Static allocation refers to system or sub-system(domains) for which memory
+areas are pre-defined by configuration using physical address ranges.
+
+## Background
+
+Cases where needs static allocation:
+
+  * Static allocation needed whenever a system has a pre-defined non-changing
+behaviour. This is usually the case in safety world where system must behave
+the same upon reboot, so memory resource for both XEN and domains should be
+static and pre-defined.
+
+  * Static allocation needed whenever a guest wants to allocate memory
+from refined memory ranges. For example, a system has one high-speed RAM
+region, and would like to assign it to one specific domain.
+
+  * Static allocation needed whenever a system needs a guest restricted to some
+known memory area due to hardware limitations reason. For example, some device
+can only do DMA to a specific part of the memory.
+
+Limitations:
+  * There is no consideration for PV devices at the moment.
+
+## Design on Static Allocation
+
+Static allocation refers to system or sub-system(domains) for which memory
+areas are pre-defined by configuration using physical address ranges.
+
+These pre-defined memory, -- Static Momery, as parts of RAM reserved in the
+beginning, shall never go to heap allocator or boot allocator for any use.
+
+### Static Allocation for Domains
+
+### New Deivce Tree Node: `xen,static_mem`
+
+Here introduces new `xen,static_mem` node to define static memory nodes for
+one specific domain.
+
+For domains on static allocation, users need to pre-define guest RAM regions in
+configuration, through `xen,static_mem` node under approriate `domUx` node.
+
+Here is one example:
+
+
+        domU1 {
+            compatible = "xen,domain";
+            #address-cells = <0x2>;
+            #size-cells = <0x2>;
+            cpus = <2>;
+            xen,static-mem = <0x0 0xa0000000 0x0 0x20000000>;
+            ...
+        };
+
+RAM at 0xa0000000 of 512 MB are static memory reserved for domU1 as its RAM.
+
+### New Page Flag: `PGC_reserved`
+
+In order to differentiate and manage pages reserved as static memory with
+those which are allocated from heap allocator for normal domains, we shall
+introduce a new page flag `PGC_reserved` to tell.
+
+Grant pages `PGC_reserved` when initializing static memory.
+
+### New linked page list: `reserved_page_list` in  `struct domain`
+
+Right now, for normal domains, on assigning pages to domain, pages allocated
+from heap allocator as guest RAM shall be inserted to one linked page
+list `page_list` for later managing and storing.
+
+So in order to tell, pages allocated from static memory, shall be inserted
+to a different linked page list `reserved_page_list`.
+
+Later, when domain get destroyed and memory relinquished, only pages in
+`page_list` go back to heap, and pages in `reserved_page_list` shall not.
+
+### Memory Allocation for Domains on Static Allocation
+
+RAM regions pre-defined as static memory for one specifc domain shall be parsed
+and reserved from the beginning. And they shall never go to any memory
+allocator for any use.
+
+Later when allocating static memory for this specific domain, after acquiring
+those reserved regions, users need to a do set of verification before
+assigning.
+For each page there, it at least includes the following steps:
+1. Check if it is in free state and has zero reference count.
+2. Check if the page is reserved(`PGC_reserved`).
+
+Then, assigning these pages to this specific domain, and all pages go to one
+new linked page list `reserved_page_list`.
+
+At last, set up guest P2M mapping. By default, it shall be mapped to the fixed
+guest RAM address `GUEST_RAM0_BASE`, `GUEST_RAM1_BASE`, just like normal
+domains. But later in 1:1 direct-map design, if `direct-map` is set, the guest
+physical address will equal to physical address.
+
+### Static Allocation for Xen itself
+
+### New Deivce Tree Node: `xen,reserved_heap`
+
+Static memory for Xen heap refers to parts of RAM reserved in the beginning
+for Xen heap only. The memory is pre-defined through XEN configuration
+using physical address ranges.
+
+The reserved memory for Xen heap is an optional feature and can be enabled
+by adding a device tree property in the `chosen` node. Currently, this feature
+is only supported on AArch64.
+
+Here is one example:
+
+
+        chosen {
+            xen,reserved-heap = <0x0 0x30000000 0x0 0x40000000>;
+            ...
+        };
+
+RAM at 0x30000000 of 1G size will be reserved as heap memory. Later, heap
+allocator will allocate memory only from this specific region.
+
+# Introduction on 1:1 direct-map
+
+## Background
+
+Cases where domU needs 1:1 direct-map memory map:
+
+  * IOMMU not present in the system.
+  * IOMMU disabled if it doesn't cover a specific device and all the guests
+are trusted. Thinking a mixed scenario, where a few devices with IOMMU and
+a few without, then guest DMA security still could not be totally guaranteed.
+So users may want to disable the IOMMU, to at least gain some performance
+improvement from IOMMU disabled.
+  * IOMMU disabled as a workaround when it doesn't have enough bandwidth.
+To be specific, in a few extreme situation, when multiple devices do DMA
+concurrently, these requests may exceed IOMMU's transmission capacity.
+  * IOMMU disabled when it adds too much latency on DMA. For example,
+TLB may be missing in some IOMMU hardware, which may bring latency in DMA
+progress, so users may want to disable it in some realtime scenario.
+
+*WARNING:
+Users should be aware that it is not always secure to assign a device without
+IOMMU/SMMU protection.
+When the device is not protected by the IOMMU/SMMU, the administrator should
+make sure that:
+ 1. The device is assigned to a trusted guest.
+ 2. Users have additional security mechanism on the platform.
+
+Limitations:
+  * There is no consideration for PV devices at the moment.
+
+## Design on 1:1 direct-map
+
+Here only supports 1:1 direct-map with user-defined memory regions.
+
+The implementation may cover following aspects:
+
+### Native Address and IRQ numbers for GIC and UART(vPL011)
+
+Today, fixed addresses and IRQ numbers are used to map GIC and UART(vPL011)
+in DomUs. And it may cause potential clash on 1:1 direct-map domains.
+So, Using native addresses and irq numbers for GIC, UART(vPL011), in
+1:1 direct-map domains is necessary.
+
+For the virtual interrupt of vPL011: instead of always using
+`GUEST_VPL011_SPI`, try to reuse the physical SPI number if possible.
+
+### New Device Tree Node: `direct-map` Option
+
+Introduce a new option `direct-map` for 1:1 direct-map domains.
+
+When users allocating an 1:1 direct-map domain, `direct-map` property needs
+to be added under the appropriate `/chosen/domUx`. For now, since only
+supporting 1:1 direct-map with user-defined memory regions, users must choose
+RAM banks as 1:1 dirct-map guest RAM, through `xen,static-mem`, which has
+been elaborated before in chapter `New Deivce Tree Node: `xen,static_mem``.
+
+Hers is one example to allocate one 1:1 direct-map domain:
+
+
+            chosen {
+                ...
+                domU1 {
+                    compatible = "xen, domain";
+                    #address-cells = <0x2>;
+                    #size-cells = <0x2>;
+                    cpus = <2>;
+                    vpl011;
+                    direct-map;
+                    xen,static-mem = <0x0 0x30000000 0x0 0x40000000>;
+                    ...
+                };
+                ...
+            };
+
+DOMU1 is an 1:1 direct-map domain with reserved RAM at 0x30000000 of 1G size.
+
+### Memory Allocation for 1:1 direct-map Domain
+
+Implementing memory allocation for 1:1 direct-map domain includes two parts:
+Static Allocation for Domain and 1:1 direct-map.
+
+The first part has been elaborated before in chapter `Memory Allocation for
+Domains on Static Allocation`. Then, to ensure 1:1 direct-map, when setting up
+guest P2M mapping, it needs to make sure that guest physical address equal to
+physical address(`gfn == mfn`).
+
+*DISCUSSION:
+
+  * Here only supports booting up one domain on static allocation or on 1:1
+direct-map through device tree, is `xl` also needed?
+
+  * Here only supports 1:1 direct-map domain with user-defined memory regions,
+is 1:1 direct-map domain with arbitrary memory regions also needed? We had
+quite a discussion [here](
+https://patchew.org/Xen/20201208052113.1641514-1-penny.zheng@arm.com/). In
+order to mitigate guest memory fragementation, we introduce static memory pool(
+same implementation as `xen,reserved-heap`) and static memory allocator(a new
+linear memory allocator, very alike boot allocator). This new allocator is also
+applied to MPU system, so I may create a new design on this to elaborate more.
-- 
2.17.1

Re: [PATCH] Xen: Design doc for 1:1 direct-map and static allocation

[Jan Beulich]

On 18.05.2021 07:07, Penny Zheng wrote:
> +## Background
> +
> +Cases where needs static allocation:
> +
> +  * Static allocation needed whenever a system has a pre-defined non-changing
> +behaviour. This is usually the case in safety world where system must behave
> +the same upon reboot, so memory resource for both XEN and domains should be
> +static and pre-defined.
> +
> +  * Static allocation needed whenever a guest wants to allocate memory
> +from refined memory ranges. For example, a system has one high-speed RAM
> +region, and would like to assign it to one specific domain.
> +
> +  * Static allocation needed whenever a system needs a guest restricted to some
> +known memory area due to hardware limitations reason. For example, some device
> +can only do DMA to a specific part of the memory.

This isn't a reason for fully static partitioning. Such devices also exist
in the x86 world, without there having been a need to statically partition
systems. All you want to guarantee is that for I/O purposes a domain has
_some_ memory in the accessible range.

> +Limitations:
> +  * There is no consideration for PV devices at the moment.

How would PV devices be affected? Drivers better would use grant
transfers, but that's about it afaics.

> +## Design on Static Allocation
> +
> +Static allocation refers to system or sub-system(domains) for which memory
> +areas are pre-defined by configuration using physical address ranges.
> +
> +These pre-defined memory, -- Static Momery, as parts of RAM reserved in the
> +beginning, shall never go to heap allocator or boot allocator for any use.
> +
> +### Static Allocation for Domains
> +
> +### New Deivce Tree Node: `xen,static_mem`
> +
> +Here introduces new `xen,static_mem` node to define static memory nodes for
> +one specific domain.
> +
> +For domains on static allocation, users need to pre-define guest RAM regions in
> +configuration, through `xen,static_mem` node under approriate `domUx` node.
> +
> +Here is one example:
> +
> +
> +        domU1 {
> +            compatible = "xen,domain";
> +            #address-cells = <0x2>;
> +            #size-cells = <0x2>;
> +            cpus = <2>;
> +            xen,static-mem = <0x0 0xa0000000 0x0 0x20000000>;
> +            ...
> +        };
> +
> +RAM at 0xa0000000 of 512 MB are static memory reserved for domU1 as its RAM.
> +
> +### New Page Flag: `PGC_reserved`
> +
> +In order to differentiate and manage pages reserved as static memory with
> +those which are allocated from heap allocator for normal domains, we shall
> +introduce a new page flag `PGC_reserved` to tell.

This contradicts you saying higher up "shall never go to heap allocator
or boot allocator for any use" - no such flag ought to be needed of the
allocators never get to see these pages. And even if such a flag was
needed, I can't see how it would be sufficient to express the page ->
domain relationship.

> +Grant pages `PGC_reserved` when initializing static memory.

I'm afraid I don't understand this sentence at all.

> +### New linked page list: `reserved_page_list` in  `struct domain`
> +
> +Right now, for normal domains, on assigning pages to domain, pages allocated
> +from heap allocator as guest RAM shall be inserted to one linked page
> +list `page_list` for later managing and storing.
> +
> +So in order to tell, pages allocated from static memory, shall be inserted
> +to a different linked page list `reserved_page_list`.
> +
> +Later, when domain get destroyed and memory relinquished, only pages in
> +`page_list` go back to heap, and pages in `reserved_page_list` shall not.

If such a domain can be destroyed (and re-created), how would the
association between memory and intended owner be retained / propagated?
Where else would the pages from reserved_page_list go (they need to go
somewhere, as the struct domain instance will go away)?

> +### Memory Allocation for Domains on Static Allocation
> +
> +RAM regions pre-defined as static memory for one specifc domain shall be parsed
> +and reserved from the beginning. And they shall never go to any memory
> +allocator for any use.
> +
> +Later when allocating static memory for this specific domain, after acquiring
> +those reserved regions, users need to a do set of verification before
> +assigning.
> +For each page there, it at least includes the following steps:
> +1. Check if it is in free state and has zero reference count.
> +2. Check if the page is reserved(`PGC_reserved`).

If this memory is reserved for a specific domain, why is such verification
necessary?

> +Then, assigning these pages to this specific domain, and all pages go to one
> +new linked page list `reserved_page_list`.
> +
> +At last, set up guest P2M mapping. By default, it shall be mapped to the fixed
> +guest RAM address `GUEST_RAM0_BASE`, `GUEST_RAM1_BASE`, just like normal
> +domains. But later in 1:1 direct-map design, if `direct-map` is set, the guest
> +physical address will equal to physical address.

I think you're missing "host" ahead of the 2nd "physical address"?

> +### Static Allocation for Xen itself
> +
> +### New Deivce Tree Node: `xen,reserved_heap`
> +
> +Static memory for Xen heap refers to parts of RAM reserved in the beginning
> +for Xen heap only. The memory is pre-defined through XEN configuration
> +using physical address ranges.
> +
> +The reserved memory for Xen heap is an optional feature and can be enabled
> +by adding a device tree property in the `chosen` node. Currently, this feature
> +is only supported on AArch64.

The earlier "Cases where needs static allocation" doesn't really seem to
cover any case where this would be needed for Xen itself. Without a need,
I don't see the point of having the feature.

> +## Background
> +
> +Cases where domU needs 1:1 direct-map memory map:
> +
> +  * IOMMU not present in the system.
> +  * IOMMU disabled if it doesn't cover a specific device and all the guests
> +are trusted. Thinking a mixed scenario, where a few devices with IOMMU and
> +a few without, then guest DMA security still could not be totally guaranteed.
> +So users may want to disable the IOMMU, to at least gain some performance
> +improvement from IOMMU disabled.
> +  * IOMMU disabled as a workaround when it doesn't have enough bandwidth.
> +To be specific, in a few extreme situation, when multiple devices do DMA
> +concurrently, these requests may exceed IOMMU's transmission capacity.
> +  * IOMMU disabled when it adds too much latency on DMA. For example,
> +TLB may be missing in some IOMMU hardware, which may bring latency in DMA
> +progress, so users may want to disable it in some realtime scenario.
> +
> +*WARNING:
> +Users should be aware that it is not always secure to assign a device without
> +IOMMU/SMMU protection.
> +When the device is not protected by the IOMMU/SMMU, the administrator should
> +make sure that:
> + 1. The device is assigned to a trusted guest.
> + 2. Users have additional security mechanism on the platform.
> +
> +Limitations:
> +  * There is no consideration for PV devices at the moment.

Again I'm struggling to see how PV devices might be impacted.

Jan

Re: [PATCH] Xen: Design doc for 1:1 direct-map and static allocation

[Julien Grall]

Hi Penny,

On 18/05/2021 06:07, Penny Zheng wrote:
> Create one design doc for 1:1 direct-map and static allocation.
> It is the first draft and aims to describe why and how we allocate
> 1:1 direct-map(guest physical == physical) domains, and why and how
> we let domains on static allocation.
> 
> Signed-off-by: Penny Zheng <penny.zheng@arm.com>
> ---
>   docs/designs/static_alloc_and_direct_map.md | 239 ++++++++++++++++++++
>   1 file changed, 239 insertions(+)
>   create mode 100644 docs/designs/static_alloc_and_direct_map.md
> 
> diff --git a/docs/designs/static_alloc_and_direct_map.md b/docs/designs/static_alloc_and_direct_map.md
> new file mode 100644
> index 0000000000..fdda162188
> --- /dev/null
> +++ b/docs/designs/static_alloc_and_direct_map.md
> @@ -0,0 +1,239 @@
> +# Preface
> +
> +The document is an early draft for 1:1 direct-map memory map
> +(`guest physical == physical`) of domUs and Static Allocation.
> +Since the implementation of these two features shares a lot, we would like
> +to introduce both in one design.
> +
> +Right now, these two features are limited to ARM architecture.
> +
> +This design aims to describe why and how the guest would be created as 1:1
> +direct-map domain, and why and what the static allocation is.
> +
> +This document is partly based on Stefano Stabellini's patch serie v1:
> +[direct-map DomUs](
> +https://lists.xenproject.org/archives/html/xen-devel/2020-04/msg00707.html).

While for the reviewer this is a useful information to have, I am not 
sure a future reader needs to know all the history. So I would move this 
to the commit message.

> +
> +This is a first draft and some questions are still unanswered. When this is
> +the case, it will be included under chapter `DISCUSSION`.
> +
> +# Introduction on Static Allocation
> +
> +Static allocation refers to system or sub-system(domains) for which memory
> +areas are pre-defined by configuration using physical address ranges.
> +
> +## Background
> +
> +Cases where needs static allocation:
> +
> +  * Static allocation needed whenever a system has a pre-defined non-changing
> +behaviour. This is usually the case in safety world where system must behave
> +the same upon reboot, so memory resource for both XEN and domains should be
> +static and pre-defined.
> +
> +  * Static allocation needed whenever a guest wants to allocate memory
> +from refined memory ranges. For example, a system has one high-speed RAM
> +region, and would like to assign it to one specific domain.
> +
> +  * Static allocation needed whenever a system needs a guest restricted to some
> +known memory area due to hardware limitations reason. For example, some device
> +can only do DMA to a specific part of the memory.
> +
> +Limitations:
> +  * There is no consideration for PV devices at the moment.
> +
> +## Design on Static Allocation
> +
> +Static allocation refers to system or sub-system(domains) for which memory
> +areas are pre-defined by configuration using physical address ranges.
> +
> +These pre-defined memory, -- Static Momery, as parts of RAM reserved in the

s/Momery/Memory/

> +beginning, shall never go to heap allocator or boot allocator for any use.

I think you mean "buddy" rather than "heap". Looking at your code, you 
are treating static memory region as domheap pages.

> +
> +### Static Allocation for Domains
> +
> +### New Deivce Tree Node: `xen,static_mem`

S/Deivce/

> +
> +Here introduces new `xen,static_mem` node to define static memory nodes for
> +one specific domain.
> +
> +For domains on static allocation, users need to pre-define guest RAM regions in
> +configuration, through `xen,static_mem` node under approriate `domUx` node.
> +
> +Here is one example:
> +
> +
> +        domU1 {
> +            compatible = "xen,domain";
> +            #address-cells = <0x2>;
> +            #size-cells = <0x2>;
> +            cpus = <2>;
> +            xen,static-mem = <0x0 0xa0000000 0x0 0x20000000>;
> +            ...
> +        };
> +
> +RAM at 0xa0000000 of 512 MB are static memory reserved for domU1 as its RAM.
> +
> +### New Page Flag: `PGC_reserved`
> +
> +In order to differentiate and manage pages reserved as static memory with
> +those which are allocated from heap allocator for normal domains, we shall
> +introduce a new page flag `PGC_reserved` to tell.
> +
> +Grant pages `PGC_reserved` when initializing static memory.
> +
> +### New linked page list: `reserved_page_list` in  `struct domain`
> +
> +Right now, for normal domains, on assigning pages to domain, pages allocated
> +from heap allocator as guest RAM shall be inserted to one linked page
> +list `page_list` for later managing and storing.
> +
> +So in order to tell, pages allocated from static memory, shall be inserted
> +to a different linked page list `reserved_page_list`.

You already have the flag ``PGC_reserved`` to indicate whether the 
memory is reserved or not. So why do you also need to link list it?

> +
> +Later, when domain get destroyed and memory relinquished, only pages in
> +`page_list` go back to heap, and pages in `reserved_page_list` shall not.

While going through the series, I could not find any code implementing 
this. However, this is not enough to prevent a page to go to the heap 
allocator because a domain can release memory at runtime using 
hypercalls like XENMEM_remove_from_physmap.

One of the use case is when the guest decides to balloon out some 
memory. This will call free_domheap_pages().

Effectively, you are treating static memory as domheap pages. So I think 
it would be better if you hook in free_domheap_pages() to decide which 
allocator is used.

Now, if a guest can balloon out memory, it can also balloon in memory. 
There are two cases:
    1) The region used to be RAM region statically allocated
    2) The region used to be unallocated.

I think for 1), we need to be able to re-use the page previously. For 
2), it is not clear to me whether a guest with memory statically 
allocated should be allowed to allocate "dynamic" pages.

> +### Memory Allocation for Domains on Static Allocation
> +
> +RAM regions pre-defined as static memory for one specifc domain shall be parsed
> +and reserved from the beginning. And they shall never go to any memory
> +allocator for any use.

Technically, you are introducing a new allocator. So do you mean they 
should not be given to neither the buddy allocator nor the bot allocator?

> +
> +Later when allocating static memory for this specific domain, after acquiring
> +those reserved regions, users need to a do set of verification before
> +assigning.
> +For each page there, it at least includes the following steps:
> +1. Check if it is in free state and has zero reference count.
> +2. Check if the page is reserved(`PGC_reserved`).
> +
> +Then, assigning these pages to this specific domain, and all pages go to one
> +new linked page list `reserved_page_list`.
> +
> +At last, set up guest P2M mapping. By default, it shall be mapped to the fixed
> +guest RAM address `GUEST_RAM0_BASE`, `GUEST_RAM1_BASE`, just like normal
> +domains. But later in 1:1 direct-map design, if `direct-map` is set, the guest
> +physical address will equal to physical address.
> +
> +### Static Allocation for Xen itself
> +
> +### New Deivce Tree Node: `xen,reserved_heap`

s/Deivce/Device/

> +
> +Static memory for Xen heap refers to parts of RAM reserved in the beginning
> +for Xen heap only. The memory is pre-defined through XEN configuration
> +using physical address ranges.
> +
> +The reserved memory for Xen heap is an optional feature and can be enabled
> +by adding a device tree property in the `chosen` node. Currently, this feature
> +is only supported on AArch64.
> +
> +Here is one example:
> +
> +
> +        chosen {
> +            xen,reserved-heap = <0x0 0x30000000 0x0 0x40000000>;
> +            ...
> +        };
> +
> +RAM at 0x30000000 of 1G size will be reserved as heap memory. Later, heap
> +allocator will allocate memory only from this specific region.

This section is quite confusing. I think we need to clearly 
differentiate heap vs allocator.

In Xen we have two heaps:
    1) Xen heap: It is always mapped in Xen virtual address space. This 
is mainly used for xen internal allocation.
    2) Domain heap: It may not always be mapped in Xen virtual address 
space. This is mainly used for domain memory and mapped on-demand.

For Arm64 (and x86), two heaps are allocated from the same region. But 
on Arm32, they are different.

We also have two allocator:
    1) Boot allocator: This is used during boot only. There is no 
concept of heap at this time.
    2) Buddy allocator: This is the current runtime allocator. This can 
either allocator from either heap.

AFAICT, this design is introducing a 3rd allocator that will return 
domain heap pages.

Now, back to this section, are you saying you will separate the two 
heaps and force the buddy allocator to allocate xen heap pages from a 
specific region?

[...]

> +### Memory Allocation for 1:1 direct-map Domain
> +
> +Implementing memory allocation for 1:1 direct-map domain includes two parts:
> +Static Allocation for Domain and 1:1 direct-map.
> +
> +The first part has been elaborated before in chapter `Memory Allocation for
> +Domains on Static Allocation`. Then, to ensure 1:1 direct-map, when setting up
> +guest P2M mapping, it needs to make sure that guest physical address equal to
> +physical address(`gfn == mfn`).
> +
> +*DISCUSSION:
> +
> +  * Here only supports booting up one domain on static allocation or on 1:1
> +direct-map through device tree, is `xl` also needed?

I think they can be separated for now.

Cheers,

-- 
Julien Grall

RE: [PATCH] Xen: Design doc for 1:1 direct-map and static allocation

[Penny Zheng]

Hi Julien

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: Tuesday, May 18, 2021 7:48 PM
> To: Penny Zheng <Penny.Zheng@arm.com>; xen-devel@lists.xenproject.org;
> sstabellini@kernel.org
> Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>; Wei Chen
> <Wei.Chen@arm.com>; nd <nd@arm.com>
> Subject: Re: [PATCH] Xen: Design doc for 1:1 direct-map and static allocation
> 
> Hi Penny,
> 
> On 18/05/2021 06:07, Penny Zheng wrote:
> > Create one design doc for 1:1 direct-map and static allocation.
> > It is the first draft and aims to describe why and how we allocate
> > 1:1 direct-map(guest physical == physical) domains, and why and how we
> > let domains on static allocation.
> >
> > Signed-off-by: Penny Zheng <penny.zheng@arm.com>
> > ---
> >   docs/designs/static_alloc_and_direct_map.md | 239
> ++++++++++++++++++++
> >   1 file changed, 239 insertions(+)
> >   create mode 100644 docs/designs/static_alloc_and_direct_map.md
> >
> > diff --git a/docs/designs/static_alloc_and_direct_map.md
> > b/docs/designs/static_alloc_and_direct_map.md
> > new file mode 100644
> > index 0000000000..fdda162188
> > --- /dev/null
> > +++ b/docs/designs/static_alloc_and_direct_map.md
> > @@ -0,0 +1,239 @@
> > +# Preface
> > +
> > +The document is an early draft for 1:1 direct-map memory map (`guest
> > +physical == physical`) of domUs and Static Allocation.
> > +Since the implementation of these two features shares a lot, we would
> > +like to introduce both in one design.
> > +
> > +Right now, these two features are limited to ARM architecture.
> > +
> > +This design aims to describe why and how the guest would be created
> > +as 1:1 direct-map domain, and why and what the static allocation is.
> > +
> > +This document is partly based on Stefano Stabellini's patch serie v1:
> > +[direct-map DomUs](
> > +https://lists.xenproject.org/archives/html/xen-devel/2020-
> 04/msg00707.html).
> 
> While for the reviewer this is a useful information to have, I am not sure a
> future reader needs to know all the history. So I would move this to the
> commit message.
> 
> > +
> > +This is a first draft and some questions are still unanswered. When
> > +this is the case, it will be included under chapter `DISCUSSION`.
> > +
> > +# Introduction on Static Allocation
> > +
> > +Static allocation refers to system or sub-system(domains) for which
> > +memory areas are pre-defined by configuration using physical address
> ranges.
> > +
> > +## Background
> > +
> > +Cases where needs static allocation:
> > +
> > +  * Static allocation needed whenever a system has a pre-defined
> > +non-changing behaviour. This is usually the case in safety world
> > +where system must behave the same upon reboot, so memory resource
> for
> > +both XEN and domains should be static and pre-defined.
> > +
> > +  * Static allocation needed whenever a guest wants to allocate
> > +memory from refined memory ranges. For example, a system has one
> > +high-speed RAM region, and would like to assign it to one specific domain.
> > +
> > +  * Static allocation needed whenever a system needs a guest
> > +restricted to some known memory area due to hardware limitations
> > +reason. For example, some device can only do DMA to a specific part of
> the memory.
> > +
> > +Limitations:
> > +  * There is no consideration for PV devices at the moment.
> > +
> > +## Design on Static Allocation
> > +
> > +Static allocation refers to system or sub-system(domains) for which
> > +memory areas are pre-defined by configuration using physical address
> ranges.
> > +
> > +These pre-defined memory, -- Static Momery, as parts of RAM reserved
> > +in the
> 
> s/Momery/Memory/
> 

Thx.

> > +beginning, shall never go to heap allocator or boot allocator for any use.
> 
> I think you mean "buddy" rather than "heap". Looking at your code, you are
> treating static memory region as domheap pages.
> 
> > +
> > +### Static Allocation for Domains
> > +
> > +### New Deivce Tree Node: `xen,static_mem`
> 
> S/Deivce/
> 

Thx.

> > +
> > +Here introduces new `xen,static_mem` node to define static memory
> > +nodes for one specific domain.
> > +
> > +For domains on static allocation, users need to pre-define guest RAM
> > +regions in configuration, through `xen,static_mem` node under approriate
> `domUx` node.
> > +
> > +Here is one example:
> > +
> > +
> > +        domU1 {
> > +            compatible = "xen,domain";
> > +            #address-cells = <0x2>;
> > +            #size-cells = <0x2>;
> > +            cpus = <2>;
> > +            xen,static-mem = <0x0 0xa0000000 0x0 0x20000000>;
> > +            ...
> > +        };
> > +
> > +RAM at 0xa0000000 of 512 MB are static memory reserved for domU1 as
> its RAM.
> > +
> > +### New Page Flag: `PGC_reserved`
> > +
> > +In order to differentiate and manage pages reserved as static memory
> > +with those which are allocated from heap allocator for normal
> > +domains, we shall introduce a new page flag `PGC_reserved` to tell.
> > +
> > +Grant pages `PGC_reserved` when initializing static memory.
> > +
> > +### New linked page list: `reserved_page_list` in  `struct domain`
> > +
> > +Right now, for normal domains, on assigning pages to domain, pages
> > +allocated from heap allocator as guest RAM shall be inserted to one
> > +linked page list `page_list` for later managing and storing.
> > +
> > +So in order to tell, pages allocated from static memory, shall be
> > +inserted to a different linked page list `reserved_page_list`.
> 
> You already have the flag ``PGC_reserved`` to indicate whether the memory
> is reserved or not. So why do you also need to link list it?
> 

Yes, I introduce this link list to try to fix issues of rebooting domain on static allocation.

And after taking your suggestion on getting static memory regions info again from
device tree configuration, I think this link list is not necessary. And I will delete here
and in codes too.

> > +
> > +Later, when domain get destroyed and memory relinquished, only pages
> > +in `page_list` go back to heap, and pages in `reserved_page_list` shall not.
> 
> While going through the series, I could not find any code implementing this.
> However, this is not enough to prevent a page to go to the heap allocator
> because a domain can release memory at runtime using hypercalls like
> XENMEM_remove_from_physmap.
> 
> One of the use case is when the guest decides to balloon out some memory.
> This will call free_domheap_pages().
> 
> Effectively, you are treating static memory as domheap pages. So I think it
> would be better if you hook in free_domheap_pages() to decide which
> allocator is used.
> 
> Now, if a guest can balloon out memory, it can also balloon in memory.
> There are two cases:
>     1) The region used to be RAM region statically allocated
>     2) The region used to be unallocated.
> 
> I think for 1), we need to be able to re-use the page previously. For 2), it is
> not clear to me whether a guest with memory statically allocated should be
> allowed to allocate "dynamic" pages.
> 

Yeah, I share the same with you of hooking in free_domheap_pages(). I'm thinking
that if pages of PGC_reserved, we may create a new func free_staticmem_pages to
free them.

For issues on ballooning out or in, it is not supported here.
Domain on Static Allocation and 1:1 direct-map are all based on dom0-less right
now, so no PV, grant table, event channel, etc, considered.

Right now, it only supports device got passthrough into the guest.

> > +### Memory Allocation for Domains on Static Allocation
> > +
> > +RAM regions pre-defined as static memory for one specifc domain shall
> > +be parsed and reserved from the beginning. And they shall never go to
> > +any memory allocator for any use.
> 
> Technically, you are introducing a new allocator. So do you mean they should
> not be given to neither the buddy allocator nor the bot allocator?
> 

Yes. These pre-defined RAM regions will not be given to any current
memory allocator. If be given there, there is no guarantee that it will
not be allocated for other use.

And right now, in my current design, these pre-defined RAM regions are either for
one specific domain as guest RAM or as XEN heap.
  
> > +
> > +Later when allocating static memory for this specific domain, after
> > +acquiring those reserved regions, users need to a do set of
> > +verification before assigning.
> > +For each page there, it at least includes the following steps:
> > +1. Check if it is in free state and has zero reference count.
> > +2. Check if the page is reserved(`PGC_reserved`).
> > +
> > +Then, assigning these pages to this specific domain, and all pages go
> > +to one new linked page list `reserved_page_list`.
> > +
> > +At last, set up guest P2M mapping. By default, it shall be mapped to
> > +the fixed guest RAM address `GUEST_RAM0_BASE`, `GUEST_RAM1_BASE`,
> > +just like normal domains. But later in 1:1 direct-map design, if
> > +`direct-map` is set, the guest physical address will equal to physical
> address.
> > +
> > +### Static Allocation for Xen itself
> > +
> > +### New Deivce Tree Node: `xen,reserved_heap`
> 
> s/Deivce/Device/
> 

Thx.

> > +
> > +Static memory for Xen heap refers to parts of RAM reserved in the
> > +beginning for Xen heap only. The memory is pre-defined through XEN
> > +configuration using physical address ranges.
> > +
> > +The reserved memory for Xen heap is an optional feature and can be
> > +enabled by adding a device tree property in the `chosen` node.
> > +Currently, this feature is only supported on AArch64.
> > +
> > +Here is one example:
> > +
> > +
> > +        chosen {
> > +            xen,reserved-heap = <0x0 0x30000000 0x0 0x40000000>;
> > +            ...
> > +        };
> > +
> > +RAM at 0x30000000 of 1G size will be reserved as heap memory. Later,
> > +heap allocator will allocate memory only from this specific region.
> 
> This section is quite confusing. I think we need to clearly differentiate heap vs
> allocator.
> 
> In Xen we have two heaps:
>     1) Xen heap: It is always mapped in Xen virtual address space. This is
> mainly used for xen internal allocation.
>     2) Domain heap: It may not always be mapped in Xen virtual address space.
> This is mainly used for domain memory and mapped on-demand.
> 
> For Arm64 (and x86), two heaps are allocated from the same region. But on
> Arm32, they are different.
> 
> We also have two allocator:
>     1) Boot allocator: This is used during boot only. There is no concept of
> heap at this time.
>     2) Buddy allocator: This is the current runtime allocator. This can either
> allocator from either heap.
> 
> AFAICT, this design is introducing a 3rd allocator that will return domain heap
> pages.
> 
> Now, back to this section, are you saying you will separate the two heaps and
> force the buddy allocator to allocate xen heap pages from a specific region?
> 
> [...]

I will try to explain clearly here. 
The intention behind this reserved heap is that for supporting total static system, we
not only want to pre-define memory resource for guests, but also for xen runtime
allocation. Any runtime behavior are more predictable.

Right now, on AArch64, all RAM, except reserved memory, will be given to buddy
allocator as heap,  like you said, guest RAM for normal domain will be allocated
from there, xmalloc eventually is get memory from there, etc. So we want to refine
the heap here, not iterating through bootinfo.mem to set up XEN heap, but like
iterating bootinfo. reserved_heap to set up XEN heap.

True, on ARM32, xen heap and domain heap are separately mapped, which is more
complicated here. That's why I only talking about implementing these features on
AArch64 as first step.

> 
> > +### Memory Allocation for 1:1 direct-map Domain
> > +
> > +Implementing memory allocation for 1:1 direct-map domain includes two
> parts:
> > +Static Allocation for Domain and 1:1 direct-map.
> > +
> > +The first part has been elaborated before in chapter `Memory
> > +Allocation for Domains on Static Allocation`. Then, to ensure 1:1
> > +direct-map, when setting up guest P2M mapping, it needs to make sure
> > +that guest physical address equal to physical address(`gfn == mfn`).
> > +
> > +*DISCUSSION:
> > +
> > +  * Here only supports booting up one domain on static allocation or
> > +on 1:1 direct-map through device tree, is `xl` also needed?
> 
> I think they can be separated for now.
> 

Agree~~

> Cheers,
> 
> --
> Julien Grall

Re: [PATCH] Xen: Design doc for 1:1 direct-map and static allocation

[Julien Grall]

On 20/05/2021 06:36, Penny Zheng wrote:
> Hi Julien

Hi Penny,

>>> +
>>> +Later, when domain get destroyed and memory relinquished, only pages
>>> +in `page_list` go back to heap, and pages in `reserved_page_list` shall not.
>>
>> While going through the series, I could not find any code implementing this.
>> However, this is not enough to prevent a page to go to the heap allocator
>> because a domain can release memory at runtime using hypercalls like
>> XENMEM_remove_from_physmap.
>>
>> One of the use case is when the guest decides to balloon out some memory.
>> This will call free_domheap_pages().
>>
>> Effectively, you are treating static memory as domheap pages. So I think it
>> would be better if you hook in free_domheap_pages() to decide which
>> allocator is used.
>>
>> Now, if a guest can balloon out memory, it can also balloon in memory.
>> There are two cases:
>>      1) The region used to be RAM region statically allocated
>>      2) The region used to be unallocated.
>>
>> I think for 1), we need to be able to re-use the page previously. For 2), it is
>> not clear to me whether a guest with memory statically allocated should be
>> allowed to allocate "dynamic" pages.
>>
> 
> Yeah, I share the same with you of hooking in free_domheap_pages(). I'm thinking
> that if pages of PGC_reserved, we may create a new func free_staticmem_pages to
> free them.
> 
> For issues on ballooning out or in, it is not supported here.

It is fine that the implementation doesn't yet implement it. However, I 
think the design document should take into account ballooning. This is 
because even if...

> Domain on Static Allocation and 1:1 direct-map are all based on dom0-less right
> now, so no PV, grant table, event channel, etc, considered.

... there is no PV support & co, a guest is still able to issue 
hypercalls (they are not hidden). Therefore your guest will be able to 
disturb your static allocation.

> 
> Right now, it only supports device got passthrough into the guest.
> 
>>> +### Memory Allocation for Domains on Static Allocation
>>> +
>>> +RAM regions pre-defined as static memory for one specifc domain shall
>>> +be parsed and reserved from the beginning. And they shall never go to
>>> +any memory allocator for any use.
>>
>> Technically, you are introducing a new allocator. So do you mean they should
>> not be given to neither the buddy allocator nor the bot allocator?
>>
> 
> Yes. These pre-defined RAM regions will not be given to any current
> memory allocator. If be given there, there is no guarantee that it will
> not be allocated for other use.
> 
> And right now, in my current design, these pre-defined RAM regions are either for
> one specific domain as guest RAM or as XEN heap.
>    
>>> +
>>> +Later when allocating static memory for this specific domain, after
>>> +acquiring those reserved regions, users need to a do set of
>>> +verification before assigning.
>>> +For each page there, it at least includes the following steps:
>>> +1. Check if it is in free state and has zero reference count.
>>> +2. Check if the page is reserved(`PGC_reserved`).
>>> +
>>> +Then, assigning these pages to this specific domain, and all pages go
>>> +to one new linked page list `reserved_page_list`.
>>> +
>>> +At last, set up guest P2M mapping. By default, it shall be mapped to
>>> +the fixed guest RAM address `GUEST_RAM0_BASE`, `GUEST_RAM1_BASE`,
>>> +just like normal domains. But later in 1:1 direct-map design, if
>>> +`direct-map` is set, the guest physical address will equal to physical
>> address.
>>> +
>>> +### Static Allocation for Xen itself
>>> +
>>> +### New Deivce Tree Node: `xen,reserved_heap`
>>
>> s/Deivce/Device/
>>
> 
> Thx.
> 
>>> +
>>> +Static memory for Xen heap refers to parts of RAM reserved in the
>>> +beginning for Xen heap only. The memory is pre-defined through XEN
>>> +configuration using physical address ranges.
>>> +
>>> +The reserved memory for Xen heap is an optional feature and can be
>>> +enabled by adding a device tree property in the `chosen` node.
>>> +Currently, this feature is only supported on AArch64.
>>> +
>>> +Here is one example:
>>> +
>>> +
>>> +        chosen {
>>> +            xen,reserved-heap = <0x0 0x30000000 0x0 0x40000000>;
>>> +            ...
>>> +        };
>>> +
>>> +RAM at 0x30000000 of 1G size will be reserved as heap memory. Later,
>>> +heap allocator will allocate memory only from this specific region.
>>
>> This section is quite confusing. I think we need to clearly differentiate heap vs
>> allocator.
>>
>> In Xen we have two heaps:
>>      1) Xen heap: It is always mapped in Xen virtual address space. This is
>> mainly used for xen internal allocation.
>>      2) Domain heap: It may not always be mapped in Xen virtual address space.
>> This is mainly used for domain memory and mapped on-demand.
>>
>> For Arm64 (and x86), two heaps are allocated from the same region. But on
>> Arm32, they are different.
>>
>> We also have two allocator:
>>      1) Boot allocator: This is used during boot only. There is no concept of
>> heap at this time.
>>      2) Buddy allocator: This is the current runtime allocator. This can either
>> allocator from either heap.
>>
>> AFAICT, this design is introducing a 3rd allocator that will return domain heap
>> pages.
>>
>> Now, back to this section, are you saying you will separate the two heaps and
>> force the buddy allocator to allocate xen heap pages from a specific region?
>>
>> [...]
> 
> I will try to explain clearly here.
> The intention behind this reserved heap is that for supporting total static system, we
> not only want to pre-define memory resource for guests, but also for xen runtime
> allocation. Any runtime behavior are more predictable.
> 
> Right now, on AArch64, all RAM, except reserved memory, will be given to buddy
> allocator as heap,  like you said, guest RAM for normal domain will be allocated
> from there, xmalloc eventually is get memory from there, etc. So we want to refine
> the heap here, not iterating through bootinfo.mem to set up XEN heap, but like
> iterating bootinfo. reserved_heap to set up XEN heap.

So effectively, you want to move to a split heap like on Arm32. Is that 
correct?

But let's take a step back from the actual code (this is implementation 
details). If the Device-Tree describes all the regions statically 
allocated to domains, why can't the memory used by Xen heap be the left 
over?

> 
> True, on ARM32, xen heap and domain heap are separately mapped, which is more
> complicated here. That's why I only talking about implementing these features on
> AArch64 as first step.

Cheers,

-- 
Julien Grall