[PATCH] design: design doc for shared memory on a dom0less system

[PATCH] design: design doc for shared memory on a dom0less system

[Penny Zheng]

This commit provides a design doc for static shared memory
on a dom0less system.

Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
 design/shm-dom0less.md | 182 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 182 insertions(+)
 create mode 100644 design/shm-dom0less.md

diff --git a/design/shm-dom0less.md b/design/shm-dom0less.md
new file mode 100644
index 0000000..b46199d
--- /dev/null
+++ b/design/shm-dom0less.md
@@ -0,0 +1,182 @@
+# Static Shared Memory between domains on a dom0less system
+
+This design aims to provide an overview of the new feature: setting up static
+shared memory between domains on a dom0less system, through device tree
+configuration.
+
+The new feature is driven by the need of finding a way to build up
+communication channels on dom0less system, since the legacy ways including
+grant table, etc are all absent there.
+
+It was inspired by the patch serie of "xl/libxl-based shared memory", see
+[1] for more details.
+
+# Static Shared Memory Device Tree Configuration
+
+The static shared memory device tree nodes allow users to statically set up
+shared memory among a group of dom0less DomUs and Dom0, enabling domains
+to do shm-based communication.
+
+- compatible
+
+    "xen,domain-shared-memory-v1"
+
+- xen,shm-id
+
+    An u32 value represents the unique identifier of the shared memory region.
+    User valuing per shared memory region shall follow the ascending order,
+    starting from xen,shm-id = <0x0>, to the maximum identifier
+    xen,shm-id = <0x126>. The special xen,shm-id = <0x127> is reserved for
+    INVALID_SHMID.
+
+- xen,shared-mem
+
+    An array takes a physical address, which is the base address of the
+    shared memory region in host physical address space, a size, and a guest
+    physical address, as the target address of the mapping.
+
+- role(Optional)
+
+    A string property specifying the ownership of a shared memory region,
+    the value must be one of the following: "owner", or "borrower"
+    A shared memory region could be explicitly backed by one domain, which is
+    called "owner domain", and all the other domains who are also sharing
+    this region are called "borrower domain".
+    If not specified, the default value is "borrower" and owner is
+    "dom_shared", a system domain.
+
+## Example
+
+chosen {
+    #address-cells = <0x1>;
+    #size-cells = <0x1>;
+    xen,xen-bootargs = "console=dtuart dtuart=serial0 bootscrub=0";
+
+    ......
+
+    /* this is for Dom0 */
+    dom0-shared-mem@10000000 {
+        compatible = "xen,domain-shared-memory-v1";
+        xen,shm-id = <0x0>;
+        role = "owner";
+        xen,shared-mem = <0x10000000 0x10000000 0x10000000>;
+    }
+
+    domU1 {
+        compatible = "xen,domain";
+        #address-cells = <0x1>;
+        #size-cells = <0x1>;
+        memory = <0 131072>;
+        cpus = <2>;
+        vpl011;
+
+        /*
+         * shared memory region identified as 0x0(xen,shm-id = <0x0>)
+         * shared between dom0.
+         */
+        domU1-shared-mem@10000000 {
+            compatible = "xen,domain-shared-memory-v1";
+            xen,shm-id = <0x0>;
+            role = "borrower";
+            xen,shared-mem = <0x10000000 0x10000000 0x50000000>;
+        }
+
+        domU1-shared-mem@50000000 {
+            compatible = "xen,domain-shared-memory-v1";
+            xen,shm-id = <0x1>;
+            xen,shared-mem = <0x50000000 0x20000000 0x60000000>;
+        }
+
+        ......
+
+    };
+
+    domU2 {
+        compatible = "xen,domain";
+        #address-cells = <0x1>;
+        #size-cells = <0x1>;
+        memory = <0 65536>;
+        cpus = <1>;
+
+        /*
+         * shared memory region identified as 0x1(xen,shm-id = <0x1>)
+         * shared between domU1.
+         */
+        domU2-shared-mem@50000000 {
+            compatible = "xen,domain-shared-memory-v1";
+            xen,shm-id = <0x1>;
+            xen,shared-mem = <0x50000000 0x20000000 0x70000000>;
+        }
+
+        ......
+    };
+};
+
+It is the example of two static shared memory regions.
+
+In terms of shared memory region identified as 0x0, host physical address
+starting at 0x10000000 of 256MB will be reserved to be shared between Dom0
+and DomU1. It will get mapped at 0x10000000 in Dom0 guest physical address
+space, and at 0x50000000 in DomU1 guest physical address space. Dom0 is the
+owner domain, and domU1 is the borrower domain.
+
+And in terms of shared memory region identified as 0x1, host physical address
+starting at 0x50000000 of 512MB will be reserved to be shared between DomU1
+and DomU2. It will get mapped at 0x60000000 in DomU1 guest physical address
+space, and at 0x70000000 in DomU2 guest physical address space. Since no owner
+domain is explicitly defined, the default "dom_shared" is the owner domain,
+and both domU1 and domU2 are the borrower domains.
+
+# Overview of Static Shared Memory Flow
+
+Static Shared Memory working flow could be classified into the following
+steps:
+ - Carve out a range of memory in host physical address space to be used
+for sharing. Define it in device tree configuration, then parse and reserve
+it to avoid other use.
+ - Create a special domain "dom_shared". It will be the owner domain which
+is owning the statically shared pages, if "role" property is not specified.
+ - Per shared memory region could be shared with multiple domains. For
+owner domain, it acquires statically shared pages and assign them to itself,
+in the same way with static memory. And other than owner domain, the others
+who are also sharing are called "borrower domain", for which foreign memory
+map of statically shared pages is required.
+ - Expose the shared memory to the domU using the "xen,shared-memory-v1"
+reserved-memory binding. See
+Documentation/devicetree/bindings/reserved-memory/xen,shared-memory.txt
+in Linux for the corresponding device tree binding.
+
+# Memory management of Shared Memory Region
+
+Each memory page needs to have an "owner" and it is likely that in many cases
+the user don't care who the owner is, so it makes sense that users don't
+need to specify the "role" in device tree if they don't want to, in which
+scenario, a default domain shall be the owner domain.
+
+We propose a new system domain "dom_shared" to be the default domain owning all
+statically unowned shared pages, assigning it dom_id 0x7FF5(DOMID_SHARED).
+
+"dom_shared" domain shall get constructed before domain construction and after
+"setup_virt_paging", during system boot-time, so it could successfully do
+p2m initialization.
+
+Owner domain acquires statically shared pages and assign them to itself,
+while borrower domains get and take reference of them, then do foreign memory
+map of these statically shared pages.
+
+When destroying or rebooting a domain, if it is a borrower domain, other than
+removing foreign memory map of statically shared pages in P2M table, we also
+need to drop according gained reference. And if it is an owner domain, since
+statically shared pages are allocated as guest normal ram, it is not needed to
+do extra removing.
+
+However if owner domain is not the default "dom_shared" domain, but specified
+explicitly in device tree, stopping itself will make shared memory region
+unaccessible to all borrower domains, so we need to remove foreign memory map
+for all borrower domains. Notice that all borrowers domains should be stopped
+before stopping the owner domain.
+
+"dom_shared" domain is destroyed when the whole system shuts down, so its
+owning statically shared pages are only freed at system shutdown.
+
+[1] https://marc.info/?l=xen-devel&m=154404821731186
-- 
2.25.1

Re: [PATCH] design: design doc for shared memory on a dom0less system

[Julien Grall]

Hi,

On 26/01/2022 10:09, Penny Zheng wrote:
> This commit provides a design doc for static shared memory
> on a dom0less system.
> 
> Signed-off-by: Penny Zheng <penny.zheng@arm.com>
> ---
>   design/shm-dom0less.md | 182 +++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 182 insertions(+)
>   create mode 100644 design/shm-dom0less.md
> 
> diff --git a/design/shm-dom0less.md b/design/shm-dom0less.md
> new file mode 100644
> index 0000000..b46199d
> --- /dev/null
> +++ b/design/shm-dom0less.md
> @@ -0,0 +1,182 @@
> +# Static Shared Memory between domains on a dom0less system
> +
> +This design aims to provide an overview of the new feature: setting up static
> +shared memory between domains on a dom0less system, through device tree
> +configuration.
> +
> +The new feature is driven by the need of finding a way to build up
> +communication channels on dom0less system, since the legacy ways including
> +grant table, etc are all absent there.

Stefano has a series to add support for grant-table [2]. So I think you 
want to justify it differently.

> +
> +It was inspired by the patch serie of "xl/libxl-based shared memory", see
> +[1] for more details.
> +
> +# Static Shared Memory Device Tree Configuration
> +
> +The static shared memory device tree nodes allow users to statically set up
> +shared memory among a group of dom0less DomUs and Dom0, enabling domains
> +to do shm-based communication.
> +
> +- compatible
> +
> +    "xen,domain-shared-memory-v1"
> +
> +- xen,shm-id

 From the document, it is not clear to me what is the purpose of the 
identifier. Could you clarify it?

> +
> +    An u32 value represents the unique identifier of the shared memory region.
> +    User valuing per shared memory region shall follow the ascending order,
> +    starting from xen,shm-id = <0x0>, to the maximum identifier
> +    xen,shm-id = <0x126>.

Why is it limit to 0x126? And also, why do they have to be allocated in 
ascending order?

> The special xen,shm-id = <0x127> is reserved for
> +    INVALID_SHMID.

Why do we need to reserve invalid?

> +
> +- xen,shared-mem
> +
> +    An array takes a physical address, which is the base address of the
> +    shared memory region in host physical address space, a size, and a guest
> +    physical address, as the target address of the mapping.

I think shared memory is useful without static allocation. So I think we 
want to make the host physical address optional.

> +
> +- role(Optional)
> +
> +    A string property specifying the ownership of a shared memory region,
> +    the value must be one of the following: "owner", or "borrower"
> +    A shared memory region could be explicitly backed by one domain, which is
> +    called "owner domain", and all the other domains who are also sharing
> +    this region are called "borrower domain".
> +    If not specified, the default value is "borrower" and owner is
> +    "dom_shared", a system domain.

I don't particularly like adding another system domain. Instead, it 
would be better to always specify the owner.

> +
> +## Example
> +
> +chosen {
> +    #address-cells = <0x1>;
> +    #size-cells = <0x1>;
> +    xen,xen-bootargs = "console=dtuart dtuart=serial0 bootscrub=0";
> +
> +    ......
> +
> +    /* this is for Dom0 */
> +    dom0-shared-mem@10000000 {
> +        compatible = "xen,domain-shared-memory-v1";
> +        xen,shm-id = <0x0>;
> +        role = "owner";
> +        xen,shared-mem = <0x10000000 0x10000000 0x10000000>;
> +    }
> +
> +    domU1 {
> +        compatible = "xen,domain";
> +        #address-cells = <0x1>;
> +        #size-cells = <0x1>;
> +        memory = <0 131072>;
> +        cpus = <2>;
> +        vpl011;
> +
> +        /*
> +         * shared memory region identified as 0x0(xen,shm-id = <0x0>)
> +         * shared between dom0.
> +         */
> +        domU1-shared-mem@10000000 {
> +            compatible = "xen,domain-shared-memory-v1";
> +            xen,shm-id = <0x0>;
> +            role = "borrower";
> +            xen,shared-mem = <0x10000000 0x10000000 0x50000000>;

Technically, you already know the physical address from the owner. In 
fact, it will only increase the risk to get the wrong binding. So I 
would like to suggest a different binding.

1) Reserve the region in the host memory using reserved-memory binding
2) Create a binding per domain that contains a phandle to the host 
memory and the role.

The advantage with this is we could easily support region that are not 
backed by a reserved-memory.


> +        }
> +
> +        domU1-shared-mem@50000000 {
> +            compatible = "xen,domain-shared-memory-v1";
> +            xen,shm-id = <0x1>;
> +            xen,shared-mem = <0x50000000 0x20000000 0x60000000>;
> +        }
> +
> +        ......
> +
> +    };
> +
> +    domU2 {
> +        compatible = "xen,domain";
> +        #address-cells = <0x1>;
> +        #size-cells = <0x1>;
> +        memory = <0 65536>;
> +        cpus = <1>;
> +
> +        /*
> +         * shared memory region identified as 0x1(xen,shm-id = <0x1>)
> +         * shared between domU1.
> +         */
> +        domU2-shared-mem@50000000 {
> +            compatible = "xen,domain-shared-memory-v1";
> +            xen,shm-id = <0x1>;
> +            xen,shared-mem = <0x50000000 0x20000000 0x70000000>;
> +        }
> +
> +        ......
> +    };
> +};
> +
> +It is the example of two static shared memory regions.
> +
> +In terms of shared memory region identified as 0x0, host physical address
> +starting at 0x10000000 of 256MB will be reserved to be shared between Dom0
> +and DomU1. It will get mapped at 0x10000000 in Dom0 guest physical address
> +space, and at 0x50000000 in DomU1 guest physical address space. Dom0 is the
> +owner domain, and domU1 is the borrower domain.
> +
> +And in terms of shared memory region identified as 0x1, host physical address
> +starting at 0x50000000 of 512MB will be reserved to be shared between DomU1
> +and DomU2. It will get mapped at 0x60000000 in DomU1 guest physical address
> +space, and at 0x70000000 in DomU2 guest physical address space. Since no owner
> +domain is explicitly defined, the default "dom_shared" is the owner domain,
> +and both domU1 and domU2 are the borrower domains.
> +
> +# Overview of Static Shared Memory Flow
> +
> +Static Shared Memory working flow could be classified into the following
> +steps:
> + - Carve out a range of memory in host physical address space to be used
> +for sharing. Define it in device tree configuration, then parse and reserve
> +it to avoid other use.
> + - Create a special domain "dom_shared". It will be the owner domain which
> +is owning the statically shared pages, if "role" property is not specified.
> + - Per shared memory region could be shared with multiple domains. For
> +owner domain, it acquires statically shared pages and assign them to itself,
> +in the same way with static memory. And other than owner domain, the others
> +who are also sharing are called "borrower domain", for which foreign memory
> +map of statically shared pages is required.
> + - Expose the shared memory to the domU using the "xen,shared-memory-v1"
> +reserved-memory binding. See
> +Documentation/devicetree/bindings/reserved-memory/xen,shared-memory.txt
> +in Linux for the corresponding device tree binding.
> +
> +# Memory management of Shared Memory Region
> +
> +Each memory page needs to have an "owner" and it is likely that in many cases
> +the user don't care who the owner is, so it makes sense that users don't
> +need to specify the "role" in device tree if they don't want to, in which
> +scenario, a default domain shall be the owner domain.
> +
> +We propose a new system domain "dom_shared" to be the default domain owning all
> +statically unowned shared pages, assigning it dom_id 0x7FF5(DOMID_SHARED).
> +
> +"dom_shared" domain shall get constructed before domain construction and after
> +"setup_virt_paging", during system boot-time, so it could successfully do
> +p2m initialization.

IHMO, this is going too much into details for a design document. The 
goal is to abstract the feature rather than mentioning the 
implementation (which may change during review or in the future).

> +
> +Owner domain acquires statically shared pages and assign them to itself,
> +while borrower domains get and take reference of them, then do foreign memory
> +map of these statically shared pages.

What happens if the borrower is seen before the owner?

> +
> +When destroying or rebooting a domain, if it is a borrower domain, other than
> +removing foreign memory map of statically shared pages in P2M table, we also
> +need to drop according gained reference. And if it is an owner domain, since
> +statically shared pages are allocated as guest normal ram, it is not needed to
> +do extra removing.
> +
> +However if owner domain is not the default "dom_shared" domain, but specified
> +explicitly in device tree, stopping itself will make shared memory region
> +unaccessible to all borrower domains, so we need to remove foreign memory map
> +for all borrower domains. Notice that all borrowers domains should be stopped
> +before stopping the owner domain.

How will you enforce that?

> +
> +"dom_shared" domain is destroyed when the whole system shuts down, so its
> +owning statically shared pages are only freed at system shutdown.
> +
> +[1] https://marc.info/?l=xen-devel&m=154404821731186

[2] <alpine.DEB.2.22.394.2201121646290.19362@ubuntu-linux-20-04-desktop>

-- 
Julien Grall

Re: [PATCH] design: design doc for shared memory on a dom0less system

[Bertrand Marquis]

Hi Julien,

Thanks a lot for the quick feedback.

> On 26 Jan 2022, at 10:58, Julien Grall <julien@xen.org> wrote:
> 
> Hi,
> 
> On 26/01/2022 10:09, Penny Zheng wrote:
>> This commit provides a design doc for static shared memory
>> on a dom0less system.
>> Signed-off-by: Penny Zheng <penny.zheng@arm.com>
>> ---
>>  design/shm-dom0less.md | 182 +++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 182 insertions(+)
>>  create mode 100644 design/shm-dom0less.md
>> diff --git a/design/shm-dom0less.md b/design/shm-dom0less.md
>> new file mode 100644
>> index 0000000..b46199d
>> --- /dev/null
>> +++ b/design/shm-dom0less.md
>> @@ -0,0 +1,182 @@
>> +# Static Shared Memory between domains on a dom0less system
>> +
>> +This design aims to provide an overview of the new feature: setting up static
>> +shared memory between domains on a dom0less system, through device tree
>> +configuration.
>> +
>> +The new feature is driven by the need of finding a way to build up
>> +communication channels on dom0less system, since the legacy ways including
>> +grant table, etc are all absent there.
> 
> Stefano has a series to add support for grant-table [2]. So I think you want to justify it differently.
> 
>> +
>> +It was inspired by the patch serie of "xl/libxl-based shared memory", see
>> +[1] for more details.
>> +
>> +# Static Shared Memory Device Tree Configuration
>> +
>> +The static shared memory device tree nodes allow users to statically set up
>> +shared memory among a group of dom0less DomUs and Dom0, enabling domains
>> +to do shm-based communication.
>> +
>> +- compatible
>> +
>> +    "xen,domain-shared-memory-v1"
>> +
>> +- xen,shm-id
> 
> From the document, it is not clear to me what is the purpose of the identifier. Could you clarify it?
> 
>> +
>> +    An u32 value represents the unique identifier of the shared memory region.
>> +    User valuing per shared memory region shall follow the ascending order,
>> +    starting from xen,shm-id = <0x0>, to the maximum identifier
>> +    xen,shm-id = <0x126>.
> 
> Why is it limit to 0x126? And also, why do they have to be allocated in ascending order?
> 
>> The special xen,shm-id = <0x127> is reserved for
>> +    INVALID_SHMID.
> 
> Why do we need to reserve invalid?
> 
>> +
>> +- xen,shared-mem
>> +
>> +    An array takes a physical address, which is the base address of the
>> +    shared memory region in host physical address space, a size, and a guest
>> +    physical address, as the target address of the mapping.
> 
> I think shared memory is useful without static allocation. So I think we want to make the host physical address optional.
> 
>> +
>> +- role(Optional)
>> +
>> +    A string property specifying the ownership of a shared memory region,
>> +    the value must be one of the following: "owner", or "borrower"
>> +    A shared memory region could be explicitly backed by one domain, which is
>> +    called "owner domain", and all the other domains who are also sharing
>> +    this region are called "borrower domain".
>> +    If not specified, the default value is "borrower" and owner is
>> +    "dom_shared", a system domain.
> 
> I don't particularly like adding another system domain. Instead, it would be better to always specify the owner.

Having an owner which is not Xen is creating a dependency so restart the owner you would need to restart the borrowers.
To remove this dependency and allow use cases where any domain having access can be restarted without the other side
needing to, having Xen as the owner is required.

Initial discussion started between Penny and Stefano went the way you said and I asked to modify it like this to have something
more looking like a standard shared memory with only users but no “owner”.
Also it fits to some of our use cases.

And it also solve some of the issues you found ...

> 
>> +
>> +## Example
>> +
>> +chosen {
>> +    #address-cells = <0x1>;
>> +    #size-cells = <0x1>;
>> +    xen,xen-bootargs = "console=dtuart dtuart=serial0 bootscrub=0";
>> +
>> +    ......
>> +
>> +    /* this is for Dom0 */
>> +    dom0-shared-mem@10000000 {
>> +        compatible = "xen,domain-shared-memory-v1";
>> +        xen,shm-id = <0x0>;
>> +        role = "owner";
>> +        xen,shared-mem = <0x10000000 0x10000000 0x10000000>;
>> +    }
>> +
>> +    domU1 {
>> +        compatible = "xen,domain";
>> +        #address-cells = <0x1>;
>> +        #size-cells = <0x1>;
>> +        memory = <0 131072>;
>> +        cpus = <2>;
>> +        vpl011;
>> +
>> +        /*
>> +         * shared memory region identified as 0x0(xen,shm-id = <0x0>)
>> +         * shared between dom0.
>> +         */
>> +        domU1-shared-mem@10000000 {
>> +            compatible = "xen,domain-shared-memory-v1";
>> +            xen,shm-id = <0x0>;
>> +            role = "borrower";
>> +            xen,shared-mem = <0x10000000 0x10000000 0x50000000>;
> 
> Technically, you already know the physical address from the owner. In fact, it will only increase the risk to get the wrong binding. So I would like to suggest a different binding.
> 
> 1) Reserve the region in the host memory using reserved-memory binding
> 2) Create a binding per domain that contains a phandle to the host memory and the role.
> 
> The advantage with this is we could easily support region that are not backed by a reserved-memory.
> 
> 
>> +        }
>> +
>> +        domU1-shared-mem@50000000 {
>> +            compatible = "xen,domain-shared-memory-v1";
>> +            xen,shm-id = <0x1>;
>> +            xen,shared-mem = <0x50000000 0x20000000 0x60000000>;
>> +        }
>> +
>> +        ......
>> +
>> +    };
>> +
>> +    domU2 {
>> +        compatible = "xen,domain";
>> +        #address-cells = <0x1>;
>> +        #size-cells = <0x1>;
>> +        memory = <0 65536>;
>> +        cpus = <1>;
>> +
>> +        /*
>> +         * shared memory region identified as 0x1(xen,shm-id = <0x1>)
>> +         * shared between domU1.
>> +         */
>> +        domU2-shared-mem@50000000 {
>> +            compatible = "xen,domain-shared-memory-v1";
>> +            xen,shm-id = <0x1>;
>> +            xen,shared-mem = <0x50000000 0x20000000 0x70000000>;
>> +        }
>> +
>> +        ......
>> +    };
>> +};
>> +
>> +It is the example of two static shared memory regions.
>> +
>> +In terms of shared memory region identified as 0x0, host physical address
>> +starting at 0x10000000 of 256MB will be reserved to be shared between Dom0
>> +and DomU1. It will get mapped at 0x10000000 in Dom0 guest physical address
>> +space, and at 0x50000000 in DomU1 guest physical address space. Dom0 is the
>> +owner domain, and domU1 is the borrower domain.
>> +
>> +And in terms of shared memory region identified as 0x1, host physical address
>> +starting at 0x50000000 of 512MB will be reserved to be shared between DomU1
>> +and DomU2. It will get mapped at 0x60000000 in DomU1 guest physical address
>> +space, and at 0x70000000 in DomU2 guest physical address space. Since no owner
>> +domain is explicitly defined, the default "dom_shared" is the owner domain,
>> +and both domU1 and domU2 are the borrower domains.
>> +
>> +# Overview of Static Shared Memory Flow
>> +
>> +Static Shared Memory working flow could be classified into the following
>> +steps:
>> + - Carve out a range of memory in host physical address space to be used
>> +for sharing. Define it in device tree configuration, then parse and reserve
>> +it to avoid other use.
>> + - Create a special domain "dom_shared". It will be the owner domain which
>> +is owning the statically shared pages, if "role" property is not specified.
>> + - Per shared memory region could be shared with multiple domains. For
>> +owner domain, it acquires statically shared pages and assign them to itself,
>> +in the same way with static memory. And other than owner domain, the others
>> +who are also sharing are called "borrower domain", for which foreign memory
>> +map of statically shared pages is required.
>> + - Expose the shared memory to the domU using the "xen,shared-memory-v1"
>> +reserved-memory binding. See
>> +Documentation/devicetree/bindings/reserved-memory/xen,shared-memory.txt
>> +in Linux for the corresponding device tree binding.
>> +
>> +# Memory management of Shared Memory Region
>> +
>> +Each memory page needs to have an "owner" and it is likely that in many cases
>> +the user don't care who the owner is, so it makes sense that users don't
>> +need to specify the "role" in device tree if they don't want to, in which
>> +scenario, a default domain shall be the owner domain.
>> +
>> +We propose a new system domain "dom_shared" to be the default domain owning all
>> +statically unowned shared pages, assigning it dom_id 0x7FF5(DOMID_SHARED).
>> +
>> +"dom_shared" domain shall get constructed before domain construction and after
>> +"setup_virt_paging", during system boot-time, so it could successfully do
>> +p2m initialization.
> 
> IHMO, this is going too much into details for a design document. The goal is to abstract the feature rather than mentioning the implementation (which may change during review or in the future).
> 
>> +
>> +Owner domain acquires statically shared pages and assign them to itself,
>> +while borrower domains get and take reference of them, then do foreign memory
>> +map of these statically shared pages.
> 
> What happens if the borrower is seen before the owner?

This is not an issue if Xen is the owner.

> 
>> +
>> +When destroying or rebooting a domain, if it is a borrower domain, other than
>> +removing foreign memory map of statically shared pages in P2M table, we also
>> +need to drop according gained reference. And if it is an owner domain, since
>> +statically shared pages are allocated as guest normal ram, it is not needed to
>> +do extra removing.
>> +
>> +However if owner domain is not the default "dom_shared" domain, but specified
>> +explicitly in device tree, stopping itself will make shared memory region
>> +unaccessible to all borrower domains, so we need to remove foreign memory map
>> +for all borrower domains. Notice that all borrowers domains should be stopped
>> +before stopping the owner domain.
> 
> How will you enforce that?

And this either.

Cheers
Bertrand

> 
>> +
>> +"dom_shared" domain is destroyed when the whole system shuts down, so its
>> +owning statically shared pages are only freed at system shutdown.
>> +
>> +[1] https://marc.info/?l=xen-devel&m=154404821731186
> 
> [2] <alpine.DEB.2.22.394.2201121646290.19362@ubuntu-linux-20-04-desktop>
> 
> -- 
> Julien Grall

Re: [PATCH] design: design doc for shared memory on a dom0less system

[Julien Grall]

Hi Bertrand,

On 26/01/2022 11:14, Bertrand Marquis wrote:
>> On 26 Jan 2022, at 10:58, Julien Grall <julien@xen.org> wrote:
>>
>> Hi,
>>
>> On 26/01/2022 10:09, Penny Zheng wrote:
>>> This commit provides a design doc for static shared memory
>>> on a dom0less system.
>>> Signed-off-by: Penny Zheng <penny.zheng@arm.com>
>>> ---
>>>   design/shm-dom0less.md | 182 +++++++++++++++++++++++++++++++++++++++++
>>>   1 file changed, 182 insertions(+)
>>>   create mode 100644 design/shm-dom0less.md
>>> diff --git a/design/shm-dom0less.md b/design/shm-dom0less.md
>>> new file mode 100644
>>> index 0000000..b46199d
>>> --- /dev/null
>>> +++ b/design/shm-dom0less.md
>>> @@ -0,0 +1,182 @@
>>> +# Static Shared Memory between domains on a dom0less system
>>> +
>>> +This design aims to provide an overview of the new feature: setting up static
>>> +shared memory between domains on a dom0less system, through device tree
>>> +configuration.
>>> +
>>> +The new feature is driven by the need of finding a way to build up
>>> +communication channels on dom0less system, since the legacy ways including
>>> +grant table, etc are all absent there.
>>
>> Stefano has a series to add support for grant-table [2]. So I think you want to justify it differently.
>>
>>> +
>>> +It was inspired by the patch serie of "xl/libxl-based shared memory", see
>>> +[1] for more details.
>>> +
>>> +# Static Shared Memory Device Tree Configuration
>>> +
>>> +The static shared memory device tree nodes allow users to statically set up
>>> +shared memory among a group of dom0less DomUs and Dom0, enabling domains
>>> +to do shm-based communication.
>>> +
>>> +- compatible
>>> +
>>> +    "xen,domain-shared-memory-v1"
>>> +
>>> +- xen,shm-id
>>
>>  From the document, it is not clear to me what is the purpose of the identifier. Could you clarify it?
>>
>>> +
>>> +    An u32 value represents the unique identifier of the shared memory region.
>>> +    User valuing per shared memory region shall follow the ascending order,
>>> +    starting from xen,shm-id = <0x0>, to the maximum identifier
>>> +    xen,shm-id = <0x126>.
>>
>> Why is it limit to 0x126? And also, why do they have to be allocated in ascending order?
>>
>>> The special xen,shm-id = <0x127> is reserved for
>>> +    INVALID_SHMID.
>>
>> Why do we need to reserve invalid?
>>
>>> +
>>> +- xen,shared-mem
>>> +
>>> +    An array takes a physical address, which is the base address of the
>>> +    shared memory region in host physical address space, a size, and a guest
>>> +    physical address, as the target address of the mapping.
>>
>> I think shared memory is useful without static allocation. So I think we want to make the host physical address optional.
>>
>>> +
>>> +- role(Optional)
>>> +
>>> +    A string property specifying the ownership of a shared memory region,
>>> +    the value must be one of the following: "owner", or "borrower"
>>> +    A shared memory region could be explicitly backed by one domain, which is
>>> +    called "owner domain", and all the other domains who are also sharing
>>> +    this region are called "borrower domain".
>>> +    If not specified, the default value is "borrower" and owner is
>>> +    "dom_shared", a system domain.
>>
>> I don't particularly like adding another system domain. Instead, it would be better to always specify the owner.
> 
> Having an owner which is not Xen is creating a dependency so restart the owner you would need to restart the borrowers.

You don't necessarily have to. You can keep the "struct domain" and the 
shared pages in place and wipe everything else.

> To remove this dependency and allow use cases where any domain having access can be restarted without the other side
> needing to, having Xen as the owner is required.

> 
> Initial discussion started between Penny and Stefano went the way you said and I asked to modify it like this to have something
> more looking like a standard shared memory with only users but no “owner”.

My main concern with dom_shared is the permissions. How do you make sure 
that a given page is only shared with the proper domain?

> Also it fits to some of our use cases.

Would you mind to briefly describe them?

Cheers,

-- 
Julien Grall

Re: [PATCH] design: design doc for shared memory on a dom0less system

[Bertrand Marquis]

Hi Julien,

> On 26 Jan 2022, at 11:35, Julien Grall <julien@xen.org> wrote:
> 
> Hi Bertrand,
> 
> On 26/01/2022 11:14, Bertrand Marquis wrote:
>>> On 26 Jan 2022, at 10:58, Julien Grall <julien@xen.org> wrote:
>>> 
>>> Hi,
>>> 
>>> On 26/01/2022 10:09, Penny Zheng wrote:
>>>> This commit provides a design doc for static shared memory
>>>> on a dom0less system.
>>>> Signed-off-by: Penny Zheng <penny.zheng@arm.com>
>>>> ---
>>>>  design/shm-dom0less.md | 182 +++++++++++++++++++++++++++++++++++++++++
>>>>  1 file changed, 182 insertions(+)
>>>>  create mode 100644 design/shm-dom0less.md
>>>> diff --git a/design/shm-dom0less.md b/design/shm-dom0less.md
>>>> new file mode 100644
>>>> index 0000000..b46199d
>>>> --- /dev/null
>>>> +++ b/design/shm-dom0less.md
>>>> @@ -0,0 +1,182 @@
>>>> +# Static Shared Memory between domains on a dom0less system
>>>> +
>>>> +This design aims to provide an overview of the new feature: setting up static
>>>> +shared memory between domains on a dom0less system, through device tree
>>>> +configuration.
>>>> +
>>>> +The new feature is driven by the need of finding a way to build up
>>>> +communication channels on dom0less system, since the legacy ways including
>>>> +grant table, etc are all absent there.
>>> 
>>> Stefano has a series to add support for grant-table [2]. So I think you want to justify it differently.
>>> 
>>>> +
>>>> +It was inspired by the patch serie of "xl/libxl-based shared memory", see
>>>> +[1] for more details.
>>>> +
>>>> +# Static Shared Memory Device Tree Configuration
>>>> +
>>>> +The static shared memory device tree nodes allow users to statically set up
>>>> +shared memory among a group of dom0less DomUs and Dom0, enabling domains
>>>> +to do shm-based communication.
>>>> +
>>>> +- compatible
>>>> +
>>>> +    "xen,domain-shared-memory-v1"
>>>> +
>>>> +- xen,shm-id
>>> 
>>> From the document, it is not clear to me what is the purpose of the identifier. Could you clarify it?
>>> 
>>>> +
>>>> +    An u32 value represents the unique identifier of the shared memory region.
>>>> +    User valuing per shared memory region shall follow the ascending order,
>>>> +    starting from xen,shm-id = <0x0>, to the maximum identifier
>>>> +    xen,shm-id = <0x126>.
>>> 
>>> Why is it limit to 0x126? And also, why do they have to be allocated in ascending order?
>>> 
>>>> The special xen,shm-id = <0x127> is reserved for
>>>> +    INVALID_SHMID.
>>> 
>>> Why do we need to reserve invalid?
>>> 
>>>> +
>>>> +- xen,shared-mem
>>>> +
>>>> +    An array takes a physical address, which is the base address of the
>>>> +    shared memory region in host physical address space, a size, and a guest
>>>> +    physical address, as the target address of the mapping.
>>> 
>>> I think shared memory is useful without static allocation. So I think we want to make the host physical address optional.
>>> 
>>>> +
>>>> +- role(Optional)
>>>> +
>>>> +    A string property specifying the ownership of a shared memory region,
>>>> +    the value must be one of the following: "owner", or "borrower"
>>>> +    A shared memory region could be explicitly backed by one domain, which is
>>>> +    called "owner domain", and all the other domains who are also sharing
>>>> +    this region are called "borrower domain".
>>>> +    If not specified, the default value is "borrower" and owner is
>>>> +    "dom_shared", a system domain.
>>> 
>>> I don't particularly like adding another system domain. Instead, it would be better to always specify the owner.
>> Having an owner which is not Xen is creating a dependency so restart the owner you would need to restart the borrowers.
> 
> You don't necessarily have to. You can keep the "struct domain" and the shared pages in place and wipe everything else.

But you need to wipe everything.

> 
>> To remove this dependency and allow use cases where any domain having access can be restarted without the other side
>> needing to, having Xen as the owner is required.
> 
>> Initial discussion started between Penny and Stefano went the way you said and I asked to modify it like this to have something
>> more looking like a standard shared memory with only users but no “owner”.
> 
> My main concern with dom_shared is the permissions. How do you make sure that a given page is only shared with the proper domain?

That is by configuration as you define those area in your config and say who has access to it.
Or maybe I misunderstood your question ?

> 
>> Also it fits to some of our use cases.
> 
> Would you mind to briefly describe them?

Sure:
- one domain real time domain writing some logs in the shared memory that is polled and stored on a storage by an other domain. 
Real-time domain might reboot and wipe the logs and this could be acceptable, Linux might reboot and restart where it left.
- lockless fifo between 2 domains. If one restarts it can start over from where it left (Only need a one time initialiser at boot)
- redundancy system, for example 2 domains providing a position through different means, you do not care who provides
 you just need the latest info so one could override the data from the other one, the reader would not mind.
- configuration data storage. One domain reads the config and then stops until it is restarted to update it, other domains can read the config

I hope those are helping, I am open to provide others if you need.

Cheers
Bertrand

> 
> Cheers,
> 
> -- 
> Julien Grall

Re: [PATCH] design: design doc for shared memory on a dom0less system

[Stefano Stabellini]

On Wed, 26 Jan 2022, Julien Grall wrote:
> > +
> > +- role(Optional)
> > +
> > +    A string property specifying the ownership of a shared memory region,
> > +    the value must be one of the following: "owner", or "borrower"
> > +    A shared memory region could be explicitly backed by one domain, which
> > is
> > +    called "owner domain", and all the other domains who are also sharing
> > +    this region are called "borrower domain".
> > +    If not specified, the default value is "borrower" and owner is
> > +    "dom_shared", a system domain.
> 
> I don't particularly like adding another system domain. Instead, it would be
> better to always specify the owner.
 
I wonder if we could reuse one of the existing system domains. DOM_IO
looks very close to what we need here, except that the current
description doesn't quite fit:

"DOMID_IO is used to restrict page-table updates to mapping I/O memory."

Also DOMID_XEN doesn't fit:

"DOMID_XEN is used to allow privileged domains to map restricted parts
of Xen's heap space (e.g., the machine_to_phys table)."


On the other hand the description of DOMID_COW seems to be exactly what
we want:

"DOMID_COW is used as the owner of sharable pages"

This is technically the description of what we need :-D  However, we
know that DOMID_COW was introduced with a different goal in mind.


> > +
> > +## Example
> > +
> > +chosen {
> > +    #address-cells = <0x1>;
> > +    #size-cells = <0x1>;
> > +    xen,xen-bootargs = "console=dtuart dtuart=serial0 bootscrub=0";
> > +
> > +    ......
> > +
> > +    /* this is for Dom0 */
> > +    dom0-shared-mem@10000000 {
> > +        compatible = "xen,domain-shared-memory-v1";
> > +        xen,shm-id = <0x0>;
> > +        role = "owner";
> > +        xen,shared-mem = <0x10000000 0x10000000 0x10000000>;
> > +    }
> > +
> > +    domU1 {
> > +        compatible = "xen,domain";
> > +        #address-cells = <0x1>;
> > +        #size-cells = <0x1>;
> > +        memory = <0 131072>;
> > +        cpus = <2>;
> > +        vpl011;
> > +
> > +        /*
> > +         * shared memory region identified as 0x0(xen,shm-id = <0x0>)
> > +         * shared between dom0.
> > +         */
> > +        domU1-shared-mem@10000000 {
> > +            compatible = "xen,domain-shared-memory-v1";
> > +            xen,shm-id = <0x0>;
> > +            role = "borrower";
> > +            xen,shared-mem = <0x10000000 0x10000000 0x50000000>;
> 
> Technically, you already know the physical address from the owner. In fact, it
> will only increase the risk to get the wrong binding. So I would like to
> suggest a different binding.
> 
> 1) Reserve the region in the host memory using reserved-memory binding
> 2) Create a binding per domain that contains a phandle to the host memory and
> the role.
> 
> The advantage with this is we could easily support region that are not backed
> by a reserved-memory.

I see what you mean but given that we have to specify the guest physical
address anyway I don't think that replacing the physical address with a
phandle would make things easier. We would still need to specify guest
physical address and maybe size. We already have xen,reg that also
works similarly. So I think it would be best to follow the same format
as xen,reg.

That would also handle the case where the shared memory region is an
sram region: we just need to use an address range that corresponds to
mmio-sram instead of memory. Note that with a phandle we would have the
problem that we could only share the entire sram region and not a part
of it (because we could only link to the full mmio-sram node and not to
a subset of it).

We just need to clearly specify the number of address cells and size
cells to use. We could introduce #shared-mem-address-cells or reuse
#address-cells. Either way parsing it should be easy.


If we wanted to use a phandle, I think it should point to a
reserved-memory region outside of /reserved-memory. Long explanation
follows.

Reserved-memory is a bit tricky to get right because reserved-memory is a
configuration node and we don't have a way to tell who the
reserved-memory configuration is for. Is it for Xen? Or for dom0? Or for
domUs? When discussing system device tree topics (system device tree is
very similar to device tree with Xen domain nodes) reserved-memory was
one of the firsts concerns that Rob raised. The solution for system
device tree was that each domain node should have its own
reserved-memory sub-node instead of using /reserved-memory (e.g.
/chosen/domU/reserved-memory).

Going back to Xen, if we take domUs out of the picture, who is
/reserved-memory for? In practice, as most drivers are in dom0, we have
been assuming /reserved-memory is for dom0. Xen copies reserved-memory
nodes to dom0.

This is why I would like to suggest that if we want to use the phandle
model then I think we should use a new and different reserved-memory
node to specify the shared memory. E.g. /chosen/reserved-memory. We
could call it differently and/or have an unambiguous compatible string:

chosen {
    reserved-memory {
        compatible = "xen,domains-reserved-memory-v1";

        ...
    };

This way we clearly distinguish it from /reserved-memory and we avoid
ownership conflicts with dom0.

 
> > +        }
> > +
> > +        domU1-shared-mem@50000000 {
> > +            compatible = "xen,domain-shared-memory-v1";
> > +            xen,shm-id = <0x1>;
> > +            xen,shared-mem = <0x50000000 0x20000000 0x60000000>;
> > +        }
> > +
> > +        ......
> > +
> > +    };
> > +
> > +    domU2 {
> > +        compatible = "xen,domain";
> > +        #address-cells = <0x1>;
> > +        #size-cells = <0x1>;
> > +        memory = <0 65536>;
> > +        cpus = <1>;
> > +
> > +        /*
> > +         * shared memory region identified as 0x1(xen,shm-id = <0x1>)
> > +         * shared between domU1.
> > +         */
> > +        domU2-shared-mem@50000000 {
> > +            compatible = "xen,domain-shared-memory-v1";
> > +            xen,shm-id = <0x1>;
> > +            xen,shared-mem = <0x50000000 0x20000000 0x70000000>;
> > +        }
> > +
> > +        ......
> > +    };
> > +};

RE: [PATCH] design: design doc for shared memory on a dom0less system

[Penny Zheng]

Hi Julien

Sorry for the late response, Since I'm considering sending the code together for better
understanding.

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: Wednesday, January 26, 2022 6:58 PM
> To: Penny Zheng <Penny.Zheng@arm.com>; xen-devel@lists.xenproject.org;
> sstabellini@kernel.org
> Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>; Wei Chen
> <Wei.Chen@arm.com>
> Subject: Re: [PATCH] design: design doc for shared memory on a dom0less
> system
> 
> Hi,
> 
> On 26/01/2022 10:09, Penny Zheng wrote:
> > This commit provides a design doc for static shared memory on a
> > dom0less system.
> >
> > Signed-off-by: Penny Zheng <penny.zheng@arm.com>
> > ---
> >   design/shm-dom0less.md | 182
> +++++++++++++++++++++++++++++++++++++++++
> >   1 file changed, 182 insertions(+)
> >   create mode 100644 design/shm-dom0less.md
> >
> > diff --git a/design/shm-dom0less.md b/design/shm-dom0less.md new file
> > mode 100644 index 0000000..b46199d
> > --- /dev/null
> > +++ b/design/shm-dom0less.md
> > @@ -0,0 +1,182 @@
> > +# Static Shared Memory between domains on a dom0less system
> > +
> > +This design aims to provide an overview of the new feature: setting
> > +up static shared memory between domains on a dom0less system, through
> > +device tree configuration.
> > +
> > +The new feature is driven by the need of finding a way to build up
> > +communication channels on dom0less system, since the legacy ways
> > +including grant table, etc are all absent there.
> 
> Stefano has a series to add support for grant-table [2]. So I think you want to
> justify it differently.
> 

The dom0less system I am referring here is that either dom0 is totally missing in the system,
or even when dom0 is there, all the other domains are statically configured and CONFIG_XEN
is not enabled, so in above scenario, all the legacy ways to communicate between domains are
absent. These scenarios are specially applied to ARMv8R, where everything is static.

I think Stefano is trying to enable PV drivers on dom0less guest with CONFIG_XEN still on.
TBH, I could not find proper nouns for them, sometimes calling these domains dom0less guests but
with dom0 still there makes me confused too. ;/

> > +
> > +It was inspired by the patch serie of "xl/libxl-based shared memory",
> > +see [1] for more details.
> > +
> > +# Static Shared Memory Device Tree Configuration
> > +
> > +The static shared memory device tree nodes allow users to statically
> > +set up shared memory among a group of dom0less DomUs and Dom0,
> > +enabling domains to do shm-based communication.
> > +
> > +- compatible
> > +
> > +    "xen,domain-shared-memory-v1"
> > +
> > +- xen,shm-id
> 
>  From the document, it is not clear to me what is the purpose of the identifier.
> Could you clarify it?
> 

It is more related to the implementation. 

I've already pushed an RFC patch for reviewing in community for better understanding.  For example,
in commit "xen/arm: introduce static shared memory", when parsing shared memory node to reserve it
in advance, we notice that a shared memory region could be shared among multiple domains, so in order to
prevent iterating over all reserved memory regions each time, we use bitmap there and shm-id is used as the
index of the bitmap.

Also, in Linux Doc about xen,shared-memory binding[3], it requires a xen,id property that identifies the
shared memory region as specified in the VM config file. 

> > +
> > +    An u32 value represents the unique identifier of the shared memory
> region.
> > +    User valuing per shared memory region shall follow the ascending order,
> > +    starting from xen,shm-id = <0x0>, to the maximum identifier
> > +    xen,shm-id = <0x126>.
> 
> Why is it limit to 0x126? And also, why do they have to be allocated in
> ascending order?
> 

In current code, I make it the same number with NR_MEM_BANKS

> > The special xen,shm-id = <0x127> is reserved for
> > +    INVALID_SHMID.
> 
> Why do we need to reserve invalid?
> 

It is removed in current codes.

> > +
> > +- xen,shared-mem
> > +
> > +    An array takes a physical address, which is the base address of the
> > +    shared memory region in host physical address space, a size, and a guest
> > +    physical address, as the target address of the mapping.
> 
> I think shared memory is useful without static allocation. So I think we want to
> make the host physical address optional.
> 

Hmm,, so later,, you want to make one domain as owner, and parts of its guest RAM
is shared with other borrower domains? So only GFN is enough here, right?

> > +
> > +- role(Optional)
> > +
> > +    A string property specifying the ownership of a shared memory region,
> > +    the value must be one of the following: "owner", or "borrower"
> > +    A shared memory region could be explicitly backed by one domain, which
> is
> > +    called "owner domain", and all the other domains who are also sharing
> > +    this region are called "borrower domain".
> > +    If not specified, the default value is "borrower" and owner is
> > +    "dom_shared", a system domain.
> 
> I don't particularly like adding another system domain. Instead, it would be
> better to always specify the owner.
> 
> > +
> > +## Example
> > +
> > +chosen {
> > +    #address-cells = <0x1>;
> > +    #size-cells = <0x1>;
> > +    xen,xen-bootargs = "console=dtuart dtuart=serial0 bootscrub=0";
> > +
> > +    ......
> > +
> > +    /* this is for Dom0 */
> > +    dom0-shared-mem@10000000 {
> > +        compatible = "xen,domain-shared-memory-v1";
> > +        xen,shm-id = <0x0>;
> > +        role = "owner";
> > +        xen,shared-mem = <0x10000000 0x10000000 0x10000000>;
> > +    }
> > +
> > +    domU1 {
> > +        compatible = "xen,domain";
> > +        #address-cells = <0x1>;
> > +        #size-cells = <0x1>;
> > +        memory = <0 131072>;
> > +        cpus = <2>;
> > +        vpl011;
> > +
> > +        /*
> > +         * shared memory region identified as 0x0(xen,shm-id = <0x0>)
> > +         * shared between dom0.
> > +         */
> > +        domU1-shared-mem@10000000 {
> > +            compatible = "xen,domain-shared-memory-v1";
> > +            xen,shm-id = <0x0>;
> > +            role = "borrower";
> > +            xen,shared-mem = <0x10000000 0x10000000 0x50000000>;
> 
> Technically, you already know the physical address from the owner. In fact, it
> will only increase the risk to get the wrong binding. So I would like to suggest a
> different binding.
> 
> 1) Reserve the region in the host memory using reserved-memory binding
> 2) Create a binding per domain that contains a phandle to the host memory
> and the role.
> 
> The advantage with this is we could easily support region that are not backed
> by a reserved-memory.
> 
> 
> > +        }
> > +
> > +        domU1-shared-mem@50000000 {
> > +            compatible = "xen,domain-shared-memory-v1";
> > +            xen,shm-id = <0x1>;
> > +            xen,shared-mem = <0x50000000 0x20000000 0x60000000>;
> > +        }
> > +
> > +        ......
> > +
> > +    };
> > +
> > +    domU2 {
> > +        compatible = "xen,domain";
> > +        #address-cells = <0x1>;
> > +        #size-cells = <0x1>;
> > +        memory = <0 65536>;
> > +        cpus = <1>;
> > +
> > +        /*
> > +         * shared memory region identified as 0x1(xen,shm-id = <0x1>)
> > +         * shared between domU1.
> > +         */
> > +        domU2-shared-mem@50000000 {
> > +            compatible = "xen,domain-shared-memory-v1";
> > +            xen,shm-id = <0x1>;
> > +            xen,shared-mem = <0x50000000 0x20000000 0x70000000>;
> > +        }
> > +
> > +        ......
> > +    };
> > +};
> > +
> > +It is the example of two static shared memory regions.
> > +
> > +In terms of shared memory region identified as 0x0, host physical address
> > +starting at 0x10000000 of 256MB will be reserved to be shared between
> Dom0
> > +and DomU1. It will get mapped at 0x10000000 in Dom0 guest physical
> address
> > +space, and at 0x50000000 in DomU1 guest physical address space. Dom0 is
> the
> > +owner domain, and domU1 is the borrower domain.
> > +
> > +And in terms of shared memory region identified as 0x1, host physical
> address
> > +starting at 0x50000000 of 512MB will be reserved to be shared between
> DomU1
> > +and DomU2. It will get mapped at 0x60000000 in DomU1 guest physical
> address
> > +space, and at 0x70000000 in DomU2 guest physical address space. Since no
> owner
> > +domain is explicitly defined, the default "dom_shared" is the owner
> domain,
> > +and both domU1 and domU2 are the borrower domains.
> > +
> > +# Overview of Static Shared Memory Flow
> > +
> > +Static Shared Memory working flow could be classified into the following
> > +steps:
> > + - Carve out a range of memory in host physical address space to be used
> > +for sharing. Define it in device tree configuration, then parse and reserve
> > +it to avoid other use.
> > + - Create a special domain "dom_shared". It will be the owner domain
> which
> > +is owning the statically shared pages, if "role" property is not specified.
> > + - Per shared memory region could be shared with multiple domains. For
> > +owner domain, it acquires statically shared pages and assign them to itself,
> > +in the same way with static memory. And other than owner domain, the
> others
> > +who are also sharing are called "borrower domain", for which foreign
> memory
> > +map of statically shared pages is required.
> > + - Expose the shared memory to the domU using the "xen,shared-memory-
> v1"
> > +reserved-memory binding. See
> > +Documentation/devicetree/bindings/reserved-memory/xen,shared-
> memory.txt
> > +in Linux for the corresponding device tree binding.
> > +
> > +# Memory management of Shared Memory Region
> > +
> > +Each memory page needs to have an "owner" and it is likely that in many
> cases
> > +the user don't care who the owner is, so it makes sense that users don't
> > +need to specify the "role" in device tree if they don't want to, in which
> > +scenario, a default domain shall be the owner domain.
> > +
> > +We propose a new system domain "dom_shared" to be the default domain
> owning all
> > +statically unowned shared pages, assigning it dom_id
> 0x7FF5(DOMID_SHARED).
> > +
> > +"dom_shared" domain shall get constructed before domain construction
> and after
> > +"setup_virt_paging", during system boot-time, so it could successfully do
> > +p2m initialization.
> 
> IHMO, this is going too much into details for a design document. The
> goal is to abstract the feature rather than mentioning the
> implementation (which may change during review or in the future).
> 

Sure, I'll delete

> > +
> > +Owner domain acquires statically shared pages and assign them to itself,
> > +while borrower domains get and take reference of them, then do foreign
> memory
> > +map of these statically shared pages.
> 
> What happens if the borrower is seen before the owner?
> 

I've a commit specially focusing on this scenario, PLZ see commit "
xen/arm: defer foreign memory map in shm_init_late", like I also comment in
the codes:
"
In a few scenarios where owner domain, is defined after borrower domain in
device tree configuration, statically shared pages haven't been properly allocated
if borrower domain here tries to do foreign memory map.
In order to cover such scenario, we defer all borrower domains' foreign memory
map after all domain construction finished.
"

> > +
> > +When destroying or rebooting a domain, if it is a borrower domain, other
> than
> > +removing foreign memory map of statically shared pages in P2M table, we
> also
> > +need to drop according gained reference. And if it is an owner domain,
> since
> > +statically shared pages are allocated as guest normal ram, it is not needed
> to
> > +do extra removing.
> > +
> > +However if owner domain is not the default "dom_shared" domain, but
> specified
> > +explicitly in device tree, stopping itself will make shared memory region
> > +unaccessible to all borrower domains, so we need to remove foreign
> memory map
> > +for all borrower domains. Notice that all borrowers domains should be
> stopped
> > +before stopping the owner domain.
> 
> How will you enforce that?
>

Yes, I agree that we could not enforce that, so in commit "
xen/arm: unmap foreign memory mapping when destroyed domain is owner domain",
Now, when destroyed domain is an owner domain of a static shared memory region, 
we need to ensure that all according borrower domains shall not have the access to this
static shared memory region too.
 
> > +
> > +"dom_shared" domain is destroyed when the whole system shuts down, so
> its
> > +owning statically shared pages are only freed at system shutdown.
> > +
> > +[1] https://marc.info/?l=xen-devel&m=154404821731186
> 
> [2] <alpine.DEB.2.22.394.2201121646290.19362@ubuntu-linux-20-04-desktop>
> 

[3] https://github.com/torvalds/linux/blob/master/Documentation/devicetree/bindings/reserved-memory/xen%2Cshared-memory.txt

> --
> Julien Grall

--
Penny Zheng