[PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Rahul Singh]

in dom0less system. This patch introduce the new feature to support the
signaling between two domUs in dom0less system.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
 docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
 1 file changed, 96 insertions(+)
 create mode 100644 docs/designs/dom0less-evtchn.md

diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
new file mode 100644
index 0000000000..6a1b7e8c22
--- /dev/null
+++ b/docs/designs/dom0less-evtchn.md
@@ -0,0 +1,96 @@
+# Signaling support between two domUs on dom0less system
+
+## Current state: Draft version
+
+## Proposer(s): Rahul Singh, Bertrand Marquis
+
+## Problem Statement:
+
+The goal of this work is to define a simple signaling system between Xen guests
+in dom0less systems.
+
+In dom0less system, we cannot make use of xenbus and xenstore that are used in
+normal systems with dynamic VMs to communicate between domains by providing a
+bus abstraction for paravirtualized drivers.
+
+One possible solution to implement the signaling system between domUs is based
+on event channels.
+
+## Proposal:
+
+Event channels are the basic primitive provided by Xen for event notifications.
+An event channel is a logical connection between 2 domains (more specifically
+between dom1,port1 and dom2,port2). They essentially store one bit of
+information, the event of interest is signalled by transitioning this bit from
+0 to 1. An event is an equivalent of a hardware interrupt.
+
+Notifications are received by a guest via an interrupt from Xen to the guest,
+indicating when an event arrives (setting the bit). Further notifications are
+masked until the bit is cleared again. When a domain wants to wait for data it
+will block until an event arrives, and then send an event to signal that data
+has been consumed. Events are delivered asynchronously to guests and are
+enqueued when the guest is not running.
+
+Event channel communication will be established statically between two domU
+guests before unpausing the domains after domain creation. Event channel
+connection information between domUs will be passed to XEN via device tree
+node.
+
+Under the /chosen node, there needs to be sub nodes with compatible
+"xen,evtchn" that descibes the event channel connection between two domUs.
+
+The event channel sub-node has the following properties:
+
+- compatible
+
+    "xen,evtchn"
+
+- xen,evtchn
+
+    The property is four numbers of tuples of
+    (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle) where:
+
+    local-port-domU1 is an integer value that will be used to allocte local
+    port for domU1 to send an event notification to the remote domain.
+
+    domU1-phandle is a single phandle to an domain to which local-port-domU1
+    will be allocated.
+
+    local-port-domU2 is an integer value that will be used to allocte local
+    port for domU2 to send an event notification to the remote domain.
+
+    domU2-phandle is a single phandle to an domain to which local-port-domU2
+    will be allocated.
+
+Example:
+
+    chosen {
+        ....
+
+        domU1: domU1 {
+            ......
+        };
+
+        domU2: domU2 {
+            ......
+        };
+
+        evtchn@1 {
+            compatible = "xen,evtchn";
+            xen,evtchn = <0xa &domU1 0xb &domU2>;
+        };
+
+        evtchn@2 {
+            compatible = "xen,evtchn";
+            xen,evtchn = <0xc &domU1 0xd &domU2>;
+        };
+    };
+
+In above example two event channel comunication will be established between
+domU1 and domU2.
+
+    domU1 (port 0xa) <-----------------> domU2 (port 0xb)
+    domU1 (port 0xc) <-----------------> domU2 (port 0xd)
+
+domU1 and domU2 can send the signal to remote domain via hypercall
+EVTCHNOP_send(.) on local port.
-- 
2.25.1

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Jan Beulich]

On 23.03.2022 16:43, Rahul Singh wrote:
> in dom0less system. This patch introduce the new feature to support the
> signaling between two domUs in dom0less system.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>  docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>  1 file changed, 96 insertions(+)
>  create mode 100644 docs/designs/dom0less-evtchn.md
> 
> diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
> new file mode 100644
> index 0000000000..6a1b7e8c22
> --- /dev/null
> +++ b/docs/designs/dom0less-evtchn.md
> @@ -0,0 +1,96 @@
> +# Signaling support between two domUs on dom0less system
> +
> +## Current state: Draft version
> +
> +## Proposer(s): Rahul Singh, Bertrand Marquis
> +
> +## Problem Statement:
> +
> +The goal of this work is to define a simple signaling system between Xen guests
> +in dom0less systems.
> +
> +In dom0less system, we cannot make use of xenbus and xenstore that are used in
> +normal systems with dynamic VMs to communicate between domains by providing a
> +bus abstraction for paravirtualized drivers.
> +
> +One possible solution to implement the signaling system between domUs is based
> +on event channels.
> +
> +## Proposal:
> +
> +Event channels are the basic primitive provided by Xen for event notifications.
> +An event channel is a logical connection between 2 domains (more specifically
> +between dom1,port1 and dom2,port2). They essentially store one bit of
> +information, the event of interest is signalled by transitioning this bit from
> +0 to 1. An event is an equivalent of a hardware interrupt.

Nit: Since you're talking of channels here, not ports, strictly speaking
it's two bits - one on each side. Alternatively instead of "They ..."
you could say "Each port ...".

> +Notifications are received by a guest via an interrupt from Xen to the guest,
> +indicating when an event arrives (setting the bit). Further notifications are
> +masked until the bit is cleared again. When a domain wants to wait for data it
> +will block until an event arrives, and then send an event to signal that data
> +has been consumed. Events are delivered asynchronously to guests and are
> +enqueued when the guest is not running.
> +
> +Event channel communication will be established statically between two domU
> +guests before unpausing the domains after domain creation. Event channel
> +connection information between domUs will be passed to XEN via device tree
> +node.
> +
> +Under the /chosen node, there needs to be sub nodes with compatible
> +"xen,evtchn" that descibes the event channel connection between two domUs.
> +
> +The event channel sub-node has the following properties:
> +
> +- compatible
> +
> +    "xen,evtchn"
> +
> +- xen,evtchn
> +
> +    The property is four numbers of tuples of
> +    (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle) where:

Nit: I think you mean "tuples of four numbers"?

> +    local-port-domU1 is an integer value that will be used to allocte local
> +    port for domU1 to send an event notification to the remote domain.
> +
> +    domU1-phandle is a single phandle to an domain to which local-port-domU1
> +    will be allocated.
> +
> +    local-port-domU2 is an integer value that will be used to allocte local
> +    port for domU2 to send an event notification to the remote domain.
> +
> +    domU2-phandle is a single phandle to an domain to which local-port-domU2
> +    will be allocated.
> +
> +Example:
> +
> +    chosen {
> +        ....
> +
> +        domU1: domU1 {
> +            ......
> +        };
> +
> +        domU2: domU2 {
> +            ......
> +        };
> +
> +        evtchn@1 {
> +            compatible = "xen,evtchn";
> +            xen,evtchn = <0xa &domU1 0xb &domU2>;
> +        };
> +
> +        evtchn@2 {
> +            compatible = "xen,evtchn";
> +            xen,evtchn = <0xc &domU1 0xd &domU2>;
> +        };
> +    };
> +
> +In above example two event channel comunication will be established between
> +domU1 and domU2.
> +
> +    domU1 (port 0xa) <-----------------> domU2 (port 0xb)
> +    domU1 (port 0xc) <-----------------> domU2 (port 0xd)
> +
> +domU1 and domU2 can send the signal to remote domain via hypercall
> +EVTCHNOP_send(.) on local port.

How do the DomU-s learn of these ports? I guess information is to be
extracted into their individual DT representation, but this could do
with spelling out, including how those nodes (or however the data
items are called in DT) would be named and what data they would
contain.

Further I assume this being encoded in the DT passed to Xen means
implicit consent of the admin security-wise. If so, this could do
with making explicit as well. If not, the security of this would
need discussing in even broader a scope.

Finally I assume such channels are established fully bound, i.e.
there's nothing to do by guests in order to establish communication.
This is another aspect that would imo better be said explicitly.

Jan

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Rahul Singh]

Hello Jan,

Thanks for reviewing the design.

> On 23 Mar 2022, at 4:07 pm, Jan Beulich <jbeulich@suse.com> wrote:
> 
> On 23.03.2022 16:43, Rahul Singh wrote:
>> in dom0less system. This patch introduce the new feature to support the
>> signaling between two domUs in dom0less system.
>> 
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>> 1 file changed, 96 insertions(+)
>> create mode 100644 docs/designs/dom0less-evtchn.md
>> 
>> diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
>> new file mode 100644
>> index 0000000000..6a1b7e8c22
>> --- /dev/null
>> +++ b/docs/designs/dom0less-evtchn.md
>> @@ -0,0 +1,96 @@
>> +# Signaling support between two domUs on dom0less system
>> +
>> +## Current state: Draft version
>> +
>> +## Proposer(s): Rahul Singh, Bertrand Marquis
>> +
>> +## Problem Statement:
>> +
>> +The goal of this work is to define a simple signaling system between Xen guests
>> +in dom0less systems.
>> +
>> +In dom0less system, we cannot make use of xenbus and xenstore that are used in
>> +normal systems with dynamic VMs to communicate between domains by providing a
>> +bus abstraction for paravirtualized drivers.
>> +
>> +One possible solution to implement the signaling system between domUs is based
>> +on event channels.
>> +
>> +## Proposal:
>> +
>> +Event channels are the basic primitive provided by Xen for event notifications.
>> +An event channel is a logical connection between 2 domains (more specifically
>> +between dom1,port1 and dom2,port2). They essentially store one bit of
>> +information, the event of interest is signalled by transitioning this bit from
>> +0 to 1. An event is an equivalent of a hardware interrupt.
> 
> Nit: Since you're talking of channels here, not ports, strictly speaking
> it's two bits - one on each side. Alternatively instead of "They ..."
> you could say "Each port ...".
> 

Ack. I will correct it in next version.

>> +Notifications are received by a guest via an interrupt from Xen to the guest,
>> +indicating when an event arrives (setting the bit). Further notifications are
>> +masked until the bit is cleared again. When a domain wants to wait for data it
>> +will block until an event arrives, and then send an event to signal that data
>> +has been consumed. Events are delivered asynchronously to guests and are
>> +enqueued when the guest is not running.
>> +
>> +Event channel communication will be established statically between two domU
>> +guests before unpausing the domains after domain creation. Event channel
>> +connection information between domUs will be passed to XEN via device tree
>> +node.
>> +
>> +Under the /chosen node, there needs to be sub nodes with compatible
>> +"xen,evtchn" that descibes the event channel connection between two domUs.
>> +
>> +The event channel sub-node has the following properties:
>> +
>> +- compatible
>> +
>> +    "xen,evtchn"
>> +
>> +- xen,evtchn
>> +
>> +    The property is four numbers of tuples of
>> +    (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle) where:
> 
> Nit: I think you mean "tuples of four numbers"?

Yes, you are right I will fix this.
> 
>> +    local-port-domU1 is an integer value that will be used to allocte local
>> +    port for domU1 to send an event notification to the remote domain.
>> +
>> +    domU1-phandle is a single phandle to an domain to which local-port-domU1
>> +    will be allocated.
>> +
>> +    local-port-domU2 is an integer value that will be used to allocte local
>> +    port for domU2 to send an event notification to the remote domain.
>> +
>> +    domU2-phandle is a single phandle to an domain to which local-port-domU2
>> +    will be allocated.
>> +
>> +Example:
>> +
>> +    chosen {
>> +        ....
>> +
>> +        domU1: domU1 {
>> +            ......
>> +        };
>> +
>> +        domU2: domU2 {
>> +            ......
>> +        };
>> +
>> +        evtchn@1 {
>> +            compatible = "xen,evtchn";
>> +            xen,evtchn = <0xa &domU1 0xb &domU2>;
>> +        };
>> +
>> +        evtchn@2 {
>> +            compatible = "xen,evtchn";
>> +            xen,evtchn = <0xc &domU1 0xd &domU2>;
>> +        };
>> +    };
>> +
>> +In above example two event channel comunication will be established between
>> +domU1 and domU2.
>> +
>> +    domU1 (port 0xa) <-----------------> domU2 (port 0xb)
>> +    domU1 (port 0xc) <-----------------> domU2 (port 0xd)
>> +
>> +domU1 and domU2 can send the signal to remote domain via hypercall
>> +EVTCHNOP_send(.) on local port.
> 
> How do the DomU-s learn of these ports? I guess information is to be
> extracted into their individual DT representation, but this could do
> with spelling out, including how those nodes (or however the data
> items are called in DT) would be named and what data they would
> contain.

There are two options how the guest learn the local ports.

1. Document the event channel connection information for the end-user in
    the end user documentation. The integrator will make use of the local port information
    to write a userspace application to send events to the remote domain. For Linux, in this
    case, we need to modify the current  "drivers/xen/evtchn.c” to include a new IOCTL to
    bind the local port to the user and set up the interrupt handler. The application can use the ioctl
    IOCTL_STATIC_EVTCHN_BIND and IOCTL_EVTCHN_NOTIFY to notify the remote domain.

	case IOCTL_STATIC_EVTCHN_BIND: {
		struct ioctl_evtchn_bind bind;

		rc = -EFAULT;
		if (copy_from_user(&bind, uarg, sizeof(bind)))
			break;

		rc = evtchn_bind_to_user(u, bind.port);
		evtchn_bind_interdom_next_vcpu(bind.port);
		break;                                                                  
    }

   We can also implement the new hypercall for error checking if the port requested from user application is static port
   and already created in XEN.

2. Create the DT node for guest and pass the local port information to guest. Guest can
    make use this DT node to create device node “ /dev/xen/eventchn-localport" or something else in Linux.
    In this case we may need to write the new driver in linux to support this.
 
> 
> Further I assume this being encoded in the DT passed to Xen means
> implicit consent of the admin security-wise. If so, this could do
> with making explicit as well. If not, the security of this would
> need discussing in even broader a scope.

Yes, it is responsibility of the admin to take care of security when 
defining the event channel in DT.
> 
> Finally I assume such channels are established fully bound, i.e.
> there's nothing to do by guests in order to establish communication.
> This is another aspect that would imo better be said explicitly.
> 

Yes, you are right event channel will be created and established beforehand in XEN before
guests started. The guest doesn’t need to do any operation to establish a connection. Guest 
only needs hypercall EVTCHNOP_send(local port) to send notifications to the remote guest.

Regards,
Rahul
> Jan

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[David Vrabel]

On 23/03/2022 15:43, Rahul Singh wrote:
> in dom0less system. This patch introduce the new feature to support the
> signaling between two domUs in dom0less system.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>   docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>   1 file changed, 96 insertions(+)
>   create mode 100644 docs/designs/dom0less-evtchn.md
> 
> diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
> new file mode 100644
> index 0000000000..6a1b7e8c22
> --- /dev/null
> +++ b/docs/designs/dom0less-evtchn.md
> @@ -0,0 +1,96 @@
> +# Signaling support between two domUs on dom0less system
> +
> +## Current state: Draft version
> +
> +## Proposer(s): Rahul Singh, Bertrand Marquis
> +
> +## Problem Statement:
> +
> +The goal of this work is to define a simple signaling system between Xen guests
> +in dom0less systems.
> +
> +In dom0less system, we cannot make use of xenbus and xenstore that are used in
> +normal systems with dynamic VMs to communicate between domains by providing a
> +bus abstraction for paravirtualized drivers.
> +
> +One possible solution to implement the signaling system between domUs is based
> +on event channels.

This problem statement could do with some example use cases that are 
usefully solved by this proposed solution.

"We don't have xenstore so can't set up shared rings, but here's a 
replacement comms mechanism that can do a single bit." Doesn't seem very 
compelling to me.

> +    chosen {
> +        ....
> +
> +        domU1: domU1 {
> +            ......
> +        };
> +
> +        domU2: domU2 {
> +            ......
> +        };
> +
> +        evtchn@1 {
> +            compatible = "xen,evtchn";
> +            xen,evtchn = <0xa &domU1 0xb &domU2>;
> +        };
> +
> +        evtchn@2 {
> +            compatible = "xen,evtchn";
> +            xen,evtchn = <0xc &domU1 0xd &domU2>;
> +        };

How is the domain supposed to know what these event channels are for?

I'm not that familiar with device tree. Is it possible to give these 
entries name?

David

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 7446 bytes --]

On Wed, 23 Mar 2022, Rahul Singh wrote:
> in dom0less system. This patch introduce the new feature to support the
> signaling between two domUs in dom0less system.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>  docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>  1 file changed, 96 insertions(+)
>  create mode 100644 docs/designs/dom0less-evtchn.md
> 
> diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
> new file mode 100644
> index 0000000000..6a1b7e8c22
> --- /dev/null
> +++ b/docs/designs/dom0less-evtchn.md
> @@ -0,0 +1,96 @@
> +# Signaling support between two domUs on dom0less system
> +
> +## Current state: Draft version
> +
> +## Proposer(s): Rahul Singh, Bertrand Marquis
> +
> +## Problem Statement:
> +
> +The goal of this work is to define a simple signaling system between Xen guests
> +in dom0less systems.
> +
> +In dom0less system, we cannot make use of xenbus and xenstore that are used in
> +normal systems with dynamic VMs to communicate between domains by providing a
> +bus abstraction for paravirtualized drivers.
> +
> +One possible solution to implement the signaling system between domUs is based
> +on event channels.

I suggest to reword this as follows:

---
Dom0less guests would benefit from a statically-defined memory sharing
and signally system for communication. One that would be immediately
available at boot without any need for dynamic configurations.

In embedded a great variety of guest operating system kernels exist,
many of which don't have support for xenstore, grant table or other
complex drivers. Some of them are small kernel-space applications (often
called "baremetal", not to be confused with the term "baremetal" used in
datacenter which means "without hypervisors") or RTOSes.  Additionally,
for safety reasons, users often need to be able to configure the full
system statically so that it can be verified statically.

Event channels are very simple and can be added even to baremetal
applications. This proposal introduces a way to define them statically
to make them suitable to dom0less embedded deployments.
---


> +## Proposal:
> +
> +Event channels are the basic primitive provided by Xen for event notifications.
> +An event channel is a logical connection between 2 domains (more specifically
> +between dom1,port1 and dom2,port2). They essentially store one bit of
> +information, the event of interest is signalled by transitioning this bit from
> +0 to 1. An event is an equivalent of a hardware interrupt.
> +
> +Notifications are received by a guest via an interrupt from Xen to the guest,
> +indicating when an event arrives (setting the bit). Further notifications are
> +masked until the bit is cleared again. When a domain wants to wait for data it
> +will block until an event arrives, and then send an event to signal that data
> +has been consumed. Events are delivered asynchronously to guests and are
> +enqueued when the guest is not running.
> +
> +Event channel communication will be established statically between two domU
> +guests before unpausing the domains after domain creation. Event channel
> +connection information between domUs will be passed to XEN via device tree
> +node.
> +
> +Under the /chosen node, there needs to be sub nodes with compatible
> +"xen,evtchn" that descibes the event channel connection between two domUs.
> +
> +The event channel sub-node has the following properties:
> +
> +- compatible
> +
> +    "xen,evtchn"
> +
> +- xen,evtchn
> +
> +    The property is four numbers of tuples of
> +    (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle) where:
> +
> +    local-port-domU1 is an integer value that will be used to allocte local
> +    port for domU1 to send an event notification to the remote domain.
> +
> +    domU1-phandle is a single phandle to an domain to which local-port-domU1
> +    will be allocated.
> +
> +    local-port-domU2 is an integer value that will be used to allocte local
> +    port for domU2 to send an event notification to the remote domain.
> +
> +    domU2-phandle is a single phandle to an domain to which local-port-domU2
> +    will be allocated.
> +
> +Example:
> +
> +    chosen {
> +        ....
> +
> +        domU1: domU1 {
> +            ......
> +        };
> +
> +        domU2: domU2 {
> +            ......
> +        };
> +
> +        evtchn@1 {
> +            compatible = "xen,evtchn";
> +            xen,evtchn = <0xa &domU1 0xb &domU2>;
> +        };
> +
> +        evtchn@2 {
> +            compatible = "xen,evtchn";
> +            xen,evtchn = <0xc &domU1 0xd &domU2>;
> +        };
> +    };

There is no need to use two evtchn nodes for this given that in device
tree it is entirely normal to have multiple tuplets in a property. Also,
it would be good to have a version number in the compatible string so
that we can change version in the future.

1)
    chosen {
        ....

        domU1: domU1 {
            ......
        };

        domU2: domU2 {
            ......
        };

        evtchn {
            compatible = "xen,evtchn-v1";
            xen,evtchn = <0xa &domU1 0xb &domU2 0xc &domU1 0xd &domU2>;
        };
    };


I should mention that it would be also possible to use sub-nodes to
express this information:

2)
        domU1: domU1 {
            ...
            /* one sub-node per local event channel */
            ec1: evtchn@a {
                compatible = "xen,evtchn-v1";
                /* local-evtchn link-to-foreign-evtchn */
                xen,evtchn = <0xa &ec3>
            };
            ec2: evtchn@c {
                compatible = "xen,evtchn-v1";
                xen,evtchn = <0xc &ec4>
            };
        };

        domU2: domU2 {
            ...
            ec3: evtchn@b {
                compatible = "xen,evtchn-v1";
                xen,evtchn = <0xb &ec1>
            };
            ec4: evtchn@d {
                compatible = "xen,evtchn-v1";
                xen,evtchn = <0xa &ec2>
            };
        };
    };

This format has the advantage that doesn't need a new top-level node
type under /chosen. That is desirable few a few reasons. Today we only
have domains (dom0 is legacy). In the future we might have nested
domains and non-Xen domains. With System Device Tree, domains are under
/domains instead of /chosen.

So normally I would argue to use the sub-node format because it doesn't
need a new top-level node under /chosen. However, in this case it looks
like the 1) format is simpler to write and also simpler to parse in Xen.

In 1), we would need to loop over xen,evtchn and for each tuplet we
would only need to fetch the foreign domid.

In 2), we would need to check the compatible string of every
"xen,evtchn-v1" node, and we would have to fetch from the phandle both
the remote event channel number but also the domain-id of the parent.

So it looks like 1) is better because it is much simpler to parse. Do
you agree?


> +In above example two event channel comunication will be established between
> +domU1 and domU2.
> +
> +    domU1 (port 0xa) <-----------------> domU2 (port 0xb)
> +    domU1 (port 0xc) <-----------------> domU2 (port 0xd)
> +
> +domU1 and domU2 can send the signal to remote domain via hypercall
> +EVTCHNOP_send(.) on local port.

I think this is fine in principle. Like Jan wrote, at some point we'll
need to specify the device tree binding to expose this information to
the guest.

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 8207 bytes --]

On Fri, 8 Apr 2022, Stefano Stabellini wrote:
> On Wed, 23 Mar 2022, Rahul Singh wrote:
> > in dom0less system. This patch introduce the new feature to support the
> > signaling between two domUs in dom0less system.
> > 
> > Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> > ---
> >  docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
> >  1 file changed, 96 insertions(+)
> >  create mode 100644 docs/designs/dom0less-evtchn.md
> > 
> > diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
> > new file mode 100644
> > index 0000000000..6a1b7e8c22
> > --- /dev/null
> > +++ b/docs/designs/dom0less-evtchn.md
> > @@ -0,0 +1,96 @@
> > +# Signaling support between two domUs on dom0less system
> > +
> > +## Current state: Draft version
> > +
> > +## Proposer(s): Rahul Singh, Bertrand Marquis
> > +
> > +## Problem Statement:
> > +
> > +The goal of this work is to define a simple signaling system between Xen guests
> > +in dom0less systems.
> > +
> > +In dom0less system, we cannot make use of xenbus and xenstore that are used in
> > +normal systems with dynamic VMs to communicate between domains by providing a
> > +bus abstraction for paravirtualized drivers.
> > +
> > +One possible solution to implement the signaling system between domUs is based
> > +on event channels.
> 
> I suggest to reword this as follows:
> 
> ---
> Dom0less guests would benefit from a statically-defined memory sharing
> and signally system for communication. One that would be immediately
> available at boot without any need for dynamic configurations.
> 
> In embedded a great variety of guest operating system kernels exist,
> many of which don't have support for xenstore, grant table or other
> complex drivers. Some of them are small kernel-space applications (often
> called "baremetal", not to be confused with the term "baremetal" used in
> datacenter which means "without hypervisors") or RTOSes.  Additionally,
> for safety reasons, users often need to be able to configure the full
> system statically so that it can be verified statically.
> 
> Event channels are very simple and can be added even to baremetal
> applications. This proposal introduces a way to define them statically
> to make them suitable to dom0less embedded deployments.
> ---
> 
> 
> > +## Proposal:
> > +
> > +Event channels are the basic primitive provided by Xen for event notifications.
> > +An event channel is a logical connection between 2 domains (more specifically
> > +between dom1,port1 and dom2,port2). They essentially store one bit of
> > +information, the event of interest is signalled by transitioning this bit from
> > +0 to 1. An event is an equivalent of a hardware interrupt.
> > +
> > +Notifications are received by a guest via an interrupt from Xen to the guest,
> > +indicating when an event arrives (setting the bit). Further notifications are
> > +masked until the bit is cleared again. When a domain wants to wait for data it
> > +will block until an event arrives, and then send an event to signal that data
> > +has been consumed. Events are delivered asynchronously to guests and are
> > +enqueued when the guest is not running.
> > +
> > +Event channel communication will be established statically between two domU
> > +guests before unpausing the domains after domain creation. Event channel
> > +connection information between domUs will be passed to XEN via device tree
> > +node.
> > +
> > +Under the /chosen node, there needs to be sub nodes with compatible
> > +"xen,evtchn" that descibes the event channel connection between two domUs.
> > +
> > +The event channel sub-node has the following properties:
> > +
> > +- compatible
> > +
> > +    "xen,evtchn"
> > +
> > +- xen,evtchn
> > +
> > +    The property is four numbers of tuples of
> > +    (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle) where:
> > +
> > +    local-port-domU1 is an integer value that will be used to allocte local
> > +    port for domU1 to send an event notification to the remote domain.
> > +
> > +    domU1-phandle is a single phandle to an domain to which local-port-domU1
> > +    will be allocated.
> > +
> > +    local-port-domU2 is an integer value that will be used to allocte local
> > +    port for domU2 to send an event notification to the remote domain.
> > +
> > +    domU2-phandle is a single phandle to an domain to which local-port-domU2
> > +    will be allocated.
> > +
> > +Example:
> > +
> > +    chosen {
> > +        ....
> > +
> > +        domU1: domU1 {
> > +            ......
> > +        };
> > +
> > +        domU2: domU2 {
> > +            ......
> > +        };
> > +
> > +        evtchn@1 {
> > +            compatible = "xen,evtchn";
> > +            xen,evtchn = <0xa &domU1 0xb &domU2>;
> > +        };
> > +
> > +        evtchn@2 {
> > +            compatible = "xen,evtchn";
> > +            xen,evtchn = <0xc &domU1 0xd &domU2>;
> > +        };
> > +    };
> 
> There is no need to use two evtchn nodes for this given that in device
> tree it is entirely normal to have multiple tuplets in a property. Also,
> it would be good to have a version number in the compatible string so
> that we can change version in the future.
> 
> 1)
>     chosen {
>         ....
> 
>         domU1: domU1 {
>             ......
>         };
> 
>         domU2: domU2 {
>             ......
>         };
> 
>         evtchn {
>             compatible = "xen,evtchn-v1";
>             xen,evtchn = <0xa &domU1 0xb &domU2 0xc &domU1 0xd &domU2>;
>         };
>     };
> 
> 
> I should mention that it would be also possible to use sub-nodes to
> express this information:
> 
> 2)
>         domU1: domU1 {
>             ...
>             /* one sub-node per local event channel */
>             ec1: evtchn@a {
>                 compatible = "xen,evtchn-v1";
>                 /* local-evtchn link-to-foreign-evtchn */
>                 xen,evtchn = <0xa &ec3>
>             };
>             ec2: evtchn@c {
>                 compatible = "xen,evtchn-v1";
>                 xen,evtchn = <0xc &ec4>
>             };
>         };
> 
>         domU2: domU2 {
>             ...
>             ec3: evtchn@b {
>                 compatible = "xen,evtchn-v1";
>                 xen,evtchn = <0xb &ec1>
>             };
>             ec4: evtchn@d {
>                 compatible = "xen,evtchn-v1";
>                 xen,evtchn = <0xa &ec2>
>             };
>         };
>     };
> 
> This format has the advantage that doesn't need a new top-level node
> type under /chosen. That is desirable few a few reasons. Today we only
> have domains (dom0 is legacy). In the future we might have nested
> domains and non-Xen domains. With System Device Tree, domains are under
> /domains instead of /chosen.
> 
> So normally I would argue to use the sub-node format because it doesn't
> need a new top-level node under /chosen. However, in this case it looks
> like the 1) format is simpler to write and also simpler to parse in Xen.
> 
> In 1), we would need to loop over xen,evtchn and for each tuplet we
> would only need to fetch the foreign domid.
> 
> In 2), we would need to check the compatible string of every
> "xen,evtchn-v1" node, and we would have to fetch from the phandle both
> the remote event channel number but also the domain-id of the parent.
> 
> So it looks like 1) is better because it is much simpler to parse. Do
> you agree?

[...]
> 
> I think this is fine in principle. Like Jan wrote, at some point we'll
> need to specify the device tree binding to expose this information to
> the guest.

Actually, thinking more about it, I think it is likely that the guest
device tree bindings will include information about how it is supposed
to be used. For instance, the domU device tree might pair an event
channel with a shared memory region so that the domU knows that they are
expected to be used together.

If Xen is to generate such a device tree for guests, then we need that
information also on the host device tree too (the one given to Xen and
discussed here.)

So, I think it would be a good idea to discuss the domU device tree
bindings for this, as part of this design document, even if we don't
implement it straight away.

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Bertrand Marquis]

Hi Stefano,

> On 9 Apr 2022, at 02:44, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Fri, 8 Apr 2022, Stefano Stabellini wrote:
>> On Wed, 23 Mar 2022, Rahul Singh wrote:
>>> in dom0less system. This patch introduce the new feature to support the
>>> signaling between two domUs in dom0less system.
>>> 
>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>> ---
>>> docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>>> 1 file changed, 96 insertions(+)
>>> create mode 100644 docs/designs/dom0less-evtchn.md
>>> 
>>> diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
>>> new file mode 100644
>>> index 0000000000..6a1b7e8c22
>>> --- /dev/null
>>> +++ b/docs/designs/dom0less-evtchn.md
>>> @@ -0,0 +1,96 @@
>>> +# Signaling support between two domUs on dom0less system
>>> +
>>> +## Current state: Draft version
>>> +
>>> +## Proposer(s): Rahul Singh, Bertrand Marquis
>>> +
>>> +## Problem Statement:
>>> +
>>> +The goal of this work is to define a simple signaling system between Xen guests
>>> +in dom0less systems.
>>> +
>>> +In dom0less system, we cannot make use of xenbus and xenstore that are used in
>>> +normal systems with dynamic VMs to communicate between domains by providing a
>>> +bus abstraction for paravirtualized drivers.
>>> +
>>> +One possible solution to implement the signaling system between domUs is based
>>> +on event channels.
>> 
>> I suggest to reword this as follows:
>> 
>> ---
>> Dom0less guests would benefit from a statically-defined memory sharing
>> and signally system for communication. One that would be immediately
>> available at boot without any need for dynamic configurations.
>> 
>> In embedded a great variety of guest operating system kernels exist,
>> many of which don't have support for xenstore, grant table or other
>> complex drivers. Some of them are small kernel-space applications (often
>> called "baremetal", not to be confused with the term "baremetal" used in
>> datacenter which means "without hypervisors") or RTOSes.  Additionally,
>> for safety reasons, users often need to be able to configure the full
>> system statically so that it can be verified statically.
>> 
>> Event channels are very simple and can be added even to baremetal
>> applications. This proposal introduces a way to define them statically
>> to make them suitable to dom0less embedded deployments.
>> ---
>> 
>> 
>>> +## Proposal:
>>> +
>>> +Event channels are the basic primitive provided by Xen for event notifications.
>>> +An event channel is a logical connection between 2 domains (more specifically
>>> +between dom1,port1 and dom2,port2). They essentially store one bit of
>>> +information, the event of interest is signalled by transitioning this bit from
>>> +0 to 1. An event is an equivalent of a hardware interrupt.
>>> +
>>> +Notifications are received by a guest via an interrupt from Xen to the guest,
>>> +indicating when an event arrives (setting the bit). Further notifications are
>>> +masked until the bit is cleared again. When a domain wants to wait for data it
>>> +will block until an event arrives, and then send an event to signal that data
>>> +has been consumed. Events are delivered asynchronously to guests and are
>>> +enqueued when the guest is not running.
>>> +
>>> +Event channel communication will be established statically between two domU
>>> +guests before unpausing the domains after domain creation. Event channel
>>> +connection information between domUs will be passed to XEN via device tree
>>> +node.
>>> +
>>> +Under the /chosen node, there needs to be sub nodes with compatible
>>> +"xen,evtchn" that descibes the event channel connection between two domUs.
>>> +
>>> +The event channel sub-node has the following properties:
>>> +
>>> +- compatible
>>> +
>>> +    "xen,evtchn"
>>> +
>>> +- xen,evtchn
>>> +
>>> +    The property is four numbers of tuples of
>>> +    (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle) where:
>>> +
>>> +    local-port-domU1 is an integer value that will be used to allocte local
>>> +    port for domU1 to send an event notification to the remote domain.
>>> +
>>> +    domU1-phandle is a single phandle to an domain to which local-port-domU1
>>> +    will be allocated.
>>> +
>>> +    local-port-domU2 is an integer value that will be used to allocte local
>>> +    port for domU2 to send an event notification to the remote domain.
>>> +
>>> +    domU2-phandle is a single phandle to an domain to which local-port-domU2
>>> +    will be allocated.
>>> +
>>> +Example:
>>> +
>>> +    chosen {
>>> +        ....
>>> +
>>> +        domU1: domU1 {
>>> +            ......
>>> +        };
>>> +
>>> +        domU2: domU2 {
>>> +            ......
>>> +        };
>>> +
>>> +        evtchn@1 {
>>> +            compatible = "xen,evtchn";
>>> +            xen,evtchn = <0xa &domU1 0xb &domU2>;
>>> +        };
>>> +
>>> +        evtchn@2 {
>>> +            compatible = "xen,evtchn";
>>> +            xen,evtchn = <0xc &domU1 0xd &domU2>;
>>> +        };
>>> +    };
>> 
>> There is no need to use two evtchn nodes for this given that in device
>> tree it is entirely normal to have multiple tuplets in a property. Also,
>> it would be good to have a version number in the compatible string so
>> that we can change version in the future.
>> 
>> 1)
>>    chosen {
>>        ....
>> 
>>        domU1: domU1 {
>>            ......
>>        };
>> 
>>        domU2: domU2 {
>>            ......
>>        };
>> 
>>        evtchn {
>>            compatible = "xen,evtchn-v1";
>>            xen,evtchn = <0xa &domU1 0xb &domU2 0xc &domU1 0xd &domU2>;
>>        };
>>    };
>> 
>> 
>> I should mention that it would be also possible to use sub-nodes to
>> express this information:
>> 
>> 2)
>>        domU1: domU1 {
>>            ...
>>            /* one sub-node per local event channel */
>>            ec1: evtchn@a {
>>                compatible = "xen,evtchn-v1";
>>                /* local-evtchn link-to-foreign-evtchn */
>>                xen,evtchn = <0xa &ec3>
>>            };
>>            ec2: evtchn@c {
>>                compatible = "xen,evtchn-v1";
>>                xen,evtchn = <0xc &ec4>
>>            };
>>        };
>> 
>>        domU2: domU2 {
>>            ...
>>            ec3: evtchn@b {
>>                compatible = "xen,evtchn-v1";
>>                xen,evtchn = <0xb &ec1>
>>            };
>>            ec4: evtchn@d {
>>                compatible = "xen,evtchn-v1";
>>                xen,evtchn = <0xa &ec2>
>>            };
>>        };
>>    };
>> 
>> This format has the advantage that doesn't need a new top-level node
>> type under /chosen. That is desirable few a few reasons. Today we only
>> have domains (dom0 is legacy). In the future we might have nested
>> domains and non-Xen domains. With System Device Tree, domains are under
>> /domains instead of /chosen.
>> 
>> So normally I would argue to use the sub-node format because it doesn't
>> need a new top-level node under /chosen. However, in this case it looks
>> like the 1) format is simpler to write and also simpler to parse in Xen.
>> 
>> In 1), we would need to loop over xen,evtchn and for each tuplet we
>> would only need to fetch the foreign domid.
>> 
>> In 2), we would need to check the compatible string of every
>> "xen,evtchn-v1" node, and we would have to fetch from the phandle both
>> the remote event channel number but also the domain-id of the parent.
>> 
>> So it looks like 1) is better because it is much simpler to parse. Do
>> you agree?
> 
> [...]
>> 
>> I think this is fine in principle. Like Jan wrote, at some point we'll
>> need to specify the device tree binding to expose this information to
>> the guest.
> 
> Actually, thinking more about it, I think it is likely that the guest
> device tree bindings will include information about how it is supposed
> to be used. For instance, the domU device tree might pair an event
> channel with a shared memory region so that the domU knows that they are
> expected to be used together.
> 
> If Xen is to generate such a device tree for guests, then we need that
> information also on the host device tree too (the one given to Xen and
> discussed here.)
> 
> So, I think it would be a good idea to discuss the domU device tree
> bindings for this, as part of this design document, even if we don't
> implement it straight away.

What you mention here is actually combining 2 different solutions inside
Xen to build a custom communication solution.
My assumption here is that the user will actually create the device tree
nodes he wants to do that and we should not create guest node entries
as it would enforce some design.

If everything can be statically defined for Xen then the user can also
statically define node entries inside his guest to make use of the events
and the shared memories.

For example one might need more than one event to build a communication
system, or more than one shared memory or could build something
communicating with multiple guest thus requiring even more events and
shared memories.

Cheers
Bertrand

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Rahul Singh]

Hi Stefano,

Thanks for reviewing the design.

> On 9 Apr 2022, at 2:00 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Wed, 23 Mar 2022, Rahul Singh wrote:
>> in dom0less system. This patch introduce the new feature to support the
>> signaling between two domUs in dom0less system.
>> 
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>> 1 file changed, 96 insertions(+)
>> create mode 100644 docs/designs/dom0less-evtchn.md
>> 
>> diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
>> new file mode 100644
>> index 0000000000..6a1b7e8c22
>> --- /dev/null
>> +++ b/docs/designs/dom0less-evtchn.md
>> @@ -0,0 +1,96 @@
>> +# Signaling support between two domUs on dom0less system
>> +
>> +## Current state: Draft version
>> +
>> +## Proposer(s): Rahul Singh, Bertrand Marquis
>> +
>> +## Problem Statement:
>> +
>> +The goal of this work is to define a simple signaling system between Xen guests
>> +in dom0less systems.
>> +
>> +In dom0less system, we cannot make use of xenbus and xenstore that are used in
>> +normal systems with dynamic VMs to communicate between domains by providing a
>> +bus abstraction for paravirtualized drivers.
>> +
>> +One possible solution to implement the signaling system between domUs is based
>> +on event channels.
> 
> I suggest to reword this as follows:
> 
> ---
> Dom0less guests would benefit from a statically-defined memory sharing
> and signally system for communication. One that would be immediately
> available at boot without any need for dynamic configurations.
> 
> In embedded a great variety of guest operating system kernels exist,
> many of which don't have support for xenstore, grant table or other
> complex drivers. Some of them are small kernel-space applications (often
> called "baremetal", not to be confused with the term "baremetal" used in
> datacenter which means "without hypervisors") or RTOSes. Additionally,
> for safety reasons, users often need to be able to configure the full
> system statically so that it can be verified statically.
> 
> Event channels are very simple and can be added even to baremetal
> applications. This proposal introduces a way to define them statically
> to make them suitable to dom0less embedded deployments.
> ---
> 

Ok. This is really a good explanation I will add this in next version.
> 
>> +## Proposal:
>> +
>> +Event channels are the basic primitive provided by Xen for event notifications.
>> +An event channel is a logical connection between 2 domains (more specifically
>> +between dom1,port1 and dom2,port2). They essentially store one bit of
>> +information, the event of interest is signalled by transitioning this bit from
>> +0 to 1. An event is an equivalent of a hardware interrupt.
>> +
>> +Notifications are received by a guest via an interrupt from Xen to the guest,
>> +indicating when an event arrives (setting the bit). Further notifications are
>> +masked until the bit is cleared again. When a domain wants to wait for data it
>> +will block until an event arrives, and then send an event to signal that data
>> +has been consumed. Events are delivered asynchronously to guests and are
>> +enqueued when the guest is not running.
>> +
>> +Event channel communication will be established statically between two domU
>> +guests before unpausing the domains after domain creation. Event channel
>> +connection information between domUs will be passed to XEN via device tree
>> +node.
>> +
>> +Under the /chosen node, there needs to be sub nodes with compatible
>> +"xen,evtchn" that descibes the event channel connection between two domUs.
>> +
>> +The event channel sub-node has the following properties:
>> +
>> +- compatible
>> +
>> + "xen,evtchn"
>> +
>> +- xen,evtchn
>> +
>> + The property is four numbers of tuples of
>> + (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle) where:
>> +
>> + local-port-domU1 is an integer value that will be used to allocte local
>> + port for domU1 to send an event notification to the remote domain.
>> +
>> + domU1-phandle is a single phandle to an domain to which local-port-domU1
>> + will be allocated.
>> +
>> + local-port-domU2 is an integer value that will be used to allocte local
>> + port for domU2 to send an event notification to the remote domain.
>> +
>> + domU2-phandle is a single phandle to an domain to which local-port-domU2
>> + will be allocated.
>> +
>> +Example:
>> +
>> + chosen {
>> + ....
>> +
>> + domU1: domU1 {
>> + ......
>> + };
>> +
>> + domU2: domU2 {
>> + ......
>> + };
>> +
>> + evtchn@1 {
>> + compatible = "xen,evtchn";
>> + xen,evtchn = <0xa &domU1 0xb &domU2>;
>> + };
>> +
>> + evtchn@2 {
>> + compatible = "xen,evtchn";
>> + xen,evtchn = <0xc &domU1 0xd &domU2>;
>> + };
>> + };
> 
> There is no need to use two evtchn nodes for this given that in device
> tree it is entirely normal to have multiple tuplets in a property. Also,
> it would be good to have a version number in the compatible string so
> that we can change version in the future.
> 
> 1)
> chosen {
> ....
> 
> domU1: domU1 {
> ......
> };
> 
> domU2: domU2 {
> ......
> };
> 
> evtchn {
> compatible = "xen,evtchn-v1";
> xen,evtchn = <0xa &domU1 0xb &domU2 0xc &domU1 0xd &domU2>;
> };
> };
> 

I agree if we can have multiple tuples in a property. I will modify the design in next version
to have multiple tuples in a property. 
> 
> I should mention that it would be also possible to use sub-nodes to
> express this information:
> 
> 2)
> domU1: domU1 {
> ...
> /* one sub-node per local event channel */
> ec1: evtchn@a {
> compatible = "xen,evtchn-v1";
> /* local-evtchn link-to-foreign-evtchn */
> xen,evtchn = <0xa &ec3>
> };
> ec2: evtchn@c {
> compatible = "xen,evtchn-v1";
> xen,evtchn = <0xc &ec4>
> };
> };
> 
> domU2: domU2 {
> ...
> ec3: evtchn@b {
> compatible = "xen,evtchn-v1";
> xen,evtchn = <0xb &ec1>
> };
> ec4: evtchn@d {
> compatible = "xen,evtchn-v1";
> xen,evtchn = <0xa &ec2>
> };
> };
> };
> 
> This format has the advantage that doesn't need a new top-level node
> type under /chosen. That is desirable few a few reasons. Today we only
> have domains (dom0 is legacy). In the future we might have nested
> domains and non-Xen domains. With System Device Tree, domains are under
> /domains instead of /chosen.
> 
> So normally I would argue to use the sub-node format because it doesn't
> need a new top-level node under /chosen. However, in this case it looks
> like the 1) format is simpler to write and also simpler to parse in Xen.
> 
> In 1), we would need to loop over xen,evtchn and for each tuplet we
> would only need to fetch the foreign domid.
> 
> In 2), we would need to check the compatible string of every
> "xen,evtchn-v1" node, and we would have to fetch from the phandle both
> the remote event channel number but also the domain-id of the parent.

> 
> So it looks like 1) is better because it is much simpler to parse. Do
> you agree?

Yes I agree with you, for this case we need to parse all the "xen,evtchn-v1” compatible node
and from that node, we need to find the remote event channel and dom-id from the phandle.

I started from this configuration and later realize that if we use this configuration code will become
more complex and defining the event-channel connection in DT will also not be simple.
> 
>> +In above example two event channel comunication will be established between
>> +domU1 and domU2.
>> +
>> + domU1 (port 0xa) <-----------------> domU2 (port 0xb)
>> + domU1 (port 0xc) <-----------------> domU2 (port 0xd)
>> +
>> +domU1 and domU2 can send the signal to remote domain via hypercall
>> +EVTCHNOP_send(.) on local port.
> 
> I think this is fine in principle. Like Jan wrote, at some point we'll
> need to specify the device tree binding to expose this information to
> the guest.

Regards,
Rahul

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Rahul Singh]

Hello David,

Thanks for reviewing the design and sorry for the late reply. 

> On 24 Mar 2022, at 12:24 pm, David Vrabel <dvrabel@cantab.net> wrote:
> 
> 
> 
> On 23/03/2022 15:43, Rahul Singh wrote:
>> in dom0less system. This patch introduce the new feature to support the
>> signaling between two domUs in dom0less system.
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>> 1 file changed, 96 insertions(+)
>> create mode 100644 docs/designs/dom0less-evtchn.md
>> diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
>> new file mode 100644
>> index 0000000000..6a1b7e8c22
>> --- /dev/null
>> +++ b/docs/designs/dom0less-evtchn.md
>> @@ -0,0 +1,96 @@
>> +# Signaling support between two domUs on dom0less system
>> +
>> +## Current state: Draft version
>> +
>> +## Proposer(s): Rahul Singh, Bertrand Marquis
>> +
>> +## Problem Statement:
>> +
>> +The goal of this work is to define a simple signaling system between Xen guests
>> +in dom0less systems.
>> +
>> +In dom0less system, we cannot make use of xenbus and xenstore that are used in
>> +normal systems with dynamic VMs to communicate between domains by providing a
>> +bus abstraction for paravirtualized drivers.
>> +
>> +One possible solution to implement the signaling system between domUs is based
>> +on event channels.
> 
> This problem statement could do with some example use cases that are usefully solved by this proposed solution.
> 
> "We don't have xenstore so can't set up shared rings, but here's a replacement comms mechanism that can do a single bit." Doesn't seem very compelling to me.

Ok. Let me add more information in next version.
> 
>> + chosen {
>> + ....
>> +
>> + domU1: domU1 {
>> + ......
>> + };
>> +
>> + domU2: domU2 {
>> + ......
>> + };
>> +
>> + evtchn@1 {
>> + compatible = "xen,evtchn";
>> + xen,evtchn = <0xa &domU1 0xb &domU2>;
>> + };
>> +
>> + evtchn@2 {
>> + compatible = "xen,evtchn";
>> + xen,evtchn = <0xc &domU1 0xd &domU2>;
>> + };
> 
> How is the domain supposed to know what these event channels are for?

As we are statically defining the event channel in XEN, we can document the event
channel connection information for the end-user in the end-user documentation and
let the user decide how he is going to use it.  

> 
> I'm not that familiar with device tree. Is it possible to give these entries name?

As per the device-tree specification, each node in the device tree is named according to the following convention
	node-name@unit-address

We can give the name to these entries but in another email, we are discussing having singe node, in that
case there is no need to give a name.

evtchn {
	compatible = "xen,evtchn-v1”;
	xen,evtchn = <0xa &domU1 0xb &domU2 0xc &domU1 0xd &domU2>;
};

Regards.
Rahul
> 
> David

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Julien Grall]

Hi Rahul,

Title: s/../.../

On 23/03/2022 15:43, Rahul Singh wrote:
> in dom0less system. This patch introduce the new feature to support the

s/introduce/introduces/
s/the new/a/

> signaling between two domUs in dom0less system.
Did you intend to add a newline before the second sentence?

> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>   docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>   1 file changed, 96 insertions(+)
>   create mode 100644 docs/designs/dom0less-evtchn.md
> 
> diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
> new file mode 100644
> index 0000000000..6a1b7e8c22
> --- /dev/null
> +++ b/docs/designs/dom0less-evtchn.md
> @@ -0,0 +1,96 @@
> +# Signaling support between two domUs on dom0less system
> +
> +## Current state: Draft version
> +
> +## Proposer(s): Rahul Singh, Bertrand Marquis
> +
> +## Problem Statement:
> +
> +The goal of this work is to define a simple signaling system between Xen guests
> +in dom0less systems.
> +
> +In dom0less system, we cannot make use of xenbus and xenstore that are used in
> +normal systems with dynamic VMs to communicate between domains by providing a
> +bus abstraction for paravirtualized drivers.
> +
> +One possible solution to implement the signaling system between domUs is based
> +on event channels.
> +
> +## Proposal:
> +
> +Event channels are the basic primitive provided by Xen for event notifications.
> +An event channel is a logical connection between 2 domains (more specifically
> +between dom1,port1 and dom2,port2). They essentially store one bit of
> +information, the event of interest is signalled by transitioning this bit from
> +0 to 1. An event is an equivalent of a hardware interrupt.
> +
> +Notifications are received by a guest via an interrupt from Xen to the guest,
> +indicating when an event arrives (setting the bit). 

I am a bit confused with the description. Are you trying to explain the 
event channel in layman term? If not, then event channel protocol is 
more complicated than that (in particular for fifo).

> Further notifications are
> +masked until the bit is cleared again.

I think "masked" is confusing here.

The event channel differentiate "mask" vs "pending". When sending an 
event, the pending bit will be set to 1. If it wasn't already pending 
and the mask bit is clear, then we will notify the guest.

If the pending bit is already set, then we will ignore.

In fact, the event channel is acting similarly to an edge interrupt. I 
wrote similarly, because IIRC they are behaving slightly differently 
(see [1] for more details).

> When a domain wants to wait for data it
> +will block until an event arrives, and then send an event to signal that data
> +has been consumed.
> Events are delivered asynchronously to guests and are
> +enqueued when the guest is not running.

s/guest/domain/ to stay consistent and also include dom0/hardware domain.

> +
> +Event channel communication will be established statically between two domU
> +guests before unpausing the domains after domain creation. Event channel
> +connection information between domUs will be passed to XEN via device tree
> +node.

Why are we limiting ourself to domUs?

> +
> +Under the /chosen node, there needs to be sub nodes with compatible
> +"xen,evtchn" that descibes the event channel connection between two domUs.

s/descibes/describes/

> +
> +The event channel sub-node has the following properties:
> +
> +- compatible
> +
> +    "xen,evtchn"
> +
> +- xen,evtchn
> +
> +    The property is four numbers of tuples of
> +    (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle) where:

This is quite difficult to read. Can we add some space before/after each 
comma?

> +
> +    local-port-domU1 is an integer value that will be used to allocte local

s/allocte/allocate/

> +    port for domU1 to send an event notification to the remote domain.

The port will be used for sending but also receiving event notification.

Also, I would suggest to replace "remote domain" with "domU2". So it is 
more explicit.

> +
> +    domU1-phandle is a single phandle to an domain to which local-port-domU1

s/an domain/a domain/ I think.

> +    will be allocated.
> +
> +    local-port-domU2 is an integer value that will be used to allocte local

s/allocte/allocate/

> +    port for domU2 to send an event notification to the remote domain.

Same as above for "remote domain".

> +
> +    domU2-phandle is a single phandle to an domain to which local-port-domU2
> +    will be allocated.
> +
> +Example:
> +
> +    chosen {
> +        ....
> +
> +        domU1: domU1 {
> +            ......
> +        };
> +
> +        domU2: domU2 {
> +            ......
> +        };
> +
> +        evtchn@1 {
> +            compatible = "xen,evtchn";
> +            xen,evtchn = <0xa &domU1 0xb &domU2>;
> +        };
> +
> +        evtchn@2 {
> +            compatible = "xen,evtchn";
> +            xen,evtchn = <0xc &domU1 0xd &domU2>;
> +        };
> +    };
> +
> +In above example two event channel comunication will be established between
> +domU1 and domU2.
> +
> +    domU1 (port 0xa) <-----------------> domU2 (port 0xb)
> +    domU1 (port 0xc) <-----------------> domU2 (port 0xd)
> +
> +domU1 and domU2 can send the signal to remote domain via hypercall
> +EVTCHNOP_send(.) on local port.

Cheers,

[1] 
https://lore.kernel.org/lkml/dbfd87e9-48fc-f641-9e24-ddb6c4f61135@arm.com/

-- 
Julien Grall

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Julien Grall]

Hi,

On 09/04/2022 02:00, Stefano Stabellini wrote:
> On Wed, 23 Mar 2022, Rahul Singh wrote:
>> in dom0less system. This patch introduce the new feature to support the
>> signaling between two domUs in dom0less system.
>>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>>   docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>>   1 file changed, 96 insertions(+)
>>   create mode 100644 docs/designs/dom0less-evtchn.md
>>
>> diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
>> new file mode 100644
>> index 0000000000..6a1b7e8c22
>> --- /dev/null
>> +++ b/docs/designs/dom0less-evtchn.md
>> @@ -0,0 +1,96 @@
>> +# Signaling support between two domUs on dom0less system
>> +
>> +## Current state: Draft version
>> +
>> +## Proposer(s): Rahul Singh, Bertrand Marquis
>> +
>> +## Problem Statement:
>> +
>> +The goal of this work is to define a simple signaling system between Xen guests
>> +in dom0less systems.
>> +
>> +In dom0less system, we cannot make use of xenbus and xenstore that are used in
>> +normal systems with dynamic VMs to communicate between domains by providing a
>> +bus abstraction for paravirtualized drivers.
>> +
>> +One possible solution to implement the signaling system between domUs is based
>> +on event channels.
> 
> I suggest to reword this as follows:
> 
> ---
> Dom0less guests would benefit from a statically-defined memory sharing
> and signally system for communication. One that would be immediately
> available at boot without any need for dynamic configurations.
> 
> In embedded a great variety of guest operating system kernels exist,
> many of which don't have support for xenstore, grant table or other
> complex drivers. Some of them are small kernel-space applications (often
> called "baremetal", not to be confused with the term "baremetal" used in
> datacenter which means "without hypervisors") or RTOSes.  Additionally,
> for safety reasons, users often need to be able to configure the full
> system statically so that it can be verified statically.
> 
> Event channels are very simple and can be added even to baremetal
> applications. This proposal introduces a way to define them statically
> to make them suitable to dom0less embedded deployments.
> ---
> 
> 
>> +## Proposal:
>> +
>> +Event channels are the basic primitive provided by Xen for event notifications.
>> +An event channel is a logical connection between 2 domains (more specifically
>> +between dom1,port1 and dom2,port2). They essentially store one bit of
>> +information, the event of interest is signalled by transitioning this bit from
>> +0 to 1. An event is an equivalent of a hardware interrupt.
>> +
>> +Notifications are received by a guest via an interrupt from Xen to the guest,
>> +indicating when an event arrives (setting the bit). Further notifications are
>> +masked until the bit is cleared again. When a domain wants to wait for data it
>> +will block until an event arrives, and then send an event to signal that data
>> +has been consumed. Events are delivered asynchronously to guests and are
>> +enqueued when the guest is not running.
>> +
>> +Event channel communication will be established statically between two domU
>> +guests before unpausing the domains after domain creation. Event channel
>> +connection information between domUs will be passed to XEN via device tree
>> +node.
>> +
>> +Under the /chosen node, there needs to be sub nodes with compatible
>> +"xen,evtchn" that descibes the event channel connection between two domUs.
>> +
>> +The event channel sub-node has the following properties:
>> +
>> +- compatible
>> +
>> +    "xen,evtchn"
>> +
>> +- xen,evtchn
>> +
>> +    The property is four numbers of tuples of
>> +    (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle) where:
>> +
>> +    local-port-domU1 is an integer value that will be used to allocte local
>> +    port for domU1 to send an event notification to the remote domain.
>> +
>> +    domU1-phandle is a single phandle to an domain to which local-port-domU1
>> +    will be allocated.
>> +
>> +    local-port-domU2 is an integer value that will be used to allocte local
>> +    port for domU2 to send an event notification to the remote domain.
>> +
>> +    domU2-phandle is a single phandle to an domain to which local-port-domU2
>> +    will be allocated.
>> +
>> +Example:
>> +
>> +    chosen {
>> +        ....
>> +
>> +        domU1: domU1 {
>> +            ......
>> +        };
>> +
>> +        domU2: domU2 {
>> +            ......
>> +        };
>> +
>> +        evtchn@1 {
>> +            compatible = "xen,evtchn";
>> +            xen,evtchn = <0xa &domU1 0xb &domU2>;
>> +        };
>> +
>> +        evtchn@2 {
>> +            compatible = "xen,evtchn";
>> +            xen,evtchn = <0xc &domU1 0xd &domU2>;
>> +        };
>> +    };
> 
> There is no need to use two evtchn nodes for this given that in device
> tree it is entirely normal to have multiple tuplets in a property. Also,
> it would be good to have a version number in the compatible string so
> that we can change version in the future.
> 
> 1)
>      chosen {
>          ....
> 
>          domU1: domU1 {
>              ......
>          };
> 
>          domU2: domU2 {
>              ......
>          };
> 
>          evtchn {
>              compatible = "xen,evtchn-v1";
>              xen,evtchn = <0xa &domU1 0xb &domU2 0xc &domU1 0xd &domU2>;
>          };
>      };
> 
> 
> I should mention that it would be also possible to use sub-nodes to
> express this information:
> 
> 2)
>          domU1: domU1 {
>              ...
>              /* one sub-node per local event channel */
>              ec1: evtchn@a {
>                  compatible = "xen,evtchn-v1";
>                  /* local-evtchn link-to-foreign-evtchn */
>                  xen,evtchn = <0xa &ec3>
>              };
>              ec2: evtchn@c {
>                  compatible = "xen,evtchn-v1";
>                  xen,evtchn = <0xc &ec4>
>              };
>          };
> 
>          domU2: domU2 {
>              ...
>              ec3: evtchn@b {
>                  compatible = "xen,evtchn-v1";
>                  xen,evtchn = <0xb &ec1>
>              };
>              ec4: evtchn@d {
>                  compatible = "xen,evtchn-v1";
>                  xen,evtchn = <0xa &ec2>
>              };
>          };
>      };

As for 1), you could combine all the ports in one node.

> 
> This format has the advantage that doesn't need a new top-level node
> type under /chosen. That is desirable few a few reasons. Today we only
> have domains (dom0 is legacy). In the future we might have nested
> domains and non-Xen domains. With System Device Tree, domains are under
> /domains instead of /chosen.
> 
> So normally I would argue to use the sub-node format because it doesn't
> need a new top-level node under /chosen. However, in this case it looks
> like the 1) format is simpler to write and also simpler to parse in Xen.

I am not sure for both. For the writing part: In one hand, it is nice to 
have all the event channels defined in one place. One the other hand, it 
is more messy if you want to visually check that you don't duplicate the 
event channels.

It may also end up to be complex to read if you have many events 
channels. So if we go with 1), I think we would want to allow multiple 
nodes to help the user to keep a clean/readable DT.

> 
> In 1), we would need to loop over xen,evtchn and for each tuplet we
> would only need to fetch the foreign domid.

You will need to loop over all the nodes in chosen to find "xen,evtchn" 
and also fetch two phandles.

> 
> In 2), we would need to check the compatible string of every
> "xen,evtchn-v1" node, and we would have to fetch from the phandle both
> the remote event channel number but also the domain-id of the parent.
> 
> So it looks like 1) is better because it is much simpler to parse. Do
> you agree?

See above, I am not sure the code to parse will end up to be much bigger 
because we would still need to loop on the nodes in chosen and fetch two 
phandles.

That said, we are potentially going to need to loop on more nodes.

So overall, I am 50/50 on which one to chose.

Cheers,

-- 
Julien Grall

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 8055 bytes --]

On Mon, 11 Apr 2022, Julien Grall wrote:
> On 09/04/2022 02:00, Stefano Stabellini wrote:
> > On Wed, 23 Mar 2022, Rahul Singh wrote:
> > > in dom0less system. This patch introduce the new feature to support the
> > > signaling between two domUs in dom0less system.
> > > 
> > > Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> > > ---
> > >   docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
> > >   1 file changed, 96 insertions(+)
> > >   create mode 100644 docs/designs/dom0less-evtchn.md
> > > 
> > > diff --git a/docs/designs/dom0less-evtchn.md
> > > b/docs/designs/dom0less-evtchn.md
> > > new file mode 100644
> > > index 0000000000..6a1b7e8c22
> > > --- /dev/null
> > > +++ b/docs/designs/dom0less-evtchn.md
> > > @@ -0,0 +1,96 @@
> > > +# Signaling support between two domUs on dom0less system
> > > +
> > > +## Current state: Draft version
> > > +
> > > +## Proposer(s): Rahul Singh, Bertrand Marquis
> > > +
> > > +## Problem Statement:
> > > +
> > > +The goal of this work is to define a simple signaling system between Xen
> > > guests
> > > +in dom0less systems.
> > > +
> > > +In dom0less system, we cannot make use of xenbus and xenstore that are
> > > used in
> > > +normal systems with dynamic VMs to communicate between domains by
> > > providing a
> > > +bus abstraction for paravirtualized drivers.
> > > +
> > > +One possible solution to implement the signaling system between domUs is
> > > based
> > > +on event channels.
> > 
> > I suggest to reword this as follows:
> > 
> > ---
> > Dom0less guests would benefit from a statically-defined memory sharing
> > and signally system for communication. One that would be immediately
> > available at boot without any need for dynamic configurations.
> > 
> > In embedded a great variety of guest operating system kernels exist,
> > many of which don't have support for xenstore, grant table or other
> > complex drivers. Some of them are small kernel-space applications (often
> > called "baremetal", not to be confused with the term "baremetal" used in
> > datacenter which means "without hypervisors") or RTOSes.  Additionally,
> > for safety reasons, users often need to be able to configure the full
> > system statically so that it can be verified statically.
> > 
> > Event channels are very simple and can be added even to baremetal
> > applications. This proposal introduces a way to define them statically
> > to make them suitable to dom0less embedded deployments.
> > ---
> > 
> > 
> > > +## Proposal:
> > > +
> > > +Event channels are the basic primitive provided by Xen for event
> > > notifications.
> > > +An event channel is a logical connection between 2 domains (more
> > > specifically
> > > +between dom1,port1 and dom2,port2). They essentially store one bit of
> > > +information, the event of interest is signalled by transitioning this bit
> > > from
> > > +0 to 1. An event is an equivalent of a hardware interrupt.
> > > +
> > > +Notifications are received by a guest via an interrupt from Xen to the
> > > guest,
> > > +indicating when an event arrives (setting the bit). Further notifications
> > > are
> > > +masked until the bit is cleared again. When a domain wants to wait for
> > > data it
> > > +will block until an event arrives, and then send an event to signal that
> > > data
> > > +has been consumed. Events are delivered asynchronously to guests and are
> > > +enqueued when the guest is not running.
> > > +
> > > +Event channel communication will be established statically between two
> > > domU
> > > +guests before unpausing the domains after domain creation. Event channel
> > > +connection information between domUs will be passed to XEN via device
> > > tree
> > > +node.
> > > +
> > > +Under the /chosen node, there needs to be sub nodes with compatible
> > > +"xen,evtchn" that descibes the event channel connection between two
> > > domUs.
> > > +
> > > +The event channel sub-node has the following properties:
> > > +
> > > +- compatible
> > > +
> > > +    "xen,evtchn"
> > > +
> > > +- xen,evtchn
> > > +
> > > +    The property is four numbers of tuples of
> > > +    (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle)
> > > where:
> > > +
> > > +    local-port-domU1 is an integer value that will be used to allocte
> > > local
> > > +    port for domU1 to send an event notification to the remote domain.
> > > +
> > > +    domU1-phandle is a single phandle to an domain to which
> > > local-port-domU1
> > > +    will be allocated.
> > > +
> > > +    local-port-domU2 is an integer value that will be used to allocte
> > > local
> > > +    port for domU2 to send an event notification to the remote domain.
> > > +
> > > +    domU2-phandle is a single phandle to an domain to which
> > > local-port-domU2
> > > +    will be allocated.
> > > +
> > > +Example:
> > > +
> > > +    chosen {
> > > +        ....
> > > +
> > > +        domU1: domU1 {
> > > +            ......
> > > +        };
> > > +
> > > +        domU2: domU2 {
> > > +            ......
> > > +        };
> > > +
> > > +        evtchn@1 {
> > > +            compatible = "xen,evtchn";
> > > +            xen,evtchn = <0xa &domU1 0xb &domU2>;
> > > +        };
> > > +
> > > +        evtchn@2 {
> > > +            compatible = "xen,evtchn";
> > > +            xen,evtchn = <0xc &domU1 0xd &domU2>;
> > > +        };
> > > +    };
> > 
> > There is no need to use two evtchn nodes for this given that in device
> > tree it is entirely normal to have multiple tuplets in a property. Also,
> > it would be good to have a version number in the compatible string so
> > that we can change version in the future.
> > 
> > 1)
> >      chosen {
> >          ....
> > 
> >          domU1: domU1 {
> >              ......
> >          };
> > 
> >          domU2: domU2 {
> >              ......
> >          };
> > 
> >          evtchn {
> >              compatible = "xen,evtchn-v1";
> >              xen,evtchn = <0xa &domU1 0xb &domU2 0xc &domU1 0xd &domU2>;
> >          };
> >      };
> > 
> > 
> > I should mention that it would be also possible to use sub-nodes to
> > express this information:
> > 
> > 2)
> >          domU1: domU1 {
> >              ...
> >              /* one sub-node per local event channel */
> >              ec1: evtchn@a {
> >                  compatible = "xen,evtchn-v1";
> >                  /* local-evtchn link-to-foreign-evtchn */
> >                  xen,evtchn = <0xa &ec3>
> >              };
> >              ec2: evtchn@c {
> >                  compatible = "xen,evtchn-v1";
> >                  xen,evtchn = <0xc &ec4>
> >              };
> >          };
> > 
> >          domU2: domU2 {
> >              ...
> >              ec3: evtchn@b {
> >                  compatible = "xen,evtchn-v1";
> >                  xen,evtchn = <0xb &ec1>
> >              };
> >              ec4: evtchn@d {
> >                  compatible = "xen,evtchn-v1";
> >                  xen,evtchn = <0xd &ec2>
> >              };
> >          };
> >      };
> 
> As for 1), you could combine all the ports in one node.

I thought about it but I couldn't come up with a way to do that which
retains the simplicity of this example. The problem is that in device
tree you can only link to nodes, not to individual properties. So I
think we would have to have separate nodes for each event channel so
that we could separately link to them. 

Otherwise, we would have to add the foreign event channel number in
addition to the link to be able to distinguish them. And that would
result in duplicated information. E.g.:

       domU1: domU1 {
           ...
           /* one sub-node per local event channel */
           ec1: evtchn@a {
               compatible = "xen,evtchn-v1";
               /* local-evtchn link-to-foreign foreign-evtchn */
               xen,evtchn = <0xa &ec2 0xa 0xc &ec2 0xd>
           };
       };

       domU2: domU2 {
           ...
           ec2: evtchn@b {
               compatible = "xen,evtchn-v1";
               xen,evtchn = <0xb &ec1 0xa 0xd &ec1 0xc>
           };
       };

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Rahul Singh]

Hello Julien,

Thanks for reviewing the design.

> On 11 Apr 2022, at 7:01 pm, Julien Grall <julien@xen.org> wrote:
> 
> Hi Rahul,
> 
> Title: s/../.../
> 
> On 23/03/2022 15:43, Rahul Singh wrote:
>> in dom0less system. This patch introduce the new feature to support the
> 
> s/introduce/introduces/
> s/the new/a/

Ack. 
> 
>> signaling between two domUs in dom0less system.
> Did you intend to add a newline before the second sentence?

No. I will fix this.

> 
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>> 1 file changed, 96 insertions(+)
>> create mode 100644 docs/designs/dom0less-evtchn.md
>> diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
>> new file mode 100644
>> index 0000000000..6a1b7e8c22
>> --- /dev/null
>> +++ b/docs/designs/dom0less-evtchn.md
>> @@ -0,0 +1,96 @@
>> +# Signaling support between two domUs on dom0less system
>> +
>> +## Current state: Draft version
>> +
>> +## Proposer(s): Rahul Singh, Bertrand Marquis
>> +
>> +## Problem Statement:
>> +
>> +The goal of this work is to define a simple signaling system between Xen guests
>> +in dom0less systems.
>> +
>> +In dom0less system, we cannot make use of xenbus and xenstore that are used in
>> +normal systems with dynamic VMs to communicate between domains by providing a
>> +bus abstraction for paravirtualized drivers.
>> +
>> +One possible solution to implement the signaling system between domUs is based
>> +on event channels.
>> +
>> +## Proposal:
>> +
>> +Event channels are the basic primitive provided by Xen for event notifications.
>> +An event channel is a logical connection between 2 domains (more specifically
>> +between dom1,port1 and dom2,port2). They essentially store one bit of
>> +information, the event of interest is signalled by transitioning this bit from
>> +0 to 1. An event is an equivalent of a hardware interrupt.
>> +
>> +Notifications are received by a guest via an interrupt from Xen to the guest,
>> +indicating when an event arrives (setting the bit). 
> 
> I am a bit confused with the description. Are you trying to explain the event channel in layman term? If not, then event channel protocol is more complicated than that (in particular for fifo).

Yes I am trying to explain the event-channel in simple term. 
> 
>> Further notifications are
>> +masked until the bit is cleared again.
> 
> I think "masked" is confusing here.
> 
> The event channel differentiate "mask" vs "pending". When sending an event, the pending bit will be set to 1. If it wasn't already pending and the mask bit is clear, then we will notify the guest.
> 
> If the pending bit is already set, then we will ignore.
> 
> In fact, the event channel is acting similarly to an edge interrupt. I wrote similarly, because IIRC they are behaving slightly differently (see [1] for more details).

Ok. Let me modify the sentence in next version.
> 
>> When a domain wants to wait for data it
>> +will block until an event arrives, and then send an event to signal that data
>> +has been consumed.
>> Events are delivered asynchronously to guests and are
>> +enqueued when the guest is not running.
> 
> s/guest/domain/ to stay consistent and also include dom0/hardware domain.
Ack.
> 
>> +
>> +Event channel communication will be established statically between two domU
>> +guests before unpausing the domains after domain creation. Event channel
>> +connection information between domUs will be passed to XEN via device tree
>> +node.
> 
> Why are we limiting ourself to domUs?

As this design is for a dom0less system I mean here to all the domains on the dom0less system.
What I understand is that all domains in the dom0less system are called as domUs.
Please correct me if I am wrong.

> 
>> +
>> +Under the /chosen node, there needs to be sub nodes with compatible
>> +"xen,evtchn" that descibes the event channel connection between two domUs.
> 
> s/descibes/describes/

Ack. 

> 
>> +
>> +The event channel sub-node has the following properties:
>> +
>> +- compatible
>> +
>> + "xen,evtchn"
>> +
>> +- xen,evtchn
>> +
>> + The property is four numbers of tuples of
>> + (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle) where:

> This is quite difficult to read. Can we add some space before/after each comma?

Ack. 
> 
>> +
>> + local-port-domU1 is an integer value that will be used to allocte local
> 
> s/allocte/allocate/

Ack. 
> 
>> + port for domU1 to send an event notification to the remote domain.
> 
> The port will be used for sending but also receiving event notification.

Yes. I will modify.
> 
> Also, I would suggest to replace "remote domain" with "domU2". So it is more explicit.

Ack. 
> 
>> +
>> + domU1-phandle is a single phandle to an domain to which local-port-domU1
> 
> s/an domain/a domain/ I think.

Ack. 
> 
>> + will be allocated.
>> +
>> + local-port-domU2 is an integer value that will be used to allocte local
> 
> s/allocte/allocate/

Ack. 
> 
>> + port for domU2 to send an event notification to the remote domain.
> 
> Same as above for "remote domain".

Ack.  

Regards,
Rahul

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Julien Grall]

Hi Rahul,

On 13/04/2022 09:25, Rahul Singh wrote:
>> On 11 Apr 2022, at 7:01 pm, Julien Grall <julien@xen.org> wrote:
>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>> ---
>>> docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>>> 1 file changed, 96 insertions(+)
>>> create mode 100644 docs/designs/dom0less-evtchn.md
>>> diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
>>> new file mode 100644
>>> index 0000000000..6a1b7e8c22
>>> --- /dev/null
>>> +++ b/docs/designs/dom0less-evtchn.md
>>> @@ -0,0 +1,96 @@
>>> +# Signaling support between two domUs on dom0less system
>>> +
>>> +## Current state: Draft version
>>> +
>>> +## Proposer(s): Rahul Singh, Bertrand Marquis
>>> +
>>> +## Problem Statement:
>>> +
>>> +The goal of this work is to define a simple signaling system between Xen guests
>>> +in dom0less systems.
>>> +
>>> +In dom0less system, we cannot make use of xenbus and xenstore that are used in
>>> +normal systems with dynamic VMs to communicate between domains by providing a
>>> +bus abstraction for paravirtualized drivers.
>>> +
>>> +One possible solution to implement the signaling system between domUs is based
>>> +on event channels.
>>> +
>>> +## Proposal:
>>> +
>>> +Event channels are the basic primitive provided by Xen for event notifications.
>>> +An event channel is a logical connection between 2 domains (more specifically
>>> +between dom1,port1 and dom2,port2). They essentially store one bit of
>>> +information, the event of interest is signalled by transitioning this bit from
>>> +0 to 1. An event is an equivalent of a hardware interrupt.
>>> +
>>> +Notifications are received by a guest via an interrupt from Xen to the guest,
>>> +indicating when an event arrives (setting the bit).
>>
>> I am a bit confused with the description. Are you trying to explain the event channel in layman term? If not, then event channel protocol is more complicated than that (in particular for fifo).
> 
> Yes I am trying to explain the event-channel in simple term.

I would suggest to make that clear and also point to the documentation 
for the Event Channel ABI (we have a doc for fifo at least).

>>> +
>>> +Event channel communication will be established statically between two domU
>>> +guests before unpausing the domains after domain creation. Event channel
>>> +connection information between domUs will be passed to XEN via device tree
>>> +node.
>>
>> Why are we limiting ourself to domUs?
> 
> As this design is for a dom0less system I mean here to all the domains on the dom0less system.
> What I understand is that all domains in the dom0less system are called as domUs.

It depends on whether an admin may have specific a dom0 kernel. Looking 
at Penny series to handle shared memory, it will be possible to create a 
shared region between dom0 and a dom0less domU. Most likely the user 
will also want to provide a notification communication.

So shouldn't we also provide a way to create an event channel between 
dom0 and another domU?

Cheers,

-- 
Julien Grall

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Julien Grall]

Hi Stefano,

On 12/04/2022 21:39, Stefano Stabellini wrote:
>>> I should mention that it would be also possible to use sub-nodes to
>>> express this information:
>>>
>>> 2)
>>>           domU1: domU1 {
>>>               ...
>>>               /* one sub-node per local event channel */
>>>               ec1: evtchn@a {
>>>                   compatible = "xen,evtchn-v1";
>>>                   /* local-evtchn link-to-foreign-evtchn */
>>>                   xen,evtchn = <0xa &ec3>
>>>               };
>>>               ec2: evtchn@c {
>>>                   compatible = "xen,evtchn-v1";
>>>                   xen,evtchn = <0xc &ec4>
>>>               };
>>>           };
>>>
>>>           domU2: domU2 {
>>>               ...
>>>               ec3: evtchn@b {
>>>                   compatible = "xen,evtchn-v1";
>>>                   xen,evtchn = <0xb &ec1>
>>>               };
>>>               ec4: evtchn@d {
>>>                   compatible = "xen,evtchn-v1";
>>>                   xen,evtchn = <0xd &ec2>
>>>               };
>>>           };
>>>       };
>>
>> As for 1), you could combine all the ports in one node.
> 
> I thought about it but I couldn't come up with a way to do that which
> retains the simplicity of this example. The problem is that in device
> tree you can only link to nodes, not to individual properties. So I
> think we would have to have separate nodes for each event channel so
> that we could separately link to them.

Ah yes. I overlooked it when making the suggestion. That said, I think 
my point in my previous e-mail stands.

If this is too verbose, then we could provide macro to generate the 
event channel node.

> 
> Otherwise, we would have to add the foreign event channel number in
> addition to the link to be able to distinguish them. And that would
> result in duplicated information. E.g.:

This is not indeed not great.

> 
>         domU1: domU1 {
>             ...
>             /* one sub-node per local event channel */
>             ec1: evtchn@a {
>                 compatible = "xen,evtchn-v1";
>                 /* local-evtchn link-to-foreign foreign-evtchn */
>                 xen,evtchn = <0xa &ec2 0xa 0xc &ec2 0xd>
>             };
>         };
> 
>         domU2: domU2 {
>             ...
>             ec2: evtchn@b {
>                 compatible = "xen,evtchn-v1";
>                 xen,evtchn = <0xb &ec1 0xa 0xd &ec1 0xc>
>             };
>         };

Cheers,

-- 
Julien Grall

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Rahul Singh]

Hello Julien,

> On 13 Apr 2022, at 10:44 am, Julien Grall <julien@xen.org> wrote:
> 
> Hi Rahul,
> 
> On 13/04/2022 09:25, Rahul Singh wrote:
>>> On 11 Apr 2022, at 7:01 pm, Julien Grall <julien@xen.org> wrote:
>>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>>> ---
>>>> docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>>>> 1 file changed, 96 insertions(+)
>>>> create mode 100644 docs/designs/dom0less-evtchn.md
>>>> diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
>>>> new file mode 100644
>>>> index 0000000000..6a1b7e8c22
>>>> --- /dev/null
>>>> +++ b/docs/designs/dom0less-evtchn.md
>>>> @@ -0,0 +1,96 @@
>>>> +# Signaling support between two domUs on dom0less system
>>>> +
>>>> +## Current state: Draft version
>>>> +
>>>> +## Proposer(s): Rahul Singh, Bertrand Marquis
>>>> +
>>>> +## Problem Statement:
>>>> +
>>>> +The goal of this work is to define a simple signaling system between Xen guests
>>>> +in dom0less systems.
>>>> +
>>>> +In dom0less system, we cannot make use of xenbus and xenstore that are used in
>>>> +normal systems with dynamic VMs to communicate between domains by providing a
>>>> +bus abstraction for paravirtualized drivers.
>>>> +
>>>> +One possible solution to implement the signaling system between domUs is based
>>>> +on event channels.
>>>> +
>>>> +## Proposal:
>>>> +
>>>> +Event channels are the basic primitive provided by Xen for event notifications.
>>>> +An event channel is a logical connection between 2 domains (more specifically
>>>> +between dom1,port1 and dom2,port2). They essentially store one bit of
>>>> +information, the event of interest is signalled by transitioning this bit from
>>>> +0 to 1. An event is an equivalent of a hardware interrupt.
>>>> +
>>>> +Notifications are received by a guest via an interrupt from Xen to the guest,
>>>> +indicating when an event arrives (setting the bit).
>>> 
>>> I am a bit confused with the description. Are you trying to explain the event channel in layman term? If not, then event channel protocol is more complicated than that (in particular for fifo).
>> Yes I am trying to explain the event-channel in simple term.
> 
> I would suggest to make that clear and also point to the documentation for the Event Channel ABI (we have a doc for fifo at least).

Ack. 
> 
>>>> +
>>>> +Event channel communication will be established statically between two domU
>>>> +guests before unpausing the domains after domain creation. Event channel
>>>> +connection information between domUs will be passed to XEN via device tree
>>>> +node.
>>> 
>>> Why are we limiting ourself to domUs?
>> As this design is for a dom0less system I mean here to all the domains on the dom0less system.
>> What I understand is that all domains in the dom0less system are called as domUs.
> 
> It depends on whether an admin may have specific a dom0 kernel. Looking at Penny series to handle shared memory, it will be possible to create a shared region between dom0 and a dom0less domU. Most likely the user will also want to provide a notification communication.
> 
> So shouldn't we also provide a way to create an event channel between dom0 and another domU?

Yes, we can create the event channel between dom0 and dom0less domU. I will add this in the next design doc.

Sorry for not explaining it correctly earlier I mean to say that all domains ( including the dom0 ) in the dom0less system can
have the static event channel.  

Regards,
Rahul

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Stefano Stabellini]

On Mon, 11 Apr 2022, Bertrand Marquis wrote:
> What you mention here is actually combining 2 different solutions inside
> Xen to build a custom communication solution.
> My assumption here is that the user will actually create the device tree
> nodes he wants to do that and we should not create guest node entries
> as it would enforce some design.
> 
> If everything can be statically defined for Xen then the user can also
> statically define node entries inside his guest to make use of the events
> and the shared memories.
> 
> For example one might need more than one event to build a communication
> system, or more than one shared memory or could build something
> communicating with multiple guest thus requiring even more events and
> shared memories.

Hi Bertrand, Rahul,

If the guests are allowed some level of dynamic discovery, this feature
is not needed. They can discover the shared memory location from the
domU device tree, then proceed to allocate evtchns as needed and tell
the other end the evtchn numbers over shared memory. I already have an
example of it here:

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/2251030537/Xen+Shared+Memory+and+Interrupts+Between+VMs

What if the guest doesn't support device tree at runtime, like baremetal
or Zephyr? The shared memory address can be hardcoded or generated from
device tree at build time. That's no problem. Then, the event channels
can still be allocated at runtime and passed to the other end over
shared memory. That's what the example on the wikipage does.


When are static event channels actually useful? When the application
cannot allocate the event channels at runtime at all. The reason for the
restriction could be related to safety (no dynamic allocations at
runtime) or convenience (everything else is fully static, why should the
event channel numbers be dynamic?)

Given the above, I can see why there is no need to describe the static
event channel info in the domU device tree: static event channels are
only useful in fully static configurations, and in those configurations
the domU device tree dynamically generated by Xen is not needed. I can
see where you are coming from.


The workflow that we have been trying to enable with the System Device
Tree effort (System Device Tree is similar to a normal Device Tree plus
the xen,domains nodes) is the following:

S-DT ---[lopper]---> Linux DT
                L--> Zephyr DT ---[Zephyr build]---> Zephyr .h files

S-DT contains all the needed information for both the regular Linux DT
generation and also the Zephyr/RTOS/baremetal header files generation,
that happens at build time.

S-DT is not the same as the Xen device tree, but so far it has been
conceptually and practically similar. I always imagine that the bindings
we have in Xen we'll also have corresponding bindings in System Device
Tree.

For this workflow to work S-DT needs all the info so that both Linux DT
and Zephyr DT and Zephyr .h files can be generated.

Does this proposal contain enough information so that Zephyr .h files
could be statically generated with the event channel numbers and static
shared memory regions addresses?

I am not sure. Maybe not?


It is possible that the shared memory usage is so application specific
that there is no point in even talking about it. But I think that
introducing a simple bundle of both event channels and shared memory
would help a lot.

Something like the following in the Xen device tree would be enough to
specify an arbitrary number of event channels connected with the same
domains sharing the memory region.

It looks like that if we did the below, we would carry a lot more useful
information compared to the original proposal alone. We could add a
similar xen,notificaiton property to the domU reserved-memory region in
device tree generated by Xen for consistency, so that everything
available to the domU is described fully in device tree.


    domU1 {
        compatible = "xen,domain";

        /* one sub-node per local event channel */
        ec1: evtchn@1 {
            compatible = "xen,evtchn-v1";
            /* local-evtchn link-to-foreign-evtchn */
            xen,evtchn = <0x1 &ec3>
        };
        ec2: evtchn@2 {
            compatible = "xen,evtchn-v1";
            xen,evtchn = <0x2 &ec4>
        };
        /*
         * shared memory region between DomU1 and DomU2.
         */
        domU1-shared-mem@50000000 {
            compatible = "xen,domain-shared-memory-v1";
            xen,shm-id = <0x1>;
            xen,shared-mem = <0x50000000 0x20000000 0x60000000>;
            /* this is new */
            xen,notification = <&ec1 &ec2>;
        }
    };

    domU2 {
        compatible = "xen,domain";

        /* one sub-node per local event channel */
        ec3: evtchn@3 {
            compatible = "xen,evtchn-v1";
            /* local-evtchn link-to-foreign-evtchn */
            xen,evtchn = <0x3 &ec1>
        };
        ec4: evtchn@4 {
            compatible = "xen,evtchn-v1";
            xen,evtchn = <0x4 &ec2>
        };
        /*
         * shared memory region between domU1 and domU2.
         */
        domU2-shared-mem@50000000 {
            compatible = "xen,domain-shared-memory-v1";
            xen,shm-id = <0x1>;
            xen,shared-mem = <0x50000000 0x20000000 0x70000000>;
            /* this is new */
            xen,notification = <&ec3 &ec4>;
        }
    };



The good thing about this is that:

- it is very flexible
- nothing to do in this series, except switching to the
  one-subnode-per-evtchn model, which we called 2) in the previous email
- there were good reasons to use the one-subnode-per-evtchn model anyway
- the xen,notification property can be added later without issues, after Penny's series

There are a couple of ways to implement the xen,notification property
but we don't need to discuss them now.


Short Summary
------------
I think it is fine to only introduce the Xen device tree binding for
static event channels without domU binding, but I prefer if we switched
to using proposal 2) "one subnode per event channel".

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Bertrand Marquis]

Hi Stefano,

> On 14 Apr 2022, at 02:14, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Mon, 11 Apr 2022, Bertrand Marquis wrote:
>> What you mention here is actually combining 2 different solutions inside
>> Xen to build a custom communication solution.
>> My assumption here is that the user will actually create the device tree
>> nodes he wants to do that and we should not create guest node entries
>> as it would enforce some design.
>> 
>> If everything can be statically defined for Xen then the user can also
>> statically define node entries inside his guest to make use of the events
>> and the shared memories.
>> 
>> For example one might need more than one event to build a communication
>> system, or more than one shared memory or could build something
>> communicating with multiple guest thus requiring even more events and
>> shared memories.
> 
> Hi Bertrand, Rahul,
> 
> If the guests are allowed some level of dynamic discovery, this feature
> is not needed. They can discover the shared memory location from the
> domU device tree, then proceed to allocate evtchns as needed and tell
> the other end the evtchn numbers over shared memory. I already have an
> example of it here:
> 
> https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/2251030537/Xen+Shared+Memory+and+Interrupts+Between+VMs
> 
> What if the guest doesn't support device tree at runtime, like baremetal
> or Zephyr? The shared memory address can be hardcoded or generated from
> device tree at build time. That's no problem. Then, the event channels
> can still be allocated at runtime and passed to the other end over
> shared memory. That's what the example on the wikipage does.
> 
> 
> When are static event channels actually useful? When the application
> cannot allocate the event channels at runtime at all. The reason for the
> restriction could be related to safety (no dynamic allocations at
> runtime) or convenience (everything else is fully static, why should the
> event channel numbers be dynamic?)

An other use case here is dom0less: you cannot have dom0 create them.

> 
> Given the above, I can see why there is no need to describe the static
> event channel info in the domU device tree: static event channels are
> only useful in fully static configurations, and in those configurations
> the domU device tree dynamically generated by Xen is not needed. I can
> see where you are coming from.
> 
> 
> The workflow that we have been trying to enable with the System Device
> Tree effort (System Device Tree is similar to a normal Device Tree plus
> the xen,domains nodes) is the following:
> 
> S-DT ---[lopper]---> Linux DT
>                L--> Zephyr DT ---[Zephyr build]---> Zephyr .h files
> 
> S-DT contains all the needed information for both the regular Linux DT
> generation and also the Zephyr/RTOS/baremetal header files generation,
> that happens at build time.
> 
> S-DT is not the same as the Xen device tree, but so far it has been
> conceptually and practically similar. I always imagine that the bindings
> we have in Xen we'll also have corresponding bindings in System Device
> Tree.
> 
> For this workflow to work S-DT needs all the info so that both Linux DT
> and Zephyr DT and Zephyr .h files can be generated.
> 
> Does this proposal contain enough information so that Zephyr .h files
> could be statically generated with the event channel numbers and static
> shared memory regions addresses?
> 
> I am not sure. Maybe not?

Yes it should be possible to have all infos as the integrator will setup the
system and will decide upfront the address and the event(s) number(s).

> 
> 
> It is possible that the shared memory usage is so application specific
> that there is no point in even talking about it. But I think that
> introducing a simple bundle of both event channels and shared memory
> would help a lot.
> 
> Something like the following in the Xen device tree would be enough to
> specify an arbitrary number of event channels connected with the same
> domains sharing the memory region.
> 
> It looks like that if we did the below, we would carry a lot more useful
> information compared to the original proposal alone. We could add a
> similar xen,notificaiton property to the domU reserved-memory region in
> device tree generated by Xen for consistency, so that everything
> available to the domU is described fully in device tree.
> 
> 
>    domU1 {
>        compatible = "xen,domain";
> 
>        /* one sub-node per local event channel */
>        ec1: evtchn@1 {
>            compatible = "xen,evtchn-v1";
>            /* local-evtchn link-to-foreign-evtchn */
>            xen,evtchn = <0x1 &ec3>
>        };
>        ec2: evtchn@2 {
>            compatible = "xen,evtchn-v1";
>            xen,evtchn = <0x2 &ec4>
>        };
>        /*
>         * shared memory region between DomU1 and DomU2.
>         */
>        domU1-shared-mem@50000000 {
>            compatible = "xen,domain-shared-memory-v1";
>            xen,shm-id = <0x1>;
>            xen,shared-mem = <0x50000000 0x20000000 0x60000000>;
>            /* this is new */
>            xen,notification = <&ec1 &ec2>;
>        }
>    };
> 
>    domU2 {
>        compatible = "xen,domain";
> 
>        /* one sub-node per local event channel */
>        ec3: evtchn@3 {
>            compatible = "xen,evtchn-v1";
>            /* local-evtchn link-to-foreign-evtchn */
>            xen,evtchn = <0x3 &ec1>
>        };
>        ec4: evtchn@4 {
>            compatible = "xen,evtchn-v1";
>            xen,evtchn = <0x4 &ec2>
>        };
>        /*
>         * shared memory region between domU1 and domU2.
>         */
>        domU2-shared-mem@50000000 {
>            compatible = "xen,domain-shared-memory-v1";
>            xen,shm-id = <0x1>;
>            xen,shared-mem = <0x50000000 0x20000000 0x70000000>;
>            /* this is new */
>            xen,notification = <&ec3 &ec4>;
>        }
>    };

Few remarks/questions on this:
- this is not a shared memory anymore as you add a notification system to it
- what if someone wants to use only a shared memory, or an event, what should xen do ?
- in xen device tree, how do you associate the event with the shared memory ?

> 
> 
> 
> The good thing about this is that:
> 
> - it is very flexible
> - nothing to do in this series, except switching to the
>  one-subnode-per-evtchn model, which we called 2) in the previous email
> - there were good reasons to use the one-subnode-per-evtchn model anyway
> - the xen,notification property can be added later without issues, after Penny's series
> 
> There are a couple of ways to implement the xen,notification property
> but we don't need to discuss them now.

I think there is something to do here but we need a bit more discussion and this can be done later.
Right now I am not quite sure we will not add something that will end up not being used.

> 
> 
> Short Summary
> ------------
> I think it is fine to only introduce the Xen device tree binding for
> static event channels without domU binding, but I prefer if we switched
> to using proposal 2) "one subnode per event channel".

I will let Rahul answer on that.

Cheers
Bertrand

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Stefano Stabellini]

On Thu, 14 Apr 2022, Bertrand Marquis wrote:
> > On 14 Apr 2022, at 02:14, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > 
> > On Mon, 11 Apr 2022, Bertrand Marquis wrote:
> >> What you mention here is actually combining 2 different solutions inside
> >> Xen to build a custom communication solution.
> >> My assumption here is that the user will actually create the device tree
> >> nodes he wants to do that and we should not create guest node entries
> >> as it would enforce some design.
> >> 
> >> If everything can be statically defined for Xen then the user can also
> >> statically define node entries inside his guest to make use of the events
> >> and the shared memories.
> >> 
> >> For example one might need more than one event to build a communication
> >> system, or more than one shared memory or could build something
> >> communicating with multiple guest thus requiring even more events and
> >> shared memories.
> > 
> > Hi Bertrand, Rahul,
> > 
> > If the guests are allowed some level of dynamic discovery, this feature
> > is not needed. They can discover the shared memory location from the
> > domU device tree, then proceed to allocate evtchns as needed and tell
> > the other end the evtchn numbers over shared memory. I already have an
> > example of it here:
> > 
> > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/2251030537/Xen+Shared+Memory+and+Interrupts+Between+VMs
> > 
> > What if the guest doesn't support device tree at runtime, like baremetal
> > or Zephyr? The shared memory address can be hardcoded or generated from
> > device tree at build time. That's no problem. Then, the event channels
> > can still be allocated at runtime and passed to the other end over
> > shared memory. That's what the example on the wikipage does.
> > 
> > 
> > When are static event channels actually useful? When the application
> > cannot allocate the event channels at runtime at all. The reason for the
> > restriction could be related to safety (no dynamic allocations at
> > runtime) or convenience (everything else is fully static, why should the
> > event channel numbers be dynamic?)
> 
> An other use case here is dom0less: you cannot have dom0 create them.
> 
> > 
> > Given the above, I can see why there is no need to describe the static
> > event channel info in the domU device tree: static event channels are
> > only useful in fully static configurations, and in those configurations
> > the domU device tree dynamically generated by Xen is not needed. I can
> > see where you are coming from.
> > 
> > 
> > The workflow that we have been trying to enable with the System Device
> > Tree effort (System Device Tree is similar to a normal Device Tree plus
> > the xen,domains nodes) is the following:
> > 
> > S-DT ---[lopper]---> Linux DT
> >                L--> Zephyr DT ---[Zephyr build]---> Zephyr .h files
> > 
> > S-DT contains all the needed information for both the regular Linux DT
> > generation and also the Zephyr/RTOS/baremetal header files generation,
> > that happens at build time.
> > 
> > S-DT is not the same as the Xen device tree, but so far it has been
> > conceptually and practically similar. I always imagine that the bindings
> > we have in Xen we'll also have corresponding bindings in System Device
> > Tree.
> > 
> > For this workflow to work S-DT needs all the info so that both Linux DT
> > and Zephyr DT and Zephyr .h files can be generated.
> > 
> > Does this proposal contain enough information so that Zephyr .h files
> > could be statically generated with the event channel numbers and static
> > shared memory regions addresses?
> > 
> > I am not sure. Maybe not?
> 
> Yes it should be possible to have all infos as the integrator will setup the
> system and will decide upfront the address and the event(s) number(s).
> 
> > 
> > 
> > It is possible that the shared memory usage is so application specific
> > that there is no point in even talking about it. But I think that
> > introducing a simple bundle of both event channels and shared memory
> > would help a lot.
> > 
> > Something like the following in the Xen device tree would be enough to
> > specify an arbitrary number of event channels connected with the same
> > domains sharing the memory region.
> > 
> > It looks like that if we did the below, we would carry a lot more useful
> > information compared to the original proposal alone. We could add a
> > similar xen,notificaiton property to the domU reserved-memory region in
> > device tree generated by Xen for consistency, so that everything
> > available to the domU is described fully in device tree.
> > 
> > 
> >    domU1 {
> >        compatible = "xen,domain";
> > 
> >        /* one sub-node per local event channel */
> >        ec1: evtchn@1 {
> >            compatible = "xen,evtchn-v1";
> >            /* local-evtchn link-to-foreign-evtchn */
> >            xen,evtchn = <0x1 &ec3>
> >        };
> >        ec2: evtchn@2 {
> >            compatible = "xen,evtchn-v1";
> >            xen,evtchn = <0x2 &ec4>
> >        };
> >        /*
> >         * shared memory region between DomU1 and DomU2.
> >         */
> >        domU1-shared-mem@50000000 {
> >            compatible = "xen,domain-shared-memory-v1";
> >            xen,shm-id = <0x1>;
> >            xen,shared-mem = <0x50000000 0x20000000 0x60000000>;
> >            /* this is new */
> >            xen,notification = <&ec1 &ec2>;
> >        }
> >    };
> > 
> >    domU2 {
> >        compatible = "xen,domain";
> > 
> >        /* one sub-node per local event channel */
> >        ec3: evtchn@3 {
> >            compatible = "xen,evtchn-v1";
> >            /* local-evtchn link-to-foreign-evtchn */
> >            xen,evtchn = <0x3 &ec1>
> >        };
> >        ec4: evtchn@4 {
> >            compatible = "xen,evtchn-v1";
> >            xen,evtchn = <0x4 &ec2>
> >        };
> >        /*
> >         * shared memory region between domU1 and domU2.
> >         */
> >        domU2-shared-mem@50000000 {
> >            compatible = "xen,domain-shared-memory-v1";
> >            xen,shm-id = <0x1>;
> >            xen,shared-mem = <0x50000000 0x20000000 0x70000000>;
> >            /* this is new */
> >            xen,notification = <&ec3 &ec4>;
> >        }
> >    };
> 
> Few remarks/questions on this:
> - this is not a shared memory anymore as you add a notification system to it
> - what if someone wants to use only a shared memory, or an event, what should xen do ?

They still can. xen,notification would only be an optional property, not
a mandatory property. So it is still possible to have shared memory
without notifications (skip the xen,notification property), or event
channels without shared memory (do not link the evtchn to
xen,notification).


> - in xen device tree, how do you associate the event with the shared memory ?

I don't think I understand the question. The example above shows how to
associate the event with the shared memory: the only additional thing
needed (compared to proposal 2 already discussed) is the new optional
property xen,notification.

Xen itself wouldn't have to do anything special when xen,notification is
specified, but would add a similar optional xen,notification property to
the generated domU device tree.


> > The good thing about this is that:
> > 
> > - it is very flexible
> > - nothing to do in this series, except switching to the
> >  one-subnode-per-evtchn model, which we called 2) in the previous email
> > - there were good reasons to use the one-subnode-per-evtchn model anyway
> > - the xen,notification property can be added later without issues, after Penny's series
> > 
> > There are a couple of ways to implement the xen,notification property
> > but we don't need to discuss them now.
> 
> I think there is something to do here but we need a bit more discussion and this can be done later.
> Right now I am not quite sure we will not add something that will end up not being used.
 
Yes, I am not asking to add xen,notification now, neither to the Xen
device tree or the domU device tree. I am only trying to make sure it
would be possible do something like it without major changes to the
existing device tree. And I think it is possible if we use proposal 2).
 

> > Short Summary
> > ------------
> > I think it is fine to only introduce the Xen device tree binding for
> > static event channels without domU binding, but I prefer if we switched
> > to using proposal 2) "one subnode per event channel".
> 
> I will let Rahul answer on that.

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Rahul Singh]

Hello Stefano,

> On 14 Apr 2022, at 2:14 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Mon, 11 Apr 2022, Bertrand Marquis wrote:
>> What you mention here is actually combining 2 different solutions inside
>> Xen to build a custom communication solution.
>> My assumption here is that the user will actually create the device tree
>> nodes he wants to do that and we should not create guest node entries
>> as it would enforce some design.
>> 
>> If everything can be statically defined for Xen then the user can also
>> statically define node entries inside his guest to make use of the events
>> and the shared memories.
>> 
>> For example one might need more than one event to build a communication
>> system, or more than one shared memory or could build something
>> communicating with multiple guest thus requiring even more events and
>> shared memories.
> 
> Hi Bertrand, Rahul,
> 
> If the guests are allowed some level of dynamic discovery, this feature
> is not needed. They can discover the shared memory location from the
> domU device tree, then proceed to allocate evtchns as needed and tell
> the other end the evtchn numbers over shared memory. I already have an
> example of it here:
> 
> https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/2251030537/Xen+Shared+Memory+and+Interrupts+Between+VMs
> 
> What if the guest doesn't support device tree at runtime, like baremetal
> or Zephyr? The shared memory address can be hardcoded or generated from
> device tree at build time. That's no problem. Then, the event channels
> can still be allocated at runtime and passed to the other end over
> shared memory. That's what the example on the wikipage does.
> 
> 
> When are static event channels actually useful? When the application
> cannot allocate the event channels at runtime at all. The reason for the
> restriction could be related to safety (no dynamic allocations at
> runtime) or convenience (everything else is fully static, why should the
> event channel numbers be dynamic?)
> 
> Given the above, I can see why there is no need to describe the static
> event channel info in the domU device tree: static event channels are
> only useful in fully static configurations, and in those configurations
> the domU device tree dynamically generated by Xen is not needed. I can
> see where you are coming from.
> 
> 
> The workflow that we have been trying to enable with the System Device
> Tree effort (System Device Tree is similar to a normal Device Tree plus
> the xen,domains nodes) is the following:
> 
> S-DT ---[lopper]---> Linux DT
>                L--> Zephyr DT ---[Zephyr build]---> Zephyr .h files
> 
> S-DT contains all the needed information for both the regular Linux DT
> generation and also the Zephyr/RTOS/baremetal header files generation,
> that happens at build time.
> 
> S-DT is not the same as the Xen device tree, but so far it has been
> conceptually and practically similar. I always imagine that the bindings
> we have in Xen we'll also have corresponding bindings in System Device
> Tree.
> 
> For this workflow to work S-DT needs all the info so that both Linux DT
> and Zephyr DT and Zephyr .h files can be generated.
> 
> Does this proposal contain enough information so that Zephyr .h files
> could be statically generated with the event channel numbers and static
> shared memory regions addresses?
> 
> I am not sure. Maybe not?
> 
> 
> It is possible that the shared memory usage is so application specific
> that there is no point in even talking about it. But I think that
> introducing a simple bundle of both event channels and shared memory
> would help a lot.
> 
> Something like the following in the Xen device tree would be enough to
> specify an arbitrary number of event channels connected with the same
> domains sharing the memory region.
> 
> It looks like that if we did the below, we would carry a lot more useful
> information compared to the original proposal alone. We could add a
> similar xen,notificaiton property to the domU reserved-memory region in
> device tree generated by Xen for consistency, so that everything
> available to the domU is described fully in device tree.
> 
> 
>    domU1 {
>        compatible = "xen,domain";
> 
>        /* one sub-node per local event channel */
>        ec1: evtchn@1 {
>            compatible = "xen,evtchn-v1";
>            /* local-evtchn link-to-foreign-evtchn */
>            xen,evtchn = <0x1 &ec3>
>        };
>        ec2: evtchn@2 {
>            compatible = "xen,evtchn-v1";
>            xen,evtchn = <0x2 &ec4>
>        };
>        /*
>         * shared memory region between DomU1 and DomU2.
>         */
>        domU1-shared-mem@50000000 {
>            compatible = "xen,domain-shared-memory-v1";
>            xen,shm-id = <0x1>;
>            xen,shared-mem = <0x50000000 0x20000000 0x60000000>;
>            /* this is new */
>            xen,notification = <&ec1 &ec2>;
>        }
>    };
> 
>    domU2 {
>        compatible = "xen,domain";
> 
>        /* one sub-node per local event channel */
>        ec3: evtchn@3 {
>            compatible = "xen,evtchn-v1";
>            /* local-evtchn link-to-foreign-evtchn */
>            xen,evtchn = <0x3 &ec1>
>        };
>        ec4: evtchn@4 {
>            compatible = "xen,evtchn-v1";
>            xen,evtchn = <0x4 &ec2>
>        };
>        /*
>         * shared memory region between domU1 and domU2.
>         */
>        domU2-shared-mem@50000000 {
>            compatible = "xen,domain-shared-memory-v1";
>            xen,shm-id = <0x1>;
>            xen,shared-mem = <0x50000000 0x20000000 0x70000000>;
>            /* this is new */
>            xen,notification = <&ec3 &ec4>;
>        }
>    };
> 
> 
> 
> The good thing about this is that:
> 
> - it is very flexible
> - nothing to do in this series, except switching to the
>  one-subnode-per-evtchn model, which we called 2) in the previous email
> - there were good reasons to use the one-subnode-per-evtchn model anyway
> - the xen,notification property can be added later without issues, after Penny's series
> 
> There are a couple of ways to implement the xen,notification property
> but we don't need to discuss them now.
> 
> 
> Short Summary
> ------------
> I think it is fine to only introduce the Xen device tree binding for
> static event channels without domU binding, but I prefer if we switched
> to using proposal 2) "one subnode per event channel".

Thanks for detailed explanation.I just did the basic implementation of the 2) proposal and
it looks simple and not very complex as I thought earlier.Therefore I am ok to switch to proposal 2).

If everyone is ok with the 2) proposal I will send the v2 of design doc after fixing all the comments.

Regards,
Rahul

Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Stefano Stabellini]

On Tue, 19 Apr 2022, Rahul Singh wrote:
> > On 14 Apr 2022, at 2:14 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > 
> > On Mon, 11 Apr 2022, Bertrand Marquis wrote:
> >> What you mention here is actually combining 2 different solutions inside
> >> Xen to build a custom communication solution.
> >> My assumption here is that the user will actually create the device tree
> >> nodes he wants to do that and we should not create guest node entries
> >> as it would enforce some design.
> >> 
> >> If everything can be statically defined for Xen then the user can also
> >> statically define node entries inside his guest to make use of the events
> >> and the shared memories.
> >> 
> >> For example one might need more than one event to build a communication
> >> system, or more than one shared memory or could build something
> >> communicating with multiple guest thus requiring even more events and
> >> shared memories.
> > 
> > Hi Bertrand, Rahul,
> > 
> > If the guests are allowed some level of dynamic discovery, this feature
> > is not needed. They can discover the shared memory location from the
> > domU device tree, then proceed to allocate evtchns as needed and tell
> > the other end the evtchn numbers over shared memory. I already have an
> > example of it here:
> > 
> > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/2251030537/Xen+Shared+Memory+and+Interrupts+Between+VMs
> > 
> > What if the guest doesn't support device tree at runtime, like baremetal
> > or Zephyr? The shared memory address can be hardcoded or generated from
> > device tree at build time. That's no problem. Then, the event channels
> > can still be allocated at runtime and passed to the other end over
> > shared memory. That's what the example on the wikipage does.
> > 
> > 
> > When are static event channels actually useful? When the application
> > cannot allocate the event channels at runtime at all. The reason for the
> > restriction could be related to safety (no dynamic allocations at
> > runtime) or convenience (everything else is fully static, why should the
> > event channel numbers be dynamic?)
> > 
> > Given the above, I can see why there is no need to describe the static
> > event channel info in the domU device tree: static event channels are
> > only useful in fully static configurations, and in those configurations
> > the domU device tree dynamically generated by Xen is not needed. I can
> > see where you are coming from.
> > 
> > 
> > The workflow that we have been trying to enable with the System Device
> > Tree effort (System Device Tree is similar to a normal Device Tree plus
> > the xen,domains nodes) is the following:
> > 
> > S-DT ---[lopper]---> Linux DT
> >                L--> Zephyr DT ---[Zephyr build]---> Zephyr .h files
> > 
> > S-DT contains all the needed information for both the regular Linux DT
> > generation and also the Zephyr/RTOS/baremetal header files generation,
> > that happens at build time.
> > 
> > S-DT is not the same as the Xen device tree, but so far it has been
> > conceptually and practically similar. I always imagine that the bindings
> > we have in Xen we'll also have corresponding bindings in System Device
> > Tree.
> > 
> > For this workflow to work S-DT needs all the info so that both Linux DT
> > and Zephyr DT and Zephyr .h files can be generated.
> > 
> > Does this proposal contain enough information so that Zephyr .h files
> > could be statically generated with the event channel numbers and static
> > shared memory regions addresses?
> > 
> > I am not sure. Maybe not?
> > 
> > 
> > It is possible that the shared memory usage is so application specific
> > that there is no point in even talking about it. But I think that
> > introducing a simple bundle of both event channels and shared memory
> > would help a lot.
> > 
> > Something like the following in the Xen device tree would be enough to
> > specify an arbitrary number of event channels connected with the same
> > domains sharing the memory region.
> > 
> > It looks like that if we did the below, we would carry a lot more useful
> > information compared to the original proposal alone. We could add a
> > similar xen,notificaiton property to the domU reserved-memory region in
> > device tree generated by Xen for consistency, so that everything
> > available to the domU is described fully in device tree.
> > 
> > 
> >    domU1 {
> >        compatible = "xen,domain";
> > 
> >        /* one sub-node per local event channel */
> >        ec1: evtchn@1 {
> >            compatible = "xen,evtchn-v1";
> >            /* local-evtchn link-to-foreign-evtchn */
> >            xen,evtchn = <0x1 &ec3>
> >        };
> >        ec2: evtchn@2 {
> >            compatible = "xen,evtchn-v1";
> >            xen,evtchn = <0x2 &ec4>
> >        };
> >        /*
> >         * shared memory region between DomU1 and DomU2.
> >         */
> >        domU1-shared-mem@50000000 {
> >            compatible = "xen,domain-shared-memory-v1";
> >            xen,shm-id = <0x1>;
> >            xen,shared-mem = <0x50000000 0x20000000 0x60000000>;
> >            /* this is new */
> >            xen,notification = <&ec1 &ec2>;
> >        }
> >    };
> > 
> >    domU2 {
> >        compatible = "xen,domain";
> > 
> >        /* one sub-node per local event channel */
> >        ec3: evtchn@3 {
> >            compatible = "xen,evtchn-v1";
> >            /* local-evtchn link-to-foreign-evtchn */
> >            xen,evtchn = <0x3 &ec1>
> >        };
> >        ec4: evtchn@4 {
> >            compatible = "xen,evtchn-v1";
> >            xen,evtchn = <0x4 &ec2>
> >        };
> >        /*
> >         * shared memory region between domU1 and domU2.
> >         */
> >        domU2-shared-mem@50000000 {
> >            compatible = "xen,domain-shared-memory-v1";
> >            xen,shm-id = <0x1>;
> >            xen,shared-mem = <0x50000000 0x20000000 0x70000000>;
> >            /* this is new */
> >            xen,notification = <&ec3 &ec4>;
> >        }
> >    };
> > 
> > 
> > 
> > The good thing about this is that:
> > 
> > - it is very flexible
> > - nothing to do in this series, except switching to the
> >  one-subnode-per-evtchn model, which we called 2) in the previous email
> > - there were good reasons to use the one-subnode-per-evtchn model anyway
> > - the xen,notification property can be added later without issues, after Penny's series
> > 
> > There are a couple of ways to implement the xen,notification property
> > but we don't need to discuss them now.
> > 
> > 
> > Short Summary
> > ------------
> > I think it is fine to only introduce the Xen device tree binding for
> > static event channels without domU binding, but I prefer if we switched
> > to using proposal 2) "one subnode per event channel".
> 
> Thanks for detailed explanation.I just did the basic implementation of the 2) proposal and
> it looks simple and not very complex as I thought earlier.Therefore I am ok to switch to proposal 2).
> 
> If everyone is ok with the 2) proposal I will send the v2 of design doc after fixing all the comments.

That's great, thank you Rahul!