Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Rahul Singh <Rahul.Singh@arm.com>]

Hi Stefano,

Thanks for reviewing the design.

> On 9 Apr 2022, at 2:00 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Wed, 23 Mar 2022, Rahul Singh wrote:
>> in dom0less system. This patch introduce the new feature to support the
>> signaling between two domUs in dom0less system.
>> 
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>> 1 file changed, 96 insertions(+)
>> create mode 100644 docs/designs/dom0less-evtchn.md
>> 
>> diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
>> new file mode 100644
>> index 0000000000..6a1b7e8c22
>> --- /dev/null
>> +++ b/docs/designs/dom0less-evtchn.md
>> @@ -0,0 +1,96 @@
>> +# Signaling support between two domUs on dom0less system
>> +
>> +## Current state: Draft version
>> +
>> +## Proposer(s): Rahul Singh, Bertrand Marquis
>> +
>> +## Problem Statement:
>> +
>> +The goal of this work is to define a simple signaling system between Xen guests
>> +in dom0less systems.
>> +
>> +In dom0less system, we cannot make use of xenbus and xenstore that are used in
>> +normal systems with dynamic VMs to communicate between domains by providing a
>> +bus abstraction for paravirtualized drivers.
>> +
>> +One possible solution to implement the signaling system between domUs is based
>> +on event channels.
> 
> I suggest to reword this as follows:
> 
> ---
> Dom0less guests would benefit from a statically-defined memory sharing
> and signally system for communication. One that would be immediately
> available at boot without any need for dynamic configurations.
> 
> In embedded a great variety of guest operating system kernels exist,
> many of which don't have support for xenstore, grant table or other
> complex drivers. Some of them are small kernel-space applications (often
> called "baremetal", not to be confused with the term "baremetal" used in
> datacenter which means "without hypervisors") or RTOSes. Additionally,
> for safety reasons, users often need to be able to configure the full
> system statically so that it can be verified statically.
> 
> Event channels are very simple and can be added even to baremetal
> applications. This proposal introduces a way to define them statically
> to make them suitable to dom0less embedded deployments.
> ---
> 

Ok. This is really a good explanation I will add this in next version.
> 
>> +## Proposal:
>> +
>> +Event channels are the basic primitive provided by Xen for event notifications.
>> +An event channel is a logical connection between 2 domains (more specifically
>> +between dom1,port1 and dom2,port2). They essentially store one bit of
>> +information, the event of interest is signalled by transitioning this bit from
>> +0 to 1. An event is an equivalent of a hardware interrupt.
>> +
>> +Notifications are received by a guest via an interrupt from Xen to the guest,
>> +indicating when an event arrives (setting the bit). Further notifications are
>> +masked until the bit is cleared again. When a domain wants to wait for data it
>> +will block until an event arrives, and then send an event to signal that data
>> +has been consumed. Events are delivered asynchronously to guests and are
>> +enqueued when the guest is not running.
>> +
>> +Event channel communication will be established statically between two domU
>> +guests before unpausing the domains after domain creation. Event channel
>> +connection information between domUs will be passed to XEN via device tree
>> +node.
>> +
>> +Under the /chosen node, there needs to be sub nodes with compatible
>> +"xen,evtchn" that descibes the event channel connection between two domUs.
>> +
>> +The event channel sub-node has the following properties:
>> +
>> +- compatible
>> +
>> + "xen,evtchn"
>> +
>> +- xen,evtchn
>> +
>> + The property is four numbers of tuples of
>> + (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle) where:
>> +
>> + local-port-domU1 is an integer value that will be used to allocte local
>> + port for domU1 to send an event notification to the remote domain.
>> +
>> + domU1-phandle is a single phandle to an domain to which local-port-domU1
>> + will be allocated.
>> +
>> + local-port-domU2 is an integer value that will be used to allocte local
>> + port for domU2 to send an event notification to the remote domain.
>> +
>> + domU2-phandle is a single phandle to an domain to which local-port-domU2
>> + will be allocated.
>> +
>> +Example:
>> +
>> + chosen {
>> + ....
>> +
>> + domU1: domU1 {
>> + ......
>> + };
>> +
>> + domU2: domU2 {
>> + ......
>> + };
>> +
>> + evtchn@1 {
>> + compatible = "xen,evtchn";
>> + xen,evtchn = <0xa &domU1 0xb &domU2>;
>> + };
>> +
>> + evtchn@2 {
>> + compatible = "xen,evtchn";
>> + xen,evtchn = <0xc &domU1 0xd &domU2>;
>> + };
>> + };
> 
> There is no need to use two evtchn nodes for this given that in device
> tree it is entirely normal to have multiple tuplets in a property. Also,
> it would be good to have a version number in the compatible string so
> that we can change version in the future.
> 
> 1)
> chosen {
> ....
> 
> domU1: domU1 {
> ......
> };
> 
> domU2: domU2 {
> ......
> };
> 
> evtchn {
> compatible = "xen,evtchn-v1";
> xen,evtchn = <0xa &domU1 0xb &domU2 0xc &domU1 0xd &domU2>;
> };
> };
> 

I agree if we can have multiple tuples in a property. I will modify the design in next version
to have multiple tuples in a property. 
> 
> I should mention that it would be also possible to use sub-nodes to
> express this information:
> 
> 2)
> domU1: domU1 {
> ...
> /* one sub-node per local event channel */
> ec1: evtchn@a {
> compatible = "xen,evtchn-v1";
> /* local-evtchn link-to-foreign-evtchn */
> xen,evtchn = <0xa &ec3>
> };
> ec2: evtchn@c {
> compatible = "xen,evtchn-v1";
> xen,evtchn = <0xc &ec4>
> };
> };
> 
> domU2: domU2 {
> ...
> ec3: evtchn@b {
> compatible = "xen,evtchn-v1";
> xen,evtchn = <0xb &ec1>
> };
> ec4: evtchn@d {
> compatible = "xen,evtchn-v1";
> xen,evtchn = <0xa &ec2>
> };
> };
> };
> 
> This format has the advantage that doesn't need a new top-level node
> type under /chosen. That is desirable few a few reasons. Today we only
> have domains (dom0 is legacy). In the future we might have nested
> domains and non-Xen domains. With System Device Tree, domains are under
> /domains instead of /chosen.
> 
> So normally I would argue to use the sub-node format because it doesn't
> need a new top-level node under /chosen. However, in this case it looks
> like the 1) format is simpler to write and also simpler to parse in Xen.
> 
> In 1), we would need to loop over xen,evtchn and for each tuplet we
> would only need to fetch the foreign domid.
> 
> In 2), we would need to check the compatible string of every
> "xen,evtchn-v1" node, and we would have to fetch from the phandle both
> the remote event channel number but also the domain-id of the parent.

> 
> So it looks like 1) is better because it is much simpler to parse. Do
> you agree?

Yes I agree with you, for this case we need to parse all the "xen,evtchn-v1” compatible node
and from that node, we need to find the remote event channel and dom-id from the phandle.

I started from this configuration and later realize that if we use this configuration code will become
more complex and defining the event-channel connection in DT will also not be simple.
> 
>> +In above example two event channel comunication will be established between
>> +domU1 and domU2.
>> +
>> + domU1 (port 0xa) <-----------------> domU2 (port 0xb)
>> + domU1 (port 0xc) <-----------------> domU2 (port 0xd)
>> +
>> +domU1 and domU2 can send the signal to remote domain via hypercall
>> +EVTCHNOP_send(.) on local port.
> 
> I think this is fine in principle. Like Jan wrote, at some point we'll
> need to specify the device tree binding to expose this information to
> the guest.

Regards,
Rahul