[RFC PATCH] xen/design: Add design for EFI dom0less system start

[RFC PATCH] xen/design: Add design for EFI dom0less system start

[Luca Fancellu]

Add a design describing a proposal to improve the EFI
configuration file, adding keywords to describe domU
guests and allowing to start a dom0less system.

Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
---
 docs/designs/efi-arm-dom0less.md | 105 +++++++++++++++++++++++++++++++
 1 file changed, 105 insertions(+)
 create mode 100644 docs/designs/efi-arm-dom0less.md

diff --git a/docs/designs/efi-arm-dom0less.md b/docs/designs/efi-arm-dom0less.md
new file mode 100644
index 0000000000..8d8fa2243f
--- /dev/null
+++ b/docs/designs/efi-arm-dom0less.md
@@ -0,0 +1,105 @@
+# Xen EFI configuration file
+
+The current configuration file used by Xen when it is started as an EFI
+application is considering only the dom0 guest and doesn't have any
+property to describe and load in memory domU guests.
+Hence currently it's impossible to start a dom0less system using EFI.
+
+# Objective
+
+This document describes the proposed improvement to the Xen EFI
+configuration file to list properly both the dom0 guest and the domU
+guests as well.
+The final goal is to be able to start a dom0less system using EFI.
+
+# Current Xen EFI configuration file
+
+The current configuration file is described by the documentation page
+https://xenbits.xenproject.org/docs/unstable/misc/efi.html.
+
+Here an example:
+
+```
+[global]
+default=section1
+
+[section1]
+options=console=vga,com1 com1=57600 loglvl=all noreboot
+kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
+ramdisk=initrd-3.0.31-0.4-xen
+xsm=<filename>
+dtb=devtree.dtb
+```
+
+# Proposed improvement
+
+The proposed improvement to the current configuration file is the
+introduction of new keywords to describe additional domUs.
+
+Here follows the proposed new keywords:
+  - domu#_kernel=<kernel file> [domU command line options]
+    - Mandatory kernel file for the domU#
+  - domu#_ramdisk=<ramdisk file>
+    - Optional ramdisk file for the domU#
+  - domu#_dtb=<dtb file>
+    - Optional dtb fragment file for the domU#, it is used for device
+      assignment (passthrough).
+  - domu#_property=cpus=2
+    - Properties that should be added to the dtb in the domU node to
+      properly describe the domU guest. Refer to the documentation:
+      https://xenbits.xenproject.org/docs/unstable/misc/arm/device-tree/booting.txt,
+      section "Creating Multiple Domains directly from Xen".
+
+For all the keywords above, the # is a number that uniquely identifies
+the guest.
+The keywords domu#_kernel, domu#_ramdisk, domu#_dtb are unique, therefore there
+must not be specified the same keyword twice in a section.
+The # number is not enforcing any domid, it is just used to link each property
+to the right guest, so there can be domu1_* guests that are started with domid 2
+and so on.
+
+The domu#_property can appear multiple times and it specifies an additional
+property to be listed in the domU node inside the device tree, Xen will
+not check if the same content is specified multiple times.
+
+There are some property whose name starts with an hash symbol (#address-cells,
+#size-cells), in this case the line will be considered as a comment, so to
+specify them, they have to be listed without the hash symbol, the documentation
+will be updated as well to see the implemented handling of these special
+properties.
+
+# Example of a configuration file describing a dom0less system
+
+The following configuration file is describing a dom0less system starting two
+guests.
+
+```
+[global]
+default=xen
+
+[xen]
+# Xen boot arguments
+options=noreboot console=dtuart dtuart=serial0 bootscrub=0
+# Xen device tree
+dtb=devtree.dtb
+
+# Guest 1
+domu1_property=address-cells=2
+domu1_property=size-cells=2
+domu1_kernel=Image-domu1.bin console=ttyAMA0 root=/dev/ram0 rw
+domu1_property=cpus=1
+domu1_property=memory=0xC0000
+domu1_dtb=domu.dtb
+
+# Guest 2
+domu2_kernel=Image-domu2.bin console=ttyAMA0 root=/dev/ram0 rw
+domu2_property=cpus=2
+domu2_property=memory=0x100000
+domu2_property=vpl011
+```
+
+In this example the Guest 1 is loading Image-domu1.bin as kernel and it is using
+the passed boot arguments, it is also loading a device tree snippet for device
+passthrough.
+The second guest, Guest 2, is using another kernel and it will use the vpl011
+device.
-- 
2.17.1

Re: [RFC PATCH] xen/design: Add design for EFI dom0less system start

[Jan Beulich]

On 07.09.2021 08:52, Luca Fancellu wrote:
> Add a design describing a proposal to improve the EFI
> configuration file, adding keywords to describe domU
> guests and allowing to start a dom0less system.
> 
> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
> ---
>  docs/designs/efi-arm-dom0less.md | 105 +++++++++++++++++++++++++++++++
>  1 file changed, 105 insertions(+)
>  create mode 100644 docs/designs/efi-arm-dom0less.md
> 
> diff --git a/docs/designs/efi-arm-dom0less.md b/docs/designs/efi-arm-dom0less.md
> new file mode 100644
> index 0000000000..8d8fa2243f
> --- /dev/null
> +++ b/docs/designs/efi-arm-dom0less.md
> @@ -0,0 +1,105 @@
> +# Xen EFI configuration file
> +
> +The current configuration file used by Xen when it is started as an EFI
> +application is considering only the dom0 guest and doesn't have any
> +property to describe and load in memory domU guests.
> +Hence currently it's impossible to start a dom0less system using EFI.
> +
> +# Objective
> +
> +This document describes the proposed improvement to the Xen EFI
> +configuration file to list properly both the dom0 guest and the domU
> +guests as well.
> +The final goal is to be able to start a dom0less system using EFI.
> +
> +# Current Xen EFI configuration file
> +
> +The current configuration file is described by the documentation page
> +https://xenbits.xenproject.org/docs/unstable/misc/efi.html.
> +
> +Here an example:
> +
> +```
> +[global]
> +default=section1
> +
> +[section1]
> +options=console=vga,com1 com1=57600 loglvl=all noreboot
> +kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
> +ramdisk=initrd-3.0.31-0.4-xen
> +xsm=<filename>
> +dtb=devtree.dtb
> +```
> +
> +# Proposed improvement
> +
> +The proposed improvement to the current configuration file is the
> +introduction of new keywords to describe additional domUs.
> +
> +Here follows the proposed new keywords:
> +  - domu#_kernel=<kernel file> [domU command line options]
> +    - Mandatory kernel file for the domU#
> +  - domu#_ramdisk=<ramdisk file>
> +    - Optional ramdisk file for the domU#
> +  - domu#_dtb=<dtb file>
> +    - Optional dtb fragment file for the domU#, it is used for device
> +      assignment (passthrough).
> +  - domu#_property=cpus=2
> +    - Properties that should be added to the dtb in the domU node to
> +      properly describe the domU guest. Refer to the documentation:
> +      https://xenbits.xenproject.org/docs/unstable/misc/arm/device-tree/booting.txt,
> +      section "Creating Multiple Domains directly from Xen".
> +
> +For all the keywords above, the # is a number that uniquely identifies
> +the guest.
> +The keywords domu#_kernel, domu#_ramdisk, domu#_dtb are unique, therefore there
> +must not be specified the same keyword twice in a section.
> +The # number is not enforcing any domid, it is just used to link each property
> +to the right guest, so there can be domu1_* guests that are started with domid 2
> +and so on.
> +
> +The domu#_property can appear multiple times and it specifies an additional
> +property to be listed in the domU node inside the device tree, Xen will
> +not check if the same content is specified multiple times.
> +
> +There are some property whose name starts with an hash symbol (#address-cells,
> +#size-cells), in this case the line will be considered as a comment, so to
> +specify them, they have to be listed without the hash symbol, the documentation
> +will be updated as well to see the implemented handling of these special
> +properties.
> +
> +# Example of a configuration file describing a dom0less system
> +
> +The following configuration file is describing a dom0less system starting two
> +guests.
> +
> +```
> +[global]
> +default=xen
> +
> +[xen]
> +# Xen boot arguments
> +options=noreboot console=dtuart dtuart=serial0 bootscrub=0
> +# Xen device tree
> +dtb=devtree.dtb
> +
> +# Guest 1
> +domu1_property=address-cells=2
> +domu1_property=size-cells=2
> +domu1_kernel=Image-domu1.bin console=ttyAMA0 root=/dev/ram0 rw
> +domu1_property=cpus=1
> +domu1_property=memory=0xC0000
> +domu1_dtb=domu.dtb
> +
> +# Guest 2
> +domu2_kernel=Image-domu2.bin console=ttyAMA0 root=/dev/ram0 rw
> +domu2_property=cpus=2
> +domu2_property=memory=0x100000
> +domu2_property=vpl011
> +```

I'd like to suggest a different scheme, not the least because I expect
the individual domains being independent of e.g. hypervisor command
line options or Dom0 kernel versions. Yet varying sets of these are,
for example, a reason to have multiple sections in the current scheme.
Every dom0less guest would then require spelling out in every such
section. Hence I think we'd be better off having a section per guest:

[guest1]
kernel=Image-domu1.bin console=ttyAMA0 root=/dev/ram0 rw
property=cpus=1
property=memory=0xC0000
dtb=domu.dtb

These sections would then be referenced by other sections, e.g. by a
new "guests" (or "domus", but this ends up looking a little odd for
its matching of an unrelated latin word) keyword:

guests=guest1,guest2

If it is deemed necessary to make sure such a section can't be
(mistakenly) used to create Dom0, such sections would need identifying
in some way. Presence of property= (or, as per below, properties=)
could be one means (allowing an empty setting would then be desirable).

As to the properties, is there anything wrong with having them all on
one line:

[guest1]
kernel=Image-domu1.bin console=ttyAMA0 root=/dev/ram0 rw
dtb=domu.dtb
properties=cpus=1 memory=0xC0000

(just like is the case for options=)?

Jan

Re: [RFC PATCH] xen/design: Add design for EFI dom0less system start

[Julien Grall]

Hi,

On 07/09/2021 09:33, Jan Beulich wrote:
> On 07.09.2021 08:52, Luca Fancellu wrote:
>> Add a design describing a proposal to improve the EFI
>> configuration file, adding keywords to describe domU
>> guests and allowing to start a dom0less system.
>>
>> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
>> ---
>>   docs/designs/efi-arm-dom0less.md | 105 +++++++++++++++++++++++++++++++
>>   1 file changed, 105 insertions(+)
>>   create mode 100644 docs/designs/efi-arm-dom0less.md
>>
>> diff --git a/docs/designs/efi-arm-dom0less.md b/docs/designs/efi-arm-dom0less.md
>> new file mode 100644
>> index 0000000000..8d8fa2243f
>> --- /dev/null
>> +++ b/docs/designs/efi-arm-dom0less.md
>> @@ -0,0 +1,105 @@
>> +# Xen EFI configuration file
>> +
>> +The current configuration file used by Xen when it is started as an EFI
>> +application is considering only the dom0 guest and doesn't have any
>> +property to describe and load in memory domU guests.
>> +Hence currently it's impossible to start a dom0less system using EFI.
>> +
>> +# Objective
>> +
>> +This document describes the proposed improvement to the Xen EFI
>> +configuration file to list properly both the dom0 guest and the domU
>> +guests as well.
>> +The final goal is to be able to start a dom0less system using EFI.
>> +
>> +# Current Xen EFI configuration file
>> +
>> +The current configuration file is described by the documentation page
>> +https://xenbits.xenproject.org/docs/unstable/misc/efi.html.
>> +
>> +Here an example:
>> +
>> +```
>> +[global]
>> +default=section1
>> +
>> +[section1]
>> +options=console=vga,com1 com1=57600 loglvl=all noreboot
>> +kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
>> +ramdisk=initrd-3.0.31-0.4-xen
>> +xsm=<filename>
>> +dtb=devtree.dtb
>> +```
>> +
>> +# Proposed improvement
>> +
>> +The proposed improvement to the current configuration file is the
>> +introduction of new keywords to describe additional domUs.
>> +
>> +Here follows the proposed new keywords:
>> +  - domu#_kernel=<kernel file> [domU command line options]
>> +    - Mandatory kernel file for the domU#
>> +  - domu#_ramdisk=<ramdisk file>
>> +    - Optional ramdisk file for the domU#
>> +  - domu#_dtb=<dtb file>
>> +    - Optional dtb fragment file for the domU#, it is used for device
>> +      assignment (passthrough).
>> +  - domu#_property=cpus=2
>> +    - Properties that should be added to the dtb in the domU node to
>> +      properly describe the domU guest. Refer to the documentation:
>> +      https://xenbits.xenproject.org/docs/unstable/misc/arm/device-tree/booting.txt,
>> +      section "Creating Multiple Domains directly from Xen".
>> +
>> +For all the keywords above, the # is a number that uniquely identifies
>> +the guest.
>> +The keywords domu#_kernel, domu#_ramdisk, domu#_dtb are unique, therefore there
>> +must not be specified the same keyword twice in a section.
>> +The # number is not enforcing any domid, it is just used to link each property
>> +to the right guest, so there can be domu1_* guests that are started with domid 2
>> +and so on.
>> +
>> +The domu#_property can appear multiple times and it specifies an additional
>> +property to be listed in the domU node inside the device tree, Xen will
>> +not check if the same content is specified multiple times.
>> +
>> +There are some property whose name starts with an hash symbol (#address-cells,
>> +#size-cells), in this case the line will be considered as a comment, so to
>> +specify them, they have to be listed without the hash symbol, the documentation
>> +will be updated as well to see the implemented handling of these special
>> +properties.
>> +
>> +# Example of a configuration file describing a dom0less system
>> +
>> +The following configuration file is describing a dom0less system starting two
>> +guests.
>> +
>> +```
>> +[global]
>> +default=xen
>> +
>> +[xen]
>> +# Xen boot arguments
>> +options=noreboot console=dtuart dtuart=serial0 bootscrub=0
>> +# Xen device tree
>> +dtb=devtree.dtb
>> +
>> +# Guest 1
>> +domu1_property=address-cells=2
>> +domu1_property=size-cells=2
>> +domu1_kernel=Image-domu1.bin console=ttyAMA0 root=/dev/ram0 rw
>> +domu1_property=cpus=1
>> +domu1_property=memory=0xC0000
>> +domu1_dtb=domu.dtb
>> +
>> +# Guest 2
>> +domu2_kernel=Image-domu2.bin console=ttyAMA0 root=/dev/ram0 rw
>> +domu2_property=cpus=2
>> +domu2_property=memory=0x100000
>> +domu2_property=vpl011
>> +```
> 
> I'd like to suggest a different scheme, not the least because I expect
> the individual domains being independent of e.g. hypervisor command
> line options or Dom0 kernel versions. Yet varying sets of these are,
> for example, a reason to have multiple sections in the current scheme.
> Every dom0less guest would then require spelling out in every such
> section. Hence I think we'd be better off having a section per guest:
> 
> [guest1]
> kernel=Image-domu1.bin console=ttyAMA0 root=/dev/ram0 rw
> property=cpus=1
> property=memory=0xC0000
> dtb=domu.dtb

I much prefer the idea of the section. This is going to be easier to 
parse the configuration file as we would not have to look for "domuX_" 
and then distinguishing X.

> 
> These sections would then be referenced by other sections, e.g. by a
> new "guests" (or "domus", but this ends up looking a little odd for
> its matching of an unrelated latin word) keyword:
> 
> guests=guest1,guest2
> 
> If it is deemed necessary to make sure such a section can't be
> (mistakenly) used to create Dom0, such sections would need identifying
> in some way. Presence of property= (or, as per below, properties=)
> could be one means (allowing an empty setting would then be desirable).

I would expect dom0 to be described in the similar fashion at some 
point. So maybe we should name the property "domains=...".

> 
> As to the properties, is there anything wrong with having them all on
> one line:
> 
> [guest1]
> kernel=Image-domu1.bin console=ttyAMA0 root=/dev/ram0 rw
> dtb=domu.dtb
> properties=cpus=1 memory=0xC0000

It depends on the number of properties for the domain, this may become 
quickly unreadable.

But... if we use sections, then I think it would be better to have:

kernel=..
dtb=...
cpu=1
memory=0xC0000

This would also allow us to create more complex setup (such as for the 
static memory allocation).

Cheers,

-- 
Julien Grall

Re: [RFC PATCH] xen/design: Add design for EFI dom0less system start

[Jan Beulich]

On 07.09.2021 11:17, Julien Grall wrote:
> On 07/09/2021 09:33, Jan Beulich wrote:
>> I'd like to suggest a different scheme, not the least because I expect
>> the individual domains being independent of e.g. hypervisor command
>> line options or Dom0 kernel versions. Yet varying sets of these are,
>> for example, a reason to have multiple sections in the current scheme.
>> Every dom0less guest would then require spelling out in every such
>> section. Hence I think we'd be better off having a section per guest:
>>
>> [guest1]
>> kernel=Image-domu1.bin console=ttyAMA0 root=/dev/ram0 rw
>> property=cpus=1
>> property=memory=0xC0000
>> dtb=domu.dtb
> 
> I much prefer the idea of the section. This is going to be easier to 
> parse the configuration file as we would not have to look for "domuX_" 
> and then distinguishing X.
> 
>>
>> These sections would then be referenced by other sections, e.g. by a
>> new "guests" (or "domus", but this ends up looking a little odd for
>> its matching of an unrelated latin word) keyword:
>>
>> guests=guest1,guest2
>>
>> If it is deemed necessary to make sure such a section can't be
>> (mistakenly) used to create Dom0, such sections would need identifying
>> in some way. Presence of property= (or, as per below, properties=)
>> could be one means (allowing an empty setting would then be desirable).
> 
> I would expect dom0 to be described in the similar fashion at some 
> point. So maybe we should name the property "domains=...".

Not sure - the order above doesn't mandate domain IDs, yet Dom0 needs
creating with ID 0. IOW I was deliberately suggesting "guests=".

>> As to the properties, is there anything wrong with having them all on
>> one line:
>>
>> [guest1]
>> kernel=Image-domu1.bin console=ttyAMA0 root=/dev/ram0 rw
>> dtb=domu.dtb
>> properties=cpus=1 memory=0xC0000
> 
> It depends on the number of properties for the domain, this may become 
> quickly unreadable.
> 
> But... if we use sections, then I think it would be better to have:
> 
> kernel=..
> dtb=...
> cpu=1
> memory=0xC0000
> 
> This would also allow us to create more complex setup (such as for the 
> static memory allocation).

If that's feasible parsing-wise - sure. I was first thinking to suggest
separate keywords, but then decided there was a reason this wasn't done
in the original proposal (with respective dom#_ prefixes).

Jan

Re: [RFC PATCH] xen/design: Add design for EFI dom0less system start

[Luca Fancellu]

> On 7 Sep 2021, at 10:24, Jan Beulich <jbeulich@suse.com> wrote:
> 
> On 07.09.2021 11:17, Julien Grall wrote:
>> On 07/09/2021 09:33, Jan Beulich wrote:
>>> I'd like to suggest a different scheme, not the least because I expect
>>> the individual domains being independent of e.g. hypervisor command
>>> line options or Dom0 kernel versions. Yet varying sets of these are,
>>> for example, a reason to have multiple sections in the current scheme.
>>> Every dom0less guest would then require spelling out in every such
>>> section. Hence I think we'd be better off having a section per guest:
>>> 
>>> [guest1]
>>> kernel=Image-domu1.bin console=ttyAMA0 root=/dev/ram0 rw
>>> property=cpus=1
>>> property=memory=0xC0000
>>> dtb=domu.dtb
>> 
>> I much prefer the idea of the section. This is going to be easier to 
>> parse the configuration file as we would not have to look for "domuX_" 
>> and then distinguishing X.
>> 
>>> 
>>> These sections would then be referenced by other sections, e.g. by a
>>> new "guests" (or "domus", but this ends up looking a little odd for
>>> its matching of an unrelated latin word) keyword:
>>> 
>>> guests=guest1,guest2
>>> 
>>> If it is deemed necessary to make sure such a section can't be
>>> (mistakenly) used to create Dom0, such sections would need identifying
>>> in some way. Presence of property= (or, as per below, properties=)
>>> could be one means (allowing an empty setting would then be desirable).
>> 
>> I would expect dom0 to be described in the similar fashion at some 
>> point. So maybe we should name the property "domains=...".
> 
> Not sure - the order above doesn't mandate domain IDs, yet Dom0 needs
> creating with ID 0. IOW I was deliberately suggesting "guests=".

Hi Jan, Julien,

Thanks for your feedbacks, really appreciated, I agree with you that sections
are easier to read.
I stand with the “guests=“ or “domus=“ or “domu_guests=“ keywords where
the references to “guest sections” can be listed, separated by commas. 

> 
>>> As to the properties, is there anything wrong with having them all on
>>> one line:
>>> 
>>> [guest1]
>>> kernel=Image-domu1.bin console=ttyAMA0 root=/dev/ram0 rw
>>> dtb=domu.dtb
>>> properties=cpus=1 memory=0xC0000
>> 
>> It depends on the number of properties for the domain, this may become 
>> quickly unreadable.
>> 
>> But... if we use sections, then I think it would be better to have:
>> 
>> kernel=..
>> dtb=...
>> cpu=1
>> memory=0xC0000
>> 
>> This would also allow us to create more complex setup (such as for the 
>> static memory allocation).
> 
> If that's feasible parsing-wise - sure. I was first thinking to suggest
> separate keywords, but then decided there was a reason this wasn't done
> in the original proposal (with respective dom#_ prefixes).

Yes I think it is feasible and easy to read, so given your feedback we can agree on
something like that:

[global]
default=xen

[xen]
options=[xen command line]
kernel=[dom0 kernel file] [domain 0 command line options]
ramdisk=[dom0 ramdisk]
xsm=[xsm filename]
dtb=[xen device tree blob]
domu_guests=guest1,...

[guest1]
address-cells=2
size-cells=2
kernel=[domU kernel file] [domU command line options]
cpus=1
memory=0xC0000
dtb=[domU passthrough dtb]

This won’t break the compatibility because on older version the domu_guests keyword
Is ignored, the user needs just to check that a guest section is not specified as default=
argument because in this case it will be loaded as dom0 kernel (and probably won’t work),
but I think we can just update the documentation to highlight this thing.

Finally this example should load a dom0less system:

[global]
default=xen

[xen]
# Xen boot arguments
options=noreboot console=dtuart dtuart=serial0 bootscrub=0
# Xen device tree
dtb=devtree.dtb
# domus to be started
domu_guests=guest1,guest2

[guest1]
address-cells=2
size-cells=2
kernel=Image-domu1.bin console=ttyAMA0 root=/dev/ram0 rw
cpus=1
memory=0xC0000
dtb=domu.dtb

[guest2]
kernel=Image-domu2.bin console=ttyAMA0 root=/dev/ram0 rw
cpus=2
memory=0x100000
vpl011


> 
> Jan
> 

Re: [RFC PATCH] xen/design: Add design for EFI dom0less system start

[Julien Grall]

Hi Luca,

On 07/09/2021 07:52, Luca Fancellu wrote:
> Add a design describing a proposal to improve the EFI
> configuration file, adding keywords to describe domU
> guests and allowing to start a dom0less system.
> 
> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
> ---
>   docs/designs/efi-arm-dom0less.md | 105 +++++++++++++++++++++++++++++++
>   1 file changed, 105 insertions(+)
>   create mode 100644 docs/designs/efi-arm-dom0less.md
> 
> diff --git a/docs/designs/efi-arm-dom0less.md b/docs/designs/efi-arm-dom0less.md
> new file mode 100644
> index 0000000000..8d8fa2243f
> --- /dev/null
> +++ b/docs/designs/efi-arm-dom0less.md
> @@ -0,0 +1,105 @@
> +# Xen EFI configuration file
> +
> +The current configuration file used by Xen when it is started as an EFI
> +application is considering only the dom0 guest and doesn't have any
> +property to describe and load in memory domU guests.

 From my understanding, the problem is less about properties (we already 
have them in the Device-Tree) but more about where are the binaries 
located in memory as we don't know in advance.

So I would like to propose something that build on top of the 
Device-Tree work we did. Note this is early thoughts.

The problematic nodes in the DT are:

         module@0x4a000000 {
             compatible = "multiboot,kernel", "multiboot,module";
             reg = <0x0 0x4a000000 0xffffff>;
             bootargs = "console=ttyAMA0 init=/bin/sh";
         };

         module@0x4b000000 {
             compatible = "multiboot,ramdisk", "multiboot,module";
             reg = <0x0 0x4b000000 0xffffff>;
         };

In particular the property "reg" cannot be known in advance because the 
UEFI stub will be responsible to load the binaries in memory.

What we could do is providing a list of binaries to load and associate a 
key for each of them. Something like:

binary=<binary> <key>
binary=<binary2> <key2>
....

We can then replace the property "reg" with a new property "uefi,key" 
that will contain the name of the binary.

What do you think?

Cheers,

-- 
Julien Grall

Re: [RFC PATCH] xen/design: Add design for EFI dom0less system start

[Luca Fancellu]

> On 7 Sep 2021, at 10:35, Julien Grall <julien@xen.org> wrote:
> 
> Hi Luca,
> 
> On 07/09/2021 07:52, Luca Fancellu wrote:
>> Add a design describing a proposal to improve the EFI
>> configuration file, adding keywords to describe domU
>> guests and allowing to start a dom0less system.
>> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
>> ---
>>  docs/designs/efi-arm-dom0less.md | 105 +++++++++++++++++++++++++++++++
>>  1 file changed, 105 insertions(+)
>>  create mode 100644 docs/designs/efi-arm-dom0less.md
>> diff --git a/docs/designs/efi-arm-dom0less.md b/docs/designs/efi-arm-dom0less.md
>> new file mode 100644
>> index 0000000000..8d8fa2243f
>> --- /dev/null
>> +++ b/docs/designs/efi-arm-dom0less.md
>> @@ -0,0 +1,105 @@
>> +# Xen EFI configuration file
>> +
>> +The current configuration file used by Xen when it is started as an EFI
>> +application is considering only the dom0 guest and doesn't have any
>> +property to describe and load in memory domU guests.
> 
> From my understanding, the problem is less about properties (we already have them in the Device-Tree) but more about where are the binaries located in memory as we don't know in advance.

Hi Julien,

I think I used the wrong word there, I meant “keyword” instead of “property” because I was referring about the
lack of keywords to describe a domu guest in the Xen EFI configuration file.

I agree with you that on systems with static allocation, the kernel and ramdisk binaries must be at certain locations
that are out of control when we use the EFI boot services, the thing we can do is provide a keyword to specify the
addresses and then use the CopyMem() function to relocate the kernel/ramdisk in the address we want.

> 
> So I would like to propose something that build on top of the Device-Tree work we did. Note this is early thoughts.
> 
> The problematic nodes in the DT are:
> 
>        module@0x4a000000 {
>            compatible = "multiboot,kernel", "multiboot,module";
>            reg = <0x0 0x4a000000 0xffffff>;
>            bootargs = "console=ttyAMA0 init=/bin/sh";
>        };
> 
>        module@0x4b000000 {
>            compatible = "multiboot,ramdisk", "multiboot,module";
>            reg = <0x0 0x4b000000 0xffffff>;
>        };
> 
> In particular the property "reg" cannot be known in advance because the UEFI stub will be responsible to load the binaries in memory.

Yes that’s true, the UEFI stub is using from the UEFI boot service the AllocatePages function that is giving back an address out of our control,
then using another function the binary is read from the disk and copied at that address, finally the UEFI stub is writing the node in the device tree that
will be used by Xen later.

> 
> What we could do is providing a list of binaries to load and associate a key for each of them. Something like:
> 
> binary=<binary> <key>
> binary=<binary2> <key2>
> ....
> 
> We can then replace the property "reg" with a new property "uefi,key" that will contain the name of the binary.
> 
> What do you think?

Here I’m lost, because I don’t understand what we are going to do with the name of the binary.

Cheers,

Luca

> 
> Cheers,
> 
> -- 
> Julien Grall

Re: [RFC PATCH] xen/design: Add design for EFI dom0less system start

[Jan Beulich]

On 07.09.2021 13:51, Luca Fancellu wrote:
>> On 7 Sep 2021, at 10:35, Julien Grall <julien@xen.org> wrote:
>> On 07/09/2021 07:52, Luca Fancellu wrote:
>>> --- /dev/null
>>> +++ b/docs/designs/efi-arm-dom0less.md
>>> @@ -0,0 +1,105 @@
>>> +# Xen EFI configuration file
>>> +
>>> +The current configuration file used by Xen when it is started as an EFI
>>> +application is considering only the dom0 guest and doesn't have any
>>> +property to describe and load in memory domU guests.
>>
>> From my understanding, the problem is less about properties (we already have them in the Device-Tree) but more about where are the binaries located in memory as we don't know in advance.
> 
> I think I used the wrong word there, I meant “keyword” instead of “property” because I was referring about the
> lack of keywords to describe a domu guest in the Xen EFI configuration file.
> 
> I agree with you that on systems with static allocation, the kernel and ramdisk binaries must be at certain locations
> that are out of control when we use the EFI boot services, the thing we can do is provide a keyword to specify the
> addresses and then use the CopyMem() function to relocate the kernel/ramdisk in the address we want.
> 
>>
>> So I would like to propose something that build on top of the Device-Tree work we did. Note this is early thoughts.
>>
>> The problematic nodes in the DT are:
>>
>>        module@0x4a000000 {
>>            compatible = "multiboot,kernel", "multiboot,module";
>>            reg = <0x0 0x4a000000 0xffffff>;
>>            bootargs = "console=ttyAMA0 init=/bin/sh";
>>        };
>>
>>        module@0x4b000000 {
>>            compatible = "multiboot,ramdisk", "multiboot,module";
>>            reg = <0x0 0x4b000000 0xffffff>;
>>        };
>>
>> In particular the property "reg" cannot be known in advance because the UEFI stub will be responsible to load the binaries in memory.
> 
> Yes that’s true, the UEFI stub is using from the UEFI boot service the AllocatePages function that is giving back an address out of our control,
> then using another function the binary is read from the disk and copied at that address, finally the UEFI stub is writing the node in the device tree that
> will be used by Xen later.

If you know the intended address is available, AllocatePages() can very well
be given a pre-determined address via the AllocateAddress allocation type.

Jan

Re: [RFC PATCH] xen/design: Add design for EFI dom0less system start

[Julien Grall]

On 07/09/2021 12:51, Luca Fancellu wrote:
> 
> 
>> On 7 Sep 2021, at 10:35, Julien Grall <julien@xen.org> wrote:
>>
>> Hi Luca,
>>
>> On 07/09/2021 07:52, Luca Fancellu wrote:
>>> Add a design describing a proposal to improve the EFI
>>> configuration file, adding keywords to describe domU
>>> guests and allowing to start a dom0less system.
>>> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
>>> ---
>>>   docs/designs/efi-arm-dom0less.md | 105 +++++++++++++++++++++++++++++++
>>>   1 file changed, 105 insertions(+)
>>>   create mode 100644 docs/designs/efi-arm-dom0less.md
>>> diff --git a/docs/designs/efi-arm-dom0less.md b/docs/designs/efi-arm-dom0less.md
>>> new file mode 100644
>>> index 0000000000..8d8fa2243f
>>> --- /dev/null
>>> +++ b/docs/designs/efi-arm-dom0less.md
>>> @@ -0,0 +1,105 @@
>>> +# Xen EFI configuration file
>>> +
>>> +The current configuration file used by Xen when it is started as an EFI
>>> +application is considering only the dom0 guest and doesn't have any
>>> +property to describe and load in memory domU guests.
>>
>>  From my understanding, the problem is less about properties (we already have them in the Device-Tree) but more about where are the binaries located in memory as we don't know in advance.
> 
> Hi Julien,
Hi Luca,

> I think I used the wrong word there, I meant “keyword” instead of “property” because I was referring about the
> lack of keywords to describe a domu guest in the Xen EFI configuration file.
> 
> I agree with you that on systems with static allocation, the kernel and ramdisk binaries must be at certain locations
> that are out of control when we use the EFI boot services, the thing we can do is provide a keyword to specify the
> addresses and then use the CopyMem() function to relocate the kernel/ramdisk in the address we want.

I wasn't specifically referring to static allocation here, sorry if this 
wasn't clear. I was pointing out that most of the information you create 
in the xen.cfg is going to be similar to what we already provide in the 
Device-Tree.

My main concern is everytime we add a new feature in Dom0less, a 
developer would need to write code for the DT and UEFI. This will 
increase the code size and maintenance.

The same can be said for the admin as if they want to switch from plain 
U-boot to UEFI, they would also need to fully rewrite the bindings.

AFAICT, most of the information provided in the Device-Tree are usable 
even when using UEFI. So I would prefer if we try to re-use what's 
existing. This is what my proposal below was about.

> 
>>
>> So I would like to propose something that build on top of the Device-Tree work we did. Note this is early thoughts.
>>
>> The problematic nodes in the DT are:
>>
>>         module@0x4a000000 {
>>             compatible = "multiboot,kernel", "multiboot,module";
>>             reg = <0x0 0x4a000000 0xffffff>;
>>             bootargs = "console=ttyAMA0 init=/bin/sh";
>>         };
>>
>>         module@0x4b000000 {
>>             compatible = "multiboot,ramdisk", "multiboot,module";
>>             reg = <0x0 0x4b000000 0xffffff>;
>>         };
>>
>> In particular the property "reg" cannot be known in advance because the UEFI stub will be responsible to load the binaries in memory.
> 
> Yes that’s true, the UEFI stub is using from the UEFI boot service the AllocatePages function that is giving back an address out of our control,
> then using another function the binary is read from the disk and copied at that address, finally the UEFI stub is writing the node in the device tree that
> will be used by Xen later.

I am not sure to follow. Are you saying the UEFI stub will create the 
dom0less node in the DT based on the xen.cfg?

> 
>>
>> What we could do is providing a list of binaries to load and associate a key for each of them. Something like:
>>
>> binary=<binary> <key>
>> binary=<binary2> <key2>
>> ....
>>
>> We can then replace the property "reg" with a new property "uefi,key" that will contain the name of the binary.
>>
>> What do you think?
> 
> Here I’m lost, because I don’t understand what we are going to do with the name of the binary.

<binaryX> would be used by the UEFI stub to load the binary in memory. 
Each binary will have a <keyX> which helps to refer them in the 
Device-Tree. To give a concrete example, let say we have two dom0less 
domains:
   - DomA: 2 vCPUs, 128MB
   - DomB: 3 vCPUs, 512MB

DomA and DomB will be using the same kernel but a different ramdisk. 
xen.cfg, would look like:

[global]
default=section1

[section1]
options=console=vga,com1 com1=57600 loglvl=all noreboot
kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
ramdisk=initrd-3.0.31-0.4-xen
xsm=<filename>
dtb=devtree.dtb
binary=vmlinuz-guest domu-kernel
binary=ramdisk-domA.img domA-ramdisk
binary=ramdisk-domB.img domB-ramdisk

The chosen node in the DT would look like:

chosen {
     domU1 {
         compatible = "xen,domain";
         #address-cells = <0x2>;
         #size-cells = <0x1>;
         memory = <0 0x8000000>;
         cpus = <2>;

         module@1 {
             compatible = "multiboot,kernel", "multiboot,module";
             uefi,binary = "domu-kernel";
             bootargs = "console=ttyAMA0 init=/bin/sh";
         };

         module@2 {
             compatible = "multiboot,ramdisk", "multiboot,module";
             uefi,binary = "domA-ramdisk";
         };
     };

     domU2 {
         compatible = "xen,domain";
         #address-cells = <0x3>;
         #size-cells = <0x1>;
         memory = <0 0x20000000>;
         cpus = <3>;

         module@1 {
             compatible = "multiboot,kernel", "multiboot,module";
             uefi,binary = "domu-kernel";
             bootargs = "console=ttyAMA0 init=/bin/sh";
         };

         module@2 {
             compatible = "multiboot,ramdisk", "multiboot,module";
             uefi,binary = "domA-ramdisk";
         };
     };
};

With this approach, the change is quite minimal to move between an 
classic U-boot boot and EFI boot.

Cheers,

-- 
Julien Grall

Re: [RFC PATCH] xen/design: Add design for EFI dom0less system start

[Luca Fancellu]

> On 7 Sep 2021, at 13:30, Julien Grall <julien@xen.org> wrote:
> 
> 
> 
> On 07/09/2021 12:51, Luca Fancellu wrote:
>>> On 7 Sep 2021, at 10:35, Julien Grall <julien@xen.org> wrote:
>>> 
>>> Hi Luca,
>>> 
>>> On 07/09/2021 07:52, Luca Fancellu wrote:
>>>> Add a design describing a proposal to improve the EFI
>>>> configuration file, adding keywords to describe domU
>>>> guests and allowing to start a dom0less system.
>>>> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
>>>> ---
>>>>  docs/designs/efi-arm-dom0less.md | 105 +++++++++++++++++++++++++++++++
>>>>  1 file changed, 105 insertions(+)
>>>>  create mode 100644 docs/designs/efi-arm-dom0less.md
>>>> diff --git a/docs/designs/efi-arm-dom0less.md b/docs/designs/efi-arm-dom0less.md
>>>> new file mode 100644
>>>> index 0000000000..8d8fa2243f
>>>> --- /dev/null
>>>> +++ b/docs/designs/efi-arm-dom0less.md
>>>> @@ -0,0 +1,105 @@
>>>> +# Xen EFI configuration file
>>>> +
>>>> +The current configuration file used by Xen when it is started as an EFI
>>>> +application is considering only the dom0 guest and doesn't have any
>>>> +property to describe and load in memory domU guests.
>>> 
>>> From my understanding, the problem is less about properties (we already have them in the Device-Tree) but more about where are the binaries located in memory as we don't know in advance.
>> Hi Julien,
> Hi Luca,
> 
>> I think I used the wrong word there, I meant “keyword” instead of “property” because I was referring about the
>> lack of keywords to describe a domu guest in the Xen EFI configuration file.
>> I agree with you that on systems with static allocation, the kernel and ramdisk binaries must be at certain locations
>> that are out of control when we use the EFI boot services, the thing we can do is provide a keyword to specify the
>> addresses and then use the CopyMem() function to relocate the kernel/ramdisk in the address we want.
> 
> I wasn't specifically referring to static allocation here, sorry if this wasn't clear. I was pointing out that most of the information you create in the xen.cfg is going to be similar to what we already provide in the Device-Tree.
> 
> My main concern is everytime we add a new feature in Dom0less, a developer would need to write code for the DT and UEFI. This will increase the code size and maintenance.
> 
> The same can be said for the admin as if they want to switch from plain U-boot to UEFI, they would also need to fully rewrite the bindings.
> 
> AFAICT, most of the information provided in the Device-Tree are usable even when using UEFI. So I would prefer if we try to re-use what's existing. This is what my proposal below was about.
> 
>>> 
>>> So I would like to propose something that build on top of the Device-Tree work we did. Note this is early thoughts.
>>> 
>>> The problematic nodes in the DT are:
>>> 
>>>        module@0x4a000000 {
>>>            compatible = "multiboot,kernel", "multiboot,module";
>>>            reg = <0x0 0x4a000000 0xffffff>;
>>>            bootargs = "console=ttyAMA0 init=/bin/sh";
>>>        };
>>> 
>>>        module@0x4b000000 {
>>>            compatible = "multiboot,ramdisk", "multiboot,module";
>>>            reg = <0x0 0x4b000000 0xffffff>;
>>>        };
>>> 
>>> In particular the property "reg" cannot be known in advance because the UEFI stub will be responsible to load the binaries in memory.
>> Yes that’s true, the UEFI stub is using from the UEFI boot service the AllocatePages function that is giving back an address out of our control,
>> then using another function the binary is read from the disk and copied at that address, finally the UEFI stub is writing the node in the device tree that
>> will be used by Xen later.
> 
> I am not sure to follow. Are you saying the UEFI stub will create the dom0less node in the DT based on the xen.cfg?

Yes, in my mind the UEFI stub would have created all necessary node into the chosen {} to create the system, now the stub does that
but only for the dom0 modules.

> 
>>> 
>>> What we could do is providing a list of binaries to load and associate a key for each of them. Something like:
>>> 
>>> binary=<binary> <key>
>>> binary=<binary2> <key2>
>>> ....
>>> 
>>> We can then replace the property "reg" with a new property "uefi,key" that will contain the name of the binary.
>>> 
>>> What do you think?
>> Here I’m lost, because I don’t understand what we are going to do with the name of the binary.
> 
> <binaryX> would be used by the UEFI stub to load the binary in memory. Each binary will have a <keyX> which helps to refer them in the Device-Tree. To give a concrete example, let say we have two dom0less domains:
>  - DomA: 2 vCPUs, 128MB
>  - DomB: 3 vCPUs, 512MB
> 
> DomA and DomB will be using the same kernel but a different ramdisk. xen.cfg, would look like:
> 
> [global]
> default=section1
> 
> [section1]
> options=console=vga,com1 com1=57600 loglvl=all noreboot
> kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
> ramdisk=initrd-3.0.31-0.4-xen
> xsm=<filename>
> dtb=devtree.dtb
> binary=vmlinuz-guest domu-kernel
> binary=ramdisk-domA.img domA-ramdisk
> binary=ramdisk-domB.img domB-ramdisk
> 
> The chosen node in the DT would look like:
> 
> chosen {
>    domU1 {
>        compatible = "xen,domain";
>        #address-cells = <0x2>;
>        #size-cells = <0x1>;
>        memory = <0 0x8000000>;
>        cpus = <2>;
> 
>        module@1 {
>            compatible = "multiboot,kernel", "multiboot,module";
>            uefi,binary = "domu-kernel";
>            bootargs = "console=ttyAMA0 init=/bin/sh";
>        };
> 
>        module@2 {
>            compatible = "multiboot,ramdisk", "multiboot,module";
>            uefi,binary = "domA-ramdisk";
>        };
>    };
> 
>    domU2 {
>        compatible = "xen,domain";
>        #address-cells = <0x3>;
>        #size-cells = <0x1>;
>        memory = <0 0x20000000>;
>        cpus = <3>;
> 
>        module@1 {
>            compatible = "multiboot,kernel", "multiboot,module";
>            uefi,binary = "domu-kernel";
>            bootargs = "console=ttyAMA0 init=/bin/sh";
>        };
> 
>        module@2 {
>            compatible = "multiboot,ramdisk", "multiboot,module";
>            uefi,binary = "domA-ramdisk";
>        };
>    };
> };
> 
> With this approach, the change is quite minimal to move between an classic U-boot boot and EFI boot.

Ok now I see, yes this approach can work and can save some code, in the current code we have that if
a "multiboot,module” is found in the dtb, the Xen EFI configuration file is skipped, but if we use the
module@XX {} without the compatible it can work, the UEFI stub will load the binary and update all
the needed properties (compatible, reg).

Please let me know if we all agree on that

Cheers,

Luca


> 
> Cheers,
> 
> -- 
> Julien Grall

Re: [RFC PATCH] xen/design: Add design for EFI dom0less system start

[Julien Grall]

Hi Luca,

On 07/09/2021 14:30, Luca Fancellu wrote:
>> On 7 Sep 2021, at 13:30, Julien Grall <julien@xen.org> wrote:
>> On 07/09/2021 12:51, Luca Fancellu wrote:
>>>> On 7 Sep 2021, at 10:35, Julien Grall <julien@xen.org> wrote:
>>>> What we could do is providing a list of binaries to load and associate a key for each of them. Something like:
>>>>
>>>> binary=<binary> <key>
>>>> binary=<binary2> <key2>
>>>> ....
>>>>
>>>> We can then replace the property "reg" with a new property "uefi,key" that will contain the name of the binary.
>>>>
>>>> What do you think?
>>> Here I’m lost, because I don’t understand what we are going to do with the name of the binary.
>>
>> <binaryX> would be used by the UEFI stub to load the binary in memory. Each binary will have a <keyX> which helps to refer them in the Device-Tree. To give a concrete example, let say we have two dom0less domains:
>>   - DomA: 2 vCPUs, 128MB
>>   - DomB: 3 vCPUs, 512MB
>>
>> DomA and DomB will be using the same kernel but a different ramdisk. xen.cfg, would look like:
>>
>> [global]
>> default=section1
>>
>> [section1]
>> options=console=vga,com1 com1=57600 loglvl=all noreboot
>> kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
>> ramdisk=initrd-3.0.31-0.4-xen
>> xsm=<filename>
>> dtb=devtree.dtb
>> binary=vmlinuz-guest domu-kernel
>> binary=ramdisk-domA.img domA-ramdisk
>> binary=ramdisk-domB.img domB-ramdisk
>>
>> The chosen node in the DT would look like:
>>
>> chosen {
>>     domU1 {
>>         compatible = "xen,domain";
>>         #address-cells = <0x2>;
>>         #size-cells = <0x1>;
>>         memory = <0 0x8000000>;
>>         cpus = <2>;
>>
>>         module@1 {
>>             compatible = "multiboot,kernel", "multiboot,module";
>>             uefi,binary = "domu-kernel";
>>             bootargs = "console=ttyAMA0 init=/bin/sh";
>>         };
>>
>>         module@2 {
>>             compatible = "multiboot,ramdisk", "multiboot,module";
>>             uefi,binary = "domA-ramdisk";
>>         };
>>     };
>>
>>     domU2 {
>>         compatible = "xen,domain";
>>         #address-cells = <0x3>;
>>         #size-cells = <0x1>;
>>         memory = <0 0x20000000>;
>>         cpus = <3>;
>>
>>         module@1 {
>>             compatible = "multiboot,kernel", "multiboot,module";
>>             uefi,binary = "domu-kernel";
>>             bootargs = "console=ttyAMA0 init=/bin/sh";
>>         };
>>
>>         module@2 {
>>             compatible = "multiboot,ramdisk", "multiboot,module";
>>             uefi,binary = "domA-ramdisk";
>>         };
>>     };
>> };
>>
>> With this approach, the change is quite minimal to move between an classic U-boot boot and EFI boot.
> 
> Ok now I see, yes this approach can work and can save some code, in the current code we have that if
> a "multiboot,module” is found in the dtb, the Xen EFI configuration file is skipped, but if we use the
> module@XX {} without the compatible it can work, the UEFI stub will load the binary and update all
> the needed properties (compatible, reg).
With my proposal, you don't know whether the binary is a kernel, 
ramdisk... So you wouldn't be able to recreate the compatible properly.

But the behavior of the UEFI stub can be modified. We could say that if 
there is a "xen,domain" then use the configuration file to fetch the 
binaries.

Cheers,

-- 
Julien Grall

Re: [RFC PATCH] xen/design: Add design for EFI dom0less system start

[Luca Fancellu]

> On 7 Sep 2021, at 15:18, Julien Grall <julien@xen.org> wrote:
> 
> Hi Luca,
> 
> On 07/09/2021 14:30, Luca Fancellu wrote:
>>> On 7 Sep 2021, at 13:30, Julien Grall <julien@xen.org> wrote:
>>> On 07/09/2021 12:51, Luca Fancellu wrote:
>>>>> On 7 Sep 2021, at 10:35, Julien Grall <julien@xen.org> wrote:
>>>>> What we could do is providing a list of binaries to load and associate a key for each of them. Something like:
>>>>> 
>>>>> binary=<binary> <key>
>>>>> binary=<binary2> <key2>
>>>>> ....
>>>>> 
>>>>> We can then replace the property "reg" with a new property "uefi,key" that will contain the name of the binary.
>>>>> 
>>>>> What do you think?
>>>> Here I’m lost, because I don’t understand what we are going to do with the name of the binary.
>>> 
>>> <binaryX> would be used by the UEFI stub to load the binary in memory. Each binary will have a <keyX> which helps to refer them in the Device-Tree. To give a concrete example, let say we have two dom0less domains:
>>>  - DomA: 2 vCPUs, 128MB
>>>  - DomB: 3 vCPUs, 512MB
>>> 
>>> DomA and DomB will be using the same kernel but a different ramdisk. xen.cfg, would look like:
>>> 
>>> [global]
>>> default=section1
>>> 
>>> [section1]
>>> options=console=vga,com1 com1=57600 loglvl=all noreboot
>>> kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
>>> ramdisk=initrd-3.0.31-0.4-xen
>>> xsm=<filename>
>>> dtb=devtree.dtb
>>> binary=vmlinuz-guest domu-kernel
>>> binary=ramdisk-domA.img domA-ramdisk
>>> binary=ramdisk-domB.img domB-ramdisk
>>> 
>>> The chosen node in the DT would look like:
>>> 
>>> chosen {
>>>    domU1 {
>>>        compatible = "xen,domain";
>>>        #address-cells = <0x2>;
>>>        #size-cells = <0x1>;
>>>        memory = <0 0x8000000>;
>>>        cpus = <2>;
>>> 
>>>        module@1 {
>>>            compatible = "multiboot,kernel", "multiboot,module";
>>>            uefi,binary = "domu-kernel";
>>>            bootargs = "console=ttyAMA0 init=/bin/sh";
>>>        };
>>> 
>>>        module@2 {
>>>            compatible = "multiboot,ramdisk", "multiboot,module";
>>>            uefi,binary = "domA-ramdisk";
>>>        };
>>>    };
>>> 
>>>    domU2 {
>>>        compatible = "xen,domain";
>>>        #address-cells = <0x3>;
>>>        #size-cells = <0x1>;
>>>        memory = <0 0x20000000>;
>>>        cpus = <3>;
>>> 
>>>        module@1 {
>>>            compatible = "multiboot,kernel", "multiboot,module";
>>>            uefi,binary = "domu-kernel";
>>>            bootargs = "console=ttyAMA0 init=/bin/sh";
>>>        };
>>> 
>>>        module@2 {
>>>            compatible = "multiboot,ramdisk", "multiboot,module";
>>>            uefi,binary = "domA-ramdisk";
>>>        };
>>>    };
>>> };
>>> 
>>> With this approach, the change is quite minimal to move between an classic U-boot boot and EFI boot.
>> Ok now I see, yes this approach can work and can save some code, in the current code we have that if
>> a "multiboot,module” is found in the dtb, the Xen EFI configuration file is skipped, but if we use the
>> module@XX {} without the compatible it can work, the UEFI stub will load the binary and update all
>> the needed properties (compatible, reg).
> With my proposal, you don't know whether the binary is a kernel, ramdisk... So you wouldn't be able to recreate the compatible properly.
> 
> But the behavior of the UEFI stub can be modified. We could say that if there is a "xen,domain" then use the configuration file to fetch the binaries.

Yes you are right, or we can introduce domu_kernel, domu_ramdisk, domu_dtb with the same syntax of your binary= keyword, that would be enough
to select the right compatible, instead the right module is identified by the string.

> 
> Cheers,
> 
> -- 
> Julien Grall

Re: [RFC PATCH] xen/design: Add design for EFI dom0less system start

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 7627 bytes --]

On Tue, 7 Sep 2021, Julien Grall wrote:
> On 07/09/2021 12:51, Luca Fancellu wrote:
> > > On 7 Sep 2021, at 10:35, Julien Grall <julien@xen.org> wrote:
> > > 
> > > Hi Luca,
> > > 
> > > On 07/09/2021 07:52, Luca Fancellu wrote:
> > > > Add a design describing a proposal to improve the EFI
> > > > configuration file, adding keywords to describe domU
> > > > guests and allowing to start a dom0less system.
> > > > Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
> > > > ---
> > > >   docs/designs/efi-arm-dom0less.md | 105 +++++++++++++++++++++++++++++++
> > > >   1 file changed, 105 insertions(+)
> > > >   create mode 100644 docs/designs/efi-arm-dom0less.md
> > > > diff --git a/docs/designs/efi-arm-dom0less.md
> > > > b/docs/designs/efi-arm-dom0less.md
> > > > new file mode 100644
> > > > index 0000000000..8d8fa2243f
> > > > --- /dev/null
> > > > +++ b/docs/designs/efi-arm-dom0less.md
> > > > @@ -0,0 +1,105 @@
> > > > +# Xen EFI configuration file
> > > > +
> > > > +The current configuration file used by Xen when it is started as an EFI
> > > > +application is considering only the dom0 guest and doesn't have any
> > > > +property to describe and load in memory domU guests.
> > > 
> > >  From my understanding, the problem is less about properties (we already
> > > have them in the Device-Tree) but more about where are the binaries
> > > located in memory as we don't know in advance.
> > 
> > Hi Julien,
> Hi Luca,
> 
> > I think I used the wrong word there, I meant “keyword” instead of “property”
> > because I was referring about the
> > lack of keywords to describe a domu guest in the Xen EFI configuration file.
> > 
> > I agree with you that on systems with static allocation, the kernel and
> > ramdisk binaries must be at certain locations
> > that are out of control when we use the EFI boot services, the thing we can
> > do is provide a keyword to specify the
> > addresses and then use the CopyMem() function to relocate the kernel/ramdisk
> > in the address we want.
> 
> I wasn't specifically referring to static allocation here, sorry if this
> wasn't clear. I was pointing out that most of the information you create in
> the xen.cfg is going to be similar to what we already provide in the
> Device-Tree.
> 
> My main concern is everytime we add a new feature in Dom0less, a developer
> would need to write code for the DT and UEFI. This will increase the code size
> and maintenance.
> 
> The same can be said for the admin as if they want to switch from plain U-boot
> to UEFI, they would also need to fully rewrite the bindings.
> 
> AFAICT, most of the information provided in the Device-Tree are usable even
> when using UEFI. So I would prefer if we try to re-use what's existing. This
> is what my proposal below was about.
> 
> > 
> > > 
> > > So I would like to propose something that build on top of the Device-Tree
> > > work we did. Note this is early thoughts.
> > > 
> > > The problematic nodes in the DT are:
> > > 
> > >         module@0x4a000000 {
> > >             compatible = "multiboot,kernel", "multiboot,module";
> > >             reg = <0x0 0x4a000000 0xffffff>;
> > >             bootargs = "console=ttyAMA0 init=/bin/sh";
> > >         };
> > > 
> > >         module@0x4b000000 {
> > >             compatible = "multiboot,ramdisk", "multiboot,module";
> > >             reg = <0x0 0x4b000000 0xffffff>;
> > >         };
> > > 
> > > In particular the property "reg" cannot be known in advance because the
> > > UEFI stub will be responsible to load the binaries in memory.
> > 
> > Yes that’s true, the UEFI stub is using from the UEFI boot service the
> > AllocatePages function that is giving back an address out of our control,
> > then using another function the binary is read from the disk and copied at
> > that address, finally the UEFI stub is writing the node in the device tree
> > that
> > will be used by Xen later.
> 
> I am not sure to follow. Are you saying the UEFI stub will create the dom0less
> node in the DT based on the xen.cfg?
> 
> > 
> > > 
> > > What we could do is providing a list of binaries to load and associate a
> > > key for each of them. Something like:
> > > 
> > > binary=<binary> <key>
> > > binary=<binary2> <key2>
> > > ....
> > > 
> > > We can then replace the property "reg" with a new property "uefi,key" that
> > > will contain the name of the binary.
> > > 
> > > What do you think?
> > 
> > Here I’m lost, because I don’t understand what we are going to do with the
> > name of the binary.
> 
> <binaryX> would be used by the UEFI stub to load the binary in memory. Each
> binary will have a <keyX> which helps to refer them in the Device-Tree. To
> give a concrete example, let say we have two dom0less domains:
>   - DomA: 2 vCPUs, 128MB
>   - DomB: 3 vCPUs, 512MB
> 
> DomA and DomB will be using the same kernel but a different ramdisk. xen.cfg,
> would look like:
> 
> [global]
> default=section1
> 
> [section1]
> options=console=vga,com1 com1=57600 loglvl=all noreboot
> kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
> ramdisk=initrd-3.0.31-0.4-xen
> xsm=<filename>
> dtb=devtree.dtb
> binary=vmlinuz-guest domu-kernel
> binary=ramdisk-domA.img domA-ramdisk
> binary=ramdisk-domB.img domB-ramdisk
> 
> The chosen node in the DT would look like:
> 
> chosen {
>     domU1 {
>         compatible = "xen,domain";
>         #address-cells = <0x2>;
>         #size-cells = <0x1>;
>         memory = <0 0x8000000>;
>         cpus = <2>;
> 
>         module@1 {
>             compatible = "multiboot,kernel", "multiboot,module";
>             uefi,binary = "domu-kernel";
>             bootargs = "console=ttyAMA0 init=/bin/sh";
>         };
> 
>         module@2 {
>             compatible = "multiboot,ramdisk", "multiboot,module";
>             uefi,binary = "domA-ramdisk";
>         };
>     };
> 
>     domU2 {
>         compatible = "xen,domain";
>         #address-cells = <0x3>;
>         #size-cells = <0x1>;
>         memory = <0 0x20000000>;
>         cpus = <3>;
> 
>         module@1 {
>             compatible = "multiboot,kernel", "multiboot,module";
>             uefi,binary = "domu-kernel";
>             bootargs = "console=ttyAMA0 init=/bin/sh";
>         };
> 
>         module@2 {
>             compatible = "multiboot,ramdisk", "multiboot,module";
>             uefi,binary = "domA-ramdisk";
>         };
>     };
> };
> 
> With this approach, the change is quite minimal to move between an classic
> U-boot boot and EFI boot.

Great idea! I think it is good to try to reuse Device Tree, and using it
as configuration is well aligned with other projects too (e.g.  Zephyr).


There are a few options for the bindings. These are some ideas.

If we are just going to specify a filename and a reference with the
"binary" key=value pair, then we could get rid of it entirely and just
write the filename directly in device tree:

    domU1 {
        compatible = "xen,domain";
        #address-cells = <0x2>;
        #size-cells = <0x1>;
        memory = <0 0x8000000>;
        cpus = <2>;

        module@1 {
            compatible = "multiboot,kernel", "multiboot,module";
            uefi,binary = "vmlinuz-guest";
            bootargs = "console=ttyAMA0 init=/bin/sh";
        };
    };

Then we don't even need the binary key=value pair, and especially the
device tree becomes "self contained" as in we don't need to resolve the
reference to "binary=value ref" to understand it.


(Also worth noting as inspiration the HyperLaunch binding that is using
the "mb-index" property as index in the multiboot module chain. Not
directly applicable, but interesting.)

Re: [RFC PATCH] xen/design: Add design for EFI dom0less system start

[Luca Fancellu]

> On 8 Sep 2021, at 02:09, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Tue, 7 Sep 2021, Julien Grall wrote:
>> On 07/09/2021 12:51, Luca Fancellu wrote:
>>>> On 7 Sep 2021, at 10:35, Julien Grall <julien@xen.org> wrote:
>>>> 
>>>> Hi Luca,
>>>> 
>>>> On 07/09/2021 07:52, Luca Fancellu wrote:
>>>>> Add a design describing a proposal to improve the EFI
>>>>> configuration file, adding keywords to describe domU
>>>>> guests and allowing to start a dom0less system.
>>>>> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
>>>>> ---
>>>>>  docs/designs/efi-arm-dom0less.md | 105 +++++++++++++++++++++++++++++++
>>>>>  1 file changed, 105 insertions(+)
>>>>>  create mode 100644 docs/designs/efi-arm-dom0less.md
>>>>> diff --git a/docs/designs/efi-arm-dom0less.md
>>>>> b/docs/designs/efi-arm-dom0less.md
>>>>> new file mode 100644
>>>>> index 0000000000..8d8fa2243f
>>>>> --- /dev/null
>>>>> +++ b/docs/designs/efi-arm-dom0less.md
>>>>> @@ -0,0 +1,105 @@
>>>>> +# Xen EFI configuration file
>>>>> +
>>>>> +The current configuration file used by Xen when it is started as an EFI
>>>>> +application is considering only the dom0 guest and doesn't have any
>>>>> +property to describe and load in memory domU guests.
>>>> 
>>>> From my understanding, the problem is less about properties (we already
>>>> have them in the Device-Tree) but more about where are the binaries
>>>> located in memory as we don't know in advance.
>>> 
>>> Hi Julien,
>> Hi Luca,
>> 
>>> I think I used the wrong word there, I meant “keyword” instead of “property”
>>> because I was referring about the
>>> lack of keywords to describe a domu guest in the Xen EFI configuration file.
>>> 
>>> I agree with you that on systems with static allocation, the kernel and
>>> ramdisk binaries must be at certain locations
>>> that are out of control when we use the EFI boot services, the thing we can
>>> do is provide a keyword to specify the
>>> addresses and then use the CopyMem() function to relocate the kernel/ramdisk
>>> in the address we want.
>> 
>> I wasn't specifically referring to static allocation here, sorry if this
>> wasn't clear. I was pointing out that most of the information you create in
>> the xen.cfg is going to be similar to what we already provide in the
>> Device-Tree.
>> 
>> My main concern is everytime we add a new feature in Dom0less, a developer
>> would need to write code for the DT and UEFI. This will increase the code size
>> and maintenance.
>> 
>> The same can be said for the admin as if they want to switch from plain U-boot
>> to UEFI, they would also need to fully rewrite the bindings.
>> 
>> AFAICT, most of the information provided in the Device-Tree are usable even
>> when using UEFI. So I would prefer if we try to re-use what's existing. This
>> is what my proposal below was about.
>> 
>>> 
>>>> 
>>>> So I would like to propose something that build on top of the Device-Tree
>>>> work we did. Note this is early thoughts.
>>>> 
>>>> The problematic nodes in the DT are:
>>>> 
>>>>        module@0x4a000000 {
>>>>            compatible = "multiboot,kernel", "multiboot,module";
>>>>            reg = <0x0 0x4a000000 0xffffff>;
>>>>            bootargs = "console=ttyAMA0 init=/bin/sh";
>>>>        };
>>>> 
>>>>        module@0x4b000000 {
>>>>            compatible = "multiboot,ramdisk", "multiboot,module";
>>>>            reg = <0x0 0x4b000000 0xffffff>;
>>>>        };
>>>> 
>>>> In particular the property "reg" cannot be known in advance because the
>>>> UEFI stub will be responsible to load the binaries in memory.
>>> 
>>> Yes that’s true, the UEFI stub is using from the UEFI boot service the
>>> AllocatePages function that is giving back an address out of our control,
>>> then using another function the binary is read from the disk and copied at
>>> that address, finally the UEFI stub is writing the node in the device tree
>>> that
>>> will be used by Xen later.
>> 
>> I am not sure to follow. Are you saying the UEFI stub will create the dom0less
>> node in the DT based on the xen.cfg?
>> 
>>> 
>>>> 
>>>> What we could do is providing a list of binaries to load and associate a
>>>> key for each of them. Something like:
>>>> 
>>>> binary=<binary> <key>
>>>> binary=<binary2> <key2>
>>>> ....
>>>> 
>>>> We can then replace the property "reg" with a new property "uefi,key" that
>>>> will contain the name of the binary.
>>>> 
>>>> What do you think?
>>> 
>>> Here I’m lost, because I don’t understand what we are going to do with the
>>> name of the binary.
>> 
>> <binaryX> would be used by the UEFI stub to load the binary in memory. Each
>> binary will have a <keyX> which helps to refer them in the Device-Tree. To
>> give a concrete example, let say we have two dom0less domains:
>>  - DomA: 2 vCPUs, 128MB
>>  - DomB: 3 vCPUs, 512MB
>> 
>> DomA and DomB will be using the same kernel but a different ramdisk. xen.cfg,
>> would look like:
>> 
>> [global]
>> default=section1
>> 
>> [section1]
>> options=console=vga,com1 com1=57600 loglvl=all noreboot
>> kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
>> ramdisk=initrd-3.0.31-0.4-xen
>> xsm=<filename>
>> dtb=devtree.dtb
>> binary=vmlinuz-guest domu-kernel
>> binary=ramdisk-domA.img domA-ramdisk
>> binary=ramdisk-domB.img domB-ramdisk
>> 
>> The chosen node in the DT would look like:
>> 
>> chosen {
>>    domU1 {
>>        compatible = "xen,domain";
>>        #address-cells = <0x2>;
>>        #size-cells = <0x1>;
>>        memory = <0 0x8000000>;
>>        cpus = <2>;
>> 
>>        module@1 {
>>            compatible = "multiboot,kernel", "multiboot,module";
>>            uefi,binary = "domu-kernel";
>>            bootargs = "console=ttyAMA0 init=/bin/sh";
>>        };
>> 
>>        module@2 {
>>            compatible = "multiboot,ramdisk", "multiboot,module";
>>            uefi,binary = "domA-ramdisk";
>>        };
>>    };
>> 
>>    domU2 {
>>        compatible = "xen,domain";
>>        #address-cells = <0x3>;
>>        #size-cells = <0x1>;
>>        memory = <0 0x20000000>;
>>        cpus = <3>;
>> 
>>        module@1 {
>>            compatible = "multiboot,kernel", "multiboot,module";
>>            uefi,binary = "domu-kernel";
>>            bootargs = "console=ttyAMA0 init=/bin/sh";
>>        };
>> 
>>        module@2 {
>>            compatible = "multiboot,ramdisk", "multiboot,module";
>>            uefi,binary = "domA-ramdisk";
>>        };
>>    };
>> };
>> 
>> With this approach, the change is quite minimal to move between an classic
>> U-boot boot and EFI boot.
> 
> Great idea! I think it is good to try to reuse Device Tree, and using it
> as configuration is well aligned with other projects too (e.g.  Zephyr).
> 
> 
> There are a few options for the bindings. These are some ideas.
> 
> If we are just going to specify a filename and a reference with the
> "binary" key=value pair, then we could get rid of it entirely and just
> write the filename directly in device tree:
> 
>    domU1 {
>        compatible = "xen,domain";
>        #address-cells = <0x2>;
>        #size-cells = <0x1>;
>        memory = <0 0x8000000>;
>        cpus = <2>;
> 
>        module@1 {
>            compatible = "multiboot,kernel", "multiboot,module";
>            uefi,binary = "vmlinuz-guest";
>            bootargs = "console=ttyAMA0 init=/bin/sh";
>        };
>    };
> 
> Then we don't even need the binary key=value pair, and especially the
> device tree becomes "self contained" as in we don't need to resolve the
> reference to "binary=value ref" to understand it.
> 
> 
> (Also worth noting as inspiration the HyperLaunch binding that is using
> the "mb-index" property as index in the multiboot module chain. Not
> directly applicable, but interesting.)

Hi Stefano,

Yes this is an amazing idea, it will solve every issue without introducing new keywords in the config file
and it maintains the compatibility.
Now in the current code there is a check for modules that are “multiboot,module”, if any is found the configuration
file is skipped, but if we change this bit to check only for modules whose parent is the chosen node, we can
distinguish very well between the dom0 and domus kernel.

If we all agree with this, I will update the design.

Cheers,

Luca