Re: [PATCH 2/2] arm/efi: Use dom0less configuration when using EFI boot

[Luca Fancellu <Luca.Fancellu@arm.com>]

> On 17 Sep 2021, at 23:33, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Fri, 17 Sep 2021, Luca Fancellu wrote:
>>> On 16 Sep 2021, at 21:16, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>> 
>>> On Thu, 16 Sep 2021, Jan Beulich wrote:
>>>> On 16.09.2021 17:07, Luca Fancellu wrote:
>>>>> I explain here my understanding on dom0less, this feature is used to start domUs at
>>>>> Xen boot in parallel, the name is misleading but it doesn’t require dom0 to be absent.
>>>>> 
>>>>> So if you have a dom0 kernel embed in the image, it's completely fine to start it and it 
>>>>> doesn’t have to be skipped.
>>>>> 
>>>>> Here the possible user cases:
>>>>> 1) start only dom0 [dom0 modules on xen.cfg or embedded in Xen image]
>>>>> 2) start only domUs, true dom0less [no dom0 modules on xen.cfg or embedded in Xen image, domUs on DT]
>>>>> 3) start dom0 and domUs [(dom0 modules on xen.cfg or embedded in Xen image) and domUs on DT]
>>>> 
>>>> If that's the intention - fine. Stefano?
>>> 
>> 
>> Hi Stefano,
>> 
>>> What do you mean by dom0 modules embedded in the Xen image? I am not
>>> familiar with it, but I imagine it doesn't involve any multiboot,module
>>> nodes in device tree, right?
>>> 
>>> Putting aside "dom0 modules embedded in Xen image" that I didn't fully
>>> understand, there are three ways to load Dom0:
>>> 
>>> - dom0 kernel (and ramdisk, optional) on xen.cfg
>>> - dom0 kernel (and ramdisk, optional) on device tree using the "reg" property
>>> - dom0 kernel (and ramdisk, optional) on device tree using the "uefi,binary" property
>> 
>> True for the #1 and #2, the last one is not implemented. The uefi,binary property
>> for now is only used to load domU modules.
> 
> Yeah, it is no problem that is not currently implemented, but from a
> device tree binding / efi interface perspective it should be possible.
> 
> 
>>> Then, the other use cases are:
>>> 
>>> - true dom0less, domUs on device tree using the "reg" property
>>> - true dom0less, domUs on device tree using the "uefi,binary" property
>>> 
>>> And of course all the possible combinations between Dom0 and DomU
>>> loading.
>>> 
>>> 
>>> Currently, patch #1 checks for the presence of a Dom0 kernel node and, if
>>> present, it decides not to proceed with xen.cfg. So if the Dom0 kernel
>>> node is *not* present, efi_arch_use_config_file returns true.
>>> 
>>> However, this could be a true dom0less configuration without any Dom0
>>> kernel. If so, you might not want to load xen.cfg at all because it is
>>> not needed. In a true dom0less configuration, we probably want
>>> efi_arch_use_config_file to return false.
>> 
>> In a true dom0less configuration we might need to read xen.cfg to retrieve the
>> Xen command line, 
> 
> The Xen command line could also be on device tree
> (/chosen/xen,xen-bootargs).
> 
> 
>> but following the actual implementation of the common code
>> there is more. I’m going to explain.
>> 
>> What efi_arch_use_config_file really does is not only choosing to read xen.cfg
>> or not. Following the common code (xen/common/efi/boot.c) and what its result activate
>> along the path, it basically decides if the UEFI stub is used as a loader from filesystem
>> or not. We need the UEFI stub as a loader to be 100% UEFI and load our modules.
>> 
>> The original check basically says “if there are multiboot,module in the DT, then some
>> bootloader has loaded in memory the required modules so I’m not gonna load anything
>> from the filesystem because I assume it puts everything in place for me to boot.”
> 
> OK, I am following. It looks like this is the source of the issue.
> 
> 
>>> From misc/efi.txt:
>> When booted as an EFI application, Xen requires a configuration file as described below unless a bootloader,
>> such as GRUB, has loaded the modules and describes them in the device tree provided to Xen. If a bootloader
>> provides a device tree containing modules then any configuration files are ignored, and the bootloader is
>> responsible for populating all relevant device tree nodes.
>> 
>> What I’m doing in patch #1 is restricting that check to just the multiboot,module that are
>> Dom0 module, why? Because with the introduction of dom0less we need to specify
>> multiboot,modules for domUs, but the presence or not of dom0 modules is the only
>> Information we need to understand if the user decided to start Xen with everything
>> in places (modules in memory, xen command line, dtb) or if the job is demanded to the
>> UEFI stub and its configuration file.
> 
> I don't think so. Imagine a case where the user has everything in device
> tree, doesn't need xen.cfg, but dom0 and domUs are specified as
> uefi,binary.
> 
> We don't want xen.cfg but we do want to be able to load files from the
> filesystem. This might not be currently implemented but from an bindings
> perspective it should be possible.
> 
> 
>> By the configuration file you can also load in memory the Xen dtb, so Xen can
>> be started as an EFI application without the DTB and then load it using the EFI stub.
> 
> This can be very useful but it would follow the !fdt check and return
> true from efi_arch_use_config_file. So it doesn't really conflict with
> anything we would around multiboot,module and xen,cfg-loading.
> 
> 
>> I’m not against this new property “xen,cfg-loading”, I just think it is not needed because
>> we have all the information we need without it and in any case we need to read the
>> configuration file because otherwise we won’t have access to the Xen command line.
> 
> We don't necessarely need to read the Xen command line from xen.cfg :-)
> 
> 
>> Now I’m going to show you examples of all use cases that are valid with the introduction
>> of this serie:
>> 
>> 1) Start Xen as EFI application and load only Dom0
>> 
>>    Xen.cfg:
>>    [global]
>>    default=xen_dom0
>> 
>>    [xen_dom0]
>>    options=<Xen command line>
>>    kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
>>    ramdisk=initrd-3.0.31-0.4-xen
>>    dtb=<xen DTB>
>> 
>>    DT:
>>    {no modification}
>> 
>> 2) Start Xen as EFI application to load a true dom0less setup
>> 
>>    Xen.cfg:
>>    [global]
>>    default=xen_true_dom0less
>> 
>>    [xen_true_dom0less]
>>    options=<Xen command line>
>>    dtb=<xen DTB>
>> 
>>    DT:
>>    chosen {
>>        #size-cells = <0x1>;
>> 	#address-cells = <0x1>;
>> 
>> 	domU1 {
>>            #size-cells = <0x1>; 
>>            #address-cells = <0x1>;
>>            compatible = "xen,domain”;
>>            cpus = <1>;
>>            memory = <0 0xC0000>;
>> 
>>            module@1 {
>>                compatible = "multiboot,kernel", "multiboot,module”;
>>                bootargs = "console=ttyAMA0 root=/dev/ram0 rw”;
>>                uefi,binary = “domU_kernel1.bin"
>>            };
>> 
>>            module@2 {
>>                compatible = “multiboot,ramdisk", "multiboot,module”;
>>                uefi,binary = “domU_ramdisk1.bin"
>>            };
>> 
>>            module@3 {
>>                compatible = "multiboot,device-tree", "multiboot,module”;
>>                uefi,binary = “domU_passthrough1.dtb"
>>            }; 
>>        };
>> 
>>        domU2 { <as above> };
>>    }
>> 
>> 3) Start Xen as EFI application to load Dom0 and DomUs
>> 
>>    Xen.cfg:
>>    [global]
>>    default=xen_dom0_domUs
>> 
>>    [xen_dom0_domUs]
>>    options=<Xen command line>
>>    kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
>>    ramdisk=initrd-3.0.31-0.4-xen
>>    dtb=<xen DTB>
>> 
>>    DT:
>>    chosen {
>>        #size-cells = <0x1>;
>> 	#address-cells = <0x1>;
>> 
>> 	domU1 {
>>            #size-cells = <0x1>; 
>>            #address-cells = <0x1>;
>>            compatible = "xen,domain”;
>>            cpus = <1>;
>>            memory = <0 0xC0000>;
>> 
>>            module@1 {
>>                compatible = "multiboot,kernel", "multiboot,module”;
>>                bootargs = "console=ttyAMA0 root=/dev/ram0 rw”;
>>                uefi,binary = “domU_kernel1.bin"
>>            };
>> 
>>            module@2 {
>>                compatible = “multiboot,ramdisk", "multiboot,module”;
>>                uefi,binary = “domU_ramdisk1.bin"
>>            };
>> 
>>            module@3 {
>>                compatible = "multiboot,device-tree", "multiboot,module”;
>>                uefi,binary = “domU_passthrough1.dtb"
>>            }; 
>>        };
>> 
>>        domU2 { <as above> };
>>    }
>> 
>> So as you see every case is covered without the introduction of the
>> property.
>> 
>> Please let me know what do you think.
> 

Hi Stefano,

> I think that from an interface perspective (not a code perspective) we
> need to be able to account for cases like:
> 
> 4) Start Xen as EFI application and load only Dom0
> no Xen.cfg
> DT:
>  xen,xen-bootargs
>  dom0/uefi,binary
>  domUs/uefi,binary
> 
> 
> But in any case, even disregarding this case, past experience has taught
> me that it is always better to have an explicit flag to trigger a new
> behavior, rather than relying on "guesswork". If we introduce
> "xen,cfg-loading", we are going to be a lot more future-proof that if we
> don't introduce it in terms of backward and forward compatibility in
> case we need to change anything.

I see your point, for sure the DT is a more powerful tool than the simple
text configuration file and it would be the best interface.
However I think we are moving into the direction where x86 and arm
are going to diverge even if we can have a common interface for them
(the configuration file).

For that reason I’m asking if you would be willing to accept a solution
where we introduce a new keyword in the configuration file:

dom0less=<dtb> OR domu_guests=<dtb> OR I’m open to suggestion.

Where the pointed dtb contains the domU domains:

/dts-v1/;

/ {
    /* #*cells are here to keep DTC happy */
    #address-cells = <2>;
    #size-cells = <2>;

    domU1 {
           #size-cells = <0x1>; 
           #address-cells = <0x1>;
           compatible = "xen,domain”;
           cpus = <1>;
           memory = <0 0xC0000>;

           module@1 {
               compatible = "multiboot,kernel", "multiboot,module”;
               bootargs = "console=ttyAMA0 root=/dev/ram0 rw”;
               uefi,binary = “domU_kernel1.bin"
           };

           module@2 {
               compatible = “multiboot,ramdisk", "multiboot,module”;
               uefi,binary = “domU_ramdisk1.bin"
           };

           module@3 {
               compatible = "multiboot,device-tree", "multiboot,module”;
               uefi,binary = “domU_passthrough1.dtb"
           }; 
    };

    domU2 { <as above> };

};


So that the user cases we discussed are valid:

1) Start Xen and load Dom0:
   
   Xen.cfg:
   [global]
   default=xen

   [xen]
   options=<Xen command line>
   kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
   ramdisk=initrd-3.0.31-0.4-xen
   dtb=<xen DTB>

2) Start Xen and load only domUs (true dom0less)

   Xen.cfg:
   [global]
   default=xen

   [xen]
   options=<Xen command line>
   dom0less=<dom0less DTB>
   dtb=<xen DTB>

3) Start Xen and load Dom0 and DomUs

   Xen.cfg:
   [global]
   default=xen

   [xen]
   options=<Xen command line>
   kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
   ramdisk=initrd-3.0.31-0.4-xen
   dom0less=<dom0less DTB>
   dtb=<xen DTB>


With this change we will be consistent across x86 and arm UEFI boot
start procedure, we won’t touch the current check on multiboot,modules
because it will be valid, we will have a way to boot dom0less and it
requires less testing for the changing in the common code.

Please let me know what do you think about that.

Cheers,
Luca