UEFI support in ARM DomUs

UEFI support in ARM DomUs

[Roman Shaposhnik]

Hi!

with a lot of help from Stefano, we're getting RPi4 support in
Project EVE pretty much on par between KVM and Xen.

One big area that still remains is supporting UEFI boot sequence
for DomUs. With KVM, given the qemu virt device model this is
as simple as using either stock UEFI build for arm or even U-Boot
EFI emulation environment and passing it via -bios option.

Obviously with Xen on ARM we don't have the device model so
my understanding is that the easiest way we can support it would
be to port UEFI's OvmfPkg/OvmfXen target to ARM (it seems to
be currently exclusively X64).

So here's my first question: if there's anybody on this list who had
a hand in implementing OvmfPkg/OvmfXen can you please share
your thoughts on how much work that port may be (or whether it is
even feasible -- I may totally be missing something really obvious here).

And as long as I've got your attention: two more questions:
   1.. compared to the above, would porting pvgrub to ARM be any
   easier or more difficult?

   2. same question for teaching u-boot about PV calls.

Thanks,
Roman.

P.S. Oh and I guess between:
   0. OvmfPkg/OvmfXen on ARM64
   1. pvgrub on ARM64
   2. u-boot/EFI emulation with PV calls backend
I didn't miss any other obvious way of making UEFI-aware VM images
to boot on Xen ARM64 DomU, right?

Re: UEFI support in ARM DomUs

[Julien Grall]

On Sun, 31 May 2020 at 23:05, Roman Shaposhnik <roman@zededa.com> wrote:
> Hi!
>
> with a lot of help from Stefano, we're getting RPi4 support in
> Project EVE pretty much on par between KVM and Xen.
>
> One big area that still remains is supporting UEFI boot sequence
> for DomUs. With KVM, given the qemu virt device model this is
> as simple as using either stock UEFI build for arm or even U-Boot
> EFI emulation environment and passing it via -bios option.
>
> Obviously with Xen on ARM we don't have the device model so
> my understanding is that the easiest way we can support it would
> be to port UEFI's OvmfPkg/OvmfXen target to ARM (it seems to
> be currently exclusively X64).

EDK2 has been supporting Xen on Arm for the past 5 years. We don't use
OvmfPkg/OvmfXen but ArmVirtPkg/ArmVirtXen (see [1]).
I haven't tried to build it recently, but I should be able to help if
there is any issue with it.

Cheers,

[1] https://github.com/tianocore/edk2/blob/master/ArmVirtPkg/ArmVirtXen.fdf

Re: UEFI support in ARM DomUs

[Roman Shaposhnik]

Hi Julien!

On Sun, May 31, 2020 at 3:24 PM Julien Grall <julien.grall.oss@gmail.com> wrote:
>
> On Sun, 31 May 2020 at 23:05, Roman Shaposhnik <roman@zededa.com> wrote:
> > Hi!
> >
> > with a lot of help from Stefano, we're getting RPi4 support in
> > Project EVE pretty much on par between KVM and Xen.
> >
> > One big area that still remains is supporting UEFI boot sequence
> > for DomUs. With KVM, given the qemu virt device model this is
> > as simple as using either stock UEFI build for arm or even U-Boot
> > EFI emulation environment and passing it via -bios option.
> >
> > Obviously with Xen on ARM we don't have the device model so
> > my understanding is that the easiest way we can support it would
> > be to port UEFI's OvmfPkg/OvmfXen target to ARM (it seems to
> > be currently exclusively X64).
>
> EDK2 has been supporting Xen on Arm for the past 5 years. We don't use
> OvmfPkg/OvmfXen but ArmVirtPkg/ArmVirtXen (see [1]).
> I haven't tried to build it recently, but I should be able to help if
> there is any issue with it.
>
> Cheers,
>
> [1] https://github.com/tianocore/edk2/blob/master/ArmVirtPkg/ArmVirtXen.fdf

This is really, really awesome -- I guess it would be really helpful to document
this someplace on the ARM/Xen wiki (I can volunteer if someone can grant
me the karma).

I've built XEN_EFI.fd and the rest worked out like a charm.

All on Raspberry Pi 4!

Thanks,
Roman.

Re: UEFI support in ARM DomUs

[Stefano Stabellini]

+ George

On Sun, 31 May 2020, Roman Shaposhnik wrote:
> Hi Julien!
> 
> On Sun, May 31, 2020 at 3:24 PM Julien Grall <julien.grall.oss@gmail.com> wrote:
> >
> > On Sun, 31 May 2020 at 23:05, Roman Shaposhnik <roman@zededa.com> wrote:
> > > Hi!
> > >
> > > with a lot of help from Stefano, we're getting RPi4 support in
> > > Project EVE pretty much on par between KVM and Xen.
> > >
> > > One big area that still remains is supporting UEFI boot sequence
> > > for DomUs. With KVM, given the qemu virt device model this is
> > > as simple as using either stock UEFI build for arm or even U-Boot
> > > EFI emulation environment and passing it via -bios option.
> > >
> > > Obviously with Xen on ARM we don't have the device model so
> > > my understanding is that the easiest way we can support it would
> > > be to port UEFI's OvmfPkg/OvmfXen target to ARM (it seems to
> > > be currently exclusively X64).
> >
> > EDK2 has been supporting Xen on Arm for the past 5 years. We don't use
> > OvmfPkg/OvmfXen but ArmVirtPkg/ArmVirtXen (see [1]).
> > I haven't tried to build it recently, but I should be able to help if
> > there is any issue with it.
> >
> > Cheers,
> >
> > [1] https://github.com/tianocore/edk2/blob/master/ArmVirtPkg/ArmVirtXen.fdf
> 
> This is really, really awesome -- I guess it would be really helpful to document
> this someplace on the ARM/Xen wiki (I can volunteer if someone can grant
> me the karma).

Regarding the wiki: yes please! Let George know if you don't have write access.


> I've built XEN_EFI.fd and the rest worked out like a charm.
> 
> All on Raspberry Pi 4!

Fantastic!

Re: UEFI support in ARM DomUs

[Roman Shaposhnik]

On Mon, Jun 1, 2020 at 9:12 AM Stefano Stabellini
<sstabellini@kernel.org> wrote:
>
> + George
>
> On Sun, 31 May 2020, Roman Shaposhnik wrote:
> > Hi Julien!
> >
> > On Sun, May 31, 2020 at 3:24 PM Julien Grall <julien.grall.oss@gmail.com> wrote:
> > >
> > > On Sun, 31 May 2020 at 23:05, Roman Shaposhnik <roman@zededa.com> wrote:
> > > > Hi!
> > > >
> > > > with a lot of help from Stefano, we're getting RPi4 support in
> > > > Project EVE pretty much on par between KVM and Xen.
> > > >
> > > > One big area that still remains is supporting UEFI boot sequence
> > > > for DomUs. With KVM, given the qemu virt device model this is
> > > > as simple as using either stock UEFI build for arm or even U-Boot
> > > > EFI emulation environment and passing it via -bios option.
> > > >
> > > > Obviously with Xen on ARM we don't have the device model so
> > > > my understanding is that the easiest way we can support it would
> > > > be to port UEFI's OvmfPkg/OvmfXen target to ARM (it seems to
> > > > be currently exclusively X64).
> > >
> > > EDK2 has been supporting Xen on Arm for the past 5 years. We don't use
> > > OvmfPkg/OvmfXen but ArmVirtPkg/ArmVirtXen (see [1]).
> > > I haven't tried to build it recently, but I should be able to help if
> > > there is any issue with it.
> > >
> > > Cheers,
> > >
> > > [1] https://github.com/tianocore/edk2/blob/master/ArmVirtPkg/ArmVirtXen.fdf
> >
> > This is really, really awesome -- I guess it would be really helpful to document
> > this someplace on the ARM/Xen wiki (I can volunteer if someone can grant
> > me the karma).
>
> Regarding the wiki: yes please! Let George know if you don't have write access.

Hey Geroge -- FWIW: my wiki account name is rvs -- please let me know
once you enable whatever needs to be enabled for my write access.

Thanks,
Roman.

Re: UEFI support in ARM DomUs

[George Dunlap]

> On Jun 2, 2020, at 11:50 PM, Roman Shaposhnik <roman@zededa.com> wrote:
> 
> On Mon, Jun 1, 2020 at 9:12 AM Stefano Stabellini
> <sstabellini@kernel.org> wrote:
>> 
>> + George
>> 
>> On Sun, 31 May 2020, Roman Shaposhnik wrote:
>>> Hi Julien!
>>> 
>>> On Sun, May 31, 2020 at 3:24 PM Julien Grall <julien.grall.oss@gmail.com> wrote:
>>>> 
>>>> On Sun, 31 May 2020 at 23:05, Roman Shaposhnik <roman@zededa.com> wrote:
>>>>> Hi!
>>>>> 
>>>>> with a lot of help from Stefano, we're getting RPi4 support in
>>>>> Project EVE pretty much on par between KVM and Xen.
>>>>> 
>>>>> One big area that still remains is supporting UEFI boot sequence
>>>>> for DomUs. With KVM, given the qemu virt device model this is
>>>>> as simple as using either stock UEFI build for arm or even U-Boot
>>>>> EFI emulation environment and passing it via -bios option.
>>>>> 
>>>>> Obviously with Xen on ARM we don't have the device model so
>>>>> my understanding is that the easiest way we can support it would
>>>>> be to port UEFI's OvmfPkg/OvmfXen target to ARM (it seems to
>>>>> be currently exclusively X64).
>>>> 
>>>> EDK2 has been supporting Xen on Arm for the past 5 years. We don't use
>>>> OvmfPkg/OvmfXen but ArmVirtPkg/ArmVirtXen (see [1]).
>>>> I haven't tried to build it recently, but I should be able to help if
>>>> there is any issue with it.
>>>> 
>>>> Cheers,
>>>> 
>>>> [1] https://github.com/tianocore/edk2/blob/master/ArmVirtPkg/ArmVirtXen.fdf
>>> 
>>> This is really, really awesome -- I guess it would be really helpful to document
>>> this someplace on the ARM/Xen wiki (I can volunteer if someone can grant
>>> me the karma).
>> 
>> Regarding the wiki: yes please! Let George know if you don't have write access.
> 
> Hey Geroge -- FWIW: my wiki account name is rvs -- please let me know
> once you enable whatever needs to be enabled for my write access.

Hmm, not sure if I have the appropriate permissions yet.  Let me look into this.

 -George

Re: UEFI support in ARM DomUs

[Julien Grall]

On 01/06/2020 05:11, Roman Shaposhnik wrote:
> Hi Julien!

Hi Roman,

> 
> On Sun, May 31, 2020 at 3:24 PM Julien Grall <julien.grall.oss@gmail.com> wrote:
>>
>> On Sun, 31 May 2020 at 23:05, Roman Shaposhnik <roman@zededa.com> wrote:
>>> Hi!
>>>
>>> with a lot of help from Stefano, we're getting RPi4 support in
>>> Project EVE pretty much on par between KVM and Xen.
>>>
>>> One big area that still remains is supporting UEFI boot sequence
>>> for DomUs. With KVM, given the qemu virt device model this is
>>> as simple as using either stock UEFI build for arm or even U-Boot
>>> EFI emulation environment and passing it via -bios option.
>>>
>>> Obviously with Xen on ARM we don't have the device model so
>>> my understanding is that the easiest way we can support it would
>>> be to port UEFI's OvmfPkg/OvmfXen target to ARM (it seems to
>>> be currently exclusively X64).
>>
>> EDK2 has been supporting Xen on Arm for the past 5 years. We don't use
>> OvmfPkg/OvmfXen but ArmVirtPkg/ArmVirtXen (see [1]).
>> I haven't tried to build it recently, but I should be able to help if
>> there is any issue with it.
>>
>> Cheers,
>>
>> [1] https://github.com/tianocore/edk2/blob/master/ArmVirtPkg/ArmVirtXen.fdf
> 
> This is really, really awesome -- I guess it would be really helpful to document
> this someplace on the ARM/Xen wiki (I can volunteer if someone can grant
> me the karma).

There used to be a page on the Linaro wiki when they did the port. But 
it looks like any Xen pages have been removed/relocated :(.

Anyway, a page on Xen Project wiki would definitely be appreciated.

> 
> I've built XEN_EFI.fd and the rest worked out like a charm.

Glad to hear that it worked :).

Cheers,

-- 
Julien Grall

RE: UEFI support in ARM DomUs

[Peng Fan]

Grall <julien@xen.org>;
> Nataliya Korovkina <malus.brandywine@gmail.com>
> Subject: UEFI support in ARM DomUs

We have made U-Boot run inside XEN DomU, but just only PV console part,
not implement other frontend drivers currently. Would this help for your
case if enable EFI in U-Boot?

Regards,
Peng.

> 
> Hi!
> 
> with a lot of help from Stefano, we're getting RPi4 support in Project EVE
> pretty much on par between KVM and Xen.
> 
> One big area that still remains is supporting UEFI boot sequence for DomUs.
> With KVM, given the qemu virt device model this is as simple as using either
> stock UEFI build for arm or even U-Boot EFI emulation environment and
> passing it via -bios option.
> 
> Obviously with Xen on ARM we don't have the device model so my
> understanding is that the easiest way we can support it would be to port
> UEFI's OvmfPkg/OvmfXen target to ARM (it seems to be currently exclusively
> X64).
> 
> So here's my first question: if there's anybody on this list who had a hand in
> implementing OvmfPkg/OvmfXen can you please share your thoughts on how
> much work that port may be (or whether it is even feasible -- I may totally be
> missing something really obvious here).
> 
> And as long as I've got your attention: two more questions:
>    1.. compared to the above, would porting pvgrub to ARM be any
>    easier or more difficult?
> 
>    2. same question for teaching u-boot about PV calls.
> 
> Thanks,
> Roman.
> 
> P.S. Oh and I guess between:
>    0. OvmfPkg/OvmfXen on ARM64
>    1. pvgrub on ARM64
>    2. u-boot/EFI emulation with PV calls backend I didn't miss any other
> obvious way of making UEFI-aware VM images to boot on Xen ARM64 DomU,
> right?

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/4/20 4:57 AM, Peng Fan wrote:
> Grall <julien@xen.org>;
>> Nataliya Korovkina <malus.brandywine@gmail.com>
>> Subject: UEFI support in ARM DomUs
> We have made U-Boot run inside XEN DomU, but just only PV console part,
> not implement other frontend drivers currently. Would this help for your
> case if enable EFI in U-Boot?

Well, we have a working PV block implementation on top of that on iMX8

platform, mostly ported from mini-os. Currently we are finalizing the work

and cleaning up (it's going to take a week or so hopefully). Then, we we'll post

it on our public github. We are also thinking about upstreaming the work, but it may

take quite some time if the whole idea fits u-boot's view on such an extension at all.

Regards,

Oleksandr

> Regards,
> Peng.
>
>> Hi!
>>
>> with a lot of help from Stefano, we're getting RPi4 support in Project EVE
>> pretty much on par between KVM and Xen.
>>
>> One big area that still remains is supporting UEFI boot sequence for DomUs.
>> With KVM, given the qemu virt device model this is as simple as using either
>> stock UEFI build for arm or even U-Boot EFI emulation environment and
>> passing it via -bios option.
>>
>> Obviously with Xen on ARM we don't have the device model so my
>> understanding is that the easiest way we can support it would be to port
>> UEFI's OvmfPkg/OvmfXen target to ARM (it seems to be currently exclusively
>> X64).
>>
>> So here's my first question: if there's anybody on this list who had a hand in
>> implementing OvmfPkg/OvmfXen can you please share your thoughts on how
>> much work that port may be (or whether it is even feasible -- I may totally be
>> missing something really obvious here).
>>
>> And as long as I've got your attention: two more questions:
>>     1.. compared to the above, would porting pvgrub to ARM be any
>>     easier or more difficult?
>>
>>     2. same question for teaching u-boot about PV calls.
>>
>> Thanks,
>> Roman.
>>
>> P.S. Oh and I guess between:
>>     0. OvmfPkg/OvmfXen on ARM64
>>     1. pvgrub on ARM64
>>     2. u-boot/EFI emulation with PV calls backend I didn't miss any other
>> obvious way of making UEFI-aware VM images to boot on Xen ARM64 DomU,
>> right?

Re: UEFI support in ARM DomUs

[Stefano Stabellini]

On Thu, 4 Jun 2020, Oleksandr Andrushchenko wrote:
> On 6/4/20 4:57 AM, Peng Fan wrote:
> > Grall <julien@xen.org>;
> >> Nataliya Korovkina <malus.brandywine@gmail.com>
> >> Subject: UEFI support in ARM DomUs
> > We have made U-Boot run inside XEN DomU, but just only PV console part,
> > not implement other frontend drivers currently. Would this help for your
> > case if enable EFI in U-Boot?
> 
> Well, we have a working PV block implementation on top of that on iMX8
> 
> platform, mostly ported from mini-os. Currently we are finalizing the work
> 
> and cleaning up (it's going to take a week or so hopefully). Then, we we'll post
> 
> it on our public github. We are also thinking about upstreaming the work, but it may
> 
> take quite some time if the whole idea fits u-boot's view on such an extension at all.

Yes please to both of you! :-)

In the meantime, while we wait for those changes to go upstream in
uboot, could you please post a branch on github and a link on this email
thread?

Maybe we should have a wikipage on wiki.xenproject.org about
work-in-progress uboot items.




> > Regards,
> > Peng.
> >
> >> Hi!
> >>
> >> with a lot of help from Stefano, we're getting RPi4 support in Project EVE
> >> pretty much on par between KVM and Xen.
> >>
> >> One big area that still remains is supporting UEFI boot sequence for DomUs.
> >> With KVM, given the qemu virt device model this is as simple as using either
> >> stock UEFI build for arm or even U-Boot EFI emulation environment and
> >> passing it via -bios option.
> >>
> >> Obviously with Xen on ARM we don't have the device model so my
> >> understanding is that the easiest way we can support it would be to port
> >> UEFI's OvmfPkg/OvmfXen target to ARM (it seems to be currently exclusively
> >> X64).
> >>
> >> So here's my first question: if there's anybody on this list who had a hand in
> >> implementing OvmfPkg/OvmfXen can you please share your thoughts on how
> >> much work that port may be (or whether it is even feasible -- I may totally be
> >> missing something really obvious here).
> >>
> >> And as long as I've got your attention: two more questions:
> >>     1.. compared to the above, would porting pvgrub to ARM be any
> >>     easier or more difficult?
> >>
> >>     2. same question for teaching u-boot about PV calls.
> >>
> >> Thanks,
> >> Roman.
> >>
> >> P.S. Oh and I guess between:
> >>     0. OvmfPkg/OvmfXen on ARM64
> >>     1. pvgrub on ARM64
> >>     2. u-boot/EFI emulation with PV calls backend I didn't miss any other
> >> obvious way of making UEFI-aware VM images to boot on Xen ARM64 DomU,
> >> right?

Re: UEFI support in ARM DomUs

[Roman Shaposhnik]

On Thu, Jun 4, 2020 at 8:31 AM Stefano Stabellini
<sstabellini@kernel.org> wrote:
>
> On Thu, 4 Jun 2020, Oleksandr Andrushchenko wrote:
> > On 6/4/20 4:57 AM, Peng Fan wrote:
> > > Grall <julien@xen.org>;
> > >> Nataliya Korovkina <malus.brandywine@gmail.com>
> > >> Subject: UEFI support in ARM DomUs
> > > We have made U-Boot run inside XEN DomU, but just only PV console part,
> > > not implement other frontend drivers currently. Would this help for your
> > > case if enable EFI in U-Boot?
> >
> > Well, we have a working PV block implementation on top of that on iMX8
> >
> > platform, mostly ported from mini-os. Currently we are finalizing the work
> >
> > and cleaning up (it's going to take a week or so hopefully). Then, we we'll post
> >
> > it on our public github. We are also thinking about upstreaming the work, but it may
> >
> > take quite some time if the whole idea fits u-boot's view on such an extension at all.
>
> Yes please to both of you! :-)
>
> In the meantime, while we wait for those changes to go upstream in
> uboot, could you please post a branch on github and a link on this email
> thread?
>
> Maybe we should have a wikipage on wiki.xenproject.org about
> work-in-progress uboot items.

Yes!!! Speaking of which -- I've been meaning to update the wiki
for quite a few things already, but I'm still waiting on someone (George?)
to give user 'rvs' proper karma.

Thanks,
Roman.

RE: UEFI support in ARM DomUs

[Peng Fan]

Hi Stefano,

> -----Original Message-----
> From: Stefano Stabellini [mailto:sstabellini@kernel.org]
> Sent: 2020年6月4日 23:32
> To: Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>
> Cc: Peng Fan <peng.fan@nxp.com>; Roman Shaposhnik
> <roman@zededa.com>; Xen-devel <xen-devel@lists.xenproject.org>; Stefano
> Stabellini <sstabellini@kernel.org>; Julien Grall <julien@xen.org>; Nataliya
> Korovkina <malus.brandywine@gmail.com>
> Subject: Re: UEFI support in ARM DomUs
> 
> On Thu, 4 Jun 2020, Oleksandr Andrushchenko wrote:
> > On 6/4/20 4:57 AM, Peng Fan wrote:
> > > Grall <julien@xen.org>;
> > >> Nataliya Korovkina <malus.brandywine@gmail.com>
> > >> Subject: UEFI support in ARM DomUs
> > > We have made U-Boot run inside XEN DomU, but just only PV console
> > > part, not implement other frontend drivers currently. Would this
> > > help for your case if enable EFI in U-Boot?
> >
> > Well, we have a working PV block implementation on top of that on iMX8
> >
> > platform, mostly ported from mini-os. Currently we are finalizing the
> > work
> >
> > and cleaning up (it's going to take a week or so hopefully). Then, we
> > we'll post
> >
> > it on our public github. We are also thinking about upstreaming the
> > work, but it may
> >
> > take quite some time if the whole idea fits u-boot's view on such an
> extension at all.
> 
> Yes please to both of you! :-)

The simple console part

https://source.codeaurora.org/external/imx/uboot-imx/tree/drivers/serial/serial_xen.c?h=imx_v2020.04_5.4.24_2.1.0

We enable U-Boot in DomU is mostly for android auto dual bootloader case.

It has the issue that only have console output after mmu enabled in U-Boot stage.

Regards,
Peng.

> 
> In the meantime, while we wait for those changes to go upstream in uboot,
> could you please post a branch on github and a link on this email thread?
> 
> Maybe we should have a wikipage on wiki.xenproject.org about
> work-in-progress uboot items.
> 
> 
> 
> 
> > > Regards,
> > > Peng.
> > >
> > >> Hi!
> > >>
> > >> with a lot of help from Stefano, we're getting RPi4 support in
> > >> Project EVE pretty much on par between KVM and Xen.
> > >>
> > >> One big area that still remains is supporting UEFI boot sequence for
> DomUs.
> > >> With KVM, given the qemu virt device model this is as simple as
> > >> using either stock UEFI build for arm or even U-Boot EFI emulation
> > >> environment and passing it via -bios option.
> > >>
> > >> Obviously with Xen on ARM we don't have the device model so my
> > >> understanding is that the easiest way we can support it would be to
> > >> port UEFI's OvmfPkg/OvmfXen target to ARM (it seems to be currently
> > >> exclusively X64).
> > >>
> > >> So here's my first question: if there's anybody on this list who
> > >> had a hand in implementing OvmfPkg/OvmfXen can you please share
> > >> your thoughts on how much work that port may be (or whether it is
> > >> even feasible -- I may totally be missing something really obvious here).
> > >>
> > >> And as long as I've got your attention: two more questions:
> > >>     1.. compared to the above, would porting pvgrub to ARM be any
> > >>     easier or more difficult?
> > >>
> > >>     2. same question for teaching u-boot about PV calls.
> > >>
> > >> Thanks,
> > >> Roman.
> > >>
> > >> P.S. Oh and I guess between:
> > >>     0. OvmfPkg/OvmfXen on ARM64
> > >>     1. pvgrub on ARM64
> > >>     2. u-boot/EFI emulation with PV calls backend I didn't miss any
> > >> other obvious way of making UEFI-aware VM images to boot on Xen
> > >> ARM64 DomU, right?

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/4/20 6:31 PM, Stefano Stabellini wrote:
> On Thu, 4 Jun 2020, Oleksandr Andrushchenko wrote:
>> On 6/4/20 4:57 AM, Peng Fan wrote:
>>> Grall <julien@xen.org>;
>>>> Nataliya Korovkina <malus.brandywine@gmail.com>
>>>> Subject: UEFI support in ARM DomUs
>>> We have made U-Boot run inside XEN DomU, but just only PV console part,
>>> not implement other frontend drivers currently. Would this help for your
>>> case if enable EFI in U-Boot?
>> Well, we have a working PV block implementation on top of that on iMX8
>>
>> platform, mostly ported from mini-os. Currently we are finalizing the work
>>
>> and cleaning up (it's going to take a week or so hopefully). Then, we we'll post
>>
>> it on our public github. We are also thinking about upstreaming the work, but it may
>>
>> take quite some time if the whole idea fits u-boot's view on such an extension at all.
> Yes please to both of you! :-)
>
> In the meantime, while we wait for those changes to go upstream in
> uboot, could you please post a branch on github and a link on this email
> thread?

It took a bit more time than we expected, but here we go [1]:

this is in form of a pull-request as we would love to hear from the

community and it is easier to discuss the code (please leave comments there)

1. There is code originating from MiniOS and work done by Peng, so we

would like to ask the respective copyright owners to raise their hands and

let us *fix inappropriate licensing* if any.

2. Please note, the series has a HACK to move the RAM base as it is expected by

our test platform (iMX8), so others will need to remove or modify that.

3. There is a limitation already noted by Peng that we do not have serial output

until MMU is setup, so we have introduced xen_early_printk helper which always

works, so you can use that for early stage debugging.

4. Minimal memory size supported is ~129M because of dtb placement by Xen tools

5. We use -D__XEN__ to access some of the hidden defines we need such as

GUEST_RAM0_BASE and the friends as there is no other way but manually defining the

same which is not cute.

Have fun,

Anastasiia, Oleksandr

>
> Maybe we should have a wikipage on wiki.xenproject.org about
> work-in-progress uboot items.
>
>
>
>
>>> Regards,
>>> Peng.
>>>
>>>> Hi!
>>>>
>>>> with a lot of help from Stefano, we're getting RPi4 support in Project EVE
>>>> pretty much on par between KVM and Xen.
>>>>
>>>> One big area that still remains is supporting UEFI boot sequence for DomUs.
>>>> With KVM, given the qemu virt device model this is as simple as using either
>>>> stock UEFI build for arm or even U-Boot EFI emulation environment and
>>>> passing it via -bios option.
>>>>
>>>> Obviously with Xen on ARM we don't have the device model so my
>>>> understanding is that the easiest way we can support it would be to port
>>>> UEFI's OvmfPkg/OvmfXen target to ARM (it seems to be currently exclusively
>>>> X64).
>>>>
>>>> So here's my first question: if there's anybody on this list who had a hand in
>>>> implementing OvmfPkg/OvmfXen can you please share your thoughts on how
>>>> much work that port may be (or whether it is even feasible -- I may totally be
>>>> missing something really obvious here).
>>>>
>>>> And as long as I've got your attention: two more questions:
>>>>      1.. compared to the above, would porting pvgrub to ARM be any
>>>>      easier or more difficult?
>>>>
>>>>      2. same question for teaching u-boot about PV calls.
>>>>
>>>> Thanks,
>>>> Roman.
>>>>
>>>> P.S. Oh and I guess between:
>>>>      0. OvmfPkg/OvmfXen on ARM64
>>>>      1. pvgrub on ARM64
>>>>      2. u-boot/EFI emulation with PV calls backend I didn't miss any other
>>>> obvious way of making UEFI-aware VM images to boot on Xen ARM64 DomU,
>>>> right?
[1] https://github.com/xen-troops/u-boot/pull/1

Re: UEFI support in ARM DomUs

[Julien Grall]

On 18/06/2020 06:22, Oleksandr Andrushchenko wrote:
> 
> On 6/4/20 6:31 PM, Stefano Stabellini wrote:
>> On Thu, 4 Jun 2020, Oleksandr Andrushchenko wrote:
>>> On 6/4/20 4:57 AM, Peng Fan wrote:
>>>> Grall <julien@xen.org>;
>>>>> Nataliya Korovkina <malus.brandywine@gmail.com>
>>>>> Subject: UEFI support in ARM DomUs
>>>> We have made U-Boot run inside XEN DomU, but just only PV console part,
>>>> not implement other frontend drivers currently. Would this help for your
>>>> case if enable EFI in U-Boot?
>>> Well, we have a working PV block implementation on top of that on iMX8
>>>
>>> platform, mostly ported from mini-os. Currently we are finalizing the work
>>>
>>> and cleaning up (it's going to take a week or so hopefully). Then, we we'll post
>>>
>>> it on our public github. We are also thinking about upstreaming the work, but it may
>>>
>>> take quite some time if the whole idea fits u-boot's view on such an extension at all.
>> Yes please to both of you! :-)
>>
>> In the meantime, while we wait for those changes to go upstream in
>> uboot, could you please post a branch on github and a link on this email
>> thread?
> 
> It took a bit more time than we expected, but here we go [1]:
> 
> this is in form of a pull-request as we would love to hear from the
> 
> community and it is easier to discuss the code (please leave comments there)
> 
> 1. There is code originating from MiniOS and work done by Peng, so we
> 
> would like to ask the respective copyright owners to raise their hands and

Not everyone are closely watching xen-devel. So if you want to find out 
who are the copyright owners, then your best solution is to go through 
the git log and CC the authors.

> 
> let us *fix inappropriate licensing* if any.
> 
> 2. Please note, the series has a HACK to move the RAM base as it is expected by
> 
> our test platform (iMX8), so others will need to remove or modify that.
> 
> 3. There is a limitation already noted by Peng that we do not have serial output
> 
> until MMU is setup, so we have introduced xen_early_printk helper which always
> 
> works, so you can use that for early stage debugging.
> 
> 4. Minimal memory size supported is ~129M because of dtb placement by Xen tools

Hmmm... Why? What's wrong with booting a guest with just 64MB?

> 
> 5. We use -D__XEN__ to access some of the hidden defines we need such as
> 
> GUEST_RAM0_BASE and the friends as there is no other way but manually defining the
> 
> same which is not cute.

I have replied to this in the pull request. But I want to bring the 
conversation here to have a wider discussion.

For a first, __XEN__ should really only be defined by the hypervisor and 
not used by the guest. This is used to gate non-stable ABI (such as the 
tools) and anything behind it hasn't been vetted to work in other build 
configuration that the one used by Xen.

In general, we expect the guest to discover everything for the 
device-tree because the memory layout is not stable (we want to be able 
to reshuffle as we add more features).

I know that EDK2/Tianocore can be built once and work on every Xen 
configuration. It would be ideal that U-boot follow the same. If it is 
really not possible, then we should explore a path that is preventing to 
define __XEN__.

How much does U-boot expect to know about the memory layout? Does it 
require to know all the memory banks? Or would it be sufficient for it 
to know the start address of the first bank and the minimum of RAM?

Cheers,

-- 
Julien Grall

Re: UEFI support in ARM DomUs

[Stefano Stabellini]

On Thu, 18 Jun 2020, Julien Grall wrote:
> On 18/06/2020 06:22, Oleksandr Andrushchenko wrote:
> > 
> > On 6/4/20 6:31 PM, Stefano Stabellini wrote:
> > > On Thu, 4 Jun 2020, Oleksandr Andrushchenko wrote:
> > > > On 6/4/20 4:57 AM, Peng Fan wrote:
> > > > > Grall <julien@xen.org>;
> > > > > > Nataliya Korovkina <malus.brandywine@gmail.com>
> > > > > > Subject: UEFI support in ARM DomUs
> > > > > We have made U-Boot run inside XEN DomU, but just only PV console
> > > > > part,
> > > > > not implement other frontend drivers currently. Would this help for
> > > > > your
> > > > > case if enable EFI in U-Boot?
> > > > Well, we have a working PV block implementation on top of that on iMX8
> > > > 
> > > > platform, mostly ported from mini-os. Currently we are finalizing the
> > > > work
> > > > 
> > > > and cleaning up (it's going to take a week or so hopefully). Then, we
> > > > we'll post
> > > > 
> > > > it on our public github. We are also thinking about upstreaming the
> > > > work, but it may
> > > > 
> > > > take quite some time if the whole idea fits u-boot's view on such an
> > > > extension at all.
> > > Yes please to both of you! :-)
> > > 
> > > In the meantime, while we wait for those changes to go upstream in
> > > uboot, could you please post a branch on github and a link on this email
> > > thread?
> > 
> > It took a bit more time than we expected, but here we go [1]:
> > 
> > this is in form of a pull-request as we would love to hear from the
> > 
> > community and it is easier to discuss the code (please leave comments there)
> > 
> > 1. There is code originating from MiniOS and work done by Peng, so we
> > 
> > would like to ask the respective copyright owners to raise their hands and
> 
> Not everyone are closely watching xen-devel. So if you want to find out who
> are the copyright owners, then your best solution is to go through the git log
> and CC the authors.

That is true. But why do you want to contact them? It doesn't look like
there would be any licensing issues.

 
> > let us *fix inappropriate licensing* if any.
> > 
> > 2. Please note, the series has a HACK to move the RAM base as it is expected
> > by
> > 
> > our test platform (iMX8), so others will need to remove or modify that.
> > 
> > 3. There is a limitation already noted by Peng that we do not have serial
> > output
> > 
> > until MMU is setup, so we have introduced xen_early_printk helper which
> > always
> > 
> > works, so you can use that for early stage debugging.
> > 
> > 4. Minimal memory size supported is ~129M because of dtb placement by Xen
> > tools
> 
> Hmmm... Why? What's wrong with booting a guest with just 64MB?

I agree that it would be nice to support 64MB at least as a minimum. We
are talking about embedded here, every byte counts :-)


> > 
> > 5. We use -D__XEN__ to access some of the hidden defines we need such as
> > 
> > GUEST_RAM0_BASE and the friends as there is no other way but manually
> > defining the
> > 
> > same which is not cute.
> 
> I have replied to this in the pull request. But I want to bring the
> conversation here to have a wider discussion.
> 
> For a first, __XEN__ should really only be defined by the hypervisor and not
> used by the guest. This is used to gate non-stable ABI (such as the tools) and
> anything behind it hasn't been vetted to work in other build configuration
> that the one used by Xen.
> 
> In general, we expect the guest to discover everything for the device-tree
> because the memory layout is not stable (we want to be able to reshuffle as we
> add more features).
> 
> I know that EDK2/Tianocore can be built once and work on every Xen
> configuration. It would be ideal that U-boot follow the same. If it is really
> not possible, then we should explore a path that is preventing to define
> __XEN__.
> 
> How much does U-boot expect to know about the memory layout? Does it require
> to know all the memory banks? Or would it be sufficient for it to know the
> start address of the first bank and the minimum of RAM?

Copy/pasting here from my comment on the pull request plus additional
thoughts.

Uboot is one of those embedded projects that typically assumes that all
the information about the platform is available at *build time*. It is
meant to be built tailored for a specific version of a specific board. A
Uboot binary is not expected to be "portable" across different versions
of the platform or different platforms. In other words, it is not
expected to be portable across Xen versions.

This is a different model meant for different use-cases. I don't think
it is a problem "philosophically" to let Uboot know about Xen details at
build time. Yes, that is not what we did historically but it is very
much in the spirit of Uboot.

But of course the least Uboot depends on these details the better
because it will be more flexible, but it could very well be that we
won't be able to reach the point where the binary works on any version
like we did with Tianocore. The two projects work differently.


That said, I think Julien is right that we need to be careful on how we
expose these information to Uboot, i.e. defining __XEN__ to build Uboot
doesn't seem like a good idea because we enable definitions that were
never meant to be used outside of a Xen build. Also, it wouldn't be easy
to know exactly which definitions are actively used by Uboot and which
ones aren't.

If we are going to make Uboot dependent on version-specific information
of the Xen interface, it would be better to be very clear about which
definitions we are using. So that one day if we need to change them, we
can find them easily.

So I think it would be better to introduce a set of defines with the
minimum amount of information required by Uboot statically. That way,
at least we have a single place where to find all the version-specific
definitions that Uboot is using. We can also manage versioning of the
Xen interface (like we do in QEMU) if we have to.

Re: UEFI support in ARM DomUs

[Julien Grall]

On Thu, 18 Jun 2020 at 23:00, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> On Thu, 18 Jun 2020, Julien Grall wrote:
> > On 18/06/2020 06:22, Oleksandr Andrushchenko wrote:
> > >
> > > On 6/4/20 6:31 PM, Stefano Stabellini wrote:
> > > > On Thu, 4 Jun 2020, Oleksandr Andrushchenko wrote:
> > > > > On 6/4/20 4:57 AM, Peng Fan wrote:
> > > > > > Grall <julien@xen.org>;
> > > > > > > Nataliya Korovkina <malus.brandywine@gmail.com>
> > > > > > > Subject: UEFI support in ARM DomUs
> > > > > > We have made U-Boot run inside XEN DomU, but just only PV console
> > > > > > part,
> > > > > > not implement other frontend drivers currently. Would this help for
> > > > > > your
> > > > > > case if enable EFI in U-Boot?
> > > > > Well, we have a working PV block implementation on top of that on iMX8
> > > > >
> > > > > platform, mostly ported from mini-os. Currently we are finalizing the
> > > > > work
> > > > >
> > > > > and cleaning up (it's going to take a week or so hopefully). Then, we
> > > > > we'll post
> > > > >
> > > > > it on our public github. We are also thinking about upstreaming the
> > > > > work, but it may
> > > > >
> > > > > take quite some time if the whole idea fits u-boot's view on such an
> > > > > extension at all.
> > > > Yes please to both of you! :-)
> > > >
> > > > In the meantime, while we wait for those changes to go upstream in
> > > > uboot, could you please post a branch on github and a link on this email
> > > > thread?
> > >
> > > It took a bit more time than we expected, but here we go [1]:
> > >
> > > this is in form of a pull-request as we would love to hear from the
> > >
> > > community and it is easier to discuss the code (please leave comments there)
> > >
> > > 1. There is code originating from MiniOS and work done by Peng, so we
> > >
> > > would like to ask the respective copyright owners to raise their hands and
> >
> > Not everyone are closely watching xen-devel. So if you want to find out who
> > are the copyright owners, then your best solution is to go through the git log
> > and CC the authors.
>
> That is true. But why do you want to contact them? It doesn't look like
> there would be any licensing issues.

From the sentence, I wasn't entirely sure whether he wanted to contact
the copyright owner for crediting them in the headers.

> > >
> > > 5. We use -D__XEN__ to access some of the hidden defines we need such as
> > >
> > > GUEST_RAM0_BASE and the friends as there is no other way but manually
> > > defining the
> > >
> > > same which is not cute.
> >
> > I have replied to this in the pull request. But I want to bring the
> > conversation here to have a wider discussion.
> >
> > For a first, __XEN__ should really only be defined by the hypervisor and not
> > used by the guest. This is used to gate non-stable ABI (such as the tools) and
> > anything behind it hasn't been vetted to work in other build configuration
> > that the one used by Xen.
> >
> > In general, we expect the guest to discover everything for the device-tree
> > because the memory layout is not stable (we want to be able to reshuffle as we
> > add more features).
> >
> > I know that EDK2/Tianocore can be built once and work on every Xen
> > configuration. It would be ideal that U-boot follow the same. If it is really
> > not possible, then we should explore a path that is preventing to define
> > __XEN__.
> >
> > How much does U-boot expect to know about the memory layout? Does it require
> > to know all the memory banks? Or would it be sufficient for it to know the
> > start address of the first bank and the minimum of RAM?
>
> Copy/pasting here from my comment on the pull request plus additional
> thoughts.
>
> Uboot is one of those embedded projects that typically assumes that all
> the information about the platform is available at *build time*. It is
> meant to be built tailored for a specific version of a specific board. A
> Uboot binary is not expected to be "portable" across different versions
> of the platform or different platforms. In other words, it is not
> expected to be portable across Xen versions.

Can you define "version" here? Do you refer to the difference in terms
of memory?

>
> This is a different model meant for different use-cases. I don't think
> it is a problem "philosophically" to let Uboot know about Xen details at
> build time. Yes, that is not what we did historically but it is very
> much in the spirit of Uboot.

TBH, I don't particularly mind that U-boot is built against a specific
version of Xen. I am more concerned about the long term implication if
we endorse it
and then try to change the memory layout in depth.

>
> But of course the least Uboot depends on these details the better
> because it will be more flexible, but it could very well be that we
> won't be able to reach the point where the binary works on any version
> like we did with Tianocore. The two projects work differently.

Can we have a list of things U-boot require to know at compile time?

In particular, I would like to understand if it would be sufficient to
only be aware of the first bank. If it is, then my preference would be
to standardize that bit of the layout.

> That said, I think Julien is right that we need to be careful on how we
> expose these information to Uboot, i.e. defining __XEN__ to build Uboot
> doesn't seem like a good idea because we enable definitions that were
> never meant to be used outside of a Xen build. Also, it wouldn't be easy
> to know exactly which definitions are actively used by Uboot and which
> ones aren't.
>
> If we are going to make Uboot dependent on version-specific information
> of the Xen interface, it would be better to be very clear about which
> definitions we are using. So that one day if we need to change them, we
> can find them easily.
>
> So I think it would be better to introduce a set of defines with the
> minimum amount of information required by Uboot statically. That way,
> at least we have a single place where to find all the version-specific
> definitions that Uboot is using.

I am not sure what you are suggesting. Can you expand a bit more?

> We can also manage versioning of the
> Xen interface (like we do in QEMU) if we have to.

Can you provide more details about the compatibility? I am quite
interested in the part where you would have to deal with an older QEMU
version built against a new Xen?

Cheers,

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/19/20 1:49 AM, Julien Grall wrote:
> On Thu, 18 Jun 2020 at 23:00, Stefano Stabellini <sstabellini@kernel.org> wrote:
>> On Thu, 18 Jun 2020, Julien Grall wrote:
>>> On 18/06/2020 06:22, Oleksandr Andrushchenko wrote:
>>>> On 6/4/20 6:31 PM, Stefano Stabellini wrote:
>>>>> On Thu, 4 Jun 2020, Oleksandr Andrushchenko wrote:
>>>>>> On 6/4/20 4:57 AM, Peng Fan wrote:
>>>>>>> Grall <julien@xen.org>;
>>>>>>>> Nataliya Korovkina <malus.brandywine@gmail.com>
>>>>>>>> Subject: UEFI support in ARM DomUs
>>>>>>> We have made U-Boot run inside XEN DomU, but just only PV console
>>>>>>> part,
>>>>>>> not implement other frontend drivers currently. Would this help for
>>>>>>> your
>>>>>>> case if enable EFI in U-Boot?
>>>>>> Well, we have a working PV block implementation on top of that on iMX8
>>>>>>
>>>>>> platform, mostly ported from mini-os. Currently we are finalizing the
>>>>>> work
>>>>>>
>>>>>> and cleaning up (it's going to take a week or so hopefully). Then, we
>>>>>> we'll post
>>>>>>
>>>>>> it on our public github. We are also thinking about upstreaming the
>>>>>> work, but it may
>>>>>>
>>>>>> take quite some time if the whole idea fits u-boot's view on such an
>>>>>> extension at all.
>>>>> Yes please to both of you! :-)
>>>>>
>>>>> In the meantime, while we wait for those changes to go upstream in
>>>>> uboot, could you please post a branch on github and a link on this email
>>>>> thread?
>>>> It took a bit more time than we expected, but here we go [1]:
>>>>
>>>> this is in form of a pull-request as we would love to hear from the
>>>>
>>>> community and it is easier to discuss the code (please leave comments there)
>>>>
>>>> 1. There is code originating from MiniOS and work done by Peng, so we
>>>>
>>>> would like to ask the respective copyright owners to raise their hands and
>>> Not everyone are closely watching xen-devel. So if you want to find out who
>>> are the copyright owners, then your best solution is to go through the git log
>>> and CC the authors.
>> That is true. But why do you want to contact them? It doesn't look like
>> there would be any licensing issues.
>  From the sentence, I wasn't entirely sure whether he wanted to contact
> the copyright owner for crediting them in the headers.
>
>>>> 5. We use -D__XEN__ to access some of the hidden defines we need such as
>>>>
>>>> GUEST_RAM0_BASE and the friends as there is no other way but manually
>>>> defining the
>>>>
>>>> same which is not cute.
>>> I have replied to this in the pull request. But I want to bring the
>>> conversation here to have a wider discussion.
>>>
>>> For a first, __XEN__ should really only be defined by the hypervisor and not
>>> used by the guest. This is used to gate non-stable ABI (such as the tools) and
>>> anything behind it hasn't been vetted to work in other build configuration
>>> that the one used by Xen.
>>>
>>> In general, we expect the guest to discover everything for the device-tree
>>> because the memory layout is not stable (we want to be able to reshuffle as we
>>> add more features).
>>>
>>> I know that EDK2/Tianocore can be built once and work on every Xen
>>> configuration. It would be ideal that U-boot follow the same. If it is really
>>> not possible, then we should explore a path that is preventing to define
>>> __XEN__.
>>>
>>> How much does U-boot expect to know about the memory layout? Does it require
>>> to know all the memory banks? Or would it be sufficient for it to know the
>>> start address of the first bank and the minimum of RAM?
>> Copy/pasting here from my comment on the pull request plus additional
>> thoughts.
>>
>> Uboot is one of those embedded projects that typically assumes that all
>> the information about the platform is available at *build time*. It is
>> meant to be built tailored for a specific version of a specific board. A
>> Uboot binary is not expected to be "portable" across different versions
>> of the platform or different platforms. In other words, it is not
>> expected to be portable across Xen versions.
> Can you define "version" here? Do you refer to the difference in terms
> of memory?
>
>> This is a different model meant for different use-cases. I don't think
>> it is a problem "philosophically" to let Uboot know about Xen details at
>> build time. Yes, that is not what we did historically but it is very
>> much in the spirit of Uboot.
> TBH, I don't particularly mind that U-boot is built against a specific
> version of Xen. I am more concerned about the long term implication if
> we endorse it
> and then try to change the memory layout in depth.
>
>> But of course the least Uboot depends on these details the better
>> because it will be more flexible, but it could very well be that we
>> won't be able to reach the point where the binary works on any version
>> like we did with Tianocore. The two projects work differently.
> Can we have a list of things U-boot require to know at compile time?
>
> In particular, I would like to understand if it would be sufficient to
> only be aware of the first bank. If it is, then my preference would be
> to standardize that bit of the layout.

Well, my bad, I was not right about PIE. We are lucky that it is supported

for ARM64, so we can avoid using GUEST_RAM0_BASE.

With respect to the number of banks I see no big issue if we do not use

GUEST_RAM_BANKS, but set it to 1.

The last thing at the moment that I am not sure of is GUEST_MAGIC_{BASE|SIZE}:

those can be retrieved from the device tree and I'll have to check if

fdt is available at the very early boot stage so we can get that when it is needed.

We'll keep you updated on this

>> That said, I think Julien is right that we need to be careful on how we
>> expose these information to Uboot, i.e. defining __XEN__ to build Uboot
>> doesn't seem like a good idea because we enable definitions that were
>> never meant to be used outside of a Xen build. Also, it wouldn't be easy
>> to know exactly which definitions are actively used by Uboot and which
>> ones aren't.
>>
>> If we are going to make Uboot dependent on version-specific information
>> of the Xen interface, it would be better to be very clear about which
>> definitions we are using. So that one day if we need to change them, we
>> can find them easily.
>>
>> So I think it would be better to introduce a set of defines with the
>> minimum amount of information required by Uboot statically. That way,
>> at least we have a single place where to find all the version-specific
>> definitions that Uboot is using.
> I am not sure what you are suggesting. Can you expand a bit more?
>
>> We can also manage versioning of the
>> Xen interface (like we do in QEMU) if we have to.
> Can you provide more details about the compatibility? I am quite
> interested in the part where you would have to deal with an older QEMU
> version built against a new Xen?
>
> Cheers,

Re: UEFI support in ARM DomUs

[Julien Grall]

Hi Oleksandr,

On 19/06/2020 13:32, Oleksandr Andrushchenko wrote:
> 
> On 6/19/20 1:49 AM, Julien Grall wrote:
>> On Thu, 18 Jun 2020 at 23:00, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>> On Thu, 18 Jun 2020, Julien Grall wrote:
>>>> On 18/06/2020 06:22, Oleksandr Andrushchenko wrote:
>>>>> On 6/4/20 6:31 PM, Stefano Stabellini wrote:
>>>>>> On Thu, 4 Jun 2020, Oleksandr Andrushchenko wrote:
>>>>>>> On 6/4/20 4:57 AM, Peng Fan wrote:
>>>>>>>> Grall <julien@xen.org>;
>>>>>>>>> Nataliya Korovkina <malus.brandywine@gmail.com>
>>>>>>>>> Subject: UEFI support in ARM DomUs
>>>>>>>> We have made U-Boot run inside XEN DomU, but just only PV console
>>>>>>>> part,
>>>>>>>> not implement other frontend drivers currently. Would this help for
>>>>>>>> your
>>>>>>>> case if enable EFI in U-Boot?
>>>>>>> Well, we have a working PV block implementation on top of that on iMX8
>>>>>>>
>>>>>>> platform, mostly ported from mini-os. Currently we are finalizing the
>>>>>>> work
>>>>>>>
>>>>>>> and cleaning up (it's going to take a week or so hopefully). Then, we
>>>>>>> we'll post
>>>>>>>
>>>>>>> it on our public github. We are also thinking about upstreaming the
>>>>>>> work, but it may
>>>>>>>
>>>>>>> take quite some time if the whole idea fits u-boot's view on such an
>>>>>>> extension at all.
>>>>>> Yes please to both of you! :-)
>>>>>>
>>>>>> In the meantime, while we wait for those changes to go upstream in
>>>>>> uboot, could you please post a branch on github and a link on this email
>>>>>> thread?
>>>>> It took a bit more time than we expected, but here we go [1]:
>>>>>
>>>>> this is in form of a pull-request as we would love to hear from the
>>>>>
>>>>> community and it is easier to discuss the code (please leave comments there)
>>>>>
>>>>> 1. There is code originating from MiniOS and work done by Peng, so we
>>>>>
>>>>> would like to ask the respective copyright owners to raise their hands and
>>>> Not everyone are closely watching xen-devel. So if you want to find out who
>>>> are the copyright owners, then your best solution is to go through the git log
>>>> and CC the authors.
>>> That is true. But why do you want to contact them? It doesn't look like
>>> there would be any licensing issues.
>>   From the sentence, I wasn't entirely sure whether he wanted to contact
>> the copyright owner for crediting them in the headers.
>>
>>>>> 5. We use -D__XEN__ to access some of the hidden defines we need such as
>>>>>
>>>>> GUEST_RAM0_BASE and the friends as there is no other way but manually
>>>>> defining the
>>>>>
>>>>> same which is not cute.
>>>> I have replied to this in the pull request. But I want to bring the
>>>> conversation here to have a wider discussion.
>>>>
>>>> For a first, __XEN__ should really only be defined by the hypervisor and not
>>>> used by the guest. This is used to gate non-stable ABI (such as the tools) and
>>>> anything behind it hasn't been vetted to work in other build configuration
>>>> that the one used by Xen.
>>>>
>>>> In general, we expect the guest to discover everything for the device-tree
>>>> because the memory layout is not stable (we want to be able to reshuffle as we
>>>> add more features).
>>>>
>>>> I know that EDK2/Tianocore can be built once and work on every Xen
>>>> configuration. It would be ideal that U-boot follow the same. If it is really
>>>> not possible, then we should explore a path that is preventing to define
>>>> __XEN__.
>>>>
>>>> How much does U-boot expect to know about the memory layout? Does it require
>>>> to know all the memory banks? Or would it be sufficient for it to know the
>>>> start address of the first bank and the minimum of RAM?
>>> Copy/pasting here from my comment on the pull request plus additional
>>> thoughts.
>>>
>>> Uboot is one of those embedded projects that typically assumes that all
>>> the information about the platform is available at *build time*. It is
>>> meant to be built tailored for a specific version of a specific board. A
>>> Uboot binary is not expected to be "portable" across different versions
>>> of the platform or different platforms. In other words, it is not
>>> expected to be portable across Xen versions.
>> Can you define "version" here? Do you refer to the difference in terms
>> of memory?
>>
>>> This is a different model meant for different use-cases. I don't think
>>> it is a problem "philosophically" to let Uboot know about Xen details at
>>> build time. Yes, that is not what we did historically but it is very
>>> much in the spirit of Uboot.
>> TBH, I don't particularly mind that U-boot is built against a specific
>> version of Xen. I am more concerned about the long term implication if
>> we endorse it
>> and then try to change the memory layout in depth.
>>
>>> But of course the least Uboot depends on these details the better
>>> because it will be more flexible, but it could very well be that we
>>> won't be able to reach the point where the binary works on any version
>>> like we did with Tianocore. The two projects work differently.
>> Can we have a list of things U-boot require to know at compile time?
>>
>> In particular, I would like to understand if it would be sufficient to
>> only be aware of the first bank. If it is, then my preference would be
>> to standardize that bit of the layout.
> 
> Well, my bad, I was not right about PIE. We are lucky that it is supported
> 
> for ARM64, so we can avoid using GUEST_RAM0_BASE.

Cool!

> 
> With respect to the number of banks I see no big issue if we do not use
> 
> GUEST_RAM_BANKS, but set it to 1.

I am guessing U-boot wouldn't be able to load modules into the second 
bank. Am I corre
> The last thing at the moment that I am not sure of is GUEST_MAGIC_{BASE|SIZE}:
> 
> those can be retrieved from the device tree and I'll have to check if
> 
> fdt is available at the very early boot stage so we can get that when it is needed.

They will not be available from the fdt, but you can retrieve them with 
an hypervisor call (see HVM_PARAM_STORE_PFN, HVM_PARAM_CONSOLE_PFN).
One question though, why do you need to map them in advance? Couldn't 
you map them on demand?
Cheers,

-- 
Julien Grall

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/19/20 3:47 PM, Julien Grall wrote:
> Hi Oleksandr,
>
> On 19/06/2020 13:32, Oleksandr Andrushchenko wrote:
>>
>> On 6/19/20 1:49 AM, Julien Grall wrote:
>>> On Thu, 18 Jun 2020 at 23:00, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>>> On Thu, 18 Jun 2020, Julien Grall wrote:
>>>>> On 18/06/2020 06:22, Oleksandr Andrushchenko wrote:
>>>>>> On 6/4/20 6:31 PM, Stefano Stabellini wrote:
>>>>>>> On Thu, 4 Jun 2020, Oleksandr Andrushchenko wrote:
>>>>>>>> On 6/4/20 4:57 AM, Peng Fan wrote:
>>>>>>>>> Grall <julien@xen.org>;
>>>>>>>>>> Nataliya Korovkina <malus.brandywine@gmail.com>
>>>>>>>>>> Subject: UEFI support in ARM DomUs
>>>>>>>>> We have made U-Boot run inside XEN DomU, but just only PV console
>>>>>>>>> part,
>>>>>>>>> not implement other frontend drivers currently. Would this help for
>>>>>>>>> your
>>>>>>>>> case if enable EFI in U-Boot?
>>>>>>>> Well, we have a working PV block implementation on top of that on iMX8
>>>>>>>>
>>>>>>>> platform, mostly ported from mini-os. Currently we are finalizing the
>>>>>>>> work
>>>>>>>>
>>>>>>>> and cleaning up (it's going to take a week or so hopefully). Then, we
>>>>>>>> we'll post
>>>>>>>>
>>>>>>>> it on our public github. We are also thinking about upstreaming the
>>>>>>>> work, but it may
>>>>>>>>
>>>>>>>> take quite some time if the whole idea fits u-boot's view on such an
>>>>>>>> extension at all.
>>>>>>> Yes please to both of you! :-)
>>>>>>>
>>>>>>> In the meantime, while we wait for those changes to go upstream in
>>>>>>> uboot, could you please post a branch on github and a link on this email
>>>>>>> thread?
>>>>>> It took a bit more time than we expected, but here we go [1]:
>>>>>>
>>>>>> this is in form of a pull-request as we would love to hear from the
>>>>>>
>>>>>> community and it is easier to discuss the code (please leave comments there)
>>>>>>
>>>>>> 1. There is code originating from MiniOS and work done by Peng, so we
>>>>>>
>>>>>> would like to ask the respective copyright owners to raise their hands and
>>>>> Not everyone are closely watching xen-devel. So if you want to find out who
>>>>> are the copyright owners, then your best solution is to go through the git log
>>>>> and CC the authors.
>>>> That is true. But why do you want to contact them? It doesn't look like
>>>> there would be any licensing issues.
>>>   From the sentence, I wasn't entirely sure whether he wanted to contact
>>> the copyright owner for crediting them in the headers.
>>>
>>>>>> 5. We use -D__XEN__ to access some of the hidden defines we need such as
>>>>>>
>>>>>> GUEST_RAM0_BASE and the friends as there is no other way but manually
>>>>>> defining the
>>>>>>
>>>>>> same which is not cute.
>>>>> I have replied to this in the pull request. But I want to bring the
>>>>> conversation here to have a wider discussion.
>>>>>
>>>>> For a first, __XEN__ should really only be defined by the hypervisor and not
>>>>> used by the guest. This is used to gate non-stable ABI (such as the tools) and
>>>>> anything behind it hasn't been vetted to work in other build configuration
>>>>> that the one used by Xen.
>>>>>
>>>>> In general, we expect the guest to discover everything for the device-tree
>>>>> because the memory layout is not stable (we want to be able to reshuffle as we
>>>>> add more features).
>>>>>
>>>>> I know that EDK2/Tianocore can be built once and work on every Xen
>>>>> configuration. It would be ideal that U-boot follow the same. If it is really
>>>>> not possible, then we should explore a path that is preventing to define
>>>>> __XEN__.
>>>>>
>>>>> How much does U-boot expect to know about the memory layout? Does it require
>>>>> to know all the memory banks? Or would it be sufficient for it to know the
>>>>> start address of the first bank and the minimum of RAM?
>>>> Copy/pasting here from my comment on the pull request plus additional
>>>> thoughts.
>>>>
>>>> Uboot is one of those embedded projects that typically assumes that all
>>>> the information about the platform is available at *build time*. It is
>>>> meant to be built tailored for a specific version of a specific board. A
>>>> Uboot binary is not expected to be "portable" across different versions
>>>> of the platform or different platforms. In other words, it is not
>>>> expected to be portable across Xen versions.
>>> Can you define "version" here? Do you refer to the difference in terms
>>> of memory?
>>>
>>>> This is a different model meant for different use-cases. I don't think
>>>> it is a problem "philosophically" to let Uboot know about Xen details at
>>>> build time. Yes, that is not what we did historically but it is very
>>>> much in the spirit of Uboot.
>>> TBH, I don't particularly mind that U-boot is built against a specific
>>> version of Xen. I am more concerned about the long term implication if
>>> we endorse it
>>> and then try to change the memory layout in depth.
>>>
>>>> But of course the least Uboot depends on these details the better
>>>> because it will be more flexible, but it could very well be that we
>>>> won't be able to reach the point where the binary works on any version
>>>> like we did with Tianocore. The two projects work differently.
>>> Can we have a list of things U-boot require to know at compile time?
>>>
>>> In particular, I would like to understand if it would be sufficient to
>>> only be aware of the first bank. If it is, then my preference would be
>>> to standardize that bit of the layout.
>>
>> Well, my bad, I was not right about PIE. We are lucky that it is supported
>>
>> for ARM64, so we can avoid using GUEST_RAM0_BASE.
>
> Cool!
>
>>
>> With respect to the number of banks I see no big issue if we do not use
>>
>> GUEST_RAM_BANKS, but set it to 1.
>
> I am guessing U-boot wouldn't be able to load modules into the second bank. Am I corre
Not sure, but this can be the case
>> The last thing at the moment that I am not sure of is GUEST_MAGIC_{BASE|SIZE}:
>>
>> those can be retrieved from the device tree and I'll have to check if
>>
>> fdt is available at the very early boot stage so we can get that when it is needed.
>
> They will not be available from the fdt, but you can retrieve them with an hypervisor call (see HVM_PARAM_STORE_PFN, HVM_PARAM_CONSOLE_PFN).
Yes, and it used in the relevant pieces of code (hyp calls)
> One question though, why do you need to map them in advance? Couldn't you map them on demand?

Well, we need to at least estimate the pg_table size so we can reserve and allocate memory later,

so I have to provide memory range from either by coding a constant or looking into the devtree at

hypervisor { reg = <>; }. It is a bit tricky though

> Cheers,
>

Re: UEFI support in ARM DomUs

[Julien Grall]

Hi,

On 19/06/2020 13:51, Oleksandr Andrushchenko wrote:
> On 6/19/20 3:47 PM, Julien Grall wrote:
>> They will not be available from the fdt, but you can retrieve them with an hypervisor call (see HVM_PARAM_STORE_PFN, HVM_PARAM_CONSOLE_PFN).
> Yes, and it used in the relevant pieces of code (hyp calls)
>> One question though, why do you need to map them in advance? Couldn't you map them on demand?
> 
> Well, we need to at least estimate the pg_table size so we can reserve and allocate memory later,

Oh, so U-boot doesn't support runtime page-table table allocation. Is 
that right?

> 
> so I have to provide memory range from either by coding a constant or looking into the devtree at
> 
> hypervisor { reg = <>; }. It is a bit tricky though

Looking for a node in the device-tree shouldn't be too difficult given 
that you have fdt_* available.

However, please not that <reg> doesn't refer to the guest magic pages. 
Instead, it provides a region you can use for mapping the grant-table frames

Cheers,

-- 
Julien Grall

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/19/20 3:59 PM, Julien Grall wrote:
> Hi,
>
> On 19/06/2020 13:51, Oleksandr Andrushchenko wrote:
>> On 6/19/20 3:47 PM, Julien Grall wrote:
>>> They will not be available from the fdt, but you can retrieve them with an hypervisor call (see HVM_PARAM_STORE_PFN, HVM_PARAM_CONSOLE_PFN).
>> Yes, and it used in the relevant pieces of code (hyp calls)
>>> One question though, why do you need to map them in advance? Couldn't you map them on demand?
>>
>> Well, we need to at least estimate the pg_table size so we can reserve and allocate memory later,
>
> Oh, so U-boot doesn't support runtime page-table table allocation. Is that right?
As per my understanding no, we provide a memory map and the tables are allocated beforehand
>
>>
>> so I have to provide memory range from either by coding a constant or looking into the devtree at
>>
>> hypervisor { reg = <>; }. It is a bit tricky though
>
> Looking for a node in the device-tree shouldn't be too difficult given that you have fdt_* available.
>
> However, please not that <reg> doesn't refer to the guest magic pages. Instead, it provides a region you can use for mapping the grant-table frames

Indeed, this is in my case 0x38000000, but the magic is at 0x39000000

So, I need the memory range set up beforehand, but I can't as there is no cute way to get that.

Of course, I can issue a hyp call to get HVM_PARAM_CONSOLE_PFN and use it as the base address,

but this smells like a hack. I can call other HVM_PARAM_ to get their pfns and set up the memory regions,

but this looks a bit weird. I need that constant badly ;)

>
> Cheers,
>

Re: UEFI support in ARM DomUs

[Julien Grall]

On 19/06/2020 14:06, Oleksandr Andrushchenko wrote:
> 
> On 6/19/20 3:59 PM, Julien Grall wrote:
>> Hi,
>>
>> On 19/06/2020 13:51, Oleksandr Andrushchenko wrote:
>>> On 6/19/20 3:47 PM, Julien Grall wrote:
>>>> They will not be available from the fdt, but you can retrieve them with an hypervisor call (see HVM_PARAM_STORE_PFN, HVM_PARAM_CONSOLE_PFN).
>>> Yes, and it used in the relevant pieces of code (hyp calls)
>>>> One question though, why do you need to map them in advance? Couldn't you map them on demand?
>>>
>>> Well, we need to at least estimate the pg_table size so we can reserve and allocate memory later,
>>
>> Oh, so U-boot doesn't support runtime page-table table allocation. Is that right?
> As per my understanding no, we provide a memory map and the tables are allocated beforehand

Ok :(.

>>
>>>
>>> so I have to provide memory range from either by coding a constant or looking into the devtree at
>>>
>>> hypervisor { reg = <>; }. It is a bit tricky though
>>
>> Looking for a node in the device-tree shouldn't be too difficult given that you have fdt_* available.
>>
>> However, please not that <reg> doesn't refer to the guest magic pages. Instead, it provides a region you can use for mapping the grant-table frames
> 
> Indeed, this is in my case 0x38000000, but the magic is at 0x39000000
> 
> So, I need the memory range set up beforehand, but I can't as there is no cute way to get that.
> 
> Of course, I can issue a hyp call to get HVM_PARAM_CONSOLE_PFN and use it as the base address,
> 
> but this smells like a hack. I can call other HVM_PARAM_ to get their pfns and set up the memory regions,
> 
> but this looks a bit weird.

Why is it weird? In general, you only want to map exactly what you need. 
Not less, not more.

In your situation, you only care about two pages, the console page and 
the xenstore page. The rest shouldn't be mapped.

> I need that constant badly ;)

No you don't ;).

Cheers,

-- 
Julien Grall

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/19/20 4:15 PM, Julien Grall wrote:
>
>
> On 19/06/2020 14:06, Oleksandr Andrushchenko wrote:
>>
>> On 6/19/20 3:59 PM, Julien Grall wrote:
>>> Hi,
>>>
>>> On 19/06/2020 13:51, Oleksandr Andrushchenko wrote:
>>>> On 6/19/20 3:47 PM, Julien Grall wrote:
>>>>> They will not be available from the fdt, but you can retrieve them with an hypervisor call (see HVM_PARAM_STORE_PFN, HVM_PARAM_CONSOLE_PFN).
>>>> Yes, and it used in the relevant pieces of code (hyp calls)
>>>>> One question though, why do you need to map them in advance? Couldn't you map them on demand?
>>>>
>>>> Well, we need to at least estimate the pg_table size so we can reserve and allocate memory later,
>>>
>>> Oh, so U-boot doesn't support runtime page-table table allocation. Is that right?
>> As per my understanding no, we provide a memory map and the tables are allocated beforehand
>
> Ok :(.
>
>>>
>>>>
>>>> so I have to provide memory range from either by coding a constant or looking into the devtree at
>>>>
>>>> hypervisor { reg = <>; }. It is a bit tricky though
>>>
>>> Looking for a node in the device-tree shouldn't be too difficult given that you have fdt_* available.
>>>
>>> However, please not that <reg> doesn't refer to the guest magic pages. Instead, it provides a region you can use for mapping the grant-table frames
>>
>> Indeed, this is in my case 0x38000000, but the magic is at 0x39000000
>>
>> So, I need the memory range set up beforehand, but I can't as there is no cute way to get that.
>>
>> Of course, I can issue a hyp call to get HVM_PARAM_CONSOLE_PFN and use it as the base address,
>>
>> but this smells like a hack. I can call other HVM_PARAM_ to get their pfns and set up the memory regions,
>>
>> but this looks a bit weird.
>
> Why is it weird? In general, you only want to map exactly what you need. Not less, not more.
>
> In your situation, you only care about two pages, the console page and the xenstore page. The rest shouldn't be mapped.
Ok, so I'll try get pfns for HVM_PARAM_CONSOLE_PFN + XENSTORE_PFN_OFFSET via hyp call and map those
>
>> I need that constant badly ;)
>
> No you don't ;).
>
> Cheers,
>
Thanks for helping with this

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/19/20 4:29 PM, Oleksandr Andrushchenko wrote:
> On 6/19/20 4:15 PM, Julien Grall wrote:
>>
>>
>> On 19/06/2020 14:06, Oleksandr Andrushchenko wrote:
>>>
>>> On 6/19/20 3:59 PM, Julien Grall wrote:
>>>> Hi,
>>>>
>>>> On 19/06/2020 13:51, Oleksandr Andrushchenko wrote:
>>>>> On 6/19/20 3:47 PM, Julien Grall wrote:
>>>>>> They will not be available from the fdt, but you can retrieve them with an hypervisor call (see HVM_PARAM_STORE_PFN, HVM_PARAM_CONSOLE_PFN).
>>>>> Yes, and it used in the relevant pieces of code (hyp calls)
>>>>>> One question though, why do you need to map them in advance? Couldn't you map them on demand?
>>>>>
>>>>> Well, we need to at least estimate the pg_table size so we can reserve and allocate memory later,
>>>>
>>>> Oh, so U-boot doesn't support runtime page-table table allocation. Is that right?
>>> As per my understanding no, we provide a memory map and the tables are allocated beforehand
>>
>> Ok :(.
>>
>>>>
>>>>>
>>>>> so I have to provide memory range from either by coding a constant or looking into the devtree at
>>>>>
>>>>> hypervisor { reg = <>; }. It is a bit tricky though
>>>>
>>>> Looking for a node in the device-tree shouldn't be too difficult given that you have fdt_* available.
>>>>
>>>> However, please not that <reg> doesn't refer to the guest magic pages. Instead, it provides a region you can use for mapping the grant-table frames
>>>
>>> Indeed, this is in my case 0x38000000, but the magic is at 0x39000000
>>>
>>> So, I need the memory range set up beforehand, but I can't as there is no cute way to get that.
>>>
>>> Of course, I can issue a hyp call to get HVM_PARAM_CONSOLE_PFN and use it as the base address,
>>>
>>> but this smells like a hack. I can call other HVM_PARAM_ to get their pfns and set up the memory regions,
>>>
>>> but this looks a bit weird.
>>
>> Why is it weird? In general, you only want to map exactly what you need. Not less, not more.
>>
>> In your situation, you only care about two pages, the console page and the xenstore page. The rest shouldn't be mapped.
> Ok, so I'll try get pfns for HVM_PARAM_CONSOLE_PFN + XENSTORE_PFN_OFFSET via hyp call and map those
>>
>>> I need that constant badly ;)
>>
>> No you don't ;).

We have managed to make use of the relevant hypercalls to get the PFNs, but for that

we need to maintain the caches as this happens (the calls) when MMU is not yet

setup and is in the process of.

>>
>> Cheers,
>>
> Thanks for helping with this

Re: UEFI support in ARM DomUs

[Julien Grall]

On 22/06/2020 14:56, Oleksandr Andrushchenko wrote:
> 
> On 6/19/20 4:29 PM, Oleksandr Andrushchenko wrote:
>> On 6/19/20 4:15 PM, Julien Grall wrote:
>>>
>>>
>>> On 19/06/2020 14:06, Oleksandr Andrushchenko wrote:
>>>>
>>>> On 6/19/20 3:59 PM, Julien Grall wrote:
>>>>> Hi,
>>>>>
>>>>> On 19/06/2020 13:51, Oleksandr Andrushchenko wrote:
>>>>>> On 6/19/20 3:47 PM, Julien Grall wrote:
>>>>>>> They will not be available from the fdt, but you can retrieve them with an hypervisor call (see HVM_PARAM_STORE_PFN, HVM_PARAM_CONSOLE_PFN).
>>>>>> Yes, and it used in the relevant pieces of code (hyp calls)
>>>>>>> One question though, why do you need to map them in advance? Couldn't you map them on demand?
>>>>>>
>>>>>> Well, we need to at least estimate the pg_table size so we can reserve and allocate memory later,
>>>>>
>>>>> Oh, so U-boot doesn't support runtime page-table table allocation. Is that right?
>>>> As per my understanding no, we provide a memory map and the tables are allocated beforehand
>>>
>>> Ok :(.
>>>
>>>>>
>>>>>>
>>>>>> so I have to provide memory range from either by coding a constant or looking into the devtree at
>>>>>>
>>>>>> hypervisor { reg = <>; }. It is a bit tricky though
>>>>>
>>>>> Looking for a node in the device-tree shouldn't be too difficult given that you have fdt_* available.
>>>>>
>>>>> However, please not that <reg> doesn't refer to the guest magic pages. Instead, it provides a region you can use for mapping the grant-table frames
>>>>
>>>> Indeed, this is in my case 0x38000000, but the magic is at 0x39000000
>>>>
>>>> So, I need the memory range set up beforehand, but I can't as there is no cute way to get that.
>>>>
>>>> Of course, I can issue a hyp call to get HVM_PARAM_CONSOLE_PFN and use it as the base address,
>>>>
>>>> but this smells like a hack. I can call other HVM_PARAM_ to get their pfns and set up the memory regions,
>>>>
>>>> but this looks a bit weird.
>>>
>>> Why is it weird? In general, you only want to map exactly what you need. Not less, not more.
>>>
>>> In your situation, you only care about two pages, the console page and the xenstore page. The rest shouldn't be mapped.
>> Ok, so I'll try get pfns for HVM_PARAM_CONSOLE_PFN + XENSTORE_PFN_OFFSET via hyp call and map those
>>>
>>>> I need that constant badly ;)
>>>
>>> No you don't ;).
> 
> We have managed to make use of the relevant hypercalls to get the PFNs, but for that
> 
> we need to maintain the caches as this happens (the calls) when MMU is not yet
> 
> setup and is in the process of.

Glad to hear it works. Yes, that's unfortunately the drawback of using 
hypercalls with MMU off. But at least you make U-boot more generic :).

Cheers,

-- 
Julien Grall

RE: UEFI support in ARM DomUs

[Peng Fan]

> Subject: Re: UEFI support in ARM DomUs
> 
> 
> On 6/19/20 3:59 PM, Julien Grall wrote:
> > Hi,
> >
> > On 19/06/2020 13:51, Oleksandr Andrushchenko wrote:
> >> On 6/19/20 3:47 PM, Julien Grall wrote:
> >>> They will not be available from the fdt, but you can retrieve them with an
> hypervisor call (see HVM_PARAM_STORE_PFN,
> HVM_PARAM_CONSOLE_PFN).
> >> Yes, and it used in the relevant pieces of code (hyp calls)
> >>> One question though, why do you need to map them in advance?
> Couldn't you map them on demand?
> >>
> >> Well, we need to at least estimate the pg_table size so we can reserve and
> allocate memory later,
> >
> > Oh, so U-boot doesn't support runtime page-table table allocation. Is that
> right?
> As per my understanding no, we provide a memory map and the tables are
> allocated beforehand

You are right.

Regards,
Peng.
> >
> >>
> >> so I have to provide memory range from either by coding a constant or
> looking into the devtree at
> >>
> >> hypervisor { reg = <>; }. It is a bit tricky though
> >
> > Looking for a node in the device-tree shouldn't be too difficult given that you
> have fdt_* available.
> >
> > However, please not that <reg> doesn't refer to the guest magic pages.
> Instead, it provides a region you can use for mapping the grant-table frames
> 
> Indeed, this is in my case 0x38000000, but the magic is at 0x39000000
> 
> So, I need the memory range set up beforehand, but I can't as there is no cute
> way to get that.
> 
> Of course, I can issue a hyp call to get HVM_PARAM_CONSOLE_PFN and use it
> as the base address,
> 
> but this smells like a hack. I can call other HVM_PARAM_ to get their pfns and
> set up the memory regions,
> 
> but this looks a bit weird. I need that constant badly ;)
> 
> >
> > Cheers,
> >

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/19/20 4:16 PM, Peng Fan wrote:
>> Subject: Re: UEFI support in ARM DomUs
>>
>>
>> On 6/19/20 3:59 PM, Julien Grall wrote:
>>> Hi,
>>>
>>> On 19/06/2020 13:51, Oleksandr Andrushchenko wrote:
>>>> On 6/19/20 3:47 PM, Julien Grall wrote:
>>>>> They will not be available from the fdt, but you can retrieve them with an
>> hypervisor call (see HVM_PARAM_STORE_PFN,
>> HVM_PARAM_CONSOLE_PFN).
>>>> Yes, and it used in the relevant pieces of code (hyp calls)
>>>>> One question though, why do you need to map them in advance?
>> Couldn't you map them on demand?
>>>> Well, we need to at least estimate the pg_table size so we can reserve and
>> allocate memory later,
>>> Oh, so U-boot doesn't support runtime page-table table allocation. Is that
>> right?
>> As per my understanding no, we provide a memory map and the tables are
>> allocated beforehand
> You are right.
ok, so then we only have a choice of probing the range via hyp calls
> Regards,
> Peng.
>>>> so I have to provide memory range from either by coding a constant or
>> looking into the devtree at
>>>> hypervisor { reg = <>; }. It is a bit tricky though
>>> Looking for a node in the device-tree shouldn't be too difficult given that you
>> have fdt_* available.
>>> However, please not that <reg> doesn't refer to the guest magic pages.
>> Instead, it provides a region you can use for mapping the grant-table frames
>>
>> Indeed, this is in my case 0x38000000, but the magic is at 0x39000000
>>
>> So, I need the memory range set up beforehand, but I can't as there is no cute
>> way to get that.
>>
>> Of course, I can issue a hyp call to get HVM_PARAM_CONSOLE_PFN and use it
>> as the base address,
>>
>> but this smells like a hack. I can call other HVM_PARAM_ to get their pfns and
>> set up the memory regions,
>>
>> but this looks a bit weird. I need that constant badly ;)
>>
>>> Cheers,
>>>

Re: UEFI support in ARM DomUs

[Stefano Stabellini]

On Thu, 18 Jun 2020, Julien Grall wrote:
> > Copy/pasting here from my comment on the pull request plus additional
> > thoughts.
> >
> > Uboot is one of those embedded projects that typically assumes that all
> > the information about the platform is available at *build time*. It is
> > meant to be built tailored for a specific version of a specific board. A
> > Uboot binary is not expected to be "portable" across different versions
> > of the platform or different platforms. In other words, it is not
> > expected to be portable across Xen versions.
> 
> Can you define "version" here? Do you refer to the difference in terms
> of memory?
 
Yes, I meant any meaningful differences in any of the external
interfaces that end up impacting the UBoot implementation. A primary
example would be the memory addresses for instance.


> > This is a different model meant for different use-cases. I don't think
> > it is a problem "philosophically" to let Uboot know about Xen details at
> > build time. Yes, that is not what we did historically but it is very
> > much in the spirit of Uboot.
> 
> TBH, I don't particularly mind that U-boot is built against a specific
> version of Xen. I am more concerned about the long term implication if
> we endorse it
> and then try to change the memory layout in depth.

I'll provide more information below.


> > But of course the least Uboot depends on these details the better
> > because it will be more flexible, but it could very well be that we
> > won't be able to reach the point where the binary works on any version
> > like we did with Tianocore. The two projects work differently.
> 
> Can we have a list of things U-boot require to know at compile time?
> 
> In particular, I would like to understand if it would be sufficient to
> only be aware of the first bank. If it is, then my preference would be
> to standardize that bit of the layout.

That would be my preference too, and it was great to read that it looks
like it can be done. Yay!


> > That said, I think Julien is right that we need to be careful on how we
> > expose these information to Uboot, i.e. defining __XEN__ to build Uboot
> > doesn't seem like a good idea because we enable definitions that were
> > never meant to be used outside of a Xen build. Also, it wouldn't be easy
> > to know exactly which definitions are actively used by Uboot and which
> > ones aren't.
> >
> > If we are going to make Uboot dependent on version-specific information
> > of the Xen interface, it would be better to be very clear about which
> > definitions we are using. So that one day if we need to change them, we
> > can find them easily.
> >
> > So I think it would be better to introduce a set of defines with the
> > minimum amount of information required by Uboot statically. That way,
> > at least we have a single place where to find all the version-specific
> > definitions that Uboot is using.
> 
> I am not sure what you are suggesting. Can you expand a bit more?
> 
> > We can also manage versioning of the
> > Xen interface (like we do in QEMU) if we have to.
> 
> Can you provide more details about the compatibility? I am quite
> interested in the part where you would have to deal with an older QEMU
> version built against a new Xen?

Sure let me expand a bit more. I'll switch "hat" to "QEMU (or UBoot)
contributor" for the sake of this discussion.

Xen Project offers certain stability guarantees on some interfaces and
not others. Let's say that for any reason you have to or want to use one
of the unstable interfaces in QEMU/UBoot/Whatever. Then it becomes your
responsibility as QEMU developer to maintain compatibility in QEMU
itself.

First I'd add code to detect the version of the interface. The detection
is based on the Xen headers/libraries provided. In QEMU it is done in
the "configure" script. It sets a preprocessor #define to the version
of the interface (e.g. CONFIG_XEN_CTRL_INTERFACE_VERSION == 41100).

Then you can have preprocessors checks in one of the headers such as:

#if CONFIG_XEN_CTRL_INTERFACE_VERSION < 40701
    #define xenevtchn_open(l, f) xc_evtchn_open(l, f);
#else
    XXX
#endif


And also like:

#if CONFIG_XEN_CTRL_INTERFACE_VERSION < 40600

#ifndef HVM_IOREQSRV_BUFIOREQ_ATOMIC
#define HVM_IOREQSRV_BUFIOREQ_ATOMIC 2
#endif

#endif


This works OK to handle differences in the interface between past
versions of Xen. What about building an older QEMU against a new version
of Xen? That is not going to work if something changes again on the Xen
side. However, it becomes much easier to add support for the new Xen
interface in QEMU, because you can go and look at that compatibility
header and just add the right #else XXX. It also makes it clear what
interfaces and definitions have been used that are unstable.

Of course it is better to use the stable interfaces when possible :-)

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/19/20 11:02 PM, Stefano Stabellini wrote:
> On Thu, 18 Jun 2020, Julien Grall wrote:
>>> Copy/pasting here from my comment on the pull request plus additional
>>> thoughts.
>>>
>>> Uboot is one of those embedded projects that typically assumes that all
>>> the information about the platform is available at *build time*. It is
>>> meant to be built tailored for a specific version of a specific board. A
>>> Uboot binary is not expected to be "portable" across different versions
>>> of the platform or different platforms. In other words, it is not
>>> expected to be portable across Xen versions.
>> Can you define "version" here? Do you refer to the difference in terms
>> of memory?
>   
> Yes, I meant any meaningful differences in any of the external
> interfaces that end up impacting the UBoot implementation. A primary
> example would be the memory addresses for instance.
>
>
>>> This is a different model meant for different use-cases. I don't think
>>> it is a problem "philosophically" to let Uboot know about Xen details at
>>> build time. Yes, that is not what we did historically but it is very
>>> much in the spirit of Uboot.
>> TBH, I don't particularly mind that U-boot is built against a specific
>> version of Xen. I am more concerned about the long term implication if
>> we endorse it
>> and then try to change the memory layout in depth.
> I'll provide more information below.
>
>
>>> But of course the least Uboot depends on these details the better
>>> because it will be more flexible, but it could very well be that we
>>> won't be able to reach the point where the binary works on any version
>>> like we did with Tianocore. The two projects work differently.
>> Can we have a list of things U-boot require to know at compile time?
>>
>> In particular, I would like to understand if it would be sufficient to
>> only be aware of the first bank. If it is, then my preference would be
>> to standardize that bit of the layout.
> That would be my preference too, and it was great to read that it looks
> like it can be done. Yay!
>
>
>>> That said, I think Julien is right that we need to be careful on how we
>>> expose these information to Uboot, i.e. defining __XEN__ to build Uboot
>>> doesn't seem like a good idea because we enable definitions that were
>>> never meant to be used outside of a Xen build. Also, it wouldn't be easy
>>> to know exactly which definitions are actively used by Uboot and which
>>> ones aren't.
>>>
>>> If we are going to make Uboot dependent on version-specific information
>>> of the Xen interface, it would be better to be very clear about which
>>> definitions we are using. So that one day if we need to change them, we
>>> can find them easily.
>>>
>>> So I think it would be better to introduce a set of defines with the
>>> minimum amount of information required by Uboot statically. That way,
>>> at least we have a single place where to find all the version-specific
>>> definitions that Uboot is using.
>> I am not sure what you are suggesting. Can you expand a bit more?
>>
>>> We can also manage versioning of the
>>> Xen interface (like we do in QEMU) if we have to.
>> Can you provide more details about the compatibility? I am quite
>> interested in the part where you would have to deal with an older QEMU
>> version built against a new Xen?
> Sure let me expand a bit more. I'll switch "hat" to "QEMU (or UBoot)
> contributor" for the sake of this discussion.
>
> Xen Project offers certain stability guarantees on some interfaces and
> not others. Let's say that for any reason you have to or want to use one
> of the unstable interfaces in QEMU/UBoot/Whatever. Then it becomes your
> responsibility as QEMU developer to maintain compatibility in QEMU
> itself.
>
> First I'd add code to detect the version of the interface. The detection
> is based on the Xen headers/libraries provided. In QEMU it is done in
> the "configure" script. It sets a preprocessor #define to the version
> of the interface (e.g. CONFIG_XEN_CTRL_INTERFACE_VERSION == 41100).

I looked at QEMU's configure script and I'm afraid we can't have the

same flexibility in U-boot: we don't have configure script, pkg-config

is not widely used etc. So, for now we have hardcoded:

ifeq ($(CONFIG_XEN),y)
arch-y += -D__XEN_INTERFACE_VERSION__=0x00040d00
endif

and we also have Xen 4.13 headers in the U-boot tree.

For the first part (__XEN_INTERFACE_VERSION__) I think we can provide it via

CFLAGS or something. This can also be done for the location of Xen headers.

We don't have strong opinion here, so would love to hear from Xen community on this

Current WIP with the fixes requested by Julien are at [1]: can be and will be force pushed

as we are still working on it, but can give you an impression of what we have as of now

>
> Then you can have preprocessors checks in one of the headers such as:
>
> #if CONFIG_XEN_CTRL_INTERFACE_VERSION < 40701
>      #define xenevtchn_open(l, f) xc_evtchn_open(l, f);
> #else
>      XXX
> #endif
>
>
> And also like:
>
> #if CONFIG_XEN_CTRL_INTERFACE_VERSION < 40600
>
> #ifndef HVM_IOREQSRV_BUFIOREQ_ATOMIC
> #define HVM_IOREQSRV_BUFIOREQ_ATOMIC 2
> #endif
>
> #endif
>
>
> This works OK to handle differences in the interface between past
> versions of Xen. What about building an older QEMU against a new version
> of Xen? That is not going to work if something changes again on the Xen
> side. However, it becomes much easier to add support for the new Xen
> interface in QEMU, because you can go and look at that compatibility
> header and just add the right #else XXX. It also makes it clear what
> interfaces and definitions have been used that are unstable.
>
> Of course it is better to use the stable interfaces when possible :-)
[1] https://github.com/andr2000/u-boot/tree/pvblock_upstream_v1

Re: UEFI support in ARM DomUs

[Julien Grall]

Hi Oleksandr,

On 22/06/2020 15:04, Oleksandr Andrushchenko wrote:
> On 6/19/20 11:02 PM, Stefano Stabellini wrote:
>> On Thu, 18 Jun 2020, Julien Grall wrote:
> ifeq ($(CONFIG_XEN),y)
> arch-y += -D__XEN_INTERFACE_VERSION__=0x00040d00
> endif
> 
> and we also have Xen 4.13 headers in the U-boot tree.

Sorry if this was already asked before. Why do you need to specify 
__XEN_INTERFACE_VERSION__?

> 
> For the first part (__XEN_INTERFACE_VERSION__) I think we can provide it via
> 
> CFLAGS or something. This can also be done for the location of Xen headers.

__XEN_INTERFACE_VERSION__ should work through the CFLAGS. An alternative 
would be to allow the user to specify through the Kconfig.
Cheers,

-- 
Julien Grall

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/22/20 5:27 PM, Julien Grall wrote:
> Hi Oleksandr,
>
> On 22/06/2020 15:04, Oleksandr Andrushchenko wrote:
>> On 6/19/20 11:02 PM, Stefano Stabellini wrote:
>>> On Thu, 18 Jun 2020, Julien Grall wrote:
>> ifeq ($(CONFIG_XEN),y)
>> arch-y += -D__XEN_INTERFACE_VERSION__=0x00040d00
>> endif
>>
>> and we also have Xen 4.13 headers in the U-boot tree.
>
> Sorry if this was already asked before. Why do you need to specify __XEN_INTERFACE_VERSION__?

This is because of include/xen/interface/xen-compat.h:

#if defined(__XEN__) || defined(__XEN_TOOLS__)

/* Xen is built with matching headers and implements the latest interface. */
#define __XEN_INTERFACE_VERSION__ __XEN_LATEST_INTERFACE_VERSION__
#elif !defined(__XEN_INTERFACE_VERSION__)
/* Guests which do not specify a version get the legacy interface. */
#define __XEN_INTERFACE_VERSION__ 0x00000000
#endif

So, one needs to specify the version (QEMU does that via its configure script after

some tests)

>
>>
>> For the first part (__XEN_INTERFACE_VERSION__) I think we can provide it via
>>
>> CFLAGS or something. This can also be done for the location of Xen headers.
>
> __XEN_INTERFACE_VERSION__ should work through the CFLAGS. An alternative would be to allow the user to specify through the Kconfig.

You mean specifying via Kconfig something like "0x00040d00"?

And what about the headers? How will we provide their location if we decide not to include those

in the tree?

> Cheers,
>

Re: UEFI support in ARM DomUs

[Julien Grall]

On 22/06/2020 15:33, Oleksandr Andrushchenko wrote:
> 
> On 6/22/20 5:27 PM, Julien Grall wrote:
>> Hi Oleksandr,
>>
>> On 22/06/2020 15:04, Oleksandr Andrushchenko wrote:
>>> On 6/19/20 11:02 PM, Stefano Stabellini wrote:
>>>> On Thu, 18 Jun 2020, Julien Grall wrote:
>>> ifeq ($(CONFIG_XEN),y)
>>> arch-y += -D__XEN_INTERFACE_VERSION__=0x00040d00
>>> endif
>>>
>>> and we also have Xen 4.13 headers in the U-boot tree.
>>
>> Sorry if this was already asked before. Why do you need to specify __XEN_INTERFACE_VERSION__?
> 
> This is because of include/xen/interface/xen-compat.h:
> 
> #if defined(__XEN__) || defined(__XEN_TOOLS__)
> 
> /* Xen is built with matching headers and implements the latest interface. */
> #define __XEN_INTERFACE_VERSION__ __XEN_LATEST_INTERFACE_VERSION__
> #elif !defined(__XEN_INTERFACE_VERSION__)
> /* Guests which do not specify a version get the legacy interface. */
> #define __XEN_INTERFACE_VERSION__ 0x00000000
> #endif

I am afraid this doesn't explain why you need to define it to a specific 
version.

__XEN_INTERFACE_VERSION__ is really mostly here to allow a guest to 
build if they rely on header from xen.git (we may have done some 
renaming). Most (if not all) the interfaces you care ought to be stable.

Older Xen will return -ENOSYS/-EOPNOTSUPP should deny any values they 
don't know.

As you pull the headers in U-boot, you could safely define 
__XEN_INTERFACE_VERSION__ as __XEN_LATEST_INTERFACE_VERSION__. FWIW, 
this is what Linux is doing to some extend.

Note that I haven't suggested to keep __XEN_INTERFACE_VERSION__ as 
0x00000000 because it looks like it is going to do the wrong thing on 
arm32 :(.

I have at least spot one issue with GNTTABOP_setup_table where it will 
use unsigned long (i.e 32-bit) for older interface. But the hypervisor 
will always 64-bits.

This probably going to result to some overwrite. I think we should fix 
the headers to force it to use xen_pfn_t for all Xen on Arm version.

Something like:

#if !(defined(__arch64__) || defined(__arm__)) && 
__XEN_INTERFACE_VERSION__ < 0x00040300
     XEN_GUEST_HANDLE(ulong) frame_list;
#else
     XEN_GUEST_HANDLE(xen_pfn_t) frame_list;
#endif

> 
> So, one needs to specify the version (QEMU does that via its configure script after
> 
> some tests)

Well libxc is definitely not stable, hence why QEMU requires to specify 
the version. But the interface with the guest is always meant to be stable.

>>
>>>
>>> For the first part (__XEN_INTERFACE_VERSION__) I think we can provide it via
>>>
>>> CFLAGS or something. This can also be done for the location of Xen headers.
>>
>> __XEN_INTERFACE_VERSION__ should work through the CFLAGS. An alternative would be to allow the user to specify through the Kconfig.
> 
> You mean specifying via Kconfig something like "0x00040d00"?

Possibly yes.

> 
> And what about the headers? How will we provide their location if we decide not to include those
> 
> in the tree?

I would do through Kconfig as well.

Cheers,

-- 
Julien Grall

Re: UEFI support in ARM DomUs

[Stefano Stabellini]

On Mon, 22 Jun 2020, Julien Grall wrote:
> > > > For the first part (__XEN_INTERFACE_VERSION__) I think we can provide it
> > > > via
> > > > 
> > > > CFLAGS or something. This can also be done for the location of Xen
> > > > headers.
> > > 
> > > __XEN_INTERFACE_VERSION__ should work through the CFLAGS. An alternative
> > > would be to allow the user to specify through the Kconfig.
> > 
> > You mean specifying via Kconfig something like "0x00040d00"?
> 
> Possibly yes.
> 
> > 
> > And what about the headers? How will we provide their location if we decide
> > not to include those
> > 
> > in the tree?
> 
> I would do through Kconfig as well.

If we specify the external location of the Xen headers via Kconfig, it
seems to me that we should be able to detect the interface version
automatically from any Makefile as part of the build. No need to ask the
user.

However, if Oleksandr is thinking of using the Xen headers for the
hypercalls definitions, then I think we might not need external headers
at all because that is a stable interface, as Julien wrote. We could
just define our own few headers for just what you need like Linux does.

If you can do that, I think it would be better because we decouple the
UBoot build from the Xen build completely. We don't even need the Xen
tree checked out to build UBoot. It would be a huge advantage because it
makes it far easier to build-test changes for others in the community
that don't know about Xen and also it becomes far easier to integrate
into any build system.

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/23/20 4:20 AM, Stefano Stabellini wrote:
> On Mon, 22 Jun 2020, Julien Grall wrote:
>>>>> For the first part (__XEN_INTERFACE_VERSION__) I think we can provide it
>>>>> via
>>>>>
>>>>> CFLAGS or something. This can also be done for the location of Xen
>>>>> headers.
>>>> __XEN_INTERFACE_VERSION__ should work through the CFLAGS. An alternative
>>>> would be to allow the user to specify through the Kconfig.
>>> You mean specifying via Kconfig something like "0x00040d00"?
>> Possibly yes.
>>
>>> And what about the headers? How will we provide their location if we decide
>>> not to include those
>>>
>>> in the tree?
>> I would do through Kconfig as well.
> If we specify the external location of the Xen headers via Kconfig, it
> seems to me that we should be able to detect the interface version
> automatically from any Makefile as part of the build. No need to ask the
> user.
>
> However, if Oleksandr is thinking of using the Xen headers for the
> hypercalls definitions, then I think we might not need external headers
> at all because that is a stable interface, as Julien wrote. We could
> just define our own few headers for just what you need like Linux does.

This is a good idea: I'll try to get the minimal set of headers from Linux

instead of Xen as those seem to be well prepared for such a use-case. This

way we'll have headers in U-boot tree and guarantee that we have a minimal

subset which is easier to maintain. I'll keep you updated on the progress

>
> If you can do that, I think it would be better because we decouple the
> UBoot build from the Xen build completely. We don't even need the Xen
> tree checked out to build UBoot. It would be a huge advantage because it
> makes it far easier to build-test changes for others in the community
> that don't know about Xen and also it becomes far easier to integrate
> into any build system.

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/23/20 8:31 AM, Oleksandr Andrushchenko wrote:
>
> On 6/23/20 4:20 AM, Stefano Stabellini wrote:
>> On Mon, 22 Jun 2020, Julien Grall wrote:
>>>>>> For the first part (__XEN_INTERFACE_VERSION__) I think we can provide it
>>>>>> via
>>>>>>
>>>>>> CFLAGS or something. This can also be done for the location of Xen
>>>>>> headers.
>>>>> __XEN_INTERFACE_VERSION__ should work through the CFLAGS. An alternative
>>>>> would be to allow the user to specify through the Kconfig.
>>>> You mean specifying via Kconfig something like "0x00040d00"?
>>> Possibly yes.
>>>
>>>> And what about the headers? How will we provide their location if we decide
>>>> not to include those
>>>>
>>>> in the tree?
>>> I would do through Kconfig as well.
>> If we specify the external location of the Xen headers via Kconfig, it
>> seems to me that we should be able to detect the interface version
>> automatically from any Makefile as part of the build. No need to ask the
>> user.
>>
>> However, if Oleksandr is thinking of using the Xen headers for the
>> hypercalls definitions, then I think we might not need external headers
>> at all because that is a stable interface, as Julien wrote. We could
>> just define our own few headers for just what you need like Linux does.
>
> This is a good idea: I'll try to get the minimal set of headers from Linux
>
> instead of Xen as those seem to be well prepared for such a use-case. This
>
> way we'll have headers in U-boot tree and guarantee that we have a minimal
>
> subset which is easier to maintain. I'll keep you updated on the progress

We've managed to strip the headers and remove __XEN__ and the rest definitions

we were talking about. So, these are now the minimal required set of headers

that allows U-boot to build serial and block drivers. Please take a look at [1]

Pull request for comments is at [2]

>
>>
>> If you can do that, I think it would be better because we decouple the
>> UBoot build from the Xen build completely. We don't even need the Xen
>> tree checked out to build UBoot. It would be a huge advantage because it
>> makes it far easier to build-test changes for others in the community
>> that don't know about Xen and also it becomes far easier to integrate
>> into any build system.

[1] https://github.com/andr2000/u-boot/tree/pvblock_upstream_v1

[2] https://github.com/xen-troops/u-boot/pull/2

RE: UEFI support in ARM DomUs

[Peng Fan]

> Subject: Re: UEFI support in ARM DomUs
> 
> 
> On 6/23/20 8:31 AM, Oleksandr Andrushchenko wrote:
> >
> > On 6/23/20 4:20 AM, Stefano Stabellini wrote:
> >> On Mon, 22 Jun 2020, Julien Grall wrote:
> >>>>>> For the first part (__XEN_INTERFACE_VERSION__) I think we can
> >>>>>> provide it via
> >>>>>>
> >>>>>> CFLAGS or something. This can also be done for the location of
> >>>>>> Xen headers.
> >>>>> __XEN_INTERFACE_VERSION__ should work through the CFLAGS. An
> >>>>> alternative would be to allow the user to specify through the Kconfig.
> >>>> You mean specifying via Kconfig something like "0x00040d00"?
> >>> Possibly yes.
> >>>
> >>>> And what about the headers? How will we provide their location if
> >>>> we decide not to include those
> >>>>
> >>>> in the tree?
> >>> I would do through Kconfig as well.
> >> If we specify the external location of the Xen headers via Kconfig,
> >> it seems to me that we should be able to detect the interface version
> >> automatically from any Makefile as part of the build. No need to ask
> >> the user.
> >>
> >> However, if Oleksandr is thinking of using the Xen headers for the
> >> hypercalls definitions, then I think we might not need external
> >> headers at all because that is a stable interface, as Julien wrote.
> >> We could just define our own few headers for just what you need like Linux
> does.
> >
> > This is a good idea: I'll try to get the minimal set of headers from
> > Linux
> >
> > instead of Xen as those seem to be well prepared for such a use-case.
> > This
> >
> > way we'll have headers in U-boot tree and guarantee that we have a
> > minimal
> >
> > subset which is easier to maintain. I'll keep you updated on the
> > progress
> 
> We've managed to strip the headers and remove __XEN__ and the rest
> definitions
> 
> we were talking about. So, these are now the minimal required set of headers
> 
> that allows U-boot to build serial and block drivers. Please take a look at [1]
> 
> Pull request for comments is at [2]

The U-Boot new merge window will be open in 2020/7/1, so I'd suggest
the patchset goes to U-Boot mail list for discussion if you wanna the patches
gonna merged soon.

Regards,
Peng.

> 
> >
> >>
> >> If you can do that, I think it would be better because we decouple
> >> the UBoot build from the Xen build completely. We don't even need the
> >> Xen tree checked out to build UBoot. It would be a huge advantage
> >> because it makes it far easier to build-test changes for others in
> >> the community that don't know about Xen and also it becomes far
> >> easier to integrate into any build system.
> 
> [1]
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.
> com%2Fandr2000%2Fu-boot%2Ftree%2Fpvblock_upstream_v1&data=0
> 2%7C01%7Cpeng.fan%40nxp.com%7C407d8af24a36483fbdce08d81805ed88
> %7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637285761021975
> 164&sdata=5vWfBbLScICXPZWU%2BU3b7DyONcgxT8iICsxrwUbORZY%
> 3D&reserved=0
> 
> [2]
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.
> com%2Fxen-troops%2Fu-boot%2Fpull%2F2&data=02%7C01%7Cpeng.fa
> n%40nxp.com%7C407d8af24a36483fbdce08d81805ed88%7C686ea1d3bc2b4
> c6fa92cd99c5c301635%7C0%7C0%7C637285761021975164&sdata=%2
> FmXheEvKssLjjaFKsHBBbqh%2B72jH3uQnE7cpN0J3k8I%3D&reserved=0

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/24/20 10:07 AM, Peng Fan wrote:
>> Subject: Re: UEFI support in ARM DomUs
>>
>>
>> On 6/23/20 8:31 AM, Oleksandr Andrushchenko wrote:
>>> On 6/23/20 4:20 AM, Stefano Stabellini wrote:
>>>> On Mon, 22 Jun 2020, Julien Grall wrote:
>>>>>>>> For the first part (__XEN_INTERFACE_VERSION__) I think we can
>>>>>>>> provide it via
>>>>>>>>
>>>>>>>> CFLAGS or something. This can also be done for the location of
>>>>>>>> Xen headers.
>>>>>>> __XEN_INTERFACE_VERSION__ should work through the CFLAGS. An
>>>>>>> alternative would be to allow the user to specify through the Kconfig.
>>>>>> You mean specifying via Kconfig something like "0x00040d00"?
>>>>> Possibly yes.
>>>>>
>>>>>> And what about the headers? How will we provide their location if
>>>>>> we decide not to include those
>>>>>>
>>>>>> in the tree?
>>>>> I would do through Kconfig as well.
>>>> If we specify the external location of the Xen headers via Kconfig,
>>>> it seems to me that we should be able to detect the interface version
>>>> automatically from any Makefile as part of the build. No need to ask
>>>> the user.
>>>>
>>>> However, if Oleksandr is thinking of using the Xen headers for the
>>>> hypercalls definitions, then I think we might not need external
>>>> headers at all because that is a stable interface, as Julien wrote.
>>>> We could just define our own few headers for just what you need like Linux
>> does.
>>> This is a good idea: I'll try to get the minimal set of headers from
>>> Linux
>>>
>>> instead of Xen as those seem to be well prepared for such a use-case.
>>> This
>>>
>>> way we'll have headers in U-boot tree and guarantee that we have a
>>> minimal
>>>
>>> subset which is easier to maintain. I'll keep you updated on the
>>> progress
>> We've managed to strip the headers and remove __XEN__ and the rest
>> definitions
>>
>> we were talking about. So, these are now the minimal required set of headers
>>
>> that allows U-boot to build serial and block drivers. Please take a look at [1]
>>
>> Pull request for comments is at [2]
> The U-Boot new merge window will be open in 2020/7/1, so I'd suggest
> the patchset goes to U-Boot mail list for discussion if you wanna the patches
> gonna merged soon.

We definitely want the patches to be upstreamed and merged, but before that

we also want to make sure that Xen community is happy with what we upstream

I believe we resolved most of the concerns such as headers, PIE etc, so this can

be done.

BTW, Peng, did you have a chance to try the pvblock driver yet?

>
> Regards,
> Peng.
>
>>>> If you can do that, I think it would be better because we decouple
>>>> the UBoot build from the Xen build completely. We don't even need the
>>>> Xen tree checked out to build UBoot. It would be a huge advantage
>>>> because it makes it far easier to build-test changes for others in
>>>> the community that don't know about Xen and also it becomes far
>>>> easier to integrate into any build system.
>> [1]
>> https://urldefense.com/v3/__https://eur01.safelinks.protection.outlook.com/?url=https*3A*2F*2Fgithub__;JSUl!!GF_29dbcQIUBPA!mwib3un6-vYBG68zMfurW6-0MuuER5tNmJtOitDpViICNkX0lhig8iPHmZokuM-BLQYpeKYAGQ$ .
>> com%2Fandr2000%2Fu-boot%2Ftree%2Fpvblock_upstream_v1&data=0
>> 2%7C01%7Cpeng.fan%40nxp.com%7C407d8af24a36483fbdce08d81805ed88
>> %7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637285761021975
>> 164&sdata=5vWfBbLScICXPZWU%2BU3b7DyONcgxT8iICsxrwUbORZY%
>> 3D&reserved=0
>>
>> [2]
>> https://urldefense.com/v3/__https://eur01.safelinks.protection.outlook.com/?url=https*3A*2F*2Fgithub__;JSUl!!GF_29dbcQIUBPA!mwib3un6-vYBG68zMfurW6-0MuuER5tNmJtOitDpViICNkX0lhig8iPHmZokuM-BLQYpeKYAGQ$ .
>> com%2Fxen-troops%2Fu-boot%2Fpull%2F2&data=02%7C01%7Cpeng.fa
>> n%40nxp.com%7C407d8af24a36483fbdce08d81805ed88%7C686ea1d3bc2b4
>> c6fa92cd99c5c301635%7C0%7C0%7C637285761021975164&sdata=%2
>> FmXheEvKssLjjaFKsHBBbqh%2B72jH3uQnE7cpN0J3k8I%3D&reserved=0

RE: UEFI support in ARM DomUs

[Peng Fan]

> Subject: Re: UEFI support in ARM DomUs
> 
> 
> On 6/24/20 10:07 AM, Peng Fan wrote:
> >> Subject: Re: UEFI support in ARM DomUs
> >>
> >>
> >> On 6/23/20 8:31 AM, Oleksandr Andrushchenko wrote:
> >>> On 6/23/20 4:20 AM, Stefano Stabellini wrote:
> >>>> On Mon, 22 Jun 2020, Julien Grall wrote:
> >>>>>>>> For the first part (__XEN_INTERFACE_VERSION__) I think we can
> >>>>>>>> provide it via
> >>>>>>>>
> >>>>>>>> CFLAGS or something. This can also be done for the location of
> >>>>>>>> Xen headers.
> >>>>>>> __XEN_INTERFACE_VERSION__ should work through the CFLAGS.
> An
> >>>>>>> alternative would be to allow the user to specify through the
> Kconfig.
> >>>>>> You mean specifying via Kconfig something like "0x00040d00"?
> >>>>> Possibly yes.
> >>>>>
> >>>>>> And what about the headers? How will we provide their location if
> >>>>>> we decide not to include those
> >>>>>>
> >>>>>> in the tree?
> >>>>> I would do through Kconfig as well.
> >>>> If we specify the external location of the Xen headers via Kconfig,
> >>>> it seems to me that we should be able to detect the interface
> >>>> version automatically from any Makefile as part of the build. No
> >>>> need to ask the user.
> >>>>
> >>>> However, if Oleksandr is thinking of using the Xen headers for the
> >>>> hypercalls definitions, then I think we might not need external
> >>>> headers at all because that is a stable interface, as Julien wrote.
> >>>> We could just define our own few headers for just what you need
> >>>> like Linux
> >> does.
> >>> This is a good idea: I'll try to get the minimal set of headers from
> >>> Linux
> >>>
> >>> instead of Xen as those seem to be well prepared for such a use-case.
> >>> This
> >>>
> >>> way we'll have headers in U-boot tree and guarantee that we have a
> >>> minimal
> >>>
> >>> subset which is easier to maintain. I'll keep you updated on the
> >>> progress
> >> We've managed to strip the headers and remove __XEN__ and the rest
> >> definitions
> >>
> >> we were talking about. So, these are now the minimal required set of
> >> headers
> >>
> >> that allows U-boot to build serial and block drivers. Please take a
> >> look at [1]
> >>
> >> Pull request for comments is at [2]
> > The U-Boot new merge window will be open in 2020/7/1, so I'd suggest
> > the patchset goes to U-Boot mail list for discussion if you wanna the
> > patches gonna merged soon.
> 
> We definitely want the patches to be upstreamed and merged, but before
> that
> 
> we also want to make sure that Xen community is happy with what we
> upstream
> 
> I believe we resolved most of the concerns such as headers, PIE etc, so this
> can
> 
> be done.
> 
> BTW, Peng, did you have a chance to try the pvblock driver yet?

Not yet. I could have time to give a test next Monday. I think you not
enable SPL, right? So android dual bootloader feature not enabled.
But SPL mostly not have MMU enabled..

Regards,
Peng.

> 
> >
> > Regards,
> > Peng.
> >
> >>>> If you can do that, I think it would be better because we decouple
> >>>> the UBoot build from the Xen build completely. We don't even need
> >>>> the Xen tree checked out to build UBoot. It would be a huge
> >>>> advantage because it makes it far easier to build-test changes for
> >>>> others in the community that don't know about Xen and also it
> >>>> becomes far easier to integrate into any build system.
> >> [1]
> >>
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldef
> ense.com%2Fv3%2F__https%3A%2F%2Feur01.safelinks.protection.outlook.c
> om%2F%3Furl%3Dhttps*3A*2F*2Fgithub__%3BJSUl!!GF_29dbcQIUBPA!mwi
> b3un6-vYBG68zMfurW6-0MuuER5tNmJtOitDpViICNkX0lhig8iPHmZokuM-BLQ
> YpeKYAGQ%24&data=02%7C01%7Cpeng.fan%40nxp.com%7C950d9c0a
> dc514927ce8b08d8180ea4ef%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C
> 0%7C0%7C637285798460563914&sdata=fMrI%2FQotknCsXV0amC4BFl
> 1Hg4vPw3zOMVdAVim2Wcs%3D&reserved=0 .
> >>
> com%2Fandr2000%2Fu-boot%2Ftree%2Fpvblock_upstream_v1&data=0
> >>
> 2%7C01%7Cpeng.fan%40nxp.com%7C407d8af24a36483fbdce08d81805ed88
> >> %7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637285761021
> 975
> >>
> 164&sdata=5vWfBbLScICXPZWU%2BU3b7DyONcgxT8iICsxrwUbORZY%
> >> 3D&reserved=0
> >>
> >> [2]
> >>
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldef
> ense.com%2Fv3%2F__https%3A%2F%2Feur01.safelinks.protection.outlook.c
> om%2F%3Furl%3Dhttps*3A*2F*2Fgithub__%3BJSUl!!GF_29dbcQIUBPA!mwi
> b3un6-vYBG68zMfurW6-0MuuER5tNmJtOitDpViICNkX0lhig8iPHmZokuM-BLQ
> YpeKYAGQ%24&data=02%7C01%7Cpeng.fan%40nxp.com%7C950d9c0a
> dc514927ce8b08d8180ea4ef%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C
> 0%7C0%7C637285798460563914&sdata=fMrI%2FQotknCsXV0amC4BFl
> 1Hg4vPw3zOMVdAVim2Wcs%3D&reserved=0 .
> >>
> com%2Fxen-troops%2Fu-boot%2Fpull%2F2&data=02%7C01%7Cpeng.fa
> >>
> n%40nxp.com%7C407d8af24a36483fbdce08d81805ed88%7C686ea1d3bc2b4
> >>
> c6fa92cd99c5c301635%7C0%7C0%7C637285761021975164&sdata=%2
> >>
> FmXheEvKssLjjaFKsHBBbqh%2B72jH3uQnE7cpN0J3k8I%3D&reserved=0

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/24/20 10:26 AM, Peng Fan wrote:
>> Subject: Re: UEFI support in ARM DomUs
>>
>>
>> On 6/24/20 10:07 AM, Peng Fan wrote:
>>>> Subject: Re: UEFI support in ARM DomUs
>>>>
>>>>
>>>> On 6/23/20 8:31 AM, Oleksandr Andrushchenko wrote:
>>>>> On 6/23/20 4:20 AM, Stefano Stabellini wrote:
>>>>>> On Mon, 22 Jun 2020, Julien Grall wrote:
>>>>>>>>>> For the first part (__XEN_INTERFACE_VERSION__) I think we can
>>>>>>>>>> provide it via
>>>>>>>>>>
>>>>>>>>>> CFLAGS or something. This can also be done for the location of
>>>>>>>>>> Xen headers.
>>>>>>>>> __XEN_INTERFACE_VERSION__ should work through the CFLAGS.
>> An
>>>>>>>>> alternative would be to allow the user to specify through the
>> Kconfig.
>>>>>>>> You mean specifying via Kconfig something like "0x00040d00"?
>>>>>>> Possibly yes.
>>>>>>>
>>>>>>>> And what about the headers? How will we provide their location if
>>>>>>>> we decide not to include those
>>>>>>>>
>>>>>>>> in the tree?
>>>>>>> I would do through Kconfig as well.
>>>>>> If we specify the external location of the Xen headers via Kconfig,
>>>>>> it seems to me that we should be able to detect the interface
>>>>>> version automatically from any Makefile as part of the build. No
>>>>>> need to ask the user.
>>>>>>
>>>>>> However, if Oleksandr is thinking of using the Xen headers for the
>>>>>> hypercalls definitions, then I think we might not need external
>>>>>> headers at all because that is a stable interface, as Julien wrote.
>>>>>> We could just define our own few headers for just what you need
>>>>>> like Linux
>>>> does.
>>>>> This is a good idea: I'll try to get the minimal set of headers from
>>>>> Linux
>>>>>
>>>>> instead of Xen as those seem to be well prepared for such a use-case.
>>>>> This
>>>>>
>>>>> way we'll have headers in U-boot tree and guarantee that we have a
>>>>> minimal
>>>>>
>>>>> subset which is easier to maintain. I'll keep you updated on the
>>>>> progress
>>>> We've managed to strip the headers and remove __XEN__ and the rest
>>>> definitions
>>>>
>>>> we were talking about. So, these are now the minimal required set of
>>>> headers
>>>>
>>>> that allows U-boot to build serial and block drivers. Please take a
>>>> look at [1]
>>>>
>>>> Pull request for comments is at [2]
>>> The U-Boot new merge window will be open in 2020/7/1, so I'd suggest
>>> the patchset goes to U-Boot mail list for discussion if you wanna the
>>> patches gonna merged soon.
>> We definitely want the patches to be upstreamed and merged, but before
>> that
>>
>> we also want to make sure that Xen community is happy with what we
>> upstream
>>
>> I believe we resolved most of the concerns such as headers, PIE etc, so this
>> can
>>
>> be done.
>>
>> BTW, Peng, did you have a chance to try the pvblock driver yet?
> Not yet. I could have time to give a test next Monday. I think you not
> enable SPL, right?

No, we decided that we can run with U-boot proper, so we can have more

flexibility comparing to what we have in SPL. It seems that was the right

approach: you can jump to Linux or you can jump to the U-boot provided

by Android anyway, whatever your setup

>   So android dual bootloader feature not enabled.

I think this step will depend on the use-case you have: at the moment we have

a base build capable of booting Linux kernel, but this can easily be extended with

specific defconfig to build in boota command or whatever else is required.

> But SPL mostly not have MMU enabled..
>
> Regards,
> Peng.
>
>>> Regards,
>>> Peng.
>>>
>>>>>> If you can do that, I think it would be better because we decouple
>>>>>> the UBoot build from the Xen build completely. We don't even need
>>>>>> the Xen tree checked out to build UBoot. It would be a huge
>>>>>> advantage because it makes it far easier to build-test changes for
>>>>>> others in the community that don't know about Xen and also it
>>>>>> becomes far easier to integrate into any build system.
>>>> [1]
>>>>
>> https://urldefense.com/v3/__https://eur01.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldef__;JSUl!!GF_29dbcQIUBPA!kJhWFr5ZO_UWn2oD_I9pDfnn64pZXw1ZBtASsxS9AZwbG65093ZydtlVPy6baPy4oeF957GBNA$
>> ense.com%2Fv3%2F__https%3A%2F%2Feur01.safelinks.protection.outlook.c
>> om%2F%3Furl%3Dhttps*3A*2F*2Fgithub__%3BJSUl!!GF_29dbcQIUBPA!mwi
>> b3un6-vYBG68zMfurW6-0MuuER5tNmJtOitDpViICNkX0lhig8iPHmZokuM-BLQ
>> YpeKYAGQ%24&data=02%7C01%7Cpeng.fan%40nxp.com%7C950d9c0a
>> dc514927ce8b08d8180ea4ef%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C
>> 0%7C0%7C637285798460563914&sdata=fMrI%2FQotknCsXV0amC4BFl
>> 1Hg4vPw3zOMVdAVim2Wcs%3D&reserved=0 .
>> com%2Fandr2000%2Fu-boot%2Ftree%2Fpvblock_upstream_v1&data=0
>> 2%7C01%7Cpeng.fan%40nxp.com%7C407d8af24a36483fbdce08d81805ed88
>>>> %7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637285761021
>> 975
>> 164&sdata=5vWfBbLScICXPZWU%2BU3b7DyONcgxT8iICsxrwUbORZY%
>>>> 3D&reserved=0
>>>>
>>>> [2]
>>>>
>> https://urldefense.com/v3/__https://eur01.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldef__;JSUl!!GF_29dbcQIUBPA!kJhWFr5ZO_UWn2oD_I9pDfnn64pZXw1ZBtASsxS9AZwbG65093ZydtlVPy6baPy4oeF957GBNA$
>> ense.com%2Fv3%2F__https%3A%2F%2Feur01.safelinks.protection.outlook.c
>> om%2F%3Furl%3Dhttps*3A*2F*2Fgithub__%3BJSUl!!GF_29dbcQIUBPA!mwi
>> b3un6-vYBG68zMfurW6-0MuuER5tNmJtOitDpViICNkX0lhig8iPHmZokuM-BLQ
>> YpeKYAGQ%24&data=02%7C01%7Cpeng.fan%40nxp.com%7C950d9c0a
>> dc514927ce8b08d8180ea4ef%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C
>> 0%7C0%7C637285798460563914&sdata=fMrI%2FQotknCsXV0amC4BFl
>> 1Hg4vPw3zOMVdAVim2Wcs%3D&reserved=0 .
>> com%2Fxen-troops%2Fu-boot%2Fpull%2F2&data=02%7C01%7Cpeng.fa
>> n%40nxp.com%7C407d8af24a36483fbdce08d81805ed88%7C686ea1d3bc2b4
>> c6fa92cd99c5c301635%7C0%7C0%7C637285761021975164&sdata=%2
>> FmXheEvKssLjjaFKsHBBbqh%2B72jH3uQnE7cpN0J3k8I%3D&reserved=0

Re: UEFI support in ARM DomUs

[Stefano Stabellini]

On Wed, 24 Jun 2020, Oleksandr Andrushchenko wrote:
> On 6/23/20 8:31 AM, Oleksandr Andrushchenko wrote:
> >
> > On 6/23/20 4:20 AM, Stefano Stabellini wrote:
> >> On Mon, 22 Jun 2020, Julien Grall wrote:
> >>>>>> For the first part (__XEN_INTERFACE_VERSION__) I think we can provide it
> >>>>>> via
> >>>>>>
> >>>>>> CFLAGS or something. This can also be done for the location of Xen
> >>>>>> headers.
> >>>>> __XEN_INTERFACE_VERSION__ should work through the CFLAGS. An alternative
> >>>>> would be to allow the user to specify through the Kconfig.
> >>>> You mean specifying via Kconfig something like "0x00040d00"?
> >>> Possibly yes.
> >>>
> >>>> And what about the headers? How will we provide their location if we decide
> >>>> not to include those
> >>>>
> >>>> in the tree?
> >>> I would do through Kconfig as well.
> >> If we specify the external location of the Xen headers via Kconfig, it
> >> seems to me that we should be able to detect the interface version
> >> automatically from any Makefile as part of the build. No need to ask the
> >> user.
> >>
> >> However, if Oleksandr is thinking of using the Xen headers for the
> >> hypercalls definitions, then I think we might not need external headers
> >> at all because that is a stable interface, as Julien wrote. We could
> >> just define our own few headers for just what you need like Linux does.
> >
> > This is a good idea: I'll try to get the minimal set of headers from Linux
> >
> > instead of Xen as those seem to be well prepared for such a use-case. This
> >
> > way we'll have headers in U-boot tree and guarantee that we have a minimal
> >
> > subset which is easier to maintain. I'll keep you updated on the progress
> 
> We've managed to strip the headers and remove __XEN__ and the rest definitions
> 
> we were talking about. So, these are now the minimal required set of headers
> 
> that allows U-boot to build serial and block drivers. Please take a look at [1]
> 
> Pull request for comments is at [2]

I think this is the right approach. There is no build-dependency on Xen
anymore, is that correct?

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/24/20 8:05 PM, Stefano Stabellini wrote:
> On Wed, 24 Jun 2020, Oleksandr Andrushchenko wrote:
>> On 6/23/20 8:31 AM, Oleksandr Andrushchenko wrote:
>>> On 6/23/20 4:20 AM, Stefano Stabellini wrote:
>>>> On Mon, 22 Jun 2020, Julien Grall wrote:
>>>>>>>> For the first part (__XEN_INTERFACE_VERSION__) I think we can provide it
>>>>>>>> via
>>>>>>>>
>>>>>>>> CFLAGS or something. This can also be done for the location of Xen
>>>>>>>> headers.
>>>>>>> __XEN_INTERFACE_VERSION__ should work through the CFLAGS. An alternative
>>>>>>> would be to allow the user to specify through the Kconfig.
>>>>>> You mean specifying via Kconfig something like "0x00040d00"?
>>>>> Possibly yes.
>>>>>
>>>>>> And what about the headers? How will we provide their location if we decide
>>>>>> not to include those
>>>>>>
>>>>>> in the tree?
>>>>> I would do through Kconfig as well.
>>>> If we specify the external location of the Xen headers via Kconfig, it
>>>> seems to me that we should be able to detect the interface version
>>>> automatically from any Makefile as part of the build. No need to ask the
>>>> user.
>>>>
>>>> However, if Oleksandr is thinking of using the Xen headers for the
>>>> hypercalls definitions, then I think we might not need external headers
>>>> at all because that is a stable interface, as Julien wrote. We could
>>>> just define our own few headers for just what you need like Linux does.
>>> This is a good idea: I'll try to get the minimal set of headers from Linux
>>>
>>> instead of Xen as those seem to be well prepared for such a use-case. This
>>>
>>> way we'll have headers in U-boot tree and guarantee that we have a minimal
>>>
>>> subset which is easier to maintain. I'll keep you updated on the progress
>> We've managed to strip the headers and remove __XEN__ and the rest definitions
>>
>> we were talking about. So, these are now the minimal required set of headers
>>
>> that allows U-boot to build serial and block drivers. Please take a look at [1]
>>
>> Pull request for comments is at [2]
> I think this is the right approach. There is no build-dependency on Xen
> anymore, is that correct?
No dependency

Re: UEFI support in ARM DomUs

[Stefano Stabellini]

On Wed, 24 Jun 2020, Oleksandr Andrushchenko wrote:
> On 6/24/20 8:05 PM, Stefano Stabellini wrote:
> > On Wed, 24 Jun 2020, Oleksandr Andrushchenko wrote:
> >> On 6/23/20 8:31 AM, Oleksandr Andrushchenko wrote:
> >>> On 6/23/20 4:20 AM, Stefano Stabellini wrote:
> >>>> On Mon, 22 Jun 2020, Julien Grall wrote:
> >>>>>>>> For the first part (__XEN_INTERFACE_VERSION__) I think we can provide it
> >>>>>>>> via
> >>>>>>>>
> >>>>>>>> CFLAGS or something. This can also be done for the location of Xen
> >>>>>>>> headers.
> >>>>>>> __XEN_INTERFACE_VERSION__ should work through the CFLAGS. An alternative
> >>>>>>> would be to allow the user to specify through the Kconfig.
> >>>>>> You mean specifying via Kconfig something like "0x00040d00"?
> >>>>> Possibly yes.
> >>>>>
> >>>>>> And what about the headers? How will we provide their location if we decide
> >>>>>> not to include those
> >>>>>>
> >>>>>> in the tree?
> >>>>> I would do through Kconfig as well.
> >>>> If we specify the external location of the Xen headers via Kconfig, it
> >>>> seems to me that we should be able to detect the interface version
> >>>> automatically from any Makefile as part of the build. No need to ask the
> >>>> user.
> >>>>
> >>>> However, if Oleksandr is thinking of using the Xen headers for the
> >>>> hypercalls definitions, then I think we might not need external headers
> >>>> at all because that is a stable interface, as Julien wrote. We could
> >>>> just define our own few headers for just what you need like Linux does.
> >>> This is a good idea: I'll try to get the minimal set of headers from Linux
> >>>
> >>> instead of Xen as those seem to be well prepared for such a use-case. This
> >>>
> >>> way we'll have headers in U-boot tree and guarantee that we have a minimal
> >>>
> >>> subset which is easier to maintain. I'll keep you updated on the progress
> >> We've managed to strip the headers and remove __XEN__ and the rest definitions
> >>
> >> we were talking about. So, these are now the minimal required set of headers
> >>
> >> that allows U-boot to build serial and block drivers. Please take a look at [1]
> >>
> >> Pull request for comments is at [2]
> > I think this is the right approach. There is no build-dependency on Xen
> > anymore, is that correct?
>
> No dependency

Great!

Re: UEFI support in ARM DomUs

[Oleksandr Andrushchenko]

On 6/18/20 5:50 PM, Julien Grall wrote:
>
>
> On 18/06/2020 06:22, Oleksandr Andrushchenko wrote:
>>
>> On 6/4/20 6:31 PM, Stefano Stabellini wrote:
>>> On Thu, 4 Jun 2020, Oleksandr Andrushchenko wrote:
>>>> On 6/4/20 4:57 AM, Peng Fan wrote:
>>>>> Grall <julien@xen.org>;
>>>>>> Nataliya Korovkina <malus.brandywine@gmail.com>
>>>>>> Subject: UEFI support in ARM DomUs
>>>>> We have made U-Boot run inside XEN DomU, but just only PV console part,
>>>>> not implement other frontend drivers currently. Would this help for your
>>>>> case if enable EFI in U-Boot?
>>>> Well, we have a working PV block implementation on top of that on iMX8
>>>>
>>>> platform, mostly ported from mini-os. Currently we are finalizing the work
>>>>
>>>> and cleaning up (it's going to take a week or so hopefully). Then, we we'll post
>>>>
>>>> it on our public github. We are also thinking about upstreaming the work, but it may
>>>>
>>>> take quite some time if the whole idea fits u-boot's view on such an extension at all.
>>> Yes please to both of you! :-)
>>>
>>> In the meantime, while we wait for those changes to go upstream in
>>> uboot, could you please post a branch on github and a link on this email
>>> thread?
>>
>> It took a bit more time than we expected, but here we go [1]:
>>
>> this is in form of a pull-request as we would love to hear from the
>>
>> community and it is easier to discuss the code (please leave comments there)
>>
>> 1. There is code originating from MiniOS and work done by Peng, so we
>>
>> would like to ask the respective copyright owners to raise their hands and
>
> Not everyone are closely watching xen-devel. So if you want to find out who are the copyright owners, then your best solution is to go through the git log and CC the authors.

We didn't want to contact the respective authors and copyright owners (some of those

are dated 2005 or so). This is to stress that we are trying hard to put all the copyrights

and not miss anyone: there is no intention to put our copyright at some inappropriate

place and remove someones else.

>
>>
>> let us *fix inappropriate licensing* if any.
>>
>> 2. Please note, the series has a HACK to move the RAM base as it is expected by
>>
>> our test platform (iMX8), so others will need to remove or modify that.
>>
>> 3. There is a limitation already noted by Peng that we do not have serial output
>>
>> until MMU is setup, so we have introduced xen_early_printk helper which always
>>
>> works, so you can use that for early stage debugging.
>>
>> 4. Minimal memory size supported is ~129M because of dtb placement by Xen tools
>
> Hmmm... Why? What's wrong with booting a guest with just 64MB?

Because of the bug in U-boot [1]: it tries to read beyond the physical memory if guest

has less than 128M+ as only then Xen tools put the fdt at 128M and not at the RAM end.

>
>>
>> 5. We use -D__XEN__ to access some of the hidden defines we need such as
>>
>> GUEST_RAM0_BASE and the friends as there is no other way but manually defining the
>>
>> same which is not cute.
>
> I have replied to this in the pull request. But I want to bring the conversation here to have a wider discussion.
>
> For a first, __XEN__ should really only be defined by the hypervisor and not used by the guest. This is used to gate non-stable ABI (such as the tools) and anything behind it hasn't been vetted to work in other build configuration that the one used by Xen.
>
> In general, we expect the guest to discover everything for the device-tree because the memory layout is not stable (we want to be able to reshuffle as we add more features).
>
> I know that EDK2/Tianocore can be built once and work on every Xen configuration. It would be ideal that U-boot follow the same. If it is really not possible, then we should explore a path that is preventing to define __XEN__.
>
> How much does U-boot expect to know about the memory layout? Does it require to know all the memory banks? Or would it be sufficient for it to know the start address of the first bank and the minimum of RAM?
>
> Cheers,
>
[1] https://github.com/xen-troops/u-boot/pull/1/commits/9c639bd514961e70cfb2ebed9d8badb7b22d21ab

Re: UEFI support in ARM DomUs

[Julien Grall]

Hi,

On 04/06/2020 16:26, Oleksandr Andrushchenko wrote:
> On 6/4/20 4:57 AM, Peng Fan wrote:
>> Grall <julien@xen.org>;
>>> Nataliya Korovkina <malus.brandywine@gmail.com>
>>> Subject: UEFI support in ARM DomUs
>> We have made U-Boot run inside XEN DomU, but just only PV console part,
>> not implement other frontend drivers currently. Would this help for your
>> case if enable EFI in U-Boot?
> 
> Well, we have a working PV block implementation on top of that on iMX8

That's a nice work and will be a good addition. However...

> 
> platform, mostly ported from mini-os. Currently we are finalizing the work

... AFAICT mini-os is licensed using BSD-2 while U-boot is using GPLv2. 
So I would be careful with the licensing here.

It might be better to consider to port Linux PV driver instead or 
rewrite them completely.

Cheers,

-- 
Julien Grall

Re: UEFI support in ARM DomUs

[Stefano Stabellini]

On Thu, 4 Jun 2020, Julien Grall wrote:
> On 04/06/2020 16:26, Oleksandr Andrushchenko wrote:
> > On 6/4/20 4:57 AM, Peng Fan wrote:
> > > Grall <julien@xen.org>;
> > > > Nataliya Korovkina <malus.brandywine@gmail.com>
> > > > Subject: UEFI support in ARM DomUs
> > > We have made U-Boot run inside XEN DomU, but just only PV console part,
> > > not implement other frontend drivers currently. Would this help for your
> > > case if enable EFI in U-Boot?
> > 
> > Well, we have a working PV block implementation on top of that on iMX8
> 
> That's a nice work and will be a good addition. However...
> 
> > 
> > platform, mostly ported from mini-os. Currently we are finalizing the work
> 
> ... AFAICT mini-os is licensed using BSD-2 while U-boot is using GPLv2. So I
> would be careful with the licensing here.
> 
> It might be better to consider to port Linux PV driver instead or rewrite them
> completely.

Julien, I might be misunderstanding what you wrote. MiniOS is BSD-2 so
MiniOS code can be integrated into a GPLv2 project without issues
(becoming GPLv2.)  So it should be OK to take MiniOS code and add it to
Uboot?

The other way around would be an issue: you cannot take GPLv2 code and
add it to a BSD-2 project.

Re: UEFI support in ARM DomUs

[Julien Grall]

Hi,

On 04/06/2020 17:27, Stefano Stabellini wrote:
> On Thu, 4 Jun 2020, Julien Grall wrote:
>> On 04/06/2020 16:26, Oleksandr Andrushchenko wrote:
>>> On 6/4/20 4:57 AM, Peng Fan wrote:
>>>> Grall <julien@xen.org>;
>>>>> Nataliya Korovkina <malus.brandywine@gmail.com>
>>>>> Subject: UEFI support in ARM DomUs
>>>> We have made U-Boot run inside XEN DomU, but just only PV console part,
>>>> not implement other frontend drivers currently. Would this help for your
>>>> case if enable EFI in U-Boot?
>>>
>>> Well, we have a working PV block implementation on top of that on iMX8
>>
>> That's a nice work and will be a good addition. However...
>>
>>>
>>> platform, mostly ported from mini-os. Currently we are finalizing the work
>>
>> ... AFAICT mini-os is licensed using BSD-2 while U-boot is using GPLv2. So I
>> would be careful with the licensing here.
>>
>> It might be better to consider to port Linux PV driver instead or rewrite them
>> completely.
> 
> Julien, I might be misunderstanding what you wrote. MiniOS is BSD-2 so
> MiniOS code can be integrated into a GPLv2 project without issues
> (becoming GPLv2.)  So it should be OK to take MiniOS code and add it to
> Uboot?

Yes I did realized that afterwards. Sorry for the noise :(.

Cheers,

-- 
Julien Grall