Question: Enable LINUX_EFI_MEMRESERVE_TABLE_GUID in EFI

Question: Enable LINUX_EFI_MEMRESERVE_TABLE_GUID in EFI

[Leo Yan]

Hi there,

Now I am working on Ampere Altra SoC platform, with Xen (4.16) and Linux
kernel (5.15.23).

I observed a warning is reported by Linux kernel in the booting flow:

[    0.403737] ------------[ cut here ]------------
[    0.403738] WARNING: CPU: 30 PID: 0 at drivers/irqchip/irq-gic-v3-its.c:3074 its_cpu_init+0x814/0xae0
[    0.403745] Modules linked in:
[    0.403748] CPU: 30 PID: 0 Comm: swapper/30 Tainted: G        W         5.15.23-ampere-lts-standard #1
[    0.403752] pstate: 600001c5 (nZCv dAIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    0.403755] pc : its_cpu_init+0x814/0xae0
[    0.403758] lr : its_cpu_init+0x810/0xae0
[    0.403761] sp : ffff800009c03ce0
[    0.403762] x29: ffff800009c03ce0 x28: 000000000000001e x27: ffff880711f43000
[    0.403767] x26: ffff80000a3c0070 x25: fffffc1ffe0a4400 x24: ffff80000a3c0000
[    0.403770] x23: ffff8000095bc998 x22: ffff8000090a6000 x21: ffff800009850cb0
[    0.403774] x20: ffff800009701a10 x19: ffff800009701000 x18: ffffffffffffffff
[    0.403777] x17: 3030303035303031 x16: 3030313030303078 x15: 303a30206e6f6967
[    0.403780] x14: 6572206530312072 x13: 3030303030353030 x12: 3130303130303030
[    0.403784] x11: 78303a30206e6f69 x10: 6765722065303120 x9 : ffff80000870e710
[    0.403788] x8 : 6964657220646e75 x7 : 0000000000000003 x6 : 0000000000000000
[    0.403791] x5 : 0000000000000000 x4 : fffffc0000000000 x3 : 0000000000000010
[    0.403794] x2 : 000000000000ffff x1 : 0000000000010000 x0 : 00000000ffffffed
[    0.403798] Call trace:
[    0.403799]  its_cpu_init+0x814/0xae0
[    0.403802]  gic_starting_cpu+0x48/0x90
[    0.403805]  cpuhp_invoke_callback+0x16c/0x5b0
[    0.403808]  cpuhp_invoke_callback_range+0x78/0xf0
[    0.403811]  notify_cpu_starting+0xbc/0xdc
[    0.403814]  secondary_start_kernel+0xe0/0x170
[    0.403817]  __secondary_switched+0x94/0x98
[    0.403821] ---[ end trace f68728a0d3053b70 ]---

Looked into the code, the GICv3 driver tries to create persistent
reservations for pending pages, and the persistent reservation table
can be used by kexec/kdump.  For the persistent reservations, it
relies on MEMRESERVE EFI configuration table, but this table is not
supported by xen.efi, I think this is the reason for the above oops.

I checked that if I boot a host Linux (without Xen), then the EFI has
provided MEMRESERVE configuration table, I can get below log:

  #  dmesg | grep MEMRESERVE
  [    0.000000] efi: TPMFinalLog=0x807f9ef0000 ACPI 2.0=0x807fa0d0018 ... MEMRESERVE=0x807f8141e98

Just want to confirm, is anyone working on enabling MEMRESERVE EFI
configuration table for Xen?  And welcome any comments and
suggestions!

Thanks,
Leo

Re: Question: Enable LINUX_EFI_MEMRESERVE_TABLE_GUID in EFI

[Rahul Singh]

Hi Leo,

> On 4 Aug 2022, at 9:12 am, Leo Yan <leo.yan@linaro.org> wrote:
> 
> Hi there,
> 
> Now I am working on Ampere Altra SoC platform, with Xen (4.16) and Linux
> kernel (5.15.23).
> 
> I observed a warning is reported by Linux kernel in the booting flow:
> 
> [    0.403737] ------------[ cut here ]------------
> [    0.403738] WARNING: CPU: 30 PID: 0 at drivers/irqchip/irq-gic-v3-its.c:3074 its_cpu_init+0x814/0xae0
> [    0.403745] Modules linked in:
> [    0.403748] CPU: 30 PID: 0 Comm: swapper/30 Tainted: G        W         5.15.23-ampere-lts-standard #1
> [    0.403752] pstate: 600001c5 (nZCv dAIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [    0.403755] pc : its_cpu_init+0x814/0xae0
> [    0.403758] lr : its_cpu_init+0x810/0xae0
> [    0.403761] sp : ffff800009c03ce0
> [    0.403762] x29: ffff800009c03ce0 x28: 000000000000001e x27: ffff880711f43000
> [    0.403767] x26: ffff80000a3c0070 x25: fffffc1ffe0a4400 x24: ffff80000a3c0000
> [    0.403770] x23: ffff8000095bc998 x22: ffff8000090a6000 x21: ffff800009850cb0
> [    0.403774] x20: ffff800009701a10 x19: ffff800009701000 x18: ffffffffffffffff
> [    0.403777] x17: 3030303035303031 x16: 3030313030303078 x15: 303a30206e6f6967
> [    0.403780] x14: 6572206530312072 x13: 3030303030353030 x12: 3130303130303030
> [    0.403784] x11: 78303a30206e6f69 x10: 6765722065303120 x9 : ffff80000870e710
> [    0.403788] x8 : 6964657220646e75 x7 : 0000000000000003 x6 : 0000000000000000
> [    0.403791] x5 : 0000000000000000 x4 : fffffc0000000000 x3 : 0000000000000010
> [    0.403794] x2 : 000000000000ffff x1 : 0000000000010000 x0 : 00000000ffffffed
> [    0.403798] Call trace:
> [    0.403799]  its_cpu_init+0x814/0xae0
> [    0.403802]  gic_starting_cpu+0x48/0x90
> [    0.403805]  cpuhp_invoke_callback+0x16c/0x5b0
> [    0.403808]  cpuhp_invoke_callback_range+0x78/0xf0
> [    0.403811]  notify_cpu_starting+0xbc/0xdc
> [    0.403814]  secondary_start_kernel+0xe0/0x170
> [    0.403817]  __secondary_switched+0x94/0x98
> [    0.403821] ---[ end trace f68728a0d3053b70 ]---
> 
> Looked into the code, the GICv3 driver tries to create persistent
> reservations for pending pages, and the persistent reservation table
> can be used by kexec/kdump.  For the persistent reservations, it
> relies on MEMRESERVE EFI configuration table, but this table is not
> supported by xen.efi, I think this is the reason for the above oops.

Yes, you are right above warning is observed because Xen does not support 
memreserve efi table. I also observed a similar warning on the N1SDP board.
> 
> I checked that if I boot a host Linux (without Xen), then the EFI has
> provided MEMRESERVE configuration table, I can get below log:
> 
>  #  dmesg | grep MEMRESERVE
>  [    0.000000] efi: TPMFinalLog=0x807f9ef0000 ACPI 2.0=0x807fa0d0018 ... MEMRESERVE=0x807f8141e98
> 
> Just want to confirm, is anyone working on enabling MEMRESERVE EFI
> configuration table for Xen?  And welcome any comments and
> suggestions!
> 


Regards,
Rahul

Re: Question: Enable LINUX_EFI_MEMRESERVE_TABLE_GUID in EFI

[Bertrand Marquis]

HI Leo

> On 5 Aug 2022, at 12:44, Rahul Singh <Rahul.Singh@arm.com> wrote:
> 
> Hi Leo,
> 
>> On 4 Aug 2022, at 9:12 am, Leo Yan <leo.yan@linaro.org> wrote:
>> 
>> Hi there,
>> 
>> Now I am working on Ampere Altra SoC platform, with Xen (4.16) and Linux
>> kernel (5.15.23).
>> 
>> I observed a warning is reported by Linux kernel in the booting flow:
>> 
>> [    0.403737] ------------[ cut here ]------------
>> [    0.403738] WARNING: CPU: 30 PID: 0 at drivers/irqchip/irq-gic-v3-its.c:3074 its_cpu_init+0x814/0xae0
>> [    0.403745] Modules linked in:
>> [    0.403748] CPU: 30 PID: 0 Comm: swapper/30 Tainted: G        W         5.15.23-ampere-lts-standard #1
>> [    0.403752] pstate: 600001c5 (nZCv dAIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
>> [    0.403755] pc : its_cpu_init+0x814/0xae0
>> [    0.403758] lr : its_cpu_init+0x810/0xae0
>> [    0.403761] sp : ffff800009c03ce0
>> [    0.403762] x29: ffff800009c03ce0 x28: 000000000000001e x27: ffff880711f43000
>> [    0.403767] x26: ffff80000a3c0070 x25: fffffc1ffe0a4400 x24: ffff80000a3c0000
>> [    0.403770] x23: ffff8000095bc998 x22: ffff8000090a6000 x21: ffff800009850cb0
>> [    0.403774] x20: ffff800009701a10 x19: ffff800009701000 x18: ffffffffffffffff
>> [    0.403777] x17: 3030303035303031 x16: 3030313030303078 x15: 303a30206e6f6967
>> [    0.403780] x14: 6572206530312072 x13: 3030303030353030 x12: 3130303130303030
>> [    0.403784] x11: 78303a30206e6f69 x10: 6765722065303120 x9 : ffff80000870e710
>> [    0.403788] x8 : 6964657220646e75 x7 : 0000000000000003 x6 : 0000000000000000
>> [    0.403791] x5 : 0000000000000000 x4 : fffffc0000000000 x3 : 0000000000000010
>> [    0.403794] x2 : 000000000000ffff x1 : 0000000000010000 x0 : 00000000ffffffed
>> [    0.403798] Call trace:
>> [    0.403799]  its_cpu_init+0x814/0xae0
>> [    0.403802]  gic_starting_cpu+0x48/0x90
>> [    0.403805]  cpuhp_invoke_callback+0x16c/0x5b0
>> [    0.403808]  cpuhp_invoke_callback_range+0x78/0xf0
>> [    0.403811]  notify_cpu_starting+0xbc/0xdc
>> [    0.403814]  secondary_start_kernel+0xe0/0x170
>> [    0.403817]  __secondary_switched+0x94/0x98
>> [    0.403821] ---[ end trace f68728a0d3053b70 ]---
>> 
>> Looked into the code, the GICv3 driver tries to create persistent
>> reservations for pending pages, and the persistent reservation table
>> can be used by kexec/kdump.  For the persistent reservations, it
>> relies on MEMRESERVE EFI configuration table, but this table is not
>> supported by xen.efi, I think this is the reason for the above oops.
> 
> Yes, you are right above warning is observed because Xen does not support 
> memreserve efi table. I also observed a similar warning on the N1SDP board.
>> 
>> I checked that if I boot a host Linux (without Xen), then the EFI has
>> provided MEMRESERVE configuration table, I can get below log:
>> 
>> #  dmesg | grep MEMRESERVE
>> [    0.000000] efi: TPMFinalLog=0x807f9ef0000 ACPI 2.0=0x807fa0d0018 ... MEMRESERVE=0x807f8141e98
>> 
>> Just want to confirm, is anyone working on enabling MEMRESERVE EFI
>> configuration table for Xen?  And welcome any comments and
>> suggestions!
>> 

No I do not think anybody is working on this at the moment.
If you want to work on adding support for this in Xen, we can provide support
and help on reviewing and testing as we have several targets on which we
observe this (N1SDP and Ava).

Cheers
Bertrand

Re: Question: Enable LINUX_EFI_MEMRESERVE_TABLE_GUID in EFI

[Leo Yan]

Hi Bertrand, Rahul,

On Fri, Aug 05, 2022 at 12:05:23PM +0000, Bertrand Marquis wrote:
> > On 5 Aug 2022, at 12:44, Rahul Singh <Rahul.Singh@arm.com> wrote:

[...]

> >> Looked into the code, the GICv3 driver tries to create persistent
> >> reservations for pending pages, and the persistent reservation table
> >> can be used by kexec/kdump.  For the persistent reservations, it
> >> relies on MEMRESERVE EFI configuration table, but this table is not
> >> supported by xen.efi, I think this is the reason for the above oops.
> > 
> > Yes, you are right above warning is observed because Xen does not support 
> > memreserve efi table. I also observed a similar warning on the N1SDP board.
> >> 
> >> I checked that if I boot a host Linux (without Xen), then the EFI has
> >> provided MEMRESERVE configuration table, I can get below log:
> >> 
> >> #  dmesg | grep MEMRESERVE
> >> [    0.000000] efi: TPMFinalLog=0x807f9ef0000 ACPI 2.0=0x807fa0d0018 ... MEMRESERVE=0x807f8141e98
> >> 
> >> Just want to confirm, is anyone working on enabling MEMRESERVE EFI
> >> configuration table for Xen?  And welcome any comments and
> >> suggestions!
> >> 
> 
> No I do not think anybody is working on this at the moment.
> If you want to work on adding support for this in Xen, we can provide support
> and help on reviewing and testing as we have several targets on which we
> observe this (N1SDP and Ava).

Thanks for your quick response.

I took time to look into the code, below are my conclusions.

For a normal UEFI boot flow, UEFI will invoke Linux kernel's EFI stub,
and the EFI stub will install MEMRESERVE EFI configuration table.
This is accomplished in the Linux function install_memreserve_table().

Secondly, Xen passes DT to kernel, it synthesizes ACPI compatible
nodes in the device tree and finally kernel parses DT and create the
ACPI table.  In this case, Xen doesn't invoke Linux EFI stub.

To be honest, I have very less knowledge for Xen and APCI; just based on
reading code, I think it's hard for Xen to invoke Linux kernel's EFI
stub, this is because Xen needs to provide the EFI runtime services, and
I don't think it's feasible for Xen to pass through UEFI's runtime
service to Linux kernel.  If we implement the EFI runtime services for
Xen, then this would introduce a big implementation.

So another option is we simply add MEMRESERVE EFI configuration table
into device tree, just like Xen does for other ACPI tables (e.g.
RSDP?).  And then in Linux kernel, we need to parse the DT binding and
initialize the corresponding variables in kernel, so we need to add
support in the Linux source drivers/firmware/efi/fdtparams.c.

Before I proceed, just want to check which option would be the right
way to move forward?  And I am open for any other solution and welcome
suggestions.

Thanks a lot!
Leo

Re: Question: Enable LINUX_EFI_MEMRESERVE_TABLE_GUID in EFI

[Bertrand Marquis]

Hi Leon,

> On 11 Aug 2022, at 15:17, Leo Yan <leo.yan@linaro.org> wrote:
> 
> Hi Bertrand, Rahul,
> 
> On Fri, Aug 05, 2022 at 12:05:23PM +0000, Bertrand Marquis wrote:
>>> On 5 Aug 2022, at 12:44, Rahul Singh <Rahul.Singh@arm.com> wrote:
> 
> [...]
> 
>>>> Looked into the code, the GICv3 driver tries to create persistent
>>>> reservations for pending pages, and the persistent reservation table
>>>> can be used by kexec/kdump.  For the persistent reservations, it
>>>> relies on MEMRESERVE EFI configuration table, but this table is not
>>>> supported by xen.efi, I think this is the reason for the above oops.
>>> 
>>> Yes, you are right above warning is observed because Xen does not support 
>>> memreserve efi table. I also observed a similar warning on the N1SDP board.
>>>> 
>>>> I checked that if I boot a host Linux (without Xen), then the EFI has
>>>> provided MEMRESERVE configuration table, I can get below log:
>>>> 
>>>> #  dmesg | grep MEMRESERVE
>>>> [    0.000000] efi: TPMFinalLog=0x807f9ef0000 ACPI 2.0=0x807fa0d0018 ... MEMRESERVE=0x807f8141e98
>>>> 
>>>> Just want to confirm, is anyone working on enabling MEMRESERVE EFI
>>>> configuration table for Xen?  And welcome any comments and
>>>> suggestions!
>>>> 
>> 
>> No I do not think anybody is working on this at the moment.
>> If you want to work on adding support for this in Xen, we can provide support
>> and help on reviewing and testing as we have several targets on which we
>> observe this (N1SDP and Ava).
> 
> Thanks for your quick response.
> 
> I took time to look into the code, below are my conclusions.
> 
> For a normal UEFI boot flow, UEFI will invoke Linux kernel's EFI stub,
> and the EFI stub will install MEMRESERVE EFI configuration table.
> This is accomplished in the Linux function install_memreserve_table().
> 
> Secondly, Xen passes DT to kernel, it synthesizes ACPI compatible
> nodes in the device tree and finally kernel parses DT and create the
> ACPI table.  In this case, Xen doesn't invoke Linux EFI stub.
> 
> To be honest, I have very less knowledge for Xen and APCI; just based on
> reading code, I think it's hard for Xen to invoke Linux kernel's EFI
> stub, this is because Xen needs to provide the EFI runtime services, and
> I don't think it's feasible for Xen to pass through UEFI's runtime
> service to Linux kernel.  If we implement the EFI runtime services for
> Xen, then this would introduce a big implementation.
> 
> So another option is we simply add MEMRESERVE EFI configuration table
> into device tree, just like Xen does for other ACPI tables (e.g.
> RSDP?).  And then in Linux kernel, we need to parse the DT binding and
> initialize the corresponding variables in kernel, so we need to add
> support in the Linux source drivers/firmware/efi/fdtparams.c.
> 
> Before I proceed, just want to check which option would be the right
> way to move forward?  And I am open for any other solution and welcome
> suggestions.

When Xen is started using EFI, Linux is then started purely using device tree
there is a standard way to define reserved memory to linux using the device
tree and Xen should decode the Memreserve entry from EFI and add the
corresponding entry in the device tree that we give to linux.

Regards
Bertrand

> 
> Thanks a lot!
> Leo

Re: Question: Enable LINUX_EFI_MEMRESERVE_TABLE_GUID in EFI

[Ard Biesheuvel]

On Thu, 11 Aug 2022 at 16:59, Bertrand Marquis <Bertrand.Marquis@arm.com> wrote:
>
> Hi Leon,
>
> > On 11 Aug 2022, at 15:17, Leo Yan <leo.yan@linaro.org> wrote:
> >
> > Hi Bertrand, Rahul,
> >
> > On Fri, Aug 05, 2022 at 12:05:23PM +0000, Bertrand Marquis wrote:
> >>> On 5 Aug 2022, at 12:44, Rahul Singh <Rahul.Singh@arm.com> wrote:
> >
> > [...]
> >
> >>>> Looked into the code, the GICv3 driver tries to create persistent
> >>>> reservations for pending pages, and the persistent reservation table
> >>>> can be used by kexec/kdump.  For the persistent reservations, it
> >>>> relies on MEMRESERVE EFI configuration table, but this table is not
> >>>> supported by xen.efi, I think this is the reason for the above oops.
> >>>
> >>> Yes, you are right above warning is observed because Xen does not support
> >>> memreserve efi table. I also observed a similar warning on the N1SDP board.
> >>>>
> >>>> I checked that if I boot a host Linux (without Xen), then the EFI has
> >>>> provided MEMRESERVE configuration table, I can get below log:
> >>>>
> >>>> #  dmesg | grep MEMRESERVE
> >>>> [    0.000000] efi: TPMFinalLog=0x807f9ef0000 ACPI 2.0=0x807fa0d0018 ... MEMRESERVE=0x807f8141e98
> >>>>
> >>>> Just want to confirm, is anyone working on enabling MEMRESERVE EFI
> >>>> configuration table for Xen?  And welcome any comments and
> >>>> suggestions!
> >>>>
> >>
> >> No I do not think anybody is working on this at the moment.
> >> If you want to work on adding support for this in Xen, we can provide support
> >> and help on reviewing and testing as we have several targets on which we
> >> observe this (N1SDP and Ava).
> >
> > Thanks for your quick response.
> >
> > I took time to look into the code, below are my conclusions.
> >
> > For a normal UEFI boot flow, UEFI will invoke Linux kernel's EFI stub,
> > and the EFI stub will install MEMRESERVE EFI configuration table.
> > This is accomplished in the Linux function install_memreserve_table().
> >
> > Secondly, Xen passes DT to kernel, it synthesizes ACPI compatible
> > nodes in the device tree and finally kernel parses DT and create the
> > ACPI table.  In this case, Xen doesn't invoke Linux EFI stub.
> >
> > To be honest, I have very less knowledge for Xen and APCI; just based on
> > reading code, I think it's hard for Xen to invoke Linux kernel's EFI
> > stub, this is because Xen needs to provide the EFI runtime services, and
> > I don't think it's feasible for Xen to pass through UEFI's runtime
> > service to Linux kernel.  If we implement the EFI runtime services for
> > Xen, then this would introduce a big implementation.
> >
> > So another option is we simply add MEMRESERVE EFI configuration table
> > into device tree, just like Xen does for other ACPI tables (e.g.
> > RSDP?).  And then in Linux kernel, we need to parse the DT binding and
> > initialize the corresponding variables in kernel, so we need to add
> > support in the Linux source drivers/firmware/efi/fdtparams.c.
> >
> > Before I proceed, just want to check which option would be the right
> > way to move forward?  And I am open for any other solution and welcome
> > suggestions.
>
> When Xen is started using EFI, Linux is then started purely using device tree
> there is a standard way to define reserved memory to linux using the device
> tree and Xen should decode the Memreserve entry from EFI and add the
> corresponding entry in the device tree that we give to linux.
>

This is not what MEMRESERVE is used for. Specifying reservations for
the current boot is straight-forward. What MEMRESERVE does is specify
a reservation that survives kexec and is passed on to the next
kernel(s), as the table is anchored in a structure that is created by
the EFI stub on the first boot. This is needed for the GICv3 on some
platforms, as memory that Linux reserves for its interrupt tables can
never be released again, even across kexec, which means that the GICv3
will be DMA'ing into that memory if the kexec kernel wants it or not)

I'd strongly recommend against doing any of the things Xen does for
ACPI boot today: both the ACPI spec and the kernel documentation about
ACPI support in the arm64 port is 100% clear that EFI boot is the only
supported boot method. Issues like this one would have never popped up
if those rules were adhered to. (/pedantic mode off)

In your case, this is a matter of allocating a structure of the right
type and size, and making it available via the configuration table
array in the EFI system table that the dom0 kernel receives from Xen
at boot.

Please don't add DT entries for this - we should be able to cover this
using the existing pseudo-EFI boot flow that Xen uses today.

-- 
Ard.

Re: Question: Enable LINUX_EFI_MEMRESERVE_TABLE_GUID in EFI

[Bertrand Marquis]

Hi,

> On 11 Aug 2022, at 16:05, Ard Biesheuvel <ardb@kernel.org> wrote:
> 
> On Thu, 11 Aug 2022 at 16:59, Bertrand Marquis <Bertrand.Marquis@arm.com> wrote:
>> 
>> Hi Leon,
>> 
>>> On 11 Aug 2022, at 15:17, Leo Yan <leo.yan@linaro.org> wrote:
>>> 
>>> Hi Bertrand, Rahul,
>>> 
>>> On Fri, Aug 05, 2022 at 12:05:23PM +0000, Bertrand Marquis wrote:
>>>>> On 5 Aug 2022, at 12:44, Rahul Singh <Rahul.Singh@arm.com> wrote:
>>> 
>>> [...]
>>> 
>>>>>> Looked into the code, the GICv3 driver tries to create persistent
>>>>>> reservations for pending pages, and the persistent reservation table
>>>>>> can be used by kexec/kdump.  For the persistent reservations, it
>>>>>> relies on MEMRESERVE EFI configuration table, but this table is not
>>>>>> supported by xen.efi, I think this is the reason for the above oops.
>>>>> 
>>>>> Yes, you are right above warning is observed because Xen does not support
>>>>> memreserve efi table. I also observed a similar warning on the N1SDP board.
>>>>>> 
>>>>>> I checked that if I boot a host Linux (without Xen), then the EFI has
>>>>>> provided MEMRESERVE configuration table, I can get below log:
>>>>>> 
>>>>>> #  dmesg | grep MEMRESERVE
>>>>>> [    0.000000] efi: TPMFinalLog=0x807f9ef0000 ACPI 2.0=0x807fa0d0018 ... MEMRESERVE=0x807f8141e98
>>>>>> 
>>>>>> Just want to confirm, is anyone working on enabling MEMRESERVE EFI
>>>>>> configuration table for Xen?  And welcome any comments and
>>>>>> suggestions!
>>>>>> 
>>>> 
>>>> No I do not think anybody is working on this at the moment.
>>>> If you want to work on adding support for this in Xen, we can provide support
>>>> and help on reviewing and testing as we have several targets on which we
>>>> observe this (N1SDP and Ava).
>>> 
>>> Thanks for your quick response.
>>> 
>>> I took time to look into the code, below are my conclusions.
>>> 
>>> For a normal UEFI boot flow, UEFI will invoke Linux kernel's EFI stub,
>>> and the EFI stub will install MEMRESERVE EFI configuration table.
>>> This is accomplished in the Linux function install_memreserve_table().
>>> 
>>> Secondly, Xen passes DT to kernel, it synthesizes ACPI compatible
>>> nodes in the device tree and finally kernel parses DT and create the
>>> ACPI table.  In this case, Xen doesn't invoke Linux EFI stub.
>>> 
>>> To be honest, I have very less knowledge for Xen and APCI; just based on
>>> reading code, I think it's hard for Xen to invoke Linux kernel's EFI
>>> stub, this is because Xen needs to provide the EFI runtime services, and
>>> I don't think it's feasible for Xen to pass through UEFI's runtime
>>> service to Linux kernel.  If we implement the EFI runtime services for
>>> Xen, then this would introduce a big implementation.
>>> 
>>> So another option is we simply add MEMRESERVE EFI configuration table
>>> into device tree, just like Xen does for other ACPI tables (e.g.
>>> RSDP?).  And then in Linux kernel, we need to parse the DT binding and
>>> initialize the corresponding variables in kernel, so we need to add
>>> support in the Linux source drivers/firmware/efi/fdtparams.c.
>>> 
>>> Before I proceed, just want to check which option would be the right
>>> way to move forward?  And I am open for any other solution and welcome
>>> suggestions.
>> 
>> When Xen is started using EFI, Linux is then started purely using device tree
>> there is a standard way to define reserved memory to linux using the device
>> tree and Xen should decode the Memreserve entry from EFI and add the
>> corresponding entry in the device tree that we give to linux.
>> 
> 
> This is not what MEMRESERVE is used for. Specifying reservations for
> the current boot is straight-forward. What MEMRESERVE does is specify
> a reservation that survives kexec and is passed on to the next
> kernel(s), as the table is anchored in a structure that is created by
> the EFI stub on the first boot. This is needed for the GICv3 on some
> platforms, as memory that Linux reserves for its interrupt tables can
> never be released again, even across kexec, which means that the GICv3
> will be DMA'ing into that memory if the kexec kernel wants it or not)
> 
> I'd strongly recommend against doing any of the things Xen does for
> ACPI boot today: both the ACPI spec and the kernel documentation about
> ACPI support in the arm64 port is 100% clear that EFI boot is the only
> supported boot method. Issues like this one would have never popped up
> if those rules were adhered to. (/pedantic mode off)

I agree with that in the long term we should find a solution to remove this
system and have something more compliant with EFI/ACPI in Xen.

> 
> In your case, this is a matter of allocating a structure of the right
> type and size, and making it available via the configuration table
> array in the EFI system table that the dom0 kernel receives from Xen
> at boot.
> 
> Please don't add DT entries for this - we should be able to cover this
> using the existing pseudo-EFI boot flow that Xen uses today.

Currently the EFI system table is passed using a device tree generated
by Xen. To add support right now we would need to make the table
available to dom0 and pass its address inside this device tree.

Cheers
Bertrand

> 
> -- 
> Ard.