Design doc of adding ACPI support for arm64 on Xen - version 4

Design doc of adding ACPI support for arm64 on Xen - version 4

[Shannon Zhao]

This document is going to explain the design details of Xen booting with
ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are
welcome.

Changes v3->v4:
* add explanation for minimal DT and the properties
* drop "linux," prefix of the properties
* add explanation for the event channel flag
* create RSDP table since the "xsdt_physical_address" is changed
* since it uses hypervisor_id introduced by ACPI 6.0 to notify Dom0 the
  hypervisor ID, so it needs to limit minimum supported ACPI version for
  Xen on ARM to 6.0.

Changes v2->v3:
* remove the two HVM_PARAMs for grant table and let linux kernel use
  xlated_setup_gnttab_pages() to setup grant table.
* don't modify GTDT table
* add definition of event-channel interrupt flag
* state that route all Xen unused interrupt to Dom0
* state that reusing existing PCI bus_notifier for PCI devices MMIO
* mapping

To Xen itself booting with ACPI, this is similar to Linux kernel except
that Xen doesn't parse DSDT table. So I'll skip this part and focus on
how Xen prepares ACPI tables for Dom0 and how Xen passes them to Dom0.

1. Create minimal DT to pass required information to Dom0
----------------------------------------------------------
Since there are no legacy interfaces like x86 for Dom0 to get the
booting required information on ARM, here we use the minimal DT which is
used by UEFI stub communicating with Linux kernel as well.

The UEFI stub is a feature that extends the Image/zImage into a valid
UEFI PE/COFF executable, including a loader application that makes it
possible to load the kernel directly from the UEFI shell, boot menu, or
one of the lightweight bootloaders like Gummiboot or rEFInd.

The kernel image built with stub support remains a valid kernel image
for booting in non-UEFI environments and the UEFI stub will be jumped
for non-UEFI environments.

When booting in UEFI mode, the UEFI stub will create a minimal DT in
order to pass the command line and other informations (such as the EFI
memory table) to the kernel. And when booting with ACPI, kernel will get
command line, ACPI root table address and memory map information from
the minimal DT. Also, it will check if the DT contains only the /chosen
node to know whether it boots with DT or ACPI.

In addition, the current names of these properties with a "linux,"
prefix in the minimal DT are Linux specified. It needs to standardize
them so that other OS(such as FreeBSD) could reuse them in the future.
So we drop the "linux," prefix of UEFI parameters and change the names
in Linux kernel as well.

An example of the minimal DT:
/ {
    #address-cells = <2>;
    #size-cells = <1>;
    chosen {
        bootargs = "kernel=Image console=hvc0 earlycon=pl011,0x1c090000
root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force efi=noruntime";
        linux,initrd-start = <0xXXXXXXXX>;
        linux,initrd-end = <0xXXXXXXXX>;
        uefi-system-table = <0xXXXXXXXX>;
        uefi-mmap-start = <0xXXXXXXXX>;
        uefi-mmap-size = <0xXXXXXXXX>;
        uefi-mmap-desc-size = <0xXXXXXXXX>;
        uefi-mmap-desc-ver = <0xXXXXXXXX>;
    };
};

For details loook at [1].

2. Copy and change some EFI and ACPI tables
-------------------------------------------
a) Create EFI_SYSTEM_TABLE table
Copy the header from the origin and change the value of FirmwareVendor.
Create only one ConfigurationTable to store VendorGuid and VendorTable.
The EFI System Table will be passed to Dom0 through the property
"uefi-system-table" in the above minimal DT. So Dom0 could get ACPI root
table address through the ConfigurationTable.

b) Create EFI_MEMORY_DESCRIPTOR table
It needs to notify Dom0 where are the RAM regions. Add memory start and
size information of Dom0 in this table. It's passed to Dom0 through the
properties "uefi-mmap-start", "uefi-mmap-size", "uefi-mmap-desc-size"
and "uefi-mmap-desc-ver" of the minimal DT. Then Dom0 will get the
memory information through this EFI table.

c) Copy FADT table
Change the value of arm_boot_flags to enable PSCI and HVC.

Since there is no cpuid like x86 on ARM, here it's through the field
hypervisor_id of FADT table to tell Dom0 that the hypervisor is Xen.
Because hypervisor_id is introduced by ACPI 6.0, so it needs to limit
minimum supported ACPI version for Xen on ARM to 6.0.

Set the hypervisor_id to "XenVMM", then Dom0 could through HVM_PARAM to
get some informations for booting necessity, such as event-channel
interrupt.

d) Copy MADT table
It needs to change MADT table to restrict the number of vCPUs. We choose
to copy the first dom0_max_vcpus GICC entries of MADT to new created
MADT table as numa is not supported currently. If numa is supported in
the future, it would change the mpidr according to the cpu topology.
Also copy GICD as well.

e) Create STAO table
This table is a new added one that's used to define a list of ACPI
namespace names that are to be ignored by the OSPM in Dom0. Currently
we use it to tell OSPM that it should ignore UART defined in SPCR table.
Look at [2] for details.

f) Copy XSDT table
Add a new table entry for STAO and change other table's entries.

g) Copy RSDP table
Change the value of xsdt_physical_address in RSDP table. As we create a
new XSDT table and the address of XSDT is changed, so it needs to update
the value of "xsdt_physical_address" in RSDP. So Dom0 could get the
right XSDT table rather than the old one. And it needs to update the
value of VendorTable in EFI Configuration Table which is created in
above step a).

h) The rest of tables are not copied or changed. They are reused
including DSDT, SPCR, GTDT, etc. It doesn't mask EL2 resources for Dom0
because the Linux kernel would not use EL2 resources when it boots at
EL1.

All these tables will be copied to Dom0 memory except that the reused
tables(DSDT, SPCR, GTDT, etc) will be mapped to Dom0.

3. Dom0 gets grant table and event channel irq information
-----------------------------------------------------------
The OS will have to find a place himself in the memory map for the grant
table region. For instance, Linux can make usage of
xlated_setup_gnttab_pages.

To event channel interrupt, reuse HVM_PARAM_CALLBACK_IRQ and add a new
delivery type to get it.
val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64
only)
The definition of flag reusing the definition of xenv table[3]. Bit 0
stands interrupt mode(1: edge triggered, 0: level triggered) and bit 1
stands interrupt polarity(1: active low, 0: active high).

As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that
it runs on Xen hypervisor. Then Dom0 could get it through hypercall
HVMOP_get_param.

4. Map MMIO regions
-------------------
Register a bus_notifier for platform and amba bus in Linux. Add a new
XENMAPSPACE "XENMAPSPACE_dev_mmio". The usage of this hypercall
parameters:
- domid: DOMID_SELF.
- space: XENMAPSPACE_dev_mmio.
- gpfns: guest physical addresses where the mapping should appear.

Within the register, check if the device is newly added, then call
hypercall XENMEM_add_to_physmap to map the mmio regions.

For PCI bus device, it could reuse the existing PCI bus_notifier like
X86.

5. Route device interrupts to Dom0
----------------------------------
Route all the SPI interrupts to Dom0 before Dom0 booting, except the
interrupts used by Xen. For uart, it could get the interrupt from SPCR
table and exclude it. For SMMU, it could get the interrupts from IORT
table and exclude them as well.

[1] https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi.txt
[2]
http://wiki.xenproject.org/mediawiki/images/0/02/Status-override-table.pdf
[3]
http://wiki.xenproject.org/mediawiki/images/c/c4/Xen-environment-table.pdf

Thanks,
-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Jan Beulich]

>>> On 19.08.15 at 14:13, <zhaoshenglong@huawei.com> wrote:
> 1. Create minimal DT to pass required information to Dom0
> ----------------------------------------------------------
> Since there are no legacy interfaces like x86 for Dom0 to get the
> booting required information on ARM, here we use the minimal DT which is
> used by UEFI stub communicating with Linux kernel as well.
> 
> The UEFI stub is a feature that extends the Image/zImage into a valid
> UEFI PE/COFF executable, including a loader application that makes it
> possible to load the kernel directly from the UEFI shell, boot menu, or
> one of the lightweight bootloaders like Gummiboot or rEFInd.
> 
> The kernel image built with stub support remains a valid kernel image
> for booting in non-UEFI environments and the UEFI stub will be jumped
> for non-UEFI environments.

Isn't this backwards? I.e. ...

> When booting in UEFI mode, the UEFI stub will create a minimal DT in
> order to pass the command line and other informations (such as the EFI
> memory table) to the kernel. And when booting with ACPI, kernel will get
> command line, ACPI root table address and memory map information from
> the minimal DT. Also, it will check if the DT contains only the /chosen
> node to know whether it boots with DT or ACPI.
> 
> In addition, the current names of these properties with a "linux,"
> prefix in the minimal DT are Linux specified. It needs to standardize
> them so that other OS(such as FreeBSD) could reuse them in the future.
> So we drop the "linux," prefix of UEFI parameters and change the names
> in Linux kernel as well.

... wouldn't it make more sense to leave the generation of these
Linux-specific tags to Linux (and allow them to continue to be Linux
specific), by the same or a second, parallel (Xen) stub? This would
then also move at least some of the awkward table creation (and
ideally also copying) to that stub.

> All these tables will be copied to Dom0 memory except that the reused
> tables(DSDT, SPCR, GTDT, etc) will be mapped to Dom0.

I continue to be puzzled by this - all of the tables should be in identical
kinds of memory, no matter whether they get mapped or copied. Yet
copying to Dom0 memory to me implies memory owned by Dom0 (and
how the kernel knows to not use it as "normal" RAM is left unsaid),
while mapping to me implies memory _not_ owned by Dom0 (much
like x86's E820 reserved type or UEFI's EfiReservedMemoryType).

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Roger Pau Monné]

El 19/08/15 a les 14.13, Shannon Zhao ha escrit:
> 4. Map MMIO regions
> -------------------
> Register a bus_notifier for platform and amba bus in Linux. Add a new

Can we make this OS agnostic? Could you explain what you mean with the
above sentence without using Linux kernel internals?

> XENMAPSPACE "XENMAPSPACE_dev_mmio". The usage of this hypercall
> parameters:
> - domid: DOMID_SELF.
> - space: XENMAPSPACE_dev_mmio.
> - gpfns: guest physical addresses where the mapping should appear.

This is not complete, you have forgotten to add the idxs field, and I
don't think the domid field needs an explanation TBH.

>
> Within the register, check if the device is newly added, then call
> hypercall XENMEM_add_to_physmap to map the mmio regions.
>
> For PCI bus device, it could reuse the existing PCI bus_notifier like
> X86.

What is a PCI bus_notifier? I don't see any mention of bus_notifier in
Xen code.

In general the text above seems more like a description of the Linux
implementation rather than a Xen design document. Could you explain
without using Linux internals how/when this new hypercall should be used?

If you want you can provide examples about how this new interface is
going to be used in Linux, but the general description should remain OS
agnostic IMHO.

Roger.

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Julien Grall]

Hi Jan,

On 19/08/2015 07:05, Jan Beulich wrote:
> ... wouldn't it make more sense to leave the generation of these
> Linux-specific tags to Linux (and allow them to continue to be Linux
> specific), by the same or a second, parallel (Xen) stub? This would
> then also move at least some of the awkward table creation (and
> ideally also copying) to that stub.

We don't want to use a specific Xen entry path. So having a Xen stub is 
has to be rule out.

Anyway as said multiple time, there is no issue to drop the "linux," in 
the properties. They are only used internally between the stub and the 
kernel (both are built in the same image).

It would be pointless to create a stub just for renaming the properties...

Regards,

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Shannon Zhao]

Hi Roger,

On 2015/8/19 23:02, Roger Pau Monné wrote:
> El 19/08/15 a les 14.13, Shannon Zhao ha escrit:
>> 4. Map MMIO regions
>> -------------------
>> Register a bus_notifier for platform and amba bus in Linux. Add a new
> 
> Can we make this OS agnostic? Could you explain what you mean with the
> above sentence without using Linux kernel internals?
> 
Ok, will change it.

>> XENMAPSPACE "XENMAPSPACE_dev_mmio". The usage of this hypercall
>> parameters:
>> - domid: DOMID_SELF.
>> - space: XENMAPSPACE_dev_mmio.
>> - gpfns: guest physical addresses where the mapping should appear.
> 
> This is not complete, you have forgotten to add the idxs field, 

Sorry, I didn't use the idx for the mmio region mapping. What's the idx
useful for here?

> and I
> don't think the domid field needs an explanation TBH.
> 
>>
>> Within the register, check if the device is newly added, then call
>> hypercall XENMEM_add_to_physmap to map the mmio regions.
>>
>> For PCI bus device, it could reuse the existing PCI bus_notifier like
>> X86.
> 
> What is a PCI bus_notifier? I don't see any mention of bus_notifier in
> Xen code.
> 
I thought You konw what I mean here. If not, what should I do! Yeah, as
you said, I will change the description.

But I want to say that the purpose of this document is to explain some
important implementation details for ACPI support on ARM, which have
some divergence before. I don't hope this doc to be official but I do
hope this lets us reach an agreement on the approach before respinning
the patches. If we focus on the words or description or even
word-splitting, it doesn't help to make this move on!

> In general the text above seems more like a description of the Linux
> implementation rather than a Xen design document. Could you explain
> without using Linux internals how/when this new hypercall should be used?
> 
> If you want you can provide examples about how this new interface is
> going to be used in Linux, but the general description should remain OS
> agnostic IMHO.
> 
> Roger.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 
> .
> 

-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Shannon Zhao]

Hi Jan,

On 2015/8/19 22:05, Jan Beulich wrote:
>>>> On 19.08.15 at 14:13, <zhaoshenglong@huawei.com> wrote:
>> 1. Create minimal DT to pass required information to Dom0
>> ----------------------------------------------------------
>> Since there are no legacy interfaces like x86 for Dom0 to get the
>> booting required information on ARM, here we use the minimal DT which is
>> used by UEFI stub communicating with Linux kernel as well.
>>
>> The UEFI stub is a feature that extends the Image/zImage into a valid
>> UEFI PE/COFF executable, including a loader application that makes it
>> possible to load the kernel directly from the UEFI shell, boot menu, or
>> one of the lightweight bootloaders like Gummiboot or rEFInd.
>>
>> The kernel image built with stub support remains a valid kernel image
>> for booting in non-UEFI environments and the UEFI stub will be jumped
>> for non-UEFI environments.
> 
> Isn't this backwards? I.e. ...
> 
Sorry, what do you mean here?
It's used for backwards compatibility. The reason I mention it here is
to explain that Xen acts as a EFI stub which creates a minimal DT and
load kernel image. Even though the kernel image built with EFI stub
which will be jumped, the kernel could boot well as before.

>> When booting in UEFI mode, the UEFI stub will create a minimal DT in
>> order to pass the command line and other informations (such as the EFI
>> memory table) to the kernel. And when booting with ACPI, kernel will get
>> command line, ACPI root table address and memory map information from
>> the minimal DT. Also, it will check if the DT contains only the /chosen
>> node to know whether it boots with DT or ACPI.
>>
>> In addition, the current names of these properties with a "linux,"
>> prefix in the minimal DT are Linux specified. It needs to standardize
>> them so that other OS(such as FreeBSD) could reuse them in the future.
>> So we drop the "linux," prefix of UEFI parameters and change the names
>> in Linux kernel as well.
> 
> ... wouldn't it make more sense to leave the generation of these
> Linux-specific tags to Linux (and allow them to continue to be Linux
> specific), by the same or a second, parallel (Xen) stub? This would
> then also move at least some of the awkward table creation (and
> ideally also copying) to that stub.
> 
>> All these tables will be copied to Dom0 memory except that the reused
>> tables(DSDT, SPCR, GTDT, etc) will be mapped to Dom0.
> 
> I continue to be puzzled by this - all of the tables should be in identical
> kinds of memory, no matter whether they get mapped or copied. Yet
> copying to Dom0 memory to me implies memory owned by Dom0 (and
> how the kernel knows to not use it as "normal" RAM is left unsaid),
> while mapping to me implies memory _not_ owned by Dom0 (much
> like x86's E820 reserved type or UEFI's EfiReservedMemoryType).
> 

These will be described in EFI_MEMORY_DESCRIPTOR table. For Dom0 memory
range, the Type is EfiConventionalMemory. For ACPI tables whether it's
new created or reused, the Type is EfiACPIReclaimMemory. For EFI System
table and Memory Descriptor table, the Type is EfiReservedMemoryType.

-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Julien Grall]

On 19/08/2015 20:07, Shannon Zhao wrote:
>> and I
>> don't think the domid field needs an explanation TBH.
>>
>>>
>>> Within the register, check if the device is newly added, then call
>>> hypercall XENMEM_add_to_physmap to map the mmio regions.
>>>
>>> For PCI bus device, it could reuse the existing PCI bus_notifier like
>>> X86.
>>
>> What is a PCI bus_notifier? I don't see any mention of bus_notifier in
>> Xen code.
>>
> I thought You konw what I mean here. If not, what should I do! Yeah, as
> you said, I will change the description.

Please remember that not everyone on the mailing list knows how work the 
internal of linux is working. The goal of the design doc is to explain 
to them in simple way what we are planning to do.

>
> But I want to say that the purpose of this document is to explain some
> important implementation details for ACPI support on ARM, which have
> some divergence before. I don't hope this doc to be official but I do
> hope this lets us reach an agreement on the approach before respinning
> the patches. If we focus on the words or description or even
> word-splitting, it doesn't help to make this move on!

I think you misunderstood what is the design doc. It's an high level
description about what will be implemented and official.

If people doesn't understand what you meant in the design doc, I don't 
see how they will be able to check that what you've implemented will be 
match what we agree. So the wording, description ... is really important.

Regards,

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Roger Pau Monné]

El 20/08/15 a les 5.07, Shannon Zhao ha escrit:
> On 2015/8/19 23:02, Roger Pau Monné wrote:
>> El 19/08/15 a les 14.13, Shannon Zhao ha escrit:
>>> XENMAPSPACE "XENMAPSPACE_dev_mmio". The usage of this hypercall
>>> parameters:
>>> - domid: DOMID_SELF.
>>> - space: XENMAPSPACE_dev_mmio.
>>> - gpfns: guest physical addresses where the mapping should appear.
>>
>> This is not complete, you have forgotten to add the idxs field, 
> 
> Sorry, I didn't use the idx for the mmio region mapping. What's the idx
> useful for here?

I've already posted this in the previous version, and you agreed on the
interface and the usage of the fields, please see:

http://marc.info/?l=xen-devel&m=143986236212359

The idxs field is explicitly mentioned there with it's usage.

Roger.

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Jan Beulich]

>>> On 19.08.15 at 20:37, <julien.grall@citrix.com> wrote:
> On 19/08/2015 07:05, Jan Beulich wrote:
>> ... wouldn't it make more sense to leave the generation of these
>> Linux-specific tags to Linux (and allow them to continue to be Linux
>> specific), by the same or a second, parallel (Xen) stub? This would
>> then also move at least some of the awkward table creation (and
>> ideally also copying) to that stub.
> 
> We don't want to use a specific Xen entry path. So having a Xen stub is 
> has to be rule out.
> 
> Anyway as said multiple time, there is no issue to drop the "linux," in 
> the properties. They are only used internally between the stub and the 
> kernel (both are built in the same image).
> 
> It would be pointless to create a stub just for renaming the properties...

Agreed. As long as it's not Xen to create these tags, I guess I'm fine
with that particular aspect.

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Jan Beulich]

>>> On 20.08.15 at 05:41, <zhaoshenglong@huawei.com> wrote:
> On 2015/8/19 22:05, Jan Beulich wrote:
>>>>> On 19.08.15 at 14:13, <zhaoshenglong@huawei.com> wrote:
>>> 1. Create minimal DT to pass required information to Dom0
>>> ----------------------------------------------------------
>>> Since there are no legacy interfaces like x86 for Dom0 to get the
>>> booting required information on ARM, here we use the minimal DT which is
>>> used by UEFI stub communicating with Linux kernel as well.
>>>
>>> The UEFI stub is a feature that extends the Image/zImage into a valid
>>> UEFI PE/COFF executable, including a loader application that makes it
>>> possible to load the kernel directly from the UEFI shell, boot menu, or
>>> one of the lightweight bootloaders like Gummiboot or rEFInd.
>>>
>>> The kernel image built with stub support remains a valid kernel image
>>> for booting in non-UEFI environments and the UEFI stub will be jumped
>>> for non-UEFI environments.
>> 
>> Isn't this backwards? I.e. ...
>> 
> Sorry, what do you mean here?
> It's used for backwards compatibility. The reason I mention it here is
> to explain that Xen acts as a EFI stub which creates a minimal DT and
> load kernel image. Even though the kernel image built with EFI stub
> which will be jumped, the kernel could boot well as before.

To me what you and Julien replied contradicts one another: You say
it's Xen to create these DT tags, whereas he assures me that these
tags are Linux internal ones.

As to the "backwards" - maybe by "jumped" you mean "jumped over"
when I understand it to mean "jumped to"?

>>> When booting in UEFI mode, the UEFI stub will create a minimal DT in
>>> order to pass the command line and other informations (such as the EFI
>>> memory table) to the kernel. And when booting with ACPI, kernel will get
>>> command line, ACPI root table address and memory map information from
>>> the minimal DT. Also, it will check if the DT contains only the /chosen
>>> node to know whether it boots with DT or ACPI.
>>>
>>> In addition, the current names of these properties with a "linux,"
>>> prefix in the minimal DT are Linux specified. It needs to standardize
>>> them so that other OS(such as FreeBSD) could reuse them in the future.
>>> So we drop the "linux," prefix of UEFI parameters and change the names
>>> in Linux kernel as well.
>> 
>> ... wouldn't it make more sense to leave the generation of these
>> Linux-specific tags to Linux (and allow them to continue to be Linux
>> specific), by the same or a second, parallel (Xen) stub? This would
>> then also move at least some of the awkward table creation (and
>> ideally also copying) to that stub.
>> 
>>> All these tables will be copied to Dom0 memory except that the reused
>>> tables(DSDT, SPCR, GTDT, etc) will be mapped to Dom0.
>> 
>> I continue to be puzzled by this - all of the tables should be in identical
>> kinds of memory, no matter whether they get mapped or copied. Yet
>> copying to Dom0 memory to me implies memory owned by Dom0 (and
>> how the kernel knows to not use it as "normal" RAM is left unsaid),
>> while mapping to me implies memory _not_ owned by Dom0 (much
>> like x86's E820 reserved type or UEFI's EfiReservedMemoryType).
>> 
> 
> These will be described in EFI_MEMORY_DESCRIPTOR table. For Dom0 memory
> range, the Type is EfiConventionalMemory. For ACPI tables whether it's
> new created or reused, the Type is EfiACPIReclaimMemory. For EFI System
> table and Memory Descriptor table, the Type is EfiReservedMemoryType.

And Dom0 will have to figure out how to establish mappings for the
two different kinds (which ought to work differently since in one case
Dom0-owned memory is to be mapped, while in the other case it's
Xen- [or even firmware-]owned memory)? Anyway - even without
that aspect, it seems conceptually wrong to place firmware tables
(even if they are fake ones) in Dom0-owned memory.

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Shannon Zhao]

Hi Roger,

On 2015/8/20 16:20, Roger Pau Monné wrote:
> El 20/08/15 a les 5.07, Shannon Zhao ha escrit:
>> On 2015/8/19 23:02, Roger Pau Monné wrote:
>>> El 19/08/15 a les 14.13, Shannon Zhao ha escrit:
>>>> XENMAPSPACE "XENMAPSPACE_dev_mmio". The usage of this hypercall
>>>> parameters:
>>>> - domid: DOMID_SELF.
>>>> - space: XENMAPSPACE_dev_mmio.
>>>> - gpfns: guest physical addresses where the mapping should appear.
>>>
>>> This is not complete, you have forgotten to add the idxs field,
>>
>> Sorry, I didn't use the idx for the mmio region mapping. What's the idx
>> useful for here?
>
> I've already posted this in the previous version, and you agreed on the
> interface and the usage of the fields, please see:
>
> http://marc.info/?l=xen-devel&m=143986236212359
>
> The idxs field is explicitly mentioned there with it's usage.
>

Yeah, I said I will add the description of hypercall parameters.
It seems that we are talking about a different parameter.
To map the mmio region, I reuse the struct xen_add_to_physmap and there 
is idx not idxs. Everytime Dom0 maps one page and it's mapped 1:1(guest 
physical address is same with real physical hardware address), so it 
only needs to tell the hypervisor the gpfn.

Since I'm not familiar with Xen general approach, if this is not right, 
please fix me.

Thanks,
-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Roger Pau Monné]

El 20/08/15 a les 13.22, Shannon Zhao ha escrit:
> Hi Roger,
> 
> On 2015/8/20 16:20, Roger Pau Monné wrote:
>> El 20/08/15 a les 5.07, Shannon Zhao ha escrit:
>>> On 2015/8/19 23:02, Roger Pau Monné wrote:
>>>> El 19/08/15 a les 14.13, Shannon Zhao ha escrit:
>>>>> XENMAPSPACE "XENMAPSPACE_dev_mmio". The usage of this hypercall
>>>>> parameters:
>>>>> - domid: DOMID_SELF.
>>>>> - space: XENMAPSPACE_dev_mmio.
>>>>> - gpfns: guest physical addresses where the mapping should appear.
>>>>
>>>> This is not complete, you have forgotten to add the idxs field,
>>>
>>> Sorry, I didn't use the idx for the mmio region mapping. What's the idx
>>> useful for here?
>>
>> I've already posted this in the previous version, and you agreed on the
>> interface and the usage of the fields, please see:
>>
>> http://marc.info/?l=xen-devel&m=143986236212359
>>
>> The idxs field is explicitly mentioned there with it's usage.
>>
> 
> Yeah, I said I will add the description of hypercall parameters.
> It seems that we are talking about a different parameter.
> To map the mmio region, I reuse the struct xen_add_to_physmap and there

You should also take into account xen_add_to_physmap_batch (or are you
planning to issue an hypercall for every single MMIO page that you want
to map?), but anyway the idx(s) field is there in both structs.

> is idx not idxs. Everytime Dom0 maps one page and it's mapped 1:1(guest
> physical address is same with real physical hardware address), so it
> only needs to tell the hypervisor the gpfn.

IMHO, I'm not sure why we should restrict this to 1:1 (although I admit
this is going to be the common case). Didn't we are that we are going to
allow non 1:1 mapping of MMIO regions?

If you want you can check in the hypercall handler that idxs[i] ==
gpfns[i], and return -EOPNOTSUPP if they don't match, but I still don't
see why this should be restricted to 1:1 mappings.

Roger.

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Jan Beulich]

>>> On 20.08.15 at 13:28, <roger.pau@citrix.com> wrote:
> If you want you can check in the hypercall handler that idxs[i] ==
> gpfns[i], and return -EOPNOTSUPP if they don't match, but I still don't
> see why this should be restricted to 1:1 mappings.

+1

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Shannon Zhao]

On 2015/8/20 19:28, Roger Pau Monné wrote:
> El 20/08/15 a les 13.22, Shannon Zhao ha escrit:
>> Hi Roger,
>>
>> On 2015/8/20 16:20, Roger Pau Monné wrote:
>>> El 20/08/15 a les 5.07, Shannon Zhao ha escrit:
>>>> On 2015/8/19 23:02, Roger Pau Monné wrote:
>>>>> El 19/08/15 a les 14.13, Shannon Zhao ha escrit:
>>>>>> XENMAPSPACE "XENMAPSPACE_dev_mmio". The usage of this hypercall
>>>>>> parameters:
>>>>>> - domid: DOMID_SELF.
>>>>>> - space: XENMAPSPACE_dev_mmio.
>>>>>> - gpfns: guest physical addresses where the mapping should appear.
>>>>>
>>>>> This is not complete, you have forgotten to add the idxs field,
>>>>
>>>> Sorry, I didn't use the idx for the mmio region mapping. What's the idx
>>>> useful for here?
>>>
>>> I've already posted this in the previous version, and you agreed on the
>>> interface and the usage of the fields, please see:
>>>
>>> http://marc.info/?l=xen-devel&m=143986236212359
>>>
>>> The idxs field is explicitly mentioned there with it's usage.
>>>
>>
>> Yeah, I said I will add the description of hypercall parameters.
>> It seems that we are talking about a different parameter.
>> To map the mmio region, I reuse the struct xen_add_to_physmap and there
>
> You should also take into account xen_add_to_physmap_batch (or are you
> planning to issue an hypercall for every single MMIO page that you want
> to map?), but anyway the idx(s) field is there in both structs.
>

Yeah, current approach is to issue an hypercall for every single MMIO 
page. But if we want to batch map MMIO pages, I think it needs the size 
parameter and what's idxs useful for? As we map the MMIO pages 1:1, it 
seems it's unnecessary to check "idxs[i] == gpfns[i]", right?

>> is idx not idxs. Everytime Dom0 maps one page and it's mapped 1:1(guest
>> physical address is same with real physical hardware address), so it
>> only needs to tell the hypervisor the gpfn.
>
> IMHO, I'm not sure why we should restrict this to 1:1 (although I admit
> this is going to be the common case). Didn't we are that we are going to
> allow non 1:1 mapping of MMIO regions?
>
> If you want you can check in the hypercall handler that idxs[i] ==
> gpfns[i], and return -EOPNOTSUPP if they don't match, but I still don't
> see why this should be restricted to 1:1 mappings.
>

For Dom0 which get the device MMIO information from the DT or ACPI DSDT 
table. To ACPI, we don't (or can't)modify anything in DSDT table. So 
actually the MMIO regions Dom0 gets are the real physical hardware MMIO 
regions and the start address and size of them are same.

-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Shannon Zhao]

On 2015/8/20 17:30, Jan Beulich wrote:
>>>> On 20.08.15 at 05:41, <zhaoshenglong@huawei.com> wrote:
>> On 2015/8/19 22:05, Jan Beulich wrote:
>>>>>> On 19.08.15 at 14:13, <zhaoshenglong@huawei.com> wrote:
>>>> 1. Create minimal DT to pass required information to Dom0
>>>> ----------------------------------------------------------
>>>> Since there are no legacy interfaces like x86 for Dom0 to get the
>>>> booting required information on ARM, here we use the minimal DT which is
>>>> used by UEFI stub communicating with Linux kernel as well.
>>>>
>>>> The UEFI stub is a feature that extends the Image/zImage into a valid
>>>> UEFI PE/COFF executable, including a loader application that makes it
>>>> possible to load the kernel directly from the UEFI shell, boot menu, or
>>>> one of the lightweight bootloaders like Gummiboot or rEFInd.
>>>>
>>>> The kernel image built with stub support remains a valid kernel image
>>>> for booting in non-UEFI environments and the UEFI stub will be jumped
>>>> for non-UEFI environments.
>>>
>>> Isn't this backwards? I.e. ...
>>>
>> Sorry, what do you mean here?
>> It's used for backwards compatibility. The reason I mention it here is
>> to explain that Xen acts as a EFI stub which creates a minimal DT and
>> load kernel image. Even though the kernel image built with EFI stub
>> which will be jumped, the kernel could boot well as before.
>
> To me what you and Julien replied contradicts one another: You say
> it's Xen to create these DT tags, whereas he assures me that these
> tags are Linux internal ones.
>

I think What Julien said is to explain that the DT properties are used 
between EFI stub and kernel image which are built together. So if we 
change the names in Linux, it won't break backward compatibility.

I'll explain What I mean here. If not right, please fix me.
Firstly, Xen doesn't offer a (complete) firmware for Dom0, so it can't 
load EFI stub, then EFI stub loads the kernel image. Therefore, there 
are no chance to create the DT which we want to use it to pass the uefi 
information.
Secondly, we make Xen act like a EFI stub to create the DT which could 
pass the information to Dom0. When we load kernel image, the kernel will 
call efi_get_fdt_params to get the information from the DT just like the 
DT is created by EFI stub for kernel.

> As to the "backwards" - maybe by "jumped" you mean "jumped over"
> when I understand it to mean "jumped to"?
>
yes, jumped over.

>>>> When booting in UEFI mode, the UEFI stub will create a minimal DT in
>>>> order to pass the command line and other informations (such as the EFI
>>>> memory table) to the kernel. And when booting with ACPI, kernel will get
>>>> command line, ACPI root table address and memory map information from
>>>> the minimal DT. Also, it will check if the DT contains only the /chosen
>>>> node to know whether it boots with DT or ACPI.
>>>>
>>>> In addition, the current names of these properties with a "linux,"
>>>> prefix in the minimal DT are Linux specified. It needs to standardize
>>>> them so that other OS(such as FreeBSD) could reuse them in the future.
>>>> So we drop the "linux," prefix of UEFI parameters and change the names
>>>> in Linux kernel as well.
>>>
>>> ... wouldn't it make more sense to leave the generation of these
>>> Linux-specific tags to Linux (and allow them to continue to be Linux
>>> specific), by the same or a second, parallel (Xen) stub? This would
>>> then also move at least some of the awkward table creation (and
>>> ideally also copying) to that stub.
>>>
>>>> All these tables will be copied to Dom0 memory except that the reused
>>>> tables(DSDT, SPCR, GTDT, etc) will be mapped to Dom0.
>>>
>>> I continue to be puzzled by this - all of the tables should be in identical
>>> kinds of memory, no matter whether they get mapped or copied. Yet
>>> copying to Dom0 memory to me implies memory owned by Dom0 (and
>>> how the kernel knows to not use it as "normal" RAM is left unsaid),
>>> while mapping to me implies memory _not_ owned by Dom0 (much
>>> like x86's E820 reserved type or UEFI's EfiReservedMemoryType).
>>>
>>
>> These will be described in EFI_MEMORY_DESCRIPTOR table. For Dom0 memory
>> range, the Type is EfiConventionalMemory. For ACPI tables whether it's
>> new created or reused, the Type is EfiACPIReclaimMemory. For EFI System
>> table and Memory Descriptor table, the Type is EfiReservedMemoryType.
>
> And Dom0 will have to figure out how to establish mappings for the
> two different kinds (which ought to work differently since in one case
> Dom0-owned memory is to be mapped, while in the other case it's
> Xen- [or even firmware-]owned memory)? Anyway - even without
> that aspect, it seems conceptually wrong to place firmware tables
> (even if they are fake ones) in Dom0-owned memory.
>
Here we do not want to supply Dom0 with a firmware. We just want to take 
advantage of the existing way to pass the information to Dom0.

If this not right, do you have any other good ideas for this? Please 
share your suggestion with me.

Thanks,
-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Roger Pau Monné]

El 20/08/15 a les 14.29, Shannon Zhao ha escrit:
> 
> 
> On 2015/8/20 19:28, Roger Pau Monné wrote:
>> El 20/08/15 a les 13.22, Shannon Zhao ha escrit:
>>> Hi Roger,
>>>
>>> On 2015/8/20 16:20, Roger Pau Monné wrote:
>>>> El 20/08/15 a les 5.07, Shannon Zhao ha escrit:
>>>>> On 2015/8/19 23:02, Roger Pau Monné wrote:
>>>>>> El 19/08/15 a les 14.13, Shannon Zhao ha escrit:
>>>>>>> XENMAPSPACE "XENMAPSPACE_dev_mmio". The usage of this hypercall
>>>>>>> parameters:
>>>>>>> - domid: DOMID_SELF.
>>>>>>> - space: XENMAPSPACE_dev_mmio.
>>>>>>> - gpfns: guest physical addresses where the mapping should appear.
>>>>>>
>>>>>> This is not complete, you have forgotten to add the idxs field,
>>>>>
>>>>> Sorry, I didn't use the idx for the mmio region mapping. What's the
>>>>> idx
>>>>> useful for here?
>>>>
>>>> I've already posted this in the previous version, and you agreed on the
>>>> interface and the usage of the fields, please see:
>>>>
>>>> http://marc.info/?l=xen-devel&m=143986236212359
>>>>
>>>> The idxs field is explicitly mentioned there with it's usage.
>>>>
>>>
>>> Yeah, I said I will add the description of hypercall parameters.
>>> It seems that we are talking about a different parameter.
>>> To map the mmio region, I reuse the struct xen_add_to_physmap and there
>>
>> You should also take into account xen_add_to_physmap_batch (or are you
>> planning to issue an hypercall for every single MMIO page that you want
>> to map?), but anyway the idx(s) field is there in both structs.
>>
> 
> Yeah, current approach is to issue an hypercall for every single MMIO
> page. But if we want to batch map MMIO pages, I think it needs the size
> parameter and what's idxs useful for? As we map the MMIO pages 1:1, it
> seems it's unnecessary to check "idxs[i] == gpfns[i]", right?

This is what I've been trying to say, why do we need to enforce 1:1
mappings in such way? Is there some kind of technical limitation in ARM
second stage translation that prevents doing non 1:1 mappings for MMIO
regions?

If for the initial design you need to enforce 1:1 for some reason (which
I'm interested in knowing), why don't you just check idxs[i] ==
gpfns[i], this way we can always add support for non 1:1 mappings later
if needed.

And yes, the "size" parameter in xen_add_to_physmap_batch indicates the
number of 4KB pages present in both the idxs and the gpfns arrays.

>>> is idx not idxs. Everytime Dom0 maps one page and it's mapped 1:1(guest
>>> physical address is same with real physical hardware address), so it
>>> only needs to tell the hypervisor the gpfn.
>>
>> IMHO, I'm not sure why we should restrict this to 1:1 (although I admit
>> this is going to be the common case). Didn't we are that we are going to
>> allow non 1:1 mapping of MMIO regions?
>>
>> If you want you can check in the hypercall handler that idxs[i] ==
>> gpfns[i], and return -EOPNOTSUPP if they don't match, but I still don't
>> see why this should be restricted to 1:1 mappings.
>>
> 
> For Dom0 which get the device MMIO information from the DT or ACPI DSDT
> table. To ACPI, we don't (or can't)modify anything in DSDT table. So
> actually the MMIO regions Dom0 gets are the real physical hardware MMIO
> regions and the start address and size of them are same.

I understand that 1:1 mappings are always going to be used with the
current approach in Linux, but I see no reason to enforce this inside of
Xen. It's not going to add more complexity to the hypercall handler, and
is something that we might want to use in the future.

Roger.

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Jan Beulich]

>>> On 20.08.15 at 14:56, <shannon.zhao@linaro.org> wrote:

> 
> On 2015/8/20 17:30, Jan Beulich wrote:
>>>>> On 20.08.15 at 05:41, <zhaoshenglong@huawei.com> wrote:
>>> On 2015/8/19 22:05, Jan Beulich wrote:
>>>>>>> On 19.08.15 at 14:13, <zhaoshenglong@huawei.com> wrote:
>>>>> 1. Create minimal DT to pass required information to Dom0
>>>>> ----------------------------------------------------------
>>>>> Since there are no legacy interfaces like x86 for Dom0 to get the
>>>>> booting required information on ARM, here we use the minimal DT which is
>>>>> used by UEFI stub communicating with Linux kernel as well.
>>>>>
>>>>> The UEFI stub is a feature that extends the Image/zImage into a valid
>>>>> UEFI PE/COFF executable, including a loader application that makes it
>>>>> possible to load the kernel directly from the UEFI shell, boot menu, or
>>>>> one of the lightweight bootloaders like Gummiboot or rEFInd.
>>>>>
>>>>> The kernel image built with stub support remains a valid kernel image
>>>>> for booting in non-UEFI environments and the UEFI stub will be jumped
>>>>> for non-UEFI environments.
>>>>
>>>> Isn't this backwards? I.e. ...
>>>>
>>> Sorry, what do you mean here?
>>> It's used for backwards compatibility. The reason I mention it here is
>>> to explain that Xen acts as a EFI stub which creates a minimal DT and
>>> load kernel image. Even though the kernel image built with EFI stub
>>> which will be jumped, the kernel could boot well as before.
>>
>> To me what you and Julien replied contradicts one another: You say
>> it's Xen to create these DT tags, whereas he assures me that these
>> tags are Linux internal ones.
>>
> 
> I think What Julien said is to explain that the DT properties are used 
> between EFI stub and kernel image which are built together. So if we 
> change the names in Linux, it won't break backward compatibility.
> 
> I'll explain What I mean here. If not right, please fix me.
> Firstly, Xen doesn't offer a (complete) firmware for Dom0, so it can't 
> load EFI stub, then EFI stub loads the kernel image. Therefore, there 
> are no chance to create the DT which we want to use it to pass the uefi 
> information.
> Secondly, we make Xen act like a EFI stub to create the DT which could 
> pass the information to Dom0. When we load kernel image, the kernel will 
> call efi_get_fdt_params to get the information from the DT just like the 
> DT is created by EFI stub for kernel.

So can the two of you please sort out whether these are Linux
internal tags (which Xen has no business generating, or even
knowing of) or some form of publicly documented interface? I'm
not going to comment on this any further until I get consistent
messages from both of you.

>>>>> When booting in UEFI mode, the UEFI stub will create a minimal DT in
>>>>> order to pass the command line and other informations (such as the EFI
>>>>> memory table) to the kernel. And when booting with ACPI, kernel will get
>>>>> command line, ACPI root table address and memory map information from
>>>>> the minimal DT. Also, it will check if the DT contains only the /chosen
>>>>> node to know whether it boots with DT or ACPI.
>>>>>
>>>>> In addition, the current names of these properties with a "linux,"
>>>>> prefix in the minimal DT are Linux specified. It needs to standardize
>>>>> them so that other OS(such as FreeBSD) could reuse them in the future.
>>>>> So we drop the "linux," prefix of UEFI parameters and change the names
>>>>> in Linux kernel as well.
>>>>
>>>> ... wouldn't it make more sense to leave the generation of these
>>>> Linux-specific tags to Linux (and allow them to continue to be Linux
>>>> specific), by the same or a second, parallel (Xen) stub? This would
>>>> then also move at least some of the awkward table creation (and
>>>> ideally also copying) to that stub.
>>>>
>>>>> All these tables will be copied to Dom0 memory except that the reused
>>>>> tables(DSDT, SPCR, GTDT, etc) will be mapped to Dom0.
>>>>
>>>> I continue to be puzzled by this - all of the tables should be in identical
>>>> kinds of memory, no matter whether they get mapped or copied. Yet
>>>> copying to Dom0 memory to me implies memory owned by Dom0 (and
>>>> how the kernel knows to not use it as "normal" RAM is left unsaid),
>>>> while mapping to me implies memory _not_ owned by Dom0 (much
>>>> like x86's E820 reserved type or UEFI's EfiReservedMemoryType).
>>>>
>>>
>>> These will be described in EFI_MEMORY_DESCRIPTOR table. For Dom0 memory
>>> range, the Type is EfiConventionalMemory. For ACPI tables whether it's
>>> new created or reused, the Type is EfiACPIReclaimMemory. For EFI System
>>> table and Memory Descriptor table, the Type is EfiReservedMemoryType.
>>
>> And Dom0 will have to figure out how to establish mappings for the
>> two different kinds (which ought to work differently since in one case
>> Dom0-owned memory is to be mapped, while in the other case it's
>> Xen- [or even firmware-]owned memory)? Anyway - even without
>> that aspect, it seems conceptually wrong to place firmware tables
>> (even if they are fake ones) in Dom0-owned memory.
>>
> Here we do not want to supply Dom0 with a firmware. We just want to take 
> advantage of the existing way to pass the information to Dom0.
> 
> If this not right, do you have any other good ideas for this? Please 
> share your suggestion with me.

I already said how I would see this done cleanly: Even if the tables
are being made up, pass them in a unique, consistent way so that
Dom0 has a unique way to access them. And the only consistent
way I can see is pass them all in memory _not_ owned by Dom0.

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Jan Beulich]

>>> On 20.08.15 at 15:46, <roger.pau@citrix.com> wrote:
> El 20/08/15 a les 14.29, Shannon Zhao ha escrit:
>> 
>> 
>> On 2015/8/20 19:28, Roger Pau Monné wrote:
>>> El 20/08/15 a les 13.22, Shannon Zhao ha escrit:
>>>> Hi Roger,
>>>>
>>>> On 2015/8/20 16:20, Roger Pau Monné wrote:
>>>>> El 20/08/15 a les 5.07, Shannon Zhao ha escrit:
>>>>>> On 2015/8/19 23:02, Roger Pau Monné wrote:
>>>>>>> El 19/08/15 a les 14.13, Shannon Zhao ha escrit:
>>>>>>>> XENMAPSPACE "XENMAPSPACE_dev_mmio". The usage of this hypercall
>>>>>>>> parameters:
>>>>>>>> - domid: DOMID_SELF.
>>>>>>>> - space: XENMAPSPACE_dev_mmio.
>>>>>>>> - gpfns: guest physical addresses where the mapping should appear.
>>>>>>>
>>>>>>> This is not complete, you have forgotten to add the idxs field,
>>>>>>
>>>>>> Sorry, I didn't use the idx for the mmio region mapping. What's the
>>>>>> idx
>>>>>> useful for here?
>>>>>
>>>>> I've already posted this in the previous version, and you agreed on the
>>>>> interface and the usage of the fields, please see:
>>>>>
>>>>> http://marc.info/?l=xen-devel&m=143986236212359 
>>>>>
>>>>> The idxs field is explicitly mentioned there with it's usage.
>>>>>
>>>>
>>>> Yeah, I said I will add the description of hypercall parameters.
>>>> It seems that we are talking about a different parameter.
>>>> To map the mmio region, I reuse the struct xen_add_to_physmap and there
>>>
>>> You should also take into account xen_add_to_physmap_batch (or are you
>>> planning to issue an hypercall for every single MMIO page that you want
>>> to map?), but anyway the idx(s) field is there in both structs.
>>>
>> 
>> Yeah, current approach is to issue an hypercall for every single MMIO
>> page. But if we want to batch map MMIO pages, I think it needs the size
>> parameter and what's idxs useful for? As we map the MMIO pages 1:1, it
>> seems it's unnecessary to check "idxs[i] == gpfns[i]", right?
> 
> This is what I've been trying to say, why do we need to enforce 1:1
> mappings in such way? Is there some kind of technical limitation in ARM
> second stage translation that prevents doing non 1:1 mappings for MMIO
> regions?

And even if there was such a limitation, this _still_ wouldn't justify
designing an inflexible interface - the interface shouldn't be tailored
to a current restriction of a particular architecture.

>>>> is idx not idxs. Everytime Dom0 maps one page and it's mapped 1:1(guest
>>>> physical address is same with real physical hardware address), so it
>>>> only needs to tell the hypervisor the gpfn.
>>>
>>> IMHO, I'm not sure why we should restrict this to 1:1 (although I admit
>>> this is going to be the common case). Didn't we are that we are going to
>>> allow non 1:1 mapping of MMIO regions?
>>>
>>> If you want you can check in the hypercall handler that idxs[i] ==
>>> gpfns[i], and return -EOPNOTSUPP if they don't match, but I still don't
>>> see why this should be restricted to 1:1 mappings.
>>>
>> 
>> For Dom0 which get the device MMIO information from the DT or ACPI DSDT
>> table. To ACPI, we don't (or can't)modify anything in DSDT table. So
>> actually the MMIO regions Dom0 gets are the real physical hardware MMIO
>> regions and the start address and size of them are same.
> 
> I understand that 1:1 mappings are always going to be used with the
> current approach in Linux, but I see no reason to enforce this inside of
> Xen.

You mean inside the ABI, I suppose? Since you suggested yourself
to bail on idx != gfn for the time being.

> It's not going to add more complexity to the hypercall handler, and
> is something that we might want to use in the future.

Indeed.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Shannon Zhao]

On 2015/8/20 22:06, Jan Beulich wrote:
>>>> On 20.08.15 at 14:56, <shannon.zhao@linaro.org> wrote:
> 
>>
>> On 2015/8/20 17:30, Jan Beulich wrote:
>>>>>> On 20.08.15 at 05:41, <zhaoshenglong@huawei.com> wrote:
>>>> On 2015/8/19 22:05, Jan Beulich wrote:
>>>>>>>> On 19.08.15 at 14:13, <zhaoshenglong@huawei.com> wrote:
>>>>>> 1. Create minimal DT to pass required information to Dom0
>>>>>> ----------------------------------------------------------
>>>>>> Since there are no legacy interfaces like x86 for Dom0 to get the
>>>>>> booting required information on ARM, here we use the minimal DT which is
>>>>>> used by UEFI stub communicating with Linux kernel as well.
>>>>>>
>>>>>> The UEFI stub is a feature that extends the Image/zImage into a valid
>>>>>> UEFI PE/COFF executable, including a loader application that makes it
>>>>>> possible to load the kernel directly from the UEFI shell, boot menu, or
>>>>>> one of the lightweight bootloaders like Gummiboot or rEFInd.
>>>>>>
>>>>>> The kernel image built with stub support remains a valid kernel image
>>>>>> for booting in non-UEFI environments and the UEFI stub will be jumped
>>>>>> for non-UEFI environments.
>>>>>
>>>>> Isn't this backwards? I.e. ...
>>>>>
>>>> Sorry, what do you mean here?
>>>> It's used for backwards compatibility. The reason I mention it here is
>>>> to explain that Xen acts as a EFI stub which creates a minimal DT and
>>>> load kernel image. Even though the kernel image built with EFI stub
>>>> which will be jumped, the kernel could boot well as before.
>>>
>>> To me what you and Julien replied contradicts one another: You say
>>> it's Xen to create these DT tags, whereas he assures me that these
>>> tags are Linux internal ones.
>>>
>>
>> I think What Julien said is to explain that the DT properties are used 
>> between EFI stub and kernel image which are built together. So if we 
>> change the names in Linux, it won't break backward compatibility.
>>
>> I'll explain What I mean here. If not right, please fix me.
>> Firstly, Xen doesn't offer a (complete) firmware for Dom0, so it can't 
>> load EFI stub, then EFI stub loads the kernel image. Therefore, there 
>> are no chance to create the DT which we want to use it to pass the uefi 
>> information.
>> Secondly, we make Xen act like a EFI stub to create the DT which could 
>> pass the information to Dom0. When we load kernel image, the kernel will 
>> call efi_get_fdt_params to get the information from the DT just like the 
>> DT is created by EFI stub for kernel.
> 
> So can the two of you please sort out whether these are Linux
> internal tags (which Xen has no business generating, or even
> knowing of) or some form of publicly documented interface?

They are Linux internal tags. But to support Xen ACPI on ARM, we just
reuse the existing mechanism to pass the information to Dom0.

Wait for Julien's reply.

> I'm
> not going to comment on this any further until I get consistent
> messages from both of you.
> 


>>>>>> When booting in UEFI mode, the UEFI stub will create a minimal DT in
>>>>>> order to pass the command line and other informations (such as the EFI
>>>>>> memory table) to the kernel. And when booting with ACPI, kernel will get
>>>>>> command line, ACPI root table address and memory map information from
>>>>>> the minimal DT. Also, it will check if the DT contains only the /chosen
>>>>>> node to know whether it boots with DT or ACPI.
>>>>>>
>>>>>> In addition, the current names of these properties with a "linux,"
>>>>>> prefix in the minimal DT are Linux specified. It needs to standardize
>>>>>> them so that other OS(such as FreeBSD) could reuse them in the future.
>>>>>> So we drop the "linux," prefix of UEFI parameters and change the names
>>>>>> in Linux kernel as well.
>>>>>
>>>>> ... wouldn't it make more sense to leave the generation of these
>>>>> Linux-specific tags to Linux (and allow them to continue to be Linux
>>>>> specific), by the same or a second, parallel (Xen) stub? This would
>>>>> then also move at least some of the awkward table creation (and
>>>>> ideally also copying) to that stub.
>>>>>
>>>>>> All these tables will be copied to Dom0 memory except that the reused
>>>>>> tables(DSDT, SPCR, GTDT, etc) will be mapped to Dom0.
>>>>>
>>>>> I continue to be puzzled by this - all of the tables should be in identical
>>>>> kinds of memory, no matter whether they get mapped or copied. Yet
>>>>> copying to Dom0 memory to me implies memory owned by Dom0 (and
>>>>> how the kernel knows to not use it as "normal" RAM is left unsaid),
>>>>> while mapping to me implies memory _not_ owned by Dom0 (much
>>>>> like x86's E820 reserved type or UEFI's EfiReservedMemoryType).
>>>>>
>>>>
>>>> These will be described in EFI_MEMORY_DESCRIPTOR table. For Dom0 memory
>>>> range, the Type is EfiConventionalMemory. For ACPI tables whether it's
>>>> new created or reused, the Type is EfiACPIReclaimMemory. For EFI System
>>>> table and Memory Descriptor table, the Type is EfiReservedMemoryType.
>>>
>>> And Dom0 will have to figure out how to establish mappings for the
>>> two different kinds (which ought to work differently since in one case
>>> Dom0-owned memory is to be mapped, while in the other case it's
>>> Xen- [or even firmware-]owned memory)? Anyway - even without
>>> that aspect, it seems conceptually wrong to place firmware tables
>>> (even if they are fake ones) in Dom0-owned memory.
>>>
>> Here we do not want to supply Dom0 with a firmware. We just want to take 
>> advantage of the existing way to pass the information to Dom0.
>>
>> If this not right, do you have any other good ideas for this? Please 
>> share your suggestion with me.
> 
> I already said how I would see this done cleanly: Even if the tables
> are being made up, pass them in a unique, consistent way so that
> Dom0 has a unique way to access them. And the only consistent
> way I can see is pass them all in memory _not_ owned by Dom0.
> 

I'm ok with placing them in the memory not owned by Dom0. Is there some
ways to find that kind of memory space in Xen?

Thanks,
-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Shannon Zhao]

On 2015/8/20 21:46, Roger Pau Monné wrote:
> El 20/08/15 a les 14.29, Shannon Zhao ha escrit:
>>
>>
>> On 2015/8/20 19:28, Roger Pau Monné wrote:
>>> El 20/08/15 a les 13.22, Shannon Zhao ha escrit:
>>>> Hi Roger,
>>>>
>>>> On 2015/8/20 16:20, Roger Pau Monné wrote:
>>>>> El 20/08/15 a les 5.07, Shannon Zhao ha escrit:
>>>>>> On 2015/8/19 23:02, Roger Pau Monné wrote:
>>>>>>> El 19/08/15 a les 14.13, Shannon Zhao ha escrit:
>>>>>>>> XENMAPSPACE "XENMAPSPACE_dev_mmio". The usage of this hypercall
>>>>>>>> parameters:
>>>>>>>> - domid: DOMID_SELF.
>>>>>>>> - space: XENMAPSPACE_dev_mmio.
>>>>>>>> - gpfns: guest physical addresses where the mapping should appear.
>>>>>>>
>>>>>>> This is not complete, you have forgotten to add the idxs field,
>>>>>>
>>>>>> Sorry, I didn't use the idx for the mmio region mapping. What's the
>>>>>> idx
>>>>>> useful for here?
>>>>>
>>>>> I've already posted this in the previous version, and you agreed on the
>>>>> interface and the usage of the fields, please see:
>>>>>
>>>>> http://marc.info/?l=xen-devel&m=143986236212359
>>>>>
>>>>> The idxs field is explicitly mentioned there with it's usage.
>>>>>
>>>>
>>>> Yeah, I said I will add the description of hypercall parameters.
>>>> It seems that we are talking about a different parameter.
>>>> To map the mmio region, I reuse the struct xen_add_to_physmap and there
>>>
>>> You should also take into account xen_add_to_physmap_batch (or are you
>>> planning to issue an hypercall for every single MMIO page that you want
>>> to map?), but anyway the idx(s) field is there in both structs.
>>>
>>
>> Yeah, current approach is to issue an hypercall for every single MMIO
>> page. But if we want to batch map MMIO pages, I think it needs the size
>> parameter and what's idxs useful for? As we map the MMIO pages 1:1, it
>> seems it's unnecessary to check "idxs[i] == gpfns[i]", right?
> 
> This is what I've been trying to say, why do we need to enforce 1:1
> mappings in such way? Is there some kind of technical limitation in ARM
> second stage translation that prevents doing non 1:1 mappings for MMIO
> regions?
> 
> If for the initial design you need to enforce 1:1 for some reason (which
> I'm interested in knowing), why don't you just check idxs[i] ==
> gpfns[i], this way we can always add support for non 1:1 mappings later
> if needed.
> 
> And yes, the "size" parameter in xen_add_to_physmap_batch indicates the
> number of 4KB pages present in both the idxs and the gpfns arrays.
> 
>>>> is idx not idxs. Everytime Dom0 maps one page and it's mapped 1:1(guest
>>>> physical address is same with real physical hardware address), so it
>>>> only needs to tell the hypervisor the gpfn.
>>>
>>> IMHO, I'm not sure why we should restrict this to 1:1 (although I admit
>>> this is going to be the common case). Didn't we are that we are going to
>>> allow non 1:1 mapping of MMIO regions?
>>>
>>> If you want you can check in the hypercall handler that idxs[i] ==
>>> gpfns[i], and return -EOPNOTSUPP if they don't match, but I still don't
>>> see why this should be restricted to 1:1 mappings.
>>>
>>
>> For Dom0 which get the device MMIO information from the DT or ACPI DSDT
>> table. To ACPI, we don't (or can't)modify anything in DSDT table. So
>> actually the MMIO regions Dom0 gets are the real physical hardware MMIO
>> regions and the start address and size of them are same.
> 
> I understand that 1:1 mappings are always going to be used with the
> current approach in Linux, but I see no reason to enforce this inside of
> Xen. It's not going to add more complexity to the hypercall handler, and
> is something that we might want to use in the future.
> 

Ok, I will change this to make it can handle non 1:1 mappings. Let guest
decide what kind of mapping it wants to use.

-- 
Shannon


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Jan Beulich]

>>> On 21.08.15 at 04:25, <zhaoshenglong@huawei.com> wrote:
> I'm ok with placing them in the memory not owned by Dom0. Is there some
> ways to find that kind of memory space in Xen?

Who do you need to be able to find such memory? Or maybe I'm not
getting what or why you're asking...

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Julien Grall]

Hi,

Sorry for the late answer.

On 20/08/2015 19:25, Shannon Zhao wrote:
> On 2015/8/20 22:06, Jan Beulich wrote:
>>>>> On 20.08.15 at 14:56, <shannon.zhao@linaro.org> wrote:
>>
>>>
>>> On 2015/8/20 17:30, Jan Beulich wrote:
>>>>>>> On 20.08.15 at 05:41, <zhaoshenglong@huawei.com> wrote:
>>>>> On 2015/8/19 22:05, Jan Beulich wrote:
>>>>>>>>> On 19.08.15 at 14:13, <zhaoshenglong@huawei.com> wrote:
>>>>>>> 1. Create minimal DT to pass required information to Dom0
>>>>>>> ----------------------------------------------------------
>>>>>>> Since there are no legacy interfaces like x86 for Dom0 to get the
>>>>>>> booting required information on ARM, here we use the minimal DT which is
>>>>>>> used by UEFI stub communicating with Linux kernel as well.
>>>>>>>
>>>>>>> The UEFI stub is a feature that extends the Image/zImage into a valid
>>>>>>> UEFI PE/COFF executable, including a loader application that makes it
>>>>>>> possible to load the kernel directly from the UEFI shell, boot menu, or
>>>>>>> one of the lightweight bootloaders like Gummiboot or rEFInd.
>>>>>>>
>>>>>>> The kernel image built with stub support remains a valid kernel image
>>>>>>> for booting in non-UEFI environments and the UEFI stub will be jumped
>>>>>>> for non-UEFI environments.
>>>>>>
>>>>>> Isn't this backwards? I.e. ...
>>>>>>
>>>>> Sorry, what do you mean here?
>>>>> It's used for backwards compatibility. The reason I mention it here is
>>>>> to explain that Xen acts as a EFI stub which creates a minimal DT and
>>>>> load kernel image. Even though the kernel image built with EFI stub
>>>>> which will be jumped, the kernel could boot well as before.
>>>>
>>>> To me what you and Julien replied contradicts one another: You say
>>>> it's Xen to create these DT tags, whereas he assures me that these
>>>> tags are Linux internal ones.
>>>>
>>>
>>> I think What Julien said is to explain that the DT properties are used
>>> between EFI stub and kernel image which are built together. So if we
>>> change the names in Linux, it won't break backward compatibility.
>>>
>>> I'll explain What I mean here. If not right, please fix me.
>>> Firstly, Xen doesn't offer a (complete) firmware for Dom0, so it can't
>>> load EFI stub, then EFI stub loads the kernel image. Therefore, there
>>> are no chance to create the DT which we want to use it to pass the uefi
>>> information.
>>> Secondly, we make Xen act like a EFI stub to create the DT which could
>>> pass the information to Dom0. When we load kernel image, the kernel will
>>> call efi_get_fdt_params to get the information from the DT just like the
>>> DT is created by EFI stub for kernel.
>>
>> So can the two of you please sort out whether these are Linux
>> internal tags (which Xen has no business generating, or even
>> knowing of) or some form of publicly documented interface?
>
> They are Linux internal tags. But to support Xen ACPI on ARM, we just
> reuse the existing mechanism to pass the information to Dom0.

My previous mail was unclear, sorry. I was explaining the current usage 
of those properties because you seemed to be concern about the backward 
compatibility.

We'd like to formalize those properties in order to use them to pass 
information between Xen and the kernel image.

This would avoid us to create a Xen specific entry path in the kernel 
and use the non-EFI one (also used by the EFI stub) and make easier to 
support new DOM0 OS (for instance FreeBSD).

So I totally agree with Shannon mails.

Regards,

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Jan Beulich]

>>> On 27.08.15 at 02:37, <julien.grall@citrix.com> wrote:
> On 20/08/2015 19:25, Shannon Zhao wrote:
>> On 2015/8/20 22:06, Jan Beulich wrote:
>>> So can the two of you please sort out whether these are Linux
>>> internal tags (which Xen has no business generating, or even
>>> knowing of) or some form of publicly documented interface?
>>
>> They are Linux internal tags. But to support Xen ACPI on ARM, we just
>> reuse the existing mechanism to pass the information to Dom0.
> 
> My previous mail was unclear, sorry. I was explaining the current usage 
> of those properties because you seemed to be concern about the backward 
> compatibility.
> 
> We'd like to formalize those properties in order to use them to pass 
> information between Xen and the kernel image.

Ah, that clarifies things. Thanks.

> This would avoid us to create a Xen specific entry path in the kernel 
> and use the non-EFI one (also used by the EFI stub) and make easier to 
> support new DOM0 OS (for instance FreeBSD).
> 
> So I totally agree with Shannon mails.

One other aspect completely left off so far is that of proper isolation:
What x86 exposes to Dom0 is specifically limited to what Dom0 is
supposed to know. I'm getting the impression that by exposing more
EFI tables this is being violated just for the purpose of avoiding any
changes to Linux. But maybe I'm misremembering, and all the extra
tables exposed are actually fake ones rather than cloned host ones.

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Shannon Zhao]

On 2015/8/27 15:52, Jan Beulich wrote:
>>>> On 27.08.15 at 02:37, <julien.grall@citrix.com> wrote:
>> On 20/08/2015 19:25, Shannon Zhao wrote:
>>> On 2015/8/20 22:06, Jan Beulich wrote:
>>>> So can the two of you please sort out whether these are Linux
>>>> internal tags (which Xen has no business generating, or even
>>>> knowing of) or some form of publicly documented interface?
>>>
>>> They are Linux internal tags. But to support Xen ACPI on ARM, we just
>>> reuse the existing mechanism to pass the information to Dom0.
>>
>> My previous mail was unclear, sorry. I was explaining the current usage
>> of those properties because you seemed to be concern about the backward
>> compatibility.
>>
>> We'd like to formalize those properties in order to use them to pass
>> information between Xen and the kernel image.
>
> Ah, that clarifies things. Thanks.
>
>> This would avoid us to create a Xen specific entry path in the kernel
>> and use the non-EFI one (also used by the EFI stub) and make easier to
>> support new DOM0 OS (for instance FreeBSD).
>>
>> So I totally agree with Shannon mails.
>
> One other aspect completely left off so far is that of proper isolation:
> What x86 exposes to Dom0 is specifically limited to what Dom0 is
> supposed to know. I'm getting the impression that by exposing more
> EFI tables this is being violated just for the purpose of avoiding any
> changes to Linux. But maybe I'm misremembering, and all the extra
> tables exposed are actually fake ones rather than cloned host ones.
>

Currently we create EFI system table and EFI memory descriptor table as 
Linux requires. I think the EFI memory descriptor table is necessary. 
What we didn't reach an agreement is only the EFI system table. Yes, we 
only use the Configure table of the EFI system table to get the ACPI 
root address. As you mentioned before, it could pass only the Configure 
table to Dom0, but we should change the process of parsing the DT and 
consider the backwards compatibility.

On the other hand, as the RUNTIME service is not supported, it could 
assign the runtime service members of EFI system table invalid values 
and let Dom0 not initialize RUNTIME service(This could be done by making 
the memory attribute not be EFI_MEMORY_RUNTIME when we create the EFI 
memory descriptor table). If the RUNTIME service is supported in the 
future, it doesn't need to change the Linux again. So it could avoid 
changing back.

Jan, how do you think about this?

Thanks,
-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Jan Beulich]

>>> On 27.08.15 at 15:50, <shannon.zhao@linaro.org> wrote:
> On 2015/8/27 15:52, Jan Beulich wrote:
>> One other aspect completely left off so far is that of proper isolation:
>> What x86 exposes to Dom0 is specifically limited to what Dom0 is
>> supposed to know. I'm getting the impression that by exposing more
>> EFI tables this is being violated just for the purpose of avoiding any
>> changes to Linux. But maybe I'm misremembering, and all the extra
>> tables exposed are actually fake ones rather than cloned host ones.
> 
> Currently we create EFI system table and EFI memory descriptor table as 
> Linux requires. I think the EFI memory descriptor table is necessary. 
> What we didn't reach an agreement is only the EFI system table. Yes, we 
> only use the Configure table of the EFI system table to get the ACPI 
> root address. As you mentioned before, it could pass only the Configure 
> table to Dom0, but we should change the process of parsing the DT and 
> consider the backwards compatibility.

A made up system table would (as said before) be fine with me too.
Just not a clone of the host one.

> On the other hand, as the RUNTIME service is not supported, it could 
> assign the runtime service members of EFI system table invalid values 
> and let Dom0 not initialize RUNTIME service(This could be done by making 
> the memory attribute not be EFI_MEMORY_RUNTIME when we create the EFI 
> memory descriptor table). If the RUNTIME service is supported in the 
> future, it doesn't need to change the Linux again. So it could avoid 
> changing back.

I'd strongly advise against such hackery - it will get you (and Xen)
into the bad firmware corner. EFI without runtime services doesn't
exist. And runtime services code/data not marked as such are a
firmware bug (sadly existing in reality on the x86 side). But remember
that under Xen the Dom0 kernel mustn't care about runtime services
(other than wanting to be able to invoke them through hypercalls).

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 4

[Shannon Zhao]

On 2015/8/27 22:13, Jan Beulich wrote:
>>>> On 27.08.15 at 15:50, <shannon.zhao@linaro.org> wrote:
>> On 2015/8/27 15:52, Jan Beulich wrote:
>>> One other aspect completely left off so far is that of proper isolation:
>>> What x86 exposes to Dom0 is specifically limited to what Dom0 is
>>> supposed to know. I'm getting the impression that by exposing more
>>> EFI tables this is being violated just for the purpose of avoiding any
>>> changes to Linux. But maybe I'm misremembering, and all the extra
>>> tables exposed are actually fake ones rather than cloned host ones.
>>
>> Currently we create EFI system table and EFI memory descriptor table as
>> Linux requires. I think the EFI memory descriptor table is necessary.
>> What we didn't reach an agreement is only the EFI system table. Yes, we
>> only use the Configure table of the EFI system table to get the ACPI
>> root address. As you mentioned before, it could pass only the Configure
>> table to Dom0, but we should change the process of parsing the DT and
>> consider the backwards compatibility.
>
> A made up system table would (as said before) be fine with me too.
> Just not a clone of the host one.
>

Yeah, it's a made up one.

>> On the other hand, as the RUNTIME service is not supported, it could
>> assign the runtime service members of EFI system table invalid values
>> and let Dom0 not initialize RUNTIME service(This could be done by making
>> the memory attribute not be EFI_MEMORY_RUNTIME when we create the EFI
>> memory descriptor table). If the RUNTIME service is supported in the
>> future, it doesn't need to change the Linux again. So it could avoid
>> changing back.
>
> I'd strongly advise against such hackery - it will get you (and Xen)
> into the bad firmware corner. EFI without runtime services doesn't
> exist. And runtime services code/data not marked as such are a
> firmware bug (sadly existing in reality on the x86 side). But remember
> that under Xen the Dom0 kernel mustn't care about runtime services
> (other than wanting to be able to invoke them through hypercalls).
>

Oh, I see. Thanks for your explanation.

-- 
Shannon