Design doc of adding ACPI support for arm64 on Xen - version 2

Design doc of adding ACPI support for arm64 on Xen - version 2

[Shannon Zhao]

This document is going to explain the design details of Xen booting with
ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are
welcome.

To Xen itself booting with ACPI, this is similar to Linux kernel except
that Xen doesn't parse DSDT table. So I'll skip this part and focus on
how Xen prepares ACPI tables for Dom0 and how Xen passes them to Dom0.

1. Copy and change some EFI and ACPI tables
-------------------------------------------
a) Copy EFI_SYSTEM_TABLE and change the value of FirmwareVendor,
VendorGuid, VendorTable, ConfigurationTable. These changes are not very
special and it just assign values to these members.

b) Create EFI_MEMORY_DESCRIPTOR table. This will add memory start and
size information of Dom0. And Dom0 will get the memory information
through this EFI table.

c) Copy FADT table. Change the value of arm_boot_flags to enable PSCI
and HVC. Let the hypervisor_id be "XenVMM" in order to tell Dom0 that it
runs on Xen hypervisor, then Dom0 could through HVM_PARAM to get some
informations for booting necessity, such as grant table start address
and size. Change header revison, length and checksum as well.

d) Copy GTDT table. Set non_secure_el2_interrupt and
non_secure_el2_flags to 0 to mask EL2 timer for Dom0.

e) Copy MADT table. According to the value of dom0_max_vcpus to change
the number GICC entries.

f) Create STAO table. This table is a new added one that's used to
define a list of ACPI namespace names that are to be ignored by the OSPM
in Dom0. Currently we use it to tell OSPM should ignore UART defined in
SPCR table.

g) Copy XSDT table. Add a new table entry for STAO and change other
table's entries.

h) Change the value of xsdt_physical_address in RSDP table.

i) The rest of tables are not copied or changed. They are reused
including DSDT, SPCR, etc.

All these tables will be copied to Dom0 memory except that the reused
tables(DSDT, SPCR, etc) will be mapped to Dom0.

2. Create minimal DT to pass required information to Dom0
----------------------------------------------------------
The minimal DT mainly passes Dom0 bootargs, address and size of initrd
(if available), address and size of uefi system table, address and size
of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver.

An example of the minimal DT:
/ {
    #address-cells = <2>;
    #size-cells = <1>;
    chosen {
        bootargs = "kernel=Image console=hvc0 earlycon=pl011,0x1c090000
root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force";
        linux,initrd-start = <0xXXXXXXXX>;
        linux,initrd-end = <0xXXXXXXXX>;
        linux,uefi-system-table = <0xXXXXXXXX>;
        linux,uefi-mmap-start = <0xXXXXXXXX>;
        linux,uefi-mmap-size = <0xXXXXXXXX>;
        linux,uefi-mmap-desc-size = <0xXXXXXXXX>;
        linux,uefi-mmap-desc-ver = <0xXXXXXXXX>;
    };
};

For details loook at
https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi.txt

3. Dom0 gets grant table and event channel irq information
-----------------------------------------------------------
As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that
it runs on Xen hypervisor.

For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS
and HVM_PARAM_GNTTAB_SIZE.

For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new
delivery type:
val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64
only)

When constructing Dom0 in Xen, save these values. Then Dom0 could get
them through hypercall HVMOP_get_param.

4. Map MMIO regions
-------------------
Register a bus_notifier for platform and amba bus in Linux. Add a new
XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the
device is newly added, then call hypercall XENMEM_add_to_physmap to map
the mmio regions.

5. Route device interrupts to Dom0
----------------------------------
Route all the SPI interrupts to Dom0 before Dom0 booting.


Look forward to your comments. If you think it has no problem, giving
your ack would be a big help. Then the patchset could move on. :)

Thanks,
-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Stefano Stabellini]

On Fri, 7 Aug 2015, Shannon Zhao wrote:
> This document is going to explain the design details of Xen booting with
> ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are
> welcome.
> 
> To Xen itself booting with ACPI, this is similar to Linux kernel except
> that Xen doesn't parse DSDT table. So I'll skip this part and focus on
> how Xen prepares ACPI tables for Dom0 and how Xen passes them to Dom0.
> 
> 1. Copy and change some EFI and ACPI tables
> -------------------------------------------
> a) Copy EFI_SYSTEM_TABLE and change the value of FirmwareVendor,
> VendorGuid, VendorTable, ConfigurationTable. These changes are not very
> special and it just assign values to these members.
> 
> b) Create EFI_MEMORY_DESCRIPTOR table. This will add memory start and
> size information of Dom0. And Dom0 will get the memory information
> through this EFI table.
> 
> c) Copy FADT table. Change the value of arm_boot_flags to enable PSCI
> and HVC. Let the hypervisor_id be "XenVMM" in order to tell Dom0 that it
> runs on Xen hypervisor, then Dom0 could through HVM_PARAM to get some
> informations for booting necessity, such as grant table start address
> and size. Change header revison, length and checksum as well.
> 
> d) Copy GTDT table. Set non_secure_el2_interrupt and
> non_secure_el2_flags to 0 to mask EL2 timer for Dom0.
> 
> e) Copy MADT table. According to the value of dom0_max_vcpus to change
> the number GICC entries.
> 
> f) Create STAO table. This table is a new added one that's used to
> define a list of ACPI namespace names that are to be ignored by the OSPM
> in Dom0. Currently we use it to tell OSPM should ignore UART defined in
> SPCR table.
> 
> g) Copy XSDT table. Add a new table entry for STAO and change other
> table's entries.
> 
> h) Change the value of xsdt_physical_address in RSDP table.
> 
> i) The rest of tables are not copied or changed. They are reused
> including DSDT, SPCR, etc.
> 
> All these tables will be copied to Dom0 memory except that the reused
> tables(DSDT, SPCR, etc) will be mapped to Dom0.
> 
> 2. Create minimal DT to pass required information to Dom0
> ----------------------------------------------------------
> The minimal DT mainly passes Dom0 bootargs, address and size of initrd
> (if available), address and size of uefi system table, address and size
> of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver.
> 
> An example of the minimal DT:
> / {
>     #address-cells = <2>;
>     #size-cells = <1>;
>     chosen {
>         bootargs = "kernel=Image console=hvc0 earlycon=pl011,0x1c090000
> root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force";
>         linux,initrd-start = <0xXXXXXXXX>;
>         linux,initrd-end = <0xXXXXXXXX>;
>         linux,uefi-system-table = <0xXXXXXXXX>;
>         linux,uefi-mmap-start = <0xXXXXXXXX>;
>         linux,uefi-mmap-size = <0xXXXXXXXX>;
>         linux,uefi-mmap-desc-size = <0xXXXXXXXX>;
>         linux,uefi-mmap-desc-ver = <0xXXXXXXXX>;
>     };
> };
> 
> For details loook at
> https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi.txt
> 
> 3. Dom0 gets grant table and event channel irq information
> -----------------------------------------------------------
> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that
> it runs on Xen hypervisor.
> 
> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS
> and HVM_PARAM_GNTTAB_SIZE.
> 
> For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new
> delivery type:
> val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64
> only)
> 
> When constructing Dom0 in Xen, save these values. Then Dom0 could get
> them through hypercall HVMOP_get_param.
> 
> 4. Map MMIO regions
> -------------------
> Register a bus_notifier for platform and amba bus in Linux. Add a new
> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the
> device is newly added, then call hypercall XENMEM_add_to_physmap to map
> the mmio regions.
> 
> 5. Route device interrupts to Dom0
> ----------------------------------
> Route all the SPI interrupts to Dom0 before Dom0 booting.
> 
> 
> Look forward to your comments. If you think it has no problem, giving
> your ack would be a big help. Then the patchset could move on. :)

Indeed.

Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

It would be nice to have an Ack from Jan too. He should be back in a
couple of days.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

Hi Shannon,

Just some clarification questions.

On 07/08/15 03:11, Shannon Zhao wrote:
> 3. Dom0 gets grant table and event channel irq information
> -----------------------------------------------------------
> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that
> it runs on Xen hypervisor.
> 
> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS
> and HVM_PARAM_GNTTAB_SIZE.
> 
> For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new
> delivery type:
> val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64
> only)

Can you describe the content of flag?

> When constructing Dom0 in Xen, save these values. Then Dom0 could get
> them through hypercall HVMOP_get_param.
> 
> 4. Map MMIO regions
> -------------------
> Register a bus_notifier for platform and amba bus in Linux. Add a new
> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the
> device is newly added, then call hypercall XENMEM_add_to_physmap to map
> the mmio regions.
> 
> 5. Route device interrupts to Dom0
> ----------------------------------
> Route all the SPI interrupts to Dom0 before Dom0 booting.

Not all the SPI will be routed to DOM0. Some are used by Xen and should
never be used by any guest. I have in mind the UART and SMMU interrupts.

You will have to find away to skip them nicely. Note that not all the
IRQs used by Xen are properly registered when we build DOM0 (see the SMMU).

Regards,

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Christoffer Dall]

On Fri, Aug 7, 2015 at 12:33 PM, Julien Grall <julien.grall@citrix.com> wrote:
> Hi Shannon,
>
> Just some clarification questions.
>
> On 07/08/15 03:11, Shannon Zhao wrote:
>> 3. Dom0 gets grant table and event channel irq information
>> -----------------------------------------------------------
>> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that
>> it runs on Xen hypervisor.
>>
>> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS
>> and HVM_PARAM_GNTTAB_SIZE.
>>
>> For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new
>> delivery type:
>> val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64
>> only)
>
> Can you describe the content of flag?
>
>> When constructing Dom0 in Xen, save these values. Then Dom0 could get
>> them through hypercall HVMOP_get_param.
>>
>> 4. Map MMIO regions
>> -------------------
>> Register a bus_notifier for platform and amba bus in Linux. Add a new
>> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the
>> device is newly added, then call hypercall XENMEM_add_to_physmap to map
>> the mmio regions.
>>
>> 5. Route device interrupts to Dom0
>> ----------------------------------
>> Route all the SPI interrupts to Dom0 before Dom0 booting.
>
> Not all the SPI will be routed to DOM0. Some are used by Xen and should
> never be used by any guest. I have in mind the UART and SMMU interrupts.
>
> You will have to find away to skip them nicely. Note that not all the
> IRQs used by Xen are properly registered when we build DOM0 (see the SMMU).
>
Just to clarify; Xen should map all SPIs which are not reserved for
Xen to Dom0, right?

-Christoffer

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Stefano Stabellini]

On Fri, 7 Aug 2015, Christoffer Dall wrote:
> On Fri, Aug 7, 2015 at 12:33 PM, Julien Grall <julien.grall@citrix.com> wrote:
> > Hi Shannon,
> >
> > Just some clarification questions.
> >
> > On 07/08/15 03:11, Shannon Zhao wrote:
> >> 3. Dom0 gets grant table and event channel irq information
> >> -----------------------------------------------------------
> >> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that
> >> it runs on Xen hypervisor.
> >>
> >> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS
> >> and HVM_PARAM_GNTTAB_SIZE.
> >>
> >> For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new
> >> delivery type:
> >> val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64
> >> only)
> >
> > Can you describe the content of flag?
> >
> >> When constructing Dom0 in Xen, save these values. Then Dom0 could get
> >> them through hypercall HVMOP_get_param.
> >>
> >> 4. Map MMIO regions
> >> -------------------
> >> Register a bus_notifier for platform and amba bus in Linux. Add a new
> >> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the
> >> device is newly added, then call hypercall XENMEM_add_to_physmap to map
> >> the mmio regions.
> >>
> >> 5. Route device interrupts to Dom0
> >> ----------------------------------
> >> Route all the SPI interrupts to Dom0 before Dom0 booting.
> >
> > Not all the SPI will be routed to DOM0. Some are used by Xen and should
> > never be used by any guest. I have in mind the UART and SMMU interrupts.
> >
> > You will have to find away to skip them nicely. Note that not all the
> > IRQs used by Xen are properly registered when we build DOM0 (see the SMMU).
> >
> Just to clarify; Xen should map all SPIs which are not reserved for
> Xen to Dom0, right?

Yes, all SPIs that Xen does not use for itself.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

On 07/08/15 11:37, Christoffer Dall wrote:
> On Fri, Aug 7, 2015 at 12:33 PM, Julien Grall <julien.grall@citrix.com> wrote:
>> Hi Shannon,
>>
>> Just some clarification questions.
>>
>> On 07/08/15 03:11, Shannon Zhao wrote:
>>> 3. Dom0 gets grant table and event channel irq information
>>> -----------------------------------------------------------
>>> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that
>>> it runs on Xen hypervisor.
>>>
>>> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS
>>> and HVM_PARAM_GNTTAB_SIZE.
>>>
>>> For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new
>>> delivery type:
>>> val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64
>>> only)
>>
>> Can you describe the content of flag?
>>
>>> When constructing Dom0 in Xen, save these values. Then Dom0 could get
>>> them through hypercall HVMOP_get_param.
>>>
>>> 4. Map MMIO regions
>>> -------------------
>>> Register a bus_notifier for platform and amba bus in Linux. Add a new
>>> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the
>>> device is newly added, then call hypercall XENMEM_add_to_physmap to map
>>> the mmio regions.
>>>
>>> 5. Route device interrupts to Dom0
>>> ----------------------------------
>>> Route all the SPI interrupts to Dom0 before Dom0 booting.
>>
>> Not all the SPI will be routed to DOM0. Some are used by Xen and should
>> never be used by any guest. I have in mind the UART and SMMU interrupts.
>>
>> You will have to find away to skip them nicely. Note that not all the
>> IRQs used by Xen are properly registered when we build DOM0 (see the SMMU).
>>
> Just to clarify; Xen should map all SPIs which are not reserved for
> Xen to Dom0, right?

Right.

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Shannon Zhao]

Hi Julien,

On 2015/8/7 18:33, Julien Grall wrote:
> Hi Shannon,
> 
> Just some clarification questions.
> 
> On 07/08/15 03:11, Shannon Zhao wrote:
>> 3. Dom0 gets grant table and event channel irq information
>> -----------------------------------------------------------
>> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that
>> it runs on Xen hypervisor.
>>
>> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS
>> and HVM_PARAM_GNTTAB_SIZE.
>>
>> For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new
>> delivery type:
>> val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64
>> only)
> 
> Can you describe the content of flag?
> 

This needs definition as well. I think it could use the definition of
xenv table. Bit 0 stands interrupt mode and bit 1 stands interrupt
polarity. And explain it in the comment of HVM_PARAM_CALLBACK_IRQ.

>> When constructing Dom0 in Xen, save these values. Then Dom0 could get
>> them through hypercall HVMOP_get_param.
>>
>> 4. Map MMIO regions
>> -------------------
>> Register a bus_notifier for platform and amba bus in Linux. Add a new
>> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the
>> device is newly added, then call hypercall XENMEM_add_to_physmap to map
>> the mmio regions.
>>
>> 5. Route device interrupts to Dom0
>> ----------------------------------
>> Route all the SPI interrupts to Dom0 before Dom0 booting.
> 
> Not all the SPI will be routed to DOM0. Some are used by Xen and should
> never be used by any guest. I have in mind the UART and SMMU interrupts.
> 
> You will have to find away to skip them nicely. Note that not all the
> IRQs used by Xen are properly registered when we build DOM0 (see the SMMU).
> 
To uart, we can get the interrupt information from SPCR table and hide
it from Dom0.

IIUC, currently Xen (as well as Linux) doesn't support use SMMU when
booting with ACPI. When it supports, it could read the interrupts
information from IORT table and Hide them from Dom0.

-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

On 11/08/15 03:09, Shannon Zhao wrote:
> Hi Julien,

Hi Shannon,

> On 2015/8/7 18:33, Julien Grall wrote:
>> Hi Shannon,
>>
>> Just some clarification questions.
>>
>> On 07/08/15 03:11, Shannon Zhao wrote:
>>> 3. Dom0 gets grant table and event channel irq information
>>> -----------------------------------------------------------
>>> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that
>>> it runs on Xen hypervisor.
>>>
>>> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS
>>> and HVM_PARAM_GNTTAB_SIZE.
>>>
>>> For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new
>>> delivery type:
>>> val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64
>>> only)
>>
>> Can you describe the content of flag?
>>
> 
> This needs definition as well. I think it could use the definition of
> xenv table. Bit 0 stands interrupt mode and bit 1 stands interrupt
> polarity. And explain it in the comment of HVM_PARAM_CALLBACK_IRQ.

That would be fine for me.

>>> When constructing Dom0 in Xen, save these values. Then Dom0 could get
>>> them through hypercall HVMOP_get_param.
>>>
>>> 4. Map MMIO regions
>>> -------------------
>>> Register a bus_notifier for platform and amba bus in Linux. Add a new
>>> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the
>>> device is newly added, then call hypercall XENMEM_add_to_physmap to map
>>> the mmio regions.
>>>
>>> 5. Route device interrupts to Dom0
>>> ----------------------------------
>>> Route all the SPI interrupts to Dom0 before Dom0 booting.
>>
>> Not all the SPI will be routed to DOM0. Some are used by Xen and should
>> never be used by any guest. I have in mind the UART and SMMU interrupts.
>>
>> You will have to find away to skip them nicely. Note that not all the
>> IRQs used by Xen are properly registered when we build DOM0 (see the SMMU).
>>
> To uart, we can get the interrupt information from SPCR table and hide
> it from Dom0.

Can you clarify your meaning of "hide from DOM0"? Did you mean avoid to
route the SPI to DOM0?

> IIUC, currently Xen (as well as Linux) doesn't support use SMMU when
> booting with ACPI. When it supports, it could read the interrupts
> information from IORT table and Hide them from Dom0.

Well for Xen we don't even have ACPI supported upstream ;). For Linux
there is some on-going work. Anyway, this is not important right now.

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Shannon Zhao]

On 2015/8/11 17:46, Julien Grall wrote:
> On 11/08/15 03:09, Shannon Zhao wrote:
>> Hi Julien,
> 
> Hi Shannon,
> 
>> On 2015/8/7 18:33, Julien Grall wrote:
>>> Hi Shannon,
>>>
>>> Just some clarification questions.
>>>
>>> On 07/08/15 03:11, Shannon Zhao wrote:
>>>> 3. Dom0 gets grant table and event channel irq information
>>>> -----------------------------------------------------------
>>>> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that
>>>> it runs on Xen hypervisor.
>>>>
>>>> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS
>>>> and HVM_PARAM_GNTTAB_SIZE.
>>>>
>>>> For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new
>>>> delivery type:
>>>> val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64
>>>> only)
>>>
>>> Can you describe the content of flag?
>>>
>>
>> This needs definition as well. I think it could use the definition of
>> xenv table. Bit 0 stands interrupt mode and bit 1 stands interrupt
>> polarity. And explain it in the comment of HVM_PARAM_CALLBACK_IRQ.
> 
> That would be fine for me.
> 
>>>> When constructing Dom0 in Xen, save these values. Then Dom0 could get
>>>> them through hypercall HVMOP_get_param.
>>>>
>>>> 4. Map MMIO regions
>>>> -------------------
>>>> Register a bus_notifier for platform and amba bus in Linux. Add a new
>>>> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the
>>>> device is newly added, then call hypercall XENMEM_add_to_physmap to map
>>>> the mmio regions.
>>>>
>>>> 5. Route device interrupts to Dom0
>>>> ----------------------------------
>>>> Route all the SPI interrupts to Dom0 before Dom0 booting.
>>>
>>> Not all the SPI will be routed to DOM0. Some are used by Xen and should
>>> never be used by any guest. I have in mind the UART and SMMU interrupts.
>>>
>>> You will have to find away to skip them nicely. Note that not all the
>>> IRQs used by Xen are properly registered when we build DOM0 (see the SMMU).
>>>
>> To uart, we can get the interrupt information from SPCR table and hide
>> it from Dom0.
> 
> Can you clarify your meaning of "hide from DOM0"? Did you mean avoid to
> route the SPI to DOM0?
> 
Yes.

>> IIUC, currently Xen (as well as Linux) doesn't support use SMMU when
>> booting with ACPI. When it supports, it could read the interrupts
>> information from IORT table and Hide them from Dom0.
> 
> Well for Xen we don't even have ACPI supported upstream ;). For Linux
> there is some on-going work. Anyway, this is not important right now.
> 

Yeah, that could be done after this patchset upstream.

Thanks,
-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
> 
[...]
> 3. Dom0 gets grant table and event channel irq information
> -----------------------------------------------------------
> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that
> it runs on Xen hypervisor.
> 
> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS
> and HVM_PARAM_GNTTAB_SIZE.

The reason we expose this range is essentially to allow OS authors to take
a short cut by telling them about an IPA range which is unused, so it is
available for remapping the grant table into. On x86 there is a BAR on the
Xen platform PCI device which serves a similar purpose.

IIRC somebody (perhaps David V, CCd) had proposed at some point to make it
so that Linux was able to pick such an IPA itself by examining the memory
map or by some other scheme.

Any _if_ there was a viable proposal along those lines then we could use it
and avoid the need for these HVM params perhaps?

Ian.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
> This document is going to explain the design details of Xen booting with
> ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are
> welcome.

Some small subsets of this seem like they might overlap with what will be
required for PVH on x86 (a new x86 guest mode not dissimilar to the sole
ARM guest mode). If so then it would be preferable IMHO if PVH x86 could
use the same interfaces.

I've trimmed the quotes to just those bits and CCd some of the PVH people
(Boris and Roger[0]) in case they have any thoughts.

Actually, having done the trimming there is only one such bit:

[...]
> 4. Map MMIO regions
> -------------------
> Register a bus_notifier for platform and amba bus in Linux. Add a new
> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the
> device is newly added, then call hypercall XENMEM_add_to_physmap to map
> the mmio regions.

Ian.

[0] Roger is away for a week or so, but I'm expect feedback to be of the
"we could use one extra field" type rather than "this needs to be done some
totally different way for x86/PVH" (in which case we wouldn't want to share
the interface anyway I suppose) so need to block on awaiting that feedback.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Tue, 2015-08-11 at 15:19 +0100, Ian Campbell wrote:
> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
> > This document is going to explain the design details of Xen booting 
> > with
> > ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are
> > welcome.
> 
> Some small subsets of this seem like they might overlap with what will be
> required for PVH on x86 (a new x86 guest mode not dissimilar to the sole
> ARM guest mode). If so then it would be preferable IMHO if PVH x86 could
> use the same interfaces.
> 
> I've trimmed the quotes to just those bits and CCd some of the PVH people
> (Boris and Roger[0]) in case they have any thoughts.
> 
> Actually, having done the trimming there is only one such bit:
> 
> [...]
> > 4. Map MMIO regions
> > -------------------
> > Register a bus_notifier for platform and amba bus in Linux.

Previously PCI was included in this scheme, which is why I thought of PVH,
having missed that PCI wasn't mentioned here now.

Is it handled some other way now or is that just an accidental omission?

> >  Add a new
> > XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the
> > device is newly added, then call hypercall XENMEM_add_to_physmap to map
> > the mmio regions.
> 
> Ian.
> 
> [0] Roger is away for a week or so, but I'm expect feedback to be of the
> "we could use one extra field" type rather than "this needs to be done 
> some
> totally different way for x86/PVH" (in which case we wouldn't want to 
> share
> the interface anyway I suppose) so need to block on awaiting that 
> feedback.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[David Vrabel]

On 11/08/15 15:12, Ian Campbell wrote:
> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
>>
> [...]
>> 3. Dom0 gets grant table and event channel irq information
>> -----------------------------------------------------------
>> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that
>> it runs on Xen hypervisor.
>>
>> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS
>> and HVM_PARAM_GNTTAB_SIZE.
> 
> The reason we expose this range is essentially to allow OS authors to take
> a short cut by telling them about an IPA range which is unused, so it is
> available for remapping the grant table into. On x86 there is a BAR on the
> Xen platform PCI device which serves a similar purpose.
> 
> IIRC somebody (perhaps David V, CCd) had proposed at some point to make it
> so that Linux was able to pick such an IPA itself by examining the memory
> map or by some other scheme.

PVH in Linux uses ballooned pages which are vmap()'d into a virtually
contiguous region.

See xlated_setup_gnttab_pages().

David

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Tue, 2015-08-11 at 15:51 +0100, David Vrabel wrote:
> On 11/08/15 15:12, Ian Campbell wrote:
> > On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
> > > 
> > [...]
> > > 3. Dom0 gets grant table and event channel irq information
> > > -----------------------------------------------------------
> > > As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 
> > > that
> > > it runs on Xen hypervisor.
> > > 
> > > For grant table, add two new HVM_PARAMs: 
> > > HVM_PARAM_GNTTAB_START_ADDRESS
> > > and HVM_PARAM_GNTTAB_SIZE.
> > 
> > The reason we expose this range is essentially to allow OS authors to 
> > take
> > a short cut by telling them about an IPA range which is unused, so it 
> > is
> > available for remapping the grant table into. On x86 there is a BAR on 
> > the
> > Xen platform PCI device which serves a similar purpose.
> > 
> > IIRC somebody (perhaps David V, CCd) had proposed at some point to make 
> > it
> > so that Linux was able to pick such an IPA itself by examining the 
> > memory
> > map or by some other scheme.
> 
> PVH in Linux uses ballooned pages which are vmap()'d into a virtually
> contiguous region.
> 
> See xlated_setup_gnttab_pages().

So somewhat more concrete than a proposal then ;-)

I don't see anything there which would be a problem on ARM, so we should
probably go that route there too (at least for ACPI, if not globally for
all ARM guests).

Ian.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[David Vrabel]

On 11/08/15 15:59, Ian Campbell wrote:
> On Tue, 2015-08-11 at 15:51 +0100, David Vrabel wrote:
>> On 11/08/15 15:12, Ian Campbell wrote:
>>> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
>>>>
>>> [...]
>>>> 3. Dom0 gets grant table and event channel irq information
>>>> -----------------------------------------------------------
>>>> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 
>>>> that
>>>> it runs on Xen hypervisor.
>>>>
>>>> For grant table, add two new HVM_PARAMs: 
>>>> HVM_PARAM_GNTTAB_START_ADDRESS
>>>> and HVM_PARAM_GNTTAB_SIZE.
>>>
>>> The reason we expose this range is essentially to allow OS authors to 
>>> take
>>> a short cut by telling them about an IPA range which is unused, so it 
>>> is
>>> available for remapping the grant table into. On x86 there is a BAR on 
>>> the
>>> Xen platform PCI device which serves a similar purpose.
>>>
>>> IIRC somebody (perhaps David V, CCd) had proposed at some point to make 
>>> it
>>> so that Linux was able to pick such an IPA itself by examining the 
>>> memory
>>> map or by some other scheme.
>>
>> PVH in Linux uses ballooned pages which are vmap()'d into a virtually
>> contiguous region.
>>
>> See xlated_setup_gnttab_pages().
> 
> So somewhat more concrete than a proposal then ;-)
> 
> I don't see anything there which would be a problem on ARM, so we should
> probably go that route there too (at least for ACPI, if not globally for
> all ARM guests).

If someone does this please move xlated_setup_gnttab_pages() into
drivers/xen/xlate_mmu.c, and not copy it into an arm specific file.

David

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

On 11/08/15 15:59, Ian Campbell wrote:
> On Tue, 2015-08-11 at 15:51 +0100, David Vrabel wrote:
>> On 11/08/15 15:12, Ian Campbell wrote:
>>> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
>>>>
>>> [...]
>>>> 3. Dom0 gets grant table and event channel irq information
>>>> -----------------------------------------------------------
>>>> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 
>>>> that
>>>> it runs on Xen hypervisor.
>>>>
>>>> For grant table, add two new HVM_PARAMs: 
>>>> HVM_PARAM_GNTTAB_START_ADDRESS
>>>> and HVM_PARAM_GNTTAB_SIZE.
>>>
>>> The reason we expose this range is essentially to allow OS authors to 
>>> take
>>> a short cut by telling them about an IPA range which is unused, so it 
>>> is
>>> available for remapping the grant table into. On x86 there is a BAR on 
>>> the
>>> Xen platform PCI device which serves a similar purpose.
>>>
>>> IIRC somebody (perhaps David V, CCd) had proposed at some point to make 
>>> it
>>> so that Linux was able to pick such an IPA itself by examining the 
>>> memory
>>> map or by some other scheme.
>>
>> PVH in Linux uses ballooned pages which are vmap()'d into a virtually
>> contiguous region.
>>
>> See xlated_setup_gnttab_pages().
> 
> So somewhat more concrete than a proposal then ;-)
> 
> I don't see anything there which would be a problem on ARM, so we should
> probably go that route there too (at least for ACPI, if not globally for
> all ARM guests).

IIRC we talked about it few months ago and you said that using balloon
page will split in 4K the 1G/2M mapping done in the stage-2 p2m.

Note that for DOM0 we never give back the ballooned page to Xen. This is
because we have to respect the 1:1 mapping when DOM0 want to get back
the page.

Regards,

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Tue, 2015-08-11 at 16:11 +0100, Julien Grall wrote:
> On 11/08/15 15:59, Ian Campbell wrote:
> > On Tue, 2015-08-11 at 15:51 +0100, David Vrabel wrote:
> > > On 11/08/15 15:12, Ian Campbell wrote:
> > > > On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
> > > > > 
> > > > [...]
> > > > > 3. Dom0 gets grant table and event channel irq information
> > > > > -----------------------------------------------------------
> > > > > As said above, we assign the hypervisor_id be "XenVMM" to tell 
> > > > > Dom0 
> > > > > that
> > > > > it runs on Xen hypervisor.
> > > > > 
> > > > > For grant table, add two new HVM_PARAMs: 
> > > > > HVM_PARAM_GNTTAB_START_ADDRESS
> > > > > and HVM_PARAM_GNTTAB_SIZE.
> > > > 
> > > > The reason we expose this range is essentially to allow OS authors 
> > > > to 
> > > > take
> > > > a short cut by telling them about an IPA range which is unused, so 
> > > > it 
> > > > is
> > > > available for remapping the grant table into. On x86 there is a BAR 
> > > > on 
> > > > the
> > > > Xen platform PCI device which serves a similar purpose.
> > > > 
> > > > IIRC somebody (perhaps David V, CCd) had proposed at some point to 
> > > > make 
> > > > it
> > > > so that Linux was able to pick such an IPA itself by examining the 
> > > > memory
> > > > map or by some other scheme.
> > > 
> > > PVH in Linux uses ballooned pages which are vmap()'d into a virtually
> > > contiguous region.
> > > 
> > > See xlated_setup_gnttab_pages().
> > 
> > So somewhat more concrete than a proposal then ;-)
> > 
> > I don't see anything there which would be a problem on ARM, so we 
> > should
> > probably go that route there too (at least for ACPI, if not globally 
> > for
> > all ARM guests).
> 
> IIRC we talked about it few months ago and you said that using balloon
> page will split in 4K the 1G/2M mapping done in the stage-2 p2m.

Did I? Odd because I'm also of the opinion that alloc_ballooned_pages
should operate in chunks of 2M at the hypercall layer and keep any
resulting spare 4K pages on a free list to use for future such allocations.

IOW it should avoid such shattering where it can.

> Note that for DOM0 we never give back the ballooned page to Xen. This is
> because we have to respect the 1:1 mapping when DOM0 want to get back
> the page.
> 
> Regards,
> 

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[David Vrabel]

On 11/08/15 16:19, Ian Campbell wrote:
> On Tue, 2015-08-11 at 16:11 +0100, Julien Grall wrote:
>> On 11/08/15 15:59, Ian Campbell wrote:
>>> On Tue, 2015-08-11 at 15:51 +0100, David Vrabel wrote:
>>>> On 11/08/15 15:12, Ian Campbell wrote:
>>>>> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
>>>>>>
>>>>> [...]
>>>>>> 3. Dom0 gets grant table and event channel irq information
>>>>>> -----------------------------------------------------------
>>>>>> As said above, we assign the hypervisor_id be "XenVMM" to tell 
>>>>>> Dom0 
>>>>>> that
>>>>>> it runs on Xen hypervisor.
>>>>>>
>>>>>> For grant table, add two new HVM_PARAMs: 
>>>>>> HVM_PARAM_GNTTAB_START_ADDRESS
>>>>>> and HVM_PARAM_GNTTAB_SIZE.
>>>>>
>>>>> The reason we expose this range is essentially to allow OS authors 
>>>>> to 
>>>>> take
>>>>> a short cut by telling them about an IPA range which is unused, so 
>>>>> it 
>>>>> is
>>>>> available for remapping the grant table into. On x86 there is a BAR 
>>>>> on 
>>>>> the
>>>>> Xen platform PCI device which serves a similar purpose.
>>>>>
>>>>> IIRC somebody (perhaps David V, CCd) had proposed at some point to 
>>>>> make 
>>>>> it
>>>>> so that Linux was able to pick such an IPA itself by examining the 
>>>>> memory
>>>>> map or by some other scheme.
>>>>
>>>> PVH in Linux uses ballooned pages which are vmap()'d into a virtually
>>>> contiguous region.
>>>>
>>>> See xlated_setup_gnttab_pages().
>>>
>>> So somewhat more concrete than a proposal then ;-)
>>>
>>> I don't see anything there which would be a problem on ARM, so we 
>>> should
>>> probably go that route there too (at least for ACPI, if not globally 
>>> for
>>> all ARM guests).
>>
>> IIRC we talked about it few months ago and you said that using balloon
>> page will split in 4K the 1G/2M mapping done in the stage-2 p2m.
> 
> Did I? Odd because I'm also of the opinion that alloc_ballooned_pages
> should operate in chunks of 2M at the hypercall layer and keep any
> resulting spare 4K pages on a free list to use for future such allocations.
> 
> IOW it should avoid such shattering where it can.

You can also (soon) enable memory hotplug and get to hotplug new (empty)
memory sections to avoid having to release any frames back to Xen.

David

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Boris Ostrovsky]

On 08/11/2015 10:21 AM, Ian Campbell wrote:
> On Tue, 2015-08-11 at 15:19 +0100, Ian Campbell wrote:
>> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
>>> This document is going to explain the design details of Xen booting
>>> with
>>> ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are
>>> welcome.
>> Some small subsets of this seem like they might overlap with what will be
>> required for PVH on x86 (a new x86 guest mode not dissimilar to the sole
>> ARM guest mode). If so then it would be preferable IMHO if PVH x86 could
>> use the same interfaces.
>>
>> I've trimmed the quotes to just those bits and CCd some of the PVH people
>> (Boris and Roger[0]) in case they have any thoughts.
>>
>> Actually, having done the trimming there is only one such bit:
>>
>> [...]
>>> 4. Map MMIO regions
>>> -------------------
>>> Register a bus_notifier for platform and amba bus in Linux.
> Previously PCI was included in this scheme, which is why I thought of PVH,
> having missed that PCI wasn't mentioned here now.
>
> Is it handled some other way now or is that just an accidental omission?

PCI already has a bus notifier which is probably why it's not explicitly 
called out here.

-borsi

>
>>>   Add a new
>>> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the
>>> device is newly added, then call hypercall XENMEM_add_to_physmap to map
>>> the mmio regions.
>> Ian.
>>
>> [0] Roger is away for a week or so, but I'm expect feedback to be of the
>> "we could use one extra field" type rather than "this needs to be done
>> some
>> totally different way for x86/PVH" (in which case we wouldn't want to
>> share
>> the interface anyway I suppose) so need to block on awaiting that
>> feedback.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Tue, 2015-08-11 at 11:29 -0400, Boris Ostrovsky wrote:
> On 08/11/2015 10:21 AM, Ian Campbell wrote:
> > On Tue, 2015-08-11 at 15:19 +0100, Ian Campbell wrote:
> > > On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
> > > > This document is going to explain the design details of Xen booting
> > > > with
> > > > ACPI on ARM. Maybe parts of it may not be appropriate. Any comments 
> > > > are
> > > > welcome.
> > > Some small subsets of this seem like they might overlap with what 
> > > will be
> > > required for PVH on x86 (a new x86 guest mode not dissimilar to the 
> > > sole
> > > ARM guest mode). If so then it would be preferable IMHO if PVH x86 
> > > could
> > > use the same interfaces.
> > > 
> > > I've trimmed the quotes to just those bits and CCd some of the PVH 
> > > people
> > > (Boris and Roger[0]) in case they have any thoughts.
> > > 
> > > Actually, having done the trimming there is only one such bit:
> > > 
> > > [...]
> > > > 4. Map MMIO regions
> > > > -------------------
> > > > Register a bus_notifier for platform and amba bus in Linux.
> > Previously PCI was included in this scheme, which is why I thought of 
> > PVH,
> > having missed that PCI wasn't mentioned here now.
> > 
> > Is it handled some other way now or is that just an accidental 
> > omission?
> 
> PCI already has a bus notifier which is probably why it's not explicitly 
> called out here.

Right, but I was more specifically thinking of the bit which followed
regarding the use of the notifier to map the MMIO, which is the more
important bit.

> 
> > > >   Add a new
> > > > XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if 
> > > > the
> > > > device is newly added, then call hypercall XENMEM_add_to_physmap to 
> > > > map
> > > > the mmio regions.
> > > Ian.
> > > 
> > > [0] Roger is away for a week or so, but I'm expect feedback to be of 
> > > the
> > > "we could use one extra field" type rather than "this needs to be 
> > > done
> > > some
> > > totally different way for x86/PVH" (in which case we wouldn't want to
> > > share
> > > the interface anyway I suppose) so need to block on awaiting that
> > > feedback.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Boris Ostrovsky]

On 08/11/2015 11:35 AM, Ian Campbell wrote:
> On Tue, 2015-08-11 at 11:29 -0400, Boris Ostrovsky wrote:
>> On 08/11/2015 10:21 AM, Ian Campbell wrote:
>>> On Tue, 2015-08-11 at 15:19 +0100, Ian Campbell wrote:
>>>> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
>>>>> This document is going to explain the design details of Xen booting
>>>>> with
>>>>> ACPI on ARM. Maybe parts of it may not be appropriate. Any comments
>>>>> are
>>>>> welcome.
>>>> Some small subsets of this seem like they might overlap with what
>>>> will be
>>>> required for PVH on x86 (a new x86 guest mode not dissimilar to the
>>>> sole
>>>> ARM guest mode). If so then it would be preferable IMHO if PVH x86
>>>> could
>>>> use the same interfaces.
>>>>
>>>> I've trimmed the quotes to just those bits and CCd some of the PVH
>>>> people
>>>> (Boris and Roger[0]) in case they have any thoughts.
>>>>
>>>> Actually, having done the trimming there is only one such bit:
>>>>
>>>> [...]
>>>>> 4. Map MMIO regions
>>>>> -------------------
>>>>> Register a bus_notifier for platform and amba bus in Linux.
>>> Previously PCI was included in this scheme, which is why I thought of
>>> PVH,
>>> having missed that PCI wasn't mentioned here now.
>>>
>>> Is it handled some other way now or is that just an accidental
>>> omission?
>> PCI already has a bus notifier which is probably why it's not explicitly
>> called out here.
> Right, but I was more specifically thinking of the bit which followed
> regarding the use of the notifier to map the MMIO, which is the more
> important bit.

This notifier calls PHYSDEVOP_pci_device_add and as far as I can tell 
MMIO is mapped there (for IOMMU, which IIUIC is the main issue here).

-boris

>
>>>>>    Add a new
>>>>> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if
>>>>> the
>>>>> device is newly added, then call hypercall XENMEM_add_to_physmap to
>>>>> map
>>>>> the mmio regions.
>>>> Ian.
>>>>
>>>> [0] Roger is away for a week or so, but I'm expect feedback to be of
>>>> the
>>>> "we could use one extra field" type rather than "this needs to be
>>>> done
>>>> some
>>>> totally different way for x86/PVH" (in which case we wouldn't want to
>>>> share
>>>> the interface anyway I suppose) so need to block on awaiting that
>>>> feedback.
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

On 11/08/15 16:19, Ian Campbell wrote:
>> IIRC we talked about it few months ago and you said that using balloon
>> page will split in 4K the 1G/2M mapping done in the stage-2 p2m.
> 
> Did I? Odd because I'm also of the opinion that alloc_ballooned_pages
> should operate in chunks of 2M at the hypercall layer and keep any
> resulting spare 4K pages on a free list to use for future such allocations.

That from what I recall from an IRL talk.

Anyway, I've looked in my archive to see why we decided to keep the
grant table parameters (in Xen ACPI table at this point). We were not
sure that the domain as all the key in hand in order to find memory hole.

I think it's quite important to not think only about Linux but all other
Operating Systems. If we ever require a parameters later, it would mean
that the OS won't be able to run as DOM0 on older Xen.

Linux is using ballooned page, which means loosing ~128KB (default of
the grant table on ARM) of memory because we never give back the page to
Xen due the 1:1 mapping. Although I guess this is not a big deal as it's
quite small and Linux, as said by David, will support memory hotplug soon.

FreeBSD is using memory hole in the address space so there is no issue here.

So I guess we could skip this parameters as 128KB doesn't seem to be a
big deal.

> IOW it should avoid such shattering where it can.

That would work too.

Regards,

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

Hi Shannon,

On 07/08/15 03:11, Shannon Zhao wrote:
> 2. Create minimal DT to pass required information to Dom0
> ----------------------------------------------------------
> The minimal DT mainly passes Dom0 bootargs, address and size of initrd
> (if available), address and size of uefi system table, address and size
> of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver.
> 
> An example of the minimal DT:
> / {
>     #address-cells = <2>;
>     #size-cells = <1>;
>     chosen {
>         bootargs = "kernel=Image console=hvc0 earlycon=pl011,0x1c090000
> root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force";
>         linux,initrd-start = <0xXXXXXXXX>;
>         linux,initrd-end = <0xXXXXXXXX>;
>         linux,uefi-system-table = <0xXXXXXXXX>;
>         linux,uefi-mmap-start = <0xXXXXXXXX>;
>         linux,uefi-mmap-size = <0xXXXXXXXX>;
>         linux,uefi-mmap-desc-size = <0xXXXXXXXX>;
>         linux,uefi-mmap-desc-ver = <0xXXXXXXXX>;
>     };
> };
> 
> For details loook at
> https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi.txt

AFAICT, the device tree properties in this documentation are only used
in order to communicate between the UEFI stub and Linux.

This means that those properties are not standardize and can change at
any time by Linux folks. They don't even live in Documentation/devicetree/

I would also expect to see the same needs for FreeBSD running as DOM0
with ACPI.

So it looks like to me that a generic name would be better for all those
properties.

Regards,

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Shannon Zhao]

On 2015/8/12 0:01, Julien Grall wrote:
> On 11/08/15 16:19, Ian Campbell wrote:
>>> IIRC we talked about it few months ago and you said that using balloon
>>> page will split in 4K the 1G/2M mapping done in the stage-2 p2m.
>>
>> Did I? Odd because I'm also of the opinion that alloc_ballooned_pages
>> should operate in chunks of 2M at the hypercall layer and keep any
>> resulting spare 4K pages on a free list to use for future such allocations.
> 
> That from what I recall from an IRL talk.
> 
> Anyway, I've looked in my archive to see why we decided to keep the
> grant table parameters (in Xen ACPI table at this point). We were not
> sure that the domain as all the key in hand in order to find memory hole.
> 
> I think it's quite important to not think only about Linux but all other
> Operating Systems. If we ever require a parameters later, it would mean
> that the OS won't be able to run as DOM0 on older Xen.
> 
> Linux is using ballooned page, which means loosing ~128KB (default of
> the grant table on ARM) of memory because we never give back the page to
> Xen due the 1:1 mapping. Although I guess this is not a big deal as it's
> quite small and Linux, as said by David, will support memory hotplug soon.
> 
> FreeBSD is using memory hole in the address space so there is no issue here.
> 
> So I guess we could skip this parameters as 128KB doesn't seem to be a
> big deal.
> 
>> IOW it should avoid such shattering where it can.
> 
> That would work too.
> 

So use xlated_setup_gnttab_pages for both DT and ACPI?

-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Shannon Zhao]

On 2015/8/11 23:52, Boris Ostrovsky wrote:
> On 08/11/2015 11:35 AM, Ian Campbell wrote:
>> On Tue, 2015-08-11 at 11:29 -0400, Boris Ostrovsky wrote:
>>> On 08/11/2015 10:21 AM, Ian Campbell wrote:
>>>> On Tue, 2015-08-11 at 15:19 +0100, Ian Campbell wrote:
>>>>> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
>>>>>> This document is going to explain the design details of Xen booting
>>>>>> with
>>>>>> ACPI on ARM. Maybe parts of it may not be appropriate. Any comments
>>>>>> are
>>>>>> welcome.
>>>>> Some small subsets of this seem like they might overlap with what
>>>>> will be
>>>>> required for PVH on x86 (a new x86 guest mode not dissimilar to the
>>>>> sole
>>>>> ARM guest mode). If so then it would be preferable IMHO if PVH x86
>>>>> could
>>>>> use the same interfaces.
>>>>>
>>>>> I've trimmed the quotes to just those bits and CCd some of the PVH
>>>>> people
>>>>> (Boris and Roger[0]) in case they have any thoughts.
>>>>>
>>>>> Actually, having done the trimming there is only one such bit:
>>>>>
>>>>> [...]
>>>>>> 4. Map MMIO regions
>>>>>> -------------------
>>>>>> Register a bus_notifier for platform and amba bus in Linux.
>>>> Previously PCI was included in this scheme, which is why I thought of
>>>> PVH,
>>>> having missed that PCI wasn't mentioned here now.
>>>>
>>>> Is it handled some other way now or is that just an accidental
>>>> omission?
>>> PCI already has a bus notifier which is probably why it's not explicitly
>>> called out here.
>> Right, but I was more specifically thinking of the bit which followed
>> regarding the use of the notifier to map the MMIO, which is the more
>> important bit.
> 
> This notifier calls PHYSDEVOP_pci_device_add and as far as I can tell
> MMIO is mapped there (for IOMMU, which IIUIC is the main issue here).

Right, at this moment we only add platform and amba bus bus_notifier.
The PCI device can reuse existing bus_notifier.

-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Shannon Zhao]

Hi Julien,

On 2015/8/12 0:19, Julien Grall wrote:
> Hi Shannon,
> 
> On 07/08/15 03:11, Shannon Zhao wrote:
>> 2. Create minimal DT to pass required information to Dom0
>> ----------------------------------------------------------
>> The minimal DT mainly passes Dom0 bootargs, address and size of initrd
>> (if available), address and size of uefi system table, address and size
>> of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver.
>>
>> An example of the minimal DT:
>> / {
>>     #address-cells = <2>;
>>     #size-cells = <1>;
>>     chosen {
>>         bootargs = "kernel=Image console=hvc0 earlycon=pl011,0x1c090000
>> root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force";
>>         linux,initrd-start = <0xXXXXXXXX>;
>>         linux,initrd-end = <0xXXXXXXXX>;
>>         linux,uefi-system-table = <0xXXXXXXXX>;
>>         linux,uefi-mmap-start = <0xXXXXXXXX>;
>>         linux,uefi-mmap-size = <0xXXXXXXXX>;
>>         linux,uefi-mmap-desc-size = <0xXXXXXXXX>;
>>         linux,uefi-mmap-desc-ver = <0xXXXXXXXX>;
>>     };
>> };
>>
>> For details loook at
>> https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi.txt
> 
> AFAICT, the device tree properties in this documentation are only used
> in order to communicate between the UEFI stub and Linux.
> 
> This means that those properties are not standardize and can change at
> any time by Linux folks. They don't even live in Documentation/devicetree/
> 
> I would also expect to see the same needs for FreeBSD running as DOM0
> with ACPI.
>
I'm not very clear about how FreeBSD communicates with UEFI. And when
booting with DT, how does FreeBSD communicate with UEFI? Not through
these properties?

> So it looks like to me that a generic name would be better for all those
> properties.
> 
If we change these name, it needs change some functions in Linux. Will
it impact the use of Linux with UEFI not on Xen?

Thanks,
-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Shannon Zhao]

Hi,

I'm working on re-spinning this patchset while encountering a werid
problem about xzalloc_bytes.

Since I need to copy some ACPI tables, I need to allocate some memory
for it. So there are a few places calling xzalloc_bytes. And it fails at
the fifth one. The log is shown as following:

(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading kernel from boot module @ 00000008fa315000
(XEN) Allocating 1:1 mappings totalling 256MB for dom0:
(XEN) BANK[0] 0x00000090000000-0x000000a0000000 (256MB)
(XEN) Grant table range: 0x00000087e00000-0x00000087e5b000
(XEN) Loading zImage from 00000008fa315000 to
0000000090080000-00000000909e0ec8
(XEN) Hypervisor Trap. HSR=0x96000044 EC=0x25 IL=1 Syndrome=0x44
(XEN) CPU0: Unexpected Trap: Hypervisor
(XEN) ----[ Xen-4.6-unstable  arm64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) PC:     0000000000238b78 xmem_pool_alloc+0x348/0x4b0
(XEN) LR:     0000000000238960
(XEN) SP:     00000000002bf4e0
(XEN) CPSR:   20000249 MODE:64-bit EL2h (Hypervisor, handler)

(XEN) Xen call trace:
(XEN)    [<0000000000238b78>] xmem_pool_alloc+0x348/0x4b0 (PC)
(XEN)    [<0000000000238960>] xmem_pool_alloc+0x130/0x4b0 (LR)
(XEN)    [<0000000000239100>] _xmalloc+0xf4/0x290
(XEN)    [<00000000002392b0>] _xzalloc+0x14/0x38
(XEN)    [<0000000000245430>] construct_dom0+0xbc0/0x14cc
(XEN)    [<000000000028c4c4>] start_xen+0xbf4/0xcb0
(XEN)    [<000000000020060c>] paging+0x84/0xbc
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) CPU0: Unexpected Trap: Hypervisor
(XEN)
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Tue, 2015-08-11 at 17:01 +0100, Julien Grall wrote:
> On 11/08/15 16:19, Ian Campbell wrote:
> > > IIRC we talked about it few months ago and you said that using 
> > > balloon
> > > page will split in 4K the 1G/2M mapping done in the stage-2 p2m.
> > 
> > Did I? Odd because I'm also of the opinion that alloc_ballooned_pages
> > should operate in chunks of 2M at the hypercall layer and keep any
> > resulting spare 4K pages on a free list to use for future such 
> > allocations.
> 
> That from what I recall from an IRL talk.
> 
> Anyway, I've looked in my archive to see why we decided to keep the
> grant table parameters (in Xen ACPI table at this point). We were not
> sure that the domain as all the key in hand in order to find memory hole.
> 
> I think it's quite important to not think only about Linux but all other
> Operating Systems. If we ever require a parameters later, it would mean
> that the OS won't be able to run as DOM0 on older Xen.
> 
> Linux is using ballooned page, which means loosing ~128KB

It's not "loosing" anything. It is _using_ 128KB for a legitimate data
structure needed to support the platform (the fact that it happens to not
really be using the underlying RAM is irrelevant here).

It's not really inherently any different to "loosing" memory in order to
allocate e.g. page tables.

Ian.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Wed, 2015-08-12 at 10:42 +0800, Shannon Zhao wrote:
> 
> On 2015/8/12 0:01, Julien Grall wrote:
> > On 11/08/15 16:19, Ian Campbell wrote:
> > > > IIRC we talked about it few months ago and you said that using 
> > > > balloon
> > > > page will split in 4K the 1G/2M mapping done in the stage-2 p2m.
> > > 
> > > Did I? Odd because I'm also of the opinion that alloc_ballooned_pages
> > > should operate in chunks of 2M at the hypercall layer and keep any
> > > resulting spare 4K pages on a free list to use for future such 
> > > allocations.
> > 
> > That from what I recall from an IRL talk.
> > 
> > Anyway, I've looked in my archive to see why we decided to keep the
> > grant table parameters (in Xen ACPI table at this point). We were not
> > sure that the domain as all the key in hand in order to find memory 
> > hole.
> > 
> > I think it's quite important to not think only about Linux but all 
> > other
> > Operating Systems. If we ever require a parameters later, it would mean
> > that the OS won't be able to run as DOM0 on older Xen.
> > 
> > Linux is using ballooned page, which means loosing ~128KB (default of
> > the grant table on ARM) of memory because we never give back the page 
> > to
> > Xen due the 1:1 mapping. Although I guess this is not a big deal as 
> > it's
> > quite small and Linux, as said by David, will support memory hotplug 
> > soon.
> > 
> > FreeBSD is using memory hole in the address space so there is no issue 
> > here.
> > 
> > So I guess we could skip this parameters as 128KB doesn't seem to be a
> > big deal.
> > 
> > > IOW it should avoid such shattering where it can.
> > 
> > That would work too.
> > 
> 
> So use xlated_setup_gnttab_pages for both DT and ACPI?

I think we might as well, yes.

Ian.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Wed, 2015-08-12 at 10:47 +0800, Shannon Zhao wrote:
> 
> On 2015/8/11 23:52, Boris Ostrovsky wrote:
> > On 08/11/2015 11:35 AM, Ian Campbell wrote:
> > > On Tue, 2015-08-11 at 11:29 -0400, Boris Ostrovsky wrote:
> > > > On 08/11/2015 10:21 AM, Ian Campbell wrote:
> > > > > On Tue, 2015-08-11 at 15:19 +0100, Ian Campbell wrote:
> > > > > > On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
> > > > > > > This document is going to explain the design details of Xen 
> > > > > > > booting
> > > > > > > with
> > > > > > > ACPI on ARM. Maybe parts of it may not be appropriate. Any 
> > > > > > > comments
> > > > > > > are
> > > > > > > welcome.
> > > > > > Some small subsets of this seem like they might overlap with 
> > > > > > what
> > > > > > will be
> > > > > > required for PVH on x86 (a new x86 guest mode not dissimilar to 
> > > > > > the
> > > > > > sole
> > > > > > ARM guest mode). If so then it would be preferable IMHO if PVH 
> > > > > > x86
> > > > > > could
> > > > > > use the same interfaces.
> > > > > > 
> > > > > > I've trimmed the quotes to just those bits and CCd some of the 
> > > > > > PVH
> > > > > > people
> > > > > > (Boris and Roger[0]) in case they have any thoughts.
> > > > > > 
> > > > > > Actually, having done the trimming there is only one such bit:
> > > > > > 
> > > > > > [...]
> > > > > > > 4. Map MMIO regions
> > > > > > > -------------------
> > > > > > > Register a bus_notifier for platform and amba bus in Linux.
> > > > > Previously PCI was included in this scheme, which is why I 
> > > > > thought of
> > > > > PVH,
> > > > > having missed that PCI wasn't mentioned here now.
> > > > > 
> > > > > Is it handled some other way now or is that just an accidental
> > > > > omission?
> > > > PCI already has a bus notifier which is probably why it's not 
> > > > explicitly
> > > > called out here.
> > > Right, but I was more specifically thinking of the bit which followed
> > > regarding the use of the notifier to map the MMIO, which is the more
> > > important bit.
> > 
> > This notifier calls PHYSDEVOP_pci_device_add and as far as I can tell
> > MMIO is mapped there (for IOMMU, which IIUIC is the main issue here).
> 
> Right, at this moment we only add platform and amba bus bus_notifier.
> The PCI device can reuse existing bus_notifier.

Super. Perhaps add a sentence to that effect so I remember that this
already exists next time?

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Wed, 2015-08-12 at 11:04 +0800, Shannon Zhao wrote:
> Hi Julien,
> 
> On 2015/8/12 0:19, Julien Grall wrote:
> > Hi Shannon,
> > 
> > On 07/08/15 03:11, Shannon Zhao wrote:
> > > 2. Create minimal DT to pass required information to Dom0
> > > ----------------------------------------------------------
> > > The minimal DT mainly passes Dom0 bootargs, address and size of 
> > > initrd
> > > (if available), address and size of uefi system table, address and 
> > > size
> > > of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver.
> > > 
> > > An example of the minimal DT:
> > > / {
> > >     #address-cells = <2>;
> > >     #size-cells = <1>;
> > >     chosen {
> > >         bootargs = "kernel=Image console=hvc0 
> > > earlycon=pl011,0x1c090000
> > > root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force";
> > >         linux,initrd-start = <0xXXXXXXXX>;
> > >         linux,initrd-end = <0xXXXXXXXX>;
> > >         linux,uefi-system-table = <0xXXXXXXXX>;
> > >         linux,uefi-mmap-start = <0xXXXXXXXX>;
> > >         linux,uefi-mmap-size = <0xXXXXXXXX>;
> > >         linux,uefi-mmap-desc-size = <0xXXXXXXXX>;
> > >         linux,uefi-mmap-desc-ver = <0xXXXXXXXX>;
> > >     };
> > > };
> > > 
> > > For details loook at
> > > https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi.
> > > txt
> > 
> > AFAICT, the device tree properties in this documentation are only used
> > in order to communicate between the UEFI stub and Linux.
> > 
> > This means that those properties are not standardize and can change at
> > any time by Linux folks. They don't even live in 
> > Documentation/devicetree/
> > 
> > I would also expect to see the same needs for FreeBSD running as DOM0
> > with ACPI.
> > 
> I'm not very clear about how FreeBSD communicates with UEFI. And when
> booting with DT, how does FreeBSD communicate with UEFI? Not through
> these properties?

These properties are in effect a Linux internal interface defined between
the "Linux UEFI stub" and the "Linux kernel proper". The stub and the
kernel are notionally separate entities, although they are in the same tree
etc there is a well defined transition/entry point between the two. Since
they are in the same tree even though they are in theory "separate" I
expect they will tend to co-evolve.

IIRC we discussed with some of the maintainers (at Connect?) making this a
more formal interface, i.e. exposing the entry point to "Linux kernel
proper" which understands these properties to other than just the "Linux
UEFI stub" specifically to external entities such as Xen.

Probably part of this work needs to formalise that, such as by moving this
binding into the proper external bindings dir.

At which point BSD can (hopefully!) choose to support the same interface.

Ian.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Shannon Zhao]

On 2015/8/12 16:47, Ian Campbell wrote:
> On Wed, 2015-08-12 at 10:47 +0800, Shannon Zhao wrote:
>>
>> On 2015/8/11 23:52, Boris Ostrovsky wrote:
>>> On 08/11/2015 11:35 AM, Ian Campbell wrote:
>>>> On Tue, 2015-08-11 at 11:29 -0400, Boris Ostrovsky wrote:
>>>>> On 08/11/2015 10:21 AM, Ian Campbell wrote:
>>>>>> On Tue, 2015-08-11 at 15:19 +0100, Ian Campbell wrote:
>>>>>>> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
>>>>>>>> This document is going to explain the design details of Xen 
>>>>>>>> booting
>>>>>>>> with
>>>>>>>> ACPI on ARM. Maybe parts of it may not be appropriate. Any 
>>>>>>>> comments
>>>>>>>> are
>>>>>>>> welcome.
>>>>>>> Some small subsets of this seem like they might overlap with 
>>>>>>> what
>>>>>>> will be
>>>>>>> required for PVH on x86 (a new x86 guest mode not dissimilar to 
>>>>>>> the
>>>>>>> sole
>>>>>>> ARM guest mode). If so then it would be preferable IMHO if PVH 
>>>>>>> x86
>>>>>>> could
>>>>>>> use the same interfaces.
>>>>>>>
>>>>>>> I've trimmed the quotes to just those bits and CCd some of the 
>>>>>>> PVH
>>>>>>> people
>>>>>>> (Boris and Roger[0]) in case they have any thoughts.
>>>>>>>
>>>>>>> Actually, having done the trimming there is only one such bit:
>>>>>>>
>>>>>>> [...]
>>>>>>>> 4. Map MMIO regions
>>>>>>>> -------------------
>>>>>>>> Register a bus_notifier for platform and amba bus in Linux.
>>>>>> Previously PCI was included in this scheme, which is why I 
>>>>>> thought of
>>>>>> PVH,
>>>>>> having missed that PCI wasn't mentioned here now.
>>>>>>
>>>>>> Is it handled some other way now or is that just an accidental
>>>>>> omission?
>>>>> PCI already has a bus notifier which is probably why it's not 
>>>>> explicitly
>>>>> called out here.
>>>> Right, but I was more specifically thinking of the bit which followed
>>>> regarding the use of the notifier to map the MMIO, which is the more
>>>> important bit.
>>>
>>> This notifier calls PHYSDEVOP_pci_device_add and as far as I can tell
>>> MMIO is mapped there (for IOMMU, which IIUIC is the main issue here).
>>
>> Right, at this moment we only add platform and amba bus bus_notifier.
>> The PCI device can reuse existing bus_notifier.
> 
> Super. Perhaps add a sentence to that effect so I remember that this
> already exists next time?
> 
Ok. Will add.

-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

On 12/08/2015 09:46, Ian Campbell wrote:
> On Tue, 2015-08-11 at 17:01 +0100, Julien Grall wrote:
>> On 11/08/15 16:19, Ian Campbell wrote:
>>>> IIRC we talked about it few months ago and you said that using
>>>> balloon
>>>> page will split in 4K the 1G/2M mapping done in the stage-2 p2m.
>>>
>>> Did I? Odd because I'm also of the opinion that alloc_ballooned_pages
>>> should operate in chunks of 2M at the hypercall layer and keep any
>>> resulting spare 4K pages on a free list to use for future such
>>> allocations.
>>
>> That from what I recall from an IRL talk.
>>
>> Anyway, I've looked in my archive to see why we decided to keep the
>> grant table parameters (in Xen ACPI table at this point). We were not
>> sure that the domain as all the key in hand in order to find memory hole.
>>
>> I think it's quite important to not think only about Linux but all other
>> Operating Systems. If we ever require a parameters later, it would mean
>> that the OS won't be able to run as DOM0 on older Xen.
>>
>> Linux is using ballooned page, which means loosing ~128KB
>
> It's not "loosing" anything. It is _using_ 128KB for a legitimate data
> structure needed to support the platform (the fact that it happens to not
> really be using the underlying RAM is irrelevant here).
>
> It's not really inherently any different to "loosing" memory in order to
> allocate e.g. page tables.

If you read my sentence until the end, you would have noticed why I say 
"loosing". It's because we never give back this memory to Xen due to the 
1:1 mapping...

The grant table are already allocated in Xen separately which means 
doubling the size of memory used for DOM0 grant table.

Anyway, that's small amount of data and can optimized later using the 
hotplug.

Cheers,

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

On 12/08/2015 08:22, Shannon Zhao wrote:
> Hi,

Hi Shannon,

It's not part of the design discussion and we are avoiding to mix 
discussion. Can you please create another thread (or at least renaming 
the subject)?

> I'm working on re-spinning this patchset while encountering a werid
> problem about xzalloc_bytes.
>
> Since I need to copy some ACPI tables, I need to allocate some memory
> for it. So there are a few places calling xzalloc_bytes. And it fails at
> the fifth one. The log is shown as following:

Do you copy data in the newly allocated memory between 2 xzalloc_bytes?

The only thing I have in mind based on the log below is your are 
overriding the metadata of the memory allocator.

> (XEN) *** LOADING DOMAIN 0 ***
> (XEN) Loading kernel from boot module @ 00000008fa315000
> (XEN) Allocating 1:1 mappings totalling 256MB for dom0:
> (XEN) BANK[0] 0x00000090000000-0x000000a0000000 (256MB)
> (XEN) Grant table range: 0x00000087e00000-0x00000087e5b000
> (XEN) Loading zImage from 00000008fa315000 to
> 0000000090080000-00000000909e0ec8
> (XEN) Hypervisor Trap. HSR=0x96000044 EC=0x25 IL=1 Syndrome=0x44
> (XEN) CPU0: Unexpected Trap: Hypervisor
> (XEN) ----[ Xen-4.6-unstable  arm64  debug=y  Not tainted ]----
> (XEN) CPU:    0
> (XEN) PC:     0000000000238b78 xmem_pool_alloc+0x348/0x4b0
> (XEN) LR:     0000000000238960
> (XEN) SP:     00000000002bf4e0
> (XEN) CPSR:   20000249 MODE:64-bit EL2h (Hypervisor, handler)
>
> (XEN) Xen call trace:
> (XEN)    [<0000000000238b78>] xmem_pool_alloc+0x348/0x4b0 (PC)
> (XEN)    [<0000000000238960>] xmem_pool_alloc+0x130/0x4b0 (LR)
> (XEN)    [<0000000000239100>] _xmalloc+0xf4/0x290
> (XEN)    [<00000000002392b0>] _xzalloc+0x14/0x38
> (XEN)    [<0000000000245430>] construct_dom0+0xbc0/0x14cc
> (XEN)    [<000000000028c4c4>] start_xen+0xbf4/0xcb0
> (XEN)    [<000000000020060c>] paging+0x84/0xbc
> (XEN)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) CPU0: Unexpected Trap: Hypervisor
> (XEN)
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
>

Regards,

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

Hi,

(Cc Andrew Turner who worked on the ACPI port for FreeBSD ARM64)

On 12/08/2015 09:52, Ian Campbell wrote:
> On Wed, 2015-08-12 at 11:04 +0800, Shannon Zhao wrote:
>> Hi Julien,
>>
>> On 2015/8/12 0:19, Julien Grall wrote:
>>> Hi Shannon,
>>>
>>> On 07/08/15 03:11, Shannon Zhao wrote:
>>>> 2. Create minimal DT to pass required information to Dom0
>>>> ----------------------------------------------------------
>>>> The minimal DT mainly passes Dom0 bootargs, address and size of
>>>> initrd
>>>> (if available), address and size of uefi system table, address and
>>>> size
>>>> of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver.
>>>>
>>>> An example of the minimal DT:
>>>> / {
>>>>      #address-cells = <2>;
>>>>      #size-cells = <1>;
>>>>      chosen {
>>>>          bootargs = "kernel=Image console=hvc0
>>>> earlycon=pl011,0x1c090000
>>>> root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force";
>>>>          linux,initrd-start = <0xXXXXXXXX>;
>>>>          linux,initrd-end = <0xXXXXXXXX>;
>>>>          linux,uefi-system-table = <0xXXXXXXXX>;
>>>>          linux,uefi-mmap-start = <0xXXXXXXXX>;
>>>>          linux,uefi-mmap-size = <0xXXXXXXXX>;
>>>>          linux,uefi-mmap-desc-size = <0xXXXXXXXX>;
>>>>          linux,uefi-mmap-desc-ver = <0xXXXXXXXX>;
>>>>      };
>>>> };
>>>>
>>>> For details loook at
>>>> https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi.
>>>> txt
>>>
>>> AFAICT, the device tree properties in this documentation are only used
>>> in order to communicate between the UEFI stub and Linux.
>>>
>>> This means that those properties are not standardize and can change at
>>> any time by Linux folks. They don't even live in
>>> Documentation/devicetree/
>>>
>>> I would also expect to see the same needs for FreeBSD running as DOM0
>>> with ACPI.
>>>
>> I'm not very clear about how FreeBSD communicates with UEFI. And when
>> booting with DT, how does FreeBSD communicate with UEFI? Not through
>> these properties?
>
> These properties are in effect a Linux internal interface defined between
> the "Linux UEFI stub" and the "Linux kernel proper". The stub and the
> kernel are notionally separate entities, although they are in the same tree
> etc there is a well defined transition/entry point between the two. Since
> they are in the same tree even though they are in theory "separate" I
> expect they will tend to co-evolve.
>
> IIRC we discussed with some of the maintainers (at Connect?) making this a
> more formal interface, i.e. exposing the entry point to "Linux kernel
> proper" which understands these properties to other than just the "Linux
> UEFI stub" specifically to external entities such as Xen.
>
> Probably part of this work needs to formalise that, such as by moving this
> binding into the proper external bindings dir.
>
> At which point BSD can (hopefully!) choose to support the same interface.

I haven't yet figure out how to exactly boot FreeBSD ARM64 as DOM0 (with 
both ACPI and DT).

But given that the only way to get the memory map on DOM0 with ACPI will 
be via those properties, we will surely needs them at some point in FreeBSD.

Regards,

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Stefano Stabellini]

On Wed, 12 Aug 2015, Ian Campbell wrote:
> On Wed, 2015-08-12 at 11:04 +0800, Shannon Zhao wrote:
> > Hi Julien,
> > 
> > On 2015/8/12 0:19, Julien Grall wrote:
> > > Hi Shannon,
> > > 
> > > On 07/08/15 03:11, Shannon Zhao wrote:
> > > > 2. Create minimal DT to pass required information to Dom0
> > > > ----------------------------------------------------------
> > > > The minimal DT mainly passes Dom0 bootargs, address and size of 
> > > > initrd
> > > > (if available), address and size of uefi system table, address and 
> > > > size
> > > > of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver.
> > > > 
> > > > An example of the minimal DT:
> > > > / {
> > > >     #address-cells = <2>;
> > > >     #size-cells = <1>;
> > > >     chosen {
> > > >         bootargs = "kernel=Image console=hvc0 
> > > > earlycon=pl011,0x1c090000
> > > > root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force";
> > > >         linux,initrd-start = <0xXXXXXXXX>;
> > > >         linux,initrd-end = <0xXXXXXXXX>;
> > > >         linux,uefi-system-table = <0xXXXXXXXX>;
> > > >         linux,uefi-mmap-start = <0xXXXXXXXX>;
> > > >         linux,uefi-mmap-size = <0xXXXXXXXX>;
> > > >         linux,uefi-mmap-desc-size = <0xXXXXXXXX>;
> > > >         linux,uefi-mmap-desc-ver = <0xXXXXXXXX>;
> > > >     };
> > > > };
> > > > 
> > > > For details loook at
> > > > https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi.
> > > > txt
> > > 
> > > AFAICT, the device tree properties in this documentation are only used
> > > in order to communicate between the UEFI stub and Linux.
> > > 
> > > This means that those properties are not standardize and can change at
> > > any time by Linux folks. They don't even live in 
> > > Documentation/devicetree/
> > > 
> > > I would also expect to see the same needs for FreeBSD running as DOM0
> > > with ACPI.
> > > 
> > I'm not very clear about how FreeBSD communicates with UEFI. And when
> > booting with DT, how does FreeBSD communicate with UEFI? Not through
> > these properties?
> 
> These properties are in effect a Linux internal interface defined between
> the "Linux UEFI stub" and the "Linux kernel proper". The stub and the
> kernel are notionally separate entities, although they are in the same tree
> etc there is a well defined transition/entry point between the two. Since
> they are in the same tree even though they are in theory "separate" I
> expect they will tend to co-evolve.
> 
> IIRC we discussed with some of the maintainers (at Connect?) making this a
> more formal interface, i.e. exposing the entry point to "Linux kernel
> proper" which understands these properties to other than just the "Linux
> UEFI stub" specifically to external entities such as Xen.
> 
> Probably part of this work needs to formalise that, such as by moving this
> binding into the proper external bindings dir.
> 
> At which point BSD can (hopefully!) choose to support the same interface.

CCing Ard, who originally proposed this.

Yes, I think it would be good to submit a patch to Linux to formalize
the bindings and clear out that the interface has become external.

As a reference, this was the original thread:

http://marc.info/?l=linux-kernel&m=142362266626403&w=2

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Andrew Turner]

On Wed, 12 Aug 2015 10:21:55 +0100
Julien Grall <julien.grall@citrix.com> wrote:

> Hi,
> 
> (Cc Andrew Turner who worked on the ACPI port for FreeBSD ARM64)
> 
> On 12/08/2015 09:52, Ian Campbell wrote:
> > On Wed, 2015-08-12 at 11:04 +0800, Shannon Zhao wrote:
> >> Hi Julien,
> >>
> >> On 2015/8/12 0:19, Julien Grall wrote:
> >>> Hi Shannon,
> >>>
> >>> On 07/08/15 03:11, Shannon Zhao wrote:
> >>>> 2. Create minimal DT to pass required information to Dom0
> >>>> ----------------------------------------------------------
> >>>> The minimal DT mainly passes Dom0 bootargs, address and size of
> >>>> initrd
> >>>> (if available), address and size of uefi system table, address
> >>>> and size
> >>>> of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver.
> >>>>
> >>>> An example of the minimal DT:
> >>>> / {
> >>>>      #address-cells = <2>;
> >>>>      #size-cells = <1>;
> >>>>      chosen {
> >>>>          bootargs = "kernel=Image console=hvc0
> >>>> earlycon=pl011,0x1c090000
> >>>> root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force";
> >>>>          linux,initrd-start = <0xXXXXXXXX>;
> >>>>          linux,initrd-end = <0xXXXXXXXX>;
> >>>>          linux,uefi-system-table = <0xXXXXXXXX>;
> >>>>          linux,uefi-mmap-start = <0xXXXXXXXX>;
> >>>>          linux,uefi-mmap-size = <0xXXXXXXXX>;
> >>>>          linux,uefi-mmap-desc-size = <0xXXXXXXXX>;
> >>>>          linux,uefi-mmap-desc-ver = <0xXXXXXXXX>;
> >>>>      };

Would it be possible to add a stdout property and node for the hvc0
device? It would help FreeBSD as we use this to find the kernel
console. We check for the stdout-path, linux,stdout-path, stdout,
stdin-path, and stdin properties, in that order, with the first
property selected as the console. If none are found we fall back to
searching for a serial0 device. You can see how we find the device at
[1].

> >>>> };
> >>>>
> >>>> For details loook at
> >>>> https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi.
> >>>> txt
> >>>
> >>> AFAICT, the device tree properties in this documentation are only
> >>> used in order to communicate between the UEFI stub and Linux.
> >>>
> >>> This means that those properties are not standardize and can
> >>> change at any time by Linux folks. They don't even live in
> >>> Documentation/devicetree/
> >>>
> >>> I would also expect to see the same needs for FreeBSD running as
> >>> DOM0 with ACPI.
> >>>
> >> I'm not very clear about how FreeBSD communicates with UEFI. And
> >> when booting with DT, how does FreeBSD communicate with UEFI? Not
> >> through these properties?

FreeBSD has a tool called loader.efi. It gets loaded by UEFI, and knows
how to communicate with it. It then loads the kernel and reads any
important data the kernel may need. Finally it puts this data into a
format the kernel understands, exits the boot services, and boots the
kernel. The kernel never communicates with UEFI, we have loaded any
data we need (however this may change in the future).

In the case of the memory may loader.efi calls GetMemoryMap from
EFI_BOOT_SERVICES. It then passes this data directly to the kernel for
the kernel to parse in the early boot code.

> >
> > These properties are in effect a Linux internal interface defined
> > between the "Linux UEFI stub" and the "Linux kernel proper". The
> > stub and the kernel are notionally separate entities, although they
> > are in the same tree etc there is a well defined transition/entry
> > point between the two. Since they are in the same tree even though
> > they are in theory "separate" I expect they will tend to co-evolve.
> >
> > IIRC we discussed with some of the maintainers (at Connect?) making
> > this a more formal interface, i.e. exposing the entry point to
> > "Linux kernel proper" which understands these properties to other
> > than just the "Linux UEFI stub" specifically to external entities
> > such as Xen.
> >
> > Probably part of this work needs to formalise that, such as by
> > moving this binding into the proper external bindings dir.
> >
> > At which point BSD can (hopefully!) choose to support the same
> > interface.

What are the advantages of these bindings over the existing UEFI calls
to get the memory map?

Andrew

[1]
https://svnweb.freebsd.org/base/head/sys/dev/uart/uart_cpu_fdt.c?revision=281438&view=markup#l127

-- 
ABT Systems Ltd
Unit 11, Hove Business Centre, Fonthill Road, Hove, BN3 6HA
Registered in England and Wales, No. 9285513

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Stefano Stabellini]

On Wed, 12 Aug 2015, Andrew Turner wrote:
> On Wed, 12 Aug 2015 10:21:55 +0100
> Julien Grall <julien.grall@citrix.com> wrote:
> 
> > Hi,
> > 
> > (Cc Andrew Turner who worked on the ACPI port for FreeBSD ARM64)
> > 
> > On 12/08/2015 09:52, Ian Campbell wrote:
> > > On Wed, 2015-08-12 at 11:04 +0800, Shannon Zhao wrote:
> > >> Hi Julien,
> > >>
> > >> On 2015/8/12 0:19, Julien Grall wrote:
> > >>> Hi Shannon,
> > >>>
> > >>> On 07/08/15 03:11, Shannon Zhao wrote:
> > >>>> 2. Create minimal DT to pass required information to Dom0
> > >>>> ----------------------------------------------------------
> > >>>> The minimal DT mainly passes Dom0 bootargs, address and size of
> > >>>> initrd
> > >>>> (if available), address and size of uefi system table, address
> > >>>> and size
> > >>>> of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver.
> > >>>>
> > >>>> An example of the minimal DT:
> > >>>> / {
> > >>>>      #address-cells = <2>;
> > >>>>      #size-cells = <1>;
> > >>>>      chosen {
> > >>>>          bootargs = "kernel=Image console=hvc0
> > >>>> earlycon=pl011,0x1c090000
> > >>>> root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force";
> > >>>>          linux,initrd-start = <0xXXXXXXXX>;
> > >>>>          linux,initrd-end = <0xXXXXXXXX>;
> > >>>>          linux,uefi-system-table = <0xXXXXXXXX>;
> > >>>>          linux,uefi-mmap-start = <0xXXXXXXXX>;
> > >>>>          linux,uefi-mmap-size = <0xXXXXXXXX>;
> > >>>>          linux,uefi-mmap-desc-size = <0xXXXXXXXX>;
> > >>>>          linux,uefi-mmap-desc-ver = <0xXXXXXXXX>;
> > >>>>      };
> 
> Would it be possible to add a stdout property and node for the hvc0
> device? It would help FreeBSD as we use this to find the kernel
> console. We check for the stdout-path, linux,stdout-path, stdout,
> stdin-path, and stdin properties, in that order, with the first
> property selected as the console. If none are found we fall back to
> searching for a serial0 device. You can see how we find the device at
> [1].
> 
> > >>>> };
> > >>>>
> > >>>> For details loook at
> > >>>> https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi.
> > >>>> txt
> > >>>
> > >>> AFAICT, the device tree properties in this documentation are only
> > >>> used in order to communicate between the UEFI stub and Linux.
> > >>>
> > >>> This means that those properties are not standardize and can
> > >>> change at any time by Linux folks. They don't even live in
> > >>> Documentation/devicetree/
> > >>>
> > >>> I would also expect to see the same needs for FreeBSD running as
> > >>> DOM0 with ACPI.
> > >>>
> > >> I'm not very clear about how FreeBSD communicates with UEFI. And
> > >> when booting with DT, how does FreeBSD communicate with UEFI? Not
> > >> through these properties?
> 
> FreeBSD has a tool called loader.efi. It gets loaded by UEFI, and knows
> how to communicate with it. It then loads the kernel and reads any
> important data the kernel may need. Finally it puts this data into a
> format the kernel understands, exits the boot services, and boots the
> kernel. The kernel never communicates with UEFI, we have loaded any
> data we need (however this may change in the future).
> 
> In the case of the memory may loader.efi calls GetMemoryMap from
> EFI_BOOT_SERVICES. It then passes this data directly to the kernel for
> the kernel to parse in the early boot code.
> 
> > >
> > > These properties are in effect a Linux internal interface defined
> > > between the "Linux UEFI stub" and the "Linux kernel proper". The
> > > stub and the kernel are notionally separate entities, although they
> > > are in the same tree etc there is a well defined transition/entry
> > > point between the two. Since they are in the same tree even though
> > > they are in theory "separate" I expect they will tend to co-evolve.
> > >
> > > IIRC we discussed with some of the maintainers (at Connect?) making
> > > this a more formal interface, i.e. exposing the entry point to
> > > "Linux kernel proper" which understands these properties to other
> > > than just the "Linux UEFI stub" specifically to external entities
> > > such as Xen.
> > >
> > > Probably part of this work needs to formalise that, such as by
> > > moving this binding into the proper external bindings dir.
> > >
> > > At which point BSD can (hopefully!) choose to support the same
> > > interface.
> 
> What are the advantages of these bindings over the existing UEFI calls
> to get the memory map?

The UEFI calls couldn't be used because ExitBootServices has already
been called.

The Linux UEFI stub is a bit like your loader.efi, except that is part
of the kernel. These bindings are used to pass data from the Linux UEFI
stub to the kernel proper, after ExitBootServices is called.

We plan to use the same bindings in Xen to pass data to the Dom0 kernel,
again after ExitBootServices is called (by Xen in this case).

The question is whether FreeBSD could support these bindings too.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Wed, 2015-08-12 at 11:48 +0100, Stefano Stabellini wrote:
> On Wed, 12 Aug 2015, Andrew Turner wrote:
> > On Wed, 12 Aug 2015 10:21:55 +0100
> > Julien Grall <julien.grall@citrix.com> wrote:
> > 
> > > Hi,
> > > 
> > > (Cc Andrew Turner who worked on the ACPI port for FreeBSD ARM64)
> > > 
> > > On 12/08/2015 09:52, Ian Campbell wrote:
> > > > On Wed, 2015-08-12 at 11:04 +0800, Shannon Zhao wrote:
> > > > > Hi Julien,
> > > > > 
> > > > > On 2015/8/12 0:19, Julien Grall wrote:
> > > > > > Hi Shannon,
> > > > > > 
> > > > > > On 07/08/15 03:11, Shannon Zhao wrote:
> > > > > > > 2. Create minimal DT to pass required information to Dom0
> > > > > > > ----------------------------------------------------------
> > > > > > > The minimal DT mainly passes Dom0 bootargs, address and size 
> > > > > > > of
> > > > > > > initrd
> > > > > > > (if available), address and size of uefi system table, 
> > > > > > > address
> > > > > > > and size
> > > > > > > of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc
> > > > > > > -ver.
> > > > > > > 
> > > > > > > An example of the minimal DT:
> > > > > > > / {
> > > > > > >      #address-cells = <2>;
> > > > > > >      #size-cells = <1>;
> > > > > > >      chosen {
> > > > > > >          bootargs = "kernel=Image console=hvc0
> > > > > > > earlycon=pl011,0x1c090000
> > > > > > > root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force";
> > > > > > >          linux,initrd-start = <0xXXXXXXXX>;
> > > > > > >          linux,initrd-end = <0xXXXXXXXX>;
> > > > > > >          linux,uefi-system-table = <0xXXXXXXXX>;
> > > > > > >          linux,uefi-mmap-start = <0xXXXXXXXX>;
> > > > > > >          linux,uefi-mmap-size = <0xXXXXXXXX>;
> > > > > > >          linux,uefi-mmap-desc-size = <0xXXXXXXXX>;
> > > > > > >          linux,uefi-mmap-desc-ver = <0xXXXXXXXX>;
> > > > > > >      };
> > 
> > Would it be possible to add a stdout property and node for the hvc0
> > device? It would help FreeBSD as we use this to find the kernel
> > console. We check for the stdout-path, linux,stdout-path, stdout,
> > stdin-path, and stdin properties, in that order, with the first
> > property selected as the console. If none are found we fall back to
> > searching for a serial0 device. You can see how we find the device at
> > [1].
> > 
> > > > > > > };
> > > > > > > 
> > > > > > > For details loook at
> > > > > > > https://github.com/torvalds/linux/blob/master/Documentation/a
> > > > > > > rm/uefi.
> > > > > > > txt
> > > > > > 
> > > > > > AFAICT, the device tree properties in this documentation are 
> > > > > > only
> > > > > > used in order to communicate between the UEFI stub and Linux.
> > > > > > 
> > > > > > This means that those properties are not standardize and can
> > > > > > change at any time by Linux folks. They don't even live in
> > > > > > Documentation/devicetree/
> > > > > > 
> > > > > > I would also expect to see the same needs for FreeBSD running 
> > > > > > as
> > > > > > DOM0 with ACPI.
> > > > > > 
> > > > > I'm not very clear about how FreeBSD communicates with UEFI. And
> > > > > when booting with DT, how does FreeBSD communicate with UEFI? Not
> > > > > through these properties?
> > 
> > FreeBSD has a tool called loader.efi. It gets loaded by UEFI, and knows
> > how to communicate with it. It then loads the kernel and reads any
> > important data the kernel may need. Finally it puts this data into a
> > format the kernel understands, exits the boot services, and boots the
> > kernel. The kernel never communicates with UEFI, we have loaded any
> > data we need (however this may change in the future).
> > 
> > In the case of the memory may loader.efi calls GetMemoryMap from
> > EFI_BOOT_SERVICES. It then passes this data directly to the kernel for
> > the kernel to parse in the early boot code.
> > 
> > > > 
> > > > These properties are in effect a Linux internal interface defined
> > > > between the "Linux UEFI stub" and the "Linux kernel proper". The
> > > > stub and the kernel are notionally separate entities, although they
> > > > are in the same tree etc there is a well defined transition/entry
> > > > point between the two. Since they are in the same tree even though
> > > > they are in theory "separate" I expect they will tend to co-evolve.
> > > > 
> > > > IIRC we discussed with some of the maintainers (at Connect?) making
> > > > this a more formal interface, i.e. exposing the entry point to
> > > > "Linux kernel proper" which understands these properties to other
> > > > than just the "Linux UEFI stub" specifically to external entities
> > > > such as Xen.
> > > > 
> > > > Probably part of this work needs to formalise that, such as by
> > > > moving this binding into the proper external bindings dir.
> > > > 
> > > > At which point BSD can (hopefully!) choose to support the same
> > > > interface.
> > 
> > What are the advantages of these bindings over the existing UEFI calls
> > to get the memory map?
> 
> The UEFI calls couldn't be used because ExitBootServices has already
> been called.
> 
> The Linux UEFI stub is a bit like your loader.efi, except that is part
> of the kernel.

Strictly it is considered a separate thing, much like loader.efi, despite
where it lives e.g. it is self contained and not allowed to call into the
kernel proper except via the formal interface provided for the hand-off.

That might seem like semantic quibbling, but I just want to clarify that
the Linux and BSD approaches here are basically the same.

Given that these device tree bindings are really just Linux's equivalent of
the "a format the kernel understands" which BSD uses as described above. I
don't know what format BSD uses, Linux just happened to have a DTB library
handy...

>  These bindings are used to pass data from the Linux UEFI
> stub to the kernel proper, after ExitBootServices is called.
> 
> We plan to use the same bindings in Xen to pass data to the Dom0 kernel,
> again after ExitBootServices is called (by Xen in this case).
> 
> The question is whether FreeBSD could support these bindings too.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

On 12/08/15 12:23, Ian Campbell wrote:
> Strictly it is considered a separate thing, much like loader.efi, despite
> where it lives e.g. it is self contained and not allowed to call into the
> kernel proper except via the formal interface provided for the hand-off.
> 
> That might seem like semantic quibbling, but I just want to clarify that
> the Linux and BSD approaches here are basically the same.
> 
> Given that these device tree bindings are really just Linux's equivalent of
> the "a format the kernel understands" which BSD uses as described above. I
> don't know what format BSD uses, Linux just happened to have a DTB library
> handy...

IIRC, on FreeBSD, the loader and the kernel is talking through a custom
format called metadata. There is no modification of device tree by the
loader.

Although, I would prefer to see a common interface between Xen and DOM0
rather than implementing a custom one for each OS we will support.

I have to think how everything will work together. AFAIK, on x86, the
loader is loading Xen and the FreeBSD kernel in the memory. The metadata
necessary for the kernel is passed as a multiboot entry.

I will speak with Royger to see what we can do here.

In the meantime, I think we should drop "linux," when we standardize
them to show that they are generic and not linux specific.

Regards,

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

On 12/08/15 11:36, Andrew Turner wrote:
> Would it be possible to add a stdout property and node for the hvc0
> device? It would help FreeBSD as we use this to find the kernel
> console. We check for the stdout-path, linux,stdout-path, stdout,
> stdin-path, and stdin properties, in that order, with the first
> property selected as the console. If none are found we fall back to
> searching for a serial0 device. You can see how we find the device at
> [1].

the stdout-path property is used by Xen in order to get the UART. The
property will be removed from the device tree passed to DOM0.

The Xen console is not UART a driver so having a property with "hvc"
wouldn't help here.

Although, when running on Xen, FreeBSD is creating a Xen console by
default with the higher priority. It will be grabbed by FreeBSD as the
default one.

Regards,

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Jan Beulich]

>>> On 07.08.15 at 04:11, <zhaoshenglong@huawei.com> wrote:
> 1. Copy and change some EFI and ACPI tables
> -------------------------------------------

A key thing I'm missing here is reasoning of why this copying approach
is needed in the first place. Remember that on the x86 side we get
away without any such copying-and-changing. Yet the farther you
diverge from x86's model, the more changes you'll need to common
Xen code in e.g. Linux.

> All these tables will be copied to Dom0 memory except that the reused
> tables(DSDT, SPCR, etc) will be mapped to Dom0.

Which again seems odd - such tables should be considered MMIO
(despite living in RAM), and hence not be part of Dom0's memory
assignment.

> 3. Dom0 gets grant table and event channel irq information
> -----------------------------------------------------------
> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that
> it runs on Xen hypervisor.
> 
> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS
> and HVM_PARAM_GNTTAB_SIZE.

Mind explaining why you need this on ARM when x86 gets away without?

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Jan Beulich]

>>> On 11.08.15 at 16:12, <ian.campbell@citrix.com> wrote:
> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
>> 
> [...]
>> 3. Dom0 gets grant table and event channel irq information
>> -----------------------------------------------------------
>> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that
>> it runs on Xen hypervisor.
>> 
>> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS
>> and HVM_PARAM_GNTTAB_SIZE.
> 
> The reason we expose this range is essentially to allow OS authors to take
> a short cut by telling them about an IPA range which is unused, so it is
> available for remapping the grant table into. On x86 there is a BAR on the
> Xen platform PCI device which serves a similar purpose.

Ah, this answers a question I just raised, but means the items are
mis-named and badly described.

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Christoffer Dall]

On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 07.08.15 at 04:11, <zhaoshenglong@huawei.com> wrote:
>> 1. Copy and change some EFI and ACPI tables
>> -------------------------------------------
>
> A key thing I'm missing here is reasoning of why this copying approach
> is needed in the first place. Remember that on the x86 side we get
> away without any such copying-and-changing. Yet the farther you
> diverge from x86's model, the more changes you'll need to common
> Xen code in e.g. Linux.
>
>> All these tables will be copied to Dom0 memory except that the reused
>> tables(DSDT, SPCR, etc) will be mapped to Dom0.
>
> Which again seems odd - such tables should be considered MMIO
> (despite living in RAM), and hence not be part of Dom0's memory
> assignment.
>
not sure if this applies to the context of your comment, but we had
issues before when trying to deal with this data as device memory
(MMIO), because ARM doesn't allow unaligned accesses etc. on device
memory, and so Dom0 would crash at memory abort exceptions during
boot, so this really does have to be mapped as RAM to Dom0.  If you
meant some internal Xen bookkeeping thing, then just ignore me.

-Christoffer

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Jan Beulich]

>>> On 12.08.15 at 17:51, <christoffer.dall@linaro.org> wrote:
> On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 07.08.15 at 04:11, <zhaoshenglong@huawei.com> wrote:
>>> All these tables will be copied to Dom0 memory except that the reused
>>> tables(DSDT, SPCR, etc) will be mapped to Dom0.
>>
>> Which again seems odd - such tables should be considered MMIO
>> (despite living in RAM), and hence not be part of Dom0's memory
>> assignment.
>>
> not sure if this applies to the context of your comment, but we had
> issues before when trying to deal with this data as device memory
> (MMIO), because ARM doesn't allow unaligned accesses etc. on device
> memory, and so Dom0 would crash at memory abort exceptions during
> boot, so this really does have to be mapped as RAM to Dom0.  If you
> meant some internal Xen bookkeeping thing, then just ignore me.

Hmm, how would native Linux avoid such unaligned accesses then?

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

On 12/08/15 16:58, Jan Beulich wrote:
>>>> On 12.08.15 at 17:51, <christoffer.dall@linaro.org> wrote:
>> On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>> On 07.08.15 at 04:11, <zhaoshenglong@huawei.com> wrote:
>>>> All these tables will be copied to Dom0 memory except that the reused
>>>> tables(DSDT, SPCR, etc) will be mapped to Dom0.
>>>
>>> Which again seems odd - such tables should be considered MMIO
>>> (despite living in RAM), and hence not be part of Dom0's memory
>>> assignment.
>>>
>> not sure if this applies to the context of your comment, but we had
>> issues before when trying to deal with this data as device memory
>> (MMIO), because ARM doesn't allow unaligned accesses etc. on device
>> memory, and so Dom0 would crash at memory abort exceptions during
>> boot, so this really does have to be mapped as RAM to Dom0.  If you
>> meant some internal Xen bookkeeping thing, then just ignore me.
> 
> Hmm, how would native Linux avoid such unaligned accesses then?

ACPI table are living on RAM on ARM. So there is no issue with Linux
baremetal.

The "problem" is from Xen where we are mapping memory region with Device
attribute. This is because until now we never had a reason to directly
map other thing than MMIO to the domain.

This could be fixed by adding new helper in Xen to directly map RAM region.

Nonetheless, we still have to copy some table in Xen in order to modify
them and/or new one. I have in mind the FADT table to set the hypervisor
field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to
avoid the kernel thinks there is hyp mode available.

Regards,

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Jan Beulich]

>>> On 12.08.15 at 18:18, <julien.grall@citrix.com> wrote:
> On 12/08/15 16:58, Jan Beulich wrote:
>>>>> On 12.08.15 at 17:51, <christoffer.dall@linaro.org> wrote:
>>> On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>> On 07.08.15 at 04:11, <zhaoshenglong@huawei.com> wrote:
>>>>> All these tables will be copied to Dom0 memory except that the reused
>>>>> tables(DSDT, SPCR, etc) will be mapped to Dom0.
>>>>
>>>> Which again seems odd - such tables should be considered MMIO
>>>> (despite living in RAM), and hence not be part of Dom0's memory
>>>> assignment.
>>>>
>>> not sure if this applies to the context of your comment, but we had
>>> issues before when trying to deal with this data as device memory
>>> (MMIO), because ARM doesn't allow unaligned accesses etc. on device
>>> memory, and so Dom0 would crash at memory abort exceptions during
>>> boot, so this really does have to be mapped as RAM to Dom0.  If you
>>> meant some internal Xen bookkeeping thing, then just ignore me.
>> 
>> Hmm, how would native Linux avoid such unaligned accesses then?
> 
> ACPI table are living on RAM on ARM. So there is no issue with Linux
> baremetal.

I'm sure our interpretation of the meaning of RAM here differs:
While from a physical perspective the tables live in RAM too on
x86, from a memory map perspective they don't. And since iirc
ACPI implies UEFI, and since UEFI has special memory types
for such tables (to prevent the OS from using the space as
normal memory), I can't believe this to be different under ARM.

> The "problem" is from Xen where we are mapping memory region with Device
> attribute. This is because until now we never had a reason to directly
> map other thing than MMIO to the domain.
> 
> This could be fixed by adding new helper in Xen to directly map RAM region.

Which would seem to be the correct route.

> Nonetheless, we still have to copy some table in Xen in order to modify
> them and/or new one. I have in mind the FADT table to set the hypervisor
> field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to
> avoid the kernel thinks there is hyp mode available.

"Have to"? Or rather "would like to"? As said in another reply to
Shannon, I didn't see any rationale spelled out for this fundamental
difference to x86.

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Christoffer Dall]

On Thu, Aug 13, 2015 at 8:41 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 12.08.15 at 18:18, <julien.grall@citrix.com> wrote:
>> On 12/08/15 16:58, Jan Beulich wrote:
>>>>>> On 12.08.15 at 17:51, <christoffer.dall@linaro.org> wrote:
>>>> On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>> On 07.08.15 at 04:11, <zhaoshenglong@huawei.com> wrote:
>>>>>> All these tables will be copied to Dom0 memory except that the reused
>>>>>> tables(DSDT, SPCR, etc) will be mapped to Dom0.
>>>>>
>>>>> Which again seems odd - such tables should be considered MMIO
>>>>> (despite living in RAM), and hence not be part of Dom0's memory
>>>>> assignment.
>>>>>
>>>> not sure if this applies to the context of your comment, but we had
>>>> issues before when trying to deal with this data as device memory
>>>> (MMIO), because ARM doesn't allow unaligned accesses etc. on device
>>>> memory, and so Dom0 would crash at memory abort exceptions during
>>>> boot, so this really does have to be mapped as RAM to Dom0.  If you
>>>> meant some internal Xen bookkeeping thing, then just ignore me.
>>>
>>> Hmm, how would native Linux avoid such unaligned accesses then?
>>
>> ACPI table are living on RAM on ARM. So there is no issue with Linux
>> baremetal.
>
> I'm sure our interpretation of the meaning of RAM here differs:
> While from a physical perspective the tables live in RAM too on
> x86, from a memory map perspective they don't. And since iirc
> ACPI implies UEFI, and since UEFI has special memory types
> for such tables (to prevent the OS from using the space as
> normal memory), I can't believe this to be different under ARM.
>
>> The "problem" is from Xen where we are mapping memory region with Device
>> attribute. This is because until now we never had a reason to directly
>> map other thing than MMIO to the domain.
>>
>> This could be fixed by adding new helper in Xen to directly map RAM region.
>
> Which would seem to be the correct route.
>
>> Nonetheless, we still have to copy some table in Xen in order to modify
>> them and/or new one. I have in mind the FADT table to set the hypervisor
>> field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to
>> avoid the kernel thinks there is hyp mode available.
>
> "Have to"? Or rather "would like to"? As said in another reply to
> Shannon, I didn't see any rationale spelled out for this fundamental
> difference to x86.
>
So the suggestion is to just edit the tables in place rather than
copying them and relying on Xen having already read whatever
'original' information it needs before modifying them?

Is there ever a case where the tables live in ROM or shouldn't be
modified for any other reason?

-Christoffer

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Jan Beulich]

>>> On 13.08.15 at 10:01, <christoffer.dall@linaro.org> wrote:
> On Thu, Aug 13, 2015 at 8:41 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 12.08.15 at 18:18, <julien.grall@citrix.com> wrote:
>>> Nonetheless, we still have to copy some table in Xen in order to modify
>>> them and/or new one. I have in mind the FADT table to set the hypervisor
>>> field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to
>>> avoid the kernel thinks there is hyp mode available.
>>
>> "Have to"? Or rather "would like to"? As said in another reply to
>> Shannon, I didn't see any rationale spelled out for this fundamental
>> difference to x86.
>>
> So the suggestion is to just edit the tables in place rather than
> copying them and relying on Xen having already read whatever
> 'original' information it needs before modifying them?

No, the suggestion is to leave the tables alone and teach Dom0
to ignore parts it has no interest in.

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Jan Beulich]

>>> On 13.08.15 at 10:01, <christoffer.dall@linaro.org> wrote:
> On Thu, Aug 13, 2015 at 8:41 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 12.08.15 at 18:18, <julien.grall@citrix.com> wrote:
>>> Nonetheless, we still have to copy some table in Xen in order to modify
>>> them and/or new one. I have in mind the FADT table to set the hypervisor
>>> field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to
>>> avoid the kernel thinks there is hyp mode available.
>>
>> "Have to"? Or rather "would like to"? As said in another reply to
>> Shannon, I didn't see any rationale spelled out for this fundamental
>> difference to x86.
>>
> So the suggestion is to just edit the tables in place rather than
> copying them and relying on Xen having already read whatever
> 'original' information it needs before modifying them?

And just to clarify / amend my reply just sent: I also don't see why
you would need to expose the System Table to Dom0. Did anyone
of you look at how we draw the boundary between Xen and Dom0
on x86? If not, I'd suggest doing so. If so, I'd like to ask for
proper reasoning for any item of data exposed to Dom0 beyond
what we currently expose.

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Thu, 2015-08-13 at 00:41 -0600, Jan Beulich wrote:
> > 
> > Nonetheless, we still have to copy some table in Xen in order to modify
> > them and/or new one. I have in mind the FADT table to set the hypervisor
> > field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to
> > avoid the kernel thinks there is hyp mode available.
> 
> "Have to"? Or rather "would like to"? As said in another reply to
> Shannon, I didn't see any rationale spelled out for this fundamental
> difference to x86.

I think the fundamental difference is that h/w virt features are not
(always) "discoverable" through h/w interfaces on ARM whereas they are
visible in e.g. cpuid on x86 and not described in ACPI (x86 is only just
gaining hardware support for virtualised interrupts so perhaps this will
change? I think it doesn't have any hypervisor dedicated timer sources).

So on ARM the firmware tables contain things like the additional
virtualisation register regions in the interrupt controller description and
the hypervisor timer interrupt in the timer block description. So we would
like to hide these from dom0.

Perhaps instead we should teach dom0 to notice that it was launched in
Kernel mode rather than Hyp mode (which is detectable) and therefore ignore
these unusable resources. Now you mention it that does sound sensible and I
imagine is even already (close to) true for a Linux kernel to handle e.g.
older firmware with newer device tree.

The other reason for modification is to hide the Xen console device (i.e.
one UART) from the dom0 kernel, since it is unusable. How does that work on
x86? Do you just not bother and you expect the admin to arrange the
configuration to work or is there some other trick?

BTW, IIRC x86 does modify at least one ACPI table which is the DMAR (I
think), to hide the IOMMU from the guest? That's another table we would
want to frob on ARM I think (or it's equivalent, which I think is IORT).

Ian.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Jan Beulich]

>>> On 13.08.15 at 11:05, <ian.campbell@citrix.com> wrote:
> The other reason for modification is to hide the Xen console device (i.e.
> one UART) from the dom0 kernel, since it is unusable. How does that work on
> x86? Do you just not bother and you expect the admin to arrange the
> configuration to work or is there some other trick?

When it's I/O port based we simply disallow access to the ports (i.e.
Dom0 reads return all ones, Dom0 writes get dropped). When it's
MMIO based (which iirc became an option only in the not so distant
past) we can't always do that, since there could be other devices
on the same MMIO page. In any event we then pci_hide_devices() or
pci_ro_device() the device (see ns16550_init_postirq()) to limit the
damage Dom0 can do.

> BTW, IIRC x86 does modify at least one ACPI table which is the DMAR (I
> think), to hide the IOMMU from the guest? That's another table we would
> want to frob on ARM I think (or it's equivalent, which I think is IORT).

Eliminating that hack is supposed to be on the VT-d maintainers'
TODO list(s) - Dom0 has no business looking at that table (and its
AMD counterpart already isn't being fiddled with in the same way).

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Stefano Stabellini]

On Thu, 13 Aug 2015, Jan Beulich wrote:
> >>> On 12.08.15 at 18:18, <julien.grall@citrix.com> wrote:
> > On 12/08/15 16:58, Jan Beulich wrote:
> >>>>> On 12.08.15 at 17:51, <christoffer.dall@linaro.org> wrote:
> >>> On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich <JBeulich@suse.com> wrote:
> >>>>>>> On 07.08.15 at 04:11, <zhaoshenglong@huawei.com> wrote:
> >>>>> All these tables will be copied to Dom0 memory except that the reused
> >>>>> tables(DSDT, SPCR, etc) will be mapped to Dom0.
> >>>>
> >>>> Which again seems odd - such tables should be considered MMIO
> >>>> (despite living in RAM), and hence not be part of Dom0's memory
> >>>> assignment.
> >>>>
> >>> not sure if this applies to the context of your comment, but we had
> >>> issues before when trying to deal with this data as device memory
> >>> (MMIO), because ARM doesn't allow unaligned accesses etc. on device
> >>> memory, and so Dom0 would crash at memory abort exceptions during
> >>> boot, so this really does have to be mapped as RAM to Dom0.  If you
> >>> meant some internal Xen bookkeeping thing, then just ignore me.
> >> 
> >> Hmm, how would native Linux avoid such unaligned accesses then?
> > 
> > ACPI table are living on RAM on ARM. So there is no issue with Linux
> > baremetal.
> 
> I'm sure our interpretation of the meaning of RAM here differs:
> While from a physical perspective the tables live in RAM too on
> x86, from a memory map perspective they don't. And since iirc
> ACPI implies UEFI, and since UEFI has special memory types
> for such tables (to prevent the OS from using the space as
> normal memory), I can't believe this to be different under ARM.
> 
> > The "problem" is from Xen where we are mapping memory region with Device
> > attribute. This is because until now we never had a reason to directly
> > map other thing than MMIO to the domain.
> > 
> > This could be fixed by adding new helper in Xen to directly map RAM region.
> 
> Which would seem to be the correct route.
> 
> > Nonetheless, we still have to copy some table in Xen in order to modify
> > them and/or new one. I have in mind the FADT table to set the hypervisor
> > field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to
> > avoid the kernel thinks there is hyp mode available.
> 
> "Have to"? Or rather "would like to"? As said in another reply to
> Shannon, I didn't see any rationale spelled out for this fundamental
> difference to x86.

Just because it was done this way before, it doesn't mean that it is the
right way of doing it.

I think is bad design to expose the presence of certain functionalities
in ACPI tables and then expect that the Dom0 kernel won't use them.
ACPI tables should describe only and all the hardware which is exposed
to Dom0. Anything else is a mistake.

For example it is only natural for the kernel to try to use the GIC hyp
functionalities if they are described, while actually they are not
emulated by Xen at all.

I would rather teach Xen to fix the tables now, than later try teach
every possible Dom0 kernel (Linux, FreeBSD, etc) to ignore a set of info
which are wrong but still passed to them anyway. Moreover the list of
things to ignore can change over time. It is just a bad ABI to maintain.

In terms of code we could share in Linux between Xen x86 and Xen ARM
regarding dealing with ACPI info we want to ignore, unfortunately there
isn't much of it, because we are mostly talking about new ARM specific
tables here (GIC, arch_timer, etc).

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Jan Beulich]

>>> On 13.08.15 at 11:43, <stefano.stabellini@eu.citrix.com> wrote:
> On Thu, 13 Aug 2015, Jan Beulich wrote:
>> >>> On 12.08.15 at 18:18, <julien.grall@citrix.com> wrote:
>> > On 12/08/15 16:58, Jan Beulich wrote:
>> >>>>> On 12.08.15 at 17:51, <christoffer.dall@linaro.org> wrote:
>> >>> On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich <JBeulich@suse.com> wrote:
>> >>>>>>> On 07.08.15 at 04:11, <zhaoshenglong@huawei.com> wrote:
>> >>>>> All these tables will be copied to Dom0 memory except that the reused
>> >>>>> tables(DSDT, SPCR, etc) will be mapped to Dom0.
>> >>>>
>> >>>> Which again seems odd - such tables should be considered MMIO
>> >>>> (despite living in RAM), and hence not be part of Dom0's memory
>> >>>> assignment.
>> >>>>
>> >>> not sure if this applies to the context of your comment, but we had
>> >>> issues before when trying to deal with this data as device memory
>> >>> (MMIO), because ARM doesn't allow unaligned accesses etc. on device
>> >>> memory, and so Dom0 would crash at memory abort exceptions during
>> >>> boot, so this really does have to be mapped as RAM to Dom0.  If you
>> >>> meant some internal Xen bookkeeping thing, then just ignore me.
>> >> 
>> >> Hmm, how would native Linux avoid such unaligned accesses then?
>> > 
>> > ACPI table are living on RAM on ARM. So there is no issue with Linux
>> > baremetal.
>> 
>> I'm sure our interpretation of the meaning of RAM here differs:
>> While from a physical perspective the tables live in RAM too on
>> x86, from a memory map perspective they don't. And since iirc
>> ACPI implies UEFI, and since UEFI has special memory types
>> for such tables (to prevent the OS from using the space as
>> normal memory), I can't believe this to be different under ARM.
>> 
>> > The "problem" is from Xen where we are mapping memory region with Device
>> > attribute. This is because until now we never had a reason to directly
>> > map other thing than MMIO to the domain.
>> > 
>> > This could be fixed by adding new helper in Xen to directly map RAM region.
>> 
>> Which would seem to be the correct route.
>> 
>> > Nonetheless, we still have to copy some table in Xen in order to modify
>> > them and/or new one. I have in mind the FADT table to set the hypervisor
>> > field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to
>> > avoid the kernel thinks there is hyp mode available.
>> 
>> "Have to"? Or rather "would like to"? As said in another reply to
>> Shannon, I didn't see any rationale spelled out for this fundamental
>> difference to x86.
> 
> Just because it was done this way before, it doesn't mean that it is the
> right way of doing it.

Correct. Which is why I'm not saying outright "no" but asking for
justification.

> I think is bad design to expose the presence of certain functionalities
> in ACPI tables and then expect that the Dom0 kernel won't use them.
> ACPI tables should describe only and all the hardware which is exposed
> to Dom0. Anything else is a mistake.

That depends on the perspective you take: Especially for a PV
Dom0 it is quite reasonable for the kernel to be aware of certain
implications from running under a (or the) hypervisor it knows of.
I agree that the perspective may shift when it comes to HVM-like
Dom0 (albeit in the spectrum I'd rather see ARM near PVH, which
again is supposed to be aware).

> For example it is only natural for the kernel to try to use the GIC hyp
> functionalities if they are described, while actually they are not
> emulated by Xen at all.

See Ian's earlier reply: It can also be considered natural for it to
be aware that when run in EL2 to not use EL1 functionality.

> I would rather teach Xen to fix the tables now, than later try teach
> every possible Dom0 kernel (Linux, FreeBSD, etc) to ignore a set of info
> which are wrong but still passed to them anyway. Moreover the list of
> things to ignore can change over time. It is just a bad ABI to maintain.

I don't think there's a clear advantage to teaching Xen of new
tables to hide over teaching Dom0 of new tables to ignore - it
much depends on release cycles and such which one would be
more beneficial.

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Stefano Stabellini]

On Thu, 13 Aug 2015, Jan Beulich wrote:
> >>> On 13.08.15 at 11:43, <stefano.stabellini@eu.citrix.com> wrote:
> > On Thu, 13 Aug 2015, Jan Beulich wrote:
> >> >>> On 12.08.15 at 18:18, <julien.grall@citrix.com> wrote:
> >> > On 12/08/15 16:58, Jan Beulich wrote:
> >> >>>>> On 12.08.15 at 17:51, <christoffer.dall@linaro.org> wrote:
> >> >>> On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich <JBeulich@suse.com> wrote:
> >> >>>>>>> On 07.08.15 at 04:11, <zhaoshenglong@huawei.com> wrote:
> >> >>>>> All these tables will be copied to Dom0 memory except that the reused
> >> >>>>> tables(DSDT, SPCR, etc) will be mapped to Dom0.
> >> >>>>
> >> >>>> Which again seems odd - such tables should be considered MMIO
> >> >>>> (despite living in RAM), and hence not be part of Dom0's memory
> >> >>>> assignment.
> >> >>>>
> >> >>> not sure if this applies to the context of your comment, but we had
> >> >>> issues before when trying to deal with this data as device memory
> >> >>> (MMIO), because ARM doesn't allow unaligned accesses etc. on device
> >> >>> memory, and so Dom0 would crash at memory abort exceptions during
> >> >>> boot, so this really does have to be mapped as RAM to Dom0.  If you
> >> >>> meant some internal Xen bookkeeping thing, then just ignore me.
> >> >> 
> >> >> Hmm, how would native Linux avoid such unaligned accesses then?
> >> > 
> >> > ACPI table are living on RAM on ARM. So there is no issue with Linux
> >> > baremetal.
> >> 
> >> I'm sure our interpretation of the meaning of RAM here differs:
> >> While from a physical perspective the tables live in RAM too on
> >> x86, from a memory map perspective they don't. And since iirc
> >> ACPI implies UEFI, and since UEFI has special memory types
> >> for such tables (to prevent the OS from using the space as
> >> normal memory), I can't believe this to be different under ARM.
> >> 
> >> > The "problem" is from Xen where we are mapping memory region with Device
> >> > attribute. This is because until now we never had a reason to directly
> >> > map other thing than MMIO to the domain.
> >> > 
> >> > This could be fixed by adding new helper in Xen to directly map RAM region.
> >> 
> >> Which would seem to be the correct route.
> >> 
> >> > Nonetheless, we still have to copy some table in Xen in order to modify
> >> > them and/or new one. I have in mind the FADT table to set the hypervisor
> >> > field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to
> >> > avoid the kernel thinks there is hyp mode available.
> >> 
> >> "Have to"? Or rather "would like to"? As said in another reply to
> >> Shannon, I didn't see any rationale spelled out for this fundamental
> >> difference to x86.
> > 
> > Just because it was done this way before, it doesn't mean that it is the
> > right way of doing it.
> 
> Correct. Which is why I'm not saying outright "no" but asking for
> justification.
> 
> > I think is bad design to expose the presence of certain functionalities
> > in ACPI tables and then expect that the Dom0 kernel won't use them.
> > ACPI tables should describe only and all the hardware which is exposed
> > to Dom0. Anything else is a mistake.
> 
> That depends on the perspective you take: Especially for a PV
> Dom0 it is quite reasonable for the kernel to be aware of certain
> implications from running under a (or the) hypervisor it knows of.
> I agree that the perspective may shift when it comes to HVM-like
> Dom0 (albeit in the spectrum I'd rather see ARM near PVH, which
> again is supposed to be aware).

PVH has always been the wrong analogy for ARM guests. Guest kernels boot
following native boot paths. They don't require emulation because the
ARM architecture is much more flexible and legacy free compared to x86,
not because we instructed the kernel to ignore certain pieces of
hardware. So they are like PV on HVM without any need for QEMU because
there is nothing to emulate. Maybe the right analogy is HVMLite.


> > For example it is only natural for the kernel to try to use the GIC hyp
> > functionalities if they are described, while actually they are not
> > emulated by Xen at all.
> 
> See Ian's earlier reply: It can also be considered natural for it to
> be aware that when run in EL2 to not use EL1 functionality.

It is not just about the GIC Hyp functionalities.


> > I would rather teach Xen to fix the tables now, than later try teach
> > every possible Dom0 kernel (Linux, FreeBSD, etc) to ignore a set of info
> > which are wrong but still passed to them anyway. Moreover the list of
> > things to ignore can change over time. It is just a bad ABI to maintain.
> 
> I don't think there's a clear advantage to teaching Xen of new
> tables to hide over teaching Dom0 of new tables to ignore - it
> much depends on release cycles and such which one would be
> more beneficial.

There are a couple of very obvious advantages:

- there is one Xen, there are many Dom0 kernels
- if Xen changes ACPI tables, Dom0 won't be surprised by the fact that
actually they don't match the hardware. Keep in mind that we don't have
10 years of PV history on ARM. People still think of this as confusing
and counter-intuitive.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Thu, 2015-08-13 at 11:13 +0100, Stefano Stabellini wrote:
> 
> > > For example it is only natural for the kernel to try to use the GIC 
> > > hyp
> > > functionalities if they are described, while actually they are not
> > > emulated by Xen at all.
> > 
> > See Ian's earlier reply: It can also be considered natural for it to
> > be aware that when run in EL2 to not use EL1 functionality.

NB EL2 == Hyp and EL1 == Kernel, so it's the other way round, FWIW.

> It is not just about the GIC Hyp functionalities.

What else is there which is not subject to this logic? Timers are too, it
even applies to IOMMU's which have both stage1 and stage2 bits.

BTW, I think kernels _already_ need to deal with a lot of this because in
reality nobody modifies the DTB when they use a firmware which launches the
kernel in EL1. IOW I think the kernel is already aware of which resources
can be used by which privilege level.

Ian.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Christoffer Dall]

On Thu, Aug 13, 2015 at 11:22:19AM +0100, Ian Campbell wrote:
> On Thu, 2015-08-13 at 11:13 +0100, Stefano Stabellini wrote:
> > 
> > > > For example it is only natural for the kernel to try to use the GIC 
> > > > hyp
> > > > functionalities if they are described, while actually they are not
> > > > emulated by Xen at all.
> > > 
> > > See Ian's earlier reply: It can also be considered natural for it to
> > > be aware that when run in EL2 to not use EL1 functionality.
> 
> NB EL2 == Hyp and EL1 == Kernel, so it's the other way round, FWIW.
> 
> > It is not just about the GIC Hyp functionalities.
> 
> What else is there which is not subject to this logic? Timers are too, it
> even applies to IOMMU's which have both stage1 and stage2 bits.
> 
> BTW, I think kernels _already_ need to deal with a lot of this because in
> reality nobody modifies the DTB when they use a firmware which launches the
> kernel in EL1. IOW I think the kernel is already aware of which resources
> can be used by which privilege level.
> 
Yes, for resources specific to EL2 I believe that is indeed the case
(the GIC driver doesn't look at the hypervisor control register address,
and KVM does not even get that far if you're not booted in EL2, and the
timer only uses the virtual timer if not booted in EL2 - we never
attempt to use the hyp timer until Marc's VHE patches land, but they
also depend on being booted in hyp mode).

However, what about for other resources?  Having code somewhere that
says "hide this random piece of hardware if you're Xen dom0" sounds
awful to me.  I know it's only the serial port right now, but still.

-Christoffer

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Stefano Stabellini]

On Thu, 13 Aug 2015, Christoffer Dall wrote:
> On Thu, Aug 13, 2015 at 11:22:19AM +0100, Ian Campbell wrote:
> > On Thu, 2015-08-13 at 11:13 +0100, Stefano Stabellini wrote:
> > > 
> > > > > For example it is only natural for the kernel to try to use the GIC 
> > > > > hyp
> > > > > functionalities if they are described, while actually they are not
> > > > > emulated by Xen at all.
> > > > 
> > > > See Ian's earlier reply: It can also be considered natural for it to
> > > > be aware that when run in EL2 to not use EL1 functionality.
> > 
> > NB EL2 == Hyp and EL1 == Kernel, so it's the other way round, FWIW.
> > 
> > > It is not just about the GIC Hyp functionalities.
> > 
> > What else is there which is not subject to this logic? Timers are too, it
> > even applies to IOMMU's which have both stage1 and stage2 bits.
> > 
> > BTW, I think kernels _already_ need to deal with a lot of this because in
> > reality nobody modifies the DTB when they use a firmware which launches the
> > kernel in EL1. IOW I think the kernel is already aware of which resources
> > can be used by which privilege level.
> > 
> Yes, for resources specific to EL2 I believe that is indeed the case
> (the GIC driver doesn't look at the hypervisor control register address,
> and KVM does not even get that far if you're not booted in EL2, and the
> timer only uses the virtual timer if not booted in EL2 - we never
> attempt to use the hyp timer until Marc's VHE patches land, but they
> also depend on being booted in hyp mode).
>
> However, what about for other resources?  Having code somewhere that
> says "hide this random piece of hardware if you're Xen dom0" sounds
> awful to me.  I know it's only the serial port right now, but still.

I completely agree. Non-EL2 specific resources should defenitely be
removed. I can see that EL2 stuff could be left there, but I think that
removing it now would avoid us trouble in the future.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Thu, 2015-08-13 at 12:29 +0200, Christoffer Dall wrote:
> On Thu, Aug 13, 2015 at 11:22:19AM +0100, Ian Campbell wrote:
> > On Thu, 2015-08-13 at 11:13 +0100, Stefano Stabellini wrote:
> > > 
> > > > > For example it is only natural for the kernel to try to use the 
> > > > > GIC 
> > > > > hyp
> > > > > functionalities if they are described, while actually they are 
> > > > > not
> > > > > emulated by Xen at all.
> > > > 
> > > > See Ian's earlier reply: It can also be considered natural for it 
> > > > to
> > > > be aware that when run in EL2 to not use EL1 functionality.
> > 
> > NB EL2 == Hyp and EL1 == Kernel, so it's the other way round, FWIW.
> > 
> > > It is not just about the GIC Hyp functionalities.
> > 
> > What else is there which is not subject to this logic? Timers are too, 
> > it
> > even applies to IOMMU's which have both stage1 and stage2 bits.
> > 
> > BTW, I think kernels _already_ need to deal with a lot of this because 
> > in
> > reality nobody modifies the DTB when they use a firmware which launches 
> > the
> > kernel in EL1. IOW I think the kernel is already aware of which 
> > resources
> > can be used by which privilege level.
> > 
> Yes, for resources specific to EL2 I believe that is indeed the case
> (the GIC driver doesn't look at the hypervisor control register address,
> and KVM does not even get that far if you're not booted in EL2, and the
> timer only uses the virtual timer if not booted in EL2 - we never
> attempt to use the hyp timer until Marc's VHE patches land, but they
> also depend on being booted in hyp mode).

Right, and I think that's almost always going to be the case by virtue of
the architecture. You can't use these resources from EL1 (however you got
there) so you need checks.

> However, what about for other resources?  Having code somewhere that
> says "hide this random piece of hardware if you're Xen dom0" sounds
> awful to me.  I know it's only the serial port right now, but still.

Right, that's the only bit I'm aware of that I'm not sure about, which is
why I was curious how x86 managed it (I've seen but not digested Jan's
answer).

At least this has the virtue of being an extra table, and tagging on a new
one of those has more limited impact on the tables I think.

Ian.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

On 13/08/15 10:20, Jan Beulich wrote:
>> BTW, IIRC x86 does modify at least one ACPI table which is the DMAR (I
>> think), to hide the IOMMU from the guest? That's another table we would
>> want to frob on ARM I think (or it's equivalent, which I think is IORT).
> 
> Eliminating that hack is supposed to be on the VT-d maintainers'
> TODO list(s) - Dom0 has no business looking at that table (and its
> AMD counterpart already isn't being fiddled with in the same way).

ARM SMMU is supporting 2 stage in order to protect the device also by
the domain.

At some point we will expose it to DOM0 and therefore may need to modify
IORT.

Regards,

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Shannon Zhao]

On 2015/8/13 18:29, Christoffer Dall wrote:
> On Thu, Aug 13, 2015 at 11:22:19AM +0100, Ian Campbell wrote:
>> On Thu, 2015-08-13 at 11:13 +0100, Stefano Stabellini wrote:
>>>
>>>>> For example it is only natural for the kernel to try to use the GIC 
>>>>> hyp
>>>>> functionalities if they are described, while actually they are not
>>>>> emulated by Xen at all.
>>>>
>>>> See Ian's earlier reply: It can also be considered natural for it to
>>>> be aware that when run in EL2 to not use EL1 functionality.
>>
>> NB EL2 == Hyp and EL1 == Kernel, so it's the other way round, FWIW.
>>
>>> It is not just about the GIC Hyp functionalities.
>>
>> What else is there which is not subject to this logic? Timers are too, it
>> even applies to IOMMU's which have both stage1 and stage2 bits.
>>
>> BTW, I think kernels _already_ need to deal with a lot of this because in
>> reality nobody modifies the DTB when they use a firmware which launches the
>> kernel in EL1. IOW I think the kernel is already aware of which resources
>> can be used by which privilege level.
>>
> Yes, for resources specific to EL2 I believe that is indeed the case
> (the GIC driver doesn't look at the hypervisor control register address,
> and KVM does not even get that far if you're not booted in EL2, and the
> timer only uses the virtual timer if not booted in EL2 - we never
> attempt to use the hyp timer until Marc's VHE patches land, but they
> also depend on being booted in hyp mode).
> 
> However, what about for other resources?  Having code somewhere that
> says "hide this random piece of hardware if you're Xen dom0" sounds
> awful to me.  I know it's only the serial port right now, but still.
> 
It needs to modify MADT table according to dom0_max_vcpus and modify EFI
MMAP table according to dom0_mem. And modify FADT table to set
hypervisor_id as "XenVMM" to tell Dom0 that it runs on Xen hypervisor.

--
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Jan Beulich]

>>> On 13.08.15 at 12:48, <zhaoshenglong@huawei.com> wrote:
> On 2015/8/13 18:29, Christoffer Dall wrote:
>> However, what about for other resources?  Having code somewhere that
>> says "hide this random piece of hardware if you're Xen dom0" sounds
>> awful to me.  I know it's only the serial port right now, but still.
>> 
> It needs to modify MADT table according to dom0_max_vcpus and modify EFI
> MMAP table according to dom0_mem. And modify FADT table to set
> hypervisor_id as "XenVMM" to tell Dom0 that it runs on Xen hypervisor.

None of this should be required:

Unaltered MADT may (or should I say will) be needed to drive P- or
C-states.

UEFI memmap shouldn't be exposed to Dom0 at all.

Detecting running on Xen should (hopefully) be possible by other
means for Dom0.

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

On 13/08/15 11:54, Jan Beulich wrote:
>>>> On 13.08.15 at 12:48, <zhaoshenglong@huawei.com> wrote:
>> On 2015/8/13 18:29, Christoffer Dall wrote:
>>> However, what about for other resources?  Having code somewhere that
>>> says "hide this random piece of hardware if you're Xen dom0" sounds
>>> awful to me.  I know it's only the serial port right now, but still.
>>>
>> It needs to modify MADT table according to dom0_max_vcpus and modify EFI
>> MMAP table according to dom0_mem. And modify FADT table to set
>> hypervisor_id as "XenVMM" to tell Dom0 that it runs on Xen hypervisor.
> 
> None of this should be required:
> 
> Unaltered MADT may (or should I say will) be needed to drive P- or
> C-states.

We have to alter MADT for different reasons:
	- restricting the number of vCPUs
	- update the CPU bring up method
	- changing the CPUID

Maybe we should pass a backup but we don't support cpufreq driver right
now and it would need quite a look of work to do it (see [1]).

> UEFI memmap shouldn't be exposed to Dom0 at all.

We have to craft a UEFI memmap in order to notify Dom0 where are the RAM
regions.

> Detecting running on Xen should (hopefully) be possible by other
> means for Dom0.

How there is no cpuid like on ARM. So no possibility to know whether you
are running on Xen or another hypervisor...

Modifying the FADT is the only viable solution we have today in order to
tell that it's running on Xen.

Regards,

[1]http://lists.xen.org/archives/html/xen-devel/2014-10/msg02883.html

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Stefano Stabellini]

On Thu, 13 Aug 2015, Julien Grall wrote:
> On 13/08/15 11:54, Jan Beulich wrote:
> >>>> On 13.08.15 at 12:48, <zhaoshenglong@huawei.com> wrote:
> >> On 2015/8/13 18:29, Christoffer Dall wrote:
> >>> However, what about for other resources?  Having code somewhere that
> >>> says "hide this random piece of hardware if you're Xen dom0" sounds
> >>> awful to me.  I know it's only the serial port right now, but still.
> >>>
> >> It needs to modify MADT table according to dom0_max_vcpus and modify EFI
> >> MMAP table according to dom0_mem. And modify FADT table to set
> >> hypervisor_id as "XenVMM" to tell Dom0 that it runs on Xen hypervisor.
> > 
> > None of this should be required:
> > 
> > Unaltered MADT may (or should I say will) be needed to drive P- or
> > C-states.
> 
> We have to alter MADT for different reasons:
> 	- restricting the number of vCPUs
> 	- update the CPU bring up method
> 	- changing the CPUID
> 
> Maybe we should pass a backup but we don't support cpufreq driver right
> now and it would need quite a look of work to do it (see [1]).
> 
> > UEFI memmap shouldn't be exposed to Dom0 at all.
> 
> We have to craft a UEFI memmap in order to notify Dom0 where are the RAM
> regions.

Right. Keep in mind that there are no legacy interfaces to get these
info on ARM. All we have is ACPI and EFI.


> > Detecting running on Xen should (hopefully) be possible by other
> > means for Dom0.
> 
> How there is no cpuid like on ARM. So no possibility to know whether you
> are running on Xen or another hypervisor...
> 
> Modifying the FADT is the only viable solution we have today in order to
> tell that it's running on Xen.

We also have the XENV table.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Jan Beulich]

>>> On 13.08.15 at 13:00, <julien.grall@citrix.com> wrote:
> On 13/08/15 11:54, Jan Beulich wrote:
>>>>> On 13.08.15 at 12:48, <zhaoshenglong@huawei.com> wrote:
>>> On 2015/8/13 18:29, Christoffer Dall wrote:
>>>> However, what about for other resources?  Having code somewhere that
>>>> says "hide this random piece of hardware if you're Xen dom0" sounds
>>>> awful to me.  I know it's only the serial port right now, but still.
>>>>
>>> It needs to modify MADT table according to dom0_max_vcpus and modify EFI
>>> MMAP table according to dom0_mem. And modify FADT table to set
>>> hypervisor_id as "XenVMM" to tell Dom0 that it runs on Xen hypervisor.
>> 
>> None of this should be required:
>> 
>> Unaltered MADT may (or should I say will) be needed to drive P- or
>> C-states.
> 
> We have to alter MADT for different reasons:
> 	- restricting the number of vCPUs
> 	- update the CPU bring up method
> 	- changing the CPUID
> 
> Maybe we should pass a backup but we don't support cpufreq driver right
> now and it would need quite a look of work to do it (see [1]).

I.e. you intend to not even leave open the road there.

Jan

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

On 13/08/15 12:55, Jan Beulich wrote:
>>>> On 13.08.15 at 13:00, <julien.grall@citrix.com> wrote:
>> On 13/08/15 11:54, Jan Beulich wrote:
>>>>>> On 13.08.15 at 12:48, <zhaoshenglong@huawei.com> wrote:
>>>> On 2015/8/13 18:29, Christoffer Dall wrote:
>>>>> However, what about for other resources?  Having code somewhere that
>>>>> says "hide this random piece of hardware if you're Xen dom0" sounds
>>>>> awful to me.  I know it's only the serial port right now, but still.
>>>>>
>>>> It needs to modify MADT table according to dom0_max_vcpus and modify EFI
>>>> MMAP table according to dom0_mem. And modify FADT table to set
>>>> hypervisor_id as "XenVMM" to tell Dom0 that it runs on Xen hypervisor.
>>>
>>> None of this should be required:
>>>
>>> Unaltered MADT may (or should I say will) be needed to drive P- or
>>> C-states.
>>
>> We have to alter MADT for different reasons:
>> 	- restricting the number of vCPUs
>> 	- update the CPU bring up method
>> 	- changing the CPUID
>>
>> Maybe we should pass a backup but we don't support cpufreq driver right
>> now and it would need quite a look of work to do it (see [1]).
> 
> I.e. you intend to not even leave open the road there.

Please beware that currently porting a guest to Xen on ARM is only a
couple of hundreds lines which is mainly detecting Xen and initializing
the Xen code. Other than that everything is the same as booting on
baremetal. I don't take into account the drivers as they usually make
usage of the kernel framework and don't touch boot code.

This is a strong advantage for Xen because developer would not have to
hack the internal OS in order to get running on Xen.

We are trying  to aim the same for ACPI. Any change compare to
baremetal boot will result to more work for the OS developer.

The MADT has to be modified in order to let DOM0 knows about his CPU
topology. If ever need the original MADT for power management, then we
should find a Xen specific way to do it (even if it's passing the
orginal MADT as a backup).

Regards,

-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Thu, 2015-08-13 at 13:08 +0100, Julien Grall wrote:
> 
> The MADT has to be modified in order to let DOM0 knows about his CPU
> topology. If ever need the original MADT for power management, then we
> should find a Xen specific way to do it (even if it's passing the
> orginal MADT as a backup).

Right, I was about to say something along these lines, the default set of
tables should be the ones which are expected for the guest's (including
dom0's) actual configuration and own use.

Any additional magic stuff (like host-cpufreq in dom0) already needs to do
Xen specific magic to bring itself up in a world which doesn't match that
and should request whatever real host information it needs itself, not
expect to find it in the "proper" places.

Ian.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Shannon Zhao]

On 2015/8/13 18:29, Christoffer Dall wrote:
> On Thu, Aug 13, 2015 at 11:22:19AM +0100, Ian Campbell wrote:
>> > On Thu, 2015-08-13 at 11:13 +0100, Stefano Stabellini wrote:
>>> > > 
>>>>> > > > > For example it is only natural for the kernel to try to use the GIC 
>>>>> > > > > hyp
>>>>> > > > > functionalities if they are described, while actually they are not
>>>>> > > > > emulated by Xen at all.
>>>> > > > 
>>>> > > > See Ian's earlier reply: It can also be considered natural for it to
>>>> > > > be aware that when run in EL2 to not use EL1 functionality.
>> > 
>> > NB EL2 == Hyp and EL1 == Kernel, so it's the other way round, FWIW.
>> > 
>>> > > It is not just about the GIC Hyp functionalities.
>> > 
>> > What else is there which is not subject to this logic? Timers are too, it
>> > even applies to IOMMU's which have both stage1 and stage2 bits.
>> > 
>> > BTW, I think kernels _already_ need to deal with a lot of this because in
>> > reality nobody modifies the DTB when they use a firmware which launches the
>> > kernel in EL1. IOW I think the kernel is already aware of which resources
>> > can be used by which privilege level.
>> > 
> Yes, for resources specific to EL2 I believe that is indeed the case
> (the GIC driver doesn't look at the hypervisor control register address,
> and KVM does not even get that far if you're not booted in EL2, and the
> timer only uses the virtual timer if not booted in EL2 - we never
> attempt to use the hyp timer until Marc's VHE patches land, but they
> also depend on being booted in hyp mode).
> 
> However, what about for other resources?  Having code somewhere that
> says "hide this random piece of hardware if you're Xen dom0" sounds
> awful to me.  I know it's only the serial port right now, but still.

If we remove the entry of SPCR table from XSDT table, would it work for
Dom0 to ignore the uart? And it doesn't need the STAO table?

-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Stefano Stabellini]

On Fri, 14 Aug 2015, Shannon Zhao wrote:
> On 2015/8/13 18:29, Christoffer Dall wrote:
> > On Thu, Aug 13, 2015 at 11:22:19AM +0100, Ian Campbell wrote:
> >> > On Thu, 2015-08-13 at 11:13 +0100, Stefano Stabellini wrote:
> >>> > > 
> >>>>> > > > > For example it is only natural for the kernel to try to use the GIC 
> >>>>> > > > > hyp
> >>>>> > > > > functionalities if they are described, while actually they are not
> >>>>> > > > > emulated by Xen at all.
> >>>> > > > 
> >>>> > > > See Ian's earlier reply: It can also be considered natural for it to
> >>>> > > > be aware that when run in EL2 to not use EL1 functionality.
> >> > 
> >> > NB EL2 == Hyp and EL1 == Kernel, so it's the other way round, FWIW.
> >> > 
> >>> > > It is not just about the GIC Hyp functionalities.
> >> > 
> >> > What else is there which is not subject to this logic? Timers are too, it
> >> > even applies to IOMMU's which have both stage1 and stage2 bits.
> >> > 
> >> > BTW, I think kernels _already_ need to deal with a lot of this because in
> >> > reality nobody modifies the DTB when they use a firmware which launches the
> >> > kernel in EL1. IOW I think the kernel is already aware of which resources
> >> > can be used by which privilege level.
> >> > 
> > Yes, for resources specific to EL2 I believe that is indeed the case
> > (the GIC driver doesn't look at the hypervisor control register address,
> > and KVM does not even get that far if you're not booted in EL2, and the
> > timer only uses the virtual timer if not booted in EL2 - we never
> > attempt to use the hyp timer until Marc's VHE patches land, but they
> > also depend on being booted in hyp mode).
> > 
> > However, what about for other resources?  Having code somewhere that
> > says "hide this random piece of hardware if you're Xen dom0" sounds
> > awful to me.  I know it's only the serial port right now, but still.
> 
> If we remove the entry of SPCR table from XSDT table, would it work for
> Dom0 to ignore the uart? And it doesn't need the STAO table?

I don't think that is enough because the very same uart is likely to be
described in the dsdt too. I am pretty sure that we need the STAO.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Shannon Zhao]

On 2015/8/12 17:11, Julien Grall wrote:
> On 12/08/2015 08:22, Shannon Zhao wrote:
>> Hi,
>
> Hi Shannon,
>
> It's not part of the design discussion and we are avoiding to mix
> discussion. Can you please create another thread (or at least renaming
> the subject)?
>
>> I'm working on re-spinning this patchset while encountering a werid
>> problem about xzalloc_bytes.
>>
>> Since I need to copy some ACPI tables, I need to allocate some memory
>> for it. So there are a few places calling xzalloc_bytes. And it fails at
>> the fifth one. The log is shown as following:
>
> Do you copy data in the newly allocated memory between 2 xzalloc_bytes?
>

No, I just use xzalloc_bytes to allocate some place and copy ACPI to the 
allocated place, modify the content, then call 
raw_copy_to_guest_flush_dcache to copy the modified tables to guest memory.

> The only thing I have in mind based on the log below is your are
> overriding the metadata of the memory allocator.
>
>> (XEN) *** LOADING DOMAIN 0 ***
>> (XEN) Loading kernel from boot module @ 00000008fa315000
>> (XEN) Allocating 1:1 mappings totalling 256MB for dom0:
>> (XEN) BANK[0] 0x00000090000000-0x000000a0000000 (256MB)
>> (XEN) Grant table range: 0x00000087e00000-0x00000087e5b000
>> (XEN) Loading zImage from 00000008fa315000 to
>> 0000000090080000-00000000909e0ec8
>> (XEN) Hypervisor Trap. HSR=0x96000044 EC=0x25 IL=1 Syndrome=0x44
>> (XEN) CPU0: Unexpected Trap: Hypervisor
>> (XEN) ----[ Xen-4.6-unstable  arm64  debug=y  Not tainted ]----
>> (XEN) CPU:    0
>> (XEN) PC:     0000000000238b78 xmem_pool_alloc+0x348/0x4b0
>> (XEN) LR:     0000000000238960
>> (XEN) SP:     00000000002bf4e0
>> (XEN) CPSR:   20000249 MODE:64-bit EL2h (Hypervisor, handler)
>>
>> (XEN) Xen call trace:
>> (XEN)    [<0000000000238b78>] xmem_pool_alloc+0x348/0x4b0 (PC)
>> (XEN)    [<0000000000238960>] xmem_pool_alloc+0x130/0x4b0 (LR)
>> (XEN)    [<0000000000239100>] _xmalloc+0xf4/0x290
>> (XEN)    [<00000000002392b0>] _xzalloc+0x14/0x38
>> (XEN)    [<0000000000245430>] construct_dom0+0xbc0/0x14cc
>> (XEN)    [<000000000028c4c4>] start_xen+0xbf4/0xcb0
>> (XEN)    [<000000000020060c>] paging+0x84/0xbc
>> (XEN)
>> (XEN)
>> (XEN) ****************************************
>> (XEN) Panic on CPU 0:
>> (XEN) CPU0: Unexpected Trap: Hypervisor
>> (XEN)
>> (XEN) ****************************************
>> (XEN)
>> (XEN) Reboot in five seconds...
>>
>
> Regards,
>

-- 
Shannon

xen/arm: Crash when allocating memory for ACPI table (Was Re: Design doc of adding ACPI support for arm64 on Xen - version 2)

[Julien Grall]

(Renaming the subject)

Hi Shannon,

On 14/08/15 15:05, Shannon Zhao wrote:
> 
> 
> On 2015/8/12 17:11, Julien Grall wrote:
>> On 12/08/2015 08:22, Shannon Zhao wrote:
>>> Hi,
>>
>> Hi Shannon,
>>
>> It's not part of the design discussion and we are avoiding to mix
>> discussion. Can you please create another thread (or at least renaming
>> the subject)?
>>
>>> I'm working on re-spinning this patchset while encountering a werid
>>> problem about xzalloc_bytes.
>>>
>>> Since I need to copy some ACPI tables, I need to allocate some memory
>>> for it. So there are a few places calling xzalloc_bytes. And it fails at
>>> the fifth one. The log is shown as following:
>>
>> Do you copy data in the newly allocated memory between 2 xzalloc_bytes?
>>
> 
> No, I just use xzalloc_bytes to allocate some place and copy ACPI to the
> allocated place, modify the content, then call
> raw_copy_to_guest_flush_dcache to copy the modified tables to guest memory.

Can you provide the code and show which call is crashing?

Regards,

-- 
Julien Grall

Re: xen/arm: Crash when allocating memory for ACPI table (Was Re: Design doc of adding ACPI support for arm64 on Xen - version 2)

[Shannon Zhao]

Hi Julien,

On 2015/8/14 22:17, Julien Grall wrote:
> (Renaming the subject)
>
> Hi Shannon,
>
> On 14/08/15 15:05, Shannon Zhao wrote:
>>
>>
>> On 2015/8/12 17:11, Julien Grall wrote:
>>> On 12/08/2015 08:22, Shannon Zhao wrote:
>>>> Hi,
>>>
>>> Hi Shannon,
>>>
>>> It's not part of the design discussion and we are avoiding to mix
>>> discussion. Can you please create another thread (or at least renaming
>>> the subject)?
>>>
>>>> I'm working on re-spinning this patchset while encountering a werid
>>>> problem about xzalloc_bytes.
>>>>
>>>> Since I need to copy some ACPI tables, I need to allocate some memory
>>>> for it. So there are a few places calling xzalloc_bytes. And it fails at
>>>> the fifth one. The log is shown as following:
>>>
>>> Do you copy data in the newly allocated memory between 2 xzalloc_bytes?
>>>
>>
>> No, I just use xzalloc_bytes to allocate some place and copy ACPI to the
>> allocated place, modify the content, then call
>> raw_copy_to_guest_flush_dcache to copy the modified tables to guest memory.
>
> Can you provide the code and show which call is crashing?
>
Oh, sorry. The code is not on hand as it stays at my working computer.
 From previous debug, it fails at the xzalloc_bytes. Because I add two 
printk before and after the xzalloc_bytes, only the before one shows.

The code calling route is like below:

acpi_create_fadt();
acpi_create_gtdt();
acpi_create_madt();
acpi_create_stao();
acpi_create_xsdt();
acpi_map_rsdp();
acpi_map_rest_table();
acpi_create_est();
acpi_create_mmap();
...

Within everyone of these functions, it will call xzalloc_bytes to 
allocate memory and call raw_copy_to_guest_flush_dcache to copy the 
modified tables to guest memory. And this failure happened at 
acpi_create_xsdt().

If I add xzalloc_bytes(1000) before acpi_create_xsdt() like below:

acpi_create_fadt();
acpi_create_gtdt();
acpi_create_madt();
acpi_create_stao();

xzalloc_bytes(1000);

acpi_create_xsdt();
acpi_map_rsdp();
acpi_map_rest_table();
acpi_create_est();
acpi_create_mmap();
...

The failure will not happen at acpi_create_xsdt() but at acpi_create_mmap().

-- 
Shannon

Re: xen/arm: Crash when allocating memory for ACPI table (Was Re: Design doc of adding ACPI support for arm64 on Xen - version 2)

[Julien Grall]

On 14/08/15 15:35, Shannon Zhao wrote:
>>>> Do you copy data in the newly allocated memory between 2 xzalloc_bytes?
>>>>
>>>
>>> No, I just use xzalloc_bytes to allocate some place and copy ACPI to the
>>> allocated place, modify the content, then call
>>> raw_copy_to_guest_flush_dcache to copy the modified tables to guest
>>> memory.
>>
>> Can you provide the code and show which call is crashing?
>>
> Oh, sorry. The code is not on hand as it stays at my working computer.
> From previous debug, it fails at the xzalloc_bytes. Because I add two
> printk before and after the xzalloc_bytes, only the before one shows.
> 
> The code calling route is like below:
> 
> acpi_create_fadt();
> acpi_create_gtdt();
> acpi_create_madt();
> acpi_create_stao();
> acpi_create_xsdt();
> acpi_map_rsdp();
> acpi_map_rest_table();
> acpi_create_est();
> acpi_create_mmap();
> ...
> 
> Within everyone of these functions, it will call xzalloc_bytes to
> allocate memory and call raw_copy_to_guest_flush_dcache to copy the
> modified tables to guest memory. And this failure happened at
> acpi_create_xsdt().

When I asked if you copy data between 2 calls of xzalloc_bytes you said
no ... But here you say the invert ... So do you copy data between two
call or not? (FIY, raw_copy_to_guest_flush_dcache is copying data).

> 
> If I add xzalloc_bytes(1000) before acpi_create_xsdt() like below:
> 
> acpi_create_fadt();
> acpi_create_gtdt();
> acpi_create_madt();
> acpi_create_stao();
> 
> xzalloc_bytes(1000);
> 
> acpi_create_xsdt();
> acpi_map_rsdp();
> acpi_map_rest_table();
> acpi_create_est();
> acpi_create_mmap();
> ...
> 
> The failure will not happen at acpi_create_xsdt() but at
> acpi_create_mmap().

Ok, so it's likely a memory corruption. You need to check the bound you
ara using when copying the data to the guest or from the ACPI in
general. Or maybe you just didn't allocate enough space.

Regards,

-- 
Julien Grall

Re: xen/arm: Crash when allocating memory for ACPI table (Was Re: Design doc of adding ACPI support for arm64 on Xen - version 2)

[Shannon Zhao]

On 2015/8/14 22:41, Julien Grall wrote:
> On 14/08/15 15:35, Shannon Zhao wrote:
>>>>> Do you copy data in the newly allocated memory between 2 xzalloc_bytes?
>>>>>
>>>>
>>>> No, I just use xzalloc_bytes to allocate some place and copy ACPI to the
>>>> allocated place, modify the content, then call
>>>> raw_copy_to_guest_flush_dcache to copy the modified tables to guest
>>>> memory.
>>>
>>> Can you provide the code and show which call is crashing?
>>>
>> Oh, sorry. The code is not on hand as it stays at my working computer.
>>  From previous debug, it fails at the xzalloc_bytes. Because I add two
>> printk before and after the xzalloc_bytes, only the before one shows.
>>
>> The code calling route is like below:
>>
>> acpi_create_fadt();
>> acpi_create_gtdt();
>> acpi_create_madt();
>> acpi_create_stao();
>> acpi_create_xsdt();
>> acpi_map_rsdp();
>> acpi_map_rest_table();
>> acpi_create_est();
>> acpi_create_mmap();
>> ...
>>
>> Within everyone of these functions, it will call xzalloc_bytes to
>> allocate memory and call raw_copy_to_guest_flush_dcache to copy the
>> modified tables to guest memory. And this failure happened at
>> acpi_create_xsdt().
>
> When I asked if you copy data between 2 calls of xzalloc_bytes you said
> no ... But here you say the invert ... So do you copy data between two
> call or not? (FIY, raw_copy_to_guest_flush_dcache is copying data).
>

Oh, I thought you mean that if I copy data between the two places 
allocated by xzalloc_bytes.

>>
>> If I add xzalloc_bytes(1000) before acpi_create_xsdt() like below:
>>
>> acpi_create_fadt();
>> acpi_create_gtdt();
>> acpi_create_madt();
>> acpi_create_stao();
>>
>> xzalloc_bytes(1000);
>>
>> acpi_create_xsdt();
>> acpi_map_rsdp();
>> acpi_map_rest_table();
>> acpi_create_est();
>> acpi_create_mmap();
>> ...
>>
>> The failure will not happen at acpi_create_xsdt() but at
>> acpi_create_mmap().
>
> Ok, so it's likely a memory corruption. You need to check the bound you
> ara using when copying the data to the guest or from the ACPI in
> general. Or maybe you just didn't allocate enough space.
>

But it fails at the xzalloc_bytes itself. not at copy function.

-- 
Shannon

Re: xen/arm: Crash when allocating memory for ACPI table (Was Re: Design doc of adding ACPI support for arm64 on Xen - version 2)

[Julien Grall]

On 14/08/15 15:49, Shannon Zhao wrote:
>> Ok, so it's likely a memory corruption. You need to check the bound you
>> ara using when copying the data to the guest or from the ACPI in
>> general. Or maybe you just didn't allocate enough space.
>>
> 
> But it fails at the xzalloc_bytes itself. not at copy function.

Because the previous copy may have overwritten the metadata of the
memory allocator...

If those metadata are corrupted, xalloc_bytes we act weirdly such as
crashing Xen.

Regards,

-- 
Julien Grall

Re: xen/arm: Crash when allocating memory for ACPI table (Was Re: Design doc of adding ACPI support for arm64 on Xen - version 2)

[Shannon Zhao]

On 2015/8/14 22:53, Julien Grall wrote:
> On 14/08/15 15:49, Shannon Zhao wrote:
>>> Ok, so it's likely a memory corruption. You need to check the bound you
>>> ara using when copying the data to the guest or from the ACPI in
>>> general. Or maybe you just didn't allocate enough space.
>>>
>>
>> But it fails at the xzalloc_bytes itself. not at copy function.
>
> Because the previous copy may have overwritten the metadata of the
> memory allocator...
>
> If those metadata are corrupted, xalloc_bytes we act weirdly such as
> crashing Xen.
>
Ok, will check it. Thanks.

-- 
Shannon

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Roger Pau Monné]

El 11/08/15 a les 16.19, Ian Campbell ha escrit:
> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
>> This document is going to explain the design details of Xen booting with
>> ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are
>> welcome.
> 
> Some small subsets of this seem like they might overlap with what will be
> required for PVH on x86 (a new x86 guest mode not dissimilar to the sole
> ARM guest mode). If so then it would be preferable IMHO if PVH x86 could
> use the same interfaces.
> 
> I've trimmed the quotes to just those bits and CCd some of the PVH people
> (Boris and Roger[0]) in case they have any thoughts.
> 
> Actually, having done the trimming there is only one such bit:
> 
> [...]
>> 4. Map MMIO regions
>> -------------------
>> Register a bus_notifier for platform and amba bus in Linux. Add a new
>> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the
>> device is newly added, then call hypercall XENMEM_add_to_physmap to map
>> the mmio regions.
> 
> Ian.
> 
> [0] Roger is away for a week or so, but I'm expect feedback to be of the
> "we could use one extra field" type rather than "this needs to be done some
> totally different way for x86/PVH" (in which case we wouldn't want to share
> the interface anyway I suppose) so need to block on awaiting that feedback.

This looks right to me. AFAICT this new memory space
(XENMAPSPACE_dev_mmio) will only be available to the hardware domain on
x86. I expect that for DomUs the toolstack will already map the
appropriate MMIO regions when creating the domain if there are
pass-through devices assigned, not sure if that's also the plan on ARM.
IMHO this document should also list the usage of the hypercall parameters:

- space: XENMAPSPACE_dev_mmio.
- idxs: native physical addresses.
- gpfns: guest physical addresses where the mapping should appear.

This is quite obvious but I think it's worth spelling it out.

Roger.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Shannon Zhao]

On 2015/8/17 18:36, Roger Pau Monné wrote:
> El 11/08/15 a les 16.19, Ian Campbell ha escrit:
>> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote:
>>> This document is going to explain the design details of Xen booting with
>>> ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are
>>> welcome.
>>
>> Some small subsets of this seem like they might overlap with what will be
>> required for PVH on x86 (a new x86 guest mode not dissimilar to the sole
>> ARM guest mode). If so then it would be preferable IMHO if PVH x86 could
>> use the same interfaces.
>>
>> I've trimmed the quotes to just those bits and CCd some of the PVH people
>> (Boris and Roger[0]) in case they have any thoughts.
>>
>> Actually, having done the trimming there is only one such bit:
>>
>> [...]
>>> 4. Map MMIO regions
>>> -------------------
>>> Register a bus_notifier for platform and amba bus in Linux. Add a new
>>> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the
>>> device is newly added, then call hypercall XENMEM_add_to_physmap to map
>>> the mmio regions.
>>
>> Ian.
>>
>> [0] Roger is away for a week or so, but I'm expect feedback to be of the
>> "we could use one extra field" type rather than "this needs to be done some
>> totally different way for x86/PVH" (in which case we wouldn't want to share
>> the interface anyway I suppose) so need to block on awaiting that feedback.
> 
> This looks right to me. AFAICT this new memory space
> (XENMAPSPACE_dev_mmio) will only be available to the hardware domain on
> x86. I expect that for DomUs the toolstack will already map the
> appropriate MMIO regions when creating the domain if there are
> pass-through devices assigned, not sure if that's also the plan on ARM.
> IMHO this document should also list the usage of the hypercall parameters:
> 
> - space: XENMAPSPACE_dev_mmio.
> - idxs: native physical addresses.
> - gpfns: guest physical addresses where the mapping should appear.
> 
> This is quite obvious but I think it's worth spelling it out.
> 
Will add. Thanks :)

-- 
Shannon


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Wed, 2015-08-12 at 13:11 +0100, Julien Grall wrote:
> On 12/08/15 12:23, Ian Campbell wrote:
> > Strictly it is considered a separate thing, much like loader.efi, despite
> > where it lives e.g. it is self contained and not allowed to call into the
> > kernel proper except via the formal interface provided for the hand-off.
> > 
> > That might seem like semantic quibbling, but I just want to clarify that
> > the Linux and BSD approaches here are basically the same.
> > 
> > Given that these device tree bindings are really just Linux's equivalent of
> > the "a format the kernel understands" which BSD uses as described above. I
> > don't know what format BSD uses, Linux just happened to have a DTB library
> > handy...
> 
> IIRC, on FreeBSD, the loader and the kernel is talking through a custom
> format called metadata. There is no modification of device tree by the
> loader.

Right, and as I say above Linux's equivalent of "a custom format called
metadata" is to declare a device tree binding.

> Although, I would prefer to see a common interface between Xen and DOM0
> rather than implementing a custom one for each OS we will support.

Agreed.

> I have to think how everything will work together. AFAIK, on x86, the
> loader is loading Xen and the FreeBSD kernel in the memory. The metadata
> necessary for the kernel is passed as a multiboot entry.
> 
> I will speak with Royger to see what we can do here.
> 
> In the meantime, I think we should drop "linux," when we standardize
> them to show that they are generic and not linux specific.

"we" here would be broader than just Xen of course.

Ian.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Wed, 2015-08-12 at 13:20 +0100, Julien Grall wrote:
> On 12/08/15 11:36, Andrew Turner wrote:
> > Would it be possible to add a stdout property and node for the hvc0
> > device? It would help FreeBSD as we use this to find the kernel
> > console. We check for the stdout-path, linux,stdout-path, stdout,
> > stdin-path, and stdin properties, in that order, with the first
> > property selected as the console. If none are found we fall back to
> > searching for a serial0 device. You can see how we find the device at
> > [1].
> 
> the stdout-path property is used by Xen in order to get the UART. The
> property will be removed from the device tree passed to DOM0.
> 
> The Xen console is not UART a driver so having a property with "hvc"
> wouldn't help here.

If it were helpful we could of course create some sort of dummy node or do
something else to make it work. But it really shouldn't be necessary
because...

> Although, when running on Xen, FreeBSD is creating a Xen console by
> default with the higher priority. It will be grabbed by FreeBSD as the
> default one.

... Linux should be doing the same. The problem is that the existing code
to call add_preferred_console doesn't happen early enough on ARM

Someone (you? Ard? Stefano?) has a patch to move Xen detection in Linux
earlier ages ago, but I think it failed to actually get in.

Ian.

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Julien Grall]

Hi Ian,

On 02/09/15 12:30, Ian Campbell wrote:
>> Although, when running on Xen, FreeBSD is creating a Xen console by
>> default with the higher priority. It will be grabbed by FreeBSD as the
>> default one.
> 
> ... Linux should be doing the same. The problem is that the existing code
> to call add_preferred_console doesn't happen early enough on ARM
> 
> Someone (you? Ard? Stefano?) has a patch to move Xen detection in Linux
> earlier ages ago, but I think it failed to actually get in.

The patch is present in Linux 4.2:

commit f1dddd118c555508ce383b7262f4e6440927bdf4
Author: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Date:   Wed May 6 14:14:22 2015 +0000

    xen/arm: allow console=hvc0 to be omitted for guests
    
    From: Ard Biesheuvel <ard.biesheuvel@linaro.org>
    
    This patch registers hvc0 as the preferred console if no console
    has been specified explicitly on the kernel command line.
    
    The purpose is to allow platform agnostic kernels and boot images
    (such as distro installers) to boot in a Xen/ARM domU without the
    need to modify the command line by hand.
    
    Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
    Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
    Reviewed-by: Julien Grall <julien.grall@linaro.org>


-- 
Julien Grall

Re: Design doc of adding ACPI support for arm64 on Xen - version 2

[Ian Campbell]

On Wed, 2015-09-02 at 12:39 +0100, Julien Grall wrote:
> Hi Ian,
> 
> On 02/09/15 12:30, Ian Campbell wrote:
> > > Although, when running on Xen, FreeBSD is creating a Xen console by
> > > default with the higher priority. It will be grabbed by FreeBSD as 
> > > the
> > > default one.
> > 
> > ... Linux should be doing the same. The problem is that the existing 
> > code
> > to call add_preferred_console doesn't happen early enough on ARM
> > 
> > Someone (you? Ard? Stefano?) has a patch to move Xen detection in Linux
> > earlier ages ago, but I think it failed to actually get in.
> 
> The patch is present in Linux 4.2:

Not the patch I was thinking of, but since it achieves the end goal another
way: Yeehaw!

> commit f1dddd118c555508ce383b7262f4e6440927bdf4
> Author: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> Date:   Wed May 6 14:14:22 2015 +0000
> 
>     xen/arm: allow console=hvc0 to be omitted for guests
>     
>     From: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>     
>     This patch registers hvc0 as the preferred console if no console
>     has been specified explicitly on the kernel command line.
>     
>     The purpose is to allow platform agnostic kernels and boot images
>     (such as distro installers) to boot in a Xen/ARM domU without the
>     need to modify the command line by hand.
>     
>     Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>     Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
>     Reviewed-by: Julien Grall <julien.grall@linaro.org>
> 
>