[RFC] ARM VM System Sepcification

[RFC] ARM VM System Sepcification

[Christoffer Dall]

ARM VM System Specification
===========================

Goal
----
The goal of this spec is to allow suitably-built OS images to run on
all ARM virtualization solutions, such as KVM or Xen.

Recommendations in this spec are valid for aarch32 and aarch64 alike, and
they aim to be hypervisor agnostic.

Note that simply adhering to the SBSA [2] is not a valid approach,
for example because the SBSA mandates EL2, which will not be available
for VMs.  Further, the SBSA mandates peripherals like the pl011, which
may be controversial for some ARM VM implementations to support.
This spec also covers the aarch32 execution mode, not covered in the
SBSA.


Image format
------------
The image format, as presented to the VM, needs to be well-defined in
order for prepared disk images to be bootable across various
virtualization implementations.

The raw disk format as presented to the VM must be partitioned with a
GUID Partition Table (GPT).  The bootable software must be placed in the
EFI System Partition (ESP), using the UEFI removable media path, and
must be an EFI application complying to the UEFI Specification 2.4
Revision A [6].

The ESP partition's GPT entry's partition type GUID must be
C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
formatted as FAT32/vfat as per Section 12.3.1.1 in [6].

The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
state.

This ensures that tools for both Xen and KVM can load a binary UEFI
firmware which can read and boot the EFI application in the disk image.

A typical scenario will be GRUB2 packaged as an EFI application, which
mounts the system boot partition and boots Linux.


Virtual Firmware
----------------
The VM system must be able to boot the EFI application in the ESP.  It
is recommended that this is achieved by loading a UEFI binary as the
first software executed by the VM, which then executes the EFI
application.  The UEFI implementation should be compliant with UEFI
Specification 2.4 Revision A [6] or later.

This document strongly recommends that the VM implementation supports
persistent environment storage for virtual firmware implementation in
order to ensure probable use cases such as adding additional disk images
to a VM or running installers to perform upgrades.

The binary UEFI firmware implementation should not be distributed as
part of the VM image, but is specific to the VM implementation.


Hardware Description
--------------------
The Linux kernel's proper entry point always takes a pointer to an FDT,
regardless of the boot mechanism, firmware, and hardware description
method.  Even on real hardware which only supports ACPI and UEFI, the kernel
entry point will still receive a pointer to a simple FDT, generated by
the Linux kernel UEFI stub, containing a pointer to the UEFI system
table.  The kernel can then discover ACPI from the system tables.  The
presence of ACPI vs. FDT is therefore always itself discoverable,
through the FDT.

Therefore, the VM implementation must provide through its UEFI
implementation, either:

  a complete FDT which describes the entire VM system and will boot
  mainline kernels driven by device tree alone, or

  no FDT.  In this case, the VM implementation must provide ACPI, and
  the OS must be able to locate the ACPI root pointer through the UEFI
  system table.

For more information about the arm and arm64 boot conventions, see
Documentation/arm/Booting and Documentation/arm64/booting.txt in the
Linux kernel source tree.

For more information about UEFI and ACPI booting, see [4] and [5].


VM Platform
-----------
The specification does not mandate any specific memory map.  The guest
OS must be able to enumerate all processing elements, devices, and
memory through HW description data (FDT, ACPI) or a bus-specific
mechanism such as PCI.

The virtual platform must support at least one of the following ARM
execution states:
  (1) aarch32 virtual CPUs on aarch32 physical CPUs
  (2) aarch32 virtual CPUs on aarch64 physical CPUs
  (3) aarch64 virtual CPUs on aarch64 physical CPUs

It is recommended to support both (2) and (3) on aarch64 capable
physical systems.

The virtual hardware platform must provide a number of mandatory
peripherals:

  Serial console:  The platform should provide a console,
  based on an emulated pl011, a virtio-console, or a Xen PV console.

  An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
  limits the the number of virtual CPUs to 8 cores, newer GIC versions
  removes this limitation.

  The ARM virtual timer and counter should be available to the VM as
  per the ARM Generic Timers specification in the ARM ARM [1].

  A hotpluggable bus to support hotplug of at least block and network
  devices.  Suitable buses include a virtual PCIe bus and the Xen PV
  bus.


We make the following recommendations for the guest OS kernel:

  The guest OS must include support for GICv2 and any available newer
  version of the GIC architecture to maintain compatibility with older
  VM implementations.

  It is strongly recommended to include support for all available
  (block, network, console, balloon) virtio-pci, virtio-mmio, and Xen PV
  drivers in the guest OS kernel or initial ramdisk.


Other common peripherals for block devices, networking, and more can
(and typically will) be provided, but OS software written and compiled
to run on ARM VMs cannot make any assumptions about which variations
of these should exist or which implementation they use (e.g. VirtIO or
Xen PV).  See "Hardware Description" above.

Note that this platform specification is separate from the Linux kernel
concept of mach-virt, which merely specifies a machine model driven
purely from device tree, but does not mandate any peripherals or have any
mention of ACPI.


References
----------
[1]: The ARM Architecture Reference Manual, ARMv8, Issue A.b
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html

[2]: ARM Server Base System Architecture
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0029/index.html

[3]: The ARM Generic Interrupt Controller Architecture Specifications v2.0
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html

[4]: http://www.secretlab.ca/archives/27

[5]: https://git.linaro.org/people/leif.lindholm/linux.git/blob/refs/heads/uefi-for-upstream:/Documentation/arm/uefi.txt

[6]: UEFI Specification 2.4 Revision A
http://www.uefi.org/sites/default/files/resources/2_4_Errata_A.pdf

[RFC] ARM VM System Sepcification

[Christoffer Dall]

ARM VM System Specification
===========================

Goal
----
The goal of this spec is to allow suitably-built OS images to run on
all ARM virtualization solutions, such as KVM or Xen.

Recommendations in this spec are valid for aarch32 and aarch64 alike, and
they aim to be hypervisor agnostic.

Note that simply adhering to the SBSA [2] is not a valid approach,
for example because the SBSA mandates EL2, which will not be available
for VMs.  Further, the SBSA mandates peripherals like the pl011, which
may be controversial for some ARM VM implementations to support.
This spec also covers the aarch32 execution mode, not covered in the
SBSA.


Image format
------------
The image format, as presented to the VM, needs to be well-defined in
order for prepared disk images to be bootable across various
virtualization implementations.

The raw disk format as presented to the VM must be partitioned with a
GUID Partition Table (GPT).  The bootable software must be placed in the
EFI System Partition (ESP), using the UEFI removable media path, and
must be an EFI application complying to the UEFI Specification 2.4
Revision A [6].

The ESP partition's GPT entry's partition type GUID must be
C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
formatted as FAT32/vfat as per Section 12.3.1.1 in [6].

The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
state.

This ensures that tools for both Xen and KVM can load a binary UEFI
firmware which can read and boot the EFI application in the disk image.

A typical scenario will be GRUB2 packaged as an EFI application, which
mounts the system boot partition and boots Linux.


Virtual Firmware
----------------
The VM system must be able to boot the EFI application in the ESP.  It
is recommended that this is achieved by loading a UEFI binary as the
first software executed by the VM, which then executes the EFI
application.  The UEFI implementation should be compliant with UEFI
Specification 2.4 Revision A [6] or later.

This document strongly recommends that the VM implementation supports
persistent environment storage for virtual firmware implementation in
order to ensure probable use cases such as adding additional disk images
to a VM or running installers to perform upgrades.

The binary UEFI firmware implementation should not be distributed as
part of the VM image, but is specific to the VM implementation.


Hardware Description
--------------------
The Linux kernel's proper entry point always takes a pointer to an FDT,
regardless of the boot mechanism, firmware, and hardware description
method.  Even on real hardware which only supports ACPI and UEFI, the kernel
entry point will still receive a pointer to a simple FDT, generated by
the Linux kernel UEFI stub, containing a pointer to the UEFI system
table.  The kernel can then discover ACPI from the system tables.  The
presence of ACPI vs. FDT is therefore always itself discoverable,
through the FDT.

Therefore, the VM implementation must provide through its UEFI
implementation, either:

  a complete FDT which describes the entire VM system and will boot
  mainline kernels driven by device tree alone, or

  no FDT.  In this case, the VM implementation must provide ACPI, and
  the OS must be able to locate the ACPI root pointer through the UEFI
  system table.

For more information about the arm and arm64 boot conventions, see
Documentation/arm/Booting and Documentation/arm64/booting.txt in the
Linux kernel source tree.

For more information about UEFI and ACPI booting, see [4] and [5].


VM Platform
-----------
The specification does not mandate any specific memory map.  The guest
OS must be able to enumerate all processing elements, devices, and
memory through HW description data (FDT, ACPI) or a bus-specific
mechanism such as PCI.

The virtual platform must support at least one of the following ARM
execution states:
  (1) aarch32 virtual CPUs on aarch32 physical CPUs
  (2) aarch32 virtual CPUs on aarch64 physical CPUs
  (3) aarch64 virtual CPUs on aarch64 physical CPUs

It is recommended to support both (2) and (3) on aarch64 capable
physical systems.

The virtual hardware platform must provide a number of mandatory
peripherals:

  Serial console:  The platform should provide a console,
  based on an emulated pl011, a virtio-console, or a Xen PV console.

  An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
  limits the the number of virtual CPUs to 8 cores, newer GIC versions
  removes this limitation.

  The ARM virtual timer and counter should be available to the VM as
  per the ARM Generic Timers specification in the ARM ARM [1].

  A hotpluggable bus to support hotplug of at least block and network
  devices.  Suitable buses include a virtual PCIe bus and the Xen PV
  bus.


We make the following recommendations for the guest OS kernel:

  The guest OS must include support for GICv2 and any available newer
  version of the GIC architecture to maintain compatibility with older
  VM implementations.

  It is strongly recommended to include support for all available
  (block, network, console, balloon) virtio-pci, virtio-mmio, and Xen PV
  drivers in the guest OS kernel or initial ramdisk.


Other common peripherals for block devices, networking, and more can
(and typically will) be provided, but OS software written and compiled
to run on ARM VMs cannot make any assumptions about which variations
of these should exist or which implementation they use (e.g. VirtIO or
Xen PV).  See "Hardware Description" above.

Note that this platform specification is separate from the Linux kernel
concept of mach-virt, which merely specifies a machine model driven
purely from device tree, but does not mandate any peripherals or have any
mention of ACPI.


References
----------
[1]: The ARM Architecture Reference Manual, ARMv8, Issue A.b
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html

[2]: ARM Server Base System Architecture
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0029/index.html

[3]: The ARM Generic Interrupt Controller Architecture Specifications v2.0
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html

[4]: http://www.secretlab.ca/archives/27

[5]: https://git.linaro.org/people/leif.lindholm/linux.git/blob/refs/heads/uefi-for-upstream:/Documentation/arm/uefi.txt

[6]: UEFI Specification 2.4 Revision A
http://www.uefi.org/sites/default/files/resources/2_4_Errata_A.pdf

Re: [RFC] ARM VM System Sepcification

[Christopher Covington]

Hi Christoffer,

On 02/26/2014 01:34 PM, Christoffer Dall wrote:
> ARM VM System Specification
> ===========================
> 
> Goal
> ----
> The goal of this spec is to allow suitably-built OS images to run on
> all ARM virtualization solutions, such as KVM or Xen.

Would you consider including simulators/emulators as well, such as QEMU in TCG
mode?

> Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> they aim to be hypervisor agnostic.
> 
> Note that simply adhering to the SBSA [2] is not a valid approach,
> for example because the SBSA mandates EL2, which will not be available
> for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> may be controversial for some ARM VM implementations to support.
> This spec also covers the aarch32 execution mode, not covered in the
> SBSA.
> 
> 
> Image format
> ------------
> The image format, as presented to the VM, needs to be well-defined in
> order for prepared disk images to be bootable across various
> virtualization implementations.
> 
> The raw disk format as presented to the VM must be partitioned with a
> GUID Partition Table (GPT).  The bootable software must be placed in the
> EFI System Partition (ESP), using the UEFI removable media path, and
> must be an EFI application complying to the UEFI Specification 2.4
> Revision A [6].
> 
> The ESP partition's GPT entry's partition type GUID must be
> C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> 
> The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> state.
> 
> This ensures that tools for both Xen and KVM can load a binary UEFI
> firmware which can read and boot the EFI application in the disk image.
> 
> A typical scenario will be GRUB2 packaged as an EFI application, which
> mounts the system boot partition and boots Linux.
> 
> 
> Virtual Firmware
> ----------------
> The VM system must be able to boot the EFI application in the ESP.  It
> is recommended that this is achieved by loading a UEFI binary as the
> first software executed by the VM, which then executes the EFI
> application.  The UEFI implementation should be compliant with UEFI
> Specification 2.4 Revision A [6] or later.
> 
> This document strongly recommends that the VM implementation supports
> persistent environment storage for virtual firmware implementation in
> order to ensure probable use cases such as adding additional disk images
> to a VM or running installers to perform upgrades.
> 
> The binary UEFI firmware implementation should not be distributed as
> part of the VM image, but is specific to the VM implementation.

Can you elaborate on the motivation for requiring that the kernel be stuffed
into a disk image and for requiring such a heavyweight bootloader/firmware? By
doing so you would seem to exclude those requiring an optimized boot process.

> Hardware Description
> --------------------
> The Linux kernel's proper entry point always takes a pointer to an FDT,
> regardless of the boot mechanism, firmware, and hardware description
> method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> entry point will still receive a pointer to a simple FDT, generated by
> the Linux kernel UEFI stub, containing a pointer to the UEFI system
> table.  The kernel can then discover ACPI from the system tables.  The
> presence of ACPI vs. FDT is therefore always itself discoverable,
> through the FDT.
> 
> Therefore, the VM implementation must provide through its UEFI
> implementation, either:
> 
>   a complete FDT which describes the entire VM system and will boot
>   mainline kernels driven by device tree alone, or
> 
>   no FDT.  In this case, the VM implementation must provide ACPI, and
>   the OS must be able to locate the ACPI root pointer through the UEFI
>   system table.
> 
> For more information about the arm and arm64 boot conventions, see
> Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> Linux kernel source tree.
> 
> For more information about UEFI and ACPI booting, see [4] and [5].
> 
> 
> VM Platform
> -----------
> The specification does not mandate any specific memory map.  The guest
> OS must be able to enumerate all processing elements, devices, and
> memory through HW description data (FDT, ACPI) or a bus-specific
> mechanism such as PCI.
> 
> The virtual platform must support at least one of the following ARM
> execution states:
>   (1) aarch32 virtual CPUs on aarch32 physical CPUs
>   (2) aarch32 virtual CPUs on aarch64 physical CPUs
>   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> 
> It is recommended to support both (2) and (3) on aarch64 capable
> physical systems.
> 
> The virtual hardware platform must provide a number of mandatory
> peripherals:
> 
>   Serial console:  The platform should provide a console,
>   based on an emulated pl011, a virtio-console, or a Xen PV console.
> 
>   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
>   limits the the number of virtual CPUs to 8 cores, newer GIC versions
>   removes this limitation.
> 
>   The ARM virtual timer and counter should be available to the VM as
>   per the ARM Generic Timers specification in the ARM ARM [1].
> 
>   A hotpluggable bus to support hotplug of at least block and network
>   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
>   bus.

Is VirtIO hotplug capable? Over PCI or MMIO transports or both?

> We make the following recommendations for the guest OS kernel:
> 
>   The guest OS must include support for GICv2 and any available newer
>   version of the GIC architecture to maintain compatibility with older
>   VM implementations.
> 
>   It is strongly recommended to include support for all available
>   (block, network, console, balloon) virtio-pci, virtio-mmio, and Xen PV
>   drivers in the guest OS kernel or initial ramdisk.

I would love to eventually see some defconfigs for this sort of thing.

> Other common peripherals for block devices, networking, and more can
> (and typically will) be provided, but OS software written and compiled
> to run on ARM VMs cannot make any assumptions about which variations
> of these should exist or which implementation they use (e.g. VirtIO or
> Xen PV).  See "Hardware Description" above.
> 
> Note that this platform specification is separate from the Linux kernel
> concept of mach-virt, which merely specifies a machine model driven
> purely from device tree, but does not mandate any peripherals or have any
> mention of ACPI.

Well, the commit message for it said it mandated a GIC and architected timers.

Regards,
Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

[RFC] ARM VM System Sepcification

[Christopher Covington]

Hi Christoffer,

On 02/26/2014 01:34 PM, Christoffer Dall wrote:
> ARM VM System Specification
> ===========================
> 
> Goal
> ----
> The goal of this spec is to allow suitably-built OS images to run on
> all ARM virtualization solutions, such as KVM or Xen.

Would you consider including simulators/emulators as well, such as QEMU in TCG
mode?

> Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> they aim to be hypervisor agnostic.
> 
> Note that simply adhering to the SBSA [2] is not a valid approach,
> for example because the SBSA mandates EL2, which will not be available
> for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> may be controversial for some ARM VM implementations to support.
> This spec also covers the aarch32 execution mode, not covered in the
> SBSA.
> 
> 
> Image format
> ------------
> The image format, as presented to the VM, needs to be well-defined in
> order for prepared disk images to be bootable across various
> virtualization implementations.
> 
> The raw disk format as presented to the VM must be partitioned with a
> GUID Partition Table (GPT).  The bootable software must be placed in the
> EFI System Partition (ESP), using the UEFI removable media path, and
> must be an EFI application complying to the UEFI Specification 2.4
> Revision A [6].
> 
> The ESP partition's GPT entry's partition type GUID must be
> C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> 
> The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> state.
> 
> This ensures that tools for both Xen and KVM can load a binary UEFI
> firmware which can read and boot the EFI application in the disk image.
> 
> A typical scenario will be GRUB2 packaged as an EFI application, which
> mounts the system boot partition and boots Linux.
> 
> 
> Virtual Firmware
> ----------------
> The VM system must be able to boot the EFI application in the ESP.  It
> is recommended that this is achieved by loading a UEFI binary as the
> first software executed by the VM, which then executes the EFI
> application.  The UEFI implementation should be compliant with UEFI
> Specification 2.4 Revision A [6] or later.
> 
> This document strongly recommends that the VM implementation supports
> persistent environment storage for virtual firmware implementation in
> order to ensure probable use cases such as adding additional disk images
> to a VM or running installers to perform upgrades.
> 
> The binary UEFI firmware implementation should not be distributed as
> part of the VM image, but is specific to the VM implementation.

Can you elaborate on the motivation for requiring that the kernel be stuffed
into a disk image and for requiring such a heavyweight bootloader/firmware? By
doing so you would seem to exclude those requiring an optimized boot process.

> Hardware Description
> --------------------
> The Linux kernel's proper entry point always takes a pointer to an FDT,
> regardless of the boot mechanism, firmware, and hardware description
> method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> entry point will still receive a pointer to a simple FDT, generated by
> the Linux kernel UEFI stub, containing a pointer to the UEFI system
> table.  The kernel can then discover ACPI from the system tables.  The
> presence of ACPI vs. FDT is therefore always itself discoverable,
> through the FDT.
> 
> Therefore, the VM implementation must provide through its UEFI
> implementation, either:
> 
>   a complete FDT which describes the entire VM system and will boot
>   mainline kernels driven by device tree alone, or
> 
>   no FDT.  In this case, the VM implementation must provide ACPI, and
>   the OS must be able to locate the ACPI root pointer through the UEFI
>   system table.
> 
> For more information about the arm and arm64 boot conventions, see
> Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> Linux kernel source tree.
> 
> For more information about UEFI and ACPI booting, see [4] and [5].
> 
> 
> VM Platform
> -----------
> The specification does not mandate any specific memory map.  The guest
> OS must be able to enumerate all processing elements, devices, and
> memory through HW description data (FDT, ACPI) or a bus-specific
> mechanism such as PCI.
> 
> The virtual platform must support at least one of the following ARM
> execution states:
>   (1) aarch32 virtual CPUs on aarch32 physical CPUs
>   (2) aarch32 virtual CPUs on aarch64 physical CPUs
>   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> 
> It is recommended to support both (2) and (3) on aarch64 capable
> physical systems.
> 
> The virtual hardware platform must provide a number of mandatory
> peripherals:
> 
>   Serial console:  The platform should provide a console,
>   based on an emulated pl011, a virtio-console, or a Xen PV console.
> 
>   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
>   limits the the number of virtual CPUs to 8 cores, newer GIC versions
>   removes this limitation.
> 
>   The ARM virtual timer and counter should be available to the VM as
>   per the ARM Generic Timers specification in the ARM ARM [1].
> 
>   A hotpluggable bus to support hotplug of at least block and network
>   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
>   bus.

Is VirtIO hotplug capable? Over PCI or MMIO transports or both?

> We make the following recommendations for the guest OS kernel:
> 
>   The guest OS must include support for GICv2 and any available newer
>   version of the GIC architecture to maintain compatibility with older
>   VM implementations.
> 
>   It is strongly recommended to include support for all available
>   (block, network, console, balloon) virtio-pci, virtio-mmio, and Xen PV
>   drivers in the guest OS kernel or initial ramdisk.

I would love to eventually see some defconfigs for this sort of thing.

> Other common peripherals for block devices, networking, and more can
> (and typically will) be provided, but OS software written and compiled
> to run on ARM VMs cannot make any assumptions about which variations
> of these should exist or which implementation they use (e.g. VirtIO or
> Xen PV).  See "Hardware Description" above.
> 
> Note that this platform specification is separate from the Linux kernel
> concept of mach-virt, which merely specifies a machine model driven
> purely from device tree, but does not mandate any peripherals or have any
> mention of ACPI.

Well, the commit message for it said it mandated a GIC and architected timers.

Regards,
Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

Re: [RFC] ARM VM System Sepcification

[Christopher Covington]

Hi Christoffer,

On 02/26/2014 01:34 PM, Christoffer Dall wrote:
> ARM VM System Specification
> ===========================
> 
> Goal
> ----
> The goal of this spec is to allow suitably-built OS images to run on
> all ARM virtualization solutions, such as KVM or Xen.

Would you consider including simulators/emulators as well, such as QEMU in TCG
mode?

> Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> they aim to be hypervisor agnostic.
> 
> Note that simply adhering to the SBSA [2] is not a valid approach,
> for example because the SBSA mandates EL2, which will not be available
> for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> may be controversial for some ARM VM implementations to support.
> This spec also covers the aarch32 execution mode, not covered in the
> SBSA.
> 
> 
> Image format
> ------------
> The image format, as presented to the VM, needs to be well-defined in
> order for prepared disk images to be bootable across various
> virtualization implementations.
> 
> The raw disk format as presented to the VM must be partitioned with a
> GUID Partition Table (GPT).  The bootable software must be placed in the
> EFI System Partition (ESP), using the UEFI removable media path, and
> must be an EFI application complying to the UEFI Specification 2.4
> Revision A [6].
> 
> The ESP partition's GPT entry's partition type GUID must be
> C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> 
> The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> state.
> 
> This ensures that tools for both Xen and KVM can load a binary UEFI
> firmware which can read and boot the EFI application in the disk image.
> 
> A typical scenario will be GRUB2 packaged as an EFI application, which
> mounts the system boot partition and boots Linux.
> 
> 
> Virtual Firmware
> ----------------
> The VM system must be able to boot the EFI application in the ESP.  It
> is recommended that this is achieved by loading a UEFI binary as the
> first software executed by the VM, which then executes the EFI
> application.  The UEFI implementation should be compliant with UEFI
> Specification 2.4 Revision A [6] or later.
> 
> This document strongly recommends that the VM implementation supports
> persistent environment storage for virtual firmware implementation in
> order to ensure probable use cases such as adding additional disk images
> to a VM or running installers to perform upgrades.
> 
> The binary UEFI firmware implementation should not be distributed as
> part of the VM image, but is specific to the VM implementation.

Can you elaborate on the motivation for requiring that the kernel be stuffed
into a disk image and for requiring such a heavyweight bootloader/firmware? By
doing so you would seem to exclude those requiring an optimized boot process.

> Hardware Description
> --------------------
> The Linux kernel's proper entry point always takes a pointer to an FDT,
> regardless of the boot mechanism, firmware, and hardware description
> method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> entry point will still receive a pointer to a simple FDT, generated by
> the Linux kernel UEFI stub, containing a pointer to the UEFI system
> table.  The kernel can then discover ACPI from the system tables.  The
> presence of ACPI vs. FDT is therefore always itself discoverable,
> through the FDT.
> 
> Therefore, the VM implementation must provide through its UEFI
> implementation, either:
> 
>   a complete FDT which describes the entire VM system and will boot
>   mainline kernels driven by device tree alone, or
> 
>   no FDT.  In this case, the VM implementation must provide ACPI, and
>   the OS must be able to locate the ACPI root pointer through the UEFI
>   system table.
> 
> For more information about the arm and arm64 boot conventions, see
> Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> Linux kernel source tree.
> 
> For more information about UEFI and ACPI booting, see [4] and [5].
> 
> 
> VM Platform
> -----------
> The specification does not mandate any specific memory map.  The guest
> OS must be able to enumerate all processing elements, devices, and
> memory through HW description data (FDT, ACPI) or a bus-specific
> mechanism such as PCI.
> 
> The virtual platform must support at least one of the following ARM
> execution states:
>   (1) aarch32 virtual CPUs on aarch32 physical CPUs
>   (2) aarch32 virtual CPUs on aarch64 physical CPUs
>   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> 
> It is recommended to support both (2) and (3) on aarch64 capable
> physical systems.
> 
> The virtual hardware platform must provide a number of mandatory
> peripherals:
> 
>   Serial console:  The platform should provide a console,
>   based on an emulated pl011, a virtio-console, or a Xen PV console.
> 
>   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
>   limits the the number of virtual CPUs to 8 cores, newer GIC versions
>   removes this limitation.
> 
>   The ARM virtual timer and counter should be available to the VM as
>   per the ARM Generic Timers specification in the ARM ARM [1].
> 
>   A hotpluggable bus to support hotplug of at least block and network
>   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
>   bus.

Is VirtIO hotplug capable? Over PCI or MMIO transports or both?

> We make the following recommendations for the guest OS kernel:
> 
>   The guest OS must include support for GICv2 and any available newer
>   version of the GIC architecture to maintain compatibility with older
>   VM implementations.
> 
>   It is strongly recommended to include support for all available
>   (block, network, console, balloon) virtio-pci, virtio-mmio, and Xen PV
>   drivers in the guest OS kernel or initial ramdisk.

I would love to eventually see some defconfigs for this sort of thing.

> Other common peripherals for block devices, networking, and more can
> (and typically will) be provided, but OS software written and compiled
> to run on ARM VMs cannot make any assumptions about which variations
> of these should exist or which implementation they use (e.g. VirtIO or
> Xen PV).  See "Hardware Description" above.
> 
> Note that this platform specification is separate from the Linux kernel
> concept of mach-virt, which merely specifies a machine model driven
> purely from device tree, but does not mandate any peripherals or have any
> mention of ACPI.

Well, the commit message for it said it mandated a GIC and architected timers.

Regards,
Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Wed, Feb 26, 2014 at 02:27:40PM -0500, Christopher Covington wrote:
> Hi Christoffer,
> 
> On 02/26/2014 01:34 PM, Christoffer Dall wrote:
> > ARM VM System Specification
> > ===========================
> > 
> > Goal
> > ----
> > The goal of this spec is to allow suitably-built OS images to run on
> > all ARM virtualization solutions, such as KVM or Xen.
> 
> Would you consider including simulators/emulators as well, such as QEMU in TCG
> mode?
> 

Yes, but I think KVM or Xen is the most common use cases for this, but
in fact, for KVM, most of the work to support this would be in QEMU
anyhow and whether you choose to enable KVM or not shouldn't make any
difference.

> > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > they aim to be hypervisor agnostic.
> > 
> > Note that simply adhering to the SBSA [2] is not a valid approach,
> > for example because the SBSA mandates EL2, which will not be available
> > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > may be controversial for some ARM VM implementations to support.
> > This spec also covers the aarch32 execution mode, not covered in the
> > SBSA.
> > 
> > 
> > Image format
> > ------------
> > The image format, as presented to the VM, needs to be well-defined in
> > order for prepared disk images to be bootable across various
> > virtualization implementations.
> > 
> > The raw disk format as presented to the VM must be partitioned with a
> > GUID Partition Table (GPT).  The bootable software must be placed in the
> > EFI System Partition (ESP), using the UEFI removable media path, and
> > must be an EFI application complying to the UEFI Specification 2.4
> > Revision A [6].
> > 
> > The ESP partition's GPT entry's partition type GUID must be
> > C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> > formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> > 
> > The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> > execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> > state.
> > 
> > This ensures that tools for both Xen and KVM can load a binary UEFI
> > firmware which can read and boot the EFI application in the disk image.
> > 
> > A typical scenario will be GRUB2 packaged as an EFI application, which
> > mounts the system boot partition and boots Linux.
> > 
> > 
> > Virtual Firmware
> > ----------------
> > The VM system must be able to boot the EFI application in the ESP.  It
> > is recommended that this is achieved by loading a UEFI binary as the
> > first software executed by the VM, which then executes the EFI
> > application.  The UEFI implementation should be compliant with UEFI
> > Specification 2.4 Revision A [6] or later.
> > 
> > This document strongly recommends that the VM implementation supports
> > persistent environment storage for virtual firmware implementation in
> > order to ensure probable use cases such as adding additional disk images
> > to a VM or running installers to perform upgrades.
> > 
> > The binary UEFI firmware implementation should not be distributed as
> > part of the VM image, but is specific to the VM implementation.
> 
> Can you elaborate on the motivation for requiring that the kernel be stuffed
> into a disk image and for requiring such a heavyweight bootloader/firmware? By
> doing so you would seem to exclude those requiring an optimized boot process.
> 

What's the alternative?  Shipping kernels externally and loading them
externally?  Sure you can do that, but then distros can't upgrade the
kernel themselves, and you have to come up with a convention for how to
ship kernels, initrd's etc.

This works well on x86 today and is going to reflect how most people see
ARM server hardware to behave as well.

> > Hardware Description
> > --------------------
> > The Linux kernel's proper entry point always takes a pointer to an FDT,
> > regardless of the boot mechanism, firmware, and hardware description
> > method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> > entry point will still receive a pointer to a simple FDT, generated by
> > the Linux kernel UEFI stub, containing a pointer to the UEFI system
> > table.  The kernel can then discover ACPI from the system tables.  The
> > presence of ACPI vs. FDT is therefore always itself discoverable,
> > through the FDT.
> > 
> > Therefore, the VM implementation must provide through its UEFI
> > implementation, either:
> > 
> >   a complete FDT which describes the entire VM system and will boot
> >   mainline kernels driven by device tree alone, or
> > 
> >   no FDT.  In this case, the VM implementation must provide ACPI, and
> >   the OS must be able to locate the ACPI root pointer through the UEFI
> >   system table.
> > 
> > For more information about the arm and arm64 boot conventions, see
> > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > Linux kernel source tree.
> > 
> > For more information about UEFI and ACPI booting, see [4] and [5].
> > 
> > 
> > VM Platform
> > -----------
> > The specification does not mandate any specific memory map.  The guest
> > OS must be able to enumerate all processing elements, devices, and
> > memory through HW description data (FDT, ACPI) or a bus-specific
> > mechanism such as PCI.
> > 
> > The virtual platform must support at least one of the following ARM
> > execution states:
> >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> > 
> > It is recommended to support both (2) and (3) on aarch64 capable
> > physical systems.
> > 
> > The virtual hardware platform must provide a number of mandatory
> > peripherals:
> > 
> >   Serial console:  The platform should provide a console,
> >   based on an emulated pl011, a virtio-console, or a Xen PV console.
> > 
> >   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
> >   limits the the number of virtual CPUs to 8 cores, newer GIC versions
> >   removes this limitation.
> > 
> >   The ARM virtual timer and counter should be available to the VM as
> >   per the ARM Generic Timers specification in the ARM ARM [1].
> > 
> >   A hotpluggable bus to support hotplug of at least block and network
> >   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
> >   bus.
> 
> Is VirtIO hotplug capable? Over PCI or MMIO transports or both?
> 

VirtIO devices attached on a PCIe bus are hotpluggable, the emulated
PCIe bus itself would not have anything to do with virtio, except that
virtio devices can hang off of it.  AFAIU.

> > We make the following recommendations for the guest OS kernel:
> > 
> >   The guest OS must include support for GICv2 and any available newer
> >   version of the GIC architecture to maintain compatibility with older
> >   VM implementations.
> > 
> >   It is strongly recommended to include support for all available
> >   (block, network, console, balloon) virtio-pci, virtio-mmio, and Xen PV
> >   drivers in the guest OS kernel or initial ramdisk.
> 
> I would love to eventually see some defconfigs for this sort of thing.
> 

Agreed, I think it's beyond the scope of this spec though.

> > Other common peripherals for block devices, networking, and more can
> > (and typically will) be provided, but OS software written and compiled
> > to run on ARM VMs cannot make any assumptions about which variations
> > of these should exist or which implementation they use (e.g. VirtIO or
> > Xen PV).  See "Hardware Description" above.
> > 
> > Note that this platform specification is separate from the Linux kernel
> > concept of mach-virt, which merely specifies a machine model driven
> > purely from device tree, but does not mandate any peripherals or have any
> > mention of ACPI.
> 
> Well, the commit message for it said it mandated a GIC and architected timers.
> 
Haven't we been down that road before?  I think everyone pretty much
agrees this is the definition of mach-virt today, but if this note
causes people to start splitting hairs, I can remove the paragraph.

Thanks,
-Christoffer

[RFC] ARM VM System Sepcification

[Christoffer Dall]

On Wed, Feb 26, 2014 at 02:27:40PM -0500, Christopher Covington wrote:
> Hi Christoffer,
> 
> On 02/26/2014 01:34 PM, Christoffer Dall wrote:
> > ARM VM System Specification
> > ===========================
> > 
> > Goal
> > ----
> > The goal of this spec is to allow suitably-built OS images to run on
> > all ARM virtualization solutions, such as KVM or Xen.
> 
> Would you consider including simulators/emulators as well, such as QEMU in TCG
> mode?
> 

Yes, but I think KVM or Xen is the most common use cases for this, but
in fact, for KVM, most of the work to support this would be in QEMU
anyhow and whether you choose to enable KVM or not shouldn't make any
difference.

> > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > they aim to be hypervisor agnostic.
> > 
> > Note that simply adhering to the SBSA [2] is not a valid approach,
> > for example because the SBSA mandates EL2, which will not be available
> > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > may be controversial for some ARM VM implementations to support.
> > This spec also covers the aarch32 execution mode, not covered in the
> > SBSA.
> > 
> > 
> > Image format
> > ------------
> > The image format, as presented to the VM, needs to be well-defined in
> > order for prepared disk images to be bootable across various
> > virtualization implementations.
> > 
> > The raw disk format as presented to the VM must be partitioned with a
> > GUID Partition Table (GPT).  The bootable software must be placed in the
> > EFI System Partition (ESP), using the UEFI removable media path, and
> > must be an EFI application complying to the UEFI Specification 2.4
> > Revision A [6].
> > 
> > The ESP partition's GPT entry's partition type GUID must be
> > C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> > formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> > 
> > The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> > execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> > state.
> > 
> > This ensures that tools for both Xen and KVM can load a binary UEFI
> > firmware which can read and boot the EFI application in the disk image.
> > 
> > A typical scenario will be GRUB2 packaged as an EFI application, which
> > mounts the system boot partition and boots Linux.
> > 
> > 
> > Virtual Firmware
> > ----------------
> > The VM system must be able to boot the EFI application in the ESP.  It
> > is recommended that this is achieved by loading a UEFI binary as the
> > first software executed by the VM, which then executes the EFI
> > application.  The UEFI implementation should be compliant with UEFI
> > Specification 2.4 Revision A [6] or later.
> > 
> > This document strongly recommends that the VM implementation supports
> > persistent environment storage for virtual firmware implementation in
> > order to ensure probable use cases such as adding additional disk images
> > to a VM or running installers to perform upgrades.
> > 
> > The binary UEFI firmware implementation should not be distributed as
> > part of the VM image, but is specific to the VM implementation.
> 
> Can you elaborate on the motivation for requiring that the kernel be stuffed
> into a disk image and for requiring such a heavyweight bootloader/firmware? By
> doing so you would seem to exclude those requiring an optimized boot process.
> 

What's the alternative?  Shipping kernels externally and loading them
externally?  Sure you can do that, but then distros can't upgrade the
kernel themselves, and you have to come up with a convention for how to
ship kernels, initrd's etc.

This works well on x86 today and is going to reflect how most people see
ARM server hardware to behave as well.

> > Hardware Description
> > --------------------
> > The Linux kernel's proper entry point always takes a pointer to an FDT,
> > regardless of the boot mechanism, firmware, and hardware description
> > method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> > entry point will still receive a pointer to a simple FDT, generated by
> > the Linux kernel UEFI stub, containing a pointer to the UEFI system
> > table.  The kernel can then discover ACPI from the system tables.  The
> > presence of ACPI vs. FDT is therefore always itself discoverable,
> > through the FDT.
> > 
> > Therefore, the VM implementation must provide through its UEFI
> > implementation, either:
> > 
> >   a complete FDT which describes the entire VM system and will boot
> >   mainline kernels driven by device tree alone, or
> > 
> >   no FDT.  In this case, the VM implementation must provide ACPI, and
> >   the OS must be able to locate the ACPI root pointer through the UEFI
> >   system table.
> > 
> > For more information about the arm and arm64 boot conventions, see
> > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > Linux kernel source tree.
> > 
> > For more information about UEFI and ACPI booting, see [4] and [5].
> > 
> > 
> > VM Platform
> > -----------
> > The specification does not mandate any specific memory map.  The guest
> > OS must be able to enumerate all processing elements, devices, and
> > memory through HW description data (FDT, ACPI) or a bus-specific
> > mechanism such as PCI.
> > 
> > The virtual platform must support at least one of the following ARM
> > execution states:
> >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> > 
> > It is recommended to support both (2) and (3) on aarch64 capable
> > physical systems.
> > 
> > The virtual hardware platform must provide a number of mandatory
> > peripherals:
> > 
> >   Serial console:  The platform should provide a console,
> >   based on an emulated pl011, a virtio-console, or a Xen PV console.
> > 
> >   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
> >   limits the the number of virtual CPUs to 8 cores, newer GIC versions
> >   removes this limitation.
> > 
> >   The ARM virtual timer and counter should be available to the VM as
> >   per the ARM Generic Timers specification in the ARM ARM [1].
> > 
> >   A hotpluggable bus to support hotplug of at least block and network
> >   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
> >   bus.
> 
> Is VirtIO hotplug capable? Over PCI or MMIO transports or both?
> 

VirtIO devices attached on a PCIe bus are hotpluggable, the emulated
PCIe bus itself would not have anything to do with virtio, except that
virtio devices can hang off of it.  AFAIU.

> > We make the following recommendations for the guest OS kernel:
> > 
> >   The guest OS must include support for GICv2 and any available newer
> >   version of the GIC architecture to maintain compatibility with older
> >   VM implementations.
> > 
> >   It is strongly recommended to include support for all available
> >   (block, network, console, balloon) virtio-pci, virtio-mmio, and Xen PV
> >   drivers in the guest OS kernel or initial ramdisk.
> 
> I would love to eventually see some defconfigs for this sort of thing.
> 

Agreed, I think it's beyond the scope of this spec though.

> > Other common peripherals for block devices, networking, and more can
> > (and typically will) be provided, but OS software written and compiled
> > to run on ARM VMs cannot make any assumptions about which variations
> > of these should exist or which implementation they use (e.g. VirtIO or
> > Xen PV).  See "Hardware Description" above.
> > 
> > Note that this platform specification is separate from the Linux kernel
> > concept of mach-virt, which merely specifies a machine model driven
> > purely from device tree, but does not mandate any peripherals or have any
> > mention of ACPI.
> 
> Well, the commit message for it said it mandated a GIC and architected timers.
> 
Haven't we been down that road before?  I think everyone pretty much
agrees this is the definition of mach-virt today, but if this note
causes people to start splitting hairs, I can remove the paragraph.

Thanks,
-Christoffer

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Wed, Feb 26, 2014 at 02:27:40PM -0500, Christopher Covington wrote:
> Hi Christoffer,
> 
> On 02/26/2014 01:34 PM, Christoffer Dall wrote:
> > ARM VM System Specification
> > ===========================
> > 
> > Goal
> > ----
> > The goal of this spec is to allow suitably-built OS images to run on
> > all ARM virtualization solutions, such as KVM or Xen.
> 
> Would you consider including simulators/emulators as well, such as QEMU in TCG
> mode?
> 

Yes, but I think KVM or Xen is the most common use cases for this, but
in fact, for KVM, most of the work to support this would be in QEMU
anyhow and whether you choose to enable KVM or not shouldn't make any
difference.

> > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > they aim to be hypervisor agnostic.
> > 
> > Note that simply adhering to the SBSA [2] is not a valid approach,
> > for example because the SBSA mandates EL2, which will not be available
> > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > may be controversial for some ARM VM implementations to support.
> > This spec also covers the aarch32 execution mode, not covered in the
> > SBSA.
> > 
> > 
> > Image format
> > ------------
> > The image format, as presented to the VM, needs to be well-defined in
> > order for prepared disk images to be bootable across various
> > virtualization implementations.
> > 
> > The raw disk format as presented to the VM must be partitioned with a
> > GUID Partition Table (GPT).  The bootable software must be placed in the
> > EFI System Partition (ESP), using the UEFI removable media path, and
> > must be an EFI application complying to the UEFI Specification 2.4
> > Revision A [6].
> > 
> > The ESP partition's GPT entry's partition type GUID must be
> > C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> > formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> > 
> > The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> > execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> > state.
> > 
> > This ensures that tools for both Xen and KVM can load a binary UEFI
> > firmware which can read and boot the EFI application in the disk image.
> > 
> > A typical scenario will be GRUB2 packaged as an EFI application, which
> > mounts the system boot partition and boots Linux.
> > 
> > 
> > Virtual Firmware
> > ----------------
> > The VM system must be able to boot the EFI application in the ESP.  It
> > is recommended that this is achieved by loading a UEFI binary as the
> > first software executed by the VM, which then executes the EFI
> > application.  The UEFI implementation should be compliant with UEFI
> > Specification 2.4 Revision A [6] or later.
> > 
> > This document strongly recommends that the VM implementation supports
> > persistent environment storage for virtual firmware implementation in
> > order to ensure probable use cases such as adding additional disk images
> > to a VM or running installers to perform upgrades.
> > 
> > The binary UEFI firmware implementation should not be distributed as
> > part of the VM image, but is specific to the VM implementation.
> 
> Can you elaborate on the motivation for requiring that the kernel be stuffed
> into a disk image and for requiring such a heavyweight bootloader/firmware? By
> doing so you would seem to exclude those requiring an optimized boot process.
> 

What's the alternative?  Shipping kernels externally and loading them
externally?  Sure you can do that, but then distros can't upgrade the
kernel themselves, and you have to come up with a convention for how to
ship kernels, initrd's etc.

This works well on x86 today and is going to reflect how most people see
ARM server hardware to behave as well.

> > Hardware Description
> > --------------------
> > The Linux kernel's proper entry point always takes a pointer to an FDT,
> > regardless of the boot mechanism, firmware, and hardware description
> > method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> > entry point will still receive a pointer to a simple FDT, generated by
> > the Linux kernel UEFI stub, containing a pointer to the UEFI system
> > table.  The kernel can then discover ACPI from the system tables.  The
> > presence of ACPI vs. FDT is therefore always itself discoverable,
> > through the FDT.
> > 
> > Therefore, the VM implementation must provide through its UEFI
> > implementation, either:
> > 
> >   a complete FDT which describes the entire VM system and will boot
> >   mainline kernels driven by device tree alone, or
> > 
> >   no FDT.  In this case, the VM implementation must provide ACPI, and
> >   the OS must be able to locate the ACPI root pointer through the UEFI
> >   system table.
> > 
> > For more information about the arm and arm64 boot conventions, see
> > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > Linux kernel source tree.
> > 
> > For more information about UEFI and ACPI booting, see [4] and [5].
> > 
> > 
> > VM Platform
> > -----------
> > The specification does not mandate any specific memory map.  The guest
> > OS must be able to enumerate all processing elements, devices, and
> > memory through HW description data (FDT, ACPI) or a bus-specific
> > mechanism such as PCI.
> > 
> > The virtual platform must support at least one of the following ARM
> > execution states:
> >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> > 
> > It is recommended to support both (2) and (3) on aarch64 capable
> > physical systems.
> > 
> > The virtual hardware platform must provide a number of mandatory
> > peripherals:
> > 
> >   Serial console:  The platform should provide a console,
> >   based on an emulated pl011, a virtio-console, or a Xen PV console.
> > 
> >   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
> >   limits the the number of virtual CPUs to 8 cores, newer GIC versions
> >   removes this limitation.
> > 
> >   The ARM virtual timer and counter should be available to the VM as
> >   per the ARM Generic Timers specification in the ARM ARM [1].
> > 
> >   A hotpluggable bus to support hotplug of at least block and network
> >   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
> >   bus.
> 
> Is VirtIO hotplug capable? Over PCI or MMIO transports or both?
> 

VirtIO devices attached on a PCIe bus are hotpluggable, the emulated
PCIe bus itself would not have anything to do with virtio, except that
virtio devices can hang off of it.  AFAIU.

> > We make the following recommendations for the guest OS kernel:
> > 
> >   The guest OS must include support for GICv2 and any available newer
> >   version of the GIC architecture to maintain compatibility with older
> >   VM implementations.
> > 
> >   It is strongly recommended to include support for all available
> >   (block, network, console, balloon) virtio-pci, virtio-mmio, and Xen PV
> >   drivers in the guest OS kernel or initial ramdisk.
> 
> I would love to eventually see some defconfigs for this sort of thing.
> 

Agreed, I think it's beyond the scope of this spec though.

> > Other common peripherals for block devices, networking, and more can
> > (and typically will) be provided, but OS software written and compiled
> > to run on ARM VMs cannot make any assumptions about which variations
> > of these should exist or which implementation they use (e.g. VirtIO or
> > Xen PV).  See "Hardware Description" above.
> > 
> > Note that this platform specification is separate from the Linux kernel
> > concept of mach-virt, which merely specifies a machine model driven
> > purely from device tree, but does not mandate any peripherals or have any
> > mention of ACPI.
> 
> Well, the commit message for it said it mandated a GIC and architected timers.
> 
Haven't we been down that road before?  I think everyone pretty much
agrees this is the definition of mach-virt today, but if this note
causes people to start splitting hairs, I can remove the paragraph.

Thanks,
-Christoffer

Re: [RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> ARM VM System Specification
> ===========================
> 
> Goal
> ----
> The goal of this spec is to allow suitably-built OS images to run on
> all ARM virtualization solutions, such as KVM or Xen.
> 
> Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> they aim to be hypervisor agnostic.
> 
> Note that simply adhering to the SBSA [2] is not a valid approach,
> for example because the SBSA mandates EL2, which will not be available
> for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> may be controversial for some ARM VM implementations to support.
> This spec also covers the aarch32 execution mode, not covered in the
> SBSA.

I would prefer if we can stay as close as possible to SBSA for individual
hardware components, and only stray from it when there is a strong reason.
pl011-subset doesn't sound like a significant problem to implement,
especially as SBSA makes the DMA part of that optional. Can you
elaborate on what hypervisor would have a problem with that?

> Hardware Description
> --------------------
> The Linux kernel's proper entry point always takes a pointer to an FDT,
> regardless of the boot mechanism, firmware, and hardware description
> method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> entry point will still receive a pointer to a simple FDT, generated by
> the Linux kernel UEFI stub, containing a pointer to the UEFI system
> table.  The kernel can then discover ACPI from the system tables.  The
> presence of ACPI vs. FDT is therefore always itself discoverable,
> through the FDT.
> 
> Therefore, the VM implementation must provide through its UEFI
> implementation, either:
> 
>   a complete FDT which describes the entire VM system and will boot
>   mainline kernels driven by device tree alone, or
> 
>   no FDT.  In this case, the VM implementation must provide ACPI, and
>   the OS must be able to locate the ACPI root pointer through the UEFI
>   system table.
> 
> For more information about the arm and arm64 boot conventions, see
> Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> Linux kernel source tree.
> 
> For more information about UEFI and ACPI booting, see [4] and [5].

What's the point of having ACPI in a virtual machine? You wouldn't
need to abstract any of the hardware in AML since you already know
what the virtual hardware is, so I can't see how this would help
anyone.

However, as ACPI will not be supported by arm32, not having the
complete FDT will prevent you from running a 32-bit guest on
a 64-bit hypervisor, which I consider an important use case.

> VM Platform
> -----------
> The specification does not mandate any specific memory map.  The guest
> OS must be able to enumerate all processing elements, devices, and
> memory through HW description data (FDT, ACPI) or a bus-specific
> mechanism such as PCI.
> 
> The virtual platform must support at least one of the following ARM
> execution states:
>   (1) aarch32 virtual CPUs on aarch32 physical CPUs
>   (2) aarch32 virtual CPUs on aarch64 physical CPUs
>   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> 
> It is recommended to support both (2) and (3) on aarch64 capable
> physical systems.

Isn't this more of a CPU capabilities question? Or maybe you
should just add 'if aarch32 mode supported is supported by
the host CPU'.

> 
> The virtual hardware platform must provide a number of mandatory
> peripherals:
> 
>   Serial console:  The platform should provide a console,
>   based on an emulated pl011, a virtio-console, or a Xen PV console.
> 
>   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
>   limits the the number of virtual CPUs to 8 cores, newer GIC versions
>   removes this limitation.
> 
>   The ARM virtual timer and counter should be available to the VM as
>   per the ARM Generic Timers specification in the ARM ARM [1].
> 
>   A hotpluggable bus to support hotplug of at least block and network
>   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
>   bus.

I think you should specify exactly what you want PCIe to look like,
if present. Otherwise you can get wildly incompatible bus discovery.

> Note that this platform specification is separate from the Linux kernel
> concept of mach-virt, which merely specifies a machine model driven
> purely from device tree, but does not mandate any peripherals or have any
> mention of ACPI.

Did you notice we are removing mach-virt now? Probably no point
mentioning it here.

	Arnd

[RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> ARM VM System Specification
> ===========================
> 
> Goal
> ----
> The goal of this spec is to allow suitably-built OS images to run on
> all ARM virtualization solutions, such as KVM or Xen.
> 
> Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> they aim to be hypervisor agnostic.
> 
> Note that simply adhering to the SBSA [2] is not a valid approach,
> for example because the SBSA mandates EL2, which will not be available
> for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> may be controversial for some ARM VM implementations to support.
> This spec also covers the aarch32 execution mode, not covered in the
> SBSA.

I would prefer if we can stay as close as possible to SBSA for individual
hardware components, and only stray from it when there is a strong reason.
pl011-subset doesn't sound like a significant problem to implement,
especially as SBSA makes the DMA part of that optional. Can you
elaborate on what hypervisor would have a problem with that?

> Hardware Description
> --------------------
> The Linux kernel's proper entry point always takes a pointer to an FDT,
> regardless of the boot mechanism, firmware, and hardware description
> method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> entry point will still receive a pointer to a simple FDT, generated by
> the Linux kernel UEFI stub, containing a pointer to the UEFI system
> table.  The kernel can then discover ACPI from the system tables.  The
> presence of ACPI vs. FDT is therefore always itself discoverable,
> through the FDT.
> 
> Therefore, the VM implementation must provide through its UEFI
> implementation, either:
> 
>   a complete FDT which describes the entire VM system and will boot
>   mainline kernels driven by device tree alone, or
> 
>   no FDT.  In this case, the VM implementation must provide ACPI, and
>   the OS must be able to locate the ACPI root pointer through the UEFI
>   system table.
> 
> For more information about the arm and arm64 boot conventions, see
> Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> Linux kernel source tree.
> 
> For more information about UEFI and ACPI booting, see [4] and [5].

What's the point of having ACPI in a virtual machine? You wouldn't
need to abstract any of the hardware in AML since you already know
what the virtual hardware is, so I can't see how this would help
anyone.

However, as ACPI will not be supported by arm32, not having the
complete FDT will prevent you from running a 32-bit guest on
a 64-bit hypervisor, which I consider an important use case.

> VM Platform
> -----------
> The specification does not mandate any specific memory map.  The guest
> OS must be able to enumerate all processing elements, devices, and
> memory through HW description data (FDT, ACPI) or a bus-specific
> mechanism such as PCI.
> 
> The virtual platform must support at least one of the following ARM
> execution states:
>   (1) aarch32 virtual CPUs on aarch32 physical CPUs
>   (2) aarch32 virtual CPUs on aarch64 physical CPUs
>   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> 
> It is recommended to support both (2) and (3) on aarch64 capable
> physical systems.

Isn't this more of a CPU capabilities question? Or maybe you
should just add 'if aarch32 mode supported is supported by
the host CPU'.

> 
> The virtual hardware platform must provide a number of mandatory
> peripherals:
> 
>   Serial console:  The platform should provide a console,
>   based on an emulated pl011, a virtio-console, or a Xen PV console.
> 
>   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
>   limits the the number of virtual CPUs to 8 cores, newer GIC versions
>   removes this limitation.
> 
>   The ARM virtual timer and counter should be available to the VM as
>   per the ARM Generic Timers specification in the ARM ARM [1].
> 
>   A hotpluggable bus to support hotplug of at least block and network
>   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
>   bus.

I think you should specify exactly what you want PCIe to look like,
if present. Otherwise you can get wildly incompatible bus discovery.

> Note that this platform specification is separate from the Linux kernel
> concept of mach-virt, which merely specifies a machine model driven
> purely from device tree, but does not mandate any peripherals or have any
> mention of ACPI.

Did you notice we are removing mach-virt now? Probably no point
mentioning it here.

	Arnd

Re: [RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> ARM VM System Specification
> ===========================
> 
> Goal
> ----
> The goal of this spec is to allow suitably-built OS images to run on
> all ARM virtualization solutions, such as KVM or Xen.
> 
> Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> they aim to be hypervisor agnostic.
> 
> Note that simply adhering to the SBSA [2] is not a valid approach,
> for example because the SBSA mandates EL2, which will not be available
> for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> may be controversial for some ARM VM implementations to support.
> This spec also covers the aarch32 execution mode, not covered in the
> SBSA.

I would prefer if we can stay as close as possible to SBSA for individual
hardware components, and only stray from it when there is a strong reason.
pl011-subset doesn't sound like a significant problem to implement,
especially as SBSA makes the DMA part of that optional. Can you
elaborate on what hypervisor would have a problem with that?

> Hardware Description
> --------------------
> The Linux kernel's proper entry point always takes a pointer to an FDT,
> regardless of the boot mechanism, firmware, and hardware description
> method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> entry point will still receive a pointer to a simple FDT, generated by
> the Linux kernel UEFI stub, containing a pointer to the UEFI system
> table.  The kernel can then discover ACPI from the system tables.  The
> presence of ACPI vs. FDT is therefore always itself discoverable,
> through the FDT.
> 
> Therefore, the VM implementation must provide through its UEFI
> implementation, either:
> 
>   a complete FDT which describes the entire VM system and will boot
>   mainline kernels driven by device tree alone, or
> 
>   no FDT.  In this case, the VM implementation must provide ACPI, and
>   the OS must be able to locate the ACPI root pointer through the UEFI
>   system table.
> 
> For more information about the arm and arm64 boot conventions, see
> Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> Linux kernel source tree.
> 
> For more information about UEFI and ACPI booting, see [4] and [5].

What's the point of having ACPI in a virtual machine? You wouldn't
need to abstract any of the hardware in AML since you already know
what the virtual hardware is, so I can't see how this would help
anyone.

However, as ACPI will not be supported by arm32, not having the
complete FDT will prevent you from running a 32-bit guest on
a 64-bit hypervisor, which I consider an important use case.

> VM Platform
> -----------
> The specification does not mandate any specific memory map.  The guest
> OS must be able to enumerate all processing elements, devices, and
> memory through HW description data (FDT, ACPI) or a bus-specific
> mechanism such as PCI.
> 
> The virtual platform must support at least one of the following ARM
> execution states:
>   (1) aarch32 virtual CPUs on aarch32 physical CPUs
>   (2) aarch32 virtual CPUs on aarch64 physical CPUs
>   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> 
> It is recommended to support both (2) and (3) on aarch64 capable
> physical systems.

Isn't this more of a CPU capabilities question? Or maybe you
should just add 'if aarch32 mode supported is supported by
the host CPU'.

> 
> The virtual hardware platform must provide a number of mandatory
> peripherals:
> 
>   Serial console:  The platform should provide a console,
>   based on an emulated pl011, a virtio-console, or a Xen PV console.
> 
>   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
>   limits the the number of virtual CPUs to 8 cores, newer GIC versions
>   removes this limitation.
> 
>   The ARM virtual timer and counter should be available to the VM as
>   per the ARM Generic Timers specification in the ARM ARM [1].
> 
>   A hotpluggable bus to support hotplug of at least block and network
>   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
>   bus.

I think you should specify exactly what you want PCIe to look like,
if present. Otherwise you can get wildly incompatible bus discovery.

> Note that this platform specification is separate from the Linux kernel
> concept of mach-virt, which merely specifies a machine model driven
> purely from device tree, but does not mandate any peripherals or have any
> mention of ACPI.

Did you notice we are removing mach-virt now? Probably no point
mentioning it here.

	Arnd

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> > ARM VM System Specification
> > ===========================
> > 
> > Goal
> > ----
> > The goal of this spec is to allow suitably-built OS images to run on
> > all ARM virtualization solutions, such as KVM or Xen.
> > 
> > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > they aim to be hypervisor agnostic.
> > 
> > Note that simply adhering to the SBSA [2] is not a valid approach,
> > for example because the SBSA mandates EL2, which will not be available
> > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > may be controversial for some ARM VM implementations to support.
> > This spec also covers the aarch32 execution mode, not covered in the
> > SBSA.
> 
> I would prefer if we can stay as close as possible to SBSA for individual
> hardware components, and only stray from it when there is a strong reason.
> pl011-subset doesn't sound like a significant problem to implement,
> especially as SBSA makes the DMA part of that optional. Can you
> elaborate on what hypervisor would have a problem with that?
> 

The Xen guys are hard-set on not supporting a pl011.  If we can convince
them or force it upon them, I'm ok with that.

I agree we should stay close to the SBSA, but I think there are
considerations for VMs beyond the SBSA that warrants this spec.

> > Hardware Description
> > --------------------
> > The Linux kernel's proper entry point always takes a pointer to an FDT,
> > regardless of the boot mechanism, firmware, and hardware description
> > method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> > entry point will still receive a pointer to a simple FDT, generated by
> > the Linux kernel UEFI stub, containing a pointer to the UEFI system
> > table.  The kernel can then discover ACPI from the system tables.  The
> > presence of ACPI vs. FDT is therefore always itself discoverable,
> > through the FDT.
> > 
> > Therefore, the VM implementation must provide through its UEFI
> > implementation, either:
> > 
> >   a complete FDT which describes the entire VM system and will boot
> >   mainline kernels driven by device tree alone, or
> > 
> >   no FDT.  In this case, the VM implementation must provide ACPI, and
> >   the OS must be able to locate the ACPI root pointer through the UEFI
> >   system table.
> > 
> > For more information about the arm and arm64 boot conventions, see
> > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > Linux kernel source tree.
> > 
> > For more information about UEFI and ACPI booting, see [4] and [5].
> 
> What's the point of having ACPI in a virtual machine? You wouldn't
> need to abstract any of the hardware in AML since you already know
> what the virtual hardware is, so I can't see how this would help
> anyone.

The most common response I've been getting so far is that people
generally want their VMs to look close to the real thing, but not sure
how valid an argument that is.

Some people feel strongly about this and seem to think that ARMv8
kernels will only work with ACPI in the future...

Another case is that it's a good development platform.  I know nothing
of developing and testing ACPI, so I won't judge one way or the other.

> 
> However, as ACPI will not be supported by arm32, not having the
> complete FDT will prevent you from running a 32-bit guest on
> a 64-bit hypervisor, which I consider an important use case.
> 

Agreed, I didn't appreciate that fact.  Hmmm, we need to consider that
case.

> > VM Platform
> > -----------
> > The specification does not mandate any specific memory map.  The guest
> > OS must be able to enumerate all processing elements, devices, and
> > memory through HW description data (FDT, ACPI) or a bus-specific
> > mechanism such as PCI.
> > 
> > The virtual platform must support at least one of the following ARM
> > execution states:
> >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> > 
> > It is recommended to support both (2) and (3) on aarch64 capable
> > physical systems.
> 
> Isn't this more of a CPU capabilities question? Or maybe you
> should just add 'if aarch32 mode supported is supported by
> the host CPU'.
> 

The recommendation is to tell people to actually have a -aarch32 (or
whatever it would be called) that works in their VM implementation.
This can certainly be reworded.

> > 
> > The virtual hardware platform must provide a number of mandatory
> > peripherals:
> > 
> >   Serial console:  The platform should provide a console,
> >   based on an emulated pl011, a virtio-console, or a Xen PV console.
> > 
> >   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
> >   limits the the number of virtual CPUs to 8 cores, newer GIC versions
> >   removes this limitation.
> > 
> >   The ARM virtual timer and counter should be available to the VM as
> >   per the ARM Generic Timers specification in the ARM ARM [1].
> > 
> >   A hotpluggable bus to support hotplug of at least block and network
> >   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
> >   bus.
> 
> I think you should specify exactly what you want PCIe to look like,
> if present. Otherwise you can get wildly incompatible bus discovery.
> 

As soon as there is more clarity on what it will actually look like,
I'll be happy to add this.  I'm afraid my PCIe understanding is too
piecemeal to fully grasp this, so concrete suggestions for the text
would be much appreciated.

> > Note that this platform specification is separate from the Linux kernel
> > concept of mach-virt, which merely specifies a machine model driven
> > purely from device tree, but does not mandate any peripherals or have any
> > mention of ACPI.
> 
> Did you notice we are removing mach-virt now? Probably no point
> mentioning it here.
> 
Yes, I'm aware.  I've just heard people say "why do we need this, isn't
mach-virt all we need", and therefore I added the note.

I can definitely can rid of this paragraph in the future if it causes
more harm than good.

Thanks!
-Christoffer

[RFC] ARM VM System Sepcification

[Christoffer Dall]

On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> > ARM VM System Specification
> > ===========================
> > 
> > Goal
> > ----
> > The goal of this spec is to allow suitably-built OS images to run on
> > all ARM virtualization solutions, such as KVM or Xen.
> > 
> > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > they aim to be hypervisor agnostic.
> > 
> > Note that simply adhering to the SBSA [2] is not a valid approach,
> > for example because the SBSA mandates EL2, which will not be available
> > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > may be controversial for some ARM VM implementations to support.
> > This spec also covers the aarch32 execution mode, not covered in the
> > SBSA.
> 
> I would prefer if we can stay as close as possible to SBSA for individual
> hardware components, and only stray from it when there is a strong reason.
> pl011-subset doesn't sound like a significant problem to implement,
> especially as SBSA makes the DMA part of that optional. Can you
> elaborate on what hypervisor would have a problem with that?
> 

The Xen guys are hard-set on not supporting a pl011.  If we can convince
them or force it upon them, I'm ok with that.

I agree we should stay close to the SBSA, but I think there are
considerations for VMs beyond the SBSA that warrants this spec.

> > Hardware Description
> > --------------------
> > The Linux kernel's proper entry point always takes a pointer to an FDT,
> > regardless of the boot mechanism, firmware, and hardware description
> > method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> > entry point will still receive a pointer to a simple FDT, generated by
> > the Linux kernel UEFI stub, containing a pointer to the UEFI system
> > table.  The kernel can then discover ACPI from the system tables.  The
> > presence of ACPI vs. FDT is therefore always itself discoverable,
> > through the FDT.
> > 
> > Therefore, the VM implementation must provide through its UEFI
> > implementation, either:
> > 
> >   a complete FDT which describes the entire VM system and will boot
> >   mainline kernels driven by device tree alone, or
> > 
> >   no FDT.  In this case, the VM implementation must provide ACPI, and
> >   the OS must be able to locate the ACPI root pointer through the UEFI
> >   system table.
> > 
> > For more information about the arm and arm64 boot conventions, see
> > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > Linux kernel source tree.
> > 
> > For more information about UEFI and ACPI booting, see [4] and [5].
> 
> What's the point of having ACPI in a virtual machine? You wouldn't
> need to abstract any of the hardware in AML since you already know
> what the virtual hardware is, so I can't see how this would help
> anyone.

The most common response I've been getting so far is that people
generally want their VMs to look close to the real thing, but not sure
how valid an argument that is.

Some people feel strongly about this and seem to think that ARMv8
kernels will only work with ACPI in the future...

Another case is that it's a good development platform.  I know nothing
of developing and testing ACPI, so I won't judge one way or the other.

> 
> However, as ACPI will not be supported by arm32, not having the
> complete FDT will prevent you from running a 32-bit guest on
> a 64-bit hypervisor, which I consider an important use case.
> 

Agreed, I didn't appreciate that fact.  Hmmm, we need to consider that
case.

> > VM Platform
> > -----------
> > The specification does not mandate any specific memory map.  The guest
> > OS must be able to enumerate all processing elements, devices, and
> > memory through HW description data (FDT, ACPI) or a bus-specific
> > mechanism such as PCI.
> > 
> > The virtual platform must support at least one of the following ARM
> > execution states:
> >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> > 
> > It is recommended to support both (2) and (3) on aarch64 capable
> > physical systems.
> 
> Isn't this more of a CPU capabilities question? Or maybe you
> should just add 'if aarch32 mode supported is supported by
> the host CPU'.
> 

The recommendation is to tell people to actually have a -aarch32 (or
whatever it would be called) that works in their VM implementation.
This can certainly be reworded.

> > 
> > The virtual hardware platform must provide a number of mandatory
> > peripherals:
> > 
> >   Serial console:  The platform should provide a console,
> >   based on an emulated pl011, a virtio-console, or a Xen PV console.
> > 
> >   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
> >   limits the the number of virtual CPUs to 8 cores, newer GIC versions
> >   removes this limitation.
> > 
> >   The ARM virtual timer and counter should be available to the VM as
> >   per the ARM Generic Timers specification in the ARM ARM [1].
> > 
> >   A hotpluggable bus to support hotplug of at least block and network
> >   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
> >   bus.
> 
> I think you should specify exactly what you want PCIe to look like,
> if present. Otherwise you can get wildly incompatible bus discovery.
> 

As soon as there is more clarity on what it will actually look like,
I'll be happy to add this.  I'm afraid my PCIe understanding is too
piecemeal to fully grasp this, so concrete suggestions for the text
would be much appreciated.

> > Note that this platform specification is separate from the Linux kernel
> > concept of mach-virt, which merely specifies a machine model driven
> > purely from device tree, but does not mandate any peripherals or have any
> > mention of ACPI.
> 
> Did you notice we are removing mach-virt now? Probably no point
> mentioning it here.
> 
Yes, I'm aware.  I've just heard people say "why do we need this, isn't
mach-virt all we need", and therefore I added the note.

I can definitely can rid of this paragraph in the future if it causes
more harm than good.

Thanks!
-Christoffer

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> > ARM VM System Specification
> > ===========================
> > 
> > Goal
> > ----
> > The goal of this spec is to allow suitably-built OS images to run on
> > all ARM virtualization solutions, such as KVM or Xen.
> > 
> > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > they aim to be hypervisor agnostic.
> > 
> > Note that simply adhering to the SBSA [2] is not a valid approach,
> > for example because the SBSA mandates EL2, which will not be available
> > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > may be controversial for some ARM VM implementations to support.
> > This spec also covers the aarch32 execution mode, not covered in the
> > SBSA.
> 
> I would prefer if we can stay as close as possible to SBSA for individual
> hardware components, and only stray from it when there is a strong reason.
> pl011-subset doesn't sound like a significant problem to implement,
> especially as SBSA makes the DMA part of that optional. Can you
> elaborate on what hypervisor would have a problem with that?
> 

The Xen guys are hard-set on not supporting a pl011.  If we can convince
them or force it upon them, I'm ok with that.

I agree we should stay close to the SBSA, but I think there are
considerations for VMs beyond the SBSA that warrants this spec.

> > Hardware Description
> > --------------------
> > The Linux kernel's proper entry point always takes a pointer to an FDT,
> > regardless of the boot mechanism, firmware, and hardware description
> > method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> > entry point will still receive a pointer to a simple FDT, generated by
> > the Linux kernel UEFI stub, containing a pointer to the UEFI system
> > table.  The kernel can then discover ACPI from the system tables.  The
> > presence of ACPI vs. FDT is therefore always itself discoverable,
> > through the FDT.
> > 
> > Therefore, the VM implementation must provide through its UEFI
> > implementation, either:
> > 
> >   a complete FDT which describes the entire VM system and will boot
> >   mainline kernels driven by device tree alone, or
> > 
> >   no FDT.  In this case, the VM implementation must provide ACPI, and
> >   the OS must be able to locate the ACPI root pointer through the UEFI
> >   system table.
> > 
> > For more information about the arm and arm64 boot conventions, see
> > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > Linux kernel source tree.
> > 
> > For more information about UEFI and ACPI booting, see [4] and [5].
> 
> What's the point of having ACPI in a virtual machine? You wouldn't
> need to abstract any of the hardware in AML since you already know
> what the virtual hardware is, so I can't see how this would help
> anyone.

The most common response I've been getting so far is that people
generally want their VMs to look close to the real thing, but not sure
how valid an argument that is.

Some people feel strongly about this and seem to think that ARMv8
kernels will only work with ACPI in the future...

Another case is that it's a good development platform.  I know nothing
of developing and testing ACPI, so I won't judge one way or the other.

> 
> However, as ACPI will not be supported by arm32, not having the
> complete FDT will prevent you from running a 32-bit guest on
> a 64-bit hypervisor, which I consider an important use case.
> 

Agreed, I didn't appreciate that fact.  Hmmm, we need to consider that
case.

> > VM Platform
> > -----------
> > The specification does not mandate any specific memory map.  The guest
> > OS must be able to enumerate all processing elements, devices, and
> > memory through HW description data (FDT, ACPI) or a bus-specific
> > mechanism such as PCI.
> > 
> > The virtual platform must support at least one of the following ARM
> > execution states:
> >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> > 
> > It is recommended to support both (2) and (3) on aarch64 capable
> > physical systems.
> 
> Isn't this more of a CPU capabilities question? Or maybe you
> should just add 'if aarch32 mode supported is supported by
> the host CPU'.
> 

The recommendation is to tell people to actually have a -aarch32 (or
whatever it would be called) that works in their VM implementation.
This can certainly be reworded.

> > 
> > The virtual hardware platform must provide a number of mandatory
> > peripherals:
> > 
> >   Serial console:  The platform should provide a console,
> >   based on an emulated pl011, a virtio-console, or a Xen PV console.
> > 
> >   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
> >   limits the the number of virtual CPUs to 8 cores, newer GIC versions
> >   removes this limitation.
> > 
> >   The ARM virtual timer and counter should be available to the VM as
> >   per the ARM Generic Timers specification in the ARM ARM [1].
> > 
> >   A hotpluggable bus to support hotplug of at least block and network
> >   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
> >   bus.
> 
> I think you should specify exactly what you want PCIe to look like,
> if present. Otherwise you can get wildly incompatible bus discovery.
> 

As soon as there is more clarity on what it will actually look like,
I'll be happy to add this.  I'm afraid my PCIe understanding is too
piecemeal to fully grasp this, so concrete suggestions for the text
would be much appreciated.

> > Note that this platform specification is separate from the Linux kernel
> > concept of mach-virt, which merely specifies a machine model driven
> > purely from device tree, but does not mandate any peripherals or have any
> > mention of ACPI.
> 
> Did you notice we are removing mach-virt now? Probably no point
> mentioning it here.
> 
Yes, I'm aware.  I've just heard people say "why do we need this, isn't
mach-virt all we need", and therefore I added the note.

I can definitely can rid of this paragraph in the future if it causes
more harm than good.

Thanks!
-Christoffer

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 26/02/2014 20:55, Arnd Bergmann ha scritto:
>> > For more information about UEFI and ACPI booting, see [4] and [5].
> What's the point of having ACPI in a virtual machine? You wouldn't
> need to abstract any of the hardware in AML since you already know
> what the virtual hardware is, so I can't see how this would help
> anyone.

In x86 land it's been certainly helpful to abstract hotplug 
capabilities.  For ARM it could be the same.  Not so much for PCI (ARM 
probably can use native PCIe hotplug and standard hotplug controllers; 
on x86 we started with PCI and also have to deal with Windows's lack of 
support for SHPC), but it could help for CPU and memory hotplug.

> Did you notice we are removing mach-virt now? Probably no point
> mentioning it here.

Peter, do we still want mach-virt support in QEMU then?

Paolo

[RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 26/02/2014 20:55, Arnd Bergmann ha scritto:
>> > For more information about UEFI and ACPI booting, see [4] and [5].
> What's the point of having ACPI in a virtual machine? You wouldn't
> need to abstract any of the hardware in AML since you already know
> what the virtual hardware is, so I can't see how this would help
> anyone.

In x86 land it's been certainly helpful to abstract hotplug 
capabilities.  For ARM it could be the same.  Not so much for PCI (ARM 
probably can use native PCIe hotplug and standard hotplug controllers; 
on x86 we started with PCI and also have to deal with Windows's lack of 
support for SHPC), but it could help for CPU and memory hotplug.

> Did you notice we are removing mach-virt now? Probably no point
> mentioning it here.

Peter, do we still want mach-virt support in QEMU then?

Paolo

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 26/02/2014 20:55, Arnd Bergmann ha scritto:
>> > For more information about UEFI and ACPI booting, see [4] and [5].
> What's the point of having ACPI in a virtual machine? You wouldn't
> need to abstract any of the hardware in AML since you already know
> what the virtual hardware is, so I can't see how this would help
> anyone.

In x86 land it's been certainly helpful to abstract hotplug 
capabilities.  For ARM it could be the same.  Not so much for PCI (ARM 
probably can use native PCIe hotplug and standard hotplug controllers; 
on x86 we started with PCI and also have to deal with Windows's lack of 
support for SHPC), but it could help for CPU and memory hotplug.

> Did you notice we are removing mach-virt now? Probably no point
> mentioning it here.

Peter, do we still want mach-virt support in QEMU then?

Paolo

Re: [RFC] ARM VM System Sepcification

[Peter Maydell]

On 26 February 2014 20:19, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 26/02/2014 20:55, Arnd Bergmann ha scritto:
>> Did you notice we are removing mach-virt now? Probably no point
>> mentioning it here.
>
>
> Peter, do we still want mach-virt support in QEMU then?

Yes, it will be about the only thing that supports PCI-e :-)
When the kernel guys say "removing mach-virt" they mean
removing a specific lump of kernel code, not removing support
for having a kernel that can handle a machine described
only by the device tree.

thanks
-- PMM

[RFC] ARM VM System Sepcification

[Peter Maydell]

On 26 February 2014 20:19, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 26/02/2014 20:55, Arnd Bergmann ha scritto:
>> Did you notice we are removing mach-virt now? Probably no point
>> mentioning it here.
>
>
> Peter, do we still want mach-virt support in QEMU then?

Yes, it will be about the only thing that supports PCI-e :-)
When the kernel guys say "removing mach-virt" they mean
removing a specific lump of kernel code, not removing support
for having a kernel that can handle a machine described
only by the device tree.

thanks
-- PMM

Re: [RFC] ARM VM System Sepcification

[Peter Maydell]

On 26 February 2014 20:19, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 26/02/2014 20:55, Arnd Bergmann ha scritto:
>> Did you notice we are removing mach-virt now? Probably no point
>> mentioning it here.
>
>
> Peter, do we still want mach-virt support in QEMU then?

Yes, it will be about the only thing that supports PCI-e :-)
When the kernel guys say "removing mach-virt" they mean
removing a specific lump of kernel code, not removing support
for having a kernel that can handle a machine described
only by the device tree.

thanks
-- PMM

Re: [RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Wednesday 26 February 2014 12:05:37 Christoffer Dall wrote:
> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> 
> The most common response I've been getting so far is that people
> generally want their VMs to look close to the real thing, but not sure
> how valid an argument that is.
> 
> Some people feel strongly about this and seem to think that ARMv8
> kernels will only work with ACPI in the future...

That is certainly a misconception that has caused a lot of trouble.
We will certainly keep supporting FDT boot in ARMv8 indefinitely,
and I expect that most systems will not use ACPI at all.

The case for ACPI is really SBSA compliant servers, where ACPI serves
to abstract the hardware differences to let you boot an OS that does
not know about the system details for things like power management.

For embedded systems that are not SBSA compliant, using ACPI doesn't
gain you anything and causes a lot of headache, so we won't do that.

> Another case is that it's a good development platform.  I know nothing
> of developing and testing ACPI, so I won't judge one way or the other.

The interesting aspects of developing and testing ACPI are all related
to the hardware specific parts. Testing ACPI on a trivial virtual
machine doesn't gain you much once the basic support is there.

> > > VM Platform
> > > -----------
> > > The specification does not mandate any specific memory map.  The guest
> > > OS must be able to enumerate all processing elements, devices, and
> > > memory through HW description data (FDT, ACPI) or a bus-specific
> > > mechanism such as PCI.
> > > 
> > > The virtual platform must support at least one of the following ARM
> > > execution states:
> > >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> > >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> > >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> > > 
> > > It is recommended to support both (2) and (3) on aarch64 capable
> > > physical systems.
> > 
> > Isn't this more of a CPU capabilities question? Or maybe you
> > should just add 'if aarch32 mode supported is supported by
> > the host CPU'.
> 
> The recommendation is to tell people to actually have a -aarch32 (or
> whatever it would be called) that works in their VM implementation.
> This can certainly be reworded.

Yes, the intention is good, it just won't work on a few systems
when the CPU designers took the shortcut to implement only the
64-bit instructions. Apparently those systems will exist, but I
expect them to be the exception, and I don't know if they will
support virtualization.

> > I think you should specify exactly what you want PCIe to look like,
> > if present. Otherwise you can get wildly incompatible bus discovery.
> > 
> 
> As soon as there is more clarity on what it will actually look like,
> I'll be happy to add this.  I'm afraid my PCIe understanding is too
> piecemeal to fully grasp this, so concrete suggestions for the text
> would be much appreciated.

Will Deacon is currently prototyping a PCI model using kvmtool, trying
to make it as simple as possible, and compliant to SBSA. How about
we wait for the next version of that to see if we're happy with it,
and then figure out if all hypervisors we care about can use the
same interface.

> > > Note that this platform specification is separate from the Linux kernel
> > > concept of mach-virt, which merely specifies a machine model driven
> > > purely from device tree, but does not mandate any peripherals or have any
> > > mention of ACPI.
> > 
> > Did you notice we are removing mach-virt now? Probably no point
> > mentioning it here.
> > 
> Yes, I'm aware.  I've just heard people say "why do we need this, isn't
> mach-virt all we need", and therefore I added the note.
> 
> I can definitely can rid of this paragraph in the future if it causes
> more harm than good.

Maybe replace it with something like this:

| On both arm32 and arm64, no platform specific kernel code must be required,
| and all device detection must happen through the device description
| passed from the hypervisor or discoverable buses.

A harder question is what peripherals we should list as mandatory or
optional beyond what you have already. We could try coming up with
an exhaustive list of devices that are supported by mainline Linux-3.10
(the current longterm release) and implemented by any of the hypervisors,
but we probably want to leave open the possibility to extend it later
as we implement new virtual devices.

	Arnd

[RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Wednesday 26 February 2014 12:05:37 Christoffer Dall wrote:
> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> 
> The most common response I've been getting so far is that people
> generally want their VMs to look close to the real thing, but not sure
> how valid an argument that is.
> 
> Some people feel strongly about this and seem to think that ARMv8
> kernels will only work with ACPI in the future...

That is certainly a misconception that has caused a lot of trouble.
We will certainly keep supporting FDT boot in ARMv8 indefinitely,
and I expect that most systems will not use ACPI at all.

The case for ACPI is really SBSA compliant servers, where ACPI serves
to abstract the hardware differences to let you boot an OS that does
not know about the system details for things like power management.

For embedded systems that are not SBSA compliant, using ACPI doesn't
gain you anything and causes a lot of headache, so we won't do that.

> Another case is that it's a good development platform.  I know nothing
> of developing and testing ACPI, so I won't judge one way or the other.

The interesting aspects of developing and testing ACPI are all related
to the hardware specific parts. Testing ACPI on a trivial virtual
machine doesn't gain you much once the basic support is there.

> > > VM Platform
> > > -----------
> > > The specification does not mandate any specific memory map.  The guest
> > > OS must be able to enumerate all processing elements, devices, and
> > > memory through HW description data (FDT, ACPI) or a bus-specific
> > > mechanism such as PCI.
> > > 
> > > The virtual platform must support at least one of the following ARM
> > > execution states:
> > >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> > >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> > >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> > > 
> > > It is recommended to support both (2) and (3) on aarch64 capable
> > > physical systems.
> > 
> > Isn't this more of a CPU capabilities question? Or maybe you
> > should just add 'if aarch32 mode supported is supported by
> > the host CPU'.
> 
> The recommendation is to tell people to actually have a -aarch32 (or
> whatever it would be called) that works in their VM implementation.
> This can certainly be reworded.

Yes, the intention is good, it just won't work on a few systems
when the CPU designers took the shortcut to implement only the
64-bit instructions. Apparently those systems will exist, but I
expect them to be the exception, and I don't know if they will
support virtualization.

> > I think you should specify exactly what you want PCIe to look like,
> > if present. Otherwise you can get wildly incompatible bus discovery.
> > 
> 
> As soon as there is more clarity on what it will actually look like,
> I'll be happy to add this.  I'm afraid my PCIe understanding is too
> piecemeal to fully grasp this, so concrete suggestions for the text
> would be much appreciated.

Will Deacon is currently prototyping a PCI model using kvmtool, trying
to make it as simple as possible, and compliant to SBSA. How about
we wait for the next version of that to see if we're happy with it,
and then figure out if all hypervisors we care about can use the
same interface.

> > > Note that this platform specification is separate from the Linux kernel
> > > concept of mach-virt, which merely specifies a machine model driven
> > > purely from device tree, but does not mandate any peripherals or have any
> > > mention of ACPI.
> > 
> > Did you notice we are removing mach-virt now? Probably no point
> > mentioning it here.
> > 
> Yes, I'm aware.  I've just heard people say "why do we need this, isn't
> mach-virt all we need", and therefore I added the note.
> 
> I can definitely can rid of this paragraph in the future if it causes
> more harm than good.

Maybe replace it with something like this:

| On both arm32 and arm64, no platform specific kernel code must be required,
| and all device detection must happen through the device description
| passed from the hypervisor or discoverable buses.

A harder question is what peripherals we should list as mandatory or
optional beyond what you have already. We could try coming up with
an exhaustive list of devices that are supported by mainline Linux-3.10
(the current longterm release) and implemented by any of the hypervisors,
but we probably want to leave open the possibility to extend it later
as we implement new virtual devices.

	Arnd

Re: [RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Wednesday 26 February 2014 12:05:37 Christoffer Dall wrote:
> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> 
> The most common response I've been getting so far is that people
> generally want their VMs to look close to the real thing, but not sure
> how valid an argument that is.
> 
> Some people feel strongly about this and seem to think that ARMv8
> kernels will only work with ACPI in the future...

That is certainly a misconception that has caused a lot of trouble.
We will certainly keep supporting FDT boot in ARMv8 indefinitely,
and I expect that most systems will not use ACPI at all.

The case for ACPI is really SBSA compliant servers, where ACPI serves
to abstract the hardware differences to let you boot an OS that does
not know about the system details for things like power management.

For embedded systems that are not SBSA compliant, using ACPI doesn't
gain you anything and causes a lot of headache, so we won't do that.

> Another case is that it's a good development platform.  I know nothing
> of developing and testing ACPI, so I won't judge one way or the other.

The interesting aspects of developing and testing ACPI are all related
to the hardware specific parts. Testing ACPI on a trivial virtual
machine doesn't gain you much once the basic support is there.

> > > VM Platform
> > > -----------
> > > The specification does not mandate any specific memory map.  The guest
> > > OS must be able to enumerate all processing elements, devices, and
> > > memory through HW description data (FDT, ACPI) or a bus-specific
> > > mechanism such as PCI.
> > > 
> > > The virtual platform must support at least one of the following ARM
> > > execution states:
> > >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> > >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> > >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> > > 
> > > It is recommended to support both (2) and (3) on aarch64 capable
> > > physical systems.
> > 
> > Isn't this more of a CPU capabilities question? Or maybe you
> > should just add 'if aarch32 mode supported is supported by
> > the host CPU'.
> 
> The recommendation is to tell people to actually have a -aarch32 (or
> whatever it would be called) that works in their VM implementation.
> This can certainly be reworded.

Yes, the intention is good, it just won't work on a few systems
when the CPU designers took the shortcut to implement only the
64-bit instructions. Apparently those systems will exist, but I
expect them to be the exception, and I don't know if they will
support virtualization.

> > I think you should specify exactly what you want PCIe to look like,
> > if present. Otherwise you can get wildly incompatible bus discovery.
> > 
> 
> As soon as there is more clarity on what it will actually look like,
> I'll be happy to add this.  I'm afraid my PCIe understanding is too
> piecemeal to fully grasp this, so concrete suggestions for the text
> would be much appreciated.

Will Deacon is currently prototyping a PCI model using kvmtool, trying
to make it as simple as possible, and compliant to SBSA. How about
we wait for the next version of that to see if we're happy with it,
and then figure out if all hypervisors we care about can use the
same interface.

> > > Note that this platform specification is separate from the Linux kernel
> > > concept of mach-virt, which merely specifies a machine model driven
> > > purely from device tree, but does not mandate any peripherals or have any
> > > mention of ACPI.
> > 
> > Did you notice we are removing mach-virt now? Probably no point
> > mentioning it here.
> > 
> Yes, I'm aware.  I've just heard people say "why do we need this, isn't
> mach-virt all we need", and therefore I added the note.
> 
> I can definitely can rid of this paragraph in the future if it causes
> more harm than good.

Maybe replace it with something like this:

| On both arm32 and arm64, no platform specific kernel code must be required,
| and all device detection must happen through the device description
| passed from the hypervisor or discoverable buses.

A harder question is what peripherals we should list as mandatory or
optional beyond what you have already. We could try coming up with
an exhaustive list of devices that are supported by mainline Linux-3.10
(the current longterm release) and implemented by any of the hypervisors,
but we probably want to leave open the possibility to extend it later
as we implement new virtual devices.

	Arnd

Re: [RFC] ARM VM System Sepcification

[Michael Hudson-Doyle]

Christoffer Dall <christoffer.dall@linaro.org> writes:

> Hardware Description
> --------------------
> The Linux kernel's proper entry point always takes a pointer to an FDT,
> regardless of the boot mechanism, firmware, and hardware description
> method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> entry point will still receive a pointer to a simple FDT, generated by
> the Linux kernel UEFI stub, containing a pointer to the UEFI system
> table.  The kernel can then discover ACPI from the system tables.  The
> presence of ACPI vs. FDT is therefore always itself discoverable,
> through the FDT.
>
> Therefore, the VM implementation must provide through its UEFI
> implementation, either:
>
>   a complete FDT which describes the entire VM system and will boot
>   mainline kernels driven by device tree alone, or
>
>   no FDT.  In this case, the VM implementation must provide ACPI, and
>   the OS must be able to locate the ACPI root pointer through the UEFI
>   system table.

Maybe I'm missing something, but should this last bit say "a trivial
FDT" instead of "no FDT"?  If not, I don't understand the first
paragraph :-)

Cheers,
mwh

[RFC] ARM VM System Sepcification

[Michael Hudson-Doyle]

Christoffer Dall <christoffer.dall@linaro.org> writes:

> Hardware Description
> --------------------
> The Linux kernel's proper entry point always takes a pointer to an FDT,
> regardless of the boot mechanism, firmware, and hardware description
> method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> entry point will still receive a pointer to a simple FDT, generated by
> the Linux kernel UEFI stub, containing a pointer to the UEFI system
> table.  The kernel can then discover ACPI from the system tables.  The
> presence of ACPI vs. FDT is therefore always itself discoverable,
> through the FDT.
>
> Therefore, the VM implementation must provide through its UEFI
> implementation, either:
>
>   a complete FDT which describes the entire VM system and will boot
>   mainline kernels driven by device tree alone, or
>
>   no FDT.  In this case, the VM implementation must provide ACPI, and
>   the OS must be able to locate the ACPI root pointer through the UEFI
>   system table.

Maybe I'm missing something, but should this last bit say "a trivial
FDT" instead of "no FDT"?  If not, I don't understand the first
paragraph :-)

Cheers,
mwh

Re: [RFC] ARM VM System Sepcification

[Michael Hudson-Doyle]

Christoffer Dall <christoffer.dall@linaro.org> writes:

> Hardware Description
> --------------------
> The Linux kernel's proper entry point always takes a pointer to an FDT,
> regardless of the boot mechanism, firmware, and hardware description
> method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> entry point will still receive a pointer to a simple FDT, generated by
> the Linux kernel UEFI stub, containing a pointer to the UEFI system
> table.  The kernel can then discover ACPI from the system tables.  The
> presence of ACPI vs. FDT is therefore always itself discoverable,
> through the FDT.
>
> Therefore, the VM implementation must provide through its UEFI
> implementation, either:
>
>   a complete FDT which describes the entire VM system and will boot
>   mainline kernels driven by device tree alone, or
>
>   no FDT.  In this case, the VM implementation must provide ACPI, and
>   the OS must be able to locate the ACPI root pointer through the UEFI
>   system table.

Maybe I'm missing something, but should this last bit say "a trivial
FDT" instead of "no FDT"?  If not, I don't understand the first
paragraph :-)

Cheers,
mwh

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Thu, Feb 27, 2014 at 10:05:05AM +1300, Michael Hudson-Doyle wrote:
> Christoffer Dall <christoffer.dall@linaro.org> writes:
> 
> > Hardware Description
> > --------------------
> > The Linux kernel's proper entry point always takes a pointer to an FDT,
> > regardless of the boot mechanism, firmware, and hardware description
> > method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> > entry point will still receive a pointer to a simple FDT, generated by
> > the Linux kernel UEFI stub, containing a pointer to the UEFI system
> > table.  The kernel can then discover ACPI from the system tables.  The
> > presence of ACPI vs. FDT is therefore always itself discoverable,
> > through the FDT.
> >
> > Therefore, the VM implementation must provide through its UEFI
> > implementation, either:
> >
> >   a complete FDT which describes the entire VM system and will boot
> >   mainline kernels driven by device tree alone, or
> >
> >   no FDT.  In this case, the VM implementation must provide ACPI, and
> >   the OS must be able to locate the ACPI root pointer through the UEFI
> >   system table.
> 
> Maybe I'm missing something, but should this last bit say "a trivial
> FDT" instead of "no FDT"?  If not, I don't understand the first
> paragraph :-)
> 
That trivial FDT would be generated by the EFI stub in the kernel - not
provided by the VM implementation.

-Christoffer

[RFC] ARM VM System Sepcification

[Christoffer Dall]

On Thu, Feb 27, 2014 at 10:05:05AM +1300, Michael Hudson-Doyle wrote:
> Christoffer Dall <christoffer.dall@linaro.org> writes:
> 
> > Hardware Description
> > --------------------
> > The Linux kernel's proper entry point always takes a pointer to an FDT,
> > regardless of the boot mechanism, firmware, and hardware description
> > method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> > entry point will still receive a pointer to a simple FDT, generated by
> > the Linux kernel UEFI stub, containing a pointer to the UEFI system
> > table.  The kernel can then discover ACPI from the system tables.  The
> > presence of ACPI vs. FDT is therefore always itself discoverable,
> > through the FDT.
> >
> > Therefore, the VM implementation must provide through its UEFI
> > implementation, either:
> >
> >   a complete FDT which describes the entire VM system and will boot
> >   mainline kernels driven by device tree alone, or
> >
> >   no FDT.  In this case, the VM implementation must provide ACPI, and
> >   the OS must be able to locate the ACPI root pointer through the UEFI
> >   system table.
> 
> Maybe I'm missing something, but should this last bit say "a trivial
> FDT" instead of "no FDT"?  If not, I don't understand the first
> paragraph :-)
> 
That trivial FDT would be generated by the EFI stub in the kernel - not
provided by the VM implementation.

-Christoffer

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Thu, Feb 27, 2014 at 10:05:05AM +1300, Michael Hudson-Doyle wrote:
> Christoffer Dall <christoffer.dall@linaro.org> writes:
> 
> > Hardware Description
> > --------------------
> > The Linux kernel's proper entry point always takes a pointer to an FDT,
> > regardless of the boot mechanism, firmware, and hardware description
> > method.  Even on real hardware which only supports ACPI and UEFI, the kernel
> > entry point will still receive a pointer to a simple FDT, generated by
> > the Linux kernel UEFI stub, containing a pointer to the UEFI system
> > table.  The kernel can then discover ACPI from the system tables.  The
> > presence of ACPI vs. FDT is therefore always itself discoverable,
> > through the FDT.
> >
> > Therefore, the VM implementation must provide through its UEFI
> > implementation, either:
> >
> >   a complete FDT which describes the entire VM system and will boot
> >   mainline kernels driven by device tree alone, or
> >
> >   no FDT.  In this case, the VM implementation must provide ACPI, and
> >   the OS must be able to locate the ACPI root pointer through the UEFI
> >   system table.
> 
> Maybe I'm missing something, but should this last bit say "a trivial
> FDT" instead of "no FDT"?  If not, I don't understand the first
> paragraph :-)
> 
That trivial FDT would be generated by the EFI stub in the kernel - not
provided by the VM implementation.

-Christoffer

Re: [RFC] ARM VM System Sepcification

[Leif Lindholm]

On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> > ARM VM System Specification
> > ===========================
> > 
> > Goal
> > ----
> > The goal of this spec is to allow suitably-built OS images to run on
> > all ARM virtualization solutions, such as KVM or Xen.
> > 
> > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > they aim to be hypervisor agnostic.
> > 
> > Note that simply adhering to the SBSA [2] is not a valid approach,
> > for example because the SBSA mandates EL2, which will not be available
> > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > may be controversial for some ARM VM implementations to support.
> > This spec also covers the aarch32 execution mode, not covered in the
> > SBSA.
> 
> I would prefer if we can stay as close as possible to SBSA for individual
> hardware components, and only stray from it when there is a strong reason.
> pl011-subset doesn't sound like a significant problem to implement,
> especially as SBSA makes the DMA part of that optional. Can you
> elaborate on what hypervisor would have a problem with that?

I believe it comes down to how much extra overhead pl011-access-trap
would be over virtio-console. If low, then sure.
(Since there are certain things we cannot provide SBSA-compliant in
the guest anyway, I wouldn't consider lack of pl011 to be a big issue.)

> >   no FDT.  In this case, the VM implementation must provide ACPI, and
> >   the OS must be able to locate the ACPI root pointer through the UEFI
> >   system table.
> > 
> > For more information about the arm and arm64 boot conventions, see
> > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > Linux kernel source tree.
> > 
> > For more information about UEFI and ACPI booting, see [4] and [5].
> 
> What's the point of having ACPI in a virtual machine? You wouldn't
> need to abstract any of the hardware in AML since you already know
> what the virtual hardware is, so I can't see how this would help
> anyone.

The point is that if we need to share any real hw then we need to use
whatever the host has.

> However, as ACPI will not be supported by arm32, not having the
> complete FDT will prevent you from running a 32-bit guest on
> a 64-bit hypervisor, which I consider an important use case.

In which case we would be making an active call to ban anything other
than virtio/xen-pv devices for 32-bit guests on hardware without DT.

However, I see the case of a 32-bit guest on 64-bit hypervisor as less
likely in the server space than in mobile, and ACPI in mobile as
unlikely, so it may end up not being a big issue.

/
    Leif

[RFC] ARM VM System Sepcification

[Leif Lindholm]

On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> > ARM VM System Specification
> > ===========================
> > 
> > Goal
> > ----
> > The goal of this spec is to allow suitably-built OS images to run on
> > all ARM virtualization solutions, such as KVM or Xen.
> > 
> > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > they aim to be hypervisor agnostic.
> > 
> > Note that simply adhering to the SBSA [2] is not a valid approach,
> > for example because the SBSA mandates EL2, which will not be available
> > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > may be controversial for some ARM VM implementations to support.
> > This spec also covers the aarch32 execution mode, not covered in the
> > SBSA.
> 
> I would prefer if we can stay as close as possible to SBSA for individual
> hardware components, and only stray from it when there is a strong reason.
> pl011-subset doesn't sound like a significant problem to implement,
> especially as SBSA makes the DMA part of that optional. Can you
> elaborate on what hypervisor would have a problem with that?

I believe it comes down to how much extra overhead pl011-access-trap
would be over virtio-console. If low, then sure.
(Since there are certain things we cannot provide SBSA-compliant in
the guest anyway, I wouldn't consider lack of pl011 to be a big issue.)

> >   no FDT.  In this case, the VM implementation must provide ACPI, and
> >   the OS must be able to locate the ACPI root pointer through the UEFI
> >   system table.
> > 
> > For more information about the arm and arm64 boot conventions, see
> > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > Linux kernel source tree.
> > 
> > For more information about UEFI and ACPI booting, see [4] and [5].
> 
> What's the point of having ACPI in a virtual machine? You wouldn't
> need to abstract any of the hardware in AML since you already know
> what the virtual hardware is, so I can't see how this would help
> anyone.

The point is that if we need to share any real hw then we need to use
whatever the host has.

> However, as ACPI will not be supported by arm32, not having the
> complete FDT will prevent you from running a 32-bit guest on
> a 64-bit hypervisor, which I consider an important use case.

In which case we would be making an active call to ban anything other
than virtio/xen-pv devices for 32-bit guests on hardware without DT.

However, I see the case of a 32-bit guest on 64-bit hypervisor as less
likely in the server space than in mobile, and ACPI in mobile as
unlikely, so it may end up not being a big issue.

/
    Leif

Re: [RFC] ARM VM System Sepcification

[Leif Lindholm]

On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> > ARM VM System Specification
> > ===========================
> > 
> > Goal
> > ----
> > The goal of this spec is to allow suitably-built OS images to run on
> > all ARM virtualization solutions, such as KVM or Xen.
> > 
> > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > they aim to be hypervisor agnostic.
> > 
> > Note that simply adhering to the SBSA [2] is not a valid approach,
> > for example because the SBSA mandates EL2, which will not be available
> > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > may be controversial for some ARM VM implementations to support.
> > This spec also covers the aarch32 execution mode, not covered in the
> > SBSA.
> 
> I would prefer if we can stay as close as possible to SBSA for individual
> hardware components, and only stray from it when there is a strong reason.
> pl011-subset doesn't sound like a significant problem to implement,
> especially as SBSA makes the DMA part of that optional. Can you
> elaborate on what hypervisor would have a problem with that?

I believe it comes down to how much extra overhead pl011-access-trap
would be over virtio-console. If low, then sure.
(Since there are certain things we cannot provide SBSA-compliant in
the guest anyway, I wouldn't consider lack of pl011 to be a big issue.)

> >   no FDT.  In this case, the VM implementation must provide ACPI, and
> >   the OS must be able to locate the ACPI root pointer through the UEFI
> >   system table.
> > 
> > For more information about the arm and arm64 boot conventions, see
> > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > Linux kernel source tree.
> > 
> > For more information about UEFI and ACPI booting, see [4] and [5].
> 
> What's the point of having ACPI in a virtual machine? You wouldn't
> need to abstract any of the hardware in AML since you already know
> what the virtual hardware is, so I can't see how this would help
> anyone.

The point is that if we need to share any real hw then we need to use
whatever the host has.

> However, as ACPI will not be supported by arm32, not having the
> complete FDT will prevent you from running a 32-bit guest on
> a 64-bit hypervisor, which I consider an important use case.

In which case we would be making an active call to ban anything other
than virtio/xen-pv devices for 32-bit guests on hardware without DT.

However, I see the case of a 32-bit guest on 64-bit hypervisor as less
likely in the server space than in mobile, and ACPI in mobile as
unlikely, so it may end up not being a big issue.

/
    Leif

Re: [RFC] ARM VM System Sepcification

[Rob Herring]

On Wed, Feb 26, 2014 at 2:22 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Wednesday 26 February 2014 12:05:37 Christoffer Dall wrote:
>> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
>> > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
>>
>> The most common response I've been getting so far is that people
>> generally want their VMs to look close to the real thing, but not sure
>> how valid an argument that is.
>>
>> Some people feel strongly about this and seem to think that ARMv8
>> kernels will only work with ACPI in the future...
>
> That is certainly a misconception that has caused a lot of trouble.
> We will certainly keep supporting FDT boot in ARMv8 indefinitely,
> and I expect that most systems will not use ACPI at all.

Furthermore, even enterprise distro kernels will boot DT based kernels
assuming the h/w support is mainlined despite statements to the
contrary. It is a requirement in mainline kernels that DT and ACPI
support to coexist. Distros are not going to go out of their way to
undo/break that. And since the boot interface is DT, you can't simply
turn off DT. :)

Rob

[RFC] ARM VM System Sepcification

[Rob Herring]

On Wed, Feb 26, 2014 at 2:22 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Wednesday 26 February 2014 12:05:37 Christoffer Dall wrote:
>> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
>> > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
>>
>> The most common response I've been getting so far is that people
>> generally want their VMs to look close to the real thing, but not sure
>> how valid an argument that is.
>>
>> Some people feel strongly about this and seem to think that ARMv8
>> kernels will only work with ACPI in the future...
>
> That is certainly a misconception that has caused a lot of trouble.
> We will certainly keep supporting FDT boot in ARMv8 indefinitely,
> and I expect that most systems will not use ACPI at all.

Furthermore, even enterprise distro kernels will boot DT based kernels
assuming the h/w support is mainlined despite statements to the
contrary. It is a requirement in mainline kernels that DT and ACPI
support to coexist. Distros are not going to go out of their way to
undo/break that. And since the boot interface is DT, you can't simply
turn off DT. :)

Rob

Re: [RFC] ARM VM System Sepcification

[Rob Herring]

On Wed, Feb 26, 2014 at 2:22 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Wednesday 26 February 2014 12:05:37 Christoffer Dall wrote:
>> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
>> > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
>>
>> The most common response I've been getting so far is that people
>> generally want their VMs to look close to the real thing, but not sure
>> how valid an argument that is.
>>
>> Some people feel strongly about this and seem to think that ARMv8
>> kernels will only work with ACPI in the future...
>
> That is certainly a misconception that has caused a lot of trouble.
> We will certainly keep supporting FDT boot in ARMv8 indefinitely,
> and I expect that most systems will not use ACPI at all.

Furthermore, even enterprise distro kernels will boot DT based kernels
assuming the h/w support is mainlined despite statements to the
contrary. It is a requirement in mainline kernels that DT and ACPI
support to coexist. Distros are not going to go out of their way to
undo/break that. And since the boot interface is DT, you can't simply
turn off DT. :)

Rob

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Wed, Feb 26, 2014 at 03:56:02PM -0600, Rob Herring wrote:
> On Wed, Feb 26, 2014 at 2:22 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> > On Wednesday 26 February 2014 12:05:37 Christoffer Dall wrote:
> >> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> >> > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> >>
> >> The most common response I've been getting so far is that people
> >> generally want their VMs to look close to the real thing, but not sure
> >> how valid an argument that is.
> >>
> >> Some people feel strongly about this and seem to think that ARMv8
> >> kernels will only work with ACPI in the future...
> >
> > That is certainly a misconception that has caused a lot of trouble.
> > We will certainly keep supporting FDT boot in ARMv8 indefinitely,
> > and I expect that most systems will not use ACPI at all.
> 
> Furthermore, even enterprise distro kernels will boot DT based kernels
> assuming the h/w support is mainlined despite statements to the
> contrary. It is a requirement in mainline kernels that DT and ACPI
> support to coexist. Distros are not going to go out of their way to
> undo/break that. And since the boot interface is DT, you can't simply
> turn off DT. :)
> 

Personally I'm all for simplicity so I don't want to push any agenda for
ACPI in VMs.

Note that the spec does not mandate the use of ACPI, it just tells you
how to do it if you wish to.

But, we can change the spec to require full FDT description of the
system, unless of course some of the ACPI-in-VM supporters manage to
convince the rest.

-Christoffer

[RFC] ARM VM System Sepcification

[Christoffer Dall]

On Wed, Feb 26, 2014 at 03:56:02PM -0600, Rob Herring wrote:
> On Wed, Feb 26, 2014 at 2:22 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> > On Wednesday 26 February 2014 12:05:37 Christoffer Dall wrote:
> >> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> >> > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> >>
> >> The most common response I've been getting so far is that people
> >> generally want their VMs to look close to the real thing, but not sure
> >> how valid an argument that is.
> >>
> >> Some people feel strongly about this and seem to think that ARMv8
> >> kernels will only work with ACPI in the future...
> >
> > That is certainly a misconception that has caused a lot of trouble.
> > We will certainly keep supporting FDT boot in ARMv8 indefinitely,
> > and I expect that most systems will not use ACPI at all.
> 
> Furthermore, even enterprise distro kernels will boot DT based kernels
> assuming the h/w support is mainlined despite statements to the
> contrary. It is a requirement in mainline kernels that DT and ACPI
> support to coexist. Distros are not going to go out of their way to
> undo/break that. And since the boot interface is DT, you can't simply
> turn off DT. :)
> 

Personally I'm all for simplicity so I don't want to push any agenda for
ACPI in VMs.

Note that the spec does not mandate the use of ACPI, it just tells you
how to do it if you wish to.

But, we can change the spec to require full FDT description of the
system, unless of course some of the ACPI-in-VM supporters manage to
convince the rest.

-Christoffer

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Wed, Feb 26, 2014 at 03:56:02PM -0600, Rob Herring wrote:
> On Wed, Feb 26, 2014 at 2:22 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> > On Wednesday 26 February 2014 12:05:37 Christoffer Dall wrote:
> >> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> >> > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> >>
> >> The most common response I've been getting so far is that people
> >> generally want their VMs to look close to the real thing, but not sure
> >> how valid an argument that is.
> >>
> >> Some people feel strongly about this and seem to think that ARMv8
> >> kernels will only work with ACPI in the future...
> >
> > That is certainly a misconception that has caused a lot of trouble.
> > We will certainly keep supporting FDT boot in ARMv8 indefinitely,
> > and I expect that most systems will not use ACPI at all.
> 
> Furthermore, even enterprise distro kernels will boot DT based kernels
> assuming the h/w support is mainlined despite statements to the
> contrary. It is a requirement in mainline kernels that DT and ACPI
> support to coexist. Distros are not going to go out of their way to
> undo/break that. And since the boot interface is DT, you can't simply
> turn off DT. :)
> 

Personally I'm all for simplicity so I don't want to push any agenda for
ACPI in VMs.

Note that the spec does not mandate the use of ACPI, it just tells you
how to do it if you wish to.

But, we can change the spec to require full FDT description of the
system, unless of course some of the ACPI-in-VM supporters manage to
convince the rest.

-Christoffer

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Wed, Feb 26, 2014 at 09:48:43PM +0000, Leif Lindholm wrote:
> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> > > ARM VM System Specification
> > > ===========================
> > > 
> > > Goal
> > > ----
> > > The goal of this spec is to allow suitably-built OS images to run on
> > > all ARM virtualization solutions, such as KVM or Xen.
> > > 
> > > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > > they aim to be hypervisor agnostic.
> > > 
> > > Note that simply adhering to the SBSA [2] is not a valid approach,
> > > for example because the SBSA mandates EL2, which will not be available
> > > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > > may be controversial for some ARM VM implementations to support.
> > > This spec also covers the aarch32 execution mode, not covered in the
> > > SBSA.
> > 
> > I would prefer if we can stay as close as possible to SBSA for individual
> > hardware components, and only stray from it when there is a strong reason.
> > pl011-subset doesn't sound like a significant problem to implement,
> > especially as SBSA makes the DMA part of that optional. Can you
> > elaborate on what hypervisor would have a problem with that?
> 
> I believe it comes down to how much extra overhead pl011-access-trap
> would be over virtio-console. If low, then sure.
> (Since there are certain things we cannot provide SBSA-compliant in
> the guest anyway, I wouldn't consider lack of pl011 to be a big issue.)
> 

I don't think it's about overhead, sure pl011 may be slower, but it's a
serial port, does anyone care about performance of a console? pl011
should be good enough for sure.

I think the issue is that Xen does no real device emulation today at
all, they don't use QEMU etc.  That being said, adding a pl011 emulation
in the Xen tools doesn't seem like the worst idea in the world, but I
will let them chime in on why they are opposed to it.

> > >   no FDT.  In this case, the VM implementation must provide ACPI, and
> > >   the OS must be able to locate the ACPI root pointer through the UEFI
> > >   system table.
> > > 
> > > For more information about the arm and arm64 boot conventions, see
> > > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > > Linux kernel source tree.
> > > 
> > > For more information about UEFI and ACPI booting, see [4] and [5].
> > 
> > What's the point of having ACPI in a virtual machine? You wouldn't
> > need to abstract any of the hardware in AML since you already know
> > what the virtual hardware is, so I can't see how this would help
> > anyone.
> 
> The point is that if we need to share any real hw then we need to use
> whatever the host has.
> 
> > However, as ACPI will not be supported by arm32, not having the
> > complete FDT will prevent you from running a 32-bit guest on
> > a 64-bit hypervisor, which I consider an important use case.
> 
> In which case we would be making an active call to ban anything other
> than virtio/xen-pv devices for 32-bit guests on hardware without DT.
> 
> However, I see the case of a 32-bit guest on 64-bit hypervisor as less
> likely in the server space than in mobile, and ACPI in mobile as
> unlikely, so it may end up not being a big issue.
> 
But you may want a networking applicance, for example, built on ARM
64-bit hardware, with an ARM 64-bit hypervisor on there, and you wish to
run some 32-bit system in a VM.  If the hardware is ACPI based, it would
be a shame if the VM implementation (or the virtual firmware) couldn't
provide a useable 32-bit FDT.  We may want to call that case out in this
document?

Thanks,
-Christoffer

[RFC] ARM VM System Sepcification

[Christoffer Dall]

On Wed, Feb 26, 2014 at 09:48:43PM +0000, Leif Lindholm wrote:
> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> > > ARM VM System Specification
> > > ===========================
> > > 
> > > Goal
> > > ----
> > > The goal of this spec is to allow suitably-built OS images to run on
> > > all ARM virtualization solutions, such as KVM or Xen.
> > > 
> > > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > > they aim to be hypervisor agnostic.
> > > 
> > > Note that simply adhering to the SBSA [2] is not a valid approach,
> > > for example because the SBSA mandates EL2, which will not be available
> > > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > > may be controversial for some ARM VM implementations to support.
> > > This spec also covers the aarch32 execution mode, not covered in the
> > > SBSA.
> > 
> > I would prefer if we can stay as close as possible to SBSA for individual
> > hardware components, and only stray from it when there is a strong reason.
> > pl011-subset doesn't sound like a significant problem to implement,
> > especially as SBSA makes the DMA part of that optional. Can you
> > elaborate on what hypervisor would have a problem with that?
> 
> I believe it comes down to how much extra overhead pl011-access-trap
> would be over virtio-console. If low, then sure.
> (Since there are certain things we cannot provide SBSA-compliant in
> the guest anyway, I wouldn't consider lack of pl011 to be a big issue.)
> 

I don't think it's about overhead, sure pl011 may be slower, but it's a
serial port, does anyone care about performance of a console? pl011
should be good enough for sure.

I think the issue is that Xen does no real device emulation today at
all, they don't use QEMU etc.  That being said, adding a pl011 emulation
in the Xen tools doesn't seem like the worst idea in the world, but I
will let them chime in on why they are opposed to it.

> > >   no FDT.  In this case, the VM implementation must provide ACPI, and
> > >   the OS must be able to locate the ACPI root pointer through the UEFI
> > >   system table.
> > > 
> > > For more information about the arm and arm64 boot conventions, see
> > > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > > Linux kernel source tree.
> > > 
> > > For more information about UEFI and ACPI booting, see [4] and [5].
> > 
> > What's the point of having ACPI in a virtual machine? You wouldn't
> > need to abstract any of the hardware in AML since you already know
> > what the virtual hardware is, so I can't see how this would help
> > anyone.
> 
> The point is that if we need to share any real hw then we need to use
> whatever the host has.
> 
> > However, as ACPI will not be supported by arm32, not having the
> > complete FDT will prevent you from running a 32-bit guest on
> > a 64-bit hypervisor, which I consider an important use case.
> 
> In which case we would be making an active call to ban anything other
> than virtio/xen-pv devices for 32-bit guests on hardware without DT.
> 
> However, I see the case of a 32-bit guest on 64-bit hypervisor as less
> likely in the server space than in mobile, and ACPI in mobile as
> unlikely, so it may end up not being a big issue.
> 
But you may want a networking applicance, for example, built on ARM
64-bit hardware, with an ARM 64-bit hypervisor on there, and you wish to
run some 32-bit system in a VM.  If the hardware is ACPI based, it would
be a shame if the VM implementation (or the virtual firmware) couldn't
provide a useable 32-bit FDT.  We may want to call that case out in this
document?

Thanks,
-Christoffer

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Wed, Feb 26, 2014 at 09:48:43PM +0000, Leif Lindholm wrote:
> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> > > ARM VM System Specification
> > > ===========================
> > > 
> > > Goal
> > > ----
> > > The goal of this spec is to allow suitably-built OS images to run on
> > > all ARM virtualization solutions, such as KVM or Xen.
> > > 
> > > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > > they aim to be hypervisor agnostic.
> > > 
> > > Note that simply adhering to the SBSA [2] is not a valid approach,
> > > for example because the SBSA mandates EL2, which will not be available
> > > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > > may be controversial for some ARM VM implementations to support.
> > > This spec also covers the aarch32 execution mode, not covered in the
> > > SBSA.
> > 
> > I would prefer if we can stay as close as possible to SBSA for individual
> > hardware components, and only stray from it when there is a strong reason.
> > pl011-subset doesn't sound like a significant problem to implement,
> > especially as SBSA makes the DMA part of that optional. Can you
> > elaborate on what hypervisor would have a problem with that?
> 
> I believe it comes down to how much extra overhead pl011-access-trap
> would be over virtio-console. If low, then sure.
> (Since there are certain things we cannot provide SBSA-compliant in
> the guest anyway, I wouldn't consider lack of pl011 to be a big issue.)
> 

I don't think it's about overhead, sure pl011 may be slower, but it's a
serial port, does anyone care about performance of a console? pl011
should be good enough for sure.

I think the issue is that Xen does no real device emulation today at
all, they don't use QEMU etc.  That being said, adding a pl011 emulation
in the Xen tools doesn't seem like the worst idea in the world, but I
will let them chime in on why they are opposed to it.

> > >   no FDT.  In this case, the VM implementation must provide ACPI, and
> > >   the OS must be able to locate the ACPI root pointer through the UEFI
> > >   system table.
> > > 
> > > For more information about the arm and arm64 boot conventions, see
> > > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > > Linux kernel source tree.
> > > 
> > > For more information about UEFI and ACPI booting, see [4] and [5].
> > 
> > What's the point of having ACPI in a virtual machine? You wouldn't
> > need to abstract any of the hardware in AML since you already know
> > what the virtual hardware is, so I can't see how this would help
> > anyone.
> 
> The point is that if we need to share any real hw then we need to use
> whatever the host has.
> 
> > However, as ACPI will not be supported by arm32, not having the
> > complete FDT will prevent you from running a 32-bit guest on
> > a 64-bit hypervisor, which I consider an important use case.
> 
> In which case we would be making an active call to ban anything other
> than virtio/xen-pv devices for 32-bit guests on hardware without DT.
> 
> However, I see the case of a 32-bit guest on 64-bit hypervisor as less
> likely in the server space than in mobile, and ACPI in mobile as
> unlikely, so it may end up not being a big issue.
> 
But you may want a networking applicance, for example, built on ARM
64-bit hardware, with an ARM 64-bit hypervisor on there, and you wish to
run some 32-bit system in a VM.  If the hardware is ACPI based, it would
be a shame if the VM implementation (or the virtual firmware) couldn't
provide a useable 32-bit FDT.  We may want to call that case out in this
document?

Thanks,
-Christoffer

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

[-- Attachment #1.1: Type: text/plain, Size: 7537 bytes --]

Hi Christoffer,

Comments below...

On 26 Feb 2014 18:35, "Christoffer Dall" <christoffer.dall-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
wrote:
>
> ARM VM System Specification
> ===========================
>
> Goal
> ----
> The goal of this spec is to allow suitably-built OS images to run on
> all ARM virtualization solutions, such as KVM or Xen.
>
> Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> they aim to be hypervisor agnostic.
>
> Note that simply adhering to the SBSA [2] is not a valid approach,
> for example because the SBSA mandates EL2, which will not be available
> for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> may be controversial for some ARM VM implementations to support.
> This spec also covers the aarch32 execution mode, not covered in the
> SBSA.
>
>
> Image format
> ------------
> The image format, as presented to the VM, needs to be well-defined in
> order for prepared disk images to be bootable across various
> virtualization implementations.
>
> The raw disk format as presented to the VM must be partitioned with a
> GUID Partition Table (GPT).  The bootable software must be placed in the
> EFI System Partition (ESP), using the UEFI removable media path, and
> must be an EFI application complying to the UEFI Specification 2.4
> Revision A [6].
>
> The ESP partition's GPT entry's partition type GUID must be
> C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
>
> The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> state.
>
> This ensures that tools for both Xen and KVM can load a binary UEFI
> firmware which can read and boot the EFI application in the disk image.
>
> A typical scenario will be GRUB2 packaged as an EFI application, which
> mounts the system boot partition and boots Linux.
>
>
> Virtual Firmware
> ----------------
> The VM system must be able to boot the EFI application in the ESP.  It
> is recommended that this is achieved by loading a UEFI binary as the
> first software executed by the VM, which then executes the EFI
> application.  The UEFI implementation should be compliant with UEFI
> Specification 2.4 Revision A [6] or later.
>
> This document strongly recommends that the VM implementation supports
> persistent environment storage for virtual firmware implementation in
> order to ensure probable use cases such as adding additional disk images
> to a VM or running installers to perform upgrades.
>
> The binary UEFI firmware implementation should not be distributed as
> part of the VM image, but is specific to the VM implementation.
>
>
> Hardware Description
> --------------------
> The Linux kernel's proper entry point always takes a pointer to an FDT,
> regardless of the boot mechanism, firmware, and hardware description
> method.  Even on real hardware which only supports ACPI and UEFI, the
kernel
> entry point will still receive a pointer to a simple FDT, generated by
> the Linux kernel UEFI stub, containing a pointer to the UEFI system
> table.  The kernel can then discover ACPI from the system tables.  The
> presence of ACPI vs. FDT is therefore always itself discoverable,
> through the FDT.

I would drop pretty much all of the above detail of the kernel entry point.
The spec should specify UEFI compliance and stop there.

What is relevant is the allowance for the UEFI implementation to provide an
FDT and/or ACPI via the configuration table.

>
> Therefore, the VM implementation must provide through its UEFI
> implementation, either:
>
>   a complete FDT which describes the entire VM system and will boot
>   mainline kernels driven by device tree alone, or
>
>   no FDT.  In this case, the VM implementation must provide ACPI, and
>   the OS must be able to locate the ACPI root pointer through the UEFI
>   system table.

It is actually valid for the VM to provide both ACPI and FDT. In that
scenario it is up to the OS to chose which it will use.

> For more information about the arm and arm64 boot conventions, see
> Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> Linux kernel source tree.
>
> For more information about UEFI and ACPI booting, see [4] and [5].
>
>
> VM Platform
> -----------
> The specification does not mandate any specific memory map.  The guest
> OS must be able to enumerate all processing elements, devices, and
> memory through HW description data (FDT, ACPI) or a bus-specific
> mechanism such as PCI.
>
> The virtual platform must support at least one of the following ARM
> execution states:
>   (1) aarch32 virtual CPUs on aarch32 physical CPUs
>   (2) aarch32 virtual CPUs on aarch64 physical CPUs
>   (3) aarch64 virtual CPUs on aarch64 physical CPUs
>
> It is recommended to support both (2) and (3) on aarch64 capable
> physical systems.
>
> The virtual hardware platform must provide a number of mandatory
> peripherals:
>
>   Serial console:  The platform should provide a console,
>   based on an emulated pl011, a virtio-console, or a Xen PV console.

For portable disk image, can Xen PV be dropped from the list? pl011 is part
of SBSA, and virtio is getting standardised, but Xen PV is implementation
specific.

>
>   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
>   limits the the number of virtual CPUs to 8 cores, newer GIC versions
>   removes this limitation.
>
>   The ARM virtual timer and counter should be available to the VM as
>   per the ARM Generic Timers specification in the ARM ARM [1].
>
>   A hotpluggable bus to support hotplug of at least block and network
>   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
>   bus.
>
>
> We make the following recommendations for the guest OS kernel:
>
>   The guest OS must include support for GICv2 and any available newer
>   version of the GIC architecture to maintain compatibility with older
>   VM implementations.
>
>   It is strongly recommended to include support for all available
>   (block, network, console, balloon) virtio-pci, virtio-mmio, and Xen PV
>   drivers in the guest OS kernel or initial ramdisk.
>
>
> Other common peripherals for block devices, networking, and more can
> (and typically will) be provided, but OS software written and compiled
> to run on ARM VMs cannot make any assumptions about which variations
> of these should exist or which implementation they use (e.g. VirtIO or
> Xen PV).  See "Hardware Description" above.
>
> Note that this platform specification is separate from the Linux kernel
> concept of mach-virt, which merely specifies a machine model driven
> purely from device tree, but does not mandate any peripherals or have any
> mention of ACPI.
>
>
> References
> ----------
> [1]: The ARM Architecture Reference Manual, ARMv8, Issue A.b
>
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html
>
> [2]: ARM Server Base System Architecture
>
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0029/index.html
>
> [3]: The ARM Generic Interrupt Controller Architecture Specifications v2.0
>
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html
>
> [4]: http://www.secretlab.ca/archives/27
>
> [5]:
https://git.linaro.org/people/leif.lindholm/linux.git/blob/refs/heads/uefi-for-upstream:/Documentation/arm/uefi.txt
>
> [6]: UEFI Specification 2.4 Revision A
> http://www.uefi.org/sites/default/files/resources/2_4_Errata_A.pdf

[-- Attachment #1.2: Type: text/html, Size: 9683 bytes --]

[-- Attachment #2: Type: text/plain, Size: 181 bytes --]

_______________________________________________
cross-distro mailing list
cross-distro-cunTk1MwBs8s++Sfvej+rw@public.gmane.org
http://lists.linaro.org/mailman/listinfo/cross-distro

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

[-- Attachment #1.1: Type: text/plain, Size: 7508 bytes --]

Hi Christoffer,

Comments below...

On 26 Feb 2014 18:35, "Christoffer Dall" <christoffer.dall@linaro.org>
wrote:
>
> ARM VM System Specification
> ===========================
>
> Goal
> ----
> The goal of this spec is to allow suitably-built OS images to run on
> all ARM virtualization solutions, such as KVM or Xen.
>
> Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> they aim to be hypervisor agnostic.
>
> Note that simply adhering to the SBSA [2] is not a valid approach,
> for example because the SBSA mandates EL2, which will not be available
> for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> may be controversial for some ARM VM implementations to support.
> This spec also covers the aarch32 execution mode, not covered in the
> SBSA.
>
>
> Image format
> ------------
> The image format, as presented to the VM, needs to be well-defined in
> order for prepared disk images to be bootable across various
> virtualization implementations.
>
> The raw disk format as presented to the VM must be partitioned with a
> GUID Partition Table (GPT).  The bootable software must be placed in the
> EFI System Partition (ESP), using the UEFI removable media path, and
> must be an EFI application complying to the UEFI Specification 2.4
> Revision A [6].
>
> The ESP partition's GPT entry's partition type GUID must be
> C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
>
> The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> state.
>
> This ensures that tools for both Xen and KVM can load a binary UEFI
> firmware which can read and boot the EFI application in the disk image.
>
> A typical scenario will be GRUB2 packaged as an EFI application, which
> mounts the system boot partition and boots Linux.
>
>
> Virtual Firmware
> ----------------
> The VM system must be able to boot the EFI application in the ESP.  It
> is recommended that this is achieved by loading a UEFI binary as the
> first software executed by the VM, which then executes the EFI
> application.  The UEFI implementation should be compliant with UEFI
> Specification 2.4 Revision A [6] or later.
>
> This document strongly recommends that the VM implementation supports
> persistent environment storage for virtual firmware implementation in
> order to ensure probable use cases such as adding additional disk images
> to a VM or running installers to perform upgrades.
>
> The binary UEFI firmware implementation should not be distributed as
> part of the VM image, but is specific to the VM implementation.
>
>
> Hardware Description
> --------------------
> The Linux kernel's proper entry point always takes a pointer to an FDT,
> regardless of the boot mechanism, firmware, and hardware description
> method.  Even on real hardware which only supports ACPI and UEFI, the
kernel
> entry point will still receive a pointer to a simple FDT, generated by
> the Linux kernel UEFI stub, containing a pointer to the UEFI system
> table.  The kernel can then discover ACPI from the system tables.  The
> presence of ACPI vs. FDT is therefore always itself discoverable,
> through the FDT.

I would drop pretty much all of the above detail of the kernel entry point.
The spec should specify UEFI compliance and stop there.

What is relevant is the allowance for the UEFI implementation to provide an
FDT and/or ACPI via the configuration table.

>
> Therefore, the VM implementation must provide through its UEFI
> implementation, either:
>
>   a complete FDT which describes the entire VM system and will boot
>   mainline kernels driven by device tree alone, or
>
>   no FDT.  In this case, the VM implementation must provide ACPI, and
>   the OS must be able to locate the ACPI root pointer through the UEFI
>   system table.

It is actually valid for the VM to provide both ACPI and FDT. In that
scenario it is up to the OS to chose which it will use.

> For more information about the arm and arm64 boot conventions, see
> Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> Linux kernel source tree.
>
> For more information about UEFI and ACPI booting, see [4] and [5].
>
>
> VM Platform
> -----------
> The specification does not mandate any specific memory map.  The guest
> OS must be able to enumerate all processing elements, devices, and
> memory through HW description data (FDT, ACPI) or a bus-specific
> mechanism such as PCI.
>
> The virtual platform must support at least one of the following ARM
> execution states:
>   (1) aarch32 virtual CPUs on aarch32 physical CPUs
>   (2) aarch32 virtual CPUs on aarch64 physical CPUs
>   (3) aarch64 virtual CPUs on aarch64 physical CPUs
>
> It is recommended to support both (2) and (3) on aarch64 capable
> physical systems.
>
> The virtual hardware platform must provide a number of mandatory
> peripherals:
>
>   Serial console:  The platform should provide a console,
>   based on an emulated pl011, a virtio-console, or a Xen PV console.

For portable disk image, can Xen PV be dropped from the list? pl011 is part
of SBSA, and virtio is getting standardised, but Xen PV is implementation
specific.

>
>   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
>   limits the the number of virtual CPUs to 8 cores, newer GIC versions
>   removes this limitation.
>
>   The ARM virtual timer and counter should be available to the VM as
>   per the ARM Generic Timers specification in the ARM ARM [1].
>
>   A hotpluggable bus to support hotplug of at least block and network
>   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
>   bus.
>
>
> We make the following recommendations for the guest OS kernel:
>
>   The guest OS must include support for GICv2 and any available newer
>   version of the GIC architecture to maintain compatibility with older
>   VM implementations.
>
>   It is strongly recommended to include support for all available
>   (block, network, console, balloon) virtio-pci, virtio-mmio, and Xen PV
>   drivers in the guest OS kernel or initial ramdisk.
>
>
> Other common peripherals for block devices, networking, and more can
> (and typically will) be provided, but OS software written and compiled
> to run on ARM VMs cannot make any assumptions about which variations
> of these should exist or which implementation they use (e.g. VirtIO or
> Xen PV).  See "Hardware Description" above.
>
> Note that this platform specification is separate from the Linux kernel
> concept of mach-virt, which merely specifies a machine model driven
> purely from device tree, but does not mandate any peripherals or have any
> mention of ACPI.
>
>
> References
> ----------
> [1]: The ARM Architecture Reference Manual, ARMv8, Issue A.b
>
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html
>
> [2]: ARM Server Base System Architecture
>
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0029/index.html
>
> [3]: The ARM Generic Interrupt Controller Architecture Specifications v2.0
>
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html
>
> [4]: http://www.secretlab.ca/archives/27
>
> [5]:
https://git.linaro.org/people/leif.lindholm/linux.git/blob/refs/heads/uefi-for-upstream:/Documentation/arm/uefi.txt
>
> [6]: UEFI Specification 2.4 Revision A
> http://www.uefi.org/sites/default/files/resources/2_4_Errata_A.pdf

[-- Attachment #1.2: Type: text/html, Size: 9625 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

Hi Grant,

Thanks for comments,

On Wed, Feb 26, 2014 at 10:35:54PM +0000, Grant Likely wrote:
> 
> On 26 Feb 2014 18:35, "Christoffer Dall" <christoffer.dall@linaro.org>
> wrote:
> >
> > ARM VM System Specification
> > ===========================
> >
> > Goal
> > ----
> > The goal of this spec is to allow suitably-built OS images to run on
> > all ARM virtualization solutions, such as KVM or Xen.
> >
> > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > they aim to be hypervisor agnostic.
> >
> > Note that simply adhering to the SBSA [2] is not a valid approach,
> > for example because the SBSA mandates EL2, which will not be available
> > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > may be controversial for some ARM VM implementations to support.
> > This spec also covers the aarch32 execution mode, not covered in the
> > SBSA.
> >
> >
> > Image format
> > ------------
> > The image format, as presented to the VM, needs to be well-defined in
> > order for prepared disk images to be bootable across various
> > virtualization implementations.
> >
> > The raw disk format as presented to the VM must be partitioned with a
> > GUID Partition Table (GPT).  The bootable software must be placed in the
> > EFI System Partition (ESP), using the UEFI removable media path, and
> > must be an EFI application complying to the UEFI Specification 2.4
> > Revision A [6].
> >
> > The ESP partition's GPT entry's partition type GUID must be
> > C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> > formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> >
> > The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> > execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> > state.
> >
> > This ensures that tools for both Xen and KVM can load a binary UEFI
> > firmware which can read and boot the EFI application in the disk image.
> >
> > A typical scenario will be GRUB2 packaged as an EFI application, which
> > mounts the system boot partition and boots Linux.
> >
> >
> > Virtual Firmware
> > ----------------
> > The VM system must be able to boot the EFI application in the ESP.  It
> > is recommended that this is achieved by loading a UEFI binary as the
> > first software executed by the VM, which then executes the EFI
> > application.  The UEFI implementation should be compliant with UEFI
> > Specification 2.4 Revision A [6] or later.
> >
> > This document strongly recommends that the VM implementation supports
> > persistent environment storage for virtual firmware implementation in
> > order to ensure probable use cases such as adding additional disk images
> > to a VM or running installers to perform upgrades.
> >
> > The binary UEFI firmware implementation should not be distributed as
> > part of the VM image, but is specific to the VM implementation.
> >
> >
> > Hardware Description
> > --------------------
> > The Linux kernel's proper entry point always takes a pointer to an FDT,
> > regardless of the boot mechanism, firmware, and hardware description
> > method.  Even on real hardware which only supports ACPI and UEFI, the
> kernel
> > entry point will still receive a pointer to a simple FDT, generated by
> > the Linux kernel UEFI stub, containing a pointer to the UEFI system
> > table.  The kernel can then discover ACPI from the system tables.  The
> > presence of ACPI vs. FDT is therefore always itself discoverable,
> > through the FDT.
> 
> I would drop pretty much all of the above detail of the kernel entry point.
> The spec should specify UEFI compliance and stop there.

That probably make sense.  The discussion started out as "should it be
DTB or ACPI" and moved on from there, and then interestingly we sort of
found out, that it doesn't really matter in terms of how to package the
kernel, hence this text.  But looking over it now, it doesn't add to the
clarity of the spec.

> 
> What is relevant is the allowance for the UEFI implementation to provide an
> FDT and/or ACPI via the configuration table.
> 
> >
> > Therefore, the VM implementation must provide through its UEFI
> > implementation, either:
> >
> >   a complete FDT which describes the entire VM system and will boot
> >   mainline kernels driven by device tree alone, or
> >
> >   no FDT.  In this case, the VM implementation must provide ACPI, and
> >   the OS must be able to locate the ACPI root pointer through the UEFI
> >   system table.
> 
> It is actually valid for the VM to provide both ACPI and FDT. In that
> scenario it is up to the OS to chose which it will use.
> 

ok, "either" becomes "at least one of" with the appropriate adjustments.

> > For more information about the arm and arm64 boot conventions, see
> > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > Linux kernel source tree.
> >
> > For more information about UEFI and ACPI booting, see [4] and [5].
> >
> >
> > VM Platform
> > -----------
> > The specification does not mandate any specific memory map.  The guest
> > OS must be able to enumerate all processing elements, devices, and
> > memory through HW description data (FDT, ACPI) or a bus-specific
> > mechanism such as PCI.
> >
> > The virtual platform must support at least one of the following ARM
> > execution states:
> >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> >
> > It is recommended to support both (2) and (3) on aarch64 capable
> > physical systems.
> >
> > The virtual hardware platform must provide a number of mandatory
> > peripherals:
> >
> >   Serial console:  The platform should provide a console,
> >   based on an emulated pl011, a virtio-console, or a Xen PV console.
> 
> For portable disk image, can Xen PV be dropped from the list? pl011 is part
> of SBSA, and virtio is getting standardised, but Xen PV is implementation
> specific.
> 

It would certainly be easier if everyone just use virtio and to mandate
a pl011, but I don't want to preclude Xen from this spec, and the Xen
folks don't seem likely to implement virtio or pl011 based on this spec.

What's the problem with the current recommendation?  Are you concerned
that people will build kernels without virtio-console and Xen PV
console?

> >
> >   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
> >   limits the the number of virtual CPUs to 8 cores, newer GIC versions
> >   removes this limitation.
> >
> >   The ARM virtual timer and counter should be available to the VM as
> >   per the ARM Generic Timers specification in the ARM ARM [1].
> >
> >   A hotpluggable bus to support hotplug of at least block and network
> >   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
> >   bus.
> >
> >
> > We make the following recommendations for the guest OS kernel:
> >
> >   The guest OS must include support for GICv2 and any available newer
> >   version of the GIC architecture to maintain compatibility with older
> >   VM implementations.
> >
> >   It is strongly recommended to include support for all available
> >   (block, network, console, balloon) virtio-pci, virtio-mmio, and Xen PV
> >   drivers in the guest OS kernel or initial ramdisk.
> >
> >
> > Other common peripherals for block devices, networking, and more can
> > (and typically will) be provided, but OS software written and compiled
> > to run on ARM VMs cannot make any assumptions about which variations
> > of these should exist or which implementation they use (e.g. VirtIO or
> > Xen PV).  See "Hardware Description" above.
> >
> > Note that this platform specification is separate from the Linux kernel
> > concept of mach-virt, which merely specifies a machine model driven
> > purely from device tree, but does not mandate any peripherals or have any
> > mention of ACPI.
> >
> >
> > References
> > ----------
> > [1]: The ARM Architecture Reference Manual, ARMv8, Issue A.b
> >
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html
> >
> > [2]: ARM Server Base System Architecture
> >
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0029/index.html
> >
> > [3]: The ARM Generic Interrupt Controller Architecture Specifications v2.0
> >
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html
> >
> > [4]: http://www.secretlab.ca/archives/27
> >
> > [5]:
> https://git.linaro.org/people/leif.lindholm/linux.git/blob/refs/heads/uefi-for-upstream:/Documentation/arm/uefi.txt
> >
> > [6]: UEFI Specification 2.4 Revision A
> > http://www.uefi.org/sites/default/files/resources/2_4_Errata_A.pdf

[RFC] ARM VM System Sepcification

[Christoffer Dall]

Hi Grant,

Thanks for comments,

On Wed, Feb 26, 2014 at 10:35:54PM +0000, Grant Likely wrote:
> 
> On 26 Feb 2014 18:35, "Christoffer Dall" <christoffer.dall@linaro.org>
> wrote:
> >
> > ARM VM System Specification
> > ===========================
> >
> > Goal
> > ----
> > The goal of this spec is to allow suitably-built OS images to run on
> > all ARM virtualization solutions, such as KVM or Xen.
> >
> > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > they aim to be hypervisor agnostic.
> >
> > Note that simply adhering to the SBSA [2] is not a valid approach,
> > for example because the SBSA mandates EL2, which will not be available
> > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > may be controversial for some ARM VM implementations to support.
> > This spec also covers the aarch32 execution mode, not covered in the
> > SBSA.
> >
> >
> > Image format
> > ------------
> > The image format, as presented to the VM, needs to be well-defined in
> > order for prepared disk images to be bootable across various
> > virtualization implementations.
> >
> > The raw disk format as presented to the VM must be partitioned with a
> > GUID Partition Table (GPT).  The bootable software must be placed in the
> > EFI System Partition (ESP), using the UEFI removable media path, and
> > must be an EFI application complying to the UEFI Specification 2.4
> > Revision A [6].
> >
> > The ESP partition's GPT entry's partition type GUID must be
> > C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> > formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> >
> > The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> > execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> > state.
> >
> > This ensures that tools for both Xen and KVM can load a binary UEFI
> > firmware which can read and boot the EFI application in the disk image.
> >
> > A typical scenario will be GRUB2 packaged as an EFI application, which
> > mounts the system boot partition and boots Linux.
> >
> >
> > Virtual Firmware
> > ----------------
> > The VM system must be able to boot the EFI application in the ESP.  It
> > is recommended that this is achieved by loading a UEFI binary as the
> > first software executed by the VM, which then executes the EFI
> > application.  The UEFI implementation should be compliant with UEFI
> > Specification 2.4 Revision A [6] or later.
> >
> > This document strongly recommends that the VM implementation supports
> > persistent environment storage for virtual firmware implementation in
> > order to ensure probable use cases such as adding additional disk images
> > to a VM or running installers to perform upgrades.
> >
> > The binary UEFI firmware implementation should not be distributed as
> > part of the VM image, but is specific to the VM implementation.
> >
> >
> > Hardware Description
> > --------------------
> > The Linux kernel's proper entry point always takes a pointer to an FDT,
> > regardless of the boot mechanism, firmware, and hardware description
> > method.  Even on real hardware which only supports ACPI and UEFI, the
> kernel
> > entry point will still receive a pointer to a simple FDT, generated by
> > the Linux kernel UEFI stub, containing a pointer to the UEFI system
> > table.  The kernel can then discover ACPI from the system tables.  The
> > presence of ACPI vs. FDT is therefore always itself discoverable,
> > through the FDT.
> 
> I would drop pretty much all of the above detail of the kernel entry point.
> The spec should specify UEFI compliance and stop there.

That probably make sense.  The discussion started out as "should it be
DTB or ACPI" and moved on from there, and then interestingly we sort of
found out, that it doesn't really matter in terms of how to package the
kernel, hence this text.  But looking over it now, it doesn't add to the
clarity of the spec.

> 
> What is relevant is the allowance for the UEFI implementation to provide an
> FDT and/or ACPI via the configuration table.
> 
> >
> > Therefore, the VM implementation must provide through its UEFI
> > implementation, either:
> >
> >   a complete FDT which describes the entire VM system and will boot
> >   mainline kernels driven by device tree alone, or
> >
> >   no FDT.  In this case, the VM implementation must provide ACPI, and
> >   the OS must be able to locate the ACPI root pointer through the UEFI
> >   system table.
> 
> It is actually valid for the VM to provide both ACPI and FDT. In that
> scenario it is up to the OS to chose which it will use.
> 

ok, "either" becomes "at least one of" with the appropriate adjustments.

> > For more information about the arm and arm64 boot conventions, see
> > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > Linux kernel source tree.
> >
> > For more information about UEFI and ACPI booting, see [4] and [5].
> >
> >
> > VM Platform
> > -----------
> > The specification does not mandate any specific memory map.  The guest
> > OS must be able to enumerate all processing elements, devices, and
> > memory through HW description data (FDT, ACPI) or a bus-specific
> > mechanism such as PCI.
> >
> > The virtual platform must support at least one of the following ARM
> > execution states:
> >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> >
> > It is recommended to support both (2) and (3) on aarch64 capable
> > physical systems.
> >
> > The virtual hardware platform must provide a number of mandatory
> > peripherals:
> >
> >   Serial console:  The platform should provide a console,
> >   based on an emulated pl011, a virtio-console, or a Xen PV console.
> 
> For portable disk image, can Xen PV be dropped from the list? pl011 is part
> of SBSA, and virtio is getting standardised, but Xen PV is implementation
> specific.
> 

It would certainly be easier if everyone just use virtio and to mandate
a pl011, but I don't want to preclude Xen from this spec, and the Xen
folks don't seem likely to implement virtio or pl011 based on this spec.

What's the problem with the current recommendation?  Are you concerned
that people will build kernels without virtio-console and Xen PV
console?

> >
> >   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
> >   limits the the number of virtual CPUs to 8 cores, newer GIC versions
> >   removes this limitation.
> >
> >   The ARM virtual timer and counter should be available to the VM as
> >   per the ARM Generic Timers specification in the ARM ARM [1].
> >
> >   A hotpluggable bus to support hotplug of at least block and network
> >   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
> >   bus.
> >
> >
> > We make the following recommendations for the guest OS kernel:
> >
> >   The guest OS must include support for GICv2 and any available newer
> >   version of the GIC architecture to maintain compatibility with older
> >   VM implementations.
> >
> >   It is strongly recommended to include support for all available
> >   (block, network, console, balloon) virtio-pci, virtio-mmio, and Xen PV
> >   drivers in the guest OS kernel or initial ramdisk.
> >
> >
> > Other common peripherals for block devices, networking, and more can
> > (and typically will) be provided, but OS software written and compiled
> > to run on ARM VMs cannot make any assumptions about which variations
> > of these should exist or which implementation they use (e.g. VirtIO or
> > Xen PV).  See "Hardware Description" above.
> >
> > Note that this platform specification is separate from the Linux kernel
> > concept of mach-virt, which merely specifies a machine model driven
> > purely from device tree, but does not mandate any peripherals or have any
> > mention of ACPI.
> >
> >
> > References
> > ----------
> > [1]: The ARM Architecture Reference Manual, ARMv8, Issue A.b
> >
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html
> >
> > [2]: ARM Server Base System Architecture
> >
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0029/index.html
> >
> > [3]: The ARM Generic Interrupt Controller Architecture Specifications v2.0
> >
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html
> >
> > [4]: http://www.secretlab.ca/archives/27
> >
> > [5]:
> https://git.linaro.org/people/leif.lindholm/linux.git/blob/refs/heads/uefi-for-upstream:/Documentation/arm/uefi.txt
> >
> > [6]: UEFI Specification 2.4 Revision A
> > http://www.uefi.org/sites/default/files/resources/2_4_Errata_A.pdf

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

Hi Grant,

Thanks for comments,

On Wed, Feb 26, 2014 at 10:35:54PM +0000, Grant Likely wrote:
> 
> On 26 Feb 2014 18:35, "Christoffer Dall" <christoffer.dall@linaro.org>
> wrote:
> >
> > ARM VM System Specification
> > ===========================
> >
> > Goal
> > ----
> > The goal of this spec is to allow suitably-built OS images to run on
> > all ARM virtualization solutions, such as KVM or Xen.
> >
> > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > they aim to be hypervisor agnostic.
> >
> > Note that simply adhering to the SBSA [2] is not a valid approach,
> > for example because the SBSA mandates EL2, which will not be available
> > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > may be controversial for some ARM VM implementations to support.
> > This spec also covers the aarch32 execution mode, not covered in the
> > SBSA.
> >
> >
> > Image format
> > ------------
> > The image format, as presented to the VM, needs to be well-defined in
> > order for prepared disk images to be bootable across various
> > virtualization implementations.
> >
> > The raw disk format as presented to the VM must be partitioned with a
> > GUID Partition Table (GPT).  The bootable software must be placed in the
> > EFI System Partition (ESP), using the UEFI removable media path, and
> > must be an EFI application complying to the UEFI Specification 2.4
> > Revision A [6].
> >
> > The ESP partition's GPT entry's partition type GUID must be
> > C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> > formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> >
> > The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> > execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> > state.
> >
> > This ensures that tools for both Xen and KVM can load a binary UEFI
> > firmware which can read and boot the EFI application in the disk image.
> >
> > A typical scenario will be GRUB2 packaged as an EFI application, which
> > mounts the system boot partition and boots Linux.
> >
> >
> > Virtual Firmware
> > ----------------
> > The VM system must be able to boot the EFI application in the ESP.  It
> > is recommended that this is achieved by loading a UEFI binary as the
> > first software executed by the VM, which then executes the EFI
> > application.  The UEFI implementation should be compliant with UEFI
> > Specification 2.4 Revision A [6] or later.
> >
> > This document strongly recommends that the VM implementation supports
> > persistent environment storage for virtual firmware implementation in
> > order to ensure probable use cases such as adding additional disk images
> > to a VM or running installers to perform upgrades.
> >
> > The binary UEFI firmware implementation should not be distributed as
> > part of the VM image, but is specific to the VM implementation.
> >
> >
> > Hardware Description
> > --------------------
> > The Linux kernel's proper entry point always takes a pointer to an FDT,
> > regardless of the boot mechanism, firmware, and hardware description
> > method.  Even on real hardware which only supports ACPI and UEFI, the
> kernel
> > entry point will still receive a pointer to a simple FDT, generated by
> > the Linux kernel UEFI stub, containing a pointer to the UEFI system
> > table.  The kernel can then discover ACPI from the system tables.  The
> > presence of ACPI vs. FDT is therefore always itself discoverable,
> > through the FDT.
> 
> I would drop pretty much all of the above detail of the kernel entry point.
> The spec should specify UEFI compliance and stop there.

That probably make sense.  The discussion started out as "should it be
DTB or ACPI" and moved on from there, and then interestingly we sort of
found out, that it doesn't really matter in terms of how to package the
kernel, hence this text.  But looking over it now, it doesn't add to the
clarity of the spec.

> 
> What is relevant is the allowance for the UEFI implementation to provide an
> FDT and/or ACPI via the configuration table.
> 
> >
> > Therefore, the VM implementation must provide through its UEFI
> > implementation, either:
> >
> >   a complete FDT which describes the entire VM system and will boot
> >   mainline kernels driven by device tree alone, or
> >
> >   no FDT.  In this case, the VM implementation must provide ACPI, and
> >   the OS must be able to locate the ACPI root pointer through the UEFI
> >   system table.
> 
> It is actually valid for the VM to provide both ACPI and FDT. In that
> scenario it is up to the OS to chose which it will use.
> 

ok, "either" becomes "at least one of" with the appropriate adjustments.

> > For more information about the arm and arm64 boot conventions, see
> > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > Linux kernel source tree.
> >
> > For more information about UEFI and ACPI booting, see [4] and [5].
> >
> >
> > VM Platform
> > -----------
> > The specification does not mandate any specific memory map.  The guest
> > OS must be able to enumerate all processing elements, devices, and
> > memory through HW description data (FDT, ACPI) or a bus-specific
> > mechanism such as PCI.
> >
> > The virtual platform must support at least one of the following ARM
> > execution states:
> >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> >
> > It is recommended to support both (2) and (3) on aarch64 capable
> > physical systems.
> >
> > The virtual hardware platform must provide a number of mandatory
> > peripherals:
> >
> >   Serial console:  The platform should provide a console,
> >   based on an emulated pl011, a virtio-console, or a Xen PV console.
> 
> For portable disk image, can Xen PV be dropped from the list? pl011 is part
> of SBSA, and virtio is getting standardised, but Xen PV is implementation
> specific.
> 

It would certainly be easier if everyone just use virtio and to mandate
a pl011, but I don't want to preclude Xen from this spec, and the Xen
folks don't seem likely to implement virtio or pl011 based on this spec.

What's the problem with the current recommendation?  Are you concerned
that people will build kernels without virtio-console and Xen PV
console?

> >
> >   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
> >   limits the the number of virtual CPUs to 8 cores, newer GIC versions
> >   removes this limitation.
> >
> >   The ARM virtual timer and counter should be available to the VM as
> >   per the ARM Generic Timers specification in the ARM ARM [1].
> >
> >   A hotpluggable bus to support hotplug of at least block and network
> >   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
> >   bus.
> >
> >
> > We make the following recommendations for the guest OS kernel:
> >
> >   The guest OS must include support for GICv2 and any available newer
> >   version of the GIC architecture to maintain compatibility with older
> >   VM implementations.
> >
> >   It is strongly recommended to include support for all available
> >   (block, network, console, balloon) virtio-pci, virtio-mmio, and Xen PV
> >   drivers in the guest OS kernel or initial ramdisk.
> >
> >
> > Other common peripherals for block devices, networking, and more can
> > (and typically will) be provided, but OS software written and compiled
> > to run on ARM VMs cannot make any assumptions about which variations
> > of these should exist or which implementation they use (e.g. VirtIO or
> > Xen PV).  See "Hardware Description" above.
> >
> > Note that this platform specification is separate from the Linux kernel
> > concept of mach-virt, which merely specifies a machine model driven
> > purely from device tree, but does not mandate any peripherals or have any
> > mention of ACPI.
> >
> >
> > References
> > ----------
> > [1]: The ARM Architecture Reference Manual, ARMv8, Issue A.b
> >
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html
> >
> > [2]: ARM Server Base System Architecture
> >
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0029/index.html
> >
> > [3]: The ARM Generic Interrupt Controller Architecture Specifications v2.0
> >
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html
> >
> > [4]: http://www.secretlab.ca/archives/27
> >
> > [5]:
> https://git.linaro.org/people/leif.lindholm/linux.git/blob/refs/heads/uefi-for-upstream:/Documentation/arm/uefi.txt
> >
> > [6]: UEFI Specification 2.4 Revision A
> > http://www.uefi.org/sites/default/files/resources/2_4_Errata_A.pdf

Re: [RFC] ARM VM System Sepcification

[Rob Herring]

On Wed, Feb 26, 2014 at 1:55 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
>> ARM VM System Specification
>> ===========================
>>
>> Goal
>> ----
>> The goal of this spec is to allow suitably-built OS images to run on
>> all ARM virtualization solutions, such as KVM or Xen.
>>
>> Recommendations in this spec are valid for aarch32 and aarch64 alike, and
>> they aim to be hypervisor agnostic.
>>
>> Note that simply adhering to the SBSA [2] is not a valid approach,
>> for example because the SBSA mandates EL2, which will not be available
>> for VMs.  Further, the SBSA mandates peripherals like the pl011, which
>> may be controversial for some ARM VM implementations to support.
>> This spec also covers the aarch32 execution mode, not covered in the
>> SBSA.
>
> I would prefer if we can stay as close as possible to SBSA for individual
> hardware components, and only stray from it when there is a strong reason.
> pl011-subset doesn't sound like a significant problem to implement,
> especially as SBSA makes the DMA part of that optional. Can you
> elaborate on what hypervisor would have a problem with that?

The SBSA only spec's a very minimal pl011 subset which is only
suitable for early serial output. Not only is there no DMA, but there
are no interrupts and maybe no input. I think it also assumes the uart
is enabled and configured already by firmware. It is all somewhat
pointless because the location is still not known or discoverable by
early code. Just mandating a real pl011 would have been better, but I
guess uart IP is value add for some. There is a downside to the pl011
which is the tty name is different from x86 uarts which gets exposed
to users and things like libvirt.

I think the VM image just has to support pl011, virtio-console, and
xen console. Arguably, an 8250 should also be included in interest of
making things just work.

Rob

[RFC] ARM VM System Sepcification

[Rob Herring]

On Wed, Feb 26, 2014 at 1:55 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
>> ARM VM System Specification
>> ===========================
>>
>> Goal
>> ----
>> The goal of this spec is to allow suitably-built OS images to run on
>> all ARM virtualization solutions, such as KVM or Xen.
>>
>> Recommendations in this spec are valid for aarch32 and aarch64 alike, and
>> they aim to be hypervisor agnostic.
>>
>> Note that simply adhering to the SBSA [2] is not a valid approach,
>> for example because the SBSA mandates EL2, which will not be available
>> for VMs.  Further, the SBSA mandates peripherals like the pl011, which
>> may be controversial for some ARM VM implementations to support.
>> This spec also covers the aarch32 execution mode, not covered in the
>> SBSA.
>
> I would prefer if we can stay as close as possible to SBSA for individual
> hardware components, and only stray from it when there is a strong reason.
> pl011-subset doesn't sound like a significant problem to implement,
> especially as SBSA makes the DMA part of that optional. Can you
> elaborate on what hypervisor would have a problem with that?

The SBSA only spec's a very minimal pl011 subset which is only
suitable for early serial output. Not only is there no DMA, but there
are no interrupts and maybe no input. I think it also assumes the uart
is enabled and configured already by firmware. It is all somewhat
pointless because the location is still not known or discoverable by
early code. Just mandating a real pl011 would have been better, but I
guess uart IP is value add for some. There is a downside to the pl011
which is the tty name is different from x86 uarts which gets exposed
to users and things like libvirt.

I think the VM image just has to support pl011, virtio-console, and
xen console. Arguably, an 8250 should also be included in interest of
making things just work.

Rob

Re: [RFC] ARM VM System Sepcification

[Rob Herring]

On Wed, Feb 26, 2014 at 1:55 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
>> ARM VM System Specification
>> ===========================
>>
>> Goal
>> ----
>> The goal of this spec is to allow suitably-built OS images to run on
>> all ARM virtualization solutions, such as KVM or Xen.
>>
>> Recommendations in this spec are valid for aarch32 and aarch64 alike, and
>> they aim to be hypervisor agnostic.
>>
>> Note that simply adhering to the SBSA [2] is not a valid approach,
>> for example because the SBSA mandates EL2, which will not be available
>> for VMs.  Further, the SBSA mandates peripherals like the pl011, which
>> may be controversial for some ARM VM implementations to support.
>> This spec also covers the aarch32 execution mode, not covered in the
>> SBSA.
>
> I would prefer if we can stay as close as possible to SBSA for individual
> hardware components, and only stray from it when there is a strong reason.
> pl011-subset doesn't sound like a significant problem to implement,
> especially as SBSA makes the DMA part of that optional. Can you
> elaborate on what hypervisor would have a problem with that?

The SBSA only spec's a very minimal pl011 subset which is only
suitable for early serial output. Not only is there no DMA, but there
are no interrupts and maybe no input. I think it also assumes the uart
is enabled and configured already by firmware. It is all somewhat
pointless because the location is still not known or discoverable by
early code. Just mandating a real pl011 would have been better, but I
guess uart IP is value add for some. There is a downside to the pl011
which is the tty name is different from x86 uarts which gets exposed
to users and things like libvirt.

I think the VM image just has to support pl011, virtio-console, and
xen console. Arguably, an 8250 should also be included in interest of
making things just work.

Rob

Re: [RFC] ARM VM System Sepcification

[Peter Maydell]

On 26 February 2014 22:49, Rob Herring <robherring2@gmail.com> wrote
> The SBSA only spec's a very minimal pl011 subset which is only
> suitable for early serial output. Not only is there no DMA, but there
> are no interrupts and maybe no input.

No interrupts on a UART in a VM (especially an emulated one)
is a good way to spend all your time bouncing around in I/O
emulation of the "hey can we send another byte yet?" register...

> I think it also assumes the uart
> is enabled and configured already by firmware. It is all somewhat
> pointless because the location is still not known or discoverable by
> early code.

This sounds like we should specify and implement something
so we can provide this information in the device tree. Telling
the kernel where the hardware is is exactly what DT is for, right?

> I think the VM image just has to support pl011, virtio-console, and
> xen console. Arguably, an 8250 should also be included in interest of
> making things just work.

What does the 8250 have to recommend it over just providing
the PL011?

thanks
-- PMM

[RFC] ARM VM System Sepcification

[Peter Maydell]

On 26 February 2014 22:49, Rob Herring <robherring2@gmail.com> wrote
> The SBSA only spec's a very minimal pl011 subset which is only
> suitable for early serial output. Not only is there no DMA, but there
> are no interrupts and maybe no input.

No interrupts on a UART in a VM (especially an emulated one)
is a good way to spend all your time bouncing around in I/O
emulation of the "hey can we send another byte yet?" register...

> I think it also assumes the uart
> is enabled and configured already by firmware. It is all somewhat
> pointless because the location is still not known or discoverable by
> early code.

This sounds like we should specify and implement something
so we can provide this information in the device tree. Telling
the kernel where the hardware is is exactly what DT is for, right?

> I think the VM image just has to support pl011, virtio-console, and
> xen console. Arguably, an 8250 should also be included in interest of
> making things just work.

What does the 8250 have to recommend it over just providing
the PL011?

thanks
-- PMM

Re: [RFC] ARM VM System Sepcification

[Peter Maydell]

On 26 February 2014 22:49, Rob Herring <robherring2@gmail.com> wrote
> The SBSA only spec's a very minimal pl011 subset which is only
> suitable for early serial output. Not only is there no DMA, but there
> are no interrupts and maybe no input.

No interrupts on a UART in a VM (especially an emulated one)
is a good way to spend all your time bouncing around in I/O
emulation of the "hey can we send another byte yet?" register...

> I think it also assumes the uart
> is enabled and configured already by firmware. It is all somewhat
> pointless because the location is still not known or discoverable by
> early code.

This sounds like we should specify and implement something
so we can provide this information in the device tree. Telling
the kernel where the hardware is is exactly what DT is for, right?

> I think the VM image just has to support pl011, virtio-console, and
> xen console. Arguably, an 8250 should also be included in interest of
> making things just work.

What does the 8250 have to recommend it over just providing
the PL011?

thanks
-- PMM

Re: [RFC] ARM VM System Sepcification

[Rob Herring]

On Wed, Feb 26, 2014 at 4:54 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
> On 26 February 2014 22:49, Rob Herring <robherring2@gmail.com> wrote
>> The SBSA only spec's a very minimal pl011 subset which is only
>> suitable for early serial output. Not only is there no DMA, but there
>> are no interrupts and maybe no input.
>
> No interrupts on a UART in a VM (especially an emulated one)
> is a good way to spend all your time bouncing around in I/O
> emulation of the "hey can we send another byte yet?" register...
>
>> I think it also assumes the uart
>> is enabled and configured already by firmware. It is all somewhat
>> pointless because the location is still not known or discoverable by
>> early code.
>
> This sounds like we should specify and implement something
> so we can provide this information in the device tree. Telling
> the kernel where the hardware is is exactly what DT is for, right?

Yes, I'm looking into that, but that's not really a concern for this
doc as early output is a debug feature.

>> I think the VM image just has to support pl011, virtio-console, and
>> xen console. Arguably, an 8250 should also be included in interest of
>> making things just work.
>
> What does the 8250 have to recommend it over just providing
> the PL011?

As I mentioned, it will just work for anything that expects the serial
port to be ttyS0 as on x86 rather than ttyAMA0. Really, I'd like to
see ttyAMA go away, but evidently that's not an easily fixed issue and
it is an ABI.

Rob

[RFC] ARM VM System Sepcification

[Rob Herring]

On Wed, Feb 26, 2014 at 4:54 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
> On 26 February 2014 22:49, Rob Herring <robherring2@gmail.com> wrote
>> The SBSA only spec's a very minimal pl011 subset which is only
>> suitable for early serial output. Not only is there no DMA, but there
>> are no interrupts and maybe no input.
>
> No interrupts on a UART in a VM (especially an emulated one)
> is a good way to spend all your time bouncing around in I/O
> emulation of the "hey can we send another byte yet?" register...
>
>> I think it also assumes the uart
>> is enabled and configured already by firmware. It is all somewhat
>> pointless because the location is still not known or discoverable by
>> early code.
>
> This sounds like we should specify and implement something
> so we can provide this information in the device tree. Telling
> the kernel where the hardware is is exactly what DT is for, right?

Yes, I'm looking into that, but that's not really a concern for this
doc as early output is a debug feature.

>> I think the VM image just has to support pl011, virtio-console, and
>> xen console. Arguably, an 8250 should also be included in interest of
>> making things just work.
>
> What does the 8250 have to recommend it over just providing
> the PL011?

As I mentioned, it will just work for anything that expects the serial
port to be ttyS0 as on x86 rather than ttyAMA0. Really, I'd like to
see ttyAMA go away, but evidently that's not an easily fixed issue and
it is an ABI.

Rob

Re: [RFC] ARM VM System Sepcification

[Rob Herring]

On Wed, Feb 26, 2014 at 4:54 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
> On 26 February 2014 22:49, Rob Herring <robherring2@gmail.com> wrote
>> The SBSA only spec's a very minimal pl011 subset which is only
>> suitable for early serial output. Not only is there no DMA, but there
>> are no interrupts and maybe no input.
>
> No interrupts on a UART in a VM (especially an emulated one)
> is a good way to spend all your time bouncing around in I/O
> emulation of the "hey can we send another byte yet?" register...
>
>> I think it also assumes the uart
>> is enabled and configured already by firmware. It is all somewhat
>> pointless because the location is still not known or discoverable by
>> early code.
>
> This sounds like we should specify and implement something
> so we can provide this information in the device tree. Telling
> the kernel where the hardware is is exactly what DT is for, right?

Yes, I'm looking into that, but that's not really a concern for this
doc as early output is a debug feature.

>> I think the VM image just has to support pl011, virtio-console, and
>> xen console. Arguably, an 8250 should also be included in interest of
>> making things just work.
>
> What does the 8250 have to recommend it over just providing
> the PL011?

As I mentioned, it will just work for anything that expects the serial
port to be ttyS0 as on x86 rather than ttyAMA0. Really, I'd like to
see ttyAMA go away, but evidently that's not an easily fixed issue and
it is an ABI.

Rob

Re: [RFC] ARM VM System Sepcification

[Christopher Covington]

On 02/26/2014 05:49 PM, Rob Herring wrote:
> On Wed, Feb 26, 2014 at 1:55 PM, Arnd Bergmann <arnd@arndb.de> wrote:
>> On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
>>> ARM VM System Specification
>>> ===========================
>>>
>>> Goal
>>> ----
>>> The goal of this spec is to allow suitably-built OS images to run on
>>> all ARM virtualization solutions, such as KVM or Xen.
>>>
>>> Recommendations in this spec are valid for aarch32 and aarch64 alike, and
>>> they aim to be hypervisor agnostic.
>>>
>>> Note that simply adhering to the SBSA [2] is not a valid approach,
>>> for example because the SBSA mandates EL2, which will not be available
>>> for VMs.  Further, the SBSA mandates peripherals like the pl011, which
>>> may be controversial for some ARM VM implementations to support.
>>> This spec also covers the aarch32 execution mode, not covered in the
>>> SBSA.
>>
>> I would prefer if we can stay as close as possible to SBSA for individual
>> hardware components, and only stray from it when there is a strong reason.
>> pl011-subset doesn't sound like a significant problem to implement,
>> especially as SBSA makes the DMA part of that optional. Can you
>> elaborate on what hypervisor would have a problem with that?
> 
> The SBSA only spec's a very minimal pl011 subset which is only
> suitable for early serial output. Not only is there no DMA, but there
> are no interrupts and maybe no input. I think it also assumes the uart
> is enabled and configured already by firmware. It is all somewhat
> pointless because the location is still not known or discoverable by
> early code. Just mandating a real pl011 would have been better, but I
> guess uart IP is value add for some. There is a downside to the pl011
> which is the tty name is different from x86 uarts which gets exposed
> to users and things like libvirt.

Can you just use /dev/console? That's what I use in my init scripts for
portability from PL011 to DCC.

Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

[RFC] ARM VM System Sepcification

[Christopher Covington]

On 02/26/2014 05:49 PM, Rob Herring wrote:
> On Wed, Feb 26, 2014 at 1:55 PM, Arnd Bergmann <arnd@arndb.de> wrote:
>> On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
>>> ARM VM System Specification
>>> ===========================
>>>
>>> Goal
>>> ----
>>> The goal of this spec is to allow suitably-built OS images to run on
>>> all ARM virtualization solutions, such as KVM or Xen.
>>>
>>> Recommendations in this spec are valid for aarch32 and aarch64 alike, and
>>> they aim to be hypervisor agnostic.
>>>
>>> Note that simply adhering to the SBSA [2] is not a valid approach,
>>> for example because the SBSA mandates EL2, which will not be available
>>> for VMs.  Further, the SBSA mandates peripherals like the pl011, which
>>> may be controversial for some ARM VM implementations to support.
>>> This spec also covers the aarch32 execution mode, not covered in the
>>> SBSA.
>>
>> I would prefer if we can stay as close as possible to SBSA for individual
>> hardware components, and only stray from it when there is a strong reason.
>> pl011-subset doesn't sound like a significant problem to implement,
>> especially as SBSA makes the DMA part of that optional. Can you
>> elaborate on what hypervisor would have a problem with that?
> 
> The SBSA only spec's a very minimal pl011 subset which is only
> suitable for early serial output. Not only is there no DMA, but there
> are no interrupts and maybe no input. I think it also assumes the uart
> is enabled and configured already by firmware. It is all somewhat
> pointless because the location is still not known or discoverable by
> early code. Just mandating a real pl011 would have been better, but I
> guess uart IP is value add for some. There is a downside to the pl011
> which is the tty name is different from x86 uarts which gets exposed
> to users and things like libvirt.

Can you just use /dev/console? That's what I use in my init scripts for
portability from PL011 to DCC.

Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

Re: [RFC] ARM VM System Sepcification

[Christopher Covington]

On 02/26/2014 05:49 PM, Rob Herring wrote:
> On Wed, Feb 26, 2014 at 1:55 PM, Arnd Bergmann <arnd@arndb.de> wrote:
>> On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
>>> ARM VM System Specification
>>> ===========================
>>>
>>> Goal
>>> ----
>>> The goal of this spec is to allow suitably-built OS images to run on
>>> all ARM virtualization solutions, such as KVM or Xen.
>>>
>>> Recommendations in this spec are valid for aarch32 and aarch64 alike, and
>>> they aim to be hypervisor agnostic.
>>>
>>> Note that simply adhering to the SBSA [2] is not a valid approach,
>>> for example because the SBSA mandates EL2, which will not be available
>>> for VMs.  Further, the SBSA mandates peripherals like the pl011, which
>>> may be controversial for some ARM VM implementations to support.
>>> This spec also covers the aarch32 execution mode, not covered in the
>>> SBSA.
>>
>> I would prefer if we can stay as close as possible to SBSA for individual
>> hardware components, and only stray from it when there is a strong reason.
>> pl011-subset doesn't sound like a significant problem to implement,
>> especially as SBSA makes the DMA part of that optional. Can you
>> elaborate on what hypervisor would have a problem with that?
> 
> The SBSA only spec's a very minimal pl011 subset which is only
> suitable for early serial output. Not only is there no DMA, but there
> are no interrupts and maybe no input. I think it also assumes the uart
> is enabled and configured already by firmware. It is all somewhat
> pointless because the location is still not known or discoverable by
> early code. Just mandating a real pl011 would have been better, but I
> guess uart IP is value add for some. There is a downside to the pl011
> which is the tty name is different from x86 uarts which gets exposed
> to users and things like libvirt.

Can you just use /dev/console? That's what I use in my init scripts for
portability from PL011 to DCC.

Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

Re: [RFC] ARM VM System Sepcification

[Peter Maydell]

On 26 February 2014 23:08, Rob Herring <robherring2@gmail.com> wrote:
> On Wed, Feb 26, 2014 at 4:54 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
>> What does the 8250 have to recommend it over just providing
>> the PL011?
>
> As I mentioned, it will just work for anything that expects the serial
> port to be ttyS0 as on x86 rather than ttyAMA0.

This doesn't seem very compelling to me. Either userspace
should just be fixed or the kernel should implement a namespace
for serial ports which doesn't randomly change just because the
particular h/w driver doing the implementation is different.
We shouldn't be papering over other peoples' problems in the VM
spec IMHO.

thanks
-- PMM

[RFC] ARM VM System Sepcification

[Peter Maydell]

On 26 February 2014 23:08, Rob Herring <robherring2@gmail.com> wrote:
> On Wed, Feb 26, 2014 at 4:54 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
>> What does the 8250 have to recommend it over just providing
>> the PL011?
>
> As I mentioned, it will just work for anything that expects the serial
> port to be ttyS0 as on x86 rather than ttyAMA0.

This doesn't seem very compelling to me. Either userspace
should just be fixed or the kernel should implement a namespace
for serial ports which doesn't randomly change just because the
particular h/w driver doing the implementation is different.
We shouldn't be papering over other peoples' problems in the VM
spec IMHO.

thanks
-- PMM

Re: [RFC] ARM VM System Sepcification

[Peter Maydell]

On 26 February 2014 23:08, Rob Herring <robherring2@gmail.com> wrote:
> On Wed, Feb 26, 2014 at 4:54 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
>> What does the 8250 have to recommend it over just providing
>> the PL011?
>
> As I mentioned, it will just work for anything that expects the serial
> port to be ttyS0 as on x86 rather than ttyAMA0.

This doesn't seem very compelling to me. Either userspace
should just be fixed or the kernel should implement a namespace
for serial ports which doesn't randomly change just because the
particular h/w driver doing the implementation is different.
We shouldn't be papering over other peoples' problems in the VM
spec IMHO.

thanks
-- PMM

Re: [RFC] ARM VM System Sepcification

[Blibbet]

On 2/26/14 10:34 AM, Christoffer Dall wrote:
> ARM VM System Specification
> ===========================

See also the thread forked off to the EFI dev list, about using existing 
EFI ByteCode (EBC) for this new purpose, especially the informative 
reply from Andrew Fish of Apple.com:

http://sourceforge.net/p/edk2/mailman/message/32031943/

Today, EBC is created by Intel, and targets Intel's 3 platforms, but no 
ARM platforms yet. Today existing EFI uses EBC "VM", existing 
implementation exists with BSD license. EBC's goal was to let IHVs share 
Option ROM style drivers, not have to ship multiple ones. Having ARM and 
Intel use the same VM/bytecode would be even better for IHVs. I'm 
unclear to your non-EFI use cases, so may not be useful outside EFI. IMO 
the main issue with EBC is only commercial Intel and Microsoft compilers 
support it, not GCC or CLang. IP clarify of this Intel creation would 
also be an issue, but apparently UEFI Forum owns the spec.

There are too many lists CC'ed already, but if this becomes a valid 
option, the linux-efi list on kernel.org needs to get invited. :-)

Thanks,
Lee

[RFC] ARM VM System Sepcification

[Blibbet]

On 2/26/14 10:34 AM, Christoffer Dall wrote:
> ARM VM System Specification
> ===========================

See also the thread forked off to the EFI dev list, about using existing 
EFI ByteCode (EBC) for this new purpose, especially the informative 
reply from Andrew Fish of Apple.com:

http://sourceforge.net/p/edk2/mailman/message/32031943/

Today, EBC is created by Intel, and targets Intel's 3 platforms, but no 
ARM platforms yet. Today existing EFI uses EBC "VM", existing 
implementation exists with BSD license. EBC's goal was to let IHVs share 
Option ROM style drivers, not have to ship multiple ones. Having ARM and 
Intel use the same VM/bytecode would be even better for IHVs. I'm 
unclear to your non-EFI use cases, so may not be useful outside EFI. IMO 
the main issue with EBC is only commercial Intel and Microsoft compilers 
support it, not GCC or CLang. IP clarify of this Intel creation would 
also be an issue, but apparently UEFI Forum owns the spec.

There are too many lists CC'ed already, but if this becomes a valid 
option, the linux-efi list on kernel.org needs to get invited. :-)

Thanks,
Lee

Re: [RFC] ARM VM System Sepcification

[Blibbet]

On 2/26/14 10:34 AM, Christoffer Dall wrote:
> ARM VM System Specification
> ===========================

See also the thread forked off to the EFI dev list, about using existing 
EFI ByteCode (EBC) for this new purpose, especially the informative 
reply from Andrew Fish of Apple.com:

http://sourceforge.net/p/edk2/mailman/message/32031943/

Today, EBC is created by Intel, and targets Intel's 3 platforms, but no 
ARM platforms yet. Today existing EFI uses EBC "VM", existing 
implementation exists with BSD license. EBC's goal was to let IHVs share 
Option ROM style drivers, not have to ship multiple ones. Having ARM and 
Intel use the same VM/bytecode would be even better for IHVs. I'm 
unclear to your non-EFI use cases, so may not be useful outside EFI. IMO 
the main issue with EBC is only commercial Intel and Microsoft compilers 
support it, not GCC or CLang. IP clarify of this Intel creation would 
also be an issue, but apparently UEFI Forum owns the spec.

There are too many lists CC'ed already, but if this becomes a valid 
option, the linux-efi list on kernel.org needs to get invited. :-)

Thanks,
Lee

Re: [RFC] ARM VM System Sepcification

[Nicolas Pitre]

On Wed, 26 Feb 2014, Peter Maydell wrote:

> On 26 February 2014 23:08, Rob Herring <robherring2@gmail.com> wrote:
> > On Wed, Feb 26, 2014 at 4:54 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
> >> What does the 8250 have to recommend it over just providing
> >> the PL011?
> >
> > As I mentioned, it will just work for anything that expects the serial
> > port to be ttyS0 as on x86 rather than ttyAMA0.
> 
> This doesn't seem very compelling to me. Either userspace
> should just be fixed or the kernel should implement a namespace
> for serial ports which doesn't randomly change just because the
> particular h/w driver doing the implementation is different.
> We shouldn't be papering over other peoples' problems in the VM
> spec IMHO.

Indeed.

This is already causing us trouble in other contexts.  Please have a 
look at this patch for a solution:

http://article.gmane.org/gmane.linux.kernel.samsung-soc/27222


Nicolas

[RFC] ARM VM System Sepcification

[Nicolas Pitre]

On Wed, 26 Feb 2014, Peter Maydell wrote:

> On 26 February 2014 23:08, Rob Herring <robherring2@gmail.com> wrote:
> > On Wed, Feb 26, 2014 at 4:54 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
> >> What does the 8250 have to recommend it over just providing
> >> the PL011?
> >
> > As I mentioned, it will just work for anything that expects the serial
> > port to be ttyS0 as on x86 rather than ttyAMA0.
> 
> This doesn't seem very compelling to me. Either userspace
> should just be fixed or the kernel should implement a namespace
> for serial ports which doesn't randomly change just because the
> particular h/w driver doing the implementation is different.
> We shouldn't be papering over other peoples' problems in the VM
> spec IMHO.

Indeed.

This is already causing us trouble in other contexts.  Please have a 
look at this patch for a solution:

http://article.gmane.org/gmane.linux.kernel.samsung-soc/27222


Nicolas

Re: [RFC] ARM VM System Sepcification

[Nicolas Pitre]

On Wed, 26 Feb 2014, Peter Maydell wrote:

> On 26 February 2014 23:08, Rob Herring <robherring2@gmail.com> wrote:
> > On Wed, Feb 26, 2014 at 4:54 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
> >> What does the 8250 have to recommend it over just providing
> >> the PL011?
> >
> > As I mentioned, it will just work for anything that expects the serial
> > port to be ttyS0 as on x86 rather than ttyAMA0.
> 
> This doesn't seem very compelling to me. Either userspace
> should just be fixed or the kernel should implement a namespace
> for serial ports which doesn't randomly change just because the
> particular h/w driver doing the implementation is different.
> We shouldn't be papering over other peoples' problems in the VM
> spec IMHO.

Indeed.

This is already causing us trouble in other contexts.  Please have a 
look at this patch for a solution:

http://article.gmane.org/gmane.linux.kernel.samsung-soc/27222


Nicolas

Re: [RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Wednesday 26 February 2014, Christoffer Dall wrote:
> Personally I'm all for simplicity so I don't want to push any agenda for
> ACPI in VMs.
> 
> Note that the spec does not mandate the use of ACPI, it just tells you
> how to do it if you wish to.
> 
> But, we can change the spec to require full FDT description of the
> system, unless of course some of the ACPI-in-VM supporters manage to
> convince the rest.

I guess the real question is whether we are interested in running Windows RT
in VM guests. I don't personally expect MS to come out with a port for
this spec, no matter what we do, but some of you may have information I don't.

	Arnd

[RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Wednesday 26 February 2014, Christoffer Dall wrote:
> Personally I'm all for simplicity so I don't want to push any agenda for
> ACPI in VMs.
> 
> Note that the spec does not mandate the use of ACPI, it just tells you
> how to do it if you wish to.
> 
> But, we can change the spec to require full FDT description of the
> system, unless of course some of the ACPI-in-VM supporters manage to
> convince the rest.

I guess the real question is whether we are interested in running Windows RT
in VM guests. I don't personally expect MS to come out with a port for
this spec, no matter what we do, but some of you may have information I don't.

	Arnd

Re: [RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Wednesday 26 February 2014, Christoffer Dall wrote:
> Personally I'm all for simplicity so I don't want to push any agenda for
> ACPI in VMs.
> 
> Note that the spec does not mandate the use of ACPI, it just tells you
> how to do it if you wish to.
> 
> But, we can change the spec to require full FDT description of the
> system, unless of course some of the ACPI-in-VM supporters manage to
> convince the rest.

I guess the real question is whether we are interested in running Windows RT
in VM guests. I don't personally expect MS to come out with a port for
this spec, no matter what we do, but some of you may have information I don't.

	Arnd

Re: [RFC] ARM VM System Sepcification

[Catalin Marinas]

On 26 February 2014 20:05, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
>> On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
>> > For more information about UEFI and ACPI booting, see [4] and [5].
>>
>> What's the point of having ACPI in a virtual machine? You wouldn't
>> need to abstract any of the hardware in AML since you already know
>> what the virtual hardware is, so I can't see how this would help
>> anyone.
>
> The most common response I've been getting so far is that people
> generally want their VMs to look close to the real thing, but not sure
> how valid an argument that is.
>
> Some people feel strongly about this and seem to think that ARMv8
> kernels will only work with ACPI in the future...

My strong feeling is that AArch64 kernels *may* support ACPI in the future ;).

On a more serious note, both FDT and ACPI will be first-class citizens
on AArch64 and I have no intention whatsoever of dropping FDT.

-- 
Catalin

[RFC] ARM VM System Sepcification

[Catalin Marinas]

On 26 February 2014 20:05, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
>> On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
>> > For more information about UEFI and ACPI booting, see [4] and [5].
>>
>> What's the point of having ACPI in a virtual machine? You wouldn't
>> need to abstract any of the hardware in AML since you already know
>> what the virtual hardware is, so I can't see how this would help
>> anyone.
>
> The most common response I've been getting so far is that people
> generally want their VMs to look close to the real thing, but not sure
> how valid an argument that is.
>
> Some people feel strongly about this and seem to think that ARMv8
> kernels will only work with ACPI in the future...

My strong feeling is that AArch64 kernels *may* support ACPI in the future ;).

On a more serious note, both FDT and ACPI will be first-class citizens
on AArch64 and I have no intention whatsoever of dropping FDT.

-- 
Catalin

Re: [RFC] ARM VM System Sepcification

[Catalin Marinas]

On 26 February 2014 20:05, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
>> On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
>> > For more information about UEFI and ACPI booting, see [4] and [5].
>>
>> What's the point of having ACPI in a virtual machine? You wouldn't
>> need to abstract any of the hardware in AML since you already know
>> what the virtual hardware is, so I can't see how this would help
>> anyone.
>
> The most common response I've been getting so far is that people
> generally want their VMs to look close to the real thing, but not sure
> how valid an argument that is.
>
> Some people feel strongly about this and seem to think that ARMv8
> kernels will only work with ACPI in the future...

My strong feeling is that AArch64 kernels *may* support ACPI in the future ;).

On a more serious note, both FDT and ACPI will be first-class citizens
on AArch64 and I have no intention whatsoever of dropping FDT.

-- 
Catalin

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 27/02/2014 08:30, Arnd Bergmann ha scritto:
> I guess the real question is whether we are interested in running Windows RT
> in VM guests. I don't personally expect MS to come out with a port for
> this spec, no matter what we do, but some of you may have information I don't.

Given enough firmware and driver support there's no reason why Windows 
and Linux should be any different---they certainly aren't on x86.

Right now Windows on ARM is just tablets and hence RT; but things might 
change for server-based ARM.  So Windows support as a VM should 
definitely be on the table, but so is writing custom drivers: you 
certainly will be able at some point to build virtio drivers for ARM, 
and use them with this spec.

Paolo

[RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 27/02/2014 08:30, Arnd Bergmann ha scritto:
> I guess the real question is whether we are interested in running Windows RT
> in VM guests. I don't personally expect MS to come out with a port for
> this spec, no matter what we do, but some of you may have information I don't.

Given enough firmware and driver support there's no reason why Windows 
and Linux should be any different---they certainly aren't on x86.

Right now Windows on ARM is just tablets and hence RT; but things might 
change for server-based ARM.  So Windows support as a VM should 
definitely be on the table, but so is writing custom drivers: you 
certainly will be able at some point to build virtio drivers for ARM, 
and use them with this spec.

Paolo

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 27/02/2014 08:30, Arnd Bergmann ha scritto:
> I guess the real question is whether we are interested in running Windows RT
> in VM guests. I don't personally expect MS to come out with a port for
> this spec, no matter what we do, but some of you may have information I don't.

Given enough firmware and driver support there's no reason why Windows 
and Linux should be any different---they certainly aren't on x86.

Right now Windows on ARM is just tablets and hence RT; but things might 
change for server-based ARM.  So Windows support as a VM should 
definitely be on the table, but so is writing custom drivers: you 
certainly will be able at some point to build virtio drivers for ARM, 
and use them with this spec.

Paolo

Re: [RFC] ARM VM System Sepcification

[Robie Basak]

[-- Attachment #1: Type: text/plain, Size: 926 bytes --]

On Wed, Feb 26, 2014 at 05:08:38PM -0600, Rob Herring wrote:
> > What does the 8250 have to recommend it over just providing
> > the PL011?
> 
> As I mentioned, it will just work for anything that expects the serial
> port to be ttyS0 as on x86 rather than ttyAMA0. Really, I'd like to
> see ttyAMA go away, but evidently that's not an easily fixed issue and
> it is an ABI.

Kernel parameters (eg. console=...) are going to be embedded inside the
guest VM disk image inside a grub configuration or something like that,
right? Is there enough here to ensure that I can ship a guest VM disk
image that will correctly work (ie. use the right device) for both
kernel console output and userspace getty, regardless of what the host
is? Reading your post I realise that I'm not sure what I need to do to
make this work, since the device may be different. Can something be
recommended as part of the spec for this?

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[RFC] ARM VM System Sepcification

[Robie Basak]

On Wed, Feb 26, 2014 at 05:08:38PM -0600, Rob Herring wrote:
> > What does the 8250 have to recommend it over just providing
> > the PL011?
> 
> As I mentioned, it will just work for anything that expects the serial
> port to be ttyS0 as on x86 rather than ttyAMA0. Really, I'd like to
> see ttyAMA go away, but evidently that's not an easily fixed issue and
> it is an ABI.

Kernel parameters (eg. console=...) are going to be embedded inside the
guest VM disk image inside a grub configuration or something like that,
right? Is there enough here to ensure that I can ship a guest VM disk
image that will correctly work (ie. use the right device) for both
kernel console output and userspace getty, regardless of what the host
is? Reading your post I realise that I'm not sure what I need to do to
make this work, since the device may be different. Can something be
recommended as part of the spec for this?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140227/45b03c17/attachment.sig>

Re: [RFC] ARM VM System Sepcification

[Robie Basak]

[-- Attachment #1.1: Type: text/plain, Size: 926 bytes --]

On Wed, Feb 26, 2014 at 05:08:38PM -0600, Rob Herring wrote:
> > What does the 8250 have to recommend it over just providing
> > the PL011?
> 
> As I mentioned, it will just work for anything that expects the serial
> port to be ttyS0 as on x86 rather than ttyAMA0. Really, I'd like to
> see ttyAMA go away, but evidently that's not an easily fixed issue and
> it is an ABI.

Kernel parameters (eg. console=...) are going to be embedded inside the
guest VM disk image inside a grub configuration or something like that,
right? Is there enough here to ensure that I can ship a guest VM disk
image that will correctly work (ie. use the right device) for both
kernel console output and userspace getty, regardless of what the host
is? Reading your post I realise that I'm not sure what I need to do to
make this work, since the device may be different. Can something be
recommended as part of the spec for this?

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] ARM VM System Sepcification

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1782 bytes --]

On Wed, 26 Feb 2014, Grant Likely wrote:
> > VM Platform
> > -----------
> > The specification does not mandate any specific memory map.  The guest
> > OS must be able to enumerate all processing elements, devices, and
> > memory through HW description data (FDT, ACPI) or a bus-specific
> > mechanism such as PCI.
> >
> > The virtual platform must support at least one of the following ARM
> > execution states:
> >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> >
> > It is recommended to support both (2) and (3) on aarch64 capable
> > physical systems.
> >
> > The virtual hardware platform must provide a number of mandatory
> > peripherals:
> >
> >   Serial console:  The platform should provide a console,
> >   based on an emulated pl011, a virtio-console, or a Xen PV console.
> 
> For portable disk image, can Xen PV be dropped from the list? pl011 is part of SBSA, and virtio is getting standardised, but Xen PV is
> implementation specific.

Does an interface need OASIS' rubber stamp to be "standard"?  If so, we
should also drop FDT from this document. The SBSA has not been published
by any OASIS-like standardization body either, so maybe we should drop
the SBSA too.

If it doesn't need OASIS nice logo on the side to be a standard, then
the Xen PV interfaces are a standard too. Give a look at
xen/include/public/io, they go as back as 2004, and they have multiple
different implementations of the frontends and backends in multiple
operating systems today.

There is no reason why another hypervisor couldn't implement the same
interface and in fact I know for a fact that it was considered for KVM.

[RFC] ARM VM System Sepcification

[Stefano Stabellini]

On Wed, 26 Feb 2014, Grant Likely wrote:
> > VM Platform
> > -----------
> > The specification does not mandate any specific memory map. ?The guest
> > OS must be able to enumerate all processing elements, devices, and
> > memory through HW description data (FDT, ACPI) or a bus-specific
> > mechanism such as PCI.
> >
> > The virtual platform must support at least one of the following ARM
> > execution states:
> > ? (1) aarch32 virtual CPUs on aarch32 physical CPUs
> > ? (2) aarch32 virtual CPUs on aarch64 physical CPUs
> > ? (3) aarch64 virtual CPUs on aarch64 physical CPUs
> >
> > It is recommended to support both (2) and (3) on aarch64 capable
> > physical systems.
> >
> > The virtual hardware platform must provide a number of mandatory
> > peripherals:
> >
> > ? Serial console: ?The platform should provide a console,
> > ? based on an emulated pl011, a virtio-console, or a Xen PV console.
> 
> For portable disk image, can Xen PV be dropped from the list? pl011 is part of SBSA, and virtio is getting standardised, but Xen PV is
> implementation specific.

Does an interface need OASIS' rubber stamp to be "standard"?  If so, we
should also drop FDT from this document. The SBSA has not been published
by any OASIS-like standardization body either, so maybe we should drop
the SBSA too.

If it doesn't need OASIS nice logo on the side to be a standard, then
the Xen PV interfaces are a standard too. Give a look at
xen/include/public/io, they go as back as 2004, and they have multiple
different implementations of the frontends and backends in multiple
operating systems today.

There is no reason why another hypervisor couldn't implement the same
interface and in fact I know for a fact that it was considered for KVM.

Re: [RFC] ARM VM System Sepcification

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1782 bytes --]

On Wed, 26 Feb 2014, Grant Likely wrote:
> > VM Platform
> > -----------
> > The specification does not mandate any specific memory map.  The guest
> > OS must be able to enumerate all processing elements, devices, and
> > memory through HW description data (FDT, ACPI) or a bus-specific
> > mechanism such as PCI.
> >
> > The virtual platform must support at least one of the following ARM
> > execution states:
> >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> >
> > It is recommended to support both (2) and (3) on aarch64 capable
> > physical systems.
> >
> > The virtual hardware platform must provide a number of mandatory
> > peripherals:
> >
> >   Serial console:  The platform should provide a console,
> >   based on an emulated pl011, a virtio-console, or a Xen PV console.
> 
> For portable disk image, can Xen PV be dropped from the list? pl011 is part of SBSA, and virtio is getting standardised, but Xen PV is
> implementation specific.

Does an interface need OASIS' rubber stamp to be "standard"?  If so, we
should also drop FDT from this document. The SBSA has not been published
by any OASIS-like standardization body either, so maybe we should drop
the SBSA too.

If it doesn't need OASIS nice logo on the side to be a standard, then
the Xen PV interfaces are a standard too. Give a look at
xen/include/public/io, they go as back as 2004, and they have multiple
different implementations of the frontends and backends in multiple
operating systems today.

There is no reason why another hypervisor couldn't implement the same
interface and in fact I know for a fact that it was considered for KVM.

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] ARM VM System Sepcification

[Stefano Stabellini]

On Wed, 26 Feb 2014, Leif Lindholm wrote:
> > >   no FDT.  In this case, the VM implementation must provide ACPI, and
> > >   the OS must be able to locate the ACPI root pointer through the UEFI
> > >   system table.
> > > 
> > > For more information about the arm and arm64 boot conventions, see
> > > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > > Linux kernel source tree.
> > > 
> > > For more information about UEFI and ACPI booting, see [4] and [5].
> > 
> > What's the point of having ACPI in a virtual machine? You wouldn't
> > need to abstract any of the hardware in AML since you already know
> > what the virtual hardware is, so I can't see how this would help
> > anyone.
> 
> The point is that if we need to share any real hw then we need to use
> whatever the host has.

That's right.

I dislike ACPI as much as the next guy, but unfortunately if the host
only supports ACPI, the Linux driver for a particular device only works
together with ACPI, and you want to assign that device to a VM, then we
might be forced to use ACPI to describe it.

[RFC] ARM VM System Sepcification

[Stefano Stabellini]

On Wed, 26 Feb 2014, Leif Lindholm wrote:
> > >   no FDT.  In this case, the VM implementation must provide ACPI, and
> > >   the OS must be able to locate the ACPI root pointer through the UEFI
> > >   system table.
> > > 
> > > For more information about the arm and arm64 boot conventions, see
> > > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > > Linux kernel source tree.
> > > 
> > > For more information about UEFI and ACPI booting, see [4] and [5].
> > 
> > What's the point of having ACPI in a virtual machine? You wouldn't
> > need to abstract any of the hardware in AML since you already know
> > what the virtual hardware is, so I can't see how this would help
> > anyone.
> 
> The point is that if we need to share any real hw then we need to use
> whatever the host has.

That's right.

I dislike ACPI as much as the next guy, but unfortunately if the host
only supports ACPI, the Linux driver for a particular device only works
together with ACPI, and you want to assign that device to a VM, then we
might be forced to use ACPI to describe it.

Re: [RFC] ARM VM System Sepcification

[Stefano Stabellini]

On Wed, 26 Feb 2014, Leif Lindholm wrote:
> > >   no FDT.  In this case, the VM implementation must provide ACPI, and
> > >   the OS must be able to locate the ACPI root pointer through the UEFI
> > >   system table.
> > > 
> > > For more information about the arm and arm64 boot conventions, see
> > > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > > Linux kernel source tree.
> > > 
> > > For more information about UEFI and ACPI booting, see [4] and [5].
> > 
> > What's the point of having ACPI in a virtual machine? You wouldn't
> > need to abstract any of the hardware in AML since you already know
> > what the virtual hardware is, so I can't see how this would help
> > anyone.
> 
> The point is that if we need to share any real hw then we need to use
> whatever the host has.

That's right.

I dislike ACPI as much as the next guy, but unfortunately if the host
only supports ACPI, the Linux driver for a particular device only works
together with ACPI, and you want to assign that device to a VM, then we
might be forced to use ACPI to describe it.

Re: [RFC] ARM VM System Sepcification

[Peter Maydell]

On 26 February 2014 22:35, Grant Likely <grant.likely@linaro.org> wrote:
> On 26 Feb 2014 18:35, "Christoffer Dall" <christoffer.dall@linaro.org>
> wrote:
>>   Serial console:  The platform should provide a console,
>>   based on an emulated pl011, a virtio-console, or a Xen PV console.
>
> For portable disk image, can Xen PV be dropped from the list? pl011 is part
> of SBSA, and virtio is getting standardised, but Xen PV is implementation
> specific.

The underlying question here is to what extent we want to
force VMs to provide a single implementation of something
and to what extent we want to force guests to cope with "any
choice from some small set". Personally I don't think it's
realistic to ask the Xen folk to drop their long-standing
PV bus implementation, and so the right answer is roughly
what we have here, ie "guest kernels need to cope with both
situations". Otherwise Xen is going to go its own way anyway,
and you just end up either (a) ruling out Xen as a platform
for running portable disk images or (b) having an unofficial
requirement to handle Xen PV anyway if you want an actually
portable image, which I would assume distros do.

thanks
-- PMM

[RFC] ARM VM System Sepcification

[Peter Maydell]

On 26 February 2014 22:35, Grant Likely <grant.likely@linaro.org> wrote:
> On 26 Feb 2014 18:35, "Christoffer Dall" <christoffer.dall@linaro.org>
> wrote:
>>   Serial console:  The platform should provide a console,
>>   based on an emulated pl011, a virtio-console, or a Xen PV console.
>
> For portable disk image, can Xen PV be dropped from the list? pl011 is part
> of SBSA, and virtio is getting standardised, but Xen PV is implementation
> specific.

The underlying question here is to what extent we want to
force VMs to provide a single implementation of something
and to what extent we want to force guests to cope with "any
choice from some small set". Personally I don't think it's
realistic to ask the Xen folk to drop their long-standing
PV bus implementation, and so the right answer is roughly
what we have here, ie "guest kernels need to cope with both
situations". Otherwise Xen is going to go its own way anyway,
and you just end up either (a) ruling out Xen as a platform
for running portable disk images or (b) having an unofficial
requirement to handle Xen PV anyway if you want an actually
portable image, which I would assume distros do.

thanks
-- PMM

Re: [RFC] ARM VM System Sepcification

[Peter Maydell]

On 26 February 2014 22:35, Grant Likely <grant.likely@linaro.org> wrote:
> On 26 Feb 2014 18:35, "Christoffer Dall" <christoffer.dall@linaro.org>
> wrote:
>>   Serial console:  The platform should provide a console,
>>   based on an emulated pl011, a virtio-console, or a Xen PV console.
>
> For portable disk image, can Xen PV be dropped from the list? pl011 is part
> of SBSA, and virtio is getting standardised, but Xen PV is implementation
> specific.

The underlying question here is to what extent we want to
force VMs to provide a single implementation of something
and to what extent we want to force guests to cope with "any
choice from some small set". Personally I don't think it's
realistic to ask the Xen folk to drop their long-standing
PV bus implementation, and so the right answer is roughly
what we have here, ie "guest kernels need to cope with both
situations". Otherwise Xen is going to go its own way anyway,
and you just end up either (a) ruling out Xen as a platform
for running portable disk images or (b) having an unofficial
requirement to handle Xen PV anyway if you want an actually
portable image, which I would assume distros do.

thanks
-- PMM

Re: [RFC] ARM VM System Sepcification

[Christopher Covington]

Hi Christoffer,

On 02/26/2014 02:51 PM, Christoffer Dall wrote:
> On Wed, Feb 26, 2014 at 02:27:40PM -0500, Christopher Covington wrote:

>>> Image format
>>> ------------
>>> The image format, as presented to the VM, needs to be well-defined in
>>> order for prepared disk images to be bootable across various
>>> virtualization implementations.
>>>
>>> The raw disk format as presented to the VM must be partitioned with a
>>> GUID Partition Table (GPT).  The bootable software must be placed in the
>>> EFI System Partition (ESP), using the UEFI removable media path, and
>>> must be an EFI application complying to the UEFI Specification 2.4
>>> Revision A [6].
>>>
>>> The ESP partition's GPT entry's partition type GUID must be
>>> C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
>>> formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
>>>
>>> The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
>>> execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
>>> state.
>>>
>>> This ensures that tools for both Xen and KVM can load a binary UEFI
>>> firmware which can read and boot the EFI application in the disk image.
>>>
>>> A typical scenario will be GRUB2 packaged as an EFI application, which
>>> mounts the system boot partition and boots Linux.
>>>
>>>
>>> Virtual Firmware
>>> ----------------
>>> The VM system must be able to boot the EFI application in the ESP.  It
>>> is recommended that this is achieved by loading a UEFI binary as the
>>> first software executed by the VM, which then executes the EFI
>>> application.  The UEFI implementation should be compliant with UEFI
>>> Specification 2.4 Revision A [6] or later.
>>>
>>> This document strongly recommends that the VM implementation supports
>>> persistent environment storage for virtual firmware implementation in
>>> order to ensure probable use cases such as adding additional disk images
>>> to a VM or running installers to perform upgrades.
>>>
>>> The binary UEFI firmware implementation should not be distributed as
>>> part of the VM image, but is specific to the VM implementation.
>>
>> Can you elaborate on the motivation for requiring that the kernel be stuffed
>> into a disk image and for requiring such a heavyweight bootloader/firmware? By
>> doing so you would seem to exclude those requiring an optimized boot process.
>>
> 
> What's the alternative?  Shipping kernels externally and loading them
> externally?  Sure you can do that, but then distros can't upgrade the
> kernel themselves, and you have to come up with a convention for how to
> ship kernels, initrd's etc.

The self-hosted upgrades use case makes sense. I can imagine using a
pass-through or network filesystem to do it in the case of external loading,
something like the following. In the case of P9, the tag could be the same as
the GPT GUID. Everything could still be in the /EFI/BOOT directory. The kernel
Image could be at BOOT(ARM|AA64).IMG, the zImage at .ZMG, and the initramfs at
.RFS. It's more work for distros to support multiple upgrade methods, though,
so maybe those who want an optimized boot process should make an external
loader capable of carving the necessary components out of a VFAT filesystem
inside a GPT partitioned image instead.

>>> VM Platform
>>> -----------
>>> The specification does not mandate any specific memory map.  The guest
>>> OS must be able to enumerate all processing elements, devices, and
>>> memory through HW description data (FDT, ACPI) or a bus-specific
>>> mechanism such as PCI.
>>>
>>> The virtual platform must support at least one of the following ARM
>>> execution states:
>>>   (1) aarch32 virtual CPUs on aarch32 physical CPUs
>>>   (2) aarch32 virtual CPUs on aarch64 physical CPUs
>>>   (3) aarch64 virtual CPUs on aarch64 physical CPUs
>>>
>>> It is recommended to support both (2) and (3) on aarch64 capable
>>> physical systems.
>>>
>>> The virtual hardware platform must provide a number of mandatory
>>> peripherals:
>>>
>>>   Serial console:  The platform should provide a console,
>>>   based on an emulated pl011, a virtio-console, or a Xen PV console.
>>>
>>>   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
>>>   limits the the number of virtual CPUs to 8 cores, newer GIC versions
>>>   removes this limitation.
>>>
>>>   The ARM virtual timer and counter should be available to the VM as
>>>   per the ARM Generic Timers specification in the ARM ARM [1].
>>>
>>>   A hotpluggable bus to support hotplug of at least block and network
>>>   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
>>>   bus.
>>
>> Is VirtIO hotplug capable? Over PCI or MMIO transports or both?
> 
> VirtIO devices attached on a PCIe bus are hotpluggable, the emulated
> PCIe bus itself would not have anything to do with virtio, except that
> virtio devices can hang off of it.  AFAIU.

So network/block device only as memory mapped peripherals (like SMSC or PL
SD/MMC) or over VirtIO-MMIO won't meet the specification? Is PCI/VirtIO-PCI on
ARM production ready? What's the motivation for requiring hotplug?

Thanks,
Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

[RFC] ARM VM System Sepcification

[Christopher Covington]

Hi Christoffer,

On 02/26/2014 02:51 PM, Christoffer Dall wrote:
> On Wed, Feb 26, 2014 at 02:27:40PM -0500, Christopher Covington wrote:

>>> Image format
>>> ------------
>>> The image format, as presented to the VM, needs to be well-defined in
>>> order for prepared disk images to be bootable across various
>>> virtualization implementations.
>>>
>>> The raw disk format as presented to the VM must be partitioned with a
>>> GUID Partition Table (GPT).  The bootable software must be placed in the
>>> EFI System Partition (ESP), using the UEFI removable media path, and
>>> must be an EFI application complying to the UEFI Specification 2.4
>>> Revision A [6].
>>>
>>> The ESP partition's GPT entry's partition type GUID must be
>>> C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
>>> formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
>>>
>>> The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
>>> execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
>>> state.
>>>
>>> This ensures that tools for both Xen and KVM can load a binary UEFI
>>> firmware which can read and boot the EFI application in the disk image.
>>>
>>> A typical scenario will be GRUB2 packaged as an EFI application, which
>>> mounts the system boot partition and boots Linux.
>>>
>>>
>>> Virtual Firmware
>>> ----------------
>>> The VM system must be able to boot the EFI application in the ESP.  It
>>> is recommended that this is achieved by loading a UEFI binary as the
>>> first software executed by the VM, which then executes the EFI
>>> application.  The UEFI implementation should be compliant with UEFI
>>> Specification 2.4 Revision A [6] or later.
>>>
>>> This document strongly recommends that the VM implementation supports
>>> persistent environment storage for virtual firmware implementation in
>>> order to ensure probable use cases such as adding additional disk images
>>> to a VM or running installers to perform upgrades.
>>>
>>> The binary UEFI firmware implementation should not be distributed as
>>> part of the VM image, but is specific to the VM implementation.
>>
>> Can you elaborate on the motivation for requiring that the kernel be stuffed
>> into a disk image and for requiring such a heavyweight bootloader/firmware? By
>> doing so you would seem to exclude those requiring an optimized boot process.
>>
> 
> What's the alternative?  Shipping kernels externally and loading them
> externally?  Sure you can do that, but then distros can't upgrade the
> kernel themselves, and you have to come up with a convention for how to
> ship kernels, initrd's etc.

The self-hosted upgrades use case makes sense. I can imagine using a
pass-through or network filesystem to do it in the case of external loading,
something like the following. In the case of P9, the tag could be the same as
the GPT GUID. Everything could still be in the /EFI/BOOT directory. The kernel
Image could be at BOOT(ARM|AA64).IMG, the zImage at .ZMG, and the initramfs at
.RFS. It's more work for distros to support multiple upgrade methods, though,
so maybe those who want an optimized boot process should make an external
loader capable of carving the necessary components out of a VFAT filesystem
inside a GPT partitioned image instead.

>>> VM Platform
>>> -----------
>>> The specification does not mandate any specific memory map.  The guest
>>> OS must be able to enumerate all processing elements, devices, and
>>> memory through HW description data (FDT, ACPI) or a bus-specific
>>> mechanism such as PCI.
>>>
>>> The virtual platform must support at least one of the following ARM
>>> execution states:
>>>   (1) aarch32 virtual CPUs on aarch32 physical CPUs
>>>   (2) aarch32 virtual CPUs on aarch64 physical CPUs
>>>   (3) aarch64 virtual CPUs on aarch64 physical CPUs
>>>
>>> It is recommended to support both (2) and (3) on aarch64 capable
>>> physical systems.
>>>
>>> The virtual hardware platform must provide a number of mandatory
>>> peripherals:
>>>
>>>   Serial console:  The platform should provide a console,
>>>   based on an emulated pl011, a virtio-console, or a Xen PV console.
>>>
>>>   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
>>>   limits the the number of virtual CPUs to 8 cores, newer GIC versions
>>>   removes this limitation.
>>>
>>>   The ARM virtual timer and counter should be available to the VM as
>>>   per the ARM Generic Timers specification in the ARM ARM [1].
>>>
>>>   A hotpluggable bus to support hotplug of at least block and network
>>>   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
>>>   bus.
>>
>> Is VirtIO hotplug capable? Over PCI or MMIO transports or both?
> 
> VirtIO devices attached on a PCIe bus are hotpluggable, the emulated
> PCIe bus itself would not have anything to do with virtio, except that
> virtio devices can hang off of it.  AFAIU.

So network/block device only as memory mapped peripherals (like SMSC or PL
SD/MMC) or over VirtIO-MMIO won't meet the specification? Is PCI/VirtIO-PCI on
ARM production ready? What's the motivation for requiring hotplug?

Thanks,
Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

Re: [RFC] ARM VM System Sepcification

[Christopher Covington]

Hi Christoffer,

On 02/26/2014 02:51 PM, Christoffer Dall wrote:
> On Wed, Feb 26, 2014 at 02:27:40PM -0500, Christopher Covington wrote:

>>> Image format
>>> ------------
>>> The image format, as presented to the VM, needs to be well-defined in
>>> order for prepared disk images to be bootable across various
>>> virtualization implementations.
>>>
>>> The raw disk format as presented to the VM must be partitioned with a
>>> GUID Partition Table (GPT).  The bootable software must be placed in the
>>> EFI System Partition (ESP), using the UEFI removable media path, and
>>> must be an EFI application complying to the UEFI Specification 2.4
>>> Revision A [6].
>>>
>>> The ESP partition's GPT entry's partition type GUID must be
>>> C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
>>> formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
>>>
>>> The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
>>> execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
>>> state.
>>>
>>> This ensures that tools for both Xen and KVM can load a binary UEFI
>>> firmware which can read and boot the EFI application in the disk image.
>>>
>>> A typical scenario will be GRUB2 packaged as an EFI application, which
>>> mounts the system boot partition and boots Linux.
>>>
>>>
>>> Virtual Firmware
>>> ----------------
>>> The VM system must be able to boot the EFI application in the ESP.  It
>>> is recommended that this is achieved by loading a UEFI binary as the
>>> first software executed by the VM, which then executes the EFI
>>> application.  The UEFI implementation should be compliant with UEFI
>>> Specification 2.4 Revision A [6] or later.
>>>
>>> This document strongly recommends that the VM implementation supports
>>> persistent environment storage for virtual firmware implementation in
>>> order to ensure probable use cases such as adding additional disk images
>>> to a VM or running installers to perform upgrades.
>>>
>>> The binary UEFI firmware implementation should not be distributed as
>>> part of the VM image, but is specific to the VM implementation.
>>
>> Can you elaborate on the motivation for requiring that the kernel be stuffed
>> into a disk image and for requiring such a heavyweight bootloader/firmware? By
>> doing so you would seem to exclude those requiring an optimized boot process.
>>
> 
> What's the alternative?  Shipping kernels externally and loading them
> externally?  Sure you can do that, but then distros can't upgrade the
> kernel themselves, and you have to come up with a convention for how to
> ship kernels, initrd's etc.

The self-hosted upgrades use case makes sense. I can imagine using a
pass-through or network filesystem to do it in the case of external loading,
something like the following. In the case of P9, the tag could be the same as
the GPT GUID. Everything could still be in the /EFI/BOOT directory. The kernel
Image could be at BOOT(ARM|AA64).IMG, the zImage at .ZMG, and the initramfs at
.RFS. It's more work for distros to support multiple upgrade methods, though,
so maybe those who want an optimized boot process should make an external
loader capable of carving the necessary components out of a VFAT filesystem
inside a GPT partitioned image instead.

>>> VM Platform
>>> -----------
>>> The specification does not mandate any specific memory map.  The guest
>>> OS must be able to enumerate all processing elements, devices, and
>>> memory through HW description data (FDT, ACPI) or a bus-specific
>>> mechanism such as PCI.
>>>
>>> The virtual platform must support at least one of the following ARM
>>> execution states:
>>>   (1) aarch32 virtual CPUs on aarch32 physical CPUs
>>>   (2) aarch32 virtual CPUs on aarch64 physical CPUs
>>>   (3) aarch64 virtual CPUs on aarch64 physical CPUs
>>>
>>> It is recommended to support both (2) and (3) on aarch64 capable
>>> physical systems.
>>>
>>> The virtual hardware platform must provide a number of mandatory
>>> peripherals:
>>>
>>>   Serial console:  The platform should provide a console,
>>>   based on an emulated pl011, a virtio-console, or a Xen PV console.
>>>
>>>   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
>>>   limits the the number of virtual CPUs to 8 cores, newer GIC versions
>>>   removes this limitation.
>>>
>>>   The ARM virtual timer and counter should be available to the VM as
>>>   per the ARM Generic Timers specification in the ARM ARM [1].
>>>
>>>   A hotpluggable bus to support hotplug of at least block and network
>>>   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
>>>   bus.
>>
>> Is VirtIO hotplug capable? Over PCI or MMIO transports or both?
> 
> VirtIO devices attached on a PCIe bus are hotpluggable, the emulated
> PCIe bus itself would not have anything to do with virtio, except that
> virtio devices can hang off of it.  AFAIU.

So network/block device only as memory mapped peripherals (like SMSC or PL
SD/MMC) or over VirtIO-MMIO won't meet the specification? Is PCI/VirtIO-PCI on
ARM production ready? What's the motivation for requiring hotplug?

Thanks,
Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

Re: [RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Thursday 27 February 2014 12:31:55 Stefano Stabellini wrote:
> On Wed, 26 Feb 2014, Leif Lindholm wrote:
> > > >   no FDT.  In this case, the VM implementation must provide ACPI, and
> > > >   the OS must be able to locate the ACPI root pointer through the UEFI
> > > >   system table.
> > > > 
> > > > For more information about the arm and arm64 boot conventions, see
> > > > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > > > Linux kernel source tree.
> > > > 
> > > > For more information about UEFI and ACPI booting, see [4] and [5].
> > > 
> > > What's the point of having ACPI in a virtual machine? You wouldn't
> > > need to abstract any of the hardware in AML since you already know
> > > what the virtual hardware is, so I can't see how this would help
> > > anyone.
> > 
> > The point is that if we need to share any real hw then we need to use
> > whatever the host has.

I would be more comfortable defining in the spec that you cannot share
hardware at all. Obviously that doesn't stop anyone from actually
sharing hardware with the guest, but at that point it would become
noncompliant with this spec, with the consequence that you couldn't
expect a compliant guest image to run on that hardware, but that is
exactly something we can't guarantee anyway because we don't know
what drivers might be needed.

Also, there is no way to generally do this with either FDT or ACPI:
In the former case, the hypervisor needs to modify any properties
that point to other device nodes so that they point to nodes visible
to the guest. That may be possible for simple things like IRQs
and reg properties, but as soon as you get into stuff like dmaengine,
pinctrl or PHY references, you just can't solve it in a generic way.

For ACPI it's probably worse: any AML methods that the host has
are unlikely to work in the guest, and it's impossible to translate
them at all.

Obviously things are different for Xen Dom0 where we share *all* devices
between host and guest, and we just use the host firmware interfaces.
That case again cannot be covered by the generic VM system specification.

> I dislike ACPI as much as the next guy, but unfortunately if the host
> only supports ACPI, the Linux driver for a particular device only works
> together with ACPI, and you want to assign that device to a VM, then we
> might be forced to use ACPI to describe it.

Can anyone think of an example where this would actually work?

The only case I can see where it's possible to share a device with
a guest without the hypervisor building up the description is for
PCI functions that are passed through with an IOMMU. Those won't
need ACPI or DT support however.

	Arnd

[RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Thursday 27 February 2014 12:31:55 Stefano Stabellini wrote:
> On Wed, 26 Feb 2014, Leif Lindholm wrote:
> > > >   no FDT.  In this case, the VM implementation must provide ACPI, and
> > > >   the OS must be able to locate the ACPI root pointer through the UEFI
> > > >   system table.
> > > > 
> > > > For more information about the arm and arm64 boot conventions, see
> > > > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > > > Linux kernel source tree.
> > > > 
> > > > For more information about UEFI and ACPI booting, see [4] and [5].
> > > 
> > > What's the point of having ACPI in a virtual machine? You wouldn't
> > > need to abstract any of the hardware in AML since you already know
> > > what the virtual hardware is, so I can't see how this would help
> > > anyone.
> > 
> > The point is that if we need to share any real hw then we need to use
> > whatever the host has.

I would be more comfortable defining in the spec that you cannot share
hardware at all. Obviously that doesn't stop anyone from actually
sharing hardware with the guest, but at that point it would become
noncompliant with this spec, with the consequence that you couldn't
expect a compliant guest image to run on that hardware, but that is
exactly something we can't guarantee anyway because we don't know
what drivers might be needed.

Also, there is no way to generally do this with either FDT or ACPI:
In the former case, the hypervisor needs to modify any properties
that point to other device nodes so that they point to nodes visible
to the guest. That may be possible for simple things like IRQs
and reg properties, but as soon as you get into stuff like dmaengine,
pinctrl or PHY references, you just can't solve it in a generic way.

For ACPI it's probably worse: any AML methods that the host has
are unlikely to work in the guest, and it's impossible to translate
them at all.

Obviously things are different for Xen Dom0 where we share *all* devices
between host and guest, and we just use the host firmware interfaces.
That case again cannot be covered by the generic VM system specification.

> I dislike ACPI as much as the next guy, but unfortunately if the host
> only supports ACPI, the Linux driver for a particular device only works
> together with ACPI, and you want to assign that device to a VM, then we
> might be forced to use ACPI to describe it.

Can anyone think of an example where this would actually work?

The only case I can see where it's possible to share a device with
a guest without the hypervisor building up the description is for
PCI functions that are passed through with an IOMMU. Those won't
need ACPI or DT support however.

	Arnd

Re: [RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Thursday 27 February 2014 12:31:55 Stefano Stabellini wrote:
> On Wed, 26 Feb 2014, Leif Lindholm wrote:
> > > >   no FDT.  In this case, the VM implementation must provide ACPI, and
> > > >   the OS must be able to locate the ACPI root pointer through the UEFI
> > > >   system table.
> > > > 
> > > > For more information about the arm and arm64 boot conventions, see
> > > > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > > > Linux kernel source tree.
> > > > 
> > > > For more information about UEFI and ACPI booting, see [4] and [5].
> > > 
> > > What's the point of having ACPI in a virtual machine? You wouldn't
> > > need to abstract any of the hardware in AML since you already know
> > > what the virtual hardware is, so I can't see how this would help
> > > anyone.
> > 
> > The point is that if we need to share any real hw then we need to use
> > whatever the host has.

I would be more comfortable defining in the spec that you cannot share
hardware at all. Obviously that doesn't stop anyone from actually
sharing hardware with the guest, but at that point it would become
noncompliant with this spec, with the consequence that you couldn't
expect a compliant guest image to run on that hardware, but that is
exactly something we can't guarantee anyway because we don't know
what drivers might be needed.

Also, there is no way to generally do this with either FDT or ACPI:
In the former case, the hypervisor needs to modify any properties
that point to other device nodes so that they point to nodes visible
to the guest. That may be possible for simple things like IRQs
and reg properties, but as soon as you get into stuff like dmaengine,
pinctrl or PHY references, you just can't solve it in a generic way.

For ACPI it's probably worse: any AML methods that the host has
are unlikely to work in the guest, and it's impossible to translate
them at all.

Obviously things are different for Xen Dom0 where we share *all* devices
between host and guest, and we just use the host firmware interfaces.
That case again cannot be covered by the generic VM system specification.

> I dislike ACPI as much as the next guy, but unfortunately if the host
> only supports ACPI, the Linux driver for a particular device only works
> together with ACPI, and you want to assign that device to a VM, then we
> might be forced to use ACPI to describe it.

Can anyone think of an example where this would actually work?

The only case I can see where it's possible to share a device with
a guest without the hypervisor building up the description is for
PCI functions that are passed through with an IOMMU. Those won't
need ACPI or DT support however.

	Arnd

Re: [RFC] ARM VM System Sepcification

[Alexander Graf]

> Am 27.02.2014 um 22:00 schrieb Arnd Bergmann <arnd@arndb.de>:
> 
>> On Thursday 27 February 2014 12:31:55 Stefano Stabellini wrote:
>> On Wed, 26 Feb 2014, Leif Lindholm wrote:
>>>>>  no FDT.  In this case, the VM implementation must provide ACPI, and
>>>>>  the OS must be able to locate the ACPI root pointer through the UEFI
>>>>>  system table.
>>>>> 
>>>>> For more information about the arm and arm64 boot conventions, see
>>>>> Documentation/arm/Booting and Documentation/arm64/booting.txt in the
>>>>> Linux kernel source tree.
>>>>> 
>>>>> For more information about UEFI and ACPI booting, see [4] and [5].
>>>> 
>>>> What's the point of having ACPI in a virtual machine? You wouldn't
>>>> need to abstract any of the hardware in AML since you already know
>>>> what the virtual hardware is, so I can't see how this would help
>>>> anyone.
>>> 
>>> The point is that if we need to share any real hw then we need to use
>>> whatever the host has.
> 
> I would be more comfortable defining in the spec that you cannot share
> hardware at all. Obviously that

Nonono we want to share hardware.

> doesn't stop anyone from actually
> sharing hardware with the guest, but at that point it would become
> noncompliant with this spec, with the consequence that you couldn't
> expect a compliant guest image to run on that hardware, but that is
> exactly something we can't guarantee anyway because we don't know
> what drivers might be needed.
> 
> Also, there is no way to generally do this with either FDT or ACPI:
> In the former case, the hypervisor needs to modify any properties
> that point to other device nodes so that they point to nodes visible
> to the guest. That may be possible for simple things like IRQs
> and reg properties, but as soon as you get into stuff like dmaengine,
> pinctrl or PHY references, you just can't solve it in a generic way.
> 
> For ACPI it's probably worse: any AML methods that the host has
> are unlikely to work in the guest, and it's impossible to translate
> them at all.
> 
> Obviously things are different for Xen Dom0 where we share *all* devices
> between host and guest, and we just use the host firmware interfaces.
> That case again cannot be covered by the generic VM system specification.
> 
>> I dislike ACPI as much as the next guy, but unfortunately if the host
>> only supports ACPI, the Linux driver for a particular device only works
>> together with ACPI, and you want to assign that device to a VM, then we
>> might be forced to use ACPI to describe it.
> 
> Can anyone think of an example where this would actually work?
> 
> The only case I can see where it's possible to share a device with
> a guest without the hypervisor building up the description is for
> PCI functions that are passed through with an IOMMU. Those won't
> need ACPI or DT support however.

If you want to assign a platform device, you need to generate a respective hardware description (fdt/dsdt) chunk in the hypervisor. You can't take the host's description - it's too tightly coupled to the overall host layout.

Imagine you get an AArch64 notebook with Windows on it. You want to run Linux there, so your host needs to understand ACPI. Now you want to run a Windows guest inside a VM, so you need ACPI in there again.

Replace Windows by "Linux with custom drivers" and you're in the same situation even when you neglect Windows. Reality will be that we will have fdt and acpi based systems.


Alex

[RFC] ARM VM System Sepcification

[Alexander Graf]

> Am 27.02.2014 um 22:00 schrieb Arnd Bergmann <arnd@arndb.de>:
> 
>> On Thursday 27 February 2014 12:31:55 Stefano Stabellini wrote:
>> On Wed, 26 Feb 2014, Leif Lindholm wrote:
>>>>>  no FDT.  In this case, the VM implementation must provide ACPI, and
>>>>>  the OS must be able to locate the ACPI root pointer through the UEFI
>>>>>  system table.
>>>>> 
>>>>> For more information about the arm and arm64 boot conventions, see
>>>>> Documentation/arm/Booting and Documentation/arm64/booting.txt in the
>>>>> Linux kernel source tree.
>>>>> 
>>>>> For more information about UEFI and ACPI booting, see [4] and [5].
>>>> 
>>>> What's the point of having ACPI in a virtual machine? You wouldn't
>>>> need to abstract any of the hardware in AML since you already know
>>>> what the virtual hardware is, so I can't see how this would help
>>>> anyone.
>>> 
>>> The point is that if we need to share any real hw then we need to use
>>> whatever the host has.
> 
> I would be more comfortable defining in the spec that you cannot share
> hardware at all. Obviously that

Nonono we want to share hardware.

> doesn't stop anyone from actually
> sharing hardware with the guest, but at that point it would become
> noncompliant with this spec, with the consequence that you couldn't
> expect a compliant guest image to run on that hardware, but that is
> exactly something we can't guarantee anyway because we don't know
> what drivers might be needed.
> 
> Also, there is no way to generally do this with either FDT or ACPI:
> In the former case, the hypervisor needs to modify any properties
> that point to other device nodes so that they point to nodes visible
> to the guest. That may be possible for simple things like IRQs
> and reg properties, but as soon as you get into stuff like dmaengine,
> pinctrl or PHY references, you just can't solve it in a generic way.
> 
> For ACPI it's probably worse: any AML methods that the host has
> are unlikely to work in the guest, and it's impossible to translate
> them at all.
> 
> Obviously things are different for Xen Dom0 where we share *all* devices
> between host and guest, and we just use the host firmware interfaces.
> That case again cannot be covered by the generic VM system specification.
> 
>> I dislike ACPI as much as the next guy, but unfortunately if the host
>> only supports ACPI, the Linux driver for a particular device only works
>> together with ACPI, and you want to assign that device to a VM, then we
>> might be forced to use ACPI to describe it.
> 
> Can anyone think of an example where this would actually work?
> 
> The only case I can see where it's possible to share a device with
> a guest without the hypervisor building up the description is for
> PCI functions that are passed through with an IOMMU. Those won't
> need ACPI or DT support however.

If you want to assign a platform device, you need to generate a respective hardware description (fdt/dsdt) chunk in the hypervisor. You can't take the host's description - it's too tightly coupled to the overall host layout.

Imagine you get an AArch64 notebook with Windows on it. You want to run Linux there, so your host needs to understand ACPI. Now you want to run a Windows guest inside a VM, so you need ACPI in there again.

Replace Windows by "Linux with custom drivers" and you're in the same situation even when you neglect Windows. Reality will be that we will have fdt and acpi based systems.


Alex

Re: [RFC] ARM VM System Sepcification

[Alexander Graf]

> Am 27.02.2014 um 22:00 schrieb Arnd Bergmann <arnd@arndb.de>:
> 
>> On Thursday 27 February 2014 12:31:55 Stefano Stabellini wrote:
>> On Wed, 26 Feb 2014, Leif Lindholm wrote:
>>>>>  no FDT.  In this case, the VM implementation must provide ACPI, and
>>>>>  the OS must be able to locate the ACPI root pointer through the UEFI
>>>>>  system table.
>>>>> 
>>>>> For more information about the arm and arm64 boot conventions, see
>>>>> Documentation/arm/Booting and Documentation/arm64/booting.txt in the
>>>>> Linux kernel source tree.
>>>>> 
>>>>> For more information about UEFI and ACPI booting, see [4] and [5].
>>>> 
>>>> What's the point of having ACPI in a virtual machine? You wouldn't
>>>> need to abstract any of the hardware in AML since you already know
>>>> what the virtual hardware is, so I can't see how this would help
>>>> anyone.
>>> 
>>> The point is that if we need to share any real hw then we need to use
>>> whatever the host has.
> 
> I would be more comfortable defining in the spec that you cannot share
> hardware at all. Obviously that

Nonono we want to share hardware.

> doesn't stop anyone from actually
> sharing hardware with the guest, but at that point it would become
> noncompliant with this spec, with the consequence that you couldn't
> expect a compliant guest image to run on that hardware, but that is
> exactly something we can't guarantee anyway because we don't know
> what drivers might be needed.
> 
> Also, there is no way to generally do this with either FDT or ACPI:
> In the former case, the hypervisor needs to modify any properties
> that point to other device nodes so that they point to nodes visible
> to the guest. That may be possible for simple things like IRQs
> and reg properties, but as soon as you get into stuff like dmaengine,
> pinctrl or PHY references, you just can't solve it in a generic way.
> 
> For ACPI it's probably worse: any AML methods that the host has
> are unlikely to work in the guest, and it's impossible to translate
> them at all.
> 
> Obviously things are different for Xen Dom0 where we share *all* devices
> between host and guest, and we just use the host firmware interfaces.
> That case again cannot be covered by the generic VM system specification.
> 
>> I dislike ACPI as much as the next guy, but unfortunately if the host
>> only supports ACPI, the Linux driver for a particular device only works
>> together with ACPI, and you want to assign that device to a VM, then we
>> might be forced to use ACPI to describe it.
> 
> Can anyone think of an example where this would actually work?
> 
> The only case I can see where it's possible to share a device with
> a guest without the hypervisor building up the description is for
> PCI functions that are passed through with an IOMMU. Those won't
> need ACPI or DT support however.

If you want to assign a platform device, you need to generate a respective hardware description (fdt/dsdt) chunk in the hypervisor. You can't take the host's description - it's too tightly coupled to the overall host layout.

Imagine you get an AArch64 notebook with Windows on it. You want to run Linux there, so your host needs to understand ACPI. Now you want to run a Windows guest inside a VM, so you need ACPI in there again.

Replace Windows by "Linux with custom drivers" and you're in the same situation even when you neglect Windows. Reality will be that we will have fdt and acpi based systems.


Alex

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Thu, Feb 27, 2014 at 08:12:35AM -0500, Christopher Covington wrote:
> Hi Christoffer,
> 
> On 02/26/2014 02:51 PM, Christoffer Dall wrote:
> > On Wed, Feb 26, 2014 at 02:27:40PM -0500, Christopher Covington wrote:
> 

[...]

> >>>
> >>> The virtual hardware platform must provide a number of mandatory
> >>> peripherals:
> >>>
> >>>   Serial console:  The platform should provide a console,
> >>>   based on an emulated pl011, a virtio-console, or a Xen PV console.
> >>>
> >>>   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
> >>>   limits the the number of virtual CPUs to 8 cores, newer GIC versions
> >>>   removes this limitation.
> >>>
> >>>   The ARM virtual timer and counter should be available to the VM as
> >>>   per the ARM Generic Timers specification in the ARM ARM [1].
> >>>
> >>>   A hotpluggable bus to support hotplug of at least block and network
> >>>   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
> >>>   bus.
> >>
> >> Is VirtIO hotplug capable? Over PCI or MMIO transports or both?
> > 
> > VirtIO devices attached on a PCIe bus are hotpluggable, the emulated
> > PCIe bus itself would not have anything to do with virtio, except that
> > virtio devices can hang off of it.  AFAIU.
> 
> So network/block device only as memory mapped peripherals (like SMSC or PL
> SD/MMC) or over VirtIO-MMIO won't meet the specification? Is PCI/VirtIO-PCI on
> ARM production ready? What's the motivation for requiring hotplug?
> 

Platform devices that don't sit on any 'real bus' are generally not
hotpluggable.

VM management systems such as OpenStack make heavy use of hotplug to add
storage to your VMs, for example.

This spec does not prohibit devices over mmio, or over virtio-mmio, in
fact it encourages guest kernels to include support for such.  But it
mandates that there is some hotpluggable bus, so a very common VM use
case is supported for ARM VMs.

PCI for ARM is not ready yet, but people are working on it.  That should
not have any bearing on what the right decision for this spec is though
- it's all early stage at this point.

-Christoffer

[RFC] ARM VM System Sepcification

[Christoffer Dall]

On Thu, Feb 27, 2014 at 08:12:35AM -0500, Christopher Covington wrote:
> Hi Christoffer,
> 
> On 02/26/2014 02:51 PM, Christoffer Dall wrote:
> > On Wed, Feb 26, 2014 at 02:27:40PM -0500, Christopher Covington wrote:
> 

[...]

> >>>
> >>> The virtual hardware platform must provide a number of mandatory
> >>> peripherals:
> >>>
> >>>   Serial console:  The platform should provide a console,
> >>>   based on an emulated pl011, a virtio-console, or a Xen PV console.
> >>>
> >>>   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
> >>>   limits the the number of virtual CPUs to 8 cores, newer GIC versions
> >>>   removes this limitation.
> >>>
> >>>   The ARM virtual timer and counter should be available to the VM as
> >>>   per the ARM Generic Timers specification in the ARM ARM [1].
> >>>
> >>>   A hotpluggable bus to support hotplug of at least block and network
> >>>   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
> >>>   bus.
> >>
> >> Is VirtIO hotplug capable? Over PCI or MMIO transports or both?
> > 
> > VirtIO devices attached on a PCIe bus are hotpluggable, the emulated
> > PCIe bus itself would not have anything to do with virtio, except that
> > virtio devices can hang off of it.  AFAIU.
> 
> So network/block device only as memory mapped peripherals (like SMSC or PL
> SD/MMC) or over VirtIO-MMIO won't meet the specification? Is PCI/VirtIO-PCI on
> ARM production ready? What's the motivation for requiring hotplug?
> 

Platform devices that don't sit on any 'real bus' are generally not
hotpluggable.

VM management systems such as OpenStack make heavy use of hotplug to add
storage to your VMs, for example.

This spec does not prohibit devices over mmio, or over virtio-mmio, in
fact it encourages guest kernels to include support for such.  But it
mandates that there is some hotpluggable bus, so a very common VM use
case is supported for ARM VMs.

PCI for ARM is not ready yet, but people are working on it.  That should
not have any bearing on what the right decision for this spec is though
- it's all early stage at this point.

-Christoffer

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Thu, Feb 27, 2014 at 08:12:35AM -0500, Christopher Covington wrote:
> Hi Christoffer,
> 
> On 02/26/2014 02:51 PM, Christoffer Dall wrote:
> > On Wed, Feb 26, 2014 at 02:27:40PM -0500, Christopher Covington wrote:
> 

[...]

> >>>
> >>> The virtual hardware platform must provide a number of mandatory
> >>> peripherals:
> >>>
> >>>   Serial console:  The platform should provide a console,
> >>>   based on an emulated pl011, a virtio-console, or a Xen PV console.
> >>>
> >>>   An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
> >>>   limits the the number of virtual CPUs to 8 cores, newer GIC versions
> >>>   removes this limitation.
> >>>
> >>>   The ARM virtual timer and counter should be available to the VM as
> >>>   per the ARM Generic Timers specification in the ARM ARM [1].
> >>>
> >>>   A hotpluggable bus to support hotplug of at least block and network
> >>>   devices.  Suitable buses include a virtual PCIe bus and the Xen PV
> >>>   bus.
> >>
> >> Is VirtIO hotplug capable? Over PCI or MMIO transports or both?
> > 
> > VirtIO devices attached on a PCIe bus are hotpluggable, the emulated
> > PCIe bus itself would not have anything to do with virtio, except that
> > virtio devices can hang off of it.  AFAIU.
> 
> So network/block device only as memory mapped peripherals (like SMSC or PL
> SD/MMC) or over VirtIO-MMIO won't meet the specification? Is PCI/VirtIO-PCI on
> ARM production ready? What's the motivation for requiring hotplug?
> 

Platform devices that don't sit on any 'real bus' are generally not
hotpluggable.

VM management systems such as OpenStack make heavy use of hotplug to add
storage to your VMs, for example.

This spec does not prohibit devices over mmio, or over virtio-mmio, in
fact it encourages guest kernels to include support for such.  But it
mandates that there is some hotpluggable bus, so a very common VM use
case is supported for ARM VMs.

PCI for ARM is not ready yet, but people are working on it.  That should
not have any bearing on what the right decision for this spec is though
- it's all early stage at this point.

-Christoffer

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Wed, Feb 26, 2014 at 9:15 PM, Dennis Gilmore <dennis@gilmore.net.au> wrote:
> On Wed, 26 Feb 2014 11:56:53 -0800
> Christoffer Dall <christoffer.dall@linaro.org> wrote:
>
>> [why did you drop everyone from cc here?]
>
> standard reply to list behavior, I would appreciate if you followed it.

Not on the Linaro, infradead or vger lists. We preserve cc's here, always have.

>> On 26 February 2014 11:42, Dennis Gilmore <dennis@gilmore.net.au>
>> wrote:
>> > On Wed, 26 Feb 2014 10:34:54 -0800
>> > Christoffer Dall <christoffer.dall@linaro.org> wrote:
>> Also, I'm afraid "u-boot and look like an existing 32 bit system" is
>> not much of a spec. How does a distro vendor ship an image based on
>> that description that they can be sure will boot?
>
> based on the work to make a standard boot environment I have been
> working on, pass in the u-boot binary and things will work by loading
> config from inside the image and acting just like any system. really
> UEFI is major overkill here and a massive divergence from the real
> world. What is the argument that justifies the divergence?

That's what I used to say all the time until I actually looked at it.
It isn't the horrid monster that many of us feared it would be. There
is a fully open source implementation hosted on sourceforge which is
what I would expect most VM vendors to use directly. It isn't
unreasonably large and it implements sane behaviour.

Remember, we are talking about what is needed to make a portable VM
ecosystem. The folks working on the UEFI spec have spent a lot of time
thinking about how to choose what image to boot from a disk and the
spec is well defined in this regard. That aspect has not been U-Boot's
focus and U-Boot isn't anywhere near as mature as UEFI in that regard
(nor would I expect it to be; embedded has never had the same
incentive to create portable boot images as general purpose machines).

Also, specifying UEFI for this spec does not in any way prevent
someone from running U-Boot in their VM, or executing the kernel
directly. This spec is about a platform for portable images and it is
important to as much as possible specify things like firmware
interfaces without a whole lot of options. Other use-cases can freely
disregard the spec and run whatever they want.

>> Personally I think keeping things uniform across both 32-bit and
>> 64-bit is better, and the GTP/EFI image is a modern standard that
>> should work well.
>
> It means that installers will need special code paths to support being
> installed into virt guests and is not sustainable or supportable. as
> hardware wont work the same way.

Installers already have the EFI code paths, the kernel patches for
both 32 and 64 bit ARM are in-flight and will get merged soon. The
grub patches are done and merged. Installers will work exactly the
same way on real hardware with EFI and on VMs with EFI. It will also
work exactly the same way between x86, ARM and ARM64. What part is
unsustainable?

g.

[RFC] ARM VM System Sepcification

[Grant Likely]

On Wed, Feb 26, 2014 at 9:15 PM, Dennis Gilmore <dennis@gilmore.net.au> wrote:
> On Wed, 26 Feb 2014 11:56:53 -0800
> Christoffer Dall <christoffer.dall@linaro.org> wrote:
>
>> [why did you drop everyone from cc here?]
>
> standard reply to list behavior, I would appreciate if you followed it.

Not on the Linaro, infradead or vger lists. We preserve cc's here, always have.

>> On 26 February 2014 11:42, Dennis Gilmore <dennis@gilmore.net.au>
>> wrote:
>> > On Wed, 26 Feb 2014 10:34:54 -0800
>> > Christoffer Dall <christoffer.dall@linaro.org> wrote:
>> Also, I'm afraid "u-boot and look like an existing 32 bit system" is
>> not much of a spec. How does a distro vendor ship an image based on
>> that description that they can be sure will boot?
>
> based on the work to make a standard boot environment I have been
> working on, pass in the u-boot binary and things will work by loading
> config from inside the image and acting just like any system. really
> UEFI is major overkill here and a massive divergence from the real
> world. What is the argument that justifies the divergence?

That's what I used to say all the time until I actually looked at it.
It isn't the horrid monster that many of us feared it would be. There
is a fully open source implementation hosted on sourceforge which is
what I would expect most VM vendors to use directly. It isn't
unreasonably large and it implements sane behaviour.

Remember, we are talking about what is needed to make a portable VM
ecosystem. The folks working on the UEFI spec have spent a lot of time
thinking about how to choose what image to boot from a disk and the
spec is well defined in this regard. That aspect has not been U-Boot's
focus and U-Boot isn't anywhere near as mature as UEFI in that regard
(nor would I expect it to be; embedded has never had the same
incentive to create portable boot images as general purpose machines).

Also, specifying UEFI for this spec does not in any way prevent
someone from running U-Boot in their VM, or executing the kernel
directly. This spec is about a platform for portable images and it is
important to as much as possible specify things like firmware
interfaces without a whole lot of options. Other use-cases can freely
disregard the spec and run whatever they want.

>> Personally I think keeping things uniform across both 32-bit and
>> 64-bit is better, and the GTP/EFI image is a modern standard that
>> should work well.
>
> It means that installers will need special code paths to support being
> installed into virt guests and is not sustainable or supportable. as
> hardware wont work the same way.

Installers already have the EFI code paths, the kernel patches for
both 32 and 64 bit ARM are in-flight and will get merged soon. The
grub patches are done and merged. Installers will work exactly the
same way on real hardware with EFI and on VMs with EFI. It will also
work exactly the same way between x86, ARM and ARM64. What part is
unsustainable?

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Wed, Feb 26, 2014 at 9:15 PM, Dennis Gilmore <dennis@gilmore.net.au> wrote:
> On Wed, 26 Feb 2014 11:56:53 -0800
> Christoffer Dall <christoffer.dall@linaro.org> wrote:
>
>> [why did you drop everyone from cc here?]
>
> standard reply to list behavior, I would appreciate if you followed it.

Not on the Linaro, infradead or vger lists. We preserve cc's here, always have.

>> On 26 February 2014 11:42, Dennis Gilmore <dennis@gilmore.net.au>
>> wrote:
>> > On Wed, 26 Feb 2014 10:34:54 -0800
>> > Christoffer Dall <christoffer.dall@linaro.org> wrote:
>> Also, I'm afraid "u-boot and look like an existing 32 bit system" is
>> not much of a spec. How does a distro vendor ship an image based on
>> that description that they can be sure will boot?
>
> based on the work to make a standard boot environment I have been
> working on, pass in the u-boot binary and things will work by loading
> config from inside the image and acting just like any system. really
> UEFI is major overkill here and a massive divergence from the real
> world. What is the argument that justifies the divergence?

That's what I used to say all the time until I actually looked at it.
It isn't the horrid monster that many of us feared it would be. There
is a fully open source implementation hosted on sourceforge which is
what I would expect most VM vendors to use directly. It isn't
unreasonably large and it implements sane behaviour.

Remember, we are talking about what is needed to make a portable VM
ecosystem. The folks working on the UEFI spec have spent a lot of time
thinking about how to choose what image to boot from a disk and the
spec is well defined in this regard. That aspect has not been U-Boot's
focus and U-Boot isn't anywhere near as mature as UEFI in that regard
(nor would I expect it to be; embedded has never had the same
incentive to create portable boot images as general purpose machines).

Also, specifying UEFI for this spec does not in any way prevent
someone from running U-Boot in their VM, or executing the kernel
directly. This spec is about a platform for portable images and it is
important to as much as possible specify things like firmware
interfaces without a whole lot of options. Other use-cases can freely
disregard the spec and run whatever they want.

>> Personally I think keeping things uniform across both 32-bit and
>> 64-bit is better, and the GTP/EFI image is a modern standard that
>> should work well.
>
> It means that installers will need special code paths to support being
> installed into virt guests and is not sustainable or supportable. as
> hardware wont work the same way.

Installers already have the EFI code paths, the kernel patches for
both 32 and 64 bit ARM are in-flight and will get merged soon. The
grub patches are done and merged. Installers will work exactly the
same way on real hardware with EFI and on VMs with EFI. It will also
work exactly the same way between x86, ARM and ARM64. What part is
unsustainable?

g.

Re: [RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Thursday 27 February 2014 22:24:13 Alexander Graf wrote:
> 
> If you want to assign a platform device, you need to generate a respective
> hardware description (fdt/dsdt) chunk in the hypervisor. You can't take
> the host's description - it's too tightly coupled to the overall host layout.

But at that point, you need hardware specific drivers in both the hypervisor
and in the guest OS. When you have that, why do you still care about a
system specification? 

Going back to the previous argument, since the hypervisor has to make up the
description for the platform device itself, it won't matter whether the host
is booted using DT or ACPI: the description that the hypervisor makes up for
the guest has to match what the hypervisor uses to describe the rest of the
guest environment, which is independent of what the host uses.

> Imagine you get an AArch64 notebook with Windows on it. You want to run
> Linux there, so your host needs to understand ACPI. Now you want to run
> a Windows guest inside a VM, so you need ACPI in there again.

And you think that Windows is going to support a VM system specification
we are writing here? Booting Windows RT in a virtual machine is certainly
an interesting use case, but I think we will have to emulate a platform
that WinRT supports then, rather than expect it to run on ours.
 
> Replace Windows by "Linux with custom drivers" and you're in the same
> situation even when you neglect Windows. Reality will be that we will
> have fdt and acpi based systems.

We will however want to boot all sorts of guests in a standardized
virtual environment:

* 32 bit Linux (since some distros don't support biarch or multiarch
  on arm64) for running applications that are either binary-only
  or not 64-bit safe.
* 32-bit Android
* big-endian Linux for running applications that are not endian-clean
  (typically network stuff ported from powerpc or mipseb.
* OS/v guests
* NOMMU Linux
* BSD based OSs
* QNX
* random other RTOSs

Most of these will not work with ACPI, or at least not in 32-bit mode.
64-bit Linux will obviously support both DT (always) and ACPI (optionally),
depending on the platform, but for a specification like this, I think
it's much easier to support fewer options, to make it easier for other
guest OSs to ensure they actually run on any compliant hypervisor.

	Arnd

[RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Thursday 27 February 2014 22:24:13 Alexander Graf wrote:
> 
> If you want to assign a platform device, you need to generate a respective
> hardware description (fdt/dsdt) chunk in the hypervisor. You can't take
> the host's description - it's too tightly coupled to the overall host layout.

But at that point, you need hardware specific drivers in both the hypervisor
and in the guest OS. When you have that, why do you still care about a
system specification? 

Going back to the previous argument, since the hypervisor has to make up the
description for the platform device itself, it won't matter whether the host
is booted using DT or ACPI: the description that the hypervisor makes up for
the guest has to match what the hypervisor uses to describe the rest of the
guest environment, which is independent of what the host uses.

> Imagine you get an AArch64 notebook with Windows on it. You want to run
> Linux there, so your host needs to understand ACPI. Now you want to run
> a Windows guest inside a VM, so you need ACPI in there again.

And you think that Windows is going to support a VM system specification
we are writing here? Booting Windows RT in a virtual machine is certainly
an interesting use case, but I think we will have to emulate a platform
that WinRT supports then, rather than expect it to run on ours.
 
> Replace Windows by "Linux with custom drivers" and you're in the same
> situation even when you neglect Windows. Reality will be that we will
> have fdt and acpi based systems.

We will however want to boot all sorts of guests in a standardized
virtual environment:

* 32 bit Linux (since some distros don't support biarch or multiarch
  on arm64) for running applications that are either binary-only
  or not 64-bit safe.
* 32-bit Android
* big-endian Linux for running applications that are not endian-clean
  (typically network stuff ported from powerpc or mipseb.
* OS/v guests
* NOMMU Linux
* BSD based OSs
* QNX
* random other RTOSs

Most of these will not work with ACPI, or at least not in 32-bit mode.
64-bit Linux will obviously support both DT (always) and ACPI (optionally),
depending on the platform, but for a specification like this, I think
it's much easier to support fewer options, to make it easier for other
guest OSs to ensure they actually run on any compliant hypervisor.

	Arnd

Re: [RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Thursday 27 February 2014 22:24:13 Alexander Graf wrote:
> 
> If you want to assign a platform device, you need to generate a respective
> hardware description (fdt/dsdt) chunk in the hypervisor. You can't take
> the host's description - it's too tightly coupled to the overall host layout.

But at that point, you need hardware specific drivers in both the hypervisor
and in the guest OS. When you have that, why do you still care about a
system specification? 

Going back to the previous argument, since the hypervisor has to make up the
description for the platform device itself, it won't matter whether the host
is booted using DT or ACPI: the description that the hypervisor makes up for
the guest has to match what the hypervisor uses to describe the rest of the
guest environment, which is independent of what the host uses.

> Imagine you get an AArch64 notebook with Windows on it. You want to run
> Linux there, so your host needs to understand ACPI. Now you want to run
> a Windows guest inside a VM, so you need ACPI in there again.

And you think that Windows is going to support a VM system specification
we are writing here? Booting Windows RT in a virtual machine is certainly
an interesting use case, but I think we will have to emulate a platform
that WinRT supports then, rather than expect it to run on ours.
 
> Replace Windows by "Linux with custom drivers" and you're in the same
> situation even when you neglect Windows. Reality will be that we will
> have fdt and acpi based systems.

We will however want to boot all sorts of guests in a standardized
virtual environment:

* 32 bit Linux (since some distros don't support biarch or multiarch
  on arm64) for running applications that are either binary-only
  or not 64-bit safe.
* 32-bit Android
* big-endian Linux for running applications that are not endian-clean
  (typically network stuff ported from powerpc or mipseb.
* OS/v guests
* NOMMU Linux
* BSD based OSs
* QNX
* random other RTOSs

Most of these will not work with ACPI, or at least not in 32-bit mode.
64-bit Linux will obviously support both DT (always) and ACPI (optionally),
depending on the platform, but for a specification like this, I think
it's much easier to support fewer options, to make it easier for other
guest OSs to ensure they actually run on any compliant hypervisor.

	Arnd

Re: [RFC] ARM VM System Sepcification

[Alexander Graf]

> Am 28.02.2014 um 03:56 schrieb Arnd Bergmann <arnd@arndb.de>:
> 
>> On Thursday 27 February 2014 22:24:13 Alexander Graf wrote:
>> 
>> If you want to assign a platform device, you need to generate a respective
>> hardware description (fdt/dsdt) chunk in the hypervisor. You can't take
>> the host's description - it's too tightly coupled to the overall host layout.
> 
> But at that point, you need hardware specific drivers in both the hypervisor
> and in the guest OS.

In our case, you need hardware specific drivers in QEMU and the guest, correct.

> When you have that, why do you still care about a
> system specification? 

Because I don't want to go back to the system level definition. To me a peripheral is a peripheral - regardless of whether it is on a platform bus or a PCI bus. I want to leverage common ground and only add the few pieces that diverge from it.

> 
> Going back to the previous argument, since the hypervisor has to make up the
> description for the platform device itself, it won't matter whether the host
> is booted using DT or ACPI: the description that the hypervisor makes up for
> the guest has to match what the hypervisor uses to describe the rest of the
> guest environment, which is independent of what the host uses.

I agree 100%. This spec should be completely independent of the host.

The reason I brought up the host is that if you want to migrate an OS from physical to virtual, you may need to pass through devices to make this work. If your host driver developers only ever worked with ACPI, there's a good chance the drivers won't work in a pure dt environment.

Brw, the same argument applies the other way around as well. I don't believe we will get around with generating and mandating a single machibe description environment.

> 
>> Imagine you get an AArch64 notebook with Windows on it. You want to run
>> Linux there, so your host needs to understand ACPI. Now you want to run
>> a Windows guest inside a VM, so you need ACPI in there again.
> 
> And you think that Windows is going to support a VM system specification
> we are writing here? Booting Windows RT in a virtual machine is certainly
> an interesting use case, but I think we will have to emulate a platform
> that WinRT supports then, rather than expect it to run on ours.

Point taken :). Though it is a real shame we are in that situation in tge first place. X86 makes life a lot easier here.

> 
>> Replace Windows by "Linux with custom drivers" and you're in the same
>> situation even when you neglect Windows. Reality will be that we will
>> have fdt and acpi based systems.
> 
> We will however want to boot all sorts of guests in a standardized
> virtual environment:
> 
> * 32 bit Linux (since some distros don't support biarch or multiarch
>  on arm64) for running applications that are either binary-only
>  or not 64-bit safe.
> * 32-bit Android
> * big-endian Linux for running applications that are not endian-clean
>  (typically network stuff ported from powerpc or mipseb.
> * OS/v guests
> * NOMMU Linux
> * BSD based OSs
> * QNX
> * random other RTOSs
> 
> Most of these will not work with ACPI, or at least not in 32-bit mode.
> 64-bit Linux will obviously support both DT (always)

Unfortunately not

> and ACPI (optionally),
> depending on the platform, but for a specification like this, I think
> it's much easier to support fewer options, to make it easier for other
> guest OSs to ensure they actually run on any compliant hypervisor.

You're forgetting a few pretty important cases here:

* Enterprise grade Linux distribution that only supports ACPI
* Maybe WinRT if we can convince MS to use it
* Non-Linux with x86/ia64 heritage and thus ACPI support

If we want to run those, we need to expose ACPI tables.

Again, I think the only reasonable thing to do is to implement and expose both. That situation sucks, but we got into it ourselves ;).


Alex

[RFC] ARM VM System Sepcification

[Alexander Graf]

> Am 28.02.2014 um 03:56 schrieb Arnd Bergmann <arnd@arndb.de>:
> 
>> On Thursday 27 February 2014 22:24:13 Alexander Graf wrote:
>> 
>> If you want to assign a platform device, you need to generate a respective
>> hardware description (fdt/dsdt) chunk in the hypervisor. You can't take
>> the host's description - it's too tightly coupled to the overall host layout.
> 
> But at that point, you need hardware specific drivers in both the hypervisor
> and in the guest OS.

In our case, you need hardware specific drivers in QEMU and the guest, correct.

> When you have that, why do you still care about a
> system specification? 

Because I don't want to go back to the system level definition. To me a peripheral is a peripheral - regardless of whether it is on a platform bus or a PCI bus. I want to leverage common ground and only add the few pieces that diverge from it.

> 
> Going back to the previous argument, since the hypervisor has to make up the
> description for the platform device itself, it won't matter whether the host
> is booted using DT or ACPI: the description that the hypervisor makes up for
> the guest has to match what the hypervisor uses to describe the rest of the
> guest environment, which is independent of what the host uses.

I agree 100%. This spec should be completely independent of the host.

The reason I brought up the host is that if you want to migrate an OS from physical to virtual, you may need to pass through devices to make this work. If your host driver developers only ever worked with ACPI, there's a good chance the drivers won't work in a pure dt environment.

Brw, the same argument applies the other way around as well. I don't believe we will get around with generating and mandating a single machibe description environment.

> 
>> Imagine you get an AArch64 notebook with Windows on it. You want to run
>> Linux there, so your host needs to understand ACPI. Now you want to run
>> a Windows guest inside a VM, so you need ACPI in there again.
> 
> And you think that Windows is going to support a VM system specification
> we are writing here? Booting Windows RT in a virtual machine is certainly
> an interesting use case, but I think we will have to emulate a platform
> that WinRT supports then, rather than expect it to run on ours.

Point taken :). Though it is a real shame we are in that situation in tge first place. X86 makes life a lot easier here.

> 
>> Replace Windows by "Linux with custom drivers" and you're in the same
>> situation even when you neglect Windows. Reality will be that we will
>> have fdt and acpi based systems.
> 
> We will however want to boot all sorts of guests in a standardized
> virtual environment:
> 
> * 32 bit Linux (since some distros don't support biarch or multiarch
>  on arm64) for running applications that are either binary-only
>  or not 64-bit safe.
> * 32-bit Android
> * big-endian Linux for running applications that are not endian-clean
>  (typically network stuff ported from powerpc or mipseb.
> * OS/v guests
> * NOMMU Linux
> * BSD based OSs
> * QNX
> * random other RTOSs
> 
> Most of these will not work with ACPI, or at least not in 32-bit mode.
> 64-bit Linux will obviously support both DT (always)

Unfortunately not

> and ACPI (optionally),
> depending on the platform, but for a specification like this, I think
> it's much easier to support fewer options, to make it easier for other
> guest OSs to ensure they actually run on any compliant hypervisor.

You're forgetting a few pretty important cases here:

* Enterprise grade Linux distribution that only supports ACPI
* Maybe WinRT if we can convince MS to use it
* Non-Linux with x86/ia64 heritage and thus ACPI support

If we want to run those, we need to expose ACPI tables.

Again, I think the only reasonable thing to do is to implement and expose both. That situation sucks, but we got into it ourselves ;).


Alex

Re: [RFC] ARM VM System Sepcification

[Alexander Graf]

> Am 28.02.2014 um 03:56 schrieb Arnd Bergmann <arnd@arndb.de>:
> 
>> On Thursday 27 February 2014 22:24:13 Alexander Graf wrote:
>> 
>> If you want to assign a platform device, you need to generate a respective
>> hardware description (fdt/dsdt) chunk in the hypervisor. You can't take
>> the host's description - it's too tightly coupled to the overall host layout.
> 
> But at that point, you need hardware specific drivers in both the hypervisor
> and in the guest OS.

In our case, you need hardware specific drivers in QEMU and the guest, correct.

> When you have that, why do you still care about a
> system specification? 

Because I don't want to go back to the system level definition. To me a peripheral is a peripheral - regardless of whether it is on a platform bus or a PCI bus. I want to leverage common ground and only add the few pieces that diverge from it.

> 
> Going back to the previous argument, since the hypervisor has to make up the
> description for the platform device itself, it won't matter whether the host
> is booted using DT or ACPI: the description that the hypervisor makes up for
> the guest has to match what the hypervisor uses to describe the rest of the
> guest environment, which is independent of what the host uses.

I agree 100%. This spec should be completely independent of the host.

The reason I brought up the host is that if you want to migrate an OS from physical to virtual, you may need to pass through devices to make this work. If your host driver developers only ever worked with ACPI, there's a good chance the drivers won't work in a pure dt environment.

Brw, the same argument applies the other way around as well. I don't believe we will get around with generating and mandating a single machibe description environment.

> 
>> Imagine you get an AArch64 notebook with Windows on it. You want to run
>> Linux there, so your host needs to understand ACPI. Now you want to run
>> a Windows guest inside a VM, so you need ACPI in there again.
> 
> And you think that Windows is going to support a VM system specification
> we are writing here? Booting Windows RT in a virtual machine is certainly
> an interesting use case, but I think we will have to emulate a platform
> that WinRT supports then, rather than expect it to run on ours.

Point taken :). Though it is a real shame we are in that situation in tge first place. X86 makes life a lot easier here.

> 
>> Replace Windows by "Linux with custom drivers" and you're in the same
>> situation even when you neglect Windows. Reality will be that we will
>> have fdt and acpi based systems.
> 
> We will however want to boot all sorts of guests in a standardized
> virtual environment:
> 
> * 32 bit Linux (since some distros don't support biarch or multiarch
>  on arm64) for running applications that are either binary-only
>  or not 64-bit safe.
> * 32-bit Android
> * big-endian Linux for running applications that are not endian-clean
>  (typically network stuff ported from powerpc or mipseb.
> * OS/v guests
> * NOMMU Linux
> * BSD based OSs
> * QNX
> * random other RTOSs
> 
> Most of these will not work with ACPI, or at least not in 32-bit mode.
> 64-bit Linux will obviously support both DT (always)

Unfortunately not

> and ACPI (optionally),
> depending on the platform, but for a specification like this, I think
> it's much easier to support fewer options, to make it easier for other
> guest OSs to ensure they actually run on any compliant hypervisor.

You're forgetting a few pretty important cases here:

* Enterprise grade Linux distribution that only supports ACPI
* Maybe WinRT if we can convince MS to use it
* Non-Linux with x86/ia64 heritage and thus ACPI support

If we want to run those, we need to expose ACPI tables.

Again, I think the only reasonable thing to do is to implement and expose both. That situation sucks, but we got into it ourselves ;).


Alex

Re: [RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Friday 28 February 2014 08:05:15 Alexander Graf wrote: 
> > Am 28.02.2014 um 03:56 schrieb Arnd Bergmann <arnd-r2nGTMty4D4@public.gmane.org>:
> >> On Thursday 27 February 2014 22:24:13 Alexander Graf wrote:
>
> > When you have that, why do you still care about a
> > system specification? 
> 
> Because I don't want to go back to the system level definition. To me a 
> peripheral is a peripheral - regardless of whether it is on a platform 
> bus or a PCI bus. I want to leverage common ground and only add the few 
> pieces that diverge from it.

You may be missing a lot of the complexity that describing platform devices
in the general case brings then. To pass through an ethernet controller, you may 
also need to add (any subset of)

* phy device
* clock controller
* voltage regulator
* gpio controller
* LED controller
* DMA engine
* an MSI irqchip
* IOMMU

Each of the above in turn is shared with other peripherals on the host,
which brings you to three options:

* change the driver to not depend on the above, but instead support an
  abstract virtualized version of the platform device that doesn't
  need them.
* Pass through all devices this one depends on, giving up on guest
  isolation. This may work for some embedded use cases, but not
  for running untrusted guests.
* Implement virtualized versions of the other interfaces and make
  the hypervisor talk to the real hardware.

I would still argue that each of those approaches is out of scope for
this specification.

> > Going back to the previous argument, since the hypervisor has to make up the
> > description for the platform device itself, it won't matter whether the host
> > is booted using DT or ACPI: the description that the hypervisor makes up for
> > the guest has to match what the hypervisor uses to describe the rest of the
> > guest environment, which is independent of what the host uses.
> 
> I agree 100%. This spec should be completely independent of the host.
> 
> The reason I brought up the host is that if you want to migrate an OS from
> physical to virtual, you may need to pass through devices to make this work.
> If your host driver developers only ever worked with ACPI, there's a good
> chance the drivers won't work in a pure dt environment.
> 
> Brw, the same argument applies the other way around as well. I don't believe
> we will get around with generating and mandating a single machibe
> description environment.

I see those two cases as completely distinct. There are good reasons
for emulating a real machine for running a guest image that expects
certain hardware, and you can easily do this with qemu. But since you
are emulating an existing platform and run an existing OS, you don't
need a VM System Specification, you just do whatever the platform would
normally do that the guest relies on.

The VM system specification on the other hand should allow you to run
any OS that is written to support this specification on any hypervisor
that implements it.

> >> Replace Windows by "Linux with custom drivers" and you're in the same
> >> situation even when you neglect Windows. Reality will be that we will
> >> have fdt and acpi based systems.
> > 
> > We will however want to boot all sorts of guests in a standardized
> > virtual environment:
> > 
> > * 32 bit Linux (since some distros don't support biarch or multiarch
> >  on arm64) for running applications that are either binary-only
> >  or not 64-bit safe.
> > * 32-bit Android
> > * big-endian Linux for running applications that are not endian-clean
> >  (typically network stuff ported from powerpc or mipseb.
> > * OS/v guests
> > * NOMMU Linux
> > * BSD based OSs
> > * QNX
> > * random other RTOSs
> > 
> > Most of these will not work with ACPI, or at least not in 32-bit mode.
> > 64-bit Linux will obviously support both DT (always)
> 
> Unfortunately not
> 
> > and ACPI (optionally),
> > depending on the platform, but for a specification like this, I think
> > it's much easier to support fewer options, to make it easier for other
> > guest OSs to ensure they actually run on any compliant hypervisor.
> 
> You're forgetting a few pretty important cases here:
> 
> * Enterprise grade Linux distribution that only supports ACPI

You can't actually turn off DT support in the kernel, and I don't think
there is any point in patching the kernel to remove it. The only sane
thing the enterprise distros can do is turn on the "SBSA" platform
that supports all compliant machines running ACPI, but turn off all
platforms that are not SBSA compliant and boot using DT. With the way
that the VM spec is written at this point, Linux will still boot on
these guests, since the hardware support is a subset of SBSA, with the
addition of a few drivers for hypervisor specific features.

> * Maybe WinRT if we can convince MS to use it

I'd argue that would be unlikely.

> * Non-Linux with x86/ia64 heritage and thus ACPI support

That assumes that x86 ACPI support is anything like ARM64 ACPI
support, which it really isn't. In particular, you can turn off
ACPI on any x86 machine and it will still work for the most
part, while on ARM64 we will need to use ACPI to describe even
the most basic aspects of the platform.

	Arnd

[RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Friday 28 February 2014 08:05:15 Alexander Graf wrote: 
> > Am 28.02.2014 um 03:56 schrieb Arnd Bergmann <arnd@arndb.de>:
> >> On Thursday 27 February 2014 22:24:13 Alexander Graf wrote:
>
> > When you have that, why do you still care about a
> > system specification? 
> 
> Because I don't want to go back to the system level definition. To me a 
> peripheral is a peripheral - regardless of whether it is on a platform 
> bus or a PCI bus. I want to leverage common ground and only add the few 
> pieces that diverge from it.

You may be missing a lot of the complexity that describing platform devices
in the general case brings then. To pass through an ethernet controller, you may 
also need to add (any subset of)

* phy device
* clock controller
* voltage regulator
* gpio controller
* LED controller
* DMA engine
* an MSI irqchip
* IOMMU

Each of the above in turn is shared with other peripherals on the host,
which brings you to three options:

* change the driver to not depend on the above, but instead support an
  abstract virtualized version of the platform device that doesn't
  need them.
* Pass through all devices this one depends on, giving up on guest
  isolation. This may work for some embedded use cases, but not
  for running untrusted guests.
* Implement virtualized versions of the other interfaces and make
  the hypervisor talk to the real hardware.

I would still argue that each of those approaches is out of scope for
this specification.

> > Going back to the previous argument, since the hypervisor has to make up the
> > description for the platform device itself, it won't matter whether the host
> > is booted using DT or ACPI: the description that the hypervisor makes up for
> > the guest has to match what the hypervisor uses to describe the rest of the
> > guest environment, which is independent of what the host uses.
> 
> I agree 100%. This spec should be completely independent of the host.
> 
> The reason I brought up the host is that if you want to migrate an OS from
> physical to virtual, you may need to pass through devices to make this work.
> If your host driver developers only ever worked with ACPI, there's a good
> chance the drivers won't work in a pure dt environment.
> 
> Brw, the same argument applies the other way around as well. I don't believe
> we will get around with generating and mandating a single machibe
> description environment.

I see those two cases as completely distinct. There are good reasons
for emulating a real machine for running a guest image that expects
certain hardware, and you can easily do this with qemu. But since you
are emulating an existing platform and run an existing OS, you don't
need a VM System Specification, you just do whatever the platform would
normally do that the guest relies on.

The VM system specification on the other hand should allow you to run
any OS that is written to support this specification on any hypervisor
that implements it.

> >> Replace Windows by "Linux with custom drivers" and you're in the same
> >> situation even when you neglect Windows. Reality will be that we will
> >> have fdt and acpi based systems.
> > 
> > We will however want to boot all sorts of guests in a standardized
> > virtual environment:
> > 
> > * 32 bit Linux (since some distros don't support biarch or multiarch
> >  on arm64) for running applications that are either binary-only
> >  or not 64-bit safe.
> > * 32-bit Android
> > * big-endian Linux for running applications that are not endian-clean
> >  (typically network stuff ported from powerpc or mipseb.
> > * OS/v guests
> > * NOMMU Linux
> > * BSD based OSs
> > * QNX
> > * random other RTOSs
> > 
> > Most of these will not work with ACPI, or at least not in 32-bit mode.
> > 64-bit Linux will obviously support both DT (always)
> 
> Unfortunately not
> 
> > and ACPI (optionally),
> > depending on the platform, but for a specification like this, I think
> > it's much easier to support fewer options, to make it easier for other
> > guest OSs to ensure they actually run on any compliant hypervisor.
> 
> You're forgetting a few pretty important cases here:
> 
> * Enterprise grade Linux distribution that only supports ACPI

You can't actually turn off DT support in the kernel, and I don't think
there is any point in patching the kernel to remove it. The only sane
thing the enterprise distros can do is turn on the "SBSA" platform
that supports all compliant machines running ACPI, but turn off all
platforms that are not SBSA compliant and boot using DT. With the way
that the VM spec is written at this point, Linux will still boot on
these guests, since the hardware support is a subset of SBSA, with the
addition of a few drivers for hypervisor specific features.

> * Maybe WinRT if we can convince MS to use it

I'd argue that would be unlikely.

> * Non-Linux with x86/ia64 heritage and thus ACPI support

That assumes that x86 ACPI support is anything like ARM64 ACPI
support, which it really isn't. In particular, you can turn off
ACPI on any x86 machine and it will still work for the most
part, while on ARM64 we will need to use ACPI to describe even
the most basic aspects of the platform.

	Arnd

Re: [RFC] ARM VM System Sepcification

[Arnd Bergmann]

On Friday 28 February 2014 08:05:15 Alexander Graf wrote: 
> > Am 28.02.2014 um 03:56 schrieb Arnd Bergmann <arnd@arndb.de>:
> >> On Thursday 27 February 2014 22:24:13 Alexander Graf wrote:
>
> > When you have that, why do you still care about a
> > system specification? 
> 
> Because I don't want to go back to the system level definition. To me a 
> peripheral is a peripheral - regardless of whether it is on a platform 
> bus or a PCI bus. I want to leverage common ground and only add the few 
> pieces that diverge from it.

You may be missing a lot of the complexity that describing platform devices
in the general case brings then. To pass through an ethernet controller, you may 
also need to add (any subset of)

* phy device
* clock controller
* voltage regulator
* gpio controller
* LED controller
* DMA engine
* an MSI irqchip
* IOMMU

Each of the above in turn is shared with other peripherals on the host,
which brings you to three options:

* change the driver to not depend on the above, but instead support an
  abstract virtualized version of the platform device that doesn't
  need them.
* Pass through all devices this one depends on, giving up on guest
  isolation. This may work for some embedded use cases, but not
  for running untrusted guests.
* Implement virtualized versions of the other interfaces and make
  the hypervisor talk to the real hardware.

I would still argue that each of those approaches is out of scope for
this specification.

> > Going back to the previous argument, since the hypervisor has to make up the
> > description for the platform device itself, it won't matter whether the host
> > is booted using DT or ACPI: the description that the hypervisor makes up for
> > the guest has to match what the hypervisor uses to describe the rest of the
> > guest environment, which is independent of what the host uses.
> 
> I agree 100%. This spec should be completely independent of the host.
> 
> The reason I brought up the host is that if you want to migrate an OS from
> physical to virtual, you may need to pass through devices to make this work.
> If your host driver developers only ever worked with ACPI, there's a good
> chance the drivers won't work in a pure dt environment.
> 
> Brw, the same argument applies the other way around as well. I don't believe
> we will get around with generating and mandating a single machibe
> description environment.

I see those two cases as completely distinct. There are good reasons
for emulating a real machine for running a guest image that expects
certain hardware, and you can easily do this with qemu. But since you
are emulating an existing platform and run an existing OS, you don't
need a VM System Specification, you just do whatever the platform would
normally do that the guest relies on.

The VM system specification on the other hand should allow you to run
any OS that is written to support this specification on any hypervisor
that implements it.

> >> Replace Windows by "Linux with custom drivers" and you're in the same
> >> situation even when you neglect Windows. Reality will be that we will
> >> have fdt and acpi based systems.
> > 
> > We will however want to boot all sorts of guests in a standardized
> > virtual environment:
> > 
> > * 32 bit Linux (since some distros don't support biarch or multiarch
> >  on arm64) for running applications that are either binary-only
> >  or not 64-bit safe.
> > * 32-bit Android
> > * big-endian Linux for running applications that are not endian-clean
> >  (typically network stuff ported from powerpc or mipseb.
> > * OS/v guests
> > * NOMMU Linux
> > * BSD based OSs
> > * QNX
> > * random other RTOSs
> > 
> > Most of these will not work with ACPI, or at least not in 32-bit mode.
> > 64-bit Linux will obviously support both DT (always)
> 
> Unfortunately not
> 
> > and ACPI (optionally),
> > depending on the platform, but for a specification like this, I think
> > it's much easier to support fewer options, to make it easier for other
> > guest OSs to ensure they actually run on any compliant hypervisor.
> 
> You're forgetting a few pretty important cases here:
> 
> * Enterprise grade Linux distribution that only supports ACPI

You can't actually turn off DT support in the kernel, and I don't think
there is any point in patching the kernel to remove it. The only sane
thing the enterprise distros can do is turn on the "SBSA" platform
that supports all compliant machines running ACPI, but turn off all
platforms that are not SBSA compliant and boot using DT. With the way
that the VM spec is written at this point, Linux will still boot on
these guests, since the hardware support is a subset of SBSA, with the
addition of a few drivers for hypervisor specific features.

> * Maybe WinRT if we can convince MS to use it

I'd argue that would be unlikely.

> * Non-Linux with x86/ia64 heritage and thus ACPI support

That assumes that x86 ACPI support is anything like ARM64 ACPI
support, which it really isn't. In particular, you can turn off
ACPI on any x86 machine and it will still work for the most
part, while on ARM64 we will need to use ACPI to describe even
the most basic aspects of the platform.

	Arnd

Re: [RFC] ARM VM System Sepcification

[Riku Voipio]

[-- Attachment #1.1: Type: text/plain, Size: 1413 bytes --]

On 28 February 2014 02:05, Alexander Graf <agraf-l3A5Bk7waGM@public.gmane.org> wrote:
> > Am 28.02.2014 um 03:56 schrieb Arnd Bergmann <arnd-r2nGTMty4D4@public.gmane.org>
> >> Replace Windows by "Linux with custom drivers" and you're in the same
> >> situation even when you neglect Windows. Reality will be that we will
> >> have fdt and acpi based systems.

> > We will however want to boot all sorts of guests in a standardized
> > virtual environment:

> > * 32 bit Linux (since some distros don't support biarch or multiarch
> >  on arm64) for running applications that are either binary-only
> >  or not 64-bit safe.
> > * 32-bit Android
> > * big-endian Linux for running applications that are not endian-clean
> >  (typically network stuff ported from powerpc or mipseb.
> > * OS/v guests
> > * NOMMU Linux
> > * BSD based OSs
> > * QNX
> > * random other RTOSs

*snip*

> You're forgetting a few pretty important cases here:
>
> * Enterprise grade Linux distribution that only supports ACPI
> * Maybe WinRT if we can convince MS to use it
> * Non-Linux with x86/ia64 heritage and thus ACPI support

I think we need limit the scope of the spec a bit here.

For a this VM system specification, we should describe what is simple to
set up and high performance for KVM, Xen and mainline Linux. For everyone
else, there is Qemu where we can mangle to provide whatever the guest OS
might want.

Re UEFI,

Riku

[-- Attachment #1.2: Type: text/html, Size: 1823 bytes --]

[-- Attachment #2: Type: text/plain, Size: 181 bytes --]

_______________________________________________
cross-distro mailing list
cross-distro-cunTk1MwBs8s++Sfvej+rw@public.gmane.org
http://lists.linaro.org/mailman/listinfo/cross-distro

Re: [RFC] ARM VM System Sepcification

[Riku Voipio]

[-- Attachment #1.1: Type: text/plain, Size: 1372 bytes --]

On 28 February 2014 02:05, Alexander Graf <agraf@suse.de> wrote:
> > Am 28.02.2014 um 03:56 schrieb Arnd Bergmann <arnd@arndb.de>
> >> Replace Windows by "Linux with custom drivers" and you're in the same
> >> situation even when you neglect Windows. Reality will be that we will
> >> have fdt and acpi based systems.

> > We will however want to boot all sorts of guests in a standardized
> > virtual environment:

> > * 32 bit Linux (since some distros don't support biarch or multiarch
> >  on arm64) for running applications that are either binary-only
> >  or not 64-bit safe.
> > * 32-bit Android
> > * big-endian Linux for running applications that are not endian-clean
> >  (typically network stuff ported from powerpc or mipseb.
> > * OS/v guests
> > * NOMMU Linux
> > * BSD based OSs
> > * QNX
> > * random other RTOSs

*snip*

> You're forgetting a few pretty important cases here:
>
> * Enterprise grade Linux distribution that only supports ACPI
> * Maybe WinRT if we can convince MS to use it
> * Non-Linux with x86/ia64 heritage and thus ACPI support

I think we need limit the scope of the spec a bit here.

For a this VM system specification, we should describe what is simple to
set up and high performance for KVM, Xen and mainline Linux. For everyone
else, there is Qemu where we can mangle to provide whatever the guest OS
might want.

Re UEFI,

Riku

[-- Attachment #1.2: Type: text/html, Size: 1761 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] ARM VM System Sepcification

[Stefano Stabellini]

On Fri, 28 Feb 2014, Alexander Graf wrote:
> > We will however want to boot all sorts of guests in a standardized
> > virtual environment:
> > 
> > * 32 bit Linux (since some distros don't support biarch or multiarch
> >  on arm64) for running applications that are either binary-only
> >  or not 64-bit safe.
> > * 32-bit Android
> > * big-endian Linux for running applications that are not endian-clean
> >  (typically network stuff ported from powerpc or mipseb.
> > * OS/v guests
> > * NOMMU Linux
> > * BSD based OSs
> > * QNX
> > * random other RTOSs

8<---

> * Enterprise grade Linux distribution that only supports ACPI
> * Maybe WinRT if we can convince MS to use it
> * Non-Linux with x86/ia64 heritage and thus ACPI support
> 
> If we want to run those, we need to expose ACPI tables.
> 
> Again, I think the only reasonable thing to do is to implement and expose both. That situation sucks, but we got into it ourselves ;).

I think we should have a clear idea on the purpose of this doc: is it a
spec that we expect Linux and other guest OSes to comply to if they want
to run on KVM/Xen? Or is it a document that describes the state of the
world at the beginning of 2014?

If it is a spec, then we should simply ignore non-collaborative vendors
and their products. If we know in advance that they are not going to
comply to the spec, what's the point of trying to accommodate them here?
We can always carry our workarounds and hacks in the hypervisor if we
want to run their products as guests.

[RFC] ARM VM System Sepcification

[Stefano Stabellini]

On Fri, 28 Feb 2014, Alexander Graf wrote:
> > We will however want to boot all sorts of guests in a standardized
> > virtual environment:
> > 
> > * 32 bit Linux (since some distros don't support biarch or multiarch
> >  on arm64) for running applications that are either binary-only
> >  or not 64-bit safe.
> > * 32-bit Android
> > * big-endian Linux for running applications that are not endian-clean
> >  (typically network stuff ported from powerpc or mipseb.
> > * OS/v guests
> > * NOMMU Linux
> > * BSD based OSs
> > * QNX
> > * random other RTOSs

8<---

> * Enterprise grade Linux distribution that only supports ACPI
> * Maybe WinRT if we can convince MS to use it
> * Non-Linux with x86/ia64 heritage and thus ACPI support
> 
> If we want to run those, we need to expose ACPI tables.
> 
> Again, I think the only reasonable thing to do is to implement and expose both. That situation sucks, but we got into it ourselves ;).

I think we should have a clear idea on the purpose of this doc: is it a
spec that we expect Linux and other guest OSes to comply to if they want
to run on KVM/Xen? Or is it a document that describes the state of the
world at the beginning of 2014?

If it is a spec, then we should simply ignore non-collaborative vendors
and their products. If we know in advance that they are not going to
comply to the spec, what's the point of trying to accommodate them here?
We can always carry our workarounds and hacks in the hypervisor if we
want to run their products as guests.

Re: [RFC] ARM VM System Sepcification

[Stefano Stabellini]

On Fri, 28 Feb 2014, Alexander Graf wrote:
> > We will however want to boot all sorts of guests in a standardized
> > virtual environment:
> > 
> > * 32 bit Linux (since some distros don't support biarch or multiarch
> >  on arm64) for running applications that are either binary-only
> >  or not 64-bit safe.
> > * 32-bit Android
> > * big-endian Linux for running applications that are not endian-clean
> >  (typically network stuff ported from powerpc or mipseb.
> > * OS/v guests
> > * NOMMU Linux
> > * BSD based OSs
> > * QNX
> > * random other RTOSs

8<---

> * Enterprise grade Linux distribution that only supports ACPI
> * Maybe WinRT if we can convince MS to use it
> * Non-Linux with x86/ia64 heritage and thus ACPI support
> 
> If we want to run those, we need to expose ACPI tables.
> 
> Again, I think the only reasonable thing to do is to implement and expose both. That situation sucks, but we got into it ourselves ;).

I think we should have a clear idea on the purpose of this doc: is it a
spec that we expect Linux and other guest OSes to comply to if they want
to run on KVM/Xen? Or is it a document that describes the state of the
world at the beginning of 2014?

If it is a spec, then we should simply ignore non-collaborative vendors
and their products. If we know in advance that they are not going to
comply to the spec, what's the point of trying to accommodate them here?
We can always carry our workarounds and hacks in the hypervisor if we
want to run their products as guests.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

Hi Christoffer. I've got another comment on the text, this time about the format of the ESP:

On Wed, 26 Feb 2014 10:34:54 -0800, Christoffer Dall <christoffer.dall@linaro.org> wrote:
[...]
> Image format
> ------------
> The image format, as presented to the VM, needs to be well-defined in
> order for prepared disk images to be bootable across various
> virtualization implementations.
> 
> The raw disk format as presented to the VM must be partitioned with a
> GUID Partition Table (GPT).  The bootable software must be placed in the
> EFI System Partition (ESP), using the UEFI removable media path, and
> must be an EFI application complying to the UEFI Specification 2.4
> Revision A [6].
> 
> The ESP partition's GPT entry's partition type GUID must be
> C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> 
> The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> state.

I would also reference section 3.3 (Boot Option Variables Default Boot
Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
restate the meaning of the requirement in this spec, but the UEFI spec
is the authoritative source. Distributed VM disk images fall under the
same scenario as the firmware not having any valid boot variables.

g.

[RFC] ARM VM System Sepcification

[Grant Likely]

Hi Christoffer. I've got another comment on the text, this time about the format of the ESP:

On Wed, 26 Feb 2014 10:34:54 -0800, Christoffer Dall <christoffer.dall@linaro.org> wrote:
[...]
> Image format
> ------------
> The image format, as presented to the VM, needs to be well-defined in
> order for prepared disk images to be bootable across various
> virtualization implementations.
> 
> The raw disk format as presented to the VM must be partitioned with a
> GUID Partition Table (GPT).  The bootable software must be placed in the
> EFI System Partition (ESP), using the UEFI removable media path, and
> must be an EFI application complying to the UEFI Specification 2.4
> Revision A [6].
> 
> The ESP partition's GPT entry's partition type GUID must be
> C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> 
> The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> state.

I would also reference section 3.3 (Boot Option Variables Default Boot
Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
restate the meaning of the requirement in this spec, but the UEFI spec
is the authoritative source. Distributed VM disk images fall under the
same scenario as the firmware not having any valid boot variables.

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

Hi Christoffer. I've got another comment on the text, this time about the format of the ESP:

On Wed, 26 Feb 2014 10:34:54 -0800, Christoffer Dall <christoffer.dall@linaro.org> wrote:
[...]
> Image format
> ------------
> The image format, as presented to the VM, needs to be well-defined in
> order for prepared disk images to be bootable across various
> virtualization implementations.
> 
> The raw disk format as presented to the VM must be partitioned with a
> GUID Partition Table (GPT).  The bootable software must be placed in the
> EFI System Partition (ESP), using the UEFI removable media path, and
> must be an EFI application complying to the UEFI Specification 2.4
> Revision A [6].
> 
> The ESP partition's GPT entry's partition type GUID must be
> C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> 
> The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> state.

I would also reference section 3.3 (Boot Option Variables Default Boot
Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
restate the meaning of the requirement in this spec, but the UEFI spec
is the authoritative source. Distributed VM disk images fall under the
same scenario as the firmware not having any valid boot variables.

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Thu, 27 Feb 2014 08:12:35 -0500, Christopher Covington <cov@codeaurora.org> wrote:
> Hi Christoffer,
> 
> On 02/26/2014 02:51 PM, Christoffer Dall wrote:
> > On Wed, Feb 26, 2014 at 02:27:40PM -0500, Christopher Covington wrote:
> 
> >>> Image format
> >>> ------------
> >>> The image format, as presented to the VM, needs to be well-defined in
> >>> order for prepared disk images to be bootable across various
> >>> virtualization implementations.
> >>>
> >>> The raw disk format as presented to the VM must be partitioned with a
> >>> GUID Partition Table (GPT).  The bootable software must be placed in the
> >>> EFI System Partition (ESP), using the UEFI removable media path, and
> >>> must be an EFI application complying to the UEFI Specification 2.4
> >>> Revision A [6].
> >>>
> >>> The ESP partition's GPT entry's partition type GUID must be
> >>> C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> >>> formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> >>>
> >>> The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> >>> execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> >>> state.
> >>>
> >>> This ensures that tools for both Xen and KVM can load a binary UEFI
> >>> firmware which can read and boot the EFI application in the disk image.
> >>>
> >>> A typical scenario will be GRUB2 packaged as an EFI application, which
> >>> mounts the system boot partition and boots Linux.
> >>>
> >>>
> >>> Virtual Firmware
> >>> ----------------
> >>> The VM system must be able to boot the EFI application in the ESP.  It
> >>> is recommended that this is achieved by loading a UEFI binary as the
> >>> first software executed by the VM, which then executes the EFI
> >>> application.  The UEFI implementation should be compliant with UEFI
> >>> Specification 2.4 Revision A [6] or later.
> >>>
> >>> This document strongly recommends that the VM implementation supports
> >>> persistent environment storage for virtual firmware implementation in
> >>> order to ensure probable use cases such as adding additional disk images
> >>> to a VM or running installers to perform upgrades.
> >>>
> >>> The binary UEFI firmware implementation should not be distributed as
> >>> part of the VM image, but is specific to the VM implementation.
> >>
> >> Can you elaborate on the motivation for requiring that the kernel be stuffed
> >> into a disk image and for requiring such a heavyweight bootloader/firmware? By
> >> doing so you would seem to exclude those requiring an optimized boot process.

This spec doesn't exclude or prevent VMs from doing that if the user
wants to. It is about specifying the base requirements for a disk image
to be portable. Any disk image conforming to this spec should boot on
any VM conforming to the spec.

> > What's the alternative?  Shipping kernels externally and loading them
> > externally?  Sure you can do that, but then distros can't upgrade the
> > kernel themselves, and you have to come up with a convention for how to
> > ship kernels, initrd's etc.
> 
> The self-hosted upgrades use case makes sense. I can imagine using a
> pass-through or network filesystem to do it in the case of external loading,
> something like the following.

Network booting is actually something else that is already supported and
doesn't use the filesystem protocol at all. DHCP+TFTP to obtain the 2nd
stage loader (which can do whatever it wants) is the preferred way to do
things. The problem with trying to provide a network filesystem

Section 3.4.2 (Boot via LOAD_FILE_PROTOCOL) and 3.4.2.1 (Network
Booting) covers that scenario. Network boot uses most of the Preboot
eXecution Environment (PXE) spec except it retrieves an EFI executable
instead of a PXE executable. It is expected that PXE server will then
supply all the boot configuration to the client. The exact same thing
can be done with VMs... all of which is out of scope for this section
because it is talking about disk images!  :-)

g.

[RFC] ARM VM System Sepcification

[Grant Likely]

On Thu, 27 Feb 2014 08:12:35 -0500, Christopher Covington <cov@codeaurora.org> wrote:
> Hi Christoffer,
> 
> On 02/26/2014 02:51 PM, Christoffer Dall wrote:
> > On Wed, Feb 26, 2014 at 02:27:40PM -0500, Christopher Covington wrote:
> 
> >>> Image format
> >>> ------------
> >>> The image format, as presented to the VM, needs to be well-defined in
> >>> order for prepared disk images to be bootable across various
> >>> virtualization implementations.
> >>>
> >>> The raw disk format as presented to the VM must be partitioned with a
> >>> GUID Partition Table (GPT).  The bootable software must be placed in the
> >>> EFI System Partition (ESP), using the UEFI removable media path, and
> >>> must be an EFI application complying to the UEFI Specification 2.4
> >>> Revision A [6].
> >>>
> >>> The ESP partition's GPT entry's partition type GUID must be
> >>> C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> >>> formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> >>>
> >>> The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> >>> execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> >>> state.
> >>>
> >>> This ensures that tools for both Xen and KVM can load a binary UEFI
> >>> firmware which can read and boot the EFI application in the disk image.
> >>>
> >>> A typical scenario will be GRUB2 packaged as an EFI application, which
> >>> mounts the system boot partition and boots Linux.
> >>>
> >>>
> >>> Virtual Firmware
> >>> ----------------
> >>> The VM system must be able to boot the EFI application in the ESP.  It
> >>> is recommended that this is achieved by loading a UEFI binary as the
> >>> first software executed by the VM, which then executes the EFI
> >>> application.  The UEFI implementation should be compliant with UEFI
> >>> Specification 2.4 Revision A [6] or later.
> >>>
> >>> This document strongly recommends that the VM implementation supports
> >>> persistent environment storage for virtual firmware implementation in
> >>> order to ensure probable use cases such as adding additional disk images
> >>> to a VM or running installers to perform upgrades.
> >>>
> >>> The binary UEFI firmware implementation should not be distributed as
> >>> part of the VM image, but is specific to the VM implementation.
> >>
> >> Can you elaborate on the motivation for requiring that the kernel be stuffed
> >> into a disk image and for requiring such a heavyweight bootloader/firmware? By
> >> doing so you would seem to exclude those requiring an optimized boot process.

This spec doesn't exclude or prevent VMs from doing that if the user
wants to. It is about specifying the base requirements for a disk image
to be portable. Any disk image conforming to this spec should boot on
any VM conforming to the spec.

> > What's the alternative?  Shipping kernels externally and loading them
> > externally?  Sure you can do that, but then distros can't upgrade the
> > kernel themselves, and you have to come up with a convention for how to
> > ship kernels, initrd's etc.
> 
> The self-hosted upgrades use case makes sense. I can imagine using a
> pass-through or network filesystem to do it in the case of external loading,
> something like the following.

Network booting is actually something else that is already supported and
doesn't use the filesystem protocol at all. DHCP+TFTP to obtain the 2nd
stage loader (which can do whatever it wants) is the preferred way to do
things. The problem with trying to provide a network filesystem

Section 3.4.2 (Boot via LOAD_FILE_PROTOCOL) and 3.4.2.1 (Network
Booting) covers that scenario. Network boot uses most of the Preboot
eXecution Environment (PXE) spec except it retrieves an EFI executable
instead of a PXE executable. It is expected that PXE server will then
supply all the boot configuration to the client. The exact same thing
can be done with VMs... all of which is out of scope for this section
because it is talking about disk images!  :-)

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Wed, 26 Feb 2014 14:21:47 -0800, Christoffer Dall <christoffer.dall-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:
> On Wed, Feb 26, 2014 at 03:56:02PM -0600, Rob Herring wrote:
> > On Wed, Feb 26, 2014 at 2:22 PM, Arnd Bergmann <arnd-r2nGTMty4D4@public.gmane.org> wrote:
> > > On Wednesday 26 February 2014 12:05:37 Christoffer Dall wrote:
> > >> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> > >> > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> > >>
> > >> The most common response I've been getting so far is that people
> > >> generally want their VMs to look close to the real thing, but not sure
> > >> how valid an argument that is.
> > >>
> > >> Some people feel strongly about this and seem to think that ARMv8
> > >> kernels will only work with ACPI in the future...
> > >
> > > That is certainly a misconception that has caused a lot of trouble.
> > > We will certainly keep supporting FDT boot in ARMv8 indefinitely,
> > > and I expect that most systems will not use ACPI at all.
> > 
> > Furthermore, even enterprise distro kernels will boot DT based kernels
> > assuming the h/w support is mainlined despite statements to the
> > contrary. It is a requirement in mainline kernels that DT and ACPI
> > support to coexist. Distros are not going to go out of their way to
> > undo/break that. And since the boot interface is DT, you can't simply
> > turn off DT. :)
> > 
> 
> Personally I'm all for simplicity so I don't want to push any agenda for
> ACPI in VMs.
> 
> Note that the spec does not mandate the use of ACPI, it just tells you
> how to do it if you wish to.
> 
> But, we can change the spec to require full FDT description of the
> system, unless of course some of the ACPI-in-VM supporters manage to
> convince the rest.

Given that we don't even have a reliable view of what ACPI is going to
look like on hardware yet, it probably is appropriate to omit talking
about it entirely for v1.0.

... although let me walk through the implications of how that could be
done:

Option 1: If v1.0 requires VM provide FDT, and v2.0 requires VM provide
          either FDT or ACPI:
- v1.0 VM shall always provide an FDT
- v2.0 VM might provide ACPI without FDT
- v1.0 OS must accept FDT
- v2.0 OS must accept both ACPI and FDT
Implications:
  - a v1.0 OS might not boot on a v2.0 VM (because it cannot handle ACPI)
  - a v2.0 OS will always boot on both v1.0 and v2.0 VMs
  - ACPI-only OS not supported by this spec (Windows; potentially RHEL)
  - FDT-only OS not supported by v2.0 of the spec. If we're only talking
    Linux this isn't a problem, but it does put additional burden on
    anyone doing a domain-specific OS. (for example, a bare-metal
    networking application using virtualized devices)

Option 2: If v1.0 requires FDT, and v2.0 requires either FDT only or FDT+ACPI
- v1.0 and v2.0 VMs shall always provide and FDT
- v2.0 VMs may additionally provide ACPI
- v1.0 and v2.0 OSes must accept FDT
- No requirement for OS to accept ACPI
Impliciations:
  - a v1.0 OS will boot on a v1.0 and v2.0 VM (FDT is always provided)
  - a v2.0 OS will boot on a v1.0 and v2.0 VM
  - ACPI-only OS not supported by this spec for all configurations.
    There would need to be a stronger version of the spec (v2.0-ACPI) to
    specify the configuration usable by ACPI OSes.

Option 3: If v1.0 requires FDT, and v2.0 requires both FDT and ACPI
- v1.0 and v2.0 VMs shall always provide and FDT
- v2.0 VMs shall always provide ACPI
- v1.0 OSes must accept FDT
- v2.0 OS may accept either ACPI or FDT
Impliciations:
  - a v1.0 OS will boot on a v1.0 and v2.0 VM (FDT is always provided)
  - a v2.0 OS will boot on a v1.0 and v2.0 VM
  - Both FDT-only and ACPI-only OSes supported by v2.0 version of spec

Someone is going to be unhappy no matter what is chosen. I think it is
critical to be really clear about who the audience is. Doing the above
also highlights for me the cost adding either/or options to the spec.
Every 'or' option given to VMs adds cost to the OS. ie. if the spec
allows the VM to implement either ACPI or FDT, then a compliant OS must
support both. Alternately, if the spec allows the OS to implement only
ACPI or only FDT, then compliant VMs are forced to implement both. In
both cases it is a non-trivial burden (As far as I know, no ACPI-on-arm
work has been done for *BSD, and it is yet to be done for all the VMs).
The spec will be the most useful if as many options as possible are
eliminated.

Right now I think option 2 above makes the most sense; require FDT and
make ACPI an optional extension. That will support almost all Linux
vendors immediately and possibly even FreeBSD. As others have said,
Windows is a complete unknown and we have no idea if they will be
interested in the spec (nor is this the right forum for that
conversation, but I will bring it up with my contacts).

g.

[RFC] ARM VM System Sepcification

[Grant Likely]

On Wed, 26 Feb 2014 14:21:47 -0800, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> On Wed, Feb 26, 2014 at 03:56:02PM -0600, Rob Herring wrote:
> > On Wed, Feb 26, 2014 at 2:22 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> > > On Wednesday 26 February 2014 12:05:37 Christoffer Dall wrote:
> > >> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> > >> > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> > >>
> > >> The most common response I've been getting so far is that people
> > >> generally want their VMs to look close to the real thing, but not sure
> > >> how valid an argument that is.
> > >>
> > >> Some people feel strongly about this and seem to think that ARMv8
> > >> kernels will only work with ACPI in the future...
> > >
> > > That is certainly a misconception that has caused a lot of trouble.
> > > We will certainly keep supporting FDT boot in ARMv8 indefinitely,
> > > and I expect that most systems will not use ACPI at all.
> > 
> > Furthermore, even enterprise distro kernels will boot DT based kernels
> > assuming the h/w support is mainlined despite statements to the
> > contrary. It is a requirement in mainline kernels that DT and ACPI
> > support to coexist. Distros are not going to go out of their way to
> > undo/break that. And since the boot interface is DT, you can't simply
> > turn off DT. :)
> > 
> 
> Personally I'm all for simplicity so I don't want to push any agenda for
> ACPI in VMs.
> 
> Note that the spec does not mandate the use of ACPI, it just tells you
> how to do it if you wish to.
> 
> But, we can change the spec to require full FDT description of the
> system, unless of course some of the ACPI-in-VM supporters manage to
> convince the rest.

Given that we don't even have a reliable view of what ACPI is going to
look like on hardware yet, it probably is appropriate to omit talking
about it entirely for v1.0.

... although let me walk through the implications of how that could be
done:

Option 1: If v1.0 requires VM provide FDT, and v2.0 requires VM provide
          either FDT or ACPI:
- v1.0 VM shall always provide an FDT
- v2.0 VM might provide ACPI without FDT
- v1.0 OS must accept FDT
- v2.0 OS must accept both ACPI and FDT
Implications:
  - a v1.0 OS might not boot on a v2.0 VM (because it cannot handle ACPI)
  - a v2.0 OS will always boot on both v1.0 and v2.0 VMs
  - ACPI-only OS not supported by this spec (Windows; potentially RHEL)
  - FDT-only OS not supported by v2.0 of the spec. If we're only talking
    Linux this isn't a problem, but it does put additional burden on
    anyone doing a domain-specific OS. (for example, a bare-metal
    networking application using virtualized devices)

Option 2: If v1.0 requires FDT, and v2.0 requires either FDT only or FDT+ACPI
- v1.0 and v2.0 VMs shall always provide and FDT
- v2.0 VMs may additionally provide ACPI
- v1.0 and v2.0 OSes must accept FDT
- No requirement for OS to accept ACPI
Impliciations:
  - a v1.0 OS will boot on a v1.0 and v2.0 VM (FDT is always provided)
  - a v2.0 OS will boot on a v1.0 and v2.0 VM
  - ACPI-only OS not supported by this spec for all configurations.
    There would need to be a stronger version of the spec (v2.0-ACPI) to
    specify the configuration usable by ACPI OSes.

Option 3: If v1.0 requires FDT, and v2.0 requires both FDT and ACPI
- v1.0 and v2.0 VMs shall always provide and FDT
- v2.0 VMs shall always provide ACPI
- v1.0 OSes must accept FDT
- v2.0 OS may accept either ACPI or FDT
Impliciations:
  - a v1.0 OS will boot on a v1.0 and v2.0 VM (FDT is always provided)
  - a v2.0 OS will boot on a v1.0 and v2.0 VM
  - Both FDT-only and ACPI-only OSes supported by v2.0 version of spec

Someone is going to be unhappy no matter what is chosen. I think it is
critical to be really clear about who the audience is. Doing the above
also highlights for me the cost adding either/or options to the spec.
Every 'or' option given to VMs adds cost to the OS. ie. if the spec
allows the VM to implement either ACPI or FDT, then a compliant OS must
support both. Alternately, if the spec allows the OS to implement only
ACPI or only FDT, then compliant VMs are forced to implement both. In
both cases it is a non-trivial burden (As far as I know, no ACPI-on-arm
work has been done for *BSD, and it is yet to be done for all the VMs).
The spec will be the most useful if as many options as possible are
eliminated.

Right now I think option 2 above makes the most sense; require FDT and
make ACPI an optional extension. That will support almost all Linux
vendors immediately and possibly even FreeBSD. As others have said,
Windows is a complete unknown and we have no idea if they will be
interested in the spec (nor is this the right forum for that
conversation, but I will bring it up with my contacts).

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Wed, 26 Feb 2014 14:21:47 -0800, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> On Wed, Feb 26, 2014 at 03:56:02PM -0600, Rob Herring wrote:
> > On Wed, Feb 26, 2014 at 2:22 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> > > On Wednesday 26 February 2014 12:05:37 Christoffer Dall wrote:
> > >> On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> > >> > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> > >>
> > >> The most common response I've been getting so far is that people
> > >> generally want their VMs to look close to the real thing, but not sure
> > >> how valid an argument that is.
> > >>
> > >> Some people feel strongly about this and seem to think that ARMv8
> > >> kernels will only work with ACPI in the future...
> > >
> > > That is certainly a misconception that has caused a lot of trouble.
> > > We will certainly keep supporting FDT boot in ARMv8 indefinitely,
> > > and I expect that most systems will not use ACPI at all.
> > 
> > Furthermore, even enterprise distro kernels will boot DT based kernels
> > assuming the h/w support is mainlined despite statements to the
> > contrary. It is a requirement in mainline kernels that DT and ACPI
> > support to coexist. Distros are not going to go out of their way to
> > undo/break that. And since the boot interface is DT, you can't simply
> > turn off DT. :)
> > 
> 
> Personally I'm all for simplicity so I don't want to push any agenda for
> ACPI in VMs.
> 
> Note that the spec does not mandate the use of ACPI, it just tells you
> how to do it if you wish to.
> 
> But, we can change the spec to require full FDT description of the
> system, unless of course some of the ACPI-in-VM supporters manage to
> convince the rest.

Given that we don't even have a reliable view of what ACPI is going to
look like on hardware yet, it probably is appropriate to omit talking
about it entirely for v1.0.

... although let me walk through the implications of how that could be
done:

Option 1: If v1.0 requires VM provide FDT, and v2.0 requires VM provide
          either FDT or ACPI:
- v1.0 VM shall always provide an FDT
- v2.0 VM might provide ACPI without FDT
- v1.0 OS must accept FDT
- v2.0 OS must accept both ACPI and FDT
Implications:
  - a v1.0 OS might not boot on a v2.0 VM (because it cannot handle ACPI)
  - a v2.0 OS will always boot on both v1.0 and v2.0 VMs
  - ACPI-only OS not supported by this spec (Windows; potentially RHEL)
  - FDT-only OS not supported by v2.0 of the spec. If we're only talking
    Linux this isn't a problem, but it does put additional burden on
    anyone doing a domain-specific OS. (for example, a bare-metal
    networking application using virtualized devices)

Option 2: If v1.0 requires FDT, and v2.0 requires either FDT only or FDT+ACPI
- v1.0 and v2.0 VMs shall always provide and FDT
- v2.0 VMs may additionally provide ACPI
- v1.0 and v2.0 OSes must accept FDT
- No requirement for OS to accept ACPI
Impliciations:
  - a v1.0 OS will boot on a v1.0 and v2.0 VM (FDT is always provided)
  - a v2.0 OS will boot on a v1.0 and v2.0 VM
  - ACPI-only OS not supported by this spec for all configurations.
    There would need to be a stronger version of the spec (v2.0-ACPI) to
    specify the configuration usable by ACPI OSes.

Option 3: If v1.0 requires FDT, and v2.0 requires both FDT and ACPI
- v1.0 and v2.0 VMs shall always provide and FDT
- v2.0 VMs shall always provide ACPI
- v1.0 OSes must accept FDT
- v2.0 OS may accept either ACPI or FDT
Impliciations:
  - a v1.0 OS will boot on a v1.0 and v2.0 VM (FDT is always provided)
  - a v2.0 OS will boot on a v1.0 and v2.0 VM
  - Both FDT-only and ACPI-only OSes supported by v2.0 version of spec

Someone is going to be unhappy no matter what is chosen. I think it is
critical to be really clear about who the audience is. Doing the above
also highlights for me the cost adding either/or options to the spec.
Every 'or' option given to VMs adds cost to the OS. ie. if the spec
allows the VM to implement either ACPI or FDT, then a compliant OS must
support both. Alternately, if the spec allows the OS to implement only
ACPI or only FDT, then compliant VMs are forced to implement both. In
both cases it is a non-trivial burden (As far as I know, no ACPI-on-arm
work has been done for *BSD, and it is yet to be done for all the VMs).
The spec will be the most useful if as many options as possible are
eliminated.

Right now I think option 2 above makes the most sense; require FDT and
make ACPI an optional extension. That will support almost all Linux
vendors immediately and possibly even FreeBSD. As others have said,
Windows is a complete unknown and we have no idea if they will be
interested in the spec (nor is this the right forum for that
conversation, but I will bring it up with my contacts).

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Wed, 26 Feb 2014 14:25:53 -0800, Christoffer Dall <christoffer.dall-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:
> On Wed, Feb 26, 2014 at 09:48:43PM +0000, Leif Lindholm wrote:
> > On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> > > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> > > > ARM VM System Specification
> > > > ===========================
> > > > 
> > > > Goal
> > > > ----
> > > > The goal of this spec is to allow suitably-built OS images to run on
> > > > all ARM virtualization solutions, such as KVM or Xen.
> > > > 
> > > > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > > > they aim to be hypervisor agnostic.
> > > > 
> > > > Note that simply adhering to the SBSA [2] is not a valid approach,
> > > > for example because the SBSA mandates EL2, which will not be available
> > > > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > > > may be controversial for some ARM VM implementations to support.
> > > > This spec also covers the aarch32 execution mode, not covered in the
> > > > SBSA.
> > > 
> > > I would prefer if we can stay as close as possible to SBSA for individual
> > > hardware components, and only stray from it when there is a strong reason.
> > > pl011-subset doesn't sound like a significant problem to implement,
> > > especially as SBSA makes the DMA part of that optional. Can you
> > > elaborate on what hypervisor would have a problem with that?
> > 
> > I believe it comes down to how much extra overhead pl011-access-trap
> > would be over virtio-console. If low, then sure.
> > (Since there are certain things we cannot provide SBSA-compliant in
> > the guest anyway, I wouldn't consider lack of pl011 to be a big issue.)
> > 
> 
> I don't think it's about overhead, sure pl011 may be slower, but it's a
> serial port, does anyone care about performance of a console? pl011
> should be good enough for sure.

The reason pl011 is specified in the SBSA is to have a reliable output
port available from the first instruction. It is a debug feature.
Hardware vendors aren't even need to wire it up to a physical serial
port. It is just as valid to wire it to the RAS controller.

g.

[RFC] ARM VM System Sepcification

[Grant Likely]

On Wed, 26 Feb 2014 14:25:53 -0800, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> On Wed, Feb 26, 2014 at 09:48:43PM +0000, Leif Lindholm wrote:
> > On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> > > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> > > > ARM VM System Specification
> > > > ===========================
> > > > 
> > > > Goal
> > > > ----
> > > > The goal of this spec is to allow suitably-built OS images to run on
> > > > all ARM virtualization solutions, such as KVM or Xen.
> > > > 
> > > > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > > > they aim to be hypervisor agnostic.
> > > > 
> > > > Note that simply adhering to the SBSA [2] is not a valid approach,
> > > > for example because the SBSA mandates EL2, which will not be available
> > > > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > > > may be controversial for some ARM VM implementations to support.
> > > > This spec also covers the aarch32 execution mode, not covered in the
> > > > SBSA.
> > > 
> > > I would prefer if we can stay as close as possible to SBSA for individual
> > > hardware components, and only stray from it when there is a strong reason.
> > > pl011-subset doesn't sound like a significant problem to implement,
> > > especially as SBSA makes the DMA part of that optional. Can you
> > > elaborate on what hypervisor would have a problem with that?
> > 
> > I believe it comes down to how much extra overhead pl011-access-trap
> > would be over virtio-console. If low, then sure.
> > (Since there are certain things we cannot provide SBSA-compliant in
> > the guest anyway, I wouldn't consider lack of pl011 to be a big issue.)
> > 
> 
> I don't think it's about overhead, sure pl011 may be slower, but it's a
> serial port, does anyone care about performance of a console? pl011
> should be good enough for sure.

The reason pl011 is specified in the SBSA is to have a reliable output
port available from the first instruction. It is a debug feature.
Hardware vendors aren't even need to wire it up to a physical serial
port. It is just as valid to wire it to the RAS controller.

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Wed, 26 Feb 2014 14:25:53 -0800, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> On Wed, Feb 26, 2014 at 09:48:43PM +0000, Leif Lindholm wrote:
> > On Wed, Feb 26, 2014 at 08:55:58PM +0100, Arnd Bergmann wrote:
> > > On Wednesday 26 February 2014 10:34:54 Christoffer Dall wrote:
> > > > ARM VM System Specification
> > > > ===========================
> > > > 
> > > > Goal
> > > > ----
> > > > The goal of this spec is to allow suitably-built OS images to run on
> > > > all ARM virtualization solutions, such as KVM or Xen.
> > > > 
> > > > Recommendations in this spec are valid for aarch32 and aarch64 alike, and
> > > > they aim to be hypervisor agnostic.
> > > > 
> > > > Note that simply adhering to the SBSA [2] is not a valid approach,
> > > > for example because the SBSA mandates EL2, which will not be available
> > > > for VMs.  Further, the SBSA mandates peripherals like the pl011, which
> > > > may be controversial for some ARM VM implementations to support.
> > > > This spec also covers the aarch32 execution mode, not covered in the
> > > > SBSA.
> > > 
> > > I would prefer if we can stay as close as possible to SBSA for individual
> > > hardware components, and only stray from it when there is a strong reason.
> > > pl011-subset doesn't sound like a significant problem to implement,
> > > especially as SBSA makes the DMA part of that optional. Can you
> > > elaborate on what hypervisor would have a problem with that?
> > 
> > I believe it comes down to how much extra overhead pl011-access-trap
> > would be over virtio-console. If low, then sure.
> > (Since there are certain things we cannot provide SBSA-compliant in
> > the guest anyway, I wouldn't consider lack of pl011 to be a big issue.)
> > 
> 
> I don't think it's about overhead, sure pl011 may be slower, but it's a
> serial port, does anyone care about performance of a console? pl011
> should be good enough for sure.

The reason pl011 is specified in the SBSA is to have a reliable output
port available from the first instruction. It is a debug feature.
Hardware vendors aren't even need to wire it up to a physical serial
port. It is just as valid to wire it to the RAS controller.

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Thu, 27 Feb 2014 15:00:44 +0100, Arnd Bergmann <arnd@arndb.de> wrote:
> On Thursday 27 February 2014 12:31:55 Stefano Stabellini wrote:
> > On Wed, 26 Feb 2014, Leif Lindholm wrote:
> > > > >   no FDT.  In this case, the VM implementation must provide ACPI, and
> > > > >   the OS must be able to locate the ACPI root pointer through the UEFI
> > > > >   system table.
> > > > > 
> > > > > For more information about the arm and arm64 boot conventions, see
> > > > > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > > > > Linux kernel source tree.
> > > > > 
> > > > > For more information about UEFI and ACPI booting, see [4] and [5].
> > > > 
> > > > What's the point of having ACPI in a virtual machine? You wouldn't
> > > > need to abstract any of the hardware in AML since you already know
> > > > what the virtual hardware is, so I can't see how this would help
> > > > anyone.
> > > 
> > > The point is that if we need to share any real hw then we need to use
> > > whatever the host has.
> 
> I would be more comfortable defining in the spec that you cannot share
> hardware at all. Obviously that doesn't stop anyone from actually
> sharing hardware with the guest, but at that point it would become
> noncompliant with this spec, with the consequence that you couldn't
> expect a compliant guest image to run on that hardware, but that is
> exactly something we can't guarantee anyway because we don't know
> what drivers might be needed.

I don't think this spec should say *anything* about sharing hardware.
This spec is about producing portable disk images. Assigning hardware
into guests is rather orthogonal to whether or not hardware is assigned.

g.

[RFC] ARM VM System Sepcification

[Grant Likely]

On Thu, 27 Feb 2014 15:00:44 +0100, Arnd Bergmann <arnd@arndb.de> wrote:
> On Thursday 27 February 2014 12:31:55 Stefano Stabellini wrote:
> > On Wed, 26 Feb 2014, Leif Lindholm wrote:
> > > > >   no FDT.  In this case, the VM implementation must provide ACPI, and
> > > > >   the OS must be able to locate the ACPI root pointer through the UEFI
> > > > >   system table.
> > > > > 
> > > > > For more information about the arm and arm64 boot conventions, see
> > > > > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > > > > Linux kernel source tree.
> > > > > 
> > > > > For more information about UEFI and ACPI booting, see [4] and [5].
> > > > 
> > > > What's the point of having ACPI in a virtual machine? You wouldn't
> > > > need to abstract any of the hardware in AML since you already know
> > > > what the virtual hardware is, so I can't see how this would help
> > > > anyone.
> > > 
> > > The point is that if we need to share any real hw then we need to use
> > > whatever the host has.
> 
> I would be more comfortable defining in the spec that you cannot share
> hardware at all. Obviously that doesn't stop anyone from actually
> sharing hardware with the guest, but at that point it would become
> noncompliant with this spec, with the consequence that you couldn't
> expect a compliant guest image to run on that hardware, but that is
> exactly something we can't guarantee anyway because we don't know
> what drivers might be needed.

I don't think this spec should say *anything* about sharing hardware.
This spec is about producing portable disk images. Assigning hardware
into guests is rather orthogonal to whether or not hardware is assigned.

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Thu, 27 Feb 2014 15:00:44 +0100, Arnd Bergmann <arnd@arndb.de> wrote:
> On Thursday 27 February 2014 12:31:55 Stefano Stabellini wrote:
> > On Wed, 26 Feb 2014, Leif Lindholm wrote:
> > > > >   no FDT.  In this case, the VM implementation must provide ACPI, and
> > > > >   the OS must be able to locate the ACPI root pointer through the UEFI
> > > > >   system table.
> > > > > 
> > > > > For more information about the arm and arm64 boot conventions, see
> > > > > Documentation/arm/Booting and Documentation/arm64/booting.txt in the
> > > > > Linux kernel source tree.
> > > > > 
> > > > > For more information about UEFI and ACPI booting, see [4] and [5].
> > > > 
> > > > What's the point of having ACPI in a virtual machine? You wouldn't
> > > > need to abstract any of the hardware in AML since you already know
> > > > what the virtual hardware is, so I can't see how this would help
> > > > anyone.
> > > 
> > > The point is that if we need to share any real hw then we need to use
> > > whatever the host has.
> 
> I would be more comfortable defining in the spec that you cannot share
> hardware at all. Obviously that doesn't stop anyone from actually
> sharing hardware with the guest, but at that point it would become
> noncompliant with this spec, with the consequence that you couldn't
> expect a compliant guest image to run on that hardware, but that is
> exactly something we can't guarantee anyway because we don't know
> what drivers might be needed.

I don't think this spec should say *anything* about sharing hardware.
This spec is about producing portable disk images. Assigning hardware
into guests is rather orthogonal to whether or not hardware is assigned.

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

[-- Attachment #1: Type: text/plain, Size: 3687 bytes --]

On Thu, 27 Feb 2014 12:27:58 +0000, Stefano Stabellini <stefano.stabellini-mvvWK6WmYclDPfheJLI6IQ@public.gmane.org> wrote:
> On Wed, 26 Feb 2014, Grant Likely wrote:
> > > VM Platform
> > > -----------
> > > The specification does not mandate any specific memory map.  The guest
> > > OS must be able to enumerate all processing elements, devices, and
> > > memory through HW description data (FDT, ACPI) or a bus-specific
> > > mechanism such as PCI.
> > >
> > > The virtual platform must support at least one of the following ARM
> > > execution states:
> > >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> > >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> > >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> > >
> > > It is recommended to support both (2) and (3) on aarch64 capable
> > > physical systems.
> > >
> > > The virtual hardware platform must provide a number of mandatory
> > > peripherals:
> > >
> > >   Serial console:  The platform should provide a console,
> > >   based on an emulated pl011, a virtio-console, or a Xen PV console.
> > 
> > For portable disk image, can Xen PV be dropped from the list? pl011 is part of SBSA, and virtio is getting standardised, but Xen PV is
> > implementation specific.
> 
> Does an interface need OASIS' rubber stamp to be "standard"?  If so, we
> should also drop FDT from this document. The SBSA has not been published
> by any OASIS-like standardization body either, so maybe we should drop
> the SBSA too.
> 
> If it doesn't need OASIS nice logo on the side to be a standard, then
> the Xen PV interfaces are a standard too. Give a look at
> xen/include/public/io, they go as back as 2004, and they have multiple
> different implementations of the frontends and backends in multiple
> operating systems today.
> 
> There is no reason why another hypervisor couldn't implement the same
> interface and in fact I know for a fact that it was considered for KVM.

Allow me to elaborate. I'm not trying to punish Xen here, but I'm
deliberately pushing back against "either/or" options in the spec. In
this case the spec says the VM must implement one of pl011 *or* virtio
*or* xenpv. That gives lots of implementation choice to VM projects.

The downside is that spec-compliant OSes are required to implement
support for *all three*. This is a non-issue for Linux guests because all
those drivers are already there*, but it a real cost for niche guests.

The reason why a cut-dowm pl011 exists in SBSA is to provide a
bare-minimum output device that is always available. I would be far
happier for this spec to not give the VMs an option at all here. Now
that I think about it, it probably isn't appropriate to allow
virtio-console to be an option either. It should flat out require the
cut-down pl011 register interface. Make everything else optional. 'sane'
linux distros will enable virtio-console and xenpv and the kernel should
use whichever it finds in normal operation. Then no matter what crazy
guest the user wants to run there is still a known-safe fallback for
console output when things go wrong.

* aside from the footprint required to enable all of them

I've just had another thought. In the majority of cases the fallback
pl011 won't get used by the guest anyway unless asked to because the
UEFI OS Loader (EFI_STUB for Linux) will use the UEFI console drivers.
I expect the Xen UEFI port will use xenpv, and the kvm UEFI port will
use virtio. After UEFI exit boot services the OS is responsible for it's
own console output. If it has drivers for virtio-console or xenpv, then
fine; it discovers the interface and all is good. But if it doesn't,
then the fallback will at least work.

g.


[-- Attachment #2: Type: text/plain, Size: 181 bytes --]

_______________________________________________
cross-distro mailing list
cross-distro-cunTk1MwBs8s++Sfvej+rw@public.gmane.org
http://lists.linaro.org/mailman/listinfo/cross-distro

[RFC] ARM VM System Sepcification

[Grant Likely]

On Thu, 27 Feb 2014 12:27:58 +0000, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
> On Wed, 26 Feb 2014, Grant Likely wrote:
> > > VM Platform
> > > -----------
> > > The specification does not mandate any specific memory map. ??The guest
> > > OS must be able to enumerate all processing elements, devices, and
> > > memory through HW description data (FDT, ACPI) or a bus-specific
> > > mechanism such as PCI.
> > >
> > > The virtual platform must support at least one of the following ARM
> > > execution states:
> > > ?? (1) aarch32 virtual CPUs on aarch32 physical CPUs
> > > ?? (2) aarch32 virtual CPUs on aarch64 physical CPUs
> > > ?? (3) aarch64 virtual CPUs on aarch64 physical CPUs
> > >
> > > It is recommended to support both (2) and (3) on aarch64 capable
> > > physical systems.
> > >
> > > The virtual hardware platform must provide a number of mandatory
> > > peripherals:
> > >
> > > ?? Serial console: ??The platform should provide a console,
> > > ?? based on an emulated pl011, a virtio-console, or a Xen PV console.
> > 
> > For portable disk image, can Xen PV be dropped from the list? pl011 is part of SBSA, and virtio is getting standardised, but Xen PV is
> > implementation specific.
> 
> Does an interface need OASIS' rubber stamp to be "standard"?  If so, we
> should also drop FDT from this document. The SBSA has not been published
> by any OASIS-like standardization body either, so maybe we should drop
> the SBSA too.
> 
> If it doesn't need OASIS nice logo on the side to be a standard, then
> the Xen PV interfaces are a standard too. Give a look at
> xen/include/public/io, they go as back as 2004, and they have multiple
> different implementations of the frontends and backends in multiple
> operating systems today.
> 
> There is no reason why another hypervisor couldn't implement the same
> interface and in fact I know for a fact that it was considered for KVM.

Allow me to elaborate. I'm not trying to punish Xen here, but I'm
deliberately pushing back against "either/or" options in the spec. In
this case the spec says the VM must implement one of pl011 *or* virtio
*or* xenpv. That gives lots of implementation choice to VM projects.

The downside is that spec-compliant OSes are required to implement
support for *all three*. This is a non-issue for Linux guests because all
those drivers are already there*, but it a real cost for niche guests.

The reason why a cut-dowm pl011 exists in SBSA is to provide a
bare-minimum output device that is always available. I would be far
happier for this spec to not give the VMs an option at all here. Now
that I think about it, it probably isn't appropriate to allow
virtio-console to be an option either. It should flat out require the
cut-down pl011 register interface. Make everything else optional. 'sane'
linux distros will enable virtio-console and xenpv and the kernel should
use whichever it finds in normal operation. Then no matter what crazy
guest the user wants to run there is still a known-safe fallback for
console output when things go wrong.

* aside from the footprint required to enable all of them

I've just had another thought. In the majority of cases the fallback
pl011 won't get used by the guest anyway unless asked to because the
UEFI OS Loader (EFI_STUB for Linux) will use the UEFI console drivers.
I expect the Xen UEFI port will use xenpv, and the kvm UEFI port will
use virtio. After UEFI exit boot services the OS is responsible for it's
own console output. If it has drivers for virtio-console or xenpv, then
fine; it discovers the interface and all is good. But if it doesn't,
then the fallback will at least work.

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

[-- Attachment #1: Type: text/plain, Size: 3661 bytes --]

On Thu, 27 Feb 2014 12:27:58 +0000, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
> On Wed, 26 Feb 2014, Grant Likely wrote:
> > > VM Platform
> > > -----------
> > > The specification does not mandate any specific memory map.  The guest
> > > OS must be able to enumerate all processing elements, devices, and
> > > memory through HW description data (FDT, ACPI) or a bus-specific
> > > mechanism such as PCI.
> > >
> > > The virtual platform must support at least one of the following ARM
> > > execution states:
> > >   (1) aarch32 virtual CPUs on aarch32 physical CPUs
> > >   (2) aarch32 virtual CPUs on aarch64 physical CPUs
> > >   (3) aarch64 virtual CPUs on aarch64 physical CPUs
> > >
> > > It is recommended to support both (2) and (3) on aarch64 capable
> > > physical systems.
> > >
> > > The virtual hardware platform must provide a number of mandatory
> > > peripherals:
> > >
> > >   Serial console:  The platform should provide a console,
> > >   based on an emulated pl011, a virtio-console, or a Xen PV console.
> > 
> > For portable disk image, can Xen PV be dropped from the list? pl011 is part of SBSA, and virtio is getting standardised, but Xen PV is
> > implementation specific.
> 
> Does an interface need OASIS' rubber stamp to be "standard"?  If so, we
> should also drop FDT from this document. The SBSA has not been published
> by any OASIS-like standardization body either, so maybe we should drop
> the SBSA too.
> 
> If it doesn't need OASIS nice logo on the side to be a standard, then
> the Xen PV interfaces are a standard too. Give a look at
> xen/include/public/io, they go as back as 2004, and they have multiple
> different implementations of the frontends and backends in multiple
> operating systems today.
> 
> There is no reason why another hypervisor couldn't implement the same
> interface and in fact I know for a fact that it was considered for KVM.

Allow me to elaborate. I'm not trying to punish Xen here, but I'm
deliberately pushing back against "either/or" options in the spec. In
this case the spec says the VM must implement one of pl011 *or* virtio
*or* xenpv. That gives lots of implementation choice to VM projects.

The downside is that spec-compliant OSes are required to implement
support for *all three*. This is a non-issue for Linux guests because all
those drivers are already there*, but it a real cost for niche guests.

The reason why a cut-dowm pl011 exists in SBSA is to provide a
bare-minimum output device that is always available. I would be far
happier for this spec to not give the VMs an option at all here. Now
that I think about it, it probably isn't appropriate to allow
virtio-console to be an option either. It should flat out require the
cut-down pl011 register interface. Make everything else optional. 'sane'
linux distros will enable virtio-console and xenpv and the kernel should
use whichever it finds in normal operation. Then no matter what crazy
guest the user wants to run there is still a known-safe fallback for
console output when things go wrong.

* aside from the footprint required to enable all of them

I've just had another thought. In the majority of cases the fallback
pl011 won't get used by the guest anyway unless asked to because the
UEFI OS Loader (EFI_STUB for Linux) will use the UEFI console drivers.
I expect the Xen UEFI port will use xenpv, and the kvm UEFI port will
use virtio. After UEFI exit boot services the OS is responsible for it's
own console output. If it has drivers for virtio-console or xenpv, then
fine; it discovers the interface and all is good. But if it doesn't,
then the fallback will at least work.

g.


[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] ARM VM System Sepcification

[Peter Maydell]

On 1 March 2014 19:54, Grant Likely <grant.likely@linaro.org> wrote:
> Allow me to elaborate. I'm not trying to punish Xen here, but I'm
> deliberately pushing back against "either/or" options in the spec. In
> this case the spec says the VM must implement one of pl011 *or* virtio
> *or* xenpv. That gives lots of implementation choice to VM projects.

>From the VM implementation side, of course, we want to push back
against "must be X and only X" settings in the spec. My feeling is
that if you're a VM that doesn't already handle X then implementing
it is potentially quite a bit of work (especially if it implies implementing
a bus framework or whatever that you don't have any support for,
as might be the case for requiring virtio in Xen). On the other hand
Linux already has support for all the options and is easy to configure
with all the options enabled. So when I was looking through / making
comments on this spec my preference generally came down on the
side of "give the VM side the flexibility to select 1 from some small
number of N".

(Sometimes, as with GIC choices, the "flexibility must be in the
guest, not the VM" approach is forced by host h/w possibilities.)

For initial console in particular it might be reasonable to require
"minimal pl011". On the other hand I don't think we want to imply
"as a guest it's OK for you not to support Xen pv console" because
that is saying "console performance will be terrible on Xen". How
about "must implement minimal pl011, and also at least one of
virtio-console, interrupt-driven pl011 or xen pv console" ?

thanks
-- PMM

[RFC] ARM VM System Sepcification

[Peter Maydell]

On 1 March 2014 19:54, Grant Likely <grant.likely@linaro.org> wrote:
> Allow me to elaborate. I'm not trying to punish Xen here, but I'm
> deliberately pushing back against "either/or" options in the spec. In
> this case the spec says the VM must implement one of pl011 *or* virtio
> *or* xenpv. That gives lots of implementation choice to VM projects.

>From the VM implementation side, of course, we want to push back
against "must be X and only X" settings in the spec. My feeling is
that if you're a VM that doesn't already handle X then implementing
it is potentially quite a bit of work (especially if it implies implementing
a bus framework or whatever that you don't have any support for,
as might be the case for requiring virtio in Xen). On the other hand
Linux already has support for all the options and is easy to configure
with all the options enabled. So when I was looking through / making
comments on this spec my preference generally came down on the
side of "give the VM side the flexibility to select 1 from some small
number of N".

(Sometimes, as with GIC choices, the "flexibility must be in the
guest, not the VM" approach is forced by host h/w possibilities.)

For initial console in particular it might be reasonable to require
"minimal pl011". On the other hand I don't think we want to imply
"as a guest it's OK for you not to support Xen pv console" because
that is saying "console performance will be terrible on Xen". How
about "must implement minimal pl011, and also at least one of
virtio-console, interrupt-driven pl011 or xen pv console" ?

thanks
-- PMM

Re: [RFC] ARM VM System Sepcification

[Peter Maydell]

On 1 March 2014 19:54, Grant Likely <grant.likely@linaro.org> wrote:
> Allow me to elaborate. I'm not trying to punish Xen here, but I'm
> deliberately pushing back against "either/or" options in the spec. In
> this case the spec says the VM must implement one of pl011 *or* virtio
> *or* xenpv. That gives lots of implementation choice to VM projects.

>From the VM implementation side, of course, we want to push back
against "must be X and only X" settings in the spec. My feeling is
that if you're a VM that doesn't already handle X then implementing
it is potentially quite a bit of work (especially if it implies implementing
a bus framework or whatever that you don't have any support for,
as might be the case for requiring virtio in Xen). On the other hand
Linux already has support for all the options and is easy to configure
with all the options enabled. So when I was looking through / making
comments on this spec my preference generally came down on the
side of "give the VM side the flexibility to select 1 from some small
number of N".

(Sometimes, as with GIC choices, the "flexibility must be in the
guest, not the VM" approach is forced by host h/w possibilities.)

For initial console in particular it might be reasonable to require
"minimal pl011". On the other hand I don't think we want to imply
"as a guest it's OK for you not to support Xen pv console" because
that is saying "console performance will be terrible on Xen". How
about "must implement minimal pl011, and also at least one of
virtio-console, interrupt-driven pl011 or xen pv console" ?

thanks
-- PMM

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
> Hi Christoffer. I've got another comment on the text, this time about the format of the ESP:
> 
> On Wed, 26 Feb 2014 10:34:54 -0800, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> [...]
> > Image format
> > ------------
> > The image format, as presented to the VM, needs to be well-defined in
> > order for prepared disk images to be bootable across various
> > virtualization implementations.
> > 
> > The raw disk format as presented to the VM must be partitioned with a
> > GUID Partition Table (GPT).  The bootable software must be placed in the
> > EFI System Partition (ESP), using the UEFI removable media path, and
> > must be an EFI application complying to the UEFI Specification 2.4
> > Revision A [6].
> > 
> > The ESP partition's GPT entry's partition type GUID must be
> > C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> > formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> > 
> > The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> > execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> > state.
> 
> I would also reference section 3.3 (Boot Option Variables Default Boot
> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
> restate the meaning of the requirement in this spec, but the UEFI spec
> is the authoritative source. Distributed VM disk images fall under the
> same scenario as the firmware not having any valid boot variables.
> 
ack, thanks.

-Christoffer

[RFC] ARM VM System Sepcification

[Christoffer Dall]

On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
> Hi Christoffer. I've got another comment on the text, this time about the format of the ESP:
> 
> On Wed, 26 Feb 2014 10:34:54 -0800, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> [...]
> > Image format
> > ------------
> > The image format, as presented to the VM, needs to be well-defined in
> > order for prepared disk images to be bootable across various
> > virtualization implementations.
> > 
> > The raw disk format as presented to the VM must be partitioned with a
> > GUID Partition Table (GPT).  The bootable software must be placed in the
> > EFI System Partition (ESP), using the UEFI removable media path, and
> > must be an EFI application complying to the UEFI Specification 2.4
> > Revision A [6].
> > 
> > The ESP partition's GPT entry's partition type GUID must be
> > C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> > formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> > 
> > The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> > execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> > state.
> 
> I would also reference section 3.3 (Boot Option Variables Default Boot
> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
> restate the meaning of the requirement in this spec, but the UEFI spec
> is the authoritative source. Distributed VM disk images fall under the
> same scenario as the firmware not having any valid boot variables.
> 
ack, thanks.

-Christoffer

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
> Hi Christoffer. I've got another comment on the text, this time about the format of the ESP:
> 
> On Wed, 26 Feb 2014 10:34:54 -0800, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> [...]
> > Image format
> > ------------
> > The image format, as presented to the VM, needs to be well-defined in
> > order for prepared disk images to be bootable across various
> > virtualization implementations.
> > 
> > The raw disk format as presented to the VM must be partitioned with a
> > GUID Partition Table (GPT).  The bootable software must be placed in the
> > EFI System Partition (ESP), using the UEFI removable media path, and
> > must be an EFI application complying to the UEFI Specification 2.4
> > Revision A [6].
> > 
> > The ESP partition's GPT entry's partition type GUID must be
> > C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
> > formatted as FAT32/vfat as per Section 12.3.1.1 in [6].
> > 
> > The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
> > execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
> > state.
> 
> I would also reference section 3.3 (Boot Option Variables Default Boot
> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
> restate the meaning of the requirement in this spec, but the UEFI spec
> is the authoritative source. Distributed VM disk images fall under the
> same scenario as the firmware not having any valid boot variables.
> 
ack, thanks.

-Christoffer

Re: [RFC] ARM VM System Sepcification

[Robie Basak]

[-- Attachment #1: Type: text/plain, Size: 1630 bytes --]

On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
> I would also reference section 3.3 (Boot Option Variables Default Boot
> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
> restate the meaning of the requirement in this spec, but the UEFI spec
> is the authoritative source. Distributed VM disk images fall under the
> same scenario as the firmware not having any valid boot variables.

What happens when the VM is first booted without boot variables, but
then the OS expects to be able to set boot variables and see them on
next boot?

AIUI, we don't have an implementation of this right now, and even if we
did, there are implications for persistent storage of this data further
up the stack (a required implementation in libvirt, OpenStack nova
providing a storage area for it, etc).

If possible, I would prefer to mandate that the host implementation is
permitted to no-op (or otherwise disable) boot variable write operations
altogether to avoid having to deal with this. In the common case, I
don't see why an OS installation shipped via a VM disk image would need
to write boot variables anyway.

Would there be any adverse consequences to doing this?

My reason is that this would save us from blocking a general OpenStack
implementation on ARM by requiring that these pieces are implemented
further up the stack first, when it would bring actual gain to doing so.

This would not preclude host implementations from implementing writeable
variables, or guests from using them. Just that for a _portable VM disk
image_, the OS on it cannot assume that this functionality is present.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[RFC] ARM VM System Sepcification

[Robie Basak]

On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
> I would also reference section 3.3 (Boot Option Variables Default Boot
> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
> restate the meaning of the requirement in this spec, but the UEFI spec
> is the authoritative source. Distributed VM disk images fall under the
> same scenario as the firmware not having any valid boot variables.

What happens when the VM is first booted without boot variables, but
then the OS expects to be able to set boot variables and see them on
next boot?

AIUI, we don't have an implementation of this right now, and even if we
did, there are implications for persistent storage of this data further
up the stack (a required implementation in libvirt, OpenStack nova
providing a storage area for it, etc).

If possible, I would prefer to mandate that the host implementation is
permitted to no-op (or otherwise disable) boot variable write operations
altogether to avoid having to deal with this. In the common case, I
don't see why an OS installation shipped via a VM disk image would need
to write boot variables anyway.

Would there be any adverse consequences to doing this?

My reason is that this would save us from blocking a general OpenStack
implementation on ARM by requiring that these pieces are implemented
further up the stack first, when it would bring actual gain to doing so.

This would not preclude host implementations from implementing writeable
variables, or guests from using them. Just that for a _portable VM disk
image_, the OS on it cannot assume that this functionality is present.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140306/6de98c9d/attachment-0001.sig>

Re: [RFC] ARM VM System Sepcification

[Robie Basak]

[-- Attachment #1.1: Type: text/plain, Size: 1630 bytes --]

On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
> I would also reference section 3.3 (Boot Option Variables Default Boot
> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
> restate the meaning of the requirement in this spec, but the UEFI spec
> is the authoritative source. Distributed VM disk images fall under the
> same scenario as the firmware not having any valid boot variables.

What happens when the VM is first booted without boot variables, but
then the OS expects to be able to set boot variables and see them on
next boot?

AIUI, we don't have an implementation of this right now, and even if we
did, there are implications for persistent storage of this data further
up the stack (a required implementation in libvirt, OpenStack nova
providing a storage area for it, etc).

If possible, I would prefer to mandate that the host implementation is
permitted to no-op (or otherwise disable) boot variable write operations
altogether to avoid having to deal with this. In the common case, I
don't see why an OS installation shipped via a VM disk image would need
to write boot variables anyway.

Would there be any adverse consequences to doing this?

My reason is that this would save us from blocking a general OpenStack
implementation on ARM by requiring that these pieces are implemented
further up the stack first, when it would bring actual gain to doing so.

This would not preclude host implementations from implementing writeable
variables, or guests from using them. Just that for a _portable VM disk
image_, the OS on it cannot assume that this functionality is present.

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 06/03/2014 09:52, Robie Basak ha scritto:
> On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
>> I would also reference section 3.3 (Boot Option Variables Default Boot
>> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
>> restate the meaning of the requirement in this spec, but the UEFI spec
>> is the authoritative source. Distributed VM disk images fall under the
>> same scenario as the firmware not having any valid boot variables.
>
> What happens when the VM is first booted without boot variables, but
> then the OS expects to be able to set boot variables and see them on
> next boot?

UEFI scans the devices; looks for an EFI system partition on the disks; 
and builds a default boot order.

> If possible, I would prefer to mandate that the host implementation is
> permitted to no-op (or otherwise disable) boot variable write operations
> altogether to avoid having to deal with this. In the common case, I
> don't see why an OS installation shipped via a VM disk image would need
> to write boot variables anyway.
>
> Would there be any adverse consequences to doing this?

Given the experience on x86 UEFI, no.

Unlike bare metal, it is common to run UEFI VMs without persistent flash 
storage.  In this case the boot variables and boot order are rebuilt on 
the fly on every boot, and it just works for both Windows and Linux; 
there's no reason why it should be any different for ARM.

> My reason is that this would save us from blocking a general OpenStack
> implementation on ARM by requiring that these pieces are implemented
> further up the stack first, when it would bring actual gain to doing so.
>
> This would not preclude host implementations from implementing writeable
> variables, or guests from using them. Just that for a _portable VM disk
> image_, the OS on it cannot assume that this functionality is present.

This is already the case for most OSes.  Otherwise you wouldn't be able 
to move a hard disk from a (physical) machine to another.

I strongly suggest that you take a look at the work done in Tiano Core's 
OvmfPkg, which has support for almost every QEMU feature thanks to the 
work of Laszlo Ersek and Jordan Justen.

In particular, OvmfPkg has support for specifying a boot order in the VM 
configuration (which maps to the "-boot" option in QEMU).  In this case, 
the UEFI boot order is overridden by a variable that is placed in some 
architecture-specific firmware configuration mechanism (on x86 we have 
one called fw_cfg, on ARM you could look at the fdt).  This predates 
UEFI and is not a UEFI variable; in fact is is a list of OpenFirmware 
device paths.  UEFI will match the OF paths to UEFI paths, and use the 
result to build a UEFI boot order.

Paolo

[RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 06/03/2014 09:52, Robie Basak ha scritto:
> On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
>> I would also reference section 3.3 (Boot Option Variables Default Boot
>> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
>> restate the meaning of the requirement in this spec, but the UEFI spec
>> is the authoritative source. Distributed VM disk images fall under the
>> same scenario as the firmware not having any valid boot variables.
>
> What happens when the VM is first booted without boot variables, but
> then the OS expects to be able to set boot variables and see them on
> next boot?

UEFI scans the devices; looks for an EFI system partition on the disks; 
and builds a default boot order.

> If possible, I would prefer to mandate that the host implementation is
> permitted to no-op (or otherwise disable) boot variable write operations
> altogether to avoid having to deal with this. In the common case, I
> don't see why an OS installation shipped via a VM disk image would need
> to write boot variables anyway.
>
> Would there be any adverse consequences to doing this?

Given the experience on x86 UEFI, no.

Unlike bare metal, it is common to run UEFI VMs without persistent flash 
storage.  In this case the boot variables and boot order are rebuilt on 
the fly on every boot, and it just works for both Windows and Linux; 
there's no reason why it should be any different for ARM.

> My reason is that this would save us from blocking a general OpenStack
> implementation on ARM by requiring that these pieces are implemented
> further up the stack first, when it would bring actual gain to doing so.
>
> This would not preclude host implementations from implementing writeable
> variables, or guests from using them. Just that for a _portable VM disk
> image_, the OS on it cannot assume that this functionality is present.

This is already the case for most OSes.  Otherwise you wouldn't be able 
to move a hard disk from a (physical) machine to another.

I strongly suggest that you take a look at the work done in Tiano Core's 
OvmfPkg, which has support for almost every QEMU feature thanks to the 
work of Laszlo Ersek and Jordan Justen.

In particular, OvmfPkg has support for specifying a boot order in the VM 
configuration (which maps to the "-boot" option in QEMU).  In this case, 
the UEFI boot order is overridden by a variable that is placed in some 
architecture-specific firmware configuration mechanism (on x86 we have 
one called fw_cfg, on ARM you could look at the fdt).  This predates 
UEFI and is not a UEFI variable; in fact is is a list of OpenFirmware 
device paths.  UEFI will match the OF paths to UEFI paths, and use the 
result to build a UEFI boot order.

Paolo

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 06/03/2014 09:52, Robie Basak ha scritto:
> On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
>> I would also reference section 3.3 (Boot Option Variables Default Boot
>> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
>> restate the meaning of the requirement in this spec, but the UEFI spec
>> is the authoritative source. Distributed VM disk images fall under the
>> same scenario as the firmware not having any valid boot variables.
>
> What happens when the VM is first booted without boot variables, but
> then the OS expects to be able to set boot variables and see them on
> next boot?

UEFI scans the devices; looks for an EFI system partition on the disks; 
and builds a default boot order.

> If possible, I would prefer to mandate that the host implementation is
> permitted to no-op (or otherwise disable) boot variable write operations
> altogether to avoid having to deal with this. In the common case, I
> don't see why an OS installation shipped via a VM disk image would need
> to write boot variables anyway.
>
> Would there be any adverse consequences to doing this?

Given the experience on x86 UEFI, no.

Unlike bare metal, it is common to run UEFI VMs without persistent flash 
storage.  In this case the boot variables and boot order are rebuilt on 
the fly on every boot, and it just works for both Windows and Linux; 
there's no reason why it should be any different for ARM.

> My reason is that this would save us from blocking a general OpenStack
> implementation on ARM by requiring that these pieces are implemented
> further up the stack first, when it would bring actual gain to doing so.
>
> This would not preclude host implementations from implementing writeable
> variables, or guests from using them. Just that for a _portable VM disk
> image_, the OS on it cannot assume that this functionality is present.

This is already the case for most OSes.  Otherwise you wouldn't be able 
to move a hard disk from a (physical) machine to another.

I strongly suggest that you take a look at the work done in Tiano Core's 
OvmfPkg, which has support for almost every QEMU feature thanks to the 
work of Laszlo Ersek and Jordan Justen.

In particular, OvmfPkg has support for specifying a boot order in the VM 
configuration (which maps to the "-boot" option in QEMU).  In this case, 
the UEFI boot order is overridden by a variable that is placed in some 
architecture-specific firmware configuration mechanism (on x86 we have 
one called fw_cfg, on ARM you could look at the fdt).  This predates 
UEFI and is not a UEFI variable; in fact is is a list of OpenFirmware 
device paths.  UEFI will match the OF paths to UEFI paths, and use the 
result to build a UEFI boot order.

Paolo

Re: [RFC] ARM VM System Sepcification

[Laszlo Ersek]

On 03/06/14 10:46, Paolo Bonzini wrote:
> Il 06/03/2014 09:52, Robie Basak ha scritto:
>> On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
>>> I would also reference section 3.3 (Boot Option Variables Default Boot
>>> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
>>> restate the meaning of the requirement in this spec, but the UEFI spec
>>> is the authoritative source. Distributed VM disk images fall under the
>>> same scenario as the firmware not having any valid boot variables.

"+1"

>>
>> What happens when the VM is first booted without boot variables, but
>> then the OS expects to be able to set boot variables and see them on
>> next boot?
> 
> UEFI scans the devices; looks for an EFI system partition on the disks;
> and builds a default boot order.
> 
>> If possible, I would prefer to mandate that the host implementation is
>> permitted to no-op (or otherwise disable) boot variable write operations
>> altogether to avoid having to deal with this. In the common case, I
>> don't see why an OS installation shipped via a VM disk image would need
>> to write boot variables anyway.
>>
>> Would there be any adverse consequences to doing this?
> 
> Given the experience on x86 UEFI, no.
> 
> Unlike bare metal, it is common to run UEFI VMs without persistent flash
> storage.  In this case the boot variables and boot order are rebuilt on
> the fly on every boot, and it just works for both Windows and Linux;
> there's no reason why it should be any different for ARM.
> 
>> My reason is that this would save us from blocking a general OpenStack
>> implementation on ARM by requiring that these pieces are implemented
>> further up the stack first, when it would bring actual gain to doing so.
>>
>> This would not preclude host implementations from implementing writeable
>> variables, or guests from using them. Just that for a _portable VM disk
>> image_, the OS on it cannot assume that this functionality is present.
> 
> This is already the case for most OSes.  Otherwise you wouldn't be able
> to move a hard disk from a (physical) machine to another.
> 
> I strongly suggest that you take a look at the work done in Tiano Core's
> OvmfPkg, which has support for almost every QEMU feature thanks to the
> work of Laszlo Ersek and Jordan Justen.
> 
> In particular, OvmfPkg has support for specifying a boot order in the VM
> configuration (which maps to the "-boot" option in QEMU).  In this case,
> the UEFI boot order is overridden by a variable that is placed in some
> architecture-specific firmware configuration mechanism (on x86 we have
> one called fw_cfg, on ARM you could look at the fdt).  This predates
> UEFI and is not a UEFI variable; in fact is is a list of OpenFirmware
> device paths.  UEFI will match the OF paths to UEFI paths, and use the
> result to build a UEFI boot order.

If I understand correctly, the question is this:

  Given a hypervisor that doesn't support non-volatile UEFI variables
  (including BootOrder and Boot####), is it possible to automatically
  boot a carefully prepared VM image, made available as a fixed media
  device?

The answer is "yes". See

  3.3 Boot Option Variables Default Boot Behavior

in the UEFI spec (already referenced by Grant Likely in the context above).

  3.3 Boot Option Variables Default Boot Behavior

  The default state of globally-defined variables is firmware vendor
  specific. However the boot options require a standard default behavior
  in the exceptional case that valid boot options are not present on a
  platform. The default behavior must be invoked any time the BootOrder
  variable does not exist or only points to nonexistent boot options.

  If no valid boot options exist, the boot manager will enumerate all
  removable media devices followed by all fixed media devices. The order
  within each group is undefined. These new default boot options are not
  saved to non volatile storage. The boot manger will then attempt to
  boot from each boot option. If the device supports the
  EFI_SIMPLE_FILE_SYSTEM_PROTOCOL then the removable media boot behavior
  (see Section 3.4.1.1) is executed. Otherwise, the firmware will
  attempt to boot the device via the EFI_LOAD_FILE_PROTOCOL.

  It is expected that this default boot will load an operating system or
  a maintenance utility. If this is an operating system setup program it
  is then responsible for setting the requisite environment variables
  for subsequent boots. The platform firmware may also decide to recover
  or set to a known set of boot options.

Basically, the "removable media boot behavior" applies to fixed media
devices as last resort. You can prepare a disk image where this behavior
will "simply" boot the OS.

See also Peter Jones' blog post about this:

http://blog.uncooperative.org/blog/2014/02/06/the-efi-system-partition/

(In short, my email is a "+1" to what Grant said.)

Thanks
Laszlo

[RFC] ARM VM System Sepcification

[Laszlo Ersek]

On 03/06/14 10:46, Paolo Bonzini wrote:
> Il 06/03/2014 09:52, Robie Basak ha scritto:
>> On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
>>> I would also reference section 3.3 (Boot Option Variables Default Boot
>>> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
>>> restate the meaning of the requirement in this spec, but the UEFI spec
>>> is the authoritative source. Distributed VM disk images fall under the
>>> same scenario as the firmware not having any valid boot variables.

"+1"

>>
>> What happens when the VM is first booted without boot variables, but
>> then the OS expects to be able to set boot variables and see them on
>> next boot?
> 
> UEFI scans the devices; looks for an EFI system partition on the disks;
> and builds a default boot order.
> 
>> If possible, I would prefer to mandate that the host implementation is
>> permitted to no-op (or otherwise disable) boot variable write operations
>> altogether to avoid having to deal with this. In the common case, I
>> don't see why an OS installation shipped via a VM disk image would need
>> to write boot variables anyway.
>>
>> Would there be any adverse consequences to doing this?
> 
> Given the experience on x86 UEFI, no.
> 
> Unlike bare metal, it is common to run UEFI VMs without persistent flash
> storage.  In this case the boot variables and boot order are rebuilt on
> the fly on every boot, and it just works for both Windows and Linux;
> there's no reason why it should be any different for ARM.
> 
>> My reason is that this would save us from blocking a general OpenStack
>> implementation on ARM by requiring that these pieces are implemented
>> further up the stack first, when it would bring actual gain to doing so.
>>
>> This would not preclude host implementations from implementing writeable
>> variables, or guests from using them. Just that for a _portable VM disk
>> image_, the OS on it cannot assume that this functionality is present.
> 
> This is already the case for most OSes.  Otherwise you wouldn't be able
> to move a hard disk from a (physical) machine to another.
> 
> I strongly suggest that you take a look at the work done in Tiano Core's
> OvmfPkg, which has support for almost every QEMU feature thanks to the
> work of Laszlo Ersek and Jordan Justen.
> 
> In particular, OvmfPkg has support for specifying a boot order in the VM
> configuration (which maps to the "-boot" option in QEMU).  In this case,
> the UEFI boot order is overridden by a variable that is placed in some
> architecture-specific firmware configuration mechanism (on x86 we have
> one called fw_cfg, on ARM you could look at the fdt).  This predates
> UEFI and is not a UEFI variable; in fact is is a list of OpenFirmware
> device paths.  UEFI will match the OF paths to UEFI paths, and use the
> result to build a UEFI boot order.

If I understand correctly, the question is this:

  Given a hypervisor that doesn't support non-volatile UEFI variables
  (including BootOrder and Boot####), is it possible to automatically
  boot a carefully prepared VM image, made available as a fixed media
  device?

The answer is "yes". See

  3.3 Boot Option Variables Default Boot Behavior

in the UEFI spec (already referenced by Grant Likely in the context above).

  3.3 Boot Option Variables Default Boot Behavior

  The default state of globally-defined variables is firmware vendor
  specific. However the boot options require a standard default behavior
  in the exceptional case that valid boot options are not present on a
  platform. The default behavior must be invoked any time the BootOrder
  variable does not exist or only points to nonexistent boot options.

  If no valid boot options exist, the boot manager will enumerate all
  removable media devices followed by all fixed media devices. The order
  within each group is undefined. These new default boot options are not
  saved to non volatile storage. The boot manger will then attempt to
  boot from each boot option. If the device supports the
  EFI_SIMPLE_FILE_SYSTEM_PROTOCOL then the removable media boot behavior
  (see Section 3.4.1.1) is executed. Otherwise, the firmware will
  attempt to boot the device via the EFI_LOAD_FILE_PROTOCOL.

  It is expected that this default boot will load an operating system or
  a maintenance utility. If this is an operating system setup program it
  is then responsible for setting the requisite environment variables
  for subsequent boots. The platform firmware may also decide to recover
  or set to a known set of boot options.

Basically, the "removable media boot behavior" applies to fixed media
devices as last resort. You can prepare a disk image where this behavior
will "simply" boot the OS.

See also Peter Jones' blog post about this:

http://blog.uncooperative.org/blog/2014/02/06/the-efi-system-partition/

(In short, my email is a "+1" to what Grant said.)

Thanks
Laszlo

Re: [RFC] ARM VM System Sepcification

[Laszlo Ersek]

On 03/06/14 10:46, Paolo Bonzini wrote:
> Il 06/03/2014 09:52, Robie Basak ha scritto:
>> On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
>>> I would also reference section 3.3 (Boot Option Variables Default Boot
>>> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
>>> restate the meaning of the requirement in this spec, but the UEFI spec
>>> is the authoritative source. Distributed VM disk images fall under the
>>> same scenario as the firmware not having any valid boot variables.

"+1"

>>
>> What happens when the VM is first booted without boot variables, but
>> then the OS expects to be able to set boot variables and see them on
>> next boot?
> 
> UEFI scans the devices; looks for an EFI system partition on the disks;
> and builds a default boot order.
> 
>> If possible, I would prefer to mandate that the host implementation is
>> permitted to no-op (or otherwise disable) boot variable write operations
>> altogether to avoid having to deal with this. In the common case, I
>> don't see why an OS installation shipped via a VM disk image would need
>> to write boot variables anyway.
>>
>> Would there be any adverse consequences to doing this?
> 
> Given the experience on x86 UEFI, no.
> 
> Unlike bare metal, it is common to run UEFI VMs without persistent flash
> storage.  In this case the boot variables and boot order are rebuilt on
> the fly on every boot, and it just works for both Windows and Linux;
> there's no reason why it should be any different for ARM.
> 
>> My reason is that this would save us from blocking a general OpenStack
>> implementation on ARM by requiring that these pieces are implemented
>> further up the stack first, when it would bring actual gain to doing so.
>>
>> This would not preclude host implementations from implementing writeable
>> variables, or guests from using them. Just that for a _portable VM disk
>> image_, the OS on it cannot assume that this functionality is present.
> 
> This is already the case for most OSes.  Otherwise you wouldn't be able
> to move a hard disk from a (physical) machine to another.
> 
> I strongly suggest that you take a look at the work done in Tiano Core's
> OvmfPkg, which has support for almost every QEMU feature thanks to the
> work of Laszlo Ersek and Jordan Justen.
> 
> In particular, OvmfPkg has support for specifying a boot order in the VM
> configuration (which maps to the "-boot" option in QEMU).  In this case,
> the UEFI boot order is overridden by a variable that is placed in some
> architecture-specific firmware configuration mechanism (on x86 we have
> one called fw_cfg, on ARM you could look at the fdt).  This predates
> UEFI and is not a UEFI variable; in fact is is a list of OpenFirmware
> device paths.  UEFI will match the OF paths to UEFI paths, and use the
> result to build a UEFI boot order.

If I understand correctly, the question is this:

  Given a hypervisor that doesn't support non-volatile UEFI variables
  (including BootOrder and Boot####), is it possible to automatically
  boot a carefully prepared VM image, made available as a fixed media
  device?

The answer is "yes". See

  3.3 Boot Option Variables Default Boot Behavior

in the UEFI spec (already referenced by Grant Likely in the context above).

  3.3 Boot Option Variables Default Boot Behavior

  The default state of globally-defined variables is firmware vendor
  specific. However the boot options require a standard default behavior
  in the exceptional case that valid boot options are not present on a
  platform. The default behavior must be invoked any time the BootOrder
  variable does not exist or only points to nonexistent boot options.

  If no valid boot options exist, the boot manager will enumerate all
  removable media devices followed by all fixed media devices. The order
  within each group is undefined. These new default boot options are not
  saved to non volatile storage. The boot manger will then attempt to
  boot from each boot option. If the device supports the
  EFI_SIMPLE_FILE_SYSTEM_PROTOCOL then the removable media boot behavior
  (see Section 3.4.1.1) is executed. Otherwise, the firmware will
  attempt to boot the device via the EFI_LOAD_FILE_PROTOCOL.

  It is expected that this default boot will load an operating system or
  a maintenance utility. If this is an operating system setup program it
  is then responsible for setting the requisite environment variables
  for subsequent boots. The platform firmware may also decide to recover
  or set to a known set of boot options.

Basically, the "removable media boot behavior" applies to fixed media
devices as last resort. You can prepare a disk image where this behavior
will "simply" boot the OS.

See also Peter Jones' blog post about this:

http://blog.uncooperative.org/blog/2014/02/06/the-efi-system-partition/

(In short, my email is a "+1" to what Grant said.)

Thanks
Laszlo

Re: [RFC] ARM VM System Sepcification

[Robie Basak]

[-- Attachment #1.1: Type: text/plain, Size: 1797 bytes --]

On Thu, Mar 06, 2014 at 12:44:57PM +0100, Laszlo Ersek wrote:
> If I understand correctly, the question is this:
> 
>   Given a hypervisor that doesn't support non-volatile UEFI variables
>   (including BootOrder and Boot####), is it possible to automatically
>   boot a carefully prepared VM image, made available as a fixed media
>   device?
> 
> The answer is "yes". See

Right, but I think there is a subsequent problem.

>   It is expected that this default boot will load an operating system or
>   a maintenance utility. If this is an operating system setup program it
>   is then responsible for setting the requisite environment variables
>   for subsequent boots. The platform firmware may also decide to recover
        ^^^^^^^^^^^^^^^^
>   or set to a known set of boot options.

It seems to me that the guest OS is permitted to assume that persistent
boot variables will work after first boot, for subsequent boots.

So, for example, the guest OS might, on bootloader or kernel upgrade,
completely replace the boot mechanism, dropping the removable path and
replacing it with a fixed disk arrangement, setting boot variables
appropriately, and assume that it can reboot and everything will
continue to work.

But if the host does not support non-volatile variables, then this will
break.

This is why I'm suggesting that the specification mandate that the guest
OS shipped in a "portable disk image" as defined by the spec must not
make this assumption.

It's either this, or mandate that all hosts must support persistent
variables. I have no objection to that in principle, but since we have
no implementation currently, it seems easier to avoid this particular
roadblock by tweaking the spec in a way that nobody seems to care about
anyway.

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC] ARM VM System Sepcification

[Robie Basak]

On Thu, Mar 06, 2014 at 12:44:57PM +0100, Laszlo Ersek wrote:
> If I understand correctly, the question is this:
> 
>   Given a hypervisor that doesn't support non-volatile UEFI variables
>   (including BootOrder and Boot####), is it possible to automatically
>   boot a carefully prepared VM image, made available as a fixed media
>   device?
> 
> The answer is "yes". See

Right, but I think there is a subsequent problem.

>   It is expected that this default boot will load an operating system or
>   a maintenance utility. If this is an operating system setup program it
>   is then responsible for setting the requisite environment variables
>   for subsequent boots. The platform firmware may also decide to recover
        ^^^^^^^^^^^^^^^^
>   or set to a known set of boot options.

It seems to me that the guest OS is permitted to assume that persistent
boot variables will work after first boot, for subsequent boots.

So, for example, the guest OS might, on bootloader or kernel upgrade,
completely replace the boot mechanism, dropping the removable path and
replacing it with a fixed disk arrangement, setting boot variables
appropriately, and assume that it can reboot and everything will
continue to work.

But if the host does not support non-volatile variables, then this will
break.

This is why I'm suggesting that the specification mandate that the guest
OS shipped in a "portable disk image" as defined by the spec must not
make this assumption.

It's either this, or mandate that all hosts must support persistent
variables. I have no objection to that in principle, but since we have
no implementation currently, it seems easier to avoid this particular
roadblock by tweaking the spec in a way that nobody seems to care about
anyway.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140306/72149884/attachment-0001.sig>

Re: [RFC] ARM VM System Sepcification

[Robie Basak]

[-- Attachment #1.1: Type: text/plain, Size: 1797 bytes --]

On Thu, Mar 06, 2014 at 12:44:57PM +0100, Laszlo Ersek wrote:
> If I understand correctly, the question is this:
> 
>   Given a hypervisor that doesn't support non-volatile UEFI variables
>   (including BootOrder and Boot####), is it possible to automatically
>   boot a carefully prepared VM image, made available as a fixed media
>   device?
> 
> The answer is "yes". See

Right, but I think there is a subsequent problem.

>   It is expected that this default boot will load an operating system or
>   a maintenance utility. If this is an operating system setup program it
>   is then responsible for setting the requisite environment variables
>   for subsequent boots. The platform firmware may also decide to recover
        ^^^^^^^^^^^^^^^^
>   or set to a known set of boot options.

It seems to me that the guest OS is permitted to assume that persistent
boot variables will work after first boot, for subsequent boots.

So, for example, the guest OS might, on bootloader or kernel upgrade,
completely replace the boot mechanism, dropping the removable path and
replacing it with a fixed disk arrangement, setting boot variables
appropriately, and assume that it can reboot and everything will
continue to work.

But if the host does not support non-volatile variables, then this will
break.

This is why I'm suggesting that the specification mandate that the guest
OS shipped in a "portable disk image" as defined by the spec must not
make this assumption.

It's either this, or mandate that all hosts must support persistent
variables. I have no objection to that in principle, but since we have
no implementation currently, it seems easier to avoid this particular
roadblock by tweaking the spec in a way that nobody seems to care about
anyway.

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 06/03/2014 13:04, Robie Basak ha scritto:
> So, for example, the guest OS might, on bootloader or kernel upgrade,
> completely replace the boot mechanism, dropping the removable path and
> replacing it with a fixed disk arrangement, setting boot variables
> appropriately, and assume that it can reboot and everything will
> continue to work.

It can, but at the same time it had better keep the fallback method 
working, where fixed media are treated like removable ones.

In practice, both Linux distributions and Windows do that.

> This is why I'm suggesting that the specification mandate that the guest
> OS shipped in a "portable disk image" as defined by the spec must not
> make this assumption.

Yeah, that's fine to specify.

Paolo

[RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 06/03/2014 13:04, Robie Basak ha scritto:
> So, for example, the guest OS might, on bootloader or kernel upgrade,
> completely replace the boot mechanism, dropping the removable path and
> replacing it with a fixed disk arrangement, setting boot variables
> appropriately, and assume that it can reboot and everything will
> continue to work.

It can, but at the same time it had better keep the fallback method 
working, where fixed media are treated like removable ones.

In practice, both Linux distributions and Windows do that.

> This is why I'm suggesting that the specification mandate that the guest
> OS shipped in a "portable disk image" as defined by the spec must not
> make this assumption.

Yeah, that's fine to specify.

Paolo

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 06/03/2014 13:04, Robie Basak ha scritto:
> So, for example, the guest OS might, on bootloader or kernel upgrade,
> completely replace the boot mechanism, dropping the removable path and
> replacing it with a fixed disk arrangement, setting boot variables
> appropriately, and assume that it can reboot and everything will
> continue to work.

It can, but at the same time it had better keep the fallback method 
working, where fixed media are treated like removable ones.

In practice, both Linux distributions and Windows do that.

> This is why I'm suggesting that the specification mandate that the guest
> OS shipped in a "portable disk image" as defined by the spec must not
> make this assumption.

Yeah, that's fine to specify.

Paolo

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Thu, 6 Mar 2014 08:52:13 +0000, Robie Basak <robie.basak@canonical.com> wrote:
> On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
> > I would also reference section 3.3 (Boot Option Variables Default Boot
> > Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
> > restate the meaning of the requirement in this spec, but the UEFI spec
> > is the authoritative source. Distributed VM disk images fall under the
> > same scenario as the firmware not having any valid boot variables.
> 
> What happens when the VM is first booted without boot variables, but
> then the OS expects to be able to set boot variables and see them on
> next boot?
> 
> AIUI, we don't have an implementation of this right now, and even if we
> did, there are implications for persistent storage of this data further
> up the stack (a required implementation in libvirt, OpenStack nova
> providing a storage area for it, etc).
> 
> If possible, I would prefer to mandate that the host implementation is
> permitted to no-op (or otherwise disable) boot variable write operations
> altogether to avoid having to deal with this. In the common case, I
> don't see why an OS installation shipped via a VM disk image would need
> to write boot variables anyway.

If a VM is booting from a distributed disk image, the boot variables
absolutely should start out empty. That's the only sane option.

It is appropriate to implement boot variable storage, but only because
it is needed if multiple OSes get installed. Those variables should not
get distributed with a disk image.

> Would there be any adverse consequences to doing this?

It would be a bad idea to inhibit variable storage. That would break
all kinds of boot and install scenarios.

> My reason is that this would save us from blocking a general OpenStack
> implementation on ARM by requiring that these pieces are implemented
> further up the stack first, when it would bring actual gain to doing so.
> 
> This would not preclude host implementations from implementing writeable
> variables, or guests from using them. Just that for a _portable VM disk
> image_, the OS on it cannot assume that this functionality is present.

Yeah, just to restate what I mean. If you're talking about bringing up a
portable disk image, the VM should start with an empty list of boot
variables.

g.

[RFC] ARM VM System Sepcification

[Grant Likely]

On Thu, 6 Mar 2014 08:52:13 +0000, Robie Basak <robie.basak@canonical.com> wrote:
> On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
> > I would also reference section 3.3 (Boot Option Variables Default Boot
> > Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
> > restate the meaning of the requirement in this spec, but the UEFI spec
> > is the authoritative source. Distributed VM disk images fall under the
> > same scenario as the firmware not having any valid boot variables.
> 
> What happens when the VM is first booted without boot variables, but
> then the OS expects to be able to set boot variables and see them on
> next boot?
> 
> AIUI, we don't have an implementation of this right now, and even if we
> did, there are implications for persistent storage of this data further
> up the stack (a required implementation in libvirt, OpenStack nova
> providing a storage area for it, etc).
> 
> If possible, I would prefer to mandate that the host implementation is
> permitted to no-op (or otherwise disable) boot variable write operations
> altogether to avoid having to deal with this. In the common case, I
> don't see why an OS installation shipped via a VM disk image would need
> to write boot variables anyway.

If a VM is booting from a distributed disk image, the boot variables
absolutely should start out empty. That's the only sane option.

It is appropriate to implement boot variable storage, but only because
it is needed if multiple OSes get installed. Those variables should not
get distributed with a disk image.

> Would there be any adverse consequences to doing this?

It would be a bad idea to inhibit variable storage. That would break
all kinds of boot and install scenarios.

> My reason is that this would save us from blocking a general OpenStack
> implementation on ARM by requiring that these pieces are implemented
> further up the stack first, when it would bring actual gain to doing so.
> 
> This would not preclude host implementations from implementing writeable
> variables, or guests from using them. Just that for a _portable VM disk
> image_, the OS on it cannot assume that this functionality is present.

Yeah, just to restate what I mean. If you're talking about bringing up a
portable disk image, the VM should start with an empty list of boot
variables.

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Thu, 6 Mar 2014 08:52:13 +0000, Robie Basak <robie.basak@canonical.com> wrote:
> On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
> > I would also reference section 3.3 (Boot Option Variables Default Boot
> > Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
> > restate the meaning of the requirement in this spec, but the UEFI spec
> > is the authoritative source. Distributed VM disk images fall under the
> > same scenario as the firmware not having any valid boot variables.
> 
> What happens when the VM is first booted without boot variables, but
> then the OS expects to be able to set boot variables and see them on
> next boot?
> 
> AIUI, we don't have an implementation of this right now, and even if we
> did, there are implications for persistent storage of this data further
> up the stack (a required implementation in libvirt, OpenStack nova
> providing a storage area for it, etc).
> 
> If possible, I would prefer to mandate that the host implementation is
> permitted to no-op (or otherwise disable) boot variable write operations
> altogether to avoid having to deal with this. In the common case, I
> don't see why an OS installation shipped via a VM disk image would need
> to write boot variables anyway.

If a VM is booting from a distributed disk image, the boot variables
absolutely should start out empty. That's the only sane option.

It is appropriate to implement boot variable storage, but only because
it is needed if multiple OSes get installed. Those variables should not
get distributed with a disk image.

> Would there be any adverse consequences to doing this?

It would be a bad idea to inhibit variable storage. That would break
all kinds of boot and install scenarios.

> My reason is that this would save us from blocking a general OpenStack
> implementation on ARM by requiring that these pieces are implemented
> further up the stack first, when it would bring actual gain to doing so.
> 
> This would not preclude host implementations from implementing writeable
> variables, or guests from using them. Just that for a _portable VM disk
> image_, the OS on it cannot assume that this functionality is present.

Yeah, just to restate what I mean. If you're talking about bringing up a
portable disk image, the VM should start with an empty list of boot
variables.

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Thu, 06 Mar 2014 10:46:22 +0100, Paolo Bonzini <pbonzini-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> Il 06/03/2014 09:52, Robie Basak ha scritto:
> > On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
> >> I would also reference section 3.3 (Boot Option Variables Default Boot
> >> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
> >> restate the meaning of the requirement in this spec, but the UEFI spec
> >> is the authoritative source. Distributed VM disk images fall under the
> >> same scenario as the firmware not having any valid boot variables.
> >
> > What happens when the VM is first booted without boot variables, but
> > then the OS expects to be able to set boot variables and see them on
> > next boot?
> 
> UEFI scans the devices; looks for an EFI system partition on the disks; 
> and builds a default boot order.
> 
> > If possible, I would prefer to mandate that the host implementation is
> > permitted to no-op (or otherwise disable) boot variable write operations
> > altogether to avoid having to deal with this. In the common case, I
> > don't see why an OS installation shipped via a VM disk image would need
> > to write boot variables anyway.
> >
> > Would there be any adverse consequences to doing this?
> 
> Given the experience on x86 UEFI, no.
> 
> Unlike bare metal, it is common to run UEFI VMs without persistent flash 
> storage.  In this case the boot variables and boot order are rebuilt on 
> the fly on every boot, and it just works for both Windows and Linux; 
> there's no reason why it should be any different for ARM.
> 
> > My reason is that this would save us from blocking a general OpenStack
> > implementation on ARM by requiring that these pieces are implemented
> > further up the stack first, when it would bring actual gain to doing so.
> >
> > This would not preclude host implementations from implementing writeable
> > variables, or guests from using them. Just that for a _portable VM disk
> > image_, the OS on it cannot assume that this functionality is present.
> 
> This is already the case for most OSes.  Otherwise you wouldn't be able 
> to move a hard disk from a (physical) machine to another.
> 
> I strongly suggest that you take a look at the work done in Tiano Core's 
> OvmfPkg, which has support for almost every QEMU feature thanks to the 
> work of Laszlo Ersek and Jordan Justen.
> 
> In particular, OvmfPkg has support for specifying a boot order in the VM 
> configuration (which maps to the "-boot" option in QEMU).  In this case, 
> the UEFI boot order is overridden by a variable that is placed in some 
> architecture-specific firmware configuration mechanism (on x86 we have 
> one called fw_cfg, on ARM you could look at the fdt).  This predates 
> UEFI and is not a UEFI variable; in fact is is a list of OpenFirmware 
> device paths.  UEFI will match the OF paths to UEFI paths, and use the 
> result to build a UEFI boot order.

I don't know why we wouldn't want to make the UEFI variable the
mechanism for exposing VM boot order to UEFI and the OS. I do completely
agree that the boot order should be owned by the VM and be able to be
manipulated from the config file and command line.

g.

[RFC] ARM VM System Sepcification

[Grant Likely]

On Thu, 06 Mar 2014 10:46:22 +0100, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 06/03/2014 09:52, Robie Basak ha scritto:
> > On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
> >> I would also reference section 3.3 (Boot Option Variables Default Boot
> >> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
> >> restate the meaning of the requirement in this spec, but the UEFI spec
> >> is the authoritative source. Distributed VM disk images fall under the
> >> same scenario as the firmware not having any valid boot variables.
> >
> > What happens when the VM is first booted without boot variables, but
> > then the OS expects to be able to set boot variables and see them on
> > next boot?
> 
> UEFI scans the devices; looks for an EFI system partition on the disks; 
> and builds a default boot order.
> 
> > If possible, I would prefer to mandate that the host implementation is
> > permitted to no-op (or otherwise disable) boot variable write operations
> > altogether to avoid having to deal with this. In the common case, I
> > don't see why an OS installation shipped via a VM disk image would need
> > to write boot variables anyway.
> >
> > Would there be any adverse consequences to doing this?
> 
> Given the experience on x86 UEFI, no.
> 
> Unlike bare metal, it is common to run UEFI VMs without persistent flash 
> storage.  In this case the boot variables and boot order are rebuilt on 
> the fly on every boot, and it just works for both Windows and Linux; 
> there's no reason why it should be any different for ARM.
> 
> > My reason is that this would save us from blocking a general OpenStack
> > implementation on ARM by requiring that these pieces are implemented
> > further up the stack first, when it would bring actual gain to doing so.
> >
> > This would not preclude host implementations from implementing writeable
> > variables, or guests from using them. Just that for a _portable VM disk
> > image_, the OS on it cannot assume that this functionality is present.
> 
> This is already the case for most OSes.  Otherwise you wouldn't be able 
> to move a hard disk from a (physical) machine to another.
> 
> I strongly suggest that you take a look at the work done in Tiano Core's 
> OvmfPkg, which has support for almost every QEMU feature thanks to the 
> work of Laszlo Ersek and Jordan Justen.
> 
> In particular, OvmfPkg has support for specifying a boot order in the VM 
> configuration (which maps to the "-boot" option in QEMU).  In this case, 
> the UEFI boot order is overridden by a variable that is placed in some 
> architecture-specific firmware configuration mechanism (on x86 we have 
> one called fw_cfg, on ARM you could look at the fdt).  This predates 
> UEFI and is not a UEFI variable; in fact is is a list of OpenFirmware 
> device paths.  UEFI will match the OF paths to UEFI paths, and use the 
> result to build a UEFI boot order.

I don't know why we wouldn't want to make the UEFI variable the
mechanism for exposing VM boot order to UEFI and the OS. I do completely
agree that the boot order should be owned by the VM and be able to be
manipulated from the config file and command line.

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Thu, 06 Mar 2014 10:46:22 +0100, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 06/03/2014 09:52, Robie Basak ha scritto:
> > On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
> >> I would also reference section 3.3 (Boot Option Variables Default Boot
> >> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's fine to
> >> restate the meaning of the requirement in this spec, but the UEFI spec
> >> is the authoritative source. Distributed VM disk images fall under the
> >> same scenario as the firmware not having any valid boot variables.
> >
> > What happens when the VM is first booted without boot variables, but
> > then the OS expects to be able to set boot variables and see them on
> > next boot?
> 
> UEFI scans the devices; looks for an EFI system partition on the disks; 
> and builds a default boot order.
> 
> > If possible, I would prefer to mandate that the host implementation is
> > permitted to no-op (or otherwise disable) boot variable write operations
> > altogether to avoid having to deal with this. In the common case, I
> > don't see why an OS installation shipped via a VM disk image would need
> > to write boot variables anyway.
> >
> > Would there be any adverse consequences to doing this?
> 
> Given the experience on x86 UEFI, no.
> 
> Unlike bare metal, it is common to run UEFI VMs without persistent flash 
> storage.  In this case the boot variables and boot order are rebuilt on 
> the fly on every boot, and it just works for both Windows and Linux; 
> there's no reason why it should be any different for ARM.
> 
> > My reason is that this would save us from blocking a general OpenStack
> > implementation on ARM by requiring that these pieces are implemented
> > further up the stack first, when it would bring actual gain to doing so.
> >
> > This would not preclude host implementations from implementing writeable
> > variables, or guests from using them. Just that for a _portable VM disk
> > image_, the OS on it cannot assume that this functionality is present.
> 
> This is already the case for most OSes.  Otherwise you wouldn't be able 
> to move a hard disk from a (physical) machine to another.
> 
> I strongly suggest that you take a look at the work done in Tiano Core's 
> OvmfPkg, which has support for almost every QEMU feature thanks to the 
> work of Laszlo Ersek and Jordan Justen.
> 
> In particular, OvmfPkg has support for specifying a boot order in the VM 
> configuration (which maps to the "-boot" option in QEMU).  In this case, 
> the UEFI boot order is overridden by a variable that is placed in some 
> architecture-specific firmware configuration mechanism (on x86 we have 
> one called fw_cfg, on ARM you could look at the fdt).  This predates 
> UEFI and is not a UEFI variable; in fact is is a list of OpenFirmware 
> device paths.  UEFI will match the OF paths to UEFI paths, and use the 
> result to build a UEFI boot order.

I don't know why we wouldn't want to make the UEFI variable the
mechanism for exposing VM boot order to UEFI and the OS. I do completely
agree that the boot order should be owned by the VM and be able to be
manipulated from the config file and command line.

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Thu, 6 Mar 2014 12:04:50 +0000, Robie Basak <robie.basak-Z7WLFzj8eWMS+FvcfC7Uqw@public.gmane.org> wrote:
> On Thu, Mar 06, 2014 at 12:44:57PM +0100, Laszlo Ersek wrote:
> > If I understand correctly, the question is this:
> > 
> >   Given a hypervisor that doesn't support non-volatile UEFI variables
> >   (including BootOrder and Boot####), is it possible to automatically
> >   boot a carefully prepared VM image, made available as a fixed media
> >   device?
> > 
> > The answer is "yes". See
> 
> Right, but I think there is a subsequent problem.
> 
> >   It is expected that this default boot will load an operating system or
> >   a maintenance utility. If this is an operating system setup program it
> >   is then responsible for setting the requisite environment variables
> >   for subsequent boots. The platform firmware may also decide to recover
>         ^^^^^^^^^^^^^^^^
> >   or set to a known set of boot options.
> 
> It seems to me that the guest OS is permitted to assume that persistent
> boot variables will work after first boot, for subsequent boots.
> 
> So, for example, the guest OS might, on bootloader or kernel upgrade,
> completely replace the boot mechanism, dropping the removable path and
> replacing it with a fixed disk arrangement, setting boot variables
> appropriately, and assume that it can reboot and everything will
> continue to work.
> 
> But if the host does not support non-volatile variables, then this will
> break.

Correct

> This is why I'm suggesting that the specification mandate that the guest
> OS shipped in a "portable disk image" as defined by the spec must not
> make this assumption.

Also correct... the installer must be aware of this constraint which is
why it is part of the draft spec.

> It's either this, or mandate that all hosts must support persistent
> variables. I have no objection to that in principle, but since we have
> no implementation currently, it seems easier to avoid this particular
> roadblock by tweaking the spec in a way that nobody seems to care about
> anyway.

Right. I guess my position is that if persistent storage is not
implemented then there are a number of install/upgrade scenarios that
won't work. Regardless, portable images must assume an empty boot list
and we can build that into the spec.

g.

[RFC] ARM VM System Sepcification

[Grant Likely]

On Thu, 6 Mar 2014 12:04:50 +0000, Robie Basak <robie.basak@canonical.com> wrote:
> On Thu, Mar 06, 2014 at 12:44:57PM +0100, Laszlo Ersek wrote:
> > If I understand correctly, the question is this:
> > 
> >   Given a hypervisor that doesn't support non-volatile UEFI variables
> >   (including BootOrder and Boot####), is it possible to automatically
> >   boot a carefully prepared VM image, made available as a fixed media
> >   device?
> > 
> > The answer is "yes". See
> 
> Right, but I think there is a subsequent problem.
> 
> >   It is expected that this default boot will load an operating system or
> >   a maintenance utility. If this is an operating system setup program it
> >   is then responsible for setting the requisite environment variables
> >   for subsequent boots. The platform firmware may also decide to recover
>         ^^^^^^^^^^^^^^^^
> >   or set to a known set of boot options.
> 
> It seems to me that the guest OS is permitted to assume that persistent
> boot variables will work after first boot, for subsequent boots.
> 
> So, for example, the guest OS might, on bootloader or kernel upgrade,
> completely replace the boot mechanism, dropping the removable path and
> replacing it with a fixed disk arrangement, setting boot variables
> appropriately, and assume that it can reboot and everything will
> continue to work.
> 
> But if the host does not support non-volatile variables, then this will
> break.

Correct

> This is why I'm suggesting that the specification mandate that the guest
> OS shipped in a "portable disk image" as defined by the spec must not
> make this assumption.

Also correct... the installer must be aware of this constraint which is
why it is part of the draft spec.

> It's either this, or mandate that all hosts must support persistent
> variables. I have no objection to that in principle, but since we have
> no implementation currently, it seems easier to avoid this particular
> roadblock by tweaking the spec in a way that nobody seems to care about
> anyway.

Right. I guess my position is that if persistent storage is not
implemented then there are a number of install/upgrade scenarios that
won't work. Regardless, portable images must assume an empty boot list
and we can build that into the spec.

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Thu, 6 Mar 2014 12:04:50 +0000, Robie Basak <robie.basak@canonical.com> wrote:
> On Thu, Mar 06, 2014 at 12:44:57PM +0100, Laszlo Ersek wrote:
> > If I understand correctly, the question is this:
> > 
> >   Given a hypervisor that doesn't support non-volatile UEFI variables
> >   (including BootOrder and Boot####), is it possible to automatically
> >   boot a carefully prepared VM image, made available as a fixed media
> >   device?
> > 
> > The answer is "yes". See
> 
> Right, but I think there is a subsequent problem.
> 
> >   It is expected that this default boot will load an operating system or
> >   a maintenance utility. If this is an operating system setup program it
> >   is then responsible for setting the requisite environment variables
> >   for subsequent boots. The platform firmware may also decide to recover
>         ^^^^^^^^^^^^^^^^
> >   or set to a known set of boot options.
> 
> It seems to me that the guest OS is permitted to assume that persistent
> boot variables will work after first boot, for subsequent boots.
> 
> So, for example, the guest OS might, on bootloader or kernel upgrade,
> completely replace the boot mechanism, dropping the removable path and
> replacing it with a fixed disk arrangement, setting boot variables
> appropriately, and assume that it can reboot and everything will
> continue to work.
> 
> But if the host does not support non-volatile variables, then this will
> break.

Correct

> This is why I'm suggesting that the specification mandate that the guest
> OS shipped in a "portable disk image" as defined by the spec must not
> make this assumption.

Also correct... the installer must be aware of this constraint which is
why it is part of the draft spec.

> It's either this, or mandate that all hosts must support persistent
> variables. I have no objection to that in principle, but since we have
> no implementation currently, it seems easier to avoid this particular
> roadblock by tweaking the spec in a way that nobody seems to care about
> anyway.

Right. I guess my position is that if persistent storage is not
implemented then there are a number of install/upgrade scenarios that
won't work. Regardless, portable images must assume an empty boot list
and we can build that into the spec.

g.

Re: [RFC] ARM VM System Sepcification

[Michael Casadevall]

On 03/06/2014 05:46 PM, Paolo Bonzini wrote:
> Il 06/03/2014 09:52, Robie Basak ha scritto:
>> On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
>>> I would also reference section 3.3 (Boot Option Variables Default Boot
>>> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's 
>>> fine to
>>> restate the meaning of the requirement in this spec, but the UEFI spec
>>> is the authoritative source. Distributed VM disk images fall under the
>>> same scenario as the firmware not having any valid boot variables.
>>
>> What happens when the VM is first booted without boot variables, but
>> then the OS expects to be able to set boot variables and see them on
>> next boot?
>
> UEFI scans the devices; looks for an EFI system partition on the 
> disks; and builds a default boot order.
>
>> If possible, I would prefer to mandate that the host implementation is
>> permitted to no-op (or otherwise disable) boot variable write operations
>> altogether to avoid having to deal with this. In the common case, I
>> don't see why an OS installation shipped via a VM disk image would need
>> to write boot variables anyway.
>>
>> Would there be any adverse consequences to doing this?
>
> Given the experience on x86 UEFI, no.
>
> Unlike bare metal, it is common to run UEFI VMs without persistent 
> flash storage.  In this case the boot variables and boot order are 
> rebuilt on the fly on every boot, and it just works for both Windows 
> and Linux; there's no reason why it should be any different for ARM.
>
While I realize in the real world, we can live with non-persistent boot 
variables, this is a *direct* violation of the UEFI spec; we can't call 
our VMs UEFI-compatible if we do this.

However, I've been looking at the spec, and I think we're within spec if 
we save the variables on the HDD itself. There's some support for this 
already (Firmware Block Volume Device), but its possible we could 
implement boot variables as a file on system partition (UEFI's default 
search order can be used to figure out which variable file to use, or 
some sorta fingerprinting system). The biggest trick though is that 
UEFI's Runtime Services will need to be able to write this file, which 
may require us move a large chunk of UEFI to runtime services so the FAT 
filesystem stuff can stick around. If we give it a proper partition, 
then we can just do raw block read/writes. This however would require us 
mandating that said partition exists, and making sure there aren't any 
hidden gotchas in invoking this.

Obviously this isn't ideal, but this might be the middle road solution 
we need here. I can dig through Tiano to get a realistic idea of how 
hard this will be in reality if we want to seriously look at this option.

>> My reason is that this would save us from blocking a general OpenStack
>> implementation on ARM by requiring that these pieces are implemented
>> further up the stack first, when it would bring actual gain to doing so.
>>
>> This would not preclude host implementations from implementing writeable
>> variables, or guests from using them. Just that for a _portable VM disk
>> image_, the OS on it cannot assume that this functionality is present.
>
> This is already the case for most OSes.  Otherwise you wouldn't be 
> able to move a hard disk from a (physical) machine to another.
>
> I strongly suggest that you take a look at the work done in Tiano 
> Core's OvmfPkg, which has support for almost every QEMU feature thanks 
> to the work of Laszlo Ersek and Jordan Justen.
>
> In particular, OvmfPkg has support for specifying a boot order in the 
> VM configuration (which maps to the "-boot" option in QEMU). In this 
> case, the UEFI boot order is overridden by a variable that is placed 
> in some architecture-specific firmware configuration mechanism (on x86 
> we have one called fw_cfg, on ARM you could look at the fdt).  This 
> predates UEFI and is not a UEFI variable; in fact is is a list of 
> OpenFirmware device paths.  UEFI will match the OF paths to UEFI 
> paths, and use the result to build a UEFI boot order.
>
This lines up with work to make Tiano itself run on FDT to handle 
varying boot configurations. Is this behaviour and the DT nodes codified 
anywhere?

> Paolo

[RFC] ARM VM System Sepcification

[Michael Casadevall]

On 03/06/2014 05:46 PM, Paolo Bonzini wrote:
> Il 06/03/2014 09:52, Robie Basak ha scritto:
>> On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
>>> I would also reference section 3.3 (Boot Option Variables Default Boot
>>> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's 
>>> fine to
>>> restate the meaning of the requirement in this spec, but the UEFI spec
>>> is the authoritative source. Distributed VM disk images fall under the
>>> same scenario as the firmware not having any valid boot variables.
>>
>> What happens when the VM is first booted without boot variables, but
>> then the OS expects to be able to set boot variables and see them on
>> next boot?
>
> UEFI scans the devices; looks for an EFI system partition on the 
> disks; and builds a default boot order.
>
>> If possible, I would prefer to mandate that the host implementation is
>> permitted to no-op (or otherwise disable) boot variable write operations
>> altogether to avoid having to deal with this. In the common case, I
>> don't see why an OS installation shipped via a VM disk image would need
>> to write boot variables anyway.
>>
>> Would there be any adverse consequences to doing this?
>
> Given the experience on x86 UEFI, no.
>
> Unlike bare metal, it is common to run UEFI VMs without persistent 
> flash storage.  In this case the boot variables and boot order are 
> rebuilt on the fly on every boot, and it just works for both Windows 
> and Linux; there's no reason why it should be any different for ARM.
>
While I realize in the real world, we can live with non-persistent boot 
variables, this is a *direct* violation of the UEFI spec; we can't call 
our VMs UEFI-compatible if we do this.

However, I've been looking at the spec, and I think we're within spec if 
we save the variables on the HDD itself. There's some support for this 
already (Firmware Block Volume Device), but its possible we could 
implement boot variables as a file on system partition (UEFI's default 
search order can be used to figure out which variable file to use, or 
some sorta fingerprinting system). The biggest trick though is that 
UEFI's Runtime Services will need to be able to write this file, which 
may require us move a large chunk of UEFI to runtime services so the FAT 
filesystem stuff can stick around. If we give it a proper partition, 
then we can just do raw block read/writes. This however would require us 
mandating that said partition exists, and making sure there aren't any 
hidden gotchas in invoking this.

Obviously this isn't ideal, but this might be the middle road solution 
we need here. I can dig through Tiano to get a realistic idea of how 
hard this will be in reality if we want to seriously look at this option.

>> My reason is that this would save us from blocking a general OpenStack
>> implementation on ARM by requiring that these pieces are implemented
>> further up the stack first, when it would bring actual gain to doing so.
>>
>> This would not preclude host implementations from implementing writeable
>> variables, or guests from using them. Just that for a _portable VM disk
>> image_, the OS on it cannot assume that this functionality is present.
>
> This is already the case for most OSes.  Otherwise you wouldn't be 
> able to move a hard disk from a (physical) machine to another.
>
> I strongly suggest that you take a look at the work done in Tiano 
> Core's OvmfPkg, which has support for almost every QEMU feature thanks 
> to the work of Laszlo Ersek and Jordan Justen.
>
> In particular, OvmfPkg has support for specifying a boot order in the 
> VM configuration (which maps to the "-boot" option in QEMU). In this 
> case, the UEFI boot order is overridden by a variable that is placed 
> in some architecture-specific firmware configuration mechanism (on x86 
> we have one called fw_cfg, on ARM you could look at the fdt).  This 
> predates UEFI and is not a UEFI variable; in fact is is a list of 
> OpenFirmware device paths.  UEFI will match the OF paths to UEFI 
> paths, and use the result to build a UEFI boot order.
>
This lines up with work to make Tiano itself run on FDT to handle 
varying boot configurations. Is this behaviour and the DT nodes codified 
anywhere?

> Paolo

Re: [RFC] ARM VM System Sepcification

[Michael Casadevall]

On 03/06/2014 05:46 PM, Paolo Bonzini wrote:
> Il 06/03/2014 09:52, Robie Basak ha scritto:
>> On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
>>> I would also reference section 3.3 (Boot Option Variables Default Boot
>>> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's 
>>> fine to
>>> restate the meaning of the requirement in this spec, but the UEFI spec
>>> is the authoritative source. Distributed VM disk images fall under the
>>> same scenario as the firmware not having any valid boot variables.
>>
>> What happens when the VM is first booted without boot variables, but
>> then the OS expects to be able to set boot variables and see them on
>> next boot?
>
> UEFI scans the devices; looks for an EFI system partition on the 
> disks; and builds a default boot order.
>
>> If possible, I would prefer to mandate that the host implementation is
>> permitted to no-op (or otherwise disable) boot variable write operations
>> altogether to avoid having to deal with this. In the common case, I
>> don't see why an OS installation shipped via a VM disk image would need
>> to write boot variables anyway.
>>
>> Would there be any adverse consequences to doing this?
>
> Given the experience on x86 UEFI, no.
>
> Unlike bare metal, it is common to run UEFI VMs without persistent 
> flash storage.  In this case the boot variables and boot order are 
> rebuilt on the fly on every boot, and it just works for both Windows 
> and Linux; there's no reason why it should be any different for ARM.
>
While I realize in the real world, we can live with non-persistent boot 
variables, this is a *direct* violation of the UEFI spec; we can't call 
our VMs UEFI-compatible if we do this.

However, I've been looking at the spec, and I think we're within spec if 
we save the variables on the HDD itself. There's some support for this 
already (Firmware Block Volume Device), but its possible we could 
implement boot variables as a file on system partition (UEFI's default 
search order can be used to figure out which variable file to use, or 
some sorta fingerprinting system). The biggest trick though is that 
UEFI's Runtime Services will need to be able to write this file, which 
may require us move a large chunk of UEFI to runtime services so the FAT 
filesystem stuff can stick around. If we give it a proper partition, 
then we can just do raw block read/writes. This however would require us 
mandating that said partition exists, and making sure there aren't any 
hidden gotchas in invoking this.

Obviously this isn't ideal, but this might be the middle road solution 
we need here. I can dig through Tiano to get a realistic idea of how 
hard this will be in reality if we want to seriously look at this option.

>> My reason is that this would save us from blocking a general OpenStack
>> implementation on ARM by requiring that these pieces are implemented
>> further up the stack first, when it would bring actual gain to doing so.
>>
>> This would not preclude host implementations from implementing writeable
>> variables, or guests from using them. Just that for a _portable VM disk
>> image_, the OS on it cannot assume that this functionality is present.
>
> This is already the case for most OSes.  Otherwise you wouldn't be 
> able to move a hard disk from a (physical) machine to another.
>
> I strongly suggest that you take a look at the work done in Tiano 
> Core's OvmfPkg, which has support for almost every QEMU feature thanks 
> to the work of Laszlo Ersek and Jordan Justen.
>
> In particular, OvmfPkg has support for specifying a boot order in the 
> VM configuration (which maps to the "-boot" option in QEMU). In this 
> case, the UEFI boot order is overridden by a variable that is placed 
> in some architecture-specific firmware configuration mechanism (on x86 
> we have one called fw_cfg, on ARM you could look at the fdt).  This 
> predates UEFI and is not a UEFI variable; in fact is is a list of 
> OpenFirmware device paths.  UEFI will match the OF paths to UEFI 
> paths, and use the result to build a UEFI boot order.
>
This lines up with work to make Tiano itself run on FDT to handle 
varying boot configurations. Is this behaviour and the DT nodes codified 
anywhere?

> Paolo

Re: [RFC] ARM VM System Sepcification

[Laszlo Ersek]

On 03/08/14 12:41, Michael Casadevall wrote:
> 
> On 03/06/2014 05:46 PM, Paolo Bonzini wrote:
>> Il 06/03/2014 09:52, Robie Basak ha scritto:
>>> On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
>>>> I would also reference section 3.3 (Boot Option Variables Default Boot
>>>> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's
>>>> fine to
>>>> restate the meaning of the requirement in this spec, but the UEFI spec
>>>> is the authoritative source. Distributed VM disk images fall under the
>>>> same scenario as the firmware not having any valid boot variables.
>>>
>>> What happens when the VM is first booted without boot variables, but
>>> then the OS expects to be able to set boot variables and see them on
>>> next boot?
>>
>> UEFI scans the devices; looks for an EFI system partition on the
>> disks; and builds a default boot order.
>>
>>> If possible, I would prefer to mandate that the host implementation is
>>> permitted to no-op (or otherwise disable) boot variable write operations
>>> altogether to avoid having to deal with this. In the common case, I
>>> don't see why an OS installation shipped via a VM disk image would need
>>> to write boot variables anyway.
>>>
>>> Would there be any adverse consequences to doing this?
>>
>> Given the experience on x86 UEFI, no.
>>
>> Unlike bare metal, it is common to run UEFI VMs without persistent
>> flash storage.  In this case the boot variables and boot order are
>> rebuilt on the fly on every boot, and it just works for both Windows
>> and Linux; there's no reason why it should be any different for ARM.
>>
> While I realize in the real world, we can live with non-persistent boot
> variables, this is a *direct* violation of the UEFI spec; we can't call
> our VMs UEFI-compatible if we do this.
> 
> However, I've been looking at the spec, and I think we're within spec if
> we save the variables on the HDD itself. There's some support for this
> already (Firmware Block Volume Device), but its possible we could
> implement boot variables as a file on system partition (UEFI's default
> search order can be used to figure out which variable file to use, or
> some sorta fingerprinting system). The biggest trick though is that
> UEFI's Runtime Services will need to be able to write this file, which
> may require us move a large chunk of UEFI to runtime services so the FAT
> filesystem stuff can stick around. If we give it a proper partition,
> then we can just do raw block read/writes. This however would require us
> mandating that said partition exists, and making sure there aren't any
> hidden gotchas in invoking this.

This is how OVMF fakes non-volatile variables when pflash is not enabled
or supported by the host. It stores variables in a file called \NvVars
in the EFI system partition. Before ExitBootServices(), changes are
synced out immediately. After ExitBootServices(), changes are kept only
in memory. If you then do an in-VM reboot, then you end up again
"before" ExitBootServices(), and then the in-memory changes are written
out. This was usually good enough to keep installers happy (because the
last step they do is install grub, run efibootmgr, and reboot).

Keeping around the UEFI FAT driver into runtime would be just one
problem. Another problem is that the OS could itself mount the partition
that holds the file, and then the two drivers (UEFI and Linux) would
corrupt the filesystem. A "raw" partition and a custom FVB driver would
probably be safer in this regard.

> Obviously this isn't ideal, but this might be the middle road solution
> we need here. I can dig through Tiano to get a realistic idea of how
> hard this will be in reality if we want to seriously look at this option.

You could check out Jordan's implementation for qemu pflash --
"OvmfPkg/QemuFlashFvbServicesRuntimeDxe".

The protocol to implement is EFI_FIRMWARE_VOLUME_BLOCK[2]_PROTOCOL,
everything else that should be seated on top "automatically" is already
present in edk2. The protocol is documented in Vol3 of the Platform Init
spec, chapter 3.4.2.

(...BLOCK2... is just a typedef to ...BLOCK..., see
MdePkg/Include/Protocol/FirmwareVolumeBlock.h)

Thanks
Laszlo

[RFC] ARM VM System Sepcification

[Laszlo Ersek]

On 03/08/14 12:41, Michael Casadevall wrote:
> 
> On 03/06/2014 05:46 PM, Paolo Bonzini wrote:
>> Il 06/03/2014 09:52, Robie Basak ha scritto:
>>> On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
>>>> I would also reference section 3.3 (Boot Option Variables Default Boot
>>>> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's
>>>> fine to
>>>> restate the meaning of the requirement in this spec, but the UEFI spec
>>>> is the authoritative source. Distributed VM disk images fall under the
>>>> same scenario as the firmware not having any valid boot variables.
>>>
>>> What happens when the VM is first booted without boot variables, but
>>> then the OS expects to be able to set boot variables and see them on
>>> next boot?
>>
>> UEFI scans the devices; looks for an EFI system partition on the
>> disks; and builds a default boot order.
>>
>>> If possible, I would prefer to mandate that the host implementation is
>>> permitted to no-op (or otherwise disable) boot variable write operations
>>> altogether to avoid having to deal with this. In the common case, I
>>> don't see why an OS installation shipped via a VM disk image would need
>>> to write boot variables anyway.
>>>
>>> Would there be any adverse consequences to doing this?
>>
>> Given the experience on x86 UEFI, no.
>>
>> Unlike bare metal, it is common to run UEFI VMs without persistent
>> flash storage.  In this case the boot variables and boot order are
>> rebuilt on the fly on every boot, and it just works for both Windows
>> and Linux; there's no reason why it should be any different for ARM.
>>
> While I realize in the real world, we can live with non-persistent boot
> variables, this is a *direct* violation of the UEFI spec; we can't call
> our VMs UEFI-compatible if we do this.
> 
> However, I've been looking at the spec, and I think we're within spec if
> we save the variables on the HDD itself. There's some support for this
> already (Firmware Block Volume Device), but its possible we could
> implement boot variables as a file on system partition (UEFI's default
> search order can be used to figure out which variable file to use, or
> some sorta fingerprinting system). The biggest trick though is that
> UEFI's Runtime Services will need to be able to write this file, which
> may require us move a large chunk of UEFI to runtime services so the FAT
> filesystem stuff can stick around. If we give it a proper partition,
> then we can just do raw block read/writes. This however would require us
> mandating that said partition exists, and making sure there aren't any
> hidden gotchas in invoking this.

This is how OVMF fakes non-volatile variables when pflash is not enabled
or supported by the host. It stores variables in a file called \NvVars
in the EFI system partition. Before ExitBootServices(), changes are
synced out immediately. After ExitBootServices(), changes are kept only
in memory. If you then do an in-VM reboot, then you end up again
"before" ExitBootServices(), and then the in-memory changes are written
out. This was usually good enough to keep installers happy (because the
last step they do is install grub, run efibootmgr, and reboot).

Keeping around the UEFI FAT driver into runtime would be just one
problem. Another problem is that the OS could itself mount the partition
that holds the file, and then the two drivers (UEFI and Linux) would
corrupt the filesystem. A "raw" partition and a custom FVB driver would
probably be safer in this regard.

> Obviously this isn't ideal, but this might be the middle road solution
> we need here. I can dig through Tiano to get a realistic idea of how
> hard this will be in reality if we want to seriously look at this option.

You could check out Jordan's implementation for qemu pflash --
"OvmfPkg/QemuFlashFvbServicesRuntimeDxe".

The protocol to implement is EFI_FIRMWARE_VOLUME_BLOCK[2]_PROTOCOL,
everything else that should be seated on top "automatically" is already
present in edk2. The protocol is documented in Vol3 of the Platform Init
spec, chapter 3.4.2.

(...BLOCK2... is just a typedef to ...BLOCK..., see
MdePkg/Include/Protocol/FirmwareVolumeBlock.h)

Thanks
Laszlo

Re: [RFC] ARM VM System Sepcification

[Laszlo Ersek]

On 03/08/14 12:41, Michael Casadevall wrote:
> 
> On 03/06/2014 05:46 PM, Paolo Bonzini wrote:
>> Il 06/03/2014 09:52, Robie Basak ha scritto:
>>> On Sat, Mar 01, 2014 at 03:27:56PM +0000, Grant Likely wrote:
>>>> I would also reference section 3.3 (Boot Option Variables Default Boot
>>>> Behavior) and 3.4.1.1 (Removable Media Boot Behavior) here. It's
>>>> fine to
>>>> restate the meaning of the requirement in this spec, but the UEFI spec
>>>> is the authoritative source. Distributed VM disk images fall under the
>>>> same scenario as the firmware not having any valid boot variables.
>>>
>>> What happens when the VM is first booted without boot variables, but
>>> then the OS expects to be able to set boot variables and see them on
>>> next boot?
>>
>> UEFI scans the devices; looks for an EFI system partition on the
>> disks; and builds a default boot order.
>>
>>> If possible, I would prefer to mandate that the host implementation is
>>> permitted to no-op (or otherwise disable) boot variable write operations
>>> altogether to avoid having to deal with this. In the common case, I
>>> don't see why an OS installation shipped via a VM disk image would need
>>> to write boot variables anyway.
>>>
>>> Would there be any adverse consequences to doing this?
>>
>> Given the experience on x86 UEFI, no.
>>
>> Unlike bare metal, it is common to run UEFI VMs without persistent
>> flash storage.  In this case the boot variables and boot order are
>> rebuilt on the fly on every boot, and it just works for both Windows
>> and Linux; there's no reason why it should be any different for ARM.
>>
> While I realize in the real world, we can live with non-persistent boot
> variables, this is a *direct* violation of the UEFI spec; we can't call
> our VMs UEFI-compatible if we do this.
> 
> However, I've been looking at the spec, and I think we're within spec if
> we save the variables on the HDD itself. There's some support for this
> already (Firmware Block Volume Device), but its possible we could
> implement boot variables as a file on system partition (UEFI's default
> search order can be used to figure out which variable file to use, or
> some sorta fingerprinting system). The biggest trick though is that
> UEFI's Runtime Services will need to be able to write this file, which
> may require us move a large chunk of UEFI to runtime services so the FAT
> filesystem stuff can stick around. If we give it a proper partition,
> then we can just do raw block read/writes. This however would require us
> mandating that said partition exists, and making sure there aren't any
> hidden gotchas in invoking this.

This is how OVMF fakes non-volatile variables when pflash is not enabled
or supported by the host. It stores variables in a file called \NvVars
in the EFI system partition. Before ExitBootServices(), changes are
synced out immediately. After ExitBootServices(), changes are kept only
in memory. If you then do an in-VM reboot, then you end up again
"before" ExitBootServices(), and then the in-memory changes are written
out. This was usually good enough to keep installers happy (because the
last step they do is install grub, run efibootmgr, and reboot).

Keeping around the UEFI FAT driver into runtime would be just one
problem. Another problem is that the OS could itself mount the partition
that holds the file, and then the two drivers (UEFI and Linux) would
corrupt the filesystem. A "raw" partition and a custom FVB driver would
probably be safer in this regard.

> Obviously this isn't ideal, but this might be the middle road solution
> we need here. I can dig through Tiano to get a realistic idea of how
> hard this will be in reality if we want to seriously look at this option.

You could check out Jordan's implementation for qemu pflash --
"OvmfPkg/QemuFlashFvbServicesRuntimeDxe".

The protocol to implement is EFI_FIRMWARE_VOLUME_BLOCK[2]_PROTOCOL,
everything else that should be seated on top "automatically" is already
present in edk2. The protocol is documented in Vol3 of the Platform Init
spec, chapter 3.4.2.

(...BLOCK2... is just a typedef to ...BLOCK..., see
MdePkg/Include/Protocol/FirmwareVolumeBlock.h)

Thanks
Laszlo

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Fri, Mar 07, 2014 at 08:24:18PM +0800, Grant Likely wrote:
> On Thu, 6 Mar 2014 12:04:50 +0000, Robie Basak <robie.basak@canonical.com> wrote:
> > On Thu, Mar 06, 2014 at 12:44:57PM +0100, Laszlo Ersek wrote:
> > > If I understand correctly, the question is this:
> > > 
> > >   Given a hypervisor that doesn't support non-volatile UEFI variables
> > >   (including BootOrder and Boot####), is it possible to automatically
> > >   boot a carefully prepared VM image, made available as a fixed media
> > >   device?
> > > 
> > > The answer is "yes". See
> > 
> > Right, but I think there is a subsequent problem.
> > 
> > >   It is expected that this default boot will load an operating system or
> > >   a maintenance utility. If this is an operating system setup program it
> > >   is then responsible for setting the requisite environment variables
> > >   for subsequent boots. The platform firmware may also decide to recover
> >         ^^^^^^^^^^^^^^^^
> > >   or set to a known set of boot options.
> > 
> > It seems to me that the guest OS is permitted to assume that persistent
> > boot variables will work after first boot, for subsequent boots.
> > 
> > So, for example, the guest OS might, on bootloader or kernel upgrade,
> > completely replace the boot mechanism, dropping the removable path and
> > replacing it with a fixed disk arrangement, setting boot variables
> > appropriately, and assume that it can reboot and everything will
> > continue to work.
> > 
> > But if the host does not support non-volatile variables, then this will
> > break.
> 
> Correct
> 
> > This is why I'm suggesting that the specification mandate that the guest
> > OS shipped in a "portable disk image" as defined by the spec must not
> > make this assumption.
> 
> Also correct... the installer must be aware of this constraint which is
> why it is part of the draft spec.
> 
> > It's either this, or mandate that all hosts must support persistent
> > variables. I have no objection to that in principle, but since we have
> > no implementation currently, it seems easier to avoid this particular
> > roadblock by tweaking the spec in a way that nobody seems to care about
> > anyway.
> 
> Right. I guess my position is that if persistent storage is not
> implemented then there are a number of install/upgrade scenarios that
> won't work. Regardless, portable images must assume an empty boot list
> and we can build that into the spec.
> 

Sorry for the delay in responding - all sorts of unexpected things
happened when I returned from LCA14.

I agree on the technical discussion going on here.  My conclusion is
that we have two options:

1. Simply mandate that VM implementations support persistent variables
   for their UEFI implementation - with whatever constraints that may
   put on higher level tools.

2. Require that OSes shipped as part of compliant VM images make no
   assumption that changes to the UEFI environment will be stored.

I feel that option number two will break in all sorts of cases, just
like Grant stated above, and it is fundamentally not practical; if a
distribution ships Linux with a UEFI stub that expects to be able to do
something, distributions must modify Linux to conform to this spec.  I
think imagining that this spec controls how UEFI support in Linux/Grub
is done in general would be overreaching.  Additionally, Michael brought
up the fact that it would be non-UEFI compliant.

I know that door #1 may be a pain in terms of libvirt/openstack support
and other related tools, but it feels by far the cleanest and most
long-term solution and is in my oppinion what we shoud shoot for.  We
are early enough in the ARM server/VM game that we should be able to
make the right decisions at this point.

-Christoffer

[RFC] ARM VM System Sepcification

[Christoffer Dall]

On Fri, Mar 07, 2014 at 08:24:18PM +0800, Grant Likely wrote:
> On Thu, 6 Mar 2014 12:04:50 +0000, Robie Basak <robie.basak@canonical.com> wrote:
> > On Thu, Mar 06, 2014 at 12:44:57PM +0100, Laszlo Ersek wrote:
> > > If I understand correctly, the question is this:
> > > 
> > >   Given a hypervisor that doesn't support non-volatile UEFI variables
> > >   (including BootOrder and Boot####), is it possible to automatically
> > >   boot a carefully prepared VM image, made available as a fixed media
> > >   device?
> > > 
> > > The answer is "yes". See
> > 
> > Right, but I think there is a subsequent problem.
> > 
> > >   It is expected that this default boot will load an operating system or
> > >   a maintenance utility. If this is an operating system setup program it
> > >   is then responsible for setting the requisite environment variables
> > >   for subsequent boots. The platform firmware may also decide to recover
> >         ^^^^^^^^^^^^^^^^
> > >   or set to a known set of boot options.
> > 
> > It seems to me that the guest OS is permitted to assume that persistent
> > boot variables will work after first boot, for subsequent boots.
> > 
> > So, for example, the guest OS might, on bootloader or kernel upgrade,
> > completely replace the boot mechanism, dropping the removable path and
> > replacing it with a fixed disk arrangement, setting boot variables
> > appropriately, and assume that it can reboot and everything will
> > continue to work.
> > 
> > But if the host does not support non-volatile variables, then this will
> > break.
> 
> Correct
> 
> > This is why I'm suggesting that the specification mandate that the guest
> > OS shipped in a "portable disk image" as defined by the spec must not
> > make this assumption.
> 
> Also correct... the installer must be aware of this constraint which is
> why it is part of the draft spec.
> 
> > It's either this, or mandate that all hosts must support persistent
> > variables. I have no objection to that in principle, but since we have
> > no implementation currently, it seems easier to avoid this particular
> > roadblock by tweaking the spec in a way that nobody seems to care about
> > anyway.
> 
> Right. I guess my position is that if persistent storage is not
> implemented then there are a number of install/upgrade scenarios that
> won't work. Regardless, portable images must assume an empty boot list
> and we can build that into the spec.
> 

Sorry for the delay in responding - all sorts of unexpected things
happened when I returned from LCA14.

I agree on the technical discussion going on here.  My conclusion is
that we have two options:

1. Simply mandate that VM implementations support persistent variables
   for their UEFI implementation - with whatever constraints that may
   put on higher level tools.

2. Require that OSes shipped as part of compliant VM images make no
   assumption that changes to the UEFI environment will be stored.

I feel that option number two will break in all sorts of cases, just
like Grant stated above, and it is fundamentally not practical; if a
distribution ships Linux with a UEFI stub that expects to be able to do
something, distributions must modify Linux to conform to this spec.  I
think imagining that this spec controls how UEFI support in Linux/Grub
is done in general would be overreaching.  Additionally, Michael brought
up the fact that it would be non-UEFI compliant.

I know that door #1 may be a pain in terms of libvirt/openstack support
and other related tools, but it feels by far the cleanest and most
long-term solution and is in my oppinion what we shoud shoot for.  We
are early enough in the ARM server/VM game that we should be able to
make the right decisions at this point.

-Christoffer

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Fri, Mar 07, 2014 at 08:24:18PM +0800, Grant Likely wrote:
> On Thu, 6 Mar 2014 12:04:50 +0000, Robie Basak <robie.basak@canonical.com> wrote:
> > On Thu, Mar 06, 2014 at 12:44:57PM +0100, Laszlo Ersek wrote:
> > > If I understand correctly, the question is this:
> > > 
> > >   Given a hypervisor that doesn't support non-volatile UEFI variables
> > >   (including BootOrder and Boot####), is it possible to automatically
> > >   boot a carefully prepared VM image, made available as a fixed media
> > >   device?
> > > 
> > > The answer is "yes". See
> > 
> > Right, but I think there is a subsequent problem.
> > 
> > >   It is expected that this default boot will load an operating system or
> > >   a maintenance utility. If this is an operating system setup program it
> > >   is then responsible for setting the requisite environment variables
> > >   for subsequent boots. The platform firmware may also decide to recover
> >         ^^^^^^^^^^^^^^^^
> > >   or set to a known set of boot options.
> > 
> > It seems to me that the guest OS is permitted to assume that persistent
> > boot variables will work after first boot, for subsequent boots.
> > 
> > So, for example, the guest OS might, on bootloader or kernel upgrade,
> > completely replace the boot mechanism, dropping the removable path and
> > replacing it with a fixed disk arrangement, setting boot variables
> > appropriately, and assume that it can reboot and everything will
> > continue to work.
> > 
> > But if the host does not support non-volatile variables, then this will
> > break.
> 
> Correct
> 
> > This is why I'm suggesting that the specification mandate that the guest
> > OS shipped in a "portable disk image" as defined by the spec must not
> > make this assumption.
> 
> Also correct... the installer must be aware of this constraint which is
> why it is part of the draft spec.
> 
> > It's either this, or mandate that all hosts must support persistent
> > variables. I have no objection to that in principle, but since we have
> > no implementation currently, it seems easier to avoid this particular
> > roadblock by tweaking the spec in a way that nobody seems to care about
> > anyway.
> 
> Right. I guess my position is that if persistent storage is not
> implemented then there are a number of install/upgrade scenarios that
> won't work. Regardless, portable images must assume an empty boot list
> and we can build that into the spec.
> 

Sorry for the delay in responding - all sorts of unexpected things
happened when I returned from LCA14.

I agree on the technical discussion going on here.  My conclusion is
that we have two options:

1. Simply mandate that VM implementations support persistent variables
   for their UEFI implementation - with whatever constraints that may
   put on higher level tools.

2. Require that OSes shipped as part of compliant VM images make no
   assumption that changes to the UEFI environment will be stored.

I feel that option number two will break in all sorts of cases, just
like Grant stated above, and it is fundamentally not practical; if a
distribution ships Linux with a UEFI stub that expects to be able to do
something, distributions must modify Linux to conform to this spec.  I
think imagining that this spec controls how UEFI support in Linux/Grub
is done in general would be overreaching.  Additionally, Michael brought
up the fact that it would be non-UEFI compliant.

I know that door #1 may be a pain in terms of libvirt/openstack support
and other related tools, but it feels by far the cleanest and most
long-term solution and is in my oppinion what we shoud shoot for.  We
are early enough in the ARM server/VM game that we should be able to
make the right decisions at this point.

-Christoffer

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 22/03/2014 03:29, Christoffer Dall ha scritto:
> 1. Simply mandate that VM implementations support persistent variables
>    for their UEFI implementation - with whatever constraints that may
>    put on higher level tools.
>
> 2. Require that OSes shipped as part of compliant VM images make no
>    assumption that changes to the UEFI environment will be stored.
>
> I feel that option number two will break in all sorts of cases, just
> like Grant stated above, and it is fundamentally not practical; if a
> distribution ships Linux with a UEFI stub that expects to be able to do
> something, distributions must modify Linux to conform to this spec.  I
> think imagining that this spec controls how UEFI support in Linux/Grub
> is done in general would be overreaching.  Additionally, Michael brought
> up the fact that it would be non-UEFI compliant.

OSes are already able to cope with loss of changes to UEFI environment 
are stored, because losing persistent variables is what happens if you 
copy an image to a new hard disk.

Asking implementations for support of persistent variables is a good 
idea; however, independent of what is in the spec, OSes should not 
expect that users will enable that support---most of them won't.

Paolo

[RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 22/03/2014 03:29, Christoffer Dall ha scritto:
> 1. Simply mandate that VM implementations support persistent variables
>    for their UEFI implementation - with whatever constraints that may
>    put on higher level tools.
>
> 2. Require that OSes shipped as part of compliant VM images make no
>    assumption that changes to the UEFI environment will be stored.
>
> I feel that option number two will break in all sorts of cases, just
> like Grant stated above, and it is fundamentally not practical; if a
> distribution ships Linux with a UEFI stub that expects to be able to do
> something, distributions must modify Linux to conform to this spec.  I
> think imagining that this spec controls how UEFI support in Linux/Grub
> is done in general would be overreaching.  Additionally, Michael brought
> up the fact that it would be non-UEFI compliant.

OSes are already able to cope with loss of changes to UEFI environment 
are stored, because losing persistent variables is what happens if you 
copy an image to a new hard disk.

Asking implementations for support of persistent variables is a good 
idea; however, independent of what is in the spec, OSes should not 
expect that users will enable that support---most of them won't.

Paolo

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 22/03/2014 03:29, Christoffer Dall ha scritto:
> 1. Simply mandate that VM implementations support persistent variables
>    for their UEFI implementation - with whatever constraints that may
>    put on higher level tools.
>
> 2. Require that OSes shipped as part of compliant VM images make no
>    assumption that changes to the UEFI environment will be stored.
>
> I feel that option number two will break in all sorts of cases, just
> like Grant stated above, and it is fundamentally not practical; if a
> distribution ships Linux with a UEFI stub that expects to be able to do
> something, distributions must modify Linux to conform to this spec.  I
> think imagining that this spec controls how UEFI support in Linux/Grub
> is done in general would be overreaching.  Additionally, Michael brought
> up the fact that it would be non-UEFI compliant.

OSes are already able to cope with loss of changes to UEFI environment 
are stored, because losing persistent variables is what happens if you 
copy an image to a new hard disk.

Asking implementations for support of persistent variables is a good 
idea; however, independent of what is in the spec, OSes should not 
expect that users will enable that support---most of them won't.

Paolo

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Fri, 21 Mar 2014 19:29:50 -0700, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> On Fri, Mar 07, 2014 at 08:24:18PM +0800, Grant Likely wrote:
> > On Thu, 6 Mar 2014 12:04:50 +0000, Robie Basak <robie.basak@canonical.com> wrote:
> > > On Thu, Mar 06, 2014 at 12:44:57PM +0100, Laszlo Ersek wrote:
> > > > If I understand correctly, the question is this:
> > > > 
> > > >   Given a hypervisor that doesn't support non-volatile UEFI variables
> > > >   (including BootOrder and Boot####), is it possible to automatically
> > > >   boot a carefully prepared VM image, made available as a fixed media
> > > >   device?
> > > > 
> > > > The answer is "yes". See
> > > 
> > > Right, but I think there is a subsequent problem.
> > > 
> > > >   It is expected that this default boot will load an operating system or
> > > >   a maintenance utility. If this is an operating system setup program it
> > > >   is then responsible for setting the requisite environment variables
> > > >   for subsequent boots. The platform firmware may also decide to recover
> > >         ^^^^^^^^^^^^^^^^
> > > >   or set to a known set of boot options.
> > > 
> > > It seems to me that the guest OS is permitted to assume that persistent
> > > boot variables will work after first boot, for subsequent boots.
> > > 
> > > So, for example, the guest OS might, on bootloader or kernel upgrade,
> > > completely replace the boot mechanism, dropping the removable path and
> > > replacing it with a fixed disk arrangement, setting boot variables
> > > appropriately, and assume that it can reboot and everything will
> > > continue to work.
> > > 
> > > But if the host does not support non-volatile variables, then this will
> > > break.
> > 
> > Correct
> > 
> > > This is why I'm suggesting that the specification mandate that the guest
> > > OS shipped in a "portable disk image" as defined by the spec must not
> > > make this assumption.
> > 
> > Also correct... the installer must be aware of this constraint which is
> > why it is part of the draft spec.
> > 
> > > It's either this, or mandate that all hosts must support persistent
> > > variables. I have no objection to that in principle, but since we have
> > > no implementation currently, it seems easier to avoid this particular
> > > roadblock by tweaking the spec in a way that nobody seems to care about
> > > anyway.
> > 
> > Right. I guess my position is that if persistent storage is not
> > implemented then there are a number of install/upgrade scenarios that
> > won't work. Regardless, portable images must assume an empty boot list
> > and we can build that into the spec.
> > 
> 
> Sorry for the delay in responding - all sorts of unexpected things
> happened when I returned from LCA14.
> 
> I agree on the technical discussion going on here.  My conclusion is
> that we have two options:
> 
> 1. Simply mandate that VM implementations support persistent variables
>    for their UEFI implementation - with whatever constraints that may
>    put on higher level tools.
> 
> 2. Require that OSes shipped as part of compliant VM images make no
>    assumption that changes to the UEFI environment will be stored.
> 
> I feel that option number two will break in all sorts of cases, just
> like Grant stated above, and it is fundamentally not practical; if a
> distribution ships Linux with a UEFI stub that expects to be able to do
> something, distributions must modify Linux to conform to this spec.  I
> think imagining that this spec controls how UEFI support in Linux/Grub
> is done in general would be overreaching.  Additionally, Michael brought
> up the fact that it would be non-UEFI compliant.

That isn't actually my position. I absolutely think that VMs /should/
implement persistent variables, but the variables are a property of a VM
instance, not of the disk image. As far as this spec is concerned, I
think portable disk images should operate under the assumption of an
empty set of variables, and therefore follow the removable disk
requirements in the UEFI spec.

I would propose a modification to option 1:

VM implementations SHOULD to implement persistent variables for
their UEFI implementation - with whatever constraints that may be put on
higher level tools. Variable storage SHALL be a property of a VM
instance, but SHALL NOT be stored as part of a portable disk image.
Portable disk images SHALL conform to the UEFI removable disk
requirements from the UEFI spec.

g.

[RFC] ARM VM System Sepcification

[Grant Likely]

On Fri, 21 Mar 2014 19:29:50 -0700, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> On Fri, Mar 07, 2014 at 08:24:18PM +0800, Grant Likely wrote:
> > On Thu, 6 Mar 2014 12:04:50 +0000, Robie Basak <robie.basak@canonical.com> wrote:
> > > On Thu, Mar 06, 2014 at 12:44:57PM +0100, Laszlo Ersek wrote:
> > > > If I understand correctly, the question is this:
> > > > 
> > > >   Given a hypervisor that doesn't support non-volatile UEFI variables
> > > >   (including BootOrder and Boot####), is it possible to automatically
> > > >   boot a carefully prepared VM image, made available as a fixed media
> > > >   device?
> > > > 
> > > > The answer is "yes". See
> > > 
> > > Right, but I think there is a subsequent problem.
> > > 
> > > >   It is expected that this default boot will load an operating system or
> > > >   a maintenance utility. If this is an operating system setup program it
> > > >   is then responsible for setting the requisite environment variables
> > > >   for subsequent boots. The platform firmware may also decide to recover
> > >         ^^^^^^^^^^^^^^^^
> > > >   or set to a known set of boot options.
> > > 
> > > It seems to me that the guest OS is permitted to assume that persistent
> > > boot variables will work after first boot, for subsequent boots.
> > > 
> > > So, for example, the guest OS might, on bootloader or kernel upgrade,
> > > completely replace the boot mechanism, dropping the removable path and
> > > replacing it with a fixed disk arrangement, setting boot variables
> > > appropriately, and assume that it can reboot and everything will
> > > continue to work.
> > > 
> > > But if the host does not support non-volatile variables, then this will
> > > break.
> > 
> > Correct
> > 
> > > This is why I'm suggesting that the specification mandate that the guest
> > > OS shipped in a "portable disk image" as defined by the spec must not
> > > make this assumption.
> > 
> > Also correct... the installer must be aware of this constraint which is
> > why it is part of the draft spec.
> > 
> > > It's either this, or mandate that all hosts must support persistent
> > > variables. I have no objection to that in principle, but since we have
> > > no implementation currently, it seems easier to avoid this particular
> > > roadblock by tweaking the spec in a way that nobody seems to care about
> > > anyway.
> > 
> > Right. I guess my position is that if persistent storage is not
> > implemented then there are a number of install/upgrade scenarios that
> > won't work. Regardless, portable images must assume an empty boot list
> > and we can build that into the spec.
> > 
> 
> Sorry for the delay in responding - all sorts of unexpected things
> happened when I returned from LCA14.
> 
> I agree on the technical discussion going on here.  My conclusion is
> that we have two options:
> 
> 1. Simply mandate that VM implementations support persistent variables
>    for their UEFI implementation - with whatever constraints that may
>    put on higher level tools.
> 
> 2. Require that OSes shipped as part of compliant VM images make no
>    assumption that changes to the UEFI environment will be stored.
> 
> I feel that option number two will break in all sorts of cases, just
> like Grant stated above, and it is fundamentally not practical; if a
> distribution ships Linux with a UEFI stub that expects to be able to do
> something, distributions must modify Linux to conform to this spec.  I
> think imagining that this spec controls how UEFI support in Linux/Grub
> is done in general would be overreaching.  Additionally, Michael brought
> up the fact that it would be non-UEFI compliant.

That isn't actually my position. I absolutely think that VMs /should/
implement persistent variables, but the variables are a property of a VM
instance, not of the disk image. As far as this spec is concerned, I
think portable disk images should operate under the assumption of an
empty set of variables, and therefore follow the removable disk
requirements in the UEFI spec.

I would propose a modification to option 1:

VM implementations SHOULD to implement persistent variables for
their UEFI implementation - with whatever constraints that may be put on
higher level tools. Variable storage SHALL be a property of a VM
instance, but SHALL NOT be stored as part of a portable disk image.
Portable disk images SHALL conform to the UEFI removable disk
requirements from the UEFI spec.

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Fri, 21 Mar 2014 19:29:50 -0700, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> On Fri, Mar 07, 2014 at 08:24:18PM +0800, Grant Likely wrote:
> > On Thu, 6 Mar 2014 12:04:50 +0000, Robie Basak <robie.basak@canonical.com> wrote:
> > > On Thu, Mar 06, 2014 at 12:44:57PM +0100, Laszlo Ersek wrote:
> > > > If I understand correctly, the question is this:
> > > > 
> > > >   Given a hypervisor that doesn't support non-volatile UEFI variables
> > > >   (including BootOrder and Boot####), is it possible to automatically
> > > >   boot a carefully prepared VM image, made available as a fixed media
> > > >   device?
> > > > 
> > > > The answer is "yes". See
> > > 
> > > Right, but I think there is a subsequent problem.
> > > 
> > > >   It is expected that this default boot will load an operating system or
> > > >   a maintenance utility. If this is an operating system setup program it
> > > >   is then responsible for setting the requisite environment variables
> > > >   for subsequent boots. The platform firmware may also decide to recover
> > >         ^^^^^^^^^^^^^^^^
> > > >   or set to a known set of boot options.
> > > 
> > > It seems to me that the guest OS is permitted to assume that persistent
> > > boot variables will work after first boot, for subsequent boots.
> > > 
> > > So, for example, the guest OS might, on bootloader or kernel upgrade,
> > > completely replace the boot mechanism, dropping the removable path and
> > > replacing it with a fixed disk arrangement, setting boot variables
> > > appropriately, and assume that it can reboot and everything will
> > > continue to work.
> > > 
> > > But if the host does not support non-volatile variables, then this will
> > > break.
> > 
> > Correct
> > 
> > > This is why I'm suggesting that the specification mandate that the guest
> > > OS shipped in a "portable disk image" as defined by the spec must not
> > > make this assumption.
> > 
> > Also correct... the installer must be aware of this constraint which is
> > why it is part of the draft spec.
> > 
> > > It's either this, or mandate that all hosts must support persistent
> > > variables. I have no objection to that in principle, but since we have
> > > no implementation currently, it seems easier to avoid this particular
> > > roadblock by tweaking the spec in a way that nobody seems to care about
> > > anyway.
> > 
> > Right. I guess my position is that if persistent storage is not
> > implemented then there are a number of install/upgrade scenarios that
> > won't work. Regardless, portable images must assume an empty boot list
> > and we can build that into the spec.
> > 
> 
> Sorry for the delay in responding - all sorts of unexpected things
> happened when I returned from LCA14.
> 
> I agree on the technical discussion going on here.  My conclusion is
> that we have two options:
> 
> 1. Simply mandate that VM implementations support persistent variables
>    for their UEFI implementation - with whatever constraints that may
>    put on higher level tools.
> 
> 2. Require that OSes shipped as part of compliant VM images make no
>    assumption that changes to the UEFI environment will be stored.
> 
> I feel that option number two will break in all sorts of cases, just
> like Grant stated above, and it is fundamentally not practical; if a
> distribution ships Linux with a UEFI stub that expects to be able to do
> something, distributions must modify Linux to conform to this spec.  I
> think imagining that this spec controls how UEFI support in Linux/Grub
> is done in general would be overreaching.  Additionally, Michael brought
> up the fact that it would be non-UEFI compliant.

That isn't actually my position. I absolutely think that VMs /should/
implement persistent variables, but the variables are a property of a VM
instance, not of the disk image. As far as this spec is concerned, I
think portable disk images should operate under the assumption of an
empty set of variables, and therefore follow the removable disk
requirements in the UEFI spec.

I would propose a modification to option 1:

VM implementations SHOULD to implement persistent variables for
their UEFI implementation - with whatever constraints that may be put on
higher level tools. Variable storage SHALL be a property of a VM
instance, but SHALL NOT be stored as part of a portable disk image.
Portable disk images SHALL conform to the UEFI removable disk
requirements from the UEFI spec.

g.

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 22/03/2014 13:23, Grant Likely ha scritto:
> VM implementations SHOULD to implement persistent variables for
> their UEFI implementation - with whatever constraints that may be put on
> higher level tools. Variable storage SHALL be a property of a VM
> instance, but SHALL NOT be stored as part of a portable disk image.
> Portable disk images SHALL conform to the UEFI removable disk
> requirements from the UEFI spec.

I fully agree with this wording.

Paolo

[RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 22/03/2014 13:23, Grant Likely ha scritto:
> VM implementations SHOULD to implement persistent variables for
> their UEFI implementation - with whatever constraints that may be put on
> higher level tools. Variable storage SHALL be a property of a VM
> instance, but SHALL NOT be stored as part of a portable disk image.
> Portable disk images SHALL conform to the UEFI removable disk
> requirements from the UEFI spec.

I fully agree with this wording.

Paolo

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 22/03/2014 13:23, Grant Likely ha scritto:
> VM implementations SHOULD to implement persistent variables for
> their UEFI implementation - with whatever constraints that may be put on
> higher level tools. Variable storage SHALL be a property of a VM
> instance, but SHALL NOT be stored as part of a portable disk image.
> Portable disk images SHALL conform to the UEFI removable disk
> requirements from the UEFI spec.

I fully agree with this wording.

Paolo

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Sat, 22 Mar 2014 20:57:58 +0100, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 22/03/2014 13:23, Grant Likely ha scritto:
> > VM implementations SHOULD to implement persistent variables for
> > their UEFI implementation - with whatever constraints that may be put on
> > higher level tools. Variable storage SHALL be a property of a VM
> > instance, but SHALL NOT be stored as part of a portable disk image.
> > Portable disk images SHALL conform to the UEFI removable disk
> > requirements from the UEFI spec.
> 
> I fully agree with this wording.

:-)

On a seperate, but related, topic. If we were to define a portable VM
spec that includes the entire hardware configuration, then variables
storage should be included in that.

g.

[RFC] ARM VM System Sepcification

[Grant Likely]

On Sat, 22 Mar 2014 20:57:58 +0100, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 22/03/2014 13:23, Grant Likely ha scritto:
> > VM implementations SHOULD to implement persistent variables for
> > their UEFI implementation - with whatever constraints that may be put on
> > higher level tools. Variable storage SHALL be a property of a VM
> > instance, but SHALL NOT be stored as part of a portable disk image.
> > Portable disk images SHALL conform to the UEFI removable disk
> > requirements from the UEFI spec.
> 
> I fully agree with this wording.

:-)

On a seperate, but related, topic. If we were to define a portable VM
spec that includes the entire hardware configuration, then variables
storage should be included in that.

g.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Sat, 22 Mar 2014 20:57:58 +0100, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 22/03/2014 13:23, Grant Likely ha scritto:
> > VM implementations SHOULD to implement persistent variables for
> > their UEFI implementation - with whatever constraints that may be put on
> > higher level tools. Variable storage SHALL be a property of a VM
> > instance, but SHALL NOT be stored as part of a portable disk image.
> > Portable disk images SHALL conform to the UEFI removable disk
> > requirements from the UEFI spec.
> 
> I fully agree with this wording.

:-)

On a seperate, but related, topic. If we were to define a portable VM
spec that includes the entire hardware configuration, then variables
storage should be included in that.

g.

Re: [RFC] ARM VM System Sepcification

[Michael Casadevall]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On 03/22/2014 03:57 PM, Paolo Bonzini wrote:
> Il 22/03/2014 13:23, Grant Likely ha scritto:
>> VM implementations SHOULD to implement persistent variables for 
>> their UEFI implementation - with whatever constraints that may be
>> put on higher level tools. Variable storage SHALL be a property
>> of a VM instance, but SHALL NOT be stored as part of a portable
>> disk image. Portable disk images SHALL conform to the UEFI
>> removable disk requirements from the UEFI spec.
> 
> I fully agree with this wording.
> 
> Paolo

+1 here

Michael
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTLh75AAoJEHM+GkLSJHY5XfwP/27RPv5+UQKgNJa0kooLEGqD
L5YkcofZOPeAdWZ0gzN9BDhzQ+4un5HhaJKTldwgKogblLmaiN9LMi/affSleF+N
HMqSjbxCUy/KduD8Ib4k7qZDkGuO3PbVGP/1RpcaE5jZ+EvQYHGqwRpzWYZyx7Ve
r4DQhSc5IyR8titm/hiCdFufDznQ+afrxuum8WQIC02H3vPYavDmVaApXrgYwnHA
vfSOjEKI4xfmBfWfrtVBFr8/2ZgP9mSuj9lYukun2nRG5Onk7fsPQSZuLCtk52x9
Hx6/hg5hskLM0cTfQ9yCyW3lUdQJkiLmmtn43EfKTJ80esrWpCzhkVD6b4qsYokd
7CpBHziupDTrjQTGundWOkw07pzsrlq3jOQ5tm4hmsr9qwtq4W2SOFfkM2q4KpfO
j0z2edqRoT32v0eAOhBvkC4cx9rfBVZR/sU6HmKqLM80zEMNcXmMfXgwZPqznlfs
bnT342tJPdQD+Mt215MbgKhxwq3kMAWsjW8/2Ad896lyhk6nU0ZbSIc4A8+HBU3M
Bn7tBmpRzuyExsjFxzkhLMfIAa3/UxQuI7o1WXD5asWHxQTKtPhOYTCyhNq4tBHo
8LU9dcP5Q1vZsAFa28MNjrNECf3hrLPxj06NI13CT6PST67FaScMqsV1ZT8Q9xm+
Cw4V5inPq5BvMpdmIu4S
=uG8J
-----END PGP SIGNATURE-----

[RFC] ARM VM System Sepcification

[Michael Casadevall]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On 03/22/2014 03:57 PM, Paolo Bonzini wrote:
> Il 22/03/2014 13:23, Grant Likely ha scritto:
>> VM implementations SHOULD to implement persistent variables for 
>> their UEFI implementation - with whatever constraints that may be
>> put on higher level tools. Variable storage SHALL be a property
>> of a VM instance, but SHALL NOT be stored as part of a portable
>> disk image. Portable disk images SHALL conform to the UEFI
>> removable disk requirements from the UEFI spec.
> 
> I fully agree with this wording.
> 
> Paolo

+1 here

Michael
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTLh75AAoJEHM+GkLSJHY5XfwP/27RPv5+UQKgNJa0kooLEGqD
L5YkcofZOPeAdWZ0gzN9BDhzQ+4un5HhaJKTldwgKogblLmaiN9LMi/affSleF+N
HMqSjbxCUy/KduD8Ib4k7qZDkGuO3PbVGP/1RpcaE5jZ+EvQYHGqwRpzWYZyx7Ve
r4DQhSc5IyR8titm/hiCdFufDznQ+afrxuum8WQIC02H3vPYavDmVaApXrgYwnHA
vfSOjEKI4xfmBfWfrtVBFr8/2ZgP9mSuj9lYukun2nRG5Onk7fsPQSZuLCtk52x9
Hx6/hg5hskLM0cTfQ9yCyW3lUdQJkiLmmtn43EfKTJ80esrWpCzhkVD6b4qsYokd
7CpBHziupDTrjQTGundWOkw07pzsrlq3jOQ5tm4hmsr9qwtq4W2SOFfkM2q4KpfO
j0z2edqRoT32v0eAOhBvkC4cx9rfBVZR/sU6HmKqLM80zEMNcXmMfXgwZPqznlfs
bnT342tJPdQD+Mt215MbgKhxwq3kMAWsjW8/2Ad896lyhk6nU0ZbSIc4A8+HBU3M
Bn7tBmpRzuyExsjFxzkhLMfIAa3/UxQuI7o1WXD5asWHxQTKtPhOYTCyhNq4tBHo
8LU9dcP5Q1vZsAFa28MNjrNECf3hrLPxj06NI13CT6PST67FaScMqsV1ZT8Q9xm+
Cw4V5inPq5BvMpdmIu4S
=uG8J
-----END PGP SIGNATURE-----

Re: [RFC] ARM VM System Sepcification

[Michael Casadevall]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On 03/22/2014 03:57 PM, Paolo Bonzini wrote:
> Il 22/03/2014 13:23, Grant Likely ha scritto:
>> VM implementations SHOULD to implement persistent variables for 
>> their UEFI implementation - with whatever constraints that may be
>> put on higher level tools. Variable storage SHALL be a property
>> of a VM instance, but SHALL NOT be stored as part of a portable
>> disk image. Portable disk images SHALL conform to the UEFI
>> removable disk requirements from the UEFI spec.
> 
> I fully agree with this wording.
> 
> Paolo

+1 here

Michael
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTLh75AAoJEHM+GkLSJHY5XfwP/27RPv5+UQKgNJa0kooLEGqD
L5YkcofZOPeAdWZ0gzN9BDhzQ+4un5HhaJKTldwgKogblLmaiN9LMi/affSleF+N
HMqSjbxCUy/KduD8Ib4k7qZDkGuO3PbVGP/1RpcaE5jZ+EvQYHGqwRpzWYZyx7Ve
r4DQhSc5IyR8titm/hiCdFufDznQ+afrxuum8WQIC02H3vPYavDmVaApXrgYwnHA
vfSOjEKI4xfmBfWfrtVBFr8/2ZgP9mSuj9lYukun2nRG5Onk7fsPQSZuLCtk52x9
Hx6/hg5hskLM0cTfQ9yCyW3lUdQJkiLmmtn43EfKTJ80esrWpCzhkVD6b4qsYokd
7CpBHziupDTrjQTGundWOkw07pzsrlq3jOQ5tm4hmsr9qwtq4W2SOFfkM2q4KpfO
j0z2edqRoT32v0eAOhBvkC4cx9rfBVZR/sU6HmKqLM80zEMNcXmMfXgwZPqznlfs
bnT342tJPdQD+Mt215MbgKhxwq3kMAWsjW8/2Ad896lyhk6nU0ZbSIc4A8+HBU3M
Bn7tBmpRzuyExsjFxzkhLMfIAa3/UxQuI7o1WXD5asWHxQTKtPhOYTCyhNq4tBHo
8LU9dcP5Q1vZsAFa28MNjrNECf3hrLPxj06NI13CT6PST67FaScMqsV1ZT8Q9xm+
Cw4V5inPq5BvMpdmIu4S
=uG8J
-----END PGP SIGNATURE-----

Re: [RFC] ARM VM System Sepcification

[Laszlo Ersek]

On 03/23/14 00:38, Michael Casadevall wrote:
> On 03/22/2014 03:57 PM, Paolo Bonzini wrote:
>> Il 22/03/2014 13:23, Grant Likely ha scritto:
>>> VM implementations SHOULD to implement persistent variables for 
>>> their UEFI implementation - with whatever constraints that may be
>>> put on higher level tools. Variable storage SHALL be a property
>>> of a VM instance, but SHALL NOT be stored as part of a portable
>>> disk image. Portable disk images SHALL conform to the UEFI
>>> removable disk requirements from the UEFI spec.
>>
>> I fully agree with this wording.
>>
>> Paolo
> 
> +1 here

I cannot resist getting productive here, so I'll mention that the first
"to" in the language should probably be dropped.

I would also replace the "but" with "and".

Happy to contribute, always.
Laszlo
:)

[RFC] ARM VM System Sepcification

[Laszlo Ersek]

On 03/23/14 00:38, Michael Casadevall wrote:
> On 03/22/2014 03:57 PM, Paolo Bonzini wrote:
>> Il 22/03/2014 13:23, Grant Likely ha scritto:
>>> VM implementations SHOULD to implement persistent variables for 
>>> their UEFI implementation - with whatever constraints that may be
>>> put on higher level tools. Variable storage SHALL be a property
>>> of a VM instance, but SHALL NOT be stored as part of a portable
>>> disk image. Portable disk images SHALL conform to the UEFI
>>> removable disk requirements from the UEFI spec.
>>
>> I fully agree with this wording.
>>
>> Paolo
> 
> +1 here

I cannot resist getting productive here, so I'll mention that the first
"to" in the language should probably be dropped.

I would also replace the "but" with "and".

Happy to contribute, always.
Laszlo
:)

Re: [RFC] ARM VM System Sepcification

[Laszlo Ersek]

On 03/23/14 00:38, Michael Casadevall wrote:
> On 03/22/2014 03:57 PM, Paolo Bonzini wrote:
>> Il 22/03/2014 13:23, Grant Likely ha scritto:
>>> VM implementations SHOULD to implement persistent variables for 
>>> their UEFI implementation - with whatever constraints that may be
>>> put on higher level tools. Variable storage SHALL be a property
>>> of a VM instance, but SHALL NOT be stored as part of a portable
>>> disk image. Portable disk images SHALL conform to the UEFI
>>> removable disk requirements from the UEFI spec.
>>
>> I fully agree with this wording.
>>
>> Paolo
> 
> +1 here

I cannot resist getting productive here, so I'll mention that the first
"to" in the language should probably be dropped.

I would also replace the "but" with "and".

Happy to contribute, always.
Laszlo
:)

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Sat, Mar 22, 2014 at 09:08:37AM +0100, Paolo Bonzini wrote:
> Il 22/03/2014 03:29, Christoffer Dall ha scritto:
> >1. Simply mandate that VM implementations support persistent variables
> >   for their UEFI implementation - with whatever constraints that may
> >   put on higher level tools.
> >
> >2. Require that OSes shipped as part of compliant VM images make no
> >   assumption that changes to the UEFI environment will be stored.
> >
> >I feel that option number two will break in all sorts of cases, just
> >like Grant stated above, and it is fundamentally not practical; if a
> >distribution ships Linux with a UEFI stub that expects to be able to do
> >something, distributions must modify Linux to conform to this spec.  I
> >think imagining that this spec controls how UEFI support in Linux/Grub
> >is done in general would be overreaching.  Additionally, Michael brought
> >up the fact that it would be non-UEFI compliant.
> 
> OSes are already able to cope with loss of changes to UEFI
> environment are stored, because losing persistent variables is what
> happens if you copy an image to a new hard disk.
> 
> Asking implementations for support of persistent variables is a good
> idea; however, independent of what is in the spec, OSes should not
> expect that users will enable that support---most of them won't.
> 
OK, fair enough, mandating support for persistent variable storage may
be overreaching but at the same time I feel it is unlikely for this spec
to reach far enough that a generic UEFI Linux loader, for example,
actually follows it, and therefore explicitly mandating that guest OSes
must be completely portable in all that they do is a non-practical
constraint.  That was the request that kicked this discussion off, and
what I was trygint to address.

-Christoffer

[RFC] ARM VM System Sepcification

[Christoffer Dall]

On Sat, Mar 22, 2014 at 09:08:37AM +0100, Paolo Bonzini wrote:
> Il 22/03/2014 03:29, Christoffer Dall ha scritto:
> >1. Simply mandate that VM implementations support persistent variables
> >   for their UEFI implementation - with whatever constraints that may
> >   put on higher level tools.
> >
> >2. Require that OSes shipped as part of compliant VM images make no
> >   assumption that changes to the UEFI environment will be stored.
> >
> >I feel that option number two will break in all sorts of cases, just
> >like Grant stated above, and it is fundamentally not practical; if a
> >distribution ships Linux with a UEFI stub that expects to be able to do
> >something, distributions must modify Linux to conform to this spec.  I
> >think imagining that this spec controls how UEFI support in Linux/Grub
> >is done in general would be overreaching.  Additionally, Michael brought
> >up the fact that it would be non-UEFI compliant.
> 
> OSes are already able to cope with loss of changes to UEFI
> environment are stored, because losing persistent variables is what
> happens if you copy an image to a new hard disk.
> 
> Asking implementations for support of persistent variables is a good
> idea; however, independent of what is in the spec, OSes should not
> expect that users will enable that support---most of them won't.
> 
OK, fair enough, mandating support for persistent variable storage may
be overreaching but at the same time I feel it is unlikely for this spec
to reach far enough that a generic UEFI Linux loader, for example,
actually follows it, and therefore explicitly mandating that guest OSes
must be completely portable in all that they do is a non-practical
constraint.  That was the request that kicked this discussion off, and
what I was trygint to address.

-Christoffer

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Sat, Mar 22, 2014 at 09:08:37AM +0100, Paolo Bonzini wrote:
> Il 22/03/2014 03:29, Christoffer Dall ha scritto:
> >1. Simply mandate that VM implementations support persistent variables
> >   for their UEFI implementation - with whatever constraints that may
> >   put on higher level tools.
> >
> >2. Require that OSes shipped as part of compliant VM images make no
> >   assumption that changes to the UEFI environment will be stored.
> >
> >I feel that option number two will break in all sorts of cases, just
> >like Grant stated above, and it is fundamentally not practical; if a
> >distribution ships Linux with a UEFI stub that expects to be able to do
> >something, distributions must modify Linux to conform to this spec.  I
> >think imagining that this spec controls how UEFI support in Linux/Grub
> >is done in general would be overreaching.  Additionally, Michael brought
> >up the fact that it would be non-UEFI compliant.
> 
> OSes are already able to cope with loss of changes to UEFI
> environment are stored, because losing persistent variables is what
> happens if you copy an image to a new hard disk.
> 
> Asking implementations for support of persistent variables is a good
> idea; however, independent of what is in the spec, OSes should not
> expect that users will enable that support---most of them won't.
> 
OK, fair enough, mandating support for persistent variable storage may
be overreaching but at the same time I feel it is unlikely for this spec
to reach far enough that a generic UEFI Linux loader, for example,
actually follows it, and therefore explicitly mandating that guest OSes
must be completely portable in all that they do is a non-practical
constraint.  That was the request that kicked this discussion off, and
what I was trygint to address.

-Christoffer

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Sat, Mar 22, 2014 at 12:23:54PM +0000, Grant Likely wrote:
> On Fri, 21 Mar 2014 19:29:50 -0700, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> > On Fri, Mar 07, 2014 at 08:24:18PM +0800, Grant Likely wrote:
> > > On Thu, 6 Mar 2014 12:04:50 +0000, Robie Basak <robie.basak@canonical.com> wrote:
> > > > On Thu, Mar 06, 2014 at 12:44:57PM +0100, Laszlo Ersek wrote:
> > > > > If I understand correctly, the question is this:
> > > > > 
> > > > >   Given a hypervisor that doesn't support non-volatile UEFI variables
> > > > >   (including BootOrder and Boot####), is it possible to automatically
> > > > >   boot a carefully prepared VM image, made available as a fixed media
> > > > >   device?
> > > > > 
> > > > > The answer is "yes". See
> > > > 
> > > > Right, but I think there is a subsequent problem.
> > > > 
> > > > >   It is expected that this default boot will load an operating system or
> > > > >   a maintenance utility. If this is an operating system setup program it
> > > > >   is then responsible for setting the requisite environment variables
> > > > >   for subsequent boots. The platform firmware may also decide to recover
> > > >         ^^^^^^^^^^^^^^^^
> > > > >   or set to a known set of boot options.
> > > > 
> > > > It seems to me that the guest OS is permitted to assume that persistent
> > > > boot variables will work after first boot, for subsequent boots.
> > > > 
> > > > So, for example, the guest OS might, on bootloader or kernel upgrade,
> > > > completely replace the boot mechanism, dropping the removable path and
> > > > replacing it with a fixed disk arrangement, setting boot variables
> > > > appropriately, and assume that it can reboot and everything will
> > > > continue to work.
> > > > 
> > > > But if the host does not support non-volatile variables, then this will
> > > > break.
> > > 
> > > Correct
> > > 
> > > > This is why I'm suggesting that the specification mandate that the guest
> > > > OS shipped in a "portable disk image" as defined by the spec must not
> > > > make this assumption.
> > > 
> > > Also correct... the installer must be aware of this constraint which is
> > > why it is part of the draft spec.
> > > 
> > > > It's either this, or mandate that all hosts must support persistent
> > > > variables. I have no objection to that in principle, but since we have
> > > > no implementation currently, it seems easier to avoid this particular
> > > > roadblock by tweaking the spec in a way that nobody seems to care about
> > > > anyway.
> > > 
> > > Right. I guess my position is that if persistent storage is not
> > > implemented then there are a number of install/upgrade scenarios that
> > > won't work. Regardless, portable images must assume an empty boot list
> > > and we can build that into the spec.
> > > 
> > 
> > Sorry for the delay in responding - all sorts of unexpected things
> > happened when I returned from LCA14.
> > 
> > I agree on the technical discussion going on here.  My conclusion is
> > that we have two options:
> > 
> > 1. Simply mandate that VM implementations support persistent variables
> >    for their UEFI implementation - with whatever constraints that may
> >    put on higher level tools.
> > 
> > 2. Require that OSes shipped as part of compliant VM images make no
> >    assumption that changes to the UEFI environment will be stored.
> > 
> > I feel that option number two will break in all sorts of cases, just
> > like Grant stated above, and it is fundamentally not practical; if a
> > distribution ships Linux with a UEFI stub that expects to be able to do
> > something, distributions must modify Linux to conform to this spec.  I
> > think imagining that this spec controls how UEFI support in Linux/Grub
> > is done in general would be overreaching.  Additionally, Michael brought
> > up the fact that it would be non-UEFI compliant.
> 
> That isn't actually my position. I absolutely think that VMs /should/
> implement persistent variables, but the variables are a property of a VM
> instance, not of the disk image. As far as this spec is concerned, I
> think portable disk images should operate under the assumption of an
> empty set of variables, and therefore follow the removable disk
> requirements in the UEFI spec.

I think we may have misunderstood each other a bit here, I didn't mean
anything in my statement above that contradicts what you say.

I am only saying that mandating what OSes do and don't do, once they've
been booted, is beyond the scope of this spec.

Portable VM images should absolutely boot with an empty set of variables
and I completely agree that the persistent variable storage is a
property of the VM.

> 
> I would propose a modification to option 1:
> 
> VM implementations SHOULD to implement persistent variables for
> their UEFI implementation - with whatever constraints that may be put on
> higher level tools. Variable storage SHALL be a property of a VM
> instance, but SHALL NOT be stored as part of a portable disk image.
> Portable disk images SHALL conform to the UEFI removable disk
> requirements from the UEFI spec.
> 
I agree with all of the above.

(For the record I wasn't proposing this text as something that should go
verbatim in the spec, but I may borrow from your wording here).

Thanks,
-Christoffer

[RFC] ARM VM System Sepcification

[Christoffer Dall]

On Sat, Mar 22, 2014 at 12:23:54PM +0000, Grant Likely wrote:
> On Fri, 21 Mar 2014 19:29:50 -0700, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> > On Fri, Mar 07, 2014 at 08:24:18PM +0800, Grant Likely wrote:
> > > On Thu, 6 Mar 2014 12:04:50 +0000, Robie Basak <robie.basak@canonical.com> wrote:
> > > > On Thu, Mar 06, 2014 at 12:44:57PM +0100, Laszlo Ersek wrote:
> > > > > If I understand correctly, the question is this:
> > > > > 
> > > > >   Given a hypervisor that doesn't support non-volatile UEFI variables
> > > > >   (including BootOrder and Boot####), is it possible to automatically
> > > > >   boot a carefully prepared VM image, made available as a fixed media
> > > > >   device?
> > > > > 
> > > > > The answer is "yes". See
> > > > 
> > > > Right, but I think there is a subsequent problem.
> > > > 
> > > > >   It is expected that this default boot will load an operating system or
> > > > >   a maintenance utility. If this is an operating system setup program it
> > > > >   is then responsible for setting the requisite environment variables
> > > > >   for subsequent boots. The platform firmware may also decide to recover
> > > >         ^^^^^^^^^^^^^^^^
> > > > >   or set to a known set of boot options.
> > > > 
> > > > It seems to me that the guest OS is permitted to assume that persistent
> > > > boot variables will work after first boot, for subsequent boots.
> > > > 
> > > > So, for example, the guest OS might, on bootloader or kernel upgrade,
> > > > completely replace the boot mechanism, dropping the removable path and
> > > > replacing it with a fixed disk arrangement, setting boot variables
> > > > appropriately, and assume that it can reboot and everything will
> > > > continue to work.
> > > > 
> > > > But if the host does not support non-volatile variables, then this will
> > > > break.
> > > 
> > > Correct
> > > 
> > > > This is why I'm suggesting that the specification mandate that the guest
> > > > OS shipped in a "portable disk image" as defined by the spec must not
> > > > make this assumption.
> > > 
> > > Also correct... the installer must be aware of this constraint which is
> > > why it is part of the draft spec.
> > > 
> > > > It's either this, or mandate that all hosts must support persistent
> > > > variables. I have no objection to that in principle, but since we have
> > > > no implementation currently, it seems easier to avoid this particular
> > > > roadblock by tweaking the spec in a way that nobody seems to care about
> > > > anyway.
> > > 
> > > Right. I guess my position is that if persistent storage is not
> > > implemented then there are a number of install/upgrade scenarios that
> > > won't work. Regardless, portable images must assume an empty boot list
> > > and we can build that into the spec.
> > > 
> > 
> > Sorry for the delay in responding - all sorts of unexpected things
> > happened when I returned from LCA14.
> > 
> > I agree on the technical discussion going on here.  My conclusion is
> > that we have two options:
> > 
> > 1. Simply mandate that VM implementations support persistent variables
> >    for their UEFI implementation - with whatever constraints that may
> >    put on higher level tools.
> > 
> > 2. Require that OSes shipped as part of compliant VM images make no
> >    assumption that changes to the UEFI environment will be stored.
> > 
> > I feel that option number two will break in all sorts of cases, just
> > like Grant stated above, and it is fundamentally not practical; if a
> > distribution ships Linux with a UEFI stub that expects to be able to do
> > something, distributions must modify Linux to conform to this spec.  I
> > think imagining that this spec controls how UEFI support in Linux/Grub
> > is done in general would be overreaching.  Additionally, Michael brought
> > up the fact that it would be non-UEFI compliant.
> 
> That isn't actually my position. I absolutely think that VMs /should/
> implement persistent variables, but the variables are a property of a VM
> instance, not of the disk image. As far as this spec is concerned, I
> think portable disk images should operate under the assumption of an
> empty set of variables, and therefore follow the removable disk
> requirements in the UEFI spec.

I think we may have misunderstood each other a bit here, I didn't mean
anything in my statement above that contradicts what you say.

I am only saying that mandating what OSes do and don't do, once they've
been booted, is beyond the scope of this spec.

Portable VM images should absolutely boot with an empty set of variables
and I completely agree that the persistent variable storage is a
property of the VM.

> 
> I would propose a modification to option 1:
> 
> VM implementations SHOULD to implement persistent variables for
> their UEFI implementation - with whatever constraints that may be put on
> higher level tools. Variable storage SHALL be a property of a VM
> instance, but SHALL NOT be stored as part of a portable disk image.
> Portable disk images SHALL conform to the UEFI removable disk
> requirements from the UEFI spec.
> 
I agree with all of the above.

(For the record I wasn't proposing this text as something that should go
verbatim in the spec, but I may borrow from your wording here).

Thanks,
-Christoffer

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Sat, Mar 22, 2014 at 12:23:54PM +0000, Grant Likely wrote:
> On Fri, 21 Mar 2014 19:29:50 -0700, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> > On Fri, Mar 07, 2014 at 08:24:18PM +0800, Grant Likely wrote:
> > > On Thu, 6 Mar 2014 12:04:50 +0000, Robie Basak <robie.basak@canonical.com> wrote:
> > > > On Thu, Mar 06, 2014 at 12:44:57PM +0100, Laszlo Ersek wrote:
> > > > > If I understand correctly, the question is this:
> > > > > 
> > > > >   Given a hypervisor that doesn't support non-volatile UEFI variables
> > > > >   (including BootOrder and Boot####), is it possible to automatically
> > > > >   boot a carefully prepared VM image, made available as a fixed media
> > > > >   device?
> > > > > 
> > > > > The answer is "yes". See
> > > > 
> > > > Right, but I think there is a subsequent problem.
> > > > 
> > > > >   It is expected that this default boot will load an operating system or
> > > > >   a maintenance utility. If this is an operating system setup program it
> > > > >   is then responsible for setting the requisite environment variables
> > > > >   for subsequent boots. The platform firmware may also decide to recover
> > > >         ^^^^^^^^^^^^^^^^
> > > > >   or set to a known set of boot options.
> > > > 
> > > > It seems to me that the guest OS is permitted to assume that persistent
> > > > boot variables will work after first boot, for subsequent boots.
> > > > 
> > > > So, for example, the guest OS might, on bootloader or kernel upgrade,
> > > > completely replace the boot mechanism, dropping the removable path and
> > > > replacing it with a fixed disk arrangement, setting boot variables
> > > > appropriately, and assume that it can reboot and everything will
> > > > continue to work.
> > > > 
> > > > But if the host does not support non-volatile variables, then this will
> > > > break.
> > > 
> > > Correct
> > > 
> > > > This is why I'm suggesting that the specification mandate that the guest
> > > > OS shipped in a "portable disk image" as defined by the spec must not
> > > > make this assumption.
> > > 
> > > Also correct... the installer must be aware of this constraint which is
> > > why it is part of the draft spec.
> > > 
> > > > It's either this, or mandate that all hosts must support persistent
> > > > variables. I have no objection to that in principle, but since we have
> > > > no implementation currently, it seems easier to avoid this particular
> > > > roadblock by tweaking the spec in a way that nobody seems to care about
> > > > anyway.
> > > 
> > > Right. I guess my position is that if persistent storage is not
> > > implemented then there are a number of install/upgrade scenarios that
> > > won't work. Regardless, portable images must assume an empty boot list
> > > and we can build that into the spec.
> > > 
> > 
> > Sorry for the delay in responding - all sorts of unexpected things
> > happened when I returned from LCA14.
> > 
> > I agree on the technical discussion going on here.  My conclusion is
> > that we have two options:
> > 
> > 1. Simply mandate that VM implementations support persistent variables
> >    for their UEFI implementation - with whatever constraints that may
> >    put on higher level tools.
> > 
> > 2. Require that OSes shipped as part of compliant VM images make no
> >    assumption that changes to the UEFI environment will be stored.
> > 
> > I feel that option number two will break in all sorts of cases, just
> > like Grant stated above, and it is fundamentally not practical; if a
> > distribution ships Linux with a UEFI stub that expects to be able to do
> > something, distributions must modify Linux to conform to this spec.  I
> > think imagining that this spec controls how UEFI support in Linux/Grub
> > is done in general would be overreaching.  Additionally, Michael brought
> > up the fact that it would be non-UEFI compliant.
> 
> That isn't actually my position. I absolutely think that VMs /should/
> implement persistent variables, but the variables are a property of a VM
> instance, not of the disk image. As far as this spec is concerned, I
> think portable disk images should operate under the assumption of an
> empty set of variables, and therefore follow the removable disk
> requirements in the UEFI spec.

I think we may have misunderstood each other a bit here, I didn't mean
anything in my statement above that contradicts what you say.

I am only saying that mandating what OSes do and don't do, once they've
been booted, is beyond the scope of this spec.

Portable VM images should absolutely boot with an empty set of variables
and I completely agree that the persistent variable storage is a
property of the VM.

> 
> I would propose a modification to option 1:
> 
> VM implementations SHOULD to implement persistent variables for
> their UEFI implementation - with whatever constraints that may be put on
> higher level tools. Variable storage SHALL be a property of a VM
> instance, but SHALL NOT be stored as part of a portable disk image.
> Portable disk images SHALL conform to the UEFI removable disk
> requirements from the UEFI spec.
> 
I agree with all of the above.

(For the record I wasn't proposing this text as something that should go
verbatim in the spec, but I may borrow from your wording here).

Thanks,
-Christoffer

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Sat, Mar 22, 2014 at 08:19:52PM -0700, Christoffer Dall wrote:
> On Sat, Mar 22, 2014 at 09:08:37AM +0100, Paolo Bonzini wrote:
> > Il 22/03/2014 03:29, Christoffer Dall ha scritto:
> > >1. Simply mandate that VM implementations support persistent variables
> > >   for their UEFI implementation - with whatever constraints that may
> > >   put on higher level tools.
> > >
> > >2. Require that OSes shipped as part of compliant VM images make no
> > >   assumption that changes to the UEFI environment will be stored.
> > >
> > >I feel that option number two will break in all sorts of cases, just
> > >like Grant stated above, and it is fundamentally not practical; if a
> > >distribution ships Linux with a UEFI stub that expects to be able to do
> > >something, distributions must modify Linux to conform to this spec.  I
> > >think imagining that this spec controls how UEFI support in Linux/Grub
> > >is done in general would be overreaching.  Additionally, Michael brought
> > >up the fact that it would be non-UEFI compliant.
> > 
> > OSes are already able to cope with loss of changes to UEFI
> > environment are stored, because losing persistent variables is what
> > happens if you copy an image to a new hard disk.
> > 
> > Asking implementations for support of persistent variables is a good
> > idea; however, independent of what is in the spec, OSes should not
> > expect that users will enable that support---most of them won't.
> > 
> OK, fair enough, mandating support for persistent variable storage may
> be overreaching but at the same time I feel it is unlikely for this spec
> to reach far enough that a generic UEFI Linux loader, for example,
> actually follows it, and therefore explicitly mandating that guest OSes
> must be completely portable in all that they do is a non-practical
> constraint.  That was the request that kicked this discussion off, and
> what I was trygint to address.
> 
After thinking about this a bit more, I think I see what we're actually
discussing.  It's obvious that if software in a VM makes changes to UEFI
variables that are required to be persistent for that VM image to boot
again, then the VM image is no longer portable, as per the spec.

It is in fact probably a good idea to specify that out clearly, I found
it intuitively obvious, but it could be spelled out nevertheless.

-Christoffer

[RFC] ARM VM System Sepcification

[Christoffer Dall]

On Sat, Mar 22, 2014 at 08:19:52PM -0700, Christoffer Dall wrote:
> On Sat, Mar 22, 2014 at 09:08:37AM +0100, Paolo Bonzini wrote:
> > Il 22/03/2014 03:29, Christoffer Dall ha scritto:
> > >1. Simply mandate that VM implementations support persistent variables
> > >   for their UEFI implementation - with whatever constraints that may
> > >   put on higher level tools.
> > >
> > >2. Require that OSes shipped as part of compliant VM images make no
> > >   assumption that changes to the UEFI environment will be stored.
> > >
> > >I feel that option number two will break in all sorts of cases, just
> > >like Grant stated above, and it is fundamentally not practical; if a
> > >distribution ships Linux with a UEFI stub that expects to be able to do
> > >something, distributions must modify Linux to conform to this spec.  I
> > >think imagining that this spec controls how UEFI support in Linux/Grub
> > >is done in general would be overreaching.  Additionally, Michael brought
> > >up the fact that it would be non-UEFI compliant.
> > 
> > OSes are already able to cope with loss of changes to UEFI
> > environment are stored, because losing persistent variables is what
> > happens if you copy an image to a new hard disk.
> > 
> > Asking implementations for support of persistent variables is a good
> > idea; however, independent of what is in the spec, OSes should not
> > expect that users will enable that support---most of them won't.
> > 
> OK, fair enough, mandating support for persistent variable storage may
> be overreaching but at the same time I feel it is unlikely for this spec
> to reach far enough that a generic UEFI Linux loader, for example,
> actually follows it, and therefore explicitly mandating that guest OSes
> must be completely portable in all that they do is a non-practical
> constraint.  That was the request that kicked this discussion off, and
> what I was trygint to address.
> 
After thinking about this a bit more, I think I see what we're actually
discussing.  It's obvious that if software in a VM makes changes to UEFI
variables that are required to be persistent for that VM image to boot
again, then the VM image is no longer portable, as per the spec.

It is in fact probably a good idea to specify that out clearly, I found
it intuitively obvious, but it could be spelled out nevertheless.

-Christoffer

Re: [RFC] ARM VM System Sepcification

[Christoffer Dall]

On Sat, Mar 22, 2014 at 08:19:52PM -0700, Christoffer Dall wrote:
> On Sat, Mar 22, 2014 at 09:08:37AM +0100, Paolo Bonzini wrote:
> > Il 22/03/2014 03:29, Christoffer Dall ha scritto:
> > >1. Simply mandate that VM implementations support persistent variables
> > >   for their UEFI implementation - with whatever constraints that may
> > >   put on higher level tools.
> > >
> > >2. Require that OSes shipped as part of compliant VM images make no
> > >   assumption that changes to the UEFI environment will be stored.
> > >
> > >I feel that option number two will break in all sorts of cases, just
> > >like Grant stated above, and it is fundamentally not practical; if a
> > >distribution ships Linux with a UEFI stub that expects to be able to do
> > >something, distributions must modify Linux to conform to this spec.  I
> > >think imagining that this spec controls how UEFI support in Linux/Grub
> > >is done in general would be overreaching.  Additionally, Michael brought
> > >up the fact that it would be non-UEFI compliant.
> > 
> > OSes are already able to cope with loss of changes to UEFI
> > environment are stored, because losing persistent variables is what
> > happens if you copy an image to a new hard disk.
> > 
> > Asking implementations for support of persistent variables is a good
> > idea; however, independent of what is in the spec, OSes should not
> > expect that users will enable that support---most of them won't.
> > 
> OK, fair enough, mandating support for persistent variable storage may
> be overreaching but at the same time I feel it is unlikely for this spec
> to reach far enough that a generic UEFI Linux loader, for example,
> actually follows it, and therefore explicitly mandating that guest OSes
> must be completely portable in all that they do is a non-practical
> constraint.  That was the request that kicked this discussion off, and
> what I was trygint to address.
> 
After thinking about this a bit more, I think I see what we're actually
discussing.  It's obvious that if software in a VM makes changes to UEFI
variables that are required to be persistent for that VM image to boot
again, then the VM image is no longer portable, as per the spec.

It is in fact probably a good idea to specify that out clearly, I found
it intuitively obvious, but it could be spelled out nevertheless.

-Christoffer

Re: [RFC] ARM VM System Sepcification

[Ian Campbell]

On Sat, 2014-03-22 at 12:23 +0000, Grant Likely wrote:
> That isn't actually my position. I absolutely think that VMs /should/
> implement persistent variables, but the variables are a property of a VM
> instance, not of the disk image. As far as this spec is concerned, I
> think portable disk images should operate under the assumption of an
> empty set of variables, and therefore follow the removable disk
> requirements in the UEFI spec.

Just to be sure I understand. You position is:
     1. A VM image downloaded from www.distro.org should neither contain
        nor expect any persistent variables to be present.
     2. After a VM image is instantiated into a specific VM instance and
        booted then it is at liberty to set persistent variables (either
        on first boot or as part of an upgrade) and the VM should ensure
        that those variables a retained over reboot for that specific
        instance.
     3. If a VM does not preserve those variables then the instance
        should have some sane functional fallback (implied by the
        removable disk requirements from the UEFI spec).

Is that right? I'm pretty sure you meant (1), reasonably sure you meant
(2) and not at all sure you meant (3) ;-)

Ian.

[RFC] ARM VM System Sepcification

[Ian Campbell]

On Sat, 2014-03-22 at 12:23 +0000, Grant Likely wrote:
> That isn't actually my position. I absolutely think that VMs /should/
> implement persistent variables, but the variables are a property of a VM
> instance, not of the disk image. As far as this spec is concerned, I
> think portable disk images should operate under the assumption of an
> empty set of variables, and therefore follow the removable disk
> requirements in the UEFI spec.

Just to be sure I understand. You position is:
     1. A VM image downloaded from www.distro.org should neither contain
        nor expect any persistent variables to be present.
     2. After a VM image is instantiated into a specific VM instance and
        booted then it is at liberty to set persistent variables (either
        on first boot or as part of an upgrade) and the VM should ensure
        that those variables a retained over reboot for that specific
        instance.
     3. If a VM does not preserve those variables then the instance
        should have some sane functional fallback (implied by the
        removable disk requirements from the UEFI spec).

Is that right? I'm pretty sure you meant (1), reasonably sure you meant
(2) and not at all sure you meant (3) ;-)

Ian.

Re: [RFC] ARM VM System Sepcification

[Ian Campbell]

On Sat, 2014-03-22 at 12:23 +0000, Grant Likely wrote:
> That isn't actually my position. I absolutely think that VMs /should/
> implement persistent variables, but the variables are a property of a VM
> instance, not of the disk image. As far as this spec is concerned, I
> think portable disk images should operate under the assumption of an
> empty set of variables, and therefore follow the removable disk
> requirements in the UEFI spec.

Just to be sure I understand. You position is:
     1. A VM image downloaded from www.distro.org should neither contain
        nor expect any persistent variables to be present.
     2. After a VM image is instantiated into a specific VM instance and
        booted then it is at liberty to set persistent variables (either
        on first boot or as part of an upgrade) and the VM should ensure
        that those variables a retained over reboot for that specific
        instance.
     3. If a VM does not preserve those variables then the instance
        should have some sane functional fallback (implied by the
        removable disk requirements from the UEFI spec).

Is that right? I'm pretty sure you meant (1), reasonably sure you meant
(2) and not at all sure you meant (3) ;-)

Ian.

Re: [RFC] ARM VM System Sepcification

[Robie Basak]

[-- Attachment #1: Type: text/plain, Size: 741 bytes --]

Thank you all for considering this case in more detail.

On Sat, Mar 22, 2014 at 08:29:39PM -0700, Christoffer Dall wrote:
> After thinking about this a bit more, I think I see what we're actually
> discussing.  It's obvious that if software in a VM makes changes to UEFI
> variables that are required to be persistent for that VM image to boot
> again, then the VM image is no longer portable, as per the spec.

No longer portable, and given the current state of implementation, no
longer bootable, since we don't support persistent storage yet;
certainly not on OpenStack? Or do we have that now?

Are we really pushing ahead with a specification that nobody can
implement today? How far away are we from a fully compliant
implementation?

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[RFC] ARM VM System Sepcification

[Robie Basak]

Thank you all for considering this case in more detail.

On Sat, Mar 22, 2014 at 08:29:39PM -0700, Christoffer Dall wrote:
> After thinking about this a bit more, I think I see what we're actually
> discussing.  It's obvious that if software in a VM makes changes to UEFI
> variables that are required to be persistent for that VM image to boot
> again, then the VM image is no longer portable, as per the spec.

No longer portable, and given the current state of implementation, no
longer bootable, since we don't support persistent storage yet;
certainly not on OpenStack? Or do we have that now?

Are we really pushing ahead with a specification that nobody can
implement today? How far away are we from a fully compliant
implementation?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140324/6e1da7b1/attachment.sig>

Re: [RFC] ARM VM System Sepcification

[Robie Basak]

[-- Attachment #1.1: Type: text/plain, Size: 741 bytes --]

Thank you all for considering this case in more detail.

On Sat, Mar 22, 2014 at 08:29:39PM -0700, Christoffer Dall wrote:
> After thinking about this a bit more, I think I see what we're actually
> discussing.  It's obvious that if software in a VM makes changes to UEFI
> variables that are required to be persistent for that VM image to boot
> again, then the VM image is no longer portable, as per the spec.

No longer portable, and given the current state of implementation, no
longer bootable, since we don't support persistent storage yet;
certainly not on OpenStack? Or do we have that now?

Are we really pushing ahead with a specification that nobody can
implement today? How far away are we from a fully compliant
implementation?

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 24/03/2014 10:03, Ian Campbell ha scritto:
>> > That isn't actually my position. I absolutely think that VMs /should/
>> > implement persistent variables, but the variables are a property of a VM
>> > instance, not of the disk image. As far as this spec is concerned, I
>> > think portable disk images should operate under the assumption of an
>> > empty set of variables, and therefore follow the removable disk
>> > requirements in the UEFI spec.
> Just to be sure I understand. You position is:
>      1. A VM image downloaded from www.distro.org should neither contain
>         nor expect any persistent variables to be present.
>      2. After a VM image is instantiated into a specific VM instance and
>         booted then it is at liberty to set persistent variables (either
>         on first boot or as part of an upgrade) and the VM should ensure
>         that those variables a retained over reboot for that specific
>         instance.
>      3. If a VM does not preserve those variables then the instance
>         should have some sane functional fallback (implied by the
>         removable disk requirements from the UEFI spec).
>
> Is that right? I'm pretty sure you meant (1), reasonably sure you meant
> (2) and not at all sure you meant (3) ;-)

At least I do. :)

Paolo

[RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 24/03/2014 10:03, Ian Campbell ha scritto:
>> > That isn't actually my position. I absolutely think that VMs /should/
>> > implement persistent variables, but the variables are a property of a VM
>> > instance, not of the disk image. As far as this spec is concerned, I
>> > think portable disk images should operate under the assumption of an
>> > empty set of variables, and therefore follow the removable disk
>> > requirements in the UEFI spec.
> Just to be sure I understand. You position is:
>      1. A VM image downloaded from www.distro.org should neither contain
>         nor expect any persistent variables to be present.
>      2. After a VM image is instantiated into a specific VM instance and
>         booted then it is at liberty to set persistent variables (either
>         on first boot or as part of an upgrade) and the VM should ensure
>         that those variables a retained over reboot for that specific
>         instance.
>      3. If a VM does not preserve those variables then the instance
>         should have some sane functional fallback (implied by the
>         removable disk requirements from the UEFI spec).
>
> Is that right? I'm pretty sure you meant (1), reasonably sure you meant
> (2) and not at all sure you meant (3) ;-)

At least I do. :)

Paolo

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 24/03/2014 10:03, Ian Campbell ha scritto:
>> > That isn't actually my position. I absolutely think that VMs /should/
>> > implement persistent variables, but the variables are a property of a VM
>> > instance, not of the disk image. As far as this spec is concerned, I
>> > think portable disk images should operate under the assumption of an
>> > empty set of variables, and therefore follow the removable disk
>> > requirements in the UEFI spec.
> Just to be sure I understand. You position is:
>      1. A VM image downloaded from www.distro.org should neither contain
>         nor expect any persistent variables to be present.
>      2. After a VM image is instantiated into a specific VM instance and
>         booted then it is at liberty to set persistent variables (either
>         on first boot or as part of an upgrade) and the VM should ensure
>         that those variables a retained over reboot for that specific
>         instance.
>      3. If a VM does not preserve those variables then the instance
>         should have some sane functional fallback (implied by the
>         removable disk requirements from the UEFI spec).
>
> Is that right? I'm pretty sure you meant (1), reasonably sure you meant
> (2) and not at all sure you meant (3) ;-)

At least I do. :)

Paolo

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 24/03/2014 10:57, Robie Basak ha scritto:
>> > After thinking about this a bit more, I think I see what we're actually
>> > discussing.  It's obvious that if software in a VM makes changes to UEFI
>> > variables that are required to be persistent for that VM image to boot
>> > again, then the VM image is no longer portable, as per the spec.
> No longer portable, and given the current state of implementation, no
> longer bootable, since we don't support persistent storage yet;
> certainly not on OpenStack? Or do we have that now?
>
> Are we really pushing ahead with a specification that nobody can
> implement today? How far away are we from a fully compliant
> implementation?

The spec says SHOULD, so I think it's fine.  While support for 
persistent variables in the KVM stack is in the early stages, there is 
request and it is not ARM-specific.  It will be implemented sooner 
rather than later at least at the libvirt level, and I suppose qemu-arm, 
xl, OpenStack/, Virt, XenServer and everything else will follow soon.

Paolo

[RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 24/03/2014 10:57, Robie Basak ha scritto:
>> > After thinking about this a bit more, I think I see what we're actually
>> > discussing.  It's obvious that if software in a VM makes changes to UEFI
>> > variables that are required to be persistent for that VM image to boot
>> > again, then the VM image is no longer portable, as per the spec.
> No longer portable, and given the current state of implementation, no
> longer bootable, since we don't support persistent storage yet;
> certainly not on OpenStack? Or do we have that now?
>
> Are we really pushing ahead with a specification that nobody can
> implement today? How far away are we from a fully compliant
> implementation?

The spec says SHOULD, so I think it's fine.  While support for 
persistent variables in the KVM stack is in the early stages, there is 
request and it is not ARM-specific.  It will be implemented sooner 
rather than later at least at the libvirt level, and I suppose qemu-arm, 
xl, OpenStack/, Virt, XenServer and everything else will follow soon.

Paolo

Re: [RFC] ARM VM System Sepcification

[Paolo Bonzini]

Il 24/03/2014 10:57, Robie Basak ha scritto:
>> > After thinking about this a bit more, I think I see what we're actually
>> > discussing.  It's obvious that if software in a VM makes changes to UEFI
>> > variables that are required to be persistent for that VM image to boot
>> > again, then the VM image is no longer portable, as per the spec.
> No longer portable, and given the current state of implementation, no
> longer bootable, since we don't support persistent storage yet;
> certainly not on OpenStack? Or do we have that now?
>
> Are we really pushing ahead with a specification that nobody can
> implement today? How far away are we from a fully compliant
> implementation?

The spec says SHOULD, so I think it's fine.  While support for 
persistent variables in the KVM stack is in the early stages, there is 
request and it is not ARM-specific.  It will be implemented sooner 
rather than later at least at the libvirt level, and I suppose qemu-arm, 
xl, OpenStack/, Virt, XenServer and everything else will follow soon.

Paolo

Re: [RFC] ARM VM System Sepcification

[Ian Campbell]

On Mon, 2014-03-24 at 11:41 +0100, Paolo Bonzini wrote:
> Il 24/03/2014 10:03, Ian Campbell ha scritto:
> >> > That isn't actually my position. I absolutely think that VMs /should/
> >> > implement persistent variables, but the variables are a property of a VM
> >> > instance, not of the disk image. As far as this spec is concerned, I
> >> > think portable disk images should operate under the assumption of an
> >> > empty set of variables, and therefore follow the removable disk
> >> > requirements in the UEFI spec.
> > Just to be sure I understand. You position is:
> >      1. A VM image downloaded from www.distro.org should neither contain
> >         nor expect any persistent variables to be present.

I suppose for completeness I should have had 1a here:
        When a VM image is instantiated into a specific VM instance then
        it must not expect or require any persistent variables to be
        present.

> >      2. After a VM image is instantiated into a specific VM instance and
> >         booted then it is at liberty to set persistent variables (either
> >         on first boot or as part of an upgrade) and the VM should ensure
> >         that those variables a retained over reboot for that specific
> >         instance.
> >      3. If a VM does not preserve those variables then the instance
> >         should have some sane functional fallback (implied by the
> >         removable disk requirements from the UEFI spec).
> >
> > Is that right? I'm pretty sure you meant (1), reasonably sure you meant
> > (2) and not at all sure you meant (3) ;-)
> 
> At least I do. :)

You did mean it, or you do think Grant meant it? ;-)

Ian.

[RFC] ARM VM System Sepcification

[Ian Campbell]

On Mon, 2014-03-24 at 11:41 +0100, Paolo Bonzini wrote:
> Il 24/03/2014 10:03, Ian Campbell ha scritto:
> >> > That isn't actually my position. I absolutely think that VMs /should/
> >> > implement persistent variables, but the variables are a property of a VM
> >> > instance, not of the disk image. As far as this spec is concerned, I
> >> > think portable disk images should operate under the assumption of an
> >> > empty set of variables, and therefore follow the removable disk
> >> > requirements in the UEFI spec.
> > Just to be sure I understand. You position is:
> >      1. A VM image downloaded from www.distro.org should neither contain
> >         nor expect any persistent variables to be present.

I suppose for completeness I should have had 1a here:
        When a VM image is instantiated into a specific VM instance then
        it must not expect or require any persistent variables to be
        present.

> >      2. After a VM image is instantiated into a specific VM instance and
> >         booted then it is at liberty to set persistent variables (either
> >         on first boot or as part of an upgrade) and the VM should ensure
> >         that those variables a retained over reboot for that specific
> >         instance.
> >      3. If a VM does not preserve those variables then the instance
> >         should have some sane functional fallback (implied by the
> >         removable disk requirements from the UEFI spec).
> >
> > Is that right? I'm pretty sure you meant (1), reasonably sure you meant
> > (2) and not at all sure you meant (3) ;-)
> 
> At least I do. :)

You did mean it, or you do think Grant meant it? ;-)

Ian.

Re: [RFC] ARM VM System Sepcification

[Ian Campbell]

On Mon, 2014-03-24 at 11:41 +0100, Paolo Bonzini wrote:
> Il 24/03/2014 10:03, Ian Campbell ha scritto:
> >> > That isn't actually my position. I absolutely think that VMs /should/
> >> > implement persistent variables, but the variables are a property of a VM
> >> > instance, not of the disk image. As far as this spec is concerned, I
> >> > think portable disk images should operate under the assumption of an
> >> > empty set of variables, and therefore follow the removable disk
> >> > requirements in the UEFI spec.
> > Just to be sure I understand. You position is:
> >      1. A VM image downloaded from www.distro.org should neither contain
> >         nor expect any persistent variables to be present.

I suppose for completeness I should have had 1a here:
        When a VM image is instantiated into a specific VM instance then
        it must not expect or require any persistent variables to be
        present.

> >      2. After a VM image is instantiated into a specific VM instance and
> >         booted then it is at liberty to set persistent variables (either
> >         on first boot or as part of an upgrade) and the VM should ensure
> >         that those variables a retained over reboot for that specific
> >         instance.
> >      3. If a VM does not preserve those variables then the instance
> >         should have some sane functional fallback (implied by the
> >         removable disk requirements from the UEFI spec).
> >
> > Is that right? I'm pretty sure you meant (1), reasonably sure you meant
> > (2) and not at all sure you meant (3) ;-)
> 
> At least I do. :)

You did mean it, or you do think Grant meant it? ;-)

Ian.

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Mon, Mar 24, 2014 at 2:03 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Sat, 2014-03-22 at 12:23 +0000, Grant Likely wrote:
>> That isn't actually my position. I absolutely think that VMs /should/
>> implement persistent variables, but the variables are a property of a VM
>> instance, not of the disk image. As far as this spec is concerned, I
>> think portable disk images should operate under the assumption of an
>> empty set of variables, and therefore follow the removable disk
>> requirements in the UEFI spec.
>
> Just to be sure I understand. You position is:
>      1. A VM image downloaded from www.distro.org should neither contain
>         nor expect any persistent variables to be present.

yes

>      2. After a VM image is instantiated into a specific VM instance and
>         booted then it is at liberty to set persistent variables (either
>         on first boot or as part of an upgrade) and the VM should ensure
>         that those variables a retained over reboot for that specific
>         instance.

yes

>      3. If a VM does not preserve those variables then the instance
>         should have some sane functional fallback (implied by the
>         removable disk requirements from the UEFI spec).

yes

> Is that right? I'm pretty sure you meant (1), reasonably sure you meant
> (2) and not at all sure you meant (3) ;-)
>
> Ian.
>

[RFC] ARM VM System Sepcification

[Grant Likely]

On Mon, Mar 24, 2014 at 2:03 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Sat, 2014-03-22 at 12:23 +0000, Grant Likely wrote:
>> That isn't actually my position. I absolutely think that VMs /should/
>> implement persistent variables, but the variables are a property of a VM
>> instance, not of the disk image. As far as this spec is concerned, I
>> think portable disk images should operate under the assumption of an
>> empty set of variables, and therefore follow the removable disk
>> requirements in the UEFI spec.
>
> Just to be sure I understand. You position is:
>      1. A VM image downloaded from www.distro.org should neither contain
>         nor expect any persistent variables to be present.

yes

>      2. After a VM image is instantiated into a specific VM instance and
>         booted then it is at liberty to set persistent variables (either
>         on first boot or as part of an upgrade) and the VM should ensure
>         that those variables a retained over reboot for that specific
>         instance.

yes

>      3. If a VM does not preserve those variables then the instance
>         should have some sane functional fallback (implied by the
>         removable disk requirements from the UEFI spec).

yes

> Is that right? I'm pretty sure you meant (1), reasonably sure you meant
> (2) and not at all sure you meant (3) ;-)
>
> Ian.
>

Re: [RFC] ARM VM System Sepcification

[Grant Likely]

On Mon, Mar 24, 2014 at 2:03 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Sat, 2014-03-22 at 12:23 +0000, Grant Likely wrote:
>> That isn't actually my position. I absolutely think that VMs /should/
>> implement persistent variables, but the variables are a property of a VM
>> instance, not of the disk image. As far as this spec is concerned, I
>> think portable disk images should operate under the assumption of an
>> empty set of variables, and therefore follow the removable disk
>> requirements in the UEFI spec.
>
> Just to be sure I understand. You position is:
>      1. A VM image downloaded from www.distro.org should neither contain
>         nor expect any persistent variables to be present.

yes

>      2. After a VM image is instantiated into a specific VM instance and
>         booted then it is at liberty to set persistent variables (either
>         on first boot or as part of an upgrade) and the VM should ensure
>         that those variables a retained over reboot for that specific
>         instance.

yes

>      3. If a VM does not preserve those variables then the instance
>         should have some sane functional fallback (implied by the
>         removable disk requirements from the UEFI spec).

yes

> Is that right? I'm pretty sure you meant (1), reasonably sure you meant
> (2) and not at all sure you meant (3) ;-)
>
> Ian.
>

Re: [RFC] ARM VM System Sepcification

[Ian Campbell]

On Mon, 2014-03-24 at 05:13 -0700, Grant Likely wrote:
> yes
> yes
> yes

Great, thanks. I think this sounds reasonable.

Ian.

[RFC] ARM VM System Sepcification

[Ian Campbell]

On Mon, 2014-03-24 at 05:13 -0700, Grant Likely wrote:
> yes
> yes
> yes

Great, thanks. I think this sounds reasonable.

Ian.

Re: [RFC] ARM VM System Sepcification

[Ian Campbell]

On Mon, 2014-03-24 at 05:13 -0700, Grant Likely wrote:
> yes
> yes
> yes

Great, thanks. I think this sounds reasonable.

Ian.

[RFC] ARM VM System Sepcification

[Christoffer Dall]

ARM VM System Specification
===========================

Goal
----
The goal of this spec is to allow suitably-built OS images to run on
all ARM virtualization solutions, such as KVM or Xen.

Recommendations in this spec are valid for aarch32 and aarch64 alike, and
they aim to be hypervisor agnostic.

Note that simply adhering to the SBSA [2] is not a valid approach,
for example because the SBSA mandates EL2, which will not be available
for VMs.  Further, the SBSA mandates peripherals like the pl011, which
may be controversial for some ARM VM implementations to support.
This spec also covers the aarch32 execution mode, not covered in the
SBSA.


Image format
------------
The image format, as presented to the VM, needs to be well-defined in
order for prepared disk images to be bootable across various
virtualization implementations.

The raw disk format as presented to the VM must be partitioned with a
GUID Partition Table (GPT).  The bootable software must be placed in the
EFI System Partition (ESP), using the UEFI removable media path, and
must be an EFI application complying to the UEFI Specification 2.4
Revision A [6].

The ESP partition's GPT entry's partition type GUID must be
C12A7328-F81F-11D2-BA4B-00A0C93EC93B and the file system must be
formatted as FAT32/vfat as per Section 12.3.1.1 in [6].

The removable media path is \EFI\BOOT\BOOTARM.EFI for the aarch32
execution state and is \EFI\BOOT\BOOTAA64.EFI for the aarch64 execution
state.

This ensures that tools for both Xen and KVM can load a binary UEFI
firmware which can read and boot the EFI application in the disk image.

A typical scenario will be GRUB2 packaged as an EFI application, which
mounts the system boot partition and boots Linux.


Virtual Firmware
----------------
The VM system must be able to boot the EFI application in the ESP.  It
is recommended that this is achieved by loading a UEFI binary as the
first software executed by the VM, which then executes the EFI
application.  The UEFI implementation should be compliant with UEFI
Specification 2.4 Revision A [6] or later.

This document strongly recommends that the VM implementation supports
persistent environment storage for virtual firmware implementation in
order to ensure probable use cases such as adding additional disk images
to a VM or running installers to perform upgrades.

The binary UEFI firmware implementation should not be distributed as
part of the VM image, but is specific to the VM implementation.


Hardware Description
--------------------
The Linux kernel's proper entry point always takes a pointer to an FDT,
regardless of the boot mechanism, firmware, and hardware description
method.  Even on real hardware which only supports ACPI and UEFI, the kernel
entry point will still receive a pointer to a simple FDT, generated by
the Linux kernel UEFI stub, containing a pointer to the UEFI system
table.  The kernel can then discover ACPI from the system tables.  The
presence of ACPI vs. FDT is therefore always itself discoverable,
through the FDT.

Therefore, the VM implementation must provide through its UEFI
implementation, either:

  a complete FDT which describes the entire VM system and will boot
  mainline kernels driven by device tree alone, or

  no FDT.  In this case, the VM implementation must provide ACPI, and
  the OS must be able to locate the ACPI root pointer through the UEFI
  system table.

For more information about the arm and arm64 boot conventions, see
Documentation/arm/Booting and Documentation/arm64/booting.txt in the
Linux kernel source tree.

For more information about UEFI and ACPI booting, see [4] and [5].


VM Platform
-----------
The specification does not mandate any specific memory map.  The guest
OS must be able to enumerate all processing elements, devices, and
memory through HW description data (FDT, ACPI) or a bus-specific
mechanism such as PCI.

The virtual platform must support at least one of the following ARM
execution states:
  (1) aarch32 virtual CPUs on aarch32 physical CPUs
  (2) aarch32 virtual CPUs on aarch64 physical CPUs
  (3) aarch64 virtual CPUs on aarch64 physical CPUs

It is recommended to support both (2) and (3) on aarch64 capable
physical systems.

The virtual hardware platform must provide a number of mandatory
peripherals:

  Serial console:  The platform should provide a console,
  based on an emulated pl011, a virtio-console, or a Xen PV console.

  An ARM Generic Interrupt Controller v2 (GICv2) [3] or newer.  GICv2
  limits the the number of virtual CPUs to 8 cores, newer GIC versions
  removes this limitation.

  The ARM virtual timer and counter should be available to the VM as
  per the ARM Generic Timers specification in the ARM ARM [1].

  A hotpluggable bus to support hotplug of at least block and network
  devices.  Suitable buses include a virtual PCIe bus and the Xen PV
  bus.


We make the following recommendations for the guest OS kernel:

  The guest OS must include support for GICv2 and any available newer
  version of the GIC architecture to maintain compatibility with older
  VM implementations.

  It is strongly recommended to include support for all available
  (block, network, console, balloon) virtio-pci, virtio-mmio, and Xen PV
  drivers in the guest OS kernel or initial ramdisk.


Other common peripherals for block devices, networking, and more can
(and typically will) be provided, but OS software written and compiled
to run on ARM VMs cannot make any assumptions about which variations
of these should exist or which implementation they use (e.g. VirtIO or
Xen PV).  See "Hardware Description" above.

Note that this platform specification is separate from the Linux kernel
concept of mach-virt, which merely specifies a machine model driven
purely from device tree, but does not mandate any peripherals or have any
mention of ACPI.


References
----------
[1]: The ARM Architecture Reference Manual, ARMv8, Issue A.b
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html

[2]: ARM Server Base System Architecture
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0029/index.html

[3]: The ARM Generic Interrupt Controller Architecture Specifications v2.0
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a.b/index.html

[4]: http://www.secretlab.ca/archives/27

[5]: https://git.linaro.org/people/leif.lindholm/linux.git/blob/refs/heads/uefi-for-upstream:/Documentation/arm/uefi.txt

[6]: UEFI Specification 2.4 Revision A
http://www.uefi.org/sites/default/files/resources/2_4_Errata_A.pdf