Re: [edk2] Passing Xen memory map and resource map to OVMF

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Laszlo Ersek]

On 11/12/13 19:33, Wei Liu wrote:
> Hi all
> 
> Currently OVMF determines memory size by consulting CMOS, then it makes
> up memory map of its own.
> 
> Consulting memory size by reading CMOS limits the RAM size to 1TB as
> there's only 3 bytes from 0x5b-0x5d in CMOS, where the upper memory size
> is stored.
> 
> And from Xen's point of view, OVMF should use the memory mapped passed
> by hypervisor (from hvmloader) instead of making up its own.
> 
> To solve the above two problems all in one go, I plan to pass necessary
> information (io resource, mmio resource) to OVMF from Xen.  I will
> construct the table / structure in hvmloader then hook up platform pei
> code when OVMF is running on Xen.
> 
> The first thing that comes in mind is to reuse E820 table for memory map
> plus some extra fields for io / mmio resources. But I guess UEFI is the
> new world so stuffs like E820 from old world will be less popular. Any
> suggestion on existing table / data structure I can use?

As far as I understand / remember, we're not really dynamic wrt. memory
layout in PEI, even on pure qemu or with KVM. I think it would be useful
to key off some memory map provided by the emulator (although I'm sure
we'd run into regressions in some spots, but those can be fixed later).

As far as I can see, qemu exports an fw_cfg table called "etc/e820". See
qemu commit

commit 7d67110f2d9a6a2d6b5215a948abc95d07258735
Author: Gerd Hoffmann <kraxel@redhat.com>
Date:   Fri Oct 18 11:31:54 2013 +0200

    pc: add etc/e820 fw_cfg file

    Unlike the existing FW_CFG_E820_TABLE entry which carries reservations
    only the new etc/e820 file also has entries for RAM.

    Format is simliar to the FW_CFG_E820_TABLE, it is a simple list of
    e820_entry structs.  Unlike FW_CFG_E820_TABLE it has no count though
    as the number of entries can be figured from the file size.

    Cc: Andrea Arcangeli <aarcange@redhat.com>
    Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>

Does this work when qemu is used in combination with Xen?

Maybe we could use a common interface here that would work independently
of the qemu accelerator.

"OvmfPkg/Library/QemuFwCfgLib" is a BASE library, so I *think* we should
be free to use it in "OvmfPkg/PlatformPei" (which is a PEIM).


OVMF already contains a structure definition for the E820 entry (called
"struct e820_entry"), since commit

Author: Jordan Justen <jordan.l.justen@intel.com>
Date:   Fri Nov 2 18:26:30 2012 +0000

    OvmfPkg: Add Linux bzimage include file

    This file is from the efilinux project where it resides
    under the path loaders/bzimage/bzimage.h.
    git://git.kernel.org/pub/scm/boot/efilinux/efilinux.git

    Contributed-under: TianoCore Contribution Agreement 1.0
    Signed-off-by: Jordan Justen <jordan.l.justen@intel.com>
    Reviewed-by: Laszlo Ersek <lersek@redhat.com>
    Acked-by: Matt Fleming <matt.fleming@intel.com>

    git-svn-id: https://svn.code.sf.net/p/edk2/code/trunk/edk2@13920 6f19259b-4bc3-4df7-8a09-765794883524

I guess this defintion could be moved to a more "neutral" location (from
"Include/IndustryStandard/LinuxBzimage.h" to
"Include/IndustryStandard/E820.h" or some such), and then you could
reuse it in the new code.

I also recall that an e820-related fw_cfg file has been introduced
recently to qemu, it has to do with the 64-bit PCI hole?... Igor? :)

Thanks,
Laszlo

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Laszlo Ersek]

On 11/12/13 20:05, Laszlo Ersek wrote:
> On 11/12/13 19:33, Wei Liu wrote:
>> Hi all
>>
>> Currently OVMF determines memory size by consulting CMOS, then it makes
>> up memory map of its own.
>>
>> Consulting memory size by reading CMOS limits the RAM size to 1TB as
>> there's only 3 bytes from 0x5b-0x5d in CMOS, where the upper memory size
>> is stored.
>>
>> And from Xen's point of view, OVMF should use the memory mapped passed
>> by hypervisor (from hvmloader) instead of making up its own.
>>
>> To solve the above two problems all in one go, I plan to pass necessary
>> information (io resource, mmio resource) to OVMF from Xen.  I will
>> construct the table / structure in hvmloader then hook up platform pei
>> code when OVMF is running on Xen.
>>
>> The first thing that comes in mind is to reuse E820 table for memory map
>> plus some extra fields for io / mmio resources. But I guess UEFI is the
>> new world so stuffs like E820 from old world will be less popular. Any
>> suggestion on existing table / data structure I can use?
> 
> As far as I understand / remember, we're not really dynamic wrt. memory
> layout in PEI, even on pure qemu or with KVM. I think it would be useful
> to key off some memory map provided by the emulator (although I'm sure
> we'd run into regressions in some spots, but those can be fixed later).
> 
> As far as I can see, qemu exports an fw_cfg table called "etc/e820". See
> qemu commit
> 
> commit 7d67110f2d9a6a2d6b5215a948abc95d07258735
> Author: Gerd Hoffmann <kraxel@redhat.com>
> Date:   Fri Oct 18 11:31:54 2013 +0200
> 
>     pc: add etc/e820 fw_cfg file
> 
>     Unlike the existing FW_CFG_E820_TABLE entry which carries reservations
>     only the new etc/e820 file also has entries for RAM.
> 
>     Format is simliar to the FW_CFG_E820_TABLE, it is a simple list of
>     e820_entry structs.  Unlike FW_CFG_E820_TABLE it has no count though
>     as the number of entries can be figured from the file size.
> 
>     Cc: Andrea Arcangeli <aarcange@redhat.com>
>     Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> 
> Does this work when qemu is used in combination with Xen?

Even if it doesn't (because Xen uses a different way to pass the
information): perhaps a common function that turns an e820 table into a
bunch of HOBs would be useful. The pointer to the first entry could be
derived differently, dependent on the specific accelerator, but the
translation could be common.

(Of course I'm only assuming that qemu's etc/e820 fw_cfg file would be
appropriate for these HOBs. I could be wrong.)

Thanks
Laszlo

Re: Passing Xen memory map and resource map to OVMF

[Konrad Rzeszutek Wilk]

On Tue, Nov 12, 2013 at 06:33:21PM +0000, Wei Liu wrote:
> Hi all
> 
> Currently OVMF determines memory size by consulting CMOS, then it makes
> up memory map of its own.
> 
> Consulting memory size by reading CMOS limits the RAM size to 1TB as
> there's only 3 bytes from 0x5b-0x5d in CMOS, where the upper memory size
> is stored.
> 
> And from Xen's point of view, OVMF should use the memory mapped passed
> by hypervisor (from hvmloader) instead of making up its own.
> 
> To solve the above two problems all in one go, I plan to pass necessary
> information (io resource, mmio resource) to OVMF from Xen.  I will
> construct the table / structure in hvmloader then hook up platform pei
> code when OVMF is running on Xen.

/me nods. The nice thing about that is that it could also allow us
to modify the E820 from the toolstack for any combination we want.

For example if you use e820_host=1 in your PV guest the E820 will
look like the host one. Great for PCI passthrough when you have some
stubborn cards.
> 
> The first thing that comes in mind is to reuse E820 table for memory map
> plus some extra fields for io / mmio resources. But I guess UEFI is the
> new world so stuffs like E820 from old world will be less popular. Any
> suggestion on existing table / data structure I can use?

I think the E820 is fine. After all the Linux kernel picks up the EFI
memmap and then stuffs it in E820.

And there are some patches that do this for HVM already:

http://article.gmane.org/gmane.comp.emulators.xen.devel/170593

for which I was hoping could be reworked as needed.
> 
> 
> Thanks
> Wei.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Passing Xen memory map and resource map to OVMF

[Jordan Justen]

On Tue, Nov 12, 2013 at 10:33 AM, Wei Liu <wei.liu2@citrix.com> wrote:
> Hi all
>
> Currently OVMF determines memory size by consulting CMOS, then it makes
> up memory map of its own.
>
> Consulting memory size by reading CMOS limits the RAM size to 1TB as
> there's only 3 bytes from 0x5b-0x5d in CMOS, where the upper memory size
> is stored.
>
> And from Xen's point of view, OVMF should use the memory mapped passed
> by hypervisor (from hvmloader) instead of making up its own.
>
> To solve the above two problems all in one go, I plan to pass necessary
> information (io resource, mmio resource) to OVMF from Xen.  I will
> construct the table / structure in hvmloader then hook up platform pei
> code when OVMF is running on Xen.
>
> The first thing that comes in mind is to reuse E820 table for memory map
> plus some extra fields for io / mmio resources. But I guess UEFI is the
> new world so stuffs like E820 from old world will be less popular. Any
> suggestion on existing table / data structure I can use?

Is OvmfPkg/Include/Guid/XenInfo.h helpful on the OVMF side?

Regarding E820 structures, I guess I'd suggest copying the E820
structures from
IntelFrameworkModulePkg/Csm/LegacyBiosDxe/LegacyBiosInterface.h into a
new OvmfPkg/Include/IndustryStandard/E820.h file.

-Jordan

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Gerd Hoffmann]

  Hi,

> > The first thing that comes in mind is to reuse E820 table for memory map
> > plus some extra fields for io / mmio resources. But I guess UEFI is the
> > new world so stuffs like E820 from old world will be less popular. Any
> > suggestion on existing table / data structure I can use?

What io/mmio ressources do you need to pass on?

> As far as I can see, qemu exports an fw_cfg table called "etc/e820". See
> qemu commit
> 
> commit 7d67110f2d9a6a2d6b5215a948abc95d07258735
> Author: Gerd Hoffmann <kraxel@redhat.com>
> Date:   Fri Oct 18 11:31:54 2013 +0200
> 
>     pc: add etc/e820 fw_cfg file

New in qemu 1.7  Was added exactly to deal with the 1TB limitation and
also to allow for non-contiguous memory (not implemented yet in qemu,
but if numa support is extended some day to support that we are fine
interface-wise).

> Does this work when qemu is used in combination with Xen?

It should, might need windup in the xen machine type though (not fully
sure how much code is shared between normal and xen machine types).

cheers,
  Gerd

Re: Passing Xen memory map and resource map to OVMF

[Ian Campbell]

On Tue, 2013-11-12 at 14:22 -0500, Konrad Rzeszutek Wilk wrote:
> On Tue, Nov 12, 2013 at 06:33:21PM +0000, Wei Liu wrote:
> > Hi all
> > 
> > Currently OVMF determines memory size by consulting CMOS, then it makes
> > up memory map of its own.
> > 
> > Consulting memory size by reading CMOS limits the RAM size to 1TB as
> > there's only 3 bytes from 0x5b-0x5d in CMOS, where the upper memory size
> > is stored.
> > 
> > And from Xen's point of view, OVMF should use the memory mapped passed
> > by hypervisor (from hvmloader) instead of making up its own.
> > 
> > To solve the above two problems all in one go, I plan to pass necessary
> > information (io resource, mmio resource) to OVMF from Xen.  I will
> > construct the table / structure in hvmloader then hook up platform pei
> > code when OVMF is running on Xen.
> 
> /me nods. The nice thing about that is that it could also allow us
> to modify the E820 from the toolstack for any combination we want.

Note that the right way to do that would be for the toolstack to
communicate the base e820 to hvmloader, which would then do what it
needs to do modifying as it went and plumb the result through to the
BIOS.

Ian.

Re: Passing Xen memory map and resource map to OVMF

[Ian Campbell]

On Tue, 2013-11-12 at 18:33 +0000, Wei Liu wrote:
> Hi all
> 
> Currently OVMF determines memory size by consulting CMOS, then it makes
> up memory map of its own.
> 
> Consulting memory size by reading CMOS limits the RAM size to 1TB as
> there's only 3 bytes from 0x5b-0x5d in CMOS, where the upper memory size
> is stored.
> 
> And from Xen's point of view, OVMF should use the memory mapped passed
> by hypervisor (from hvmloader) instead of making up its own.
> 
> To solve the above two problems all in one go, I plan to pass necessary
> information (io resource, mmio resource) to OVMF from Xen.  I will
> construct the table / structure in hvmloader then hook up platform pei
> code when OVMF is running on Xen.

FWIW when hvmloader launches SeaBIOS it passes this stuff in a struct at
a known location:

struct xen_seabios_info {
    char signature[14]; /* XenHVMSeaBIOS\0 */
    u8 length;     /* Length of this struct */
    u8 checksum;   /* Set such that the sum over bytes 0..length == 0 */
    /*
     * Physical address of an array of tables_nr elements.
     *
     * Each element is a 32 bit value contianing the physical address
     * of a BIOS table.
     */
    u32 tables;
    u32 tables_nr;
    /*
     * Physical address of the e820 table, contains e820_nr entries.
     */
    u32 e820;
    u32 e820_nr;
} PACKED;

The e820 is not self identifiable so it gets its own field. The "tables"
are the ACPI RSDP, the mptables, the smbios tables and the PIR tables.
SeaBIOS iterates over these and identifies them by their unique
signature, ignoring any which it doesn't understand for forwards
compatibility.

I don't know which if any of those tables are suitable to be passed into
OVMF, I guess some of them are legacy BIOS things...

But the general shape of the scheme is probably worth following...

Ian.

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Wei Liu]

On Wed, Nov 13, 2013 at 08:03:39AM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > > The first thing that comes in mind is to reuse E820 table for memory map
> > > plus some extra fields for io / mmio resources. But I guess UEFI is the
> > > new world so stuffs like E820 from old world will be less popular. Any
> > > suggestion on existing table / data structure I can use?
> 
> What io/mmio ressources do you need to pass on?
> 

MMIO holes, IO range created by hvmloader.

OVMF should make use of that information instead of creating MMIO holes
etc of its own. See OvmfPkg/PlatformPei/Platform.c:MemMapInitialization.

> > As far as I can see, qemu exports an fw_cfg table called "etc/e820". See
> > qemu commit
> > 
> > commit 7d67110f2d9a6a2d6b5215a948abc95d07258735
> > Author: Gerd Hoffmann <kraxel@redhat.com>
> > Date:   Fri Oct 18 11:31:54 2013 +0200
> > 
> >     pc: add etc/e820 fw_cfg file
> 
> New in qemu 1.7  Was added exactly to deal with the 1TB limitation and
> also to allow for non-contiguous memory (not implemented yet in qemu,
> but if numa support is extended some day to support that we are fine
> interface-wise).
> 

This is probably too new for us. We're still fighting a regression in
1.6.

Wei.

> > Does this work when qemu is used in combination with Xen?
> 
> It should, might need windup in the xen machine type though (not fully
> sure how much code is shared between normal and xen machine types).
> 
> cheers,
>   Gerd
> 

Re: Passing Xen memory map and resource map to OVMF

[Wei Liu]

On Tue, Nov 12, 2013 at 11:51:10AM -0800, Jordan Justen wrote:
> On Tue, Nov 12, 2013 at 10:33 AM, Wei Liu <wei.liu2@citrix.com> wrote:
> > Hi all
> >
> > Currently OVMF determines memory size by consulting CMOS, then it makes
> > up memory map of its own.
> >
> > Consulting memory size by reading CMOS limits the RAM size to 1TB as
> > there's only 3 bytes from 0x5b-0x5d in CMOS, where the upper memory size
> > is stored.
> >
> > And from Xen's point of view, OVMF should use the memory mapped passed
> > by hypervisor (from hvmloader) instead of making up its own.
> >
> > To solve the above two problems all in one go, I plan to pass necessary
> > information (io resource, mmio resource) to OVMF from Xen.  I will
> > construct the table / structure in hvmloader then hook up platform pei
> > code when OVMF is running on Xen.
> >
> > The first thing that comes in mind is to reuse E820 table for memory map
> > plus some extra fields for io / mmio resources. But I guess UEFI is the
> > new world so stuffs like E820 from old world will be less popular. Any
> > suggestion on existing table / data structure I can use?
> 
> Is OvmfPkg/Include/Guid/XenInfo.h helpful on the OVMF side?
> 

It needs to be extended to hold more information.

> Regarding E820 structures, I guess I'd suggest copying the E820
> structures from
> IntelFrameworkModulePkg/Csm/LegacyBiosDxe/LegacyBiosInterface.h into a
> new OvmfPkg/Include/IndustryStandard/E820.h file.
> 

Thanks for the pointer.

Wei.

> -Jordan

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Gerd Hoffmann]

On Mi, 2013-11-13 at 11:58 +0000, Wei Liu wrote:
> On Wed, Nov 13, 2013 at 08:03:39AM +0100, Gerd Hoffmann wrote:
> >   Hi,
> > 
> > > > The first thing that comes in mind is to reuse E820 table for memory map
> > > > plus some extra fields for io / mmio resources. But I guess UEFI is the
> > > > new world so stuffs like E820 from old world will be less popular. Any
> > > > suggestion on existing table / data structure I can use?
> > 
> > What io/mmio ressources do you need to pass on?
> > 
> 
> MMIO holes, IO range created by hvmloader.

Why?  All (memory) address space not backed by ram effectively is mmio.
There is no need to explicitly declare holes ...

Also why io ranges?  Isn't pc io address space pretty much fixed anyway?

cheers,
  Gerd

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Wei Liu]

On Wed, Nov 13, 2013 at 02:53:34PM +0100, Gerd Hoffmann wrote:
> On Mi, 2013-11-13 at 11:58 +0000, Wei Liu wrote:
> > On Wed, Nov 13, 2013 at 08:03:39AM +0100, Gerd Hoffmann wrote:
> > >   Hi,
> > > 
> > > > > The first thing that comes in mind is to reuse E820 table for memory map
> > > > > plus some extra fields for io / mmio resources. But I guess UEFI is the
> > > > > new world so stuffs like E820 from old world will be less popular. Any
> > > > > suggestion on existing table / data structure I can use?
> > > 
> > > What io/mmio ressources do you need to pass on?
> > > 
> > 
> > MMIO holes, IO range created by hvmloader.
> 
> Why?  All (memory) address space not backed by ram effectively is mmio.
> There is no need to explicitly declare holes ...
> 

Yes there is, because hvmloader is in charge of declaring holes. OVMF
should just use whatever it gets.

> Also why io ranges?  Isn't pc io address space pretty much fixed anyway?
> 

This is not actually necessary, but I want to pass this anyway, just to
be consistent.

Wei.

> cheers,
>   Gerd
> 
> 

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Gerd Hoffmann]

  Hi,

> > > MMIO holes, IO range created by hvmloader.
> > 
> > Why?  All (memory) address space not backed by ram effectively is mmio.
> > There is no need to explicitly declare holes ...
> > 
> 
> Yes there is, because hvmloader is in charge of declaring holes.

Why does it declare holes in the first place?
Just for ACPI _CRS?  Or other reasons too?

> OVMF
> should just use whatever it gets.

What would OVMF use them for?

As far I know hvmloader creates the acpi tables and initializes pci
device mappings, so there isn't much to manage for ovmf ...

> > Also why io ranges?  Isn't pc io address space pretty much fixed anyway?
> > 
> 
> This is not actually necessary, but I want to pass this anyway, just to
> be consistent.

I don't see the point in passing it ...

cheers,
  Gerd

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Konrad Rzeszutek Wilk]

On Wed, Nov 13, 2013 at 08:03:39AM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > > The first thing that comes in mind is to reuse E820 table for memory map
> > > plus some extra fields for io / mmio resources. But I guess UEFI is the
> > > new world so stuffs like E820 from old world will be less popular. Any
> > > suggestion on existing table / data structure I can use?
> 
> What io/mmio ressources do you need to pass on?

Say the ones for the emulated PCI cards or when PCI passthrough has been
setup?

> 
> > As far as I can see, qemu exports an fw_cfg table called "etc/e820". See
> > qemu commit
> > 
> > commit 7d67110f2d9a6a2d6b5215a948abc95d07258735
> > Author: Gerd Hoffmann <kraxel@redhat.com>
> > Date:   Fri Oct 18 11:31:54 2013 +0200
> > 
> >     pc: add etc/e820 fw_cfg file
> 
> New in qemu 1.7  Was added exactly to deal with the 1TB limitation and
> also to allow for non-contiguous memory (not implemented yet in qemu,
> but if numa support is extended some day to support that we are fine
> interface-wise).
> 
> > Does this work when qemu is used in combination with Xen?
> 
> It should, might need windup in the xen machine type though (not fully
> sure how much code is shared between normal and xen machine types).
> 
> cheers,
>   Gerd
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Konrad Rzeszutek Wilk]

On Wed, Nov 13, 2013 at 02:53:34PM +0100, Gerd Hoffmann wrote:
> On Mi, 2013-11-13 at 11:58 +0000, Wei Liu wrote:
> > On Wed, Nov 13, 2013 at 08:03:39AM +0100, Gerd Hoffmann wrote:
> > >   Hi,
> > > 
> > > > > The first thing that comes in mind is to reuse E820 table for memory map
> > > > > plus some extra fields for io / mmio resources. But I guess UEFI is the
> > > > > new world so stuffs like E820 from old world will be less popular. Any
> > > > > suggestion on existing table / data structure I can use?
> > > 
> > > What io/mmio ressources do you need to pass on?
> > > 
> > 
> > MMIO holes, IO range created by hvmloader.
> 
> Why?  All (memory) address space not backed by ram effectively is mmio.
> There is no need to explicitly declare holes ...

That is the case.

The question is where should RAM start and stop. And if you want to
extend the MMIO (for odd cards that you are passing in.

> 
> Also why io ranges?  Isn't pc io address space pretty much fixed anyway?

I think most BIOSes mark that area as E820_RSV?

 000000000009e800 - 00000000000a0000 (reserved)

Oh wait, that is not PC IO address. You are right those are
most certainly marked as normal RAM.

> 
> cheers,
>   Gerd
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Wei Liu]

On Wed, Nov 13, 2013 at 03:23:05PM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > > > MMIO holes, IO range created by hvmloader.
> > > 
> > > Why?  All (memory) address space not backed by ram effectively is mmio.
> > > There is no need to explicitly declare holes ...
> > > 
> > 
> > Yes there is, because hvmloader is in charge of declaring holes.
> 
> Why does it declare holes in the first place?
> Just for ACPI _CRS?  Or other reasons too?
> 

The reason is hvmloader knows the constraints of where MMIO holes can be
put, so it arranges the holes itself to avoid tripping over Xen
limitation. If OVMF creates its own map it's going to cause problem.

One example is, when the guest has <2GB RAM, OVMF will allocate PCI
resources starting from 2GB. But hvmloader has already set host bridge
window to start at 4GB. When Linux boots up it finds existing BAR
incompatible with bridge window so it decides to relocate BAR to start
from 4GB. One of the many BARs is framebuffer, Xen cannot handle the
situation that framebuffer is moved from one location to another.

> > OVMF
> > should just use whatever it gets.
> 
> What would OVMF use them for?
> 

To reserve range for MMIO holes, so that later PCI resource allocation
protocol can only use those ranges.

Wei.

> As far I know hvmloader creates the acpi tables and initializes pci
> device mappings, so there isn't much to manage for ovmf ...
> 
> > > Also why io ranges?  Isn't pc io address space pretty much fixed anyway?
> > > 
> > 
> > This is not actually necessary, but I want to pass this anyway, just to
> > be consistent.
> 
> I don't see the point in passing it ...
> 
> cheers,
>   Gerd
> 

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Gerd Hoffmann]

  Hi,

> > > OVMF
> > > should just use whatever it gets.
> > 
> > What would OVMF use them for?
> > 
> 
> To reserve range for MMIO holes, so that later PCI resource allocation
> protocol can only use those ranges.

I'm still not convinced you need that in the first place.

When booting seabios @ xen it is not needed.  pci ressource allocation
is handled by hvmloader.  apci tables (and the ressources declared
therein) are handled by hvmloader.  seabios doesn't touch the pci bars
and passes through the apci tables -> guest os is happy.

So why ovmf should be different?   IMHO it should operate like seabios
and NOT do pci ressource allocation when running on xen.  Ressources are
already handled already by hvmloader.  Doing it again is (a) pointless
and (b) creates problems like the one we are discussion right now.

cheers,
  Gerd

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Wei Liu]

On Thu, Nov 14, 2013 at 08:58:18AM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > > > OVMF
> > > > should just use whatever it gets.
> > > 
> > > What would OVMF use them for?
> > > 
> > 
> > To reserve range for MMIO holes, so that later PCI resource allocation
> > protocol can only use those ranges.
> 
> I'm still not convinced you need that in the first place.
> 
> When booting seabios @ xen it is not needed.  pci ressource allocation
> is handled by hvmloader.  apci tables (and the ressources declared
> therein) are handled by hvmloader.  seabios doesn't touch the pci bars
> and passes through the apci tables -> guest os is happy.
> 
> So why ovmf should be different?   IMHO it should operate like seabios
> and NOT do pci ressource allocation when running on xen.  Ressources are
> already handled already by hvmloader.  Doing it again is (a) pointless
> and (b) creates problems like the one we are discussion right now.
> 

Agreed. But the protocol to allocate PCI resources seems to be mandatory
according to UEFI spec [0].

--- page 16
The following protocols are mandatory if the system supports PCI devices
or slots: 
• EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL 
• EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL 
--

I'm more than happy to disable that allocation protocol if there's a way.

Wei.

[0] http://www.intel.com/content/dam/doc/reference-guide/efi-pci-host-bridge-allocation-protocol-specification.pdf

> cheers,
>   Gerd
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Laszlo Ersek]

On 11/14/13 08:58, Gerd Hoffmann wrote:
>   Hi,
>
>>>> OVMF
>>>> should just use whatever it gets.
>>>
>>> What would OVMF use them for?
>>>
>>
>> To reserve range for MMIO holes, so that later PCI resource
>> allocation protocol can only use those ranges.
>
> I'm still not convinced you need that in the first place.
>
> When booting seabios @ xen it is not needed.  pci ressource allocation
> is handled by hvmloader.  apci tables (and the ressources declared
> therein) are handled by hvmloader.  seabios doesn't touch the pci bars
> and passes through the apci tables -> guest os is happy.
>
> So why ovmf should be different?   IMHO it should operate like seabios
> and NOT do pci ressource allocation when running on xen.  Ressources
> are already handled already by hvmloader.  Doing it again is (a)
> pointless and (b) creates problems like the one we are discussion
> right now.

The PEI (Pre-EFI Initialization) phase in UEFI has to produce a series
of hand-off blocks (HOBs) that describe a bunch of things for the later
phases of UEFI. One such set of information is the memory map used eg.
by the memory management functions called from drivers and applications.
ACPI tables are not enough.

So, these HOBs in PEI must come from somewhere. (Platform initialization
is officially platform-specific, so if on your platform ACPI tables come
to existence first, at some location in memory, you can key off those
when exporting the HOBs.)

The memmap-related HOBs we currently export in
"OvmfPkg/PlatformPei/Platform.c" are basically static, they only depend
on the memory size retrieved from the CMOS. I can imagine that more
flexibility is needed.

Also, PCI resource allocation (enumeration etc.) is just part of the PCI
bus driver in UEFI. Sometime earlier I wrote

> The MMIO range is allocated like this:
>
> PciEnumerator()                                                   [MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumerator.c]
>   PciHostBridgeResourceAllocator()                                [MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c]
>     NotifyPhase (PciResAlloc, EfiPciHostBridgeAllocateResources)  [MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumerator.c]
>       NotifyPhase()                                               [PcAtChipsetPkg/PciHostBridgeDxe/PciHostBridge.c]
>         CoreAllocateMemorySpace()                                 [MdeModulePkg/Core/Dxe/Gcd/Gcd.c]
>           CoreAllocateSpace()
>
> The memory map base comes from CoreInitializeGcdServices() /
> MemMapInitialization().

The MemMapInitialization() function (in
"OvmfPkg/PlatformPei/Platform.c") mentioned in the quote is what we're
discussing now, that's what produces those early HOBs.

(Corrections welcome to whatever I said in this email.)

Thanks
Laszlo

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Gerd Hoffmann]

  Hi,

> > So why ovmf should be different?   IMHO it should operate like seabios
> > and NOT do pci ressource allocation when running on xen.  Ressources are
> > already handled already by hvmloader.  Doing it again is (a) pointless
> > and (b) creates problems like the one we are discussion right now.
> > 
> 
> Agreed. But the protocol to allocate PCI resources seems to be mandatory
> according to UEFI spec [0].
> 
> --- page 16
> The following protocols are mandatory if the system supports PCI devices
> or slots: 
> • EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL 
> • EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL 
> --
> 
> I'm more than happy to disable that allocation protocol if there's a way.

Disabling isn't going to fly I suspect.  But maybe it is possible to add
a read-only mode to these protocols, so they just enumerate the pci
bus(es), read the pci bar registers and create whatever data structures
ovmf needs to drive the pci devices.  But don't try to manage the pci
ressources.

That would also be useful when running tianocore as coreboot payload,
which is a simliar situation: coreboot already did the pci enumeration
and tianocore doesn't need to do it again (Cc'ing coreboot list).

cheers,
  Gerd



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Andrew Fish]

On Nov 14, 2013, at 3:06 AM, Laszlo Ersek <lersek@redhat.com> wrote:

> On 11/14/13 08:58, Gerd Hoffmann wrote:
>>  Hi,
>> 
>>>>> OVMF
>>>>> should just use whatever it gets.
>>>> 
>>>> What would OVMF use them for?
>>>> 
>>> 
>>> To reserve range for MMIO holes, so that later PCI resource
>>> allocation protocol can only use those ranges.
>> 
>> I'm still not convinced you need that in the first place.
>> 
>> When booting seabios @ xen it is not needed.  pci ressource allocation
>> is handled by hvmloader.  apci tables (and the ressources declared
>> therein) are handled by hvmloader.  seabios doesn't touch the pci bars
>> and passes through the apci tables -> guest os is happy.
>> 
>> So why ovmf should be different?   IMHO it should operate like seabios
>> and NOT do pci ressource allocation when running on xen.  Ressources
>> are already handled already by hvmloader.  Doing it again is (a)
>> pointless and (b) creates problems like the one we are discussion
>> right now.
> 
> The PEI (Pre-EFI Initialization) phase in UEFI has to produce a series
> of hand-off blocks (HOBs) that describe a bunch of things for the later
> phases of UEFI. One such set of information is the memory map used eg.
> by the memory management functions called from drivers and applications.
> ACPI tables are not enough.
> 
> So, these HOBs in PEI must come from somewhere. (Platform initialization
> is officially platform-specific, so if on your platform ACPI tables come
> to existence first, at some location in memory, you can key off those
> when exporting the HOBs.)
> 

Assuming the info you need is in static ACPI tables. I thought the abstraction of PCI in ACPI was via a root bridge method and you need a full blown interpreter to read it. There is an Open Source ACPI interpreter, from Intel, but it seems like overkill. From a standards point of view we have gotten requests in the past from OS vendors to expose some information that was in the ACPI name space as it was need before the OS ACPI interpreter would be available….

Also assuming everything is behind PCI that has been enumerated seems like a very PC centric view of the world. A lot of ARM SoC devices don’t have PCI, but just memory mapped registers. Hiding everything behind PCI is a PC trick to make Windows work. So for generic hardware some kind of table that describes the memory map seems useful. 

> The memmap-related HOBs we currently export in
> "OvmfPkg/PlatformPei/Platform.c" are basically static, they only depend
> on the memory size retrieved from the CMOS. I can imagine that more
> flexibility is needed.
> 
> Also, PCI resource allocation (enumeration etc.) is just part of the PCI
> bus driver in UEFI. Sometime earlier I wrote
> 
>> The MMIO range is allocated like this:
>> 
>> PciEnumerator()                                                   [MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumerator.c]
>>  PciHostBridgeResourceAllocator()                                [MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c]
>>    NotifyPhase (PciResAlloc, EfiPciHostBridgeAllocateResources)  [MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumerator.c]
>>      NotifyPhase()                                               [PcAtChipsetPkg/PciHostBridgeDxe/PciHostBridge.c]
>>        CoreAllocateMemorySpace()                                 [MdeModulePkg/Core/Dxe/Gcd/Gcd.c]
>>          CoreAllocateSpace()
>> 

The PI Spec. GCD (Global Coherency Domain) services manage address space. So some chunk of address space needs to be allocated to system memory, to PCI config space, to PCI MMIO space ect. 

>> The memory map base comes from CoreInitializeGcdServices() /
>> MemMapInitialization().
> 
> The MemMapInitialization() function (in
> "OvmfPkg/PlatformPei/Platform.c") mentioned in the quote is what we're
> discussing now, that's what produces those early HOBs.
> 
> (Corrections welcome to whatever I said in this email.)
> 

>From a UEFI spec perspective you only need to produce EFI_PCI_ROOT_BRIGE_IO_PROTOCOL and a EFI_PCI_IO_PROTOCOL per device. This is all that is required to make generic UEFI code (all the PCI drivers, shell commands, etc.) work. 

The https://svn.code.sf.net/p/edk2/code/trunk/edk2/MdeModulePkg/Bus/Pci/PciBusDxe/ driver assumes you are following the PI Spec model and a full PCI enumeration is requires and a set of chipset/platform specific protocols are provided to make the PCI enumerate code generic. 

There are other examples in the edk2 where PCI enumeration is not required:
1) DUET, as the PCI enumeration has already been done. You can see in https://svn.code.sf.net/p/edk2/code/trunk/edk2/DuetPkg/DuetPkgX64.dsc that the UEFI required protocols for PCI are implemented in DuetPkg/PciRootBridgeNoEnumerationDxe/PciRootBridgeNoEnumeration.inf and DuetPkg/PciBusNoEnumerationDxe/PciBusNoEnumeration.inf
2) The BeagleBoard does not have PCI, but it produces a fake set of PCI interfaces to reuse some of the EFI PCI drivers. See https://svn.code.sf.net/p/edk2/code/trunk/edk2/Omap35xxPkg/PciEmulation/

Hope this helps,

Andrew Fish


> Thanks
> Laszlo
> 
> ------------------------------------------------------------------------------
> DreamFactory - Open Source REST & JSON Services for HTML5 & Native Apps
> OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
> Free app hosting. Or install the open source package on any LAMP server.
> Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
> http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/edk2-devel

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Gerd Hoffmann]

  Hi,

> There are other examples in the edk2 where PCI enumeration is not required:

> 1) DUET, as the PCI enumeration has already been done.

That is very simliar to the situation we have when running edk2 on xen
(and coreboot), so its worth investigating that route IMHO.

cheers,
  Gerd

Re: [edk2] Passing Xen memory map and resource map to OVMF

[Wei Liu]

On Thu, Nov 14, 2013 at 08:42:39AM -0800, Andrew Fish wrote:
[...]
> > 
> 
> >From a UEFI spec perspective you only need to produce EFI_PCI_ROOT_BRIGE_IO_PROTOCOL and a EFI_PCI_IO_PROTOCOL per device. This is all that is required to make generic UEFI code (all the PCI drivers, shell commands, etc.) work. 
> 
> The https://svn.code.sf.net/p/edk2/code/trunk/edk2/MdeModulePkg/Bus/Pci/PciBusDxe/ driver assumes you are following the PI Spec model and a full PCI enumeration is requires and a set of chipset/platform specific protocols are provided to make the PCI enumerate code generic. 
> 
> There are other examples in the edk2 where PCI enumeration is not required:
> 1) DUET, as the PCI enumeration has already been done. You can see in https://svn.code.sf.net/p/edk2/code/trunk/edk2/DuetPkg/DuetPkgX64.dsc that the UEFI required protocols for PCI are implemented in DuetPkg/PciRootBridgeNoEnumerationDxe/PciRootBridgeNoEnumeration.inf and DuetPkg/PciBusNoEnumerationDxe/PciBusNoEnumeration.inf
> 2) The BeagleBoard does not have PCI, but it produces a fake set of PCI interfaces to reuse some of the EFI PCI drivers. See https://svn.code.sf.net/p/edk2/code/trunk/edk2/Omap35xxPkg/PciEmulation/
> 
> Hope this helps,
> 

Thanks for the pointer, I will see what I can do. :-)

Wei.