[RFC] OVMF on PVH

[RFC] OVMF on PVH

[Anthony PERARD]

Hi,

I've been working on booting OVMF in a PVH guest. There are few changes
that I'd like your comments on. Those are changes that I've already made
in my private branch, there are either required or will make things
easier.

## Goal

Have a single blob that can be use for both HVM and PVH guests. And be
able to start UEFI compatible guests in PVH.


## New binary, separated from QEMU/KVM one.

Right now, the rules to build the firmware are located in
`OvmfPkg/OvmfPkgX64.dsc`, a platform. I've created a new platform, without
KVM support, and would like to retire the Xen support from the current
platform.  For now, I have created OvmfPkg/XenOvmf.dsc. (The change to a
new platform had been proposed by upstream.)

That would simplify Xen support in OVMF, and allow to diverge more from
OVMF where needed. That would mostly be useful during the initial boot
phase where there lots of `if xen do that, else do that`.

We can also choose some drivers that can work on both PVH and HVM, for
example, use a LAPIC timer instead of ACPI Timer.

All this mean that building OVMF for Xen would need to be change.


## ELF header

Adding an ELF header to OVMF so the blob can be loaded by libxl/libxc
without modifying those libs.

That replace a section of the OVMF binary called VarStore by that ELF
header.  It's empty and libxl doesn't know how to use it. (QEMU/KVM would
have a flash device loading the store, so it can be written to.)

With the ELF header, I can test OVMF by simply adding this in the config
file:

    type='pvh'
    kernel='/path/to/OVMF.fd'

We could use a modified `hvmloader` to load OVMF on PVH or teach libxc to
load it at a hardcoded location, but I don't see it as necessary, and if
we have one less firmware to load, it's probably better.

With the use of an ELF header comes another issue, which as to do with
were the binary needs to be loaded initially.

## Loading binary at 0x10000 (1MB, like hvmloader on HVM)

Right now, the OVMF blob is loaded (by hvmloader) just below 4GB, with
hvmloader moving some RAM around to allow that.  But with the use of an
ELF header, and let libxc load the blob, it was not possible to load the
blob just below 4GB. (Or at least without modification of the toolstack I
guess.) So I've attempted to have OVMF been started from a different place
in memory. Then I had to hack a bit more in order to be able to start OVMF
from two different location (the current one for HVM guest, and a new one
that can be used for PVH). That works fine, I'll just have to find out
what upstream think about that.

## Current WIP

You can find my current branch on xenbits, it still needs to be cleanup
but it works and can boot a PVH guest:
https://xenbits.xen.org/git-http/people/aperard/ovmf.git
branch: wip.platform-xen-pvh

To build: OvmfPkg/build.sh -p OvmfPkg/XenOvmf.dsc -n $number_of_cpu

Thanks,

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC] OVMF on PVH

[Jan Beulich]

>>> On 02.08.18 at 13:24, <anthony.perard@citrix.com> wrote:
> ## New binary, separated from QEMU/KVM one.
> 
> Right now, the rules to build the firmware are located in
> `OvmfPkg/OvmfPkgX64.dsc`, a platform. I've created a new platform, without
> KVM support, and would like to retire the Xen support from the current
> platform.  For now, I have created OvmfPkg/XenOvmf.dsc. (The change to a
> new platform had been proposed by upstream.)
> 
> That would simplify Xen support in OVMF, and allow to diverge more from
> OVMF where needed. That would mostly be useful during the initial boot
> phase where there lots of `if xen do that, else do that`.
> 
> We can also choose some drivers that can work on both PVH and HVM, for
> example, use a LAPIC timer instead of ACPI Timer.
> 
> All this mean that building OVMF for Xen would need to be change.

And users / tool stacks having to pick the right binary.

> ## Loading binary at 0x10000 (1MB, like hvmloader on HVM)
> 
> Right now, the OVMF blob is loaded (by hvmloader) just below 4GB, with
> hvmloader moving some RAM around to allow that.  But with the use of an
> ELF header, and let libxc load the blob, it was not possible to load the
> blob just below 4GB. (Or at least without modification of the toolstack I
> guess.) So I've attempted to have OVMF been started from a different place
> in memory. Then I had to hack a bit more in order to be able to start OVMF
> from two different location (the current one for HVM guest, and a new one
> that can be used for PVH). That works fine, I'll just have to find out
> what upstream think about that.

I'm of two minds here - as a firmware binary, it doesn't seem to make
sense to load at 0x100000, yet as a hvmloader replacement it might.
However, parts of it will need to stay (runtime services code/data),
and such permanent data better lives at higher addresses imo.

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC] OVMF on PVH

[Wei Liu]

On Thu, Aug 02, 2018 at 12:24:35PM +0100, Anthony PERARD wrote:
> Hi,
> 
> I've been working on booting OVMF in a PVH guest. There are few changes
> that I'd like your comments on. Those are changes that I've already made
> in my private branch, there are either required or will make things
> easier.
> 
> ## Goal
> 
> Have a single blob that can be use for both HVM and PVH guests. And be
> able to start UEFI compatible guests in PVH.
> 
> 
> ## New binary, separated from QEMU/KVM one.
> 
> Right now, the rules to build the firmware are located in
> `OvmfPkg/OvmfPkgX64.dsc`, a platform. I've created a new platform, without
> KVM support, and would like to retire the Xen support from the current
> platform.  For now, I have created OvmfPkg/XenOvmf.dsc. (The change to a
> new platform had been proposed by upstream.)
> 
> That would simplify Xen support in OVMF, and allow to diverge more from
> OVMF where needed. That would mostly be useful during the initial boot
> phase where there lots of `if xen do that, else do that`.
> 
> We can also choose some drivers that can work on both PVH and HVM, for
> example, use a LAPIC timer instead of ACPI Timer.
> 
> All this mean that building OVMF for Xen would need to be change.

Changing things in xen.git is easy and fine. This is more about
introducing a burden for downstream packagers because they need to build
a separate packages.

This is not saying I'm against this idea in any way, just something that
comes to my mind.

> 
> 
> ## ELF header
> 
> Adding an ELF header to OVMF so the blob can be loaded by libxl/libxc
> without modifying those libs.
> 
> That replace a section of the OVMF binary called VarStore by that ELF
> header.  It's empty and libxl doesn't know how to use it. (QEMU/KVM would
> have a flash device loading the store, so it can be written to.)

But long term we might want to support VarStore (that is used to stored
config, right?). How are we going to do that in the future if we
repurpose it now?

> 
> With the ELF header, I can test OVMF by simply adding this in the config
> file:
> 
>     type='pvh'
>     kernel='/path/to/OVMF.fd'
> 
> We could use a modified `hvmloader` to load OVMF on PVH or teach libxc to
> load it at a hardcoded location, but I don't see it as necessary, and if
> we have one less firmware to load, it's probably better.
> 
> With the use of an ELF header comes another issue, which as to do with
> were the binary needs to be loaded initially.
> 
> ## Loading binary at 0x10000 (1MB, like hvmloader on HVM)
> 
> Right now, the OVMF blob is loaded (by hvmloader) just below 4GB, with
> hvmloader moving some RAM around to allow that.  But with the use of an
> ELF header, and let libxc load the blob, it was not possible to load the
> blob just below 4GB. (Or at least without modification of the toolstack I
> guess.) So I've attempted to have OVMF been started from a different place
> in memory. Then I had to hack a bit more in order to be able to start OVMF
> from two different location (the current one for HVM guest, and a new one
> that can be used for PVH). That works fine, I'll just have to find out
> what upstream think about that.

If you specify the virt_addr (?) in the ELF header, libxc should know
how to deal with that? What is missing in the current toolstack code?

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC] OVMF on PVH

[Roger Pau Monné]

On Thu, Aug 02, 2018 at 12:24:35PM +0100, Anthony PERARD wrote:
> Hi,
> 
> I've been working on booting OVMF in a PVH guest. There are few changes
> that I'd like your comments on. Those are changes that I've already made
> in my private branch, there are either required or will make things
> easier.
> 
> ## Goal
> 
> Have a single blob that can be use for both HVM and PVH guests. And be
> able to start UEFI compatible guests in PVH.
> 
> 
> ## New binary, separated from QEMU/KVM one.
> 
> Right now, the rules to build the firmware are located in
> `OvmfPkg/OvmfPkgX64.dsc`, a platform. I've created a new platform, without
> KVM support, and would like to retire the Xen support from the current
> platform.  For now, I have created OvmfPkg/XenOvmf.dsc. (The change to a
> new platform had been proposed by upstream.)

Is it my understanding that after this there will be a generic
platform (for QEMU, KVM and everything else) and a platform for Xen?

> That would simplify Xen support in OVMF, and allow to diverge more from
> OVMF where needed. That would mostly be useful during the initial boot
> phase where there lots of `if xen do that, else do that`.

I fear this might be dangerous because most developers will only test
against the generic platform, so breakage could be introduced more
easily to the Xen platform without upstream noticing?

> We can also choose some drivers that can work on both PVH and HVM, for
> example, use a LAPIC timer instead of ACPI Timer.
> 
> All this mean that building OVMF for Xen would need to be change.

And distros will likely have to provide two different packages, or a
single package with the generic binary and the Xen specific one.

> 
> ## ELF header
> 
> Adding an ELF header to OVMF so the blob can be loaded by libxl/libxc
> without modifying those libs.
> 
> That replace a section of the OVMF binary called VarStore by that ELF
> header.  It's empty and libxl doesn't know how to use it. (QEMU/KVM would
> have a flash device loading the store, so it can be written to.)

Can't you just add the header without replacing anything? The ELF
header is quite small, so I don't think size should be a problem (if
that's why it's placed on top of the VarStore).

> With the ELF header, I can test OVMF by simply adding this in the config
> file:
> 
>     type='pvh'
>     kernel='/path/to/OVMF.fd'

Yes, that's the expectation and is what firmware='ovmf|uefi' should
do behind the scenes.

> We could use a modified `hvmloader` to load OVMF on PVH or teach libxc to
> load it at a hardcoded location, but I don't see it as necessary, and if
> we have one less firmware to load, it's probably better.
> 
> With the use of an ELF header comes another issue, which as to do with
> were the binary needs to be loaded initially.
> 
> ## Loading binary at 0x10000 (1MB, like hvmloader on HVM)

That works well for hvmloader because after it has finished running
the memory can be overwritten without issues. The same is not true for
ovmf which needs to keep data around so that boot services and runtime
services can continue working.

> Right now, the OVMF blob is loaded (by hvmloader) just below 4GB, with
> hvmloader moving some RAM around to allow that.  But with the use of an
> ELF header, and let libxc load the blob, it was not possible to load the
> blob just below 4GB. (Or at least without modification of the toolstack I
> guess.)

libxc/libelf will load the binary in the position specified by the ELF
program headers PhysAddr field (also called LMA). If you execute
`readelf -l <ovmf binary>` you should see the program headers and the
address where they should be loaded.

You can easily modify the LMA in the linker script using the AT
keyword.

> So I've attempted to have OVMF been started from a different place
> in memory. Then I had to hack a bit more in order to be able to start OVMF
> from two different location (the current one for HVM guest, and a new one
> that can be used for PVH). That works fine, I'll just have to find out
> what upstream think about that.

I think the address where ovmf is loaded should be the same for HVM or
PVH if possible. We know that loading at this address works for HVM,
so I think we should keep it for PVH, it's less likely to cause issues
in the future if both are kept in sync.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC] OVMF on PVH

[Anthony PERARD]

On Thu, Aug 02, 2018 at 06:23:54AM -0600, Jan Beulich wrote:
> >>> On 02.08.18 at 13:24, <anthony.perard@citrix.com> wrote:
> > ## Loading binary at 0x10000 (1MB, like hvmloader on HVM)
> > 
> > Right now, the OVMF blob is loaded (by hvmloader) just below 4GB, with
> > hvmloader moving some RAM around to allow that.  But with the use of an
> > ELF header, and let libxc load the blob, it was not possible to load the
> > blob just below 4GB. (Or at least without modification of the toolstack I
> > guess.) So I've attempted to have OVMF been started from a different place
> > in memory. Then I had to hack a bit more in order to be able to start OVMF
> > from two different location (the current one for HVM guest, and a new one
> > that can be used for PVH). That works fine, I'll just have to find out
> > what upstream think about that.
> 
> I'm of two minds here - as a firmware binary, it doesn't seem to make
> sense to load at 0x100000, yet as a hvmloader replacement it might.
> However, parts of it will need to stay (runtime services code/data),
> and such permanent data better lives at higher addresses imo.
> 
> Jan

On Thu, Aug 02, 2018 at 06:21:31PM +0200, Roger Pau Monné wrote:
> That works well for hvmloader because after it has finished running
> the memory can be overwritten without issues. The same is not true for
> ovmf which needs to keep data around so that boot services and runtime
> services can continue working.
> 
> Thanks, Roger.

So there is something to know about the way the OVMF blob (it's actually
called flash device image, or FD), it that there is almost no text (code
to execute). Most of the FD is compressed, with a small section (called
SEC Phase modules) which takes care of bringing up the CPU from 16bits
to 64bits, then some code to find where the next phase is and decompress
the main module.

I don't think any of the initial blob is kept around once the
decompression is done. I did build my OVMF, and use the same exacte blob
for both HVM and PVH, in HVM the blob would be loaded below 4G (no
change), and for PVH it would be loaded at 1M. They both worked fine, so
it doesn't really matter where the initial blob is.

As for where the permanent data actually lives, I am not changing that.
OVMF setup a lot of stuff between 0x800000-0x1400000 as this is where
its page tables are initialy setup (I don't know if they move later),
and all its code is.

Thanks,

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC] OVMF on PVH

[Anthony PERARD]

On Thu, Aug 02, 2018 at 04:44:56PM +0100, Wei Liu wrote:
> On Thu, Aug 02, 2018 at 12:24:35PM +0100, Anthony PERARD wrote:
> > ## New binary, separated from QEMU/KVM one.
> > 
> > Right now, the rules to build the firmware are located in
> > `OvmfPkg/OvmfPkgX64.dsc`, a platform. I've created a new platform, without
> > KVM support, and would like to retire the Xen support from the current
> > platform.  For now, I have created OvmfPkg/XenOvmf.dsc. (The change to a
> > new platform had been proposed by upstream.)
> > 
> > That would simplify Xen support in OVMF, and allow to diverge more from
> > OVMF where needed. That would mostly be useful during the initial boot
> > phase where there lots of `if xen do that, else do that`.
> > 
> > We can also choose some drivers that can work on both PVH and HVM, for
> > example, use a LAPIC timer instead of ACPI Timer.
> > 
> > All this mean that building OVMF for Xen would need to be change.
> 
> Changing things in xen.git is easy and fine. This is more about
> introducing a burden for downstream packagers because they need to build
> a separate packages.

Yeah, that would be a bit annoying for downstream. Maybe we could keep
in the current OVMF implementation for a little while longer and create
the new one which would be Xen specific, but which would introduce PVH
support. That would mean duplicated code upstream.

I don't think it would be possible to add PVH to the current OVMF.

> This is not saying I'm against this idea in any way, just something that
> comes to my mind.
> 
> > 
> > 
> > ## ELF header
> > 
> > Adding an ELF header to OVMF so the blob can be loaded by libxl/libxc
> > without modifying those libs.
> > 
> > That replace a section of the OVMF binary called VarStore by that ELF
> > header.  It's empty and libxl doesn't know how to use it. (QEMU/KVM would
> > have a flash device loading the store, so it can be written to.)
> 
> But long term we might want to support VarStore (that is used to stored
> config, right?). How are we going to do that in the future if we
> repurpose it now?

So, normally on QEMU/KVM, in order to use this VarStore, libvirt (or
user) would add two flash device to qemu, one which gives access to this
VarStore that a guest can write to it (and so would be baked by a file
in the host), and the second flash device would be the rest of OVMF, all
its code.

In Xen, we only ever load the sum of both, and I think hvmloader might
not work if the OVMF blob is smaller than expected (meaning the
toolstack load ovmf without the VarStore so it can attach to QEMU
instead).

I think that VarStore is just a placeholder anyway, in the current
binary.

> > 
> > With the ELF header, I can test OVMF by simply adding this in the config
> > file:
> > 
> >     type='pvh'
> >     kernel='/path/to/OVMF.fd'
> > 
> > We could use a modified `hvmloader` to load OVMF on PVH or teach libxc to
> > load it at a hardcoded location, but I don't see it as necessary, and if
> > we have one less firmware to load, it's probably better.
> > 
> > With the use of an ELF header comes another issue, which as to do with
> > were the binary needs to be loaded initially.
> > 
> > ## Loading binary at 0x10000 (1MB, like hvmloader on HVM)
> > 
> > Right now, the OVMF blob is loaded (by hvmloader) just below 4GB, with
> > hvmloader moving some RAM around to allow that.  But with the use of an
> > ELF header, and let libxc load the blob, it was not possible to load the
> > blob just below 4GB. (Or at least without modification of the toolstack I
> > guess.) So I've attempted to have OVMF been started from a different place
> > in memory. Then I had to hack a bit more in order to be able to start OVMF
> > from two different location (the current one for HVM guest, and a new one
> > that can be used for PVH). That works fine, I'll just have to find out
> > what upstream think about that.
> 
> If you specify the virt_addr (?) in the ELF header, libxc should know
> how to deal with that? What is missing in the current toolstack code?

RAM.

If I set the ELF header properly so that it is written that the blob
must be loaded in the guest RAM just below 4G, libxc will not be able to
do anything because there is no RAM, that space is suppose to be a mem
hole, or something like that, I forgot the exact detail :(. I think also
that libxl would try to load other data like the startinfo page past the
kernel (OVMF here) and that would mean past 4G.

(In my notes from when I tried, I've written "libxc will not move ram
like hvmloader would; [and with >4G for the guest] libxc can not map
anything past 0xfee00000, and I want 0xffe00000", whatever that mean,
can find out where I found that 0xfee00000 address.)

Anyway, it didn't make sense to be to move the RAM around for some
code/data that is only use very early and never used again. And it seams
easier to have initial blob somewhere else rather trying to teach libxc
about this weirdness.

Thanks,

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC] OVMF on PVH

[Roger Pau Monné]

On Wed, Aug 08, 2018 at 04:02:09PM +0100, Anthony PERARD wrote:
> On Thu, Aug 02, 2018 at 04:44:56PM +0100, Wei Liu wrote:
> > On Thu, Aug 02, 2018 at 12:24:35PM +0100, Anthony PERARD wrote:
> > > ## Loading binary at 0x10000 (1MB, like hvmloader on HVM)
> > > 
> > > Right now, the OVMF blob is loaded (by hvmloader) just below 4GB, with
> > > hvmloader moving some RAM around to allow that.  But with the use of an
> > > ELF header, and let libxc load the blob, it was not possible to load the
> > > blob just below 4GB. (Or at least without modification of the toolstack I
> > > guess.) So I've attempted to have OVMF been started from a different place
> > > in memory. Then I had to hack a bit more in order to be able to start OVMF
> > > from two different location (the current one for HVM guest, and a new one
> > > that can be used for PVH). That works fine, I'll just have to find out
> > > what upstream think about that.
> > 
> > If you specify the virt_addr (?) in the ELF header, libxc should know
> > how to deal with that? What is missing in the current toolstack code?
> 
> RAM.
> 
> If I set the ELF header properly so that it is written that the blob
> must be loaded in the guest RAM just below 4G, libxc will not be able to
> do anything because there is no RAM, that space is suppose to be a mem
> hole, or something like that, I forgot the exact detail :(. I think also
> that libxl would try to load other data like the startinfo page past the
> kernel (OVMF here) and that would mean past 4G.
> 
> (In my notes from when I tried, I've written "libxc will not move ram
> like hvmloader would; [and with >4G for the guest] libxc can not map
> anything past 0xfee00000, and I want 0xffe00000", whatever that mean,
> can find out where I found that 0xfee00000 address.)

This is arguably a bug of the current libxc implementation. libxc
should always populate the place where the kernel will be loaded
first, and then populate the rest of the memory map using the
remaining memory.

Given your other reply that mentions that the initial load region is
discarded afterwards because ovmf relocates itself I think I'm OK with
loading it wherever it's easier.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC] OVMF on PVH

[Anthony PERARD]

On Thu, Aug 02, 2018 at 06:21:31PM +0200, Roger Pau Monné wrote:
> On Thu, Aug 02, 2018 at 12:24:35PM +0100, Anthony PERARD wrote:
> > ## New binary, separated from QEMU/KVM one.
> > 
> > Right now, the rules to build the firmware are located in
> > `OvmfPkg/OvmfPkgX64.dsc`, a platform. I've created a new platform, without
> > KVM support, and would like to retire the Xen support from the current
> > platform.  For now, I have created OvmfPkg/XenOvmf.dsc. (The change to a
> > new platform had been proposed by upstream.)
> 
> Is it my understanding that after this there will be a generic
> platform (for QEMU, KVM and everything else) and a platform for Xen?

It's not really generic. As far as I can tell, it's only QEMU/KVM (and
Xen now). But yes, there would be a separate platform (or file when
built) for Xen.

It would be possible to keep Xen support in the current "generic"
platform, but that would mean that code would be duplicated.

I think the OVMF maintainer would be happier if the Xen support was
separated, even now without PVH.

> > That would simplify Xen support in OVMF, and allow to diverge more from
> > OVMF where needed. That would mostly be useful during the initial boot
> > phase where there lots of `if xen do that, else do that`.
> 
> I fear this might be dangerous because most developers will only test
> against the generic platform, so breakage could be introduced more
> easily to the Xen platform without upstream noticing?

That would not change anything. Upstream maintainers doesn't test booting
OVMF in a Xen guest so there are sometime where only Xen is broken, even
if they try hard to not break it.

As for build, I actually don't know if that would change anything.
Anyway, we've got osstest.

> > We can also choose some drivers that can work on both PVH and HVM, for
> > example, use a LAPIC timer instead of ACPI Timer.
> > 
> > All this mean that building OVMF for Xen would need to be change.
> 
> And distros will likely have to provide two different packages, or a
> single package with the generic binary and the Xen specific one.

Yes. But it is already an issue sometime. For example, I cann't use the
ovmf package from Arch Linux because it is cut in two part (vars and
code), and Xen only works when both part are together (/bin/cat vars
code).

> > 
> > ## ELF header
> > 
> > Adding an ELF header to OVMF so the blob can be loaded by libxl/libxc
> > without modifying those libs.
> > 
> > That replace a section of the OVMF binary called VarStore by that ELF
> > header.  It's empty and libxl doesn't know how to use it. (QEMU/KVM would
> > have a flash device loading the store, so it can be written to.)
> 
> Can't you just add the header without replacing anything? The ELF
> header is quite small, so I don't think size should be a problem (if
> that's why it's placed on top of the VarStore).

I don't think hvmloader can cope with a different size right now. And
that section doesn't contain anything useful, it would need to be
separated in order to be useful.

> > With the ELF header, I can test OVMF by simply adding this in the config
> > file:
> > 
> >     type='pvh'
> >     kernel='/path/to/OVMF.fd'
> 
> Yes, that's the expectation and is what firmware='ovmf|uefi' should
> do behind the scenes.
> 
> > We could use a modified `hvmloader` to load OVMF on PVH or teach libxc to
> > load it at a hardcoded location, but I don't see it as necessary, and if
> > we have one less firmware to load, it's probably better.
> > 
> > With the use of an ELF header comes another issue, which as to do with
> > were the binary needs to be loaded initially.
> > 
> > ## Loading binary at 0x10000 (1MB, like hvmloader on HVM)
> 
> That works well for hvmloader because after it has finished running
> the memory can be overwritten without issues. The same is not true for
> ovmf which needs to keep data around so that boot services and runtime
> services can continue working.
> 
> > Right now, the OVMF blob is loaded (by hvmloader) just below 4GB, with
> > hvmloader moving some RAM around to allow that.  But with the use of an
> > ELF header, and let libxc load the blob, it was not possible to load the
> > blob just below 4GB. (Or at least without modification of the toolstack I
> > guess.)
> 
> libxc/libelf will load the binary in the position specified by the ELF
> program headers PhysAddr field (also called LMA). If you execute
> `readelf -l <ovmf binary>` you should see the program headers and the
> address where they should be loaded.
> 
> You can easily modify the LMA in the linker script using the AT
> keyword.

I don't know what to answer to that. OVMF isn't an ELF. There isn't an
easy way to tell where it should be loaded. There is no linker script
(at least that link everything together), but a collection python script
and some program, and configuration. I don't need to run `readelf`
because it either doesn't work or it would print the value that I've set
manually.

The issue it that, if I set LMA to 0xffe00000, libxc fails.

To reuse names from elf.h, what my header would be if I try to load ovmf
where it is currently (just bellow 4G), that would be (with only
addr/size, not the flags):
Elf32_Ehdr hdr = {
.e_entry = 0xffffffd0,
};
Elf32_Phdr phdr = {
.p_paddr = 0xffe00000,
.p_vaddr = 0xffe00000;
.p_filesz = 0x00200000,
.p_memsz = 0x00200000,
};

> > So I've attempted to have OVMF been started from a different place
> > in memory. Then I had to hack a bit more in order to be able to start OVMF
> > from two different location (the current one for HVM guest, and a new one
> > that can be used for PVH). That works fine, I'll just have to find out
> > what upstream think about that.
> 
> I think the address where ovmf is loaded should be the same for HVM or
> PVH if possible. We know that loading at this address works for HVM,
> so I think we should keep it for PVH, it's less likely to cause issues
> in the future if both are kept in sync.

Indeed, but that doesn't work well, and I find it strange that we have
to move RAM around for something that is only use very early and never
used again.

Thanks,

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC] OVMF on PVH

[Anthony PERARD]

On Thu, Aug 02, 2018 at 12:24:35PM +0100, Anthony PERARD wrote:
> That replace a section of the OVMF binary called VarStore by that ELF
> header.  It's empty and libxl doesn't know how to use it. (QEMU/KVM would
> have a flash device loading the store, so it can be written to.)

> ## Loading binary at 0x10000 (1MB, like hvmloader on HVM)

Mostly notes for myself:
I've realize that moving the initial binary location might also move the
VarStore (depending on how OVMF build is configured). The VarStore
location is used later by OVMF, if that location looks like a VarStore
(there is a header) then OVMF use it, otherwise OVMF simply initialise
it.

Anyway, just a reminder that I need to pay attention to where it is, as
maybe someone is trying to use it on HVM.

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC] OVMF on PVH

[Wei Liu]

On Wed, Aug 08, 2018 at 04:02:09PM +0100, Anthony PERARD wrote:
> > But long term we might want to support VarStore (that is used to stored
> > config, right?). How are we going to do that in the future if we
> > repurpose it now?
> 
> So, normally on QEMU/KVM, in order to use this VarStore, libvirt (or
> user) would add two flash device to qemu, one which gives access to this
> VarStore that a guest can write to it (and so would be baked by a file
> in the host), and the second flash device would be the rest of OVMF, all
> its code.
> 
> In Xen, we only ever load the sum of both, and I think hvmloader might
> not work if the OVMF blob is smaller than expected (meaning the
> toolstack load ovmf without the VarStore so it can attach to QEMU
> instead).
> 
> I think that VarStore is just a placeholder anyway, in the current
> binary.

OK. I think I understand better now. I'm fine with repurposing the
in-binary VarStore for something else, because if we really want to
support it we should expose another flash device to the guest just like
KVM does.

> 
> > If you specify the virt_addr (?) in the ELF header, libxc should know
> > how to deal with that? What is missing in the current toolstack code?
> 
> RAM.
> 
> If I set the ELF header properly so that it is written that the blob
> must be loaded in the guest RAM just below 4G, libxc will not be able to
> do anything because there is no RAM, that space is suppose to be a mem
> hole, or something like that, I forgot the exact detail :(. I think also
> that libxl would try to load other data like the startinfo page past the
> kernel (OVMF here) and that would mean past 4G.
> 
> (In my notes from when I tried, I've written "libxc will not move ram
> like hvmloader would; [and with >4G for the guest] libxc can not map
> anything past 0xfee00000, and I want 0xffe00000", whatever that mean,
> can find out where I found that 0xfee00000 address.)
> 
> Anyway, it didn't make sense to be to move the RAM around for some
> code/data that is only use very early and never used again. And it seams
> easier to have initial blob somewhere else rather trying to teach libxc
> about this weirdness.
> 

OK. This makes sense.

Wei.


> Thanks,
> 
> -- 
> Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel