Xen PVH support in grub2

Xen PVH support in grub2

[Juergen Gross]

I'm thinking about to add support for Xen PVH guests to grub2.

Basically I see two options how to do it:

a) add PVH support to current grub.xen (both, 32- and 64-bit), in
   order to use the same grub binary for either pv-domains or
   pvh-domains
b) create a new variant grub.xenpvh capable only to boot pvh-domains

Are there any preferences which way to go?


Juergen

Re: Xen PVH support in grub2

[George Dunlap]

On Fri, Sep 29, 2017 at 3:46 PM, Juergen Gross <jgross@suse.com> wrote:
> I'm thinking about to add support for Xen PVH guests to grub2.
>
> Basically I see two options how to do it:
>
> a) add PVH support to current grub.xen (both, 32- and 64-bit), in
>    order to use the same grub binary for either pv-domains or
>    pvh-domains
> b) create a new variant grub.xenpvh capable only to boot pvh-domains
>
> Are there any preferences which way to go?

Well from a user perspective, it's better to have fewer binaries, and
from a builder / distro perspective it's better to have fewer builds.
So I would only do 'b' if 'a' is significantly more complicated.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[George Dunlap]

On Fri, Sep 29, 2017 at 3:46 PM, Juergen Gross <jgross@suse.com> wrote:
> I'm thinking about to add support for Xen PVH guests to grub2.
>
> Basically I see two options how to do it:
>
> a) add PVH support to current grub.xen (both, 32- and 64-bit), in
>    order to use the same grub binary for either pv-domains or
>    pvh-domains
> b) create a new variant grub.xenpvh capable only to boot pvh-domains
>
> Are there any preferences which way to go?

Well from a user perspective, it's better to have fewer binaries, and
from a builder / distro perspective it's better to have fewer builds.
So I would only do 'b' if 'a' is significantly more complicated.

 -George

Re: Xen PVH support in grub2

[Roger Pau Monné]

On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
> I'm thinking about to add support for Xen PVH guests to grub2.
> 
> Basically I see two options how to do it:
> 
> a) add PVH support to current grub.xen (both, 32- and 64-bit), in
>    order to use the same grub binary for either pv-domains or
>    pvh-domains
> b) create a new variant grub.xenpvh capable only to boot pvh-domains
> 
> Are there any preferences which way to go?

IMHO, it should be possible to create a grub binary that can be loaded
in PVH mode but that's not tied to a specific bitness. Let's avoid
having a grub32 and grub64, that's not needed at all for PVH.

Then, I also wonder whether it would make sense for this grub to load
the kernel using the PVH entry point or the native entry point. Would
it be possible to boot a Linux kernel up to the point where cpuid can
be used inside of a PVH container?

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Roger Pau Monné]

On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
> I'm thinking about to add support for Xen PVH guests to grub2.
> 
> Basically I see two options how to do it:
> 
> a) add PVH support to current grub.xen (both, 32- and 64-bit), in
>    order to use the same grub binary for either pv-domains or
>    pvh-domains
> b) create a new variant grub.xenpvh capable only to boot pvh-domains
> 
> Are there any preferences which way to go?

IMHO, it should be possible to create a grub binary that can be loaded
in PVH mode but that's not tied to a specific bitness. Let's avoid
having a grub32 and grub64, that's not needed at all for PVH.

Then, I also wonder whether it would make sense for this grub to load
the kernel using the PVH entry point or the native entry point. Would
it be possible to boot a Linux kernel up to the point where cpuid can
be used inside of a PVH container?

Thanks, Roger.

Re: Xen PVH support in grub2

[Juergen Gross]

On 29/09/17 17:24, Roger Pau Monné wrote:
> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>> I'm thinking about to add support for Xen PVH guests to grub2.
>>
>> Basically I see two options how to do it:
>>
>> a) add PVH support to current grub.xen (both, 32- and 64-bit), in
>>    order to use the same grub binary for either pv-domains or
>>    pvh-domains
>> b) create a new variant grub.xenpvh capable only to boot pvh-domains
>>
>> Are there any preferences which way to go?
> 
> IMHO, it should be possible to create a grub binary that can be loaded
> in PVH mode but that's not tied to a specific bitness. Let's avoid
> having a grub32 and grub64, that's not needed at all for PVH.

Right. OTOH we have those two variants already for PV. So the question
is whether we want grub32 and grub64 or do we want grub32, grub64 and
grub.pvh.

> Then, I also wonder whether it would make sense for this grub to load
> the kernel using the PVH entry point or the native entry point. Would
> it be possible to boot a Linux kernel up to the point where cpuid can
> be used inside of a PVH container?

I don't think today's Linux allows that. This has been discussed
very thoroughly at the time Boris added PVH V2 support to the kernel.

So my plan was to use the PVH entry point of the kernel.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Juergen Gross]

On 29/09/17 17:24, Roger Pau Monné wrote:
> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>> I'm thinking about to add support for Xen PVH guests to grub2.
>>
>> Basically I see two options how to do it:
>>
>> a) add PVH support to current grub.xen (both, 32- and 64-bit), in
>>    order to use the same grub binary for either pv-domains or
>>    pvh-domains
>> b) create a new variant grub.xenpvh capable only to boot pvh-domains
>>
>> Are there any preferences which way to go?
> 
> IMHO, it should be possible to create a grub binary that can be loaded
> in PVH mode but that's not tied to a specific bitness. Let's avoid
> having a grub32 and grub64, that's not needed at all for PVH.

Right. OTOH we have those two variants already for PV. So the question
is whether we want grub32 and grub64 or do we want grub32, grub64 and
grub.pvh.

> Then, I also wonder whether it would make sense for this grub to load
> the kernel using the PVH entry point or the native entry point. Would
> it be possible to boot a Linux kernel up to the point where cpuid can
> be used inside of a PVH container?

I don't think today's Linux allows that. This has been discussed
very thoroughly at the time Boris added PVH V2 support to the kernel.

So my plan was to use the PVH entry point of the kernel.


Juergen

Re: Xen PVH support in grub2

[George Dunlap]

On Fri, Sep 29, 2017 at 4:24 PM, Roger Pau Monné <roger.pau@citrix.com> wrote:
> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>> I'm thinking about to add support for Xen PVH guests to grub2.
>>
>> Basically I see two options how to do it:
>>
>> a) add PVH support to current grub.xen (both, 32- and 64-bit), in
>>    order to use the same grub binary for either pv-domains or
>>    pvh-domains
>> b) create a new variant grub.xenpvh capable only to boot pvh-domains
>>
>> Are there any preferences which way to go?
>
> IMHO, it should be possible to create a grub binary that can be loaded
> in PVH mode but that's not tied to a specific bitness. Let's avoid
> having a grub32 and grub64, that's not needed at all for PVH.

Oh, right -- yeah, a single binary that can do both is definitely
worth having a separate binary from the PV one.

 -george

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[George Dunlap]

On Fri, Sep 29, 2017 at 4:24 PM, Roger Pau Monné <roger.pau@citrix.com> wrote:
> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>> I'm thinking about to add support for Xen PVH guests to grub2.
>>
>> Basically I see two options how to do it:
>>
>> a) add PVH support to current grub.xen (both, 32- and 64-bit), in
>>    order to use the same grub binary for either pv-domains or
>>    pvh-domains
>> b) create a new variant grub.xenpvh capable only to boot pvh-domains
>>
>> Are there any preferences which way to go?
>
> IMHO, it should be possible to create a grub binary that can be loaded
> in PVH mode but that's not tied to a specific bitness. Let's avoid
> having a grub32 and grub64, that's not needed at all for PVH.

Oh, right -- yeah, a single binary that can do both is definitely
worth having a separate binary from the PV one.

 -george

Re: Xen PVH support in grub2

[Roger Pau Monné]

On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
> On 29/09/17 17:24, Roger Pau Monné wrote:
> > On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
> > Then, I also wonder whether it would make sense for this grub to load
> > the kernel using the PVH entry point or the native entry point. Would
> > it be possible to boot a Linux kernel up to the point where cpuid can
> > be used inside of a PVH container?
> 
> I don't think today's Linux allows that. This has been discussed
> very thoroughly at the time Boris added PVH V2 support to the kernel.

OK, I'm not going to insist on that, but my plans for FreeBSD is to
make the native entry point capable of booting inside of a PVH
container up to the point where cpuid (or whatever method) can be used
to detect the environment.

Do you recall what's preventing the native entry point from booting
inside of a PVH container? If certain emulated devices not present are
needed at early boot we could look into either replacing them with
other options available inside of a PVH container, or as a last resort
making them available on a PVH container.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Roger Pau Monné]

On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
> On 29/09/17 17:24, Roger Pau Monné wrote:
> > On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
> > Then, I also wonder whether it would make sense for this grub to load
> > the kernel using the PVH entry point or the native entry point. Would
> > it be possible to boot a Linux kernel up to the point where cpuid can
> > be used inside of a PVH container?
> 
> I don't think today's Linux allows that. This has been discussed
> very thoroughly at the time Boris added PVH V2 support to the kernel.

OK, I'm not going to insist on that, but my plans for FreeBSD is to
make the native entry point capable of booting inside of a PVH
container up to the point where cpuid (or whatever method) can be used
to detect the environment.

Do you recall what's preventing the native entry point from booting
inside of a PVH container? If certain emulated devices not present are
needed at early boot we could look into either replacing them with
other options available inside of a PVH container, or as a last resort
making them available on a PVH container.

Roger.

Re: Xen PVH support in grub2

[Boris Ostrovsky]

On 09/29/2017 11:51 AM, Roger Pau Monné wrote:
> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>> Then, I also wonder whether it would make sense for this grub to load
>>> the kernel using the PVH entry point or the native entry point. Would
>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>> be used inside of a PVH container?
>> I don't think today's Linux allows that. This has been discussed
>> very thoroughly at the time Boris added PVH V2 support to the kernel.
> OK, I'm not going to insist on that, but my plans for FreeBSD is to
> make the native entry point capable of booting inside of a PVH
> container up to the point where cpuid (or whatever method) can be used
> to detect the environment.
>
> Do you recall what's preventing the native entry point from booting
> inside of a PVH container? If certain emulated devices not present are
> needed at early boot we could look into either replacing them with
> other options available inside of a PVH container, or as a last resort
> making them available on a PVH container.

Very much IIRC one of the reasons was the fact that zeropage
(bootparams) needed to be properly formatted. And the hypercall page
needs to be set up.



-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Boris Ostrovsky]

On 09/29/2017 11:51 AM, Roger Pau Monné wrote:
> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>> Then, I also wonder whether it would make sense for this grub to load
>>> the kernel using the PVH entry point or the native entry point. Would
>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>> be used inside of a PVH container?
>> I don't think today's Linux allows that. This has been discussed
>> very thoroughly at the time Boris added PVH V2 support to the kernel.
> OK, I'm not going to insist on that, but my plans for FreeBSD is to
> make the native entry point capable of booting inside of a PVH
> container up to the point where cpuid (or whatever method) can be used
> to detect the environment.
>
> Do you recall what's preventing the native entry point from booting
> inside of a PVH container? If certain emulated devices not present are
> needed at early boot we could look into either replacing them with
> other options available inside of a PVH container, or as a last resort
> making them available on a PVH container.

Very much IIRC one of the reasons was the fact that zeropage
(bootparams) needed to be properly formatted. And the hypercall page
needs to be set up.



-boris

Re: Xen PVH support in grub2

[Roger Pau Monné]

On Fri, Sep 29, 2017 at 05:02:48PM +0000, Boris Ostrovsky wrote:
> On 09/29/2017 11:51 AM, Roger Pau Monné wrote:
> > On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
> >> On 29/09/17 17:24, Roger Pau Monné wrote:
> >>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
> >>> Then, I also wonder whether it would make sense for this grub to load
> >>> the kernel using the PVH entry point or the native entry point. Would
> >>> it be possible to boot a Linux kernel up to the point where cpuid can
> >>> be used inside of a PVH container?
> >> I don't think today's Linux allows that. This has been discussed
> >> very thoroughly at the time Boris added PVH V2 support to the kernel.
> > OK, I'm not going to insist on that, but my plans for FreeBSD is to
> > make the native entry point capable of booting inside of a PVH
> > container up to the point where cpuid (or whatever method) can be used
> > to detect the environment.
> >
> > Do you recall what's preventing the native entry point from booting
> > inside of a PVH container? If certain emulated devices not present are
> > needed at early boot we could look into either replacing them with
> > other options available inside of a PVH container, or as a last resort
> > making them available on a PVH container.
> 
> Very much IIRC one of the reasons was the fact that zeropage
> (bootparams) needed to be properly formatted. And the hypercall page
> needs to be set up.

But in this case bootparams is going to be setup by grub, so it should
be fine (just like it's done on bare metal).

Couldn't the hypercall page be setup at some point during early boot?
Not sure if setting it up at the same point HVM does would be fine for
PVH?

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Roger Pau Monné]

On Fri, Sep 29, 2017 at 05:02:48PM +0000, Boris Ostrovsky wrote:
> On 09/29/2017 11:51 AM, Roger Pau Monné wrote:
> > On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
> >> On 29/09/17 17:24, Roger Pau Monné wrote:
> >>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
> >>> Then, I also wonder whether it would make sense for this grub to load
> >>> the kernel using the PVH entry point or the native entry point. Would
> >>> it be possible to boot a Linux kernel up to the point where cpuid can
> >>> be used inside of a PVH container?
> >> I don't think today's Linux allows that. This has been discussed
> >> very thoroughly at the time Boris added PVH V2 support to the kernel.
> > OK, I'm not going to insist on that, but my plans for FreeBSD is to
> > make the native entry point capable of booting inside of a PVH
> > container up to the point where cpuid (or whatever method) can be used
> > to detect the environment.
> >
> > Do you recall what's preventing the native entry point from booting
> > inside of a PVH container? If certain emulated devices not present are
> > needed at early boot we could look into either replacing them with
> > other options available inside of a PVH container, or as a last resort
> > making them available on a PVH container.
> 
> Very much IIRC one of the reasons was the fact that zeropage
> (bootparams) needed to be properly formatted. And the hypercall page
> needs to be set up.

But in this case bootparams is going to be setup by grub, so it should
be fine (just like it's done on bare metal).

Couldn't the hypercall page be setup at some point during early boot?
Not sure if setting it up at the same point HVM does would be fine for
PVH?

Roger.

Re: Xen PVH support in grub2

[Boris Ostrovsky]

On 09/29/2017 01:07 PM, Roger Pau Monné wrote:
> On Fri, Sep 29, 2017 at 05:02:48PM +0000, Boris Ostrovsky wrote:
>> On 09/29/2017 11:51 AM, Roger Pau Monné wrote:
>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>> be used inside of a PVH container?
>>>> I don't think today's Linux allows that. This has been discussed
>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>> make the native entry point capable of booting inside of a PVH
>>> container up to the point where cpuid (or whatever method) can be used
>>> to detect the environment.
>>>
>>> Do you recall what's preventing the native entry point from booting
>>> inside of a PVH container? If certain emulated devices not present are
>>> needed at early boot we could look into either replacing them with
>>> other options available inside of a PVH container, or as a last resort
>>> making them available on a PVH container.
>> Very much IIRC one of the reasons was the fact that zeropage
>> (bootparams) needed to be properly formatted. And the hypercall page
>> needs to be set up.
> But in this case bootparams is going to be setup by grub, so it should
> be fine (just like it's done on bare metal).

Yes, I think so.

>
> Couldn't the hypercall page be setup at some point during early boot?
> Not sure if setting it up at the same point HVM does would be fine for
> PVH?

Probably --- I think the only reason we set it early is because we need
to call XENMEM_memory_map to set bootparams. One other thing I noticed
is that we need to set acpi_irq_model before hypervisor is discovered
(can't remember why) but I suppose this can be worked around.

Having said that --- since for direct boot we still need to go via
PVH-specific entry point I am not sure how much we will gain by having
grub avoid it.

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Boris Ostrovsky]

On 09/29/2017 01:07 PM, Roger Pau Monné wrote:
> On Fri, Sep 29, 2017 at 05:02:48PM +0000, Boris Ostrovsky wrote:
>> On 09/29/2017 11:51 AM, Roger Pau Monné wrote:
>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>> be used inside of a PVH container?
>>>> I don't think today's Linux allows that. This has been discussed
>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>> make the native entry point capable of booting inside of a PVH
>>> container up to the point where cpuid (or whatever method) can be used
>>> to detect the environment.
>>>
>>> Do you recall what's preventing the native entry point from booting
>>> inside of a PVH container? If certain emulated devices not present are
>>> needed at early boot we could look into either replacing them with
>>> other options available inside of a PVH container, or as a last resort
>>> making them available on a PVH container.
>> Very much IIRC one of the reasons was the fact that zeropage
>> (bootparams) needed to be properly formatted. And the hypercall page
>> needs to be set up.
> But in this case bootparams is going to be setup by grub, so it should
> be fine (just like it's done on bare metal).

Yes, I think so.

>
> Couldn't the hypercall page be setup at some point during early boot?
> Not sure if setting it up at the same point HVM does would be fine for
> PVH?

Probably --- I think the only reason we set it early is because we need
to call XENMEM_memory_map to set bootparams. One other thing I noticed
is that we need to set acpi_irq_model before hypervisor is discovered
(can't remember why) but I suppose this can be worked around.

Having said that --- since for direct boot we still need to go via
PVH-specific entry point I am not sure how much we will gain by having
grub avoid it.

-boris

Re: Xen PVH support in grub2

[Roger Pau Monné]

On Fri, Sep 29, 2017 at 05:22:25PM +0000, Boris Ostrovsky wrote:
> On 09/29/2017 01:07 PM, Roger Pau Monné wrote:
> > On Fri, Sep 29, 2017 at 05:02:48PM +0000, Boris Ostrovsky wrote:
> >> On 09/29/2017 11:51 AM, Roger Pau Monné wrote:
> >>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
> >>>> On 29/09/17 17:24, Roger Pau Monné wrote:
> >>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
> >>>>> Then, I also wonder whether it would make sense for this grub to load
> >>>>> the kernel using the PVH entry point or the native entry point. Would
> >>>>> it be possible to boot a Linux kernel up to the point where cpuid can
> >>>>> be used inside of a PVH container?
> >>>> I don't think today's Linux allows that. This has been discussed
> >>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
> >>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
> >>> make the native entry point capable of booting inside of a PVH
> >>> container up to the point where cpuid (or whatever method) can be used
> >>> to detect the environment.
> >>>
> >>> Do you recall what's preventing the native entry point from booting
> >>> inside of a PVH container? If certain emulated devices not present are
> >>> needed at early boot we could look into either replacing them with
> >>> other options available inside of a PVH container, or as a last resort
> >>> making them available on a PVH container.
> >> Very much IIRC one of the reasons was the fact that zeropage
> >> (bootparams) needed to be properly formatted. And the hypercall page
> >> needs to be set up.
> > But in this case bootparams is going to be setup by grub, so it should
> > be fine (just like it's done on bare metal).
> 
> Yes, I think so.
> 
> >
> > Couldn't the hypercall page be setup at some point during early boot?
> > Not sure if setting it up at the same point HVM does would be fine for
> > PVH?
> 
> Probably --- I think the only reason we set it early is because we need
> to call XENMEM_memory_map to set bootparams. One other thing I noticed
> is that we need to set acpi_irq_model before hypervisor is discovered
> (can't remember why) but I suppose this can be worked around.
> 
> Having said that --- since for direct boot we still need to go via
> PVH-specific entry point I am not sure how much we will gain by having
> grub avoid it.

Being able to boot inside of a PVH container using the native entry
point would prevent us from having to add PVH loader specific code to
each firmware we plan to support in PVH mode.

If Linux must be started using the PVH entry point in order to run
inside of a PVH container, it means we would have to add a PVH loader
to OVFM and grub at least.

OTOH if Linux is capable of booting from the native entry point inside
of a PVH container, we would only have to port OVMF and grub in order
to work inside of a PVH container, leaving the rest of the logic
untouched.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Roger Pau Monné]

On Fri, Sep 29, 2017 at 05:22:25PM +0000, Boris Ostrovsky wrote:
> On 09/29/2017 01:07 PM, Roger Pau Monné wrote:
> > On Fri, Sep 29, 2017 at 05:02:48PM +0000, Boris Ostrovsky wrote:
> >> On 09/29/2017 11:51 AM, Roger Pau Monné wrote:
> >>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
> >>>> On 29/09/17 17:24, Roger Pau Monné wrote:
> >>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
> >>>>> Then, I also wonder whether it would make sense for this grub to load
> >>>>> the kernel using the PVH entry point or the native entry point. Would
> >>>>> it be possible to boot a Linux kernel up to the point where cpuid can
> >>>>> be used inside of a PVH container?
> >>>> I don't think today's Linux allows that. This has been discussed
> >>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
> >>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
> >>> make the native entry point capable of booting inside of a PVH
> >>> container up to the point where cpuid (or whatever method) can be used
> >>> to detect the environment.
> >>>
> >>> Do you recall what's preventing the native entry point from booting
> >>> inside of a PVH container? If certain emulated devices not present are
> >>> needed at early boot we could look into either replacing them with
> >>> other options available inside of a PVH container, or as a last resort
> >>> making them available on a PVH container.
> >> Very much IIRC one of the reasons was the fact that zeropage
> >> (bootparams) needed to be properly formatted. And the hypercall page
> >> needs to be set up.
> > But in this case bootparams is going to be setup by grub, so it should
> > be fine (just like it's done on bare metal).
> 
> Yes, I think so.
> 
> >
> > Couldn't the hypercall page be setup at some point during early boot?
> > Not sure if setting it up at the same point HVM does would be fine for
> > PVH?
> 
> Probably --- I think the only reason we set it early is because we need
> to call XENMEM_memory_map to set bootparams. One other thing I noticed
> is that we need to set acpi_irq_model before hypervisor is discovered
> (can't remember why) but I suppose this can be worked around.
> 
> Having said that --- since for direct boot we still need to go via
> PVH-specific entry point I am not sure how much we will gain by having
> grub avoid it.

Being able to boot inside of a PVH container using the native entry
point would prevent us from having to add PVH loader specific code to
each firmware we plan to support in PVH mode.

If Linux must be started using the PVH entry point in order to run
inside of a PVH container, it means we would have to add a PVH loader
to OVFM and grub at least.

OTOH if Linux is capable of booting from the native entry point inside
of a PVH container, we would only have to port OVMF and grub in order
to work inside of a PVH container, leaving the rest of the logic
untouched.

Thanks, Roger.

Re: Xen PVH support in grub2

[Boris Ostrovsky]

On 10/03/2017 04:56 AM, Roger Pau Monné wrote:
> On Fri, Sep 29, 2017 at 05:22:25PM +0000, Boris Ostrovsky wrote:
>> On 09/29/2017 01:07 PM, Roger Pau Monné wrote:
>>> On Fri, Sep 29, 2017 at 05:02:48PM +0000, Boris Ostrovsky wrote:
>>>> On 09/29/2017 11:51 AM, Roger Pau Monné wrote:
>>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>>> be used inside of a PVH container?
>>>>>> I don't think today's Linux allows that. This has been discussed
>>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>>> make the native entry point capable of booting inside of a PVH
>>>>> container up to the point where cpuid (or whatever method) can be used
>>>>> to detect the environment.
>>>>>
>>>>> Do you recall what's preventing the native entry point from booting
>>>>> inside of a PVH container? If certain emulated devices not present are
>>>>> needed at early boot we could look into either replacing them with
>>>>> other options available inside of a PVH container, or as a last resort
>>>>> making them available on a PVH container.
>>>> Very much IIRC one of the reasons was the fact that zeropage
>>>> (bootparams) needed to be properly formatted. And the hypercall page
>>>> needs to be set up.
>>> But in this case bootparams is going to be setup by grub, so it should
>>> be fine (just like it's done on bare metal).
>> Yes, I think so.
>>
>>> Couldn't the hypercall page be setup at some point during early boot?
>>> Not sure if setting it up at the same point HVM does would be fine for
>>> PVH?
>> Probably --- I think the only reason we set it early is because we need
>> to call XENMEM_memory_map to set bootparams. One other thing I noticed
>> is that we need to set acpi_irq_model before hypervisor is discovered
>> (can't remember why) but I suppose this can be worked around.
>>
>> Having said that --- since for direct boot we still need to go via
>> PVH-specific entry point I am not sure how much we will gain by having
>> grub avoid it.
> Being able to boot inside of a PVH container using the native entry
> point would prevent us from having to add PVH loader specific code to
> each firmware we plan to support in PVH mode.
>
> If Linux must be started using the PVH entry point in order to run
> inside of a PVH container, it means we would have to add a PVH loader
> to OVFM and grub at least.
>
> OTOH if Linux is capable of booting from the native entry point inside
> of a PVH container, we would only have to port OVMF and grub in order
> to work inside of a PVH container, leaving the rest of the logic
> untouched.

Right, I understand that. I was simply trying to say that PVH-specific 
entry point is likely to stay for us to continue booting PVH guests 
directly.

-boris

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Boris Ostrovsky]

On 10/03/2017 04:56 AM, Roger Pau Monné wrote:
> On Fri, Sep 29, 2017 at 05:22:25PM +0000, Boris Ostrovsky wrote:
>> On 09/29/2017 01:07 PM, Roger Pau Monné wrote:
>>> On Fri, Sep 29, 2017 at 05:02:48PM +0000, Boris Ostrovsky wrote:
>>>> On 09/29/2017 11:51 AM, Roger Pau Monné wrote:
>>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>>> be used inside of a PVH container?
>>>>>> I don't think today's Linux allows that. This has been discussed
>>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>>> make the native entry point capable of booting inside of a PVH
>>>>> container up to the point where cpuid (or whatever method) can be used
>>>>> to detect the environment.
>>>>>
>>>>> Do you recall what's preventing the native entry point from booting
>>>>> inside of a PVH container? If certain emulated devices not present are
>>>>> needed at early boot we could look into either replacing them with
>>>>> other options available inside of a PVH container, or as a last resort
>>>>> making them available on a PVH container.
>>>> Very much IIRC one of the reasons was the fact that zeropage
>>>> (bootparams) needed to be properly formatted. And the hypercall page
>>>> needs to be set up.
>>> But in this case bootparams is going to be setup by grub, so it should
>>> be fine (just like it's done on bare metal).
>> Yes, I think so.
>>
>>> Couldn't the hypercall page be setup at some point during early boot?
>>> Not sure if setting it up at the same point HVM does would be fine for
>>> PVH?
>> Probably --- I think the only reason we set it early is because we need
>> to call XENMEM_memory_map to set bootparams. One other thing I noticed
>> is that we need to set acpi_irq_model before hypervisor is discovered
>> (can't remember why) but I suppose this can be worked around.
>>
>> Having said that --- since for direct boot we still need to go via
>> PVH-specific entry point I am not sure how much we will gain by having
>> grub avoid it.
> Being able to boot inside of a PVH container using the native entry
> point would prevent us from having to add PVH loader specific code to
> each firmware we plan to support in PVH mode.
>
> If Linux must be started using the PVH entry point in order to run
> inside of a PVH container, it means we would have to add a PVH loader
> to OVFM and grub at least.
>
> OTOH if Linux is capable of booting from the native entry point inside
> of a PVH container, we would only have to port OVMF and grub in order
> to work inside of a PVH container, leaving the rest of the logic
> untouched.

Right, I understand that. I was simply trying to say that PVH-specific 
entry point is likely to stay for us to continue booting PVH guests 
directly.

-boris

-boris

Re: Xen PVH support in grub2

[Juergen Gross]

On 29/09/17 17:51, Roger Pau Monné wrote:
> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>> Then, I also wonder whether it would make sense for this grub to load
>>> the kernel using the PVH entry point or the native entry point. Would
>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>> be used inside of a PVH container?
>>
>> I don't think today's Linux allows that. This has been discussed
>> very thoroughly at the time Boris added PVH V2 support to the kernel.
> 
> OK, I'm not going to insist on that, but my plans for FreeBSD is to
> make the native entry point capable of booting inside of a PVH
> container up to the point where cpuid (or whatever method) can be used
> to detect the environment.

Looking more thoroughly into the Linux boot code I think this could
work for Linux, too. But only if we can tell PVH from HVM in the guest.
How would you do that in FreeBSD? Via flags in the boot params? This
would the have to be done in the boot loader (e.g. grub or OVMF).


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Juergen Gross]

On 29/09/17 17:51, Roger Pau Monné wrote:
> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>> Then, I also wonder whether it would make sense for this grub to load
>>> the kernel using the PVH entry point or the native entry point. Would
>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>> be used inside of a PVH container?
>>
>> I don't think today's Linux allows that. This has been discussed
>> very thoroughly at the time Boris added PVH V2 support to the kernel.
> 
> OK, I'm not going to insist on that, but my plans for FreeBSD is to
> make the native entry point capable of booting inside of a PVH
> container up to the point where cpuid (or whatever method) can be used
> to detect the environment.

Looking more thoroughly into the Linux boot code I think this could
work for Linux, too. But only if we can tell PVH from HVM in the guest.
How would you do that in FreeBSD? Via flags in the boot params? This
would the have to be done in the boot loader (e.g. grub or OVMF).


Juergen

Re: Xen PVH support in grub2

[Roger Pau Monné]

On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
> On 29/09/17 17:51, Roger Pau Monné wrote:
> > On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
> >> On 29/09/17 17:24, Roger Pau Monné wrote:
> >>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
> >>> Then, I also wonder whether it would make sense for this grub to load
> >>> the kernel using the PVH entry point or the native entry point. Would
> >>> it be possible to boot a Linux kernel up to the point where cpuid can
> >>> be used inside of a PVH container?
> >>
> >> I don't think today's Linux allows that. This has been discussed
> >> very thoroughly at the time Boris added PVH V2 support to the kernel.
> > 
> > OK, I'm not going to insist on that, but my plans for FreeBSD is to
> > make the native entry point capable of booting inside of a PVH
> > container up to the point where cpuid (or whatever method) can be used
> > to detect the environment.
> 
> Looking more thoroughly into the Linux boot code I think this could
> work for Linux, too. But only if we can tell PVH from HVM in the guest.
> How would you do that in FreeBSD? Via flags in the boot params? This
> would the have to be done in the boot loader (e.g. grub or OVMF).

My plan was not to differentiate between HVM and PVH, but rather to
make use of the ACPI information in order to decide which devices are
available and which are not inside of a PVH guest.

For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
we already set "VGA Not Present" and "CMOS RTC Not Present". There
might be other flags/fields that must be set, but I would like to
avoid having a CPUID bit or similar saying "PVH", because then Xen
will be tied to always providing the same set of devices in PVH
containers.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Roger Pau Monné]

On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
> On 29/09/17 17:51, Roger Pau Monné wrote:
> > On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
> >> On 29/09/17 17:24, Roger Pau Monné wrote:
> >>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
> >>> Then, I also wonder whether it would make sense for this grub to load
> >>> the kernel using the PVH entry point or the native entry point. Would
> >>> it be possible to boot a Linux kernel up to the point where cpuid can
> >>> be used inside of a PVH container?
> >>
> >> I don't think today's Linux allows that. This has been discussed
> >> very thoroughly at the time Boris added PVH V2 support to the kernel.
> > 
> > OK, I'm not going to insist on that, but my plans for FreeBSD is to
> > make the native entry point capable of booting inside of a PVH
> > container up to the point where cpuid (or whatever method) can be used
> > to detect the environment.
> 
> Looking more thoroughly into the Linux boot code I think this could
> work for Linux, too. But only if we can tell PVH from HVM in the guest.
> How would you do that in FreeBSD? Via flags in the boot params? This
> would the have to be done in the boot loader (e.g. grub or OVMF).

My plan was not to differentiate between HVM and PVH, but rather to
make use of the ACPI information in order to decide which devices are
available and which are not inside of a PVH guest.

For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
we already set "VGA Not Present" and "CMOS RTC Not Present". There
might be other flags/fields that must be set, but I would like to
avoid having a CPUID bit or similar saying "PVH", because then Xen
will be tied to always providing the same set of devices in PVH
containers.

Roger.

Re: Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 13:17, Roger Pau Monné wrote:
> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>> be used inside of a PVH container?
>>>>
>>>> I don't think today's Linux allows that. This has been discussed
>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>
>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>> make the native entry point capable of booting inside of a PVH
>>> container up to the point where cpuid (or whatever method) can be used
>>> to detect the environment.
>>
>> Looking more thoroughly into the Linux boot code I think this could
>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>> How would you do that in FreeBSD? Via flags in the boot params? This
>> would the have to be done in the boot loader (e.g. grub or OVMF).
> 
> My plan was not to differentiate between HVM and PVH, but rather to
> make use of the ACPI information in order to decide which devices are
> available and which are not inside of a PVH guest.
> 
> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
> we already set "VGA Not Present" and "CMOS RTC Not Present". There
> might be other flags/fields that must be set, but I would like to
> avoid having a CPUID bit or similar saying "PVH", because then Xen
> will be tied to always providing the same set of devices in PVH
> containers.

Why? This would depend on the semantics tied to the flag. It could just
mean "don't assume availability of legacy stuff" (e.g. BIOS calls).

Linux would have a problem with the ACPI approach as it would try BIOS
calls way before it is initializing its ACPI handling. So in Linux I'd
need another way to tell I'm running in PVH mode, e.g. a "no legacy"
bit in the Xen HVM cpuid leaf.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 13:17, Roger Pau Monné wrote:
> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>> be used inside of a PVH container?
>>>>
>>>> I don't think today's Linux allows that. This has been discussed
>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>
>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>> make the native entry point capable of booting inside of a PVH
>>> container up to the point where cpuid (or whatever method) can be used
>>> to detect the environment.
>>
>> Looking more thoroughly into the Linux boot code I think this could
>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>> How would you do that in FreeBSD? Via flags in the boot params? This
>> would the have to be done in the boot loader (e.g. grub or OVMF).
> 
> My plan was not to differentiate between HVM and PVH, but rather to
> make use of the ACPI information in order to decide which devices are
> available and which are not inside of a PVH guest.
> 
> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
> we already set "VGA Not Present" and "CMOS RTC Not Present". There
> might be other flags/fields that must be set, but I would like to
> avoid having a CPUID bit or similar saying "PVH", because then Xen
> will be tied to always providing the same set of devices in PVH
> containers.

Why? This would depend on the semantics tied to the flag. It could just
mean "don't assume availability of legacy stuff" (e.g. BIOS calls).

Linux would have a problem with the ACPI approach as it would try BIOS
calls way before it is initializing its ACPI handling. So in Linux I'd
need another way to tell I'm running in PVH mode, e.g. a "no legacy"
bit in the Xen HVM cpuid leaf.


Juergen

Re: Xen PVH support in grub2

[Roger Pau Monné]

On Fri, Nov 03, 2017 at 01:50:11PM +0100, Juergen Gross wrote:
> On 03/11/17 13:17, Roger Pau Monné wrote:
> > On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
> >> On 29/09/17 17:51, Roger Pau Monné wrote:
> >>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
> >>>> On 29/09/17 17:24, Roger Pau Monné wrote:
> >>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
> >>>>> Then, I also wonder whether it would make sense for this grub to load
> >>>>> the kernel using the PVH entry point or the native entry point. Would
> >>>>> it be possible to boot a Linux kernel up to the point where cpuid can
> >>>>> be used inside of a PVH container?
> >>>>
> >>>> I don't think today's Linux allows that. This has been discussed
> >>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
> >>>
> >>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
> >>> make the native entry point capable of booting inside of a PVH
> >>> container up to the point where cpuid (or whatever method) can be used
> >>> to detect the environment.
> >>
> >> Looking more thoroughly into the Linux boot code I think this could
> >> work for Linux, too. But only if we can tell PVH from HVM in the guest.
> >> How would you do that in FreeBSD? Via flags in the boot params? This
> >> would the have to be done in the boot loader (e.g. grub or OVMF).
> > 
> > My plan was not to differentiate between HVM and PVH, but rather to
> > make use of the ACPI information in order to decide which devices are
> > available and which are not inside of a PVH guest.
> > 
> > For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
> > we already set "VGA Not Present" and "CMOS RTC Not Present". There
> > might be other flags/fields that must be set, but I would like to
> > avoid having a CPUID bit or similar saying "PVH", because then Xen
> > will be tied to always providing the same set of devices in PVH
> > containers.
> 
> Why? This would depend on the semantics tied to the flag. It could just
> mean "don't assume availability of legacy stuff" (e.g. BIOS calls).
> 
> Linux would have a problem with the ACPI approach as it would try BIOS
> calls way before it is initializing its ACPI handling. So in Linux I'd
> need another way to tell I'm running in PVH mode, e.g. a "no legacy"
> bit in the Xen HVM cpuid leaf.

If you are booted from the PVH entry point, there's no BIOS or UEFI
(ie: no firmware), if you are booted from the BIOS entry point there's
a BIOS and the same applies to UEFI. How does Linux differentiate
whether it's booted from BIOS or UEFI?

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Roger Pau Monné]

On Fri, Nov 03, 2017 at 01:50:11PM +0100, Juergen Gross wrote:
> On 03/11/17 13:17, Roger Pau Monné wrote:
> > On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
> >> On 29/09/17 17:51, Roger Pau Monné wrote:
> >>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
> >>>> On 29/09/17 17:24, Roger Pau Monné wrote:
> >>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
> >>>>> Then, I also wonder whether it would make sense for this grub to load
> >>>>> the kernel using the PVH entry point or the native entry point. Would
> >>>>> it be possible to boot a Linux kernel up to the point where cpuid can
> >>>>> be used inside of a PVH container?
> >>>>
> >>>> I don't think today's Linux allows that. This has been discussed
> >>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
> >>>
> >>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
> >>> make the native entry point capable of booting inside of a PVH
> >>> container up to the point where cpuid (or whatever method) can be used
> >>> to detect the environment.
> >>
> >> Looking more thoroughly into the Linux boot code I think this could
> >> work for Linux, too. But only if we can tell PVH from HVM in the guest.
> >> How would you do that in FreeBSD? Via flags in the boot params? This
> >> would the have to be done in the boot loader (e.g. grub or OVMF).
> > 
> > My plan was not to differentiate between HVM and PVH, but rather to
> > make use of the ACPI information in order to decide which devices are
> > available and which are not inside of a PVH guest.
> > 
> > For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
> > we already set "VGA Not Present" and "CMOS RTC Not Present". There
> > might be other flags/fields that must be set, but I would like to
> > avoid having a CPUID bit or similar saying "PVH", because then Xen
> > will be tied to always providing the same set of devices in PVH
> > containers.
> 
> Why? This would depend on the semantics tied to the flag. It could just
> mean "don't assume availability of legacy stuff" (e.g. BIOS calls).
> 
> Linux would have a problem with the ACPI approach as it would try BIOS
> calls way before it is initializing its ACPI handling. So in Linux I'd
> need another way to tell I'm running in PVH mode, e.g. a "no legacy"
> bit in the Xen HVM cpuid leaf.

If you are booted from the PVH entry point, there's no BIOS or UEFI
(ie: no firmware), if you are booted from the BIOS entry point there's
a BIOS and the same applies to UEFI. How does Linux differentiate
whether it's booted from BIOS or UEFI?

Roger.

Re: Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 15:07, Roger Pau Monné wrote:
> On Fri, Nov 03, 2017 at 01:50:11PM +0100, Juergen Gross wrote:
>> On 03/11/17 13:17, Roger Pau Monné wrote:
>>> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>>>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>>> be used inside of a PVH container?
>>>>>>
>>>>>> I don't think today's Linux allows that. This has been discussed
>>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>>
>>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>>> make the native entry point capable of booting inside of a PVH
>>>>> container up to the point where cpuid (or whatever method) can be used
>>>>> to detect the environment.
>>>>
>>>> Looking more thoroughly into the Linux boot code I think this could
>>>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>>>> How would you do that in FreeBSD? Via flags in the boot params? This
>>>> would the have to be done in the boot loader (e.g. grub or OVMF).
>>>
>>> My plan was not to differentiate between HVM and PVH, but rather to
>>> make use of the ACPI information in order to decide which devices are
>>> available and which are not inside of a PVH guest.
>>>
>>> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
>>> we already set "VGA Not Present" and "CMOS RTC Not Present". There
>>> might be other flags/fields that must be set, but I would like to
>>> avoid having a CPUID bit or similar saying "PVH", because then Xen
>>> will be tied to always providing the same set of devices in PVH
>>> containers.
>>
>> Why? This would depend on the semantics tied to the flag. It could just
>> mean "don't assume availability of legacy stuff" (e.g. BIOS calls).
>>
>> Linux would have a problem with the ACPI approach as it would try BIOS
>> calls way before it is initializing its ACPI handling. So in Linux I'd
>> need another way to tell I'm running in PVH mode, e.g. a "no legacy"
>> bit in the Xen HVM cpuid leaf.
> 
> If you are booted from the PVH entry point, there's no BIOS or UEFI
> (ie: no firmware), if you are booted from the BIOS entry point there's
> a BIOS and the same applies to UEFI. How does Linux differentiate
> whether it's booted from BIOS or UEFI?

They use different entries.


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 15:07, Roger Pau Monné wrote:
> On Fri, Nov 03, 2017 at 01:50:11PM +0100, Juergen Gross wrote:
>> On 03/11/17 13:17, Roger Pau Monné wrote:
>>> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>>>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>>> be used inside of a PVH container?
>>>>>>
>>>>>> I don't think today's Linux allows that. This has been discussed
>>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>>
>>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>>> make the native entry point capable of booting inside of a PVH
>>>>> container up to the point where cpuid (or whatever method) can be used
>>>>> to detect the environment.
>>>>
>>>> Looking more thoroughly into the Linux boot code I think this could
>>>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>>>> How would you do that in FreeBSD? Via flags in the boot params? This
>>>> would the have to be done in the boot loader (e.g. grub or OVMF).
>>>
>>> My plan was not to differentiate between HVM and PVH, but rather to
>>> make use of the ACPI information in order to decide which devices are
>>> available and which are not inside of a PVH guest.
>>>
>>> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
>>> we already set "VGA Not Present" and "CMOS RTC Not Present". There
>>> might be other flags/fields that must be set, but I would like to
>>> avoid having a CPUID bit or similar saying "PVH", because then Xen
>>> will be tied to always providing the same set of devices in PVH
>>> containers.
>>
>> Why? This would depend on the semantics tied to the flag. It could just
>> mean "don't assume availability of legacy stuff" (e.g. BIOS calls).
>>
>> Linux would have a problem with the ACPI approach as it would try BIOS
>> calls way before it is initializing its ACPI handling. So in Linux I'd
>> need another way to tell I'm running in PVH mode, e.g. a "no legacy"
>> bit in the Xen HVM cpuid leaf.
> 
> If you are booted from the PVH entry point, there's no BIOS or UEFI
> (ie: no firmware), if you are booted from the BIOS entry point there's
> a BIOS and the same applies to UEFI. How does Linux differentiate
> whether it's booted from BIOS or UEFI?

They use different entries.


Juergen

Re: Xen PVH support in grub2

[Boris Ostrovsky]

On 11/03/2017 10:24 AM, Juergen Gross wrote:
> On 03/11/17 15:07, Roger Pau Monné wrote:
>> On Fri, Nov 03, 2017 at 01:50:11PM +0100, Juergen Gross wrote:
>>> On 03/11/17 13:17, Roger Pau Monné wrote:
>>>> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>>>>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>>>> be used inside of a PVH container?
>>>>>>> I don't think today's Linux allows that. This has been discussed
>>>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>>>> make the native entry point capable of booting inside of a PVH
>>>>>> container up to the point where cpuid (or whatever method) can be used
>>>>>> to detect the environment.
>>>>> Looking more thoroughly into the Linux boot code I think this could
>>>>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>>>>> How would you do that in FreeBSD? Via flags in the boot params? This
>>>>> would the have to be done in the boot loader (e.g. grub or OVMF).
>>>> My plan was not to differentiate between HVM and PVH, but rather to
>>>> make use of the ACPI information in order to decide which devices are
>>>> available and which are not inside of a PVH guest.
>>>>
>>>> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
>>>> we already set "VGA Not Present" and "CMOS RTC Not Present". There
>>>> might be other flags/fields that must be set, but I would like to
>>>> avoid having a CPUID bit or similar saying "PVH", because then Xen
>>>> will be tied to always providing the same set of devices in PVH
>>>> containers.
>>> Why? This would depend on the semantics tied to the flag. It could just
>>> mean "don't assume availability of legacy stuff" (e.g. BIOS calls).
>>>
>>> Linux would have a problem with the ACPI approach as it would try BIOS
>>> calls way before it is initializing its ACPI handling. So in Linux I'd
>>> need another way to tell I'm running in PVH mode, e.g. a "no legacy"
>>> bit in the Xen HVM cpuid leaf.
>> If you are booted from the PVH entry point, there's no BIOS or UEFI
>> (ie: no firmware), if you are booted from the BIOS entry point there's
>> a BIOS and the same applies to UEFI. How does Linux differentiate
>> whether it's booted from BIOS or UEFI?
> They use different entries.

In fact, we had a discussion with Matt Fleming (Linux EFI maintainer) to
see if we can use EFI entry point to also be able to boot PVH guest but
found some issues with that approach, which is why we ended up with a
dedicated PVH entry point.

I am curious though, Juergen --- what do we need besides zeropage to
allow us to boot PVH from startup_64?



-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Boris Ostrovsky]

On 11/03/2017 10:24 AM, Juergen Gross wrote:
> On 03/11/17 15:07, Roger Pau Monné wrote:
>> On Fri, Nov 03, 2017 at 01:50:11PM +0100, Juergen Gross wrote:
>>> On 03/11/17 13:17, Roger Pau Monné wrote:
>>>> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>>>>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>>>> be used inside of a PVH container?
>>>>>>> I don't think today's Linux allows that. This has been discussed
>>>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>>>> make the native entry point capable of booting inside of a PVH
>>>>>> container up to the point where cpuid (or whatever method) can be used
>>>>>> to detect the environment.
>>>>> Looking more thoroughly into the Linux boot code I think this could
>>>>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>>>>> How would you do that in FreeBSD? Via flags in the boot params? This
>>>>> would the have to be done in the boot loader (e.g. grub or OVMF).
>>>> My plan was not to differentiate between HVM and PVH, but rather to
>>>> make use of the ACPI information in order to decide which devices are
>>>> available and which are not inside of a PVH guest.
>>>>
>>>> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
>>>> we already set "VGA Not Present" and "CMOS RTC Not Present". There
>>>> might be other flags/fields that must be set, but I would like to
>>>> avoid having a CPUID bit or similar saying "PVH", because then Xen
>>>> will be tied to always providing the same set of devices in PVH
>>>> containers.
>>> Why? This would depend on the semantics tied to the flag. It could just
>>> mean "don't assume availability of legacy stuff" (e.g. BIOS calls).
>>>
>>> Linux would have a problem with the ACPI approach as it would try BIOS
>>> calls way before it is initializing its ACPI handling. So in Linux I'd
>>> need another way to tell I'm running in PVH mode, e.g. a "no legacy"
>>> bit in the Xen HVM cpuid leaf.
>> If you are booted from the PVH entry point, there's no BIOS or UEFI
>> (ie: no firmware), if you are booted from the BIOS entry point there's
>> a BIOS and the same applies to UEFI. How does Linux differentiate
>> whether it's booted from BIOS or UEFI?
> They use different entries.

In fact, we had a discussion with Matt Fleming (Linux EFI maintainer) to
see if we can use EFI entry point to also be able to boot PVH guest but
found some issues with that approach, which is why we ended up with a
dedicated PVH entry point.

I am curious though, Juergen --- what do we need besides zeropage to
allow us to boot PVH from startup_64?



-boris

Re: Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 15:36, Boris Ostrovsky wrote:
> On 11/03/2017 10:24 AM, Juergen Gross wrote:
>> On 03/11/17 15:07, Roger Pau Monné wrote:
>>> On Fri, Nov 03, 2017 at 01:50:11PM +0100, Juergen Gross wrote:
>>>> On 03/11/17 13:17, Roger Pau Monné wrote:
>>>>> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>>>>>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>>>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>>>>> be used inside of a PVH container?
>>>>>>>> I don't think today's Linux allows that. This has been discussed
>>>>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>>>>> make the native entry point capable of booting inside of a PVH
>>>>>>> container up to the point where cpuid (or whatever method) can be used
>>>>>>> to detect the environment.
>>>>>> Looking more thoroughly into the Linux boot code I think this could
>>>>>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>>>>>> How would you do that in FreeBSD? Via flags in the boot params? This
>>>>>> would the have to be done in the boot loader (e.g. grub or OVMF).
>>>>> My plan was not to differentiate between HVM and PVH, but rather to
>>>>> make use of the ACPI information in order to decide which devices are
>>>>> available and which are not inside of a PVH guest.
>>>>>
>>>>> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
>>>>> we already set "VGA Not Present" and "CMOS RTC Not Present". There
>>>>> might be other flags/fields that must be set, but I would like to
>>>>> avoid having a CPUID bit or similar saying "PVH", because then Xen
>>>>> will be tied to always providing the same set of devices in PVH
>>>>> containers.
>>>> Why? This would depend on the semantics tied to the flag. It could just
>>>> mean "don't assume availability of legacy stuff" (e.g. BIOS calls).
>>>>
>>>> Linux would have a problem with the ACPI approach as it would try BIOS
>>>> calls way before it is initializing its ACPI handling. So in Linux I'd
>>>> need another way to tell I'm running in PVH mode, e.g. a "no legacy"
>>>> bit in the Xen HVM cpuid leaf.
>>> If you are booted from the PVH entry point, there's no BIOS or UEFI
>>> (ie: no firmware), if you are booted from the BIOS entry point there's
>>> a BIOS and the same applies to UEFI. How does Linux differentiate
>>> whether it's booted from BIOS or UEFI?
>> They use different entries.
> 
> In fact, we had a discussion with Matt Fleming (Linux EFI maintainer) to
> see if we can use EFI entry point to also be able to boot PVH guest but
> found some issues with that approach, which is why we ended up with a
> dedicated PVH entry point.
> 
> I am curious though, Juergen --- what do we need besides zeropage to
> allow us to boot PVH from startup_64?

Oh, you are right. I managed to get lost in the early boot paths.

Only setting up the hyperpage seems to be missing, but this should be
doable. And setting xen_pvh, of course.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 15:36, Boris Ostrovsky wrote:
> On 11/03/2017 10:24 AM, Juergen Gross wrote:
>> On 03/11/17 15:07, Roger Pau Monné wrote:
>>> On Fri, Nov 03, 2017 at 01:50:11PM +0100, Juergen Gross wrote:
>>>> On 03/11/17 13:17, Roger Pau Monné wrote:
>>>>> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>>>>>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>>>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>>>>> be used inside of a PVH container?
>>>>>>>> I don't think today's Linux allows that. This has been discussed
>>>>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>>>>> make the native entry point capable of booting inside of a PVH
>>>>>>> container up to the point where cpuid (or whatever method) can be used
>>>>>>> to detect the environment.
>>>>>> Looking more thoroughly into the Linux boot code I think this could
>>>>>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>>>>>> How would you do that in FreeBSD? Via flags in the boot params? This
>>>>>> would the have to be done in the boot loader (e.g. grub or OVMF).
>>>>> My plan was not to differentiate between HVM and PVH, but rather to
>>>>> make use of the ACPI information in order to decide which devices are
>>>>> available and which are not inside of a PVH guest.
>>>>>
>>>>> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
>>>>> we already set "VGA Not Present" and "CMOS RTC Not Present". There
>>>>> might be other flags/fields that must be set, but I would like to
>>>>> avoid having a CPUID bit or similar saying "PVH", because then Xen
>>>>> will be tied to always providing the same set of devices in PVH
>>>>> containers.
>>>> Why? This would depend on the semantics tied to the flag. It could just
>>>> mean "don't assume availability of legacy stuff" (e.g. BIOS calls).
>>>>
>>>> Linux would have a problem with the ACPI approach as it would try BIOS
>>>> calls way before it is initializing its ACPI handling. So in Linux I'd
>>>> need another way to tell I'm running in PVH mode, e.g. a "no legacy"
>>>> bit in the Xen HVM cpuid leaf.
>>> If you are booted from the PVH entry point, there's no BIOS or UEFI
>>> (ie: no firmware), if you are booted from the BIOS entry point there's
>>> a BIOS and the same applies to UEFI. How does Linux differentiate
>>> whether it's booted from BIOS or UEFI?
>> They use different entries.
> 
> In fact, we had a discussion with Matt Fleming (Linux EFI maintainer) to
> see if we can use EFI entry point to also be able to boot PVH guest but
> found some issues with that approach, which is why we ended up with a
> dedicated PVH entry point.
> 
> I am curious though, Juergen --- what do we need besides zeropage to
> allow us to boot PVH from startup_64?

Oh, you are right. I managed to get lost in the early boot paths.

Only setting up the hyperpage seems to be missing, but this should be
doable. And setting xen_pvh, of course.


Juergen

Re: Xen PVH support in grub2

[Boris Ostrovsky]

On 11/03/2017 10:59 AM, Juergen Gross wrote:
> On 03/11/17 15:36, Boris Ostrovsky wrote:
>> On 11/03/2017 10:24 AM, Juergen Gross wrote:
>>> On 03/11/17 15:07, Roger Pau Monné wrote:
>>>> On Fri, Nov 03, 2017 at 01:50:11PM +0100, Juergen Gross wrote:
>>>>> On 03/11/17 13:17, Roger Pau Monné wrote:
>>>>>> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>>>>>>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>>>>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>>>>>> be used inside of a PVH container?
>>>>>>>>> I don't think today's Linux allows that. This has been discussed
>>>>>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>>>>>> make the native entry point capable of booting inside of a PVH
>>>>>>>> container up to the point where cpuid (or whatever method) can be used
>>>>>>>> to detect the environment.
>>>>>>> Looking more thoroughly into the Linux boot code I think this could
>>>>>>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>>>>>>> How would you do that in FreeBSD? Via flags in the boot params? This
>>>>>>> would the have to be done in the boot loader (e.g. grub or OVMF).
>>>>>> My plan was not to differentiate between HVM and PVH, but rather to
>>>>>> make use of the ACPI information in order to decide which devices are
>>>>>> available and which are not inside of a PVH guest.
>>>>>>
>>>>>> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
>>>>>> we already set "VGA Not Present" and "CMOS RTC Not Present". There
>>>>>> might be other flags/fields that must be set, but I would like to
>>>>>> avoid having a CPUID bit or similar saying "PVH", because then Xen
>>>>>> will be tied to always providing the same set of devices in PVH
>>>>>> containers.
>>>>> Why? This would depend on the semantics tied to the flag. It could just
>>>>> mean "don't assume availability of legacy stuff" (e.g. BIOS calls).
>>>>>
>>>>> Linux would have a problem with the ACPI approach as it would try BIOS
>>>>> calls way before it is initializing its ACPI handling. So in Linux I'd
>>>>> need another way to tell I'm running in PVH mode, e.g. a "no legacy"
>>>>> bit in the Xen HVM cpuid leaf.
>>>> If you are booted from the PVH entry point, there's no BIOS or UEFI
>>>> (ie: no firmware), if you are booted from the BIOS entry point there's
>>>> a BIOS and the same applies to UEFI. How does Linux differentiate
>>>> whether it's booted from BIOS or UEFI?
>>> They use different entries.
>> In fact, we had a discussion with Matt Fleming (Linux EFI maintainer) to
>> see if we can use EFI entry point to also be able to boot PVH guest but
>> found some issues with that approach, which is why we ended up with a
>> dedicated PVH entry point.
>>
>> I am curious though, Juergen --- what do we need besides zeropage to
>> allow us to boot PVH from startup_64?
> Oh, you are right. I managed to get lost in the early boot paths.
>
> Only setting up the hyperpage seems to be missing, but this should be
> doable. And setting xen_pvh, of course.

That last part was actually my question --- do we need to have xen_pvh
set before we get to xen-specific code for the first time (which I think
is init_hypervisor_platform()) from startup_64?

Because if we do --- who will set it?


-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Boris Ostrovsky]

On 11/03/2017 10:59 AM, Juergen Gross wrote:
> On 03/11/17 15:36, Boris Ostrovsky wrote:
>> On 11/03/2017 10:24 AM, Juergen Gross wrote:
>>> On 03/11/17 15:07, Roger Pau Monné wrote:
>>>> On Fri, Nov 03, 2017 at 01:50:11PM +0100, Juergen Gross wrote:
>>>>> On 03/11/17 13:17, Roger Pau Monné wrote:
>>>>>> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>>>>>>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>>>>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>>>>>> be used inside of a PVH container?
>>>>>>>>> I don't think today's Linux allows that. This has been discussed
>>>>>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>>>>>> make the native entry point capable of booting inside of a PVH
>>>>>>>> container up to the point where cpuid (or whatever method) can be used
>>>>>>>> to detect the environment.
>>>>>>> Looking more thoroughly into the Linux boot code I think this could
>>>>>>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>>>>>>> How would you do that in FreeBSD? Via flags in the boot params? This
>>>>>>> would the have to be done in the boot loader (e.g. grub or OVMF).
>>>>>> My plan was not to differentiate between HVM and PVH, but rather to
>>>>>> make use of the ACPI information in order to decide which devices are
>>>>>> available and which are not inside of a PVH guest.
>>>>>>
>>>>>> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
>>>>>> we already set "VGA Not Present" and "CMOS RTC Not Present". There
>>>>>> might be other flags/fields that must be set, but I would like to
>>>>>> avoid having a CPUID bit or similar saying "PVH", because then Xen
>>>>>> will be tied to always providing the same set of devices in PVH
>>>>>> containers.
>>>>> Why? This would depend on the semantics tied to the flag. It could just
>>>>> mean "don't assume availability of legacy stuff" (e.g. BIOS calls).
>>>>>
>>>>> Linux would have a problem with the ACPI approach as it would try BIOS
>>>>> calls way before it is initializing its ACPI handling. So in Linux I'd
>>>>> need another way to tell I'm running in PVH mode, e.g. a "no legacy"
>>>>> bit in the Xen HVM cpuid leaf.
>>>> If you are booted from the PVH entry point, there's no BIOS or UEFI
>>>> (ie: no firmware), if you are booted from the BIOS entry point there's
>>>> a BIOS and the same applies to UEFI. How does Linux differentiate
>>>> whether it's booted from BIOS or UEFI?
>>> They use different entries.
>> In fact, we had a discussion with Matt Fleming (Linux EFI maintainer) to
>> see if we can use EFI entry point to also be able to boot PVH guest but
>> found some issues with that approach, which is why we ended up with a
>> dedicated PVH entry point.
>>
>> I am curious though, Juergen --- what do we need besides zeropage to
>> allow us to boot PVH from startup_64?
> Oh, you are right. I managed to get lost in the early boot paths.
>
> Only setting up the hyperpage seems to be missing, but this should be
> doable. And setting xen_pvh, of course.

That last part was actually my question --- do we need to have xen_pvh
set before we get to xen-specific code for the first time (which I think
is init_hypervisor_platform()) from startup_64?

Because if we do --- who will set it?


-boris

Re: Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 16:10, Boris Ostrovsky wrote:
> On 11/03/2017 10:59 AM, Juergen Gross wrote:
>> On 03/11/17 15:36, Boris Ostrovsky wrote:
>>> On 11/03/2017 10:24 AM, Juergen Gross wrote:
>>>> On 03/11/17 15:07, Roger Pau Monné wrote:
>>>>> On Fri, Nov 03, 2017 at 01:50:11PM +0100, Juergen Gross wrote:
>>>>>> On 03/11/17 13:17, Roger Pau Monné wrote:
>>>>>>> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>>>>>>>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>>>>>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>>>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>>>>>>> be used inside of a PVH container?
>>>>>>>>>> I don't think today's Linux allows that. This has been discussed
>>>>>>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>>>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>>>>>>> make the native entry point capable of booting inside of a PVH
>>>>>>>>> container up to the point where cpuid (or whatever method) can be used
>>>>>>>>> to detect the environment.
>>>>>>>> Looking more thoroughly into the Linux boot code I think this could
>>>>>>>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>>>>>>>> How would you do that in FreeBSD? Via flags in the boot params? This
>>>>>>>> would the have to be done in the boot loader (e.g. grub or OVMF).
>>>>>>> My plan was not to differentiate between HVM and PVH, but rather to
>>>>>>> make use of the ACPI information in order to decide which devices are
>>>>>>> available and which are not inside of a PVH guest.
>>>>>>>
>>>>>>> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
>>>>>>> we already set "VGA Not Present" and "CMOS RTC Not Present". There
>>>>>>> might be other flags/fields that must be set, but I would like to
>>>>>>> avoid having a CPUID bit or similar saying "PVH", because then Xen
>>>>>>> will be tied to always providing the same set of devices in PVH
>>>>>>> containers.
>>>>>> Why? This would depend on the semantics tied to the flag. It could just
>>>>>> mean "don't assume availability of legacy stuff" (e.g. BIOS calls).
>>>>>>
>>>>>> Linux would have a problem with the ACPI approach as it would try BIOS
>>>>>> calls way before it is initializing its ACPI handling. So in Linux I'd
>>>>>> need another way to tell I'm running in PVH mode, e.g. a "no legacy"
>>>>>> bit in the Xen HVM cpuid leaf.
>>>>> If you are booted from the PVH entry point, there's no BIOS or UEFI
>>>>> (ie: no firmware), if you are booted from the BIOS entry point there's
>>>>> a BIOS and the same applies to UEFI. How does Linux differentiate
>>>>> whether it's booted from BIOS or UEFI?
>>>> They use different entries.
>>> In fact, we had a discussion with Matt Fleming (Linux EFI maintainer) to
>>> see if we can use EFI entry point to also be able to boot PVH guest but
>>> found some issues with that approach, which is why we ended up with a
>>> dedicated PVH entry point.
>>>
>>> I am curious though, Juergen --- what do we need besides zeropage to
>>> allow us to boot PVH from startup_64?
>> Oh, you are right. I managed to get lost in the early boot paths.
>>
>> Only setting up the hyperpage seems to be missing, but this should be
>> doable. And setting xen_pvh, of course.
> 
> That last part was actually my question --- do we need to have xen_pvh
> set before we get to xen-specific code for the first time (which I think
> is init_hypervisor_platform()) from startup_64?
> 
> Because if we do --- who will set it?

This should be easily testable: Just set an internal indicator in
xen_prepare_pvh() and copy that to xen_pvh in init_hypervisor_platform()
and see whether the kernel still comes up in PVH mode.

The same can be done with setting up the hypercall page.

Just building a kernel trying that...


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 16:10, Boris Ostrovsky wrote:
> On 11/03/2017 10:59 AM, Juergen Gross wrote:
>> On 03/11/17 15:36, Boris Ostrovsky wrote:
>>> On 11/03/2017 10:24 AM, Juergen Gross wrote:
>>>> On 03/11/17 15:07, Roger Pau Monné wrote:
>>>>> On Fri, Nov 03, 2017 at 01:50:11PM +0100, Juergen Gross wrote:
>>>>>> On 03/11/17 13:17, Roger Pau Monné wrote:
>>>>>>> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>>>>>>>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>>>>>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>>>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>>>>>>> be used inside of a PVH container?
>>>>>>>>>> I don't think today's Linux allows that. This has been discussed
>>>>>>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>>>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>>>>>>> make the native entry point capable of booting inside of a PVH
>>>>>>>>> container up to the point where cpuid (or whatever method) can be used
>>>>>>>>> to detect the environment.
>>>>>>>> Looking more thoroughly into the Linux boot code I think this could
>>>>>>>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>>>>>>>> How would you do that in FreeBSD? Via flags in the boot params? This
>>>>>>>> would the have to be done in the boot loader (e.g. grub or OVMF).
>>>>>>> My plan was not to differentiate between HVM and PVH, but rather to
>>>>>>> make use of the ACPI information in order to decide which devices are
>>>>>>> available and which are not inside of a PVH guest.
>>>>>>>
>>>>>>> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
>>>>>>> we already set "VGA Not Present" and "CMOS RTC Not Present". There
>>>>>>> might be other flags/fields that must be set, but I would like to
>>>>>>> avoid having a CPUID bit or similar saying "PVH", because then Xen
>>>>>>> will be tied to always providing the same set of devices in PVH
>>>>>>> containers.
>>>>>> Why? This would depend on the semantics tied to the flag. It could just
>>>>>> mean "don't assume availability of legacy stuff" (e.g. BIOS calls).
>>>>>>
>>>>>> Linux would have a problem with the ACPI approach as it would try BIOS
>>>>>> calls way before it is initializing its ACPI handling. So in Linux I'd
>>>>>> need another way to tell I'm running in PVH mode, e.g. a "no legacy"
>>>>>> bit in the Xen HVM cpuid leaf.
>>>>> If you are booted from the PVH entry point, there's no BIOS or UEFI
>>>>> (ie: no firmware), if you are booted from the BIOS entry point there's
>>>>> a BIOS and the same applies to UEFI. How does Linux differentiate
>>>>> whether it's booted from BIOS or UEFI?
>>>> They use different entries.
>>> In fact, we had a discussion with Matt Fleming (Linux EFI maintainer) to
>>> see if we can use EFI entry point to also be able to boot PVH guest but
>>> found some issues with that approach, which is why we ended up with a
>>> dedicated PVH entry point.
>>>
>>> I am curious though, Juergen --- what do we need besides zeropage to
>>> allow us to boot PVH from startup_64?
>> Oh, you are right. I managed to get lost in the early boot paths.
>>
>> Only setting up the hyperpage seems to be missing, but this should be
>> doable. And setting xen_pvh, of course.
> 
> That last part was actually my question --- do we need to have xen_pvh
> set before we get to xen-specific code for the first time (which I think
> is init_hypervisor_platform()) from startup_64?
> 
> Because if we do --- who will set it?

This should be easily testable: Just set an internal indicator in
xen_prepare_pvh() and copy that to xen_pvh in init_hypervisor_platform()
and see whether the kernel still comes up in PVH mode.

The same can be done with setting up the hypercall page.

Just building a kernel trying that...


Juergen

Re: Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 16:27, Juergen Gross wrote:
> On 03/11/17 16:10, Boris Ostrovsky wrote:
>> On 11/03/2017 10:59 AM, Juergen Gross wrote:
>>> On 03/11/17 15:36, Boris Ostrovsky wrote:
>>>> On 11/03/2017 10:24 AM, Juergen Gross wrote:
>>>>> On 03/11/17 15:07, Roger Pau Monné wrote:
>>>>>> On Fri, Nov 03, 2017 at 01:50:11PM +0100, Juergen Gross wrote:
>>>>>>> On 03/11/17 13:17, Roger Pau Monné wrote:
>>>>>>>> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>>>>>>>>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>>>>>>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>>>>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>>>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>>>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>>>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>>>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>>>>>>>> be used inside of a PVH container?
>>>>>>>>>>> I don't think today's Linux allows that. This has been discussed
>>>>>>>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>>>>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>>>>>>>> make the native entry point capable of booting inside of a PVH
>>>>>>>>>> container up to the point where cpuid (or whatever method) can be used
>>>>>>>>>> to detect the environment.
>>>>>>>>> Looking more thoroughly into the Linux boot code I think this could
>>>>>>>>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>>>>>>>>> How would you do that in FreeBSD? Via flags in the boot params? This
>>>>>>>>> would the have to be done in the boot loader (e.g. grub or OVMF).
>>>>>>>> My plan was not to differentiate between HVM and PVH, but rather to
>>>>>>>> make use of the ACPI information in order to decide which devices are
>>>>>>>> available and which are not inside of a PVH guest.
>>>>>>>>
>>>>>>>> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
>>>>>>>> we already set "VGA Not Present" and "CMOS RTC Not Present". There
>>>>>>>> might be other flags/fields that must be set, but I would like to
>>>>>>>> avoid having a CPUID bit or similar saying "PVH", because then Xen
>>>>>>>> will be tied to always providing the same set of devices in PVH
>>>>>>>> containers.
>>>>>>> Why? This would depend on the semantics tied to the flag. It could just
>>>>>>> mean "don't assume availability of legacy stuff" (e.g. BIOS calls).
>>>>>>>
>>>>>>> Linux would have a problem with the ACPI approach as it would try BIOS
>>>>>>> calls way before it is initializing its ACPI handling. So in Linux I'd
>>>>>>> need another way to tell I'm running in PVH mode, e.g. a "no legacy"
>>>>>>> bit in the Xen HVM cpuid leaf.
>>>>>> If you are booted from the PVH entry point, there's no BIOS or UEFI
>>>>>> (ie: no firmware), if you are booted from the BIOS entry point there's
>>>>>> a BIOS and the same applies to UEFI. How does Linux differentiate
>>>>>> whether it's booted from BIOS or UEFI?
>>>>> They use different entries.
>>>> In fact, we had a discussion with Matt Fleming (Linux EFI maintainer) to
>>>> see if we can use EFI entry point to also be able to boot PVH guest but
>>>> found some issues with that approach, which is why we ended up with a
>>>> dedicated PVH entry point.
>>>>
>>>> I am curious though, Juergen --- what do we need besides zeropage to
>>>> allow us to boot PVH from startup_64?
>>> Oh, you are right. I managed to get lost in the early boot paths.
>>>
>>> Only setting up the hyperpage seems to be missing, but this should be
>>> doable. And setting xen_pvh, of course.
>>
>> That last part was actually my question --- do we need to have xen_pvh
>> set before we get to xen-specific code for the first time (which I think
>> is init_hypervisor_platform()) from startup_64?
>>
>> Because if we do --- who will set it?
> 
> This should be easily testable: Just set an internal indicator in
> xen_prepare_pvh() and copy that to xen_pvh in init_hypervisor_platform()
> and see whether the kernel still comes up in PVH mode.
> 
> The same can be done with setting up the hypercall page.
> 
> Just building a kernel trying that...

Okay, setting xen_pvh late is working. I can't omit setting up the
hypercall_page right now, of course, as then I wouldn't get the memory
map. But this shouldn't be a problem for the grub/OVMF case as those
would setup the memory map for the guest in zeropage.

So again the question: how to tell whether we are PVH or HVM in
init_hypervisor_platform()? ACPi tables are scanned way later...


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 16:27, Juergen Gross wrote:
> On 03/11/17 16:10, Boris Ostrovsky wrote:
>> On 11/03/2017 10:59 AM, Juergen Gross wrote:
>>> On 03/11/17 15:36, Boris Ostrovsky wrote:
>>>> On 11/03/2017 10:24 AM, Juergen Gross wrote:
>>>>> On 03/11/17 15:07, Roger Pau Monné wrote:
>>>>>> On Fri, Nov 03, 2017 at 01:50:11PM +0100, Juergen Gross wrote:
>>>>>>> On 03/11/17 13:17, Roger Pau Monné wrote:
>>>>>>>> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>>>>>>>>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>>>>>>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>>>>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>>>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>>>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>>>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>>>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>>>>>>>> be used inside of a PVH container?
>>>>>>>>>>> I don't think today's Linux allows that. This has been discussed
>>>>>>>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>>>>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>>>>>>>> make the native entry point capable of booting inside of a PVH
>>>>>>>>>> container up to the point where cpuid (or whatever method) can be used
>>>>>>>>>> to detect the environment.
>>>>>>>>> Looking more thoroughly into the Linux boot code I think this could
>>>>>>>>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>>>>>>>>> How would you do that in FreeBSD? Via flags in the boot params? This
>>>>>>>>> would the have to be done in the boot loader (e.g. grub or OVMF).
>>>>>>>> My plan was not to differentiate between HVM and PVH, but rather to
>>>>>>>> make use of the ACPI information in order to decide which devices are
>>>>>>>> available and which are not inside of a PVH guest.
>>>>>>>>
>>>>>>>> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
>>>>>>>> we already set "VGA Not Present" and "CMOS RTC Not Present". There
>>>>>>>> might be other flags/fields that must be set, but I would like to
>>>>>>>> avoid having a CPUID bit or similar saying "PVH", because then Xen
>>>>>>>> will be tied to always providing the same set of devices in PVH
>>>>>>>> containers.
>>>>>>> Why? This would depend on the semantics tied to the flag. It could just
>>>>>>> mean "don't assume availability of legacy stuff" (e.g. BIOS calls).
>>>>>>>
>>>>>>> Linux would have a problem with the ACPI approach as it would try BIOS
>>>>>>> calls way before it is initializing its ACPI handling. So in Linux I'd
>>>>>>> need another way to tell I'm running in PVH mode, e.g. a "no legacy"
>>>>>>> bit in the Xen HVM cpuid leaf.
>>>>>> If you are booted from the PVH entry point, there's no BIOS or UEFI
>>>>>> (ie: no firmware), if you are booted from the BIOS entry point there's
>>>>>> a BIOS and the same applies to UEFI. How does Linux differentiate
>>>>>> whether it's booted from BIOS or UEFI?
>>>>> They use different entries.
>>>> In fact, we had a discussion with Matt Fleming (Linux EFI maintainer) to
>>>> see if we can use EFI entry point to also be able to boot PVH guest but
>>>> found some issues with that approach, which is why we ended up with a
>>>> dedicated PVH entry point.
>>>>
>>>> I am curious though, Juergen --- what do we need besides zeropage to
>>>> allow us to boot PVH from startup_64?
>>> Oh, you are right. I managed to get lost in the early boot paths.
>>>
>>> Only setting up the hyperpage seems to be missing, but this should be
>>> doable. And setting xen_pvh, of course.
>>
>> That last part was actually my question --- do we need to have xen_pvh
>> set before we get to xen-specific code for the first time (which I think
>> is init_hypervisor_platform()) from startup_64?
>>
>> Because if we do --- who will set it?
> 
> This should be easily testable: Just set an internal indicator in
> xen_prepare_pvh() and copy that to xen_pvh in init_hypervisor_platform()
> and see whether the kernel still comes up in PVH mode.
> 
> The same can be done with setting up the hypercall page.
> 
> Just building a kernel trying that...

Okay, setting xen_pvh late is working. I can't omit setting up the
hypercall_page right now, of course, as then I wouldn't get the memory
map. But this shouldn't be a problem for the grub/OVMF case as those
would setup the memory map for the guest in zeropage.

So again the question: how to tell whether we are PVH or HVM in
init_hypervisor_platform()? ACPi tables are scanned way later...


Juergen

Re: Xen PVH support in grub2

[Boris Ostrovsky]

On 11/03/2017 02:05 PM, Juergen Gross wrote:
>
> So again the question: how to tell whether we are PVH or HVM in
> init_hypervisor_platform()? ACPi tables are scanned way later...

Can we make grub/OVMF append a boot option?

Or set setup_header.hardware_subarch to something? We already have
X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
think).

-boris


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Boris Ostrovsky]

On 11/03/2017 02:05 PM, Juergen Gross wrote:
>
> So again the question: how to tell whether we are PVH or HVM in
> init_hypervisor_platform()? ACPi tables are scanned way later...

Can we make grub/OVMF append a boot option?

Or set setup_header.hardware_subarch to something? We already have
X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
think).

-boris

Re: Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 19:19, Boris Ostrovsky wrote:
> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>
>> So again the question: how to tell whether we are PVH or HVM in
>> init_hypervisor_platform()? ACPi tables are scanned way later...
> 
> Can we make grub/OVMF append a boot option?
> 
> Or set setup_header.hardware_subarch to something? We already have
> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
> think).

But wouldn't this break the idea to reuse the native boot paths in
grub/OVMF without further modifications?

I'd rather have a way to ask the hypervisor whether we are in PVH mode
(e.g. via CPUID or a hypercall to test for a devicemodel having itself
registered).

In case we need to set an indicator in zeropage I'd opt for reusing
X86_SUBARCH_XEN.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 19:19, Boris Ostrovsky wrote:
> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>
>> So again the question: how to tell whether we are PVH or HVM in
>> init_hypervisor_platform()? ACPi tables are scanned way later...
> 
> Can we make grub/OVMF append a boot option?
> 
> Or set setup_header.hardware_subarch to something? We already have
> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
> think).

But wouldn't this break the idea to reuse the native boot paths in
grub/OVMF without further modifications?

I'd rather have a way to ask the hypervisor whether we are in PVH mode
(e.g. via CPUID or a hypercall to test for a devicemodel having itself
registered).

In case we need to set an indicator in zeropage I'd opt for reusing
X86_SUBARCH_XEN.


Juergen

Re: Xen PVH support in grub2

[Boris Ostrovsky]

On 11/03/2017 02:23 PM, Juergen Gross wrote:
> On 03/11/17 19:19, Boris Ostrovsky wrote:
>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>> So again the question: how to tell whether we are PVH or HVM in
>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>> Can we make grub/OVMF append a boot option?
>>
>> Or set setup_header.hardware_subarch to something? We already have
>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>> think).
> But wouldn't this break the idea to reuse the native boot paths in
> grub/OVMF without further modifications?

WDYM? We will have to have some sort of a plugin in either one to build
the zeropage anyway. So we'd set hardware_subarch there, in addition to
other things like setting memory and such.

-boris

>
> I'd rather have a way to ask the hypervisor whether we are in PVH mode
> (e.g. via CPUID or a hypercall to test for a devicemodel having itself
> registered).
>
> In case we need to set an indicator in zeropage I'd opt for reusing
> X86_SUBARCH_XEN.
>
>
> Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Boris Ostrovsky]

On 11/03/2017 02:23 PM, Juergen Gross wrote:
> On 03/11/17 19:19, Boris Ostrovsky wrote:
>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>> So again the question: how to tell whether we are PVH or HVM in
>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>> Can we make grub/OVMF append a boot option?
>>
>> Or set setup_header.hardware_subarch to something? We already have
>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>> think).
> But wouldn't this break the idea to reuse the native boot paths in
> grub/OVMF without further modifications?

WDYM? We will have to have some sort of a plugin in either one to build
the zeropage anyway. So we'd set hardware_subarch there, in addition to
other things like setting memory and such.

-boris

>
> I'd rather have a way to ask the hypervisor whether we are in PVH mode
> (e.g. via CPUID or a hypercall to test for a devicemodel having itself
> registered).
>
> In case we need to set an indicator in zeropage I'd opt for reusing
> X86_SUBARCH_XEN.
>
>
> Juergen

Re: Xen PVH support in grub2

[Roger Pau Monné]

On Fri, Nov 03, 2017 at 07:23:50PM +0100, Juergen Gross wrote:
> On 03/11/17 19:19, Boris Ostrovsky wrote:
> > On 11/03/2017 02:05 PM, Juergen Gross wrote:
> >>
> >> So again the question: how to tell whether we are PVH or HVM in
> >> init_hypervisor_platform()? ACPi tables are scanned way later...
> > 
> > Can we make grub/OVMF append a boot option?
> > 
> > Or set setup_header.hardware_subarch to something? We already have
> > X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
> > hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
> > think).
> 
> But wouldn't this break the idea to reuse the native boot paths in
> grub/OVMF without further modifications?
> 
> I'd rather have a way to ask the hypervisor whether we are in PVH mode
> (e.g. via CPUID or a hypercall to test for a devicemodel having itself
> registered).

Keep in mind that from Xen's PoV PVH is just a HVM guest. Xen
currently keeps a bitmap of the emulated devices that are available to
a guest, but that's so far an internal field. We could consider
exporting this on a cpuid leaf, but then we need to make it a fixed
ABI.

Maybe we can make a list of platform devices that are not available on
PVH and that Linux assumes to be present?

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Roger Pau Monné]

On Fri, Nov 03, 2017 at 07:23:50PM +0100, Juergen Gross wrote:
> On 03/11/17 19:19, Boris Ostrovsky wrote:
> > On 11/03/2017 02:05 PM, Juergen Gross wrote:
> >>
> >> So again the question: how to tell whether we are PVH or HVM in
> >> init_hypervisor_platform()? ACPi tables are scanned way later...
> > 
> > Can we make grub/OVMF append a boot option?
> > 
> > Or set setup_header.hardware_subarch to something? We already have
> > X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
> > hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
> > think).
> 
> But wouldn't this break the idea to reuse the native boot paths in
> grub/OVMF without further modifications?
> 
> I'd rather have a way to ask the hypervisor whether we are in PVH mode
> (e.g. via CPUID or a hypercall to test for a devicemodel having itself
> registered).

Keep in mind that from Xen's PoV PVH is just a HVM guest. Xen
currently keeps a bitmap of the emulated devices that are available to
a guest, but that's so far an internal field. We could consider
exporting this on a cpuid leaf, but then we need to make it a fixed
ABI.

Maybe we can make a list of platform devices that are not available on
PVH and that Linux assumes to be present?

Roger.

Re: Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 19:35, Boris Ostrovsky wrote:
> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>> So again the question: how to tell whether we are PVH or HVM in
>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>> Can we make grub/OVMF append a boot option?
>>>
>>> Or set setup_header.hardware_subarch to something? We already have
>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>> think).
>> But wouldn't this break the idea to reuse the native boot paths in
>> grub/OVMF without further modifications?
> 
> WDYM? We will have to have some sort of a plugin in either one to build
> the zeropage anyway. So we'd set hardware_subarch there, in addition to
> other things like setting memory and such.

But isn't the zeropage already being built? I admit that setting subarch
isn't a big deal, but using another entry with a passed-through pvh
start struct isn't either...


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 19:35, Boris Ostrovsky wrote:
> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>> So again the question: how to tell whether we are PVH or HVM in
>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>> Can we make grub/OVMF append a boot option?
>>>
>>> Or set setup_header.hardware_subarch to something? We already have
>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>> think).
>> But wouldn't this break the idea to reuse the native boot paths in
>> grub/OVMF without further modifications?
> 
> WDYM? We will have to have some sort of a plugin in either one to build
> the zeropage anyway. So we'd set hardware_subarch there, in addition to
> other things like setting memory and such.

But isn't the zeropage already being built? I admit that setting subarch
isn't a big deal, but using another entry with a passed-through pvh
start struct isn't either...


Juergen

Re: Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 19:37, Roger Pau Monné wrote:
> On Fri, Nov 03, 2017 at 07:23:50PM +0100, Juergen Gross wrote:
>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>
>>>> So again the question: how to tell whether we are PVH or HVM in
>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>
>>> Can we make grub/OVMF append a boot option?
>>>
>>> Or set setup_header.hardware_subarch to something? We already have
>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>> think).
>>
>> But wouldn't this break the idea to reuse the native boot paths in
>> grub/OVMF without further modifications?
>>
>> I'd rather have a way to ask the hypervisor whether we are in PVH mode
>> (e.g. via CPUID or a hypercall to test for a devicemodel having itself
>> registered).
> 
> Keep in mind that from Xen's PoV PVH is just a HVM guest. Xen
> currently keeps a bitmap of the emulated devices that are available to
> a guest, but that's so far an internal field. We could consider
> exporting this on a cpuid leaf, but then we need to make it a fixed
> ABI.
> 
> Maybe we can make a list of platform devices that are not available on
> PVH and that Linux assumes to be present?

The main problem in Linux is that we have to know whether we are a HVM
or a PVH guest. Okay, we could scan for the Xen PCI-device to make the
distinction, but this is again rather late in the boot process.

Using implicit assumptions is normally a bad way to make such a
decision. I'd rather be either told by the boot loader I'm running as
PVH guest (and the boot loader does know it), or I want to ask the
hypervisor (which right now doesn't know), or I need another clear
distinction like the existence of the Xen PCI-device (which might be
a problem in the future for a PVH dom0 in a nested Xen environment).


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 19:37, Roger Pau Monné wrote:
> On Fri, Nov 03, 2017 at 07:23:50PM +0100, Juergen Gross wrote:
>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>
>>>> So again the question: how to tell whether we are PVH or HVM in
>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>
>>> Can we make grub/OVMF append a boot option?
>>>
>>> Or set setup_header.hardware_subarch to something? We already have
>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>> think).
>>
>> But wouldn't this break the idea to reuse the native boot paths in
>> grub/OVMF without further modifications?
>>
>> I'd rather have a way to ask the hypervisor whether we are in PVH mode
>> (e.g. via CPUID or a hypercall to test for a devicemodel having itself
>> registered).
> 
> Keep in mind that from Xen's PoV PVH is just a HVM guest. Xen
> currently keeps a bitmap of the emulated devices that are available to
> a guest, but that's so far an internal field. We could consider
> exporting this on a cpuid leaf, but then we need to make it a fixed
> ABI.
> 
> Maybe we can make a list of platform devices that are not available on
> PVH and that Linux assumes to be present?

The main problem in Linux is that we have to know whether we are a HVM
or a PVH guest. Okay, we could scan for the Xen PCI-device to make the
distinction, but this is again rather late in the boot process.

Using implicit assumptions is normally a bad way to make such a
decision. I'd rather be either told by the boot loader I'm running as
PVH guest (and the boot loader does know it), or I want to ask the
hypervisor (which right now doesn't know), or I need another clear
distinction like the existence of the Xen PCI-device (which might be
a problem in the future for a PVH dom0 in a nested Xen environment).


Juergen

Re: Xen PVH support in grub2

[Boris Ostrovsky]

On 11/03/2017 02:40 PM, Juergen Gross wrote:
> On 03/11/17 19:35, Boris Ostrovsky wrote:
>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>> So again the question: how to tell whether we are PVH or HVM in
>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>> Can we make grub/OVMF append a boot option?
>>>>
>>>> Or set setup_header.hardware_subarch to something? We already have
>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>>> think).
>>> But wouldn't this break the idea to reuse the native boot paths in
>>> grub/OVMF without further modifications?
>> WDYM? We will have to have some sort of a plugin in either one to build
>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
>> other things like setting memory and such.
> But isn't the zeropage already being built? I admit that setting subarch
> isn't a big deal, but using another entry with a passed-through pvh
> start struct isn't either...

I don't follow, sorry. My understanding is that zeropage will be built
by PVH-enlightened grub so part of this process would be setting the
subarch bit.

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Boris Ostrovsky]

On 11/03/2017 02:40 PM, Juergen Gross wrote:
> On 03/11/17 19:35, Boris Ostrovsky wrote:
>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>> So again the question: how to tell whether we are PVH or HVM in
>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>> Can we make grub/OVMF append a boot option?
>>>>
>>>> Or set setup_header.hardware_subarch to something? We already have
>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>>> think).
>>> But wouldn't this break the idea to reuse the native boot paths in
>>> grub/OVMF without further modifications?
>> WDYM? We will have to have some sort of a plugin in either one to build
>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
>> other things like setting memory and such.
> But isn't the zeropage already being built? I admit that setting subarch
> isn't a big deal, but using another entry with a passed-through pvh
> start struct isn't either...

I don't follow, sorry. My understanding is that zeropage will be built
by PVH-enlightened grub so part of this process would be setting the
subarch bit.

-boris

Re: Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 20:00, Boris Ostrovsky wrote:
> On 11/03/2017 02:40 PM, Juergen Gross wrote:
>> On 03/11/17 19:35, Boris Ostrovsky wrote:
>>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>>> So again the question: how to tell whether we are PVH or HVM in
>>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>>> Can we make grub/OVMF append a boot option?
>>>>>
>>>>> Or set setup_header.hardware_subarch to something? We already have
>>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>>>> think).
>>>> But wouldn't this break the idea to reuse the native boot paths in
>>>> grub/OVMF without further modifications?
>>> WDYM? We will have to have some sort of a plugin in either one to build
>>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
>>> other things like setting memory and such.
>> But isn't the zeropage already being built? I admit that setting subarch
>> isn't a big deal, but using another entry with a passed-through pvh
>> start struct isn't either...
> 
> I don't follow, sorry. My understanding is that zeropage will be built
> by PVH-enlightened grub so part of this process would be setting the
> subarch bit.

My reasoning was based on Roger's remark:

"OTOH if Linux is capable of booting from the native entry point inside
of a PVH container, we would only have to port OVMF and grub in order
to work inside of a PVH container, leaving the rest of the logic
untouched."


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 20:00, Boris Ostrovsky wrote:
> On 11/03/2017 02:40 PM, Juergen Gross wrote:
>> On 03/11/17 19:35, Boris Ostrovsky wrote:
>>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>>> So again the question: how to tell whether we are PVH or HVM in
>>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>>> Can we make grub/OVMF append a boot option?
>>>>>
>>>>> Or set setup_header.hardware_subarch to something? We already have
>>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>>>> think).
>>>> But wouldn't this break the idea to reuse the native boot paths in
>>>> grub/OVMF without further modifications?
>>> WDYM? We will have to have some sort of a plugin in either one to build
>>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
>>> other things like setting memory and such.
>> But isn't the zeropage already being built? I admit that setting subarch
>> isn't a big deal, but using another entry with a passed-through pvh
>> start struct isn't either...
> 
> I don't follow, sorry. My understanding is that zeropage will be built
> by PVH-enlightened grub so part of this process would be setting the
> subarch bit.

My reasoning was based on Roger's remark:

"OTOH if Linux is capable of booting from the native entry point inside
of a PVH container, we would only have to port OVMF and grub in order
to work inside of a PVH container, leaving the rest of the logic
untouched."


Juergen

Re: Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 13:17, Roger Pau Monné wrote:
> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>> be used inside of a PVH container?
>>>>
>>>> I don't think today's Linux allows that. This has been discussed
>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>
>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>> make the native entry point capable of booting inside of a PVH
>>> container up to the point where cpuid (or whatever method) can be used
>>> to detect the environment.
>>
>> Looking more thoroughly into the Linux boot code I think this could
>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>> How would you do that in FreeBSD? Via flags in the boot params? This
>> would the have to be done in the boot loader (e.g. grub or OVMF).
> 
> My plan was not to differentiate between HVM and PVH, but rather to
> make use of the ACPI information in order to decide which devices are
> available and which are not inside of a PVH guest.
> 
> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
> we already set "VGA Not Present" and "CMOS RTC Not Present". There
> might be other flags/fields that must be set, but I would like to
> avoid having a CPUID bit or similar saying "PVH", because then Xen
> will be tied to always providing the same set of devices in PVH
> containers.

I looked through the xen_pvh_domain() use cases in the Linux kernel
again.

Maybe we can really manage to not need differentiating PVH from HVM
until ACPI table scan. We'd need another hook for Xen, but this should
be easy as KVM already has a hook where we'd need one. So this can be
made more general and we are fine.

I even think we can drop some of the PVH tests, as the PVH-specific
handling (e.g. for grant table initialization) should work for HVM, too.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Juergen Gross]

On 03/11/17 13:17, Roger Pau Monné wrote:
> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>> be used inside of a PVH container?
>>>>
>>>> I don't think today's Linux allows that. This has been discussed
>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>
>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>> make the native entry point capable of booting inside of a PVH
>>> container up to the point where cpuid (or whatever method) can be used
>>> to detect the environment.
>>
>> Looking more thoroughly into the Linux boot code I think this could
>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>> How would you do that in FreeBSD? Via flags in the boot params? This
>> would the have to be done in the boot loader (e.g. grub or OVMF).
> 
> My plan was not to differentiate between HVM and PVH, but rather to
> make use of the ACPI information in order to decide which devices are
> available and which are not inside of a PVH guest.
> 
> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
> we already set "VGA Not Present" and "CMOS RTC Not Present". There
> might be other flags/fields that must be set, but I would like to
> avoid having a CPUID bit or similar saying "PVH", because then Xen
> will be tied to always providing the same set of devices in PVH
> containers.

I looked through the xen_pvh_domain() use cases in the Linux kernel
again.

Maybe we can really manage to not need differentiating PVH from HVM
until ACPI table scan. We'd need another hook for Xen, but this should
be easy as KVM already has a hook where we'd need one. So this can be
made more general and we are fine.

I even think we can drop some of the PVH tests, as the PVH-specific
handling (e.g. for grant table initialization) should work for HVM, too.


Juergen

Re: Xen PVH support in grub2

[Boris Ostrovsky]

On 11/06/2017 02:16 AM, Juergen Gross wrote:
> On 03/11/17 20:00, Boris Ostrovsky wrote:
>> On 11/03/2017 02:40 PM, Juergen Gross wrote:
>>> On 03/11/17 19:35, Boris Ostrovsky wrote:
>>>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>>>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>>>> So again the question: how to tell whether we are PVH or HVM in
>>>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>>>> Can we make grub/OVMF append a boot option?
>>>>>>
>>>>>> Or set setup_header.hardware_subarch to something? We already have
>>>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>>>>> think).
>>>>> But wouldn't this break the idea to reuse the native boot paths in
>>>>> grub/OVMF without further modifications?
>>>> WDYM? We will have to have some sort of a plugin in either one to build
>>>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
>>>> other things like setting memory and such.
>>> But isn't the zeropage already being built? I admit that setting subarch
>>> isn't a big deal, but using another entry with a passed-through pvh
>>> start struct isn't either...
>> I don't follow, sorry. My understanding is that zeropage will be built
>> by PVH-enlightened grub so part of this process would be setting the
>> subarch bit.
> My reasoning was based on Roger's remark:
>
> "OTOH if Linux is capable of booting from the native entry point inside
> of a PVH container, we would only have to port OVMF and grub in order
> to work inside of a PVH container, leaving the rest of the logic
> untouched."

Right, and in my mind porting OVMF/grub includes creating proper zeropage.

BTW, another option might be to "type_of_loader = (9 << 4) | 0", which
is what init_pvh_bootparams() does. In fact, whatever is done in the
firmware should probably match what that routine does.

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Boris Ostrovsky]

On 11/06/2017 02:16 AM, Juergen Gross wrote:
> On 03/11/17 20:00, Boris Ostrovsky wrote:
>> On 11/03/2017 02:40 PM, Juergen Gross wrote:
>>> On 03/11/17 19:35, Boris Ostrovsky wrote:
>>>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>>>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>>>> So again the question: how to tell whether we are PVH or HVM in
>>>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>>>> Can we make grub/OVMF append a boot option?
>>>>>>
>>>>>> Or set setup_header.hardware_subarch to something? We already have
>>>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>>>>> think).
>>>>> But wouldn't this break the idea to reuse the native boot paths in
>>>>> grub/OVMF without further modifications?
>>>> WDYM? We will have to have some sort of a plugin in either one to build
>>>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
>>>> other things like setting memory and such.
>>> But isn't the zeropage already being built? I admit that setting subarch
>>> isn't a big deal, but using another entry with a passed-through pvh
>>> start struct isn't either...
>> I don't follow, sorry. My understanding is that zeropage will be built
>> by PVH-enlightened grub so part of this process would be setting the
>> subarch bit.
> My reasoning was based on Roger's remark:
>
> "OTOH if Linux is capable of booting from the native entry point inside
> of a PVH container, we would only have to port OVMF and grub in order
> to work inside of a PVH container, leaving the rest of the logic
> untouched."

Right, and in my mind porting OVMF/grub includes creating proper zeropage.

BTW, another option might be to "type_of_loader = (9 << 4) | 0", which
is what init_pvh_bootparams() does. In fact, whatever is done in the
firmware should probably match what that routine does.

-boris

Re: Xen PVH support in grub2

[Juergen Gross]

On 06/11/17 15:51, Boris Ostrovsky wrote:
> On 11/06/2017 02:16 AM, Juergen Gross wrote:
>> On 03/11/17 20:00, Boris Ostrovsky wrote:
>>> On 11/03/2017 02:40 PM, Juergen Gross wrote:
>>>> On 03/11/17 19:35, Boris Ostrovsky wrote:
>>>>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>>>>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>>>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>>>>> So again the question: how to tell whether we are PVH or HVM in
>>>>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>>>>> Can we make grub/OVMF append a boot option?
>>>>>>>
>>>>>>> Or set setup_header.hardware_subarch to something? We already have
>>>>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>>>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>>>>>> think).
>>>>>> But wouldn't this break the idea to reuse the native boot paths in
>>>>>> grub/OVMF without further modifications?
>>>>> WDYM? We will have to have some sort of a plugin in either one to build
>>>>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
>>>>> other things like setting memory and such.
>>>> But isn't the zeropage already being built? I admit that setting subarch
>>>> isn't a big deal, but using another entry with a passed-through pvh
>>>> start struct isn't either...
>>> I don't follow, sorry. My understanding is that zeropage will be built
>>> by PVH-enlightened grub so part of this process would be setting the
>>> subarch bit.
>> My reasoning was based on Roger's remark:
>>
>> "OTOH if Linux is capable of booting from the native entry point inside
>> of a PVH container, we would only have to port OVMF and grub in order
>> to work inside of a PVH container, leaving the rest of the logic
>> untouched."
> 
> Right, and in my mind porting OVMF/grub includes creating proper zeropage.

Aah, okay. I reasoned on the assumption to just enable OVMF/grub to run
in PVH environment without touching the parts setting up anything for
the new kernel.

> BTW, another option might be to "type_of_loader = (9 << 4) | 0", which
> is what init_pvh_bootparams() does. In fact, whatever is done in the
> firmware should probably match what that routine does.

So it wouldn't be possible any longer to tell whether the kernel has
been booted directly or via grub. I don't like this. The loader type
is accessible via sysfs after all.


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Juergen Gross]

On 06/11/17 15:51, Boris Ostrovsky wrote:
> On 11/06/2017 02:16 AM, Juergen Gross wrote:
>> On 03/11/17 20:00, Boris Ostrovsky wrote:
>>> On 11/03/2017 02:40 PM, Juergen Gross wrote:
>>>> On 03/11/17 19:35, Boris Ostrovsky wrote:
>>>>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>>>>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>>>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>>>>> So again the question: how to tell whether we are PVH or HVM in
>>>>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>>>>> Can we make grub/OVMF append a boot option?
>>>>>>>
>>>>>>> Or set setup_header.hardware_subarch to something? We already have
>>>>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>>>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>>>>>> think).
>>>>>> But wouldn't this break the idea to reuse the native boot paths in
>>>>>> grub/OVMF without further modifications?
>>>>> WDYM? We will have to have some sort of a plugin in either one to build
>>>>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
>>>>> other things like setting memory and such.
>>>> But isn't the zeropage already being built? I admit that setting subarch
>>>> isn't a big deal, but using another entry with a passed-through pvh
>>>> start struct isn't either...
>>> I don't follow, sorry. My understanding is that zeropage will be built
>>> by PVH-enlightened grub so part of this process would be setting the
>>> subarch bit.
>> My reasoning was based on Roger's remark:
>>
>> "OTOH if Linux is capable of booting from the native entry point inside
>> of a PVH container, we would only have to port OVMF and grub in order
>> to work inside of a PVH container, leaving the rest of the logic
>> untouched."
> 
> Right, and in my mind porting OVMF/grub includes creating proper zeropage.

Aah, okay. I reasoned on the assumption to just enable OVMF/grub to run
in PVH environment without touching the parts setting up anything for
the new kernel.

> BTW, another option might be to "type_of_loader = (9 << 4) | 0", which
> is what init_pvh_bootparams() does. In fact, whatever is done in the
> firmware should probably match what that routine does.

So it wouldn't be possible any longer to tell whether the kernel has
been booted directly or via grub. I don't like this. The loader type
is accessible via sysfs after all.


Juergen

Re: Xen PVH support in grub2

[Boris Ostrovsky]

On 11/06/2017 10:05 AM, Juergen Gross wrote:
> On 06/11/17 15:51, Boris Ostrovsky wrote:
>> On 11/06/2017 02:16 AM, Juergen Gross wrote:
>>> On 03/11/17 20:00, Boris Ostrovsky wrote:
>>>> On 11/03/2017 02:40 PM, Juergen Gross wrote:
>>>>> On 03/11/17 19:35, Boris Ostrovsky wrote:
>>>>>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>>>>>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>>>>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>>>>>> So again the question: how to tell whether we are PVH or HVM in
>>>>>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>>>>>> Can we make grub/OVMF append a boot option?
>>>>>>>>
>>>>>>>> Or set setup_header.hardware_subarch to something? We already have
>>>>>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>>>>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>>>>>>> think).
>>>>>>> But wouldn't this break the idea to reuse the native boot paths in
>>>>>>> grub/OVMF without further modifications?
>>>>>> WDYM? We will have to have some sort of a plugin in either one to build
>>>>>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
>>>>>> other things like setting memory and such.
>>>>> But isn't the zeropage already being built? I admit that setting subarch
>>>>> isn't a big deal, but using another entry with a passed-through pvh
>>>>> start struct isn't either...
>>>> I don't follow, sorry. My understanding is that zeropage will be built
>>>> by PVH-enlightened grub so part of this process would be setting the
>>>> subarch bit.
>>> My reasoning was based on Roger's remark:
>>>
>>> "OTOH if Linux is capable of booting from the native entry point inside
>>> of a PVH container, we would only have to port OVMF and grub in order
>>> to work inside of a PVH container, leaving the rest of the logic
>>> untouched."
>> Right, and in my mind porting OVMF/grub includes creating proper zeropage.
> Aah, okay. I reasoned on the assumption to just enable OVMF/grub to run
> in PVH environment without touching the parts setting up anything for
> the new kernel.

Someone needs to do what xen_prepare_pvh() does.

And, for 64-bit, we also may need to build early pagetables since
startup_64() (unlike startup_32()) expects paging to be on. (I don't
know whether this is already part of standard FW codepath)

>
>> BTW, another option might be to "type_of_loader = (9 << 4) | 0", which
>> is what init_pvh_bootparams() does. In fact, whatever is done in the
>> firmware should probably match what that routine does.
> So it wouldn't be possible any longer to tell whether the kernel has
> been booted directly or via grub. I don't like this. The loader type
> is accessible via sysfs after all.

I didn't know that. What is the path?

-boris


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Boris Ostrovsky]

On 11/06/2017 10:05 AM, Juergen Gross wrote:
> On 06/11/17 15:51, Boris Ostrovsky wrote:
>> On 11/06/2017 02:16 AM, Juergen Gross wrote:
>>> On 03/11/17 20:00, Boris Ostrovsky wrote:
>>>> On 11/03/2017 02:40 PM, Juergen Gross wrote:
>>>>> On 03/11/17 19:35, Boris Ostrovsky wrote:
>>>>>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>>>>>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>>>>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>>>>>> So again the question: how to tell whether we are PVH or HVM in
>>>>>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>>>>>> Can we make grub/OVMF append a boot option?
>>>>>>>>
>>>>>>>> Or set setup_header.hardware_subarch to something? We already have
>>>>>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>>>>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>>>>>>> think).
>>>>>>> But wouldn't this break the idea to reuse the native boot paths in
>>>>>>> grub/OVMF without further modifications?
>>>>>> WDYM? We will have to have some sort of a plugin in either one to build
>>>>>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
>>>>>> other things like setting memory and such.
>>>>> But isn't the zeropage already being built? I admit that setting subarch
>>>>> isn't a big deal, but using another entry with a passed-through pvh
>>>>> start struct isn't either...
>>>> I don't follow, sorry. My understanding is that zeropage will be built
>>>> by PVH-enlightened grub so part of this process would be setting the
>>>> subarch bit.
>>> My reasoning was based on Roger's remark:
>>>
>>> "OTOH if Linux is capable of booting from the native entry point inside
>>> of a PVH container, we would only have to port OVMF and grub in order
>>> to work inside of a PVH container, leaving the rest of the logic
>>> untouched."
>> Right, and in my mind porting OVMF/grub includes creating proper zeropage.
> Aah, okay. I reasoned on the assumption to just enable OVMF/grub to run
> in PVH environment without touching the parts setting up anything for
> the new kernel.

Someone needs to do what xen_prepare_pvh() does.

And, for 64-bit, we also may need to build early pagetables since
startup_64() (unlike startup_32()) expects paging to be on. (I don't
know whether this is already part of standard FW codepath)

>
>> BTW, another option might be to "type_of_loader = (9 << 4) | 0", which
>> is what init_pvh_bootparams() does. In fact, whatever is done in the
>> firmware should probably match what that routine does.
> So it wouldn't be possible any longer to tell whether the kernel has
> been booted directly or via grub. I don't like this. The loader type
> is accessible via sysfs after all.

I didn't know that. What is the path?

-boris

Re: Xen PVH support in grub2

[Juergen Gross]

On 06/11/17 17:42, Boris Ostrovsky wrote:
> On 11/06/2017 10:05 AM, Juergen Gross wrote:
>> On 06/11/17 15:51, Boris Ostrovsky wrote:
>>> On 11/06/2017 02:16 AM, Juergen Gross wrote:
>>>> On 03/11/17 20:00, Boris Ostrovsky wrote:
>>>>> On 11/03/2017 02:40 PM, Juergen Gross wrote:
>>>>>> On 03/11/17 19:35, Boris Ostrovsky wrote:
>>>>>>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>>>>>>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>>>>>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>>>>>>> So again the question: how to tell whether we are PVH or HVM in
>>>>>>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>>>>>>> Can we make grub/OVMF append a boot option?
>>>>>>>>>
>>>>>>>>> Or set setup_header.hardware_subarch to something? We already have
>>>>>>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>>>>>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>>>>>>>> think).
>>>>>>>> But wouldn't this break the idea to reuse the native boot paths in
>>>>>>>> grub/OVMF without further modifications?
>>>>>>> WDYM? We will have to have some sort of a plugin in either one to build
>>>>>>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
>>>>>>> other things like setting memory and such.
>>>>>> But isn't the zeropage already being built? I admit that setting subarch
>>>>>> isn't a big deal, but using another entry with a passed-through pvh
>>>>>> start struct isn't either...
>>>>> I don't follow, sorry. My understanding is that zeropage will be built
>>>>> by PVH-enlightened grub so part of this process would be setting the
>>>>> subarch bit.
>>>> My reasoning was based on Roger's remark:
>>>>
>>>> "OTOH if Linux is capable of booting from the native entry point inside
>>>> of a PVH container, we would only have to port OVMF and grub in order
>>>> to work inside of a PVH container, leaving the rest of the logic
>>>> untouched."
>>> Right, and in my mind porting OVMF/grub includes creating proper zeropage.
>> Aah, okay. I reasoned on the assumption to just enable OVMF/grub to run
>> in PVH environment without touching the parts setting up anything for
>> the new kernel.
> 
> Someone needs to do what xen_prepare_pvh() does.

As the loader is filling in the memory map information the only thing
remaining would be setting xen_pvh. And this could be delayed as my test
have shown, so we only need to detect the PVH case from inside the
kernel. One possibility would be the flags in the ACPI FADT table as
Roger mentioned, another idea would be a flag in zeropage set by the
loader.

> And, for 64-bit, we also may need to build early pagetables since
> startup_64() (unlike startup_32()) expects paging to be on. (I don't
> know whether this is already part of standard FW codepath)

This would be done the same way as for a native kernel.

>>> BTW, another option might be to "type_of_loader = (9 << 4) | 0", which
>>> is what init_pvh_bootparams() does. In fact, whatever is done in the
>>> firmware should probably match what that routine does.
>> So it wouldn't be possible any longer to tell whether the kernel has
>> been booted directly or via grub. I don't like this. The loader type
>> is accessible via sysfs after all.
> 
> I didn't know that. What is the path?

/proc/sys/kernel/bootloader_type


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Juergen Gross]

On 06/11/17 17:42, Boris Ostrovsky wrote:
> On 11/06/2017 10:05 AM, Juergen Gross wrote:
>> On 06/11/17 15:51, Boris Ostrovsky wrote:
>>> On 11/06/2017 02:16 AM, Juergen Gross wrote:
>>>> On 03/11/17 20:00, Boris Ostrovsky wrote:
>>>>> On 11/03/2017 02:40 PM, Juergen Gross wrote:
>>>>>> On 03/11/17 19:35, Boris Ostrovsky wrote:
>>>>>>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>>>>>>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>>>>>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>>>>>>> So again the question: how to tell whether we are PVH or HVM in
>>>>>>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>>>>>>> Can we make grub/OVMF append a boot option?
>>>>>>>>>
>>>>>>>>> Or set setup_header.hardware_subarch to something? We already have
>>>>>>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>>>>>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>>>>>>>> think).
>>>>>>>> But wouldn't this break the idea to reuse the native boot paths in
>>>>>>>> grub/OVMF without further modifications?
>>>>>>> WDYM? We will have to have some sort of a plugin in either one to build
>>>>>>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
>>>>>>> other things like setting memory and such.
>>>>>> But isn't the zeropage already being built? I admit that setting subarch
>>>>>> isn't a big deal, but using another entry with a passed-through pvh
>>>>>> start struct isn't either...
>>>>> I don't follow, sorry. My understanding is that zeropage will be built
>>>>> by PVH-enlightened grub so part of this process would be setting the
>>>>> subarch bit.
>>>> My reasoning was based on Roger's remark:
>>>>
>>>> "OTOH if Linux is capable of booting from the native entry point inside
>>>> of a PVH container, we would only have to port OVMF and grub in order
>>>> to work inside of a PVH container, leaving the rest of the logic
>>>> untouched."
>>> Right, and in my mind porting OVMF/grub includes creating proper zeropage.
>> Aah, okay. I reasoned on the assumption to just enable OVMF/grub to run
>> in PVH environment without touching the parts setting up anything for
>> the new kernel.
> 
> Someone needs to do what xen_prepare_pvh() does.

As the loader is filling in the memory map information the only thing
remaining would be setting xen_pvh. And this could be delayed as my test
have shown, so we only need to detect the PVH case from inside the
kernel. One possibility would be the flags in the ACPI FADT table as
Roger mentioned, another idea would be a flag in zeropage set by the
loader.

> And, for 64-bit, we also may need to build early pagetables since
> startup_64() (unlike startup_32()) expects paging to be on. (I don't
> know whether this is already part of standard FW codepath)

This would be done the same way as for a native kernel.

>>> BTW, another option might be to "type_of_loader = (9 << 4) | 0", which
>>> is what init_pvh_bootparams() does. In fact, whatever is done in the
>>> firmware should probably match what that routine does.
>> So it wouldn't be possible any longer to tell whether the kernel has
>> been booted directly or via grub. I don't like this. The loader type
>> is accessible via sysfs after all.
> 
> I didn't know that. What is the path?

/proc/sys/kernel/bootloader_type


Juergen

Re: Xen PVH support in grub2

[Juergen Gross]

On 06/11/17 12:36, Juergen Gross wrote:
> On 03/11/17 13:17, Roger Pau Monné wrote:
>> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>> be used inside of a PVH container?
>>>>>
>>>>> I don't think today's Linux allows that. This has been discussed
>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>
>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>> make the native entry point capable of booting inside of a PVH
>>>> container up to the point where cpuid (or whatever method) can be used
>>>> to detect the environment.
>>>
>>> Looking more thoroughly into the Linux boot code I think this could
>>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>>> How would you do that in FreeBSD? Via flags in the boot params? This
>>> would the have to be done in the boot loader (e.g. grub or OVMF).
>>
>> My plan was not to differentiate between HVM and PVH, but rather to
>> make use of the ACPI information in order to decide which devices are
>> available and which are not inside of a PVH guest.
>>
>> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
>> we already set "VGA Not Present" and "CMOS RTC Not Present". There
>> might be other flags/fields that must be set, but I would like to
>> avoid having a CPUID bit or similar saying "PVH", because then Xen
>> will be tied to always providing the same set of devices in PVH
>> containers.
> 
> I looked through the xen_pvh_domain() use cases in the Linux kernel
> again.
> 
> Maybe we can really manage to not need differentiating PVH from HVM
> until ACPI table scan. We'd need another hook for Xen, but this should
> be easy as KVM already has a hook where we'd need one. So this can be
> made more general and we are fine.
> 
> I even think we can drop some of the PVH tests, as the PVH-specific
> handling (e.g. for grant table initialization) should work for HVM, too.

So I did a little test now: with some small patches added I've managed
to boot a PVH Linux kernel without any special handling in the early
PVH paths of the kernel (only setting up initial page tables and faking
a memory map similar to the one grub would deliver). PVH detection is
done via ACPI table ("VGA Not Present" and "CMOS RTC Not Present" in a
HVM domain will result in PVH assumed).

If nobody objects to this handling I'll send patches soon.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Juergen Gross]

On 06/11/17 12:36, Juergen Gross wrote:
> On 03/11/17 13:17, Roger Pau Monné wrote:
>> On Fri, Nov 03, 2017 at 01:00:46PM +0100, Juergen Gross wrote:
>>> On 29/09/17 17:51, Roger Pau Monné wrote:
>>>> On Fri, Sep 29, 2017 at 03:33:58PM +0000, Juergen Gross wrote:
>>>>> On 29/09/17 17:24, Roger Pau Monné wrote:
>>>>>> On Fri, Sep 29, 2017 at 02:46:53PM +0000, Juergen Gross wrote:
>>>>>> Then, I also wonder whether it would make sense for this grub to load
>>>>>> the kernel using the PVH entry point or the native entry point. Would
>>>>>> it be possible to boot a Linux kernel up to the point where cpuid can
>>>>>> be used inside of a PVH container?
>>>>>
>>>>> I don't think today's Linux allows that. This has been discussed
>>>>> very thoroughly at the time Boris added PVH V2 support to the kernel.
>>>>
>>>> OK, I'm not going to insist on that, but my plans for FreeBSD is to
>>>> make the native entry point capable of booting inside of a PVH
>>>> container up to the point where cpuid (or whatever method) can be used
>>>> to detect the environment.
>>>
>>> Looking more thoroughly into the Linux boot code I think this could
>>> work for Linux, too. But only if we can tell PVH from HVM in the guest.
>>> How would you do that in FreeBSD? Via flags in the boot params? This
>>> would the have to be done in the boot loader (e.g. grub or OVMF).
>>
>> My plan was not to differentiate between HVM and PVH, but rather to
>> make use of the ACPI information in order to decide which devices are
>> available and which are not inside of a PVH guest.
>>
>> For example in the FADT "IA-PC Boot Architecture Flags" field for PVH
>> we already set "VGA Not Present" and "CMOS RTC Not Present". There
>> might be other flags/fields that must be set, but I would like to
>> avoid having a CPUID bit or similar saying "PVH", because then Xen
>> will be tied to always providing the same set of devices in PVH
>> containers.
> 
> I looked through the xen_pvh_domain() use cases in the Linux kernel
> again.
> 
> Maybe we can really manage to not need differentiating PVH from HVM
> until ACPI table scan. We'd need another hook for Xen, but this should
> be easy as KVM already has a hook where we'd need one. So this can be
> made more general and we are fine.
> 
> I even think we can drop some of the PVH tests, as the PVH-specific
> handling (e.g. for grant table initialization) should work for HVM, too.

So I did a little test now: with some small patches added I've managed
to boot a PVH Linux kernel without any special handling in the early
PVH paths of the kernel (only setting up initial page tables and faking
a memory map similar to the one grub would deliver). PVH detection is
done via ACPI table ("VGA Not Present" and "CMOS RTC Not Present" in a
HVM domain will result in PVH assumed).

If nobody objects to this handling I'll send patches soon.


Juergen

Re: Xen PVH support in grub2

[Boris Ostrovsky]

On 11/07/2017 02:42 AM, Juergen Gross wrote:
> On 06/11/17 17:42, Boris Ostrovsky wrote:
>> On 11/06/2017 10:05 AM, Juergen Gross wrote:
>>> On 06/11/17 15:51, Boris Ostrovsky wrote:
>>>> On 11/06/2017 02:16 AM, Juergen Gross wrote:
>>>>> On 03/11/17 20:00, Boris Ostrovsky wrote:
>>>>>> On 11/03/2017 02:40 PM, Juergen Gross wrote:
>>>>>>> On 03/11/17 19:35, Boris Ostrovsky wrote:
>>>>>>>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>>>>>>>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>>>>>>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>>>>>>>> So again the question: how to tell whether we are PVH or HVM in
>>>>>>>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>>>>>>>> Can we make grub/OVMF append a boot option?
>>>>>>>>>>
>>>>>>>>>> Or set setup_header.hardware_subarch to something? We already have
>>>>>>>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>>>>>>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>>>>>>>>> think).
>>>>>>>>> But wouldn't this break the idea to reuse the native boot paths in
>>>>>>>>> grub/OVMF without further modifications?
>>>>>>>> WDYM? We will have to have some sort of a plugin in either one to build
>>>>>>>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
>>>>>>>> other things like setting memory and such.
>>>>>>> But isn't the zeropage already being built? I admit that setting subarch
>>>>>>> isn't a big deal, but using another entry with a passed-through pvh
>>>>>>> start struct isn't either...
>>>>>> I don't follow, sorry. My understanding is that zeropage will be built
>>>>>> by PVH-enlightened grub so part of this process would be setting the
>>>>>> subarch bit.
>>>>> My reasoning was based on Roger's remark:
>>>>>
>>>>> "OTOH if Linux is capable of booting from the native entry point inside
>>>>> of a PVH container, we would only have to port OVMF and grub in order
>>>>> to work inside of a PVH container, leaving the rest of the logic
>>>>> untouched."
>>>> Right, and in my mind porting OVMF/grub includes creating proper zeropage.
>>> Aah, okay. I reasoned on the assumption to just enable OVMF/grub to run
>>> in PVH environment without touching the parts setting up anything for
>>> the new kernel.
>> Someone needs to do what xen_prepare_pvh() does.
> As the loader is filling in the memory map information 

That's the thing that I thought may need to be done by us (setting
commandline too) . But I haven't looked at Xen support in grub so maybe
it's already there.

> the only thing
> remaining would be setting xen_pvh. And this could be delayed as my test
> have shown, so we only need to detect the PVH case from inside the
> kernel. One possibility would be the flags in the ACPI FADT table as
> Roger mentioned, another idea would be a flag in zeropage set by the
> loader.
>
>> And, for 64-bit, we also may need to build early pagetables since
>> startup_64() (unlike startup_32()) expects paging to be on. (I don't
>> know whether this is already part of standard FW codepath)
> This would be done the same way as for a native kernel.

This is done by Linux trampoline code. AFAIK grub loads kernel in realmode.


-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Boris Ostrovsky]

On 11/07/2017 02:42 AM, Juergen Gross wrote:
> On 06/11/17 17:42, Boris Ostrovsky wrote:
>> On 11/06/2017 10:05 AM, Juergen Gross wrote:
>>> On 06/11/17 15:51, Boris Ostrovsky wrote:
>>>> On 11/06/2017 02:16 AM, Juergen Gross wrote:
>>>>> On 03/11/17 20:00, Boris Ostrovsky wrote:
>>>>>> On 11/03/2017 02:40 PM, Juergen Gross wrote:
>>>>>>> On 03/11/17 19:35, Boris Ostrovsky wrote:
>>>>>>>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
>>>>>>>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
>>>>>>>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
>>>>>>>>>>> So again the question: how to tell whether we are PVH or HVM in
>>>>>>>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
>>>>>>>>>> Can we make grub/OVMF append a boot option?
>>>>>>>>>>
>>>>>>>>>> Or set setup_header.hardware_subarch to something? We already have
>>>>>>>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
>>>>>>>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
>>>>>>>>>> think).
>>>>>>>>> But wouldn't this break the idea to reuse the native boot paths in
>>>>>>>>> grub/OVMF without further modifications?
>>>>>>>> WDYM? We will have to have some sort of a plugin in either one to build
>>>>>>>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
>>>>>>>> other things like setting memory and such.
>>>>>>> But isn't the zeropage already being built? I admit that setting subarch
>>>>>>> isn't a big deal, but using another entry with a passed-through pvh
>>>>>>> start struct isn't either...
>>>>>> I don't follow, sorry. My understanding is that zeropage will be built
>>>>>> by PVH-enlightened grub so part of this process would be setting the
>>>>>> subarch bit.
>>>>> My reasoning was based on Roger's remark:
>>>>>
>>>>> "OTOH if Linux is capable of booting from the native entry point inside
>>>>> of a PVH container, we would only have to port OVMF and grub in order
>>>>> to work inside of a PVH container, leaving the rest of the logic
>>>>> untouched."
>>>> Right, and in my mind porting OVMF/grub includes creating proper zeropage.
>>> Aah, okay. I reasoned on the assumption to just enable OVMF/grub to run
>>> in PVH environment without touching the parts setting up anything for
>>> the new kernel.
>> Someone needs to do what xen_prepare_pvh() does.
> As the loader is filling in the memory map information 

That's the thing that I thought may need to be done by us (setting
commandline too) . But I haven't looked at Xen support in grub so maybe
it's already there.

> the only thing
> remaining would be setting xen_pvh. And this could be delayed as my test
> have shown, so we only need to detect the PVH case from inside the
> kernel. One possibility would be the flags in the ACPI FADT table as
> Roger mentioned, another idea would be a flag in zeropage set by the
> loader.
>
>> And, for 64-bit, we also may need to build early pagetables since
>> startup_64() (unlike startup_32()) expects paging to be on. (I don't
>> know whether this is already part of standard FW codepath)
> This would be done the same way as for a native kernel.

This is done by Linux trampoline code. AFAIK grub loads kernel in realmode.


-boris

Re: Xen PVH support in grub2

[Konrad Rzeszutek Wilk]

On Tue, Nov 07, 2017 at 11:10:29AM -0500, Boris Ostrovsky wrote:
> On 11/07/2017 02:42 AM, Juergen Gross wrote:
> > On 06/11/17 17:42, Boris Ostrovsky wrote:
> >> On 11/06/2017 10:05 AM, Juergen Gross wrote:
> >>> On 06/11/17 15:51, Boris Ostrovsky wrote:
> >>>> On 11/06/2017 02:16 AM, Juergen Gross wrote:
> >>>>> On 03/11/17 20:00, Boris Ostrovsky wrote:
> >>>>>> On 11/03/2017 02:40 PM, Juergen Gross wrote:
> >>>>>>> On 03/11/17 19:35, Boris Ostrovsky wrote:
> >>>>>>>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
> >>>>>>>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
> >>>>>>>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
> >>>>>>>>>>> So again the question: how to tell whether we are PVH or HVM in
> >>>>>>>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
> >>>>>>>>>> Can we make grub/OVMF append a boot option?
> >>>>>>>>>>
> >>>>>>>>>> Or set setup_header.hardware_subarch to something? We already have
> >>>>>>>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
> >>>>>>>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
> >>>>>>>>>> think).
> >>>>>>>>> But wouldn't this break the idea to reuse the native boot paths in
> >>>>>>>>> grub/OVMF without further modifications?
> >>>>>>>> WDYM? We will have to have some sort of a plugin in either one to build
> >>>>>>>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
> >>>>>>>> other things like setting memory and such.
> >>>>>>> But isn't the zeropage already being built? I admit that setting subarch
> >>>>>>> isn't a big deal, but using another entry with a passed-through pvh
> >>>>>>> start struct isn't either...
> >>>>>> I don't follow, sorry. My understanding is that zeropage will be built
> >>>>>> by PVH-enlightened grub so part of this process would be setting the
> >>>>>> subarch bit.
> >>>>> My reasoning was based on Roger's remark:
> >>>>>
> >>>>> "OTOH if Linux is capable of booting from the native entry point inside
> >>>>> of a PVH container, we would only have to port OVMF and grub in order
> >>>>> to work inside of a PVH container, leaving the rest of the logic
> >>>>> untouched."
> >>>> Right, and in my mind porting OVMF/grub includes creating proper zeropage.
> >>> Aah, okay. I reasoned on the assumption to just enable OVMF/grub to run
> >>> in PVH environment without touching the parts setting up anything for
> >>> the new kernel.
> >> Someone needs to do what xen_prepare_pvh() does.
> > As the loader is filling in the memory map information 
> 
> That's the thing that I thought may need to be done by us (setting
> commandline too) . But I haven't looked at Xen support in grub so maybe
> it's already there.
> 
> > the only thing
> > remaining would be setting xen_pvh. And this could be delayed as my test
> > have shown, so we only need to detect the PVH case from inside the
> > kernel. One possibility would be the flags in the ACPI FADT table as
> > Roger mentioned, another idea would be a flag in zeropage set by the
> > loader.
> >
> >> And, for 64-bit, we also may need to build early pagetables since
> >> startup_64() (unlike startup_32()) expects paging to be on. (I don't
> >> know whether this is already part of standard FW codepath)
> > This would be done the same way as for a native kernel.
> 
> This is done by Linux trampoline code. AFAIK grub loads kernel in realmode.

Adding in Maran who is looking at re-using PVH for qemu launching
the kernels.

> 
> 
> -boris
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen PVH support in grub2

[Konrad Rzeszutek Wilk]

On Tue, Nov 07, 2017 at 11:10:29AM -0500, Boris Ostrovsky wrote:
> On 11/07/2017 02:42 AM, Juergen Gross wrote:
> > On 06/11/17 17:42, Boris Ostrovsky wrote:
> >> On 11/06/2017 10:05 AM, Juergen Gross wrote:
> >>> On 06/11/17 15:51, Boris Ostrovsky wrote:
> >>>> On 11/06/2017 02:16 AM, Juergen Gross wrote:
> >>>>> On 03/11/17 20:00, Boris Ostrovsky wrote:
> >>>>>> On 11/03/2017 02:40 PM, Juergen Gross wrote:
> >>>>>>> On 03/11/17 19:35, Boris Ostrovsky wrote:
> >>>>>>>> On 11/03/2017 02:23 PM, Juergen Gross wrote:
> >>>>>>>>> On 03/11/17 19:19, Boris Ostrovsky wrote:
> >>>>>>>>>> On 11/03/2017 02:05 PM, Juergen Gross wrote:
> >>>>>>>>>>> So again the question: how to tell whether we are PVH or HVM in
> >>>>>>>>>>> init_hypervisor_platform()? ACPi tables are scanned way later...
> >>>>>>>>>> Can we make grub/OVMF append a boot option?
> >>>>>>>>>>
> >>>>>>>>>> Or set setup_header.hardware_subarch to something? We already have
> >>>>>>>>>> X86_SUBARCH_XEN but it is only used by PV.  Or we might be able to use
> >>>>>>>>>> hardware_subarch_data (will need to get a buy-in from x86 maintainers, I
> >>>>>>>>>> think).
> >>>>>>>>> But wouldn't this break the idea to reuse the native boot paths in
> >>>>>>>>> grub/OVMF without further modifications?
> >>>>>>>> WDYM? We will have to have some sort of a plugin in either one to build
> >>>>>>>> the zeropage anyway. So we'd set hardware_subarch there, in addition to
> >>>>>>>> other things like setting memory and such.
> >>>>>>> But isn't the zeropage already being built? I admit that setting subarch
> >>>>>>> isn't a big deal, but using another entry with a passed-through pvh
> >>>>>>> start struct isn't either...
> >>>>>> I don't follow, sorry. My understanding is that zeropage will be built
> >>>>>> by PVH-enlightened grub so part of this process would be setting the
> >>>>>> subarch bit.
> >>>>> My reasoning was based on Roger's remark:
> >>>>>
> >>>>> "OTOH if Linux is capable of booting from the native entry point inside
> >>>>> of a PVH container, we would only have to port OVMF and grub in order
> >>>>> to work inside of a PVH container, leaving the rest of the logic
> >>>>> untouched."
> >>>> Right, and in my mind porting OVMF/grub includes creating proper zeropage.
> >>> Aah, okay. I reasoned on the assumption to just enable OVMF/grub to run
> >>> in PVH environment without touching the parts setting up anything for
> >>> the new kernel.
> >> Someone needs to do what xen_prepare_pvh() does.
> > As the loader is filling in the memory map information 
> 
> That's the thing that I thought may need to be done by us (setting
> commandline too) . But I haven't looked at Xen support in grub so maybe
> it's already there.
> 
> > the only thing
> > remaining would be setting xen_pvh. And this could be delayed as my test
> > have shown, so we only need to detect the PVH case from inside the
> > kernel. One possibility would be the flags in the ACPI FADT table as
> > Roger mentioned, another idea would be a flag in zeropage set by the
> > loader.
> >
> >> And, for 64-bit, we also may need to build early pagetables since
> >> startup_64() (unlike startup_32()) expects paging to be on. (I don't
> >> know whether this is already part of standard FW codepath)
> > This would be done the same way as for a native kernel.
> 
> This is done by Linux trampoline code. AFAIK grub loads kernel in realmode.

Adding in Maran who is looking at re-using PVH for qemu launching
the kernels.

> 
> 
> -boris
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel

Xen PVH support in grub2

[Juergen Gross]

I'm thinking about to add support for Xen PVH guests to grub2.

Basically I see two options how to do it:

a) add PVH support to current grub.xen (both, 32- and 64-bit), in
   order to use the same grub binary for either pv-domains or
   pvh-domains
b) create a new variant grub.xenpvh capable only to boot pvh-domains

Are there any preferences which way to go?


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel