RFC: making the PVH 64bit ABI as stable

RFC: making the PVH 64bit ABI as stable

[Roger Pau Monné]

Hello,

The document describing the PVH interface was committed 9 months ago
[1], and since then there hasn't been any change regarding the
interface. PVH is still missing features in order to have feature parity
with pure PV, mainly:

 - DomU miration support.
 - PCI passthrough support.
 - 32bit support.
 - AMD support.

AFAICT however none of these features are going to change the current
ABI. PCI passthrough might expand it, by adding new hypercalls, but I
don't think this should prevent us from marking the current ABI as
stable. ARM for example doesn't have PCI passthrough yet or migration
support, but the ABI has been marked as stable.

To that end, I would like to request the 64bit PVH ABI to be marked as
stable for DomUs. This is needed so external projects (like PVH support
for grub2) can progress.

Roger.

[1]
http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=a9be04f504bafd9b30094d7c825ed43933b028af

Re: RFC: making the PVH 64bit ABI as stable

[Andrew Cooper]

On 02/06/15 16:11, Roger Pau Monné wrote:
> Hello,
>
> The document describing the PVH interface was committed 9 months ago
> [1], and since then there hasn't been any change regarding the
> interface. PVH is still missing features in order to have feature parity
> with pure PV, mainly:
>
>  - DomU miration support.
>  - PCI passthrough support.
>  - 32bit support.
>  - AMD support.
>
> AFAICT however none of these features are going to change the current
> ABI. PCI passthrough might expand it, by adding new hypercalls, but I
> don't think this should prevent us from marking the current ABI as
> stable. ARM for example doesn't have PCI passthrough yet or migration
> support, but the ABI has been marked as stable.
>
> To that end, I would like to request the 64bit PVH ABI to be marked as
> stable for DomUs. This is needed so external projects (like PVH support
> for grub2) can progress.

This would be exceedingly short sighted until 32bit and AMD support are
present.

Therefore, -1 from me, especially as I am not convinced that 32bit will
get in without ABI changes.

~Andrew


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: RFC: making the PVH 64bit ABI as stable

[Jan Beulich]

>>> On 02.06.15 at 17:11, <roger.pau@citrix.com> wrote:
> Hello,
> 
> The document describing the PVH interface was committed 9 months ago
> [1], and since then there hasn't been any change regarding the
> interface. PVH is still missing features in order to have feature parity
> with pure PV, mainly:
> 
>  - DomU miration support.
>  - PCI passthrough support.
>  - 32bit support.
>  - AMD support.
> 
> AFAICT however none of these features are going to change the current
> ABI.

This your guess; I don't think there's any guarantee. The more
that talk was about making PVH uniformly enter the kernel in 32-bit
mode.

> PCI passthrough might expand it, by adding new hypercalls, but I
> don't think this should prevent us from marking the current ABI as
> stable. ARM for example doesn't have PCI passthrough yet or migration
> support, but the ABI has been marked as stable.
> 
> To that end, I would like to request the 64bit PVH ABI to be marked as
> stable for DomUs. This is needed so external projects (like PVH support
> for grub2) can progress.

Understandable, but no, not before all the fixmes in the tree got
dealt with.

Jan

Re: RFC: making the PVH 64bit ABI as stableo

[Stefano Stabellini]

On Tue, 2 Jun 2015, Jan Beulich wrote:
> >>> On 02.06.15 at 17:11, <roger.pau@citrix.com> wrote:
> > Hello,
> > 
> > The document describing the PVH interface was committed 9 months ago
> > [1], and since then there hasn't been any change regarding the
> > interface. PVH is still missing features in order to have feature parity
> > with pure PV, mainly:
> > 
> >  - DomU miration support.
> >  - PCI passthrough support.
> >  - 32bit support.
> >  - AMD support.
> > 
> > AFAICT however none of these features are going to change the current
> > ABI.
> 
> This your guess; I don't think there's any guarantee.

Let's make it a guarantee.


> The more that talk was about making PVH uniformly enter the kernel in
> 32-bit mode.

What talk? IRL talks are irrelevant in this context. If it is not on the
list, it doesn't exist.


> > PCI passthrough might expand it, by adding new hypercalls, but I
> > don't think this should prevent us from marking the current ABI as
> > stable. ARM for example doesn't have PCI passthrough yet or migration
> > support, but the ABI has been marked as stable.
> > 
> > To that end, I would like to request the 64bit PVH ABI to be marked as
> > stable for DomUs. This is needed so external projects (like PVH support
> > for grub2) can progress.
> 
> Understandable, but no, not before all the fixmes in the tree got
> dealt with.

What is your timeline for that? In fact does anybody have any timelines
for it?

We need to have a clear idea of what exactly needs to happen. We also
need to have confidence that is going to happen in a reasonable time
frame. At the moment we have some various mumblings about things, but we
don't have a clear breakdown of the outstanding work and names
associated with each work item.

Is anybody going to volunteer writing that todo list?

Are we going to be able to find enough volunteers with the right skills
to be confident that PVH is going to be out of "experimental" within a
reasonable time frame? It is clear that some of the clean-ups require an
hypervisor expert.

If not, I suggest we rethink our priorities and we consider dropping PVH
entirely. I don't think is fair to expect Roger or anybody else to keep
their efforts up on PVH, when actually we don't know if we'll be able to
land it.

Maybe we could focus on improving PV on HVM and its security. Maybe we
could resurrect Intel's HVM Dom0 project
(http://events.linuxfoundation.org/sites/events/files/slides/HVM%20Dom0.pdf).
Think of how much farther we would be if we didn't start PVH in the
first place.


P.S.
This message is not addressed to Jan in particular but to the larger Xen
community.

Re: RFC: making the PVH 64bit ABI as stableo

[Andrew Cooper]

On 02/06/15 17:51, Stefano Stabellini wrote:
> On Tue, 2 Jun 2015, Jan Beulich wrote:
>>>>> On 02.06.15 at 17:11, <roger.pau@citrix.com> wrote:
>>> Hello,
>>>
>>> The document describing the PVH interface was committed 9 months ago
>>> [1], and since then there hasn't been any change regarding the
>>> interface. PVH is still missing features in order to have feature parity
>>> with pure PV, mainly:
>>>
>>>  - DomU miration support.
>>>  - PCI passthrough support.
>>>  - 32bit support.
>>>  - AMD support.
>>>
>>> AFAICT however none of these features are going to change the current
>>> ABI.
>> This your guess; I don't think there's any guarantee.
> Let's make it a guarantee.

No.  Lets fix it.

>
>> The more that talk was about making PVH uniformly enter the kernel in
>> 32-bit mode.
> What talk?

At the hackathon.

>  IRL talks are irrelevant in this context. If it is not on the
> list, it doesn't exist.
>
>
>>> PCI passthrough might expand it, by adding new hypercalls, but I
>>> don't think this should prevent us from marking the current ABI as
>>> stable. ARM for example doesn't have PCI passthrough yet or migration
>>> support, but the ABI has been marked as stable.
>>>
>>> To that end, I would like to request the 64bit PVH ABI to be marked as
>>> stable for DomUs. This is needed so external projects (like PVH support
>>> for grub2) can progress.
>> Understandable, but no, not before all the fixmes in the tree got
>> dealt with.
> What is your timeline for that? In fact does anybody have any timelines
> for it?
>
> We need to have a clear idea of what exactly needs to happen. We also
> need to have confidence that is going to happen in a reasonable time
> frame. At the moment we have some various mumblings about things, but we
> don't have a clear breakdown of the outstanding work and names
> associated with each work item.
>
> Is anybody going to volunteer writing that todo list?
>
> Are we going to be able to find enough volunteers with the right skills
> to be confident that PVH is going to be out of "experimental" within a
> reasonable time frame? It is clear that some of the clean-ups require an
> hypervisor expert.
>
> If not, I suggest we rethink our priorities and we consider dropping PVH
> entirely. I don't think is fair to expect Roger or anybody else to keep
> their efforts up on PVH, when actually we don't know if we'll be able to
> land it.
>
> Maybe we could focus on improving PV on HVM and its security. Maybe we
> could resurrect Intel's HVM Dom0 project
> (http://events.linuxfoundation.org/sites/events/files/slides/HVM%20Dom0.pdf).
> Think of how much farther we would be if we didn't start PVH in the
> first place.
>
>
> P.S.
> This message is not addressed to Jan in particular but to the larger Xen
> community.

Boris: sorry if this comes as news to you, but you were volunteered in
absentia by your manager at the hackathon ;)

With my x86 maintainer hat on, the following is an absolute minimum set
of prerequisite for PVH.

* 32bit support
* AMD support
* Removal of all /* TODO pvhfixme */ from the code

OTOH, I do not see PCI Passthough or Migration as blockers to tech preview.

All three of these issues can be addressed by ceasing to think of PVH as
PV + hardware virt, and by thinking of PVH as HVM without qemu.  At this
point, PVH and HVM are basically identical from Xens point of view.

It is my strong oppinion that all PVH guests should start in 32bit flat
mode (curiously the same as a multiboot entry, hint hint, which I expect
all mainstream kernels to already have entry points for) at which point
they can do whatever they like 32/64bit wise and pagetable wise, and we
are not in the same position as we are with current PV guests where we
need to know the bitness of them ahead of domain construction.

~Andrew

Re: RFC: making the PVH 64bit ABI as stableo

[Boris Ostrovsky]

On 06/02/2015 12:51 PM, Stefano Stabellini wrote:
> On Tue, 2 Jun 2015, Jan Beulich wrote:
>>>>> On 02.06.15 at 17:11, <roger.pau@citrix.com> wrote:
>>> Hello,
>>>
>>> The document describing the PVH interface was committed 9 months ago
>>> [1], and since then there hasn't been any change regarding the
>>> interface. PVH is still missing features in order to have feature parity
>>> with pure PV, mainly:
>>>
>>>   - DomU miration support.
>>>   - PCI passthrough support.
>>>   - 32bit support.
>>>   - AMD support.
>>>
>>> AFAICT however none of these features are going to change the current
>>> ABI.
>>
>> This your guess; I don't think there's any guarantee.
>
> Let's make it a guarantee.
>
>
>> The more that talk was about making PVH uniformly enter the kernel in
>> 32-bit mode.
>
> What talk? IRL talks are irrelevant in this context. If it is not on the
> list, it doesn't exist.
>
>
>>> PCI passthrough might expand it, by adding new hypercalls, but I
>>> don't think this should prevent us from marking the current ABI as
>>> stable. ARM for example doesn't have PCI passthrough yet or migration
>>> support, but the ABI has been marked as stable.
>>>
>>> To that end, I would like to request the 64bit PVH ABI to be marked as
>>> stable for DomUs. This is needed so external projects (like PVH support
>>> for grub2) can progress.
>>
>> Understandable, but no, not before all the fixmes in the tree got
>> dealt with.
>
> What is your timeline for that? In fact does anybody have any timelines
> for it?
>
> We need to have a clear idea of what exactly needs to happen. We also
> need to have confidence that is going to happen in a reasonable time
> frame. At the moment we have some various mumblings about things, but we
> don't have a clear breakdown of the outstanding work and names
> associated with each work item.
>
> Is anybody going to volunteer writing that todo list?
>
> Are we going to be able to find enough volunteers with the right skills
> to be confident that PVH is going to be out of "experimental" within a
> reasonable time frame? It is clear that some of the clean-ups require an
> hypervisor expert.
>
> If not, I suggest we rethink our priorities and we consider dropping PVH
> entirely. I don't think is fair to expect Roger or anybody else to keep
> their efforts up on PVH, when actually we don't know if we'll be able to
> land it.



Roger, Tim, Elena, Konrad and I had a conversation a few months ago and 
at that time we came up with a (somewhat informal) list of what we knew 
was broken:

   - 32-bit cannot boot.
   - Does not work under AMD.
   - Migration
   - PCI passthrough
   - Memory ballooning
   - Multiple VBDs, NICs, etc.
   - CPUID filtering. There are no filtering done at all which means
that certain cpuid flags are exposed to the guest.
   - The x2apic will cause a crash if the NMI handler is invoked.
   - The APERF will cause inferior scheduling decisions.
   - working with shadow code (which is what we use when migrating HVM
guests). But the nice side-benefit is that we can then run PVH on
machines without VMX or SVM support.
   - TSC modes are broken.

Obviously some things in this list are large(er) projects and some are 
simply bugs.

I picked 32-bit support, Elena is looking into AMD and Roger agreed to 
look at migration and passthrough for now. Plus, at some point we will
probably need to think about how to move PVH support into a 
"feature-flag" support that Tim proposed at a hackathon last year (where 
we don't have HVM/PV/PVH guests but rather guests with various features 
enabled).

(Incidentally, I booted a 32-bit PVH guest yesterday finally. UP only 
for now)

So there is a more than one person working on it (for a specific 
definition of the word "working" since we all are constantly preempted 
by other things that also preemptable by more important things).

-boris


>
> Maybe we could focus on improving PV on HVM and its security. Maybe we
> could resurrect Intel's HVM Dom0 project
> (http://events.linuxfoundation.org/sites/events/files/slides/HVM%20Dom0.pdf).
> Think of how much farther we would be if we didn't start PVH in the
> first place.
>
>
> P.S.
> This message is not addressed to Jan in particular but to the larger Xen
> community.
>

Re: RFC: making the PVH 64bit ABI as stableo

[Jan Beulich]

>>> On 02.06.15 at 19:12, <boris.ostrovsky@oracle.com> wrote:
>    - working with shadow code (which is what we use when migrating HVM
> guests). But the nice side-benefit is that we can then run PVH on
> machines without VMX or SVM support.

Without VMX/SVM? What's the H then in PVH? Or did you mean
without EPT/NPT?

Jan

Re: RFC: making the PVH 64bit ABI as stableo

[Ian Campbell]

On Tue, 2015-06-02 at 18:09 +0100, Andrew Cooper wrote:
> * Removal of all /* TODO pvhfixme */ from the code

I guess this is not the literal tag:
ianc@cosworth:xen.git$ git grep -i TODO.pvhfixme xen
ianc@cosworth:xen.git$ git grep -i TODO.*pvh xen
ianc@cosworth:xen.git$

I think you meant:
ianc@cosworth:xen.git$ git grep -i pvh.fixme xen
xen/arch/x86/domain_build.c: * pvh fixme: The following doesn't map MMIO ranges when they sit above the
xen/arch/x86/domain_build.c:     * PVH Fixme: XENFEAT_supervisor_mode_kernel has been reused in PVH with a
xen/arch/x86/hvm/vmx/vmx.c:            if ( unlikely(is_pvh_vcpu(v)) /* PVH fixme */ ||
xen/arch/x86/mm/p2m-pt.c:        /* pvh fixme: foreign types are only supported on ept at present */
xen/arch/x86/mm/p2m.c: * pvh fixme: when adding support for pvh non-hardware domains, this path must
xen/arch/x86/mm/p2m.c:     * pvh fixme: until support is added to p2m teardown code to cleanup any
xen/arch/x86/time.c:         * PVH fixme: support more tsc modes.
xen/arch/x86/traps.c:    if ( !d || !d->vcpu || !d->vcpu[0] || !is_pv_domain(d) /* PVH fixme */ )
xen/common/vm_event.c:            /* pvh fixme: p2m_is_foreign types need addressing */
xen/common/vm_event.c:            /* pvh fixme: p2m_is_foreign types need addressing */

Are there other variants or is this the list?

Re: RFC: making the PVH 64bit ABI as stableo

[Ian Campbell]

On Tue, 2015-06-02 at 13:12 -0400, Boris Ostrovsky wrote:
> On 06/02/2015 12:51 PM, Stefano Stabellini wrote:
> > On Tue, 2 Jun 2015, Jan Beulich wrote:
> >>>>> On 02.06.15 at 17:11, <roger.pau@citrix.com> wrote:
> >>> Hello,
> >>>
> >>> The document describing the PVH interface was committed 9 months ago
> >>> [1], and since then there hasn't been any change regarding the
> >>> interface. PVH is still missing features in order to have feature parity
> >>> with pure PV, mainly:
> >>>
> >>>   - DomU miration support.
> >>>   - PCI passthrough support.
> >>>   - 32bit support.
> >>>   - AMD support.
> >>>
> >>> AFAICT however none of these features are going to change the current
> >>> ABI.
> >>
> >> This your guess; I don't think there's any guarantee.
> >
> > Let's make it a guarantee.
> >
> >
> >> The more that talk was about making PVH uniformly enter the kernel in
> >> 32-bit mode.
> >
> > What talk? IRL talks are irrelevant in this context. If it is not on the
> > list, it doesn't exist.
> >
> >
> >>> PCI passthrough might expand it, by adding new hypercalls, but I
> >>> don't think this should prevent us from marking the current ABI as
> >>> stable. ARM for example doesn't have PCI passthrough yet or migration
> >>> support, but the ABI has been marked as stable.
> >>>
> >>> To that end, I would like to request the 64bit PVH ABI to be marked as
> >>> stable for DomUs. This is needed so external projects (like PVH support
> >>> for grub2) can progress.
> >>
> >> Understandable, but no, not before all the fixmes in the tree got
> >> dealt with.
> >
> > What is your timeline for that? In fact does anybody have any timelines
> > for it?
> >
> > We need to have a clear idea of what exactly needs to happen. We also
> > need to have confidence that is going to happen in a reasonable time
> > frame. At the moment we have some various mumblings about things, but we
> > don't have a clear breakdown of the outstanding work and names
> > associated with each work item.
> >
> > Is anybody going to volunteer writing that todo list?
> >
> > Are we going to be able to find enough volunteers with the right skills
> > to be confident that PVH is going to be out of "experimental" within a
> > reasonable time frame? It is clear that some of the clean-ups require an
> > hypervisor expert.
> >
> > If not, I suggest we rethink our priorities and we consider dropping PVH
> > entirely. I don't think is fair to expect Roger or anybody else to keep
> > their efforts up on PVH, when actually we don't know if we'll be able to
> > land it.
> 
> 
> 
> Roger, Tim, Elena, Konrad and I had a conversation a few months ago and 
> at that time we came up with a (somewhat informal) list of what we knew 
> was broken:
> 
>    - 32-bit cannot boot.
>    - Does not work under AMD.
>    - Migration
>    - PCI passthrough
>    - Memory ballooning
>    - Multiple VBDs, NICs, etc.
>    - CPUID filtering. There are no filtering done at all which means
> that certain cpuid flags are exposed to the guest.
>    - The x2apic will cause a crash if the NMI handler is invoked.
>    - The APERF will cause inferior scheduling decisions.
>    - working with shadow code (which is what we use when migrating HVM
> guests). But the nice side-benefit is that we can then run PVH on
> machines without VMX or SVM support.
>    - TSC modes are broken.

This is missing at least:

        - Remove all /* pvh fixme */ from the code

What I'm hearing from the x86 maintainers is that this is actually a
high priority and not a "nice to have cleanup".

> I picked 32-bit support, Elena is looking into AMD

With the TODOs + these 2 being the things which the x86 maintainers have
highlighted in this thread as being most critical for marking the ABI as
stable (or at least moving experimental->tech preview) let me ask
explicotly:

        What are the current time frames on these two items?

Ian.

Re: RFC: making the PVH 64bit ABI as stableo

[Andrew Cooper]

On 03/06/15 10:00, Ian Campbell wrote:
> On Tue, 2015-06-02 at 18:09 +0100, Andrew Cooper wrote:
>> * Removal of all /* TODO pvhfixme */ from the code
> I guess this is not the literal tag:
> ianc@cosworth:xen.git$ git grep -i TODO.pvhfixme xen
> ianc@cosworth:xen.git$ git grep -i TODO.*pvh xen
> ianc@cosworth:xen.git$
>
> I think you meant:
> ianc@cosworth:xen.git$ git grep -i pvh.fixme xen
> xen/arch/x86/domain_build.c: * pvh fixme: The following doesn't map MMIO ranges when they sit above the
> xen/arch/x86/domain_build.c:     * PVH Fixme: XENFEAT_supervisor_mode_kernel has been reused in PVH with a
> xen/arch/x86/hvm/vmx/vmx.c:            if ( unlikely(is_pvh_vcpu(v)) /* PVH fixme */ ||
> xen/arch/x86/mm/p2m-pt.c:        /* pvh fixme: foreign types are only supported on ept at present */
> xen/arch/x86/mm/p2m.c: * pvh fixme: when adding support for pvh non-hardware domains, this path must
> xen/arch/x86/mm/p2m.c:     * pvh fixme: until support is added to p2m teardown code to cleanup any
> xen/arch/x86/time.c:         * PVH fixme: support more tsc modes.
> xen/arch/x86/traps.c:    if ( !d || !d->vcpu || !d->vcpu[0] || !is_pv_domain(d) /* PVH fixme */ )
> xen/common/vm_event.c:            /* pvh fixme: p2m_is_foreign types need addressing */
> xen/common/vm_event.c:            /* pvh fixme: p2m_is_foreign types need addressing */
>
> Are there other variants or is this the list?
>
>

At least one more comes to mind.

andrewcoop@andrewcoop:/local/xen.git/xen$ git grep -i "PVH 32bitfixme"
arch/x86/domain.c:768:        /* PVH 32bitfixme */
arch/x86/hvm/hvm.c:2318:        v->arch.hvm_vcpu.hcall_64bit = 1;    /*
PVH 32bitfixme. */
arch/x86/hvm/hvm.c:4898:/* PVH 32bitfixme. */
arch/x86/hvm/hvm.c:5003:        regs->_eax = -ENOSYS; /* PVH 32bitfixme. */
arch/x86/hvm/vmx/vmcs.c:987:        /* Start in 64-bit mode. PVH
32bitfixme. */
arch/x86/hvm/vmx/vmcs.c:1128:        /* CS.L == 1, exec, read/write,
accessed. PVH 32bitfixme. */

~Andrew

Re: RFC: making the PVH 64bit ABI as stableo

[Ian Campbell]

On Wed, 2015-06-03 at 10:06 +0100, Andrew Cooper wrote:
> On 03/06/15 10:00, Ian Campbell wrote:
> > On Tue, 2015-06-02 at 18:09 +0100, Andrew Cooper wrote:
> >> * Removal of all /* TODO pvhfixme */ from the code
> > I guess this is not the literal tag:
> > ianc@cosworth:xen.git$ git grep -i TODO.pvhfixme xen
> > ianc@cosworth:xen.git$ git grep -i TODO.*pvh xen
> > ianc@cosworth:xen.git$
> >
> > I think you meant:
> > ianc@cosworth:xen.git$ git grep -i pvh.fixme xen
> > xen/arch/x86/domain_build.c: * pvh fixme: The following doesn't map MMIO ranges when they sit above the
> > xen/arch/x86/domain_build.c:     * PVH Fixme: XENFEAT_supervisor_mode_kernel has been reused in PVH with a
> > xen/arch/x86/hvm/vmx/vmx.c:            if ( unlikely(is_pvh_vcpu(v)) /* PVH fixme */ ||
> > xen/arch/x86/mm/p2m-pt.c:        /* pvh fixme: foreign types are only supported on ept at present */
> > xen/arch/x86/mm/p2m.c: * pvh fixme: when adding support for pvh non-hardware domains, this path must
> > xen/arch/x86/mm/p2m.c:     * pvh fixme: until support is added to p2m teardown code to cleanup any
> > xen/arch/x86/time.c:         * PVH fixme: support more tsc modes.
> > xen/arch/x86/traps.c:    if ( !d || !d->vcpu || !d->vcpu[0] || !is_pv_domain(d) /* PVH fixme */ )
> > xen/common/vm_event.c:            /* pvh fixme: p2m_is_foreign types need addressing */
> > xen/common/vm_event.c:            /* pvh fixme: p2m_is_foreign types need addressing */
> >
> > Are there other variants or is this the list?
> >
> >
> 
> At least one more comes to mind.
> 
> andrewcoop@andrewcoop:/local/xen.git/xen$ git grep -i "PVH 32bitfixme"
> arch/x86/domain.c:768:        /* PVH 32bitfixme */
> arch/x86/hvm/hvm.c:2318:        v->arch.hvm_vcpu.hcall_64bit = 1;    /*
> PVH 32bitfixme. */
> arch/x86/hvm/hvm.c:4898:/* PVH 32bitfixme. */
> arch/x86/hvm/hvm.c:5003:        regs->_eax = -ENOSYS; /* PVH 32bitfixme. */
> arch/x86/hvm/vmx/vmcs.c:987:        /* Start in 64-bit mode. PVH
> 32bitfixme. */
> arch/x86/hvm/vmx/vmcs.c:1128:        /* CS.L == 1, exec, read/write,
> accessed. PVH 32bitfixme. */

OK, so "git grep -i pvh.*fixme" (with an extra *) is closer to the
canonical list, thanks.

(and it seems that at least some of these come other under feature work
anyway, like 32 bit or amd support).

Ian.

Re: RFC: making the PVH 64bit ABI as stableo

[Stefano Stabellini]

On Tue, 2 Jun 2015, Andrew Cooper wrote:
> With my x86 maintainer hat on, the following is an absolute minimum set
> of prerequisite for PVH.
> 
> * 32bit support

Could you please explain why 32bit is important to get PVH out of tech
preview? I don't see 32 bit OSes as an important use case. Maybe there
is more behind it that I cannot see.


> * AMD support

Given that it is pretty difficult for any AMD related changes to affect
the interface, why is this item in the list of things needed to get out
of experimental?


> * Removal of all /* TODO pvhfixme */ from the code
> 
> OTOH, I do not see PCI Passthough or Migration as blockers to tech preview.
> 
> All three of these issues can be addressed by ceasing to think of PVH as
> PV + hardware virt, and by thinking of PVH as HVM without qemu.  At this
> point, PVH and HVM are basically identical from Xens point of view.
> 
> It is my strong oppinion that all PVH guests should start in 32bit flat
> mode (curiously the same as a multiboot entry, hint hint, which I expect
> all mainstream kernels to already have entry points for) at which point
> they can do whatever they like 32/64bit wise and pagetable wise, and we
> are not in the same position as we are with current PV guests where we
> need to know the bitness of them ahead of domain construction.
> 
> ~Andrew
> 

Re: RFC: making the PVH 64bit ABI as stableo

[Jan Beulich]

>>> On 03.06.15 at 12:02, <stefano.stabellini@eu.citrix.com> wrote:
> On Tue, 2 Jun 2015, Andrew Cooper wrote:
>> With my x86 maintainer hat on, the following is an absolute minimum set
>> of prerequisite for PVH.
>> 
>> * 32bit support
> 
> Could you please explain why 32bit is important to get PVH out of tech
> preview? I don't see 32 bit OSes as an important use case. Maybe there
> is more behind it that I cannot see.

The primary reason was named before: 32-bit support will likely
end up changing the way 64-bit guests get launched.

>> * AMD support
> 
> Given that it is pretty difficult for any AMD related changes to affect
> the interface, why is this item in the list of things needed to get out
> of experimental?

Because we cannot rule out that adding SVM support may require
interface changes. And even if we promoted it to tech preview,
the interface would still not be frozen.

Apart from that (and I also said so on the hackathon session, just
like you considering to rip the whole thing out if it doesn't make
any progress) already during initial patch review I was hesitant to
accept changes with so many loose ends; I accepted them on the
basis that they would get dealt with in a timely manner, which
didn't happen, and which leaves us in this sad, half working (or
should I say half broken) state.

Jan

Re: RFC: making the PVH 64bit ABI as stableo

[Stefano Stabellini]

On Wed, 3 Jun 2015, Jan Beulich wrote:
> >>> On 03.06.15 at 12:02, <stefano.stabellini@eu.citrix.com> wrote:
> > On Tue, 2 Jun 2015, Andrew Cooper wrote:
> >> With my x86 maintainer hat on, the following is an absolute minimum set
> >> of prerequisite for PVH.
> >> 
> >> * 32bit support
> > 
> > Could you please explain why 32bit is important to get PVH out of tech
> > preview? I don't see 32 bit OSes as an important use case. Maybe there
> > is more behind it that I cannot see.
> 
> The primary reason was named before: 32-bit support will likely
> end up changing the way 64-bit guests get launched.

I would rather have 64-bit only PVH, then no PVH. No PVH is what we have
today.


> >> * AMD support
> > 
> > Given that it is pretty difficult for any AMD related changes to affect
> > the interface, why is this item in the list of things needed to get out
> > of experimental?
> 
> Because we cannot rule out that adding SVM support may require
> interface changes. And even if we promoted it to tech preview,
> the interface would still not be frozen.

Without a "frozen" interface, the thing is unusable. I would rather have
an Intel only, DomU only, 64-bit only PVH, than no PVH at all, that is
the current state of things, which is unlikely to change in the short
and medium terms.

Nobody replied with a timeline for these two features yet, but from the
look of things, if we maintain the current pace, I would guess we are
looking at late 2017. In my option that is too late and would rather rip
PVH off.


> Apart from that (and I also said so on the hackathon session, just
> like you considering to rip the whole thing out if it doesn't make
> any progress) already during initial patch review I was hesitant to
> accept changes with so many loose ends; I accepted them on the
> basis that they would get dealt with in a timely manner, which
> didn't happen, and which leaves us in this sad, half working (or
> should I say half broken) state.

I completely agree.

Also it would be nice if any discussions at the hackathon, or anywhere
else, were written down and sent to the list. Otherwise people like me,
that didn't attend, will just think of them as rumors.

Re: RFC: making the PVH 64bit ABI as stableo

[Boris Ostrovsky]

On 06/03/2015 02:09 AM, Jan Beulich wrote:
>>>> On 02.06.15 at 19:12, <boris.ostrovsky@oracle.com> wrote:
>>     - working with shadow code (which is what we use when migrating HVM
>> guests). But the nice side-benefit is that we can then run PVH on
>> machines without VMX or SVM support.
> Without VMX/SVM? What's the H then in PVH? Or did you mean
> without EPT/NPT?


Yes, this must have been EPT/NPT (I don't remember where this item came 
from, TBH)

-boris

Re: RFC: making the PVH 64bit ABI as stableo

[Boris Ostrovsky]

On 06/03/2015 05:03 AM, Ian Campbell wrote:
> On Tue, 2015-06-02 at 13:12 -0400, Boris Ostrovsky wrote:
>> On 06/02/2015 12:51 PM, Stefano Stabellini wrote:
>>> On Tue, 2 Jun 2015, Jan Beulich wrote:
>>>>>>> On 02.06.15 at 17:11, <roger.pau@citrix.com> wrote:
>>>>> Hello,
>>>>>
>>>>> The document describing the PVH interface was committed 9 months ago
>>>>> [1], and since then there hasn't been any change regarding the
>>>>> interface. PVH is still missing features in order to have feature parity
>>>>> with pure PV, mainly:
>>>>>
>>>>>    - DomU miration support.
>>>>>    - PCI passthrough support.
>>>>>    - 32bit support.
>>>>>    - AMD support.
>>>>>
>>>>> AFAICT however none of these features are going to change the current
>>>>> ABI.
>>>> This your guess; I don't think there's any guarantee.
>>> Let's make it a guarantee.
>>>
>>>
>>>> The more that talk was about making PVH uniformly enter the kernel in
>>>> 32-bit mode.
>>> What talk? IRL talks are irrelevant in this context. If it is not on the
>>> list, it doesn't exist.
>>>
>>>
>>>>> PCI passthrough might expand it, by adding new hypercalls, but I
>>>>> don't think this should prevent us from marking the current ABI as
>>>>> stable. ARM for example doesn't have PCI passthrough yet or migration
>>>>> support, but the ABI has been marked as stable.
>>>>>
>>>>> To that end, I would like to request the 64bit PVH ABI to be marked as
>>>>> stable for DomUs. This is needed so external projects (like PVH support
>>>>> for grub2) can progress.
>>>> Understandable, but no, not before all the fixmes in the tree got
>>>> dealt with.
>>> What is your timeline for that? In fact does anybody have any timelines
>>> for it?
>>>
>>> We need to have a clear idea of what exactly needs to happen. We also
>>> need to have confidence that is going to happen in a reasonable time
>>> frame. At the moment we have some various mumblings about things, but we
>>> don't have a clear breakdown of the outstanding work and names
>>> associated with each work item.
>>>
>>> Is anybody going to volunteer writing that todo list?
>>>
>>> Are we going to be able to find enough volunteers with the right skills
>>> to be confident that PVH is going to be out of "experimental" within a
>>> reasonable time frame? It is clear that some of the clean-ups require an
>>> hypervisor expert.
>>>
>>> If not, I suggest we rethink our priorities and we consider dropping PVH
>>> entirely. I don't think is fair to expect Roger or anybody else to keep
>>> their efforts up on PVH, when actually we don't know if we'll be able to
>>> land it.
>>
>>
>> Roger, Tim, Elena, Konrad and I had a conversation a few months ago and
>> at that time we came up with a (somewhat informal) list of what we knew
>> was broken:
>>
>>     - 32-bit cannot boot.
>>     - Does not work under AMD.
>>     - Migration
>>     - PCI passthrough
>>     - Memory ballooning
>>     - Multiple VBDs, NICs, etc.
>>     - CPUID filtering. There are no filtering done at all which means
>> that certain cpuid flags are exposed to the guest.
>>     - The x2apic will cause a crash if the NMI handler is invoked.
>>     - The APERF will cause inferior scheduling decisions.
>>     - working with shadow code (which is what we use when migrating HVM
>> guests). But the nice side-benefit is that we can then run PVH on
>> machines without VMX or SVM support.
>>     - TSC modes are broken.
> This is missing at least:
>
>          - Remove all /* pvh fixme */ from the code

I considered those would be gone as part of working out the item on the 
list above (I know that many, if not most, of them are for 32-bit guests 
and I removed them in my semi-working tree)

>
> What I'm hearing from the x86 maintainers is that this is actually a
> high priority and not a "nice to have cleanup".
>
>> I picked 32-bit support, Elena is looking into AMD
> With the TODOs + these 2 being the things which the x86 maintainers have
> highlighted in this thread as being most critical for marking the ABI as
> stable (or at least moving experimental->tech preview) let me ask
> explicotly:
>
>          What are the current time frames on these two items?

For 32-bit support, just to get it to work in the within current 
framework I think can be done for 4.7 release (which is late this year 
IIRC).

I can't tell you how much it will take to make it a part of a "unified" 
32/64-bit guest launching as I haven't looked at this at all yet.

-boris

Re: RFC: making the PVH 64bit ABI as stableo

[Boris Ostrovsky]

On 06/03/2015 09:11 AM, Stefano Stabellini wrote:
> Also it would be nice if any discussions at the hackathon, or anywhere 
> else, were written down and sent to the list. Otherwise people like 
> me, that didn't attend, will just think of them as rumors. 


Here is the closest that I could find. This is about last year's 
hackathon, not from last month (which I also didn't attend) if this is 
what you are after):

http://lists.xen.org/archives/html/xen-devel/2014-12/msg00596.html

-boris

Re: RFC: making the PVH 64bit ABI as stableo

[Roger Pau Monné]

El 03/06/15 a les 14.08, Jan Beulich ha escrit:
>>>> On 03.06.15 at 12:02, <stefano.stabellini@eu.citrix.com> wrote:
>> On Tue, 2 Jun 2015, Andrew Cooper wrote:
>>> With my x86 maintainer hat on, the following is an absolute minimum set
>>> of prerequisite for PVH.
>>>
>>> * 32bit support
>>
>> Could you please explain why 32bit is important to get PVH out of tech
>> preview? I don't see 32 bit OSes as an important use case. Maybe there
>> is more behind it that I cannot see.
> 
> The primary reason was named before: 32-bit support will likely
> end up changing the way 64-bit guests get launched.

I can work on the new boot ABI, even if it's just a design document now,
but the actual implementation needs to be done on top of the 32-bit
support series.

Boris, do you think you could send an early RFC of your 32-bit support
series in a couple of weeks at most?

Roger.

Re: RFC: making the PVH 64bit ABI as stableo

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1134 bytes --]

On Fri, 5 Jun 2015, Roger Pau Monné wrote:
> El 03/06/15 a les 14.08, Jan Beulich ha escrit:
> >>>> On 03.06.15 at 12:02, <stefano.stabellini@eu.citrix.com> wrote:
> >> On Tue, 2 Jun 2015, Andrew Cooper wrote:
> >>> With my x86 maintainer hat on, the following is an absolute minimum set
> >>> of prerequisite for PVH.
> >>>
> >>> * 32bit support
> >>
> >> Could you please explain why 32bit is important to get PVH out of tech
> >> preview? I don't see 32 bit OSes as an important use case. Maybe there
> >> is more behind it that I cannot see.
> >
> > The primary reason was named before: 32-bit support will likely
> > end up changing the way 64-bit guests get launched.
>
> I can work on the new boot ABI, even if it's just a design document now,
> but the actual implementation needs to be done on top of the 32-bit
> support series.
>
> Boris, do you think you could send an early RFC of your 32-bit support
> series in a couple of weeks at most?

Besides these, we still don't have any timelines for AMD support, and
nobody stepping up to fix any remaining /* TODO pvh fixme */ in the code
afterwards.

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: RFC: making the PVH 64bit ABI as stableo

[Ian Campbell]

On Wed, 2015-06-03 at 09:35 -0400, Boris Ostrovsky wrote:
> > What I'm hearing from the x86 maintainers is that this is actually a
> > high priority and not a "nice to have cleanup".
> >
> >> I picked 32-bit support, Elena is looking into AMD
> > With the TODOs + these 2 being the things which the x86 maintainers have
> > highlighted in this thread as being most critical for marking the ABI as
> > stable (or at least moving experimental->tech preview) let me ask
> > explicotly:
> >
> >          What are the current time frames on these two items?
> 
> For 32-bit support, just to get it to work in the within current 
> framework I think can be done for 4.7 release (which is late this year 
> IIRC).
> 
> I can't tell you how much it will take to make it a part of a "unified" 
> 32/64-bit guest launching as I haven't looked at this at all yet.

Thanks. What about AMD support then? Elena?

Ian.

Re: RFC: making the PVH 64bit ABI as stableo

[Boris Ostrovsky]

On 06/05/2015 12:16 PM, Roger Pau Monné wrote:
> El 03/06/15 a les 14.08, Jan Beulich ha escrit:
>>>>> On 03.06.15 at 12:02, <stefano.stabellini@eu.citrix.com> wrote:
>>> On Tue, 2 Jun 2015, Andrew Cooper wrote:
>>>> With my x86 maintainer hat on, the following is an absolute minimum set
>>>> of prerequisite for PVH.
>>>>
>>>> * 32bit support
>>> Could you please explain why 32bit is important to get PVH out of tech
>>> preview? I don't see 32 bit OSes as an important use case. Maybe there
>>> is more behind it that I cannot see.
>> The primary reason was named before: 32-bit support will likely
>> end up changing the way 64-bit guests get launched.
> I can work on the new boot ABI, even if it's just a design document now,
> but the actual implementation needs to be done on top of the 32-bit
> support series.
>
> Boris, do you think you could send an early RFC of your 32-bit support
> series in a couple of weeks at most?

That's highly unlikely. For one, I am still unable to boot MP guests. In 
addition, it is all held together by rubber bands and matchsticks so 
calling it an RFC would be an insult to RFCs. (for example, I apparently 
broke HVM somewhere along the way).


-boris

Re: RFC: making the PVH 64bit ABI as stableo

[Boris Ostrovsky]

On 06/05/2015 12:21 PM, Stefano Stabellini wrote:
> On Fri, 5 Jun 2015, Roger Pau Monné wrote:
>> El 03/06/15 a les 14.08, Jan Beulich ha escrit:
>>>>>> On 03.06.15 at 12:02, <stefano.stabellini@eu.citrix.com> wrote:
>>>> On Tue, 2 Jun 2015, Andrew Cooper wrote:
>>>>> With my x86 maintainer hat on, the following is an absolute minimum set
>>>>> of prerequisite for PVH.
>>>>>
>>>>> * 32bit support
>>>> Could you please explain why 32bit is important to get PVH out of tech
>>>> preview? I don't see 32 bit OSes as an important use case. Maybe there
>>>> is more behind it that I cannot see.
>>> The primary reason was named before: 32-bit support will likely
>>> end up changing the way 64-bit guests get launched.
>> I can work on the new boot ABI, even if it's just a design document now,
>> but the actual implementation needs to be done on top of the 32-bit
>> support series.
>>
>> Boris, do you think you could send an early RFC of your 32-bit support
>> series in a couple of weeks at most?
> Besides these, we still don't have any timelines for AMD support, and
> nobody stepping up to fix any remaining /* TODO pvh fixme */ in the code
> afterwards.

Most of "fixme"s should be taken care of as part of 32-bit and AMD support.

I will look at whatever is left (some of them IIRC are related to 
migration and it sounds like those don't *have* to be fixed right away).

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: RFC: making the PVH 64bit ABI as stableo

[Andrew Cooper]

On 05/06/15 17:43, Boris Ostrovsky wrote:
> On 06/05/2015 12:16 PM, Roger Pau Monné wrote:
>> El 03/06/15 a les 14.08, Jan Beulich ha escrit:
>>>>>> On 03.06.15 at 12:02, <stefano.stabellini@eu.citrix.com> wrote:
>>>> On Tue, 2 Jun 2015, Andrew Cooper wrote:
>>>>> With my x86 maintainer hat on, the following is an absolute
>>>>> minimum set
>>>>> of prerequisite for PVH.
>>>>>
>>>>> * 32bit support
>>>> Could you please explain why 32bit is important to get PVH out of tech
>>>> preview? I don't see 32 bit OSes as an important use case. Maybe there
>>>> is more behind it that I cannot see.
>>> The primary reason was named before: 32-bit support will likely
>>> end up changing the way 64-bit guests get launched.
>> I can work on the new boot ABI, even if it's just a design document now,
>> but the actual implementation needs to be done on top of the 32-bit
>> support series.
>>
>> Boris, do you think you could send an early RFC of your 32-bit support
>> series in a couple of weeks at most?
>
> That's highly unlikely. For one, I am still unable to boot MP guests.
> In addition, it is all held together by rubber bands and matchsticks
> so calling it an RFC would be an insult to RFCs. (for example, I
> apparently broke HVM somewhere along the way).

How about working it the other way around.

Start with an HVM guest and start with a sane method of booting.  I
highly suggest multiboot1 as it is very easy and we have most of the
code already.  Whomever actually gets around to doing this gets leeway,
subject to it being sane (which the current method very certainly is not).

Start the domain without qemu, and expose some of the PV hypercalls to
HVM guests, and see how far it gets.  One will find suddenly that all
32bit and AMD problems have already been solved.  All the PV(h) kernel
needs to know is that there is no real hardware, and not to touch it.

~Andrew

Re: RFC: making the PVH 64bit ABI as stableo

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 2334 bytes --]

On Fri, 5 Jun 2015, Andrew Cooper wrote:
> On 05/06/15 17:43, Boris Ostrovsky wrote:
> > On 06/05/2015 12:16 PM, Roger Pau Monné wrote:
> >> El 03/06/15 a les 14.08, Jan Beulich ha escrit:
> >>>>>> On 03.06.15 at 12:02, <stefano.stabellini@eu.citrix.com> wrote:
> >>>> On Tue, 2 Jun 2015, Andrew Cooper wrote:
> >>>>> With my x86 maintainer hat on, the following is an absolute
> >>>>> minimum set
> >>>>> of prerequisite for PVH.
> >>>>>
> >>>>> * 32bit support
> >>>> Could you please explain why 32bit is important to get PVH out of tech
> >>>> preview? I don't see 32 bit OSes as an important use case. Maybe there
> >>>> is more behind it that I cannot see.
> >>> The primary reason was named before: 32-bit support will likely
> >>> end up changing the way 64-bit guests get launched.
> >> I can work on the new boot ABI, even if it's just a design document now,
> >> but the actual implementation needs to be done on top of the 32-bit
> >> support series.
> >>
> >> Boris, do you think you could send an early RFC of your 32-bit support
> >> series in a couple of weeks at most?
> >
> > That's highly unlikely. For one, I am still unable to boot MP guests.
> > In addition, it is all held together by rubber bands and matchsticks
> > so calling it an RFC would be an insult to RFCs. (for example, I
> > apparently broke HVM somewhere along the way).
>
> How about working it the other way around.
>
> Start with an HVM guest and start with a sane method of booting.  I
> highly suggest multiboot1 as it is very easy and we have most of the
> code already.  Whomever actually gets around to doing this gets leeway,
> subject to it being sane (which the current method very certainly is not).
>
> Start the domain without qemu, and expose some of the PV hypercalls to
> HVM guests, and see how far it gets.  One will find suddenly that all
> 32bit and AMD problems have already been solved.  All the PV(h) kernel
> needs to know is that there is no real hardware, and not to touch it.

This seems like a clean and nice way forward, but rather than PVH is
actually something else.  Am I the only one to think that making this
drastic change in the design at this stange (3 years in) is too late?
At that point, it might be better to rip all the current code off and
start from scratch.

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: RFC: making the PVH 64bit ABI as stableo

[Boris Ostrovsky]

On 06/05/2015 01:16 PM, Stefano Stabellini wrote:
> On Fri, 5 Jun 2015, Andrew Cooper wrote:
>> On 05/06/15 17:43, Boris Ostrovsky wrote:
>>> On 06/05/2015 12:16 PM, Roger Pau Monné wrote:
>>>> El 03/06/15 a les 14.08, Jan Beulich ha escrit:
>>>>>>>> On 03.06.15 at 12:02, <stefano.stabellini@eu.citrix.com> wrote:
>>>>>> On Tue, 2 Jun 2015, Andrew Cooper wrote:
>>>>>>> With my x86 maintainer hat on, the following is an absolute
>>>>>>> minimum set
>>>>>>> of prerequisite for PVH.
>>>>>>>
>>>>>>> * 32bit support
>>>>>> Could you please explain why 32bit is important to get PVH out of tech
>>>>>> preview? I don't see 32 bit OSes as an important use case. Maybe there
>>>>>> is more behind it that I cannot see.
>>>>> The primary reason was named before: 32-bit support will likely
>>>>> end up changing the way 64-bit guests get launched.
>>>> I can work on the new boot ABI, even if it's just a design document now,
>>>> but the actual implementation needs to be done on top of the 32-bit
>>>> support series.
>>>>
>>>> Boris, do you think you could send an early RFC of your 32-bit support
>>>> series in a couple of weeks at most?
>>> That's highly unlikely. For one, I am still unable to boot MP guests.
>>> In addition, it is all held together by rubber bands and matchsticks
>>> so calling it an RFC would be an insult to RFCs. (for example, I
>>> apparently broke HVM somewhere along the way).
>> How about working it the other way around.
>>
>> Start with an HVM guest and start with a sane method of booting.  I
>> highly suggest multiboot1 as it is very easy and we have most of the
>> code already.  Whomever actually gets around to doing this gets leeway,
>> subject to it being sane (which the current method very certainly is not).
>>
>> Start the domain without qemu, and expose some of the PV hypercalls to
>> HVM guests, and see how far it gets.  One will find suddenly that all
>> 32bit and AMD problems have already been solved.  All the PV(h) kernel
>> needs to know is that there is no real hardware, and not to touch it.
> This seems like a clean and nice way forward, but rather than PVH is
> actually something else.  Am I the only one to think that making this
> drastic change in the design at this stange (3 years in) is too late?
> At that point, it might be better to rip all the current code off and
> start from scratch.

I suspect that doing what Andrew is suggesting will result in most/large 
portion of the code to be replaced (i.e. it's not "better to rip" it 
off, it will just happen organically).

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: RFC: making the PVH 64bit ABI as stableo

[Andrew Cooper]

On 05/06/15 18:16, Stefano Stabellini wrote:
> On Fri, 5 Jun 2015, Andrew Cooper wrote:
>> On 05/06/15 17:43, Boris Ostrovsky wrote:
>>> On 06/05/2015 12:16 PM, Roger Pau Monné wrote:
>>>> El 03/06/15 a les 14.08, Jan Beulich ha escrit:
>>>>>>>> On 03.06.15 at 12:02, <stefano.stabellini@eu.citrix.com> wrote:
>>>>>> On Tue, 2 Jun 2015, Andrew Cooper wrote:
>>>>>>> With my x86 maintainer hat on, the following is an absolute
>>>>>>> minimum set
>>>>>>> of prerequisite for PVH.
>>>>>>>
>>>>>>> * 32bit support
>>>>>> Could you please explain why 32bit is important to get PVH out of tech
>>>>>> preview? I don't see 32 bit OSes as an important use case. Maybe there
>>>>>> is more behind it that I cannot see.
>>>>> The primary reason was named before: 32-bit support will likely
>>>>> end up changing the way 64-bit guests get launched.
>>>> I can work on the new boot ABI, even if it's just a design document now,
>>>> but the actual implementation needs to be done on top of the 32-bit
>>>> support series.
>>>>
>>>> Boris, do you think you could send an early RFC of your 32-bit support
>>>> series in a couple of weeks at most?
>>> That's highly unlikely. For one, I am still unable to boot MP guests.
>>> In addition, it is all held together by rubber bands and matchsticks
>>> so calling it an RFC would be an insult to RFCs. (for example, I
>>> apparently broke HVM somewhere along the way).
>> How about working it the other way around.
>>
>> Start with an HVM guest and start with a sane method of booting.  I
>> highly suggest multiboot1 as it is very easy and we have most of the
>> code already.  Whomever actually gets around to doing this gets leeway,
>> subject to it being sane (which the current method very certainly is not).
>>
>> Start the domain without qemu, and expose some of the PV hypercalls to
>> HVM guests, and see how far it gets.  One will find suddenly that all
>> 32bit and AMD problems have already been solved.  All the PV(h) kernel
>> needs to know is that there is no real hardware, and not to touch it.
> This seems like a clean and nice way forward, but rather than PVH is
> actually something else.  Am I the only one to think that making this
> drastic change in the design at this stange (3 years in) is too late?

There was no design in the slightest, which is why we have got 3 years
in and are in this position.

> At that point, it might be better to rip all the current code off and
> start from scratch.

With a maintainers hat on, I am happy with any solution (including
ripping it all out and starting from scratch) which ends up in the
correct position.

However, I expect it to turn into (HVM - Qemu + very few extra PV
hypercalls) which would probably be better developed straight on top of
HVM guests.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: RFC: making the PVH 64bit ABI as stableo

[Tim Deegan]

At 18:21 +0100 on 05 Jun (1433528517), Andrew Cooper wrote:
> On 05/06/15 18:16, Stefano Stabellini wrote:
> > On Fri, 5 Jun 2015, Andrew Cooper wrote:
> >> On 05/06/15 17:43, Boris Ostrovsky wrote:
> >>> On 06/05/2015 12:16 PM, Roger Pau Monné wrote:
> >>>> El 03/06/15 a les 14.08, Jan Beulich ha escrit:
> >>>>>>>> On 03.06.15 at 12:02, <stefano.stabellini@eu.citrix.com> wrote:
> >>>>>> On Tue, 2 Jun 2015, Andrew Cooper wrote:
> >>>>>>> With my x86 maintainer hat on, the following is an absolute
> >>>>>>> minimum set
> >>>>>>> of prerequisite for PVH.
> >>>>>>>
> >>>>>>> * 32bit support
> >>>>>> Could you please explain why 32bit is important to get PVH out of tech
> >>>>>> preview? I don't see 32 bit OSes as an important use case. Maybe there
> >>>>>> is more behind it that I cannot see.
> >>>>> The primary reason was named before: 32-bit support will likely
> >>>>> end up changing the way 64-bit guests get launched.
> >>>> I can work on the new boot ABI, even if it's just a design document now,
> >>>> but the actual implementation needs to be done on top of the 32-bit
> >>>> support series.
> >>>>
> >>>> Boris, do you think you could send an early RFC of your 32-bit support
> >>>> series in a couple of weeks at most?
> >>> That's highly unlikely. For one, I am still unable to boot MP guests.
> >>> In addition, it is all held together by rubber bands and matchsticks
> >>> so calling it an RFC would be an insult to RFCs. (for example, I
> >>> apparently broke HVM somewhere along the way).
> >> How about working it the other way around.
> >>
> >> Start with an HVM guest and start with a sane method of booting.  I
> >> highly suggest multiboot1 as it is very easy and we have most of the
> >> code already.  Whomever actually gets around to doing this gets leeway,
> >> subject to it being sane (which the current method very certainly is not).
> >>
> >> Start the domain without qemu, and expose some of the PV hypercalls to
> >> HVM guests, and see how far it gets.  One will find suddenly that all
> >> 32bit and AMD problems have already been solved.  All the PV(h) kernel
> >> needs to know is that there is no real hardware, and not to touch it.
> > This seems like a clean and nice way forward, but rather than PVH is
> > actually something else.  Am I the only one to think that making this
> > drastic change in the design at this stange (3 years in) is too late?
> 
> There was no design in the slightest, which is why we have got 3 years
> in and are in this position.

Please try to keep things friendly and contructive on this list.  Yes,
there was design; it was discussed on this list and at the Xen summit.
With hindsight, it turned out that "PV guest that uses a lightweight
HVM container" took a lot more work/code than was originally expected.

I suspect that an implementation of "HVM without qemu and some
hypercalls" will also turn out more complex than it sounds.  I believe
I've made my opinion clear that that's where PVH ought to end up, but
I'm unconvinced that starting from scratch will be the fastest way.

Tim.

Re: RFC: making the PVH 64bit ABI as stableo

[Roger Pau Monné]

El 05/06/15 a les 23.52, Tim Deegan ha escrit:
> At 18:21 +0100 on 05 Jun (1433528517), Andrew Cooper wrote:
>> On 05/06/15 18:16, Stefano Stabellini wrote:
>>> On Fri, 5 Jun 2015, Andrew Cooper wrote:
>>>> On 05/06/15 17:43, Boris Ostrovsky wrote:
>>>>> On 06/05/2015 12:16 PM, Roger Pau Monné wrote:
>>>>>> El 03/06/15 a les 14.08, Jan Beulich ha escrit:
>>>>>>>>>> On 03.06.15 at 12:02, <stefano.stabellini@eu.citrix.com> wrote:
>>>>>>>> On Tue, 2 Jun 2015, Andrew Cooper wrote:
>>>>>>>>> With my x86 maintainer hat on, the following is an absolute
>>>>>>>>> minimum set
>>>>>>>>> of prerequisite for PVH.
>>>>>>>>>
>>>>>>>>> * 32bit support
>>>>>>>> Could you please explain why 32bit is important to get PVH out of tech
>>>>>>>> preview? I don't see 32 bit OSes as an important use case. Maybe there
>>>>>>>> is more behind it that I cannot see.
>>>>>>> The primary reason was named before: 32-bit support will likely
>>>>>>> end up changing the way 64-bit guests get launched.
>>>>>> I can work on the new boot ABI, even if it's just a design document now,
>>>>>> but the actual implementation needs to be done on top of the 32-bit
>>>>>> support series.
>>>>>>
>>>>>> Boris, do you think you could send an early RFC of your 32-bit support
>>>>>> series in a couple of weeks at most?
>>>>> That's highly unlikely. For one, I am still unable to boot MP guests.
>>>>> In addition, it is all held together by rubber bands and matchsticks
>>>>> so calling it an RFC would be an insult to RFCs. (for example, I
>>>>> apparently broke HVM somewhere along the way).
>>>> How about working it the other way around.
>>>>
>>>> Start with an HVM guest and start with a sane method of booting.  I
>>>> highly suggest multiboot1 as it is very easy and we have most of the
>>>> code already.  Whomever actually gets around to doing this gets leeway,
>>>> subject to it being sane (which the current method very certainly is not).

I agree that using a boot ABI similar to multiboot1 is going to solve
some of the issues that we currently have, while probably simplifying
the code to build a domain. There are also several multiboot1
implementations around which can be used as a basis for this for guest
OSes that don't have native multiboot support.

>>>> Start the domain without qemu, and expose some of the PV hypercalls to
>>>> HVM guests, and see how far it gets.  One will find suddenly that all
>>>> 32bit and AMD problems have already been solved.  All the PV(h) kernel
>>>> needs to know is that there is no real hardware, and not to touch it.
>>> This seems like a clean and nice way forward, but rather than PVH is
>>> actually something else.  Am I the only one to think that making this
>>> drastic change in the design at this stange (3 years in) is too late?

I don't think the ABI is going to change much, most of this plumbing is
going to be in Xen internals, so I wouldn't call it a drastic change.

>> There was no design in the slightest, which is why we have got 3 years
>> in and are in this position.
> 
> Please try to keep things friendly and contructive on this list.  Yes,
> there was design; it was discussed on this list and at the Xen summit.
> With hindsight, it turned out that "PV guest that uses a lightweight
> HVM container" took a lot more work/code than was originally expected.
> 
> I suspect that an implementation of "HVM without qemu and some
> hypercalls" will also turn out more complex than it sounds.  I believe
> I've made my opinion clear that that's where PVH ought to end up, but
> I'm unconvinced that starting from scratch will be the fastest way.

I believe the right way to move forward is to start implementing this
new boot ABI on top of HVM, without axing out the PVH code. I think most
of the current PVH code would still be needed for the HVM-without-dm
kind of guest, and that at some point both will meet.

I will send a design document for this boot ABI next week, but the plan
is as follows:

 - Start the guest in protected mode without paging.
 - Fill the hypercall page using wrmsr (HVM).
 - Map the shared info page using XENMEM_add_to_physmap (HVM).

That means we can get rid of some of the ELFNOTES, the ones that come to
mind right now are:

 - XEN_ELFNOTE_VIRT_BASE
 - XEN_ELFNOTE_HYPERCALL_PAGE
 - XEN_ELFNOTE_HV_START_LOW
 - XEN_ELFNOTE_PAE_MODE
 - XEN_ELFNOTE_L1_MFN_VALID

And probably some more.

Roger.

Re: RFC: making the PVH 64bit ABI as stableo

[Andrew Cooper]

On 06/06/15 10:57, Roger Pau Monné wrote:
> El 05/06/15 a les 23.52, Tim Deegan ha escrit:
>> At 18:21 +0100 on 05 Jun (1433528517), Andrew Cooper wrote:
>>> On 05/06/15 18:16, Stefano Stabellini wrote:
>>>> On Fri, 5 Jun 2015, Andrew Cooper wrote:
>>>>> On 05/06/15 17:43, Boris Ostrovsky wrote:
>>>>>> On 06/05/2015 12:16 PM, Roger Pau Monné wrote:
>>>>>>> El 03/06/15 a les 14.08, Jan Beulich ha escrit:
>>>>>>>>>>> On 03.06.15 at 12:02, <stefano.stabellini@eu.citrix.com> wrote:
>>>>>>>>> On Tue, 2 Jun 2015, Andrew Cooper wrote:
>>>>>>>>>> With my x86 maintainer hat on, the following is an absolute
>>>>>>>>>> minimum set
>>>>>>>>>> of prerequisite for PVH.
>>>>>>>>>>
>>>>>>>>>> * 32bit support
>>>>>>>>> Could you please explain why 32bit is important to get PVH out of tech
>>>>>>>>> preview? I don't see 32 bit OSes as an important use case. Maybe there
>>>>>>>>> is more behind it that I cannot see.
>>>>>>>> The primary reason was named before: 32-bit support will likely
>>>>>>>> end up changing the way 64-bit guests get launched.
>>>>>>> I can work on the new boot ABI, even if it's just a design document now,
>>>>>>> but the actual implementation needs to be done on top of the 32-bit
>>>>>>> support series.
>>>>>>>
>>>>>>> Boris, do you think you could send an early RFC of your 32-bit support
>>>>>>> series in a couple of weeks at most?
>>>>>> That's highly unlikely. For one, I am still unable to boot MP guests.
>>>>>> In addition, it is all held together by rubber bands and matchsticks
>>>>>> so calling it an RFC would be an insult to RFCs. (for example, I
>>>>>> apparently broke HVM somewhere along the way).
>>>>> How about working it the other way around.
>>>>>
>>>>> Start with an HVM guest and start with a sane method of booting.  I
>>>>> highly suggest multiboot1 as it is very easy and we have most of the
>>>>> code already.  Whomever actually gets around to doing this gets leeway,
>>>>> subject to it being sane (which the current method very certainly is not).
> I agree that using a boot ABI similar to multiboot1 is going to solve
> some of the issues that we currently have, while probably simplifying
> the code to build a domain. There are also several multiboot1
> implementations around which can be used as a basis for this for guest
> OSes that don't have native multiboot support.
>
>>>>> Start the domain without qemu, and expose some of the PV hypercalls to
>>>>> HVM guests, and see how far it gets.  One will find suddenly that all
>>>>> 32bit and AMD problems have already been solved.  All the PV(h) kernel
>>>>> needs to know is that there is no real hardware, and not to touch it.
>>>> This seems like a clean and nice way forward, but rather than PVH is
>>>> actually something else.  Am I the only one to think that making this
>>>> drastic change in the design at this stange (3 years in) is too late?
> I don't think the ABI is going to change much, most of this plumbing is
> going to be in Xen internals, so I wouldn't call it a drastic change.

To my mind, the ABI covers the boot method, but I would agree that the
runtime environment is unlikely to need changing.  Most of what is
involved along these lines is to lift the current restrictions.

>
>>> There was no design in the slightest, which is why we have got 3 years
>>> in and are in this position.
>> Please try to keep things friendly and contructive on this list.  Yes,
>> there was design; it was discussed on this list and at the Xen summit.
>> With hindsight, it turned out that "PV guest that uses a lightweight
>> HVM container" took a lot more work/code than was originally expected.
>>
>> I suspect that an implementation of "HVM without qemu and some
>> hypercalls" will also turn out more complex than it sounds.  I believe
>> I've made my opinion clear that that's where PVH ought to end up, but
>> I'm unconvinced that starting from scratch will be the fastest way.
> I believe the right way to move forward is to start implementing this
> new boot ABI on top of HVM, without axing out the PVH code. I think most
> of the current PVH code would still be needed for the HVM-without-dm
> kind of guest, and that at some point both will meet.

+1.  I agree, and think this is a very sensible way forwards.

>
> I will send a design document for this boot ABI next week, but the plan
> is as follows:
>
>  - Start the guest in protected mode without paging.

A step in the middle here is finding finding the Xen cpuid leaves.

An elfnote signifying that the kernel is PVH-capable and a cpuid leaf
indicating that the environment is PVH seems like a good combination.

~Andrew

>  - Fill the hypercall page using wrmsr (HVM).
>  - Map the shared info page using XENMEM_add_to_physmap (HVM).
>
> That means we can get rid of some of the ELFNOTES, the ones that come to
> mind right now are:
>
>  - XEN_ELFNOTE_VIRT_BASE
>  - XEN_ELFNOTE_HYPERCALL_PAGE
>  - XEN_ELFNOTE_HV_START_LOW
>  - XEN_ELFNOTE_PAE_MODE
>  - XEN_ELFNOTE_L1_MFN_VALID
>
> And probably some more.

Re: RFC: making the PVH 64bit ABI as stableo

[Boris Ostrovsky]

On 06/06/2015 10:41 AM, Andrew Cooper wrote:
> On 06/06/15 10:57, Roger Pau Monné wrote:
>> El 05/06/15 a les 23.52, Tim Deegan ha escrit:
>>> At 18:21 +0100 on 05 Jun (1433528517), Andrew Cooper wrote:
>>>> On 05/06/15 18:16, Stefano Stabellini wrote:
>>>>> On Fri, 5 Jun 2015, Andrew Cooper wrote:
>>>>>> On 05/06/15 17:43, Boris Ostrovsky wrote:
>>>>>>> On 06/05/2015 12:16 PM, Roger Pau Monné wrote:
>>>>>>>> El 03/06/15 a les 14.08, Jan Beulich ha escrit:
>>>>>>>>>>>> On 03.06.15 at 12:02, <stefano.stabellini@eu.citrix.com> wrote:
>>>>>>>>>> On Tue, 2 Jun 2015, Andrew Cooper wrote:
>>>>>>>>>>> With my x86 maintainer hat on, the following is an absolute
>>>>>>>>>>> minimum set
>>>>>>>>>>> of prerequisite for PVH.
>>>>>>>>>>>
>>>>>>>>>>> * 32bit support
>>>>>>>>>> Could you please explain why 32bit is important to get PVH out of tech
>>>>>>>>>> preview? I don't see 32 bit OSes as an important use case. Maybe there
>>>>>>>>>> is more behind it that I cannot see.
>>>>>>>>> The primary reason was named before: 32-bit support will likely
>>>>>>>>> end up changing the way 64-bit guests get launched.
>>>>>>>> I can work on the new boot ABI, even if it's just a design document now,
>>>>>>>> but the actual implementation needs to be done on top of the 32-bit
>>>>>>>> support series.
>>>>>>>>
>>>>>>>> Boris, do you think you could send an early RFC of your 32-bit support
>>>>>>>> series in a couple of weeks at most?
>>>>>>> That's highly unlikely. For one, I am still unable to boot MP guests.
>>>>>>> In addition, it is all held together by rubber bands and matchsticks
>>>>>>> so calling it an RFC would be an insult to RFCs. (for example, I
>>>>>>> apparently broke HVM somewhere along the way).
>>>>>> How about working it the other way around.
>>>>>>
>>>>>> Start with an HVM guest and start with a sane method of booting.  I
>>>>>> highly suggest multiboot1 as it is very easy and we have most of the
>>>>>> code already.  Whomever actually gets around to doing this gets leeway,
>>>>>> subject to it being sane (which the current method very certainly is not).
>> I agree that using a boot ABI similar to multiboot1 is going to solve
>> some of the issues that we currently have, while probably simplifying
>> the code to build a domain. There are also several multiboot1
>> implementations around which can be used as a basis for this for guest
>> OSes that don't have native multiboot support.
>>
>>>>>> Start the domain without qemu, and expose some of the PV hypercalls to
>>>>>> HVM guests, and see how far it gets.  One will find suddenly that all
>>>>>> 32bit and AMD problems have already been solved.  All the PV(h) kernel
>>>>>> needs to know is that there is no real hardware, and not to touch it.
>>>>> This seems like a clean and nice way forward, but rather than PVH is
>>>>> actually something else.  Am I the only one to think that making this
>>>>> drastic change in the design at this stange (3 years in) is too late?
>> I don't think the ABI is going to change much, most of this plumbing is
>> going to be in Xen internals, so I wouldn't call it a drastic change.
> To my mind, the ABI covers the boot method, but I would agree that the
> runtime environment is unlikely to need changing.  Most of what is
> involved along these lines is to lift the current restrictions.
>
>>>> There was no design in the slightest, which is why we have got 3 years
>>>> in and are in this position.
>>> Please try to keep things friendly and contructive on this list.  Yes,
>>> there was design; it was discussed on this list and at the Xen summit.
>>> With hindsight, it turned out that "PV guest that uses a lightweight
>>> HVM container" took a lot more work/code than was originally expected.
>>>
>>> I suspect that an implementation of "HVM without qemu and some
>>> hypercalls" will also turn out more complex than it sounds.  I believe
>>> I've made my opinion clear that that's where PVH ought to end up, but
>>> I'm unconvinced that starting from scratch will be the fastest way.
>> I believe the right way to move forward is to start implementing this
>> new boot ABI on top of HVM, without axing out the PVH code. I think most
>> of the current PVH code would still be needed for the HVM-without-dm
>> kind of guest, and that at some point both will meet.


This is a good thing to do in the medium-to-long term but it seems to me 
that we still should get current PVH implementation to work first (i.e. 
AMD/32-bit). If past experience is any indication, this rework will take 
a while while we have something working right now (warts and all).

For example, the diffstat of what I have now for 32-bit (i.e. UP only) is
     20 files changed, 503 insertions(+), 95 deletions(-)
and that is probably 75% debugging/crud.

Linux is even less:
     17 files changed, 221 insertions(+), 28 deletions(-)
(although I did some fairly awful things there).

So adding that part appears to be doable without major disruption on the 
Xen side. And I don't think AMD support will be any worse.


-boris


> +1.  I agree, and think this is a very sensible way forwards.
>
>> I will send a design document for this boot ABI next week, but the plan
>> is as follows:
>>
>>   - Start the guest in protected mode without paging.
> A step in the middle here is finding finding the Xen cpuid leaves.
>
> An elfnote signifying that the kernel is PVH-capable and a cpuid leaf
> indicating that the environment is PVH seems like a good combination.
>
> ~Andrew
>
>>   - Fill the hypercall page using wrmsr (HVM).
>>   - Map the shared info page using XENMEM_add_to_physmap (HVM).
>>
>> That means we can get rid of some of the ELFNOTES, the ones that come to
>> mind right now are:
>>
>>   - XEN_ELFNOTE_VIRT_BASE
>>   - XEN_ELFNOTE_HYPERCALL_PAGE
>>   - XEN_ELFNOTE_HV_START_LOW
>>   - XEN_ELFNOTE_PAE_MODE
>>   - XEN_ELFNOTE_L1_MFN_VALID
>>
>> And probably some more.

Re: RFC: making the PVH 64bit ABI as stableo

[Ian Campbell]

On Fri, 2015-06-05 at 18:21 +0100, Andrew Cooper wrote:

> However, I expect it to turn into (HVM - Qemu + very few extra PV
> hypercalls)

Don't forget "- <in Xen device models (e.g. PIT etc)>" and "- some other
stuff which I'm sure I'm forgetting from Tim's original list of things
to be made conditional".

Ian.

Re: RFC: making the PVH 64bit ABI as stableo

[Konrad Rzeszutek Wilk]

On Sat, Jun 06, 2015 at 11:57:31AM +0200, Roger Pau Monné wrote:
> El 05/06/15 a les 23.52, Tim Deegan ha escrit:
> > At 18:21 +0100 on 05 Jun (1433528517), Andrew Cooper wrote:
> >> On 05/06/15 18:16, Stefano Stabellini wrote:
> >>> On Fri, 5 Jun 2015, Andrew Cooper wrote:
> >>>> On 05/06/15 17:43, Boris Ostrovsky wrote:
> >>>>> On 06/05/2015 12:16 PM, Roger Pau Monné wrote:
> >>>>>> El 03/06/15 a les 14.08, Jan Beulich ha escrit:
> >>>>>>>>>> On 03.06.15 at 12:02, <stefano.stabellini@eu.citrix.com> wrote:
> >>>>>>>> On Tue, 2 Jun 2015, Andrew Cooper wrote:
> >>>>>>>>> With my x86 maintainer hat on, the following is an absolute
> >>>>>>>>> minimum set
> >>>>>>>>> of prerequisite for PVH.
> >>>>>>>>>
> >>>>>>>>> * 32bit support
> >>>>>>>> Could you please explain why 32bit is important to get PVH out of tech
> >>>>>>>> preview? I don't see 32 bit OSes as an important use case. Maybe there
> >>>>>>>> is more behind it that I cannot see.
> >>>>>>> The primary reason was named before: 32-bit support will likely
> >>>>>>> end up changing the way 64-bit guests get launched.
> >>>>>> I can work on the new boot ABI, even if it's just a design document now,
> >>>>>> but the actual implementation needs to be done on top of the 32-bit
> >>>>>> support series.
> >>>>>>
> >>>>>> Boris, do you think you could send an early RFC of your 32-bit support
> >>>>>> series in a couple of weeks at most?
> >>>>> That's highly unlikely. For one, I am still unable to boot MP guests.
> >>>>> In addition, it is all held together by rubber bands and matchsticks
> >>>>> so calling it an RFC would be an insult to RFCs. (for example, I
> >>>>> apparently broke HVM somewhere along the way).
> >>>> How about working it the other way around.
> >>>>
> >>>> Start with an HVM guest and start with a sane method of booting.  I
> >>>> highly suggest multiboot1 as it is very easy and we have most of the
> >>>> code already.  Whomever actually gets around to doing this gets leeway,
> >>>> subject to it being sane (which the current method very certainly is not).
> 
> I agree that using a boot ABI similar to multiboot1 is going to solve
> some of the issues that we currently have, while probably simplifying
> the code to build a domain. There are also several multiboot1
> implementations around which can be used as a basis for this for guest
> OSes that don't have native multiboot support.

Multiboot1 requires that the header be within 8K of the start of the kernel.
Linux has an PE header, bootparams and multiboot1 would have to fit afterwards
in it.

Now from a implementation and political side:
 - In Linux you would need multiboot1 copy all the paramters in the bootparams
   type. And then if were to use the generic bootup path we need to track
   any changes in the early bootup code - which means we MUST at startup
   look like a bootloader (whatever that means).
 - Adding in extra bootloader support could be blocked by the Linux x86
   maintainers. As in they would prefer to have all the booting code
   related to this lay in arch/x86/xen and just call Linux code the same
   way as it is doing now (setup the pvops, x86_* function tables, etc;
   and then call x86_64_start_reservations (or i386_start_kernel).
   I can see them asking: "Why two entry points for Xen?!" And eliminating
   the old one is not yet an option, unless we are ready to make Linux
   upstream not boot on Amazon or other clouds that provide PV guest support.

Either way the code from multiboot1 (or XEN_ELFNOTE_ENTRY mechanism)
needs to do the same thing - fix up function tables such that the platform
is OK booting without much hardware. And then also setup Linux specific
changes (pass in EFI data, bootparams, x86_init, etc, stack protector).

I think the issue folks see with PVH bootup code is that a lot of 
the setup (GDT, CR registers, etc) is done on the hypervisor side.

And Andrew - I think you would like it to be done as much as possible on
the guest side - and by having the entry point to the OS have the state be
like an bootloader - it can be "done".

I think if you want to go that route it is going to delay PVH by
another year. It will also require the nod/approval from the Linux and
FreeBSD kernel folks.

The "done" is being skeptical. I think the Xen hypervisor part would
still need to setup GDT, CR registers, etc - so you would not
change much on the hypervisor side anyhow - except add more code
to deal with multiboot1 headers.

> 
> >>>> Start the domain without qemu, and expose some of the PV hypercalls to
> >>>> HVM guests, and see how far it gets.  One will find suddenly that all
> >>>> 32bit and AMD problems have already been solved.  All the PV(h) kernel
> >>>> needs to know is that there is no real hardware, and not to touch it.
> >>> This seems like a clean and nice way forward, but rather than PVH is
> >>> actually something else.  Am I the only one to think that making this
> >>> drastic change in the design at this stange (3 years in) is too late?
> 
> I don't think the ABI is going to change much, most of this plumbing is
> going to be in Xen internals, so I wouldn't call it a drastic change.
> 
> >> There was no design in the slightest, which is why we have got 3 years
> >> in and are in this position.
> > 
> > Please try to keep things friendly and contructive on this list.  Yes,
> > there was design; it was discussed on this list and at the Xen summit.
> > With hindsight, it turned out that "PV guest that uses a lightweight
> > HVM container" took a lot more work/code than was originally expected.
> > 
> > I suspect that an implementation of "HVM without qemu and some
> > hypercalls" will also turn out more complex than it sounds.  I believe
> > I've made my opinion clear that that's where PVH ought to end up, but
> > I'm unconvinced that starting from scratch will be the fastest way.
> 
> I believe the right way to move forward is to start implementing this
> new boot ABI on top of HVM, without axing out the PVH code. I think most
> of the current PVH code would still be needed for the HVM-without-dm
> kind of guest, and that at some point both will meet.
> 
> I will send a design document for this boot ABI next week, but the plan
> is as follows:
> 
>  - Start the guest in protected mode without paging.
>  - Fill the hypercall page using wrmsr (HVM).
>  - Map the shared info page using XENMEM_add_to_physmap (HVM).
> 
> That means we can get rid of some of the ELFNOTES, the ones that come to
> mind right now are:
> 
>  - XEN_ELFNOTE_VIRT_BASE
>  - XEN_ELFNOTE_HYPERCALL_PAGE
>  - XEN_ELFNOTE_HV_START_LOW
>  - XEN_ELFNOTE_PAE_MODE
>  - XEN_ELFNOTE_L1_MFN_VALID
> 
> And probably some more.
> 
> Roger.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: RFC: making the PVH 64bit ABI as stableo

[Elena Ufimtseva]

On Fri, Jun 05, 2015 at 05:29:21PM +0100, Ian Campbell wrote:
> On Wed, 2015-06-03 at 09:35 -0400, Boris Ostrovsky wrote:
> > > What I'm hearing from the x86 maintainers is that this is actually a
> > > high priority and not a "nice to have cleanup".
> > >
> > >> I picked 32-bit support, Elena is looking into AMD
> > > With the TODOs + these 2 being the things which the x86 maintainers have
> > > highlighted in this thread as being most critical for marking the ABI as
> > > stable (or at least moving experimental->tech preview) let me ask
> > > explicotly:
> > >
> > >          What are the current time frames on these two items?
> > 
> > For 32-bit support, just to get it to work in the within current 
> > framework I think can be done for 4.7 release (which is late this year 
> > IIRC).
> > 
> > I can't tell you how much it will take to make it a part of a "unified" 
> > 32/64-bit guest launching as I haven't looked at this at all yet.
> 
> Thanks. What about AMD support then? Elena?

Hi Ian.
I am working on debugging PVH AMD and looks like its movingi, slowly.
Not sure how many other issues will be on the way, but hopefully similar
timeframe, ie late this year.

> 
> Ian.
>