Regression building HVM domains following "x86: add bitmap of enabled emulated devices"

Regression building HVM domains following "x86: add bitmap of enabled emulated devices"

[Andrew Cooper]

Actually CC-ing xen-devel this time.

On 11/11/15 19:13, Andrew Cooper wrote:
> Hello,
>
> Xapi uses the Ocaml stub_xc_domain_create() which uses
> xc_domain_create().  xc_domain_create() itself zeros the arch
> configuration but passes flags straight through.
>
> As a result of c/s 171946ab "x86: add bitmap of enabled emulated
> devices", xc_domain_create() can no longer be used to construct HVM
> domains, failing the hypervisor-side sanity check.
>
> Needless to say, this has put a dent in XenServer's automated testing.
>
>
> There are a couple of options, but neither of them are fantastic.
>
> 1) Have xc_domain_create() fill in XEN_X86_EMU_ALL based on
> XEN_DOMCTL_CDF_hvm_guest and XEN_DOMCTL_CDF_pvh_guest
>
> or
>
> 2) Mandate that all callers provide a valid arch configuration,
> essentially turning xc_domain_create() into xc_domain_create_config()
>
>
> Longterm, what is the plan wrt guest construction?  With my x86
> maintainership hat on, I don't want to keep XEN_DOMCTL_CDF_pvh_guest in
> the interface, so I do not like 1) as an option.
>
> `git grep` indicates that the 3 users of xc_domain_create() are the
> Ocaml/Python stubs and init-xenstore-domain.c which only constructs a PV
> guest (which bypasses the issue), whereas libxl uses
> xc_domain_create_config().  (For the python stubs, I expect this will
> hit Oracle who are still using Xend to my knowledge).
>
> Option 2) is a better alternative, but will have a knock-on effect for
> downstream consumers of the stubs.
>
> It is a worthwhile disruption?  I can't spot a compatible option which
> preserves the stubs' ability to build DMLite domains.
>
> ~Andrew

Re: Regression building HVM domains following "x86: add bitmap of enabled emulated devices"

[Roger Pau Monné]

El 11/11/15 a les 20.22, Andrew Cooper ha escrit:
> Actually CC-ing xen-devel this time.
> 
> On 11/11/15 19:13, Andrew Cooper wrote:
>> Hello,
>>
>> Xapi uses the Ocaml stub_xc_domain_create() which uses
>> xc_domain_create().  xc_domain_create() itself zeros the arch
>> configuration but passes flags straight through.
>>
>> As a result of c/s 171946ab "x86: add bitmap of enabled emulated
>> devices", xc_domain_create() can no longer be used to construct HVM
>> domains, failing the hypervisor-side sanity check.
>>
>> Needless to say, this has put a dent in XenServer's automated testing.
>>
>>
>> There are a couple of options, but neither of them are fantastic.
>>
>> 1) Have xc_domain_create() fill in XEN_X86_EMU_ALL based on
>> XEN_DOMCTL_CDF_hvm_guest and XEN_DOMCTL_CDF_pvh_guest
>>
>> or
>>
>> 2) Mandate that all callers provide a valid arch configuration,
>> essentially turning xc_domain_create() into xc_domain_create_config()
>>
>>
>> Longterm, what is the plan wrt guest construction?  With my x86
>> maintainership hat on, I don't want to keep XEN_DOMCTL_CDF_pvh_guest in
>> the interface, so I do not like 1) as an option.
>>
>> `git grep` indicates that the 3 users of xc_domain_create() are the
>> Ocaml/Python stubs and init-xenstore-domain.c which only constructs a PV
>> guest (which bypasses the issue), whereas libxl uses
>> xc_domain_create_config().  (For the python stubs, I expect this will
>> hit Oracle who are still using Xend to my knowledge).

Qemu upstream also uses xc_domain_create (see
hw/xenpv/xen_domainbuild.c:xen_domain_build_pv).

IMHO, I would just remove xc_domain_create and switch all users to
xc_domain_create_config. I can take care of this if it seems like the
best solution.

Roger.

Re: Regression building HVM domains following "x86: add bitmap of enabled emulated devices"

[Andrew Cooper]

On 11/11/2015 20:57, Konrad Rzeszutek Wilk wrote:
> On Wed, Nov 11, 2015 at 07:13:25PM +0000, Andrew Cooper wrote:
>> Hello,
>>
>> Xapi uses the Ocaml stub_xc_domain_create() which uses
>> xc_domain_create().  xc_domain_create() itself zeros the arch
>> configuration but passes flags straight through.
> Ooops.
>> As a result of c/s 171946ab "x86: add bitmap of enabled emulated
>> devices", xc_domain_create() can no longer be used to construct HVM
>> domains, failing the hypervisor-side sanity check.
>>
>> Needless to say, this has put a dent in XenServer's automated testing.
>>
>>
>> There are a couple of options, but neither of them are fantastic.
>>
>> 1) Have xc_domain_create() fill in XEN_X86_EMU_ALL based on
>> XEN_DOMCTL_CDF_hvm_guest and XEN_DOMCTL_CDF_pvh_guest
>>
>> or
>>
>> 2) Mandate that all callers provide a valid arch configuration,
>> essentially turning xc_domain_create() into xc_domain_create_config()
>>
>>
>> Longterm, what is the plan wrt guest construction?  With my x86
>> maintainership hat on, I don't want to keep XEN_DOMCTL_CDF_pvh_guest in
>> the interface, so I do not like 1) as an option.
>>
>> `git grep` indicates that the 3 users of xc_domain_create() are the
>> Ocaml/Python stubs and init-xenstore-domain.c which only constructs a PV
>> guest (which bypasses the issue), whereas libxl uses
>> xc_domain_create_config().  (For the python stubs, I expect this will
>> hit Oracle who are still using Xend to my knowledge).
> We are moving to 'xl'.. and there are no Xend bits anymore.
>> Option 2) is a better alternative, but will have a knock-on effect for
>> downstream consumers of the stubs.
> But aren't xc_* calls not-release-stable?

They are indeed not, which offers the option to change the API.

>
> Here is a third idea::
>
>  Make 'xc_domain_create' call 'xc_domain_create_config'. The xc_domain_create
>  would synthesis the flags and we would put an 'deprecated' flag on it
>  (whatever that means?) and remove 'xc_domain_create' in 4.7?

This is option 1.  xc_domain_create() already calls
xc_domain_create_config() but with a zeroed arch configuration.

The issue is that modifying xc_domain_create() will preclude their
construction of DMLite domains.

~Andrew

>
>> It is a worthwhile disruption?  I can't spot a compatible option which
>> preserves the stubs' ability to build DMLite domains.
>>
>> ~Andrew

Re: Regression building HVM domains following "x86: add bitmap of enabled emulated devices"

[Konrad Rzeszutek Wilk]

On Wed, Nov 11, 2015 at 09:07:34PM +0000, Andrew Cooper wrote:
> On 11/11/2015 20:57, Konrad Rzeszutek Wilk wrote:
> > On Wed, Nov 11, 2015 at 07:13:25PM +0000, Andrew Cooper wrote:
> >> Hello,
> >>
> >> Xapi uses the Ocaml stub_xc_domain_create() which uses
> >> xc_domain_create().  xc_domain_create() itself zeros the arch
> >> configuration but passes flags straight through.
> > Ooops.
> >> As a result of c/s 171946ab "x86: add bitmap of enabled emulated
> >> devices", xc_domain_create() can no longer be used to construct HVM
> >> domains, failing the hypervisor-side sanity check.
> >>
> >> Needless to say, this has put a dent in XenServer's automated testing.
> >>
> >>
> >> There are a couple of options, but neither of them are fantastic.
> >>
> >> 1) Have xc_domain_create() fill in XEN_X86_EMU_ALL based on
> >> XEN_DOMCTL_CDF_hvm_guest and XEN_DOMCTL_CDF_pvh_guest
> >>
> >> or
> >>
> >> 2) Mandate that all callers provide a valid arch configuration,
> >> essentially turning xc_domain_create() into xc_domain_create_config()
> >>
> >>
> >> Longterm, what is the plan wrt guest construction?  With my x86
> >> maintainership hat on, I don't want to keep XEN_DOMCTL_CDF_pvh_guest in
> >> the interface, so I do not like 1) as an option.
> >>
> >> `git grep` indicates that the 3 users of xc_domain_create() are the
> >> Ocaml/Python stubs and init-xenstore-domain.c which only constructs a PV
> >> guest (which bypasses the issue), whereas libxl uses
> >> xc_domain_create_config().  (For the python stubs, I expect this will
> >> hit Oracle who are still using Xend to my knowledge).
> > We are moving to 'xl'.. and there are no Xend bits anymore.
> >> Option 2) is a better alternative, but will have a knock-on effect for
> >> downstream consumers of the stubs.
> > But aren't xc_* calls not-release-stable?
> 
> They are indeed not, which offers the option to change the API.
> 
> >
> > Here is a third idea::
> >
> >  Make 'xc_domain_create' call 'xc_domain_create_config'. The xc_domain_create
> >  would synthesis the flags and we would put an 'deprecated' flag on it
> >  (whatever that means?) and remove 'xc_domain_create' in 4.7?
> 
> This is option 1.  xc_domain_create() already calls
> xc_domain_create_config() but with a zeroed arch configuration.

When I wrote 'flag' I mean 'arch'. Sorry about that brainfart.
> 
> The issue is that modifying xc_domain_create() will preclude their
> construction of DMLite domains.

/me nods.
> 
> ~Andrew
> 
> >
> >> It is a worthwhile disruption?  I can't spot a compatible option which
> >> preserves the stubs' ability to build DMLite domains.
> >>
> >> ~Andrew
> 

Re: Regression building HVM domains following "x86: add bitmap of enabled emulated devices"

[Ian Campbell]

On Wed, 2015-11-11 at 21:07 +0000, Andrew Cooper wrote:
> On 11/11/2015 20:57, Konrad Rzeszutek Wilk wrote:
> > On Wed, Nov 11, 2015 at 07:13:25PM +0000, Andrew Cooper wrote:
> > > Hello,
> > > 
> > > Xapi uses the Ocaml stub_xc_domain_create() which uses
> > > xc_domain_create().  xc_domain_create() itself zeros the arch
> > > configuration but passes flags straight through.
> > Ooops.
> > > As a result of c/s 171946ab "x86: add bitmap of enabled emulated
> > > devices", xc_domain_create() can no longer be used to construct HVM
> > > domains, failing the hypervisor-side sanity check.
> > > 
> > > Needless to say, this has put a dent in XenServer's automated
> > > testing.
> > > 
> > > 
> > > There are a couple of options, but neither of them are fantastic.
> > > 
> > > 1) Have xc_domain_create() fill in XEN_X86_EMU_ALL based on
> > > XEN_DOMCTL_CDF_hvm_guest and XEN_DOMCTL_CDF_pvh_guest
> > > 
> > > or
> > > 
> > > 2) Mandate that all callers provide a valid arch configuration,
> > > essentially turning xc_domain_create() into xc_domain_create_config()
> > > 
> > > 
> > > Longterm, what is the plan wrt guest construction?  With my x86
> > > maintainership hat on, I don't want to keep XEN_DOMCTL_CDF_pvh_guest
> > > in
> > > the interface, so I do not like 1) as an option.
> > > 
> > > `git grep` indicates that the 3 users of xc_domain_create() are the
> > > Ocaml/Python stubs and init-xenstore-domain.c which only constructs a
> > > PV
> > > guest (which bypasses the issue), whereas libxl uses
> > > xc_domain_create_config().  (For the python stubs, I expect this will
> > > hit Oracle who are still using Xend to my knowledge).
> > We are moving to 'xl'.. and there are no Xend bits anymore.
> > > Option 2) is a better alternative, but will have a knock-on effect
> > > for
> > > downstream consumers of the stubs.
> > But aren't xc_* calls not-release-stable?
> 
> They are indeed not, which offers the option to change the API.
> 
> > 
> > Here is a third idea::
> > 
> >  Make 'xc_domain_create' call 'xc_domain_create_config'. The
> > xc_domain_create
> >  would synthesis the flags and we would put an 'deprecated' flag on it
> >  (whatever that means?) and remove 'xc_domain_create' in 4.7?
> 
> This is option 1.  xc_domain_create() already calls
> xc_domain_create_config() but with a zeroed arch configuration.
> 
> The issue is that modifying xc_domain_create() will preclude their
> construction of DMLite domains.

I think that's ok, we would essentially be declaring that
xc_domain_create_config() is the formally supported interface which can do
everything while xc_domain_create() is essentially a compat shim which can
only do things up to a certain point.

Users who want more would need to switch to the _config variant.

I'm half considering suggesting removing xc_domain_create(), renaming
xc_domain_create_config() without the _config() and having it do some
compat thing if the passed config==NULL, such that existing callers of
xc_domain_create() just need to add NULL to retain their current behaviour.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Regression building HVM domains following "x86: add bitmap of enabled emulated devices"

[Andrew Cooper]

On 12/11/15 09:25, Ian Campbell wrote:
> On Wed, 2015-11-11 at 21:07 +0000, Andrew Cooper wrote:
>> On 11/11/2015 20:57, Konrad Rzeszutek Wilk wrote:
>>> On Wed, Nov 11, 2015 at 07:13:25PM +0000, Andrew Cooper wrote:
>>>> Hello,
>>>>
>>>> Xapi uses the Ocaml stub_xc_domain_create() which uses
>>>> xc_domain_create().  xc_domain_create() itself zeros the arch
>>>> configuration but passes flags straight through.
>>> Ooops.
>>>> As a result of c/s 171946ab "x86: add bitmap of enabled emulated
>>>> devices", xc_domain_create() can no longer be used to construct HVM
>>>> domains, failing the hypervisor-side sanity check.
>>>>
>>>> Needless to say, this has put a dent in XenServer's automated
>>>> testing.
>>>>
>>>>
>>>> There are a couple of options, but neither of them are fantastic.
>>>>
>>>> 1) Have xc_domain_create() fill in XEN_X86_EMU_ALL based on
>>>> XEN_DOMCTL_CDF_hvm_guest and XEN_DOMCTL_CDF_pvh_guest
>>>>
>>>> or
>>>>
>>>> 2) Mandate that all callers provide a valid arch configuration,
>>>> essentially turning xc_domain_create() into xc_domain_create_config()
>>>>
>>>>
>>>> Longterm, what is the plan wrt guest construction?  With my x86
>>>> maintainership hat on, I don't want to keep XEN_DOMCTL_CDF_pvh_guest
>>>> in
>>>> the interface, so I do not like 1) as an option.
>>>>
>>>> `git grep` indicates that the 3 users of xc_domain_create() are the
>>>> Ocaml/Python stubs and init-xenstore-domain.c which only constructs a
>>>> PV
>>>> guest (which bypasses the issue), whereas libxl uses
>>>> xc_domain_create_config().  (For the python stubs, I expect this will
>>>> hit Oracle who are still using Xend to my knowledge).
>>> We are moving to 'xl'.. and there are no Xend bits anymore.
>>>> Option 2) is a better alternative, but will have a knock-on effect
>>>> for
>>>> downstream consumers of the stubs.
>>> But aren't xc_* calls not-release-stable?
>> They are indeed not, which offers the option to change the API.
>>
>>> Here is a third idea::
>>>
>>>  Make 'xc_domain_create' call 'xc_domain_create_config'. The
>>> xc_domain_create
>>>  would synthesis the flags and we would put an 'deprecated' flag on it
>>>  (whatever that means?) and remove 'xc_domain_create' in 4.7?
>> This is option 1.  xc_domain_create() already calls
>> xc_domain_create_config() but with a zeroed arch configuration.
>>
>> The issue is that modifying xc_domain_create() will preclude their
>> construction of DMLite domains.
> I think that's ok, we would essentially be declaring that
> xc_domain_create_config() is the formally supported interface which can do
> everything while xc_domain_create() is essentially a compat shim which can
> only do things up to a certain point.
>
> Users who want more would need to switch to the _config variant.
>
> I'm half considering suggesting removing xc_domain_create(), renaming
> xc_domain_create_config() without the _config() and having it do some
> compat thing if the passed config==NULL, such that existing callers of
> xc_domain_create() just need to add NULL to retain their current behaviour.

That sounds best, with one modification.

In the case that NULL is passed, there should be some default filled in
locally (as currently done for ARM).  For x86, this should now be flags
& hvm => X86_EMU_ALL, so they retain their ability to build HMV guests
by passing a NULL config.

~Andrew

Re: Regression building HVM domains following "x86: add bitmap of enabled emulated devices"

[Ian Campbell]

On Thu, 2015-11-12 at 10:07 +0000, Andrew Cooper wrote:
> On 12/11/15 09:25, Ian Campbell wrote:
> > On Wed, 2015-11-11 at 21:07 +0000, Andrew Cooper wrote:
> > > On 11/11/2015 20:57, Konrad Rzeszutek Wilk wrote:
> > > > On Wed, Nov 11, 2015 at 07:13:25PM +0000, Andrew Cooper wrote:
> > > > > Hello,
> > > > > 
> > > > > Xapi uses the Ocaml stub_xc_domain_create() which uses
> > > > > xc_domain_create().  xc_domain_create() itself zeros the arch
> > > > > configuration but passes flags straight through.
> > > > Ooops.
> > > > > As a result of c/s 171946ab "x86: add bitmap of enabled emulated
> > > > > devices", xc_domain_create() can no longer be used to construct
> > > > > HVM
> > > > > domains, failing the hypervisor-side sanity check.
> > > > > 
> > > > > Needless to say, this has put a dent in XenServer's automated
> > > > > testing.
> > > > > 
> > > > > 
> > > > > There are a couple of options, but neither of them are fantastic.
> > > > > 
> > > > > 1) Have xc_domain_create() fill in XEN_X86_EMU_ALL based on
> > > > > XEN_DOMCTL_CDF_hvm_guest and XEN_DOMCTL_CDF_pvh_guest
> > > > > 
> > > > > or
> > > > > 
> > > > > 2) Mandate that all callers provide a valid arch configuration,
> > > > > essentially turning xc_domain_create() into
> > > > > xc_domain_create_config()
> > > > > 
> > > > > 
> > > > > Longterm, what is the plan wrt guest construction?  With my x86
> > > > > maintainership hat on, I don't want to keep
> > > > > XEN_DOMCTL_CDF_pvh_guest
> > > > > in
> > > > > the interface, so I do not like 1) as an option.
> > > > > 
> > > > > `git grep` indicates that the 3 users of xc_domain_create() are
> > > > > the
> > > > > Ocaml/Python stubs and init-xenstore-domain.c which only
> > > > > constructs a
> > > > > PV
> > > > > guest (which bypasses the issue), whereas libxl uses
> > > > > xc_domain_create_config().  (For the python stubs, I expect this
> > > > > will
> > > > > hit Oracle who are still using Xend to my knowledge).
> > > > We are moving to 'xl'.. and there are no Xend bits anymore.
> > > > > Option 2) is a better alternative, but will have a knock-on
> > > > > effect
> > > > > for
> > > > > downstream consumers of the stubs.
> > > > But aren't xc_* calls not-release-stable?
> > > They are indeed not, which offers the option to change the API.
> > > 
> > > > Here is a third idea::
> > > > 
> > > >  Make 'xc_domain_create' call 'xc_domain_create_config'. The
> > > > xc_domain_create
> > > >  would synthesis the flags and we would put an 'deprecated' flag on
> > > > it
> > > >  (whatever that means?) and remove 'xc_domain_create' in 4.7?
> > > This is option 1.  xc_domain_create() already calls
> > > xc_domain_create_config() but with a zeroed arch configuration.
> > > 
> > > The issue is that modifying xc_domain_create() will preclude their
> > > construction of DMLite domains.
> > I think that's ok, we would essentially be declaring that
> > xc_domain_create_config() is the formally supported interface which can
> > do
> > everything while xc_domain_create() is essentially a compat shim which
> > can
> > only do things up to a certain point.
> > 
> > Users who want more would need to switch to the _config variant.
> > 
> > I'm half considering suggesting removing xc_domain_create(), renaming
> > xc_domain_create_config() without the _config() and having it do some
> > compat thing if the passed config==NULL, such that existing callers of
> > xc_domain_create() just need to add NULL to retain their current
> > behaviour.
> 
> That sounds best, with one modification.
> 
> In the case that NULL is passed, there should be some default filled in
> locally (as currently done for ARM).  For x86, this should now be flags
> & hvm => X86_EMU_ALL, so they retain their ability to build HMV guests
> by passing a NULL config.

Yes, that was the "some compat thing" I was thinking of.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel