staging: unable to restore HVM with Viridian param set

staging: unable to restore HVM with Viridian param set

[Tamas K Lengyel]

Hi all,
trying to restore a Windows VM saved on Xen 4.14 with Xen staging results in:

# xl restore -p /shared/cfg/windows10.save
Loading new save file /shared/cfg/windows10.save (new xl fmt info 0x3/0x0/1475)
 Savefile contains xl domain config in JSON format
Parsing config from <saved>
xc: info: Found x86 HVM domain from Xen 4.14
xc: info: Restoring domain
xc: error: set HVM param 9 = 0x0000000000000065 (17 = File exists):
Internal error
xc: error: Restore failed (17 = File exists): Internal error
libxl: error: libxl_stream_read.c:850:libxl__xc_domain_restore_done:
restoring domain: File exists
libxl: error: libxl_create.c:1581:domcreate_rebuild_done: Domain
8:cannot (re-)build domain: -3
libxl: error: libxl_domain.c:1182:libxl__destroy_domid: Domain
8:Non-existant domain
libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain
8:Unable to destroy guest
libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain
8:Destruction of domain failed

Running on staging 419cd07895891c6642f29085aee07be72413f08c

Tamas

Re: staging: unable to restore HVM with Viridian param set

[Andrew Cooper]

On 01/02/2021 22:51, Tamas K Lengyel wrote:
> Hi all,
> trying to restore a Windows VM saved on Xen 4.14 with Xen staging results in:
>
> # xl restore -p /shared/cfg/windows10.save
> Loading new save file /shared/cfg/windows10.save (new xl fmt info 0x3/0x0/1475)
>  Savefile contains xl domain config in JSON format
> Parsing config from <saved>
> xc: info: Found x86 HVM domain from Xen 4.14
> xc: info: Restoring domain
> xc: error: set HVM param 9 = 0x0000000000000065 (17 = File exists):
> Internal error
> xc: error: Restore failed (17 = File exists): Internal error
> libxl: error: libxl_stream_read.c:850:libxl__xc_domain_restore_done:
> restoring domain: File exists
> libxl: error: libxl_create.c:1581:domcreate_rebuild_done: Domain
> 8:cannot (re-)build domain: -3
> libxl: error: libxl_domain.c:1182:libxl__destroy_domid: Domain
> 8:Non-existant domain
> libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain
> 8:Unable to destroy guest
> libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain
> 8:Destruction of domain failed
>
> Running on staging 419cd07895891c6642f29085aee07be72413f08c

CC'ing maintainers and those who've edited the code recently.

What is happening is xl/libxl is selecting some viridian settings,
applying them to the domain, and then the migrations stream has a
different set of viridian settings.

For a migrating-in VM, nothing should be set during domain build. 
Viridian state has been part of the migrate stream since before mig-v2,
so can be considered to be everywhere relevant now.

~Andrew

Re: staging: unable to restore HVM with Viridian param set

[Igor Druzhinin]

n 01/02/2021 22:57, Andrew Cooper wrote:
> On 01/02/2021 22:51, Tamas K Lengyel wrote:
>> Hi all,
>> trying to restore a Windows VM saved on Xen 4.14 with Xen staging results in:
>>
>> # xl restore -p /shared/cfg/windows10.save
>> Loading new save file /shared/cfg/windows10.save (new xl fmt info 0x3/0x0/1475)
>>  Savefile contains xl domain config in JSON format
>> Parsing config from <saved>
>> xc: info: Found x86 HVM domain from Xen 4.14
>> xc: info: Restoring domain
>> xc: error: set HVM param 9 = 0x0000000000000065 (17 = File exists):
>> Internal error
>> xc: error: Restore failed (17 = File exists): Internal error
>> libxl: error: libxl_stream_read.c:850:libxl__xc_domain_restore_done:
>> restoring domain: File exists
>> libxl: error: libxl_create.c:1581:domcreate_rebuild_done: Domain
>> 8:cannot (re-)build domain: -3
>> libxl: error: libxl_domain.c:1182:libxl__destroy_domid: Domain
>> 8:Non-existant domain
>> libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain
>> 8:Unable to destroy guest
>> libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain
>> 8:Destruction of domain failed
>>
>> Running on staging 419cd07895891c6642f29085aee07be72413f08c
> 
> CC'ing maintainers and those who've edited the code recently.
> 
> What is happening is xl/libxl is selecting some viridian settings,
> applying them to the domain, and then the migrations stream has a
> different set of viridian settings.
> 
> For a migrating-in VM, nothing should be set during domain build. 
> Viridian state has been part of the migrate stream since before mig-v2,
> so can be considered to be everywhere relevant now.

The fallout is likely from my changes that modified default set of viridian
settings. The relevant commits:
983524671031fcfdb24a6c0da988203ebb47aebe
7e5cffcd1e9300cab46a1816b5eb676caaeed2c1

The same config from migrated domains now implies different set of viridian
extensions then those set at the source side. That creates inconsistency in
libxl. I don't really know how to address it properly in libxl other than
don't extend the default set ever.

Igor

RE: staging: unable to restore HVM with Viridian param set

[Paul Durrant]

> -----Original Message-----
> From: Igor Druzhinin <igor.druzhinin@citrix.com>
> Sent: 02 February 2021 00:20
> To: Andrew Cooper <andrew.cooper3@citrix.com>; Tamas K Lengyel <tamas.k.lengyel@gmail.com>; Xen-devel
> <xen-devel@lists.xenproject.org>; Wei Liu <wl@xen.org>; Ian Jackson <iwj@xenproject.org>; Anthony
> PERARD <anthony.perard@citrix.com>; Paul Durrant <paul@xen.org>
> Subject: Re: staging: unable to restore HVM with Viridian param set
> 
> n 01/02/2021 22:57, Andrew Cooper wrote:
> > On 01/02/2021 22:51, Tamas K Lengyel wrote:
> >> Hi all,
> >> trying to restore a Windows VM saved on Xen 4.14 with Xen staging results in:
> >>
> >> # xl restore -p /shared/cfg/windows10.save
> >> Loading new save file /shared/cfg/windows10.save (new xl fmt info 0x3/0x0/1475)
> >>  Savefile contains xl domain config in JSON format
> >> Parsing config from <saved>
> >> xc: info: Found x86 HVM domain from Xen 4.14
> >> xc: info: Restoring domain
> >> xc: error: set HVM param 9 = 0x0000000000000065 (17 = File exists):
> >> Internal error
> >> xc: error: Restore failed (17 = File exists): Internal error
> >> libxl: error: libxl_stream_read.c:850:libxl__xc_domain_restore_done:
> >> restoring domain: File exists
> >> libxl: error: libxl_create.c:1581:domcreate_rebuild_done: Domain
> >> 8:cannot (re-)build domain: -3
> >> libxl: error: libxl_domain.c:1182:libxl__destroy_domid: Domain
> >> 8:Non-existant domain
> >> libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain
> >> 8:Unable to destroy guest
> >> libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain
> >> 8:Destruction of domain failed
> >>
> >> Running on staging 419cd07895891c6642f29085aee07be72413f08c
> >
> > CC'ing maintainers and those who've edited the code recently.
> >
> > What is happening is xl/libxl is selecting some viridian settings,
> > applying them to the domain, and then the migrations stream has a
> > different set of viridian settings.
> >
> > For a migrating-in VM, nothing should be set during domain build.
> > Viridian state has been part of the migrate stream since before mig-v2,
> > so can be considered to be everywhere relevant now.
> 
> The fallout is likely from my changes that modified default set of viridian
> settings. The relevant commits:
> 983524671031fcfdb24a6c0da988203ebb47aebe
> 7e5cffcd1e9300cab46a1816b5eb676caaeed2c1
> 
> The same config from migrated domains now implies different set of viridian
> extensions then those set at the source side. That creates inconsistency in
> libxl. I don't really know how to address it properly in libxl other than
> don't extend the default set ever.
> 

Surely it should be addressed properly in libxl by not messing with the viridian settings on migrate-in/resume, as Andrew says? TBH I thought this was already the case. There should be no problem with adding to the default set as this is just an xl/libxl concept; the param flags in Xen are always define the *exact* set of enabled enlightenments.

  Paul


> Igor

Re: staging: unable to restore HVM with Viridian param set

[Igor Druzhinin]

On 02/02/2021 08:35, Paul Durrant wrote:
>> -----Original Message-----
>> From: Igor Druzhinin <igor.druzhinin@citrix.com>
>> Sent: 02 February 2021 00:20
>> To: Andrew Cooper <andrew.cooper3@citrix.com>; Tamas K Lengyel <tamas.k.lengyel@gmail.com>; Xen-devel
>> <xen-devel@lists.xenproject.org>; Wei Liu <wl@xen.org>; Ian Jackson <iwj@xenproject.org>; Anthony
>> PERARD <anthony.perard@citrix.com>; Paul Durrant <paul@xen.org>
>> Subject: Re: staging: unable to restore HVM with Viridian param set
>>
>> n 01/02/2021 22:57, Andrew Cooper wrote:
>>> On 01/02/2021 22:51, Tamas K Lengyel wrote:
>>>> Hi all,
>>>> trying to restore a Windows VM saved on Xen 4.14 with Xen staging results in:
>>>>
>>>> # xl restore -p /shared/cfg/windows10.save
>>>> Loading new save file /shared/cfg/windows10.save (new xl fmt info 0x3/0x0/1475)
>>>>  Savefile contains xl domain config in JSON format
>>>> Parsing config from <saved>
>>>> xc: info: Found x86 HVM domain from Xen 4.14
>>>> xc: info: Restoring domain
>>>> xc: error: set HVM param 9 = 0x0000000000000065 (17 = File exists):
>>>> Internal error
>>>> xc: error: Restore failed (17 = File exists): Internal error
>>>> libxl: error: libxl_stream_read.c:850:libxl__xc_domain_restore_done:
>>>> restoring domain: File exists
>>>> libxl: error: libxl_create.c:1581:domcreate_rebuild_done: Domain
>>>> 8:cannot (re-)build domain: -3
>>>> libxl: error: libxl_domain.c:1182:libxl__destroy_domid: Domain
>>>> 8:Non-existant domain
>>>> libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain
>>>> 8:Unable to destroy guest
>>>> libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain
>>>> 8:Destruction of domain failed
>>>>
>>>> Running on staging 419cd07895891c6642f29085aee07be72413f08c
>>>
>>> CC'ing maintainers and those who've edited the code recently.
>>>
>>> What is happening is xl/libxl is selecting some viridian settings,
>>> applying them to the domain, and then the migrations stream has a
>>> different set of viridian settings.
>>>
>>> For a migrating-in VM, nothing should be set during domain build.
>>> Viridian state has been part of the migrate stream since before mig-v2,
>>> so can be considered to be everywhere relevant now.
>>
>> The fallout is likely from my changes that modified default set of viridian
>> settings. The relevant commits:
>> 983524671031fcfdb24a6c0da988203ebb47aebe
>> 7e5cffcd1e9300cab46a1816b5eb676caaeed2c1
>>
>> The same config from migrated domains now implies different set of viridian
>> extensions then those set at the source side. That creates inconsistency in
>> libxl. I don't really know how to address it properly in libxl other than
>> don't extend the default set ever.
>>
> 
> Surely it should be addressed properly in libxl by not messing with the viridian settings on migrate-in/resume, as Andrew says? TBH I thought this was already the case. There should be no problem with adding to the default set as this is just an xl/libxl concept; the param flags in Xen are always define the *exact* set of enabled enlightenments.

If maintainers agree with this approach I can make a patch.

Igor

Re: staging: unable to restore HVM with Viridian param set

[Ian Jackson]

Igor Druzhinin writes ("Re: staging: unable to restore HVM with Viridian param set"):
> On 02/02/2021 08:35, Paul Durrant wrote:
> > Surely it should be addressed properly in libxl by not messing with the viridian settings on migrate-in/resume, as Andrew says? TBH I thought this was already the case. There should be no problem with adding to the default set as this is just an xl/libxl concept; the param flags in Xen are always define the *exact* set of enabled enlightenments.
> 
> If maintainers agree with this approach I can make a patch.

If Andy is in favour of this approach then certainly it is fine by me.

FTR, preemptively,

Release-Acked-by: Ian Jackson <iwj@xenproject.org>

although as this is a bugfix it probably doesn't need one.

Ian.

Re: staging: unable to restore HVM with Viridian param set

[Andrew Cooper]

On 02/02/2021 11:51, Ian Jackson wrote:
> Igor Druzhinin writes ("Re: staging: unable to restore HVM with Viridian param set"):
>> On 02/02/2021 08:35, Paul Durrant wrote:
>>> Surely it should be addressed properly in libxl by not messing with the viridian settings on migrate-in/resume, as Andrew says? TBH I thought this was already the case. There should be no problem with adding to the default set as this is just an xl/libxl concept; the param flags in Xen are always define the *exact* set of enabled enlightenments.
>> If maintainers agree with this approach I can make a patch.
> If Andy is in favour of this approach then certainly it is fine by me.

Yeah - in the case that we're restoring from a suspend image or migrate,
just skip setting the viridian flags in build.  They are in the the
stream, and will be restored later.

~Andrew

P.S. This is going to get even more complicated when it all shifts into
CPUID settings, but it needs to happen at some point.