live migration from new platform to old platform

live migration from new platform to old platform

[Hao, Xudong]

Does anyone experience for HVM live migration from new generation platform to old generation platform? For X86 got a failure from Broadwell to Haswell, both platforms used same Xen version 4.7.0 RC3.

[root@bdw-ep1 feng]# xl migrate 21 hsw-ep1 
...
migration target: Ready to receive domain.
Saving to migration stream new xl format (info 0x3/0x0/1537) Loading new save file <incoming migration stream> (new xl fmt info 0x3/0x0/1537)  Savefile contains xl domain config in JSON format Parsing config from <saved>
xc: info: Saving domain 21, type x86 HVM
xc: info: Found x86 HVM domain from Xen 4.7
xc: info: Restoring domain
xc: error: Unable to restore HVM context (1 = Operation not permitted): Internal error
xc: error: Restore failed (1 = Operation not permitted): Internal error
libxl: error: libxl_stream_read.c:852:libxl__xc_domain_restore_done: restoring domain: Operation not permitted
libxl: error: libxl_create.c:1220:domcreate_rebuild_done: cannot (re-)build domain: -3
libxl: error: libxl.c:1564:libxl__destroy_domid: non-existant domain 31
libxl: error: libxl.c:1523:domain_destroy_callback: unable to destroy guest with domid 31
libxl: error: libxl.c:1452:domain_destroy_cb: destruction of domain 31 failed migration target: Domain creation failed (code -3).
libxl: error: libxl_utils.c:507:libxl_read_exactly: file/stream truncated reading ready message from migration receiver stream
libxl: info: libxl_exec.c:118:libxl_report_child_exitstatus: migration transport process [20867] exited with error status 1 Migration failed, resuming at sender.


Guessing HVM context of some new instruction can't be restored to an old platform. Current Xen xl has cupid parameter parsing implement "cupid" in guest config, like 
cupid='host,xsave=0'
or
cpuid=[ '1:ecx=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0' ]

So I tried to disable some new cupid in guest config, both of the two below method doesn't work, the HVM guest didn't disable those cupid such as "hle, rtm, rdseed...".
cpuid='host,3dnowprefetch=0,hle=0,rtm=0,rdseed=0,adx=0,smap=0'
cpuid=['7:ebx=xxxxxxxxxxx000xxxxxx0xxxxxx0xxxx']

From function libxl_cpuid_parse_config() of tools/libxl/libxl_cpuid.c, only some cupid are implemented. Feature lacking issue?

Best Regards,
Xudong


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: live migration from new platform to old platform

[Jan Beulich]

>>> On 07.06.16 at 09:02, <xudong.hao@intel.com> wrote:
> Guessing HVM context of some new instruction can't be restored to an old 
> platform. Current Xen xl has cupid parameter parsing implement "cupid" in 
> guest config, like 
> cupid='host,xsave=0'
> or
> cpuid=[ '1:ecx=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0' ]
> 
> So I tried to disable some new cupid in guest config, both of the two below 
> method doesn't work, the HVM guest didn't disable those cupid such as "hle, 
> rtm, rdseed...".
> cpuid='host,3dnowprefetch=0,hle=0,rtm=0,rdseed=0,adx=0,smap=0'
> cpuid=['7:ebx=xxxxxxxxxxx000xxxxxx0xxxxxx0xxxx']

This derivation of yours doesn't look right: Leaf 7 requires a sub-leaf
to be specified (I don't recall off the top of my head what the exact
syntax is). So I first of all wonder - did you verify that your CPUID
adjustments actually too effect (in the sense of making it into
d->arch.cpuids[])? Because once there, I can't see how the guest
could get presented anything other than those values.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: live migration from new platform to old platform

[Hao, Xudong]

> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Tuesday, June 7, 2016 3:39 PM
> To: Hao, Xudong <xudong.hao@intel.com>
> Cc: Wei Liu <wei.liu2@citrix.com>; Zhang, Haozhong
> <haozhong.zhang@intel.com>; Xen-devel <xen-devel@lists.xenproject.org>
> Subject: Re: [Xen-devel] live migration from new platform to old platform
> 
> >>> On 07.06.16 at 09:02, <xudong.hao@intel.com> wrote:
> > Guessing HVM context of some new instruction can't be restored to an
> > old platform. Current Xen xl has cupid parameter parsing implement
> > "cupid" in guest config, like cupid='host,xsave=0'
> > or
> > cpuid=[ '1:ecx=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0' ]
> >
> > So I tried to disable some new cupid in guest config, both of the two
> > below method doesn't work, the HVM guest didn't disable those cupid
> > such as "hle, rtm, rdseed...".
> > cpuid='host,3dnowprefetch=0,hle=0,rtm=0,rdseed=0,adx=0,smap=0'
> > cpuid=['7:ebx=xxxxxxxxxxx000xxxxxx0xxxxxx0xxxx']
> 
> This derivation of yours doesn't look right: Leaf 7 requires a sub-leaf to be
> specified (I don't recall off the top of my head what the exact syntax is). So I first
> of all wonder - did you verify that your CPUID adjustments actually too effect (in
> the sense of making it into
> d->arch.cpuids[])? Because once there, I can't see how the guest
> could get presented anything other than those values.
> 

Jan, Thanks.
Yes, there is a Sub-leaf after leaf 7, with below additional line, migration succeed from Broadwell to Haswell.
cpuid=['7,0:ebx=xxxxxxxxxxx000xxxxxx0xxxxxx0xxxx']

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: live migration from new platform to old platform

[Andrew Cooper]

On 07/06/16 08:02, Hao, Xudong wrote:
> Does anyone experience for HVM live migration from new generation platform to old generation platform? For X86 got a failure from Broadwell to Haswell, both platforms used same Xen version 4.7.0 RC3.
>
> [root@bdw-ep1 feng]# xl migrate 21 hsw-ep1 
> ...
> migration target: Ready to receive domain.
> Saving to migration stream new xl format (info 0x3/0x0/1537) Loading new save file <incoming migration stream> (new xl fmt info 0x3/0x0/1537)  Savefile contains xl domain config in JSON format Parsing config from <saved>
> xc: info: Saving domain 21, type x86 HVM
> xc: info: Found x86 HVM domain from Xen 4.7
> xc: info: Restoring domain
> xc: error: Unable to restore HVM context (1 = Operation not permitted): Internal error
> xc: error: Restore failed (1 = Operation not permitted): Internal error
> libxl: error: libxl_stream_read.c:852:libxl__xc_domain_restore_done: restoring domain: Operation not permitted
> libxl: error: libxl_create.c:1220:domcreate_rebuild_done: cannot (re-)build domain: -3
> libxl: error: libxl.c:1564:libxl__destroy_domid: non-existant domain 31
> libxl: error: libxl.c:1523:domain_destroy_callback: unable to destroy guest with domid 31
> libxl: error: libxl.c:1452:domain_destroy_cb: destruction of domain 31 failed migration target: Domain creation failed (code -3).
> libxl: error: libxl_utils.c:507:libxl_read_exactly: file/stream truncated reading ready message from migration receiver stream
> libxl: info: libxl_exec.c:118:libxl_report_child_exitstatus: migration transport process [20867] exited with error status 1 Migration failed, resuming at sender.

I see you have a solution to this problem.

In general, `xl dmesg` on hsw-ep1 will provide some information as to
why the restore failed.  Given your hardware, I expect the problem was
with CR4.SMAP.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: live migration from new platform to old platform

[Hao, Xudong]

> -----Original Message-----
> From: Andrew Cooper [mailto:andrew.cooper3@citrix.com]
> Sent: Tuesday, June 7, 2016 5:47 PM
> To: Hao, Xudong <xudong.hao@intel.com>; Xen-devel <xen-
> devel@lists.xenproject.org>
> Cc: Zhang, Haozhong <haozhong.zhang@intel.com>; Wei Liu
> <wei.liu2@citrix.com>
> Subject: Re: [Xen-devel] live migration from new platform to old platform
> 
> On 07/06/16 08:02, Hao, Xudong wrote:
> > Does anyone experience for HVM live migration from new generation
> platform to old generation platform? For X86 got a failure from Broadwell to
> Haswell, both platforms used same Xen version 4.7.0 RC3.
> >
> > [root@bdw-ep1 feng]# xl migrate 21 hsw-ep1 ...
> > migration target: Ready to receive domain.
> > Saving to migration stream new xl format (info 0x3/0x0/1537) Loading
> > new save file <incoming migration stream> (new xl fmt info
> > 0x3/0x0/1537)  Savefile contains xl domain config in JSON format
> > Parsing config from <saved>
> > xc: info: Saving domain 21, type x86 HVM
> > xc: info: Found x86 HVM domain from Xen 4.7
> > xc: info: Restoring domain
> > xc: error: Unable to restore HVM context (1 = Operation not
> > permitted): Internal error
> > xc: error: Restore failed (1 = Operation not permitted): Internal
> > error
> > libxl: error: libxl_stream_read.c:852:libxl__xc_domain_restore_done:
> > restoring domain: Operation not permitted
> > libxl: error: libxl_create.c:1220:domcreate_rebuild_done: cannot
> > (re-)build domain: -3
> > libxl: error: libxl.c:1564:libxl__destroy_domid: non-existant domain
> > 31
> > libxl: error: libxl.c:1523:domain_destroy_callback: unable to destroy
> > guest with domid 31
> > libxl: error: libxl.c:1452:domain_destroy_cb: destruction of domain 31 failed
> migration target: Domain creation failed (code -3).
> > libxl: error: libxl_utils.c:507:libxl_read_exactly: file/stream
> > truncated reading ready message from migration receiver stream
> > libxl: info: libxl_exec.c:118:libxl_report_child_exitstatus: migration transport
> process [20867] exited with error status 1 Migration failed, resuming at sender.
> 
> I see you have a solution to this problem.
> 
Yes.

> In general, `xl dmesg` on hsw-ep1 will provide some information as to why the
> restore failed.  Given your hardware, I expect the problem was with CR4.SMAP.
> 
Actually, it report bad CR4 but not show SMAP explicitly, of course the CR4 include SMAP control bit in below error log.
(XEN) HVM4 restore: bad CR4 0x3406f0

Thanks,
-Xudong

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel