From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chenxiao Zhao Subject: questions of vm save/restore on arm64 Date: Fri, 27 May 2016 10:08:23 +0000 Message-ID: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5973850146651517570==" Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: "xen-devel@lists.xen.org" Cc: Stefano Stabellini List-Id: xen-devel@lists.xenproject.org --===============5973850146651517570== Content-Type: multipart/alternative; boundary=047d7bf0ebfcde519b0533d016ee --047d7bf0ebfcde519b0533d016ee Content-Type: text/plain; charset=UTF-8 Hi, My board is Hikey on which have octa-core of arm cortex-a53. I have applied patches [1] to try vm save/restore on arm. These patches originally do not working on arm64. I have made some changes based on patch set [2]. What I have got so far is 1. if I run 'xl save -p guest memState' to leave guest in suspend state, then run 'xl unpause guest'. the guest can resume successfully. so I suppose the guest works find on suspend/resume. 2. if I run 'xl restore -p memState' to restore guest and use xenctx to dump all vcpu's registers. all the registers are identical to the state on save. After I run 'xl unpause guest', I got no error but can not connect to console. After restore the guest's PC is at a function called user_disable_single_step(), which is called by single_step_handler(). My question is 1. How could I debug guest on restore progress? are there any tools available? 2. From my understanding, the restore not working is because some status is missing when saving. e.g. on cpu_save, it know the domain is 64bit, but on cpu_load, it always think it is a 32bit domain. so I have hard coded the domain type to DOMAIN_64BIT. Am I correct? 3. How could I dump all VM's status? I only found xenctx can dump vcpu's registers. I have attached my patch and log below. Looking forward for your feedback. Thanks xl list Name ID Mem VCPUs State Time(s) Domain-0 0 1024 8 r----- 11.7 root@linaro-alip:~# xl create guest.cfg Parsing config from guest.cfg [ 39.238723] xen-blkback: ring-ref 8, event-channel 3, protocol 1 (arm-abi) persistent grants root@linaro-alip:~# xl save -p guest memState Saving to memState new xl format (info 0x3/0x0/931) xc: info: Saving domain 1, type ARM (XEN) HVM1 save: VCPU (XEN) HVM1 save: A15_TIMER (XEN) HVM1 save: GICV2_GICD (XEN) HVM1 save: GICV2_GICC (XEN) HVM1 save: GICV3 root@linaro-alip:~# /usr/lib/xen/bin/xenctx -a 1 PC: ffffffc0000ab028 LR: ffffffc00050458c ELR_EL1: ffffffc000086b34 CPSR: 200001c5 SPSR_EL1: 60000145 SP_EL0: 0000007ff6f2a850 SP_EL1: ffffffc0140a7ca0 x0: 0000000000000001 x1: 00000000deadbeef x2: 0000000000000002 x3: 0000000000000002 x4: 0000000000000004 x5: 0000000000000000 x6: 000000000000001b x7: 0000000000000001 x8: 000000618e589e00 x9: 0000000000000000 x10: 0000000000000000 x11: 0000000000000000 x12: 00000000000001a3 x13: 000000001911a7d9 x14: 0000000000002ee0 x15: 0000000000000005 x16: 00000000deadbeef x17: 0000000000000001 x18: 0000000000000007 x19: 0000000000000000 x20: ffffffc014163d58 x21: ffffffc014163cd8 x22: 0000000000000001 x23: 0000000000000140 x24: ffffffc000d5bb18 x25: ffffffc014163cd8 x26: 0000000000000000 x27: 0000000000000000 x28: 0000000000000000 x29: ffffffc0140a7ca0 SCTLR: 34d5d91d TTBCR: 00000032b5193519 TTBR0: 002d000054876000 TTBR1: 0000000040dcf000 root@linaro-alip:~# xl destroy guest (XEN) mm.c:1265:d0v1 gnttab_mark_dirty not implemented yet root@linaro-alip:~# xl restore -p memState Loading new save file memState (new xl fmt info 0x3/0x0/931) Savefile contains xl domain config in JSON format Parsing config from xc: info: (XEN) HVM2 restore: VCPU 0 Found ARM domain from Xen 4.7 xc: info: Restoring domain (XEN) HVM2 restore: A15_TIMER 0 (XEN) HVM2 restore: A15_TIMER 0 (XEN) HVM2 restore: GICV2_GICD 0 (XEN) HVM2 restore: GICV2_GICC 0 (XEN) GICH_LRs (vcpu 0) mask=0 (XEN) VCPU_LR[0]=0 (XEN) VCPU_LR[1]=0 (XEN) VCPU_LR[2]=0 (XEN) VCPU_LR[3]=0 xc: info: Restore successful xc: info: XenStore: mfn 0x39001, dom 0, evt 1 xc: info: Console: mfn 0x39000, dom 0, evt 2 root@linaro-alip:~# /usr/lib/xen/bin/xenctx -a 2 PC: ffffffc0000ab028 LR: ffffffc00050458c ELR_EL1: ffffffc000086b34 CPSR: 200001c5 SPSR_EL1: 60000145 SP_EL0: 0000007ff6f2a850 SP_EL1: ffffffc0140a7ca0 x0: 0000000000000000 x1: 00000000deadbeef x2: 0000000000000002 x3: 0000000000000002 x4: 0000000000000004 x5: 0000000000000000 x6: 000000000000001b x7: 0000000000000001 x8: 000000618e589e00 x9: 0000000000000000 x10: 0000000000000000 x11: 0000000000000000 x12: 00000000000001a3 x13: 000000001911a7d9 x14: 0000000000002ee0 x15: 0000000000000005 x16: 00000000deadbeef x17: 0000000000000001 x18: 0000000000000007 x19: 0000000000000000 x20: ffffffc014163d58 x21: ffffffc014163cd8 x22: 0000000000000001 x23: 0000000000000140 x24: ffffffc000d5bb18 x25: ffffffc014163cd8 x26: 0000000000000000 x27: 0000000000000000 x28: 0000000000000000 x29: ffffffc0140a7ca0 SCTLR: 34d5d91d TTBCR: 00000000b5193519 TTBR0: 002d000054876000 TTBR1: 0000000040dcf000 root@linaro-alip:~# xl unpause guest root@linaro-alip:~# xl list Name ID Mem VCPUs State Time(s) Domain-0 0 1024 8 r----- 22.2 guest 2 0 1 r----- 4.8 root@linaro-alip:~# /usr/lib/xen/bin/xenctx -a 2 PC: ffffffc000084a00 LR: ffffffc00050458c ELR_EL1: ffffffc000084a00 CPSR: 000003c5 SPSR_EL1: 000003c5 SP_EL0: 0000007ff6f2a850 SP_EL1: ffffffc0140a7ca0 x0: 0000000000000000 x1: 00000000deadbeef x2: 0000000000000002 x3: 0000000000000002 x4: 0000000000000004 x5: 0000000000000000 x6: 000000000000001b x7: 0000000000000001 x8: 000000618e589e00 x9: 0000000000000000 x10: 0000000000000000 x11: 0000000000000000 x12: 00000000000001a3 x13: 000000001911a7d9 x14: 0000000000002ee0 x15: 0000000000000005 x16: 00000000deadbeef x17: 0000000000000001 x18: 0000000000000007 x19: 0000000000000000 x20: ffffffc014163d58 x21: ffffffc014163cd8 x22: 0000000000000001 x23: 0000000000000140 x24: ffffffc000d5bb18 x25: ffffffc014163cd8 x26: 0000000000000000 x27: 0000000000000000 x28: 0000000000000000 x29: ffffffc0140a7ca0 SCTLR: 34d5d91d TTBCR: 00000000b5193519 TTBR0: 002d000054876000 TTBR1: 0000000040dcf000 root@linaro-alip:~# xl console guest xenconsole: Could not read tty from store: Success root@linaro-alip:~# diff --git a/xen/arch/arm/hvm.c b/xen/arch/arm/hvm.c index aee3353..411bab4 100644 --- a/xen/arch/arm/hvm.c +++ b/xen/arch/arm/hvm.c @@ -120,7 +120,8 @@ static int cpu_save(struct domain *d, hvm_domain_context_t *h) ctxt.dfar = v->arch.dfar; ctxt.dfsr = v->arch.dfsr; #else - /* XXX 64-bit */ + ctxt.far = v->arch.far; + ctxt.esr = v->arch.esr; #endif #ifdef CONFIG_ARM_32 @@ -187,6 +188,9 @@ static int cpu_load(struct domain *d, hvm_domain_context_t *h) if ( hvm_load_entry(VCPU, h, &ctxt) != 0 ) return -EINVAL; +#ifdef CONFIG_ARM64 + v->arch.type = DOMAIN_64BIT; +#endif v->arch.sctlr = ctxt.sctlr; v->arch.ttbr0 = ctxt.ttbr0; v->arch.ttbr1 = ctxt.ttbr1; @@ -199,7 +203,8 @@ static int cpu_load(struct domain *d, hvm_domain_context_t *h) v->arch.dfar = ctxt.dfar; v->arch.dfsr = ctxt.dfsr; #else - /* XXX 64-bit */ + v->arch.far = ctxt.far; + v->arch.esr = ctxt.esr; #endif #ifdef CONFIG_ARM_32 diff --git a/xen/include/public/arch-arm/hvm/save.h b/xen/include/public/arch-arm/hvm/save.h index db916b1..89e6e89 100644 --- a/xen/include/public/arch-arm/hvm/save.h +++ b/xen/include/public/arch-arm/hvm/save.h @@ -46,8 +46,12 @@ DECLARE_HVM_SAVE_TYPE(HEADER, 1, struct hvm_save_header); struct hvm_hw_cpu { +#ifdef CONFIG_ARM_32 uint64_t vfp[34]; /* Vector floating pointer */ /* VFP v3 state is 34x64 bit, VFP v4 is not yet supported */ +#else + uint64_t vfp[66]; +#endif /* Guest core registers */ struct vcpu_guest_core_regs core_regs; @@ -60,6 +64,9 @@ struct hvm_hw_cpu uint32_t dacr; uint64_t par; + uint64_t far; + uint64_t esr; + uint64_t mair0, mair1; uint64_t tpidr_el0; uint64_t tpidr_el1; [1] http://lists.xen.org/archives/html/xen-devel/2015-12/msg01053.html [2] http://lists.xen.org/archives/html/xen-devel/2014-04/msg01544.html --047d7bf0ebfcde519b0533d016ee Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,=C2=A0

My board is Hikey = on which have octa-core of arm cortex-a53.=C2=A0I have applied patches [1] to try vm save/restore on arm.
=
These patches originally do not workin= g on arm64. I have made some changes based on patch set [2].

What I have got so far is=

1. if I run 'xl save -p guest memState'= ; to leave guest in suspend state, then run 'xl unpause guest'.
=C2=A0 =C2=A0 the guest can r= esume successfully. so I=C2=A0suppose the guest works find on suspen= d/resume.=C2=A0

2. if I run 'xl restore -p memState' to restore guest and use xenc= tx to dump all vcpu's registers.
=C2=A0 =C2=A0 all the registers are identical to the state on sa= ve. After I run 'xl unpause guest', I got no error but can not conn= ect to console.
After rest= ore the guest's PC is at a function called=C2=A0user_disable_single_ste= p(), which is called by=C2=A0single_step_handler().=C2=A0
<= div>
My question is

1. How could I de= bug guest on restore progress? are there any tools available?
<= div>2. From my understanding, the restore n= ot working is because some status is missing when saving.
= =C2=A0e.g. on cpu_save, it know the domain = is 64bit, but on cpu_load, it always think it is a 32bit domain. so I have = hard coded the domain type to DOMAIN_64BIT.
Am I correct?
3. How could I dump all VM= 's status? I only found xenctx can dump vcpu's registers.

I have attached my patch and log below.

Looking forward for your feedback.
Thanks

xl list
Name =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0ID =C2=A0 Mem VCPUs =C2=A0 =C2=A0 =C2=A0State = =C2=A0 Time(s)
Domain-0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 0 =C2=A01024 =C2=A0 =C2=A0 8 =C2=A0 =C2=A0 r----- =C2=A0 =C2=A0 = =C2=A011.7
root@linaro-alip:~# xl create guest.cfg
Pars= ing config from guest.cfg
[ =C2=A0 39.238723] xen-blkback: ring-r= ef 8, event-channel 3, protocol 1 (arm-abi) persistent grants
root@linaro-alip:~# xl save -p guest memState
Saving = to memState new xl format (info 0x3/0x0/931)
xc: info: Saving dom= ain 1, type ARM
(XEN) HVM1 save: VCPU
(XEN) HVM1 save: = A15_TIMER
(XEN) HVM1 save: GICV2_GICD
(XEN) HVM1 save: = GICV2_GICC
(XEN) HVM1 save: GICV3
root@linaro-alip:~# /= usr/lib/xen/bin/xenctx -a 1
PC: =C2=A0 =C2=A0 =C2=A0 ffffffc0000a= b028
LR: =C2=A0 =C2=A0 =C2=A0 ffffffc00050458c
ELR_EL1:= =C2=A0ffffffc000086b34
CPSR: =C2=A0 =C2=A0 200001c5
SP= SR_EL1: 60000145
SP_EL0: =C2=A0 0000007ff6f2a850
SP_EL1= : =C2=A0 ffffffc0140a7ca0

=C2=A0x0: 00000000000000= 01 =C2=A0 =C2=A0x1: 00000000deadbeef =C2=A0 =C2=A0x2: 0000000000000002
=C2=A0x3: 0000000000000002 =C2=A0 =C2=A0x4: 0000000000000004 =C2=A0 = =C2=A0x5: 0000000000000000
=C2=A0x6: 000000000000001b =C2=A0 =C2= =A0x7: 0000000000000001 =C2=A0 =C2=A0x8: 000000618e589e00
=C2=A0x= 9: 0000000000000000 =C2=A0 x10: 0000000000000000 =C2=A0 x11: 00000000000000= 00
x12: 00000000000001a3 =C2=A0 x13: 000000001911a7d9 =C2=A0 x14:= 0000000000002ee0
x15: 0000000000000005 =C2=A0 x16: 00000000deadb= eef =C2=A0 x17: 0000000000000001
x18: 0000000000000007 =C2=A0 x19= : 0000000000000000 =C2=A0 x20: ffffffc014163d58
x21: ffffffc01416= 3cd8 =C2=A0 x22: 0000000000000001 =C2=A0 x23: 0000000000000140
x2= 4: ffffffc000d5bb18 =C2=A0 x25: ffffffc014163cd8 =C2=A0 x26: 00000000000000= 00
x27: 0000000000000000 =C2=A0 x28: 0000000000000000 =C2=A0 x29:= ffffffc0140a7ca0

SCTLR: 34d5d91d
TTBCR:= 00000032b5193519
TTBR0: 002d000054876000
TTBR1: 000000= 0040dcf000
root@linaro-alip:~# xl destroy guest
(XEN) m= m.c:1265:d0v1 gnttab_mark_dirty not implemented yet
root@linaro-a= lip:~# xl restore -p memState
Loading new save file memState (new= xl fmt info 0x3/0x0/931)
=C2=A0Savefile contains xl domain confi= g in JSON format
Parsing config from <saved>
xc: = info: (XEN) HVM2 restore: VCPU 0
Found ARM domain from Xen 4.7
xc: info: Restoring domain
(XEN) HVM2 restore: A15_TIMER = 0
(XEN) HVM2 restore: A15_TIMER 0
(XEN) HVM2 restore: G= ICV2_GICD 0
(XEN) HVM2 restore: GICV2_GICC 0
(XEN) GICH= _LRs (vcpu 0) mask=3D0
(XEN) =C2=A0 =C2=A0VCPU_LR[0]=3D0
(XEN) =C2=A0 =C2=A0VCPU_LR[1]=3D0
(XEN) =C2=A0 =C2=A0VCPU_LR[2]= =3D0
(XEN) =C2=A0 =C2=A0VCPU_LR[3]=3D0
xc: info: Restor= e successful
xc: info: XenStore: mfn 0x39001, dom 0, evt 1
<= div>xc: info: Console: mfn 0x39000, dom 0, evt 2
root@linaro-alip= :~# /usr/lib/xen/bin/xenctx -a 2
PC: =C2=A0 =C2=A0 =C2=A0 ffffffc= 0000ab028
LR: =C2=A0 =C2=A0 =C2=A0 ffffffc00050458c
ELR= _EL1: =C2=A0ffffffc000086b34
CPSR: =C2=A0 =C2=A0 200001c5
SPSR_EL1: 60000145
SP_EL0: =C2=A0 0000007ff6f2a850
S= P_EL1: =C2=A0 ffffffc0140a7ca0

=C2=A0x0: 000000000= 0000000 =C2=A0 =C2=A0x1: 00000000deadbeef =C2=A0 =C2=A0x2: 0000000000000002=
=C2=A0x3: 0000000000000002 =C2=A0 =C2=A0x4: 0000000000000004 =C2= =A0 =C2=A0x5: 0000000000000000
=C2=A0x6: 000000000000001b =C2=A0 = =C2=A0x7: 0000000000000001 =C2=A0 =C2=A0x8: 000000618e589e00
=C2= =A0x9: 0000000000000000 =C2=A0 x10: 0000000000000000 =C2=A0 x11: 0000000000= 000000
x12: 00000000000001a3 =C2=A0 x13: 000000001911a7d9 =C2=A0 = x14: 0000000000002ee0
x15: 0000000000000005 =C2=A0 x16: 00000000d= eadbeef =C2=A0 x17: 0000000000000001
x18: 0000000000000007 =C2=A0= x19: 0000000000000000 =C2=A0 x20: ffffffc014163d58
x21: ffffffc0= 14163cd8 =C2=A0 x22: 0000000000000001 =C2=A0 x23: 0000000000000140
x24: ffffffc000d5bb18 =C2=A0 x25: ffffffc014163cd8 =C2=A0 x26: 0000000000= 000000
x27: 0000000000000000 =C2=A0 x28: 0000000000000000 =C2=A0 = x29: ffffffc0140a7ca0

SCTLR: 34d5d91d
TT= BCR: 00000000b5193519
TTBR0: 002d000054876000
TTBR1: 00= 00000040dcf000
root@linaro-alip:~# xl unpause guest
roo= t@linaro-alip:~# xl list
Name =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ID =C2=A0 Mem VCPUs =C2=A0 =C2=A0 =C2=A0Stat= e =C2=A0 Time(s)
Domain-0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 0 =C2=A01024 =C2=A0 =C2=A0 8 =C2=A0 =C2=A0 r----- =C2=A0 =C2= =A0 =C2=A022.2
guest =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A02 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 1 =C2=A0 =C2=A0 r--= --- =C2=A0 =C2=A0 =C2=A0 4.8
root@linaro-alip:~# /usr/lib/xen/bin= /xenctx -a 2
PC: =C2=A0 =C2=A0 =C2=A0 ffffffc000084a00
= LR: =C2=A0 =C2=A0 =C2=A0 ffffffc00050458c
ELR_EL1: =C2=A0ffffffc0= 00084a00
CPSR: =C2=A0 =C2=A0 000003c5
SPSR_EL1: 000003c= 5
SP_EL0: =C2=A0 0000007ff6f2a850
SP_EL1: =C2=A0 ffffff= c0140a7ca0

=C2=A0x0: 0000000000000000 =C2=A0 =C2= =A0x1: 00000000deadbeef =C2=A0 =C2=A0x2: 0000000000000002
=C2=A0x= 3: 0000000000000002 =C2=A0 =C2=A0x4: 0000000000000004 =C2=A0 =C2=A0x5: 0000= 000000000000
=C2=A0x6: 000000000000001b =C2=A0 =C2=A0x7: 00000000= 00000001 =C2=A0 =C2=A0x8: 000000618e589e00
=C2=A0x9: 000000000000= 0000 =C2=A0 x10: 0000000000000000 =C2=A0 x11: 0000000000000000
x1= 2: 00000000000001a3 =C2=A0 x13: 000000001911a7d9 =C2=A0 x14: 0000000000002e= e0
x15: 0000000000000005 =C2=A0 x16: 00000000deadbeef =C2=A0 x17:= 0000000000000001
x18: 0000000000000007 =C2=A0 x19: 0000000000000= 000 =C2=A0 x20: ffffffc014163d58
x21: ffffffc014163cd8 =C2=A0 x22= : 0000000000000001 =C2=A0 x23: 0000000000000140
x24: ffffffc000d5= bb18 =C2=A0 x25: ffffffc014163cd8 =C2=A0 x26: 0000000000000000
x2= 7: 0000000000000000 =C2=A0 x28: 0000000000000000 =C2=A0 x29: ffffffc0140a7c= a0

SCTLR: 34d5d91d
TTBCR: 00000000b51935= 19
TTBR0: 002d000054876000
TTBR1: 0000000040dcf000
root@linaro-alip:~# xl console guest
xenconsole: Could not = read tty from store: Success
root@linaro-alip:~#
=

diff --git a/xen/arch/arm/hvm.c b/xen= /arch/arm/hvm.c
index aee3353..411bab4 100644
--- a/xen= /arch/arm/hvm.c
+++ b/xen/arch/arm/hvm.c
@@ -120,7 +120= ,8 @@ static int cpu_save(struct domain *d, hvm_domain_context_t *h)
<= div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ctxt.dfar =3D v->arch.dfar;
<= div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ctxt.dfsr =3D v->arch.dfsr;
<= div>=C2=A0#else
- =C2=A0 =C2=A0 =C2=A0 =C2=A0/* XXX 64-bit */
+ =C2=A0 =C2=A0 =C2=A0 ctxt.far =3D v->arch.far;
+ =C2= =A0 =C2=A0 =C2=A0 ctxt.esr =3D v->arch.esr;
=C2=A0#endif
=

=C2=A0#ifdef CONFIG_ARM_32
@@ -187,6 +188,9 @= @ static int cpu_load(struct domain *d, hvm_domain_context_t *h)
= =C2=A0 =C2=A0 =C2=A0if ( hvm_load_entry(VCPU, h, &ctxt) !=3D 0 )
<= div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return -EINVAL;

<= div>+#ifdef CONFIG_ARM64
+ =C2=A0 =C2=A0v->arch.type =3D DOMAI= N_64BIT;
+#endif
=C2=A0 =C2=A0 =C2=A0v->arch.sctlr = =3D ctxt.sctlr;
=C2=A0 =C2=A0 =C2=A0v->arch.ttbr0 =3D ctxt.ttb= r0;
=C2=A0 =C2=A0 =C2=A0v->arch.ttbr1 =3D ctxt.ttbr1;
@@ -199,7 +203,8 @@ static int cpu_load(struct domain *d, hvm_domain_cont= ext_t *h)
=C2=A0 =C2=A0 =C2=A0v->arch.dfar =3D ctxt.dfar;
=C2=A0 =C2=A0 =C2=A0v->arch.dfsr =3D ctxt.dfsr;
=C2=A0#e= lse
- =C2=A0 =C2=A0/* XXX 64-bit */
+ =C2=A0 =C2=A0v-&g= t;arch.far =3D ctxt.far;
+ =C2=A0 =C2=A0v->arch.esr =3D ctxt.e= sr;
=C2=A0#endif

=C2=A0#ifdef CONFIG_ARM= _32
diff --git a/xen/include/public/arch-arm/hvm/save.h b/xen/inc= lude/public/arch-arm/hvm/save.h
index db916b1..89e6e89 100644
--- a/xen/include/public/arch-arm/hvm/save.h
+++ b/xen/inc= lude/public/arch-arm/hvm/save.h
@@ -46,8 +46,12 @@ DECLARE_HVM_SA= VE_TYPE(HEADER, 1, struct hvm_save_header);

=C2=A0= struct hvm_hw_cpu
=C2=A0{
+#ifdef CONFIG_ARM_32
=C2=A0 =C2=A0 =C2=A0uint64_t vfp[34]; /* Vector floating pointer */
=C2=A0 =C2=A0 =C2=A0/* VFP v3 state is 34x64 bit, VFP v4 is not yet s= upported */
+#else
+ =C2=A0 =C2=A0uint64_t vfp[66];
+#endif

=C2=A0 =C2=A0 =C2=A0/* Guest core r= egisters */
=C2=A0 =C2=A0 =C2=A0struct vcpu_guest_core_regs core_= regs;
@@ -60,6 +64,9 @@ struct hvm_hw_cpu
=C2=A0 =C2=A0= =C2=A0uint32_t dacr;
=C2=A0 =C2=A0 =C2=A0uint64_t par;

+ =C2=A0 =C2=A0uint64_t far;
+ =C2=A0 =C2=A0uint6= 4_t esr;
+
=C2=A0 =C2=A0 =C2=A0uint64_t mair0, mair1;
=C2=A0 =C2=A0 =C2=A0uint64_t tpidr_el0;
=C2=A0 =C2=A0 = =C2=A0uint64_t tpidr_el1;



[1]= http://lists.xen.org/archives/html/xen-devel/2015-12/msg01053.html=
--047d7bf0ebfcde519b0533d016ee-- --===============5973850146651517570== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwOi8vbGlzdHMueGVuLm9y Zy94ZW4tZGV2ZWwK --===============5973850146651517570==--