Re: questions of vm save/restore on arm64

[Chenxiao Zhao <chenxiao.zhao@gmail.com>]

On 6/7/2016 9:17 AM, Chenxiao Zhao wrote:
>
>
> On 6/6/2016 7:58 PM, Stefano Stabellini wrote:
>> On Fri, 3 Jun 2016, Chenxiao Zhao wrote:
>>> On 6/3/2016 4:02 AM, Julien Grall wrote:
>>>> Hello,
>>>>
>>>> First thing, the time in the mail headers seems to be wrong. Maybe
>>>> because of a wrong timezone?
>>>>
>>>> I got: 04/06/16 02:32 however we are still the 3rd in my timezone.
>>>>
>>>> On 04/06/16 02:32, Chenxiao Zhao wrote:
>>>>>
>>>>>
>>>>> On 6/3/2016 3:16 AM, Julien Grall wrote:
>>>>>> Hello,
>>>>>>
>>>>>> On 03/06/16 18:05, Chenxiao Zhao wrote:
>>>>>>> I finally found out that the problem is that the toolstack did
>>>>>>> not get
>>>>>>> corret p2m_size while sending all pages on save(always be zero).
>>>>>>> After I
>>>>>>> fixed that, the guest could be restored but guest kernel caught
>>>>>>> handle_mm_fault().
>>>>>>>
>>>>>>> where do you think I'm going to investigate, guest kernel
>>>>>>> hibernation
>>>>>>> restore or xen?
>>>>>>
>>>>>> The hibernation support for ARM64 has only been merged recently in
>>>>>> the
>>>>>> kernel. Which kernel are you using?
>>>>>
>>>>> Hi Julien,
>>>>>
>>>>> I'm using a linaro ported Linux kernel 4.1 for hikey from this link.
>>>>>
>>>>> https://github.com/96boards/linux/tree/android-hikey-linaro-4.1
>>>>>
>>>>> I also applied following patches to make the kernel support
>>>>> hibernation.
>>>>
>>>> This looks the wrong way to do it as this series may requires some
>>>> patches which have been upstreamed before hand.
>>>>
>>>> Linux upstream seems support to the hikey board [1]. Any reason to not
>>>> using it?
>>>
>>> I tried a newer version of kernel 4.4, but got no luck to start dom0
>>> with xen.
>>> so I decide to stay in 4.1 for now.
>>>
>>>>
>>>>> [1] http://www.spinics.net/lists/arm-kernel/msg477769.html
>>>>> [2] http://lists.xen.org/archives/html/xen-devel/2015-12/msg01068.html
>>>>>
>>>>>>
>>>>>> Also, what are the modifications you have made to support Xen
>>>>>> suspend/resume for ARM64?
>>>>>
>>>>> I believe I have posted my modifications on xen in the first mail of
>>>>> this thread.
>>>>
>>>> I mean in Linux. The patch from Ian Campbell does not have any kind of
>>>> support for ARM64.
>>>>
>>>> For instance arch/arm/xen/suspend.c needs to be built for ARM64. So
>>>> I am
>>>> wondering if your kernel has support of hibernation...
>>>
>>> Oh, yes, I most forgot I added this file in arch/arm64/xen/Makefile
>>> to let it
>>> build for arm64.
>>>>
>>>>>
>>>>>  From my understanding, a kernel hibernation will cause kernel to save
>>>>> memories to disk(swap partition). But on guest save progress, the
>>>>> hibernation for domU does not make the guest save memories to disk.
>>>>> it's
>>>>> more like suspend all processes in guest, and memors actually
>>>>> depends on
>>>>> xen toolstack to save the pages to file. Am I correct?
>>>>
>>>> You are using an older tree with a patch series based on a newer tree.
>>>>
>>>> So I would recommend you to move to a newer tree. If it is not
>>>> possible,
>>>> please test that hibernation works on baremetal.
>>>
>>> I think the suspend/resume in guest is working, cause I can use
>>> pause/unpause
>>> command in toolstack to suspend/resume guest without problem. I can
>>> also see
>>> the suspend/resume kernel messages from guest's console. The only
>>> problem is
>>> it's can not resume from restore.
>>
>> But can you still connect to the guest after resume, maybe over the
>> network?
>> If you cannot, then something is likely wrong.
>
> Hi Stefano,
>
> I can connect to the guest after resume from xen console. It responds by
> 'return' key, but I can not run any other commands, e.g. ls or ps. I
> think the guest is not 'fully' restored.
>
>>
>>
>>> One thing that confused me is that the kernel's hibernation means the
>>> guest
>>> kernel will save the memory state to disk and power off VM at last.
>>> The guest
>>> will also take care of the memory restore itself. But I do not see the
>>> save/restore on xen works that way. So my question is why it requires
>>> hibernation (aka. suspend to disk) instead of the real suspend (aka.
>>> suspend
>>> to RAM and standby)?
>>
>> Xen suspend/resume has nothing to do with guest suspend to RAM or guest
>> hibernation.
>>
>> Xen suspend/resume is a way for the hypervisor to save to file the
>> entire state of the VM, including RAM and the state of any devices.
>> Guest suspend to RAM and guest hibernation are two guest driven
>> technologies to save the state of the operating system to RAM or to
>> disk. The only link between Xen suspend and guest suspend is that when
>> Xen issues a domain suspend, it notifies the guest of it so that it can
>> ease the process.  The code in Linux to support Xen suspend/resume is:
>>
>> drivers/xen/manage.c:do_suspend
>>
>> and makes use of some of the Linux internal hooks provided for
>> hibernations (see CONFIG_HIBERNATE_CALLBACKS). But that's just for
>> better integration with the rest of Linux: hibernation is NOT what is
>> happening.
>>
>> I hope that this clarifies things a bit, I realize that it is confusing.
>>
>
> Thanks for your explanation, It clear enough and just as my
> understanding from the code. I think the problem might caused by
> incompatible of arm p2m and xen save/restore mechanism. I'll try a
> core-dump and compare the memory after save and restore. I suppose this
> two dumps should be identical but there already pages are different.
> I'll let you know if I got some progress.
>
> Regards.

Hi all,

I finally got save/restore working on arm64, but it only works when I 
assign only one vCPU to VM. If I set vcpus=4 in configure file, the 
restored VM does not work properly.

Got any ideas?

Best Regards.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel