xen-devel.lists.xenproject.org archive mirror
 help / color / mirror / Atom feed
From: Chenxiao Zhao <chenxiao.zhao@gmail.com>
To: Stefano Stabellini <sstabellini@kernel.org>
Cc: Julien Grall <julien.grall@arm.com>,
	"xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
Subject: Re: questions of vm save/restore on arm64
Date: Sun, 12 Jun 2016 17:46:06 +0800	[thread overview]
Message-ID: <d9a6214a-d79e-9fa0-5dff-600a4f6c191d@gmail.com> (raw)
In-Reply-To: <c036d51f-7a54-04c3-ce76-f1f6292a43bd@gmail.com>

On 6/7/2016 9:17 AM, Chenxiao Zhao wrote:
> On 6/6/2016 7:58 PM, Stefano Stabellini wrote:
>> On Fri, 3 Jun 2016, Chenxiao Zhao wrote:
>>> On 6/3/2016 4:02 AM, Julien Grall wrote:
>>>> Hello,
>>>> First thing, the time in the mail headers seems to be wrong. Maybe
>>>> because of a wrong timezone?
>>>> I got: 04/06/16 02:32 however we are still the 3rd in my timezone.
>>>> On 04/06/16 02:32, Chenxiao Zhao wrote:
>>>>> On 6/3/2016 3:16 AM, Julien Grall wrote:
>>>>>> Hello,
>>>>>> On 03/06/16 18:05, Chenxiao Zhao wrote:
>>>>>>> I finally found out that the problem is that the toolstack did
>>>>>>> not get
>>>>>>> corret p2m_size while sending all pages on save(always be zero).
>>>>>>> After I
>>>>>>> fixed that, the guest could be restored but guest kernel caught
>>>>>>> handle_mm_fault().
>>>>>>> where do you think I'm going to investigate, guest kernel
>>>>>>> hibernation
>>>>>>> restore or xen?
>>>>>> The hibernation support for ARM64 has only been merged recently in
>>>>>> the
>>>>>> kernel. Which kernel are you using?
>>>>> Hi Julien,
>>>>> I'm using a linaro ported Linux kernel 4.1 for hikey from this link.
>>>>> https://github.com/96boards/linux/tree/android-hikey-linaro-4.1
>>>>> I also applied following patches to make the kernel support
>>>>> hibernation.
>>>> This looks the wrong way to do it as this series may requires some
>>>> patches which have been upstreamed before hand.
>>>> Linux upstream seems support to the hikey board [1]. Any reason to not
>>>> using it?
>>> I tried a newer version of kernel 4.4, but got no luck to start dom0
>>> with xen.
>>> so I decide to stay in 4.1 for now.
>>>>> [1] http://www.spinics.net/lists/arm-kernel/msg477769.html
>>>>> [2] http://lists.xen.org/archives/html/xen-devel/2015-12/msg01068.html
>>>>>> Also, what are the modifications you have made to support Xen
>>>>>> suspend/resume for ARM64?
>>>>> I believe I have posted my modifications on xen in the first mail of
>>>>> this thread.
>>>> I mean in Linux. The patch from Ian Campbell does not have any kind of
>>>> support for ARM64.
>>>> For instance arch/arm/xen/suspend.c needs to be built for ARM64. So
>>>> I am
>>>> wondering if your kernel has support of hibernation...
>>> Oh, yes, I most forgot I added this file in arch/arm64/xen/Makefile
>>> to let it
>>> build for arm64.
>>>>>  From my understanding, a kernel hibernation will cause kernel to save
>>>>> memories to disk(swap partition). But on guest save progress, the
>>>>> hibernation for domU does not make the guest save memories to disk.
>>>>> it's
>>>>> more like suspend all processes in guest, and memors actually
>>>>> depends on
>>>>> xen toolstack to save the pages to file. Am I correct?
>>>> You are using an older tree with a patch series based on a newer tree.
>>>> So I would recommend you to move to a newer tree. If it is not
>>>> possible,
>>>> please test that hibernation works on baremetal.
>>> I think the suspend/resume in guest is working, cause I can use
>>> pause/unpause
>>> command in toolstack to suspend/resume guest without problem. I can
>>> also see
>>> the suspend/resume kernel messages from guest's console. The only
>>> problem is
>>> it's can not resume from restore.
>> But can you still connect to the guest after resume, maybe over the
>> network?
>> If you cannot, then something is likely wrong.
> Hi Stefano,
> I can connect to the guest after resume from xen console. It responds by
> 'return' key, but I can not run any other commands, e.g. ls or ps. I
> think the guest is not 'fully' restored.
>>> One thing that confused me is that the kernel's hibernation means the
>>> guest
>>> kernel will save the memory state to disk and power off VM at last.
>>> The guest
>>> will also take care of the memory restore itself. But I do not see the
>>> save/restore on xen works that way. So my question is why it requires
>>> hibernation (aka. suspend to disk) instead of the real suspend (aka.
>>> suspend
>>> to RAM and standby)?
>> Xen suspend/resume has nothing to do with guest suspend to RAM or guest
>> hibernation.
>> Xen suspend/resume is a way for the hypervisor to save to file the
>> entire state of the VM, including RAM and the state of any devices.
>> Guest suspend to RAM and guest hibernation are two guest driven
>> technologies to save the state of the operating system to RAM or to
>> disk. The only link between Xen suspend and guest suspend is that when
>> Xen issues a domain suspend, it notifies the guest of it so that it can
>> ease the process.  The code in Linux to support Xen suspend/resume is:
>> drivers/xen/manage.c:do_suspend
>> and makes use of some of the Linux internal hooks provided for
>> hibernations (see CONFIG_HIBERNATE_CALLBACKS). But that's just for
>> better integration with the rest of Linux: hibernation is NOT what is
>> happening.
>> I hope that this clarifies things a bit, I realize that it is confusing.
> Thanks for your explanation, It clear enough and just as my
> understanding from the code. I think the problem might caused by
> incompatible of arm p2m and xen save/restore mechanism. I'll try a
> core-dump and compare the memory after save and restore. I suppose this
> two dumps should be identical but there already pages are different.
> I'll let you know if I got some progress.
> Regards.

Hi all,

I finally got save/restore working on arm64, but it only works when I 
assign only one vCPU to VM. If I set vcpus=4 in configure file, the 
restored VM does not work properly.

Got any ideas?

Best Regards.

Xen-devel mailing list

  reply	other threads:[~2016-06-12  9:46 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-05-27 10:08 questions of vm save/restore on arm64 Chenxiao Zhao
2016-05-30 11:40 ` Stefano Stabellini
2016-06-01  0:28   ` Chenxiao Zhao
2016-06-02 12:29     ` Julien Grall
2016-06-03 17:05       ` Chenxiao Zhao
2016-06-03 10:16         ` Julien Grall
2016-06-04  1:32           ` Chenxiao Zhao
2016-06-03 11:02             ` Julien Grall
2016-06-04  2:37               ` Chenxiao Zhao
2016-06-03 12:33                 ` Julien Grall
2016-06-06 11:58                 ` Stefano Stabellini
2016-06-07  1:17                   ` Chenxiao Zhao
2016-06-12  9:46                     ` Chenxiao Zhao [this message]
2016-06-12 15:31                       ` Julien Grall
2016-06-13  0:55                         ` Chenxiao Zhao
2016-06-13  9:59                           ` Julien Grall

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d9a6214a-d79e-9fa0-5dff-600a4f6c191d@gmail.com \
    --to=chenxiao.zhao@gmail.com \
    --cc=julien.grall@arm.com \
    --cc=sstabellini@kernel.org \
    --cc=xen-devel@lists.xen.org \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).