linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed
From: Anchal Agarwal <anchalag@amazon.com>
To: <tglx@linutronix.de>, <mingo@redhat.com>, <bp@alien8.de>,
	<hpa@zytor.com>, <x86@kernel.org>, <boris.ostrovsky@oracle.com>,
	<jgross@suse.com>, <linux-pm@vger.kernel.org>,
	<linux-mm@kvack.org>, <kamatam@amazon.com>,
	<sstabellini@kernel.org>, <konrad.wilk@oracle.com>,
	<roger.pau@citrix.com>, <axboe@kernel.dk>, <davem@davemloft.net>,
	<rjw@rjwysocki.net>, <len.brown@intel.com>, <pavel@ucw.cz>,
	<peterz@infradead.org>, <eduval@amazon.com>, <sblbir@amazon.com>,
	<anchalag@amazon.com>, <xen-devel@lists.xenproject.org>,
	<vkuznets@redhat.com>, <netdev@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <dwmw@amazon.co.uk>,
	<benh@kernel.crashing.org>
Subject: [PATCH 00/12] Fix PM hibernation in Xen guests
Date: Tue, 19 May 2020 23:24:28 +0000	[thread overview]
Message-ID: <cover.1589926004.git.anchalag@amazon.com> (raw)

Hello,
This series fixes PM hibernation for hvm guests running on xen hypervisor.
The running guest could now be hibernated and resumed successfully at a
later time. The fixes for PM hibernation are added to block and
network device drivers i.e xen-blkfront and xen-netfront. Any other driver
that needs to add S4 support if not already, can follow same method of
introducing freeze/thaw/restore callbacks.
The patches had been tested against upstream kernel and xen4.11. Large
scale testing is also done on Xen based Amazon EC2 instances. All this testing
involved running memory exhausting workload in the background.

Doing guest hibernation does not involve any support from hypervisor and
this way guest has complete control over its state. Infrastructure
restrictions for saving up guest state can be overcome by guest initiated
hibernation.

These patches were send out as RFC before and all the feedback had been
incorporated in the patches. The last RFCV3 could be found here:
https://lkml.org/lkml/2020/2/14/2789

Known issues:
1.KASLR causes intermittent hibernation failures. VM fails to resumes and
has to be restarted. I will investigate this issue separately and shouldn't
be a blocker for this patch series.
2. During hibernation, I observed sometimes that freezing of tasks fails due
to busy XFS workqueuei[xfs-cil/xfs-sync]. This is also intermittent may be 1
out of 200 runs and hibernation is aborted in this case. Re-trying hibernation
may work. Also, this is a known issue with hibernation and some
filesystems like XFS has been discussed by the community for years with not an
effectve resolution at this point.

Testing How to:
---------------
1. Setup xen hypervisor on a physical machine[ I used Ubuntu 16.04 +upstream
xen-4.11]
2. Bring up a HVM guest w/t kernel compiled with hibernation patches
[I used ubuntu18.04 netboot bionic images and also Amazon Linux on-prem images].
3. Create a swap file size=RAM size
4. Update grub parameters and reboot
5. Trigger pm-hibernation from within the VM

Example:
Set up a file-backed swap space. Swap file size>=Total memory on the system
sudo dd if=/dev/zero of=/swap bs=$(( 1024 * 1024 )) count=4096 # 4096MiB
sudo chmod 600 /swap
sudo mkswap /swap
sudo swapon /swap

Update resume device/resume offset in grub if using swap file:
resume=/dev/xvda1 resume_offset=200704 no_console_suspend=1

Execute:
--------
sudo pm-hibernate
OR
echo disk > /sys/power/state && echo reboot > /sys/power/disk

Compute resume offset code:
"
#!/usr/bin/env python
import sys
import array
import fcntl

#swap file
f = open(sys.argv[1], 'r')
buf = array.array('L', [0])

#FIBMAP
ret = fcntl.ioctl(f.fileno(), 0x01, buf)
print buf[0]
"


Anchal Agarwal (5):
  x86/xen: Introduce new function to map HYPERVISOR_shared_info on
    Resume
  genirq: Shutdown irq chips in suspend/resume during hibernation
  xen: Introduce wrapper for save/restore sched clock offset
  xen: Update sched clock offset to avoid system instability in
    hibernation
  PM / hibernate: update the resume offset on SNAPSHOT_SET_SWAP_AREA

Munehisa Kamata (7):
  xen/manage: keep track of the on-going suspend mode
  xenbus: add freeze/thaw/restore callbacks support
  x86/xen: add system core suspend and resume callbacks
  xen-blkfront: add callbacks for PM suspend and hibernation
  xen-netfront: add callbacks for PM suspend and hibernation
  xen/time: introduce xen_{save,restore}_steal_clock
  x86/xen: save and restore steal clock

 arch/x86/xen/enlighten_hvm.c      |   8 ++
 arch/x86/xen/suspend.c            |  72 ++++++++++++++++++
 arch/x86/xen/time.c               |  18 ++++-
 arch/x86/xen/xen-ops.h            |   3 +
 drivers/block/xen-blkfront.c      | 122 ++++++++++++++++++++++++++++--
 drivers/net/xen-netfront.c        |  98 +++++++++++++++++++++++-
 drivers/xen/events/events_base.c  |   1 +
 drivers/xen/manage.c              |  73 ++++++++++++++++++
 drivers/xen/time.c                |  29 ++++++-
 drivers/xen/xenbus/xenbus_probe.c |  99 +++++++++++++++++++-----
 include/linux/irq.h               |   2 +
 include/xen/xen-ops.h             |   8 ++
 include/xen/xenbus.h              |   3 +
 kernel/irq/chip.c                 |   2 +-
 kernel/irq/internals.h            |   1 +
 kernel/irq/pm.c                   |  31 +++++---
 kernel/power/user.c               |   6 +-
 17 files changed, 536 insertions(+), 40 deletions(-)

-- 
2.24.1.AMZN



             reply	other threads:[~2020-05-19 23:25 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-19 23:24 Anchal Agarwal [this message]
2020-05-19 23:24 ` [PATCH 01/12] xen/manage: keep track of the on-going suspend mode Anchal Agarwal
2020-05-30 22:26   ` Boris Ostrovsky
2020-06-01 21:00     ` Agarwal, Anchal
2020-06-01 22:39       ` Boris Ostrovsky
2020-05-19 23:25 ` [PATCH 02/12] xenbus: add freeze/thaw/restore callbacks support Anchal Agarwal
2020-05-30 22:56   ` Boris Ostrovsky
2020-06-01 23:36     ` Agarwal, Anchal
2020-05-19 23:25 ` [PATCH 03/12] x86/xen: Introduce new function to map HYPERVISOR_shared_info on Resume Anchal Agarwal
2020-05-30 23:02   ` Boris Ostrovsky
2020-06-04 23:03     ` Anchal Agarwal
2020-06-05 21:39       ` Boris Ostrovsky
2020-06-08 16:52         ` Anchal Agarwal
2020-06-08 18:49           ` Boris Ostrovsky
2020-05-19 23:26 ` [PATCH 04/12] x86/xen: add system core suspend and resume callbacks Anchal Agarwal
2020-05-30 23:10   ` Boris Ostrovsky
2020-06-03 22:40     ` Agarwal, Anchal
2020-06-05 21:24       ` Boris Ostrovsky
2020-06-08 17:09         ` Anchal Agarwal
2020-05-19 23:26 ` [PATCH 05/12] genirq: Shutdown irq chips in suspend/resume during hibernation Anchal Agarwal
2020-05-19 23:29   ` Singh, Balbir
2020-05-19 23:36     ` Agarwal, Anchal
2020-05-19 23:34   ` Anchal Agarwal
2020-05-30 23:17   ` Boris Ostrovsky
2020-06-01 20:46     ` Agarwal, Anchal
2020-05-19 23:27 ` [PATCH 06/12] xen-blkfront: add callbacks for PM suspend and hibernation Anchal Agarwal
2020-05-20  5:00   ` kbuild test robot
2020-05-20  5:07   ` kbuild test robot
2020-05-21 23:48   ` Anchal Agarwal
2020-05-22  1:43     ` Singh, Balbir
2020-05-23 12:32   ` kbuild test robot
2020-05-28 12:30   ` Roger Pau Monné
2020-05-19 23:28 ` [PATCH 07/12] xen-netfront: " Anchal Agarwal
2020-05-19 23:28 ` [PATCH 08/12] xen/time: introduce xen_{save,restore}_steal_clock Anchal Agarwal
2020-05-30 23:32   ` Boris Ostrovsky
2020-05-19 23:28 ` [PATCH 09/12] x86/xen: save and restore steal clock Anchal Agarwal
2020-05-30 23:44   ` Boris Ostrovsky
2020-06-04 18:33     ` Anchal Agarwal
2020-05-19 23:29 ` [PATCH 10/12] xen: Introduce wrapper for save/restore sched clock offset Anchal Agarwal
2020-05-19 23:29 ` [PATCH 11/12] xen: Update sched clock offset to avoid system instability in hibernation Anchal Agarwal
2020-05-19 23:29 ` [PATCH 12/12] PM / hibernate: update the resume offset on SNAPSHOT_SET_SWAP_AREA Anchal Agarwal
2020-05-28 17:59 ` [PATCH 00/12] Fix PM hibernation in Xen guests Agarwal, Anchal

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1589926004.git.anchalag@amazon.com \
    --to=anchalag@amazon.com \
    --cc=axboe@kernel.dk \
    --cc=benh@kernel.crashing.org \
    --cc=boris.ostrovsky@oracle.com \
    --cc=bp@alien8.de \
    --cc=davem@davemloft.net \
    --cc=dwmw@amazon.co.uk \
    --cc=eduval@amazon.com \
    --cc=hpa@zytor.com \
    --cc=jgross@suse.com \
    --cc=kamatam@amazon.com \
    --cc=konrad.wilk@oracle.com \
    --cc=len.brown@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-pm@vger.kernel.org \
    --cc=mingo@redhat.com \
    --cc=netdev@vger.kernel.org \
    --cc=pavel@ucw.cz \
    --cc=peterz@infradead.org \
    --cc=rjw@rjwysocki.net \
    --cc=roger.pau@citrix.com \
    --cc=sblbir@amazon.com \
    --cc=sstabellini@kernel.org \
    --cc=tglx@linutronix.de \
    --cc=vkuznets@redhat.com \
    --cc=x86@kernel.org \
    --cc=xen-devel@lists.xenproject.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).