From: Anchal Agarwal <anchalag@amazon.com>
To: "Rafael J. Wysocki" <rafael@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
"H. Peter Anvin" <hpa@zytor.com>,
"the arch/x86 maintainers" <x86@kernel.org>,
Boris Ostrovsky <boris.ostrovsky@oracle.com>,
Juergen Gross <jgross@suse.com>,
Linux PM <linux-pm@vger.kernel.org>,
Linux Memory Management List <linux-mm@kvack.org>,
"Kamata, Munehisa" <kamatam@amazon.com>,
"Stefano Stabellini" <sstabellini@kernel.org>,
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>,
<roger.pau@citrix.com>, Jens Axboe <axboe@kernel.dk>,
David Miller <davem@davemloft.net>,
"Rafael J. Wysocki" <rjw@rjwysocki.net>,
Len Brown <len.brown@intel.com>, Pavel Machek <pavel@ucw.cz>,
Peter Zijlstra <peterz@infradead.org>,
Eduardo Valentin <eduval@amazon.com>,
"Singh, Balbir" <sblbir@amazon.com>,
<xen-devel@lists.xenproject.org>,
Vitaly Kuznetsov <vkuznets@redhat.com>,
netdev <netdev@vger.kernel.org>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
David Woodhouse <dwmw@amazon.co.uk>,
"Benjamin Herrenschmidt" <benh@kernel.crashing.org>
Subject: Re: [PATCH v3 00/11] Fix PM hibernation in Xen guests
Date: Fri, 11 Sep 2020 20:44:47 +0000 [thread overview]
Message-ID: <20200911204447.GA23729@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com> (raw)
In-Reply-To: <20200828183945.GA22160@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com>
On Fri, Aug 28, 2020 at 06:39:45PM +0000, Anchal Agarwal wrote:
> On Fri, Aug 28, 2020 at 08:29:24PM +0200, Rafael J. Wysocki wrote:
> > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> >
> >
> >
> > On Fri, Aug 28, 2020 at 8:26 PM Anchal Agarwal <anchalag@amazon.com> wrote:
> > >
> > > On Fri, Aug 21, 2020 at 10:22:43PM +0000, Anchal Agarwal wrote:
> > > > Hello,
> > > > This series fixes PM hibernation for hvm guests running on xen hypervisor.
> > > > The running guest could now be hibernated and resumed successfully at a
> > > > later time. The fixes for PM hibernation are added to block and
> > > > network device drivers i.e xen-blkfront and xen-netfront. Any other driver
> > > > that needs to add S4 support if not already, can follow same method of
> > > > introducing freeze/thaw/restore callbacks.
> > > > The patches had been tested against upstream kernel and xen4.11. Large
> > > > scale testing is also done on Xen based Amazon EC2 instances. All this testing
> > > > involved running memory exhausting workload in the background.
> > > >
> > > > Doing guest hibernation does not involve any support from hypervisor and
> > > > this way guest has complete control over its state. Infrastructure
> > > > restrictions for saving up guest state can be overcome by guest initiated
> > > > hibernation.
> > > >
> > > > These patches were send out as RFC before and all the feedback had been
> > > > incorporated in the patches. The last v1 & v2 could be found here:
> > > >
> > > > [v1]: https://lkml.org/lkml/2020/5/19/1312
> > > > [v2]: https://lkml.org/lkml/2020/7/2/995
> > > > All comments and feedback from v2 had been incorporated in v3 series.
> > > >
> > > > Known issues:
> > > > 1.KASLR causes intermittent hibernation failures. VM fails to resumes and
> > > > has to be restarted. I will investigate this issue separately and shouldn't
> > > > be a blocker for this patch series.
> > > > 2. During hibernation, I observed sometimes that freezing of tasks fails due
> > > > to busy XFS workqueuei[xfs-cil/xfs-sync]. This is also intermittent may be 1
> > > > out of 200 runs and hibernation is aborted in this case. Re-trying hibernation
> > > > may work. Also, this is a known issue with hibernation and some
> > > > filesystems like XFS has been discussed by the community for years with not an
> > > > effectve resolution at this point.
> > > >
> > > > Testing How to:
> > > > ---------------
> > > > 1. Setup xen hypervisor on a physical machine[ I used Ubuntu 16.04 +upstream
> > > > xen-4.11]
> > > > 2. Bring up a HVM guest w/t kernel compiled with hibernation patches
> > > > [I used ubuntu18.04 netboot bionic images and also Amazon Linux on-prem images].
> > > > 3. Create a swap file size=RAM size
> > > > 4. Update grub parameters and reboot
> > > > 5. Trigger pm-hibernation from within the VM
> > > >
> > > > Example:
> > > > Set up a file-backed swap space. Swap file size>=Total memory on the system
> > > > sudo dd if=/dev/zero of=/swap bs=$(( 1024 * 1024 )) count=4096 # 4096MiB
> > > > sudo chmod 600 /swap
> > > > sudo mkswap /swap
> > > > sudo swapon /swap
> > > >
> > > > Update resume device/resume offset in grub if using swap file:
> > > > resume=/dev/xvda1 resume_offset=200704 no_console_suspend=1
> > > >
> > > > Execute:
> > > > --------
> > > > sudo pm-hibernate
> > > > OR
> > > > echo disk > /sys/power/state && echo reboot > /sys/power/disk
> > > >
> > > > Compute resume offset code:
> > > > "
> > > > #!/usr/bin/env python
> > > > import sys
> > > > import array
> > > > import fcntl
> > > >
> > > > #swap file
> > > > f = open(sys.argv[1], 'r')
> > > > buf = array.array('L', [0])
> > > >
> > > > #FIBMAP
> > > > ret = fcntl.ioctl(f.fileno(), 0x01, buf)
> > > > print buf[0]
> > > > "
> > > >
> > > > Aleksei Besogonov (1):
> > > > PM / hibernate: update the resume offset on SNAPSHOT_SET_SWAP_AREA
> > > >
> > > > Anchal Agarwal (4):
> > > > x86/xen: Introduce new function to map HYPERVISOR_shared_info on
> > > > Resume
> > > > x86/xen: save and restore steal clock during PM hibernation
> > > > xen: Introduce wrapper for save/restore sched clock offset
> > > > xen: Update sched clock offset to avoid system instability in
> > > > hibernation
> > > >
> > > > Munehisa Kamata (5):
> > > > xen/manage: keep track of the on-going suspend mode
> > > > xenbus: add freeze/thaw/restore callbacks support
> > > > x86/xen: add system core suspend and resume callbacks
> > > > xen-blkfront: add callbacks for PM suspend and hibernation
> > > > xen-netfront: add callbacks for PM suspend and hibernation
> > > >
> > > > Thomas Gleixner (1):
> > > > genirq: Shutdown irq chips in suspend/resume during hibernation
> > > >
> > > > arch/x86/xen/enlighten_hvm.c | 7 +++
> > > > arch/x86/xen/suspend.c | 63 ++++++++++++++++++++
> > > > arch/x86/xen/time.c | 15 ++++-
> > > > arch/x86/xen/xen-ops.h | 3 +
> > > > drivers/block/xen-blkfront.c | 122 ++++++++++++++++++++++++++++++++++++--
> > > > drivers/net/xen-netfront.c | 96 +++++++++++++++++++++++++++++-
> > > > drivers/xen/events/events_base.c | 1 +
> > > > drivers/xen/manage.c | 46 ++++++++++++++
> > > > drivers/xen/xenbus/xenbus_probe.c | 96 +++++++++++++++++++++++++-----
> > > > include/linux/irq.h | 2 +
> > > > include/xen/xen-ops.h | 3 +
> > > > include/xen/xenbus.h | 3 +
> > > > kernel/irq/chip.c | 2 +-
> > > > kernel/irq/internals.h | 1 +
> > > > kernel/irq/pm.c | 31 +++++++---
> > > > kernel/power/user.c | 7 ++-
> > > > 16 files changed, 464 insertions(+), 34 deletions(-)
> > > >
> > > > --
> > > > 2.16.6
> > > >
> > > A gentle ping on the series in case there is any more feedback or can we plan to
> > > merge this? I can then send the series with minor fixes pointed by tglx@
> >
> > Some more time, please!
> >
> Sure happy to answer any more questions and fix more BUGS!!
>
> --
> Anchal
>
A gentle ping on this one again!
--
Anchal
next prev parent reply other threads:[~2020-09-11 20:45 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-21 22:22 [PATCH v3 00/11] Fix PM hibernation in Xen guests Anchal Agarwal
2020-08-21 22:25 ` [PATCH v3 01/11] xen/manage: keep track of the on-going suspend mode Anchal Agarwal
2020-09-13 15:43 ` boris.ostrovsky
2020-09-14 21:47 ` Anchal Agarwal
2020-09-15 0:24 ` boris.ostrovsky
2020-09-15 18:00 ` Anchal Agarwal
2020-09-15 19:58 ` boris.ostrovsky
2020-09-21 21:54 ` Anchal Agarwal
2020-09-22 16:18 ` boris.ostrovsky
2020-09-22 23:17 ` Anchal Agarwal
[not found] ` <20200925190423.GA31885@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com>
2020-09-25 20:02 ` boris.ostrovsky
2020-09-25 22:28 ` Anchal Agarwal
2020-09-28 18:49 ` boris.ostrovsky
2020-09-30 21:29 ` Anchal Agarwal
2020-10-01 12:43 ` boris.ostrovsky
2021-05-21 5:26 ` Anchal Agarwal
2021-05-25 22:23 ` Boris Ostrovsky
2021-05-26 4:40 ` Anchal Agarwal
2021-05-26 18:29 ` Boris Ostrovsky
[not found] ` <20210528215008.GA19622@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com>
2021-06-01 14:18 ` Boris Ostrovsky
2021-06-02 19:37 ` Anchal Agarwal
2021-06-03 20:11 ` Boris Ostrovsky
2021-06-03 23:27 ` Anchal Agarwal
2021-06-04 1:49 ` Boris Ostrovsky
2020-09-13 17:07 ` boris.ostrovsky
2020-08-21 22:26 ` [PATCH v3 02/11] xenbus: add freeze/thaw/restore callbacks support Anchal Agarwal
2020-09-13 16:11 ` boris.ostrovsky
2020-09-15 19:56 ` Anchal Agarwal
2020-08-21 22:26 ` [PATCH v3 03/11] x86/xen: Introduce new function to map HYPERVISOR_shared_info on Resume Anchal Agarwal
2020-08-21 22:27 ` [PATCH v3 04/11] x86/xen: add system core suspend and resume callbacks Anchal Agarwal
2020-09-13 17:25 ` boris.ostrovsky
2020-08-21 22:27 ` [PATCH v3 05/11] genirq: Shutdown irq chips in suspend/resume during hibernation Thomas Gleixner
2020-08-22 0:36 ` Thomas Gleixner
2020-08-24 17:25 ` Anchal Agarwal
2020-08-25 13:20 ` Christoph Hellwig
2020-08-25 15:25 ` Thomas Gleixner
2020-08-21 22:28 ` [PATCH v3 06/11] xen-blkfront: add callbacks for PM suspend and hibernation Anchal Agarwal
2020-08-21 22:29 ` [PATCH v3 07/11] xen-netfront: " Anchal Agarwal
2020-08-21 22:29 ` [PATCH v3 08/11] x86/xen: save and restore steal clock during PM hibernation Anchal Agarwal
2020-08-21 22:30 ` [PATCH v3 09/11] xen: Introduce wrapper for save/restore sched clock offset Anchal Agarwal
2020-08-21 22:30 ` [PATCH v3 10/11] xen: Update sched clock offset to avoid system instability in hibernation Anchal Agarwal
2020-09-13 17:52 ` boris.ostrovsky
2020-08-21 22:31 ` [PATCH v3 11/11] PM / hibernate: update the resume offset on SNAPSHOT_SET_SWAP_AREA Anchal Agarwal
2020-08-28 18:26 ` [PATCH v3 00/11] Fix PM hibernation in Xen guests Anchal Agarwal
2020-08-28 18:29 ` Rafael J. Wysocki
2020-08-28 18:39 ` Anchal Agarwal
2020-09-11 20:44 ` Anchal Agarwal [this message]
2020-09-11 15:19 ` boris.ostrovsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200911204447.GA23729@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com \
--to=anchalag@amazon.com \
--cc=axboe@kernel.dk \
--cc=benh@kernel.crashing.org \
--cc=boris.ostrovsky@oracle.com \
--cc=bp@alien8.de \
--cc=davem@davemloft.net \
--cc=dwmw@amazon.co.uk \
--cc=eduval@amazon.com \
--cc=hpa@zytor.com \
--cc=jgross@suse.com \
--cc=kamatam@amazon.com \
--cc=konrad.wilk@oracle.com \
--cc=len.brown@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-pm@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=netdev@vger.kernel.org \
--cc=pavel@ucw.cz \
--cc=peterz@infradead.org \
--cc=rafael@kernel.org \
--cc=rjw@rjwysocki.net \
--cc=roger.pau@citrix.com \
--cc=sblbir@amazon.com \
--cc=sstabellini@kernel.org \
--cc=tglx@linutronix.de \
--cc=vkuznets@redhat.com \
--cc=x86@kernel.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).