[PATCH 0/2] clarify limitations of hibernation

[PATCH 0/2] clarify limitations of hibernation

[Luigi Semenzato]

These patches aim to make it clearer under which circumstances
hibernation will function as expected.  They include one documentation
change, and one change which logs more information on failure to
enter hibernation.

Luigi Semenzato (2):
  Documentation: clarify limitations of hibernation
  pm: add more logging on hibernation failure

 Documentation/admin-guide/pm/sleep-states.rst | 18 +++++++++++++++++-
 kernel/power/snapshot.c                       | 18 ++++++++++++------
 2 files changed, 29 insertions(+), 7 deletions(-)

-- 
2.24.1.735.g03f4e72817-goog

[PATCH 1/2] Documentation: clarify limitations of hibernation

[Luigi Semenzato]

Entering hibernation (suspend-to-disk) will fail if the kernel
cannot allocate enough memory to create a snapshot of all pages
in use; i.e., if memory in use is over 1/2 of total RAM.  This
patch makes this limitation clearer in the documentation.  Without
it, users may assume that hibernation can replace suspend-to-RAM
when in fact its functionality is more limited.

Signed-off-by: Luigi Semenzato <semenzato@google.com>
---
 Documentation/admin-guide/pm/sleep-states.rst | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/Documentation/admin-guide/pm/sleep-states.rst b/Documentation/admin-guide/pm/sleep-states.rst
index cd3a28cb81f4..fd0072eb8c03 100644
--- a/Documentation/admin-guide/pm/sleep-states.rst
+++ b/Documentation/admin-guide/pm/sleep-states.rst
@@ -112,7 +112,9 @@ Hibernation
 This state (also referred to as Suspend-to-Disk or STD) offers the greatest
 energy savings and can be used even in the absence of low-level platform support
 for system suspend.  However, it requires some low-level code for resuming the
-system to be present for the underlying CPU architecture.
+system to be present for the underlying CPU architecture.  Additionally, the
+current implementation can enter the hibernation state only when memory
+pressure is low (see "Limitations" below).
 
 Hibernation is significantly different from any of the system suspend variants.
 It takes three system state changes to put it into hibernation and two system
@@ -149,6 +151,20 @@ Hibernation is supported if the :c:macro:`CONFIG_HIBERNATION` kernel
 configuration option is set.  However, this option can only be set if support
 for the given CPU architecture includes the low-level code for system resume.
 
+Limitations of Hibernation
+==========================
+
+When entering hibernation, the kernel tries to allocate a chunk of memory large
+enough to contain a copy of all pages in use, to use it for the system
+snapshot.  If the allocation fails, the system cannot hibernate and the
+operation fails with ENOMEM.  This will happen, for instance, when the total
+amount of anonymous pages (process data) exceeds 1/2 of total RAM.
+
+One possible workaround (besides terminating enough processes) is to force
+excess anonymous pages out to swap before hibernating.  This can be achieved
+with memcgroups, by lowering memory usage limits with ``echo <new limit> >
+/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``.  However, the latter
+operation is not guaranteed to succeed.
 
 Basic ``sysfs`` Interfaces for System Suspend and Hibernation
 =============================================================
-- 
2.24.1.735.g03f4e72817-goog

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

[Michal Hocko]

On Thu 26-12-19 14:02:04, Luigi Semenzato wrote:
[...]
> +Limitations of Hibernation
> +==========================
> +
> +When entering hibernation, the kernel tries to allocate a chunk of memory large
> +enough to contain a copy of all pages in use, to use it for the system
> +snapshot.  If the allocation fails, the system cannot hibernate and the
> +operation fails with ENOMEM.  This will happen, for instance, when the total
> +amount of anonymous pages (process data) exceeds 1/2 of total RAM.
> +
> +One possible workaround (besides terminating enough processes) is to force
> +excess anonymous pages out to swap before hibernating.  This can be achieved
> +with memcgroups, by lowering memory usage limits with ``echo <new limit> >
> +/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``.  However, the latter
> +operation is not guaranteed to succeed.

I am not familiar with the hibernation process much. But what prevents
those allocations to reclaim memory and push out the anonymous memory to
the swap on demand during the hibernation's allocations?
-- 
Michal Hocko
SUSE Labs

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

[Luigi Semenzato]

On Mon, Jan 6, 2020 at 4:53 AM Michal Hocko <mhocko@kernel.org> wrote:
>
> On Thu 26-12-19 14:02:04, Luigi Semenzato wrote:
> [...]
> > +Limitations of Hibernation
> > +==========================
> > +
> > +When entering hibernation, the kernel tries to allocate a chunk of memory large
> > +enough to contain a copy of all pages in use, to use it for the system
> > +snapshot.  If the allocation fails, the system cannot hibernate and the
> > +operation fails with ENOMEM.  This will happen, for instance, when the total
> > +amount of anonymous pages (process data) exceeds 1/2 of total RAM.
> > +
> > +One possible workaround (besides terminating enough processes) is to force
> > +excess anonymous pages out to swap before hibernating.  This can be achieved
> > +with memcgroups, by lowering memory usage limits with ``echo <new limit> >
> > +/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``.  However, the latter
> > +operation is not guaranteed to succeed.
>
> I am not familiar with the hibernation process much. But what prevents
> those allocations to reclaim memory and push out the anonymous memory to
> the swap on demand during the hibernation's allocations?

Good question, thanks.

The hibernation image is stored into a swap device (or partition), so
I suppose one could set up two swap devices, giving a lower priority
to the hibernation device, so that it remains unused while the kernel
reclaims pages for the hibernation image.

If that works, then it may be appropriate to describe this technique
in Documentation/power/swsusp.rst.  There's a brief mention of this
situation in the Q/A section, but maybe this deserves more visibility.

In my experience, the page allocator is prone to giving up in this
kind of situation.  But my experience is up to 4.X kernels.  Is this
guaranteed to work now?

Note that I removed the cgroup suggestion in later versions of this patch:
https://marc.info/?l=linux-pm&m=157800718520897

Also, there was some related discussion here:
https://marc.info/?l=linux-kernel&m=157177497015315

Thanks!



> --
> Michal Hocko
> SUSE Labs

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

[Michal Hocko]

On Mon 06-01-20 11:08:56, Luigi Semenzato wrote:
> On Mon, Jan 6, 2020 at 4:53 AM Michal Hocko <mhocko@kernel.org> wrote:
> >
> > On Thu 26-12-19 14:02:04, Luigi Semenzato wrote:
> > [...]
> > > +Limitations of Hibernation
> > > +==========================
> > > +
> > > +When entering hibernation, the kernel tries to allocate a chunk of memory large
> > > +enough to contain a copy of all pages in use, to use it for the system
> > > +snapshot.  If the allocation fails, the system cannot hibernate and the
> > > +operation fails with ENOMEM.  This will happen, for instance, when the total
> > > +amount of anonymous pages (process data) exceeds 1/2 of total RAM.
> > > +
> > > +One possible workaround (besides terminating enough processes) is to force
> > > +excess anonymous pages out to swap before hibernating.  This can be achieved
> > > +with memcgroups, by lowering memory usage limits with ``echo <new limit> >
> > > +/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``.  However, the latter
> > > +operation is not guaranteed to succeed.
> >
> > I am not familiar with the hibernation process much. But what prevents
> > those allocations to reclaim memory and push out the anonymous memory to
> > the swap on demand during the hibernation's allocations?
> 
> Good question, thanks.
> 
> The hibernation image is stored into a swap device (or partition), so
> I suppose one could set up two swap devices, giving a lower priority
> to the hibernation device, so that it remains unused while the kernel
> reclaims pages for the hibernation image.

I do not think hibernation can choose which swap device to use. Having
an additional swap device migh help though because there will be more
space to swap out to.

> If that works, then it may be appropriate to describe this technique
> in Documentation/power/swsusp.rst.  There's a brief mention of this
> situation in the Q/A section, but maybe this deserves more visibility.
> 
> In my experience, the page allocator is prone to giving up in this
> kind of situation.  But my experience is up to 4.X kernels.  Is this
> guaranteed to work now?

OK, I can see it now. I forgot about the ugly hack in the page allocator
that hibernation is using. If there is no way to make a forward progress
for the allocation and we enter the allocator oom path (__alloc_pages_may_oom)
pm_suspended_storage() bails out early and the allocator gives up.

That being said allocator would swap out processes so it doesn't make
much sense to do that pro-actively. It can still fail if the swap is
depleted though and then the hibernation gives up. This makes some sense
because you wouldn't like to have something killed by the oom killer
while hibernating right? Graceful failure should be a preferable action
and let you decide what to do IMHO.
-- 
Michal Hocko
SUSE Labs

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

[Luigi Semenzato]

Thank you all for the replies and sorry for the delay (vacation + flu).

This has given me various ideas for experiments and I will try to get
to them in the future.  For now, the cgroup workaround (described in
my first version of the patch, but removed later) will do for us.

The purpose of my documentation patch was to make it clearer that
hibernation may fail in situations in which suspend-to-RAM works; for
instance, when there is no swap, and anonymous pages are over 50% of
total RAM.  I will send a new version of the patch which hopefully
makes this clearer.

From this discussion, it seems that it should be possible to set up
swap and hibernation in a way that increases the probability of
success when entering hibernation (or maybe make it a certainty?).  It
would be useful to include such setup in the documentation.  I don't
know how to do this (yet) but if anybody does, it would be a great
contribution.

Thanks!

On Wed, Jan 8, 2020 at 3:49 AM Michal Hocko <mhocko@kernel.org> wrote:
>
> On Mon 06-01-20 11:08:56, Luigi Semenzato wrote:
> > On Mon, Jan 6, 2020 at 4:53 AM Michal Hocko <mhocko@kernel.org> wrote:
> > >
> > > On Thu 26-12-19 14:02:04, Luigi Semenzato wrote:
> > > [...]
> > > > +Limitations of Hibernation
> > > > +==========================
> > > > +
> > > > +When entering hibernation, the kernel tries to allocate a chunk of memory large
> > > > +enough to contain a copy of all pages in use, to use it for the system
> > > > +snapshot.  If the allocation fails, the system cannot hibernate and the
> > > > +operation fails with ENOMEM.  This will happen, for instance, when the total
> > > > +amount of anonymous pages (process data) exceeds 1/2 of total RAM.
> > > > +
> > > > +One possible workaround (besides terminating enough processes) is to force
> > > > +excess anonymous pages out to swap before hibernating.  This can be achieved
> > > > +with memcgroups, by lowering memory usage limits with ``echo <new limit> >
> > > > +/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``.  However, the latter
> > > > +operation is not guaranteed to succeed.
> > >
> > > I am not familiar with the hibernation process much. But what prevents
> > > those allocations to reclaim memory and push out the anonymous memory to
> > > the swap on demand during the hibernation's allocations?
> >
> > Good question, thanks.
> >
> > The hibernation image is stored into a swap device (or partition), so
> > I suppose one could set up two swap devices, giving a lower priority
> > to the hibernation device, so that it remains unused while the kernel
> > reclaims pages for the hibernation image.
>
> I do not think hibernation can choose which swap device to use. Having
> an additional swap device migh help though because there will be more
> space to swap out to.
>
> > If that works, then it may be appropriate to describe this technique
> > in Documentation/power/swsusp.rst.  There's a brief mention of this
> > situation in the Q/A section, but maybe this deserves more visibility.
> >
> > In my experience, the page allocator is prone to giving up in this
> > kind of situation.  But my experience is up to 4.X kernels.  Is this
> > guaranteed to work now?
>
> OK, I can see it now. I forgot about the ugly hack in the page allocator
> that hibernation is using. If there is no way to make a forward progress
> for the allocation and we enter the allocator oom path (__alloc_pages_may_oom)
> pm_suspended_storage() bails out early and the allocator gives up.
>
> That being said allocator would swap out processes so it doesn't make
> much sense to do that pro-actively. It can still fail if the swap is
> depleted though and then the hibernation gives up. This makes some sense
> because you wouldn't like to have something killed by the oom killer
> while hibernating right? Graceful failure should be a preferable action
> and let you decide what to do IMHO.
> --
> Michal Hocko
> SUSE Labs

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

[Michal Hocko]

On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]
> The purpose of my documentation patch was to make it clearer that
> hibernation may fail in situations in which suspend-to-RAM works; for
> instance, when there is no swap, and anonymous pages are over 50% of
> total RAM.  I will send a new version of the patch which hopefully
> makes this clearer.

I was under impression that s2disk is pretty much impossible without any
swap.
-- 
Michal Hocko
SUSE Labs

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

[Luigi Semenzato]

On Mon, Jan 27, 2020 at 6:16 AM Michal Hocko <mhocko@kernel.org> wrote:
>
> On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
> [...]
> > The purpose of my documentation patch was to make it clearer that
> > hibernation may fail in situations in which suspend-to-RAM works; for
> > instance, when there is no swap, and anonymous pages are over 50% of
> > total RAM.  I will send a new version of the patch which hopefully
> > makes this clearer.
>
> I was under impression that s2disk is pretty much impossible without any
> swap.

I am not sure what you mean by "swap" here.  S2disk needs a swap
partition for storing the image, but that partition is not used for
regular swap.  If there is no swap, but more than 50% of RAM is free
or reclaimable, s2disk works fine.  If anonymous is more than 50%,
hibernation can still work, but swap needs to be set up (in addition
to the space for the hibernation image).  The setup is not obvious and
I don't think that the documentation is clear on this.

> --
> Michal Hocko
> SUSE Labs

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

[Rafael J. Wysocki]

On Mon, Jan 27, 2020 at 5:13 PM Luigi Semenzato <semenzato@google.com> wrote:
>
> On Mon, Jan 27, 2020 at 6:16 AM Michal Hocko <mhocko@kernel.org> wrote:
> >
> > On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
> > [...]
> > > The purpose of my documentation patch was to make it clearer that
> > > hibernation may fail in situations in which suspend-to-RAM works; for
> > > instance, when there is no swap, and anonymous pages are over 50% of
> > > total RAM.  I will send a new version of the patch which hopefully
> > > makes this clearer.
> >
> > I was under impression that s2disk is pretty much impossible without any
> > swap.
>
> I am not sure what you mean by "swap" here.  S2disk needs a swap
> partition for storing the image, but that partition is not used for
> regular swap.

That's not correct.

The swap partition (or file) used by s2disk needs to be made active
before it can use it and the mm subsystem is also able to use it for
regular swap then.

>  If there is no swap, but more than 50% of RAM is free
> or reclaimable, s2disk works fine.  If anonymous is more than 50%,
> hibernation can still work, but swap needs to be set up (in addition
> to the space for the hibernation image).  The setup is not obvious and
> I don't think that the documentation is clear on this.

Well, the entire contents of RAM must be preserved, this way or
another, during hibernation.  That should be totally obvious to anyone
using it really.

Some of the RAM contents is copies of data already there in the
filesystems on persistent storage and that does not need to be saved
again.  Everything else must be saved and s2disk (and Linux
hibernation in general) uses active swap space to save these things.
This implies that in order to hibernate the system, you generally need
the amount of swap space equal to the size of RAM minus the size of
files mapped into memory.

So, to be on the safe side, the total amount of swap space to be used
for hibernation needs to match the size of RAM (even though
realistically it may be smaller than that in the majority of cases).

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

[Luigi Semenzato]

On Mon, Jan 27, 2020 at 8:28 AM Rafael J. Wysocki <rafael@kernel.org> wrote:
>
> On Mon, Jan 27, 2020 at 5:13 PM Luigi Semenzato <semenzato@google.com> wrote:
> >
> > On Mon, Jan 27, 2020 at 6:16 AM Michal Hocko <mhocko@kernel.org> wrote:
> > >
> > > On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
> > > [...]
> > > > The purpose of my documentation patch was to make it clearer that
> > > > hibernation may fail in situations in which suspend-to-RAM works; for
> > > > instance, when there is no swap, and anonymous pages are over 50% of
> > > > total RAM.  I will send a new version of the patch which hopefully
> > > > makes this clearer.
> > >
> > > I was under impression that s2disk is pretty much impossible without any
> > > swap.
> >
> > I am not sure what you mean by "swap" here.  S2disk needs a swap
> > partition for storing the image, but that partition is not used for
> > regular swap.
>
> That's not correct.
>
> The swap partition (or file) used by s2disk needs to be made active
> before it can use it and the mm subsystem is also able to use it for
> regular swap then.

OK---I had this wrong, thanks.

> >  If there is no swap, but more than 50% of RAM is free
> > or reclaimable, s2disk works fine.  If anonymous is more than 50%,
> > hibernation can still work, but swap needs to be set up (in addition
> > to the space for the hibernation image).  The setup is not obvious and
> > I don't think that the documentation is clear on this.
>
> Well, the entire contents of RAM must be preserved, this way or
> another, during hibernation.  That should be totally obvious to anyone
> using it really.

Yes, that's obvious.

> Some of the RAM contents is copies of data already there in the
> filesystems on persistent storage and that does not need to be saved
> again.  Everything else must be saved and s2disk (and Linux
> hibernation in general) uses active swap space to save these things.
> This implies that in order to hibernate the system, you generally need
> the amount of swap space equal to the size of RAM minus the size of
> files mapped into memory.
>
> So, to be on the safe side, the total amount of swap space to be used
> for hibernation needs to match the size of RAM (even though
> realistically it may be smaller than that in the majority of cases).

This all makes sense, but we do this:

-- add resume=/dev/sdc to the command line
-- attach a disk (/dev/sdc) with size equal to RAM
-- mkswap /dev/sdc
-- swapon /dev/sdc
-- echo disk > /sys/power/state

and the last operation fails with ENOMEM.  Are we doing something
wrong?  Are we hitting some other mm bug?

Thanks!

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

[Rafael J. Wysocki]

On Mon, Jan 27, 2020 at 6:21 PM Luigi Semenzato <semenzato@google.com> wrote:
>
> On Mon, Jan 27, 2020 at 8:28 AM Rafael J. Wysocki <rafael@kernel.org> wrote:
> >
> > On Mon, Jan 27, 2020 at 5:13 PM Luigi Semenzato <semenzato@google.com> wrote:
> > >
> > > On Mon, Jan 27, 2020 at 6:16 AM Michal Hocko <mhocko@kernel.org> wrote:
> > > >
> > > > On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
> > > > [...]
> > > > > The purpose of my documentation patch was to make it clearer that
> > > > > hibernation may fail in situations in which suspend-to-RAM works; for
> > > > > instance, when there is no swap, and anonymous pages are over 50% of
> > > > > total RAM.  I will send a new version of the patch which hopefully
> > > > > makes this clearer.
> > > >
> > > > I was under impression that s2disk is pretty much impossible without any
> > > > swap.
> > >
> > > I am not sure what you mean by "swap" here.  S2disk needs a swap
> > > partition for storing the image, but that partition is not used for
> > > regular swap.
> >
> > That's not correct.
> >
> > The swap partition (or file) used by s2disk needs to be made active
> > before it can use it and the mm subsystem is also able to use it for
> > regular swap then.
>
> OK---I had this wrong, thanks.
>
> > >  If there is no swap, but more than 50% of RAM is free
> > > or reclaimable, s2disk works fine.  If anonymous is more than 50%,
> > > hibernation can still work, but swap needs to be set up (in addition
> > > to the space for the hibernation image).  The setup is not obvious and
> > > I don't think that the documentation is clear on this.
> >
> > Well, the entire contents of RAM must be preserved, this way or
> > another, during hibernation.  That should be totally obvious to anyone
> > using it really.
>
> Yes, that's obvious.
>
> > Some of the RAM contents is copies of data already there in the
> > filesystems on persistent storage and that does not need to be saved
> > again.  Everything else must be saved and s2disk (and Linux
> > hibernation in general) uses active swap space to save these things.
> > This implies that in order to hibernate the system, you generally need
> > the amount of swap space equal to the size of RAM minus the size of
> > files mapped into memory.
> >
> > So, to be on the safe side, the total amount of swap space to be used
> > for hibernation needs to match the size of RAM (even though
> > realistically it may be smaller than that in the majority of cases).
>
> This all makes sense, but we do this:
>
> -- add resume=/dev/sdc to the command line
> -- attach a disk (/dev/sdc) with size equal to RAM
> -- mkswap /dev/sdc
> -- swapon /dev/sdc
> -- echo disk > /sys/power/state
>
> and the last operation fails with ENOMEM.  Are we doing something
> wrong?  Are we hitting some other mm bug?

I would expect this to work, so the fact that it doesn't work for you
indicates a bug somewhere or at least an assumption that doesn't hold.

Can you please remind me what you do to trigger the unexpected behavior?

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

[Luigi Semenzato]

On Thu, Jan 30, 2020 at 12:50 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
>
> On Mon, Jan 27, 2020 at 6:21 PM Luigi Semenzato <semenzato@google.com> wrote:
> >
> > On Mon, Jan 27, 2020 at 8:28 AM Rafael J. Wysocki <rafael@kernel.org> wrote:
> > >
> > > On Mon, Jan 27, 2020 at 5:13 PM Luigi Semenzato <semenzato@google.com> wrote:
> > > >
> > > > On Mon, Jan 27, 2020 at 6:16 AM Michal Hocko <mhocko@kernel.org> wrote:
> > > > >
> > > > > On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
> > > > > [...]
> > > > > > The purpose of my documentation patch was to make it clearer that
> > > > > > hibernation may fail in situations in which suspend-to-RAM works; for
> > > > > > instance, when there is no swap, and anonymous pages are over 50% of
> > > > > > total RAM.  I will send a new version of the patch which hopefully
> > > > > > makes this clearer.
> > > > >
> > > > > I was under impression that s2disk is pretty much impossible without any
> > > > > swap.
> > > >
> > > > I am not sure what you mean by "swap" here.  S2disk needs a swap
> > > > partition for storing the image, but that partition is not used for
> > > > regular swap.
> > >
> > > That's not correct.
> > >
> > > The swap partition (or file) used by s2disk needs to be made active
> > > before it can use it and the mm subsystem is also able to use it for
> > > regular swap then.
> >
> > OK---I had this wrong, thanks.
> >
> > > >  If there is no swap, but more than 50% of RAM is free
> > > > or reclaimable, s2disk works fine.  If anonymous is more than 50%,
> > > > hibernation can still work, but swap needs to be set up (in addition
> > > > to the space for the hibernation image).  The setup is not obvious and
> > > > I don't think that the documentation is clear on this.
> > >
> > > Well, the entire contents of RAM must be preserved, this way or
> > > another, during hibernation.  That should be totally obvious to anyone
> > > using it really.
> >
> > Yes, that's obvious.
> >
> > > Some of the RAM contents is copies of data already there in the
> > > filesystems on persistent storage and that does not need to be saved
> > > again.  Everything else must be saved and s2disk (and Linux
> > > hibernation in general) uses active swap space to save these things.
> > > This implies that in order to hibernate the system, you generally need
> > > the amount of swap space equal to the size of RAM minus the size of
> > > files mapped into memory.
> > >
> > > So, to be on the safe side, the total amount of swap space to be used
> > > for hibernation needs to match the size of RAM (even though
> > > realistically it may be smaller than that in the majority of cases).
> >
> > This all makes sense, but we do this:
> >
> > -- add resume=/dev/sdc to the command line
> > -- attach a disk (/dev/sdc) with size equal to RAM
> > -- mkswap /dev/sdc
> > -- swapon /dev/sdc
> > -- echo disk > /sys/power/state
> >
> > and the last operation fails with ENOMEM.  Are we doing something
> > wrong?  Are we hitting some other mm bug?
>
> I would expect this to work, so the fact that it doesn't work for you
> indicates a bug somewhere or at least an assumption that doesn't hold.
>
> Can you please remind me what you do to trigger the unexpected behavior?

Yes, I create processes that use a large amount of anon memory, more
than 50% of RAM, like this:

dd if=/dev/zero bs=1G count=1 | sleep infinity

I think dd has a 2 GB limit, or around that number, so you'll need a
few of those.

Thanks!

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

[Rafael J. Wysocki]

On Thu, Jan 30, 2020 at 10:11 PM Luigi Semenzato <semenzato@google.com> wrote:
>
> On Thu, Jan 30, 2020 at 12:50 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
> >
> > On Mon, Jan 27, 2020 at 6:21 PM Luigi Semenzato <semenzato@google.com> wrote:
> > >
> > > On Mon, Jan 27, 2020 at 8:28 AM Rafael J. Wysocki <rafael@kernel.org> wrote:
> > > >
> > > > On Mon, Jan 27, 2020 at 5:13 PM Luigi Semenzato <semenzato@google.com> wrote:
> > > > >
> > > > > On Mon, Jan 27, 2020 at 6:16 AM Michal Hocko <mhocko@kernel.org> wrote:
> > > > > >
> > > > > > On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
> > > > > > [...]
> > > > > > > The purpose of my documentation patch was to make it clearer that
> > > > > > > hibernation may fail in situations in which suspend-to-RAM works; for
> > > > > > > instance, when there is no swap, and anonymous pages are over 50% of
> > > > > > > total RAM.  I will send a new version of the patch which hopefully
> > > > > > > makes this clearer.
> > > > > >
> > > > > > I was under impression that s2disk is pretty much impossible without any
> > > > > > swap.
> > > > >
> > > > > I am not sure what you mean by "swap" here.  S2disk needs a swap
> > > > > partition for storing the image, but that partition is not used for
> > > > > regular swap.
> > > >
> > > > That's not correct.
> > > >
> > > > The swap partition (or file) used by s2disk needs to be made active
> > > > before it can use it and the mm subsystem is also able to use it for
> > > > regular swap then.
> > >
> > > OK---I had this wrong, thanks.
> > >
> > > > >  If there is no swap, but more than 50% of RAM is free
> > > > > or reclaimable, s2disk works fine.  If anonymous is more than 50%,
> > > > > hibernation can still work, but swap needs to be set up (in addition
> > > > > to the space for the hibernation image).  The setup is not obvious and
> > > > > I don't think that the documentation is clear on this.
> > > >
> > > > Well, the entire contents of RAM must be preserved, this way or
> > > > another, during hibernation.  That should be totally obvious to anyone
> > > > using it really.
> > >
> > > Yes, that's obvious.
> > >
> > > > Some of the RAM contents is copies of data already there in the
> > > > filesystems on persistent storage and that does not need to be saved
> > > > again.  Everything else must be saved and s2disk (and Linux
> > > > hibernation in general) uses active swap space to save these things.
> > > > This implies that in order to hibernate the system, you generally need
> > > > the amount of swap space equal to the size of RAM minus the size of
> > > > files mapped into memory.
> > > >
> > > > So, to be on the safe side, the total amount of swap space to be used
> > > > for hibernation needs to match the size of RAM (even though
> > > > realistically it may be smaller than that in the majority of cases).
> > >
> > > This all makes sense, but we do this:
> > >
> > > -- add resume=/dev/sdc to the command line
> > > -- attach a disk (/dev/sdc) with size equal to RAM
> > > -- mkswap /dev/sdc
> > > -- swapon /dev/sdc
> > > -- echo disk > /sys/power/state
> > >
> > > and the last operation fails with ENOMEM.  Are we doing something
> > > wrong?  Are we hitting some other mm bug?
> >
> > I would expect this to work, so the fact that it doesn't work for you
> > indicates a bug somewhere or at least an assumption that doesn't hold.
> >
> > Can you please remind me what you do to trigger the unexpected behavior?
>
> Yes, I create processes that use a large amount of anon memory, more
> than 50% of RAM, like this:
>
> dd if=/dev/zero bs=1G count=1 | sleep infinity
>
> I think dd has a 2 GB limit, or around that number, so you'll need a
> few of those.

And then you get -ENOMEM from hibernate_preallocate_memory(), or from
somewhere else?

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

[Luigi Semenzato]

On Thu, Jan 30, 2020 at 1:29 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
>
> On Thu, Jan 30, 2020 at 10:11 PM Luigi Semenzato <semenzato@google.com> wrote:
> >
> > On Thu, Jan 30, 2020 at 12:50 PM Rafael J. Wysocki <rafael@kernel.org> wrote:
> > >
> > > On Mon, Jan 27, 2020 at 6:21 PM Luigi Semenzato <semenzato@google.com> wrote:
> > > >
> > > > On Mon, Jan 27, 2020 at 8:28 AM Rafael J. Wysocki <rafael@kernel.org> wrote:
> > > > >
> > > > > On Mon, Jan 27, 2020 at 5:13 PM Luigi Semenzato <semenzato@google.com> wrote:
> > > > > >
> > > > > > On Mon, Jan 27, 2020 at 6:16 AM Michal Hocko <mhocko@kernel.org> wrote:
> > > > > > >
> > > > > > > On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
> > > > > > > [...]
> > > > > > > > The purpose of my documentation patch was to make it clearer that
> > > > > > > > hibernation may fail in situations in which suspend-to-RAM works; for
> > > > > > > > instance, when there is no swap, and anonymous pages are over 50% of
> > > > > > > > total RAM.  I will send a new version of the patch which hopefully
> > > > > > > > makes this clearer.
> > > > > > >
> > > > > > > I was under impression that s2disk is pretty much impossible without any
> > > > > > > swap.
> > > > > >
> > > > > > I am not sure what you mean by "swap" here.  S2disk needs a swap
> > > > > > partition for storing the image, but that partition is not used for
> > > > > > regular swap.
> > > > >
> > > > > That's not correct.
> > > > >
> > > > > The swap partition (or file) used by s2disk needs to be made active
> > > > > before it can use it and the mm subsystem is also able to use it for
> > > > > regular swap then.
> > > >
> > > > OK---I had this wrong, thanks.
> > > >
> > > > > >  If there is no swap, but more than 50% of RAM is free
> > > > > > or reclaimable, s2disk works fine.  If anonymous is more than 50%,
> > > > > > hibernation can still work, but swap needs to be set up (in addition
> > > > > > to the space for the hibernation image).  The setup is not obvious and
> > > > > > I don't think that the documentation is clear on this.
> > > > >
> > > > > Well, the entire contents of RAM must be preserved, this way or
> > > > > another, during hibernation.  That should be totally obvious to anyone
> > > > > using it really.
> > > >
> > > > Yes, that's obvious.
> > > >
> > > > > Some of the RAM contents is copies of data already there in the
> > > > > filesystems on persistent storage and that does not need to be saved
> > > > > again.  Everything else must be saved and s2disk (and Linux
> > > > > hibernation in general) uses active swap space to save these things.
> > > > > This implies that in order to hibernate the system, you generally need
> > > > > the amount of swap space equal to the size of RAM minus the size of
> > > > > files mapped into memory.
> > > > >
> > > > > So, to be on the safe side, the total amount of swap space to be used
> > > > > for hibernation needs to match the size of RAM (even though
> > > > > realistically it may be smaller than that in the majority of cases).
> > > >
> > > > This all makes sense, but we do this:
> > > >
> > > > -- add resume=/dev/sdc to the command line
> > > > -- attach a disk (/dev/sdc) with size equal to RAM
> > > > -- mkswap /dev/sdc
> > > > -- swapon /dev/sdc
> > > > -- echo disk > /sys/power/state
> > > >
> > > > and the last operation fails with ENOMEM.  Are we doing something
> > > > wrong?  Are we hitting some other mm bug?
> > >
> > > I would expect this to work, so the fact that it doesn't work for you
> > > indicates a bug somewhere or at least an assumption that doesn't hold.
> > >
> > > Can you please remind me what you do to trigger the unexpected behavior?
> >
> > Yes, I create processes that use a large amount of anon memory, more
> > than 50% of RAM, like this:
> >
> > dd if=/dev/zero bs=1G count=1 | sleep infinity
> >
> > I think dd has a 2 GB limit, or around that number, so you'll need a
> > few of those.
>
> And then you get -ENOMEM from hibernate_preallocate_memory(), or from
> somewhere else?

That is correct.  More precisely, preallocate_image_memory() doesn't
get enough pages, and then preallocate_image_highmem() either gets
nothing, or in any case too little.

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

[Rafael J. Wysocki]

On Mon, Jan 27, 2020 at 3:16 PM Michal Hocko <mhocko@kernel.org> wrote:
>
> On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
> [...]
> > The purpose of my documentation patch was to make it clearer that
> > hibernation may fail in situations in which suspend-to-RAM works; for
> > instance, when there is no swap, and anonymous pages are over 50% of
> > total RAM.  I will send a new version of the patch which hopefully
> > makes this clearer.
>
> I was under impression that s2disk is pretty much impossible without any
> swap.

And you were correct. :-)

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

[Rafael J. Wysocki]

On Mon, Jan 6, 2020 at 1:53 PM Michal Hocko <mhocko@kernel.org> wrote:
>
> On Thu 26-12-19 14:02:04, Luigi Semenzato wrote:
> [...]
> > +Limitations of Hibernation
> > +==========================
> > +
> > +When entering hibernation, the kernel tries to allocate a chunk of memory large
> > +enough to contain a copy of all pages in use, to use it for the system
> > +snapshot.  If the allocation fails, the system cannot hibernate and the
> > +operation fails with ENOMEM.  This will happen, for instance, when the total
> > +amount of anonymous pages (process data) exceeds 1/2 of total RAM.
> > +
> > +One possible workaround (besides terminating enough processes) is to force
> > +excess anonymous pages out to swap before hibernating.  This can be achieved
> > +with memcgroups, by lowering memory usage limits with ``echo <new limit> >
> > +/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``.  However, the latter
> > +operation is not guaranteed to succeed.
>
> I am not familiar with the hibernation process much. But what prevents
> those allocations to reclaim memory and push out the anonymous memory to
> the swap on demand during the hibernation's allocations?

Nothing in particular AFAICS, at least in theory.

The approach taken by the hibernation code is rather straightforward:
allocate enough memory to store a copy of every page (in RAM) that
needs to be saved.  These allocations are made one page at a time, so
in theory they should not fail as long as there is enough swap space
in the system, but I'm probably missing something here.

[PATCH 2/2] pm: add more logging on hibernation failure

[Luigi Semenzato]

Hibernation fails when the kernel cannot allocate enough memory
to copy all pages in use.  This patch ensures that the failure
reason is clearly logged.

Signed-off-by: Luigi Semenzato <semenzato@google.com>
---
 kernel/power/snapshot.c | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c
index 26b9168321e7..df498717a97e 100644
--- a/kernel/power/snapshot.c
+++ b/kernel/power/snapshot.c
@@ -1705,16 +1705,20 @@ int hibernate_preallocate_memory(void)
 	ktime_t start, stop;
 	int error;
 
-	pr_info("Preallocating image memory... ");
+	pr_info("Preallocating hibernation image memory\n");
 	start = ktime_get();
 
 	error = memory_bm_create(&orig_bm, GFP_IMAGE, PG_ANY);
-	if (error)
+	if (error) {
+		pr_err("Cannot allocate original bitmap\n");
 		goto err_out;
+	}
 
 	error = memory_bm_create(&copy_bm, GFP_IMAGE, PG_ANY);
-	if (error)
+	if (error) {
+		pr_err("Cannot allocate copy bitmap\n");
 		goto err_out;
+	}
 
 	alloc_normal = 0;
 	alloc_highmem = 0;
@@ -1804,8 +1808,11 @@ int hibernate_preallocate_memory(void)
 		alloc -= pages;
 		pages += pages_highmem;
 		pages_highmem = preallocate_image_highmem(alloc);
-		if (pages_highmem < alloc)
+		if (pages_highmem < alloc) {
+			pr_err("Image allocation is %lu pages short\n",
+				alloc - pages_highmem);
 			goto err_out;
+		}
 		pages += pages_highmem;
 		/*
 		 * size is the desired number of saveable pages to leave in
@@ -1836,13 +1843,12 @@ int hibernate_preallocate_memory(void)
 
  out:
 	stop = ktime_get();
-	pr_cont("done (allocated %lu pages)\n", pages);
+	pr_info("Allocated %lu pages for hibernation shapshot\n", pages);
 	swsusp_show_speed(start, stop, pages, "Allocated");
 
 	return 0;
 
  err_out:
-	pr_cont("\n");
 	swsusp_free();
 	return -ENOMEM;
 }
-- 
2.24.1.735.g03f4e72817-goog