2.6.37-rc1+: hibernate regression, claims not enough swap space

2.6.37-rc1+: hibernate regression, claims not enough swap space

[Pavel Machek]

Hi!

...but there's enough -- or at least it was enough to fit previous
versions. 32-bit machine, so it has highmem.

System is in console mode, very lightly loaded.

Mem:   2054716k total,   736548k used,  1318168k free,    15368k buffers
Swap:   779148k total,     2360k used,   776788k free,   546388k cached

thinkpad_acpi: ACPI backlight control delay disabled
PM: thaw of devices complete after 539.577 msecs
PM: writing image.
PM: Free swap pages: 194166
PM: Not enough free swap
Restarting tasks ...

Aha, and it is the new default /sys/power/image_size .. setting it to
0 lets machine hibernate. I guess the new default is very wrong for
highmem machine...

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: 2.6.37-rc1+: hibernate regression, claims not enough swap space

[Rafael J. Wysocki]

On Sunday, November 14, 2010, Pavel Machek wrote:
> Hi!

Hi,

> ...but there's enough -- or at least it was enough to fit previous
> versions. 32-bit machine, so it has highmem.
> 
> System is in console mode, very lightly loaded.
> 
> Mem:   2054716k total,   736548k used,  1318168k free,    15368k buffers
> Swap:   779148k total,     2360k used,   776788k free,   546388k cached

Well, the swap is rather in short supply.  Below the 50% of RAM recommendation.

> thinkpad_acpi: ACPI backlight control delay disabled
> PM: thaw of devices complete after 539.577 msecs
> PM: writing image.
> PM: Free swap pages: 194166
> PM: Not enough free swap
> Restarting tasks ...
> 
> Aha, and it is the new default /sys/power/image_size .. setting it to
> 0 lets machine hibernate. I guess the new default is very wrong for
> highmem machine...

The old default did the wrong thing for everyone with sufficient swap (it made
the OOM code trigger every time while preparing to create an image), so I think
the new one it's better overall.

Thanks,
Rafael

Re: 2.6.37-rc1+: hibernate regression, claims not enough swap space

[Pavel Machek]

Hi!

> > ...but there's enough -- or at least it was enough to fit previous
> > versions. 32-bit machine, so it has highmem.
> > 
> > System is in console mode, very lightly loaded.
> > 
> > Mem:   2054716k total,   736548k used,  1318168k free,    15368k buffers
> > Swap:   779148k total,     2360k used,   776788k free,   546388k cached
> 
> Well, the swap is rather in short supply.  Below the 50% of RAM recommendation.

Well, but the biggest image we can write is not 50% of RAM, but 50% of
lowmem... and swap is nearly as big as lowmem in this case -- so this
machine is not misconfigured.

root@amd:/# cat /proc/meminfo 
MemTotal:        2054716 kB
MemFree:          192048 kB
Buffers:           97628 kB
Cached:          1427088 kB
SwapCached:            0 kB
Active:           547580 kB
Inactive:        1091744 kB
Active(anon):      57060 kB
Inactive(anon):    58908 kB
Active(file):     490520 kB
Inactive(file):  1032836 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:       1178440 kB
HighFree:         128092 kB
LowTotal:         876276 kB
LowFree:           63956 kB
SwapTotal:        779148 kB
SwapFree:         779148 kB

> > thinkpad_acpi: ACPI backlight control delay disabled
> > PM: thaw of devices complete after 539.577 msecs
> > PM: writing image.
> > PM: Free swap pages: 194166
> > PM: Not enough free swap
> > Restarting tasks ...
> > 
> > Aha, and it is the new default /sys/power/image_size .. setting it to
> > 0 lets machine hibernate. I guess the new default is very wrong for
> > highmem machine...
> 
> The old default did the wrong thing for everyone with sufficient swap (it made
> the OOM code trigger every time while preparing to create an image), so I think
> the new one it's better overall.

OOM? No... image_size of 0 should have written "as small image as
possible"; slow, but should not OOM.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: 2.6.37-rc1+: hibernate regression, claims not enough swap space

[Pavel Machek]

On Tue 2010-11-16 06:26:34, Pavel Machek wrote:
> Hi!
> 
> > > ...but there's enough -- or at least it was enough to fit previous
> > > versions. 32-bit machine, so it has highmem.
> > > 
> > > System is in console mode, very lightly loaded.
> > > 
> > > Mem:   2054716k total,   736548k used,  1318168k free,    15368k buffers
> > > Swap:   779148k total,     2360k used,   776788k free,   546388k cached
> > 
> > Well, the swap is rather in short supply.  Below the 50% of RAM recommendation.
> 
> Well, but the biggest image we can write is not 50% of RAM, but 50% of
> lowmem... and swap is nearly as big as lowmem in this case -- so this
> machine is not misconfigured.

BTW swsusp.txt may need updating, it says:

echo N > /sys/power/image_size

before suspend (it is limited to 500 MB by default).

...
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: 2.6.37-rc1+: hibernate regression, claims not enough swap space

[Rafael J. Wysocki]

On Tuesday, November 16, 2010, Pavel Machek wrote:
> Hi!
> 
> > > ...but there's enough -- or at least it was enough to fit previous
> > > versions. 32-bit machine, so it has highmem.
> > > 
> > > System is in console mode, very lightly loaded.
> > > 
> > > Mem:   2054716k total,   736548k used,  1318168k free,    15368k buffers
> > > Swap:   779148k total,     2360k used,   776788k free,   546388k cached
> > 
> > Well, the swap is rather in short supply.  Below the 50% of RAM recommendation.
> 
> Well, but the biggest image we can write is not 50% of RAM, but 50% of
> lowmem... 

No, it is not.  It's been 50% of RAM for a couple of years now. :-)

> and swap is nearly as big as lowmem in this case -- so this
> machine is not misconfigured.
> 
> root@amd:/# cat /proc/meminfo 
> MemTotal:        2054716 kB
> MemFree:          192048 kB
> Buffers:           97628 kB
> Cached:          1427088 kB
> SwapCached:            0 kB
> Active:           547580 kB
> Inactive:        1091744 kB
> Active(anon):      57060 kB
> Inactive(anon):    58908 kB
> Active(file):     490520 kB
> Inactive(file):  1032836 kB
> Unevictable:           0 kB
> Mlocked:               0 kB
> HighTotal:       1178440 kB
> HighFree:         128092 kB
> LowTotal:         876276 kB
> LowFree:           63956 kB
> SwapTotal:        779148 kB
> SwapFree:         779148 kB
> 
> > > thinkpad_acpi: ACPI backlight control delay disabled
> > > PM: thaw of devices complete after 539.577 msecs
> > > PM: writing image.
> > > PM: Free swap pages: 194166
> > > PM: Not enough free swap
> > > Restarting tasks ...
> > > 
> > > Aha, and it is the new default /sys/power/image_size .. setting it to
> > > 0 lets machine hibernate. I guess the new default is very wrong for
> > > highmem machine...
> > 
> > The old default did the wrong thing for everyone with sufficient swap (it made
> > the OOM code trigger every time while preparing to create an image), so I think
> > the new one it's better overall.
> 
> OOM? No... image_size of 0 should have written "as small image as
> possible"; slow, but should not OOM.

This is not how it works now.  We preallocate memory to create memory pressure,
so if image_size is 0, we need to preallocate until we run out of pages that
can be freed, which means OOM.

Thanks,
Rafael

Re: 2.6.37-rc1+: hibernate regression, claims not enough swap space

[Rafael J. Wysocki]

On Tuesday, November 16, 2010, Pavel Machek wrote:
> On Tue 2010-11-16 06:26:34, Pavel Machek wrote:
> > Hi!
> > 
> > > > ...but there's enough -- or at least it was enough to fit previous
> > > > versions. 32-bit machine, so it has highmem.
> > > > 
> > > > System is in console mode, very lightly loaded.
> > > > 
> > > > Mem:   2054716k total,   736548k used,  1318168k free,    15368k buffers
> > > > Swap:   779148k total,     2360k used,   776788k free,   546388k cached
> > > 
> > > Well, the swap is rather in short supply.  Below the 50% of RAM recommendation.
> > 
> > Well, but the biggest image we can write is not 50% of RAM, but 50% of
> > lowmem... and swap is nearly as big as lowmem in this case -- so this
> > machine is not misconfigured.
> 
> BTW swsusp.txt may need updating, it says:
> 
> echo N > /sys/power/image_size
> 
> before suspend (it is limited to 500 MB by default).

Yes, this needs to be updated.

Thanks,
Rafael

Re: 2.6.37-rc1+: hibernate regression, claims not enough swap space

[Pavel Machek]

Hi!

> > > > ...but there's enough -- or at least it was enough to fit previous
> > > > versions. 32-bit machine, so it has highmem.
> > > > 
> > > > System is in console mode, very lightly loaded.
> > > > 
> > > > Mem:   2054716k total,   736548k used,  1318168k free,    15368k buffers
> > > > Swap:   779148k total,     2360k used,   776788k free,   546388k cached
> > > 
> > > Well, the swap is rather in short supply.  Below the 50% of RAM recommendation.
> > 
> > Well, but the biggest image we can write is not 50% of RAM, but 50% of
> > lowmem... 
> 
> No, it is not.  It's been 50% of RAM for a couple of years now. :-)




> > > > PM: thaw of devices complete after 539.577 msecs
> > > > PM: writing image.
> > > > PM: Free swap pages: 194166
> > > > PM: Not enough free swap
> > > > Restarting tasks ...
> > > > 
> > > > Aha, and it is the new default /sys/power/image_size .. setting it to
> > > > 0 lets machine hibernate. I guess the new default is very wrong for
> > > > highmem machine...
> > > 
> > > The old default did the wrong thing for everyone with sufficient swap (it made
> > > the OOM code trigger every time while preparing to create an image), so I think
> > > the new one it's better overall.
> > 
> > OOM? No... image_size of 0 should have written "as small image as
> > possible"; slow, but should not OOM.
> 
> This is not how it works now.  We preallocate memory to create memory pressure,
> so if image_size is 0, we need to preallocate until we run out of pages that
> can be freed, which means OOM.

That's bad, right? Instead of killing 

Anyway, it does not work at all.

root@amd:~# echo 300000000 > /sys/power/image_size
root@amd:~# echo disk > /sys/power/state 
-su: echo: write error: No space left on device

(And dmesg full of failed allocations).

I can write 400M there, and it fails, too.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: 2.6.37-rc1+: hibernate regression, claims not enough swap space

[Rafael J. Wysocki]

On Wednesday, November 17, 2010, Pavel Machek wrote:
> Hi!
> 
> > > > > ...but there's enough -- or at least it was enough to fit previous
> > > > > versions. 32-bit machine, so it has highmem.
> > > > > 
> > > > > System is in console mode, very lightly loaded.
> > > > > 
> > > > > Mem:   2054716k total,   736548k used,  1318168k free,    15368k buffers
> > > > > Swap:   779148k total,     2360k used,   776788k free,   546388k cached
> > > > 
> > > > Well, the swap is rather in short supply.  Below the 50% of RAM recommendation.
> > > 
> > > Well, but the biggest image we can write is not 50% of RAM, but 50% of
> > > lowmem... 
> > 
> > No, it is not.  It's been 50% of RAM for a couple of years now. :-)
> 
> 
> 
> 
> > > > > PM: thaw of devices complete after 539.577 msecs
> > > > > PM: writing image.
> > > > > PM: Free swap pages: 194166
> > > > > PM: Not enough free swap
> > > > > Restarting tasks ...
> > > > > 
> > > > > Aha, and it is the new default /sys/power/image_size .. setting it to
> > > > > 0 lets machine hibernate. I guess the new default is very wrong for
> > > > > highmem machine...
> > > > 
> > > > The old default did the wrong thing for everyone with sufficient swap (it made
> > > > the OOM code trigger every time while preparing to create an image), so I think
> > > > the new one it's better overall.
> > > 
> > > OOM? No... image_size of 0 should have written "as small image as
> > > possible"; slow, but should not OOM.
> > 
> > This is not how it works now.  We preallocate memory to create memory pressure,
> > so if image_size is 0, we need to preallocate until we run out of pages that
> > can be freed, which means OOM.
> 
> That's bad, right? Instead of killing 

I'm not sure what you mean.

> Anyway, it does not work at all.
> 
> root@amd:~# echo 300000000 > /sys/power/image_size
> root@amd:~# echo disk > /sys/power/state 
> -su: echo: write error: No space left on device
> 
> (And dmesg full of failed allocations).
> 
> I can write 400M there, and it fails, too.

Well, in that case your swap is smaller than the requested image size, isn't it?

There's a check in there that should catch that and it apparently doesn't work.

Do you have that dmesg by chance?

Rafael

Re: 2.6.37-rc1+: hibernate regression, claims not enough swap space

[Pavel Machek]

Hi!

> > > This is not how it works now.  We preallocate memory to create memory pressure,
> > > so if image_size is 0, we need to preallocate until we run out of pages that
> > > can be freed, which means OOM.
> > 
> > That's bad, right? Instead of killing 
> 
> I'm not sure what you mean.

I meant "instead of failing hibernation, you get OOM -- killing random
processes".

> > Anyway, it does not work at all.
> > 
> > root@amd:~# echo 300000000 > /sys/power/image_size
> > root@amd:~# echo disk > /sys/power/state 
> > -su: echo: write error: No space left on device
> > 
> > (And dmesg full of failed allocations).
> > 
> > I can write 400M there, and it fails, too.
> 
> Well, in that case your swap is smaller than the requested image size, isn't it?

No, swap is 

pavel@amd:~/misc$ cat /proc/swaps 
Filename				Type		Size	Used
Priority
/dev/sda1                               partition	779148	0
-1

... 700M.

> There's a check in there that should catch that and it apparently doesn't work.
> 
> Do you have that dmesg by chance?

dmesg is useless -- it is full of backtraces :-(. ACPI allocations
fail.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: 2.6.37-rc1+: hibernate regression, claims not enough swap space

[Rafael J. Wysocki]

On Thursday, November 18, 2010, Pavel Machek wrote:
> Hi!
> 
> > > > This is not how it works now.  We preallocate memory to create memory pressure,
> > > > so if image_size is 0, we need to preallocate until we run out of pages that
> > > > can be freed, which means OOM.
> > > 
> > > That's bad, right? Instead of killing 
> > 
> > I'm not sure what you mean.
> 
> I meant "instead of failing hibernation, you get OOM -- killing random
> processes".

That's not the case.  The OOM _killer_ is disabled during hibernation, but
there's some more OOM code that's trying very hard to free whatever pages
it can get without killing any processes.

> > > Anyway, it does not work at all.
> > > 
> > > root@amd:~# echo 300000000 > /sys/power/image_size
> > > root@amd:~# echo disk > /sys/power/state 
> > > -su: echo: write error: No space left on device
> > > 
> > > (And dmesg full of failed allocations).
> > > 
> > > I can write 400M there, and it fails, too.
> > 
> > Well, in that case your swap is smaller than the requested image size, isn't it?
> 
> No, swap is 
> 
> pavel@amd:~/misc$ cat /proc/swaps 
> Filename				Type		Size	Used
> Priority
> /dev/sda1                               partition	779148	0
> -1
> 
> ... 700M.
> 
> > There's a check in there that should catch that and it apparently doesn't work.
> > 
> > Do you have that dmesg by chance?
> 
> dmesg is useless -- it is full of backtraces :-(. ACPI allocations
> fail.

What about:

# echo core > /sys/power/pm_test
# echo disk > /sys/power/state

Then the messages should be there in the logs.

Thanks,
Rafael