3.7-rc6 soft lockup in kswapd0

3.7-rc6 soft lockup in kswapd0

[George Spelvin]

I'm having an interesting issue with a uniprocessor Pentium 4 machine
locking up overnight.  3.6.5 didn't do that, but 3.7-rc6 is not doing
so well.

It's kind of a funny lockup.  Some things work:

- TCP SYN handshake
- Alt-SysRq

And others don't:

- Caps lock
- Shift-PgUp
- Alt-Fn
- Screen unblanking
- Actually talking to a daemon

This is a "headless" machine that boots to a text console and has zero
console activity until the lockup.

This has happened overnight, three nights in a row.  I had to turn screen
blanking off to see anything on the screen.  Running the daily cron jobs
manually just now didn't trigger it, so I haven't found a proximate cause.

The *first* error has scrolled off the screen, but what I can see
an infinite stream (at about 20s intervals) of:

BUG: soft lockup - CPU#0 stuck for 22s! [kswapd0:317]
Pid: 317, comm: kswapd0 Not tainted 3.7.0-RC6 #224 HP Pavilion 04 P6319A-ABA 750N/P4B266LA
EIP: 0060:[<c10571f7>] EFLAGS: 00000202 CPU: 0
EIP is at __zone_watermark_ok+0x5f/7e, 0x67/7e, 0x6e/0x7e, or 0x74/7e
(Didn't type registers & stack)
Call Trace:
 [<c105774f>] ? zone_watermark_ok_safe+0x34/0x3a
 [<c105ec7e>] ? kswapd+0x2fa/0x6f6
 [<c105e984>] ? try_to_free_pages+0x4b8/0x4b8
 [<c103106b>] ? kthread+0x67/0x6c
 [<c12559b7>] ? ret_from_kernel_thread+0x1b/0x28
 [<c1031004>] ? -_kthread_parkme+0x4c.0x4c
Code: (didn't type in first line)
                                5f                        67                     6e                  74                             7e
 c9 39 d6 7f 14 eb 1c 6b c1 2c <8b> 44 05 60 d3 e0 29 c6 <d1> fb 39 de 7e 09 41 <39> f9 7c ea b0 01 <eb> 02 31 c0 5a 5b 5e 5f 5d c3 01 14 85 7c 16


The lack of scrollback limits me to 49 lines of SysRq output, and usually the most interesting
part disappears off the screen.  Two things I can see:

- SysRq-W shows no blocked tasks
- SysRq-M shows zero swap in use, and apparently adequate free memory
	DMA: <various segments> = 9048kB
	Normal: <various> = 116312kB
	HighMem: <various> = 41660kB
	416557 total pagecache pages
	0 pages in swap cache
	Swap cache stats: add 0, delete 0, find 0/0
	Free swap  = 4883724kB
	Total swap = 4883724kB
	524260 pages RAM
	296958 pages HighMem
	5221 pages reserved
	406417 pages shared
	351419 pages non-shared

Does anyone have any debugging suggestions?  Waiting overnight to
make a good/bad decision makes bisecting pretty slow...

Re: 3.7-rc6 soft lockup in kswapd0

[Jan Kara]

On Thu 22-11-12 12:58:24, George Spelvin wrote:
> I'm having an interesting issue with a uniprocessor Pentium 4 machine
> locking up overnight.  3.6.5 didn't do that, but 3.7-rc6 is not doing
> so well.
  I've added some CCs which are hopefully relevant. Specifially I remember
Mel fixing some -mm lockup recently although after googling for a while
that is likely something different.

> It's kind of a funny lockup.  Some things work:
> 
> - TCP SYN handshake
> - Alt-SysRq
> 
> And others don't:
> 
> - Caps lock
> - Shift-PgUp
> - Alt-Fn
> - Screen unblanking
> - Actually talking to a daemon
> 
> This is a "headless" machine that boots to a text console and has zero
> console activity until the lockup.
> 
> This has happened overnight, three nights in a row.  I had to turn screen
> blanking off to see anything on the screen.  Running the daily cron jobs
> manually just now didn't trigger it, so I haven't found a proximate cause.
> 
> The *first* error has scrolled off the screen, but what I can see
> an infinite stream (at about 20s intervals) of:
> 
> BUG: soft lockup - CPU#0 stuck for 22s! [kswapd0:317]
> Pid: 317, comm: kswapd0 Not tainted 3.7.0-RC6 #224 HP Pavilion 04 P6319A-ABA 750N/P4B266LA
> EIP: 0060:[<c10571f7>] EFLAGS: 00000202 CPU: 0
> EIP is at __zone_watermark_ok+0x5f/7e, 0x67/7e, 0x6e/0x7e, or 0x74/7e
> (Didn't type registers & stack)
> Call Trace:
>  [<c105774f>] ? zone_watermark_ok_safe+0x34/0x3a
>  [<c105ec7e>] ? kswapd+0x2fa/0x6f6
>  [<c105e984>] ? try_to_free_pages+0x4b8/0x4b8
>  [<c103106b>] ? kthread+0x67/0x6c
>  [<c12559b7>] ? ret_from_kernel_thread+0x1b/0x28
>  [<c1031004>] ? -_kthread_parkme+0x4c.0x4c
> Code: (didn't type in first line)
>                                 5f                        67                     6e                  74                             7e
>  c9 39 d6 7f 14 eb 1c 6b c1 2c <8b> 44 05 60 d3 e0 29 c6 <d1> fb 39 de 7e 09 41 <39> f9 7c ea b0 01 <eb> 02 31 c0 5a 5b 5e 5f 5d c3 01 14 85 7c 16
  Taking picture of the screen with a digital camera can usually save you
some typing :)

> The lack of scrollback limits me to 49 lines of SysRq output, and usually the most interesting
> part disappears off the screen.  Two things I can see:
> 
> - SysRq-W shows no blocked tasks
> - SysRq-M shows zero swap in use, and apparently adequate free memory
> 	DMA: <various segments> = 9048kB
> 	Normal: <various> = 116312kB
> 	HighMem: <various> = 41660kB
> 	416557 total pagecache pages
> 	0 pages in swap cache
> 	Swap cache stats: add 0, delete 0, find 0/0
> 	Free swap  = 4883724kB
> 	Total swap = 4883724kB
> 	524260 pages RAM
> 	296958 pages HighMem
> 	5221 pages reserved
> 	406417 pages shared
> 	351419 pages non-shared
> 
> Does anyone have any debugging suggestions?  Waiting overnight to
> make a good/bad decision makes bisecting pretty slow...

							Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: 3.7-rc6 soft lockup in kswapd0

[Jan Kara]

On Thu 22-11-12 23:06:00, Jan Kara wrote:
> On Thu 22-11-12 12:58:24, George Spelvin wrote:
> > I'm having an interesting issue with a uniprocessor Pentium 4 machine
> > locking up overnight.  3.6.5 didn't do that, but 3.7-rc6 is not doing
> > so well.
>   I've added some CCs which are hopefully relevant. Specifially I remember
> Mel fixing some -mm lockup recently although after googling for a while
> that is likely something different.
  Actually, https://lkml.org/lkml/2012/11/20/567 looks more relevant to
your problem...

> > It's kind of a funny lockup.  Some things work:
> > 
> > - TCP SYN handshake
> > - Alt-SysRq
> > 
> > And others don't:
> > 
> > - Caps lock
> > - Shift-PgUp
> > - Alt-Fn
> > - Screen unblanking
> > - Actually talking to a daemon
> > 
> > This is a "headless" machine that boots to a text console and has zero
> > console activity until the lockup.
> > 
> > This has happened overnight, three nights in a row.  I had to turn screen
> > blanking off to see anything on the screen.  Running the daily cron jobs
> > manually just now didn't trigger it, so I haven't found a proximate cause.
> > 
> > The *first* error has scrolled off the screen, but what I can see
> > an infinite stream (at about 20s intervals) of:
> > 
> > BUG: soft lockup - CPU#0 stuck for 22s! [kswapd0:317]
> > Pid: 317, comm: kswapd0 Not tainted 3.7.0-RC6 #224 HP Pavilion 04 P6319A-ABA 750N/P4B266LA
> > EIP: 0060:[<c10571f7>] EFLAGS: 00000202 CPU: 0
> > EIP is at __zone_watermark_ok+0x5f/7e, 0x67/7e, 0x6e/0x7e, or 0x74/7e
> > (Didn't type registers & stack)
> > Call Trace:
> >  [<c105774f>] ? zone_watermark_ok_safe+0x34/0x3a
> >  [<c105ec7e>] ? kswapd+0x2fa/0x6f6
> >  [<c105e984>] ? try_to_free_pages+0x4b8/0x4b8
> >  [<c103106b>] ? kthread+0x67/0x6c
> >  [<c12559b7>] ? ret_from_kernel_thread+0x1b/0x28
> >  [<c1031004>] ? -_kthread_parkme+0x4c.0x4c
> > Code: (didn't type in first line)
> >                                 5f                        67                     6e                  74                             7e
> >  c9 39 d6 7f 14 eb 1c 6b c1 2c <8b> 44 05 60 d3 e0 29 c6 <d1> fb 39 de 7e 09 41 <39> f9 7c ea b0 01 <eb> 02 31 c0 5a 5b 5e 5f 5d c3 01 14 85 7c 16
>   Taking picture of the screen with a digital camera can usually save you
> some typing :)
> 
> > The lack of scrollback limits me to 49 lines of SysRq output, and usually the most interesting
> > part disappears off the screen.  Two things I can see:
> > 
> > - SysRq-W shows no blocked tasks
> > - SysRq-M shows zero swap in use, and apparently adequate free memory
> > 	DMA: <various segments> = 9048kB
> > 	Normal: <various> = 116312kB
> > 	HighMem: <various> = 41660kB
> > 	416557 total pagecache pages
> > 	0 pages in swap cache
> > 	Swap cache stats: add 0, delete 0, find 0/0
> > 	Free swap  = 4883724kB
> > 	Total swap = 4883724kB
> > 	524260 pages RAM
> > 	296958 pages HighMem
> > 	5221 pages reserved
> > 	406417 pages shared
> > 	351419 pages non-shared
> > 
> > Does anyone have any debugging suggestions?  Waiting overnight to
> > make a good/bad decision makes bisecting pretty slow...
> 
> 							Honza
> -- 
> Jan Kara <jack@suse.cz>
> SUSE Labs, CR
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: 3.7-rc6 soft lockup in kswapd0

[Mel Gorman]

On Thu, Nov 22, 2012 at 11:06:00PM +0100, Jan Kara wrote:
> On Thu 22-11-12 12:58:24, George Spelvin wrote:
> > I'm having an interesting issue with a uniprocessor Pentium 4 machine

heh, those P4s are great for keeping the room warm in winter. Legacy
high five?

Joking aside, the UP aspect of this is the most relevant.

> > locking up overnight.  3.6.5 didn't do that, but 3.7-rc6 is not doing
> > so well.
>   I've added some CCs which are hopefully relevant. Specifially I remember
> Mel fixing some -mm lockup recently although after googling for a while
> that is likely something different.
> 

Thanks Jan.

> > It's kind of a funny lockup.  Some things work:
> > 
> > - TCP SYN handshake
> > - Alt-SysRq
> > 
> > And others don't:
> > 
> > - Caps lock
> > - Shift-PgUp
> > - Alt-Fn
> > - Screen unblanking
> > - Actually talking to a daemon
> > 

So basically interrupts work but the machine has otherwise locked up. On
a uniprocessor, it's possible it is infinite looping in kswapd and
nothing else is getting the chance to run if it never hits a
cond_resched().

> > This is a "headless" machine that boots to a text console and has zero
> > console activity until the lockup.
> > 
> > This has happened overnight, three nights in a row.  I had to turn screen
> > blanking off to see anything on the screen.  Running the daily cron jobs
> > manually just now didn't trigger it, so I haven't found a proximate cause.
> > 
> > The *first* error has scrolled off the screen, but what I can see
> > an infinite stream (at about 20s intervals) of:
> > 
> > BUG: soft lockup - CPU#0 stuck for 22s! [kswapd0:317]
> > Pid: 317, comm: kswapd0 Not tainted 3.7.0-RC6 #224 HP Pavilion 04 P6319A-ABA 750N/P4B266LA
> > EIP: 0060:[<c10571f7>] EFLAGS: 00000202 CPU: 0
> > EIP is at __zone_watermark_ok+0x5f/7e, 0x67/7e, 0x6e/0x7e, or 0x74/7e
> > (Didn't type registers & stack)
> > Call Trace:
> >  [<c105774f>] ? zone_watermark_ok_safe+0x34/0x3a
> >  [<c105ec7e>] ? kswapd+0x2fa/0x6f6
> >  [<c105e984>] ? try_to_free_pages+0x4b8/0x4b8
> >  [<c103106b>] ? kthread+0x67/0x6c
> >  [<c12559b7>] ? ret_from_kernel_thread+0x1b/0x28
> >  [<c1031004>] ? -_kthread_parkme+0x4c.0x4c
> > Code: (didn't type in first line)
> >                                 5f                        67                     6e                  74                             7e
> >  c9 39 d6 7f 14 eb 1c 6b c1 2c <8b> 44 05 60 d3 e0 29 c6 <d1> fb 39 de 7e 09 41 <39> f9 7c ea b0 01 <eb> 02 31 c0 5a 5b 5e 5f 5d c3 01 14 85 7c 16

Ok, is there any chance you can capture more of sysrq+m, particularly the
bits that say how much free memory there is and many pages of each order
that is free? If you can't, it's ok. I ask because my kernel bug dowsing
rod is twitching in the direction of the recent free page accounting bug
Dave Hansen identified and fixed -- https://lkml.org/lkml/2012/11/21/504

You might have a machine that is able to hit this particular bug faster. It's
not a memory leak as such, but it acts like one. The kernel would think
the watermarks are not met because it's using NR_FREE_PAGES instead of
checking the free lists.

Can you try that patch out please?

> > The lack of scrollback limits me to 49 lines of SysRq output, and usually the most interesting
> > part disappears off the screen.  Two things I can see:
> > 
> > - SysRq-W shows no blocked tasks
> > - SysRq-M shows zero swap in use, and apparently adequate free memory
> > 	DMA: <various segments> = 9048kB

The interesting information in this case is further up. First look for
the line that looks kinda like this

[2322019.463089]  free:83907 slab_reclaimable:89351 slab_unreclaimable:17263

That's the number of free pages. Further down is the free list contents
and looks kinda like this

[2322019.463159] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15904kB
[2322019.463180] Node 0 DMA32: 11398*4kB 7805*8kB 2186*16kB 3*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 143104kB
[2322019.463201] Node 0 Normal: 28595*4kB 7807*8kB 2*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 176868kB

The free page counter and these free lists should be close together. If
there is a big gap then it's almost certainly the bug Dave identified.

There is another potential infinite loop in kswapd that Johannes has
identified and it could also be that. However, lets rule out Dave's bug
first.

-- 
Mel Gorman
SUSE Labs

Re: 3.7-rc6 soft lockup in kswapd0

[George Spelvin]

tl;dr: Have installed Dave Hansen's patch as requested, rebooted.
       Now it's a matter of waiting for lockup...

Mel Gorman wrote:
> heh, those P4s are great for keeping the room warm in winter. Legacy
> high five?

I wanted a physically separate box for some lightly used outside-facing
network services, and it was lying around.  Since then, if it ain't broke,
don't fix it.

If you want *legacy*, a few months ago I installed recent kernels on
an original F00F-bug Pentium (96 MB RAM,bit only 64 MB cacheable!),
and an original MCM PPro.  They aren't actually in service, though.

> Joking aside, the UP aspect of this is the most relevant.

Yeah, I wondered how much testing that got these days. :-)

>> It's kind of a funny lockup.  Some things work:
>> 
>> - TCP SYN handshake
>> - Alt-SysRq
>> 
>> And others don't:
>> 
>> - Caps lock
>> - Shift-PgUp
>> - Alt-Fn
>> - Screen unblanking
>> - Actually talking to a daemon
>> 

> So basically interrupts work but the machine has otherwise locked up. On
> a uniprocessor, it's possible it is infinite looping in kswapd and
> nothing else is getting the chance to run if it never hits a
> cond_resched().

Did caps lock LED handling get moved to something above interrupt context?
I used to use that as the test of "is the machine locked hard".

It might be worth seeing if that functionality can be restored.  The fact
that I can make the console scroll down with Alt-SysRq, but can't scroll
back up to see what just got printed, is maddening.

> Ok, is there any chance you can capture more of sysrq+m, particularly the
> bits that say how much free memory there is and many pages of each order
> that is free? If you can't, it's ok. I ask because my kernel bug dowsing
> rod is twitching in the direction of the recent free page accounting bug
> Dave Hansen identified and fixed -- https://lkml.org/lkml/2012/11/21/504

Will do when I get in front of the machine again.  I had rebooted with
2.6.5, but I can remotely reboot with 2.7-rc6, then it's just a matter
of waiting.

> You might have a machine that is able to hit this particular bug faster. It's
> not a memory leak as such, but it acts like one. The kernel would think
> the watermarks are not met because it's using NR_FREE_PAGES instead of
> checking the free lists.
> 
> Can you try that patch out please?

Okay, so I've cherry-picked ef6c5be658f6a70c1256fbd18e18ee0dc24c3386
from mainline, and rebooted.

I've never tried disabling console blanking remotely, though.  I did
	# echo '^[[9;0]' > /dev/tty0
	# echo '^[[9;0]' > /dev/tty1
	# echo '^[[14;0]' > /dev/tty1
	# echo '^[[14;0]' > /dev/tty0
I hope that works...

> The interesting information in this case is further up. First look for
> the line that looks kinda like this

Will do if it locks up again.  I did notice that all three zones had
at least one free page of size 4096kb, FWIW.

> The free page counter and these free lists should be close together. If
> there is a big gap then it's almost certainly the bug Dave identified.
> 
> There is another potential infinite loop in kswapd that Johannes has
> identified and it could also be that. However, lets rule out Dave's bug
> first.

Thanks a lot!

Re: 3.7-rc6 soft lockup in kswapd0

[George Spelvin]

> tl;dr: Have installed Dave Hansen's patch as requested, rebooted.
>        Now it's a matter of waiting for lockup...

Well, the machine locked up again; it looks like Dave Hansen's patch
isn't the whie story.  Too bad I'm away from the office and can't get
at the console right now.

I'll get the extra information when I have the chance.

Re: 3.7-rc6 soft lockup in kswapd0

[George Spelvin]

Sorry for the delay; was AF(that)K for the weekend.

> Ok, is there any chance you can capture more of sysrq+m, particularly the
> bits that say how much free memory there is and many pages of each order
> that is free? If you can't, it's ok. I ask because my kernel bug dowsing
> rod is twitching in the direction of the recent free page accounting bug
> Dave Hansen identified and fixed -- https://lkml.org/lkml/2012/11/21/504

Okay; as mentioned, I installed that patch and it didn't make any obvious
difference to the symptoms.

The hang IP is still either in __zone_watermark_ok or kswapd (address varies).

> The free page counter and these free lists should be close together. If
> there is a big gap then it's almost certainly the bug Dave identified.

The full sysrq-M output (at least the 49 lines I could read without
working scrollback) is here.  I went over it twice (at least, all the
varying numbers; I may have a typo in some of the fixed text), so I'm
pretty sure it's right.

DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd: 181
HighMem per-cpu:
CPU    0: hi:  186, btch:  31 usd: 172
active_anon:14759 inactive_anon:6504 isolated_anon:0
 active_file:126610 inactive_file:283340 isolated_file:0
 unevictable:0 dirty:289 writeback:0 unstable:0
 free:39060 slab_reclaimable:44235 slab_unreclaimable:1079
 mapped:4468 shmem:580 pagetables:301 bounce:0
 free_cma:0
DMA free:9044kB min:784kB low:980kB high:1176kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:6868kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15804kB mlocked:0kB dirty:4kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 865 2016 2016
Normal free:107016kB min:44012kB low:55012kB high:66016kB active_anon:9096kB inactive_anon:9344kB active_file:296716kB inactive_file:257136kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:885944kB mlocked:0kB dirty:1152kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:176940kB slab_unreclaimable:4316kB kernel_stack:1104kB pagetables:1204kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 9207 9207
HighMem free:40180kB min:512kB low:15148kB high:29784kB active_anon:49940kB inactive_anon:16672kB active_file:209724kB inactive_file:869356kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1178552kB mlocked:0kB dirty:4kB writeback:0kB mapped:17868kB shmem:2320kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_Reserve[]: 0 0 0 0
DMA: 1*4kB 2*8kB 2*16kB 1*32kB 2*64kB 1*128kB 2*256kB 2*512kB 1*1024kB 1*2048kB 1*4096kB = 9044kB
Normal: 572*4kB 129*8kB 33*16kB 42*32kB 33*64kB 15*128kB 8*256kB 3*512kB 6*1024kB 3*2048kB 20*4096kB = 107016kB
HighMem: 407*4kB 199*8kB 124*16kB 107*32kB 53*64kB 10*128kB 5*256kB 4*512kB 3*1024kB 2*2048kB 4*4096kB = 40180kB
410530 total pagecahce pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 4883724kB
Total swap = 4883724kB
524268 pages RAM
296958 pages HighMem
5331 pages reserved
390516 pages shared	FLUCTUATES: 390508, 468, 412, 492, 484, 428, 524, 452, 468, ...
366021 pages non-shared

As mentioned, the "shared" number fluctuates, and the "Normal" free pages
count also fluctuates a little bit, but less.  (The 32kB free count is
sometimes 43 or 41 rather than 42, bringing the total to 107048 or 106984.)

I do see that the free space figures appear to agree.  I couldn't find a
number that appeared to change in sync with the "pages shared" figure.

I could remove Dave Hansen's patch and test again, if that would
help.

Any other ideas?

Thank you very much for the assistance!

Re: 3.7-rc6 soft lockup in kswapd0

[Mel Gorman]

On Sun, Nov 25, 2012 at 10:58:41PM -0500, George Spelvin wrote:
> Sorry for the delay; was AF(that)K for the weekend.
> 
> > Ok, is there any chance you can capture more of sysrq+m, particularly the
> > bits that say how much free memory there is and many pages of each order
> > that is free? If you can't, it's ok. I ask because my kernel bug dowsing
> > rod is twitching in the direction of the recent free page accounting bug
> > Dave Hansen identified and fixed -- https://lkml.org/lkml/2012/11/21/504
> 
> Okay; as mentioned, I installed that patch and it didn't make any obvious
> difference to the symptoms.
> 
> The hang IP is still either in __zone_watermark_ok or kswapd (address varies).
> 

Ok, can you try this patch from Rik on top as well please? This is in
addition to Dave Hansen's accounting fix.

---8<---
From: Rik van Riel <riel@redhat.com>
Subject: mm,vmscan: only loop back if compaction would fail in all zones

Kswapd frees memory to satisfy two goals:
1) allow allocations to succeed, and
2) balance memory pressure between zones 

Currently, kswapd has an issue where it will loop back to free
more memory if any memory zone in the pgdat has not enough free
memory for compaction.  This can lead to unnecessary overhead,
and even infinite loops in kswapd.

It is better to only loop back to free more memory if all of
the zones in the pgdat have insufficient free memory for
compaction.  That satisfies both of kswapd's goals with less
overhead.

Signed-off-by: Rik van Riel <riel@redhat.com>
---
 mm/vmscan.c |   11 ++++++++---
 1 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index b99ecba..f0d111b 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2790,6 +2790,7 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
 	 */
 	if (order) {
 		int zones_need_compaction = 1;
+		int compaction_needs_memory = 1;
 
 		for (i = 0; i <= end_zone; i++) {
 			struct zone *zone = pgdat->node_zones + i;
@@ -2801,10 +2802,10 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
 			    sc.priority != DEF_PRIORITY)
 				continue;
 
-			/* Would compaction fail due to lack of free memory? */
+			/* Is there enough memory for compaction? */
 			if (COMPACTION_BUILD &&
-			    compaction_suitable(zone, order) == COMPACT_SKIPPED)
-				goto loop_again;
+			    compaction_suitable(zone, order) != COMPACT_SKIPPED)
+				compaction_needs_memory = 0;
 
 			/* Confirm the zone is balanced for order-0 */
 			if (!zone_watermark_ok(zone, 0,
@@ -2822,6 +2823,10 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
 			zone_clear_flag(zone, ZONE_CONGESTED);
 		}
 
+		/* None of the zones had enough free memory for compaction. */
+		if (compaction_needs_memory)
+			goto loop_again;
+
 		if (zones_need_compaction)
 			compact_pgdat(pgdat, order);
 	}

Re: 3.7-rc6 soft lockup in kswapd0

[George Spelvin]

Mel Gorman <mgorman@suse.de> wrote:
> Ok, can you try this patch from Rik on top as well please? This is in
> addition to Dave Hansen's accounting fix.
> 
> ---8<---
> From: Rik van Riel <riel@redhat.com>
> Subject: mm,vmscan: only loop back if compaction would fail in all zones

Booted and running.  Judging from the patch, the expected result is
"stops hanging", as opposed to more informative diagnostics, so I'll
keep you posted.

Peraonally, I like to use "bool" for such flags where possible;
it helps document the intent of the variable better.

Re: 3.7-rc6 soft lockup in kswapd0

[Johannes Weiner]

Hi George,

On Mon, Nov 26, 2012 at 08:05:04AM -0500, George Spelvin wrote:
> Mel Gorman <mgorman@suse.de> wrote:
> > Ok, can you try this patch from Rik on top as well please? This is in
> > addition to Dave Hansen's accounting fix.
> > 
> > ---8<---
> > From: Rik van Riel <riel@redhat.com>
> > Subject: mm,vmscan: only loop back if compaction would fail in all zones
> 
> Booted and running.  Judging from the patch, the expected result is
> "stops hanging", as opposed to more informative diagnostics, so I'll
> keep you posted.
> 
> Peraonally, I like to use "bool" for such flags where possible;
> it helps document the intent of the variable better.

Any chance you could test with this fix instead, in addition to Dave's
accounting fix?  It's got bool and everything!

---
From: Johannes Weiner <hannes@cmpxchg.org>
Subject: [patch] mm: vmscan: fix endless loop in kswapd balancing

Kswapd does not in all places have the same criteria for when it
considers a zone balanced.  This leads to zones being not reclaimed
because they are considered just fine and other checks to loop over
the zonelist again because they are considered unbalanced.

Add a function, zone_balanced(), that checks the watermark, and, for
higher order allocations, if compaction has enough free memory.  Then
use it uniformly for when kswapd needs to check if a zone is balanced.

Reported-by: George Spelvin <linux@horizon.com>
Reported-by: Johannes Hirte <johannes.hirte@fem.tu-ilmenau.de>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Tested-by: Johannes Hirte <johannes.hirte@fem.tu-ilmenau.de>
Reviewed-by: Rik van Riel <riel@redhat.com>
Cc: stable@kernel.org [3.4+]
---
 mm/vmscan.c | 27 ++++++++++++++++++---------
 1 file changed, 18 insertions(+), 9 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 48550c6..3b0aef4 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2397,6 +2397,19 @@ static void age_active_anon(struct zone *zone, struct scan_control *sc)
 	} while (memcg);
 }
 
+static bool zone_balanced(struct zone *zone, int order,
+			  unsigned long balance_gap, int classzone_idx)
+{
+	if (!zone_watermark_ok_safe(zone, order, high_wmark_pages(zone) +
+				    balance_gap, classzone_idx, 0))
+		return false;
+
+	if (COMPACTION_BUILD && order && !compaction_suitable(zone, order))
+		return false;
+
+	return true;
+}
+
 /*
  * pgdat_balanced is used when checking if a node is balanced for high-order
  * allocations. Only zones that meet watermarks and are in a zone allowed
@@ -2475,8 +2488,7 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
 			continue;
 		}
 
-		if (!zone_watermark_ok_safe(zone, order, high_wmark_pages(zone),
-							i, 0))
+		if (!zone_balanced(zone, order, 0, i))
 			all_zones_ok = false;
 		else
 			balanced += zone->present_pages;
@@ -2585,8 +2597,7 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
 				break;
 			}
 
-			if (!zone_watermark_ok_safe(zone, order,
-					high_wmark_pages(zone), 0, 0)) {
+			if (!zone_balanced(zone, order, 0, 0)) {
 				end_zone = i;
 				break;
 			} else {
@@ -2662,9 +2673,8 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
 				testorder = 0;
 
 			if ((buffer_heads_over_limit && is_highmem_idx(i)) ||
-				    !zone_watermark_ok_safe(zone, testorder,
-					high_wmark_pages(zone) + balance_gap,
-					end_zone, 0)) {
+			    !zone_balanced(zone, testorder,
+					   balance_gap, end_zone)) {
 				shrink_zone(zone, &sc);
 
 				reclaim_state->reclaimed_slab = 0;
@@ -2691,8 +2701,7 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
 				continue;
 			}
 
-			if (!zone_watermark_ok_safe(zone, testorder,
-					high_wmark_pages(zone), end_zone, 0)) {
+			if (!zone_balanced(zone, testorder, 0, end_zone)) {
 				all_zones_ok = 0;
 				/*
 				 * We are still under min water mark.  This
-- 
1.7.11.7

Re: 3.7-rc6 soft lockup in kswapd0

[George Spelvin]

Johannes Weiner <hannes@cmpxchg.org> wrote:
> Any chance you could test with this fix instead, in addition to Dave's
> accounting fix?  It's got bool and everything!

Okay.  Mel, speak up if you object.  I also rebased on top of 3.7-rc7,
which already includes Dave's fix.  Again, speak up if that's a bad idea.

> ---
> From: Johannes Weiner <hannes@cmpxchg.org>
> Subject: [patch] mm: vmscan: fix endless loop in kswapd balancing

Re: 3.7-rc6 soft lockup in kswapd0

[Mel Gorman]

On Mon, Nov 26, 2012 at 01:53:17PM -0500, George Spelvin wrote:
> Johannes Weiner <hannes@cmpxchg.org> wrote:
> > Any chance you could test with this fix instead, in addition to Dave's
> > accounting fix?  It's got bool and everything!
> 
> Okay.  Mel, speak up if you object.  I also rebased on top of 3.7-rc7,
> which already includes Dave's fix.  Again, speak up if that's a bad idea.
> 

No objections all round.

-- 
Mel Gorman
SUSE Labs

Re: 3.7-rc6 soft lockup in kswapd0

[George Spelvin]

Mel Gorman <mgorman@suse.de> wrote:
> On Mon, Nov 26, 2012 at 01:53:17PM -0500, George Spelvin wrote:
>> Johannes Weiner <hannes@cmpxchg.org> wrote:
>>> Any chance you could test with this fix instead, in addition to Dave's
>>> accounting fix?  It's got bool and everything!
> 
>> Okay.  Mel, speak up if you object.  I also rebased on top of 3.7-rc7,
>> which already includes Dave's fix.  Again, speak up if that's a bad idea.
> 
> No objections all round.

Well, it just made it to 24 hours, 
it did before.  I'm going to wait a couple more days before declaring
victory, but it looks good so far.

 19:19:10 up 1 day, 0 min,  2 users,  load average: 0.15, 0.20, 0.22
 21:24:05 up 1 day,  2:05,  2 users,  load average: 0.25, 0.19, 0.18

Re: 3.7-rc6 soft lockup in kswapd0

[Mel Gorman]

On Tue, Nov 27, 2012 at 04:25:14PM -0500, George Spelvin wrote:
> Mel Gorman <mgorman@suse.de> wrote:
> > On Mon, Nov 26, 2012 at 01:53:17PM -0500, George Spelvin wrote:
> >> Johannes Weiner <hannes@cmpxchg.org> wrote:
> >>> Any chance you could test with this fix instead, in addition to Dave's
> >>> accounting fix?  It's got bool and everything!
> > 
> >> Okay.  Mel, speak up if you object.  I also rebased on top of 3.7-rc7,
> >> which already includes Dave's fix.  Again, speak up if that's a bad idea.
> > 
> > No objections all round.
> 
> Well, it just made it to 24 hours, 
> it did before.  I'm going to wait a couple more days before declaring
> victory, but it looks good so far.
> 
>  19:19:10 up 1 day, 0 min,  2 users,  load average: 0.15, 0.20, 0.22
>  21:24:05 up 1 day,  2:05,  2 users,  load average: 0.25, 0.19, 0.18

Superb. The relevant patches *should* be in flight for 3.7 assuming they
make it through the confusion of last-minute fixes.

Thanks

-- 
Mel Gorman
SUSE Labs

Re: 3.7-rc6 soft lockup in kswapd0

[George Spelvin]

Mel Gorman <mgorman@suse.de> wrote:
> On Tue, Nov 27, 2012 at 04:25:14PM -0500, George Spelvin wrote:
>> Well, it just made it to 24 hours, 
>> it did before.  I'm going to wait a couple more days before declaring
>> victory, but it looks good so far.
>> 
>>  19:19:10 up 1 day, 0 min,  2 users,  load average: 0.15, 0.20, 0.22
>>  21:24:05 up 1 day,  2:05,  2 users,  load average: 0.25, 0.19, 0.18
>
> Superb. The relevant patches *should* be in flight for 3.7 assuming they
> make it through the confusion of last-minute fixes.

 14:53:54 up 2 days, 19:35,  2 users,  load average: 0.20, 0.24, 0.23

Almost three days, when it wouldn't live overnight before.
As promised, I'm declaring victory.

The patch that worked (on top of -rc7) was Johannes Weiner's
"mm: vmscan: fix endless loop in kswapd balancing"
that added the zone_balanced() function to mm/vmscan.c:2400.

Thank you all very much!

Re: 3.7-rc6 soft lockup in kswapd0

[Mel Gorman]

On Thu, Nov 29, 2012 at 09:54:14AM -0500, George Spelvin wrote:
> Mel Gorman <mgorman@suse.de> wrote:
> > On Tue, Nov 27, 2012 at 04:25:14PM -0500, George Spelvin wrote:
> >> Well, it just made it to 24 hours, 
> >> it did before.  I'm going to wait a couple more days before declaring
> >> victory, but it looks good so far.
> >> 
> >>  19:19:10 up 1 day, 0 min,  2 users,  load average: 0.15, 0.20, 0.22
> >>  21:24:05 up 1 day,  2:05,  2 users,  load average: 0.25, 0.19, 0.18
> >
> > Superb. The relevant patches *should* be in flight for 3.7 assuming they
> > make it through the confusion of last-minute fixes.
> 
>  14:53:54 up 2 days, 19:35,  2 users,  load average: 0.20, 0.24, 0.23
> 
> Almost three days, when it wouldn't live overnight before.
> As promised, I'm declaring victory.
> 
> The patch that worked (on top of -rc7) was Johannes Weiner's
> "mm: vmscan: fix endless loop in kswapd balancing"
> that added the zone_balanced() function to mm/vmscan.c:2400.
> 
> Thank you all very much!

Excellent, thanks for getting back quickly. The necessary patches in
question are sortof-in-flight but I expect they'll make it in for 3.7.
If that happens, it would be very nice if you could test with 3.7 to confirm
and if all goes according to plan, I'll do a backport for 3.6-stable and
hopefully squash most of the THP-causes-all-and-sundry-to-go-nuts bugs.


-- 
Mel Gorman
SUSE Labs

Re: 3.7-rc6 soft lockup in kswapd0

[Johannes Weiner]

On Thu, Nov 29, 2012 at 09:54:14AM -0500, George Spelvin wrote:
> Mel Gorman <mgorman@suse.de> wrote:
> > On Tue, Nov 27, 2012 at 04:25:14PM -0500, George Spelvin wrote:
> >> Well, it just made it to 24 hours, 
> >> it did before.  I'm going to wait a couple more days before declaring
> >> victory, but it looks good so far.
> >> 
> >>  19:19:10 up 1 day, 0 min,  2 users,  load average: 0.15, 0.20, 0.22
> >>  21:24:05 up 1 day,  2:05,  2 users,  load average: 0.25, 0.19, 0.18
> >
> > Superb. The relevant patches *should* be in flight for 3.7 assuming they
> > make it through the confusion of last-minute fixes.
> 
>  14:53:54 up 2 days, 19:35,  2 users,  load average: 0.20, 0.24, 0.23
> 
> Almost three days, when it wouldn't live overnight before.
> As promised, I'm declaring victory.
> 
> The patch that worked (on top of -rc7) was Johannes Weiner's
> "mm: vmscan: fix endless loop in kswapd balancing"
> that added the zone_balanced() function to mm/vmscan.c:2400.

Thanks for testing!

Love,
Georgina

Re: 3.7-rc6 soft lockup in kswapd0

[George Spelvin]

> Almost three days, when it wouldn't live overnight before.
> As promised, I'm declaring victory.
> 
> The patch that worked (on top of -rc7) was Johannes Weiner's
> "mm: vmscan: fix endless loop in kswapd balancing"
> that added the zone_balanced() function to mm/vmscan.c:2400.
> 
> Thank you all very much!

Further update, uptime is now 1 week with no more problems.

Just checked dmesg; no complaints.  And kswapd0 is showing 0:09.08 of
CPU time for the week, more than cron but less than init.

Tested-by: George Spelvin <linux@horizon.com>