[3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Mikael Pettersson]

Since updating my ARAnym VMs from 3.12.16 to 3.13.11 I see kswapd0
and ksoftirqd/0 consume inordinate amounts of CPU.  kswapd0 often
rises to about 30-50% CPU even though RAM shouldn't be anywhere near
depleted (768GB, a gcc bootstrap running in a screen session).
kswapd0 tends to stay this way until I drop caches, but doing that
doesn't always fix it.  I also sometimes see ksoftirqd/0 consume
5-30% CPU.

Reverting to 3.12.16 completely eliminates these problems.

I haven't tested 3.14 or 3.15-rc yet.

/Mikael

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Geert Uytterhoeven]

Hi Mikael,

On Fri, Jun 6, 2014 at 1:38 PM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
> Since updating my ARAnym VMs from 3.12.16 to 3.13.11 I see kswapd0
> and ksoftirqd/0 consume inordinate amounts of CPU.  kswapd0 often
> rises to about 30-50% CPU even though RAM shouldn't be anywhere near
> depleted (768GB, a gcc bootstrap running in a screen session).
> kswapd0 tends to stay this way until I drop caches, but doing that
> doesn't always fix it.  I also sometimes see ksoftirqd/0 consume
> 5-30% CPU.
>
> Reverting to 3.12.16 completely eliminates these problems.

Any chance to bisect it?

> I haven't tested 3.14 or 3.15-rc yet.

Would be good to know, though, as it may have been fixed.
However, as v3.13.11 is the most recent stable version of v3.13, this may be
an elsewhere unknown and thus unfixed issue.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Mikael Pettersson]

Geert Uytterhoeven writes:
 > Hi Mikael,
 > 
 > On Fri, Jun 6, 2014 at 1:38 PM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
 > > Since updating my ARAnym VMs from 3.12.16 to 3.13.11 I see kswapd0
 > > and ksoftirqd/0 consume inordinate amounts of CPU.  kswapd0 often
 > > rises to about 30-50% CPU even though RAM shouldn't be anywhere near
 > > depleted (768GB, a gcc bootstrap running in a screen session).
 > > kswapd0 tends to stay this way until I drop caches, but doing that
 > > doesn't always fix it.  I also sometimes see ksoftirqd/0 consume
 > > 5-30% CPU.
 > >
 > > Reverting to 3.12.16 completely eliminates these problems.
 > 
 > Any chance to bisect it?
 > 
 > > I haven't tested 3.14 or 3.15-rc yet.
 > 
 > Would be good to know, though, as it may have been fixed.
 > However, as v3.13.11 is the most recent stable version of v3.13, this may be
 > an elsewhere unknown and thus unfixed issue.

I've just started a gcc-4.8 bootstrap on 3.14.5, and then I'll try 3.15
if it isn't fixed in 3.14.  If 3.15 too is broken, I'll do the bisect,
but it will be a slow process since it takes anywhere from a few hours
to a couple of days for the bug to appear.

BTW, the presence of some unknown bug causing kswapd0 to hog the CPU has
been mentioned on the Debian m68k list earlier this year.

/Mikael

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Mikael Pettersson]

Mikael Pettersson writes:
 > Geert Uytterhoeven writes:
 >  > Hi Mikael,
 >  > 
 >  > On Fri, Jun 6, 2014 at 1:38 PM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
 >  > > Since updating my ARAnym VMs from 3.12.16 to 3.13.11 I see kswapd0
 >  > > and ksoftirqd/0 consume inordinate amounts of CPU.  kswapd0 often
 >  > > rises to about 30-50% CPU even though RAM shouldn't be anywhere near
 >  > > depleted (768GB, a gcc bootstrap running in a screen session).
 >  > > kswapd0 tends to stay this way until I drop caches, but doing that
 >  > > doesn't always fix it.  I also sometimes see ksoftirqd/0 consume
 >  > > 5-30% CPU.
 >  > >
 >  > > Reverting to 3.12.16 completely eliminates these problems.
 >  > 
 >  > Any chance to bisect it?
 >  > 
 >  > > I haven't tested 3.14 or 3.15-rc yet.
 >  > 
 >  > Would be good to know, though, as it may have been fixed.
 >  > However, as v3.13.11 is the most recent stable version of v3.13, this may be
 >  > an elsewhere unknown and thus unfixed issue.
 > 
 > I've just started a gcc-4.8 bootstrap on 3.14.5, and then I'll try 3.15
 > if it isn't fixed in 3.14.

3.14.5 is also affected by the bug.  Dropping caches fixed kswapd0, but
instead ksoftirqd/0 jumped to 20-40%, and it refuses to calm down.

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Mikael Pettersson]

Mikael Pettersson writes:
 > Mikael Pettersson writes:
 >  > Geert Uytterhoeven writes:
 >  >  > Hi Mikael,
 >  >  > 
 >  >  > On Fri, Jun 6, 2014 at 1:38 PM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
 >  >  > > Since updating my ARAnym VMs from 3.12.16 to 3.13.11 I see kswapd0
 >  >  > > and ksoftirqd/0 consume inordinate amounts of CPU.  kswapd0 often
 >  >  > > rises to about 30-50% CPU even though RAM shouldn't be anywhere near
 >  >  > > depleted (768GB, a gcc bootstrap running in a screen session).
 >  >  > > kswapd0 tends to stay this way until I drop caches, but doing that
 >  >  > > doesn't always fix it.  I also sometimes see ksoftirqd/0 consume
 >  >  > > 5-30% CPU.
 >  >  > >
 >  >  > > Reverting to 3.12.16 completely eliminates these problems.
 >  >  > 
 >  >  > Any chance to bisect it?
 >  >  > 
 >  >  > > I haven't tested 3.14 or 3.15-rc yet.
 >  >  > 
 >  >  > Would be good to know, though, as it may have been fixed.
 >  >  > However, as v3.13.11 is the most recent stable version of v3.13, this may be
 >  >  > an elsewhere unknown and thus unfixed issue.
 >  > 
 >  > I've just started a gcc-4.8 bootstrap on 3.14.5, and then I'll try 3.15
 >  > if it isn't fixed in 3.14.
 > 
 > 3.14.5 is also affected by the bug.  Dropping caches fixed kswapd0, but
 > instead ksoftirqd/0 jumped to 20-40%, and it refuses to calm down.

3.15-rc8 is also affected.  It took a while before triggering, but I just
got a kswapd0 CPU hog there too.  Dropping page cache helped this time.

Another observation is that ksoftirqd/0 seem to consistently consume much
more CPU in 3.14/3.15 than in 3.12.  Usually not hugely so at specific time
points, but the accumulated CPU for it on a system that's been up a couple
of days with constant load is looking a bit scary.

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Andreas Schwab]

Mikael Pettersson <mikpelinux@gmail.com> writes:

> Reverting to 3.12.16 completely eliminates these problems.

Even 3.11 has the kswapd0 cpu hog problem.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Mikael Pettersson]

Andreas Schwab writes:
 > Mikael Pettersson <mikpelinux@gmail.com> writes:
 > 
 > > Reverting to 3.12.16 completely eliminates these problems.
 > 
 > Even 3.11 has the kswapd0 cpu hog problem.

Hmm, I just got the kswapd0 CPU hog on 3.12.16 too (while compiling
java code during a gcc package rebuild).

So kernels >= 3.11 have the kswapd0 CPU hog bug, and kernels >= 3.13
also have the ksoftirdq/0 CPU hog bug.

What's the last known-good kernel? 3.10?

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Michael Schmitz]

Hi,

has anyone found a solution to this one?

3.18-rc5 has kswapd0 hogging the CPU - haven't seen ksoftirqd0 yet.
Unpacking a large tarball tends to trigger this for me.

Cheers,

  Michael


On Tue, Jul 1, 2014 at 11:43 PM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
> Andreas Schwab writes:
>  > Mikael Pettersson <mikpelinux@gmail.com> writes:
>  >
>  > > Reverting to 3.12.16 completely eliminates these problems.
>  >
>  > Even 3.11 has the kswapd0 cpu hog problem.
>
> Hmm, I just got the kswapd0 CPU hog on 3.12.16 too (while compiling
> java code during a gcc package rebuild).
>
> So kernels >= 3.11 have the kswapd0 CPU hog bug, and kernels >= 3.13
> also have the ksoftirdq/0 CPU hog bug.
>
> What's the last known-good kernel? 3.10?
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Mikael Pettersson]

Michael Schmitz writes:
 > Hi,
 > 
 > has anyone found a solution to this one?
 > 
 > 3.18-rc5 has kswapd0 hogging the CPU - haven't seen ksoftirqd0 yet.
 > Unpacking a large tarball tends to trigger this for me.

Alas, no.  I went back to the 3.10.xx kernels and they work Ok for me
(they tend to hang during shutdown, but I can live with that).

I should do a git bisect...

/Mikael

 > 
 > Cheers,
 > 
 >   Michael
 > 
 > 
 > On Tue, Jul 1, 2014 at 11:43 PM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
 > > Andreas Schwab writes:
 > >  > Mikael Pettersson <mikpelinux@gmail.com> writes:
 > >  >
 > >  > > Reverting to 3.12.16 completely eliminates these problems.
 > >  >
 > >  > Even 3.11 has the kswapd0 cpu hog problem.
 > >
 > > Hmm, I just got the kswapd0 CPU hog on 3.12.16 too (while compiling
 > > java code during a gcc package rebuild).
 > >
 > > So kernels >= 3.11 have the kswapd0 CPU hog bug, and kernels >= 3.13
 > > also have the ksoftirdq/0 CPU hog bug.
 > >
 > > What's the last known-good kernel? 3.10?
 > > --
 > > To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
 > > the body of a message to majordomo@vger.kernel.org
 > > More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Michael Schmitz]

Hi Mikael,
>  > has anyone found a solution to this one?
>  > 
>  > 3.18-rc5 has kswapd0 hogging the CPU - haven't seen ksoftirqd0 yet.
>  > Unpacking a large tarball tends to trigger this for me.
>
> Alas, no.  I went back to the 3.10.xx kernels and they work Ok for me
> (they tend to hang during shutdown, but I can live with that).
>   

I've followed the vm stats while running gunzip -c on a large file. I 
get an 'invalid compressed data' error at the very end of the gunzip 
run, and the file md5sum does not match what I get when uncompressing 
that file on another system with no error

As long as that file will fit into available memory, all that happens is 
kswapd0 and/or kswapd1 running forever after gunzip has finished. kswapd 
running full tilt appears to coincide with the number of free pages 
hitting the min_free_kbytes limit. The number of dirty pages never grows 
very large (hovers around 1000, less than 1% of RAM size) and remains 
below the nr_dirty_threshold (10800) and nr_dirty_background_threshold 
(5400) limits at all time. (Both limits progressively shrink over time - 
is that normal?).

If I force the VM to only use part of the RAM (by setting 
min_free_kbytes to say 10% of RAM size), the system becomes unresponsive 
as soon as the limit is reached. The swap tasks start to eat substantial 
amounts of CPU once the number of free pages approaches  that limit , 
nr_dirty drops at that time, and nr_dirty_threshold as well as 
nr_dirty_background_threshold begin to rise again  - above the initial 
values, in fact.
> I should do a git bisect...
>   

Would be nice to be able to force this a lot quicker. I'll try with 
smaller files to uncompress, and larger min_free limit.

Cheers,

    Michael

> /Mikael
>
>  > 
>  > Cheers,
>  > 
>  >   Michael
>  > 
>  > 
>  > On Tue, Jul 1, 2014 at 11:43 PM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
>  > > Andreas Schwab writes:
>  > >  > Mikael Pettersson <mikpelinux@gmail.com> writes:
>  > >  >
>  > >  > > Reverting to 3.12.16 completely eliminates these problems.
>  > >  >
>  > >  > Even 3.11 has the kswapd0 cpu hog problem.
>  > >
>  > > Hmm, I just got the kswapd0 CPU hog on 3.12.16 too (while compiling
>  > > java code during a gcc package rebuild).
>  > >
>  > > So kernels >= 3.11 have the kswapd0 CPU hog bug, and kernels >= 3.13
>  > > also have the ksoftirdq/0 CPU hog bug.
>  > >
>  > > What's the last known-good kernel? 3.10?
>  > > --
>  > > To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>  > > the body of a message to majordomo@vger.kernel.org
>  > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>   

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Finn Thain]

On Wed, 1 Apr 2015, Michael Schmitz wrote:

> Hi Mikael,
> > > has anyone found a solution to this one?
> > > 
> > > 3.18-rc5 has kswapd0 hogging the CPU - haven't seen ksoftirqd0 yet. 
> > > Unpacking a large tarball tends to trigger this for me.
> >
> > Alas, no.  I went back to the 3.10.xx kernels and they work Ok for me 
> > (they tend to hang during shutdown, but I can live with that).
> >   
> 
> I've followed the vm stats while running gunzip -c on a large file. I 
> get an 'invalid compressed data' error at the very end of the gunzip 
> run, and the file md5sum does not match what I get when uncompressing 
> that file on another system with no error

Was that an aranym virtual machine or a physical one? If physical, can the 
error be reproduced using a virtual one (given same RAM size, kernel etc)?

-- 

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Michael Schmitz]

Hi Finn,

>> I've followed the vm stats while running gunzip -c on a large file. I 
>> get an 'invalid compressed data' error at the very end of the gunzip 
>> run, and the file md5sum does not match what I get when uncompressing 
>> that file on another system with no error
>>     
>
> Was that an aranym virtual machine or a physical one? If physical, can the 
> error be reproduced using a virtual one (given same RAM size, kernel etc)?
>
>   

Physical - and I haven't tried to reproduce on a VM yet. Cloning the 
system for a VM won't be trivial but I can give it a shot.

    Michael

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Andreas Schwab]

Michael Schmitz <schmitzmic@gmail.com> writes:

> has anyone found a solution to this one?
>
> 3.18-rc5 has kswapd0 hogging the CPU - haven't seen ksoftirqd0 yet.
> Unpacking a large tarball tends to trigger this for me.

The only workaround I have is to drop caches regularily
(/proc/sys/vm/drop_caches) when MemFree is getting low.  All my OBS
build workers are stable running 3.11.6 with this workaround.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Michael Schmitz]

Hi Finn,

> On Wed, 1 Apr 2015, Michael Schmitz wrote:
>
>   
>> I've followed the vm stats while running gunzip -c on a large file. I 
>> get an 'invalid compressed data' error at the very end of the gunzip 
>> run, and the file md5sum does not match what I get when uncompressing 
>> that file on another system with no error
>>     
>
> Was that an aranym virtual machine or a physical one? If physical, can the 
> error be reproduced using a virtual one (given same RAM size, kernel etc)?
>   

The gunzip error cannot be reproduced on any of my ARAnyM VMs. Might be 
a RAM error or other hardware related problem. Same kernel but 
configured slightly different (added 030 and 040 support, plus ARAnyM 
support).

The general behaviour (gunzip eats up all free RAM, then kswapd spins 
doing nothing very apparent, with no dirty pages to be flushed and 
cached pages never released) remains the same. Though I've seen the 
gunzip complete without kicking off kswapd on occasion (had set 
dirty_background_ratio and dirty_ratio half the default for that).

And yes, dropping cached pages as Andreas suggested, does free up 
significant (i.e. most of all) RAM and shuts up kswapd.

    Michael

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Finn Thain]

On Tue, 7 Apr 2015, Michael Schmitz wrote:

> Hi Finn,
> 
> > On Wed, 1 Apr 2015, Michael Schmitz wrote:
> >
> >   
> > > I've followed the vm stats while running gunzip -c on a large file. 
> > > I get an 'invalid compressed data' error at the very end of the 
> > > gunzip run, and the file md5sum does not match what I get when 
> > > uncompressing that file on another system with no error
> > >     
> >
> > Was that an aranym virtual machine or a physical one? If physical, can 
> > the error be reproduced using a virtual one (given same RAM size, 
> > kernel etc)?
> >   
> 
> The gunzip error cannot be reproduced on any of my ARAnyM VMs. Might be 
> a RAM error or other hardware related problem. Same kernel but 
> configured slightly different (added 030 and 040 support, plus ARAnyM 
> support).

The configuration differences could be cancelled by booting your new 
aranym kernel on the physical hardware, and reproducing the fault that 
way. BTW, is this the same physical machine that has DMA issues, which we 
discussed off-list in the past?

> 
> The general behaviour (gunzip eats up all free RAM, then kswapd spins 
> doing nothing very apparent, with no dirty pages to be flushed and 
> cached pages never released) remains the same. Though I've seen the 
> gunzip complete without kicking off kswapd on occasion (had set 
> dirty_background_ratio and dirty_ratio half the default for that).
> 
> And yes, dropping cached pages as Andreas suggested, does free up 
> significant (i.e. most of all) RAM and shuts up kswapd.

This bug has been reported to Red Hat in the past (on x86_64). They closed 
the bugzilla entry in 2011, but the bug was still being reported by Fedora 
users in 2014. https://bugzilla.redhat.com/show_bug.cgi?id=712019

-- 

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Michael Schmitz]

Hi Finn,
> On Tue, 7 Apr 2015, Michael Schmitz wrote:
>
>   
>> The gunzip error cannot be reproduced on any of my ARAnyM VMs. Might be 
>> a RAM error or other hardware related problem. Same kernel but 
>> configured slightly different (added 030 and 040 support, plus ARAnyM 
>> support).
>>     
>
> The configuration differences could be cancelled by booting your new 
> aranym kernel on the physical hardware, and reproducing the fault that 
>   

Sure, and I'll make certain to do that once I've finished the current 
task (updating stuff in a current unstable chroot).

> way. BTW, is this the same physical machine that has DMA issues, which we 
> discussed off-list in the past?
>   

The very same. Runs fairly stable otherwise though - I would have 
expected filesystem corruption or other more drastic errors if the RAM 
was faulty.

>   
>> The general behaviour (gunzip eats up all free RAM, then kswapd spins 
>> doing nothing very apparent, with no dirty pages to be flushed and 
>> cached pages never released) remains the same. Though I've seen the 
>> gunzip complete without kicking off kswapd on occasion (had set 
>> dirty_background_ratio and dirty_ratio half the default for that).
>>
>> And yes, dropping cached pages as Andreas suggested, does free up 
>> significant (i.e. most of all) RAM and shuts up kswapd.
>>     
>
> This bug has been reported to Red Hat in the past (on x86_64). They closed 
> the bugzilla entry in 2011, but the bug was still being reported by Fedora 
> users in 2014. https://bugzilla.redhat.com/show_bug.cgi?id=712019
>   

Thanks for pointing that out!

The patches attached to this report made it into Linus' git tree at that 
time so I presume we are seeing something closely related. The 
discussion on LRML makes my head spin - not a chance to debug this in a 
meaningful way, I suppose.

    Michael

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Mikael Pettersson]

Mikael Pettersson writes:
 > Michael Schmitz writes:
 >  > Hi,
 >  > 
 >  > has anyone found a solution to this one?
 >  > 
 >  > 3.18-rc5 has kswapd0 hogging the CPU - haven't seen ksoftirqd0 yet.
 >  > Unpacking a large tarball tends to trigger this for me.
 > 
 > Alas, no.  I went back to the 3.10.xx kernels and they work Ok for me
 > (they tend to hang during shutdown, but I can live with that).
 > 
 > I should do a git bisect...

I've done two git bisects on this.  The first one was inconclusive
(pointed to a harmless commit), but the second one ended up with:

# first bad commit: [ac4de9543aca59f2b763746647577302fbedd57e] Merge branch 'akpm' (patches from Andrew Morton)

That's a big pile of VM changes, so I think it could be the culprit.

Are people still seeing the kswapd bug with current kernels?  I've
stayed with the 3.10 kernel series on my aranym VMs, except for the
one VM that did the bisection tests.

/Mikael

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Michael Schmitz]

Hi Mikael,

I've since changed too much around my system to be certain, but would
suspect the bug to still be there. I'll see whether changing my config
to use SLUB retriggers the bug when uncompressing a huge file (but I'm
running with only 14 MB noe, which makes traversing the entire VM
metadata a lot faster).

Cheers,

  Michael


On Mon, Feb 22, 2016 at 6:06 AM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
> Mikael Pettersson writes:
>  > Michael Schmitz writes:
>  >  > Hi,
>  >  >
>  >  > has anyone found a solution to this one?
>  >  >
>  >  > 3.18-rc5 has kswapd0 hogging the CPU - haven't seen ksoftirqd0 yet.
>  >  > Unpacking a large tarball tends to trigger this for me.
>  >
>  > Alas, no.  I went back to the 3.10.xx kernels and they work Ok for me
>  > (they tend to hang during shutdown, but I can live with that).
>  >
>  > I should do a git bisect...
>
> I've done two git bisects on this.  The first one was inconclusive
> (pointed to a harmless commit), but the second one ended up with:
>
> # first bad commit: [ac4de9543aca59f2b763746647577302fbedd57e] Merge branch 'akpm' (patches from Andrew Morton)
>
> That's a big pile of VM changes, so I think it could be the culprit.
>
> Are people still seeing the kswapd bug with current kernels?  I've
> stayed with the 3.10 kernel series on my aranym VMs, except for the
> one VM that did the bisection tests.
>
> /Mikael

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Geert Uytterhoeven]

Hi Mikael,

On Sun, Feb 21, 2016 at 6:06 PM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
> Mikael Pettersson writes:
>  > Michael Schmitz writes:
>  >  > has anyone found a solution to this one?
>  >  >
>  >  > 3.18-rc5 has kswapd0 hogging the CPU - haven't seen ksoftirqd0 yet.
>  >  > Unpacking a large tarball tends to trigger this for me.
>  >
>  > Alas, no.  I went back to the 3.10.xx kernels and they work Ok for me
>  > (they tend to hang during shutdown, but I can live with that).
>  >
>  > I should do a git bisect...
>
> I've done two git bisects on this.  The first one was inconclusive
> (pointed to a harmless commit), but the second one ended up with:

Thanks a lot for doing this!

> # first bad commit: [ac4de9543aca59f2b763746647577302fbedd57e] Merge branch 'akpm' (patches from Andrew Morton)
>
> That's a big pile of VM changes, so I think it could be the culprit.

So git bisect pointed to the merge commit itself, not to any of the commits in
the akpm branch?

I redid that merge myself, and the result is the same as ac4de9543aca5.
There could still be a semantical merge conflict that cannot be detected by
git, though.

Could you try cherry-picking the 36 commits from the akpm branch and
bisecting that?
I.e.
    git checkout 26935fb06ee88f11
    git cherry-pick 26935fb06ee88f11..de32a8177f64bc62
    git bisect start
    git bisect bad
    git bisect good 26935fb06ee88f11

Thanks again!

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Mikael Pettersson]

Geert Uytterhoeven writes:
 > Hi Mikael,
 > 
 > On Sun, Feb 21, 2016 at 6:06 PM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
 > > Mikael Pettersson writes:
 > >  > Michael Schmitz writes:
 > >  >  > has anyone found a solution to this one?
 > >  >  >
 > >  >  > 3.18-rc5 has kswapd0 hogging the CPU - haven't seen ksoftirqd0 yet.
 > >  >  > Unpacking a large tarball tends to trigger this for me.
 > >  >
 > >  > Alas, no.  I went back to the 3.10.xx kernels and they work Ok for me
 > >  > (they tend to hang during shutdown, but I can live with that).
 > >  >
 > >  > I should do a git bisect...
 > >
 > > I've done two git bisects on this.  The first one was inconclusive
 > > (pointed to a harmless commit), but the second one ended up with:
 > 
 > Thanks a lot for doing this!
 > 
 > > # first bad commit: [ac4de9543aca59f2b763746647577302fbedd57e] Merge branch 'akpm' (patches from Andrew Morton)
 > >
 > > That's a big pile of VM changes, so I think it could be the culprit.
 > 
 > So git bisect pointed to the merge commit itself, not to any of the commits in
 > the akpm branch?
 > 
 > I redid that merge myself, and the result is the same as ac4de9543aca5.
 > There could still be a semantical merge conflict that cannot be detected by
 > git, though.
 > 
 > Could you try cherry-picking the 36 commits from the akpm branch and
 > bisecting that?
 > I.e.
 >     git checkout 26935fb06ee88f11
 >     git cherry-pick 26935fb06ee88f11..de32a8177f64bc62
 >     git bisect start
 >     git bisect bad
 >     git bisect good 26935fb06ee88f11

I ran these exact commands and restarted my bisection + test loop.

However, git told me it had some 50000+ commits to go through in 16 steps,
so it looks like it selected a much larger range than those 36 commits.

/Mikael

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Geert Uytterhoeven]

Hi Mikael,

On Sun, Mar 6, 2016 at 8:21 AM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
> Geert Uytterhoeven writes:
>  > On Sun, Feb 21, 2016 at 6:06 PM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
>  > > # first bad commit: [ac4de9543aca59f2b763746647577302fbedd57e] Merge branch 'akpm' (patches from Andrew Morton)
>  > >
>  > > That's a big pile of VM changes, so I think it could be the culprit.
>  >
>  > So git bisect pointed to the merge commit itself, not to any of the commits in
>  > the akpm branch?
>  >
>  > I redid that merge myself, and the result is the same as ac4de9543aca5.
>  > There could still be a semantical merge conflict that cannot be detected by
>  > git, though.
>  >
>  > Could you try cherry-picking the 36 commits from the akpm branch and
>  > bisecting that?
>  > I.e.
>  >     git checkout 26935fb06ee88f11
>  >     git cherry-pick 26935fb06ee88f11..de32a8177f64bc62
>  >     git bisect start
>  >     git bisect bad
>  >     git bisect good 26935fb06ee88f11
>
> I ran these exact commands and restarted my bisection + test loop.
>
> However, git told me it had some 50000+ commits to go through in 16 steps,
> so it looks like it selected a much larger range than those 36 commits.

Are you sure you did exactly that?

$ git checkout 26935fb06ee8
[...]
$ git cherry-pick 26935fb06ee88f11..de32a8177f64bc62
[...]
$ git bisect start
$ git bisect bad
$ git bisect good 26935fb06ee88f11
Bisecting: 17 revisions left to test after this (roughly 4 steps)
[8969e7b3b3302ea668d300d0fa593108003b908b] mm: memcg: do not trap
chargers with full callstack on OOM
$

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Mikael Pettersson]

Geert Uytterhoeven writes:
 > Hi Mikael,
 > 
 > On Sun, Mar 6, 2016 at 8:21 AM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
 > > Geert Uytterhoeven writes:
 > >  > On Sun, Feb 21, 2016 at 6:06 PM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
 > >  > > # first bad commit: [ac4de9543aca59f2b763746647577302fbedd57e] Merge branch 'akpm' (patches from Andrew Morton)
 > >  > >
 > >  > > That's a big pile of VM changes, so I think it could be the culprit.
 > >  >
 > >  > So git bisect pointed to the merge commit itself, not to any of the commits in
 > >  > the akpm branch?
 > >  >
 > >  > I redid that merge myself, and the result is the same as ac4de9543aca5.
 > >  > There could still be a semantical merge conflict that cannot be detected by
 > >  > git, though.
 > >  >
 > >  > Could you try cherry-picking the 36 commits from the akpm branch and
 > >  > bisecting that?
 > >  > I.e.
 > >  >     git checkout 26935fb06ee88f11
 > >  >     git cherry-pick 26935fb06ee88f11..de32a8177f64bc62
 > >  >     git bisect start
 > >  >     git bisect bad
 > >  >     git bisect good 26935fb06ee88f11
 > >
 > > I ran these exact commands and restarted my bisection + test loop.
 > >
 > > However, git told me it had some 50000+ commits to go through in 16 steps,
 > > so it looks like it selected a much larger range than those 36 commits.
 > 
 > Are you sure you did exactly that?
 > 
 > $ git checkout 26935fb06ee8
 > [...]
 > $ git cherry-pick 26935fb06ee88f11..de32a8177f64bc62
 > [...]
 > $ git bisect start
 > $ git bisect bad
 > $ git bisect good 26935fb06ee88f11
 > Bisecting: 17 revisions left to test after this (roughly 4 steps)
 > [8969e7b3b3302ea668d300d0fa593108003b908b] mm: memcg: do not trap
 > chargers with full callstack on OOM
 > $

Yes, I copy-pasted those commands exactly.  However, git got confused
because I didn't 'git bisect reset' first.  Now it has the correct
range to bisect :-)

/Mikael

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Mikael Pettersson]

Mikael Pettersson writes:
 > Geert Uytterhoeven writes:
 >  > Hi Mikael,
 >  > 
 >  > On Sun, Mar 6, 2016 at 8:21 AM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
 >  > > Geert Uytterhoeven writes:
 >  > >  > On Sun, Feb 21, 2016 at 6:06 PM, Mikael Pettersson <mikpelinux@gmail.com> wrote:
 >  > >  > > # first bad commit: [ac4de9543aca59f2b763746647577302fbedd57e] Merge branch 'akpm' (patches from Andrew Morton)
 >  > >  > >
 >  > >  > > That's a big pile of VM changes, so I think it could be the culprit.
 >  > >  >
 >  > >  > So git bisect pointed to the merge commit itself, not to any of the commits in
 >  > >  > the akpm branch?
 >  > >  >
 >  > >  > I redid that merge myself, and the result is the same as ac4de9543aca5.
 >  > >  > There could still be a semantical merge conflict that cannot be detected by
 >  > >  > git, though.
 >  > >  >
 >  > >  > Could you try cherry-picking the 36 commits from the akpm branch and
 >  > >  > bisecting that?
 >  > >  > I.e.
 >  > >  >     git checkout 26935fb06ee88f11
 >  > >  >     git cherry-pick 26935fb06ee88f11..de32a8177f64bc62
 >  > >  >     git bisect start
 >  > >  >     git bisect bad
 >  > >  >     git bisect good 26935fb06ee88f11
 >  > >
 >  > > I ran these exact commands and restarted my bisection + test loop.
 >  > >
 >  > > However, git told me it had some 50000+ commits to go through in 16 steps,
 >  > > so it looks like it selected a much larger range than those 36 commits.
 >  > 
 >  > Are you sure you did exactly that?
 >  > 
 >  > $ git checkout 26935fb06ee8
 >  > [...]
 >  > $ git cherry-pick 26935fb06ee88f11..de32a8177f64bc62
 >  > [...]
 >  > $ git bisect start
 >  > $ git bisect bad
 >  > $ git bisect good 26935fb06ee88f11
 >  > Bisecting: 17 revisions left to test after this (roughly 4 steps)
 >  > [8969e7b3b3302ea668d300d0fa593108003b908b] mm: memcg: do not trap
 >  > chargers with full callstack on OOM
 >  > $
 > 
 > Yes, I copy-pasted those commands exactly.  However, git got confused
 > because I didn't 'git bisect reset' first.  Now it has the correct
 > range to bisect :-)

First bisection run completed:

5ee28828d1b5f8d036dc5793be5321dd6f26d344 is the first bad commit
commit 5ee28828d1b5f8d036dc5793be5321dd6f26d344
Author: Michal Hocko <mhocko@suse.cz>
Date:   Thu Sep 12 15:13:26 2013 -0700

    memcg: enhance memcg iterator to support predicates

    The caller of the iterator might know that some nodes or even subtrees
    should be skipped but there is no way to tell iterators about that so the
    only choice left is to let iterators to visit each node and do the
    selection outside of the iterating code.  This, however, doesn't scale
    well with hierarchies with many groups where only few groups are
    interesting.

    This patch adds mem_cgroup_iter_cond variant of the iterator with a
    callback which gets called for every visited node.  There are three
    possible ways how the callback can influence the walk.  Either the node is
    visited, it is skipped but the tree walk continues down the tree or the
    whole subtree of the current group is skipped.

    [hughd@google.com: fix memcg-less page reclaim]
    Signed-off-by: Michal Hocko <mhocko@suse.cz>
    Cc: Balbir Singh <bsingharora@gmail.com>
    Cc: Glauber Costa <glommer@openvz.org>
    Cc: Greg Thelen <gthelen@google.com>
    Cc: Johannes Weiner <hannes@cmpxchg.org>
    Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
    Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
    Cc: Michel Lespinasse <walken@google.com>
    Cc: Tejun Heo <tj@kernel.org>
    Cc: Ying Han <yinghan@google.com>
    Signed-off-by: Hugh Dickins <hughd@google.com>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

:040000 040000 46799adf458e8a2db998ef099dae907af03e1595 29638de8593a98add32f56fb9aea16cd8227f6a4 M      include
:040000 040000 27e4202d2f2a785e35454ef93e75919997219130 8c5973443d35e65eb88a0155b2c78bf90c650a21 M      mm

I'll reset and re-bisect with the known bad points pre-seeded.

/Mikael

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Finn Thain]

On Sun, 21 Feb 2016, Mikael Pettersson wrote:

> I've done two git bisects on this.  The first one was inconclusive 
> (pointed to a harmless commit), but the second one ended up with:
> 
> # first bad commit: [ac4de9543aca59f2b763746647577302fbedd57e] Merge branch 'akpm' (patches from Andrew Morton)
> 
> That's a big pile of VM changes, so I think it could be the culprit.

I think this issue may date back to v2.6.38 or earlier.

The redhat.com bug report was closed in 2012 but Fedora users were still 
seeing the problem after it was supposedly fixed.
  https://bugzilla.redhat.com/show_bug.cgi?id=712019

That page also has a link to the bug report for Ubuntu:
  https://bugs.launchpad.net/ubuntu/+source/linux/+bug/484045

BTW, I came across this recently: "Rik van Riel pointed out that [the 
kswapd thread] tends to be slow for [the purpose of compaction], and it 
can get stuck in a shrinker somewhere waiting for a lock."
  http://lwn.net/Articles/684611/

Perhaps a stack trace would help to ascertain whether this is the same 
known bug or not (?)

-- 

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Mikael Pettersson]

Finn Thain writes:
 > 
 > On Sun, 21 Feb 2016, Mikael Pettersson wrote:
 > 
 > > I've done two git bisects on this.  The first one was inconclusive 
 > > (pointed to a harmless commit), but the second one ended up with:
 > > 
 > > # first bad commit: [ac4de9543aca59f2b763746647577302fbedd57e] Merge branch 'akpm' (patches from Andrew Morton)
 > > 
 > > That's a big pile of VM changes, so I think it could be the culprit.
 > 
 > I think this issue may date back to v2.6.38 or earlier.
 > 
 > The redhat.com bug report was closed in 2012 but Fedora users were still 
 > seeing the problem after it was supposedly fixed.
 >   https://bugzilla.redhat.com/show_bug.cgi?id=712019
 > 
 > That page also has a link to the bug report for Ubuntu:
 >   https://bugs.launchpad.net/ubuntu/+source/linux/+bug/484045
 > 
 > BTW, I came across this recently: "Rik van Riel pointed out that [the 
 > kswapd thread] tends to be slow for [the purpose of compaction], and it 
 > can get stuck in a shrinker somewhere waiting for a lock."
 >   http://lwn.net/Articles/684611/
 > 
 > Perhaps a stack trace would help to ascertain whether this is the same 
 > known bug or not (?)
 > 
 > -- 

FWIW, my latest round(s) of bisects identified the following:

fdbadebec27cc92358ed4f593e8763cf10b82687 is the first bad commit
commit fdbadebec27cc92358ed4f593e8763cf10b82687
Author: Li Zefan <lizefan@huawei.com>
Date:   Thu Sep 12 15:13:19 2013 -0700

    memcg: remove redundant code in mem_cgroup_force_empty_write()

    vfs guarantees the cgroup won't be destroyed, so it's redundant to get a
    css reference.

    Signed-off-by: Li Zefan <lizefan@huawei.com>
    Acked-by: Michal Hocko <mhocko@suse.cz>
    Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
    Cc: Johannes Weiner <hannes@cmpxchg.org>
    Cc: Tejun Heo <tj@kernel.org>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

:040000 040000 1f6b5b056995067c7c60e6f87e9cd1f181e8fbeb ea29d63e70ce2320e144fac7b157a146d41360bf M      mm

This appears to be the first commit in the merge (git bisect refuses to
bisect before it), so either it's it or the problem predates the merge.

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Geert Uytterhoeven]

Hi Mikael,

On Tue, May 31, 2016 at 12:06 PM, Mikael Pettersson
<mikpelinux@gmail.com> wrote:
> Finn Thain writes:
>  > On Sun, 21 Feb 2016, Mikael Pettersson wrote:
>  > > I've done two git bisects on this.  The first one was inconclusive
>  > > (pointed to a harmless commit), but the second one ended up with:
>  > >
>  > > # first bad commit: [ac4de9543aca59f2b763746647577302fbedd57e] Merge branch 'akpm' (patches from Andrew Morton)
>  > >
>  > > That's a big pile of VM changes, so I think it could be the culprit.
>  >
>  > I think this issue may date back to v2.6.38 or earlier.
>  >
>  > The redhat.com bug report was closed in 2012 but Fedora users were still
>  > seeing the problem after it was supposedly fixed.
>  >   https://bugzilla.redhat.com/show_bug.cgi?id=712019
>  >
>  > That page also has a link to the bug report for Ubuntu:
>  >   https://bugs.launchpad.net/ubuntu/+source/linux/+bug/484045
>  >
>  > BTW, I came across this recently: "Rik van Riel pointed out that [the
>  > kswapd thread] tends to be slow for [the purpose of compaction], and it
>  > can get stuck in a shrinker somewhere waiting for a lock."
>  >   http://lwn.net/Articles/684611/
>  >
>  > Perhaps a stack trace would help to ascertain whether this is the same
>  > known bug or not (?)
>  >
>  > --
>
> FWIW, my latest round(s) of bisects identified the following:
>
> fdbadebec27cc92358ed4f593e8763cf10b82687 is the first bad commit
> commit fdbadebec27cc92358ed4f593e8763cf10b82687
> Author: Li Zefan <lizefan@huawei.com>
> Date:   Thu Sep 12 15:13:19 2013 -0700
>
>     memcg: remove redundant code in mem_cgroup_force_empty_write()
>
>     vfs guarantees the cgroup won't be destroyed, so it's redundant to get a
>     css reference.
>
>     Signed-off-by: Li Zefan <lizefan@huawei.com>
>     Acked-by: Michal Hocko <mhocko@suse.cz>
>     Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
>     Cc: Johannes Weiner <hannes@cmpxchg.org>
>     Cc: Tejun Heo <tj@kernel.org>
>     Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
>     Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
>
> :040000 040000 1f6b5b056995067c7c60e6f87e9cd1f181e8fbeb ea29d63e70ce2320e144fac7b157a146d41360bf M      mm
>
> This appears to be the first commit in the merge (git bisect refuses to
> bisect before it), so either it's it or the problem predates the merge.

That's upstream commit c33bd8354f3a3bb26a98d2b6bf600b7b35657328?

Well done! Looks indeed like a suspect for the behavior you're seeing.
I suppose you will follow up with the mm people?

Thx!

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[John Paul Adrian Glaubitz]

On 05/31/2016 12:21 PM, Geert Uytterhoeven wrote:
> That's upstream commit c33bd8354f3a3bb26a98d2b6bf600b7b35657328?
> 
> Well done! Looks indeed like a suspect for the behavior you're seeing.
> I suppose you will follow up with the mm people?

Yay, great news. I'm currently building a native GHC package for Debian/m68k
on Aranym and [kswapd] is taking around half of the CPU load. The suggestion
with drop_caches doesn't help all the time, unfortunately, so I'm really
looking forward getting this fixed :).

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Geert Uytterhoeven]

Hi Adrian,

On Tue, May 31, 2016 at 12:39 PM, John Paul Adrian Glaubitz
<glaubitz@physik.fu-berlin.de> wrote:
> On 05/31/2016 12:21 PM, Geert Uytterhoeven wrote:
>> That's upstream commit c33bd8354f3a3bb26a98d2b6bf600b7b35657328?
>>
>> Well done! Looks indeed like a suspect for the behavior you're seeing.
>> I suppose you will follow up with the mm people?
>
> Yay, great news. I'm currently building a native GHC package for Debian/m68k
> on Aranym and [kswapd] is taking around half of the CPU load. The suggestion
> with drop_caches doesn't help all the time, unfortunately, so I'm really
> looking forward getting this fixed :).

I expect the bad commit can just be reverted, as the surrounding code hasn't
changed?

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [3.13 regression] kswapd0 and ksoftirqd/0 CPU hogs

[Mikael Pettersson]

Geert Uytterhoeven writes:
 > Hi Mikael,
 > 
 > On Tue, May 31, 2016 at 12:06 PM, Mikael Pettersson
 > <mikpelinux@gmail.com> wrote:
 > > Finn Thain writes:
 > >  > On Sun, 21 Feb 2016, Mikael Pettersson wrote:
 > >  > > I've done two git bisects on this.  The first one was inconclusive
 > >  > > (pointed to a harmless commit), but the second one ended up with:
 > >  > >
 > >  > > # first bad commit: [ac4de9543aca59f2b763746647577302fbedd57e] Merge branch 'akpm' (patches from Andrew Morton)
 > >  > >
 > >  > > That's a big pile of VM changes, so I think it could be the culprit.
 > >  >
 > >  > I think this issue may date back to v2.6.38 or earlier.
 > >  >
 > >  > The redhat.com bug report was closed in 2012 but Fedora users were still
 > >  > seeing the problem after it was supposedly fixed.
 > >  >   https://bugzilla.redhat.com/show_bug.cgi?id=712019
 > >  >
 > >  > That page also has a link to the bug report for Ubuntu:
 > >  >   https://bugs.launchpad.net/ubuntu/+source/linux/+bug/484045
 > >  >
 > >  > BTW, I came across this recently: "Rik van Riel pointed out that [the
 > >  > kswapd thread] tends to be slow for [the purpose of compaction], and it
 > >  > can get stuck in a shrinker somewhere waiting for a lock."
 > >  >   http://lwn.net/Articles/684611/
 > >  >
 > >  > Perhaps a stack trace would help to ascertain whether this is the same
 > >  > known bug or not (?)
 > >  >
 > >  > --
 > >
 > > FWIW, my latest round(s) of bisects identified the following:
 > >
 > > fdbadebec27cc92358ed4f593e8763cf10b82687 is the first bad commit
 > > commit fdbadebec27cc92358ed4f593e8763cf10b82687
 > > Author: Li Zefan <lizefan@huawei.com>
 > > Date:   Thu Sep 12 15:13:19 2013 -0700
 > >
 > >     memcg: remove redundant code in mem_cgroup_force_empty_write()
 > >
 > >     vfs guarantees the cgroup won't be destroyed, so it's redundant to get a
 > >     css reference.
 > >
 > >     Signed-off-by: Li Zefan <lizefan@huawei.com>
 > >     Acked-by: Michal Hocko <mhocko@suse.cz>
 > >     Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
 > >     Cc: Johannes Weiner <hannes@cmpxchg.org>
 > >     Cc: Tejun Heo <tj@kernel.org>
 > >     Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
 > >     Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
 > >
 > > :040000 040000 1f6b5b056995067c7c60e6f87e9cd1f181e8fbeb ea29d63e70ce2320e144fac7b157a146d41360bf M      mm
 > >
 > > This appears to be the first commit in the merge (git bisect refuses to
 > > bisect before it), so either it's it or the problem predates the merge.
 > 
 > That's upstream commit c33bd8354f3a3bb26a98d2b6bf600b7b35657328?
 > 
 > Well done! Looks indeed like a suspect for the behavior you're seeing.
 > I suppose you will follow up with the mm people?

Alas, this was a false find.  This commit is definitely in the bad range,
but reverting it from 3.12.60 doesn't eliminate the kswapd bug.

I'll have to bisect mainline again.