Excessive swapping under Linux 4.14.91 (no issues in 4.14.90).

Excessive swapping under Linux 4.14.91 (no issues in 4.14.90).

[Hussam Al-Tayeb]

Hello. My system has 16GB of ram. Before kernel 4.14.91, it would use about 700KBytes to 1MB of the swap if the memory usage was 4 to 5GB in use out of 16GB (15.6 actually).

After upgrading to 4.14.91, I am seeing 500 to 700MB of swap usage even under low memory pressure situations. For example, 3.5GB used out of 16GB.

I downgraded to 4.14.90 and the excessive swapping stopped. I upgraded to 4.14.91 and the heavy swapping came back.

Any idea how I can gather information to tell what is happening?

The machine always has the same services and applications running so I am sure this is a kernel issue. It hasn't received any non-kernel updates in months. This narrows the issue down to the kernel.

Thank you.

Re: Excessive swapping under Linux 4.14.91 (no issues in 4.14.90).

[Vito Caputo]

On Thu, Jan 03, 2019 at 09:33:09PM +0100, Hussam Al-Tayeb wrote:
> > Sent: Thursday, January 03, 2019 at 10:12 PM
> > From: "Vito Caputo" <vcaputo@pengaru.com>
> > To: "Hussam Al-Tayeb" <hussam.altayeb@gmx.com>
> > Subject: Re: Excessive swapping under Linux 4.14.91 (no issues in 4.14.90).
> >
> > 
> > The diff between 4.14.90 and 4.14.91 is rather small, appended below is
> > the entire shortlog.
> > 
> > There's only mm one commit:
> > 
> > > commit 36f93a2e7dce0a4f58b96a7ecb3af4e5897a60d4
> > > Author: Roman Gushchin <guro@fb.com>
> > > Date:   Fri Oct 26 15:03:27 2018 -0700
> > > 
> > >     mm: don't miss the last page because of round-off error
> > >     
> > >     commit 68600f623d69da428c6163275f97ca126e1a8ec5 upstream.
> > >     
> > >     I've noticed, that dying memory cgroups are often pinned in memory by a
> > >     single pagecache page.  Even under moderate memory pressure they sometimes
> > >     stayed in such state for a long time.  That looked strange.
> > >     
> > >     My investigation showed that the problem is caused by applying the LRU
> > >     pressure balancing math:
> > >     
> > >       scan = div64_u64(scan * fraction[lru], denominator),
> > >     
> > >     where
> > >     
> > >       denominator = fraction[anon] + fraction[file] + 1.
> > >     
> > >     Because fraction[lru] is always less than denominator, if the initial scan
> > >     size is 1, the result is always 0.
> > >     
> > >     This means the last page is not scanned and has
> > >     no chances to be reclaimed.
> > >     
> > >     Fix this by rounding up the result of the division.
> > >     
> > >     In practice this change significantly improves the speed of dying cgroups
> > >     reclaim.
> > >     
> > >     [guro@fb.com: prevent double calculation of DIV64_U64_ROUND_UP() arguments]
> > >       Link: http://lkml.kernel.org/r/20180829213311.GA13501@castle
> > >     Link: http://lkml.kernel.org/r/20180827162621.30187-3-guro@fb.com
> > >     Signed-off-by: Roman Gushchin <guro@fb.com>
> > >     Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
> > >     Cc: Johannes Weiner <hannes@cmpxchg.org>
> > >     Cc: Michal Hocko <mhocko@kernel.org>
> > >     Cc: Tejun Heo <tj@kernel.org>
> > >     Cc: Rik van Riel <riel@surriel.com>
> > >     Cc: Konstantin Khlebnikov <koct9i@gmail.com>
> > >     Cc: Matthew Wilcox <willy@infradead.org>
> > >     Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> > >     Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
> > >     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> > 
> > If you're up for compiling a kernel, you could try reverting just
> > 36f93a2e from 4.14.91 and seeing if your problem goes away.
> > 
> > Regards,
> > Vito Caputo
> 
> I will do that. Thank you.

I just realized I didn't include lkml in replying to you, so I'm adding
them now for posterity.  Please include the list in any further
discussion.

Regards,
Vito Caputo