From: Michal Hocko <mhocko@kernel.org> To: Gerhard Wiesinger <lists@wiesinger.com> Cc: lkml@pengaru.com, Minchan Kim <minchan@kernel.org>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Linus Torvalds <torvalds@linux-foundation.org> Subject: Re: Still OOM problems with 4.9er/4.10er kernels Date: Sun, 19 Mar 2017 11:18:38 -0400 [thread overview] Message-ID: <20170319151837.GD12414@dhcp22.suse.cz> (raw) In-Reply-To: <8cb1d796-aff3-0063-3ef8-880e76d437c0@wiesinger.com> On Fri 17-03-17 21:08:31, Gerhard Wiesinger wrote: > On 17.03.2017 18:13, Michal Hocko wrote: > >On Fri 17-03-17 17:37:48, Gerhard Wiesinger wrote: > >[...] > >>Why does the kernel prefer to swapin/out and not use > >> > >>a.) the free memory? > >It will use all the free memory up to min watermark which is set up > >based on min_free_kbytes. > > Makes sense, how is /proc/sys/vm/min_free_kbytes default value calculated? See init_per_zone_wmark_min > >>b.) the buffer/cache? > >the memory reclaim is strongly biased towards page cache and we try to > >avoid swapout as much as possible (see get_scan_count). > > If I understand it correctly, swapping is preferred over dropping the > cache, right. Can this behaviour be changed to prefer dropping the > cache to some minimum amount? Is this also configurable in a way? No, we enforce swapping if the amount of free + file pages are below the cumulative high watermark. > (As far as I remember e.g. kernel 2.4 dropped the caches well). > > >>There is ~100M memory available but kernel swaps all the time ... > >> > >>Any ideas? > >> > >>Kernel: 4.9.14-200.fc25.x86_64 > >> > >>top - 17:33:43 up 28 min, 3 users, load average: 3.58, 1.67, 0.89 > >>Tasks: 145 total, 4 running, 141 sleeping, 0 stopped, 0 zombie > >>%Cpu(s): 19.1 us, 56.2 sy, 0.0 ni, 4.3 id, 13.4 wa, 2.0 hi, 0.3 si, 4.7 > >>st > >>KiB Mem : 230076 total, 61508 free, 123472 used, 45096 buff/cache > >> > >>procs -----------memory---------- ---swap-- -----io---- -system-- > >>------cpu----- > >> r b swpd free buff cache si so bi bo in cs us sy id wa st > >> 3 5 303916 60372 328 43864 27828 200 41420 236 6984 11138 11 47 6 23 14 > >I am really surprised to see any reclaim at all. 26% of free memory > >doesn't sound as if we should do a reclaim at all. Do you have an > >unusual configuration of /proc/sys/vm/min_free_kbytes ? Or is there > >anything running inside a memory cgroup with a small limit? > > nothing special set regarding /proc/sys/vm/min_free_kbytes (default values), > detailed config below. Regarding cgroups, none of I know. How to check (I > guess nothing is set because cg* commands are not available)? be careful because systemd started to use some controllers. You can easily check cgroup mount points. > /proc/sys/vm/min_free_kbytes > 45056 So at least 45M will be kept reserved for the system. Your data indicated you had more memory. How does /proc/zoneinfo look like? Btw. you seem to be using fc kernel, are there any patches applied on top of Linus tree? Could you try to retest vanilla kernel? -- Michal Hocko SUSE Labs
WARNING: multiple messages have this Message-ID (diff)
From: Michal Hocko <mhocko@kernel.org> To: Gerhard Wiesinger <lists@wiesinger.com> Cc: lkml@pengaru.com, Minchan Kim <minchan@kernel.org>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Linus Torvalds <torvalds@linux-foundation.org> Subject: Re: Still OOM problems with 4.9er/4.10er kernels Date: Sun, 19 Mar 2017 11:18:38 -0400 [thread overview] Message-ID: <20170319151837.GD12414@dhcp22.suse.cz> (raw) In-Reply-To: <8cb1d796-aff3-0063-3ef8-880e76d437c0@wiesinger.com> On Fri 17-03-17 21:08:31, Gerhard Wiesinger wrote: > On 17.03.2017 18:13, Michal Hocko wrote: > >On Fri 17-03-17 17:37:48, Gerhard Wiesinger wrote: > >[...] > >>Why does the kernel prefer to swapin/out and not use > >> > >>a.) the free memory? > >It will use all the free memory up to min watermark which is set up > >based on min_free_kbytes. > > Makes sense, how is /proc/sys/vm/min_free_kbytes default value calculated? See init_per_zone_wmark_min > >>b.) the buffer/cache? > >the memory reclaim is strongly biased towards page cache and we try to > >avoid swapout as much as possible (see get_scan_count). > > If I understand it correctly, swapping is preferred over dropping the > cache, right. Can this behaviour be changed to prefer dropping the > cache to some minimum amount? Is this also configurable in a way? No, we enforce swapping if the amount of free + file pages are below the cumulative high watermark. > (As far as I remember e.g. kernel 2.4 dropped the caches well). > > >>There is ~100M memory available but kernel swaps all the time ... > >> > >>Any ideas? > >> > >>Kernel: 4.9.14-200.fc25.x86_64 > >> > >>top - 17:33:43 up 28 min, 3 users, load average: 3.58, 1.67, 0.89 > >>Tasks: 145 total, 4 running, 141 sleeping, 0 stopped, 0 zombie > >>%Cpu(s): 19.1 us, 56.2 sy, 0.0 ni, 4.3 id, 13.4 wa, 2.0 hi, 0.3 si, 4.7 > >>st > >>KiB Mem : 230076 total, 61508 free, 123472 used, 45096 buff/cache > >> > >>procs -----------memory---------- ---swap-- -----io---- -system-- > >>------cpu----- > >> r b swpd free buff cache si so bi bo in cs us sy id wa st > >> 3 5 303916 60372 328 43864 27828 200 41420 236 6984 11138 11 47 6 23 14 > >I am really surprised to see any reclaim at all. 26% of free memory > >doesn't sound as if we should do a reclaim at all. Do you have an > >unusual configuration of /proc/sys/vm/min_free_kbytes ? Or is there > >anything running inside a memory cgroup with a small limit? > > nothing special set regarding /proc/sys/vm/min_free_kbytes (default values), > detailed config below. Regarding cgroups, none of I know. How to check (I > guess nothing is set because cg* commands are not available)? be careful because systemd started to use some controllers. You can easily check cgroup mount points. > /proc/sys/vm/min_free_kbytes > 45056 So at least 45M will be kept reserved for the system. Your data indicated you had more memory. How does /proc/zoneinfo look like? Btw. you seem to be using fc kernel, are there any patches applied on top of Linus tree? Could you try to retest vanilla kernel? -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2017-03-19 15:19 UTC|newest] Thread overview: 88+ messages / expand[flat|nested] mbox.gz Atom feed top 2016-11-30 7:10 Still OOM problems with 4.9er kernels Gerhard Wiesinger 2016-11-30 7:20 ` Gerhard Wiesinger 2016-12-09 7:06 ` Gerhard Wiesinger 2016-12-09 13:40 ` Michal Hocko 2016-12-09 13:40 ` Michal Hocko 2016-12-09 15:52 ` Gerhard Wiesinger 2016-12-09 15:52 ` Gerhard Wiesinger 2016-12-09 15:58 ` Gerhard Wiesinger 2016-12-09 15:58 ` Gerhard Wiesinger 2016-12-09 16:09 ` Michal Hocko 2016-12-09 16:09 ` Michal Hocko 2016-12-09 16:58 ` Gerhard Wiesinger 2016-12-09 17:30 ` Michal Hocko 2016-12-09 17:30 ` Michal Hocko 2016-12-09 18:01 ` Gerhard Wiesinger 2016-12-09 18:01 ` Gerhard Wiesinger 2016-12-09 21:42 ` Vlastimil Babka 2016-12-09 21:42 ` Vlastimil Babka 2016-12-10 13:50 ` Gerhard Wiesinger 2016-12-10 13:50 ` Gerhard Wiesinger 2016-12-12 8:24 ` Michal Hocko 2016-12-12 8:24 ` Michal Hocko 2016-12-23 2:55 ` Minchan Kim 2016-12-23 2:55 ` Minchan Kim 2017-01-01 17:20 ` Gerhard Wiesinger 2017-01-01 17:20 ` Gerhard Wiesinger 2017-01-04 8:40 ` Gerhard Wiesinger 2017-01-04 9:11 ` Michal Hocko 2017-01-04 9:11 ` Michal Hocko 2017-02-26 8:40 ` Still OOM problems with 4.9er/4.10er kernels Gerhard Wiesinger 2017-02-27 8:27 ` Michal Hocko 2017-02-27 8:27 ` Michal Hocko 2017-02-28 6:06 ` Gerhard Wiesinger 2017-02-28 6:06 ` Gerhard Wiesinger 2017-02-28 8:14 ` Michal Hocko 2017-02-28 8:14 ` Michal Hocko 2017-02-27 9:02 ` Minchan Kim 2017-02-27 9:02 ` Minchan Kim 2017-02-27 9:44 ` Michal Hocko 2017-02-27 9:44 ` Michal Hocko 2017-02-28 5:17 ` Minchan Kim 2017-02-28 5:17 ` Minchan Kim 2017-02-28 8:12 ` Michal Hocko 2017-02-28 8:12 ` Michal Hocko 2017-03-02 7:17 ` Minchan Kim 2017-03-02 7:17 ` Minchan Kim 2017-03-16 6:38 ` Gerhard Wiesinger 2017-03-16 6:38 ` Gerhard Wiesinger 2017-03-16 8:27 ` Michal Hocko 2017-03-16 8:27 ` Michal Hocko 2017-03-16 8:47 ` lkml 2017-03-16 8:47 ` lkml 2017-03-16 9:08 ` Michal Hocko 2017-03-16 9:08 ` Michal Hocko 2017-03-16 9:23 ` lkml 2017-03-16 9:23 ` lkml 2017-03-16 9:39 ` Michal Hocko 2017-03-16 9:39 ` Michal Hocko 2017-03-17 16:37 ` Gerhard Wiesinger 2017-03-17 16:37 ` Gerhard Wiesinger 2017-03-17 17:13 ` Michal Hocko 2017-03-17 17:13 ` Michal Hocko 2017-03-17 20:08 ` Gerhard Wiesinger 2017-03-17 20:08 ` Gerhard Wiesinger 2017-03-19 8:17 ` Gerhard Wiesinger 2017-03-19 8:17 ` Gerhard Wiesinger 2017-03-20 1:54 ` Tetsuo Handa 2017-03-20 1:54 ` Tetsuo Handa 2017-03-19 15:18 ` Michal Hocko [this message] 2017-03-19 15:18 ` Michal Hocko 2017-03-19 16:02 ` Gerhard Wiesinger 2017-03-19 16:02 ` Gerhard Wiesinger 2017-03-20 3:05 ` Mike Galbraith 2017-03-20 3:05 ` Mike Galbraith 2017-03-21 5:59 ` Gerhard Wiesinger 2017-03-21 5:59 ` Gerhard Wiesinger 2017-03-21 7:13 ` Mike Galbraith 2017-03-21 7:13 ` Mike Galbraith 2017-03-23 7:16 ` Gerhard Wiesinger 2017-03-23 7:16 ` Gerhard Wiesinger 2017-03-23 8:38 ` Mike Galbraith 2017-03-23 8:38 ` Mike Galbraith 2017-03-23 14:46 ` Tetsuo Handa 2017-03-23 14:46 ` Tetsuo Handa 2017-03-26 8:36 ` Gerhard Wiesinger 2017-03-26 8:36 ` Gerhard Wiesinger 2016-12-09 16:03 ` Still OOM problems with 4.9er kernels Gerhard Wiesinger 2016-12-09 16:03 ` Gerhard Wiesinger
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20170319151837.GD12414@dhcp22.suse.cz \ --to=mhocko@kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=lists@wiesinger.com \ --cc=lkml@pengaru.com \ --cc=minchan@kernel.org \ --cc=torvalds@linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.