From: Michal Hocko <mhocko@kernel.org> To: Gerhard Wiesinger <lists@wiesinger.com> Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Linus Torvalds <torvalds@linux-foundation.org> Subject: Re: Still OOM problems with 4.9er kernels Date: Fri, 9 Dec 2016 18:30:18 +0100 [thread overview] Message-ID: <20161209173018.GA31809@dhcp22.suse.cz> (raw) In-Reply-To: <fd029311-f0fe-3d1f-26d2-1f87576b14da@wiesinger.com> On Fri 09-12-16 17:58:14, Gerhard Wiesinger wrote: > On 09.12.2016 17:09, Michal Hocko wrote: [...] > > > [97883.882611] Mem-Info: > > > [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 > > > active_file:3902 inactive_file:3639 isolated_file:0 > > > unevictable:0 dirty:205 writeback:0 unstable:0 > > > slab_reclaimable:9856 slab_unreclaimable:9682 > > > mapped:3722 shmem:59 pagetables:2080 bounce:0 > > > free:748 free_pcp:15 free_cma:0 > > there is still some page cache which doesn't seem to be neither dirty > > nor under writeback. So it should be theoretically reclaimable but for > > some reason we cannot seem to reclaim that memory. > > There is still some anonymous memory and free swap so we could reclaim > > it as well but it all seems pretty down and the memory pressure is > > really large > > Yes, it might be large on the update situation, but that should be handled > by a virtual memory system by the kernel, right? Well this is what we try and call it memory reclaim. But if we are not able to reclaim anything then we eventually have to give up and trigger the OOM killer. Now the information that 4.4 made a difference is interesting. I do not really see any major differences in the reclaim between 4.3 and 4.4 kernels. The reason might be somewhere else as well. E.g. some of the subsystem consumes much more memory than before. Just curious, what kind of filesystem are you using? Could you try some additional debugging. Enabling reclaim related tracepoints might tell us more. The following should tell us more mount -t tracefs none /trace echo 1 > /trace/events/vmscan/enable echo 1 > /trace/events/writeback/writeback_congestion_wait/enable cat /trace/trace_pipe > trace.log Collecting /proc/vmstat over time might be helpful as well mkdir logs while true do cp /proc/vmstat vmstat.$(date +%s) sleep 1s done -- Michal Hocko SUSE Labs
WARNING: multiple messages have this Message-ID (diff)
From: Michal Hocko <mhocko@kernel.org> To: Gerhard Wiesinger <lists@wiesinger.com> Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Linus Torvalds <torvalds@linux-foundation.org> Subject: Re: Still OOM problems with 4.9er kernels Date: Fri, 9 Dec 2016 18:30:18 +0100 [thread overview] Message-ID: <20161209173018.GA31809@dhcp22.suse.cz> (raw) In-Reply-To: <fd029311-f0fe-3d1f-26d2-1f87576b14da@wiesinger.com> On Fri 09-12-16 17:58:14, Gerhard Wiesinger wrote: > On 09.12.2016 17:09, Michal Hocko wrote: [...] > > > [97883.882611] Mem-Info: > > > [97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0 > > > active_file:3902 inactive_file:3639 isolated_file:0 > > > unevictable:0 dirty:205 writeback:0 unstable:0 > > > slab_reclaimable:9856 slab_unreclaimable:9682 > > > mapped:3722 shmem:59 pagetables:2080 bounce:0 > > > free:748 free_pcp:15 free_cma:0 > > there is still some page cache which doesn't seem to be neither dirty > > nor under writeback. So it should be theoretically reclaimable but for > > some reason we cannot seem to reclaim that memory. > > There is still some anonymous memory and free swap so we could reclaim > > it as well but it all seems pretty down and the memory pressure is > > really large > > Yes, it might be large on the update situation, but that should be handled > by a virtual memory system by the kernel, right? Well this is what we try and call it memory reclaim. But if we are not able to reclaim anything then we eventually have to give up and trigger the OOM killer. Now the information that 4.4 made a difference is interesting. I do not really see any major differences in the reclaim between 4.3 and 4.4 kernels. The reason might be somewhere else as well. E.g. some of the subsystem consumes much more memory than before. Just curious, what kind of filesystem are you using? Could you try some additional debugging. Enabling reclaim related tracepoints might tell us more. The following should tell us more mount -t tracefs none /trace echo 1 > /trace/events/vmscan/enable echo 1 > /trace/events/writeback/writeback_congestion_wait/enable cat /trace/trace_pipe > trace.log Collecting /proc/vmstat over time might be helpful as well mkdir logs while true do cp /proc/vmstat vmstat.$(date +%s) sleep 1s done -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2016-12-09 17:30 UTC|newest] Thread overview: 88+ messages / expand[flat|nested] mbox.gz Atom feed top 2016-11-30 7:10 Still OOM problems with 4.9er kernels Gerhard Wiesinger 2016-11-30 7:20 ` Gerhard Wiesinger 2016-12-09 7:06 ` Gerhard Wiesinger 2016-12-09 13:40 ` Michal Hocko 2016-12-09 13:40 ` Michal Hocko 2016-12-09 15:52 ` Gerhard Wiesinger 2016-12-09 15:52 ` Gerhard Wiesinger 2016-12-09 15:58 ` Gerhard Wiesinger 2016-12-09 15:58 ` Gerhard Wiesinger 2016-12-09 16:09 ` Michal Hocko 2016-12-09 16:09 ` Michal Hocko 2016-12-09 16:58 ` Gerhard Wiesinger 2016-12-09 17:30 ` Michal Hocko [this message] 2016-12-09 17:30 ` Michal Hocko 2016-12-09 18:01 ` Gerhard Wiesinger 2016-12-09 18:01 ` Gerhard Wiesinger 2016-12-09 21:42 ` Vlastimil Babka 2016-12-09 21:42 ` Vlastimil Babka 2016-12-10 13:50 ` Gerhard Wiesinger 2016-12-10 13:50 ` Gerhard Wiesinger 2016-12-12 8:24 ` Michal Hocko 2016-12-12 8:24 ` Michal Hocko 2016-12-23 2:55 ` Minchan Kim 2016-12-23 2:55 ` Minchan Kim 2017-01-01 17:20 ` Gerhard Wiesinger 2017-01-01 17:20 ` Gerhard Wiesinger 2017-01-04 8:40 ` Gerhard Wiesinger 2017-01-04 9:11 ` Michal Hocko 2017-01-04 9:11 ` Michal Hocko 2017-02-26 8:40 ` Still OOM problems with 4.9er/4.10er kernels Gerhard Wiesinger 2017-02-27 8:27 ` Michal Hocko 2017-02-27 8:27 ` Michal Hocko 2017-02-28 6:06 ` Gerhard Wiesinger 2017-02-28 6:06 ` Gerhard Wiesinger 2017-02-28 8:14 ` Michal Hocko 2017-02-28 8:14 ` Michal Hocko 2017-02-27 9:02 ` Minchan Kim 2017-02-27 9:02 ` Minchan Kim 2017-02-27 9:44 ` Michal Hocko 2017-02-27 9:44 ` Michal Hocko 2017-02-28 5:17 ` Minchan Kim 2017-02-28 5:17 ` Minchan Kim 2017-02-28 8:12 ` Michal Hocko 2017-02-28 8:12 ` Michal Hocko 2017-03-02 7:17 ` Minchan Kim 2017-03-02 7:17 ` Minchan Kim 2017-03-16 6:38 ` Gerhard Wiesinger 2017-03-16 6:38 ` Gerhard Wiesinger 2017-03-16 8:27 ` Michal Hocko 2017-03-16 8:27 ` Michal Hocko 2017-03-16 8:47 ` lkml 2017-03-16 8:47 ` lkml 2017-03-16 9:08 ` Michal Hocko 2017-03-16 9:08 ` Michal Hocko 2017-03-16 9:23 ` lkml 2017-03-16 9:23 ` lkml 2017-03-16 9:39 ` Michal Hocko 2017-03-16 9:39 ` Michal Hocko 2017-03-17 16:37 ` Gerhard Wiesinger 2017-03-17 16:37 ` Gerhard Wiesinger 2017-03-17 17:13 ` Michal Hocko 2017-03-17 17:13 ` Michal Hocko 2017-03-17 20:08 ` Gerhard Wiesinger 2017-03-17 20:08 ` Gerhard Wiesinger 2017-03-19 8:17 ` Gerhard Wiesinger 2017-03-19 8:17 ` Gerhard Wiesinger 2017-03-20 1:54 ` Tetsuo Handa 2017-03-20 1:54 ` Tetsuo Handa 2017-03-19 15:18 ` Michal Hocko 2017-03-19 15:18 ` Michal Hocko 2017-03-19 16:02 ` Gerhard Wiesinger 2017-03-19 16:02 ` Gerhard Wiesinger 2017-03-20 3:05 ` Mike Galbraith 2017-03-20 3:05 ` Mike Galbraith 2017-03-21 5:59 ` Gerhard Wiesinger 2017-03-21 5:59 ` Gerhard Wiesinger 2017-03-21 7:13 ` Mike Galbraith 2017-03-21 7:13 ` Mike Galbraith 2017-03-23 7:16 ` Gerhard Wiesinger 2017-03-23 7:16 ` Gerhard Wiesinger 2017-03-23 8:38 ` Mike Galbraith 2017-03-23 8:38 ` Mike Galbraith 2017-03-23 14:46 ` Tetsuo Handa 2017-03-23 14:46 ` Tetsuo Handa 2017-03-26 8:36 ` Gerhard Wiesinger 2017-03-26 8:36 ` Gerhard Wiesinger 2016-12-09 16:03 ` Still OOM problems with 4.9er kernels Gerhard Wiesinger 2016-12-09 16:03 ` Gerhard Wiesinger
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20161209173018.GA31809@dhcp22.suse.cz \ --to=mhocko@kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=lists@wiesinger.com \ --cc=torvalds@linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.