From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933519Ab3GPQJJ (ORCPT ); Tue, 16 Jul 2013 12:09:09 -0400 Received: from cantor2.suse.de ([195.135.220.15]:42166 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932807Ab3GPQJH (ORCPT ); Tue, 16 Jul 2013 12:09:07 -0400 Date: Tue, 16 Jul 2013 18:09:05 +0200 From: Michal Hocko To: Johannes Weiner Cc: azurIt , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups mailinglist , KAMEZAWA Hiroyuki , righi.andrea@gmail.com Subject: Re: [PATCH for 3.2] memcg: do not trap chargers with full callstack on OOM Message-ID: <20130716160905.GA20018@dhcp22.suse.cz> References: <20130709131029.GH20281@dhcp22.suse.cz> <20130709151921.5160C199@pobox.sk> <20130709135450.GI20281@dhcp22.suse.cz> <20130710182506.F25DF461@pobox.sk> <20130711072507.GA21667@dhcp22.suse.cz> <20130714012641.C2DA4E05@pobox.sk> <20130714015112.FFCB7AF7@pobox.sk> <20130715154119.GA32435@dhcp22.suse.cz> <20130715160006.GB32435@dhcp22.suse.cz> <20130716153544.GX17812@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130716153544.GX17812@cmpxchg.org> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue 16-07-13 11:35:44, Johannes Weiner wrote: > On Mon, Jul 15, 2013 at 06:00:06PM +0200, Michal Hocko wrote: > > On Mon 15-07-13 17:41:19, Michal Hocko wrote: > > > On Sun 14-07-13 01:51:12, azurIt wrote: > > > > > CC: "Johannes Weiner" , linux-kernel@vger.kernel.org, linux-mm@kvack.org, "cgroups mailinglist" , "KAMEZAWA Hiroyuki" , righi.andrea@gmail.com > > > > >> CC: "Johannes Weiner" , linux-kernel@vger.kernel.org, linux-mm@kvack.org, "cgroups mailinglist" , "KAMEZAWA Hiroyuki" , righi.andrea@gmail.com > > > > >>On Wed 10-07-13 18:25:06, azurIt wrote: > > > > >>> >> Now i realized that i forgot to remove UID from that cgroup before > > > > >>> >> trying to remove it, so cgroup cannot be removed anyway (we are using > > > > >>> >> third party cgroup called cgroup-uid from Andrea Righi, which is able > > > > >>> >> to associate all user's processes with target cgroup). Look here for > > > > >>> >> cgroup-uid patch: > > > > >>> >> https://www.develer.com/~arighi/linux/patches/cgroup-uid/cgroup-uid-v8.patch > > > > >>> >> > > > > >>> >> ANYWAY, i'm 101% sure that 'tasks' file was empty and 'under_oom' was > > > > >>> >> permanently '1'. > > > > >>> > > > > > >>> >This is really strange. Could you post the whole diff against stable > > > > >>> >tree you are using (except for grsecurity stuff and the above cgroup-uid > > > > >>> >patch)? > > > > >>> > > > > >>> > > > > >>> Here are all patches which i applied to kernel 3.2.48 in my last test: > > > > >>> http://watchdog.sk/lkml/patches3/ > > > > >> > > > > >>The two patches from Johannes seem correct. > > > > >> > > > > >>From a quick look even grsecurity patchset shouldn't interfere as it > > > > >>doesn't seem to put any code between handle_mm_fault and mm_fault_error > > > > >>and there also doesn't seem to be any new handle_mm_fault call sites. > > > > >> > > > > >>But I cannot tell there aren't other code paths which would lead to a > > > > >>memcg charge, thus oom, without proper FAULT_FLAG_KERNEL handling. > > > > > > > > > > > > > > >Michal, > > > > > > > > > >now i can definitely confirm that problem with unremovable cgroups > > > > >persists. What info do you need from me? I applied also your little > > > > >'WARN_ON' patch. > > > > > > > > Ok, i think you want this: > > > > http://watchdog.sk/lkml/kern4.log > > > > > > Jul 14 01:11:39 server01 kernel: [ 593.589087] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name > > > Jul 14 01:11:39 server01 kernel: [ 593.589451] [12021] 1333 12021 172027 64723 4 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.589647] [12030] 1333 12030 172030 64748 2 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.589836] [12031] 1333 12031 172030 64749 3 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590025] [12032] 1333 12032 170619 63428 3 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590213] [12033] 1333 12033 167934 60524 2 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590401] [12034] 1333 12034 170747 63496 4 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590588] [12035] 1333 12035 169659 62451 1 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590776] [12036] 1333 12036 167614 60384 3 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590984] [12037] 1333 12037 166342 58964 3 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.591178] Memory cgroup out of memory: Kill process 12021 (apache2) score 847 or sacrifice child > > > Jul 14 01:11:39 server01 kernel: [ 593.591370] Killed process 12021 (apache2) total-vm:688108kB, anon-rss:255472kB, file-rss:3420kB > > > Jul 14 01:11:41 server01 kernel: [ 595.392920] ------------[ cut here ]------------ > > > Jul 14 01:11:41 server01 kernel: [ 595.393096] WARNING: at kernel/exit.c:888 do_exit+0x7d0/0x870() > > > Jul 14 01:11:41 server01 kernel: [ 595.393256] Hardware name: S5000VSA > > > Jul 14 01:11:41 server01 kernel: [ 595.393415] Pid: 12037, comm: apache2 Not tainted 3.2.48-grsec #1 > > > Jul 14 01:11:41 server01 kernel: [ 595.393577] Call Trace: > > > Jul 14 01:11:41 server01 kernel: [ 595.393737] [] warn_slowpath_common+0x7a/0xb0 > > > Jul 14 01:11:41 server01 kernel: [ 595.393903] [] warn_slowpath_null+0x1a/0x20 > > > Jul 14 01:11:41 server01 kernel: [ 595.394068] [] do_exit+0x7d0/0x870 > > > Jul 14 01:11:41 server01 kernel: [ 595.394231] [] ? thread_group_times+0x44/0xb0 > > > Jul 14 01:11:41 server01 kernel: [ 595.394392] [] do_group_exit+0x51/0xc0 > > > Jul 14 01:11:41 server01 kernel: [ 595.394551] [] sys_exit_group+0x17/0x20 > > > Jul 14 01:11:41 server01 kernel: [ 595.394714] [] system_call_fastpath+0x18/0x1d > > > Jul 14 01:11:41 server01 kernel: [ 595.394921] ---[ end trace 738570e688acf099 ]--- > > > > > > OK, so you had an OOM which has been handled by in-kernel oom handler > > > (it killed 12021) and 12037 was in the same group. The warning tells us > > > that it went through mem_cgroup_oom as well (otherwise it wouldn't have > > > memcg_oom.wait_on_memcg set and the warning wouldn't trigger) and then > > > it exited on the userspace request (by exit syscall). > > > > > > I do not see any way how, this could happen though. If mem_cgroup_oom > > > is called then we always return CHARGE_NOMEM which turns into ENOMEM > > > returned by __mem_cgroup_try_charge (invoke_oom must have been set to > > > true). So if nobody screwed the return value on the way up to page > > > fault handler then there is no way to escape. > > > > > > I will check the code. > > > > OK, I guess I found it: > > __do_fault > > fault = filemap_fault > > do_async_mmap_readahead > > page_cache_async_readahead > > ondemand_readahead > > __do_page_cache_readahead > > read_pages > > readpages = ext3_readpages > > mpage_readpages # Doesn't propagate ENOMEM > > add_to_page_cache_lru > > add_to_page_cache > > add_to_page_cache_locked > > mem_cgroup_cache_charge > > > > So the read ahead most probably. Again! Duhhh. I will try to think > > about a fix for this. One obvious place is mpage_readpages but > > __do_page_cache_readahead ignores read_pages return value as well and > > page_cache_async_readahead, even worse, is just void and exported as > > such. > > > > So this smells like a hard to fix bugger. One possible, and really ugly > > way would be calling mem_cgroup_oom_synchronize even if handle_mm_fault > > doesn't return VM_FAULT_ERROR, but that is a crude hack. > > Ouch, good spot. > > I don't think we need to handle an OOM from the readahead code. If > readahead does not produce the desired page, we retry synchroneously > in page_cache_read() and handle the OOM properly. We should not > signal an OOM for optional pages anyway. > > So either we pass a flag from the readahead code down to > add_to_page_cache and mem_cgroup_cache_charge that tells the charge > code to ignore OOM conditions and do not set up an OOM context. That was my previous attempt and it was sooo painful. > Or we DO call mem_cgroup_oom_synchronize() from the read_cache_pages, > with an argument that makes it only clean up the context and not wait. Yes, I was playing with this idea as well. I just do not like how fragile this is. We need some way to catch all possible places which might leak it. > It would not be completely outlandish to place it there, since it's > right next to where an error from add_to_page_cache() is not further > propagated back through the fault stack. > > I'm travelling right now, I'll send a patch when I get back > (Thursday). Unless you beat me to it :) I can cook something up but there is quite a big pile on my desk currently (as always :/). -- Michal Hocko SUSE Labs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx203.postini.com [74.125.245.203]) by kanga.kvack.org (Postfix) with SMTP id 592BA6B0034 for ; Tue, 16 Jul 2013 12:09:07 -0400 (EDT) Date: Tue, 16 Jul 2013 18:09:05 +0200 From: Michal Hocko Subject: Re: [PATCH for 3.2] memcg: do not trap chargers with full callstack on OOM Message-ID: <20130716160905.GA20018@dhcp22.suse.cz> References: <20130709131029.GH20281@dhcp22.suse.cz> <20130709151921.5160C199@pobox.sk> <20130709135450.GI20281@dhcp22.suse.cz> <20130710182506.F25DF461@pobox.sk> <20130711072507.GA21667@dhcp22.suse.cz> <20130714012641.C2DA4E05@pobox.sk> <20130714015112.FFCB7AF7@pobox.sk> <20130715154119.GA32435@dhcp22.suse.cz> <20130715160006.GB32435@dhcp22.suse.cz> <20130716153544.GX17812@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130716153544.GX17812@cmpxchg.org> Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: azurIt , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups mailinglist , KAMEZAWA Hiroyuki , righi.andrea@gmail.com On Tue 16-07-13 11:35:44, Johannes Weiner wrote: > On Mon, Jul 15, 2013 at 06:00:06PM +0200, Michal Hocko wrote: > > On Mon 15-07-13 17:41:19, Michal Hocko wrote: > > > On Sun 14-07-13 01:51:12, azurIt wrote: > > > > > CC: "Johannes Weiner" , linux-kernel@vger.kernel.org, linux-mm@kvack.org, "cgroups mailinglist" , "KAMEZAWA Hiroyuki" , righi.andrea@gmail.com > > > > >> CC: "Johannes Weiner" , linux-kernel@vger.kernel.org, linux-mm@kvack.org, "cgroups mailinglist" , "KAMEZAWA Hiroyuki" , righi.andrea@gmail.com > > > > >>On Wed 10-07-13 18:25:06, azurIt wrote: > > > > >>> >> Now i realized that i forgot to remove UID from that cgroup before > > > > >>> >> trying to remove it, so cgroup cannot be removed anyway (we are using > > > > >>> >> third party cgroup called cgroup-uid from Andrea Righi, which is able > > > > >>> >> to associate all user's processes with target cgroup). Look here for > > > > >>> >> cgroup-uid patch: > > > > >>> >> https://www.develer.com/~arighi/linux/patches/cgroup-uid/cgroup-uid-v8.patch > > > > >>> >> > > > > >>> >> ANYWAY, i'm 101% sure that 'tasks' file was empty and 'under_oom' was > > > > >>> >> permanently '1'. > > > > >>> > > > > > >>> >This is really strange. Could you post the whole diff against stable > > > > >>> >tree you are using (except for grsecurity stuff and the above cgroup-uid > > > > >>> >patch)? > > > > >>> > > > > >>> > > > > >>> Here are all patches which i applied to kernel 3.2.48 in my last test: > > > > >>> http://watchdog.sk/lkml/patches3/ > > > > >> > > > > >>The two patches from Johannes seem correct. > > > > >> > > > > >>From a quick look even grsecurity patchset shouldn't interfere as it > > > > >>doesn't seem to put any code between handle_mm_fault and mm_fault_error > > > > >>and there also doesn't seem to be any new handle_mm_fault call sites. > > > > >> > > > > >>But I cannot tell there aren't other code paths which would lead to a > > > > >>memcg charge, thus oom, without proper FAULT_FLAG_KERNEL handling. > > > > > > > > > > > > > > >Michal, > > > > > > > > > >now i can definitely confirm that problem with unremovable cgroups > > > > >persists. What info do you need from me? I applied also your little > > > > >'WARN_ON' patch. > > > > > > > > Ok, i think you want this: > > > > http://watchdog.sk/lkml/kern4.log > > > > > > Jul 14 01:11:39 server01 kernel: [ 593.589087] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name > > > Jul 14 01:11:39 server01 kernel: [ 593.589451] [12021] 1333 12021 172027 64723 4 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.589647] [12030] 1333 12030 172030 64748 2 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.589836] [12031] 1333 12031 172030 64749 3 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590025] [12032] 1333 12032 170619 63428 3 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590213] [12033] 1333 12033 167934 60524 2 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590401] [12034] 1333 12034 170747 63496 4 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590588] [12035] 1333 12035 169659 62451 1 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590776] [12036] 1333 12036 167614 60384 3 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590984] [12037] 1333 12037 166342 58964 3 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.591178] Memory cgroup out of memory: Kill process 12021 (apache2) score 847 or sacrifice child > > > Jul 14 01:11:39 server01 kernel: [ 593.591370] Killed process 12021 (apache2) total-vm:688108kB, anon-rss:255472kB, file-rss:3420kB > > > Jul 14 01:11:41 server01 kernel: [ 595.392920] ------------[ cut here ]------------ > > > Jul 14 01:11:41 server01 kernel: [ 595.393096] WARNING: at kernel/exit.c:888 do_exit+0x7d0/0x870() > > > Jul 14 01:11:41 server01 kernel: [ 595.393256] Hardware name: S5000VSA > > > Jul 14 01:11:41 server01 kernel: [ 595.393415] Pid: 12037, comm: apache2 Not tainted 3.2.48-grsec #1 > > > Jul 14 01:11:41 server01 kernel: [ 595.393577] Call Trace: > > > Jul 14 01:11:41 server01 kernel: [ 595.393737] [] warn_slowpath_common+0x7a/0xb0 > > > Jul 14 01:11:41 server01 kernel: [ 595.393903] [] warn_slowpath_null+0x1a/0x20 > > > Jul 14 01:11:41 server01 kernel: [ 595.394068] [] do_exit+0x7d0/0x870 > > > Jul 14 01:11:41 server01 kernel: [ 595.394231] [] ? thread_group_times+0x44/0xb0 > > > Jul 14 01:11:41 server01 kernel: [ 595.394392] [] do_group_exit+0x51/0xc0 > > > Jul 14 01:11:41 server01 kernel: [ 595.394551] [] sys_exit_group+0x17/0x20 > > > Jul 14 01:11:41 server01 kernel: [ 595.394714] [] system_call_fastpath+0x18/0x1d > > > Jul 14 01:11:41 server01 kernel: [ 595.394921] ---[ end trace 738570e688acf099 ]--- > > > > > > OK, so you had an OOM which has been handled by in-kernel oom handler > > > (it killed 12021) and 12037 was in the same group. The warning tells us > > > that it went through mem_cgroup_oom as well (otherwise it wouldn't have > > > memcg_oom.wait_on_memcg set and the warning wouldn't trigger) and then > > > it exited on the userspace request (by exit syscall). > > > > > > I do not see any way how, this could happen though. If mem_cgroup_oom > > > is called then we always return CHARGE_NOMEM which turns into ENOMEM > > > returned by __mem_cgroup_try_charge (invoke_oom must have been set to > > > true). So if nobody screwed the return value on the way up to page > > > fault handler then there is no way to escape. > > > > > > I will check the code. > > > > OK, I guess I found it: > > __do_fault > > fault = filemap_fault > > do_async_mmap_readahead > > page_cache_async_readahead > > ondemand_readahead > > __do_page_cache_readahead > > read_pages > > readpages = ext3_readpages > > mpage_readpages # Doesn't propagate ENOMEM > > add_to_page_cache_lru > > add_to_page_cache > > add_to_page_cache_locked > > mem_cgroup_cache_charge > > > > So the read ahead most probably. Again! Duhhh. I will try to think > > about a fix for this. One obvious place is mpage_readpages but > > __do_page_cache_readahead ignores read_pages return value as well and > > page_cache_async_readahead, even worse, is just void and exported as > > such. > > > > So this smells like a hard to fix bugger. One possible, and really ugly > > way would be calling mem_cgroup_oom_synchronize even if handle_mm_fault > > doesn't return VM_FAULT_ERROR, but that is a crude hack. > > Ouch, good spot. > > I don't think we need to handle an OOM from the readahead code. If > readahead does not produce the desired page, we retry synchroneously > in page_cache_read() and handle the OOM properly. We should not > signal an OOM for optional pages anyway. > > So either we pass a flag from the readahead code down to > add_to_page_cache and mem_cgroup_cache_charge that tells the charge > code to ignore OOM conditions and do not set up an OOM context. That was my previous attempt and it was sooo painful. > Or we DO call mem_cgroup_oom_synchronize() from the read_cache_pages, > with an argument that makes it only clean up the context and not wait. Yes, I was playing with this idea as well. I just do not like how fragile this is. We need some way to catch all possible places which might leak it. > It would not be completely outlandish to place it there, since it's > right next to where an error from add_to_page_cache() is not further > propagated back through the fault stack. > > I'm travelling right now, I'll send a patch when I get back > (Thursday). Unless you beat me to it :) I can cook something up but there is quite a big pile on my desk currently (as always :/). -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michal Hocko Subject: Re: [PATCH for 3.2] memcg: do not trap chargers with full callstack on OOM Date: Tue, 16 Jul 2013 18:09:05 +0200 Message-ID: <20130716160905.GA20018@dhcp22.suse.cz> References: <20130709131029.GH20281@dhcp22.suse.cz> <20130709151921.5160C199@pobox.sk> <20130709135450.GI20281@dhcp22.suse.cz> <20130710182506.F25DF461@pobox.sk> <20130711072507.GA21667@dhcp22.suse.cz> <20130714012641.C2DA4E05@pobox.sk> <20130714015112.FFCB7AF7@pobox.sk> <20130715154119.GA32435@dhcp22.suse.cz> <20130715160006.GB32435@dhcp22.suse.cz> <20130716153544.GX17812@cmpxchg.org> Mime-Version: 1.0 Return-path: Content-Disposition: inline In-Reply-To: <20130716153544.GX17812-druUgvl0LCNAfugRpC6u6w@public.gmane.org> Sender: cgroups-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-ID: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Johannes Weiner Cc: azurIt , linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, cgroups mailinglist , KAMEZAWA Hiroyuki , righi.andrea-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org On Tue 16-07-13 11:35:44, Johannes Weiner wrote: > On Mon, Jul 15, 2013 at 06:00:06PM +0200, Michal Hocko wrote: > > On Mon 15-07-13 17:41:19, Michal Hocko wrote: > > > On Sun 14-07-13 01:51:12, azurIt wrote: > > > > > CC: "Johannes Weiner" , linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, "cgroups mailinglist" , "KAMEZAWA Hiroyuki" , righi.andrea-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org > > > > >> CC: "Johannes Weiner" , linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, "cgroups mailinglist" , "KAMEZAWA Hiroyuki" , righi.andrea-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org > > > > >>On Wed 10-07-13 18:25:06, azurIt wrote: > > > > >>> >> Now i realized that i forgot to remove UID from that cgroup before > > > > >>> >> trying to remove it, so cgroup cannot be removed anyway (we are using > > > > >>> >> third party cgroup called cgroup-uid from Andrea Righi, which is able > > > > >>> >> to associate all user's processes with target cgroup). Look here for > > > > >>> >> cgroup-uid patch: > > > > >>> >> https://www.develer.com/~arighi/linux/patches/cgroup-uid/cgroup-uid-v8.patch > > > > >>> >> > > > > >>> >> ANYWAY, i'm 101% sure that 'tasks' file was empty and 'under_oom' was > > > > >>> >> permanently '1'. > > > > >>> > > > > > >>> >This is really strange. Could you post the whole diff against stable > > > > >>> >tree you are using (except for grsecurity stuff and the above cgroup-uid > > > > >>> >patch)? > > > > >>> > > > > >>> > > > > >>> Here are all patches which i applied to kernel 3.2.48 in my last test: > > > > >>> http://watchdog.sk/lkml/patches3/ > > > > >> > > > > >>The two patches from Johannes seem correct. > > > > >> > > > > >>From a quick look even grsecurity patchset shouldn't interfere as it > > > > >>doesn't seem to put any code between handle_mm_fault and mm_fault_error > > > > >>and there also doesn't seem to be any new handle_mm_fault call sites. > > > > >> > > > > >>But I cannot tell there aren't other code paths which would lead to a > > > > >>memcg charge, thus oom, without proper FAULT_FLAG_KERNEL handling. > > > > > > > > > > > > > > >Michal, > > > > > > > > > >now i can definitely confirm that problem with unremovable cgroups > > > > >persists. What info do you need from me? I applied also your little > > > > >'WARN_ON' patch. > > > > > > > > Ok, i think you want this: > > > > http://watchdog.sk/lkml/kern4.log > > > > > > Jul 14 01:11:39 server01 kernel: [ 593.589087] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name > > > Jul 14 01:11:39 server01 kernel: [ 593.589451] [12021] 1333 12021 172027 64723 4 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.589647] [12030] 1333 12030 172030 64748 2 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.589836] [12031] 1333 12031 172030 64749 3 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590025] [12032] 1333 12032 170619 63428 3 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590213] [12033] 1333 12033 167934 60524 2 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590401] [12034] 1333 12034 170747 63496 4 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590588] [12035] 1333 12035 169659 62451 1 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590776] [12036] 1333 12036 167614 60384 3 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.590984] [12037] 1333 12037 166342 58964 3 0 0 apache2 > > > Jul 14 01:11:39 server01 kernel: [ 593.591178] Memory cgroup out of memory: Kill process 12021 (apache2) score 847 or sacrifice child > > > Jul 14 01:11:39 server01 kernel: [ 593.591370] Killed process 12021 (apache2) total-vm:688108kB, anon-rss:255472kB, file-rss:3420kB > > > Jul 14 01:11:41 server01 kernel: [ 595.392920] ------------[ cut here ]------------ > > > Jul 14 01:11:41 server01 kernel: [ 595.393096] WARNING: at kernel/exit.c:888 do_exit+0x7d0/0x870() > > > Jul 14 01:11:41 server01 kernel: [ 595.393256] Hardware name: S5000VSA > > > Jul 14 01:11:41 server01 kernel: [ 595.393415] Pid: 12037, comm: apache2 Not tainted 3.2.48-grsec #1 > > > Jul 14 01:11:41 server01 kernel: [ 595.393577] Call Trace: > > > Jul 14 01:11:41 server01 kernel: [ 595.393737] [] warn_slowpath_common+0x7a/0xb0 > > > Jul 14 01:11:41 server01 kernel: [ 595.393903] [] warn_slowpath_null+0x1a/0x20 > > > Jul 14 01:11:41 server01 kernel: [ 595.394068] [] do_exit+0x7d0/0x870 > > > Jul 14 01:11:41 server01 kernel: [ 595.394231] [] ? thread_group_times+0x44/0xb0 > > > Jul 14 01:11:41 server01 kernel: [ 595.394392] [] do_group_exit+0x51/0xc0 > > > Jul 14 01:11:41 server01 kernel: [ 595.394551] [] sys_exit_group+0x17/0x20 > > > Jul 14 01:11:41 server01 kernel: [ 595.394714] [] system_call_fastpath+0x18/0x1d > > > Jul 14 01:11:41 server01 kernel: [ 595.394921] ---[ end trace 738570e688acf099 ]--- > > > > > > OK, so you had an OOM which has been handled by in-kernel oom handler > > > (it killed 12021) and 12037 was in the same group. The warning tells us > > > that it went through mem_cgroup_oom as well (otherwise it wouldn't have > > > memcg_oom.wait_on_memcg set and the warning wouldn't trigger) and then > > > it exited on the userspace request (by exit syscall). > > > > > > I do not see any way how, this could happen though. If mem_cgroup_oom > > > is called then we always return CHARGE_NOMEM which turns into ENOMEM > > > returned by __mem_cgroup_try_charge (invoke_oom must have been set to > > > true). So if nobody screwed the return value on the way up to page > > > fault handler then there is no way to escape. > > > > > > I will check the code. > > > > OK, I guess I found it: > > __do_fault > > fault = filemap_fault > > do_async_mmap_readahead > > page_cache_async_readahead > > ondemand_readahead > > __do_page_cache_readahead > > read_pages > > readpages = ext3_readpages > > mpage_readpages # Doesn't propagate ENOMEM > > add_to_page_cache_lru > > add_to_page_cache > > add_to_page_cache_locked > > mem_cgroup_cache_charge > > > > So the read ahead most probably. Again! Duhhh. I will try to think > > about a fix for this. One obvious place is mpage_readpages but > > __do_page_cache_readahead ignores read_pages return value as well and > > page_cache_async_readahead, even worse, is just void and exported as > > such. > > > > So this smells like a hard to fix bugger. One possible, and really ugly > > way would be calling mem_cgroup_oom_synchronize even if handle_mm_fault > > doesn't return VM_FAULT_ERROR, but that is a crude hack. > > Ouch, good spot. > > I don't think we need to handle an OOM from the readahead code. If > readahead does not produce the desired page, we retry synchroneously > in page_cache_read() and handle the OOM properly. We should not > signal an OOM for optional pages anyway. > > So either we pass a flag from the readahead code down to > add_to_page_cache and mem_cgroup_cache_charge that tells the charge > code to ignore OOM conditions and do not set up an OOM context. That was my previous attempt and it was sooo painful. > Or we DO call mem_cgroup_oom_synchronize() from the read_cache_pages, > with an argument that makes it only clean up the context and not wait. Yes, I was playing with this idea as well. I just do not like how fragile this is. We need some way to catch all possible places which might leak it. > It would not be completely outlandish to place it there, since it's > right next to where an error from add_to_page_cache() is not further > propagated back through the fault stack. > > I'm travelling right now, I'll send a patch when I get back > (Thursday). Unless you beat me to it :) I can cook something up but there is quite a big pile on my desk currently (as always :/). -- Michal Hocko SUSE Labs