From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753487Ab2GQGdK (ORCPT ); Tue, 17 Jul 2012 02:33:10 -0400 Received: from cantor2.suse.de ([195.135.220.15]:48381 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751721Ab2GQGdI (ORCPT ); Tue, 17 Jul 2012 02:33:08 -0400 Date: Tue, 17 Jul 2012 08:33:01 +0200 From: Michal Hocko To: Hugh Dickins Cc: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, KAMEZAWA Hiroyuki , Mel Gorman , Minchan Kim , Rik van Riel , Ying Han , Greg Thelen , Johannes Weiner , Fengguang Wu Subject: Re: [PATCH mmotm] memcg: further prevent OOM with too many dirty pages Message-ID: <20120717063301.GA25435@tiehlicka.suse.cz> References: <20120620101119.GC5541@tiehlicka.suse.cz> <20120712070501.GB21013@tiehlicka.suse.cz> <20120712141343.e1cb7776.akpm@linux-foundation.org> <20120713082150.GA1448@tiehlicka.suse.cz> <20120716092631.GC14664@tiehlicka.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon 16-07-12 21:52:51, Hugh Dickins wrote: > On Mon, 16 Jul 2012, Michal Hocko wrote: > > On Mon 16-07-12 01:35:34, Hugh Dickins wrote: > > > But even so, the test still OOMs sometimes: when originally testing > > > on 3.5-rc6, it OOMed about one time in five or ten; when testing > > > just now on 3.5-rc6-mm1, it OOMed on the first iteration. > > > > > > This residual problem comes from an accumulation of pages under > > > ordinary writeback, not marked PageReclaim, so rightly not causing > > > the memcg check to wait on their writeback: these too can prevent > > > shrink_page_list() from freeing any pages, so many times that memcg > > > reclaim fails and OOMs. > > > > I guess you managed to trigger this with 20M limit, right? > > That's right. > > > I have tested > > with different group sizes but the writeback didn't trigger for most of > > them and all the dirty data were flushed from the reclaim. > > I didn't examine writeback stats to confirm, but I guess that just > occasionally it managed to come in and do enough work to confound us. > > > Have you used any special setting the dirty ratio? > > No, I wasn't imaginative enough to try that. > > > Or was it with xfs (IIUC that one > > does ignore writeback from the direct reclaim completely). > > No, just ext4 at that point. > > I have since tested the final patch with ext4, ext3 (by ext3 driver > and by ext4 driver), ext2 (by ext2 driver and by ext4 driver), xfs, > btrfs, vfat, tmpfs (with swap on the USB stick) and block device: > about an hour on each, no surprises, all okay. > > But I didn't experiment beyond the 20M memcg. Great coverage anyway. Thanks a lot Hugh! > > Hugh -- Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx150.postini.com [74.125.245.150]) by kanga.kvack.org (Postfix) with SMTP id A28BE6B005A for ; Tue, 17 Jul 2012 02:33:08 -0400 (EDT) Date: Tue, 17 Jul 2012 08:33:01 +0200 From: Michal Hocko Subject: Re: [PATCH mmotm] memcg: further prevent OOM with too many dirty pages Message-ID: <20120717063301.GA25435@tiehlicka.suse.cz> References: <20120620101119.GC5541@tiehlicka.suse.cz> <20120712070501.GB21013@tiehlicka.suse.cz> <20120712141343.e1cb7776.akpm@linux-foundation.org> <20120713082150.GA1448@tiehlicka.suse.cz> <20120716092631.GC14664@tiehlicka.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Hugh Dickins Cc: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, KAMEZAWA Hiroyuki , Mel Gorman , Minchan Kim , Rik van Riel , Ying Han , Greg Thelen , Johannes Weiner , Fengguang Wu On Mon 16-07-12 21:52:51, Hugh Dickins wrote: > On Mon, 16 Jul 2012, Michal Hocko wrote: > > On Mon 16-07-12 01:35:34, Hugh Dickins wrote: > > > But even so, the test still OOMs sometimes: when originally testing > > > on 3.5-rc6, it OOMed about one time in five or ten; when testing > > > just now on 3.5-rc6-mm1, it OOMed on the first iteration. > > > > > > This residual problem comes from an accumulation of pages under > > > ordinary writeback, not marked PageReclaim, so rightly not causing > > > the memcg check to wait on their writeback: these too can prevent > > > shrink_page_list() from freeing any pages, so many times that memcg > > > reclaim fails and OOMs. > > > > I guess you managed to trigger this with 20M limit, right? > > That's right. > > > I have tested > > with different group sizes but the writeback didn't trigger for most of > > them and all the dirty data were flushed from the reclaim. > > I didn't examine writeback stats to confirm, but I guess that just > occasionally it managed to come in and do enough work to confound us. > > > Have you used any special setting the dirty ratio? > > No, I wasn't imaginative enough to try that. > > > Or was it with xfs (IIUC that one > > does ignore writeback from the direct reclaim completely). > > No, just ext4 at that point. > > I have since tested the final patch with ext4, ext3 (by ext3 driver > and by ext4 driver), ext2 (by ext2 driver and by ext4 driver), xfs, > btrfs, vfat, tmpfs (with swap on the USB stick) and block device: > about an hour on each, no surprises, all okay. > > But I didn't experiment beyond the 20M memcg. Great coverage anyway. Thanks a lot Hugh! > > Hugh -- Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org