From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932948AbcLSXUa (ORCPT ); Mon, 19 Dec 2016 18:20:30 -0500 Received: from mail.linuxfoundation.org ([140.211.169.12]:48608 "EHLO mail.linuxfoundation.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932585AbcLSXU0 (ORCPT ); Mon, 19 Dec 2016 18:20:26 -0500 Date: Mon, 19 Dec 2016 15:21:25 -0800 From: Andrew Morton To: Michal Hocko Cc: Tetsuo Handa , , LKML , Michal Hocko Subject: Re: [PATCH] mm: throttle show_mem from warn_alloc Message-Id: <20161219152125.f77ddf79f3c89e5cdd0e02d6@linux-foundation.org> In-Reply-To: <20161215101510.9030-1-mhocko@kernel.org> References: <20161215101510.9030-1-mhocko@kernel.org> X-Mailer: Sylpheed 3.4.1 (GTK+ 2.24.23; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 15 Dec 2016 11:15:10 +0100 Michal Hocko wrote: > Tetsuo has been stressing OOM killer path with many parallel allocation > requests when he has noticed that it is not all that hard to swamp > kernel logs with warn_alloc messages caused by allocation stalls. Even > though the allocation stall message is triggered only once in 10s there > might be many different tasks hitting it roughly around the same time. > > A big part of the output is show_mem() which can generate a lot of > output even on a small machines. There is no reason to show the state of > memory counter for each allocation stall, especially when multiple of > them are reported in a short time period. Chances are that not much has > changed since the last report. This patch simply rate limits show_mem > called from warn_alloc to only dump something once per second. This > should be enough to give us a clue why an allocation might be stalling > while burst of warnings will not swamp log with too much data. > > While we are at it, extract all the show_mem related handling (filters) > into a separate function warn_alloc_show_mem. This will make the code > cleaner and as a bonus point we can distinguish which part of warn_alloc > got throttled due to rate limiting as ___ratelimit dumps the caller. These guys don't need file-wide scope... --- a/mm/page_alloc.c~mm-throttle-show_mem-from-warn_alloc-fix +++ a/mm/page_alloc.c @@ -3018,15 +3018,10 @@ static inline bool should_suppress_show_ return ret; } -static DEFINE_RATELIMIT_STATE(nopage_rs, - DEFAULT_RATELIMIT_INTERVAL, - DEFAULT_RATELIMIT_BURST); - -static DEFINE_RATELIMIT_STATE(show_mem_rs, HZ, 1); - static void warn_alloc_show_mem(gfp_t gfp_mask) { unsigned int filter = SHOW_MEM_FILTER_NODES; + static DEFINE_RATELIMIT_STATE(show_mem_rs, HZ, 1); if (should_suppress_show_mem() || !__ratelimit(&show_mem_rs)) return; @@ -3050,6 +3045,8 @@ void warn_alloc(gfp_t gfp_mask, const ch { struct va_format vaf; va_list args; + static DEFINE_RATELIMIT_STATE(nopage_rs, DEFAULT_RATELIMIT_INTERVAL, + DEFAULT_RATELIMIT_BURST); if ((gfp_mask & __GFP_NOWARN) || !__ratelimit(&nopage_rs) || debug_guardpage_minorder() > 0) _ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-it0-f72.google.com (mail-it0-f72.google.com [209.85.214.72]) by kanga.kvack.org (Postfix) with ESMTP id CC3046B02CD for ; Mon, 19 Dec 2016 18:20:26 -0500 (EST) Received: by mail-it0-f72.google.com with SMTP id 75so96423188ite.7 for ; Mon, 19 Dec 2016 15:20:26 -0800 (PST) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org. [140.211.169.12]) by mx.google.com with ESMTPS id a125si14620123iog.216.2016.12.19.15.20.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Dec 2016 15:20:26 -0800 (PST) Date: Mon, 19 Dec 2016 15:21:25 -0800 From: Andrew Morton Subject: Re: [PATCH] mm: throttle show_mem from warn_alloc Message-Id: <20161219152125.f77ddf79f3c89e5cdd0e02d6@linux-foundation.org> In-Reply-To: <20161215101510.9030-1-mhocko@kernel.org> References: <20161215101510.9030-1-mhocko@kernel.org> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko Cc: Tetsuo Handa , linux-mm@kvack.org, LKML , Michal Hocko On Thu, 15 Dec 2016 11:15:10 +0100 Michal Hocko wrote: > Tetsuo has been stressing OOM killer path with many parallel allocation > requests when he has noticed that it is not all that hard to swamp > kernel logs with warn_alloc messages caused by allocation stalls. Even > though the allocation stall message is triggered only once in 10s there > might be many different tasks hitting it roughly around the same time. > > A big part of the output is show_mem() which can generate a lot of > output even on a small machines. There is no reason to show the state of > memory counter for each allocation stall, especially when multiple of > them are reported in a short time period. Chances are that not much has > changed since the last report. This patch simply rate limits show_mem > called from warn_alloc to only dump something once per second. This > should be enough to give us a clue why an allocation might be stalling > while burst of warnings will not swamp log with too much data. > > While we are at it, extract all the show_mem related handling (filters) > into a separate function warn_alloc_show_mem. This will make the code > cleaner and as a bonus point we can distinguish which part of warn_alloc > got throttled due to rate limiting as ___ratelimit dumps the caller. These guys don't need file-wide scope... --- a/mm/page_alloc.c~mm-throttle-show_mem-from-warn_alloc-fix +++ a/mm/page_alloc.c @@ -3018,15 +3018,10 @@ static inline bool should_suppress_show_ return ret; } -static DEFINE_RATELIMIT_STATE(nopage_rs, - DEFAULT_RATELIMIT_INTERVAL, - DEFAULT_RATELIMIT_BURST); - -static DEFINE_RATELIMIT_STATE(show_mem_rs, HZ, 1); - static void warn_alloc_show_mem(gfp_t gfp_mask) { unsigned int filter = SHOW_MEM_FILTER_NODES; + static DEFINE_RATELIMIT_STATE(show_mem_rs, HZ, 1); if (should_suppress_show_mem() || !__ratelimit(&show_mem_rs)) return; @@ -3050,6 +3045,8 @@ void warn_alloc(gfp_t gfp_mask, const ch { struct va_format vaf; va_list args; + static DEFINE_RATELIMIT_STATE(nopage_rs, DEFAULT_RATELIMIT_INTERVAL, + DEFAULT_RATELIMIT_BURST); if ((gfp_mask & __GFP_NOWARN) || !__ratelimit(&nopage_rs) || debug_guardpage_minorder() > 0) _ -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org