From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22138C55193 for ; Thu, 23 Apr 2020 07:34:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C46BB208E4 for ; Thu, 23 Apr 2020 07:34:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C46BB208E4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 253548E0007; Thu, 23 Apr 2020 03:34:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1DCE88E0003; Thu, 23 Apr 2020 03:34:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 07F3B8E0007; Thu, 23 Apr 2020 03:34:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0248.hostedemail.com [216.40.44.248]) by kanga.kvack.org (Postfix) with ESMTP id DE9F18E0003 for ; Thu, 23 Apr 2020 03:34:42 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 9C7C76D73 for ; Thu, 23 Apr 2020 07:34:42 +0000 (UTC) X-FDA: 76738307604.15.boat49_2911c7c1b8a62 X-HE-Tag: boat49_2911c7c1b8a62 X-Filterd-Recvd-Size: 4678 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Thu, 23 Apr 2020 07:34:42 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id BCB2FAF19; Thu, 23 Apr 2020 07:34:39 +0000 (UTC) Date: Thu, 23 Apr 2020 09:34:38 +0200 From: Michal Hocko To: Yafang Shao Cc: Tetsuo Handa , Andrew Morton , Roman Gushchin , linux-mm , David Rientjes , Shakeel Butt Subject: Re: [nacked] mm-oom-avoid-printk-iteration-under-rcu.patch removed from -mm tree Message-ID: <20200423073438.GC4206@dhcp22.suse.cz> References: <20200131043324.wkArXTf6-%akpm@linux-foundation.org> <20200417152637.GQ26707@dhcp22.suse.cz> <68371d79-dfae-e9b9-38df-dbb916607a82@i-love.sakura.ne.jp> <20200420073305.GD27314@dhcp22.suse.cz> <041d4158-f3dc-a5b5-546e-dd3f71f67b5f@i-love.sakura.ne.jp> <20200422065950.GA30312@dhcp22.suse.cz> <0aa5e490-c5b8-dc5c-334e-9a8d37da215c@i-love.sakura.ne.jp> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu 23-04-20 14:35:22, Yafang Shao wrote: > On Thu, Apr 23, 2020 at 1:35 PM Tetsuo Handa [...] > > dump_tasks() remains definitely a printk() abuser which is capable of pushing > > many thousands of printk() messages in one second if async printk were available. > > Async printk CANNOT deal with the problem that too much backlog causes important > > messages to be delayed for too long. Please read my explanation carefully. > > > > Agreed. Too much oom reports still be a issue even if the printk() is asyn. I believe nobody is disputing this part. We are talking about two things here and I believe that contributes to a confusion considerably 1) dump_tasks being a large noise generator to the kernel log buffer 2) a heavy printk load from the oom context There is no good answer for 1). We simply print a lot of data that scales with the number eligible tasks and that might be thousands. We have done quite a lot of work to make the data collecting part of the process as optimal as possible but having this feature enabled by default is simply a package we have to carry with us. printk doesn't cope with such a load really great currently. There might be some future changes but the underline is that no matter how printk gets optimized there is still the payload to be printed. No matter this happens transparently async or explicitly done in a detached context. 2) is about the sync nature of the printk _right_now_ and that causes delays in the allocator context while the system is OOM. There are locks held both by the OOM context and in the call chain to the allocator potentially. The longer the oom context is going to take the longer is the agony going to take. Here is where the async printing might help because it would push out the heavy lifting to a different context. There is a clear agreement in this part. The whole discussion in this thread is about how to achieve that. There are two ways. Develop a code to do that for this very specific case (aka push out to a worker) or rely on printk doing that for us and potentially many other places in a similar situation. I am definitely for the later option because a) it adds less code we have to maintain and b) it is a more generic solution. For the current or older kernels there are two ways to workaround for the problem and floods of oom killer events doesn't seem to be be a regular production system state (I would even dare to claim that something is terribly wrong if yes) so no quick&dirty hacks are due. Either tune the log level or simply disable dump_tasks. It is an useful tool in some cases but not really necessary in the vast majority of cases. > I think the aysnc printk() won't care about wheter the data is > improtant or not, so the user of printk() (even if it is asyn) should > have a good management of these data especially if these data may > consume all or most of the printk buffer. Not sure what you mean here. We do have an option to tune the ring buffer (both size and log levels) and dump_tasks specifically. -- Michal Hocko SUSE Labs