From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.6 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB866C2BA19 for ; Thu, 23 Apr 2020 10:23:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8456A20704 for ; Thu, 23 Apr 2020 10:23:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WqjTKr4m" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8456A20704 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DBDD08E0005; Thu, 23 Apr 2020 06:23:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D1F778E0003; Thu, 23 Apr 2020 06:23:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE6558E0005; Thu, 23 Apr 2020 06:23:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0135.hostedemail.com [216.40.44.135]) by kanga.kvack.org (Postfix) with ESMTP id 9E9CF8E0003 for ; Thu, 23 Apr 2020 06:23:01 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 5A6775DE6 for ; Thu, 23 Apr 2020 10:23:01 +0000 (UTC) X-FDA: 76738731762.20.sky39_377091a220e32 X-HE-Tag: sky39_377091a220e32 X-Filterd-Recvd-Size: 7952 Received: from mail-il1-f194.google.com (mail-il1-f194.google.com [209.85.166.194]) by imf08.hostedemail.com (Postfix) with ESMTP for ; Thu, 23 Apr 2020 10:23:00 +0000 (UTC) Received: by mail-il1-f194.google.com with SMTP id x2so5001469ilp.13 for ; Thu, 23 Apr 2020 03:23:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=HNJVumb3clPUqpxVmQlr+qLDN0rRuMf02GGBdoAwDXE=; b=WqjTKr4mdUZLEdPruWLpLBC2Hgq9GRkMhv6V9ZNSqUoq5OGqnfFYEUKNEdR7y8Xmob WBrUoE2WaFHeU+ViBkKbRmC6JxPl18C6e2OpJ+FHicFX9glqeT45uuUxhvZulddUibRT DzOS2M6QTfbncg6uQIGj2+2+sXZpOn2bAy8k3xk4HJtKH+wVA87nSQJYWV7zRqDgKUO7 T3LlFpSQGKMqhRphbkIAEsPQfL8RJ553x1oGsoMjaUL819Koa/in77Od2xkdgse5V28y UTOqZKO95SJz1EjKFlet0CTpyWsZzgXzL71GCliZIU+KM5GCmk7xWFDGPf6iprxLlLq8 H7RA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=HNJVumb3clPUqpxVmQlr+qLDN0rRuMf02GGBdoAwDXE=; b=k1nLSKhc4Fgurpz3Lfm88NQkdnSXlA0b3kM7lerr3kn9YODixZQCSdNR6wEM8xh7Xx js7wX0HAyZLKvXoh+yvgVhWg0VGE6Lcv9l6D+h/EP7R7T0W+CJ/oEntMD80mC/hrhV// brCIRGkQoWRlvYrIMUb9QsiqnlQLPcymNcWMlZizfz7YaC2TJsX0LjAexcEkuHC7nfM/ ouvks7hPG0wc4A38EwLVS2pty3EHFDX+sMvliAi/c7zFDXRQvoQGplgqHJuwH9AB5ZvH KDipYFDMB4kjXBTzW+e1JppRgT0Vu6SDC01utU0Kt71RBWlYGaiL45oBGYNlcDTdEOI0 3FLQ== X-Gm-Message-State: AGi0PuZrzaHTLD7Gf48lFAK05emykNQkePJnzx/ujJUBzMirZY1Sq/K5 TF55TzkFL8OrwYN3eZvohvK8mDiUkN3mXj4RyNk= X-Google-Smtp-Source: APiQypLD6EbLpbIDc6NFboux3qKBFi9WwyX4y2Ae1Oph6z2XlVNXZv5xXovq+UDehP2Vv7OhNVP/TMuprYt6xUkD+rY= X-Received: by 2002:a92:9e11:: with SMTP id q17mr2706320ili.137.1587637380246; Thu, 23 Apr 2020 03:23:00 -0700 (PDT) MIME-Version: 1.0 References: <20200131043324.wkArXTf6-%akpm@linux-foundation.org> <20200417152637.GQ26707@dhcp22.suse.cz> <68371d79-dfae-e9b9-38df-dbb916607a82@i-love.sakura.ne.jp> <20200420073305.GD27314@dhcp22.suse.cz> <041d4158-f3dc-a5b5-546e-dd3f71f67b5f@i-love.sakura.ne.jp> <20200422065950.GA30312@dhcp22.suse.cz> <0aa5e490-c5b8-dc5c-334e-9a8d37da215c@i-love.sakura.ne.jp> <20200423073438.GC4206@dhcp22.suse.cz> In-Reply-To: <20200423073438.GC4206@dhcp22.suse.cz> From: Yafang Shao Date: Thu, 23 Apr 2020 18:22:22 +0800 Message-ID: Subject: Re: [nacked] mm-oom-avoid-printk-iteration-under-rcu.patch removed from -mm tree To: Michal Hocko Cc: Tetsuo Handa , Andrew Morton , Roman Gushchin , linux-mm , David Rientjes , Shakeel Butt Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Apr 23, 2020 at 3:34 PM Michal Hocko wrote: > > On Thu 23-04-20 14:35:22, Yafang Shao wrote: > > On Thu, Apr 23, 2020 at 1:35 PM Tetsuo Handa > [...] > > > dump_tasks() remains definitely a printk() abuser which is capable of= pushing > > > many thousands of printk() messages in one second if async printk wer= e available. > > > Async printk CANNOT deal with the problem that too much backlog cause= s important > > > messages to be delayed for too long. Please read my explanation caref= ully. > > > > > > > Agreed. Too much oom reports still be a issue even if the printk() is a= syn. > > I believe nobody is disputing this part. We are talking about two things > here and I believe that contributes to a confusion considerably > 1) dump_tasks being a large noise generator to the kernel log buffer > 2) a heavy printk load from the oom context > > There is no good answer for 1). We simply print a lot of data that > scales with the number eligible tasks and that might be thousands. > We have done quite a lot of work to make the data collecting part of the > process as optimal as possible but having this feature enabled by > default is simply a package we have to carry with us. printk doesn't > cope with such a load really great currently. There might be some future > changes but the underline is that no matter how printk gets optimized > there is still the payload to be printed. No matter this happens > transparently async or explicitly done in a detached context. > > 2) is about the sync nature of the printk _right_now_ and that causes > delays in the allocator context while the system is OOM. There are locks > held both by the OOM context and in the call chain to the allocator > potentially. The longer the oom context is going to take the longer is > the agony going to take. Here is where the async printing might help > because it would push out the heavy lifting to a different context. > > There is a clear agreement in this part. The whole discussion in this > thread is about how to achieve that. There are two ways. Develop a code > to do that for this very specific case (aka push out to a worker) or > rely on printk doing that for us and potentially many other places in a > similar situation. I am definitely for the later option because a) it > adds less code we have to maintain and b) it is a more generic solution. > > For the current or older kernels there are two ways to workaround for > the problem and floods of oom killer events doesn't seem to be be a > regular production system state (I would even dare to claim that > something is terribly wrong if yes) so no quick&dirty hacks are due. > Either tune the log level or simply disable dump_tasks. It is an useful > tool in some cases but not really necessary in the vast majority of > cases. > Thanks for your explanation. We have an agreement here. > > I think the aysnc printk() won't care about wheter the data is > > improtant or not, so the user of printk() (even if it is asyn) should > > have a good management of these data especially if these data may > > consume all or most of the printk buffer. > > Not sure what you mean here. We do have an option to tune the ring > buffer (both size and log levels) and dump_tasks specifically. > There're drawbacks in both of these two options. printk() is multiple-producer and mutiple-consumer. OOM is one of the producers. logfile (/var/log/messages) and console are two of the consumers. Now let's see the drawback of each option. - tune the ring buffer (both size and log levels) All the producers are effected. For example, if you tune the log levels, then all producers have the same loglevel with dump_stack() can't show in the console. Tuning the size may be not scalable, because we don't know how slow the console is and tuning it too big is a waste of memory. - tune the dump_tasks specifically =EF=BC=88vm.oom_dump_tasks=EF=BC=89 All the consumers are effected. The logfile is fast enough, so we expect that these dump_tasks could be printed into the logfile. The console is so slow that we don't want to print into it. A possilbe way to fix it is improve vm.oom_dump_tasks. vm.oom_dump_tasks : 1 - dump into all consumers 2 - don't dump into console 0 - don't dump into any of the consumers But someone may still needs these dump_tasks from the console. As I'm not familiar with asyn printk(), my understanding may be not correct= . --=20 Thanks Yafang