From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754392AbeATMUF (ORCPT ); Sat, 20 Jan 2018 07:20:05 -0500 Received: from mail-qt0-f169.google.com ([209.85.216.169]:44211 "EHLO mail-qt0-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751146AbeATMT5 (ORCPT ); Sat, 20 Jan 2018 07:19:57 -0500 X-Google-Smtp-Source: AH8x225rydAPhsv9dedbDrU8Dh2Ljv7TCYZQU9xNRkO9arC8K7lBgU13VilPAzSJE5g0wB2GcuM22Q== Date: Sat, 20 Jan 2018 04:19:53 -0800 From: Tejun Heo To: Steven Rostedt Cc: Petr Mladek , Sergey Senozhatsky , Sergey Senozhatsky , akpm@linux-foundation.org, linux-mm@kvack.org, Cong Wang , Dave Hansen , Johannes Weiner , Mel Gorman , Michal Hocko , Vlastimil Babka , Peter Zijlstra , Linus Torvalds , Jan Kara , Mathieu Desnoyers , Tetsuo Handa , rostedt@rostedt.homelinux.com, Byungchul Park , Pavel Machek , linux-kernel@vger.kernel.org Subject: Re: [PATCH v5 0/2] printk: Console owner and waiter logic cleanup Message-ID: <20180120121953.GA1096857@devbig577.frc2.facebook.com> References: <20180111103845.GB477@jagdpanzerIV> <20180111112908.50de440a@vmware.local.home> <20180111203057.5b1a8f8f@gandalf.local.home> <20180111215547.2f66a23a@gandalf.local.home> <20180116194456.GS3460072@devbig577.frc2.facebook.com> <20180117091208.ezvuhumnsarz5thh@pathway.suse.cz> <20180117151509.GT3460072@devbig577.frc2.facebook.com> <20180117121251.7283a56e@gandalf.local.home> <20180117134201.0a9cbbbf@gandalf.local.home> <20180119132052.02b89626@gandalf.local.home> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180119132052.02b89626@gandalf.local.home> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hello, Steven. On Fri, Jan 19, 2018 at 01:20:52PM -0500, Steven Rostedt wrote: > I was thinking about this a bit more, and instead of offloading a > recursive printk, perhaps its best to simply throttle it. Because the > problem may not go away if a printk thread takes over, because the bug > is really the printk infrastructure filling the printk buffer keeping > printk from ever stopping. > > This patch detects that printk is causing itself to print more and > throttles it after 3 messages have printed due to recursion. Could you > see if this helps your test cases? Sure, if this is the approach we're gonna take, I can try it with the silly test code and also try to reproduce the original problem and see whether this helps. I'm a bit worried tho because this essentially seems like "detect recursion, ignore messages" approach. netcons can have a very large surface for bugs. Suppressing those messages would make them difficult to debug. For example, all our machines have both serial console (thus the slowness) and netconsole hooked up and netcons code has had its fair share of issues. This would likely make tracking down those problems more challenging. Can we discuss pros and cons of this approach against offloading before committing to this? Thanks. -- tejun From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt0-f200.google.com (mail-qt0-f200.google.com [209.85.216.200]) by kanga.kvack.org (Postfix) with ESMTP id D68546B0033 for ; Sat, 20 Jan 2018 07:19:57 -0500 (EST) Received: by mail-qt0-f200.google.com with SMTP id c20so6808872qtn.22 for ; Sat, 20 Jan 2018 04:19:57 -0800 (PST) Received: from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41]) by mx.google.com with SMTPS id q4sor8652961qtl.154.2018.01.20.04.19.56 for (Google Transport Security); Sat, 20 Jan 2018 04:19:57 -0800 (PST) Date: Sat, 20 Jan 2018 04:19:53 -0800 From: Tejun Heo Subject: Re: [PATCH v5 0/2] printk: Console owner and waiter logic cleanup Message-ID: <20180120121953.GA1096857@devbig577.frc2.facebook.com> References: <20180111103845.GB477@jagdpanzerIV> <20180111112908.50de440a@vmware.local.home> <20180111203057.5b1a8f8f@gandalf.local.home> <20180111215547.2f66a23a@gandalf.local.home> <20180116194456.GS3460072@devbig577.frc2.facebook.com> <20180117091208.ezvuhumnsarz5thh@pathway.suse.cz> <20180117151509.GT3460072@devbig577.frc2.facebook.com> <20180117121251.7283a56e@gandalf.local.home> <20180117134201.0a9cbbbf@gandalf.local.home> <20180119132052.02b89626@gandalf.local.home> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180119132052.02b89626@gandalf.local.home> Sender: owner-linux-mm@kvack.org List-ID: To: Steven Rostedt Cc: Petr Mladek , Sergey Senozhatsky , Sergey Senozhatsky , akpm@linux-foundation.org, linux-mm@kvack.org, Cong Wang , Dave Hansen , Johannes Weiner , Mel Gorman , Michal Hocko , Vlastimil Babka , Peter Zijlstra , Linus Torvalds , Jan Kara , Mathieu Desnoyers , Tetsuo Handa , rostedt@home.goodmis.org, Byungchul Park , Pavel Machek , linux-kernel@vger.kernel.org Hello, Steven. On Fri, Jan 19, 2018 at 01:20:52PM -0500, Steven Rostedt wrote: > I was thinking about this a bit more, and instead of offloading a > recursive printk, perhaps its best to simply throttle it. Because the > problem may not go away if a printk thread takes over, because the bug > is really the printk infrastructure filling the printk buffer keeping > printk from ever stopping. > > This patch detects that printk is causing itself to print more and > throttles it after 3 messages have printed due to recursion. Could you > see if this helps your test cases? Sure, if this is the approach we're gonna take, I can try it with the silly test code and also try to reproduce the original problem and see whether this helps. I'm a bit worried tho because this essentially seems like "detect recursion, ignore messages" approach. netcons can have a very large surface for bugs. Suppressing those messages would make them difficult to debug. For example, all our machines have both serial console (thus the slowness) and netconsole hooked up and netcons code has had its fair share of issues. This would likely make tracking down those problems more challenging. Can we discuss pros and cons of this approach against offloading before committing to this? Thanks. -- tejun -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org