Re: [PATCH 00/50] Add log level to show_stack()

From: Dmitry Safonov <dima@arista.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>,
	linux-sh@vger.kernel.org,
	Catalin Marinas <catalin.marinas@arm.com>,
	Ben Segall <bsegall@google.com>, Guo Ren <guoren@kernel.org>,
	Pavel Machek <pavel@ucw.cz>,
	Vincent Guittot <vincent.guittot@linaro.org>,
	Paul Burton <paulburton@kernel.org>,
	Michael Ellerman <mpe@ellerman.id.au>,
	Geert Uytterhoeven <geert@linux-m68k.org>,
	Mel Gorman <mgorman@suse.de>, Jiri Slaby <jslaby@suse.com>,
	Matt Turner <mattst88@gmail.com>,
	uclinux-h8-devel@lists.sourceforge.jp,
	Len Brown <len.brown@intel.com>,
	linux-pm@vger.kernel.org,
	Heiko Carstens <heiko.carstens@de.ibm.com>,
	linux-um@lists.infradead.org,
	Thomas Gleixner <tglx@linutronix.de>,
	Dietmar Eggemann <dietmar.eggemann@arm.com>,
	Richard Henderson <rth@twiddle.net>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	"Rafael J. Wysocki" <rjw@rjwysocki.net>,
	LKML <linux-kernel@vger.kernel.org>,
	Ralf Baechle <ralf@linux-mips.org>,
	Paul Mackerras <paulus@samba.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	linux-ia64@vger.kernel.org,
	Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>,
	James Hogan <jhogan@kernel.org>,
	"James E.J. Bottomley" <James.Bottomley@hansenpartnership.com>,
	Max Filippov <jcmvbkbc@gmail.com>,
	Vincent Chen <deanbo422@gmail.com>,
	Ingo Molnar <mingo@kernel.org>,
	linux-s390@vger.kernel.org, linux-c6x-dev@linux-c6x.org,
	Yoshinori Sato <ysato@users.sourceforge.jp>,
	linux-hexagon@vger.kernel.org, Helge Deller <deller@gmx.de>,
	"open list:TENSILICA XTENSA PORT \(xtensa\)"
	<linux-xtensa@linux-xtensa.org>,
	Vasily Gorbik <gor@linux.ibm.com>,
	Aurelien Jacquiot <jacquiot.aurelien@gmail.com>,
	linux-m68k@lists.linux-m68k.org,
	Stafford Horne <shorne@gmail.com>,
	linux-arm-kernel@lists.infradead.org,
	Chris Zankel <chris@zankel.net>, Tony Luck <tony.luck@intel.com>,
	Douglas Anderson <dianders@chromium.org>,
	Benjamin Herrenschmidt <benh@kernel.crashing.org>,
	Dmitry Safonov <0x7f454c46@gmail.com>,
	Will Deacon <will@kernel.org>,
	Daniel Thompson <daniel.thompson@linaro.org>,
	Brian Cain <bcain@codeaurora.org>,
	Christian Borntraeger <borntraeger@de.ibm.com>,
	kgdb-bugreport@lists.sourceforge.net,
	linux-snps-arc@lists.infradead.org,
	Fenghua Yu <fenghua.yu@intel.com>, Borislav Petkov <bp@alien8.de>,
	Jeff Dike <jdike@addtoit.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Ivan Kokshaysky <ink@jurassic.park.msu.ru>,
	Greentime Hu <green.hu@gmail.com>, Guan Xuetao <gxt@pku.edu.cn>,
	linux-parisc@vger.kernel.org, linux-alpha@vger.kernel.org,
	Ley Foon Tan <lftan@altera.com>,
	"David S. Miller" <davem@davemloft.net>,
	Rich Felker <dalias@libc.org>, Petr Mladek <pmladek@suse.com>,
	"H. Peter Anvin" <hpa@zytor.com>,
	sparclinux@vger.kernel.org, linux-riscv@lists.infradead.org,
	Anton Ivanov <anton.ivanov@cambridgegreys.com>,
	Jonas Bonn <jonas@southpole.se>,
	Richard Weinberger <richard@nod.at>, X86 ML <x86@kernel.org>,
	Russell King <linux@armlinux.org.uk>,
	clang-built-linux@googlegroups.com,
	Ingo Molnar <mingo@redhat.com>, Mark Salter <msalter@redhat.com>,
	Albert Ou <aou@eecs.berkeley.edu>,
	Stefan Kristiansson <stefan.kristiansson@saunalahti.fi>,
	openrisc@lists.librecores.org,
	Paul Walmsley <paul.walmsley@sifive.com>,
	Michal Simek <monstr@monstr.eu>,
	Vineet Gupta <vgupta@synopsys.com>,
	linux-mips@vger.kernel.org,
	Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
	Palmer Dabbelt <palmer@dabbelt.com>,
	Jason Wessel <jason.wessel@windriver.com>,
	nios2-dev@lists.rocketboards.org, linuxppc-dev@lists.ozlabs.org
Subject: Re: [PATCH 00/50] Add log level to show_stack()
Date: Fri, 8 Nov 2019 16:28:30 +0000	[thread overview]
Message-ID: <CAGrbwDRgX1BZoFrVYSXhAeeUKHrB=q3w8jXFPOuRvO7HV4Ciqw@mail.gmail.com> (raw)
In-Reply-To: <20191106203440.GH3079@worktop.programming.kicks-ass.net>

On 11/6/19 8:34 PM, Peter Zijlstra wrote:
> On Wed, Nov 06, 2019 at 04:27:33PM +0000, Dmitry Safonov wrote:
[..]
>> Sorry, I should have tried to describe better.
>>
>> I'm trying to remove external users of console_loglevel by following
>> reasons:
>
> I suppose since all my machines have 'debug ignore_loglevel
> earlyprintk=serial,ttyS0,115200 console=ttyS0,115200' I don't have this
> experience.

Yeah, I remember you avoid all those functionalities of printk(), fair
enough. On the other side, regular users and I'm betting most of
the non-tuned distributions use /proc/sys/kernel/printk by default.
(Checking on my Arch & Fedora - loglevel 4 from the box)

>> - changing console_loglevel on SMP means that unwanted messages from
>> other CPUs will appear (that have lower log level)
>> - on UMP unwanted messages may appear if the code is preempted while it
>> hasn't set the console_loglevel back to old
>> - rising console_loglevel to print wanted message(s) may not work at all
>> if printk() has being delayed and the console_loglevel is already set
>> back to old value
>
> Sure, frobbing the global console_loglevel is bad.
>
>> I also have patches in wip those needs to print backtrace with specific
>> loglevel (higher when it's critical, lower when it's notice and
>> shouldn't go to serial console).
>
> (everything always should go to serial, serial is awesome :-)

Personally I agree. Unfortunately, here @Arista there are switches (I'm
speaking about the order of thousands at least) those have baud-rate 9600.
It's a bit expensive being elaborate with such setup.

>> Besides on local tests I see hits those have headers (messages like
>> "Backtrace: ") without an actual backtrace and the reverse - a backtrace
>> without a reason for it. It's quite annoying and worth addressing by
>> syncing headers log levels to backtraces.
>
> I suppose I'm surprised there are backtraces that are not important.
> Either badness happened and it needs printing, or the user asked for it
> and it needs printing.
>
> Perhaps we should be removing backtraces if they're not important
> instead of allowing to print them as lower loglevels?

Well, the use-case for lower log-level is that everything goes into logs
(/var/log/dmesg or /var/log/messages whatever rsyslog has settting).

That has it's value:
- after a failure (i.e. panic) messages, those were only signs that
something goes wrong can be seen in logs which can give ideas what has
happened.
- before the failure, those messages go to telemetry and can be
auto-magically processed remotely to take a decision (e.g. balance the
traffic away).

I wish all the information could be gathered in the userspace, but many
existing kernel/driver messages are valuable. A more detailed example is
hung task detector: we want to have the backtrace for a task that is
in uninterruptible state too long, but only in logs as printing on
serial console may lead to holding console lock and watchdog.
Besides customer notifications and investigations, I see the value of such
"bottleneck" warnings during long-running integration tests.

Thanks,
          Dmitry

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv