LinuxPPC-Dev Archive on lore.kernel.org
 help / color / Atom feed
From: Dmitry Safonov <dima@arista.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>,
	linux-sh@vger.kernel.org,
	Catalin Marinas <catalin.marinas@arm.com>,
	Ben Segall <bsegall@google.com>, Guo Ren <guoren@kernel.org>,
	Pavel Machek <pavel@ucw.cz>,
	Vincent Guittot <vincent.guittot@linaro.org>,
	Paul Burton <paulburton@kernel.org>,
	Geert Uytterhoeven <geert@linux-m68k.org>,
	Mel Gorman <mgorman@suse.de>, Jiri Slaby <jslaby@suse.com>,
	Matt Turner <mattst88@gmail.com>,
	uclinux-h8-devel@lists.sourceforge.jp,
	Len Brown <len.brown@intel.com>,
	linux-pm@vger.kernel.org,
	Heiko Carstens <heiko.carstens@de.ibm.com>,
	linux-um@lists.infradead.org,
	Thomas Gleixner <tglx@linutronix.de>,
	Dietmar Eggemann <dietmar.eggemann@arm.com>,
	Richard Henderson <rth@twiddle.net>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	"Rafael J. Wysocki" <rjw@rjwysocki.net>,
	LKML <linux-kernel@vger.kernel.org>,
	Ralf Baechle <ralf@linux-mips.org>,
	Paul Mackerras <paulus@samba.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	linux-ia64@vger.kernel.org,
	Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>,
	James Hogan <jhogan@kernel.org>,
	"James E.J. Bottomley" <James.Bottomley@hansenpartnership.com>,
	Max Filippov <jcmvbkbc@gmail.com>,
	Vincent Chen <deanbo422@gmail.com>,
	Ingo Molnar <mingo@kernel.org>,
	linux-s390@vger.kernel.org, linux-c6x-dev@linux-c6x.org,
	Yoshinori Sato <ysato@users.sourceforge.jp>,
	linux-hexagon@vger.kernel.org, Helge Deller <deller@gmx.de>,
	"open list:TENSILICA XTENSA PORT \(xtensa\)"
	<linux-xtensa@linux-xtensa.org>,
	Vasily Gorbik <gor@linux.ibm.com>,
	Aurelien Jacquiot <jacquiot.aurelien@gmail.com>,
	linux-m68k@lists.linux-m68k.org,
	Stafford Horne <shorne@gmail.com>,
	linux-arm-kernel@lists.infradead.org,
	Chris Zankel <chris@zankel.net>, Tony Luck <tony.luck@intel.com>,
	Douglas Anderson <dianders@chromium.org>,
	Dmitry Safonov <0x7f454c46@gmail.com>,
	Will Deacon <will@kernel.org>,
	Daniel Thompson <daniel.thompson@linaro.org>,
	Brian Cain <bcain@codeaurora.org>,
	Christian Borntraeger <borntraeger@de.ibm.com>,
	kgdb-bugreport@lists.sourceforge.net,
	linux-snps-arc@lists.infradead.org,
	Fenghua Yu <fenghua.yu@intel.com>, Borislav Petkov <bp@alien8.de>,
	Jeff Dike <jdike@addtoit.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Ivan Kokshaysky <ink@jurassic.park.msu.ru>,
	Greentime Hu <green.hu@gmail.com>, Guan Xuetao <gxt@pku.edu.cn>,
	linux-parisc@vger.kernel.org, linux-alpha@vger.kernel.org,
	Ley Foon Tan <lftan@altera.com>,
	"David S. Miller" <davem@davemloft.net>,
	Rich Felker <dalias@libc.org>, Petr Mladek <pmladek@suse.com>,
	"H. Peter Anvin" <hpa@zytor.com>,
	sparclinux@vger.kernel.org, linux-riscv@lists.infradead.org,
	Anton Ivanov <anton.ivanov@cambridgegreys.com>,
	Jonas Bonn <jonas@southpole.se>,
	Richard Weinberger <richard@nod.at>, X86 ML <x86@kernel.org>,
	Russell King <linux@armlinux.org.uk>,
	clang-built-linux@googlegroups.com,
	Ingo Molnar <mingo@redhat.com>, Mark Salter <msalter@redhat.com>,
	Albert Ou <aou@eecs.berkeley.edu>,
	Stefan Kristiansson <stefan.kristiansson@saunalahti.fi>,
	openrisc@lists.librecores.org,
	Paul Walmsley <paul.walmsley@sifive.com>,
	Michal Simek <monstr@monstr.eu>,
	Vineet Gupta <vgupta@synopsys.com>,
	linux-mips@vger.kernel.org,
	Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
	Palmer Dabbelt <palmer@dabbelt.com>,
	Jason Wessel <jason.wessel@windriver.com>,
	nios2-dev@lists.rocketboards.org, linuxppc-dev@lists.ozlabs.org
Subject: Re: [PATCH 00/50] Add log level to show_stack()
Date: Fri, 8 Nov 2019 16:28:30 +0000
Message-ID: <CAGrbwDRgX1BZoFrVYSXhAeeUKHrB=q3w8jXFPOuRvO7HV4Ciqw@mail.gmail.com> (raw)
In-Reply-To: <20191106203440.GH3079@worktop.programming.kicks-ass.net>

On 11/6/19 8:34 PM, Peter Zijlstra wrote:
> On Wed, Nov 06, 2019 at 04:27:33PM +0000, Dmitry Safonov wrote:
[..]
>> Sorry, I should have tried to describe better.
>>
>> I'm trying to remove external users of console_loglevel by following
>> reasons:
>
> I suppose since all my machines have 'debug ignore_loglevel
> earlyprintk=serial,ttyS0,115200 console=ttyS0,115200' I don't have this
> experience.

Yeah, I remember you avoid all those functionalities of printk(), fair
enough. On the other side, regular users and I'm betting most of
the non-tuned distributions use /proc/sys/kernel/printk by default.
(Checking on my Arch & Fedora - loglevel 4 from the box)

>> - changing console_loglevel on SMP means that unwanted messages from
>> other CPUs will appear (that have lower log level)
>> - on UMP unwanted messages may appear if the code is preempted while it
>> hasn't set the console_loglevel back to old
>> - rising console_loglevel to print wanted message(s) may not work at all
>> if printk() has being delayed and the console_loglevel is already set
>> back to old value
>
> Sure, frobbing the global console_loglevel is bad.
>
>> I also have patches in wip those needs to print backtrace with specific
>> loglevel (higher when it's critical, lower when it's notice and
>> shouldn't go to serial console).
>
> (everything always should go to serial, serial is awesome :-)

Personally I agree. Unfortunately, here @Arista there are switches (I'm
speaking about the order of thousands at least) those have baud-rate 9600.
It's a bit expensive being elaborate with such setup.

>> Besides on local tests I see hits those have headers (messages like
>> "Backtrace: ") without an actual backtrace and the reverse - a backtrace
>> without a reason for it. It's quite annoying and worth addressing by
>> syncing headers log levels to backtraces.
>
> I suppose I'm surprised there are backtraces that are not important.
> Either badness happened and it needs printing, or the user asked for it
> and it needs printing.
>
> Perhaps we should be removing backtraces if they're not important
> instead of allowing to print them as lower loglevels?

Well, the use-case for lower log-level is that everything goes into logs
(/var/log/dmesg or /var/log/messages whatever rsyslog has settting).

That has it's value:
- after a failure (i.e. panic) messages, those were only signs that
something goes wrong can be seen in logs which can give ideas what has
happened.
- before the failure, those messages go to telemetry and can be
auto-magically processed remotely to take a decision (e.g. balance the
traffic away).

I wish all the information could be gathered in the userspace, but many
existing kernel/driver messages are valuable. A more detailed example is
hung task detector: we want to have the backtrace for a task that is
in uninterruptible state too long, but only in logs as printing on
serial console may lead to holding console lock and watchdog.
Besides customer notifications and investigations, I see the value of such
"bottleneck" warnings during long-running integration tests.

Thanks,
          Dmitry

  parent reply index

Thread overview: 39+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-11-06  3:04 Dmitry Safonov
2019-11-06  3:05 ` [PATCH 26/50] powerpc: Add show_stack_loglvl() Dmitry Safonov
2019-11-06  9:52   ` Michael Ellerman
2019-11-06 16:01     ` Dmitry Safonov
2019-11-06  8:35 ` [PATCH 00/50] Add log level to show_stack() Petr Mladek
2019-11-06 16:12   ` Dmitry Safonov
2019-11-08 10:37   ` Sergey Senozhatsky
2019-11-08 13:04     ` Petr Mladek
2019-11-11  1:23       ` Sergey Senozhatsky
2019-11-11  9:12         ` Petr Mladek
2019-11-12  4:44           ` Sergey Senozhatsky
2019-11-12  4:57             ` Sergey Senozhatsky
2019-11-12  8:35               ` Petr Mladek
2019-11-12 10:12                 ` Sergey Senozhatsky
2019-11-13  1:23                   ` Sergey Senozhatsky
2019-11-13  2:25                     ` Dmitry Safonov
2019-11-13  6:33                       ` Sergey Senozhatsky
2019-11-13  8:47                         ` Petr Mladek
2019-11-13 16:24                           ` Steven Rostedt
2019-11-13 16:40                         ` Dmitry Safonov
2019-11-15  0:50                           ` Sergey Senozhatsky
2019-11-13 15:32             ` Steven Rostedt
2019-11-11 19:47         ` Dmitry Safonov
2019-11-12  2:17           ` Sergey Senozhatsky
2019-11-12  2:40             ` Dmitry Safonov
2019-11-12  4:25               ` Sergey Senozhatsky
2019-11-13  2:41                 ` Dmitry Safonov
2019-11-13  2:51                   ` Sergey Senozhatsky
2019-11-13 15:39             ` Steven Rostedt
2019-11-15  3:36               ` Sergey Senozhatsky
2019-11-06  9:20 ` Peter Zijlstra
2019-11-06 16:27   ` Dmitry Safonov
2019-11-06 20:34     ` Peter Zijlstra
2019-11-06 23:25       ` Russell King - ARM Linux admin
2019-11-13 15:44         ` Steven Rostedt
2019-11-08 16:28       ` Dmitry Safonov [this message]
2019-11-08 17:30         ` Russell King - ARM Linux admin
2019-11-08 21:08           ` Dmitry Safonov
2019-11-13 15:41       ` Steven Rostedt

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAGrbwDRgX1BZoFrVYSXhAeeUKHrB=q3w8jXFPOuRvO7HV4Ciqw@mail.gmail.com' \
    --to=dima@arista.com \
    --cc=0x7f454c46@gmail.com \
    --cc=James.Bottomley@hansenpartnership.com \
    --cc=akpm@linux-foundation.org \
    --cc=anton.ivanov@cambridgegreys.com \
    --cc=aou@eecs.berkeley.edu \
    --cc=bcain@codeaurora.org \
    --cc=borntraeger@de.ibm.com \
    --cc=bp@alien8.de \
    --cc=bsegall@google.com \
    --cc=catalin.marinas@arm.com \
    --cc=chris@zankel.net \
    --cc=clang-built-linux@googlegroups.com \
    --cc=dalias@libc.org \
    --cc=daniel.thompson@linaro.org \
    --cc=davem@davemloft.net \
    --cc=deanbo422@gmail.com \
    --cc=deller@gmx.de \
    --cc=dianders@chromium.org \
    --cc=dietmar.eggemann@arm.com \
    --cc=fenghua.yu@intel.com \
    --cc=geert@linux-m68k.org \
    --cc=gor@linux.ibm.com \
    --cc=green.hu@gmail.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=guoren@kernel.org \
    --cc=gxt@pku.edu.cn \
    --cc=heiko.carstens@de.ibm.com \
    --cc=hpa@zytor.com \
    --cc=ink@jurassic.park.msu.ru \
    --cc=jacquiot.aurelien@gmail.com \
    --cc=jason.wessel@windriver.com \
    --cc=jcmvbkbc@gmail.com \
    --cc=jdike@addtoit.com \
    --cc=jhogan@kernel.org \
    --cc=jonas@southpole.se \
    --cc=jslaby@suse.com \
    --cc=juri.lelli@redhat.com \
    --cc=kgdb-bugreport@lists.sourceforge.net \
    --cc=len.brown@intel.com \
    --cc=lftan@altera.com \
    --cc=linux-alpha@vger.kernel.org \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-c6x-dev@linux-c6x.org \
    --cc=linux-hexagon@vger.kernel.org \
    --cc=linux-ia64@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-m68k@lists.linux-m68k.org \
    --cc=linux-mips@vger.kernel.org \
    --cc=linux-parisc@vger.kernel.org \
    --cc=linux-pm@vger.kernel.org \
    --cc=linux-riscv@lists.infradead.org \
    --cc=linux-s390@vger.kernel.org \
    --cc=linux-sh@vger.kernel.org \
    --cc=linux-snps-arc@lists.infradead.org \
    --cc=linux-um@lists.infradead.org \
    --cc=linux-xtensa@linux-xtensa.org \
    --cc=linux@armlinux.org.uk \
    --cc=linuxppc-dev@lists.ozlabs.org \
    --cc=mattst88@gmail.com \
    --cc=mgorman@suse.de \
    --cc=mingo@kernel.org \
    --cc=mingo@redhat.com \
    --cc=monstr@monstr.eu \
    --cc=msalter@redhat.com \
    --cc=nios2-dev@lists.rocketboards.org \
    --cc=openrisc@lists.librecores.org \
    --cc=palmer@dabbelt.com \
    --cc=paul.walmsley@sifive.com \
    --cc=paulburton@kernel.org \
    --cc=paulus@samba.org \
    --cc=pavel@ucw.cz \
    --cc=penguin-kernel@i-love.sakura.ne.jp \
    --cc=peterz@infradead.org \
    --cc=pmladek@suse.com \
    --cc=ralf@linux-mips.org \
    --cc=richard@nod.at \
    --cc=rjw@rjwysocki.net \
    --cc=rostedt@goodmis.org \
    --cc=rth@twiddle.net \
    --cc=sergey.senozhatsky@gmail.com \
    --cc=shorne@gmail.com \
    --cc=sparclinux@vger.kernel.org \
    --cc=stefan.kristiansson@saunalahti.fi \
    --cc=tglx@linutronix.de \
    --cc=tony.luck@intel.com \
    --cc=uclinux-h8-devel@lists.sourceforge.jp \
    --cc=vgupta@synopsys.com \
    --cc=vincent.guittot@linaro.org \
    --cc=will@kernel.org \
    --cc=x86@kernel.org \
    --cc=ysato@users.sourceforge.jp \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LinuxPPC-Dev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linuxppc-dev/0 linuxppc-dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linuxppc-dev linuxppc-dev/ https://lore.kernel.org/linuxppc-dev \
		linuxppc-dev@lists.ozlabs.org linuxppc-dev@ozlabs.org
	public-inbox-index linuxppc-dev

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.ozlabs.lists.linuxppc-dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git