All of lore.kernel.org
 help / color / mirror / Atom feed
From: Pavel Machek <pavel@ucw.cz>
To: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Cc: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>,
	Jan Kara <jack@suse.cz>,
	"Eric W. Biederman" <ebiederm@xmission.com>,
	Ye Xiaolong <xiaolong.ye@intel.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Petr Mladek <pmladek@suse.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Peter Zijlstra <peterz@infradead.org>,
	"Rafael J . Wysocki" <rjw@rjwysocki.net>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Jiri Slaby <jslaby@suse.com>, Len Brown <len.brown@intel.com>,
	linux-kernel@vger.kernel.org, lkp@01.org
Subject: Re: [printk]  fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
Date: Sun, 9 Apr 2017 12:12:30 +0200	[thread overview]
Message-ID: <20170409101230.GB27363@amd> (raw)
In-Reply-To: <20170407151306.GA384@tigerII.localdomain>

[-- Attachment #1: Type: text/plain, Size: 2819 bytes --]

On Sat 2017-04-08 00:13:06, Sergey Senozhatsky wrote:
> On (04/07/17 14:44), Pavel Machek wrote:
> [..]
> > > [..]
> > > > I believe "spend at most 2 seconds in printk(), then print a warning
> > > > and offload" is a solution closer to what we had before.
> > > 
> > > a warning here can be very noisy.
> > 
> > Well, on normally-configured it should be ok. We don't commonly see
> > printk problems... If it is too noisy, perhaps we should increase from
> > 2 seconds, but I don't think it will be problem.
> 
> we are looking at different typical setups :) serial console being 45
> seconds behind logbuf does not surprise me anymore.
> 
> [..]
> > > what we have been thinking about is something like printk-stall detection.
> > > we probably (there are some if-s) can detect in printk() that offloading
> > > does not work and we must automatically switch to printk_emergency mode.
> > > that, in theory, can relax our dependency on printk_emergency_begin/end
> > > being in the right place at the right time. need to think more about it.
> > 
> > So... I don't really like the begin/end interface. I would rather have
> > printk_emergency(KERN_ ...).
> 
> you mean a single printk_emergency() switches printk to emergency mode
> or printk_emergency(KERN_ ... ) is a single message that must be printed
> in emergency mode?

The latter. Having state is ugly.

> printk() depends on console_trylock(). we can't expect printk_emergency(KERN_ ...)
> to always do more than just log_store().
> 
> the idea behind begin/end interface is that you can do
> 
> 	emergency_begin
> 	printk
> 	pr_cont
> 	pr_cont
> 	pr_cont
> 	printk
> 	dump_stack
> 	emergency_end
> 
> with out the need of rewriting dump_stack() or anything else to use
> printk_emergency(). we, for example, do this in sysrq patch from this
> series.

Well.. I guess it is less work to include emergency_begin/end() but I
also believe result will state-less solution will be cleaner.

> > Second... I don't think "stuck detector" is that helpful. What I
> > usually seen was some rather innocent kernel message followed by
> > hard-lock. That's where "message delayed" is useful..
> 
> a side note,
> that's rather unclear to me how would "message delayed" really help.
> if your system hard-lockup so badly and there are no printk messages
> even from NMI watchdog, then we won't be able to print that message.

We are talking about

   printk("unusual condition");
   do_something_clever(); /* Which unfortunately hard-crashes the machine */

that works with my proposal, but not with yours. Seen it happen many
times before.

									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

WARNING: multiple messages have this Message-ID (diff)
From: Pavel Machek <pavel@ucw.cz>
To: lkp@lists.01.org
Subject: Re: [printk] fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage
Date: Sun, 09 Apr 2017 12:12:30 +0200	[thread overview]
Message-ID: <20170409101230.GB27363@amd> (raw)
In-Reply-To: <20170407151306.GA384@tigerII.localdomain>

[-- Attachment #1: Type: text/plain, Size: 2819 bytes --]

On Sat 2017-04-08 00:13:06, Sergey Senozhatsky wrote:
> On (04/07/17 14:44), Pavel Machek wrote:
> [..]
> > > [..]
> > > > I believe "spend at most 2 seconds in printk(), then print a warning
> > > > and offload" is a solution closer to what we had before.
> > > 
> > > a warning here can be very noisy.
> > 
> > Well, on normally-configured it should be ok. We don't commonly see
> > printk problems... If it is too noisy, perhaps we should increase from
> > 2 seconds, but I don't think it will be problem.
> 
> we are looking at different typical setups :) serial console being 45
> seconds behind logbuf does not surprise me anymore.
> 
> [..]
> > > what we have been thinking about is something like printk-stall detection.
> > > we probably (there are some if-s) can detect in printk() that offloading
> > > does not work and we must automatically switch to printk_emergency mode.
> > > that, in theory, can relax our dependency on printk_emergency_begin/end
> > > being in the right place at the right time. need to think more about it.
> > 
> > So... I don't really like the begin/end interface. I would rather have
> > printk_emergency(KERN_ ...).
> 
> you mean a single printk_emergency() switches printk to emergency mode
> or printk_emergency(KERN_ ... ) is a single message that must be printed
> in emergency mode?

The latter. Having state is ugly.

> printk() depends on console_trylock(). we can't expect printk_emergency(KERN_ ...)
> to always do more than just log_store().
> 
> the idea behind begin/end interface is that you can do
> 
> 	emergency_begin
> 	printk
> 	pr_cont
> 	pr_cont
> 	pr_cont
> 	printk
> 	dump_stack
> 	emergency_end
> 
> with out the need of rewriting dump_stack() or anything else to use
> printk_emergency(). we, for example, do this in sysrq patch from this
> series.

Well.. I guess it is less work to include emergency_begin/end() but I
also believe result will state-less solution will be cleaner.

> > Second... I don't think "stuck detector" is that helpful. What I
> > usually seen was some rather innocent kernel message followed by
> > hard-lock. That's where "message delayed" is useful..
> 
> a side note,
> that's rather unclear to me how would "message delayed" really help.
> if your system hard-lockup so badly and there are no printk messages
> even from NMI watchdog, then we won't be able to print that message.

We are talking about

   printk("unusual condition");
   do_something_clever(); /* Which unfortunately hard-crashes the machine */

that works with my proposal, but not with yours. Seen it happen many
times before.

									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

  parent reply	other threads:[~2017-04-09 10:12 UTC|newest]

Thread overview: 111+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-03-29  9:25 [RFC][PATCHv2 0/8] printk: introduce printing kernel thread Sergey Senozhatsky
2017-03-29  9:25 ` [RFC][PATCHv2 1/8] printk: move printk_pending out of per-cpu Sergey Senozhatsky
2017-03-31 13:09   ` Petr Mladek
2017-03-31 13:33     ` Peter Zijlstra
2017-04-03 11:23       ` Sergey Senozhatsky
2017-04-03 12:43         ` Petr Mladek
2017-03-29  9:25 ` [RFC][PATCHv2 2/8] printk: introduce printing kernel thread Sergey Senozhatsky
2017-04-04  9:01   ` Petr Mladek
2017-04-04  9:36     ` Sergey Senozhatsky
2017-04-06 17:14   ` Pavel Machek
2017-04-07  5:12     ` Sergey Senozhatsky
2017-04-07  7:21       ` Pavel Machek
2017-04-07  8:15         ` Sergey Senozhatsky
2017-04-07 12:06           ` Pavel Machek
2017-03-29  9:25 ` [RFC][PATCHv2 3/8] printk: offload printing from wake_up_klogd_work_func() Sergey Senozhatsky
2017-03-31 14:56   ` Petr Mladek
2017-04-04 16:15     ` Sergey Senozhatsky
2017-03-29  9:25 ` [RFC][PATCHv2 4/8] pm: switch to printk.emergency mode in unsafe places Sergey Senozhatsky
2017-03-31 15:06   ` Petr Mladek
2017-04-06 17:20   ` Pavel Machek
2017-04-09 10:59     ` Andreas Mohr
2017-04-10 12:20       ` Petr Mladek
2017-04-10 14:38         ` Sergey Senozhatsky
2017-03-29  9:25 ` [RFC][PATCHv2 5/8] sysrq: " Sergey Senozhatsky
2017-03-31 15:37   ` Petr Mladek
2017-04-01  0:04     ` Sergey Senozhatsky
2017-03-29  9:25 ` [RFC][PATCHv2 6/8] kexec: " Sergey Senozhatsky
2017-03-31 15:39   ` Petr Mladek
2017-03-29  9:25 ` [RFC][PATCHv2 7/8] printk: add printk emergency_mode parameter Sergey Senozhatsky
2017-04-03 15:29   ` Petr Mladek
2017-04-04  8:29     ` Sergey Senozhatsky
2017-03-29  9:25 ` [RFC][PATCHv2 8/8] printk: enable printk offloading Sergey Senozhatsky
2017-03-30 21:38   ` [printk] fbc14616f4: BUG:kernel_reboot-without-warning_in_test_stage kernel test robot
2017-03-30 21:38     ` kernel test robot
2017-03-31  2:35     ` Sergey Senozhatsky
2017-03-31  2:35       ` Sergey Senozhatsky
2017-03-31  4:04       ` Sergey Senozhatsky
2017-03-31  4:04         ` Sergey Senozhatsky
2017-03-31  6:39         ` Ye Xiaolong
2017-03-31  6:39           ` Ye Xiaolong
2017-03-31 14:47           ` Sergey Senozhatsky
2017-03-31 14:47             ` Sergey Senozhatsky
2017-03-31 15:28             ` Eric W. Biederman
2017-03-31 15:28               ` Eric W. Biederman
2017-04-03  9:31               ` Jan Kara
2017-04-03  9:31                 ` Jan Kara
2017-04-03 10:06                 ` Petr Mladek
2017-04-03 10:06                   ` Petr Mladek
2017-04-06 17:33                 ` Pavel Machek
2017-04-06 17:33                   ` Pavel Machek
2017-04-07  4:44                   ` Sergey Senozhatsky
2017-04-07  4:44                     ` Sergey Senozhatsky
2017-04-07  7:15                     ` Pavel Machek
2017-04-07  7:15                       ` Pavel Machek
2017-04-07  7:46                       ` Sergey Senozhatsky
2017-04-07  7:46                         ` Sergey Senozhatsky
2017-04-07  8:14                         ` Pavel Machek
2017-04-07  8:14                           ` Pavel Machek
2017-04-07 12:10                           ` Sergey Senozhatsky
2017-04-07 12:10                             ` Sergey Senozhatsky
2017-04-07 12:44                             ` Pavel Machek
2017-04-07 12:44                               ` Pavel Machek
2017-04-07 14:40                               ` Steven Rostedt
2017-04-07 14:40                                 ` Steven Rostedt
2017-05-08  6:37                                 ` Sergey Senozhatsky
2017-05-08  6:37                                   ` Sergey Senozhatsky
2017-05-17 13:13                                   ` Petr Mladek
2017-05-17 13:13                                     ` Petr Mladek
2017-04-07 15:13                               ` Sergey Senozhatsky
2017-04-07 15:13                                 ` Sergey Senozhatsky
2017-04-07 15:23                                 ` Peter Zijlstra
2017-04-07 15:23                                   ` Peter Zijlstra
2017-04-07 15:40                                   ` Sergey Senozhatsky
2017-04-07 15:40                                     ` Sergey Senozhatsky
2017-04-09 18:21                                     ` Eric W. Biederman
2017-04-09 18:21                                       ` Eric W. Biederman
2017-04-10  4:46                                       ` Sergey Senozhatsky
2017-04-10  4:46                                         ` Sergey Senozhatsky
2017-04-09 10:12                                 ` Pavel Machek [this message]
2017-04-09 10:12                                   ` Pavel Machek
2017-04-10  4:53                                   ` Sergey Senozhatsky
2017-04-10  4:53                                     ` Sergey Senozhatsky
2017-04-10 11:54                                     ` Petr Mladek
2017-04-10 11:54                                       ` Petr Mladek
2017-04-10 15:08                                       ` Sergey Senozhatsky
2017-04-10 15:08                                         ` Sergey Senozhatsky
2017-04-10 18:48                                     ` Pavel Machek
2017-04-10 18:48                                       ` Pavel Machek
2017-04-11  1:46                                       ` Sergey Senozhatsky
2017-04-11  1:46                                         ` Sergey Senozhatsky
2017-04-11 16:19                                         ` Sergey Senozhatsky
2017-04-12 18:43                                           ` Pavel Machek
2017-04-13  4:34                                             ` Sergey Senozhatsky
2017-04-13  5:50                                           ` Sergey Senozhatsky
2017-04-13  8:19                                             ` Sergey Senozhatsky
2017-04-13 14:03                                           ` Petr Mladek
2017-04-14  4:42                                             ` Sergey Senozhatsky
2017-04-07 14:29                           ` Steven Rostedt
2017-04-07 14:29                             ` Steven Rostedt
2017-04-09  9:57                             ` Pavel Machek
2017-04-09  9:57                               ` Pavel Machek
2017-04-03 10:51               ` Sergey Senozhatsky
2017-04-03 10:51                 ` Sergey Senozhatsky
2017-04-05  7:29           ` Ye Xiaolong
2017-04-05  7:29             ` Ye Xiaolong
2017-04-05  8:40             ` Sergey Senozhatsky
2017-04-05  8:40               ` Sergey Senozhatsky
2017-04-03 15:42   ` [RFC][PATCHv2 8/8] printk: enable printk offloading Petr Mladek
2017-04-04 13:20     ` Sergey Senozhatsky
2017-04-02  4:13 [printk] fbc14616f4 BUG: kernel reboot-without-warning in test stage Fengguang Wu
2017-04-03  2:14 ` Sergey Senozhatsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170409101230.GB27363@amd \
    --to=pavel@ucw.cz \
    --cc=akpm@linux-foundation.org \
    --cc=ebiederm@xmission.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=jack@suse.cz \
    --cc=jslaby@suse.com \
    --cc=len.brown@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=lkp@01.org \
    --cc=peterz@infradead.org \
    --cc=pmladek@suse.com \
    --cc=rjw@rjwysocki.net \
    --cc=rostedt@goodmis.org \
    --cc=sergey.senozhatsky.work@gmail.com \
    --cc=sergey.senozhatsky@gmail.com \
    --cc=torvalds@linux-foundation.org \
    --cc=xiaolong.ye@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.