All of lore.kernel.org
 help / color / mirror / Atom feed
From: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
To: linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI
Date: Fri, 21 Apr 2017 01:57:25 +0000	[thread overview]
Message-ID: <20170421015724.GA586@jagdpanzerIV.localdomain> (raw)
In-Reply-To: <20170420131154.GL3452@pathway.suse.cz>

Hello,

On (04/20/17 15:11), Petr Mladek wrote:
[..]
> Good analyze. I would summarize it that we need to be careful of:
> 
>   + logbug_lock
>   + PRINTK_SAFE_CONTEXT
>   + locks used by console drivers
> 
> The first two things are easy to check. Except that a check for logbuf_lock
> might produce false negatives. The last check is very hard.
> 
> > so at the moment what I can think of is something like
> > 
> >   -- check this_cpu_read(printk_context) in NMI prink
> > 
> > 	-- if we are NOT in printk_safe on this CPU, then do printk_deferred()
> > 	   and bypass `nmi_print_seq' buffer
> 
> I would add also a check for logbuf_lock.
> > 	-- if we are in printk_safe
> > 	  -- well... bad luck... have a bigger buffer.
> 
> Yup, we do the best effort while still trying to stay on the safe
> side.
> 
> I have cooked up a patch based on this. It uses printk_deferred()
> in NMI when it is safe. Note that console_flush_on_panic() will
> try to get them on the console when a kdump is not generated.
> I believe that it will help Steven.


OK. I need to look more at the patch. It does more than I'd expected/imagined.


[..]
>  void printk_nmi_enter(void)
>  {
> -	this_cpu_or(printk_context, PRINTK_NMI_CONTEXT_MASK);
> +	/*
> +	 * The size of the extra per-CPU buffer is limited. Use it
> +	 * only when really needed.
> +	 */
> +	if (this_cpu_read(printk_context) & PRINTK_SAFE_CONTEXT_MASK ||
> +	    raw_spin_is_locked(&logbuf_lock)) {
> +		this_cpu_or(printk_context, PRINTK_NMI_CONTEXT_MASK);
> +	} else {
> +		this_cpu_or(printk_context, PRINTK_NMI_DEFERRED_CONTEXT_MASK);
> +	}
>  }

well... the logbuf_lock can temporarily be locked from another CPU. I'd say
that spin_is_locked() has better chances for false positive than
this_cpu_read(printk_context). because this_cpu_read(printk_context) depends
only on this CPU state, while spin_is_locked() depends on all CPUs. and the
idea with this_cpu_read(printk_context) was that we check if the logbuf_lock
was locked from this particular CPU.

I agree that this_cpu_read(printk_context) covers slightly more than
logbuf_lock scope, so we may get positive this_cpu_read(printk_context)
with unlocked logbuf_lock, but I don't tend to think that it's a big
problem.


wouldn't something as simple as below do the trick?
// absolutely and completely untested //


diff --git a/kernel/printk/printk_safe.c b/kernel/printk/printk_safe.c
index 033e50a7d706..c7477654c5b1 100644
--- a/kernel/printk/printk_safe.c
+++ b/kernel/printk/printk_safe.c
@@ -303,7 +303,10 @@ static int vprintk_nmi(const char *fmt, va_list args)
 {
        struct printk_safe_seq_buf *s = this_cpu_ptr(&nmi_print_seq);
 
-       return printk_safe_log_store(s, fmt, args);
+       if (this_cpu_read(printk_context) & PRINTK_SAFE_CONTEXT_MASK)
+               return printk_safe_log_store(s, fmt, args);
+
+       return vprintk_emit(0, LOGLEVEL_SCHED, NULL, 0, fmt, args);
 }

	-ss

WARNING: multiple messages have this Message-ID (diff)
From: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
To: Petr Mladek <pmladek@suse.com>
Cc: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Russell King <rmk+kernel@arm.linux.org.uk>,
	Daniel Thompson <daniel.thompson@linaro.org>,
	Jiri Kosina <jkosina@suse.com>, Ingo Molnar <mingo@redhat.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Chris Metcalf <cmetcalf@ezchip.com>,
	linux-kernel@vger.kernel.org, x86@kernel.org,
	linux-arm-kernel@lists.infradead.org,
	adi-buildroot-devel@lists.sourceforge.net,
	linux-cris-kernel@axis.com, linux-mips@linux-mips.org,
	linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org,
	linux-sh@vger.kernel.org, sparclinux@vger.kernel.org,
	Jan Kara <jack@suse.cz>, Ralf Baechle <ralf@linux-mips.org>,
	Benjamin Herrenschmidt <benh@kernel.crashing.org>,
	Martin Schwidefsky <schwidefsky@de.ibm.com>,
	David Miller <davem@davemloft.net>
Subject: Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI
Date: Fri, 21 Apr 2017 10:57:25 +0900	[thread overview]
Message-ID: <20170421015724.GA586@jagdpanzerIV.localdomain> (raw)
In-Reply-To: <20170420131154.GL3452@pathway.suse.cz>

Hello,

On (04/20/17 15:11), Petr Mladek wrote:
[..]
> Good analyze. I would summarize it that we need to be careful of:
> 
>   + logbug_lock
>   + PRINTK_SAFE_CONTEXT
>   + locks used by console drivers
> 
> The first two things are easy to check. Except that a check for logbuf_lock
> might produce false negatives. The last check is very hard.
> 
> > so at the moment what I can think of is something like
> > 
> >   -- check this_cpu_read(printk_context) in NMI prink
> > 
> > 	-- if we are NOT in printk_safe on this CPU, then do printk_deferred()
> > 	   and bypass `nmi_print_seq' buffer
> 
> I would add also a check for logbuf_lock.
> > 	-- if we are in printk_safe
> > 	  -- well... bad luck... have a bigger buffer.
> 
> Yup, we do the best effort while still trying to stay on the safe
> side.
> 
> I have cooked up a patch based on this. It uses printk_deferred()
> in NMI when it is safe. Note that console_flush_on_panic() will
> try to get them on the console when a kdump is not generated.
> I believe that it will help Steven.


OK. I need to look more at the patch. It does more than I'd expected/imagined.


[..]
>  void printk_nmi_enter(void)
>  {
> -	this_cpu_or(printk_context, PRINTK_NMI_CONTEXT_MASK);
> +	/*
> +	 * The size of the extra per-CPU buffer is limited. Use it
> +	 * only when really needed.
> +	 */
> +	if (this_cpu_read(printk_context) & PRINTK_SAFE_CONTEXT_MASK ||
> +	    raw_spin_is_locked(&logbuf_lock)) {
> +		this_cpu_or(printk_context, PRINTK_NMI_CONTEXT_MASK);
> +	} else {
> +		this_cpu_or(printk_context, PRINTK_NMI_DEFERRED_CONTEXT_MASK);
> +	}
>  }

well... the logbuf_lock can temporarily be locked from another CPU. I'd say
that spin_is_locked() has better chances for false positive than
this_cpu_read(printk_context). because this_cpu_read(printk_context) depends
only on this CPU state, while spin_is_locked() depends on all CPUs. and the
idea with this_cpu_read(printk_context) was that we check if the logbuf_lock
was locked from this particular CPU.

I agree that this_cpu_read(printk_context) covers slightly more than
logbuf_lock scope, so we may get positive this_cpu_read(printk_context)
with unlocked logbuf_lock, but I don't tend to think that it's a big
problem.


wouldn't something as simple as below do the trick?
// absolutely and completely untested //


diff --git a/kernel/printk/printk_safe.c b/kernel/printk/printk_safe.c
index 033e50a7d706..c7477654c5b1 100644
--- a/kernel/printk/printk_safe.c
+++ b/kernel/printk/printk_safe.c
@@ -303,7 +303,10 @@ static int vprintk_nmi(const char *fmt, va_list args)
 {
        struct printk_safe_seq_buf *s = this_cpu_ptr(&nmi_print_seq);
 
-       return printk_safe_log_store(s, fmt, args);
+       if (this_cpu_read(printk_context) & PRINTK_SAFE_CONTEXT_MASK)
+               return printk_safe_log_store(s, fmt, args);
+
+       return vprintk_emit(0, LOGLEVEL_SCHED, NULL, 0, fmt, args);
 }

	-ss

WARNING: multiple messages have this Message-ID (diff)
From: sergey.senozhatsky.work@gmail.com (Sergey Senozhatsky)
To: linux-arm-kernel@lists.infradead.org
Subject: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI
Date: Fri, 21 Apr 2017 10:57:25 +0900	[thread overview]
Message-ID: <20170421015724.GA586@jagdpanzerIV.localdomain> (raw)
In-Reply-To: <20170420131154.GL3452@pathway.suse.cz>

Hello,

On (04/20/17 15:11), Petr Mladek wrote:
[..]
> Good analyze. I would summarize it that we need to be careful of:
> 
>   + logbug_lock
>   + PRINTK_SAFE_CONTEXT
>   + locks used by console drivers
> 
> The first two things are easy to check. Except that a check for logbuf_lock
> might produce false negatives. The last check is very hard.
> 
> > so at the moment what I can think of is something like
> > 
> >   -- check this_cpu_read(printk_context) in NMI prink
> > 
> > 	-- if we are NOT in printk_safe on this CPU, then do printk_deferred()
> > 	   and bypass `nmi_print_seq' buffer
> 
> I would add also a check for logbuf_lock.
> > 	-- if we are in printk_safe
> > 	  -- well... bad luck... have a bigger buffer.
> 
> Yup, we do the best effort while still trying to stay on the safe
> side.
> 
> I have cooked up a patch based on this. It uses printk_deferred()
> in NMI when it is safe. Note that console_flush_on_panic() will
> try to get them on the console when a kdump is not generated.
> I believe that it will help Steven.


OK. I need to look more at the patch. It does more than I'd expected/imagined.


[..]
>  void printk_nmi_enter(void)
>  {
> -	this_cpu_or(printk_context, PRINTK_NMI_CONTEXT_MASK);
> +	/*
> +	 * The size of the extra per-CPU buffer is limited. Use it
> +	 * only when really needed.
> +	 */
> +	if (this_cpu_read(printk_context) & PRINTK_SAFE_CONTEXT_MASK ||
> +	    raw_spin_is_locked(&logbuf_lock)) {
> +		this_cpu_or(printk_context, PRINTK_NMI_CONTEXT_MASK);
> +	} else {
> +		this_cpu_or(printk_context, PRINTK_NMI_DEFERRED_CONTEXT_MASK);
> +	}
>  }

well... the logbuf_lock can temporarily be locked from another CPU. I'd say
that spin_is_locked() has better chances for false positive than
this_cpu_read(printk_context). because this_cpu_read(printk_context) depends
only on this CPU state, while spin_is_locked() depends on all CPUs. and the
idea with this_cpu_read(printk_context) was that we check if the logbuf_lock
was locked from this particular CPU.

I agree that this_cpu_read(printk_context) covers slightly more than
logbuf_lock scope, so we may get positive this_cpu_read(printk_context)
with unlocked logbuf_lock, but I don't tend to think that it's a big
problem.


wouldn't something as simple as below do the trick?
// absolutely and completely untested //


diff --git a/kernel/printk/printk_safe.c b/kernel/printk/printk_safe.c
index 033e50a7d706..c7477654c5b1 100644
--- a/kernel/printk/printk_safe.c
+++ b/kernel/printk/printk_safe.c
@@ -303,7 +303,10 @@ static int vprintk_nmi(const char *fmt, va_list args)
 {
        struct printk_safe_seq_buf *s = this_cpu_ptr(&nmi_print_seq);
 
-       return printk_safe_log_store(s, fmt, args);
+       if (this_cpu_read(printk_context) & PRINTK_SAFE_CONTEXT_MASK)
+               return printk_safe_log_store(s, fmt, args);
+
+       return vprintk_emit(0, LOGLEVEL_SCHED, NULL, 0, fmt, args);
 }

	-ss

  reply	other threads:[~2017-04-21  1:57 UTC|newest]

Thread overview: 97+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-04-21 11:48 [PATCH v5 0/4] Cleaning printk stuff in NMI context Petr Mladek
2016-04-21 11:48 ` Petr Mladek
2016-04-21 11:48 ` Petr Mladek
2016-04-21 11:48 ` [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI Petr Mladek
2016-04-21 11:48   ` Petr Mladek
2016-04-21 11:48   ` Petr Mladek
2016-04-27  9:31   ` Russell King - ARM Linux
2016-04-27  9:31     ` Russell King - ARM Linux
2016-04-27  9:31     ` Russell King - ARM Linux
2017-04-19 17:13   ` Steven Rostedt
2017-04-19 17:13     ` Steven Rostedt
2017-04-19 17:13     ` Steven Rostedt
2017-04-19 17:21     ` Peter Zijlstra
2017-04-19 17:21       ` Peter Zijlstra
2017-04-19 17:21       ` Peter Zijlstra
2017-04-20  3:31     ` Sergey Senozhatsky
2017-04-20  3:31       ` Sergey Senozhatsky
2017-04-20  3:31       ` Sergey Senozhatsky
2017-04-20 13:11       ` Petr Mladek
2017-04-20 13:11         ` Petr Mladek
2017-04-20 13:11         ` Petr Mladek
2017-04-20 13:11         ` Petr Mladek
2017-04-21  1:57         ` Sergey Senozhatsky [this message]
2017-04-21  1:57           ` Sergey Senozhatsky
2017-04-21  1:57           ` Sergey Senozhatsky
2017-04-21 12:06           ` Petr Mladek
2017-04-21 12:06             ` Petr Mladek
2017-04-21 12:06             ` Petr Mladek
2017-04-24  2:17             ` Sergey Senozhatsky
2017-04-24  2:17               ` Sergey Senozhatsky
2017-04-24  2:17               ` Sergey Senozhatsky
2017-04-27 13:38               ` Petr Mladek
2017-04-27 13:38                 ` Petr Mladek
2017-04-27 13:38                 ` Petr Mladek
2017-04-27 14:31                 ` Steven Rostedt
2017-04-27 14:31                   ` Steven Rostedt
2017-04-27 14:31                   ` Steven Rostedt
2017-04-27 15:28                   ` Petr Mladek
2017-04-27 15:28                     ` Petr Mladek
2017-04-27 15:28                     ` Petr Mladek
2017-04-27 15:42                     ` Steven Rostedt
2017-04-27 15:42                       ` Steven Rostedt
2017-04-27 15:42                       ` Steven Rostedt
2017-04-28  9:02                 ` Peter Zijlstra
2017-04-28  9:02                   ` Peter Zijlstra
2017-04-28  9:02                   ` Peter Zijlstra
2017-04-28 13:44                   ` Petr Mladek
2017-04-28 13:44                     ` Petr Mladek
2017-04-28 13:44                     ` Petr Mladek
2017-04-28 13:58                     ` Peter Zijlstra
2017-04-28 13:58                       ` Peter Zijlstra
2017-04-28 13:58                       ` Peter Zijlstra
2017-04-28 14:47                       ` Steven Rostedt
2017-04-28 14:47                         ` Steven Rostedt
2017-04-28 14:47                         ` Steven Rostedt
2017-04-27 16:14         ` Steven Rostedt
2017-04-27 16:14           ` Steven Rostedt
2017-04-27 16:14           ` Steven Rostedt
2017-04-28  1:35           ` Sergey Senozhatsky
2017-04-28  1:35             ` Sergey Senozhatsky
2017-04-28  1:35             ` Sergey Senozhatsky
2017-04-28 12:57             ` Petr Mladek
2017-04-28 12:57               ` Petr Mladek
2017-04-28 12:57               ` Petr Mladek
2017-04-28 14:16               ` Steven Rostedt
2017-04-28 14:16                 ` Steven Rostedt
2017-04-28 14:16                 ` Steven Rostedt
2017-04-28  1:25         ` Sergey Senozhatsky
2017-04-28  1:25           ` Sergey Senozhatsky
2017-04-28  1:25           ` Sergey Senozhatsky
2017-04-28 12:38           ` Petr Mladek
2017-04-28 12:38             ` Petr Mladek
2017-04-28 12:38             ` Petr Mladek
2016-04-21 11:48 ` [PATCH v5 2/4] printk/nmi: warn when some message has been lost in NMI context Petr Mladek
2016-04-21 11:48   ` Petr Mladek
2016-04-21 11:48   ` Petr Mladek
2016-04-27  9:34   ` Russell King - ARM Linux
2016-04-27  9:34     ` Russell King - ARM Linux
2016-04-27  9:34     ` Russell King - ARM Linux
2016-04-21 11:48 ` [PATCH v5 3/4] printk/nmi: increase the size of NMI buffer and make it configurable Petr Mladek
2016-04-21 11:48   ` Petr Mladek
2016-04-21 11:48   ` Petr Mladek
2016-04-21 11:48 ` [PATCH v5 4/4] printk/nmi: flush NMI messages on the system panic Petr Mladek
2016-04-21 11:48   ` Petr Mladek
2016-04-21 11:48   ` Petr Mladek
2016-04-23  3:49   ` Sergey Senozhatsky
2016-04-23  3:49     ` Sergey Senozhatsky
2016-04-23  3:49     ` Sergey Senozhatsky
2016-04-26 14:21     ` Petr Mladek
2016-04-26 14:21       ` Petr Mladek
2016-04-26 14:21       ` Petr Mladek
2016-04-27  0:34       ` Sergey Senozhatsky
2016-04-27  0:34         ` Sergey Senozhatsky
2016-04-27  0:34         ` Sergey Senozhatsky
2016-04-27  0:36 ` [PATCH v5 0/4] Cleaning printk stuff in NMI context Sergey Senozhatsky
2016-04-27  0:36   ` Sergey Senozhatsky
2016-04-27  0:36   ` Sergey Senozhatsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170421015724.GA586@jagdpanzerIV.localdomain \
    --to=sergey.senozhatsky.work@gmail.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.