linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Andrew Morton <akpm@linux-foundation.org>
To: Jan Kara <jack@suse.cz>
Cc: Steven Rostedt <rostedt@goodmis.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Frederic Weisbecker <fweisbec@gmail.com>,
	jslaby@suse.cz, Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Ingo Molnar <mingo@kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	"kay.sievers" <kay.sievers@vrfy.org>
Subject: Re: [PATCH 3/3] printk: Avoid softlockups in console_unlock()
Date: Fri, 15 Feb 2013 14:22:19 -0800	[thread overview]
Message-ID: <20130215142219.01915825.akpm@linux-foundation.org> (raw)
In-Reply-To: <20130215165710.GB12734@quack.suse.cz>

On Fri, 15 Feb 2013 17:57:10 +0100
Jan Kara <jack@suse.cz> wrote:

> A CPU can be caught in console_unlock() for a long time (tens of seconds are
> reported by our customers) when other CPUs are using printk heavily and serial
> console makes printing slow. Despite serial console drivers are calling
> touch_nmi_watchdog() this triggers softlockup warnings because
> interrupts are disabled for the whole time console_unlock() runs (e.g.
> vprintk() calls console_unlock() with interrupts disabled). Thus IPIs
> cannot be processed and other CPUs get stuck spinning in calls like
> smp_call_function_many(). Also RCU eventually starts reporting lockups.
> 
> In my artifical testing I also managed to trigger a situation when disk
> disappeared from the system apparently because commands to / from it
> could not be delivered for long enough. This is why just silencing
> watchdogs isn't a reliable solution to the problem and we simply have to
> avoid spending too long in console_unlock().
> 
> We fix the issue by limiting the time we spend in console_unlock() to
> watchdog_thresh() / 4 (unless we are in an early boot stage or oops is
> happening). The rest of the buffer will be printed either by further
> callers to printk() or during next timer tick.
> 

It still gives me tummy ache :(

The patch adds additional tests of oops_in_progress.  Some description
of your thinking on that matter would be appropriate?

> --- a/kernel/printk.c
> +++ b/kernel/printk.c
> @@ -1990,17 +1990,31 @@ int is_console_locked(void)
>  #define PRINTK_PENDING_OUTPUT	2
>  
>  static unsigned long printk_pending;
> +static int last_printing_cpu = -1;
> +
> +static bool __console_unlock(void);
>  
>  void printk_tick(void)

printk_tick() no longer exists in linux-next.


  reply	other threads:[~2013-02-15 22:22 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-02-06  1:05 [RFC][PATCH] printk: Remove separate printk_sched buffers and use printk buf instead Steven Rostedt
2013-02-06 14:32 ` Jan Kara
2013-02-06 23:02 ` Jan Kara
2013-02-12 12:22   ` Jan Kara
2013-02-14 18:48     ` Steven Rostedt
2013-02-15 16:50       ` Jan Kara
2013-02-15 16:53         ` Steven Rostedt
2013-02-15 17:00           ` Jan Kara
2013-02-15 16:54   ` Jan Kara
2013-02-15 16:57   ` [PATCH 3/3] printk: Avoid softlockups in console_unlock() Jan Kara
2013-02-15 22:22     ` Andrew Morton [this message]
2013-02-18 16:31       ` Jan Kara

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20130215142219.01915825.akpm@linux-foundation.org \
    --to=akpm@linux-foundation.org \
    --cc=fweisbec@gmail.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=jack@suse.cz \
    --cc=jslaby@suse.cz \
    --cc=kay.sievers@vrfy.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).