From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter Zijlstra Date: Wed, 19 Apr 2017 17:21:39 +0000 Subject: Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI Message-Id: <20170419172139.hipylcdsarhvnsby@hirez.programming.kicks-ass.net> List-Id: References: <1461239325-22779-1-git-send-email-pmladek@suse.com> <1461239325-22779-2-git-send-email-pmladek@suse.com> <20170419131341.76bc7634@gandalf.local.home> In-Reply-To: <20170419131341.76bc7634@gandalf.local.home> MIME-Version: 1.0 Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: quoted-printable To: linux-arm-kernel@lists.infradead.org On Wed, Apr 19, 2017 at 01:13:41PM -0400, Steven Rostedt wrote: > On Thu, 21 Apr 2016 13:48:42 +0200 > Petr Mladek wrote: >=20 > > printk() takes some locks and could not be used a safe way in NMI conte= xt. >=20 > I just found a problem with this solution. It kills ftrace dumps from > NMI context :-( >=20 > [ 1295.168495] <...>-67423 10dNh1 382171111us : do_raw_spin_lock <-_r= aw_spin_lock > [ 1295.168495] <...>-67423 10dNh1 382171111us : sched_stat_runtime: c= omm=CC1 pidg423 runtime=96858 [ns] vruntime=11924198270 [ns] > [ 1295.168496] <...>-67423 10dNh1 382171111us : lock_acquire: fffffff= f81c5c940 read rcu_read_lock > [ 1295.168497] > [ 1295.168498] Lost 4890096 message(s)! > [ 1296.805063] ---[ end Kernel panic - not syncing: Hard LOCKUP > [ 1296.811553] unchecked MSR access error: WRMSR to 0x83f (tried to write= 0x00000000000000f6) at rIP: 0xffffffff81046fc7 (native_apic_msr_write+0x27= /0x40) > [ 1296.811553] Call Trace: > [ 1296.811553] >=20 > I was hoping to see a cause of a hard lockup by enabling > ftrace_dump_on_oops. But as NMIs now have a very small buffer that > gets flushed, we need to find a new way to print out the full ftrace > buffer over serial. >=20 > Thoughts? early_printk ;-) From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S968364AbdDSRWZ (ORCPT ); Wed, 19 Apr 2017 13:22:25 -0400 Received: from bombadil.infradead.org ([65.50.211.133]:59889 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S968203AbdDSRWT (ORCPT ); Wed, 19 Apr 2017 13:22:19 -0400 Date: Wed, 19 Apr 2017 19:21:39 +0200 From: Peter Zijlstra To: Steven Rostedt Cc: Petr Mladek , Andrew Morton , Russell King , Daniel Thompson , Jiri Kosina , Ingo Molnar , Thomas Gleixner , Sergey Senozhatsky , Chris Metcalf , linux-kernel@vger.kernel.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, adi-buildroot-devel@lists.sourceforge.net, linux-cris-kernel@axis.com, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, Jan Kara , Ralf Baechle , Benjamin Herrenschmidt , Martin Schwidefsky , David Miller Subject: Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI Message-ID: <20170419172139.hipylcdsarhvnsby@hirez.programming.kicks-ass.net> References: <1461239325-22779-1-git-send-email-pmladek@suse.com> <1461239325-22779-2-git-send-email-pmladek@suse.com> <20170419131341.76bc7634@gandalf.local.home> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170419131341.76bc7634@gandalf.local.home> User-Agent: NeoMutt/20170113 (1.7.2) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Apr 19, 2017 at 01:13:41PM -0400, Steven Rostedt wrote: > On Thu, 21 Apr 2016 13:48:42 +0200 > Petr Mladek wrote: > > > printk() takes some locks and could not be used a safe way in NMI context. > > I just found a problem with this solution. It kills ftrace dumps from > NMI context :-( > > [ 1295.168495] <...>-67423 10dNh1 382171111us : do_raw_spin_lock <-_raw_spin_lock > [ 1295.168495] <...>-67423 10dNh1 382171111us : sched_stat_runtime: comm=cc1 pid=67423 runtime=96858 [ns] vruntime=11924198270 [ns] > [ 1295.168496] <...>-67423 10dNh1 382171111us : lock_acquire: ffffffff81c5c940 read rcu_read_lock > [ 1295.168497] > [ 1295.168498] Lost 4890096 message(s)! > [ 1296.805063] ---[ end Kernel panic - not syncing: Hard LOCKUP > [ 1296.811553] unchecked MSR access error: WRMSR to 0x83f (tried to write 0x00000000000000f6) at rIP: 0xffffffff81046fc7 (native_apic_msr_write+0x27/0x40) > [ 1296.811553] Call Trace: > [ 1296.811553] > > I was hoping to see a cause of a hard lockup by enabling > ftrace_dump_on_oops. But as NMIs now have a very small buffer that > gets flushed, we need to find a new way to print out the full ftrace > buffer over serial. > > Thoughts? early_printk ;-) From mboxrd@z Thu Jan 1 00:00:00 1970 From: peterz@infradead.org (Peter Zijlstra) Date: Wed, 19 Apr 2017 19:21:39 +0200 Subject: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI In-Reply-To: <20170419131341.76bc7634@gandalf.local.home> References: <1461239325-22779-1-git-send-email-pmladek@suse.com> <1461239325-22779-2-git-send-email-pmladek@suse.com> <20170419131341.76bc7634@gandalf.local.home> Message-ID: <20170419172139.hipylcdsarhvnsby@hirez.programming.kicks-ass.net> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org On Wed, Apr 19, 2017 at 01:13:41PM -0400, Steven Rostedt wrote: > On Thu, 21 Apr 2016 13:48:42 +0200 > Petr Mladek wrote: > > > printk() takes some locks and could not be used a safe way in NMI context. > > I just found a problem with this solution. It kills ftrace dumps from > NMI context :-( > > [ 1295.168495] <...>-67423 10dNh1 382171111us : do_raw_spin_lock <-_raw_spin_lock > [ 1295.168495] <...>-67423 10dNh1 382171111us : sched_stat_runtime: comm=cc1 pid=67423 runtime=96858 [ns] vruntime=11924198270 [ns] > [ 1295.168496] <...>-67423 10dNh1 382171111us : lock_acquire: ffffffff81c5c940 read rcu_read_lock > [ 1295.168497] > [ 1295.168498] Lost 4890096 message(s)! > [ 1296.805063] ---[ end Kernel panic - not syncing: Hard LOCKUP > [ 1296.811553] unchecked MSR access error: WRMSR to 0x83f (tried to write 0x00000000000000f6) at rIP: 0xffffffff81046fc7 (native_apic_msr_write+0x27/0x40) > [ 1296.811553] Call Trace: > [ 1296.811553] > > I was hoping to see a cause of a hard lockup by enabling > ftrace_dump_on_oops. But as NMIs now have a very small buffer that > gets flushed, we need to find a new way to print out the full ftrace > buffer over serial. > > Thoughts? early_printk ;-)