From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752866AbdCHPTN (ORCPT ); Wed, 8 Mar 2017 10:19:13 -0500 Received: from g9t1613g.houston.hpe.com ([15.241.32.99]:37510 "EHLO g9t1613g.houston.hpe.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750822AbdCHPRo (ORCPT ); Wed, 8 Mar 2017 10:17:44 -0500 Subject: Re: [PATCH 1/2] x86/platform: Add a low priority low frequency NMI call chain To: Ingo Molnar References: <20170306181737.059578494@asylum.americas.sgi.com> <20170306181737.322206440@asylum.americas.sgi.com> <20170307074210.GA24782@gmail.com> <20170308102829.GA11864@gmail.com> Cc: Ingo Molnar , Thomas Gleixner , "H. Peter Anvin" , Don Zickus , Peter Zijlstra , Dimitri Sivanich , Frank Ramsay , Russ Anderson , Tony Ernst , x86@kernel.org, linux-kernel@vger.kernel.org From: Mike Travis Message-ID: <562edb0b-56f8-0ca4-c9ed-841a79e0cf2a@hpe.com> Date: Wed, 8 Mar 2017 07:17:13 -0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 In-Reply-To: <20170308102829.GA11864@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 3/8/2017 2:28 AM, Ingo Molnar wrote: > > * Mike Travis wrote: > >> >> >> On 3/6/2017 11:42 PM, Ingo Molnar wrote: >>> >>> * Mike Travis wrote: >>> >>>> Add a new NMI call chain that is called last after all other NMI handlers >>>> have been checked and did not "handle" the NMI. This mimics the current >>>> NMI_UNKNOWN call chain except it eliminates the WARNING message about >>>> multiple NMI handlers registering on this call chain. >>>> >>>> This call chain dramatically lowers the NMI call frequency when high >>>> frequency NMI tools are in use, notably the perf tools. It is required >>>> for NMI handlers that cannot sustain a high NMI call rate without >>>> ramifications to the system operability. >>> >>> So how about we just turn off that warning instead? I don't remember the last time >>> it actually _helped_ us find any kernel or hardware bug - and it has caused tons >>> of problems... >> >> I can do that, with an even simpler patch... >> >>> >>> It's not like we warn about excess regular IRQs either - we either handle them or >>> at most increase a counter somewhere. We could do the same for NMIs: introduce a >>> counter somewhere that counts the number of seemingly unhandled NMIs. >> >> Really "unknown" NMI errors are reported by either the "dazed and >> confused" message or if the panic on unknown nmi is set, then the >> system will panic. So unknown NMI occurrences are already being >> dealt with. > > So I'd even remove the 'dazed and confused' message - has it ever helped us? I can remove it though it seems to have become an institution, or more correctly, a common term of reference. :) It does precede the decision to either attempt to continue system operation, or panic the system immediately. > If NMIs are generated but not handled properly then developers and users will > notice it just like when IRQs are lost: either through bad system behavior or via > weird stats in procfs. The kernel log should not get spammed. Having some notice is probably a good thing even if for archaic reasons. We recently discovered that an internal system error triggered an NMI event. Without any notice, the system would not have been suspected of acting strangely, but data could potentially have been silently lost. (NMI seems by far the least standard standard in the x86 architecture.) Also, I don't think IRQs and NMIs are in the same league. Missing an IRQ means an expected I/O operation did not occur. Prudent drivers can set a timeout to notice missing interrupts. Missing an NMI usually means that something unexpected occurred but was not dealt with. Losing perf interrupts is recoverable since there will be another along shortly. But missing an NMI due to a system failure event is not. (Why NMI is heavily overloaded, and not very standardized.) > > So if you could expose the lost NMI stats via procfs or debugfs then we could > remove both the warning and the dazed-and-confused spam on the system log. I can add this. > > This should make perf all around more usable on UV systems, right? I'm not sure this is accurate. Perf is currently very usable on UV. But as we increase our online fault analysis procedures, this warning message stood out as a glaring example of a false positive. Note it is not warning of anything except there is more than one NMI handler registering on this "call after all other handlers have been called and did not claim the NMI" chain. So let me know if I should go ahead with the above (remove some or all indications that an unclaimed NMI event occurred, and add a reporting facility for NMI stats.) Thanks! Mike