All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v2] Add hard/soft lockup debugger entry points
@ 2016-01-26 17:04 Jeff Merkey
  2016-01-26 17:22 ` Chris Metcalf
                   ` (2 more replies)
  0 siblings, 3 replies; 6+ messages in thread
From: Jeff Merkey @ 2016-01-26 17:04 UTC (permalink / raw)
  To: linux-kernel
  Cc: Andrew Morton, Don Zickus, Ulrich Obergfell, Aaron Tomlin,
	Michal Hocko, Frederic Weisbecker, Tejun Heo, Hidehiro Kawai,
	Chris Metcalf

This patch adds an export which can be set by system debuggers to direct
the hard lockup and soft lockup detector to trigger a breakpoint exception
and enter a debugger if one is active.  It is assumed that if someone
sets this variable, then an breakpoint handler of some sort will be actively
loaded or registered via the notify die handler chain.

This addition is extremely useful for debugging hard and soft lockups
real time and quickly from a console debugger.

Signed-off-by: Jeff Merkey <linux.mdb@gmail.com>
---
 kernel/watchdog.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index b3ace6e..c28e58c 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -23,6 +23,7 @@
 #include <linux/workqueue.h>
 
 #include <asm/irq_regs.h>
+#include <linux/kgdb.h>
 #include <linux/kvm_para.h>
 #include <linux/perf_event.h>
 #include <linux/kthread.h>
@@ -108,6 +109,9 @@ static DEFINE_PER_CPU(struct perf_event *, watchdog_ev);
 #endif
 static unsigned long soft_lockup_nmi_warn;
 
+int debug_watchdog_lockups;
+EXPORT_SYMBOL_GPL(debug_watchdog_lockups);
+
 /* boot commands */
 /*
  * Should we panic when a soft-lockup or hard-lockup occurs:
@@ -358,6 +362,9 @@ static void watchdog_overflow_callback(struct perf_event *event,
 		else
 			dump_stack();
 
+		if (debug_watchdog_lockups)
+			arch_kgdb_breakpoint();
+
 		/*
 		 * Perform all-CPU dump only once to avoid multiple hardlockups
 		 * generating interleaving traces
@@ -478,6 +485,9 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 		else
 			dump_stack();
 
+		if (debug_watchdog_lockups)
+			arch_kgdb_breakpoint();
+
 		if (softlockup_all_cpu_backtrace) {
 			/* Avoid generating two back traces for current
 			 * given that one is already made above
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [PATCH v2] Add hard/soft lockup debugger entry points
  2016-01-26 17:04 [PATCH v2] Add hard/soft lockup debugger entry points Jeff Merkey
@ 2016-01-26 17:22 ` Chris Metcalf
  2016-01-26 17:26   ` Jeff Merkey
  2016-01-26 17:45 ` kbuild test robot
  2016-01-26 18:46 ` kbuild test robot
  2 siblings, 1 reply; 6+ messages in thread
From: Chris Metcalf @ 2016-01-26 17:22 UTC (permalink / raw)
  To: Jeff Merkey, linux-kernel
  Cc: Andrew Morton, Don Zickus, Ulrich Obergfell, Aaron Tomlin,
	Michal Hocko, Frederic Weisbecker, Tejun Heo, Hidehiro Kawai

On 01/26/2016 12:04 PM, Jeff Merkey wrote:
> This patch adds an export which can be set by system debuggers to direct
> the hard lockup and soft lockup detector to trigger a breakpoint exception
> and enter a debugger if one is active.  It is assumed that if someone
> sets this variable, then an breakpoint handler of some sort will be actively
> loaded or registered via the notify die handler chain.
>
> This addition is extremely useful for debugging hard and soft lockups
> real time and quickly from a console debugger.
>
> Signed-off-by: Jeff Merkey<linux.mdb@gmail.com>
> ---
>   kernel/watchdog.c | 10 ++++++++++
>   1 file changed, 10 insertions(+)

You probably should reach out to someone who uses this stuff more
regularly - I actually wonder if the kgdb_breakpoint() API is the
right thing, though, not the internal arch_kgdb_breakpoint().

Of course any of these strategies also assume you are building
the kernel with CONFIG_KGDB set, and I'm pretty sure will cause
your build to fail if it isn't.  You likely need to guard this
stuff locally within watchdog.c for !CONFIG_KGDB.

-- 
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH v2] Add hard/soft lockup debugger entry points
  2016-01-26 17:22 ` Chris Metcalf
@ 2016-01-26 17:26   ` Jeff Merkey
  0 siblings, 0 replies; 6+ messages in thread
From: Jeff Merkey @ 2016-01-26 17:26 UTC (permalink / raw)
  To: Chris Metcalf
  Cc: linux-kernel, Andrew Morton, Don Zickus, Ulrich Obergfell,
	Aaron Tomlin, Michal Hocko, Frederic Weisbecker, Tejun Heo,
	Hidehiro Kawai

On 1/26/16, Chris Metcalf <cmetcalf@ezchip.com> wrote:
> On 01/26/2016 12:04 PM, Jeff Merkey wrote:
>> This patch adds an export which can be set by system debuggers to direct
>> the hard lockup and soft lockup detector to trigger a breakpoint
>> exception
>> and enter a debugger if one is active.  It is assumed that if someone
>> sets this variable, then an breakpoint handler of some sort will be
>> actively
>> loaded or registered via the notify die handler chain.
>>
>> This addition is extremely useful for debugging hard and soft lockups
>> real time and quickly from a console debugger.
>>
>> Signed-off-by: Jeff Merkey<linux.mdb@gmail.com>
>> ---
>>   kernel/watchdog.c | 10 ++++++++++
>>   1 file changed, 10 insertions(+)
>
> You probably should reach out to someone who uses this stuff more
> regularly - I actually wonder if the kgdb_breakpoint() API is the
> right thing, though, not the internal arch_kgdb_breakpoint().
>
> Of course any of these strategies also assume you are building
> the kernel with CONFIG_KGDB set, and I'm pretty sure will cause
> your build to fail if it isn't.  You likely need to guard this
> stuff locally within watchdog.c for !CONFIG_KGDB.
>
> --
> Chris Metcalf, EZChip Semiconductor
> http://www.ezchip.com
>
>

I'll go review that code carefully.  What would ideal is to have the
breakpoint abstraction independent of kgdb entirely.  That would not
be a simple patch but more involved.  We really just need the
breakpoint instruction for each arch, not all of kgdb and it should
not depend on KGB being enabled.  I looked over those includes and
this may work, but you are correct, it's not squeaky clean to do it
that way.  I'll get to work on v3.

:-)

Jeff

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH v2] Add hard/soft lockup debugger entry points
  2016-01-26 17:04 [PATCH v2] Add hard/soft lockup debugger entry points Jeff Merkey
  2016-01-26 17:22 ` Chris Metcalf
@ 2016-01-26 17:45 ` kbuild test robot
  2016-01-26 18:46 ` kbuild test robot
  2 siblings, 0 replies; 6+ messages in thread
From: kbuild test robot @ 2016-01-26 17:45 UTC (permalink / raw)
  To: Jeff Merkey
  Cc: kbuild-all, linux-kernel, Andrew Morton, Don Zickus,
	Ulrich Obergfell, Aaron Tomlin, Michal Hocko,
	Frederic Weisbecker, Tejun Heo, Hidehiro Kawai, Chris Metcalf

[-- Attachment #1: Type: text/plain, Size: 1409 bytes --]

Hi Jeff,

[auto build test ERROR on v4.5-rc1]
[also build test ERROR on next-20160125]
[if your patch is applied to the wrong git tree, please drop us a note to help improving the system]

url:    https://github.com/0day-ci/linux/commits/Jeff-Merkey/Add-hard-soft-lockup-debugger-entry-points/20160127-010930
config: xtensa-allyesconfig (attached as .config)
reproduce:
        wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=xtensa 

All errors (new ones prefixed by >>):

   kernel/watchdog.c: In function 'watchdog_timer_fn':
>> kernel/watchdog.c:489:4: error: implicit declaration of function 'arch_kgdb_breakpoint' [-Werror=implicit-function-declaration]
       arch_kgdb_breakpoint();
       ^
   cc1: some warnings being treated as errors

vim +/arch_kgdb_breakpoint +489 kernel/watchdog.c

   483			if (regs)
   484				show_regs(regs);
   485			else
   486				dump_stack();
   487	
   488			if (debug_watchdog_lockups)
 > 489				arch_kgdb_breakpoint();
   490	
   491			if (softlockup_all_cpu_backtrace) {
   492				/* Avoid generating two back traces for current

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 44066 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH v2] Add hard/soft lockup debugger entry points
  2016-01-26 17:04 [PATCH v2] Add hard/soft lockup debugger entry points Jeff Merkey
  2016-01-26 17:22 ` Chris Metcalf
  2016-01-26 17:45 ` kbuild test robot
@ 2016-01-26 18:46 ` kbuild test robot
  2016-01-26 19:33   ` Jeff Merkey
  2 siblings, 1 reply; 6+ messages in thread
From: kbuild test robot @ 2016-01-26 18:46 UTC (permalink / raw)
  To: Jeff Merkey
  Cc: kbuild-all, linux-kernel, Andrew Morton, Don Zickus,
	Ulrich Obergfell, Aaron Tomlin, Michal Hocko,
	Frederic Weisbecker, Tejun Heo, Hidehiro Kawai, Chris Metcalf

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

Hi Jeff,

[auto build test ERROR on v4.5-rc1]
[also build test ERROR on next-20160125]
[if your patch is applied to the wrong git tree, please drop us a note to help improving the system]

url:    https://github.com/0day-ci/linux/commits/Jeff-Merkey/Add-hard-soft-lockup-debugger-entry-points/20160127-010930
config: sparc64-defconfig (attached as .config)
reproduce:
        wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=sparc64 

All errors (new ones prefixed by >>):

   kernel/built-in.o: In function `watchdog_timer_fn':
>> watchdog.c:(.text+0x736ec): undefined reference to `arch_kgdb_breakpoint'

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 16661 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH v2] Add hard/soft lockup debugger entry points
  2016-01-26 18:46 ` kbuild test robot
@ 2016-01-26 19:33   ` Jeff Merkey
  0 siblings, 0 replies; 6+ messages in thread
From: Jeff Merkey @ 2016-01-26 19:33 UTC (permalink / raw)
  To: kbuild test robot
  Cc: kbuild-all, linux-kernel, Andrew Morton, Don Zickus,
	Ulrich Obergfell, Aaron Tomlin, Michal Hocko,
	Frederic Weisbecker, Tejun Heo, Hidehiro Kawai, Chris Metcalf

On 1/26/16, kbuild test robot <lkp@intel.com> wrote:
> Hi Jeff,
>
> [auto build test ERROR on v4.5-rc1]
> [also build test ERROR on next-20160125]
> [if your patch is applied to the wrong git tree, please drop us a note to
> help improving the system]
>
> url:
> https://github.com/0day-ci/linux/commits/Jeff-Merkey/Add-hard-soft-lockup-debugger-entry-points/20160127-010930
> config: sparc64-defconfig (attached as .config)
> reproduce:
>         wget
> https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross
> -O ~/bin/make.cross
>         chmod +x ~/bin/make.cross
>         # save the attached .config to linux build tree
>         make.cross ARCH=sparc64
>
> All errors (new ones prefixed by >>):
>
>    kernel/built-in.o: In function `watchdog_timer_fn':
>>> watchdog.c:(.text+0x736ec): undefined reference to
>>> `arch_kgdb_breakpoint'
>
> ---
> 0-DAY kernel test infrastructure                Open Source Technology
> Center
> https://lists.01.org/pipermail/kbuild-all                   Intel
> Corporation
>

I love this test robot.  What an incredibly useful tool.  Well, looks
like a I quite a bit of work to do.

Jeff

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2016-01-26 19:33 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-01-26 17:04 [PATCH v2] Add hard/soft lockup debugger entry points Jeff Merkey
2016-01-26 17:22 ` Chris Metcalf
2016-01-26 17:26   ` Jeff Merkey
2016-01-26 17:45 ` kbuild test robot
2016-01-26 18:46 ` kbuild test robot
2016-01-26 19:33   ` Jeff Merkey

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.