Novell Linux Kernel Debugger (NLKD)

Novell Linux Kernel Debugger (NLKD)

[Clyde Griffin]

Novell engineering is introducing the Novell Linux Kernel Debugger (NLKD) as an open source project intended to provide an enhanced and robust debugging experience for Linux kernel developers. 

The project (nlkd) at http://forge.novell.com/modules/xfmod/project/?nlkd currently has code patches to SUSE Linux Enterprise Server v9 SP1 and SP2, which when applied allow kernel developers to recompile the Linux kernel exposing the functionality provided by NLKD.  

Patches against the mainline will be forthcoming after community feedback.

At this point the intent of this project is to gather comments and suggestions on how to improve NLKD and make it possible for NLKD to be included as a native (mainline) kernel debugger for Linux.  Novell engineering will be actively involved in reviewing comments regarding NLKD and submitting and receiving patches for the support of NLKD.
 
NLKD provides an architecture for supporting a stable debugging experience as well as debug agents supporting local and remote debugging.  NLKD's Console Debug Agent (CDA) supports on-box kernel debugging and is extremely rich in functionality and easy to use.  The Remote Debug Agent (RDA) supports remote source level debugging via gdbtransport to your favorite gdb based client.  Both agents benefit from the Core Debug Engine (CDE), which provides the state machine logic supporting the debug agents.

A detailed description of the architecture will be presented in a paper titled "The Novell Linux Kernel Debugger, NLKD" at the Ottawa Linux Symposium July 20-23, 2005.

NLKD provides debugging capabilities that will be a benefit to the whole development community and we invite developers interested in using NLKD to download the patches, provide feedback, and benefit from the use of this great tool.

Developers interested in commenting on, contributing to, testing, documenting, and improving NLKD should subscribe to the nlkd-dev mailing list at http://forge.novell.com/modules/xfmod/project/?nlkd and post to nlkd-dev@forge.novell.com.

Re: Novell Linux Kernel Debugger (NLKD)

[Parag Warudkar]

On Thursday 23 June 2005 18:54, Clyde Griffin wrote:
> At this point the intent of this project is to gather comments and
> suggestions on how to improve NLKD and make it possible for NLKD to be
> included as a native (mainline) kernel debugger for Linux.  Novell
> engineering will be actively involved in reviewing comments regarding NLKD
> and submitting and receiving patches for the support of NLKD.

Does / Will it support remote debugging over firewire ports? I think this 
would  be a great opportunity to get rid of the serial port requirement for 
debugging.

Parag

Re: Novell Linux Kernel Debugger (NLKD)

[Greg KH]

On Thu, Jun 23, 2005 at 04:54:09PM -0600, Clyde Griffin wrote:
> 
> Patches against the mainline will be forthcoming after community feedback.

Please provide patches against a mainline kernel, otherwise it's pretty
much impossible to review.

And try to reduce the ammount of #ifdev CONFIG_CDE you have placed in
your patch, it's quite excessive and against the kernel coding style
rules.

thanks,

greg k-h

Re: Novell Linux Kernel Debugger (NLKD)

[Christoph Lameter]

On Thu, 23 Jun 2005, Clyde Griffin wrote:

> Novell engineering is introducing the Novell Linux Kernel Debugger 
> (NLKD) as an open source project intended to provide an enhanced and 
> robust debugging experience for Linux kernel developers.

Umm... How does this differ from KDB?

Re: Novell Linux Kernel Debugger (NLKD)

[John W. Linville]

On Thu, Jun 23, 2005 at 05:24:04PM -0700, Christoph Lameter wrote:
> On Thu, 23 Jun 2005, Clyde Griffin wrote:
> 
> > Novell engineering is introducing the Novell Linux Kernel Debugger 
> > (NLKD) as an open source project intended to provide an enhanced and 
> > robust debugging experience for Linux kernel developers.
> 
> Umm... How does this differ from KDB?

Sounds like it overlaps w/ both KDB and KGDB...still, the question
stands...

John
-- 
John W. Linville
linville@tuxdriver.com

Re: Novell Linux Kernel Debugger (NLKD)

[jmerkey]

I have downloaded and reviewed the code. It's a GBD replacement and is 
not fully open source.
KDB is at present more capable. It has a lot of promise, but it does not 
have the all the architectural
features necessary to replace either KDB or GDB at present.

Jeff

>On Thu, Jun 23, 2005 at 05:24:04PM -0700, Christoph Lameter wrote:
>  
>
>>On Thu, 23 Jun 2005, Clyde Griffin wrote:
>>
>>    
>>
>>>Novell engineering is introducing the Novell Linux Kernel Debugger 
>>>(NLKD) as an open source project intended to provide an enhanced and 
>>>robust debugging experience for Linux kernel developers.
>>>      
>>>
>>Umm... How does this differ from KDB?
>>    
>>
>
>Sounds like it overlaps w/ both KDB and KGDB...still, the question
>stands...
>
>John
>  
>

Re: Novell Linux Kernel Debugger (NLKD)

[Rik Van Riel]

On Thu, 23 Jun 2005, jmerkey wrote:

> I have downloaded and reviewed the code. It's a GBD replacement and is 
> not fully open source.

Thanks for that info.  Guess I really won't have to bother
looking at it any more now ;)

I'll go back to waiting for systemtap...

-- 
The Theory of Escalating Commitment: "The cost of continuing mistakes is
borne by others, while the cost of admitting mistakes is borne by yourself."
  -- Joseph Stiglitz, Nobel Laureate in Economics

Re: Novell Linux Kernel Debugger (NLKD)

[jmerkey]

Rik Van Riel wrote:

>On Thu, 23 Jun 2005, jmerkey wrote:
>
>  
>
>>I have downloaded and reviewed the code. It's a GBD replacement and is 
>>not fully open source.
>>    
>>
>
>Thanks for that info.  Guess I really won't have to bother
>looking at it any more now ;)
>
>I'll go back to waiting for systemtap...
>
>  
>
Rik,

You are most welcome. You should encourage Clyde though to keep working 
to convince Novell to release all
of it open source. Clyde is a really good guy, and he knows what he's doing.

:-)

Jeff

Re: Novell Linux Kernel Debugger (NLKD)

[Jan Beulich]

>It's a GBD replacement and is not fully open source.

What is not open source in it ()?

>KDB is at present more capable. It has a lot of promise, but it does not 
>have the all the architectural
>features necessary to replace either KDB or GDB at present.

While I never used or saw kdb, I'd be curious about what you immediately saw missing...

Jan

Re: Novell Linux Kernel Debugger (NLKD)

[jmerkey]

Jan Beulich wrote:

>>It's a GBD replacement and is not fully open source.
>>    
>>
>
>What is not open source in it ()?
>
>  
>
>>KDB is at present more capable. It has a lot of promise, but it does not 
>>have the all the architectural
>>features necessary to replace either KDB or GDB at present.
>>    
>>
>
>While I never used or saw kdb, I'd be curious about what you immediately saw missing...
>  
>

1. No back trace
2. Doesn't run standalone fully embeded in the kernel
3. Not fully open source (since it's not embeded in the kernel)
4. IA64 doesn't really matter, since IA64 is basically dead anyway
5. No advanced recursive descent parser for conditional breakpoints

Jeff

>Jan
>
>
>  
>

Re: Novell Linux Kernel Debugger (NLKD)

[Chris Mason]

On Friday 24 June 2005 00:59, jmerkey wrote:
> Jan Beulich wrote:
> >>It's a GBD replacement and is not fully open source.
> >
> >What is not open source in it ()?
> >
> >>KDB is at present more capable. It has a lot of promise, but it does not
> >>have the all the architectural
> >>features necessary to replace either KDB or GDB at present.
> >
> >While I never used or saw kdb, I'd be curious about what you immediately
> > saw missing...
>
> 1. No back trace
> 2. Doesn't run standalone fully embeded in the kernel
> 3. Not fully open source (since it's not embeded in the kernel)
> 4. IA64 doesn't really matter, since IA64 is basically dead anyway
> 5. No advanced recursive descent parser for conditional breakpoints

This is more or less completely inaccurate.

-chris

Re: Novell Linux Kernel Debugger (NLKD)

[jmerkey]

Chris Mason wrote:

>On Friday 24 June 2005 00:59, jmerkey wrote:
>  
>
>>Jan Beulich wrote:
>>    
>>
>>>>It's a GBD replacement and is not fully open source.
>>>>        
>>>>
>>>What is not open source in it ()?
>>>
>>>      
>>>
>>>>KDB is at present more capable. It has a lot of promise, but it does not
>>>>have the all the architectural
>>>>features necessary to replace either KDB or GDB at present.
>>>>        
>>>>
>>>While I never used or saw kdb, I'd be curious about what you immediately
>>>saw missing...
>>>      
>>>
>>1. No back trace
>>2. Doesn't run standalone fully embeded in the kernel
>>3. Not fully open source (since it's not embeded in the kernel)
>>4. IA64 doesn't really matter, since IA64 is basically dead anyway
>>5. No advanced recursive descent parser for conditional breakpoints
>>    
>>
>
>This is more or less completely inaccurate.
>
>  
>

Which one -- more ?  or less ? 

Jeff

>-chris
>
>
>
>  
>

Re: Novell Linux Kernel Debugger (NLKD)

[Chris Mason]

On Friday 24 June 2005 11:55, jmerkey wrote:

> >>1. No back trace
> >>2. Doesn't run standalone fully embeded in the kernel
> >>3. Not fully open source (since it's not embeded in the kernel)
> >>4. IA64 doesn't really matter, since IA64 is basically dead anyway
> >>5. No advanced recursive descent parser for conditional breakpoints
> >
> >This is more or less completely inaccurate.
>
> Which one -- more ?  or less ?

I think Jan answered this in detail later in the thread.

-chirs

Re: Novell Linux Kernel Debugger (NLKD)

[jmerkey]

Chris Mason wrote:

>On Friday 24 June 2005 11:55, jmerkey wrote:
>
>  
>
>>>>1. No back trace
>>>>2. Doesn't run standalone fully embeded in the kernel
>>>>3. Not fully open source (since it's not embeded in the kernel)
>>>>4. IA64 doesn't really matter, since IA64 is basically dead anyway
>>>>5. No advanced recursive descent parser for conditional breakpoints
>>>>        
>>>>
>>>This is more or less completely inaccurate.
>>>      
>>>
>>Which one -- more ?  or less ?
>>    
>>
>
>I think Jan answered this in detail later in the thread.
>
>  
>

Jan was very professional and interested and solicited suggestions, and 
got them.  :-)

Jeff

>-chirs
>
>  
>

Re: Novell Linux Kernel Debugger (NLKD)

[Lee Revell]

On Fri, 2005-06-24 at 08:21 +0200, Jan Beulich wrote:
> While I never used or saw kdb, I'd be curious about what you immediately saw missing...

Um, you guys developed your own kernel debugger without looking at KDB
first?  How would you know you're not just reinventing the wheel?

Lee

Re: Novell Linux Kernel Debugger (NLKD)

[Christoph Lameter]

On Fri, 24 Jun 2005, Lee Revell wrote:

> On Fri, 2005-06-24 at 08:21 +0200, Jan Beulich wrote:
> > While I never used or saw kdb, I'd be curious about what you immediately saw missing...
> 
> Um, you guys developed your own kernel debugger without looking at KDB
> first?  How would you know you're not just reinventing the wheel?

Isnt the name "Novell ..." already a giveway? This is some kind of 
corporate management artifact. Someone wants to do something good for the 
poor Linux community that has no debuggers available and they just got 
something from somewhere that is going nowhere (merger?). They are in the 
period clueing in to opensource. Just give them some time .....

Hint to Novell Management : You own SuSE who 
has provided KDB with their kernel for quite some time.

Re: Novell Linux Kernel Debugger (NLKD)

[Christoph Lameter]

On Thu, 23 Jun 2005, John W. Linville wrote:

> On Thu, Jun 23, 2005 at 05:24:04PM -0700, Christoph Lameter wrote:
> > On Thu, 23 Jun 2005, Clyde Griffin wrote:
> > 
> > > Novell engineering is introducing the Novell Linux Kernel Debugger 
> > > (NLKD) as an open source project intended to provide an enhanced and 
> > > robust debugging experience for Linux kernel developers.
> > 
> > Umm... How does this differ from KDB?
> 
> Sounds like it overlaps w/ both KDB and KGDB...still, the question
> stands...

I would appreciate efforts to consolidate all of these and finally get one 
into the kernel.

Re: Novell Linux Kernel Debugger (NLKD)

[Rik Van Riel]

On Thu, 23 Jun 2005, Clyde Griffin wrote:

> Patches against the mainline will be forthcoming after community feedback.

I see that the tar ball is 2.7MB in size.  This is too big to
effectively give any feedback on.  When you create patches
against the mainline kernel, please split them up into logical
pieces that are each small enough to properly review.

-- 
The Theory of Escalating Commitment: "The cost of continuing mistakes is
borne by others, while the cost of admitting mistakes is borne by yourself."
  -- Joseph Stiglitz, Nobel Laureate in Economics

Re: Novell Linux Kernel Debugger (NLKD)

[Keith Owens]

On Thu, 23 Jun 2005 16:54:09 -0600, 
"Clyde Griffin" <CGRIFFIN@novell.com> wrote:
>
>Novell engineering is introducing the Novell Linux Kernel Debugger
>(NLKD) as an open source project intended to provide an enhanced and
>robust debugging experience for Linux kernel developers.

Hmm, no backtrace, turns off kprobes, does not handle ia64 MCA/INIT.
And don't get me started about the trailing white space in the patch.
I'll stick with KDB for now.

Re: Novell Linux Kernel Debugger (NLKD)

[Pavel Machek]

Hi!

(please wrap your lines after 72 characters or so)

> Novell engineering is introducing the Novell Linux Kernel Debugger (NLKD) as an open source project intended to provide an enhanced and robust debugging experience for Linux kernel developers. 
> 
> The project (nlkd) at http://forge.novell.com/modules/xfmod/project/?nlkd currently has code patches to SUSE Linux Enterprise Server v9 SP1 and SP2, which when applied allow kernel developers to recompile the Linux kernel exposing the functionality provided by NLKD.  
> 
> Patches against the mainline will be forthcoming after community
> feedback.

gettingRidOfFunkyCapitalisation and rid of hungarian notation_t would
be a good start. (I only seen the docs, but it has some code snippets
that look very ugly).
								Pavel
-- 
teflon -- maybe it is a trademark, but it should not be.

Re: Novell Linux Kernel Debugger (NLKD)

[Jan Beulich]

>>> Keith Owens <kaos@sgi.com> 24.06.05 04:15:19 >>>
>On Thu, 23 Jun 2005 16:54:09 -0600, 
>"Clyde Griffin" <CGRIFFIN@novell.com> wrote:
>>
>>Novell engineering is introducing the Novell Linux Kernel Debugger
>>(NLKD) as an open source project intended to provide an enhanced and
>>robust debugging experience for Linux kernel developers.

>Hmm, no backtrace,

That's certainly wrong: for x86 it uses .eh_frame info, for ia64 it would use the native one.

>turns off kprobes,

A temporary measure; enabling this again is on the todo list.

>does not handle ia64 MCA/INIT.

It doesn't handle ia64 at all at present; the code in there is yet-to-be-ported code. Only x86{,-64} has been dealt with so far.

>And don't get me started about the trailing white space in the patch.

While I wasn't aware of any (and honestly, I hate discovering such, too), that is certainly the easiest thing to take care of.

>I'll stick with KDB for now.

Your choice, obviously.

Jan

Re: Novell Linux Kernel Debugger (NLKD)

[Jan Beulich]

>1. No back trace

You're the second one to mention this, and I wonder where you take this from.

>2. Doesn't run standalone fully embeded in the kernel

What do you mean with 'fully embedded'? As long as the agents aren't compiled as modules, everything's right in the kernel (of course, not the mainline one, but that's same as for kdb and kgdb).

>3. Not fully open source (since it's not embeded in the kernel)

What has embedding things in the kernel to do with being open source?

>5. No advanced recursive descent parser for conditional breakpoints

Where've you seen this? What functionality does the parser miss?

Jan

Re: Novell Linux Kernel Debugger (NLKD)

[jmerkey]

Jan Beulich wrote:

>>1. No back trace
>>    
>>
>
>You're the second one to mention this, and I wonder where you take this from.
>  
>
Check the version binaries on NovellForge. Didn't work. Suse Linux perhaps?

>  
>
>>2. Doesn't run standalone fully embeded in the kernel
>>    
>>
>
>What do you mean with 'fully embedded'? As long as the agents aren't compiled as modules, everything's right in the kernel (of course, not the mainline one, but that's same as for kdb and kgdb).
>  
>

Right.

>  
>
>>3. Not fully open source (since it's not embeded in the kernel)
>>    
>>
>
>What has embedding things in the kernel to do with being open source?
>  
>

Debuggers need to be open source to work on Linux (socially work, not 
technically). They also need to use kdb
and gdb as a base, so you can have one patch that runs on all the kdb 
versions for the kernels. Then you can use kdb
and let it do all the interfacing beneath you and leverage KDB and the 
porting effort there. Hook kdb() and traps.c
for events to userspace.


>  
>
>>5. No advanced recursive descent parser for conditional breakpoints
>>    
>>
>
>Where've you seen this? What functionality does the parser miss?
>  
>

Your architecture is event based, but there's no source code to change 
this, just a white paper. How do I know a conditional
breakpoint is running real time when it triggers. No source code, no 
interest.... Social Problem.

There is ****NO**** commerical debugger business on Linux. I don't agree 
that the GPL contaminates most
kernel modules, but a kernel debugger is one exception, because it has 
to include key kernel header files and truly is
an "integrated" component (at least a decent one is).

If you want to promote an interesting project and make no money off of 
it, open source your debugger, put it on KDB
as a base, and lots of folks will use it. We tried the whole kernel 
debugger product thing, and there are actual legal
issues if you choose not to open source it. File Systems, drivers, etc. 
-- not the case. Kernel debugger is the case.

:-)

Jeff


>Jan
>
>
>  
>

Re: Novell Linux Kernel Debugger (NLKD)

[Keith Owens]

On Fri, 24 Jun 2005 09:38:41 +0200, 
"Jan Beulich" <JBeulich@novell.com> wrote:
>>1. No back trace
>
>You're the second one to mention this, and I wonder where you take this from.

Probably from this line in your patch.

debugger-2.6.5-7.183.patch:+//todo?   COMMAND(bt),

Which is misleading.  There is some backtrace code, under F12, where it
is called Stack Trace.

Re: Novell Linux Kernel Debugger (NLKD)

[Rik Van Riel]

On Fri, 24 Jun 2005, Keith Owens wrote:

> Which is misleading.  There is some backtrace code, under F12, where it 
> is called Stack Trace.

It's using function keys?   Do those work over serial console?

-- 
The Theory of Escalating Commitment: "The cost of continuing mistakes is
borne by others, while the cost of admitting mistakes is borne by yourself."
  -- Joseph Stiglitz, Nobel Laureate in Economics

Re: Novell Linux Kernel Debugger (NLKD)

[Jan Beulich]

>>> Rik Van Riel <riel@redhat.com> 24.06.05 14:02:16 >>>
>On Fri, 24 Jun 2005, Keith Owens wrote:
>
>> Which is misleading.  There is some backtrace code, under F12, where it 
>> is called Stack Trace.
>
>It's using function keys?   Do those work over serial console?

It's not "the" debugger that's using function keys. The console frontend (one of the two currently available agents) is, the serial interface (the other available agent) obviously isn't, but that one doesn't know about keys at all. The console one in turn isn't intended to be run over a serial console...

Jan

Re: Novell Linux Kernel Debugger (NLKD)

[Clyde Griffin]

>>>> Parag Warudkar <kernel-stuff@comcast.net> 6/23/2005 3:14:26 PM >>>
>On Thursday 23 June 2005 18:54, Clyde Griffin wrote:
>> At this point the intent of this project is to gather comments and
>> suggestions on how to improve NLKD and make it possible for NLKD to be
>> included as a native (mainline) kernel debugger for Linux.  Novell
>> engineering will be actively involved in reviewing comments regarding NLKD
>> and submitting and receiving patches for the support of NLKD.
>
>Does / Will it support remote debugging over firewire ports? I think this 
>would  be a great opportunity to get rid of the serial port requirement for 
>debugging.

NLKD's architecture supports pluggable remote agents that talk directly
to hardware.  For remote debugging we currently we only have support 
for the serial port, but supporting other device types should be
fairly straightforward and doing so is on the todo list.

The main effort it is in writing the driver which must operate outside 
kernel context.  As we start to see support for additional hardware in remote 
debug agents we may spend the time to further abstract the remote 
debug protocol stack support away from the hardware driver.  

This architecture will be explained in our OLS paper.

In the mean time someone could simply take the RDA serial port code
and replace it with support for other hardware (NIC or whatever) and get 
the desired results.  

Again this sort of thing is on our todo list and we would really appreciate 
help from persons familiar with writing drivers for various device types 
to help jump start this.

Clyde

>
>Parag