Linux Kernel GDB tracepoint module demo release

Linux Kernel GDB tracepoint module demo release

[Hui Zhu]

Hi,

GDB Tracepoint (http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html)
is new feature that make GDB can debug real-time program.
I think this feature is very fit for debug Linux Kernel.  So I make a
lkm to make kernel support it. You can get it in
https://code.google.com/p/kgtp/

I just release the demo for it.  It just support collect $reg in
x86_32.  I will add the other tracepoint support in the future.

For example:
tar vxjf gtp_20100811.tar.bz2
cd gtp
make
insmod gtp.ko
gdb ./vmlinux
GNU gdb (GDB) 7.2.50.20100720-cvs
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /vmlinux...done.
(gdb) target remote /proc/gtp
Remote debugging using /proc/gtp
0x00000000 in ?? ()
(gdb) trace vfs_readdir
Tracepoint 1 at 0xc01a1ac0: file
/home/teawater/kernel/linux-2.6/fs/readdir.c, line 23.
(gdb) actions
Enter actions for tracepoint 1, one per line.
End with a line saying just "end".
>collect $reg
>end
(gdb) tstart
(gdb) shell ls
(gdb) tstop
(gdb) tfind
Found trace frame 0, tracepoint 1
#0  0xc01a1ac1 in vfs_readdir (file=0xc5528d00, filler=0xc01a1900 <filldir64>,
    buf=0xc0d09f90) at /home/teawater/kernel/linux-2.6/fs/readdir.c:23
23	/home/teawater/kernel/linux-2.6/fs/readdir.c: No such file or directory.
	in /home/teawater/kernel/linux-2.6/fs/readdir.c
(gdb) info reg
eax            0xc5528d00	-984445696
ecx            0xc0d09f90	-1060069488
edx            0xc01a1900	-1072031488
ebx            0xfffffff7	-9
esp            0xc0d09f8c	0xc0d09f8c
ebp            0x0	0x0
esi            0x8061480	134616192
edi            0xc5528d00	-984445696
eip            0xc01a1ac1	0xc01a1ac1 <vfs_readdir+1>
eflags         0x286	[ PF SF IF ]
cs             0x60	96
ss             0x8061480	134616192
ds             0x7b	123
es             0x7b	123
fs             0x0	0
gs             0x0	0
(gdb) tfind
Found trace frame 1, tracepoint 1
0xc01a1ac1	23	in /home/teawater/kernel/linux-2.6/fs/readdir.c
(gdb) info reg
eax            0xc5528d00	-984445696
ecx            0xc0d09f90	-1060069488
edx            0xc01a1900	-1072031488
ebx            0xfffffff7	-9
esp            0xc0d09f8c	0xc0d09f8c
ebp            0x0	0x0
esi            0x8061480	134616192
edi            0xc5528d00	-984445696
eip            0xc01a1ac1	0xc01a1ac1 <vfs_readdir+1>
eflags         0x286	[ PF SF IF ]
cs             0x60	96
ss             0x8061480	134616192
ds             0x7b	123
es             0x7b	123
fs             0x0	0
gs             0x0	0
(gdb) tfind
Found trace frame 2, tracepoint 1
#0  0xc01a1ac1 in vfs_readdir (file=0xc5528300, filler=0xc01a19e0 <filldir>,
    buf=0xc6c2df90) at /home/teawater/kernel/linux-2.6/fs/readdir.c:23
23	in /home/teawater/kernel/linux-2.6/fs/readdir.c
(gdb)
Found trace frame 3, tracepoint 1
0xc01a1ac1	23	in /home/teawater/kernel/linux-2.6/fs/readdir.c
(gdb)
Target failed to find requested trace frame.
(gdb) quit
A debugging session is active.

	Inferior 1 [Remote target] will be killed.

Quit anyway? (y or n) y

Thanks,
Hui

Re: Linux Kernel GDB tracepoint module demo release

[Thiago Jung Bauermann]

Hi Hui,

On Thu, 2010-08-12 at 00:34 +0800, Hui Zhu wrote:
> GDB Tracepoint (http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html)
> is new feature that make GDB can debug real-time program.
> I think this feature is very fit for debug Linux Kernel.  So I make a
> lkm to make kernel support it. You can get it in
> https://code.google.com/p/kgtp/

Jim Blandy worked on GDB tracepoint support for the kernel too. I don't
know how his work compares to yours, you might be interested in having a
look:

http://www.red-bean.com/trac/tracepoints/

-- 
[]'s
Thiago Jung Bauermann

Re: Linux Kernel GDB tracepoint module demo release

[Michael Snyder]

Thiago Jung Bauermann wrote:
> Hi Hui,
> 
> On Thu, 2010-08-12 at 00:34 +0800, Hui Zhu wrote:
>> GDB Tracepoint (http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html)
>> is new feature that make GDB can debug real-time program.
>> I think this feature is very fit for debug Linux Kernel.  So I make a
>> lkm to make kernel support it. You can get it in
>> https://code.google.com/p/kgtp/
> 
> Jim Blandy worked on GDB tracepoint support for the kernel too. I don't
> know how his work compares to yours, you might be interested in having a
> look:
> 
> http://www.red-bean.com/trac/tracepoints/
> 

There's also the LTTng project: http://lttng.org/

Re: Linux Kernel GDB tracepoint module demo release

[Michael Snyder]

Actually, this is kind of interesting.
Can your system set a tracepoint at any arbitrary location in the
kernel?  If so, I think that makes it more powerful than existing
systems such as LTTng, which can only trace statically pre-determined
locations.

Hui Zhu wrote:
> Hi,
> 
> GDB Tracepoint (http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html)
> is new feature that make GDB can debug real-time program.
> I think this feature is very fit for debug Linux Kernel.  So I make a
> lkm to make kernel support it. You can get it in
> https://code.google.com/p/kgtp/
> 
> I just release the demo for it.  It just support collect $reg in
> x86_32.  I will add the other tracepoint support in the future.
> 
> For example:
> tar vxjf gtp_20100811.tar.bz2
> cd gtp
> make
> insmod gtp.ko
> gdb ./vmlinux
> GNU gdb (GDB) 7.2.50.20100720-cvs
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "i686-pc-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /vmlinux...done.
> (gdb) target remote /proc/gtp
> Remote debugging using /proc/gtp
> 0x00000000 in ?? ()
> (gdb) trace vfs_readdir
> Tracepoint 1 at 0xc01a1ac0: file
> /home/teawater/kernel/linux-2.6/fs/readdir.c, line 23.
> (gdb) actions
> Enter actions for tracepoint 1, one per line.
> End with a line saying just "end".
>> collect $reg
>> end
> (gdb) tstart
> (gdb) shell ls
> (gdb) tstop
> (gdb) tfind
> Found trace frame 0, tracepoint 1
> #0  0xc01a1ac1 in vfs_readdir (file=0xc5528d00, filler=0xc01a1900 <filldir64>,
>     buf=0xc0d09f90) at /home/teawater/kernel/linux-2.6/fs/readdir.c:23
> 23	/home/teawater/kernel/linux-2.6/fs/readdir.c: No such file or directory.
> 	in /home/teawater/kernel/linux-2.6/fs/readdir.c
> (gdb) info reg
> eax            0xc5528d00	-984445696
> ecx            0xc0d09f90	-1060069488
> edx            0xc01a1900	-1072031488
> ebx            0xfffffff7	-9
> esp            0xc0d09f8c	0xc0d09f8c
> ebp            0x0	0x0
> esi            0x8061480	134616192
> edi            0xc5528d00	-984445696
> eip            0xc01a1ac1	0xc01a1ac1 <vfs_readdir+1>
> eflags         0x286	[ PF SF IF ]
> cs             0x60	96
> ss             0x8061480	134616192
> ds             0x7b	123
> es             0x7b	123
> fs             0x0	0
> gs             0x0	0
> (gdb) tfind
> Found trace frame 1, tracepoint 1
> 0xc01a1ac1	23	in /home/teawater/kernel/linux-2.6/fs/readdir.c
> (gdb) info reg
> eax            0xc5528d00	-984445696
> ecx            0xc0d09f90	-1060069488
> edx            0xc01a1900	-1072031488
> ebx            0xfffffff7	-9
> esp            0xc0d09f8c	0xc0d09f8c
> ebp            0x0	0x0
> esi            0x8061480	134616192
> edi            0xc5528d00	-984445696
> eip            0xc01a1ac1	0xc01a1ac1 <vfs_readdir+1>
> eflags         0x286	[ PF SF IF ]
> cs             0x60	96
> ss             0x8061480	134616192
> ds             0x7b	123
> es             0x7b	123
> fs             0x0	0
> gs             0x0	0
> (gdb) tfind
> Found trace frame 2, tracepoint 1
> #0  0xc01a1ac1 in vfs_readdir (file=0xc5528300, filler=0xc01a19e0 <filldir>,
>     buf=0xc6c2df90) at /home/teawater/kernel/linux-2.6/fs/readdir.c:23
> 23	in /home/teawater/kernel/linux-2.6/fs/readdir.c
> (gdb)
> Found trace frame 3, tracepoint 1
> 0xc01a1ac1	23	in /home/teawater/kernel/linux-2.6/fs/readdir.c
> (gdb)
> Target failed to find requested trace frame.
> (gdb) quit
> A debugging session is active.
> 
> 	Inferior 1 [Remote target] will be killed.
> 
> Quit anyway? (y or n) y
> 
> Thanks,
> Hui

Re: Linux Kernel GDB tracepoint module demo release

[Hui Zhu]

Thanks Thiago and Michael, both of them have some limit.  I try to
make gdb tracepoint and kernel work together directly. :)

Thanks,
Hui


On Thu, Aug 12, 2010 at 03:06, Michael Snyder <msnyder@vmware.com> wrote:
> Thiago Jung Bauermann wrote:
>>
>> Hi Hui,
>>
>> On Thu, 2010-08-12 at 00:34 +0800, Hui Zhu wrote:
>>>
>>> GDB Tracepoint
>>> (http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html)
>>> is new feature that make GDB can debug real-time program.
>>> I think this feature is very fit for debug Linux Kernel.  So I make a
>>> lkm to make kernel support it. You can get it in
>>> https://code.google.com/p/kgtp/
>>
>> Jim Blandy worked on GDB tracepoint support for the kernel too. I don't
>> know how his work compares to yours, you might be interested in having a
>> look:
>>
>> http://www.red-bean.com/trac/tracepoints/
>>
>
> There's also the LTTng project: http://lttng.org/
>
>
>

Re: Linux Kernel GDB tracepoint module demo release

[Hui Zhu]

KGTP use Kprobes, I think it can work with most part of kernel.

Thanks,
Hui

On Thu, Aug 12, 2010 at 05:10, Michael Snyder <msnyder@vmware.com> wrote:
> Actually, this is kind of interesting.
> Can your system set a tracepoint at any arbitrary location in the
> kernel?  If so, I think that makes it more powerful than existing
> systems such as LTTng, which can only trace statically pre-determined
> locations.
>
> Hui Zhu wrote:
>>
>> Hi,
>>
>> GDB Tracepoint
>> (http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html)
>> is new feature that make GDB can debug real-time program.
>> I think this feature is very fit for debug Linux Kernel.  So I make a
>> lkm to make kernel support it. You can get it in
>> https://code.google.com/p/kgtp/
>>
>> I just release the demo for it.  It just support collect $reg in
>> x86_32.  I will add the other tracepoint support in the future.
>>
>> For example:
>> tar vxjf gtp_20100811.tar.bz2
>> cd gtp
>> make
>> insmod gtp.ko
>> gdb ./vmlinux
>> GNU gdb (GDB) 7.2.50.20100720-cvs
>> Copyright (C) 2010 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "i686-pc-linux-gnu".
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>...
>> Reading symbols from /vmlinux...done.
>> (gdb) target remote /proc/gtp
>> Remote debugging using /proc/gtp
>> 0x00000000 in ?? ()
>> (gdb) trace vfs_readdir
>> Tracepoint 1 at 0xc01a1ac0: file
>> /home/teawater/kernel/linux-2.6/fs/readdir.c, line 23.
>> (gdb) actions
>> Enter actions for tracepoint 1, one per line.
>> End with a line saying just "end".
>>>
>>> collect $reg
>>> end
>>
>> (gdb) tstart
>> (gdb) shell ls
>> (gdb) tstop
>> (gdb) tfind
>> Found trace frame 0, tracepoint 1
>> #0  0xc01a1ac1 in vfs_readdir (file=0xc5528d00, filler=0xc01a1900
>> <filldir64>,
>>    buf=0xc0d09f90) at /home/teawater/kernel/linux-2.6/fs/readdir.c:23
>> 23      /home/teawater/kernel/linux-2.6/fs/readdir.c: No such file or
>> directory.
>>        in /home/teawater/kernel/linux-2.6/fs/readdir.c
>> (gdb) info reg
>> eax            0xc5528d00       -984445696
>> ecx            0xc0d09f90       -1060069488
>> edx            0xc01a1900       -1072031488
>> ebx            0xfffffff7       -9
>> esp            0xc0d09f8c       0xc0d09f8c
>> ebp            0x0      0x0
>> esi            0x8061480        134616192
>> edi            0xc5528d00       -984445696
>> eip            0xc01a1ac1       0xc01a1ac1 <vfs_readdir+1>
>> eflags         0x286    [ PF SF IF ]
>> cs             0x60     96
>> ss             0x8061480        134616192
>> ds             0x7b     123
>> es             0x7b     123
>> fs             0x0      0
>> gs             0x0      0
>> (gdb) tfind
>> Found trace frame 1, tracepoint 1
>> 0xc01a1ac1      23      in /home/teawater/kernel/linux-2.6/fs/readdir.c
>> (gdb) info reg
>> eax            0xc5528d00       -984445696
>> ecx            0xc0d09f90       -1060069488
>> edx            0xc01a1900       -1072031488
>> ebx            0xfffffff7       -9
>> esp            0xc0d09f8c       0xc0d09f8c
>> ebp            0x0      0x0
>> esi            0x8061480        134616192
>> edi            0xc5528d00       -984445696
>> eip            0xc01a1ac1       0xc01a1ac1 <vfs_readdir+1>
>> eflags         0x286    [ PF SF IF ]
>> cs             0x60     96
>> ss             0x8061480        134616192
>> ds             0x7b     123
>> es             0x7b     123
>> fs             0x0      0
>> gs             0x0      0
>> (gdb) tfind
>> Found trace frame 2, tracepoint 1
>> #0  0xc01a1ac1 in vfs_readdir (file=0xc5528300, filler=0xc01a19e0
>> <filldir>,
>>    buf=0xc6c2df90) at /home/teawater/kernel/linux-2.6/fs/readdir.c:23
>> 23      in /home/teawater/kernel/linux-2.6/fs/readdir.c
>> (gdb)
>> Found trace frame 3, tracepoint 1
>> 0xc01a1ac1      23      in /home/teawater/kernel/linux-2.6/fs/readdir.c
>> (gdb)
>> Target failed to find requested trace frame.
>> (gdb) quit
>> A debugging session is active.
>>
>>        Inferior 1 [Remote target] will be killed.
>>
>> Quit anyway? (y or n) y
>>
>> Thanks,
>> Hui
>
>

Re: Linux Kernel GDB tracepoint module demo release

[Michael Snyder]

Very interesting then.  I'd like to see you continue this work.


Hui Zhu wrote:
> KGTP use Kprobes, I think it can work with most part of kernel.
> 
> Thanks,
> Hui
> 
> On Thu, Aug 12, 2010 at 05:10, Michael Snyder <msnyder@vmware.com> wrote:
>> Actually, this is kind of interesting.
>> Can your system set a tracepoint at any arbitrary location in the
>> kernel?  If so, I think that makes it more powerful than existing
>> systems such as LTTng, which can only trace statically pre-determined
>> locations.
>>
>> Hui Zhu wrote:
>>> Hi,
>>>
>>> GDB Tracepoint
>>> (http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html)
>>> is new feature that make GDB can debug real-time program.
>>> I think this feature is very fit for debug Linux Kernel.  So I make a
>>> lkm to make kernel support it. You can get it in
>>> https://code.google.com/p/kgtp/
>>>
>>> I just release the demo for it.  It just support collect $reg in
>>> x86_32.  I will add the other tracepoint support in the future.
>>>
>>> For example:
>>> tar vxjf gtp_20100811.tar.bz2
>>> cd gtp
>>> make
>>> insmod gtp.ko
>>> gdb ./vmlinux
>>> GNU gdb (GDB) 7.2.50.20100720-cvs
>>> Copyright (C) 2010 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later
>>> <http://gnu.org/licenses/gpl.html>
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>>> and "show warranty" for details.
>>> This GDB was configured as "i686-pc-linux-gnu".
>>> For bug reporting instructions, please see:
>>> <http://www.gnu.org/software/gdb/bugs/>...
>>> Reading symbols from /vmlinux...done.
>>> (gdb) target remote /proc/gtp
>>> Remote debugging using /proc/gtp
>>> 0x00000000 in ?? ()
>>> (gdb) trace vfs_readdir
>>> Tracepoint 1 at 0xc01a1ac0: file
>>> /home/teawater/kernel/linux-2.6/fs/readdir.c, line 23.
>>> (gdb) actions
>>> Enter actions for tracepoint 1, one per line.
>>> End with a line saying just "end".
>>>> collect $reg
>>>> end
>>> (gdb) tstart
>>> (gdb) shell ls
>>> (gdb) tstop
>>> (gdb) tfind
>>> Found trace frame 0, tracepoint 1
>>> #0  0xc01a1ac1 in vfs_readdir (file=0xc5528d00, filler=0xc01a1900
>>> <filldir64>,
>>>    buf=0xc0d09f90) at /home/teawater/kernel/linux-2.6/fs/readdir.c:23
>>> 23      /home/teawater/kernel/linux-2.6/fs/readdir.c: No such file or
>>> directory.
>>>        in /home/teawater/kernel/linux-2.6/fs/readdir.c
>>> (gdb) info reg
>>> eax            0xc5528d00       -984445696
>>> ecx            0xc0d09f90       -1060069488
>>> edx            0xc01a1900       -1072031488
>>> ebx            0xfffffff7       -9
>>> esp            0xc0d09f8c       0xc0d09f8c
>>> ebp            0x0      0x0
>>> esi            0x8061480        134616192
>>> edi            0xc5528d00       -984445696
>>> eip            0xc01a1ac1       0xc01a1ac1 <vfs_readdir+1>
>>> eflags         0x286    [ PF SF IF ]
>>> cs             0x60     96
>>> ss             0x8061480        134616192
>>> ds             0x7b     123
>>> es             0x7b     123
>>> fs             0x0      0
>>> gs             0x0      0
>>> (gdb) tfind
>>> Found trace frame 1, tracepoint 1
>>> 0xc01a1ac1      23      in /home/teawater/kernel/linux-2.6/fs/readdir.c
>>> (gdb) info reg
>>> eax            0xc5528d00       -984445696
>>> ecx            0xc0d09f90       -1060069488
>>> edx            0xc01a1900       -1072031488
>>> ebx            0xfffffff7       -9
>>> esp            0xc0d09f8c       0xc0d09f8c
>>> ebp            0x0      0x0
>>> esi            0x8061480        134616192
>>> edi            0xc5528d00       -984445696
>>> eip            0xc01a1ac1       0xc01a1ac1 <vfs_readdir+1>
>>> eflags         0x286    [ PF SF IF ]
>>> cs             0x60     96
>>> ss             0x8061480        134616192
>>> ds             0x7b     123
>>> es             0x7b     123
>>> fs             0x0      0
>>> gs             0x0      0
>>> (gdb) tfind
>>> Found trace frame 2, tracepoint 1
>>> #0  0xc01a1ac1 in vfs_readdir (file=0xc5528300, filler=0xc01a19e0
>>> <filldir>,
>>>    buf=0xc6c2df90) at /home/teawater/kernel/linux-2.6/fs/readdir.c:23
>>> 23      in /home/teawater/kernel/linux-2.6/fs/readdir.c
>>> (gdb)
>>> Found trace frame 3, tracepoint 1
>>> 0xc01a1ac1      23      in /home/teawater/kernel/linux-2.6/fs/readdir.c
>>> (gdb)
>>> Target failed to find requested trace frame.
>>> (gdb) quit
>>> A debugging session is active.
>>>
>>>        Inferior 1 [Remote target] will be killed.
>>>
>>> Quit anyway? (y or n) y
>>>
>>> Thanks,
>>> Hui
>>

Re: Linux Kernel GDB tracepoint module demo release

[Hui Zhu]

Support x86-64 now.

On Fri, Aug 13, 2010 at 01:13, Michael Snyder <msnyder@vmware.com> wrote:
> Very interesting then.  I'd like to see you continue this work.
>
>
> Hui Zhu wrote:
>>
>> KGTP use Kprobes, I think it can work with most part of kernel.
>>
>> Thanks,
>> Hui
>>
>> On Thu, Aug 12, 2010 at 05:10, Michael Snyder <msnyder@vmware.com> wrote:
>>>
>>> Actually, this is kind of interesting.
>>> Can your system set a tracepoint at any arbitrary location in the
>>> kernel?  If so, I think that makes it more powerful than existing
>>> systems such as LTTng, which can only trace statically pre-determined
>>> locations.
>>>
>>> Hui Zhu wrote:
>>>>
>>>> Hi,
>>>>
>>>> GDB Tracepoint
>>>> (http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html)
>>>> is new feature that make GDB can debug real-time program.
>>>> I think this feature is very fit for debug Linux Kernel.  So I make a
>>>> lkm to make kernel support it. You can get it in
>>>> https://code.google.com/p/kgtp/
>>>>
>>>> I just release the demo for it.  It just support collect $reg in
>>>> x86_32.  I will add the other tracepoint support in the future.
>>>>
>>>> For example:
>>>> tar vxjf gtp_20100811.tar.bz2
>>>> cd gtp
>>>> make
>>>> insmod gtp.ko
>>>> gdb ./vmlinux
>>>> GNU gdb (GDB) 7.2.50.20100720-cvs
>>>> Copyright (C) 2010 Free Software Foundation, Inc.
>>>> License GPLv3+: GNU GPL version 3 or later
>>>> <http://gnu.org/licenses/gpl.html>
>>>> This is free software: you are free to change and redistribute it.
>>>> There is NO WARRANTY, to the extent permitted by law.  Type "show
>>>> copying"
>>>> and "show warranty" for details.
>>>> This GDB was configured as "i686-pc-linux-gnu".
>>>> For bug reporting instructions, please see:
>>>> <http://www.gnu.org/software/gdb/bugs/>...
>>>> Reading symbols from /vmlinux...done.
>>>> (gdb) target remote /proc/gtp
>>>> Remote debugging using /proc/gtp
>>>> 0x00000000 in ?? ()
>>>> (gdb) trace vfs_readdir
>>>> Tracepoint 1 at 0xc01a1ac0: file
>>>> /home/teawater/kernel/linux-2.6/fs/readdir.c, line 23.
>>>> (gdb) actions
>>>> Enter actions for tracepoint 1, one per line.
>>>> End with a line saying just "end".
>>>>>
>>>>> collect $reg
>>>>> end
>>>>
>>>> (gdb) tstart
>>>> (gdb) shell ls
>>>> (gdb) tstop
>>>> (gdb) tfind
>>>> Found trace frame 0, tracepoint 1
>>>> #0  0xc01a1ac1 in vfs_readdir (file=0xc5528d00, filler=0xc01a1900
>>>> <filldir64>,
>>>>   buf=0xc0d09f90) at /home/teawater/kernel/linux-2.6/fs/readdir.c:23
>>>> 23      /home/teawater/kernel/linux-2.6/fs/readdir.c: No such file or
>>>> directory.
>>>>       in /home/teawater/kernel/linux-2.6/fs/readdir.c
>>>> (gdb) info reg
>>>> eax            0xc5528d00       -984445696
>>>> ecx            0xc0d09f90       -1060069488
>>>> edx            0xc01a1900       -1072031488
>>>> ebx            0xfffffff7       -9
>>>> esp            0xc0d09f8c       0xc0d09f8c
>>>> ebp            0x0      0x0
>>>> esi            0x8061480        134616192
>>>> edi            0xc5528d00       -984445696
>>>> eip            0xc01a1ac1       0xc01a1ac1 <vfs_readdir+1>
>>>> eflags         0x286    [ PF SF IF ]
>>>> cs             0x60     96
>>>> ss             0x8061480        134616192
>>>> ds             0x7b     123
>>>> es             0x7b     123
>>>> fs             0x0      0
>>>> gs             0x0      0
>>>> (gdb) tfind
>>>> Found trace frame 1, tracepoint 1
>>>> 0xc01a1ac1      23      in /home/teawater/kernel/linux-2.6/fs/readdir.c
>>>> (gdb) info reg
>>>> eax            0xc5528d00       -984445696
>>>> ecx            0xc0d09f90       -1060069488
>>>> edx            0xc01a1900       -1072031488
>>>> ebx            0xfffffff7       -9
>>>> esp            0xc0d09f8c       0xc0d09f8c
>>>> ebp            0x0      0x0
>>>> esi            0x8061480        134616192
>>>> edi            0xc5528d00       -984445696
>>>> eip            0xc01a1ac1       0xc01a1ac1 <vfs_readdir+1>
>>>> eflags         0x286    [ PF SF IF ]
>>>> cs             0x60     96
>>>> ss             0x8061480        134616192
>>>> ds             0x7b     123
>>>> es             0x7b     123
>>>> fs             0x0      0
>>>> gs             0x0      0
>>>> (gdb) tfind
>>>> Found trace frame 2, tracepoint 1
>>>> #0  0xc01a1ac1 in vfs_readdir (file=0xc5528300, filler=0xc01a19e0
>>>> <filldir>,
>>>>   buf=0xc6c2df90) at /home/teawater/kernel/linux-2.6/fs/readdir.c:23
>>>> 23      in /home/teawater/kernel/linux-2.6/fs/readdir.c
>>>> (gdb)
>>>> Found trace frame 3, tracepoint 1
>>>> 0xc01a1ac1      23      in /home/teawater/kernel/linux-2.6/fs/readdir.c
>>>> (gdb)
>>>> Target failed to find requested trace frame.
>>>> (gdb) quit
>>>> A debugging session is active.
>>>>
>>>>       Inferior 1 [Remote target] will be killed.
>>>>
>>>> Quit anyway? (y or n) y
>>>>
>>>> Thanks,
>>>> Hui
>>>
>
>