[PATCH v4 00/13] Add gdb python scripts as kernel debugging helpers

* [PATCH v4 00/13] Add gdb python scripts as kernel debugging helpers
@ 2013-01-21 17:06 ` Jan Kiszka
  0 siblings, 0 replies; 40+ messages in thread
From: Jan Kiszka @ 2013-01-21 17:06 UTC (permalink / raw)
  To: Andrew Morton, linux-kernel
  Cc: Jason Wessel, kgdb-bugreport, Andi Kleen, Tom Tromey,
	Ben Widawsky, David S. Miller, Fenghua Yu, Kay Sievers,
	linux-ia64, linux-kbuild, Michal Marek, sparclinux, Tony Luck

Version 4 of this series is a rebase over latest 3.8-rc4+. Moreover, I
updated the mechanism that implements automatic symbol loading for new
modules. It was affected by the refactorings around finit_module.

While waiting for feedback who could imagine picking this up for merge,
I wrote a tiny tutorial, see below.

Here is the original series intro again:

This adds the infrastructure and first tools that make kernel debugging
through gdb more comfortable. Since 7.0, gdb supports python scripting.
And this opens the doors to automate steps like the tedious loading of
module symbols at the right address, resolving per-cpu variables or even
retrieving the current kernel log without resuming an stopped target.

Many of the helpers naturally depend on the layout of structures or
internal mechanics of the kernel. So the best place to maintain such
things, keeping them consistent with the corresponding kernel is, well,
the kernel itself.

While these scripts have been originally developed for debugging via
QEMU/KVM, I've now also added the required bits for KGDB. Works fine,
but as QEMU/KVM tends to outperform KGDB it remains the recommendation
- when available.

There are two architecture dependencies so far, one regarding per-cpu,
the other regarding thread_info calculation. None of them I was able to
test on a target, so I'm counting on review/testing by the corresponding
communities.

This series should be considered the foundation of much more kernel
state exploration helpers, e.g. around tasks, timers, locks, sockets -
I guess people will have even more ideas.

And this is a tutorial for the gdb extension using QEMU/KVM as target
platform:

 o Set up a virtual Linux machine for KVM (see www.linux-kvm.org and
   www.qemu.org for more details)

 o Build the kernel with this series applied, enabling CONFIG_DEBUG_INFO
   (but leave CONFIG_DEBUG_INFO_REDUCED off)

 o Install that kernel on the guest

 o Enable the gdb stub of QEMU/KVM, either
    - at VM startup time by appending "-s" to the QEMU command line
   or
    - during runtime by issuing "gdbserver" from the QEMU monitor
      console

 o cd /path/to/linux-build

 o Start gdb: gdb vmlinux

 o Attach to the booted guest:
    (gdb) target remote :1234

 o Load module (and main kernel) symbols:
    (gdb) lx-symbols
    loading vmlinux
    scanning for modules in /home/user/linux/build
    loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
    loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
    loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
    loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
    loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
    ...
    loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko

 o Set a breakpoint on some not yet loaded module function, e.g.:
    (gdb) b btrfs_init_sysfs
    Function "btrfs_init_sysfs" not defined.
    Make breakpoint pending on future shared library load? (y or [n]) y
    Breakpoint 1 (btrfs_init_sysfs) pending.

 o Continue the target

 o Load the module on the target and watch what happens:
    loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
    loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
    loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
    loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko

    Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
    36              btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);

 o Let's examine the current task a bit:
    (gdb) p ().pid
     = 4998
    (gdb) p ().comm
     = "modprobe\000\000\000\000\000\000\000"

 o Dump the log buffer of target kernel:
    (gdb) lx-dmesg
    [     0.000000] Initializing cgroup subsys cpuset
    [     0.000000] Initializing cgroup subsys cpu
    [     0.000000] Linux version 3.8.0-rc4-dbg+ (...
    [     0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
    [     0.000000] e820: BIOS-provided physical RAM map:
    [     0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
    [     0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
    ....

 o Make use of the per-cpu helper for the current or a specified CPU:
    (gdb) p ("runqueues").nr_running
     = 1
    (gdb) p ("runqueues", 2).nr_running
     = 0

 o And now we are digging deep into hrtimers using the container_of
   helper:
    (gdb) set  = ("hrtimer_bases").clock_base[0].active.next
    (gdb) p *(, "struct hrtimer", "node")
     = {
      node = {
        node = {
          __rb_parent_color = 18446612133355256072,
          rb_right = 0x0 <irq_stack_union>,
          rb_left = 0x0 <irq_stack_union>
        },
        expires = {
          tv64 = 1835268000000
        }
      },
      _softexpires = {
        tv64 = 1835268000000
      },
      function = 0xffffffff81078232 <tick_sched_timer>,
      base = 0xffff88003fd0d6f0,
      state = 1,
      start_pid = 0,
      start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
      start_comm = "swapper/2\000\000\000\000\000\000"
    }

Hope this provided some ideas and inspirations on how the commands and
helper functions can support kernel development.

Enjoy,
Jan

PS: Also available via git://git.kiszka.org/linux.git queues/gdb-scripts

CC: "David S. Miller" <davem@davemloft.net>
CC: Fenghua Yu <fenghua.yu@intel.com>
CC: Kay Sievers <kay@vrfy.org>
CC: linux-ia64@vger.kernel.org
CC: linux-kbuild@vger.kernel.org
CC: Michal Marek <mmarek@suse.cz>
CC: sparclinux@vger.kernel.org
CC: Tony Luck <tony.luck@intel.com>

Jan Kiszka (13):
  scripts/gdb: Add infrastructure
  scripts/gdb: Add container_of helper and convenience function
  scripts/gdb: Add lx-symbols command
  scripts/gdb: Add get_target_endianness helper
  scripts/gdb: Add read_u16/32/64 helpers
  scripts/gdb: Add lx-dmesg command
  scripts/gdb: Add task iteration helper
  scripts/gdb: Add helper and convenience function to look up tasks
  scripts/gdb: Add is_target_arch helper
  scripts/gdb: Add internal helper and convenience function to retrieve
    thread_info
  scripts/gdb: Add get_gdbserver_type helper
  scripts/gdb: Add internal helper and convenience function for per-cpu
    lookup
  scripts/gdb: Add lx_current convenience function

 Makefile                   |    5 +-
 scripts/Makefile           |    3 +-
 scripts/gdb/Makefile       |    9 +++
 scripts/gdb/dmesg.py       |   63 ++++++++++++++++++
 scripts/gdb/percpu.py      |   76 ++++++++++++++++++++++
 scripts/gdb/symbols.py     |  153 ++++++++++++++++++++++++++++++++++++++++++++
 scripts/gdb/task.py        |  108 +++++++++++++++++++++++++++++++
 scripts/gdb/utils.py       |  137 +++++++++++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |   28 ++++++++
 9 files changed, 580 insertions(+), 2 deletions(-)
 create mode 100644 scripts/gdb/Makefile
 create mode 100644 scripts/gdb/dmesg.py
 create mode 100644 scripts/gdb/percpu.py
 create mode 100644 scripts/gdb/symbols.py
 create mode 100644 scripts/gdb/task.py
 create mode 100644 scripts/gdb/utils.py
 create mode 100644 scripts/gdb/vmlinux-gdb.py

-- 
1.7.3.4

^ permalink raw reply	[flat|nested] 40+ messages in thread