[PATCH v5 00/20] Add gdb python scripts as kernel debugging helpers

[PATCH v5 00/20] Add gdb python scripts as kernel debugging helpers

[Jan Kiszka]

Version 5 comes with the following changes:
 - moved tutorial into Documentation/gdb-kernel-debugging.txt
 - improved caching of gdb.Type objects, ensure they are in sync with
   currently loaded symbols
 - added new functions and commands
    - lx_module -- Find module by name and return the module variable
    - lx_modvar -- Return global variable of a module
    - lx-lsmod -- List currently loaded modules

See http://lkml.indiana.edu/hypermail/linux/kernel/1210.0/01598.html for
the original description and

    git://git.kiszka.org/linux.git queues/gdb-scripts

for the latest version.

Jan

CC: "David S. Miller" <davem@davemloft.net>
CC: Fenghua Yu <fenghua.yu@intel.com>
CC: Kay Sievers <kay@vrfy.org>
CC: linux-doc@vger.kernel.org
CC: linux-ia64@vger.kernel.org
CC: linux-kbuild@vger.kernel.org
CC: Michal Marek <mmarek@suse.cz>
CC: Rob Landley <rob@landley.net>
CC: sparclinux@vger.kernel.org
CC: Tony Luck <tony.luck@intel.com>

Jan Kiszka (20):
  scripts/gdb: Add infrastructure
  scripts/gdb: Add cache for type objects
  scripts/gdb: Add container_of helper and convenience function
  scripts/gdb: Add module iteration helper
  scripts/gdb: Add lx-symbols command
  scripts/gdb: Add internal helper and convenience function to look up
    a module
  scripts/gdb: Add lx_modvar convenience function
  scripts/gdb: Add get_target_endianness helper
  scripts/gdb: Add read_u16/32/64 helpers
  scripts/gdb: Add lx-dmesg command
  scripts/gdb: Add task iteration helper
  scripts/gdb: Add helper and convenience function to look up tasks
  scripts/gdb: Add is_target_arch helper
  scripts/gdb: Add internal helper and convenience function to retrieve
    thread_info
  scripts/gdb: Add get_gdbserver_type helper
  scripts/gdb: Add internal helper and convenience function for per-cpu
    lookup
  scripts/gdb: Add lx_current convenience function
  scripts/gdb: Add helper to iterate over CPU masks
  scripts/gdb: Add lx-lsmod command
  scripts/gdb: Add basic documentation

 Documentation/gdb-kernel-debugging.txt |  155 +++++++++++++++++++++++++++++++
 Makefile                               |    5 +-
 scripts/Makefile                       |    3 +-
 scripts/gdb/Makefile                   |    9 ++
 scripts/gdb/dmesg.py                   |   63 +++++++++++++
 scripts/gdb/module.py                  |  149 ++++++++++++++++++++++++++++++
 scripts/gdb/percpu.py                  |  110 ++++++++++++++++++++++
 scripts/gdb/symbols.py                 |  147 +++++++++++++++++++++++++++++
 scripts/gdb/task.py                    |   99 ++++++++++++++++++++
 scripts/gdb/utils.py                   |  158 ++++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py             |   29 ++++++
 11 files changed, 925 insertions(+), 2 deletions(-)
 create mode 100644 Documentation/gdb-kernel-debugging.txt
 create mode 100644 scripts/gdb/Makefile
 create mode 100644 scripts/gdb/dmesg.py
 create mode 100644 scripts/gdb/module.py
 create mode 100644 scripts/gdb/percpu.py
 create mode 100644 scripts/gdb/symbols.py
 create mode 100644 scripts/gdb/task.py
 create mode 100644 scripts/gdb/utils.py
 create mode 100644 scripts/gdb/vmlinux-gdb.py

-- 
1.7.3.4

[PATCH v5 01/20] scripts/gdb: Add infrastructure

[Jan Kiszka]

This provides the basic infrastructure to load kernel-specific python
helper scripts when debugging the kernel in gdb.

The loading mechanism is based on gdb loading for <objfile>-gdb.py when
opening <objfile>. Therefore, this places a corresponding link to the
main helper script into the output directory that contains vmlinux.

The main scripts will pull in submodules containing Linux specific gdb
commands and functions. To avoid polluting the source directory with
compiled python modules, we link to them from the object directory.

Due to gdb.parse_and_eval, we depend on gdb >= 7.1. We need to
pre-process the version string returned by gdb as some distros tend to
prefix it with their name.

This feature depends on CONFIG_DEBUG_INFO.

CC: Michal Marek <mmarek@suse.cz>
CC: linux-kbuild@vger.kernel.org
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 Makefile                   |    5 ++++-
 scripts/Makefile           |    3 ++-
 scripts/gdb/Makefile       |    9 +++++++++
 scripts/gdb/utils.py       |   17 +++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |   22 ++++++++++++++++++++++
 5 files changed, 54 insertions(+), 2 deletions(-)
 create mode 100644 scripts/gdb/Makefile
 create mode 100644 scripts/gdb/utils.py
 create mode 100644 scripts/gdb/vmlinux-gdb.py

diff --git a/Makefile b/Makefile
index 2d3c92c..f0d2b2d 100644
--- a/Makefile
+++ b/Makefile
@@ -774,6 +774,9 @@ endif
 ifdef CONFIG_BUILD_DOCSRC
 	$(Q)$(MAKE) $(build)=Documentation
 endif
+ifdef CONFIG_DEBUG_INFO
+	$(Q)ln -fsn $(srctree)/scripts/gdb/vmlinux-gdb.py
+endif
 	+$(call if_changed,link-vmlinux)
 
 # The actual objects are generated when descending, 
@@ -1019,7 +1022,7 @@ MRPROPER_FILES += .config .config.old .version .old_version $(version_h) \
 		  Module.symvers tags TAGS cscope* GPATH GTAGS GRTAGS GSYMS \
 		  signing_key.priv signing_key.x509 x509.genkey		\
 		  extra_certificates signing_key.x509.keyid		\
-		  signing_key.x509.signer
+		  signing_key.x509.signer vmlinux-gdb.py
 
 # clean - Delete most, but leave enough to build external modules
 #
diff --git a/scripts/Makefile b/scripts/Makefile
index 01e7adb..3204b91 100644
--- a/scripts/Makefile
+++ b/scripts/Makefile
@@ -37,6 +37,7 @@ subdir-$(CONFIG_MODVERSIONS) += genksyms
 subdir-y                     += mod
 subdir-$(CONFIG_SECURITY_SELINUX) += selinux
 subdir-$(CONFIG_DTC)         += dtc
+subdir-$(CONFIG_DEBUG_INFO)  += gdb
 
 # Let clean descend into subdirs
-subdir-	+= basic kconfig package selinux
+subdir-	+= basic kconfig package selinux gdb
diff --git a/scripts/gdb/Makefile b/scripts/gdb/Makefile
new file mode 100644
index 0000000..34ccd06
--- /dev/null
+++ b/scripts/gdb/Makefile
@@ -0,0 +1,9 @@
+always := gdb-scripts
+
+$(obj)/gdb-scripts:
+ifneq ($(KBUILD_SRC),)
+	$(Q)ln -fsn $(srctree)/$(obj)/*.py $(objtree)/$(obj)
+endif
+	@:
+
+clean-files := *.pyc $(if $(KBUILD_SRC),*.py)
diff --git a/scripts/gdb/utils.py b/scripts/gdb/utils.py
new file mode 100644
index 0000000..f75eae6
--- /dev/null
+++ b/scripts/gdb/utils.py
@@ -0,0 +1,17 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  common utilities
+#
+# Copyright (c) Siemens AG, 2011-2013
+#
+# Authors:
+#  Jan Kiszka <jan.kiszka@siemens.com>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import gdb
+import re
+
+gdb_version = re.sub("^[^0-9]*", "", gdb.VERSION)
diff --git a/scripts/gdb/vmlinux-gdb.py b/scripts/gdb/vmlinux-gdb.py
new file mode 100644
index 0000000..bcb45cc
--- /dev/null
+++ b/scripts/gdb/vmlinux-gdb.py
@@ -0,0 +1,22 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  loader module
+#
+# Copyright (c) Siemens AG, 2012
+#
+# Authors:
+#  Jan Kiszka <jan.kiszka@siemens.com>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import os
+
+sys.path = [ os.path.dirname(__file__) + "/scripts/gdb" ] + sys.path
+
+from utils import gdb_version
+
+if gdb_version < "7.1":
+	print "NOTE: gdb 7.1 or later required for Linux helper scripts " \
+	      "to work."
-- 
1.7.3.4

[PATCH v5 02/20] scripts/gdb: Add cache for type objects

[Jan Kiszka]

Type lookups are very slow in gdb-python which is often noticeable when
iterating over a number of objects. Introduce the helper class
CachedType that keeps a reference to a gdb.Type object but also
refreshes it after an object file has been loaded.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/utils.py |   21 +++++++++++++++++++++
 1 files changed, 21 insertions(+), 0 deletions(-)

diff --git a/scripts/gdb/utils.py b/scripts/gdb/utils.py
index f75eae6..fdff85e 100644
--- a/scripts/gdb/utils.py
+++ b/scripts/gdb/utils.py
@@ -15,3 +15,24 @@ import gdb
 import re
 
 gdb_version = re.sub("^[^0-9]*", "", gdb.VERSION)
+
+
+class CachedType:
+	_type = None
+
+	def _new_objfile_handler(self, event):
+		self._type = None
+		gdb.events.new_objfile.disconnect(self._new_objfile_handler)
+
+	def __init__(self, name):
+		self._name = name
+
+	def get_type(self):
+		if self._type == None:
+			self._type = gdb.lookup_type(self._name)
+			if self._type == None:
+				raise gdb.GdbError("cannot resolve " \
+						   "type '%s'" % self._name)
+			gdb.events.new_objfile.connect(
+				self._new_objfile_handler)
+		return self._type
-- 
1.7.3.4

[PATCH v5 03/20] scripts/gdb: Add container_of helper and convenience function

[Jan Kiszka]

Provide an internal helper with container_of semantics. As type lookups
are very slow in gdb-python and we need a type "long" for this, cache
the reference to this type object. Then export the helper also as a
convenience function form use at the gdb command line.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/utils.py       |   34 ++++++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |    2 ++
 2 files changed, 36 insertions(+), 0 deletions(-)

diff --git a/scripts/gdb/utils.py b/scripts/gdb/utils.py
index fdff85e..d1f6f12 100644
--- a/scripts/gdb/utils.py
+++ b/scripts/gdb/utils.py
@@ -36,3 +36,37 @@ class CachedType:
 			gdb.events.new_objfile.connect(
 				self._new_objfile_handler)
 		return self._type
+
+
+long_type = CachedType("long")
+
+def get_long_type():
+	global long_type
+	return long_type.get_type()
+
+
+def offset_of(typeobj, field):
+	element = gdb.Value(0).cast(typeobj)
+	return int(str(element[field].address).split()[0], 16)
+
+def container_of(ptr, typeobj, member):
+	return (ptr.cast(get_long_type()) -
+		offset_of(typeobj, member)).cast(typeobj)
+
+
+class ContainerOf(gdb.Function):
+	__doc__ = "Return pointer to containing data structure.\n" \
+		  "\n" \
+		  "$container_of(PTR, \"TYPE\", \"ELEMENT\"): Given PTR, return a pointer to the\n" \
+		  "data structure of the type TYPE in which PTR is the address of ELEMENT.\n" \
+		  "Note that TYPE and ELEMENT have to be quoted as strings."
+
+	def __init__(self):
+		super(ContainerOf, self).__init__("container_of")
+
+	def invoke(self, ptr, typename, elementname):
+		return container_of(ptr,
+				    gdb.lookup_type(typename.string()).pointer(),
+				    elementname.string())
+
+ContainerOf()
diff --git a/scripts/gdb/vmlinux-gdb.py b/scripts/gdb/vmlinux-gdb.py
index bcb45cc..8ef76c9 100644
--- a/scripts/gdb/vmlinux-gdb.py
+++ b/scripts/gdb/vmlinux-gdb.py
@@ -20,3 +20,5 @@ from utils import gdb_version
 if gdb_version < "7.1":
 	print "NOTE: gdb 7.1 or later required for Linux helper scripts " \
 	      "to work."
+else:
+	import utils
-- 
1.7.3.4

[PATCH v5 04/20] scripts/gdb: Add module iteration helper

[Jan Kiszka]

Will soon be used for loading symbols, printing global variables or
listing modules.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/module.py |   28 ++++++++++++++++++++++++++++
 1 files changed, 28 insertions(+), 0 deletions(-)
 create mode 100644 scripts/gdb/module.py

diff --git a/scripts/gdb/module.py b/scripts/gdb/module.py
new file mode 100644
index 0000000..01b23df
--- /dev/null
+++ b/scripts/gdb/module.py
@@ -0,0 +1,28 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  module tools
+#
+# Copyright (c) Siemens AG, 2013
+#
+# Authors:
+#  Jan Kiszka <jan.kiszka@siemens.com>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import gdb
+
+from utils import *
+
+module_type = CachedType("struct module")
+
+def for_each_module(func, arg = None):
+	global module_type
+	module_ptr_type = module_type.get_type().pointer()
+	modules = gdb.parse_and_eval("modules")
+	entry = modules['next']
+	while entry != modules.address:
+		module = container_of(entry, module_ptr_type, "list")
+		func(module, arg)
+		entry = entry['next']
-- 
1.7.3.4

[PATCH v5 05/20] scripts/gdb: Add lx-symbols command

[Jan Kiszka]

This is probably the most useful helper when debugging kernel modules:
lx-symbols will first reload vmlinux. Then it searches recursively for
*.ko files in the specified paths and the current directory. Finally it
walks the kernel's module list, issuing the necessary add-symbol-file
command for each loaded module so that gdb know which module symbol
correspond to which address.

In addition, the command installs a silent breakpoint in the load_module
function at the point where the module was loaded but not yet
initialized. The breakpoint handler will try to load symbols from the
module files found during lx-symbols execution. This way, breakpoints
can be set to module initialization functions, and there is usually no
need to explicitly call lx-symbols after (re-)loading a module.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/symbols.py     |  147 ++++++++++++++++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |    1 +
 2 files changed, 148 insertions(+), 0 deletions(-)
 create mode 100644 scripts/gdb/symbols.py

diff --git a/scripts/gdb/symbols.py b/scripts/gdb/symbols.py
new file mode 100644
index 0000000..8b672e2
--- /dev/null
+++ b/scripts/gdb/symbols.py
@@ -0,0 +1,147 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  load kernel and module symbols
+#
+# Copyright (c) Siemens AG, 2011-2013
+#
+# Authors:
+#  Jan Kiszka <jan.kiszka@siemens.com>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import gdb
+import os
+import re
+import string
+
+from module import for_each_module
+from utils import *
+
+class LinuxSymbols(gdb.Command):
+	__doc__ = "(Re-)load symbols of Linux kernel and currently loaded modules.\n" \
+		  "\n" \
+		  "The kernel (vmlinux) is taken from the current working directly. Modules (.ko)\n" \
+		  "are scanned recursively, starting in the same directory. Optionally, the module\n" \
+		  "search path can be extended by a space separated list of paths passed to the\n" \
+		  "lx-symbols command."
+
+	module_files = []
+	breakpoint = None
+
+	def __init__(self):
+		super(LinuxSymbols, self).__init__("lx-symbols",
+						   gdb.COMMAND_FILES,
+						   gdb.COMPLETE_FILENAME)
+
+	def _load_module_symbols(self, module, arg = None):
+		module_name = module['name'].string()
+		module_addr = str(module['module_core']).split()[0]
+		module_pattern = ".*/" + \
+			string.replace(module_name, "_", r"[_\-]") + r"\.ko$"
+		module_path = ""
+		for name in self.module_files:
+			if re.match(module_pattern, name):
+				module_path = name
+				break
+
+		if module_path:
+			print "loading @" + module_addr + ": " + module_path
+			if gdb.VERSION >= "7.2":
+				gdb.execute("add-symbol-file " + \
+					    module_path + " " + module_addr,
+					    to_string = True)
+			else:
+				gdb.execute("add-symbol-file " + \
+					    module_path + " " + module_addr)
+		else:
+			print "no module object found for '" + \
+			      module_name + "'"
+
+	if gdb_version >= "7.3":
+		class _LoadModuleBreakpoint(gdb.Breakpoint):
+			def __init__(self, spec, gdb_command):
+				super(LinuxSymbols._LoadModuleBreakpoint,
+				      self).__init__(spec, internal = True)
+				self.silent = True
+				self.gdb_command = gdb_command
+
+			def stop(self):
+				module = gdb.parse_and_eval("mod")
+				self.gdb_command._load_module_symbols(module)
+				return False
+
+		def _find_breakpoint_location(self):
+			breakpoint_match = "^[0-9]*[\t]*err = parse_args\(.*"
+
+			src = gdb.execute("list kernel/module.c:load_module",
+					  to_string = True)
+			done = False
+			lineno = None
+			while not done:
+				src = gdb.execute("list", to_string = True)
+				for line in src.splitlines():
+					if re.match(breakpoint_match, line):
+						done = True
+						lineno = line.split()[0]
+						break
+					elif re.match("^[0-9]*\t}$", line):
+						done = True
+						break
+			return lineno
+
+	def _load_all_module_symbols(self, arg):
+		if gdb.parse_and_eval("modules.next == &modules"):
+			print "no modules found"
+			return
+
+		self.module_files = []
+		paths = arg.split()
+		paths.append(os.getcwd())
+		for path in paths:
+			print "scanning for modules in " + path
+			for root, dirs, files in os.walk(path):
+				for name in files:
+					if re.match(r".*\.ko$", name):
+						self.module_files.append(
+							root + "/" + name)
+
+		for_each_module(self._load_module_symbols)
+
+	def invoke(self, arg, from_tty):
+		print "loading vmlinux"
+
+		saved_states = []
+		if (gdb.breakpoints() != None):
+			for breakpoint in gdb.breakpoints():
+				saved_states.append({
+					'breakpoint': breakpoint,
+					'enabled': breakpoint.enabled })
+
+		# drop all current symbols and reload vmlinux
+		gdb.execute("symbol-file", to_string = True)
+		gdb.execute("symbol-file vmlinux")
+
+		self._load_all_module_symbols(arg)
+
+		for saved_state in saved_states:
+			saved_state['breakpoint'].enabled = \
+				saved_state['enabled']
+
+		if gdb_version < "7.3":
+			print "Note: symbol update on module loading not " \
+			      "supported with this gdb version"
+		else:
+			if self.breakpoint != None:
+				self.breakpoint.delete()
+				self.breakpoint = None
+			lineno = self._find_breakpoint_location()
+			if lineno != None:
+				self.breakpoint = self._LoadModuleBreakpoint(
+					lineno, self)
+			else:
+				print "unable to detect breakpoint location " \
+				      "of load module completion"
+
+LinuxSymbols()
diff --git a/scripts/gdb/vmlinux-gdb.py b/scripts/gdb/vmlinux-gdb.py
index 8ef76c9..3553b7d 100644
--- a/scripts/gdb/vmlinux-gdb.py
+++ b/scripts/gdb/vmlinux-gdb.py
@@ -22,3 +22,4 @@ if gdb_version < "7.1":
 	      "to work."
 else:
 	import utils
+	import symbols
-- 
1.7.3.4

[PATCH v5 06/20] scripts/gdb: Add internal helper and convenience function to look up a module

[Jan Kiszka]

Add the internal helper get_module_by_name to obtain the module
structure corresponding to the given name. Also export this service as a
convenience function.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/module.py      |   30 ++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |    1 +
 2 files changed, 31 insertions(+), 0 deletions(-)

diff --git a/scripts/gdb/module.py b/scripts/gdb/module.py
index 01b23df..e309c0a 100644
--- a/scripts/gdb/module.py
+++ b/scripts/gdb/module.py
@@ -26,3 +26,33 @@ def for_each_module(func, arg = None):
 		module = container_of(entry, module_ptr_type, "list")
 		func(module, arg)
 		entry = entry['next']
+
+def find_module_by_name(name):
+	def match_name(module, arg):
+		if module['name'].string() == arg['name']:
+			arg['module'] = module
+
+	arg = { 'name': name, 'module': None }
+	for_each_module(match_name, arg)
+
+	return arg['module']
+
+
+class LxModule(gdb.Function):
+	__doc__ = "Find module by name and return the module variable.\n" \
+		  "\n" \
+		  "$lx_module(MODULE): Given the name MODULE, iterate over all loaded modules of\n" \
+		  "the target and return that module variable which MODULE matches."
+
+	def __init__(self):
+		super(LxModule, self).__init__("lx_module")
+
+	def invoke(self, mod_name):
+		mod_name = mod_name.string()
+		module = find_module_by_name(mod_name)
+		if module:
+			return module.dereference()
+		else:
+			raise gdb.GdbError("Unable to find MODULE " + mod_name)
+
+LxModule()
diff --git a/scripts/gdb/vmlinux-gdb.py b/scripts/gdb/vmlinux-gdb.py
index 3553b7d..2a466a0 100644
--- a/scripts/gdb/vmlinux-gdb.py
+++ b/scripts/gdb/vmlinux-gdb.py
@@ -23,3 +23,4 @@ if gdb_version < "7.1":
 else:
 	import utils
 	import symbols
+	import module
-- 
1.7.3.4

[PATCH v5 07/20] scripts/gdb: Add lx_modvar convenience function

[Jan Kiszka]

This function looks up and returns global module variables. Gdb is only
aware of their types and section offsets, not their absolute addresses.
The function either searches the symbol table of a specified module or
it derives the module name from the object file of the current frame.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/module.py |   48 ++++++++++++++++++++++++++++++++++++++++++++++++
 scripts/gdb/utils.py  |    7 +++++++
 2 files changed, 55 insertions(+), 0 deletions(-)

diff --git a/scripts/gdb/module.py b/scripts/gdb/module.py
index e309c0a..333729a 100644
--- a/scripts/gdb/module.py
+++ b/scripts/gdb/module.py
@@ -12,6 +12,8 @@
 #
 
 import gdb
+import os
+import string
 
 from utils import *
 
@@ -56,3 +58,49 @@ class LxModule(gdb.Function):
 			raise gdb.GdbError("Unable to find MODULE " + mod_name)
 
 LxModule()
+
+
+class LxModvar(gdb.Function):
+	__doc__ = "Return global variable of a module.\n" \
+		  "\n" \
+		  "$lx_modvar(\"VAR\"[, \"MODULE\"]): Return the global variable called VAR that is\n" \
+		  "defined by given MODULE. If MODULE is omitted, the current frame is used to\n" \
+		  "try finding the corresponding module name."
+
+	def __init__(self):
+		super(LxModvar, self).__init__("lx_modvar")
+
+	def _lookup_mod_symbol(self, module, var_name):
+		char_ptr_type = get_char_type().pointer()
+		for i in range(0, int(module['num_symtab'])):
+			symtab_entry = module['symtab'][i]
+			idx = int(symtab_entry['st_name'])
+			synname_addr = module['strtab'][idx].address
+			symname = synname_addr.cast(char_ptr_type)
+			if symname.string() == var_name:
+				return symtab_entry['st_value']
+		return None
+
+	def invoke(self, var_name, mod_name = None):
+		if (mod_name == None):
+			obj = gdb.selected_frame().function().symtab.objfile
+			mod_name = string.replace(
+				os.path.basename(obj.filename), ".ko", "")
+			module = find_module_by_name(mod_name)
+			if module == None:
+				raise gdb.GdbError("Current frame does not " \
+						   "belong to a module")
+		else:
+			mod_name = mod_name.string()
+			module = find_module_by_name(mod_name)
+			if module == None:
+				raise gdb.GdbError("Unable to find MODULE " +
+						   mod_name)
+		var_name = var_name.string()
+		var_addr = self._lookup_mod_symbol(module, var_name)
+		if var_addr == None:
+			raise gdb.GdbError("Unable to find VAR " + var_name)
+		var = gdb.parse_and_eval(var_name)
+		return var_addr.cast(var.type.pointer()).dereference()
+
+LxModvar()
diff --git a/scripts/gdb/utils.py b/scripts/gdb/utils.py
index d1f6f12..13f903a 100644
--- a/scripts/gdb/utils.py
+++ b/scripts/gdb/utils.py
@@ -45,6 +45,13 @@ def get_long_type():
 	return long_type.get_type()
 
 
+char_type = CachedType("char")
+
+def get_char_type():
+	global char_type
+	return char_type.get_type()
+
+
 def offset_of(typeobj, field):
 	element = gdb.Value(0).cast(typeobj)
 	return int(str(element[field].address).split()[0], 16)
-- 
1.7.3.4

[PATCH v5 08/20] scripts/gdb: Add get_target_endianness helper

[Jan Kiszka]

Parse the target endianness from the output of "show endian" and cache
the result to return it via the new helper get_target_endiannes. We will
need it for reading integers from buffers that contain target memory.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/utils.py |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)

diff --git a/scripts/gdb/utils.py b/scripts/gdb/utils.py
index 13f903a..1c7cb42 100644
--- a/scripts/gdb/utils.py
+++ b/scripts/gdb/utils.py
@@ -77,3 +77,20 @@ class ContainerOf(gdb.Function):
 				    elementname.string())
 
 ContainerOf()
+
+
+BIG_ENDIAN = 0
+LITTLE_ENDIAN = 1
+target_endianness = None
+
+def get_target_endianness():
+	global target_endianness
+	if target_endianness == None:
+		endian = gdb.execute("show endian", False, True)
+		if endian.find("little endian") >= 0:
+			target_endianness = LITTLE_ENDIAN
+		elif endian.find("big endian") >= 0:
+			target_endianness = BIG_ENDIAN
+		else:
+			raise gdb.GdgError("unknown endianness '%s'" % endian)
+	return target_endianness
-- 
1.7.3.4

[PATCH v5 09/20] scripts/gdb: Add read_u16/32/64 helpers

[Jan Kiszka]

Add helpers for reading integers from target memory buffers. Required
when caching the memory access is more efficient than reading individual
values via gdb.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/utils.py |   18 ++++++++++++++++++
 1 files changed, 18 insertions(+), 0 deletions(-)

diff --git a/scripts/gdb/utils.py b/scripts/gdb/utils.py
index 1c7cb42..9b23052 100644
--- a/scripts/gdb/utils.py
+++ b/scripts/gdb/utils.py
@@ -94,3 +94,21 @@ def get_target_endianness():
 		else:
 			raise gdb.GdgError("unknown endianness '%s'" % endian)
 	return target_endianness
+
+def read_u16(buffer):
+	if get_target_endianness() == LITTLE_ENDIAN:
+		return ord(buffer[0]) + (ord(buffer[1]) << 8)
+	else:
+		return ord(buffer[1]) + (ord(buffer[0]) << 8)
+
+def read_u32(buffer):
+	if get_target_endianness() == LITTLE_ENDIAN:
+		return read_u16(buffer[0:2]) + (read_u16(buffer[2:4]) << 16)
+	else:
+		return read_u16(buffer[2:4]) + (read_u16(buffer[0:2]) << 16)
+
+def read_u64(buffer):
+	if get_target_endianness() == LITTLE_ENDIAN:
+		return read_u32(buffer[0:4]) + (read_u32(buffer[4:8]) << 32)
+	else:
+		return read_u32(buffer[4:8]) + (read_u32(buffer[0:4]) << 32)
-- 
1.7.3.4

[PATCH v5 10/20] scripts/gdb: Add lx-dmesg command

[Jan Kiszka]

This pokes into the log buffer of the debugged kernel, dumping it to the
gdb console. Helping in case the target should or can no longer execute
dmesg itself.

CC: Kay Sievers <kay@vrfy.org>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/dmesg.py       |   63 ++++++++++++++++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |    1 +
 2 files changed, 64 insertions(+), 0 deletions(-)
 create mode 100644 scripts/gdb/dmesg.py

diff --git a/scripts/gdb/dmesg.py b/scripts/gdb/dmesg.py
new file mode 100644
index 0000000..1bc3368
--- /dev/null
+++ b/scripts/gdb/dmesg.py
@@ -0,0 +1,63 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  kernel log buffer dump
+#
+# Copyright (c) Siemens AG, 2011, 2012
+#
+# Authors:
+#  Jan Kiszka <jan.kiszka@siemens.com>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import gdb
+import string
+
+from utils import *
+
+class LinuxDmesg(gdb.Command):
+	__doc__ = "Print Linux kernel log buffer."
+
+	def __init__(self):
+		super(LinuxDmesg, self).__init__("lx-dmesg", gdb.COMMAND_DATA)
+
+	def invoke(self, arg, from_tty):
+		log_buf_addr = \
+			int(str(gdb.parse_and_eval("log_buf")).split()[0], 16)
+		log_first_idx = int(gdb.parse_and_eval("log_first_idx"))
+		log_next_idx = int(gdb.parse_and_eval("log_next_idx"))
+		log_buf_len = int(gdb.parse_and_eval("log_buf_len"))
+
+		inf = gdb.inferiors()[0]
+		start = log_buf_addr + log_first_idx
+		if log_first_idx < log_next_idx:
+			log_buf_2nd_half = -1
+			length = log_next_idx - log_first_idx
+			log_buf = inf.read_memory(start, length)
+		else:
+			log_buf_2nd_half = log_buf_len - log_first_idx
+			log_buf = inf.read_memory(start, log_buf_2nd_half) + \
+				  inf.read_memory(log_buf_addr, log_next_idx)
+
+		pos = 0
+		while pos < log_buf.__len__():
+			length = read_u16(log_buf[pos + 8 : pos + 10])
+			if length == 0:
+				if log_buf_2nd_half == -1:
+					print "Corrupted log buffer!"
+					break
+				pos = log_buf_2nd_half
+				continue
+
+			text_len = read_u16(log_buf[pos + 10 : pos + 12])
+			time_stamp = read_u64(log_buf[pos : pos + 8])
+
+			for line in log_buf[pos + 16 :
+					    pos + 16 + text_len].splitlines():
+				print "[%13.6f] " % \
+				      (time_stamp / 1000000000.0) + line
+
+			pos += length
+
+LinuxDmesg()
diff --git a/scripts/gdb/vmlinux-gdb.py b/scripts/gdb/vmlinux-gdb.py
index 2a466a0..4a5685b 100644
--- a/scripts/gdb/vmlinux-gdb.py
+++ b/scripts/gdb/vmlinux-gdb.py
@@ -24,3 +24,4 @@ else:
 	import utils
 	import symbols
 	import module
+	import dmesg
-- 
1.7.3.4

[PATCH v5 11/20] scripts/gdb: Add task iteration helper

[Jan Kiszka]

The internal helper for_each_task iterates over all tasks of the target,
calling the provided function on each.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/task.py |   35 +++++++++++++++++++++++++++++++++++
 1 files changed, 35 insertions(+), 0 deletions(-)
 create mode 100644 scripts/gdb/task.py

diff --git a/scripts/gdb/task.py b/scripts/gdb/task.py
new file mode 100644
index 0000000..6b73dbf
--- /dev/null
+++ b/scripts/gdb/task.py
@@ -0,0 +1,35 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  task & thread tools
+#
+# Copyright (c) Siemens AG, 2011-2013
+#
+# Authors:
+#  Jan Kiszka <jan.kiszka@siemens.com>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import gdb
+
+from utils import *
+
+task_type = CachedType("struct task_struct")
+
+def for_each_task(func, arg = None):
+	global task_type
+	task_ptr_type = task_type.get_type().pointer()
+	init_task = gdb.parse_and_eval("init_task")
+	g = init_task.address
+	while True:
+		g =  container_of(g['tasks']['next'], task_ptr_type, "tasks")
+		if g == init_task.address:
+			break;
+		t = g
+		while True:
+			func(t, arg)
+			t = container_of(t['thread_group']['next'],
+					 task_ptr_type, "thread_group")
+			if t == g:
+				break
-- 
1.7.3.4

[PATCH v5 12/20] scripts/gdb: Add helper and convenience function to look up tasks

[Jan Kiszka]

Add the helper task_by_pid that can look up a task by its PID. Also
export it as a convenience function.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/task.py        |   29 +++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |    1 +
 2 files changed, 30 insertions(+), 0 deletions(-)

diff --git a/scripts/gdb/task.py b/scripts/gdb/task.py
index 6b73dbf..fe42eb0 100644
--- a/scripts/gdb/task.py
+++ b/scripts/gdb/task.py
@@ -33,3 +33,32 @@ def for_each_task(func, arg = None):
 					 task_ptr_type, "thread_group")
 			if t == g:
 				break
+
+def get_task_by_pid(pid):
+	def match_pid(t, arg):
+		if int(t['pid']) == arg['pid']:
+			arg['task'] = t
+
+	arg = { 'pid': pid, 'task': None }
+	for_each_task(match_pid, arg)
+
+	return arg['task']
+
+
+class LxTaskByPidFunc(gdb.Function):
+	__doc__ = "Find Linux task by PID and return the task_struct variable.\n" \
+		  "\n" \
+		  "$lx_task_by_pid(PID): Given PID, iterate over all tasks of the target and\n" \
+		  "return that task_struct variable which PID matches."
+
+	def __init__(self):
+		super(LxTaskByPidFunc, self).__init__("lx_task_by_pid")
+
+	def invoke(self, pid):
+		task = get_task_by_pid(pid)
+		if task:
+			return task.dereference()
+		else:
+			raise gdb.GdbError("No task of PID " + str(pid))
+
+LxTaskByPidFunc()
diff --git a/scripts/gdb/vmlinux-gdb.py b/scripts/gdb/vmlinux-gdb.py
index 4a5685b..e1c5dbc 100644
--- a/scripts/gdb/vmlinux-gdb.py
+++ b/scripts/gdb/vmlinux-gdb.py
@@ -25,3 +25,4 @@ else:
 	import symbols
 	import module
 	import dmesg
+	import task
-- 
1.7.3.4

[PATCH v5 13/20] scripts/gdb: Add is_target_arch helper

[Jan Kiszka]

This helper caches to result of "show architecture" and matches the
provided arch (sub-)string against that output.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/utils.py |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/scripts/gdb/utils.py b/scripts/gdb/utils.py
index 9b23052..bcf4278 100644
--- a/scripts/gdb/utils.py
+++ b/scripts/gdb/utils.py
@@ -112,3 +112,12 @@ def read_u64(buffer):
 		return read_u32(buffer[0:4]) + (read_u32(buffer[4:8]) << 32)
 	else:
 		return read_u32(buffer[4:8]) + (read_u32(buffer[0:4]) << 32)
+
+
+target_arch = None
+
+def is_target_arch(arch):
+	global target_arch
+	if target_arch == None:
+		target_arch = gdb.execute("show architecture", False, True)
+	return target_arch.find(arch) >= 0
-- 
1.7.3.4

[PATCH v5 14/20] scripts/gdb: Add internal helper and convenience function to retrieve thread_info

[Jan Kiszka]

Add the internal helper get_thread_info that calculated the thread_info
from a given task variable. Also export this service as a convenience
function.

Note: ia64 version is untested.

CC: Tony Luck <tony.luck@intel.com>
CC: Fenghua Yu <fenghua.yu@intel.com>
CC: linux-ia64@vger.kernel.org
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/task.py |   35 +++++++++++++++++++++++++++++++++++
 1 files changed, 35 insertions(+), 0 deletions(-)

diff --git a/scripts/gdb/task.py b/scripts/gdb/task.py
index fe42eb0..8e96238 100644
--- a/scripts/gdb/task.py
+++ b/scripts/gdb/task.py
@@ -62,3 +62,38 @@ class LxTaskByPidFunc(gdb.Function):
 			raise gdb.GdbError("No task of PID " + str(pid))
 
 LxTaskByPidFunc()
+
+
+thread_info_type = CachedType("struct thread_info")
+
+ia64_task_size = None
+
+def get_thread_info(task):
+	global thread_info_type
+	thread_info_ptr_type = thread_info_type.get_type().pointer()
+	if is_target_arch("ia64"):
+		global ia64_task_size
+		if ia64_task_size == None:
+			ia64_task_size = gdb.parse_and_eval(
+						"sizeof(struct task_struct)")
+		thread_info_addr = task.address + ia64_task_size
+		thread_info = thread_info_addr.cast(thread_info_ptr_type)
+	else:
+		thread_info = task['stack'].cast(thread_info_ptr_type)
+	return thread_info.dereference()
+
+
+class LxThreadInfoFunc (gdb.Function):
+	# Calculate Linux thread_info from task variable.
+	__doc__ = "Calculate Linux thread_info from task variable.\n" \
+		  "\n" \
+		  "$lx_thread_info(TASK): Given TASK, return the corresponding thread_info\n" \
+		  "variable."
+
+	def __init__(self):
+		super(LxThreadInfoFunc, self).__init__("lx_thread_info")
+
+	def invoke(self, task):
+		return get_thread_info(task)
+
+LxThreadInfoFunc()
-- 
1.7.3.4

[PATCH v5 15/20] scripts/gdb: Add get_gdbserver_type helper

[Jan Kiszka]

This helper probes the type of the gdb server. Supported are QEMU and
KGDB so far. Knowledge about the gdb server is required e.g. to retrieve
the current CPU or current task.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/utils.py |   35 +++++++++++++++++++++++++++++++++++
 1 files changed, 35 insertions(+), 0 deletions(-)

diff --git a/scripts/gdb/utils.py b/scripts/gdb/utils.py
index bcf4278..d381f63 100644
--- a/scripts/gdb/utils.py
+++ b/scripts/gdb/utils.py
@@ -121,3 +121,38 @@ def is_target_arch(arch):
 	if target_arch == None:
 		target_arch = gdb.execute("show architecture", False, True)
 	return target_arch.find(arch) >= 0
+
+
+GDBSERVER_QEMU = 0
+GDBSERVER_KGDB = 1
+gdbserver_type = None
+
+def get_gdbserver_type():
+	def exit_handler(event):
+		global gdbserver_type
+		gdbserver_type = None
+		gdb.events.exited.disconnect(exit_handler)
+
+	def probe_qemu():
+		try:
+			return gdb.execute("monitor info version", False,
+					   True) != ""
+		except:
+			return False
+
+	def probe_kgdb():
+		try:
+			thread_info = gdb.execute("info thread 2", False, True)
+			return thread_info.find("shadowCPU0") >= 0
+		except:
+			return False
+
+	global gdbserver_type
+	if gdbserver_type == None:
+		if probe_qemu():
+			gdbserver_type = GDBSERVER_QEMU
+		elif probe_kgdb():
+			gdbserver_type = GDBSERVER_KGDB
+		if gdbserver_type != None:
+			gdb.events.exited.connect(exit_handler)
+	return gdbserver_type
-- 
1.7.3.4

[PATCH v5 16/20] scripts/gdb: Add internal helper and convenience function for per-cpu lookup

[Jan Kiszka]

This function allows to obtain a per-cpu variable, either of the current
or an explicitly specified CPU.

Note: sparc64 version is untested.

CC: "David S. Miller" <davem@davemloft.net>
CC: sparclinux@vger.kernel.org
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/percpu.py      |   61 ++++++++++++++++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |    1 +
 2 files changed, 62 insertions(+), 0 deletions(-)
 create mode 100644 scripts/gdb/percpu.py

diff --git a/scripts/gdb/percpu.py b/scripts/gdb/percpu.py
new file mode 100644
index 0000000..864962c
--- /dev/null
+++ b/scripts/gdb/percpu.py
@@ -0,0 +1,61 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  per-cpu tools
+#
+# Copyright (c) Siemens AG, 2011-2013
+#
+# Authors:
+#  Jan Kiszka <jan.kiszka@siemens.com>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import gdb
+
+from utils import *
+from task import *
+
+MAX_CPUS = 4096
+
+def get_current_cpu():
+	if get_gdbserver_type() == GDBSERVER_QEMU:
+		return gdb.selected_thread().num - 1
+	elif get_gdbserver_type() == GDBSERVER_KGDB:
+		tid = gdb.selected_thread().ptid[2]
+		if tid > (0x100000000 - MAX_CPUS - 2):
+			return 0x100000000 - tid - 2
+		else:
+			return get_thread_info(get_task_by_pid(tid))['cpu']
+	else:
+		raise gdb.GdbError("Sorry, obtaining the current CPU is "
+				   "not yet supported with this gdb server.")
+
+def per_cpu(var_ptr, cpu):
+	if cpu == -1:
+		cpu = get_current_cpu()
+	if is_target_arch("sparc:v9"):
+		offset = gdb.parse_and_eval("trap_block[" + str(cpu) +
+					    "].__per_cpu_base")
+	else:
+		offset = gdb.parse_and_eval("__per_cpu_offset[" + str(cpu) +
+					    "]")
+	pointer = var_ptr.cast(get_long_type()) + offset
+	return pointer.cast(var_ptr.type).dereference()
+
+
+class PerCpu(gdb.Function):
+	__doc__ = "Return per-cpu variable.\n" \
+		  "\n" \
+		  "$lx_per_cpu(\"VAR\"[, CPU]): Return the per-cpu variable called VAR for the\n" \
+		  "given CPU number. If CPU is omitted, the CPU of the current context is used.\n" \
+		  "Note that VAR has to be quoted as string."
+
+	def __init__(self):
+		super(PerCpu, self).__init__("lx_per_cpu")
+
+	def invoke(self, var_name, cpu = -1):
+		var_ptr = gdb.parse_and_eval("&" + var_name.string())
+		return per_cpu(var_ptr, cpu)
+
+PerCpu()
diff --git a/scripts/gdb/vmlinux-gdb.py b/scripts/gdb/vmlinux-gdb.py
index e1c5dbc..b143808 100644
--- a/scripts/gdb/vmlinux-gdb.py
+++ b/scripts/gdb/vmlinux-gdb.py
@@ -26,3 +26,4 @@ else:
 	import module
 	import dmesg
 	import task
+	import percpu
-- 
1.7.3.4

[PATCH v5 17/20] scripts/gdb: Add lx_current convenience function

[Jan Kiszka]

This is a shorthand for *$lx_per_cpu("current_task"), i.e. a convenience
function to retrieve the currently running task of the active context.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/percpu.py |   16 ++++++++++++++++
 1 files changed, 16 insertions(+), 0 deletions(-)

diff --git a/scripts/gdb/percpu.py b/scripts/gdb/percpu.py
index 864962c..4dab8d3 100644
--- a/scripts/gdb/percpu.py
+++ b/scripts/gdb/percpu.py
@@ -59,3 +59,19 @@ class PerCpu(gdb.Function):
 		return per_cpu(var_ptr, cpu)
 
 PerCpu()
+
+
+class LxCurrentFunc(gdb.Function):
+	__doc__ = "Return current task.\n" \
+		  "\n" \
+		  "$lx_current([CPU]): Return the per-cpu task variable for the given CPU\n" \
+		  "number. If CPU is omitted, the CPU of the current context is used."
+
+	def __init__(self):
+		super(LxCurrentFunc, self).__init__("lx_current")
+
+	def invoke(self, cpu = -1):
+		var_ptr = gdb.parse_and_eval("&current_task")
+		return per_cpu(var_ptr, cpu).dereference()
+
+LxCurrentFunc()
-- 
1.7.3.4

[PATCH v5 18/20] scripts/gdb: Add helper to iterate over CPU masks

[Jan Kiszka]

Will be used first to count module references. It is optimized to read
the mask only once per stop and to minimize the loop lengths.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/percpu.py |   33 +++++++++++++++++++++++++++++++++
 1 files changed, 33 insertions(+), 0 deletions(-)

diff --git a/scripts/gdb/percpu.py b/scripts/gdb/percpu.py
index 4dab8d3..bc04188 100644
--- a/scripts/gdb/percpu.py
+++ b/scripts/gdb/percpu.py
@@ -43,6 +43,39 @@ def per_cpu(var_ptr, cpu):
 	pointer = var_ptr.cast(get_long_type()) + offset
 	return pointer.cast(var_ptr.type).dereference()
 
+cpu_mask = { }
+
+def for_each_cpu(mask_name, func, arg = None):
+	def invalidate_handler(event):
+		global cpu_online_mask
+		cpu_mask = { }
+		gdb.events.stop.disconnect(invalidate_handler)
+		gdb.events.new_objfile.disconnect(invalidate_handler)
+
+	global cpu_mask
+	mask = None
+	if mask_name in cpu_mask:
+		mask = cpu_mask[mask_name]
+	if mask == None:
+		mask = gdb.parse_and_eval(mask_name + ".bits")
+		cpu_mask[mask_name] = mask
+		gdb.events.stop.connect(invalidate_handler)
+		gdb.events.new_objfile.connect(invalidate_handler)
+
+	max_cpu_id = mask.type.sizeof * 8
+	bits_per_entry = mask[0].type.sizeof * 8
+	for entry in range(max_cpu_id / bits_per_entry):
+		bits = mask[entry]
+		if bits == 0:
+			continue
+		for bit in range(bits_per_entry):
+			if bits & 1:
+				cpu = entry * bits_per_entry + bit
+				func(cpu, arg)
+			bits >>= 1
+			if bits == 0:
+				break
+
 
 class PerCpu(gdb.Function):
 	__doc__ = "Return per-cpu variable.\n" \
-- 
1.7.3.4

[PATCH v5 19/20] scripts/gdb: Add lx-lsmod command

[Jan Kiszka]

This adds a lsmod-like command to list all currently loaded modules of
the target.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/gdb/module.py |   43 +++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 43 insertions(+), 0 deletions(-)

diff --git a/scripts/gdb/module.py b/scripts/gdb/module.py
index 333729a..101d2e0 100644
--- a/scripts/gdb/module.py
+++ b/scripts/gdb/module.py
@@ -15,6 +15,7 @@ import gdb
 import os
 import string
 
+from percpu import *
 from utils import *
 
 module_type = CachedType("struct module")
@@ -104,3 +105,45 @@ class LxModvar(gdb.Function):
 		return var_addr.cast(var.type.pointer()).dereference()
 
 LxModvar()
+
+
+class LinuxLsmod(gdb.Command):
+	__doc__ = "List currently loaded modules."
+
+	_module_use_type = CachedType("struct module_use")
+
+	def __init__(self):
+		super(LinuxLsmod, self).__init__("lx-lsmod", gdb.COMMAND_DATA)
+
+	def invoke(self, arg, from_tty):
+		def print_module(module, arg):
+			def collect_ref(cpu, arg):
+				refptr = per_cpu(arg['refptr'], cpu)
+				arg['ref'] += refptr['incs']
+				arg['ref'] -= refptr['decs']
+
+			arg = { 'refptr': module['refptr'], 'ref': 0 }
+			for_each_cpu("cpu_possible_mask", collect_ref, arg)
+
+			print "%s" % module['module_core'] + \
+			      " %-19s" % module['name'].string() + \
+			      " %8s" % module['core_size'] + \
+			      "  %d" % arg['ref'],
+
+			source_list = module['source_list']
+			t = self._module_use_type.get_type().pointer()
+			entry = source_list['next']
+			first = True
+			while entry != source_list.address:
+				use = container_of(entry, t, "source_list")
+				gdb.write((" " if first else ",") + \
+					  use['source']['name'].string())
+				first = False
+				entry = entry['next']
+			print
+
+		print "Address%s    Module                  Size  Used by" % \
+		      "        " if get_long_type().sizeof == 8 else ""
+		for_each_module(print_module)
+
+LinuxLsmod()
-- 
1.7.3.4

[PATCH v5 20/20] scripts/gdb: Add basic documentation

[Jan Kiszka]

CC: Rob Landley <rob@landley.net>
CC: linux-doc@vger.kernel.org
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 Documentation/gdb-kernel-debugging.txt |  155 ++++++++++++++++++++++++++++++++
 1 files changed, 155 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/gdb-kernel-debugging.txt

diff --git a/Documentation/gdb-kernel-debugging.txt b/Documentation/gdb-kernel-debugging.txt
new file mode 100644
index 0000000..0ea46e1
--- /dev/null
+++ b/Documentation/gdb-kernel-debugging.txt
@@ -0,0 +1,155 @@
+Debugging kernel and modules via gdb
+====================================
+
+The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
+interfaces allow to debug the Linux kernel and its modules during runtime
+using gdb. Gdb comes with a powerful scripting interface for python. The
+kernel provides a collection of helper scripts that can simplify typical
+kernel debugging steps. This is a short tutorial about how to enable and use
+them. It focuses on QEMU/KVM virtual machines as target, but the examples can
+be transferred to the other gdb stubs as well.
+
+
+Requirements
+------------
+
+ o gdb 7.1+ (recommended: 7.3+) with python support enabled (typically true
+   for distributions)
+
+
+Setup
+-----
+
+ o Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
+   www.qemu.org for more details)
+
+ o Build the kernel with CONFIG_DEBUG_INFO enabled, but leave
+   CONFIG_DEBUG_INFO_REDUCED off
+
+ o Install that kernel on the guest.
+
+   Alternatively, QEMU allows to boot the kernel directly using -kernel,
+   -append, -initrd command line switches. This is generally only useful if
+   you do not depend on modules. See QEMU documentation for more details on
+   this mode.
+
+ o Enable the gdb stub of QEMU/KVM, either
+    - at VM startup time by appending "-s" to the QEMU command line
+   or
+    - during runtime by issuing "gdbserver" from the QEMU monitor
+      console
+
+ o cd /path/to/linux-build
+
+ o Start gdb: gdb vmlinux
+
+   Note: Some distros may restrict auto-loading of gdb scripts to known safe
+   directories. In case gdb reports to refuse loading vmlinux-gdb.py, add
+
+    add-add-auto-load-safe-path /path/to/linux-build
+
+   to ~/.gdbinit. See gdb help for more details.
+
+ o Attach to the booted guest:
+    (gdb) target remote :1234
+
+
+Examples of using the Linux-provided gdb helpers
+------------------------------------------------
+
+ o Load module (and main kernel) symbols:
+    (gdb) lx-symbols
+    loading vmlinux
+    scanning for modules in /home/user/linux/build
+    loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
+    loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
+    loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
+    loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
+    loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
+    ...
+    loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
+
+ o Set a breakpoint on some not yet loaded module function, e.g.:
+    (gdb) b btrfs_init_sysfs
+    Function "btrfs_init_sysfs" not defined.
+    Make breakpoint pending on future shared library load? (y or [n]) y
+    Breakpoint 1 (btrfs_init_sysfs) pending.
+
+ o Continue the target
+    (gdb) c
+
+ o Load the module on the target and watch the symbols being loaded as well as
+   the breakpoint hit:
+    loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
+    loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
+    loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
+    loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
+
+    Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
+    36              btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
+
+ o Dump the log buffer of the target kernel:
+    (gdb) lx-dmesg
+    [     0.000000] Initializing cgroup subsys cpuset
+    [     0.000000] Initializing cgroup subsys cpu
+    [     0.000000] Linux version 3.8.0-rc4-dbg+ (...
+    [     0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
+    [     0.000000] e820: BIOS-provided physical RAM map:
+    [     0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
+    [     0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
+    ....
+
+ o Examine fields of the current task struct:
+    (gdb) p $lx_current().pid
+    $1 = 4998
+    (gdb) p $lx_current().comm
+    $2 = "modprobe\000\000\000\000\000\000\000"
+
+ o Make use of the per-cpu helper for the current or a specified CPU:
+    (gdb) p $lx_per_cpu("runqueues").nr_running
+    $3 = 1
+    (gdb) p $lx_per_cpu("runqueues", 2).nr_running
+    $4 = 0
+
+ o Dig into hrtimers using the container_of helper:
+    (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
+    (gdb) p *$container_of($next, "struct hrtimer", "node")
+    $5 = {
+      node = {
+        node = {
+          __rb_parent_color = 18446612133355256072,
+          rb_right = 0x0 <irq_stack_union>,
+          rb_left = 0x0 <irq_stack_union>
+        },
+        expires = {
+          tv64 = 1835268000000
+        }
+      },
+      _softexpires = {
+        tv64 = 1835268000000
+      },
+      function = 0xffffffff81078232 <tick_sched_timer>,
+      base = 0xffff88003fd0d6f0,
+      state = 1,
+      start_pid = 0,
+      start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
+      start_comm = "swapper/2\000\000\000\000\000\000"
+    }
+
+
+List of commands and helper
+---------------------------
+
+The number of commands and convenience helpers may evolve over the time, this
+is just a snapshot of the initial version:
+
+ (gdb) apropos lx
+ function lx_current -- Return current task
+ function lx_module -- Find module by name and return the module variable
+ function lx_modvar -- Return global variable of a module
+ function lx_per_cpu -- Return per-cpu variable
+ function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
+ function lx_thread_info -- Calculate Linux thread_info from task variable
+ lx-dmesg -- Print Linux kernel log buffer
+ lx-lsmod -- List currently loaded modules
+ lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
-- 
1.7.3.4

Re: [PATCH v5 16/20] scripts/gdb: Add internal helper and convenience function for per-cpu lookup

[Borislav Petkov]

On Tue, Jan 29, 2013 at 01:37:59PM +0100, Jan Kiszka wrote:
> This function allows to obtain a per-cpu variable, either of the current
> or an explicitly specified CPU.
> 
> Note: sparc64 version is untested.
> 
> CC: "David S. Miller" <davem@davemloft.net>
> CC: sparclinux@vger.kernel.org
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  scripts/gdb/percpu.py      |   61 ++++++++++++++++++++++++++++++++++++++++++++
>  scripts/gdb/vmlinux-gdb.py |    1 +
>  2 files changed, 62 insertions(+), 0 deletions(-)
>  create mode 100644 scripts/gdb/percpu.py
> 
> diff --git a/scripts/gdb/percpu.py b/scripts/gdb/percpu.py
> new file mode 100644
> index 0000000..864962c
> --- /dev/null
> +++ b/scripts/gdb/percpu.py
> @@ -0,0 +1,61 @@
> +#
> +# gdb helper commands and functions for Linux kernel debugging
> +#
> +#  per-cpu tools

This is actually a very cool thing for CPU guys: it can show what kind
of hybrid CPUs they've been glueing together, like the following, for
example:

...
(gdb) p $lx_per_cpu("cpu_info").x86_virt_bits
$5 = 48 '0'
(gdb) p $lx_per_cpu("cpu_info").x86_vendor_id
$6 = "GenuineIntel\000\000\000"
(gdb) p $lx_per_cpu("cpu_info").x86_model_id
$7 = "AMD Phenom(tm) 9550 Quad-Core Processor", '\000' <repeats 24 times>
(gdb)

Since when does Intel produce CPUs called "AMD Phenom(tm) 9550 Quad-Core
Processor"? .. hahaha..

> +#
> +# Copyright (c) Siemens AG, 2011-2013
> +#
> +# Authors:
> +#  Jan Kiszka <jan.kiszka@siemens.com>
> +#
> +# This work is licensed under the terms of the GNU GPL version 2.
> +#
> +
> +import gdb
> +
> +from utils import *
> +from task import *
> +
> +MAX_CPUS = 4096
> +
> +def get_current_cpu():
> +	if get_gdbserver_type() == GDBSERVER_QEMU:
> +		return gdb.selected_thread().num - 1
> +	elif get_gdbserver_type() == GDBSERVER_KGDB:
> +		tid = gdb.selected_thread().ptid[2]
> +		if tid > (0x100000000 - MAX_CPUS - 2):
> +			return 0x100000000 - tid - 2
> +		else:
> +			return get_thread_info(get_task_by_pid(tid))['cpu']
> +	else:
> +		raise gdb.GdbError("Sorry, obtaining the current CPU is "
> +				   "not yet supported with this gdb server.")
> +
> +def per_cpu(var_ptr, cpu):
> +	if cpu == -1:
> +		cpu = get_current_cpu()
> +	if is_target_arch("sparc:v9"):
> +		offset = gdb.parse_and_eval("trap_block[" + str(cpu) +
> +					    "].__per_cpu_base")
> +	else:
> +		offset = gdb.parse_and_eval("__per_cpu_offset[" + str(cpu) +
> +					    "]")
> +	pointer = var_ptr.cast(get_long_type()) + offset
> +	return pointer.cast(var_ptr.type).dereference()
> +
> +
> +class PerCpu(gdb.Function):
> +	__doc__ = "Return per-cpu variable.\n" \
> +		  "\n" \
> +		  "$lx_per_cpu(\"VAR\"[, CPU]): Return the per-cpu variable called VAR for the\n" \
> +		  "given CPU number. If CPU is omitted, the CPU of the current context is used.\n" \
> +		  "Note that VAR has to be quoted as string."


Ok, seriously now:

apropos shows the "Return per-cpu... " line above. Have you found out
which gdb command shows the rest? help and info both say "Undefined
command".

Thanks.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH v5 16/20] scripts/gdb: Add internal helper and convenience function for per-cpu lookup

[Jan Kiszka]

On 2013-01-29 14:51, Borislav Petkov wrote:
> On Tue, Jan 29, 2013 at 01:37:59PM +0100, Jan Kiszka wrote:
>> This function allows to obtain a per-cpu variable, either of the current
>> or an explicitly specified CPU.
>>
>> Note: sparc64 version is untested.
>>
>> CC: "David S. Miller" <davem@davemloft.net>
>> CC: sparclinux@vger.kernel.org
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  scripts/gdb/percpu.py      |   61 ++++++++++++++++++++++++++++++++++++++++++++
>>  scripts/gdb/vmlinux-gdb.py |    1 +
>>  2 files changed, 62 insertions(+), 0 deletions(-)
>>  create mode 100644 scripts/gdb/percpu.py
>>
>> diff --git a/scripts/gdb/percpu.py b/scripts/gdb/percpu.py
>> new file mode 100644
>> index 0000000..864962c
>> --- /dev/null
>> +++ b/scripts/gdb/percpu.py
>> @@ -0,0 +1,61 @@
>> +#
>> +# gdb helper commands and functions for Linux kernel debugging
>> +#
>> +#  per-cpu tools
> 
> This is actually a very cool thing for CPU guys: it can show what kind
> of hybrid CPUs they've been glueing together, like the following, for
> example:
> 
> ...
> (gdb) p $lx_per_cpu("cpu_info").x86_virt_bits
> $5 = 48 '0'
> (gdb) p $lx_per_cpu("cpu_info").x86_vendor_id
> $6 = "GenuineIntel\000\000\000"
> (gdb) p $lx_per_cpu("cpu_info").x86_model_id
> $7 = "AMD Phenom(tm) 9550 Quad-Core Processor", '\000' <repeats 24 times>
> (gdb)
> 
> Since when does Intel produce CPUs called "AMD Phenom(tm) 9550 Quad-Core
> Processor"? .. hahaha..

Let me guess: You are dumping a weird QEMU/KVM CPU, right?

> 
>> +#
>> +# Copyright (c) Siemens AG, 2011-2013
>> +#
>> +# Authors:
>> +#  Jan Kiszka <jan.kiszka@siemens.com>
>> +#
>> +# This work is licensed under the terms of the GNU GPL version 2.
>> +#
>> +
>> +import gdb
>> +
>> +from utils import *
>> +from task import *
>> +
>> +MAX_CPUS = 4096
>> +
>> +def get_current_cpu():
>> +	if get_gdbserver_type() == GDBSERVER_QEMU:
>> +		return gdb.selected_thread().num - 1
>> +	elif get_gdbserver_type() == GDBSERVER_KGDB:
>> +		tid = gdb.selected_thread().ptid[2]
>> +		if tid > (0x100000000 - MAX_CPUS - 2):
>> +			return 0x100000000 - tid - 2
>> +		else:
>> +			return get_thread_info(get_task_by_pid(tid))['cpu']
>> +	else:
>> +		raise gdb.GdbError("Sorry, obtaining the current CPU is "
>> +				   "not yet supported with this gdb server.")
>> +
>> +def per_cpu(var_ptr, cpu):
>> +	if cpu == -1:
>> +		cpu = get_current_cpu()
>> +	if is_target_arch("sparc:v9"):
>> +		offset = gdb.parse_and_eval("trap_block[" + str(cpu) +
>> +					    "].__per_cpu_base")
>> +	else:
>> +		offset = gdb.parse_and_eval("__per_cpu_offset[" + str(cpu) +
>> +					    "]")
>> +	pointer = var_ptr.cast(get_long_type()) + offset
>> +	return pointer.cast(var_ptr.type).dereference()
>> +
>> +
>> +class PerCpu(gdb.Function):
>> +	__doc__ = "Return per-cpu variable.\n" \
>> +		  "\n" \
>> +		  "$lx_per_cpu(\"VAR\"[, CPU]): Return the per-cpu variable called VAR for the\n" \
>> +		  "given CPU number. If CPU is omitted, the CPU of the current context is used.\n" \
>> +		  "Note that VAR has to be quoted as string."
> 
> 
> Ok, seriously now:
> 
> apropos shows the "Return per-cpu... " line above. Have you found out
> which gdb command shows the rest? help and info both say "Undefined
> command".

help function lx_...

It took me a while to find this, too.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v5 14/20] scripts/gdb: Add internal helper and convenience function to retrieve thread_info

[Borislav Petkov]

On Tue, Jan 29, 2013 at 01:37:57PM +0100, Jan Kiszka wrote:
> Add the internal helper get_thread_info that calculated the thread_info
> from a given task variable. Also export this service as a convenience
> function.
> 
> Note: ia64 version is untested.
> 
> CC: Tony Luck <tony.luck@intel.com>
> CC: Fenghua Yu <fenghua.yu@intel.com>
> CC: linux-ia64@vger.kernel.org
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---

[ … ]

> +class LxThreadInfoFunc (gdb.Function):
> +	# Calculate Linux thread_info from task variable.
> +	__doc__ = "Calculate Linux thread_info from task variable.\n" \
> +		  "\n" \
> +		  "$lx_thread_info(TASK): Given TASK, return the corresponding thread_info\n" \
> +		  "variable."
> +
> +	def __init__(self):
> +		super(LxThreadInfoFunc, self).__init__("lx_thread_info")
> +
> +	def invoke(self, task):
> +		return get_thread_info(task)
> +
> +LxThreadInfoFunc()

Nice, you can plug commands into one-another:

(gdb) p $lx_thread_info($lx_current())
$12 = {task = 0xffffffff81a14440, exec_domain = 0xffffffff81a21e00, flags = 0, status = 0, cpu = 0, 
  preempt_count = 1, addr_limit = {seg = 18446744073709551615}, restart_block = {
    fn = 0xffffffff81054cd0 <do_no_restart_syscall>, {futex = {uaddr = 0x0, val = 0, flags = 0, bitset = 0, 
        time = 0, uaddr2 = 0x0}, nanosleep = {clockid = 0, rmtp = 0x0, compat_rmtp = 0x0, expires = 0}, 
      poll = {ufds = 0x0, nfds = 0, has_timeout = 0, tv_sec = 0, tv_nsec = 0}}}, sysenter_return = 0x0, 
  sig_on_uaccess_error = 0, uaccess_err = 0}

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH v5 16/20] scripts/gdb: Add internal helper and convenience function for per-cpu lookup

[Borislav Petkov]

On Tue, Jan 29, 2013 at 02:56:56PM +0100, Jan Kiszka wrote:
> Let me guess: You are dumping a weird QEMU/KVM CPU, right?

Nah, I actually have the silicon! :-)

Joking, of course. I wish. I'm booting the guest with -cpu phenom (it
has been like that since forever in my boot-kernel-in-kvm script) but
the host is Intel.

If I do this on an AMD host, all is ok:

(gdb) p $lx_per_cpu("cpu_info").x86_vendor_id
$1 = "AuthenticAMD\000\000\000"
(gdb) p $lx_per_cpu("cpu_info").x86_model_id
$2 = "AMD Phenom(tm) 9550 Quad-Core Processor", '\000' <repeats 24 times>
(gdb)

[ … ]

> >> +class PerCpu(gdb.Function):
> >> +	__doc__ = "Return per-cpu variable.\n" \
> >> +		  "\n" \
> >> +		  "$lx_per_cpu(\"VAR\"[, CPU]): Return the per-cpu variable called VAR for the\n" \
> >> +		  "given CPU number. If CPU is omitted, the CPU of the current context is used.\n" \
> >> +		  "Note that VAR has to be quoted as string."
> > 
> > 
> > Ok, seriously now:
> > 
> > apropos shows the "Return per-cpu... " line above. Have you found out
> > which gdb command shows the rest? help and info both say "Undefined
> > command".
> 
> help function lx_...
> 
> It took me a while to find this, too.

Maybe worth documenting it then at the end of gdb-kernel-debugging.txt?

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH v5 00/20] Add gdb python scripts as kernel debugging helpers

[Borislav Petkov]

On Tue, Jan 29, 2013 at 01:37:43PM +0100, Jan Kiszka wrote:
> Version 5 comes with the following changes:
>  - moved tutorial into Documentation/gdb-kernel-debugging.txt
>  - improved caching of gdb.Type objects, ensure they are in sync with
>    currently loaded symbols
>  - added new functions and commands
>     - lx_module -- Find module by name and return the module variable
>     - lx_modvar -- Return global variable of a module
>     - lx-lsmod -- List currently loaded modules
> 
> See http://lkml.indiana.edu/hypermail/linux/kernel/1210.0/01598.html for
> the original description and
> 
>     git://git.kiszka.org/linux.git queues/gdb-scripts
> 
> for the latest version.

Ok, I've seldomly shown any feelings when acking a patch{,set} besides
maybe "hm, ok, fine" but this patchset is very very cool and it is an
enormous fun playing with it.

So maybe this is the most enthusiasm I've shown sofar:

Emphatically-acked-by: Borislav Petkov <bp@suse.de>

Thanks Jan, very good work!

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH v5 16/20] scripts/gdb: Add internal helper and convenience function for per-cpu lookup

[Jan Kiszka]

On 2013-01-29 15:12, Borislav Petkov wrote:
> On Tue, Jan 29, 2013 at 02:56:56PM +0100, Jan Kiszka wrote:
>> Let me guess: You are dumping a weird QEMU/KVM CPU, right?
> 
> Nah, I actually have the silicon! :-)
> 
> Joking, of course. I wish. I'm booting the guest with -cpu phenom (it
> has been like that since forever in my boot-kernel-in-kvm script) but
> the host is Intel.
> 
> If I do this on an AMD host, all is ok:
> 
> (gdb) p $lx_per_cpu("cpu_info").x86_vendor_id
> $1 = "AuthenticAMD\000\000\000"
> (gdb) p $lx_per_cpu("cpu_info").x86_model_id
> $2 = "AMD Phenom(tm) 9550 Quad-Core Processor", '\000' <repeats 24 times>
> (gdb)

Remains a bug of QEMU, though possibly a minor one.

> 
> [ … ]
> 
>>>> +class PerCpu(gdb.Function):
>>>> +	__doc__ = "Return per-cpu variable.\n" \
>>>> +		  "\n" \
>>>> +		  "$lx_per_cpu(\"VAR\"[, CPU]): Return the per-cpu variable called VAR for the\n" \
>>>> +		  "given CPU number. If CPU is omitted, the CPU of the current context is used.\n" \
>>>> +		  "Note that VAR has to be quoted as string."
>>>
>>>
>>> Ok, seriously now:
>>>
>>> apropos shows the "Return per-cpu... " line above. Have you found out
>>> which gdb command shows the rest? help and info both say "Undefined
>>> command".
>>
>> help function lx_...
>>
>> It took me a while to find this, too.
> 
> Maybe worth documenting it then at the end of gdb-kernel-debugging.txt?
> 

Done. Will come with v6 (if needed), otherwise as an add-on patch.

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

[PATCH v6 20/20] scripts/gdb: Add basic documentation

[Jan Kiszka]

CC: Rob Landley <rob@landley.net>
CC: linux-doc@vger.kernel.org
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

Just to close the request Boris had regarding help on the commands.

Changes in v6:
 - explain how to obtain help on commands and functions
 - minor wording fixes

 Documentation/gdb-kernel-debugging.txt |  158 ++++++++++++++++++++++++++++++++
 1 files changed, 158 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/gdb-kernel-debugging.txt

diff --git a/Documentation/gdb-kernel-debugging.txt b/Documentation/gdb-kernel-debugging.txt
new file mode 100644
index 0000000..1b5b3da
--- /dev/null
+++ b/Documentation/gdb-kernel-debugging.txt
@@ -0,0 +1,158 @@
+Debugging kernel and modules via gdb
+====================================
+
+The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
+interfaces allow to debug the Linux kernel and its modules during runtime
+using gdb. Gdb comes with a powerful scripting interface for python. The
+kernel provides a collection of helper scripts that can simplify typical
+kernel debugging steps. This is a short tutorial about how to enable and use
+them. It focuses on QEMU/KVM virtual machines as target, but the examples can
+be transferred to the other gdb stubs as well.
+
+
+Requirements
+------------
+
+ o gdb 7.1+ (recommended: 7.3+) with python support enabled (typically true
+   for distributions)
+
+
+Setup
+-----
+
+ o Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
+   www.qemu.org for more details)
+
+ o Build the kernel with CONFIG_DEBUG_INFO enabled, but leave
+   CONFIG_DEBUG_INFO_REDUCED off
+
+ o Install that kernel on the guest.
+
+   Alternatively, QEMU allows to boot the kernel directly using -kernel,
+   -append, -initrd command line switches. This is generally only useful if
+   you do not depend on modules. See QEMU documentation for more details on
+   this mode.
+
+ o Enable the gdb stub of QEMU/KVM, either
+    - at VM startup time by appending "-s" to the QEMU command line
+   or
+    - during runtime by issuing "gdbserver" from the QEMU monitor
+      console
+
+ o cd /path/to/linux-build
+
+ o Start gdb: gdb vmlinux
+
+   Note: Some distros may restrict auto-loading of gdb scripts to known safe
+   directories. In case gdb reports to refuse loading vmlinux-gdb.py, add
+
+    add-add-auto-load-safe-path /path/to/linux-build
+
+   to ~/.gdbinit. See gdb help for more details.
+
+ o Attach to the booted guest:
+    (gdb) target remote :1234
+
+
+Examples of using the Linux-provided gdb helpers
+------------------------------------------------
+
+ o Load module (and main kernel) symbols:
+    (gdb) lx-symbols
+    loading vmlinux
+    scanning for modules in /home/user/linux/build
+    loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
+    loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
+    loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
+    loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
+    loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
+    ...
+    loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
+
+ o Set a breakpoint on some not yet loaded module function, e.g.:
+    (gdb) b btrfs_init_sysfs
+    Function "btrfs_init_sysfs" not defined.
+    Make breakpoint pending on future shared library load? (y or [n]) y
+    Breakpoint 1 (btrfs_init_sysfs) pending.
+
+ o Continue the target
+    (gdb) c
+
+ o Load the module on the target and watch the symbols being loaded as well as
+   the breakpoint hit:
+    loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
+    loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
+    loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
+    loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
+
+    Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
+    36              btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
+
+ o Dump the log buffer of the target kernel:
+    (gdb) lx-dmesg
+    [     0.000000] Initializing cgroup subsys cpuset
+    [     0.000000] Initializing cgroup subsys cpu
+    [     0.000000] Linux version 3.8.0-rc4-dbg+ (...
+    [     0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
+    [     0.000000] e820: BIOS-provided physical RAM map:
+    [     0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
+    [     0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
+    ....
+
+ o Examine fields of the current task struct:
+    (gdb) p $lx_current().pid
+    $1 = 4998
+    (gdb) p $lx_current().comm
+    $2 = "modprobe\000\000\000\000\000\000\000"
+
+ o Make use of the per-cpu function for the current or a specified CPU:
+    (gdb) p $lx_per_cpu("runqueues").nr_running
+    $3 = 1
+    (gdb) p $lx_per_cpu("runqueues", 2).nr_running
+    $4 = 0
+
+ o Dig into hrtimers using the container_of helper:
+    (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
+    (gdb) p *$container_of($next, "struct hrtimer", "node")
+    $5 = {
+      node = {
+        node = {
+          __rb_parent_color = 18446612133355256072,
+          rb_right = 0x0 <irq_stack_union>,
+          rb_left = 0x0 <irq_stack_union>
+        },
+        expires = {
+          tv64 = 1835268000000
+        }
+      },
+      _softexpires = {
+        tv64 = 1835268000000
+      },
+      function = 0xffffffff81078232 <tick_sched_timer>,
+      base = 0xffff88003fd0d6f0,
+      state = 1,
+      start_pid = 0,
+      start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
+      start_comm = "swapper/2\000\000\000\000\000\000"
+    }
+
+
+List of commands and functions
+------------------------------
+
+The number of commands and convenience functions may evolve over the time,
+this is just a snapshot of the initial version:
+
+ (gdb) apropos lx
+ function lx_current -- Return current task
+ function lx_module -- Find module by name and return the module variable
+ function lx_modvar -- Return global variable of a module
+ function lx_per_cpu -- Return per-cpu variable
+ function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
+ function lx_thread_info -- Calculate Linux thread_info from task variable
+ lx-dmesg -- Print Linux kernel log buffer
+ lx-lsmod -- List currently loaded modules
+ lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
+
+Detailed help can be obtained via "help <command-name>" for commands and "help
+function <function-name>" for convenience functions.
-- 
1.7.3.4

Re: [PATCH v5 20/20] scripts/gdb: Add basic documentation

[Rob Landley]

On 01/29/2013 06:38:03 AM, Jan Kiszka wrote:
> CC: Rob Landley <rob@landley.net>
> CC: linux-doc@vger.kernel.org
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  Documentation/gdb-kernel-debugging.txt |  155  
> ++++++++++++++++++++++++++++++++
>  1 files changed, 155 insertions(+), 0 deletions(-)
>  create mode 100644 Documentation/gdb-kernel-debugging.txt
> 
> diff --git a/Documentation/gdb-kernel-debugging.txt  
> b/Documentation/gdb-kernel-debugging.txt
> new file mode 100644
> index 0000000..0ea46e1
> --- /dev/null
> +++ b/Documentation/gdb-kernel-debugging.txt
> @@ -0,0 +1,155 @@
> +Debugging kernel and modules via gdb
> +====================================
> +
> +The kernel debugger kgdb, hypervisors like QEMU or JTAG-based  
> hardware
> +interfaces allow to debug the Linux kernel and its modules during  
> runtime
> +using gdb.

This could use some clarification.

Technically they're attaching gdb to the hardware using the gdbserver  
protocol, instead of attaching to a specific process like the gdbserver  
client program does. (I think of it as "pid 0": your register state  
isn't a "register profile" but the actual contents of the hardware  
registers, and your memory map is physical.)

Two of these three methods (jtag and emulator) are independent of  
linux, and the fact the hardware you're attached to may or may not be  
running linux (or u-boot, or a bare metal hello world) is irrelevant.  
Only kgdb requires linux to be running to produce/consume gdbserver  
procol with information about current hardware state.

When Linux _is_ running, if you want symbol data you can point gdb at  
an unstripped vmlinux. (Which is an ELF image, the build usually feeds  
that through objdump and glues a wrapper on the front to make it  
bootable on bare metal, but this is the one with symbols still in it.)  
But even with symbols, trying to dig through Linux's various  
abstractions and data structures to find anything belonging to an  
actual process from outside the kernel is a flaming pain, so you've  
made some gdb plugins to help out there. Cool.

> Gdb comes with a powerful scripting interface for python. The
> +kernel provides a collection of helper scripts that can simplify  
> typical
> +kernel debugging steps. This is a short tutorial about how to enable  
> and use
> +them. It focuses on QEMU/KVM virtual machines as target, but the  
> examples can
> +be transferred to the other gdb stubs as well.
> +
> +
> +Requirements
> +------------
> +
> + o gdb 7.1+ (recommended: 7.3+) with python support enabled  
> (typically true
> +   for distributions)
> +
> +
> +Setup
> +-----
> +
> + o Create a virtual Linux machine for QEMU/KVM (see  
> www.linux-kvm.org and
> +   www.qemu.org for more details)

If it helps, I've got a dozen different targets as  
"system-image-*.tar.bz2" at
http://landley.net/aboriginal/bin and the "run-emulator.sh" script  
inside each one gives the qemu command line to boot it to a shell  
prompt. At that shell prompt you can grab the relevant kernel config  
with "zcat /proc/config.gz" to build your own kernel, and the  
toolchains I did so with are in the same directory. (The README in  
there describes other categories of file in the directory.)

That at least gives you a "known working" version to compare your own  
attempts against.

> + o Build the kernel with CONFIG_DEBUG_INFO enabled, but leave
> +   CONFIG_DEBUG_INFO_REDUCED off
> +
> + o Install that kernel on the guest.
> +
> +   Alternatively, QEMU allows to boot the kernel directly using  
> -kernel,
> +   -append, -initrd command line switches. This is generally only  
> useful if
> +   you do not depend on modules. See QEMU documentation for more  
> details on
> +   this mode.

It's qemu's built-in bootloader, no different than any other  
bootloader. If your hardware needs a module to access the block device  
you want to keep the rest of your modules on, A) you're making a  
peverse architectural decision, B) stick it in the initramfs.

(I could do up an example of that sort of setup if you like. The images  
I linked to above keep a squashfs on a virtual hard drive, I could do  
an initramfs that loads the IDE module and then switch_root to it. I  
just haven't bothered because the easy way works.)

> + o Enable the gdb stub of QEMU/KVM, either
> +    - at VM startup time by appending "-s" to the QEMU command line
> +   or
> +    - during runtime by issuing "gdbserver" from the QEMU monitor
> +      console

Initiallly I thought this document would be more about kgdb, but this  
is about how to use a direct hardware attached debugger (or kgdb  
emulating one). Hence the clarification up top.

> + o cd /path/to/linux-build
> +
> + o Start gdb: gdb vmlinux
> +
> +   Note: Some distros may restrict auto-loading of gdb scripts to  
> known safe
> +   directories. In case gdb reports to refuse loading  
> vmlinux-gdb.py, add
> +
> +    add-add-auto-load-safe-path /path/to/linux-build
> +
> +   to ~/.gdbinit. See gdb help for more details.
> +
> + o Attach to the booted guest:
> +    (gdb) target remote :1234

I've found the "target remote |" option useful, where you pipe the I/O  
to another command that forwards data to the target board.

(I once had to debug a program on a board that only had a serial  
console, and that was always on and attached to another machine. So I  
piped to ssh which ran a term program which launched netcat on the  
board to connect the remote end because the FSF didn't bother to make a  
stdio option for gdbserver so you _have_ to use loopback. Although that  
was a few versions back, maybe they fixed this...)

*shrug* Possibly off topic...

The rest is about what the plugins do, looks fine to me. :)

Rob

> +
> +Examples of using the Linux-provided gdb helpers
> +------------------------------------------------
> +
> + o Load module (and main kernel) symbols:
> +    (gdb) lx-symbols
> +    loading vmlinux
> +    scanning for modules in /home/user/linux/build
> +    loading @0xffffffffa0020000:  
> /home/user/linux/build/net/netfilter/xt_tcpudp.ko
> +    loading @0xffffffffa0016000:  
> /home/user/linux/build/net/netfilter/xt_pkttype.ko
> +    loading @0xffffffffa0002000:  
> /home/user/linux/build/net/netfilter/xt_limit.ko
> +    loading @0xffffffffa00ca000:  
> /home/user/linux/build/net/packet/af_packet.ko
> +    loading @0xffffffffa003c000:  
> /home/user/linux/build/fs/fuse/fuse.ko
> +    ...
> +    loading @0xffffffffa0000000:  
> /home/user/linux/build/drivers/ata/ata_generic.ko
> +
> + o Set a breakpoint on some not yet loaded module function, e.g.:
> +    (gdb) b btrfs_init_sysfs
> +    Function "btrfs_init_sysfs" not defined.
> +    Make breakpoint pending on future shared library load? (y or  
> [n]) y
> +    Breakpoint 1 (btrfs_init_sysfs) pending.
> +
> + o Continue the target
> +    (gdb) c
> +
> + o Load the module on the target and watch the symbols being loaded  
> as well as
> +   the breakpoint hit:
> +    loading @0xffffffffa0034000:  
> /home/user/linux/build/lib/libcrc32c.ko
> +    loading @0xffffffffa0050000:  
> /home/user/linux/build/lib/lzo/lzo_compress.ko
> +    loading @0xffffffffa006e000:  
> /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
> +    loading @0xffffffffa01b1000:  
> /home/user/linux/build/fs/btrfs/btrfs.ko
> +
> +    Breakpoint 1, btrfs_init_sysfs () at  
> /home/user/linux/fs/btrfs/sysfs.c:36
> +    36              btrfs_kset = kset_create_and_add("btrfs", NULL,  
> fs_kobj);
> +
> + o Dump the log buffer of the target kernel:
> +    (gdb) lx-dmesg
> +    [     0.000000] Initializing cgroup subsys cpuset
> +    [     0.000000] Initializing cgroup subsys cpu
> +    [     0.000000] Linux version 3.8.0-rc4-dbg+ (...
> +    [     0.000000] Command line: root=/dev/sda2 resume=/dev/sda1  
> vga=0x314
> +    [     0.000000] e820: BIOS-provided physical RAM map:
> +    [     0.000000] BIOS-e820: [mem  
> 0x0000000000000000-0x000000000009fbff] usable
> +    [     0.000000] BIOS-e820: [mem  
> 0x000000000009fc00-0x000000000009ffff] reserved
> +    ....
> +
> + o Examine fields of the current task struct:
> +    (gdb) p $lx_current().pid
> +    $1 = 4998
> +    (gdb) p $lx_current().comm
> +    $2 = "modprobe\000\000\000\000\000\000\000"
> +
> + o Make use of the per-cpu helper for the current or a specified CPU:
> +    (gdb) p $lx_per_cpu("runqueues").nr_running
> +    $3 = 1
> +    (gdb) p $lx_per_cpu("runqueues", 2).nr_running
> +    $4 = 0
> +
> + o Dig into hrtimers using the container_of helper:
> +    (gdb) set $next =  
> $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
> +    (gdb) p *$container_of($next, "struct hrtimer", "node")
> +    $5 = {
> +      node = {
> +        node = {
> +          __rb_parent_color = 18446612133355256072,
> +          rb_right = 0x0 <irq_stack_union>,
> +          rb_left = 0x0 <irq_stack_union>
> +        },
> +        expires = {
> +          tv64 = 1835268000000
> +        }
> +      },
> +      _softexpires = {
> +        tv64 = 1835268000000
> +      },
> +      function = 0xffffffff81078232 <tick_sched_timer>,
> +      base = 0xffff88003fd0d6f0,
> +      state = 1,
> +      start_pid = 0,
> +      start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
> +      start_comm = "swapper/2\000\000\000\000\000\000"
> +    }
> +
> +
> +List of commands and helper
> +---------------------------
> +
> +The number of commands and convenience helpers may evolve over the  
> time, this
> +is just a snapshot of the initial version:
> +
> + (gdb) apropos lx
> + function lx_current -- Return current task
> + function lx_module -- Find module by name and return the module  
> variable
> + function lx_modvar -- Return global variable of a module
> + function lx_per_cpu -- Return per-cpu variable
> + function lx_task_by_pid -- Find Linux task by PID and return the  
> task_struct variable
> + function lx_thread_info -- Calculate Linux thread_info from task  
> variable
> + lx-dmesg -- Print Linux kernel log buffer
> + lx-lsmod -- List currently loaded modules
> + lx-symbols -- (Re-)load symbols of Linux kernel and currently  
> loaded modules
> --
> 1.7.3.4
> 
> 
> 

Re: [PATCH v5 13/20] scripts/gdb: Add is_target_arch helper

[Tom Tromey]

>>>>> "Jan" == Jan Kiszka <jan.kiszka@siemens.com> writes:

Jan> +def is_target_arch(arch):
Jan> +	global target_arch
Jan> +	if target_arch == None:
Jan> +		target_arch = gdb.execute("show architecture", False, True)
Jan> +	return target_arch.find(arch) >= 0

FYI - the next gdb will have a Frame.architecture method for getting the
architecture of a frame.  You might consider using it when it is
available.  "show architecture" is mildly unfortunate to cache given its
"auto" behavior.

Tom

Re: [PATCH v5 01/20] scripts/gdb: Add infrastructure

[Tom Tromey]

>>>>> "Jan" == Jan Kiszka <jan.kiszka@siemens.com> writes:

Jan> +if gdb_version < "7.1":
Jan> +	print "NOTE: gdb 7.1 or later required for Linux helper scripts " \
Jan> +	      "to work."

FWIW you can just directly feature-test for the things you know you need.
Like:  if not hasattr(gdb, 'parse_and_eval'): ...
In this case, 7.1 is old enough that it probably doesn't matter.
For newer things, though, it is better to feature-test because distros
sometimes forward-port patches, so the version number isn't a reliable
indicator.

Tom

Re: [PATCH v5 05/20] scripts/gdb: Add lx-symbols command

[Tom Tromey]

>>>>> "Jan" == Jan Kiszka <jan.kiszka@siemens.com> writes:

Jan> In addition, the command installs a silent breakpoint in the load_module
Jan> function at the point where the module was loaded but not yet
Jan> initialized. The breakpoint handler will try to load symbols from the
Jan> module files found during lx-symbols execution. This way, breakpoints
Jan> can be set to module initialization functions, and there is usually no
Jan> need to explicitly call lx-symbols after (re-)loading a module.

It is a nice approach, but I wonder whether the kernel's gdb stub could
use the existing qXfer:libraries remote protocol packet.  And, if not,
could we extend gdb to make it work.

Jan> +		def _find_breakpoint_location(self):
Jan> +			breakpoint_match = "^[0-9]*[\t]*err = parse_args\(.*"
Jan> +
Jan> +			src = gdb.execute("list kernel/module.c:load_module",
Jan> +					  to_string = True)

Whatever works -- but I think there are better ways.

The simplest is introducing a function that is called at the right spot
with the right arguments.  It doesn't need to do anything, just be a
name where you can put a breakpoint.

Jan> +	import symbols

I think it's better to put everything into its own package, e.g. 'import
linux.symbols', to try to avoid conflicts with other python modules that
may get loaded.

Tom

Re: [PATCH v5 05/20] scripts/gdb: Add lx-symbols command

[Jan Kiszka]

On 2013-02-14 16:40, Tom Tromey wrote:
>>>>>> "Jan" == Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
> Jan> In addition, the command installs a silent breakpoint in the load_module
> Jan> function at the point where the module was loaded but not yet
> Jan> initialized. The breakpoint handler will try to load symbols from the
> Jan> module files found during lx-symbols execution. This way, breakpoints
> Jan> can be set to module initialization functions, and there is usually no
> Jan> need to explicitly call lx-symbols after (re-)loading a module.
> 
> It is a nice approach, but I wonder whether the kernel's gdb stub could
> use the existing qXfer:libraries remote protocol packet.  And, if not,
> could we extend gdb to make it work.

That would help kgdb, but not QEMU or hardware debuggers. They have no
"Linux awareness" built in.

> 
> Jan> +		def _find_breakpoint_location(self):
> Jan> +			breakpoint_match = "^[0-9]*[\t]*err = parse_args\(.*"
> Jan> +
> Jan> +			src = gdb.execute("list kernel/module.c:load_module",
> Jan> +					  to_string = True)
> 
> Whatever works -- but I think there are better ways.
> 
> The simplest is introducing a function that is called at the right spot
> with the right arguments.  It doesn't need to do anything, just be a
> name where you can put a breakpoint.

Yes, I have this on my todo list. I already have a local config here
where the current heuristic broke - once again. We likely just need to
enforce un-inlining of do_init_module.

> 
> Jan> +	import symbols
> 
> I think it's better to put everything into its own package, e.g. 'import
> linux.symbols', to try to avoid conflicts with other python modules that
> may get loaded.

OK.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux