[PATCH 0/4] Revised the debug info patches...

[PATCH 0/4] Revised the debug info patches...

[Mark Hatle]

After talking with Richard a bit, I've revised the configuration of the
debug info, patch 2 of 4.  1, 3 and 4 remain unchanged.

The primary difference is to change the naming from "debian style", and
"fedora style" to a more accurate representation based on the GDB
documentation and usage I recently found.  In addition to the simple
name changing, the source code copying is now defined for both styles
and simply places the code in /usr/src/debug.

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: mhatle/debuginfo
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/debuginfo

Thanks,
    Mark Hatle <mark.hatle@windriver.com>
---


Mark Hatle (4):
  package.bbclass: Breakout the strip from the populate_packages
  Revise stripping and splitting of debug information
  package.bbclass: Refactor split and strip function
  recipes: pre/post actions

 meta/classes/insane.bbclass                        |    4 +-
 meta/classes/package.bbclass                       |  325 +++++++++++++++++---
 meta/conf/bitbake.conf                             |    9 +-
 meta/conf/local.conf.sample                        |   31 ++
 meta/recipes-bsp/hostap/hostap-conf_1.0.bb         |    4 +-
 meta/recipes-bsp/lrzsz/lrzsz_0.12.20.bb            |    4 +-
 meta/recipes-bsp/orinoco/orinoco-conf_1.0.bb       |    4 +-
 .../libnss-mdns/libnss-mdns_0.10.bb                |    4 +-
 .../ppp-dialin/ppp-dialin_0.1.bb                   |    4 +-
 .../recipes-core/base-passwd/base-passwd_3.5.22.bb |    2 +-
 meta/recipes-core/dropbear/dropbear.inc            |    4 +-
 .../sgml-common/sgml-common_0.6.3.bb               |    4 +-
 meta/recipes-extended/bash/bash.inc                |    2 +-
 meta/recipes-extended/bash/bash_3.2.48.bb          |    2 +-
 meta/recipes-extended/sudo/sudo.inc                |    2 +-
 .../ttf-fonts/liberation-fonts_1.04.bb             |    2 +-
 .../ttf-fonts/liberation-fonts_1.06.bb             |    2 +-
 .../ttf-fonts/ttf-bitstream-vera_1.10.bb           |    2 +-
 .../update-modules/update-modules_1.0.bb           |    2 +-
 19 files changed, 344 insertions(+), 69 deletions(-)

-- 
1.7.3.4

[PATCH 2/4] Revise stripping and splitting of debug information

[Mark Hatle]

We now support two styles of debug information generation, the '.debug' style,
which is the same as previously implemented.  This style simply splits the
debug information and makes it available in the same general directory.

  /bin/foo -> /bin/.debug/foo

The new 'debug-file-directory' style splits the debug information and places
it into the single debug-file-directory, /usr/lib/debug:

  /bin/foo -> /usr/lib/debug/bin/foo.debug

Both also find and copy all referenced source code to a new /usr/src/debug
directory.  This allows the -dbg files to be used for stand-a-lone debugging
on or off the target device.

File stripping is now handled as a seperate operation from file splitting.
This allows us to split the debug information, but also leave it in the
original file -- or prevent the debug information from being split.

Also enhance the comments within local.conf.sample to provide a better
understanding of the control the user has over debug file generation.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/insane.bbclass  |    4 +-
 meta/classes/package.bbclass |  233 +++++++++++++++++++++++++++++++++++++-----
 meta/conf/bitbake.conf       |    9 ++-
 meta/conf/local.conf.sample  |   31 ++++++
 4 files changed, 246 insertions(+), 31 deletions(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 5d3ef92..b376470 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -192,8 +192,8 @@ def package_qa_check_dev(path, name,d, elf):
 
     sane = True
 
-    if not name.endswith("-dev") and path.endswith(".so") and os.path.islink(path):
-        error_msg = "non -dev package contains symlink .so: %s path '%s'" % \
+    if not name.endswith("-dev") and not name.endswith("-dbg") and path.endswith(".so") and os.path.islink(path):
+        error_msg = "non -dev/-dbg package contains symlink .so: %s path '%s'" % \
                  (name, package_qa_clean_path(path,d))
         sane = package_qa_handle_error(0, error_msg, name, path, d)
 
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index fa1abf0..6c5c03d 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -165,32 +165,88 @@ python () {
         d.setVar("PACKAGERDEPTASK", "")
 }
 
-def runstrip(file, d):
-    # Function to strip a single file, called from split_and_strip_files below
+def splitfile(file, debugfile, debugsrcdir, d):
+    # Function to split a single file, called from split_and_strip_files below
     # A working 'file' (one which works on the target architecture)
-    # is necessary for this stuff to work, hence the addition to do_package[depends]
+    # is split and the split off portions go to debugfile.
+    #
+    # The debug information is then processed for src references.  These
+    # references are copied to debugsrcdir, if defined.
 
     import commands, stat
 
+    dvar = bb.data.getVar('PKGD', d, True)
     pathprefix = "export PATH=%s; " % bb.data.getVar('PATH', d, True)
+    objcopy = bb.data.getVar("OBJCOPY", d, True)
+    debugedit = bb.data.expand("${STAGING_LIBDIR_NATIVE}/rpm/bin/debugedit", d)
+    workdir = bb.data.expand("${WORKDIR}", d)
+    sourcefile = bb.data.expand("${WORKDIR}/debugsources.list", d)
 
-    ret, result = commands.getstatusoutput("%sfile '%s'" % (pathprefix, file))
+    # We ignore kernel modules, we don't generate debug info files.
+    if file.find("/lib/modules/") != -1 and file.endswith(".ko"):
+	return 0
 
-    if ret:
-        bb.error("runstrip: 'file %s' failed (forced strip)" % file)
+    newmode = None
+    if not os.access(file, os.W_OK) or os.access(file, os.R_OK):
+        origmode = os.stat(file)[stat.ST_MODE]
+        newmode = origmode | stat.S_IWRITE | stat.S_IREAD
+        os.chmod(file, newmode)
 
-    if "not stripped" not in result:
-        bb.debug(1, "runstrip: skip %s" % file)
-        return 0
+    # We need to extract the debug src information here...
+    if debugsrcdir:
+	os.system("%s'%s' -b '%s' -d '%s' -i -l '%s' '%s'" % (pathprefix, debugedit, workdir, debugsrcdir, sourcefile, file))
 
-    # If the file is in a .debug directory it was already stripped,
-    # don't do it again...
-    if os.path.dirname(file).endswith(".debug"):
-        bb.note("Already ran strip")
-        return 0
+    bb.mkdirhier(os.path.dirname(debugfile))
+
+    os.system("%s'%s' --only-keep-debug '%s' '%s'" % (pathprefix, objcopy, file, debugfile))
+
+    # Set the debuglink to have the view of the file path on the target
+    os.system("%s'%s' --add-gnu-debuglink='%s' '%s'" % (pathprefix, objcopy, debugfile, file))
 
+    if newmode:
+        os.chmod(file, origmode)
+
+    return 0
+
+def splitfile2(debugsrcdir, d):
+    # Function to split a single file, called from split_and_strip_files below
+    #
+    # The debug src information processed in the splitfile2 is further procecessed
+    # and copied to the destination here.
+
+    import commands, stat
+
+    dvar = bb.data.getVar('PKGD', d, True)
+    pathprefix = "export PATH=%s; " % bb.data.getVar('PATH', d, True)
     strip = bb.data.getVar("STRIP", d, True)
     objcopy = bb.data.getVar("OBJCOPY", d, True)
+    debugedit = bb.data.expand("${STAGING_LIBDIR_NATIVE}/rpm/bin/debugedit", d)
+    workdir = bb.data.expand("${WORKDIR}", d)
+    sourcefile = bb.data.expand("${WORKDIR}/debugsources.list", d)
+
+    if debugsrcdir:
+       bb.mkdirhier(debugsrcdir)
+
+       processdebugsrc =  "LC_ALL=C ; sort -z -u '%s' | egrep -v -z '(<internal>|<built-in>)$' | "
+       processdebugsrc += "(cd '%s' ; cpio -pd0mL '%s%s' 2>/dev/null)"
+
+       os.system(processdebugsrc % (sourcefile, workdir, dvar, debugsrcdir))
+
+       # The copy by cpio may have resulted in some empty directories!  Remove these
+       for root, dirs, files in os.walk("%s%s" % (dvar, debugsrcdir)):
+          for d in dirs:
+              dir = os.path.join(root, d)
+              #bb.note("rmdir -p %s" % dir)
+              os.system("rmdir -p %s 2>/dev/null" % dir)
+
+def runstrip(file, d):
+    # Function to strip a single file, called from split_and_strip_files below
+    # A working 'file' (one which works on the target architecture)
+
+    import commands, stat
+
+    pathprefix = "export PATH=%s; " % bb.data.getVar('PATH', d, True)
+    strip = bb.data.getVar("STRIP", d, True)
 
     # Handle kernel modules specifically - .debug directories here are pointless
     if file.find("/lib/modules/") != -1 and file.endswith(".ko"):
@@ -202,21 +258,22 @@ def runstrip(file, d):
         newmode = origmode | stat.S_IWRITE | stat.S_IREAD
         os.chmod(file, newmode)
 
+    ret, result = commands.getstatusoutput("%sfile '%s'" % (pathprefix, file))
+
+    if ret:
+        bb.error("runstrip: 'file %s' failed" % file)
+        return 0
+
     extraflags = ""
     if ".so" in file and "shared" in result:
         extraflags = "--remove-section=.comment --remove-section=.note --strip-unneeded"
     elif "shared" in result or "executable" in result:
         extraflags = "--remove-section=.comment --remove-section=.note"
 
-    bb.mkdirhier(os.path.join(os.path.dirname(file), ".debug"))
-    debugfile=os.path.join(os.path.dirname(file), ".debug", os.path.basename(file))
-
     stripcmd = "'%s' %s '%s'" % (strip, extraflags, file)
     bb.debug(1, "runstrip: %s" % stripcmd)
 
-    os.system("%s'%s' --only-keep-debug '%s' '%s'" % (pathprefix, objcopy, file, debugfile))
     ret = os.system("%s%s" % (pathprefix, stripcmd))
-    os.system("%s'%s' --add-gnu-debuglink='%s' '%s'" % (pathprefix, objcopy, debugfile, file))
 
     if newmode:
         os.chmod(file, origmode)
@@ -224,7 +281,7 @@ def runstrip(file, d):
     if ret:
         bb.error("runstrip: '%s' strip command failed" % stripcmd)
 
-    return 1
+    return 0
 
 #
 # Package data handling routines
@@ -333,10 +390,24 @@ python perform_packagecopy () {
 }
 
 python split_and_strip_files () {
-	import stat
+	import commands, stat, errno
 
 	dvar = bb.data.getVar('PKGD', d, True)
 
+	# We default to '.debug' style
+	if bb.data.getVar('PACKAGE_DEBUG_SPLIT_STYLE', d, True) == 'debug-file-directory':
+		# Single debug-file-directory style debug info
+		debugappend = ".debug"
+		debugdir = ""
+		debuglibdir = "/usr/lib/debug"
+		debugsrcdir = "/usr/src/debug"
+	else:
+		# Original Poky, a.k.a. ".debug", style debug info
+		debugappend = ""
+		debugdir = "/.debug"
+		debuglibdir = ""
+		debugsrcdir = "/usr/src/debug"
+
 	os.chdir(dvar)
 
 	def isexec(path):
@@ -344,16 +415,124 @@ python split_and_strip_files () {
 			s = os.stat(path)
 		except (os.error, AttributeError):
 			return 0
-		return (s[stat.ST_MODE] & stat.S_IEXEC)
+		return ((s[stat.ST_MODE] & stat.S_IXUSR) or (s[stat.ST_MODE] & stat.S_IXGRP) or (s[stat.ST_MODE] & stat.S_IXOTH))
+
+	# Return 0 - not elf, 1 - ELF & not stripped, 2 - ELF & stripped
+	def isELF(path):
+		pathprefix = "export PATH=%s; " % bb.data.getVar('PATH', d, True)
+		ret, result = commands.getstatusoutput("%sfile '%s'" % (pathprefix, path))
+
+		if ret:
+			bb.error("split_and_strip_files: 'file %s' failed" % path)
+			return 0
+
+		# Not stripped
+		if "ELF" in result and "not stripped" in result:
+			return 1
+
+		# Stripped
+		if "ELF" in result:
+			return 2
+
+		return 0;
+
+	#
+	# First lets process debug splitting
+	#
+	if (bb.data.getVar('INHIBIT_PACKAGE_DEBUG_SPLIT', d, True) != '1'):
+		file_links = {}
 
-	# Figure out which packages we want to process
-	if (bb.data.getVar('INHIBIT_PACKAGE_STRIP', d, True) != '1'):
 		for root, dirs, files in os.walk(dvar):
 			for f in files:
 				file = os.path.join(root, f)
-				if not os.path.islink(file) and not os.path.isdir(file) and isexec(file):
-					runstrip(file, d)
+				# Skip debug files, it must be executable, and must be a file (or link)
+				if not (debugappend != "" and file.endswith(debugappend)) and not (debugdir != "" and debugdir in os.path.dirname(file[len(dvar):])) and isexec(file) and os.path.isfile(file):
+					src = file[len(dvar):]
+					dest = debuglibdir + os.path.dirname(src) + debugdir + "/" + os.path.basename(src) + debugappend
+					fpath = dvar + dest
+					# Preserve symlinks in debug area...
+					if os.path.islink(file):
+						target = os.readlink(file)
+						if not os.path.isabs(target):
+							target = os.path.join(os.path.dirname(file), target)
+						if isELF(target):
+							ltarget = os.readlink(file)
+							lpath = os.path.dirname(ltarget)
+							lbase = os.path.basename(ltarget)
+							ftarget = ""
+							if lpath and lpath != ".":
+								ftarget += lpath + debugdir + "/"
+							ftarget += lbase + debugappend
+							bb.mkdirhier(os.path.dirname(fpath))
+							#bb.note("Symlink %s -> %s" % (fpath, ftarget))
+							os.symlink(ftarget, fpath)
+						continue
+
+					# If the file is elf we need to check it for hard links
+					elf_file = isELF(file)
+					if elf_file:
+						# Preserve hard links in debug area...
+						s = os.stat(file)
+						if s.st_nlink > 1:
+							file_reference = "%d_%d" % (s.st_dev, s.st_ino)
+							if file_reference not in file_links:
+								# If this is new, and already stripped we avoid recording it
+								# as we'll be unable to set the hard link later, because it
+								# won't be split/stripped...
+								if elf_file != 2:
+									file_links[file_reference] = fpath
+							else:
+								bb.mkdirhier(os.path.dirname(fpath))
+								#bb.note("Link %s -> %s" % (fpath, file_links[file_reference]))
+								os.link(file_links[file_reference], fpath)
+								continue
+
+						if elf_file == 2:
+							bb.warn("File '%s' was already stripped, this will prevent future debugging!" % (src))
+							continue
 
+						# Split and Strip
+						bb.mkdirhier(os.path.dirname(fpath))
+						#bb.note("Split %s -> %s" % (file, fpath))
+						splitfile(file, fpath, debugsrcdir, d)
+
+		# Process the debugsrcdir if requested...
+		splitfile2(debugsrcdir, d)
+
+		# The above may have generated dangling symlinks 
+		for root, dirs, files in os.walk(dvar):
+			for f in files:
+				file = os.path.join(root, f)
+				# We ONLY strip dangling links if they're debug generated!
+				if (debugappend != "" and file.endswith(debugappend)) or (debugdir != "" and debugdir in os.path.dirname(file[len(dvar):])):
+					try:
+						s = os.stat(file)
+					except OSError, (err, strerror):
+						if err != errno.ENOENT:
+							raise
+						#bb.note("Remove dangling link %s" % file)
+						os.unlink(file)
+
+	#
+	# End of debug splitting
+	#
+
+	#
+	# Now lets go back over things and strip them
+	#
+	if (bb.data.getVar('INHIBIT_PACKAGE_STRIP', d, True) != '1'):	
+		for root, dirs, files in os.walk(dvar):
+			for f in files:
+				file = os.path.join(root, f)
+				# if not a debugfile, is executable, is a file, and not a symlink
+				if not (debugappend != "" and file.endswith(debugappend)) and not (debugdir != "" and debugdir in os.path.dirname(file[len(dvar):])) and isexec(file) and os.path.isfile(file) and not os.path.islink(file):
+					elf_file = isELF(file)
+					if elf_file and elf_file != 2:
+						#bb.note("Strip %s" % file)
+						runstrip(file, d)
+	#
+	# End of strip
+	#
 }
 
 python populate_packages () {
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index da50fe5..b682ea2 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -242,9 +242,14 @@ SECTION_${PN}-dev = "devel"
 ALLOW_EMPTY_${PN}-dev = "1"
 RDEPENDS_${PN}-dev = "${PN} (= ${EXTENDPV})"
 
-FILES_${PN}-dbg = "${bindir}/.debug ${sbindir}/.debug ${libexecdir}/.debug ${libdir}/.debug \
+DOTDEBUG-dbg = "${bindir}/.debug ${sbindir}/.debug ${libexecdir}/.debug ${libdir}/.debug \
             ${base_bindir}/.debug ${base_sbindir}/.debug ${base_libdir}/.debug ${libdir}/${PN}/.debug \
-            ${libdir}/matchbox-panel/.debug"
+            ${libdir}/matchbox-panel/.debug /usr/src/debug"
+
+DEBUGFILEDIRECTORY-dbg = "/usr/lib/debug /usr/src/debug"
+
+FILES_${PN}-dbg = "${@bb.data.getVar(['DOTDEBUG-dbg', 'DEBUGFILEDIRECTORY-dbg'][bb.data.getVar('PACKAGE_DEBUG_SPLIT_STYLE', d, 1) == 'debug-file-directory'], d, 1)}"
+
 SECTION_${PN}-dbg = "devel"
 ALLOW_EMPTY_${PN}-dbg = "1"
 RRECOMMENDS_${PN}-dbg = "${PN} (= ${EXTENDPV})"
diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
index 16571fe..c5bedce 100644
--- a/meta/conf/local.conf.sample
+++ b/meta/conf/local.conf.sample
@@ -106,6 +106,37 @@ USER_CLASSES ?= "image-mklibs image-prelink"
 # <build directory>/tmp
 #TMPDIR = "${POKYBASE}/build/tmp"
 
+# The following are used to control options related to debugging.
+#
+# Uncomment this to change the optimization to make debugging easer, at the 
+# possible cost of performance.
+# DEBUG_BUILD = "1"
+#
+# Uncomment this to disable the stripping of the installed binaries
+# INHIBIT_PACKAGE_STRIP = "1"
+#
+# Uncomment this to disable the split of the debug information into -dbg files
+# INHIBIT_PACKAGE_DEBUG_SPLIT = "1"
+#
+# When splitting debug information, the following controls the results of the
+# file splitting.
+#
+#  .debug (default):
+#    When splitting the debug information will be placed into
+#    a .debug directory in the same dirname of the binary produced:
+#      /bin/foo -> /bin/.debug/foo
+#
+#  debug-file-directory:
+#    When splitting the debug information will be placed into
+#    a central debug-file-directory, /usr/lib/debug:
+#      /bin/foo -> /usr/lib/debug/bin/foo.debug
+#
+#    Any source code referenced in the debug symbols will be copied
+#    and made available within the /usr/src/debug directory
+#
+PACKAGE_DEBUG_SPLIT_STYLE = '.debug'
+# PACKAGE_DEBUG_SPLIT_STYLE = 'debug-file-directory'
+
 # Uncomment these to build a package such that you can use gprof to profile it.
 # NOTE: This will only work with 'linux' targets, not
 # 'linux-uclibc', as uClibc doesn't provide the necessary
-- 
1.7.3.4

Re: [PATCH 0/4] Revised the debug info patches...

[Koen Kooi]

Op 11 feb 2011, om 16:44 heeft Mark Hatle het volgende geschreven:

> After talking with Richard a bit, I've revised the configuration of the
> debug info, patch 2 of 4.  1, 3 and 4 remain unchanged.
> 
> The primary difference is to change the naming from "debian style", and
> "fedora style" to a more accurate representation based on the GDB
> documentation and usage I recently found.  In addition to the simple
> name changing, the source code copying is now defined for both styles
> and simply places the code in /usr/src/debug.

Is there a bbclass planned to populate /usr/src(/debug)? It would help people wanting to do external rebuilds of our (patched) sources.

regards,

Koen

Re: [PATCH 0/4] Revised the debug info patches...

[Mark Hatle]

On 2/11/11 10:15 AM, Koen Kooi wrote:
> 
> Op 11 feb 2011, om 16:44 heeft Mark Hatle het volgende geschreven:
> 
>> After talking with Richard a bit, I've revised the configuration of the
>> debug info, patch 2 of 4.  1, 3 and 4 remain unchanged.
>>
>> The primary difference is to change the naming from "debian style", and
>> "fedora style" to a more accurate representation based on the GDB
>> documentation and usage I recently found.  In addition to the simple
>> name changing, the source code copying is now defined for both styles
>> and simply places the code in /usr/src/debug.
> 
> Is there a bbclass planned to populate /usr/src(/debug)? It would help people wanting to do external rebuilds of our (patched) sources.

The two halves of the splitfile function (splitfile and splitfile2) generate the
/usr/src/debug.

In the splitfile, before the files are actually split, a component called
"debugedit" (a standard part of rpm) is run.

os.system("%s'%s' -b '%s' -d '%s' -i -l '%s' '%s'" % (pathprefix, debugedit,
workdir, debugsrcdir, sourcefile, file))

debugedit -b ${WORKDIR} -d '/usr/src/debug' -i -l '${WORKDIR}/debugsources.list'
'<file>'

The debugedit program does two things.  The first is it scans the <file> looking
for dwarf source references.  Upon finding those references it captures the
original file name (places it into the ${WORKDIR}/debugsources.list), and then
replaces the ${WORKDIR} part of the file name with the /usr/src/debug path.

Later after everything is processed and the debugsources.list contains a full
list of referenced sources the following script is executed:

processdebugsrc =  "LC_ALL=C ; sort -z -u '%s' | egrep -v -z
'(<internal>|<built-in>)$' | "
processdebugsrc += "(cd '%s' ; cpio -pd0mL '%s%s' 2>/dev/null)"

os.system(processdebugsrc % (sourcefile, workdir, dvar, debugsrcdir))

In otherwords, we sort the list (null separated), and remove references to gcc
internal and builtin functions.  The result of the list is passed to cpio which
performs a copy of all of the referenced files to the dvar/usr/src/debug directory.

----

So that's how it works, and what it does..  As far as a bbclass, it's part of
the standard packaging BBCLASS, I would have thought this could be used, even by
external people given the right inputs to package/repackage external files.
There is nothing Poky specific in this code -- other then it expects the
binaries coming in to not be stripped.  (If they are, a warning is issues.)

--Mark

> regards,
> 
> Koen

Re: [PATCH 0/4] Revised the debug info patches...

[Richard Purdie]

On Fri, 2011-02-11 at 09:44 -0600, Mark Hatle wrote:
> After talking with Richard a bit, I've revised the configuration of the
> debug info, patch 2 of 4.  1, 3 and 4 remain unchanged.
> 
> The primary difference is to change the naming from "debian style", and
> "fedora style" to a more accurate representation based on the GDB
> documentation and usage I recently found.  In addition to the simple
> name changing, the source code copying is now defined for both styles
> and simply places the code in /usr/src/debug.
> 
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>   Branch: mhatle/debuginfo
>   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/debuginfo

I've taken patch 1 and 4, I'll leave 2 and 3 until after the weekend as
they need a little more review.

Cheers,

Richard

[PATCH 2/4] Revise stripping and splitting of debug information

[Mark Hatle]

We now support two styles of debug information generation, the 'debian' style,
which is the same as previously implemented.  This style simply splits the
debug information and makes it available in the same general directory.

The new 'fedora' style splits the debug information and places it into
/usr/lib/debug/<path>/<file>.debug

It also has the ability to include an referenced source code into a new
/usr/src/debug/<path> directory.

File stripping is now handled as a seperate operation from file splitting.
This allows us to split the debug information, but also leave it in the
original file -- or prevent the debug information from being split.

Also enhance the comments within local.conf.sample to provide a better
understanding of the control the user has over debug file generation.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/insane.bbclass  |    4 +-
 meta/classes/package.bbclass |  233 +++++++++++++++++++++++++++++++++++++-----
 meta/conf/bitbake.conf       |    7 +-
 meta/conf/local.conf.sample  |   28 +++++
 4 files changed, 242 insertions(+), 30 deletions(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 5d3ef92..b376470 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -192,8 +192,8 @@ def package_qa_check_dev(path, name,d, elf):
 
     sane = True
 
-    if not name.endswith("-dev") and path.endswith(".so") and os.path.islink(path):
-        error_msg = "non -dev package contains symlink .so: %s path '%s'" % \
+    if not name.endswith("-dev") and not name.endswith("-dbg") and path.endswith(".so") and os.path.islink(path):
+        error_msg = "non -dev/-dbg package contains symlink .so: %s path '%s'" % \
                  (name, package_qa_clean_path(path,d))
         sane = package_qa_handle_error(0, error_msg, name, path, d)
 
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index fa1abf0..94dbdcc 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -165,32 +165,88 @@ python () {
         d.setVar("PACKAGERDEPTASK", "")
 }
 
-def runstrip(file, d):
-    # Function to strip a single file, called from split_and_strip_files below
+def splitfile(file, debugfile, debugsrcdir, d):
+    # Function to split a single file, called from split_and_strip_files below
     # A working 'file' (one which works on the target architecture)
-    # is necessary for this stuff to work, hence the addition to do_package[depends]
+    # is split and the split off portions go to debugfile.
+    #
+    # The debug information is then processed for src references.  These
+    # references are copied to debugsrcdir, if defined.
 
     import commands, stat
 
+    dvar = bb.data.getVar('PKGD', d, True)
     pathprefix = "export PATH=%s; " % bb.data.getVar('PATH', d, True)
+    objcopy = bb.data.getVar("OBJCOPY", d, True)
+    debugedit = bb.data.expand("${STAGING_LIBDIR_NATIVE}/rpm/bin/debugedit", d)
+    workdir = bb.data.expand("${WORKDIR}", d)
+    sourcefile = bb.data.expand("${WORKDIR}/debugsources.list", d)
 
-    ret, result = commands.getstatusoutput("%sfile '%s'" % (pathprefix, file))
+    # We ignore kernel modules, we don't generate debug info files.
+    if file.find("/lib/modules/") != -1 and file.endswith(".ko"):
+	return 0
 
-    if ret:
-        bb.error("runstrip: 'file %s' failed (forced strip)" % file)
+    newmode = None
+    if not os.access(file, os.W_OK) or os.access(file, os.R_OK):
+        origmode = os.stat(file)[stat.ST_MODE]
+        newmode = origmode | stat.S_IWRITE | stat.S_IREAD
+        os.chmod(file, newmode)
 
-    if "not stripped" not in result:
-        bb.debug(1, "runstrip: skip %s" % file)
-        return 0
+    # We need to extract the debug src information here...
+    if debugsrcdir:
+	os.system("%s'%s' -b '%s' -d '%s' -i -l '%s' '%s'" % (pathprefix, debugedit, workdir, debugsrcdir, sourcefile, file))
 
-    # If the file is in a .debug directory it was already stripped,
-    # don't do it again...
-    if os.path.dirname(file).endswith(".debug"):
-        bb.note("Already ran strip")
-        return 0
+    bb.mkdirhier(os.path.dirname(debugfile))
+
+    os.system("%s'%s' --only-keep-debug '%s' '%s'" % (pathprefix, objcopy, file, debugfile))
+
+    # Set the debuglink to have the view of the file path on the target
+    os.system("%s'%s' --add-gnu-debuglink='%s' '%s'" % (pathprefix, objcopy, debugfile, file))
 
+    if newmode:
+        os.chmod(file, origmode)
+
+    return 0
+
+def splitfile2(debugsrcdir, d):
+    # Function to split a single file, called from split_and_strip_files below
+    #
+    # The debug src information processed in the splitfile2 is further procecessed
+    # and copied to the destination here.
+
+    import commands, stat
+
+    dvar = bb.data.getVar('PKGD', d, True)
+    pathprefix = "export PATH=%s; " % bb.data.getVar('PATH', d, True)
     strip = bb.data.getVar("STRIP", d, True)
     objcopy = bb.data.getVar("OBJCOPY", d, True)
+    debugedit = bb.data.expand("${STAGING_LIBDIR_NATIVE}/rpm/bin/debugedit", d)
+    workdir = bb.data.expand("${WORKDIR}", d)
+    sourcefile = bb.data.expand("${WORKDIR}/debugsources.list", d)
+
+    if debugsrcdir:
+       bb.mkdirhier(debugsrcdir)
+
+       processdebugsrc =  "LC_ALL=C ; sort -z -u '%s' | egrep -v -z '(<internal>|<built-in>)$' | "
+       processdebugsrc += "(cd '%s' ; cpio -pd0mL '%s%s' 2>/dev/null)"
+
+       os.system(processdebugsrc % (sourcefile, workdir, dvar, debugsrcdir))
+
+       # The copy by cpio may have resulted in some empty directories!  Remove these
+       for root, dirs, files in os.walk("%s%s" % (dvar, debugsrcdir)):
+          for d in dirs:
+              dir = os.path.join(root, d)
+              #bb.note("rmdir -p %s" % dir)
+              os.system("rmdir -p %s 2>/dev/null" % dir)
+
+def runstrip(file, d):
+    # Function to strip a single file, called from split_and_strip_files below
+    # A working 'file' (one which works on the target architecture)
+
+    import commands, stat
+
+    pathprefix = "export PATH=%s; " % bb.data.getVar('PATH', d, True)
+    strip = bb.data.getVar("STRIP", d, True)
 
     # Handle kernel modules specifically - .debug directories here are pointless
     if file.find("/lib/modules/") != -1 and file.endswith(".ko"):
@@ -202,21 +258,22 @@ def runstrip(file, d):
         newmode = origmode | stat.S_IWRITE | stat.S_IREAD
         os.chmod(file, newmode)
 
+    ret, result = commands.getstatusoutput("%sfile '%s'" % (pathprefix, file))
+
+    if ret:
+        bb.error("runstrip: 'file %s' failed" % file)
+        return 0
+
     extraflags = ""
     if ".so" in file and "shared" in result:
         extraflags = "--remove-section=.comment --remove-section=.note --strip-unneeded"
     elif "shared" in result or "executable" in result:
         extraflags = "--remove-section=.comment --remove-section=.note"
 
-    bb.mkdirhier(os.path.join(os.path.dirname(file), ".debug"))
-    debugfile=os.path.join(os.path.dirname(file), ".debug", os.path.basename(file))
-
     stripcmd = "'%s' %s '%s'" % (strip, extraflags, file)
     bb.debug(1, "runstrip: %s" % stripcmd)
 
-    os.system("%s'%s' --only-keep-debug '%s' '%s'" % (pathprefix, objcopy, file, debugfile))
     ret = os.system("%s%s" % (pathprefix, stripcmd))
-    os.system("%s'%s' --add-gnu-debuglink='%s' '%s'" % (pathprefix, objcopy, debugfile, file))
 
     if newmode:
         os.chmod(file, origmode)
@@ -224,7 +281,7 @@ def runstrip(file, d):
     if ret:
         bb.error("runstrip: '%s' strip command failed" % stripcmd)
 
-    return 1
+    return 0
 
 #
 # Package data handling routines
@@ -333,10 +390,24 @@ python perform_packagecopy () {
 }
 
 python split_and_strip_files () {
-	import stat
+	import commands, stat, errno
 
 	dvar = bb.data.getVar('PKGD', d, True)
 
+	# We default to Debian style
+	if bb.data.getVar('PACKAGE_DEBUG_SPLIT_STYLE', d, True) == 'fedora':
+		# Fedora Style debug info
+		debugappend = ".debug"
+		debugdir = ""
+		debuglibdir = "/usr/lib/debug"
+		debugsrcdir = "/usr/src/debug"
+	else:
+		# Debian Style debug info
+		debugappend = ""
+		debugdir = "/.debug"
+		debuglibdir = ""
+		debugsrcdir = ""
+
 	os.chdir(dvar)
 
 	def isexec(path):
@@ -344,16 +415,124 @@ python split_and_strip_files () {
 			s = os.stat(path)
 		except (os.error, AttributeError):
 			return 0
-		return (s[stat.ST_MODE] & stat.S_IEXEC)
+		return ((s[stat.ST_MODE] & stat.S_IXUSR) or (s[stat.ST_MODE] & stat.S_IXGRP) or (s[stat.ST_MODE] & stat.S_IXOTH))
+
+	# Return 0 - not elf, 1 - ELF & not stripped, 2 - ELF & stripped
+	def isELF(path):
+		pathprefix = "export PATH=%s; " % bb.data.getVar('PATH', d, True)
+		ret, result = commands.getstatusoutput("%sfile '%s'" % (pathprefix, path))
+
+		if ret:
+			bb.error("split_and_strip_files: 'file %s' failed" % path)
+			return 0
+
+		# Not stripped
+		if "ELF" in result and "not stripped" in result:
+			return 1
+
+		# Stripped
+		if "ELF" in result:
+			return 2
+
+		return 0;
+
+	#
+	# First lets process debug splitting
+	#
+	if (bb.data.getVar('INHIBIT_PACKAGE_DEBUG_SPLIT', d, True) != '1'):
+		file_links = {}
 
-	# Figure out which packages we want to process
-	if (bb.data.getVar('INHIBIT_PACKAGE_STRIP', d, True) != '1'):
 		for root, dirs, files in os.walk(dvar):
 			for f in files:
 				file = os.path.join(root, f)
-				if not os.path.islink(file) and not os.path.isdir(file) and isexec(file):
-					runstrip(file, d)
+				# Skip debug files, it must be executable, and must be a file (or link)
+				if not (debugappend != "" and file.endswith(debugappend)) and not (debugdir != "" and debugdir in os.path.dirname(file[len(dvar):])) and isexec(file) and os.path.isfile(file):
+					src = file[len(dvar):]
+					dest = debuglibdir + os.path.dirname(src) + debugdir + "/" + os.path.basename(src) + debugappend
+					fpath = dvar + dest
+					# Preserve symlinks in debug area...
+					if os.path.islink(file):
+						target = os.readlink(file)
+						if not os.path.isabs(target):
+							target = os.path.join(os.path.dirname(file), target)
+						if isELF(target):
+							ltarget = os.readlink(file)
+							lpath = os.path.dirname(ltarget)
+							lbase = os.path.basename(ltarget)
+							ftarget = ""
+							if lpath and lpath != ".":
+								ftarget += lpath + debugdir + "/"
+							ftarget += lbase + debugappend
+							bb.mkdirhier(os.path.dirname(fpath))
+							#bb.note("Symlink %s -> %s" % (fpath, ftarget))
+							os.symlink(ftarget, fpath)
+						continue
+
+					# If the file is elf we need to check it for hard links
+					elf_file = isELF(file)
+					if elf_file:
+						# Preserve hard links in debug area...
+						s = os.stat(file)
+						if s.st_nlink > 1:
+							file_reference = "%d_%d" % (s.st_dev, s.st_ino)
+							if file_reference not in file_links:
+								# If this is new, and already stripped we avoid recording it
+								# as we'll be unable to set the hard link later, because it
+								# won't be split/stripped...
+								if elf_file != 2:
+									file_links[file_reference] = fpath
+							else:
+								bb.mkdirhier(os.path.dirname(fpath))
+								#bb.note("Link %s -> %s" % (fpath, file_links[file_reference]))
+								os.link(file_links[file_reference], fpath)
+								continue
+
+						if elf_file == 2:
+							bb.warn("File '%s' was already stripped, this will prevent future debugging!" % (src))
+							continue
 
+						# Split and Strip
+						bb.mkdirhier(os.path.dirname(fpath))
+						#bb.note("Split %s -> %s" % (file, fpath))
+						splitfile(file, fpath, debugsrcdir, d)
+
+		# Process the debugsrcdir if requested...
+		splitfile2(debugsrcdir, d)
+
+		# The above may have generated dangling symlinks 
+		for root, dirs, files in os.walk(dvar):
+			for f in files:
+				file = os.path.join(root, f)
+				# We ONLY strip dangling links if they're debug generated!
+				if (debugappend != "" and file.endswith(debugappend)) or (debugdir != "" and debugdir in os.path.dirname(file[len(dvar):])):
+					try:
+						s = os.stat(file)
+					except OSError, (err, strerror):
+						if err != errno.ENOENT:
+							raise
+						#bb.note("Remove dangling link %s" % file)
+						os.unlink(file)
+
+	#
+	# End of debug splitting
+	#
+
+	#
+	# Now lets go back over things and strip them
+	#
+	if (bb.data.getVar('INHIBIT_PACKAGE_STRIP', d, True) != '1'):	
+		for root, dirs, files in os.walk(dvar):
+			for f in files:
+				file = os.path.join(root, f)
+				# if not a debugfile, is executable, is a file, and not a symlink
+				if not (debugappend != "" and file.endswith(debugappend)) and not (debugdir != "" and debugdir in os.path.dirname(file[len(dvar):])) and isexec(file) and os.path.isfile(file) and not os.path.islink(file):
+					elf_file = isELF(file)
+					if elf_file and elf_file != 2:
+						#bb.note("Strip %s" % file)
+						runstrip(file, d)
+	#
+	# End of strip
+	#
 }
 
 python populate_packages () {
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index da50fe5..7ec02cd 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -242,9 +242,14 @@ SECTION_${PN}-dev = "devel"
 ALLOW_EMPTY_${PN}-dev = "1"
 RDEPENDS_${PN}-dev = "${PN} (= ${EXTENDPV})"
 
-FILES_${PN}-dbg = "${bindir}/.debug ${sbindir}/.debug ${libexecdir}/.debug ${libdir}/.debug \
+DEBIANSTYLE-dbg = "${bindir}/.debug ${sbindir}/.debug ${libexecdir}/.debug ${libdir}/.debug \
             ${base_bindir}/.debug ${base_sbindir}/.debug ${base_libdir}/.debug ${libdir}/${PN}/.debug \
             ${libdir}/matchbox-panel/.debug"
+
+FEDORASTYLE-dbg = "/usr/lib/debug /usr/src/debug"
+
+FILES_${PN}-dbg = "${@bb.data.getVar(['DEBIANSTYLE-dbg', 'FEDORASTYLE-dbg'][bb.data.getVar('PACKAGE_DEBUG_SPLIT_STYLE', d, 1) == 'fedora'], d, 1)}"
+
 SECTION_${PN}-dbg = "devel"
 ALLOW_EMPTY_${PN}-dbg = "1"
 RRECOMMENDS_${PN}-dbg = "${PN} (= ${EXTENDPV})"
diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
index 16571fe..b4863c5 100644
--- a/meta/conf/local.conf.sample
+++ b/meta/conf/local.conf.sample
@@ -106,6 +106,34 @@ USER_CLASSES ?= "image-mklibs image-prelink"
 # <build directory>/tmp
 #TMPDIR = "${POKYBASE}/build/tmp"
 
+# The following are used to control options related to debugging.
+#
+# Uncomment this to change the optimization to make debugging easer, at the 
+# possible cost of performance.
+# DEBUG_BUILD = "1"
+#
+# Uncomment this to disable the stripping of the installed binaries
+# INHIBIT_PACKAGE_STRIP = "1"
+#
+# Uncomment this to disable the split of the debug information into -dbg files
+# INHIBIT_PACKAGE_DEBUG_SPLIT = "1"
+#
+# When splitting debug information, the following controls the results of the
+# file splitting.
+#
+#  debian (default):
+#    When splitting /bin/foo, the debug information will be:
+#      /bin/.debug/foo
+#    No referenced sources will be added to the -dbg file
+#
+#  fedora:
+#    When splitting /bin/foo, the debug information will be:
+#      /usr/lib/debug/bin/foo.debug
+#    Any source code referenced in the debug symbols will be copied
+#    and made available within the /usr/src/debug directory
+#
+PACKAGE_DEBUG_SPLIT_STYLE = 'debian'
+
 # Uncomment these to build a package such that you can use gprof to profile it.
 # NOTE: This will only work with 'linux' targets, not
 # 'linux-uclibc', as uClibc doesn't provide the necessary
-- 
1.7.3.4