[PATCH V7 00/17] fix buildpaths QA issue && dbp package source generation

[PATCH V7 00/17] fix buildpaths QA issue && dbp package source generation

[Hongxu Jia]

Changed in V7:
- Improve -fdebug-prefix-map to support to remap relative path 

- Improve dbg package sources generation from work-shared

- Fix ifupdown do_compile failed while GCCVERSION = "4.9%"

- Rebase to latest master

Changed in V6:
- Drop kernel build path fix

- Tweak debug location to /usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR}
  to keep syn with past

- Tweak PACKAGE_BUILDPATH_TEXT_PATTERNS in fix_buildpaths.bbclass to
  adapt the above fix;


Test steps:

1. vim local.conf
------
IMAGE_INSTALL_append = " gdb gcc gzip"
EXTRA_IMAGE_FEATURES_append = " dbg-pkgs"
------

2. bitbake core-image-minimal

3. runqemu qemux86-64 slirp nographic

4. gdb gzip
(gdb) list
387	    printf ("Written by Jean-loup Gailly.\n");
388	}
389	
390	local void progerror (char const *string)
391	{
392	    int e = errno;
393	    fprintf (stderr, "%s: ", program_name);
394	    errno = e;
395	    perror(string);
396	    exit_code = ERROR;

5. gdb x86_64-poky-linux-gcc
(gdb) list
38	extern int main (int, char **);
39	
40	int
41	main (int argc, char **argv)
42	{
43	  driver d;
44	
45	  return d.main (argc, argv);
46	}
(gdb) b 45
Breakpoint 1 at 0x402ce7: file /usr/src/debug/gcc/5.3.0-r0/work-shared/gcc-5.3.0-r0/gcc-5.3.0/gcc/gcc-main.c, line 45.
(gdb) r
Starting program: /usr/bin/x86_64-poky-linux-gcc 

Breakpoint 1, main (argc=1, argv=0x7fffffffecf8) at /usr/src/debug/gcc/5.3.0-r0/work-shared/gcc-5.3.0-r0/gcc-5.3.0/gcc/gcc-main.c:45
45	  return d.main (argc, argv);
(gdb) step
driver::main (this=this@entry=0x7fffffffebf0, argc=1, argv=0x7fffffffecf8)
    at /usr/src/debug/gcc/5.3.0-r0/work-shared/gcc-5.3.0-r0/gcc-5.3.0/gcc/gcc.c:6935
6935	{
(gdb) bt
#0  driver::main (this=this@entry=0x7fffffffebf0, argc=1, argv=0x7fffffffecf8)
    at /usr/src/debug/gcc/5.3.0-r0/work-shared/gcc-5.3.0-r0/gcc-5.3.0/gcc/gcc.c:6935
#1  0x0000000000402cf1 in main (argc=<optimized out>, argv=<optimized out>)
    at /usr/src/debug/gcc/5.3.0-r0/work-shared/gcc-5.3.0-r0/gcc-5.3.0/gcc/gcc-main.c:45

//Hongxu

The following changes since commit 5005cab723bc488bcf3f982a6401ecf49bdb9ca3:

  add !meta-poky to .gitignore file (2016-03-26 08:06:58 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib hongxu/fix-buildpath
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=hongxu/fix-buildpath

Hongxu Jia (17):
  conf/bitbake.conf package.bbclass: fix dbg package not contain sources
    while -fdebug-prefix-map used
  gcc-5.3/gcc-4.9: -fdebug-prefix-map support to remap relative path
  conf/bitbake.conf package.bbclass: improve dbg package sources
    generation from work-shared
  dtc.inc: fix buildpaths QA issue
  fix_buildpaths.bbclass: add bbclass to fix build path
  icu: fix buildpaths QA issue
  tcl: fix buildpath QA issue
  python2/3: fix buildpath QA issue
  bbclass distutils/distutils3: fix .pyc/.pyo buildpath
  bbclass distutils/distutils3/setuptools/setuptools3: clean up
    DISTUTILS_INSTALL_ARGS
  python-setuptools/python3-setuptools: use old-style install
  python3-pip: use old-style install
  waf.bbclass: support do patch on extracted files
  python-pycairo: fix buildpath QA issue
  openssl: fix buildpath QA issue
  epiphany: fix buildpaths QA issue
  ifupdown: fix do_compile failed while GCCVERSION = "4.9%"

 meta/classes/distutils.bbclass                     |   8 +-
 meta/classes/distutils3.bbclass                    |   8 +-
 meta/classes/fix_buildpaths.bbclass                |  66 +++++++++++++
 meta/classes/package.bbclass                       | 106 ++++++++++++++++-----
 meta/classes/setuptools.bbclass                    |   5 -
 meta/classes/setuptools3.bbclass                   |   5 -
 meta/classes/waf.bbclass                           |  11 ++-
 meta/conf/bitbake.conf                             |   4 +-
 .../openssl/remove-build-path-in-source-code.patch |  44 +++++++++
 .../recipes-connectivity/openssl/openssl_1.0.2g.bb |   7 ++
 meta/recipes-core/ifupdown/ifupdown_0.8.2.bb       |   2 +-
 meta/recipes-devtools/gcc/gcc-4.9.inc              |   1 +
 ...prefix-map-support-to-remap-relative-path.patch |  51 ++++++++++
 meta/recipes-devtools/gcc/gcc-5.3.inc              |   1 +
 ...prefix-map-support-to-remap-relative-path.patch |  51 ++++++++++
 ...trip-off-build-path-in-purported-filename.patch |  62 ++++++++++++
 .../python/python-pycairo_1.10.0.bb                |  25 ++---
 .../python/python-setuptools_19.4.bb               |   2 -
 meta/recipes-devtools/python/python3-pip_8.0.0.bb  |  12 ---
 .../python/python3-setuptools_19.4.bb              |   8 +-
 meta/recipes-devtools/python/python3_3.5.1.bb      |  20 +++-
 meta/recipes-devtools/python/python_2.7.11.bb      |  29 +++++-
 meta/recipes-devtools/tcltk/tcl_8.6.4.bb           |   9 +-
 meta/recipes-gnome/epiphany/epiphany_3.18.4.bb     |   8 +-
 .../files/use-relative-path-for-build-dir.patch    |  57 +++++++++++
 meta/recipes-kernel/dtc/dtc.inc                    |   3 +
 .../dtc/dtc/pass-DEBUG_FLAGS-to-CFLAGS.patch       |  30 ++++++
 .../icu/icu/add-variable-U_CC-and-U_CXX.patch      |  61 ++++++++++++
 meta/recipes-support/icu/icu_56.1.bb               |  12 +++
 29 files changed, 613 insertions(+), 95 deletions(-)
 create mode 100644 meta/classes/fix_buildpaths.bbclass
 create mode 100644 meta/recipes-connectivity/openssl/openssl/remove-build-path-in-source-code.patch
 create mode 100644 meta/recipes-devtools/gcc/gcc-4.9/0074-fdebug-prefix-map-support-to-remap-relative-path.patch
 create mode 100644 meta/recipes-devtools/gcc/gcc-5.3/0056-fdebug-prefix-map-support-to-remap-relative-path.patch
 create mode 100644 meta/recipes-devtools/python/python-pycairo/strip-off-build-path-in-purported-filename.patch
 create mode 100644 meta/recipes-gnome/epiphany/files/use-relative-path-for-build-dir.patch
 create mode 100644 meta/recipes-kernel/dtc/dtc/pass-DEBUG_FLAGS-to-CFLAGS.patch
 create mode 100644 meta/recipes-support/icu/icu/add-variable-U_CC-and-U_CXX.patch

-- 
1.9.1

[PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Hongxu Jia]

Tweak DEBUG_FLAGS to use "/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR}"
as source target path in DWARF. While use gdb to debug binary, it could
work with sources in dbg package.

While -fdebug-prefix-map is used for compiling, we do not need invoking
debugedit to edit DWARF at do_package time, but list where sources files
are.

The copydebugsources uses the list to copy sources to dbg package. It
works whether -fdebug-prefix-map used or not.

[YOCTO #9305]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/package.bbclass | 24 ++++++++++++++++++++----
 meta/conf/bitbake.conf       |  3 +--
 2 files changed, 21 insertions(+), 6 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 5a84255..bdbe96d 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -300,6 +300,15 @@ def get_conffiles(pkg, d):
     os.chdir(cwd)
     return conf_list
 
+def checkbuildpath(file, d):
+    tmpdir = d.getVar('TMPDIR', True)
+    with open(file) as f:
+        file_content = f.read()
+        if tmpdir in file_content:
+            return True
+
+    return False
+
 def splitdebuginfo(file, debugfile, debugsrcdir, sourcefile, d):
     # Function to split a single file into two components, one is the stripped
     # target system binary, the other contains any debugging information. The
@@ -312,8 +321,6 @@ def splitdebuginfo(file, debugfile, debugsrcdir, sourcefile, d):
     dvar = d.getVar('PKGD', True)
     objcopy = d.getVar("OBJCOPY", True)
     debugedit = d.expand("${STAGING_LIBDIR_NATIVE}/rpm/bin/debugedit")
-    workdir = d.getVar("WORKDIR", True)
-    workparentdir = d.getVar("DEBUGSRC_OVERRIDE_PATH", True) or os.path.dirname(os.path.dirname(workdir))
 
     # We ignore kernel modules, we don't generate debug info files.
     if file.find("/lib/modules/") != -1 and file.endswith(".ko"):
@@ -327,7 +334,7 @@ def splitdebuginfo(file, debugfile, debugsrcdir, sourcefile, d):
 
     # We need to extract the debug src information here...
     if debugsrcdir:
-        cmd = "'%s' -b '%s' -d '%s' -i -l '%s' '%s'" % (debugedit, workparentdir, debugsrcdir, sourcefile, file)
+        cmd = "'%s' -i -l '%s' '%s'" % (debugedit, sourcefile, file)
         (retval, output) = oe.utils.getstatusoutput(cmd)
         if retval:
             bb.fatal("debugedit failed with exit code %s (cmd was %s)%s" % (retval, cmd, ":\n%s" % output if output else ""))
@@ -366,6 +373,13 @@ def copydebugsources(debugsrcdir, d):
         workparentdir = os.path.dirname(os.path.dirname(workdir))
         workbasedir = os.path.basename(os.path.dirname(workdir)) + "/" + os.path.basename(workdir)
 
+        # If build path exists in sourcefile, it means toolchain did not use
+        # -fdebug-prefix-map to compile
+        if checkbuildpath(sourcefile, d):
+            localsrc_prefix = workparentdir + "/"
+        else:
+            localsrc_prefix = "/usr/src/debug/"
+
         nosuchdir = []
         basepath = dvar
         for p in debugsrcdir.split("/"):
@@ -379,9 +393,11 @@ def copydebugsources(debugsrcdir, d):
         # We need to ignore files that are not actually ours
         # we do this by only paying attention to items from this package
         processdebugsrc += "fgrep -zw '%s' | "
+        # Remove prefix in the source paths
+        processdebugsrc += "sed 's#%s##g' | "
         processdebugsrc += "(cd '%s' ; cpio -pd0mlL --no-preserve-owner '%s%s' 2>/dev/null)"
 
-        cmd = processdebugsrc % (sourcefile, workbasedir, workparentdir, dvar, debugsrcdir)
+        cmd = processdebugsrc % (sourcefile, workbasedir, localsrc_prefix, workparentdir, dvar, debugsrcdir)
         (retval, output) = oe.utils.getstatusoutput(cmd)
         # Can "fail" if internal headers/transient sources are attempted
         #if retval:
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index e3db504..0eb288e 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -550,8 +550,7 @@ EXTRA_OEMAKE_prepend_task-install = "${PARALLEL_MAKEINST} "
 # Optimization flags.
 ##################################################################
 DEBUG_FLAGS ?= "-g -feliminate-unused-debug-types \
-                -fdebug-prefix-map=${B}=/usr/src/${BPN} \
-                -fdebug-prefix-map=${S}=/usr/src/${BPN} \
+                -fdebug-prefix-map=${WORKDIR}=/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR} \
                 -fdebug-prefix-map=${STAGING_DIR_NATIVE}= \
                 -fdebug-prefix-map=${STAGING_DIR_HOST}= \
 "
-- 
1.9.1

[PATCH 02/17] gcc-5.3/gcc-4.9: -fdebug-prefix-map support to remap relative path

[Hongxu Jia]

For relative path in DWARF, -fdebug-prefix-map could not remap it,
so translate to real path before mapping.

[YOCTO #9305]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-devtools/gcc/gcc-4.9.inc              |  1 +
 ...prefix-map-support-to-remap-relative-path.patch | 51 ++++++++++++++++++++++
 meta/recipes-devtools/gcc/gcc-5.3.inc              |  1 +
 ...prefix-map-support-to-remap-relative-path.patch | 51 ++++++++++++++++++++++
 4 files changed, 104 insertions(+)
 create mode 100644 meta/recipes-devtools/gcc/gcc-4.9/0074-fdebug-prefix-map-support-to-remap-relative-path.patch
 create mode 100644 meta/recipes-devtools/gcc/gcc-5.3/0056-fdebug-prefix-map-support-to-remap-relative-path.patch

diff --git a/meta/recipes-devtools/gcc/gcc-4.9.inc b/meta/recipes-devtools/gcc/gcc-4.9.inc
index 87aa7da..7e03f31 100644
--- a/meta/recipes-devtools/gcc/gcc-4.9.inc
+++ b/meta/recipes-devtools/gcc/gcc-4.9.inc
@@ -89,6 +89,7 @@ SRC_URI = "\
     file://0071-Ignore-fdebug-prefix-map-in-producer-string-by-Danie.patch \
     file://0072-support-ffile-prefix-map.patch \
     file://0073-Reuse-fdebug-prefix-map-to-replace-ffile-prefix-map.patch \
+    file://0074-fdebug-prefix-map-support-to-remap-relative-path.patch \
 "
 SRC_URI[md5sum] = "6f831b4d251872736e8e9cc09746f327"
 SRC_URI[sha256sum] = "2332b2a5a321b57508b9031354a8503af6fdfb868b8c1748d33028d100a8b67e"
diff --git a/meta/recipes-devtools/gcc/gcc-4.9/0074-fdebug-prefix-map-support-to-remap-relative-path.patch b/meta/recipes-devtools/gcc/gcc-4.9/0074-fdebug-prefix-map-support-to-remap-relative-path.patch
new file mode 100644
index 0000000..0b91fdb
--- /dev/null
+++ b/meta/recipes-devtools/gcc/gcc-4.9/0074-fdebug-prefix-map-support-to-remap-relative-path.patch
@@ -0,0 +1,51 @@
+From 289ad2969a5966c603bf6928ce442db74c4cbb25 Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <hongxu.jia@windriver.com>
+Date: Thu, 24 Mar 2016 11:23:14 -0400
+Subject: [PATCH] gcc/final.c: -fdebug-prefix-map support to remap sources with
+ relative path
+
+PR other/70428
+* final.c (remap_debug_filename): Use lrealpath to translate
+relative path before remapping
+
+https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70428
+Upstream-Status: Submitted [gcc-patches@gcc.gnu.org]
+
+Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
+---
+diff --git a/gcc/final.c b/gcc/final.c
+index 55cf509..c3594c2 100644
+--- a/gcc/final.c
++++ b/gcc/final.c
+@@ -1554,16 +1554,25 @@ remap_debug_filename (const char *filename)
+   const char *name;
+   size_t name_len;
+ 
++  /* Support to remap filename with relative path  */
++  char *realpath = lrealpath (filename);
++  if (realpath == NULL)
++    return filename;
++
+   for (map = debug_prefix_maps; map; map = map->next)
+-    if (filename_ncmp (filename, map->old_prefix, map->old_len) == 0)
++    if (filename_ncmp (realpath, map->old_prefix, map->old_len) == 0)
+       break;
+   if (!map)
+-    return filename;
+-  name = filename + map->old_len;
++    {
++      free (realpath);
++      return filename;
++    }
++  name = realpath + map->old_len;
+   name_len = strlen (name) + 1;
+   s = (char *) alloca (name_len + map->new_len);
+   memcpy (s, map->new_prefix, map->new_len);
+   memcpy (s + map->new_len, name, name_len);
++  free (realpath);
+   return ggc_strdup (s);
+ }
+ \f
+-- 
+2.7.4
+
diff --git a/meta/recipes-devtools/gcc/gcc-5.3.inc b/meta/recipes-devtools/gcc/gcc-5.3.inc
index aa011fd..aa5645e 100644
--- a/meta/recipes-devtools/gcc/gcc-5.3.inc
+++ b/meta/recipes-devtools/gcc/gcc-5.3.inc
@@ -85,6 +85,7 @@ SRC_URI = "\
            file://0053-expr.c-PR-target-65358-Avoid-clobbering-partial-argu.patch \
            file://0054-support-ffile-prefix-map.patch \
            file://0055-Reuse-fdebug-prefix-map-to-replace-ffile-prefix-map.patch \
+           file://0056-fdebug-prefix-map-support-to-remap-relative-path.patch \
 "
 
 BACKPORTS = ""
diff --git a/meta/recipes-devtools/gcc/gcc-5.3/0056-fdebug-prefix-map-support-to-remap-relative-path.patch b/meta/recipes-devtools/gcc/gcc-5.3/0056-fdebug-prefix-map-support-to-remap-relative-path.patch
new file mode 100644
index 0000000..0b91fdb
--- /dev/null
+++ b/meta/recipes-devtools/gcc/gcc-5.3/0056-fdebug-prefix-map-support-to-remap-relative-path.patch
@@ -0,0 +1,51 @@
+From 289ad2969a5966c603bf6928ce442db74c4cbb25 Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <hongxu.jia@windriver.com>
+Date: Thu, 24 Mar 2016 11:23:14 -0400
+Subject: [PATCH] gcc/final.c: -fdebug-prefix-map support to remap sources with
+ relative path
+
+PR other/70428
+* final.c (remap_debug_filename): Use lrealpath to translate
+relative path before remapping
+
+https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70428
+Upstream-Status: Submitted [gcc-patches@gcc.gnu.org]
+
+Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
+---
+diff --git a/gcc/final.c b/gcc/final.c
+index 55cf509..c3594c2 100644
+--- a/gcc/final.c
++++ b/gcc/final.c
+@@ -1554,16 +1554,25 @@ remap_debug_filename (const char *filename)
+   const char *name;
+   size_t name_len;
+ 
++  /* Support to remap filename with relative path  */
++  char *realpath = lrealpath (filename);
++  if (realpath == NULL)
++    return filename;
++
+   for (map = debug_prefix_maps; map; map = map->next)
+-    if (filename_ncmp (filename, map->old_prefix, map->old_len) == 0)
++    if (filename_ncmp (realpath, map->old_prefix, map->old_len) == 0)
+       break;
+   if (!map)
+-    return filename;
+-  name = filename + map->old_len;
++    {
++      free (realpath);
++      return filename;
++    }
++  name = realpath + map->old_len;
+   name_len = strlen (name) + 1;
+   s = (char *) alloca (name_len + map->new_len);
+   memcpy (s, map->new_prefix, map->new_len);
+   memcpy (s + map->new_len, name, name_len);
++  free (realpath);
+   return ggc_strdup (s);
+ }
+ \f
+-- 
+2.7.4
+
-- 
1.9.1

[PATCH 03/17] conf/bitbake.conf package.bbclass: improve dbg package sources generation from work-shared

[Hongxu Jia]

Previously, the dbg package could not generate sources if they
are in work-shared dir (toolchain, kernel).

The fix add a new fdebug-prefix-map to remap work-shared dir
and collect sources to dbg package.

[YOCTO #9305]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/package.bbclass | 100 +++++++++++++++++++++++++++++++------------
 meta/conf/bitbake.conf       |   1 +
 2 files changed, 73 insertions(+), 28 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index bdbe96d..f862d0b 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -300,6 +300,14 @@ def get_conffiles(pkg, d):
     os.chdir(cwd)
     return conf_list
 
+def checkworkshared(file, d):
+    with open(file) as f:
+        file_content = f.read()
+        if "work-shared" in file_content:
+            return True
+
+    return False
+
 def checkbuildpath(file, d):
     tmpdir = d.getVar('TMPDIR', True)
     with open(file) as f:
@@ -309,6 +317,50 @@ def checkbuildpath(file, d):
 
     return False
 
+def collectsources(sourcefile, prefix_new, prefix_old, dest, d):
+    bb.utils.mkdirhier(dest)
+
+    # List all available sources
+    cmd =  "LC_ALL=C ; sort -z -u '%s' | " % sourcefile
+
+    # Filter out <internal> and <built-in>
+    cmd +=  "egrep -v -z '(<internal>|<built-in>)$' | "
+
+    # We need to ignore files that are not actually ours
+    # we do this by only paying attention to items from this package
+    cmd += "fgrep -zw '%s' | " % prefix_new
+
+    # Remove prefix in the source paths
+    cmd += "sed 's#%s/##g' | " % prefix_new
+
+    # If sources in WORKDIR, it ignores work-shared;
+    # If sources in work-shared, it filters nothing;
+    cmd += " sed -z '/work-shared/d' | "
+
+    # Enter dir prefix_old to copy sources
+    cmd += "(cd '%s' ; cpio -pd0mlL --no-preserve-owner '%s' 2>/dev/null)" % (prefix_old, dest)
+
+    (retval, output) = oe.utils.getstatusoutput(cmd)
+    # Can "fail" if internal headers/transient sources are attempted
+    #if retval:
+    #    bb.fatal("debug source copy failed with exit code %s (cmd was %s)" % (retval, cmd))
+
+# cpio seems to have a bug with -lL together and symbolic links are just copied, not dereferenced.
+# Work around this by manually finding and copying any symbolic links that made it through.
+def dereferencesymlink(prefix_old, dest, d):
+    # Enter dir dest, list all available symlinks that were removed prefix dest
+    cmd = "find %s -type l -print0 -delete | sed 's#%s/##g' | " % (dest, dest)
+
+    # If sources in WORKDIR, it ignores work-shared;
+    # If sources in work-shared, it filters nothing;
+    cmd += " sed '/work-shared/d' | "
+
+    # Enter dir prefix_old to copy sources to dest
+    cmd += "(cd '%s'; cpio -pd0mL --no-preserve-owner '%s' 2>/dev/null)" % (prefix_old, dest)
+    (retval, output) = oe.utils.getstatusoutput(cmd)
+    if retval:
+        bb.fatal("debugsrc symlink fixup failed with exit code %s (cmd was %s)" % (retval, cmd))
+
 def splitdebuginfo(file, debugfile, debugsrcdir, sourcefile, d):
     # Function to split a single file into two components, one is the stripped
     # target system binary, the other contains any debugging information. The
@@ -370,15 +422,6 @@ def copydebugsources(debugsrcdir, d):
         objcopy = d.getVar("OBJCOPY", True)
         debugedit = d.expand("${STAGING_LIBDIR_NATIVE}/rpm/bin/debugedit")
         workdir = d.getVar("WORKDIR", True)
-        workparentdir = os.path.dirname(os.path.dirname(workdir))
-        workbasedir = os.path.basename(os.path.dirname(workdir)) + "/" + os.path.basename(workdir)
-
-        # If build path exists in sourcefile, it means toolchain did not use
-        # -fdebug-prefix-map to compile
-        if checkbuildpath(sourcefile, d):
-            localsrc_prefix = workparentdir + "/"
-        else:
-            localsrc_prefix = "/usr/src/debug/"
 
         nosuchdir = []
         basepath = dvar
@@ -389,26 +432,27 @@ def copydebugsources(debugsrcdir, d):
         bb.utils.mkdirhier(basepath)
         cpath.updatecache(basepath)
 
-        processdebugsrc =  "LC_ALL=C ; sort -z -u '%s' | egrep -v -z '(<internal>|<built-in>)$' | "
-        # We need to ignore files that are not actually ours
-        # we do this by only paying attention to items from this package
-        processdebugsrc += "fgrep -zw '%s' | "
-        # Remove prefix in the source paths
-        processdebugsrc += "sed 's#%s##g' | "
-        processdebugsrc += "(cd '%s' ; cpio -pd0mlL --no-preserve-owner '%s%s' 2>/dev/null)"
-
-        cmd = processdebugsrc % (sourcefile, workbasedir, localsrc_prefix, workparentdir, dvar, debugsrcdir)
-        (retval, output) = oe.utils.getstatusoutput(cmd)
-        # Can "fail" if internal headers/transient sources are attempted
-        #if retval:
-        #    bb.fatal("debug source copy failed with exit code %s (cmd was %s)" % (retval, cmd))
+        # Collect sources in work-shared
+        if checkworkshared(sourcefile, d):
+            if checkbuildpath(sourcefile, d):
+                prefix_new = prefix_old = d.expand("${TMPDIR}/work-shared")
+            else:
+                prefix_old = d.expand("${TMPDIR}/work-shared")
+                prefix_new = d.expand("/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR}/work-shared")
+            dest = "%s%s" % (dvar, d.expand("/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR}/work-shared"))
+            collectsources(sourcefile, prefix_new, prefix_old, dest, d)
+            dereferencesymlink(prefix_old, dest, d)
 
-        # cpio seems to have a bug with -lL together and symbolic links are just copied, not dereferenced.
-        # Work around this by manually finding and copying any symbolic links that made it through.
-        cmd = "find %s%s -type l -print0 -delete | sed s#%s%s/##g | (cd '%s' ; cpio -pd0mL --no-preserve-owner '%s%s' 2>/dev/null)" % (dvar, debugsrcdir, dvar, debugsrcdir, workparentdir, dvar, debugsrcdir)
-        (retval, output) = oe.utils.getstatusoutput(cmd)
-        if retval:
-            bb.fatal("debugsrc symlink fixup failed with exit code %s (cmd was %s)" % (retval, cmd))
+        # If build path exists in sourcefile, it means toolchain did not use
+        # -fdebug-prefix-map to compile
+        if checkbuildpath(sourcefile, d):
+            prefix_new = prefix_old = workdir
+        else:
+            prefix_old = workdir
+            prefix_new = d.expand("/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR}")
+        dest = "%s%s" % (dvar, d.expand("/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR}"))
+        collectsources(sourcefile, prefix_new, prefix_old, dest, d)
+        dereferencesymlink(prefix_old, dest, d)
 
         # The copy by cpio may have resulted in some empty directories!  Remove these
         cmd = "find %s%s -empty -type d -delete" % (dvar, debugsrcdir)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 0eb288e..007ae0f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -550,6 +550,7 @@ EXTRA_OEMAKE_prepend_task-install = "${PARALLEL_MAKEINST} "
 # Optimization flags.
 ##################################################################
 DEBUG_FLAGS ?= "-g -feliminate-unused-debug-types \
+                -fdebug-prefix-map=${TMPDIR}/work-shared=/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR}/work-shared \
                 -fdebug-prefix-map=${WORKDIR}=/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR} \
                 -fdebug-prefix-map=${STAGING_DIR_NATIVE}= \
                 -fdebug-prefix-map=${STAGING_DIR_HOST}= \
-- 
1.9.1

[PATCH 04/17] dtc.inc: fix buildpaths QA issue

[Hongxu Jia]

Previous Makefile define its own CFLAGS, pass DEBUG_FLAGS to
it could fix build path issue in DWARF.

[YOCTO #8457]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-kernel/dtc/dtc.inc                    |  3 +++
 .../dtc/dtc/pass-DEBUG_FLAGS-to-CFLAGS.patch       | 30 ++++++++++++++++++++++
 2 files changed, 33 insertions(+)
 create mode 100644 meta/recipes-kernel/dtc/dtc/pass-DEBUG_FLAGS-to-CFLAGS.patch

diff --git a/meta/recipes-kernel/dtc/dtc.inc b/meta/recipes-kernel/dtc/dtc.inc
index 0c409b0..a1a30fc 100644
--- a/meta/recipes-kernel/dtc/dtc.inc
+++ b/meta/recipes-kernel/dtc/dtc.inc
@@ -7,8 +7,11 @@ DEPENDS = "flex-native bison-native"
 SRC_URI = "git://git.kernel.org/pub/scm/utils/dtc/dtc.git \
            file://make_install.patch \
 	  "
+SRC_URI_append_class-target = " file://pass-DEBUG_FLAGS-to-CFLAGS.patch \
+"
 
 EXTRA_OEMAKE='PREFIX="${prefix}" LIBDIR="${libdir}" DESTDIR="${D}"'
+EXTRA_OEMAKE_class-target += 'DEBUG_FLAGS="${DEBUG_FLAGS}"'
 
 S = "${WORKDIR}/git"
 
diff --git a/meta/recipes-kernel/dtc/dtc/pass-DEBUG_FLAGS-to-CFLAGS.patch b/meta/recipes-kernel/dtc/dtc/pass-DEBUG_FLAGS-to-CFLAGS.patch
new file mode 100644
index 0000000..3f17ee9
--- /dev/null
+++ b/meta/recipes-kernel/dtc/dtc/pass-DEBUG_FLAGS-to-CFLAGS.patch
@@ -0,0 +1,30 @@
+From 4019e29b0c1ec441fa43d9969197c24ad73f3b1a Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <hongxu.jia@windriver.com>
+Date: Fri, 26 Feb 2016 03:28:15 -0500
+Subject: [PATCH] Makefile: pass DEBUG_FLAGS to CFLAGS
+
+It fixes buildpaths issue while compiling.
+
+Upstream-Status: Inappropriate [openembedded specific]
+
+Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
+---
+ Makefile | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/Makefile b/Makefile
+index 18838cc..4954f32 100644
+--- a/Makefile
++++ b/Makefile
+@@ -18,7 +18,7 @@ CONFIG_LOCALVERSION =
+ CPPFLAGS = -I libfdt -I .
+ WARNINGS = -Werror -Wall -Wpointer-arith -Wcast-qual -Wnested-externs \
+ 	-Wstrict-prototypes -Wmissing-prototypes -Wredundant-decls -Wshadow
+-CFLAGS = -g -Os -fPIC -Werror $(WARNINGS)
++CFLAGS = $(DEBUG_FLAGS) -Os -fPIC -Werror $(WARNINGS)
+ 
+ BISON = bison
+ LEX = flex
+-- 
+1.9.1
+
-- 
1.9.1

[PATCH 05/17] fix_buildpaths.bbclass: add bbclass to fix build path

[Hongxu Jia]

Define variable PACKAGE_BUILDPATH_TEXT_FILES to list files that have
build paths and remove these paths at do_package time. It supports
package override while files exist in conditionally generated package.
(such as ${PN}-ptest is conditionally generated)

Define variable PACKAGE_BUILDPATH_TEXT_PATTERNS to list build path
patterns, which used by sed, it removes --sysroot and
-fdebug-prefix-map in text files by default.

Define python function to remove build path in variable.

Define python function to remove build path in python compiled
code.

[YOCTO #9169]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/fix_buildpaths.bbclass | 66 +++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)
 create mode 100644 meta/classes/fix_buildpaths.bbclass

diff --git a/meta/classes/fix_buildpaths.bbclass b/meta/classes/fix_buildpaths.bbclass
new file mode 100644
index 0000000..7ab7c66
--- /dev/null
+++ b/meta/classes/fix_buildpaths.bbclass
@@ -0,0 +1,66 @@
+PACKAGE_PREPROCESS_FUNCS += 'remove_buildpath_package_preprocess'
+
+def list_package_buildpath_text_files(d):
+    files = d.getVar('PACKAGE_BUILDPATH_TEXT_FILES', True) or ''
+
+    # Support package override
+    for pn in d.getVar('PACKAGES', True).split():
+        files += d.getVar('PACKAGE_BUILDPATH_TEXT_FILES_%s' % pn, True) or ''
+
+    return files
+
+remove_buildpath_package_preprocess () {
+	buildpath_files="${@list_package_buildpath_text_files(d)}"
+
+	# Remove build paths in text files
+	for file in $buildpath_files;do
+		sed -i ${@get_package_buildpath_text_patterns(d)} ${PKGD}$file
+	done
+}
+
+def get_package_buildpath_text_patterns(d):
+    result = d.getVar("PACKAGE_BUILDPATH_TEXT_PATTERNS", True) or ""
+
+    # Add --sysroot as default pattern
+    toolchain_opt = d.getVar("TOOLCHAIN_OPTIONS", True).split()
+    for opt in toolchain_opt:
+        if '--sysroot' in opt:
+            result += " -e 's:%s::g'" % opt
+
+    # Add -fdebug-prefix-map as default pattern
+    debug_flags = d.getVar('DEBUG_FLAGS', True).split()
+    for opt in debug_flags:
+        if '-fdebug-prefix-map=' in opt:
+            result += " -e 's:%s::g'" % opt
+
+    return result
+
+# List build path patterns, which used by sed
+PACKAGE_BUILDPATH_TEXT_PATTERNS = ""
+
+# List files which have build paths, and remove these paths.
+PACKAGE_BUILDPATH_TEXT_FILES ??= ""
+
+# Remove build path in variable
+def remove_buildpath_variable(d, var):
+    val = d.getVar(var, True) or ''
+
+    toolchain_opts = d.getVar('TOOLCHAIN_OPTIONS', True)
+    val = val.replace(toolchain_opts, '')
+
+    debug_flags = d.getVar('DEBUG_FLAGS', True).split()
+    for opt in debug_flags:
+        if '-fdebug-prefix-map=' in opt:
+            val = val.replace(opt, '')
+
+    return val
+
+# Remove build path in python compiled code which
+# located in root_path
+def remove_buildpath_bytecode(root_path, byte_code):
+    import py_compile
+
+    file = root_path + byte_code[0:-1]
+    dfile = byte_code[0:-1]
+    py_compile.compile(file, dfile=dfile)
+
-- 
1.9.1

[PATCH 06/17] icu: fix buildpaths QA issue

[Hongxu Jia]

- Define new variable U_CC and U_CXX which do not have build path.
  The output of icuinfo will not have build path in it:
  ...
  root@qemux86-64:~# icuinfo
   <icuSystemParams type="icu4c">
      [snip]
      <param name="build.cc">x86_64-poky-linux-gcc -m64 -march=core2 -mtune=core2 -msse3 -mfpmath=sse
      <param name="build.cxx">x86_64-poky-linux-g++ -m64 -march=core2 -mtune=core2 -msse3 -mfpmath=sse
      [snip]
   </icuSystemParams>
  ...

- Remove build paths in pkgdata.inc, Makefile.inc and
  icu-config.

[YOCTO #7058]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 .../icu/icu/add-variable-U_CC-and-U_CXX.patch      | 61 ++++++++++++++++++++++
 meta/recipes-support/icu/icu_56.1.bb               | 12 +++++
 2 files changed, 73 insertions(+)
 create mode 100644 meta/recipes-support/icu/icu/add-variable-U_CC-and-U_CXX.patch

diff --git a/meta/recipes-support/icu/icu/add-variable-U_CC-and-U_CXX.patch b/meta/recipes-support/icu/icu/add-variable-U_CC-and-U_CXX.patch
new file mode 100644
index 0000000..826d934
--- /dev/null
+++ b/meta/recipes-support/icu/icu/add-variable-U_CC-and-U_CXX.patch
@@ -0,0 +1,61 @@
+From 3221b108a18edeff5424bc5c66589eb20148d9b2 Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <hongxu.jia@windriver.com>
+Date: Wed, 2 Mar 2016 21:18:34 -0500
+Subject: [PATCH] add variable U_CC and U_CXX
+
+Define new variable U_CC and U_CXX which do not have build path.
+The output of icuinfo will not have build path in it:
+...
+root@qemux86-64:~# icuinfo
+ <icuSystemParams type="icu4c">
+    [snip]
+    <param name="build.cc">x86_64-poky-linux-gcc -m64 -march=core2 -mtune=core2 -msse3 -mfpmath=sse
+    <param name="build.cxx">x86_64-poky-linux-g++ -m64 -march=core2 -mtune=core2 -msse3 -mfpmath=sse
+    [snip]
+ </icuSystemParams>
+...
+
+Upstream-Status: Inappropriate [openembedded specific]
+
+Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
+---
+ configure.ac               | 9 +++++++++
+ tools/toolutil/Makefile.in | 2 +-
+ 2 files changed, 10 insertions(+), 1 deletion(-)
+
+diff --git a/configure.ac b/configure.ac
+index 595bd12..ec05c19 100644
+--- a/configure.ac
++++ b/configure.ac
+@@ -673,6 +673,15 @@ AC_SUBST(GENCCODE_ASSEMBLY)
+ 
+ AC_MSG_RESULT($GENCCODE_ASSEMBLY)
+ 
++# U_CC and U_CXX are used for icuinfo which remove
++# --sysroot for cross-compilation
++U_CC=${U_CC}
++AC_SUBST(U_CC)
++AC_MSG_RESULT(U_CC)
++U_CXX=${U_CXX}
++AC_SUBST(U_CXX)
++AC_MSG_RESULT(U_CXX)
++
+ # Checks for header files
+ AC_CHECK_HEADERS(inttypes.h)
+ if test $ac_cv_header_inttypes_h = no; then
+diff --git a/tools/toolutil/Makefile.in b/tools/toolutil/Makefile.in
+index 1fb42ed..a083522 100644
+--- a/tools/toolutil/Makefile.in
++++ b/tools/toolutil/Makefile.in
+@@ -46,7 +46,7 @@ CXXFLAGS += $(LIBCXXFLAGS)
+ CPPFLAGS += -I$(top_srcdir)/common -I$(top_srcdir)/i18n $(LIBCPPFLAGS)
+ 
+ # from icuinfo
+-CPPFLAGS+=  "-DU_BUILD=\"@build@\"" "-DU_HOST=\"@host@\"" "-DU_CC=\"@CC@\"" "-DU_CXX=\"@CXX@\""
++CPPFLAGS+=  "-DU_BUILD=\"@build@\"" "-DU_HOST=\"@host@\"" "-DU_CC=\"@U_CC@\"" "-DU_CXX=\"@U_CXX@\""
+ CPPFLAGS += -DUNISTR_FROM_CHAR_EXPLICIT=explicit -DUNISTR_FROM_STRING_EXPLICIT=explicit
+ 
+ DEFS += -DU_TOOLUTIL_IMPLEMENTATION
+-- 
+1.9.1
+
diff --git a/meta/recipes-support/icu/icu_56.1.bb b/meta/recipes-support/icu/icu_56.1.bb
index db0de10..1fd4dfc 100644
--- a/meta/recipes-support/icu/icu_56.1.bb
+++ b/meta/recipes-support/icu/icu_56.1.bb
@@ -17,6 +17,7 @@ SRC_URI = "${BASE_SRC_URI} \
            file://icu-pkgdata-large-cmd.patch \
            file://fix-install-manx.patch \
            file://icu-release-56-1-flagparser-fix.patch \
+           file://add-variable-U_CC-and-U_CXX.patch \
           "
 
 SRC_URI_append_class-target = "\
@@ -27,3 +28,14 @@ SRC_URI[sha256sum] = "3a64e9105c734dcf631c0b3ed60404531bce6c0f5a64bfe1a6402a4cc2
 
 UPSTREAM_CHECK_REGEX = "(?P<pver>\d+(\.\d+)+)/"
 UPSTREAM_CHECK_URI = "http://download.icu-project.org/files/icu4c/"
+
+inherit fix_buildpaths
+CACHED_CONFIGUREVARS += "U_CC="${@remove_buildpath_variable(d, 'CC')}" \
+                         U_CXX="${@remove_buildpath_variable(d, 'CXX')}" \
+"
+PACKAGE_BUILDPATH_TEXT_FILES_${PN}-dev ?= " \
+    ${libdir}/${BPN}/${PV}/pkgdata.inc \
+    ${libdir}/${BPN}/${PV}/Makefile.inc \
+    ${bindir}/icu-config \
+"
+
-- 
1.9.1

[PATCH 07/17] tcl: fix buildpath QA issue

[Hongxu Jia]

- Tweak DEBUG_FLAGS, since ${S} is not normal format

- Remove build path in tclConfig.sh

- Tweak sed expression at tclConfig.sh to strict the pattern

[YOCTO #8457]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-devtools/tcltk/tcl_8.6.4.bb | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-devtools/tcltk/tcl_8.6.4.bb b/meta/recipes-devtools/tcltk/tcl_8.6.4.bb
index 8e92b3e..2aeb952 100644
--- a/meta/recipes-devtools/tcltk/tcl_8.6.4.bb
+++ b/meta/recipes-devtools/tcltk/tcl_8.6.4.bb
@@ -54,8 +54,8 @@ do_install() {
 	autotools_do_install install-private-headers
 	ln -sf ./tclsh${VER} ${D}${bindir}/tclsh
 	ln -sf tclsh8.6 ${D}${bindir}/tclsh${VER}
-	sed -i "s+-L${B}+-L${STAGING_LIBDIR}+g" tclConfig.sh
-	sed -i "s+${WORKDIR}+${STAGING_INCDIR}+g" tclConfig.sh
+	sed -i "s,-L${B},-L${STAGING_LIBDIR},g" tclConfig.sh
+	sed -i "s,='${WORKDIR},='${STAGING_INCDIR},g" tclConfig.sh
 	sed -i "s,-L${libdir},-L=${libdir},g" tclConfig.sh
 	sed -i "s,-I${includedir},-I=${includedir},g" tclConfig.sh
 	install -d ${D}${bindir_crossscripts}
@@ -73,6 +73,11 @@ tcl_sysroot_preprocess () {
 	sysroot_stage_dir ${D}${bindir_crossscripts} ${SYSROOT_DESTDIR}${bindir_crossscripts}
 }
 
+inherit fix_buildpaths
+PACKAGE_BUILDPATH_TEXT_FILES ?= "${bindir_crossscripts}/tclConfig.sh \
+                                 ${libdir}/tclConfig.sh \
+"
+
 PACKAGES =+ "tcl-lib"
 FILES_tcl-lib = "${libdir}/libtcl8.6.so.*"
 FILES_${PN} += "${libdir}/tcl${VER} ${libdir}/tcl8.6 ${libdir}/tcl8"
-- 
1.9.1

[PATCH 08/17] python2/3: fix buildpath QA issue

[Hongxu Jia]

- Remove build paths in target file

- Since _sysconfigdata.py was modified at do_package,
  so update _sysconfigdata.pyc manually (Python2 only).

[YOCTO #8446]
[YOCTO #9169]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-devtools/python/python3_3.5.1.bb | 20 ++++++++++++++----
 meta/recipes-devtools/python/python_2.7.11.bb | 29 ++++++++++++++++++++++-----
 2 files changed, 40 insertions(+), 9 deletions(-)

diff --git a/meta/recipes-devtools/python/python3_3.5.1.bb b/meta/recipes-devtools/python/python3_3.5.1.bb
index 2fbb3cb..bfcea85 100644
--- a/meta/recipes-devtools/python/python3_3.5.1.bb
+++ b/meta/recipes-devtools/python/python3_3.5.1.bb
@@ -186,12 +186,24 @@ PACKAGE_PREPROCESS_FUNCS += "py_package_preprocess"
 py_package_preprocess () {
 	# copy back the old Makefile to fix target package
 	install -m 0644 ${B}/Makefile.orig ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile
-	# Remove references to buildmachine paths in target Makefile and _sysconfigdata
-	sed -i -e 's:--sysroot=${STAGING_DIR_TARGET}::g' -e s:'--with-libtool-sysroot=${STAGING_DIR_TARGET}'::g \
-		${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile \
-		${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata.py
 }
 
+# Remove references to buildmachine paths in target
+inherit fix_buildpaths
+PACKAGE_BUILDPATH_TEXT_FILES ?= " \
+    ${libdir}/python${PYTHON_MAJMIN}/config/Makefile \
+    ${libdir}/python${PYTHON_MAJMIN}/config-${PYTHON_BINABI}/Makefile \
+    ${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata.py \
+    ${bindir}/python${PYTHON_BINABI}-config \
+"
+PACKAGE_BUILDPATH_TEXT_PATTERNS += " \
+    -e 's:--with-libtool-sysroot=${STAGING_DIR_TARGET}::g' \
+    -e 's:${STAGING_DIR}/${MACHINE}${datadir}/pkgconfig:${datadir}/pkgconfig:g' \
+    -e 's:${STAGING_DIR_HOST}::g' \
+    -e 's:${STAGING_DIR_NATIVE}::g' \
+    -e 's:${WORKDIR}::g' \
+"
+
 require python-${PYTHON_MAJMIN}-manifest.inc
 
 # manual dependency additions
diff --git a/meta/recipes-devtools/python/python_2.7.11.bb b/meta/recipes-devtools/python/python_2.7.11.bb
index 0c00d6e..3cf8642 100644
--- a/meta/recipes-devtools/python/python_2.7.11.bb
+++ b/meta/recipes-devtools/python/python_2.7.11.bb
@@ -128,15 +128,34 @@ do_install_append_class-nativesdk () {
 
 SSTATE_SCAN_FILES += "Makefile"
 PACKAGE_PREPROCESS_FUNCS += "py_package_preprocess"
-
 py_package_preprocess () {
 	# copy back the old Makefile to fix target package
 	install -m 0644 ${B}/Makefile.orig ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile
+}
+
+# Remove references to buildmachine paths in target
+inherit fix_buildpaths
+PACKAGE_BUILDPATH_TEXT_FILES ?= " \
+    ${libdir}/python${PYTHON_MAJMIN}/config/Makefile \
+    ${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata.py \
+"
+PACKAGE_BUILDPATH_TEXT_FILES_${PN}-ptest ?= " \
+    ${PTEST_PATH}/Makefile \
+"
+PACKAGE_BUILDPATH_TEXT_PATTERNS += " \
+    -e 's:--with-libtool-sysroot=${STAGING_DIR_TARGET}::g' \
+    -e 's:${STAGING_DIR}/${MACHINE}${datadir}/pkgconfig:${datadir}/pkgconfig:g' \
+    -e 's:${STAGING_DIR_HOST}::g' \
+    -e 's:${STAGING_DIR_NATIVE}::g' \
+    -e 's:${WORKDIR}::g' \
+"
 
-	# Remove references to buildmachine paths in target Makefile and _sysconfigdata
-	sed -i -e 's:--sysroot=${STAGING_DIR_TARGET}::g' -e s:'--with-libtool-sysroot=${STAGING_DIR_TARGET}'::g \
-		${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile \
-		${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata.py
+PACKAGE_PREPROCESS_FUNCS += "update_pyc_package_preprocess"
+python update_pyc_package_preprocess () {
+    # Update _sysconfigdata.pyc
+    root_path = d.expand("${PKGD}")
+    byte_code = d.expand("${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata.pyc")
+    remove_buildpath_bytecode(root_path, byte_code)
 }
 
 require python-${PYTHON_MAJMIN}-manifest.inc
-- 
1.9.1

[PATCH 09/17] bbclass distutils/distutils3: fix .pyc/.pyo buildpath

[Hongxu Jia]

Get the "--root" directory supplied to the "install" command,
and use it as a prefix to strip off the purported filename
encoded in bytecode files.

Since --root added, we need to tweak --install-data,
--prefix and --install-lib to use relative path.

[YOCTO #8446]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/distutils.bbclass  | 6 +++---
 meta/classes/distutils3.bbclass | 6 +++---
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/meta/classes/distutils.bbclass b/meta/classes/distutils.bbclass
index da48a2e..8427c1a 100644
--- a/meta/classes/distutils.bbclass
+++ b/meta/classes/distutils.bbclass
@@ -4,8 +4,8 @@ DISTUTILS_BUILD_ARGS ?= ""
 DISTUTILS_STAGE_HEADERS_ARGS ?= "--install-dir=${STAGING_INCDIR}/${PYTHON_DIR}"
 DISTUTILS_STAGE_ALL_ARGS ?= "--prefix=${STAGING_DIR_HOST}${prefix} \
     --install-data=${STAGING_DATADIR}"
-DISTUTILS_INSTALL_ARGS ?= "--prefix=${D}/${prefix} \
-    --install-data=${D}/${datadir}"
+DISTUTILS_INSTALL_ARGS ?= "--prefix=${prefix} \
+    --install-data=${datadir}"
 
 distutils_do_compile() {
          STAGING_INCDIR=${STAGING_INCDIR} \
@@ -38,7 +38,7 @@ distutils_do_install() {
         STAGING_LIBDIR=${STAGING_LIBDIR} \
         PYTHONPATH=${D}${PYTHON_SITEPACKAGES_DIR} \
         BUILD_SYS=${BUILD_SYS} HOST_SYS=${HOST_SYS} \
-        ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py install --install-lib=${D}/${PYTHON_SITEPACKAGES_DIR} ${DISTUTILS_INSTALL_ARGS} || \
+        ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py install --root=${D} --install-lib=${PYTHON_SITEPACKAGES_DIR} ${DISTUTILS_INSTALL_ARGS} || \
         bbfatal_log "${PYTHON_PN} setup.py install execution failed."
 
         # support filenames with *spaces*
diff --git a/meta/classes/distutils3.bbclass b/meta/classes/distutils3.bbclass
index 4f6ca44..a98606b 100644
--- a/meta/classes/distutils3.bbclass
+++ b/meta/classes/distutils3.bbclass
@@ -5,8 +5,8 @@ DISTUTILS_BUILD_EXT_ARGS ?= ""
 DISTUTILS_STAGE_HEADERS_ARGS ?= "--install-dir=${STAGING_INCDIR}/${PYTHON_DIR}"
 DISTUTILS_STAGE_ALL_ARGS ?= "--prefix=${STAGING_DIR_HOST}${prefix} \
     --install-data=${STAGING_DATADIR}"
-DISTUTILS_INSTALL_ARGS ?= "--prefix=${D}/${prefix} \
-    --install-data=${D}/${datadir}"
+DISTUTILS_INSTALL_ARGS ?= "--prefix=${prefix} \
+    --install-data=${datadir}"
 
 distutils3_do_compile() {
         if [ ${BUILD_SYS} != ${HOST_SYS} ]; then
@@ -63,7 +63,7 @@ distutils3_do_install() {
         STAGING_LIBDIR=${STAGING_LIBDIR} \
         PYTHONPATH=${D}${PYTHON_SITEPACKAGES_DIR} \
         BUILD_SYS=${BUILD_SYS} HOST_SYS=${SYS} \
-        ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py install --install-lib=${D}/${PYTHON_SITEPACKAGES_DIR} ${DISTUTILS_INSTALL_ARGS} || \
+        ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py install --root=${D} --install-lib=${PYTHON_SITEPACKAGES_DIR} ${DISTUTILS_INSTALL_ARGS} || \
         bbfatal_log "${PYTHON_PN} setup.py install execution failed."
 
         # support filenames with *spaces*
-- 
1.9.1

[PATCH 10/17] bbclass distutils/distutils3/setuptools/setuptools3: clean up DISTUTILS_INSTALL_ARGS

[Hongxu Jia]

Since fix .pyc/.pyo buildpaths issue, we have the same
DISTUTILS_INSTALL_ARGS in these bbclass files, so clean
up the duplicate.

[YOCTO #8446]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/distutils.bbclass   | 4 +++-
 meta/classes/distutils3.bbclass  | 4 +++-
 meta/classes/setuptools.bbclass  | 5 -----
 meta/classes/setuptools3.bbclass | 5 -----
 4 files changed, 6 insertions(+), 12 deletions(-)

diff --git a/meta/classes/distutils.bbclass b/meta/classes/distutils.bbclass
index 8427c1a..407ccd4 100644
--- a/meta/classes/distutils.bbclass
+++ b/meta/classes/distutils.bbclass
@@ -5,6 +5,8 @@ DISTUTILS_STAGE_HEADERS_ARGS ?= "--install-dir=${STAGING_INCDIR}/${PYTHON_DIR}"
 DISTUTILS_STAGE_ALL_ARGS ?= "--prefix=${STAGING_DIR_HOST}${prefix} \
     --install-data=${STAGING_DATADIR}"
 DISTUTILS_INSTALL_ARGS ?= "--prefix=${prefix} \
+    --root=${D} \
+    --install-lib=${PYTHON_SITEPACKAGES_DIR} \
     --install-data=${datadir}"
 
 distutils_do_compile() {
@@ -38,7 +40,7 @@ distutils_do_install() {
         STAGING_LIBDIR=${STAGING_LIBDIR} \
         PYTHONPATH=${D}${PYTHON_SITEPACKAGES_DIR} \
         BUILD_SYS=${BUILD_SYS} HOST_SYS=${HOST_SYS} \
-        ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py install --root=${D} --install-lib=${PYTHON_SITEPACKAGES_DIR} ${DISTUTILS_INSTALL_ARGS} || \
+        ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py install ${DISTUTILS_INSTALL_ARGS} || \
         bbfatal_log "${PYTHON_PN} setup.py install execution failed."
 
         # support filenames with *spaces*
diff --git a/meta/classes/distutils3.bbclass b/meta/classes/distutils3.bbclass
index a98606b..5da93fa 100644
--- a/meta/classes/distutils3.bbclass
+++ b/meta/classes/distutils3.bbclass
@@ -6,6 +6,8 @@ DISTUTILS_STAGE_HEADERS_ARGS ?= "--install-dir=${STAGING_INCDIR}/${PYTHON_DIR}"
 DISTUTILS_STAGE_ALL_ARGS ?= "--prefix=${STAGING_DIR_HOST}${prefix} \
     --install-data=${STAGING_DATADIR}"
 DISTUTILS_INSTALL_ARGS ?= "--prefix=${prefix} \
+    --root=${D} \
+    --install-lib=${PYTHON_SITEPACKAGES_DIR} \
     --install-data=${datadir}"
 
 distutils3_do_compile() {
@@ -63,7 +65,7 @@ distutils3_do_install() {
         STAGING_LIBDIR=${STAGING_LIBDIR} \
         PYTHONPATH=${D}${PYTHON_SITEPACKAGES_DIR} \
         BUILD_SYS=${BUILD_SYS} HOST_SYS=${SYS} \
-        ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py install --root=${D} --install-lib=${PYTHON_SITEPACKAGES_DIR} ${DISTUTILS_INSTALL_ARGS} || \
+        ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py install ${DISTUTILS_INSTALL_ARGS} || \
         bbfatal_log "${PYTHON_PN} setup.py install execution failed."
 
         # support filenames with *spaces*
diff --git a/meta/classes/setuptools.bbclass b/meta/classes/setuptools.bbclass
index 56343b1..0a3f0f9 100644
--- a/meta/classes/setuptools.bbclass
+++ b/meta/classes/setuptools.bbclass
@@ -1,8 +1,3 @@
 inherit distutils
 
 DEPENDS += "python-distribute-native"
-
-DISTUTILS_INSTALL_ARGS = "--root=${D} \
-    --prefix=${prefix} \
-    --install-lib=${PYTHON_SITEPACKAGES_DIR} \
-    --install-data=${datadir}"
diff --git a/meta/classes/setuptools3.bbclass b/meta/classes/setuptools3.bbclass
index de6dd94..38e175a 100644
--- a/meta/classes/setuptools3.bbclass
+++ b/meta/classes/setuptools3.bbclass
@@ -1,8 +1,3 @@
 inherit distutils3
 
 DEPENDS += "python3-setuptools-native"
-
-DISTUTILS_INSTALL_ARGS = "--root=${D} \
-    --prefix=${prefix} \
-    --install-lib=${PYTHON_SITEPACKAGES_DIR} \
-    --install-data=${datadir}"
-- 
1.9.1

[PATCH 11/17] python-setuptools/python3-setuptools: use old-style install

[Hongxu Jia]

Since '--root' is added to 'setup.py install' to fix .pyc/.pyo
buildpaths issue. For python setuptools, the '--root' option
means explicit request for old-style install. There will be
no egg generated. And it do not need setuptools.pth any more.

Rename easy_install to easy_install3.

Ubuntu or Debian did the same thing like us:
https://packages.debian.org/sid/all/python3-setuptools/filelist

[YOCTO #8446]
[YOCTO #9162]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-devtools/python/python-setuptools_19.4.bb  | 2 --
 meta/recipes-devtools/python/python3-setuptools_19.4.bb | 8 ++------
 2 files changed, 2 insertions(+), 8 deletions(-)

diff --git a/meta/recipes-devtools/python/python-setuptools_19.4.bb b/meta/recipes-devtools/python/python-setuptools_19.4.bb
index c172039..01763d4 100644
--- a/meta/recipes-devtools/python/python-setuptools_19.4.bb
+++ b/meta/recipes-devtools/python/python-setuptools_19.4.bb
@@ -7,8 +7,6 @@ DEPENDS_class-native += "python-native"
 
 inherit distutils
 
-DISTUTILS_INSTALL_ARGS += "--install-lib=${D}${libdir}/${PYTHON_DIR}/site-packages"
-
 RDEPENDS_${PN} = "\
   python-stringold \
   python-email \
diff --git a/meta/recipes-devtools/python/python3-setuptools_19.4.bb b/meta/recipes-devtools/python/python3-setuptools_19.4.bb
index fb2931c..12b0d93 100644
--- a/meta/recipes-devtools/python/python3-setuptools_19.4.bb
+++ b/meta/recipes-devtools/python/python3-setuptools_19.4.bb
@@ -5,13 +5,9 @@ DEPENDS_class-native += "python3-native"
 
 inherit distutils3
 
-DISTUTILS_INSTALL_ARGS += "--install-lib=${D}${libdir}/${PYTHON_DIR}/site-packages"
-
-# The installer puts the wrong path in the setuptools.pth file.  Correct it.
+# Rename easy_install to avoid confliction with python2's setuptools
 do_install_append() {
-    rm ${D}${PYTHON_SITEPACKAGES_DIR}/setuptools.pth
-    mv ${D}${bindir}/easy_install ${D}${bindir}/easy3_install
-    echo "./${SRCNAME}-${PV}-py${PYTHON_BASEVERSION}.egg" > ${D}${PYTHON_SITEPACKAGES_DIR}/setuptools.pth
+    mv ${D}${bindir}/easy_install ${D}${bindir}/easy_install3
 }
 
 RDEPENDS_${PN} = "\
-- 
1.9.1

[PATCH 12/17] python3-pip: use old-style install

[Hongxu Jia]

Since '--root' is added to 'setup.py install' to fix .pyc/.pyo
buildpaths issue. For python3-pip, the '--root' option
means there will be no eggs generated.

Ubuntu or Debian did the same thing like us:
https://packages.debian.org/sid/all/python3-pip/filelist

[YOCTO #8446]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-devtools/python/python3-pip_8.0.0.bb | 12 ------------
 1 file changed, 12 deletions(-)

diff --git a/meta/recipes-devtools/python/python3-pip_8.0.0.bb b/meta/recipes-devtools/python/python3-pip_8.0.0.bb
index 834ecda..b45aeea 100644
--- a/meta/recipes-devtools/python/python3-pip_8.0.0.bb
+++ b/meta/recipes-devtools/python/python3-pip_8.0.0.bb
@@ -19,22 +19,10 @@ S = "${WORKDIR}/${SRCNAME}-${PV}"
 
 inherit distutils3
 
-DISTUTILS_INSTALL_ARGS += "--install-lib=${D}${libdir}/${PYTHON_DIR}/site-packages"
-
-do_install_prepend() {
-    install -d ${D}/${libdir}/${PYTHON_DIR}/site-packages
-}
-
 # Use setuptools site.py instead, avoid shared state issue
 do_install_append() {
-    rm ${D}/${libdir}/${PYTHON_DIR}/site-packages/site.py
-    rm ${D}/${libdir}/${PYTHON_DIR}/site-packages/__pycache__/site.cpython-*.pyc
-
     # Install as pip3 and leave pip2 as default
     rm ${D}/${bindir}/pip
-
-    # Installed eggs need to be passed directly to the interpreter via a pth file
-    echo "./${SRCNAME}-${PV}-py${PYTHON_BASEVERSION}.egg" > ${D}${PYTHON_SITEPACKAGES_DIR}/${SRCNAME}-${PV}.pth
 }
 
 RDEPENDS_${PN} = "\
-- 
1.9.1

[PATCH 13/17] waf.bbclass: support do patch on extracted files

[Hongxu Jia]

Along with do_unpack, run './waf --help' to do the self extract,
so we could do patch on the extracted files later.

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/waf.bbclass | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/meta/classes/waf.bbclass b/meta/classes/waf.bbclass
index 5e55833..a36bdd4 100644
--- a/meta/classes/waf.bbclass
+++ b/meta/classes/waf.bbclass
@@ -23,8 +23,17 @@ def get_waf_parallel_make(d):
 
     return ""
 
+do_unpack[postfuncs] += "unpack_waf"
+do_unpack[vardeps] += "unpack_waf"
+unpack_waf() {
+	# Just self extract tarball embedded in waf, do not care help message.
+	# Do not generate bytecode, becuase we may need to patch python file
+        # later
+	PYTHONDONTWRITEBYTECODE=True ./waf --help >/dev/null
+}
+
 waf_do_configure() {
-	${S}/waf configure --prefix=${prefix} ${EXTRA_OECONF}
+	${CACHED_CONFIGUREVARS} ${S}/waf configure --prefix=${prefix} ${EXTRA_OECONF}
 }
 
 waf_do_compile()  {
-- 
1.9.1

[PATCH 14/17] python-pycairo: fix buildpath QA issue

[Hongxu Jia]

- Inherit waf.bbclass, remove duplicate tasks

- Get "--destdir" directory supplied to the "install" command,
  and use it as a prefix to strip off the purported filename
  encoded in bytecode files.

- Do not need to generate pycairo.pc manually

[YOCTO #8446]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 ...trip-off-build-path-in-purported-filename.patch | 62 ++++++++++++++++++++++
 .../python/python-pycairo_1.10.0.bb                | 25 +++------
 2 files changed, 68 insertions(+), 19 deletions(-)
 create mode 100644 meta/recipes-devtools/python/python-pycairo/strip-off-build-path-in-purported-filename.patch

diff --git a/meta/recipes-devtools/python/python-pycairo/strip-off-build-path-in-purported-filename.patch b/meta/recipes-devtools/python/python-pycairo/strip-off-build-path-in-purported-filename.patch
new file mode 100644
index 0000000..647c4be
--- /dev/null
+++ b/meta/recipes-devtools/python/python-pycairo/strip-off-build-path-in-purported-filename.patch
@@ -0,0 +1,62 @@
+From 86faf604fc8a7f17320db48734cda464119da387 Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <hongxu.jia@windriver.com>
+Date: Thu, 25 Feb 2016 04:22:07 -0500
+Subject: [PATCH] strip off build path in purported filename
+
+Get the "--destdir" directory supplied to the "install" command,
+and use it as a prefix to strip off the purported filename
+encoded in bytecode files.
+
+Upstream-Status: Inappropriate [openembedded specific]
+
+Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
+---
+diff --git a/.waf-1.6.3-3c3129a3ec8fb4a5bbc7ba3161463b22/waflib/Tools/python.py b/.waf-1.6.3-3c3129a3ec8fb4a5bbc7ba3161463b22/waflib/Tools/python.py
+--- a/.waf-1.6.3-3c3129a3ec8fb4a5bbc7ba3161463b22/waflib/Tools/python.py
++++ b/.waf-1.6.3-3c3129a3ec8fb4a5bbc7ba3161463b22/waflib/Tools/python.py
+@@ -26,7 +26,7 @@ int main()
+ INST='''
+ import sys, py_compile
+ for pyfile in sys.argv[1:]:
+-	py_compile.compile(pyfile, pyfile + %r)
++	py_compile.compile(pyfile, pyfile + %r, %r)
+ '''
+ def process_py(self,node):
+ 	try:
+@@ -39,6 +39,16 @@ def process_py(self,node):
+ 	def inst_py(ctx):
+ 		install_pyfile(self,node)
+ 	self.bld.add_post_fun(inst_py)
++
++def get_dfile(dfile):
++	if getattr(Options.options,'destdir'):
++		destdir = Options.options.destdir
++		if dfile[:len(destdir)] != destdir:
++			raise Errors.WafError("invalid destdir: filename %r doesn't start with %r" % ((dfile, destdir)))
++		dfile = dfile[len(destdir):]
++
++	return dfile
++
+ def install_pyfile(self,node):
+ 	tsk=self.bld.install_files(self.install_path,[node],postpone=False)
+ 	path=os.path.join(tsk.get_install_path(),node.name)
+@@ -53,12 +63,14 @@ def install_pyfile(self,node):
+ 		if self.env['PYC']or self.env['PYO']:
+ 			info("+ byte compiling %r"%path)
+ 		if self.env['PYC']:
+-			argv=self.env['PYTHON']+['-c',INST%'c',path]
++			dfile = get_dfile(path+'c')
++			argv=self.env['PYTHON']+['-c',INST%('c',dfile),path]
+ 			ret=Utils.subprocess.Popen(argv).wait()
+ 			if ret:
+ 				raise Errors.WafError('pyc compilation failed %r'%path)
+ 		if self.env['PYO']:
+-			argv=self.env['PYTHON']+[self.env['PYFLAGS_OPT'],'-c',INST%'o',path]
++			dfile = get_dfile(path+'o')
++			argv=self.env['PYTHON']+[self.env['PYFLAGS_OPT'],'-c',INST%('o',dfile),path]
+ 			ret=Utils.subprocess.Popen(argv).wait()
+ 			if ret:
+ 				raise Errors.WafError('pyo compilation failed %r'%path)
+-- 
+1.9.1
+
diff --git a/meta/recipes-devtools/python/python-pycairo_1.10.0.bb b/meta/recipes-devtools/python/python-pycairo_1.10.0.bb
index 70fe25c..2fbd033 100644
--- a/meta/recipes-devtools/python/python-pycairo_1.10.0.bb
+++ b/meta/recipes-devtools/python/python-pycairo_1.10.0.bb
@@ -11,31 +11,18 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=f2e071ab72978431b294a0d696327421 \
 DEPENDS = "cairo"
 PR = "r2"
 
-SRC_URI = "http://cairographics.org/releases/py2cairo-${PV}.tar.bz2"
+SRC_URI = "http://cairographics.org/releases/py2cairo-${PV}.tar.bz2 \
+           file://strip-off-build-path-in-purported-filename.patch \
+"
 
 SRC_URI[md5sum] = "20337132c4ab06c1146ad384d55372c5"
 SRC_URI[sha256sum] = "d30439f06c2ec1a39e27464c6c828b6eface3b22ee17b2de05dc409e429a7431"
 
 S = "${WORKDIR}/py2cairo-${PV}"
 
-inherit distutils pkgconfig
+inherit distutils pkgconfig waf
 
 BBCLASSEXTEND = "native"
 
-do_configure() {
-	BUILD_SYS=${BUILD_SYS} HOST_SYS=${HOST_SYS} ./waf configure --prefix=${D}${prefix} --libdir=${D}${libdir}
-}
-
-do_compile() {
-	./waf build ${PARALLEL_MAKE}
-}
-
-do_install() {
-	./waf install
-	sed \
-		-e 's:@prefix@:${prefix}:' \
-		-e 's:@VERSION@:${PV}:' \
-		-e 's:@includedir@:${includedir}:' \
-		pycairo.pc.in > pycairo.pc
-	install -m 0644 pycairo.pc ${D}${libdir}/pkgconfig/
-}
+CACHED_CONFIGUREVARS = "BUILD_SYS=${BUILD_SYS} HOST_SYS=${HOST_SYS}"
+EXTRA_OECONF += "--libdir=${libdir}"
-- 
1.9.1

[PATCH 15/17] openssl: fix buildpath QA issue

[Hongxu Jia]

- The openssl records $CC and $CFLAGS in source code which
  caused buildpaths QA issue:
  In crypto/cversion.c
  ...
      if (t == SSLEAY_CFLAGS) {
          return (CFLAGS);
          return ("compiler: information not available");
      }
  ...
  We should remove build path in the above source code.

- Remove build paths in ${PTEST_PATH}/Makefile  and
  ${PTEST_PATH}/Configure

[YOCTO #7058]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 .../openssl/remove-build-path-in-source-code.patch | 44 ++++++++++++++++++++++
 .../recipes-connectivity/openssl/openssl_1.0.2g.bb |  7 ++++
 2 files changed, 51 insertions(+)
 create mode 100644 meta/recipes-connectivity/openssl/openssl/remove-build-path-in-source-code.patch

diff --git a/meta/recipes-connectivity/openssl/openssl/remove-build-path-in-source-code.patch b/meta/recipes-connectivity/openssl/openssl/remove-build-path-in-source-code.patch
new file mode 100644
index 0000000..fd5531b
--- /dev/null
+++ b/meta/recipes-connectivity/openssl/openssl/remove-build-path-in-source-code.patch
@@ -0,0 +1,44 @@
+From b937d7581a1793cdebbf5829bdaea787ed4128db Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <hongxu.jia@windriver.com>
+Date: Mon, 7 Mar 2016 01:11:42 -0500
+Subject: [PATCH] crypto/Makefile: remove build path in source code
+
+Openssl records $CC and $CFLAGS in source code which
+caused buildpaths QA issue:
+In crypto/cversion.c
+...
+    if (t == SSLEAY_CFLAGS) {
+        return (CFLAGS);
+        return ("compiler: information not available");
+    }
+...
+
+We should remove build path in the above source code.
+
+Upstream-Status: Inappropriate [embedded specific]
+
+Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
+---
+ crypto/Makefile | 5 ++++-
+ 1 file changed, 4 insertions(+), 1 deletion(-)
+
+diff --git a/crypto/Makefile b/crypto/Makefile
+index 17a87f8..ee630e8 100644
+--- a/crypto/Makefile
++++ b/crypto/Makefile
+@@ -54,8 +54,11 @@ top:
+ 
+ all: shared
+ 
++NO_BUILDPATH_CCFLAGS=`echo "$(CC) $(CFLAGS)" | \
++         sed -e 's:--sysroot=[^ ]*::g' \
++             -e 's:-fdebug-prefix-map=[^ ]*::g'`
+ buildinf.h: ../Makefile
+-	$(PERL) $(TOP)/util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)" >buildinf.h
++	$(PERL) $(TOP)/util/mkbuildinf.pl "$(NO_BUILDPATH_CCFLAGS)" "$(PLATFORM)" >buildinf.h
+ 
+ x86cpuid.s:	x86cpuid.pl perlasm/x86asm.pl
+ 	$(PERL) x86cpuid.pl $(PERLASM_SCHEME) $(CFLAGS) $(PROCESSOR) > $@
+-- 
+1.9.1
+
diff --git a/meta/recipes-connectivity/openssl/openssl_1.0.2g.bb b/meta/recipes-connectivity/openssl/openssl_1.0.2g.bb
index 290f129..078f60e 100644
--- a/meta/recipes-connectivity/openssl/openssl_1.0.2g.bb
+++ b/meta/recipes-connectivity/openssl/openssl_1.0.2g.bb
@@ -39,6 +39,7 @@ SRC_URI += "file://find.pl;subdir=${BP}/util/ \
             file://ptest_makefile_deps.patch  \
             file://configure-musl-target.patch \
             file://parallel.patch \
+            file://remove-build-path-in-source-code.patch \
            "
 
 SRC_URI[md5sum] = "f3c710c045cdee5fd114feb69feba7aa"
@@ -47,6 +48,12 @@ SRC_URI[sha256sum] = "b784b1b3907ce39abf4098702dade6365522a253ad1552e267a9a0e895
 PACKAGES =+ "${PN}-engines"
 FILES_${PN}-engines = "${libdir}/ssl/engines/*.so ${libdir}/engines"
 
+inherit fix_buildpaths
+PACKAGE_BUILDPATH_TEXT_FILES_${PN}-ptest ?= " \
+        ${PTEST_PATH}/Makefile \
+        ${PTEST_PATH}/Configure \
+"
+
 # The crypto_use_bigint patch means that perl's bignum module needs to be
 # installed, but some distributions (for example Fedora 23) don't ship it by
 # default.  As the resulting error is very misleading check for bignum before
-- 
1.9.1

[PATCH 16/17] epiphany: fix buildpaths QA issue

[Hongxu Jia]

In order to fix build path issue, we reverts the following commit
in https://github.com/GNOME/epiphany.git
...
commit 54a3122d8626f78338033a704d5849738f6dd2e6
Author: Xan Lopez <xan@igalia.com>
Date:   Wed May 16 18:21:21 2012 +0900

    ephy-profile-utils: use absolute path for build dir

    That way we can run the tests from any directory in debug mode.
...
It did not consider cross-compile situation which we do not need
to run tests in host build.

BTW, the tests suits were compiled but not installed to ${D},
so we explicitly disable it.

[YOCTO #7058]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-gnome/epiphany/epiphany_3.18.4.bb     |  8 ++-
 .../files/use-relative-path-for-build-dir.patch    | 57 ++++++++++++++++++++++
 2 files changed, 63 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-gnome/epiphany/files/use-relative-path-for-build-dir.patch

diff --git a/meta/recipes-gnome/epiphany/epiphany_3.18.4.bb b/meta/recipes-gnome/epiphany/epiphany_3.18.4.bb
index 12d3b07..0b8335b 100644
--- a/meta/recipes-gnome/epiphany/epiphany_3.18.4.bb
+++ b/meta/recipes-gnome/epiphany/epiphany_3.18.4.bb
@@ -9,11 +9,15 @@ inherit gnomebase gsettings distro_features_check upstream-version-is-even
 # libwnck3 is x11 only
 REQUIRED_DISTRO_FEATURES = "x11"
 
-SRC_URI += "file://0001-yelp.m4-drop-the-check-for-itstool.patch"
+SRC_URI += "file://0001-yelp.m4-drop-the-check-for-itstool.patch \
+            file://use-relative-path-for-build-dir.patch \
+"
 SRC_URI[archive.md5sum] = "172b78256100e8d3c629764abd0e1495"
 SRC_URI[archive.sha256sum] = "be699d484371111abae754e669187215df73e21533f461e513b79537d7a1c1c1"
 
-EXTRA_OECONF += " --disable-nss --with-distributor-name=${DISTRO}"
+EXTRA_OECONF += " --disable-nss --with-distributor-name=${DISTRO} \
+                  --disable-tests \
+"
 
 do_configure_prepend() {
     touch ${S}/gnome-doc-utils.make
diff --git a/meta/recipes-gnome/epiphany/files/use-relative-path-for-build-dir.patch b/meta/recipes-gnome/epiphany/files/use-relative-path-for-build-dir.patch
new file mode 100644
index 0000000..21c0912
--- /dev/null
+++ b/meta/recipes-gnome/epiphany/files/use-relative-path-for-build-dir.patch
@@ -0,0 +1,57 @@
+From 73e46aeea35c7bda7833e635f1bb2dbd39e2b43e Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <hongxu.jia@windriver.com>
+Date: Wed, 2 Mar 2016 02:00:23 -0500
+Subject: [PATCH] ephy-profile-utils: use relative path for build dir
+
+It reverts the following commit in https://github.com/GNOME/epiphany.git
+...
+commit 54a3122d8626f78338033a704d5849738f6dd2e6
+Author: Xan Lopez <xan@igalia.com>
+Date:   Wed May 16 18:21:21 2012 +0900
+
+    ephy-profile-utils: use absolute path for build dir
+
+    That way we can run the tests from any directory in debug mode.
+...
+It did not consider cross-compile situation which we do not need
+to run tests in host build.
+
+It also caused build path issue.
+
+Upstream-Status: Inappropriate [openembedded specific]
+
+Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
+---
+ lib/Makefile.am          | 2 +-
+ lib/ephy-profile-utils.c | 2 +-
+ 2 files changed, 2 insertions(+), 2 deletions(-)
+
+diff --git a/lib/Makefile.am b/lib/Makefile.am
+index 24d684b..849a07f 100644
+--- a/lib/Makefile.am
++++ b/lib/Makefile.am
+@@ -88,7 +88,7 @@ libephymisc_la_CPPFLAGS = \
+ 	-DDATADIR="\"$(datadir)\""		    \
+ 	-DSHARE_DIR=\"$(pkgdatadir)\" 		    \
+ 	-DTOP_SRC_DATADIR=\"$(top_srcdir)/data\"    \
+-	-DABS_TOP_BUILD_DIR=\"$(abs_top_builddir)\" \
++	-DTOP_BUILD_DIR=\"$(top_builddir)\"         \
+ 	-DLOCALEDIR=\"$(localedir)\"		    \
+ 	$(AM_CPPFLAGS)
+ 
+diff --git a/lib/ephy-profile-utils.c b/lib/ephy-profile-utils.c
+index 0430582..a42ec4e 100644
+--- a/lib/ephy-profile-utils.c
++++ b/lib/ephy-profile-utils.c
+@@ -128,7 +128,7 @@ ephy_profile_utils_do_migration (const char *profile_directory, int test_to_run,
+   argv[i++] = NULL;
+ 
+   if (debug)
+-    argv[0] = ABS_TOP_BUILD_DIR"/lib/"EPHY_PROFILE_MIGRATOR;
++    argv[0] = TOP_BUILD_DIR"/lib/"EPHY_PROFILE_MIGRATOR;
+ 
+   ret = g_spawn_sync (NULL, argv, envp, G_SPAWN_SEARCH_PATH,
+                       NULL, NULL, NULL, NULL,
+-- 
+1.9.1
+
-- 
1.9.1

[PATCH 17/17] ifupdown: fix do_compile failed while GCCVERSION = "4.9%"

[Hongxu Jia]

In repo https://anonscm.debian.org/git/collab-maint/ifupdown.git
...
commit 4bfa7e5ae6175469d1049b6ace86c259814add16
Author: Guus Sliepen <guus@debian.org>
Date:   Wed Jun 3 12:43:59 2015 +0200

    Compile with -std=c99.

    This prepares for the use of C99 features in the source code. With
    -std=c99, some GNU extensions are disabled, and the standard library
    headers are more strict in what they define, so we need to compile with
    -D_DEFAULT_SOURCE.
...

We override CFLAGS in recipe, so we should add above to keep
sync with upstream.

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-core/ifupdown/ifupdown_0.8.2.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-core/ifupdown/ifupdown_0.8.2.bb b/meta/recipes-core/ifupdown/ifupdown_0.8.2.bb
index 732b295..70ff10d 100644
--- a/meta/recipes-core/ifupdown/ifupdown_0.8.2.bb
+++ b/meta/recipes-core/ifupdown/ifupdown_0.8.2.bb
@@ -22,7 +22,7 @@ inherit update-rc.d update-alternatives
 
 do_compile () {
 	chmod a+rx *.pl *.sh
-	oe_runmake 'CC=${CC}' "CFLAGS=${CFLAGS} -Wall -W -D'IFUPDOWN_VERSION=\"${PV}\"'"
+	oe_runmake 'CC=${CC}' "CFLAGS=${CFLAGS} -std=c99 -D_DEFAULT_SOURCE -Wall -W -D'IFUPDOWN_VERSION=\"${PV}\"'"
 }
 
 do_install () {
-- 
1.9.1

Re: [PATCH 03/17] conf/bitbake.conf package.bbclass: improve dbg package sources generation from work-shared

[Richard Purdie]

On Mon, 2016-03-28 at 04:18 -0400, Hongxu Jia wrote:
> Previously, the dbg package could not generate sources if they
> are in work-shared dir (toolchain, kernel).
> 
> The fix add a new fdebug-prefix-map to remap work-shared dir
> and collect sources to dbg package.
> 
> [YOCTO #9305]
> 
> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>

I wasn't really thrilled to be reading whole files into memory with the
original patch. To start doing this twice in succession for shared work
means we really need to rethink how this is being done :(.

Its possible software may mix from shared-work and from some local
kernel module build too, not sure if/how well the code copes with that
case.

Cheers,

Richard

Re: [PATCH 03/17] conf/bitbake.conf package.bbclass: improve dbg package sources generation from work-shared

[Hongxu Jia]

On 03/30/2016 06:38 AM, Richard Purdie wrote:
> On Mon, 2016-03-28 at 04:18 -0400, Hongxu Jia wrote:
>> Previously, the dbg package could not generate sources if they
>> are in work-shared dir (toolchain, kernel).
>>
>> The fix add a new fdebug-prefix-map to remap work-shared dir
>> and collect sources to dbg package.
>>
>> [YOCTO #9305]
>>
>> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> I wasn't really thrilled to be reading whole files into memory with the
> original patch. To start doing this twice in succession for shared work
> means we really need to rethink how this is being done :(.

Yes, I did not consider the work-shared situation in original patch neither,
thanks Mark to point it out, and I have to rework in this fix.

The idea of design is:

1. Try to collect sources from work-shared, and then from WORKDIR
     (The reason is the gcc case has sources in work-shared and WORKDIR)

2. While collecting:
     - Invoke debugedit to list source and header file names

     - Translate these file names to replace prefix_new with prefix_old
        (Revert -fdebug-prefix=<prefix_old>=<prefix_new>)

     - Follow prefix_old file names, and copy them to dest.

     - Dereference symlink in dest

3. Tweak prefix_new, prefix_old, and dest for work-shared/ WORKDIR 
situation,
     also for with/without -fdebug-prefix-map situation.

>
> Its possible software may mix from shared-work and from some local
> kernel module build too, not sure if/how well the code copes with that
> case.

Currently kernel module has no dbg package generated, but I will
try to open it, and please wait my report for this case.

//Hongxu

> Cheers,
>
> Richard

Re: [PATCH 03/17] conf/bitbake.conf package.bbclass: improve dbg package sources generation from work-shared

[Hongxu Jia]

On 03/30/2016 09:58 AM, Hongxu Jia wrote:
>
>>
>> Its possible software may mix from shared-work and from some local
>> kernel module build too, not sure if/how well the code copes with that
>> case.
>
> Currently kernel module has no dbg package generated, but I will
> try to open it, and please wait my report for this case.

Hi Richard,

It is hardly to generate dbg for kernel module, it explicitly ignore 
them and strip them.

In meta/classes/package.bbclass:
...
  377     # We ignore kernel modules, we don't generate debug info files.
  378     if file.find("/lib/modules/") != -1 and file.endswith(".ko"):
  379         return 1
...
  943     if (d.getVar('INHIBIT_PACKAGE_STRIP', True) != '1'):
  944         for root, dirs, files in cpath.walk(dvar):
  945             for f in files:
  946                 file = os.path.join(root, f)
  947                 if file.endswith(".ko") and 
file.find("/lib/modules/") != -1:
  948                     kernmods.append(file)
  949                     continue
...
(I tried, but lots of failures and make no sense)

I could not get a case that software gets sources from shared-work and
local kernel module build. But if local kernel module build does not in
work-shared or software's WORKDIR, this fix could not do the collection
from local kernel module.

This fix could collect sources from work-shared and recipes's WORKDIR
(such as gcc).

//Hongxu

Re: [PATCH V7 00/17] fix buildpaths QA issue && dbp package source generation

[Hongxu Jia]

On 03/28/2016 04:18 PM, Hongxu Jia wrote:
> are available in the git repository at:
>
>    git://git.openembedded.org/openembedded-core-contrib hongxu/fix-buildpath
>    http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=hongxu/fix-buildpath

Rebase to latest master to fix gcc-5.3 conflict.

//Hongxu

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Andreas Müller]

On Mon, Mar 28, 2016 at 10:18 AM, Hongxu Jia <hongxu.jia@windriver.com> wrote:
> Tweak DEBUG_FLAGS to use "/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR}"
> as source target path in DWARF. While use gdb to debug binary, it could
> work with sources in dbg package.
>
> While -fdebug-prefix-map is used for compiling, we do not need invoking
> debugedit to edit DWARF at do_package time, but list where sources files
> are.
>
> The copydebugsources uses the list to copy sources to dbg package. It
> works whether -fdebug-prefix-map used or not.
>
> [YOCTO #9305]
>
> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> ---
>  meta/classes/package.bbclass | 24 ++++++++++++++++++++----
>  meta/conf/bitbake.conf       |  3 +--
>  2 files changed, 21 insertions(+), 6 deletions(-)
>
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index 5a84255..bdbe96d 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -300,6 +300,15 @@ def get_conffiles(pkg, d):
>      os.chdir(cwd)
>      return conf_list
>
> +def checkbuildpath(file, d):
> +    tmpdir = d.getVar('TMPDIR', True)
> +    with open(file) as f:
> +        file_content = f.read()
> +        if tmpdir in file_content:
> +            return True
> +
> +    return False
> +
>  def splitdebuginfo(file, debugfile, debugsrcdir, sourcefile, d):
>      # Function to split a single file into two components, one is the stripped
>      # target system binary, the other contains any debugging information. The
> @@ -312,8 +321,6 @@ def splitdebuginfo(file, debugfile, debugsrcdir, sourcefile, d):
>      dvar = d.getVar('PKGD', True)
>      objcopy = d.getVar("OBJCOPY", True)
>      debugedit = d.expand("${STAGING_LIBDIR_NATIVE}/rpm/bin/debugedit")
> -    workdir = d.getVar("WORKDIR", True)
> -    workparentdir = d.getVar("DEBUGSRC_OVERRIDE_PATH", True) or os.path.dirname(os.path.dirname(workdir))
>
>      # We ignore kernel modules, we don't generate debug info files.
>      if file.find("/lib/modules/") != -1 and file.endswith(".ko"):
> @@ -327,7 +334,7 @@ def splitdebuginfo(file, debugfile, debugsrcdir, sourcefile, d):
>
>      # We need to extract the debug src information here...
>      if debugsrcdir:
> -        cmd = "'%s' -b '%s' -d '%s' -i -l '%s' '%s'" % (debugedit, workparentdir, debugsrcdir, sourcefile, file)
> +        cmd = "'%s' -i -l '%s' '%s'" % (debugedit, sourcefile, file)
>          (retval, output) = oe.utils.getstatusoutput(cmd)
>          if retval:
>              bb.fatal("debugedit failed with exit code %s (cmd was %s)%s" % (retval, cmd, ":\n%s" % output if output else ""))
> @@ -366,6 +373,13 @@ def copydebugsources(debugsrcdir, d):
>          workparentdir = os.path.dirname(os.path.dirname(workdir))
>          workbasedir = os.path.basename(os.path.dirname(workdir)) + "/" + os.path.basename(workdir)
>
> +        # If build path exists in sourcefile, it means toolchain did not use
> +        # -fdebug-prefix-map to compile
> +        if checkbuildpath(sourcefile, d):
> +            localsrc_prefix = workparentdir + "/"
> +        else:
> +            localsrc_prefix = "/usr/src/debug/"
> +
>          nosuchdir = []
>          basepath = dvar
>          for p in debugsrcdir.split("/"):
> @@ -379,9 +393,11 @@ def copydebugsources(debugsrcdir, d):
>          # We need to ignore files that are not actually ours
>          # we do this by only paying attention to items from this package
>          processdebugsrc += "fgrep -zw '%s' | "
> +        # Remove prefix in the source paths
> +        processdebugsrc += "sed 's#%s##g' | "
>          processdebugsrc += "(cd '%s' ; cpio -pd0mlL --no-preserve-owner '%s%s' 2>/dev/null)"
>
> -        cmd = processdebugsrc % (sourcefile, workbasedir, workparentdir, dvar, debugsrcdir)
> +        cmd = processdebugsrc % (sourcefile, workbasedir, localsrc_prefix, workparentdir, dvar, debugsrcdir)
>          (retval, output) = oe.utils.getstatusoutput(cmd)
>          # Can "fail" if internal headers/transient sources are attempted
>          #if retval:
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index e3db504..0eb288e 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -550,8 +550,7 @@ EXTRA_OEMAKE_prepend_task-install = "${PARALLEL_MAKEINST} "
>  # Optimization flags.
>  ##################################################################
>  DEBUG_FLAGS ?= "-g -feliminate-unused-debug-types \
> -                -fdebug-prefix-map=${B}=/usr/src/${BPN} \
> -                -fdebug-prefix-map=${S}=/usr/src/${BPN} \
> +                -fdebug-prefix-map=${WORKDIR}=/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR} \
>                  -fdebug-prefix-map=${STAGING_DIR_NATIVE}= \
>                  -fdebug-prefix-map=${STAGING_DIR_HOST}= \
>  "
> --
> 1.9.1
>
Has anybody tried remote debugging with this patch applied? For me the
source files are no more found and I think this patch looks
suspicious...

Andreas

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Hongxu Jia]

On 04/18/2016 04:09 PM, Andreas Müller wrote:
>> >
> Has anybody tried remote debugging with this patch applied? For me the
> source files are no more found and I think this patch looks
> suspicious...

What's your build steps?

//Hongxu

>
> Andreas

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Andreas Müller]

On Mon, Apr 18, 2016 at 10:11 AM, Hongxu Jia <hongxu.jia@windriver.com> wrote:
> On 04/18/2016 04:09 PM, Andreas Müller wrote:
>>>
>>> >
>>
>> Has anybody tried remote debugging with this patch applied? For me the
>> source files are no more found and I think this patch looks
>> suspicious...
>
>
> What's your build steps?
>
Don't understand your question I guess. I build by bitbake <image> with

INHIBIT_SYSROOT_STRIP = "1"

and set sysroot to build sysroot - see [1].

[1] http://lists.openembedded.org/pipermail/openembedded-core/2016-March/118635.html

Andreas

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Hongxu Jia]

On 04/18/2016 04:23 PM, Andreas Müller wrote:
> On Mon, Apr 18, 2016 at 10:11 AM, Hongxu Jia <hongxu.jia@windriver.com> wrote:
>> On 04/18/2016 04:09 PM, Andreas Müller wrote:
>>> Has anybody tried remote debugging with this patch applied? For me the
>>> source files are no more found and I think this patch looks
>>> suspicious...
>>
>> What's your build steps?
>>
> Don't understand your question I guess. I build by bitbake <image> with
>
> INHIBIT_SYSROOT_STRIP = "1"
>
> and set sysroot to build sysroot - see [1].
>
> [1] http://lists.openembedded.org/pipermail/openembedded-core/2016-March/118635.html

Do you install *-dbp pkgs to your image? In the following example, we 
use gdb to
debug gzip.

Test steps:

1. vim local.conf
------
IMAGE_INSTALL_append = " gdb gcc gzip"
EXTRA_IMAGE_FEATURES_append = " dbg-pkgs"
------

Currently, The ${S} locates in 'work-shared' dir could not generate 
sources (such as gcc),
and ${S} located in WORKDIR works (such as above gzip).

And now I start a new building to verify the suspicious you report.

//Hongxu

>
> Andreas

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Andreas Müller]

On Mon, Apr 18, 2016 at 10:32 AM, Hongxu Jia <hongxu.jia@windriver.com> wrote:
> On 04/18/2016 04:23 PM, Andreas Müller wrote:
>>
>> On Mon, Apr 18, 2016 at 10:11 AM, Hongxu Jia <hongxu.jia@windriver.com>
>> wrote:
>>>
>>> On 04/18/2016 04:09 PM, Andreas Müller wrote:
>>>>
>>>> Has anybody tried remote debugging with this patch applied? For me the
>>>> source files are no more found and I think this patch looks
>>>> suspicious...
>>>
>>>
>>> What's your build steps?
>>>
>> Don't understand your question I guess. I build by bitbake <image> with
>>
>> INHIBIT_SYSROOT_STRIP = "1"
>>
>> and set sysroot to build sysroot - see [1].
>>
>> [1]
>> http://lists.openembedded.org/pipermail/openembedded-core/2016-March/118635.html
>
>
> Do you install *-dbp pkgs to your image? In the following example, we use
> gdb to
> debug gzip.
Ehm dbg packages?? No - I just want remote debugging
>
> Test steps:
>
> 1. vim local.conf
> ------
> IMAGE_INSTALL_append = " gdb gcc gzip"
> EXTRA_IMAGE_FEATURES_append = " dbg-pkgs"
> ------
>
> Currently, The ${S} locates in 'work-shared' dir could not generate sources
> (such as gcc),
> and ${S} located in WORKDIR works (such as above gzip).
OK no problem - the application I wanted to debug is qt5-creator. This
has worked in the past.
>
> And now I start a new building to verify the suspicious you report.
>
> //Hongxu
>
>>
>> Andreas
>
I cannot check further at the moment: At work currently I have jethro
which debugs fine. The machine with which I detected the issue is at
home close to master.

Andreas

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Hongxu Jia]

On 04/18/2016 04:47 PM, Andreas Müller wrote:
>> debug gzip.
> Ehm dbg packages?? No - I just want remote debugging

Since -fdebug-prefix-map were added to DEBUG_FLAGS, the prefix in 
debugging info
will be remapped, I am afraid your remote debug with build path sysroot 
could not work.

I am afraid you need to tweak sysroot location to target path. Or 
directly remove -fdebug-prefix-map
from DEBUG_FLAGS.

//Hongxu

>> >

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Andreas Müller]

On Mon, Apr 18, 2016 at 11:04 AM, Hongxu Jia <hongxu.jia@windriver.com> wrote:
> On 04/18/2016 04:47 PM, Andreas Müller wrote:
>>>
>>> debug gzip.
>>
>> Ehm dbg packages?? No - I just want remote debugging
>
>
> Since -fdebug-prefix-map were added to DEBUG_FLAGS, the prefix in debugging
> info
> will be remapped, I am afraid your remote debug with build path sysroot
> could not work.
>
> I am afraid you need to tweak sysroot location to target path. Or directly
> remove -fdebug-prefix-map
> from DEBUG_FLAGS.
>
> //Hongxu
>
So remote debugging is no more available for standard settings. I am
tempted to send a revert: Installing -dbg causes bloated images and
debugging on machine is slow and last resort only.

Andreas

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Hongxu Jia]

On 04/18/2016 05:15 PM, Andreas Müller wrote:
> So remote debugging is no more available for standard settings. I am
> tempted to send a revert: Installing -dbg causes bloated images and
> debugging on machine is slow and last resort only.

I think we support remote debugging better since -fdebug-prefix-map used,
It is easy to construct sources paths.

Yours remote debug was limited to original build environment.

//Hongxu

> Andreas

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Andreas Müller]

On Mon, Apr 18, 2016 at 11:34 AM, Hongxu Jia <hongxu.jia@windriver.com> wrote:
> On 04/18/2016 05:15 PM, Andreas Müller wrote:
>>
>> So remote debugging is no more available for standard settings. I am
>> tempted to send a revert: Installing -dbg causes bloated images and
>> debugging on machine is slow and last resort only.
>
>
> I think we support remote debugging better since -fdebug-prefix-map used,
> It is easy to construct sources paths.
>
> Yours remote debug was limited to original build environment.
>
> //Hongxu
>
>> Andreas
>
OK I checked the documentation for this. You say setting sysroot is
broken now. How exactly is your suggested option for *remote*
debugging? And please note: Installing -dbg packages is not an option:

* It explodes image's size
* problems are caused by having only one -dbg per recipe: due to
dependencies conflicting packages are installed causing build to break
* it is terribly slow: All source code is transferred by network
during debugging
* a quick starting up of debugger is impossible - a new image has to be created.

Andreas

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Hongxu Jia]

On 04/18/2016 05:52 PM, Andreas Müller wrote:
> On Mon, Apr 18, 2016 at 11:34 AM, Hongxu Jia <hongxu.jia@windriver.com> wrote:
>> On 04/18/2016 05:15 PM, Andreas Müller wrote:
>>> So remote debugging is no more available for standard settings. I am
>>> tempted to send a revert: Installing -dbg causes bloated images and
>>> debugging on machine is slow and last resort only.
>>
>> I think we support remote debugging better since -fdebug-prefix-map used,
>> It is easy to construct sources paths.
>>
>> Yours remote debug was limited to original build environment.
>>
>> //Hongxu
>>
>>> Andreas
> OK I checked the documentation for this. You say setting sysroot is
> broken now. How exactly is your suggested option for *remote*
> debugging? And please note: Installing -dbg packages is not an option:
In your case for remote debug with build environment,

My suggestions is:

Plan1:

If sysroot is not required, just install extract *-dpb to /usr/src/ in host

Plang2:

If sysroot is required, override DEBUG_FLAGS to remove -fdebug-prefix-map

//Hongxu


> * It explodes image's size
> * problems are caused by having only one -dbg per recipe: due to
> dependencies conflicting packages are installed causing build to break
> * it is terribly slow: All source code is transferred by network
> during debugging
> * a quick starting up of debugger is impossible - a new image has to be created.
>
> Andreas

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 647 bytes --]

On 18 April 2016 at 14:08, Hongxu Jia <hongxu.jia@windriver.com> wrote:

> If sysroot is required, override DEBUG_FLAGS to remove -fdebug-prefix-map
>

We should probably get this in the release notes, as many people do indeed
want remote debug to work.

Can you tell gdb the base path to use when looking for symbols?  I've not
done remote GDB for some time but wouldn't "set substitute-path /
//my/sysroot/" work?  Or maybe /usr /my/sysroot/usr?

If gdb can't be told then instead of having to replace all of DEBUG_FLAGS
it would be neat if the prefix mapping variables where in another variable
that could just be unset.

Ross

[-- Attachment #2: Type: text/html, Size: 1084 bytes --]

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Andreas Müller]

On Mon, Apr 18, 2016 at 3:55 PM, Burton, Ross <ross.burton@intel.com> wrote:
>
> On 18 April 2016 at 14:08, Hongxu Jia <hongxu.jia@windriver.com> wrote:
>>
>> If sysroot is required, override DEBUG_FLAGS to remove -fdebug-prefix-map
>
>
> We should probably get this in the release notes, as many people do indeed
> want remote debug to work.
>
> Can you tell gdb the base path to use when looking for symbols?  I've not
> done remote GDB for some time but wouldn't "set substitute-path /
> //my/sysroot/" work?  Or maybe /usr /my/sysroot/usr?
>
> If gdb can't be told then instead of having to replace all of DEBUG_FLAGS it
> would be neat if the prefix mapping variables where in another variable that
> could just be unset.
>
> Ross
OK I think I could live with removing -fdebug-prefix-map for now.

A thought: We have the setting IMAGE_GEN_DEBUGFS - I have not yet
tested. As far as I understand it creates an unstripped sysroot and
does not affect target rootfs - is that correct? Does the sysroot
created by IMAGE_GEN_DEBUGFS contain the sources? If yes we could set
that as sysroot for gdb.

I think one of the problems we have here is that there is no reference
way documented (or it is outdated) how remote debugging is meant to be
performed.

Andreas

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Richard Purdie]

On Mon, 2016-04-18 at 16:16 +0200, Andreas Müller wrote:
> On Mon, Apr 18, 2016 at 3:55 PM, Burton, Ross <ross.burton@intel.com>
> wrote:
> > 
> > On 18 April 2016 at 14:08, Hongxu Jia <hongxu.jia@windriver.com>
> > wrote:
> > > 
> > > If sysroot is required, override DEBUG_FLAGS to remove -fdebug
> > > -prefix-map
> > 
> > 
> > We should probably get this in the release notes, as many people do
> > indeed
> > want remote debug to work.
> > 
> > Can you tell gdb the base path to use when looking for symbols? 
> >  I've not
> > done remote GDB for some time but wouldn't "set substitute-path /
> > //my/sysroot/" work?  Or maybe /usr /my/sysroot/usr?
> > 
> > If gdb can't be told then instead of having to replace all of
> > DEBUG_FLAGS it
> > would be neat if the prefix mapping variables where in another
> > variable that
> > could just be unset.
> > 
> > Ross
> OK I think I could live with removing -fdebug-prefix-map for now.
> 
> A thought: We have the setting IMAGE_GEN_DEBUGFS - I have not yet
> tested. As far as I understand it creates an unstripped sysroot and
> does not affect target rootfs - is that correct? Does the sysroot
> created by IMAGE_GEN_DEBUGFS contain the sources? If yes we could set
> that as sysroot for gdb.
> 
> I think one of the problems we have here is that there is no
> reference
> way documented (or it is outdated) how remote debugging is meant to
> be
> performed.

FWIW I do believe the prefix mapping variables are the right thing to
do, it makes sense that the paths in the -dbg files should be /usr/src/
rather than some build time path. We've aimed to do that for a long
time, first using debugedit and more recently directly when compiling
rather than editing later.

We do need to figure out how to explain that to gdb, or if that can't
be made to work, even consider patching gdb and upstreaming the patch
such that it can look at a sysroot and work correctly.

I think it comes down to more people getting involved as we're
struggling to do everything with the current people and as I've said in
other threads, its also aboutbetter test cases so that when something
regresses that people care about we do get to know sooner.

Cheers,

Richard

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Mark Hatle]

On 4/18/16 9:16 AM, Andreas Müller wrote:
> On Mon, Apr 18, 2016 at 3:55 PM, Burton, Ross <ross.burton@intel.com> wrote:
>>
>> On 18 April 2016 at 14:08, Hongxu Jia <hongxu.jia@windriver.com> wrote:
>>>
>>> If sysroot is required, override DEBUG_FLAGS to remove -fdebug-prefix-map
>>
>>
>> We should probably get this in the release notes, as many people do indeed
>> want remote debug to work.
>>
>> Can you tell gdb the base path to use when looking for symbols?  I've not
>> done remote GDB for some time but wouldn't "set substitute-path /
>> //my/sysroot/" work?  Or maybe /usr /my/sysroot/usr?
>>
>> If gdb can't be told then instead of having to replace all of DEBUG_FLAGS it
>> would be neat if the prefix mapping variables where in another variable that
>> could just be unset.
>>
>> Ross
> OK I think I could live with removing -fdebug-prefix-map for now.
> 
> A thought: We have the setting IMAGE_GEN_DEBUGFS - I have not yet
> tested. As far as I understand it creates an unstripped sysroot and
> does not affect target rootfs - is that correct? Does the sysroot
> created by IMAGE_GEN_DEBUGFS contain the sources? If yes we could set
> that as sysroot for gdb.
> 
> I think one of the problems we have here is that there is no reference
> way documented (or it is outdated) how remote debugging is meant to be
> performed.

Remote debugging is setup to always assume the remote side has access to a
debugable sysroot.  This isn't the project's sysroot, since that is transient,
but a REAL matching sysroot to the main image.

The easiest way to do this is to add the following to your local.conf:
   IMAGE_GEN_DEBUGFS = '1'

After this, look in your deploy directory and you will see the image to deploy
to the target, and also a matching 'debug' chunk that you keep on your local
host.  This way you -know- they are always in sync.

There is class material from the various Yocto Project Dev Day classes on using
this.


The alternative is tell GDB itself to map directories.  I don't remember offhand
the right option, but GDB supports remapping a directory reference to an actual
directory.  It should be pretty simple to re-map it -- but due to the transient
nature of the sysroot directory in the build, I don't recommend doing it this way.

--Mark

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Andreas Müller]

On Mon, Apr 18, 2016 at 8:50 PM, Mark Hatle <mark.hatle@windriver.com> wrote:
> On 4/18/16 9:16 AM, Andreas Müller wrote:
>> On Mon, Apr 18, 2016 at 3:55 PM, Burton, Ross <ross.burton@intel.com> wrote:
>>>
>>> On 18 April 2016 at 14:08, Hongxu Jia <hongxu.jia@windriver.com> wrote:
>>>>
>>>> If sysroot is required, override DEBUG_FLAGS to remove -fdebug-prefix-map
>>>
>>>
>>> We should probably get this in the release notes, as many people do indeed
>>> want remote debug to work.
>>>
>>> Can you tell gdb the base path to use when looking for symbols?  I've not
>>> done remote GDB for some time but wouldn't "set substitute-path /
>>> //my/sysroot/" work?  Or maybe /usr /my/sysroot/usr?
>>>
>>> If gdb can't be told then instead of having to replace all of DEBUG_FLAGS it
>>> would be neat if the prefix mapping variables where in another variable that
>>> could just be unset.
>>>
>>> Ross
>> OK I think I could live with removing -fdebug-prefix-map for now.
>>
>> A thought: We have the setting IMAGE_GEN_DEBUGFS - I have not yet
>> tested. As far as I understand it creates an unstripped sysroot and
>> does not affect target rootfs - is that correct? Does the sysroot
>> created by IMAGE_GEN_DEBUGFS contain the sources? If yes we could set
>> that as sysroot for gdb.
>>
>> I think one of the problems we have here is that there is no reference
>> way documented (or it is outdated) how remote debugging is meant to be
>> performed.
>
> Remote debugging is setup to always assume the remote side has access to a
> debugable sysroot.
Just that I get this right: remote = target? If yes: Where do you get
this from? Or - have you ever run a remote session with files taken
from target? I have: see caffeine/colleagues section below.
> This isn't the project's sysroot, since that is transient,
> but a REAL matching sysroot to the main image.
If you say transient: Do you mean status of last build. Isn't that
what I am usually looking for when debugging?
>
> The easiest way to do this is to add the following to your local.conf:
>    IMAGE_GEN_DEBUGFS = '1'
This exactly what I've asked for in my previous mail - further comments below.
>
> After this, look in your deploy directory and you will see the image to deploy
> to the target, and also a matching 'debug' chunk that you keep on your local
> host.  This way you -know- they are always in sync.
You really want me to wait for building an full image after each
modification - I will either die for caffeine overdose or my
colleagues will kill me...
>
> There is class material from the various Yocto Project Dev Day classes on using
> this.
>
>
> The alternative is tell GDB itself to map directories.  I don't remember offhand
> the right option, but GDB supports remapping a directory reference to an actual
> directory.  It should be pretty simple to re-map it -- but due to the transient
> nature of the sysroot directory in the build, I don't recommend doing it this way.
>
We are using remote debugging with build sysroot almost daily and we
do that since we use openembedded. The more I learn here it seems
nobody else does remote debugging. How do you fix bugs?

I'll remove -fdebug-prefix-map and wait till remote debugging breaks
again - maybe then completely. That scares me as I am loosing the
possiblity to start a remote debug session on an productive image
within seconds.

Andreas

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Mark Hatle]

On 4/18/16 4:05 PM, Andreas Müller wrote:
> On Mon, Apr 18, 2016 at 8:50 PM, Mark Hatle <mark.hatle@windriver.com> wrote:
>> On 4/18/16 9:16 AM, Andreas Müller wrote:
>>> On Mon, Apr 18, 2016 at 3:55 PM, Burton, Ross <ross.burton@intel.com> wrote:
>>>>
>>>> On 18 April 2016 at 14:08, Hongxu Jia <hongxu.jia@windriver.com> wrote:
>>>>>
>>>>> If sysroot is required, override DEBUG_FLAGS to remove -fdebug-prefix-map
>>>>
>>>>
>>>> We should probably get this in the release notes, as many people do indeed
>>>> want remote debug to work.
>>>>
>>>> Can you tell gdb the base path to use when looking for symbols?  I've not
>>>> done remote GDB for some time but wouldn't "set substitute-path /
>>>> //my/sysroot/" work?  Or maybe /usr /my/sysroot/usr?
>>>>
>>>> If gdb can't be told then instead of having to replace all of DEBUG_FLAGS it
>>>> would be neat if the prefix mapping variables where in another variable that
>>>> could just be unset.
>>>>
>>>> Ross
>>> OK I think I could live with removing -fdebug-prefix-map for now.
>>>
>>> A thought: We have the setting IMAGE_GEN_DEBUGFS - I have not yet
>>> tested. As far as I understand it creates an unstripped sysroot and
>>> does not affect target rootfs - is that correct? Does the sysroot
>>> created by IMAGE_GEN_DEBUGFS contain the sources? If yes we could set
>>> that as sysroot for gdb.
>>>
>>> I think one of the problems we have here is that there is no reference
>>> way documented (or it is outdated) how remote debugging is meant to be
>>> performed.
>>
>> Remote debugging is setup to always assume the remote side has access to a
>> debugable sysroot.

Remote (target from host) is setup to always assume the -host- side has access
to a debuggable sysroot.  The target -never- needs access, unless you are trying
to run GDB on the target.

> Just that I get this right: remote = target? If yes: Where do you get
> this from? Or - have you ever run a remote session with files taken
> from target? I have: see caffeine/colleagues section below.
>> This isn't the project's sysroot, since that is transient,
>> but a REAL matching sysroot to the main image.
> If you say transient: Do you mean status of last build. Isn't that
> what I am usually looking for when debugging?

It's transient.  There is no direct correlation between the contents of the
image you are booting and the sysroot directory int he project/tmp/sysroot area.
 The files MIGHT be related, but they are not the same.  The build project is
only setup for actually building software.

To debug, you must you the -dbg packages that are created as part of the build.
 Using the IMAGE_GEN_DEBUGFS automates this process.  Or you can individually
grab (and unpack) the -dbg for the components you wish to debug.

>> The easiest way to do this is to add the following to your local.conf:
>>    IMAGE_GEN_DEBUGFS = '1'
> This exactly what I've asked for in my previous mail - further comments below.
>>
>> After this, look in your deploy directory and you will see the image to deploy
>> to the target, and also a matching 'debug' chunk that you keep on your local
>> host.  This way you -know- they are always in sync.
> You really want me to wait for building an full image after each
> modification - I will either die for caffeine overdose or my
> colleagues will kill me...

How big is your filesystem?  Typically when I make a small change, it only takes
me minutes to build a filesystem.. (I'm roughly similar to the core-image-base.)
 I'm not changing key things like the toolchain or glibc, I'm changing
individual applications with few downstream dependencies.

You can always build the component by itself and simply pull the -dbg components
out if you want.  That is certainly a short cut method to this.

The non-dbg have the correct versions for deployment for runtime, while the -dbg
has the corresponding debug information.  The sysroot though is not the right
place for this, if you want to work around the system use the actual work
location for the components themselves and copy the pieces from there.

>>
>> There is class material from the various Yocto Project Dev Day classes on using
>> this.
>>
>>
>> The alternative is tell GDB itself to map directories.  I don't remember offhand
>> the right option, but GDB supports remapping a directory reference to an actual
>> directory.  It should be pretty simple to re-map it -- but due to the transient
>> nature of the sysroot directory in the build, I don't recommend doing it this way.
>>
> We are using remote debugging with build sysroot almost daily and we
> do that since we use openembedded. The more I learn here it seems
> nobody else does remote debugging. How do you fix bugs?

I do remote debugging all the time, note -- for much of my work I'm still on
2.0.  But the design of 2.1 (master) actually makes this easier then before, as
all of the file references are finally correct -- instead of a mismatched set of
target and host dependencies.

I -always- use the -dbg versions when doing local (on target) or remote (host to
target) debugging.  It takes no additional time, and it's easy to shortcut as I
mentioned above.

The sysroot is only appropriate for compiling against.

> I'll remove -fdebug-prefix-map and wait till remote debugging breaks
> again - maybe then completely. That scares me as I am loosing the
> possiblity to start a remote debug session on an productive image
> within seconds.

Without the prefix maps, you won't get functional -dbg packages for remote
debugging of deployed production systems.

This is really the key here, we facilitate both development -and- production
remote debug.  I don't know of any other (embedded) build system that is
designed with production remote debug built in.  You never have to deploy
symbols to your production images, but yet you can still cross debug them by
simply keeping a copy of the -dbg (and as of recently the corresponding debugfs
image.)  This ensures the code is not only debugable today, but in the future
when your product has been deployed for a long time.

--Mark

> Andreas
> 

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Hongxu Jia]

[-- Attachment #1: Type: text/plain, Size: 852 bytes --]

On 04/18/2016 09:55 PM, Burton, Ross wrote:
>
> On 18 April 2016 at 14:08, Hongxu Jia <hongxu.jia@windriver.com 
> <mailto:hongxu.jia@windriver.com>> wrote:
>
>     If sysroot is required, override DEBUG_FLAGS to remove
>     -fdebug-prefix-map
>
>
> We should probably get this in the release notes, as many people do 
> indeed want remote debug to work.
>
> Can you tell gdb the base path to use when looking for symbols?  I've 
> not done remote GDB for some time but wouldn't "set substitute-path / 
> //my/sysroot/" work?  Or maybe /usr /my/sysroot/usr?
>
> If gdb can't be told then instead of having to replace all of 
> DEBUG_FLAGS it would be neat if the prefix mapping variables where in 
> another variable that could just be unset.
>

OK, I will try to work on GDB to let remote debug to work.

//Hongxu

> Ross


[-- Attachment #2: Type: text/html, Size: 2431 bytes --]

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Paul Eggleton]

On Mon, 18 Apr 2016 13:50:15 Mark Hatle wrote:
> On 4/18/16 9:16 AM, Andreas Müller wrote:
> > On Mon, Apr 18, 2016 at 3:55 PM, Burton, Ross <ross.burton@intel.com> 
wrote:
> >> On 18 April 2016 at 14:08, Hongxu Jia <hongxu.jia@windriver.com> wrote:
> >>> If sysroot is required, override DEBUG_FLAGS to remove
> >>> -fdebug-prefix-map
> >> 
> >> We should probably get this in the release notes, as many people do
> >> indeed
> >> want remote debug to work.
> >> 
> >> Can you tell gdb the base path to use when looking for symbols?  I've not
> >> done remote GDB for some time but wouldn't "set substitute-path /
> >> //my/sysroot/" work?  Or maybe /usr /my/sysroot/usr?
> >> 
> >> If gdb can't be told then instead of having to replace all of DEBUG_FLAGS
> >> it would be neat if the prefix mapping variables where in another
> >> variable that could just be unset.
> >> 
> >> Ross
> > 
> > OK I think I could live with removing -fdebug-prefix-map for now.
> > 
> > A thought: We have the setting IMAGE_GEN_DEBUGFS - I have not yet
> > tested. As far as I understand it creates an unstripped sysroot and
> > does not affect target rootfs - is that correct? Does the sysroot
> > created by IMAGE_GEN_DEBUGFS contain the sources? If yes we could set
> > that as sysroot for gdb.
> > 
> > I think one of the problems we have here is that there is no reference
> > way documented (or it is outdated) how remote debugging is meant to be
> > performed.
> 
> Remote debugging is setup to always assume the remote side has access to a
> debugable sysroot.  This isn't the project's sysroot, since that is
> transient, but a REAL matching sysroot to the main image.
> 
> The easiest way to do this is to add the following to your local.conf:
>    IMAGE_GEN_DEBUGFS = '1'
> 
> After this, look in your deploy directory and you will see the image to
> deploy to the target, and also a matching 'debug' chunk that you keep on
> your local host.  This way you -know- they are always in sync.
> 
> There is class material from the various Yocto Project Dev Day classes on
> using this.

I thought I'd seen this go into the manual but I can't seem to find it; I also 
can't find said Dev Day class material online. Mark, would you be able to work 
with Scott R to get this documented both within the variable reference and in 
a task-oriented manner?

In the mean time, Jefro - is the recent dev day material available somewhere? 
I can only find 2014 (ELCE Dusseldorf) under "Training" on the YP website.

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Hongxu Jia]

[-- Attachment #1: Type: text/plain, Size: 3644 bytes --]

Hi all,

With some investigation, there is an example to explain how to do remote 
debug.

Also file [YOCTO #9481] for document.

Hi Andreas,

Would you please have a look the example, to see if it could fit your case.

Example:

1. Build a image with gdbserver and gzip installed. The gzip
    is the debug binary. Append the following to local.conf
...
IMAGE_INSTALL_append = " gdbserver gzip"
...

$ bitbake core-image-sato

2. Build gdb-cross for remote debug on host

$ bitbake gdb-cross-i586

3. start qemu

$ runqemu qemux86 core-image-sato

4. On host side, create symlink /usr/src/debug to
    ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS} which the
    location of sources is /usr/src/debug.

$ ln -snf /TOPDIR/tmp/work/i586-poky-linux /usr/src/debug

5. On target side, ipaddr is 128.224.163.187,
    run gdbserver to debug "gzip -h"

root@qemux86-64:~# gdbserver localhost:1024 gzip -h
Process gzip created; pid = 530
Listening on port 1024

6. On host side, enter gzip's devshell
$ bitbake -cdevshell gzip

7. On host side, run gdb-cross to remote debug gzip

gzip-1.6# i586-poky-linux-gdb ../build/gzip

8. Use path ${STAGING_DIR_HOST} as the system root for
    the program being debugged. Any absolute shared
    library paths will be prefixed with path;

(gdb) set sysroot /TOPDIR/tmp/sysroots/qemux86


9. Start remote target, set break point at help()

(gdb) target remote 128.224.163.187:1024
(gdb) break gzip.c:325
(gdb) info break
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x0000000000401ec1 in main at 
/usr/src/debug/gzip/1.6-r0/gzip-1.6/gzip.c:325
(gdb) c
Continuing.

Breakpoint 1, main (argc=2, argv=0x7fffffffeca8) at 
/usr/src/debug/gzip/1.6-r0/gzip-1.6/gzip.c:465
465                 help(); do_exit(OK); break;
(gdb) list
460             case 'd':
461                 decompress = 1; break;
462             case 'f':
463                 force++; break;
464             case 'h': case 'H':
465                 help(); do_exit(OK); break;
466             case 'k':
467                 keep = 1; break;
468             case 'l':
469                 list = decompress = to_stdout = 1; break;
(gdb) c
Continuing.
[Inferior 1 (process 1246) exited normally]


10. On target side, get output of "gzip -h"
[snip]
Remote debugging from host 128.224.153.74
Usage: gzip [OPTION]... [FILE]...
Compress or uncompress FILEs (by default, compress FILES in-place).

Mandatory arguments to long options are mandatory for short options too.

   -c, --stdout      write on standard output, keep original files unchanged
   -d, --decompress  decompress
   -f, --force       force overwrite of output file and compress links
   -h, --help        give this help
   -k, --keep        keep (don't delete) input files
   -l, --list        list compressed file contents
   -L, --license     display software license
   -n, --no-name     do not save or restore the original name and time stamp
   -N, --name        save or restore the original name and time stamp
   -q, --quiet       suppress all warnings
   -r, --recursive   operate recursively on directories
   -S, --suffix=SUF  use suffix SUF on compressed files
   -t, --test        test compressed file integrity
   -v, --verbose     verbose mode
   -V, --version     display version number
   -1, --fast        compress faster
   -9, --best        compress better

With no FILE, or when FILE is -, read standard input.

Report bugs to <bug-gzip@gnu.org>.

Child exited with status 0
GDBserver exiting
[snip]

//Hongxu


[-- Attachment #2: Type: text/html, Size: 5227 bytes --]

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Mark Hatle]

On 4/18/16 11:32 PM, Paul Eggleton wrote:
> On Mon, 18 Apr 2016 13:50:15 Mark Hatle wrote:
>> On 4/18/16 9:16 AM, Andreas Müller wrote:
>>> On Mon, Apr 18, 2016 at 3:55 PM, Burton, Ross <ross.burton@intel.com> 
> wrote:
>>>> On 18 April 2016 at 14:08, Hongxu Jia <hongxu.jia@windriver.com> wrote:
>>>>> If sysroot is required, override DEBUG_FLAGS to remove
>>>>> -fdebug-prefix-map
>>>>
>>>> We should probably get this in the release notes, as many people do
>>>> indeed
>>>> want remote debug to work.
>>>>
>>>> Can you tell gdb the base path to use when looking for symbols?  I've not
>>>> done remote GDB for some time but wouldn't "set substitute-path /
>>>> //my/sysroot/" work?  Or maybe /usr /my/sysroot/usr?
>>>>
>>>> If gdb can't be told then instead of having to replace all of DEBUG_FLAGS
>>>> it would be neat if the prefix mapping variables where in another
>>>> variable that could just be unset.
>>>>
>>>> Ross
>>>
>>> OK I think I could live with removing -fdebug-prefix-map for now.
>>>
>>> A thought: We have the setting IMAGE_GEN_DEBUGFS - I have not yet
>>> tested. As far as I understand it creates an unstripped sysroot and
>>> does not affect target rootfs - is that correct? Does the sysroot
>>> created by IMAGE_GEN_DEBUGFS contain the sources? If yes we could set
>>> that as sysroot for gdb.
>>>
>>> I think one of the problems we have here is that there is no reference
>>> way documented (or it is outdated) how remote debugging is meant to be
>>> performed.
>>
>> Remote debugging is setup to always assume the remote side has access to a
>> debugable sysroot.  This isn't the project's sysroot, since that is
>> transient, but a REAL matching sysroot to the main image.
>>
>> The easiest way to do this is to add the following to your local.conf:
>>    IMAGE_GEN_DEBUGFS = '1'
>>
>> After this, look in your deploy directory and you will see the image to
>> deploy to the target, and also a matching 'debug' chunk that you keep on
>> your local host.  This way you -know- they are always in sync.
>>
>> There is class material from the various Yocto Project Dev Day classes on
>> using this.
> 
> I thought I'd seen this go into the manual but I can't seem to find it; I also 
> can't find said Dev Day class material online. Mark, would you be able to work 
> with Scott R to get this documented both within the variable reference and in 
> a task-oriented manner?
> 
> In the mean time, Jefro - is the recent dev day material available somewhere? 
> I can only find 2014 (ELCE Dusseldorf) under "Training" on the YP website.

(I thought these steps were documented as well, and also were part of the
beginner material.  But I've also not been able to find it -- as I've only got
the advanced class materials for Ireland and San Diego.)


The steps are in the Dusseldorf slides:

http://www.yoctoproject.org/bulk/devday-eu-2014/ypdd14-reyna-iot-hands-on-lab.pdf

Slide 28 shows the old way of producing the debugfs.  The new way is to use the
IMAGE_GEN_DEBUGFS = "1".  You then extract it (and the original rootfs) into a
single directory.  In the slides that directory is called "debugfs".


Start on slide 44.

The setup for GDB (on the host side):

(gdb) set sysroot debugfs
(gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
(gdb) file debugfs/usr/bin/morseapp

This assumes your CWD is the place with the exracted debugfs (produced by the
IMAGE_GEN_DEBUGFS = "1").

The substitute-path is what allows the (host) to find the matching symbols for
the items.  You simply set:

set substitute-path <piece of target path> <host equivalent>

By using the debugfs, you don't have to set this to 100 different locations, you
can simply use a single location.

--Mark

> Thanks,
> Paul
> 

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Mark Hatle]

On 4/19/16 2:33 AM, Hongxu Jia wrote:
> Hi all,
> 
> With some investigation, there is an example to explain how to do remote debug.
> 
> Also file [YOCTO #9481] for document.
> 
> Hi Andreas,
> 
> Would you please have a look the example, to see if it could fit your case.
> 
> Example:
> 
> 1. Build a image with gdbserver and gzip installed. The gzip
>    is the debug binary. Append the following to local.conf
> ...
> IMAGE_INSTALL_append = " gdbserver gzip"
> ...
> 
> $ bitbake core-image-sato
> 
> 2. Build gdb-cross for remote debug on host
> 
> $ bitbake gdb-cross-i586
> 
> 3. start qemu
> 
> $ runqemu qemux86 core-image-sato
> 
> 4. On host side, create symlink /usr/src/debug to
>    ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS} which the
>    location of sources is /usr/src/debug.
> 
> $ ln -snf /TOPDIR/tmp/work/i586-poky-linux /usr/src/debug

This is incorrect.  On the host you use the combination of:

(gdb) set sysroot <your debug capable fs path>
(gdb) set substitute-path /usr/src/debug <your debug capable fs path>/usr/src/debug

> 5. On target side, ipaddr is 128.224.163.187,
>    run gdbserver to debug "gzip -h"
> 
> root@qemux86-64:~# gdbserver localhost:1024 gzip -h
> Process gzip created; pid = 530
> Listening on port 1024
> 
> 6. On host side, enter gzip's devshell
> $ bitbake -cdevshell gzip
> 
> 7. On host side, run gdb-cross to remote debug gzip
> 
> gzip-1.6# i586-poky-linux-gdb ../build/gzip
> 
> 8. Use path ${STAGING_DIR_HOST} as the system root for
>    the program being debugged. Any absolute shared
>    library paths will be prefixed with path;
> 
> (gdb) set sysroot /TOPDIR/tmp/sysroots/qemux86
> 
> 
> 9. Start remote target, set break point at help()
> 
> (gdb) target remote 128.224.163.187:1024
> (gdb) break gzip.c:325
> (gdb) info break
> Num     Type           Disp Enb Address            What
> 1       breakpoint     keep y   0x0000000000401ec1 in main at
> /usr/src/debug/gzip/1.6-r0/gzip-1.6/gzip.c:325
> (gdb) c
> Continuing.
> 
> Breakpoint 1, main (argc=2, argv=0x7fffffffeca8) at
> /usr/src/debug/gzip/1.6-r0/gzip-1.6/gzip.c:465
> 465                 help(); do_exit(OK); break;
> (gdb) list
> 460             case 'd':
> 461                 decompress = 1; break;
> 462             case 'f':
> 463                 force++; break;
> 464             case 'h': case 'H':
> 465                 help(); do_exit(OK); break;
> 466             case 'k':
> 467                 keep = 1; break;
> 468             case 'l':
> 469                 list = decompress = to_stdout = 1; break;
> (gdb) c
> Continuing.
> [Inferior 1 (process 1246) exited normally]
> 
> 
> 10. On target side, get output of "gzip -h"
> [snip]
> Remote debugging from host 128.224.153.74
> Usage: gzip [OPTION]... [FILE]...
> Compress or uncompress FILEs (by default, compress FILES in-place).
> 
> Mandatory arguments to long options are mandatory for short options too.
> 
>   -c, --stdout      write on standard output, keep original files unchanged
>   -d, --decompress  decompress
>   -f, --force       force overwrite of output file and compress links
>   -h, --help        give this help
>   -k, --keep        keep (don't delete) input files
>   -l, --list        list compressed file contents
>   -L, --license     display software license
>   -n, --no-name     do not save or restore the original name and time stamp
>   -N, --name        save or restore the original name and time stamp
>   -q, --quiet       suppress all warnings
>   -r, --recursive   operate recursively on directories
>   -S, --suffix=SUF  use suffix SUF on compressed files
>   -t, --test        test compressed file integrity
>   -v, --verbose     verbose mode
>   -V, --version     display version number
>   -1, --fast        compress faster
>   -9, --best        compress better
> 
> With no FILE, or when FILE is -, read standard input.
> 
> Report bugs to <bug-gzip@gnu.org>.
> 
> Child exited with status 0
> GDBserver exiting
> [snip]
> 
> //Hongxu
> 

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Hongxu Jia]

On 04/19/2016 08:31 PM, Mark Hatle wrote:
> On 4/19/16 2:33 AM, Hongxu Jia wrote:
>> Hi all,
>>
>> With some investigation, there is an example to explain how to do remote debug.
>>
>> Also file [YOCTO #9481] for document.
>>
>> Hi Andreas,
>>
>> Would you please have a look the example, to see if it could fit your case.
>>
>> Example:
>>
>> 1. Build a image with gdbserver and gzip installed. The gzip
>>     is the debug binary. Append the following to local.conf
>> ...
>> IMAGE_INSTALL_append = " gdbserver gzip"
>> ...
>>
>> $ bitbake core-image-sato
>>
>> 2. Build gdb-cross for remote debug on host
>>
>> $ bitbake gdb-cross-i586
>>
>> 3. start qemu
>>
>> $ runqemu qemux86 core-image-sato
>>
>> 4. On host side, create symlink /usr/src/debug to
>>     ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS} which the
>>     location of sources is /usr/src/debug.
>>
>> $ ln -snf /TOPDIR/tmp/work/i586-poky-linux /usr/src/debug
> This is incorrect.  On the host you use the combination of:
>
> (gdb) set sysroot <your debug capable fs path>
> (gdb) set substitute-path /usr/src/debug <your debug capable fs path>/usr/src/debug
>

Yes! Yours is more reasonable. It is verified on my case:

4. On host side, Set a substitution rule for finding source files.
    Replace /usr/src/debug with ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}

(gdb) set substitute-path /usr/src/debug /TOPDIR/tmp/work/i586-poky-linux

//Hongxu

>> 5. On target side, ipaddr is 128.224.163.187,
>>     run gdbserver to debug "gzip -h"
>>
>> root@qemux86-64:~# gdbserver localhost:1024 gzip -h
>> Process gzip created; pid = 530
>> Listening on port 1024
>>
>> 6. On host side, enter gzip's devshell
>> $ bitbake -cdevshell gzip
>>
>> 7. On host side, run gdb-cross to remote debug gzip
>>
>> gzip-1.6# i586-poky-linux-gdb ../build/gzip
>>
>> 8. Use path ${STAGING_DIR_HOST} as the system root for
>>     the program being debugged. Any absolute shared
>>     library paths will be prefixed with path;
>>
>> (gdb) set sysroot /TOPDIR/tmp/sysroots/qemux86
>>
>>
>> 9. Start remote target, set break point at help()
>>
>> (gdb) target remote 128.224.163.187:1024
>> (gdb) break gzip.c:325
>> (gdb) info break
>> Num     Type           Disp Enb Address            What
>> 1       breakpoint     keep y   0x0000000000401ec1 in main at
>> /usr/src/debug/gzip/1.6-r0/gzip-1.6/gzip.c:325
>> (gdb) c
>> Continuing.
>>
>> Breakpoint 1, main (argc=2, argv=0x7fffffffeca8) at
>> /usr/src/debug/gzip/1.6-r0/gzip-1.6/gzip.c:465
>> 465                 help(); do_exit(OK); break;
>> (gdb) list
>> 460             case 'd':
>> 461                 decompress = 1; break;
>> 462             case 'f':
>> 463                 force++; break;
>> 464             case 'h': case 'H':
>> 465                 help(); do_exit(OK); break;
>> 466             case 'k':
>> 467                 keep = 1; break;
>> 468             case 'l':
>> 469                 list = decompress = to_stdout = 1; break;
>> (gdb) c
>> Continuing.
>> [Inferior 1 (process 1246) exited normally]
>>
>>
>> 10. On target side, get output of "gzip -h"
>> [snip]
>> Remote debugging from host 128.224.153.74
>> Usage: gzip [OPTION]... [FILE]...
>> Compress or uncompress FILEs (by default, compress FILES in-place).
>>
>> Mandatory arguments to long options are mandatory for short options too.
>>
>>    -c, --stdout      write on standard output, keep original files unchanged
>>    -d, --decompress  decompress
>>    -f, --force       force overwrite of output file and compress links
>>    -h, --help        give this help
>>    -k, --keep        keep (don't delete) input files
>>    -l, --list        list compressed file contents
>>    -L, --license     display software license
>>    -n, --no-name     do not save or restore the original name and time stamp
>>    -N, --name        save or restore the original name and time stamp
>>    -q, --quiet       suppress all warnings
>>    -r, --recursive   operate recursively on directories
>>    -S, --suffix=SUF  use suffix SUF on compressed files
>>    -t, --test        test compressed file integrity
>>    -v, --verbose     verbose mode
>>    -V, --version     display version number
>>    -1, --fast        compress faster
>>    -9, --best        compress better
>>
>> With no FILE, or when FILE is -, read standard input.
>>
>> Report bugs to <bug-gzip@gnu.org>.
>>
>> Child exited with status 0
>> GDBserver exiting
>> [snip]
>>
>> //Hongxu
>>

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Andreas Müller]

On Tue, Apr 19, 2016 at 9:33 AM, Hongxu Jia <hongxu.jia@windriver.com> wrote:
> Hi all,
>
> With some investigation, there is an example to explain how to do remote
> debug.
>
> Also file [YOCTO #9481] for document.
>
> Hi Andreas,
>
> Would you please have a look the example, to see if it could fit your case.
>
> Example:
>
> 1. Build a image with gdbserver and gzip installed. The gzip
>    is the debug binary. Append the following to local.conf
> ...
> IMAGE_INSTALL_append = " gdbserver gzip"
> ...
>
> $ bitbake core-image-sato
>
> 2. Build gdb-cross for remote debug on host
>
> $ bitbake gdb-cross-i586
>
> 3. start qemu
>
> $ runqemu qemux86 core-image-sato
>
> 4. On host side, create symlink /usr/src/debug to
>    ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS} which the
>    location of sources is /usr/src/debug.
>
> $ ln -snf /TOPDIR/tmp/work/i586-poky-linux /usr/src/debug
>
> 5. On target side, ipaddr is 128.224.163.187,
>    run gdbserver to debug "gzip -h"
>
> root@qemux86-64:~# gdbserver localhost:1024 gzip -h
> Process gzip created; pid = 530
> Listening on port 1024
>
> 6. On host side, enter gzip's devshell
> $ bitbake -cdevshell gzip
>
> 7. On host side, run gdb-cross to remote debug gzip
>
> gzip-1.6# i586-poky-linux-gdb ../build/gzip
>
> 8. Use path ${STAGING_DIR_HOST} as the system root for
>    the program being debugged. Any absolute shared
>    library paths will be prefixed with path;
>
> (gdb) set sysroot /TOPDIR/tmp/sysroots/qemux86
>
>
> 9. Start remote target, set break point at help()
>
> (gdb) target remote 128.224.163.187:1024
> (gdb) break gzip.c:325
> (gdb) info break
> Num     Type           Disp Enb Address            What
> 1       breakpoint     keep y   0x0000000000401ec1 in main at
> /usr/src/debug/gzip/1.6-r0/gzip-1.6/gzip.c:325
> (gdb) c
> Continuing.
>
> Breakpoint 1, main (argc=2, argv=0x7fffffffeca8) at
> /usr/src/debug/gzip/1.6-r0/gzip-1.6/gzip.c:465
> 465                 help(); do_exit(OK); break;
> (gdb) list
> 460             case 'd':
> 461                 decompress = 1; break;
> 462             case 'f':
> 463                 force++; break;
> 464             case 'h': case 'H':
> 465                 help(); do_exit(OK); break;
> 466             case 'k':
> 467                 keep = 1; break;
> 468             case 'l':
> 469                 list = decompress = to_stdout = 1; break;
> (gdb) c
> Continuing.
> [Inferior 1 (process 1246) exited normally]
>
>
> 10. On target side, get output of "gzip -h"
> [snip]
> Remote debugging from host 128.224.153.74
> Usage: gzip [OPTION]... [FILE]...
> Compress or uncompress FILEs (by default, compress FILES in-place).
>
> Mandatory arguments to long options are mandatory for short options too.
>
>   -c, --stdout      write on standard output, keep original files unchanged
>   -d, --decompress  decompress
>   -f, --force       force overwrite of output file and compress links
>   -h, --help        give this help
>   -k, --keep        keep (don't delete) input files
>   -l, --list        list compressed file contents
>   -L, --license     display software license
>   -n, --no-name     do not save or restore the original name and time stamp
>   -N, --name        save or restore the original name and time stamp
>   -q, --quiet       suppress all warnings
>   -r, --recursive   operate recursively on directories
>   -S, --suffix=SUF  use suffix SUF on compressed files
>   -t, --test        test compressed file integrity
>   -v, --verbose     verbose mode
>   -V, --version     display version number
>   -1, --fast        compress faster
>   -9, --best        compress better
>
> With no FILE, or when FILE is -, read standard input.
>
> Report bugs to <bug-gzip@gnu.org>.
>
> Child exited with status 0
> GDBserver exiting
> [snip]
>
> //Hongxu
>
thanks a lot for your efforts. Do I understand this right: You suggest
to use build sysroot (on my own risk - as I did before) and make gdb
happy by linking sources?

Problem I see is that we have multiple package archs e.g
ARM/ARMThumb/Machine so ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}
expands to multiple paths. But a script symlinking all together + set
substitute-path might help here...

I will play around with this in the later...

Andreas

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Mark Hatle]

On 4/19/16 9:11 AM, Andreas Müller wrote:
> thanks a lot for your efforts. Do I understand this right: You suggest
> to use build sysroot (on my own risk - as I did before) and make gdb
> happy by linking sources?
> 
> Problem I see is that we have multiple package archs e.g
> ARM/ARMThumb/Machine so ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}
> expands to multiple paths. But a script symlinking all together + set
> substitute-path might help here...
> 
> I will play around with this in the later...
> 
> Andreas
> 

See Yocto Project bug #9481: https://bugzilla.yoctoproject.org/show_bug.cgi?id=9481

Below are the steps that I have added that I hope will be documented in the YP
2.1 documentation. (This matches current master.)

I just went through and verified the steps on my local machine. As noted, there
is currently a bug in the debugfs creation. RP helped me fix the 'rpm' case, but
it appears ipkg support is broken in some way.



1. Configure your build system to construct the companion debug filesystem

In the local.conf:

IMAGE_GEN_DEBUGFS = '1'
IMAGE_FSTYPES_DEBUGFS = 'tar.bz2'

The options above will cause the system to generate a special companion
filesystem fragment, that contains the matching source and debug symbols to your
deployable filesystem. It does this by looking at what is in the deployed
filesystem, and pulling the corresponding -dbg packages.

The companion debug filesystem is NOT a complete filesystem, but only contains
the debug fragements. It must be combined with the full filesystem for
debugging. (Later steps will show this.)

2. Configure the system to include gdbserver in the target filesystem

In the local.conf (or an image recipe):

IMAGE_INSTALL_append = “ gdbserver"

3. Build the environment

Construct the image (and companion Debug Filesystem)
bitbake <image>

Build the cross GDB component and make it available for debugging

Build the SDK that matches the image (this is best for a production build for
later debugging, especially during long term maintenance)
bitbake -c populate_sdk <image>

or

Build the minimal toolchain components that match the target (this is smaller
then a typical SDK and only contains a minimal set of components to build simple
test applications, as well as run the debugger)
bitbake meta-toolchain

or

Build gdb itself within the build system (this produces a temporary copy of
cross-gdb that can be used for debugging during development. It is the quickest
approach, but the other methods are better for long term maintenance strategies.)
bitbake gdb-cross-<arch>

(Note: If you run bitbake gdb-cross, the system will give you a suggestion like
gdb-cross-i586, the suggestion is likely the actual name you want to use.)

4. Setup your environment

4.1 Setup the debugfs

$ mkdir debugfs
$ cd debugfs
$ tar xvfj <build dir>/tmp-glibc/deploy/images/<machine>/<image>.rootfs.tar.bz2
$ tar xvfj <build dir>/tmp-glibc/deploy/images/<machine>/<image>-dbg.rootfs.tar.bz2

4.2 Setup GDB

Install the SDK (if you built one) and source the correct environment file. This
will put it in your path.

if using the build system gdb will be in
<build>/<tmp>/sysroots/<host>/usr/bin/<arch>/<arch>-gdb

5. Boot the target

For QEMU, <link to runqemu docs…>

(verify that your host can access the target via TCP)

6. Debug a program

6.1 Run gdbserver on the target, such as:

(We’ll debug gzip in this example, see gdbserver docs for additional options)

root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help

6.2 On the host, run gdb, configure it and connect to the target:

$ cd <directory holding the debugfs dir>
$ <arch>-gdb

(gdb) set sysroot debugfs
(gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
(gdb) target remote <ip of target>:1234

At this point everything should automatically load (matching binaries, symbols
and headers.)

Note: the gdb ‘set’ commands above can be placed into the users ~/.gdbinit GDB
will automatically run whatever commands are in that file when it is started.


7. Deploying WITHOUT a full image rebuild

In many cases, during development you want a quick method to deploy a new binary
to the target and debug it, without waiting for a full image build.

One approach to this is to build the component you want to debug. Then directly
copy the executable to BOTH the target and the host ‘debugfs’. If the binary is
processed through the debug splitting in OE, you should also copy the debug
items (the .debug contents and corresponding /usr/src/debug) from the work dir.

Such as:

$ bitbake bash
$ bitbake -c devshell bash
$ cd ..
$ scp packages-split/bash/bin/bash <target>:/bin/bash
$ cp -a packages-split/bash-dbg/* <path>/debugfs

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Mark Hatle]

Of course after hitting send, I noticed the example in 7 was missing on step.. :P

On 4/19/16 9:47 AM, Mark Hatle wrote:
> On 4/19/16 9:11 AM, Andreas Müller wrote:
>> thanks a lot for your efforts. Do I understand this right: You suggest
>> to use build sysroot (on my own risk - as I did before) and make gdb
>> happy by linking sources?
>>
>> Problem I see is that we have multiple package archs e.g
>> ARM/ARMThumb/Machine so ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}
>> expands to multiple paths. But a script symlinking all together + set
>> substitute-path might help here...
>>
>> I will play around with this in the later...
>>
>> Andreas
>>
> 
> See Yocto Project bug #9481: https://bugzilla.yoctoproject.org/show_bug.cgi?id=9481
> 
> Below are the steps that I have added that I hope will be documented in the YP
> 2.1 documentation. (This matches current master.)
> 
> I just went through and verified the steps on my local machine. As noted, there
> is currently a bug in the debugfs creation. RP helped me fix the 'rpm' case, but
> it appears ipkg support is broken in some way.
> 
> 
> 
> 1. Configure your build system to construct the companion debug filesystem
> 
> In the local.conf:
> 
> IMAGE_GEN_DEBUGFS = '1'
> IMAGE_FSTYPES_DEBUGFS = 'tar.bz2'
> 
> The options above will cause the system to generate a special companion
> filesystem fragment, that contains the matching source and debug symbols to your
> deployable filesystem. It does this by looking at what is in the deployed
> filesystem, and pulling the corresponding -dbg packages.
> 
> The companion debug filesystem is NOT a complete filesystem, but only contains
> the debug fragements. It must be combined with the full filesystem for
> debugging. (Later steps will show this.)
> 
> 2. Configure the system to include gdbserver in the target filesystem
> 
> In the local.conf (or an image recipe):
> 
> IMAGE_INSTALL_append = “ gdbserver"
> 
> 3. Build the environment
> 
> Construct the image (and companion Debug Filesystem)
> bitbake <image>
> 
> Build the cross GDB component and make it available for debugging
> 
> Build the SDK that matches the image (this is best for a production build for
> later debugging, especially during long term maintenance)
> bitbake -c populate_sdk <image>
> 
> or
> 
> Build the minimal toolchain components that match the target (this is smaller
> then a typical SDK and only contains a minimal set of components to build simple
> test applications, as well as run the debugger)
> bitbake meta-toolchain
> 
> or
> 
> Build gdb itself within the build system (this produces a temporary copy of
> cross-gdb that can be used for debugging during development. It is the quickest
> approach, but the other methods are better for long term maintenance strategies.)
> bitbake gdb-cross-<arch>
> 
> (Note: If you run bitbake gdb-cross, the system will give you a suggestion like
> gdb-cross-i586, the suggestion is likely the actual name you want to use.)
> 
> 4. Setup your environment
> 
> 4.1 Setup the debugfs
> 
> $ mkdir debugfs
> $ cd debugfs
> $ tar xvfj <build dir>/tmp-glibc/deploy/images/<machine>/<image>.rootfs.tar.bz2
> $ tar xvfj <build dir>/tmp-glibc/deploy/images/<machine>/<image>-dbg.rootfs.tar.bz2
> 
> 4.2 Setup GDB
> 
> Install the SDK (if you built one) and source the correct environment file. This
> will put it in your path.
> 
> if using the build system gdb will be in
> <build>/<tmp>/sysroots/<host>/usr/bin/<arch>/<arch>-gdb
> 
> 5. Boot the target
> 
> For QEMU, <link to runqemu docs…>
> 
> (verify that your host can access the target via TCP)
> 
> 6. Debug a program
> 
> 6.1 Run gdbserver on the target, such as:
> 
> (We’ll debug gzip in this example, see gdbserver docs for additional options)
> 
> root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
> 
> 6.2 On the host, run gdb, configure it and connect to the target:
> 
> $ cd <directory holding the debugfs dir>
> $ <arch>-gdb
> 
> (gdb) set sysroot debugfs
> (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
> (gdb) target remote <ip of target>:1234
> 
> At this point everything should automatically load (matching binaries, symbols
> and headers.)
> 
> Note: the gdb ‘set’ commands above can be placed into the users ~/.gdbinit GDB
> will automatically run whatever commands are in that file when it is started.
> 
> 
> 7. Deploying WITHOUT a full image rebuild
> 
> In many cases, during development you want a quick method to deploy a new binary
> to the target and debug it, without waiting for a full image build.
> 
> One approach to this is to build the component you want to debug. Then directly
> copy the executable to BOTH the target and the host ‘debugfs’. If the binary is
> processed through the debug splitting in OE, you should also copy the debug
> items (the .debug contents and corresponding /usr/src/debug) from the work dir.
> 
> Such as:
> 
> $ bitbake bash
> $ bitbake -c devshell bash
> $ cd ..
> $ scp packages-split/bash/bin/bash <target>:/bin/bash

(missing step)
$ cp packages-split/bash/bin/bash <path>/debugfs/bin/bash

> $ cp -a packages-split/bash-dbg/* <path>/debugfs
> 
> 

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Richard Purdie]

On Tue, 2016-04-19 at 16:11 +0200, Andreas Müller wrote:
> thanks a lot for your efforts. Do I understand this right: You
> suggest
> to use build sysroot (on my own risk - as I did before) and make gdb
> happy by linking sources?
> 
> Problem I see is that we have multiple package archs e.g
> ARM/ARMThumb/Machine so ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}
> expands to multiple paths. But a script symlinking all together + set
> substitute-path might help here...
> 
> I will play around with this in the later...

I was thinking about this a bit more and there is another option
available to you which would be to pass all the debug sources into the
sysroot as hardlinks to the files in workdir, in much the same way as
we do the other sysroot files (which you don't strip).

Since those files are collected by do_package you'd probably have to
force do_populate_sysroot after do_package but such an additional class
to make these tweaks probably wouldn't be that large and would just
affect performance a bit (with the benefit of the debugging you want).

Just thinking out loud really. My big worry is all the different
codepaths we have people using, equally, you can't get performance
(disk footprint and speed) and functionality in some cases and people
do want to choose different compromises.

Cheers,

Richard

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Andreas Müller]

On Tue, Apr 19, 2016 at 4:54 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2016-04-19 at 16:11 +0200, Andreas Müller wrote:
>> thanks a lot for your efforts. Do I understand this right: You
>> suggest
>> to use build sysroot (on my own risk - as I did before) and make gdb
>> happy by linking sources?
>>
>> Problem I see is that we have multiple package archs e.g
>> ARM/ARMThumb/Machine so ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}
>> expands to multiple paths. But a script symlinking all together + set
>> substitute-path might help here...
>>
>> I will play around with this in the later...
>
> I was thinking about this a bit more and there is another option
> available to you which would be to pass all the debug sources into the
> sysroot as hardlinks to the files in workdir, in much the same way as
> we do the other sysroot files (which you don't strip).
>
> Since those files are collected by do_package you'd probably have to
> force do_populate_sysroot after do_package but such an additional class
> to make these tweaks probably wouldn't be that large and would just
> affect performance a bit (with the benefit of the debugging you want).
>
> Just thinking out loud really. My big worry is all the different
> codepaths we have people using, equally, you can't get performance
> (disk footprint and speed) and functionality in some cases and people
> do want to choose different compromises.
>
> Cheers,
>
> Richard
Yes such an option would make both possible and is worth spending
efforts: Releasable and properly linked debug sysroot/-dbg packs and
'quick' debugging on top of current build-sysroot.

I can't promise when (daily coding for food eats up my time) but I'll
try to get something to work on this matter. If this will lead to
something (and even if not), the chapter for remote debugging in docs
needs love. The IMAGE_GEN_DEBUGFS / IMAGE_FSTYPES_DEBUGFS can be taken
from this thread :).

I would like to append some notes how to remote debug with qt-creator
- it is really easy to set up and does so much - not only when
debugging qt applications...

Andreas

Re: [PATCH 01/17] conf/bitbake.conf package.bbclass: fix dbg package not contain sources while -fdebug-prefix-map used

[Andreas Müller]

On Tue, Apr 19, 2016 at 4:54 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2016-04-19 at 16:11 +0200, Andreas Müller wrote:
>> thanks a lot for your efforts. Do I understand this right: You
>> suggest
>> to use build sysroot (on my own risk - as I did before) and make gdb
>> happy by linking sources?
>>
>> Problem I see is that we have multiple package archs e.g
>> ARM/ARMThumb/Machine so ${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}
>> expands to multiple paths. But a script symlinking all together + set
>> substitute-path might help here...
>>
>> I will play around with this in the later...
>
> I was thinking about this a bit more and there is another option
> available to you which would be to pass all the debug sources into the
> sysroot as hardlinks to the files in workdir, in much the same way as
> we do the other sysroot files (which you don't strip).
>
> Since those files are collected by do_package you'd probably have to
> force do_populate_sysroot after do_package but such an additional class
> to make these tweaks probably wouldn't be that large and would just
> affect performance a bit (with the benefit of the debugging you want).
Would like to come back to this. Would something like:

addtask do_stage_sources after do_package do_populate_sysroot

in a class enabled by local.conf do the job? I wouldn't have expected
this to be possible until I found something similar in insane.bbclass.

Andreas