All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] package.bbclass: Include dbgsrc for static libs
@ 2018-02-23  7:46 Ola x Nilsson
  2018-03-27 12:28 ` Ola x Nilsson
  0 siblings, 1 reply; 6+ messages in thread
From: Ola x Nilsson @ 2018-02-23  7:46 UTC (permalink / raw)
  To: openembedded-core

The debugsource must be added from the package providing the static
lib, because any package using that lib does not have access to the
source code.

Fixes [YOCTO #12558]

Signed-off-by: Ola x Nilsson <olani@axis.com>
---
 meta/classes/package.bbclass | 42 ++++++++++++++++++++++++++++++------------
 1 file changed, 30 insertions(+), 12 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index e7e93a067a..ec6fca87b5 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -344,6 +344,20 @@ def parse_debugsources_from_dwarfsrcfiles_output(dwarfsrcfiles_output):
 
     return debugfiles.keys()
 
+def append_source_info(file, sourcefile, d):
+    cmd = "'dwarfsrcfiles' '%s'" % (file)
+    (retval, output) = oe.utils.getstatusoutput(cmd)
+    # 255 means a specific file wasn't fully parsed to get the debug file list, which is not a fatal failure
+    if retval != 0 and retval != 255:
+        bb.fatal("dwarfsrcfiles failed with exit code %s (cmd was %s)%s" % (retval, cmd, ":\n%s" % output if output else ""))
+
+    debugsources = parse_debugsources_from_dwarfsrcfiles_output(output)
+    # filenames are null-separated - this is an artefact of the previous use
+    # of rpm's debugedit, which was writing them out that way, and the code elsewhere
+    # is still assuming that.
+    debuglistoutput = '\0'.join(debugsources) + '\0'
+    open(sourcefile, 'a').write(debuglistoutput)
+
 
 def splitdebuginfo(file, debugfile, debugsrcdir, sourcefile, d):
     # Function to split a single file into two components, one is the stripped
@@ -369,18 +383,7 @@ def splitdebuginfo(file, debugfile, debugsrcdir, sourcefile, d):
 
     # We need to extract the debug src information here...
     if debugsrcdir:
-        cmd = "'dwarfsrcfiles' '%s'" % (file)
-        (retval, output) = oe.utils.getstatusoutput(cmd)
-        # 255 means a specific file wasn't fully parsed to get the debug file list, which is not a fatal failure
-        if retval != 0 and retval != 255:
-            bb.fatal("dwarfsrcfiles failed with exit code %s (cmd was %s)%s" % (retval, cmd, ":\n%s" % output if output else ""))
-
-        debugsources = parse_debugsources_from_dwarfsrcfiles_output(output)
-        # filenames are null-separated - this is an artefact of the previous use
-        # of rpm's debugedit, which was writing them out that way, and the code elsewhere
-        # is still assuming that.
-        debuglistoutput = '\0'.join(debugsources) + '\0'
-        open(sourcefile, 'a').write(debuglistoutput)
+        append_source_info(file, sourcefile, d)
 
     bb.utils.mkdirhier(os.path.dirname(debugfile))
 
@@ -936,6 +939,13 @@ python split_and_strip_files () {
                 type |= 8
         return type
 
+    def isStaticLib(path):
+        if file.endswith('.a'):
+            with open(path, 'rb') as fh:
+                magic = b'!<arch>\n'
+                start = fh.read(len(magic))
+                return start == magic
+        return False
 
     #
     # First lets figure out all of the files we may have to process ... do this only once!
@@ -943,6 +953,7 @@ python split_and_strip_files () {
     elffiles = {}
     symlinks = {}
     kernmods = []
+    staticlibs = []
     inodes = {}
     libdir = os.path.abspath(dvar + os.sep + d.getVar("libdir"))
     baselibdir = os.path.abspath(dvar + os.sep + d.getVar("base_libdir"))
@@ -955,6 +966,9 @@ python split_and_strip_files () {
                 if file.endswith(".ko") and file.find("/lib/modules/") != -1:
                     kernmods.append(file)
                     continue
+                if isStaticLib(file):
+                    staticlibs.append(file)
+                    continue
 
                 # Skip debug files
                 if debugappend and file.endswith(debugappend):
@@ -1033,6 +1047,10 @@ python split_and_strip_files () {
             # Only store off the hard link reference if we successfully split!
             splitdebuginfo(file, fpath, debugsrcdir, sourcefile, d)
 
+        if debugsrcdir:
+            for file in staticlibs:
+                append_source_info(file, sourcefile, d)
+
         # Hardlink our debug symbols to the other hardlink copies
         for ref in inodes:
             if len(inodes[ref]) == 1:
-- 
2.11.0



^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [PATCH] package.bbclass: Include dbgsrc for static libs
  2018-02-23  7:46 [PATCH] package.bbclass: Include dbgsrc for static libs Ola x Nilsson
@ 2018-03-27 12:28 ` Ola x Nilsson
  2018-04-04 13:43   ` Richard Purdie
  0 siblings, 1 reply; 6+ messages in thread
From: Ola x Nilsson @ 2018-03-27 12:28 UTC (permalink / raw)
  To: Ola x Nilsson, openembedded-core

ping?

-- 
Ola Nilsson
________________________________________
From: openembedded-core-bounces@lists.openembedded.org <openembedded-core-bounces@lists.openembedded.org> on behalf of Ola x Nilsson <olani@axis.com>
Sent: Friday, February 23, 2018 08:46
To: openembedded-core@lists.openembedded.org
Subject: [OE-core] [PATCH] package.bbclass: Include dbgsrc for static libs

The debugsource must be added from the package providing the static
lib, because any package using that lib does not have access to the
source code.

Fixes [YOCTO #12558]

Signed-off-by: Ola x Nilsson <olani@axis.com>
---
 meta/classes/package.bbclass | 42 ++++++++++++++++++++++++++++++------------
 1 file changed, 30 insertions(+), 12 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index e7e93a067a..ec6fca87b5 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -344,6 +344,20 @@ def parse_debugsources_from_dwarfsrcfiles_output(dwarfsrcfiles_output):

     return debugfiles.keys()

+def append_source_info(file, sourcefile, d):
+    cmd = "'dwarfsrcfiles' '%s'" % (file)
+    (retval, output) = oe.utils.getstatusoutput(cmd)
+    # 255 means a specific file wasn't fully parsed to get the debug file list, which is not a fatal failure
+    if retval != 0 and retval != 255:
+        bb.fatal("dwarfsrcfiles failed with exit code %s (cmd was %s)%s" % (retval, cmd, ":\n%s" % output if output else ""))
+
+    debugsources = parse_debugsources_from_dwarfsrcfiles_output(output)
+    # filenames are null-separated - this is an artefact of the previous use
+    # of rpm's debugedit, which was writing them out that way, and the code elsewhere
+    # is still assuming that.
+    debuglistoutput = '\0'.join(debugsources) + '\0'
+    open(sourcefile, 'a').write(debuglistoutput)
+

 def splitdebuginfo(file, debugfile, debugsrcdir, sourcefile, d):
     # Function to split a single file into two components, one is the stripped
@@ -369,18 +383,7 @@ def splitdebuginfo(file, debugfile, debugsrcdir, sourcefile, d):

     # We need to extract the debug src information here...
     if debugsrcdir:
-        cmd = "'dwarfsrcfiles' '%s'" % (file)
-        (retval, output) = oe.utils.getstatusoutput(cmd)
-        # 255 means a specific file wasn't fully parsed to get the debug file list, which is not a fatal failure
-        if retval != 0 and retval != 255:
-            bb.fatal("dwarfsrcfiles failed with exit code %s (cmd was %s)%s" % (retval, cmd, ":\n%s" % output if output else ""))
-
-        debugsources = parse_debugsources_from_dwarfsrcfiles_output(output)
-        # filenames are null-separated - this is an artefact of the previous use
-        # of rpm's debugedit, which was writing them out that way, and the code elsewhere
-        # is still assuming that.
-        debuglistoutput = '\0'.join(debugsources) + '\0'
-        open(sourcefile, 'a').write(debuglistoutput)
+        append_source_info(file, sourcefile, d)

     bb.utils.mkdirhier(os.path.dirname(debugfile))

@@ -936,6 +939,13 @@ python split_and_strip_files () {
                 type |= 8
         return type

+    def isStaticLib(path):
+        if file.endswith('.a'):
+            with open(path, 'rb') as fh:
+                magic = b'!<arch>\n'
+                start = fh.read(len(magic))
+                return start == magic
+        return False

     #
     # First lets figure out all of the files we may have to process ... do this only once!
@@ -943,6 +953,7 @@ python split_and_strip_files () {
     elffiles = {}
     symlinks = {}
     kernmods = []
+    staticlibs = []
     inodes = {}
     libdir = os.path.abspath(dvar + os.sep + d.getVar("libdir"))
     baselibdir = os.path.abspath(dvar + os.sep + d.getVar("base_libdir"))
@@ -955,6 +966,9 @@ python split_and_strip_files () {
                 if file.endswith(".ko") and file.find("/lib/modules/") != -1:
                     kernmods.append(file)
                     continue
+                if isStaticLib(file):
+                    staticlibs.append(file)
+                    continue

                 # Skip debug files
                 if debugappend and file.endswith(debugappend):
@@ -1033,6 +1047,10 @@ python split_and_strip_files () {
             # Only store off the hard link reference if we successfully split!
             splitdebuginfo(file, fpath, debugsrcdir, sourcefile, d)

+        if debugsrcdir:
+            for file in staticlibs:
+                append_source_info(file, sourcefile, d)
+
         # Hardlink our debug symbols to the other hardlink copies
         for ref in inodes:
             if len(inodes[ref]) == 1:
--
2.11.0

--
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [PATCH] package.bbclass: Include dbgsrc for static libs
  2018-03-27 12:28 ` Ola x Nilsson
@ 2018-04-04 13:43   ` Richard Purdie
  2018-04-19 11:17     ` Ola x Nilsson
  0 siblings, 1 reply; 6+ messages in thread
From: Richard Purdie @ 2018-04-04 13:43 UTC (permalink / raw)
  To: Ola x Nilsson, Ola x Nilsson, openembedded-core

On Tue, 2018-03-27 at 12:28 +0000, Ola x Nilsson wrote:
> ping?

Fails if you enable ptest in openssl. Suspect you need to check if its
a real file rather than a symlink and only check real files.

Cheers,

Richard

ERROR: openssl-1.0.2o-r0 do_package: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:split_and_strip_files(d)
     0003:
File: '/media/build1/poky/meta/classes/package.bbclass', lineno: 969, function: split_and_strip_files
     0965:                file = os.path.join(root, f)
     0966:                if file.endswith(".ko") and file.find("/lib/modules/") != -1:
     0967:                    kernmods.append(file)
     0968:                    continue
 *** 0969:                if isStaticLib(file):
     0970:                    staticlibs.append(file)
     0971:                    continue
     0972:
     0973:                # Skip debug files
File: '/media/build1/poky/meta/classes/package.bbclass', lineno: 944, function: isStaticLib
     0940:        return type
     0941:
     0942:    def isStaticLib(path):
     0943:        if file.endswith('.a'):
 *** 0944:            with open(path, 'rb') as fh:
     0945:                magic = b'!<arch>\n'
     0946:                start = fh.read(len(magic))
     0947:                return start == magic
     0948:        return False
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/media/build1/poky/build/tmp/work/core2-64-poky-linux/openssl/1.0.2o-r0/package/usr/lib/openssl/ptest/libssl.a'

ERROR: openssl-1.0.2o-r0 do_package: Function failed: split_and_strip_files
ERROR: Logfile of failure stored in: /media/build1/poky/build/tmp/work/core2-64-poky-linux/openssl/1.0.2o-r0/temp/log.do_package.20107
ERROR: Task (/media/build1/poky/meta/recipes-connectivity/openssl/openssl_1.0.2o.bb:do_package) failed with exit code '1'
NOTE: Tasks Summary: Attempted 1308 tasks of which 1303 didn't need to be rerun and 1 failed.


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] package.bbclass: Include dbgsrc for static libs
  2018-04-04 13:43   ` Richard Purdie
@ 2018-04-19 11:17     ` Ola x Nilsson
  2018-04-19 11:17       ` [PATCH v2] " Ola x Nilsson
  0 siblings, 1 reply; 6+ messages in thread
From: Ola x Nilsson @ 2018-04-19 11:17 UTC (permalink / raw)
  To: openembedded-core

You were right, I needed to check if it was a proper file.

I stumbled into another issue, that golang .a files can not be read by
dwarfsrcfiles.  I'm not familiar with go, so I don't really know what
to do about it.  This patch is still an improvement in my book, even
if it cannot handle go archives.

New in v2:
 * do not process symbolic links
 * do not process golang .a files



^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH v2] package.bbclass: Include dbgsrc for static libs
  2018-04-19 11:17     ` Ola x Nilsson
@ 2018-04-19 11:17       ` Ola x Nilsson
  2018-04-20  7:13         ` Richard Purdie
  0 siblings, 1 reply; 6+ messages in thread
From: Ola x Nilsson @ 2018-04-19 11:17 UTC (permalink / raw)
  To: openembedded-core

The debugsource must be added from the package providing the static
lib, because any package using that lib does not have access to the
source code.

Fixes [YOCTO #12558]

Signed-off-by: Ola x Nilsson <olani@axis.com>
---
 meta/classes/package.bbclass | 44 ++++++++++++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 12 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 9bba021efb..505a80da8b 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -344,6 +344,20 @@ def parse_debugsources_from_dwarfsrcfiles_output(dwarfsrcfiles_output):
 
     return debugfiles.keys()
 
+def append_source_info(file, sourcefile, d):
+    cmd = "'dwarfsrcfiles' '%s'" % (file)
+    (retval, output) = oe.utils.getstatusoutput(cmd)
+    # 255 means a specific file wasn't fully parsed to get the debug file list, which is not a fatal failure
+    if retval != 0 and retval != 255:
+        bb.fatal("dwarfsrcfiles failed with exit code %s (cmd was %s)%s" % (retval, cmd, ":\n%s" % output if output else ""))
+
+    debugsources = parse_debugsources_from_dwarfsrcfiles_output(output)
+    # filenames are null-separated - this is an artefact of the previous use
+    # of rpm's debugedit, which was writing them out that way, and the code elsewhere
+    # is still assuming that.
+    debuglistoutput = '\0'.join(debugsources) + '\0'
+    open(sourcefile, 'a').write(debuglistoutput)
+
 
 def splitdebuginfo(file, debugfile, debugsrcdir, sourcefile, d):
     # Function to split a single file into two components, one is the stripped
@@ -369,18 +383,7 @@ def splitdebuginfo(file, debugfile, debugsrcdir, sourcefile, d):
 
     # We need to extract the debug src information here...
     if debugsrcdir:
-        cmd = "'dwarfsrcfiles' '%s'" % (file)
-        (retval, output) = oe.utils.getstatusoutput(cmd)
-        # 255 means a specific file wasn't fully parsed to get the debug file list, which is not a fatal failure
-        if retval != 0 and retval != 255:
-            bb.fatal("dwarfsrcfiles failed with exit code %s (cmd was %s)%s" % (retval, cmd, ":\n%s" % output if output else ""))
-
-        debugsources = parse_debugsources_from_dwarfsrcfiles_output(output)
-        # filenames are null-separated - this is an artefact of the previous use
-        # of rpm's debugedit, which was writing them out that way, and the code elsewhere
-        # is still assuming that.
-        debuglistoutput = '\0'.join(debugsources) + '\0'
-        open(sourcefile, 'a').write(debuglistoutput)
+        append_source_info(file, sourcefile, d)
 
     bb.utils.mkdirhier(os.path.dirname(debugfile))
 
@@ -936,6 +939,15 @@ python split_and_strip_files () {
                 type |= 8
         return type
 
+    def isStaticLib(path):
+        if path.endswith('.a') and not os.path.islink(path):
+            with open(path, 'rb') as fh:
+                # The magic must include the first slash to avoid
+                # matching golang static libraries
+                magic = b'!<arch>\x0a/'
+                start = fh.read(len(magic))
+                return start == magic
+        return False
 
     #
     # First lets figure out all of the files we may have to process ... do this only once!
@@ -943,6 +955,7 @@ python split_and_strip_files () {
     elffiles = {}
     symlinks = {}
     kernmods = []
+    staticlibs = []
     inodes = {}
     libdir = os.path.abspath(dvar + os.sep + d.getVar("libdir"))
     baselibdir = os.path.abspath(dvar + os.sep + d.getVar("base_libdir"))
@@ -955,6 +968,9 @@ python split_and_strip_files () {
                 if file.endswith(".ko") and file.find("/lib/modules/") != -1:
                     kernmods.append(file)
                     continue
+                if isStaticLib(file):
+                    staticlibs.append(file)
+                    continue
 
                 # Skip debug files
                 if debugappend and file.endswith(debugappend):
@@ -1033,6 +1049,10 @@ python split_and_strip_files () {
             # Only store off the hard link reference if we successfully split!
             splitdebuginfo(file, fpath, debugsrcdir, sourcefile, d)
 
+        if debugsrcdir:
+            for file in staticlibs:
+                append_source_info(file, sourcefile, d)
+
         # Hardlink our debug symbols to the other hardlink copies
         for ref in inodes:
             if len(inodes[ref]) == 1:
-- 
2.11.0



^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [PATCH v2] package.bbclass: Include dbgsrc for static libs
  2018-04-19 11:17       ` [PATCH v2] " Ola x Nilsson
@ 2018-04-20  7:13         ` Richard Purdie
  0 siblings, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2018-04-20  7:13 UTC (permalink / raw)
  To: Ola x Nilsson, openembedded-core

On Thu, 2018-04-19 at 13:17 +0200, Ola x Nilsson wrote:
> The debugsource must be added from the package providing the static
> lib, because any package using that lib does not have access to the
> source code.
> 
> Fixes [YOCTO #12558]
> 
> Signed-off-by: Ola x Nilsson <olani@axis.com>
> ---
>  meta/classes/package.bbclass | 44 ++++++++++++++++++++++++++++++++
> ------------
>  1 file changed, 32 insertions(+), 12 deletions(-)

I messed up and accidentally merged this onto OE-Core early and then we
had test failures on mingw. 

To resolve that I've added a tweak to disable this on mingw since it
doesn't work there.

Cheers,

Richard


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2018-04-20  7:13 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-23  7:46 [PATCH] package.bbclass: Include dbgsrc for static libs Ola x Nilsson
2018-03-27 12:28 ` Ola x Nilsson
2018-04-04 13:43   ` Richard Purdie
2018-04-19 11:17     ` Ola x Nilsson
2018-04-19 11:17       ` [PATCH v2] " Ola x Nilsson
2018-04-20  7:13         ` Richard Purdie

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.