* [PATCH 0/3] Fix some package issues, and add new features
@ 2020-02-07 20:20 Mark Hatle
2020-02-07 20:20 ` [PATCH 1/3] package.bbclass: Fix debug source processing for static libraries Mark Hatle
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Mark Hatle @ 2020-02-07 20:20 UTC (permalink / raw)
To: openembedded-core
While working on baremetal toolchains, I found that the SDKs were potentially
very very large. I investigated and found it was due to included debug info,
but realize the associated source files were not actually present.
Patch 1/3 - Fixes identification of corresponding dwarf debug sources
Patch 2/3 - Allow us to skip kernel modules and static libraries
Patch 3/3 - Add a -dbg like behavior for static libraries (off by default)
Mark Hatle (3):
package.bbclass: Fix debug source processing for static libraries
package.bbclass: Allow INHIBIT_PACKAGE_STRIP_FILES to skip .ko and
static libs
package.bbclass: Support stripping and debug copy of static libraries
meta/classes/insane.bbclass | 2 +-
meta/classes/package.bbclass | 80 ++++++++++++++++++++++++++++++++----
meta/conf/bitbake.conf | 2 +-
3 files changed, 73 insertions(+), 11 deletions(-)
--
2.17.1
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH 1/3] package.bbclass: Fix debug source processing for static libraries 2020-02-07 20:20 [PATCH 0/3] Fix some package issues, and add new features Mark Hatle @ 2020-02-07 20:20 ` Mark Hatle 2020-02-07 20:20 ` [PATCH 2/3] package.bbclass: Allow INHIBIT_PACKAGE_STRIP_FILES to skip .ko and static libs Mark Hatle 2020-02-07 20:20 ` [PATCH 3/3] package.bbclass: Support stripping and debug copy of static libraries Mark Hatle 2 siblings, 0 replies; 5+ messages in thread From: Mark Hatle @ 2020-02-07 20:20 UTC (permalink / raw) To: openembedded-core Format of the sources list is the [ (file, [source, ...]), ... ] before this change, the static libraries were processed but the items were included incorrectly causing no sources for static libraries to be included. Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> --- meta/classes/package.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass index ef3de35961..46ec9b6b3a 100644 --- a/meta/classes/package.bbclass +++ b/meta/classes/package.bbclass @@ -1051,7 +1051,7 @@ python split_and_strip_files () { if debugsrcdir and not targetos.startswith("mingw"): for file in staticlibs: - results.extend(source_info(file, d, fatal=False)) + results.append( (file, source_info(file, d, fatal=False)) ) sources = set() for r in results: -- 2.17.1 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/3] package.bbclass: Allow INHIBIT_PACKAGE_STRIP_FILES to skip .ko and static libs 2020-02-07 20:20 [PATCH 0/3] Fix some package issues, and add new features Mark Hatle 2020-02-07 20:20 ` [PATCH 1/3] package.bbclass: Fix debug source processing for static libraries Mark Hatle @ 2020-02-07 20:20 ` Mark Hatle 2020-02-07 20:20 ` [PATCH 3/3] package.bbclass: Support stripping and debug copy of static libraries Mark Hatle 2 siblings, 0 replies; 5+ messages in thread From: Mark Hatle @ 2020-02-07 20:20 UTC (permalink / raw) To: openembedded-core Change the order of the skip processing to happen before any .ko and static library processing. This will allow these types of files to be individually skipped if necessary. Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> --- meta/classes/package.bbclass | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass index 46ec9b6b3a..7080d63287 100644 --- a/meta/classes/package.bbclass +++ b/meta/classes/package.bbclass @@ -955,12 +955,6 @@ python split_and_strip_files () { for root, dirs, files in cpath.walk(dvar): for f in files: file = os.path.join(root, f) - if file.endswith(".ko") and file.find("/lib/modules/") != -1: - kernmods.append(file) - continue - if oe.package.is_static_lib(file): - staticlibs.append(file) - continue # Skip debug files if debugappend and file.endswith(debugappend): @@ -971,6 +965,13 @@ python split_and_strip_files () { if file in skipfiles: continue + if file.endswith(".ko") and file.find("/lib/modules/") != -1: + kernmods.append(file) + continue + if oe.package.is_static_lib(file): + staticlibs.append(file) + continue + try: ltarget = cpath.realpath(file, dvar, False) s = cpath.lstat(ltarget) -- 2.17.1 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 3/3] package.bbclass: Support stripping and debug copy of static libraries 2020-02-07 20:20 [PATCH 0/3] Fix some package issues, and add new features Mark Hatle 2020-02-07 20:20 ` [PATCH 1/3] package.bbclass: Fix debug source processing for static libraries Mark Hatle 2020-02-07 20:20 ` [PATCH 2/3] package.bbclass: Allow INHIBIT_PACKAGE_STRIP_FILES to skip .ko and static libs Mark Hatle @ 2020-02-07 20:20 ` Mark Hatle 2020-04-16 16:43 ` [OE-core] " Peter Kjellerstedt 2 siblings, 1 reply; 5+ messages in thread From: Mark Hatle @ 2020-02-07 20:20 UTC (permalink / raw) To: openembedded-core By default, we won't copy and strip static libraries. However, this functionality can be useful in some cases where people are doing development on the target, and don't generally want the larger debug capable static libraries. To enable the new functionality set: PACKAGE_DEBUG_STATIC_SPLIT = '1' Add a new function splitstaticdebuginfo. Thus function will copy the unmodified static library into the specific debug directory location. By keeping an unmodified version, it is possible for a user trying to debug something to use -L /usr/lib/.debug-static and their existing build commands to switch from stripped to full debug versions. The PACKAGE_DEBUG_SPLIT_STYLE will select between two different approaches, /usr/lib/debug-static or <path>/.debug-static. Additionally you can now choose to strip static libraries to conserve space. If either 'PACKAGE_DEBUG_STATIC_SPLIT' or 'PACKAGE_STRIP_STATIC' is set to 1, the static library will be stripped. (This is not on by default, as it could make diagnosing static library usage difficult in some cases.) Add to insane.bbclass a skip to the staticdev warning for the specific -dbg package versions. Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> --- meta/classes/insane.bbclass | 2 +- meta/classes/package.bbclass | 67 ++++++++++++++++++++++++++++++++++-- meta/conf/bitbake.conf | 2 +- 3 files changed, 66 insertions(+), 5 deletions(-) diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass index 795c7b9212..7425b8cbd5 100644 --- a/meta/classes/insane.bbclass +++ b/meta/classes/insane.bbclass @@ -182,7 +182,7 @@ def package_qa_check_staticdev(path, name, d, elf, messages): libgcc.a, libgcov.a will be skipped in their packages """ - if not name.endswith("-pic") and not name.endswith("-staticdev") and not name.endswith("-ptest") and path.endswith(".a") and not path.endswith("_nonshared.a"): + if not name.endswith("-pic") and not name.endswith("-staticdev") and not name.endswith("-ptest") and path.endswith(".a") and not path.endswith("_nonshared.a") and not '/usr/lib/debug-static/' in path and not '/.debug-static/' in path: package_qa_add_message(messages, "staticdev", "non -staticdev package contains static .a library: %s path '%s'" % \ (name, package_qa_clean_path(path,d))) diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass index 7080d63287..1efc396ac6 100644 --- a/meta/classes/package.bbclass +++ b/meta/classes/package.bbclass @@ -416,6 +416,49 @@ def splitdebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir, return (file, sources) +def splitstaticdebuginfo(file, dvar, debugstaticdir, debugstaticlibdir, debugstaticappend, debugsrcdir, d): + # Unlike the function above, there is no way to split a static library + # two components. So to get similar results we will copy the unmodified + # static library (containing the debug symbols) into a new directory. + # We will then strip (preserving symbols) the static library in the + # typical location. + # + # return a mapping of files:debugsources + + import stat + import shutil + + src = file[len(dvar):] + dest = debugstaticlibdir + os.path.dirname(src) + debugstaticdir + "/" + os.path.basename(src) + debugstaticappend + debugfile = dvar + dest + sources = [] + + # Copy the file... + bb.utils.mkdirhier(os.path.dirname(debugfile)) + #bb.note("Copy %s -> %s" % (file, debugfile)) + + dvar = d.getVar('PKGD') + + newmode = None + if not os.access(file, os.W_OK) or os.access(file, os.R_OK): + origmode = os.stat(file)[stat.ST_MODE] + newmode = origmode | stat.S_IWRITE | stat.S_IREAD + os.chmod(file, newmode) + + # We need to extract the debug src information here... + if debugsrcdir: + sources = source_info(file, d) + + bb.utils.mkdirhier(os.path.dirname(debugfile)) + + # Copy the unmodified item to the debug directory + shutil.copy2(file, debugfile) + + if newmode: + os.chmod(file, origmode) + + return (file, sources) + def copydebugsources(debugsrcdir, sources, d): # The debug src information written out to sourcefile is further processed # and copied to the destination here. @@ -916,25 +959,37 @@ python split_and_strip_files () { if d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-file-directory': # Single debug-file-directory style debug info debugappend = ".debug" + debugstaticappend = "" debugdir = "" + debugstaticdir = "" debuglibdir = "/usr/lib/debug" + debugstaticlibdir = "/usr/lib/debug-static" debugsrcdir = "/usr/src/debug" elif d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-without-src': # Original OE-core, a.k.a. ".debug", style debug info, but without sources in /usr/src/debug debugappend = "" + debugstaticappend = "" debugdir = "/.debug" + debugstaticdir = "/.debug-static" debuglibdir = "" + debugstaticlibdir = "" debugsrcdir = "" elif d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-with-srcpkg': debugappend = "" + debugstaticappend = "" debugdir = "/.debug" + debugstaticdir = "/.debug-static" debuglibdir = "" + debugstaticlibdir = "" debugsrcdir = "/usr/src/debug" else: # Original OE-core, a.k.a. ".debug", style debug info debugappend = "" + debugstaticappend = "" debugdir = "/.debug" + debugstaticdir = "/.debug-static" debuglibdir = "" + debugstaticlibdir = "" debugsrcdir = "/usr/src/debug" # @@ -1051,8 +1106,11 @@ python split_and_strip_files () { results = oe.utils.multiprocess_launch(splitdebuginfo, list(elffiles), d, extraargs=(dvar, debugdir, debuglibdir, debugappend, debugsrcdir, d)) if debugsrcdir and not targetos.startswith("mingw"): - for file in staticlibs: - results.append( (file, source_info(file, d, fatal=False)) ) + if (d.getVar('PACKAGE_DEBUG_STATIC_SPLIT') == '1'): + results = oe.utils.multiprocess_launch(splitstaticdebuginfo, staticlibs, d, extraargs=(dvar, debugstaticdir, debugstaticlibdir, debugstaticappend, debugsrcdir, d)) + else: + for file in staticlibs: + results.append( (file,source_info(file, d)) ) sources = set() for r in results: @@ -1121,6 +1179,9 @@ python split_and_strip_files () { sfiles.append((file, elf_file, strip)) for f in kernmods: sfiles.append((f, 16, strip)) + if (d.getVar('PACKAGE_STRIP_STATIC') == '1' or d.getVar('PACKAGE_DEBUG_STATIC_SPLIT') == '1'): + for f in staticlibs: + sfiles.append((f, 16, strip)) oe.utils.multiprocess_launch(oe.package.runstrip, sfiles, d) @@ -1188,7 +1249,7 @@ python populate_packages () { dir = os.sep for f in (files + dirs): path = "." + os.path.join(dir, f) - if "/.debug/" in path or path.endswith("/.debug"): + if "/.debug/" in path or "/.debug-static/" in path or path.endswith("/.debug"): debug.append(path) for pkg in packages: diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 8b4621f9b9..954c06b313 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -325,7 +325,7 @@ FILES_${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a" SECTION_${PN}-staticdev = "devel" RDEPENDS_${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})" -FILES_${PN}-dbg = "/usr/lib/debug /usr/src/debug" +FILES_${PN}-dbg = "/usr/lib/debug /usr/lib/debug-static /usr/src/debug" SECTION_${PN}-dbg = "devel" ALLOW_EMPTY_${PN}-dbg = "1" -- 2.17.1 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [OE-core] [PATCH 3/3] package.bbclass: Support stripping and debug copy of static libraries 2020-02-07 20:20 ` [PATCH 3/3] package.bbclass: Support stripping and debug copy of static libraries Mark Hatle @ 2020-04-16 16:43 ` Peter Kjellerstedt 0 siblings, 0 replies; 5+ messages in thread From: Peter Kjellerstedt @ 2020-04-16 16:43 UTC (permalink / raw) To: Mark Hatle; +Cc: openembedded-core > -----Original Message----- > From: openembedded-core-bounces@lists.openembedded.org <openembedded- > core-bounces@lists.openembedded.org> On Behalf Of Mark Hatle > Sent: den 7 februari 2020 21:20 > To: openembedded-core@lists.openembedded.org > Subject: [OE-core] [PATCH 3/3] package.bbclass: Support stripping and > debug copy of static libraries > > By default, we won't copy and strip static libraries. However, this > functionality can be useful in some cases where people are doing > development on the target, and don't generally want the larger debug > capable static libraries. To enable the new functionality set: > > PACKAGE_DEBUG_STATIC_SPLIT = '1' > > Add a new function splitstaticdebuginfo. Thus function will copy the > unmodified static library into the specific debug directory location. > By keeping an unmodified version, it is possible for a user trying > to debug something to use -L /usr/lib/.debug-static and their existing > build commands to switch from stripped to full debug versions. > > The PACKAGE_DEBUG_SPLIT_STYLE will select between two different > approaches, /usr/lib/debug-static or <path>/.debug-static. > > Additionally you can now choose to strip static libraries to conserve > space. If either 'PACKAGE_DEBUG_STATIC_SPLIT' or 'PACKAGE_STRIP_STATIC' > is set to 1, the static library will be stripped. (This is not on by > default, as it could make diagnosing static library usage difficult in > some cases.) > > Add to insane.bbclass a skip to the staticdev warning for the specific > -dbg package versions. > > Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> > --- > meta/classes/insane.bbclass | 2 +- > meta/classes/package.bbclass | 67 ++++++++++++++++++++++++++++++++++-- > meta/conf/bitbake.conf | 2 +- > 3 files changed, 66 insertions(+), 5 deletions(-) [cut] > diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass > index 7080d63287..1efc396ac6 100644 > --- a/meta/classes/package.bbclass > +++ b/meta/classes/package.bbclass [cut] > @@ -1051,8 +1106,11 @@ python split_and_strip_files () { > results = oe.utils.multiprocess_launch(splitdebuginfo, list(elffiles), d, extraargs=(dvar, debugdir, debuglibdir, debugappend, debugsrcdir, d)) > > if debugsrcdir and not targetos.startswith("mingw"): > - for file in staticlibs: > - results.append( (file, source_info(file, d, fatal=False)) ) > + if (d.getVar('PACKAGE_DEBUG_STATIC_SPLIT') == '1'): > + results = oe.utils.multiprocess_launch(splitstaticdebuginfo, staticlibs, d, extraargs=(dvar, debugstaticdir, debugstaticlibdir, debugstaticappend, debugsrcdir, d)) > + else: > + for file in staticlibs: > + results.append( (file,source_info(file, d)) ) Why was fatal=False dropped from the call to source_info()? //Peter ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-04-16 16:43 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-02-07 20:20 [PATCH 0/3] Fix some package issues, and add new features Mark Hatle 2020-02-07 20:20 ` [PATCH 1/3] package.bbclass: Fix debug source processing for static libraries Mark Hatle 2020-02-07 20:20 ` [PATCH 2/3] package.bbclass: Allow INHIBIT_PACKAGE_STRIP_FILES to skip .ko and static libs Mark Hatle 2020-02-07 20:20 ` [PATCH 3/3] package.bbclass: Support stripping and debug copy of static libraries Mark Hatle 2020-04-16 16:43 ` [OE-core] " Peter Kjellerstedt
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.