[PATCH 0/8] Extensible SDK improvements

[PATCH 0/8] Extensible SDK improvements

[Paul Eggleton]

Some fixes and enhancements centered around how the toolchain is
incorporated in the eSDK, as well as a couple of tweaks for buildhistory
that I found we needed during development.

Note: this is on top of my earlier as-yet unmerged eSDK progress patch.


The following changes since commit 7e4a33933d39f6406922231d64414637a5f52d66:

  classes/populate_sdk_ext: show progress when preparing build system (2016-07-23 00:34:44 +1200)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/extsdkfixes12-oe
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/extsdkfixes12-oe

Paul Eggleton (8):
  classes/populate_sdk_ext: set default for SDK_INCLUDE_PKGDATA
  meta-extsdk-toolchain: add meta-recipe to install toolchain into eSDK
  classes/populate_sdk_ext: allow including toolchain in eSDK on install
  scripts: add oe-check-sstate script
  classes/populate_sdk_ext: filter sstate within the extensible SDK
  classes/populate_sdk_ext: add gdb to full extensible SDK
  classes/buildhistory: add additional variables for eSDK
  classes/buildhistory: ensure eSDK sstate lists sorted secondarily by name

 meta/classes/buildhistory.bbclass               |   6 +-
 meta/classes/populate_sdk_ext.bbclass           |  92 +++++++++++++++++-
 meta/classes/sstate.bbclass                     |   3 +
 meta/lib/oe/copy_buildsystem.py                 |  27 +++++-
 meta/recipes-core/meta/meta-extsdk-toolchain.bb |  28 ++++++
 scripts/gen-lockedsig-cache                     |  16 +++-
 scripts/oe-check-sstate                         | 121 ++++++++++++++++++++++++
 7 files changed, 280 insertions(+), 13 deletions(-)
 create mode 100644 meta/recipes-core/meta/meta-extsdk-toolchain.bb
 create mode 100755 scripts/oe-check-sstate

-- 
2.5.5

[PATCH 1/8] classes/populate_sdk_ext: set default for SDK_INCLUDE_PKGDATA

[Paul Eggleton]

We don't absolutely need this - it doesn't change the default
behaviour, but it seems to me we have a convention to set default values
so we should add one here.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 62f2c3e..720142f 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -20,6 +20,7 @@ SDK_EXT_task-populate-sdk-ext = "-ext"
 
 # Options are full or minimal
 SDK_EXT_TYPE ?= "full"
+SDK_INCLUDE_PKGDATA ?= "0"
 
 SDK_RECRDEP_TASKS ?= ""
 
-- 
2.5.5

[PATCH 2/8] meta-extsdk-toolchain: add meta-recipe to install toolchain into eSDK

[Paul Eggleton]

Add a meta-recipe to bring the toolchain into the extensible SDK. This
was modelled on meta-ide-support but some adjustments were needed to the
dependency validation function in sstate.bbclass to ensure that all of
the toolchain gets installed into the sysroot. With this, after
installing a minimal eSDK you only need to run the following after
sourcing the environment setup script to get the toolchain:

  devtool sdk-install meta-extsdk-toolchain

Addresses [YOCTO #9257].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/sstate.bbclass                     |  3 +++
 meta/recipes-core/meta/meta-extsdk-toolchain.bb | 13 +++++++++++++
 2 files changed, 16 insertions(+)
 create mode 100644 meta/recipes-core/meta/meta-extsdk-toolchain.bb

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index d706d75..2496928 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -927,6 +927,9 @@ def setscene_depvalid(task, taskdependees, notneeded, d):
             # Nothing need depend on libc-initial/gcc-cross-initial
             if "-initial" in taskdependees[task][0]:
                 continue
+            # For meta-extsdk-toolchain we want all sysroot dependencies
+            if taskdependees[dep][0] == 'meta-extsdk-toolchain':
+                return False
             # Native/Cross populate_sysroot need their dependencies
             if isNativeCross(taskdependees[task][0]) and isNativeCross(taskdependees[dep][0]):
                 return False
diff --git a/meta/recipes-core/meta/meta-extsdk-toolchain.bb b/meta/recipes-core/meta/meta-extsdk-toolchain.bb
new file mode 100644
index 0000000..9bff220
--- /dev/null
+++ b/meta/recipes-core/meta/meta-extsdk-toolchain.bb
@@ -0,0 +1,13 @@
+SUMMARY = "Extensible SDK toolchain meta-recipe"
+DESCRIPTION = "Meta-recipe for ensuring the build directory contains all appropriate toolchain packages for using an IDE"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690 \
+                    file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+DEPENDS = "virtual/libc gdb-cross-${TARGET_ARCH} qemu-native qemu-helper-native"
+
+do_populate_sysroot[deptask] = "do_populate_sysroot"
+
+# NOTE: There is logic specific to this recipe in setscene_depvalid()
+# within sstate.bbclass, so if you copy or rename this and expect the same
+# functionality you'll need to modify that as well.
-- 
2.5.5

[PATCH 3/8] classes/populate_sdk_ext: allow including toolchain in eSDK on install

[Paul Eggleton]

If we're to completely replace the standard SDK with the extensible SDK,
we need to be able to provide the standard toolchain on install without
doing anything other than installing it, so that you can install the SDK
and then point your IDE at it. This is particularly applicable to the
minimal SDK which normally installs nothing by default.

NOTE: enabling this option currently adds ~280MB to the size of the
minimal eSDK installer. If we need to reduce this further we would have
to look at adjusting the dependencies and/or the sstate_depvalid()
function in sstate.bbclass which eliminates dependencies, or look at
reducing the size of the artifacts themselves.

Implements [YOCTO #9751].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass           | 19 ++++++++++++++++++-
 meta/lib/oe/copy_buildsystem.py                 |  5 +++--
 meta/recipes-core/meta/meta-extsdk-toolchain.bb | 15 +++++++++++++++
 3 files changed, 36 insertions(+), 3 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 720142f..211a022 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -21,6 +21,7 @@ SDK_EXT_task-populate-sdk-ext = "-ext"
 # Options are full or minimal
 SDK_EXT_TYPE ?= "full"
 SDK_INCLUDE_PKGDATA ?= "0"
+SDK_INCLUDE_TOOLCHAIN ?= "0"
 
 SDK_RECRDEP_TASKS ?= ""
 
@@ -54,6 +55,8 @@ def get_sdk_install_targets(d, images_only=False):
     if not images_only:
         if d.getVar('SDK_INCLUDE_PKGDATA', True) == '1':
             sdk_install_targets += ' meta-world-pkgdata:do_allpackagedata'
+        if d.getVar('SDK_INCLUDE_TOOLCHAIN', True) == '1':
+            sdk_install_targets += ' meta-extsdk-toolchain:do_populate_sysroot'
 
     return sdk_install_targets
 
@@ -309,6 +312,19 @@ python copy_buildsystem () {
                                              lockedsigs_pruned,
                                              lockedsigs_copy)
 
+    if d.getVar('SDK_INCLUDE_TOOLCHAIN', True) == '1':
+        lockedsigs_base = d.getVar('WORKDIR', True) + '/locked-sigs-base2.inc'
+        lockedsigs_toolchain = d.getVar('STAGING_DIR_HOST', True) + '/locked-sigs/locked-sigs-extsdk-toolchain.inc'
+        shutil.move(lockedsigs_pruned, lockedsigs_base)
+        oe.copy_buildsystem.merge_lockedsigs(['do_populate_sysroot'],
+                                             lockedsigs_base,
+                                             lockedsigs_toolchain,
+                                             lockedsigs_pruned)
+        oe.copy_buildsystem.create_locked_sstate_cache(lockedsigs_toolchain,
+                                                       d.getVar('SSTATE_DIR', True),
+                                                       sstate_out, d,
+                                                       fixedlsbstring)
+
     if d.getVar('SDK_EXT_TYPE', True) == 'minimal':
         if derivative:
             # Assume the user is not going to set up an additional sstate
@@ -486,7 +502,8 @@ do_populate_sdk_ext[dirs] = "${@d.getVarFlag('do_populate_sdk', 'dirs', False)}"
 
 do_populate_sdk_ext[depends] = "${@d.getVarFlag('do_populate_sdk', 'depends', False)} \
                                 buildtools-tarball:do_populate_sdk uninative-tarball:do_populate_sdk \
-                                ${@'meta-world-pkgdata:do_collect_packagedata' if d.getVar('SDK_INCLUDE_PKGDATA', True) == '1' else ''}"
+                                ${@'meta-world-pkgdata:do_collect_packagedata' if d.getVar('SDK_INCLUDE_PKGDATA', True) == '1' else ''} \
+                                ${@'meta-extsdk-toolchain:do_locked_sigs' if d.getVar('SDK_INCLUDE_TOOLCHAIN', True) == '1' else ''}"
 
 do_populate_sdk_ext[rdepends] += "${@' '.join([x + ':do_build' for x in d.getVar('SDK_TARGETS', True).split()])}"
 
diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index eddf5bb..b5f546f 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -145,7 +145,7 @@ def prune_lockedsigs(excluded_tasks, excluded_targets, lockedsigs, pruned_output
                     invalue = True
                     f.write(line)
 
-def merge_lockedsigs(copy_tasks, lockedsigs_main, lockedsigs_extra, merged_output, copy_output):
+def merge_lockedsigs(copy_tasks, lockedsigs_main, lockedsigs_extra, merged_output, copy_output=None):
     merged = {}
     arch_order = []
     with open(lockedsigs_main, 'r') as f:
@@ -195,7 +195,8 @@ def merge_lockedsigs(copy_tasks, lockedsigs_main, lockedsigs_extra, merged_outpu
                     fulltypes.append(typename)
             f.write('SIGGEN_LOCKEDSIGS_TYPES = "%s"\n' % ' '.join(fulltypes))
 
-    write_sigs_file(copy_output, list(tocopy.keys()), tocopy)
+    if copy_output:
+        write_sigs_file(copy_output, list(tocopy.keys()), tocopy)
     if merged_output:
         write_sigs_file(merged_output, arch_order, merged)
 
diff --git a/meta/recipes-core/meta/meta-extsdk-toolchain.bb b/meta/recipes-core/meta/meta-extsdk-toolchain.bb
index 9bff220..886ff07 100644
--- a/meta/recipes-core/meta/meta-extsdk-toolchain.bb
+++ b/meta/recipes-core/meta/meta-extsdk-toolchain.bb
@@ -11,3 +11,18 @@ do_populate_sysroot[deptask] = "do_populate_sysroot"
 # NOTE: There is logic specific to this recipe in setscene_depvalid()
 # within sstate.bbclass, so if you copy or rename this and expect the same
 # functionality you'll need to modify that as well.
+
+LOCKED_SIGS_INDIR = "${D}/locked-sigs"
+
+addtask do_locked_sigs after do_populate_sysroot
+SSTATETASKS += "do_locked_sigs"
+do_locked_sigs[sstate-inputdirs] = "${LOCKED_SIGS_INDIR}"
+do_locked_sigs[sstate-outputdirs] = "${STAGING_DIR_HOST}/locked-sigs"
+
+python do_locked_sigs() {
+    import oe.copy_buildsystem
+    outdir = os.path.join(d.getVar('LOCKED_SIGS_INDIR', True))
+    bb.utils.mkdirhier(outdir)
+    sigfile = os.path.join(outdir, 'locked-sigs-extsdk-toolchain.inc')
+    oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
+}
-- 
2.5.5

[PATCH 4/8] scripts: add oe-check-sstate script

[Paul Eggleton]

Add a script to check which sstate artifacts would be installed by
building a given target - by default this is done with a separate
TMPDIR to ensure we get the "from scratch" result. The script produces a
list of tasks that will be restored from the sstate cache. This can also
be combined with BB_SETSCENE_ENFORCE* to check if sstate artifacts are
available.

The implementation is a little crude - we're running bitbake -n and
looking at the output. In future when we have the ability to execute
tasks from tinfoil-based scripts we can look at rewriting that part of
it to use that instead.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/oe-check-sstate | 121 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 121 insertions(+)
 create mode 100755 scripts/oe-check-sstate

diff --git a/scripts/oe-check-sstate b/scripts/oe-check-sstate
new file mode 100755
index 0000000..8aab86a
--- /dev/null
+++ b/scripts/oe-check-sstate
@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+
+# Query which tasks will be restored from sstate
+#
+# Copyright 2016 Intel Corporation
+# Authored-by:  Paul Eggleton <paul.eggleton@intel.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import sys
+import os
+import subprocess
+import tempfile
+import shutil
+import re
+
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+lib_path = scripts_path + '/lib'
+sys.path = sys.path + [lib_path]
+import scriptutils
+import scriptpath
+scriptpath.add_bitbake_lib_path()
+import argparse_oe
+
+
+def translate_virtualfns(tasks):
+    import bb.tinfoil
+    tinfoil = bb.tinfoil.Tinfoil()
+    try:
+        tinfoil.prepare(False)
+
+        pkg_fn = tinfoil.cooker.recipecache.pkg_fn
+        outtasks = []
+        for task in tasks:
+            fn, taskname = task.rsplit(':', 1)
+            if taskname.endswith('_setscene'):
+                taskname = taskname[:-9]
+            outtasks.append('%s:%s' % (pkg_fn[fn], taskname))
+    finally:
+        tinfoil.shutdown()
+    return outtasks
+
+
+def check(args):
+    tmpdir = tempfile.mkdtemp(prefix='oe-check-sstate-')
+    try:
+        env = os.environ.copy()
+        if not args.same_tmpdir:
+            env['BB_ENV_EXTRAWHITE'] = env.get('BB_ENV_EXTRAWHITE', '') + ' TMPDIR_forcevariable'
+            env['TMPDIR_forcevariable'] = tmpdir
+
+        try:
+            output = subprocess.check_output(
+                    'bitbake -n %s' % ' '.join(args.target),
+                    stderr=subprocess.STDOUT,
+                    env=env,
+                    shell=True)
+
+            task_re = re.compile('NOTE: Running setscene task [0-9]+ of [0-9]+ \(([^)]+)\)')
+            tasks = []
+            for line in output.decode('utf-8').splitlines():
+                res = task_re.match(line)
+                if res:
+                    tasks.append(res.group(1))
+            outtasks = translate_virtualfns(tasks)
+        except subprocess.CalledProcessError as e:
+            print('ERROR: bitbake failed:\n%s' % e.output.decode('utf-8'))
+            return e.returncode
+    finally:
+        shutil.rmtree(tmpdir)
+
+    if args.log:
+        with open(args.log, 'wb') as f:
+            f.write(output)
+
+    if args.outfile:
+        with open(args.outfile, 'w') as f:
+            for task in outtasks:
+                f.write('%s\n' % task)
+    else:
+        for task in outtasks:
+            print(task)
+
+    return 0
+
+
+def main():
+    parser = argparse_oe.ArgumentParser(description='OpenEmbedded sstate check tool. Does a dry-run to check restoring the specified targets from shared state, and lists the tasks that would be restored. Set BB_SETSCENE_ENFORCE=1 in the environment if you wish to ensure real tasks are disallowed.')
+
+    parser.add_argument('target', nargs='+', help='Target to check')
+    parser.add_argument('-o', '--outfile', help='Write list to a file instead of stdout')
+    parser.add_argument('-l', '--log', help='Write full log to a file')
+    parser.add_argument('-s', '--same-tmpdir', action='store_true', help='Use same TMPDIR for check (list will then be dependent on what tasks have executed previously)')
+
+    parser.set_defaults(func=check)
+
+    args = parser.parse_args()
+
+    ret = args.func(args)
+    return ret
+
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+        traceback.print_exc()
+    sys.exit(ret)
-- 
2.5.5

[PATCH 5/8] classes/populate_sdk_ext: filter sstate within the extensible SDK

[Paul Eggleton]

Use the new oe-check-sstate to filter the sstate artifacts shipped with
the extensible SDK by effectively running bitbake within the produced
eSDK and and getting it to tell us which tasks it will restore from
sstate. This has several benefits:

1) We drop the *-initial artifacts from the minimal + toolchain eSDK.
   This still leaves us with a reasonably large SDK for this
   configuration, however it does pave the way for future reductions
   since we are actually filtering by what will be expected to be there
   on install rather than hoping that whatever cuts we make will match.

2) We verify bitbake's basic operation within the eSDK, i.e. that
   we haven't messed up the configuration

3) We verify that the sstate artifacts we expect to be present are
   present (at least in the sstate cache for the build producing the
   eSDK). Outside deletion of sstate artifacts has been a problem up to
   now, and this should at least catch that earlier i.e. during the
   build rather than when someone tries to install the eSDK.

This does add a couple of minutes to the do_populate_sdk_ext time, but
it seems like the most appropriate way to handle this.

Should mostly address [YOCTO #9083] and [YOCTO #9626].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 78 ++++++++++++++++++++++++++++++++---
 meta/lib/oe/copy_buildsystem.py       | 22 +++++++++-
 scripts/gen-lockedsig-cache           | 16 ++++++-
 3 files changed, 106 insertions(+), 10 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 211a022..ce8c8ff 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -85,6 +85,60 @@ SDK_EXT_HOST_MANIFEST = "${SDK_DEPLOY}/${TOOLCHAINEXT_OUTPUTNAME}.host.manifest"
 
 SDK_TITLE_task-populate-sdk-ext = "${@d.getVar('DISTRO_NAME', True) or d.getVar('DISTRO', True)} Extensible SDK"
 
+def clean_esdk_builddir(sdkbasepath):
+    """Clean up traces of the fake build for create_filtered_tasklist()"""
+    import shutil
+    cleanpaths = 'tmp cache conf/sanity_info conf/templateconf.cfg downloads'.split()
+    for pth in cleanpaths:
+        fullpth = os.path.join(sdkbasepath, pth)
+        if os.path.isdir(fullpth):
+            shutil.rmtree(fullpth)
+        elif os.path.isfile(fullpth):
+            os.remove(fullpth)
+
+def create_filtered_tasklist(d, sdkbasepath, tasklistfile, conf_initpath):
+    """
+    Create a filtered list of tasks. Also double-checks that the build system
+    within the SDK basically works and required sstate artifacts are available.
+    """
+    import tempfile
+    import shutil
+    import oe.copy_buildsystem
+
+    # Create a temporary build directory that we can pass to the env setup script
+    shutil.copyfile(sdkbasepath + '/conf/local.conf', sdkbasepath + '/conf/local.conf.bak')
+    try:
+        with open(sdkbasepath + '/conf/local.conf', 'a') as f:
+            f.write('\nSSTATE_DIR_forcevariable = "%s"\n' % d.getVar('SSTATE_DIR', True))
+            f.write('SSTATE_MIRRORS_forcevariable = ""\n')
+
+        # Unfortunately the default SDKPATH (or even a custom value) may contain characters that bitbake
+        # will not allow in its COREBASE path, so we need to rename the directory temporarily
+        temp_sdkbasepath = d.getVar('SDK_OUTPUT', True) + '/tmp-renamed-sdk'
+        # Delete any existing temp dir
+        try:
+            shutil.rmtree(temp_sdkbasepath)
+        except FileNotFoundError:
+            pass
+        os.rename(sdkbasepath, temp_sdkbasepath)
+        try:
+            cmdprefix = '. %s .; ' % conf_initpath
+            logfile = d.getVar('WORKDIR', True) + '/tasklist_bb_log.txt'
+            try:
+                oe.copy_buildsystem.check_sstate_task_list(d, get_sdk_install_targets(d), tasklistfile, cmdprefix=cmdprefix, cwd=temp_sdkbasepath, logfile=logfile)
+            except bb.process.ExecutionError as e:
+                msg = 'Failed to generate filtered task list for extensible SDK:\n%s' %  e.stdout.rstrip()
+                if 'attempted to execute unexpectedly and should have been setscened' in e.stdout:
+                    msg += '\n----------\n\nNOTE: "attempted to execute unexpectedly and should have been setscened" errors indicate this may be caused by missing sstate artifacts that were likely produced in earlier builds, but have been subsequently deleted for some reason.\n'
+                bb.fatal(msg)
+        finally:
+            os.rename(temp_sdkbasepath, sdkbasepath)
+        # Clean out residue of running bitbake, which check_sstate_task_list()
+        # will effectively do
+        clean_esdk_builddir(sdkbasepath)
+    finally:
+        os.replace(sdkbasepath + '/conf/local.conf.bak', sdkbasepath + '/conf/local.conf')
+
 python copy_buildsystem () {
     import re
     import shutil
@@ -301,6 +355,15 @@ python copy_buildsystem () {
     # uninative.bbclass sets NATIVELSBSTRING to 'universal'
     fixedlsbstring = 'universal'
 
+    sdk_include_toolchain = (d.getVar('SDK_INCLUDE_TOOLCHAIN', True) == '1')
+    sdk_ext_type = d.getVar('SDK_EXT_TYPE', True)
+    if sdk_ext_type != 'minimal' or sdk_include_toolchain or derivative:
+        # Create the filtered task list used to generate the sstate cache shipped with the SDK
+        tasklistfn = d.getVar('WORKDIR', True) + '/tasklist.txt'
+        create_filtered_tasklist(d, baseoutpath, tasklistfn, conf_initpath)
+    else:
+        tasklistfn = None
+
     # Add packagedata if enabled
     if d.getVar('SDK_INCLUDE_PKGDATA', True) == '1':
         lockedsigs_base = d.getVar('WORKDIR', True) + '/locked-sigs-base.inc'
@@ -312,20 +375,21 @@ python copy_buildsystem () {
                                              lockedsigs_pruned,
                                              lockedsigs_copy)
 
-    if d.getVar('SDK_INCLUDE_TOOLCHAIN', True) == '1':
+    if sdk_include_toolchain:
         lockedsigs_base = d.getVar('WORKDIR', True) + '/locked-sigs-base2.inc'
         lockedsigs_toolchain = d.getVar('STAGING_DIR_HOST', True) + '/locked-sigs/locked-sigs-extsdk-toolchain.inc'
         shutil.move(lockedsigs_pruned, lockedsigs_base)
-        oe.copy_buildsystem.merge_lockedsigs(['do_populate_sysroot'],
+        oe.copy_buildsystem.merge_lockedsigs([],
                                              lockedsigs_base,
                                              lockedsigs_toolchain,
                                              lockedsigs_pruned)
         oe.copy_buildsystem.create_locked_sstate_cache(lockedsigs_toolchain,
                                                        d.getVar('SSTATE_DIR', True),
                                                        sstate_out, d,
-                                                       fixedlsbstring)
+                                                       fixedlsbstring,
+                                                       filterfile=tasklistfn)
 
-    if d.getVar('SDK_EXT_TYPE', True) == 'minimal':
+    if sdk_ext_type == 'minimal':
         if derivative:
             # Assume the user is not going to set up an additional sstate
             # mirror, thus we need to copy the additional artifacts (from
@@ -341,12 +405,14 @@ python copy_buildsystem () {
                 oe.copy_buildsystem.create_locked_sstate_cache(lockedsigs_extra,
                                                                d.getVar('SSTATE_DIR', True),
                                                                sstate_out, d,
-                                                               fixedlsbstring)
+                                                               fixedlsbstring,
+                                                               filterfile=tasklistfn)
     else:
         oe.copy_buildsystem.create_locked_sstate_cache(lockedsigs_pruned,
                                                        d.getVar('SSTATE_DIR', True),
                                                        sstate_out, d,
-                                                       fixedlsbstring)
+                                                       fixedlsbstring,
+                                                       filterfile=tasklistfn)
 
     # We don't need sstate do_package files
     for root, dirs, files in os.walk(sstate_out):
diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index b5f546f..349819e 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -200,11 +200,11 @@ def merge_lockedsigs(copy_tasks, lockedsigs_main, lockedsigs_extra, merged_outpu
     if merged_output:
         write_sigs_file(merged_output, arch_order, merged)
 
-def create_locked_sstate_cache(lockedsigs, input_sstate_cache, output_sstate_cache, d, fixedlsbstring=""):
+def create_locked_sstate_cache(lockedsigs, input_sstate_cache, output_sstate_cache, d, fixedlsbstring="", filterfile=None):
     bb.note('Generating sstate-cache...')
 
     nativelsbstring = d.getVar('NATIVELSBSTRING', True)
-    bb.process.run("gen-lockedsig-cache %s %s %s %s" % (lockedsigs, input_sstate_cache, output_sstate_cache, nativelsbstring))
+    bb.process.run("gen-lockedsig-cache %s %s %s %s %s" % (lockedsigs, input_sstate_cache, output_sstate_cache, nativelsbstring, filterfile or ''))
     if fixedlsbstring:
         nativedir = output_sstate_cache + '/' + nativelsbstring
         if os.path.isdir(nativedir):
@@ -216,3 +216,21 @@ def create_locked_sstate_cache(lockedsigs, input_sstate_cache, output_sstate_cac
                 src = os.path.join(nativedir, i)
                 dest = os.path.join(destdir, i)
                 os.rename(src, dest)
+
+def check_sstate_task_list(d, targets, filteroutfile, cmdprefix='', cwd=None, logfile=None):
+    import subprocess
+
+    bb.note('Generating sstate task list...')
+
+    if not cwd:
+        cwd = os.getcwd()
+    if logfile:
+        logparam = '-l %s' % logfile
+    else:
+        logparam = ''
+    cmd = "%sBB_SETSCENE_ENFORCE=1 PSEUDO_DISABLED=1 oe-check-sstate %s -s -o %s %s" % (cmdprefix, targets, filteroutfile, logparam)
+    env = dict(d.getVar('BB_ORIGENV', False))
+    env.pop('BUILDDIR', '')
+    pathitems = env['PATH'].split(':')
+    env['PATH'] = ':'.join([item for item in pathitems if not item.endswith('/bitbake/bin')])
+    bb.process.run(cmd, stderr=subprocess.STDOUT, env=env, cwd=cwd)
diff --git a/scripts/gen-lockedsig-cache b/scripts/gen-lockedsig-cache
index 26e9b63..de8a20c 100755
--- a/scripts/gen-lockedsig-cache
+++ b/scripts/gen-lockedsig-cache
@@ -15,15 +15,27 @@ def mkdir(d):
 
 if len(sys.argv) < 5:
     print("Incorrect number of arguments specified")
-    print("syntax: gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir> <nativelsbstring>")
+    print("syntax: gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir> <nativelsbstring> [filterfile]")
     sys.exit(1)
 
+filterlist = []
+if len(sys.argv) > 5:
+    print('Reading filter file %s' % sys.argv[5])
+    with open(sys.argv[5]) as f:
+        for l in f.readlines():
+            if ":" in l:
+                filterlist.append(l.rstrip())
+
 print('Reading %s' % sys.argv[1])
 sigs = []
 with open(sys.argv[1]) as f:
     for l in f.readlines():
         if ":" in l:
-            sigs.append(l.split(":")[2].split()[0])
+            task, sig = l.split()[0].rsplit(':', 1)
+            if filterlist and not task in filterlist:
+                print('Filtering out %s' % task)
+            else:
+                sigs.append(sig)
 
 print('Gathering file list')
 files = set()
-- 
2.5.5

[PATCH 6/8] classes/populate_sdk_ext: add gdb to full extensible SDK

[Paul Eggleton]

If SDK_EXT_TYPE is set to "full" then we really ought to be shipping
everything that is expected to be in the SDK, and that includes gdb
(it's already referred to by the environment setup script if nothing
else). This is implemented by using the SDK_INCLUDE_TOOLCHAIN
functionality I just added, since the only material thing that adds on
top of a full SDK is gdb and we should always have the rest of it in a
full SDK anyway.

Fixes [YOCTO #9850].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index ce8c8ff..f6b0834 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -21,7 +21,7 @@ SDK_EXT_task-populate-sdk-ext = "-ext"
 # Options are full or minimal
 SDK_EXT_TYPE ?= "full"
 SDK_INCLUDE_PKGDATA ?= "0"
-SDK_INCLUDE_TOOLCHAIN ?= "0"
+SDK_INCLUDE_TOOLCHAIN ?= "${@'1' if d.getVar('SDK_EXT_TYPE', True) == 'full' else '0'}"
 
 SDK_RECRDEP_TASKS ?= ""
 
-- 
2.5.5

[PATCH 7/8] classes/buildhistory: add additional variables for eSDK

[Paul Eggleton]

Add SDK_INCLUDE_PKGDATA and SDK_INCLUDE_TOOLCHAIN to the variables that
we put into sdk-info.txt

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 6995d06..943bfc8 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -641,7 +641,7 @@ def buildhistory_get_sdkvars(d):
     sdkvars = "DISTRO DISTRO_VERSION SDK_NAME SDK_VERSION SDKMACHINE SDKIMAGE_FEATURES BAD_RECOMMENDATIONS NO_RECOMMENDATIONS PACKAGE_EXCLUDE"
     if d.getVar('BB_CURRENTTASK', True) == 'populate_sdk_ext':
         # Extensible SDK uses some additional variables
-        sdkvars += " SDK_LOCAL_CONF_WHITELIST SDK_LOCAL_CONF_BLACKLIST SDK_INHERIT_BLACKLIST SDK_UPDATE_URL SDK_EXT_TYPE SDK_RECRDEP_TASKS"
+        sdkvars += " SDK_LOCAL_CONF_WHITELIST SDK_LOCAL_CONF_BLACKLIST SDK_INHERIT_BLACKLIST SDK_UPDATE_URL SDK_EXT_TYPE SDK_RECRDEP_TASKS SDK_INCLUDE_PKGDATA SDK_INCLUDE_TOOLCHAIN"
     listvars = "SDKIMAGE_FEATURES BAD_RECOMMENDATIONS PACKAGE_EXCLUDE SDK_LOCAL_CONF_WHITELIST SDK_LOCAL_CONF_BLACKLIST SDK_INHERIT_BLACKLIST"
     return outputvars(sdkvars, listvars, d)
 
-- 
2.5.5

[PATCH 8/8] classes/buildhistory: ensure eSDK sstate lists sorted secondarily by name

[Paul Eggleton]

I got fed up with seeing items dance around in sstate-package-sizes.txt
in the buildhistory git repo simply because they have the same size.
Let's sort the list first by size and then also by name to ensure items
with the same size are deterministically sorted.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 943bfc8..2db9145 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -561,11 +561,11 @@ python buildhistory_get_extra_sdkinfo() {
                     tasksizes[task] = origtotal + fsize
                     filesizes[fn] = fsize
         with open(d.expand('${BUILDHISTORY_DIR_SDK}/sstate-package-sizes.txt'), 'w') as f:
-            filesizes_sorted = sorted(filesizes.items(), key=operator.itemgetter(1), reverse=True)
+            filesizes_sorted = sorted(filesizes.items(), key=operator.itemgetter(1, 0), reverse=True)
             for fn, size in filesizes_sorted:
                 f.write('%10d KiB %s\n' % (size, fn))
         with open(d.expand('${BUILDHISTORY_DIR_SDK}/sstate-task-sizes.txt'), 'w') as f:
-            tasksizes_sorted = sorted(tasksizes.items(), key=operator.itemgetter(1), reverse=True)
+            tasksizes_sorted = sorted(tasksizes.items(), key=operator.itemgetter(1, 0), reverse=True)
             for task, size in tasksizes_sorted:
                 f.write('%10d KiB %s\n' % (size, task))
 }
-- 
2.5.5

Re: [PATCH 0/8] Extensible SDK improvements

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 457 bytes --]

On 22 July 2016 at 13:38, Paul Eggleton <paul.eggleton@linux.intel.com>
wrote:

> Some fixes and enhancements centered around how the toolchain is
> incorporated in the eSDK, as well as a couple of tweaks for buildhistory
> that I found we needed during development.
>

This is happening in mut, could your branch be responsible:

http://errors.yoctoproject.org/Errors/Details/73022/

Ross (who will look further when its not 23:30 on Friday!)

[-- Attachment #2: Type: text/html, Size: 1039 bytes --]

Re: [PATCH 0/8] Extensible SDK improvements

[Paul Eggleton]

On Fri, 22 Jul 2016 23:31:38 Burton, Ross wrote:
> On 22 July 2016 at 13:38, Paul Eggleton <paul.eggleton@linux.intel.com>
> wrote:
> > Some fixes and enhancements centered around how the toolchain is
> > incorporated in the eSDK, as well as a couple of tweaks for buildhistory
> > that I found we needed during development.
> 
> This is happening in mut, could your branch be responsible:
> 
> http://errors.yoctoproject.org/Errors/Details/73022/
> 
> Ross (who will look further when its not 23:30 on Friday!)

Hmm, it will be, but I can't tell immediately from the error message what 
could be wrong...

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/8] Extensible SDK improvements

[Paul Eggleton]

On Sat, 23 Jul 2016 13:11:39 Paul Eggleton wrote:
> On Fri, 22 Jul 2016 23:31:38 Burton, Ross wrote:
> > On 22 July 2016 at 13:38, Paul Eggleton <paul.eggleton@linux.intel.com>
> > 
> > wrote:
> > > Some fixes and enhancements centered around how the toolchain is
> > > incorporated in the eSDK, as well as a couple of tweaks for buildhistory
> > > that I found we needed during development.
> > 
> > This is happening in mut, could your branch be responsible:
> > 
> > http://errors.yoctoproject.org/Errors/Details/73022/
> > 
> > Ross (who will look further when its not 23:30 on Friday!)
> 
> Hmm, it will be, but I can't tell immediately from the error message what
> could be wrong...

Just noticed something that looks odd - why do all these paths have a literal 
TOPDIR at the start? Is that normal?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/8] Extensible SDK improvements

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 1289 bytes --]

On Jul 24, 2016 2:43 PM, "Paul Eggleton" <paul.eggleton@linux.intel.com>
wrote:
>
> On Sat, 23 Jul 2016 13:11:39 Paul Eggleton wrote:
> > On Fri, 22 Jul 2016 23:31:38 Burton, Ross wrote:
> > > On 22 July 2016 at 13:38, Paul Eggleton <paul.eggleton@linux.intel.com
>
> > >
> > > wrote:
> > > > Some fixes and enhancements centered around how the toolchain is
> > > > incorporated in the eSDK, as well as a couple of tweaks for
buildhistory
> > > > that I found we needed during development.
> > >
> > > This is happening in mut, could your branch be responsible:
> > >
> > > http://errors.yoctoproject.org/Errors/Details/73022/
> > >
> > > Ross (who will look further when its not 23:30 on Friday!)
> >
> > Hmm, it will be, but I can't tell immediately from the error message
what
> > could be wrong...
>
> Just noticed something that looks odd - why do all these paths have a
literal
> TOPDIR at the start? Is that normal?

Its normal. Used To hide the person paths.
>
> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

[-- Attachment #2: Type: text/html, Size: 2005 bytes --]

Re: [PATCH 0/8] Extensible SDK improvements

[Paul Eggleton]

On Sun, 24 Jul 2016 16:31:25 Khem Raj wrote:
> On Jul 24, 2016 2:43 PM, "Paul Eggleton" <paul.eggleton@linux.intel.com>
> wrote:
> > Just noticed something that looks odd - why do all these paths have a
> > literal TOPDIR at the start? Is that normal?
> 
> Its normal. Used To hide the person paths.

OK, thanks.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/8] Extensible SDK improvements

[Paul Eggleton]

On Sat, 23 Jul 2016 13:11:39 Paul Eggleton wrote:
> On Fri, 22 Jul 2016 23:31:38 Burton, Ross wrote:
> > On 22 July 2016 at 13:38, Paul Eggleton <paul.eggleton@linux.intel.com>
> > 
> > wrote:
> > > Some fixes and enhancements centered around how the toolchain is
> > > incorporated in the eSDK, as well as a couple of tweaks for buildhistory
> > > that I found we needed during development.
> > 
> > This is happening in mut, could your branch be responsible:
> > 
> > http://errors.yoctoproject.org/Errors/Details/73022/
> > 
> > Ross (who will look further when its not 23:30 on Friday!)
> 
> Hmm, it will be, but I can't tell immediately from the error message what
> could be wrong...

Ah, I believe this is bash vs. dash. Fix incoming, just need to test it.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

[PATCH v2] classes/populate_sdk_ext: filter sstate within the extensible SDK

[Paul Eggleton]

Use the new oe-check-sstate to filter the sstate artifacts shipped with
the extensible SDK by effectively running bitbake within the produced
eSDK and and getting it to tell us which tasks it will restore from
sstate. This has several benefits:

1) We drop the *-initial artifacts from the minimal + toolchain eSDK.
   This still leaves us with a reasonably large SDK for this
   configuration, however it does pave the way for future reductions
   since we are actually filtering by what will be expected to be there
   on install rather than hoping that whatever cuts we make will match.

2) We verify bitbake's basic operation within the eSDK, i.e. that
   we haven't messed up the configuration

3) We verify that the sstate artifacts we expect to be present are
   present (at least in the sstate cache for the build producing the
   eSDK). Outside deletion of sstate artifacts has been a problem up to
   now, and this should at least catch that earlier i.e. during the
   build rather than when someone tries to install the eSDK.

This does add a couple of minutes to the do_populate_sdk_ext time, but
it seems like the most appropriate way to handle this.

Should mostly address [YOCTO #9083] and [YOCTO #9626].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---

Changes since v1 - fixed to run the oe-check-sstate command under /bin/bash
so that it works on distros that use dash such as Ubuntu.

 meta/classes/populate_sdk_ext.bbclass | 78 ++++++++++++++++++++++++++++++++---
 meta/lib/oe/copy_buildsystem.py       | 22 +++++++++-
 scripts/gen-lockedsig-cache           | 16 ++++++-
 3 files changed, 106 insertions(+), 10 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 211a022..ce8c8ff 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -85,6 +85,60 @@ SDK_EXT_HOST_MANIFEST = "${SDK_DEPLOY}/${TOOLCHAINEXT_OUTPUTNAME}.host.manifest"
 
 SDK_TITLE_task-populate-sdk-ext = "${@d.getVar('DISTRO_NAME', True) or d.getVar('DISTRO', True)} Extensible SDK"
 
+def clean_esdk_builddir(sdkbasepath):
+    """Clean up traces of the fake build for create_filtered_tasklist()"""
+    import shutil
+    cleanpaths = 'tmp cache conf/sanity_info conf/templateconf.cfg downloads'.split()
+    for pth in cleanpaths:
+        fullpth = os.path.join(sdkbasepath, pth)
+        if os.path.isdir(fullpth):
+            shutil.rmtree(fullpth)
+        elif os.path.isfile(fullpth):
+            os.remove(fullpth)
+
+def create_filtered_tasklist(d, sdkbasepath, tasklistfile, conf_initpath):
+    """
+    Create a filtered list of tasks. Also double-checks that the build system
+    within the SDK basically works and required sstate artifacts are available.
+    """
+    import tempfile
+    import shutil
+    import oe.copy_buildsystem
+
+    # Create a temporary build directory that we can pass to the env setup script
+    shutil.copyfile(sdkbasepath + '/conf/local.conf', sdkbasepath + '/conf/local.conf.bak')
+    try:
+        with open(sdkbasepath + '/conf/local.conf', 'a') as f:
+            f.write('\nSSTATE_DIR_forcevariable = "%s"\n' % d.getVar('SSTATE_DIR', True))
+            f.write('SSTATE_MIRRORS_forcevariable = ""\n')
+
+        # Unfortunately the default SDKPATH (or even a custom value) may contain characters that bitbake
+        # will not allow in its COREBASE path, so we need to rename the directory temporarily
+        temp_sdkbasepath = d.getVar('SDK_OUTPUT', True) + '/tmp-renamed-sdk'
+        # Delete any existing temp dir
+        try:
+            shutil.rmtree(temp_sdkbasepath)
+        except FileNotFoundError:
+            pass
+        os.rename(sdkbasepath, temp_sdkbasepath)
+        try:
+            cmdprefix = '. %s .; ' % conf_initpath
+            logfile = d.getVar('WORKDIR', True) + '/tasklist_bb_log.txt'
+            try:
+                oe.copy_buildsystem.check_sstate_task_list(d, get_sdk_install_targets(d), tasklistfile, cmdprefix=cmdprefix, cwd=temp_sdkbasepath, logfile=logfile)
+            except bb.process.ExecutionError as e:
+                msg = 'Failed to generate filtered task list for extensible SDK:\n%s' %  e.stdout.rstrip()
+                if 'attempted to execute unexpectedly and should have been setscened' in e.stdout:
+                    msg += '\n----------\n\nNOTE: "attempted to execute unexpectedly and should have been setscened" errors indicate this may be caused by missing sstate artifacts that were likely produced in earlier builds, but have been subsequently deleted for some reason.\n'
+                bb.fatal(msg)
+        finally:
+            os.rename(temp_sdkbasepath, sdkbasepath)
+        # Clean out residue of running bitbake, which check_sstate_task_list()
+        # will effectively do
+        clean_esdk_builddir(sdkbasepath)
+    finally:
+        os.replace(sdkbasepath + '/conf/local.conf.bak', sdkbasepath + '/conf/local.conf')
+
 python copy_buildsystem () {
     import re
     import shutil
@@ -301,6 +355,15 @@ python copy_buildsystem () {
     # uninative.bbclass sets NATIVELSBSTRING to 'universal'
     fixedlsbstring = 'universal'
 
+    sdk_include_toolchain = (d.getVar('SDK_INCLUDE_TOOLCHAIN', True) == '1')
+    sdk_ext_type = d.getVar('SDK_EXT_TYPE', True)
+    if sdk_ext_type != 'minimal' or sdk_include_toolchain or derivative:
+        # Create the filtered task list used to generate the sstate cache shipped with the SDK
+        tasklistfn = d.getVar('WORKDIR', True) + '/tasklist.txt'
+        create_filtered_tasklist(d, baseoutpath, tasklistfn, conf_initpath)
+    else:
+        tasklistfn = None
+
     # Add packagedata if enabled
     if d.getVar('SDK_INCLUDE_PKGDATA', True) == '1':
         lockedsigs_base = d.getVar('WORKDIR', True) + '/locked-sigs-base.inc'
@@ -312,20 +375,21 @@ python copy_buildsystem () {
                                              lockedsigs_pruned,
                                              lockedsigs_copy)
 
-    if d.getVar('SDK_INCLUDE_TOOLCHAIN', True) == '1':
+    if sdk_include_toolchain:
         lockedsigs_base = d.getVar('WORKDIR', True) + '/locked-sigs-base2.inc'
         lockedsigs_toolchain = d.getVar('STAGING_DIR_HOST', True) + '/locked-sigs/locked-sigs-extsdk-toolchain.inc'
         shutil.move(lockedsigs_pruned, lockedsigs_base)
-        oe.copy_buildsystem.merge_lockedsigs(['do_populate_sysroot'],
+        oe.copy_buildsystem.merge_lockedsigs([],
                                              lockedsigs_base,
                                              lockedsigs_toolchain,
                                              lockedsigs_pruned)
         oe.copy_buildsystem.create_locked_sstate_cache(lockedsigs_toolchain,
                                                        d.getVar('SSTATE_DIR', True),
                                                        sstate_out, d,
-                                                       fixedlsbstring)
+                                                       fixedlsbstring,
+                                                       filterfile=tasklistfn)
 
-    if d.getVar('SDK_EXT_TYPE', True) == 'minimal':
+    if sdk_ext_type == 'minimal':
         if derivative:
             # Assume the user is not going to set up an additional sstate
             # mirror, thus we need to copy the additional artifacts (from
@@ -341,12 +405,14 @@ python copy_buildsystem () {
                 oe.copy_buildsystem.create_locked_sstate_cache(lockedsigs_extra,
                                                                d.getVar('SSTATE_DIR', True),
                                                                sstate_out, d,
-                                                               fixedlsbstring)
+                                                               fixedlsbstring,
+                                                               filterfile=tasklistfn)
     else:
         oe.copy_buildsystem.create_locked_sstate_cache(lockedsigs_pruned,
                                                        d.getVar('SSTATE_DIR', True),
                                                        sstate_out, d,
-                                                       fixedlsbstring)
+                                                       fixedlsbstring,
+                                                       filterfile=tasklistfn)
 
     # We don't need sstate do_package files
     for root, dirs, files in os.walk(sstate_out):
diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index b5f546f..4d3faf6 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -200,11 +200,11 @@ def merge_lockedsigs(copy_tasks, lockedsigs_main, lockedsigs_extra, merged_outpu
     if merged_output:
         write_sigs_file(merged_output, arch_order, merged)
 
-def create_locked_sstate_cache(lockedsigs, input_sstate_cache, output_sstate_cache, d, fixedlsbstring=""):
+def create_locked_sstate_cache(lockedsigs, input_sstate_cache, output_sstate_cache, d, fixedlsbstring="", filterfile=None):
     bb.note('Generating sstate-cache...')
 
     nativelsbstring = d.getVar('NATIVELSBSTRING', True)
-    bb.process.run("gen-lockedsig-cache %s %s %s %s" % (lockedsigs, input_sstate_cache, output_sstate_cache, nativelsbstring))
+    bb.process.run("gen-lockedsig-cache %s %s %s %s %s" % (lockedsigs, input_sstate_cache, output_sstate_cache, nativelsbstring, filterfile or ''))
     if fixedlsbstring:
         nativedir = output_sstate_cache + '/' + nativelsbstring
         if os.path.isdir(nativedir):
@@ -216,3 +216,21 @@ def create_locked_sstate_cache(lockedsigs, input_sstate_cache, output_sstate_cac
                 src = os.path.join(nativedir, i)
                 dest = os.path.join(destdir, i)
                 os.rename(src, dest)
+
+def check_sstate_task_list(d, targets, filteroutfile, cmdprefix='', cwd=None, logfile=None):
+    import subprocess
+
+    bb.note('Generating sstate task list...')
+
+    if not cwd:
+        cwd = os.getcwd()
+    if logfile:
+        logparam = '-l %s' % logfile
+    else:
+        logparam = ''
+    cmd = "%sBB_SETSCENE_ENFORCE=1 PSEUDO_DISABLED=1 oe-check-sstate %s -s -o %s %s" % (cmdprefix, targets, filteroutfile, logparam)
+    env = dict(d.getVar('BB_ORIGENV', False))
+    env.pop('BUILDDIR', '')
+    pathitems = env['PATH'].split(':')
+    env['PATH'] = ':'.join([item for item in pathitems if not item.endswith('/bitbake/bin')])
+    bb.process.run(cmd, stderr=subprocess.STDOUT, env=env, cwd=cwd, executable='/bin/bash')
diff --git a/scripts/gen-lockedsig-cache b/scripts/gen-lockedsig-cache
index 26e9b63..de8a20c 100755
--- a/scripts/gen-lockedsig-cache
+++ b/scripts/gen-lockedsig-cache
@@ -15,15 +15,27 @@ def mkdir(d):
 
 if len(sys.argv) < 5:
     print("Incorrect number of arguments specified")
-    print("syntax: gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir> <nativelsbstring>")
+    print("syntax: gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir> <nativelsbstring> [filterfile]")
     sys.exit(1)
 
+filterlist = []
+if len(sys.argv) > 5:
+    print('Reading filter file %s' % sys.argv[5])
+    with open(sys.argv[5]) as f:
+        for l in f.readlines():
+            if ":" in l:
+                filterlist.append(l.rstrip())
+
 print('Reading %s' % sys.argv[1])
 sigs = []
 with open(sys.argv[1]) as f:
     for l in f.readlines():
         if ":" in l:
-            sigs.append(l.split(":")[2].split()[0])
+            task, sig = l.split()[0].rsplit(':', 1)
+            if filterlist and not task in filterlist:
+                print('Filtering out %s' % task)
+            else:
+                sigs.append(sig)
 
 print('Gathering file list')
 files = set()
-- 
2.5.5

Re: [PATCH 0/8] Extensible SDK improvements

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 579 bytes --]

On 24 July 2016 at 22:42, Paul Eggleton <paul.eggleton@linux.intel.com>
wrote:

> Just noticed something that looks odd - why do all these paths have a
> literal
> TOPDIR at the start? Is that normal?
>

The error reporting tool use to replace TOPDIR with " " but that got
confusing as it looked like either whitespace was being added, or the
sysroot was missing in various places.  It now replaces the paths with
TOPDIR or WORKDIR to hide these personal paths in the logs for both privacy
and to let errors.yp have a better chance of finding matching logs.

Ross

[-- Attachment #2: Type: text/html, Size: 982 bytes --]

Re: [PATCH 0/8] Extensible SDK improvements

[Paul Eggleton]

On Mon, 25 Jul 2016 17:39:42 Burton, Ross wrote:
> On 24 July 2016 at 22:42, Paul Eggleton <paul.eggleton@linux.intel.com>
> wrote:
> > Just noticed something that looks odd - why do all these paths have a
> > literal TOPDIR at the start? Is that normal?
> 
> The error reporting tool use to replace TOPDIR with " " but that got
> confusing as it looked like either whitespace was being added, or the
> sysroot was missing in various places.  It now replaces the paths with
> TOPDIR or WORKDIR to hide these personal paths in the logs for both privacy
> and to let errors.yp have a better chance of finding matching logs.

Yep that makes sense. For the autobuilder it's a minor inconvenience but you 
can just click through and find the real log so no big deal.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/8] Extensible SDK improvements

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 766 bytes --]

On 25 July 2016 at 21:33, Paul Eggleton <paul.eggleton@linux.intel.com>
wrote:

> > The error reporting tool use to replace TOPDIR with " " but that got
> > confusing as it looked like either whitespace was being added, or the
> > sysroot was missing in various places.  It now replaces the paths with
> > TOPDIR or WORKDIR to hide these personal paths in the logs for both
> privacy
> > and to let errors.yp have a better chance of finding matching logs.
>
> Yep that makes sense. For the autobuilder it's a minor inconvenience but
> you
> can just click through and find the real log so no big deal.


I find that it actually makes the logs easier to read as there are less
giant paths to read, but yeah the full logs are on the AB still.

Ross

[-- Attachment #2: Type: text/html, Size: 1181 bytes --]

Re: [PATCH 0/8] Extensible SDK improvements

[Richard Purdie]

On Mon, 2016-07-25 at 19:42 +1200, Paul Eggleton wrote:
> On Sat, 23 Jul 2016 13:11:39 Paul Eggleton wrote:
> > On Fri, 22 Jul 2016 23:31:38 Burton, Ross wrote:
> > > On 22 July 2016 at 13:38, Paul Eggleton <
> > > paul.eggleton@linux.intel.com>
> > > 
> > > wrote:
> > > > Some fixes and enhancements centered around how the toolchain
> > > > is
> > > > incorporated in the eSDK, as well as a couple of tweaks for
> > > > buildhistory
> > > > that I found we needed during development.
> > > 
> > > This is happening in mut, could your branch be responsible:
> > > 
> > > http://errors.yoctoproject.org/Errors/Details/73022/
> > > 
> > > Ross (who will look further when its not 23:30 on Friday!)
> > 
> > Hmm, it will be, but I can't tell immediately from the error
> > message what
> > could be wrong...
> 
> Ah, I believe this is bash vs. dash. Fix incoming, just need to test
> it.

Thanks for that fix. I think but am not 100% sure that this patchset
may have also caused:

https://autobuilder.yoctoproject.org/main/builders/nightly-oecore/build
s/865

which happens only on OE-Core testing :/.

Cheers,

Richard

Re: [PATCH 0/8] Extensible SDK improvements

[Paul Eggleton]

On Fri, 29 Jul 2016 08:26:13 Richard Purdie wrote:
> On Mon, 2016-07-25 at 19:42 +1200, Paul Eggleton wrote:
> > On Sat, 23 Jul 2016 13:11:39 Paul Eggleton wrote:
> > > On Fri, 22 Jul 2016 23:31:38 Burton, Ross wrote:
> > > > On 22 July 2016 at 13:38, Paul Eggleton <
> > > > paul.eggleton@linux.intel.com>
> > > > 
> > > > wrote:
> > > > > Some fixes and enhancements centered around how the toolchain
> > > > > is
> > > > > incorporated in the eSDK, as well as a couple of tweaks for
> > > > > buildhistory
> > > > > that I found we needed during development.
> > > > 
> > > > This is happening in mut, could your branch be responsible:
> > > > 
> > > > http://errors.yoctoproject.org/Errors/Details/73022/
> > > > 
> > > > Ross (who will look further when its not 23:30 on Friday!)
> > > 
> > > Hmm, it will be, but I can't tell immediately from the error
> > > message what
> > > could be wrong...
> > 
> > Ah, I believe this is bash vs. dash. Fix incoming, just need to test
> > it.
> 
> Thanks for that fix. I think but am not 100% sure that this patchset
> may have also caused:
> 
> https://autobuilder.yoctoproject.org/main/builders/nightly-oecore/build
> s/865
> 
> which happens only on OE-Core testing :/.

I'm a 100% sure it's *exposed* the issue, but I can't say what would be 
causing the error itself. I'll investigate on Monday.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/8] Extensible SDK improvements

[Paul Eggleton]

On Sat, 30 Jul 2016 22:01:07 Paul Eggleton wrote:
> On Fri, 29 Jul 2016 08:26:13 Richard Purdie wrote:
> > On Mon, 2016-07-25 at 19:42 +1200, Paul Eggleton wrote:
> > > On Sat, 23 Jul 2016 13:11:39 Paul Eggleton wrote:
> > > > On Fri, 22 Jul 2016 23:31:38 Burton, Ross wrote:
> > > > > On 22 July 2016 at 13:38, Paul Eggleton <
> > > > > paul.eggleton@linux.intel.com>
> > > > > 
> > > > > wrote:
> > > > > > Some fixes and enhancements centered around how the toolchain
> > > > > > is
> > > > > > incorporated in the eSDK, as well as a couple of tweaks for
> > > > > > buildhistory
> > > > > > that I found we needed during development.
> > > > > 
> > > > > This is happening in mut, could your branch be responsible:
> > > > > 
> > > > > http://errors.yoctoproject.org/Errors/Details/73022/
> > > > > 
> > > > > Ross (who will look further when its not 23:30 on Friday!)
> > > > 
> > > > Hmm, it will be, but I can't tell immediately from the error
> > > > message what
> > > > could be wrong...
> > > 
> > > Ah, I believe this is bash vs. dash. Fix incoming, just need to test
> > > it.
> > 
> > Thanks for that fix. I think but am not 100% sure that this patchset
> > may have also caused:
> > 
> > https://autobuilder.yoctoproject.org/main/builders/nightly-oecore/build
> > s/865
> > 
> > which happens only on OE-Core testing :/.
> 
> I'm a 100% sure it's *exposed* the issue, but I can't say what would be
> causing the error itself. I'll investigate on Monday.

OK, so I have a fix for this - I was attempting to run the build system before
UNINATIVE_CHECKSUM was set. If you're using poky or something based on it then
that is already true, hence it worked there and thus the issue wasn't apparent.

However this still won't work, for two reasons:

1) The change to support gcc < 5 on the host in uninative.bbclass [1] causes
the signatures for all native targets to change. That shouldn't be a serious
problem, except...

2) As far as I can tell, locked signatures aren't working. You get
"taskhash mismatch" errors, but worse than that the tasks inexplicably still
execute instead of being restored from shared state, except the hash used for
the stamp is the locked value.

Clearly we need to examine the tests at least for #2 as well because they seem
to be either incomplete or broken.

Cheers,
Paul

[1] http://cgit.openembedded.org/openembedded-core/commit/?id=1925ead3828dcd50ef96212c2d1ea9c35bc9f13c

-- 

Paul Eggleton
Intel Open Source Technology Centre