[PATCH 00/14] Extensible SDK improvements redux

[PATCH 00/14] Extensible SDK improvements redux

[Paul Eggleton]

* Fixes around ensuring pkgdata is present to help find dependencies
* Fixes for the SDK update process
* Add the ability to install prebuilt items into the SDK (from sstate)
* Fix a few other issues I noticed in the process

Note that the world pkgdata implementation relies on a BitBake patch I
just sent.


The following changes since commit 8084c185c43cb7e7f5e29d31939dfd308f8a0f93:

  udev: Add 2 patches to support 4.4 kernel (2016-01-20 09:20:06 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/extsdkfixes5-oe
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/extsdkfixes5-oe

Paul Eggleton (14):
  classes/populate_sdk_ext: add option to bring in pkgdata for world
  devtool: build: ensure pkgdata is written out
  devtool: sdk-update: improve temp directory handling
  devtool: sdk-update: improve SDK update process robustness
  devtool: add sdk-install subcommand
  classes/populate_sdk_ext: drop ext-sdk-prepare.py when installing
  classes/populate_sdk*: add dependencies on script files
  toolchain-shar-extract.sh: improve behaviour when xz is not installed
  gen-lockedsig-cache: copy correct native sstate into ext SDK
  recipetool: create: strip quotes from values extracted from
    CMakeLists.txt
  devtool / recipetool: support specifying a subdirectory within the
    fetched source
  recipetool: create: extract SRC_URI from local git repositories
  recipetool: create: fix extraction of name from URLs ending in /
  recipetool: create: better fix for fetch error handling

 meta/classes/populate_sdk_base.bbclass       |   3 +
 meta/classes/populate_sdk_ext.bbclass        |  46 +++++-
 meta/files/toolchain-shar-extract.sh         |   7 +-
 meta/lib/oe/copy_buildsystem.py              |  61 +++++++-
 meta/lib/oeqa/selftest/devtool.py            |   7 +-
 meta/recipes-core/meta/meta-world-pkgdata.bb |  50 +++++++
 scripts/gen-lockedsig-cache                  |   6 +-
 scripts/lib/devtool/build.py                 |  16 +-
 scripts/lib/devtool/sdk.py                   | 212 ++++++++++++++++++++-------
 scripts/lib/devtool/standard.py              |   6 +
 scripts/lib/recipetool/create.py             |  47 ++++--
 scripts/lib/recipetool/create_buildsys.py    |  13 +-
 12 files changed, 392 insertions(+), 82 deletions(-)
 create mode 100644 meta/recipes-core/meta/meta-world-pkgdata.bb

-- 
2.5.0

[PATCH 01/14] classes/populate_sdk_ext: add option to bring in pkgdata for world

[Paul Eggleton]

Add a variable SDK_INCLUDE_PKGDATA which you can set to "1" to include
pkgdata for all recipes in the world target. There are a couple of uses
for this:

1) If you use "devtool add" to add a recipe that builds something which
   depends on anything in world, the dependency can then be correctly
   mapped to the recipe providing it and that recipe can be added to
   DEPENDS, since we have the pkg-config and shared library dependency
   data within pkgdata.
2) You'll be able to search for these recipes and any files they
   package for the target with "devtool search" since that also uses
   pkgdata

This of course assumes you've tailored world through EXCLUDE_FROM_WORLD
to only include recipes you'd want built in your distro, but I think
that's a reasonable assumption; failing that there is a
WORLD_PKGDATA_EXCLUDE variable that you can set to exclude any recipes
you don't want.

Note that this patch relies on functionality implemented in a recent
BitBake patch and will not work without it.

Implements [YOCTO #8600].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass        | 24 +++++++++++-
 meta/lib/oe/copy_buildsystem.py              | 58 +++++++++++++++++++++++++++-
 meta/recipes-core/meta/meta-world-pkgdata.bb | 50 ++++++++++++++++++++++++
 scripts/lib/devtool/sdk.py                   |  6 +--
 4 files changed, 131 insertions(+), 7 deletions(-)
 create mode 100644 meta/recipes-core/meta/meta-world-pkgdata.bb

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index e5e55b3..ebb57bb 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -24,6 +24,7 @@ SDK_INHERIT_BLACKLIST ?= "buildhistory icecc"
 SDK_UPDATE_URL ?= ""
 
 SDK_TARGETS ?= "${PN}"
+SDK_INSTALL_TARGETS = "${SDK_TARGETS} ${@'meta-world-pkgdata:do_allpackagedata' if d.getVar('SDK_INCLUDE_PKGDATA', True) == '1' else ''}"
 OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"
 
 # The files from COREBASE that you want preserved in the COREBASE copied
@@ -45,6 +46,7 @@ SDK_TITLE_task-populate-sdk-ext = "${@d.getVar('DISTRO_NAME', True) or d.getVar(
 
 python copy_buildsystem () {
     import re
+    import shutil
     import oe.copy_buildsystem
 
     oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT', True)
@@ -91,6 +93,7 @@ python copy_buildsystem () {
     config.set('General', 'core_meta_subdir', core_meta_subdir)
     config.add_section('SDK')
     config.set('SDK', 'sdk_targets', d.getVar('SDK_TARGETS', True))
+    config.set('SDK', 'sdk_update_targets', d.getVar('SDK_INSTALL_TARGETS', True))
     updateurl = d.getVar('SDK_UPDATE_URL', True)
     if updateurl:
         config.set('SDK', 'updateserver', updateurl)
@@ -199,6 +202,22 @@ python copy_buildsystem () {
                                                    d.getVar('SSTATE_DIR', True),
                                                    sstate_out, d,
                                                    fixedlsbstring)
+
+    # Add packagedata if enabled
+    if d.getVar('SDK_INCLUDE_PKGDATA', True) == '1':
+        lockedsigs_base = d.getVar('WORKDIR', True) + '/locked-sigs-base.inc'
+        lockedsigs_copy = d.getVar('WORKDIR', True) + '/locked-sigs-copy.inc'
+        shutil.move(lockedsigs_pruned, lockedsigs_base)
+        oe.copy_buildsystem.merge_lockedsigs(['do_packagedata'],
+                                             lockedsigs_base,
+                                             d.getVar('STAGING_DIR_HOST', True) + '/world-pkgdata/locked-sigs-pkgdata.inc',
+                                             lockedsigs_pruned,
+                                             lockedsigs_copy)
+        oe.copy_buildsystem.create_locked_sstate_cache(lockedsigs_copy,
+                                                       d.getVar('SSTATE_DIR', True),
+                                                       sstate_out, d,
+                                                       fixedlsbstring)
+
     # We don't need sstate do_package files
     for root, dirs, files in os.walk(sstate_out):
         for name in files:
@@ -268,7 +287,7 @@ sdk_ext_postinst() {
 		# current working directory when first ran, nor will it set $1 when
 		# sourcing a script. That is why this has to look so ugly.
 		LOGFILE="$target_sdk_dir/preparing_build_system.log"
-		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py '${SDK_TARGETS}' >> $LOGFILE 2>&1" || { echo "ERROR: SDK preparation failed: see $LOGFILE"; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
+		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py '${SDK_INSTALL_TARGETS}' >> $LOGFILE 2>&1" || { echo "ERROR: SDK preparation failed: see $LOGFILE"; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
 	fi
 	echo done
 }
@@ -314,7 +333,8 @@ def get_sdk_ext_rdepends(d):
 do_populate_sdk_ext[dirs] = "${@d.getVarFlag('do_populate_sdk', 'dirs', False)}"
 
 do_populate_sdk_ext[depends] = "${@d.getVarFlag('do_populate_sdk', 'depends', False)} \
-                                buildtools-tarball:do_populate_sdk uninative-tarball:do_populate_sdk"
+                                buildtools-tarball:do_populate_sdk uninative-tarball:do_populate_sdk \
+                                ${@'meta-world-pkgdata:do_collect_packagedata' if d.getVar('SDK_INCLUDE_PKGDATA', True) == '1' else ''}"
 
 do_populate_sdk_ext[rdepends] += "${@' '.join([x + ':do_build' for x in d.getVar('SDK_TARGETS', True).split()])}"
 
diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index a5ca3df..6475510 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -93,10 +93,64 @@ def prune_lockedsigs(excluded_tasks, excluded_targets, lockedsigs, pruned_output
                     invalue = True
                     f.write(line)
 
+def merge_lockedsigs(copy_tasks, lockedsigs_main, lockedsigs_extra, merged_output, copy_output):
+    merged = {}
+    arch_order = []
+    with open(lockedsigs_main, 'r') as f:
+        invalue = None
+        for line in f:
+            if invalue:
+                if line.endswith('\\\n'):
+                    merged[invalue].append(line)
+                else:
+                    invalue = None
+            elif line.startswith('SIGGEN_LOCKEDSIGS_t-'):
+                invalue = line[18:].split('=', 1)[0].rstrip()
+                merged[invalue] = []
+                arch_order.append(invalue)
+
+    with open(lockedsigs_extra, 'r') as f:
+        invalue = None
+        tocopy = {}
+        for line in f:
+            if invalue:
+                if line.endswith('\\\n'):
+                    if not line in merged[invalue]:
+                        target, task = line.strip().split(':')[:2]
+                        if task in copy_tasks:
+                            tocopy[invalue].append(line)
+                        merged[invalue].append(line)
+                else:
+                    invalue = None
+            elif line.startswith('SIGGEN_LOCKEDSIGS_t-'):
+                invalue = line[18:].split('=', 1)[0].rstrip()
+                if not invalue in merged:
+                    merged[invalue] = []
+                    arch_order.append(invalue)
+                tocopy[invalue] = []
+
+    def write_sigs_file(fn, types, sigs):
+        fulltypes = []
+        bb.utils.mkdirhier(os.path.dirname(fn))
+        with open(fn, 'w') as f:
+            for typename in types:
+                lines = sigs[typename]
+                if lines:
+                    f.write('SIGGEN_LOCKEDSIGS_%s = "\\\n' % typename)
+                    for line in lines:
+                        f.write(line)
+                    f.write('    "\n')
+                    fulltypes.append(typename)
+            f.write('SIGGEN_LOCKEDSIGS_TYPES = "%s"\n' % ' '.join(fulltypes))
+
+    write_sigs_file(copy_output, tocopy.keys(), tocopy)
+    write_sigs_file(merged_output, arch_order, merged)
+
 def create_locked_sstate_cache(lockedsigs, input_sstate_cache, output_sstate_cache, d, fixedlsbstring=""):
     bb.note('Generating sstate-cache...')
 
     bb.process.run("gen-lockedsig-cache %s %s %s" % (lockedsigs, input_sstate_cache, output_sstate_cache))
     if fixedlsbstring:
-        os.rename(output_sstate_cache + '/' + d.getVar('NATIVELSBSTRING', True),
-        output_sstate_cache + '/' + fixedlsbstring)
+        nativedir = output_sstate_cache + '/' + d.getVar('NATIVELSBSTRING', True)
+        if os.path.isdir(nativedir):
+            os.rename(nativedir, output_sstate_cache + '/' + fixedlsbstring)
diff --git a/meta/recipes-core/meta/meta-world-pkgdata.bb b/meta/recipes-core/meta/meta-world-pkgdata.bb
new file mode 100644
index 0000000..48a9027
--- /dev/null
+++ b/meta/recipes-core/meta/meta-world-pkgdata.bb
@@ -0,0 +1,50 @@
+SUMMARY = "Pulls in pkgdata for world"
+LICENSE = "MIT"
+INHIBIT_DEFAULT_DEPS = "1"
+
+addtask do_allpackagedata before do_build
+do_allpackagedata() {
+	:
+}
+do_allpackagedata[recrdeptask] = "do_packagedata do_allpackagedata"
+do_allpackagedata[noexec] = "1"
+
+WORLD_PKGDATADIR = "${D}/world-pkgdata"
+
+addtask do_collect_packagedata after do_allpackagedata
+SSTATETASKS += "do_collect_packagedata"
+do_collect_packagedata[sstate-inputdirs] = "${WORLD_PKGDATADIR}"
+do_collect_packagedata[sstate-outputdirs] = "${STAGING_DIR_HOST}/world-pkgdata"
+
+python do_collect_packagedata() {
+    import oe.copy_buildsystem
+    outdir = os.path.join(d.getVar('WORLD_PKGDATADIR', True))
+    bb.utils.mkdirhier(outdir)
+    sigfile = os.path.join(outdir, 'locked-sigs-pkgdata.inc')
+    oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
+}
+
+do_fetch[noexec] = "1"
+do_unpack[noexec] = "1"
+do_patch[noexec] = "1"
+do_configure[noexec] = "1"
+do_compile[noexec] = "1"
+do_install[noexec] = "1"
+
+do_configure[deptask] = ""
+
+WORLD_PKGDATA_EXCLUDE ?= "adt-installer"
+
+python calculate_extra_depends() {
+    exclude = '${WORLD_PKGDATA_EXCLUDE}'.split()
+    for p in world_target:
+        if p == self_pn:
+            continue
+
+        if p in exclude:
+            continue
+
+        deps.append(p)
+}
+
+PACKAGES = ""
diff --git a/scripts/lib/devtool/sdk.py b/scripts/lib/devtool/sdk.py
index 0872df6..68139aa 100644
--- a/scripts/lib/devtool/sdk.py
+++ b/scripts/lib/devtool/sdk.py
@@ -175,12 +175,12 @@ def sdk_update(args, config, basepath, workspace):
 
     if not args.skip_prepare:
         # Run bitbake command for the whole SDK
-        sdk_targets = config.get('SDK', 'sdk_targets')
+        sdk_update_targets = config.get('SDK', 'sdk_update_targets', config.get('SDK', 'sdk_targets'))
         logger.info("Preparing build system... (This may take some time.)")
         try:
-            exec_build_env_command(config.init_path, basepath, 'bitbake %s --setscene-only' % sdk_targets)
+            exec_build_env_command(config.init_path, basepath, 'bitbake %s --setscene-only' % sdk_update_targets)
         except:
-            logger.error('bitbake %s failed' % sdk_targets)
+            logger.error('bitbake %s failed' % sdk_update_targets)
             return -1
     return 0
 
-- 
2.5.0

[PATCH 02/14] devtool: build: ensure pkgdata is written out

[Paul Eggleton]

When you run devtool build, you need to have the pkgdata written out at
the end, so that if what you're adding is a library and the next thing
you add is something that depends on that library, the necessary
information to map the dependency back to the recipe is present. In
practical terms all this means is we need do_packagedata to run in
addition to do_populate_sysroot.

This does mean that do_package needs to run which wasn't running before,
and that means that the few package QA tests that run within do_package
such as installed-vs-shipped will now be run. This may be a bit
bothersome, and prompted a fix for one of our oe-selftest tests as a
result, but I don't see an easy way around it. Ultimately if you care
about using the recipe in an image you'll need to fix any such errors
anyway.

Fixes [YOCTO #8887].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/devtool.py |  7 ++++++-
 scripts/lib/devtool/build.py      | 16 +++++++++++-----
 2 files changed, 17 insertions(+), 6 deletions(-)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 43c7cda..d95cb08 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -226,8 +226,13 @@ class DevtoolTests(DevtoolBase):
         # Clean up anything in the workdir/sysroot/sstate cache (have to do this *after* devtool add since the recipe only exists then)
         bitbake('libftdi -c cleansstate')
         # libftdi's python/CMakeLists.txt is a bit broken, so let's just disable it
+        # There's also the matter of it installing cmake files to a path we don't
+        # normally cover, which triggers the installed-vs-shipped QA test we have
+        # within do_package
         recipefile = '%s/recipes/libftdi/libftdi_%s.bb' % (self.workspacedir, version)
-        result = runCmd('recipetool setvar %s EXTRA_OECMAKE -- "-DPYTHON_BINDINGS=OFF"' % recipefile)
+        result = runCmd('recipetool setvar %s EXTRA_OECMAKE -- \'-DPYTHON_BINDINGS=OFF -DLIBFTDI_CMAKE_CONFIG_DIR=${datadir}/cmake/Modules\'' % recipefile)
+        with open(recipefile, 'a') as f:
+            f.write('\nFILES_${PN}-dev += "${datadir}/cmake/Modules"\n')
         # Test devtool build
         result = runCmd('devtool build libftdi')
         staging_libdir = get_bb_var('STAGING_LIBDIR', 'libftdi')
diff --git a/scripts/lib/devtool/build.py b/scripts/lib/devtool/build.py
index c4c0c9f..b10a6a9 100644
--- a/scripts/lib/devtool/build.py
+++ b/scripts/lib/devtool/build.py
@@ -46,21 +46,27 @@ def _set_file_values(fn, values):
             f.writelines(newlines)
     return updated
 
-def _get_build_task(config):
-    return config.get('Build', 'build_task', 'populate_sysroot')
+def _get_build_tasks(config):
+    tasks = config.get('Build', 'build_task', 'populate_sysroot,packagedata').split(',')
+    return ['do_%s' % task.strip() for task in tasks]
 
 def build(args, config, basepath, workspace):
     """Entry point for the devtool 'build' subcommand"""
     workspacepn = check_workspace_recipe(workspace, args.recipename, bbclassextend=True)
 
-    build_task = _get_build_task(config)
+    build_tasks = _get_build_tasks(config)
 
     bbappend = workspace[workspacepn]['bbappend']
     if args.disable_parallel_make:
         logger.info("Disabling 'make' parallelism")
         _set_file_values(bbappend, {'PARALLEL_MAKE': ''})
     try:
-        exec_build_env_command(config.init_path, basepath, 'bitbake -c %s %s' % (build_task, args.recipename), watch=True)
+        bbargs = []
+        for task in build_tasks:
+            if args.recipename.endswith('-native') and 'package' in task:
+                continue
+            bbargs.append('%s:%s' % (args.recipename, task))
+        exec_build_env_command(config.init_path, basepath, 'bitbake %s' % ' '.join(bbargs), watch=True)
     except bb.process.ExecutionError as e:
         # We've already seen the output since watch=True, so just ensure we return something to the user
         return e.exitcode
@@ -73,7 +79,7 @@ def build(args, config, basepath, workspace):
 def register_commands(subparsers, context):
     """Register devtool subcommands from this plugin"""
     parser_build = subparsers.add_parser('build', help='Build a recipe',
-                                         description='Builds the specified recipe using bitbake (up to and including do_%s)' % _get_build_task(context.config))
+                                         description='Builds the specified recipe using bitbake (up to and including %s)' % ', '.join(_get_build_tasks(context.config)))
     parser_build.add_argument('recipename', help='Recipe to build')
     parser_build.add_argument('-s', '--disable-parallel-make', action="store_true", help='Disable make parallelism')
     parser_build.set_defaults(func=build)
-- 
2.5.0

[PATCH 03/14] devtool: sdk-update: improve temp directory handling

[Paul Eggleton]

* Use tempfile.mkdtemp() instead of hardcoding temp dir
* Set a variable early for the temp locked sigs file and use that
  everywhere
* Delete the temp dir at the end

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/sdk.py | 75 +++++++++++++++++++++++-----------------------
 1 file changed, 38 insertions(+), 37 deletions(-)

diff --git a/scripts/lib/devtool/sdk.py b/scripts/lib/devtool/sdk.py
index 68139aa..4fcd36a 100644
--- a/scripts/lib/devtool/sdk.py
+++ b/scripts/lib/devtool/sdk.py
@@ -7,6 +7,7 @@ import glob
 import shutil
 import errno
 import sys
+import tempfile
 from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
 
 logger = logging.getLogger('devtool')
@@ -133,45 +134,45 @@ def sdk_update(args, config, basepath, workspace):
             return ret
     else:
         # devtool sdk-update http://myhost/sdk
-        tmpsdk_dir = '/tmp/sdk-ext'
-        if os.path.exists(tmpsdk_dir):
-            shutil.rmtree(tmpsdk_dir)
-        os.makedirs(tmpsdk_dir)
-        os.makedirs(os.path.join(tmpsdk_dir, 'conf'))
-        # Fetch locked-sigs.inc from update server
-        ret = subprocess.call("wget -q -O - %s/conf/locked-sigs.inc > %s/locked-sigs.inc" % (updateserver, os.path.join(tmpsdk_dir, 'conf')), shell=True)
-        if ret != 0:
-            logger.error("Fetching conf/locked-sigs.inc from %s to %s/locked-sigs.inc failed" % (updateserver, os.path.join(tmpsdk_dir, 'conf')))
-            return ret
-        else:
-            logger.info("Fetching conf/locked-sigs.inc from %s to %s/locked-sigs.inc succeeded" % (updateserver, os.path.join(tmpsdk_dir, 'conf')))
-        new_locked_sig_file_path = os.path.join(tmpsdk_dir, 'conf/locked-sigs.inc')
-        update_dict = generate_update_dict(new_locked_sig_file_path, old_locked_sig_file_path)
-        logger.debug("update_dict = %s" % update_dict)
-        if len(update_dict) == 0:
-            logger.info("No need to update.")
-            return 0
-        # Update metadata
-        logger.debug("Updating meta data via git ...")
-        # Try using 'git pull', if failed, use 'git clone'
-        if os.path.exists(os.path.join(basepath, 'layers/.git')):
-            ret = subprocess.call("cd layers && git pull %s/layers/.git" % updateserver, shell=True)
-        else:
-            ret = -1
-        if ret != 0:
-            ret = subprocess.call("rm -rf layers && git clone %s/layers/.git" % updateserver, shell=True)
-        if ret != 0:
-            logger.error("Updating meta data via git failed")
-            return ret
-        logger.debug("Updating conf files ...")
-        conf_files = ['local.conf', 'bblayers.conf', 'devtool.conf', 'locked-sigs.inc']
-        for conf in conf_files:
-            ret = subprocess.call("wget -q -O - %s/conf/%s > conf/%s" % (updateserver, conf, conf), shell=True)
+        tmpsdk_dir = tempfile.mkdtemp()
+        try:
+            os.makedirs(os.path.join(tmpsdk_dir, 'conf'))
+            new_locked_sig_file_path = os.path.join(tmpsdk_dir, 'conf', 'locked-sigs.inc')
+            # Fetch locked-sigs.inc from update server
+            ret = subprocess.call("wget -q -O - %s/conf/locked-sigs.inc > %s" % (updateserver, new_locked_sig_file_path), shell=True)
             if ret != 0:
-                logger.error("Update %s failed" % conf)
+                logger.error("Fetching conf/locked-sigs.inc from %s to %s failed" % (updateserver, new_locked_sig_file_path))
                 return ret
-        with open(os.path.join(basepath, 'conf/local.conf'), 'a') as f:
-            f.write('SSTATE_MIRRORS_append = " file://.* %s/sstate-cache/PATH \\n "\n' % updateserver)
+            else:
+                logger.info("Fetching conf/locked-sigs.inc from %s to %s succeeded" % (updateserver, new_locked_sig_file_path))
+            update_dict = generate_update_dict(new_locked_sig_file_path, old_locked_sig_file_path)
+            logger.debug("update_dict = %s" % update_dict)
+            if len(update_dict) == 0:
+                logger.info("No need to update.")
+                return 0
+            # Update metadata
+            logger.debug("Updating meta data via git ...")
+            # Try using 'git pull', if failed, use 'git clone'
+            if os.path.exists(os.path.join(basepath, 'layers/.git')):
+                ret = subprocess.call("cd layers && git pull %s/layers/.git" % updateserver, shell=True)
+            else:
+                ret = -1
+            if ret != 0:
+                ret = subprocess.call("rm -rf layers && git clone %s/layers/.git" % updateserver, shell=True)
+            if ret != 0:
+                logger.error("Updating meta data via git failed")
+                return ret
+            logger.debug("Updating conf files ...")
+            conf_files = ['local.conf', 'bblayers.conf', 'devtool.conf', 'locked-sigs.inc']
+            for conf in conf_files:
+                ret = subprocess.call("wget -q -O - %s/conf/%s > conf/%s" % (updateserver, conf, conf), shell=True)
+                if ret != 0:
+                    logger.error("Update %s failed" % conf)
+                    return ret
+            with open(os.path.join(basepath, 'conf/local.conf'), 'a') as f:
+                f.write('SSTATE_MIRRORS_append = " file://.* %s/sstate-cache/PATH \\n "\n' % updateserver)
+        finally:
+            shutil.rmtree(tmpsdk_dir)
 
     if not args.skip_prepare:
         # Run bitbake command for the whole SDK
-- 
2.5.0

[PATCH 04/14] devtool: sdk-update: improve SDK update process robustness

[Paul Eggleton]

Make the following improvements to the SDK update process:

* Use a manifest file with sha256sums to track files other than sstate
  and metadata that we need to update - e.g. conf files. This allows us
  to handle where files such as auto.conf may or may not be present,
  as well as the configuration changing without affecting task signatures
  - we still want the config files copied in that case rather than it
  saying nothing needs to be done.
* Write the SSTATE_MIRRORS_append to site.conf rather than local.conf
  so that local.conf remains static (since we don't want to trigger an
  update every time). Also, If there is an SSTATE_MIRRORS value already
  set in the configuration we can skip this and assume it contains the
  needed packages.
* Allow the update process to be run in any directory, don't assume
  we're already at the base of the SDK
* Where practical, fetch remote files into a temporary location and
  then move them to the desired location at the end, to avoid a
  failed update leaving the SDK in a broken state.
* Update all installed do_populate_sysroot / do_packagedata tasks
  instead of using the SDK targets. This ensures any item installed
  through dependencies after installation (e.g. when running
  "devtool build") won't go stale.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass |  16 +++++
 scripts/lib/devtool/sdk.py            | 118 ++++++++++++++++++++++++----------
 2 files changed, 100 insertions(+), 34 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index ebb57bb..bb8831b 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -47,6 +47,7 @@ SDK_TITLE_task-populate-sdk-ext = "${@d.getVar('DISTRO_NAME', True) or d.getVar(
 python copy_buildsystem () {
     import re
     import shutil
+    import glob
     import oe.copy_buildsystem
 
     oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT', True)
@@ -224,6 +225,21 @@ python copy_buildsystem () {
             if name.endswith("_package.tgz"):
                 f = os.path.join(root, name)
                 os.remove(f)
+
+    # Write manifest file
+    # Note: at the moment we cannot include the env setup script here to keep
+    # it updated, since it gets modified during SDK installation (see
+    # sdk_ext_postinst() below) thus the checksum we take here would always
+    # be different.
+    manifest_file_list = ['conf/*']
+    manifest_file = os.path.join(baseoutpath, 'conf', 'sdk-conf-manifest')
+    with open(manifest_file, 'w') as f:
+        for item in manifest_file_list:
+            for fn in glob.glob(os.path.join(baseoutpath, item)):
+                if fn == manifest_file:
+                    continue
+                chksum = bb.utils.sha256_file(fn)
+                f.write('%s\t%s\n' % (chksum, os.path.relpath(fn, baseoutpath)))
 }
 
 def extsdk_get_buildtools_filename(d):
diff --git a/scripts/lib/devtool/sdk.py b/scripts/lib/devtool/sdk.py
index 4fcd36a..80ea871 100644
--- a/scripts/lib/devtool/sdk.py
+++ b/scripts/lib/devtool/sdk.py
@@ -75,6 +75,21 @@ def install_sstate_objects(sstate_objects, src_sdk, dest_sdk):
         logger.debug("Copying %s to %s" % (sb, dst))
         shutil.copy(sb, dst)
 
+def check_manifest(fn, basepath):
+    import bb.utils
+    changedfiles = []
+    with open(fn, 'r') as f:
+        for line in f:
+            splitline = line.split()
+            if len(splitline) > 1:
+                chksum = splitline[0]
+                fpath = splitline[1]
+                curr_chksum = bb.utils.sha256_file(os.path.join(basepath, fpath))
+                if chksum != curr_chksum:
+                    logger.debug('File %s changed: old csum = %s, new = %s' % (os.path.join(basepath, fpath), curr_chksum, chksum))
+                    changedfiles.append(fpath)
+    return changedfiles
+
 def sdk_update(args, config, basepath, workspace):
     # Fetch locked-sigs.inc file from remote/local destination
     updateserver = args.updateserver
@@ -98,6 +113,18 @@ def sdk_update(args, config, basepath, workspace):
     else:
         is_remote = False
 
+    layers_dir = os.path.join(basepath, 'layers')
+    conf_dir = os.path.join(basepath, 'conf')
+
+    # Grab variable values
+    tinfoil = setup_tinfoil(config_only=True, basepath=basepath)
+    try:
+        stamps_dir = tinfoil.config_data.getVar('STAMPS_DIR', True)
+        sstate_mirrors = tinfoil.config_data.getVar('SSTATE_MIRRORS', True)
+        site_conf_version = tinfoil.config_data.getVar('SITE_CONF_VERSION', True)
+    finally:
+        tinfoil.shutdown()
+
     if not is_remote:
         # devtool sdk-update /local/path/to/latest/sdk
         new_locked_sig_file_path = os.path.join(updateserver, 'conf/locked-sigs.inc')
@@ -121,16 +148,14 @@ def sdk_update(args, config, basepath, workspace):
         install_sstate_objects(sstate_objects, updateserver.rstrip('/'), basepath)
         logger.info("Updating configuration files")
         new_conf_dir = os.path.join(updateserver, 'conf')
-        old_conf_dir = os.path.join(basepath, 'conf')
-        shutil.rmtree(old_conf_dir)
-        shutil.copytree(new_conf_dir, old_conf_dir)
+        shutil.rmtree(conf_dir)
+        shutil.copytree(new_conf_dir, conf_dir)
         logger.info("Updating layers")
         new_layers_dir = os.path.join(updateserver, 'layers')
-        old_layers_dir = os.path.join(basepath, 'layers')
-        shutil.rmtree(old_layers_dir)
-        ret = subprocess.call("cp -a %s %s" % (new_layers_dir, old_layers_dir), shell=True)
+        shutil.rmtree(layers_dir)
+        ret = subprocess.call("cp -a %s %s" % (new_layers_dir, layers_dir), shell=True)
         if ret != 0:
-            logger.error("Copying %s to %s failed" % (new_layers_dir, old_layers_dir))
+            logger.error("Copying %s to %s failed" % (new_layers_dir, layers_dir))
             return ret
     else:
         # devtool sdk-update http://myhost/sdk
@@ -138,50 +163,75 @@ def sdk_update(args, config, basepath, workspace):
         try:
             os.makedirs(os.path.join(tmpsdk_dir, 'conf'))
             new_locked_sig_file_path = os.path.join(tmpsdk_dir, 'conf', 'locked-sigs.inc')
-            # Fetch locked-sigs.inc from update server
-            ret = subprocess.call("wget -q -O - %s/conf/locked-sigs.inc > %s" % (updateserver, new_locked_sig_file_path), shell=True)
-            if ret != 0:
-                logger.error("Fetching conf/locked-sigs.inc from %s to %s failed" % (updateserver, new_locked_sig_file_path))
-                return ret
-            else:
-                logger.info("Fetching conf/locked-sigs.inc from %s to %s succeeded" % (updateserver, new_locked_sig_file_path))
-            update_dict = generate_update_dict(new_locked_sig_file_path, old_locked_sig_file_path)
-            logger.debug("update_dict = %s" % update_dict)
-            if len(update_dict) == 0:
-                logger.info("No need to update.")
+            # Fetch manifest from server
+            tmpmanifest = os.path.join(tmpsdk_dir, 'conf', 'sdk-conf-manifest')
+            ret = subprocess.call("wget -q -O %s %s/conf/sdk-conf-manifest" % (tmpmanifest, updateserver), shell=True)
+            changedfiles = check_manifest(tmpmanifest, basepath)
+            if not changedfiles:
+                logger.info("Already up-to-date")
                 return 0
             # Update metadata
-            logger.debug("Updating meta data via git ...")
+            logger.debug("Updating metadata via git ...")
             # Try using 'git pull', if failed, use 'git clone'
             if os.path.exists(os.path.join(basepath, 'layers/.git')):
-                ret = subprocess.call("cd layers && git pull %s/layers/.git" % updateserver, shell=True)
+                ret = subprocess.call("git pull %s/layers/.git" % updateserver, shell=True, cwd=layers_dir)
             else:
                 ret = -1
             if ret != 0:
-                ret = subprocess.call("rm -rf layers && git clone %s/layers/.git" % updateserver, shell=True)
-            if ret != 0:
-                logger.error("Updating meta data via git failed")
-                return ret
+                ret = subprocess.call("git clone %s/layers/.git" % updateserver, shell=True, cwd=tmpsdk_dir)
+                if ret != 0:
+                    logger.error("Updating metadata via git failed")
+                    return ret
             logger.debug("Updating conf files ...")
-            conf_files = ['local.conf', 'bblayers.conf', 'devtool.conf', 'locked-sigs.inc']
-            for conf in conf_files:
-                ret = subprocess.call("wget -q -O - %s/conf/%s > conf/%s" % (updateserver, conf, conf), shell=True)
+            for changedfile in changedfiles:
+                ret = subprocess.call("wget -q -O %s %s/%s" % (changedfile, updateserver, changedfile), shell=True, cwd=tmpsdk_dir)
                 if ret != 0:
-                    logger.error("Update %s failed" % conf)
+                    logger.error("Updating %s failed" % changedfile)
                     return ret
-            with open(os.path.join(basepath, 'conf/local.conf'), 'a') as f:
-                f.write('SSTATE_MIRRORS_append = " file://.* %s/sstate-cache/PATH \\n "\n' % updateserver)
+
+            # Ok, all is well at this point - move everything over
+            tmplayers_dir = os.path.join(tmpsdk_dir, 'layers')
+            if os.path.exists(tmplayers_dir):
+                shutil.rmtree(layers_dir)
+                shutil.move(tmplayers_dir, layers_dir)
+            for changedfile in changedfiles:
+                destfile = os.path.join(basepath, changedfile)
+                os.remove(destfile)
+                shutil.move(os.path.join(tmpsdk_dir, changedfile), destfile)
+            os.remove(os.path.join(conf_dir, 'sdk-conf-manifest'))
+            shutil.move(tmpmanifest, conf_dir)
+
+            if not sstate_mirrors:
+                with open(os.path.join(conf_dir, 'site.conf'), 'a') as f:
+                    f.write('SCONF_VERSION = "%s"\n' % site_conf_version)
+                    f.write('SSTATE_MIRRORS_append = " file://.* %s/sstate-cache/PATH \\n "\n' % updateserver)
         finally:
             shutil.rmtree(tmpsdk_dir)
 
     if not args.skip_prepare:
+        # Find all potentially updateable tasks
+        sdk_update_targets = []
+        tasks = ['do_populate_sysroot', 'do_packagedata']
+        for root, _, files in os.walk(stamps_dir):
+            for fn in files:
+                if not '.sigdata.' in fn:
+                    for task in tasks:
+                        if '.%s.' % task in fn or '.%s_setscene.' % task in fn:
+                            sdk_update_targets.append('%s:%s' % (os.path.basename(root), task))
         # Run bitbake command for the whole SDK
-        sdk_update_targets = config.get('SDK', 'sdk_update_targets', config.get('SDK', 'sdk_targets'))
         logger.info("Preparing build system... (This may take some time.)")
         try:
-            exec_build_env_command(config.init_path, basepath, 'bitbake %s --setscene-only' % sdk_update_targets)
-        except:
-            logger.error('bitbake %s failed' % sdk_update_targets)
+            exec_build_env_command(config.init_path, basepath, 'bitbake --setscene-only %s' % ' '.join(sdk_update_targets), stderr=subprocess.STDOUT)
+            output, _ = exec_build_env_command(config.init_path, basepath, 'bitbake -n %s' % ' '.join(sdk_update_targets), stderr=subprocess.STDOUT)
+            runlines = []
+            for line in output.splitlines():
+                if 'Running task ' in line:
+                    runlines.append(line)
+            if runlines:
+                logger.error('Unexecuted tasks found in preparation log:\n  %s' % '\n  '.join(runlines))
+                return -1
+        except bb.process.ExecutionError as e:
+            logger.error('Preparation failed:\n%s' % e.stdout)
             return -1
     return 0
 
-- 
2.5.0

[PATCH 05/14] devtool: add sdk-install subcommand

[Paul Eggleton]

Add the ability to install additional pre-built items (from shared
state) into the extensible SDK. This can already be done implicitly by
adding something to DEPENDS within a recipe you're working on and then
running "devtool build", but it's useful to be able to explicitly
install things particularly if you're using the extensible SDK as a
traditional toolchain.

Note that for this command to be useful you need to have SSTATE_MIRRORS
set in your SDK configuration, and that mirror needs to be populated
with sstate artifacts for recipes you wish to be able to install.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/sdk.py | 63 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 62 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/sdk.py b/scripts/lib/devtool/sdk.py
index 80ea871..12de942 100644
--- a/scripts/lib/devtool/sdk.py
+++ b/scripts/lib/devtool/sdk.py
@@ -8,7 +8,7 @@ import shutil
 import errno
 import sys
 import tempfile
-from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
+from devtool import exec_build_env_command, setup_tinfoil, parse_recipe, DevtoolError
 
 logger = logging.getLogger('devtool')
 
@@ -235,6 +235,64 @@ def sdk_update(args, config, basepath, workspace):
             return -1
     return 0
 
+def sdk_install(args, config, basepath, workspace):
+    """Entry point for the devtool sdk-install command"""
+
+    import oe.recipeutils
+    import bb.process
+
+    for recipe in args.recipename:
+        if recipe in workspace:
+            raise DevtoolError('recipe %s is a recipe in your workspace' % recipe)
+
+    tasks = ['do_populate_sysroot', 'do_packagedata']
+    stampprefixes = {}
+    def checkstamp(recipe):
+        stampprefix = stampprefixes[recipe]
+        stamps = glob.glob(stampprefix + '*')
+        for stamp in stamps:
+            if '.sigdata.' not in stamp and stamp.startswith((stampprefix + '.', stampprefix + '_setscene.')):
+                return True
+        else:
+            return False
+
+    install_recipes = []
+    tinfoil = setup_tinfoil(config_only=False, basepath=basepath)
+    try:
+        for recipe in args.recipename:
+            rd = parse_recipe(config, tinfoil, recipe, True)
+            if not rd:
+                return 1
+            stampprefixes[recipe] = '%s.%s' % (rd.getVar('STAMP', True), tasks[0])
+            if checkstamp(recipe):
+                logger.info('%s is already installed' % recipe)
+            else:
+                install_recipes.append(recipe)
+    finally:
+        tinfoil.shutdown()
+
+    if install_recipes:
+        logger.info('Installing %s...' % ', '.join(install_recipes))
+        install_tasks = []
+        for recipe in install_recipes:
+            for task in tasks:
+                if recipe.endswith('-native') and 'package' in task:
+                    continue
+                install_tasks.append('%s:%s' % (recipe, task))
+        try:
+            exec_build_env_command(config.init_path, basepath, 'bitbake --setscene-only %s' % ' '.join(install_tasks))
+        except bb.process.ExecutionError as e:
+            raise DevtoolError('Failed to install %s:\n%s' % (recipe, str(e)))
+        failed = False
+        for recipe in install_recipes:
+            if checkstamp(recipe):
+                logger.info('Successfully installed %s' % recipe)
+            else:
+                raise DevtoolError('Failed to install %s - unavailable' % recipe)
+                failed = True
+        if failed:
+            return 2
+
 def register_commands(subparsers, context):
     """Register devtool subcommands from the sdk plugin"""
     if context.fixed_setup:
@@ -242,3 +300,6 @@ def register_commands(subparsers, context):
         parser_sdk.add_argument('updateserver', help='The update server to fetch latest SDK components from', nargs='?')
         parser_sdk.add_argument('--skip-prepare', action="store_true", help='Skip re-preparing the build system after updating (for debugging only)')
         parser_sdk.set_defaults(func=sdk_update)
+        parser_sdk_install = subparsers.add_parser('sdk-install', help='Install additional SDK components', description='Installs additional recipe development files into the SDK. (You can use "devtool search" to find available recipes.)')
+        parser_sdk_install.add_argument('recipename', help='Name of the recipe to install the development artifacts for', nargs='+')
+        parser_sdk_install.set_defaults(func=sdk_install)
-- 
2.5.0

[PATCH 06/14] classes/populate_sdk_ext: drop ext-sdk-prepare.py when installing

[Paul Eggleton]

At the end of the extensible SDK installation, if we've successfully
prepared the build system then we don't need ext-sdk-prepare.py. I had
thought earlier that this would be used when updating, but a different
mechanism was needed there so this script isn't used for that.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index bb8831b..2944576 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -304,6 +304,7 @@ sdk_ext_postinst() {
 		# sourcing a script. That is why this has to look so ugly.
 		LOGFILE="$target_sdk_dir/preparing_build_system.log"
 		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py '${SDK_INSTALL_TARGETS}' >> $LOGFILE 2>&1" || { echo "ERROR: SDK preparation failed: see $LOGFILE"; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
+		rm $target_sdk_dir/ext-sdk-prepare.py
 	fi
 	echo done
 }
-- 
2.5.0

[PATCH 07/14] classes/populate_sdk*: add dependencies on script files

[Paul Eggleton]

If the script files we use to construct the SDK installer change then
that really ought to trigger re-execution of the do_populate_sdk(_ext)
task, so add file-checksums varflags to ensure that happens.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_base.bbclass | 3 +++
 meta/classes/populate_sdk_ext.bbclass  | 5 +++++
 2 files changed, 8 insertions(+)

diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
index 99b64f7..966161f 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -243,6 +243,9 @@ populate_sdk_log_check() {
 	done
 }
 
+do_populate_sdk[file-checksums] += "${COREBASE}/meta/files/toolchain-shar-relocate.sh:True \
+                                    ${COREBASE}/meta/files/toolchain-shar-extract.sh:True"
+
 do_populate_sdk[dirs] = "${PKGDATA_DIR} ${TOPDIR}"
 do_populate_sdk[depends] += "${@' '.join([x + ':do_populate_sysroot' for x in d.getVar('SDK_DEPENDS', True).split()])}  ${@d.getVarFlag('do_rootfs', 'depends', False)}"
 do_populate_sdk[rdepends] = "${@' '.join([x + ':do_populate_sysroot' for x in d.getVar('SDK_RDEPENDS', True).split()])}"
diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 2944576..a115127 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -259,6 +259,7 @@ install_tools() {
 
 	install -m 0644 ${COREBASE}/meta/files/ext-sdk-prepare.py ${SDK_OUTPUT}/${SDKPATH}
 }
+do_populate_sdk_ext[file-checksums] += "${COREBASE}/meta/files/ext-sdk-prepare.py:True"
 
 # Since bitbake won't run as root it doesn't make sense to try and install
 # the extensible sdk as root.
@@ -358,4 +359,8 @@ do_populate_sdk_ext[rdepends] += "${@' '.join([x + ':do_build' for x in d.getVar
 # Make sure codes change in copy_buildsystem can result in rebuilt
 do_populate_sdk_ext[vardeps] += "copy_buildsystem"
 
+do_populate_sdk_ext[file-checksums] += "${COREBASE}/meta/files/toolchain-shar-relocate.sh:True \
+                                        ${COREBASE}/meta/files/toolchain-shar-extract.sh:True \
+                                        ${COREBASE}/scripts/gen-lockedsig-cache:True"
+
 addtask populate_sdk_ext after do_sdk_depends
-- 
2.5.0

[PATCH 08/14] toolchain-shar-extract.sh: improve behaviour when xz is not installed

[Paul Eggleton]

Currently if xz isn't installed when installing the SDK you get an error
from tar and then the SDK installation continues, cascading into a load
of other failures since files are missing. Add a check to see if xz can
be run and error out if it can't, and also exit if tar fails.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/files/toolchain-shar-extract.sh | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/meta/files/toolchain-shar-extract.sh b/meta/files/toolchain-shar-extract.sh
index b1ef4b7..d844771 100644
--- a/meta/files/toolchain-shar-extract.sh
+++ b/meta/files/toolchain-shar-extract.sh
@@ -29,6 +29,11 @@ if [ "$INST_ARCH" != "$SDK_ARCH" ]; then
 	fi
 fi
 
+if ! xz -V > /dev/null 2>&1; then
+	echo "Error: xz is required for installation of this SDK, please install it first"
+	exit 1
+fi
+
 DEFAULT_INSTALL_DIR="@SDKPATH@"
 SUDO_EXEC=""
 target_sdk_dir=""
@@ -168,7 +173,7 @@ fi
 payload_offset=$(($(grep -na -m1 "^MARKER:$" $0|cut -d':' -f1) + 1))
 
 printf "Extracting SDK..."
-tail -n +$payload_offset $0| $SUDO_EXEC tar xJ -C $target_sdk_dir --checkpoint=.2500
+tail -n +$payload_offset $0| $SUDO_EXEC tar xJ -C $target_sdk_dir --checkpoint=.2500 || exit 1
 echo "done"
 
 printf "Setting it up..."
-- 
2.5.0

[PATCH 09/14] gen-lockedsig-cache: copy correct native sstate into ext SDK

[Paul Eggleton]

When constructing the sstate-cache directory for the extensible SDK,
we were copying in any matching native sstate packages, and as the
signature doesn't actually change when the distro changes (since
NATIVELSBSTRING is just a path separator for the artifacts and is not
part of the signature) we ended up copying duplicated packages when the
distro changed e.g. upon host distro upgrade. Only search in the
NATIVELSBSTRING-named subdirectory for native packages and the issue
goes away.

Fixes [YOCTO #8885].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/copy_buildsystem.py | 5 +++--
 scripts/gen-lockedsig-cache     | 6 +++---
 2 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index 6475510..fb51b51 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -149,8 +149,9 @@ def merge_lockedsigs(copy_tasks, lockedsigs_main, lockedsigs_extra, merged_outpu
 def create_locked_sstate_cache(lockedsigs, input_sstate_cache, output_sstate_cache, d, fixedlsbstring=""):
     bb.note('Generating sstate-cache...')
 
-    bb.process.run("gen-lockedsig-cache %s %s %s" % (lockedsigs, input_sstate_cache, output_sstate_cache))
+    nativelsbstring = d.getVar('NATIVELSBSTRING', True)
+    bb.process.run("gen-lockedsig-cache %s %s %s %s" % (lockedsigs, input_sstate_cache, output_sstate_cache, nativelsbstring))
     if fixedlsbstring:
-        nativedir = output_sstate_cache + '/' + d.getVar('NATIVELSBSTRING', True)
+        nativedir = output_sstate_cache + '/' + nativelsbstring
         if os.path.isdir(nativedir):
             os.rename(nativedir, output_sstate_cache + '/' + fixedlsbstring)
diff --git a/scripts/gen-lockedsig-cache b/scripts/gen-lockedsig-cache
index 9c16506..a4e9ded 100755
--- a/scripts/gen-lockedsig-cache
+++ b/scripts/gen-lockedsig-cache
@@ -13,9 +13,9 @@ def mkdir(d):
         if e.errno != errno.EEXIST:
             raise e
 
-if len(sys.argv) < 3:
+if len(sys.argv) < 5:
     print("Incorrect number of arguments specified")
-    print("syntax: gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir>")
+    print("syntax: gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir> <nativelsbstring>")
     sys.exit(1)
 
 print('Reading %s' % sys.argv[1])
@@ -30,7 +30,7 @@ files = set()
 for s in sigs:
     p = sys.argv[2] + "/" + s[:2] + "/*" + s + "*"
     files |= set(glob.glob(p))
-    p = sys.argv[2] + "/*/" + s[:2] + "/*" + s + "*"
+    p = sys.argv[2] + "/%s/" % sys.argv[4] + s[:2] + "/*" + s + "*"
     files |= set(glob.glob(p))
 
 print('Processing files')
-- 
2.5.0

[PATCH 10/14] recipetool: create: strip quotes from values extracted from CMakeLists.txt

[Paul Eggleton]

Quoting is optional in CMakeLists.txt and is occasionally used, so strip
out quotes if they are present.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create_buildsys.py | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
index 6afb5de..ba393a8 100644
--- a/scripts/lib/recipetool/create_buildsys.py
+++ b/scripts/lib/recipetool/create_buildsys.py
@@ -142,6 +142,9 @@ class CmakeRecipeHandler(RecipeHandler):
         subdir_re = re.compile('add_subdirectory\s*\(\s*([^)\s]*)\s*([^)\s]*)\s*\)', re.IGNORECASE)
         dep_re = re.compile('([^ ><=]+)( *[<>=]+ *[^ ><=]+)?')
 
+        def interpret_value(value):
+            return value.strip('"')
+
         def parse_cmake_file(fn, paths=None):
             searchpaths = (paths or []) + [os.path.dirname(fn)]
             logger.debug('Parsing file %s' % fn)
@@ -166,13 +169,13 @@ class CmakeRecipeHandler(RecipeHandler):
                         continue
                     res = proj_re.match(line)
                     if res:
-                        extravalues['PN'] = res.group(1).split()[0]
+                        extravalues['PN'] = interpret_value(res.group(1).split()[0])
                         continue
                     res = pkgcm_re.match(line)
                     if res:
                         res = dep_re.findall(res.group(2))
                         if res:
-                            pcdeps.extend([x[0] for x in res])
+                            pcdeps.extend([interpret_value(x[0]) for x in res])
                         inherits.append('pkgconfig')
                         continue
                     res = pkgsm_re.match(line)
@@ -180,7 +183,7 @@ class CmakeRecipeHandler(RecipeHandler):
                         res = dep_re.findall(res.group(2))
                         if res:
                             # Note: appending a tuple here!
-                            item = tuple((x[0] for x in res))
+                            item = tuple((interpret_value(x[0]) for x in res))
                             if len(item) == 1:
                                 item = item[0]
                             pcdeps.append(item)
@@ -189,7 +192,7 @@ class CmakeRecipeHandler(RecipeHandler):
                     res = findpackage_re.match(line)
                     if res:
                         origpkg = res.group(1)
-                        pkg = origpkg.lower()
+                        pkg = interpret_value(origpkg.lower())
                         if pkg == 'gettext':
                             inherits.append('gettext')
                         elif pkg == 'perl':
@@ -209,7 +212,7 @@ class CmakeRecipeHandler(RecipeHandler):
                         continue
                     res = checklib_re.match(line)
                     if res:
-                        lib = res.group(1)
+                        lib = interpret_value(res.group(1))
                         if not lib.startswith('$'):
                             libdeps.append(lib)
                     if line.lower().startswith('useswig'):
-- 
2.5.0

[PATCH 11/14] devtool / recipetool: support specifying a subdirectory within the fetched source

[Paul Eggleton]

Sometimes you don't want to build an entire project, just a subdirectory
of it; add a --src-subdir option to make that easier. (We still look for
a single subdirectory in what gets unpacked, e.g. what you might find
within a tarball, so whatever you specify with this option is added onto
the end of that.)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py  |  6 ++++++
 scripts/lib/recipetool/create.py | 11 +++++++++--
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 5390f51..f19de27 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -141,6 +141,8 @@ def add(args, config, basepath, workspace):
         extracmdopts += ' -b'
     if args.also_native:
         extracmdopts += ' --also-native'
+    if args.src_subdir:
+        extracmdopts += ' --src-subdir "%s"' % args.src_subdir
 
     tempdir = tempfile.mkdtemp(prefix='devtool')
     try:
@@ -208,6 +210,9 @@ def add(args, config, basepath, workspace):
     if not rd:
         return 1
 
+    if args.src_subdir:
+        srctree = os.path.join(srctree, args.src_subdir)
+
     bb.utils.mkdirhier(os.path.dirname(appendfile))
     with open(appendfile, 'w') as f:
         f.write('inherit externalsrc\n')
@@ -1308,6 +1313,7 @@ def register_commands(subparsers, context):
     parser_add.add_argument('--no-git', '-g', help='If fetching source, do not set up source tree as a git repository', action="store_true")
     parser_add.add_argument('--binary', '-b', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure). Useful with binary packages e.g. RPMs.', action='store_true')
     parser_add.add_argument('--also-native', help='Also add native variant (i.e. support building recipe for the build host as well as the target machine)', action='store_true')
+    parser_add.add_argument('--src-subdir', help='Specify subdirectory within source tree to use', metavar='SUBDIR')
     parser_add.set_defaults(func=add)
 
     parser_modify = subparsers.add_parser('modify', help='Modify the source for an existing recipe',
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 43861ee..9c3a63d 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -324,6 +324,12 @@ def create_recipe(args):
         srcuri = ''
         srctree = args.source
 
+    if args.src_subdir:
+        srcsubdir = os.path.join(srcsubdir, args.src_subdir)
+        srctree_use = os.path.join(srctree, args.src_subdir)
+    else:
+        srctree_use = srctree
+
     if args.outfile and os.path.isdir(args.outfile):
         outfile = None
         outdir = args.outfile
@@ -343,7 +349,7 @@ def create_recipe(args):
     lines_before.append('# (Feel free to remove these comments when editing.)')
     lines_before.append('#')
 
-    licvalues = guess_license(srctree)
+    licvalues = guess_license(srctree_use)
     lic_files_chksum = []
     if licvalues:
         licenses = []
@@ -472,7 +478,7 @@ def create_recipe(args):
 
     extravalues = {}
     for handler in handlers:
-        handler.process(srctree, classes, lines_before, lines_after, handled, extravalues)
+        handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
 
     if not realpv:
         realpv = extravalues.get('PV', None)
@@ -759,5 +765,6 @@ def register_commands(subparsers):
     parser_create.add_argument('-V', '--version', help='Version to use within recipe (PV)')
     parser_create.add_argument('-b', '--binary', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure)', action='store_true')
     parser_create.add_argument('--also-native', help='Also add native variant (i.e. support building recipe for the build host as well as the target machine)', action='store_true')
+    parser_create.add_argument('--src-subdir', help='Specify subdirectory within source tree to use', metavar='SUBDIR')
     parser_create.set_defaults(func=create_recipe)
 
-- 
2.5.0

[PATCH 12/14] recipetool: create: extract SRC_URI from local git repositories

[Paul Eggleton]

If you specify a local directory which happens to be a git repository
with an origin remote (and it is in fact remote), we can use that for
SRC_URI rather than leaving it blank in the recipe.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 30 +++++++++++++++++++++++-------
 1 file changed, 23 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 9c3a63d..5caf374 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -260,6 +260,14 @@ def supports_srcrev(uri):
             return True
     return False
 
+def reformat_git_uri(uri):
+    '''Convert any http[s]://....git URI into git://...;protocol=http[s]'''
+    res = re.match('(https?)://([^;]+\.git)(;.*)?$', uri)
+    if res:
+        # Need to switch the URI around so that the git fetcher is used
+        return 'git://%s;protocol=%s%s' % (res.group(2), res.group(1), res.group(3) or '')
+    return uri
+
 def create_recipe(args):
     import bb.process
     import tempfile
@@ -275,16 +283,11 @@ def create_recipe(args):
     srcrev = '${AUTOREV}'
     if '://' in args.source:
         # Fetch a URL
-        fetchuri = urlparse.urldefrag(args.source)[0]
+        fetchuri = reformat_git_uri(urlparse.urldefrag(args.source)[0])
         if args.binary:
             # Assume the archive contains the directory structure verbatim
             # so we need to extract to a subdirectory
             fetchuri += ';subdir=%s' % os.path.splitext(os.path.basename(urlparse.urlsplit(fetchuri).path))[0]
-        git_re = re.compile('(https?)://([^;]+\.git)(;.*)?$')
-        res = git_re.match(fetchuri)
-        if res:
-            # Need to switch the URI around so that the git fetcher is used
-            fetchuri = 'git://%s;protocol=%s%s' % (res.group(2), res.group(1), res.group(3) or '')
         srcuri = fetchuri
         rev_re = re.compile(';rev=([^;]+)')
         res = rev_re.search(srcuri)
@@ -321,8 +324,21 @@ def create_recipe(args):
         if not os.path.isdir(args.source):
             logger.error('Invalid source directory %s' % args.source)
             sys.exit(1)
-        srcuri = ''
         srctree = args.source
+        srcuri = ''
+        if os.path.exists(os.path.join(srctree, '.git')):
+            # Try to get upstream repo location from origin remote
+            try:
+                stdout, _ = bb.process.run('git remote -v', cwd=srctree, shell=True)
+            except bb.process.ExecutionError as e:
+                stdout = None
+            if stdout:
+                for line in stdout.splitlines():
+                    splitline = line.split()
+                    if len(splitline) > 1:
+                        if splitline[0] == 'origin' and '://' in splitline[1]:
+                            srcuri = reformat_git_uri(splitline[1])
+                            break
 
     if args.src_subdir:
         srcsubdir = os.path.join(srcsubdir, args.src_subdir)
-- 
2.5.0

[PATCH 13/14] recipetool: create: fix extraction of name from URLs ending in /

[Paul Eggleton]

If the URL ends in a / then we want to strip that off the path we split
out of the URL before calling os.path.basename() on it.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 5caf374..f0bb58e 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -432,7 +432,7 @@ def create_recipe(args):
     if srcuri and not realpv or not pn:
         parseres = urlparse.urlparse(srcuri)
         if parseres.path:
-            srcfile = os.path.basename(parseres.path)
+            srcfile = os.path.basename(parseres.path.rstrip('/'))
             name_pn, name_pv = determine_from_filename(srcfile)
             logger.debug('Determined from filename: name = "%s", version = "%s"' % (name_pn, name_pv))
             if name_pn and not pn:
-- 
2.5.0

[PATCH 14/14] recipetool: create: better fix for fetch error handling

[Paul Eggleton]

I was a little bit hasty in OE-Core revision
c2cc5abe34169eae92067d97ce1e747e7c1413f5 - it turns out BitBake's
fetcher code is not consistent in whether it logs something useful or
not; when fetching from an http URL it does but with a git repository
it doesn't. In advance of any major reworking of fetch error handling in
BitBake, let's just print the text of the exception and then we know we
have shown something to the user.

Additionally, we were only catching FetchException here but there are
several other classes of exception that the fetcher can raise (e.g.
MalformedUrl); catch the parent BBFetchException class instead so we
avoid tracebacks for those other classes as well.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index f0bb58e..5f90b10 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -299,8 +299,8 @@ def create_recipe(args):
         logger.info('Fetching %s...' % srcuri)
         try:
             checksums = scriptutils.fetch_uri(tinfoil.config_data, fetchuri, srctree, srcrev)
-        except bb.fetch2.FetchError:
-            # Error already printed
+        except bb.fetch2.BBFetchException as e:
+            logger.error(str(e).rstrip())
             sys.exit(1)
         dirlist = os.listdir(srctree)
         if 'git.indirectionsymlink' in dirlist:
-- 
2.5.0