[PATCH 0/4] Extensible SDK fixes

[PATCH 0/4] Extensible SDK fixes

[Paul Eggleton]

The following changes since commit 541315d6c56df6448f64c262f99d43d5c1e9400b:

  update_font_cache: only scan system font directories (2016-01-11 23:23:18 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/extsdkfixes4-oe
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/extsdkfixes4-oe

Paul Eggleton (4):
  classes/populate_sdk_ext.bbclass: handle if local.conf doesn't end with a newline
  classes/populate_sdk_ext: support auto.conf
  classes/buildhistory: save auto.conf and bblayers.conf for extensible SDK
  classes/populate_sdk_ext: check that extensible SDK prepared correctly

 meta/classes/buildhistory.bbclass     |  2 +-
 meta/classes/populate_sdk_ext.bbclass | 20 ++++++--
 meta/files/ext-sdk-prepare.py         | 92 +++++++++++++++++++++++++++++++++++
 meta/files/ext-sdk-prepare.sh         |  8 ---
 4 files changed, 110 insertions(+), 12 deletions(-)
 create mode 100644 meta/files/ext-sdk-prepare.py
 delete mode 100644 meta/files/ext-sdk-prepare.sh

-- 
2.5.0

[PATCH 1/4] classes/populate_sdk_ext.bbclass: handle if local.conf doesn't end with a newline

[Paul Eggleton]

If there is no newline at the end of local.conf, appending
INHERIT = "uninative" won't work, it will corrupt the line and the
installed eSDK will build things, making the "Preparing build system..."
step take an age.

Fixes [YOCTO #8897].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 87fb767..984f538 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -142,6 +142,8 @@ python copy_buildsystem () {
         for line in newlines:
             if line.strip() and not line.startswith('#'):
                 f.write(line)
+        # Write a newline just in case there's none at the end of the original
+        f.write('\n')
 
         f.write('INHERIT += "%s"\n\n' % 'uninative')
         f.write('CONF_VERSION = "%s"\n\n' % d.getVar('CONF_VERSION', False))
-- 
2.5.0

[PATCH 2/4] classes/populate_sdk_ext: support auto.conf

[Paul Eggleton]

If auto.conf exists in the user's configuration we need to also run it
through the same filter and write the result into the ext SDK, or we
risk missing configuration applied on an autobuilder.

Fixes [YOCTO #8904].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 984f538..69e13ab 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -169,6 +169,19 @@ python copy_buildsystem () {
 
         f.write('require conf/locked-sigs.inc\n')
 
+    if os.path.exists(builddir + '/conf/auto.conf'):
+        with open(builddir + '/conf/auto.conf', 'r') as f:
+            oldlines = f.readlines()
+        (updated, newlines) = bb.utils.edit_metadata(oldlines, varlist, handle_var)
+        with open(baseoutpath + '/conf/auto.conf', 'w') as f:
+            f.write('# WARNING: this configuration has been automatically generated and in\n')
+            f.write('# most cases should not be edited. If you need more flexibility than\n')
+            f.write('# this configuration provides, it is strongly suggested that you set\n')
+            f.write('# up a proper instance of the full build system and use that instead.\n\n')
+            for line in newlines:
+                if line.strip() and not line.startswith('#'):
+                    f.write(line)
+
     sigfile = d.getVar('WORKDIR', True) + '/locked-sigs.inc'
     oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
 
-- 
2.5.0

[PATCH 3/4] classes/buildhistory: save auto.conf and bblayers.conf for extensible SDK

[Paul Eggleton]

These form part of the configuration for the extensible SDK, we should
really be recording what goes into them.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 4153e58..2c144ab 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -13,7 +13,7 @@ BUILDHISTORY_DIR_IMAGE = "${BUILDHISTORY_DIR}/images/${MACHINE_ARCH}/${TCLIBC}/$
 BUILDHISTORY_DIR_PACKAGE = "${BUILDHISTORY_DIR}/packages/${MULTIMACH_TARGET_SYS}/${PN}"
 BUILDHISTORY_DIR_SDK = "${BUILDHISTORY_DIR}/sdk/${SDK_NAME}${SDK_EXT}/${IMAGE_BASENAME}"
 BUILDHISTORY_IMAGE_FILES ?= "/etc/passwd /etc/group"
-BUILDHISTORY_SDK_FILES ?= "conf/local.conf conf/locked-sigs.inc conf/devtool.conf"
+BUILDHISTORY_SDK_FILES ?= "conf/local.conf conf/bblayers.conf conf/auto.conf conf/locked-sigs.inc conf/devtool.conf"
 BUILDHISTORY_COMMIT ?= "0"
 BUILDHISTORY_COMMIT_AUTHOR ?= "buildhistory <buildhistory@${DISTRO}>"
 BUILDHISTORY_PUSH_REPO ?= ""
-- 
2.5.0

[PATCH 4/4] classes/populate_sdk_ext: check that extensible SDK prepared correctly

[Paul Eggleton]

After the change to use --setscene-only when running bitbake to prepare
the SDK at the end of installation, add a check that the SDK got
prepared correctly by doing a dry-run and looking at the output for any
real tasks that we don't expect. In order to make this easier, the
preparation shell script was rewritten in python.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass |  5 +-
 meta/files/ext-sdk-prepare.py         | 92 +++++++++++++++++++++++++++++++++++
 meta/files/ext-sdk-prepare.sh         |  8 ---
 3 files changed, 94 insertions(+), 11 deletions(-)
 create mode 100644 meta/files/ext-sdk-prepare.py
 delete mode 100644 meta/files/ext-sdk-prepare.sh

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 69e13ab..4d8d2a6 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -224,7 +224,7 @@ install_tools() {
 
 	install ${SDK_DEPLOY}/${BUILD_ARCH}-nativesdk-libc.tar.bz2 ${SDK_OUTPUT}/${SDKPATH}
 
-	install -m 0755 ${COREBASE}/meta/files/ext-sdk-prepare.sh ${SDK_OUTPUT}/${SDKPATH}
+	install -m 0644 ${COREBASE}/meta/files/ext-sdk-prepare.py ${SDK_OUTPUT}/${SDKPATH}
 }
 
 # Since bitbake won't run as root it doesn't make sense to try and install
@@ -270,9 +270,8 @@ sdk_ext_postinst() {
 		# current working directory when first ran, nor will it set $1 when
 		# sourcing a script. That is why this has to look so ugly.
 		LOGFILE="$target_sdk_dir/preparing_build_system.log"
-		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && $target_sdk_dir/ext-sdk-prepare.sh $target_sdk_dir '${SDK_TARGETS}' >> $LOGFILE 2>&1" || { echo "ERROR: SDK preparation failed: see $LOGFILE"; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
+		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py '${SDK_TARGETS}' >> $LOGFILE 2>&1" || { echo "ERROR: SDK preparation failed: see $LOGFILE"; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
 	fi
-	rm -f $target_sdk_dir/ext-sdk-prepare.sh
 	echo done
 }
 
diff --git a/meta/files/ext-sdk-prepare.py b/meta/files/ext-sdk-prepare.py
new file mode 100644
index 0000000..143e0fe
--- /dev/null
+++ b/meta/files/ext-sdk-prepare.py
@@ -0,0 +1,92 @@
+#!/usr/bin/env python
+
+# Prepare the build system within the extensible SDK
+
+import sys
+import os
+import subprocess
+
+def exec_watch(cmd, **options):
+    """Run program with stdout shown on sys.stdout"""
+    if isinstance(cmd, basestring) and not "shell" in options:
+        options["shell"] = True
+
+    process = subprocess.Popen(
+        cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, **options
+    )
+
+    buf = ''
+    while True:
+        out = process.stdout.read(1)
+        if out:
+            sys.stdout.write(out)
+            sys.stdout.flush()
+            buf += out
+        elif out == '' and process.poll() != None:
+            break
+
+    return process.returncode, buf
+
+
+def main():
+    if len(sys.argv) < 2:
+        print('Please specify target to prepare with')
+        return 1
+
+    sdk_targets = ' '.join(sys.argv[1:]).split()
+    print('Preparing SDK for %s...' % ', '.join(sdk_targets))
+
+    ret, out = exec_watch('bitbake %s --setscene-only' % ' '.join(sdk_targets))
+    if ret:
+        return ret
+
+    targetlist = []
+    for target in sdk_targets:
+        if ':' in target:
+            target = target.split(':')[0]
+        if not target in targetlist:
+            targetlist.append(target)
+
+    recipes = []
+    for target in targetlist:
+        try:
+            out = subprocess.check_output(('bitbake -e %s' % target).split(), stderr=subprocess.STDOUT)
+            for line in out.splitlines():
+                if line.startswith('FILE='):
+                    splitval = line.rstrip().split('=')
+                    if len(splitval) > 1:
+                        recipes.append(splitval[1].strip('"'))
+                    break
+        except subprocess.CalledProcessError as e:
+            print('ERROR: Failed to get recipe for target %s:\n%s' % (target, e.output))
+            return 1
+
+    try:
+        out = subprocess.check_output('bitbake %s -n' % ' '.join(sdk_targets), stderr=subprocess.STDOUT, shell=True)
+        unexpected = []
+        for line in out.splitlines():
+            if 'Running task' in line:
+                for recipe in recipes:
+                    if recipe in line:
+                        break
+                else:
+                    line = line.split('Running', 1)[-1]
+                    unexpected.append(line.rstrip())
+    except subprocess.CalledProcessError as e:
+        print('ERROR: Failed to execute dry-run:\n%s' % e.output)
+        return 1
+
+    if unexpected:
+        print('ERROR: Unexpected tasks left over to be executed:')
+        for line in unexpected:
+            print('  ' + line)
+        return 1
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+        traceback.print_exc(5)
+    sys.exit(ret)
diff --git a/meta/files/ext-sdk-prepare.sh b/meta/files/ext-sdk-prepare.sh
deleted file mode 100644
index b3f5d93..0000000
--- a/meta/files/ext-sdk-prepare.sh
+++ /dev/null
@@ -1,8 +0,0 @@
-#!/bin/sh
-
-# Prepare the build system within the extensible SDK
-
-target_sdk_dir="$1"
-sdk_targets="$2"
-
-bitbake $sdk_targets --setscene-only || exit 1
-- 
2.5.0