[PATCH 0/9] Extensible SDK fixes

[PATCH 0/9] Extensible SDK fixes

[Paul Eggleton]

Some improvements for the extensible SDK, including fixes for some issues
within devtool.


The following changes since commit d3ab7005f0c899da9f9f132b22861bd5d4f952ba:

  tzdata: update to 2016b (2016-03-20 22:58:00 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/extsdkfixes9-oe
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/extsdkfixes9-oe

Paul Eggleton (9):
  devtool: sdk-install: add option to allow building from source
  classes/populate_sdk_ext: parse metadata on minimal SDK install
  bitbake.conf: whitelist proxy variables in config hash
  devtool: configure-help: fix error if do_configure not already run
  devtool: add: delete externalsrc files on npm recipe do_install
  devtool: add: create git repository if URL specified as positional argument
  oe-publish-sdk: drop SDK installer file from published output
  oe-publish-sdk: improve help output slightly
  oe-publish-sdk: fix remote publishing

 meta/classes/npm.bbclass              |  6 ++++--
 meta/classes/populate_sdk_ext.bbclass |  2 +-
 meta/conf/bitbake.conf                |  4 +++-
 meta/files/ext-sdk-prepare.py         | 10 +++++++---
 meta/lib/oeqa/selftest/devtool.py     |  2 ++
 scripts/lib/devtool/sdk.py            | 21 ++++++++++++++++++++-
 scripts/lib/devtool/standard.py       | 13 ++++++++++++-
 scripts/lib/devtool/utilcmds.py       |  2 +-
 scripts/oe-publish-sdk                | 34 ++++++++++++++++++++++------------
 9 files changed, 72 insertions(+), 22 deletions(-)

-- 
2.5.5

[PATCH 1/9] devtool: sdk-install: add option to allow building from source

[Paul Eggleton]

By default the sdk-install subcommand expects to restore the requested
items from sstate and fails if it can't. If the user is OK with building
from source, add a -s/--allow-build option to allow them to do that. In
the process, ensure we show the status output while we're installing.

Also add the missing header to the top of the file.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/sdk.py | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/sdk.py b/scripts/lib/devtool/sdk.py
index 22c5279..b1905f9 100644
--- a/scripts/lib/devtool/sdk.py
+++ b/scripts/lib/devtool/sdk.py
@@ -1,4 +1,19 @@
 # Development tool - sdk-update command plugin
+#
+# Copyright (C) 2015-2016 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import os
 import subprocess
@@ -278,8 +293,11 @@ def sdk_install(args, config, basepath, workspace):
                 if recipe.endswith('-native') and 'package' in task:
                     continue
                 install_tasks.append('%s:%s' % (recipe, task))
+        options = ''
+        if not args.allow_build:
+            options += ' --setscene-only'
         try:
-            exec_build_env_command(config.init_path, basepath, 'bitbake --setscene-only %s' % ' '.join(install_tasks))
+            exec_build_env_command(config.init_path, basepath, 'bitbake %s %s' % (options, ' '.join(install_tasks)), watch=True)
         except bb.process.ExecutionError as e:
             raise DevtoolError('Failed to install %s:\n%s' % (recipe, str(e)))
         failed = False
@@ -312,4 +330,5 @@ def register_commands(subparsers, context):
                                                    description='Installs additional recipe development files into the SDK. (You can use "devtool search" to find available recipes.)',
                                                    group='sdk')
         parser_sdk_install.add_argument('recipename', help='Name of the recipe to install the development artifacts for', nargs='+')
+        parser_sdk_install.add_argument('-s', '--allow-build', help='Allow building requested item(s) from source', action='store_true')
         parser_sdk_install.set_defaults(func=sdk_install)
-- 
2.5.5

[PATCH 2/9] classes/populate_sdk_ext: parse metadata on minimal SDK install

[Paul Eggleton]

Instead of skipping the build system preparation step within the
extensible SDK install process when SDK_EXT_TYPE is "minimal", run
bitbake -p so that the cache is populated ready for the first time
devtool is run.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass |  2 +-
 meta/files/ext-sdk-prepare.py         | 10 +++++++---
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index be4bcc0..cabf815 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -374,7 +374,7 @@ sdk_ext_postinst() {
 	# Warn if trying to use external bitbake and the ext SDK together
 	echo "(which bitbake > /dev/null 2>&1 && echo 'WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session instead.') || true" >> $env_setup_script
 
-	if [ "$prepare_buildsystem" != "no" -a -n "${@SDK_INSTALL_TARGETS.strip()}" ]; then
+	if [ "$prepare_buildsystem" != "no" ]; then
 		printf "Preparing build system...\n"
 		# dash which is /bin/sh on Ubuntu will not preserve the
 		# current working directory when first ran, nor will it set $1 when
diff --git a/meta/files/ext-sdk-prepare.py b/meta/files/ext-sdk-prepare.py
index c99e34f..7887696 100644
--- a/meta/files/ext-sdk-prepare.py
+++ b/meta/files/ext-sdk-prepare.py
@@ -30,10 +30,14 @@ def exec_watch(cmd, **options):
 
 def main():
     if len(sys.argv) < 2:
-        print('Please specify target to prepare with')
-        return 1
+        sdk_targets = []
+    else:
+        sdk_targets = ' '.join(sys.argv[1:]).split()
+    if not sdk_targets:
+        # Just do a parse so the cache is primed
+        ret, _ = exec_watch('bitbake -p')
+        return ret
 
-    sdk_targets = ' '.join(sys.argv[1:]).split()
     print('Preparing SDK for %s...' % ', '.join(sdk_targets))
 
     ret, out = exec_watch('bitbake %s --setscene-only' % ' '.join(sdk_targets))
-- 
2.5.5

[PATCH 3/9] bitbake.conf: whitelist proxy variables in config hash

[Paul Eggleton]

We don't need these variables incorporated in the config hash (i.e. we
don't need to reparse if they change).

This fixes an issue with the extensible SDK where the cache generated
at installation time not to be used afterwards due to the way we're
allowing through the proxy variables into the install environment (such
that they get set to "" if not set externally).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/conf/bitbake.conf | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 7ed5ffb..e3db504 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -807,7 +807,9 @@ BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
     WARN_QA ERROR_QA WORKDIR STAMPCLEAN PKGDATA_DIR BUILD_ARCH SSTATE_PKGARCH"
 BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \
     SSH_AUTH_SOCK PSEUDO_BUILD BB_ENV_EXTRAWHITE DISABLE_SANITY_CHECKS \
-    PARALLEL_MAKE BB_NUMBER_THREADS BB_ORIGENV BB_INVALIDCONF BBINCLUDED"
+    PARALLEL_MAKE BB_NUMBER_THREADS BB_ORIGENV BB_INVALIDCONF BBINCLUDED \
+    GIT_PROXY_COMMAND ALL_PROXY all_proxy NO_PROXY no_proxy FTP_PROXY ftp_proxy \
+    HTTP_PROXY http_proxy HTTPS_PROXY https_proxy SOCKS5_USER SOCKS5_PASSWD"
 BB_SIGNATURE_EXCLUDE_FLAGS ?= "doc deps depends \
     lockfiles type vardepsexclude vardeps vardepvalue vardepvalueexclude \
     file-checksums python func task export unexport noexec nostamp dirs cleandirs \
-- 
2.5.5

[PATCH 4/9] devtool: configure-help: fix error if do_configure not already run

[Paul Eggleton]

The code here for running do_configure if it hadn't already been run was
using the wrong string substitution parameters; fix it and test it.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/utilcmds.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/utilcmds.py b/scripts/lib/devtool/utilcmds.py
index 905d6d2..b761a80 100644
--- a/scripts/lib/devtool/utilcmds.py
+++ b/scripts/lib/devtool/utilcmds.py
@@ -86,7 +86,7 @@ def configure_help(args, config, basepath, workspace):
         logger.info('Running do_configure to generate configure script')
         try:
             stdout, _ = exec_build_env_command(config.init_path, basepath,
-                                               'bitbake -c configure %s' % msg, args.recipename,
+                                               'bitbake -c configure %s' % args.recipename,
                                                stderr=subprocess.STDOUT)
         except bb.process.ExecutionError:
             pass
-- 
2.5.5

[PATCH 5/9] devtool: add: delete externalsrc files on npm recipe do_install

[Paul Eggleton]

The npm class just installs whatever is in ${S}; if you're using
externalsrc in conjunction with it the symlinks (oe-workdir and oe-logs
by default) plus singletask.lock will end up in the final package, which
isn't really right. Introduce a variable so we know the path the files
will be installed into within npm.bbclass, and append to do_install
within the workspace bbappend to delete the files.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/npm.bbclass        |  6 ++++--
 scripts/lib/devtool/standard.py | 11 +++++++++++
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index b5db99d..33ff5e3 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -1,6 +1,8 @@
 DEPENDS_prepend = "nodejs-native "
 S = "${WORKDIR}/npmpkg"
 
+NPM_INSTALLDIR = "${D}${libdir}/node_modules/${PN}"
+
 npm_do_compile() {
 	# changing the home directory to the working directory, the .npmrc will
 	# be created in this directory
@@ -14,8 +16,8 @@ npm_do_compile() {
 }
 
 npm_do_install() {
-	mkdir -p ${D}${libdir}/node_modules/${PN}/
-	cp -a ${S}/* ${D}${libdir}/node_modules/${PN}/ --no-preserve=ownership
+	mkdir -p ${NPM_INSTALLDIR}/
+	cp -a ${S}/* ${NPM_INSTALLDIR}/ --no-preserve=ownership
 }
 
 python populate_packages_prepend () {
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 7600a8f..9ac2ad5 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -238,6 +238,17 @@ def add(args, config, basepath, workspace):
             f.write('    rm -f ${D}/singletask.lock\n')
             f.write('}\n')
 
+        if bb.data.inherits_class('npm', rd):
+            f.write('do_install_append() {\n')
+            f.write('    # Remove files added to source dir by devtool/externalsrc\n')
+            f.write('    rm -f ${NPM_INSTALLDIR}/singletask.lock\n')
+            f.write('    rm -rf ${NPM_INSTALLDIR}/.git\n')
+            f.write('    rm -rf ${NPM_INSTALLDIR}/oe-local-files\n')
+            f.write('    for symlink in ${EXTERNALSRC_SYMLINKS} ; do\n')
+            f.write('        rm -f ${NPM_INSTALLDIR}/${symlink%%:*}\n')
+            f.write('    done\n')
+            f.write('}\n')
+
     _add_md5(config, recipename, appendfile)
 
     logger.info('Recipe %s has been automatically created; further editing may be required to make it fully functional' % recipefile)
-- 
2.5.5

[PATCH 6/9] devtool: add: create git repository if URL specified as positional argument

[Paul Eggleton]

When we add from a fetched URL we are supposed to turn the resulting
source tree into a git repository (if it isn't already one). However, we
were using the older deprecated option name here instead of the
positional argument, so "devtool add -f <url>" resulted in the repo
being created but "devtool add <url>" didn't, which was wrong.

Also update the oe-selftest tests to check that this worked.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/devtool.py | 2 ++
 scripts/lib/devtool/standard.py   | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 2e4b6da..fe9f7f7 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -301,6 +301,7 @@ class DevtoolTests(DevtoolBase):
         result = runCmd('devtool add %s %s -f %s' % (testrecipe, srcdir, url))
         self.assertTrue(os.path.exists(os.path.join(self.workspacedir, 'conf', 'layer.conf')), 'Workspace directory not created. %s' % result.output)
         self.assertTrue(os.path.isfile(os.path.join(srcdir, 'setup.py')), 'Unable to find setup.py in source directory')
+        self.assertTrue(os.path.isdir(os.path.join(srcdir, '.git')), 'git repository for external source tree was not created')
         # Test devtool status
         result = runCmd('devtool status')
         self.assertIn(testrecipe, result.output)
@@ -394,6 +395,7 @@ class DevtoolTests(DevtoolBase):
         result = runCmd('devtool add %s' % url)
         self.assertTrue(os.path.exists(os.path.join(self.workspacedir, 'conf', 'layer.conf')), 'Workspace directory not created. %s' % result.output)
         self.assertTrue(os.path.isfile(os.path.join(srcdir, 'configure')), 'Unable to find configure script in source directory')
+        self.assertTrue(os.path.isdir(os.path.join(srcdir, '.git')), 'git repository for external source tree was not created')
         # Test devtool status
         result = runCmd('devtool status')
         self.assertIn(testrecipe, result.output)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 9ac2ad5..f3f3e98 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -205,7 +205,7 @@ def add(args, config, basepath, workspace):
     for fn in os.listdir(recipedir):
         _add_md5(config, recipename, os.path.join(recipedir, fn))
 
-    if args.fetch and not args.no_git:
+    if args.fetchuri and not args.no_git:
         setup_git_repo(srctree, args.version, 'devtool')
 
     initial_rev = None
-- 
2.5.5

[PATCH 7/9] oe-publish-sdk: drop SDK installer file from published output

[Paul Eggleton]

We don't really need the SDK installer in the published output, for two
reasons:

1) The directory produced is for the consumption of devtool sdk-update,
   and the installer isn't used by that at all
2) It wouldn't really make sense to point users at the update directory
   to download the SDK installer because it contains a bunch of things
   that aren't meant for manual download, so it wouldn't be very tidy.

Leaving the file present can mislead you into thinking the opposite of
both of the above.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/oe-publish-sdk | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/scripts/oe-publish-sdk b/scripts/oe-publish-sdk
index 9e4f1bf..e91888f 100755
--- a/scripts/oe-publish-sdk
+++ b/scripts/oe-publish-sdk
@@ -53,6 +53,7 @@ def publish(args):
     else:
         is_remote = False
         dest_sdk = os.path.join(destination, sdk_basename)
+        destdir = destination
 
     # Making sure the directory exists
     logger.debug("Making sure the destination directory exists")
@@ -83,16 +84,19 @@ def publish(args):
 
     # Unpack the SDK
     logger.info("Unpacking SDK")
+    cleanupfiles = [dest_sdk, os.path.join(destdir, 'ext-sdk-prepare.py')]
     if not is_remote:
         cmd = "sh %s -n -y -d %s" % (dest_sdk, destination)
         ret = subprocess.call(cmd, shell=True)
         if ret == 0:
             logger.info('Successfully unpacked %s to %s' % (dest_sdk, destination))
+            for cleanupfile in cleanupfiles:
+                os.remove(cleanupfile)
         else:
             logger.error('Failed to unpack %s to %s' % (dest_sdk, destination))
             return ret
     else:
-        cmd = "ssh %s 'sh %s -n -y -d %s'" % (host, dest_sdk, destdir)
+        cmd = "ssh %s 'sh %s -n -y -d %s && rm -f %s'" % (host, dest_sdk, destdir, ' '.join(cleanupfiles))
         ret = subprocess.call(cmd, shell=True)
         if ret == 0:
             logger.info('Successfully unpacked %s to %s' % (dest_sdk, destdir))
-- 
2.5.5

[PATCH 8/9] oe-publish-sdk: improve help output slightly

[Paul Eggleton]

This was just copy-and-pasted from devtool - set a reasonable
description and tweak the sdk parameter help text.

Also add a copyright statement and drop the opening comment describing
the command-line syntax that duplicated the help output (with at least
one mistake in it).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/oe-publish-sdk | 26 ++++++++++++++++----------
 1 file changed, 16 insertions(+), 10 deletions(-)

diff --git a/scripts/oe-publish-sdk b/scripts/oe-publish-sdk
index e91888f..5dbd1e5 100755
--- a/scripts/oe-publish-sdk
+++ b/scripts/oe-publish-sdk
@@ -2,13 +2,20 @@
 
 # OpenEmbedded SDK publishing tool
 
-# oe-publish-sdk publish <ext-sdk> <destination>
-# <ext-sdk>: extensible SDK to publish (path to the installer shell script)
-# <destination>: local or remote location which servers as an SDK update server
-# e.g.
-# oe-publish-sdk /path/to/sdk-ext.sh /mnt/poky/sdk-ext
-# oe-publish-sdk /path/to/sdk-ext.sh user@host:/opt/poky/sdk-ext
+# Copyright (C) 2015-2016 Intel Corporation
 #
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import sys
 import os
@@ -118,13 +125,12 @@ def publish(args):
 
 
 def main():
-    parser = argparse_oe.ArgumentParser(description="OpenEmbedded development tool",
-                                        epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
+    parser = argparse_oe.ArgumentParser(description="OpenEmbedded extensible SDK publishing tool - writes server-side data to support the extensible SDK update process to a specified location")
     parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
     parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true')
 
-    parser.add_argument('sdk', help='Extensible SDK to publish')
-    parser.add_argument('dest', help='Destination to publish SDK to')
+    parser.add_argument('sdk', help='Extensible SDK to publish (path to .sh installer file)')
+    parser.add_argument('dest', help='Destination to publish SDK to; can be local path or remote in the form of user@host:/path (in the latter case ssh/scp will be used).')
 
     parser.set_defaults(func=publish)
 
-- 
2.5.5

[PATCH 9/9] oe-publish-sdk: fix remote publishing

[Paul Eggleton]

A typo in the command prevented the last part of the publish from
succeeding.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/oe-publish-sdk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/oe-publish-sdk b/scripts/oe-publish-sdk
index 5dbd1e5..1729a0d 100755
--- a/scripts/oe-publish-sdk
+++ b/scripts/oe-publish-sdk
@@ -115,7 +115,7 @@ def publish(args):
     if not is_remote:
         cmd = 'set -e; mkdir -p %s/layers; cd %s/layers; if [ ! -e .git ]; then git init .; mv .git/hooks/post-update.sample .git/hooks/post-update; fi; git add -A .; git commit -q -m "init repo" || true; git update-server-info' % (destination, destination)
     else:
-        cmd = "ssh %s 'set -e; mkdir-p %s/layers; cd %s/layers; if [ ! -e .git ]; then git init .; mv .git/hooks/post-update.sample .git/hooks/post-update; fi; git add -A .; git commit -q -m \"init repo\" || true; git update-server-info'" % (host, destdir, destdir)
+        cmd = "ssh %s 'set -e; mkdir -p %s/layers; cd %s/layers; if [ ! -e .git ]; then git init .; mv .git/hooks/post-update.sample .git/hooks/post-update; fi; git add -A .; git commit -q -m \"init repo\" || true; git update-server-info'" % (host, destdir, destdir)
     ret = subprocess.call(cmd, shell=True)
     if ret == 0:
         logger.info('SDK published successfully')
-- 
2.5.5