[PATCH 00/13] devtool / recipetool fixes

[PATCH 00/13] devtool / recipetool fixes

[Paul Eggleton]

Some fixes for devtool / recipetool issues, along with a couple of
related changes to the oe-selftest tests. I have run the oe-selftest
tests for devtool and recipetool and they pass with these changes
applied.

NOTE: this patchset depends upon the bitbake patchset I just sent to the
bitbake-devel list.


The following changes since commit 62f1122ef166eba56441d669c6b3b3fe5f367418:

  bluez5: cosmetic fixes to bluetooth bootscript (2017-08-30 11:13:52 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/devtool29-oe
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/devtool29-oe

Paul Eggleton (12):
  devtool: update-recipe: ensure patches get deleted in srcrev mode
  devtool: upgrade: fix handling of non-absolute paths
  devtool: upgrade: workaround for recipes which apply patches conditional upon class
  devtool: edit-recipe: fix regression
  oe-selftest: devtool: test find-recipe and edit-recipe
  devtool: add: add explicit srcrev/branch options
  recipetool: create: make recently added branch/tag handling git specific
  recipetool: create: fix SRCPV prefix for non-git SCMs
  recipetool: create: suppress npm shrinkwrap/lockdown warnings again
  recipetool: create: detect Eclipse licenses
  oe-selftest: tinfoil: add a test for variable history
  scriptutils: fix fetch_url() to use lowercase dummy recipe name

paul (1):
  devtool: upgrade: check that user has configured git properly

 meta/lib/oe/recipeutils.py              | 43 +++++++++++++++++++++++-------
 meta/lib/oeqa/selftest/cases/devtool.py | 25 ++++++++++++------
 meta/lib/oeqa/selftest/cases/tinfoil.py | 34 ++++++++++++++++++++++++
 scripts/lib/devtool/standard.py         | 20 +++++++++++---
 scripts/lib/devtool/upgrade.py          | 32 ++++++++++++++++++++--
 scripts/lib/devtool/utilcmds.py         | 15 ++++++++---
 scripts/lib/recipetool/create.py        | 47 ++++++++++++++++++++++++++++-----
 scripts/lib/scriptutils.py              |  2 +-
 8 files changed, 183 insertions(+), 35 deletions(-)

-- 
2.9.5

[PATCH 01/13] devtool: update-recipe: ensure patches get deleted in srcrev mode

[Paul Eggleton]

Patches that we identify as having been "deleted" (i.e. patches in
SRC_URI that no longer appear in the git tree) need to be dropped even
if we're updating in srcrev mode. This fixes the case where HEAD of the
git tree is valid upstream (i.e. no extra commits), but there are
patches left over in the recipe, e.g. when we do devtool upgrade and
then all of the commits rebased on top of the new branch get skipped.

Fixes [YOCTO #11972].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index fa9d347..d79ebf1 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1352,9 +1352,10 @@ def _update_recipe_srcrev(srctree, rd, appendlayerdir, wildcard_version, no_remo
             old_srcrev = (rd.getVar('SRCREV', False) or '')
             upd_p, new_p, del_p = _export_patches(srctree, rd, old_srcrev,
                                                   patches_dir)
+            logger.debug('Patches: update %s, new %s, delete %s' % (dict(upd_p), dict(new_p), dict(del_p)))
 
             # Remove deleted local files and "overlapping" patches
-            remove_files = list(del_f.values()) + list(upd_p.values())
+            remove_files = list(del_f.values()) + list(upd_p.values()) + list(del_p.values())
             if remove_files:
                 removedentries = _remove_file_entries(srcuri, remove_files)[0]
                 update_srcuri = True
-- 
2.9.5

[PATCH 02/13] devtool: upgrade: fix handling of non-absolute paths

[Paul Eggleton]

If your BBLAYERS has non-absolute paths in it (e.g.
"${COREBASE}/../something") then none of the paths matched in
copy_recipe_files() with the result that no files got copied and you
ended up with an error later on because the recipe file couldn't be
found at the destination. Fix this as well as adding an explicit check
to see if no files got copied - error out earlier if so.

Fixes [YOCTO #10981].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/recipeutils.py     | 8 +++++---
 scripts/lib/devtool/upgrade.py | 3 +++
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index a7fdd36..b946128 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -334,11 +334,13 @@ def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True):
         fetch.download()
 
     # Copy local files to target directory and gather any remote files
-    bb_dir = os.path.dirname(d.getVar('FILE')) + os.sep
+    bb_dir = os.path.abspath(os.path.dirname(d.getVar('FILE'))) + os.sep
     remotes = []
     copied = []
-    includes = [path for path in d.getVar('BBINCLUDED').split() if
-                path.startswith(bb_dir) and os.path.exists(path)]
+    # Need to do this in two steps since we want to check against the absolute path
+    includes = [os.path.abspath(path) for path in d.getVar('BBINCLUDED').split() if os.path.exists(path)]
+    # We also check this below, but we don't want any items in this list being considered remotes
+    includes = [path for path in includes if path.startswith(bb_dir)]
     for path in fetch.localpaths() + includes:
         # Only import files that are under the meta directory
         if path.startswith(bb_dir):
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 1f11d47..24937dc 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -301,6 +301,9 @@ def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, workspace, tinfoil
     path = os.path.join(workspace, 'recipes', bpn)
     bb.utils.mkdirhier(path)
     copied, _ = oe.recipeutils.copy_recipe_files(rd, path)
+    if not copied:
+        raise DevtoolError('Internal error - no files were copied for recipe %s' % bpn)
+    logger.debug('Copied %s to %s' % (copied, path))
 
     oldpv = rd.getVar('PV')
     if not newpv:
-- 
2.9.5

[PATCH 03/13] devtool: upgrade: check that user has configured git properly

[Paul Eggleton]

From: paul <paul@peggleto-mobl.ger.corp.intel.com>

If user.name or user.email haven't been set then git rebase can't really
work properly. Check that the user has set these and error out if not.
(Elsewhere we are relying on OE's git patch functionality which forces
a dummy OE value - that's OK there as it's completely under OE's control
and therefore it's OK for a dummy OE user to be the committer, but here
the rebase may require intervention so it's reasonable to have the
user's actual name and email on the operation.)

Fixes [YOCTO #11947].

Signed-off-by: paul <paul@peggleto-mobl.ger.corp.intel.com>
---
 scripts/lib/devtool/upgrade.py | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 24937dc..e334d2b 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -1,6 +1,6 @@
 # Development tool - upgrade command plugin
 #
-# Copyright (C) 2014-2015 Intel Corporation
+# Copyright (C) 2014-2017 Intel Corporation
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 2 as
@@ -355,6 +355,29 @@ def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, workspace, tinfoil
 
     return fullpath, copied
 
+
+def _check_git_config():
+    def getconfig(name):
+        try:
+            value = bb.process.run('git config --global %s' % name)[0].strip()
+        except bb.process.ExecutionError as e:
+            if e.exitcode == 1:
+                value = None
+            else:
+                raise
+        return value
+
+    username = getconfig('user.name')
+    useremail = getconfig('user.email')
+    configerr = []
+    if not username:
+        configerr.append('Please set your name using:\n  git config --global user.name')
+    if not useremail:
+        configerr.append('Please set your email using:\n  git config --global user.email')
+    if configerr:
+        raise DevtoolError('Your git configuration is incomplete which will prevent rebases from working:\n' + '\n'.join(configerr))
+
+
 def upgrade(args, config, basepath, workspace):
     """Entry point for the devtool 'upgrade' subcommand"""
 
@@ -365,6 +388,8 @@ def upgrade(args, config, basepath, workspace):
     if args.srcbranch and not args.srcrev:
         raise DevtoolError("If you specify --srcbranch/-B then you must use --srcrev/-S to specify the revision" % args.recipename)
 
+    _check_git_config()
+
     tinfoil = setup_tinfoil(basepath=basepath, tracking=True)
     try:
         rd = parse_recipe(config, tinfoil, args.recipename, True)
-- 
2.9.5

[PATCH 04/13] devtool: upgrade: workaround for recipes which apply patches conditional upon class

[Paul Eggleton]

If we're upgrading a recipe that appends additional patches for, say,
class-native, and we're just upgrading the target variant, then when we
copied the recipe into the workspace we skipped copying the additional patches
for the native variant. This caused warnings because the workspace
recipe is preferred. Look at SRC_URI for all variants when copying files
to work around this.

More work is needed to make it easier to work with recipes that use
BBCLASSEXTEND where you need to build more than one variant at once, but
this at least fixes the immediate ugliness.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/recipeutils.py     | 35 ++++++++++++++++++++++++++++-------
 scripts/lib/devtool/upgrade.py |  2 +-
 2 files changed, 29 insertions(+), 8 deletions(-)

diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index b946128..c8570ac 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -2,7 +2,7 @@
 #
 # Some code borrowed from the OE layer index
 #
-# Copyright (C) 2013-2016 Intel Corporation
+# Copyright (C) 2013-2017 Intel Corporation
 #
 
 import sys
@@ -320,7 +320,7 @@ def patch_recipe(d, fn, varvalues, patch=False, relpath=''):
 
 
 
-def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True):
+def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True, all_variants=False):
     """Copy (local) recipe files, including both files included via include/require,
     and files referred to in the SRC_URI variable."""
     import bb.fetch2
@@ -328,10 +328,31 @@ def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True):
 
     # FIXME need a warning if the unexpanded SRC_URI value contains variable references
 
-    uris = (d.getVar('SRC_URI') or "").split()
-    fetch = bb.fetch2.Fetch(uris, d)
-    if download:
-        fetch.download()
+    uri_values = []
+    localpaths = []
+    def fetch_urls(rdata):
+        # Collect the local paths from SRC_URI
+        srcuri = rdata.getVar('SRC_URI') or ""
+        if srcuri not in uri_values:
+            fetch = bb.fetch2.Fetch(srcuri.split(), rdata)
+            if download:
+                fetch.download()
+            for pth in fetch.localpaths():
+                if pth not in localpaths:
+                    localpaths.append(pth)
+            uri_values.append(srcuri)
+
+    fetch_urls(d)
+    if all_variants:
+        # Get files for other variants e.g. in the case of a SRC_URI_append
+        localdata = bb.data.createCopy(d)
+        variants = (localdata.getVar('BBCLASSEXTEND') or '').split()
+        if variants:
+            # Ensure we handle class-target if we're dealing with one of the variants
+            variants.append('target')
+            for variant in variants:
+                localdata.setVar('CLASSOVERRIDE', 'class-%s' % variant)
+                fetch_urls(localdata)
 
     # Copy local files to target directory and gather any remote files
     bb_dir = os.path.abspath(os.path.dirname(d.getVar('FILE'))) + os.sep
@@ -341,7 +362,7 @@ def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True):
     includes = [os.path.abspath(path) for path in d.getVar('BBINCLUDED').split() if os.path.exists(path)]
     # We also check this below, but we don't want any items in this list being considered remotes
     includes = [path for path in includes if path.startswith(bb_dir)]
-    for path in fetch.localpaths() + includes:
+    for path in localpaths + includes:
         # Only import files that are under the meta directory
         if path.startswith(bb_dir):
             if not whole_dir:
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index e334d2b..41bd34e 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -300,7 +300,7 @@ def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, workspace, tinfoil
     bpn = rd.getVar('BPN')
     path = os.path.join(workspace, 'recipes', bpn)
     bb.utils.mkdirhier(path)
-    copied, _ = oe.recipeutils.copy_recipe_files(rd, path)
+    copied, _ = oe.recipeutils.copy_recipe_files(rd, path, all_variants=True)
     if not copied:
         raise DevtoolError('Internal error - no files were copied for recipe %s' % bpn)
     logger.debug('Copied %s to %s' % (copied, path))
-- 
2.9.5

[PATCH 05/13] devtool: edit-recipe: fix regression

[Paul Eggleton]

OE-Core commit 5a16b3c804c5eca331a1c08a7ce31a54909af105 attempted to use
the same function to get the path to a recipe as the new "find-recipe"
command it implemented, except that cannot work because (a) it didn't
return anything and (b) event if it had tried, a command function can
only return an exit code and we don't want that for find-recipe if it
succeeded. Split out a separate reusable function for both commands.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/utilcmds.py | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/devtool/utilcmds.py b/scripts/lib/devtool/utilcmds.py
index c8cb760..b745116 100644
--- a/scripts/lib/devtool/utilcmds.py
+++ b/scripts/lib/devtool/utilcmds.py
@@ -30,14 +30,13 @@ from devtool import parse_recipe
 
 logger = logging.getLogger('devtool')
 
-def find_recipe(args, config, basepath, workspace):
-    """Entry point for the devtool 'find-recipe' subcommand"""
+def _find_recipe_path(args, config, basepath, workspace):
     if args.any_recipe:
         tinfoil = setup_tinfoil(config_only=False, basepath=basepath)
         try:
             rd = parse_recipe(config, tinfoil, args.recipename, True)
             if not rd:
-                return 1
+                raise DevtoolError("Failed to find specified recipe")
             recipefile = rd.getVar('FILE')
         finally:
             tinfoil.shutdown()
@@ -47,11 +46,19 @@ def find_recipe(args, config, basepath, workspace):
         if not recipefile:
             raise DevtoolError("Recipe file for %s is not under the workspace" %
                                args.recipename)
+    return recipefile
+
+
+def find_recipe(args, config, basepath, workspace):
+    """Entry point for the devtool 'find-recipe' subcommand"""
+    recipefile = _find_recipe_path(args, config, basepath, workspace)
+    print(recipefile)
+    return 0
 
 
 def edit_recipe(args, config, basepath, workspace):
     """Entry point for the devtool 'edit-recipe' subcommand"""
-    return scriptutils.run_editor(find_recipe(args, config, basepath, workspace), logger)
+    return scriptutils.run_editor(_find_recipe_path(args, config, basepath, workspace), logger)
 
 
 def configure_help(args, config, basepath, workspace):
-- 
2.9.5

[PATCH 06/13] oe-selftest: devtool: test find-recipe and edit-recipe

[Paul Eggleton]

We weren't testing the devtool find-recipe and edit-recipe subcommands,
with the result that when they regressed recently we didn't notice. Add
some code into the test_devtool_add to test this (since we need a
recipe in the workspace, and adding a new test with all that preamble
would seem a bit excessive for these simple checks). Also take the
opportunity to refactor the test a little bit so that the recipe name
and version are variables rather than hardcoding them everywhere.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/cases/devtool.py | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py
index 88d6972..c17131a 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -194,26 +194,35 @@ class DevtoolTests(DevtoolBase):
         # Fetch source
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
         self.track_for_cleanup(tempdir)
+        pn = 'pv'
+        pv = '1.5.3'
         url = 'http://www.ivarch.com/programs/sources/pv-1.5.3.tar.bz2'
         result = runCmd('wget %s' % url, cwd=tempdir)
-        result = runCmd('tar xfv pv-1.5.3.tar.bz2', cwd=tempdir)
-        srcdir = os.path.join(tempdir, 'pv-1.5.3')
+        result = runCmd('tar xfv %s' % os.path.basename(url), cwd=tempdir)
+        srcdir = os.path.join(tempdir, '%s-%s' % (pn, pv))
         self.assertTrue(os.path.isfile(os.path.join(srcdir, 'configure')), 'Unable to find configure script in source directory')
         # Test devtool add
         self.track_for_cleanup(self.workspacedir)
-        self.add_command_to_tearDown('bitbake -c cleansstate pv')
+        self.add_command_to_tearDown('bitbake -c cleansstate %s' % pn)
         self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
-        result = runCmd('devtool add pv %s' % srcdir)
+        result = runCmd('devtool add %s %s' % (pn, srcdir))
         self.assertExists(os.path.join(self.workspacedir, 'conf', 'layer.conf'), 'Workspace directory not created')
         # Test devtool status
         result = runCmd('devtool status')
-        self.assertIn('pv', result.output)
+        recipepath = '%s/recipes/%s/%s_%s.bb' % (self.workspacedir, pn, pn, pv)
+        self.assertIn(recipepath, result.output)
         self.assertIn(srcdir, result.output)
+        # Test devtool find-recipe
+        result = runCmd('devtool -q find-recipe %s' % pn)
+        self.assertEqual(recipepath, result.output.strip())
+        # Test devtool edit-recipe
+        result = runCmd('EDITOR="echo 123" devtool -q edit-recipe %s' % pn)
+        self.assertEqual('123 %s' % recipepath, result.output.strip())
         # Clean up anything in the workdir/sysroot/sstate cache (have to do this *after* devtool add since the recipe only exists then)
-        bitbake('pv -c cleansstate')
+        bitbake('%s -c cleansstate' % pn)
         # Test devtool build
-        result = runCmd('devtool build pv')
-        bb_vars = get_bb_vars(['D', 'bindir'], 'pv')
+        result = runCmd('devtool build %s' % pn)
+        bb_vars = get_bb_vars(['D', 'bindir'], pn)
         installdir = bb_vars['D']
         self.assertTrue(installdir, 'Could not query installdir variable')
         bindir = bb_vars['bindir']
-- 
2.9.5

[PATCH 07/13] devtool: add: add explicit srcrev/branch options

[Paul Eggleton]

At the moment when fetching source from a git repository you have to
know that you can specify the revision and branch in the URL with
';rev=' and ';branch=' respectively, and you can also get thrown off by
the shell splitting on the ; character if you forget to surround the URL
in quotes. Add explicit -S/--srcrev and -B/--srcbranch options
(consistent with devtool upgrade) to make this easier for the user to
discover and use. (The rev and branch URL parameters will continue to
work, however.)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py  | 17 +++++++++++++++--
 scripts/lib/recipetool/create.py | 25 ++++++++++++++++++++++---
 2 files changed, 37 insertions(+), 5 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index d79ebf1..174cc86 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1,6 +1,6 @@
 # Development tool - standard commands plugin
 #
-# Copyright (C) 2014-2016 Intel Corporation
+# Copyright (C) 2014-2017 Intel Corporation
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 2 as
@@ -66,6 +66,12 @@ def add(args, config, basepath, workspace):
         elif os.path.isdir(args.recipename):
             logger.warn('Ambiguous argument "%s" - assuming you mean it to be the recipe name' % args.recipename)
 
+    if not args.fetchuri:
+        if args.srcrev:
+            raise DevtoolError('The -S/--srcrev option is only valid when fetching from an SCM repository')
+        if args.srcbranch:
+            raise DevtoolError('The -B/--srcbranch option is only valid when fetching from an SCM repository')
+
     if args.srctree and os.path.isfile(args.srctree):
         args.fetchuri = 'file://' + os.path.abspath(args.srctree)
         args.srctree = ''
@@ -151,6 +157,10 @@ def add(args, config, basepath, workspace):
         extracmdopts += ' --fetch-dev'
     if args.mirrors:
         extracmdopts += ' --mirrors'
+    if args.srcrev:
+        extracmdopts += ' --srcrev %s' % args.srcrev
+    if args.srcbranch:
+        extracmdopts += ' --srcbranch %s' % args.srcbranch
 
     tempdir = tempfile.mkdtemp(prefix='devtool')
     try:
@@ -1809,7 +1819,10 @@ def register_commands(subparsers, context):
     parser_add.add_argument('--fetch-dev', help='For npm, also fetch devDependencies', action="store_true")
     parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
     parser_add.add_argument('--no-git', '-g', help='If fetching source, do not set up source tree as a git repository', action="store_true")
-    parser_add.add_argument('--autorev', '-a', help='When fetching from a git repository, set SRCREV in the recipe to a floating revision instead of fixed', action="store_true")
+    group = parser_add.add_mutually_exclusive_group()
+    group.add_argument('--srcrev', '-S', help='Source revision to fetch if fetching from an SCM such as git (default latest)')
+    group.add_argument('--autorev', '-a', help='When fetching from a git repository, set SRCREV in the recipe to a floating revision instead of fixed', action="store_true")
+    parser_add.add_argument('--srcbranch', '-B', help='Branch in source repository if fetching from an SCM such as git (default master)')
     parser_add.add_argument('--binary', '-b', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure). Useful with binary packages e.g. RPMs.', action='store_true')
     parser_add.add_argument('--also-native', help='Also add native variant (i.e. support building recipe for the build host as well as the target machine)', action='store_true')
     parser_add.add_argument('--src-subdir', help='Specify subdirectory within source tree to use', metavar='SUBDIR')
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 2601145..8346336 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1,6 +1,6 @@
 # Recipe creation tool - create command plugin
 #
-# Copyright (C) 2014-2016 Intel Corporation
+# Copyright (C) 2014-2017 Intel Corporation
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 2 as
@@ -440,14 +440,30 @@ def create_recipe(args):
         rev_re = re.compile(';rev=([^;]+)')
         res = rev_re.search(srcuri)
         if res:
+            if args.srcrev:
+                logger.error('rev= parameter and -S/--srcrev option cannot both be specified - use one or the other')
+                sys.exit(1)
+            if args.autorev:
+                logger.error('rev= parameter and -a/--autorev option cannot both be specified - use one or the other')
+                sys.exit(1)
             srcrev = res.group(1)
             srcuri = rev_re.sub('', srcuri)
+        elif args.srcrev:
+            srcrev = args.srcrev
 
         # Check whether users provides any branch info in fetchuri.
         # If true, we will skip all branch checking process to honor all user's input.
         scheme, network, path, user, passwd, params = bb.fetch2.decodeurl(fetchuri)
         srcbranch = params.get('branch')
+        if args.srcbranch:
+            if srcbranch:
+                logger.error('branch= parameter and -B/--srcbranch option cannot both be specified - use one or the other')
+                sys.exit(1)
+            srcbranch = args.srcbranch
         nobranch = params.get('nobranch')
+        if nobranch and srcbranch:
+            logger.error('nobranch= cannot be used if you specify a branch')
+            sys.exit(1)
         tag = params.get('tag')
         if not srcbranch and not nobranch and srcrev != '${AUTOREV}':
             # Append nobranch=1 in the following conditions:
@@ -523,7 +539,7 @@ def create_recipe(args):
             else:
                 # If get_branch contains more than one objects, then display error and exit.
                 mbrch = '\n  ' + '\n  '.join(get_branch)
-                logger.error('Revision %s was found on multiple branches: %s\nPlease provide the correct branch in the source URL with ;branch=<branch> (and ensure you use quotes around the URL to avoid the shell interpreting the ";")' % (srcrev, mbrch))
+                logger.error('Revision %s was found on multiple branches: %s\nPlease provide the correct branch with -B/--srcbranch' % (srcrev, mbrch))
                 sys.exit(1)
 
         # Since we might have a value in srcbranch, we need to
@@ -1277,7 +1293,10 @@ def register_commands(subparsers):
     parser_create.add_argument('-b', '--binary', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure)', action='store_true')
     parser_create.add_argument('--also-native', help='Also add native variant (i.e. support building recipe for the build host as well as the target machine)', action='store_true')
     parser_create.add_argument('--src-subdir', help='Specify subdirectory within source tree to use', metavar='SUBDIR')
-    parser_create.add_argument('-a', '--autorev', help='When fetching from a git repository, set SRCREV in the recipe to a floating revision instead of fixed', action="store_true")
+    group = parser_create.add_mutually_exclusive_group()
+    group.add_argument('-a', '--autorev', help='When fetching from a git repository, set SRCREV in the recipe to a floating revision instead of fixed', action="store_true")
+    group.add_argument('-S', '--srcrev', help='Source revision to fetch if fetching from an SCM such as git (default latest)')
+    parser_create.add_argument('-B', '--srcbranch', help='Branch in source repository if fetching from an SCM such as git (default master)')
     parser_create.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)')
     parser_create.add_argument('--fetch-dev', action="store_true", help='For npm, also fetch devDependencies')
     parser_create.add_argument('--devtool', action="store_true", help=argparse.SUPPRESS)
-- 
2.9.5

[PATCH 08/13] recipetool: create: make recently added branch/tag handling git specific

[Paul Eggleton]

The branch and tag handling code that was recently added in OE-Core revs
ecca596b75cfda2f798a0bdde75f4f774e23a95b and
3afdcbdc9a3e65bc925ec61717784ffec67d529d is specific to git, so only
apply it when we're fetching from a git URL.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 8346336..4da745b 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -518,7 +518,7 @@ def create_recipe(args):
         # We need this checking mechanism to improve the recipe created by recipetool and devtool
         # is able to parse and build by bitbake.
         # If there is no input for branch name, then check for branch name with SRCREV provided.
-        if not srcbranch and not nobranch and srcrev and (srcrev != '${AUTOREV}'):
+        if not srcbranch and not nobranch and srcrev and (srcrev != '${AUTOREV}') and scheme in ['git', 'gitsm']:
             try:
                 cmd = 'git branch -r --contains'
                 check_branch, check_branch_err = bb.process.run('%s %s' % (cmd, srcrev), cwd=srctree)
@@ -549,7 +549,7 @@ def create_recipe(args):
             params['branch'] = srcbranch
             srcuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd, params))
 
-        if storeTagName:
+        if storeTagName and scheme in ['git', 'gitsm']:
             # Re-introduced tag variable from storeTagName
             # Check srcrev using tag and check validity of the tag
             cmd = ('git rev-parse --verify %s' % (storeTagName))
-- 
2.9.5

[PATCH 09/13] recipetool: create: fix SRCPV prefix for non-git SCMs

[Paul Eggleton]

If you're fetching from an SCM other than git (for example subversion or
mercurial) then we need to use a different prefix for the SRCPV in PV
instead of +git.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 4da745b..d8cfcbd 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -423,6 +423,7 @@ def create_recipe(args):
     srcsubdir = ''
     srcrev = '${AUTOREV}'
     srcbranch = ''
+    scheme = ''
     storeTagName = ''
     pv_srcpv = False
 
@@ -682,7 +683,14 @@ def create_recipe(args):
         lines_before.append('')
         lines_before.append('# Modify these as desired')
         # Note: we have code to replace realpv further down if it gets set to some other value
-        lines_before.append('PV = "%s+git${SRCPV}"' % (realpv or '1.0'))
+        scheme, _, _, _, _, _ = bb.fetch2.decodeurl(srcuri)
+        if scheme in ['git', 'gitsm']:
+            srcpvprefix = 'git'
+        elif scheme == 'svn':
+            srcpvprefix = 'svnr'
+        else:
+            srcpvprefix = scheme
+        lines_before.append('PV = "%s+%s${SRCPV}"' % (realpv or '1.0', srcpvprefix))
         pv_srcpv = True
         if not args.autorev and srcrev == '${AUTOREV}':
             if os.path.exists(os.path.join(srctree, '.git')):
-- 
2.9.5

[PATCH 10/13] recipetool: create: suppress npm shrinkwrap/lockdown warnings again

[Paul Eggleton]

Since OE-Core revision 9a47a6690052ef943c0d4760630ee630fb012153 the
mechanism we were using to suppress the warnings about
NPM_LOCKDOWN and NPM_SHRINKWRAP not being set on the first fetch of the
source is no longer available since we are using the normal fetch/unpack
tasks to do the job. Use the newly added noverify parameter to suppress
the warnings again.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index d8cfcbd..2b30385 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -479,6 +479,8 @@ def create_recipe(args):
             storeTagName = params['tag']
             params['nobranch'] = '1'
             del params['tag']
+        if scheme == 'npm':
+            params['noverify'] = '1'
         fetchuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd, params))
 
         tmpparent = tinfoil.config_data.getVar('BASE_WORKDIR')
-- 
2.9.5

[PATCH 11/13] recipetool: create: detect Eclipse licenses

[Paul Eggleton]

Add detection of EPL 1.0 and EDL 1.0 license files.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 2b30385..4788691 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1099,6 +1099,10 @@ def crunch_license(licfile):
     crunched_md5sums['1daebd9491d1e8426900b4fa5a422814'] = 'LGPLv2.1'
     # https://github.com/FFmpeg/FFmpeg/blob/master/COPYING.LGPLv3
     crunched_md5sums['2ebfb3bb49b9a48a075cc1425e7f4129'] = 'LGPLv3'
+    # https://raw.githubusercontent.com/eclipse/mosquitto/v1.4.14/epl-v10
+    crunched_md5sums['efe2cb9a35826992b9df68224e3c2628'] = 'EPL-1.0'
+    # https://raw.githubusercontent.com/eclipse/mosquitto/v1.4.14/edl-v10
+    crunched_md5sums['0a9c78c0a398d1bbce4a166757d60387'] = 'EDL-1.0'
     lictext = []
     with open(licfile, 'r', errors='surrogateescape') as f:
         for line in f:
@@ -1131,7 +1135,7 @@ def guess_license(srctree, d):
     md5sums = get_license_md5sums(d)
 
     licenses = []
-    licspecs = ['*LICEN[CS]E*', 'COPYING*', '*[Ll]icense*', 'LEGAL*', '[Ll]egal*', '*GPL*', 'README.lic*', 'COPYRIGHT*', '[Cc]opyright*']
+    licspecs = ['*LICEN[CS]E*', 'COPYING*', '*[Ll]icense*', 'LEGAL*', '[Ll]egal*', '*GPL*', 'README.lic*', 'COPYRIGHT*', '[Cc]opyright*', 'e[dp]l-v10']
     licfiles = []
     for root, dirs, files in os.walk(srctree):
         for fn in files:
-- 
2.9.5

[PATCH 12/13] oe-selftest: tinfoil: add a test for variable history

[Paul Eggleton]

I recently found that variable history wasn't working properly for
recipes when we enable history tracking, resulting in minor
functionality loss in devtool upgrade, so add a test to ensure this
doesn't regress now that it's fixed.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/cases/tinfoil.py | 34 +++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/tinfoil.py b/meta/lib/oeqa/selftest/cases/tinfoil.py
index 471517f..f889a47 100644
--- a/meta/lib/oeqa/selftest/cases/tinfoil.py
+++ b/meta/lib/oeqa/selftest/cases/tinfoil.py
@@ -195,3 +195,37 @@ class TinfoilTests(OESelftestTestCase):
             tinfoil.config_data.appendVar('OVERRIDES', ':overrideone')
             value = tinfoil.config_data.getVar('TESTVAR')
             self.assertEqual(value, 'one', 'Variable overrides not functioning correctly')
+
+    def test_variable_history(self):
+        # Basic test to ensure that variable history works when tracking=True
+        with bb.tinfoil.Tinfoil(tracking=True) as tinfoil:
+            tinfoil.prepare(config_only=False, quiet=2)
+            # Note that _tracking for any datastore we get will be
+            # false here, that's currently expected - so we can't check
+            # for that
+            history = tinfoil.config_data.varhistory.variable('DL_DIR')
+            for entry in history:
+                if entry['file'].endswith('/bitbake.conf'):
+                    if entry['op'] in ['set', 'set?']:
+                        break
+            else:
+                self.fail('Did not find history entry setting DL_DIR in bitbake.conf. History: %s' % history)
+            # Check it works for recipes as well
+            testrecipe = 'zlib'
+            rd = tinfoil.parse_recipe(testrecipe)
+            history = rd.varhistory.variable('LICENSE')
+            bbfound = -1
+            recipefound = -1
+            for i, entry in enumerate(history):
+                if entry['file'].endswith('/bitbake.conf'):
+                    if entry['detail'] == 'INVALID' and entry['op'] in ['set', 'set?']:
+                        bbfound = i
+                elif entry['file'].endswith('.bb'):
+                    if entry['op'] == 'set':
+                        recipefound = i
+            if bbfound == -1:
+                self.fail('Did not find history entry setting LICENSE in bitbake.conf parsing %s recipe. History: %s' % (testrecipe, history))
+            if recipefound == -1:
+                self.fail('Did not find history entry setting LICENSE in %s recipe. History: %s' % (testrecipe, history))
+            if bbfound > recipefound:
+                self.fail('History entry setting LICENSE in %s recipe and in bitbake.conf in wrong order. History: %s' % (testrecipe, history))
-- 
2.9.5

[PATCH 13/13] scriptutils: fix fetch_url() to use lowercase dummy recipe name

[Paul Eggleton]

recipetool create (and hence devtool add) and devtool upgrade use
fetch_url() which creates a dummy recipe in order to fetch source.
Previously the random part of the name was using uppercase characters,
and this triggers a QA warning after OE-Core commit
4713f8b2c4f2c74239d284adcf1e59e61aa66576, so use lowercase instead as I
really should have in the first place.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/scriptutils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
index 11f1a78..85b1c94 100644
--- a/scripts/lib/scriptutils.py
+++ b/scripts/lib/scriptutils.py
@@ -134,7 +134,7 @@ def fetch_url(tinfoil, srcuri, srcrev, destdir, logger, preserve_tmp=False, mirr
             # for do_fetch and do_unpack
             # I'd use tempfile functions here but underscores can be produced by that and those
             # aren't allowed in recipe file names except to separate the version
-            rndstring = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(8))
+            rndstring = ''.join(random.choice(string.ascii_lowercase + string.digits) for _ in range(8))
             fetchrecipe = os.path.join(fetchrecipedir, 'tmp-recipetool-%s.bb' % rndstring)
             fetchrecipepn = os.path.splitext(os.path.basename(fetchrecipe))[0]
             logger.debug('Generating initial recipe %s for fetching' % fetchrecipe)
-- 
2.9.5

Re: [PATCH 03/13] devtool: upgrade: check that user has configured git properly

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 714 bytes --]

On 31 August 2017 at 00:54, Paul Eggleton <paul.eggleton@linux.intel.com>
wrote:

> From: paul <paul@peggleto-mobl.ger.corp.intel.com>
>
> If user.name or user.email haven't been set then git rebase can't really
> work properly. Check that the user has set these and error out if not.
> (Elsewhere we are relying on OE's git patch functionality which forces
> a dummy OE value - that's OK there as it's completely under OE's control
> and therefore it's OK for a dummy OE user to be the committer, but here
> the rebase may require intervention so it's reasonable to have the
> user's actual name and email on the operation.)
>

The irony is strong in this one.

(I'll fixup when merging)

Ross

[-- Attachment #2: Type: text/html, Size: 1286 bytes --]

Re: [PATCH 03/13] devtool: upgrade: check that user has configured git properly

[Paul Eggleton]

On Friday, 1 September 2017 1:29:38 AM NZST Burton, Ross wrote:
> On 31 August 2017 at 00:54, Paul Eggleton <paul.eggleton@linux.intel.com>
> wrote:
> 
> > From: paul <paul@peggleto-mobl.ger.corp.intel.com>
> >
> > If user.name or user.email haven't been set then git rebase can't really
> > work properly. Check that the user has set these and error out if not.
> > (Elsewhere we are relying on OE's git patch functionality which forces
> > a dummy OE value - that's OK there as it's completely under OE's control
> > and therefore it's OK for a dummy OE user to be the committer, but here
> > the rebase may require intervention so it's reasonable to have the
> > user's actual name and email on the operation.)
> >
> 
> The irony is strong in this one.
> 
> (I'll fixup when merging)

Shoot. Of course I committed the change just after I'd cleared out the 
settings to test the darn thing ;)

Cheers,
Paul


-- 

Paul Eggleton
Intel Open Source Technology Centre