[PATCH V3 00/11] Several fixes for layerindex

[PATCH V3 00/11] Several fixes for layerindex

[Robert Yang]

* V3:
  - Better error messages when required collections not found
  - Delete layerbranch for non-existed branch
  - update_layer.py: remove --update-dependencies

* V2:
  - Remove "import bb" from update.py to avoid causing confusions when switch branches.

* V1:
  - The "-a ACTUAL_BRANCH" is a litle different from what we had talked, now it
    respects "-l" and "-b", we can use -l to specify layers or default to all,
    and it requires "-b" to work, only one branch is supported in a run, for example:

    $ update.py -b master -a branch_20170526
    All the layers which have branch master and actual_branch branch_20170526
    will be updated to branch_20170526.

    $ update.py -b master -l meta-oe -a branch_20170526
    Only meta-oe layer will be updated.

    $ update.py -b master -l bitbake -a branch_20170526
    The bitbake's bitbake_branch will be updated.

// Robert

The following changes since commit 8dfe44ed5e618d517d4acb4a3107cb3907e773c8:

  TODO: drop implemented items (2017-03-22 14:21:22 +1300)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib rbt/layerindex-8fixes
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/layerindex-8fixes

Robert Yang (11):
  layerconfparse.py: remove unused layerbranch from parse_layer()
  utils.py: remove obsolete dependencies
  update_layer.py: set layerbranch's collection before add_dependencies
  update.py: fix -l option
  recipeparse.py: restore cwd when the parsing is done
  update.py: update layers orderly
  update_layer.py: remove --update-dependencies
  update.py: add option -a to update actual branch
  update_layer.py: move the location of transaction.atomic()
  update.py: add -p to git fetch
  update_layer.py: delete layerbranch for non-existed branch

 layerindex/layerconfparse.py     |   2 +-
 layerindex/recipeparse.py        |   3 +
 layerindex/tools/import_layer.py |   2 +-
 layerindex/update.py             | 193 +++++++++++++++++++++++++++++++--------
 layerindex/update_layer.py       |  92 +++++++------------
 layerindex/utils.py              |  63 ++++++++++++-
 6 files changed, 255 insertions(+), 100 deletions(-)

-- 
2.10.2

[PATCH V3 01/11] layerconfparse.py: remove unused layerbranch from parse_layer()

[Robert Yang]

The layerbranch is not used in parse_layer(), so remove it.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 layerindex/layerconfparse.py     | 2 +-
 layerindex/tools/import_layer.py | 2 +-
 layerindex/update_layer.py       | 4 ++--
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/layerindex/layerconfparse.py b/layerindex/layerconfparse.py
index d599ed5..526d2c2 100644
--- a/layerindex/layerconfparse.py
+++ b/layerindex/layerconfparse.py
@@ -32,7 +32,7 @@ class LayerConfParse:
 
         self.config_data_copy = bb.data.createCopy(self.tinfoil.config_data)
 
-    def parse_layer(self, layerbranch, layerdir):
+    def parse_layer(self, layerdir):
 
         # This is not a valid layer, parsing will cause exception.
         if not utils.is_layer_valid(layerdir):
diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/import_layer.py
index ad16e40..a806dd1 100755
--- a/layerindex/tools/import_layer.py
+++ b/layerindex/tools/import_layer.py
@@ -377,7 +377,7 @@ def main():
                         layerdep.save()
                     layerconfparser = LayerConfParse(logger=logger)
                     try:
-                        config_data = layerconfparser.parse_layer(layerbranch, layerdir)
+                        config_data = layerconfparser.parse_layer(layerdir)
                         if config_data:
                             utils.add_dependencies(layerbranch, config_data, logger=logger)
                             utils.add_recommends(layerbranch, config_data, logger=logger)
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index 9fe96fb..05ab3e7 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -267,7 +267,7 @@ def main():
                     sys.exit(0)
 
                 layerconfparser = layerconfparse.LayerConfParse(logger=logger, bitbakepath=bitbakepath, tinfoil=tinfoil)
-                config_data = layerconfparser.parse_layer(layerbranch, layerdir)
+                config_data = layerconfparser.parse_layer(layerdir)
                 if not config_data:
                     logger.debug("Layer %s does not appear to be valid for branch %s" % (layer.name, branchdesc))
                     sys.exit(0)
@@ -355,7 +355,7 @@ def main():
                 logger.info("Collecting data for layer %s on branch %s" % (layer.name, branchdesc))
 
                 layerconfparser = layerconfparse.LayerConfParse(logger=logger, tinfoil=tinfoil)
-                layer_config_data = layerconfparser.parse_layer(layerbranch, layerdir)
+                layer_config_data = layerconfparser.parse_layer(layerdir)
                 if not layer_config_data:
                     logger.info("Skipping update of layer %s for branch %s - conf/layer.conf may have parse issues" % (layer.name, branchdesc))
                     layerconfparser.shutdown()
-- 
2.10.2

[PATCH V3 02/11] utils.py: remove obsolete dependencies

[Robert Yang]

Fixed:
  - set LAYERDEPENDS_openembedded-layer = "core"
  - $ "update.py -l meta-oe -b master"
    Check from web, its dependency is "openembedded-core"
  - Change LAYERDEPENDS_openembedded-layer = "foo"
  - Run "update.py -l meta-oe -b master"
    Check from web, its dependency is "openembedded-core and foo", this is
    wrong, it should be "foo" only, this patch can fix the problem.

And also the existing checking should filter(required=required),
otherwise it can't work well when a layer is in both depends and
recommends, this can't happen in a normal case, but it would surprise the
user when this happens.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 layerindex/utils.py | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/layerindex/utils.py b/layerindex/utils.py
index 3b45b07..1a57c07 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -88,6 +88,7 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None, required=T
         logger.debug('Error parsing %s_%s for %s\n%s' % (var, var_name, layer_name, str(vse)))
         return
 
+    need_delete = None
     for dep, ver_list in list(dep_dict.items()):
         ver_str = None
         if ver_list:
@@ -106,8 +107,14 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None, required=T
                 logger.error('Cannot resolve %s %s (version %s) for %s' % (name, dep, ver_str, layer_name))
                 continue
 
+        # Preparing to remove obsolete ones
+        if not need_delete:
+            need_delete = LayerDependency.objects.filter(layerbranch=layerbranch).filter(required=required).exclude(dependency=dep_layer)
+        else:
+            need_delete = need_delete.exclude(dependency=dep_layer)
+
         # Skip existing entries.
-        existing = list(LayerDependency.objects.filter(layerbranch=layerbranch).filter(dependency=dep_layer))
+        existing = list(LayerDependency.objects.filter(layerbranch=layerbranch).filter(required=required).filter(dependency=dep_layer))
         if existing:
             logger.debug('Skipping %s - already a dependency for %s' % (dep, layer_name))
             continue
@@ -121,6 +128,10 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None, required=T
         layerdep.required = required
         layerdep.save()
 
+    if need_delete:
+        logger.debug("Removing obsolete dependencies: %s" % need_delete)
+        need_delete.delete()
+
 def set_layerbranch_collection_version(layerbranch, config_data, logger=None):
     layerbranch.collection = config_data.getVar('BBFILE_COLLECTIONS', True)
     ver_str = "LAYERVERSION_"
-- 
2.10.2

[PATCH V3 03/11] update_layer.py: set layerbranch's collection before add_dependencies

[Robert Yang]

The _add_dependency() uses:
if layerbranch.collection:
    var_name = layerbranch.collection

The layerbranch.collection is none if it is newly created, thus it can't get
LAYERDEPENDS, because what defined in layer.conf is LAYERDEPENDS_<collection>,
but what it would get is LAYERDEPENDS_<layer_name>, this patch can fix the
problem.

Reproducer:
$ python3 update_layer.py -l mete-xfce -b <newbranch> --fullreload -d

It would get None LAYERDEPENDS.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 layerindex/update_layer.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index 05ab3e7..bcf7056 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -360,9 +360,9 @@ def main():
                     logger.info("Skipping update of layer %s for branch %s - conf/layer.conf may have parse issues" % (layer.name, branchdesc))
                     layerconfparser.shutdown()
                     sys.exit(1)
+                utils.set_layerbranch_collection_version(layerbranch, layer_config_data, logger=logger)
                 utils.add_dependencies(layerbranch, layer_config_data, logger=logger)
                 utils.add_recommends(layerbranch, layer_config_data, logger=logger)
-                utils.set_layerbranch_collection_version(layerbranch, layer_config_data, logger=logger)
                 layerbranch.save()
 
                 try:
-- 
2.10.2

[PATCH V3 04/11] update.py: fix -l option

[Robert Yang]

Fixed:
$ ./update.py -l meta-oe,not_existed

It should report error that "not_existed" doesn't exist, but it didn't, this was
because __in matches any of them.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 layerindex/update.py | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/layerindex/update.py b/layerindex/update.py
index d5c56cd..54b9f87 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -152,10 +152,13 @@ def main():
         sys.exit(1)
 
     if options.layers:
-        layerquery = LayerItem.objects.filter(classic=False).filter(name__in=options.layers.split(','))
-        if layerquery.count() == 0:
-            logger.error('No layers matching specified query "%s"' % options.layers)
-            sys.exit(1)
+        layers = options.layers.split(',')
+        for layer in layers:
+            layerquery = LayerItem.objects.filter(classic=False).filter(name=layer)
+            if layerquery.count() == 0:
+                logger.error('No layers matching specified query "%s"' % layer)
+                sys.exit(1)
+        layerquery = LayerItem.objects.filter(classic=False).filter(name__in=layers)
     else:
         # We deliberately exclude status == 'X' ("no update") here
         layerquery = LayerItem.objects.filter(classic=False).filter(status='P')
-- 
2.10.2

[PATCH V3 05/11] recipeparse.py: restore cwd when the parsing is done

[Robert Yang]

Othewise it may cause troubles to the function who calls it.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 layerindex/recipeparse.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/layerindex/recipeparse.py b/layerindex/recipeparse.py
index dd85bc3..f2a5235 100644
--- a/layerindex/recipeparse.py
+++ b/layerindex/recipeparse.py
@@ -70,10 +70,13 @@ def init_parser(settings, branch, bitbakepath, enable_tracking=False, nocheckout
     if not os.path.exists(settings.TEMP_BASE_DIR):
         os.makedirs(settings.TEMP_BASE_DIR)
     tempdir = tempfile.mkdtemp(dir=settings.TEMP_BASE_DIR)
+    saved_cwd = os.getcwd()
     os.chdir(tempdir)
 
     tinfoil = utils.setup_tinfoil(bitbakepath, enable_tracking)
 
+    os.chdir(saved_cwd)
+
     # Ensure TMPDIR exists (or insane.bbclass will blow up trying to write to the QA log)
     oe_tmpdir = tinfoil.config_data.getVar('TMPDIR', True)
     if not os.path.exists(oe_tmpdir):
-- 
2.10.2

[PATCH V3 06/11] update.py: update layers orderly

[Robert Yang]

* Problems
The update.py couldn't handle new (not only new branch, in fact, but also
existing branches, see below for more info) branch well, for example, there are
3 layers: layer_A, layer_B and layer_C, and create new branch "branch_1" for
them, and they have depends:

layer_A -> layer_B -> layer_C

The "->" means depends on.

Then run "update.py -b branch_1", there would be errors like:

ERROR: Dependency layer_B of layer_A does not have branch record for branch branch_1

Though update.py runs "update_layer.py" twice, but it didn't help since
layerbranch was None when it was failed to create in the first run.

The reason is if update.py updates layer_A firstly, it would fail since it
can't find layer_B:branch_1 in database (not added to database yet), similarly,
if add layer_B before layer_C, it would also fail. Only layer_C can be added
(assume it has no dependencies). So we have to re-run update.py again and again
to make it work, here we may have to run update.py 3 times, and more runs are
needed if the dependency chain is longer.

* Solutions:
Make update.py pass layers to update_layer.py orderly can fix the problem, we
can get LAYERDEPENDS and LAYERRECOMMENDS info from tinfoil.

Not only new branch, but also existing branches may have the problem, because
BBFILE_COLLECTIONS maybe changed in the coming update, so we can't trust
database when the layer is going to be updated, for example, if there are 10
layers in database, and 3 of them will be updated (-l layer1,layer2,layer3),
then we can not use the 3 layers' collection data from database, we need get
them from info again, so the code doesn't check whether it is a new branch or
not.

* Performance Improvement:
  It should be faster than before in theory, since it ran update_layer.py
  twice in the past, but now only once, I have tested it with 76 layers:
  - Before: 4m25.912s, but only 30 layers were added, 46 ones were failed, I
    have to re-run update.py again and again (maybe 4 times to make all of
    them added). So:
    (4 * 60 + 25)/30*76/60 = 11.19m

  - Now 8m5.315s, all the layers are added in the first run.

  It improves from 11m to 8m.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 layerindex/update.py       | 116 +++++++++++++++++++++++++++++++++------------
 layerindex/update_layer.py |   3 +-
 layerindex/utils.py        |  39 +++++++++++++++
 3 files changed, 127 insertions(+), 31 deletions(-)

diff --git a/layerindex/update.py b/layerindex/update.py
index 54b9f87..45dcb86 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -18,6 +18,8 @@ import signal
 from datetime import datetime, timedelta
 from distutils.version import LooseVersion
 import utils
+import operator
+import recipeparse
 
 import warnings
 warnings.filterwarnings("ignore", category=DeprecationWarning)
@@ -151,6 +153,11 @@ def main():
         logger.error("Please set LAYER_FETCH_DIR in settings.py")
         sys.exit(1)
 
+    layerquery_all = LayerItem.objects.filter(classic=False).filter(status='P')
+    if layerquery_all.count() == 0:
+        logger.info("No published layers to update")
+        sys.exit(1)
+
     if options.layers:
         layers = options.layers.split(',')
         for layer in layers:
@@ -161,10 +168,7 @@ def main():
         layerquery = LayerItem.objects.filter(classic=False).filter(name__in=layers)
     else:
         # We deliberately exclude status == 'X' ("no update") here
-        layerquery = LayerItem.objects.filter(classic=False).filter(status='P')
-        if layerquery.count() == 0:
-            logger.info("No published layers to update")
-            sys.exit(1)
+        layerquery = layerquery_all
 
     if not os.path.exists(fetchdir):
         os.makedirs(fetchdir)
@@ -186,6 +190,7 @@ def main():
     try:
         lockfn = os.path.join(fetchdir, "layerindex.lock")
         lockfile = utils.lock_file(lockfn)
+        tinfoil = None
         if not lockfile:
             logger.error("Layer index lock timeout expired")
             sys.exit(1)
@@ -228,8 +233,85 @@ def main():
             # they never get used during normal operation).
             last_rev = {}
             for branch in branches:
+                # If layer_A depends(or recommends) on layer_B, add layer_B before layer_A
+                deps_dict_all = {}
+                layerquery_sorted = []
+                collections_done = set()
                 branchobj = utils.get_branch(branch)
+                try:
+                    utils.shutdown_tinfoil(tinfoil)
+                    (tinfoil, tempdir) = recipeparse.init_parser(settings, branchobj, bitbakepath, nocheckout=options.nocheckout, logger=logger)
+                except recipeparse.RecipeParseError as e:
+                    logger.error(str(e))
+                    sys.exit(1)
+                for layer in layerquery_all:
+                    # Get all collections from database, but we can't trust the
+                    # one which will be updated since its collections maybe
+                    # changed (different from database).
+                    if layer in layerquery:
+                        continue
+                    layerbranch = layer.get_layerbranch(branch)
+                    if layerbranch:
+                        collections_done.add((layerbranch.collection, layerbranch.version))
+
                 for layer in layerquery:
+                    errmsg = failedrepos.get(layer.vcs_url, '')
+                    if errmsg:
+                        continue
+                    config_data = utils.copy_tinfoil_data(tinfoil.config_data)
+                    layerbranch_source = layer.get_layerbranch(None)
+                    if not layerbranch_source:
+                        logger.error('Failed to get layerbranch_source for %s' % layer.name)
+                        sys.exit(1)
+                    urldir = layer.get_fetch_dir()
+                    repodir = os.path.join(fetchdir, urldir)
+                    layerdir = os.path.join(repodir, layerbranch_source.vcs_subdir)
+                    utils.parse_layer_conf(layerdir, config_data, logger=logger)
+
+                    deps = utils.get_layer_var(config_data, 'LAYERDEPENDS') or ''
+                    recs = utils.get_layer_var(config_data, 'LAYERRECOMMENDS') or ''
+                    col = (utils.get_layer_var(config_data, 'BBFILE_COLLECTIONS') or '').strip()
+                    ver = utils.get_layer_var(config_data, 'LAYERVERSION') or ''
+
+                    deps_dict = utils.explode_dep_versions2(deps + ' ' + recs)
+                    if len(deps_dict) == 0:
+                        # No depends, add it firstly
+                        layerquery_sorted.append(layer)
+                        collections_done.add((col, ver))
+                        continue
+                    deps_dict_all[layer] = {'requires': deps_dict, 'collection': col, 'version': ver}
+
+                # Move deps_dict_all to layerquery_sorted orderly
+                logger.info("Sorting layers for branch %s" % branch)
+                while True:
+                    deps_dict_all_copy = deps_dict_all.copy()
+                    for layer, value in deps_dict_all_copy.items():
+                        for req_col, req_ver_list in value['requires'].copy().items():
+                            matched = False
+                            if req_ver_list:
+                                req_ver = req_ver_list[0]
+                            else:
+                                req_ver = None
+                            if utils.is_deps_satisfied(req_col, req_ver, collections_done):
+                                del(value['requires'][req_col])
+                        if not value['requires']:
+                            # All the depends are in collections_done:
+                            del(deps_dict_all[layer])
+                            layerquery_sorted.append(layer)
+                            collections_done.add((value['collection'], value['version']))
+
+                    if not len(deps_dict_all):
+                        break
+
+                    # Something is wrong if nothing changed after a run
+                    if operator.eq(deps_dict_all_copy, deps_dict_all):
+                        logger.error("Cannot find required collections on branch %s:" % branch)
+                        for layer, value in deps_dict_all.items():
+                            logger.error('%s: %s' % (layer.name, value['requires']))
+                        logger.error("Known collections: %s" % collections_done)
+                        sys.exit(1)
+
+                for layer in layerquery_sorted:
                     layerupdate = LayerUpdate()
                     layerupdate.update = update
 
@@ -269,33 +351,9 @@ def main():
                     if ret == 254:
                         # Interrupted by user, break out of loop
                         break
-
-            # Since update_layer may not be called in the correct order to have the
-            # dependencies created before trying to link them, we now have to loop
-            # back through all the branches and layers and try to link in the
-            # dependencies that may have been missed.  Note that creating the
-            # dependencies is a best-effort and continues if they are not found.
-            for branch in branches:
-                branchobj = utils.get_branch(branch)
-                for layer in layerquery:
-                    layerbranch = layer.get_layerbranch(branch)
-                    if layerbranch:
-                        if not (options.reload or options.fullreload):
-                            # Skip layers that did not change.
-                            layer_last_rev = last_rev.get(layerbranch, None)
-                            if layer_last_rev is None or layer_last_rev == layerbranch.vcs_last_rev:
-                                continue
-
-                        logger.info('Updating layer dependencies for %s on branch %s' % (layer.name, branch))
-                        cmd = prepare_update_layer_command(options, branchobj, layer, updatedeps=True)
-                        logger.debug('Running update dependencies command: %s' % cmd)
-                        ret, output = run_command_interruptible(cmd)
-                        if ret == 254:
-                            # Interrupted by user, break out of loop
-                            break
-
         finally:
             utils.unlock_file(lockfile)
+            utils.shutdown_tinfoil(tinfoil)
 
     finally:
         update.log = ''.join(listhandler.read())
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index bcf7056..0373109 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -716,8 +716,7 @@ def main():
         import traceback
         traceback.print_exc()
     finally:
-        if LooseVersion(bb.__version__) > LooseVersion("1.27"):
-            tinfoil.shutdown()
+        utils.shutdown_tinfoil(tinfoil)
 
     shutil.rmtree(tempdir)
     sys.exit(0)
diff --git a/layerindex/utils.py b/layerindex/utils.py
index 1a57c07..7887bae 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -27,6 +27,33 @@ def get_layer(layername):
         return res[0]
     return None
 
+def get_layer_var(config_data, var):
+    collection = config_data.getVar('BBFILE_COLLECTIONS', True)
+    if collection:
+        collection = collection.strip()
+    value = config_data.getVar('%s_%s' % (var, collection), True)
+    if not value:
+        value = config_data.getVar(var, True)
+    return value
+
+def is_deps_satisfied(req_col, req_ver, collections):
+    """ Check whether required collection and version are in collections"""
+    for existed_col, existed_ver in collections:
+        if req_col == existed_col:
+            # If there is no version constraint, return True when collection matches
+            if not req_ver:
+                return True
+            else:
+                # If there is no version in the found layer, then don't use this layer.
+                if not existed_ver:
+                    continue
+                (op, dep_version) = req_ver.split()
+                success = bb.utils.vercmp_string_op(existed_ver, dep_version, op)
+                if success:
+                    return True
+    # Return False when not found
+    return False
+
 def get_dependency_layer(depname, version_str=None, logger=None):
     from layerindex.models import LayerItem, LayerBranch
 
@@ -156,6 +183,18 @@ def setup_tinfoil(bitbakepath, enable_tracking):
 
     return tinfoil
 
+def shutdown_tinfoil(tinfoil):
+    if tinfoil and hasattr(tinfoil, 'shutdown'):
+        tinfoil.shutdown()
+
+def copy_tinfoil_data(config_data):
+    import bb.data
+    return bb.data.createCopy(config_data)
+
+def explode_dep_versions2(deps):
+    import bb.utils
+    return bb.utils.explode_dep_versions2(deps)
+
 def checkout_layer_branch(layerbranch, repodir, logger=None):
 
     branchname = layerbranch.branch.name
-- 
2.10.2

[PATCH V3 07/11] update_layer.py: remove --update-dependencies

[Robert Yang]

It never works since it is in the middle of transaction.atomic() block, and
update.py doesn't need it any more, so remove it.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 layerindex/update.py       |  4 +---
 layerindex/update_layer.py | 27 ---------------------------
 2 files changed, 1 insertion(+), 30 deletions(-)

diff --git a/layerindex/update.py b/layerindex/update.py
index 45dcb86..42dd22b 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -64,15 +64,13 @@ def run_command_interruptible(cmd):
     return process.returncode, buf
 
 
-def prepare_update_layer_command(options, branch, layer, updatedeps=False):
+def prepare_update_layer_command(options, branch, layer):
     """Prepare the update_layer.py command line"""
     if branch.update_environment:
         cmdprefix = branch.update_environment.get_command()
     else:
         cmdprefix = 'python3'
     cmd = '%s update_layer.py -l %s -b %s' % (cmdprefix, layer.name, branch.name)
-    if updatedeps:
-        cmd += ' --update-dependencies'
     if options.reload:
         cmd += ' --reload'
     if options.fullreload:
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index 0373109..20c27be 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -182,9 +182,6 @@ def main():
     parser.add_option("-n", "--dry-run",
             help = "Don't write any data back to the database",
             action="store_true", dest="dryrun")
-    parser.add_option("", "--update-dependencies",
-            help = "Update layer dependencies only",
-            action="store_true", dest="updatedeps")
     parser.add_option("", "--nocheckout",
             help = "Don't check out branches",
             action="store_true", dest="nocheckout")
@@ -253,30 +250,6 @@ def main():
                     branchname = layerbranch.actual_branch
                     branchdesc = "%s (%s)" % (options.branch, branchname)
 
-            if options.updatedeps:
-                # Update layer dependencies only
-                if not layerbranch:
-                    logger.debug('Skipping dependency update for layer %s on branch %s - no layerbranch record' % (layer, branchdesc))
-                    sys.exit(0)
-                if not options.nocheckout:
-                    utils.checkout_layer_branch(layerbranch, repodir, logger=logger)
-                layerdir = os.path.join(repodir, layerbranch.vcs_subdir)
-                if not os.path.exists(layerdir):
-                    # If this happens it was already flagged during the main update, so ignore it
-                    logger.debug('Skipping dependency update for layer %s on branch %s - layer directory not found' % (layer, branchdesc))
-                    sys.exit(0)
-
-                layerconfparser = layerconfparse.LayerConfParse(logger=logger, bitbakepath=bitbakepath, tinfoil=tinfoil)
-                config_data = layerconfparser.parse_layer(layerdir)
-                if not config_data:
-                    logger.debug("Layer %s does not appear to be valid for branch %s" % (layer.name, branchdesc))
-                    sys.exit(0)
-
-                utils.add_dependencies(layerbranch, config_data, logger=logger)
-                utils.add_recommends(layerbranch, config_data, logger=logger)
-
-                sys.exit(0)
-
             # Collect repo info
             repo = git.Repo(repodir)
             assert repo.bare == False
-- 
2.10.2

[PATCH V3 08/11] update.py: add option -a to update actual branch

[Robert Yang]

Add an option "-a" to update actual branch for layer and bitbake, it is
useful when there are many layers and need update actual branches
frequantly. We only can update them via website without this patch,
which is not funny and easy to make mistakes.

* It works with "-l", and "-l bitbake" means update bitbake branch.
* It requires "-b" to work, and only one branch is supported in a run.

For example:
$ update.py -b master -a branch_20170526
All the layers which have branch master and actual_branch branch_20170526
will be updated to branch_20170526.

$ update.py -b master -l meta-oe -a branch_20170526
Only meta-oe layer will be updated.

$ update.py -b master -l bitbake -a branch_20170526
The bitbake's bitbake_branch will be updated.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 layerindex/update.py | 60 +++++++++++++++++++++++++++++++++++++++++++++++++++-
 layerindex/utils.py  | 11 ++++++++++
 2 files changed, 70 insertions(+), 1 deletion(-)

diff --git a/layerindex/update.py b/layerindex/update.py
index 42dd22b..f781d33 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -85,6 +85,43 @@ def prepare_update_layer_command(options, branch, layer):
         cmd += ' -q'
     return cmd
 
+def update_actual_branch(layerquery, fetchdir, branch, options, update_bitbake, bitbakepath):
+    """Update actual branch for layers and bitbake in database"""
+    to_save = set()
+    actual_branch = options.actual_branch
+    if update_bitbake:
+        branchobj = utils.get_branch(branch)
+        if actual_branch != branchobj.bitbake_branch:
+            if utils.is_branch_valid(bitbakepath, actual_branch):
+                logger.info("bitbake: %s.bitbake_branch: %s -> %s" % (branch, branchobj.bitbake_branch, actual_branch))
+                branchobj.bitbake_branch = actual_branch
+                to_save.add(branchobj)
+            else:
+                logger.info("Skipping update bitbake_branch for bitbake - branch %s doens't exist" % actual_branch)
+        else:
+            logger.info("bitbake: %s.bitbake_branch is already %s, so no change" % (branch, actual_branch))
+
+    for layer in layerquery:
+        urldir = layer.get_fetch_dir()
+        repodir = os.path.join(fetchdir, urldir)
+        if not utils.is_branch_valid(repodir, actual_branch):
+            logger.info("Skipping update actual_branch for %s - branch %s doens't exist" % (layer.name, actual_branch))
+            continue
+        layerbranch = layer.get_layerbranch(branch)
+        if not layerbranch:
+            logger.info("Skipping update actual_branch for %s - layerbranch %s doens't exist" % (layer.name, branch))
+            continue
+        if actual_branch != layerbranch.actual_branch:
+            logger.info("%s: %s.actual_branch: %s -> %s" % (layer.name, branch, layerbranch.actual_branch, actual_branch))
+            layerbranch.actual_branch = actual_branch
+            to_save.add(layerbranch)
+        else:
+            logger.info("%s: %s.actual_branch is already %s, so no change" % (layer.name, branch, actual_branch))
+
+    # At last, do the save
+    if not options.dryrun:
+        for s in to_save:
+            s.save()
 
 def main():
     if LooseVersion(git.__version__) < '0.3.1':
@@ -102,6 +139,9 @@ def main():
     parser.add_option("-l", "--layer",
             help = "Specify layers to update (use commas to separate multiple). Default is all published layers.",
             action="store", dest="layers")
+    parser.add_option("-a", "--actual-branch",
+            help = "Update actual branch for layer and bitbake",
+            action="store", dest="actual_branch", default='')
     parser.add_option("-r", "--reload",
             help = "Reload recipe data instead of updating since last update",
             action="store_true", dest="reload")
@@ -156,8 +196,13 @@ def main():
         logger.info("No published layers to update")
         sys.exit(1)
 
+    # For -a option to update bitbake branch
+    update_bitbake = False
     if options.layers:
         layers = options.layers.split(',')
+        if 'bitbake' in layers:
+            update_bitbake = True
+            layers.remove('bitbake')
         for layer in layers:
             layerquery = LayerItem.objects.filter(classic=False).filter(name=layer)
             if layerquery.count() == 0:
@@ -166,8 +211,17 @@ def main():
         layerquery = LayerItem.objects.filter(classic=False).filter(name__in=layers)
     else:
         # We deliberately exclude status == 'X' ("no update") here
+        update_bitbake = True
         layerquery = layerquery_all
 
+    if options.actual_branch:
+        if not options.branch:
+            logger.error("-a option requires -b")
+            sys.exit(1)
+        elif len(branches) != 1:
+            logger.error("Only one branch should be used with -a")
+            sys.exit(1)
+
     if not os.path.exists(fetchdir):
         os.makedirs(fetchdir)
     fetchedrepos = []
@@ -215,7 +269,7 @@ def main():
                             continue
                         fetchedrepos.append(layer.vcs_url)
 
-                if not fetchedrepos:
+                if not (fetchedrepos or update_bitbake):
                     logger.error("No repositories could be fetched, exiting")
                     sys.exit(1)
 
@@ -225,6 +279,10 @@ def main():
                 else:
                     out = utils.runcmd("git fetch", bitbakepath, logger=logger)
 
+            if options.actual_branch:
+                update_actual_branch(layerquery, fetchdir, branches[0], options, update_bitbake, bitbakepath)
+                return
+
             # Process and extract data from each layer
             # We now do this by calling out to a separate script; doing otherwise turned out to be
             # unreliable due to leaking memory (we're using bitbake internals in a manner in which
diff --git a/layerindex/utils.py b/layerindex/utils.py
index 7887bae..4bceae0 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -210,6 +210,17 @@ def is_layer_valid(layerdir):
         return False
     return True
 
+def is_branch_valid(layerdir, branch):
+    import git
+
+    g = git.cmd.Git(layerdir)
+    assert g.rev_parse('--is-bare-repository') == 'false'
+    try:
+        g.rev_parse('--verify', 'origin/%s' % branch)
+    except Exception:
+        return False
+    return True
+
 def parse_conf(conf_file, d):
     if hasattr(bb.parse, "handle"):
         # Newer BitBake
-- 
2.10.2

[PATCH V3 09/11] update_layer.py: move the location of transaction.atomic()

[Robert Yang]

It doesn't need to be so ahead since we only need it when writing database, and
a following patch will remove layerbranch from database when the branch had
been removed from the repo, it's not easy to do the work in
transaction.atomic() block.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 layerindex/update_layer.py | 56 +++++++++++++++++++++++-----------------------
 1 file changed, 28 insertions(+), 28 deletions(-)

diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index 20c27be..68ca836 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -235,36 +235,36 @@ def main():
     # why won't they just fix that?!)
     tinfoil.config_data.setVar('LICENSE', '')
 
+    layer = utils.get_layer(options.layer)
+    urldir = layer.get_fetch_dir()
+    repodir = os.path.join(fetchdir, urldir)
+
+    layerbranch = layer.get_layerbranch(options.branch)
+
+    branchname = options.branch
+    branchdesc = options.branch
+    if layerbranch:
+        if layerbranch.actual_branch:
+            branchname = layerbranch.actual_branch
+            branchdesc = "%s (%s)" % (options.branch, branchname)
+
+    # Collect repo info
+    repo = git.Repo(repodir)
+    assert repo.bare == False
     try:
-        with transaction.atomic():
-            layer = utils.get_layer(options.layer)
-            urldir = layer.get_fetch_dir()
-            repodir = os.path.join(fetchdir, urldir)
-
-            layerbranch = layer.get_layerbranch(options.branch)
-
-            branchname = options.branch
-            branchdesc = options.branch
-            if layerbranch:
-                if layerbranch.actual_branch:
-                    branchname = layerbranch.actual_branch
-                    branchdesc = "%s (%s)" % (options.branch, branchname)
-
-            # Collect repo info
-            repo = git.Repo(repodir)
-            assert repo.bare == False
-            try:
-                if options.nocheckout:
-                    topcommit = repo.commit('HEAD')
-                else:
-                    topcommit = repo.commit('origin/%s' % branchname)
-            except:
-                if layerbranch:
-                    logger.error("Failed update of layer %s - branch %s no longer exists" % (layer.name, branchdesc))
-                else:
-                    logger.info("Skipping update of layer %s - branch %s doesn't exist" % (layer.name, branchdesc))
-                sys.exit(1)
+        if options.nocheckout:
+            topcommit = repo.commit('HEAD')
+        else:
+            topcommit = repo.commit('origin/%s' % branchname)
+    except:
+        if layerbranch:
+            logger.error("Failed update of layer %s - branch %s no longer exists" % (layer.name, branchdesc))
+        else:
+            logger.info("Skipping update of layer %s - branch %s doesn't exist" % (layer.name, branchdesc))
+        sys.exit(1)
 
+    try:
+        with transaction.atomic():
             newbranch = False
             if not layerbranch:
                 # LayerBranch doesn't exist for this branch, create it
-- 
2.10.2

[PATCH V3 10/11] update.py: add -p to git fetch

[Robert Yang]

-p, --prune
    Before fetching, remove any remote-tracking references that no longer exist on the remote.

Fixed:
$ git push origin :test_branch
$ ./update.py

The test_branch was still in fetched local repo which was incorrect, it should
be gone since it has been removed by upstream.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 layerindex/update.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/layerindex/update.py b/layerindex/update.py
index f781d33..e2206f7 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -262,7 +262,7 @@ def main():
                             if not os.path.exists(repodir):
                                 out = utils.runcmd("git clone %s %s" % (layer.vcs_url, urldir), fetchdir, logger=logger, printerr=False)
                             else:
-                                out = utils.runcmd("git fetch", repodir, logger=logger, printerr=False)
+                                out = utils.runcmd("git fetch -p", repodir, logger=logger, printerr=False)
                         except subprocess.CalledProcessError as e:
                             logger.error("Fetch of layer %s failed: %s" % (layer.name, e.output))
                             failedrepos[layer.vcs_url] = e.output
@@ -277,7 +277,7 @@ def main():
                 if not os.path.exists(bitbakepath):
                     out = utils.runcmd("git clone %s %s" % (settings.BITBAKE_REPO_URL, 'bitbake'), fetchdir, logger=logger)
                 else:
-                    out = utils.runcmd("git fetch", bitbakepath, logger=logger)
+                    out = utils.runcmd("git fetch -p", bitbakepath, logger=logger)
 
             if options.actual_branch:
                 update_actual_branch(layerquery, fetchdir, branches[0], options, update_bitbake, bitbakepath)
-- 
2.10.2

[PATCH V3 11/11] update_layer.py: delete layerbranch for non-existed branch

[Robert Yang]

The branch is not needed any more when it has been removed from the repo, so we
also need remove its layerbranch, otherwise it still can be got from the web,
which causes confusions.

Note, we have to move the location of tinfiol's code to make it can be shutdown
correctly.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 layerindex/update_layer.py | 34 ++++++++++++++++++----------------
 1 file changed, 18 insertions(+), 16 deletions(-)

diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index 68ca836..34ed1e0 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -220,21 +220,6 @@ def main():
 
     bitbakepath = os.path.join(fetchdir, 'bitbake')
 
-    try:
-        (tinfoil, tempdir) = recipeparse.init_parser(settings, branch, bitbakepath, nocheckout=options.nocheckout, logger=logger)
-    except recipeparse.RecipeParseError as e:
-        logger.error(str(e))
-        sys.exit(1)
-    # Clear the default value of SUMMARY so that we can use DESCRIPTION instead if it hasn't been set
-    tinfoil.config_data.setVar('SUMMARY', '')
-    # Clear the default value of DESCRIPTION so that we can see where it's not set
-    tinfoil.config_data.setVar('DESCRIPTION', '')
-    # Clear the default value of HOMEPAGE ('unknown')
-    tinfoil.config_data.setVar('HOMEPAGE', '')
-    # Set a blank value for LICENSE so that it doesn't cause the parser to die (e.g. with meta-ti -
-    # why won't they just fix that?!)
-    tinfoil.config_data.setVar('LICENSE', '')
-
     layer = utils.get_layer(options.layer)
     urldir = layer.get_fetch_dir()
     repodir = os.path.join(fetchdir, urldir)
@@ -258,12 +243,29 @@ def main():
             topcommit = repo.commit('origin/%s' % branchname)
     except:
         if layerbranch:
-            logger.error("Failed update of layer %s - branch %s no longer exists" % (layer.name, branchdesc))
+            logger.info("layer %s - branch %s no longer exists, removing it from database" % (layer.name, branchdesc))
+            if not options.dryrun:
+                layerbranch.delete()
         else:
             logger.info("Skipping update of layer %s - branch %s doesn't exist" % (layer.name, branchdesc))
         sys.exit(1)
 
     try:
+        (tinfoil, tempdir) = recipeparse.init_parser(settings, branch, bitbakepath, nocheckout=options.nocheckout, logger=logger)
+    except recipeparse.RecipeParseError as e:
+        logger.error(str(e))
+        sys.exit(1)
+    # Clear the default value of SUMMARY so that we can use DESCRIPTION instead if it hasn't been set
+    tinfoil.config_data.setVar('SUMMARY', '')
+    # Clear the default value of DESCRIPTION so that we can see where it's not set
+    tinfoil.config_data.setVar('DESCRIPTION', '')
+    # Clear the default value of HOMEPAGE ('unknown')
+    tinfoil.config_data.setVar('HOMEPAGE', '')
+    # Set a blank value for LICENSE so that it doesn't cause the parser to die (e.g. with meta-ti -
+    # why won't they just fix that?!)
+    tinfoil.config_data.setVar('LICENSE', '')
+
+    try:
         with transaction.atomic():
             newbranch = False
             if not layerbranch:
-- 
2.10.2

Re: [PATCH V3 00/11] Several fixes for layerindex

[Robert Yang]

Hi Paul,

I updated the following patch a little:

[PATCH V3 06/11] update.py: update layers orderly

Checkout the branch before parse layer:

+if not options.nocheckout:
+    utils.checkout_layer_branch(layerbranch, repodir, logger=logger)

Updated in the repo:
   git://git.pokylinux.org/poky-contrib rbt/layerindex-8fixes
   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/layerindex-8fixes

Robert Yang (11):
   layerconfparse.py: remove unused layerbranch from parse_layer()
   utils.py: remove obsolete dependencies
   update_layer.py: set layerbranch's collection before add_dependencies
   update.py: fix -l option
   recipeparse.py: restore cwd when the parsing is done
   update.py: update layers orderly
   update_layer.py: remove --update-dependencies
   update.py: add option -a to update actual branch
   update_layer.py: move the location of transaction.atomic()
   update.py: add -p to git fetch
   update_layer.py: delete layerbranch for non-existed branch

// Robert

On 06/02/2017 02:28 PM, Robert Yang wrote:
> * V3:
>   - Better error messages when required collections not found
>   - Delete layerbranch for non-existed branch
>   - update_layer.py: remove --update-dependencies
>
> * V2:
>   - Remove "import bb" from update.py to avoid causing confusions when switch branches.
>
> * V1:
>   - The "-a ACTUAL_BRANCH" is a litle different from what we had talked, now it
>     respects "-l" and "-b", we can use -l to specify layers or default to all,
>     and it requires "-b" to work, only one branch is supported in a run, for example:
>
>     $ update.py -b master -a branch_20170526
>     All the layers which have branch master and actual_branch branch_20170526
>     will be updated to branch_20170526.
>
>     $ update.py -b master -l meta-oe -a branch_20170526
>     Only meta-oe layer will be updated.
>
>     $ update.py -b master -l bitbake -a branch_20170526
>     The bitbake's bitbake_branch will be updated.
>
> // Robert
>
> The following changes since commit 8dfe44ed5e618d517d4acb4a3107cb3907e773c8:
>
>   TODO: drop implemented items (2017-03-22 14:21:22 +1300)
>
> are available in the git repository at:
>
>   git://git.pokylinux.org/poky-contrib rbt/layerindex-8fixes
>   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/layerindex-8fixes
>
> Robert Yang (11):
>   layerconfparse.py: remove unused layerbranch from parse_layer()
>   utils.py: remove obsolete dependencies
>   update_layer.py: set layerbranch's collection before add_dependencies
>   update.py: fix -l option
>   recipeparse.py: restore cwd when the parsing is done
>   update.py: update layers orderly
>   update_layer.py: remove --update-dependencies
>   update.py: add option -a to update actual branch
>   update_layer.py: move the location of transaction.atomic()
>   update.py: add -p to git fetch
>   update_layer.py: delete layerbranch for non-existed branch
>
>  layerindex/layerconfparse.py     |   2 +-
>  layerindex/recipeparse.py        |   3 +
>  layerindex/tools/import_layer.py |   2 +-
>  layerindex/update.py             | 193 +++++++++++++++++++++++++++++++--------
>  layerindex/update_layer.py       |  92 +++++++------------
>  layerindex/utils.py              |  63 ++++++++++++-
>  6 files changed, 255 insertions(+), 100 deletions(-)
>